Mapping to abs doesn't work

[YtvwlD]

I first encountered this issue with Python 3.6.7 and mypy 0.560 from my distribution's repositories, but it also occurs on Python 3.7 and mypy 0.701, and the current master.

This is the code in question:

map(abs, [1])

It works, of course. But running mypy on it raises an error:

test.py:2: error: Argument 1 to "map" has incompatible type "Callable[[SupportsAbs[_T]], _T]"; expected "Callable[[int], _T]"

If I change it to the following code, there are no errors:

map(lambda x: abs(x), [1])

(This is a minimal example, my original code was something like sum(map(abs, list_of_floats)).)

So, what's the problem here? int and float have .__abs__.

[ilevkivskyi]

This looks like a type inference bug. It is hard to tell what exactly went wrong, but mypy complains because it wanted SupportsAbs[int], not SupportsAbs[_T].

[ilevkivskyi]

This happened again in #6703, and looks like a fundamental issue, so raising priority to high.

[sambhav]

I was able to recreate it with something more generic -

from typing import TypeVar, Callable

T = TypeVar("T")
S = TypeVar("S")

def f(x: T) -> T:
    return x

def g(f: Callable[[T], S], x: T) -> S:
    return f(x)

g(f, 1)

Output:
error: Argument 1 to "g" has incompatible type "Callable[[T], T]"; expected "Callable[[int], T]"

[ilevkivskyi]

I was able to recreate it with something more generic

Yes, this is a nice repro.

[ilevkivskyi]

For future reference, there is a similar example in #4584

This is essentially the same as #1317, but I would like to keep this open as an important use case.

[RemyLau]

Has this been fixed yet? I tried to follow the linked issues but I can't find an obvious answer. I'm running in a similar issue where I'm trying to map set over a list of lists:

map(set, mylist)

And similar to the original issue here, it complains about the first argument to map having an incompatible type, which can be similarly resolved by:

map(lambda x: set(x), list_of_lists)

Is there any practical workaround at this point without having to convert to lambda (that just doesn't make much sense)?

[randolf-scholz]

Just encountered this using mypy 1.4 with

def f(t: tuple[int, ...]) -> int:
    return max(map(abs, t), default=0)

https://mypy-play.net/?mypy=latest&python=3.11

[randolf-scholz]

I think I got something: It seems mypy fails to substitute the return type:

reveal_type(map(abs, [1,2,3]))   # builtins.map[_T`-1], not builtins.map[int] !

If we look at the definition of map:

def __init__(self, __func: Callable[[_T1], _S], __iter1: Iterable[_T1]) -> None: ...

and in our case, __func receives type Callable[[SupportsAbs[_T]], _T]. It seems that the issue is that mypy fails to propagate the int type forward, as we can see it does work if we manually type hint it as an Callable[[SupportsAbs[int]], int] function:

abs_hinted: Callable[[SupportsAbs[int]], int] = abs  # ✔
map(abs_hinted, [1,2,3])  # ✔

I created a mypy-playground with a minimal reproduction.

definitions of the relevant builtins

Definition of abs

def abs(__x: SupportsAbs[_T]) -> _T: ...

Definition of SupportsAbs

@runtime_checkable
class SupportsAbs(Protocol[_T_co]):
    @abstractmethod
    def __abs__(self) -> _T_co: ...

Definition of map

def __init__(self, __func: Callable[[_T1], _S], __iter1: Iterable[_T1]) -> None: ...

[randolf-scholz]

I don't know much about the internals of mypy, but when a Callable[[T], S] is expected and a Callable[[T], T] is received, then a symbolic substitution must take place, since the input yields a constraint T = S, which it seems is not happening.

[ilevkivskyi]

I just tried examples in this issue and they all seem to work on master if I use --new-type-inference, so I am going to close this. Note this flag is not public yet (because it is actually still WIP), it will be public in the next release (and in release after that it will be on by default).

[randolf-scholz]

@ilevkivskyi Does this new inference solve issues like the following?

from typing import Callable, TypeVar, assert_type

T = TypeVar("T")

identity: Callable[[T], T] = lambda x: x
assert_type(identity(0), int)  # ✔

def foo(func: Callable[[T], T]) -> None:
    assert_type(func(0), int)  # ✘ incompatible type "int"; expected "T"

https://mypy-play.net/?mypy=master&python=3.11&gist=5e696ae12fe41e704d5df7d70ca187c2

[ilevkivskyi]

@randolf-scholz No, because there is no issue. When you define foo like this, it becomes generic itself, not its argument func. If you want to define a non-generic function that expects a generic function as an argument you will need to specify type variable binding explicitly, for example by using a callback protocol:

class Identity(Protocol):
    def __call__(self, __x: T) -> T: ...

def foo(func: Identity) -> None:
    assert_type(func(0), int)  # OK

[randolf-scholz]

@ilevkivskyi Oh wow, I tried this earlier and it failed, seems I wrote Protocol[T] instead of Protocol. I guess this leads to the same issue you mentioned here.