ParamSpec only matches the first overload

[mvanderlee]

Playground

I see 2 issues here:

1. The sync function myfn incorrectly reports an overlapping overload, interestingly the async function does not.
2. task.execute and aio_task.execute only allow the first overload value.

from collections.abc import Awaitable, Callable
from dataclasses import dataclass
from typing import overload, Literal, TypeVar, Generic, ParamSpec, reveal_type

P = ParamSpec("P")
ReturnType = TypeVar("ReturnType", bound=int | str)


# Sync
@overload
def myfn(
    prompt: str,
    is_string: Literal[True] = True,
) -> str: ...


@overload
def myfn(
    prompt: str,
    is_string: Literal[False] = False,
) -> int: ...


def myfn(
    prompt: str,
    is_string: bool = True,
) -> str | int:
    return "foo" if is_string else 4


# Async
@overload
async def aio_myfn(
    prompt: str,
    is_string: Literal[True] = True,
) -> str: ...


@overload
async def aio_myfn(
    prompt: str,
    is_string: Literal[False] = False,
) -> int: ...


async def aio_myfn(
    prompt: str,
    is_string: bool = True,
) -> str | int:
    return "foo" if is_string else 4


@dataclass
class TaskWithCache(Generic[P, ReturnType]):
    fn: Callable[P, ReturnType]

    def execute(self, *args: P.args, **kwargs: P.kwargs) -> ReturnType:
        return self.fn(*args, **kwargs)


@dataclass
class AIOTaskWithCache(Generic[P, ReturnType]):
    fn: Callable[P, Awaitable[ReturnType]]

    async def execute(self, *args: P.args, **kwargs: P.kwargs) -> ReturnType:
        return await self.fn(*args, **kwargs)


async def execute():
    task = TaskWithCache(myfn)
    task.execute("prompt", True)
    task.execute("prompt", False)


async def aio_execute():
    aio_task = AIOTaskWithCache(aio_myfn)
    await aio_task.execute("prompt", True)
    await aio_task.execute("prompt", False)

[Eric Traut (erictraut)]

When a ParamSpec is used to capture the input signature of a function, the captured input signature is "detached" from its return type. An overloaded function is effectively a mapping between input signatures and return types, so it's not clear what it would mean for a ParamSpec to capture an overload — or if doing so even makes sense. I could make an argument that this should produce an error, but I could also argue that the ParamSpec should capture all of the overloaded input signatures or capture only the first. PEP 612 (which introduced ParamSpec to the type system) and the typing spec are silent on this matter.

Today's type checkers (including pyright and mypy) capture only the first overloaded signature, as you see above. This is the most straightforward approach, from an implementation standpoint, which is probably why it's the behavior that everyone landed on.

If you'd like to see a different behavior standardized here, my recommendation is that you propose an update to the typing spec. There is a well-defined process for doing this, which you can find here.

[mvanderlee]

I'll look into the process.
I would expect type checker to use the implementation itself, or all overloads. But taking only one overload leads to type errors.

And item 1, is that a bug or am I doing something wrong?

[Eric Traut (erictraut)]

Item 1 isn't a bug. The problem here is that you've supplied two overloads that both accept a default argument for the is_string parameter. If you call the overload without providing a value for this parameter (as in myfn("")), then both overloads apply, but they return incompatible types. To fix this, simply remove the default argument from the second overload (i.e. delete the = False).

The fact that the async overload doesn't produce an "overlapping overload" error does appear to be a bug — or at least an omitted check. If I delete the async keyword and modify the return type from str to CoroutineType[Any, Any, str], then I do see an "overlapping overload" error. If you'd like me to look into that more, feel free to file a bug report.

[mvanderlee]

I disagree, but opened a bug report here: #10759

[Eric Traut (erictraut)]

I saw your comments in the bug report (thanks for filing it!). To clarify, you should delete only one of the default arguments, not both. The one you retain should reflect the reality of your function's implementation. If the default argument for parameter is_string is True, then you should include = True in overload 1 and omit = False in overload 2. If the default argument for parameter is_string is False, then you should include = False in overload 2 and omit the = True in overload 1.

If you are not interested in pyright reporting unsound overlapping overloads, you can disable reportOverlappingOverload check. You will still get correct behavior in most cases, but you may see inconsistent or incorrect behavior in certain edge cases. That's what the reportOverlappingOverload check is detecting.