Polymorphic value annotation

[decorator-factory]

Is there any way to declare a polymorphic value type, not function type? For example, if I have a function like this:

def f() -> Callable[[A, A], A]:
    if random.random() > 0.5:
        return lambda x, y: x
    else:
        return lambda x, y: y

I can then call f and store the return value:

g = f()
x = g(4, 6)

g's type is inferred to be Callable[[A, A], A], and x's value is inferred as Literal[4, 6].
However, I cannot create this annotation "from scratch":

h: Callable[[A, A], A] = lambda x, y: x # error
x = h(4, 6)  # x: A

I guess I could do it with a construction like this:

def _h() -> Callable[[A, A], A]:
    return lambda x, y: x
h = _h()

but is there a way without a function?

The use I have in mind for it is a container that stores a polymorphic function. Example:

@dataclass
class Lens(Generic[S, T, A, B]):
    getter: Callable[[S], A]
    setter: Callable[[S, B], T]

first: Lens[tuple[A, B], tuple[C, B], A, C] =\
    Lens( 
        lambda ab: ab[0],
        lambda ab, c: (a, ab[1])
    )

first.getter((1, "a"))  # 1
first.setter((1, "a"), "!") # ("!", "a")

Here I can't say that first has type Lens[tuple[Any, Any], tuple[Any, Any], Any, Any], because there's a clear relation in it (given a tuple (A, C), first can extract the first element A or replace the first element with B, yielding (B, C))

[Eric Traut (erictraut)]

Python's support for lambdas is pretty limited. Lambdas provide no way to bind to a symbol name, annotate parameter types, serve as a scope for type variables, define overloads, apply decorators, use async functionality, etc. A lambda is really just a syntactic shortcut for extremely simple functions that do not require any of these more advanced pieces of functionality. In cases where you need any of this added functionality, you should use a function rather than a lambda.

[decorator-factory]

Hm, upon further investigation, I found some strange behaviour. This works:

h = make_h()
# h inferred as: Callable[[A, A], A]

one = h(1, 2)
# one: Literal[1, 2]

But:

def make_x() -> FunctionPair[tuple[A, str], A]:
    return FunctionPair(first_element, first_element)
x = make_x()
# x inferred as: FunctionPair[tuple[A, str], A]

y = x.f((1, "x"))
# Argument of type "tuple[Literal[1], Literal['x']]" cannot be assigned to parameter of type "tuple[A@first_element, str]"
#  Tuple entry 1 is incorrect type
#    Type "Literal[1]" cannot be assigned to type "A@first_element"

I expected that this would typecheck, and that y would be inferred as Literal[1] or int.

[Eric Traut (erictraut)]

No, this isn't allowed by PEP 484. It's very clear in saying that type variables cannot appear in contexts where they are not bound to a scope. The use of A in the statement h: Callable[[A, A], A] = ... is an illegal use of a type variable.

You can apply a type annotation on x to make the generic FunctionPair instance concrete when it is constructed, like this:

x: FunctionPair[tuple[int, str], int] = FunctionPair(first_element, first_element)

You can also use a type variable in this annotation if it is bound to a valid TypeVar context, such as a class, like this:

class ClassA(Generic[A]):
    x: FunctionPair[tuple[A, str], A] = FunctionPair(first_element, first_element)

You can also use a generic callable protocol, like this:

class MyCallable(Protocol[A]):
    def __call__(self, v1: A, v2: A) -> A:
        ...

Not sure if any of these ideas help you out. I still don't fully understand what you're trying to do. Your examples are using a very strange design pattern, and I'm guessing that there's a more straightforward solution to achieve your goal.

[decorator-factory]

It's pretty disappointing that it's impossible to annotate that
I found that that's also an issue with typescript, hopefully it explains what I want more clearly: microsoft/TypeScript#17574

[Eric Traut (erictraut)]

Keep in mind also that TypeVars are "solved" at the time you call a method / constructor. "Solving" a TypeVar refers to the process by which all the TypeVars are replaced with types that satisfy all of the constraints in the function signature. Once a function / constructor has been called, any TypeVars whose values were not solved are left as TypeVars, but they can no longer be "solved".

In your example above, make_x returns a FunctionPair instance that has an unresolved TypeVar (A@make_x). A@make_x cannot be "solved" after the call to make_x because make_x is the context in which that TypeVar is defined. That's why you're seeing an error for the expression x.f((1, "x")).

[decorator-factory]

Right, but we also have polymorphic functions. So you can truly have a value that has "unfilled" type variables, like Callable[[A, A], A], which is a function like this:

def f(x: A, y: A) -> A:
    ...

and then you can specialize f in other context, for example, the result of f(1, 2) is Literal[1, 2]

So my issue is that you can't do the same with something other than a function. The example given in the TypeScript issue (microsoft/TypeScript#17574) is a lens (as a coincidence, that's exactly what I'm writing), which is something along these lines:

Basically, it allows you to read from and change nested immutable values. Sort of like x[0][1][0][0] = 5. Source: https://gist.github.com/decorator-factory/3883edbadd5bbe240b2c852b44aa7a79

In order to build this object, we need two functions: a "getter" and a "setter".
Let's create a type Lens[S, T, A, B] that will have a getter: (S -> A) and a setter : B -> (S -> T). In other words, Lens[S, T, A, B] can translate a change of A into B into a change from S into T.

For example, you can have a lens that turns the first element of an int pair to a string: Lens[tuple[int, int], tuple[str, int], int, str].
You could define it as lens = Lens(getter=lambda ii: i[0], setter=lambda s: lambda ii: (s, ii[1])), and use it like this:

>>> lens.getter((1, 2))
1
>>> lens.setter("x")((1, 2))
('x', 2)

Let's now write a lens that changes the first element of a tuple.

def first_getter(ab: tuple[A, Any]) -> A:
    return ab[0]
    
def first_setter(c: C) -> Fn[tuple[Any, B], tuple[C, B]]:
    return lambda ab: (c, ab[1])

first = Lens(getter=first_getter, setter=first_setter)

It works:

>>> first.getter(((1, 2), 3))
(1, 2)
>>> (first * first).getter(((1, 2), 3))
1
>>> first.setter("4")((1, 2))
('4', 2)

What's the type of first? Well, if I have a pair with any types A and B, I can get an A, or I can provide some C and end up with a pair of C and B. In the example execution on the picture, first() takes on different types in different contexts. For example, Lens[tuple[int, int], tuple[str, int], int, str] (in In[]) and Lens[tuple[tuple[int, int], int], tuple[tuple[int, int], int], int, int].
So we could say that its type is Lens[tuple[A, B], tuple[C, B], A, C].

But because I can't have a value with unfilled type variables, I can't assign an accurate type to first. So alas, first has to be assigned a type of Lens[tuple[Any, Any], tuple[Any, Any], Any, Any].

Of course you can fix this with protocols:

class LensLike(Protocol[S, T, A, B]):
    def getter(self, s: S) -> A: ...
    def setter(self, b: B) -> Fn[S, T]: ...
    def __mul__(self, other: LensLike[A, B, X, Y]) -> LensLike[S, T, X, Y]: ...

class FirstLens(LensLike[tuple[A, B], tuple[C, B], A, C], Protocol[A, B, C]):
    def getter(self, a: tuple[A, Any]) -> A: ...
    def setter(self, c: C) -> Fn[tuple[Any, A], tuple[C, A]]: ...
    def __mul__(self, other: LensLike[A, C, X, Y]) -> LensLike[tuple[A, B], tuple[C, B], X, Y]: ...

def first2(at: type[A], bt: type[B], ct: type[C]) -> FirstLens[A, B, C]:
    return first  # type: ignore

lens = first2(tuple[str, int], int, tuple[int, int]) * first2(str, int, int)

x = lens.getter((("a", 1), 2))     # x: str
y = lens.setter(4)((("a", 1), 2))  # y: tuple[tuple[str, int], int]

If polymorphic values besides functions were allowed, then the above snippet would collapse to:

# first : Lens[tuple[A, B], tuple[C, B], A, C]
x = (first * first).getter((("a", 1), 2))
y = (first * first).setter(4)((("a", 1), 2))

with the same results. The Protocol approach will require writing a protocol like that for every polymorphic lens; and the amount of type information you have to supply for this to work makes lenses useless.

Hopefully this example explains why there isn't always an easy replacement using other tools. I do understand that this is a very niche idea in Python, so it probably won't get accepted anywhere, but just to provide the context of why I asked the question.