PEP: 718
Title: Subscriptable functions
Author: James Hilton-Balfe <gobot1234yt@gmail.com>, Pablo Ruiz Cuevas <pablo.r.c@live.com>
Sponsor: Guido van Rossum <guido@python.org>
Discussions-To: https://discuss.python.org/t/28457/
Status: Draft
Type: Standards Track
Topic: Typing
Created: 23-Jun-2023
Python-Version: 3.15
Post-History: `24-Jun-2023 <https://discuss.python.org/t/28457/>`__

Abstract
--------

This PEP proposes making function objects subscriptable for typing purposes. Doing so
gives developers explicit control over the types produced by the type checker where
bi-directional inference (which allows for the types of parameters of anonymous
functions to be inferred) and other methods than specialisation are insufficient. It
also makes functions consistent with regular classes in their ability to be
subscripted.

Motivation
----------

Currently, classes allow passing type annotations for generic containers. This
is especially useful in common constructors such as ``list``\, ``tuple`` and ``dict``
etc.

.. code-block:: python

   my_integer_list = list[int]()
   reveal_type(my_integer_list)  # type is list[int]

At runtime ``list[int]`` returns a ``GenericAlias`` that can be later called, returning
an empty list.

Another example of this is creating a specialised ``dict`` type for a section of our
code where we want to ensure that keys are ``str`` and values are ``int``:

.. code-block:: python

   NameNumberDict = dict[str, int]

   NameNumberDict(
      one=1,
      two=2,
      three="3"  # Invalid: Literal["3"] is not of type int
   )

In spite of the utility of this syntax, when trying to use it with a function, an error
is raised, as functions are not subscriptable.

.. code-block:: python

   def my_list[T](arr: Iterable[T]) -> list[T]:
      # do something...
      return list(arr)

   my_integer_list = my_list[int]()  # TypeError: 'function' object is not subscriptable

There are a few workarounds:

1. Making a callable class:

.. code-block:: python

   class my_list[T]:
        def __call__(self, arr: Iterable[T]) -> list[T]:
            # do something...
            return list(arr)

   my_string_list = my_list[str]([])

2. Using :pep:`747`\'s TypeForm, with an extra unused argument:

.. code-block:: python

   from typing import TypeForm

   def my_list(*arr: Iterable[T], typ: TypeForm[T]) -> list[T]:
      # do something...
      return list(arr)

   my_string_list = my_list([], str)

As we can see this solution increases the complexity with an extra argument.
Additionally it requires the user to understand a new concept ``TypeForm``.

3. Annotating the assignment:

.. code-block:: python

   my_integer_list: list[int] = my_list()

This solution isn't optimal as the return type is repeated, is more verbose and would
require the type updating in multiple places if the return type changes. Additionally,
it adds unnecesary and distracting verbossity when the intention is to pass the
specialized value into another call.

In conclusion, the current workarounds are too complex or verbose, especially compared
to syntax that is consistent with the rest of the language.

Generic Specialisation
^^^^^^^^^^^^^^^^^^^^^^

As in the previous example currently we can create generic aliases for different
specialised usages:

.. code-block:: python

   NameNumberDict = dict[str, int]
   NameNumberDict(one=1, two=2, three="3")  # Invalid: Literal["3"] is not of type int``

This not currently possible for functions but if allowed we could easily
specialise operations in certain sections of the codebase:

.. code-block:: python

   def constrained_addition[T](a: T, b: T) -> T: ...

   # where we work exclusively with ints
   int_addition = constrained_addition[int]
   int_addition(2, 4+8j)  # Invalid: complex is not of type int

Unknown Types
^^^^^^^^^^^^^

Currently, it is not possible to infer the type parameters to generic functions in
certain situations.

In this example ``T`` cannot currently be meaningfully inferred, so ``x`` is
untyped without an extra assignment:

.. code-block:: python

   def factory[T](func: Callable[[T], Any]) -> Foo[T]: ...

   reveal_type(factory(lambda x: "Hello World" * x)) # type is Foo[Unknown]

If function objects were subscriptable, however, a more specific type could be given:

.. code-block:: python

   reveal_type(factory[int](lambda x: "Hello World" * x))  # type is Foo[int]

Unsolvable Type Parameters
^^^^^^^^^^^^^^^^^^^^^^^^^^

Currently, with unspecialised literals, it is not possible to determine a type for
situations similar to:

.. code-block:: python

   def foo[T](x: list[T]) -> T: ...
   reveal_type(foo([]))  # type checker cannot infer T (yet again)

.. code-block:: python

   reveal_type(foo[int]([]))  # type is int

It is also useful to be able to specify in cases in which a certain type must be passed
to a function beforehand:

.. code-block:: python

   words = ["hello", "world"]
   foo[int](words)  # Invalid: list[str] is incompatible with list[int]

Allowing subscription makes functions and methods consistent with generic classes where
they weren't already. Whilst all of the proposed changes can be implemented using
callable generic classes, syntactic sugar would be highly welcome.

Due to this, specialising the function and using it as a new factory is fine

.. code-block:: python

   make_int_list = make_list[int]
   reveal_type(make_int_list())  # type is list[int]

Monomorphisation and Reification
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

This proposal also opens the door to
`monomorphisation <https://en.wikipedia.org/wiki/Monomorphization>`_ and
`reified types <https://en.wikipedia.org/wiki/Reification_(computer_science)>`_.

This would allow for a functionality which anecdotally has been requested many times.

*Please note this feature is not being proposed by the PEP, but may be implemented in
the future.*

The syntax for such a feature may look something like:

.. code-block:: python

   def foo[T]():
      return T.__value__

   assert foo[int]() is int

Rationale
---------

This proposal improves the consistency of the type system, by allowing syntax that
already looks and feels like a natural of the existing syntax for classes.

If accepted, this syntax will reduce the necessity to learn about :pep:`747`\s
``TypeForm``,  reduce verbosity and cognitive load of safely typed python.

Specification
-------------

In this PEP "Function objects" is used to refer to ``FunctionType``\ , ``MethodType``\ ,
``BuiltinFunctionType``\ , ``BuiltinMethodType`` and ``MethodWrapperType``\ .

For ``MethodType`` you should be able to write:

.. code-block:: python

   class Foo:
       def make_list[T](self, *args: T) -> list[T]: ...

   Foo().make_list[int]()

and have it work similarly to a ``FunctionType``.

For ``BuiltinFunctionType``, so builtin generic functions (e.g. ``max`` and ``min``)
work like ones defined in Python. Built-in functions should behave as much like
functions implemented in Python as possible.

``BuiltinMethodType`` is the same type as ``BuiltinFunctionType``.

``MethodWrapperType`` (e.g. the type of ``object().__str__``) is useful for
generic magic methods.

Function objects should implement ``__getitem__`` to allow for subscription at runtime
and return an instance of ``types.GenericAlias`` with ``__origin__`` set as the
callable and ``__args__`` as the types passed.

Type checkers should support subscripting functions and understand that the parameters
passed to the function subscription should follow the same rules as a generic callable
class.

Setting ``__orig_class__``
^^^^^^^^^^^^^^^^^^^^^^^^^^

Currently, ``__orig_class__`` is an attribute set in ``GenericAlias.__call__`` to the
instance of the ``GenericAlias`` that created the called class e.g.

.. code-block:: python

   class Foo[T]: ...

   assert Foo[int]().__orig_class__ == Foo[int]

Currently, ``__orig_class__`` is unconditionally set; however, to avoid potential
erasure on any created instances, this attribute should not be set if ``__origin__`` is
an instance of any function object.

The following code snippet would fail at runtime without this change as
``__orig_class__`` would be ``bar[str]`` and not ``Foo[int]``.

.. code-block:: python

   def bar[U]():
       return Foo[int]()

   assert bar[str]().__orig_class__  == Foo[int]

Interactions with ``@typing.overload``
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

This PEP allows type checkers to do overloading based on type variables:

.. code-block:: python

   @overload
   def serializer_for[T: str]() -> StringSerializer: ...
   @overload
   def serializer_for[T: list]() -> ListSerializer: ...

   def serializer_for():
       ...

For overload resolution a new step will be required previous to any other, where the resolver
will match only the overloads where the subscription may succeed.

.. code-block:: python

   @overload
   def make[*Ts]() -> float: ...
   @overload
   def make[T]() -> int: ...

   make[int]  # matches first and second overload
   make[int, str]  # matches only first


Functions Parameterized by ``TypeVarTuple``\ s
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Currently, type checkers disallow the use of multiple ``TypeVarTuple``\s in their
generic parameters; however, it is currently valid to have a function as such:

.. code-block:: python

   def foo[*T, *U](bar: Bar[*T], baz: Baz[*U]): ...
   def spam[*T](bar: Bar[*T]): ...

This PEP does not allow functions like ``foo`` to be subscripted, for the same reason
as defined in :pep:`PEP 646<646#multiple-type-variable-tuples-not-allowed>`, the type
variables cannot be resolved unambiguously with the current syntax.

.. code-block:: python

   foo[int, str, bool, complex](Bar(), Baz())  # Invalid: cannot determine which parameters are passed to *T and *U. Explicitly parameterise the instances individually
   spam[int, str, bool, complex](Bar())        # OK

Binding Rules
^^^^^^^^^^^^^
Method subscription (including ``classmethods`` and ``staticmethods``), should only
allow their function's type parameters and not the enclosing class's.
Subscription should follow the rules specified in :pep:`PEP 696<696#binding-rules>`;
methods should bind type parameters on attribute access.

.. code-block:: python

   class C[T]:
       def method[U](self, x: T, y: U): ...
       @classmethod
       def cls[U](cls, x: T, y: U): ...

   C[int].method[str](0, "")  # OK
   C[int].cls[str](0, "")     # OK
   C.cls[int, str](0, "")     # Invalid: too many type parameters
   C.cls[str](0, "")          # OK, U will be matched to str

Backwards Compatibility
-----------------------
Currently these classes are not subclassable and so there are no backwards
compatibility concerns with regards to classes already implementing
``__getitem__``.

Reference Implementation
------------------------

The runtime changes proposed can be found here
https://github.com/Gobot1234/cpython/tree/function-subscript

Acknowledgements
----------------

Thank you to Alex Waygood and Jelle Zijlstra for their feedback on this PEP and Guido
for some motivating examples.

Copyright
---------

This document is placed in the public domain or under the CC0-1.0-Universal license,
whichever is more permissive.
