PEP: 743
Title: Add Py_OMIT_LEGACY_API to the Python C API
Author: Victor Stinner <vstinner@python.org>,
        Petr Viktorin <encukou@gmail.com>,
Discussions-To: https://discuss.python.org/t/59323
Status: Rejected
Type: Standards Track
Created: 11-Mar-2024
Python-Version: 3.15
Post-History: `11-Mar-2024 <https://discuss.python.org/t/48243>`__,
              `27-Jul-2024 <https://discuss.python.org/t/59323>`__,
Resolution: `20-Feb-2026 <https://discuss.python.org/t/59323/20>`__

.. highlight:: c


Rejection Notice
================

The Steering Council has rejected this PEP in its current form:

   While we agree that the problem the PEP tries to solve is one worth solving,
   we’re not convinced this is the correct approach.

See `the full post <https://discuss.python.org/t/59323/20>`_.


Abstract
========

Add ``Py_OMIT_LEGACY_API`` C macro that hides deprecated and
soft-deprecated symbols, allowing users to opt out of using API with known
issues that other API solves.

Also, add namespaced alternatives for API without the ``Py_`` prefix,
and soft-deprecate the original names.


Motivation
==========

Some of Python's C API has flaws that are only obvious in hindsight.

If an API prevents adding features or optimizations, or presents a serious
security risk or maintenance burden, we can deprecate and remove it as
described in :pep:`387`.

However, this leaves us with some API that has “sharp edges” -- it works fine
for its current users, but should be avoided in new code.
For example:

- API that cannot signal an exception, so failures are either ignored or
  exit the process with a fatal error. For example ``PyObject_HasAttr``.
- API that is not thread-safe, for example by borrowing references from
  mutable objects, or exposing unfinished mutable objects. For example
  ``PyDict_GetItemWithError``.
- API with names that don't use the ``Py``/``_Py`` prefix, and so can clash
  with other code. For example: ``setter``.

It is important to note that despite such flaws, it's usually possible
to use the API correctly. For example, in a single-threaded environment,
thread safety is not an issue.
We do not want to break working code, even if it uses API that would be wrong
in some -- or even *most* -- other contexts.

On the other hand, we want to steer users away from such “undesirable” API
in *new* code, especially if a safer alternative exists.


Adding the ``Py`` prefix
------------------------

Some names defined in CPython headers is not namespaced: it that lacks the
``Py`` prefix (or a variant: ``_Py``, and alternative capitalizations).
For example, we declare a function type named simply ``setter``.

While such names are not exported in the ABI (as checked by ``make smelly``),
they can clash with user code and, more importantly, with libraries linked
to third-party extensions.

While it would be possible to provide namespaced aliases and (soft-)deprecate
these names, the only way to make them not clash with third-party code is to
not define them in Python headers at all.


Rationale
=========

We want to allow an easy way for users to avoid “undesirable” API if they
choose to do so.

It might be be sufficient to leave this to third-party linters.
For that we'd need a good way to expose a list of (soft-)deprecated
API to such linters.
While adding that, we can -- rather easily -- do the linter's job directly
in CPython headers, avoiding the need for an extra tool.
Unlike Python, C makes it rather easy to limit available API -- for a whole
project or for each individual source file -- by having users define
an “opt-in” macro.

We already do something similar with ``Py_LIMITED_API``, which limits the
available API to a subset that compiles to stable ABI. (In hindsight, we should
have used a different macro name for that particular kind of limiting, but it's
too late to change that now.)

To be clear, this mechanism is *not* a replacement for deprecation.
Deprecation is for API that prevents new features or optimizations, or
presents a security risk or maintenance burden.
This mechanism, on the other hand, is meant for cases where “we found
a slightly better way of doing things” -- perhaps one that's harder to misuse,
or just has a less misleading name.
(On a lighter note: many people configure a code quality checker to shout at
them about the number of blank lines between functions. Let's help them
identify more substantial “code smells”!)

The proposed macro does not *change* any API definitions; it only *hides* them.
So, if code compiles with the macro, it'll also compile without it, with
identical behaviour.
This has implications for core devs: to deal with undesirable behaviour,
we'll need to introduce new, better API, and *then* discourage the old one.
In turn, this implies that we should look at an individual API and fix all its
known issues at once, rather than do codebase-wide sweeps for a single kind of
issue, so that we avoid multiple renames of the same function.


Specification
=============

We introduce a ``Py_OMIT_LEGACY_API`` macro.
If this macro is defined before ``#include <Python.h>``, some API definitions
-- as described below -- will be omitted from the Python header files.

The macro only omits complete top-level definitions exposed from ``<Python.h>``.
Other things (the ABI, structure definitions, macro expansions, static inline
function bodies, etc.) are not affected.

The C API working group (:pep:`731`) has authority over the set of omitted
definitions.

The set of omitted definitions will be tied to a particular feature release
of CPython, and is finalized in each 3.x.0 Beta 1 release.
In rare cases, entries can be removed (i.e. made available for use) at any
time.


Requirements for omitted API
----------------------------

An API that is omitted with ``Py_OMIT_LEGACY_API`` must:

- be soft-deprecated (see :pep:`387`);
- for all known use cases of the API, have a documented alternative
  or workaround;
- have tests to ensure it keeps working (except for 1:1 renames using
  ``#define`` or ``typedef``);
- be documented (except if it was never mentioned in previous versions of the
  documentation); and
- be approved by the C API working group. (The WG may give blanket approvals
  for groups of related API; see *Initial set* below for examples.)

Note that ``Py_OMIT_LEGACY_API`` is meant for API that can be trivially
replaced by a better alternative.
API without a replacement should generally be deprecated instead.


Location
--------

All API definitions omitted by ``Py_OMIT_LEGACY_API`` will be moved to
a new header, ``Include/legacy.h``.

This is meant to help linter authors compile lists, so they can flag the API
with warnings rather than errors.

Note that for simple renaming of source-only constructs (macros, types), we
expect names to be omitted in the same version -- or the same PR -- that adds
a replacement.
This means that the original definition will be renamed, and a ``typedef``
or ``#define`` for the old name added to ``Include/legacy.h``.


Documentation
-------------

Documentation for omitted API should generally:

- appear after the recommended replacement,
- reference the replacement (e.g. “Similar to X, but…”), and
- focus on differences from the replacement and migration advice.

Exceptions are possible if there is a good reason for them.


Initial set
-----------

The following API will be omitted with ``Py_OMIT_LEGACY_API`` set:

- Omit API returning borrowed references:

  ====================================  ==============================
  Omitted API                           Replacement
  ====================================  ==============================
  ``PyDict_GetItem()``                  ``PyDict_GetItemRef()``
  ``PyDict_GetItemString()``            ``PyDict_GetItemStringRef()``
  ``PyImport_AddModule()``              ``PyImport_AddModuleRef()``
  ``PyList_GetItem()``                  ``PyList_GetItemRef()``
  ====================================  ==============================

- Omit deprecated APIs:

  ====================================  ==============================
  Omitted Deprecated API                Replacement
  ====================================  ==============================
  ``PY_FORMAT_SIZE_T``                  ``"z"``
  ``PY_UNICODE_TYPE``                   ``wchar_t``
  ``PyCode_GetFirstFree()``             ``PyUnstable_Code_GetFirstFree()``
  ``PyCode_New()``                      ``PyUnstable_Code_New()``
  ``PyCode_NewWithPosOnlyArgs()``       ``PyUnstable_Code_NewWithPosOnlyArgs()``
  ``PyImport_ImportModuleNoBlock()``    ``PyImport_ImportModule()``
  ``PyMem_DEL()``                       ``PyMem_Free()``
  ``PyMem_Del()``                       ``PyMem_Free()``
  ``PyMem_FREE()``                      ``PyMem_Free()``
  ``PyMem_MALLOC()``                    ``PyMem_Malloc()``
  ``PyMem_NEW()``                       ``PyMem_New()``
  ``PyMem_REALLOC()``                   ``PyMem_Realloc()``
  ``PyMem_RESIZE()``                    ``PyMem_Resize()``
  ``PyModule_GetFilename()``            ``PyModule_GetFilenameObject()``
  ``PyOS_AfterFork()``                  ``PyOS_AfterFork_Child()``
  ``PyObject_DEL()``                    ``PyObject_Free()``
  ``PyObject_Del()``                    ``PyObject_Free()``
  ``PyObject_FREE()``                   ``PyObject_Free()``
  ``PyObject_MALLOC()``                 ``PyObject_Malloc()``
  ``PyObject_REALLOC()``                ``PyObject_Realloc()``
  ``PySlice_GetIndicesEx()``            (two calls; see current docs)
  ``PyThread_ReInitTLS()``              (no longer needed)
  ``PyThread_create_key()``             ``PyThread_tss_alloc()``
  ``PyThread_delete_key()``             ``PyThread_tss_free()``
  ``PyThread_delete_key_value()``       ``PyThread_tss_delete()``
  ``PyThread_get_key_value()``          ``PyThread_tss_get()``
  ``PyThread_set_key_value()``          ``PyThread_tss_set()``
  ``PyUnicode_AsDecodedObject()``       ``PyUnicode_Decode()``
  ``PyUnicode_AsDecodedUnicode()``      ``PyUnicode_Decode()``
  ``PyUnicode_AsEncodedObject()``       ``PyUnicode_AsEncodedString()``
  ``PyUnicode_AsEncodedUnicode()``      ``PyUnicode_AsEncodedString()``
  ``PyUnicode_IS_READY()``              (no longer needed)
  ``PyUnicode_READY()``                 (no longer needed)
  ``PyWeakref_GET_OBJECT()``            ``PyWeakref_GetRef()``
  ``PyWeakref_GetObject()``             ``PyWeakref_GetRef()``
  ``Py_UNICODE``                        ``wchar_t``
  ``_PyCode_GetExtra()``                ``PyUnstable_Code_GetExtra()``
  ``_PyCode_SetExtra()``                ``PyUnstable_Code_SetExtra()``
  ``_PyDict_GetItemStringWithError()``  ``PyDict_GetItemStringRef()``
  ``_PyEval_RequestCodeExtraIndex()``   ``PyUnstable_Eval_RequestCodeExtraIndex()``
  ``_PyHASH_BITS``                      ``PyHASH_BITS``
  ``_PyHASH_IMAG``                      ``PyHASH_IMAG``
  ``_PyHASH_INF``                       ``PyHASH_INF``
  ``_PyHASH_MODULUS``                   ``PyHASH_MODULUS``
  ``_PyHASH_MULTIPLIER``                ``PyHASH_MULTIPLIER``
  ``_PyObject_EXTRA_INIT``              (no longer needed)
  ``_PyThreadState_UncheckedGet()``     ``PyThreadState_GetUnchecked()``
  ``_PyUnicode_AsString()``             ``PyUnicode_AsUTF8()``
  ``_Py_HashPointer()``                 ``Py_HashPointer()``
  ``_Py_T_OBJECT``                      (``tp_getset``; docs to be written)
  ``_Py_WRITE_RESTRICTED``              (no longer needed)
  ====================================  ==============================

- Soft-deprecate and omit APIs:

  ====================================  ==============================
  Omitted Deprecated API                Replacement
  ====================================  ==============================
  ``PyDict_GetItemWithError()``         ``PyDict_GetItemRef()``
  ``PyDict_SetDefault()``               ``PyDict_SetDefaultRef()``
  ``PyMapping_HasKey()``                ``PyMapping_HasKeyWithError()``
  ``PyMapping_HasKeyString()``          ``PyMapping_HasKeyStringWithError()``
  ``PyObject_HasAttr()``                ``PyObject_HasAttrWithError()``
  ``PyObject_HasAttrString()``          ``PyObject_HasAttrStringWithError()``
  ====================================  ==============================

- Omit ``<structmember.h>`` legacy API:

  The header file ``structmember.h``, which is not included from ``<Python.h>``
  and must be included separately, will ``#error`` if
  ``Py_OMIT_LEGACY_API`` is defined.
  This affects the following API:

  ====================================  ==============================
  Omitted Deprecated API                Replacement
  ====================================  ==============================
  ``T_SHORT``                           ``Py_T_SHORT``
  ``T_INT``                             ``Py_T_INT``
  ``T_LONG``                            ``Py_T_LONG``
  ``T_FLOAT``                           ``Py_T_FLOAT``
  ``T_DOUBLE``                          ``Py_T_DOUBLE``
  ``T_STRING``                          ``Py_T_STRING``
  ``T_OBJECT``                          (``tp_getset``; docs to be written)
  ``T_CHAR``                            ``Py_T_CHAR``
  ``T_BYTE``                            ``Py_T_BYTE``
  ``T_UBYTE``                           ``Py_T_UBYTE``
  ``T_USHORT``                          ``Py_T_USHORT``
  ``T_UINT``                            ``Py_T_UINT``
  ``T_ULONG``                           ``Py_T_ULONG``
  ``T_STRING_INPLACE``                  ``Py_T_STRING_INPLACE``
  ``T_BOOL``                            ``Py_T_BOOL``
  ``T_OBJECT_EX``                       ``Py_T_OBJECT_EX``
  ``T_LONGLONG``                        ``Py_T_LONGLONG``
  ``T_ULONGLONG``                       ``Py_T_ULONGLONG``
  ``T_PYSSIZET``                        ``Py_T_PYSSIZET``
  ``T_NONE``                            (``tp_getset``; docs to be written)
  ``READONLY``                          ``Py_READONLY``
  ``PY_AUDIT_READ``                     ``Py_AUDIT_READ``
  ``READ_RESTRICTED``                   ``Py_AUDIT_READ``
  ``PY_WRITE_RESTRICTED``               (no longer needed)
  ``RESTRICTED``                        ``Py_AUDIT_READ``
  ====================================  ==============================

- Omit soft deprecated macros:

  ======================  =====================================
  Omitted Macros          Replacement
  ======================  =====================================
  ``Py_IS_NAN()``         ``isnan()`` (C99+ ``<math.h>``)
  ``Py_IS_INFINITY()``    ``isinf(X)`` (C99+ ``<math.h>``)
  ``Py_IS_FINITE()``      ``isfinite(X)`` (C99+ ``<math.h>``)
  ``Py_MEMCPY()``         ``memcpy()`` (C ``<string.h>``)
  ======================  =====================================

- Soft-deprecate and omit typedefs without the ``Py``/``_Py`` prefix
  (``getter``, ``setter``, ``allocfunc``, …), in favour of *new* ones
  that add the prefix (``Py_getter`` , etc.)

- Soft-deprecate and omit macros without the ``Py``/``_Py`` prefix
  (``METH_O``, ``CO_COROUTINE``, ``FUTURE_ANNOTATIONS``, ``WAIT_LOCK``, …),
  favour of *new* ones that add the prefix  (``Py_METH_O`` , etc.).

- Any others approved by the C API workgroup


If any of these proposed replacements, or associated documentation,
are not added in time for 3.14.0b1, they'll be omitted with later versions
of ``Py_OMIT_LEGACY_API``.
(We expect this for macros generated by ``configure``: ``HAVE_*``, ``WITH_*``,
``ALIGNOF_*``, ``SIZEOF_*``, and several without a common prefix.)


Implementation
==============

TBD


Backwards Compatibility
=======================

The macro is backwards compatible.
Developers can introduce and update the macro on their own pace, potentially
for one source file at a time.

Future versions of CPython may add more API to the set that
``Py_OMIT_LEGACY_API`` hides, breaking user code.
The fix is to undefine the macro (which is safe to do) or rework the
code.


Discussions
===========

* `PEP 743 – Add Py_COMPAT_API_VERSION to the Python C API (take 2)
  <https://discuss.python.org/t/pep-743-add-py-compat-api-version-to-the-python-c-api-take-2/59323>`__
  (July 2024)
* `Finishing the Great Renaming
  <https://discuss.python.org/t/finishing-the-great-renaming/54082>`_
  (May 2024)
* `PEP 743: Add Py_COMPAT_API_VERSION to the Python C API
  <https://discuss.python.org/t/pep-743-add-py-compat-api-version-to-the-python-c-api/48243>`_
  (March 2024)
* C API Evolutions: `Macro to hide deprecated functions
  <https://github.com/capi-workgroup/api-evolution/issues/24>`_
  (October 2023)
* C API Problems: `Opt-in macro for a new clean API? Subset of functions
  with no known issues
  <https://github.com/capi-workgroup/problems/issues/54>`_
  (June 2023)


Prior Art
=========

* ``Py_LIMITED_API`` macro of :pep:`384` "Defining a Stable ABI".
* Rejected :pep:`606` "Python Compatibility Version" which has a global
  scope.


Copyright
=========

This document is placed in the public domain or under the
CC0-1.0-Universal license, whichever is more permissive.
