PEP: 780
Title: ABI features as environment markers
Author: Klaus Zimmermann <klaus_zimmermann@gmx.de>,
        Ralf Gommers <ralf.gommers@gmail.com>
Sponsor: Lysandros Nikolaou <lisandrosnik@gmail.com>
Discussions-To: https://discuss.python.org/t/86013
Status: Draft
Type: Standards Track
Topic: Packaging
Created: 21-Mar-2025
Python-Version: 3.14
Post-History:
  `05-Aug-2024 <https://discuss.python.org/t/60007>`__,
  `26-Mar-2025 <https://discuss.python.org/t/86013>`__,

..
    If accepted, the contents of this PEP should be integrated into the
    canonical documentation in the Python Packaging User Guide and the
    following directive should be added here:
    .. canonical-pypa-spec:: :ref:`packaging:dependency-specifiers`

Abstract
========

This PEP defines using ABI features as environment markers for project
dependencies, through a new ``sys_abi_features`` environment marker. :pep:`508`
(later moved to :ref:`packaging:dependency-specifiers`) introduced environment
markers to specify dependencies based on rules that describe when the
dependency should be used. This PEP extends the environment markers to allow
specifying dependencies based on specific ABI features of the Python
interpreter. For this, it defines a set of `ABI Features`_ and specifies how
they are made available for `environment markers <pep-780-sys_abi_features_>`_
as a new marker variable, ``sys_abi_features``.

Motivation
==========

In 2015, :pep:`508` established environment markers to specify dependencies
based on environment conditions. The development of free-threaded CPython
[#python-free-threading]_ has underlined the need for an environment marker to
discriminate between different ABI features that the interpreter was built
with. For example, currently there is no way to distinguish between a
GIL-enabled and a free-threaded CPython interpreter with an environment marker.
This leads to real world issues for the adoption of free-threading and its
incremental rollout. When a Python package is being made compatible with
free-threaded CPython, it also needs all its build and runtime dependencies to
be compatible. Capturing the first version of a dependency that is compatible
precisely in metadata is currently not possible, and increasing the minimum
version of a dependency also for the GIL-enabled build is usually undesirable
since it unnecessarily limits compatibility between packages.

Some concrete examples of such issues have been discussed in the `Environment
marker for free-threading`_ Discourse thread:

- Cython has (experimental) support for free-threading only in its master
  branch, and is used by a lot of projects that already publish ``cp313t``
  wheels. Picking up the wrong Cython version is causing a lot of obscure build
  failures and runtime crashes. It would be beneficial if the metadata could
  express that (c.f. `Require Cython Pre-release for Free-Threaded Python`_).
- CFFI has no support for free-threading yet, and `Armin Rigo, one of the
  maintainers, has stated
  <https://github.com/python-cffi/cffi/pull/143#issuecomment-2580781899>`__
  that it may be a good idea to fork ``cffi``, implement support for
  free-threading, and only come back to the CFFI project with a single large PR
  that adds support after the functionality "is reasonably well-tested (either
  as tests or, better in this case, tested by being in use in various other
  projects)". There are a lot of projects that depend on ``cffi``. They are
  likely fine to start depending on a fork for free-threading only, however
  depending on a fork for >=3.13 or for all Python versions seems like a much
  larger ask, and more disruptive for distribution packagers.

While these concrete examples may be addressed later this year by Cython and
CFFI making compatible releases, the same issue is going to repeat further up
the stack. The free-threading rollout is expected to take several years, and an
environment marker for free-threading will make that rollout significantly
easier.

Another important ABI feature that is not yet covered by environment markers is
the bitness of the interpreter. In most cases, the ``sys_platform`` or
``platform_system`` markers are enough, because there is only a single bitness
in use per platform. This is not the case on Windows however: both 32-bit and
64-bit Python interpreters are widely used on x86-64 Windows. Not being able to
distinguish between the two may be relevant for packages that provide compiled
extensions. For example, SciPy does not provide ``win32`` wheels (it isn't able
to due to the lack of a suitable 32-bit compiler toolchain with Fortran
support). Those wheels lacking can be awkward especially for projects where
SciPy is an optional dependency only. In that case, it would be useful to be
able to specify that SciPy is required *unless* the interpreter is a 32-bit one
on Windows (c.f. `Require SciPy Unless on 32-bit win32`_), to avoid failed
from-source installations due to the missing wheels.


Rationale
=========

The intention of this PEP is to introduce its core features with minimal impact
on the existing ecosystem. The existing grammar proposed in :pep:`508` lends
itself to a straightforward extension to include the new environment marker.

A Forward Looking View on Free-Threaded Python
----------------------------------------------

:pep:`703`, the accepted proposal for free threading, states that the rollout
of free-threaded Python should be gradual, which has been clarified by the
Python Steering Council in `the PEP 703 acceptance post`_ to mean a three stage
process over multiple releases. It is therefore important to make sure that the
mechanisms in this PEP are useable for Python interpreters where either
free-threading or non-free-threading could be the default or the only option.

At the time of writing, free-threaded Python is in Phase I: experimental phase.
In this phase, there is an acute need for the proposed environment markers to
help with the transition to free-threaded Python as package authors gradually
add support.

As the number of packages with support increases, and particularly during
Phase II: Supported-but-not-default phase, we still anticipate a strong need
for the environment markers to help with the transition.

As free-threaded Python enters into Phase III: Default phase, the need for the
environment markers will decrease, though at this point it is not clear that
the GIL-enabled Python will be completely phased out (it may remain available
as a non standard build option). If it persists, the inverse need for the ABI
feature detection may arise.

Indeed, in all three phases it may be necessary for package authors to choose
specific versions of their dependencies based on the ABI features, with a shift
from GIL-enabled as default to free-threading as default over time.

The ABI features are designed with this in mind to guarantee usefulness and
simplicity for the foreseeable future in a changing Python ecosystem.

Relation to Other PEPs
----------------------

This PEP extends environment markers with set semantics for ABI features.
:pep:`751#additions-to-marker-expression-syntax` includes a similar extension
for lock file specific environment markers; although the two have been
developed indepedently, they are compatible where they overlap in terms of the
new set semantics.

Specification
=============

The keywords "**MUST**", "**MUST NOT**", "**REQUIRED**", "**SHALL**",
"**SHALL NOT**", "**SHOULD**", "**SHOULD NOT**", "**RECOMMENDED**", "**MAY**",
and "**OPTIONAL**"" in this document are to be interpreted as described in
:rfc:`RFC 2119 <2119>`.

ABI Features
------------

ABI features are intrinsic properties of the Python interpreter, expressed as
simple, understandable strings. However, not all features are equally
applicable to all Python interpreters or Python versions. For example, the
distinction between free-threaded and GIL-enabled interpreters is only relevant
for CPython 3.13 onwards, but the bitness of the interpreter is relevant for
all interpreters.

All interpreters MUST handle the following ABI features as stated. ABI features
that are restricted to particular interpreters MUST NOT be provided by other
interpreters. The features are subdivided into groups and for each group there
MUST be exactly one feature present, except when the group is marked as
optional, in which case there MUST be at most one feature present.

``free-threading`` or ``gil-enabled`` (only CPython)
    If the Python interpreter is free-threaded, the ``free-threading`` feature
    MUST be present and the ``gil-enabled`` feature MUST NOT be present.
    Otherwise, the ``gil-enabled`` feature MUST be present and the
    ``free-threading`` feature MUST NOT be present.

``debug`` (only CPython, optional)
    This ABI feature is reserved for the ``--with-pydebug`` build of CPython.
    If the interpreter is a CPython interpreter with ``Py_DEBUG`` capabilities,
    the ``debug`` feature MUST be present. On POSIX systems, this corresponds
    to the Python expression ``"d" in sys.abiflags``.

``32-bit`` or ``64-bit`` (optional)
    The bitness of the interpreter, that is, whether it is a 32-bit or 64-bit
    build [#bitness]_. If the bitness is unknown or neither 32-bit nor 64-bit,
    this feature MUST NOT be present.

.. _pep-780-sys_abi_features:

The ``sys_abi_features`` Environment Marker
-------------------------------------------

To make ABI features available in dependency specifications, a new environment
marker variable, ``sys_abi_features``, is added to the format of dependency
specifiers.

To do this, we need to extend the grammar laid out in :pep:`508` and maintained
in the :ref:`packaging:dependency-specifiers` and document the possible values.

The grammar is extended to include the ``sys_abi_features`` marker variable by
augmenting the definition of ``env_var`` as follows::

    env_var       = ('python_version' | 'python_full_version' |
                     'os_name' | 'sys_platform' | 'platform_release' |
                     'platform_system' | 'platform_version' |
                     'platform_machine' | 'platform_python_implementation' |
                     'implementation_name' | 'implementation_version' |
                     'sys_abi_features' |
                     'extra' # ONLY when defined by a containing layer
                     )

Like the grammar, also the overview table of environment markers in
:ref:`packaging:dependency-specifiers` is augmented to add the following row:

.. list-table::
    :header-rows: 1

    * - Marker
      - Python equivalent
      - Sample values
    * - ``sys_abi_features``
      - no direct equivalent available
      - ``{'free-threading', '64-bit'}``,
        ``{'gil-enabled', 'debug', '32-bit'}``

With these additions, ABI features can be used in dependency specifications via
the ``in`` operator to test for the presence of a feature, or the ``not in``
operator to test for the absence of a feature.

Examples
========

Require Cython Pre-release for Free-Threaded Python
----------------------------------------------------
To require a pre-release of Cython only for a free-threaded Python interpreter,
the following dependency specification can be used::

    cython >3.1.0a1; "free-threading" in sys_abi_features
    cython ==3.0.*; "free-threading" not in sys_abi_features

Require SciPy Unless on 32-bit ``win32``
----------------------------------------
To require SciPy unless on a 32-bit interpreter on Windows, the following
dependency specification can be used::

    scipy; platform_system != "Windows" or "32-bit" not in sys_abi_features

Require NumPy for a Free-Threaded Interpreter With Debugging Capabilities
--------------------------------------------------------------------------
To require NumPy only for a free-threaded interpreter with debugging
capabilities, the following dependency can be used::

    numpy; "free-threading" in sys_abi_features and "debug" in sys_abi_features

Backwards Compatibility
=======================

This is a pure extension to the existing environment markers and does not
affect existing environment markers or dependency specifications, hence there
are no direct backwards compatibility concerns.

However, the introduction of the feature has implications for a number of
ecosystem tools, especially those which attempt to support examination of data
in ``pyproject.toml`` and ``requirements.txt``.

Audit and Update Tools
----------------------

A wide range of tools understand Python dependency data as expressed in
``requirements.txt`` files. (e.g., Dependabot, Tidelift, etc)

Such tools inspect dependency data and, in some cases, offer tool-assisted or
fully automated updates. It is our expectation that no such tools would support
the new environment markers at first, and broad ecosystem support could take
many months or even some number of years to arrive.

As a result, users of the new environment markers would experience a
degradation in their workflows and tool support at the time that they start
using them. This is true of any new standard for where and how dependency data
are encoded.

Security Implications
=====================

This PEP introduces new syntax for specifying dependency information in
projects. However, it does not introduce newly specified mechanisms for
handling or resolving dependencies.

It therefore does not carry security concerns other than those inherent in any
tools which may already be used to install dependencies---i.e. malicious
dependencies may be specified here, just as they may be specified in
``requirements.txt`` files.

How to Teach This
=================

The use of environment markers is well established and communicated chiefly in
:ref:`packaging:dependency-specifiers`. The new environment marker can be
introduced in the same document. Additionally, both for package authors and
users, free-threading specific guidance can be provided at the
`Python free-threading guide`_.

Reference Implementation
========================

The reference implementation for the environment markers is available in a fork
of the ``packaging`` library at `Environment markers for ABI features
<https://github.com/zklaus/packaging/pull/1>`__.

`A demonstration package <https://github.com/zklaus/env-marker-example>`__ is
also available.

Since ``pip`` uses a vendored copy of ``packaging`` internally, we also provide
`a patched version of pip`__, which replaces the vendored ``packaging`` with
the reference implementation linked above.

__ https://github.com/zklaus/pip/tree/env-marker-free-threading

Rejected Ideas
==============

Extension Mechanism
-------------------

In an early discussion of the topic (`Environment marker for free-threading`_),
the idea of a general extension mechanism for environment markers was brought
up. While it is appealing to forego a whole PEP process should the need for
new environment markers arise in the future, there are two main challenges.

First, a completely dynamic mechanism would present difficulties for tools that
rely on static analysis of dependency specifications.

This means that even if a dynamic mechanism were to be adopted, new environment
markers would likely still need to be spelled out in a PEP.

Second, the introduction of a dynamic mechanism would require a more complex
implementation in the packaging library, which would be a significant departure
from the current approach.

Open Issues
===========

Other Environment Markers
-------------------------
If other environment markers are needed right now, this PEP could be extended
to include them.

Other Tooling
-------------
The reference implementation is based on the ``packaging`` library and ``pip``.
We have confirmed that this allows for building and installing packages with
several build backends. It is possible that other tools should be added to the
reference implementation.


Footnotes
=========

.. [#python-free-threading] Python experimental support for free threading is
   available in Python 3.13 and later. For more information, see `Python
   experimental support for free threading`_.

.. [#bitness] While there are some related environment markers available, such
   as ``platform_machine`` and ``platform_python_implementation``, these are
   not sufficient to reliably determine the bitness of the interpreter,
   particularly on platforms that allow the execution of either kind of binary.


.. _Python experimental support for free threading: https://docs.python.org/3/howto/free-threading-python.html
.. _Python free-threading guide: https://py-free-threading.github.io/
.. _Environment marker for free-threading: https://discuss.python.org/t/environment-marker-for-free-threading/60007
.. _Compatibility Status Tracking: https://py-free-threading.github.io/tracking/
.. _Free-Treaded Wheels: https://hugovk.github.io/free-threaded-wheels/
.. _the PEP 703 acceptance post: https://discuss.python.org/t/pep-703-making-the-global-interpreter-lock-optional-in-cpython-acceptance/37075

Acknowledgements
================

Thanks to Filipe Laíns for the suggestion of the ``abi_features`` attribute and
to Stephen Rosen for the Backwards Compatibility section of :pep:`735`, which
served as a template for the corresponding section in this PEP.

Copyright
=========

This document is placed in the public domain or under the
CC0-1.0-Universal license, whichever is more permissive.
