PEP: 770
Title: Improving measurability of Python packages with Software Bill-of-Materials
Author: Seth Larson <seth@python.org>
Sponsor: Brett Cannon <brett@python.org>
PEP-Delegate: Brett Cannon <brett@python.org>
Discussions-To: https://discuss.python.org/t/76308
Status: Final
Type: Standards Track
Topic: Packaging
Created: 02-Jan-2025
Post-History:
  `05-Nov-2024 <https://discuss.python.org/t/70261>`__,
  `06-Jan-2025 <https://discuss.python.org/t/76308>`__,
Resolution: `11-Apr-2025 <https://discuss.python.org/t/76308/112>`__

.. canonical-pypa-spec:: `The .dist-info/sboms/ directory <https://packaging.python.org/en/latest/specifications/binary-distribution-format/#the-dist-info-sboms-directory>`__

Abstract
========

Almost all Python packages today are accurately measurable by software
composition analysis (SCA) tools. For projects that are not accurately
measurable, there is no existing mechanism to annotate a Python package
with composition data to improve measurability.

Software Bill-of-Materials (SBOM) is a technology-and-ecosystem-agnostic
method for describing software composition, provenance, heritage, and more.
SBOMs are used as inputs for SCA tools, such as scanners for vulnerabilities and
licenses, and have been gaining traction in global software regulations and
frameworks.

This PEP proposes using SBOM documents included in Python packages as a
means to improve automated software measurability for Python packages.

Motivation
==========

Measurability and Phantom Dependencies
--------------------------------------

Python packages are particularly affected by the "`phantom dependency`_"
problem, where software components that aren't written in Python are included
in Python packages for many reasons, such as ease of installation and
compatibility with standards:

* Python serves scientific, data, web, and machine-learning use-cases which
  use compiled or non-Python languages like Rust, C, C++, Fortran, JavaScript,
  and others.
* The Python wheel format is preferred by users due to the ease-of-installation.
  No code is executed during the installation step, only extracting the archive.
* The Python wheel format requires bundling shared compiled libraries without
  a method to encode metadata about these libraries.
* Packages related to Python packaging sometimes need to solve the
  "bootstrapping" problem, so include pure Python projects inside their
  source code.

These software components can't be described using Python package metadata and
thus are likely to be missed by software composition analysis (SCA) software
which can mean vulnerable software components aren't reported accurately.

`For example <https://sethmlarson.dev/early-promising-results-with-sboms-and-python-packages>`__,
the Python package Pillow includes 16 shared object libraries in the wheel that
were bundled by auditwheel as a part of the build. None of those shared object
libraries are detected when using common SCA tools like Syft and Grype.
If an SBOM document is included annotating all the included shared libraries
then SCA tools can identify the included software reliably.

Build Tools, Environment, and Reproducibility
---------------------------------------------

Going beyond the runtime dependencies of a package: SBOMs can also record the
tools and environments used to build a package. Recording the exact tools
and versions used to build a package is often required to establish
`build reproducibility <https://reproducible-builds.org>`__.
Build reproducibility is a property of software that can be used to detect
incorrectly or maliciously modified software components when compared to their
upstream sources. Without a recorded list of build tools and versions it can
become difficult to impossible for a third-party to verify build reproducibility.

Regulations
-----------

SBOMs are required by recent software security regulations, like the
`Secure Software Development Framework`_ (SSDF) and the
`Cyber Resilience Act`_ (CRA). Due to their inclusion in these regulations,
the demand for SBOM documents of open source projects is expected to be high.
One goal is to minimize the demands on open source project maintainers by
enabling open source users that need SBOMs to self-serve using existing
tooling.

Another goal is to enable contributions from users who need SBOMs to annotate
projects they depend on with SBOM information. Today there is no mechanism to
propagate the results of those contributions for a Python package so there is
no incentive for users to contribute this type of work.

.. _Cyber Resilience Act: https://digital-strategy.ec.europa.eu/en/policies/cyber-resilience-act
.. _Secure Software Development Framework: https://csrc.nist.gov/Projects/ssdf

Rationale
=========

Using SBOM standards instead of Core Metadata fields
----------------------------------------------------

Attempting to add every field offered by SBOM standards into Python package
Core Metadata would result in an explosion of new Core Metadata fields,
including the need to keep up-to-date as SBOM standards continue to evolve
to suit new needs in that space.

Instead, this proposal delegates SBOM-specific metadata to SBOM documents that
are included in Python packages into a named directory under dist-info.

This standard also doesn't aim to replace Core Metadata with SBOMs,
instead focusing on the SBOM information being supplemental to Core Metadata.
Included SBOMs only contain information about dependencies included in the
package archive or information about the top-level software in the package that
can't be encoded into Core Metadata but is relevant for the SBOM use-case
("software identifiers", "purpose", "support level", etc).

Zero-or-more opaque SBOM documents
----------------------------------

Rather than requiring at most one included SBOM document per Python package,
this PEP proposes that one or more SBOM documents may be included in a Python
package. This means that code attempting to annotate a Python package with SBOM
data may do so without being concerned about corrupting data already contained
within other SBOM documents.

Additionally, this PEP treats SBOM document data opaquely instead relying on
final end-users of the SBOM data to process the contained SBOM data.
This choice acknowledges that SBOM standards are an active area of development
where there is not yet (and may never be) a single definitive SBOM standard
and that SBOM standards can continue to evolve independent of Python packaging
standards. Already tools that consume SBOM documents support a multitude of
SBOM standards to handle this reality.

These decisions mean this PEP is capable of supporting any SBOM standard
and does not favor one over the other, instead deferring the decision to
producing projects and tools and consuming user tooling.

Adding data to Python packages without new metadata versions
------------------------------------------------------------

The rollout of a new metadata version and field requires that many different
projects and teams need to adopt the metadata version in sequence to avoid
widespread breakage. This effect usually means a substantial delay in how
quickly users and tools can start using new packaging features.

For example, a single metadata version bump requires
updates to PyPI, various ``pyproject.toml`` parsing and schema projects,
the ``packaging`` library, wait for releases, then ``pip`` and other installers
need to bundle the changes to ``packaging`` and release, then build backends can
begin emitting the new metadata version, again wait for releases, and only then
can projects begin using the new features. Even with this careful approach it's
not guaranteed that tools won't break on new metadata versions and fields.

To avoid this delay, simplify overall how to include SBOMs, and to give
flexibility to build backends and tools, this PEP proposes using a subdirectory
under ``.dist-info`` to safely add data to a Python package while avoiding the
need for new metadata fields and versions. This mechanism allows build backends
and tools to begin using the feature described in this PEP immediately after
acceptance without the head-of-line blocking on other projects adopting the PEP.

Storing files in the ``.dist-info`` or ``.data`` directory
----------------------------------------------------------

There are two top-level directories in binary distributions where files beyond
the software itself can be stored: ``.dist-info`` and ``.data``.
This specification chose to use the ``.dist-info`` directory for storing
subdirectories and files.

Firstly, the ``.data`` directory has no corresponding location in the installed
package, compared to ``.dist-info`` which does preserve the link between the
binary distribution to the installed package in an environment. The ``.data``
directory instead has all its contents merged between all installed packages in
an environment which can lead to collisions between similarly named files.

Secondly, subdirectories under the ``.data`` directory require new definitions
to the Python `sysconfig <https://docs.python.org/3/library/sysconfig.html>`__
module. This means defining additional directories require waiting for a change
to Python and *using* the directory requires waiting for adoption of the new
Python version by users. Subdirectories under ``.dist-info`` don't have these
requirements, they can be used by any user, build backend, and installer
immediately after a new subdirectory name is registered regardless of Python
or metadata version.

What are the differences between PEP 770 and PEP 725?
-----------------------------------------------------

:pep:`725`
("Specifying external dependencies in pyproject.toml") is a different
PEP with some similarities to PEP 770, such as attempting to describe non-Python
software within Python packaging metadata. This section aims to show how these
two PEPs are tracking different information and serving different use-cases:

* PEP 725 describes **abstract dependencies**, such as requiring "a C compiler"
  as a build-time dependency (``virtual:compiler/c``) or needing to link "the
  OpenSSL library" at build time (``pkg:generic/openssl``). PEP 770 describes
  **concrete dependencies**, more akin to dependencies in a "lock file", such as
  an exact name, version, architecture, and
  hash of a software library distributed through AlmaLinux distribution
  (``pkg:rpm/almalinux/libssl3@3.2.0``). For cases like build dependencies this
  might result in a dependency being requested via PEP 725 and then recorded
  concretely in an SBOM post-build with PEP 770.
* PEP 725 is for describing **external dependencies**, provided by the system
  being used to either build or run the software. PEP 770 is for describing
  **bundled software inside Python package archives**, the SBOM documents
  don't describe software on the system.
* **PEP 725 is primarily about identification**, using a list of software
  identifiers. PEP 770 provides the **complete functionality of SBOM standards**
  to describe various software attributes such as license, checksum, download
  location, etc.
* **PEP 725 and PEP 770 have different users and use-cases**. PEP 725 is
  primarily for humans writing dependencies in ``pyproject.toml`` by hand.
  The users of the information are build backends and users who want to build
  software from source.
  PEP 770 is primarily for tools which are capable of generating SBOM documents
  to be included in a Python package archive and SBOM/SCA tools which want to
  SBOM documents about installed software to do some other task such as
  vulnerability scanning or software analysis.

.. _770-spec:

Specification
=============

The changes necessary to implement this PEP include:

* Explicitly reserving the subdirectory ``.dist-info/sboms``.
* `Additions <770-spec-project-formats_>`_ to the built distribution (wheel),
  and installed project specifications

In addition to the above, an informational PEP will be created for tools
consuming included SBOM documents and other Python package metadata to
generate complete SBOM documents for Python packages.

.. _770-spec-dist-info-subdirs-registry:

Reserving the ``.dist-info/sboms`` directory
--------------------------------------------

This PEP introduces a new registry of reserved subdirectory names allowed in
the ``.dist-info`` directory for the :term:`distribution archive`
and :term:`installed project` s project types. Future additions to this registry
will be made through the PEP process. The initial values in this registry are:

================= ==============
Subdirectory name PEP / Standard
================= ==============
``licenses``      :pep:`639`
``license_files`` :pep:`639` (draft-only)
``LICENSES``      `REUSE licensing framework <https://reuse.software>`__
``sboms``         :pep:`770`
================= ==============

See :ref:`770-backwards-compat` for a complete methodology for
avoiding backwards incompatibilities with selecting this directory name.

.. _770-spec-project-formats:

SBOM files in project formats
-----------------------------

A few additions will be made to the existing specifications.

:term:`Built distributions <Built distribution>` (:term:`wheels <wheel>`)
  The wheel specification will be updated to add the new registry of reserved
  directory names and to reflect that if the ``.dist-info/sboms`` subdirectory
  is specified that the directory contains SBOM files.

:term:`Installed projects <Installed project>`
  The Recording Installed Projects specification will be updated to reflect
  that if the ``.dist-info/sboms`` subdirectory is specified that the directory
  contains SBOM files and that any files in this directory MUST be copied from
  wheels by install tools.

SBOM data interoperability
--------------------------

This PEP treats data contained within SBOM documents as opaque, recognizing
that SBOM standards are an active area of development. However, there are some
considerations for SBOM data producers that when followed will improve the
interoperability and usability of SBOM data made available in Python packages:

* SBOM documents SHOULD use a widely-accepted SBOM standard, such as
  `CycloneDX <cyclonedxspec_>`_ or `SPDX <spdxspec_>`_.
* SBOM documents SHOULD use UTF-8-encoded JSON (:rfc:`8259`) when available
  for the SBOM standard in use.
* SBOM documents SHOULD include all required fields for the SBOM standard in
  use.
* SBOM documents SHOULD include a "time of creation" and "creating tool" field
  for the SBOM standard in use. This information is important for users
  attempting to reconstruct different stages for a Python package being built.
* The primary component described by the SBOM document SHOULD be the top-level
  software within the Python package (for example,
  "pkg:pypi/pillow" for the Pillow package).
* All non-primary components SHOULD have one or more paths in the relationship
  graph showing the relationship between components. If this information isn't
  included, SCA tools might exclude components outside of the relationship graph.
* All software components SHOULD have a name, version, and one or more software
  identifiers (PURL, CPE, download URL).

PyPI and other indices MAY validate the contents of SBOM documents specified by
this PEP, but MUST NOT validate or reject data for unknown
SBOM standards, versions, or fields.

.. _770-backwards-compat:

Backwards Compatibility
=======================

Reserved ``.dist-info/sboms`` subdirectory
------------------------------------------

The new reserved ``.dist-info/sboms`` subdirectory represents
a new reservation that wasn't previously documented, thus has the potential to
break assumptions being made by already existing tools.

To check what ``.dist-info`` subdirectory names are in use today
a query across
`all files in package archives on PyPI <https://sethmlarson.dev/security-developer-in-residence-weekly-report-18>`__
was executed:

.. code-block:: sql

    SELECT (
      regexp_extract(archive_path, '.*\.dist-info/([^/]+)/', 1) AS dirname,
      COUNT(DISTINCT project_name) AS projects
    )
    FROM '*.parquet'
    WHERE archive_path LIKE '%.dist-info/%/%'
    GROUP BY dirname ORDER BY projects DESC;

Note that this only includes records for
*files* and thus won't return results for empty directories. Empty directories
being pervasively used and somehow load-bearing is unlikely, so is an accepted
risk of using this method. This query yielded the following results:

====================== ===============
Subdirectory           Unique Projects
====================== ===============
``licenses``           22,026
``license_files``      1,828
``LICENSES``           170
``.ipynb_checkpoints`` 85
``license``            18
``.wex``               9
``dist``               8
``include``            6
``build``              5
``tmp``                4
``src``                3
``calmjs_artifacts``   3
``.idea``              2
====================== ===============

Not shown above are around ~50 other subdirectory names that are used in a
single project. From these results we can see:

* Most subdirectories under ``.dist-info`` are to do with licensing,
  one of which (``licenses``) is specified by :pep:`639` and others
  (``license_files``, ``LICENSES``) are from draft implementations
  of :pep:`639`.
* The ``sboms`` subdirectory doesn't collide with existing use.
* Other subdirectory names under ``.dist-info`` appear to be either not
  widespread or accidental.

As a result of this query we can see there are already some projects placing
directories under ``.dist-info``, so we can't require that build frontends
raise errors for unregistered subdirectories. Instead the recommendation is
that build frontends MAY warn the user or raise an error in this scenario.

Security Implications
=====================

SBOM documents are only as useful as the information encoded in them.
If an SBOM document contains incorrect information then this can result in
incorrect downstream analysis by SCA tools. For this reason, it's important
for tools including SBOM data into Python packages to be confident in the
information they are recording. SBOMs are capable of recording "known unknowns"
in addition to known data. This practice is recommended when not certain about
the data being recorded to allow for further analysis by users.

Because SBOM documents can encode information about the original system
where a Python package is built (for example, the operating system name and
version, less commonly the names of paths). This information has the potential
to "leak" through the Python package to installers via SBOMs. If this
information is sensitive, then that could represent a security risk.

How to Teach This
=================

Most typical users of Python and Python packages won't need to know the details
of this standard. The details of this standard are most important to either
maintainers of Python packages and developers of SCA tools such as
SBOM generation tools and vulnerability scanners.

What do Python package maintainers need to know?
------------------------------------------------

Python package metadata can already describe the top-level software included in
a package archive, but what if a package archive contains other software
components beyond the top-level software? For example, the Python wheel for
"Pillow" contains a handful of other software libraries bundled inside, like
``libjpeg``, ``libpng``, ``libwebp``, and so on. This scenario is where this PEP
is most useful, for adding metadata about bundled software to a Python package.

Some build tools may be able to automatically annotate bundled dependencies.
Typically tools can automatically annotate bundled dependencies when those
dependencies come from a "packaging ecosystem" (such as PyPI, Linux distros,
Crates.io, NPM, etc).

What do SBOM tool authors need to know?
---------------------------------------

Developers of SBOM generation tooling will need to know about the existence
of this PEP and that Python packages may begin publishing SBOM documents
within package archives. This information needs to be included as a part of
generating an SBOM document for a particular Python package or Python
environment.

A follow-up informational PEP will be authored to describe how to transform
Python packaging metadata, including the mechanism described in this PEP,
into an SBOM document describing Python packages. Once the informational PEP is
complete, tracking issues will be opened specifically linking to the
informational PEP to spur the adoption of PEP 770 by SBOM tools.

A `benchmark is being created <https://github.com/psf/sboms-for-python-packages/tree/main/benchmark>`__
to compare the outputs of different SBOM tools when run with various Python
packaging inputs (package archive, installed package, environment, container
image) is being created to track the progress of different SBOM generation
tools. This benchmark will inform where tools have gaps in support
of this PEP and Python packages.

What do users of SBOM documents need to know?
---------------------------------------------

Many users of this PEP won't know of its existence, instead their software
composition analysis tools, SBOM tools, or vulnerability scanners will simply
begin giving more comprehensive information after an upgrade. For users that are
interested in the sources of this new information, the "tool" field of SBOM
metadata already provides linkages to the projects generating their SBOMs.

For users who need SBOM documents describing their open source dependencies the
first step should always be "create them yourself". Using the benchmarks above
a list of tools that are known to be accurate for Python packages can be
documented and recommended to users. For projects which require
additional manual SBOM annotation: tips for contributing this data and tools for
maintaining the data can be recommended.

Note that SBOM documents can vary across different Python package archives
due to variance in dependencies, Python version, platform, architecture, etc.
For this reason users SHOULD only use the SBOM documents contained within
the actual downloaded and installed Python package archive and not assume that
the SBOM documents are the same for all archives in a given package release.

Reference Implementation
========================

`Auditwheel fork <https://sethmlarson.dev/early-promising-results-with-sboms-and-python-packages>`_
which generates CycloneDX SBOM documents to include in wheels describing
bundled shared library files. These SBOM documents worked as expected for the
Syft and Grype SBOM and vulnerability scanners.

Rejected Ideas
==============

Requiring a single SBOM standard
--------------------------------

There is no universally accepted SBOM standard and this area is still
rapidly evolving (for example, SPDX released a new major version of their
standard in April 2024). Most discussion and development around SBOMs today
focuses on two SBOM standards: `CycloneDX <cyclonedxspec_>`_ and
`SPDX <spdxspec_>`_.

To avoid locking the Python ecosystem into a specific
standard ahead of when a clear winner emerges this PEP treats SBOM documents
as opaque and only makes recommendations to promote compatibility with
downstream consumers of SBOM document data.

None of the decisions in this PEP restrict a future PEP to select
a single SBOM standard. Tools that use SBOM data today already need to support
multiple formats to handle this situation, so a future standard that updates to
require only one standard would have no effect on downstream SBOM tools.

Using metadata fields to specify SBOM files in archives
-------------------------------------------------------

A previous iteration of this specification used an ``Sbom-File`` metadata
field to specify an SBOM file within a source or binary distribution archive.
This would make the implementation similar to :pep:`639` which uses the
``License-File`` field to enumerate license files in archives.

The primary issue with this approach is that SBOM files can originate from both
static and dynamic sources: like versioned source code, the build backend,
or from tools adding SBOM files after the build has completed (like auditwheel).

Metadata fields must either be static or dynamic, not both. This is
in direct conflict with the best-case scenario for SBOM data: that SBOM files
are added automatically by tools during the build of a Python package without
user-involvement or knowledge. Compare this situation to license files which
are almost always static.

The 639-style approach was ultimately dropped in favor of defining SBOMs simply
by their presence in the ``.dist-info/sboms`` directory. This approach allows
build backends and tools to add their own SBOM data without the static/dynamic
conflict.

A future PEP will define the process for statically defining SBOM files to be
added to the ``.dist-info/sboms`` directory.

References
==========

* `Visualizing the Python package SBOM data flow <https://sethmlarson.dev/visualizing-the-python-package-sbom-data-flow>`_.
  This is a graphic that shows how this PEP fits into the bigger picture of
  Python packaging's SBOM data story.

* `Adding SBOMs to Python wheels with auditwheel <https://sethmlarson.dev/early-promising-results-with-sboms-and-python-packages>`_.
  This was some early results from a fork of auditwheel to add SBOM data to a
  wheel and then use an SBOM generation tool Syft to detect the SBOM in the
  installed package.

* `Querying every file in every release on PyPI <https://sethmlarson.dev/security-developer-in-residence-weekly-report-18>`_.
  The dataset available on `py-code.org <py-code.org>`__ from Tom Forbes was
  used to check subdirectory usage in ``.dist-info`` files.

.. _phantom dependency: https://www.endorlabs.com/learn/dependency-resolution-in-python-beware-the-phantom-dependency
.. _coremetadataspec: https://packaging.python.org/specifications/core-metadata
.. _pyprojecttoml: https://packaging.python.org/en/latest/specifications/pyproject-toml/
.. _spdxspec: https://spdx.dev/use/specifications/
.. _cyclonedxspec: https://cyclonedx.org/specification/overview/
.. _pypi-data: https://github.com/sethmlarson/pypi-data

Acknowledgements
================

Thanks to Karolina Surma for authoring and leading :pep:`639` to acceptance.
This PEP's initial design was heavily inspired by :pep:`639` and adopts a
similar approach of using a subdirectory under ``.dist-info`` to store files.

Copyright
=========

This document is placed in the public domain or under the
CC0-1.0-Universal license, whichever is more permissive.
