ITADN
NVIDIA/TransformerEngine
代码 动态 Issues506 合并请求1532 版本23
NVIDIA/TransformerEngine · 文件 下载 ZIP
文件最后提交记录最后更新时间
README.md
以下内容由 AI 翻译，如有问题请点此提交 issue 反馈

.. 版权所有 (c) 2022-2026, NVIDIA CORPORATION & AFFILIATES. 保留所有权利。

有关许可信息，请参阅 LICENSE。

|License|

Transformer Engine

Quickstart <#examples>_ | Installation <#installation>_ | User Guide <https://docs.nvidia.com/deeplearning/transformer-engine/user-guide/index.html>_ | Examples <https://github.com/NVIDIA/TransformerEngine/tree/main/examples>_ | Convergence <#convergence>_ | Integrations <#integrations>_ | Release notes <https://docs.nvidia.com/deeplearning/transformer-engine/documentation-archive.html>_

最新新闻

  • [06/2026] Boosting MoE Training Throughput with Advanced Fusion Kernels <https://developer.nvidia.com/blog/boosting-moe-training-throughput-with-advanced-fusion-kernels/>_
  • [06/2026] Nemotron 3 Ultra: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoning <https://research.nvidia.com/labs/nemotron/files/NVIDIA-Nemotron-3-Ultra-Technical-Report.pdf>_
  • [06/2026] Train Models Faster with JAX and MaxText Using NVFP4 on NVIDIA Blackwell <https://developer.nvidia.com/blog/train-models-faster-with-jax-and-maxtext-using-nvfp4-on-nvidia-blackwell/>_
  • [04/2026] Run High-Throughput Reinforcement Learning Training with End-to-End FP8 Precision <https://developer.nvidia.com/blog/run-high-throughput-reinforcement-learning-training-with-end-to-end-fp8-precision/>_
  • [02/2026] Using NVFP4 Low-Precision Model Training for Higher Throughput Without Losing Accuracy <https://developer.nvidia.com/blog/using-nvfp4-low-precision-model-training-for-higher-throughput-without-losing-accuracy/>_
  • [12/2025] NVIDIA Nemotron 3: Efficient and Open Intelligence <https://arxiv.org/abs/2512.20856>_ - 使用 Transformer Engine 以 NVFP4 进行训练
  • [11/2025] NVIDIA Blackwell Architecture Sweeps MLPerf Training v5.1 Benchmarks <https://developer.nvidia.com/blog/nvidia-blackwell-architecture-sweeps-mlperf-training-v5-1-benchmarks/>_
  • [11/2025] Scale Biology Transformer Models with PyTorch and NVIDIA BioNeMo Recipes <https://developer.nvidia.com/blog/scale-biology-transformer-models-with-pytorch-and-nvidia-bionemo-recipes/>_
  • [11/2025] FP8 Training of Large-Scale RL Models <https://lmsys.org/blog/2025-11-25-fp8-rl/>_
  • [09/2025] Pretraining Large Language Models with NVFP4 <https://www.arxiv.org/pdf/2509.25149>_
  • [09/2025] Native FP8 Mixed Precision Training for Ling 2.0, Open Sourced! <https://huggingface.co/blog/im0qianqian/ling-mini-2-fp8-mixed-precision-training-solution>_
  • [09/2025] Faster Training Throughput in FP8 Precision with NVIDIA NeMo <https://developer.nvidia.com/blog/faster-training-throughput-in-fp8-precision-with-nvidia-nemo/>_
  • [08/2025] How we built DeepL's next-generation LLMs with FP8 for training and inference <https://www.deepl.com/en/blog/tech/next-generation-llm-fp8-training>_
  • [08/2025] NVFP4 Trains with Precision of 16-bit and Speed and Efficiency of 4-bit <https://developer.nvidia.com/blog/nvfp4-trains-with-precision-of-16-bit-and-speed-and-efficiency-of-4-bit/>_

Previous News <#previous-news>_

什么是 Transformer Engine？

.. overview-begin-marker-do-not-remove

Transformer Engine (TE) 是一个用于在 NVIDIA GPU 上加速 Transformer 模型的库，包括 在 Hopper、Ada 和 Blackwell GPU 上使用 8 位浮点 (FP8) 精度，以在训练和推理中提供 更好的性能并降低内存占用。在 Blackwell GPU 上，TE 还支持 MXFP8 (Microscaling FP8) 和 NVFP4 格式，以实现更高的效率。TE 提供了一组针对流行 Transformer 架构的高度优化构建模块，以及一个类似自动混合 精度的 API，可与特定于框架的代码无缝配合使用。TE 还包括一个 与框架无关的 C++ API，可与其他深度学习库集成，以启用 Transformer 的 FP8 支持。

随着 Transformer 模型在大型语言模型、 MoE 架构和多模态模型中扩展到数千亿参数， 训练和推理变得越来越 内存和计算密集。混合精度训练将单精度 (FP32) 与 较低精度格式相结合，在几乎不影响精度的情况下带来显著的速度提升。FP8 是 随 Hopper GPU 架构引入的，与 FP16 相比提供了进一步的性能提升，且精度 没有下降，而 Blackwell 上更新的格式如 MXFP8 和 NVFP4 则进一步提升了效率。

TE 与主流 LLM 框架集成，并提供优化，使低精度训练能够与 MoE、张量/序列/上下文并行以及融合操作等高级功能无缝配合。它提供了一个 Python API，包含用于轻松构建 Transformer 层的模块，以及一个与框架无关的 C++ 库，其中包含 FP8 支持所需的结构体和内核。TE 提供的模块内部维护 FP8 训练所需的缩放因子和其他值，极大地简化了用户的混合精度训练。

Highlights

  • 易于使用的模块，用于构建支持 FP8 的 Transformer 层
  • 针对 Transformer 模型的优化（例如融合内核）
  • 支持在 NVIDIA Hopper、Ada 和 Blackwell GPU 上使用 FP8
  • 支持在 NVIDIA Blackwell GPU 上使用 MXFP8 和 NVFP4
  • 支持在 NVIDIA Ampere GPU 架构及后续架构的所有精度（FP16、BF16）上进行优化

Examples

PyTorch ^^^^^^^

.. code-block:: python

import torch import transformer_engine.pytorch as te from transformer_engine.common import recipe

Set dimensions.

in_features = 768 out_features = 3072 hidden_size = 2048

Initialize model and inputs.

model = te.Linear(in_features, out_features, bias=True) inp = torch.randn(hidden_size, in_features, device="cuda")

Create an FP8 recipe. Note: All input args are optional.

fp8_recipe = recipe.DelayedScaling(margin=0, fp8_format=recipe.Format.E4M3)

为前向传播启用自动转换

with te.autocast(enabled=True, recipe=fp8_recipe): out = model(inp)

loss = out.sum() loss.backward()

JAX ^^^

Flax


.. code-block:: python

  import flax
  import jax
  import jax.numpy as jnp
  import transformer_engine.jax as te
  import transformer_engine.jax.flax as te_flax
  from transformer_engine.common import recipe

  BATCH = 32
  SEQLEN = 128
  HIDDEN = 1024

  # 初始化 RNG 和输入。
  rng = jax.random.PRNGKey(0)
  init_rng, data_rng = jax.random.split(rng)
  inp = jax.random.normal(data_rng, [BATCH, SEQLEN, HIDDEN], jnp.float32)

  # 创建一个 FP8 配方。注意：所有输入参数均为可选。
  fp8_recipe = recipe.DelayedScaling(margin=0, fp8_format=recipe.Format.HYBRID)

  # 为前向传播启用自动转换
  with te.autocast(enabled=True, recipe=fp8_recipe):
      model = te_flax.DenseGeneral(features=HIDDEN)

      def loss_fn(params, other_vars, inp):
        out = model.apply({'params':params, **other_vars}, inp)
        return jnp.mean(out)

      # 初始化模型。
      variables = model.init(init_rng, inp)
      other_variables, params = flax.core.pop(variables, 'params')

      # 构建前向和反向函数
      fwd_bwd_fn = jax.value_and_grad(loss_fn, argnums=(0, 1))

      for _ in range(10):
        loss, (param_grads, other_grads) = fwd_bwd_fn(params, other_variables, inp)

如需更全面的教程，请参阅我们的 `Getting Started Guide <https://docs.nvidia.com/deeplearning/transformer-engine/user-guide/getting_started/index.html>`_。

.. overview-end-marker-do-not-remove

安装
============

系统要求
^^^^^^^^^^^^^^^^^^^

* **硬件：** Blackwell, Hopper, Grace Hopper/Blackwell, Ada, Ampere

* **操作系统：** Linux（官方支持），WSL2（有限支持）

* **软件：**

  * CUDA: 12.1+ (Hopper/Ada/Ampere), 12.8+ (Blackwell) 及兼容的 NVIDIA 驱动程序
  * cuDNN: 9.3+
  * 编译器：支持 C++17 的 GCC 9+ 或 Clang 10+
  * Python: 推荐 3.12

* **源码构建要求：** CMake 3.18+, Ninja, Git 2.17+, pybind11 2.6.0+, nvidia-cudnn-frontend 1.25.0+

* **备注：** FP8 功能需要 Compute Capability 8.9+ (Ada/Hopper/Blackwell)

安装方法
^^^^^^^^^^^^^^^^^^^^

Docker（推荐）
^^^^^^^^^^^^^^^^^^^^
使用 `NVIDIA GPU Cloud (NGC) Catalog <https://catalog.ngc.nvidia.com/orgs/nvidia/containers/pytorch>`_ 上的 Docker 镜像是快速开始使用 Transformer Engine 的最快方式。

例如，要交互式地使用 NGC PyTorch 容器，

.. code-block:: bash

    docker run --gpus all -it --rm nvcr.io/nvidia/pytorch:26.01-py3

例如，要交互式地使用 NGC JAX 容器，

.. code-block:: bash

    docker run --gpus all -it --rm nvcr.io/nvidia/jax:26.01-py3

其中 26.01（对应 2026 年 1 月发布）是容器版本。

我们建议更新到此处可用的最新 NGC 容器：

* https://catalog.ngc.nvidia.com/orgs/nvidia/containers/pytorch
* https://catalog.ngc.nvidia.com/orgs/nvidia/containers/jax

如果您运行任何示例，请确保使用的是与 TransformerEngine 匹配的版本。TransformerEngine 已预先构建并打包在容器中，示例可在 ``/opt/transformerengine`` or ``/opt/transformer-engine`` 中找到。

**使用 NGC 容器的优势：**

* 所有依赖项均已预装，版本兼容且配置经过优化

pip 安装
^^^^^^^^^^^^^^^^

**pip 安装的前置条件：**

* 兼容的 C++ 编译器
* 如果从源代码安装，则需要包含 cuDNN 和 NVCC（NVIDIA CUDA 编译器）的 CUDA Toolkit。

要使用 pip 安装最新稳定版本：

.. code-block:: bash

    # For PyTorch integration
    pip install --no-build-isolation transformer_engine[pytorch]

    # For JAX integration
    pip install --no-build-isolation transformer_engine[jax]

    # For both frameworks
    pip install --no-build-isolation transformer_engine[pytorch,jax]

或者，直接从 GitHub 仓库安装：

.. code-block:: bash

    pip install --no-build-isolation git+https://github.com/NVIDIA/TransformerEngine.git@stable

从 GitHub 安装时，可以使用环境变量明确指定框架：

.. code-block:: bash

    NVTE_FRAMEWORK=pytorch,jax pip install --no-build-isolation git+https://github.com/NVIDIA/TransformerEngine.git@stable

conda 安装
^^^^^^^^^^^^^^^^^^

要使用 conda 从 conda-forge 安装最新稳定版本：

.. code-block:: bash

# 用于 PyTorch 集成
    conda install -c conda-forge transformer-engine-torch

    # JAX 集成（即将推出）

源码安装
^^^^^^^^^^^^^^^^^^^

`See the installation guide <https://docs.nvidia.com/deeplearning/transformer-engine/user-guide/installation.html#installation-from-source>`_

环境变量
^^^^^^^^^^^^^^^^^^^^^
可以在安装前设置以下环境变量以自定义构建过程：

* **CUDA_PATH**：CUDA 安装路径
* **CUDNN_PATH**：cuDNN 安装路径
* **CXX**：C++ 编译器路径
* **NVTE_FRAMEWORK**：要构建的框架的逗号分隔列表（例如，``pytorch,jax``）
* **MAX_JOBS**：限制并行构建任务的数量（默认值因系统而异）
* **NVTE_BUILD_THREADS_PER_JOB**：控制每个构建任务的线程数
* **NVTE_CUDA_ARCHS**：要编译的 CUDA 计算架构的分号分隔列表（例如，``80;90`` 用于 A100 和 H100）。如果未设置，则根据 CUDA 版本自动确定。设置此项可以显著减少构建时间和二进制文件大小。

使用 FlashAttention 编译
^^^^^^^^^^^^^^^^^^^^^^^^^
Transformer Engine 在 PyTorch 中同时支持 FlashAttention-2 和 FlashAttention-3 以提升性能。FlashAttention-3 在 v1.11 版本中添加，当环境中同时存在两者时，优先使用 FlashAttention-3。

您可以通过设置以下环境变量来验证正在使用的 FlashAttention 版本：

.. code-block:: bash

    NVTE_DEBUG=1 NVTE_DEBUG_LEVEL=1 python your_script.py

已知 FlashAttention-2 的编译过程资源消耗较大，需要大量 RAM（参见 `bug <https://github.com/Dao-AILab/flash-attention/issues/358>`_），这可能导致在安装 Transformer Engine 时出现内存不足错误。请尝试在环境中设置 **MAX_JOBS=1** 以规避此问题。

.. troubleshooting-begin-marker-do-not-remove

故障排除
^^^^^^^^^^^^^^^

**常见问题及解决方案：**

1. **ABI 兼容性问题：**

   * **症状：** 导入 transformer_engine 时出现带有未定义符号的 ``ImportError``
   * **解决方案：** 确保 PyTorch 和 Transformer Engine 使用相同的 C++ ABI 设置进行构建。使用匹配的 ABI 从源码重新构建 PyTorch。
   * **背景：** 如果您使用的 PyTorch 与系统默认的 C++ ABI 不同，可能会遇到这些未定义符号错误。这在容器外通过 pip 安装的 PyTorch 中尤为常见。

2. **缺少头文件或库：**

   * **症状：** CMake 报告缺少头文件（``cudnn.h``, ``cublas_v2.h``, ``filesystem`` 等）
   * **解决方案：** 安装缺失的开发包，或设置环境变量以指向正确的位置：

     .. code-block:: bash

         export CUDA_PATH=/path/to/cuda
         export CUDNN_PATH=/path/to/cudnn

   * 如果 CMake 找不到 C++ 编译器，请设置 ``CXX`` 环境变量。
   * 确保在安装前所有路径都已正确设置。

3. **构建资源问题：**

* **症状：** 编译挂起、系统冻结或内存不足错误
   * **解决方案：** 限制并行构建：

     .. code-block:: bash

         MAX_JOBS=1 NVTE_BUILD_THREADS_PER_JOB=1 pip install ...

4. **详细构建日志：**

   * 获取详细的构建日志以帮助诊断问题：

     .. code-block:: bash

         cd transformer_engine
         pip install -v -v -v --no-build-isolation .

**使用 UV 或虚拟环境时的问题：**

1. **导入错误：**

   * **症状：** 无法导入 ``transformer_engine``
   * **解决方案：** 确保您的 UV 环境已激活，并且您使用的是 ``uv pip install --no-build-isolation <te_pypi_package_or_wheel_or_source_dir>`` 而不是向系统环境进行常规 pip 安装。

2. **cuDNN 子库加载失败：**

   * **症状：** 运行时出现 ``CUDNN_STATUS_SUBLIBRARY_LOADING_FAILED`` 错误
   * **解决方案：** 当 TE 针对容器中的系统 cuDNN 安装进行构建，但虚拟环境中的 pip 包引入了 ``nvidia-cudnn-cu12/cu13`` 的 pip 包时，可能会发生此问题。要解决此问题，在从源代码构建 TE 时，请指定以下环境变量以指向虚拟环境中的 cuDNN。

     .. code-block:: bash

        export CUDNN_PATH=$(pwd)/.venv/lib/python3.12/site-packages/nvidia/cudnn
        export CUDNN_HOME=$CUDNN_PATH
        export LD_LIBRARY_PATH=$CUDNN_PATH/lib:$LD_LIBRARY_PATH

3. **构建 Wheels：**

* **症状：** 常规 TE 安装运行正常，但 UV wheel 构建在运行时失败。
   * **解决方案：** 确保使用 ``uv build --wheel --no-build-isolation -v`` is used during the wheel build as well as the pip installation of the wheel. Use ``-v`` 以获取详细输出，从而验证 TE 是否引入了与 UV 环境版本不一致的 PyTorch 或 JAX 版本。

**JAX 特定常见问题及解决方案：**

1. **FFI 问题：**

   * **症状：** ``No registered implementation for custom call to <some_te_ffi> for platform CUDA``
   * **解决方案：** 确保 ``--no-build-isolation`` is used during installation. If pre-building wheels, ensure that the wheel is both built and installed with ``--no-build-isolation``。如果使用 UV，请参阅上文“使用 UV 或虚拟环境时的问题”。

.. troubleshooting-end-marker-do-not-remove

Breaking Changes
================

v1.7: PyTorch 的 Padding mask 定义
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
为了统一 Transformer Engine 中所有三个框架中 attention mask 的定义和使用，PyTorch 实现中的 padding mask 已从 `True` 表示包含注意力中的相应位置，更改为表示排除该位置。自 v1.7 起，所有 attention mask 类型遵循相同的定义，其中 `True` 表示屏蔽相应位置，而 `False` 表示在注意力计算中包含该位置。

此变更的一个示例如下，

.. code-block:: bash

# 对于包含 3 个序列的批次，其中 `a`s、`b`s 和 `c`s 是有用 token
    # 而 `0`s 是填充 token，
    [a, a, a, 0, 0,
     b, b, 0, 0, 0,
     c, c, c, c, 0]
    # 该批次在 v1.7 之前的填充掩码为，
    [ True,  True,  True, False, False,
      True,  True, False, False, False,
      True,  True,  True,  True, False]
    # 而在 v1.7 及之后版本中应为，
    [False, False, False,  True,  True,
     False, False,  True,  True,  True,
     False, False, False, False,  True]

收敛性
===========

FP8 和 MXFP8 已在不同的模型架构和配置中进行了广泛测试，我们发现 FP8/MXFP8 与 BF16 的训练损失曲线之间**没有显著差异**。FP8 和 MXFP8 的精度也已在下游 LLM 任务（例如 LAMBADA 和 WikiText）上得到验证。以下是跨不同框架测试收敛性的模型示例。

+------------+------------------+---------------------------------------------------------------------------------------------------------+
| 模型       | 框架             | 来源                                                                                                    |
+============+==================+=========================================================================================================+
| MPT-1.3B   |  Mosaic Composer | https://www.databricks.com/blog/coreweave-nvidia-h100-part-1                                            |
+------------+------------------+---------------------------------------------------------------------------------------------------------+
| LLama2-7B  |  Alibaba Pai     | https://mp.weixin.qq.com/s/NQT0uKXLbXyh5031zBdeBQ                                                       |
+------------+------------------+---------------------------------------------------------------------------------------------------------+
| LLM-8B     |  Megatron Core   | https://arxiv.org/abs/2506.08027                                                                        |
+------------+------------------+---------------------------------------------------------------------------------------------------------+
| MPT-13B    |  Mosaic Composer | https://www.databricks.com/blog/turbocharged-training-optimizing-databricks-mosaic-ai-stack-fp8         |
+------------+------------------+---------------------------------------------------------------------------------------------------------+
| MoE-16B    |  Megatron Core   | https://arxiv.org/abs/2506.08027                                                                        |
+------------+------------------+---------------------------------------------------------------------------------------------------------+
| LLama2-70B |  Alibaba Pai     | https://mp.weixin.qq.com/s/NQT0uKXLbXyh5031zBdeBQ                                                       |
+------------+------------------+---------------------------------------------------------------------------------------------------------+

集成
============

Transformer Engine 已与流行的 LLM 框架集成，例如：

* `DeepSpeed <https://github.com/deepspeedai/DeepSpeed>`_
* `Hugging Face Accelerate <https://huggingface.co/docs/accelerate/main/en/usage_guides/low_precision_training#configuring-transformersengine>`_
* `Lightning <https://lightning.ai/docs/pytorch/stable/common/precision.html>`_
* `MosaicML Composer <https://github.com/mosaicml/composer/releases/tag/v0.13.1>`_
* `NVIDIA JAX Toolbox <https://github.com/NVIDIA/JAX-Toolbox>`_
* `NVIDIA Megatron-LM <https://github.com/NVIDIA/Megatron-LM>`_
* `NVIDIA NeMo Megatron Bridge <https://github.com/NVIDIA-NeMo/Megatron-Bridge>`_
* `Amazon SageMaker Model Parallel Library <https://docs.aws.amazon.com/sagemaker/latest/dg/model-parallel-core-features-v2-tensor-parallelism.html>`_
* `Levanter <https://github.com/stanford-crfm/levanter>`_
* `GPT-NeoX <https://github.com/EleutherAI/gpt-neox>`_
* `Hugging Face Nanotron <https://github.com/huggingface/nanotron>`_

贡献
============

我们欢迎对 Transformer Engine 的贡献！若要为 Transformer Engine 做出贡献并提交拉取请求，
请遵循 `<CONTRIBUTING.rst>`_ 指南中概述的准则。

论文
======

* `Attention original paper <https://proceedings.neurips.cc/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf>`_
* `Megatron-LM tensor parallel <https://arxiv.org/pdf/1909.08053.pdf>`_
* `Megatron-LM sequence parallel <https://arxiv.org/pdf/2205.05198.pdf>`_
* `FP8 Formats for Deep Learning <https://arxiv.org/abs/2209.05433>`_

视频
======

* `Stable and Scalable FP8 Deep Learning Training on Blackwell | GTC 2025 <https://www.nvidia.com/en-us/on-demand/session/gtc25-s72778/>`_
* `Blackwell Numerics for AI | GTC 2025 <https://www.nvidia.com/en-us/on-demand/session/gtc25-s72458/>`_
* `Building LLMs: Accelerating Pretraining of Foundational Models With FP8 Precision | GTC 2025 <https://www.nvidia.com/gtc/session-catalog/?regcode=no-ncid&ncid=no-ncid&tab.catalogallsessionstab=16566177511100015Kus&search=zoho#/session/1726152813607001vnYK>`_
* `From FP8 LLM Training to Inference: Language AI at Scale | GTC 2025 <https://www.nvidia.com/en-us/on-demand/session/gtc25-s72799/>`_
* `What's New in Transformer Engine and FP8 Training | GTC 2024 <https://www.nvidia.com/en-us/on-demand/session/gtc24-s62457/>`_
* `FP8 Training with Transformer Engine | GTC 2023 <https://www.nvidia.com/en-us/on-demand/session/gtcspring23-s51393>`_
* `FP8 for Deep Learning | GTC 2023 <https://www.nvidia.com/en-us/on-demand/session/gtcspring23-s52166/>`_
* `Inside the Hopper Architecture | GTC 2022 <https://www.nvidia.com/en-us/on-demand/session/gtcspring22-s42663/>`_

.. |License| image:: https://img.shields.io/badge/License-Apache%202.0-blue.svg
   :target: https://opensource.org/licenses/Apache-2.0

过往新闻
=============

* [06/2025] `Floating Point 8: An Introduction to Efficient, Lower-Precision AI Training <https://developer.nvidia.com/blog/floating-point-8-an-introduction-to-efficient-lower-precision-ai-training/>`_
* [05/2025] `Advanced Optimization Strategies for LLM Training on NVIDIA Grace Hopper <https://developer.nvidia.com/blog/advanced-optimization-strategies-for-llm-training-on-nvidia-grace-hopper/>`_
* [03/2025] `Stable and Scalable FP8 Deep Learning Training on Blackwell | GTC 2025 <https://www.nvidia.com/en-us/on-demand/session/gtc25-s72778/>`_
* [03/2025] `Measure and Improve AI Workload Performance with NVIDIA DGX Cloud Benchmarking <https://developer.nvidia.com/blog/measure-and-improve-ai-workload-performance-with-nvidia-dgx-cloud-benchmarking/>`_

.. image:: docs/examples/comparison-fp8-bf16-training-nvidia-dgx-cloud-benchmarking-performance-explorer.jpg
  :width: 600
  :alt: 在 NVIDIA DGX Cloud Benchmarking Performance Explorer 中查看的 FP8 与 BF16 训练对比

* [02/2025] `Understanding the Language of Life's Biomolecules Across Evolution at a New Scale with Evo 2 <https://developer.nvidia.com/blog/understanding-the-language-of-lifes-biomolecules-across-evolution-at-a-new-scale-with-evo-2/>`_
* [02/2025] `NVIDIA DGX Cloud Introduces Ready-To-Use Templates to Benchmark AI Platform Performance <https://developer.nvidia.com/blog/nvidia-dgx-cloud-introduces-ready-to-use-templates-to-benchmark-ai-platform-performance/>`_
* [01/2025] `Continued Pretraining of State-of-the-Art LLMs for Sovereign AI and Regulated Industries with iGenius and NVIDIA DGX Cloud <https://developer.nvidia.com/blog/continued-pretraining-of-state-of-the-art-llms-for-sovereign-ai-and-regulated-industries-with-igenius-and-nvidia-dgx-cloud/>`_
* [11/2024] `Developing a 172B LLM with Strong Japanese Capabilities Using NVIDIA Megatron-LM <https://developer.nvidia.com/blog/developing-a-172b-llm-with-strong-japanese-capabilities-using-nvidia-megatron-lm/>`_
* [11/2024] `How FP8 boosts LLM training by 18% on Amazon SageMaker P5 instances <https://aws.amazon.com/blogs/machine-learning/how-fp8-boosts-llm-training-by-18-on-amazon-sagemaker-p5-instances/>`_
* [11/2024] `Efficiently train models with large sequence lengths using Amazon SageMaker model parallel <https://aws.amazon.com/blogs/machine-learning/efficiently-train-models-with-large-sequence-lengths-using-amazon-sagemaker-model-parallel/>`_
* [09/2024] `Reducing AI large model training costs by 30% requires just a single line of code from FP8 mixed precision training upgrades <https://company.hpc-ai.com/blog/reducing-ai-large-model-training-costs-by-30-requires-just-a-single-line-of-code-from-fp8-mixed-precision-training-upgrades>`_
* [05/2024] `Accelerating Transformers with NVIDIA cuDNN 9 <https://developer.nvidia.com/blog/accelerating-transformers-with-nvidia-cudnn-9/>`_
* [03/2024] `Turbocharged Training: Optimizing the Databricks Mosaic AI stack with FP8 <https://www.databricks.com/blog/turbocharged-training-optimizing-databricks-mosaic-ai-stack-fp8>`_
* [03/2024] `FP8 Training Support in SageMaker Model Parallelism Library <https://docs.aws.amazon.com/sagemaker/latest/dg/model-parallel-release-notes.html>`_
* [12/2023] `New NVIDIA NeMo Framework Features and NVIDIA H200 <https://developer.nvidia.com/blog/new-nvidia-nemo-framework-features-and-nvidia-h200-supercharge-llm-training-performance-and-versatility/>`_

.. image:: docs/examples/H200-NeMo-performance.png
  :width: 600
  :alt: H200

* [11/2023] `Inflection-2: The Next Step Up <https://inflection.ai/inflection-2>`_
* [11/2023] `Unleashing The Power Of Transformers With NVIDIA Transformer Engine <https://lambda.ai/blog/unleashing-the-power-of-transformers-with-nvidia-transformer-engine>`_
* [11/2023] `Accelerating PyTorch Training Workloads with FP8 <https://towardsdatascience.com/accelerating-pytorch-training-workloads-with-fp8-5a5123aec7d7>`_
* [09/2023] `Transformer Engine added to AWS DL Container for PyTorch Training <https://github.com/aws/deep-learning-containers/pull/3315>`_
* [06/2023] `Breaking MLPerf Training Records with NVIDIA H100 GPUs <https://developer.nvidia.com/blog/breaking-mlperf-training-records-with-nvidia-h100-gpus/>`_
* [04/2023] `Benchmarking Large Language Models on NVIDIA H100 GPUs with CoreWeave (Part 1) <https://www.databricks.com/blog/coreweave-nvidia-h100-part-1>`_