# Transformer Engine v2.12 Release Notes ## Key Features and Enhancements - Made miscellaneous improvements and fixes to the documentation. - [C] Improved performance of NVFP4 quantization kernels. ([#2412](https://github.com/NVIDIA/TransformerEngine/pull/2412)) - [C] Documented environment variables. ([#2552](https://github.com/NVIDIA/TransformerEngine/pull/2552)) - [PyTorch] Added fused permute+pad and unpermute+unpad operations for FP8 optimization. ([#1921](https://github.com/NVIDIA/TransformerEngine/pull/1921)) - [PyTorch] Improved the performance in CPU-limited scenarios. - [PyTorch] Added support for Sliding Window Attention (left, right) with fused attention. ([#2477](https://github.com/NVIDIA/TransformerEngine/pull/2477)) - [PyTorch] Improved the performance of MXFP8 and NVFP4 by fusing the swizzling into the quantization ([#2486](https://github.com/NVIDIA/TransformerEngine/pull/2486)) - [PyTorch] Added cudagraph support for activation recomputation. ([#2518](https://github.com/NVIDIA/TransformerEngine/pull/2518)) - [JAX] Added a tutorial for integrating TE/JAX quantization into existing frameworks. ([#2423](https://github.com/NVIDIA/TransformerEngine/pull/2423)) - [JAX] Added custom partitioning for permutation primitives. ([#2591](https://github.com/NVIDIA/TransformerEngine/pull/2591)) ## Fixed Issues - [C] Fixed SM120 compilation with CUDA 12. ([#2482](https://github.com/NVIDIA/TransformerEngine/pull/2482)) - [C] Fixed overflow in padding and unpadding kernels. ([#2548](https://github.com/NVIDIA/TransformerEngine/pull/2548)) - [C] Fixed a numerical issue in ``sort_chunks_by_index``. ([#2566](https://github.com/NVIDIA/TransformerEngine/pull/2566)) - [C] Fixed a numerical issue in swizzling blockwise E8 scales. ([#2589](https://github.com/NVIDIA/TransformerEngine/pull/2589)) - [PyTorch] Fixed an AttributeError issue when checkpointing the model with MXFP8 parameters. ([#2427](https://github.com/NVIDIA/TransformerEngine/pull/2427)) - [PyTorch] Fixed cross-entropy loss calculation when some tokens are ignored. ([#2476](https://github.com/NVIDIA/TransformerEngine/pull/2476)) - [PyTorch] Fixed ``Float8Tensor.contiguous`` autograd support. ([#2533](https://github.com/NVIDIA/TransformerEngine/pull/2533)) - [PyTorch] Fixed multiple CPU offloading issues. ([#2535](https://github.com/NVIDIA/TransformerEngine/pull/2535)) - [PyTorch] Fixed uninitialized ``permuted_scale`` values. ([#2547](https://github.com/NVIDIA/TransformerEngine/pull/2547)) - [PyTorch] Fixed FP8 quantization for the second MLP in ``LayerNormMLP``. ([#2577](https://github.com/NVIDIA/TransformerEngine/pull/2577)) - [PyTorch] Fixed ONNX tests and added FP8 attention export support. ([#2598](https://github.com/NVIDIA/TransformerEngine/pull/2598)) - [JAX] Removed unused TE DPA dtype handling to improve cuDNN backend dtype detection. ([#2485](https://github.com/NVIDIA/TransformerEngine/pull/2485)) - [JAX] Fixed segment-position calculation from segment IDs in `SequenceDescriptor` class. ([#2523](https://github.com/NVIDIA/TransformerEngine/pull/2523)) - [JAX] Fixed bugs in permutation custom partitioning. ([#2617](https://github.com/NVIDIA/TransformerEngine/pull/2617)) - [JAX] Fixed issue in encoder and MNIST examples due to dataset path moving. ([#2625](https://github.com/NVIDIA/TransformerEngine/pull/2625)) ## Breaking Changes in This Release No breaking changes in this release. ## Deprecated Features No features deprecated in this release.

# Transformer Engine v2.11 Release Notes ## Key Features and Enhancements - [PyTorch] Enabled the reference Current Scaling recipe for FP8 training. ([#2368](https://github.com/NVIDIA/TransformerEngine/pull/2368)) - [PyTorch] Improved Random Hadamard Transform (RHT) device tensor caching to reduce memory allocations and improve performance for NVFP4 quantization. ([#2395](https://github.com/NVIDIA/TransformerEngine/pull/2395)) - [PyTorch] Implemented selective activation checkpointing for LayerNormMLP module ([#2311](https://github.com/NVIDIA/TransformerEngine/pull/2311)) - [C, PyTorch, JAX] Improved performance of MXFP8 quantization. ([#2062](https://github.com/NVIDIA/TransformerEngine/pull/2062)) - [C, PyTorch] Improved performance of NVFP4 quantization. ([#2351](https://github.com/NVIDIA/TransformerEngine/pull/2351)) - [PyTorch] Improved FSDP2 all-gather performance and added support for FusedAdam optimizer with FSDP2. ([#2370](https://github.com/NVIDIA/TransformerEngine/pull/2370)) - [PyTorch] Extended debug tools to support GroupedLinear layers. ([#1953](https://github.com/NVIDIA/TransformerEngine/pull/1953)) - [JAX] Added Triton kernel bindings for JAX, enabling custom Triton kernels in JAX workflows. ([#2437](https://github.com/NVIDIA/TransformerEngine/pull/2437)) - [C] Introduced experimental NVTEGroupedTensor class and helper functions. ([#2388](https://github.com/NVIDIA/TransformerEngine/pull/2388)) - [C, PyTorch, JAX] Added FP8 support for primary weights in MXFP8 format with partial casting and amax calculations. ([#2055](https://github.com/NVIDIA/TransformerEngine/pull/2055)) - [JAX] Added support for context parallelism (CP) for THD format and sliding window attention (SWA) using all-gather (AG), striped load balancing with stripe size greater than 1. ([#2379](https://github.com/NVIDIA/TransformerEngine/pull/2379)) - [JAX] Implemented JAX primitives for token permutation operations on single GPU for mixture-of-experts routing. ([#2473](https://github.com/NVIDIA/TransformerEngine/pull/2473)) - [PyTorch] Added THD format support for max_logit clipping and MuonClip gradient clipping operations. ([#2480](https://github.com/NVIDIA/TransformerEngine/pull/2480)) ## Fixed Issues - [PyTorch] Fixed a numerical issue when noncontiguous tensor was passed to cross_entropy backward pass. ([#2402](https://github.com/NVIDIA/TransformerEngine/pull/2402)) - [PyTorch] Fixed CUDA graph execution order for backward weight gradient computation when using chunked layers. ([#2376](https://github.com/NVIDIA/TransformerEngine/pull/2376)) - [C] Fixed runtime library loading logic to properly handle missing dependencies and load order. ([#2297](https://github.com/NVIDIA/TransformerEngine/pull/2297)) - [Jax] Removed use of scan loop as the default for ring attention due for improved performance ([#2503](https://github.com/NVIDIA/TransformerEngine/pull/2503)). ## Breaking Changes in This Release No breaking changes in this release. ## Deprecated Features No features deprecated in this release.

# Release v2.10 ## Key Features and Enhancements * [PyTorch] Added support for the NVFP4 training recipe for the `GroupedLinear` module. * [PyTorch] Added support for CUDA graphs when using quantized weights with Tensor Parallelism. * [PyTorch] Added support for CUDA graphs when using `delay_wgrad_compute`. * [PyTorch] Expanded debug tools to support more statistics. * [PyTorch] Reduced the overhead of using debug tools. * [PyTorch] Added support for clamped SwiGLU in the `TransformerLayer` module. * [PyTorch] Added backwards compatibility for older Megatron-Core versions by introducing a `keep_columnwise` parameter to `cast_master_weights_to_fp8` and related helper functions. * [PyTorch] Added a `reset` interface to `make_graphed_callables` that clears internal CUDA graphs before distributed process group cleanup, preventing hangs. * [PyTorch] Added support for FSDP2 with quantized weights. * [PyTorch] Added support for Sliding Window Attention (SWA) with Context Parallelism with THD input format. * [PyTorch] Integrated Flash Attention's `num_splits` parameters into the attention backend. * [PyTorch] Made various improvements to mitigate CPU overhead, especially for the `GroupedLinear` module. * [C][PyTorch] Enabled RoPE (Rotary Position Embedding) application with position offsets during training, removing the previous restriction that `start_positions` could only be used with `cp_size=1` (context parallelism disabled). * [Jax] Added options to disable Stochastic Rounding, Randomized Hadamard Transform, and 2D weight quantization in the NVFP4 training recipe. * [Jax] Improved performance by using Transformer Engine quantization when fused normalization or fused activation are disabled. * [Jax] Performance Improvement for NVFP4 via TE kernels for scaling factor swizzles. * [Jax] Added support for checkpointing quantization operations in JAX. * [Jax] Added support for sink attention. * [Jax] Added support for concurrent use of Data Parallelism (DP) and Fully-Sharded Data Parallelism (FSDP). ## Fixed Issues * Fixed an occasional crash when loading cuDNN library during runtime. * [C] Fixed an out of bounds access in the NVFP4 dequantization kernel. * [C] Fixed a numerical error in the amax computation in normalization kernels. * [PyTorch] Fixed a crash in the permute kernel when using `triton` v3.5. * [PyTorch] Fixed a numerical issue when using gradient accumulation fusion with FSDP. * [PyTorch] Fixed a crash when exporting modules via ONNX when using RMSNorm. * [Jax] Fixed a partitioning issue for the NVFP4 training recipe with 1D Mesh. * [Jax] Fixed a bug where the bias parameter could be added twice when using unfused attention backend. * [Jax] Fixed a sharding bug in ring attention primitives when using packed sequences where segment position tensors were not properly sharded to match their corresponding segment ID tensors. * [PyTorch][Jax] Fixed various logical issues in the backend selection process for attention. ## Known Issues in This Release There are no known issues in this release. ## Breaking Changes in This Release * [Jax] Default value for `intermediate_dropout` changed from 0.1 to 0.0. * [Jax] Default value for `return_layernorm_output` changed from `True` to `False`. * [Jax] Default activation changed from ReLU to GeLU. * [Jax] Default input type for `DotProductAttention` is changed to BSHD. ## Deprecated Features No features are deprecated in this release.

# Release Notes – Release 2.8 ## Key Features and Enhancements - [C][PyTorch] Added support for the NVFP4 training recipe. - [C][PyTorch] Added support for FP8 attention with the current scaling recipe. - [PyTorch] Added support for mixing recipes for different modules when using the make_graphed_callables function. - [C] Added 8-bit RNG support to the dropout kernel. - [C][PyTorch] Added the nvte_rmsnorm_bwd_add function to the C API and added support for fusing RMSNorm and add operation in the sequential Transformer Engine operations API. - [C] Added more robust error checking and handling when calling CUDA and driver APIs. - [C][PyTorch] Added support for using FP8 and non-FP8 quantization modes in the same model when overlapping tensor parallel communication and GEMM using userbuffers. - [PyTorch] Added support for the qgeglu and sreglu activation in the Transformer Engine fused operations API and the LayerNormMLP module. - [C][PyTorch] Added support for FP8 GEMM output for the MXFP8 and current scaling recipe. - [PyTorch] Added support for FP8 all-gather when using Tensor Parallel with the GroupedLinear module. - [PyTorch] Added support for the current scaling FP8 recipe for module export via ONNX. - [PyTorch] Made miscellaneous improvements to MoE workloads to reduce CPU overhead. - [PyTorch] Improved performance of CUDA graphs using FP8 weight cache in quantization kernels. - [PyTorch] Added support for FlashAttention v3 for MLA with context parallelism. - [PyTorch] Added support activation CPU offloading for Transformer Engine sequential operations API. - [PyTorch] Made miscellaneous performance improvements when using RoPE (rotary positional embeddings) - [C] Added support for BF16 and FP32 inputs to the kernel that calculates auxiliary loss for MoE. - [C] Added support for sink attention from cuDNN. - [Jax] Fused swizzling operation for the scaling factor inverse and transpose calculation of the data. ## Fixed Issues - [Jax] Fixed a crash when the user calls global_shard_guard before setting the JAX mesh. - [Jax] Fixed an issue in the mesh logic such that if an axis is undefined in the mesh, Transformer Engine still applies the sharding constraint for the given tensor on other axes instead of skipping. - [Jax] Fixed a crash in GroupedScaledTensor due to incorrect arguments being passed. - [PyTorch] Fixed a bug in the cross entropy loss kernel that resulted in vanishing gradients. - [C][PyTorch] Fixed incorrect calculation of tensor parallel rank when using userbuffers. - [PyTorch] Fixed redundant memory overheads when using FP8 all-gather with sequence parallelism. ## Known Issues in This Release - [PyTorch] For distributed workloads using the Float8CurrentScaling recipe without FP8 attention, there are some performance overheads due to redundant amax reductions across the tensor parallel and context parallel groups. This issue has been fixed (https://github.com/NVIDIA/TransformerEngine/pull/2234), and will be available in the next release (v2.9). As a workaround, you can run the workload with export NVTE_DPA_FP8_RECIPE="F16" in the environment. ## Breaking Changes in This Release There are no breaking changes in this release. ## Deprecated Features There are no deprecated features in this release.

# Release v2.7 ## Key Features and Enhancements - [PyTorch] Added support for applying LayerNorm and RMSNorm to key and query tensors. - [PyTorch] Improved performance for FP8 per tensor current scaling recipe by fusing amax computation into activation kernel. - [PyTorch] Added support for multi-tensor swizzle kernels for MXFP8 grouped GEMMs. - [PyTorch] Fused zero-padding and swizzle operation for MXFP8 scale inverses for improved performance. - [PyTorch] Expanded the debug API using `nvdlfw-inpect` in order to log more advanced tensor statistics. - [PyTorch] Reduced the number of calls to CUDA driver for improved performance of the core library. - [Jax] Added new checkpointing policies that allow users to switch to TE GEMMs seamlessly without unnecessary recomputations. - [Core] Added support for cublasMP backend for overlapping TP communication and GEMM. ## Fixed Issues - [PyTorch].Fixed a potential illegal memory access when using TP overlap. - [PyTorch] Fixed the logic for choosing the correct attention backend depending on the cuDNN version. - [PyTorch] Fixed a crash when using CUDA graphs by disabling garbage collection during capture. - [PyTorch] Fixed a bug when using double buffering for CPU offloading. - [PyTorch] Fixed a bug when overlapping gradient reduction and fusing weight gradient accumulation simultaneously. - [PyTorch] Made multiple improvements and fixes to TE sequential API, including expanding supported operations to cover dropout, constant scale, etc. - [PyTorch] Fixed a bug in the `make_graphed_callables` function when applied to multiple modules with different input requirements. - [PyTorch] Fixed the crash in the permute operation when running with the FP8 datatype for input sizes requiring padding. - [PyTorch] Fixed a bug when using the Triton cross entropy kernel with cuda graphs. - [PyTorch] Fixed a bug when exporting an MXFP8 model to ONNX. - [PyTorch/Core] Disabled cuDNN attention backend for cuDNN v9.12 onwards on blackwell if the user requests a deterministic config. - [Core] Fixed integer overflow in quantization kernels when computing offsets for large tensors. - [Jax] Fixed partition rules for GEMM to correctly handle sequence parallelism. - [Jax] Fixed sharding specs for TE GEMM custom call operands when using DP. - [Jax] Fixed a crash when using `GroupedQuantizeFFI` with cuda graphs - [Jax] Fixed the fused_attn sharding constraint so that it can be used under the JAX shard_map.. ## Known Issues in This Release There are no known issues in this release. ## Breaking Changes in This Release The deprecated device_id argument for multi tensor C APIs has been removed. ## Deprecated Features There are no deprecated features in this release.

# Release Notes – Release 2.4 ## Key Features and Enhancements * [Jax] Added support for Float8CurrentScaling recipe. * [Jax] Added support for logical partitioning axes in TE Flax modules. * [Core] Added multiple experimental functions to the C API. * [PyTorch] Improved performance by caching device properties. * [PyTorch] Made miscellaneous minor improvements to reduce memory consumption for certain workloads. * [PyTorch] Added support for MXFP8 recipe when using userbuffers for overlapping TP communication and GEMMs. * [PyTorch] Reduced the binary size of the framework extension library from 108 MB to 2 MB. * [PyTorch] Introduced a Boolean parameter, rotary_pos_interleaved, in the MultiheadAttention and TransformerLayer modules for interleaved RoPE. * [PyTorch] Added support for ignoring tokens in the cross-entropy loss function. * [PyTorch] Added support for switching among all supported FP8 recipes during training and checkpointing. * [PyTorch] Added various debugging tools via NVIDIA-DL-Framework-Inspect. ## Fixed Issues * [PyTorch] Fixed a numerical issue when using activation recompute with FP8. * [PyTorch] Fixed incorrect output dimensions when using return_layernorm_output in the LayerNormLinear and LayerNormMLP modules. * [PyTorch] Fixed a numerical bug when using sequence parallelism in the LayerNorm and RMSNorm modules with Megatron-LM. * [PyTorch/Jax] Fixed miscellaneous crashes at import time due to library loading. * [Jax] Fixed a crash due to partitioning error when using the LayerNorm or LayerNormMLP module with tensor parallelism. * [PyTorch] Fixed an issue where GIL was held during the entirety of C API calls from the framework extensions, including during NVIDIA® CUDA® kernel execution. ## Known Issues in This Release There are no known issues in this release. ## Breaking Changes in This Release There are no breaking changes in this release. ## Deprecated Features There are no deprecated features in this release. ## Miscellaneous There are no miscellaneous issues in this release.

# Release Notes – Release 2.3 ## Key Features and Enhancements - [PyTorch] Sped up import of transformer_engine module by moving to a lazy compilation of functions using torch.compile. - [PyTorch] Enabled FP8 weights when using FSDP. - [C][PyTorch] Added support for Float8 block scaling recipe, as used in the [Deepseek v3 paper](https://arxiv.org/abs/2412.19437), for Hopper GPUs. - [PyTorch] Made miscellaneous fixes to reduce CPU overhead. - [PyTorch] Added support for CPU offloading for activation tensors when using FP8 attention. - [PyTorch] Enabled MXFP8 recipe for the GroupedLinear module. - [PyTorch] Added a feature to support decoupling the weight gradient compute from the backward function of Transformer Engine modules. This allows users to call backward wgrad and gives them finer-grained control over when gradients are called to support certain advanced parallelism/overlap schemes. - [PyTorch] Added support for staggered application of rope embedding to a sequence of inputs in a batch, depending on their starting positions. - [All] Added support for RTX 5090. ## Fixed Issues - [PyTorch] Fixed a numerical bug with use of custom DDP from megatron-core. - [PyTorch] Fixed a crash when using the checkpoint method for activation recompute on non-Transformer Engine modules. ## Known Issues in This Release There are no known issues in this release. ## Breaking Changes in This Release - [Jax] Praxis layers have been removed, as PAXML is no longer supported. ## Deprecated Features - The installation for Transformer Engine now requires use of the –no-build-isolation flag when using PyPI package or building from source. Support for installations with build isolation will be removed in a future release. - [PyTorch] CPU offloading weight tensors is deprecated.

# Release Notes – Release 2.2 ## Key Features and Enhancements - [PyTorch] Added support for per-tensor current scaling recipe. - [PyTorch] Implemented cross-entropy loss with support for splitting computation across multiple devices. - [PyTorch] Added support for CPU offloading with Megatron-Core style distributed optimizers. - [PyTorch] Added support for KV cache for FusedAttention, FlashAttention, and UnfusedDotProductAttention backends. - [PyTorch] Improved bulk TP communication overlap by launching GEMMs on lower priority streams. - [C/PyTorch] Improved performance for P2P-based Tensor Parallel (TP) communication overlap. - [Jax] Added support for THD format with ring attention. - [Jax] Improved performance and memory usage for causal mask in the cuDNN attention backend. - [C] Added multi-node support for NVIDIA® NVLink for TP overlap with userbuffers. ## Fixed Issues - [PyTorch] Fixed convergence when using context parallelism with a fused attention backend. - [PyTorch] Fixed a crash using GroupedLinear when the last input has no tokens. - [PyTorch] Made miscellaneous fixes to improve overall performance of the MXFP8 recipe. - [PyTorch] Reintroduced support for return_bias argument to all modules, which was silently ignored in v2.0 and v2.1. - [PyTorch] Reintroduced support for FP8 communication for overlapping reduce-scatter and GEMM when using TP overlap with userbuffers. - [PyTorch] Fixed gradient accumulation fusion in the LayerNormMLP module. - [C/PyTorch] Made miscellaneous numerical fixes to the fused attention backend. - [C] Avoided creating a new cublasLtHandle for every GEMM call to avoid memory leaks. - [Jax] Fixed shape and sharding inference in fused-attention C++ extension. - [Jax] Fixed an import error in the encoder example. ## Known Issues in This Release - RTX 5090 is currently unsupported for FP8 execution. Support will be added in v2.3.0. - Transformer Engine may crash when it is installed via the PyPI registry but is run in an environment with CUDA version < 12.8. A temporary workaround is to install from source until the issue is fixed. ## Breaking Changes in This Release - [PyTorch] The deprecated interval argument for the DelayedScaling recipe has been removed. - [PyTorch] There are multiple breaking changes in the InferenceParams class. - New arguments num_heads_kv, head_dim_k, and dtype are required during initialization. - The user must call a pre_step method to update the InferenceParams state. - The swap_key_value_dict method has been removed, as the step method now automatically reorders the key/value sequences according to their batch indices. ## Deprecated Features There are no deprecated features in this release. ## Miscellaneous - [PyTorch] The minimum required PyTorch version is changed to 2.1.