## Features ### Qwen 3.6 integration <img width="1536" height="1024" alt="ChatGPT Image Apr 26, 2026 at 11_16_18 AM" src="https://github.com/user-attachments/assets/789aad15-03b2-4ece-9828-d5c1dfed1f1e" /> TRL v1.3 ships training support for the new **Qwen 3.6** family (`Qwen/Qwen3.6-27B`, `Qwen/Qwen3.6-35B-A3B`). Qwen 3.6 reuses the `Qwen3_5Moe*` architecture but ships a slightly different chat template (adds a `preserve_thinking` flag, tweaks tool-arg stringification), so exact-string template matching needed updates across the stack. What landed: - **Chat templates**: `qwen3_6.jinja` (verbatim from upstream) and `qwen3_6_training.jinja` (prefix-preserving + `{% generation %}` markers for `assistant_only_loss=True`) - **Response schema**: routes to the existing `qwen3_5_schema` for tool-call parsing — output format unchanged - **Tiny test models** for VLM training: `tiny-Qwen3_5MoeForConditionalGeneration-3.6` (with MoE-specific shrinking) - **Test matrix** updated across SFT/DPO/GRPO/RLOO `test_(train|training)_vlm` cases ```python from trl import SFTConfig, SFTTrainer trainer = SFTTrainer( model="Qwen/Qwen3.6-27B", args=SFTConfig(assistant_only_loss=True), # works out of the box train_dataset=dataset, ) trainer.train() ``` Tool-calling agent training also works end-to-end via the existing Qwen 3.5 response schema: ```python from trl import GRPOConfig, GRPOTrainer def multiply(a: int, b: int) -> int: """ Multiplies two integers. Args: a: The first integer. b: The second integer. Returns: The product of the two integers. """ return a * b trainer = GRPOTrainer( model="Qwen/Qwen3.6-27B", reward_funcs=my_reward_fn, args=GRPOConfig(...), train_dataset=dataset, tools=[multiply], ) trainer.train() ``` by @qgallouedec in https://github.com/huggingface/trl/pull/5642 ### New experimental TPO trainer <img width="711" height="177" alt="Screenshot 2026-04-26 at 11 37 28 AM" src="https://github.com/user-attachments/assets/6090212e-5c95-45c1-b137-87333d91daa6" /> A new experimental `TPOTrainer` implements [Triple Preference Optimization](https://huggingface.co/papers/2405.16681), which augments DPO with a `reference` (gold) completion alongside `chosen`/`rejected`. The paper reports +7-19 points over DPO/SimPO on Arena-Hard, MixEval-Hard, MMLU-Pro and GSM8K, with less data. ```python from trl.experimental.tpo import TPOConfig, TPOTrainer trainer = TPOTrainer( model="Qwen/Qwen3-0.6B", args=TPOConfig(output_dir="Qwen3-0.6B-TPO"), train_dataset=load_dataset("tpo-alignment/triple-preference-ultrafeedback-40K", split="train"), ) trainer.train() ``` by @kashif in https://github.com/huggingface/trl/pull/5506 ### Speculative decoding in `trl vllm-serve` A new `--speculative_config` JSON flag exposes vLLM's [speculative decoding](https://docs.vllm.ai/en/latest/features/spec_decode.html) directly through `trl vllm-serve` — works with native MTP heads (Qwen3 Next), Eagle3 drafts, etc. — without forking the serve script. ```bash # Qwen3 native MTP (no extra draft model) trl vllm-serve --model Qwen/Qwen3-Next-80B-A3B-Instruct \ --speculative_config '{"method": "qwen3_next_mtp", "num_speculative_tokens": 5}' # Eagle3 draft model trl vllm-serve --model Qwen/Qwen3-32B \ --speculative_config '{"model": "RedHatAI/Qwen3-32B-speculator.eagle3", "method": "eagle3", "num_speculative_tokens": 3}' ``` by @Ofir408 in https://github.com/huggingface/trl/pull/5605 ### KTO ↔ DPO alignment: nearing the finish line Twelve more alignment PRs this cycle, bringing `KTOTrainer` and `DPOTrainer` essentially into structural parity. Notable shifts include moving completion assembly out of `_prepare_dataset` into a new `DataCollatorForKTO`, inlining the two-pass tokenization into a single pass, removing BOS/EOS handling, and supporting `IterableDataset` and dict `eval_dataset`. The goal — promoting **KTO out of experimental and into stable** — is now within reach for an upcoming release. PRs (all by @albertvillanova): #5582, #5578, #5579, #5583, #5587, #5599, #5601, #5600, #5606, #5612, #5632, #5635 ### More `{% generation %}` training chat templates Three more model families gain training-compatible chat templates with `{% generation %}` markers, so `assistant_only_loss=True` works out of the box: - **Gemma / Gemma 2** by @ps-abhi in https://github.com/huggingface/trl/pull/5523 - **Phi-3** by @RudrenduPaul in https://github.com/huggingface/trl/pull/5526 - **GLM-4-MoE** by @casinca in https://github.com/huggingface/trl/pull/5519 ### Other * Support processor in `maybe_apply_chat_template` by @albertvillanova in https://github.com/huggingface/trl/pull/5567 * Support VLM processors in `is_chat_template_prefix_preserving` by @qgallouedec in https://github.com/huggingface/trl/pull/5558 * Check prefix preservation at the **token** level (not string level) by @qgallouedec in https://github.com/huggingface/trl/pull/5559 * Drop vLLM 0.11 support by @qgallouedec in https://github.com/huggingface/trl/pull/5549 * Remove `forward_masked_logits` by @qgallouedec in https://github.com/huggingface/trl/pull/5626 * Remove dead token attributes from experimental trainers by @albertvillanova in https://github.com/huggingface/trl/pull/5565 * Set `_tokenizer` as trainer attribute by @albertvillanova in https://github.com/huggingface/trl/pull/5489 * Use `PreTrainedTokenizerBase` for tokenizer type hints by @qgallouedec in https://github.com/huggingface/trl/pull/5629 * Renaming of internal variables: `async_reward_X` to `async_X` by @qgallouedec in https://github.com/huggingface/trl/pull/5616 ## Fixes * **Fix entropy calculation in SFT** — three bugs at once: misaligned by one position (next-token shift), averaged over the wrong tokens (used `attention_mask` instead of `label != -100`), and wrong cross-rank aggregation (unweighted mean instead of sum/count). The reported entropy under `completion_only_loss=True` and sequence parallelism is now correct. Same fix applied to DPO entropy logging. By @qgallouedec in https://github.com/huggingface/trl/pull/5620 * Pass `AsyncGRPOTrainer`'s `processing_class` to `AsyncRolloutWorker` by @xuanduy04 in https://github.com/huggingface/trl/pull/5538 * Fix `generate_tiny_models` for gpt-oss by @albertvillanova in https://github.com/huggingface/trl/pull/5622 * Fix docstring style in vllm-serve script by @albertvillanova in https://github.com/huggingface/trl/pull/5628 * Replace wrong comment about chat template with EOS by @albertvillanova in https://github.com/huggingface/trl/pull/5607 ## Documentation and Examples * Add chat templates page to web docs by @sergiopaniego in https://github.com/huggingface/trl/pull/5581 * Update AsyncGRPO example with GSM8K and tested hyperparameters by @sergiopaniego in https://github.com/huggingface/trl/pull/5580 * Update RapidFire AI integration with FSDP and multi-backend tracking by @kamran-rapidfireAI in https://github.com/huggingface/trl/pull/5618 ## CI * Add doc-builder style check to pre-commit and CI by @albertvillanova in https://github.com/huggingface/trl/pull/5630 * Align and update doc-builder commit hash in CI GitHub Actions by @albertvillanova in https://github.com/huggingface/trl/pull/5631 * Hotfix CI: Add ruff dependency to doc-builder style check by @albertvillanova in https://github.com/huggingface/trl/pull/5634 * Fix CI with dev dependencies for Llava models by @albertvillanova in https://github.com/huggingface/trl/pull/5499 * Add additional model parameters to `TestSupportsToolCalling` for improved coverage by @qgallouedec in https://github.com/huggingface/trl/pull/5537 * Differentiate Phi-3 and Phi-3.5 in tests by @qgallouedec in https://github.com/huggingface/trl/pull/5546 ## New Contributors * @Ofir408 made their first contribution in https://github.com/huggingface/trl/pull/5605 * @ps-abhi made their first contribution in https://github.com/huggingface/trl/pull/5523 ## What's Changed * ⬆️ Bump dev version by @qgallouedec in https://github.com/huggingface/trl/pull/5577 * Support processor in maybe_apply_chat_template by @albertvillanova in https://github.com/huggingface/trl/pull/5567 * Remove dead token attributes from experimental trainers by @albertvillanova in https://github.com/huggingface/trl/pull/5565 * Support VLM processors in `is_chat_template_prefix_preserving` by @qgallouedec in https://github.com/huggingface/trl/pull/5558 * Align KTO with DPO: Align add_model_tags by @albertvillanova in https://github.com/huggingface/trl/pull/5582 * Align KTO with DPO: Align processing_class initialization by @albertvillanova in https://github.com/huggingface/trl/pull/5578 * Align KTO with DPO: Align _prepare_dataset by @albertvillanova in https://github.com/huggingface/trl/pull/5579 * Align KTO with DPO: Align ref_model preparation for distributed training by @albertvillanova in https://github.com/huggingface/trl/pull/5583 * Align KTO with DPO: Make conditional prompt extraction and unpairing in _prepare_dataset by @albertvillanova in https://github.com/huggingface/trl/pull/5587 * Update AsyncGRPO example with GSM8K and tested hyperparameters by @sergiopaniego in https://github.com/huggingface/trl/pull/5580 * [docs] Add chat templates page to web docs by @sergiopaniego in https://github.com/huggingface/trl/pull/5581 * Add additional model parameters to `TestSupportsToolCalling` for improved coverage by @qgallouedec in https://github.com/huggingface/trl/pull/5537 * Fix CI with dev dependencies for Llava models by @albertvillanova in https://github.com/huggingface/trl/pull/5499 * Differentiate Phi-3 and Phi-3.5 in tests by @qgallouedec in https://github.com/huggingface/trl/pull/5546 * Set _tokenizer as trainer attribute by @albertvillanova in https://github.com/huggingface/trl/pull/5489 * Align KTO with DPO: Support dict eval_dataset by @albertvillanova in https://github.com/huggingface/trl/pull/5599 * Align KTO with DPO: Align tokenization by @albertvillanova in https://github.com/huggingface/trl/pull/5601 * Check prefix preservation at the token level by @qgallouedec in https://github.com/huggingface/trl/pull/5559 * Replace wrong comment about chat template with EOS by @albertvillanova in https://github.com/huggingface/trl/pull/5607 * Align KTO with DPO: Support IterableDataset by @albertvillanova in https://github.com/huggingface/trl/pull/5600 * Drop vLLM 0.11 support by @qgallouedec in https://github.com/huggingface/trl/pull/5549 * Align KTO with DPO: Remove maybe_apply_chat_template by @albertvillanova in https://github.com/huggingface/trl/pull/5606 * [TPO] experimental TPO trainer by @kashif in https://github.com/huggingface/trl/pull/5506 * fix: Pass AsyncGRPOTrainer's processing_class to AsyncRolloutWorker by @xuanduy04 in https://github.com/huggingface/trl/pull/5538 * docs: update RapidFire AI integration with FSDP and multi-backend tracking by @kamran-rapidfireAI in https://github.com/huggingface/trl/pull/5618 * Fix generate_tiny_models for gpt-oss by @albertvillanova in https://github.com/huggingface/trl/pull/5622 * Added speculative_config to vllm-serve by @Ofir408 in https://github.com/huggingface/trl/pull/5605 * feat(glm-4-moe): Add `{% generation %}` markers for training chat template by @casinca in https://github.com/huggingface/trl/pull/5519 * Fix docstring style in vllm-serve script by @albertvillanova in https://github.com/huggingface/trl/pull/5628 * feat: add Gemma/Gemma2 training chat templates with generation markers by @ps-abhi in https://github.com/huggingface/trl/pull/5523 * Align KTO with DPO: Inline tokenization, new output format, DataCollatorForKTO by @albertvillanova in https://github.com/huggingface/trl/pull/5612 * feat: add Phi-3 training chat template with generation markers by @RudrenduPaul in https://github.com/huggingface/trl/pull/5526 * Remove `forward_masked_logits` by @qgallouedec in https://github.com/huggingface/trl/pull/5626 * Use `PreTrainedTokenizerBase` for tokenizer type hints by @qgallouedec in https://github.com/huggingface/trl/pull/5629 * Add doc-builder style check to pre-commit and CI by @albertvillanova in https://github.com/huggingface/trl/pull/5630 * Align and update doc-builder commit hash in CI GitHub Actions by @albertvillanova in https://github.com/huggingface/trl/pull/5631 * Align KTO with DPO: Move completion assembly from _prepare_dataset to data collator by @albertvillanova in https://github.com/huggingface/trl/pull/5632 * Hotfix CI: Add ruff dependency to doc-builder style check by @albertvillanova in https://github.com/huggingface/trl/pull/5634 * Fix entropy calculation in SFT by @qgallouedec in https://github.com/huggingface/trl/pull/5620 * Renaming of internal variables: `async_reward_X` to `async_X` by @qgallouedec in https://github.com/huggingface/trl/pull/5616 * Align KTO with DPO: Remove BOS/EOS handling by @albertvillanova in https://github.com/huggingface/trl/pull/5635 * Qwen3.6 integration by @qgallouedec in https://github.com/huggingface/trl/pull/5642 * Release: v1.3 by @qgallouedec in https://github.com/huggingface/trl/pull/5647 ## New Contributors * @Ofir408 made their first contribution in https://github.com/huggingface/trl/pull/5605 * @ps-abhi made their first contribution in https://github.com/huggingface/trl/pull/5523 **Full Changelog**: https://github.com/huggingface/trl/compare/v1.2.0...v1.3.0

## What's Changed * Fix: undefined `current_gradient_accumulation_steps` by @qgallouedec in https://github.com/huggingface/trl/pull/4852 * fix(DeepSeek OPSM): passing correct (vLLM) logprobs by @casinca in https://github.com/huggingface/trl/pull/4857 * Fix SFT training for prompt-completion type and transformers v5 by @qgallouedec in https://github.com/huggingface/trl/pull/4880 * Bugfix: Logprob drift in vLLM serving mode (compared to colocate mode) by @kdubovikov in https://github.com/huggingface/trl/pull/4873 * Fix RewardTrainer's results not reproducible by @liyc-ai in https://github.com/huggingface/trl/pull/4887 ## New Contributors * @kdubovikov made their first contribution in https://github.com/huggingface/trl/pull/4873 * @liyc-ai made their first contribution in https://github.com/huggingface/trl/pull/4887 **Full Changelog**: https://github.com/huggingface/trl/compare/v0.27.0...v0.27.1

## What's Changed * Fix vLLM error for tools usage not supported when running GRPO training by @apalmas-saifh in https://github.com/huggingface/trl/pull/4663 * Fix GRPO config validation in case `num_generations_eval` is specified and different than `num_generations` by @apalmas-saifh in https://github.com/huggingface/trl/pull/4682 ## New Contributors * @apalmas-saifh made their first contribution in https://github.com/huggingface/trl/pull/4663 **Full Changelog**: https://github.com/huggingface/trl/compare/v0.26.0...v0.26.1

## What's Changed * Replace accelerate logging with stdlib in CLI by @lewtun in https://github.com/huggingface/trl/pull/4512 * Add temporary workaround for `lr_scheduler_kwargs` dtype issue in Transformers 4.57.0 by @qgallouedec in https://github.com/huggingface/trl/pull/4513 **Full Changelog**: https://github.com/huggingface/trl/compare/v0.25.0...0.25.1

## Features * Add accuracy reward by @pramodith in https://github.com/huggingface/trl/pull/4270 * Add support for `token_type_ids` in `DPOTrainer` by @aweers in https://github.com/huggingface/trl/pull/4285 * 💰 `RichProgressCallback` enhancement by @qgallouedec in https://github.com/huggingface/trl/pull/4245 * Include `chat_template_kwargs` in `apply_chat_template` by @cmpatino in https://github.com/huggingface/trl/pull/4233 * 🏷️ Account for `token_type_ids` in `DataCollatorForVisionLanguageModeling` by @qgallouedec in https://github.com/huggingface/trl/pull/4190 * 🎨 Support mixing image+text and text-only examples by @qgallouedec in https://github.com/huggingface/trl/pull/4203 * 🎁 `RewardTrainer` refactor by @qgallouedec in https://github.com/huggingface/trl/pull/4093 * 🎞️ Support sequence classification models in `clone_chat_template` by @qgallouedec in https://github.com/huggingface/trl/pull/4097 * ✨ Add logging for training completion and model saving in training scripts by @qgallouedec in https://github.com/huggingface/trl/pull/4048 * 🖨️ Print rich table for messages by @qgallouedec in https://github.com/huggingface/trl/pull/4160 * 😴 Add `vllm_enable_sleep_mode` to RLOO Trainer by @sergiopaniego in https://github.com/huggingface/trl/pull/4107 * 📽 Multi image support for GRPO/RLOO by @qgallouedec in https://github.com/huggingface/trl/pull/4113 * 👁️ Add VLM support to RLOO trainer by @behroozazarkhalili in https://github.com/huggingface/trl/pull/4067 * ℹ️ Enable XPU for vLLM client by @jiqing-feng in https://github.com/huggingface/trl/pull/4031 * 🧶 feat: Add WeaveCallback for W&B Weave integration by @parambharat in https://github.com/huggingface/trl/pull/4089 ## Fixes * [Online-DPO] fix the completion_len == max_new_tokens crash by @kashif in https://github.com/huggingface/trl/pull/4193 * Fix entropy and accuracy calculation for prompt_tuning techniques. by @pramodith in https://github.com/huggingface/trl/pull/4196 * Fix prompt-completion labeling with add_generation_prompt and warning by @behroozazarkhalili in https://github.com/huggingface/trl/pull/4201 * 🌡️ Have vLLM return processed (temperature scaled) log probs by @YonatanGideoni in https://github.com/huggingface/trl/pull/4163 * Fix handling of f_divergence_type in DPO by @albertvillanova in https://github.com/huggingface/trl/pull/4171 * ⚡ Fix Flash Attention x Padding-Free loss by @qgallouedec in https://github.com/huggingface/trl/pull/4170 * Pass required token_type_ids by @albertvillanova in https://github.com/huggingface/trl/pull/4148 * 👩‍🦯 Fix usage of VLM using text only by @SamuelBarryCS in https://github.com/huggingface/trl/pull/4080 * ⚓ [vllm] ensure MASTER_ADDR/MASTER_PORT are set safely by @kashif in https://github.com/huggingface/trl/pull/4057 * 📤 Fix a dataset loading bug in scripts by @singing-cat in https://github.com/huggingface/trl/pull/4124 * 🐯 fix: use_liger_kernel with IterableDataset by @jue-jue-zi in https://github.com/huggingface/trl/pull/4087 * [GKD] Fix `batchmean` reduce op in GKDTrainer's loss by @cmpatino in https://github.com/huggingface/trl/pull/4105 * Fix get_peft_model() so that prepare_model_for_kbit_training does not reapply to an instance of PeftModel, thus freezing all the layers by @Hoesu in https://github.com/huggingface/trl/pull/4081 * Aux loss is already included in the loss returned by Transformers by @pramodith in https://github.com/huggingface/trl/pull/4078 * ♨️ [GRPO] Fix potential hang in `get_high_entropy_mask` by @akakakakakaa in https://github.com/huggingface/trl/pull/4041 ## Documentation * Remove logging.md: trainer-specific metrics documentation by @behroozazarkhalili in https://github.com/huggingface/trl/pull/4269 * Remove using_llama_models.md: outdated Llama2-specific documentation by @behroozazarkhalili in https://github.com/huggingface/trl/pull/4268 * Remove how_to_train.md: outdated training FAQ by @behroozazarkhalili in https://github.com/huggingface/trl/pull/4267 * Add Qwen3-VL notebooks (SFT, GRPO) by @sergiopaniego in https://github.com/huggingface/trl/pull/4275 * Remove obsolete research_projects directory by @behroozazarkhalili in https://github.com/huggingface/trl/pull/4243 * Add Efficient Online Training with GRPO and vLLM in TRL to community tutorials by @sergiopaniego in https://github.com/huggingface/trl/pull/4219 * Add trainers taxonomy to docs by @sergiopaniego in https://github.com/huggingface/trl/pull/4195 * Updated vLLM integration guide by @sergiopaniego in https://github.com/huggingface/trl/pull/4162 * [DOCS] Lora without regret by @burtenshaw in https://github.com/huggingface/trl/pull/4181 * Add docstring for OnlineTrainerState by @albertvillanova in https://github.com/huggingface/trl/pull/4166 * ⚖️ Align SFT and DPO for model creation and deprecate `DPOConfig.padding_value` in favour or `pad_token_id` by @qgallouedec in https://github.com/huggingface/trl/pull/4006 * 🏞️ Context Parallelism benchmark guide by @sergiopaniego in https://github.com/huggingface/trl/pull/4075 * ▶️ Add video to community tutorials by @qgallouedec in https://github.com/huggingface/trl/pull/4090 * Reviewed HF jobs updated docs by @sergiopaniego in https://github.com/huggingface/trl/pull/4088 ## Deprecations * Deprecate `BestOfNSampler` by @qgallouedec in https://github.com/huggingface/trl/pull/4291 * Raise deprecation warning for Python 3.9 by @albertvillanova in https://github.com/huggingface/trl/pull/4226 * Deprecate unused dataset_formatting module by @behroozazarkhalili in https://github.com/huggingface/trl/pull/4242 * Warnings pointing to RFC by @qgallouedec in https://github.com/huggingface/trl/pull/4224 * 🅰️ Remove apex by @qgallouedec in https://github.com/huggingface/trl/pull/4139 * 🗑️ Remove deprecated `AlignPropTrainer`, `DDPOTrainer` and `IterativeSFTTrainer` by @qgallouedec in https://github.com/huggingface/trl/pull/4068 ## Experimental * 🧪 Add `trl.experimental` Submodule by @August-murr in https://github.com/huggingface/trl/pull/4073 * [GRPO]: Sample from a Replay Buffer To Substitute Groups with 0 std. by @pramodith in https://github.com/huggingface/trl/pull/4060 * 🪙 [Experimental] Support GSPO-token by @hjh0119 in https://github.com/huggingface/trl/pull/3820 * 🌪️ [GFPO]: implement GFPO in GRPOTrainer by @Peter-Chou in https://github.com/huggingface/trl/pull/3989 * 🌾 [Experimental] BEMA for ref model by @qgallouedec in https://github.com/huggingface/trl/pull/3898 ## What's Changed * ⬆️ Bump dev version by @qgallouedec in https://github.com/huggingface/trl/pull/4054 * Remove redundant 'None' from docstrings by @albertvillanova in https://github.com/huggingface/trl/pull/4058 * Hotfix: Add ParallelismConfig fallback for transformers with old accelerate by @albertvillanova in https://github.com/huggingface/trl/pull/4063 * Fix CI failure in slow GRPO test due to missing pillow dependency by @albertvillanova in https://github.com/huggingface/trl/pull/4064 * 💡 Fix type hint to `make_parser` function in multiple scripts by @qgallouedec in https://github.com/huggingface/trl/pull/4050 * Improve docstring of AlignPropTrainer by @albertvillanova in https://github.com/huggingface/trl/pull/4059 * ♨️ [GRPO] Fix potential hang in `get_high_entropy_mask` by @akakakakakaa in https://github.com/huggingface/trl/pull/4041 * Set Ruff src for first-party imports by @albertvillanova in https://github.com/huggingface/trl/pull/4074 * 🧪 Add `trl.experimental` Submodule by @August-murr in https://github.com/huggingface/trl/pull/4073 * 🌾 [Experimental] BEMA for ref model by @qgallouedec in https://github.com/huggingface/trl/pull/3898 * ✂️ [GRPO VLM] Update split sizes to generalize by @zucchini-nlp in https://github.com/huggingface/trl/pull/4032 * 🛠️ Fix CI by @qgallouedec in https://github.com/huggingface/trl/pull/4076 * 🐳 Docker update + Simplify Jobs doc by @qgallouedec in https://github.com/huggingface/trl/pull/3931 * Aux loss is already included in the loss returned by Transformers by @pramodith in https://github.com/huggingface/trl/pull/4078 * Reviewed HF jobs updated docs by @sergiopaniego in https://github.com/huggingface/trl/pull/4088 * 🗑️ Remove deprecated `AlignPropTrainer`, `DDPOTrainer` and `IterativeSFTTrainer` by @qgallouedec in https://github.com/huggingface/trl/pull/4068 * ▶️ Add video to community tutorials by @qgallouedec in https://github.com/huggingface/trl/pull/4090 * Align slow tests with regular tests by @albertvillanova in https://github.com/huggingface/trl/pull/4085 * Add support for testing experimental features by @albertvillanova in https://github.com/huggingface/trl/pull/4082 * Community Tutorials design adaptation for videos by @sergiopaniego in https://github.com/huggingface/trl/pull/4095 * 🏞️ Context Parallelism benchmark guide by @sergiopaniego in https://github.com/huggingface/trl/pull/4075 * ⌨️ Pin num2words by @lewtun in https://github.com/huggingface/trl/pull/4094 * Add deprecation warnings to docstrings by @albertvillanova in https://github.com/huggingface/trl/pull/4083 * 📜 Convert `set` to `list` of tags by @qgallouedec in https://github.com/huggingface/trl/pull/4092 * 🧶 feat: Add WeaveCallback for W&B Weave integration by @parambharat in https://github.com/huggingface/trl/pull/4089 * ⚖️ Align SFT and DPO for model creation and deprecate `DPOConfig.padding_value` in favour or `pad_token_id` by @qgallouedec in https://github.com/huggingface/trl/pull/4006 * 🌪️ [GFPO]: implement GFPO in GRPOTrainer by @Peter-Chou in https://github.com/huggingface/trl/pull/3989 * ℹ️ feat: Add NPU and XPU support for activation offloading by @zilongzheng in https://github.com/huggingface/trl/pull/4056 * ℹ️ Enable XPU for vLLM client by @jiqing-feng in https://github.com/huggingface/trl/pull/4031 * Fix get_peft_model() so that prepare_model_for_kbit_training does not reapply to an instance of PeftModel, thus freezing all the layers by @Hoesu in https://github.com/huggingface/trl/pull/4081 * [GKD] Fix `batchmean` reduce op in GKDTrainer's loss by @cmpatino in https://github.com/huggingface/trl/pull/4105 * 👁️ Add VLM support to RLOO trainer by @behroozazarkhalili in https://github.com/huggingface/trl/pull/4067 * Some nits GRPO and RLOO trainer docs by @sergiopaniego in https://github.com/huggingface/trl/pull/4108 * Fix typos by @cyyever in https://github.com/huggingface/trl/pull/4106 * Fix typos by @qgallouedec in https://github.com/huggingface/trl/pull/4109 * Fix VLM configs in generate_tiny_models by @albertvillanova in https://github.com/huggingface/trl/pull/4101 * docs: correct option name to enable vllm sleep mode by @muupan in https://github.com/huggingface/trl/pull/4102 * CI hotfix: xfail test_training_with_transformers_paged for transformers<4.57.0 by @albertvillanova in https://github.com/huggingface/trl/pull/4120 * Fix code style with make precommit by @albertvillanova in https://github.com/huggingface/trl/pull/4119 * 🟩 Drop `image_split_sizes` in favour of `image_grid_thw` by @qgallouedec in https://github.com/huggingface/trl/pull/4111 * 🔭 Align param passing to VLM configs in generate_tiny_models by @albertvillanova in https://github.com/huggingface/trl/pull/4118 * 📽 Multi image support for GRPO/RLOO by @qgallouedec in https://github.com/huggingface/trl/pull/4113 * 😴 Add `vllm_enable_sleep_mode` to RLOO Trainer by @sergiopaniego in https://github.com/huggingface/trl/pull/4107 * 🐯 fix: use_liger_kernel with IterableDataset by @jue-jue-zi in https://github.com/huggingface/trl/pull/4087 * 📤 Fix a dataset loading bug in scripts by @singing-cat in https://github.com/huggingface/trl/pull/4124 * ⚓ [vllm] ensure MASTER_ADDR/MASTER_PORT are set safely by @kashif in https://github.com/huggingface/trl/pull/4057 * 📌 Pin vLLM version by @qgallouedec in https://github.com/huggingface/trl/pull/4122 * 👋 Remove `backend` parameter from `GuidedDecodingParams` by @qgallouedec in https://github.com/huggingface/trl/pull/4123 * 🧹 Remove `max_batch_tokens`, `num_blocks` and `block_size` from generation kwargs by @qgallouedec in https://github.com/huggingface/trl/pull/4065 * Remove Python version < 3.13 constraint from vllm extra dependencies by @albertvillanova in https://github.com/huggingface/trl/pull/4125 * 👩‍🦯 Fix usage of VLM using text only by @SamuelBarryCS in https://github.com/huggingface/trl/pull/4080 * [SFTrainer]: Fix DFT Loss by @pramodith in https://github.com/huggingface/trl/pull/4112 * Improve typing of SFT trainer by @cyyever in https://github.com/huggingface/trl/pull/4007 * 🌺 Fix GPT-OSS test by @qgallouedec in https://github.com/huggingface/trl/pull/4134 * 🪙 [Experimental] Support GSPO-token by @hjh0119 in https://github.com/huggingface/trl/pull/3820 * Fix CI: torch.AcceleratorError: CUDA error: device-side assert triggered by @albertvillanova in https://github.com/huggingface/trl/pull/4138 * 🤸‍♀️ Fix DFT test by @qgallouedec in https://github.com/huggingface/trl/pull/4135 * 🌵 Mark GKD trainer test as expected failure due to OOM issue by @qgallouedec in https://github.com/huggingface/trl/pull/4126 * [GRPO]: Sample from a Replay Buffer To Substitute Groups with 0 std. by @pramodith in https://github.com/huggingface/trl/pull/4060 * Fix import statement and GRPO test case by @qgallouedec in https://github.com/huggingface/trl/pull/4141 * Refactor trainers classes to use BaseTrainer with shared functionality by @albertvillanova in https://github.com/huggingface/trl/pull/4128 * Fixed some <Tip> rendering issues by @sergiopaniego in https://github.com/huggingface/trl/pull/4143 * 😷 Refactor GRPO/RLOO to isolate `_generate` by @qgallouedec in https://github.com/huggingface/trl/pull/4114 * 🟩 Drop `image_split_sizes` in favour of `image_grid_thw` by @qgallouedec in https://github.com/huggingface/trl/pull/4156 * 📽 Multi image support for GRPO replay buffer by @qgallouedec in https://github.com/huggingface/trl/pull/4157 * 😷 Refactor GRPO/RLOO to isolate `_generate` for GRPO with replay buffer by @qgallouedec in https://github.com/huggingface/trl/pull/4158 * Add docstring for OnlineTrainerState by @albertvillanova in https://github.com/huggingface/trl/pull/4166 * Pass required token_type_ids by @albertvillanova in https://github.com/huggingface/trl/pull/4148 * 💡 Replace `<Tip>` with new markdown syntax by @qgallouedec in https://github.com/huggingface/trl/pull/4161 * Remove unnecessary list comprehensions by @albertvillanova in https://github.com/huggingface/trl/pull/4164 * Add missing FDivergenceType docstring by @albertvillanova in https://github.com/huggingface/trl/pull/4165 * Fix docstrings with 'deprecated' Sphinx directive by @albertvillanova in https://github.com/huggingface/trl/pull/4174 * Fix docstring interlink to parent class for NashMDTrainer and XPOTrainer by @albertvillanova in https://github.com/huggingface/trl/pull/4179 * Fix link in docstring of RLOOTrainer by @albertvillanova in https://github.com/huggingface/trl/pull/4180 * 🖨️ Print rich table for messages by @qgallouedec in https://github.com/huggingface/trl/pull/4160 * 🅰️ Remove apex by @qgallouedec in https://github.com/huggingface/trl/pull/4139 * Fix CI ValueError: Unknown loss type: dapo by @albertvillanova in https://github.com/huggingface/trl/pull/4173 * Fix PEFT interlinks in docstrings by @albertvillanova in https://github.com/huggingface/trl/pull/4178 * ✨ Add logging for training completion and model saving in training scripts by @qgallouedec in https://github.com/huggingface/trl/pull/4048 * 👾 Use our own `require_bitsandbytes` by @qgallouedec in https://github.com/huggingface/trl/pull/4137 * 🎞️ Support sequence classification models in `clone_chat_template` by @qgallouedec in https://github.com/huggingface/trl/pull/4097 * ⚡ Fix Flash Attention x Padding-Free loss by @qgallouedec in https://github.com/huggingface/trl/pull/4170 * 🎁 `RewardTrainer` refactor by @qgallouedec in https://github.com/huggingface/trl/pull/4093 * 🧺 [1/N] Refactor `_generate` in GRPO/RLOO: list of ints instead of tensors by @qgallouedec in https://github.com/huggingface/trl/pull/4146 * Fix handling of f_divergence_type in DPO by @albertvillanova in https://github.com/huggingface/trl/pull/4171 * 🔣 Fix test: replace `trainer.tokenizer` by `trainer.processing_class` by @qgallouedec in https://github.com/huggingface/trl/pull/4185 * Fix CI ImportError: FlashAttention2 and decorator order for all parameterized tests by @albertvillanova in https://github.com/huggingface/trl/pull/4176 * Hotfix wrong formatting of docstrings with blockquote tips by @albertvillanova in https://github.com/huggingface/trl/pull/4187 * 🌡️ Have vLLM return processed (temperature scaled) log probs by @YonatanGideoni in https://github.com/huggingface/trl/pull/4163 * Replace remaining trainer.tokenizer with trainer.processing_class in GRPO test by @albertvillanova in https://github.com/huggingface/trl/pull/4192 * [DOCS] Lora without regret by @burtenshaw in https://github.com/huggingface/trl/pull/4181 * [DOCS/FIX] lora without regrets - fix lr by @burtenshaw in https://github.com/huggingface/trl/pull/4207 * Remove custome_container for building the docs by @albertvillanova in https://github.com/huggingface/trl/pull/4198 * Remove tokenizer creation from `sft` example script by @sergiopaniego in https://github.com/huggingface/trl/pull/4197 * Hotfix: Exclude transformers 4.57.0 for Python 3.9 by @albertvillanova in https://github.com/huggingface/trl/pull/4209 * Replace unittest with pytest by @albertvillanova in https://github.com/huggingface/trl/pull/4188 * Updated vLLM integration guide by @sergiopaniego in https://github.com/huggingface/trl/pull/4162 * Remove `Optional` from `processing_class` in `PPOTrainer` by @sergiopaniego in https://github.com/huggingface/trl/pull/4212 * Replace setup with pyproject and fix packaging unintended modules by @albertvillanova in https://github.com/huggingface/trl/pull/4194 * Removed tokenizer/processor creation from example scripts by @sergiopaniego in https://github.com/huggingface/trl/pull/4211 * Apply style and revert change in `sft_video_llm` example by @qgallouedec in https://github.com/huggingface/trl/pull/4214 * Fix `trl-internal-testing/tiny-DbrxForCausalLM` by @qgallouedec in https://github.com/huggingface/trl/pull/4213 * Fix prompt-completion labeling with add_generation_prompt and warning by @behroozazarkhalili in https://github.com/huggingface/trl/pull/4201 * Fix LoRA params in Python in LoRA without regret by @sergiopaniego in https://github.com/huggingface/trl/pull/4215 * [DOCS] fix prose in lora guide by @burtenshaw in https://github.com/huggingface/trl/pull/4217 * Add trainers taxonomy to docs by @sergiopaniego in https://github.com/huggingface/trl/pull/4195 * 🎨 Support mixing image+text and text-only examples by @qgallouedec in https://github.com/huggingface/trl/pull/4203 * 🧺 [2/N] Refactor `_generate` in GRPO/RLOO: Use `prompt_ids` from generation by @qgallouedec in https://github.com/huggingface/trl/pull/4152 * Fix entropy and accuracy calculation for prompt_tuning techniques. by @pramodith in https://github.com/huggingface/trl/pull/4196 * Add Efficient Online Training with GRPO and vLLM in TRL to community tutorials by @sergiopaniego in https://github.com/huggingface/trl/pull/4219 * 🏷️ Account for `token_type_ids` in `DataCollatorForVisionLanguageModeling` by @qgallouedec in https://github.com/huggingface/trl/pull/4190 * Exclude vllm dependencies from dev extra by @albertvillanova in https://github.com/huggingface/trl/pull/4229 * Fix CI unittest asserts by @albertvillanova in https://github.com/huggingface/trl/pull/4234 * Fix callable annotations by @albertvillanova in https://github.com/huggingface/trl/pull/4216 * Remove unused Path import in __init__.py by @albertvillanova in https://github.com/huggingface/trl/pull/4227 * Update CI Docker image to pytorch/pytorch:2.8.0 by @albertvillanova in https://github.com/huggingface/trl/pull/4232 * Replace setup with pyproject in CI tests paths by @albertvillanova in https://github.com/huggingface/trl/pull/4230 * Fix CI IndentationError for Python 3.13.8 by @albertvillanova in https://github.com/huggingface/trl/pull/4240 * Remove unused log_example_reports.py script by @behroozazarkhalili in https://github.com/huggingface/trl/pull/4241 * 🧘 Enhance markdown style by @qgallouedec in https://github.com/huggingface/trl/pull/4235 * Warnings pointing to RFC by @qgallouedec in https://github.com/huggingface/trl/pull/4224 * Fix CI slow test ValueError: Backward pass should have cleared tracker of all tensors by @sywangyi in https://github.com/huggingface/trl/pull/4236 * Fix CI CUDA out of memory errors by improving GPU memory management by @albertvillanova in https://github.com/huggingface/trl/pull/4238 * Install peft from main for CI tests with dev dependencies by @albertvillanova in https://github.com/huggingface/trl/pull/4250 * Fix CI ImportError for 'require_torch_gpu_if_bnb_not_multi_backend_enabled' by @albertvillanova in https://github.com/huggingface/trl/pull/4253 * Fix CI slow test ValueError: Unknown loss type: dapo by @albertvillanova in https://github.com/huggingface/trl/pull/4254 * 🧺 [3/N] Refactor `_generate` in GRPO/RLOO: Rely on generator for prompt truncation by @qgallouedec in https://github.com/huggingface/trl/pull/4153 * Remove obsolete research_projects directory by @behroozazarkhalili in https://github.com/huggingface/trl/pull/4243 * Deprecate unused dataset_formatting module by @behroozazarkhalili in https://github.com/huggingface/trl/pull/4242 * Fix CI slow test AttributeError: 'TestSFTTrainerSlow' object has no attribute 'addCleanup' by @albertvillanova in https://github.com/huggingface/trl/pull/4255 * [Online-DPO] fix the completion_len == max_new_tokens crash by @kashif in https://github.com/huggingface/trl/pull/4193 * Include `chat_template_kwargs` in `apply_chat_template` by @cmpatino in https://github.com/huggingface/trl/pull/4233 * Fix Python version check for skipping tests on Python 3.13.8 by @albertvillanova in https://github.com/huggingface/trl/pull/4246 * Raise deprecation warning for Python 3.9 by @albertvillanova in https://github.com/huggingface/trl/pull/4226 * Fix docstring interlinks by @albertvillanova in https://github.com/huggingface/trl/pull/4221 * Use FutureWarning instead of DeprecationWarning by @albertvillanova in https://github.com/huggingface/trl/pull/4266 * Fix style with make precommit by @albertvillanova in https://github.com/huggingface/trl/pull/4265 * Add Qwen3-VL notebooks (SFT, GRPO) by @sergiopaniego in https://github.com/huggingface/trl/pull/4275 * Fix typo in Colab link by @sergiopaniego in https://github.com/huggingface/trl/pull/4276 * Fix docstrings with Sphinx 'deprecated' directive by @albertvillanova in https://github.com/huggingface/trl/pull/4279 * Fix CI slow test OSError: You are trying to access a gated repo by @albertvillanova in https://github.com/huggingface/trl/pull/4283 * 💰 `RichProgressCallback` enhancement by @qgallouedec in https://github.com/huggingface/trl/pull/4245 * Fix CI dev test TypeError: unexpected keyword argument 'load_in_4bit' by @albertvillanova in https://github.com/huggingface/trl/pull/4262 * Replace unittest skipTest with pytest.skip by @albertvillanova in https://github.com/huggingface/trl/pull/4263 * Fix CI slow tests: ImportError: vLLM is not installed by @albertvillanova in https://github.com/huggingface/trl/pull/4287 * Remove logging.md: trainer-specific metrics documentation by @behroozazarkhalili in https://github.com/huggingface/trl/pull/4269 * Remove using_llama_models.md: outdated Llama2-specific documentation by @behroozazarkhalili in https://github.com/huggingface/trl/pull/4268 * Add support for `token_type_ids` in `DPOTrainer` by @aweers in https://github.com/huggingface/trl/pull/4285 * Remove how_to_train.md: outdated training FAQ by @behroozazarkhalili in https://github.com/huggingface/trl/pull/4267 * Add accuracy reward by @pramodith in https://github.com/huggingface/trl/pull/4270 * Remove unused commands directory by @behroozazarkhalili in https://github.com/huggingface/trl/pull/4258 * Deprecate `BestOfNSampler` by @qgallouedec in https://github.com/huggingface/trl/pull/4291 * Release: v0.24 by @qgallouedec in https://github.com/huggingface/trl/pull/4292 ## New Contributors * @zucchini-nlp made their first contribution in https://github.com/huggingface/trl/pull/4032 * @parambharat made their first contribution in https://github.com/huggingface/trl/pull/4089 * @zilongzheng made their first contribution in https://github.com/huggingface/trl/pull/4056 * @jiqing-feng made their first contribution in https://github.com/huggingface/trl/pull/4031 * @Hoesu made their first contribution in https://github.com/huggingface/trl/pull/4081 * @cmpatino made their first contribution in https://github.com/huggingface/trl/pull/4105 * @singing-cat made their first contribution in https://github.com/huggingface/trl/pull/4124 * @SamuelBarryCS made their first contribution in https://github.com/huggingface/trl/pull/4080 * @YonatanGideoni made their first contribution in https://github.com/huggingface/trl/pull/4163 * @aweers made their first contribution in https://github.com/huggingface/trl/pull/4285 **Full Changelog**: https://github.com/huggingface/trl/compare/v0.23.0...v0.24.0

## Major ### 🥓 Context Parallelism SFT now supports Context Parallelism (CP) for training large language models on very large sequences. You can now train with an arbitrarily long sequence length. <img width="844" height="336" alt="Screenshot 2025-09-09 at 10 39 30 PM" src="https://github.com/user-attachments/assets/f1dfc349-440a-4e05-aac9-439a3c286f08" /> by @kashif in https://github.com/huggingface/trl/pull/3994 ### 🧨 Dynamic Fine-Tuning Dynamic Fine-Tuning (DFT) is a nnow supported in TRL. ```python from trl import SFTConfig training_args = SFTConfig( loss_type="dft", ... ) ``` <img width="692" height="472" alt="Screenshot 2025-09-09 at 10 37 36 PM" src="https://github.com/user-attachments/assets/4ee2b4ab-7cc6-4578-bfac-c38124891510" /> by @qgallouedec in https://github.com/huggingface/trl/pull/4042 ### 🪵 Truncated Importance Sampling (TIS) to address rollout-training mismatch Different implementations are used for rollout generation (vLLM) and model training. The implementation gap implicitly turns the on-policy RL to be off-policy. Truncated Importance Sampling (TIS) a simple yet effective importance sampling technique for handling such discrepancy. This is now implemented in GRPO. ```python from trl import GRPOConfig training_args = GRPOConfig( ... use_vllm=True, vllm_importance_sampling_correction=True, # default True vllm_importance_sampling_cap=2.0, # hyper-parameter C ) ``` by @LeonEricsson in https://github.com/huggingface/trl/pull/3867 ### 🥣 [SFTTrainer]: Add Aux Loss for MoE models Mixture of Experts (MoE) models require an auxiliary loss to ensure that the different experts are used evenly. This auxiliary loss is now supported in SFTTrainer. ```python training_args = SFTConfig( model_init_kwargs={"output_router_logits": True}, ... ) ``` by @pramodith in https://github.com/huggingface/trl/pull/4012 ### 💤 [GRPO/RLOO] Adds an option to sleep vllm when running in colocated mode When running GRPO (or RLOO) with vLLM in colocated mode, the vLLM server consume VRAM during optimization while not being used. We now have an option to put the vLLM server to sleep during optimization to free up VRAM. ```python from trl import GRPOConfig training_args = GRPOConfig(..., vllm_sleep_enabled=True) ``` by @edbeeching in https://github.com/huggingface/trl/pull/3968 ### ⚖️ Add vLLM server mode and VLM support to OnlineDPOTrainer You can now use vLLM server mode with OnlineDPOTrainer. Additionally, VLM models are now supported. by @vaelev in https://github.com/huggingface/trl/pull/3783 ### Comprehensive Paper Index Enhancement with 9 New Algorithm Implementations The paper index has been significantly enhanced with the addition of 9+ new algorithm implementations, providing a more comprehensive resource for users. by @behroozazarkhalili in https://github.com/huggingface/trl/pull/3990 ### Other Notable Changes * 👷 Added Kernels on the Hub x TRL guide by @sergiopaniego in https://github.com/huggingface/trl/pull/3969 * 🌵 Refactor entropy_from_logits for memory efficiency by @qgallouedec in https://github.com/huggingface/trl/pull/4013 ## What's Changed * ⬆️ Bump dev version by @qgallouedec in https://github.com/huggingface/trl/pull/3978 * 👮 Fix GRPO CLI by setting parameters for `get_soft_overlong_punishment` by @qgallouedec in https://github.com/huggingface/trl/pull/3972 * 🪃 `args.gradient_checkpointing = False` instead of `args = dataclasses.replace(args, gradient_checkpointing=False)` by @qgallouedec in https://github.com/huggingface/trl/pull/3981 * [GRPO] Adds an option to sleep vllm when running in colocated mode by @edbeeching in https://github.com/huggingface/trl/pull/3968 * 🎯 Add Trackio integration documentation and update TOC by @qgallouedec in https://github.com/huggingface/trl/pull/3971 * ⚖️ Fix scale_rewards issue in GRPO by @Peter-Chou in https://github.com/huggingface/trl/pull/3992 * ⏰ fix: add return to shift_tokens_right by @ginkyenglee in https://github.com/huggingface/trl/pull/3987 * Add pre-commit and hf-doc-builder as dev dependencies by @albertvillanova in https://github.com/huggingface/trl/pull/3993 * [GRPO] Truncated Importance Sampling to address rollout-training mismatch by @LeonEricsson in https://github.com/huggingface/trl/pull/3867 * Fixed tags shown problem in memory usage docs by @sergiopaniego in https://github.com/huggingface/trl/pull/3999 * ✖️ Support pad-to-multiple-of and padding-free by @qgallouedec in https://github.com/huggingface/trl/pull/3996 * 💾 [bugfix] fix PPO save_checkpoint by @hjh0119 in https://github.com/huggingface/trl/pull/3998 * [GRPO]: Fix Multi-GPU training for Entropy based masking of tokens. by @pramodith in https://github.com/huggingface/trl/pull/3964 * 📏 `torch_dype` to `dtype` everywhere by @sergiopaniego in https://github.com/huggingface/trl/pull/4000 * Comprehensive Paper Index Enhancement with 9 New Algorithm Implementations by @behroozazarkhalili in https://github.com/huggingface/trl/pull/3990 * [SFT] fix: collator docstring by @LeonEricsson in https://github.com/huggingface/trl/pull/4011 * 👷 Added Kernels on the Hub x TRL guide by @sergiopaniego in https://github.com/huggingface/trl/pull/3969 * 🌵 Refactor entropy_from_logits for memory efficiency by @qgallouedec in https://github.com/huggingface/trl/pull/4013 * [SFTTrainer]: Add Aux Loss for MoE models. by @pramodith in https://github.com/huggingface/trl/pull/4012 * Add missing doc strings in SFTrainer by @pramodith in https://github.com/huggingface/trl/pull/4003 * ⚖️ Add vLLM server mode and VLM support to OnlineDPOTrainer by @vaelev in https://github.com/huggingface/trl/pull/3783 * Fix typo in GRPO quickstart by @dwisdom0 in https://github.com/huggingface/trl/pull/4020 * Align docstring parameters with function definitions by @albertvillanova in https://github.com/huggingface/trl/pull/4017 * Fix formatting errors in docstrings by @albertvillanova in https://github.com/huggingface/trl/pull/4025 * [doc] Paper index for Truncated Importance Sampling by @LeonEricsson in https://github.com/huggingface/trl/pull/4026 * [doc] Group paper index by trainer by @LeonEricsson in https://github.com/huggingface/trl/pull/4027 * Add missing trainer docstrings by @albertvillanova in https://github.com/huggingface/trl/pull/4030 * Add autodoc for AlignPropTrainer and AlignPropConfig by @albertvillanova in https://github.com/huggingface/trl/pull/4033 * 🥓 [docs] add CP docs by @kashif in https://github.com/huggingface/trl/pull/3994 * ⚖️ Remove `average_tokens_across_devices` default replacement by @qgallouedec in https://github.com/huggingface/trl/pull/4039 * CI hotfix: xfail test_training_with_transformers_paged by @albertvillanova in https://github.com/huggingface/trl/pull/4046 * Update transformers minimum version to 4.56.1 by @albertvillanova in https://github.com/huggingface/trl/pull/4047 * 🧨 DFT by @qgallouedec in https://github.com/huggingface/trl/pull/4042 * Update VLM arch check to `AutoModelForImageTextToText` for DPO and Online DPO by @sergiopaniego in https://github.com/huggingface/trl/pull/4049 * 🏂 Fix label shifting logic in `SFTTrainer` for compatibility with CP by @qgallouedec in https://github.com/huggingface/trl/pull/4038 * Add autodoc for BestOfNSampler and improve docstrings by @albertvillanova in https://github.com/huggingface/trl/pull/4034 * ✨ Improve SFT doc by @qgallouedec in https://github.com/huggingface/trl/pull/4005 * 💬 Remove setting chat template in sft script by @qgallouedec in https://github.com/huggingface/trl/pull/4037 * 🪪 Update SFTTrainer to handle labels correctly and add configuration example in paper index by @qgallouedec in https://github.com/huggingface/trl/pull/4051 * 🗜 Hotfix: avoid passing `quantization_config=None` by @qgallouedec in https://github.com/huggingface/trl/pull/4019 * Release: 0.23 by @qgallouedec in https://github.com/huggingface/trl/pull/4053 ## New Contributors * @Peter-Chou made their first contribution in https://github.com/huggingface/trl/pull/3992 * @ginkyenglee made their first contribution in https://github.com/huggingface/trl/pull/3987 * @albertvillanova made their first contribution in https://github.com/huggingface/trl/pull/3993 * @hjh0119 made their first contribution in https://github.com/huggingface/trl/pull/3998 * @vaelev made their first contribution in https://github.com/huggingface/trl/pull/3783 * @dwisdom0 made their first contribution in https://github.com/huggingface/trl/pull/4020 **Full Changelog**: https://github.com/huggingface/trl/compare/v0.22.0...v0.23.0

## What changed - Refactor version retrieval to use `importlib.metadata` by @qgallouedec - Release: 0.22.1 by @qgallouedec **Full Changelog**: https://github.com/huggingface/trl/compare/v0.22.0...v0.22.1

## Breaking and major changes ### 🎞️ GSPO GSPO is a GRPO variant that computes importance sampling weights at the sequence level instead of per-token. <img width="930" height="538" alt="Screenshot 2025-07-28 at 10 54 15 PM" src="https://github.com/user-attachments/assets/923835af-dc61-4fd4-8a99-44242d02bb7b" /> 📜 Paper: https://huggingface.co/papers/2507.18071 To reproduce the paper's setting, use this configuration: ```python from trl import GRPOConfig training_args = GRPOConfig( importance_sampling_level="sequence", loss_type="grpo", steps_per_generation=..., beta=0.04, # not explicitly specified in the paper, but they likely used the same value as in the GRPO paper epsilon=3e-4, # https://x.com/ChujieZheng/status/1948933507696525392 ) ``` by @qgallouedec in https://github.com/huggingface/trl/pull/3775 ### 👁️ [GRPO] Add VLM training capabilities to the GRPO trainer <img width="1136" height="594" alt="Group 291-4" src="https://github.com/user-attachments/assets/04850e80-9689-472d-acd7-fda331e66dc3" /> The GRPOTrainer can now be used for VLM training. Give a try with this dummy example: ```python from trl import GRPOTrainer from datasets import load_dataset # Dummy vision-language dataset dataset = load_dataset("trl-internal-testing/zen-image", "conversational_prompt_only", split="train") # Dummy reward function: count the number of unique characters in the completions def reward_num_unique_chars(completions, **kwargs): return [len(set(c[0]["content"])) for c in completions] trainer = GRPOTrainer( model="Qwen/Qwen2.5-VL-3B-Instruct", reward_funcs=[reward_num_unique_chars], train_dataset=dataset, ) trainer.train() ``` by @CompN3rd and @kashif in https://github.com/huggingface/trl/pull/3072 in https://github.com/huggingface/trl/pull/3760 ### 🐙 MPO <img width="440" height="438" alt="Screenshot 2025-07-28 at 10 52 15 PM" src="https://github.com/user-attachments/assets/e07a7936-c4c5-480d-9ffd-db5b77a5445e" /> The DPO trainer supports combining multiple loss functions with different weights, enabling more sophisticated optimization strategies. This is particularly useful for implementing algorithms like MPO (Mixed Preference Optimization). MPO is a training approach that combines multiple optimization objectives, as described in the paper [Enhancing the Reasoning Ability of Multimodal Large Language Models via Mixed Preference Optimization](https://huggingface.co/papers/2411.10442). To combine multiple losses, specify the loss types and corresponding weights as lists: ```python from trl import DPOConfig # MPO: Combines DPO (sigmoid) for preference and BCO (bco_pair) for quality training_args = DPOConfig( loss_type=["sigmoid", "bco_pair", "sft"], # Loss types to combine loss_weights=[0.8, 0.2, 1.0] # Corresponding weights, as used in the MPO paper ) ``` by @qgallouedec in https://github.com/huggingface/trl/pull/2544 ### Add support for CB with native transformers Continuous Batching allows for faster generation using the `transformers` backend. You can now use it with the `GRPOTrainer` by setting `use_transformers_paged=True` in the config. ```python use_transformers_paged = True from trl import GRPOConfig training_args = GRPOConfig( # ... other args use_transformers_paged=Ture, ) ``` by @ArthurZucker in https://github.com/huggingface/trl/pull/3471 ### Add entropy based filtering inside the GRPOTrainer <img width="788" height="438" alt="Screenshot 2025-07-28 at 10 27 20 PM" src="https://github.com/user-attachments/assets/8073a5db-a98e-4534-aea9-d3dbd2e75f4a" /> In [Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoning](https://huggingface.co/papers/2506.01939), it is shown that utilizing only 20% of the highest entropy tokens leads to similar performance as using all tokens. You can now enable this feature in the `GRPOTrainer` by setting `entropy_filtering=True` in the config. ```python from trl import GRPOConfig training_args = GRPOConfig( # ... other args top_entropy_quantile=0.2, # Use only the top 20% of tokens based on entropy ) ``` by @pramodith in https://github.com/huggingface/trl/pull/3563 ### 👐 FSDP2+GRPO GRPO now supports FSDP2 training. Just run your script with an FSDP2 config: ```bash accelerate launch --config_file examples/accelerate_configs/fsdp2.yaml run_grpo.py ``` by @SalmanMohammadi in https://github.com/huggingface/trl/pull/3687 ## What's Changed * ⬆️ Bump dev version by @qgallouedec in https://github.com/huggingface/trl/pull/3626 * fix grpo generation_kwargs by @ahatamiz in https://github.com/huggingface/trl/pull/3634 * fixing num_processes by @shirinyamani in https://github.com/huggingface/trl/pull/3637 * env var for vllm colocate exp added by @shirinyamani in https://github.com/huggingface/trl/pull/3638 * Update dpo_vlm.py by @Clement25 in https://github.com/huggingface/trl/pull/3629 * ☕️ GRPO script reward_funcs error by @tcapelle in https://github.com/huggingface/trl/pull/3639 * 🤝 validate gradient_accumulation_steps vs steps_per_generation for on-policy GRPO by @HarryHsing in https://github.com/huggingface/trl/pull/3493 * Add entropy based filtering inside the GRPOTrainer. by @pramodith in https://github.com/huggingface/trl/pull/3563 * Make sure chat template isn't lost when truncating prompt. by @pramodith in https://github.com/huggingface/trl/pull/3651 * Add paranthesis to correct the check. by @pramodith in https://github.com/huggingface/trl/pull/3658 * Add support for CB with native transformers by @ArthurZucker in https://github.com/huggingface/trl/pull/3471 * feat: Pass trainer state to reward functions by @seungduk-yanolja in https://github.com/huggingface/trl/pull/3669 * Enable completion-only loss in SFTTrainer when using Liger Kernel by @kswhitecross in https://github.com/huggingface/trl/pull/3674 * Add mlflow support for generate_during_eval DPOTrainer by @dhruvmullick in https://github.com/huggingface/trl/pull/3660 * [SFT] drop attention_mask if we have position ids for fa2 by @kashif in https://github.com/huggingface/trl/pull/3673 * Faster `position_ids` computation for FFD packing by @mariosasko in https://github.com/huggingface/trl/pull/3649 * Support datasets 4 by @lhoestq in https://github.com/huggingface/trl/pull/3688 * Update steps_per_generation default description grpo_config.py by @wa008 in https://github.com/huggingface/trl/pull/3685 * Fix non-serializable torch.dtype bug in VLLM weight sync by @CarlosArguilar in https://github.com/huggingface/trl/pull/3690 * fix: support dict access in SFT Trainer by @jannisborn in https://github.com/huggingface/trl/pull/3677 * [fix] type error of quantile by @gitabtion in https://github.com/huggingface/trl/pull/3667 * [CI] Fix slow grpo CI by @kashif in https://github.com/huggingface/trl/pull/3693 * Restore the effect of liger_kernel's monkey_patch on global modules in UT. by @YangKai0616 in https://github.com/huggingface/trl/pull/3680 * Add type hints to `dpo_trainer.py` by @bvantuan in https://github.com/huggingface/trl/pull/3631 * Fix mislabeling: "First-fit decreasing" is actually "Best-fit-decreasing" by @LeonEricsson in https://github.com/huggingface/trl/pull/3696 * ✂️ [BUG when vllm and prompt_truncation are used]: Strip out pad tokens in truncated prompt text by @pramodith in https://github.com/huggingface/trl/pull/3698 * 📣 Use explicit version for checking datasets version by @qgallouedec in https://github.com/huggingface/trl/pull/3702 * 🔭 Fix package discovery configuration in setup.cfg by @qgallouedec in https://github.com/huggingface/trl/pull/3703 * [SFT] Add `seq_lengths` to signature columns by @LeonEricsson in https://github.com/huggingface/trl/pull/3699 * ⚗️ Tiny MoE for test by @qgallouedec in https://github.com/huggingface/trl/pull/3712 * BUG: Disregard pad token entropies for entropy threshold calculation by @pramodith in https://github.com/huggingface/trl/pull/3715 * Fix ORPOTrainer loss scaling with gradient accumulation by @Aratako in https://github.com/huggingface/trl/pull/3716 * [Online DPO] Safeguard logit slice against empty prompt by @LeonEricsson in https://github.com/huggingface/trl/pull/3719 * Remove deprecated `processor.tokenizer` by @Tavish9 in https://github.com/huggingface/trl/pull/3720 * 👋 Remove `--bf16` flag from training scripts by @qgallouedec in https://github.com/huggingface/trl/pull/3724 * ↔️ Fix CB in GRPO by @qgallouedec in https://github.com/huggingface/trl/pull/3722 * 📥 Set environment variables for vLLM distributed training in GRPOTrainer by @qgallouedec in https://github.com/huggingface/trl/pull/3723 * [GRPO] remove common activation offloading substring in all cases by @winglian in https://github.com/huggingface/trl/pull/3738 * 🔧 Fix GRPO sampling logic by @qgallouedec in https://github.com/huggingface/trl/pull/3725 * 🕸 Use `wandb.run.url` instead of `wandb.run.get_url()` (deprecated) by @qgallouedec in https://github.com/huggingface/trl/pull/3726 * Updated `processing_class` docs for trainers by @sergiopaniego in https://github.com/huggingface/trl/pull/3737 * Updated missing `processing_class` docs for rest of trainers by @sergiopaniego in https://github.com/huggingface/trl/pull/3745 * Add comment for `average_tokens_across_devices` by @qgallouedec in https://github.com/huggingface/trl/pull/3746 * uses `steps_per_generation` in vllm max_num_seqs by @akakakakakaa in https://github.com/huggingface/trl/pull/3747 * 🏗️ Refactor top-entropy in GRPO by @qgallouedec in https://github.com/huggingface/trl/pull/3727 * [GRPO] Fix: Processing ref logprobs in batches by @idanshen in https://github.com/huggingface/trl/pull/3740 * Add Object detection grounding recipe to Community tutorials by @sergiopaniego in https://github.com/huggingface/trl/pull/3752 * 🐙 MPO by @qgallouedec in https://github.com/huggingface/trl/pull/2544 * ⚰️ Remove deprecated by @qgallouedec in https://github.com/huggingface/trl/pull/3704 * 👨‍💼 [SFT] Packing with completion_only and assistant_only training by @LeonEricsson in https://github.com/huggingface/trl/pull/3749 * 👁️ [GRPO] Add VLM training capabilities to the trainer by @CompN3rd in https://github.com/huggingface/trl/pull/3072 * Add MPO recipe to Community tutorials by @sergiopaniego in https://github.com/huggingface/trl/pull/3766 * ✋ Prevent NCCL Device Conflicts Between vLLM Server and Trainers by @CarlosArguilar in https://github.com/huggingface/trl/pull/3762 * 🔔 Add deprecation warnings for `AlignPropTrainer` and `DDPOTrainer` by @qgallouedec in https://github.com/huggingface/trl/pull/3755 * 🔠 Support model str in OnlineDPO by @kashif in https://github.com/huggingface/trl/pull/3765 * 🌌 [GRPO] Log generation entropy by @LeonEricsson in https://github.com/huggingface/trl/pull/3700 * 🤓 [GRPO] Documentation for entropy metric by @LeonEricsson in https://github.com/huggingface/trl/pull/3770 * Add uv scripts headers by @lhoestq in https://github.com/huggingface/trl/pull/3767 * Update missing uv dep by @lhoestq in https://github.com/huggingface/trl/pull/3772 * 📐 Fix CI and `GeometricMixtureWrapper` by @qgallouedec in https://github.com/huggingface/trl/pull/3779 * 🩹 [Hotfix] Fix pynccl communicator assertion error with VLLMClient by @CarlosArguilar in https://github.com/huggingface/trl/pull/3774 * 🍿 [SFT] Fix dataset indexing which crashed with a IterableDataset by @LeonEricsson in https://github.com/huggingface/trl/pull/3771 * 🎞️ GSPO by @qgallouedec in https://github.com/huggingface/trl/pull/3775 * 🤏 [SFT] Improve doc on training on assistant only messages by @lewtun in https://github.com/huggingface/trl/pull/3784 * 📐 Add epsilon hyperparameter recommendation to GSPO by @qgallouedec in https://github.com/huggingface/trl/pull/3790 * 📍 Support training peft model with gradient checkpointing by @qgallouedec in https://github.com/huggingface/trl/pull/3785 * 💬 Fix `clone_chat_template` vocab size and support PEFT instruction tuning by @qgallouedec in https://github.com/huggingface/trl/pull/3763 * 🌋 [GRPO] add support for `pixel_attention_mask` (SmolVLM2) and `image_sizes` (LLaVa-Next) by @kashif in https://github.com/huggingface/trl/pull/3760 * 🔍 Add guidance on choosing `max_length` value and include visualization tool by @qgallouedec in https://github.com/huggingface/trl/pull/3630 * 📘 SFT doc rewrite by @qgallouedec in https://github.com/huggingface/trl/pull/3619 * 👐 FSDP2+GRPO by @SalmanMohammadi in https://github.com/huggingface/trl/pull/3687 * Release: v0.20 by @qgallouedec in https://github.com/huggingface/trl/pull/3792 ## New Contributors * @ahatamiz made their first contribution in https://github.com/huggingface/trl/pull/3634 * @Clement25 made their first contribution in https://github.com/huggingface/trl/pull/3629 * @HarryHsing made their first contribution in https://github.com/huggingface/trl/pull/3493 * @ArthurZucker made their first contribution in https://github.com/huggingface/trl/pull/3471 * @seungduk-yanolja made their first contribution in https://github.com/huggingface/trl/pull/3669 * @kswhitecross made their first contribution in https://github.com/huggingface/trl/pull/3674 * @lhoestq made their first contribution in https://github.com/huggingface/trl/pull/3688 * @CarlosArguilar made their first contribution in https://github.com/huggingface/trl/pull/3690 * @jannisborn made their first contribution in https://github.com/huggingface/trl/pull/3677 * @gitabtion made their first contribution in https://github.com/huggingface/trl/pull/3667 * @YangKai0616 made their first contribution in https://github.com/huggingface/trl/pull/3680 * @Aratako made their first contribution in https://github.com/huggingface/trl/pull/3716 * @CompN3rd made their first contribution in https://github.com/huggingface/trl/pull/3072 **Full Changelog**: https://github.com/huggingface/trl/compare/v0.19.0...v0.20.0