Added primary support for Qwen3.5/Qwen3.6/Gemma4 models and compatibility with Transformers v5. ## What's Changed * [misc] set dev version by @hiyouga in https://github.com/hiyouga/LlamaFactory/pull/9703 * fix(fp8): add Transformer Engine backend support by @sbhavani in https://github.com/hiyouga/LlamaFactory/pull/9705 * [misc] Compatible with an empty architectures field in config.json by @tangefly in https://github.com/hiyouga/LlamaFactory/pull/9709 * [model] support Youtu-LLM-2B by @isLinXu in https://github.com/hiyouga/LlamaFactory/pull/9707 * [misc] lint by @hiyouga in https://github.com/hiyouga/LlamaFactory/pull/9710 * Update pyproject.toml and requirements by @jiaqiw09 in https://github.com/hiyouga/LlamaFactory/pull/9714 * [v1] add init plugin by @hiyouga in https://github.com/hiyouga/LlamaFactory/pull/9716 * [misc] Add a PyTorch version warning for Conv3D. by @tangefly in https://github.com/hiyouga/LlamaFactory/pull/9715 * [feature] add support for EAFT loss by @ymxyll in https://github.com/hiyouga/LlamaFactory/pull/9720 * [v1] add cli sampler by @hiyouga in https://github.com/hiyouga/LlamaFactory/pull/9721 * [v1] add renderer ut by @hiyouga in https://github.com/hiyouga/LlamaFactory/pull/9722 * Update README.md by @tangefly in https://github.com/hiyouga/LlamaFactory/pull/9724 * [CI]improve cuda ci cache by @frozenleaves in https://github.com/hiyouga/LlamaFactory/pull/9725 * Add support for LiquidAI's LFM2.5 (Liquid Foundation Models) to LLaMA-Factory. by @vovanphuc in https://github.com/hiyouga/LlamaFactory/pull/9726 * Add support for LiquidAI's LFM2.5-VL vision-language model by @vovanphuc in https://github.com/hiyouga/LlamaFactory/pull/9729 * [misc] fix parser by @hiyouga in https://github.com/hiyouga/LlamaFactory/pull/9730 * [refactor] rename lfm template to lfm2 and add LFM 2.5 to README by @vovanphuc in https://github.com/hiyouga/LlamaFactory/pull/9731 * [fix] correct ktransformers example config paths and templates by @JimmyPeilinLi in https://github.com/hiyouga/LlamaFactory/pull/9732 * [model] support for microsoft's Phi-4-mini by @ctx289 in https://github.com/hiyouga/LlamaFactory/pull/9734 * [misc] fix fp8 by @hiyouga in https://github.com/hiyouga/LlamaFactory/pull/9742 * [v1] add batch generator by @hiyouga in https://github.com/hiyouga/LlamaFactory/pull/9744 * [deps] fix package by @hiyouga in https://github.com/hiyouga/LlamaFactory/pull/9745 * [model] support HY-MT model by @isLinXu in https://github.com/hiyouga/LlamaFactory/pull/9746 * [v1] upgrade batching by @hiyouga in https://github.com/hiyouga/LlamaFactory/pull/9751 * [model] fixed&added Hunyuan models by @isLinXu in https://github.com/hiyouga/LlamaFactory/pull/9750 * [v1] add sft by @hiyouga in https://github.com/hiyouga/LlamaFactory/pull/9752 * using mp to run kernel test by @frozenleaves in https://github.com/hiyouga/LlamaFactory/pull/9754 * [v1] fix kernel moe patch by @jiaqiw09 in https://github.com/hiyouga/LlamaFactory/pull/9867 * [misc] update mcore related docker and mca supported models by @Kuangdd01 in https://github.com/hiyouga/LlamaFactory/pull/10114 * [feat] support `all_exhausted_without_replacement` in datasets.interleave_datasets by @Moenupa in https://github.com/hiyouga/LlamaFactory/pull/10112 * chore: Update outdated GitHub Actions versions by @pgoslatara in https://github.com/hiyouga/LlamaFactory/pull/10123 * [v1] support training with fsdp2 by @frozenleaves in https://github.com/hiyouga/LlamaFactory/pull/9773 * [v0] Fix reward model training safetensors saving by @jiaqiw09 in https://github.com/hiyouga/LlamaFactory/pull/10137 * Fix : add visual.pos_embed to Qwen3-VL visual model keys by @je1lee in https://github.com/hiyouga/LlamaFactory/pull/10139 * [feature] support using ray.remote to start distributed training. by @xvxuopop in https://github.com/hiyouga/LlamaFactory/pull/10109 * update peft, deepspeed, adapt transformers v5 by @frozenleaves in https://github.com/hiyouga/LlamaFactory/pull/10147 * [model] support youtu-vl model by @isLinXu in https://github.com/hiyouga/LlamaFactory/pull/10152 * Fix race condition in LoggerHandler during multi-GPU training by @yurekami in https://github.com/hiyouga/LlamaFactory/pull/10156 * [assets] update readme by @hiyouga in https://github.com/hiyouga/LlamaFactory/pull/10159 * [model] support MiniCPM-o-4.5 by @isLinXu in https://github.com/hiyouga/LlamaFactory/pull/10163 * add dpo/kto fsdp fsdp2 support by @UsernameFull in https://github.com/hiyouga/LlamaFactory/pull/10127 * [model] support GLM-4.7-Flash SFT by @Shanay-Mehta in https://github.com/hiyouga/LlamaFactory/pull/10173 * [v1] init commit for v1 docs by @frozenleaves in https://github.com/hiyouga/LlamaFactory/pull/10145 * [model] support GLM-OCR SFT by @Ataraxy33 in https://github.com/hiyouga/LlamaFactory/pull/10183 * [model] add liger kernel support for Qwen3-Next by @Shanay-Mehta in https://github.com/hiyouga/LlamaFactory/pull/10176 * [V1] Add v1 LoRA/Freeze support and merge workflow by @jiaqiw09 in https://github.com/hiyouga/LlamaFactory/pull/10157 * Add ASFT by @susjunyou in https://github.com/hiyouga/LlamaFactory/pull/10174 * [V1] support deepspeed by @frozenleaves in https://github.com/hiyouga/LlamaFactory/pull/10181 * [v1] support quantization by @sunyi0505 in https://github.com/hiyouga/LlamaFactory/pull/10161 * [v0/v1] fix ut huggingface hub 429 error when transformers>=5.0.0 by @jiaqiw09 in https://github.com/hiyouga/LlamaFactory/pull/10155 * [mca] update supported models by @Kuangdd01 in https://github.com/hiyouga/LlamaFactory/pull/10196 * fix: remove safe_serialization arg for transformers v5 compatibility by @Alm0stSurely in https://github.com/hiyouga/LlamaFactory/pull/10208 * Add DeepSpeed Z3 leaf module for Qwen3-Next by @Shanay-Mehta in https://github.com/hiyouga/LlamaFactory/pull/10194 * [model] Adapt Qwen3.5 by @frozenleaves in https://github.com/hiyouga/LlamaFactory/pull/10213 * [model] update constants by @hiyouga in https://github.com/hiyouga/LlamaFactory/pull/10220 * [model] support Aeva by @louzongzhi in https://github.com/hiyouga/LlamaFactory/pull/10214 * upgrade to ROCm 7.2 base image, drop PyTorch reinstall by @mjkvaak-amd in https://github.com/hiyouga/LlamaFactory/pull/10223 * [fix] register visual part for Qwen3.5 by @Kuangdd01 in https://github.com/hiyouga/LlamaFactory/pull/10227 * [V1] add seed for training and fix gradient checkpointing by @jiaqiw09 in https://github.com/hiyouga/LlamaFactory/pull/10211 * fix(vllm): support mixed multimodal payloads by @phiott in https://github.com/hiyouga/LlamaFactory/pull/10225 * [misc] fix constants by @hiyouga in https://github.com/hiyouga/LlamaFactory/pull/10232 * Add Trackio Integration for LlamaFactory by @ParagEkbote in https://github.com/hiyouga/LlamaFactory/pull/10165 * [model] support Qwen3.5 all series models by @isLinXu in https://github.com/hiyouga/LlamaFactory/pull/10237 * fix: qwen3.5 projector path by @LittleYanlin in https://github.com/hiyouga/LlamaFactory/pull/10242 * fix: get ray head ip by @SnowCharmQ in https://github.com/hiyouga/LlamaFactory/pull/10252 * [V1] Support meta loading for full and free by @jiaqiw09 in https://github.com/hiyouga/LlamaFactory/pull/10236 * fix: Fix compatibility issue with HuggingFace Dataset Column when sav… by @pyxnpyx in https://github.com/hiyouga/LlamaFactory/pull/10254 * docs: fix Python version requirement from 3.10 to >=3.11.0 by @ll0v0ll in https://github.com/hiyouga/LlamaFactory/pull/10259 * fix: convert filter() to list in read_cloud_json to fix broken empty-check by @jnMetaCode in https://github.com/hiyouga/LlamaFactory/pull/10260 * [mca] support qwen3.5 by @Kuangdd01 in https://github.com/hiyouga/LlamaFactory/pull/10265 * fix(mm): fallback to audio_processor when feature_extractor is missing by @xxddccaa in https://github.com/hiyouga/LlamaFactory/pull/10267 * update npu docker by @frozenleaves in https://github.com/hiyouga/LlamaFactory/pull/10268 * fix(template): correct gpt_oss format_assistant by @RuijieH in https://github.com/hiyouga/LlamaFactory/pull/10269 * fix: make position_id_per_seconds configurable for Qwen2OmniPlugin by @LincolnBurrows2017 in https://github.com/hiyouga/LlamaFactory/pull/10281 * fix: unused keys in ray example by @SnowCharmQ in https://github.com/hiyouga/LlamaFactory/pull/10290 * [v1] add qwen3 templates and fix rendering plugin. by @xvxuopop in https://github.com/hiyouga/LlamaFactory/pull/10212 * fix: handle empty content list in system message by @LincolnBurrows2017 in https://github.com/hiyouga/LlamaFactory/pull/10291 * fix(MiniCPMVPlugin): fix IndexError in process_messages when training with video by @xxddccaa in https://github.com/hiyouga/LlamaFactory/pull/10276 * feat(data): add SGSC zero-hallucination B2B dataset (NOO-Protocol) by @robertglools in https://github.com/hiyouga/LlamaFactory/pull/10284 * [fix] fit neat_packing & mrope model packing by @Kuangdd01 in https://github.com/hiyouga/LlamaFactory/pull/10283 * chore: mca workflow compatible with qwen-vl series by @Kuangdd01 in https://github.com/hiyouga/LlamaFactory/pull/10303 * [liger_kernel] support Qwen3.5. by @wyt2000 in https://github.com/hiyouga/LlamaFactory/pull/10313 * fix: mimo-v2 tool call by @isLinXu in https://github.com/hiyouga/LlamaFactory/pull/10315 * [v1] add callbacks by @jiaqiw09 in https://github.com/hiyouga/LlamaFactory/pull/10255 * ci: add nginx cache config for Ascend NPU CI environment by @Goalina in https://github.com/hiyouga/LlamaFactory/pull/10323 * [V1]add init on rank0 for fsdp2 by @jiaqiw09 in https://github.com/hiyouga/LlamaFactory/pull/10264 * [v1] support ulysses cp for fsdp2 by @sunyi0505 in https://github.com/hiyouga/LlamaFactory/pull/10262 * [feat] support LlamaFactory SFT training by HyperParallel FSDP2 backend by @Cui-yshoho in https://github.com/hiyouga/LlamaFactory/pull/10289 * fix moe by @frozenleaves in https://github.com/hiyouga/LlamaFactory/pull/10334 * fix: qwen3vl timestamp by @Kuangdd01 in https://github.com/hiyouga/LlamaFactory/pull/10338 * [model] gemma4 by @Kuangdd01 in https://github.com/hiyouga/LlamaFactory/pull/10346 * fix: gemma4 mm_token_type_ids padding by @Kuangdd01 in https://github.com/hiyouga/LlamaFactory/pull/10359 * fix: set mm_projectors for omni models by @Kuangdd01 in https://github.com/hiyouga/LlamaFactory/pull/10378 * fix: projector lookup for gemma4 modules by @Kuangdd01 in https://github.com/hiyouga/LlamaFactory/pull/10382 * fix(data): SeedToolUtils.tool_extractor returns content when no tool calls found by @kuishou68 in https://github.com/hiyouga/LlamaFactory/pull/10408 * [V1]support resume training from checkpoint by @frozenleaves in https://github.com/hiyouga/LlamaFactory/pull/10280 * [v1] fix device mesh and clip_grad_norm for ulysses cp by @sunyi0505 in https://github.com/hiyouga/LlamaFactory/pull/10366 * [v1] add deepspeed zero3 trigger for low memory usage weight loading by @jiaqiw09 in https://github.com/hiyouga/LlamaFactory/pull/10300 * support qwen3.6 models by @frozenleaves in https://github.com/hiyouga/LlamaFactory/pull/10415 * [v1] fix epoch and steps by @jiaqiw09 in https://github.com/hiyouga/LlamaFactory/pull/10422 * [packing] add qwen35 patch for neat_packing by @Kuangdd01 in https://github.com/hiyouga/LlamaFactory/pull/10436 * [data] support discard history cot for multiturn by @Kuangdd01 in https://github.com/hiyouga/LlamaFactory/pull/10435 * [v1] fix init on meta in transformers v5 by @jiaqiw09 in https://github.com/hiyouga/LlamaFactory/pull/10414 * [misc] code lint by @Kuangdd01 in https://github.com/hiyouga/LlamaFactory/pull/10439 * feat(npu): add Qwen3.5 support with Partial RoPE and Hybrid Attention by @curnane-lab in https://github.com/hiyouga/LlamaFactory/pull/10421 * fix: handle NotImplementedError in export_model for transformers>=5.0 (fixes #10410) by @octo-patch in https://github.com/hiyouga/LlamaFactory/pull/10438 * [v1] fix device_mesh and sp for fsdp2 by @sunyi0505 in https://github.com/hiyouga/LlamaFactory/pull/10429 * [fa2] fix IMA when train qwen3_5 by @Kuangdd01 in https://github.com/hiyouga/LlamaFactory/pull/10448 * [model] support Hy3-Preview by @isLinXu in https://github.com/hiyouga/LlamaFactory/pull/10432 * [misc] bump transformers version upperbound by @Kuangdd01 in https://github.com/hiyouga/LlamaFactory/pull/10446 * Add KTransformers AMX MoE SFT support via Accelerate by @JimmyPeilinLi in https://github.com/hiyouga/LlamaFactory/pull/10430 * [packing] fix gdn crash when meeting dummy image by @Kuangdd01 in https://github.com/hiyouga/LlamaFactory/pull/10453 * Optimize Qwen video token metadata preprocessing by @luca-888 in https://github.com/hiyouga/LlamaFactory/pull/10404 * fix(docs): correct typo in examples/README_zh.md by @simulikeit in https://github.com/hiyouga/LlamaFactory/pull/10462 * fix(data/converter): handle None tool_calls in OpenAI-style messages by @Anai-Guo in https://github.com/hiyouga/LlamaFactory/pull/10455 * [fix] fix qwen3_6 template doc by @frozenleaves in https://github.com/hiyouga/LlamaFactory/pull/10470 * [model] support MiniCPM-V-4.6 by @tsjyma in https://github.com/hiyouga/LlamaFactory/pull/10472 * [fix] Fix MiniCPM-V-4.6 image preprocessing behavior by @tsjyma in https://github.com/hiyouga/LlamaFactory/pull/10478 * [docker] update npu docker by @xvxuopop in https://github.com/hiyouga/LlamaFactory/pull/10479 * Fix: add missing return statement in MiniCPMVPlugin.get_mm_inputs by @ZMXJJ in https://github.com/hiyouga/LlamaFactory/pull/10500 * [V1] support reward training stage by @frozenleaves in https://github.com/hiyouga/LlamaFactory/pull/10431 * add torch profiler callback by @frozenleaves in https://github.com/hiyouga/LlamaFactory/pull/10463 * [V1] add cuda fused moe kernel, implementing with triton by @frozenleaves in https://github.com/hiyouga/LlamaFactory/pull/10481 * [v1] support liger_kernel by @sunyi0505 in https://github.com/hiyouga/LlamaFactory/pull/10493 * [v1] Add FlashAttention selection and implement normal / padding-free / dynamic batching by @jiaqiw09 in https://github.com/hiyouga/LlamaFactory/pull/10469 * fix: use getattr for profiler attrs to support MCA TrainingArguments by @Copilot in https://github.com/hiyouga/LlamaFactory/pull/10506 * [v1] Implement dynamic padding-free stretrgy for batching by @XuanyuChen-SEU in https://github.com/hiyouga/LlamaFactory/pull/10507 * [v1] fix padding free with sp by @jiaqiw09 in https://github.com/hiyouga/LlamaFactory/pull/10513 * [v0] fix non-packing batch (bsz>1) for Qwen3.5 with flash attention by @jiaqiw09 in https://github.com/hiyouga/LlamaFactory/pull/10529 * [fix] Fix NPU FusedMoE and RMSNorm by @xvxuopop in https://github.com/hiyouga/LlamaFactory/pull/10512 * [version] release v0.9.5 by @hiyouga in https://github.com/hiyouga/LlamaFactory/pull/10532 ## New Contributors * @sbhavani made their first contribution in https://github.com/hiyouga/LlamaFactory/pull/9705 * @ymxyll made their first contribution in https://github.com/hiyouga/LlamaFactory/pull/9720 * @vovanphuc made their first contribution in https://github.com/hiyouga/LlamaFactory/pull/9726 * @ctx289 made their first contribution in https://github.com/hiyouga/LlamaFactory/pull/9734 * @Moenupa made their first contribution in https://github.com/hiyouga/LlamaFactory/pull/10112 * @pgoslatara made their first contribution in https://github.com/hiyouga/LlamaFactory/pull/10123 * @je1lee made their first contribution in https://github.com/hiyouga/LlamaFactory/pull/10139 * @yurekami made their first contribution in https://github.com/hiyouga/LlamaFactory/pull/10156 * @Shanay-Mehta made their first contribution in https://github.com/hiyouga/LlamaFactory/pull/10173 * @Ataraxy33 made their first contribution in https://github.com/hiyouga/LlamaFactory/pull/10183 * @susjunyou made their first contribution in https://github.com/hiyouga/LlamaFactory/pull/10174 * @Alm0stSurely made their first contribution in https://github.com/hiyouga/LlamaFactory/pull/10208 * @louzongzhi made their first contribution in https://github.com/hiyouga/LlamaFactory/pull/10214 * @mjkvaak-amd made their first contribution in https://github.com/hiyouga/LlamaFactory/pull/10223 * @phiott made their first contribution in https://github.com/hiyouga/LlamaFactory/pull/10225 * @ParagEkbote made their first contribution in https://github.com/hiyouga/LlamaFactory/pull/10165 * @LittleYanlin made their first contribution in https://github.com/hiyouga/LlamaFactory/pull/10242 * @SnowCharmQ made their first contribution in https://github.com/hiyouga/LlamaFactory/pull/10252 * @pyxnpyx made their first contribution in https://github.com/hiyouga/LlamaFactory/pull/10254 * @ll0v0ll made their first contribution in https://github.com/hiyouga/LlamaFactory/pull/10259 * @jnMetaCode made their first contribution in https://github.com/hiyouga/LlamaFactory/pull/10260 * @xxddccaa made their first contribution in https://github.com/hiyouga/LlamaFactory/pull/10267 * @RuijieH made their first contribution in https://github.com/hiyouga/LlamaFactory/pull/10269 * @LincolnBurrows2017 made their first contribution in https://github.com/hiyouga/LlamaFactory/pull/10281 * @robertglools made their first contribution in https://github.com/hiyouga/LlamaFactory/pull/10284 * @wyt2000 made their first contribution in https://github.com/hiyouga/LlamaFactory/pull/10313 * @Goalina made their first contribution in https://github.com/hiyouga/LlamaFactory/pull/10323 * @Cui-yshoho made their first contribution in https://github.com/hiyouga/LlamaFactory/pull/10289 * @kuishou68 made their first contribution in https://github.com/hiyouga/LlamaFactory/pull/10408 * @curnane-lab made their first contribution in https://github.com/hiyouga/LlamaFactory/pull/10421 * @octo-patch made their first contribution in https://github.com/hiyouga/LlamaFactory/pull/10438 * @luca-888 made their first contribution in https://github.com/hiyouga/LlamaFactory/pull/10404 * @simulikeit made their first contribution in https://github.com/hiyouga/LlamaFactory/pull/10462 * @Anai-Guo made their first contribution in https://github.com/hiyouga/LlamaFactory/pull/10455 * @tsjyma made their first contribution in https://github.com/hiyouga/LlamaFactory/pull/10472 * @XuanyuChen-SEU made their first contribution in https://github.com/hiyouga/LlamaFactory/pull/10507 **Full Changelog**: https://github.com/hiyouga/LlamaFactory/compare/v0.9.4...v0.9.5

### New features - 🔥 **[APOLLO optimizer](https://arxiv.org/abs/2412.05270)** by @zhuhanqing in #6617 - 🔥 **[SwanLab](https://github.com/SwanHubX/SwanLab)** experiment tracker by @Zeyi-Lin in #6401 - 🔥 **[Ray Trainer](https://docs.ray.io/en/latest/train/train.html)** by @erictang000 in #6542 - Batch inference with **vLLM** TP by @JieShenAI in #6190 - QLoRA on Ascend NPU by @codemayq in #6601 - Yarn and Llama3 rope scaling by @hiyouga in #6693 - Support `uv run` by @erictang000 in #6907 - Ollama modelfile auto-generation by @codemayq in #4686 - Mistral tool prompt by @AlongWY in #5473 - Llama3 and Qwen2 tool prompt by @hiyouga in #6367 and #6369 ### New models - Base models - GPT2 (0.1B/0.4B/0.8B/1.5B) 📄 - Granite 3.0-3.1 (1B/2B/3B/8B) 📄 - PaliGemma2 (3B/10B/28B) 📄🖼️ - Moonlight (16B) 📄 - DeepSeek V2-V2.5 Base (236B) 📄 - DeepSeek V3 Base (671B) 📄 - Instruct/Chat models - Granite 3.0-3.1 (1B/2B/3B/8B) by @Tuyohai in #5922 📄🤖 - DeepSeek R1 (1.5B/7B/8B/14B/32B/70B/671B) by @Qwtdgh in #6767 📄🤖 - TeleChat2 (3B/7B/12B/35B/115B) @ge-xing in #6313 📄🤖 - Qwen2.5-VL (3B/7B/72B) by @hiyouga in #6779 📄🤖🖼️ - PaliGemma2-mix (3B/10B/28B) by @Kuangdd01 in #7060 📄🤖🖼️ - Qwen2 Audio (7B) by @BUAADreamer in #6701 📄🤖🔈 - MiniCPM-V/MiniCPM-o (8B) by @BUAADreamer in #6598 and #6631 📄🤖🖼️🔈 - InternLM3-Instruct (8B) by @hhaAndroid in #6640 📄🤖 - Marco-o1 (8B) 📄🤖 - Skywork-o1 (8B) 📄🤖 - Phi-4 (14B) 📄🤖 - Moonlight Instruct (16B) 📄 - Mistral Small (24B) 📄🤖 - QwQ (32B) 📄🤖 - Llama-3.3-Instruct (70B) 📄🤖 - QvQ (72B) 📄🤖🖼️ - DeepSeek V2-V2.5 (236B) 📄🤖 - DeepSeek V3 (671B) 📄🤖 ### New datasets - Supervised fine-tuning datasets - OpenO1 (en) 📄 - Open Thoughts (en) 📄 - Open-R1-Math (en) 📄 - Chinese-DeepSeek-R1-Distill (zh) 📄 ### Changes - Refactor VLMs register by @hiyouga in #6600 - Refactor mm plugin by @hiyouga in #6895 - Refactor template by @hiyouga in #6896 - Refactor data pipeline by @hiyouga in #6901 - Update vlm arguments by @hiyouga in #6976 ### Bug fix - Add `trust_remote_code` option by @yafshar in #5819 - Fix mllama config by @hiyouga in #6137 and #6140 - Fix mllama pad by @hiyouga in #6151 and #6874 - Pin tokenizers version by @hiyouga in #6157 - Fix tokenized data loading by @village-way in #6160 - Show hostname in webui by @hykilpikonna in #6170 - Fix VLMs zero3 training by @hiyouga in #6233 - Add `skip_special_tokens` by @hiyouga in #6363 - Support non-reenterent-gc by @hiyouga in #6364 - Add `disable_shuffling` option by @hiyouga in #6388 - Fix gen kwargs by @hiyouga in #6395 - Enable module run by @youkaichao in #6457 - Fix eval loss value by @hiyouga in #6465 - Fix paligemma inference by @hiyouga in #6483 - Add deepseek v3 template by @piamo in #5507 - Add http proxy argument in dockerfile by @shibingli in #6462 - Fix trainer generate by @hiyouga in #6512 - Fix pixtral DPO training by @hiyouga in #6547 - Fix ray args by @stephen-nju in #6564 - Fix minicpm template by @BUAADreamer in #6620 - Fix stop tokens for visual detection by @hiyouga in #6624 - Pin vllm version by @hiyouga in #6629 - Fix mllama any image by @hiyouga in #6637 and #7053 - Fix tokenizer max length by @xiaosu-zhu in #6632 - Fix webui locale by @steveepreston in #6653 - Fix MiniCPM-o DPO training by @BUAADreamer in #6657 - Fix Qwen2 MoE training by @hiyouga in #6684 - Upgrade to gradio 5 by @hiyouga in #6688 - Support Japanese local file by @engchina in #6698 - Fix DPO loss by @yinpu in #6722 - Webui thinking mode by @hiyouga in #6778 - Upgrade to transformers 4.48 by @hiyouga in #6628 - Fix ci by @hiyouga in #6787 - Fix instructions about installing fa2 on win platform in readme by @neavo in #6788 - Fix minicpmv plugin by @BUAADreamer in #6801, #6890, #6946 and #6998 - Fix qwen2 tool prompt by @yueqis in #6796 - Fix llama pro by @hiyouga in #6814 - Allow thought in function call by @yueqis in #6797 - Add `ALLOW_EXTRA_ARGS` by @hiyouga in #6831 - Fix Qwen2vl plugin by @hiyouga in #6855 - Upgrade vllm to 0.7.2 by @hiyouga in #6857 - Fix unit test for tool using by @hiyouga in #6865 - Skip broken data in sharegpt converter by @JJJYmmm in #6879 - Fix qwen2.5 plugin for video by @JJJYmmm in #6868 - Parsing chat template from tokenizer by @hiyouga in #6905 (experimental) - Fix mllama KTO training by @marko1616 in #6904 - Fix grad checkpointing by @hiyouga in #6916 and #6931 - Fix ollama template by @hiyouga in #6902 - Fix ray example by @erictang000 in #6906 - Improve error handling for media by @noahc1510 in #6128 - Support split on each dataset by @SrWYG in #5522 - Fix gen kwargs in training by @aliencaocao in #5451 - Liger kernel for qwen2.5vl by @hiyouga in #6930 - Fix lora target modules by @hiyouga in #6944 - Add `ray_storage_path` by @erictang000 in #6920 - Fix trainer.predict by @hiyouga in #6972 - Add min resolution control by @hiyouga in #6975 - Upgrade transformers to 4.49 by @hiyouga in #6982 - Add seed in vllm batch predict by @JieShenAI in #7058 - Fix pyproject.toml by @hiyouga in #7067 - Upgrade CANN images by @leo-pony in #7061 - Display swanlab link by @Zeyi-Lin in #7089 - Fix hf engine by @hiyouga in #7120 - Add bailing chat template by @oldstree in #7117 - Use bicubic resampler instead of nearest by @hiyouga in #7143 - Fix Qwen2Audio plugin by @lsrami in #7166 - Destroy process group by @hiyouga in #7174 - Fix swanlab callback by @Zeyi-Lin in #7176 - Fix paligemma plugin by @hiyouga in #7181 - Escape html tag in webui by @hiyouga in #7190 - Upgrade vllm to 0.7.3 by @hiyouga in #7183 and #7193 - Fix parser by @hiyouga in #7204 - Fix function formatter by @zhangch-ss in #7201 - Fix deepspeed config by @hiyouga in #7205 - Fix dataloader by @hiyouga in #7207 - Fix export tokenizer by @hiyouga in #7230 - Update arguments by @hiyouga in #7231 - Add `swanlab_logdir` by @Zeyi-Lin in #7219 - Fix vllm batch prediction by @hiyouga in #7235 - Avoid exit after saving tokenized data by @hiyouga in #7244 - Support commit in env by @hiyouga in #7247 - Release v0.9.2 by @hiyouga in #7242 - Fix #1204 #3306 #3462 #5121 #5270 #5404 #5444 #5472 #5518 #5616 #5712 #5714 #5756 #5944 #5986 #6020 #6056 #6092 #6136 #6139 #6149 #6165 #6213 #6287 #6320 #6345 #6345 #6346 #6348 #6358 #6362 #6391 #6415 #6439 #6448 #6452 #6482 #6499 #6543 #6546 #6551 #6552 #6610 #6612 #6636 #6639 #6662 #6669 #6738 #6772 #6776 #6780 #6782 #6793 #6806 #6812 #6819 #6826 #6833 #6839 #6850 #6854 #6860 #6878 #6885 #6889 #6937 #6948 #6952 #6960 #6966 #6973 #6981 #7036 #7064 #7072 #7116 #7125 #7130 #7171 #7173 #7180 #7182 #7184 #7192 #7198 #7213 #7234 #7243

### New features - 🔥Support **Llama-3.2** and **Llama-3.2-Vision** by @marko1616 in #5547 and #5555 - 🔥Support **LLaVA-NeXT**, **LLaVA-NeXT-Video** and **Video-LLaVA** by @BUAADreamer in #5574 - 🔥Support **Pixtral** model by @Kuangdd01 in #5581 - Support EXAONE3.0 by @shing100 in #5585 - Support Index-series models by @Cuiyn in #5910 - Support Liger-Kernel for Qwen2-VL by @aliencaocao in #5438 - Support download models from ModelHub by @huniu20 in #5642 - Fix abnormal loss values in transformers 4.46.* by @hiyouga in #5852 #5871 - Support multi-image inference by @hiyouga in #5895 - Support calculating effective tokens for SFT and DPO by @wtmlon in #6078 Note: now you can install `transformers>=4.46.0,<=4.46.1` to make the [gradient accumulation fix](https://github.com/huggingface/transformers/pull/34191) enabled. ### New models - Base models - Qwen2.5 (0.5B/1.5B/3B/7B/14B/32B/72B) 📄 - Qwen2.5-Coder (0.5B/1.5B/3B/7B/14B/32B) 📄🖥️ - Llama-3.2 (1B/3B) 📄 - OpenCoder (1.5B/8B) 📄🖥️ - Index (1.9B) 📄 - Instruct/Chat models - Qwen2.5-Instruct (0.5B/1.5B/3B/7B/14B/32B/72B) 📄🤖 - Qwen2.5-Coder-Instruct (0.5B/1.5B/3B/7B/14B/32B) 📄🤖🖥️ - Llama-3.2-Instruct (1B/3B) 📄🤖 - OpenCoder-Instruct (1.5B/8B) 📄🤖🖥️ - Index-Chat (1.9B) 📄🤖 - LLaVA-NeXT (7B/8B/13B/34B/72B/110B) 📄🤖🖼️ - LLaVA-NeXT-Video (7B/34B) 📄🤖🖼️ - Video-LLaVA (7B) 📄🤖🖼️ - Pixtral (12B) 📄🤖🖼️ - EXAONE-3.0-Instruct (8B) 📄🤖 ### Bug fix - Update version of rocm docker by @HardAndHeavy in #5427 - Fix Phi-3-small template by @menibrief in #5475 - Fix function call dataset process function by @whybeyoung in #5483 - Add docker args by @StrangeBytesDev in #5533 - Fix logger by @chengchengpei in #5546 - Fix Gemma2 flash attention warning by @amrear in #5580 - Update setup by @johnnynunez in #5615 #5665 - Add project by @NLPJCL in #5801 - Fix saving Qwen2-VL processor by @hiyouga in #5857 - Support change base image in dockerfile by @sd3ntato in #5880 - Fix template replace behaviour by @hiyouga in #5907 - Add `image_dir` argument by @hiyouga in #5909 - Add rank0 logger by @hiyouga in #5912 - Fix DPO metrics by @hiyouga in #5913 #6052 - Update datasets version by @hiyouga in #5926 - Fix chat engines by @hiyouga in #5927 - Fix vllm 0.6.3 by @hiyouga in #5970 - Fix extra args in llamaboard by @hiyouga in #5971 - Fix vllm input args by @JJJJerry in #5973 - Add `vllm_config` args by @hiyouga in #5982 #5990 - Add shm_size in docker compose config by @XYZliang in #6010 - Fix tyro version by @hiyouga in #6065 - Fix ci by @hiyouga in #6120 - Fix Qwen2-VL inference on vLLM by @hiyouga in #6123 #6126 - Release v0.9.1 by @hiyouga in #6124 - Fix #3881 #4712 #5411 #5542 #5549 #5611 #5668 #5705 #5747 #5749 #5768 #5796 #5797 #5883 #5904 #5966 #5988 #6050 #6061

### Congratulations on 30,000 stars 🎉 Follow us at *[X](https://twitter.com/llamafactory_ai)* ### New features - 🔥Support fine-tuning **[Qwen2-VL](https://github.com/QwenLM/Qwen2-VL)** model on multi-image datasets by @simonJJJ in #5290 - 🔥Support time&memory-efficient **[Liger-Kernel](https://github.com/linkedin/Liger-Kernel)** via the `enable_liger_kernel` argument by @hiyouga - 🔥Support memory-efficient **[Adam-mini](https://github.com/zyushun/Adam-mini)** optimizer via the `use_adam_mini` argument by @relic-yuexi in #5095 - Support fine-tuning Qwen2-VL model on video datasets by @hiyouga in #5365 and @BUAADreamer in #4136 (needs patch https://github.com/huggingface/transformers/pull/33307) - Support fine-tuning vision language models (VLMs) using RLHF/DPO/ORPO/SimPO approaches by @hiyouga - Support [Unsloth](https://unsloth.ai/blog/long-context)'s asynchronous activation offloading method via the `use_unsloth_gc` argument - Support [vLLM](https://github.com/vllm-project/vllm) 0.6.0 version - Support MFU calculation by @yzoaim in #5388 ### New models - Base models - Qwen2-Math (1.5B/7B/72B) 📄🔢 - Yi-Coder (1.5B/9B) 📄 - InternLM2.5 (1.8B/7B/20B) 📄 - Gemma-2-2B 📄 - Meta-Llama-3.1 (8B/70B) 📄 - Instruct/Chat models - MiniCPM/MiniCPM3 (1B/2B/4B) by @LDLINGLINGLING in #4996 #5372 📄🤖 - Qwen2-Math-Instruct (1.5B/7B/72B) 📄🤖🔢 - Yi-Coder-Chat (1.5B/9B) 📄🤖 - InternLM2.5-Chat (1.8B/7B/20B) 📄🤖 - Qwen2-VL-Instruct (2B/7B) 📄🤖🖼️ - Gemma-2-2B-it by @codemayq in #5037 📄🤖 - Meta-Llama-3.1-Instruct (8B/70B) 📄🤖 - Mistral-Nemo-Instruct (12B) 📄🤖 ### New datasets - Supervised fine-tuning datasets - Magpie-ultra-v0.1 (en) 📄 - Pokemon-gpt4o-captions (en&zh) 📄🖼️ - Preference datasets - RLHF-V (en) 📄🖼️ - VLFeedback (en) 📄🖼️ ### Changes - Due to compatibility consideration, fine-tuning vision language models (VLMs) needs `transformers>=4.35.0.dev0`, try `pip install git+https://github.com/huggingface/transformers.git` to install it. - `visual_inputs` has been deprecated, now you do not need to specify this argument. - LlamaFactory now adopts lazy loading for multimodal inputs, see #5346 for details. Please use `preprocessing_batch_size` to restrict the batch size in dataset pre-processing (supported by @naem1023 in #5323 ). - LlamaFactory now supports `lmf` (equivalent to `llamafactory-cli`) as a shortcut command. ### Bug fix - Fix LlamaBoard export by @liuwwang in #4950 - Add ROCm dockerfiles by @HardAndHeavy in #4970 - Fix deepseek template by @piamo in #4892 - Fix pissa savecallback by @codemayq in #4995 - Add Korean display language in LlamaBoard by @Eruly in #5010 - Fix deepseekcoder template by @relic-yuexi in #5072 - Fix examples by @codemayq in #5109 - Fix `mask_history` truncate from last by @YeQiuO in #5115 - Fix jinja template by @YeQiuO in #5156 - Fix PPO optimizer and lr scheduler by @liu-zichen in #5163 - Add SailorLLM template by @chenhuiyu in #5185 - Fix XPU device count by @Zxilly in #5188 - Fix bf16 check in NPU by @Ricardo-L-C in #5193 - Update NPU docker image by @MengqingCao in #5230 - Fix image input api by @marko1616 in #5237 - Add liger-kernel link by @ByronHsu in #5317 - Fix #4684 #4696 #4917 #4925 #4928 #4944 #4959 #4992 #5035 #5048 #5060 #5092 #5228 #5252 #5292 #5295 #5305 #5307 #5308 #5324 #5331 #5334 #5338 #5344 #5366 #5384

### New features - 🔥Support [contamination-free packing](https://github.com/MeetKai/functionary/tree/main/functionary/train/packing) via the `neat_packing` argument by @chuan298 in #4224 - 🔥Support split evaluation via the `eval_dataset` argument by @codemayq in #4691 - 🔥Support HQQ/EETQ quantization via the `quantization_method` argument by @hiyouga - 🔥Support ZeRO-3 when using BAdam by @Ledzy in #4352 - Support train on the last turn via the `mask_history` argument by @aofengdaxia in #4878 - Add NPU Dockerfile by @MengqingCao in #4355 - Support building FlashAttention2 in Dockerfile by @hzhaoy in #4461 - Support `batch_eval_metrics` at evaluation by @hiyouga ### New models - Base models - InternLM2.5-7B 📄 - Gemma2 (9B/27B) 📄 - Instruct/Chat models - TeleChat-1B-Chat by @hzhaoy in #4651 📄🤖 - InternLM2.5-7B-Chat 📄🤖 - CodeGeeX4-9B-Chat 📄🤖 - Gemma2-it (9B/27B) 📄🤖 ### Changes - Fix DPO cutoff len and deprecate `reserved_label_len` argument - Improve loss function for reward modeling ### Bug fix - Fix numpy version by @MengqingCao in #4382 - Improve cli by @kno10 in #4409 - Add `tool_format` parameter to control prompt by @mMrBun in #4417 - Automatically label npu issue by @MengqingCao in #4445 - Fix flash_attn args by @stceum in #4446 - Fix docker-compose path by @MengqingCao in #4544 - Fix torch-npu dependency by @hashstone in #4561 - Fix deepspeed + pissa by @hzhaoy in #4580 - Improve cli by @injet-zhou in #4590 - Add project by @wzh1994 in #4662 - Fix docstring by @hzhaoy in #4673 - Fix Windows command preview in WebUI by @marko1616 in #4700 - Fix vllm 0.5.1 by @T-Atlas in #4706 - Fix save value head model callback by @yzoaim in #4746 - Fix CUDA Dockerfile by @hzhaoy in #4781 - Fix examples by @codemayq in #4804 - Fix evaluation data split by @codemayq in #4821 - Fix CI by @codemayq in #4822 - Fix #2290 #3974 #4113 #4379 #4398 #4402 #4410 #4419 #4432 #4456 #4458 #4549 #4556 #4579 #4592 #4609 #4617 #4674 #4677 #4683 #4684 #4699 #4705 #4731 #4742 #4779 #4780 #4786 #4792 #4820 #4826

### New features - Support GLM-4 tools and parallel function calling by @mMrBun in #4173 - Support **PiSSA** fine-tuning by @hiyouga in #4307 ### New models - Base models - DeepSeek-Coder-V2 (16B MoE/236B MoE) 📄 - Instruct/Chat models - MiniCPM-2B 📄🤖 - DeepSeek-Coder-V2-Instruct (16B MoE/236B MoE) 📄🤖 ### New datasets - Supervised fine-tuning datasets - Neo-sft (zh) - Magpie-Pro-300K-Filtered (en) by @EliMCosta in #4309 - WebInstruct (en) by @EliMCosta in #4309 ### Bug fix - Add MANIFEST.in by @iamthebot in #4191 - Fix eos_token in llama3 pretrain by @dignfei in #4204 - Fix vllm version by @kimdwkimdw and @hzhaoy in #4234 and #4246 - Fix Dockerfile by @EliMCosta in #4314 - Fix pandas version by @zzxzz12345 in #4334 - Fix #3162 #3196 #3778 #4198 #4209 #4221 #4227 #4238 #4242 #4271 #4292 #4295 #4326 #4346 #4357 #4362

- Fix #2666: Unsloth+DoRA - Fix #4145: The PyTorch version of the docker image does not match the vLLM requirement - Fix #4160: The problem in LongLoRA implementation with the help of @f-q23 - Fix #4167: The installation problem in the Windows system by @yzoaim

### Stronger [LlamaBoard](https://github.com/hiyouga/LLaMA-Factory?tab=readme-ov-file#fine-tuning-with-llama-board-gui-powered-by-gradio) 💪😀 - Support single-node distributed training in Web UI - Support dropdown menu for easily resuming from checkpoints and picking saved configurations by @hiyouga and @hzhaoy in #4053 - Support selecting checkpoints of full/freeze tuning - Add throughput metrics to LlamaBoard by @injet-zhou in #4066 - Faster UI loading ### New features - Add KTO algorithm by @enji-zhou in #3785 - Add SimPO algorithm by @hiyouga - Support passing `max_lora_rank` to the vLLM backend by @jue-jue-zi in #3794 - Support preference datasets in sharegpt format and remove big files from git repo by @hiyouga in #3799 - Support setting system messages in CLI inference by @ycjcl868 in #3812 - Add `num_samples` option in `dataset_info.json` by @seanzhang-zhichen in #3829 - Add NPU docker image by @dongdongqiang2018 in #3876 - Improve NPU document by @MengqingCao in #3930 - Support SFT packing with greedy knapsack algorithm by @AlongWY in #4009 - Add `llamafactory-cli env` for bug report - Support image input in the API mode - Support random initialization via the `train_from_scratch` argument - Initialize CI ### New models - Base models - Qwen2 (0.5B/1.5B/7B/72B/MoE) 📄 - PaliGemma-3B (pt/mix) 📄🖼️ - GLM-4-9B 📄 - Falcon-11B 📄 - DeepSeek-V2-Lite (16B) 📄 - Instruct/Chat models - Qwen2-Instruct (0.5B/1.5B/7B/72B/MoE) 📄🤖 - Mistral-7B-Instruct-v0.3 📄🤖 - Phi-3-small-8k-instruct (7B) 📄🤖 - Aya-23 (8B/35B) 📄🤖 - OpenChat-3.6-8B 📄🤖 - GLM-4-9B-Chat 📄🤖 - TeleChat-12B-Chat by @hzhaoy in #3958 📄🤖 - Phi-3-medium-8k-instruct (14B) 📄🤖 - DeepSeek-V2-Lite-Chat (16B) 📄🤖 - Codestral-22B-v0.1 📄🤖 ### New datasets - Pre-training datasets - FineWeb (en) - FineWeb-Edu (en) - Supervised fine-tuning datasets - Ruozhiba-GPT4 (zh) - STEM-Instruction (zh) - Preference datasets - Argilla-KTO-mix-15K (en) - UltraFeedback (en) ### Bug fix - Fix RLHF for multimodal finetuning - Fix LoRA target in multimodal finetuning by @BUAADreamer in #3835 - Fix `yi` template by @Yimi81 in #3925 - Fix abort issue in LlamaBoard by @injet-zhou in #3987 - Pass `scheduler_specific_kwargs` to `get_scheduler` by @Uminosachi in #4006 - Fix hyperparameters help by @xu-song in #4007 - Update issue template by @statelesshz in #4011 - Fix vllm dtype parameter - Fix exporting hyperparameters by @MengqingCao in #4080 - Fix DeepSpeed ZeRO3 in PPO trainer - Fix #3108 #3387 #3646 #3717 #3764 #3769 #3803 #3807 #3818 #3837 #3847 #3853 #3873 #3900 #3931 #3965 #3971 #3978 #3992 #4005 #4012 #4013 #4022 #4033 #4043 #4061 #4075 #4077 #4079 #4085 #4090 #4120 #4132 #4137 #4139