## What's Changed ### Release * [release] release version by @flybird11111 in https://github.com/hpcaitech/ColossalAI/pull/6330 ### Upgrade * [upgrade]Upgrade transformers by @flybird11111 in https://github.com/hpcaitech/ColossalAI/pull/6320 * Update README.md by @Yanjia0 in https://github.com/hpcaitech/ColossalAI/pull/6268 * [ci] update ci by @flybird11111 in https://github.com/hpcaitech/ColossalAI/pull/6254 ### HotFix * [HotFix] update load lora model Readme; by @duanjunwen in https://github.com/hpcaitech/ColossalAI/pull/6240 **Full Changelog**: https://github.com/hpcaitech/ColossalAI/compare/v0.4.9...v0.5.0

## What's Changed ### Release - [release] update version (#6236) by [Hongxin Liu](https://api.github.com/users/ver217) ### Hotfix - [hotfix] fix lora load (#6231) by [Hongxin Liu](https://api.github.com/users/ver217) ### Misc - [misc] update torch version (#6206) by [Hongxin Liu](https://api.github.com/users/ver217) ### Chat - Merge pull request #6208 from hpcaitech/grpo_dev by [YeAnbang](https://api.github.com/users/YeAnbang) ### Pre-commit.ci - [pre-commit.ci] auto fixes from pre-commit.com hooks by [pre-commit-ci[bot]](https://api.github.com/users/pre-commit-ci%5Bbot%5D) **Full Changelog**: https://github.com/hpcaitech/ColossalAI/compare/v0.4.9...v0.4.8

## What's Changed ### Release - [release] update version (#6195) by [Hongxin Liu](https://api.github.com/users/ver217) ### Doc - [doc] DeepSeek V3/R1 news (#6199) by [binmakeswell](https://api.github.com/users/binmakeswell) ### Application - [application] add lora sft example data (#6198) by [Hongxin Liu](https://api.github.com/users/ver217) - [application] Update README (#6196) by [Tong Li](https://api.github.com/users/TongLi3701) - [application] add lora sft example (#6192) by [Hongxin Liu](https://api.github.com/users/ver217) ### Pre-commit.ci - Add GRPO and Support RLVR for PPO (#6186) by [YeAnbang](https://api.github.com/users/YeAnbang) ### Checkpointio - [checkpointio] fix for async io (#6189) by [flybird11111](https://api.github.com/users/flybird11111) - [checkpointio] fix checkpoint for 3d (#6187) by [flybird11111](https://api.github.com/users/flybird11111) - [checkpointio] gather tensor before unpad it if the tensor is both padded and distributed (#6168) by [Lemon Qin](https://api.github.com/users/Lemon-412) - [checkpointio] support load-pin overlap (#6177) by [Hongxin Liu](https://api.github.com/users/ver217) ### Hotfix - [hotfix] fix zero optim save (#6191) by [Hongxin Liu](https://api.github.com/users/ver217) - [hotfix] fix hybrid checkpointio for sp+dp (#6184) by [flybird11111](https://api.github.com/users/flybird11111) ### Shardformer - [shardformer] support pipeline for deepseek v3 and optimize lora save (#6188) by [Hongxin Liu](https://api.github.com/users/ver217) - [shardformer] support ep for deepseek v3 (#6185) by [Hongxin Liu](https://api.github.com/users/ver217) ### Ci - [CI] Cleanup Dist Optim tests with shared helper funcs (#6125) by [Wenxuan Tan](https://api.github.com/users/Edenzzzz) ### Issue template - [Issue template] Add checkbox asking for details to reproduce error (#6104) by [Wenxuan Tan](https://api.github.com/users/Edenzzzz) ### Inference - [Inference]Fix example in readme (#6178) by [Guangyao Zhang](https://api.github.com/users/GuangyaoZhang) **Full Changelog**: https://github.com/hpcaitech/ColossalAI/compare/v0.4.8...v0.4.7

## What's Changed ### Release - [release] update version (#6174) by [Hongxin Liu](https://api.github.com/users/ver217) ### Pre-commit.ci - [pre-commit.ci] pre-commit autoupdate (#6113) by [pre-commit-ci[bot]](https://api.github.com/users/pre-commit-ci%5Bbot%5D) ### Sharderformer - [Sharderformer] Support zbv in Sharderformer Policy (#6150) by [duanjunwen](https://api.github.com/users/duanjunwen) ### Checkpointio - [checkpointio] support non blocking pin load (#6172) by [Hongxin Liu](https://api.github.com/users/ver217) - [checkpointio]support asyncio for 3d (#6152) by [flybird11111](https://api.github.com/users/flybird11111) - [checkpointio] fix async io (#6155) by [flybird11111](https://api.github.com/users/flybird11111) - [checkpointio] support debug log (#6153) by [Hongxin Liu](https://api.github.com/users/ver217) - [checkpointio] fix zero optimizer async save memory (#6151) by [Hongxin Liu](https://api.github.com/users/ver217) - Merge pull request #6149 from ver217/hotfix/ckpt by [Wang Binluo](https://api.github.com/users/wangbluo) - [checkpointio] disable buffering by [ver217](https://api.github.com/users/ver217) - [checkpointio] fix pinned state dict by [ver217](https://api.github.com/users/ver217) - [checkpointio] fix size compute by [ver217](https://api.github.com/users/ver217) - [checkpointio] fix performance issue (#6139) by [Hongxin Liu](https://api.github.com/users/ver217) - [checkpointio] support async model save (#6131) by [Hongxin Liu](https://api.github.com/users/ver217) ### News - [news] release colossalai for sora (#6166) by [binmakeswell](https://api.github.com/users/binmakeswell) ### Hotfix - [hotfix] improve compatibility (#6165) by [Hongxin Liu](https://api.github.com/users/ver217) - [Hotfix] hotfix normalization (#6163) by [duanjunwen](https://api.github.com/users/duanjunwen) - [hotfix] fix zero comm buffer init (#6154) by [Hongxin Liu](https://api.github.com/users/ver217) - [hotfix] fix flash attn window_size err (#6132) by [duanjunwen](https://api.github.com/users/duanjunwen) ### Doc - [doc] add bonus event (#6164) by [binmakeswell](https://api.github.com/users/binmakeswell) - [doc] update cloud link (#6148) by [Sze-qq](https://api.github.com/users/Sze-qq) - [doc] add hpc cloud intro (#6147) by [Sze-qq](https://api.github.com/users/Sze-qq) ### Device - [Device]Support npu (#6159) by [flybird11111](https://api.github.com/users/flybird11111) ### Fix - [fix] fix bug caused by perf version (#6156) by [duanjunwen](https://api.github.com/users/duanjunwen) - [fix] multi-node backward slowdown (#6134) by [Hanks](https://api.github.com/users/BurkeHulk) ### Optim - [optim] hotfix adam load (#6146) by [Hongxin Liu](https://api.github.com/users/ver217) ### Zerobubble - [Zerobubble] merge main. (#6142) by [duanjunwen](https://api.github.com/users/duanjunwen) ### Async io - [async io]supoort async io (#6137) by [flybird11111](https://api.github.com/users/flybird11111) ### Ckpt - [ckpt] Add async ckpt api (#6136) by [Wang Binluo](https://api.github.com/users/wangbluo) ### Cli - [cli] support run as module option (#6135) by [Hongxin Liu](https://api.github.com/users/ver217) ### Zero - [zero] support extra dp (#6123) by [Hongxin Liu](https://api.github.com/users/ver217) ### Coati - [Coati] Refine prompt for better inference (#6117) by [Tong Li](https://api.github.com/users/TongLi3701) ### Plugin - [plugin] support get_grad_norm (#6115) by [Hongxin Liu](https://api.github.com/users/ver217) **Full Changelog**: https://github.com/hpcaitech/ColossalAI/compare/v0.4.7...v0.4.6

## What's Changed ### Release - [release] update version (#6109) by [Hongxin Liu](https://api.github.com/users/ver217) ### Pre-commit.ci - [pre-commit.ci] pre-commit autoupdate (#6078) by [pre-commit-ci[bot]](https://api.github.com/users/pre-commit-ci%5Bbot%5D) ### Checkpointio - [checkpointio] fix hybrid plugin model save (#6106) by [Hongxin Liu](https://api.github.com/users/ver217) ### Mcts - [MCTS] Add self-refined MCTS (#6098) by [Tong Li](https://api.github.com/users/TongLi3701) ### Doc - [doc] sora solution news (#6100) by [binmakeswell](https://api.github.com/users/binmakeswell) ### Extension - [extension] hotfix compile check (#6099) by [Hongxin Liu](https://api.github.com/users/ver217) ### Hotfix - Merge pull request #6096 from BurkeHulk/hotfix/lora_ckpt by [Hanks](https://api.github.com/users/BurkeHulk) **Full Changelog**: https://github.com/hpcaitech/ColossalAI/compare/v0.4.6...v0.4.5

## What's Changed ### Release - [release] update version (#6094) by [Hongxin Liu](https://api.github.com/users/ver217) ### Misc - [misc] fit torch api upgradation and remove legecy import (#6093) by [Hongxin Liu](https://api.github.com/users/ver217) ### Fp8 - [fp8] add fallback and make compile option configurable (#6092) by [Hongxin Liu](https://api.github.com/users/ver217) ### Chore - [chore] refactor by [botbw](https://api.github.com/users/botbw) ### Ckpt - [ckpt] add safetensors util by [botbw](https://api.github.com/users/botbw) ### Pipeline - [pipeline] hotfix backward for multiple outputs (#6090) by [Hongxin Liu](https://api.github.com/users/ver217) ### Ring attention - [Ring Attention] Improve comments (#6085) by [Wenxuan Tan](https://api.github.com/users/Edenzzzz) - Merge pull request #6071 from wangbluo/ring_attention by [Wang Binluo](https://api.github.com/users/wangbluo) ### Coati - [Coati] Train DPO using PP (#6054) by [Tong Li](https://api.github.com/users/TongLi3701) ### Shardformer - [shardformer] optimize seq parallelism (#6086) by [Hongxin Liu](https://api.github.com/users/ver217) - [shardformer] fix linear 1d row and support uneven splits for fused qkv linear (#6084) by [Hongxin Liu](https://api.github.com/users/ver217) **Full Changelog**: https://github.com/hpcaitech/ColossalAI/compare/v0.4.5...v0.4.4

## What's Changed ### Release - [release] update version (#6062) by [Hongxin Liu](https://api.github.com/users/ver217) ### Colossaleval - [ColossalEval] support for vllm (#6056) by [Camille Zhong](https://api.github.com/users/Camille7777) ### Moe - [moe] add parallel strategy for shared_expert && fix test for deepseek (#6063) by [botbw](https://api.github.com/users/botbw) ### Sp - Merge pull request #6064 from wangbluo/fix_attn by [Wang Binluo](https://api.github.com/users/wangbluo) - Merge pull request #6061 from wangbluo/sp_fix by [Wang Binluo](https://api.github.com/users/wangbluo) ### Doc - [doc] FP8 training and communication document (#6050) by [Guangyao Zhang](https://api.github.com/users/GuangyaoZhang) - [doc] update sp doc (#6055) by [flybird11111](https://api.github.com/users/flybird11111) ### Fp8 - [fp8] Disable all_gather intranode. Disable Redundant all_gather fp8 (#6059) by [Guangyao Zhang](https://api.github.com/users/GuangyaoZhang) - [fp8] fix missing fp8_comm flag in mixtral (#6057) by [botbw](https://api.github.com/users/botbw) - [fp8] hotfix backward hook (#6053) by [Hongxin Liu](https://api.github.com/users/ver217) ### Pre-commit.ci - [pre-commit.ci] auto fixes from pre-commit.com hooks by [pre-commit-ci[bot]](https://api.github.com/users/pre-commit-ci%5Bbot%5D) ### Hotfix - [hotfix] moe hybrid parallelism benchmark & follow-up fix (#6048) by [botbw](https://api.github.com/users/botbw) ### Feature - [Feature] Split cross-entropy computation in SP (#5959) by [Wenxuan Tan](https://api.github.com/users/Edenzzzz) **Full Changelog**: https://github.com/hpcaitech/ColossalAI/compare/v0.4.4...v0.4.3

## What's Changed ### Release - [release] update version (#6041) by [Hongxin Liu](https://api.github.com/users/ver217) ### Fp8 - [fp8] disable all_to_all_fp8 in intranode (#6045) by [Hanks](https://api.github.com/users/BurkeHulk) - [fp8] fix linear hook (#6046) by [Hongxin Liu](https://api.github.com/users/ver217) - [fp8] optimize all-gather (#6043) by [Hongxin Liu](https://api.github.com/users/ver217) - [FP8] unsqueeze scale to make it compatible with torch.compile (#6040) by [Guangyao Zhang](https://api.github.com/users/GuangyaoZhang) - Merge pull request #6012 from hpcaitech/feature/fp8_comm by [Hongxin Liu](https://api.github.com/users/ver217) - Merge pull request #6033 from wangbluo/fix by [Wang Binluo](https://api.github.com/users/wangbluo) - Merge pull request #6024 from wangbluo/fix_merge by [Wang Binluo](https://api.github.com/users/wangbluo) - Merge pull request #6023 from wangbluo/fp8_merge by [Wang Binluo](https://api.github.com/users/wangbluo) - [fp8] Merge feature/fp8_comm to main branch of Colossalai (#6016) by [Wang Binluo](https://api.github.com/users/wangbluo) - [fp8] zero support fp8 linear. (#6006) by [flybird11111](https://api.github.com/users/flybird11111) - [fp8] add use_fp8 option for MoeHybridParallelPlugin (#6009) by [Wang Binluo](https://api.github.com/users/wangbluo) - [fp8]update reduce-scatter test (#6002) by [flybird11111](https://api.github.com/users/flybird11111) - [fp8] linear perf enhancement by [botbw](https://api.github.com/users/botbw) - [fp8] update torch.compile for linear_fp8 to >= 2.4.0 (#6004) by [botbw](https://api.github.com/users/botbw) - [fp8] support asynchronous FP8 communication (#5997) by [flybird11111](https://api.github.com/users/flybird11111) - [fp8] refactor fp8 linear with compile (#5993) by [Hongxin Liu](https://api.github.com/users/ver217) - [fp8] support hybrid parallel plugin (#5982) by [Wang Binluo](https://api.github.com/users/wangbluo) - [fp8]Moe support fp8 communication (#5977) by [flybird11111](https://api.github.com/users/flybird11111) - [fp8] use torch compile (torch >= 2.3.0) (#5979) by [botbw](https://api.github.com/users/botbw) - [fp8] support gemini plugin (#5978) by [Hongxin Liu](https://api.github.com/users/ver217) - [fp8] support fp8 amp for hybrid parallel plugin (#5975) by [Hongxin Liu](https://api.github.com/users/ver217) - [fp8] add fp8 linear (#5967) by [Hongxin Liu](https://api.github.com/users/ver217) - [fp8]support all2all fp8 (#5953) by [flybird11111](https://api.github.com/users/flybird11111) - [FP8] rebase main (#5963) by [flybird11111](https://api.github.com/users/flybird11111) - Merge pull request #5961 from ver217/feature/zeor-fp8 by [Hanks](https://api.github.com/users/BurkeHulk) - [fp8] add fp8 comm for low level zero by [ver217](https://api.github.com/users/ver217) ### Hotfix - [Hotfix] Remove deprecated install (#6042) by [Tong Li](https://api.github.com/users/TongLi3701) - [Hotfix] Fix llama fwd replacement bug (#6031) by [Wenxuan Tan](https://api.github.com/users/Edenzzzz) - [Hotfix] Avoid fused RMSnorm import error without apex (#5985) by [Edenzzzz](https://api.github.com/users/Edenzzzz) - [Hotfix] README link (#5966) by [Tong Li](https://api.github.com/users/TongLi3701) - [hotfix] Remove unused plan section (#5957) by [Tong Li](https://api.github.com/users/TongLi3701) ### Colossalai/checkpoint_io/... - [colossalai/checkpoint_io/...] fix bug in load_state_dict_into_model; format error msg (#6020) by [Gao, Ruiyuan](https://api.github.com/users/flymin) ### Colossal-llama - [Colossal-LLaMA] Refactor latest APIs (#6030) by [Tong Li](https://api.github.com/users/TongLi3701) ### Plugin - [plugin] hotfix zero plugin (#6036) by [Hongxin Liu](https://api.github.com/users/ver217) - [plugin] add cast inputs option for zero (#6003) (#6022) by [Hongxin Liu](https://api.github.com/users/ver217) - [plugin] add cast inputs option for zero (#6003) by [Hongxin Liu](https://api.github.com/users/ver217) ### Ci - [CI] Remove triton version for compatibility bug; update req torch >=2.2 (#6018) by [Wenxuan Tan](https://api.github.com/users/Edenzzzz) ### Pre-commit.ci - [pre-commit.ci] auto fixes from pre-commit.com hooks by [pre-commit-ci[bot]](https://api.github.com/users/pre-commit-ci%5Bbot%5D) - [pre-commit.ci] auto fixes from pre-commit.com hooks by [pre-commit-ci[bot]](https://api.github.com/users/pre-commit-ci%5Bbot%5D) - [pre-commit.ci] auto fixes from pre-commit.com hooks by [pre-commit-ci[bot]](https://api.github.com/users/pre-commit-ci%5Bbot%5D) - [pre-commit.ci] auto fixes from pre-commit.com hooks by [pre-commit-ci[bot]](https://api.github.com/users/pre-commit-ci%5Bbot%5D) - [pre-commit.ci] auto fixes from pre-commit.com hooks by [pre-commit-ci[bot]](https://api.github.com/users/pre-commit-ci%5Bbot%5D) - [pre-commit.ci] pre-commit autoupdate (#5995) by [pre-commit-ci[bot]](https://api.github.com/users/pre-commit-ci%5Bbot%5D) - [pre-commit.ci] auto fixes from pre-commit.com hooks by [pre-commit-ci[bot]](https://api.github.com/users/pre-commit-ci%5Bbot%5D) ### Colossalchat - [ColossalChat] Add PP support (#6001) by [Tong Li](https://api.github.com/users/TongLi3701) ### Misc - [misc] Use dist logger in plugins (#6011) by [Edenzzzz](https://api.github.com/users/Edenzzzz) - [misc] update compatibility (#6008) by [Hongxin Liu](https://api.github.com/users/ver217) - [misc] Bypass the huggingface bug to solve the mask mismatch problem (#5991) by [Haze188](https://api.github.com/users/Hz188) - [misc] remove useless condition by [haze188](https://api.github.com/users/Hz188) - [misc] fix ci failure: change default value to false in moe plugin by [haze188](https://api.github.com/users/Hz188) - [misc] remove incompatible test config by [haze188](https://api.github.com/users/Hz188) - [misc] remove debug/print code by [haze188](https://api.github.com/users/Hz188) - [misc] skip redunant test by [haze188](https://api.github.com/users/Hz188) - [misc] solve booster hang by rename the variable by [haze188](https://api.github.com/users/Hz188) ### Feature - [Feature] Zigzag Ring attention (#5905) by [Edenzzzz](https://api.github.com/users/Edenzzzz) - [Feature]: support FP8 communication in DDP, FSDP, Gemini (#5928) by [Hanks](https://api.github.com/users/BurkeHulk) - [Feature] llama shardformer fp8 support (#5938) by [Guangyao Zhang](https://api.github.com/users/GuangyaoZhang) - [Feature] MoE Ulysses Support (#5918) by [Haze188](https://api.github.com/users/Hz188) ### Chat - [Chat] fix readme (#5989) by [YeAnbang](https://api.github.com/users/YeAnbang) - Merge pull request #5962 from hpcaitech/colossalchat by [YeAnbang](https://api.github.com/users/YeAnbang) - [Chat] Fix lora (#5946) by [YeAnbang](https://api.github.com/users/YeAnbang) ### Test ci - [test ci]Feature/fp8 comm (#5981) by [flybird11111](https://api.github.com/users/flybird11111) ### Docs - [Docs] clarify launch port by [Edenzzzz](https://api.github.com/users/Edenzzzz) ### Test - [test] add zero fp8 test case by [ver217](https://api.github.com/users/ver217) - [test] add check by [hxwang](https://api.github.com/users/botbw) - [test] fix test: test_zero1_2 by [hxwang](https://api.github.com/users/botbw) - [test] add mixtral modelling test by [botbw](https://api.github.com/users/botbw) - [test] pass mixtral shardformer test by [botbw](https://api.github.com/users/botbw) - [test] mixtra pp shard test by [hxwang](https://api.github.com/users/botbw) - [test] add mixtral transformer test by [hxwang](https://api.github.com/users/botbw) - [test] add mixtral for sequence classification by [hxwang](https://api.github.com/users/botbw) ### Lora - [lora] lora support hybrid parallel plugin (#5956) by [Wang Binluo](https://api.github.com/users/wangbluo) ### Feat - [feat] Dist Loader for Eval (#5950) by [Tong Li](https://api.github.com/users/TongLi3701) ### Chore - [chore] remove redundant test case, print string & reduce test tokens by [botbw](https://api.github.com/users/botbw) - [chore] docstring by [hxwang](https://api.github.com/users/botbw) - [chore] change moe_pg_mesh to private by [hxwang](https://api.github.com/users/botbw) - [chore] solve moe ckpt test failure and some other arg pass failure by [hxwang](https://api.github.com/users/botbw) - [chore] minor fix after rebase by [hxwang](https://api.github.com/users/botbw) - [chore] minor fix by [hxwang](https://api.github.com/users/botbw) - [chore] arg pass & remove drop token by [hxwang](https://api.github.com/users/botbw) - [chore] trivial fix by [botbw](https://api.github.com/users/botbw) - [chore] manually revert unintended commit by [botbw](https://api.github.com/users/botbw) - [chore] handle non member group by [hxwang](https://api.github.com/users/botbw) ### Moe - [moe] solve dp axis issue by [botbw](https://api.github.com/users/botbw) - [moe] remove force_overlap_comm flag and add warning instead by [hxwang](https://api.github.com/users/botbw) - Revert "[moe] implement submesh initialization" by [hxwang](https://api.github.com/users/botbw) - [moe] refactor mesh assignment by [hxwang](https://api.github.com/users/botbw) - [moe] deepseek moe sp support by [haze188](https://api.github.com/users/Hz188) - [moe] remove ops by [hxwang](https://api.github.com/users/botbw) - [moe] full test for deepseek and mixtral (pp + sp to fix) by [hxwang](https://api.github.com/users/botbw) - [moe] finalize test (no pp) by [hxwang](https://api.github.com/users/botbw) - [moe] init moe plugin comm setting with sp by [hxwang](https://api.github.com/users/botbw) - [moe] clean legacy code by [hxwang](https://api.github.com/users/botbw) - [moe] test deepseek by [hxwang](https://api.github.com/users/botbw) - [moe] implement tp by [botbw](https://api.github.com/users/botbw) - [moe] add mixtral dp grad scaling when not all experts are activated by [botbw](https://api.github.com/users/botbw) - [moe] implement submesh initialization by [botbw](https://api.github.com/users/botbw) - [moe] implement transit between non moe tp and ep by [botbw](https://api.github.com/users/botbw) - [moe] fix plugin by [hxwang](https://api.github.com/users/botbw) ### Doc - [doc] add MoeHybridParallelPlugin docstring by [botbw](https://api.github.com/users/botbw) ### Deepseek - [deepseek] replace attn (a workaround for bug in transformers) by [hxwang](https://api.github.com/users/botbw) ### Bug - [bug] fix: somehow logger hangs the program by [botbw](https://api.github.com/users/botbw) ### Zero - [zero] solve hang by [botbw](https://api.github.com/users/botbw) - [zero] solve hang by [hxwang](https://api.github.com/users/botbw) **Full Changelog**: https://github.com/hpcaitech/ColossalAI/compare/v0.4.3...v0.4.2