<!-- Release notes generated using configuration in .github/release.yml at main --> ## What's Changed ### 🚀 Features * Add Gloo communication to turbomind by @irexyc in https://github.com/InternLM/lmdeploy/pull/3362 * [Feat] Support llm-compressor AWQ models in TurboMind by @43758726 in https://github.com/InternLM/lmdeploy/pull/4290 * Router replay for gpt oss by @RunningLeon in https://github.com/InternLM/lmdeploy/pull/4298 * Support llm-compressor symmetric quantized model inference in TurboMind by @43758726 in https://github.com/InternLM/lmdeploy/pull/4305 * Support Intern-S1-Pro by @CUHKSZzxy in https://github.com/InternLM/lmdeploy/pull/4318 ### 💥 Improvements * Configurable max CTAs and NVLS usage for CUDA IPC communicator by @lzhangzz in https://github.com/InternLM/lmdeploy/pull/4227 * Improve aborting all sessions by @lvhan028 in https://github.com/InternLM/lmdeploy/pull/4215 * Moe Reduce kernel by @grimoire in https://github.com/InternLM/lmdeploy/pull/4228 * Refactor attn by @grimoire in https://github.com/InternLM/lmdeploy/pull/4238 * Optimize exception raising and error process by @grimoire in https://github.com/InternLM/lmdeploy/pull/4236 * [AsyncEngine Refactor 1/N] define MultimodalProcessor to handle multimodal data processing by @lvhan028 in https://github.com/InternLM/lmdeploy/pull/4250 * [AsyncEngine Refactor 2/N] Remove deprecates from chat template by @lvhan028 in https://github.com/InternLM/lmdeploy/pull/4252 * Configurable uvicorn timeout by @CUHKSZzxy in https://github.com/InternLM/lmdeploy/pull/4255 * Adapt to dlsime v0.0.2 by @JimyMa in https://github.com/InternLM/lmdeploy/pull/4242 * [Fix] fix quant calibration dataset by @43758726 in https://github.com/InternLM/lmdeploy/pull/4256 * lmdeploy suppport parrllel embedding by @Tsundoku958 in https://github.com/InternLM/lmdeploy/pull/4192 * Refactor turbomind engine by @lzhangzz in https://github.com/InternLM/lmdeploy/pull/4223 * Refactor Engine & ModelAgent interact by @grimoire in https://github.com/InternLM/lmdeploy/pull/4265 * Support sleep and destroy deepep buffer by @RunningLeon in https://github.com/InternLM/lmdeploy/pull/4246 * add yarn truncate by @grimoire in https://github.com/InternLM/lmdeploy/pull/4301 * [AsyncEngine Refactor 3/N] Introduce Session and SessionManager by @lvhan028 in https://github.com/InternLM/lmdeploy/pull/4253 * Add warning about NCCL 2.27 memory leaks by @lzhangzz in https://github.com/InternLM/lmdeploy/pull/4313 ### 🐞 Bug fixes * Fix fope cos/sin coef device type by @CUHKSZzxy in https://github.com/InternLM/lmdeploy/pull/4240 * Fix include_stop_str_in_output with output_logits Exception by @windreamer in https://github.com/InternLM/lmdeploy/pull/4244 * fix logit softcapping is None by @grimoire in https://github.com/InternLM/lmdeploy/pull/4247 * Fix performance regression for prefix caching by @lzhangzz in https://github.com/InternLM/lmdeploy/pull/4270 * convert float16 weight to bfloat16 for FP8 models by @lvhan028 in https://github.com/InternLM/lmdeploy/pull/4276 * [ascend] fix dp multinode rank_table mapping by @tangzhiyi11 in https://github.com/InternLM/lmdeploy/pull/4268 * [Fix] move calibrate load dataset location by @43758726 in https://github.com/InternLM/lmdeploy/pull/4289 * fix ignore-eos by @grimoire in https://github.com/InternLM/lmdeploy/pull/4282 * fix MPEngine poll by @grimoire in https://github.com/InternLM/lmdeploy/pull/4287 * Fix prefix caching by @lzhangzz in https://github.com/InternLM/lmdeploy/pull/4292 * Fix gemma chat template by @lvhan028 in https://github.com/InternLM/lmdeploy/pull/4280 * Fix scheduler metrics by @lzhangzz in https://github.com/InternLM/lmdeploy/pull/4294 * Fix NVLS init for mixed DP+TP by @lzhangzz in https://github.com/InternLM/lmdeploy/pull/4296 * [side-effect] The tool message dump is incomplete by @lvhan028 in https://github.com/InternLM/lmdeploy/pull/4299 * Fix mla with spec tokens by @RunningLeon in https://github.com/InternLM/lmdeploy/pull/4302 * fix stop long context by @grimoire in https://github.com/InternLM/lmdeploy/pull/4309 * fix crash on client disconnect (Ctrl+C) by @lvhan028 in https://github.com/InternLM/lmdeploy/pull/4308 * Ensure the pipe benchmark uses kwargs when calling `pipe.stream_infer` by @lvhan028 in https://github.com/InternLM/lmdeploy/pull/4312 * fix get_ppl for long context by @lvhan028 in https://github.com/InternLM/lmdeploy/pull/4314 * fix sleep engine for dp=1 by @RunningLeon in https://github.com/InternLM/lmdeploy/pull/4315 ### 🌐 Other * [ci] fix fail testcase and add generate testcase in pr test by @zhulinJulia24 in https://github.com/InternLM/lmdeploy/pull/4231 * Pin nvshmem version by @CUHKSZzxy in https://github.com/InternLM/lmdeploy/pull/4257 * fix: Pin `timm` version to avoid failed tests by @windreamer in https://github.com/InternLM/lmdeploy/pull/4258 * docs: add generated openapi spec documentation by @windreamer in https://github.com/InternLM/lmdeploy/pull/4251 * fix: get rid of buggy timm-1.0.23 by @windreamer in https://github.com/InternLM/lmdeploy/pull/4260 * [ascend] fix paged prefill by @tangzhiyi11 in https://github.com/InternLM/lmdeploy/pull/4254 * Fix ascend/maca/camb runtime_requirements by @jinminxi104 in https://github.com/InternLM/lmdeploy/pull/4262 * docs: refine the documents by @windreamer in https://github.com/InternLM/lmdeploy/pull/4259 * docs: add cli docs by @windreamer in https://github.com/InternLM/lmdeploy/pull/4264 * Drop support for Python 3.9 as it has reached end-of-life by @lvhan028 in https://github.com/InternLM/lmdeploy/pull/4281 * bump version to v0.12.0 by @lvhan028 in https://github.com/InternLM/lmdeploy/pull/4300 ## New Contributors * @43758726 made their first contribution in https://github.com/InternLM/lmdeploy/pull/4256 **Full Changelog**: https://github.com/InternLM/lmdeploy/compare/v0.11.1...v0.12.0

<!-- Release notes generated using configuration in .github/release.yml at main --> ## What's Changed ### 🚀 Features * [ascend] support dptp by @tangzhiyi11 in https://github.com/InternLM/lmdeploy/pull/4218 * Support Deepseek v32 by @grimoire in https://github.com/InternLM/lmdeploy/pull/4026 ### 💥 Improvements * Improve metrics by @CUHKSZzxy in https://github.com/InternLM/lmdeploy/pull/4178 * reserve blocks for dummy inputs by @grimoire in https://github.com/InternLM/lmdeploy/pull/4157 * Add vision id for Qwen3-VL by @CUHKSZzxy in https://github.com/InternLM/lmdeploy/pull/4183 * [Enhance]: Return routed experts when request canceled by @RunningLeon in https://github.com/InternLM/lmdeploy/pull/4197 * Add mm processor args for Qwen3-VL by @CUHKSZzxy in https://github.com/InternLM/lmdeploy/pull/4196 * support chat_template_kwargs in v1/chat/completions by @lvhan028 in https://github.com/InternLM/lmdeploy/pull/4201 * Refactor scheduler and engine.py by @grimoire in https://github.com/InternLM/lmdeploy/pull/4163 * update dp timeout by @grimoire in https://github.com/InternLM/lmdeploy/pull/4204 * Improve Qwen3-VL by @CUHKSZzxy in https://github.com/InternLM/lmdeploy/pull/4207 ### 🐞 Bug fixes * [Fix]: Split routed experts with query lens by @RunningLeon in https://github.com/InternLM/lmdeploy/pull/4180 * [Maca] fix ray and memory sync by @wanfengcxz in https://github.com/InternLM/lmdeploy/pull/4164 * Build block trie in prefill and add hit rate by @RunningLeon in https://github.com/InternLM/lmdeploy/pull/4184 * fix fope by @CUHKSZzxy in https://github.com/InternLM/lmdeploy/pull/4191 * fix hf modules read/write conflicts by multi processors by @lvhan028 in https://github.com/InternLM/lmdeploy/pull/4188 * Some Minor fix by @windreamer in https://github.com/InternLM/lmdeploy/pull/4185 * fix insecure deserialization when calling torch.load() by @lvhan028 in https://github.com/InternLM/lmdeploy/pull/4202 * Fix processor args by @CUHKSZzxy in https://github.com/InternLM/lmdeploy/pull/4200 * remove get_model_config to avoid pickle hf_config error in rpc calling by @lvhan028 in https://github.com/InternLM/lmdeploy/pull/4217 * Fix quant scale-fmt by @grimoire in https://github.com/InternLM/lmdeploy/pull/4212 * Fix requests of mix return_logprobs by @RunningLeon in https://github.com/InternLM/lmdeploy/pull/4222 * fix fillkv quant8 by @grimoire in https://github.com/InternLM/lmdeploy/pull/4229 * fix scale-fmt by @grimoire in https://github.com/InternLM/lmdeploy/pull/4230 ### 📚 Documentations * [Docs]: Add guide for VLMEvalKit by @CUHKSZzxy in https://github.com/InternLM/lmdeploy/pull/4156 ### 🌐 Other * Add FA3 by @CUHKSZzxy in https://github.com/InternLM/lmdeploy/pull/4166 * Add distributed test cases by @littlegy in https://github.com/InternLM/lmdeploy/pull/4161 * Add generate test by @littlegy in https://github.com/InternLM/lmdeploy/pull/4181 * [ci] add mllm eval by @zhulinJulia24 in https://github.com/InternLM/lmdeploy/pull/4194 * [ascend] refactor code by @yao-fengchen in https://github.com/InternLM/lmdeploy/pull/4176 * install serve.txt when building the docker image by @lvhan028 in https://github.com/InternLM/lmdeploy/pull/4219 * bump version to v0.11.1 by @lvhan028 in https://github.com/InternLM/lmdeploy/pull/4221 **Full Changelog**: https://github.com/InternLM/lmdeploy/compare/v0.11.0...v0.11.1

<!-- Release notes generated using configuration in .github/release.yml at main --> ## What's Changed ### 🚀 Features * Add ROCm support: installation guide and FlashAttention compatibility for AMD GPUs by @Vivicai1005 in https://github.com/InternLM/lmdeploy/pull/3925 * support gpt-oss basic output by @irexyc in https://github.com/InternLM/lmdeploy/pull/3956 * Add FP8*(B)F16 GEMM by @lzhangzz in https://github.com/InternLM/lmdeploy/pull/3960 * Support GLM-4.5 by @CUHKSZzxy in https://github.com/InternLM/lmdeploy/pull/3863 * [Refactor]: Remove tokenizer when building engine by @RunningLeon in https://github.com/InternLM/lmdeploy/pull/3978 * Support InternVL3.5-Flash by @CUHKSZzxy in https://github.com/InternLM/lmdeploy/pull/3952 * support gpt-oss function/reasoning in /v1/chat/completions by @irexyc in https://github.com/InternLM/lmdeploy/pull/3962 * support returning stop_str in output by @lvhan028 in https://github.com/InternLM/lmdeploy/pull/3984 * Support SDAR by @grimoire in https://github.com/InternLM/lmdeploy/pull/3922 ### 💥 Improvements * specify installation on GeForce RTX 50 series by @lvhan028 in https://github.com/InternLM/lmdeploy/pull/3947 * cherry pick PR-3708 to return token_id by @lvhan028 in https://github.com/InternLM/lmdeploy/pull/3976 * Optimize AsyncEngine generation method by @shell-nlp in https://github.com/InternLM/lmdeploy/pull/3982 * Use blocking sync when TP engine is idling by @lzhangzz in https://github.com/InternLM/lmdeploy/pull/3974 * add openai_harmony to requirements by @irexyc in https://github.com/InternLM/lmdeploy/pull/4006 ### 🐞 Bug fixes * fix bugs with triton3.4.0 by @grimoire in https://github.com/InternLM/lmdeploy/pull/3946 * fix longrope by @grimoire in https://github.com/InternLM/lmdeploy/pull/3968 * Fix tm rl usage in xtuner by @irexyc in https://github.com/InternLM/lmdeploy/pull/3912 * Disable prefix caching when serving a VLM model by @lvhan028 in https://github.com/InternLM/lmdeploy/pull/3990 * remove NCCL_LAUNCH_MODE by @irexyc in https://github.com/InternLM/lmdeploy/pull/3994 * return the last token's logprobs, logits and last_hidden_states if include_stop_str_in_output is requested by @lvhan028 in https://github.com/InternLM/lmdeploy/pull/4000 * [Fix] device args in chat cli when using pytorch engine by @CyCle1024 in https://github.com/InternLM/lmdeploy/pull/3999 * fix internvl by @CUHKSZzxy in https://github.com/InternLM/lmdeploy/pull/3997 * fix not-returned iterator in SequenceManager::Erase by @irexyc in https://github.com/InternLM/lmdeploy/pull/4001 * fix cudagraph without warmup by @grimoire in https://github.com/InternLM/lmdeploy/pull/4005 * fix internvl flash long context acc by @CUHKSZzxy in https://github.com/InternLM/lmdeploy/pull/4003 ### 🌐 Other * [ci] update daily testcase by @zhulinJulia24 in https://github.com/InternLM/lmdeploy/pull/3944 * [maca] change kv layout from pagedattn to flashattn by @yuchiwang in https://github.com/InternLM/lmdeploy/pull/3958 * remove cudnn by @irexyc in https://github.com/InternLM/lmdeploy/pull/3969 * build(pypi): add cuda 12.8 support for wheels by @windreamer in https://github.com/InternLM/lmdeploy/pull/3948 * [CI] add ascend test by @littlegy in https://github.com/InternLM/lmdeploy/pull/3959 * update serve requirement by @RunningLeon in https://github.com/InternLM/lmdeploy/pull/3986 * [ci] add h800 function test workflow by @zhulinJulia24 in https://github.com/InternLM/lmdeploy/pull/3985 * bump version to v0.10.1 by @lvhan028 in https://github.com/InternLM/lmdeploy/pull/3989 ## New Contributors * @Vivicai1005 made their first contribution in https://github.com/InternLM/lmdeploy/pull/3925 * @shell-nlp made their first contribution in https://github.com/InternLM/lmdeploy/pull/3982 * @littlegy made their first contribution in https://github.com/InternLM/lmdeploy/pull/3959 **Full Changelog**: https://github.com/InternLM/lmdeploy/compare/v0.10.0...v0.10.1

<!-- Release notes generated using configuration in .github/release.yml at main --> ## What's Changed ### 🚀 Features * support offloading weights & kv_cache for turbomind by @irexyc in https://github.com/InternLM/lmdeploy/pull/3798 * Add PPU backend support by @guozixu2001 in https://github.com/InternLM/lmdeploy/pull/3807 * Add turbomind metrics by @lvhan028 in https://github.com/InternLM/lmdeploy/pull/3811 * PytorchEngine support gpt-oss bf16 by @grimoire in https://github.com/InternLM/lmdeploy/pull/3820 * support sleep/wakeup for pt engine by @irexyc in https://github.com/InternLM/lmdeploy/pull/3687 * [ascend] run intern-s1 on A3 by @yao-fengchen in https://github.com/InternLM/lmdeploy/pull/3831 * Initial gpt-oss support for turbomind by @lzhangzz in https://github.com/InternLM/lmdeploy/pull/3839 * Support GLM-4-0414 and GLM-4.1V by @CUHKSZzxy in https://github.com/InternLM/lmdeploy/pull/3846 * support internvl3.5 by @lvhan028 in https://github.com/InternLM/lmdeploy/pull/3886 * Update turbomind communication library by @lzhangzz in https://github.com/InternLM/lmdeploy/pull/3736 * MXFP4 support for turbomind GEMM library by @lzhangzz in https://github.com/InternLM/lmdeploy/pull/3927 * Dispatch MXFP4 weight conversion for sm70 & sm75 by @lzhangzz in https://github.com/InternLM/lmdeploy/pull/3937 ### 💥 Improvements * fix: turbomind backend config in cli serve by @PeymanRM in https://github.com/InternLM/lmdeploy/pull/3784 * remove deprecated codes by @lvhan028 in https://github.com/InternLM/lmdeploy/pull/3759 * Refactor FP8 MoE GEMM by @lzhangzz in https://github.com/InternLM/lmdeploy/pull/3795 * Fix build rope params by @grimoire in https://github.com/InternLM/lmdeploy/pull/3760 * Optimize rmsnorm with head_dim=128 by @grimoire in https://github.com/InternLM/lmdeploy/pull/3814 * Simplify GEMM interface by @lzhangzz in https://github.com/InternLM/lmdeploy/pull/3818 * Optimize create_model_inputs and schedule_decoding by @grimoire in https://github.com/InternLM/lmdeploy/pull/3766 * add remote logs;optimize forward lock by @grimoire in https://github.com/InternLM/lmdeploy/pull/3737 * support deepgemm new api by @grimoire in https://github.com/InternLM/lmdeploy/pull/3827 * remove serving with gradio by @lvhan028 in https://github.com/InternLM/lmdeploy/pull/3829 * Deprecate interactive mode from api_server by @lvhan028 in https://github.com/InternLM/lmdeploy/pull/3830 * build(docker): Try to optimize docker by @windreamer in https://github.com/InternLM/lmdeploy/pull/3779 * Make a common chat.py to replace each engine's by @lvhan028 in https://github.com/InternLM/lmdeploy/pull/3836 * Ray mp engine backend by @grimoire in https://github.com/InternLM/lmdeploy/pull/3790 * [Feat] support using external ray pg with bundles by @CyCle1024 in https://github.com/InternLM/lmdeploy/pull/3850 * Remove unused code in PT Engine by @grimoire in https://github.com/InternLM/lmdeploy/pull/3858 * support logprobs by @grimoire in https://github.com/InternLM/lmdeploy/pull/3852 * optimize prefill preprocess by @grimoire in https://github.com/InternLM/lmdeploy/pull/3869 * fix flash-attn bc by @grimoire in https://github.com/InternLM/lmdeploy/pull/3873 * Graph warmup by @grimoire in https://github.com/InternLM/lmdeploy/pull/3851 * Improve turbomind's prefix cache by @lvhan028 in https://github.com/InternLM/lmdeploy/pull/3835 * Support OpenAI compatible parameter max_completion_tokens by @Huarong in https://github.com/InternLM/lmdeploy/pull/3876 * [ascend] add env to set rt visable by ray and disable warmup by @tangzhiyi11 in https://github.com/InternLM/lmdeploy/pull/3894 * support cache_max_entry_count >= 1 for Turbomind backend by @lh9171338 in https://github.com/InternLM/lmdeploy/pull/3913 * adjust default values by @lvhan028 in https://github.com/InternLM/lmdeploy/pull/3921 * [refactor][chat_template][1/N] adopt tokenizer's apply_chat_template by @lvhan028 in https://github.com/InternLM/lmdeploy/pull/3845 * use FA 2.8.3 which is compatible with torch 2.8.0 by @lvhan028 in https://github.com/InternLM/lmdeploy/pull/3936 * refactor ascend Dockerfile by @yao-fengchen in https://github.com/InternLM/lmdeploy/pull/3926 ### 🐞 Bug fixes * fix gemma3 by @grimoire in https://github.com/InternLM/lmdeploy/pull/3772 * fix head_dim=None by @grimoire in https://github.com/InternLM/lmdeploy/pull/3793 * fix user-specified max_session_len by @grimoire in https://github.com/InternLM/lmdeploy/pull/3785 * remove 'lmdeploy convert' from CLI by @lvhan028 in https://github.com/InternLM/lmdeploy/pull/3813 * Fix EP with large batch size by @grimoire in https://github.com/InternLM/lmdeploy/pull/3808 * fix internvl disable_vision_encoder by @grimoire in https://github.com/InternLM/lmdeploy/pull/3800 * Align response behavior across both engines by @lvhan028 in https://github.com/InternLM/lmdeploy/pull/3821 * fix: set text_config.tie_word_embedding = False in qwen2vl by @zenosai in https://github.com/InternLM/lmdeploy/pull/3824 * Fix v1 comp protocol by @CUHKSZzxy in https://github.com/InternLM/lmdeploy/pull/3828 * [dlinfer] fix get_backend err by @yao-fengchen in https://github.com/InternLM/lmdeploy/pull/3847 * Update internvl.py to fix InternLM/lmdeploy#3528 by @zodiacg in https://github.com/InternLM/lmdeploy/pull/3837 * fix partial rotary factor by @CUHKSZzxy in https://github.com/InternLM/lmdeploy/pull/3861 * fix: duplicated token usage in /chat/completions stream mode by @Huarong in https://github.com/InternLM/lmdeploy/pull/3859 * fix chatting with VLM model via CLI by @lvhan028 in https://github.com/InternLM/lmdeploy/pull/3862 * fix inference on windows platform by @irexyc in https://github.com/InternLM/lmdeploy/pull/3865 * fix prebuild on cuda12.8 by @lvhan028 in https://github.com/InternLM/lmdeploy/pull/3857 * Fix uninitialized members in cuBLAS wrapper by @lzhangzz in https://github.com/InternLM/lmdeploy/pull/3874 * fix flashmla build for cuda12.4 by @CUHKSZzxy in https://github.com/InternLM/lmdeploy/pull/3872 * [Fix] ray mp engine on ascend platform by @CyCle1024 in https://github.com/InternLM/lmdeploy/pull/3877 * fix bug: leaves empty by @Tsundoku958 in https://github.com/InternLM/lmdeploy/pull/3868 * Fix side effect brought by gpt-oss support by @lvhan028 in https://github.com/InternLM/lmdeploy/pull/3880 * fix pytorch metrics in mp engine by @CUHKSZzxy in https://github.com/InternLM/lmdeploy/pull/3882 * Fix stream assert error when wakeup 30+ times by @CyCle1024 in https://github.com/InternLM/lmdeploy/pull/3883 * fix batched prefill by @grimoire in https://github.com/InternLM/lmdeploy/pull/3887 * fix side effect brought by #3821 by @lvhan028 in https://github.com/InternLM/lmdeploy/pull/3888 * check_env in multiprocess by @grimoire in https://github.com/InternLM/lmdeploy/pull/3879 * fix cli serve --help by @RunningLeon in https://github.com/InternLM/lmdeploy/pull/3895 * 1. [PD Disaggregation] Some Bug Fix (adapte p2p_initialize, metrics, uniexecutor with pd disagg) by @JimyMa in https://github.com/InternLM/lmdeploy/pull/3893 * Resolve a crash in the `sleep` endpoint by casting the `level` parameter from string to int by @irexyc in https://github.com/InternLM/lmdeploy/pull/3897 * Fix nccl for docker cu11 by @RunningLeon in https://github.com/InternLM/lmdeploy/pull/3896 * disable check_env in multiprocess on dlinfer devices by @tangzhiyi11 in https://github.com/InternLM/lmdeploy/pull/3914 * [dlinfer] fix nn layout typo and scale t by @yuchiwang in https://github.com/InternLM/lmdeploy/pull/3915 * fix chat and warmup of lora adapter by @grimoire in https://github.com/InternLM/lmdeploy/pull/3911 * build(acsend): try to fix acsend CI docker build by @windreamer in https://github.com/InternLM/lmdeploy/pull/3906 * fix internvl3 hf by @CUHKSZzxy in https://github.com/InternLM/lmdeploy/pull/3932 * build(docker): fix ascend tag name by @windreamer in https://github.com/InternLM/lmdeploy/pull/3939 * put eot_token to stop_words by @lvhan028 in https://github.com/InternLM/lmdeploy/pull/3941 ### 📚 Documentations * update proxy docs by @CUHKSZzxy in https://github.com/InternLM/lmdeploy/pull/3796 * add missing docs by @CUHKSZzxy in https://github.com/InternLM/lmdeploy/pull/3871 * fix docs by @CUHKSZzxy in https://github.com/InternLM/lmdeploy/pull/3885 * update news and citation by @lvhan028 in https://github.com/InternLM/lmdeploy/pull/3889 ### 🌐 Other * add prometheus client by @CUHKSZzxy in https://github.com/InternLM/lmdeploy/pull/3792 * fix: add dummy_prefill guard for PD connection operations by @FirwoodLin in https://github.com/InternLM/lmdeploy/pull/3803 * minor fix about the log level and logs by @lvhan028 in https://github.com/InternLM/lmdeploy/pull/3758 * assert PytorchEngineConfig block size by @Tsundoku958 in https://github.com/InternLM/lmdeploy/pull/3826 * [ci] change restful api into openai and add more testcase by @zhulinJulia24 in https://github.com/InternLM/lmdeploy/pull/3866 * remove ppu backend by @yao-fengchen in https://github.com/InternLM/lmdeploy/pull/3904 * [ci] remove flash attn installation in ete test workflow by @zhulinJulia24 in https://github.com/InternLM/lmdeploy/pull/3908 * dlinfer backend support ray by @yao-fengchen in https://github.com/InternLM/lmdeploy/pull/3903 * style(types): fix return type annotation for get_all_requests by @xiaoajie738 in https://github.com/InternLM/lmdeploy/pull/3919 * upgrade torch to 2.8.0 and triton 3.4.0 by @lvhan028 in https://github.com/InternLM/lmdeploy/pull/3930 * Dlinfer readme by @jinminxi104 in https://github.com/InternLM/lmdeploy/pull/3938 * bump version to v0.10.0 by @lvhan028 in https://github.com/InternLM/lmdeploy/pull/3933 ## New Contributors * @PeymanRM made their first contribution in https://github.com/InternLM/lmdeploy/pull/3784 * @FirwoodLin made their first contribution in https://github.com/InternLM/lmdeploy/pull/3803 * @zenosai made their first contribution in https://github.com/InternLM/lmdeploy/pull/3824 * @Tsundoku958 made their first contribution in https://github.com/InternLM/lmdeploy/pull/3826 * @guozixu2001 made their first contribution in https://github.com/InternLM/lmdeploy/pull/3807 * @zodiacg made their first contribution in https://github.com/InternLM/lmdeploy/pull/3837 * @Huarong made their first contribution in https://github.com/InternLM/lmdeploy/pull/3859 * @yuchiwang made their first contribution in https://github.com/InternLM/lmdeploy/pull/3915 * @lh9171338 made their first contribution in https://github.com/InternLM/lmdeploy/pull/3913 **Full Changelog**: https://github.com/InternLM/lmdeploy/compare/v0.9.2...v0.10.0

<!-- Release notes generated using configuration in .github/release.yml at dev-0.9.2post1 --> ## What's Changed ### 🌐 Other * Fix interns1 LLM mapping for turbomind engine by @lvhan028 in https://github.com/InternLM/lmdeploy/pull/3848 * bump version to v0.9.2.post1 by @lvhan028 in https://github.com/InternLM/lmdeploy/pull/3849 **Full Changelog**: https://github.com/InternLM/lmdeploy/compare/v0.9.2...v0.9.2.post1

<!-- Release notes generated using configuration in .github/release.yml at main --> ## What's Changed ### 🚀 Features * [Feature] metrics support by @CUHKSZzxy in https://github.com/InternLM/lmdeploy/pull/3534 * Relax FP8 TP requirement by @lzhangzz in https://github.com/InternLM/lmdeploy/pull/3697 * FA3 by @zhaochaoxing in https://github.com/InternLM/lmdeploy/pull/3623 * support qwen2/2.5-vl in turbomind by @irexyc in https://github.com/InternLM/lmdeploy/pull/3744 * feat: add pytorch_engine_qwen2_5vl_sm120 by @kolmogorov-quyet in https://github.com/InternLM/lmdeploy/pull/3750 * Internvl pt by @RunningLeon in https://github.com/InternLM/lmdeploy/pull/3765 * Improve internvl for turbomind engine by @lvhan028 in https://github.com/InternLM/lmdeploy/pull/3769 ### 💥 Improvements * Refactor linear by @grimoire in https://github.com/InternLM/lmdeploy/pull/3653 * remove python3.8 support and add python3.13 support by @lvhan028 in https://github.com/InternLM/lmdeploy/pull/3638 * refactor vl inputs split by @grimoire in https://github.com/InternLM/lmdeploy/pull/3699 * [Fix]: Replace mutable default with default_factory for scheduler_stats by @ConvolutedDog in https://github.com/InternLM/lmdeploy/pull/3730 * Fix the logic of calculating max_new_tokens and determining finish_reason by @lvhan028 in https://github.com/InternLM/lmdeploy/pull/3727 * Override HF config.json via CLI by @CUHKSZzxy in https://github.com/InternLM/lmdeploy/pull/3722 * feat(build): Integrate and build turbomind backend directly in setup.py by @windreamer in https://github.com/InternLM/lmdeploy/pull/3726 * Generate the benchmark output filename with given arguments by @lvhan028 in https://github.com/InternLM/lmdeploy/pull/3740 * Make loading llm without vlm as an option by @grimoire in https://github.com/InternLM/lmdeploy/pull/3745 ### 🐞 Bug fixes * add ray to ascend requirements by @sigma-plus in https://github.com/InternLM/lmdeploy/pull/3713 * fix accessing undefined attribute `seq_aux` of deepseek-r1-0528 by @lvhan028 in https://github.com/InternLM/lmdeploy/pull/3728 * [Fix]: Avoid quantize qk norm for qwen3 dense models by @taishan1994 in https://github.com/InternLM/lmdeploy/pull/3733 * fix py313 env creation failed when building lmdeploy-builder image by @lvhan028 in https://github.com/InternLM/lmdeploy/pull/3739 * [Fix]: kernel meta retrieval for SM7X does not work by @xiaoajie738 in https://github.com/InternLM/lmdeploy/pull/3746 * limit max_session_len by @grimoire in https://github.com/InternLM/lmdeploy/pull/3751 * fix internvl norm by @grimoire in https://github.com/InternLM/lmdeploy/pull/3756 * support qwen3 moe yarn and vlm hf_overrides by @grimoire in https://github.com/InternLM/lmdeploy/pull/3757 * [PD Disaggregation] fix double unshelf by @JimyMa in https://github.com/InternLM/lmdeploy/pull/3762 * fix(build): fix version parse regex to support post-release versions by @windreamer in https://github.com/InternLM/lmdeploy/pull/3764 * adapt transformers>=v4.52.0 to loading qwen2.5-vl with turbomind by @irexyc in https://github.com/InternLM/lmdeploy/pull/3771 * fix chat template with tool call by @RunningLeon in https://github.com/InternLM/lmdeploy/pull/3773 * fix vl nothink mode by @RunningLeon in https://github.com/InternLM/lmdeploy/pull/3776 ### 📚 Documentations * update reward model docs by @CUHKSZzxy in https://github.com/InternLM/lmdeploy/pull/3721 ### 🌐 Other * update twomicrobatch by @SHshenhao in https://github.com/InternLM/lmdeploy/pull/3651 * [CI]: Upgrade to py310 for ut by @RunningLeon in https://github.com/InternLM/lmdeploy/pull/3718 * [ci] update dailytest environment and scripts by @zhulinJulia24 in https://github.com/InternLM/lmdeploy/pull/3716 * Preliminary Blackwell (sm_120a, RTX 50 series) support by @lzhangzz in https://github.com/InternLM/lmdeploy/pull/3701 * [ci] add fp8 evaluation workflow by @zhulinJulia24 in https://github.com/InternLM/lmdeploy/pull/3729 * Add VRAM bandwidth utilization stat to attention test by @lzhangzz in https://github.com/InternLM/lmdeploy/pull/3731 * doc: fix dead links to MindX DL to recover CI. by @windreamer in https://github.com/InternLM/lmdeploy/pull/3741 * fix free cache in MPEngine branch by @JimyMa in https://github.com/InternLM/lmdeploy/pull/3670 * fix: make RelWithDebInfo default cmake build type by @windreamer in https://github.com/InternLM/lmdeploy/pull/3774 * bump version to v0.9.2 by @lvhan028 in https://github.com/InternLM/lmdeploy/pull/3770 ## New Contributors * @sigma-plus made their first contribution in https://github.com/InternLM/lmdeploy/pull/3713 * @ConvolutedDog made their first contribution in https://github.com/InternLM/lmdeploy/pull/3730 * @windreamer made their first contribution in https://github.com/InternLM/lmdeploy/pull/3726 * @taishan1994 made their first contribution in https://github.com/InternLM/lmdeploy/pull/3733 * @xiaoajie738 made their first contribution in https://github.com/InternLM/lmdeploy/pull/3746 * @kolmogorov-quyet made their first contribution in https://github.com/InternLM/lmdeploy/pull/3750 **Full Changelog**: https://github.com/InternLM/lmdeploy/compare/v0.9.1...v0.9.2

<!-- Release notes generated using configuration in .github/release.yml at main --> ## What's Changed ### 🚀 Features * LMDeploy Distserve by @JimyMa in https://github.com/InternLM/lmdeploy/pull/3304 * allow api server terminated through requests from clients by @RunningLeon in https://github.com/InternLM/lmdeploy/pull/3533 * support update params for pytorch backend from api server by @irexyc in https://github.com/InternLM/lmdeploy/pull/3535 * support eplb for Qwen3-MoE by @zhaochaoxing in https://github.com/InternLM/lmdeploy/pull/3582 * support update params for turbomind backend by @irexyc in https://github.com/InternLM/lmdeploy/pull/3566 * Quantize Qwen3 MoE bf16 model to fp8 model at runtime by @grimoire in https://github.com/InternLM/lmdeploy/pull/3631 * [Feat]: Support internvl3-8b-hf by @RunningLeon in https://github.com/InternLM/lmdeploy/pull/3633 * Add FP8 MoE for turbomind by @lzhangzz in https://github.com/InternLM/lmdeploy/pull/3601 ### 💥 Improvements * reduce ray memory usage by @grimoire in https://github.com/InternLM/lmdeploy/pull/3487 * use dlblas by @zhaochaoxing in https://github.com/InternLM/lmdeploy/pull/3489 * internlm3 dense fp8 by @CUHKSZzxy in https://github.com/InternLM/lmdeploy/pull/3527 * random pad input ids by @grimoire in https://github.com/InternLM/lmdeploy/pull/3530 * ray nsys profile support by @grimoire in https://github.com/InternLM/lmdeploy/pull/3448 * update blockedfp8 scale name by @CUHKSZzxy in https://github.com/InternLM/lmdeploy/pull/3532 * start engine loop on server startup event by @grimoire in https://github.com/InternLM/lmdeploy/pull/3523 * update two microbatch by @SHshenhao in https://github.com/InternLM/lmdeploy/pull/3540 * [ascend]set transdata dynamic shape true by @JackWeiw in https://github.com/InternLM/lmdeploy/pull/3531 * ray safe exit by @grimoire in https://github.com/InternLM/lmdeploy/pull/3545 * support update params with dp=1 for pytorch engine by @irexyc in https://github.com/InternLM/lmdeploy/pull/3562 * Skip dp dummy input forward by @grimoire in https://github.com/InternLM/lmdeploy/pull/3552 * Unclock mutual exclusivity of argument: `tool-call-parser` and `reasoning-parser` by @jingyibo123 in https://github.com/InternLM/lmdeploy/pull/3550 * perform torch.cuda.empty_cache() after conversion by @bltcn in https://github.com/InternLM/lmdeploy/pull/3570 * pipeline warmup by @irexyc in https://github.com/InternLM/lmdeploy/pull/3548 * Launch multiple api servers for dp > 1 by @RunningLeon in https://github.com/InternLM/lmdeploy/pull/3414 * support awq for Qwen2.5-VL by @RunningLeon in https://github.com/InternLM/lmdeploy/pull/3559 * support qwen3 /think & /no_think & enable_thinking parameter by @BUJIDAOVS in https://github.com/InternLM/lmdeploy/pull/3564 * Eplb by @zhaochaoxing in https://github.com/InternLM/lmdeploy/pull/3572 * Update benchmark by @lvhan028 in https://github.com/InternLM/lmdeploy/pull/3578 * block output when prefetch next forward inputs. by @grimoire in https://github.com/InternLM/lmdeploy/pull/3573 * support both eplb and microbatch simultaneously by @zhaochaoxing in https://github.com/InternLM/lmdeploy/pull/3591 * Add log_file and set loglevel in launch_servers by @RunningLeon in https://github.com/InternLM/lmdeploy/pull/3596 * 1. add migration flow control by @JimyMa in https://github.com/InternLM/lmdeploy/pull/3599 * sampling on the tokenizer's vocab by @grimoire in https://github.com/InternLM/lmdeploy/pull/3604 * update deepgemm version by @grimoire in https://github.com/InternLM/lmdeploy/pull/3606 * [Ascend] set default distrbuted backend as ray for ascend device by @JackWeiw in https://github.com/InternLM/lmdeploy/pull/3603 * Blocked fp8 tma by @grimoire in https://github.com/InternLM/lmdeploy/pull/3470 * [PDDisaggreagtion] Async migration by @JimyMa in https://github.com/InternLM/lmdeploy/pull/3610 * move dp loop to model agent by @grimoire in https://github.com/InternLM/lmdeploy/pull/3598 * update some logs of proxy_server and pt engine by @lvhan028 in https://github.com/InternLM/lmdeploy/pull/3621 * improve loading model performance by shuffling the weight files by @irexyc in https://github.com/InternLM/lmdeploy/pull/3625 * add benchmark scripts about pipeline api and inference engines according to the config file by @lvhan028 in https://github.com/InternLM/lmdeploy/pull/3622 ### 🐞 Bug fixes * [ascend] fix recompile on different rank by @jinminxi104 in https://github.com/InternLM/lmdeploy/pull/3513 * fix attention sm86 by @grimoire in https://github.com/InternLM/lmdeploy/pull/3519 * fix stopwords kv cache by @grimoire in https://github.com/InternLM/lmdeploy/pull/3494 * [bug fix] fix PD Disaggregation in DSV3 by @JimyMa in https://github.com/InternLM/lmdeploy/pull/3547 * fix proxy server heart beat by @irexyc in https://github.com/InternLM/lmdeploy/pull/3543 * fix dp>1 tp=1 ep=1 by @grimoire in https://github.com/InternLM/lmdeploy/pull/3555 * fix mixtral on new transformers by @grimoire in https://github.com/InternLM/lmdeploy/pull/3580 * [Fix]: reset step after eviction by @RunningLeon in https://github.com/InternLM/lmdeploy/pull/3589 * fix parsing dynamic rope param failed by @lvhan028 in https://github.com/InternLM/lmdeploy/pull/3575 * Fix batch infer for gemma3vl by @RunningLeon in https://github.com/InternLM/lmdeploy/pull/3592 * Fix symbol error when dlBLAS is not imported by @zhaochaoxing in https://github.com/InternLM/lmdeploy/pull/3597 * read distributed envs by @grimoire in https://github.com/InternLM/lmdeploy/pull/3600 * fix side-effect caused by PR 3590 by @lvhan028 in https://github.com/InternLM/lmdeploy/pull/3608 * fix bug in qwen2 by @LKJacky in https://github.com/InternLM/lmdeploy/pull/3614 * fix awq kernel by @grimoire in https://github.com/InternLM/lmdeploy/pull/3618 * fix flash mla interface by @grimoire in https://github.com/InternLM/lmdeploy/pull/3617 * add sampling_vocab_size by @irexyc in https://github.com/InternLM/lmdeploy/pull/3607 * fix for default quant by @grimoire in https://github.com/InternLM/lmdeploy/pull/3640 * Fix log file env in ray worker by @RunningLeon in https://github.com/InternLM/lmdeploy/pull/3624 * fix qwen3 chat template by @lvhan028 in https://github.com/InternLM/lmdeploy/pull/3641 * fix vlm runtime quant by @grimoire in https://github.com/InternLM/lmdeploy/pull/3644 * Fix 'Namespace' object has no attribute 'num_tokens_per_iter' when serving by gradio by @lvhan028 in https://github.com/InternLM/lmdeploy/pull/3647 * Synchronize weight processing by @lzhangzz in https://github.com/InternLM/lmdeploy/pull/3649 * Fix zero scale in fp8 quantization by @lzhangzz in https://github.com/InternLM/lmdeploy/pull/3652 ### 🌐 Other * update doc for ascend 300I Duo docker image by @jinminxi104 in https://github.com/InternLM/lmdeploy/pull/3526 * simulate EPLB for benchmark only by @lvhan028 in https://github.com/InternLM/lmdeploy/pull/3490 * [ci] add test workflow for 3090 machine by @zhulinJulia24 in https://github.com/InternLM/lmdeploy/pull/3561 * [ci] fix transformers version in prtest by @zhulinJulia24 in https://github.com/InternLM/lmdeploy/pull/3584 * [Misc] minor api_server and tm loader, and upgrade docformatter to resolve lint error by @lvhan028 in https://github.com/InternLM/lmdeploy/pull/3590 * [ci] add qwen3 models into testcase by @zhulinJulia24 in https://github.com/InternLM/lmdeploy/pull/3593 * update Dockerfile by @CUHKSZzxy in https://github.com/InternLM/lmdeploy/pull/3634 * check in lmdeploy-builder on cuda 12.4 and 12.8 platform by @lvhan028 in https://github.com/InternLM/lmdeploy/pull/3630 * fix blocked fp8 overflow by @grimoire in https://github.com/InternLM/lmdeploy/pull/3650 * Bump version to v0.9.0 by @lvhan028 in https://github.com/InternLM/lmdeploy/pull/3609 ## New Contributors * @JimyMa made their first contribution in https://github.com/InternLM/lmdeploy/pull/3304 * @jingyibo123 made their first contribution in https://github.com/InternLM/lmdeploy/pull/3550 * @bltcn made their first contribution in https://github.com/InternLM/lmdeploy/pull/3570 * @BUJIDAOVS made their first contribution in https://github.com/InternLM/lmdeploy/pull/3564 * @LKJacky made their first contribution in https://github.com/InternLM/lmdeploy/pull/3614 **Full Changelog**: https://github.com/InternLM/lmdeploy/compare/v0.8.0...v0.9.0

<!-- Release notes generated using configuration in .github/release.yml at main --> ## What's Changed ### 🚀 Features * Torch dp support by @grimoire in https://github.com/InternLM/lmdeploy/pull/3207 * Add deep gemm with tma pre allocated by @AllentDan in https://github.com/InternLM/lmdeploy/pull/3287 * Add mixed DP + TP by @lzhangzz in https://github.com/InternLM/lmdeploy/pull/3229 * Add Qwen3 and Qwen3MoE by @lzhangzz in https://github.com/InternLM/lmdeploy/pull/3305 * [ascend] support multi nodes on ascend device by @tangzhiyi11 in https://github.com/InternLM/lmdeploy/pull/3260 * [Feature] support qwen3 and qwen3-moe for pytorch engine by @CUHKSZzxy in https://github.com/InternLM/lmdeploy/pull/3315 * [ascend]support deepseekv2 by @yao-fengchen in https://github.com/InternLM/lmdeploy/pull/3206 * add deepep by @zhaochaoxing in https://github.com/InternLM/lmdeploy/pull/3313 * support ascend w8a8 graph_mode by @yao-fengchen in https://github.com/InternLM/lmdeploy/pull/3267 * support all2all ep by @zhaochaoxing in https://github.com/InternLM/lmdeploy/pull/3370 * optimize ep in decoding stage by @zhaochaoxing in https://github.com/InternLM/lmdeploy/pull/3383 * Warmup deepgemm by @grimoire in https://github.com/InternLM/lmdeploy/pull/3387 * support Llama4 by @grimoire in https://github.com/InternLM/lmdeploy/pull/3408 * add twomicrobatch support by @SHshenhao in https://github.com/InternLM/lmdeploy/pull/3381 * Support phi4 mini by @RunningLeon in https://github.com/InternLM/lmdeploy/pull/3467 * [Dlinfer][Ascend] support 310P by @JackWeiw in https://github.com/InternLM/lmdeploy/pull/3484 * support qwen3 fp8 by @CUHKSZzxy in https://github.com/InternLM/lmdeploy/pull/3505 ### 💥 Improvements * Add spaces_between_special_tokens to /v1/interactive and make compatible with empty text by @AllentDan in https://github.com/InternLM/lmdeploy/pull/3283 * add env var to control timeout by @CUHKSZzxy in https://github.com/InternLM/lmdeploy/pull/3291 * refactor attn param by @irexyc in https://github.com/InternLM/lmdeploy/pull/3164 * Verbose log by @grimoire in https://github.com/InternLM/lmdeploy/pull/3329 * optimize mla, remove load `v` by @grimoire in https://github.com/InternLM/lmdeploy/pull/3334 * support dp decoding with cudagraph by @grimoire in https://github.com/InternLM/lmdeploy/pull/3311 * optimize quant-fp8 kernel by @grimoire in https://github.com/InternLM/lmdeploy/pull/3345 * refactor dlinfer rope by @yao-fengchen in https://github.com/InternLM/lmdeploy/pull/3326 * enable qwenvl2.5 graph mode on ascend by @jinminxi104 in https://github.com/InternLM/lmdeploy/pull/3367 * Add AIOHTTP_TIMEOUT env var for proxy server by @AllentDan in https://github.com/InternLM/lmdeploy/pull/3355 * disable sync batch on dp eager mode by @grimoire in https://github.com/InternLM/lmdeploy/pull/3382 * fix for deepgemm update by @grimoire in https://github.com/InternLM/lmdeploy/pull/3380 * Add string before hash tokens in blocktrie by @RunningLeon in https://github.com/InternLM/lmdeploy/pull/3386 * optimize moe get sorted idx by @grimoire in https://github.com/InternLM/lmdeploy/pull/3356 * use half/bf16 lm_head output by @irexyc in https://github.com/InternLM/lmdeploy/pull/3213 * remove ep eager check by @grimoire in https://github.com/InternLM/lmdeploy/pull/3392 * Optimize ascend moe by @yao-fengchen in https://github.com/InternLM/lmdeploy/pull/3364 * optimize fp8 moe kernel by @grimoire in https://github.com/InternLM/lmdeploy/pull/3419 * ray async forward execute by @grimoire in https://github.com/InternLM/lmdeploy/pull/3443 * map internvl3 chat template to builtin chat template internvl2_5 by @lvhan028 in https://github.com/InternLM/lmdeploy/pull/3450 * Refactor turbomind (low-level abstractions) by @lzhangzz in https://github.com/InternLM/lmdeploy/pull/3423 * remove barely used code to improve maintenance by @lvhan028 in https://github.com/InternLM/lmdeploy/pull/3462 * optimize sm80 long context by @grimoire in https://github.com/InternLM/lmdeploy/pull/3465 * move partial_json_parser from ’serve.txt‘ to ‘runtime.txt‘ by @lvhan028 in https://github.com/InternLM/lmdeploy/pull/3493 * support qwen3-dense models awq quantization by @lvhan028 in https://github.com/InternLM/lmdeploy/pull/3503 * Optimize MoE gate for Qwen3 by @lzhangzz in https://github.com/InternLM/lmdeploy/pull/3500 * Pass num_tokens_per_iter and max_prefill_iters params through in `lmdeploy serve api_server` by @josephrocca in https://github.com/InternLM/lmdeploy/pull/3504 * [Dlinfer][Ascend] Optimize performance of 310P device by @JackWeiw in https://github.com/InternLM/lmdeploy/pull/3486 * optimize longcontext decoding by @grimoire in https://github.com/InternLM/lmdeploy/pull/3510 * Support min_p in openai completions_v1 by @josephrocca in https://github.com/InternLM/lmdeploy/pull/3506 ### 🐞 Bug fixes * fix activation grid oversize by @grimoire in https://github.com/InternLM/lmdeploy/pull/3282 * Set ensure_ascii=False for tool calling by @AllentDan in https://github.com/InternLM/lmdeploy/pull/3295 * fix sliding window multi chat by @grimoire in https://github.com/InternLM/lmdeploy/pull/3302 * add `v` check by @grimoire in https://github.com/InternLM/lmdeploy/pull/3307 * Fix Qwen3MoE config parsing by @lzhangzz in https://github.com/InternLM/lmdeploy/pull/3336 * Fix finish reasons by @AllentDan in https://github.com/InternLM/lmdeploy/pull/3338 * remove think_end_token_id in streaming content by @AllentDan in https://github.com/InternLM/lmdeploy/pull/3327 * Fix the finish_reason by @AllentDan in https://github.com/InternLM/lmdeploy/pull/3350 * set cmake policy minimum version as 3.5 by @lvhan028 in https://github.com/InternLM/lmdeploy/pull/3376 * fix dp cudagraph by @grimoire in https://github.com/InternLM/lmdeploy/pull/3372 * fix flashmla eagermode by @grimoire in https://github.com/InternLM/lmdeploy/pull/3375 * close engine after each benchmark-generation iter by @grimoire in https://github.com/InternLM/lmdeploy/pull/3269 * [Fix] fix `image_token_id` error of qwen2-vl and deepseek by @ao-zz in https://github.com/InternLM/lmdeploy/pull/3358 * fix stopping criteria by @grimoire in https://github.com/InternLM/lmdeploy/pull/3384 * support List[dict] prompt input without do_preprocess by @irexyc in https://github.com/InternLM/lmdeploy/pull/3385 * add rayexecutor release timeout by @grimoire in https://github.com/InternLM/lmdeploy/pull/3403 * fix tensor dispatch in dynamo by @wanfengcxz in https://github.com/InternLM/lmdeploy/pull/3417 * fix linting error by upgrade to ubuntu-latest by @lvhan028 in https://github.com/InternLM/lmdeploy/pull/3442 * fix awq tp for pytorch engine by @RunningLeon in https://github.com/InternLM/lmdeploy/pull/3435 * fix mllm testcase fail by @caikun-pjlab in https://github.com/InternLM/lmdeploy/pull/3458 * remove paged attention autotune by @grimoire in https://github.com/InternLM/lmdeploy/pull/3452 * Remove empty prompts in benchmark scripts by @lvhan028 in https://github.com/InternLM/lmdeploy/pull/3460 * failed to end session properly by @lvhan028 in https://github.com/InternLM/lmdeploy/pull/3471 * fix qwen2.5-vl chat template by @CUHKSZzxy in https://github.com/InternLM/lmdeploy/pull/3475 * Align forward arguments of deepgemm blockedf8 by @RunningLeon in https://github.com/InternLM/lmdeploy/pull/3474 * fix turbomind lib missing to link nccl by exporting nccl path by @lvhan028 in https://github.com/InternLM/lmdeploy/pull/3479 * fix dsvl2 no attr config error by @CUHKSZzxy in https://github.com/InternLM/lmdeploy/pull/3477 * fix flash attention crash on triton3.1.0 by @grimoire in https://github.com/InternLM/lmdeploy/pull/3478 * Fix disorder of ray execution by @RunningLeon in https://github.com/InternLM/lmdeploy/pull/3481 * update dockerfile by @CUHKSZzxy in https://github.com/InternLM/lmdeploy/pull/3482 * fix output logprobs by @irexyc in https://github.com/InternLM/lmdeploy/pull/3488 * Fix Qwen2MoE shared expert gate by @lzhangzz in https://github.com/InternLM/lmdeploy/pull/3491 * fix replicate kv for qwen3-moe by @grimoire in https://github.com/InternLM/lmdeploy/pull/3499 * fix sampling if data overflow after temperature penalty by @irexyc in https://github.com/InternLM/lmdeploy/pull/3508 ### 📚 Documentations * update qwen2.5-vl-32b docs by @CUHKSZzxy in https://github.com/InternLM/lmdeploy/pull/3446 ### 🌐 Other * bump version to v0.7.2.post1 by @lvhan028 in https://github.com/InternLM/lmdeploy/pull/3298 * [ci] add think function testcase by @zhulinJulia24 in https://github.com/InternLM/lmdeploy/pull/3299 * merge dev into main by @lvhan028 in https://github.com/InternLM/lmdeploy/pull/3348 * [ci] add vl models into pipeline interface testcase by @zhulinJulia24 in https://github.com/InternLM/lmdeploy/pull/3374 * merge dev to main branch by @lvhan028 in https://github.com/InternLM/lmdeploy/pull/3378 * opt experts memory and permute by @zhaochaoxing in https://github.com/InternLM/lmdeploy/pull/3390 * Revert "opt experts memory and permute" by @zhaochaoxing in https://github.com/InternLM/lmdeploy/pull/3406 * merge dev to main by @lvhan028 in https://github.com/InternLM/lmdeploy/pull/3400 * add Hopper GPU dockerfile by @CUHKSZzxy in https://github.com/InternLM/lmdeploy/pull/3415 * optimize internvit by @caikun-pjlab in https://github.com/InternLM/lmdeploy/pull/3433 * fix stop/bad words by @irexyc in https://github.com/InternLM/lmdeploy/pull/3492 * [ci] testcase bugfix and add more models into testcase by @zhulinJulia24 in https://github.com/InternLM/lmdeploy/pull/3463 * bump version to v0.8.0 by @lvhan028 in https://github.com/InternLM/lmdeploy/pull/3432 ## New Contributors * @zhaochaoxing made their first contribution in https://github.com/InternLM/lmdeploy/pull/3313 * @ao-zz made their first contribution in https://github.com/InternLM/lmdeploy/pull/3358 * @wanfengcxz made their first contribution in https://github.com/InternLM/lmdeploy/pull/3417 * @SHshenhao made their first contribution in https://github.com/InternLM/lmdeploy/pull/3381 * @josephrocca made their first contribution in https://github.com/InternLM/lmdeploy/pull/3504 **Full Changelog**: https://github.com/InternLM/lmdeploy/compare/v0.7.2...v0.8.0