# Highlights - **DeepSeek V4 support**: Full inference path for DeepSeek-V4 (#23882), including: Day-0 Features: #23882 - Parallelism: Tensor Parallelism/Expert Parallelism/Context Parallelism/Data Parallel Attention - Hardware: Nvidia B300/B200/H200/H100/GB200/GB300, AMD MI35X - Prefill-Decode Disaggregation - HiSparse for offloading inactive KV cache to CPU memory - Reasoning parser and Tool Call Parser - DeepGemm and FlashMLA kernels for DeepSeek V4, including MegaMoE Post-Day-0 additions: - HiCache for DeepSeek V4 under unified Radix Tree [UnifiedTree]: #24691 - W4A4 MegaMoE kernels — faster speed with negligible accuracy drop: #25052 - Marlin/FlashInfer W4A8 MoE kernels on Hopper: #24816 #24986 - Faster V2 fused compression kernels: #24890 - TP16 support on H100/H20: #24949 - Fused SiLU+clamp+FP8 quant kernel: #24897 - Optimized MHC + DeepGemm pipeline (fused norm, fused hc_head): #24775 - Non-standard chat template support for DSv4: #23915 - Multi-detokenizer support: #24944 - Pipeline Parallelism + PD support for DeepSeek-V4: #24700 - A unified docker tag `lmsysorg/sglang:v0.5.12` for all Nvidia GPUs See the [LMSYS blog](https://www.lmsys.org/blog/2026-04-25-deepseek-v4/) and the [DeepSeek-V4 cookbook](https://docs.sglang.io/cookbook/autoregressive/DeepSeek/DeepSeek-V4) for more details. - **TokenSpeed MLA attention backend (Blackwell, FP8 KV cache)**: New MLA prefill/decode kernels integrated as an attention backend on SM100, with FP8 KV cache support for low-latency MLA serving: #24925 - **DSv3.2 / GLM-5 FP4 low-latency perf**: PDL enabled across DSv3.2 / GLM-5 kernels, `torch.mm` for the DeepSeek V3.2 indexer GEMM, and a reland of the Cute-DSL FP4 dense GEMM — materially trimming low-latency overheads on FP4 paths: #23965, #23856, #23590, #25311 - **New Model Support**: DeepSeek V4 #23882, Intern-S2-Preview #24875, MiniCPM-V 4.6 #24855, Laguna-XS.2 #24204, Ring-2.6-1T #25360, and Gemma 4 MTP #24436 — with cookbook recipes for tuned deployment commands. See [docs.sglang.io/cookbook](https://docs.sglang.io/cookbook) - **HiCache + UnifiedRadixTree**: HiCache framework support for UnifiedRadixTree (with SWA), HiCache for DeepSeek V4, SSD offload through Mooncake store, and stability fixes across cascade eviction, tombstone replay, and partial-match paths: #23316, #23391, #24691, #24277, #24943, #24972, #25068, #25277 - **Speculative Decoding V2 maturation**: Adaptive Spec V2, EAGLE-3 SWA + newer drafters, Kimi K2.5 EAGLE-3 MLA, Gemma 3/4 + EAGLE-3, and an extensive naming / shape-handling refactor across draft-extend paths: #23336, #24663, #24664, #24826, #23976, #24859 - **CUDA 13 DeepEP migration**: Gateway DeepEP source swapped from a community fork to `deepseek-ai/DeepEP@hybrid-ep` so DeepEP builds and runs cleanly on the CUDA 13 default; FlashInfer pinned at 0.6.11.post1 alongside a gpt-oss triton-kernel fix: #25113 ## New Model Support Entries with a published cookbook recipe come first; entries whose cookbook page is still pending are grouped at the bottom. * DeepSeek V4 (see [cookbook](https://docs.sglang.io/cookbook/autoregressive/DeepSeek/DeepSeek-V4); [LMSYS blog](https://www.lmsys.org/blog/2026-04-25-deepseek-v4/)) * Intern-S2-Preview: #24875, #25115, #25134 (see [cookbook](https://docs.sglang.io/cookbook/autoregressive/InternLM/Intern-S2-Preview)) * MiniCPM-V 4.6: #24855, #24876, #24991, #24998 (see [cookbook](https://docs.sglang.io/cookbook/autoregressive/OpenBMB/MiniCPM-V-4_6)) * Laguna-XS.2 (Poolside): #24204, #24730 (see [cookbook](https://docs.sglang.io/cookbook/autoregressive/Poolside/Laguna-XS.2)) * Ring-2.6-1T (InclusionAI, trillion-param reasoning): #25360, #25370 (see [cookbook](https://docs.sglang.io/cookbook/autoregressive/InclusionAI/Ring-2.6-1T)) * Gemma 4 MTP (MTP head for Gemma 4): #24436, #24433 * Trinity-mini (Ascend NPU, ~90% accuracy): #18172 * HunyuanVideo ModelOpt FP8 (Diffusion): #23199 * Qwen Image ModelOpt FP8 (Diffusion): #23155 ## Speculative Decoding * TokenSpeed MLA prefill/decode kernels integrated as attention backend (FP8 KV cache, Blackwell): #24925 * Adaptive Spec V2 (2/N): #23336 * SWA support for EAGLE-3 drafter: #24664 * Support newer EAGLE-3 drafters: #24663 * Kimi K2.5 EAGLE-3 MLA spec decoding: #24826 * Gemma 3 / Gemma 4 + EAGLE-3 support: #23976 * Spec V1 — split draft-extend into `EagleDraftExtendInput`: #24859 * Custom speculative-algorithm registry: #23991 * Spec-V2 overlap stale-state fix: #23456 * `trtllm` decode kernel for draft extend: #24566 * AMD: EAGLE on Qwen3.5 FP8/MXFP4 via aiter unified attention: #23146 * Fix Kimi K2.5 MLA EAGLE + DP attention: #25033 * Fix `ngram` metric off-by-1 in `num_accepted_drafts_per_req_cpu`: #24965 * Fix frozen-KV MTP crash when `bonus_tokens` is None: #25204 * Fix stuck-MTP on DSA models: #24635 * Reduce specdec CPU overhead: #23321 * Spec-decoding naming-convention rule + refactors: #24094, #25014, #25038, #24081, #24724, #24735, #24881, #25010, #25012, #25030, #25029, #25037, #25109 ## PD Disaggregation * DSv4 Flash disaggregation test: #24973 * Unify DSv4 dispatch with SWA: #24888 * DSv4 mooncake `state_type` branch: #24878 * Hybrid state transfer refactor: #24932 * Priority scheduling in PD mode fix: #25062 * NIXL: staging buffer for heterogeneous-TP KV transfer: #22536 * NIXL: async transfer: #23967 * NIXL XPU: uint64 pointer overflow + mismatched P/D TP fixes: #24188, #24648 * Mooncake: incremental transfer + SSD offload: #24257, #24277 * Multi-node prefill bootstrap-port broadcast: #24378 * Add retry-with-backoff for prefill bootstrap registration: #25125 * `PrefillDelayer`: NCCL all-gather for cross-DP info sync: #24768 * MORI-IO: state transfer + high-concurrency fixes: #22665 * Per-room cleanup centralization; prevent `update_status` from cleared entries; fix abort `update_status` across KV backends: #24601, #24539, #24522 * PD KV transfer metrics fix: #24416 * SWA memory preallocation for disaggregated decode: #24857 * IntraNode NVLink configuration docs: #23329 ## HiCache & Radix Cache * HiCache framework for UnifiedRadixTree: #23316 * SWA HiCache for unified radix cache: #23391 * HiCache for DeepSeek V4 + nightly CI for DSA model: #24691, #25369, #25348 * SSD offload through Mooncake store: #24277 * HiSparse FP8 KV cache via flashmla_kv backend: #23013 * Default storage prefetch timeout: #23309 * UnifiedRadixCache device match semantics with HiCache: #25277 * UnifiedTree partial match on evicted+backuped nodes: #24943 * UnifiedTree tombstone lock release replay fix: #24972 * UnifiedTree `_cascade_evict` leaf determination fix: #25068 * UnifiedRadixTree align `cache_empty_result` with RadixTree: #24779 * Mamba radix cache KV events; SWA radix cache events: #23678, #24718 * SWA chunk req deferred fix; SWA component host hit fix: #24318, #25085 ## LoRA * MLA attention LoRA (q_b_proj / kv_b_proj): #25001 * CSGMV backend with virtual experts for MoE LoRA: #24007 * MoE LoRA: remove CPU-GPU sync barriers and duplicate code (prefill optimize 2/n, 3/n): #24246, #24262 * LoRADrainer for high P99 TTFT: #17913 * `qkv_proj` buffer sizing when `tp_size > num_key_value_heads`: #24420 * Torch-Native LoRA: embedding + graph optimization: #21885 * Deterministic `lora_id` for multi-node `--lora-paths`: #24555 * Fix broken `sgemm_lora_a_graph_fwd` due to invalid `torch.mm()`: #24760 * Diffusion: fix RowParallel LoRA merged forwarding: #24410 ## Performance * TMA bulk-store `set_mla_kv_buffer` (up to 12× over baseline): #25311 * Kimi tokenizer TTFT optimization: #25265 * Avoid hidden-states D2H copy when `return_hidden_states=false`: #25155 * DeepseekV2MoE: defer shared experts when routed kernel is non-mutating: #25279 * `SGLANG_OPT_FP8_WO_A_GEMM` on by default: #25181 * `--prefill-only-disable-kv-cache` to skip KV pool allocation: #23675 * Gemma 4 MoE: fused Q/K/V RMSNorm + per-expert FP8 ckpt loader: #24696 * Gemma 4 VLM: PCG + fused RMSNorm + residual: #24048 * MHC pipeline: DeepGemm + fused norm + fused hc_head: #24775 * JIT custom all-reduce default; non-NVL follow-up: #24363, #24742 * `SGLANG_USE_JIT_ALL_REDUCE` → `SGLANG_OPT_USE_CUSTOM_ALL_REDUCE_V2`: #24297 * Eliminate logits H2D blocking copy: #24627 * Cache empty `MatchResult` in RadixCache: #24470 * Breakable CUDA graph for `bs > 1`: #24662 * FA3: skip `scheduler_metadata` precompute under DP attention: #24632 * `aten::rms_norm` / `aten::mm.dtype` registration in batch-invariant mode: #24459 * Optimize Helios fused norm modulation: #24059 * Z-Image packed QKV optimization: #24117 * KDA prefill kernels: diagonal + recompute fuse: #24271 ## Observability * `sglang:get_loads_duration_seconds` Prometheus metric: #25163 * Per-iteration forward-pass metrics via ZMQ PUB: #22789 * `SGLANG_TRACE_LEVEL` env for startup trace level: #24716 * `fwd_occupancy` metric in `SchedulerStats` + Prometheus collector: #24458 * SWA / Mamba cache metrics: #24396 * Mamba radix cache + SWA radix cache KV events: #23678, #24718 * PD KV transfer metrics fix: #24416 * CP allgather buffer registered with symmetric memory: #24040 * Decode-side bootstrap/alloc metrics + non-int token-id filter: #24684 ## Frontend & API * `/v1/tokenize` chat-completion-style support: #23981 * Multi-detokenizer support: #24944 * Structural tags for strict tool calling & reasoning across more models: #21722 * Auto-detect reasoning / tool-call parser from chat template: #23952 * Two-phase reasoning grammar + `--enable-strict-thinking`: #23953 * OpenAI `reasoning.enabled` mapping to `thinking` + `enable_thinking`: #23951 * Kimi-K2.5 bare-numeric tool-call IDs: #23950 * Crusoe managed-inference backend: #20475 * Azure Blob Storage connector (`az://` and `*.blob.core.windows.net`): #23995 * Adaptive queue-based prefill-delayer trigger: #23189 * `SGLANG_MAX_KV_CHUNK_CAPACITY` env: #25120 * `SGLANG_RADIX_FORCE_MISS` env: #24726, #24950 * Reject `repetition_penalty=0` in `SamplingParams.verify()`: #24874 * `--random-input-len` for `send_one.py`: #24464 ## SGLang-Diffusion * New model support: HunyuanVideo ModelOpt FP8 (#23199), Qwen Image ModelOpt FP8 (#23155) * CFG parallelism framework + multi-branch CFG for LTX-2: #23736 * Initial dynamic batching: #18764 * Performance-mode server args: #24491 * `dit_precision` config respected (no hardcoded bf16): #24988 * Cache-DiT: mount before `torch.compile` in native denoising: #25328 * Z-Image Cache-DiT sequence-parallel override fix: #25305 * USP: direct all-to-all collectives; NCCL deadlock fix for remainder seq lengths: #24366, #24694 * FA3 varlen `out` argument handling: #24688 * RowParallel LoRA merged forwarding fix: #24410 * CFG communication: handle non-contiguous tensors: #24332 * LTX-2.3 alignment with official + HQ denoising split passes: #24313, #24298 * LTX-2 feed-forward TP optimization (#23221) + Hunyuan3D shape denoising / export chunks: #24287, #24358 * Encoder result cache for default negative prompt: #24304 * Channels-last 3D VAE convs by default; disable VAE CPU offload by default: #23200, #24315 * Component attention-backend override CLI: #24320 * AMD: online MXFP4 + FP8 diffusion quantization; aiter RMSNorm; temporal-unfolded batched Conv2D for ROCm VAE decode; dual-stream MoE: #21431, #24360, #22971, #24005, #24677 * NPU: MXFP8 quantization for Wan2.2 (#20922, #24918); fused-operator E2E perf for Wan (#24028); selectable parallel VAE decode strategies (#23248); SANA fix (#24798); Z-Image negative-branch rotary embed CFG fix (#23538) * MUSA: sage attention backend (#24752) ## AMD / ROCm * DSv4 Flash / Pro nightly tests on MI35x ROCm 7.2: #24203, #24825, #25039 * NSA indexer fallbacks + preshuffle paged MQA + GLM-5 NSA TileLang: #24125, #23562, #25205 * `fp8` blockwise quantization combine for MoRI EP: #24879 * gfx950 + aiter `_skip_rope_for_aiter_fused_mla`: #24148 * aiter `fused_qk_rmsnorm` API shim (pre/post #2958): #24799 * TBO Spec-V2 `seq_lens_cpu` None handling: #24319 * Kimi-K2.6 nightly tests (MI30x / MI35x): #23848 * JIT kernel PR-CI through `run_suite.py`: #24987 * AMD JIT benches: clamp position + resolve-token-ids: #24209, #24210 (#25209, #25210) * AMD CI hygiene (registration + cleanup + VRAM): #24569, #24572, #24586, #24612, #24614, #24615, #24665, #24924, #24981, #25112 * Docker: cache-dit 1.3.0 pin; `archive.ubuntu.com` fallback: #24924, #24407 ## NPU / Ascend * `zbal` support: #24575 * Trinity-mini support (~90% accuracy): #18172 * Shared-expert dual-stream optimization: #23827 * Mamba-extra-buffer radix cache (Qwen3.5): #23891 * MLA KV transfer in pipeline parallel: #23893 * Multi-batch FIA ops: #20177 * GLM-5 docs: DeepEP enabled by default: #23708 * GLM-4.5V / GLM-4.7-Flash NPU support / fixes (carry-over): existing * `--disable-cuda-graph` + MTP warmup fix: #23819 * MRoPE position fix in Eagle Worker v2 with PlanStream: #23423 * Z-Image negative-branch rotary embeddings for CFG: #23538 * Wan quantization fix: #24540 * `causal_conv1d_update_v2` for performance: #24595 * `sgl-kernel-npu` 2026.05.01 bump: #24951 * Profiler revert + re-add: #24685, #24815 * Doc / accuracy / FAQ work: #21537, #24658, #24676, #24777, #25114, #25130, #25268, #24668, #24918 ## CPU / Intel / MUSA / MLX / Apple Silicon * MUSA: FlashInfer sampling backend: #24978 * MUSA: optimized kernels for piecewise CUDA graph: #23633 * MUSA: optimized kernels for hot ops: #23255 * MUSA: `torchada` 0.1.54 bump: #24592 * MLX: on-the-fly `--quantization mlx_q4` / `mlx_q8` on Apple Silicon: #24907 * MLX: auto-detect MLX-format `quantization_config` dict: #25191 * MLX: thread `--quantization` through `MlxModelRunner` in `bench_one_batch`: #25221 * MLX: Apple Silicon Metal kernel support in `sgl-kernel`: #23449 * `sgl-kernel/cpu`: w8a8 int8 model support for arm cpu: #16045 * Intel CPU tests migrated to `test/registered` (re-applied after revert): #25139, #22670, #25044 * Arm64 CPU Phase-1A CI bootstrap: #22123 * XPU pipeline parallelism on Intel: #23472 ## Quantization & Kernels * NVFP4 hot-reload-safe weight loading (alias-when-same-shape): #25190 * NVFP4: free unused source scales after weight processing: #25107 * Cute-DSL NVFP4 quantization kernels: #23745 * Cute-DSL FP4 dense GEMM (reland): #23590 * DSv3.2 indexer GEMM via `torch.mm`: #23856 * PDL for DSv3.2 / GLM-5 kernels: #23965 * DSv4: W4A4 MegaMoE; W4(MXFP4)A16 on Hopper: #25052, #24986 * FlashInfer SM90 cutlass MXFP4 MoE backend (W4A16) for GPT-OSS + DSv4: #24816 * Port KV Compression V2 + fused SiLU+clamp+FP8 quant from DSV4 dev branch: #24890, #24897 * BF16 EP-MoE for DeepGEMM: #17392 * DeepGEMM deprecated in sgl-kernel; custom `sgl-deep-gemm` wheel + release workflow: #24268, #24348, #24385 * TRT-LLM A2A dispatch: NaN sanitization in padding slots: #24850 * TRT-LLM BF16 MoE for MTP: #24260 * MegaMoE decoupled from DeepEP backend (subsequently reverted): #24884, #25317 * DeepEP waterfill load balancing for shared-expert dispatch: #19290 * DeepEP support for `--enable-return-routed-experts`: #16859 ## Dependencies * FlashInfer 0.6.8.post1 → 0.6.11 → 0.6.11.post1 (with intermediate revert): #24452, #25129, #25310, #25335 * `sgl-kernel` 0.4.2.post1, 0.4.2.post2: #24457, #25326 * `sgl-kernel`: SM90 flashmla compile fix: #24130 * Custom `sgl-deep-gemm` wheel + release workflow: #24268, #24348, #24385 * `sgl-kernel-build` x86 + arm merged into reusable workflow; disk-reclaim cleanup: #25135, #25206 * DeepEP swapped from `fzyzcjy` fork to `deepseek-ai/DeepEP@hybrid-ep` (CUDA 13): #25113 * Torch 2.11 Docker prep + dependency cleanup: #23593 * `nixl` stub installation alongside `nixl-cuXX` binary: #24369 * aarch64 cubin handling + masked-failure fix: #24234 * H20 stage on CUDA 13: #24916 * CUDA-13 kernel installation docs: #24181, #24516 * FlashInfer autotune cache: #24156 * FlashInfer workspace OOM fix: #24172 * FlashInfer allreduce fusion disabled under deterministic inference: #24629 * `trtllm` allreduce fusion with PDL: #23765 * TRTLLM MHA routing fix for draft-extend: #24856 * `torchcodec` → `soundfile` WAV fallback for trailing metadata: #24185 * `sgl-kernel-npu` 2026.05.01: #24951 ## Security _No security-tagged PRs in this window._ **All PRs included in this release**: https://github.com/sgl-project/sglang/compare/v0.5.11...v0.5.12 ## New Contributors * @Seven-Streams made their first contribution in https://github.com/sgl-project/sglang/pull/21722 * @stargazerZJ made their first contribution in https://github.com/sgl-project/sglang/pull/24344 * @Jianhong-Zhang made their first contribution in https://github.com/sgl-project/sglang/pull/24188 * @gh1595 made their first contribution in https://github.com/sgl-project/sglang/pull/24420 * @revanthreddy-hai made their first contribution in https://github.com/sgl-project/sglang/pull/24329 * @TallMessiWu made their first contribution in https://github.com/sgl-project/sglang/pull/20922 * @ranimandepudi made their first contribution in https://github.com/sgl-project/sglang/pull/22123 * @Joey-gvwal made their first contribution in https://github.com/sgl-project/sglang/pull/23255 * @fanxingran made their first contribution in https://github.com/sgl-project/sglang/pull/24129 * @xz-keg made their first contribution in https://github.com/sgl-project/sglang/pull/24604 * @zhongdaor-nv made their first contribution in https://github.com/sgl-project/sglang/pull/23678 * @chfeng-cs made their first contribution in https://github.com/sgl-project/sglang/pull/24434 * @sglang-npu-bot made their first contribution in https://github.com/sgl-project/sglang/pull/24815 * @brian030128 made their first contribution in https://github.com/sgl-project/sglang/pull/24217 * @tjdharamsi made their first contribution in https://github.com/sgl-project/sglang/pull/24871 * @sytianhe made their first contribution in https://github.com/sgl-project/sglang/pull/24716 * @Dogacel made their first contribution in https://github.com/sgl-project/sglang/pull/24663 * @tangcy98 made their first contribution in https://github.com/sgl-project/sglang/pull/24967 * @1pikachu made their first contribution in https://github.com/sgl-project/sglang/pull/22670 * @flutist made their first contribution in https://github.com/sgl-project/sglang/pull/24760 * @acheamponge made their first contribution in https://github.com/sgl-project/sglang/pull/20475 * @taegeonum made their first contribution in https://github.com/sgl-project/sglang/pull/25022 * @RulinJuice made their first contribution in https://github.com/sgl-project/sglang/pull/24874 * @lluki made their first contribution in https://github.com/sgl-project/sglang/pull/24671 * @Religious-J made their first contribution in https://github.com/sgl-project/sglang/pull/20930 * @ltcs11 made their first contribution in https://github.com/sgl-project/sglang/pull/24575 * @damahua made their first contribution in https://github.com/sgl-project/sglang/pull/24907 * @ziang663 made their first contribution in https://github.com/sgl-project/sglang/pull/25126 * @Emmanuel0612 made their first contribution in https://github.com/sgl-project/sglang/pull/25209 * @Jialin made their first contribution in https://github.com/sgl-project/sglang/pull/25234 * @jlee5814 made their first contribution in https://github.com/sgl-project/sglang/pull/25191 * @unseenmars made their first contribution in https://github.com/sgl-project/sglang/pull/24935 * @nano8259 made their first contribution in https://github.com/sgl-project/sglang/pull/25125 * @imp2002 made their first contribution in https://github.com/sgl-project/sglang/pull/24130 * @liuxianglong17 made their first contribution in https://github.com/sgl-project/sglang/pull/25080 **Full Changelog**: https://github.com/sgl-project/sglang/compare/v0.5.11...v0.5.12

# Highlights - **Piecewise CUDA Graph Enabled by Default**: Piecewise CUDA graph capture is now the default execution mode, reducing memory overhead and improving throughput for models with complex control flow patterns: #16331 - **Elastic EP for Partial Failure Tolerance**: Integrate Elastic NIXL-EP into SGLang, enabling partial failure tolerance for DeepSeek MoE deployments — when a GPU fails, the system redistributes expert weights and continues serving without full restart: #19248, #17374, #12068 [blog](https://lmsys.org/blog/2026-03-25-eep-partial-failure-tolerance/) - **GPU Staging Buffer for PD Disaggregation**: Gathers scattered head slices into contiguous memory for bulk RDMA transfer, reducing RDMA request count on GQA models by ~1000x. TPS/GPU on large concurrency increased by ~5x with Prefill TP4+Decode DEP4 on Qwen3.5: #19890 - **HiSparse for Sparse Attention**: Integrate HiSparse sparse attention backend for efficient long-context inference with reduced compute through sparsity-aware attention: #20343 - **SGLang-Diffusion Update**: * Model support: LTX-2, Hunyuan3D-2, Helios * Performance improvements on Qwen-image, Z-image increased by 1.5x * New platform: macOS * New feature: enhance the performance of diffusers backend by integrating all optimization from Cache-DiT * SKILLs: feel free to explore the curated skill for developing and optimizing sglang-diffusion! - **FlashInfer MXFP8 Kernel Support**: Integrate FlashInfer mxfp8 kernels for GEMM and MoE operations, enabling mixed-precision FP8 inference with higher accuracy through microscaling for RL and general workloads: #19537 - **Transformers 5.3.0 Upgrade**: Major upgrade from transformers 4.57.1 to 5.3.0, unlocking support for the latest model architectures and features from HuggingFace. GLM-5 model is now supported in this image instead of the custom built image: #17784 - **DeepSeek V3.2 / GLM-5 Optimization**: **GLM-5 runnable on main branch (with upgraded transformers).** Fused Triton kernel for prefill KV cache fetching, NSA fuse store indexer for K cache, TRT-LLM prefill/decode DSA kernels as default on SM100/SM103, and IndexCache for improved throughput by more than 10% on high workloads: #19319, #19148, #20062, #21914, #21405 - **Qwen3.5 GDN/KDA Optimization**: Transpose linear attention state layout from [N, HV, K, V] to [N, HV, V, K] and fuse split/reshape/cat ops in GDN projection with Triton kernel, plus CuTeDSL KDA decode kernel support for improved Qwen3.5 performance: #20283, #21019, #21203 - **LoRA Support for MoE Layers**: Add LoRA fine-tuning support for Mixture-of-Experts layers with JIT alignment kernels, fused Triton kernels, TP support, CUDA graph support, and auto-detection of LoRA target modules — enabling efficient adapter-based tuning on MoE models like DeepSeek: #19710, #19711, #14105, #21439, #21647 - **Prefill Context Parallel for MHA (Qwen3)**: Enable context parallelism during prefill for multi-head attention models like Qwen3 MoE, distributing long sequences across GPUs to reduce per-GPU memory and accelerate prefill: #18233 - **Flash Attention 4 Official Library Support**: Upgrade to the official FlashAttention 4 package, bringing the latest attention optimizations and Blackwell GPU support: #20303 - **Skip-Softmax Attention for FlashInfer TRT-LLM Kernels**: Reduce computation overhead in attention layers by skipping redundant softmax normalization: #19089 - **Speculative Decoding with FA4 Backend**: Enable speculative decoding for the FA4 attention backend, combining speculative inference with next-generation flash attention for faster generation: #21080 - **MM Attention FA4 Default on SM100**: Multi-modal attention now uses FA4 by default on Blackwell hardware for improved VLM performance: #21595 - **Stronger Transformers Modeling Backend**: Enhanced transformers backend with full TP, PP, MoE, VLM support, and torch.compile compatibility: #19163 - **sglang-kernel 0.4.1**: Major kernel package release with renamed package (sgl-kernel → sglang-kernel), consolidated kernels, and cleanup of deprecated ops: #20440, #22009 - **Native MLX Backend for Apple Silicon**: Add native MLX execution backend enabling SGLang to run inference directly on Apple Silicon Macs without CUDA: #20342 ## New Model Support * Nemotron-3-Super (bf16/fp8/nvfp4): #20407, [cookbook](https://cookbook.sglang.io/autoregressive/NVIDIA/Nemotron3-Super) * Mistral Small 4 (Pixtral): #20708 * LFM2-VL (Liquid Foundation Model 2 Vision-Language): #21230 * Voxtral (speech-to-text): #21635 * GLM-5: Supported on main branch with transformers 5.3.0 * Helios (Diffusion - Real-Time Long Video Generation): #19782 * Hunyuan3D-2 (Diffusion): #18170 * LTX-2 (Diffusion): #19295 * MOVA (Diffusion): #19489, #20430 * FireRed-Image-Edit (Diffusion): #20862 ## DeepSeek V3.2 / GLM-5 Optimization * Fused get_k_and_s Triton kernel for prefill KV cache fetching: #19319 * Support NSA fuse store indexer K cache: #19148 * `SGLANG_NSA_DENSE_ATTN_KV_LEN_THRESHOLD` environ for controlling KV length threshold of applying sparse MLA attention kernel at prefill: #20062 * Support TRT-LLM prefill/decode DSA kernels as default for Blackwell (SM100/SM103): #21914, #21783 * Enable IndexCache for improved throughput by more than 10% on high workloads: #21405 * Change default setting of V3.2 nvfp4 on TP4: #20086 ## Qwen3.5 Optimization * GDN attention state layout transposed from [N, HV, K, V] to [N, HV, V, K]: #20283 * Fuse split/reshape/cat ops in GDN projection with Triton kernel: #21019 * CuTeDSL KDA decode kernel support: #21203 * Fuse GDN kkt + solve_tril and KDA kernels: #21411, #21604 * GDN packed decode support: #20627 ## Performance * Piecewise CUDA graph enabled by default: #16331 * FlashInfer MXFP8 kernels for GEMM and MoE: #19537 * Skip-softmax attention for FlashInfer TRT-LLM kernels: #19089 * NCCL/RCCL pre-warming to reduce P99 TTFT cold-start latency: #20477 * Overlap NSA-CP key all-gather with query computation for DeepSeek-V3.2: #20438 * CUTLASS FP8 Blockwise GEMM improvement for SM120: #20887 * CUTLASS NVFP4 GEMM improvement for SM120: #21314 * Enable multi-thread weight loading by default: #20289 * Optimize CUDA IPC for multimodal transfer by caching IPC pool handles: #21418 ## LoRA * LoRA support for MoE layers with JIT alignment kernel, fused Triton kernel, and TP: #19710, #19711, #14105 * Auto-detect LoRA target modules: #21439 * LoRA support for CUDA graph: #21647 * LoRA support for Qwen3-VL-30B-A3B and GPT-OSS 20B: #21469, #21570 ## Elastic EP * Integrate Elastic NIXL-EP into SGLang: #19248 * Back up Expert Weights in DRAM: #17374 * Use GPU P2P to exchange expert weights during EPLB: #12068 * Add EPLB rebalance support for Kimi K2.5: #21004 ## SGLang-Diffusion * Model support: LTX-2 (#19295), Hunyuan3D-2 (#18170), Helios (#19782), FireRed-Image-Edit (#20862), MOVA (#19489, #20430) * Performance: Optimized Qwen-image with fused residual/layernorm/scale/shift/gate/select01 kernel (#20395), Z-Image with fused Triton rotary embedding and select01 kernels (#21387, #21318) — up to 1.5x speedup * Platform: macOS support for diffusion models (#19549, #20607) * Feature: Enhance diffusers backend by integrating all optimizations from Cache-DiT: #20361 * NVFP4 support for Flux.2: #20137 * Diffusion norm fusion for Z-Image: #18762 * LTX-2 two-stage pipeline support: #20707 ## Speculative Decoding * Reference-based speculative decoding refactor: #20393 * Add FA4-based speculative decoding support: #21080 ## Disaggregation (PD) * GPU staging buffer with dynamic ring allocator for heterogeneous TP KV transfer: #19890 * HiSparse direct cache transfer from Prefill to Decode DRAM: #21591 * Non-blocking `try_ensure_parallel_info` in pending queue: #20785 * Add kv_cache_dtype consistency check for PD disaggregation: #19407 ## HiCache * HiSparse for sparse attention: #20343 * HybridCacheController for mamba state offloading: #20457 ## VLM * Replace decord with torchcodec for video decoding: #20055 * Replace soundfile+torchaudio with torchcodec AudioDecoder in load_audio: #20190 * Chunk-aware ViT encoding with per-image cache and lazy device transfer: #22038 * Compute M-RoPE positions for preprocessed VL inputs (gRPC): #21244 ## Bug Fixes * Fix streaming session with paged KV cache (SWA/MLA): #20070 * Fix VRAM leak in overlap scheduling with structured output: #20697 * Fix chunked prefill and KV cache leaks for streaming sessions: #20476 * Fix streaming logprobs corruption caused by shared mutable list reference: #21030 * Fix TRT-LLM MHA CUDA illegal address with EAGLE v2 + DP attention: #21649 * Fix Mistral Small 4 config/weight format mismatch: #21620 * Fix mamba cache leak when adder fails to add a matched req: #21404 * Propagate grammar errors and improve llguidance backend: #20467 ## Features * Add reasoning tokens usage: #15562 * Add `--stream-response-default-include-usage` server flag: #16711 * Subprocess liveness monitor to detect scheduler crashes: #18582 * Score API — implement EngineScoreMixin: #21342 * Direct model loading from object storage with RunAI Model Streamer: #17948 * MFU metrics in Prometheus: #19395 ## Network / IPv6 * Add `NetworkAddress` abstraction for IPv6-safe address handling: #20306 * Fix socket utilities and reserve_port for IPv6 dual-stack support: #20491 * Add `--strict-ports` option for predictable port assignment: #21320 ## AMD Hardware * FP8 prefill integration with radix cache path for DeepSeek models: #20187 * Add MHA FP8-KV support: #21253 * Support AMD MXFP4 Qwen3.5-397B-A17B model: #21234 * Fused rope KV store: #21315 * Optimize Qwen3-VL decode — fuse QK-norm + 3D mRoPE + KV cache write: #21458 * Enable FP8 KV cache and FP8 attention kernel for NSA on MI300/MI355 with TileLang: #21511 * Improve openai/gpt-oss performance: #21020 ## NPU/Ascend * GLM-5 optimize with fused kernels: #18617 * Support GLM-4.7-Flash on NPU: #21408 * Replace swiglu with custom kernel: #20192 * Support Kimi-K2.5-w4a8 on Ascend: #20131 * NPU support for diffusion models with enable_torch_compile: #20687 ## CPU Backend * Add kernel apply_rotary_pos_emb_cpu for Qwen3-VL and Qwen3-Omni: #13121 * Implement MXFP4 GEMM kernels for Intel AMX to support GPT-OSS series: #14385 * Enable DeepSeek R1 inference on XPU [Intel GPU]: #18461 ## MPS (Apple Silicon) * Native MLX execution backend for Apple Silicon Mac: #20342 * Fix Triton stub sub-module imports on Python 3.12+: #21551 ## Dependencies * sgl-kernel 0.3.21 → sglang-kernel 0.4.1: #20440, #22009 * FlashInfer 0.6.3 → 0.6.7.post2: #20480, #22097 * Transformers 4.57.1 → 5.3.0: #17784 * xgrammar 0.1.25 → 0.1.32: #21032 * mooncake-transfer-engine 0.3.9 → 0.3.10.post1: #20942, #21844 * Flash Attention 4 (official release): #20303 * Diffusers 0.36.0 → 0.37.0: #20318 ## Security * Fix CVE-2026-3989: Replace unsafe pickle.loads with SafeUnpickler in replay_request_dump.py: #20904 * Fix CVE-2026-3059 / CVE-2026-3060: Bind ZMQ sockets to localhost to prevent unauthenticated remote access (multimodal generation broker and encoder parallel disaggregation): #21435 ## New Contributors * @842974287 made their first contribution in https://github.com/sgl-project/sglang/pull/21917 * @adityavaid made their first contribution in https://github.com/sgl-project/sglang/pull/21209 * @alexnails made their first contribution in https://github.com/sgl-project/sglang/pull/21818 * @Alisehen made their first contribution in https://github.com/sgl-project/sglang/pull/20687 * @AMD-yanfeiwang made their first contribution in https://github.com/sgl-project/sglang/pull/19416 * @aramasethu made their first contribution in https://github.com/sgl-project/sglang/pull/19395 * @avjves made their first contribution in https://github.com/sgl-project/sglang/pull/20178 * @chadvoegele made their first contribution in https://github.com/sgl-project/sglang/pull/20870 * @ChuanLi1101 made their first contribution in https://github.com/sgl-project/sglang/pull/20409 * @Cishoon made their first contribution in https://github.com/sgl-project/sglang/pull/20697 * @cs-cat made their first contribution in https://github.com/sgl-project/sglang/pull/20368 * @dubin555 made their first contribution in https://github.com/sgl-project/sglang/pull/20686 * @e-martirosian made their first contribution in https://github.com/sgl-project/sglang/pull/20352 * @fanghao566 made their first contribution in https://github.com/sgl-project/sglang/pull/20625 * @foraxe made their first contribution in https://github.com/sgl-project/sglang/pull/21842 * @froststeam made their first contribution in https://github.com/sgl-project/sglang/pull/21296 * @Godmook made their first contribution in https://github.com/sgl-project/sglang/pull/19234 * @iammrj made their first contribution in https://github.com/sgl-project/sglang/pull/20714 * @JackZeng0208 made their first contribution in https://github.com/sgl-project/sglang/pull/21050 * @Jacob0226 made their first contribution in https://github.com/sgl-project/sglang/pull/20175 * @jasperjiaguo made their first contribution in https://github.com/sgl-project/sglang/pull/19818 * @Javtor made their first contribution in https://github.com/sgl-project/sglang/pull/20556 * @jellysnack made their first contribution in https://github.com/sgl-project/sglang/pull/20467 * @jhchouuu made their first contribution in https://github.com/sgl-project/sglang/pull/21673 * @jiabinwa made their first contribution in https://github.com/sgl-project/sglang/pull/21002 * @jszzr made their first contribution in https://github.com/sgl-project/sglang/pull/20605 * @karanb192 made their first contribution in https://github.com/sgl-project/sglang/pull/21551 * @Kare0638 made their first contribution in https://github.com/sgl-project/sglang/pull/20403 * @kitft made their first contribution in https://github.com/sgl-project/sglang/pull/20376 * @kpham-sgl made their first contribution in https://github.com/sgl-project/sglang/pull/19807 * @lawrence-harmonic made their first contribution in https://github.com/sgl-project/sglang/pull/20273 * @libowen2121 made their first contribution in https://github.com/sgl-project/sglang/pull/20778 * @LinyuanLi0046 made their first contribution in https://github.com/sgl-project/sglang/pull/20134 * @litmei made their first contribution in https://github.com/sgl-project/sglang/pull/19939 * @LiYomi made their first contribution in https://github.com/sgl-project/sglang/pull/21620 * @LLThomas made their first contribution in https://github.com/sgl-project/sglang/pull/20365 * @lucifer1004 made their first contribution in https://github.com/sgl-project/sglang/pull/20868 * @lviy made their first contribution in https://github.com/sgl-project/sglang/pull/18213 * @mvanhorn made their first contribution in https://github.com/sgl-project/sglang/pull/20326 * @Naveassaf made their first contribution in https://github.com/sgl-project/sglang/pull/21416 * @noa-neria made their first contribution in https://github.com/sgl-project/sglang/pull/17948 * @nv-anants made their first contribution in https://github.com/sgl-project/sglang/pull/21032 * @power-more made their first contribution in https://github.com/sgl-project/sglang/pull/20194 * @psaab made their first contribution in https://github.com/sgl-project/sglang/pull/20491 * @qy-seu made their first contribution in https://github.com/sgl-project/sglang/pull/20256 * @randgun made their first contribution in https://github.com/sgl-project/sglang/pull/19246 * @Ricardo-M-L made their first contribution in https://github.com/sgl-project/sglang/pull/22007 * @roopaksrivastav made their first contribution in https://github.com/sgl-project/sglang/pull/20345 * @saatwiknagpal made their first contribution in https://github.com/sgl-project/sglang/pull/21418 * @sawsa307 made their first contribution in https://github.com/sgl-project/sglang/pull/21463 * @sbeurnier made their first contribution in https://github.com/sgl-project/sglang/pull/17905 * @shadowxz109 made their first contribution in https://github.com/sgl-project/sglang/pull/19879 * @Shunkangz made their first contribution in https://github.com/sgl-project/sglang/pull/18233 * @syd520zy made their first contribution in https://github.com/sgl-project/sglang/pull/16711 * @SYaoJun made their first contribution in https://github.com/sgl-project/sglang/pull/20723 * @vaibhawvipul made their first contribution in https://github.com/sgl-project/sglang/pull/21010 * @wang-zhuoran made their first contribution in https://github.com/sgl-project/sglang/pull/20419 * @wduan-hai made their first contribution in https://github.com/sgl-project/sglang/pull/21705 * @Wenjun7J made their first contribution in https://github.com/sgl-project/sglang/pull/21413 * @wili-65535 made their first contribution in https://github.com/sgl-project/sglang/pull/19749 * @xiazhahe made their first contribution in https://github.com/sgl-project/sglang/pull/21507 * @xieminghe1 made their first contribution in https://github.com/sgl-project/sglang/pull/19672 * @xingsy97 made their first contribution in https://github.com/sgl-project/sglang/pull/19630 * @xq25478 made their first contribution in https://github.com/sgl-project/sglang/pull/21727 * @xyuzh made their first contribution in https://github.com/sgl-project/sglang/pull/17684 * @yafengio made their first contribution in https://github.com/sgl-project/sglang/pull/19959 * @yang1002378395-cmyk made their first contribution in https://github.com/sgl-project/sglang/pull/21319 * @yejinkimis made their first contribution in https://github.com/sgl-project/sglang/pull/20210 * @ykcai-daniel made their first contribution in https://github.com/sgl-project/sglang/pull/20137 * @youngrok-XCENA made their first contribution in https://github.com/sgl-project/sglang/pull/20560 * @yuumn made their first contribution in https://github.com/sgl-project/sglang/pull/20862 * @yyy1000 made their first contribution in https://github.com/sgl-project/sglang/pull/18907 * @YzXiao101 made their first contribution in https://github.com/sgl-project/sglang/pull/22054 * @ZhaiFeiyue made their first contribution in https://github.com/sgl-project/sglang/pull/20410 * @zhentaocc made their first contribution in https://github.com/sgl-project/sglang/pull/18242 * @Zijun9 made their first contribution in https://github.com/sgl-project/sglang/pull/20891 * @zzhpro made their first contribution in https://github.com/sgl-project/sglang/pull/20132 **Full Changelog**: https://github.com/sgl-project/sglang/compare/v0.5.9...v0.5.10

## 🚀 SGLang Model Gateway v0.3.0 Released! We're thrilled to announce SGLang Model Gateway v0.3.0 – a major release with powerful new features, architectural improvements, and important breaking changes! ## ⚠️ Breaking Changes ### 📊 Metrics Architecture Redesigned Complete overhaul with new 6-layer metrics architecture covering protocol (HTTP/gRPC), router, worker, streaming (TTFT/TPOT), circuit breaker, and policy metrics with unified error codes. **Action Required**: Update your Prometheus dashboards and alerting rules. Metric names and structure have changed. ### 🔧 UUID-Based Worker Resource Management Workers are now identified by UUIDs instead of endpoints for cleaner resource management. **Action Required**: Update any tooling or scripts that interact with the worker API. ## ✨ New Features ### 🌐 Unified Inference Gateway Mode (IGW) Single gateway, entire fleet. IGW now supports ALL router types in a single deployment with Kubernetes service discovery: - gRPC router (PD and regular mode) - HTTP router (PD and regular mode) - OpenAI router Auto-enabled with service discovery. Deploy once, route everything - handle all traffic patterns across your entire inference fleet from a single gateway instance. ### 🔤 Tokenize/Detokenize HTTP Endpoints - Direct HTTP endpoints for tokenization operations - Dynamic tokenizer control plane: add, list, get, and remove tokenizers on-the-fly - TokenizerRegistry for efficient dynamic loading ### 🧠 Parser Endpoints - `/parse/reasoning` - Parse reasoning outputs - `/parse/function_call` - Parse function call responses - GLM-4 function call parser - Contributed directly by the GLM team for latest GLM models ### 📊 Embeddings Support Native embeddings endpoint for gRPC router - expand beyond text generation to embedding workloads. ### 🔐 Server-Side TLS Support Secure your gateway deployments with native TLS support. ### 🌐 Go Implementation, contributed by iFlytek MaaS team. Complete Go SGLang Model Gateway with OpenAI-compatible API server - bringing SGLang to the Go ecosystem! ## ⚡ Major Enhancements ### Control Plane - Workflow Engine Intelligent lifecycle orchestration with: - DAG-based parallel execution with pre-computed dependency graphs - Concurrent event processing for maximum throughput - Modular add/remove/update workflows ### Performance Optimization - Lock-free data structures: DashMap for policy lookups, lock-free router snapshots - Reduced CPU overhead: Optimized worker registry, gRPC client fetch, and worker selection - Optimized router management: Improved selection algorithms and state management ### Resilience & Reliability: - Retry and circuit breaker support for OpenAI and gRPC routers - Enhanced circuit breaker with better state management - Graceful shutdown for TLS and non-TLS servers - Unified error responses with error codes and X-SMG-Error-Code headers ### Infrastructure: - Multi-architecture Docker builds (Linux, macOS, Windows, ARM) - Custom Prometheus duration buckets - Improved logging across all modules ## 🐛 Bug Fixes & Stability - Fixed cache-aware routing in gRPC mode - Resolved load metric tracking and double-decrease issues for cache aware load balancing - Improved backward compatibility for GET endpoints - Fixed gRPC scheduler launcher issues - Fixed token bucket negative duration panics - Resolved MCP server initialization issues ## 📚 Documentation Major documentation update with comprehensive guides, examples, and best practices for SGLang Model Gateway. ## ⚠️ Migration checklist: - Update Prometheus dashboards for new metrics - Update worker API integrations for UUID-based management - Review new error response format ## ⚡ Built for speed. Engineered for scale. Production-proven. ### Gateway Changes (108 commits) - [model-gateway] release smg 0.3.0 (#15781) by @slin1237 in https://github.com/sgl-project/sglang/pull/15781 - [model-gateway] Fix logging module name, parse endpoint context, and tokenizer factory (#15782) by @slin1237 in https://github.com/sgl-project/sglang/pull/15782 - [model-gateway] Implement Zero-Copy Vision Tensor Access (#15750) by @ppraneth in https://github.com/sgl-project/sglang/pull/15750 - [model-gateway] Fix IGW routing and optimize RouterManager (#15741) by @slin1237 in https://github.com/sgl-project/sglang/pull/15741 - Fix smg_http_requests_total semantics (#15655) by @fzyzcjy in https://github.com/sgl-project/sglang/pull/15655 - [model-gateway]Enable IGW mode with gRPC router and auto enable IGW when service discovery is turned on (#15459) by @YouNeedCryDear in https://github.com/sgl-project/sglang/pull/15459 - [docs] major SGL Model Gateway documentation update (#15715) by @slin1237 in https://github.com/sgl-project/sglang/pull/15715 - [model-gateway] add back router worker health metric and fix init state (#15622) by @fzyzcjy in https://github.com/sgl-project/sglang/pull/15622 - [mode;-gateway] add back fixes of incorrect metrics after worker removal (#15624) by @fzyzcjy in https://github.com/sgl-project/sglang/pull/15624 - [model-gateway] Add tokenize/detokenize HTTP endpoints and tokenizer management (#15702) by @slin1237 in https://github.com/sgl-project/sglang/pull/15702 - [model-gateway] Fix tokenizer caching and improve error handling (#15695) by @slin1237 in https://github.com/sgl-project/sglang/pull/15695 - [model-gateway]: add gRPC router embeddings endpoint implementation (#15273) by @Ratish1 in https://github.com/sgl-project/sglang/pull/15273 - [model-gateway] Optimize router selection with lock-free snapshots (#15672) by @ppraneth in https://github.com/sgl-project/sglang/pull/15672 - [model-gateway] Replace tokenizer with tokenizer registry for dynamic tokenizer loading in gRPC router (#12968) by @YouNeedCryDear in https://github.com/sgl-project/sglang/pull/12968 - Improve engine customization interface (#15635) by @merrymercy in https://github.com/sgl-project/sglang/pull/15635 - Tiny add back missing router per attempt response metric (#15621) by @fzyzcjy in https://github.com/sgl-project/sglang/pull/15621 - Fix router gRPC mode launch error caused by async loading (#15368) by @fzyzcjy in https://github.com/sgl-project/sglang/pull/15368 - [model-gateway] return 503 when all workers are circuit-broken (#15611) by @slin1237 in https://github.com/sgl-project/sglang/pull/15611 - [model-gateway] add retry support to OpenAI router chat endpoint (#15589) by @slin1237 in https://github.com/sgl-project/sglang/pull/15589 - Optimize Rust CI builds with proper sccache configuration (#15581) by @slin1237 in https://github.com/sgl-project/sglang/pull/15581 - [model-gateway] add retry and circuit breaker support to gRPC routers (#15585) by @slin1237 in https://github.com/sgl-project/sglang/pull/15585 - [model-gateway] refactor WorkerManager with fan_out helper and thin handlers (#15583) by @slin1237 in https://github.com/sgl-project/sglang/pull/15583 - [model-gateway] add WorkerService abstraction for worker business logic (#15580) by @slin1237 in https://github.com/sgl-project/sglang/pull/15580 - [model-gateway] minor code clean up (#15578) by @slin1237 in https://github.com/sgl-project/sglang/pull/15578 - [model-gateway] Use UUIDs for router-managed worker resources (#15540) by @alphabetc1 in https://github.com/sgl-project/sglang/pull/15540 - [model-gateway] /parse/easoning and parse/function_call for sgl-model-gateway (#15568) by @UbeCc in https://github.com/sgl-project/sglang/pull/15568 - [model-gateway]: Tool parser for glm47 (#15520) by @UbeCc in https://github.com/sgl-project/sglang/pull/15520 - [model-gateway] bugfix: backward compatibility for GET endpoints (#15413) by @alphabetc1 in https://github.com/sgl-project/sglang/pull/15413 - [model-gateway] Optimize WASM Runtime with Instance Pooling and Component Caching (#15515) by @ppraneth in https://github.com/sgl-project/sglang/pull/15515 - [model-gateway] add model gateway multi-arch docker build, test and document docker image (#15544) by @slin1237 in https://github.com/sgl-project/sglang/pull/15544 - [model-gateway] Implement RAII load guard with response body attachment (#15507) by @slin1237 in https://github.com/sgl-project/sglang/pull/15507 - [router] bugfix: cache_aware in grpc inbalance forward (#15473) by @llfl in https://github.com/sgl-project/sglang/pull/15473 - [model-gateway] simplify workflow engine backoff and reduce duplicate reads (#15505) by @slin1237 in https://github.com/sgl-project/sglang/pull/15505 - [model-gateway] Run workflow event subscribers concurrently (#15504) by @slin1237 in https://github.com/sgl-project/sglang/pull/15504 - [model-gateway] Optimize workflow engine with pre-computed dependency graph (#15503) by @slin1237 in https://github.com/sgl-project/sglang/pull/15503 - [model-gateway] Improve logging across core modules (#15497) by @slin1237 in https://github.com/sgl-project/sglang/pull/15497 - [model-gateway] Improve logging in policies module (#15496) by @slin1237 in https://github.com/sgl-project/sglang/pull/15496 - [model-gateway] Improve logging in data_connector module (#15495) by @slin1237 in https://github.com/sgl-project/sglang/pull/15495 - [model-gateway] refactor: extract common graceful shutdown code before TLS branch (#15494) by @slin1237 in https://github.com/sgl-project/sglang/pull/15494 - [model-gateway] fix graceful shutdown for TLS/Non-TLS server (#15491) by @slin1237 in https://github.com/sgl-project/sglang/pull/15491 - [model-gateway] Replace PolicyRegistry RwLock with DashMap for lock-free policy lookups (#15361) by @slin1237 in https://github.com/sgl-project/sglang/pull/15361 - [model-gateway] optimize worker registry and reduce lock contention in grpc client fetch (#15336) by @slin1237 in https://github.com/sgl-project/sglang/pull/15336 - [model-gateway] reduce cpu overhead (#15316) by @slin1237 in https://github.com/sgl-project/sglang/pull/15316 - Super tiny rename failure_count for consistency (#15186) by @fzyzcjy in https://github.com/sgl-project/sglang/pull/15186 - [model-gateway] Remove legacy RouterMetrics and Rename SmgMetrics to Metrics and smg_labels to metrics_labels (#15160) by @slin1237 in https://github.com/sgl-project/sglang/pull/15160 - Fix num running requests (load) wrong cleared for ongoing requests (#15116) by @fzyzcjy in https://github.com/sgl-project/sglang/pull/15116 - [model-gateway] add mcp and discovery metrics (#15156) by @slin1237 in https://github.com/sgl-project/sglang/pull/15156 - [model-gateway] Add streaming metrics for harmony gRPC router (#15147) by @slin1237 in https://github.com/sgl-project/sglang/pull/15147 - [model-gateway] upgrade axum and axum server (#15146) by @slin1237 in https://github.com/sgl-project/sglang/pull/15146 - [model-gateway] Add Layer 3 worker metrics (smg_worker_*) (#15130) by @slin1237 in https://github.com/sgl-project/sglang/pull/15130 - Fix cache aware wrong routing caused by incorrect load tracking (#15101) by @fzyzcjy in https://github.com/sgl-project/sglang/pull/15101 - [model-gateway] fix circuit breaker metrics (#15099) by @fzyzcjy in https://github.com/sgl-project/sglang/pull/15099 - [model-gateway] extract circuit breaker state struct (#15098) by @fzyzcjy in https://github.com/sgl-project/sglang/pull/15098 - [model-gateway] Parallelize metrics requests (#14953) by @ppraneth in https://github.com/sgl-project/sglang/pull/14953 - feat(gateway): Add server-side TLS support (#15052) by @Ratish1 in https://github.com/sgl-project/sglang/pull/15052 - [model-gateway] add streaming metrics (TTFT, TPOT, tokens, duration) for gRPC router (#15125) by @slin1237 in https://github.com/sgl-project/sglang/pull/15125 - [model-gateway] feat(metrics): implement Layer 2 router metrics (smg_router_*) (#15124) by @slin1237 in https://github.com/sgl-project/sglang/pull/15124 - [model-gateway] Implement Layer 1 HTTP metrics instrumentation (#15121) by @slin1237 in https://github.com/sgl-project/sglang/pull/15121 - [model-gateway] Add new SMG metrics architecture with 6 layers (#15106) by @slin1237 in https://github.com/sgl-project/sglang/pull/15106 - Avoid confusing zero value metric when worker is removed (#15096) by @fzyzcjy in https://github.com/sgl-project/sglang/pull/15096 - Fix issue not reported when load decrement is incorrect (#15061) by @fzyzcjy in https://github.com/sgl-project/sglang/pull/15061 - [model-gateway] optimize metric labels to avoid unnecessary allocations (#15095) by @slin1237 in https://github.com/sgl-project/sglang/pull/15095 - [model-gateway] Add circuit breaker and discovery watcher metrics (#15094) by @slin1237 in https://github.com/sgl-project/sglang/pull/15094 - [model-gateway] Fix metric emission gaps and name mismatch (#15093) by @slin1237 in https://github.com/sgl-project/sglang/pull/15093 - [model-gateway] Remove unused TokenizerMetrics to reduce CPU overhead (#15087) by @slin1237 in https://github.com/sgl-project/sglang/pull/15087 - [model-gateway] Refactor worker steps and add update workflow (#15085) by @slin1237 in https://github.com/sgl-project/sglang/pull/15085 - [model-gateway] Avoid MCP Server Initialization Issue (#15065) by @xuwenyihust in https://github.com/sgl-project/sglang/pull/15065 - [bug] fix grpc secheduler launcher breaking change (#15080) by @slin1237 in https://github.com/sgl-project/sglang/pull/15080 - [model-gateway] Simplify error response creation (#15079) by @slin1237 in https://github.com/sgl-project/sglang/pull/15079 - Fix double decrease load (#15060) by @fzyzcjy in https://github.com/sgl-project/sglang/pull/15060 - Fix load metric not updated when using guard (#15059) by @fzyzcjy in https://github.com/sgl-project/sglang/pull/15059 - Add sgl_router_attempt_http_responses_total for single attempt information (#15037) by @fzyzcjy in https://github.com/sgl-project/sglang/pull/15037 - Add error code in prometheus metrics and add X-SMG-Error-Code header (#15036) by @fzyzcjy in https://github.com/sgl-project/sglang/pull/15036 - Provide more fine grained error reason for reqwest error (#15032) by @fzyzcjy in https://github.com/sgl-project/sglang/pull/15032 - Tiny change http router response format to unify (#15031) by @fzyzcjy in https://github.com/sgl-project/sglang/pull/15031 - Tiny unify grpc existing error responses into new format (#15030) by @fzyzcjy in https://github.com/sgl-project/sglang/pull/15030 - Add `code` field and unify error responses for router (#15028) by @fzyzcjy in https://github.com/sgl-project/sglang/pull/15028 - Super tiny remove unused log_request (#15035) by @fzyzcjy in https://github.com/sgl-project/sglang/pull/15035 - [model-gateway] refactor: unify worker management into modular workflow structure (#15010) by @slin1237 in https://github.com/sgl-project/sglang/pull/15010 - Super tiny extract route_typed_request_once (#14951) by @fzyzcjy in https://github.com/sgl-project/sglang/pull/14951 - [model-gateway] refactor: workflow engine cleanup and minor optimization (#15001) by @slin1237 in https://github.com/sgl-project/sglang/pull/15001 - [model-gateway] fix: handle workflow deadlock and optimize cycle detection (#15000) by @slin1237 in https://github.com/sgl-project/sglang/pull/15000 - [model-gateway] feat: add DAG parallel execution support and workflow optimization (#14999) by @slin1237 in https://github.com/sgl-project/sglang/pull/14999 - [model-gateway] refactor: extract workflow engine to src/workflow module (#14996) by @slin1237 in https://github.com/sgl-project/sglang/pull/14996 - Super tiny refactor error.rs logic (#14949) by @fzyzcjy in https://github.com/sgl-project/sglang/pull/14949 - Super tiny move error.rs (#14944) by @fzyzcjy in https://github.com/sgl-project/sglang/pull/14944 - Super tiny remove non-updated sgl_router_worker_load (#14888) by @fzyzcjy in https://github.com/sgl-project/sglang/pull/14888 - Tiny add e2e http request arrival metric (#14893) by @fzyzcjy in https://github.com/sgl-project/sglang/pull/14893 - Tiny add router e2e duration histogram (#14892) by @fzyzcjy in https://github.com/sgl-project/sglang/pull/14892 - Fix negative duration panic in token bucket wait time calculation (#14941) by @xiaguan in https://github.com/sgl-project/sglang/pull/14941 - [model-gateway] optimize worker selection (#14894) by @ppraneth in https://github.com/sgl-project/sglang/pull/14894 - Super tiny remove sgl_router_active_workers (#14891) by @fzyzcjy in https://github.com/sgl-project/sglang/pull/14891 - [SMG][DS32][fix] support dsv32, add role developer (#14307) by @jimmy-evo in https://github.com/sgl-project/sglang/pull/14307 - [model-gateway] fix imports and delete unused code (#14911) by @slin1237 in https://github.com/sgl-project/sglang/pull/14911 - [model-gateway] fix annotation error and code formating (#14910) by @slin1237 in https://github.com/sgl-project/sglang/pull/14910 - [model-gateway] code clean up on oai router in responses (#14852) by @slin1237 in https://github.com/sgl-project/sglang/pull/14852 - Tiny clean router load report logic (#14889) by @fzyzcjy in https://github.com/sgl-project/sglang/pull/14889 - [model-gateway] refactor cleanup WorkflowContext.get_or_err (#14890) by @fzyzcjy in https://github.com/sgl-project/sglang/pull/14890 - [model-gateway] fix import order in oai conversation (#14851) by @slin1237 in https://github.com/sgl-project/sglang/pull/14851 - [model-gateway] code clean up on oai router (#14850) by @slin1237 in https://github.com/sgl-project/sglang/pull/14850 - [model-gateway] adds default implementations to RouterTrait in mod.rs (#14841) by @slin1237 in https://github.com/sgl-project/sglang/pull/14841 - [model-gateway] Fix incompatible metric comparison in` PowerOfTwo` policy (#14823) by @ppraneth in https://github.com/sgl-project/sglang/pull/14823 - [model-gateway] support engine response http status statistics in router (#14712) by @fzyzcjy in https://github.com/sgl-project/sglang/pull/14712 - [model-gateway] support customizing Prometheus duration buckets (#14716) by @fzyzcjy in https://github.com/sgl-project/sglang/pull/14716 - [model-gateway] add anthropic message api spec (#14834) by @slin1237 in https://github.com/sgl-project/sglang/pull/14834 - Fix router keep nonzero metrics after worker is deleted (#14819) by @fzyzcjy in https://github.com/sgl-project/sglang/pull/14819 - [model-gateway] Dynamically Populate Tool Call Parser Choices (#14807) by @xuwenyihust in https://github.com/sgl-project/sglang/pull/14807 - [SMG-GO] implement a Go SGLang Model Gateway - OpenAI Compatible API Server (#14770) by @whybeyoung in https://github.com/sgl-project/sglang/pull/14770 ### New Contributors * @ppraneth made their first contribution in https://github.com/sgl-project/sglang/commit/e99ee0c69 * @xuwenyihust made their first contribution in https://github.com/sgl-project/sglang/commit/d7f6320bb * @alphabetc1 made their first contribution in https://github.com/sgl-project/sglang/commit/1d90b194b * @Ratish1 made their first contribution in https://github.com/sgl-project/sglang/commit/0e4108ba2 * @UbeCc made their first contribution in https://github.com/sgl-project/sglang/commit/26704c23c * @whybeyoung made their first contribution in https://github.com/sgl-project/sglang/commit/766476f52 * @llfl made their first contribution in https://github.com/sgl-project/sglang/commit/3c116d5e5 * @xiaguan made their first contribution in https://github.com/sgl-project/sglang/commit/22fe5da13 * @YouNeedCryDear made their first contribution in https://github.com/sgl-project/sglang/commit/dd620987d * @YouNeedCryDear made their first contribution in https://github.com/sgl-project/sglang/commit/f65fa0474 **Full Changelog**: https://github.com/sgl-project/sglang/compare/gateway-v0.2.4...gateway-v0.3.0

## 🚀 SGLang Model Gateway - New Release! We're excited to announce another powerful update to **SGLang Model Gateway** with performance improvements and expanded database support! ### ✨ **Headline Features** **⚡ Bucket Mode Routing - 20-30% Performance Boost** Introducing our new **bucket-based routing algorithm** that dramatically improves performance in PD mode. See up to **20-30% improvements in TTFT (Time To First Token) and overall throughput** **💾 PostgreSQL Support for Chat History Management** Flexibility in data storage! We now support **PostgreSQL** alongside **OracleDB** and **in-memory storage** for chat history management. **🛠️ Enhanced Model Tool & Structured Output Support** - **MinMax M2** model support! - **Structured model output** for OpenAI and gRPC router - **Streaming parsing with Tool Choice** in chat completions API - **Tool_choice support** for Responses API - **OutputItemDone events** with output item array storage for better observability ### 🐛 **Stability & Quality Improvements** Multiple bug fixes for model validation, streaming logic, reasoning content indexing, and CI stability enhancements. ### 🔧 **Code Quality Enhancements** Refactored builders for chat and responses, restructured modules for better maintainability, and consolidated error handling. Try the latest version: `pip install sglang-router --upgrade` ## What's Changed in Gateway ### Gateway Changes (45 commits) - [model-gateway] smg release 0.2.3 (#13312) by @slin1237 in https://github.com/sgl-project/sglang/pull/13312 - [router]Replace requests lib with openai in e2e_response_api (#13293) by @XinyueZhang369 in https://github.com/sgl-project/sglang/pull/13293 - fix outdated router doc (#13255) by @fzyzcjy in https://github.com/sgl-project/sglang/pull/13255 - [router][grpc] Refine docs in minimax_m2 to match other parsers (#13218) by @CatherineSue in https://github.com/sgl-project/sglang/pull/13218 - fix: display served_model_name in /v1/models (#13155) by @Sunhaihua1 in https://github.com/sgl-project/sglang/pull/13155 - [router] minmax-m2 xml tool parser (#13148) by @slin1237 in https://github.com/sgl-project/sglang/pull/13148 - [router] remove worker url requirement (#13172) by @slin1237 in https://github.com/sgl-project/sglang/pull/13172 - [router] Fix Flaky test_circuit_breaker_opens_and_recovers (#13164) by @XinyueZhang369 in https://github.com/sgl-project/sglang/pull/13164 - [router] Add comprehensive validation to Responses API (#13127) by @key4ng in https://github.com/sgl-project/sglang/pull/13127 - bugfix: multi-model routing for /generate api (#12979) by @SYChen123 in https://github.com/sgl-project/sglang/pull/12979 - [router][grpc] Support vllm backend for grpc router (#13120) by @CatherineSue in https://github.com/sgl-project/sglang/pull/13120 - [router] add minmax m2 reasoning parser (#13137) by @slin1237 in https://github.com/sgl-project/sglang/pull/13137 - [router] Support complex assistant and tool messages in /chat/completions (#12860) by @hellodanylo in https://github.com/sgl-project/sglang/pull/12860 - [router] move radix tree to policy crate and addreses some code styles (#13131) by @slin1237 in https://github.com/sgl-project/sglang/pull/13131 - [Router] use call_id instead of id for matching function calls in Responses API for Harmony (#13056) by @zhaowenzi in https://github.com/sgl-project/sglang/pull/13056 - Revert "fix: display served_model_name in /v1/models" (#13093) by @CatherineSue in https://github.com/sgl-project/sglang/pull/13093 - fix: display served_model_name in /v1/models (#13063) by @Sunhaihua1 in https://github.com/sgl-project/sglang/pull/13063 - [router] add postgres databases data connector (#12218) by @lengrongfu in https://github.com/sgl-project/sglang/pull/12218 - [router][ci] Quick Improvement to make CI more stable (#12869) by @key4ng in https://github.com/sgl-project/sglang/pull/12869 - [router][ci] Fix maturin build (#13012) by @key4ng in https://github.com/sgl-project/sglang/pull/13012 - [router] bucket policy (#11719) by @syy-hw in https://github.com/sgl-project/sglang/pull/11719 - [router] Switch MCP tests from DeepWiki to self-hosted Brave search server (#12849) by @key4ng in https://github.com/sgl-project/sglang/pull/12849 - [router][grpc] Move all error logs to their call sites (#12859) by @CatherineSue in https://github.com/sgl-project/sglang/pull/12859 - [router][grpc] Refactor: Add builders for chat and responses (#12852) by @CatherineSue in https://github.com/sgl-project/sglang/pull/12852 - [router] Support structured model output for openai and grpc router (#12431) by @key4ng in https://github.com/sgl-project/sglang/pull/12431 - [router][grpc] Add more mcp test cases to responses api (#12749) by @CatherineSue in https://github.com/sgl-project/sglang/pull/12749 - fix ci (#12760) by @key4ng in https://github.com/sgl-project/sglang/pull/12760 - Add timing metrics for requests (#12646) by @cicirori in https://github.com/sgl-project/sglang/pull/12646 - [router][ci] Disable cache (#12752) by @key4ng in https://github.com/sgl-project/sglang/pull/12752 - [router][grpc] Support mixin tool calls in Responses API (#12736) by @CatherineSue in https://github.com/sgl-project/sglang/pull/12736 - Revert "[router] web_search_preview tool basic implementation" (#12716) by @key4ng in https://github.com/sgl-project/sglang/pull/12716 - [router] add basic ci tests for gpt-oss model support (#12651) by @key4ng in https://github.com/sgl-project/sglang/pull/12651 - [router][quick fix] Add minimal option for reasoning effort in spec (#12711) by @key4ng in https://github.com/sgl-project/sglang/pull/12711 - [router][grpc] Make harmony parser checks recipient first before channel (#12713) by @CatherineSue in https://github.com/sgl-project/sglang/pull/12713 - [router][ci] speed up python binding to 1.5 min (#12673) by @key4ng in https://github.com/sgl-project/sglang/pull/12673 - [router] fix: validate HTTP status codes in health check (#12631) by @wyx-0203 in https://github.com/sgl-project/sglang/pull/12631 - [router][grpc] Support streaming parsing with Tool Choice in chat completions API (#12677) by @CatherineSue in https://github.com/sgl-project/sglang/pull/12677 - [router][grpc] Implement tool_choice support for Responses API (#12668) by @CatherineSue in https://github.com/sgl-project/sglang/pull/12668 - [router][grpc] Emit OutputItemDone event and store output item array (#12656) by @CatherineSue in https://github.com/sgl-project/sglang/pull/12656 - [router][grpc] Fix index issues in reasoning content and missing streaming events (#12650) by @CatherineSue in https://github.com/sgl-project/sglang/pull/12650 - [router][grpc] Fix model validation, tool call check, streaming logic and misc in responses (#12616) by @CatherineSue in https://github.com/sgl-project/sglang/pull/12616 - Support aggregating engine metrics in sgl-router (#11456) by @fzyzcjy in https://github.com/sgl-project/sglang/pull/11456 - [router][grpc] Restructure modules and code clean up (#12598) by @CatherineSue in https://github.com/sgl-project/sglang/pull/12598 - [router][grpc] Consolidate error messages build in error.rs (#12301) by @CatherineSue in https://github.com/sgl-project/sglang/pull/12301 - [ci] install released version router (#12410) by @key4ng in https://github.com/sgl-project/sglang/pull/12410 ### New Contributors * @XinyueZhang369 made their first contribution in https://github.com/sgl-project/sglang/commit/2cdde3d46 * @Sunhaihua1 made their first contribution in https://github.com/sgl-project/sglang/commit/a06c44f90 * @zhaowenzi made their first contribution in https://github.com/sgl-project/sglang/commit/7b877ab83 * @cicirori made their first contribution in https://github.com/sgl-project/sglang/commit/58095cb00 * @wyx-0203 made their first contribution in https://github.com/sgl-project/sglang/commit/3651cfbf6 * @syy-hw made their first contribution in https://github.com/sgl-project/sglang/commit/611a4fd08 * @SYChen123 made their first contribution in https://github.com/sgl-project/sglang/commit/4ef439054 * @hellodanylo made their first contribution in https://github.com/sgl-project/sglang/commit/d28caaf60 ### Paths Included - `sgl-router` - `python/sglang/srt/grpc` - `python/sglang/srt/entrypoints/grpc_server.py` **Full Changelog**: https://github.com/sgl-project/sglang/compare/gateway-v0.2.2...gateway-v0.2.3

## What's Changed in Gateway ### Gateway/Router Changes (11 commits) - [router] update router pypi version (#8628) by @slin1237 in https://github.com/sgl-project/sglang/pull/8628 - [router] migrate router from actix to axum (#8479) by @slin1237 in https://github.com/sgl-project/sglang/pull/8479 - [feature] [sgl-router] Add a dp-aware routing strategy (#6869) by @oldsharp in https://github.com/sgl-project/sglang/pull/6869 - [router] improve router logs and request id header (#8415) by @slin1237 in https://github.com/sgl-project/sglang/pull/8415 - [router] add different policies for p node and d node (#8395) by @slin1237 in https://github.com/sgl-project/sglang/pull/8395 - [router] add request format unit test (#8300) by @slin1237 in https://github.com/sgl-project/sglang/pull/8300 - [router] add streaming unit test (#8299) by @slin1237 in https://github.com/sgl-project/sglang/pull/8299 - [router] add endpoint unit test (#8298) by @slin1237 in https://github.com/sgl-project/sglang/pull/8298 - [router] fix pd model completion request (#8303) by @slin1237 in https://github.com/sgl-project/sglang/pull/8303 - [router] add common ut infra to mock worker and app (#8295) by @slin1237 in https://github.com/sgl-project/sglang/pull/8295 - fix: sgl-router remove dead code (#8257) by @oldsharp in https://github.com/sgl-project/sglang/pull/8257 ### New Contributors * @oldsharp made their first contribution in https://github.com/sgl-project/sglang/commit/a730ce816 * @oldsharp made their first contribution in https://github.com/sgl-project/sglang/commit/c33499a67 ### Paths Included - `sgl-router` - `python/sglang/srt/grpc` - `python/sglang/srt/entrypoints/grpc_server.py` **Full Changelog**: https://github.com/sgl-project/sglang/compare/gateway-v0.1.6...gateway-v0.1.7

## What's Changed in Gateway ### Gateway/Router Changes (12 commits) - [router] upgade router version to 0.1.6 (#8209) by Simo Lin in https://github.com/sgl-project/sglang/pull/8209 - [router] add ut for pd router (#8208) by Simo Lin in https://github.com/sgl-project/sglang/pull/8208 - [router] add ut for pd request, metrics and config (#8184) by Simo Lin in https://github.com/sgl-project/sglang/pull/8184 - [router] add ut for worker and errors (#8170) by Simo Lin in https://github.com/sgl-project/sglang/pull/8170 - Fix suffix mismatch for the metrics. (#8168) by Charles Chen in https://github.com/sgl-project/sglang/pull/8168 - [router] allow router to have empty workers (#8160) by Simo Lin in https://github.com/sgl-project/sglang/pull/8160 - [router] router metrics cleanup (#8158) by Simo Lin in https://github.com/sgl-project/sglang/pull/8158 - [router] Refactor router and policy traits with dependency injection (#7987) by Simo Lin in https://github.com/sgl-project/sglang/pull/7987 - [router] add worker abstraction (#7960) by Simo Lin in https://github.com/sgl-project/sglang/pull/7960 - [bugfix] add pd router policy validation (#7904) by Simo Lin in https://github.com/sgl-project/sglang/pull/7904 - [router] Update metrics when request completes (#7899) by Shuaiyi Zhang in https://github.com/sgl-project/sglang/pull/7899 - [router] forward stream_options in request (#7860) by Shuaiyi Zhang in https://github.com/sgl-project/sglang/pull/7860 ### New Contributors * Shuaiyi Zhang made their first contribution in https://github.com/sgl-project/sglang/commit/51ae40306 * Charles Chen made their first contribution in https://github.com/sgl-project/sglang/commit/f3d973615 ### Paths Included - `sgl-router` - `python/sglang/srt/grpc` - `python/sglang/srt/entrypoints/grpc_server.py` **Full Changelog**: https://github.com/sgl-project/sglang/compare/gateway-v0.1.5...gateway-v0.1.6

## Highlights - Day 0 support for Kimi-K2-Thinking https://huggingface.co/moonshotai/Kimi-K2-Thinking - Q4 Roadmap: https://github.com/sgl-project/sglang/issues/12780 - Multimodal-gen support https://github.com/sgl-project/sglang/pull/12484 - Blackwell kernel optimizations and MoE runner backend refactor - Overlap spec and prefill cuda graph support more models ## What's Changed * [8/n] decouple quantization impl from vllm dependency - gguf srt by @FlamingoPg in https://github.com/sgl-project/sglang/pull/11964 * lang: support direct video inference by @mickqian in https://github.com/sgl-project/sglang/pull/9936 * Enable Llama 4 + TRTLLM MHA by @b8zhong in https://github.com/sgl-project/sglang/pull/12003 * Refactor Triton-kernel MoE runner integration by @Jonahcb in https://github.com/sgl-project/sglang/pull/11795 * use flashinfer_trtllm moe runner backend to gain around 10% perf on b200 fp8 dpsk by @b8zhong in https://github.com/sgl-project/sglang/pull/11816 * Fix(security): block unsafe pickle deserialization to mitigate CVE-2025-10164 by @thelongestusernameofall in https://github.com/sgl-project/sglang/pull/11909 * Revert "lang: support direct video inference" by @merrymercy in https://github.com/sgl-project/sglang/pull/12038 * support more model in piecewise cuda graph by @narutolhy in https://github.com/sgl-project/sglang/pull/11745 * [Fix] Fix lint to pass CI by @Fridge003 in https://github.com/sgl-project/sglang/pull/12037 * Revert "[Fix] Fix lint to pass CI" by @Fridge003 in https://github.com/sgl-project/sglang/pull/12042 * fix: fix MMMU loading issue by @ZailiWang in https://github.com/sgl-project/sglang/pull/11759 * Opt MHA chunked prefix: merge prefix and extend kv cache to run mha once by @xu-yfei in https://github.com/sgl-project/sglang/pull/10953 * Add gguf dependency for cpu/xpu by @ZailiWang in https://github.com/sgl-project/sglang/pull/12041 * fix: the hardcode hf repo name comparison for deepseek-ocr by @rainj-me in https://github.com/sgl-project/sglang/pull/12031 * Install numactl in Dockerfile for GH200/GB200/GB300 by @fzyzcjy in https://github.com/sgl-project/sglang/pull/11853 * [router] Add mTLS Support for Router-to-Worker Communication by @slin1237 in https://github.com/sgl-project/sglang/pull/12019 * Tiny cleanup send_single by @fzyzcjy in https://github.com/sgl-project/sglang/pull/12056 * Refactoring GLM-4.5 and GLM-4.5V related implementations by @zRzRzRzRzRzRzR in https://github.com/sgl-project/sglang/pull/11800 * [Fix] fix missing `ipc_name` of `__getitem__` in some IO structs by @whybeyoung in https://github.com/sgl-project/sglang/pull/12053 * fix: bench_serving ITL calculation when using spec-decoding by @JustinTong0323 in https://github.com/sgl-project/sglang/pull/12064 * Fix dpsk-r1-fp4 launching crash by @Qiaolin-Yu in https://github.com/sgl-project/sglang/pull/12063 * Revise POINTSV15Chat model by @yuan-luo in https://github.com/sgl-project/sglang/pull/12049 * Add 'gguf' to project dependencies by @Muqi1029 in https://github.com/sgl-project/sglang/pull/12046 * [Profiler] expand '~' by @Muqi1029 in https://github.com/sgl-project/sglang/pull/11999 * [b200] fix piecewise cuda graph launch bug by @BBuf in https://github.com/sgl-project/sglang/pull/12067 * Fix multi processing serializer bug by @fzyzcjy in https://github.com/sgl-project/sglang/pull/11958 * [Fix]: HiCache hasher failed when EAGLE mode enabled by @leavelet in https://github.com/sgl-project/sglang/pull/12025 * adjust dynamic vs static outputs comparison in test_lora_update.py by @glenliu21 in https://github.com/sgl-project/sglang/pull/11884 * [router] implement response api get input item function and refactor input/output store by @key4ng in https://github.com/sgl-project/sglang/pull/11924 * fix(compile_utils, ep_moe): update environment variable and dtype check by @ishandhanani in https://github.com/sgl-project/sglang/pull/12034 * [router] fix ut router config init to use build pattern by @slin1237 in https://github.com/sgl-project/sglang/pull/12084 * docs(server-arguments): add allowed options for each argument by @Jonahcb in https://github.com/sgl-project/sglang/pull/11560 * [router] migrate app context to builder pattern 1/n by @slin1237 in https://github.com/sgl-project/sglang/pull/12086 * [router] migrate app context to builder pattern 2/n by @slin1237 in https://github.com/sgl-project/sglang/pull/12089 * [router][grpc] Remove gpt_oss parsers and remove _parser suffix in tool parser files by @CatherineSue in https://github.com/sgl-project/sglang/pull/12091 * [1/2] deepseek deterministic: support deterministic inference for deepseek arch models on a single GPU by @zminglei in https://github.com/sgl-project/sglang/pull/12000 * Fix: Update blog link by @LucaLow in https://github.com/sgl-project/sglang/pull/12071 * perf: trtllm_mla attention backend spec decoding speedup w/ cuda graph by @cicirori in https://github.com/sgl-project/sglang/pull/12093 * [2/N]Support DeepSeek-R1 w4a8 low latency deepep by @ayrnb in https://github.com/sgl-project/sglang/pull/8464 * Enhance tests in deterministic kernels by @fzyzcjy in https://github.com/sgl-project/sglang/pull/12070 * [Doc] Add documentation for DeepSeek V3.2 by @Fridge003 in https://github.com/sgl-project/sglang/pull/11877 * [10/N] MoE Refactor: reorganize deepgemm runner in DeepEPMoE by @ch-wan in https://github.com/sgl-project/sglang/pull/12054 * Support true on-policy by @fzyzcjy in https://github.com/sgl-project/sglang/pull/12058 * [Docs] update sgl-kernel readme by @FlamingoPg in https://github.com/sgl-project/sglang/pull/11379 * Fix 'KeyError' for per_token expert distribution recorder by @vipwangerxiao in https://github.com/sgl-project/sglang/pull/9501 * Fix kernel version bump file by @Kangyan-Zhou in https://github.com/sgl-project/sglang/pull/12087 * [Fix] Set global args in cpu test by @Fridge003 in https://github.com/sgl-project/sglang/pull/12105 * chore: bump sgl-kernel version to 0.3.16.post4 by @sglang-bot in https://github.com/sgl-project/sglang/pull/12103 * [Auto Sync] Update test_deterministic.py, test_deterministi... (20251024) by @merrymercy in https://github.com/sgl-project/sglang/pull/12083 * [router] Refactor data connector architecture with unified storage modules by @key4ng in https://github.com/sgl-project/sglang/pull/12096 * fix: release workflow should work on both archs by @ishandhanani in https://github.com/sgl-project/sglang/pull/12110 * [bugs] docker file name should be <name>.Dockerfile so it can properly render by @slin1237 in https://github.com/sgl-project/sglang/pull/11869 * Clean up server args & Add CI scripts by @merrymercy in https://github.com/sgl-project/sglang/pull/12124 * [Misc] Improve the error message of failed import by @DarkSharpness in https://github.com/sgl-project/sglang/pull/12119 * [CI] Add ci monitor balance workflow by @BBuf in https://github.com/sgl-project/sglang/pull/11962 * Skip TestLlama4LoRA in CI by @lifuhuang in https://github.com/sgl-project/sglang/pull/12098 * clean up github tokens by @merrymercy in https://github.com/sgl-project/sglang/pull/12126 * Fix Illegal Instruction/IMA errors when using DP attention -- num_tokens_for_logprob calculation by @YAMY1234 in https://github.com/sgl-project/sglang/pull/12115 * Fix token for CI monitor by @merrymercy in https://github.com/sgl-project/sglang/pull/12127 * Reenable b200 tests by @Kangyan-Zhou in https://github.com/sgl-project/sglang/pull/11814 * Update document index for DeepSeek-v32 docs by @Fridge003 in https://github.com/sgl-project/sglang/pull/12101 * Update sgl-kernel version to 0.3.16.post4 by @Fridge003 in https://github.com/sgl-project/sglang/pull/12125 * [Doc] Fix format for deepseek v3.2 document by @Fridge003 in https://github.com/sgl-project/sglang/pull/12130 * Accelerate deepseek fp4 b200 ci by @Qiaolin-Yu in https://github.com/sgl-project/sglang/pull/11993 * Clean up server launch code and multi tokenizer by @merrymercy in https://github.com/sgl-project/sglang/pull/12132 * [Test] Add dsv3.2 nsa backend testing by @Johnsonms in https://github.com/sgl-project/sglang/pull/11936 * [docs] upd docker files names everywhere by @vincentzed in https://github.com/sgl-project/sglang/pull/12133 * Make bmm batch invariant injection optional by @fzyzcjy in https://github.com/sgl-project/sglang/pull/12118 * [Doc] Small update of DeepSeek v3.2 document by @Fridge003 in https://github.com/sgl-project/sglang/pull/12138 * docs: update README by @zhyncs in https://github.com/sgl-project/sglang/pull/12139 * [router] MCP Manager - Support Connection Pooling, Tool Inventory and Proxy by @slin1237 in https://github.com/sgl-project/sglang/pull/12097 * [NVIDIA] Change default quant method for model_opt by @kaixih in https://github.com/sgl-project/sglang/pull/11991 * [router] update smg code owners for each component by @slin1237 in https://github.com/sgl-project/sglang/pull/12141 * [router] cleaned up all the redundant comments in the config module by @CatherineSue in https://github.com/sgl-project/sglang/pull/12147 * Clean up attention backend selection code & Other minor rename by @merrymercy in https://github.com/sgl-project/sglang/pull/12136 * [log] Make forward iter count optional by @hnyls2002 in https://github.com/sgl-project/sglang/pull/12116 * [misc] depdencies & enviroment flag by @hnyls2002 in https://github.com/sgl-project/sglang/pull/12113 * [quantization] AWQ Marlin doesn't work when dtype is bfloat16 by @kevin85421 in https://github.com/sgl-project/sglang/pull/11494 * [HiCache]Page head layout IO kernel by @huangtingwei9988 in https://github.com/sgl-project/sglang/pull/11615 * Do not use `MagicMock` to mock `server_args` in tests by @hnyls2002 in https://github.com/sgl-project/sglang/pull/12154 * [router][grpc] Fix tool call id in `parse_json_schema_response` by @CatherineSue in https://github.com/sgl-project/sglang/pull/12152 * [router] centralize mcp tool args handling by @slin1237 in https://github.com/sgl-project/sglang/pull/12155 * Fix ITL metrics when using openai endpoint with spec by @hnyls2002 in https://github.com/sgl-project/sglang/pull/12156 * [Fix] fix allreduce bug in Piecewise Graph by @zyksir in https://github.com/sgl-project/sglang/pull/12106 * Support DeepGEMM for deterministic inference by @fzyzcjy in https://github.com/sgl-project/sglang/pull/12142 * model: support NVILA and NVILA Lite by @futrime in https://github.com/sgl-project/sglang/pull/10399 * Avoid using flashinfer_allreduce_fusion when dp attention is enabled. by @elfiegg in https://github.com/sgl-project/sglang/pull/11632 * transfer mrope_position_delta to device when first running by @ash-sigh in https://github.com/sgl-project/sglang/pull/11047 * add gitignore for claude code and serena mcp by @slin1237 in https://github.com/sgl-project/sglang/pull/12166 * Support MiniMax M2 model by @zhaochenyang20 in https://github.com/sgl-project/sglang/pull/12129 * [misc][grpc] Remove duplicate log by @CatherineSue in https://github.com/sgl-project/sglang/pull/12168 * [router][grpc] Add `ResponsesContext` and fix error propagation in responses api by @CatherineSue in https://github.com/sgl-project/sglang/pull/12164 * [router] Remove SharedXxxStorage type aliases to make Arc explicit by @CatherineSue in https://github.com/sgl-project/sglang/pull/12171 * Remove deprecated --enable-beta-spec argument and fix b200 test by @Kangyan-Zhou in https://github.com/sgl-project/sglang/pull/12167 * fix broken deepep/flashmla install in container by adding `--no-build-isolation` by @ishandhanani in https://github.com/sgl-project/sglang/pull/12170 * Remove description for `--enable-beta-spec` argument by @JustinTong0323 in https://github.com/sgl-project/sglang/pull/12177 * chore: bump SGLang version to 0.5.4.post1 by @sglang-bot in https://github.com/sgl-project/sglang/pull/12169 * [doc] add example of using w4fp8 for Deepseek by @Kevin-XiongC in https://github.com/sgl-project/sglang/pull/12057 * [sgl-route] Optimize the use of constant slices and retain to simplif… by @lengrongfu in https://github.com/sgl-project/sglang/pull/12159 * [Fix] Fix cu130 sgl-kernel wheel renaming by @Fridge003 in https://github.com/sgl-project/sglang/pull/12173 * docs: update contact by @zhyncs in https://github.com/sgl-project/sglang/pull/12192 * [sgl-kernel] feat: Support sm120 cutlass fp8 gemm kernel by @kaln27 in https://github.com/sgl-project/sglang/pull/9403 * [sgl-kernel][4/N]Support Expert Specialization Grouped GEMM by @HydraQYH in https://github.com/sgl-project/sglang/pull/12080 * GLM-4-0414 and GLM-4.1V Code Refactor by @zRzRzRzRzRzRzR in https://github.com/sgl-project/sglang/pull/12117 * Add support for AutoRound quantized models by @WeiweiZhang1 in https://github.com/sgl-project/sglang/pull/10153 * Optimize triton_mrope with torch compile by @yuan-luo in https://github.com/sgl-project/sglang/pull/12112 * Fix crash after flush cache by @cctry in https://github.com/sgl-project/sglang/pull/12107 * [Detokenizer Manager] Cleanup state when reqs are finished by @Muqi1029 in https://github.com/sgl-project/sglang/pull/12205 * fix(metrics): double times add_latency for DECODE_BOOTSTRAP by @jinmingyi1998 in https://github.com/sgl-project/sglang/pull/12209 * improve mimax-m2 rmsnorm precision by @haichao592 in https://github.com/sgl-project/sglang/pull/12186 * check_offload_progress more frequently by @pansicheng in https://github.com/sgl-project/sglang/pull/11656 * [Feature] PD-Multiplexing Context and Scheduler. by @ykcombat in https://github.com/sgl-project/sglang/pull/11592 * rope xpu: fix missing argument 'fused_set_kv_buffer_arg' and replace native with sgl_kernel_xpu impl by @chunyuan-w in https://github.com/sgl-project/sglang/pull/12006 * Add support for Matryoshka embeddings (#126) by @satyamk7054 in https://github.com/sgl-project/sglang/pull/11142 * fix: AttributeError: 'NixlKVManager' object has no attribute 'prefill_tp_size_table' by @gongwei-130 in https://github.com/sgl-project/sglang/pull/12234 * Compiling rope while preserving true on policy by @fzyzcjy in https://github.com/sgl-project/sglang/pull/12161 * [Auto Sync] Update scheduler.py, spec_info.py, run_suite.py... (20251027) by @zhyncs in https://github.com/sgl-project/sglang/pull/12235 * Support running FP4 Deepseek on SM120. by @weireweire in https://github.com/sgl-project/sglang/pull/11708 * Add env var to control custom Triton kernel cache and set CSGMV as default backend. by @lifuhuang in https://github.com/sgl-project/sglang/pull/12176 * Use explicit uint64 dtype for Tensor data_ptr() to avoid overflow by @jianan-gu in https://github.com/sgl-project/sglang/pull/11994 * Update openai package version to 2.6.1 by @JustinTong0323 in https://github.com/sgl-project/sglang/pull/12222 * [2/2] Use moe_sum_reduce cuda kernel by @yuan-luo in https://github.com/sgl-project/sglang/pull/10654 * docker: add CUDA13 support in dockerfile and update GDRCopy/NVSHMEM for blackwell support by @ishandhanani in https://github.com/sgl-project/sglang/pull/11517 * [router] remove code duplication by @slin1237 in https://github.com/sgl-project/sglang/pull/12245 * [DeepseekV32] Enable flashmla_prefill kernel with fp8 kvcache by @hlu1 in https://github.com/sgl-project/sglang/pull/11655 * Add per-request retraction count by @scottjlee in https://github.com/sgl-project/sglang/pull/11177 * Opt fused triton moe: add tma for down proj kernel by @xu-yfei in https://github.com/sgl-project/sglang/pull/10567 * Support releasing CUDA graph memory when paused by @fzyzcjy in https://github.com/sgl-project/sglang/pull/7873 * [router] use mcp struct from sdk and clean up code across codebase by @slin1237 in https://github.com/sgl-project/sglang/pull/12249 * [router] configure workflow retries and timeout based on routerConfig by @slin1237 in https://github.com/sgl-project/sglang/pull/12252 * Feature/Add GET endpoint to query loaded LoRA adapters by @ConnorLi96 in https://github.com/sgl-project/sglang/pull/12229 * [hotfix] Incorrect CombineOverlapArgs in SBO by @ch-wan in https://github.com/sgl-project/sglang/pull/12230 * [Feature] Sglang Tracing: Fine-Grained Tracking for Request Latency - Part 2 by @sufeng-buaa in https://github.com/sgl-project/sglang/pull/10804 * [Bug fix] [PP] fix wrong dtype for quantified model by @XucSh in https://github.com/sgl-project/sglang/pull/12247 * Fix potential eos bug on decode instance when PD is enabled by @ShangmingCai in https://github.com/sgl-project/sglang/pull/12206 * Revert "[Feature] PD-Multiplexing Context and Scheduler." by @zhyncs in https://github.com/sgl-project/sglang/pull/12267 * chore: cleanup quant deps by @zhyncs in https://github.com/sgl-project/sglang/pull/12268 * [router] Fix type unmatch during validation by @key4ng in https://github.com/sgl-project/sglang/pull/12257 * Modify rocm.Dockerfile by @sogalin in https://github.com/sgl-project/sglang/pull/12274 * [router] upgrade grpc dependency and py 3.13 3.14 support by @slin1237 in https://github.com/sgl-project/sglang/pull/12284 * Fix 'BypassedTopKOutput' object has no attribute 'topk_weights' for DeepEP by @trevor-m in https://github.com/sgl-project/sglang/pull/12231 * Tiny fix sgl-kernel related CI installing the wrong binary by @fzyzcjy in https://github.com/sgl-project/sglang/pull/12283 * doc for logit_bias by @whybeyoung in https://github.com/sgl-project/sglang/pull/12188 * Use Flashinfer TRT-LLM as Llama 4 compatible MoE backend by @b8zhong in https://github.com/sgl-project/sglang/pull/11928 * [rust][ci] Add end-to-end tests for Oracle history backend by @key4ng in https://github.com/sgl-project/sglang/pull/12233 * [router] support arm, windows, mac, linux, reduce wheel size and number by @slin1237 in https://github.com/sgl-project/sglang/pull/12285 * fix seqlen bug for trtllm_mla's draft_extend by @bmac3 in https://github.com/sgl-project/sglang/pull/12295 * Update deepseek_v32.md by @hlu1 in https://github.com/sgl-project/sglang/pull/12296 * Super tiny fix expert distribution dump error by @fzyzcjy in https://github.com/sgl-project/sglang/pull/12271 * [router][grpc] Fix inconsistent behavior of conversation_id not found by @CatherineSue in https://github.com/sgl-project/sglang/pull/12299 * fix: Llama 4 BF16 load on Blackwell by @b8zhong in https://github.com/sgl-project/sglang/pull/12308 * Add continuous_usage_stats support for streaming responses by @BBuf in https://github.com/sgl-project/sglang/pull/12241 * [hotfix] missing `w13_weight_fp8` and `w2_weight_fp8` in UE8M0 requantization by @ch-wan in https://github.com/sgl-project/sglang/pull/12259 * [hotfix] Fix pytest not found in CI by @Fridge003 in https://github.com/sgl-project/sglang/pull/12311 * a tiny fix for support deepseek bf16 weights by @Gao016 in https://github.com/sgl-project/sglang/pull/12313 * [metrics][EPLB]: Support selected count of physical experts on each GPU by @acelyc111 in https://github.com/sgl-project/sglang/pull/9825 * doc: improve modelopt error description by @lianakoleva in https://github.com/sgl-project/sglang/pull/12269 * EPLB: prefer to use physical experts in the same gpu or node by @acelyc111 in https://github.com/sgl-project/sglang/pull/10874 * Add Batch‑Invariant RMSNorm by @zyzshishui in https://github.com/sgl-project/sglang/pull/12144 * followup fix for llama 4 trtllm flashinfer backend by @b8zhong in https://github.com/sgl-project/sglang/pull/12314 * [Deepseek V3.2] Enable flashmla_auto with MTP by @hlu1 in https://github.com/sgl-project/sglang/pull/12294 * feat: preview filename from tuning_fused_moe_triton.py by @lianakoleva in https://github.com/sgl-project/sglang/pull/12276 * [ci] Try fixing broken CIs by @Fridge003 in https://github.com/sgl-project/sglang/pull/12317 * Refactor abortion in event loop by @hnyls2002 in https://github.com/sgl-project/sglang/pull/12312 * [Test] Fix session control test by @hnyls2002 in https://github.com/sgl-project/sglang/pull/12336 * Eagle3 DP attention for Qwen3 MoE by @qhsc in https://github.com/sgl-project/sglang/pull/12002 * feat: return partial generation results when aborting requests in waiting queue by @guoyuhong in https://github.com/sgl-project/sglang/pull/11673 * [Bug fix] trace: fix import error in mini_lb if sgl-router image does not install sglang by @sufeng-buaa in https://github.com/sgl-project/sglang/pull/12338 * [router] fix router release workflow and add build test in PR by @CatherineSue in https://github.com/sgl-project/sglang/pull/12315 * Triton fused_moe_kernel support ep moe tuning by @BBuf in https://github.com/sgl-project/sglang/pull/12343 * [Fix] fix type issue of env flag value MODELOPT_MAX_TOKENS_PER_EXPERT by @zejunchen-zejun in https://github.com/sgl-project/sglang/pull/11709 * [bug] fix router pypi license file by @slin1237 in https://github.com/sgl-project/sglang/pull/12345 * fix: llama 4 + trtllm gen + fp8 kv cache incompatibility by @b8zhong in https://github.com/sgl-project/sglang/pull/12347 * [2/2] Deepseek deterministic: support deepseek v3 deterministic inference on 8 x H200 by @zminglei in https://github.com/sgl-project/sglang/pull/12095 * Fix Flashinfer Backend for SM120 Usage by @weireweire in https://github.com/sgl-project/sglang/pull/12325 * [router] refactor mcp to use LRU and fix pooling bug by @CatherineSue in https://github.com/sgl-project/sglang/pull/12346 * support cutlass fp4 kernel in sm120 by @AichenF in https://github.com/sgl-project/sglang/pull/11737 * [bug] fix router installation to include additional dependency by @slin1237 in https://github.com/sgl-project/sglang/pull/12348 * [router] update router docker to use maturin and build from local by @CatherineSue in https://github.com/sgl-project/sglang/pull/12350 * Fix Duplicate Classmethod in spec_info.py by @hebiao064 in https://github.com/sgl-project/sglang/pull/12354 * [CI] Add Llama 3.1 8B FP4 to B200 CI by @b8zhong in https://github.com/sgl-project/sglang/pull/12182 * Fuse wk and weight_proj in Indexer for DeepSeekV3.2-FP4 by @trevor-m in https://github.com/sgl-project/sglang/pull/12094 * [router] Harmony Pipeline: Chat Completion & Responses API with MCP Support by @slin1237 in https://github.com/sgl-project/sglang/pull/12153 * [bugfix] fix deepseekvl2 and deepseek_ocr model type conflict by @leihuang-sketch in https://github.com/sgl-project/sglang/pull/12050 * [Ckpt Engine] feat: new sglang entrypoint support for update by @stmatengss in https://github.com/sgl-project/sglang/pull/12216 * [Perf] Optimize multimodal mm_inputs process in scheduler by @yuan-luo in https://github.com/sgl-project/sglang/pull/11910 * [NPU] fix pp_size>1 by @Makcum888e in https://github.com/sgl-project/sglang/pull/12195 * Super tiny add tag for benchmark scripts by @fzyzcjy in https://github.com/sgl-project/sglang/pull/12340 * Allow benchmarking tool to handle empty response by @Kangyan-Zhou in https://github.com/sgl-project/sglang/pull/12174 * Super tiny fix AMD ci by @fzyzcjy in https://github.com/sgl-project/sglang/pull/12378 * Import flash_mla from sgl-kernel by @Fridge003 in https://github.com/sgl-project/sglang/pull/12135 * [Bug fix][PP] fix deadlock with tie_word_embeddings by @XucSh in https://github.com/sgl-project/sglang/pull/12362 * [fix] added image token as prefix for deepseek-ocr by @Tushar-ml in https://github.com/sgl-project/sglang/pull/12358 * Fix DeepSeek chat templates to handle tool call arguments type checking (#11700) by @Kangyan-Zhou in https://github.com/sgl-project/sglang/pull/12123 * [Feature] Initial eagle3 support for Deepseek-like models by @JensenFire in https://github.com/sgl-project/sglang/pull/12319 * Enable fast silu-and-mul-and-quant fused kernel by @fzyzcjy in https://github.com/sgl-project/sglang/pull/11806 * [Test] Enhance radix cache test for spec cases by @hnyls2002 in https://github.com/sgl-project/sglang/pull/12394 * [NPU] bugfix for Qwen3-Next and performance update by @iforgetmyname in https://github.com/sgl-project/sglang/pull/11969 * [Feature] Support DeepSeek MTP on NPU by @iforgetmyname in https://github.com/sgl-project/sglang/pull/11897 * Revert "Triton fused_moe_kernel support ep moe tuning" by @BBuf in https://github.com/sgl-project/sglang/pull/12377 * [sgl-kernel] upd deepgemm hash to rebased commit by @FlamingoPg in https://github.com/sgl-project/sglang/pull/11960 * [router] harmony responses api streaming support by @slin1237 in https://github.com/sgl-project/sglang/pull/12395 * [docker] clean up main dockerfile for router and dev configurations by @CatherineSue in https://github.com/sgl-project/sglang/pull/12364 * feat: add EP support in tuning by @Chen-0210 in https://github.com/sgl-project/sglang/pull/12012 * [router] use safety_identifier replace user on chat history storage by @lengrongfu in https://github.com/sgl-project/sglang/pull/12185 * [CI Monitor] Fix ci_monitor perf analyzer bug by @BBuf in https://github.com/sgl-project/sglang/pull/12281 * [router] Fix safety_identifier missing by @key4ng in https://github.com/sgl-project/sglang/pull/12404 * [ci] Fix ci_install_deepep by @Fridge003 in https://github.com/sgl-project/sglang/pull/12375 * Update news section in README.md by @merrymercy in https://github.com/sgl-project/sglang/pull/12409 * [router] Function call support for openai router Responses API by @key4ng in https://github.com/sgl-project/sglang/pull/12386 * minor code sync by @merrymercy in https://github.com/sgl-project/sglang/pull/12403 * [Bug fix][PD Dissaggregation] fix prefill hanging issue with PP and DP Attention, by @popsiclexu in https://github.com/sgl-project/sglang/pull/12368 * [NVIDIA] Add CI workloads for GB200 by @kaixih in https://github.com/sgl-project/sglang/pull/12242 * [router] web_search_preview tool basic implementation by @key4ng in https://github.com/sgl-project/sglang/pull/12290 * [router] 0.2.2 release by @slin1237 in https://github.com/sgl-project/sglang/pull/12399 * enable cudaProfilerApi for one batch benchmarking by @lpc0220 in https://github.com/sgl-project/sglang/pull/11116 * [Refactor] tuning_fused_moe for MLLM and small refactor by @JustinTong0323 in https://github.com/sgl-project/sglang/pull/11224 * [DeepSeekV32] Bug fix to ensure `page_table` and `result` in same type by @Johnsonms in https://github.com/sgl-project/sglang/pull/12300 * [CI] fix tests' time estimation by @hnyls2002 in https://github.com/sgl-project/sglang/pull/12401 * Reserved abortion API when retracting by @hnyls2002 in https://github.com/sgl-project/sglang/pull/12425 * Fix the shared expert & routed expert overlap in Llama 4 by @b8zhong in https://github.com/sgl-project/sglang/pull/12405 * feat: Add Non-intrusive Tensor Dumping for Model Inference by @guoyuhong in https://github.com/sgl-project/sglang/pull/10566 * feat: support trtllm_mha FP8 query attention kernel by @elvischenv in https://github.com/sgl-project/sglang/pull/12307 * [Bugfix]: distinguish processors for deepseek_vl2 and deepseek_ocr to p… by @bppps in https://github.com/sgl-project/sglang/pull/12384 * [ci] install released version router by @key4ng in https://github.com/sgl-project/sglang/pull/12410 * Revert "fix llama4 kv cache layout" by @b8zhong in https://github.com/sgl-project/sglang/pull/12437 * Add trait for `BasePrefixCache` by @hnyls2002 in https://github.com/sgl-project/sglang/pull/12436 * [CI] Add more bins for 1-gpu CI test by @Fridge003 in https://github.com/sgl-project/sglang/pull/12422 * [bugfix] set is_prefill_only=false when mixed_chunk by @Bruce-x-1997 in https://github.com/sgl-project/sglang/pull/10889 * Clean up sgl kernel by @merrymercy in https://github.com/sgl-project/sglang/pull/12413 * [CI] fix possible port conflicts. by @hnyls2002 in https://github.com/sgl-project/sglang/pull/12452 * Fix ci install to allow prerelease by @merrymercy in https://github.com/sgl-project/sglang/pull/12449 * fix: Add default value for backend in sample_mmmu_requests by @ZailiWang in https://github.com/sgl-project/sglang/pull/12256 * Enable bailing_moe to support TP=16 by @guoyuhong in https://github.com/sgl-project/sglang/pull/12369 * fix:watchdog thread exception by @Kindyaa in https://github.com/sgl-project/sglang/pull/12328 * Simplify watchdog by @hnyls2002 in https://github.com/sgl-project/sglang/pull/12463 * [Bug fix] Fix severe memory waste issue with torch.empty pin_memory by @sjtushenhai in https://github.com/sgl-project/sglang/pull/12266 * Feat: deepseek-ocr logits processor by @JustinTong0323 in https://github.com/sgl-project/sglang/pull/12415 * Fix lint in deepseek-ocr by @ispobock in https://github.com/sgl-project/sglang/pull/12470 * [Test] Add Functional Tests for Penalty Parameters by @neelabhsinha in https://github.com/sgl-project/sglang/pull/11931 * [Bug] OOM (Out-of-Memory) errors for extreme testing scenarios (min_tokens=2) by @LuYanFCP in https://github.com/sgl-project/sglang/pull/11757 * [Feature] PD-Multiplexing Context and Scheduler, lazy import spatial. by @ykcombat in https://github.com/sgl-project/sglang/pull/12275 * [VLM] Optimize async mm data process mechanism by @yuan-luo in https://github.com/sgl-project/sglang/pull/12066 * fix default env var for mooncake store by @huangtingwei9988 in https://github.com/sgl-project/sglang/pull/12429 * add served model name in bench serving by @carolove in https://github.com/sgl-project/sglang/pull/12428 * Tiny assert no running requests when releasing memory to avoid IMA by @fzyzcjy in https://github.com/sgl-project/sglang/pull/12341 * fix: dummy health check server not accessible on non-zero rank nodes by @ishandhanani in https://github.com/sgl-project/sglang/pull/12297 * Fix run benchmark by @ispobock in https://github.com/sgl-project/sglang/pull/12473 * Add env var to disable FA4 warmup by @cicirori in https://github.com/sgl-project/sglang/pull/12430 * Try to allow NCCL cumem for multi node nvlink case by @fzyzcjy in https://github.com/sgl-project/sglang/pull/11987 * Support Kimi Linear by @ispobock in https://github.com/sgl-project/sglang/pull/12469 * [CI] Fix kernel installation on aarch runners by @Fridge003 in https://github.com/sgl-project/sglang/pull/12475 * fa3 & trtllm_mha spec overlap by @JustinTong0323 in https://github.com/sgl-project/sglang/pull/11874 * chore: bump SGLang version to 0.5.4.post2 by @sglang-bot in https://github.com/sgl-project/sglang/pull/12439 * Tiny fix eos handling for PD disaggregation by @ShangmingCai in https://github.com/sgl-project/sglang/pull/12334 * Forward unknown tool calls instead of dropping by @Surya-Gunukula in https://github.com/sgl-project/sglang/pull/12226 * Use sgl fp4 quant kernel by default by @Qiaolin-Yu in https://github.com/sgl-project/sglang/pull/12482 * [hot fix] Remove `from python.sglang.xxx` by @hnyls2002 in https://github.com/sgl-project/sglang/pull/12483 * perf: trtllm mla performance minor improvements by @cicirori in https://github.com/sgl-project/sglang/pull/12435 * Filter tokenizer warning for kimi models by @ispobock in https://github.com/sgl-project/sglang/pull/12485 * [CI] Build aarch64 kernels for sgl-kernel test by @Fridge003 in https://github.com/sgl-project/sglang/pull/12480 * [Hotfix] Remove extra comment in sgl-kernel README by @Fridge003 in https://github.com/sgl-project/sglang/pull/12500 * [feat] Add SGLANG_TOOL_STRICT_LEVEL for tool-call behavior control by @JustinTong0323 in https://github.com/sgl-project/sglang/pull/12423 * Reduce docker image size. mount cache when use pip/cargo build by @whybeyoung in https://github.com/sgl-project/sglang/pull/12238 * [HICache / PD]: Support offloading incremental KV cache in decode side. by @hzh0425 in https://github.com/sgl-project/sglang/pull/11966 * [Deterministic] add deepseek v3 deterministic inference CI test by @zminglei in https://github.com/sgl-project/sglang/pull/12412 * [Bug] test_flashattn_mla_backend errors in Hopper #12487 by @Johnsonms in https://github.com/sgl-project/sglang/pull/12488 * Update Mooncake EP's a2a interface by @UNIDY2002 in https://github.com/sgl-project/sglang/pull/12391 * [CI][NPU] remove pypi mirror site that hangs ci dependency installation by @iforgetmyname in https://github.com/sgl-project/sglang/pull/12499 * [Ascend] Add Ascend NPU support for sglang.check_env & rework proposal by @Alexhaoge in https://github.com/sgl-project/sglang/pull/11052 * [Feature] Qwen3-Next & FLA: Support MTP topk>1; Up to 6% faster by @byjiang1996 in https://github.com/sgl-project/sglang/pull/11133 * [CI] Move some Lora/Deterministic CI tests to nightly by @Fridge003 in https://github.com/sgl-project/sglang/pull/12507 * Migrate weak_ref_tensor to sgl-kernel by @BBuf in https://github.com/sgl-project/sglang/pull/12505 * feat: Add FP4 (E2M1) KV Cache Support with Quantization Utilities for MLA by @JackChuang in https://github.com/sgl-project/sglang/pull/10078 * chore: bump sgl-kernel version to 0.3.16.post5 by @sglang-bot in https://github.com/sgl-project/sglang/pull/12511 * [FEAT] Shared mem pool based cuda ipc for multi-modal data transport by @kousakawang in https://github.com/sgl-project/sglang/pull/11917 * Add prefix for torch symm mem by @yuan-luo in https://github.com/sgl-project/sglang/pull/12506 * [ServerArgs] allow --mamba-ssm-dtype extend by @hanming-lu in https://github.com/sgl-project/sglang/pull/12481 * [Fix] `concat_mla_absorb_q_kernel` fails for long inputs by @bingps in https://github.com/sgl-project/sglang/pull/12453 * Super tiny fix naming in bench serving scripts by @fzyzcjy in https://github.com/sgl-project/sglang/pull/12515 * move all get_stream in sgl_kernel to c++ to reduce the launch overhead by @merrymercy in https://github.com/sgl-project/sglang/pull/12521 * [Refact] Remove hardcoded KV cache dimension in MLATokenToKVPool by @Johnsonms in https://github.com/sgl-project/sglang/pull/12502 * [Bug] Fix Intern-S1 model accuracy and support /generate interface with input_ids by @hhaAndroid in https://github.com/sgl-project/sglang/pull/12367 * chore: upgrade flashinfer 0.5.0 by @zhyncs in https://github.com/sgl-project/sglang/pull/12523 * [hotfix] Remove flashinfer-jit-cache from pyproject by @Fridge003 in https://github.com/sgl-project/sglang/pull/12530 * fix: move dummy format loader check before quantization checks by @cicirori in https://github.com/sgl-project/sglang/pull/12532 * chore: upgrade mooncake 0.3.7.post1 by @ShangmingCai in https://github.com/sgl-project/sglang/pull/12541 * fix: Fix KTransformers hybrid inference with int8 quantization and format by @Atream in https://github.com/sgl-project/sglang/pull/12536 * Conditionally recapture cuda graph after model weight update from disk by @harrisonlimh in https://github.com/sgl-project/sglang/pull/12060 * [spec v2] Fix output repetition by speculative sampling error by @hnyls2002 in https://github.com/sgl-project/sglang/pull/12561 * [hot-fix] Fix broken CI by @hnyls2002 in https://github.com/sgl-project/sglang/pull/12564 * fix: fix the bug which leads qwen2_5_vl to crash with mixed_chunk by @PanJason in https://github.com/sgl-project/sglang/pull/11330 * Fix error when calling quantization by @fzyzcjy in https://github.com/sgl-project/sglang/pull/12548 * [Test] Add parameters to SRTRunner by @Jonahcb in https://github.com/sgl-project/sglang/pull/12227 * [ROCm] Update Mooncake to v0.3.7.post1 and add -DUSE_HIP=ON to rocm.Dockerfile by @yeahdongcn in https://github.com/sgl-project/sglang/pull/12560 * Reduce the overhead of nccl symmetric memory by @merrymercy in https://github.com/sgl-project/sglang/pull/12524 * tiny optimize for bench serving by @yizhang2077 in https://github.com/sgl-project/sglang/pull/12553 * Super tiny allow profile activities in bench_serving by @fzyzcjy in https://github.com/sgl-project/sglang/pull/12549 * Super tiny dump server info such as args in bench for post analysis by @fzyzcjy in https://github.com/sgl-project/sglang/pull/12550 * update usage of `trtllm_fp8_per_tensor_scale_moe` by @b8zhong in https://github.com/sgl-project/sglang/pull/12569 * [router][grpc] Consolidate error messages build in error.rs by @CatherineSue in https://github.com/sgl-project/sglang/pull/12301 * Remove the dependency of nccl.h in symmetric memory by @merrymercy in https://github.com/sgl-project/sglang/pull/12571 * [chore] Fix update_kernel_whl_index script for multiple cuda version by @Fridge003 in https://github.com/sgl-project/sglang/pull/12519 * Enable mixed type LayerNorm kernel for NSA indexer by @akhilg-nv in https://github.com/sgl-project/sglang/pull/12044 * Super tiny add UT for copy_to_gpu_no_ce by @fzyzcjy in https://github.com/sgl-project/sglang/pull/12270 * [Doc] fix miss index for production request trace by @stmatengss in https://github.com/sgl-project/sglang/pull/12547 * [GDN/SWA] mamba and swa radix cache edge case fix by @hanming-lu in https://github.com/sgl-project/sglang/pull/12111 * [Qwen3 VL] Add LoRA support for Qwen 3 VL by @Jonahcb in https://github.com/sgl-project/sglang/pull/12165 * test: support return logprobs in bench_offline_throughput test by @aftersnow in https://github.com/sgl-project/sglang/pull/12462 * Tiny fix ExpertDistributionReq error by @fzyzcjy in https://github.com/sgl-project/sglang/pull/11760 * fix: respect `--ignore-eos` in PD case for benchmarking by @ishandhanani in https://github.com/sgl-project/sglang/pull/12597 * Improve the metrics for PD by @merrymercy in https://github.com/sgl-project/sglang/pull/12580 * Enable memory saver for hybrid model by @ocss884 in https://github.com/sgl-project/sglang/pull/11974 * Restore torch defaults between sgl-kernel tests by @benbarsdell in https://github.com/sgl-project/sglang/pull/11131 * feat: limit peak memory usage when computing logprobs by @aftersnow in https://github.com/sgl-project/sglang/pull/6318 * [router][grpc] Restructure modules and code clean up by @CatherineSue in https://github.com/sgl-project/sglang/pull/12598 * Add --speculative-moe-runner-backend server arg by @trevor-m in https://github.com/sgl-project/sglang/pull/10183 * [Deterministic] Optimize bmm_batch_invariant op by @zminglei in https://github.com/sgl-project/sglang/pull/12522 * chore: bump mooncake version to 0.3.7.post2 by @ShangmingCai in https://github.com/sgl-project/sglang/pull/12599 * [sepc-v2] Fix imcompatibility with constrained decoding by @hnyls2002 in https://github.com/sgl-project/sglang/pull/12615 * Support aggregating engine metrics in sgl-router by @fzyzcjy in https://github.com/sgl-project/sglang/pull/11456 * Ensure GPU work is finished when release memory occupation call is finished by @fzyzcjy in https://github.com/sgl-project/sglang/pull/12592 * Add sanity checks when a test file is not added to CI (reland) by @fzyzcjy in https://github.com/sgl-project/sglang/pull/12594 * [router][grpc] Fix model validation, tool call check, streaming logic and misc in responses by @CatherineSue in https://github.com/sgl-project/sglang/pull/12616 * [HotFix] Disable torch dynamo for mrope_triton kernel by @yuan-luo in https://github.com/sgl-project/sglang/pull/12593 * Fix skip layer in get_quant_method by @ispobock in https://github.com/sgl-project/sglang/pull/12632 * [Test] Merge all constrained decoding tests. by @hnyls2002 in https://github.com/sgl-project/sglang/pull/12633 * Add io struct naming check back by @hnyls2002 in https://github.com/sgl-project/sglang/pull/12634 * Fix `output_ids` inconsistency by @hnyls2002 in https://github.com/sgl-project/sglang/pull/12628 * fix: Lazy import mooncake-ep to fix extra gpu contexts being created by @trevor-m in https://github.com/sgl-project/sglang/pull/12641 * [hotfix] Fix deepep w4a8 bug by @Fridge003 in https://github.com/sgl-project/sglang/pull/12642 * [Auto Sync] Update scheduler_metrics_mixin.py, collector.py (20251104) by @merrymercy in https://github.com/sgl-project/sglang/pull/12647 * [Bug] Fix NSA Backend KV-Buffer Shape Mismatch in DeepSeek-V3.2 by @Johnsonms in https://github.com/sgl-project/sglang/pull/12645 * [NVIDIA] Fix wrong symmetric sizes for fp4 cases by @kaixih in https://github.com/sgl-project/sglang/pull/12640 * [router][grpc] Fix index issues in reasoning content and missing streaming events by @CatherineSue in https://github.com/sgl-project/sglang/pull/12650 * Revert "Enable memory saver for hybrid model" by @Fridge003 in https://github.com/sgl-project/sglang/pull/12648 * Add multi-GPU configurations to nightly-test.yml by @alisonshao in https://github.com/sgl-project/sglang/pull/12585 * [fix] Handle escaped characters in GLM tool call parser to prevent double serialization by @soaringk in https://github.com/sgl-project/sglang/pull/12456 * [router][grpc] Emit OutputItemDone event and store output item array by @CatherineSue in https://github.com/sgl-project/sglang/pull/12656 * Register allgather/reducescatter buffers with symm memory by @nvcastet in https://github.com/sgl-project/sglang/pull/12572 * chore: bump SGLang version to 0.5.4.post3 by @sglang-bot in https://github.com/sgl-project/sglang/pull/12639 * [NVIDIA] Fix cutedsl backend of MoE by @kaixih in https://github.com/sgl-project/sglang/pull/12353 * [PD-Disagg] Check finish after pop tranferred by @hnyls2002 in https://github.com/sgl-project/sglang/pull/12638 * fix typo of args description in sglang.profiler by @ai-easy-cpu in https://github.com/sgl-project/sglang/pull/12486 * [Dockerfile] Speed up docker image building by @acelyc111 in https://github.com/sgl-project/sglang/pull/8784 * Fix VLLM dependency test by @Kangyan-Zhou in https://github.com/sgl-project/sglang/pull/12670 * [Feature] add --lora-request-distribution arg to bench_serving.py and support skewed and distinct workloads by @glenliu21 in https://github.com/sgl-project/sglang/pull/12175 * [router][grpc] Implement tool_choice support for Responses API by @CatherineSue in https://github.com/sgl-project/sglang/pull/12668 * Expand and update test coverage for AMD CI by @hubertlu-tw in https://github.com/sgl-project/sglang/pull/10044 * fix: add `seed` bench_serving to cache key, remove redundant function definition. by @cicirori in https://github.com/sgl-project/sglang/pull/12680 * [Profiler] Add SGLANG_PROFILE_RECORD_SHAPES for recording shapes when profiling by @zejunchen-zejun in https://github.com/sgl-project/sglang/pull/11641 * fix trtllm_mla attention backend when disabling cuda graph. by @cicirori in https://github.com/sgl-project/sglang/pull/12687 * Refactor `--debug-tensor-dump-layers` to list by @guoyuhong in https://github.com/sgl-project/sglang/pull/12691 * [Grammar Fix] GLM-4-MOE self.first_k_dense_replace is undefined. by @zRzRzRzRzRzRzR in https://github.com/sgl-project/sglang/pull/12455 * add Kimi k2 reasoning parser by @MoyanZitto in https://github.com/sgl-project/sglang/pull/12702 * Commented out b200 tests due to runner shortage by @Kangyan-Zhou in https://github.com/sgl-project/sglang/pull/12609 * [CI] Fix qwen3-vl lora nightly ci by @Fridge003 in https://github.com/sgl-project/sglang/pull/12708 * Fix server args for gpt oss so users can override the moe runner backend by @merrymercy in https://github.com/sgl-project/sglang/pull/12696 * [router][grpc] Support streaming parsing with Tool Choice in chat completions API by @CatherineSue in https://github.com/sgl-project/sglang/pull/12677 * feat: initial multimodal-gen support by @mickqian in https://github.com/sgl-project/sglang/pull/12484 * Enable Aiter Attention for VL model by @Yuechguo in https://github.com/sgl-project/sglang/pull/12699 * [router] fix: validate HTTP status codes in health check by @wyx-0203 in https://github.com/sgl-project/sglang/pull/12631 * Support Expert Deferral Mechanism in KTransformers by @Atream in https://github.com/sgl-project/sglang/pull/12586 * Add mm_fp4 trtllm backend by @wenscarl in https://github.com/sgl-project/sglang/pull/12406 * [NVIDIA] Fix unit test of MoE and add it to nightly ci by @kaixih in https://github.com/sgl-project/sglang/pull/12709 * [misc] Add labeler for automatic labeling by @CatherineSue in https://github.com/sgl-project/sglang/pull/12710 * [router][ci] speed up python binding to 1.5 min by @key4ng in https://github.com/sgl-project/sglang/pull/12673 * Fix CI and style by @merrymercy in https://github.com/sgl-project/sglang/pull/12658 * Revert "Commented out b200 tests due to runner shortage (#12609)" by @Kangyan-Zhou in https://github.com/sgl-project/sglang/pull/12712 * [misc] Change sync-labels to false by @CatherineSue in https://github.com/sgl-project/sglang/pull/12714 * [router][grpc] Make harmony parser checks recipient first before channel by @CatherineSue in https://github.com/sgl-project/sglang/pull/12713 * [router][quick fix] Add minimal option for reasoning effort in spec by @key4ng in https://github.com/sgl-project/sglang/pull/12711 * [router] add basic ci tests for gpt-oss model support by @key4ng in https://github.com/sgl-project/sglang/pull/12651 * fix labeler by @key4ng in https://github.com/sgl-project/sglang/pull/12718 * [ci] fix permission by @key4ng in https://github.com/sgl-project/sglang/pull/12729 * [chore]Remove dockerfile from target file of bump kernel version by @Fridge003 in https://github.com/sgl-project/sglang/pull/12728 * [CPU] Upgrade default PT version to 2.9 by @ZailiWang in https://github.com/sgl-project/sglang/pull/12611 * Revert "[ci] fix permission" by @key4ng in https://github.com/sgl-project/sglang/pull/12732 * Revert "[router] web_search_preview tool basic implementation" by @key4ng in https://github.com/sgl-project/sglang/pull/12716 * fix sgl-kernel version by @gongwei-130 in https://github.com/sgl-project/sglang/pull/12723 * [chore] SGLang tag management in Dockerfile by @Fridge003 in https://github.com/sgl-project/sglang/pull/12734 * Add nightly test multi gpu configs by @alisonshao in https://github.com/sgl-project/sglang/pull/12721 * DeepSeek-V3.2: Add Adaptive MHA Attention Pathway for Short-Sequence Prefill by @YAMY1234 in https://github.com/sgl-project/sglang/pull/11892 * Temporarily fix missing routed_scaling_factor for CompressedTensorsWNA16MoEMethod by @Atream in https://github.com/sgl-project/sglang/pull/12738 * [chore] Fix triton installation for cu13 image by @Fridge003 in https://github.com/sgl-project/sglang/pull/12742 * keep attention backend document up to date by @b8zhong in https://github.com/sgl-project/sglang/pull/12741 * [Fix]Tiny fix in Dockerfile by @Fridge003 in https://github.com/sgl-project/sglang/pull/12748 * [router][grpc] Support mixin tool calls in Responses API by @CatherineSue in https://github.com/sgl-project/sglang/pull/12736 * fix: tiny fix cli by @mickqian in https://github.com/sgl-project/sglang/pull/12744 * [router][ci] Disable cache by @key4ng in https://github.com/sgl-project/sglang/pull/12752 * fix mamba prefix cache leak caused by abort by @yizhang2077 in https://github.com/sgl-project/sglang/pull/12693 * [BUGFIX] fix output_ids in abort by @yizhang2077 in https://github.com/sgl-project/sglang/pull/12737 * [GDN] Fuse b.sigmoid(), fused_gdn_gating and unsqueeze into one kernel: up to 0.85% e2e speedup by @byjiang1996 in https://github.com/sgl-project/sglang/pull/12508 * [VLM] Optimize qwen_vl preprocess_video by @yuan-luo in https://github.com/sgl-project/sglang/pull/12240 * Add timing metrics for requests by @cicirori in https://github.com/sgl-project/sglang/pull/12646 * fix qwen3-omni audio length < 30s by @jiapingW in https://github.com/sgl-project/sglang/pull/12674 * docs: document video-capable multimodal models by @WazupSteve in https://github.com/sgl-project/sglang/pull/12565 * fix ci by @key4ng in https://github.com/sgl-project/sglang/pull/12760 * [Refactor] Refactor fused_moe_triton tuning tools: extract shared utils, add EP/MLLM support, reduce overhead by @BBuf in https://github.com/sgl-project/sglang/pull/12440 * Update dsv3 quantization auto setting for sm100 by @ispobock in https://github.com/sgl-project/sglang/pull/12778 * chore: bump SGLang version to 0.5.5 by @sglang-bot in https://github.com/sgl-project/sglang/pull/12739 ## New Contributors * @thelongestusernameofall made their first contribution in https://github.com/sgl-project/sglang/pull/11909 * @LucaLow made their first contribution in https://github.com/sgl-project/sglang/pull/12071 * @vipwangerxiao made their first contribution in https://github.com/sgl-project/sglang/pull/9501 * @Johnsonms made their first contribution in https://github.com/sgl-project/sglang/pull/11936 * @ash-sigh made their first contribution in https://github.com/sgl-project/sglang/pull/11047 * @Kevin-XiongC made their first contribution in https://github.com/sgl-project/sglang/pull/12057 * @kaln27 made their first contribution in https://github.com/sgl-project/sglang/pull/9403 * @haichao592 made their first contribution in https://github.com/sgl-project/sglang/pull/12186 * @satyamk7054 made their first contribution in https://github.com/sgl-project/sglang/pull/11142 * @weireweire made their first contribution in https://github.com/sgl-project/sglang/pull/11708 * @bmac3 made their first contribution in https://github.com/sgl-project/sglang/pull/12295 * @Gao016 made their first contribution in https://github.com/sgl-project/sglang/pull/12313 * @lianakoleva made their first contribution in https://github.com/sgl-project/sglang/pull/12269 * @zyzshishui made their first contribution in https://github.com/sgl-project/sglang/pull/12144 * @zejunchen-zejun made their first contribution in https://github.com/sgl-project/sglang/pull/11709 * @AichenF made their first contribution in https://github.com/sgl-project/sglang/pull/11737 * @JensenFire made their first contribution in https://github.com/sgl-project/sglang/pull/12319 * @Chen-0210 made their first contribution in https://github.com/sgl-project/sglang/pull/12012 * @popsiclexu made their first contribution in https://github.com/sgl-project/sglang/pull/12368 * @lpc0220 made their first contribution in https://github.com/sgl-project/sglang/pull/11116 * @elvischenv made their first contribution in https://github.com/sgl-project/sglang/pull/12307 * @sjtushenhai made their first contribution in https://github.com/sgl-project/sglang/pull/12266 * @LuYanFCP made their first contribution in https://github.com/sgl-project/sglang/pull/11757 * @carolove made their first contribution in https://github.com/sgl-project/sglang/pull/12428 * @Surya-Gunukula made their first contribution in https://github.com/sgl-project/sglang/pull/12226 * @Alexhaoge made their first contribution in https://github.com/sgl-project/sglang/pull/11052 * @JackChuang made their first contribution in https://github.com/sgl-project/sglang/pull/10078 * @bingps made their first contribution in https://github.com/sgl-project/sglang/pull/12453 * @hhaAndroid made their first contribution in https://github.com/sgl-project/sglang/pull/12367 * @yeahdongcn made their first contribution in https://github.com/sgl-project/sglang/pull/12560 * @akhilg-nv made their first contribution in https://github.com/sgl-project/sglang/pull/12044 * @alisonshao made their first contribution in https://github.com/sgl-project/sglang/pull/12585 * @soaringk made their first contribution in https://github.com/sgl-project/sglang/pull/12456 * @ai-easy-cpu made their first contribution in https://github.com/sgl-project/sglang/pull/12486 * @MoyanZitto made their first contribution in https://github.com/sgl-project/sglang/pull/12702 * @wyx-0203 made their first contribution in https://github.com/sgl-project/sglang/pull/12631 * @WazupSteve made their first contribution in https://github.com/sgl-project/sglang/pull/12565 **Full Changelog**: https://github.com/sgl-project/sglang/compare/v0.5.4...v0.5.5

## Highlights - Model gateway v0.2 release: https://docs.sglang.ai/advanced_features/router.html - [beta] Overlap scheduler for speculative decoding: https://github.com/sgl-project/sglang/issues/11762 - [beta] Piecewise CUDA graph for prefill: https://github.com/sgl-project/sglang/issues/11490 - Prefix cache for qwen3 next and GDN/mamba models: https://github.com/sgl-project/sglang/pull/11214 - Fullset optimizations for DeepSeek-V3.2 (MTP, PD-Disagg, Function Calling) (https://docs.sglang.ai/basic_usage/deepseek_v32.html, https://github.com/sgl-project/sglang/issues/11989) - Various Blackwell kernel optimizations - DGX Spark Support: https://lmsys.org/blog/2025-10-13-nvidia-dgx-spark/ - KTransformer integration: https://lmsys.org/blog/2025-10-22-KTransformers/ - New model support: Nemotron, DeepSeek OCR, Qwen3-Omni, Olmo 3 - Native ModelOpt quantization support ## What's Changed * [router] add ipv6 support across all components by @slin1237 in https://github.com/sgl-project/sglang/pull/11219 * Remove env var warnings for release by @merrymercy in https://github.com/sgl-project/sglang/pull/11262 * Enable native ModelOpt quantization support (1/3) by @Edwardf0t1 in https://github.com/sgl-project/sglang/pull/7149 * [router][tool call] Clean up redundant `detect_format` and `has_tool_markers` by @CatherineSue in https://github.com/sgl-project/sglang/pull/11270 * disable sm100 for FlashMLA and fast-hadamard-transform in cuda12.6.1 by @gongwei-130 in https://github.com/sgl-project/sglang/pull/11274 * docker: add manifest to versioned docker releases by @ishandhanani in https://github.com/sgl-project/sglang/pull/11268 * [Bug] Fix incorrect assertion in FA4 and add UT. by @lifuhuang in https://github.com/sgl-project/sglang/pull/11182 * [router][grpc] Refine streaming processes by @CatherineSue in https://github.com/sgl-project/sglang/pull/11277 * Fix code sync scripts by @merrymercy in https://github.com/sgl-project/sglang/pull/11276 * [Auto Sync] Update test_utils.py (20251006) by @merrymercy in https://github.com/sgl-project/sglang/pull/11280 * Rename max_micro_batch_size -> pp_max_micro_batch_size by @merrymercy in https://github.com/sgl-project/sglang/pull/11279 * Reverse the AMD CI test back to 1200s and split the 8-gpu deepseek job into two. by @sunxxuns in https://github.com/sgl-project/sglang/pull/11238 * Fix LoRA support for multimodal models (VLMs) by implementing a consistent pattern for skipping vision components by @ConnorLi96 in https://github.com/sgl-project/sglang/pull/11261 * fix: correct scale parameter remapping logic in Llama4ForConditionalGeneration by @JustinTong0323 in https://github.com/sgl-project/sglang/pull/11282 * docs: update sgl-kernel README by @zhyncs in https://github.com/sgl-project/sglang/pull/11286 * chore: bump sgl-kernel version to 0.3.15 by @sglang-bot in https://github.com/sgl-project/sglang/pull/11281 * [router][grpc] Fix proto3 default value mismatches and cleanup unused fields by @CatherineSue in https://github.com/sgl-project/sglang/pull/11283 * convert test_deterministic into unit tests by @skyzh in https://github.com/sgl-project/sglang/pull/11095 * Feature/longbench v2 evaluation utils by @alhridoy in https://github.com/sgl-project/sglang/pull/10949 * [ci] fix pp test by @hnyls2002 in https://github.com/sgl-project/sglang/pull/11294 * EAGLE cache fix for SWARadixCache by @ispobock in https://github.com/sgl-project/sglang/pull/11231 * Remove overlap thread by @hnyls2002 in https://github.com/sgl-project/sglang/pull/11210 * [router] add reasoning and tool parser argument in router by @slin1237 in https://github.com/sgl-project/sglang/pull/11290 * Remove sampling info events and overlap thread file by @hnyls2002 in https://github.com/sgl-project/sglang/pull/11300 * Introduce future indices by @hnyls2002 in https://github.com/sgl-project/sglang/pull/11301 * [sgl-kernel] Support float64 moe_sum_reduce cuda kernel by @yuan-luo in https://github.com/sgl-project/sglang/pull/11068 * [Docs] [Router] Update Observability and Common Issues Section by @xuwenyihust in https://github.com/sgl-project/sglang/pull/11302 * [router] add get server info and get model info in grpc server by @slin1237 in https://github.com/sgl-project/sglang/pull/11303 * [router][grpc] Refactor chat template content format detection by @CatherineSue in https://github.com/sgl-project/sglang/pull/11288 * [Doc] HiCache Design Documents by @ykwd in https://github.com/sgl-project/sglang/pull/11027 * [Doc]: Best Practice for HICache by @hzh0425 in https://github.com/sgl-project/sglang/pull/11001 * [router] fix grpc connection conversion and add optimization by @slin1237 in https://github.com/sgl-project/sglang/pull/11305 * [router][grpc] Fix sampling_params.stop_strs is None by @CatherineSue in https://github.com/sgl-project/sglang/pull/11306 * Update tool parser and related documentation by @JustinTong0323 in https://github.com/sgl-project/sglang/pull/11223 * [router][grpc] Fix error message format in grpc chat handler by @CatherineSue in https://github.com/sgl-project/sglang/pull/11307 * [quantization] Properly ignore quantization for layers excluded in quant_config by @BowenBao in https://github.com/sgl-project/sglang/pull/11205 * [router] support Openai router conversation API CRUD by @key4ng in https://github.com/sgl-project/sglang/pull/11297 * [router][grpc] Fix request_id extraction when n > 1 by @CatherineSue in https://github.com/sgl-project/sglang/pull/11311 * [router] cleanup worker health check to return early by @slin1237 in https://github.com/sgl-project/sglang/pull/11310 * [oai serving chat] Add argument `--sampling-defaults` and fix `ChatCompletionRequest` defaults by @CatherineSue in https://github.com/sgl-project/sglang/pull/11304 * Clean match_prefix and prepare_for_extend for mem cache V2 by @cctry in https://github.com/sgl-project/sglang/pull/11200 * ci: unify the model launch method of nightly ci by @mickqian in https://github.com/sgl-project/sglang/pull/11230 * [Chore] Update xgrammar 0.1.24 -> 0.1.25 by @DarkSharpness in https://github.com/sgl-project/sglang/pull/10710 * update sampling_params documentation with defaults by @JustinTong0323 in https://github.com/sgl-project/sglang/pull/11315 * Optimize copy_kv_cache for spec decoding by @YAMY1234 in https://github.com/sgl-project/sglang/pull/11126 * Rename `ngram_utils` -> `ngram_info` by @hnyls2002 in https://github.com/sgl-project/sglang/pull/11316 * [router][grpc] Refactor chat handler in grpc/ to use centralized orchestrator by @CatherineSue in https://github.com/sgl-project/sglang/pull/11314 * [Feature] Add /tokenize and /detokenize OpenAI compatible endpoints by @adarshxs in https://github.com/sgl-project/sglang/pull/9545 * [8/N] MoE Refactor: deprecate `EPMoE` by @ch-wan in https://github.com/sgl-project/sglang/pull/11211 * Skip weight loading in deepgemm compilation by @ch-wan in https://github.com/sgl-project/sglang/pull/11312 * [2/2] Support MHA prefill with FlashAttention 4. by @lifuhuang in https://github.com/sgl-project/sglang/pull/10937 * [Doc] Update mooncake nvlink transport doc for PD disaggregation by @ShangmingCai in https://github.com/sgl-project/sglang/pull/11321 * fix(decode): adjust ServerArgs import to explicit module path by @xiaguan in https://github.com/sgl-project/sglang/pull/11007 * Support LoRA in bench_serving oai interface by @lifuhuang in https://github.com/sgl-project/sglang/pull/11318 * benchmark: enhance configurable multimodal benchmarking in bench_serving by @AlienKevin in https://github.com/sgl-project/sglang/pull/9812 * [CI] improve disaggregation CI. by @hnyls2002 in https://github.com/sgl-project/sglang/pull/11264 * model: Support Hybrid Mamba2 NemotronHForCausalLM (nvidia/NVIDIA-Nemotron-Nano-9B-v2) by @netanel-haber in https://github.com/sgl-project/sglang/pull/10909 * [router] refactor generate to use new pipeline arch by @slin1237 in https://github.com/sgl-project/sglang/pull/11323 * [router] improve reasoning parser lock and reduce req cloning by @slin1237 in https://github.com/sgl-project/sglang/pull/11336 * [router][grpc] Cleanup debug logs in grpc_server and grpc_router by @CatherineSue in https://github.com/sgl-project/sglang/pull/11340 * [router] Fix all unused_qualifications by @CatherineSue in https://github.com/sgl-project/sglang/pull/11341 * [router] Support history management using conversation by @key4ng in https://github.com/sgl-project/sglang/pull/11339 * [router][grpc] Add dependencies in Cargo.toml to support chat template rendering by @CatherineSue in https://github.com/sgl-project/sglang/pull/11342 * fix: fix revision for sgl-flash-attn in sgl-kernel by @mickqian in https://github.com/sgl-project/sglang/pull/11327 * [Auto Sync] Update scheduler.py (20251009) by @zhyncs in https://github.com/sgl-project/sglang/pull/11350 * [Generative Score API] Multi-Item scoring with custom attention mask. by @sundar24295s in https://github.com/sgl-project/sglang/pull/10979 * [router][grpc] disable health check generation and increase timeout by @slin1237 in https://github.com/sgl-project/sglang/pull/11353 * [router] Refactor OpenAI router: split monolithic file and move location by @key4ng in https://github.com/sgl-project/sglang/pull/11359 * [router][lint] Add unused_qualifications to cargo lint warnings by @CatherineSue in https://github.com/sgl-project/sglang/pull/11366 * [DeepSeek-V3.2] Include indexer kv cache when estimating kv cache size by @trevor-m in https://github.com/sgl-project/sglang/pull/11309 * [router][grpc] Fix tool call streaming bugs: empty tool names, state pollution, and panics by @CatherineSue in https://github.com/sgl-project/sglang/pull/11373 * add code pp support for nixl by @shaharmor98 in https://github.com/sgl-project/sglang/pull/11375 * fix bench_serving mishandling of internal states by @shaharmor98 in https://github.com/sgl-project/sglang/pull/11376 * [router][grpc] Replace fake health check with correct ones by @CatherineSue in https://github.com/sgl-project/sglang/pull/11387 * [router] change grpc client from mutable to clone by @slin1237 in https://github.com/sgl-project/sglang/pull/11394 * chore: upgrade flashinfer 0.4.0 by @zhyncs in https://github.com/sgl-project/sglang/pull/11364 * [router] conversation item API: create, retrieve and delete by @key4ng in https://github.com/sgl-project/sglang/pull/11369 * chore: bump SGLang version to 0.5.3.post1 by @sglang-bot in https://github.com/sgl-project/sglang/pull/11324 * move more files under srt/utils by @merrymercy in https://github.com/sgl-project/sglang/pull/11285 * [grammar] Avoid server crash when grammar backend is None by @JustinTong0323 in https://github.com/sgl-project/sglang/pull/11401 * fix: fix gpu-proc affinity set incorrectly when pp_size > 1 by @acelyc111 in https://github.com/sgl-project/sglang/pull/11389 * [Bug Fix] prevent lora adapter from being loaded into LoRAManager if it is already loaded by @glenliu21 in https://github.com/sgl-project/sglang/pull/11365 * [CI] Refactor PD disaggregation test suite by @ShangmingCai in https://github.com/sgl-project/sglang/pull/11363 * Replace pad with cat for better performance by @yuan-luo in https://github.com/sgl-project/sglang/pull/11388 * fix: reinstall torch in deps install by @zhyncs in https://github.com/sgl-project/sglang/pull/11414 * feat(hicache): Support passing prefix keys for l3 store. by @hzh0425 in https://github.com/sgl-project/sglang/pull/9045 * fix file and object naming scheme in HiCacheNixl to avoid data corruption by @ziruiliu in https://github.com/sgl-project/sglang/pull/10969 * Dedicated toml files for CPU/XPU by @ZailiWang in https://github.com/sgl-project/sglang/pull/10734 * Add metrics for speculative decoding (acceptance rate, average acceptance length) by @scottjlee in https://github.com/sgl-project/sglang/pull/11144 * chore: update pyproject by @zhyncs in https://github.com/sgl-project/sglang/pull/11420 * fix: fix video input for qwen3-vl by @mickqian in https://github.com/sgl-project/sglang/pull/11361 * perf: optimize qwen-vl with symm mem allreduce by @yuan-luo in https://github.com/sgl-project/sglang/pull/11381 * [HiCache] feat: add multi tenant with prefix tag by @stmatengss in https://github.com/sgl-project/sglang/pull/9256 * [CI] Merge build-dev into workflow matrix by @csahithi in https://github.com/sgl-project/sglang/pull/11345 * Revert "perf: optimize qwen-vl with symm mem allreduce" by @ch-wan in https://github.com/sgl-project/sglang/pull/11436 * Revert "fix: fix video input for qwen3-vl" by @merrymercy in https://github.com/sgl-project/sglang/pull/11437 * Revert "Add metrics for speculative decoding (acceptance rate, average acceptance length)" by @scottjlee in https://github.com/sgl-project/sglang/pull/11433 * [router] Fix ci nvcc not found error by @key4ng in https://github.com/sgl-project/sglang/pull/11411 * feat(mooncake): support GB suffix for global_segment_size by @xiaguan in https://github.com/sgl-project/sglang/pull/10745 * Separate allocation logic from scheduler by @cctry in https://github.com/sgl-project/sglang/pull/11313 * [router] disable rate limiter by default by @slin1237 in https://github.com/sgl-project/sglang/pull/11435 * [router] leverage RAII to actively cancel request during client disconnect by @slin1237 in https://github.com/sgl-project/sglang/pull/11399 * [router][grpc] Consolidate parser checks for chat completions by @CatherineSue in https://github.com/sgl-project/sglang/pull/11439 * Reorder PD disagg CI tests by @merrymercy in https://github.com/sgl-project/sglang/pull/11438 * fix: Change dsv32 hack temporary path to use system temp directory by @wxsms in https://github.com/sgl-project/sglang/pull/11445 * Fix batch invariant ops by @hebiao064 in https://github.com/sgl-project/sglang/pull/11368 * [BugFix] test_mla_fp8.py fails on Cublas 12.9 by @Liu-congo in https://github.com/sgl-project/sglang/pull/11360 * [DPSKv3.2] Rewrite nsa tilelang act_quant kernel to triton by @byjiang1996 in https://github.com/sgl-project/sglang/pull/11450 * Remove tilelang dependency in Dockerfile by @Fridge003 in https://github.com/sgl-project/sglang/pull/11455 * Enable native ModelOpt quantization support (2/3) by @Edwardf0t1 in https://github.com/sgl-project/sglang/pull/9991 * Reland [1/2] Optimizations and refactors about quant kernel by @fzyzcjy in https://github.com/sgl-project/sglang/pull/10312 * Super tiny delete unused openai router in sgl-router by @fzyzcjy in https://github.com/sgl-project/sglang/pull/11448 * Adjust logits metada init for target verify by @hnyls2002 in https://github.com/sgl-project/sglang/pull/11467 * [Documentation][Configuration] Server args and documentation of PD-Multiplexing. by @ykcombat in https://github.com/sgl-project/sglang/pull/11427 * Fix enable_v2 in int8 quant by @fzyzcjy in https://github.com/sgl-project/sglang/pull/11470 * [Fix] Fix split prefill with fa3. by @ykcombat in https://github.com/sgl-project/sglang/pull/11428 * fix stop when stream by @whybeyoung in https://github.com/sgl-project/sglang/pull/11462 * Add option to disable `any_whitespace` for `xgrammar` and `llguidance` backends. by @lulor in https://github.com/sgl-project/sglang/pull/8919 * [7/n] decouple quantization impl from vllm dependency - gguf kernel by @FlamingoPg in https://github.com/sgl-project/sglang/pull/11019 * fix Xeon CI by @ZailiWang in https://github.com/sgl-project/sglang/pull/11454 * [CI] Add nightly builds to dockerhub by @csahithi in https://github.com/sgl-project/sglang/pull/9804 * [Feature] support regex strings as a stopping condition by @glenliu21 in https://github.com/sgl-project/sglang/pull/10635 * Beta spec-overlap for EAGLE by @hnyls2002 in https://github.com/sgl-project/sglang/pull/11398 * Piecewise CUDA Graph Support & Torch Compile Backend by @Oasis-Git in https://github.com/sgl-project/sglang/pull/10062 * [Router]: Small Typo in a comment within tree.rs by @xuwenyihust in https://github.com/sgl-project/sglang/pull/11489 * chore: bump sgl-kernel version to 0.3.16 by @sglang-bot in https://github.com/sgl-project/sglang/pull/11476 * [smol] [perf] Qwen3-VL in place op. by @vincentzed in https://github.com/sgl-project/sglang/pull/11481 * [chore][1/N] Avoid using default mutable parameters by @kevin85421 in https://github.com/sgl-project/sglang/pull/11478 * [bugfix]: use correct causality condition for flashattention, flashinfer, and triton backends by @MahmoudAshraf97 in https://github.com/sgl-project/sglang/pull/10172 * [ perf ] Replace json-> orjson in hot path by @vincentzed in https://github.com/sgl-project/sglang/pull/11221 * [chore][2/N] Avoid using default mutable parameters by @kevin85421 in https://github.com/sgl-project/sglang/pull/11479 * Fix the GPT function calling regex to allow dash in the name by @antoine-roux in https://github.com/sgl-project/sglang/pull/10577 * bailingMoE: Fix Key error of deepep_mode by @QiuMike in https://github.com/sgl-project/sglang/pull/11465 * Fix CI break by express-laned PRs. by @hnyls2002 in https://github.com/sgl-project/sglang/pull/11499 * Move args from `global_config` to `environ` by @hnyls2002 in https://github.com/sgl-project/sglang/pull/11332 * move fla env check position by @yizhang2077 in https://github.com/sgl-project/sglang/pull/11500 * Temporarily remove b200 tests by @merrymercy in https://github.com/sgl-project/sglang/pull/11501 * Fix port conflicts in CI by @merrymercy in https://github.com/sgl-project/sglang/pull/11497 * temporarily remove b200 tests by @merrymercy in https://github.com/sgl-project/sglang/pull/11502 * Fix unit tests by @merrymercy in https://github.com/sgl-project/sglang/pull/11503 * Bugfix: Fix Type consistency for KV indices in SWARadixCache by @hzh0425 in https://github.com/sgl-project/sglang/pull/11452 * doc: add doc for adding new models into nightly-ci by @mickqian in https://github.com/sgl-project/sglang/pull/11443 * [CI] fix lint by @hnyls2002 in https://github.com/sgl-project/sglang/pull/11509 * Deprecate `global_server_args_dict` by @hnyls2002 in https://github.com/sgl-project/sglang/pull/11331 * chore: remove flashinfer cleanup cache by @zhyncs in https://github.com/sgl-project/sglang/pull/11514 * fix: revert temporarily remove b200 tests by @zhyncs in https://github.com/sgl-project/sglang/pull/11515 * [Fix] Improve longbench prompt and other logics by @byjiang1996 in https://github.com/sgl-project/sglang/pull/11474 * Sync changes on io_struct.py and deterministic ops by @merrymercy in https://github.com/sgl-project/sglang/pull/11498 * [lint] Fix the lint issue by @ch-wan in https://github.com/sgl-project/sglang/pull/11516 * Revert "Deprecate `global_server_args_dict`" by @ch-wan in https://github.com/sgl-project/sglang/pull/11520 * Improve dp attention port assignment scheme by @jokerwyt in https://github.com/sgl-project/sglang/pull/5889 * [router] openai router: support grok model by @key4ng in https://github.com/sgl-project/sglang/pull/11511 * docs(router): add token-bucket rate limiting to the docs by @Jonahcb in https://github.com/sgl-project/sglang/pull/11485 * [sgl-kernel][1/N]Support Expert Specialization Grouped GEMM by @HydraQYH in https://github.com/sgl-project/sglang/pull/11432 * Update DeepSeek-R1-FP4 default config on blackwell by @Qiaolin-Yu in https://github.com/sgl-project/sglang/pull/11512 * [Fix]: add missing device attribute to ChunkCache by @leavelet in https://github.com/sgl-project/sglang/pull/11493 * [Feature] Support mamba radix cache v0 by @yizhang2077 in https://github.com/sgl-project/sglang/pull/11214 * ci: improve nightly-ci by @mickqian in https://github.com/sgl-project/sglang/pull/11385 * [CI monitor] Improve CI analyzer: fix job failure tracking and add CUDA-focused filtering by @BBuf in https://github.com/sgl-project/sglang/pull/11505 * [HICache]: Support 3FS-Store with page_first_direct layout by @hzh0425 in https://github.com/sgl-project/sglang/pull/11460 * Tiny fix test run estimated time by @ShangmingCai in https://github.com/sgl-project/sglang/pull/11544 * [Reland] perf: optimize qwen-vl with symm mem allreduce by @yuan-luo in https://github.com/sgl-project/sglang/pull/11457 * Depreate `global_server_args_dict` by @hnyls2002 in https://github.com/sgl-project/sglang/pull/11528 * [Fix] Add per_channel_quant parameter to MoE config functions by @mmangkad in https://github.com/sgl-project/sglang/pull/11201 * [router][ci] Add Nightly Release Workflow for SGLang Router by @slin1237 in https://github.com/sgl-project/sglang/pull/11527 * [router] add tokenizer path to be dir by @slin1237 in https://github.com/sgl-project/sglang/pull/11530 * Remove `tp_worker.worker` by @hnyls2002 in https://github.com/sgl-project/sglang/pull/11548 * fix: fix video input for qwen3-vl by @mickqian in https://github.com/sgl-project/sglang/pull/11442 * [NVIDIA] BUMP FA3 by @johnnynunez in https://github.com/sgl-project/sglang/pull/11444 * [Fix] Include grpc reflection runtime dependency by @ai-jz in https://github.com/sgl-project/sglang/pull/11419 * Adjust overlap event loop by @hnyls2002 in https://github.com/sgl-project/sglang/pull/11507 * Move deep gemm related arguments to `sglang.srt.environ` by @hnyls2002 in https://github.com/sgl-project/sglang/pull/11547 * [router][grpc] Further delegate non-stream processing to `processing.rs` by @CatherineSue in https://github.com/sgl-project/sglang/pull/11553 * [router] allow user to specify chat template path by @slin1237 in https://github.com/sgl-project/sglang/pull/11549 * Minor: improve sampler & remove unused fields from model_config.py by @merrymercy in https://github.com/sgl-project/sglang/pull/11531 * [router] Add Rust CLI flags for queue size, timeout, and rate limit for token bucket rate limiter by @Jonahcb in https://github.com/sgl-project/sglang/pull/11483 * Add metrics for speculative decoding (acceptance rate, average acceptance length) by @scottjlee in https://github.com/sgl-project/sglang/pull/11441 * Fix DeepSeek-v3.2 default config (ValueError: not enough values to unpack (expected 4, got 3)) by @trevor-m in https://github.com/sgl-project/sglang/pull/11557 * [CI] Add Basic Test for DeepSeek V3.2 by @Fridge003 in https://github.com/sgl-project/sglang/pull/11308 * [router][grpc] Add error handling to `generate_tool_constraints` by @CatherineSue in https://github.com/sgl-project/sglang/pull/11562 * [NVIDIA] update pyproject.toml to support cu130 option by @johnnynunez in https://github.com/sgl-project/sglang/pull/11521 * [CI Monitor] Ci monitor only deal with main branch in default by @BBuf in https://github.com/sgl-project/sglang/pull/11538 * Tiny cleanup fp4 gemm calls by @fzyzcjy in https://github.com/sgl-project/sglang/pull/11537 * [router][grpc] Add `serve_grpc` to `launch_server` and log id for HealthCheck by @CatherineSue in https://github.com/sgl-project/sglang/pull/11564 * [router] Add BRANCH_TYPE=local support to Dockerfile.router for local builds by @YouNeedCryDear in https://github.com/sgl-project/sglang/pull/11571 * [sgl-kernel][2/N]Support Expert Specialization Grouped GEMM by @HydraQYH in https://github.com/sgl-project/sglang/pull/11534 * chore: bump sgl-kernel version to 0.3.16.post1 by @sglang-bot in https://github.com/sgl-project/sglang/pull/11573 * Fix accept rate in speculative decoding metrics by @Qiaolin-Yu in https://github.com/sgl-project/sglang/pull/11572 * Compilation Folder Reset by @Oasis-Git in https://github.com/sgl-project/sglang/pull/11539 * [FEATURE] Add Profile Trace Merger for Distributed Traces by @neelabhsinha in https://github.com/sgl-project/sglang/pull/11413 * [DSv32] Use torch.compile for _get_logits_head_gate by @trevor-m in https://github.com/sgl-project/sglang/pull/11565 * Make DeepEP combine recv do not overlap by @fzyzcjy in https://github.com/sgl-project/sglang/pull/11535 * bench_serving support PD Disaggregation by @BBuf in https://github.com/sgl-project/sglang/pull/11542 * Implement LRU eviction policy for LoRA adapters by @ConnorLi96 in https://github.com/sgl-project/sglang/pull/11041 * Revert "[NVIDIA] BUMP FA3 (#11444)" by @zhyncs in https://github.com/sgl-project/sglang/pull/11582 * chore: bump sgl-kernel version to 0.3.16.post2 by @sglang-bot in https://github.com/sgl-project/sglang/pull/11583 * [Auto Sync] Update model_config.py (20251014) by @merrymercy in https://github.com/sgl-project/sglang/pull/11580 * Add fused_moe_triton config: triton_3_4_0/E=256,N=256,device_name=NVIDIA_B200.json by @Qiaolin-Yu in https://github.com/sgl-project/sglang/pull/11587 * [router][protocols] Add Axum validate extractor and use it for `/v1/chat/completions` endpoint by @CatherineSue in https://github.com/sgl-project/sglang/pull/11588 * [router] update generate spec to align with sgl io struct by @slin1237 in https://github.com/sgl-project/sglang/pull/11591 * [router] change worker api to async instead of sync by @slin1237 in https://github.com/sgl-project/sglang/pull/11566 * Update news section in README.md by @merrymercy in https://github.com/sgl-project/sglang/pull/11598 * [router] delete useless table content comment in spec by @slin1237 in https://github.com/sgl-project/sglang/pull/11597 * [router] allow router launch server to use grpc mode by @slin1237 in https://github.com/sgl-project/sglang/pull/11600 * [Docs] [Router]: Update sg-router doc on circuit breaker by @xuwenyihust in https://github.com/sgl-project/sglang/pull/11449 * [router] when given both local tokenizer and chat template, log all by @slin1237 in https://github.com/sgl-project/sglang/pull/11601 * [AMD CI] Add image and weights caching. by @saienduri in https://github.com/sgl-project/sglang/pull/11593 * Update release-docker-dev.yml by @sglang-bot in https://github.com/sgl-project/sglang/pull/11603 * Optimize Triton Draft Backend by @hnyls2002 in https://github.com/sgl-project/sglang/pull/11556 * Refactor spec decoding metrics calculation into separate `TokenizerManager` utility function by @scottjlee in https://github.com/sgl-project/sglang/pull/11586 * make radix cache deterministic by @skyzh in https://github.com/sgl-project/sglang/pull/10721 * move eagle draft post process to cuda graph by @cicirori in https://github.com/sgl-project/sglang/pull/11434 * Reduce one step decode for draft model. by @hnyls2002 in https://github.com/sgl-project/sglang/pull/11561 * [router] add py binding and readme for openai router and history backend by @key4ng in https://github.com/sgl-project/sglang/pull/11453 * [router] cleanup app context and move to startup by @slin1237 in https://github.com/sgl-project/sglang/pull/11617 * [router] add chang and keyang to sgl router author by @slin1237 in https://github.com/sgl-project/sglang/pull/11620 * use non_blocking h2d in ForwardBatch.prepare_mlp_sync_batch. by @strgrb in https://github.com/sgl-project/sglang/pull/11605 * [router] update router readme to latest features by @slin1237 in https://github.com/sgl-project/sglang/pull/11619 * Fix log for chunked prefix cache by @Fridge003 in https://github.com/sgl-project/sglang/pull/11624 * [Auto Sync] Update scheduler.py, server_args.py (20251014) by @merrymercy in https://github.com/sgl-project/sglang/pull/11623 * [Auto Sync] Update collector.py (20251014) by @merrymercy in https://github.com/sgl-project/sglang/pull/11625 * [Minor] Update xgrammar dependency by @DarkSharpness in https://github.com/sgl-project/sglang/pull/11622 * Update install.md by @merrymercy in https://github.com/sgl-project/sglang/pull/11631 * fix: Update SGL_KERNEL_VERSION to 0.3.15 by @zhyncs in https://github.com/sgl-project/sglang/pull/11633 * [router][grpc] add warm up to grpc server by @slin1237 in https://github.com/sgl-project/sglang/pull/11627 * Refactor kv cache free by @cctry in https://github.com/sgl-project/sglang/pull/11351 * [router] update router doc to latest features by @slin1237 in https://github.com/sgl-project/sglang/pull/11639 * fix: upgrade transformers to 4.57.1 by @csahithi in https://github.com/sgl-project/sglang/pull/11628 * [router] add worker self discovery for metadata by @slin1237 in https://github.com/sgl-project/sglang/pull/11638 * [router] upgrade to 0.2.0 by @slin1237 in https://github.com/sgl-project/sglang/pull/11642 * [1/N] Introduce Mooncake Backend and Mooncake EP to Support Elastic EP by @UNIDY2002 in https://github.com/sgl-project/sglang/pull/10423 * [1/N]Support DeepSeek-R1 w4a8 normal deepep by @ayrnb in https://github.com/sgl-project/sglang/pull/8247 * [Fix] Fix accuracy bug in CSGMV kernel caching key. by @lifuhuang in https://github.com/sgl-project/sglang/pull/11579 * feat: add add_chunked_prefix_cache_attention_backend by @zhyncs in https://github.com/sgl-project/sglang/pull/11636 * Super tiny improve FA3 import error message by @fzyzcjy in https://github.com/sgl-project/sglang/pull/11590 * [BugFix][Qwen3-VL]: fix cu_seqlens in qwen3-vl by @ZhengWG in https://github.com/sgl-project/sglang/pull/11458 * [Doc] Update support matrix for attn and hybrid attn by @b8zhong in https://github.com/sgl-project/sglang/pull/11293 * Clean up some Qwen3-Next and deterministic code by @hebiao064 in https://github.com/sgl-project/sglang/pull/11585 * docs: update sglang installation guide by @zhyncs in https://github.com/sgl-project/sglang/pull/11659 * Tiny cleanup some eagle unused codes by @hnyls2002 in https://github.com/sgl-project/sglang/pull/11660 * Fix 1-step draft model forward by @ShangmingCai in https://github.com/sgl-project/sglang/pull/11653 * [tool call] Fix prev_tool_call_arr management in base_format_detector.py by @CatherineSue in https://github.com/sgl-project/sglang/pull/11367 * [router] Fix response api related spec by @key4ng in https://github.com/sgl-project/sglang/pull/11621 * Fix missing json imports in serving_responses.py by @CatherineSue in https://github.com/sgl-project/sglang/pull/11681 * [sgl-kernel][3/N]Support Expert Specialization Grouped GEMM by @HydraQYH in https://github.com/sgl-project/sglang/pull/11674 * [sgl-kernel] Optimize gguf test by @FlamingoPg in https://github.com/sgl-project/sglang/pull/11667 * [router][grpc] Simplify model_id determination by @CatherineSue in https://github.com/sgl-project/sglang/pull/11684 * [router] Refactor StopSequenceDecoder to Use Sequence for Incremental Decoding by @slin1237 in https://github.com/sgl-project/sglang/pull/11676 * chore: bump SGLang version to 0.5.3.post2 by @sglang-bot in https://github.com/sgl-project/sglang/pull/11680 * [CI][XPU]enable sglang CI on Intel XPU by @DiweiSun in https://github.com/sgl-project/sglang/pull/9493 * enable rmsnorm on XPU by @huaiyuzh in https://github.com/sgl-project/sglang/pull/10248 * Sync code and test CI; rename some env vars by @merrymercy in https://github.com/sgl-project/sglang/pull/11686 * docs: Add Contributor Covenant Code of Conduct by @zhyncs in https://github.com/sgl-project/sglang/pull/11689 * [Mamba] Increase default mamba_full_memory_ratio to 0.9 by @hanming-lu in https://github.com/sgl-project/sglang/pull/11679 * [PD] Add PD support for hybrid model (Qwen3-Next, DeepSeek V3.2 Exp) by @ShangmingCai in https://github.com/sgl-project/sglang/pull/10912 * [sgl-kernel] support hadamard by @FlamingoPg in https://github.com/sgl-project/sglang/pull/11663 * Fix missing a2a backend init of GLM4.5 MoE Block by @ShangmingCai in https://github.com/sgl-project/sglang/pull/11692 * Split test_intel_amx_attention_backend.py to pass CI of timeout by @yanbing-j in https://github.com/sgl-project/sglang/pull/11370 * Set csgmv as default lora backend. by @lifuhuang in https://github.com/sgl-project/sglang/pull/11488 * [Bugfix] Fix Qwen3/DSV3/DSV3.2 model support by @iforgetmyname in https://github.com/sgl-project/sglang/pull/11510 * [CI] Add GLM4MoE model test by @ShangmingCai in https://github.com/sgl-project/sglang/pull/11706 * [router] fix get_models endpoint for openai router by @key4ng in https://github.com/sgl-project/sglang/pull/11687 * [ci]use H20 to run disaggregation test by @HanHan009527 in https://github.com/sgl-project/sglang/pull/11543 * chore: bump SGLang version to 0.5.3.post3 by @sglang-bot in https://github.com/sgl-project/sglang/pull/11693 * model: qwen3-omni (thinker-only) by @mickqian in https://github.com/sgl-project/sglang/pull/10911 * [Router] Refactor protocol definitions: split spec.rs into modular files by @key4ng in https://github.com/sgl-project/sglang/pull/11677 * [router] fix p and d worker filtering and bootstrap port handling by @slin1237 in https://github.com/sgl-project/sglang/pull/11729 * [router][grpc] add dissag info to warm up in grpc server by @slin1237 in https://github.com/sgl-project/sglang/pull/11727 * [router] Fix tool_choice normalization in ChatCompletionRequest and fix ut by @CatherineSue in https://github.com/sgl-project/sglang/pull/11731 * Revert "make radix cache deterministic" by @Fridge003 in https://github.com/sgl-project/sglang/pull/11728 * Reduce the image processing latency in VLM by @zhooooong in https://github.com/sgl-project/sglang/pull/11541 * [router] add spec.rs to enables tests under spec folder by @key4ng in https://github.com/sgl-project/sglang/pull/11734 * [router] Add rustfmt and set group imports by default by @CatherineSue in https://github.com/sgl-project/sglang/pull/11732 * Revert "[router] fix get_models endpoint for openai router (#11687)" by @key4ng in https://github.com/sgl-project/sglang/pull/11740 * [router][CI] Clean up deprecated fields in `pr-test-pd-router.yml` by @CatherineSue in https://github.com/sgl-project/sglang/pull/11739 * [CI] Fix broken event loop creation by @hnyls2002 in https://github.com/sgl-project/sglang/pull/11746 * [overlap-spec] Make plan stream an option by @hnyls2002 in https://github.com/sgl-project/sglang/pull/11724 * ci: reduce and refactor vlm ut and combine test files by @mickqian in https://github.com/sgl-project/sglang/pull/11062 * Abstraction for spec worker and code cleanup by @hnyls2002 in https://github.com/sgl-project/sglang/pull/11643 * add tuned fuse moe kernel for qwen3 235b fp8 on h200 by @pdasgup in https://github.com/sgl-project/sglang/pull/11730 * Revert "Set csgmv as default lora backend. (#11488)" by @zhyncs in https://github.com/sgl-project/sglang/pull/11735 * [router] Fix UTF-8 Boundary Panic in Stop Sequence Decoder by @slin1237 in https://github.com/sgl-project/sglang/pull/11766 * [router] fix grpc client time out to 1h by @slin1237 in https://github.com/sgl-project/sglang/pull/11768 * [doc] update router document by @key4ng in https://github.com/sgl-project/sglang/pull/11767 * [Feature] Reuse flashinfer workspace for PD-Multiplexing. by @ykcombat in https://github.com/sgl-project/sglang/pull/11540 * Turn on shm_allreduce and shm_allgather for fp16 by @chunyuan-w in https://github.com/sgl-project/sglang/pull/10725 * [Auto Sync] Update scheduler.py (20251017) by @zhyncs in https://github.com/sgl-project/sglang/pull/11738 * [router][grpc] Remove timeout for connections and remove `max_tokens` deprecation warning log by @CatherineSue in https://github.com/sgl-project/sglang/pull/11775 * Cleaning indexer for DeepSeek V3.2 by @Fridge003 in https://github.com/sgl-project/sglang/pull/11682 * [minor] sync code on python/sglang/test/test_deterministic.py and improve ci tests by @merrymercy in https://github.com/sgl-project/sglang/pull/11777 * [Auto Sync] Update common.py (20251017) by @merrymercy in https://github.com/sgl-project/sglang/pull/11782 * [Fix] Skip visual layers when applying LoRA to Qwen2VL modules by @anvdn in https://github.com/sgl-project/sglang/pull/11519 * [Lint] Add `python/sglang` to ruff F401 checks and remove unused imports in files by @CatherineSue in https://github.com/sgl-project/sglang/pull/11685 * Super tiny fix missing input throughput by @fzyzcjy in https://github.com/sgl-project/sglang/pull/11607 * Support shared experts overlap in cutlass moe by @fzyzcjy in https://github.com/sgl-project/sglang/pull/11611 * Support casting bf16 NextN moe to fp8 by @fzyzcjy in https://github.com/sgl-project/sglang/pull/11613 * Manually flip deepep_mode for cuda_graph by @zhuzilin in https://github.com/sgl-project/sglang/pull/11666 * Set CUDA_VISIBLE_DEVICES to achieve one GPU per process by @merrymercy in https://github.com/sgl-project/sglang/pull/9170 * Super tiny fix CI by @fzyzcjy in https://github.com/sgl-project/sglang/pull/11788 * Make single-batch overlap compatible with offloading by @fzyzcjy in https://github.com/sgl-project/sglang/pull/11614 * completely remove mixed mode deterministic test as prefix mode could cover it by @zminglei in https://github.com/sgl-project/sglang/pull/11783 * [Refactor] move `deep_gemm_wrapper` out of `quantization` by @ch-wan in https://github.com/sgl-project/sglang/pull/11784 * Enable lint on main by @fzyzcjy in https://github.com/sgl-project/sglang/pull/11794 * [router][grpc] Support parallel queue puts in grpc_request_manager and remove mutex for grpc_client by @CatherineSue in https://github.com/sgl-project/sglang/pull/11798 * Try add back no-commit-to-branch by @fzyzcjy in https://github.com/sgl-project/sglang/pull/11799 * fix(glm45): disable reduce scatter by @jinmingyi1998 in https://github.com/sgl-project/sglang/pull/11665 * fix command line usage of profiling by @Qiaolin-Yu in https://github.com/sgl-project/sglang/pull/11793 * [RL] support weight update with DP attention by @zhuzilin in https://github.com/sgl-project/sglang/pull/11669 * [RL] use cpu group to prepare_mlp_sync_batch_raw when the server is offloaded by @zhuzilin in https://github.com/sgl-project/sglang/pull/10152 * set default attention backend for deterministic inference by @zminglei in https://github.com/sgl-project/sglang/pull/11801 * Eager Compiler for Torch Compile by @Oasis-Git in https://github.com/sgl-project/sglang/pull/11803 * Fix install instructions and pyproject.tomls by @merrymercy in https://github.com/sgl-project/sglang/pull/11781 * Bump torch_memory_saver to avoid installing pre-release versions by @fzyzcjy in https://github.com/sgl-project/sglang/pull/11797 * [HiCache] feat: add more eviction policy by @stmatengss in https://github.com/sgl-project/sglang/pull/11506 * [overlap-spec] support page size > 1 by @hnyls2002 in https://github.com/sgl-project/sglang/pull/11772 * support server arg override KV cache to bf16 to avoid slow cases by @b8zhong in https://github.com/sgl-project/sglang/pull/11749 * feat(example/fastapi): support --startup-timeout using Qwen3-Next-80B-A3B-Instruct as example by @Kindyaa in https://github.com/sgl-project/sglang/pull/11710 * ci: update `lmms-eval` to speed up multimodal CI by @b8zhong in https://github.com/sgl-project/sglang/pull/11000 * Use cutlass fp4 gemm by default by @Qiaolin-Yu in https://github.com/sgl-project/sglang/pull/11813 * Fix Dockerfile not installing correct version of DeepEP for arm build by @kyleliang-nv in https://github.com/sgl-project/sglang/pull/11773 * [router] Add Configurable L0 and L1 Tokenizer Caching by @slin1237 in https://github.com/sgl-project/sglang/pull/11688 * [2/2] [feature] support openai like classification api in router by @whybeyoung in https://github.com/sgl-project/sglang/pull/11670 * [1/2][feature] support openai like classification api by @whybeyoung in https://github.com/sgl-project/sglang/pull/11618 * make sure logit bias is applied during eagle spec decoding verification by @petricevich in https://github.com/sgl-project/sglang/pull/11555 * fix: do not wrap invalid grammar objects during constrained generation by @tazjin in https://github.com/sgl-project/sglang/pull/11328 * Improve `send_sone` script by @hnyls2002 in https://github.com/sgl-project/sglang/pull/11817 * Fix: Dynamic RoPE Cache Expansion to Prevent Position-ID Out-of-Bounds in EAGLE + Long-Sequence Workloads by @YAMY1234 in https://github.com/sgl-project/sglang/pull/10788 * Update CODEOWNERS for layer quantization path by @merrymercy in https://github.com/sgl-project/sglang/pull/11818 * support tokenized batch request by @narutolhy in https://github.com/sgl-project/sglang/pull/11091 * Tiny add hints when users send requests to wrong place by @fzyzcjy in https://github.com/sgl-project/sglang/pull/11808 * Make single-batch overlap compatible with NextN by @fzyzcjy in https://github.com/sgl-project/sglang/pull/11804 * Support not officially supported high sgl-kernel version with low srt version by @fzyzcjy in https://github.com/sgl-project/sglang/pull/11786 * Avoid generation gets hanging when user specifies multiple event loops by @fzyzcjy in https://github.com/sgl-project/sglang/pull/5162 * Change bf16 to fp8 for some gemms in attention for DeepSeek ckpt v2 by @fzyzcjy in https://github.com/sgl-project/sglang/pull/11805 * Revert "Fix: Dynamic RoPE Cache Expansion to Prevent Position-ID Out-of-Bounds in EAGLE + Long-Sequence Workloads" by @hnyls2002 in https://github.com/sgl-project/sglang/pull/11827 * [overlap-spec] fix stop condition and trimming by @hnyls2002 in https://github.com/sgl-project/sglang/pull/11819 * [Spec Decoding] Support MTP for dsv3.2 by @Paiiiiiiiiiiiiii in https://github.com/sgl-project/sglang/pull/11652 * [CI] always print back trace in `retry()` by @hnyls2002 in https://github.com/sgl-project/sglang/pull/11834 * [Test] Add basic matched stop for beta eagle by @hnyls2002 in https://github.com/sgl-project/sglang/pull/11833 * Deterministic Mode: Add 1-stage triton kernel for prefill by @hebiao064 in https://github.com/sgl-project/sglang/pull/11147 * [logprobs] Enable local deterministic logrprobs testing with strict threshold by @PrinsYin in https://github.com/sgl-project/sglang/pull/10994 * [CI] Add CI test for DeepSeek V3.2 MTP by @Fridge003 in https://github.com/sgl-project/sglang/pull/11835 * [NVIDIA] FA3/FA4 Fix by @johnnynunez in https://github.com/sgl-project/sglang/pull/11606 * [DeepseekV32] Add fast_topk_transform_ragged_fused kernel by @hlu1 in https://github.com/sgl-project/sglang/pull/11815 * Fix triton_kernels import error on some hardwares by @fzyzcjy in https://github.com/sgl-project/sglang/pull/11831 * Tiny bump DeepEP version in ARM blackwell by @fzyzcjy in https://github.com/sgl-project/sglang/pull/11810 * [BugFix] replace the input_to_float8 used in dsv2 by @Liu-congo in https://github.com/sgl-project/sglang/pull/11612 * [Doc] Update documents for FA4 by @Fridge003 in https://github.com/sgl-project/sglang/pull/11778 * fix(ci): Fix CI Monitor limit parameter and add CI Analysis to summary by @BBuf in https://github.com/sgl-project/sglang/pull/11832 * Fix version bump script to handle TOML files with outdated versions by @Kangyan-Zhou in https://github.com/sgl-project/sglang/pull/11787 * Improve Kernel Build Time by @Kangyan-Zhou in https://github.com/sgl-project/sglang/pull/11508 * check master server for mooncake store by @huangtingwei9988 in https://github.com/sgl-project/sglang/pull/10510 * chore: bump sgl-kernel version to 0.3.16.post3 by @sglang-bot in https://github.com/sgl-project/sglang/pull/11733 * Recapture cuda graph after model weight update to resolve IMA error by @harrisonlimh in https://github.com/sgl-project/sglang/pull/11780 * [Feature] Use current greenctx stream to communicate in PD-Multiplexing. by @ykcombat in https://github.com/sgl-project/sglang/pull/11594 * Support mrope triton kernel and add unit test by @yuan-luo in https://github.com/sgl-project/sglang/pull/11722 * [PD] Improve eagle acceptance rate by transferring draft model hidden states by @ZeldaHuang in https://github.com/sgl-project/sglang/pull/10801 * Tiny clean up for PD module and doc by @ShangmingCai in https://github.com/sgl-project/sglang/pull/11747 * Revert "[CI Monitor] Ci monitor only deal with main branch in default" by @BBuf in https://github.com/sgl-project/sglang/pull/11846 * [Model] Add Olmo 3 model support by @2015aroras in https://github.com/sgl-project/sglang/pull/11396 * Update amd gpu install docs. by @saienduri in https://github.com/sgl-project/sglang/pull/11849 * [AMD CI] Populate image cache in nightly docker release. by @saienduri in https://github.com/sgl-project/sglang/pull/11822 * fix(server_args): handle tokenizer init conflicts by @ishandhanani in https://github.com/sgl-project/sglang/pull/11776 * [Feature] New structural tag support by @DarkSharpness in https://github.com/sgl-project/sglang/pull/10691 * Tiny fix main lint by @hnyls2002 in https://github.com/sgl-project/sglang/pull/11862 * [9/N] MoE Refactor: cleanup dispatcher interfaces by @ch-wan in https://github.com/sgl-project/sglang/pull/11847 * Fix acc len and gen throughput metrics when enabling overlap-spec by @Qiaolin-Yu in https://github.com/sgl-project/sglang/pull/11823 * Replace function call with set literal by @penguin-wwy in https://github.com/sgl-project/sglang/pull/11867 * Support mixing cutedsl and deepgemm backend by @fzyzcjy in https://github.com/sgl-project/sglang/pull/11807 * [router] Worker Management Workflow Engine by @slin1237 in https://github.com/sgl-project/sglang/pull/11868 * [router] remove encoding header for oai router by @slin1237 in https://github.com/sgl-project/sglang/pull/11881 * [Auto Sync] Update scheduler.py, server_args.py (20251020) by @merrymercy in https://github.com/sgl-project/sglang/pull/11875 * [router][grpc] Remove `continue_final_message` in `ChatTemplateParams` and add `minijinja-contrib` by @CatherineSue in https://github.com/sgl-project/sglang/pull/11882 * fix(sql-router): fix conflict port in test by @htiennv in https://github.com/sgl-project/sglang/pull/11826 * [router] clean up workflow logs to debug for implementation details logs by @slin1237 in https://github.com/sgl-project/sglang/pull/11886 * [code move] move pp into a separate mixin by @merrymercy in https://github.com/sgl-project/sglang/pull/11838 * [router][grpc] Fix wram-up random token ids for small models by @CatherineSue in https://github.com/sgl-project/sglang/pull/11887 * Revise MRotaryEmbedding's forward by @yuan-luo in https://github.com/sgl-project/sglang/pull/11859 * piecewise cuda graph support qwen3-moe by @BBuf in https://github.com/sgl-project/sglang/pull/11845 * Fix RotaryEmbedding for fp32 input by @zhangdonghao-zdh in https://github.com/sgl-project/sglang/pull/11843 * Init attention backend for Intel XPU by @airMeng in https://github.com/sgl-project/sglang/pull/10656 * Use trtllm_mla decode kernel for draft extend in speculative decoding by @Qiaolin-Yu in https://github.com/sgl-project/sglang/pull/11664 * [router] release router 0.2.1 by @slin1237 in https://github.com/sgl-project/sglang/pull/11885 * [AMD] Update wave-lang to 3.8.0 by @xintin in https://github.com/sgl-project/sglang/pull/11878 * init support for KTransformers Heterogeneous Computing by @Atream in https://github.com/sgl-project/sglang/pull/11487 * [FEATURE] Add OpenAI-Compatible LoRA Adapter Selection by @neelabhsinha in https://github.com/sgl-project/sglang/pull/11570 * [fix] fix ci uv install dependency by @HanHan009527 in https://github.com/sgl-project/sglang/pull/11895 * Support Thinking Budget (via custom_logit_processor for OpenAI API) [Fix #6572] by @whybeyoung in https://github.com/sgl-project/sglang/pull/11416 * Simplify multi-tokenizer by @zhengkezhou1 in https://github.com/sgl-project/sglang/pull/11295 * [CI] disable glm4.1v and fix the flashinfer installation by @ShangmingCai in https://github.com/sgl-project/sglang/pull/11902 * vlm: enforce pybase64 for image and str encode/decode by @b8zhong in https://github.com/sgl-project/sglang/pull/10700 * [smol] [perf] Inverse perm improvement by @vincentzed in https://github.com/sgl-project/sglang/pull/11482 * [quantization][MoE] fix the check for `tp_size` / `moe_ep_size` / `moe_intermediate_size` / `weight_block_size_n` by @kevin85421 in https://github.com/sgl-project/sglang/pull/11702 * [CI] Fix b200 flashinfer installation by @ShangmingCai in https://github.com/sgl-project/sglang/pull/11915 * Fix flush cache API for spec v2 by @hnyls2002 in https://github.com/sgl-project/sglang/pull/11918 * [NVIDIA] Add new SMs support for Spark & Thor by @Kh4L in https://github.com/sgl-project/sglang/pull/11287 * Update sgl-kernel and remove fast hadamard depedency by @Fridge003 in https://github.com/sgl-project/sglang/pull/11844 * Rename flashmla kernel options of nsa backend for better readability by @Fridge003 in https://github.com/sgl-project/sglang/pull/11876 * chore: upgrade flashinfer 0.4.1 by @zhyncs in https://github.com/sgl-project/sglang/pull/11933 * [BugFix][Qwen3-VL]: add metadata for video in qwen3-vl by @ZhengWG in https://github.com/sgl-project/sglang/pull/11377 * [Auto Sync] Update forward_batch_info.py (20251021) by @zhyncs in https://github.com/sgl-project/sglang/pull/11934 * Fix openai input_text type compatibility by @key4ng in https://github.com/sgl-project/sglang/pull/11935 * fix: resolve flashinfer 0.4.1 import by @zhyncs in https://github.com/sgl-project/sglang/pull/11940 * [router][grpc] Support `v1/responses` API by @CatherineSue in https://github.com/sgl-project/sglang/pull/11926 * [router] Add gRPC E2E test suite by @key4ng in https://github.com/sgl-project/sglang/pull/11790 * [router][grpc] Fix background tasks stored with wrong id by @CatherineSue in https://github.com/sgl-project/sglang/pull/11945 * [lint] improve ruff check by @hnyls2002 in https://github.com/sgl-project/sglang/pull/11922 * [sgl-kernel] support flashmla libtorch by @FlamingoPg in https://github.com/sgl-project/sglang/pull/11717 * [NVIDIA] upstream FA4 and fix cccl path by @johnnynunez in https://github.com/sgl-project/sglang/pull/11929 * Enable native ModelOpt quantization support (3/3) by @Edwardf0t1 in https://github.com/sgl-project/sglang/pull/10154 * Fix mooncake dispatcher by @UNIDY2002 in https://github.com/sgl-project/sglang/pull/11908 * [2/N] Added the core structure of elastic EP and the eplb algorithm with faulty rank by @HanHan009527 in https://github.com/sgl-project/sglang/pull/10606 * [model] Support POINTSV15Chat model by @josephydu in https://github.com/sgl-project/sglang/pull/9651 * Fix flaky hicache test with mooncake backend by @ShangmingCai in https://github.com/sgl-project/sglang/pull/11953 * [Fix] Remove unused import from triton_kernels_moe.py by @FlamingoPg in https://github.com/sgl-project/sglang/pull/11967 * [router] Support multiple worker URLs for OpenAI router by @key4ng in https://github.com/sgl-project/sglang/pull/11723 * [Documentation] add doc for deterministic inference by @zminglei in https://github.com/sgl-project/sglang/pull/11956 * [6/n]decouple quantization implementation from vLLM dependency by @Hongbosherlock in https://github.com/sgl-project/sglang/pull/10750 * [BUG] AttributeError: 'DeepEPMoE' object has no attribute 'use_w4a… by @yuho8818 in https://github.com/sgl-project/sglang/pull/11977 * Revert "Recapture cuda graph after model weight update to resolve IMA error " by @merrymercy in https://github.com/sgl-project/sglang/pull/11980 * [NVIDIA] Update to leverage flashinfer trtllm FP4 MOE throughput kernel by @jiahanc in https://github.com/sgl-project/sglang/pull/11563 * [router] create worker removal step and clean up worker manager by @slin1237 in https://github.com/sgl-project/sglang/pull/11921 * Implement BGE-M3 Sparse Embeddings in SGLang by @approximated-intelligence in https://github.com/sgl-project/sglang/pull/10869 * [Doc] Update deterministic inference flag in server_arguments.md by @Fridge003 in https://github.com/sgl-project/sglang/pull/11978 * [grpc] Support gRPC standard health check by @CatherineSue in https://github.com/sgl-project/sglang/pull/11955 * [AMD] Support a new flag to disable quant on parallelLinear layer if required by @yichiche in https://github.com/sgl-project/sglang/pull/11811 * [ROCm] Remove vLLM rope dependency & use AITER impl by @b8zhong in https://github.com/sgl-project/sglang/pull/11322 * [NVIDIA] Build CUDA 13 by @johnnynunez in https://github.com/sgl-project/sglang/pull/11299 * Bump grace blackwell DeepEP version by @fzyzcjy in https://github.com/sgl-project/sglang/pull/11990 * [CPU] misc updates by @ZailiWang in https://github.com/sgl-project/sglang/pull/11906 * fix(deepep): resolve benchmark failure on 4×IB-card setup by aligning tuning config with DeepEP commit bdd119f8 by @zheng1 in https://github.com/sgl-project/sglang/pull/11965 * [CPU] Optimize FP16 decode_attention_cpu by @blzheng in https://github.com/sgl-project/sglang/pull/10652 * Allow to disable batch decoding. by @LorrinWWW in https://github.com/sgl-project/sglang/pull/11944 * Fix incorrect KV indices creation when page_size=32 in TRTLLM MLA backend by @cicirori in https://github.com/sgl-project/sglang/pull/11985 * aiter update to v0.1.6.post1 by @HaiShaw in https://github.com/sgl-project/sglang/pull/12004 * Support overlap-spec-v2 with trtllm_mla attention backend by @Qiaolin-Yu in https://github.com/sgl-project/sglang/pull/11821 * Support nvidia/NVIDIA-Nemotron-Nano-9B-v2-FP8/NVFP4 by @netanel-haber in https://github.com/sgl-project/sglang/pull/11866 * [router] Add comprehensive E2E tests for Response API by @key4ng in https://github.com/sgl-project/sglang/pull/11988 * [Router] Consolidate ConnectionMode enum to core module by @YouNeedCryDear in https://github.com/sgl-project/sglang/pull/11937 * Move memory runtime checker to mixin class by @hnyls2002 in https://github.com/sgl-project/sglang/pull/12014 * Revert "Support nvidia/NVIDIA-Nemotron-Nano-9B-v2-FP8/NVFP4" by @hnyls2002 in https://github.com/sgl-project/sglang/pull/12015 * [Fix] memory leak by overlap + retract by @cctry in https://github.com/sgl-project/sglang/pull/11981 * [Feature] Support loading weights from ckpt engine worker by @stmatengss in https://github.com/sgl-project/sglang/pull/11755 * [router] change ci names and update log level in ci by @slin1237 in https://github.com/sgl-project/sglang/pull/12021 * Feature/nano v2 offline modelopt fp8 and nvfp4 by @netanel-haber in https://github.com/sgl-project/sglang/pull/12018 * [Auto Sync] Update test_deterministic_utils.py (20251023) by @merrymercy in https://github.com/sgl-project/sglang/pull/12022 * ci: fix night-ci with push retry mechanism by @mickqian in https://github.com/sgl-project/sglang/pull/11765 * [router][CI] Clean up imports and prints statements in sgl-router/py_test by @CatherineSue in https://github.com/sgl-project/sglang/pull/12024 * Add AWQ quantization support for NPU. by @ErvinXie in https://github.com/sgl-project/sglang/pull/10158 * model: support deepseek-ocr by @mickqian in https://github.com/sgl-project/sglang/pull/11891 * Log iteration # for prefill and decode by @nvcastet in https://github.com/sgl-project/sglang/pull/9366 * Revert "[ROCm] Remove vLLM rope dependency & use AITER impl" by @b8zhong in https://github.com/sgl-project/sglang/pull/12028 * Fix mamba radix cache eviction logic in `alloc_req_slots` by @rogeryoungh in https://github.com/sgl-project/sglang/pull/11616 * Update Github action title for kernel build by @Kangyan-Zhou in https://github.com/sgl-project/sglang/pull/12029 * [router] Add builder pattern for RouterConfig with zero duplication by @slin1237 in https://github.com/sgl-project/sglang/pull/12030 * Fixed aarch64 flash-mla by @nvjullin in https://github.com/sgl-project/sglang/pull/12009 * chore: bump SGLang version to 0.5.4 by @sglang-bot in https://github.com/sgl-project/sglang/pull/12027 ## New Contributors * @xuwenyihust made their first contribution in https://github.com/sgl-project/sglang/pull/11302 * @ziruiliu made their first contribution in https://github.com/sgl-project/sglang/pull/10969 * @scottjlee made their first contribution in https://github.com/sgl-project/sglang/pull/11144 * @Liu-congo made their first contribution in https://github.com/sgl-project/sglang/pull/11360 * @lulor made their first contribution in https://github.com/sgl-project/sglang/pull/8919 * @antoine-roux made their first contribution in https://github.com/sgl-project/sglang/pull/10577 * @QiuMike made their first contribution in https://github.com/sgl-project/sglang/pull/11465 * @ai-jz made their first contribution in https://github.com/sgl-project/sglang/pull/11419 * @neelabhsinha made their first contribution in https://github.com/sgl-project/sglang/pull/11413 * @UNIDY2002 made their first contribution in https://github.com/sgl-project/sglang/pull/10423 * @zhooooong made their first contribution in https://github.com/sgl-project/sglang/pull/11541 * @pdasgup made their first contribution in https://github.com/sgl-project/sglang/pull/11730 * @anvdn made their first contribution in https://github.com/sgl-project/sglang/pull/11519 * @Kindyaa made their first contribution in https://github.com/sgl-project/sglang/pull/11710 * @petricevich made their first contribution in https://github.com/sgl-project/sglang/pull/11555 * @tazjin made their first contribution in https://github.com/sgl-project/sglang/pull/11328 * @Paiiiiiiiiiiiiii made their first contribution in https://github.com/sgl-project/sglang/pull/11652 * @2015aroras made their first contribution in https://github.com/sgl-project/sglang/pull/11396 * @zhangdonghao-zdh made their first contribution in https://github.com/sgl-project/sglang/pull/11843 * @xintin made their first contribution in https://github.com/sgl-project/sglang/pull/11878 * @zhengkezhou1 made their first contribution in https://github.com/sgl-project/sglang/pull/11295 * @Kh4L made their first contribution in https://github.com/sgl-project/sglang/pull/11287 * @yuho8818 made their first contribution in https://github.com/sgl-project/sglang/pull/11977 * @jiahanc made their first contribution in https://github.com/sgl-project/sglang/pull/11563 * @approximated-intelligence made their first contribution in https://github.com/sgl-project/sglang/pull/10869 * @zheng1 made their first contribution in https://github.com/sgl-project/sglang/pull/11965 * @ErvinXie made their first contribution in https://github.com/sgl-project/sglang/pull/10158 * @rogeryoungh made their first contribution in https://github.com/sgl-project/sglang/pull/11616 * @nvjullin made their first contribution in https://github.com/sgl-project/sglang/pull/12009 **Full Changelog**: https://github.com/sgl-project/sglang/compare/v0.5.3...v0.5.4