## What's Changed * misc(gha): expose action cache url and runtime as secrets by @mfuntowicz in https://github.com/huggingface/text-generation-inference/pull/2964 * feat: support max_image_fetch_size to limit by @drbh in https://github.com/huggingface/text-generation-inference/pull/3339 * Maintenance mode by @LysandreJik in https://github.com/huggingface/text-generation-inference/pull/3344 * Maintenance mode by @LysandreJik in https://github.com/huggingface/text-generation-inference/pull/3345 * fix(num_devices): fix num_shard/num device auto compute when NVIDIA_VISIBLE_DEVICES == "all" or "void" by @oOraph in https://github.com/huggingface/text-generation-inference/pull/3346 **Full Changelog**: https://github.com/huggingface/text-generation-inference/compare/v3.3.6...v3.3.7

## What's Changed * Add missing backslash by @philsupertramp in https://github.com/huggingface/text-generation-inference/pull/3311 * Revert "feat: bump flake including transformers and huggingface_hub versions" by @drbh in https://github.com/huggingface/text-generation-inference/pull/3323 * fix: remove azure by @drbh in https://github.com/huggingface/text-generation-inference/pull/3325 * Fix mask passed to flashinfer by @danieldk in https://github.com/huggingface/text-generation-inference/pull/3324 * Update iframe sources for streaming demo by @coyotte508 in https://github.com/huggingface/text-generation-inference/pull/3327 * Revert "Revert "feat: bump flake including transformers and huggingfa… by @drbh in https://github.com/huggingface/text-generation-inference/pull/3326 * Revert "feat: bump flake including transformers and huggingface_hub versions" by @drbh in https://github.com/huggingface/text-generation-inference/pull/3330 * Patch version 3.3.6 by @tengomucho in https://github.com/huggingface/text-generation-inference/pull/3329 ## New Contributors * @philsupertramp made their first contribution in https://github.com/huggingface/text-generation-inference/pull/3311 * @coyotte508 made their first contribution in https://github.com/huggingface/text-generation-inference/pull/3327 **Full Changelog**: https://github.com/huggingface/text-generation-inference/compare/v3.3.5...v3.3.6

## What's Changed * [gaudi] Refine rope memory, do not need to keep sin/cos cache per layer by @sywangyi in https://github.com/huggingface/text-generation-inference/pull/3274 * Gaudi: add CI by @baptistecolle in https://github.com/huggingface/text-generation-inference/pull/3160 * [gaudi] Gemma3 sliding window support by @sywangyi in https://github.com/huggingface/text-generation-inference/pull/3280 * xpu lora support by @sywangyi in https://github.com/huggingface/text-generation-inference/pull/3232 * Optimum neuron 0.2.2 by @dacorvo in https://github.com/huggingface/text-generation-inference/pull/3281 * [gaudi] Remove unnecessary reinitialize to HeterogeneousNextTokenChooser to m… by @sywangyi in https://github.com/huggingface/text-generation-inference/pull/3284 * [gaudi] Deepseek v2 mla and add ep to unquantized moe by @sywangyi in https://github.com/huggingface/text-generation-inference/pull/3287 * [gaudi] Fix the CI test errors by @yuanwu2017 in https://github.com/huggingface/text-generation-inference/pull/3286 * Hpu gptq gidx support by @sywangyi in https://github.com/huggingface/text-generation-inference/pull/3297 * Migrate to V2 Pydantic interface by @emmanuel-ferdman in https://github.com/huggingface/text-generation-inference/pull/3262 * Xccl by @sywangyi in https://github.com/huggingface/text-generation-inference/pull/3252 * Multi modality fix by @sywangyi in https://github.com/huggingface/text-generation-inference/pull/3283 * some gptq case could not be handled by ipex. but could be handle by t… by @sywangyi in https://github.com/huggingface/text-generation-inference/pull/3298 * fix outline import issue by @sywangyi in https://github.com/huggingface/text-generation-inference/pull/3282 * HuggingFaceM4/Idefics3-8B-Llama3 crash fix by @sywangyi in https://github.com/huggingface/text-generation-inference/pull/3267 * Optimum neuron 0.3.0 by @tengomucho in https://github.com/huggingface/text-generation-inference/pull/3308 * Disable Cachix pushes by @danieldk in https://github.com/huggingface/text-generation-inference/pull/3312 * chore: prepare version 3.3.5 by @tengomucho in https://github.com/huggingface/text-generation-inference/pull/3314 * feat: bump flake including transformers and huggingface_hub versions by @drbh in https://github.com/huggingface/text-generation-inference/pull/3313 **Full Changelog**: https://github.com/huggingface/text-generation-inference/compare/v3.3.4...git

Neuron backend update. ## What's Changed * Remove useless packages by @yuanwu2017 in https://github.com/huggingface/text-generation-inference/pull/3253 * Bump neuron SDK version by @dacorvo in https://github.com/huggingface/text-generation-inference/pull/3260 * Perf opt by @sywangyi in https://github.com/huggingface/text-generation-inference/pull/3256 * [gaudi] Vlm rebase and issue fix in benchmark test by @sywangyi in https://github.com/huggingface/text-generation-inference/pull/3263 * Move the _update_cos_sin_cache into get_cos_sin by @yuanwu2017 in https://github.com/huggingface/text-generation-inference/pull/3254 * [Gaudi] Remove optimum-habana by @yuanwu2017 in https://github.com/huggingface/text-generation-inference/pull/3261 * [gaudi] HuggingFaceM4/idefics2-8b issue fix by @sywangyi in https://github.com/huggingface/text-generation-inference/pull/3264 * [Gaudi] Enable Qwen3_moe model by @yuanwu2017 in https://github.com/huggingface/text-generation-inference/pull/3244 * [Gaudi]Fix the integration-test issues by @yuanwu2017 in https://github.com/huggingface/text-generation-inference/pull/3265 * [Gaudi] use pad_token_id to pad input id by @sywangyi in https://github.com/huggingface/text-generation-inference/pull/3268 * chore: prepare release 3.3.3 by @dacorvo in https://github.com/huggingface/text-generation-inference/pull/3269 * [gaudi] Refine logging for Gaudi warmup by @regisss in https://github.com/huggingface/text-generation-inference/pull/3222 * doc: fix README by @dacorvo in https://github.com/huggingface/text-generation-inference/pull/3271 **Full Changelog**: https://github.com/huggingface/text-generation-inference/compare/v3.3.2...v3.3.3

## Notable changes * Prefill chunking for VLMs. ## What's Changed * Fixing Qwen 2.5 VL (32B). by @Narsil in https://github.com/huggingface/text-generation-inference/pull/3157 * Fixing tokenization like https://github.com/huggingface/text-embeddin… by @Narsil in https://github.com/huggingface/text-generation-inference/pull/3156 * Gaudi: clean cuda/rocm code in hpu backend, enable flat_hpu by @sywangyi in https://github.com/huggingface/text-generation-inference/pull/3113 * L4 fixes by @mht-sharma in https://github.com/huggingface/text-generation-inference/pull/3161 * setuptools <= 70.0 is vulnerable: CVE-2024-6345 by @Narsil in https://github.com/huggingface/text-generation-inference/pull/3171 * transformers flash llm/vlm enabling in ipex by @sywangyi in https://github.com/huggingface/text-generation-inference/pull/3152 * Upgrading the dependencies in Gaudi backend. by @Narsil in https://github.com/huggingface/text-generation-inference/pull/3170 * Hotfixing gaudi deps. by @Narsil in https://github.com/huggingface/text-generation-inference/pull/3174 * Hotfix gaudi2 with newer transformers. by @Narsil in https://github.com/huggingface/text-generation-inference/pull/3176 * Support flashinfer for Gemma3 prefill by @danieldk in https://github.com/huggingface/text-generation-inference/pull/3167 * Get opentelemetry trace id from request headers instead of creating a new trace by @kozistr in https://github.com/huggingface/text-generation-inference/pull/2648 * Bump `sccache` to 0.10.0 by @alvarobartt in https://github.com/huggingface/text-generation-inference/pull/3179 * Fixing CI by @Narsil in https://github.com/huggingface/text-generation-inference/pull/3184 * Add option to configure prometheus port by @mht-sharma in https://github.com/huggingface/text-generation-inference/pull/3187 * Warmup gaudi backend by @sywangyi in https://github.com/huggingface/text-generation-inference/pull/3172 * Put more wiggle room. by @Narsil in https://github.com/huggingface/text-generation-inference/pull/3189 * Fixing the router + template for Qwen3. by @Narsil in https://github.com/huggingface/text-generation-inference/pull/3200 * Skip `{% generation %}` and `{% endgeneration %}` template handling by @alvarobartt in https://github.com/huggingface/text-generation-inference/pull/3204 * doc typo by @julien-c in https://github.com/huggingface/text-generation-inference/pull/3206 * Pr 2982 ci branch by @drbh in https://github.com/huggingface/text-generation-inference/pull/3046 * fix: bump snaps for mllama by @drbh in https://github.com/huggingface/text-generation-inference/pull/3202 * Update client SDK snippets by @julien-c in https://github.com/huggingface/text-generation-inference/pull/3207 * Fix `HF_HUB_OFFLINE=1` for Gaudi backend by @regisss in https://github.com/huggingface/text-generation-inference/pull/3193 * IPEX support FP8 kvcache/softcap/slidingwindow by @sywangyi in https://github.com/huggingface/text-generation-inference/pull/3144 * forward and tokenize chooser use the same shape by @sywangyi in https://github.com/huggingface/text-generation-inference/pull/3196 * Chunked Prefill VLM by @mht-sharma in https://github.com/huggingface/text-generation-inference/pull/3188 * Prepare for 3.3.0 by @danieldk in https://github.com/huggingface/text-generation-inference/pull/3220 ## New Contributors * @kozistr made their first contribution in https://github.com/huggingface/text-generation-inference/pull/2648 * @julien-c made their first contribution in https://github.com/huggingface/text-generation-inference/pull/3206 **Full Changelog**: https://github.com/huggingface/text-generation-inference/compare/v3.2.3...v3.3.0

## Main changes - Patching Llama 4 ## What's Changed * Use ROCM 6.3.1 by @mht-sharma in https://github.com/huggingface/text-generation-inference/pull/3141 * Update transformers to 4.51 by @mht-sharma in https://github.com/huggingface/text-generation-inference/pull/3148 * Gaudi: Add Integration Test for Gaudi Backend by @baptistecolle in https://github.com/huggingface/text-generation-inference/pull/3142 * fix: compute type typo by @oOraph in https://github.com/huggingface/text-generation-inference/pull/3150 * 3.2.3 by @Narsil in https://github.com/huggingface/text-generation-inference/pull/3151 **Full Changelog**: https://github.com/huggingface/text-generation-inference/compare/v3.2.2...v3.2.3

## What's Changed * Minor fixes. by @Narsil in https://github.com/huggingface/text-generation-inference/pull/3125 * configurable termination timeout by @ErikKaum in https://github.com/huggingface/text-generation-inference/pull/3126 * CI: enable server tests for backends by @baptistecolle in https://github.com/huggingface/text-generation-inference/pull/3128 * Torch 2.6 by @Narsil in https://github.com/huggingface/text-generation-inference/pull/3134 * Gaudi: Fix llava-next and mllama crash issue by @yuanwu2017 in https://github.com/huggingface/text-generation-inference/pull/3127 * nix-v3.2.1 -> v3.2.1-nix by @co42 in https://github.com/huggingface/text-generation-inference/pull/3129 * Gaudi: Use exponential growth to replace BATCH_BUCKET_SIZE by @yuanwu2017 in https://github.com/huggingface/text-generation-inference/pull/3131 * Add llama4 by @mht-sharma in https://github.com/huggingface/text-generation-inference/pull/3145 * Preparing for release. by @Narsil in https://github.com/huggingface/text-generation-inference/pull/3147 ## New Contributors * @co42 made their first contribution in https://github.com/huggingface/text-generation-inference/pull/3129 **Full Changelog**: https://github.com/huggingface/text-generation-inference/compare/v3.2.1...v3.2.2

## What's Changed * Update to `kernels` 0.2.1 by @danieldk in https://github.com/huggingface/text-generation-inference/pull/3084 * Router: add `gemma3-text` model type by @danieldk in https://github.com/huggingface/text-generation-inference/pull/3107 * We need gcc during runtime to enable triton to compile kernels. by @Narsil in https://github.com/huggingface/text-generation-inference/pull/3103 * Release of Gaudi Backend for TGI by @baptistecolle in https://github.com/huggingface/text-generation-inference/pull/3091 * Fixing the docker build. by @Narsil in https://github.com/huggingface/text-generation-inference/pull/3108 * Make the Nix-based Docker container work on non-NixOS by @danieldk in https://github.com/huggingface/text-generation-inference/pull/3109 * xpu 2.6 update by @sywangyi in https://github.com/huggingface/text-generation-inference/pull/3051 * launcher: correctly get the head dimension for VLMs by @danieldk in https://github.com/huggingface/text-generation-inference/pull/3116 * Gaudi: Sync TGI with the latest changes from the TGI-Gaudi fork by @baptistecolle in https://github.com/huggingface/text-generation-inference/pull/3117 * Bug Fix: Sliding Window Attention by @mht-sharma in https://github.com/huggingface/text-generation-inference/pull/3112 * Publish nix docker image. by @Narsil in https://github.com/huggingface/text-generation-inference/pull/3122 * Prepare for patch release. by @Narsil in https://github.com/huggingface/text-generation-inference/pull/3124 * Intel docker. by @Narsil in https://github.com/huggingface/text-generation-inference/pull/3121 **Full Changelog**: https://github.com/huggingface/text-generation-inference/compare/v3.2.0...v3.2.1