[Bug]: Guided decoding with Kimi-K2.5
bug
### System Info
CUDA 13.1
Python 3.12.3
TensorRT-LLM v1.3.0rc17 release container
4xB300-270GB GPUs
### Who can help?
_No response_
### Information
- [x] The official example scripts
- [x] My own modified scripts
### Tasks
- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [x] My own task or dataset (give details below)
### Reproduction
Launch command:
```
trtllm-serve nvidia/Kimi-K2.6-NVFP4 --tp_size=4 --backend=pytorch --host=0.0.0.0 \
--port=8000 --max_batch_size=96 --max_seq_len=262144 --ep_size=4 \
--max_num_tokens=16384 --max_seq_len=262144 \
--enable_chunked_prefill --trust_remote_code --max_batch_size=64 \
--extra_llm_api_options llm_extra.yaml
```
llm_extra.yaml:
```
print_iter_log: true
enable_iter_perf_stats: true
kv_cache_config:
enable_partial_reuse: false
sampler_type: auto
moe_config:
backend: CUTLASS
scheduler_config:
capacity_scheduler_policy: MAX_UTILIZATION
guided_decoding_backend: xgrammar
```
### Expected behavior
Engine properly launched
### actual behavior
Error log as follows (trimmed):
```
Skipping import of cpp extensions due to incompatible torch version 2.11.0a0+eb65b36914.nv26.02 for torchao version 0.15.0 Please see https://github.com/pytorch/ao/issues/2919 for more info
/usr/local/lib/python3.12/dist-packages/modelopt/torch/__init__.py:36: UserWarning: transformers version 5.5.4 is incompatible with nvidia-modelopt and may cause issues. Please install recommended version with `pip install nvidia-modelopt[hf]` if working with HF models.
_warnings.warn(
[TensorRT-LLM] TensorRT LLM version: 1.3.0rc17
/usr/local/lib/python3.12/dist-packages/tensorrt_llm/serve/openai_protocol.py:137: UserWarning: Field name "schema" in "ResponseFormat" shadows an attribute in parent "OpenAIBaseModel"
class ResponseFormat(OpenAIBaseModel):
[06/09/2026-22:31:33] [TRT-LLM] [W] [llmapi] Overriding moe_config
[06/09/2026-22:31:33] [TRT-LLM] [W] [llmapi] Overriding kv_cache_config
[06/09/2026-22:31:33] [TRT-LLM] [I] [llmapi] Using LLM with PyTorch backend
[06/09/2026-22:31:33] [TRT-LLM] [I] [llmapi] neither checkpoint_format nor checkpoint_loader were provided, checkpoint_format will be set to HF.
[06/09/2026-22:31:33] [TRT-LLM] [I] [llmapi] start MpiSession with 4 workers
[06/09/2026-22:31:33] [TRT-LLM] [I] [llmapi] Found /data/weights/pytrtllm-moonshotai--Kimi-K2.6-NVFP4/002/hf_quant_config.json, pre-quantized checkpoint is used.
[06/09/2026-22:31:33] [TRT-LLM] [W] [quantize] Inline 'config.json.quantization_config' diverges from 'hf_quant_config.json':
- group_size: file=16, inline=None
[06/09/2026-22:31:33] [TRT-LLM] [I] [llmapi] Setting quant_algo=NVFP4 from HF quant config.
[06/09/2026-22:31:33] [TRT-LLM] [I] [llmapi] Setting kv_cache_quant_algo=FP8 from HF quant config.
[06/09/2026-22:31:33] [TRT-LLM] [I] [llmapi] Setting group_size=16 from HF quant config.
[06/09/2026-22:31:33] [TRT-LLM] [I] [llmapi] Setting exclude_modules=['language_model.lm_head', 'language_model.model.layers.0.self_attn*', 'language_model.model.layers.... from HF quant config.
[06/09/2026-22:31:34] [TRT-LLM] [W] [tokenizr] cloudpickle is not installed. Objects from trust_remote_code dynamic modules may not be serializable across nodes in multi-node setups. Install cloudpickle to fix this: pip install cloudpickle
`torch_dtype` is deprecated! Use `dtype` instead!
[06/09/2026-22:31:34] [TRT-LLM] [W] [quantize] Inline 'config.json.quantization_config' diverges from 'hf_quant_config.json':
- group_size: file=16, inline=None
`KimiK25Processor` defines `image_processor_class = 'AutoImageProcessor'`, which is deprecated. Register the correct mapping in `AutoImageProcessor` instead.
The `use_fast` parameter is deprecated and will be removed in a future version. Use `backend="torchvision"` instead of `use_fast=True`, or `backend="pil"` instead of `use_fast=False`.
[06/09/2026-22:31:35] [TRT-LLM] [W] [executor] Orchestrator is creating IPC executor
[33;20mrank 0 using MpiPoolSession to spawn MPI processes
[0m[06/09/2026-22:31:35] [TRT-LLM] [I] [executor] Generating a new HMAC key for server proxy_request_queue
[06/09/2026-22:31:35] [TRT-LLM] [I] [executor] Generating a new HMAC key for server worker_init_status_queue
[06/09/2026-22:31:35] [TRT-LLM] [I] [executor] Generating a new HMAC key for server proxy_result_queue
Skipping import of cpp extensions due to incompatible torch version 2.11.0a0+eb65b36914.nv26.02 for torchao version 0.15.0 Please see https://github.com/pytorch/ao/issues/2919 for more info
Skipping import of cpp extensions due to incompatible torch version 2.11.0a0+eb65b36914.nv26.02 for torchao version 0.15.0 Please see https://github.com/pytorch/ao/issues/2919 for more info
Skipping import of cpp extensions due to incompatible torch version 2.11.0a0+eb65b36914.nv26.02 for torchao version 0.15.0 Please see https://github.com/pytorch/ao/issues/2919 for more info
Skipping import of cpp extensions due to incompatible torch version 2.11.0a0+eb65b36914.nv26.02 for torchao version 0.15.0 Please see https://github.com/pytorch/ao/issues/2919 for more info
Multiple distributions found for package optimum. Picked distribution: optimum
Multiple distributions found for package modelopt. Picked distribution: nvidia-modelopt
/usr/local/lib/python3.12/dist-packages/modelopt/torch/__init__.py:36: UserWarning: transformers version 5.5.4 is incompatible with nvidia-modelopt and may cause issues. Please install recommended version with `pip install nvidia-modelopt[hf]` if working with HF models.
_warnings.warn(
Multiple distributions found for package optimum. Picked distribution: optimum
Multiple distributions found for package optimum. Picked distribution: optimum
Multiple distributions found for package modelopt. Picked distribution: nvidia-modelopt
Multiple distributions found for package modelopt. Picked distribution: nvidia-modelopt
Multiple distributions found for package optimum. Picked distribution: optimum
Multiple distributions found for package modelopt. Picked distribution: nvidia-modelopt
/usr/local/lib/python3.12/dist-packages/modelopt/torch/__init__.py:36: UserWarning: transformers version 5.5.4 is incompatible with nvidia-modelopt and may cause issues. Please install recommended version with `pip install nvidia-modelopt[hf]` if working with HF models.
_warnings.warn(
/usr/local/lib/python3.12/dist-packages/modelopt/torch/__init__.py:36: UserWarning: transformers version 5.5.4 is incompatible with nvidia-modelopt and may cause issues. Please install recommended version with `pip install nvidia-modelopt[hf]` if working with HF models.
_warnings.warn(
/usr/local/lib/python3.12/dist-packages/modelopt/torch/__init__.py:36: UserWarning: transformers version 5.5.4 is incompatible with nvidia-modelopt and may cause issues. Please install recommended version with `pip install nvidia-modelopt[hf]` if working with HF models.
_warnings.warn(
[TensorRT-LLM] TensorRT LLM version: 1.3.0rc17
[TensorRT-LLM] TensorRT LLM version: 1.3.0rc17
[TensorRT-LLM] TensorRT LLM version: 1.3.0rc17
[TensorRT-LLM] TensorRT LLM version: 1.3.0rc17
/usr/local/lib/python3.12/dist-packages/tensorrt_llm/serve/openai_protocol.py:137: UserWarning: Field name "schema" in "ResponseFormat" shadows an attribute in parent "OpenAIBaseModel"
class ResponseFormat(OpenAIBaseModel):
/usr/local/lib/python3.12/dist-packages/tensorrt_llm/serve/openai_protocol.py:137: UserWarning: Field name "schema" in "ResponseFormat" shadows an attribute in parent "OpenAIBaseModel"
class ResponseFormat(OpenAIBaseModel):
/usr/local/lib/python3.12/dist-packages/tensorrt_llm/serve/openai_protocol.py:137: UserWarning: Field name "schema" in "ResponseFormat" shadows an attribute in parent "OpenAIBaseModel"
class ResponseFormat(OpenAIBaseModel):
/usr/local/lib/python3.12/dist-packages/tensorrt_llm/serve/openai_protocol.py:137: UserWarning: Field name "schema" in "ResponseFormat" shadows an attribute in parent "OpenAIBaseModel"
class ResponseFormat(OpenAIBaseModel):
[TRT-LLM] [I] [runtime][RANK 2] Refreshed the MPI local session
[TRT-LLM] [I] [runtime][RANK 1] Refreshed the MPI local session
[TRT-LLM] [I] [runtime][RANK 0] Refreshed the MPI local session
[TRT-LLM] [I] [runtime][RANK 3] Refreshed the MPI local session
[06/09/2026-22:31:55] [TRT-LLM] [W] [executor][RANK 0] Worker process 26182 is affined to run on the following CPUs: [0, 2, 4, 6, 8, 10, 12, 14, 16, 18, 20, 22, 24, 26, 28, 30, 32, 34, 36, 38, 40, 42, 44, 46, 48, 50, 52, 54, 56, 58, 60, 62, 64, 66, 68, 70, 72, 74, 76, 78, 80, 82, 84, 86, 88, 90, 92, 94, 96, 98, 100, 102, 104, 106, 108, 110, 112, 114, 116, 118, 120, 122, 124, 126, 128, 130, 132, 134, 136, 138, 140, 142, 144, 146, 148, 150, 152, 154, 156, 158, 160, 162, 164, 166, 168, 170, 172, 174, 176, 178, 180, 182, 184, 186, 188, 190, 192, 194, 196, 198, 200, 202, 204, 206, 208, 210, 212, 214, 216, 218, 220, 222, 224, 226, 228, 230, 232, 234, 236, 238, 240, 242, 244, 246, 248, 250, 252, 254] (subset of all logical CPUs). This may harm performance if set incorrectly.
[06/09/2026-22:31:55] [TRT-LLM] [W] [executor][RANK 0] Worker process 26182 has constrained CPU affinity but `TLLM_NUMA_AWARE_WORKER_AFFINITY` is not set. Removing CPU affinity constraints.
`torch_dtype` is deprecated! Use `dtype` instead!
`torch_dtype` is deprecated! Use `dtype` instead!
`torch_dtype` is deprecated! Use `dtype` instead!
[06/09/2026-22:31:55] [TRT-LLM] [W] [quantize][RANK 0] Inline 'config.json.quantization_config' diverges from 'hf_quant_config.json':
- group_size: file=16, inline=None
[06/09/2026-22:31:55] [TRT-LLM] [I] [_torch][RANK 0] Tokenizer not provided; loading from checkpoint for guided decoding
`torch_dtype` is deprecated! Use `dtype` instead!
[06/09/2026-22:31:56] [TRT-LLM] [I] [_torch][RANK 0] ATTENTION RUNTIME FEATURES: AttentionRuntimeFeatures(chunked_prefill=True, cache_reuse=True, has_speculative_draft_tokens=False, chunk_size=16384, chunked_prefill_buffer_batch_size=4)
[06/09/2026-22:31:56] [TRT-LLM] [W] [quantize][RANK 0] Inline 'config.json.quantization_config' diverges from 'hf_quant_config.json':
- group_size: file=16, inline=None
`KimiK25Processor` defines `image_processor_class = 'AutoImageProcessor'`, which is deprecated. Register the correct mapping in `AutoImageProcessor` instead.
The `use_fast` parameter is deprecated and will be removed in a future version. Use `backend="torchvision"` instead of `use_fast=True`, or `backend="pil"` instead of `use_fast=False`.
`KimiK25Processor` defines `image_processor_class = 'AutoImageProcessor'`, which is deprecated. Register the correct mapping in `AutoImageProcessor` instead.
The `use_fast` parameter is deprecated and will be removed in a future version. Use `backend="torchvision"` instead of `use_fast=True`, or `backend="pil"` instead of `use_fast=False`.
`KimiK25Processor` defines `image_processor_class = 'AutoImageProcessor'`, which is deprecated. Register the correct mapping in `AutoImageProcessor` instead.
The `use_fast` parameter is deprecated and will be removed in a future version. Use `backend="torchvision"` instead of `use_fast=True`, or `backend="pil"` instead of `use_fast=False`.
`KimiK25Processor` defines `image_processor_class = 'AutoImageProcessor'`, which is deprecated. Register the correct mapping in `AutoImageProcessor` instead.
The `use_fast` parameter is deprecated and will be removed in a future version. Use `backend="torchvision"` instead of `use_fast=True`, or `backend="pil"` instead of `use_fast=False`.
[06/09/2026-22:31:57] [TRT-LLM] [W] [quantize][RANK 0] Inline 'config.json.quantization_config' diverges from 'hf_quant_config.json':
- group_size: file=16, inline=None
[06/09/2026-22:31:57] [TRT-LLM] [I] [_torch][RANK 0] Validating KV Cache config against kv_cache_dtype="auto"
[06/09/2026-22:31:57] [TRT-LLM] [I] [_torch][RANK 0] KV cache quantization set to "auto". Using checkpoint KV quantization.
[06/09/2026-22:32:09] [TRT-LLM] [I] [_torch][RANK 0] Use 140.73 GB for model weights.
[06/09/2026-22:32:09] [TRT-LLM] [I] [_torch][RANK 0] Prefetching 550.24GB checkpoint files.
...
Loading weights: 100%|██████████| 1831/1831 [01:11<00:00, 25.50it/s]
Model init total -- 121.26s
Loading weights: 99%|█████████▉| 1818/1831 [01:12<00:00, 44.95it/s]
Loading weights: 100%|█████████▉| 1830/1831 [01:12<00:00, 50.47it/s]
Loading weights: 100%|██████████| 1831/1831 [01:12<00:00, 25.22it/s]
Model init total -- 121.54s
[06/09/2026-22:33:59] [TRT-LLM] [E] [executor][RANK 3] Failed to initialize executor on rank 3: 'KimiK25ForConditionalGeneration' object has no attribute 'vocab_size_padded'
[06/09/2026-22:33:59] [TRT-LLM] [E] [executor][RANK 3] Traceback (most recent call last):
File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/executor/worker.py", line 302, in worker_main
worker: GenerationExecutorWorker = worker_cls(
^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/executor/worker.py", line 67, in __init__
self.setup_engine()
File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/executor/base_worker.py", line 286, in setup_engine
self.engine = _create_py_executor(
^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/executor/base_worker.py", line 257, in _create_py_executor
_executor = create_executor(**args)
^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/_torch/pyexecutor/py_executor_creator.py", line 711, in create_py_executor
"vocab_size_padded": model_engine.model.vocab_size_padded,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1967, in __getattr__
raise AttributeError(
AttributeError: 'KimiK25ForConditionalGeneration' object has no attribute 'vocab_size_padded'
[06/09/2026-22:34:00] [TRT-LLM] [E] [executor][RANK 1] Failed to initialize executor on rank 1: 'KimiK25ForConditionalGeneration' object has no attribute 'vocab_size_padded'
[06/09/2026-22:34:00] [TRT-LLM] [E] [executor][RANK 1] Traceback (most recent call last):
File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/executor/worker.py", line 302, in worker_main
worker: GenerationExecutorWorker = worker_cls(
^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/executor/worker.py", line 67, in __init__
self.setup_engine()
File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/executor/base_worker.py", line 286, in setup_engine
self.engine = _create_py_executor(
^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/executor/base_worker.py", line 257, in _create_py_executor
_executor = create_executor(**args)
^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/_torch/pyexecutor/py_executor_creator.py", line 711, in create_py_executor
"vocab_size_padded": model_engine.model.vocab_size_padded,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1967, in __getattr__
raise AttributeError(
AttributeError: 'KimiK25ForConditionalGeneration' object has no attribute 'vocab_size_padded'
[06/09/2026-22:34:00] [TRT-LLM] [E] [executor] Executor worker died during initialization.
[06/09/2026-22:34:02] [TRT-LLM] [E] [executor][RANK 2] Failed to initialize executor on rank 2: 'KimiK25ForConditionalGeneration' object has no attribute 'vocab_size_padded'
[06/09/2026-22:34:02] [TRT-LLM] [E] [executor][RANK 2] Traceback (most recent call last):
File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/executor/worker.py", line 302, in worker_main
worker: GenerationExecutorWorker = worker_cls(
^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/executor/worker.py", line 67, in __init__
self.setup_engine()
File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/executor/base_worker.py", line 286, in setup_engine
self.engine = _create_py_executor(
^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/executor/base_worker.py", line 257, in _create_py_executor
_executor = create_executor(**args)
^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/_torch/pyexecutor/py_executor_creator.py", line 711, in create_py_executor
"vocab_size_padded": model_engine.model.vocab_size_padded,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1967, in __getattr__
raise AttributeError(
AttributeError: 'KimiK25ForConditionalGeneration' object has no attribute 'vocab_size_padded'
[06/09/2026-22:34:02] [TRT-LLM] [E] [executor][RANK 0] Failed to initialize executor on rank 0: 'KimiK25ForConditionalGeneration' object has no attribute 'vocab_size_padded'
[06/09/2026-22:34:02] [TRT-LLM] [E] [executor][RANK 0] Traceback (most recent call last):
File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/executor/worker.py", line 302, in worker_main
worker: GenerationExecutorWorker = worker_cls(
^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/executor/worker.py", line 67, in __init__
self.setup_engine()
File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/executor/base_worker.py", line 286, in setup_engine
self.engine = _create_py_executor(
^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/executor/base_worker.py", line 257, in _create_py_executor
_executor = create_executor(**args)
^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/_torch/pyexecutor/py_executor_creator.py", line 711, in create_py_executor
"vocab_size_padded": model_engine.model.vocab_size_padded,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1967, in __getattr__
raise AttributeError(
AttributeError: 'KimiK25ForConditionalGeneration' object has no attribute 'vocab_size_padded'
[06/09/2026-22:34:07] [TRT-LLM] [E] [executor][RANK 0] Failed to deliver error message to proxy
Traceback (most recent call last):
File "/usr/local/bin/trtllm-serve", line 6, in <module>
sys.exit(main())
^^^^^^
File "/usr/local/lib/python3.12/dist-packages/click/core.py", line 1524, in __call__
return self.main(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/click/core.py", line 1445, in main
rv = self.invoke(ctx)
^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/click/core.py", line 1912, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/click/core.py", line 1308, in invoke
return ctx.invoke(self.callback, **ctx.params)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/click/core.py", line 877, in invoke
return callback(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/commands/serve.py", line 1063, in serve
_serve_llm()
File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/commands/serve.py", line 1037, in _serve_llm
launch_server(host,
File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/commands/serve.py", line 331, in launch_server
llm = PyTorchLLM(**llm_args)
^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/llmapi/llm.py", line 1572, in __init__
super().__init__(model, tokenizer, tokenizer_mode, skip_tokenizer_init,
File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/llmapi/llm.py", line 1412, in __init__
super().__init__(model,
File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/llmapi/llm.py", line 271, in __init__
self._build_model()
File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/llmapi/llm.py", line 1519, in _build_model
self._executor = self._executor_cls.create(
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/executor/executor.py", line 584, in create
return GenerationExecutor._create_ipc_executor(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/executor/executor.py", line 500, in _create_ipc_executor
return GenerationExecutorProxy(
^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/executor/proxy.py", line 150, in __init__
self._start_executor_workers(worker_kwargs)
File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/executor/proxy.py", line 431, in _start_executor_workers
raise RuntimeError("Executor worker died during initialization")
RuntimeError: Executor worker died during initialization
```
### additional notes
None
### Before submitting a new issue...
- [x] Make sure you already searched for relevant issues, and checked the [documentation](https://nvidia.github.io/TensorRT-LLM/) and [examples](https://github.com/NVIDIA/TensorRT-LLM/tree/main/examples) for answers to frequently asked questions.
0 条评论