ITADN

Tool-call parsers lose or corrupt data at streaming chunk boundaries (multiple detectors)

#31915Openv-code01 创建于 28 天前
V
v-code01commented
A sweep of `function_call/` found several streaming/parse bugs, all reproduced by feeding text to the real detector classes. Common root causes: (a) streaming state machines with no end-of-stream `_buffer` flush (`function_call_parser.parse_stream_chunk` never flushes, and `serving_chat._check_for_unstreamed_tool_args` only rescues the last tool's trailing args and bails when `prev_tool_call_arr` is empty); (b) hand-rolled brace/bracket/token counting that ignores string-literal context; (c) regex written for the happy path applied to the whole buffer / untrusted output. **Data loss (drops arguments or whole tool calls):** - `base_format_detector.parse_streaming_increment` (name-send branch, ~`:244-271`) emits the tool name and returns without emitting args, without populating `prev_tool_call_arr`, and without advancing `_buffer`. With coarse chunking (batched detok, `stream_interval>1`, speculative decode) 2 parallel tool calls lose data in **71% (Qwen25) / 91% (Hermes)** of random 1-4 chunk splits; a single tool call in one chunk streams `arguments: ""`. Affects Qwen25/Hermes/Mistral/Llama32/GLM4/DeepSeekV3. - `step3_detector.py:264,318`: `param_regex.finditer(invoke_part)` is not bounded by `</steptml:invoke>`, so with two calls the second is dropped and its params are grafted onto the first. - `kimik2_detector.py:394`: `_split_pending_start` range stops at 2, so a lone trailing `<` (prefix of both section/call begin tokens) is not held back -> the whole call is dropped and raw tokens leak to normal_text. **Corruption (emits invalid JSON / mangled text):** - `kimik2_detector.py:316-320`: when `<|tool_call_end|>` splits across a chunk, `args_full = buffer[args_start:]` includes the partial end-token prefix and streams it as argument content (never retracted) -> `json.loads` raises `Extra data`. - `deepseekv3_detector.py:114-118`: the greedy DOTALL streaming regex `...<sep>(.*)\n\`\`\`json...` spans two calls in one increment, so `tool_index 0` gets the second tool's name. - `llama32_detector.py:124-125`: `re.sub(r"'([^']*)':", ...)` over the whole buffer injects a `\"` inside JSON string values that contain `'x':` or `: 'x'`, truncating them (e.g. an arg echoing a dict literal). - `deepseekv3_detector.py:72-86` (+ v31): the per-call loop is in one `try`; a single malformed call jumps to `except`, discarding already-parsed valid calls and returning the raw markup as normal_text. **Prose corruption / off-by-one:** - `base_format_detector.py:160-161` / `mistral_detector.py:133-134`: `normal_text.replace(self.eot_token, "")` with Mistral `eot_token="]"` deletes every `]` in ordinary assistant text (`"item[0] done"` -> `"item[0 done"`). - `llama32_detector.py:72`: `idx += end + len(tool_call_separator)` assumes a `;` always follows a decoded object, so trailing normal text with no separator loses its first char. - `pythonic_detector._find_matching_bracket:113-133` counts `[`/`]` without respecting string literals, so a string arg containing `]` mis-slices the call and it is dropped in streaming. Each is individually reproducible against the real detector. Same family as the recently-fixed streaming-detok duplication -- streaming state that loses/corrupts data at chunk boundaries. Happy to send PRs for the self-contained ones (e.g. the Mistral `]` strip, the Llama32 first-char drop, the DeepSeek discard-all-on-error) if the direction is welcome. cc @JustinTong0323 @CatherineSue
1 条评论