Tool-call parsers lose or corrupt data at streaming chunk boundaries (multiple detectors)
A sweep of `function_call/` found several streaming/parse bugs, all reproduced by feeding text to the real detector classes. Common root causes: (a) streaming state machines with no end-of-stream `_buffer` flush (`function_call_parser.parse_stream_chunk` never flushes, and `serving_chat._check_for_unstreamed_tool_args` only rescues the last tool's trailing args and bails when `prev_tool_call_arr` is empty); (b) hand-rolled brace/bracket/token counting that ignores string-literal context; (c) regex written for the happy path applied to the whole buffer / untrusted output.
**Data loss (drops arguments or whole tool calls):**
- `base_format_detector.parse_streaming_increment` (name-send branch, ~`:244-271`) emits the tool name and returns without emitting args, without populating `prev_tool_call_arr`, and without advancing `_buffer`. With coarse chunking (batched detok, `stream_interval>1`, speculative decode) 2 parallel tool calls lose data in **71% (Qwen25) / 91% (Hermes)** of random 1-4 chunk splits; a single tool call in one chunk streams `arguments: ""`. Affects Qwen25/Hermes/Mistral/Llama32/GLM4/DeepSeekV3.
- `step3_detector.py:264,318`: `param_regex.finditer(invoke_part)` is not bounded by `</steptml:invoke>`, so with two calls the second is dropped and its params are grafted onto the first.
- `kimik2_detector.py:394`: `_split_pending_start` range stops at 2, so a lone trailing `<` (prefix of both section/call begin tokens) is not held back -> the whole call is dropped and raw tokens leak to normal_text.
**Corruption (emits invalid JSON / mangled text):**
- `kimik2_detector.py:316-320`: when `<|tool_call_end|>` splits across a chunk, `args_full = buffer[args_start:]` includes the partial end-token prefix and streams it as argument content (never retracted) -> `json.loads` raises `Extra data`.
- `deepseekv3_detector.py:114-118`: the greedy DOTALL streaming regex `...<sep>(.*)\n\`\`\`json...` spans two calls in one increment, so `tool_index 0` gets the second tool's name.
- `llama32_detector.py:124-125`: `re.sub(r"'([^']*)':", ...)` over the whole buffer injects a `\"` inside JSON string values that contain `'x':` or `: 'x'`, truncating them (e.g. an arg echoing a dict literal).
- `deepseekv3_detector.py:72-86` (+ v31): the per-call loop is in one `try`; a single malformed call jumps to `except`, discarding already-parsed valid calls and returning the raw markup as normal_text.
**Prose corruption / off-by-one:**
- `base_format_detector.py:160-161` / `mistral_detector.py:133-134`: `normal_text.replace(self.eot_token, "")` with Mistral `eot_token="]"` deletes every `]` in ordinary assistant text (`"item[0] done"` -> `"item[0 done"`).
- `llama32_detector.py:72`: `idx += end + len(tool_call_separator)` assumes a `;` always follows a decoded object, so trailing normal text with no separator loses its first char.
- `pythonic_detector._find_matching_bracket:113-133` counts `[`/`]` without respecting string literals, so a string arg containing `]` mis-slices the call and it is dropped in streaming.
Each is individually reproducible against the real detector. Same family as the recently-fixed streaming-detok duplication -- streaming state that loses/corrupts data at chunk boundaries. Happy to send PRs for the self-contained ones (e.g. the Mistral `]` strip, the Llama32 first-char drop, the DeepSeek discard-all-on-error) if the direction is welcome.
cc @JustinTong0323 @CatherineSue
1 条评论