ITADN

0.84.3 regression: reasoning retention/replay bloats sessions; with subagents leads to 20GB+ parent OOM

#8746Closedgmanzienko 创建于 1 天前
untriaged
G
gmanzienkocommented
### Environment - pi: `@earendil-works/pi-coding-agent` **0.84.3** (also verified on 0.84.2 where the retention half is absent) - node v26.5.1, Arch Linux, 32GB RAM + 4GB zram - Heavy use of `pi-subagents` (0.56.0 and 0.58.0 — identical behavior, A/B tested) - Providers: `xai` (grok-4.6, direct) and a custom OpenAI-completions proxy (GLM-5.3, gpt-5.6-sol, grok-4.6), thinking level `xhigh` ### Summary Since updating pi 0.84.2 → 0.84.3 (Aug 25), every long-running session that orchestrates subagents gets killed by the kernel OOM killer. Five kills in three days, each a `node-MainThread` pi process at **21–27 GB anon RSS** while the persisted session on disk is only **0.5–2.4 MB**. Rolling back to pi **0.84.2** + pi-subagents **0.50.0** (the exact pre-update pair) makes the same workloads (same `xhigh` thinking) stable again. ### What changed in 0.84.3 (suspected root cause) Two changelog entries in 0.84.3 both cause assistant reasoning to be **retained in the session and resent on every subsequent request**, instead of being dropped after each turn: - "Changed inherited built-in xAI models to use the Responses API with encrypted reasoning replay" (#8124) - "Fixed inherited OpenAI-compatible Chat Completions reasoning replay to preserve and resend assistant-level `reasoning_details` verbatim and in order" (#7994) Measured on real sessions of the same project, same thinking level: | era | pi | avg assistant msg | msgs with retained reasoning | |---|---|---|---| | Aug 15–25 (stable) | 0.84.2 | 1.8 KB | **0 / 160** | | Aug 26–27 (OOM kills) | 0.84.3 | 4.0–5.7 KB | **296 / 298** (encrypted/reasoning_content) | Sessions created via the OpenAI-completions path (GLM-5.3, gpt-5.6-sol through a proxy) show the same retention: every thinking block carries `thinkingSignature: "reasoning_content"` (e.g. 29/29, 32/32 messages). So both adapter changes are implicated, not just the xAI one. ### Why it turns into 20GB OOM with subagents 1. pi-subagents spawns children as `pi --mode json -p`. In json mode, the child's final **`agent_end` event serializes the entire session as a single line**. I measured: a 30-turn child conversation of ~60 MB produces a **57 MB single-line `agent_end`** (stdout total 304 MB for 60 MB of conversation). This behavior is identical on 0.84.2 and 0.84.3 — the protocol is not new. 2. What is new in 0.84.3 is that **real sessions actually get that fat**, because reasoning is no longer dropped. Children hit pi-subagents' 16 MiB live-parser cap (`protocol_output_limit: child stdout line exceeded 16777216 bytes`), the run fails, the parent resumes/revives the child (now with an even bigger session), and the cycle repeats. 3. Additionally, the parent retains roughly **7–11 MB per subagent round** (measured on a synthetic loop, both pi-subagents 0.56 and 0.58) where the round's own content is ~2 MB. Over hours this accumulates. Kernel evidence (representative): ``` Out of memory: Killed process 486746 (node-MainThread) total-vm:25857632kB, anon-rss:22473464kB task_memcg=/user.slice/.../app-ghostty-surface-transient-424241.scope ``` Five kills total, all after the Aug 25 update; zero in the ten days before it on 0.84.2. One killed parent had run **22.5 GB in 9 minutes** with only ~1 min CPU; its on-disk transcript was 0.47 MB / 79 messages. ### Reproduction (no API costs) A local OpenAI-compatible SSE mock (streams N reasoning deltas per turn, drives the agent loop with tool calls) + `systemd-run --user --scope -p MemoryMax=3G`: - Parent orchestrating one 30-turn child with a 60 MB conversation (2 MB reasoning per turn) produces a **2.6–3.2 GB transient memory peak in the parent's cgroup** — a 40–50× amplification of the child's data — and the child's run fails on the 16 MiB line cap exactly as in production. - `pi --mode json -p` against the same mock: 57 MB `agent_end` line, 304 MB total stdout, 617–747 MB peak RSS for the child process alone. - Baseline control (no subagents, plain tool loop): memory grows ~linearly with conversation only — clean. ### Related issues - #8648 / PR #8671 — O(n²) `reasoning_details` accumulation in the openai-completions stream handler (fixed Aug 26, **not yet released** — npm latest is still 0.84.3 at the time of writing). This is likely the main per-chunk memory/CPU engine behind the explosions reported here. - #8711 — GLM-5.3 per-token `reasoning_details` object bloat (TUI freeze while streaming thinking). - #8619 — per-token `reasoning_details` stored unmerged in `thinkingSignature`. - #8714 — proposal to drop reasoning older than the last user message (closed Not planned). - #7994 / #8124 — the 0.84.3 changes that introduced reasoning retention/replay on the openai-completions and xAI paths. This report adds the production end-to-end consequence those issues don't cover: kernel OOM kills of 20GB+ processes through long sessions and subagent orchestration, the version bisect proving the 0.84.2→0.84.3 regression window, and the `agent_end` full-session single-line emission that turns fat sessions into child failures. ### Workaround Downgrade to the last-known-good pair: `@earendil-works/pi-coding-agent@0.84.2` + `pi-subagents@0.50.0` (pinned exact). Same models, same `xhigh` thinking, stable again. ### Suggestions - Make reasoning retention opt-in (or bounded) per provider/model, especially for OpenAI-completions proxies where `reasoning_content`/`reasoning_details` replay is often not required for correctness. - In `--mode json`, avoid emitting the full session inside a single `agent_end` line (chunk it, or emit a reference) — even without the retention regression, any long child crosses the 16 MiB line cap that consumers like pi-subagents impose. - Consider documenting that reasoning replay multiplies request payload size O(n²)-ish over a session (each request resends all accumulated reasoning). Happy to share the full repro stand (mock server + driver scripts, ~250 lines) if useful.
关闭于 1 天前 1 条评论