ITADN

[zephyr] Default coordinator exits 137 with 4,575 path results

#8186Openravwojdyla-agent 创建于 8 天前
bugagent-generated
The 1 GiB default coordinator exits 137 near the end of a 4,575-shard map stage. Each shard returns one `(shard_path, None)` tuple. The failure starts an automatic retry after workers save all shard output. Reproduce: 1. Execute a 4,575-shard map stage that returns one `(shard_path, None)` tuple for each shard. 2. Use the [default coordinator resources](https://github.com/marin-community/marin/blob/9fec04fecc2af1d941304dc8ffff7f067c246564/lib/zephyr/src/zephyr/execution.py#L262-L264) and set `max_execution_retries=0`. 3. Observe `4574/4575 tasks completed`, followed by coordinator exit 137. Iris records one failure and no preemption. Expected: The default coordinator completes this small-payload result set. If 1 GiB is insufficient, Zephyr sets RAM from expected stage size or rejects the configuration before execution. The [default-resource reproduction](https://iris.oa.dev/#/job/%2Frav%2Fdatakit-token-store-default-coord-repro-20260812) reused saved shard markers and did not write token data. The same stage [completed with 8 GiB](https://iris.oa.dev/#/job/%2Frav%2Fdatakit-token-store-smoke-nemotron3-focus-full-coord8g-fed-20260812). Kubernetes removed the pod before inspection, so the direct `OOMKilled` reason is not available. The [incident record](https://echo.oa.dev/wiki/106) contains the full evidence. Related issue #8025 changes how callers pass coordinator resources. It does not address the default RAM or coordinator memory use.
2 条评论