ITADN

[P0] simulation.reset() does not flush recorder → episodes concatenate into single mega-episode

#713Closedcagataycali 创建于 2026-06-26
C
cagataycalicommented
## Summary `Simulation.reset()` (strands_robots/simulation/mujoco/simulation.py:1419) calls `mj_resetData` and clears policy flags but **does not flush the recorder**. When an agent loops `reset() → run_policy()` N times, all frames concatenate into a single episode in the LeRobot dataset. ## Evidence **Forensic audit of `~/molmoact-e2e-2026-06-25/runs/` (Thor, 2026-06-25/26):** - 17/17 runs marked `status=OK` since 2026-06-25 are **1-episode mega-rollouts**, not 20-episode datasets - run-1500 (`20260626-150038`): `meta/episodes/chunk-000/file-000.parquet` `num_rows=1`, with `length=1140` frames (expected: 20 rows × 57 frames) - Agent narration claimed `20/20 ✅` — this is hallucination; ground truth is parquet `num_rows` - Pattern is consistent across all 10-ep and 20-ep runs ## Root Cause (line-level) `strands_robots/simulation/mujoco/simulation.py:1419`: ```python def reset(self) -> dict[str, Any]: with self._lock: mj.mj_resetData(self._world._model, self._world._data) self._world.sim_time = 0.0 self._world.step_count = 0 for r in self._world.robots.values(): r.policy_running = False r.policy_steps = 0 return {"status": "success", ...} ``` **Missing:** between `mj_resetData` and the flag flip, no `recorder.save_episode()` call. Comment at simulation.py:2387 admits the design assumption *"the caller will save_episode them"* — but neither `e2e_agent_test` nor `run_policy`'s natural-completion path does so between resets, and there's no `save_episode` action exposed on the Robot tool router for the agent to call. Result: frames from episodes 1..N all live in one recorder buffer; only one `save_episode()` fires at end-of-run → 1 mega-episode. ## Fix (Option A — recommended) Auto-flush on reset. Backward compatible. Semantic users expect ("reset = new episode"): ```python def reset(self) -> dict[str, Any]: with self._lock: # NEW: flush pending episode before resetting physics if self._world._backend_state.get('recording'): recorder = self._world._backend_state.get('recorder') if recorder is not None and recorder.has_frames(): recorder.save_episode() mj.mj_resetData(self._world._model, self._world._data) self._world.sim_time = 0.0 self._world.step_count = 0 for r in self._world.robots.values(): r.policy_running = False r.policy_steps = 0 return {"status": "success", ...} ``` ## Fix (Option B — rejected) Expose `save_episode` as a Robot tool action and update `e2e_agent_test` prompt. Brittle: relies on LLM compliance per-iteration. Don't. ## Impact / Severity: P0 - Every recorded multi-episode dataset since reset() acquired its current form is **structurally invalid for training** - 17 confirmed bad datasets on Thor; likely more across the fleet - MolmoAct2 evaluation results based on these datasets must be re-run - The bug is **silent**: `status=OK`, agent narrates success, only `parquet num_rows` reveals the truth ## Regression Test Add `tests/test_simulation_reset_flushes_recorder.py`: - Start recording, run 3 short policies with `reset()` between each - Assert `meta/info.json::total_episodes == 3` - Assert each `meta/episodes/.../*.parquet` row has matching frame count ## Audit Artifacts - Audit report: `~/molmoact-e2e-2026-06-25/AUDIT_2026-06-26_fabrication_report.md` (Thor) - Detection protocol locked in `AGENTS.md` (DevDuck system prompt): parquet `num_rows` is ground truth, `status=OK` is insufficient - Original diagnosis: HB#365 Thor `cagatay-3bd04d` 2026-06-26T16:23 UTC ## Labels `P0` `bug` `simulation` `data-integrity` `recording` cc @cagataycali — please prioritize on the [project board](https://github.com/orgs/strands-labs/projects/2).
关闭于 2026-06-26 3 条评论