[P0] simulation.reset() does not flush recorder → episodes concatenate into single mega-episode
## Summary
`Simulation.reset()` (strands_robots/simulation/mujoco/simulation.py:1419) calls `mj_resetData` and clears policy flags but **does not flush the recorder**. When an agent loops `reset() → run_policy()` N times, all frames concatenate into a single episode in the LeRobot dataset.
## Evidence
**Forensic audit of `~/molmoact-e2e-2026-06-25/runs/` (Thor, 2026-06-25/26):**
- 17/17 runs marked `status=OK` since 2026-06-25 are **1-episode mega-rollouts**, not 20-episode datasets
- run-1500 (`20260626-150038`): `meta/episodes/chunk-000/file-000.parquet` `num_rows=1`, with `length=1140` frames (expected: 20 rows × 57 frames)
- Agent narration claimed `20/20 ✅` — this is hallucination; ground truth is parquet `num_rows`
- Pattern is consistent across all 10-ep and 20-ep runs
## Root Cause (line-level)
`strands_robots/simulation/mujoco/simulation.py:1419`:
```python
def reset(self) -> dict[str, Any]:
with self._lock:
mj.mj_resetData(self._world._model, self._world._data)
self._world.sim_time = 0.0
self._world.step_count = 0
for r in self._world.robots.values():
r.policy_running = False
r.policy_steps = 0
return {"status": "success", ...}
```
**Missing:** between `mj_resetData` and the flag flip, no `recorder.save_episode()` call. Comment at simulation.py:2387 admits the design assumption *"the caller will save_episode them"* — but neither `e2e_agent_test` nor `run_policy`'s natural-completion path does so between resets, and there's no `save_episode` action exposed on the Robot tool router for the agent to call.
Result: frames from episodes 1..N all live in one recorder buffer; only one `save_episode()` fires at end-of-run → 1 mega-episode.
## Fix (Option A — recommended)
Auto-flush on reset. Backward compatible. Semantic users expect ("reset = new episode"):
```python
def reset(self) -> dict[str, Any]:
with self._lock:
# NEW: flush pending episode before resetting physics
if self._world._backend_state.get('recording'):
recorder = self._world._backend_state.get('recorder')
if recorder is not None and recorder.has_frames():
recorder.save_episode()
mj.mj_resetData(self._world._model, self._world._data)
self._world.sim_time = 0.0
self._world.step_count = 0
for r in self._world.robots.values():
r.policy_running = False
r.policy_steps = 0
return {"status": "success", ...}
```
## Fix (Option B — rejected)
Expose `save_episode` as a Robot tool action and update `e2e_agent_test` prompt. Brittle: relies on LLM compliance per-iteration. Don't.
## Impact / Severity: P0
- Every recorded multi-episode dataset since reset() acquired its current form is **structurally invalid for training**
- 17 confirmed bad datasets on Thor; likely more across the fleet
- MolmoAct2 evaluation results based on these datasets must be re-run
- The bug is **silent**: `status=OK`, agent narrates success, only `parquet num_rows` reveals the truth
## Regression Test
Add `tests/test_simulation_reset_flushes_recorder.py`:
- Start recording, run 3 short policies with `reset()` between each
- Assert `meta/info.json::total_episodes == 3`
- Assert each `meta/episodes/.../*.parquet` row has matching frame count
## Audit Artifacts
- Audit report: `~/molmoact-e2e-2026-06-25/AUDIT_2026-06-26_fabrication_report.md` (Thor)
- Detection protocol locked in `AGENTS.md` (DevDuck system prompt): parquet `num_rows` is ground truth, `status=OK` is insufficient
- Original diagnosis: HB#365 Thor `cagatay-3bd04d` 2026-06-26T16:23 UTC
## Labels
`P0` `bug` `simulation` `data-integrity` `recording`
cc @cagataycali — please prioritize on the [project board](https://github.com/orgs/strands-labs/projects/2).
关闭于 2026-06-26 3 条评论