[P0 data-corruption] Simulation.reset() does not flush dataset episode buffer → multi-episode runs collapse to 1 episode
bug
## Summary
`Simulation.reset()` does **not** flush the dataset recorder's episode buffer. When an agent (or any caller) records a dataset across multiple `reset → run_policy` cycles, all frames from N cycles concatenate into **one giant episode** with `total_episodes=1`.
## Repro
`molmoact-e2e-2026-06-25/e2e_agent_test.py` runs the natural-language prompt:
> "Run the allenai/MolmoAct2-SO100_101 policy for 20 episodes with 60 steps per episode..."
The agent correctly emits the structure (forensics from `agent_messages.json` of run `20260626-150038`):
```
[0] so101_sim: {action: 'reset'}
[2] so101_sim: {action: 'run_policy', n_steps: 60, instruction: 'pick up the red cube', ...}
[4] so101_sim: {action: 'reset'}
[6] so101_sim: {action: 'run_policy', n_steps: 60, ...}
... (10 reset+run_policy pairs total) ...
[32] so101_sim: {action: 'stop_recording'}
[34] so101_sim: {action: 'render', ...}
[36] so101_sim: {action: 'destroy'}
```
Dataset result (`runs/20260626-150038/dataset/meta/info.json`):
```json
{"total_episodes": 1, "total_frames": 1140}
```
Expected: `total_episodes=10, total_frames=600` (10 × 60). Got `1 × 1140` (one continuous concatenation; ~114 frames per cycle because n_substeps yields ~1.9 frames/step).
**Reproduced identically in `runs/20260626-140044`** — same fingerprint. This is a stable repro, not flaky.
## Root cause
`strands_robots/simulation/mujoco/simulation.py:1419`:
```python
def reset(self) -> dict[str, Any]:
# ...
with self._lock:
mj.mj_resetData(self._world._model, self._world._data)
self._world.sim_time = 0.0
self._world.step_count = 0
for r in self._world.robots.values():
r.policy_running = False
r.policy_steps = 0
return {"status": "success", "content": [{"text": "Reset to initial state."}]}
```
There is no `recorder.save_episode()` call. Comment at line 2387 confirms the design assumption: *"the caller will save_episode them"* — but no caller does. The Robot tool router does not expose `save_episode` as an action either, so the LLM has no path to emit it even if it knew to.
## Impact
- **P0 silent data corruption**: every multi-episode MolmoAct/π/SmolVLA/ACT eval run that uses `reset + run_policy + record` produces a dataset that lies about episode count.
- **MolmoAct2 has been wrongly suspected** of "not moving / not segmenting" for ~2 days — it actually ran the requested 10 episodes; the dataset just collapsed them.
- Any downstream training/eval that filters by `episode_index` gets garbage.
- Affects all sim backends? Need to check `newton/simulation.py` reset() too.
## Recommended fix (option A — auto-flush on reset)
Inject before `mj_resetData`:
```python
def reset(self) -> dict[str, Any]:
if self._world is None or self._world._model is None or self._world._data is None:
return {"status": "error", "content": [{"text": _NO_WORLD_MSG}]}
if err := self._require_no_running_policy("reset"):
return err
# NEW: episode boundary — flush buffered frames as a complete episode
# before clearing physics state.
if self._world._backend_state.get("recording") and self._recorder is not None:
if self._recorder.has_buffered_frames(): # add this helper
self._recorder.save_episode()
mj = self._mj
with self._lock:
mj.mj_resetData(...)
...
```
Rationale: "reset = new episode" is the LeRobot semantic users expect; explicit `save_episode` actions would require LLM prompt changes and are brittle. Auto-flush is backward compatible (no recording → no-op; no buffered frames → no-op).
## Alternative (option B): expose `save_episode` as a router action
Less invasive to existing reset() but requires updating `e2e_agent_test.py` prompt to instruct the agent: *"After each run_policy, before the next reset, call save_episode."* Trusts LLM compliance — not great for replayable benches.
## Acceptance criteria
- [ ] `reset()` auto-flushes buffered frames as a sealed episode when recording is active
- [ ] Add `has_buffered_frames()` helper to `DatasetRecorder` (or use existing `len(frames) > 0` accessor)
- [ ] Unit test: `reset()` between two `run_policy(60)` calls produces 2 episodes in `info.json`
- [ ] Re-run `molmoact-e2e-2026-06-25` end-to-end → expect `total_episodes=10` (or 20 if the secondary agent-iteration-budget issue is also fixed)
- [ ] Audit `newton/simulation.py` reset() for same bug
- [ ] Add CHANGELOG entry: silent data-corruption fix
## Secondary observation (separate ticket)
The agent did **10** reset/run_policy pairs, not 20 (prompt asked for 20). Likely max_iterations / token budget cap on the agent loop. File separately once the dataset bug is fixed, since we can't measure the true episode count until then.
## Evidence files
- `~/molmoact-e2e-2026-06-25/runs/20260626-150038/agent_messages.json` (39 messages, 19 tool calls, full pattern)
- `~/molmoact-e2e-2026-06-25/runs/20260626-150038/dataset/meta/info.json` (total_episodes=1, total_frames=1140)
- `~/molmoact-e2e-2026-06-25/runs/20260626-140044/` (identical pattern, prior day)
关闭于 2026-06-26 5 条评论