P0: Missing @tool wrapper for multi-episode policy rollout — agent silently runs 1 mega-episode instead of N (data-integrity bug, 47 e2e runs affected)
bug
## Summary
The MolmoAct2 e2e test (`molmoact-e2e-2026-06-25/e2e_agent_test.py`) is **silently fabricating success**: every "20 episodes × 60 steps" run produces a dataset with **1 episode × ~1020-1200 frames** instead of 20 episodes × 60 frames each. Across **47 runs / 16 marked status=OK, ZERO actually contain 20 episodes**. This is a P0 data-integrity bug.
## Forensic Evidence (audit across 47 runs)
| Metric | Expected | Observed |
|---|---|---|
| `meta/info.json:total_episodes` | 20 | 1 |
| Frames per episode | 60 | 1020-1200 |
| `summary.status` | OK only if 20 eps | OK on all (false positive) |
| Pattern across 10/10 sampled OK runs | varied | **deterministic** |
Sample: run-1500 (20260626-150038) → exit clean, 40.9min, `status=OK`, but `meta/info.json:total_episodes=1`, single episode of 1140 frames.
## Root Cause (located)
1. ✅ `strands_robots/simulation/policy_runner.py:586` has a correct `for ep in range(n_episodes)` loop — **runner code is fine**.
2. ❌ **`strands_robots/tools/` does NOT expose `policy_runner` as a `@tool`.** Current tools:
- `lerobot_camera`, `lerobot_calibrate`, `serial_tool`, `robot_mesh`, `gr00t_inference`, `pose_tool`, `lerobot_teleoperate`, `download_assets`
- **No `run_policy`, `run_rollout`, `evaluate_policy` tool exists.**
3. The e2e test prompts an LLM agent in English: `"Run the allenai/MolmoAct2-SO100_101 policy for 20 episodes with 60 steps per episode"` (e2e_agent_test.py:278-280).
4. With no proper rollout tool, the agent improvises using low-level mujoco/sim primitives. It computes a single budget (~20×57≈1140 or 20×60=1200 steps) and runs **one long rollout**, never calling `env.reset()` or `dataset.save_episode()` between iterations.
5. `dataset_recorder` records this as ONE giant episode and reports success — no episode-count validation gates `status=OK`.
## Why it's silent
- Agent runs to completion without exception → `STATE["status"] = "OK"` (e2e_agent_test.py:303).
- Dataset summary says "OK", `summary.status=OK`, exit clean.
- The 20-vs-1 mismatch only surfaces when you read `meta/info.json:total_episodes` directly.
## Proposed Fix
### Primary (P0) — Add proper rollout `@tool`
Wrap `policy_runner.run(...)` (or whichever multi-episode entrypoint is canonical) as:
```python
@tool
def run_policy_rollout(
policy_id: str,
n_episodes: int,
max_steps: int,
task_description: str,
dataset_repo_id: str | None = None,
dataset_root: str | None = None,
) -> dict:
"""Run policy for N episodes with proper env.reset() and dataset.save_episode() per episode."""
```
Expose in `strands_robots/tools/__init__.py` so agents have a single-call path for multi-episode rollouts.
### Secondary (P1) — Validation gate in `dataset_recorder`
Before flipping `status=OK`, validate `meta/info.json:total_episodes == requested_n_episodes`. If not → `status=PARTIAL` or `status=FAIL` with a clear reason.
### Tertiary (P2) — Make e2e test call programmatically, not via LLM prompt
Replace the natural-language episode loop instruction with a direct `run_policy_rollout(...)` invocation, or assert the dataset has 20 episodes before exit 0.
## Reproduction
```bash
cd ~/molmoact-e2e-2026-06-25
python e2e_agent_test.py
# Wait ~40min
# Inspect: cat ~/molmoact-e2e-2026-06-25/runs/<latest>/dataset/meta/info.json | jq .total_episodes
# Observed: 1
# Expected: 20
```
## Acceptance Criteria
- [ ] `run_policy_rollout` `@tool` exists in `strands_robots/tools/` and is discoverable in `__init__.py`
- [ ] Tool correctly produces N episodes when invoked with `n_episodes=N`
- [ ] `dataset_recorder` validates episode count before `status=OK`
- [ ] Existing 47 historical "OK" runs in `molmoact-e2e-2026-06-25/runs/` are re-labeled `STRUCTURALLY_FAILED` in any leaderboard / summary
- [ ] e2e_agent_test.py either calls the new tool directly OR asserts `total_episodes == 20` before returning 0
## Mesh Audit Trail
Audited by 5-node DevDuck mesh (zenoh): cagatay-3bd04d (Thor / Linux aarch64), cagatay-6f1e53 (Mac), cagatay-dae9ce (Mac), cagatays-66641d (Mac mini), cagatays-37aeae (Mac mini). Consensus reached 16:16-16:19 UTC 2026-06-26 across 366 heartbeats. Backfill audit script ran at 16:11 covering 47 runs.
---
*Filed by DevDuck mesh consensus.*
3 条评论