{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "7fb27b941602401d91542211134fc71a",
   "metadata": {},
   "source": [
    "# From Hugging Face Hub to robot hardware\n",
    "\n",
    "Notebook companion to the blog post of the same name. Walks through the LeRobot integration in **Strands Robots** end to end:\n",
    "\n",
    "1. Build a Strands agent (Claude Opus 4.8 on Bedrock by default) with the LeRobot AgentTools and a `Robot('so100')` simulation.\n",
    "2. Record a demonstration as a `LeRobotDataset` (local cache, optional Hub push).\n",
    "3. Run a policy on the same robot, ask the agent to render the final frame.\n",
    "4. (Reference only) Deploy the same agent code to a physical SO-101 by flipping `mode='real'`.\n",
    "5. Broadcast a command to every peer on the local Zenoh mesh.\n",
    "\n",
    "The defaults run fully in simulation with the Mock policy: no GPU, no Docker, no Hub credentials required. To use a trained policy or push to your Hub account, see the notes in the relevant cells.\n",
    "\n",
    "**Repository:** https://github.com/strands-labs/robots  \n",
    "**Blog post:** *From Hugging Face Hub to robot hardware with Strands Agents and LeRobot*"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "acae54e37e7d407bbb7b55eff062a284",
   "metadata": {},
   "source": [
    "## Prerequisites\n",
    "\n",
    "```bash\n",
    "pip install \"strands-robots[sim-mujoco,lerobot,mesh]\" strands-agents\n",
    "```\n",
    "\n",
    "Python 3.12+. The first run downloads MuJoCo Menagerie assets for the SO-100 (~30 seconds, one-time).\n",
    "\n",
    "Environment variables (dev/lab posture for the local mesh):\n",
    "\n",
    "```bash\n",
    "export STRANDS_MESH_LOCAL_DEV=1\n",
    "```\n",
    "\n",
    "AWS Bedrock access for Claude Opus 4.8 (`global.anthropic.claude-opus-4-8`). If your access is in a different region or account, override `MODEL_ID` and `AWS_REGION` in the next cell."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9a63283cbaf04dbcab1f6479b197f3a8",
   "metadata": {},
   "outputs": [],
   "source": [
    "import logging\n",
    "import os\n",
    "from pathlib import Path\n",
    "\n",
    "logging.basicConfig(level=logging.INFO, format=\"%(asctime)s [%(name)s] %(levelname)s: %(message)s\", datefmt=\"%H:%M:%S\")\n",
    "\n",
    "# Dev/lab mesh posture so Step 5 has a mesh to talk to.\n",
    "os.environ.setdefault(\"STRANDS_MESH_LOCAL_DEV\", \"1\")\n",
    "\n",
    "# Bedrock model + region. Verify the exact ID in your AWS Bedrock console.\n",
    "# AWS_REGION resolves from your AWS environment; set it explicitly here\n",
    "# (e.g. \"us-east-1\") if you need to override.\n",
    "MODEL_ID = \"global.anthropic.claude-opus-4-8\"\n",
    "AWS_REGION = os.environ.get(\"AWS_REGION\") or os.environ.get(\"AWS_DEFAULT_REGION\")\n",
    "\n",
    "# Recording target. Set HF_USER to push the dataset to your Hub account.\n",
    "HF_USER = None  # e.g. \"my_username\"\n",
    "DATASET_NAME = \"strands-cube-pick\"\n",
    "REPO_ID = f\"{HF_USER}/{DATASET_NAME}\" if HF_USER else f\"local/{DATASET_NAME}\"\n",
    "PUSH_TO_HUB = HF_USER is not None\n",
    "TASK = \"pick up the red cube\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8dd0d8092fe74a7c96281538738b07e2",
   "metadata": {},
   "source": [
    "## Step 1: Build the agent\n",
    "\n",
    "One `Robot('so100')` call returns a MuJoCo simulation by default. Pass `mode='real'` to get a hardware-backed robot driven by LeRobot, and the rest of the workflow is identical.\n",
    "\n",
    "The agent gets two tools: the `Robot` itself (handles simulation or hardware uniformly) and `robot_mesh` (peer-to-peer fleet messaging). LeRobot's CLIs handle calibration and hardware teleop outside this notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "72eea5119410473aa328ad9291626812",
   "metadata": {},
   "outputs": [],
   "source": [
    "from strands import Agent\n",
    "from strands.models import BedrockModel\n",
    "\n",
    "from strands_robots import Robot, robot_mesh\n",
    "\n",
    "model = BedrockModel(model_id=MODEL_ID, region_name=AWS_REGION) if AWS_REGION else BedrockModel(model_id=MODEL_ID)\n",
    "robot = Robot(\"so100\", data_config=\"so100_dualcam\")  # sim by default\n",
    "\n",
    "agent = Agent(\n",
    "    model=model,\n",
    "    tools=[robot, robot_mesh],\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8edb47106e1a46a883d545849b8ab81b",
   "metadata": {},
   "source": [
    "## Step 2: Record a demonstration\n",
    "\n",
    "One agent prompt → one tool sequence → one `LeRobotDataset` episode. The agent composes the scene (red cube + front camera), starts recording, runs the Mock policy, and finalizes the episode with `stop_recording`.\n",
    "\n",
    "The `Robot()` factory already initialised the simulation world, so the prompt acknowledges that to avoid the agent re-creating it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "10185d26023b46108eb7d9f57d49d2b3",
   "metadata": {},
   "outputs": [],
   "source": [
    "NUM_STEPS = 1000  # ≈ 33s of data at 30fps\n",
    "\n",
    "push_clause = (\n",
    "    f\"Push the result to {REPO_ID} when done.\" if PUSH_TO_HUB else \"Keep the dataset local; do not push to the Hub.\"\n",
    ")\n",
    "\n",
    "prompt = (\n",
    "    f\"A simulation world is already set up with the so100 robot \"\n",
    "    f\"(initialised by the Robot() factory). Compose the scene and \"\n",
    "    f\"record one demonstration:\\n\"\n",
    "    f\"\\n\"\n",
    "    f\"Scene:\\n\"\n",
    "    f\"  - Add a small red cube near the robot (about 3cm on a side).\\n\"\n",
    "    f\"  - Add a front-facing camera named 'front' looking at the cube.\\n\"\n",
    "    f\"\\n\"\n",
    "    f\"Recording:\\n\"\n",
    "    f\"  - Start recording to repo_id={REPO_ID!r} at FPS 30 with task={TASK!r}.\\n\"\n",
    "    f\"  - Run the Mock policy for {NUM_STEPS} steps with instruction={TASK!r}.\\n\"\n",
    "    f\"  - Call stop_recording to finalize the episode. {push_clause}\"\n",
    ")\n",
    "\n",
    "agent(prompt)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8763a12b2bbd4a93a75aff182afb95dc",
   "metadata": {},
   "source": [
    "### Verify the recording\n",
    "\n",
    "The dataset is on disk at `~/.cache/huggingface/lerobot/<repo_id>/`. Loading it through the `LeRobotDataset` class confirms the episode and frame counts:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7623eae2785240b9bd12b16a66d81610",
   "metadata": {},
   "outputs": [],
   "source": [
    "from lerobot.datasets.lerobot_dataset import LeRobotDataset\n",
    "\n",
    "ds = LeRobotDataset(REPO_ID)\n",
    "cache_path = Path.home() / \".cache\" / \"huggingface\" / \"lerobot\" / REPO_ID\n",
    "\n",
    "print(f\"Dataset: {REPO_ID}\")\n",
    "print(f\"  Location:  {cache_path}\")\n",
    "print(f\"  Episodes:  {ds.num_episodes}\")\n",
    "print(f\"  Frames:    {ds.num_frames}\")\n",
    "print(f\"  FPS:       {ds.fps}\")\n",
    "print(f\"  Features:  {list(ds.features.keys())}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7cdc8c89c7104fffa095e18ddfef8986",
   "metadata": {},
   "source": [
    "Inside the cache directory you'll find:\n",
    "\n",
    "- `data/chunk-000/file-000.parquet` holds joint states and actions\n",
    "- `videos/observation.images.front/chunk-000/file-000.mp4` is the recording\n",
    "- `meta/` holds schema and episode metadata\n",
    "\n",
    "Open the MP4 in QuickTime/VLC to see the recorded rollout."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b118ea5561624da68c537baed56e602f",
   "metadata": {},
   "source": [
    "## Step 3: Run a policy on the robot\n",
    "\n",
    "The same agent now drives the policy. With the Mock policy (the default), this is purely a workflow sanity-check: the arm moves through placeholder actions rather than executing a trained grasp.\n",
    "\n",
    "For real cube-picking behaviour, switch the prompt to drive `gr00t_inference` (NVIDIA GR00T container, requires Docker + GPU) or `create_policy('lerobot_local', ...)` with a checkpoint trained for the task. See this folder's README for the exact wiring."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "938c804e27f84196a10c8828c723f798",
   "metadata": {},
   "outputs": [],
   "source": [
    "agent(f\"Run the Mock policy on the robot for 200 steps with the instruction '{TASK}'. Render the final frame.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "504fb2a444614c0babb325280ed9130a",
   "metadata": {},
   "source": [
    "## Step 4: Deploy to physical hardware (reference)\n",
    "\n",
    "Switching from simulation to a physical SO-101 is one flag at construction time:\n",
    "\n",
    "```python\n",
    "robot = Robot(\n",
    "    \"so100\",\n",
    "    mode=\"real\",\n",
    "    port=\"/dev/ttyACM0\",\n",
    "    data_config=\"so100_dualcam\",\n",
    "    cameras={\n",
    "        \"front\": {\"type\": \"opencv\", \"index_or_path\": \"/dev/video0\", \"fps\": 30},\n",
    "        \"wrist\": {\"type\": \"opencv\", \"index_or_path\": \"/dev/video2\", \"fps\": 30},\n",
    "    },\n",
    ")\n",
    "```\n",
    "\n",
    "The rest of the code (`Agent(tools=[robot, ...])`, the recording prompt, the policy prompt) is unchanged. The same `LeRobotDataset` writer handles real-camera frames and hardware joint reads through LeRobot's driver layer.\n",
    "\n",
    "(Skipped in this notebook because hardware isn't typically attached to a Jupyter kernel.)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "59bbdb311c014d738909a11f9e486628",
   "metadata": {},
   "source": [
    "## Step 5: Mesh broadcast\n",
    "\n",
    "Every `Robot()` auto-joins a local Zenoh mesh, so fleets coordinate out of the box. The agent uses the `robot_mesh` tool to enumerate peers and send commands.\n",
    "\n",
    "In this single-process demo, only the local `so100_sim` joined the mesh, so peer count will be small (likely zero remote peers). On a multi-machine setup, the same prompt would discover and address every robot on the network."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b43b363d81ae4b689946ece5c682cd59",
   "metadata": {},
   "outputs": [],
   "source": [
    "agent(\n",
    "    \"Use robot_mesh to list every peer and local robot, then broadcast \"\n",
    "    \"the instruction 'go to home pose' to each one with a 5-second timeout.\"\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8a65eabff63a45729fe45fb5ade58bdc",
   "metadata": {},
   "source": [
    "## Cleanup\n",
    "\n",
    "Nothing long-running was started in this Mock-policy run. If you switched the policy to GR00T (which launches a Docker container) or LerobotLocal (which keeps GPU memory resident), the cleanup prompt for those is:\n",
    "\n",
    "```python\n",
    "agent(\"Stop the GR00T inference service on port 5555.\")\n",
    "```\n",
    "\n",
    "or the equivalent shutdown call on the policy you started.\n",
    "\n",
    "## Next steps\n",
    "\n",
    "- **Push to the Hub.** Set `HF_USER` in the second cell, export `HF_TOKEN=hf_...` with write scope, and re-run from Step 2.\n",
    "- **Train on the dataset.** The `LeRobotDataset` you just produced is in the same on-disk format the upstream LeRobot training CLI expects; `lerobot-train` consumes it directly.\n",
    "- **Use a real trained policy.** Swap the Mock prompt in Step 3 for a `lerobot_local` or `groot` checkpoint. See this folder's `README.md`.\n",
    "- **Run on physical hardware.** Calibrate your SO-101 once on the shell with `lerobot-calibrate --robot.type=so101_follower --robot.id=my_follower`, then re-run Step 1 with `mode='real'`."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
