[mcp] MVP benchmark: pytest eval harness mounting Phoenix MCP on pydantic-ai

First slice of #14089. Stand up a rudimentary pytest eval that drives an agent against the Phoenix MCP server and records token cost, latency, and goal achievement — so the "measured, not vibes" acceptance criteria on #13975 has a harness behind it. Built on the `evals/pxi/` pattern (pytest + pydantic-ai + experiments published to Phoenix), not the harbor Docker-task pattern. ## Steps - [ ] **1. Rudimentary eval pytest that mounts our MCP on pydantic-ai** - [ ] **2. Boot seeds traces from a dataset into Phoenix** - [ ] **3. Ask questions against that data — traces must export cost and latency** - [ ] **4. Publish results to a shared Phoenix instance** ## Grounding **Step 1** has a direct precedent. `evals/pxi/harness/agent_task.py:211` (`build_shared_docs_mcp_server`) already mounts an MCP toolset on pydantic-ai for the PXI evals. The new work is pointing it at the Phoenix MCP server (`/mcp`, streamable HTTP) via `pydantic_ai.mcp.MCPServerStreamableHTTP` instead of the Mintlify docs server. `pydantic-ai-slim[...,mcp,...]` is already a production dependency. Reuse the session-scoped event-loop setup in `evals/pxi/pytest.ini` (`asyncio_default_fixture_loop_scope = session`). A fresh loop per test trips anyio's cancel-scope rule with streamable-HTTP clients — this is documented at `agent_task.py:212-221` and will bite otherwise. **Step 2** can start from `src/phoenix/trace/fixtures.py`, plus the fixture generators under `evals/harbor/` (`generate_fixture_data.py`, `fetch_fixtures.py`). The benchmark needs deterministic, known-answer trace data so goal achievement is gradeable. **Step 3 has a blocker — see below.** **Step 4** is close to free: `evals/pxi/harness/run_experiment.py` already publishes runs via `phoenix.client.AsyncClient` / `RanExperiment`, and `evals/pxi/pytest.ini` defines a `phoenix` marker that records a test as a Phoenix experiment run. ## Blocker on step 3 **The REST v1 surface does not expose cost.** There are zero `cost` references across `src/phoenix/server/api/routers/v1/`, while GraphQL exposes it on `Span`, `Project`, `Experiment`, and `GenerativeModel` (`Span.cost_summary`, backed by `src/phoenix/server/cost_tracking/`). The Phoenix MCP server derives its tools from the `/v1` OpenAPI spec (`_install_progressive_disclosure`, `_v1_group_sizes`), so **MCP tools cannot return cost today**. Step 3 needs cost (and confirmed latency) exported on the REST span surface first — an API change, not wiring. Latency needs the same check: span start/end timestamps are present, but confirm a usable duration is exposed rather than derived client-side. ## Out of scope Deferred to follow-ups under #14089: - More than one task or model - Tool-group / code-mode variant sweeps - CI gating and thresholds - Statistical treatment of run-to-run variance ## Acceptance criteria `pytest` runs an agent against the Phoenix MCP server over seeded trace data, answers a small set of known-answer questions, and publishes one experiment per run to the shared Phoenix instance carrying token cost, latency, and a goal-achievement score.