# Dashboard Evaluation Agent Instructions

**Your role:** Analyze dashboard screenshots and produce a comprehensive evaluation report. You are NOT responsible for implementing fixes—an outer agent will handle modifications based on your findings.

## Evaluation Context

The dashboard must provide everything a human operator needs to debug:
- **Why is a job failed/not running?** Error messages, task states, logs
- **Why is a job not scheduled?** Pending reasons, constraint mismatches, capacity issues
- **Why is a VM up/down?** VM states, bootstrap logs, autoscaler actions
- **Why is a scale group available/unavailable?** Quota limits, demand vs capacity, recent autoscaler decisions

## Your Task

You will receive a directory of dashboard screenshots generated by `tests/e2e/test_dashboard.py`. The test suite:
1. Starts a local Iris cluster with multiple scale groups (including edge cases)
2. Submits diverse jobs covering key scenarios
3. Captures screenshots of all dashboard pages

Your job is to analyze these screenshots and generate a structured evaluation report.

## Report Structure

Produce a structured markdown report with the following sections:

### A. Screenshot-by-Screenshot Assessment

For each screenshot file:

```markdown
### Screenshot: `tab-jobs.png`

**PASS/FAIL:** [PASS|FAIL]

**Information Completeness:**
- ✅ Shows job states (succeeded, failed, running, pending)
- ✅ Shows task counts per job
- ❌ Missing: failure reason preview in table (need to click into job)
- ❌ Missing: pagination controls not visible (need 50+ jobs to test)

**Visual Clarity:**
- ✅ State colors distinguishable (green=success, red=fail, blue=running, gray=pending)
- ⚠️  Issue: pending vs killed states look similar
- ✅ Job IDs are visibly clickable (underlined/colored)

**Data Richness:**
- ✅ Multiple job states represented
- ⚠️  Limited variety: only 2-3 jobs visible, need more for realistic view

**Usability:**
- ✅ Job IDs link to detail pages
- ❌ Missing: bulk actions (kill multiple jobs)
- ❌ Missing: search/filter controls

**Edge Cases:**
- ⚠️  Not tested: empty state (no jobs)
- ⚠️  Not tested: very long job names
- ⚠️  Not tested: pagination with 100+ jobs

**Debugging Capability:**
- ❌ Cannot see *why* job failed from table (must click through)
- ❌ Cannot see *why* job is pending from table
```

### B. Critical Debugging Gaps

List common scenarios where debugging is NOT possible or exercised with the current screenshots.

For example:

```markdown
## Critical Debugging Gaps

1. **Job pending reason not visible in main table**
   - Impact: User must click into every pending job to understand why it's not scheduled
   - Fix: Add "Diagnostic" column to jobs table with truncated reason
   - Files: `src/iris/cluster/controller/dashboard.py` (add diagnostic to jobs RPC response)

2. **No visibility into autoscaler decision-making**
   - Impact: User cannot understand why autoscaler didn't scale up for pending job
   - Fix: Add "Autoscaler Reasoning" section showing per-group scaling decision
   - Files: Add new RPC endpoint for autoscaler decision log

3. **No logging available for failed task**
   - Impact: User can't see why task failed
   - Fix: Adjust task attempt detail page to show worker logs
   - Files: Adjust task attempt in dashboard

4. **No logging for build failures**
   - Impact: User can't see why job didn't start
   - Fix: Ensure worker docker build logs show up in task attempts
   - Files: iris/cluster/worker
```

### C. Recommended Improvements

#### Additional Jobs to Add to Script

```markdown
1. **Preempted job** - simulate VM preemption, verify dashboard shows PREEMPTED state
2. **Worker-failed job** - worker crashes mid-task, verify job shows WORKER_FAILED with diagnostic
3. **Job with 50+ tasks** - verify task table pagination works
4. **Job with very long name** - verify UI doesn't break with overflow
5. **Endpoint-registered job** - verify endpoints tab shows registered actors
```

#### Cluster Config Changes

```markdown
1. **Add scale group with init failures** - simulate VM that fails to bootstrap
2. **Add manual provider group** - test non-cloud worker display
3. **Add more realistic quota** - reduce max_slices on some groups to simulate real constraints
```

#### Dashboard Code Changes

```markdown
1. **Jobs table**: Add diagnostic preview column (truncated to 100 chars)
2. **Workers table**: Fix worker attributes serialization (convert dict to key=value strings)
3. **Autoscaler tab**: Add "Decision Log" section explaining why each group scaled/didn't scale
4. **Job detail**: Add "Scheduling Timeline" showing state transitions with timestamps
5. **VM detail**: Add "Related Jobs" section showing jobs that ran on this VM
6. **All pages**: Add refresh button (not just auto-refresh)
```

### D. Final Assessment

Provide an overall PASS/FAIL grade with rationale. Example:

```markdown
## Overall Assessment: PASS/FAIL

**Grade:** FAIL

**Rationale:**
While the dashboard covers basic states (jobs running/failed, workers present, VMs listed),
it fails to support critical debugging workflows:
- Pending job diagnosis requires too many clicks (not visible in table)
- Worker attributes broken (`[object Object]`)
- No autoscaler decision visibility (why didn't it scale?)
- Missing quota exhaustion messaging
- No task log access visible in screenshots

**Must-Fix Before Production:**
1. Fix worker attributes rendering
2. Add diagnostic column to jobs table
3. Add autoscaler decision log
4. Verify task logs are accessible

**Priority:**
- P0: Worker attributes fix (broken feature)
- P1: Jobs table diagnostics (usability)
- P1: Autoscaler decision log (critical for debugging scale issues)
- P2: Remaining improvements
```

## Important Notes

- **You are NOT responsible for implementing fixes.** Your job is to identify gaps and recommend improvements.
- **Be thorough.** Check every screenshot for information completeness, visual clarity, data richness, usability, edge cases, and debugging capability.
- **Provide actionable recommendations.** For each gap, suggest specific changes with relevant file paths when possible.
- **Focus on debugging workflows.** The dashboard's primary purpose is helping operators debug job/VM/autoscaler issues.

## Success Criteria for PASS Grade

You should grade the dashboard as PASS when all of the following are true:

✅ **Every screenshot shows actionable information** - user can understand state and next steps
✅ **Zero critical gaps** - all debugging scenarios are supported
✅ **No broken rendering** - no `[object Object]`, overflow issues, or missing data
✅ **Edge cases handled** - empty states, long strings, large datasets render gracefully
✅ **Minimal click-depth to diagnosis** - job failure reason visible in max 2 clicks from dashboard home

If any of these criteria are not met, grade as FAIL and document the gaps in your report.

## Reference: Dashboard Goals from README

From `README.md`:
- Controller includes "HTTP dashboard (monitoring, status)"
- Workers include "Heartbeat reporter (health monitoring)"
- Architecture emphasizes debuggability and visibility

Key principles:
- **Visibility into failures**: Why did job fail? Why didn't VM start?
- **Autoscaler transparency**: Why did/didn't it scale? What's blocking capacity?
- **Worker health**: Which workers are healthy? What are they running?
- **Log accessibility**: Can access controller, worker, and task logs

The dashboard should make these principles visible without requiring SSH or log diving.
