[azsdk-cli code review] Build the before-and-after evaluation harness
enhancementCentral-EngSys
## Parent
- #16764
## Goal
Build a frozen, reproducible before/after benchmark that measures whether the azsdk-cli code-review skill improves review quality rather than merely changing review output.
## Deliverables
- [ ] Select at least 8 maintainer-approved replayable diffs: 4 defect-bearing and 4 clean/analyzer-only controls.
- [ ] Cover behavioral, cross-file integration, reliability/concurrency, and error/response-contract defects.
- [ ] Record expected category, changed line, failure path, and analyzer/CI ownership for each fixture.
- [ ] Capture the no-skill baseline using a fixed model and review effort.
- [ ] Run the skill treatment against identical fixtures and settings.
- [ ] Run 3 repetitions per fixture for the final comparison.
- [ ] Preserve `results.jsonl`, comments, skill/tool trajectories, grader output, and raw counts as artifacts.
- [ ] Add deterministic grading for category + location, using a custom grader only if existing Vally graders are insufficient.
- [ ] Publish a before/after scorecard and rejection reasons.
## Metrics
| Metric | Target |
|---|---|
| Defect recall | >= 80%, no baseline regression |
| Review precision | >= 80% |
| Evidence rate | 100% |
| Duplicate analyzer/CI finding rate | <= 10% |
| Unrelated-change scope accuracy | 100% |
| Repeated-run stability | >= 75% |
Use raw numerator/denominator counts alongside percentages. Classify rejected findings as `false-positive`, `duplicate`, `unsupported`, `stale-rule`, `wrong-line`, or `unclear-impact`.
## Acceptance criteria
- Another maintainer can reproduce baseline and treatment results from documented commands.
- Fixtures and labels are frozen before final treatment scoring.
- Artifacts explain both what improved and what did not.
- Confirmed misses and false positives can be converted directly into regression fixtures.
0 条评论