ITADN

[azsdk-cli code review] Build the before-and-after evaluation harness

#16767Opensamvaity 创建于 4 天前
enhancementCentral-EngSys
S
samvaitycommented
## Parent - #16764 ## Goal Build a frozen, reproducible before/after benchmark that measures whether the azsdk-cli code-review skill improves review quality rather than merely changing review output. ## Deliverables - [ ] Select at least 8 maintainer-approved replayable diffs: 4 defect-bearing and 4 clean/analyzer-only controls. - [ ] Cover behavioral, cross-file integration, reliability/concurrency, and error/response-contract defects. - [ ] Record expected category, changed line, failure path, and analyzer/CI ownership for each fixture. - [ ] Capture the no-skill baseline using a fixed model and review effort. - [ ] Run the skill treatment against identical fixtures and settings. - [ ] Run 3 repetitions per fixture for the final comparison. - [ ] Preserve `results.jsonl`, comments, skill/tool trajectories, grader output, and raw counts as artifacts. - [ ] Add deterministic grading for category + location, using a custom grader only if existing Vally graders are insufficient. - [ ] Publish a before/after scorecard and rejection reasons. ## Metrics | Metric | Target | |---|---| | Defect recall | >= 80%, no baseline regression | | Review precision | >= 80% | | Evidence rate | 100% | | Duplicate analyzer/CI finding rate | <= 10% | | Unrelated-change scope accuracy | 100% | | Repeated-run stability | >= 75% | Use raw numerator/denominator counts alongside percentages. Classify rejected findings as `false-positive`, `duplicate`, `unsupported`, `stale-rule`, `wrong-line`, or `unclear-impact`. ## Acceptance criteria - Another maintainer can reproduce baseline and treatment results from documented commands. - Fixtures and labels are frozen before final treatment scoring. - Artifacts explain both what improved and what did not. - Confirmed misses and false positives can be converted directly into regression fixtures.
0 条评论