ITADN

Improve CCR quality across all SDK repos

#16034OpenJennyPng 创建于 2026-06-16
dev inner loop
J
JennyPngcommented
## Goal - Reduce human burden for code reviews by improving reliability of CCR - Improve CCR quality by automatically encoding repeated missed patterns in prompts in a repeated agentic workflow - Track quantitative metrics to measure progress ## Solution Standardize organization of code review prompts across repos for CCR to leverage. Periodically mine recently closed/merged PRs, separate Copilot-reviewer comments from human comments, use an LLM judge to find substantive, issues that humans caught but Copilot missed, cluster those into themes, and track metrics over time. Based on themes and patterns, suggests generalizable prompt improvements. ## Implementation overview ### Reviewer skill - A universal reviewer skill lives in and syncs from `azure-sdk-tools`. It points towards language-specific skills - Each repo has its own specific rules in its own skill file - Enforce sub-agent usage for different focus areas, e.g. security review, architecture review, etc. ### Automated prompt enhancement workflow 1. Ingest merged PR data from a custom date range. 2. Split bot/Copilot comments vs. human comments. Key idea: consider what humans had to catch that Copilot missed 3. Filter low-signal noise, nitpicky comments, only consider comments from core contributors, etc. 4. Cluster comments into themes/patterns 5. Normalize themes to avoid overfitting (does this missed pattern occur in multiple PRs?) 6. Extract and report metrics 7. Open PR/issue with suggested prompt updates ## Open questions / ideas - Some code review patterns are specific to codepaths or packages, not generalizable to the entire repo. Should we attempt to automate suggestions to package-specific review prompts? - What metrics do we track? e.g. # human comments over time, # human-caught patterns per 'category' over time (security, bug, architecture) that Copilot missed, time it takes to merge a PR - How do we aggregate/present the metrics? - What PR data are we considering when analyzing comments? e.g. should we consider if a comment was acted on (a later commit touches the file line range covered by that comment), - Should we also consider Copilot comments that were not acted upon / had negative responses? aka false positives? - Do we want a DB to store run information ? maybe not, can chain agentic workflows and look at outputs? ## Related references: - https://github.com/Azure/azure-sdk-for-python/pull/47506 - https://github.com/dotnet/fsharp/blob/822e80493c45d55706357c5b6670295e99f77eec/.github/agents/extraction-pipeline.md
1 条评论