# CI Pipeline Monitor — Debug Log Template

This is the template for the debug log file generated by `ci-pipeline-monitor`.
The debug log records every step, API call, and decision made during a
monitoring run. It is always generated (unlike the test report which is
on-demand).

Output file name: `ci-pipeline-monitor-<YYYY-MM-DD-HHmmss>.log`

---

```
================================================================================
CI Pipeline Monitor — Debug Log
Started:    <YYYY-MM-DD HH:mm:ss UTC>
Org:        dnceng-public
Project:    public
Branch:     refs/heads/main
================================================================================

================================================================================
Prerequisites
================================================================================

[HH:mm:ss] pip install requests — OK
[HH:mm:ss] ADO_TOKEN — <source: env var | az cli | user-provided>

================================================================================
Load Pipeline Definitions
================================================================================

[HH:mm:ss] Reading pipelines.md...
[HH:mm:ss] Loaded <N> pipelines (<M> monitored, <K> skipped)
[HH:mm:ss] Skipped: <pipeline1> (private), <pipeline2> (private)

================================================================================
Fetch Latest Builds
================================================================================

[HH:mm:ss] Running: python scripts/setup_and_fetch_builds.py --pipelines pipelines.md --db scripts/monitor.db
[HH:mm:ss]   PASS  <pipeline_name> — build <buildId> (succeeded)
[HH:mm:ss]   FAIL  <pipeline_name> — build <buildId> (failed)
[HH:mm:ss]   SKIP  <pipeline_name> (private)
... (repeat for each pipeline)

[HH:mm:ss] Build results: <P> PASS, <F> FAIL
[HH:mm:ss] Failing pipelines: <pipeline1>, <pipeline2>, ...

================================================================================
Extract Failed Tests via Test Results API
================================================================================

[HH:mm:ss] Running: python scripts/extract_failed_tests.py --db scripts/monitor.db
[HH:mm:ss] Dispatching <N> sub-agents for <F> failing pipelines...

--- Sub-agent 1: <pipeline1>, <pipeline2>, <pipeline3> ---

  [HH:mm:ss] === Processing: <pipeline1> (build <buildId>) ===
  [HH:mm:ss] GET https://dev.azure.com/dnceng-public/public/_apis/test/runs?buildUri=vstfs:///Build/Build/<buildId>&api-version=7.1
             → <N> test runs, <M> with failures
  [HH:mm:ss] Runs with failures:
             - Run <runId>: <run name> (Failed=<N>)
             - Run <runId>: <run name> (Failed=<N>)
  [HH:mm:ss] Skipping <K> runs with 0 failures

  [HH:mm:ss] GET https://dev.azure.com/dnceng-public/public/_apis/test/runs/<runId>/results?outcomes=Failed&$top=200&api-version=7.1
             → <N> failed results
  [HH:mm:ss] Failed tests:
             - <automatedTestName1>
             - <automatedTestName2>
             - ... (<N> total)
  [HH:mm:ss] failingSince: <date> (build <buildId>)

  [HH:mm:ss] GET https://dev.azure.com/dnceng-public/public/_apis/test/runs/<runId2>/results?outcomes=Failed&$top=200&api-version=7.1
             → <N> failed results
  [HH:mm:ss] Deduplicating: <testName> already seen from run <runId1>, skipping
  [HH:mm:ss] New failures from this run: <list or "none — all duplicates">

  [HH:mm:ss] === Processing: <pipeline2> (build <buildId>) ===
  ... (repeat for each pipeline)

--- Sub-agent 2: <pipeline4>, <pipeline5>, <pipeline6> ---
  ... (repeat)

[HH:mm:ss] All sub-agents completed.
[HH:mm:ss] Total failures inserted into test_results: <N> rows
[HH:mm:ss] Unique test names: <M>

================================================================================
Fetch Helix Console Logs → helix-logs/
================================================================================

[HH:mm:ss] Running: python scripts/fetch_helix_logs.py --db scripts/monitor.db
[HH:mm:ss] Found <N> test_results rows needing log download
[HH:mm:ss]   <M> unique console URLs to fetch

[HH:mm:ss] Fetching <test_name_1>...
           → OK: <N> lines, exit=<code>, saved to helix-logs/<filename>.log, updated <K> rows
[HH:mm:ss] Fetching <test_name_2>...
           → OK: <N> lines, exit=<code>, saved to helix-logs/<filename>.log, updated <K> rows
... (repeat for each unique URL)

[HH:mm:ss] All console logs downloaded.
[HH:mm:ss] Files saved to: helix-logs/ (<M> files, <N> total rows updated)
[HH:mm:ss] Remaining with console_log_path IS NULL: <should be 0>

================================================================================
Triage — Read Console Logs + Classify + Group + Search GitHub
================================================================================

[HH:mm:ss] Starting triage. test_results rows with failure_id IS NULL: <N>

--- Failure Group 1 ---

[HH:mm:ss] Reading console log: helix-logs/<filename>.log (<N> lines)
[HH:mm:ss] Extracted error_message (verbatim, <N> chars)
[HH:mm:ss] Extracted stack_trace (verbatim, <N> chars)
[HH:mm:ss] UPDATE test_results SET error_message, stack_trace WHERE id=<id>

[HH:mm:ss] Classification: <failure_category> (exit_code=<code>)
[HH:mm:ss] Grouped <K> test_results rows under this failure:
           - <test_name_1> in <pipeline1>
           - <test_name_2> in <pipeline2>

[HH:mm:ss] Search pass 1 — full test name: "<test_name_1>"
           → Found: #<N> "<issue title>" (OPEN, assigned: @<user>)
           → URL: https://github.com/dotnet/runtime/issues/<N>
[HH:mm:ss] Verifying issue #<N>: error pattern matches? YES/NO
[HH:mm:ss] Search pass 2 — class/method: "<ClassName.MethodName>"
           → <N> results
... (repeat passes as needed)

[HH:mm:ss] Decision: Matched to issue #<N> / Flagged as NEW
[HH:mm:ss] INSERT INTO failures (id=<id>, title="...", source_test_result_id=<id>, ...)
[HH:mm:ss] INSERT INTO failure_pipelines (<K> rows)
[HH:mm:ss] INSERT INTO failure_tests (<K> rows)
[HH:mm:ss] UPDATE test_results SET failure_id=<id> WHERE id IN (<ids>)

--- Failure Group 2 ---
... (repeat for each group)

[HH:mm:ss] Triage complete.
[HH:mm:ss] Validation: SELECT COUNT(*) FROM test_results WHERE failure_id IS NULL → <must be 0>
[HH:mm:ss] Total failure groups: <N>
[HH:mm:ss] Issue mapping:
           - <K1> failures → #<N1>
           - <K2> failures → #<N2>
           - <K3> failures → NEW (no issue)

================================================================================
Validate DB
================================================================================

[HH:mm:ss] Running: python scripts/validate_results.py --db scripts/monitor.db --pipelines pipelines.md --log logs/ci-pipeline-monitor-<timestamp>.log
[HH:mm:ss] <X>/<Y> checks passed

================================================================================
Fix Validation Failures (if any — up to 3 retries)
================================================================================

[HH:mm:ss] Fixing <N> validation failures...
[HH:mm:ss] Re-reading console log for failure_id=<N>, test_results id=<id>
[HH:mm:ss] UPDATE failures SET stack_trace=<corrected> WHERE id=<N>
[HH:mm:ss] Re-running validator...
[HH:mm:ss] <X>/<Y> checks passed

// Repeat if failures decreased (up to 3 retries total).
// If checks still fail after retries:
[WARN] Validation error persists after retry — <check description>
  Pipeline: [<name> <build_number>](<ado_test_results_tab_url>)
  Console Log: [Console Log](<helix_url>)
  Field: <field_name>, failure_id=<N>

================================================================================
Generate Report
================================================================================

[HH:mm:ss] Running: python scripts/generate_report.py --db scripts/monitor.db
[HH:mm:ss] Report generated: logs/test-report-<timestamp>.md

================================================================================
SUMMARY
================================================================================

Run completed: <YYYY-MM-DD HH:mm:ss UTC>
Duration:      <M> minutes
Pipelines:     <P> PASS, <F> FAIL (of <N> monitored)
Unique tests:  <M> failing
Failure groups: <N> (after grouping by root cause)
Console logs:  <N> files in helix-logs/ (<M> unique URLs)
GitHub issues: <K> known, <J> new
API calls:     <total> (builds: <n>, test runs: <n>, test results: <n>, GitHub: <n>, Helix: <n>)

================================================================================
End of debug log. Generated by ci-pipeline-monitor skill.
================================================================================
```
