[Cosmos] Diagnostics polish: per-attempt `activity_id: null` on transport failure, and `system_usage.cpu` is the literal string `"empty"`

**Severity:** Medium (debuggability — not customer-facing latency, but materially slows live-site triage of regional events) **Area:** `area: cosmos`, `client: azure_data_cosmos_driver`, `diagnostics`, `observability` **Account observed:** `cosmos-rust-drdrill-workload-2` (multi-write, [centralus, eastus2]) during indefinite DR drill, 2025‑11. **Crate:** `azure_data_cosmos_driver` **Branches checked:** `main`. Not addressed by `tvaron3/cosmos-regional-failover-tests`. --- ## Summary Two independent diagnostics-fidelity gaps surfaced in the same captured 503 DR-drill diagnostic. Each is small individually but they compound during live-site triage — both live in `diagnostics_context.rs` / `system/cpu_memory.rs`, have low blast radius, and should be reviewed together by the same owner. 1. **Per-attempt `activity_id` is `null` on every transport failure** even though the SDK knows the value it sent on `x-ms-activity-id`. We send it, we receive nothing back, we never record what we sent — so the per-attempt activity-id field that operators correlate with backend logs is silently empty for the very failure cases that need correlation most. 2. **`system_usage.cpu` is the literal string `"empty"`** when the CPU sampler hasn't accumulated samples yet (or isn't running in the host environment). This is a type-punned sentinel that looks like real data and breaks downstream JSON consumers expecting a number/object. --- ## Reproduction (diagnostic excerpt — verbatim) From `/diagnostics/503-4retry-sample.json`: ```jsonc { "activity_id": "12c4cc48-e484-4feb-b3f4-539318de8f70", // envelope: present "requests": [ { "activity_id": null, "endpoint": "…-centralus.…", "events": [ … "transport_failed" … ], … }, { "activity_id": null, "endpoint": "…-centralus.…", "events": [ … "transport_failed" … ], … }, { "activity_id": null, "endpoint": "…-eastus2.…", "events": [ … "transport_failed" … ], … }, { "activity_id": null, "endpoint": "…-centralus.…", "events": [ … "transport_failed" … ], … } ], "system_usage": { "cpu": "empty", "cpu_overloaded": false, "processor_count": 4 } } ``` - Per-attempt `activity_id` is `null` 4/4 times even though the SDK does inject `x-ms-activity-id` (see citation below). - `cpu` is the string `"empty"` — a type-punned sentinel. --- ## Root cause — code chain on `main` ### (A) Per-attempt `activity_id` only assigned from response headers `sdk/cosmos/azure_data_cosmos_driver/src/diagnostics/diagnostics_context.rs:1380-1400` (`record_response`) The only path that writes the per-attempt `RequestDiagnostics::activity_id` field is in `record_response`, which extracts it from the response headers via `cosmos_headers.activity_id()`. There is no analogous assignment in `record_request_start` / `fail_transport_request`. Meanwhile, the SDK *does* know the value it sent: `sdk/cosmos/azure_data_cosmos_driver/src/driver/pipeline/operation_pipeline.rs:1280-1286` ```rust headers.insert( request_header_names::X_MS_ACTIVITY_ID, HeaderValue::from(ctx.activity_id.to_string()), ); ``` `ctx.activity_id` is *the* value placed on the wire. On a transport failure (`event_type: "transport_failed"`), there is no response and `record_response` never runs → per-attempt `activity_id` stays `None` → serializes as `null`. Aside (explicitly out-of-scope here): peer SDKs (.NET / Java) mint a fresh activity-id *per retry attempt*, while Rust uses the operation-level `ctx.activity_id` across all attempts. That is a larger parity discussion (cross-attempt correlation vs per-call uniqueness). **This issue only asks that whatever value the SDK sent on the wire is recorded into the per-attempt diagnostic** — full stop. ### (B) `cpu: "empty"` sentinel `sdk/cosmos/azure_data_cosmos_driver/src/system/cpu_memory.rs:219-232` `CpuMemoryHistory::Display::fmt` writes `"empty"` when `samples` is empty. This `Display` impl is what `serde` uses (via the `Serialize` wrapper for the diagnostics struct in `diagnostics_context.rs:1132-1175` `SystemUsageSnapshot`). Two failure modes flow into this: 1. The CPU/memory background sampler hasn't accumulated samples yet (cold start). 2. The sampler isn't running at all in the host environment (e.g. some container configurations, macOS in some test paths). The diagnostic from the DR-drill workload has been running for hours, so this is most likely case (2). Either way, emitting the literal string `"empty"` into the `cpu` field is a type pun — downstream JSON consumers expect a numeric percentage / structured object and break. --- ## Expected vs Actual | | Expected | Actual | |---|---|---| | Per-attempt `activity_id` on transport failure | The id we sent on `x-ms-activity-id` | `null` | | `system_usage.cpu` when sampler has no data | Omit field, or `null`, or a structured object like `{ "samples": 0, "status": "unavailable" }` | Literal string `"empty"` | --- ## Cross-SDK reference - **.NET / Java:** Per-attempt activity-id is always present in `CosmosDiagnostics` regardless of whether the request received a response — populated from the outgoing request context. - **.NET:** `CpuLoadHistory` serializes to a structured object with `samples: []` rather than a sentinel string when empty. Java's `CpuLoadHistory` does similar. --- ## Suggested labels `area: cosmos` · `bug` · `severity: medium` · `client: azure_data_cosmos_driver` · `diagnostics` · `observability`