Investigation: HPK partial partition key query silently returns empty results during/after partition split

## Symptom Queries against an **HPK (Hierarchical Partition Key)** container using a **partial partition key** (e.g., 2 of 3 components) silently return **zero results** with no backend contact, no error, and no exception. The issue appears to coincide with or follow a partition split event. **Observed behavior:** - `FeedIterator.ReadNextAsync()` returns an empty `FeedResponse` with 0 items - `RequestCharge = 0` (no RU consumed — no backend contact) - Duration is extremely short (~0.27ms) compared to a successful query (~3ms) **Expected behavior:** - Query returns the correct results - Or throws a retriable exception so the caller can retry --- ## Diagnostic Evidence ### Failing query diagnostic (silent empty result) Key observations from the failing trace: - ✅ `Try Get Overlapping Ranges` completes in 0.008ms — fast, suggesting routing map IS in cache (not null) - ❌ No `MoveNextAsync` child trace — the query pipeline never iterated - ❌ No backend contact (no RNTBD/gateway calls) - ✅ `Query Response Serialization` appears immediately — meaning `EmptyQueryPipelineStage` was used ### Successful query diagnostic (same time window) The same account/collection/query succeeds ~11 minutes earlier, routing to PKRange `[,20D8EBCD...)`, returning 1 document with `RequestCharge=6.46`. ### Backend partition split timeline (account: mfsppcuapcdb, collection: fileShareManagementDataV2) Collection split from a single range (PKRange0: `[0, FF)`) into multiple ranges during the failure window: | PKRange | Min EPK | Max EPK | Parents | |---------|---------|---------|---------| | 1 | `0` | `20D8EBCD...` (128 chars) | `["0"]` | | 2 | `20D8EBCD...` (128 chars) | `FF` | `["0"]` | | 3 | `20D8EBCD...` (128 chars) | `31DA13FC...` (128 chars) | `["0","2"]` | | 4 | `31DA13FC...` (128 chars) | `FF` | `["0","2"]` | | ... | (further splits within same hour) | | | Boundaries are **128-char** hex strings (3-component HPK: 3×32-char hashes + 32-char zero padding). **Query's target EPK:** `00EB57A7EE7D5CAFE2751C18938111BC33F19FD36AD3B8EFC4A4AA4A09878654` (64 chars — 2 of 3 HPK components specified) --- ## Code Path Leading to Silent Empty Result ``` CosmosQueryExecutionContextFactory.GetTargetPartitionKeyRangesAsync → CosmosQueryClientCore.GetTargetPartitionKeyRangesAsync (forceRefresh: false) → IRoutingMapProviderExtensions.TryGetOverlappingRangesAsync → PartitionKeyRangeCache.TryGetOverlappingRangesAsync → CollectionRoutingMap.GetOverlappingRanges(range) → returns empty list (NOT null) → targetRanges.Count == 0 → EmptyQueryPipelineStage → MoveNextAsync returns false immediately → FeedResponse with 0 items, 0 RU ``` > **Critical distinction:** `null` from `TryGetOverlappingRangesAsync` correctly throws `NotFoundException` and triggers retry. An **empty list** does NOT trigger any retry — this is the silent failure path. --- ## Possible Causes (Under Investigation) ### Hypothesis 1: Stale routing map after rapid multi-level partition split The client's cached `CollectionRoutingMap` may contain **pre-split or intermediate-state boundaries** that, when used in `GetOverlappingRanges`, produce no overlap with the query's EPK range. The cache is not proactively invalidated on split — it relies on `NotFoundException` (404/1002) from the backend to trigger a refresh. But since the query never reaches the backend (returns empty first), no refresh is triggered — a self-reinforcing failure. ### Hypothesis 2: LengthAware range comparator edge case with mixed-length boundaries `CollectionRoutingMap` uses `LengthAwareMinComparer`/`LengthAwareMaxComparer` (enabled by default via `CosmosClientOptions.UseLengthAwareRangeComparer = true`). These comparators **TrimEnd('0')** before comparing. For a **64-char partial EPK** queried against **128-char boundaries with 32 trailing zeros**, the TrimEnd behavior may cause the binary search in `GetOverlappingRanges` to compute `minIndex > maxIndex`, returning an empty range list. ### Hypothesis 3: Transient inconsistent routing map state during TryCombine `CollectionRoutingMap.TryCombine` is called when a split is detected. During rapid multi-level splits (PKRange0 → 1+2 → 3+4 → 5+6 → 7+8, all within ~1 hour), the client may observe an intermediate `TryCombine` result that has removed old parent ranges but not yet fully added all child ranges — producing a routing map with a coverage gap at the query's EPK position. ### Hypothesis 4: forceRefresh not propagated to routing map cache `CosmosQueryClientCore.GetTargetPartitionKeyRangesAsync` accepts a `forceRefresh` parameter but does **not** pass it to `routingMapProvider.TryGetOverlappingRangesAsync`. This means even if a retry path explicitly passes `forceRefresh: true`, the routing map cache is never actually refreshed — perpetuating the stale state across retries. --- ## Investigation Tasks - [ ] Obtain **full untruncated SDK diagnostics JSON** from customer at time of failure - [ ] Obtain **SDK ETW/trace logs** — check for `"Routing Map Null for collection"` warnings (null map vs. empty-result map distinction) - [ ] Write a **unit test** reproducing `GetOverlappingRanges` with 128-char HPK boundaries and a 64-char partial EPK under both LengthAware and ordinal comparators (Hypothesis 2) - [ ] Trace `TryCombine` behavior during rapid multi-level splits to confirm/deny Hypothesis 3 - [ ] Confirm whether `forceRefresh` is correctly propagated through the routing map call chain (Hypothesis 4) - [ ] Determine if an empty `targetRanges` for a specified partition key should trigger a forced routing map refresh before falling back to `EmptyQueryPipelineStage` --- ## Environment - **SDK Version:** 3.58.0 - **Connection Mode:** Direct - **Container:** HPK with 3 partition key components - **Account:** mfsppcuapcdb (CentralUSEUAP) - **Collection:** fileShareManagementDataV2 - **Reproduced:** In production, intermittently during/after partition split