Python stub aborted with glibc heap corruption, leading to 0 RPS on gRPC serving thread

**Description** Under sustained production load, a Python-backend process in an ensemble aborted with glibc heap corruption: ``` malloc_consolidate(): unaligned fastbin chunk detected Signal (6) received. ``` After the SIGABRT, the main `tritonserver` process (PID 1 in the container) did **not** exit — the container was never restarted (`restartCount: 0`) — but the server stopped serving inference: throughput dropped to 0, the ensemble's queue depth and request latency spiked, and the success/failure counters showed **zero** completions and **zero** failures (requests appear to have been accepted and then held in-flight indefinitely rather than failed). The HTTP readiness endpoint `/v2/health/ready` continued returning `200` throughout, so the pod remained `Ready` and nothing externally detected the failure. The server stayed in this wedged state for ~1h47m until it was manually deleted. No `"Stub process is unhealthy and it will be restarted"` log line was ever emitted after the SIGABRT. We want to flag two things: (a) the heap corruption itself, and (b) that a dead/wedged Python-backend process does not surface in the health endpoints and is not auto-recovered. (b) appears closely related to #8604; this report focuses on (a) but includes the wedge behavior since they occurred together. **What we have confirmed vs. what is still hypothesis** (we want to be precise): Confirmed from logs/metrics: - The abort and signal above, on the Triton container's stderr, at a precise timestamp that coincides exactly with the throughput collapse. - PID 1 survived (no container restart), server wedged, HTTP health stayed 200, zero recorded failures, no stub-restart log line. - GPU is not involved: DCGM Xid errors = 0, GPU memory flat (~2.6 GB of 24 GB). - Our pre/post model code is pure Python + NumPy. We verified neither model writes to the input tensor buffer in place; all operations allocate fresh arrays. So the corrupting write is in the native layer (the `triton_python_backend_stub` C++/pybind bridge, tensor (de)serialization, or a native dependency), not in our Python. Not yet determined (we are working on a core dump to resolve this): - Whether the aborting process was a Python backend **stub** child or the main server process. Container stderr is shared between PID 1 and stub children, so the log alone does not distinguish them. - The exact corrupting allocation/free. glibc only detects the corruption later during `malloc_consolidate`, so the detection point is decoupled from the offending write (~9.5h of clean serving preceded it). One possibly-relevant detail: the line immediately preceding the abort was a CUDA→pinned-memory fallback: ``` W ... memory.cc:212 "Failed to allocate CUDA memory with byte size 224580 on GPU 0: CNMEM_STATUS_OUT_OF_MEMORY, falling back to pinned system memory" ``` We are not claiming causation, but note the corruption is detected one log line after this host-memory allocator path. This resembles the fastbin-corruption-after-pinned-fallback pattern in triton-inference-server/tensorrtllm_backend#587, and the `dtype 17 (O)` / object-string angle in #7517 (our postprocessing model is the only stage that emits a `TYPE_STRING` output). **Triton Information** Triton version: 2.59.0 (NGC container 25.05). Using the Triton container (custom image built **FROM** the NGC release; no changes to Triton itself or to the python backend). Image digest available on request. Backends in use: `tensorrt` and `python`. Platform: GKE, NVIDIA L4 (g2-standard), control plane 1.34. **To Reproduce** We do **not** yet have a deterministic minimal reproducer — the corruption surfaces only after several hours of sustained mixed production traffic, which is consistent with a low-probability native heap bug detected late. We are attempting a load-replay repro and a core-dump capture and will update this issue with the faulting stack when we have it. Filing now in case the signature is already recognized. Setup that produced it: 1. An ensemble `ensemble_model` → `preprocessing` (Python) → `model` (TensorRT) → `postprocessing` (Python). 2. Python backend instance counts are high (preprocessing and postprocessing each run many CPU instances per pod). 3. Served sustained gRPC inference (hundreds of RPS/pod, dynamic batching ~30 on the TRT step) for ~9.5h. 4. Observed the abort + wedge described above. Model descriptions: - **preprocessing** (Python backend, CPU): input `pre_input__0` (event embeddings, FP-typed, variable-length sequence); outputs `pre_output__0` (mask, INT32) and `pre_output__1` (padded embeddings). Pure NumPy: consecutive-duplicate filtering, truncation to seq_len=30, zero-padding, mask computation. - **postprocessing** (Python backend, CPU): input `post_input__0` (FP16 scores, dims `[30, 3743]`); output `post_output__0` (**TYPE_STRING**, dims `[1]`). Pure NumPy + Python: top-k selection, eligibility filtering, softmax, formats a string. This is the only stage emitting an object/string tensor. - **model** (TensorRT, GPU): not implicated (GPU clean). Representative config — postprocessing `config.pbtxt`: ```protobuf name: "postprocessing_model" backend: "python" max_batch_size: 200 input [ { name: "post_input__0", data_type: TYPE_FP16, dims: [30, 3743] } ] output [ { name: "post_output__0", data_type: TYPE_STRING, dims: [1] } ] instance_group [ { count: 24, kind: KIND_CPU } ] ``` (Ensemble and preprocessing config.pbtxt available; can attach. The `post_input__0` dims and instance counts are the operative details.) **Expected behavior** 1. A Python backend stub that dies (SIGABRT/heap corruption) should not leave the server in a state where inference is silently wedged while `/v2/health/ready` keeps returning `200`. The health/readiness endpoints should reflect that the backend can no longer serve, OR the stub should be detected as unhealthy and restarted (we saw neither — no restart log line, health stayed green). This is the part overlapping #8604. 2. Separately, we would like guidance on capturing the corrupting stack — e.g. a supported way to run the python backend stub under AddressSanitizer or with `GLIBC_TUNABLES=glibc.malloc.check=3` in the NGC image, and whether core dumps from the stub are written anywhere by default. That would let us localize whether the corruption is in Triton's python backend, in tensor (de)serialization, or in a dependency. Happy to provide the full pod spec, image digest, both `model.py` files (to show the model code is pure Python/NumPy with no in-place tensor mutation), and the Grafana panels (queue-depth-spike with zero recorded failures) if useful.