Postgres CDC: JDBC resource leak on engine-startup failure
type/bug
### Describe the bug
Running a shared Postgres CDC source (`pg_source`) against an AWS RDS Postgres instance, when the Debezium engine fails to start within the timeout, teardown raises `IllegalStateException: Cannot stop engine while tasks are starting…`, and the source actor retries every ~1 s. Each retry leaks a JDBC pool inside the compute JVM. In ~5 minutes a single compute pod accumulated **33 distinct JDBC thread pools**, eventually exhausting `max_connections` on the upstream RDS and locking out other clients. Only a compute pod restart frees the sockets.
This is a high-blast-radius defect: a single failing CDC source can exhaust `max_connections` on the upstream Postgres for every other client (other RW sources, ops sessions, application traffic).
### Error message/log
```text
ERROR ... Debezium streaming source of RW_CDC_<id> failed to start in timeout 60
ERROR ... Cdc engine failed. ... IllegalStateException: Cannot stop engine while
tasks are starting, this may lead to leaked resource.
Wait for the tasks to be fully started.
In compute pod thread dumps, the leaked pools appear as `pool-NNNN-thread-1` (one per failed startup). On RDS, `pg_stat_activity` shows the connection count climbing roughly linearly with retry count until `max_connections` is hit.
```
### To Reproduce
1. Configure a shared Postgres CDC source against an RDS Postgres where the configured replication slot is unreachable — e.g. its WAL has been recycled past `confirmed_flush_lsn` (`wal_status = 'lost'`).
2. Let the source attempt to start. Debezium engine startup will fail within the timeout.
3. Observe that the source actor retries every ~1 s and that each retry leaks a JDBC pool inside the compute JVM (visible in thread dumps as `pool-NNNN-thread-1`, and on the upstream as a steadily climbing connection count in `pg_stat_activity`).
### Expected behavior
A failed engine startup should fully release any JDBC connections, thread pools, and other resources it acquired before the source actor schedules its next retry attempt. The retry-on-failure path is correct in principle; it just needs to be safe to call repeatedly.
### How did you deploy RisingWave?
Kubernetes on AWS EKS, deployed via the RisingWave Operator. The affected component is `statefulset/risingwave-compute`. Upstream source is AWS RDS for PostgreSQL.
### The version of RisingWave
2.8.2
### Additional context
Mitigation we used during the incident:
1. `kubectl scale statefulset risingwave-compute --replicas=0` — drained the JVMs and freed all leaked JDBC sockets on RDS.
2. Resolved the underlying upstream-slot problem.
3. Scaled compute back up.
Outcome: connection saturation resolved and the retry loop now fails cleanly without leaking.
Full compute logs and `rw_event_logs` from the incident window are available on request.
0 条评论