Connection pool exhaustion in 1.19.0/1.19.1 (ee-postgresql)

**Summary** After updating the ee-postgresql image from 1.18.4 → 1.19.1, Pangolin became unable to acquire database connections within minutes of startup. Every query — both getResourceByDomain (per-request session verification) and getTraefikConfig (the Traefik config provider) — fails with timeout exceeded when trying to connect. This takes down all proxied resources at once. The PostgreSQL server is healthy and idle (33/100 connections), yet Pangolin's pool holds ~20 connections that PostgreSQL reports as idle while Pangolin simultaneously reports it cannot obtain a connection. This is the signature of pooled clients being acquired but never released (or the event loop being starved), saturating the pool at its max. A container restart restores service for a few minutes, then it re-degrades as the pool re-saturates. Downgrading to ee-postgresql-1.18.4 resolves it completely. 1.19.2 does not touch the relevant code and is also affected. **Logs** Both the per-request path and the config provider fail the same way: ``` [cause]: Error: timeout exceeded when trying to connect at async getResourceByDomain (server/db/queries/verifySessionQueries.ts:102:30) at async verifyResourceSession (server/routers/badger/verifySession.ts:156:28) [cause]: Error: timeout exceeded when trying to connect at async getTraefikConfig (drizzle-orm/node-postgres/session.js:117) at async traefikConfigProvider ``` **Evidence it is pool saturation, not the database** PostgreSQL is healthy and far from any limit, but Pangolin's connections are all parked idle while it still can't get one: ``` -- total connections vs limit (Postgres is fine) total | max -------+----- 33 | 100 -- pangolin DB: ~20 connections, effectively all idle, while every new query times out state | count | oldest --------+-------+---------- active | 3 | idle | 17 | 00:00:16 ``` A request that does get a connection completes in ~67 ms, so the database and query are fast — the failure is purely connection acquisition. PostgreSQL's own log shows no too many clients/slot exhaustion. **Suspected cause** The regression window points squarely at the 1.19.0 "improve performance in hot paths / API endpoints with thousands of sites and users" rework: - Introduced in 1.19.0; 1.19.1 (hotfix) and 1.19.2 do not modify the DB session/query layer. - 1.18.4 is unaffected. This suggests a code path in the reworked getResourceByDomain / Traefik config provider that checks out a pool client and does not release it on all branches (e.g., on early return or error), so the default pool (max ≈ 20) saturates. Pinning to 1.18.4 is a clean workaround. **Ask** Could you review connection acquire/release in the 1.19.0 hot-path changes — specifically whether pooled clients are released on every path (including error/early-return) in verifySessionQueries/the Traefik config provider — and whether a default pool max of ~20 is expected to be held indefinitely under normal traffic? Happy to provide additional pg_stat_activity snapshots, full logs, or test a patched build. ### Environment - OS Type & Version: Ubuntu 24.04.4 LTS (kernel 6.8.0-124-generic) - Pangolin Version: 1.19.1 (ee-postgresql) — bug present; also reproduces on 1.19.0; NOT present on 1.18.4 - Edition: Enterprise (ee-postgresql image) - Gerbil Version: 1.4.1 (:latest) - Traefik Version: v3.6.10 - Newt Version: 1.13.0 (:latest) - Client Version: N/A ### To Reproduce 1. Run ee-postgresql-1.19.1 against a PostgreSQL backend under normal traffic. 2. Within a few minutes, all resources become unreachable and the logs fill with the error below. 3. Restart the Pangolin container → service recovers briefly, then degrades again as the pool re-saturates. ### Expected Behavior - Expected: each query checks out a pooled connection and returns it; steady-state pool stays well below max. - Actual: the pool grows to ~20 and stays there; every new acquisition times out.