Jobs permanently stuck in "doing" — two scenarios with no recovery mechanism

Our versions : - openaleph 5.1.0 (Latest : 5.2.0-rc3) / procrastinate 3.6.0 (Latest : 3.7.2) **Scenario 1 — Orphaned jobs (worker row pruned before stalled job detection)** `prune_stalled_workers` hard-deletes the dead worker row. Then `select_stalled_jobs_by_heartbeat` uses a `LEFT JOIN` on `procrastinate_workers` to find stalled jobs — but the worker is gone, so the join finds nothing. The job stays doing forever, invisible to any retry logic. The SQL in [queries.sql (lines 39–53)](https://github.com/procrastinate-org/procrastinate/blob/3.6.0/procrastinate/sql/queries.sql#L39-L53): ``` WITH stalled_workers AS ( SELECT id FROM procrastinate_workers WHERE last_heartbeat < NOW() - (%(seconds_since_heartbeat)s || ' SECOND')::INTERVAL ) SELECT job.id, ... FROM procrastinate_jobs job LEFT JOIN stalled_workers sw ON sw.id = job.worker_id WHERE job.status = 'doing' AND (job.worker_id IS NULL OR sw.id IS NOT NULL) ``` Once the worker row is deleted: `worker_id IS NULL → FALSE` (still points to deleted ID), `sw.id IS NOT NULL → FALSE` (worker gone from table). Job is never returned. **Scenario 2 — Stale jobs (worker restarts with same ID)** In Kubernetes, a restarted pod keeps the same name. Procrastinate uses the pod name as worker ID. The new process re-registers with the same ID and sends fresh heartbeats, but knows nothing about jobs the old process was running. Both cleanup mechanisms pass: the worker exists (not orphaned) and has fresh heartbeats (not stalled). But the old jobs are abandoned — stuck `doing` forever. In production we observed jobs stuck 5–8 days this way. Issue seems not to be fixed even on the latest version. Powered by **LE MONDE**