face_search table completely empty after repeated OOM crashes during bulk facial recognition, causing degenerate single-cluster grouping

### I have searched the existing issues, both open and closed, to make sure this is not a duplicate report. - [x] Yes ### The bug ### I have searched the existing issues, both open and closed, to make sure this is not a duplicate report. - [x] Yes ### The bug After processing a large external library (~280,000 assets) with Facial Recognition enabled, the `asset_face` table was correctly populated (123,945 rows), but the `face_search` table (which stores the 512-dim vector embeddings used for clustering) remained **completely empty (0 rows)**. As a direct consequence, the "Facial Recognition" (grouping) job produced a degenerate result: instead of meaningful person clusters, 123,943 of 123,945 faces were grouped into a single `person`, with only 2 other faces left as singleton persons. Total `person` count: 3. This occurred after the server underwent multiple forced restarts (`pct stop` / `pct start` at the LXC level) during active Facial Recognition / OCR processing, triggered by host-level memory exhaustion (swap filling to 100%, load average >35 on a 4-core host, system becoming unresponsive to the point that even `systemctl`/process listing commands would hang for minutes). I suspect the embedding-write step (writing to `face_search`) is either skipped, silently failing, or not transactionally tied to the `asset_face` insert, so an interrupted/killed ML process can leave `asset_face` populated while `face_search` stays empty — with the clustering job then falling back to some degenerate behavior (single mega-cluster) instead of erroring out. This is related to but distinct from #18779, which reports "bad values" in `face_search` after a crash — in my case the table was entirely empty (0 rows), not populated with bad values. Running "Reset" on Facial Recognition (both from the UI and via the underlying job) did **not** repopulate `face_search` correctly — the same collapse into a single person occurred again after reprocessing all 123,945 faces a second time under (by then) more stable resource conditions (memory limit raised, CPU cores reduced from 4 to 3 to leave headroom for the host). ### Diagnostic queries used to confirm ```sql SELECT COUNT(*) FROM asset_face; -- 123945 SELECT COUNT(*) FROM face_search; -- 0 SELECT COUNT(*) FROM person; -- 3 SELECT "personId", COUNT(*) FROM asset_face GROUP BY "personId" ORDER BY COUNT(*) DESC LIMIT 10; -- personId | count -- --------------------------------------+-------- -- f96b4ac1-ba08-4d2e-befa-7654d27052b4 | 123943 -- | 1 -- 0d963c13-af3f-4105-b281-094581f653c7 | 1 ``` ### The OS that Immich Server is running on Debian GNU/Linux 13 (trixie), running inside an unprivileged Proxmox VE LXC container (deployed via the community-scripts ProxmoxVE `immich.sh` install script — **native systemd install, not Docker/Docker Compose**). Services: `immich-web.service`, `immich-ml.service`, `postgresql@16-main.service` (all systemd-managed, not containerized). Host: Intel Pentium Silver N6005 (4 cores @ 2.0GHz), 32GB RAM, 8GB swap. LXC allocated: initially 8GB RAM / 4GB swap / 4 cores, later increased to 16GB RAM / 6GB swap, cores reduced to 3 to preserve host stability. ### Version of Immich Server 3.0.1 (`IMMICH_VERSION=release` in `.env`, i.e. tracking latest release tag) ### Version of Immich Mobile App Not applicable to this bug (server-side issue) ### Platform with the issue - [x] Server ### Reproduction steps 1. Set up a large external library (~280,000 assets, mixed JPEG/RAW/video, spanning 20+ years) on a resource-constrained host (4 CPU cores, 32GB RAM) 2. Enable Facial Recognition with the `buffalo_l` model and OpenVINO GPU acceleration 3. Trigger a full library scan + Facial Recognition run 4. Allow the host to run out of memory (swap fills to 100%, load average exceeds core count several-fold) during active facial recognition processing 5. Force-restart the Immich services (`systemctl restart` / container restart) to regain responsiveness, one or more times, while the Facial Recognition queue is still processing 6. Once the Facial Recognition queue reaches 0 pending jobs, check `face_search` row count vs `asset_face` row count 7. Observe that `face_search` remains at 0 rows despite `asset_face` being fully populated, and that the `person` table shows a single dominant cluster containing nearly all faces ### Relevant log output Machine-learning service showed no explicit errors after the fact; the failures occurred silently during the periods where the host was unresponsive due to memory pressure (no clean error trace was captured before forced restarts, as the shell itself was unresponsive at the time — this may itself be worth noting, since it means the failure mode leaves no diagnosable trace in application logs). ### Additional context - Related: #18779 (bad values in `face_search` after crash — similar trigger, different failure mode) - Related: #26739 (OCR memory leak, still open as of March 2026 — same general class of ML-service memory exhaustion under sustained load) - Workaround currently in use: Facial Recognition disabled entirely until this is resolved, rest of the library (timeline, search, maps, albums) functions normally. ### The OS that Immich Server is running on Debian GNU/Linux 13 (trixie), running inside an unprivileged Proxmox VE LXC container ### Version of Immich Server 3.0.1 ### Version of Immich Mobile App N/A - this is a server-side only issue, not related to the mobile app ### Platform with the issue - [x] Server - [ ] Web - [ ] Mobile ### Device make and model N/A (server-side issue, no specific client device involved) ### Your docker-compose.yml content ```YAML N/A - not a Docker Compose deployment. This is a native (non-containerized) install via the community-scripts ProxmoxVE `immich.sh` script, running as systemd services (immich-web.service, immich-ml.service, postgresql@16-main.service) inside an unprivileged Proxmox LXC. Relevant .env content: TZ=Europe/Berlin IMMICH_VERSION=release NODE_ENV=production IMMICH_ALLOW_SETUP=true IMMICH_HELMET_FILE=true DB_HOSTNAME=127.0.0.1 DB_USERNAME=immich DB_PASSWORD=<redacted> DB_DATABASE_NAME=immich DB_VECTOR_EXTENSION=vectorchord REDIS_HOSTNAME=127.0.0.1 IMMICH_MACHINE_LEARNING_URL=http://127.0.0.1:3003 MACHINE_LEARNING_CACHE_FOLDER=/opt/immich/cache IMMICH_MEDIA_LOCATION=/opt/immich/upload ``` ### Your .env content ```Shell TZ=Europe/Berlin IMMICH_VERSION=release NODE_ENV=production IMMICH_ALLOW_SETUP=true IMMICH_HELMET_FILE=true DB_HOSTNAME=127.0.0.1 DB_USERNAME=immich DB_PASSWORD=<redacted> DB_DATABASE_NAME=immich DB_VECTOR_EXTENSION=vectorchord REDIS_HOSTNAME=127.0.0.1 IMMICH_MACHINE_LEARNING_URL=http://127.0.0.1:3003 MACHINE_LEARNING_CACHE_FOLDER=/opt/immich/cache IMMICH_MEDIA_LOCATION=/opt/immich/upload Note: This is a native (non-Docker) installation via the community-scripts ProxmoxVE `immich.sh` script, running as systemd services (immich-web.service, immich-ml.service, postgresql@16-main.service) inside an unprivileged Proxmox LXC container. ``` ### Reproduction steps 1. Set up a large external library (~280,000 assets, mixed JPEG/RAW/video, spanning 20+ years) on a resource-constrained host (4 CPU cores, 32GB RAM, Intel Pentium Silver N6005) 2. Enable Facial Recognition with the `buffalo_l` model and OpenVINO GPU acceleration 3. Trigger a full library scan + Facial Recognition run 4. Allow the host to run out of memory (swap fills to 100%, load average exceeds core count several-fold) during active facial recognition processing 5. Force-restart the Immich services (`pct stop`/`pct start` at the Proxmox LXC level) to regain responsiveness, one or more times, while the Facial Recognition queue is still processing 6. Once the Facial Recognition queue reaches 0 pending jobs, check `face_search` row count vs `asset_face` row count: SELECT COUNT(*) FROM asset_face; -- returned 123945 SELECT COUNT(*) FROM face_search; -- returned 0 SELECT COUNT(*) FROM person; -- returned 3 7. Observe that `face_search` remains at 0 rows despite `asset_face` being fully populated, and that the `person` table shows a single dominant cluster containing nearly all faces: SELECT "personId", COUNT(*) FROM asset_face GROUP BY "personId" ORDER BY COUNT(*) DESC LIMIT 10; personId | count --------------------------------------+-------- f96b4ac1-ba08-4d2e-befa-7654d27052b4 | 123943 | 1 0d963c13-af3f-4105-b281-094581f653c7 | 1 8. Running "Reset" on Facial Recognition (both via UI and re-running the underlying job) did NOT repopulate `face_search` correctly — the same collapse into a single person occurred again after fully reprocessing all 123,945 faces a second time, even under more stable resource conditions (memory limit raised from 8GB to 16GB, CPU cores reduced from 4 to 3 to leave headroom for the host) ### Relevant log output ```shell No explicit error trace was captured at the time of failure. The machine-learning service (immich-ml.service) produced no error output after the fact. The failures occurred silently during periods where the host was completely unresponsive due to memory pressure (load average 35-40 on 4 cores, swap at 100%) — even basic commands like `systemctl status` or `docker ps`-equivalent process listing would hang for minutes during these episodes, meaning no clean diagnostic log could be captured before a forced restart was necessary. This lack of any diagnosable trace in application logs may itself be a relevant data point: the failure mode appears to be a silent/hard OOM-kill rather than an application-level exception that would normally be logged. Related issues: - #18779 (bad values in face_search after crash — similar trigger, different failure mode: in my case face_search was entirely empty, not populated with bad values) - #26739 (OCR memory leak in immich-machine-learning, still open as of March 2026 — same general class of ML-service memory exhaustion under sustained load) ``` ### Additional information I am a beginner with immich, but I would love to use it with all features without problems like this in the future. Thx a lot for your reading, fixing and your work in general. Love it! :-)