Text encoder (LTXAVTEModel_) evicted/reloaded far more aggressively on v0.30 vs v0.22, causing 3-10x slower prompt times on VRAM-constrained GPUs

### Custom Node Testing - [x] I have tried disabling custom nodes and the issue persists (see [how to disable custom nodes](https://docs.comfy.org/troubleshooting/custom-node-issues#step-1%3A-test-with-all-custom-nodes-disabled) if you need help) ### Expected Behavior Once warmed up over a few consecutive runs, prompt execution time on v0.30 should converge to a stable, low value comparable to v0.22 (~36-40 seconds for this workflow), not remain erratic and repeatedly spike between ~42 and ~366 seconds across consecutive identical prompts with no changes to the workflow, model files, or settings. Specifically, the text encoder (LTXAVTEModel_, ~13GB) should stay resident in memory across consecutive prompts once loaded — similar to the behavior observed on v0.22 — rather than being evicted and fully reloaded from disk on seemingly arbitrary runs, which is the primary driver of the slow/inconsistent timing on v0.30. ### Actual Behavior On v0.30.0, prompt execution time does not converge to a stable value even after multiple consecutive identical prompts (same workflow, same model files, no restart between runs). Across 4 consecutive runs: 365.99s → 42.78s → 136.56s → 123.17s The text encoder (LTXAVTEModel_, ~13GB) is not staying resident in memory reliably: - Runs 1 and 2 show no "Requested to load LTXAVTEModel_" line at all in the console - Run 3 shows the text encoder being requested and fully reloaded from disk (13764MB staged), and this run jumps to 136.56s - Run 4 remains high (123.17s) even though the encoder should still be "warm" from run 3 By comparison, on v0.22.x with the identical model files and workflow, 7 consecutive runs converge steadily downward and stabilize: 86.71s → 62.95s → 50.00s → 49.97s → 44.87s → 44.04s → 36.47s Sampler step time itself is consistent and fast on both versions (~1.8-2.9s/it for 8 steps) — the discrepancy is entirely in model loading/staging overhead between prompts, not in compute/sampling time. ### Steps to Reproduce 1. Use a GPU where combined VRAM-constrained models exceed available VRAM (tested on RTX 2080 Ti, 11GB) — this forces ComfyUI's dynamic VRAM loading/paging for both the diffusion model and text encoder. 2. Load the following in a standard LTXAV text-to-video workflow: - LTXAV diffusion model (~13.3GB staged) - LTXAVTEModel_ text encoder (~13.0-13.8GB staged) - VideoVAE, AudioVAE 3. Set resolution to 1280x720, 8 sampling steps. 4. Queue the same prompt 4-5 times consecutively, WITHOUT restarting ComfyUI or changing the workflow/models between runs. 5. Watch the console log for each run: - Does "Requested to load LTXAVTEModel_" appear on every run, or only some? - Record the "Prompt executed in X seconds" line for each run. 6. Repeat the same 4-5 consecutive runs on v0.22.x with the identical model files and workflow for comparison. 7. Compare the sequence of execution times between versions — v0.22 should converge downward to a stable low value; v0.30 does not. ### Debug Logs ```powershell === v0.22.x (portable, fresh install) — 7 consecutive runs === Model VideoVAE prepared for dynamic VRAM loading. 2769MB Staged. 0 patches attached. Prompt executed in 86.71 seconds got prompt 0 models unloaded. Model LTXAVTEModel_ prepared for dynamic VRAM loading. 12662MB Staged. 0 patches attached. Force pre-loaded 400 weights: 1745 KB. Model LTXAV prepared for dynamic VRAM loading. 13279MB Staged. 0 patches attached. Force pre-loaded 608 weights: 1641 KB. 100%| 8/8 [00:23<00:00, 2.94s/it] Requested to load AudioVAE loaded completely; 346.73 MB loaded, full load: True 0 models unloaded. Model VideoVAE prepared for dynamic VRAM loading. 2769MB Staged. 0 patches attached. Prompt executed in 62.95 seconds got prompt 0 models unloaded. Model LTXAVTEModel_ prepared for dynamic VRAM loading. 12662MB Staged. 0 patches attached. Force pre-loaded 290 weights: 1497 KB. Model LTXAV prepared for dynamic VRAM loading. 13282MB Staged. 0 patches attached. 100%| 8/8 [00:15<00:00, 1.93s/it] Requested to load AudioVAE loaded completely; 346.73 MB loaded, full load: True Model VideoVAE prepared for dynamic VRAM loading. 2769MB Staged. 0 patches attached. Prompt executed in 50.00 seconds got prompt VAE load device: cuda:0, offload device: cpu, dtype: torch.bfloat16 Model LTXAV prepared for dynamic VRAM loading. 13282MB Staged. 0 patches attached. 100%| 8/8 [00:14<00:00, 1.86s/it] Requested to load AudioVAE loaded completely; 346.73 MB loaded, full load: True VAE load device: cuda:0, offload device: cpu, dtype: torch.bfloat16 Requested to load VideoVAE Model VideoVAE prepared for dynamic VRAM loading. 1384MB Staged. 0 patches attached. Prompt executed in 28.73 seconds got prompt Model LTXAVTEModel_ prepared for dynamic VRAM loading. 12662MB Staged. 0 patches attached. Force pre-loaded 290 weights: 1497 KB. Model LTXAV prepared for dynamic VRAM loading. 13282MB Staged. 0 patches attached. 100%| 8/8 [00:14<00:00, 1.84s/it] Requested to load AudioVAE loaded completely; 346.73 MB loaded, full load: True Model VideoVAE prepared for dynamic VRAM loading. 1384MB Staged. 0 patches attached. Prompt executed in 50.00 seconds got prompt Model LTXAVTEModel_ prepared for dynamic VRAM loading. 12662MB Staged. 0 patches attached. Force pre-loaded 290 weights: 1497 KB. Model LTXAV prepared for dynamic VRAM loading. 13282MB Staged. 0 patches attached. 100%| 8/8 [00:14<00:00, 1.87s/it] Requested to load AudioVAE loaded completely; 346.73 MB loaded, full load: True Model VideoVAE prepared for dynamic VRAM loading. 1384MB Staged. 0 patches attached. Prompt executed in 49.97 seconds got prompt Model LTXAVTEModel_ prepared for dynamic VRAM loading. 12662MB Staged. 0 patches attached. Force pre-loaded 290 weights: 1497 KB. Model LTXAV prepared for dynamic VRAM loading. 13282MB Staged. 0 patches attached. 100%| 8/8 [00:14<00:00, 1.84s/it] Requested to load AudioVAE loaded completely; 346.73 MB loaded, full load: True Model VideoVAE prepared for dynamic VRAM loading. 1384MB Staged. 0 patches attached. Prompt executed in 36.47 seconds === v0.30.0 (portable, same model files, same workflow) — 4 consecutive runs === [INFO] Requested to load LTXAV [INFO] Model LTXAV prepared for dynamic VRAM loading. 13279MB Staged. 0 patches attached. Force pre-loaded 608 weights: 1641 KB. 100%| 8/8 [00:17<00:00, 2.23s/it] [INFO] Requested to load AudioVAE [INFO] loaded completely; 346.73 MB loaded, full load: True [INFO] VAE load device: cuda:0, offload device: cpu, dtype: torch.bfloat16 [INFO] Requested to load VideoVAE [INFO] 0 models unloaded. [INFO] Model VideoVAE prepared for dynamic VRAM loading. 1384MB Staged. 0 patches attached. [INFO] Prompt executed in 365.99 seconds [INFO] got prompt [INFO] Model LTXAV prepared for dynamic VRAM loading. 13279MB Staged. 0 patches attached. Force pre-loaded 608 weights: 1641 KB. 100%| 8/8 [00:16<00:00, 2.08s/it] [INFO] Requested to load AudioVAE [INFO] loaded completely; 346.73 MB loaded, full load: True [INFO] VAE load device: cuda:0, offload device: cpu, dtype: torch.float32 [INFO] Requested to load VideoVAE [INFO] 0 models unloaded. [INFO] Model VideoVAE prepared for dynamic VRAM loading. 2769MB Staged. 0 patches attached. [INFO] Prompt executed in 42.78 seconds [INFO] got prompt [INFO] Requested to load LTXAVTEModel_ [INFO] 0 models unloaded. [INFO] Model LTXAVTEModel_ prepared for dynamic VRAM loading. 13764MB Staged. 0 patches attached. Force pre-loaded 400 weights: 1745 KB. [INFO] Model LTXAV prepared for dynamic VRAM loading. 13279MB Staged. 0 patches attached. Force pre-loaded 608 weights: 1641 KB. 100%| 8/8 [00:16<00:00, 2.09s/it] [INFO] Requested to load AudioVAE [INFO] loaded completely; 346.73 MB loaded, full load: True [INFO] 0 models unloaded. [INFO] Model VideoVAE prepared for dynamic VRAM loading. 2769MB Staged. 0 patches attached. [INFO] Prompt executed in 136.56 seconds [INFO] got prompt [INFO] Requested to load LTXAVTEModel_ [INFO] 0 models unloaded. [INFO] Model LTXAVTEModel_ prepared for dynamic VRAM loading. 13764MB Staged. 0 patches attached. Force pre-loaded 400 weights: 1745 KB. [INFO] Model LTXAV prepared for dynamic VRAM loading. 13279MB Staged. 0 patches attached. Force pre-loaded 608 weights: 1641 KB. 100%| 8/8 [00:16<00:00, 2.08s/it] [INFO] Requested to load AudioVAE [INFO] loaded completely; 346.73 MB loaded, full load: True [INFO] 0 models unloaded. [INFO] Model VideoVAE prepared for dynamic VRAM loading. 2769MB Staged. 0 patches attached. [INFO] Prompt executed in 123.17 seconds ``` ### Other _No response_