ITADN

LoRA training cannot resume from checkpoint — Missing key(s) in state_dict for TextToSemantic

#1295Openduchengyao 创建于 2026-06-04
bugstale
D
duchengyaocommented
### Self Checks - [x] This template is only for bug reports. For questions, please visit [Discussions](https://github.com/fishaudio/fish-speech/discussions). - [x] I have thoroughly reviewed the project documentation (installation, training, inference) but couldn't find information to solve my problem. [English](https://speech.fish.audio/) [中文](https://speech.fish.audio/zh/) [日本語](https://speech.fish.audio/ja/) [Portuguese (Brazil)](https://speech.fish.audio/pt/) - [x] I have searched for existing issues, including closed ones. [Search issues](https://github.com/fishaudio/fish-speech/issues) - [x] I confirm that I am using English to submit this report (我已阅读并同意 [Language Policy](https://github.com/fishaudio/fish-speech/issues/515)). - [x] [FOR CHINESE USERS] 请务必使用英文提交 Issue,否则会被关闭。谢谢!:) - [x] Please do not modify this template and fill in all required fields. ### Cloud or Self Hosted Self Hosted (Source) ### Environment Details - fish-speech version: 2.0.0 - PyTorch: 2.8.0 - Lightning: >= 2.1.0 - GPU: NVIDIA RTX 4090 (48GB) - LoRA config: r_8_alpha_16 ### Steps to Reproduce 1. Start LoRA training: ```bash python fish_speech/train.py --config-name text2semantic_finetune \ project=test_lora +lora@model.model.lora_config=r_8_alpha_16 ``` 2. Let it run for a few steps, then interrupt (e.g., Ctrl+C) 3. Re-run the exact same command to resume from the saved checkpoint ### ✔️ Expected Behavior The training should resume from the latest checkpoint, restoring: - LoRA weights (trainable params) - Optimizer states (AdamW momentum/variance) - LR scheduler states - Global step count Loss curve should be continuous — the first step after resume should produce similar loss to the last step before interruption. ### ❌ Actual Behavior Lightning's `CheckpointConnector.restore_model()` fails with: ``` RuntimeError: Error(s) in loading state_dict for TextToSemantic: Missing key(s) in state_dict: "model.embeddings.weight", "model.codebook_embeddings.weight", "model.layers.0.attention.wqkv.weight", "model.layers.0.attention.wo.weight", "model.layers.0.attention.q_norm.weight", "model.layers.0.attention.k_norm.weight", "model.layers.0.feed_forward.w1.weight", ... ``` (All 36 layers × 3 attention + 3 FFN + 2 norms = hundreds of missing keys, plus embeddings, fast layers, norm, and output.)
1 条评论