LoRA training cannot resume from checkpoint — Missing key(s) in state_dict for TextToSemantic
bugstale
### Self Checks
- [x] This template is only for bug reports. For questions, please visit [Discussions](https://github.com/fishaudio/fish-speech/discussions).
- [x] I have thoroughly reviewed the project documentation (installation, training, inference) but couldn't find information to solve my problem. [English](https://speech.fish.audio/) [中文](https://speech.fish.audio/zh/) [日本語](https://speech.fish.audio/ja/) [Portuguese (Brazil)](https://speech.fish.audio/pt/)
- [x] I have searched for existing issues, including closed ones. [Search issues](https://github.com/fishaudio/fish-speech/issues)
- [x] I confirm that I am using English to submit this report (我已阅读并同意 [Language Policy](https://github.com/fishaudio/fish-speech/issues/515)).
- [x] [FOR CHINESE USERS] 请务必使用英文提交 Issue,否则会被关闭。谢谢!:)
- [x] Please do not modify this template and fill in all required fields.
### Cloud or Self Hosted
Self Hosted (Source)
### Environment Details
- fish-speech version: 2.0.0
- PyTorch: 2.8.0
- Lightning: >= 2.1.0
- GPU: NVIDIA RTX 4090 (48GB)
- LoRA config: r_8_alpha_16
### Steps to Reproduce
1. Start LoRA training:
```bash
python fish_speech/train.py --config-name text2semantic_finetune \
project=test_lora +lora@model.model.lora_config=r_8_alpha_16
```
2. Let it run for a few steps, then interrupt (e.g., Ctrl+C)
3. Re-run the exact same command to resume from the saved checkpoint
### ✔️ Expected Behavior
The training should resume from the latest checkpoint, restoring:
- LoRA weights (trainable params)
- Optimizer states (AdamW momentum/variance)
- LR scheduler states
- Global step count
Loss curve should be continuous — the first step after resume should produce similar loss to the last step before interruption.
### ❌ Actual Behavior
Lightning's `CheckpointConnector.restore_model()` fails with:
```
RuntimeError: Error(s) in loading state_dict for TextToSemantic:
Missing key(s) in state_dict: "model.embeddings.weight", "model.codebook_embeddings.weight", "model.layers.0.attention.wqkv.weight", "model.layers.0.attention.wo.weight", "model.layers.0.attention.q_norm.weight", "model.layers.0.attention.k_norm.weight", "model.layers.0.feed_forward.w1.weight", ...
```
(All 36 layers × 3 attention + 3 FFN + 2 norms = hundreds of missing keys, plus embeddings, fast layers, norm, and output.)
1 条评论