[Question] Eagle3 training seems to freeze target embeddings but train a new draft lm_head
Hi, thanks for open-sourcing this project.
While reading the Eagle3 training code, I noticed a behavior that seems different from my understanding of the Eagle3 paper / method, and I wanted to confirm whether this is intentional.
## What I observed
In the current Eagle3 training path:
1. The draft model loads the **target model embeddings**:
- `scripts/train_eagle3.py`
```python
draft_model.load_embedding(args.target_model_path, embedding_key=args.embedding_key)
draft_model.freeze_embedding()
```
- so the draft embedding is copied from the target model and then frozen.
2. The draft model defines its **own lm_head**:
- `specforge/modeling/draft/llama3_eagle.py`
```python
self.lm_head = nn.Linear(config.hidden_size, config.draft_vocab_size, bias=False)
```
- and logits are produced from this draft `lm_head`:
```python
return self.lm_head(norm_hidden_states)
```
This seems a bit surprising to me, especially since the embedding is reused from the target model, but the output head is not. Is this behavior intentional?
Thanks.
1 条评论