ITADN

[Question] Eagle3 training seems to freeze target embeddings but train a new draft lm_head

#478OpenHannibal046 创建于 2026-03-02
H
Hannibal046commented
Hi, thanks for open-sourcing this project. While reading the Eagle3 training code, I noticed a behavior that seems different from my understanding of the Eagle3 paper / method, and I wanted to confirm whether this is intentional. ## What I observed In the current Eagle3 training path: 1. The draft model loads the **target model embeddings**: - `scripts/train_eagle3.py` ```python draft_model.load_embedding(args.target_model_path, embedding_key=args.embedding_key) draft_model.freeze_embedding() ``` - so the draft embedding is copied from the target model and then frozen. 2. The draft model defines its **own lm_head**: - `specforge/modeling/draft/llama3_eagle.py` ```python self.lm_head = nn.Linear(config.hidden_size, config.draft_vocab_size, bias=False) ``` - and logits are produced from this draft `lm_head`: ```python return self.lm_head(norm_hidden_states) ``` This seems a bit surprising to me, especially since the embedding is reused from the target model, but the output head is not. Is this behavior intentional? Thanks.
1 条评论