ITADN

Use rope_theta from config & correct llama3 freqs

#117Pull Requestrdyro 创建于 2024-12-28
R
rdyrocommented
As pointed out by `antra` on the LLM discord `rope_theta` is never passed to the `apply_rotary_emb` function. This PR fixes that by passing the parameter. Additionally, Llama3 requires a frequency correction: [https://github.com/huggingface/transformers/blob/5c75087aeee7081025370e10d1f571a11600f1ae/src/transformers/modeling_rope_utils.py#L310]
合并状态:未合并 0 条评论