ITADN

[Bug] dflash cross block data leak

#470OpenTechxGenus 创建于 2026-02-13
T
TechxGenuscommented
### Checklist - [x] 1. I have searched related issues but cannot get the expected help. - [x] 2. The bug has not been fixed in the latest version. - [x] 3. Please note that if the bug-related issue you submitted lacks corresponding environment info and a minimal reproducible demo, it will be challenging for us to reproduce and resolve the issue, reducing the likelihood of receiving feedback. - [x] 4. If the issue you raised is not a bug but a question, please raise a discussion at https://github.com/sgl-project/SpecForge/discussions/new/choose Otherwise, it will be closed. - [x] 5. Please use English, otherwise it will be closed. ### Describe the bug I tried training a dflash model using the script in this repo, and noticed that performance is much better when `num-anchors` is small. When `num-anchors` is large, the training loss/acc look abnormally good (loss unusually low / acc unusually high compared to expect), but all the evaluation results are much worse than reported in its paper. So I dug into the code and found the `random-anchor` path: cross-block context visibility is determined purely by the `block index` (`k_ctx < q_block_id`). When the anchor spacing becomes `smaller than `block_size`, blocks can overlap. This overlap allows a later block to see future target hidden states from the previous block. So the larger `num-anchors` making more overlaps and training loss/acc look unusually better. ### Reproduction same as https://github.com/sgl-project/SpecForge/issues/465 ### Environment same as https://github.com/sgl-project/SpecForge/issues/465
0 条评论