ITADN

Fix FP8 linear layer dimension check to prevent runtime error

#6393Pull Requestssam18 创建于 2025-11-12
S
ssam18commented
Fixes #6390 ## Problem When `use_fp8=True` is enabled in `HybridParallelPlugin` and the model has output layers with dimensions not divisible by 16 (e.g., binary classification with 2 outputs), the training fails with: ``` Expected both dimensions of mat2 to be divisible by 16 but got torch.Size([768, 2]) ``` ## Root Cause `torch._scaled_mm` requires **both dimensions** of the weight matrix to be divisible by 16. The existing check in `linear_fp8()` only validated: - Input dimension (`input.shape[-1]`) - Batch dimensions (`np.prod(input.shape[:-1])`) But it did **not** check the output dimension (`weight.shape[0]`). When using `GPT2ForSequenceClassification` with `num_labels=2`, the score layer has weight shape `[768, 2]`, where 2 is not divisible by 16. ## Solution Added a check for `weight.shape[0] % 16 != 0` to fallback to regular `F.linear` when the output dimension is not compatible with FP8. ```python if input.shape[-1] % 16 != 0 or np.prod(input.shape[:-1]) % 16 != 0 or weight.shape[0] % 16 != 0: return F.linear(input, weight, bias) ``` ## Testing This fix allows the model to: - Use FP8 for layers with compatible dimensions (performance benefit) - Fallback to standard FP16/BF16 for incompatible layers (correctness) - Run successfully with small output dimensions (e.g., binary classification) The change is backward compatible and doesn't affect existing working configurations.
合并状态:未合并 2 条评论