ITADN

SGEMM benchmark may hit CUBLAS_STATUS_ALLOC_FAILED due to cublas handle leak in kernels/sgemm/sgemm_cublas.cu

#414Closedlnxtree 创建于 2026-04-05
L
lnxtreecommented
- Summary Running kernels/sgemm/sgemm.py can fail at torch.matmul with: RuntimeError: CUDA error: CUBLAS_STATUS_ALLOC_FAILED when calling cublasCreate(handle) - Reproduction Build/run SGEMM benchmark in kernels/sgemm Observe custom cublas path works first, then torch.matmul fails with cublasCreate(handle) Observed output out_f32x4(t8x4dbuf): ... out_f32(cublas): ... Traceback ... RuntimeError: CUDA error: CUBLAS_STATUS_ALLOC_FAILED when calling cublasCreate(handle) - Root cause In kernels/sgemm/sgemm_cublas.cu, cublasCreate is called per invocation in cublas_sgemm and cublas_sgemm_tf32, but handle is not destroyed, causing resource leak. - Proposed fix Pair each cublasCreate with cublasDestroy Add return-status checks for cublasCreate, cublasSetMathMode, cublasGemmEx, cublasDestroy - Impact Prevents handle/resource accumulation and avoids later cublasCreate failure in torch.matmul.
关闭于 2026-04-05 0 条评论