ITADN

Issues with row-major vs col-major in the repo and blog.

#15OpenKillianMelsen 创建于 2025-03-07
K
KillianMelsencommented
The naive kernel uses `x` for the x dimension and `y` for the y dimension. https://github.com/siboehm/SGEMM_CUDA/blob/60cba6f9b20a198116c76f18de8047f44df8c8b8/src/kernels/1_naive.cuh#L17-L18 Looking at the CUDA programming guide (e.g., here: https://docs.nvidia.com/cuda/cuda-c-programming-guide/#shared-memory) shows that `x` thus corresponds to the `col` and `y` to the `row`. This is also what the figures in the blogpost use. The kernel itself, however, then proceeds to assume column-major formatting: https://github.com/siboehm/SGEMM_CUDA/blob/60cba6f9b20a198116c76f18de8047f44df8c8b8/src/kernels/1_naive.cuh#L20-L28 C[4, 0], (x = 4, y = 0, first row, fifth column, 10x10 matrix so N = 10), would be the fifth element in linear memory for row-major formatting, but `C[x * N + y = 40]` makes it the 41st element, which corresponds to column-major formatting. The programming guide assumes row-major formatting in the kernel examples. Why the choice to make the kernels in this repo use column-major formatting? Perhaps because cuBLAS also uses column-major instead of row-major storage? This also causes the `if` statement for the tile quantization to be wrong I think: https://github.com/siboehm/SGEMM_CUDA/blob/60cba6f9b20a198116c76f18de8047f44df8c8b8/src/kernels/1_naive.cuh#L10-L11 https://github.com/siboehm/SGEMM_CUDA/blob/60cba6f9b20a198116c76f18de8047f44df8c8b8/src/kernels/1_naive.cuh#L21 It checks whether `x` (which is the column index) is smaller than `M` (which is the number of rows of C). Similarly `y < N`, which compares the row index to the number of columns... The [figure just before the introduction of kernel 2 on the blog](https://siboehm.com/articles/22/CUDA-MMM#:~:text=Below%20is%20a%20visualization%20of%20the%20memory%20access%20pattern%20of%20our%20naive%20kernel%2C%20taking%20two%20threads%20A%20(red)%20and%20B%20(green)%20as%20an%20example%3A) also seems wrong to me. It states that "A, B, C, are stored in row-major order. This means that the last index (here y) is the one that iterates continuous through memory (=has stride 1)." An L x R two dimensional matrix indexed by (x, y) uses x for columns and y for rows. In row-major format the x then iterates continuously while the y only changes after each full row. So x has a stride of 1 while y has a stride of R if I'm not mistaken.
2 条评论