ITADN

[CUDA] Recycle CUDA events

#2604Pull Requestzcbenz 创建于 2025-09-17已合并
Z
zcbenzcommented
While the `cudaEventCreate` call is very cheap (takes about 1~2µs) and we almost get no performance gain by eliminating it during inference, it is disturbing when doing profiling because measuring the kernel/graph time requires a pair of events for each kernel/graph. This PR makes the `CudaEvent` class cache the underlying handle so the native CUDA events are reused instead of being recreated every time. For command `mlx_lm.generate --model meta-llama/Llama-3.2-1B-Instruct --prompt 'Write a story about Einstein' -m 64`, the number of `cudaEventCreate` calls reduced from 69 to 7, and there is no observable change in performance.
合并状态:已合并 合并于 2025-09-23 关闭于 2025-09-23 2 条评论