## Highlights * Wider support for cuda quantized matmuls (#3352, #3268, #3321, #3417, #3255) * MLX can be used by multiple threads for independent computations (#3405, #3348, #3281, #3423) * Added CUDA FFT support * JACCL is now a standalone lib (#3412) ## What's Changed * Bump by @angeloskath in https://github.com/ml-explore/mlx/pull/3244 * win: re-enable and fix cuDNN performance by @dhiltgen in https://github.com/ml-explore/mlx/pull/3242 * Fix crashes in multi-threaded process teardown by @louen in https://github.com/ml-explore/mlx/pull/3167 * [CUDA] Add FFT support by @lucasnewman in https://github.com/ml-explore/mlx/pull/3243 * [CUDA] Implement MaskedScatter by @Lyxot in https://github.com/ml-explore/mlx/pull/3151 * docs: fix PyTorch to MLX conversion example by @LxYuan0420 in https://github.com/ml-explore/mlx/pull/3265 * update requirements for Macbook Neo by @tosh in https://github.com/ml-explore/mlx/pull/3257 * fix comparison op JVP returning bool tangents instead of input dtype by @mm65x in https://github.com/ml-explore/mlx/pull/3253 * fix nn.GRU skipping bhn bias when hidden is None by @mm65x in https://github.com/ml-explore/mlx/pull/3252 * [CUDA] Pipelined QMM by @zcbenz in https://github.com/ml-explore/mlx/pull/3255 * tests: harden memory leak check in test_siblings_without_eval by @booxter in https://github.com/ml-explore/mlx/pull/3088 * Slice update with operation by @angeloskath in https://github.com/ml-explore/mlx/pull/3266 * Nax Refactor by @jagrit06 in https://github.com/ml-explore/mlx/pull/3271 * Fix building with CUDA toolkit 13.2 by @zcbenz in https://github.com/ml-explore/mlx/pull/3273 * [CUDA] fp and int4 quants for qmm_sm80 by @zcbenz in https://github.com/ml-explore/mlx/pull/3268 * Fix repr of conv layers by @angeloskath in https://github.com/ml-explore/mlx/pull/3275 * Merge DeviceStream into CommandEncoder by @zcbenz in https://github.com/ml-explore/mlx/pull/3264 * [CUDA] Search system-installed CUDA toolkit for headers by @zcbenz in https://github.com/ml-explore/mlx/pull/3277 * Create default random key lazily by @zcbenz in https://github.com/ml-explore/mlx/pull/3278 * Support indexing with any type which implmented `__index__` by @aisk in https://github.com/ml-explore/mlx/pull/3210 * Fix sort NaN handling for float16 and bfloat16 by @Lyxot in https://github.com/ml-explore/mlx/pull/3269 * Use thread local storage for frontend compile cache by @zcbenz in https://github.com/ml-explore/mlx/pull/3280 * [Metal][Performance]: Add split-K for quantized matmul (small M) by @Ziqiao-git in https://github.com/ml-explore/mlx/pull/3120 * [Metal] Fix depthwise conv 1D kernel name for large variant by @Brooooooklyn in https://github.com/ml-explore/mlx/pull/3289 * Fix stale transform copy-chain leaks by @Brooooooklyn in https://github.com/ml-explore/mlx/pull/3290 * Implement Pad::vmap to replace NYI stub by @Aristide021 in https://github.com/ml-explore/mlx/pull/3304 * logo files by @andresy in https://github.com/ml-explore/mlx/pull/3308 * Fix vmap + floor_divide: preserve integer dtype by @robert-johansson in https://github.com/ml-explore/mlx/pull/3292 * Fix moved-from shape bug in broadcast_arrays causing vmap bus error by @Aristide021 in https://github.com/ml-explore/mlx/pull/3310 * Use nb::ndarray for checking arrays by @zcbenz in https://github.com/ml-explore/mlx/pull/3283 * Add output_shapes for AddMM by @pHequals7 in https://github.com/ml-explore/mlx/pull/3262 * Manage Metal objects with smart pointers by @zcbenz in https://github.com/ml-explore/mlx/pull/3282 * [CUDA] support sorting complex numbers by @Lyxot in https://github.com/ml-explore/mlx/pull/3286 * Add norm parameter to FFT transforms (backward/ortho/forward) by @Aristide021 in https://github.com/ml-explore/mlx/pull/3287 * Make each thread have its own default stream by @zcbenz in https://github.com/ml-explore/mlx/pull/3281 * [CUDA] Implement BlockMaskedMM by @Lyxot in https://github.com/ml-explore/mlx/pull/3299 * Fix np bfloat16 misinterpreted as complex by @kellen-sun in https://github.com/ml-explore/mlx/pull/3146 * Remove no longer needed const_cast by @zcbenz in https://github.com/ml-explore/mlx/pull/3325 * Bump actions/deploy-pages from 4 to 5 by @dependabot[bot] in https://github.com/ml-explore/mlx/pull/3334 * Fix use after move by @angeloskath in https://github.com/ml-explore/mlx/pull/3343 * Decouple CommandEncoder from Device by @zcbenz in https://github.com/ml-explore/mlx/pull/3316 * Add vmap for BroadcastAxes by @angeloskath in https://github.com/ml-explore/mlx/pull/3344 * Add fftfreq, rfftfreq and scalar axes for fftshift/ifftshift by @declanhealy2 in https://github.com/ml-explore/mlx/pull/3298 * [Metal] Support sorting complex numbers by @Lyxot in https://github.com/ml-explore/mlx/pull/3314 * [CUDA] Fallback QMM by @zcbenz in https://github.com/ml-explore/mlx/pull/3315 * Make CommandEncoder thread local by @zcbenz in https://github.com/ml-explore/mlx/pull/3348 * [CUDA] 3/5/6-bit quants for qmm_naive by @zcbenz in https://github.com/ml-explore/mlx/pull/3352 * Fix regression in array creation by @angeloskath in https://github.com/ml-explore/mlx/pull/3353 * Use `metal` as the front-end for the metal linker by @louen in https://github.com/ml-explore/mlx/pull/3354 * Add printoptions by @ChristophePRAT in https://github.com/ml-explore/mlx/pull/3333 * Add a convenience for making local streams in python by @angeloskath in https://github.com/ml-explore/mlx/pull/3355 * Fix CMake finding wrong Python during pip install by @fijimunkii in https://github.com/ml-explore/mlx/pull/3375 * [CUDA] Add GatherQMM for quantized gather matmul by @Lyxot in https://github.com/ml-explore/mlx/pull/3321 * fix: fail build when Metal compiler header resolution fails by @dogukanveziroglu in https://github.com/ml-explore/mlx/pull/3332 * Fix: Correct cross-attention query routing in Post-LN TransformerDecoderLayer by @suryawanshishantanu6 in https://github.com/ml-explore/mlx/pull/3382 * [CUDA] Thread safety by @zcbenz in https://github.com/ml-explore/mlx/pull/3367 * Fix test "test get streams" missing initialization by @dseredkin in https://github.com/ml-explore/mlx/pull/3376 * Conjugate VJP and JVP support by @CameronChurchwell in https://github.com/ml-explore/mlx/pull/3386 * Fix int16 overflow in SDPA NAX mask indexing for KV sequences > 32K by @Clydingus in https://github.com/ml-explore/mlx/pull/3361 * Avoid joining threads on exit by @zcbenz in https://github.com/ml-explore/mlx/pull/3388 * Add clear_streams API for cleanup before exit by @zcbenz in https://github.com/ml-explore/mlx/pull/3395 * Update nanobind version to v2.12.0 by @jrp2014 in https://github.com/ml-explore/mlx/pull/3396 * Jaccl refactor by @angeloskath in https://github.com/ml-explore/mlx/pull/3412 * Fixes for CUDA CI by @zcbenz in https://github.com/ml-explore/mlx/pull/3413 * Validate safetensors data offsets by @MillaFleurs in https://github.com/ml-explore/mlx/pull/3364 * Validate safetensors data offsets against file boundaries by @matinsaurralde in https://github.com/ml-explore/mlx/pull/3410 * Document sort stability and NaN handling by @NeuralNoble in https://github.com/ml-explore/mlx/pull/3400 * ThreadLocalStream in C++ by @zcbenz in https://github.com/ml-explore/mlx/pull/3405 * Fix jaccl init bug by @angeloskath in https://github.com/ml-explore/mlx/pull/3418 * Segmented mm nax kernel by @angeloskath in https://github.com/ml-explore/mlx/pull/3419 * [CUDA] gather_mm by @zcbenz in https://github.com/ml-explore/mlx/pull/3414 * [CUDA] GatherQMM matrix-matrix sm80/naive path by @Lyxot in https://github.com/ml-explore/mlx/pull/3417 * [CUDA] Handle residue k in qmm_naive by @zcbenz in https://github.com/ml-explore/mlx/pull/3379 * Speed up NAX split-K by better tuning and routing and fix NAX addmm by @angeloskath in https://github.com/ml-explore/mlx/pull/3422 * Make Scheduler::enqueue thread safe by @zcbenz in https://github.com/ml-explore/mlx/pull/3423 * Fix flaky TestVmap.test_vmap_masked_scatter by @zcbenz in https://github.com/ml-explore/mlx/pull/3421 * Fix synchronize for ThreadLocalStream by @angeloskath in https://github.com/ml-explore/mlx/pull/3429 * Fix bytes_per_key truncation in random kernels (Metal + CUDA) by @dogukanveziroglu in https://github.com/ml-explore/mlx/pull/3432 * Throw meaningful error when Metal device is not found by @dogukanveziroglu in https://github.com/ml-explore/mlx/pull/3428 * Fix kernel cache collision in Compiled constructor by @dogukanveziroglu in https://github.com/ml-explore/mlx/pull/3427 * Fix mx.prod vjp for complex types by @CameronChurchwell in https://github.com/ml-explore/mlx/pull/3433 ## New Contributors * @LxYuan0420 made their first contribution in https://github.com/ml-explore/mlx/pull/3265 * @tosh made their first contribution in https://github.com/ml-explore/mlx/pull/3257 * @mm65x made their first contribution in https://github.com/ml-explore/mlx/pull/3253 * @booxter made their first contribution in https://github.com/ml-explore/mlx/pull/3088 * @Ziqiao-git made their first contribution in https://github.com/ml-explore/mlx/pull/3120 * @Brooooooklyn made their first contribution in https://github.com/ml-explore/mlx/pull/3289 * @Aristide021 made their first contribution in https://github.com/ml-explore/mlx/pull/3304 * @pHequals7 made their first contribution in https://github.com/ml-explore/mlx/pull/3262 * @declanhealy2 made their first contribution in https://github.com/ml-explore/mlx/pull/3298 * @fijimunkii made their first contribution in https://github.com/ml-explore/mlx/pull/3375 * @dogukanveziroglu made their first contribution in https://github.com/ml-explore/mlx/pull/3332 * @suryawanshishantanu6 made their first contribution in https://github.com/ml-explore/mlx/pull/3382 * @dseredkin made their first contribution in https://github.com/ml-explore/mlx/pull/3376 * @CameronChurchwell made their first contribution in https://github.com/ml-explore/mlx/pull/3386 * @Clydingus made their first contribution in https://github.com/ml-explore/mlx/pull/3361 * @jrp2014 made their first contribution in https://github.com/ml-explore/mlx/pull/3396 * @matinsaurralde made their first contribution in https://github.com/ml-explore/mlx/pull/3410 * @NeuralNoble made their first contribution in https://github.com/ml-explore/mlx/pull/3400 **Full Changelog**: https://github.com/ml-explore/mlx/compare/v0.31.1...v0.31.2

## Highlights * Initial version of QMMs for CUDA (#3160) * JACCL mesh bandwidth improvements (#3174) * Massive speedups for 3D cones (#3147) * Continued improvements to qqmm (#3106, #3022) ## What's Changed * Patch bump by @angeloskath in https://github.com/ml-explore/mlx/pull/3102 * is_available() should check the device index too by @andresy in https://github.com/ml-explore/mlx/pull/3107 * Fix residency set with user provided buffer by @awni in https://github.com/ml-explore/mlx/pull/3108 * Cleanup test_fast_sdpa.py by @zcbenz in https://github.com/ml-explore/mlx/pull/3112 * [CUDA] Set current device before allocating memory by @zcbenz in https://github.com/ml-explore/mlx/pull/3110 * Quantize module to QQLinear by @nastya236 in https://github.com/ml-explore/mlx/pull/3106 * [CUDA] Use cuDNN SDPA for decoding when using fixed-size KV cache by @zcbenz in https://github.com/ml-explore/mlx/pull/3113 * register pressure by @nastya236 in https://github.com/ml-explore/mlx/pull/3116 * Fix precision in Metal fused attention by @awni in https://github.com/ml-explore/mlx/pull/3119 * [CUDA] Attention sinks in cuDNN SDPA by @zcbenz in https://github.com/ml-explore/mlx/pull/3118 * Fix donation in sdpa vector by @angeloskath in https://github.com/ml-explore/mlx/pull/3121 * Manage stream placement in import function by @awni in https://github.com/ml-explore/mlx/pull/3127 * fix: propagate quantization mode in QuantizedAllToShardedLinear / QuantizedShardedToAllLinear by @vskiwi in https://github.com/ml-explore/mlx/pull/3133 * [featuring] - add hanning window function by @Vlor999 in https://github.com/ml-explore/mlx/pull/3124 * feat: adding the hamming function by @Vlor999 in https://github.com/ml-explore/mlx/pull/3135 * Tensor scale nvfp4 by @nastya236 in https://github.com/ml-explore/mlx/pull/3022 * Fix fence synchronization accross command buffers by @awni in https://github.com/ml-explore/mlx/pull/3144 * Export: preserve Dtype state values in export callback arguments by @skryl in https://github.com/ml-explore/mlx/pull/3145 * [Metal] Fix 32-bit integer overflow in conv3d unfold kernel by @kellen-sun in https://github.com/ml-explore/mlx/pull/3143 * [Metal][Performance] Add implicit matmul pathway for mx.conv3d by @belkakari in https://github.com/ml-explore/mlx/pull/3147 * [Metal] Fix event leak by @awni in https://github.com/ml-explore/mlx/pull/3159 * [CUDA] FPxINT quantized matmul for Hopper by @zcbenz in https://github.com/ml-explore/mlx/pull/3160 * feat: implement mlx.core.blackman by @Vlor999 in https://github.com/ml-explore/mlx/pull/3136 * Enable setting thread block cluster for Hopper and later by @zcbenz in https://github.com/ml-explore/mlx/pull/3168 * [CUDA][NCCL] group split by @nastya236 in https://github.com/ml-explore/mlx/pull/3172 * JACCL refactor and small update by @angeloskath in https://github.com/ml-explore/mlx/pull/3174 * [CUDA] Heuristics for Hopper QMM by @zcbenz in https://github.com/ml-explore/mlx/pull/3173 * Fix compile_fuse broadcast split aliasing bug by @robert-johansson in https://github.com/ml-explore/mlx/pull/3166 * Enable passing in a GPU architecture string via env var by @angeloskath in https://github.com/ml-explore/mlx/pull/3176 * Bump the minor version by @angeloskath in https://github.com/ml-explore/mlx/pull/3183 ## New Contributors * @vskiwi made their first contribution in https://github.com/ml-explore/mlx/pull/3133 * @Vlor999 made their first contribution in https://github.com/ml-explore/mlx/pull/3124 * @skryl made their first contribution in https://github.com/ml-explore/mlx/pull/3145 * @kellen-sun made their first contribution in https://github.com/ml-explore/mlx/pull/3143 * @belkakari made their first contribution in https://github.com/ml-explore/mlx/pull/3147 * @robert-johansson made their first contribution in https://github.com/ml-explore/mlx/pull/3166 **Full Changelog**: https://github.com/ml-explore/mlx/compare/v0.30.6...v0.31.0

## Highlights * Much faster bandwidth with JACCL on macOS >= 26.3 ([some numbers](https://github.com/ml-explore/mlx/pull/3094)) ## What's Changed * patch by @awni in https://github.com/ml-explore/mlx/pull/3093 * Disable managed memory on WSL when concurrentManagedAccess is not supported by @jessegross in https://github.com/ml-explore/mlx/pull/3095 * Fix non simd f16 build by @awni in https://github.com/ml-explore/mlx/pull/3097 * Fix 2pass sdpa on < M2 by @awni in https://github.com/ml-explore/mlx/pull/3099 * JACCL update by @angeloskath in https://github.com/ml-explore/mlx/pull/3094 * Fix qmv_impl for small N by @manuelcandales in https://github.com/ml-explore/mlx/pull/3096 * Patch for multi device CUDA by @awni in https://github.com/ml-explore/mlx/pull/3100 ## New Contributors * @manuelcandales made their first contribution in https://github.com/ml-explore/mlx/pull/3096 **Full Changelog**: https://github.com/ml-explore/mlx/compare/v0.30.5...v0.30.6

## What's Changed * patch by @awni in https://github.com/ml-explore/mlx/pull/3074 * [CUDA] Fallback Event impl when there is no hardware cpu/gpu coherency by @zcbenz in https://github.com/ml-explore/mlx/pull/3070 * Tune CUDA gaph sizes on B200 and H100 by @awni in https://github.com/ml-explore/mlx/pull/3077 * [Docs] Simple example of using MLX distributed by @stefpi in https://github.com/ml-explore/mlx/pull/2973 * Use lower-right causal mask alignment consistently by @Anri-Lombard in https://github.com/ml-explore/mlx/pull/2967 * Fix ALiBi slopes for non-power-of-2 num_heads by @vovw in https://github.com/ml-explore/mlx/pull/3071 * More useful error for large indices by @awni in https://github.com/ml-explore/mlx/pull/3079 * Fix nax condition for iphone by @awni in https://github.com/ml-explore/mlx/pull/3083 * Fallback to pinned host memory when managed memory is not supported by @zcbenz in https://github.com/ml-explore/mlx/pull/3075 * Fix failing python tests on Windows by @zcbenz in https://github.com/ml-explore/mlx/pull/3076 * [Metal] Tune splitk gemm dispatch conditions and partition sizes by @awni in https://github.com/ml-explore/mlx/pull/3087 * Fix for NAX overflow. by @awni in https://github.com/ml-explore/mlx/pull/3092 ## New Contributors * @stefpi made their first contribution in https://github.com/ml-explore/mlx/pull/2973 * @vovw made their first contribution in https://github.com/ml-explore/mlx/pull/3071 **Full Changelog**: https://github.com/ml-explore/mlx/compare/v0.30.4...v0.30.5

## Highlights - Support nvfp4 and mxfp8 quantized ops on Metal - Support nvfp4 and mxfp8 quantized-quantized matrix-matrix multiplication on CUDA ## What's Changed * Bump the patch version by @angeloskath in https://github.com/ml-explore/mlx/pull/2922 * Faster copy for col contig to row contig by @awni in https://github.com/ml-explore/mlx/pull/2917 * Fix cuda release by @awni in https://github.com/ml-explore/mlx/pull/2925 * Metal logging by @CC-Yeh in https://github.com/ml-explore/mlx/pull/2904 * fix cuda release part 2 by @awni in https://github.com/ml-explore/mlx/pull/2926 * new[CI]: add linux sanitizer tests by @incertum in https://github.com/ml-explore/mlx/pull/2860 * patch bump by @awni in https://github.com/ml-explore/mlx/pull/2927 * Fix CUDA pypi release by @awni in https://github.com/ml-explore/mlx/pull/2929 * Move allocate_workspace to cuda/utils.h by @zcbenz in https://github.com/ml-explore/mlx/pull/2923 * Allow dry run for PyPI release workflow by @zcbenz in https://github.com/ml-explore/mlx/pull/2928 * Set rpath with cmake for CUDA build by @zcbenz in https://github.com/ml-explore/mlx/pull/2932 * Fix nightly build by @zcbenz in https://github.com/ml-explore/mlx/pull/2933 * Set install rpath of python bindings with cmake by @zcbenz in https://github.com/ml-explore/mlx/pull/2934 * Fix pid in local launch by @angeloskath in https://github.com/ml-explore/mlx/pull/2936 * Make CUDA CI run faster by @zcbenz in https://github.com/ml-explore/mlx/pull/2939 * refactor: use perf_counter for accurate benchmarking by @Satyam12singh in https://github.com/ml-explore/mlx/pull/2940 * Fix for non row-contig scales by @awni in https://github.com/ml-explore/mlx/pull/2941 * Fix stubgen by @zcbenz in https://github.com/ml-explore/mlx/pull/2942 * ci: add macOS 26 target by @madrob in https://github.com/ml-explore/mlx/pull/2937 * Fix float64 size in data_types.rst by @pdevine in https://github.com/ml-explore/mlx/pull/2948 * Fixes in mlx.distributed_config by @angeloskath in https://github.com/ml-explore/mlx/pull/2947 * Metal/CPU nvfp4 and mxfp8 by @awni in https://github.com/ml-explore/mlx/pull/2946 * [CUDA] Implement gather_mm_rhs by @zcbenz in https://github.com/ml-explore/mlx/pull/2902 * Fetch nanobind with cmake by @zcbenz in https://github.com/ml-explore/mlx/pull/2949 * refactor: use time.perf_counter for consistent and accurate benchmarking by @Satyam12singh in https://github.com/ml-explore/mlx/pull/2943 * BUG FIX - Addition of missing parameter in random::uniform by @hwiesmann in https://github.com/ml-explore/mlx/pull/2963 * Fix doc issues in `mlx.nn.init.he_normal` and `mlx.nn.hard_tanh` by @Redempt1onzzZZ in https://github.com/ml-explore/mlx/pull/2968 * fix numpy dtype bug by @awni in https://github.com/ml-explore/mlx/pull/2960 * QQ linear by @nastya236 in https://github.com/ml-explore/mlx/pull/2931 * fix array allocator with user buffer and deleter by @andresy in https://github.com/ml-explore/mlx/pull/2971 * Swizzle scales by @nastya236 in https://github.com/ml-explore/mlx/pull/2979 * Fix `grid_dim_x` calculations by @CC-Yeh in https://github.com/ml-explore/mlx/pull/2980 * Add asarray to __array_namespace__ by @Anri-Lombard in https://github.com/ml-explore/mlx/pull/2966 * fix doc by @CC-Yeh in https://github.com/ml-explore/mlx/pull/2988 * replace MLX_IBV_COORDINATOR with MLX_JACCL_COORDINATOR by @Evanev7 in https://github.com/ml-explore/mlx/pull/2986 * Fix RandomBits::is_equivalent to include width by @MillaFleurs in https://github.com/ml-explore/mlx/pull/2978 * Don't try to use NAX at run-time if kernels aren't there by @awni in https://github.com/ml-explore/mlx/pull/2982 * Expose to/from fp8 in Python and don't auto-convert fp8 when loading from safetensors by @awni in https://github.com/ml-explore/mlx/pull/2985 * Allow some non 2D inputs in qqmm by @awni in https://github.com/ml-explore/mlx/pull/2981 ## New Contributors * @pdevine made their first contribution in https://github.com/ml-explore/mlx/pull/2948 * @hwiesmann made their first contribution in https://github.com/ml-explore/mlx/pull/2963 * @Anri-Lombard made their first contribution in https://github.com/ml-explore/mlx/pull/2966 * @Evanev7 made their first contribution in https://github.com/ml-explore/mlx/pull/2986 * @MillaFleurs made their first contribution in https://github.com/ml-explore/mlx/pull/2978 **Full Changelog**: https://github.com/ml-explore/mlx/compare/v0.30.1...v0.30.3

⏭️

⬆️

🚀