GridNeighborhoodSearch update on GPU intermittent bounds error

I found some odd behavior when updating the neighborhood search on the GPU, wondering if anyone has seen this before. Sometimes when I call `update!()` on the grid on the GPU and I've moved the neighboring particles far away from their previous positions (but still within the grid bounds), I get a bounds error. Weirdly, this doesn't happen all the time. Below is a MWE and the error I see. When I run the code below (in VSCode), sometimes the error shows up, sometimes it does not. Weirdly this happens on my computer with an RTX4090. My other computer with a GTX1070 always runs the code fine. Any ideas why I might be seeing this behavior? Am I missing something? I tried re-installing packages on both computers. Any ideas would be greatly appreciated! Happy to provide more details if needed. ```julia using PointNeighbors using CUDA using StaticArrays using Adapt # Domain parameters dp = 0.001 min_corner = SA[0.0, 0.0] max_corner = SA[1.0, 10.0] # Grid parameters xs = 0:dp:1 ys = 0:dp:0.3 nx = length(xs); ny = length(ys) # Create a first grid of points at the bottom of the domain points_cpu = zeros(Float32, 2, nx*ny) for i in eachindex(xs) for j in eachindex(ys) points_cpu[1, (i-1)*ny + j] = xs[i] points_cpu[2, (i-1)*ny + j] = ys[j] end end # Create a second grid of points, but offset upwards by y_offset # y_offset = 0.5 # Offsetting by a smaller amount always works y_offset = 6.0 # Offsetting by a larger amount can leads to some errors on the GPU points_cpu_top = copy(points_cpu) points_cpu_top[2,:] .= points_cpu_top[2,:] .+ y_offset # Copy the grids on the GPU points_gpu = cu(points_cpu) points_gpu_top = cu(points_cpu_top) # Create neighbor search search_radius = dp*3 cell_list = FullGridCellList(min_corner=min_corner.-2*search_radius, max_corner=max_corner.+2*search_radius, search_radius=search_radius, backend=PointNeighbors.DynamicVectorOfVectors{Int32}, max_points_per_cell = 200) nhs = GridNeighborhoodSearch{2}(search_radius=dp*3, n_points=length(points_cpu), cell_list=cell_list, update_strategy=nothing) PointNeighbors.initialize!(nhs, points_cpu, points_cpu) # Copy neighbor search onto GPU nhs_gpu = Adapt.adapt(CuArray, nhs); # Update using the second grid as the neighboring points list PointNeighbors.update!(nhs, points_cpu, points_cpu_top) # The CPU always works PointNeighbors.update!(nhs_gpu, points_gpu, points_gpu_top) # But the GPU sometimes errors println("success!") ``` ```julia ERROR: a BoundsError was thrown during kernel execution on thread (385, 1, 1) in block (1061, 1, 1). Out-of-bounds array access Stacktrace not available, run Julia on debug level 2 for more details (by passing -g2 to the executable). ERROR: KernelException: exception thrown during kernel execution on device NVIDIA GeForce RTX 4090 Stacktrace: [1] check_exceptions() @ CUDA C:\Users\user\.julia\packages\CUDA\1kIOw\src\compiler\exceptions.jl:39 [2] synchronize(stream::CuStream; blocking::Bool, spin::Bool) @ CUDA C:\Users\user\.julia\packages\CUDA\1kIOw\lib\cudadrv\synchronization.jl:207 [3] synchronize (repeats 2 times) @ C:\Users\user\.julia\packages\CUDA\1kIOw\lib\cudadrv\synchronization.jl:194 [inlined] [4] synchronize @ C:\Users\user\.julia\packages\CUDA\1kIOw\src\CUDAKernels.jl:29 [inlined] [5] parallel_foreach @ C:\Users\user\.julia\packages\PointNeighbors\VUQSm\src\util.jl:151 [inlined] [6] macro expansion @ C:\Users\user\.julia\packages\PointNeighbors\VUQSm\src\util.jl:103 [inlined] [4] synchronize @ C:\Users\user\.julia\packages\CUDA\1kIOw\src\CUDAKernels.jl:29 [inlined] [5] parallel_foreach @ C:\Users\user\.julia\packages\PointNeighbors\VUQSm\src\util.jl:151 [inlined] [6] macro expansion @ C:\Users\user\.julia\packages\PointNeighbors\VUQSm\src\util.jl:103 [inlined] [7] update_grid!(neighborhood_search::GridNeighborhoodSearch{…}, coords_fun::PointNeighbors.var"#44#45"{…}; parallelization_backend::CuArray{…}) @ PointNeighbors C:\Users\user\.julia\packages\PointNeighbors\VUQSm\src\nhs_grid.jl:312 [8] update_grid! @ C:\Users\user\.julia\packages\PointNeighbors\VUQSm\src\nhs_grid.jl:303 [inlined] [9] #update_grid!#43 @ C:\Users\user\.julia\packages\PointNeighbors\VUQSm\src\nhs_grid.jl:206 [inlined] [10] update_grid! @ C:\Users\user\.julia\packages\PointNeighbors\VUQSm\src\nhs_grid.jl:204 [inlined] [11] #update!#42 @ C:\Users\user\.julia\packages\PointNeighbors\VUQSm\src\nhs_grid.jl:200 [inlined] [12] update!(neighborhood_search::GridNeighborhoodSearch{…}, x::CuArray{…}, y::CuArray{…}) @ PointNeighbors C:\Users\user\.julia\packages\PointNeighbors\VUQSm\src\nhs_grid.jl:193 Some type information was truncated. Use `show(err)` to see complete types. ```