Bug in filling distributed halos

In addition to distributed tests failing with a `-1` (agent lost) error, there seems to be an actual bug there too. Some of those are failing with: ``` Distributed Transpose: Error During Test at /resnick/scratch/esm/slurm-buildkite/oceananigans-distributed/14626/oceananigans-distributed/test/test_distributed_transpose.jl:52 Test threw exception Expression: test_transpose((16, 44, 8), (2, 2, 1), topology, child_arch) MethodError: no method matching (::DistributedFillHalo{North})(::OffsetArray{ComplexF64, 3, Array{ComplexF64, 3}}, ::DistributedCommunicationBoundaryCondition{Oceananigans.BoundaryConditions.DistributedCommunication, Oceananigans.DistributedComputations.HaloCommunicationRanks{Int64, Int64}}, ::Tuple{Center, Center, Center}, ::RectilinearGrid{Float64, RightConnected, RightConnected, Bounded, Oceananigans.Grids.StaticVerticalDiscretization{OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, Float64, Float64}, Float64, Float64, OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, CPU}, ::Tuple{}) The object of type `DistributedFillHalo{North}` exists, but no method is defined for this combination of argument types when trying to treat it as a callable object. Closest candidates are: (::DistributedFillHalo{<:North})(::Any, ::Any, ::Any, ::Any, ::Any, ::Any) @ Oceananigans /resnick/scratch/esm/slurm-buildkite/oceananigans-distributed/14626/oceananigans-distributed/src/DistributedComputations/halo_communication.jl:274 Stacktrace: [1] fill_halo_event!(::OffsetArray{ComplexF64, 3, Array{ComplexF64, 3}}, ::DistributedFillHalo{North}, ::Tuple{DistributedCommunicationBoundaryCondition{Oceananigans.BoundaryConditions.DistributedCommunication, Oceananigans.DistributedComputations.HaloCommunicationRanks{Int64, Int64}}}, ::Tuple{Center, Center, Center}, ::RectilinearGrid{Float64, RightConnected, RightConnected, Bounded, Oceananigans.Grids.StaticVerticalDiscretization{OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, Float64, Float64}, Float64, Float64, OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, CPU}) @ Oceananigans.BoundaryConditions /resnick/scratch/esm/slurm-buildkite/oceananigans-distributed/14626/oceananigans-distributed/src/BoundaryConditions/fill_halo_regions.jl:37 [2] fill_halo_regions!(::OffsetArray{ComplexF64, 3, Array{ComplexF64, 3}}, ::FieldBoundaryConditions{NoFluxBoundaryCondition, DistributedCommunicationBoundaryCondition{Oceananigans.BoundaryConditions.DistributedCommunication, Oceananigans.DistributedComputations.HaloCommunicationRanks{Int64, Int64}}, NoFluxBoundaryCondition, DistributedCommunicationBoundaryCondition{Oceananigans.BoundaryConditions.DistributedCommunication, Oceananigans.DistributedComputations.HaloCommunicationRanks{Int64, Int64}}, NoFluxBoundaryCondition, NoFluxBoundaryCondition, Nothing, @NamedTuple{bottom_and_top::KernelAbstractions.Kernel{KernelAbstractions.CPU, KernelAbstractions.NDIteration.StaticSize{(16, 16)}, Oceananigans.Utils.OffsetStaticSize{(1:8, 1:22)}, typeof(Oceananigans.BoundaryConditions.cpu__fill_bottom_and_top_halo!)}, south::KernelAbstractions.Kernel{KernelAbstractions.CPU, KernelAbstractions.NDIteration.StaticSize{(16, 16)}, Oceananigans.Utils.OffsetStaticSize{(1:8, 1:8)}, typeof(Oceananigans.BoundaryConditions.cpu__fill_only_south_halo!)}, west::KernelAbstractions.Kernel{KernelAbstractions.CPU, KernelAbstractions.NDIteration.StaticSize{(16, 16)}, Oceananigans.Utils.OffsetStaticSize{(1:22, 1:8)}, typeof(Oceananigans.BoundaryConditions.cpu__fill_only_west_halo!)}, north::DistributedFillHalo{North}, east::DistributedFillHalo{East}}, @NamedTuple{bottom_and_top::Tuple{NoFluxBoundaryCondition, NoFluxBoundaryCondition}, south::Tuple{NoFluxBoundaryCondition}, west::Tuple{NoFluxBoundaryCondition}, north::Tuple{DistributedCommunicationBoundaryCondition{Oceananigans.BoundaryConditions.DistributedCommunication, Oceananigans.DistributedComputations.HaloCommunicationRanks{Int64, Int64}}}, east::Tuple{DistributedCommunicationBoundaryCondition{Oceananigans.BoundaryConditions.DistributedCommunication, Oceananigans.DistributedComputations.HaloCommunicationRanks{Int64, Int64}}}}}, ::Tuple{Colon, Colon, Colon}, ::Tuple{Center, Center, Center}, ::RectilinearGrid{Float64, RightConnected, RightConnected, Bounded, Oceananigans.Grids.StaticVerticalDiscretization{OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, Float64, Float64}, Float64, Float64, OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, CPU}; kwargs::@Kwargs{}) ``` I haven't seen any issue of PR open for that so I figured I'd report it in case no one had noticed.