4 tests fail (P2P related, 2 x Blackwell GPUs)

**Issue** 4 tests fail ``` Test Summary: Ran 151 test runs for 147 executables. Failed runs (4): simpleP2P : Failed (code 1) conjugateGradientMultiDeviceCG : Timeout (code -1) simpleCUFFT_MGPU : Failed (code 1) streamOrderedAllocationP2P : Failed (code 1) ``` **Environment** ``` OS: Ubuntu 24.04.3 LTS, kernel 6.14.0-33-generic CUDA Toolkit: CUDA 13.0 Update 2 GPU 0: NVIDIA RTX PRO 6000 Blackwell Workstation Edition GPU 1: NVIDIA RTX PRO 6000 Blackwell Workstation Edition Driver: 580.95.05 CUDA Version (nvidia-smi): 13.0 CMake: 3.28.3 GCC: 13.3.0 ``` **Notes** Output from simpleP2P: ``` [./simpleP2P] - Starting... Checking for multiple GPUs... CUDA-capable device count: 2 Checking GPU(s) for support of peer to peer memory access... > Peer access from NVIDIA RTX PRO 6000 Blackwell Workstation Edition (GPU0) -> NVIDIA RTX PRO 6000 Blackwell Workstation Edition (GPU1) : Yes > Peer access from NVIDIA RTX PRO 6000 Blackwell Workstation Edition (GPU1) -> NVIDIA RTX PRO 6000 Blackwell Workstation Edition (GPU0) : Yes Enabling peer access between GPU0 and GPU1... Allocating buffers (64MB on GPU0, GPU1 and CPU Host)... Creating event handles... cudaMemcpyPeer / cudaMemcpy between GPU0 and GPU1: 103.88GB/s Preparing host buffer and memcpy to GPU0... Run kernel on GPU1, taking source data from GPU0 and writing to GPU1... Run kernel on GPU0, taking source data from GPU1 and writing to GPU0... Copy data back to host from GPU0 and verify results... Verification error @ element 0: val = -0.003189, ref = 0.000000 Verification error @ element 1: val = 0.000000, ref = 4.000000 Verification error @ element 2: val = 0.000000, ref = 8.000000 Verification error @ element 3: val = 0.000000, ref = 12.000000 Verification error @ element 4: val = 0.000000, ref = 16.000000 Verification error @ element 5: val = 0.000000, ref = 20.000000 Verification error @ element 6: val = 0.000000, ref = 24.000000 Verification error @ element 7: val = 0.000000, ref = 28.000000 Verification error @ element 8: val = 0.000000, ref = 32.000000 Verification error @ element 9: val = 0.000000, ref = 36.000000 Verification error @ element 10: val = 0.000000, ref = 40.000000 Verification error @ element 11: val = 0.000000, ref = 44.000000 Disabling peer access... Shutting down... Test failed! ``` Output from conjugateGradientMultiDeviceCG: **_Both GPUs run at 100% utilization and the process stalls indefinitely._** ``` Starting [conjugateGradientMultiDeviceCG]... GPU Device 0: "NVIDIA RTX PRO 6000 Blackwell Workstation Edition" with compute capability 12.0 GPU Device 1: "NVIDIA RTX PRO 6000 Blackwell Workstation Edition" with compute capability 12.0 Device=0 CAN Access Peer Device=1 Selected p2p capable devices - deviceId = 0 deviceId = 1 Running on GPUs = 2 Total threads per GPU = 192512 numBlocksPerSm = 2 Launching kernel ``` Output from simpleCUFFT_MGPU: ``` [simpleCUFFT_MGPU] is starting... GPU Device 0: "NVIDIA RTX PRO 6000 Blackwell Workstation Edition" with compute capability 12.0 GPU Device 1: "NVIDIA RTX PRO 6000 Blackwell Workstation Edition" with compute capability 12.0 Running on GPUs GPU Device 0: "NVIDIA RTX PRO 6000 Blackwell Workstation Edition" with compute capability 12.0 GPU Device 1: "NVIDIA RTX PRO 6000 Blackwell Workstation Edition" with compute capability 12.0 Value of Library Descriptor Number of GPUs 2 Device id 0 1 Data size on GPU 512 512 Launching ComplexPointwiseMulAndScale<<< >>> Transforming signal back cufftExecC2C value of TestResult 0 ``` Output from streamOrderedAllocationP2P: ``` Device=0 CAN Access Peer Device=1 Selected p2p capable devices - deviceId = 0 deviceId = 1 selected devices = 0 & 1 > copyP2PAndScale kernel running ... > Checking the results from copyP2PAndScale() ... mismatch i = 0 expected = 0 val = 1973548544 ``` Any advice is appreciated!