[Bug] Compatibility issues with PP and Hicache causing errors

### Checklist - [x] I searched related issues but found no solution. - [x] The bug persists in the latest version. - [x] Issues without environment info and a minimal reproducible demo are hard to resolve and may receive no feedback. - [x] If this is not a bug report but a general question, please start a discussion at https://github.com/sgl-project/sglang/discussions. Otherwise, it will be closed. - [x] Please use English. Otherwise, it will be closed. ### Describe the bug It seems that the error is unrelated to traffic cause it will happen under no requests. And raising hicache-size will cause error happend more frequently. Error 1 : [2026-07-02 02:00:57] [/bin/sh]: [rank6]:[W701 18:00:57.351060385 ProcessGroupNCCL.cpp:1826] [PG ID 0 PG GUID 0 Rank 6] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe [2026-07-02 02:00:57] [/bin/sh]: [2026-07-01 18:00:57 PP0 TP0] Decode batch, #running-req: 1, #token: 178880, token usage: 0.18, cuda graph: True, gen throughput (token/s): 85.93, #queue-req: 0 [2026-07-02 02:00:57] [/bin/sh]: [2026-07-01 18:00:57 PP1 TP0] Decode batch, #running-req: 1, #token: 178880, token usage: 0.18, cuda graph: True, gen throughput (token/s): 85.97, #queue-req: 0 [2026-07-02 02:00:57] [/bin/sh]: [2026-07-01 18:00:57 PP0 TP0] Decode batch, #running-req: 1, #token: 178816, token usage: 0.18, cuda graph: True, gen throughput (token/s): 86.37, #queue-req: 0 [2026-07-02 02:00:57] [/bin/sh]: [2026-07-01 18:00:57 PP1 TP0] Decode batch, #running-req: 1, #token: 178816, token usage: 0.18, cuda graph: True, gen throughput (token/s): 86.33, #queue-req: 0 [2026-07-02 02:00:58] [/bin/sh]: glm-52-fp8-mc-clone-789548749c-lvlzt:1038:1038 [2] NCCL INFO ENV/Plugin: Closing env plugin ncclEnvDefault [2026-07-02 02:00:58] [/bin/sh]: [2026-07-01 18:00:58 PP1 TP2] Scheduler hit an exception: Traceback (most recent call last): [2026-07-02 02:00:58] [/bin/sh]: File "/sgl-workspace/sglang/python/sglang/srt/managers/scheduler.py", line 4227, in run_scheduler_process [2026-07-02 02:00:58] [/bin/sh]: scheduler.run_event_loop() [2026-07-02 02:00:58] [/bin/sh]: File "/sgl-workspace/sglang/python/sglang/srt/managers/scheduler.py", line 1486, in run_event_loop [2026-07-02 02:00:58] [/bin/sh]: dispatch_event_loop(self) [2026-07-02 02:00:58] [/bin/sh]: File "/sgl-workspace/sglang/python/sglang/srt/managers/scheduler.py", line 4088, in dispatch_event_loop [2026-07-02 02:00:58] [/bin/sh]: scheduler.event_loop_pp() [2026-07-02 02:00:58] [/bin/sh]: File "/usr/local/lib/python3.12/dist-packages/torch/utils/_contextlib.py", line 124, in decorate_context [2026-07-02 02:00:58] [/bin/sh]: return func(*args, **kwargs) [2026-07-02 02:00:58] [/bin/sh]: ^^^^^^^^^^^^^^^^^^^^^ [2026-07-02 02:00:58] [/bin/sh]: File "/sgl-workspace/sglang/python/sglang/srt/managers/scheduler_pp_mixin.py", line 110, in event_loop_pp [2026-07-02 02:00:58] [/bin/sh]: self.mbs[mb_id] = self.get_next_batch_to_run() [2026-07-02 02:00:58] [/bin/sh]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [2026-07-02 02:00:58] [/bin/sh]: File "/sgl-workspace/sglang/python/sglang/srt/utils/nvtx_utils.py", line 109, in wrapper [2026-07-02 02:00:58] [/bin/sh]: return func(*args, **kwargs) [2026-07-02 02:00:58] [/bin/sh]: ^^^^^^^^^^^^^^^^^^^^^ [2026-07-02 02:00:58] [/bin/sh]: File "/sgl-workspace/sglang/python/sglang/srt/managers/scheduler.py", line 2640, in get_next_batch_to_run [2026-07-02 02:00:58] [/bin/sh]: new_batch = self.get_new_batch_prefill() [2026-07-02 02:00:58] [/bin/sh]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [2026-07-02 02:00:58] [/bin/sh]: File "/sgl-workspace/sglang/python/sglang/srt/managers/scheduler.py", line 2713, in get_new_batch_prefill [2026-07-02 02:00:58] [/bin/sh]: ret = self._get_new_batch_prefill_raw( [2026-07-02 02:00:58] [/bin/sh]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [2026-07-02 02:00:58] [/bin/sh]: File "/sgl-workspace/sglang/python/sglang/srt/managers/scheduler.py", line 2732, in _get_new_batch_prefill_raw [2026-07-02 02:00:58] [/bin/sh]: self.tree_cache.check_hicache_events() [2026-07-02 02:00:58] [/bin/sh]: File "/sgl-workspace/sglang/python/sglang/srt/mem_cache/hiradix_cache.py", line 1282, in check_hicache_events [2026-07-02 02:00:58] [/bin/sh]: self.writing_check() [2026-07-02 02:00:58] [/bin/sh]: File "/sgl-workspace/sglang/python/sglang/srt/mem_cache/hiradix_cache.py", line 930, in writing_check [2026-07-02 02:00:58] [/bin/sh]: self._all_reduce(finish_count_tensor, torch.distributed.ReduceOp.MIN) [2026-07-02 02:00:58] [/bin/sh]: File "/sgl-workspace/sglang/python/sglang/srt/mem_cache/hiradix_cache.py", line 234, in _all_reduce [2026-07-02 02:00:58] [/bin/sh]: self._pp_sync(data) [2026-07-02 02:00:58] [/bin/sh]: File "/sgl-workspace/sglang/python/sglang/srt/mem_cache/hiradix_cache.py", line 263, in _pp_sync [2026-07-02 02:00:58] [/bin/sh]: torch.distributed.recv( [2026-07-02 02:00:58] [/bin/sh]: File "/usr/local/lib/python3.12/dist-packages/torch/distributed/c10d_logger.py", line 83, in wrapper [2026-07-02 02:00:58] [/bin/sh]: return func(*args, **kwargs) [2026-07-02 02:00:58] [/bin/sh]: ^^^^^^^^^^^^^^^^^^^^^ [2026-07-02 02:00:58] [/bin/sh]: File "/usr/local/lib/python3.12/dist-packages/torch/distributed/distributed_c10d.py", line 2662, in recv [2026-07-02 02:00:58] [/bin/sh]: work.wait() [2026-07-02 02:00:58] [/bin/sh]: RuntimeError: [/pytorch/third_party/gloo/gloo/transport/tcp/pair.cc:547] Connection closed by peer [11.224.19.34]:7303 Error 2 : [2026-06-29 17:44:04] [/bin/sh]: [2026-06-29 09:44:04 PP1 TP3] Scheduler hit an exception: Traceback (most recent call last): [2026-06-29 17:44:04] [/bin/sh]: File "/sgl-workspace/sglang/python/sglang/srt/managers/scheduler.py", line 4227, in run_scheduler_process [2026-06-29 17:44:04] [/bin/sh]: scheduler.run_event_loop() [2026-06-29 17:44:04] [/bin/sh]: File "/sgl-workspace/sglang/python/sglang/srt/managers/scheduler.py", line 1486, in run_event_loop [2026-06-29 17:44:04] [/bin/sh]: dispatch_event_loop(self) [2026-06-29 17:44:04] [/bin/sh]: File "/sgl-workspace/sglang/python/sglang/srt/managers/scheduler.py", line 4088, in dispatch_event_loop [2026-06-29 17:44:04] [/bin/sh]: scheduler.event_loop_pp() [2026-06-29 17:44:04] [/bin/sh]: File "/usr/local/lib/python3.12/dist-packages/torch/utils/_contextlib.py", line 124, in decorate_context [2026-06-29 17:44:04] [/bin/sh]: return func(*args, **kwargs) [2026-06-29 17:44:04] [/bin/sh]: ^^^^^^^^^^^^^^^^^^^^^ [2026-06-29 17:44:04] [/bin/sh]: File "/sgl-workspace/sglang/python/sglang/srt/managers/scheduler_pp_mixin.py", line 110, in event_loop_pp [2026-06-29 17:44:04] [/bin/sh]: self.mbs[mb_id] = self.get_next_batch_to_run() [2026-06-29 17:44:04] [/bin/sh]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [2026-06-29 17:44:04] [/bin/sh]: File "/sgl-workspace/sglang/python/sglang/srt/utils/nvtx_utils.py", line 109, in wrapper [2026-06-29 17:44:04] [/bin/sh]: return func(*args, **kwargs) [2026-06-29 17:44:04] [/bin/sh]: ^^^^^^^^^^^^^^^^^^^^^ [2026-06-29 17:44:04] [/bin/sh]: File "/sgl-workspace/sglang/python/sglang/srt/managers/scheduler.py", line 2640, in get_next_batch_to_run [2026-06-29 17:44:04] [/bin/sh]: new_batch = self.get_new_batch_prefill() [2026-06-29 17:44:04] [/bin/sh]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [2026-06-29 17:44:04] [/bin/sh]: File "/sgl-workspace/sglang/python/sglang/srt/managers/scheduler.py", line 2713, in get_new_batch_prefill [2026-06-29 17:44:04] [/bin/sh]: ret = self._get_new_batch_prefill_raw( [2026-06-29 17:44:04] [/bin/sh]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [2026-06-29 17:44:04] [/bin/sh]: File "/sgl-workspace/sglang/python/sglang/srt/managers/scheduler.py", line 2732, in _get_new_batch_prefill_raw [2026-06-29 17:44:04] [/bin/sh]: self.tree_cache.check_hicache_events() [2026-06-29 17:44:04] [/bin/sh]: File "/sgl-workspace/sglang/python/sglang/srt/mem_cache/hiradix_cache.py", line 1283, in check_hicache_events [2026-06-29 17:44:04] [/bin/sh]: self.loading_check() [2026-06-29 17:44:04] [/bin/sh]: File "/sgl-workspace/sglang/python/sglang/srt/mem_cache/hiradix_cache.py", line 956, in loading_check [2026-06-29 17:44:04] [/bin/sh]: _, finish_event, ack_list = self.cache_controller.ack_load_queue.pop(0) [2026-06-29 17:44:04] [/bin/sh]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [2026-06-29 17:44:04] [/bin/sh]: IndexError: pop from empty list Error 3: Health check failed. Server couldn't get a response from detokenizer for last 20 seconds. tic start time: 01:46:26. last_heartbeat time: 01:45:04 ### Reproduction python3 -m sglang.launch_server --model-path /model_dir --host 0.0.0.0 --port 8000 --tp-size 4 --pp-size 2 --kv-cache-dtype fp8_e4m3 --mem-fraction-static 0.88 --max-running-requests 16 --enable-metrics --trust-remote-code --tool-call-parser glm47 --reasoning-parser glm45 --served-model-name GLM-5.2-FP8 --enable-hierarchical-cache --hicache-size 80 ### Environment single node of H200 with 8 GPU sglang:v0.5.14