[CI Failure]: AMD Entrypoints Integration (LLM) hangs in test_guidance_no_additional_properties on mi300
rocm
### Name of failing test
`entrypoints/llm/test_struct_output_generate.py::test_guidance_no_additional_properties`
### Basic information
- [x] Flaky test
- [ ] Can reproduce locally
- [ ] Caused by external libraries (e.g. bug in `transformers`)
### 🧪 Describe the failing test
Job `AMD: Entrypoints Integration Test (LLM) (mi300_1)` hangs rather than failing. The last test to report progress is `test_guidance_no_additional_properties`, then nothing for 25 minutes until pytest's faulthandler fires and Buildkite cancels:
```
Timeout (0:25:00)!
Thread 0x00007bdcadaed640 (most recent call first):
File "/usr/lib/python3.12/threading.py", line 359 in wait
...
# Received cancellation signal, interrupting
🚨 Error: The command was interrupted by a signal: signal: terminated
```
The job ran ~55 minutes, the last 25 of which were the hang. No test is reported as FAILED, so the red state comes from the cancellation.
Possibly related, earlier in the same job several structured-output requests were terminated by the scheduler:
```
ERROR [scheduler.py:1835] Unexpected: grammar rejected tokens [2] for request 6-bf3c244b. Terminating request.
ERROR [scheduler.py:1835] Unexpected: grammar rejected tokens [151645] for request 0-8cea7b0e. Terminating request.
```
Those appear before the hang and may be unrelated, but they are in the same structured-output area.
Seen on https://buildkite.com/vllm/ci/builds/82685, on a PR touching only the KV offloading scheduler (`OffloadingConnector` appears 0 times in the job log).
### 📝 History of failing test
Noticed on build 82685. Not bisected, and I do not have MI300 hardware to reproduce locally.
### CC List.
_No response_
关闭于 11 天前 3 条评论