XDNA driver/XRT regression causing `ERT_CMD_STATE_TIMEOUT` in runlist execution
```
(iron-venv) llm@dagger ~/tools/IRON/applications/llama_3.2_1b $ xrt-smi validate
WARNING: User doesn't have admin permissions to set performance mode. Running validate in Default mode
Validate Device : [0000:c5:00.1]
Platform : NPU Strix
Power Mode : Default
-------------------------------------------------------------------------------
Test 1 [0000:c5:00.1] : gemm
Details : TOPS: 51.0
Test Status : [PASSED]
-------------------------------------------------------------------------------
Test 2 [0000:c5:00.1] : latency
Details : Average latency: 50.0 us
Test Status : [PASSED]
-------------------------------------------------------------------------------
Test 3 [0000:c5:00.1] : throughput
Details : Average throughput: 87380.0 op/s
Test Status : [PASSED]
-------------------------------------------------------------------------------
Validation completed. Please run the command '--verbose' option for more details
(iron-venv) llm@dagger ~/tools/IRON/applications/llama_3.2_1b $
```
```
/$$$$$$ /$$$$$$$ /$$$$$$ /$$ /$$
|_ $$_/| $$__ $$ /$$__ $$| $$$ | $$
| $$ | $$ \ $$| $$ \ $$| $$$$| $$
| $$ | $$$$$$$/| $$ | $$| $$ $$ $$
| $$ | $$__ $$| $$ | $$| $$ $$$$
| $$ | $$ \ $$| $$ | $$| $$\ $$$
/$$$$$$| $$ | $$| $$$$$$/| $$ \ $$
|______/|__/ |__/ \______/ |__/ \__/
/$$ /$$ /$$$$$$ /$$ /$$ /$$$$$$
| $$ | $$ /$$__ $$| $$$ /$$$ /$$__ $$
| $$ | $$ | $$ \ $$| $$$$ /$$$$| $$ \ $$
| $$ | $$ | $$$$$$$$| $$ $$/$$ $$| $$$$$$$$
| $$ | $$ | $$__ $$| $$ $$$| $$| $$__ $$
| $$ | $$ | $$ | $$| $$\ $ | $$| $$ | $$
| $$$$$$$$| $$$$$$$$| $$ | $$| $$ \/ | $$| $$ | $$
|________/|________/|__/ |__/|__/ |__/|__/ |__/
AIE Configuration (✔ = AIE NPU / ✘ = CPU):
[Model] KV Cache ✔
[Decode] GEMV ✔
[Attention] Rope ✔
[Attention] QKV GEMM ✔
[Attention] Fused MHA ✔
[FFN] Runlist-based SwiGLU ✔
[Transformer] Residual Addition ✔
[Transformer] Pre Norm ✔
[Transformer] Post Norm ✔
[Transformer] Final Norm ✔
[Transformer] Final GEMM ✘
Starting text generation...
Generating 1 tokens...
=======================================================
Starting prefill inference...
Traceback (most recent call last):
File "/home/llm/tools/IRON/applications/llama_3.2_1b/inference.py", line 438, in <module>
inference(
~~~~~~~~~^
args.weights_file_path,
^^^^^^^^^^^^^^^^^^^^^^^
...<6 lines>...
args.prompt_len,
^^^^^^^^^^^^^^^^
)
^
File "/home/llm/tools/IRON/applications/llama_3.2_1b/inference.py", line 319, in inference
token_ids = generate(
model=model,
...<9 lines>...
prefill_done_callback=set_prefill_time,
)
File "/home/llm/tools/IRON/applications/llama_3.2_1b/src/utils.py", line 232, in generate
logits = model(idx_cond, input_pos=input_pos, use_kv_cache=use_kv_cache)
File "/home/llm/tools/iron-venv/lib/python3.13/site-packages/torch/nn/modules/module.py", line 1775, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
File "/home/llm/tools/iron-venv/lib/python3.13/site-packages/torch/nn/modules/module.py", line 1786, in _call_impl
return forward_call(*args, **kwargs)
File "/home/llm/tools/IRON/applications/llama_3.2_1b/src/model_with_json.py", line 211, in forward
x = block(x, mask, self.angles, input_pos)
File "/home/llm/tools/iron-venv/lib/python3.13/site-packages/torch/nn/modules/module.py", line 1775, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
File "/home/llm/tools/iron-venv/lib/python3.13/site-packages/torch/nn/modules/module.py", line 1786, in _call_impl
return forward_call(*args, **kwargs)
File "/home/llm/tools/IRON/applications/llama_3.2_1b/src/block/transformer.py", line 163, in forward
x = self.ff(x)
File "/home/llm/tools/iron-venv/lib/python3.13/site-packages/torch/nn/modules/module.py", line 1775, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
File "/home/llm/tools/iron-venv/lib/python3.13/site-packages/torch/nn/modules/module.py", line 1786, in _call_impl
return forward_call(*args, **kwargs)
File "/home/llm/tools/IRON/applications/llama_3.2_1b/src/block/feed_forward.py", line 175, in forward
return self.aie_swiglu_prefill(x)
~~~~~~~~~~~~~~~~~~~~~~~^^^
File "/home/llm/tools/IRON/operators/common/aie_base.py", line 208, in __call__
return self.forward(*args, **kwargs)
~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
File "/home/llm/tools/IRON/operators/swiglu_prefill/op.py", line 200, in forward
out = self._execute_aie_operation(x_flat)
File "/home/llm/tools/IRON/operators/swiglu_prefill/op.py", line 216, in _execute_aie_operation
self.run_runlist()
~~~~~~~~~~~~~~~~^^
File "/home/llm/tools/IRON/operators/common/aie_base.py", line 332, in run_runlist
self.xrt_runlist.wait()
~~~~~~~~~~~~~~~~~~~~~^^
RuntimeError: runlist failed execution (ERT_CMD_STATE_TIMEOUT)
txn_op_idx = 0xFFFFFFFF
ctx_pc = 0x28B04659
fatal_error_type = 0x00000000
fatal_error_exception_type = 0x00000000
fatal_error_exception_pc = 0x00000000
fatal_error_app_module = 0x00000000
double free or corruption (out)
```
```
[18672.742934] amd_iommu_report_page_fault: 1029 callbacks suppressed
[18672.742940] amdxdna 0000:c5:00.1: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0001 address=0x7f5f245a0000 flags=0x0007]
[18672.742951] amdxdna 0000:c5:00.1: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0001 address=0x7f5f245a8000 flags=0x0007]
[18672.742958] amdxdna 0000:c5:00.1: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0001 address=0x7f5f245a1000 flags=0x0007]
[18672.742964] amdxdna 0000:c5:00.1: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0001 address=0x7f5f245a9000 flags=0x0007]
[18672.742970] amdxdna 0000:c5:00.1: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0001 address=0x7f5f245a2000 flags=0x0007]
[18672.742976] amdxdna 0000:c5:00.1: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0001 address=0x7f5f245a3000 flags=0x0007]
[18672.742982] amdxdna 0000:c5:00.1: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0001 address=0x7f5f245a4000 flags=0x0007]
[18672.742988] amdxdna 0000:c5:00.1: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0001 address=0x7f5f245a5000 flags=0x0007]
[18672.742994] amdxdna 0000:c5:00.1: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0001 address=0x7f5f24596000 flags=0x0007]
[18672.743000] amdxdna 0000:c5:00.1: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0001 address=0x7f5f245a6000 flags=0x0007]
[18672.745668] AMD-Vi: IOMMU Event log restarting
[18672.748430] AMD-Vi: IOMMU Event log restarting
[18680.784113] amdxdna 0000:c5:00.1: aie2_tdr_work: Device isn't making progress... Count 5
[18680.784126] amdxdna 0000:c5:00.1: aie2_dump_ctx: Dumping ctx ctx.472334.7, hwctx 10, sub=1, comp=0
[18680.784252] amdxdna 0000:c5:00.1: aie2_dump_ctx: Firmware timeout state capture:
[18680.784257] amdxdna 0000:c5:00.1: aie2_dump_ctx: Version: 2.0
[18680.784260] amdxdna 0000:c5:00.1: aie2_dump_ctx: Report size: 0x230
[18680.784262] amdxdna 0000:c5:00.1: aie2_dump_ctx: Context ID: 10
[18680.784264] amdxdna 0000:c5:00.1: aie2_dump_ctx: DPU PC: 0xffffffff
[18680.784265] amdxdna 0000:c5:00.1: aie2_dump_ctx: TXN OP ID: 0xffffffff
[18680.784267] amdxdna 0000:c5:00.1: aie2_dump_ctx: Context PC: 0x28b04659
[18680.784268] amdxdna 0000:c5:00.1: aie2_dump_ctx: Fatal error type: 0x0
[18680.784270] amdxdna 0000:c5:00.1: aie2_dump_ctx: Fatal error exception type: 0x0
[18680.784271] amdxdna 0000:c5:00.1: aie2_dump_ctx: Fatal error exception PC: 0x0
[18680.784272] amdxdna 0000:c5:00.1: aie2_dump_ctx: Fatal error app module: 0x0
[18680.784274] amdxdna 0000:c5:00.1: aie2_dump_ctx: Fatal error task ID: 0
[18680.784293] amdxdna 0000:c5:00.1: aie2_dump_ctx: JOB[0]:
[18680.784295] amdxdna 0000:c5:00.1: aie2_dump_ctx: seq: 0
[18680.784297] amdxdna 0000:c5:00.1: aie2_dump_ctx: op: 0x13
[18680.784299] amdxdna 0000:c5:00.1: aie2_dump_ctx: msg: 0x1d000001
[18680.784301] amdxdna 0000:c5:00.1: aie2_dump_ctx: fence: unsignaled
[18680.784303] amdxdna 0000:c5:00.1: aie2_dump_ctx: out_fence: unsignaled
[18680.795515] traps: python[472334] general protection fault ip:7f5faf1656c2 sp:7ffff61b9060 error:0 in libpython3.13.so.1.0[1656c2,7f5faf085000+2b2000]
```
9 条评论