---
hide:
  - tags
tags:
  - GPU
  - CUDA
  - Metal
  - ROCm
  - Vulkan
icon: octicons/bug-16
---

# GPU debugging playbook

This page aims to provide general approaches and practical tips for debugging
GPU compiler/runtime correctness/performance issues in IREE.

GPUs fundamentally have similar architectures and software stacks.
We target GPUs from various vendors using different GPU APIs, but they share
quite a lot common infrastructure in IREE.
So the approaches and tips here should be widely applicable.

For the ones that are specific to a particular kind of problem/component/GPU,
they are prefixed with proper icons to be clear.
Here are what those icons represents--

Icon | Category
:--: | :------:
:material-check-bold: | Correctness
:material-trending-up: | Performance
:simple-amd: | AMD HIP/ROCm
:simple-apple: | Apple Metal
:simple-microsoft: | Microsoft DirectX
:simple-nvidia: | NVIDIA CUDA
:simple-vulkan: | Vulkan

## General methodology

The difficulties associated with debugging typically arise from **isolating the
problematic component and pinpointing the culprit**.
Once done, the solution typically derives naturally.

There are many components in the IREE stack; hierarchically we can categorize
them into either the compiler or runtime bucket:

* For compilers, there are multiple layers from the top to the bottom--frontend
  input importers, IREE flow/stream compilation, IREE host/device compilation,
  GPU backend in LLVM proper or GPU driver compiler for SPIR-V.
* For runtime, we have fewer layers--IREE HAL drivers, and GPU driver.

Any of the above components/layers can have bugs.
It's important to reduce the potential surface area to make the problem more
tractable.

Once we have a more isolated case, the general methodology to pinpoint the
exact culprit is to

1. **collect and inspect the symptoms,**
2. **form hypothesis and run experiments to prove/refute the hypothesis, and**
3. **iterate**.

### .. with shortcuts

The above procedure is for facing a large problem with no clue, for example,
when bringing up a new model end-to-end via IREE.

Though most of the time, we can leverage existing facilities to avoid going down
the full top-down hierarchical debugging procedure.
For example, for regression happening on an existing model, CI or `git bitsect`
might tell us directly the culprit commit.

### .. using tools

For issues with strong signals like crashing, it's also easier to pinpoint the
exact culprit with dedicated tools--we can leverage various
[sanitizers](./sanitizers.md) or debuggers.

## Isolating the problematic component

If we are facing a large problem without a clear clue, we need to isolate the
problematic compiler or runtime layer first, typically by comparing with a
working solution:

!!! tip "[correctness/performance]"

    Sanitize the environment first.
    Asking these questions and making sure the environment is proper can save
    you hours of debugging sometimes:

    * Did you recently updated the GPU SDK or driver?
    * Are others able to reproduce the issue?
    * If not what SDK / driver versions they are using?
    * Is your machine drawing enough power when benchmarking?
    * Is your machine connected with a monitor (e.g., for Vulkan)?
    * How long since you last rebooted your machine? 👻
    * (Windows only) Did you change the `TdrDelay` value to something more lenient (e.g., 600 seconds)?
      `TdrDelay` (`HKEY_LOCAL_MACHINE\System\CurrentControlSet\Control\GraphicsDrivers`) is
      the timeout for a response to a preempt request from the GPU scheduler.
      A large enough computation can outlive the default value of 2 seconds, leading to a GPU reset.

!!! tip "[correctness/performance]"

    We have multiple GPU targets/drivers in IREE--LLVMGPU/CUDA, LLVMGPU/HIP,
    SPIR-V/Vulkan, SPIR-V/Metal.

    For the _same_ GPU, we typically have two paths to target, e.g., CUDA/HIP
    or Vulkan for NVIDIA/AMD GPUs, Metal or Vulkan for Apple GPUs.

    If one path is correct/performant, we can diff against it to try isolate
    the problem--the common/shared compiler/runtime code is likely okay; what
    differs between paths is likely problematic.

!!! tip "[correctness/performance] [vulkan]"

    Vulkan supports different GPUs.
    Similarly, if one GPU gives correct/performant result, we diff against it
    to find clues.

    Even more code in compiler/runtime are shared here; what's problematic is
    likely different capabilities triggering different CodeGen pipelines so
    revealing bugs in a particular CodeGen pipeline.
    Or there are driver issues from a particular vendor.

!!! tip "[correctness]"

    If the CPU is working properly, we can use the same dispatch region formation
    and diff against the CPU dispatches one by one to isolate the problem. See
    [this issue](https://github.com/iree-org/iree/issues/14739) as an example.

!!! tip "[correctness]"

    `--iree-flow-trace-dispatch-tensors` and/or `--iree-flow-break-dispatch=` to
    `iree-compile` is quite helpful to inspect the output after all/each
    dispatch(es).

!!! tip "[correctness]"

    `iree-reduce` is a great tool to reduce and isolate issues programmatically.
    See more details [here](https://github.com/iree-org/iree/blob/main/samples/reducer/README.md).

## Pinpointing compiler issues

Once we identified that the problem is due to some compiler issue, we can
investigate by comparing with different paths and inputs:

!!! tip "[correctness]"

    For the same dispatch, we may have different CodeGen pipelines, e.g.,
    for matmul we can have simple SIMT pipeline or using tensor/matrix cores.
    We can try to switch between different pipelines to isolate the problem.

!!! tip "[correctness]"

    Assuming we have a small repro, we can also try to see if there are
    "patterns" in the wrong result (e.g.,
    [this issue](https://github.com/iree-org/iree/issues/14739#issuecomment-1685149869)).
    Or mutate the input to see if the failure has some "consistency".

!!! tip "[correctness/performance]"

    `--mlir-print-ir-*` and `--debug*` to `iree-opt` is our best friend.
    Sometimes it just takes eyeballing the IRs between stages to find clues.

!!! tip "[performance]"

    For identifying performance issues, we typically need to use:

    * [Tracy profiling](../performance/profiling-with-tracy.md) to get a
      course-grained command-buffer timing to understand what's the most
      time-consuming kernels.
      Typical big performance issues include but not limit to going down a
      incorrect CodeGen pipeline, missing tiling/vectorization, having an
      improper tiling/vectorization configuration, and so on.
      If the course-grained information is not enough, then we need to
    * [vendor-specific tools](../performance/profiling-gpu-vulkan.md) to
      understand kernel internal counters to identify the bottleneck.

!!! tip "[correctness]"

    Some targets support the `gpu.printf` operation for printing out values from
    within GPU code, and many of the targets that don't _could_ support it with
    some work in IREE or upstream MLIR.

!!! tip "[correctness]"

    If you suspect an issue in an LLVM backend, check
    [the LLVM debugging playbook](./llvm.md) for general recommendations.

    [:simple-amd:] An occasional source of failures has been disagreements about
    code object version. Ensure that both `amdhsa_code_object_version` metadata
    and `__oclc_ABI_version` are set and agree.

## Pinpointing runtime issues

On the other side, if we suspect that it's a runtime issue, here are some
useful approaches and tips:

!!! tip "[correctness/performance]"

    [Tracy profiling](../performance/profiling-with-tracy.md) is a great way to
    view how the application runs dynamically.
    It can help to show problematic GPU API call sequences and performance
    bottlenecks.

    * It requires adding `-DIREE_ENABLE_RUNTIME_TRACING=ON` during CMake
      configuration, or use the `IREE_PY_RUNTIME=tracy` environment variable
      when invoking IREE runtime installed via Python packages.

!!! tip "[correctness]"

    GPU validation can sometimes give us hints:

    * [:simple-apple:] Enable validation via `export METAL_DEVICE_WRAPPER_TYPE=1`.
    * [:simple-vulkan:] Use `--vulkan_validation_layers=true` to `iree-run-module`, or
    * [:simple-vulkan:] Force enable via environment variables to the Vulkan loader:
      `export VK_INSTANCE_LAYERS=VK_LAYER_LUNARG_standard_validation`
      (may additionally need
      `export VK_LAYER_PATH=$VULKAN_SDK/etc/vulkan/explicit_layer.d` and
      `export LD_LIBRARY_PATH=$VULKAN_SDK/lib` if Vulkan SDK is not installed
      to a system path).

!!! tip "[correctness]"

    Turning on verbose output can give us more information:

    * When compiling IREE runtime, add
      `-DCMAKE_C_FLAGS=-DIREE_VM_EXECUTION_TRACING_FORCE_ENABLE=1` in CMake
      configuration to enable VM op tracing.
    * [:simple-vulkan:] Use `--vulkan_debug_verbosity=4` to `iree-run-module`.
    * [:simple-vulkan:] Print all Vulkan APIs calls with detailed arguments:
      `export VK_INSTANCE_LAYERS=VK_LAYER_LUNARG_api_dump`
      (may additionally need
      `export VK_LAYER_PATH=$VULKAN_SDK/etc/vulkan/explicit_layer.d` and
      `export LD_LIBRARY_PATH=$VULKAN_SDK/lib` if Vulkan SDK is not installed
      to a system path).

!!! tip "[correctness]"

    Try different "debugging modes" provided by HAL drivers:

    * [:simple-nvidia:] Switch `--cuda_use_streams=` between `true` and `false` to
      `iree-run-module` to see whether the issue comes from the stream/graph
      command buffer implementation.
    * [:simple-nvidia:] Switch `--cuda_async_allocations=false` to `iree-run-module` to
      see if the issue comes from async allocation.
    * [:simple-apple:] Use `--metal_serial_command_dispatch=true`,
      `--metal_command_buffer_retain_resources=true`, or
      `--metal_resource_hazard_tracking=true` to `iree-run-module` to see
      if any of the above "fixes" the issue.
      It can help to isolate the potential problem.
    * [:simple-vulkan:] Use `--vulkan_robust_buffer_access=true` to `iree-run-module`
      especially when seeing undeterministic/corrupted contents in buffers and
      suspecting there are buffer allocation/indexing issues.

## Binary substitution for ROCm

[:simple-amd:] The AMD ROCm target supports binary substitution on HSA code objects
(`.hsaco` files).

These files are, under the hood, ELF shared libraries containing kernel code.

If you have manually produced a binary you want to test, such as by manually
running `llc` with different optimization flags, you can turn the `.o` into
a `.hsaco` with

``` shell
ld.lld -o [filename].hsaco -shared [filename].o
```

In full, if you have a dispatch in `dispatch.mlir` and want to recompile it with
while potentially making modifications, the process is

``` shell
# A PATH edit is not strictly required. It is used here to point out that the
# LLVM binaries used should me built from the same LLVM sources IREE uses.
export PATH="[build-directory]/llvm-project/bin:[build-directory]/tools/bin:$PATH"

iree-compile dispatch.mlir \
  --iree-hal-target-device=hip \
  --iree-rocm-target=<target> \
  -o original.vmfb \
  --iree-hal-dump-executable-files-to=odump
# Opt flags are in dump/[...].optimized.ll to a file.
opt -o - [opt flags] <dump/[...].linked.ll >altered.opt.ll
# llc flags are in dump/[...].rocmsasm.
# To produce an assembly file.
llc [llc flags] altered.opt.ll -o altered.rocmasm
# To produce an object file.
llc [llc flags] altered.opt.ll -o altered.o --filetype=obj
# Linking to an HSACO.
ld.lld -o altered.hsaco -shared altered.o
# Re-compile with substitution. [dispatch_name] is the name of the
# `hal.executable` op symbol, not the variant within it. This can
# be found by looking at the relevant configured_*.mlir file in dump/, for
# example.
iree-compile dispatch.mlir \
  --iree-hal-target-device=hip \
  --iree-rocm-target=<target> \
  -o altered.vmfb \
  --iree-hal-substitute-executable-object=[dispatch_name]=altered.hsaco
```

If successful, `iree-compile` will print a message stating

``` shell
NOTE: hal.executable `[executable name]` substituted with object file at`altered.hsaco`
```

During each of these steps, you can change the flags or manually edit the `.ll`
(or even `.rocmasm`) files to attempt to get potentially-different behavior.

!!! note

    The binary substitution process could be used to replace a dispatch with a
    completely foreign implementation, such as one written in C, so long as the
    function names and argument handling schemes agree. If you do this, please
    document the steps here.
