---
hide:
  - tags
tags:
  - CPU
  - CUDA
  - GPU
  - ROCm
icon: octicons/bug-16
---

# LLVM debugging playbook

This page aims to collect notes on how to debug or reduce issues that
appear to arise from within LLVM itself and how to generate useful LLVM
bug reports.

This guide contains platform-independent notes applicable to both
CPU and GPU compilation. Additional GPU-specific notes (such as how
to perform binary substitutions in an AMD GPU context) are contained
within [the GPU debugging playbook](./gpu.md).

## Generating LLVM IR

Wthen bisecting, reducing, or debugging an issue that might manifest within
LLVM, it can be helpful to use the
`--iree-hal-dump-executable-intermediates-to=[directory]` (or the more general
`--iree-hal-dump-executable-files-to=[directory]`) flags to `iree-compile`
or `iree-opt`. These flags will cause IREE to write out the compiled LLVM module
to the specified directory so you can operate on it directly.

Generally, there will be a `.linked` file, which contains the LLVM IR shortly after
it was generated by MLIR (though after steps like bitcode library linking where
applicable) and a `.optimized` file, which contains the IR after the `opt` passes
have been run. The `.optimized` file may include reproduction instructions (if
it doesn't, the relevant compiler plugin hasn't been updated to add them).

Similarly, the final generated assembly (a `.s` or `.rocmasm` or so on) may
include reproduction instructions. Where those are present, they should be helpful
in manually recreating the LLVM compilation so that you no longer have to
route any changes through IREE.

For more details and other related flags, see
[the documentation on dumping intermediate files](../general/developer-tips.md#dumping-executable-files).

!!! tip

    While `opt` is "target-independent", many passes (such as vectorization)
    have substantial dependencies on target information. Ensure your LLVM IR
    contains a `target triple` or that you're passing `-mtriple=` to your `opt`
    invocations. There are fewer dependencies on `-mcpu=`, but it should also be
    preserved to reduce debug variability.

## LLVM binaries

To create LLVM binaries that run on the same commit as your IREE checkout,
use

``` shell
cmake --build [build-directory] --tragtet opt llc
```

to produce binaries in `[build-directory]/llvm-project/bin`. You can similarly
produce other utility binaries such as `llvm-reduce`, which aren't built by default.

## Reducing optimization levels in IREE

If you suspect an LLVM bug, try disabling (or reducing) one or both optimization
levels. LLVM has two places where a `-O[n]` is applied: the middle-end
(`opt`) and the backend/codegen (`llc`). The backend optimization level
is selected by values of the `llvm::CodeGenOptLevel` enum, which is passed to
a `createTargetMachine` call in the compiler plugin. This level defaults to
`-O3`. On the other hand, the generic/middle-end `opt` optimization level
is controlled by `llvm::OptimizationLevel` and defaults to `-O2` currently.

Setting one or both of these values to the `-O0` or `-O1` equivalent
and seeing the issue go away is an indicator that there may be a LLVM bug
in play. It may, however, also indicate that there's a race condition or
other correctness issue in the generated LLVM IR that is masked by a lack
of compiler optimizations.

## Useful flags for `opt` and `llc`

- `-print-after-all` and `-print-before/after=[passname]` can help
  locate places were suspect IR is introduced or where crashes occur, just as their
  MLIR equivalents can be used in IREE.
- `-print-module-scope` ensures IR dumps include attributes and metadata
  if those are relevant
- The exact process for feeding a binary back into IREE after manually compiling
  it is target-specific, but will generally involve
  `--iree-hal-substitute-executable-object=[executable]=[filename]`.
- `-global-isel=1` (changing the instruction selection system in `llc`)
  can be helpful in localizing a bug to instruction selection. If it solves
  your problem (or turns it into a different bug), you've substantially narrowed
  down the code that needs to be searched.
- `opt` produces human-readable output whin passed the `-S` flag, and often
  needs a `-o -` to send its results to standard output. `llc` takes a
  `--filetype={asm,obj}` argument to control whether assembly or
  assembled objects are produced.

## `llvm-diff`

When adjusting an `opt` invocation to isolate misbehaving passes or when
comparing LLVM IR from a working and a broken commit, you may be able to
use the `llvm-diff` tool to compare two LLVM IR files without the noise that
is induced by LLVM's IR numbering scheme. Note that `llvm-diff` output is written
to standard error and should be redirected with a `2>&1`.

## `llvm-reduce`

In some cases - particularly compiler crashes, the `llvm-reduce` program
(part of LLVM) may be useful. It takes a LLVM IR file and an "interestingness
script" which returns 0 (success) if there **is** a problem with a proposed
reduced input and fails otherwise.

When writing such a script, the `not` tool (especially its
`not --crash [program] [args]` mode) and `FileCheck` from the LLVM test suite
are often useful.

For example, the interestingness script used to reduce the crash in
[issue #22001](https://github.com/iree-org/iree/issues/22001) was

``` shell
#!/usr/bin/env bash

[llvm-bin]/not --crash [llvm-bin]/opt -passes='amdgpu-lower-buffer-fat-pointers' -disable-output "$@" 2>/dev/null
```

which was used with `llvm-reduce -test=interesting.sh pre-buffer-loads.ll`
on output created by adding
`--print-before=amdgpu-lower-buffer-fat-pointers --print-module-scope`
to the crashing `llc` invocation (the crashing pass was located through
backtraces and `--print-after-all`).

[This is a helpful LLVM slide deck on how to operate `llvm-reduce`](https://www.llvm.org/devmtg/2025-04/slides/tutorial/arsenault_reduce.pdf).
These slides include other useful flags and tips.

## Creating reproducers

If you're planning to file a bug against LLVM, it's helpful to create a small reproducer.

In many cases, an input that demonstrates the behavior you've identified as a bug
can be either created with `llvm-reduce` or by hand.

However, in some cases (such as incorrect dispatch results that aren't
clearly attributable to a particular change) all you can do is create a
reproduction harness. The exact process for creating these is target-specific,
but such a harness should be a piece of standalone code that links against / loads
different versions of the misbehaving input, calls the function at issue,
and reports the results (likely checking against a naive implementation).

This wrapper program should be accompanied by a simple build process that doesn't
depend on IREE and instructions on how to run it. The build should produce binaries
from LLVM IR - ideally, the post-`opt` IR, by calling `llc` (or, if needed, `opt`).

If the bug goes away at different optimization levels, you should build a working
and a non-working binary.
