# External Profiler Support

Julia provides explicit support for some external tracing profilers, enabling you to obtain a high-level overview of the runtime's execution behavior.

The currently supported profilers are:
- [Tracy](https://github.com/wolfpld/tracy)
- [Intel VTune (ITTAPI)](https://github.com/intel/ittapi)
- [NVIDIA Nsight Systems (NVTX)](#NVIDIA-Nsight-Systems-(NVTX))
- [Apple Instruments](#Apple-Instruments-(OSLog)) (macOS only)

### Adding New Zones

#### From C/C++ code

To add new zones, use the `JL_TIMING` macro. You can find numerous examples throughout the codebase by searching for `JL_TIMING`. To add a new type of zone
you add it to `JL_TIMING_OWNERS` (and possibly `JL_TIMING_EVENTS`).

#### From Julia code

The `Compiler.@zone` macro can be used to add a zone from Julia code, it is used as:

```julia
Compiler.@zone "ZONE NAME" begin
    ...
end
```

### Dynamically Enabling and Disabling Zones

The [`JULIA_TIMING_SUBSYSTEMS`](@ref JULIA_TIMING_SUBSYSTEMS) environment variable allows you to enable or disable zones for a specific Julia run. For instance, setting the variable to `+GC,-INFERENCE` will enable the `GC` zones and disable the `INFERENCE`
zones.

## Tracy Profiler

[Tracy](https://github.com/wolfpld/tracy)  is a flexible profiler that can be optionally integrated with Julia.

A typical Tracy session might look like this:

![Typical Tracy usage](tracy.png)

### Building Julia with Tracy

To enable Tracy integration, build Julia with the extra option `WITH_TRACY=1` in the `Make.user` file.

!!! note
    Julia currently bundles Tracy 0.13.1. The Tracy client and the profile viewer must use the same protocol version, so make sure the viewer (e.g. from `TracyProfiler_jll`) matches this version.

### Installing the Tracy Profile Viewer

The easiest way to obtain the profile viewer is by adding the `TracyProfiler_jll` package and launching the profiler with:

```julia
run(TracyProfiler_jll.tracy())
```

!!! note
    On macOS, you may want to set the `TRACY_DPI_SCALE` environment variable to `1.0` if the UI elements in the profiler appear excessively large.

To run a "headless" instance that saves the trace to disk, use

```julia
run(`$(TracyProfiler_jll.capture()) -o mytracefile.tracy`)
```

instead.

For information on using the Tracy UI, refer to the Tracy manual.

### Profiling Julia with Tracy

A typical workflow for profiling Julia with Tracy involves starting Julia using:

```julia
JULIA_WAIT_FOR_TRACY=1 ./julia -e '...'
```

The environment variable ensures that Julia waits until it has successfully connected to the Tracy profiler before continuing execution. Afterward, use the Tracy profiler UI, click `Connect`, and Julia execution should resume and profiling should start.

### Profiling package precompilation with Tracy

To profile a package precompilation process it is easiest to explicitly call into `Base.compilecache` with the package you want to precompile:

```julia
pkg = Base.identify_package("SparseArrays")
withenv("JULIA_WAIT_FOR_TRACY" => 1, "TRACY_PORT" => 9001) do
    Base.compilecache(pkg)
end
```

Here, we use a custom port for tracy which makes it easier to find the correct client in the Tracy UI to connect to.

### Adding metadata to zones

The various `jl_timing_show_*` and `jl_timing_printf` functions can be used to attach a string (or strings) to a zone. For example, the trace zone for inference shows the method instance that is being inferred.

The `TracyCZoneColor` function can be used to set the color of a certain zone. Search through the codebase to see how it is used.

### Viewing Tracy files in your browser

Visit [https://maleadt.github.io/trace-viewer/](https://maleadt.github.io/trace-viewer/) for an (experimental) web viewer for Tracy traces.

You can open a local `.tracy` file or provide a URL from the web (e.g. a file in a GitHub repo). If you load a trace file from the web, you can also share the page URL directly with others, enabling them to view the same trace.

### Exporting trace data to CSV

Since `.tracy` files are binary, it is often useful to export aggregated zone statistics to CSV for scripting or quick analysis. The `tracy_csvexport` tool from `TracyProfiler_jll` does this:

```julia
using TracyProfiler_jll
run(`$(TracyProfiler_jll.tracy_csvexport()) -e mytracefile.tracy`)
```

The output columns are `name, src_file, src_line, total_ns, total_perc, counts, mean_ns, min_ns, max_ns, std_ns`. The `-e` flag reports self time (time spent in a zone excluding its children), which is usually what you want for finding where time is actually spent. The `counts` column is helpful for spotting zones that are entered many times with small durations.

Useful flags include `-f <name>` to filter by zone name, `-u` to report each zone event individually (unwrapped), `-p` to include plot data (with `-u`), `-m` to report only messages, and `-t <percentile>` to report a truncated mean. Run with `--help` for the full list.

### Enabling stack trace samples

To enable call stack sampling in Tracy, build Julia with these options in your `Make.user` file:
```
WITH_TRACY := 1
WITH_TRACY_CALLSTACKS := 1
USE_BINARYBUILDER_LIBTRACYCLIENT := 0
```

You may also need to run `make -C deps clean-libtracyclient` to force a re-build of Tracy.

This feature has a significant impact on trace size and profiling overhead, so it is recommended to leave call stack sampling off when possible, especially if you intend to share your trace files online.

Note that the Julia JIT runtime does not yet have integration for Tracy's symbolification, so Julia functions will typically be unknown in these stack traces.

## Intel VTune (ITTAPI) Profiler

*This section is yet to be written.*

## NVIDIA Nsight Systems (NVTX)

[NVTX](https://github.com/NVIDIA/NVTX) is NVIDIA's lightweight instrumentation API for annotating events, code ranges, and resources so they can be visualized by NVIDIA profiling tools.
Julia emits the same `JL_TIMING` zones used by Tracy and ITTAPI as NVTX ranges, grouped and colored by subsystem, which can then be viewed in [NVIDIA Nsight Systems](https://developer.nvidia.com/nsight-systems).

NVTX is header-only and its calls are cheap no-ops unless a tool is actively attached and capturing, so building with NVTX support does not require a CUDA toolkit installation and has minimal overhead when no profiler is attached.

### Building Julia with NVTX support

To enable NVTX instrumentation of the Julia runtime, build Julia with the extra option `WITH_NVTX := 1` in the `Make.user` file, or with `make ... WITH_NVTX=1` from the command line.

### Profiling Julia with Nsight Systems

With [Nsight Systems](https://developer.nvidia.com/nsight-systems) installed, you can record a trace from the command line using `nsys`:

```
nsys profile -o my_julia_trace ./julia -e '...'
```

Open the resulting `my_julia_trace.nsys-rep` file in the Nsight Systems UI to view Julia's timing zones on the NVTX row of the timeline, grouped and colored by subsystem in the same way as with Tracy.

### Instrumenting arbitrary Julia code

If you want to additionally annotate your Julia code with custom markers see [`NVTX.jl`](https://github.com/JuliaGPU/NVTX.jl), which works independently of the `WITH_NVTX=1` build option.
You can still use the Nsight Systems profiler as explained above.

## Apple Instruments (OSLog)

On macOS, Julia can emit [os_signpost](https://developer.apple.com/documentation/os/recording-performance-data#Review-Signposts-in-Instruments) intervals that are visible in Apple Instruments. Each timing zone appears as a signpost interval in the Instruments timeline, grouped by subsystem (e.g. `GC`, `INFERENCE`, `CODEGEN`).

### Building Julia with OSLog support

Add the following to your `Make.user` file:

```
WITH_APPLE_OSLOG := 1
```

### Profiling Julia with Instruments

1. Open Instruments.app (included with Xcode).
2. Create a new trace document and add the **os_signpost** instrument.
3. Select the Julia process as the target, or launch Julia directly from Instruments.
4. Record the trace. Julia's timing zones will appear as signpost intervals grouped under the subsystem names defined in the runtime (e.g. `JL_TIMING_GC`, `JL_TIMING_INFERENCE`).

You can also record a trace from the command line using `xctrace`:

```
xctrace record --template 'Logging' --output my_julia_trace.trace --launch -- ./julia -e '...'
```

The resulting `.trace` file can be opened in Instruments for analysis (e.g. with `open my_julia_trace.trace`).

![Typical Instruments usage](os_signpost.png)
