---
title: CLI reference
sidebarTitle: CLI
description: Run one-off generation tasks and launch the HTTP server from the command line.
---
Use the CLI for one-off generation with `sglang generate` or to start a persistent HTTP server with `sglang serve`.

### Overlay repos for non-diffusers models

If `--model-path` points to a supported non-diffusers source repo, SGLang can resolve it
through a self-hosted overlay repo.

SGLang first checks a built-in overlay registry. Concrete built-in mappings can be added over time without changing the CLI surface.

Override example:

```bash Command
export SGLANG_DIFFUSION_MODEL_OVERLAY_REGISTRY='{
  "Wan-AI/Wan2.2-S2V-14B": {
    "overlay_repo_id": "your-org/Wan2.2-S2V-14B-overlay",
    "overlay_revision": "main"
  }
}'

sglang generate \
  --model-path Wan-AI/Wan2.2-S2V-14B \
  --config configs/wan_s2v.yaml
```

The overlay repo should be a complete diffusers-style/componentized repo

You can also pass the overlay repo itself as `--model-path` if it contains `_overlay/overlay_manifest.json`.

Notes:
1. `SGLANG_DIFFUSION_MODEL_OVERLAY_REGISTRY` is only an optional override for
development and debugging. It accepts either a JSON object or a path to a JSON
file, and can extend or replace built-in entries for the current process.
2. On the first load, SGLang will:
   - download overlay metadata from the overlay repo
   - download the required files from the original source repo
   - materialize a local standard component repo under `~/.cache/sgl_diffusion/materialized_models/`
3. Later loads reuse the materialized local repo. The materialized repo is what the runtime loads as a normal componentized model directory.


## Quick Start

### Generate

```bash Command
sglang generate \
  --model-path Qwen/Qwen-Image \
  --prompt "A beautiful sunset over the mountains" \
  --save-output
```

### Serve

```bash Command
sglang serve \
  --model-path Wan-AI/Wan2.1-T2V-1.3B-Diffusers \
  --num-gpus 4 \
  --ulysses-degree 2 \
  --ring-degree 2 \
  --port 30010
```

For request and response examples, see [OpenAI-Compatible API](./openai_api).

<Tip>
Use `sglang generate --help` and `sglang serve --help` for the full argument list. The CLI help output is the source of truth for exhaustive flags.
</Tip>

## Common Options

### Model and runtime

- `--model-path {MODEL}`: model path or Hugging Face model ID
- `--model-variant {NAME}`: semantic checkpoint variant to load when one model repository contains multiple weight partitions. The pipeline maps this stable name to the repository layout before loading; for example, MiniMax-H3 accepts `fl2va` and `ref2va`. This is a server/load-time choice, unlike a request's `task`.
- `--model-subfolder {PATH}`: advanced direct override for a component subfolder inside the model repository. Prefer `--model-variant` when the pipeline exposes semantic routing. If both are supplied, they must resolve to the same weight partition.
- `--lora-path {PATH}` and `--lora-nickname {NAME}`: load a LoRA adapter
- `--lora-merge-mode {auto|merge|dynamic}`: choose how LoRA is applied. `auto` statically merges regular weights and uses dynamic LoRA for FSDP-sharded weights to avoid full-gather peaks.
- `--num-gpus {N}`: number of GPUs to use
- `--performance-mode {manual|auto|speed|memory}` / `--mode`: preset for latency/throughput and memory defaults. `auto` is the default and keeps safe offload defaults, using FSDP only for validated DiT-offload replacement paths; `speed` also enables `--enable-torch-compile` unless the model-specific deployment config opts out or you explicitly disable it. Use `manual` to keep performance-related server args under explicit user control. Explicit offload, FSDP, and parallelism flags take precedence in all modes.
- `--tp-size {N}`: tensor parallelism size. Depending on the pipeline, it can shard the DiT, one or more encoders, or both.
- `--sp-degree {N}`: sequence parallelism size
- `--dp-size {N}` (alias `--data-parallel-size`): number of data-parallel replicas. Each replica is a full copy of the engine on `num_gpus / N` GPUs with its own ingress; generation requests round-robin across replicas, realtime sessions stick to the replica holding their state, and control operations (weights, LoRA, memory occupation, shutdown) apply to every replica. Combines with the other parallelism axes (`num_gpus = dp × cfg × tp × sp`); monolithic serving only.
- `--ulysses-degree {N}` and `--ring-degree {N}`: USP parallelism controls
- `--kv-gather-degree {N}`: sequence-parallel degree that splits rows inside attention and exchanges with one K/V all-gather (queries stay local) instead of Ulysses all-to-all. Non-causal attention only; does not compose with `--ulysses-degree`/`--ring-degree` yet. When no SP degree is set explicitly, `sp_degree=2` defaults to `kv_gather_degree=2` (its measured-win zone) and higher degrees default to Ulysses; under that auto assignment, attention calls the gather path cannot take fall back to the Ulysses exchange, while an explicit degree fails instead of degrading.
- `--enable-cfg-parallel {true|false}`: enable or explicitly disable CFG parallelism
- `--encoder-parallel {auto|fold|dp|replicate}`: how the text/image encoders use the GPUs the DiT replica leaves idle during encoding. `auto` (the default for both `generate` and `serve`) TP-folds an encoder wide enough to pay for the per-layer all-reduce, selects DP for a server batch when it can engage, and otherwise replicates; `fold` forces the shard whenever the dims allow it; `dp` splits a batched encode across ranks and needs `--batching-max-size > 1` to engage; `replicate` encodes redundantly on every rank. `fold` and `replicate` are bitwise-identical to single-GPU encoding. See [Encoder Parallelism](/docs/sglang-diffusion/encoder_parallel).
- `--warmup-mode {off|request|server}`: control startup warmup for `sglang serve`; `off` skips warmup, `request` primes the request path, and `server` runs a full synthetic server warmup before serving traffic
- `--enable-torch-compile {true|false}`: compile native diffusion hot paths. When no warmup mode is configured, this also enables server warmup so first real requests do not pay compile latency.
- `--offload-during-compile {true|false}`: when compile warmup is active, temporarily layerwise-offload DiT weights and move resident non-DiT components off-device so `max-autotune` fits on tighter-memory GPUs; the configured serving residency is restored before real traffic. Skipped under existing layerwise offload, Cache-DiT, or FSDP.
- `--enable-breakable-cuda-graph {true|false}`: capture supported DiT forwards as breakable CUDA graph segments to reduce launch overhead. Requires `--warmup-resolutions` for every served resolution because each resolution is captured separately.
- `--bcg-text-buckets {N...}`: prompt-length padding buckets for breakable CUDA graph capture/replay reuse.
- `--attention-backend {BACKEND}`: attention backend for native SGLang and diffusers pipelines
- `--component-attention-backends {MAP}`: per-component attention backend overrides, for example `text_encoder=torch_sdpa,transformer=fa`
- `--attention-backend-config {CONFIG}`: attention backend configuration
- `--srt-encoder-url {HTTPADDRESS}`: address of SGLang srt server with AR model for GLM-Image like models. See [Models with AR Stage](../models_with_ar).
- `--srt-encoder-timeout {SECONDS}`: Timeout in seconds for HTTP requests to the SGLang encoder server
- `--srt-encoder-connection-timeout {SECONDS}`: TCP connection timeout in seconds for SGLang encoder server
- `--scheduler-rpc-timeout {SECONDS}`: optional end-to-end deadline for an internal scheduler RPC, including scheduler queue time. It is unset by default so valid long-running and queued video jobs are not failed by the transport layer. Set it only when the deployment requires a bounded request deadline; caller cancellation and server shutdown remain effective without it.
- `--pe-server-url {HTTPADDRESS}`: url of SGLang server hosting the PE model (e.g., for ERNIE-Image). See [Models with Prompt Enhancement](../models_with_pe).

### Sampling and output

- `--prompt {PROMPT}` and `--negative-prompt {PROMPT}`
- `--image-path {PATH} [{PATH} ...]`: input image(s) for image-to-video or image-to-image generation
- `--num-inference-steps {STEPS}` and `--seed {SEED}`
- `--num-outputs-per-prompt {N}` / `--num-outputs {N}`: generate multiple outputs for each prompt. A scalar seed expands as `seed + output_index`.
- `--quality {lossless,high}`: request-level quality. `lossless` (default) keeps the exact reference path, bit-exact against the reference implementation; `high` opts into the model-owned validated accelerated path, whose quality stays guaranteed but is not bit-exact. Support and validated deployment constraints are model-specific.
- `--height {HEIGHT}`, `--width {WIDTH}`, `--num-frames {N}`, `--fps {FPS}`
- `--output-path {PATH}`, `--output-file-name {NAME}`, `--save-output`, `--return-frames`

For frame interpolation and upscaling, see [Post-Processing](./post_processing).

### Quantized transformers

For quantized transformer checkpoints, prefer:

- `--model-path` for the base pipeline
- `--transformer-path` for a quantized `transformers` transformer component folder
- `--transformer-weights-path` for a quantized safetensors file, directory, or repo
- `--quantization` for online quantization (apply quantization to unquantized models at load time, activations are quantized dynamically)
- `--quantization-ignored-layers` layer name patterns to keep unquantized (e.g. `attention.to_`)

See [Quantization](../quantization) for supported quantization families and examples.

### Request logging

- `--log-requests`: Log user-facing fields of all requests (default: `False`). The verbosity is decided by `--log-requests-level`.
- `--log-requests-level &#123;0|1|2|3&#125;`: Verbosity level for request logging (default: `2`). 0: Log metadata (request id). 1: Log metadata and sampling config (seed, steps, guidance, resolution, frames, fps, ...). 2: Log metadata, sampling config and prompt (truncated to 2 KiB). 3: Log metadata, sampling config and full prompt.
- `--log-requests-format &#123;text|json&#125;`: Format for request logging (default: `text`). `text` is human-readable; `json` outputs structured JSON lines.
- `--log-requests-target &#123;TARGET...&#125;`: Target(s) for request logging. Use `stdout` for console output and/or directory path(s) for file output. Can specify multiple targets, e.g., `--log-requests-target stdout /my/log/dir`.

## Configuration Files

Use `--config` to load JSON or YAML configuration. Command-line flags override values from the config file.

```bash Command
sglang generate --config config.yaml
```

Example:

```yaml Config
model_path: FastVideo/FastHunyuan-diffusers
prompt: A beautiful woman in a red dress walking down a street
output_path: outputs/
num_gpus: 2
sp_degree: 2
tp_size: 1
num_frames: 45
height: 720
width: 1280
num_inference_steps: 6
seed: 1024
fps: 24
precision: bf16
vae_precision: fp16
vae_tiling: true
vae_sp: true
enable_torch_compile: false
```

## Generate

`sglang generate` runs a single generation job and exits when the job finishes.

```bash Command
sglang generate \
  --model-path Wan-AI/Wan2.2-T2V-A14B-Diffusers \
  --text-encoder-cpu-offload \
  --pin-cpu-memory \
  --num-gpus 4 \
  --ulysses-degree 2 \
  --ring-degree 2 \
  --prompt "A curious raccoon" \
  --save-output \
  --output-path outputs \
  --output-file-name "a-curious-raccoon.mp4"
```

<Note>
HTTP server-only arguments are ignored by `sglang generate`.
</Note>

For supported native pipelines, set `SGLANG_CACHE_DIT_ENABLED=true` to enable Cache-DiT. For the diffusers backend, use `--backend diffusers --cache-dit-config ...`. See [Cache-DiT](../cache_dit).

For supported image pipelines, breakable CUDA graph can be enabled with `--enable-breakable-cuda-graph`, but you must declare every served resolution in `--warmup-resolutions` so warmup captures matching graph signatures.

### Layerwise Offload

Use layerwise offload when a large component does not fit comfortably in GPU memory. By default, `--dit-layerwise-offload` only applies to legacy DiT components. Use `--layerwise-offload-components` to select pipeline component names explicitly (`--layerwise-offload-modules` is accepted as an alias):

```bash Command
sglang generate \
  --model-path Wan-AI/Wan2.2-T2V-A14B-Diffusers \
  --dit-layerwise-offload \
  --layerwise-offload-components transformer text_encoder \
  --dit-offload-prefetch-size 0 \
  --prompt "A quiet city street after rain"
```

The values must match keys in the selected pipeline's `pipeline.modules`, such as `transformer`, `text_encoder`, `image_encoder`, `vae`, `condition_image_encoder`, `spatial_upsampler`, or `vocoder`. Use `all` to select every layerwise-offloadable component. Prefer the smallest component set that solves the memory issue because layerwise offload can increase latency.

## Serve

`sglang serve` starts the HTTP server and keeps the model loaded for repeated requests.

```bash Command
sglang serve \
  --model-path Wan-AI/Wan2.1-T2V-1.3B-Diffusers \
  --text-encoder-cpu-offload \
  --pin-cpu-memory \
  --num-gpus 4 \
  --ulysses-degree 2 \
  --ring-degree 2 \
  --port 30010
```

### Health endpoints

SGLang Diffusion separates process liveness from inference readiness:

| Endpoint | Success condition | Recommended use |
| --- | --- | --- |
| `GET /liveness` | The HTTP server is accepting requests. It remains `200` during server warmup. | Kubernetes liveness probe |
| `GET /health` | The server is ready for normal inference traffic. It returns `503` while server-based synthetic warmup is running and `200` after it completes. | Startup and readiness probes |
| `GET /health_generate` | Compatibility alias for `/health`. It does not currently issue a generation request in SGLang Diffusion. | Existing integrations only |

`/health` gates only server-based warmup. With `--warmup-mode off` or
`--warmup-mode request`, it returns `200` once the HTTP server starts; those modes
do not promise that compilation or other first-request work has completed. If
server-based warmup fails, the server terminates instead of reporting ready.

Do not use `/health` as a liveness probe: a long server warmup can legitimately
keep it at `503` for several minutes.

### Cloud Storage

SGLang Diffusion can upload generated images and videos to S3-compatible object storage after generation.

```bash Command
export SGLANG_CLOUD_STORAGE_TYPE=s3
export SGLANG_S3_BUCKET_NAME=my-bucket
export SGLANG_S3_ACCESS_KEY_ID=your-access-key
export SGLANG_S3_SECRET_ACCESS_KEY=your-secret-key
export SGLANG_S3_ENDPOINT_URL=https://minio.example.com
```

See [Environment Variables](../environment_variables) for the full set of storage options.

## Component Path Overrides

Override individual pipeline components such as `vae`, `transformer`, or `text_encoder` with `--<component>-path`.

```bash Command
sglang serve \
  --model-path black-forest-labs/FLUX.2-dev \
  --vae-path fal/FLUX.2-Tiny-AutoEncoder
```

The component key must match the key in the model's `model_index.json`, and the path must be either a Hugging Face repo ID or a complete component directory.

## Component Attention Backend Overrides

Use `--component-attention-backends` when one pipeline component needs a different native attention backend from the global `--attention-backend`.

```bash Command
sglang generate \
  --model-path Lightricks/LTX-2.3 \
  --attention-backend fa \
  --component-attention-backends text_encoder=torch_sdpa
```

The component key must match a pipeline module key such as `text_encoder`, `text_encoder_2`, `transformer`, `transformer_2`, or `connectors`. Component overrides take precedence over the global `--attention-backend` only while that component is being constructed.

You can also pass dotted CLI entries:

```bash Command
sglang generate \
  --model-path <MODEL_PATH_OR_ID> \
  --component-attention-backends.text_encoder torch_sdpa \
  --component-attention-backends.transformer fa
```

## Diffusers Backend

Use `--backend diffusers` to force vanilla diffusers pipelines when no native SGLang implementation exists or when a model requires a custom pipeline class.

### Key Options

<table>
  <thead>
    <tr>
      <th>Argument</th>
      <th>Values</th>
      <th>Description</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td><code>--backend</code></td>
      <td><code>auto</code>, <code>sglang</code>, <code>diffusers</code></td>
      <td>Choose native SGLang, force native, or force diffusers</td>
    </tr>
    <tr>
      <td><code>--attention-backend</code></td>
      <td><code>flash</code>, <code>_flash_3_hub</code>, <code>sage</code>, <code>xformers</code>, <code>native</code></td>
      <td>Attention backend for diffusers pipelines</td>
    </tr>
    <tr>
      <td><code>--trust-remote-code</code></td>
      <td>flag</td>
      <td>Required for models with custom pipeline classes</td>
    </tr>
    <tr>
      <td><code>--vae-tiling</code> and <code>--vae-slicing</code></td>
      <td>flag</td>
      <td>Lower memory usage for VAE decode</td>
    </tr>
    <tr>
      <td><code>--dit-precision</code> and <code>--vae-precision</code></td>
      <td><code>fp16</code>, <code>bf16</code>, <code>fp32</code></td>
      <td>Precision controls</td>
    </tr>
    <tr>
      <td><code>--enable-torch-compile</code></td>
      <td>flag</td>
      <td>Enable <code>torch.compile</code></td>
    </tr>
    <tr>
      <td><code>--cache-dit-config</code></td>
      <td><code>{PATH}</code></td>
      <td>Cache-DiT config for diffusers pipelines</td>
    </tr>
  </tbody>
</table>

### Example

```bash
sglang generate \
  --model-path AIDC-AI/Ovis-Image-7B \
  --backend diffusers \
  --trust-remote-code \
  --attention-backend flash \
  --prompt "A serene Japanese garden with cherry blossoms" \
  --height 1024 \
  --width 1024 \
  --num-inference-steps 30 \
  --save-output \
  --output-path outputs \
  --output-file-name ovis_garden.png
```

For pipeline-specific arguments not exposed in the CLI, pass `diffusers_kwargs` in a config file.
