# Volcano vGPU User Guide

## Background Knowledge of GPU Sharing Modes in Volcano

Volcano supports **two GPU sharing modes** for virtual GPU (vGPU) scheduling:

### 1. HAMI-core (Software-based vGPU)

**Description**:
Leverages **VCUDA**, a CUDA API hijacking technique to enforce GPU core and memory usage limits, enabling **software-level virtual GPU slicing**.

**Use case**:
Ideal for environments requiring **fine-grained GPU sharing**. Compatible with all GPU types.

---

### 2. Dynamic MIG (Hardware-level GPU Slicing)

**Description**:
Utilizes **NVIDIA's MIG (Multi-Instance GPU)** technology to partition a physical GPU into isolated instances with **hardware-level performance guarantees**.

**Use case**:
Best for **performance-sensitive** workloads. Requires **MIG-capable GPUs** (e.g., A100, H100).

---

GPU Sharing mode is a node configuration. Volcano supports heterogeneous cluster(i.e a part of node uses HAMi-core while another part uses dynamic MIG), See [volcano-vgpu-device-plugin](https://github.com/Project-HAMi/volcano-vgpu-device-plugin) for configuration and details.

## Installation

To enable vGPU scheduling, the following components must be set up based on the selected mode:

### Common Requirements

* **Prerequisites**:

  * NVIDIA driver > 440
  * nvidia-docker > 2.0
  * Docker configured with `nvidia` as the default runtime
  * Kubernetes >= 1.16
  * Volcano >= 1.9

* **Install Volcano**:

  * Follow instructions in [Volcano Installer Guide](https://github.com/volcano-sh/volcano?tab=readme-ov-file#quick-start-guide)

* **Install Device Plugin**:

  * Deploy [`volcano-vgpu-device-plugin`](https://github.com/Project-HAMi/volcano-vgpu-device-plugin)

  **Note:** the [vgpu device plugin yaml](https://github.com/Project-HAMi/volcano-vgpu-device-plugin/blob/main/volcano-vgpu-device-plugin.yml) also includes the ***Node GPU mode*** and the ***MIG geometry*** configuration. Please refer to the [vgpu device plugin config](https://github.com/Project-HAMi/volcano-vgpu-device-plugin/blob/main/doc/config.md).

* **Validate Setup**:
  Ensure node allocatable resources include:

```yaml
volcano.sh/vgpu-memory: "89424"
volcano.sh/vgpu-number: "8"
```
* **Scheduler Config Update**:

```yaml
kind: ConfigMap
apiVersion: v1
metadata:
  name: volcano-scheduler-configmap
  namespace: volcano-system
data:
  volcano-scheduler.conf: |
    actions: "enqueue, allocate, backfill"
    tiers:
    - plugins:
      - name: predicates
      - name: deviceshare
        arguments:
          deviceshare.VGPUEnable: true   # enable vgpu plugin
          deviceshare.SchedulePolicy: binpack  # scheduling policy. binpack / spread
```

Check with:

```bash
kubectl get node {node-name} -o yaml
```

---

### PodGroup Device Spread

To prevent two pods from the same PodGroup from sharing the same vGPU device, add the
`volcano.sh/vgpu-podgroup-policy: spread` annotation to the pod. Pods without this
annotation continue to use the default binpacking behavior.

```yaml
metadata:
  name: spread-pod
  annotations:
    volcano.sh/vgpu-podgroup-policy: "spread"
```

---

### Selecting vGPU Devices by UUID

Use `volcano.sh/vgpu-use-gpuuuid` as a comma-separated allowlist of physical GPU UUIDs:

```yaml
metadata:
  annotations:
    volcano.sh/vgpu-use-gpuuuid: "GPU-03f69c50-207a-2038-9b45-23cac89cb67d"
```

Use `volcano.sh/vgpu-nouse-gpuuuid` to exclude physical GPU UUIDs:

```yaml
metadata:
  annotations:
    volcano.sh/vgpu-nouse-gpuuuid: "GPU-03f69c50-207a-2038-9b45-23cac89cb67d"
```

UUIDs are matched exactly after trimming whitespace. If no permitted device can satisfy the request, the pod remains unschedulable. When the same UUID appears in both annotations, the denylist takes precedence.

---

### HAMI-core Usage

* **Pod Spec**:

```yaml
metadata:
  name: hami-pod
  annotations:
    volcano.sh/vgpu-mode: "hami-core"
spec:
  schedulerName: volcano
  containers:
  - name: cuda-container
    image: nvidia/cuda:9.0-devel
    resources:
      limits:
        volcano.sh/vgpu-number: 1    # requesting 1 gpu cards
        volcano.sh/vgpu-cores: 50    # (optional)each vGPU uses 50%
        volcano.sh/vgpu-memory: 3000 # (optional)each vGPU uses 3G GPU memory
```

---

### Dynamic MIG Usage

* **Enable MIG Mode**:

If you need to use MIG (Multi-Instance GPU), you must run the following command on the GPU node.

```bash
sudo nvidia-smi -mig 1
```

* **Geometry Config (Optional)**:
  The volcano-vgpu-device-plugin automatically generates an initial MIG configuration, which is stored in the `volcano-vgpu-device-config` ConfigMap under the `kube-system` namespace. You can customize this configuration as needed. For more details, refer to the [vgpu device plugin yaml](https://github.com/Project-HAMi/volcano-vgpu-device-plugin/blob/main/volcano-vgpu-device-plugin.yml).

* **Pod Spec with MIG Annotation**:

```yaml
metadata:
  name: mig-pod
  annotations:
    volcano.sh/vgpu-mode: "mig"
spec:
  schedulerName: volcano
  containers:
  - name: cuda-container
    image: nvidia/cuda:9.0-devel
    resources:
      limits:
        volcano.sh/vgpu-number: 1
        volcano.sh/vgpu-memory: 3000
```

Note: Actual memory allocated depends on best-fit MIG slice (e.g., request 3GB → 5GB slice used).

---

## GPU Exclusivity (HAMI-core only)

GPU exclusivity ensures that pods matching configured label rules get **dedicated physical GPUs** — no other rule-matching pod can share those GPUs. Non-matching pods can still share GPUs normally. This feature only applies to **hami-core** nodes; dynamic MIG nodes are skipped since they already provide hardware-level isolation. Unlike the `spread` scheduling policy, which distributes pods across different nodes for load balancing, GPU exclusivity operates at the **GPU device level within a single node** — it prevents rule-matching pods from sharing the same physical GPU, even when they land on the same node. The two features are orthogonal and can be used together. If a rule-matching pod cannot find a GPU that satisfies exclusivity (i.e., all GPUs on a node are already reserved by other rule-matching pods), the pod will **fail FilterNode** for that node and remain pending until a suitable node with available exclusive GPUs becomes available.

### Configuration

Add `deviceshare.GPUExclusiveRules` to the deviceshare plugin arguments:

```yaml
- name: deviceshare
  arguments:
    deviceshare.VGPUEnable: true
    deviceshare.GPUExclusiveRules:
      - workloadType: training
      - workloadType: batch
        priority: high
```

Each rule is a set of label key-value pairs. A pod must match **all** labels in a rule to receive exclusivity. Exclusivity is enforced within the same rule only — pods matching **different** rules can still share GPUs with each other.

### Pod Example

```yaml
metadata:
  name: training-job
  labels:
    workloadType: training
spec:
  schedulerName: volcano
  containers:
  - name: trainer
    image: nvidia/cuda:12.0-base
    resources:
      limits:
        volcano.sh/vgpu-number: "4"
        volcano.sh/vgpu-memory: "16384"
```

---

## Scheduler Mode Selection

* **Explicit Mode**:

  * Use annotation `volcano.sh/vgpu-mode` to force hami-core or MIG mode.
  * If annotation is absent, scheduler selects mode based on resource fit and policy.

* **Scheduling Policy**:

  * Modes like `binpack` or `spread` influence node selection.

---

## Summary Table

| Mode        | Isolation        | MIG GPU Required | Annotation | Core/Memory Control | Recommended For            |
| ----------- | ---------------- | ---------------- | ---------- | ------------------- | -------------------------- |
| HAMI-core   | Software (VCUDA) | No               | No         | Yes                 | General workloads          |
| Dynamic MIG | Hardware         | Yes              | Yes        | MIG-controlled      | Performance-sensitive jobs |

---

## Monitoring

* **Scheduler Metrics**:

```bash
curl http://<volcano-scheduler-ip>:8080/metrics
```

* **Device Plugin Metrics**:

```bash
curl http://<plugin-pod-ip>:9394/metrics
```

Metrics include GPU utilization, pod memory usage, and limits.

---

## Issues and Contributions

* File bugs: [Volcano Issues](https://github.com/volcano-sh/volcano/issues)
* Contribute: [Pull Requests Guide](https://help.github.com/articles/using-pull-requests/)
