// Module included in the following assemblies:
//
// * hardware_accelerators/rdma-remote-direct-memory-access.adoc

:_mod-docs-content-type: PROCEDURE
[id="rdma-configuring-the-nvidia-network-operator_{context}"]

= Configuring the NVIDIA network Operator

The NVIDIA network Operator manages NVIDIA networking resources and networking related components such as drivers and device plugins to enable NVIDIA GPUDirect RDMA workloads.

.Prerequisites

* You have installed the NVIDIA network Operator.

.Procedure 

. Validate that the network Operator is installed and running by confirming the controller is running in the `nvidia-network-operator` namespace by running the following command:
+
[source,terminal]
----
$ oc get pods -n nvidia-network-operator
----
+
.Example output
[source,terminal]
----
NAME                                                          READY   STATUS             RESTARTS        AGE
nvidia-network-operator-controller-manager-6f7d6956cd-fw5wg   1/1     Running            0                5m
----

. With the Operator running, create the `NicClusterPolicy` custom resource file. The device you choose depends on your system configuration. In this example, the Infiniband interface `ibs2f0` is hard coded and is used as the shared NVIDIA GPUDirect RDMA device. 
+
[source,yaml]
----
apiVersion: mellanox.com/v1alpha1
kind: NicClusterPolicy
metadata:
  name: nic-cluster-policy
spec:
  nicFeatureDiscovery:
    image: nic-feature-discovery
    repository: ghcr.io/mellanox
    version: v0.0.1
  docaTelemetryService:
    image: doca_telemetry
    repository: nvcr.io/nvidia/doca
    version: 1.16.5-doca2.6.0-host
  rdmaSharedDevicePlugin:
    config: |
      {
        "configList": [
          {
            "resourceName": "rdma_shared_device_ib",
            "rdmaHcaMax": 63,
            "selectors": {
              "ifNames": ["ibs2f0"]
            }
          },
          {
            "resourceName": "rdma_shared_device_eth",
            "rdmaHcaMax": 63,
            "selectors": {
              "ifNames": ["ens8f0np0"]
            }
          }
        ]
      }
    image: k8s-rdma-shared-dev-plugin
    repository: ghcr.io/mellanox
    version: v1.5.1
  secondaryNetwork:
    ipoib:
      image: ipoib-cni
      repository: ghcr.io/mellanox
      version: v1.2.0
  nvIpam:
    enableWebhook: false
    image: nvidia-k8s-ipam
    repository: ghcr.io/mellanox
    version: v0.2.0
  ofedDriver:
    readinessProbe:
      initialDelaySeconds: 10
      periodSeconds: 30
    forcePrecompiled: false
    terminationGracePeriodSeconds: 300
    livenessProbe:
      initialDelaySeconds: 30
      periodSeconds: 30
    upgradePolicy:
      autoUpgrade: true
      drain:
        deleteEmptyDir: true
        enable: true
        force: true
        timeoutSeconds: 300
        podSelector: ''
      maxParallelUpgrades: 1
      safeLoad: false
      waitForCompletion:
        timeoutSeconds: 0
    startupProbe:
      initialDelaySeconds: 10
      periodSeconds: 20
    image: doca-driver
    repository: nvcr.io/nvidia/mellanox
    version: 24.10-0.7.0.0-0
    env:
    - name: UNLOAD_STORAGE_MODULES
      value: "true"
    - name: RESTORE_DRIVER_ON_POD_TERMINATION
      value: "true"
    - name: CREATE_IFNAMES_UDEV
      value: "true"
----

. Create the `NicClusterPolicy` custom resource on the cluster by running the following command:
+
[source,terminal]
----
$ oc create -f network-sharedrdma-nic-cluster-policy.yaml 
----
+
.Example output
[source,terminal]
----
nicclusterpolicy.mellanox.com/nic-cluster-policy created
----

. Validate the `NicClusterPolicy` by running the following command in the DOCA/MOFED container:
+
[source,terminal]
----
$ oc get pods -n nvidia-network-operator
----
+
.Example output
[source,terminal]
----
NAME                                                          READY   STATUS    RESTARTS   AGE
doca-telemetry-service-hwj65                                  1/1     Running   2          160m
kube-ipoib-cni-ds-fsn8g                                       1/1     Running   2          160m
mofed-rhcos4.16-9b5ddf4c6-ds-ct2h5                            2/2     Running   4          160m
nic-feature-discovery-ds-dtksz                                1/1     Running   2          160m
nv-ipam-controller-854585f594-c5jpp                           1/1     Running   2          160m
nv-ipam-controller-854585f594-xrnp5                           1/1     Running   2          160m
nv-ipam-node-xqttl                                            1/1     Running   2          160m
nvidia-network-operator-controller-manager-5798b564cd-5cq99   1/1     Running   2         5d23h
rdma-shared-dp-ds-p9vvg                                       1/1     Running   0          85m
----

. `rsh` into the `mofed` container to check the status by running the following command:
+
[source,terminal]
----
$ MOFED_POD=$(oc get pods -n nvidia-network-operator -o name | grep mofed)
$ oc rsh -n nvidia-network-operator -c mofed-container ${MOFED_POD}
sh-5.1# ofed_info -s
----
+
.Example output
[source,terminal]
----
OFED-internal-24.07-0.6.1:
----
+
[source,terminal]
----
sh-5.1# ibdev2netdev -v
----
+
.Example output
[source,terminal]
----
0000:0d:00.0 mlx5_0 (MT41692 - 900-9D3B4-00EN-EA0) BlueField-3 E-series SuperNIC 400GbE/NDR single port QSFP112, PCIe Gen5.0 x16 FHHL, Crypto Enabled, 16GB DDR5, BMC, Tall Bracket                                                       fw 32.42.1000 port 1 (ACTIVE) ==> ibs2f0 (Up)
0000:a0:00.0 mlx5_1 (MT41692 - 900-9D3B4-00EN-EA0) BlueField-3 E-series SuperNIC 400GbE/NDR single port QSFP112, PCIe Gen5.0 x16 FHHL, Crypto Enabled, 16GB DDR5, BMC, Tall Bracket                                                       fw 32.42.1000 port 1 (ACTIVE) ==> ens8f0np0 (Up)
----

. Create a `IPoIBNetwork` custom resource file:
+
[source,yaml]
----
apiVersion: mellanox.com/v1alpha1
kind: IPoIBNetwork
metadata:
  name: example-ipoibnetwork
spec:
  ipam: |
    {
      "type": "whereabouts",
      "range": "192.168.6.225/28",
      "exclude": [
       "192.168.6.229/30",
       "192.168.6.236/32"
      ]
    }
  master: ibs2f0
  networkNamespace: default
----

. Create the `IPoIBNetwork` resource on the cluster by running the following command:
+
[source,terminal]
----
$ oc create -f ipoib-network.yaml 
----
+
.Example output
[source,terminal]
----
ipoibnetwork.mellanox.com/example-ipoibnetwork created
----

. Create a `MacvlanNetwork` custom resource file for your other interface: 
+
[source,yaml]
----
apiVersion: mellanox.com/v1alpha1
kind: MacvlanNetwork
metadata:
  name: rdmashared-net
spec:
  networkNamespace: default
  master: ens8f0np0
  mode: bridge
  mtu: 1500
  ipam: '{"type": "whereabouts", "range": "192.168.2.0/24", "gateway": "192.168.2.1"}'
----

. Create the resource on the cluster by running the following command:
+
[source,terminal]
----
$ oc create -f macvlan-network.yaml 
----
+
.Example output
[source,terminal]
----
macvlannetwork.mellanox.com/rdmashared-net created
----
