// Module included in the following assemblies:
//
// * networking/hardware_networks/configuring-sriov-rdma-cni.adoc

:_mod-docs-content-type: PROCEDURE
[id="nw-sriov-configuring-sriov-rdma-cni_{context}"]
= Configuring SR-IOV RDMA CNI

[role="_abstract"]
Configure an RDMA CNI on SR-IOV.

[NOTE]
====
This procedure applies only to Mellanox devices.
====

.Prerequisites

* You have installed the OpenShift CLI (`oc`).
* You have access to the cluster as a user with the `cluster-admin` role.
* You have installed the SR-IOV Network Operator.

.Procedure

. Create an `SriovNetworkPoolConfig` CR and save it as `sriov-nw-pool.yaml`, as shown in the following example:
+
.Example `SriovNetworkPoolConfig` CR
[source,yaml]
----
apiVersion: sriovnetwork.openshift.io/v1
kind: SriovNetworkPoolConfig
metadata:
  name: worker
  namespace: openshift-sriov-network-operator
spec:
  maxUnavailable: 1 
  nodeSelector:
    matchLabels:
      node-role.kubernetes.io/worker: ""
  rdmaMode: <rdma_mode>
----
+
--
`spec.rdmaMode`:: Specifies the RDMA network namespace mode. Set to `exclusive`.
--

. Create the `SriovNetworkPoolConfig` resource by running the following command:
+
[source,terminal]
----
$ oc create -f sriov-nw-pool.yaml
----

. Create an `SriovNetworkNodePolicy` CR and save it as `sriov-node-policy.yaml`, as shown in the following example: 
+
.Example `SriovNetworkNodePolicy` CR
[source,yaml]
----
apiVersion: sriovnetwork.openshift.io/v1
kind: SriovNetworkNodePolicy
metadata:
  name: sriov-nic-pf1
  namespace: openshift-sriov-network-operator
spec:
  deviceType: netdevice
  isRdma: <is_rdma>
  nicSelector:
    pfNames: ["ens3f0np0"]
  nodeSelector:
    node-role.kubernetes.io/worker: ""
  numVfs: 4
  priority: 99
  resourceName: sriov_nic_pf1
----
+
--
`spec.isRdma`:: Specifies whether to activate RDMA mode. Set to `true`.
--

. Create the `SriovNetworkNodePolicy` resource by running the following command:
+
[source,terminal]
----
$ oc create -f sriov-node-policy.yaml
----

. Create an `SriovNetwork` CR and save it as `sriov-network.yaml`, as shown in the following example: 
+
.Example `SriovNetwork` CR
[source,yaml]
----
apiVersion: sriovnetwork.openshift.io/v1
kind: SriovNetwork
metadata:
  name: sriov-nic-pf1
  namespace: openshift-sriov-network-operator
spec:
  networkNamespace: sriov-tests
  resourceName: sriov_nic_pf1
    ipam: |- 
  metaPlugins: |
    {
      "type": "<plugin_type>"
    }
----
+
--
`spec.metaPlugins.type`:: Specifies the type of meta plugin. Set to `rdma` to create the RDMA plugin.
--

. Create the `SriovNetwork` resource by running the following command:
+
[source,terminal]
----
$ oc create -f sriov-network.yaml
----

.Verification

. Create a  `Pod` CR and save it as `sriov-test-pod.yaml`, as shown in the following example: 
+
.Example runtime configuration
[source,yaml]
----
apiVersion: v1
kind: Pod
metadata:
  name: sample-pod
  annotations:
    k8s.v1.cni.cncf.io/networks: |-
      [
        {
          "name": "net1",
          "mac": "20:04:0f:f1:88:01",
          "ips": ["192.168.10.1/24", "2001::1/64"]
        }
      ]
spec:
  containers:
  - name: sample-container
    image: <image>
    imagePullPolicy: IfNotPresent
    command: ["sleep", "infinity"]
----

. Create the test pod by running the following command:
+
[source,terminal]
----
$ oc create -f sriov-test-pod.yaml
----

. Log in to the test pod by running the following command:
+
[source,terminal]
----
$ oc rsh testpod1 -n sriov-tests
----

. Verify that the path to the `hw-counters` directory exists by running the following command:
+
[source,terminal]
----
$ ls /sys/bus/pci/devices/${PCIDEVICE_OPENSHIFT_IO_SRIOV_NIC_PF1}/infiniband/*/ports/1/hw_counters/
----
+
.Example output
[source,terminal]
----
duplicate_request       out_of_buffer req_cqe_flush_error           resp_cqe_flush_error        roce_adp_retrans        roce_slow_restart_trans
implied_nak_seq_err     out_of_sequence req_remote_access_errors    resp_local_length_error     roce_adp_retrans_to     rx_atomic_requests
lifespan                packet_seq_err req_remote_invalid_request   resp_remote_access_errors   roce_slow_restart       rx_read_requests
local_ack_timeout_err  req_cqe_error resp_cqe_error                 rnr_nak_retry_err           roce_slow_restart_cnps  rx_write_requests
----