// Module included in the following assemblies:
//
// networking/multiple_networks/secondary_networks/about-chaining.adoc

:_mod-docs-content-type: PROCEDURE
[id="configuring-plugin-chaining-with-multus-cni_{context}"]
= Configuring plugin chaining with the route-override CNI plugin

[role="_abstract"]
Plugin chaining allows you to configure multiple CNI plugins to be applied sequentially to the same network interface, where each plugin in the chain processes the interface in order.

When you define a `NetworkAttachmentDefinition` (NAD) with a `plugins` array, the first plugin can create the interface, and a second plugin can modify its routing configuration.

The `route-override` CNI plugin is commonly used as the second plugin in a chain to modify the routing configuration of an interface created by the first plugin. It supports the following operations:

* `addroutes:` Add static routes to direct traffic for specific destination networks through the interface.
* `delroutes:` Remove specific routes from the interface.
* `flushroutes:` Remove all routes from the interface.
* `flushgateway:` Remove the default gateway route from the interface.

The following example demonstrates plugin chaining by configuring a pod with two additional network interfaces, each on a separate VLAN with custom routing:

* `eth1` on the `192.168.100.0/24` network (VLAN 100), with a static route directing `10.0.0.0/8` traffic through this interface.
* `eth2` on the `192.168.200.0/24` network (VLAN 200), with a static route directing `172.16.0.0/12` traffic through this interface.

Each interface uses a chain of two plugins: `macvlan` to create the interface on a VLAN, and `route-override` to add static routes that direct specific traffic through that interface.

.Prerequisites

* Install the OpenShift CLI (`oc`).
* An account with `cluster-admin` privileges.

.Procedure

. Create a namespace for the example by running the following command:
+
[source,terminal]
----
$ oc create namespace chain-example
----

. Create the first NetworkAttachmentDefinition (NAD) with a chained plugin configuration.

.. Create a YAML file, such as `management.yaml`, to define a NAD that configures a new interface, `eth1`, on VLAN 100 with the following configuration:
+
[source,yaml]
----
apiVersion: k8s.cni.cncf.io/v1
kind: NetworkAttachmentDefinition
metadata:
  name: management-net
  namespace: chain-example
spec:
  config: '{
    "cniVersion": "1.0.0",
    "name": "management-net",
    "plugins": [
      {
        "type": "macvlan",
        "master": "br-ex",
        "vlan": 100,
        "mode": "bridge",
        "ipam": {
          "type": "static",
          "addresses": [
            {
              "address": "192.168.100.10/24",
              "gateway": "192.168.100.1"
            }
          ]
        }
      },
      {
        "type": "route-override",
        "addroutes": [
          {
            "dst": "10.0.0.0/8",
            "gw": "192.168.100.1"
          }
        ]
      }
    ]
  }'
----

. Create the NAD by running the following command:
+
[source,terminal]
----
$ oc apply -f management.yaml
----

. Create the second NAD with a chained plugin configuration.

.. Create a YAML file, such as `sip.yaml`, to define a NAD that configures a new interface, `eth2`, on VLAN 200 with the following configuration:
+
[source,yaml]
----
apiVersion: k8s.cni.cncf.io/v1
kind: NetworkAttachmentDefinition
metadata:
  name: sip-net
  namespace: chain-example
spec:
  config: '{
    "cniVersion": "1.0.0",
    "name": "sip-net",
    "plugins": [
      {
        "type": "macvlan",
        "master": "br-ex",
        "vlan": 200,
        "mode": "bridge",
        "ipam": {
          "type": "static",
          "addresses": [
            {
              "address": "192.168.200.10/24",
              "gateway": "192.168.200.1"
            }
          ]
        }
      },
      {
        "type": "route-override",
        "addroutes": [
          {
            "dst": "172.16.0.0/12",
            "gw": "192.168.200.1"
          }
        ]
      }
    ]
  }'
----

. Create the NAD by running the following command:
+
[source,terminal]
----
$ oc apply -f sip.yaml
----

. Attach the `NetworkAttachmentDefinition` resources to a pod by creating a pod definition file, such as `pod.yaml`, with the following configuration:
+
[source,yaml]
----
apiVersion: v1
kind: Pod
metadata:
  name: chain-test-pod
  namespace: chain-example
  labels:
    app: chain-test
  annotations:
    k8s.v1.cni.cncf.io/networks: '[
      { "name": "management-net", "interface": "eth1" },
      { "name": "sip-net", "interface": "eth2" }
    ]'
spec:
  securityContext:
    runAsNonRoot: true
    seccompProfile:
      type: RuntimeDefault
  containers:
  - name: test-container
    image: registry.access.redhat.com/ubi9/ubi:latest
    command: ["sleep", "infinity"]
    securityContext:
      allowPrivilegeEscalation: false
      capabilities:
        drop: ["ALL"]
----

. Create the pod by running the following command:
+
[source,terminal]
----
$ oc apply -f pod.yaml
----

. Verify the pod is running with the following command:
+
[source,terminal]
----
$ oc wait --for=condition=Ready pod/chain-test-pod -n chain-example --timeout=120s
----
+
Example output:
+
[source,terminal]
----
pod/chain-test-pod condition met
----

.Verification

. Run the following command to list all network interfaces and their assigned IP addresses inside the pod. This verifies that the pod has the additional interfaces configured by plugin chaining:
+
[source,terminal]
----
$ oc exec chain-test-pod -n chain-example -- ip a
----
+
Example output:
+
[source,terminal]
----
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0@if31: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 8901 qdisc noqueue state UP
    link/ether 0a:58:0a:83:02:19 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 10.131.2.25/23 brd 10.131.3.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::858:aff:fe83:219/64 scope link
       valid_lft forever preferred_lft forever
3: eth1@if5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc noqueue state UP qlen 1000
    link/ether aa:25:73:ff:a7:00 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 192.168.100.10/24 brd 192.168.100.255 scope global eth1
       valid_lft forever preferred_lft forever
    inet6 fe80::a825:73ff:feff:a700/64 scope link
       valid_lft forever preferred_lft forever
4: eth2@if5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc noqueue state UP qlen 1000
    link/ether aa:a4:6c:4e:e8:97 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 192.168.200.10/24 brd 192.168.200.255 scope global eth2
       valid_lft forever preferred_lft forever
    inet6 fe80::a8a4:6cff:fe4e:e897/64 scope link
       valid_lft forever preferred_lft forever
----
+
This output shows the pod has three network interfaces:

* `eth0:` The default interface, connected to the cluster network.
* `eth1:` The first additional interface from `management-net`, with IP `192.168.100.10`.
* `eth2:` The second additional interface from `sip-net`, with IP `192.168.200.10`.

. Run the following command to verify that the `route-override` plugin added the expected static routes:
+
[source,terminal]
----
$ oc exec chain-test-pod -n chain-example -- ip route
----
+
Example output:
+
[source,terminal]
----
default via 10.132.0.1 dev eth0
10.0.0.0/8 via 192.168.100.1 dev eth1
10.132.0.0/23 dev eth0 proto kernel scope link src 10.132.1.97
10.132.0.0/14 via 10.132.0.1 dev eth0
100.64.0.0/16 via 10.132.0.1 dev eth0
169.254.0.5 via 10.132.0.1 dev eth0
172.16.0.0/12 via 192.168.200.1 dev eth2
172.30.0.0/16 via 10.132.0.1 dev eth0
192.168.100.0/24 dev eth1 proto kernel scope link src 192.168.100.10
192.168.200.0/24 dev eth2 proto kernel scope link src 192.168.200.10
----
+
This output confirms that the `route-override` plugin in each chain added the expected static routes:

* For `10.0.0.0/8 via 192.168.100.1 dev eth1`, traffic destined for `10.0.0.0/8` is routed through `eth1` via the `management-net` gateway. This route was added by the `route-override` plugin in the `management-net` chain.
* For `172.16.0.0/12 via 192.168.200.1 dev eth2`, traffic destined for `172.16.0.0/12` is routed through `eth2` via the `sip-net` gateway. This route was added by the `route-override` plugin in the `sip-net` chain.
* The connected subnet routes (`192.168.100.0/24` and `192.168.200.0/24`) were created by the `macvlan` plugin, while the default route uses `eth0`, the cluster network interface.
