// Module included in the following assemblies:
//
// * machine_management/
// * machine_management/
// * post_installation_configuration/node-tasks.adoc

:_mod-docs-content-type: PROCEDURE
[id="machine-node-custom-partition_{context}"]
= Adding a new {op-system} worker node with a custom `/var` partition in AWS

[role="_abstract"]
You can add an {op-system} worker node with a custom `/var` partition in AWS by creating a user data secret and a compute machine set that use the required device naming schema.

{product-title} supports partitioning devices during installation by using machine configs that are processed during the bootstrap. However, if you use `/var` partitioning, the device name must be determined at installation and cannot be changed. You cannot add different instance types as nodes if they have a different device naming schema. For example, if you configured the `/var` partition with the default AWS device name for `m4.large` instances, `dev/xvdb`, you cannot directly add an AWS `m5.large` instance, as `m5.large` instances use a `/dev/nvme1n1` device by default. The device might fail to partition due to the different naming schema.

The procedure in this section shows how to add a new {op-system-first} compute node with an instance that uses a different device name from what was configured at installation. You create a custom user data secret and configure a new compute machine set. These steps are specific to an AWS cluster. The principles apply to other cloud deployments also. However, the device naming schema is different for other deployments and should be determined on a per-case basis.

.Procedure

. On a command line, change to the `openshift-machine-api` namespace:
+
[source,terminal]
----
$ oc project openshift-machine-api
----

. Create a new secret from the `worker-user-data` secret:

.. Export the `userData` section of the secret to a text file:
+
[source,terminal]
----
$ oc get secret worker-user-data --template='{{index .data.userData | base64decode}}' | jq > userData.txt
----

.. Edit the text file to add the `storage`, `filesystems`, and `systemd` stanzas for the partitions you want to use for the new node. You can specify any link:https://coreos.github.io/ignition/configuration-v3_2/[Ignition configuration parameters] as needed.
+
[NOTE]
====
Do not change the values in the `ignition` stanza.
====
+
[source,terminal]
----
{
  "ignition": {
    "config": {
      "merge": [
        {
          "source": "https:...."
        }
      ]
    },
    "security": {
      "tls": {
        "certificateAuthorities": [
          {
            "source": "data:text/plain;charset=utf-8;base64,.....=="
          }
        ]
      }
    },
    "version": "3.2.0"
  },
  "storage": {
    "disks": [
      {
        "device": "/dev/nvme1n1", <1>
        "partitions": [
          {
            "label": "var",
            "sizeMiB": 50000, <2>
            "startMiB": 0 <3>
          }
        ]
      }
    ],
    "filesystems": [
      {
        "device": "/dev/disk/by-partlabel/var", <4>
        "format": "xfs", <5>
        "path": "/var" <6>
      }
    ]
  },
  "systemd": {
    "units": [ <7>
      {
        "contents": "[Unit]\nBefore=local-fs.target\n[Mount]\nWhere=/var\nWhat=/dev/disk/by-partlabel/var\nOptions=defaults,pquota\n[Install]\nWantedBy=local-fs.target\n",
        "enabled": true,
        "name": "var.mount"
      }
    ]
  }
}
----
//Copied from installation-disk-partitioning-upi-templates.adoc
<1> Specifies an absolute path to the AWS block device.
<2> Specifies the size of the data partition in Mebibytes.
<3> Specifies the start of the partition in Mebibytes. When adding a data partition to the boot disk, a minimum value of 25000 MB (Mebibytes) is recommended. The root file system is automatically resized to fill all available space up to the specified offset. If no value is specified, or if the specified value is smaller than the recommended minimum, the resulting root file system will be too small, and future reinstalls of {op-system} might overwrite the beginning of the data partition.
<4> Specifies an absolute path to the `/var` partition.
<5> Specifies the filesystem format.
<6> Specifies the mount-point of the filesystem while Ignition is running relative to where the root filesystem will be mounted. This is not necessarily the same as where it should be mounted in the real root, but it is encouraged to make it the same.
<7> Defines a systemd mount unit that mounts the `/dev/disk/by-partlabel/var` device to the `/var` partition.

.. Extract the `disableTemplating` section from the `work-user-data` secret to a text file:
+
[source,terminal]
----
$ oc get secret worker-user-data --template='{{index .data.disableTemplating | base64decode}}' | jq > disableTemplating.txt
----

.. Create the new user data secret file from the two text files. This user data secret passes the additional node partition information in the `userData.txt` file to the newly created node.
+
[source,terminal]
----
$ oc create secret generic worker-user-data-x5 --from-file=userData=userData.txt --from-file=disableTemplating=disableTemplating.txt
----

. Create a new compute machine set for the new node:

.. Create a new compute machine set YAML file, similar to the following, which is configured for AWS. Add the required partitions and the newly-created user data secret:
+
[TIP]
====
Use an existing compute machine set as a template and change the parameters as needed for the new node.
====
+
[source,terminal]
----
apiVersion: machine.openshift.io/v1beta1
kind: MachineSet
metadata:
  labels:
    machine.openshift.io/cluster-api-cluster: auto-52-92tf4
  name: worker-us-east-2-nvme1n1 <1>
  namespace: openshift-machine-api
spec:
  replicas: 1
  selector:
    matchLabels:
      machine.openshift.io/cluster-api-cluster: auto-52-92tf4
      machine.openshift.io/cluster-api-machineset: auto-52-92tf4-worker-us-east-2b
  template:
    metadata:
      labels:
        machine.openshift.io/cluster-api-cluster: auto-52-92tf4
        machine.openshift.io/cluster-api-machine-role: worker
        machine.openshift.io/cluster-api-machine-type: worker
        machine.openshift.io/cluster-api-machineset: auto-52-92tf4-worker-us-east-2b
    spec:
      metadata: {}
      providerSpec:
        value:
          ami:
            id: ami-0c2dbd95931a
          apiVersion: awsproviderconfig.openshift.io/v1beta1
          blockDevices:
          - DeviceName: /dev/nvme1n1 <2>
            ebs:
              encrypted: true
              iops: 0
              volumeSize: 120
              volumeType: gp2
          - DeviceName: /dev/nvme1n2 <3>
            ebs:
              encrypted: true
              iops: 0
              volumeSize: 50
              volumeType: gp2
          credentialsSecret:
            name: aws-cloud-credentials
          deviceIndex: 0
          iamInstanceProfile:
            id: auto-52-92tf4-worker-profile
          instanceType: m6i.large
          kind: AWSMachineProviderConfig
          metadata:
            creationTimestamp: null
          placement:
            availabilityZone: us-east-2b
            region: us-east-2
          securityGroups:
          - filters:
            - name: tag:Name
              values:
              - auto-52-92tf4-worker-sg
          subnet:
            id: subnet-07a90e5db1
          tags:
          - name: kubernetes.io/cluster/auto-52-92tf4
            value: owned
          userDataSecret:
            name: worker-user-data-x5 <4>
----
<1> Specifies a name for the new node.
<2> Specifies an absolute path to the AWS block device, here an encrypted EBS volume.
<3> Optional. Specifies an additional EBS volume.
<4> Specifies the user data secret file.

.. Create the compute machine set:
+
[source,yaml]
----
$ oc create -f <file-name>.yaml
----
+
The machines might take a few moments to become available.

. Verify that the new partition and nodes are created:

.. Verify that the compute machine set is created:
+
[source,terminal]
----
$ oc get machineset
----
+
.Example output
+
[source,terminal]
----
NAME                                               DESIRED   CURRENT   READY   AVAILABLE   AGE
ci-ln-2675bt2-76ef8-bdgsc-worker-us-east-1a        1         1         1       1           124m
ci-ln-2675bt2-76ef8-bdgsc-worker-us-east-1b        2         2         2       2           124m
worker-us-east-2-nvme1n1                           1         1         1       1           2m35s <1>
----
<1> This is the new compute machine set.

.. Verify that the new node is created:
+
[source,terminal]
----
$ oc get nodes
----
+
.Example output
+
[source,terminal]
----
NAME                           STATUS   ROLES    AGE     VERSION
ip-10-0-128-78.ec2.internal    Ready    worker   117m    v1.35.4
ip-10-0-146-113.ec2.internal   Ready    master   127m    v1.35.4
ip-10-0-153-35.ec2.internal    Ready    worker   118m    v1.35.4
ip-10-0-176-58.ec2.internal    Ready    master   126m    v1.35.4
ip-10-0-217-135.ec2.internal   Ready    worker   2m57s   v1.35.4 <1>
ip-10-0-225-248.ec2.internal   Ready    master   127m    v1.35.4
ip-10-0-245-59.ec2.internal    Ready    worker   116m    v1.35.4
----
<1> This is new new node.

.. Verify that the custom `/var` partition is created on the new node:
+
[source,terminal]
----
$ oc debug node/<node-name> -- chroot /host lsblk
----
+
For example:
+
[source,terminal]
----
$ oc debug node/ip-10-0-217-135.ec2.internal -- chroot /host lsblk
----
+
.Example output
+
[source,terminal]
----
NAME        MAJ:MIN  RM  SIZE RO TYPE MOUNTPOINT
nvme0n1     202:0    0   120G  0 disk
|-nvme0n1p1 202:1    0     1M  0 part
|-nvme0n1p2 202:2    0   127M  0 part
|-nvme0n1p3 202:3    0   384M  0 part /boot
`-nvme0n1p4 202:4    0 119.5G  0 part /sysroot
nvme1n1     202:16   0    50G  0 disk
`-nvme1n1p1 202:17   0  48.8G  0 part /var <1>
----
<1> The `nvme1n1` device is mounted to the `/var` partition.
