// Module included in the following assemblies:
//
// * machine_configuration/coreos-layering.adoc

:_mod-docs-content-type: PROCEDURE
[id="coreos-layering-configuring_{context}"]
= Using {image-mode-os-out-caps} to apply a custom layered image

[role="_abstract"]
You can use the {image-mode-os-lower} out-of-cluster build process to apply a custom layered image to your nodes by creating a `MachineOSConfig` custom resource (CR). 

When you create the object, the Machine Config Operator (MCO) reboots those nodes with the new custom layered image, overriding the base {op-system-first} image.

To apply a custom layered image to your cluster, you must have the custom layered image in a repository that your cluster can access. Then, create a `MachineConfig` object that points to the custom layered image. You need a separate `MachineConfig` object for each machine config pool that you want to configure.

[IMPORTANT]
====
As soon as you apply an out-of-cluster custom image to your cluster, you effectively _take ownership_ of your custom layered images and those nodes. {product-title} no longer automatically updates any node that uses the custom layered image. You become responsible for maintaining and updating your nodes as appropriate. If you roll back the custom layer, {product-title} resumes automatically updating the node. See the "Updating with a RHCOS custom layered image" for important information about updating nodes that use a custom layered image.
====

.Prerequisites

* You must create a custom layered image that is based on an {product-title} image digest, not a tag.
+
[NOTE]
====
You should use the same base {op-system} image that is installed on the rest of your cluster. Use the `oc adm release info --image-for rhel-coreos` command to obtain the base image being used in your cluster.
====
+
For example, the following Containerfile creates a custom layered image from an {product-title} {product-version} image and overrides the kernel package with one from CentOS 9 Stream:
+
.Example Containerfile for a custom layer image
[source,yaml,subs="+attributes"]
----
# Using a {product-version}.0 image
FROM quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256...
#Install hotfix rpm
RUN rpm-ostree override replace http://mirror.stream.centos.org/9-stream/BaseOS/x86_64/os/Packages/kernel-{,core-,modules-,modules-core-,modules-extra-}5.14.0-295.el9.x86_64.rpm && \
    rpm-ostree cleanup -m && \
    bootc container lint
----
+
where:
+
--
`FROM`:: Specifies the {op-system} base image of your cluster.
`RUN`:: Replaces the kernel packages.
--
+
[NOTE]
====
Instructions on how to create a Containerfile are beyond the scope of this documentation.
====

* Because the process for building a custom layered image is performed outside of the cluster, you must use the `--authfile /path/to/pull-secret` option with Podman or Buildah. Alternatively, to have the pull secret read by these tools automatically, you can add it to one of the default file locations: `~/.docker/config.json`, `$XDG_RUNTIME_DIR/containers/auth.json`, `~/.docker/config.json`, or `~/.dockercfg`. Refer to the `containers-auth.json` man page for more information.

* You must push the custom layered image to a repository that your cluster can access.

.Procedure

. Create a machine config file.

.. Create a YAML file similar to the following:
+
[source,yaml]
----
apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfig
metadata:
  labels:
    machineconfiguration.openshift.io/role: worker
  name: os-layer-custom
spec:
  osImageURL: quay.io/my-registry/custom-image@sha256...
----
+
where:

`metadata.labels`:: Specifies the machine config pool to deploy the custom layered image.
`spec.osImageURL`:: Specifies the path to the custom layered image in the repository.

.. Create the `MachineConfig` object:
+
[source,terminal]
----
$ oc create -f <file_name>.yaml
----
+
[IMPORTANT]
====
It is strongly recommended that you test your images outside of your production environment before rolling out to your cluster.
====

.Verification

You can verify that the custom layered image is applied by performing any of the following checks:

. Check that the worker machine config pool has rolled out with the new machine config:

.. Check that the new machine config is created:
+
[source,terminal]
----
$ oc get mc
----
+
.Sample output
[source,terminal]
----
NAME                                               GENERATEDBYCONTROLLER                      IGNITIONVERSION   AGE
00-master                                          5bdb57489b720096ef912f738b46330a8f577803   3.5.0             95m
00-worker                                          5bdb57489b720096ef912f738b46330a8f577803   3.5.0             95m
01-master-container-runtime                        5bdb57489b720096ef912f738b46330a8f577803   3.5.0             95m
01-master-kubelet                                  5bdb57489b720096ef912f738b46330a8f577803   3.5.0             95m
01-worker-container-runtime                        5bdb57489b720096ef912f738b46330a8f577803   3.5.0             95m
01-worker-kubelet                                  5bdb57489b720096ef912f738b46330a8f577803   3.5.0             95m
99-master-generated-registries                     5bdb57489b720096ef912f738b46330a8f577803   3.5.0             95m
99-master-ssh                                                                                 3.2.0             98m
99-worker-generated-registries                     5bdb57489b720096ef912f738b46330a8f577803   3.5.0             95m
99-worker-ssh                                                                                 3.2.0             98m
os-layer-custom                                                                                                 10s
rendered-master-15961f1da260f7be141006404d17d39b   5bdb57489b720096ef912f738b46330a8f577803   3.5.0             95m
rendered-worker-5aff604cb1381a4fe07feaf1595a797e   5bdb57489b720096ef912f738b46330a8f577803   3.5.0             95m
rendered-worker-5de4837625b1cbc237de6b22bc0bc873   5bdb57489b720096ef912f738b46330a8f577803   3.5.0             4s
----
+
The `os-layer-custom` object is the newly created machine config. The `rendered-worker-5de4837625b1cbc237de6b22bc0bc873` object is the newly created rendered machine config.

.. Check that the `osImageURL` value in the new machine config points to the expected image:
+
[source,terminal]
----
$ oc describe mc rendered-worker-5de4837625b1cbc237de6b22bc0bc873
----
+
.Example output
[source,terminal,subs="attributes+"]
----
Name:         rendered-worker-5de4837625b1cbc237de6b22bc0bc873
Namespace:
Labels:       <none>
Annotations:  machineconfiguration.openshift.io/generated-by-controller-version: 5bdb57489b720096ef912f738b46330a8f577803
              machineconfiguration.openshift.io/release-image-version: {product-version}.0-ec.3
API Version:  machineconfiguration.openshift.io/v1
Kind:         MachineConfig
...
  Os Image URL: quay.io/my-registry/custom-image@sha256...
----

.. Check that the associated machine config pool is updated with the new machine config:
+
[source,terminal]
----
$ oc get mcp
----
+
.Sample output
[source,terminal]
----
NAME     CONFIG                                             UPDATED   UPDATING   DEGRADED   MACHINECOUNT   READYMACHINECOUNT   UPDATEDMACHINECOUNT   DEGRADEDMACHINECOUNT   AGE
master   rendered-master-15961f1da260f7be141006404d17d39b   True      False      False      3              3                   3                     0                      39m
worker   rendered-worker-5de4837625b1cbc237de6b22bc0bc873   True      False      False      3              0                   0                     0                      39m
----
+
When the `UPDATING` field is `True`, the machine config pool is updating with the new machine config. In this case, you will not see the new machine config listed in the output. When the field becomes `False`, the worker machine config pool has rolled out to the new machine config.

.. Check the nodes to see that scheduling on the nodes is disabled. This indicates that the change is being applied:
+
[source,terminal]
----
$ oc get nodes
----
+
.Example output
[source,terminal]
----
NAME                                         STATUS                     ROLES                  AGE   VERSION
ip-10-0-148-79.us-west-1.compute.internal    Ready                      worker                 32m   v1.35.4
ip-10-0-155-125.us-west-1.compute.internal   Ready,SchedulingDisabled   worker                 35m   v1.35.4
ip-10-0-170-47.us-west-1.compute.internal    Ready                      control-plane,master   42m   v1.35.4
ip-10-0-174-77.us-west-1.compute.internal    Ready                      control-plane,master   42m   v1.35.4
ip-10-0-211-49.us-west-1.compute.internal    Ready                      control-plane,master   42m   v1.35.4
ip-10-0-218-151.us-west-1.compute.internal   Ready                      worker                 31m   v1.35.4
----

. When the node is back in the `Ready` state, check that the node is using the custom layered image:

.. Open an `oc debug` session to the node. For example:
+
[source,terminal]
----
$ oc debug node/ip-10-0-155-125.us-west-1.compute.internal
----

.. Set `/host` as the root directory within the debug shell:
+
[source,terminal]
----
sh-4.4# chroot /host
----

.. Run the `rpm-ostree status` command to view that the custom layered image is in use:
+
[source,terminal]
----
sh-4.4# sudo rpm-ostree status
----
+
.Example output
----
State: idle
Deployments:
* ostree-unverified-registry:quay.io/my-registry/...
                   Digest: sha256:...
----

