// Module included in the following assemblies:
//
// * machine_configuration/mco-coreos-layering.adoc

:_mod-docs-content-type: PROCEDURE
[id="coreos-layering-configuring-on-proc_{context}"]
= Using the {image-mode-os-on-lower} to apply a custom layered image

[role="_abstract"]
You can use the on-cluster build process to apply a custom layered image to your cluster by creating a `MachineOSConfig` custom resource (CR). 

The `MachineOSConfig` CR specifies the following parameters: 

* the Containerfile to build
* the machine config pool to associate the build
* where the final image should be pushed and pulled from
* the push and pull secrets to use

You can create only one `MachineOSConfig` CR for each machine config pool.

.Prerequisites

* You have the pull secret in the `openshift-machine-config-operator` namespace that the Machine Config Operator (MCO) needs in order to pull the base operating system image from your repository. By default, the MCO uses the cluster global pull secret, which it synchronizes into the `openshift-machine-config-operator` namespace. You can add your pull secret to the {product-title} global pull secret or you can use a different pull secret. For information on modifying the global pull secret, see "Updating the global cluster pull secret".

* You have the push secret of the registry that the MCO needs to push the new custom layered image to. The credentials provided by the secret must also grant permission to delete an image from the registry.
+
[NOTE]
====
In a disconnected environment, ensure that the disconnected cluster can access the registry where you want to push the image. Image mirroring applies only to pulling images.
====

* You have the pull secret that your nodes need to pull the new custom layered image from your registry. This should be a different secret than the one used to push the image to the repository.

* You are familiar with how to configure a Containerfile. Instructions on how to create a Containerfile are beyond the scope of this documentation.

* Optional: You have a separate machine config pool for the nodes where you want to apply the custom layered image. One benefit to having a custom machine config pool for the nodes it that you can easily revert to the base image, if needed. For more information, see "Reverting an on-cluster layered node".

.Procedure

. Create a `MachineOSconfig` object:

.. Create a YAML file similar to the following:
+
[source,yaml]
----
apiVersion: machineconfiguration.openshift.io/v1
kind: MachineOSConfig
metadata:
  name: layered-image
spec:
  machineConfigPool:
    name: layered-image
  containerFile:
  - containerfileArch: NoArch
    content: |-
      FROM configs AS final
      RUN dnf install -y cowsay && \
        dnf clean all && \
        bootc container lint
  imageBuilder:
    imageBuilderType: Job
  baseImagePullSecret:
    name: global-pull-secret-copy
  renderedImagePushSpec: image-registry.openshift-image-registry.svc:5000/openshift/os-image:latest
  renderedImagePushSecret:
    name: builder-dockercfg-mtcl23
----
+
where:

`apiVersion`:: Specifies the `machineconfiguration.openshift.io/v1` API that is required for `MachineConfig` CRs.
`metadata.name`:: Specifies a name for the `MachineOSConfig` object. The name must match the name of the associated machine config pool. This name is used with other {image-mode-os-on-lower} resources. The examples in this documentation use the name `layered-image`. 
`spec.machineConfigPool.name`:: Specifies the name of the machine config pool associated with the nodes where you want to deploy the custom layered image. The examples in this documentation use the `layered-image` machine config pool.
`spec.containerFile`:: Specifies the Containerfile to configure the custom layered image.
`spec.containerFile.containerfileArch`:: Specifies the architecture this containerfile is to be built for: `ARM64`, `AMD64`, `PPC64LE`, `S390X`, or `NoArch`. The default is `NoArch`, which defines a Containerfile that can be applied to any architecture. 
`spec.imageBuilder`:: Specifies the name of the image builder to use. This must be `Job`, which is a reference to the `job` object that is managing the image build.
`spec.baseImagePullSecret`:: Specifies the name of the pull secret that the MCO needs to pull the base operating system image from the registry. By default, the global pull secret is used. This parameter is optional.
`spec.renderedImagePushSpec`:: Specifies the image registry to push the newly-built custom layered image to. This can be any registry that your cluster has access to in the `host[:port][/namespace]/name` or `svc_name.namespace.svc[:port]/repository/name:<tag>` format. This example uses the internal {product-title} registry. You can specify a mirror registry if you cluster is properly configured to use a mirror registry.
`spec.renderedImagePushSecret`:: Specifies the name of the push secret that the MCO needs to push the newly-built custom layered image to that registry.

.. Create the `MachineOSConfig` object:
+
[source,terminal]
----
$ oc create -f <filename>.yaml
----

. If necessary, when the `MachineOSBuild` object has been created and is in the `READY` state, modify the node spec for the nodes where you want to use the new custom layered image:
+
.. Check that the `MachineOSBuild` object is ready, by running the following command:
+
[source,terminal]
----
$ oc get machineosbuild
----
+
When the `SUCCEEDED` value is `True`, the build is complete:
+
.Example output showing that the `MachineOSBuild` object is ready
[source,terminal]
----
NAME                                                     PREPARED   BUILDING   SUCCEEDED   INTERRUPTED   FAILED   AGE
layered-image-ad5a3cad36303c363cf458ab0524e7c0-builder   False      False      True        False         False    43s
----

.. Edit the nodes where you want to deploy the custom layered image by adding a label for the machine config pool you specified in the `MachineOSConfig` object:
+
[source,terminal]
----
$ oc label node <node_name> 'node-role.kubernetes.io/<mcp_name>='
----
+
--
where:

node-role.kubernetes.io/<mcp_name>=:: Specifies a node selector that identifies the nodes to deploy the custom layered image. 
--
+
When you save the changes, the MCO drains, cordons, and reboots the nodes. After the reboot, the node uses the new custom layered image.

.Verification

. Verify that the new pods are ready by running the following command:
+
[source,terminal]
----
$ oc get pods -n openshift-machine-config-operator
----
+
.Example output
[source,terminal]
----
NAME                                                                    READY   STATUS    RESTARTS   AGE
build-layered-image-ad5a3cad36303c363cf458ab0524e7c0-hxrws              2/2     Running   0          2m40s
# ...
machine-os-builder-6fb66cfb99-zcpvq                                     1/1     Running   0          2m42s
----
+
The build pod where the custom layered image is building is named in the `build-<MachineOSConfig_CR_name>-<hash>` format. The `machine-os-builder-*` pod can be used for troubleshooting.

. Verify the custom layered image build by running a command similar to the following:
+
[source,terminal]
----
$ oc get machineconfigpool <mcp_name> -o yaml
----
+
.Example output
[source,terminal]
----
apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfigPool
metadata:
  labels:
    machineconfiguration.openshift.io/mco-built-in: ""
    pools.operator.machineconfiguration.openshift.io/layered: ""
  name: layered
# ...
status:
# ...
  conditions
# ...
  - lastTransitionTime: "2025-09-09T13:43:35Z"
    message: 'Failed to build OS image for pool worker (MachineOSBuild: worker-2d03dc921ff0c242c5892a3ef1ed1608):
      Failed: Build Failed'
    reason: BuildFailed
    status: "True"
    type: ImageBuildDegraded
----
+
The `reason: BuildFailed` status indicates whether the custom layered image build failed. If `False`, the build succeeded. If `True`, the build failed. You can use the build pod logs to troubleshoot any build failures.

. Verify the current stage of your layered build by running the following command:
+
[source,terminal]
----
$ oc get machineosbuilds
----
+
.Example output
[source,terminal]
----
NAME                                             PREPARED   BUILDING   SUCCEEDED   INTERRUPTED   FAILED   AGE
layered-image-ad5a3cad36303c363cf458ab0524e7c0   False      True       False       False         False    12m
----
+
The `MachineOSBuild` object is named in the `<MachineOSConfig_CR_name>-<hash>` format.
+
The build is complete when `BUILDING` is `False` and `SUCCEEDED` is `True`.

. When the build is complete, verify that the image has been applied to the nodes in the affected pool by running a command similar to the following:
+
[source,terminal]
----
$ oc describe machineconfignode/<machine_config_node_name>
----
+
--
include::snippets/mco-mcn-ocl-example.adoc[]
--
+
--
:FeatureName: The `ImagePulledFromRegistry` condition
include::snippets/technology-preview.adoc[]
--

. Verify that the `MachineOSConfig` object contains a reference to the new custom layered image by running the following command:
+
[source,terminal]
----
$ oc describe machineosconfig <object_name>
----
+
.Example digested image pull spec
[source,terminal]
----
apiVersion: machineconfiguration.openshift.io/v1
kind: MachineOSConfig
metadata:
  annotations:
    machineconfiguration.openshift.io/current-machine-os-build: layered-9a8f89455246fa0c42ecee6ff1fa1a45
  labels:
    machineconfiguration.openshift.io/createdByOnClusterBuildsHelper: ""
  name: layered-image
# ...
status:
  currentImagePullSpec: image-registry.openshift-image-registry.svc:5000/openshift-machine-config-operator/os-image@sha256:3c8fc667adcb432ce0c83581f16086afec08a961dd28fed69bb6bad6db0a0754
----
+
where:

`status.currentImagePullSpec`:: Specifies the digested image pull spec for the new custom layered image.

. Verify that the appropriate nodes are using the new custom layered image:

.. Start a debug session as root for a control plane node by running the following command:
+
[source,terminal]
----
$ oc debug node/<node_name>
----

.. Set `/host` as the root directory within the debug shell:
+
[source,terminal]
----
sh-4.4# chroot /host
----

.. Run the `rpm-ostree status` command to view that the custom layered image is in use:
+
[source,terminal]
----
sh-5.1# rpm-ostree status
----
+
.Example output
[source,terminal]
----
# ...
Deployments:
* ostree-unverified-registry:image-registry.openshift-image-registry.svc:5000/openshift-machine-config-operator/os-images@sha256:3c8fc667adcb432ce0c83581f16086afec08a961dd28fed69bb6bad6db0a0754
                   Digest: sha256:3c8fc667adcb432ce0c83581f16086afec08a961dd28fed69bb6bad6db0a0754
                  Version: 419.94.202502100215-0 (2025-02-12T19:20:44Z)
----
+
The `Deployments` stanza includes the digested image pull spec for the new custom layered image.
