// Module included in the following assemblies:
//
// * machine_configuration/mco-coreos-layering.adoc

:_mod-docs-content-type: CONCEPT
[id="coreos-layering-configuring-on_{context}"]
= About {image-mode-os-on-lower}

[role="_abstract"]
You can use the {image-mode-os-lower} on-cluster build process to apply a custom layered image to your nodes by creating a `MachineOSConfig` custom resource (CR).

When you create the object, the Machine Config Operator (MCO) creates a `MachineOSBuild` object and a builder pod. The process also creates transient objects, such as config maps, which are cleaned up after the build is complete. The `MachineOSBuild` object and the associated `builder-*` pod use the same naming scheme, `<MachineOSConfig_CR_name>-<hash>`, for example:

.Example `MachineOSBuild` object
[source,terminal]
----
NAME                                             PREPARED   BUILDING   SUCCEEDED   INTERRUPTED   FAILED
layered-image-c8765e26ebc87e1e17a7d6e0a78e8bae   False      False      True        False         False
----

.Example builder pod
[source,terminal]
----
NAME                                                      READY   STATUS      RESTARTS        AGE
build-layered-image-c8765e26ebc87e1e17a7d6e0a78e8bae      2/2     Running     0               11m
----

You should not need to interact with these new objects or the `machine-os-builder` pod. However, you can use all of these resources for troubleshooting, if necessary.

When the build is complete, the MCO pushes the new custom layered image to your repository and rolls the image out to the nodes in the associated machine config pool. You can see the digested image pull spec for the new custom layered image in the `MachineOSConfig` object. This is now the active image pull spec for this `MachineOSConfig`.

.Example digested image pull spec
[source,terminal]
----
apiVersion: machineconfiguration.openshift.io/v1
kind: MachineOSConfig
metadata:
  annotations:
    machineconfiguration.openshift.io/current-machine-os-build: layered-9a8f89455246fa0c42ecee6ff1fa1a45
  labels:
    machineconfiguration.openshift.io/createdByOnClusterBuildsHelper: ""
  name: layered-image
# ...
status:
  currentImagePullSpec: image-registry.openshift-image-registry.svc:5000/openshift-machine-config-operator/os-image@sha256:3c8fc667adcb432ce0c83581f16086afec08a961dd28fed69bb6bad6db0a0754
----

[TIP]
====
You can test a `MachineOSBuild` object to make sure it builds correctly without rolling out the custom layered image to active nodes by using a custom machine config pool that contains non-production nodes. Alternatively, you can use a custom machine config pool that has no nodes. The `MachineOSBuild` object builds even if there are no nodes for the MCO to deploy the custom layered image onto.
====

You can apply a custom layered image to any machine config pool in your cluster, including the control plane, worker, or custom pools.

[NOTE]
====
For {sno} clusters, you can apply a custom layered image to the control plane node only.
====

include::snippets//coreos-layering-configuring-on-pause.adoc[]

In the case of a build failure, for example due to network issues or an invalid secret, the MCO retries the build three additional times before the job fails. The MCO creates a different build pod for each build attempt. Note that the MCO automatically removes these build pods after a short period of time. Also, the affected machine config pool reports a build failure through the `ImageBuildDegraded` status condition. You can use the build pod logs to troubleshoot any build failures.

.Example failed `MachineOSBuild` object
[source,terminal]
----
NAME                                             PREPARED   BUILDING   SUCCEEDED   INTERRUPTED   FAILED   AGE
layered-image-c8765e26ebc87e1e17a7d6e0a78e8bae   False      False      False        False        True     12m
----

You can manually rebuild your custom layered image by either modifying your `MachineOSConfig` object or applying an annotation to the `MachineOSConfig` object. For more information, see "Rebuilding an on-cluster custom layered image".

If you used a custom machine config pool to apply an on-cluster layered image to a node, you can remove the custom layered image from the node and revert to the base image. For more information, see "Reverting an on-cluster layered node".

You can modify an on-custom layered image as needed, to install additional packages, remove existing packages, change repositories, update secrets, or other similar changes, by editing the MachineOSConfig object. For more information, see "Modifying a custom layered image".


[id="coreos-layering-configuring-on-limitations_{context}"]
== {image-mode-os-on-caps} known limitations

Note the following limitations when working with the on-cluster layering feature:

* {image-mode-os-on-caps} is not supported on multi-architecture compute machines.
* Using multiple `MachineOSConfig` objects on the same machine config pool is not supported. You need a separate `MachineOSConfig` CR for each machine config pool where you want to use a distinct custom layered image. 
* If you scale up a machine set that uses a custom layered image, the nodes reboot two times. The first, when the node is initially created with the base image and a second time when the custom layered image is applied.
* Node disruption policies are not supported on nodes with a custom layered image. However, the following machine configuration changes do not cause a new image build or the reboot of a node with an on-cluster custom layered image:
** Modifying the configuration files in the `/var` or `/etc` directory
** Adding or modifying a systemd service
** Changing SSH keys
** Removing mirroring rules from `ICSP`, `ITMS`, and `IDMS` objects
** Changing the trusted CA, by updating the `user-ca-bundle` configmap in the `openshift-config` namespace
* The following machine configuration changes do cause a new image build and a node reboot:
** Changing the kernel arguments 
** Changing the `OSImageURL` parameter 
** Adding or removing extensions 

* The images used in creating custom layered images take up space in your push registry. Always be aware of the free space in your registry and prune the images as needed. You can automatically remove an on-cluster custom layered image from the repository by deleting the `MachineOSBuild` object that created the image. Note that the credentials provided by the registry push secret must also grant permission to delete an image from the registry. For more information, see "Removing an on-cluster custom layered image".
