// Module included in the following assemblies:
//
// * operators/operator-reference.adoc
// * machine_configuration/index.adoc

:_mod-docs-content-type: CONCEPT
[id="machine-config-overview_{context}"]
= Machine config overview

[role="_abstract"]
You should understand what the Machine Config Operator (MCO) does and how it interacts with other components before making advanced, system-level changes to an {product-title} cluster. 

The Machine Config Operator (MCO) manages updates to systemd, CRI-O and Kubelet, the kernel, Network Manager, and other system features. It also offers a `MachineConfig` CRD that can write configuration files onto the host.

You should know the following details about the MCO, machine configs, and how they are used:

* A machine config can make a specific change to a file or service on the operating system of each system representing a pool of {product-title} nodes.

* MCO applies changes to operating systems in pools of machines. All {product-title} clusters start with worker and control plane node pools. By adding more role labels, you can configure custom pools of nodes. For example, you can set up a custom pool of worker nodes that includes particular hardware features needed by an application. However, examples in this section focus on changes to the default pool types.
+
[IMPORTANT]
====
A node can have multiple labels applied that indicate its type, such as `master` or `worker`, however it can be a member of only a *single* machine config pool.
====

* Machine configs are processed alphabetically, in lexicographically increasing order, by their name. The render controller uses the first machine config in the list as the base and appends the rest to the base machine config into a rendered machine config, which is then applied to the appropriate nodes.

* When you create a machine config for the worker nodes, the changes are also applied to the nodes in all custom pools.
+
However, as of {product-title} 4.15, any machine configs that target custom pools always override worker machine configs if the worker machine configs contain definitions for the same fields.

* After a machine config change, the MCO updates the affected nodes alphabetically by zone, based on the `topology.kubernetes.io/zone` label. If a zone has more than one node, the oldest nodes are updated first. For nodes that do not use zones, such as in bare metal deployments, the nodes are upgraded by age, with the oldest nodes updated first. The MCO updates the number of nodes as specified by the `maxUnavailable` field on the machine configuration pool at a time.

* Some machine configuration must be in place before {product-title} is installed to disk. In most cases, this can be accomplished by creating
a machine config that is injected directly into the {product-title} installer process, instead of running as a postinstallation machine config. In other cases, you might need to do bare metal installation where you pass kernel arguments at {product-title} installer startup, to do such things as setting per-node individual IP addresses or advanced disk partitioning.

* MCO manages items that are set in machine configs. Manual changes you do to your systems will not be overwritten by MCO, unless MCO is explicitly told to manage a conflicting file. In other words, MCO only makes specific updates you request, it does not claim control over the whole node.

* Manual changes to nodes are strongly discouraged. If you need to decommission a node and start a new one, those direct changes would be lost.

* MCO is only supported for writing to files in `/etc` and `/var` directories, although there are symbolic links to some directories that can be writeable by being symbolically linked to one of those areas. The `/opt` and `/usr/local` directories are examples.

* Ignition is the configuration format used in MachineConfigs. For details, see the "Configuration Specification v3.5.0 (Ignition documentation)" in the _Additional resources_ section.

* Although Ignition config settings can be delivered directly at {product-title} installation time, and are formatted in the same way that MCO delivers Ignition configs, MCO has no way of seeing what those original Ignition configs are. Therefore, you should wrap Ignition config settings into a machine config before deploying them.

* When a file managed by MCO changes outside of MCO, the Machine Config Daemon (MCD) sets the node as `degraded`. It will not overwrite the
offending file, however, and should continue to operate in a `degraded` state.

* A key reason for using a machine config is that it will be applied when you spin up new nodes for a pool in your {product-title} cluster. The `machine-api-operator` provisions a new machine and MCO configures it.

MCO uses Ignition as the configuration format. {product-title} 4.6 moved from Ignition config specification version 2 to version 3. For more information, see the "Configuration Specification v3.5.0 (Ignition documentation)" in the _Additional resources_ section.

== What can you change with machine configs?
The kinds of components that MCO can change include:

* **config**: Create Ignition config objects to do things like modify files, systemd services, and other features on {product-title} machines, including:
- **Configuration files**: Create or overwrite files in the `/var` or `/etc` directory.
- **systemd units**: Create and set the status of a systemd service or add to an existing systemd service by dropping in additional settings.
- **users and groups**: Change SSH keys in the passwd section postinstallation.
+
[IMPORTANT]
====
* Changing SSH keys by using a machine config is supported only for the `core` user.
* Adding new users by using a machine config is not supported.
====
* **kernelArguments**: Add arguments to the kernel command line when {product-title} nodes boot.
* **kernelType**: Optionally identify a non-standard kernel to use instead of the standard kernel. Use `realtime` to use the RT kernel (for RAN). This is only supported on select platforms. Use the `64k-pages` parameter to enable the 64k page size kernel. This setting is exclusive to machines with 64-bit ARM architectures.
ifndef::openshift-origin[]
* **fips**: Enable FIPS mode. FIPS should be set at installation-time setting and not a postinstallation procedure. For more information, see "Using system-wide cryptographic policies" in the _Additional resources_ section.
+
--
include::snippets/fips-snippet.adoc[]
--

endif::openshift-origin[]
* **extensions**: Extend {op-system} features by adding selected pre-packaged software. For this feature, available extensions include usbguard and kernel modules. For more information, see "Protecting systems against intrusive USB devices" in the _Additional resources_ section.
* **Custom resources (for `ContainerRuntime` and `Kubelet`)**: Outside of machine configs, MCO manages two special custom resources for modifying CRI-O container runtime settings (`ContainerRuntime` CR) and the Kubelet service (`Kubelet` CR).

The MCO is not the only Operator that can change operating system components on {product-title} nodes. Other Operators can modify operating system-level features as well. One example is the Node Tuning Operator, which allows you to do node-level tuning through Tuned daemon profiles.

Tasks for the MCO configuration that can be done after installation are included in the following procedures. See descriptions of {op-system} bare metal installation for system configuration tasks that must be done during or before {product-title} installation. By default, many of the changes you make with the MCO require a reboot. 

include::snippets/node-icsp-no-drain.adoc[]

In other cases, you can mitigate the disruption to your workload when the MCO makes changes by using _node disruption policies_. For information, see _Understanding node restart behaviors after machine config changes_.  

There might be situations where the configuration on a node does not fully match what the currently-applied machine config specifies. This state is called _configuration drift_. The Machine Config Daemon (MCD) regularly checks the nodes for configuration drift. If the MCD detects configuration drift, the MCO marks the node `degraded` until an administrator corrects the node configuration. A degraded node is online and operational, but, it cannot be updated. For more information on configuration drift, see _Understanding configuration drift detection_.
