// Module included in the following assemblies:
//
// * scalability_and_performance/ztp_far_edge/ztp-precaching-tool.adoc

:_mod-docs-content-type: CONCEPT
[id="ztp-pre-caching-config-con_{context}"]
= Pre-caching images in {ztp}

[role="_abstract"]
The `ClusterInstance` manifest defines the installation and configuration parameters for an {product-title} cluster.
In the {ztp-first} provisioning workflow, the {factory-prestaging-tool} uses the following fields in the `ClusterInstance` manifest to load the pre-cached images:

* `spec.ignitionConfigOverride`
* `spec.nodes[].ignitionConfigOverride`
* `spec.nodes[].installerArgs`

The following example shows the `ClusterInstance` CR with the pre-caching fields:

[source,yaml]
----
apiVersion: siteconfig.open-cluster-management.io/v1alpha1
kind: ClusterInstance
metadata:
  name: "sno-worker-0"
  namespace: "sno-worker-0"
spec:
  baseDomain: "example.domain.redhat.com"
  pullSecretRef:
    name: "assisted-deployment-pull-secret"
  clusterImageSetNameRef: "openshift-4.22"
  sshPublicKey: "ssh-rsa ..."
  clusterName: "sno-worker-0"
  extraLabels:
    ManagedCluster:
      group-du-sno: ""
      common-411: "true"
      sites: "example-5g-lab"
      vendor: "OpenShift"
  clusterNetwork:
    - cidr: 10.128.0.0/14
      hostPrefix: 23
  machineNetwork:
    - cidr: 10.19.32.192/26
  serviceNetwork:
    - cidr: 172.30.0.0/16
  networkType: "OVNKubernetes"
  additionalNTPSources:
    - 1111:2222:3333:4444::2
  templateRefs:
    - name: ai-cluster-templates-v1
      namespace: siteconfig-system
  ignitionConfigOverride: |
    {
      "ignition": {
        "version": "3.2.0"
      },
      "systemd": {
        "units": [
          {
            "name": "var-mnt.mount",
            "enabled": true,
            "contents": "[Unit]\nDescription=Mount partition with artifacts\nBefore=precache-images.service\nBindsTo=precache-images.service\nStopWhenUnneeded=true\n\n[Mount]\nWhat=/dev/disk/by-partlabel/data\nWhere=/var/mnt\nType=xfs\nTimeoutSec=30\n\n[Install]\nRequiredBy=precache-images.service"
          },
          {
            "name": "precache-images.service",
            "enabled": true,
            "contents": "[Unit]\nDescription=Extracts the precached images in discovery stage\nAfter=var-mnt.mount\nBefore=agent.service\n\n[Service]\nType=oneshot\nUser=root\nWorkingDirectory=/var/mnt\nExecStart=bash /usr/local/bin/extract-ai.sh\n\n[Install]\nWantedBy=multi-user.target default.target\nWantedBy=agent.service"
          }
        ]
      },
      "storage": {
        "files": [
          {
            "overwrite": true,
            "path": "/usr/local/bin/extract-ai.sh",
            "mode": 755,
            "user": {
              "name": "root"
            },
            "contents": {
              "source": "data:,%23%21%2Fbin%2Fbash%0A%0AFOLDER%3D%22%24%7BFOLDER%3A-%24%28pwd%29%7D%22%0AOCP_RELEASE_LIST%3D%22%24%7BOCP_RELEASE_LIST%3A-ai-images.txt%7D%22%0ABINARY_FOLDER%3D%2Fvar%2Fmnt%0A%0Apushd%20%24FOLDER%0A%0Atotal_copies%3D%24%28sort%20-u%20%24BINARY_FOLDER%2F%24OCP_RELEASE_LIST%20%7C%20wc%20-l%29%20%20%23%20Required%20to%20keep%20track%20of%20the%20pull%20task%20vs%20total%0Acurrent_copy%3D1%0A%0Awhile%20read%20-r%20line%3B%0Ado%0A%20%20uri%3D%24%28echo%20%22%24line%22%20%7C%20awk%20%27%7Bprint%241%7D%27%29%0A%20%20%23tar%3D%24%28echo%20%22%24line%22%20%7C%20awk%20%27%7Bprint%242%7D%27%29%0A%20%20podman%20image%20exists%20%24uri%0A%20%20if%20%5B%5B%20%24%3F%20-eq%200%20%5D%5D%3B%20then%0A%20%20%20%20%20%20echo%20%22Skipping%20existing%20image%20%24tar%22%0A%20%20%20%20%20%20echo%20%22Copying%20%24%7Buri%7D%20%5B%24%7Bcurrent_copy%7D%2F%24%7Btotal_copies%7D%5D%22%0A%20%20%20%20%20%20current_copy%3D%24%28%28current_copy%20%2B%201%29%29%0A%20%20%20%20%20%20continue%0A%20%20fi%0A%20%20tar%3D%24%28echo%20%22%24uri%22%20%7C%20%20rev%20%7C%20cut%20-d%20%22%2F%22%20-f1%20%7C%20rev%20%7C%20tr%20%22%3A%22%20%22_%22%29%0A%20%20tar%20zxvf%20%24%7Btar%7D.tgz%0A%20%20if%20%5B%20%24%3F%20-eq%200%20%5D%3B%20then%20rm%20-f%20%24%7Btar%7D.gz%3B%20fi%0A%20%20echo%20%22Copying%20%24%7Buri%7D%20%5B%24%7Bcurrent_copy%7D%2F%24%7Btotal_copies%7D%5D%22%0A%20%20skopeo%20copy%20dir%3A%2F%2F%24%28pwd%29%2F%24%7Btar%7D%20containers-storage%3A%24%7Buri%7D%0A%20%20if%20%5B%20%24%3F%20-eq%200%20%5D%3B%20then%20rm%20-rf%20%24%7Btar%7D%3B%20current_copy%3D%24%28%28current_copy%20%2B%201%29%29%3B%20fi%0Adone%20%3C%20%24%7BBINARY_FOLDER%7D%2F%24%7BOCP_RELEASE_LIST%7D%0A%0A%23%20workaround%20while%20https%3A%2F%2Fgithub.com%2Fopenshift%2Fassisted-service%2Fpull%2F3546%0A%23cp%20%2Fvar%2Fmnt%2Fmodified-rhcos-4.10.3-x86_64-metal.x86_64.raw.gz%20%2Fvar%2Ftmp%2F.%0A%0Aexit%200"
            }
          },
          {
            "overwrite": true,
            "path": "/usr/local/bin/agent-fix-bz1964591",
            "mode": 755,
            "user": {
              "name": "root"
            },
            "contents": {
              "source": "data:,%23%21%2Fusr%2Fbin%2Fsh%0A%0A%23%20This%20script%20is%20a%20workaround%20for%20bugzilla%201964591%20where%20symlinks%20inside%20%2Fvar%2Flib%2Fcontainers%2F%20get%0A%23%20corrupted%20under%20some%20circumstances.%0A%23%0A%23%20In%20order%20to%20let%20agent.service%20start%20correctly%20we%20are%20checking%20here%20whether%20the%20requested%0A%23%20container%20image%20exists%20and%20in%20case%20%22podman%20images%22%20returns%20an%20error%20we%20try%20removing%20the%20faulty%0A%23%20image.%0A%23%0A%23%20In%20such%20a%20scenario%20agent.service%20will%20detect%20the%20image%20is%20not%20present%20and%20pull%20it%20again.%20In%20case%0A%23%20the%20image%20is%20present%20and%20can%20be%20detected%20correctly%2C%20no%20any%20action%20is%20required.%0A%0AIMAGE%3D%24%28echo%20%241%20%7C%20sed%20%27s%2F%3A.%2A%2F%2F%27%29%0Apodman%20image%20exists%20%24IMAGE%20%7C%7C%20echo%20%22already%20loaded%22%20%7C%7C%20echo%20%22need%20to%20be%20pulled%22%0A%23podman%20images%20%7C%20grep%20%24IMAGE%20%7C%7C%20podman%20rmi%20--force%20%241%20%7C%7C%20true"
            }
          }
        ]
      }
    }
  nodes:
    - hostName: "snonode.sno-worker-0.example.domain.redhat.com"
      role: "master"
      bmcAddress: "idrac-virtualmedia+https://[1111:2222:3333:4444::bbbb:1]/redfish/v1/Systems/System.Embedded.1"
      bmcCredentialsName:
        name: "worker0-bmh-secret"
      bootMACAddress: "AA:BB:CC:DD:EE:11"
      bootMode: "UEFI"
      rootDeviceHints:
        deviceName: /dev/disk/by-path/pci-0000:01:00.0-scsi-0:2:0:0
      installerArgs: '["--save-partlabel", "data"]'
      ignitionConfigOverride: |
        {
          "ignition": {
            "version": "3.2.0"
          },
          "systemd": {
            "units": [
              {
                "name": "var-mnt.mount",
                "enabled": true,
                "contents": "[Unit]\nDescription=Mount partition with artifacts\nBefore=precache-ocp-images.service\nBindsTo=precache-ocp-images.service\nStopWhenUnneeded=true\n\n[Mount]\nWhat=/dev/disk/by-partlabel/data\nWhere=/var/mnt\nType=xfs\nTimeoutSec=30\n\n[Install]\nRequiredBy=precache-ocp-images.service"
              },
              {
                "name": "precache-ocp-images.service",
                "enabled": true,
                "contents": "[Unit]\nDescription=Extracts the precached OCP images into containers storage\nAfter=var-mnt.mount\nBefore=machine-config-daemon-pull.service nodeip-configuration.service\n\n[Service]\nType=oneshot\nUser=root\nWorkingDirectory=/var/mnt\nExecStart=bash /usr/local/bin/extract-ocp.sh\nTimeoutStopSec=60\n\n[Install]\nWantedBy=multi-user.target"
              }
            ]
          },
          "storage": {
            "files": [
              {
                "overwrite": true,
                "path": "/usr/local/bin/extract-ocp.sh",
                "mode": 755,
                "user": {
                  "name": "root"
                },
                "contents": {
                  "source": "data:,%23%21%2Fbin%2Fbash%0A%0AFOLDER%3D%22%24%7BFOLDER%3A-%24%28pwd%29%7D%22%0AOCP_RELEASE_LIST%3D%22%24%7BOCP_RELEASE_LIST%3A-ocp-images.txt%7D%22%0ABINARY_FOLDER%3D%2Fvar%2Fmnt%0A%0Apushd%20%24FOLDER%0A%0Atotal_copies%3D%24%28sort%20-u%20%24BINARY_FOLDER%2F%24OCP_RELEASE_LIST%20%7C%20wc%20-l%29%20%20%23%20Required%20to%20keep%20track%20of%20the%20pull%20task%20vs%20total%0Acurrent_copy%3D1%0A%0Awhile%20read%20-r%20line%3B%0Ado%0A%20%20uri%3D%24%28echo%20%22%24line%22%20%7C%20awk%20%27%7Bprint%241%7D%27%29%0A%20%20%23tar%3D%24%28echo%20%22%24line%22%20%7C%20awk%20%27%7Bprint%242%7D%27%29%0A%20%20podman%20image%20exists%20%24uri%0A%20%20if%20%5B%5B%20%24%3F%20-eq%200%20%5D%5D%3B%20then%0A%20%20%20%20%20%20echo%20%22Skipping%20existing%20image%20%24tar%22%0A%20%20%20%20%20%20echo%20%22Copying%20%24%7Buri%7D%20%5B%24%7Bcurrent_copy%7D%2F%24%7Btotal_copies%7D%5D%22%0A%20%20%20%20%20%20current_copy%3D%24%28%28current_copy%20%2B%201%29%29%0A%20%20%20%20%20%20continue%0A%20%20fi%0A%20%20tar%3D%24%28echo%20%22%24uri%22%20%7C%20%20rev%20%7C%20cut%20-d%20%22%2F%22%20-f1%20%7C%20rev%20%7C%20tr%20%22%3A%22%20%22_%22%29%0A%20%20tar%20zxvf%20%24%7Btar%7D.tgz%0A%20%20if%20%5B%20%24%3F%20-eq%200%20%5D%3B%20then%20rm%20-f%20%24%7Btar%7D.gz%3B%20fi%0A%20%20echo%20%22Copying%20%24%7Buri%7D%20%5B%24%7Bcurrent_copy%7D%2F%24%7Btotal_copies%7D%5D%22%0A%20%20skopeo%20copy%20dir%3A%2F%2F%24%28pwd%29%2F%24%7Btar%7D%20containers-storage%3A%24%7Buri%7D%0A%20%20if%20%5B%20%24%3F%20-eq%200%20%5D%3B%20then%20rm%20-rf%20%24%7Btar%7D%3B%20current_copy%3D%24%28%28current_copy%20%2B%201%29%29%3B%20fi%0Adone%20%3C%20%24%7BBINARY_FOLDER%7D%2F%24%7BOCP_RELEASE_LIST%7D%0A%0Aexit%200"
                }
              }
            ]
          }
        }
      nodeNetwork:
        interfaces:
          - name: "ens1f0"
            macAddress: "AA:BB:CC:11:22:33"
        config:
          interfaces:
            - name: ens1f0
              type: ethernet
              state: up
              macAddress: "AA:BB:CC:11:22:33"
              ipv4:
                enabled: true
                dhcp: true
              ipv6:
                enabled: false
      templateRefs:
        - name: ai-node-templates-v1
          namespace: siteconfig-system
----

Where:

`spec.clusterImageSetNameRef`:: Specifies the cluster image set used for deployment.
`spec.ignitionConfigOverride`:: Configures the cluster-level ignition config override for the discovery stage.
`spec.nodes[].installerArgs`:: Specifies the installation program arguments to preserve the data partition.
`spec.nodes[].ignitionConfigOverride`:: Configures the node-level ignition config override for the installation stage.

[id="ztp-pre-caching-config-spec-ignitionconfigoverride_{context}"]
== Understanding the spec.ignitionConfigOverride field

The `spec.ignitionConfigOverride` field adds a configuration in Ignition format during the {ztp} discovery stage.
The configuration includes `systemd` services in the ISO mounted in virtual media. This way, the scripts are part of the discovery {op-system} live ISO and they can be used to load the Assisted Installer (AI) images.

`systemd` services:: The `systemd` services are `var-mnt.mount` and `precache-images.services`. The `precache-images.service` depends on the disk partition to be mounted in `/var/mnt` by the `var-mnt.mount` unit.
The service calls a script called `extract-ai.sh`.
`extract-ai.sh`:: The `extract-ai.sh` script extracts and loads the required images from the disk partition to the local container storage.
When the script finishes successfully, you can use the images locally.
`agent-fix-bz1964591`:: The `agent-fix-bz1964591` script is a workaround for an AI issue.
To prevent AI from removing the images, which can force the `agent.service` to pull the images again from the registry, the `agent-fix-bz1964591` script checks if the requested container images exist.

[id="ztp-pre-caching-config-spec-nodes-installerargs_{context}"]
== Understanding the spec.nodes[].installerArgs field

The `spec.nodes[].installerArgs` field allows you to configure how the `coreos-installer` utility writes the {op-system} live ISO to disk. You need to indicate to save the disk partition labeled as `data` because the artifacts saved in the `data` partition are needed during the {product-title} installation stage.

The extra parameters are passed directly to the `coreos-installer` utility that writes the live {op-system} to disk.
On the next reboot, the operating system starts from the disk.

You can pass several options to the `coreos-installer` utility:

[source,terminal]
----
OPTIONS:
...
    -u, --image-url <URL>
            Manually specify the image URL

    -f, --image-file <path>
            Manually specify a local image file

    -i, --ignition-file <path>
            Embed an Ignition config from a file

    -I, --ignition-url <URL>
            Embed an Ignition config from a URL
...
        --save-partlabel <lx>...
            Save partitions with this label glob

        --save-partindex <id>...
            Save partitions with this number or range
...
        --insecure-ignition
            Allow Ignition URL without HTTPS or hash
----

[id="ztp-pre-caching-config-spec-nodes-ignitionconfigoverride_{context}"]
== Understanding the spec.nodes[].ignitionConfigOverride field

Similarly to `spec.ignitionConfigOverride`, the `spec.nodes[].ignitionConfigOverride` field allows the addition of configurations in Ignition format to the `coreos-installer` utility, but at the {product-title} installation stage.
When the {op-system} is written to disk, the extra configuration included in the {ztp} discovery ISO is no longer available. During the discovery stage, the extra configuration is stored in the memory of the live OS.

[NOTE]
====
At this stage, the number of container images extracted and loaded is bigger than in the discovery stage. Depending on the {product-title} release and whether you install the Day-2 Operators, the installation time can vary.
====

At the installation stage, the `var-mnt.mount` and `precache-ocp.services` `systemd` services are used.

`precache-ocp.service`:: The `precache-ocp.service` depends on the disk partition to be mounted in `/var/mnt` by the `var-mnt.mount` unit.
The `precache-ocp.service` service calls a script called `extract-ocp.sh`.
+
[IMPORTANT]
====
To extract all the images before the {product-title} installation, you must execute `precache-ocp.service` before executing the `machine-config-daemon-pull.service` and `nodeip-configuration.service` services.
====

`extract-ocp.sh`:: The `extract-ocp.sh` script extracts and loads the required images from the disk partition to the local container storage.

When you commit the `ClusterInstance` and optional `PolicyGenerator` or `PolicyGenTemplate` custom resources (CRs) to the Git repo that Argo CD is monitoring, you can start the {ztp} workflow by syncing the CRs with the hub cluster.
