// Module included in the following assemblies:
//
// * post_installation_configuration/configuring-multi-arch-compute-machines/creating-multi-arch-compute-nodes-ibm-z.adoc
ifeval::["{context}" == "creating-multi-arch-compute-nodes-ibm-z"]
:ibm-z:
endif::[]
ifeval::["{context}" == "creating-multi-arch-compute-nodes-ibm-z-lpar"]
:ibm-z-lpar:
endif::[]
:_mod-docs-content-type: PROCEDURE
// Assumption is that attribute once outside ifdef works for several level one headings.
[id="machine-user-infra-machines-ibm-z_{context}"]
ifdef::ibm-z[]
= Creating {op-system} machines on {ibm-z-title} with z/VM

[role="_abstract"]
You can create more {op-system-first} compute machines running on {ibm-z-name} with z/VM and attach them to your existing cluster.
endif::ibm-z[]
ifdef::ibm-z-lpar[]
= Creating {op-system} machines on {ibm-z-title} in an LPAR

[role="_abstract"]
You can create more {op-system-first} compute machines running on {ibm-z-name} in a logical partition (LPAR) and attach them to your existing cluster.
endif::ibm-z-lpar[]

.Prerequisites

* You have a domain name server (DNS) that can perform hostname and reverse lookup for the nodes.
* You have an HTTP or HTTPS server running on your provisioning machine that is accessible to the machines you create.

.Procedure

. Extract the Ignition config file from the cluster by running the following command:
+
[source,terminal]
----
$ oc extract -n openshift-machine-api secret/worker-user-data-managed --keys=userData --to=- > worker.ign
----

. Upload the `worker.ign` Ignition config file you exported from your cluster to your HTTP server. Note the URL of this file.

. Validate that the Ignition file is available on the URL. The following example gets the Ignition config file for the compute node:
+
[source,terminal]
----
$ curl -k http://<http_server>/worker.ign
----

. Download the {op-system-base} live `kernel`, `initramfs`, and `rootfs` files by running the following commands:
+
[source,terminal]
----
$ curl -LO $(oc -n openshift-machine-config-operator get configmap/coreos-bootimages -o jsonpath='{.data.stream}' \
| jq -r '.architectures.s390x.artifacts.metal.formats.pxe.kernel.location')
----
+
[source,terminal]
----
$ curl -LO $(oc -n openshift-machine-config-operator get configmap/coreos-bootimages -o jsonpath='{.data.stream}' \
| jq -r '.architectures.s390x.artifacts.metal.formats.pxe.initramfs.location')
----
+
[source,terminal]
----
$ curl -LO $(oc -n openshift-machine-config-operator get configmap/coreos-bootimages -o jsonpath='{.data.stream}' \
| jq -r '.architectures.s390x.artifacts.metal.formats.pxe.rootfs.location')
----

. Move the downloaded {op-system-base} live `kernel`, `initramfs`, and `rootfs` files to an HTTP or HTTPS server that is accessible from the {op-system} guest you want to add.

. Create a parameter file for the guest. The following parameters are specific to the virtual machine:
+
** Optional: To specify a static IP address, add an `ip=` parameter with the following entries, with each separated by a colon:
+
... The IP address for the machine.
+
... An empty string.
+
... The gateway.
+
... The netmask.
+
... The machine host and domain name in the form `hostname.domainname`. If you omit this value, {op-system} obtains the hostname through a reverse DNS lookup.
+
... The network interface name. If you omit this value, {op-system} applies the IP configuration to all available interfaces.
+
... The value `none`.
+
** For `coreos.inst.ignition_url=`, specify the URL to the `worker.ign` file. Only HTTP and HTTPS protocols are supported.
+
** For `coreos.live.rootfs_url=`, specify the matching rootfs artifact for the `kernel` and `initramfs` you are booting. Only HTTP and HTTPS protocols are supported.
+
** For installations on DASD-type disks, complete the following tasks:
+
... For `coreos.inst.install_dev=`, specify `/dev/dasda`.
+
... Use `rd.dasd=` to specify the DASD where {op-system} is to be installed.
+
... You can adjust further parameters if required.
+
The following is an example parameter file, `additional-worker-dasd.parm`:
+
[source,terminal]
----
cio_ignore=all,!condev rd.neednet=1 \
console=ttysclp0 \
coreos.inst.install_dev=/dev/dasda \
coreos.inst.ignition_url=http://<http_server>/worker.ign \
coreos.live.rootfs_url=http://<http_server>/rhcos-<version>-live-rootfs.<architecture>.img \
ip=<ip>::<gateway>:<netmask>:<hostname>::none nameserver=<dns> \
rd.znet=qeth,0.0.bdf0,0.0.bdf1,0.0.bdf2,layer2=1,portno=0 \
rd.dasd=0.0.3490 \
zfcp.allow_lun_scan=0
----
+
Write all options in the parameter file as a single line and make sure that you have no newline characters.
+
** For installations on FCP-type disks, complete the following tasks:
+
... Use `rd.zfcp=<adapter>,<wwpn>,<lun>` to specify the FCP disk where {op-system} is to be installed. For multipathing, repeat this step for each additional path.
+
[NOTE]
====
When you install with multiple paths, you must enable multipathing directly after the installation. Enabling multipathing much later can cause problems for your cluster.
====
+
... Set the install device as: `coreos.inst.install_dev=/dev/sda`.
+
[NOTE]
====
If additional LUNs are configured with NPIV, FCP requires `zfcp.allow_lun_scan=0`. If you must enable `zfcp.allow_lun_scan=1` because you use a CSI driver, for example, you must configure your NPIV so that each node cannot access the boot partition of another node.
====
+
... You can adjust further parameters if required.
+
[IMPORTANT]
====
Additional postinstallation steps are required to fully enable multipathing. For more information, see “Enabling multipathing with kernel arguments on {op-system}" in _Machine configuration_.
====
// Add xref once it's allowed.
+
The following is an example parameter file, `additional-worker-fcp.parm` for a worker node with multipathing:
+
[source,terminal]
----
cio_ignore=all,!condev rd.neednet=1 \
console=ttysclp0 \
coreos.inst.install_dev=/dev/sda \
coreos.live.rootfs_url=http://<http_server>/rhcos-<version>-live-rootfs.<architecture>.img \
coreos.inst.ignition_url=http://<http_server>/worker.ign \
ip=<ip>::<gateway>:<netmask>:<hostname>::none nameserver=<dns> \
rd.znet=qeth,0.0.bdf0,0.0.bdf1,0.0.bdf2,layer2=1,portno=0 \
zfcp.allow_lun_scan=0 \
rd.zfcp=0.0.1987,0x50050763070bc5e3,0x4008400B00000000 \
rd.zfcp=0.0.19C7,0x50050763070bc5e3,0x4008400B00000000 \
rd.zfcp=0.0.1987,0x50050763071bc5e3,0x4008400B00000000 \
rd.zfcp=0.0.19C7,0x50050763071bc5e3,0x4008400B00000000
----
+
Write all options in the parameter file as a single line and make sure that you have no newline characters.

ifdef::ibm-z[]
. Transfer the `initramfs`, `kernel`, parameter files, and {op-system} images to z/VM, for example, by using FTP. For details about how to transfer the files with FTP and boot from the virtual reader, see link:https://docs.redhat.com/en/documentation/red_hat_enterprise_linux/9/html-single/interactively_installing_rhel_over_the_network/index#installing-under-z-vm_booting-the-installation-media[Booting the installation on {ibm-z-name} to install {op-system-base} in z/VM].

. Punch the files to the virtual reader of the z/VM guest virtual machine.
+
See link:https://www.ibm.com/docs/en/zvm/latest?topic=commands-punch[PUNCH] in {ibm-name} Documentation.
+
[TIP]
====
You can use the CP PUNCH command or, if you use Linux, the `vmur` command to transfer files between two z/VM guest virtual machines.
====

. Log in to CMS on the bootstrap machine.

. IPL the bootstrap machine from the reader by running the following command:
+
[source,terminal]
----
$ ipl c
----
+
See link:https://www.ibm.com/docs/en/zvm/latest?topic=commands-ipl[IPL] in {ibm-name} Documentation.
endif::ibm-z[]

ifdef::ibm-z-lpar[]
. As an example for FTP, transfer the initramfs, kernel, parameter files, and {op-system} images to the LPAR. For details about how to transfer the files with FTP and boot, see link:https://docs.redhat.com/en/documentation/red_hat_enterprise_linux/9/html-single/interactively_installing_rhel_over_the_network/index#installing-in-an-lpar_booting-the-installation-media[Booting the installation on {ibm-z-name} to install {op-system-base} in an LPAR].

. Boot the machine.
endif::ibm-z-lpar[]

ifeval::["{context}" == "creating-multi-arch-compute-nodes-ibm-z"]
:!ibm-z:
endif::[]
ifeval::["{context}" == "creating-multi-arch-compute-nodes-ibm-z-lpar"]
:!ibm-z-lpar:
endif::[]
