// Module included in the following assemblies:
//
// * installing/installing_bare_metal/ipi/ipi-install-prerequisites.adoc

:_mod-docs-content-type: CONCEPT
[id="network-requirements_{context}"]
= Network requirements

[role="_abstract"]
Installer-provisioned installation of {product-title} involves multiple network requirements. First, installer-provisioned installation involves an optional non-routable `provisioning` network for provisioning the operating system on each bare-metal node. Second, installer-provisioned installation involves a routable `baremetal` network.

image::210_OpenShift_Baremetal_IPI_Deployment_updates_0122_2.png[Installer-provisioned networking]

[IMPORTANT]
====
Red Hat supports any valid host network configuration that meets the documented requirements, regardless of how that configuration is applied. However, Red Hat only supports the configuration application process when approved tools are used. The following list details examples of these tools:

* A documented tool, such as NMState or Kubernetes-NMState.
* Other tools might be supported to perform a specific task. Red Hat does not support these other tools for general network configurations.
====

[id="network-requirements-one-ip-per-subnet_{context}"]
== One IP address for each subnet

For the primary interface, assign one IP address for each subnet. This configuration ensures the node IP address selection handles traffic without experiencing traffic handling issues.

You can assign multiple IP addresses from the same subnet to a single interface, in a process known as _IP aliasing_. However, the node IP address selection process cannot consistently determine which address to use, causing routing conflicts and unpredictable IP selection.

[id="network-requirements-ensuring-required-ports-are-open_{context}"]
== Ensuring required ports are open

Certain ports must be open between cluster nodes for installer-provisioned installations to complete successfully. In certain situations, such as using separate subnets for far edge worker nodes, you must ensure that the nodes in these subnets can communicate with nodes in the other subnets on the following required ports.

.Required ports
[options="header"]
|====
|Port|Description

|`67`,`68` | When using a provisioning network, cluster nodes access the `dnsmasq` DHCP server over their provisioning network interfaces using ports `67` and `68`.

| `69` | When using a provisioning network, cluster nodes communicate with the TFTP server on port `69` using their provisioning network interfaces. The TFTP server runs on the bootstrap VM. The bootstrap VM runs on the provisioner node.

| `80` | When not using the image caching option or when using virtual media, the provisioner node must have port `80` open on the `baremetal` machine network interface to stream the {op-system-first} image from the provisioner node to the cluster nodes.

| `123` | The cluster nodes must access the NTP server on port `123` using the `baremetal` machine network.

|`5050`| The Ironic Inspector API runs on the control plane nodes and listens on port `5050`. The Inspector API is responsible for hardware introspection, which collects information about the hardware characteristics of the bare-metal nodes.

|`5051`| Port `5050` uses port `5051` as a proxy.

|`6180`| When deploying with virtual media and not using TLS, the provisioner node and the control plane nodes must have port `6180` open on the `baremetal` machine network interface so that the baseboard management controller (BMC) of the worker nodes can access the {op-system} image. Starting with {product-title} 4.13, the default HTTP port is `6180`.

|`6183`| When deploying with virtual media and using TLS, the provisioner node and the control plane nodes must have port `6183` open on the `baremetal` machine network interface so that the BMC of the worker nodes can access the {op-system} image.

|`6385`| The Ironic API server runs initially on the bootstrap VM and later on the control plane nodes and listens on port `6385`. The Ironic API allows clients to interact with Ironic for bare-metal node provisioning and management, including operations such as enrolling new nodes, managing their power state, deploying images, and cleaning the hardware.

|`6388`| Port `6385` uses port `6388` as a proxy.

|`8080`| When using image caching without TLS, port `8080` must be open on the provisioner node and accessible by the BMC interfaces of the cluster nodes.

|`8083`| When using the image caching option with TLS, port `8083` must be open on the provisioner node and accessible by the BMC interfaces of the cluster nodes.

|`9999`| By default, the Ironic Python Agent (IPA) listens on TCP port `9999` for API calls from the Ironic conductor service. Communication between the bare-metal node where IPA is running and the Ironic conductor service uses this port.

|====

[id="network-requirements-increase-mtu_{context}"]
== Increase the network MTU

Before deploying {product-title}, increase the network maximum transmission unit (MTU) to 1500 or more. If the MTU is lower than 1500, the Ironic image that is used to boot the node might fail to communicate with the Ironic inspector pod, and inspection will fail. If this occurs, installation stops because the nodes are not available for installation.

[id="network-requirements-config-nics_{context}"]
== Configuring NICs

{product-title} deploys with two networks:

- `provisioning`: The `provisioning` network is an optional non-routable network used for provisioning the underlying operating system on each node that is a part of the {product-title} cluster. The network interface for the `provisioning` network on each cluster node must have the BIOS or UEFI configured to PXE boot.
+
The `provisioningNetworkInterface` configuration setting specifies the `provisioning` network NIC name on the control plane nodes, which must be identical on the control plane nodes. The `bootMACAddress` configuration setting provides a means to specify a particular NIC on each node for the `provisioning` network.
+
The `provisioning` network is optional, but it is required for PXE booting. If you deploy without a `provisioning` network, you must use a virtual media BMC addressing option such as `redfish-virtualmedia` or `idrac-virtualmedia`.

- `baremetal`: The `baremetal` network is a routable network. You can use any NIC to interface with the `baremetal` network provided the NIC is not configured to use the `provisioning` network.

[IMPORTANT]
====
When using a VLAN, each NIC must be on a separate VLAN corresponding to the appropriate network.
====

[id="network-requirements-dns_{context}"]
== DNS requirements

include::snippets/dns-requirements.adoc[leveloffset=+1]

[id="network-requirements-dhcp-reqs_{context}"]
== Dynamic Host Configuration Protocol (DHCP) requirements

By default, installer-provisioned installation deploys `ironic-dnsmasq` with DHCP enabled for the `provisioning` network. No other DHCP servers should be running on the `provisioning` network when the `provisioningNetwork` configuration setting is set to `managed`, which is the default value. If you have a DHCP server running on the `provisioning` network, you must set the `provisioningNetwork` configuration setting to `unmanaged` in the `install-config.yaml` file.

Network administrators must reserve IP addresses for each node in the {product-title} cluster for the `baremetal` network on an external DHCP server.

[id="network-requirements-reserving-ip-addresses_{context}"]
== Reserving IP addresses for nodes with the DHCP server

For the `baremetal` network, a network administrator must reserve several IP addresses, including:

. Two unique virtual IP addresses.
+
- One virtual IP address for the API endpoint.
- One virtual IP address for the wildcard ingress endpoint.
+
. One IP address for the provisioner node.
. One IP address for each control plane node.
. One IP address for each worker node, if applicable.

[IMPORTANT]
====
Reserving IP addresses so they become static IP addresses example:

Some administrators prefer to use static IP addresses so that each node's IP address remains constant in the absence of a DHCP server. To configure static IP addresses with NMState, see "(Optional) Configuring node network interfaces" in the "Setting up the environment for an OpenShift installation" section.
====

[IMPORTANT]
====
Networking between external load balancers and control plane nodes example:

External load balancing services and the control plane nodes must run on the same L2 network, and on the same VLAN when using VLANs to route traffic between the load balancing services and the control plane nodes.
====

[IMPORTANT]
====
The storage interface requires a DHCP reservation or a static IP.
====

The following table provides an exemplary embodiment of fully qualified domain names. The API and name server addresses begin with canonical name extensions. The hostnames of the control plane and worker nodes are exemplary, so you can use any host naming convention you prefer.

[width="100%", cols="3,5,2", options="header"]
|=====
| Usage | Host Name | IP
| API | `api.<cluster_name>.<base_domain>` | `<ip>`
| Ingress LB (apps) |  `*.apps.<cluster_name>.<base_domain>`  | `<ip>`
| Provisioner node | `provisioner.<cluster_name>.<base_domain>` | `<ip>`
| Control-plane-0 | `openshift-control-plane-0.<cluster_name>.<base_domain>` | `<ip>`
| Control-plane-1 | `openshift-control-plane-1.<cluster_name>-.<base_domain>` | `<ip>`
| Control-plane-2 | `openshift-control-plane-2.<cluster_name>.<base_domain>` | `<ip>`
| Worker-0 | `openshift-worker-0.<cluster_name>.<base_domain>` | `<ip>`
| Worker-1 | `openshift-worker-1.<cluster_name>.<base_domain>` | `<ip>`
| Worker-n | `openshift-worker-n.<cluster_name>.<base_domain>` | `<ip>`
|=====

[NOTE]
====
If you do not create DHCP reservations, the installation program requires reverse DNS resolution to set the hostnames for the Kubernetes API node, the provisioner node, the control plane nodes, and the worker nodes.
====

[id="network-requirements-provisioner_{context}"]
== Provisioner node requirements

You must specify the MAC address for the provisioner node in your installation configuration. The `bootMacAddress` specification is typically associated with PXE network booting. However, the Ironic provisioning service also requires the `bootMacAddress` specification to identify nodes during the inspection of the cluster, or during node redeployment in the cluster.

The provisioner node requires layer 2 connectivity for network booting, DHCP and DNS resolution, and local network communication. The provisioner node requires layer 3 connectivity for virtual media booting.

[id="network-requirements-ntp_{context}"]
== Network Time Protocol (NTP)

Each {product-title} node in the cluster must have access to an NTP server. {product-title} nodes use NTP to synchronize their clocks. For example, cluster nodes use SSL/TLS certificates that require validation, which might fail if the date and time between the nodes are not in sync.

[IMPORTANT]
====
Define a consistent clock date and time format in each cluster node's BIOS settings, or installation might fail.
====

You can reconfigure the control plane nodes to act as NTP servers on disconnected clusters, and reconfigure worker nodes to retrieve time from the control plane nodes.

[id="network-requirements-out-of-band_{context}"]
== Port access for the out-of-band management IP address

The out-of-band management IP address is on a separate network from the node. To ensure that the out-of-band management can communicate with the provisioner node during installation, the out-of-band management IP address must be granted access to port `6180` on the provisioner node and on the {product-title} control plane nodes. TLS port `6183` is required for virtual media installation, for example, by using Redfish.
