//NOTE TO CONTRIBUTORS:
//
//If you update any of the content in this assembly file, be sure to also make the same changes in the assemblies in the following directory: backup_and_restore/control_plane_backup_and_restore/disaster-recovery/.

:_mod-docs-content-type: ASSEMBLY
[id="etcd-disaster-recovery"]
include::_attributes/common-attributes.adoc[]
= Disaster recovery
:context: etcd-disaster-recovery

toc::[]

The disaster recovery documentation provides information for administrators on how to recover from several disaster situations that might occur with their {product-title} cluster. As an administrator, you might need to follow one or more of the following procedures to return your cluster to a working state.

[IMPORTANT]
====
Disaster recovery requires you to have at least one healthy control plane host.
====

[id="etcd-dr-quorum"]
== Quorum restoration

You can use the `quorum-restore.sh` script to restore etcd quorum on clusters that are offline due to quorum loss. When quorum is lost, the {product-title} API becomes read-only. After quorum is restored, the {product-title} API returns to read/write mode.

// Restoring etcd quorum for high availability clusters
include::modules/dr-restoring-etcd-quorum-ha.adoc[leveloffset=+2]

[role="_additional-resources"]
.Additional resources
* xref:../../installing/installing_bare_metal/upi/installing-bare-metal.adoc#installing-bare-metal[Installing a user-provisioned cluster on bare metal]
* xref:../../installing/installing_bare_metal/bare-metal-expanding-the-cluster.adoc#replacing-a-bare-metal-control-plane-node_bare-metal-expanding[Replacing a bare-metal control plane node]

[NOTE]
====
If you have a majority of your control plane nodes still available and have an etcd quorum, xref:../../backup_and_restore/control_plane_backup_and_restore/replacing-unhealthy-etcd-member.adoc#replacing-unhealthy-etcd-member[replace a single unhealthy etcd member].
====

[id="etcd-dr-restore"]
== Restoring to a previous cluster state

To restore the cluster to a previous state, you must have previously backed up the `etcd` data by creating a snapshot. You will use this snapshot to restore the cluster state. For more information, see "Backing up etcd data".

If applicable, you might also need to xref:../../backup_and_restore/control_plane_backup_and_restore/disaster_recovery/scenario-3-expired-certs.adoc#dr-recovering-expired-certs[recover from expired control plane certificates].

[WARNING]
====
Restoring to a previous cluster state is a destructive and destablizing action to take on a running cluster. This procedure should only be used as a last resort.

Before performing a restore, see "About restoring to a previous cluster state" for more information on the impact to the cluster.
====

// About restoring to a previous cluster state
include::modules/dr-restoring-cluster-state-about.adoc[leveloffset=+2]

// Restoring to a previous cluster state for a single node
include::modules/dr-restoring-cluster-state-sno.adoc[leveloffset=+2]

// Restoring to a previous cluster state
include::modules/dr-restoring-cluster-state.adoc[leveloffset=+2]

[role="_additional-resources"]
.Additional resources
* xref:../../machine_management/control_plane_machine_management/cpmso-troubleshooting.adoc#cpmso-ts-etcd-degraded_cpmso-troubleshooting[Recovering a degraded etcd Operator]

include::modules/dr-scenario-cluster-state-issues.adoc[leveloffset=+2]

// Recovering from expired control plane certificates
include::modules/dr-recover-expired-control-plane-certs.adoc[leveloffset=+1]

//Testing restore procedures
include::modules/dr-testing-restore-procedures.adoc[leveloffset=+1]

[role="_additional-resources"]
.Additional resources
* xref:../../backup_and_restore/control_plane_backup_and_restore/disaster_recovery/scenario-2-restoring-cluster-state.adoc#dr-restoring-cluster-state[Restoring to a previous cluster state]