// Module included in the following assemblies:
//
// * backup_and_restore/control_plane_backup_and_restore/disaster_recovery/about-disaster-recovery.adoc
// * etcd/etcd-backup-restore/etcd-disaster-recovery.adoc

:_mod-docs-content-type: PROCEDURE
[id="dr-testing-restore-procedures_{context}"]
= Testing restore procedures

[role="_abstract"]
You can test your cluster restore workflow by simulating etcd failure on nonrecovery nodes and restoring from backup. Use this test to confirm that your etcd backup and restore process works as expected.

[WARNING]
====
You must have SSH access to the cluster. Without SSH access, you cannot disable etcd or manage the `kubelet` service on nonrecovery nodes.
====

.Prerequisites

* You have SSH access to control plane hosts.
* You have installed the {oc-first}.

.Procedure

. Use SSH to connect to each of your nonrecovery nodes to disable etcd and the `kubelet` service:

.. Disable etcd by running the following command:
+
[source,terminal]
----
$ sudo /usr/local/bin/disable-etcd.sh
----

.. Delete variable data for etcd by running the following command:
+
[source,terminal]
----
$ sudo rm -rf /var/lib/etcd
----

.. Disable the `kubelet` service by running the following command:
+
[source,terminal]
----
$ sudo systemctl disable kubelet.service
----

. Exit every SSH session.

. Ensure that your nonrecovery nodes are in a `NOT READY` state by running the following command:
+
[source,terminal]
----
$ oc get nodes
----

. Restore your cluster to an earlier cluster state using an etcd backup. For more information, see "Restoring to an earlier cluster state".

. After you restore the cluster and the API responds, use SSH to connect to each nonrecovery node and enable the `kubelet` service by running the following command:
+
[source,terminal]
----
$ sudo systemctl enable kubelet.service
----

. Exit every SSH session.

. Verify that your nodes return to the `READY` state by running the following command:
+
[source,terminal]
----
$ oc get nodes
----

. Verify that etcd is available by running the following command:
+
[source,terminal]
----
$ oc get pods -n openshift-etcd
----