// Module included in the following assemblies:
//
// security/nbde-implementation-guide.adoc

:_mod-docs-content-type: PROCEDURE
[id="nbde-troubleshooting-permanent-error-conditions_{context}"]
= Troubleshooting permanent rekeying errors for Tang servers

If, after rekeying the Tang servers, the `READY` count does not equal the `DESIRED` count after an extended period of time, it might indicate a permanent failure condition. In this case, the following conditions might apply:

* A typographical error in the Tang server URL or thumbprint in the `NEW_TANG_PIN` definition.
* The Tang server is decommissioned or the keys are permanently lost.

.Prerequisites

* The commands shown in this procedure can be run on the Tang server or on any Linux system that has network
access to the Tang server.

.Procedure

. Validate the Tang server configuration by performing a simple encrypt and decrypt operation on each Tang
server’s configuration as defined in the daemon set.
+
This is an example of an encryption and decryption attempt with a bad thumbprint:
+
[source,terminal]
----
$ echo "okay" | clevis encrypt tang \
  '{"url":"http://tangserver02:7500","thp":"badthumbprint"}' | \
  clevis decrypt
----
+
.Example output
+
[source,terminal]
----
Unable to fetch advertisement: 'http://tangserver02:7500/adv/badthumbprint'!
----
+
This is an example of an encryption and decryption attempt with a good thumbprint:
+
[source,terminal]
----
$ echo "okay" | clevis encrypt tang \
  '{"url":"http://tangserver03:7500","thp":"goodthumbprint"}' | \
  clevis decrypt
----
+
.Example output

+
[source,terminal]
----
okay
----

. After you identify the root cause, remedy the underlying situation:

.. Delete the non-working daemon set.
.. Edit the daemon set definition to fix the underlying issue.  This might include any of the following actions:
+
* Edit a Tang server entry to correct the URL and thumbprint.
* Remove a Tang server that is no longer in service.
* Add a new Tang server that is a replacement for a decommissioned server.

. Distribute the updated daemon set again.

[NOTE]
====
When replacing, removing, or adding a Tang server from a configuration, the rekeying operation will succeed as long as at least one original server is still functional, including the server currently being rekeyed. If none of the original Tang servers are functional or can be recovered, recovery of the system is impossible and you must redeploy the affected nodes.
====

.Verification

Check the logs from each pod in the daemon set to determine whether the rekeying completed successfully. If the rekeying is not successful, the logs might indicate the failure condition.

. Locate the name of the container that was created by the daemon set:
+
[source,terminal]
----
$ oc get pods -A | grep tang-rekey
----
+
.Example output
[source,terminal]
----
openshift-machine-config-operator  tang-rekey-7ks6h  1/1  Running   20 (8m39s ago)  89m
----

. Print the logs from the container. The following log is from a completed successful rekeying operation:
+
[source,terminal]
----
$ oc logs tang-rekey-7ks6h
----
+
.Example output
[source,terminal]
----
Current tang pin:
1: sss '{"t":1,"pins":{"tang":[{"url":"http://10.46.55.192:7500"},{"url":"http://10.46.55.192:7501"},{"url":"http://10.46.55.192:7502"}]}}'
Applying new tang pin: {"t":1,"pins":{"tang":[
  {"url":"http://tangserver01:7500","thp":"WOjQYkyK7DxY_T5pMncMO5w0f6E"},
  {"url":"http://tangserver02:7500","thp":"I5Ynh2JefoAO3tNH9TgI4obIaXI"},
  {"url":"http://tangserver03:7500","thp":"38qWZVeDKzCPG9pHLqKzs6k1ons"}
]}}
Updating binding...
Binding edited successfully
Pin applied successfully
----
