// Module included in the following assemblies:
//
// * etcd/KMS_v2/kms-configuring.adoc

:_mod-docs-content-type: REFERENCE
[id="kms-troubleshooting_{context}"]
= KMS encryption troubleshooting

[role="_abstract"]
You can diagnose and resolve common KMS encryption issues to maintain secure key management and cluster availability.

[id="kms-invalid-configuration_{context}"]
== Invalid KMS configuration

*Symptom:* APIServer resource shows validation errors during KMS encryption configuration.

*Diagnosis:* Check kube-apiserver Operator logs:
[source,terminal]
----
$ oc logs -n openshift-kube-apiserver-operator deploy/kube-apiserver-operator | grep -i "kms\|validation\|error"
----

*Solutions:*

* Verify plugin configuration follows provider requirements
* Ensure all required fields are specified
* Verify plugin is running on all control plane nodes:
+
[source,terminal]
----
$ oc debug node/<node_name> -- chroot /host ls -la /var/run/kmsplugin/kms.sock
----

[id="kms-permissions-errors_{context}"]
== KMS permissions errors

*Symptom:* Encryption migration fails with permission errors.

*Diagnosis:* Check Operator and plugin logs:
[source,terminal]
----
$ oc logs -n openshift-kube-apiserver-operator deploy/kube-apiserver-operator | grep -i "kms\|permission\|access denied"
----

[source,terminal]
----
$ oc debug node/<node_name> -- chroot /host journalctl -u kms-plugin
----

*Solutions:*

* Verify plugin has valid authentication credentials
* Check if credentials have expired
* Ensure plugin principal has encrypt and decrypt permissions
* Verify KMS provider key policy allows plugin access
* Confirm encryption key is enabled and not scheduled for deletion

[id="kms-expired-deleted-key_{context}"]
== Expired or deleted KMS key

*Symptom:* API server cannot decrypt secrets when accessing encrypted resources.

*Diagnosis:* Check logs and verify key status:
[source,terminal]
----
$ oc logs -n openshift-kube-apiserver -l apiserver=true | grep -i "decrypt\|kms.*error"
----

[source,terminal]
----
$ oc debug node/<node_name> -- chroot /host journalctl -u kms-plugin | grep -i "key\|error"
----

*Solutions:*

* Re-enable the encryption key if disabled
* Restore from backup if key was permanently deleted
* Cancel key deletion if scheduled
* Ensure KMS provider maintains access to previous key versions

[WARNING]
====
Deleted KMS keys prevent data recovery. Align key retention with backup policies.
====

[id="kms-api-server-degraded_{context}"]
== API server degraded or unavailable

*Symptom:* API server becomes degraded or unresponsive after enabling KMS encryption.

*Diagnosis:* Check Operator status and logs:
[source,terminal]
----
$ oc get clusteroperator kube-apiserver
----

[source,terminal]
----
$ oc logs -n openshift-kube-apiserver -l apiserver=true --tail=200 | grep -i kms
----

*Solutions:*

* Check network connectivity between control plane and KMS provider
* Verify network policies, firewalls, and routes allow communication
* Monitor KMS provider rate limits and request increases if needed
* Verify DNS resolution and TLS certificate validation
* Confirm plugin is running on all control plane nodes:
+
[source,terminal]
----
$ oc debug node/<node_name> -- chroot /host systemctl status kms-plugin
----

[id="kms-migration-stuck-slow_{context}"]
== Encryption migration stuck or slow

*Symptom:* KMS encryption migration takes unusually long or becomes stuck.

*Diagnosis:* Check Operator status and migration logs:
[source,terminal]
----
$ oc get clusteroperator kube-apiserver
----

[source,terminal]
----
$ oc logs -n openshift-kube-apiserver-operator deploy/kube-apiserver-operator | grep -i migration
----

*Solutions:*

* Migration time depends on data size; monitor progress
* Monitor KMS provider audit logs for rate limiting or throttling events
* Check network performance between control plane and KMS provider

[id="kms-collecting-debug-info_{context}"]
== Collecting debug information

Collect cluster logs:

[source,terminal]
----
$ oc adm must-gather
----

[source,terminal]
----
$ oc get apiserver cluster -o yaml > apiserver.yaml
----

[source,terminal]
----
$ oc logs -n openshift-kube-apiserver-operator deploy/kube-apiserver-operator > kube-apiserver-operator.log
----

[source,terminal]
----
$ oc logs -n openshift-kube-apiserver -l apiserver=true --tail=500 > kube-apiserver.log
----

Collect KMS provider information:

* KMS plugin logs from control plane nodes
* KMS provider audit logs
* KMS provider key policy and permissions
* Authentication credentials status
* Network connectivity test results

[NOTE]
====
Redact credentials, tokens, and sensitive data before sharing logs.
====
