// Module included in the following assemblies:
//
// * observability/logging/troubleshooting/troubleshooting-logging-alerts.adoc

:_mod-docs-content-type: PROCEDURE
[id="es-disk-space-low_{context}"]
= Elasticsearch disk space is running low

Elasticsearch is predicted to run out of disk space within the next 6 hours based on current disk usage. Use the following procedure to troubleshoot this alert.

.Procedure

. Get the disk space of the Elasticsearch node:
+
[source,terminal]
----
$ for pod in `oc -n openshift-logging get po -l component=elasticsearch -o jsonpath='{.items[*].metadata.name}'`; \
  do echo $pod; oc -n openshift-logging exec -c elasticsearch $pod \
  -- df -h /elasticsearch/persistent; done
----

. In the command output, check the `Avail` column to determine the free disk space on that node.
+
.Example output
[source,terminal]
----
elasticsearch-cdm-kcrsda6l-1-586cc95d4f-h8zq8
Filesystem      Size  Used Avail Use% Mounted on
/dev/nvme1n1     19G  522M   19G   3% /elasticsearch/persistent
elasticsearch-cdm-kcrsda6l-2-5b548fc7b-cwwk7
Filesystem      Size  Used Avail Use% Mounted on
/dev/nvme2n1     19G  522M   19G   3% /elasticsearch/persistent
elasticsearch-cdm-kcrsda6l-3-5dfc884d99-59tjw
Filesystem      Size  Used Avail Use% Mounted on
/dev/nvme3n1     19G  528M   19G   3% /elasticsearch/persistent
----

. Increase the disk space on all nodes. If increasing the disk space is not possible, try adding a new data node to the cluster, or decrease the total cluster redundancy policy.

. To check the current `redundancyPolicy`, run the following command:
+
[source,terminal]
----
$ oc -n openshift-logging get es elasticsearch -o jsonpath='{.spec.redundancyPolicy}'
----
+
If you are using a `ClusterLogging` resource on your cluster, run the following command:
+
[source,terminal]
----
$ oc -n openshift-logging get cl \
  -o jsonpath='{.items[*].spec.logStore.elasticsearch.redundancyPolicy}'
----
+
If the cluster `redundancyPolicy` value is higher than the `SingleRedundancy` value, set it to the `SingleRedundancy` value and save this change.

. If the preceding steps do not fix the issue, delete the old indices.
.. Check the status of all indices on Elasticsearch by running the following command:
+
[source,terminal]
----
$ oc exec -n openshift-logging -c elasticsearch $ES_POD_NAME -- indices
----

.. Identify an old index that can be deleted.
.. Delete the index by running the following command:
+
[source,terminal]
----
$ oc exec -n openshift-logging -c elasticsearch $ES_POD_NAME \
  -- es_util --query=<elasticsearch_index_name> -X DELETE
----
