// Module included in the following assemblies:
//
// * observability/monitoring/troubleshooting-monitoring-issues.adoc
// * support/troubleshooting/investigating-monitoring-issues.adoc

:_mod-docs-content-type: PROCEDURE
[id="investigating-why-user-defined-metrics-are-unavailable_{context}"]
= Investigating why user-defined project metrics are unavailable

[role="_abstract"]
`ServiceMonitor` resources enable you to determine how to use the metrics exposed by a service in user-defined projects. Follow the steps outlined in this procedure if you have created a `ServiceMonitor` resource but cannot see any corresponding metrics in the Metrics UI.

.Prerequisites

ifndef::openshift-rosa,openshift-dedicated[]
* You have access to the cluster as a user with the `cluster-admin` role.
endif::openshift-rosa,openshift-dedicated[]
ifdef::openshift-rosa,openshift-dedicated[]
* You have access to the cluster as a user with the `dedicated-admin` role.
endif::openshift-rosa,openshift-dedicated[]
* You have installed the {oc-first}.
* You have enabled and configured monitoring for user-defined projects.
* You have created a `ServiceMonitor` resource.

.Procedure

. Ensure that your project and resources are not excluded from user workload monitoring. The following examples use the `ns1` project.

.. Verify that the project _does not_ have the `openshift.io/user-monitoring=false` label attached:
+
[source,terminal]
----
$ oc get namespace ns1 --show-labels | grep 'openshift.io/user-monitoring=false'
----
+
[NOTE]
====
The default label set for user workload projects is `openshift.io/user-monitoring=true`. However, the label is not visible unless you manually apply it.
====

.. Verify that the `ServiceMonitor` and `PodMonitor` resources _do not_ have the `openshift.io/user-monitoring=false` label attached. The following example checks the `prometheus-example-monitor` service monitor.
+
[source,terminal]
----
$ oc -n ns1 get servicemonitor prometheus-example-monitor --show-labels | grep 'openshift.io/user-monitoring=false'
----

.. If the label is attached, remove the label:
+
.Example of removing the label from the project
[source,terminal]
----
$ oc label namespace ns1 'openshift.io/user-monitoring-'
----
+
.Example of removing the label from the resource
[source,terminal]
----
$ oc -n ns1 label servicemonitor prometheus-example-monitor 'openshift.io/user-monitoring-'
----
+
.Example output
[source,terminal]
----
namespace/ns1 unlabeled
----

. Check that the corresponding labels match in the service and `ServiceMonitor` resource configurations. The following examples use the `prometheus-example-app` service, the `prometheus-example-monitor` service monitor, and the `ns1` project.
.. Obtain the label defined in the service.
+
[source,terminal]
----
$ oc -n ns1 get service prometheus-example-app -o yaml
----
+
.Example output
[source,terminal]
----
  labels:
    app: prometheus-example-app
----
+
.. Check that the `matchLabels` definition in the `ServiceMonitor` resource configuration matches the label output in the previous step.
+
[source,terminal]
----
$ oc -n ns1 get servicemonitor prometheus-example-monitor -o yaml
----
+
.Example output
[source,yaml]
----
apiVersion: v1
kind: ServiceMonitor
metadata:
  name: prometheus-example-monitor
  namespace: ns1
spec:
  endpoints:
  - interval: 30s
    port: web
    scheme: http
  selector:
    matchLabels:
      app: prometheus-example-app
----
+
[NOTE]
====
You can check service and `ServiceMonitor` resource labels as a developer with view permissions for the project.
====

. Inspect the logs for the Prometheus Operator in the `openshift-user-workload-monitoring` project.
.. List the pods in the `openshift-user-workload-monitoring` project:
+
[source,terminal]
----
$ oc -n openshift-user-workload-monitoring get pods
----
+
.Example output
[source,terminal]
----
NAME                                   READY   STATUS    RESTARTS   AGE
prometheus-operator-776fcbbd56-2nbfm   2/2     Running   0          132m
prometheus-user-workload-0             5/5     Running   1          132m
prometheus-user-workload-1             5/5     Running   1          132m
thanos-ruler-user-workload-0           3/3     Running   0          132m
thanos-ruler-user-workload-1           3/3     Running   0          132m
----
+
.. Obtain the logs from the `prometheus-operator` container in the `prometheus-operator` pod. In the following example, the pod is called `prometheus-operator-776fcbbd56-2nbfm`:
+
[source,terminal]
----
$ oc -n openshift-user-workload-monitoring logs prometheus-operator-776fcbbd56-2nbfm -c prometheus-operator
----
+
If there is a issue with the service monitor, the logs might include an error similar to this example:
+
[source,terminal]
----
level=warn ts=2020-08-10T11:48:20.906739623Z caller=operator.go:1829 component=prometheusoperator msg="skipping servicemonitor" error="it accesses file system via bearer token file which Prometheus specification prohibits" servicemonitor=eagle/eagle namespace=openshift-user-workload-monitoring prometheus=user-workload
----

. Review the target status for your endpoint on the *Metrics targets* page in the {product-title} web console UI.
.. Log in to the {product-title} web console and go to *Observe* → *Targets*.

.. Locate the metrics endpoint in the list, and review the status of the target in the *Status* column.

.. If the *Status* is *Down*, click the URL for the endpoint to view more information on the *Target Details* page for that metrics target.

. Configure debug level logging for the Prometheus Operator in the `openshift-user-workload-monitoring` project.
.. Edit the `user-workload-monitoring-config` `ConfigMap` object in the `openshift-user-workload-monitoring` project:
+
[source,terminal]
----
$ oc -n openshift-user-workload-monitoring edit configmap user-workload-monitoring-config
----
+
.. Add `logLevel: debug` for `prometheusOperator` under `data/config.yaml` to set the log level to `debug`:
+
[source,yaml]
----
apiVersion: v1
kind: ConfigMap
metadata:
  name: user-workload-monitoring-config
  namespace: openshift-user-workload-monitoring
data:
  config.yaml: |
    prometheusOperator:
      logLevel: debug
# ...
----
+
.. Save the file to apply the changes. The affected `prometheus-operator` pod is automatically redeployed.
+
.. Confirm that the `debug` log-level has been applied to the `prometheus-operator` deployment in the `openshift-user-workload-monitoring` project:
+
[source,terminal]
----
$ oc -n openshift-user-workload-monitoring get deploy prometheus-operator -o yaml |  grep "log-level"
----
+
.Example output
[source,terminal]
----
        - --log-level=debug
----
+
Debug level logging will show all calls made by the Prometheus Operator.
+
.. Check that the `prometheus-operator` pod is running:
+
[source,terminal]
----
$ oc -n openshift-user-workload-monitoring get pods
----
+
[NOTE]
====
If an unrecognized Prometheus Operator `loglevel` value is included in the config map, the `prometheus-operator` pod might not restart successfully.
====
+
.. Review the debug logs to see if the Prometheus Operator is using the `ServiceMonitor` resource. Review the logs for other related errors.
