// Module included in the following assemblies:
//
// *scalability_and_performance/cnf-numa-aware-scheduling.adoc

:_mod-docs-content-type: PROCEDURE
[id="cnf-checking-numa-aware-scheduler-logs_{context}"]
= Checking the NUMA-aware scheduler logs

[role="_abstract"]
To troubleshoot problems with the NUMA-aware scheduler, review the scheduler logs. If necessary, increase the log level in the `NUMAResourcesScheduler` custom resource (CR) to capture more detailed diagnostic data.

Acceptable values are `Normal`, `Debug`, and `Trace`, with `Trace` being the most verbose option.

[NOTE]
====
To change the log level of the secondary scheduler, delete the running scheduler resource and re-deploy it with the changed log level. The scheduler is unavailable for scheduling new workloads during this downtime.
====

.Prerequisites

* Installed the {oc-first}.
* You are logged in as a user with `cluster-admin` privileges.

.Procedure

. Delete the currently running `NUMAResourcesScheduler` resource:
+
.. Get the active `NUMAResourcesScheduler` by running the following command:
+
[source,terminal]
----
$ oc get NUMAResourcesScheduler
----
+
.Example output
[source,terminal]
----
NAME                     AGE
numaresourcesscheduler   90m
----
+
.. Delete the secondary scheduler resource by running the following command:
+
[source,terminal]
----
$ oc delete NUMAResourcesScheduler numaresourcesscheduler
----
+
.Example output
[source,terminal]
----
numaresourcesscheduler.nodetopology.openshift.io "numaresourcesscheduler" deleted
----

. Save the following YAML in the file `nro-scheduler-debug.yaml`. This example changes the log level to `Debug`:
+
[source,yaml,subs="attributes+"]
----
apiVersion: nodetopology.openshift.io/v1
kind: NUMAResourcesScheduler
metadata:
  name: numaresourcesscheduler
spec:
  imageSpec: "registry.redhat.io/openshift4/noderesourcetopology-scheduler-container-rhel8:v{product-version}"
  logLevel: Debug
# ...
----

. Create the updated `Debug` logging `NUMAResourcesScheduler` resource by running the following command:
+
[source,terminal]
----
$ oc create -f nro-scheduler-debug.yaml
----
+
.Example output
[source,terminal]
----
numaresourcesscheduler.nodetopology.openshift.io/numaresourcesscheduler created
----

.Verification

. Check that the NUMA-aware scheduler was successfully deployed:
+
.. Run the following command to check that the CRD is created successfully:
+
[source,terminal]
----
$ oc get crd | grep numaresourcesschedulers
----
+
.Example output
[source,terminal]
----
NAME                                                              CREATED AT
numaresourcesschedulers.nodetopology.openshift.io                 2022-02-25T11:57:03Z
----
+
.. Check that the new custom scheduler is available by running the following command:
+
[source,terminal]
----
$ oc get numaresourcesschedulers.nodetopology.openshift.io
----
+
.Example output
[source,terminal]
----
NAME                     AGE
numaresourcesscheduler   3h26m
----

. Check that the logs for the scheduler shows the increased log level:
+
.. Get the list of pods running in the `openshift-numaresources` namespace by running the following command:
+
[source,terminal]
----
$ oc get pods -n openshift-numaresources
----
+
.Example output
[source,terminal]
----
NAME                                               READY   STATUS    RESTARTS   AGE
numaresources-controller-manager-d87d79587-76mrm   1/1     Running   0          46h
numaresourcesoperator-worker-5wm2k                 2/2     Running   0          45h
numaresourcesoperator-worker-pb75c                 2/2     Running   0          45h
secondary-scheduler-7976c4d466-qm4sc               1/1     Running   0          21m
----
+
.. Get the logs for the secondary scheduler pod by running the following command:
+
[source,terminal]
----
$ oc logs secondary-scheduler-7976c4d466-qm4sc -n openshift-numaresources
----
+
.Example output
[source,terminal]
----
...
I0223 11:04:55.614788       1 reflector.go:535] k8s.io/client-go/informers/factory.go:134: Watch close - *v1.Namespace total 11 items received
I0223 11:04:56.609114       1 reflector.go:535] k8s.io/client-go/informers/factory.go:134: Watch close - *v1.ReplicationController total 10 items received
I0223 11:05:22.626818       1 reflector.go:535] k8s.io/client-go/informers/factory.go:134: Watch close - *v1.StorageClass total 7 items received
I0223 11:05:31.610356       1 reflector.go:535] k8s.io/client-go/informers/factory.go:134: Watch close - *v1.PodDisruptionBudget total 7 items received
I0223 11:05:31.713032       1 eventhandlers.go:186] "Add event for scheduled pod" pod="openshift-marketplace/certified-operators-thtvq"
I0223 11:05:53.461016       1 eventhandlers.go:244] "Delete event for scheduled pod" pod="openshift-marketplace/certified-operators-thtvq"
----
