// Module included in the following assemblies:
//
// * scalability_and_performance/low_latency_tuning/cnf-performing-platform-verification-latency-tests.adoc

:_mod-docs-content-type: REFERENCE
[id="cnf-measuring-latency_{context}"]
= Measuring latency

[role="_abstract"]
To accurately measure system latency, use the `hwlatdetect`, `cyclictest`, and `oslat` tools provided in the `cnf-tests` image. Evaluating these metrics helps you identify and resolve performance delays in your environment.

Each tool has a specific use. Use the tools in sequence to achieve reliable test results.

hwlatdetect:: Measures the baseline that the bare-metal hardware can achieve. Before proceeding with the next latency test, ensure that the latency reported by `hwlatdetect` meets the required threshold because you cannot fix hardware latency spikes by operating system tuning.

cyclictest:: Verifies the real-time kernel scheduler latency after `hwlatdetect` passes validation. The `cyclictest` tool schedules a repeated timer and measures the difference between the desired and the actual trigger times. The difference can uncover basic issues with the tuning caused by interrupts or process priorities. The tool must run on a real-time kernel.

oslat:: Behaves similarly to a CPU-intensive DPDK application and measures all the interruptions and disruptions to the busy loop that simulates CPU heavy data processing.

The tests introduce the following environment variables:

.Latency test environment variables
[cols="1,3a", options="header"]
|====
|Environment variables
|Description

|`LATENCY_TEST_DELAY`
|Specifies the amount of time in seconds after which the test starts running. You can use the variable to allow the CPU manager reconcile loop to update the default CPU pool. The default value is 0.

|`LATENCY_TEST_CPUS`
|Specifies the number of CPUs that the pod running the latency tests uses. If you do not set the variable, the default configuration includes all isolated CPUs. When `LATENCY_TEST_MEMORY` is unset, this value is also used to calculate the default memory request and limit for the latency test pod.

|`LATENCY_TEST_MEMORY`
|Specifies the memory request and limit for the pod that runs the latency tests. If you do not set the variable, the test allocates 32Mi of memory per `LATENCY_TEST_CPUS`, with a minimum of 1Gi. To override the calculated value, set `LATENCY_TEST_MEMORY` to a valid Kubernetes quantity greater than `0`, for example `2Gi`.

|`LATENCY_TEST_RUNTIME`
|Specifies the amount of time in seconds that the latency test must run. The default value is 300 seconds.

[NOTE]
====
To prevent the Ginkgo 2.0 test suite from timing out before the latency tests complete, set the `-ginkgo.timeout` flag to a value greater than `LATENCY_TEST_RUNTIME` + 2 minutes. If you also set a `LATENCY_TEST_DELAY` value then you must set `-ginkgo.timeout` to a value greater than `LATENCY_TEST_RUNTIME` + `LATENCY_TEST_DELAY` + 2 minutes. The default timeout value for the Ginkgo 2.0 test suite is 1 hour.
====

|`HWLATDETECT_MAXIMUM_LATENCY`
|Specifies the maximum acceptable hardware latency in microseconds for the workload and operating system. If you do not set the value of `HWLATDETECT_MAXIMUM_LATENCY` or `MAXIMUM_LATENCY`, the tool compares the default expected threshold (20μs) and the actual maximum latency in the tool itself. Then, the test fails or succeeds accordingly.

|`CYCLICTEST_MAXIMUM_LATENCY`
|Specifies the maximum latency in microseconds that all threads expect before waking up during the `cyclictest` run. If you do not set the value of `CYCLICTEST_MAXIMUM_LATENCY` or `MAXIMUM_LATENCY`, the tool skips the comparison of the expected and the actual maximum latency.

|`OSLAT_MAXIMUM_LATENCY`
|Specifies the maximum acceptable latency in microseconds for the `oslat` test results. If you do not set the value of `OSLAT_MAXIMUM_LATENCY` or `MAXIMUM_LATENCY`, the tool skips the comparison of the expected and the actual maximum latency.

|`MAXIMUM_LATENCY`
|Unified variable that specifies the maximum acceptable latency in microseconds. Applicable for all available latency tools.
|====

[NOTE]
====
Variables that are specific to a latency tool take precedence over unified variables. For example, if `OSLAT_MAXIMUM_LATENCY` is set to 30 microseconds and `MAXIMUM_LATENCY` is set to 10 microseconds, the `oslat` test will run with maximum acceptable latency of 30 microseconds.
====
