// Module included in the following assemblies:
//
// * scalability_and_performance/telco_core_ref_design_specs/telco-core-rds.adoc

:_mod-docs-content-type: REFERENCE
[id="telco-hub-observability_{context}"]
= Observability

[role="_abstract"]
The {rh-rhacm-first} multicluster engine Observability component provides centralized aggregation and visualization of metrics and alerts for all managed clusters.
To balance performance and data analysis, the monitoring service maintains a subset list of aggregated metrics that are collected at a downsampled interval.
The metrics can be accessed on the hub through a set of different preconfigured dashboards.

Observability installation::
The primary custom resource (CR) to enable and configure the observability service is the `MulticlusterObservability` CR, which defines the following settings:

* Configurable retention settings.
* Storage for the different components: `thanos receive`, `thanos compact`, `thanos rule`, `thanos store` sharding, `alertmanager`.
* The `metadata.annotations.mco-disable-alerting="true"` annotation that enables tuning for the monitoring configuration on managed clusters.
+
[NOTE]
====
Without this setting the Observability component attempts to configure the managed cluster monitoring configuration.
With this value set you can merge your desired configuration with the necessary Observability configuration of alert forwarding into the managed cluster monitoring `ConfigMap` object.
When the Observability service is enabled {rh-rhacm} will deploy to each managed cluster a workload to push metrics and alerts generated by local Monitoring to the hub cluster.
The metrics and alerts to be forwarded from the managed cluster to the hub, are defined by a `ConfigMap` CR in the `open-cluster-management-addon-observability` namespace.
You can also specify custom metrics. For more information, see link:https://docs.redhat.com/en/documentation/red_hat_advanced_cluster_management_for_kubernetes/2.15/html-single/observability/index#adding-custom-metrics[Adding custom metrics].
====

Alertmananger configuration::
+
--
* The hub cluster provides an Observability Alertmanager that can be configured to push alerts to external systems, for example, email.
The Alertmanager is enabled by default.
* You must configure alert forwarding.
* When the Alertmanager is enabled but not configured, the hub Alertmanager does not forward alerts externally.
* When Observability is enabled, the managed clusters can be configured to send alerts to any endpoint including the hub Alertmanager.
* When a managed cluster is configured to forward alerts to external sources, alerts are not routed through the hub cluster Alertmanager.
* Alert state is available as a metric.
* When observability is enabled, the managed cluster alert states are included in the subset of metrics forwarded to the hub cluster and are available through Observability dashboards.
--

Limits and requirements::
* Observability requires persistent object storage for long-term metrics.
For more information, see "Storage requirements".

Engineering considerations::
* Forwarding of metrics is a subset of the full metric data.
It includes only the metrics defined in the `observability-metrics-allowlist` config map and any custom metrics added by the user.
* Metrics are forwarded at a downsampled rate.
Metrics are forwarded by taking the latest datapoint at a 5 minute interval (or as defined by the `MultiClusterObservability` CR configuration).
* A network outage may lead to a loss of metrics forwarded to the hub cluster during that interval.
This can be mitigated if metrics are also forwarded directly from managed clusters to an external metrics collector in the providers network.
Full resolution metrics are available on the managed cluster.
* In addition to default metrics dashboards on the hub, users may define custom dashboards.
* The reference configuration is sized based on 15 days of metrics storage by the hub cluster for 3500 {sno} clusters.
If longer retention or other managed cluster topology or sizing is required, the storage calculations must be updated and sufficient storage capacity be maintained.
For more information about calculating new values, see "Storage requirements".
