
// Module included in the following assemblies:
//
// * rosa_architecture/rosa_policy_service_definition/rosa-policy-shared-responsibility.adoc

:_mod-docs-content-type: CONCEPT
[id="rosa-policy-incident_{context}"]
= Incident and operations management

Red{nbsp}Hat is responsible for overseeing the service components required for default platform networking.
AWS is responsible for protecting the hardware infrastructure that runs all of the services offered in the AWS Cloud. The customer is responsible for incident and operations management of customer application data and any custom networking the customer has configured for the cluster network or virtual network.

[cols= "2a,3a,3a",options="header"]
|===

|Resource
|Service responsibilities
|Customer responsibilities

|Application networking
|**Red{nbsp}Hat**

- Monitor native OpenShift router
service, and respond to alerts.
|- Monitor health of application routes, and the endpoints behind them.
- Report outages to Red{nbsp}Hat and AWS.

|Cluster networking
|**Red{nbsp}Hat**

- Monitor, alert, and address incidents related to cluster DNS, network plugin connectivity between cluster components, and the default Ingress Controller.
|- Monitor and address incidents related to optional Ingress Controllers, additional Operators installed through the software catalog, and network plugins replacing the default OpenShift CNI plugins.

|Virtual networking management
|**Red{nbsp}Hat**

- Monitor AWS load balancers, Amazon VPC subnets, and AWS service components necessary for default
platform networking. Respond to alerts.
|- Monitor health of AWS load balancer endpoints.
- Monitor network traffic that is optionally configured through Amazon VPC-to-VPC connection, AWS VPN connection, or AWS
Direct Connect for potential issues or
security threats.

|Virtual storage management
|**Red{nbsp}Hat**

- Monitor Amazon EBS volumes attached to cluster nodes and Amazon S3 buckets used for the ROSA service’s built-in container image
registry. Respond to alerts.
|- Monitor health of application data.
- If customer managed AWS KMS keys are
used, create and control the key lifecycle and
key policies for Amazon EBS encryption.

|Platform monitoring
|**Red{nbsp}Hat**

- Maintain a centralized monitoring and alerting system for all ROSA cluster components, site reliability engineer (SRE) services, and underlying AWS accounts.
|
ifdef::openshift-rosa-hcp[]
- Monitor capacity of worker nodes.
- Configure cluster monitoring stack components for monitoring and alerts. 
endif::openshift-rosa-hcp[]

|Incident management
|**Red{nbsp}Hat**

- Raise and manage known incidents.
- Share root cause analysis (RCA) drafts with the customer.
|- Raise known incidents through a support case.

|Infrastructure and data resiliency
|**Red{nbsp}Hat**

- There is no Red{nbsp}Hat-provided backup method available for ROSA clusters with STS.
- Red{nbsp}Hat does not commit to any Recovery Point Objective (RPO) or Recovery Time Objective (RTO).
|- Take regular backups of data and deploy multi-AZ clusters with workloads that follow Kubernetes best practices to ensure high availability within a region.
- If an entire cloud region is unavailable, install a new cluster in a different region and restore apps using backup data.

|Cluster capacity
|**Red{nbsp}Hat**

- Manage the capacity of all control plane and infrastructure nodes on the cluster.
- Evaluate cluster capacity during upgrades and in response to cluster alerts.
|

|AWS software (public AWS services)
|**AWS**

- For information regarding AWS incident and operations management, see link:https://docs.aws.amazon.com/whitepapers/latest/aws-operational-resilience/how-aws-maintains-operational-resilience-and-continuity-of-service.html#incident-management[How AWS maintains operational resilience and continuity of service] in the AWS whitepaper.
|- Monitor health of AWS resources in the
customer account.
- Use IAM tools to apply the appropriate
permissions to AWS resources in the customer account.

|Hardware/AWS global infrastructure
|**AWS**

- For information regarding AWS incident and operations management, see link:https://docs.aws.amazon.com/whitepapers/latest/aws-operational-resilience/how-aws-maintains-operational-resilience-and-continuity-of-service.html#incident-management[How AWS maintains operational
resilience and continuity of service] in the AWS white paper.

|- Configure, manage, and monitor customer applications and data to ensure application and data security controls are properly enforced.

|===

[id="rosa-policy-platform-monitoring_{context}"]
== Platform monitoring
Platform audit logs are securely forwarded to a centralized security information and event monitoring (SIEM) system, where they may trigger configured alerts to the Red{nbsp}Hat SRE team and are also subject to manual review. Audit logs are retained in the SIEM system for one year. Audit logs for a given cluster are not deleted at the time the cluster is deleted.

Red{nbsp}Hat monitors the cluster using monitoring and alerting systems that run on Red{nbsp}Hat managed infrastructure and operate independently of the cluster. Customers retain full access to the Cluster Monitoring Operator stack for their own in-cluster monitoring, alerting, and observability needs.

[id="rosa-policy-incident-management_{context}"]
== Incident management
An incident is an event that results in a degradation or outage of one or more Red{nbsp}Hat services.

An incident can be raised by a customer or a Customer Experience and Engagement (CEE) member through a support case, directly by the centralized monitoring and alerting system, or directly by a member of the SRE team.

Depending on the impact on the service and customer, the incident is categorized in terms of link:https://access.redhat.com/support/offerings/production/sla[severity].

When managing a new incident, Red{nbsp}Hat uses the following general workflow:

. An SRE first responder is alerted to a new incident and begins an initial investigation.
. After the initial investigation, the incident is assigned an incident lead, who coordinates the recovery efforts.
. The incident lead manages all communication and coordination around recovery, including any relevant notifications and support case updates.
. When the incident is resolved a brief summary of the incident and resolution are provided in the customer-initiated support ticket. This summary helps the customers understand the incident and its resolution in more detail.

If customers require more information in addition to what is provided in the support ticket, they can request the following workflow:

. The customer must make a request for the additional information within 5 business days of the incident resolution.
. Depending on the severity of the incident, Red{nbsp}Hat may provide customers with a root cause summary, or a root cause analysis (RCA) in the support ticket. The additional information will be provided within 7 business days for root cause summary and 30 business days for root cause analysis from the incident resolution.

Red{nbsp}Hat also assists with customer incidents raised through support cases.
Red{nbsp}Hat can assist with activities including but not limited to:

* Forensic gathering, including isolating virtual compute
* Guiding compute image collection
* Providing collected audit logs

//Note: The following content will be used again in the future (per OSDOCS:4654)
//[id="backup-recovery_{context}"]
//== Backup and recovery
//All Red Hat OpenShift Service on AWS cluster metadata from OpenShift Cluster Manager is securely backed up by Red Hat. The following table outlines backup and recovery strategies:

//Verify if the corresponding tables in rosa-sdpolicy-platform.adoc and policy-incident.adoc also need to be updated.

//[cols= "3a,2a,2a,3a",options="header"]

//|===
//|Component
//|Snapshot frequency
//|Retention
//|Notes

//.2+|Full object store backup, all cluster persistent volumes (PVs)
//|Daily
//|7 days
//.2+|This is a full backup of all Kubernetes objects like etcd, as well as all PVs in the cluster.

//|Weekly
//|30 days


//|Full object store backup
//|Hourly
//|24 hour
//|This is a full backup of all Kubernetes objects like etcd. No PVs are backed up in this backup schedule.

//|Node root volume
//|Never
//|N/A
//|Nodes are considered to be short-term. Nothing critical should be stored on a node's root volume.

//|===

[id="rosa-policy-cluster-capacity_{context}"]
== Cluster capacity

The impact of a cluster upgrade on capacity is evaluated as part of the upgrade testing process to ensure that capacity is not negatively impacted by new additions to the cluster. During a cluster upgrade, additional worker nodes are added to make sure that total cluster capacity is maintained during the upgrade process.

Capacity evaluations by the Red{nbsp}Hat SRE staff also happen in response to alerts from the cluster, after usage thresholds are exceeded for a certain period of time. Such alerts can also result in a notification to the customer.
