# January 23rd, 2019 Prow Outage

Created By: Benjamin Elder <bentheelder@google.com>
Last Modified: 2019-01-23

## Summary

The main Kubernetes production Prow deployment, [prow.k8s.io] stopped responding
to GitHub events due to the webhook handler ("[hook]") being down.

This resulted in an outage of unknown length bounded to at maximum 16 hours.
It appears to _actually_ be bounded by around 4-5 hours based on user reports,
but the start time is not fully known.

## Impact

Prow did not respond to any github events, including our various slash commands,
test triggering due to PR changes, etc.

Already scheduled, and time-based scheduled test running continued, 
but using config prior to merging the non-validating config.

## Root Cause

When the job config was [automatically updated] from a [preset refactor PR] 
components failed to load the new configuration due to errors like:

> {"component":"hook","error":"duplicated preset label : preset-aws-credential","level":"fatal","msg":"Error starting config agent.","time":"2019-01-23T16:08:30Z"}

This is because the PR refactored some job configuration presets to use
a common label key with different label values. The config "agent" validation
rejected this format, but for some reason this was not caught in presubmit.

A [previous PR][preset config behavior] enabled this behavior but had not 
been deployed tor prow.k8s.io yet.

At some time after this config update, the [hook] pods went into 
CrashloopBackoff with this error, causing our webhook endpoint 
(a GKE load balancer in front of the deployment) to serve 502s.


## Detection

This was "detected" by project members noticing various failures of Prow to 
respond to events. After informing #sig-testing and #sig-testing-ops,
@stevekuznetsov pinged @BenTheElder to check on this, as our oncall was
not online yet.

## Resolution

Initially a revert of a related configuration change was made, manually
as the CI could not respond to GitHub. This was insufficient, and a follow
up PR refactoring the configuration to use distinct preset keys was made.

Both of these changes were manually merged due to inability to trigger the
presubmits.

The configuration was then manually re-synced to the cluster using a break-glass
[config repair script].


## Lessons Learned

### What went well

Most prow components continued to function correctly while using the previuosly
valid configuration they had already loaded. Unfortunately for some reason
hook did not, possibly because it restarted and no longer was able to load
any configuration.

### What went wrong

PR review failure + config versioning:

A PR merged that changed [preset config behavior] along with validation behavior,
and took advantage of this new behavior. Because we do not deploy a new prow
instance on every merge, this validation was now not correct for the deployed
prow version.

Currently the only safe way to do this would be:

 - Loosen the config loading behavior in a PR, ensure that tight validation
   matching the current Prow deployment remains in effect. This can be tricky
   when changing that behavior.
 - Deploy an updated Prow
 - Loosen the config validation behavior
 - Begin leveraging the new behavior

This should have been caught in review.

### Where we got lucky

We already had tooling to resolve this from the previous outage 
(the [config repair script])

## Action Items

- We should consider using a versioned API for Prow config, and banning 
potentially incompatible changes from merging without moving to a new API version

- We should have alerting on config load failures. We should be able to configure
  stackdriver logging to send an alert when these occur.

## Timeline

All times in PST

2019-01-22

- 2:54 PM PST: [config behavior change PR][preset config behavior] merged

- 3:28 PM PST: bad config [uploaded][automatically updated]

2019-01-23

- 4:49 AM: first [recorded instance] of a user seeing Prow being unresponsive to events

- 7:10 AM: first [#testing-ops recorded instance], a channel some have notifications enabled for

- 7:18 AM: [oncall first pinged]

- 8:02 AM: @stevekuznetsov reports that GitHub webhooks appear to be working fine on their end,
           first report that 503 responses are being served in response to our webhooks.

- 8:04 AM: @stevekuznetsov pings @BenTheElder in #testing-ops to take a look

- 8:07 AM: @BenTheElder begins looking, identifies that [hook] is in crash loop backoff

- 8:12 AM: @BentTheElder: root cause identified, revert / config fix begins, 
           configmap is rewritten with the repair script

- 8:27 AM: @BenTheElder announces resolution is confirmed complete

- 8:36 AM: kubernetes-dev mailing list is notified of the outage, fixed status,
           and potential impact, along with links to track details.


## Appendix

[Discussion in #testing-ops]
TODO: k-dev email


[prow.k8s.io]: https://prow.k8s.io
[hook]: https://github.com/kubernetes/test-infra/tree/master/prow/cmd/hook
[preset refactor PR]: https://github.com/kubernetes/test-infra/pull/10886
[preset config behavior]: https://github.com/kubernetes/test-infra/pull/10868/files
[automatically updated]: https://github.com/kubernetes/test-infra/pull/10886#issuecomment-456605785
[config repair script]: https://github.com/kubernetes/test-infra/blob/1cdb83860cd2e96a4da45bcf88c543530c84ffb1/experiment/maintenance/recreate_prow_configmaps.py
[Dicussion in #testing-ops]: https://kubernetes.slack.com/archives/C7J9RP96G/p1548256258161200
[recorded instance]: https://kubernetes.slack.com/archives/C09QZ4DQB/p1548247766835300
[#testing-ops recorded instance]: https://kubernetes.slack.com/archives/C7J9RP96G/p1548256258161200
[oncall first pinged]: https://kubernetes.slack.com/archives/C7J9RP96G/p1548256737161700
