[fm] File a case, create active problem for "sagas which are stuck"
fault-management
Context:
- We've seen systems which are stuck in unwinding
- We've seen steno issues (e.g. https://github.com/oxidecomputer/steno/issues/17)
- Stuck sagas are particularly problematic because our upgrade story expects us to drain all sagas before a nexus upgrade
As a result, "stuck sagas" have a plethora of risks: they can effectively leak resources they've provisioned, cause a risk towards update availability, etc.
This issue tracks the integration with fault management. We should first answer:
- "What does stuck mean", in the context of a saga? This could be based on age, progress, state, etc.
- "When we encounter a stuck saga, what should we do?" in the short-term, this may simply be "contact support". It seems difficult to have more specific guidance here, considering that sagas are a framework for arbitrary code execution. However, surfacing the risk to operators seems worthwhile, considering the downsides of letting stuck sagas fester.
6 条评论