ITADN

Operator infinite loop on transient errors from kube API

#378Closedporridge 创建于 2024-08-20
P
porridgecommented
[This change](https://github.com/operator-framework/helm-operator-plugins/compare/main...porridge:helm-operator-plugins:repro) to unit tests shows the issue: ``` • [FAILED] [0.011 seconds] Updater when an update is a change [It] should apply an update status function /home/mowsiany/go/src/github.com/operator-framework/helm-operator-plugins/pkg/reconciler/internal/updater/updater_test.go:102 [FAILED] HaveLen matcher expects a string/array/map/channel/slice. Got: <nil>: nil In [It] at: /home/mowsiany/go/src/github.com/operator-framework/helm-operator-plugins/pkg/reconciler/internal/updater/updater_test.go:108 @ 08/20/24 07:39:09.843 ``` When there is a transient error from the API server, the updater never retries updates, as long as the update function correctly returns `false` if it has no effect. This is because the Apply function [reuses](https://github.com/operator-framework/helm-operator-plugins/blob/07ca439983245f8480f7dd8b6c4cbfd063ad85c4/pkg/reconciler/internal/updater/updater.go#L91) the same object on each update attempt. After the first attempt, subsequent invocations of the update functions show nothing is changed, so subsequent attempts are not made even if the first one failed. What is worse, and could be considered a different issue on its own, in the deletion case the code proceeds to [wait forever on the deletion to happen](https://github.com/operator-framework/helm-operator-plugins/blob/07ca439983245f8480f7dd8b6c4cbfd063ad85c4/pkg/reconciler/reconciler.go#L733). But this never happens, because the updates done by `doUninstall` correctly return `false` on subsequent updates. In my case the transient error was a 500 from the API server, caused in turn by a misbehaving validation webhook.
关闭于 2024-09-04 1 条评论