Re-reconcile latest desired state on Executor side after an FE controller is fully shutdown
There's a bug when Server sometimes sends empty desired state to Executor with a quick follow of a correct full state. This results in all FEs getting terminated first. Terminating FEs takes time, while this happens Server sends another correct desired state with the FEs that are currently getting terminated. As the FEs still exist Executor reconciliation passes and does nothing to create the FEs again. Then if there's no new allocs assigned to Executor then it takes 5 minutes for Executor to learn that it needs to create the FEs again (this happens after 5 minutes when Executor reconnects on idle desired states stream).
The fix to this is for Executor to re-reconcile latest desired state it received from Server after each FE is deleted.
关闭于 2025-12-16 2 条评论