ITADN

privateapi: seed self into peers to avoid discovery self-wait

#75Pull Requesthakman 创建于 2026-06-28
size/Lcncf-cla: yes
H
hakmancommented
## Problem On a single-member **first start**, the controller waits for peer discovery to "find itself" before it can act. Measured from kops e2e CI, this dominates cold-start time: - Single-member first start is **~97s on Azure**, of which **~62s** is the controller logging `cannot find self in list of peers []` ([`pkg/controller/controller.go`](https://github.com/kubernetes-sigs/etcd-manager/blob/main/pkg/controller/controller.go) ~L204) once per 10s cycle. - Root cause: the Azure discovery provider's **first** poll fails because the just-mounted etcd data disk isn't visible yet (`no data disk found`), and `DiscoveryPollInterval` is **60s**, so the next (successful) poll is a full minute later. - AWS/GCE happen to return self on the first poll, so they don't stall — but the fix helps them too (and removes the flat 2s startup sleep everywhere). ## Changes (startup/discovery only) 1. **Seed self into the peer set at construction.** `NewServer` now registers our own identity (`myInfo`) as a peer immediately, marks it healthy, and starts the normal loopback self-ping to keep it healthy — reusing the same "create peer + start ping loop" path as a discovered peer. `Peers()` includes self without waiting for `discovery.Poll()` to return it. **Other peers still come only from discovery.** 2. **Fast-retry discovery before the first hit.** `runDiscovery` retries every **2s** until the first successful, non-empty poll, then settles back to the **60s** steady-state interval. A transient cloud-API miss on the initial poll now costs seconds, not a minute. 3. **Bounded self-ready poll instead of the flat 2s sleep.** The pre-controller `time.Sleep(2s)` is replaced with a poll (every 50ms, capped at 2s) that waits until `Peers()` contains `MyPeerId()`. It's a bounded *poll*, not a removal, so the controller never starts before self is visible and then sleeps a full `CycleInterval` (~10s regression). ## Why it's safe Leader election semantics are unchanged. Lowest-id election ([`controller.go`](https://github.com/kubernetes-sigs/etcd-manager/blob/main/pkg/controller/controller.go) ~L209), `BecomeLeader` all-peers-ack with omitted-peer reject ([`pkg/privateapi/leadership.go`](https://github.com/kubernetes-sigs/etcd-manager/blob/main/pkg/privateapi/leadership.go)), and resign-on-unacked ([`controller.go`](https://github.com/kubernetes-sigs/etcd-manager/blob/main/pkg/controller/controller.go) ~L279) all still gate multi-member correctness. The transient "`Peers()` = [self only]" state is **already reachable today**: on an AWS/GCE cold start, discovery returns all volumes, but peers only become healthy after a successful ping, so a node that boots first already sees only itself. This change just makes self appear immediately rather than after the first discovery poll — it does not introduce a new state, and the election protocol already self-corrects it. Seeding self also slightly *strengthens* split-brain protection: a node always knows itself, so it rejects any leader whose view omits it. Quorum and quarantine logic are untouched. Only `cmd/etcd-manager/main.go` and `pkg/privateapi/*` are changed. ## Tests - New `pkg/privateapi/server_test.go`: - `Peers()` includes self immediately after construction (before any discovery poll). - Multi-member discovery still adds other peers (self-seeding doesn't block them). - A failed first discovery poll (the Azure case) keeps self in the peer list and reports the error/count for fast-retry. - `go build ./...`, `go vet`, and `go test -race ./pkg/privateapi/... ./cmd/...` pass. - Full suite passes, including `test/integration` — notably `TestClusterWithThreeMembers` (multi-member formation) and `TestWeOnlyFormASingleCluster` (split-brain prevention). Expected saving: **~62s** on Azure single-member cold start, plus removal of the `cannot find self` stalls and the ~2s startup sleep on all providers.
合并状态:未合并 1 条评论