grpc-go sometimes may always route requests to one endpoint
Type: Question
In etcd client SDK, users may configure multiple endpoints, but etcd always [connect/dial the first target](https://github.com/etcd-io/etcd/blob/95f9d8d3f1bafed56b7e59c4b2a669e3b732b8bc/client/v3/client.go#L333-L334) initially, and set all endpoints to grpc-go using [Resolver.UpdateState](https://github.com/grpc/grpc-go/blob/v1.78.0/resolver/manual/manual.go#L110) afterwards. Everything is working well.
The etcd client SDK has a [retry mechanism](https://github.com/etcd-io/etcd/blob/0cbc031e7f224102bd1ec42a81f659a2b8c3350c/client/v3/retry_interceptor.go#L54-L66) in the unary interceptor configured via [grpc.WithUnaryInterceptor](https://github.com/etcd-io/etcd/blob/0cbc031e7f224102bd1ec42a81f659a2b8c3350c/client/v3/client.go#L271)).
When a unary RPC fails, the interceptor automatically retries the request, and it is supposed to route the retry to another endpoint in round-robin mode. I tested this many times locally and confirmed it works as expected.
However, in production, sometimes retries are always routed to the same endpoint, even though multiple endpoints are available.
I have seen this issue multiple times (not consistently). The latest case was with grpc-go 1.71.1. The OS should be Linux, but I do not yet know the exact version.
Does anyone have any idea what could cause this behavior?
2 条评论