Intermittent request aborted from proxied service
Hi Andy -- we've been running Piko in Prod for a couple of weeks now, and have noticed that the services proxied by Piko will sometimes throw `request aborted` errors when responding to requests.
The error message is:
```
External BadRequestError: BadRequestError: request aborted
at IncomingMessage.onAborted (/main/.yarn/cache/raw-body-npm-2.5.1-9dd1d9fff9-280bedc12d.zip/node_modules/raw-body/index.js:238:10)
at IncomingMessage.emit (node:events:517:28)
at IncomingMessage.emit (node:domain:489:12)
at IncomingMessage._destroy (node:_http_incoming:224:10)
at _destroy (node:internal/streams/destroy:109:10)
at IncomingMessage.destroy (node:internal/streams/destroy:71:5)
at abortIncoming (node:_http_server:781:9)
at socketOnClose (node:_http_server:775:3)
at TLSSocket.emit (node:events:529:35)
at TLSSocket.emit (node:domain:489:12)
at node:net:350:12
at Socket.done (node:_tls_wrap:657:7)
at Object.onceWrapper (node:events:632:26)
at Socket.emit (node:events:517:28)
at Socket.emit (node:domain:489:12)
at TCP.<anonymous> (node:net:350:12)
at TCP.callbackTrampoline (node:internal/async_hooks:128:17)
```
Our setup is:
1. Service to be proxied running in ECS, with the Piko agent running as a process in the same container
2. 2 or more instances of the same service run at any given time in our ECS service, and they all use the same endpoint-id to connect to our Piko cluster (hosted on EKS)
3. Our backend attempts to contact the service through the Piko cluster, and about 5-10% of all requests end in `request aborted` issues
1. We've noticed that while the NodeJS service throws a `request aborted` status message, with an error code of 400, Piko converts that into a 502 `upstream unreachable` when reading the response from the Piko clsuter -- presumably due to [this](https://github.com/andydunstall/piko/blob/c4d56d2b09d751bf40828c8bc3c4477f4464a341/agent/reverseproxy/reverseproxy.go#L86) code
Things we've tried:
1. Confirmed that request headers and content-types are respected, including using `Accept-Encoding: identity` to prevent the Golang builtin http transporter from messing around with response types
2. Increased the connection idle timeout on the Piko cluster's ingress to 120 seconds
3. Confirmed that our services proxied by the Piko agent, nor the Piko agent itself, are dying, and are still up and running when the requests are abruptly aborted
Do you have any thoughts around why such a thing might be happening? Please let me know if you'd like more details!
关闭于 2025-04-15 12 条评论