High xray client memory consumption for xhttp/http2 server
### Integrity requirements
- [x] I have read all the comments in the issue template and ensured that this issue meet the requirements.
- [x] I confirm that I have read the documentation, understand the meaning of all the configuration items I wrote, and did not pile up seemingly useful options or default values.
- [x] I provided the complete config and logs, rather than just providing the truncated parts based on my own judgment.
- [x] I searched issues and did not find any similar issues.
- [x] The problem can be successfully reproduced in the latest Release
### Description
I've investigated high memory consumption of an iOS Network Extension (NE) for an app based on xray-core and found an interesting quirk specific to HTTP/2.
The overall stack pins significant amount of RAM, 0.5+ MiB per conn, to maintain idle(?) connections.
The largest heap user (inuse_space per pprof) was `http2.(*clientStream).writeRequestBody` function:
<img width="2206" height="1384" alt="Image" src="https://github.com/user-attachments/assets/0cb4b312-aacd-468b-88a9-1c0550690d22" />
or (another crash of the same kind)
<img width="2206" height="1384" alt="Image" src="https://github.com/user-attachments/assets/3a4ee81e-8204-4368-ab4a-acaffc7de040" />
These goroutines were blocked on `io.(*pipe).Read()`. OOM was happening as soon as there were ≈25 of them. I suspect, that's the Pipe() from splithttp.Dial():
https://github.com/XTLS/Xray-core/blob/fdb9b616fc0edf8fb4c3285870388947b92669fc/transport/internet/splithttp/dialer.go#L441
The `writeRequestBody()` comes from http2 client in x/net/http2/transport.go. It allocates some memory for scratch buffer to read from Pipe:
https://github.com/golang/net/blob/8ecbaa95fea823c19fa74c5c3b53e0bccd473828/http2/transport.go#L1506-L1515
`writeRequestBody()` does a reasonable thing. The API author probably assumed, that the pipe is a "local" (e.g. file), so it was unexpected for the x/net/http2 code to block for a long time, waiting for Pipe to be written to.
Content-Length is not known in advance in xHTTP, so the `frameScratchBufferLen()` defaults to `min(512KiB, SETTINGS_MAX_FRAME_SIZE)`:
https://github.com/golang/net/blob/8ecbaa95fea823c19fa74c5c3b53e0bccd473828/http2/transport.go#L1451
Default `SETTINGS_MAX_FRAME_SIZE` seems to be 1MiB Go x/net/http2. It's `http2.defaultMaxReadFrameSize` and server sends `{SettingMaxFrameSize, conf.MaxReadFrameSize}` as a part of SETTINGS frame to avoid default 16 KiB limit:
https://github.com/golang/net/blob/8ecbaa95fea823c19fa74c5c3b53e0bccd473828/http2/http2.go#L85
So, if I understand the case correctly, each idle connection pins at least 512 KiB of buffers on the client side just to maintain possibility to upload data via possibly-idle HTTP/2 stream.
#4749 expressed concerns about MaxReadFrameSize tuning in high-bandwidth scenarios. I've done two tests with iPhone 12 running xray-core with frameScratchBufferLen reduced from 512 KiB to 16 KiB. Both tests were using landline connections, with Wi-Fi 6, running on top of Gigabit Ethernet
- link soft-capped at 100 Mbit/s - iPhone was still able to saturate 100 Mbit/s just fine, OOKLA's Speedtest resulted in 110 Mbit/s (upload)
- link soft-capped at 800 Mbit/s - (WIP: I'll update with the exact number tomorrow, when I'll be back to the network)
The 512-to-16 KiB patch changed heap profile:
<img width="2206" height="1384" alt="Image" src="https://github.com/user-attachments/assets/1f556715-7869-4a33-882a-cf9b3fa9004f" />
The thing that changes the most is the number of goroutines waiting in writeRequestBody right before OOM: the number went from ≈25 to ≈200. Yet, this number looks strange on its own and it's another thing I'm going to investigate.
<img width="2206" height="1384" alt="Image" src="https://github.com/user-attachments/assets/63dd635e-e864-45d2-8b0e-be34c0112af0" />
The takeaways are the following:
- xray-core might provide an option to tune `MaxReadFrameSize` for the h2 listener and, maybe, other options of [http2.Server/http2.Transport](https://pkg.go.dev/golang.org/x/net/http2#section-readme), it'll not help when h2 is terminated at a CDN, but will help self-hosters with REALITY
- iOS apps maintainers may want to patch `frameScratchBufferLen()` in Go http2 library to reduce per-connection overhead
- Go http2 library might provide a setting to cap frameScratchBufferLen and/or provide a better platform-specific default
- mux seems to have potential to help with such setups
### Reproduction Method
My test-case was specific:
- feed refresh in [VK](https://en.wikipedia.org/wiki/VK_(service)) app triggered a small connection spike
- it was enough to hit 50 MiB limit after a two-three refreshes
- I polled `/debug/pprof/heap?gc=1` every two seconds and investigated the profile
### Client config
xray-core versoin: v26.6.1, go1.26.3
WIP: I'll update with the exact config later
### Server config
n/a
### Client log
n/a
### Server log
n/a
4 条评论