ITADN

regexp matcher can't match arbitrary bytes (invalid UTF-8)

#419Openbcat 创建于 2026-05-13
B
bcatcommented
I'm trying to use `layer4` to route the [Syncthing Relay Protocol](https://docs.syncthing.net/specs/relay-v1.html), which is actually _two_ protocols multiplexed on a single port: * "Protocol mode" for control messages: TLS with ALPN `bep-relay`. I confirmed this can be handled with a `tls alpn bep-relay` matcher. * "Session mode" for relay sessions: a header starting with magic bytes `9E 79 BC 40`, followed by an arbitrary sequence of bytes. According to the docs, I _should_ be able to match the "session mode" initiation packet with `@bep-relay-session regexp \x9E\x79\xBC\x4`, but that doesn't work. I factored out a minimal `Caddyfile` (ignoring the TLS "protocol mode", because that works fine): ``` { layer4 { localhost:10000 { @cats regexp \x43\x41\x54\x53 route @cats { echo } @bep-relay-session regexp \x9E\x79\xBC\x40 route @bep-relay-session { echo } route { close } } } } ``` Test it as follows: ``` $ printf '\x43\x41\x54\x53: echo me\n' | nc -N localhost 10000 | hexdump -C 00000000 43 41 54 53 3a 20 65 63 68 6f 20 6d 65 0a |CATS: echo me.| 0000000e $ printf '\x9E\x79\xBC\x40: echo me\n' | nc -N localhost 10000 | hexdump -C ``` The `regexp \x43\x41\x54\x53` matcher succeeds, routing to the `echo` handler as expected. However, the `regexp \x9E\x79\xBC\x40` matcher never matches, so the connection gets routed to the catchall `close` handler instead. I think the issue is that [Go's `regexp` package assumes bytes to check for a match are valid UTF-8](https://stackoverflow.com/a/70771294), which means it can't properly match against non-UTF-8 sequences (which are common in application protocol headers). A quick scan of the `regexp` source code [seems to confirm this](https://cs.opensource.google/go/go/+/master:src/regexp/regexp.go;l=423;drc=0b87c1d350c7842bb3c75c89f482bd107ef7ea59). I can see a couple paths forward here: 1. Find an alternate regular expressions package that properly supports "binary regex". However, I am not familiar with Go and am not sure if a good candidate exists, especially given that matching is performance sensitive. 2. Write a simpler `magic` matcher that only looks for binary substrings in a specific position of the initial packet sent by the client. I envision syntax like `magic bytes [start]`, so for my use case, just `magic 9E79BC40`. It'd be a shame to lose the flexibility of regex, but I suspect many protocols don't actually need it. (It might also be slightly more efficient, though I don't know if that matters in practice.) Personally, (2) seems like the simplest option to resolve the issue I'm facing; however, it doesn't help someone who needs _both_ the flexibility of regex _and_ real byte string matching. (The fact nobody seems to have reported this previously in the couple years the `regexp` matcher has existed suggests to me that needing both those things at once is uncommon, though. I wouldn't be surprised if binary protocols tend to have a simple, strict magic number like the Syncthing protocol does....)
2 条评论