regexp matcher can't match arbitrary bytes (invalid UTF-8)
I'm trying to use `layer4` to route the [Syncthing Relay Protocol](https://docs.syncthing.net/specs/relay-v1.html), which is actually _two_ protocols multiplexed on a single port:
* "Protocol mode" for control messages: TLS with ALPN `bep-relay`. I confirmed this can be handled with a `tls alpn bep-relay` matcher.
* "Session mode" for relay sessions: a header starting with magic bytes `9E 79 BC 40`, followed by an arbitrary sequence of bytes.
According to the docs, I _should_ be able to match the "session mode" initiation packet with `@bep-relay-session regexp \x9E\x79\xBC\x4`, but that doesn't work.
I factored out a minimal `Caddyfile` (ignoring the TLS "protocol mode", because that works fine):
```
{
layer4 {
localhost:10000 {
@cats regexp \x43\x41\x54\x53
route @cats {
echo
}
@bep-relay-session regexp \x9E\x79\xBC\x40
route @bep-relay-session {
echo
}
route {
close
}
}
}
}
```
Test it as follows:
```
$ printf '\x43\x41\x54\x53: echo me\n' | nc -N localhost 10000 | hexdump -C
00000000 43 41 54 53 3a 20 65 63 68 6f 20 6d 65 0a |CATS: echo me.|
0000000e
$ printf '\x9E\x79\xBC\x40: echo me\n' | nc -N localhost 10000 | hexdump -C
```
The `regexp \x43\x41\x54\x53` matcher succeeds, routing to the `echo` handler as expected. However, the `regexp \x9E\x79\xBC\x40` matcher never matches, so the connection gets routed to the catchall `close` handler instead.
I think the issue is that [Go's `regexp` package assumes bytes to check for a match are valid UTF-8](https://stackoverflow.com/a/70771294), which means it can't properly match against non-UTF-8 sequences (which are common in application protocol headers). A quick scan of the `regexp` source code [seems to confirm this](https://cs.opensource.google/go/go/+/master:src/regexp/regexp.go;l=423;drc=0b87c1d350c7842bb3c75c89f482bd107ef7ea59).
I can see a couple paths forward here:
1. Find an alternate regular expressions package that properly supports "binary regex". However, I am not familiar with Go and am not sure if a good candidate exists, especially given that matching is performance sensitive.
2. Write a simpler `magic` matcher that only looks for binary substrings in a specific position of the initial packet sent by the client. I envision syntax like `magic bytes [start]`, so for my use case, just `magic 9E79BC40`. It'd be a shame to lose the flexibility of regex, but I suspect many protocols don't actually need it. (It might also be slightly more efficient, though I don't know if that matters in practice.)
Personally, (2) seems like the simplest option to resolve the issue I'm facing; however, it doesn't help someone who needs _both_ the flexibility of regex _and_ real byte string matching. (The fact nobody seems to have reported this previously in the couple years the `regexp` matcher has existed suggests to me that needing both those things at once is uncommon, though. I wouldn't be surprised if binary protocols tend to have a simple, strict magic number like the Syncthing protocol does....)
2 条评论