bug: `tls_*` options break proxying HTTP/2

### Summary While I was working on #67, I came across an issue with proxying HTTP/2 from the Layer 4 app to the HTTP app. It's either I'm missing some important bit in the configuration, or there is some strange incompatibility with the protocol that I don't understand. In short, the problem is that `tls_insecure_skip_verify` breaks proxying HTTP/2 for me. Not only is it an unexpected behaviour of Caddy, but also counter-intuitive because this is a security switch off commonly used to solve connection problems rather than cause them. ### Configuration In the config below, an HTTP/2 client successfully fetches `OK` from `https://localhost`, `https://localhost:1443` and `https://localhost:2443` (`tls` is enabled for the upstream), while it fails to fetch `OK` from `https://localhost:3443` (`tls_insecure_skip_verify` is used instead). ```caddyfile { debug layer4 { tcp/:1443 { route { proxy tcp/localhost:443 } } tcp/:2443 { route { tls proxy { upstream tcp/localhost:443 { tls } } } } tcp/:3443 { route { tls proxy { upstream tcp/localhost:443 { tls_insecure_skip_verify } } } } } servers :443 { protocols h1 h2 } } localhost { tls { issuer internal } respond "OK" 200 } ``` ### Testing I've tested this config with Curl 8.18.0 on Windows. My results are as follows: | host:port | http/1.1 [^1] | http/2 [^2] | |-----------|----------|--------| | localhost:443 | OK | OK | | localhost:1443 | OK | OK | | localhost:2443 | OK | OK | | localhost:3443 | OK | Error | Curl says `Remote peer returned unexpected data while we expected SETTINGS frame. Perhaps, peer does not support HTTP/2 properly.`, while Caddy complains that `rfc9112 forbids empty Host`. The weird thing is the config above never touches the HTTP host field. <details><summary>Curl output</summary> ``` Note: Using embedded CA bundle (225076 bytes) Note: Using embedded CA bundle, for proxies (225076 bytes) * Host localhost:3443 was resolved. * IPv6: ::1 * IPv4: 127.0.0.1 * Trying [::1]:3443... * ALPN: curl offers h2,http/1.1 * TLSv1.3 (OUT), TLS handshake, Client hello (1): * SSL Trust: peer verification disabled * TLSv1.3 (IN), TLS handshake, Server hello (2): * TLSv1.3 (IN), TLS handshake, Unknown (8): * TLSv1.3 (IN), TLS handshake, Certificate (11): * TLSv1.3 (IN), TLS handshake, CERT verify (15): * TLSv1.3 (IN), TLS handshake, Finished (20): * TLSv1.3 (OUT), TLS handshake, Finished (20): * SSL connection using TLSv1.3 / TLS_CHACHA20_POLY1305_SHA256 / [blank] / UNDEF * ALPN: server accepted h2 * Server certificate: * subject: * start date: Mar 5 06:51:50 2026 GMT * expire date: Mar 5 18:51:50 2026 GMT * issuer: CN=Caddy Local Authority - ECC Intermediate * Certificate level 0: Public key type ? (256/128 Bits/secBits), signed using ecdsa-with-SHA256 * Certificate level 1: Public key type ? (256/128 Bits/secBits), signed using ecdsa-with-SHA256 * SSL certificate verification failed, continuing anyway! * Established connection to localhost (::1 port 3443) from ::1 port 58475 * using HTTP/2 * [HTTP/2] [1] OPENED stream for https://localhost:3443/ * [HTTP/2] [1] [:method: GET] * [HTTP/2] [1] [:scheme: https] * [HTTP/2] [1] [:authority: localhost:3443] * [HTTP/2] [1] [:path: /] * [HTTP/2] [1] [user-agent: curl/8.18.0] * [HTTP/2] [1] [accept: */*] > GET / HTTP/2 > Host: localhost:3443 > User-Agent: curl/8.18.0 > Accept: */* > * Request completely sent off * Remote peer returned unexpected data while we expected SETTINGS frame. Perhaps, peer does not support HTTP/2 properly. * TLSv1.3 (IN), TLS alert, close notify (256): * nghttp2 shuts down connection with error 1: PROTOCOL_ERROR * closing connection #0 curl: (16) Remote peer returned unexpected data while we expected SETTINGS frame. Perhaps, peer does not support HTTP/2 properly. ``` </details> <details><summary>Caddy log</summary> ``` {"level":"info","ts":1772716811.5244532,"msg":"maxprocs: Leaving GOMAXPROCS=6: CPU quota undefined"} {"level":"info","ts":1772716811.5244532,"msg":"GOMEMLIMIT is updated","GOMEMLIMIT":9658073088,"previous":9223372036854775807} {"level":"info","ts":1772716811.5244532,"msg":"using config from file","file":"C:\\Users\\vnxme\\GolandProjects\\caddy-l4\\cmd\\master\\issue.Caddyfile"} {"level":"info","ts":1772716811.5244532,"msg":"adapted config to JSON","adapter":"caddyfile"} {"level":"warn","ts":1772716811.5244532,"msg":"Caddyfile input is not formatted; run 'caddy fmt --overwrite' to fix inconsistencies","adapter":"caddyfile","file":"C:\\Users\\vnxme\\GolandProjects\\caddy-l4\\cmd\\master\\issue.Caddyfile","line":2} {"level":"info","ts":1772716811.5359802,"logger":"admin","msg":"admin endpoint started","address":"localhost:2019","enforce_origin":false,"origins":["//localhost:2019","//[::1]:2019","//127.0.0.1:2019"]} {"level":"info","ts":1772716811.5365279,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0x536f55104200"} {"level":"info","ts":1772716811.5392795,"logger":"http.auto_https","msg":"server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS","server_name":"srv0","https_port":443} {"level":"info","ts":1772716811.5392795,"logger":"http.auto_https","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"} {"level":"debug","ts":1772716811.5392795,"logger":"http.auto_https","msg":"adjusted config","tls":{"automation":{"policies":[{"subjects":["localhost"]},{}]}},"http":{"servers":{"remaining_auto_https_redirects":{"listen":[":80"],"routes":[{},{}]},"srv0":{"listen":[":443"],"routes":[{"handle":[{"handler":"subroute","routes":[{"handle":[{"body":"OK","handler":"static_response","status_code":200}]}]}],"terminal":true}],"tls_connection_policies":[{}],"automatic_https":{},"protocols":["h1","h2"]}}}} {"level":"debug","ts":1772716811.542169,"logger":"layer4","msg":"started handling listener socket","network":"tcp","address":"[::]:2443"} {"level":"debug","ts":1772716811.542169,"logger":"layer4","msg":"started handling listener socket","network":"tcp","address":"[::]:1443"} {"level":"debug","ts":1772716811.542169,"logger":"layer4","msg":"started handling listener socket","network":"tcp","address":"[::]:3443"} {"level":"info","ts":1772716811.5500875,"logger":"tls","msg":"storage cleaning happened too recently; skipping for now","storage":"FileStorage:C:\\Users\\vnxme\\AppData\\Roaming\\Caddy","instance":"5687cf66-4f26-4518-ae31-a8c01af643e3","try_again":1772803211.5500875,"try_again_in":86400} {"level":"info","ts":1772716811.553095,"logger":"tls","msg":"finished cleaning storage units"} {"level":"info","ts":1772716811.5767474,"logger":"pki.ca.local","msg":"root certificate is already trusted by system","path":"storage:pki/authorities/local/root.crt"} {"level":"debug","ts":1772716811.5767474,"logger":"http","msg":"starting server loop","address":"[::]:443","tls":true,"http3":false} {"level":"info","ts":1772716811.5767474,"logger":"http.log","msg":"server running","name":"srv0","protocols":["h1","h2"]} {"level":"debug","ts":1772716811.5777695,"logger":"http","msg":"starting server loop","address":"[::]:80","tls":false,"http3":false} {"level":"warn","ts":1772716811.5777695,"logger":"http","msg":"HTTP/2 skipped because it requires TLS","network":"tcp","addr":":80"} {"level":"warn","ts":1772716811.5782785,"logger":"http","msg":"HTTP/3 skipped because it requires TLS","network":"tcp","addr":":80"} {"level":"info","ts":1772716811.5782785,"logger":"http.log","msg":"server running","name":"remaining_auto_https_redirects","protocols":["h1","h2","h3"]} {"level":"info","ts":1772716811.5782785,"logger":"http","msg":"enabling automatic TLS certificate management","domains":["localhost"]} {"level":"warn","ts":1772716811.5792885,"logger":"tls","msg":"stapling OCSP","identifiers":["localhost"]} {"level":"debug","ts":1772716811.5792885,"logger":"tls.cache","msg":"added certificate to cache","subjects":["localhost"],"expiration":1772736711,"managed":true,"issuer_key":"local","hash":"a9cabbb8dad32aa1e788d6ebf65020c8cb7b0ca0b5cb26d554b597b5f5e208b8","cache_size":1,"cache_capacity":10000} {"level":"debug","ts":1772716811.579993,"logger":"events","msg":"event","name":"cached_managed_cert","id":"c4ce11a7-634b-4486-826a-a36d168d1e7e","origin":"tls","data":{"sans":["localhost"]}} {"level":"debug","ts":1772716811.579993,"logger":"events","msg":"event","name":"started","id":"40fd4766-3f78-4dd8-acfc-8ba8c176b833","origin":"","data":null} {"level":"info","ts":1772716811.579993,"msg":"autosaved config (load with --resume flag)","file":"C:\\Users\\vnxme\\AppData\\Roaming\\Caddy\\autosave.json"} {"level":"info","ts":1772716811.579993,"msg":"serving initial configuration"} {"level":"debug","ts":1772716814.4857123,"logger":"layer4","msg":"started handling connection","network":"tcp","local":"[::1]:3443","remote":"[::1]:62746"} {"level":"debug","ts":1772716814.4890287,"logger":"events","msg":"event","name":"tls_get_certificate","id":"87b2e252-d318-4537-9e57-d2e2c524a334","origin":"tls","data":{"client_hello":{"CipherSuites":[4867,4866,4865,52393,52392,52394,49200,49196,49192,49188,49172,49162,159,107,57,196,136,157,61,53,192,132,49199,49195,49191,49187,49171,49161,158,103,51,190,69,156,60,47,186,65,49169,49159,5,49170,49160,22,10,255],"ServerName":"localhost","SupportedCurves":[29,23,24,25],"SupportedPoints":"AA==","SignatureSchemes":[2054,1537,1539,2053,1281,1283,2052,1025,1027,513,515],"SupportedProtos":["h2","http/1.1"],"SupportedVersions":[772,771],"RemoteAddr":{"IP":"::1","Port":62746,"Zone":""},"LocalAddr":{"IP":"::1","Port":3443,"Zone":""}}}} {"level":"debug","ts":1772716814.4890287,"logger":"tls.handshake","msg":"choosing certificate","identifier":"localhost","num_choices":1} {"level":"debug","ts":1772716814.4890287,"logger":"tls.handshake","msg":"default certificate selection results","identifier":"localhost","subjects":["localhost"],"managed":true,"issuer_key":"local","hash":"a9cabbb8dad32aa1e788d6ebf65020c8cb7b0ca0b5cb26d554b597b5f5e208b8"} {"level":"debug","ts":1772716814.4890287,"logger":"tls.handshake","msg":"matched certificate in cache","remote_ip":"::1","remote_port":"62746","subjects":["localhost"],"managed":true,"expiration":1772736711,"hash":"a9cabbb8dad32aa1e788d6ebf65020c8cb7b0ca0b5cb26d554b597b5f5e208b8"} {"level":"debug","ts":1772716814.5033348,"logger":"layer4.handlers.tls","msg":"terminated TLS","remote":"[::1]:62746","server_name":"localhost"} {"level":"debug","ts":1772716814.5048025,"logger":"events","msg":"event","name":"tls_get_certificate","id":"02396125-9339-4b9b-9ee9-72f17ccdf332","origin":"tls","data":{"client_hello":{"CipherSuites":[52393,52392,49195,49199,49196,49200,49161,49171,49162,49172,4867,4865,4866],"ServerName":"localhost","SupportedCurves":[4588,29,23,24,25],"SupportedPoints":"AA==","SignatureSchemes":[2052,1027,2055,2053,2054,1025,1281,1537,1283,1539],"SupportedProtos":null,"SupportedVersions":[772,771],"RemoteAddr":{"IP":"::1","Port":62747,"Zone":""},"LocalAddr":{"IP":"::1","Port":443,"Zone":""}}}} {"level":"debug","ts":1772716814.5048025,"logger":"tls.handshake","msg":"choosing certificate","identifier":"localhost","num_choices":1} {"level":"debug","ts":1772716814.5048025,"logger":"tls.handshake","msg":"default certificate selection results","identifier":"localhost","subjects":["localhost"],"managed":true,"issuer_key":"local","hash":"a9cabbb8dad32aa1e788d6ebf65020c8cb7b0ca0b5cb26d554b597b5f5e208b8"} {"level":"debug","ts":1772716814.5048025,"logger":"tls.handshake","msg":"matched certificate in cache","remote_ip":"::1","remote_port":"62747","subjects":["localhost"],"managed":true,"expiration":1772736711,"hash":"a9cabbb8dad32aa1e788d6ebf65020c8cb7b0ca0b5cb26d554b597b5f5e208b8"} {"level":"debug","ts":1772716814.505952,"logger":"layer4.handlers.proxy","msg":"dial upstream","remote":"[::1]:62746","upstream":"localhost:443"} {"level":"debug","ts":1772716814.541517,"logger":"http.log.error","msg":"rfc9112 forbids empty Host","request":{"remote_ip":"::1","remote_port":"62747","client_ip":"::1","proto":"HTTP/2.0","method":"PRI","host":"","uri":"*","headers":{},"tls":{"resumed":false,"version":772,"cipher_suite":4867,"proto":"","server_name":"localhost","ech":false}},"duration":0,"status":400,"err_id":"","err_trace":""} {"level":"debug","ts":1772716814.5555696,"logger":"layer4","msg":"stopped handling connection; connection stats","network":"tcp","local":"[::1]:3443","remote":"[::1]:62746","read":554,"written":1447,"duration":0.0692977} ``` </details> [^1]: `curl.exe --insecure --verbose --http1.1 https://{host:port}/` [^2]: `curl.exe --insecure --verbose --http2 https://{host:port}/` ### Background I'm not the only one who has faced this problem so far, but I haven't found an acceptable solution in these discussions: - [2021, the closest match, but barely solves the issue](https://caddy.community/t/combining-the-layer4-and-http-apps-ssl-pass-through-http-file-server-reverse-proxy/12698/5) - [2023, but adding `h2c` won't help](https://github.com/caddyserver/caddy/issues/5992) - [2024, probably less relevant](https://caddy.community/t/using-caddy-layer4-for-tls-passthrough/22256) ### Workarounds I know, it's possible to specify `alpn http/1.1` in the `tls` handler's `connection_policy` block, but it's out of scope, because both HTTP/1.1 and HTTP/2 proxying works fine with `tls` instead of `tls_insecure_skip_verify`.