Breaking: Underscore headers silently dropped in v2.11.4 — no opt-out, no warning, no migration path

### Issue Details ## Breaking: Underscore headers silently dropped in v2.11.4 — no opt-out, no warning, no migration path Ref: [`3eb8e48`](https://github.com/caddyserver/caddy/commit/3eb8e48) (v2.11.4, @dunglas) ## Summary Caddy v2.11.4 silently drops all HTTP header fields containing underscores. There is no configuration option to opt out, no deprecation warning in prior versions, no log entry when a header is dropped, and no migration path for existing deployments. This is a breaking change that was shipped as a security patch with zero consideration for the thousands of production systems that rely on underscore-containing headers. ## The Impact Our production system serves **hundreds of thousands of paying users**. After a routine `apt upgrade` on Ubuntu 24.04, every single user was silently locked out of their paid content. No error. No log. No indication whatsoever that Caddy was swallowing authentication headers before they reached our application. It took hours to trace the issue to Caddy because **there is no log output when a header is dropped**. The request arrives, the header is gone, and the application sees an unauthenticated user. Silently. ## The Technical Problem Our clients send `user_id` as a custom HTTP header. This is **valid per RFC 7230**, which explicitly permits underscores in header field names. Caddy has decided unilaterally that a valid HTTP header should be silently discarded. The stated rationale — CGI variable collision — is a 1990s concern. No modern backend framework maps headers to environment variables via CGI. Bun, Deno, Node, Go's own `net/http` — none of them have this ambiguity. This is a phantom threat used to justify a real-world breaking change. ## What Makes This Inexcusable 1. **You built the opt-out and removed it.** The commit history of `3eb8e48` shows `allow_underscore_in_headers` was implemented, tested, and then deliberately stripped before merge. You knew this would break people. You chose not to give them a way out. 2. **Silent failure.** When Caddy drops a header, it produces zero output — no warning log, no debug message, nothing. This is the worst possible failure mode for a reverse proxy. A proxy's entire job is to faithfully pass data between client and server. Silently eating data is a fundamental violation of that contract. 3. **No deprecation path.** Prior versions of Caddy passed underscore headers without issue. There was no deprecation warning, no "this will stop working in the next major version" notice. One day it works, the next day it doesn't. For a tool that markets itself as production-grade, this is unacceptable. 4. **Not your decision to make.** A reverse proxy should not have opinions about what my headers are named. If I choose to name a header `user_id`, that is between my client and my server. Caddy's job is to forward it faithfully. If you want to offer sanitisation, make it opt-in — not the other way around. ## What Needs to Happen - **Restore `allow_underscore_in_headers`** as a global server option. You already wrote it. Ship it. - **Add a warning log** when an underscore header is dropped, so the next person doesn't waste hours debugging a ghost. - **Document this as a breaking change** prominently, not buried in a security patch bullet point. ## Environment - Caddy version: v2.11.4 (installed via apt on Ubuntu 24.04) - Previous version: pre-2.11.4 (worked without issue) - Backend: Bun/Elysia - Affected header: `user_id` (valid per RFC 7230) ### Assistance Disclosure AI used ### If AI was used, describe the extent to which it was used. Research-only (CGI era, the rationale (!) behind removing underscore, etc.)