Nginx Worker Process SIGSEGV (Signal 11) with Passenger Module

# Nginx Worker Process SIGSEGV (Signal 11) with Passenger Module **Date:** 2026-06-25 **Platform:** RHEL 9.8, CIS Benchmark Level 1, AWS EC2 --- ## Symptoms - Nginx worker processes crash immediately on first HTTP request with `signal 11 (SIGSEGV)` - Core dump shows crash in `ngx_escape_uri()` called from Passenger's `prepare_request_buffer_construction()` - Passenger watchdog/core start fine, but workers die on any request ## Root Cause **ABI mismatch between RHEL's nginx binary and Phusion's pre-built `mod_passenger` RPM.** RHEL 9's nginx (`nginx-core-1.26.3-9.module+el9.8.0`) is compiled with `NGX_HTTP_CACHE=1`, which adds an 8-byte pointer field (`ngx_http_cache_t *cache`) at offset 64 in `struct ngx_http_request_s`. Phusion's `nginx-mod-http-passenger-6.1.5-1.el9` RPM was compiled against nginx headers **without** `NGX_HTTP_CACHE` defined. This causes every struct field after offset 64 to be shifted by 8 bytes in the module's compiled code vs. the actual runtime binary layout. **Effect:** When Passenger reads `r->uri.data`, it actually reads `r->request_line.len` (garbage pointer). Passing this to `ngx_escape_uri()` causes the segfault. | Field | Binary (actual) | Passenger module (expected) | Delta | |-------|----------------|-----------------------------|-------| | `cache` | offset 64 | *not present* | +8 | | `method` | offset 856 | offset 848 | +8 | | `uri` | offset 888 | offset 880 | +8 | ## Fix (Manual CLI Procedure) Recompile the Passenger nginx module from gem source against RHEL's actual nginx source+config: ```bash # 1. Install build dependencies dnf install -y gcc gcc-c++ make pcre2-devel openssl-devel zlib-devel \ libxml2-devel libxslt-devel gd-devel perl-ExtUtils-Embed \ gperftools-devel ruby-devel curl-devel # 2. Download and prep RHEL nginx SRPM yumdownloader --source nginx-core rpm -ivh nginx-core-*.src.rpm cd ~/rpmbuild rpmbuild -bp SPECS/nginx.spec # 3. Configure with RHEL's exact flags + Passenger dynamic module cd ~/rpmbuild/BUILD/nginx-1.26.3 eval ./configure $(nginx -V 2>&1 | grep -oP '(?<=configure arguments: ).*') \ --add-dynamic-module=/usr/local/rvm/gems/ruby-3.4.2/gems/passenger-6.1.5/src/nginx_module # 4. Build just the modules make modules # 5. Install the correctly-compiled module cp objs/ngx_http_passenger_module.so /usr/lib64/nginx/modules/ngx_http_passenger_module.so cp objs/ngx_http_passenger_module.so /usr/share/nginx/modules/ngx_http_passenger_module.so # 6. Restart nginx nginx -t && systemctl restart nginx ``` ## Verification ```bash # Confirm no SIGSEGV in logs tail -20 /var/log/nginx/error.log | grep -c 'signal 11' # should be 0 # Confirm app responds curl -sk https://localhost/ | head -1 # should return HTML ``` ## Prevention - Do NOT use Phusion's pre-built `nginx-mod-http-passenger` RPM on RHEL 9 — it has an ABI mismatch - Always compile the Passenger nginx module from source against the installed RHEL nginx source - Pin nginx and passenger versions in Ansible; recompile module on any nginx update - Consider adding a molecule test that verifies nginx workers remain stable after startup ## Affected Packages | Package | Version | Source | |---------|---------|--------| | nginx-core | 1.26.3-9.module+el9.8.0+24374+c0e15309.1 | RHEL AppStream | | passenger | 6.1.5-1.el9 | Phusion repo | | nginx-mod-http-passenger | 6.1.5-1.el9 | Phusion repo (BROKEN) | | passenger gem | 6.1.5 | RVM ruby-3.4.2 |