ITADN

26.7.3 update deadlocks on WireGuard model cache after interface assign-opts timeout

#10807Opendjstroh 创建于 2 天前
support
D
djstrohcommented
**Important notices** Before you add a new report, we ask you kindly to acknowledge the following: * [x] I have read the contributing guidelines at https://github.com/opnsense/core/blob/master/CONTRIBUTING.md * [x] I am convinced that my issue is new after having checked both open and closed issues at https://github.com/opnsense/core/issues?q=is%3Aissue * [x] AI tools were used to create at least part of the text submitted herewith. If AI was used, please disclose: * Model used: OpenAI GPT-5.6 Sol (Codex in ChatGPT Work) * Extent of AI involvement: AI helped organize and draft this report from terminal output and screenshots that I supplied. I personally performed all diagnostics and recovery commands and verified the results. **Describe the bug** While upgrading from OPNsense 26.7.2_2 to 26.7.3 through **System → Firmware → Updates**, the update stalled indefinitely during the `rc.configure_firmware` post-installation step. OPNsense 26.7.2_2 had previously been operating normally. The problem appeared while installing 26.7.3. All 94 packages had been extracted successfully, including: ```text [94/94] Upgrading opnsense from 26.7.2_2 to 26.7.3... [94/94] Extracting opnsense-26.7.3: .......... done ``` The last update output was: ```text Starting configd. >>> Invoking update script 'refresh.sh' Timeout (20) executing : interface list assign-opts ``` The router, Internet connection, and web interface remained operational, but the update made no further progress for approximately one hour. The update was blocked in `run_migrations.php`, which was waiting in `lockf` on the WireGuard model-cache file. The timed-out `list_assign_options.php` process was also still present and waiting in `lockf`. Terminating only the stale `list_assign_options.php` process allowed the migration and update to resume immediately. **To Reproduce** These are the steps observed during the actual upgrade. I do not yet have a reliable minimal reproduction outside the firmware update. 1. Open `https://<firewall-address>/ui/core/firmware#updates`. 2. Start the update from OPNsense 26.7.2_2 to 26.7.3. 3. Allow all packages to install. 4. During the OPNsense core package post-installation step, observe: ```text Timeout (20) executing : interface list assign-opts ``` 5. Observe that the update remains indefinitely in: ```text /usr/local/etc/rc.syshook.d/update/10-refresh.sh ``` The blocked update process chain was: ```text update.sh └─ opnsense-update -pt opnsense └─ pkg-static upgrade -y └─ 10-refresh.sh └─ rc.configure_firmware └─ pluginctl -m └─ run_migrations.php -v ``` **Expected behavior** A timed-out `interface list assign-opts` action should terminate cleanly and release any associated model-cache locks. The firmware migration and package update should not remain blocked indefinitely behind a timed-out configd child process. **Describe alternatives you considered** I initially waited approximately one hour, but the update did not progress. I did not reboot the firewall and did not terminate `pkg`, `opnsense-update`, `run_migrations.php`, or any other update process because doing so could have damaged the package transaction. After identifying the stale interface-enumeration process, I terminated only that process: ```sh kill -TERM 10915 ``` The migration acquired the released resource, and the update resumed immediately. It completed normally and performed its controlled reboot. **Screenshots** I have attached a screenshot of the firmware update stalled at: ```text Timeout (20) executing : interface list assign-opts ``` **Relevant log files** The update processes had been stalled for approximately 58 minutes: ```text 46310 I /usr/local/bin/flock -n -o /tmp/pkg_upgrade.progress /usr/local/opnsense/scripts/firmware/update.sh 60781 IC /usr/bin/tee -a /tmp/pkg_upgrade.progress 61160 I /bin/sh /usr/local/sbin/opnsense-update -pt opnsense 71148 I pkg-static upgrade -y 71289 S pkg-static upgrade -y 97972 I /bin/sh /usr/local/etc/rc.syshook.d/update/10-refresh.sh ``` The direct child of `10-refresh.sh` was: ```text 97990 97972 I piperd /usr/local/bin/php /usr/local/etc/rc.configure_firmware ``` The child process chain continued as: ```text 578 97990 I piperd /usr/local/bin/php /usr/local/sbin/pluginctl -m 820 578 I lockf /usr/local/bin/php /usr/local/opnsense/mvc/script/run_migrations.php -v ``` The timed-out interface action was still running after more than one hour: ```text PID PPID STAT WCHAN COMMAND 10915 96845 I lockf /usr/local/bin/php /usr/local/opnsense/scripts/interfaces/list_assign_options.php ``` `procstat -f 820` showed the migration accessing: ```text /var/lib/php/tmp/mdl_cache_OPNsense_Wireguard_Server.json ``` That same file was open by PID 820, PID 10915, and multiple `php-cgi` workers. The PHP process running `rc.configure_firmware` was blocked reading from its child: ```text PID STAT WCHAN 97990 I piperd ``` Its kernel stack was: ```text mi_switch sleepq_catch_signals sleepq_wait_sig _sleep pipe_read dofileread sys_read amd64_syscall fast_syscall_common ``` **Additional context** After terminating only PID 10915, the update continued, completed, and rebooted successfully. After reboot, the system reported: ```text OPNsense 26.7.3-amd64 FreeBSD 15.1-RELEASE-p3 OpenSSL 3.5.8 ``` A firmware health audit found: ```text No missing or altered kernel files No missing or altered base files No missing package dependencies No missing or altered package files No locked packages ``` The audit initially reported only that `opnsense-26.7.3` was older than the newly published `26.7.3_1` hotfix. That hotfix was released while this upgrade was underway and appears unrelated to the deadlock. The router and web GUI remained operational throughout the stalled update. **Environment** ```text Upgrade: OPNsense 26.7.2_2 to 26.7.3 Architecture: amd64 Filesystem: ZFS Hardware: CWWK Intel N150 Memory: 32 GB FreeBSD before update: 15.1-RELEASE-p2 FreeBSD after update: 15.1-RELEASE-p3 WireGuard: configured Network: multiple VLAN interfaces Zenarmor/Sensei: installed ``` <img width="1378" height="901" alt="Image" src="https://github.com/user-attachments/assets/42942de4-e3d7-420d-ac62-9d86a335af97" />
2 条评论