26.7.3 update deadlocks on WireGuard model cache after interface assign-opts timeout
support
**Important notices**
Before you add a new report, we ask you kindly to acknowledge the following:
* [x] I have read the contributing guidelines at https://github.com/opnsense/core/blob/master/CONTRIBUTING.md
* [x] I am convinced that my issue is new after having checked both open and closed issues at https://github.com/opnsense/core/issues?q=is%3Aissue
* [x] AI tools were used to create at least part of the text submitted herewith.
If AI was used, please disclose:
* Model used: OpenAI GPT-5.6 Sol (Codex in ChatGPT Work)
* Extent of AI involvement: AI helped organize and draft this report from terminal output and screenshots that I supplied. I personally performed all diagnostics and recovery commands and verified the results.
**Describe the bug**
While upgrading from OPNsense 26.7.2_2 to 26.7.3 through **System → Firmware → Updates**, the update stalled indefinitely during the `rc.configure_firmware` post-installation step.
OPNsense 26.7.2_2 had previously been operating normally. The problem appeared while installing 26.7.3.
All 94 packages had been extracted successfully, including:
```text
[94/94] Upgrading opnsense from 26.7.2_2 to 26.7.3...
[94/94] Extracting opnsense-26.7.3: .......... done
```
The last update output was:
```text
Starting configd.
>>> Invoking update script 'refresh.sh'
Timeout (20) executing : interface list assign-opts
```
The router, Internet connection, and web interface remained operational, but the update made no further progress for approximately one hour.
The update was blocked in `run_migrations.php`, which was waiting in `lockf` on the WireGuard model-cache file. The timed-out `list_assign_options.php` process was also still present and waiting in `lockf`.
Terminating only the stale `list_assign_options.php` process allowed the migration and update to resume immediately.
**To Reproduce**
These are the steps observed during the actual upgrade. I do not yet have a reliable minimal reproduction outside the firmware update.
1. Open `https://<firewall-address>/ui/core/firmware#updates`.
2. Start the update from OPNsense 26.7.2_2 to 26.7.3.
3. Allow all packages to install.
4. During the OPNsense core package post-installation step, observe:
```text
Timeout (20) executing : interface list assign-opts
```
5. Observe that the update remains indefinitely in:
```text
/usr/local/etc/rc.syshook.d/update/10-refresh.sh
```
The blocked update process chain was:
```text
update.sh
└─ opnsense-update -pt opnsense
└─ pkg-static upgrade -y
└─ 10-refresh.sh
└─ rc.configure_firmware
└─ pluginctl -m
└─ run_migrations.php -v
```
**Expected behavior**
A timed-out `interface list assign-opts` action should terminate cleanly and release any associated model-cache locks.
The firmware migration and package update should not remain blocked indefinitely behind a timed-out configd child process.
**Describe alternatives you considered**
I initially waited approximately one hour, but the update did not progress.
I did not reboot the firewall and did not terminate `pkg`, `opnsense-update`, `run_migrations.php`, or any other update process because doing so could have damaged the package transaction.
After identifying the stale interface-enumeration process, I terminated only that process:
```sh
kill -TERM 10915
```
The migration acquired the released resource, and the update resumed immediately. It completed normally and performed its controlled reboot.
**Screenshots**
I have attached a screenshot of the firmware update stalled at:
```text
Timeout (20) executing : interface list assign-opts
```
**Relevant log files**
The update processes had been stalled for approximately 58 minutes:
```text
46310 I /usr/local/bin/flock -n -o /tmp/pkg_upgrade.progress /usr/local/opnsense/scripts/firmware/update.sh
60781 IC /usr/bin/tee -a /tmp/pkg_upgrade.progress
61160 I /bin/sh /usr/local/sbin/opnsense-update -pt opnsense
71148 I pkg-static upgrade -y
71289 S pkg-static upgrade -y
97972 I /bin/sh /usr/local/etc/rc.syshook.d/update/10-refresh.sh
```
The direct child of `10-refresh.sh` was:
```text
97990 97972 I piperd /usr/local/bin/php /usr/local/etc/rc.configure_firmware
```
The child process chain continued as:
```text
578 97990 I piperd /usr/local/bin/php /usr/local/sbin/pluginctl -m
820 578 I lockf /usr/local/bin/php /usr/local/opnsense/mvc/script/run_migrations.php -v
```
The timed-out interface action was still running after more than one hour:
```text
PID PPID STAT WCHAN COMMAND
10915 96845 I lockf /usr/local/bin/php /usr/local/opnsense/scripts/interfaces/list_assign_options.php
```
`procstat -f 820` showed the migration accessing:
```text
/var/lib/php/tmp/mdl_cache_OPNsense_Wireguard_Server.json
```
That same file was open by PID 820, PID 10915, and multiple `php-cgi` workers.
The PHP process running `rc.configure_firmware` was blocked reading from its child:
```text
PID STAT WCHAN
97990 I piperd
```
Its kernel stack was:
```text
mi_switch
sleepq_catch_signals
sleepq_wait_sig
_sleep
pipe_read
dofileread
sys_read
amd64_syscall
fast_syscall_common
```
**Additional context**
After terminating only PID 10915, the update continued, completed, and rebooted successfully.
After reboot, the system reported:
```text
OPNsense 26.7.3-amd64
FreeBSD 15.1-RELEASE-p3
OpenSSL 3.5.8
```
A firmware health audit found:
```text
No missing or altered kernel files
No missing or altered base files
No missing package dependencies
No missing or altered package files
No locked packages
```
The audit initially reported only that `opnsense-26.7.3` was older than the newly published `26.7.3_1` hotfix. That hotfix was released while this upgrade was underway and appears unrelated to the deadlock.
The router and web GUI remained operational throughout the stalled update.
**Environment**
```text
Upgrade: OPNsense 26.7.2_2 to 26.7.3
Architecture: amd64
Filesystem: ZFS
Hardware: CWWK Intel N150
Memory: 32 GB
FreeBSD before update: 15.1-RELEASE-p2
FreeBSD after update: 15.1-RELEASE-p3
WireGuard: configured
Network: multiple VLAN interfaces
Zenarmor/Sensei: installed
```
<img width="1378" height="901" alt="Image" src="https://github.com/user-attachments/assets/42942de4-e3d7-420d-ac62-9d86a335af97" />
2 条评论