Sled has a reservoir size of 0 after reboot
Sled Agent
We have seen four sleds in rack3 that have come up after a recent restart with a reservoir size of 0 set in the database.
```
root@[fd00:1122:3344:116::3]:32221/omicron> select id, serial_number, reservoir_size, time_created, time_modified from sled where reservoir_size = 0;
id | serial_number | reservoir_size | time_created | time_modified
---------------------------------------+---------------+----------------+-------------------------------+--------------------------------
2c33e8ef-ae07-4191-804d-52a1840ce53c | BRM42220042 | 0 | 2023-09-02 13:07:22.168451+00 | 2026-08-12 15:39:45.228129+00
857bd97b-8147-4d26-813b-d80ab8f36cd5 | BRM42220083 | 0 | 2023-09-02 13:07:20.885306+00 | 2026-08-12 17:45:59.820146+00
eaad4483-ea93-48d1-99d3-bf43267b9eb7 | BRM42220015 | 0 | 2023-09-02 13:07:20.852863+00 | 2026-08-12 21:23:28.448887+00
fac5180b-232a-489e-b7c5-313a0742d1cc | BRM42220054 | 0 | 2023-09-02 13:07:20.82203+00 | 2026-08-12 16:04:53.65634+00
```
All four of these sleds have sled-agent logs that look like:
```
{"msg":"Calculated eligible VMM reservoir size","v":0,"name":"SledAgent","level":30,"time":"1986-12-28T00:01:11.231465519Z","hostname":"BRM42220015","pid":665,"component":"MemoryReservations","file":"sled-hardware/src/lib.rs:154","control_plane_earmark_bytes":"47244640256","max_page_t_bytes":"31834245120","physical_ram_bytes":"1086608900096","vmm_eligible":"1007530014720"}
{"msg":"91% of 1007530014720 VMM eligible ram = 916852313395 bytes)","v":0,"name":"SledAgent","level":30,"time":"1986-12-28T00:01:11.23220472Z","hostname":"BRM42220015","pid":665,"component":"VmmReservoirManager","sled_id":"eaad4483-ea93-48d1-99d3-bf43267b9eb7","component":"SledAgent","file":"sled-agent/src/vmm_reservoir.rs:258"}
{"msg":"Setting reservoir size to 874378 MiB bytes","v":0,"name":"SledAgent","level":30,"time":"1986-12-28T00:01:11.232244165Z","hostname":"BRM42220015","pid":665,"component":"VmmReservoirManager","sled_id":"eaad4483-ea93-48d1-99d3-bf43267b9eb7","component":"SledAgent","file":"sled-agent/src/vmm_reservoir.rs:266"}
{"msg":"Retrieved SledAgentInfo from Nexus: SledAgentInfo { baseboard: Baseboard { part: \"913-0000019\", revision: 6, serial: \"BRM42220015\" }, cpu_family: AmdMilan, decommissioned: false, generation: Generation(27), repo_depot_port: 12348, reservoir_size: ByteCount(916851785728), role: Gimlet, sa_address: \"[fd00:1122:3344:109::1]:12345\", usable_hardware_threads: 128, usable_physical_ram: ByteCount(1086608900096) }","v":0,"name":"SledAgent","level":30,"time":"1986-12-28T00:01:11.256005206Z","hostname":"BRM42220015","pid":665,"component":"NexusNotifierTask","sled_id":"eaad4483-ea93-48d1-99d3-bf43267b9eb7","component":"SledAgent","file":"sled-agent/src/nexus.rs:500"}
{"msg":"Failed to setup VMM reservoir","v":0,"name":"SledAgent","level":50,"time":"2026-08-12T21:27:06.348569867Z","hostname":"BRM42220015","pid":665,"component":"VmmReservoirManager","sled_id":"eaad4483-ea93-48d1-99d3-bf43267b9eb7","component":"SledAgent","file":"sled-agent/src/vmm_reservoir.rs:191","error":"Failed to create reservoir: Failed to resize reservoir: Not enough space (os error 12)"}
```
We do know that earlier in the day after a reboot we did successfully allocate the reservoir:
```
{"msg":"Calculated eligible VMM reservoir size","v":0,"name":"SledAgent","level":30,"time":"1986-12-28T00:00:59.461675821Z","hostname":"BRM42220015","pid":666,"component":"MemoryReservations","file":"sled-hardware/src/lib.rs:154","control_plane_earmark_bytes":"47244640256","max_page_t_bytes":"31834245120","physical_ram_bytes":"1086608900096","vmm_eligible":"1007530014720"}
{"msg":"91% of 1007530014720 VMM eligible ram = 916852313395 bytes)","v":0,"name":"SledAgent","level":30,"time":"1986-12-28T00:00:59.462365518Z","hostname":"BRM42220015","pid":666,"component":"VmmReservoirManager","sled_id":"eaad4483-ea93-48d1-99d3-bf43267b9eb7","component":"SledAgent","file":"sled-agent/src/vmm_reservoir.rs:258"}
{"msg":"Setting reservoir size to 874378 MiB bytes","v":0,"name":"SledAgent","level":30,"time":"1986-12-28T00:00:59.462378383Z","hostname":"BRM42220015","pid":666,"component":"VmmReservoirManager","sled_id":"eaad4483-ea93-48d1-99d3-bf43267b9eb7","component":"SledAgent","file":"sled-agent/src/vmm_reservoir.rs:266"}
{"msg":"Retrieved SledAgentInfo from Nexus: SledAgentInfo { baseboard: Baseboard { part: \"913-0000019\", revision: 6, serial: \"BRM42220015\" }, cpu_family: AmdMilan, decommissioned: false, generation: Generation(25), repo_depot_port: 12348, reservoir_size: ByteCount(916851785728), role: Gimlet, sa_address: \"[fd00:1122:3344:109::1]:12345\", usable_hardware_threads: 128, usable_physical_ram: ByteCount(1086608900096) }","v":0,"name":"SledAgent","level":30,"time":"1986-12-28T00:00:59.484793759Z","hostname":"BRM42220015","pid":666,"component":"NexusNotifierTask","sled_id":"eaad4483-ea93-48d1-99d3-bf43267b9eb7","component":"SledAgent","file":"sled-agent/src/nexus.rs:500"}
{"msg":"Finished setting reservoir size to 874378 MiB bytes","v":0,"name":"SledAgent","level":30,"time":"2026-08-12T14:37:53.242773137Z","hostname":"BRM42220015","pid":666,"component":"VmmReservoirManager","sled_id":"eaad4483-ea93-48d1-99d3-bf43267b9eb7","component":"SledAgent","file":"sled-agent/src/vmm_reservoir.rs:270"}
```
After the failure we see the following:
```
BRM42220015 # mdb -ke '::memstat'
Page Summary Pages MB %Tot
------------ ---------------- ---------------- ----
Kernel 15050243 58790 6%
Boot pages 11 0 0%
ZFS File Data 23488784 91753 9%
VMM Memory 214695936 838656 81%
Anon 6306679 24635 2%
Exec and libs 88068 344 0%
Page cache 6815 26 0%
Free (cachelist) 313123 1223 0%
Free (freelist) 5333668 20834 2%
Total 265283327 1036262
Physical 265283325 1036262
```
We do not currently see this on rack2, but we have a sled on Berlin that has reproduced this behavior multiple times as noted by @leftwo .
```
BRM44220007 # grep reservoir $(svcs -L sled-agent) | looker
00:00:52.446Z INFO SledAgent (MemoryReservations): Calculated eligible VMM reservoir size
control_plane_earmark_bytes = 47244640256
file = sled-hardware/src/lib.rs:154
max_page_t_bytes = 31834245120
physical_ram_bytes = 1086608900096
vmm_eligible = 1007530014720
00:00:52.446Z INFO SledAgent: 91% of 1007530014720 VMM eligible ram = 916852313395 bytes)
file = sled-agent/src/vmm_reservoir.rs:258
sled_id = 8f335797-3472-461d-b106-a17e8dc7f255
00:00:52.447Z INFO SledAgent: Setting reservoir size to 874378 MiB bytes
file = sled-agent/src/vmm_reservoir.rs:266
sled_id = 8f335797-3472-461d-b106-a17e8dc7f255
00:00:52.491Z INFO SledAgent: Retrieved SledAgentInfo from Nexus: SledAgentInfo { baseboard: Baseboard { part: "913-0000019", revision: 6, serial: "BRM44220007" }, cpu_family: AmdMilan, decommissioned: false, generation: Generation(2), repo_depot_port: 12348, reservoir_size: ByteCount(0), role: Gimlet, sa_address: "[fd35:e7fa:ed75:103::1]:12345", usable_hardware_threads: 128, usable_physical_ram: ByteCount(1086608900096) }
file = sled-agent/src/nexus.rs:500
sled_id = 8f335797-3472-461d-b106-a17e8dc7f255
20:28:38.000Z ERRO SledAgent: Failed to setup VMM reservoir
error = Failed to create reservoir: Failed to resize reservoir: Not enough space (os error 12)
file = sled-agent/src/vmm_reservoir.rs:191
sled_id = 8f335797-3472-461d-b106-a17e8dc7f255
```
Specifically we are failing at https://github.com/oxidecomputer/omicron/blob/main/illumos-utils/src/vmm_reservoir.rs#L50 from our current understanding.
2 条评论