# Mini-Plan: Azure stuck-VM recovery hardening (SCT-632)

**Date:** 2026-07-16
**Estimated LOC:** ~300 (≈120 src, ≈180 tests)
**Related issue:** [SCT-632](https://scylladb.atlassian.net/browse/SCT-632)
**Design spec:** `docs/superpowers/specs/2026-07-16-sct632-azure-stuck-vm-recovery-design.md`

## Problem

The stuck-VM recovery added in #15004 loses to Azure eastus degradation in three ways:
the redeploy step never rescues `Creating`-stuck VMs (0/14 observed attempts, wastes up
to 5 min each), per-VM 900s waits are serialized (2 stuck VMs ≈ 31 min per round), and
recovery has no total time budget, so the Jenkins 90-min "Provision Resources" stage
SIGTERMs hydra mid-recovery — no error raised, no events flushed, no logs collected
(builds #1020/#1023/#1025 of longevity-10gb-3h-azure-test).

## Approach

Ordered steps; one PR, logically split commits (1–2 / 3 / 4–5 / 6).

1. **Remove the redeploy path** (`provisioner.py`): delete `_redeploy_stuck_node()`,
   `DEFAULT_REDEPLOY_TIMEOUT`, redeploy wording in the `_provision_with_stuck_recreate`
   docstring; recovery round = `InstanceProvisionStuckEvent` (message: "recreating stuck
   VM on fresh capacity") → `_delete_stuck_node()` → loop re-enters
   `_provision_resources`.
2. **Remove orphaned wait helpers** (`virtual_machine_provider.py`): delete
   `recheck_provisioned()` (only caller was `_redeploy_stuck_node`); reword the
   "will be redeployed to a new node" warning (~line 190) to "will be recreated on
   fresh capacity".
3. **Shared-deadline wait** (`virtual_machine_provider.py`): replace the serialized
   per-VM `_wait_until_provisioned()` calls in `get_or_create()` with
   `_wait_all_provisioned(names, pricing_model, deadline)`:
   - one shared deadline: `min(monotonic() + self._stuck_vm_timeout, deadline)`;
   - sweep all pending VMs each `_stuck_vm_poll_interval`, collect on `Succeeded`,
     classify still-pending as stuck at deadline → caller raises
     `StuckVMProvisioningError` (keep `ProvisionError` precedence from the quick
     poller check);
   - per-poll exception contract: spot 404 → `SpotTerminationEvent` + immediate
     `OperationPreemptedError` (deliberate fast-fail, was after full window); any other
     exception → WARNING log, VM stays pending;
   - delete `_wait_until_provisioned()` once absorbed.
4. **Total recovery budget** (`provisioner.py`): new
   `DEFAULT_STUCK_VM_TOTAL_TIMEOUT = 4500` next to `DEFAULT_STUCK_VM_RECREATE_ATTEMPTS`
   (line 51); read `azure_provision_stuck_vm_total_timeout` from config in `__init__`
   like the existing stuck params (lines 65–70); in `_provision_with_stuck_recreate`:
   capture `deadline = monotonic() + total_timeout` at entry; before each recovery
   round require `monotonic() + self._stuck_vm_timeout <= deadline` else give up;
   pass `deadline` down through `_provision_resources` → `get_or_create` to cap the
   in-round wait; give-up (attempts or budget) raises `ProvisionUnrecoverableError`
   with the exhausted budget named in the event message.
5. **Config param wiring**: field `azure_provision_stuck_vm_total_timeout` in
   `sdcm/sct_config.py` azure options (after `azure_provision_stuck_vm_recreate_attempts`,
   ~line 1465) — description states the interaction with `recreate_attempts` (whichever
   budget exhausts first wins); default `4500` in `defaults/azure_config.yaml`;
   pass-through in `sdcm/tester.py` (~line 1749) and
   `sdcm/sct_provision/azure/azure_region_definition_builder.py` (~line 59);
   `docs/configuration_options.md` regenerated by the `update-conf-docs` hook.
6. **Tests** (`unit_tests/unit/provisioner/`): drop redeploy tests and the
   `begin_redeploy` / `VMCreateBehavior.redeploy_fails` hooks in
   `fake_azure_service.py`; add/port per the spec's Testing section.
7. **Nemesis capacity-skip for Azure** (added after build #1027 evidence): teach
   `skip_on_capacity_issues` and `critical_on_capacity_issues`
   (`sdcm/utils/decorators.py`) to handle `ProvisionUnrecoverableError` like the
   existing AWS/GCE capacity branches, so a mid-test stuck-VM give-up (e.g.
   `AddRemoveDcNemesis` adding a new-DC node) skips the nemesis with
   `UnsupportedNemesis("Capacity Issue")` instead of failing the test with a
   `DisruptionEvent` ERROR. Import is deferred inside the wrappers —
   `sdcm.provision` package init imports `decorators` back (circular import).

## Files to Modify

- `sdcm/provision/azure/provisioner.py` — remove redeploy; budget-aware recovery loop;
  new default constant + config read; deadline plumbed into `_provision_resources`
- `sdcm/provision/azure/virtual_machine_provider.py` — `_wait_all_provisioned()` with
  shared deadline + per-poll exception contract; remove `_wait_until_provisioned()`
  and `recheck_provisioned()`
- `sdcm/sct_config.py` — new `azure_provision_stuck_vm_total_timeout` field
- `defaults/azure_config.yaml` — default `azure_provision_stuck_vm_total_timeout: 4500`
- `sdcm/tester.py` — pass new param to provisioner config (existing stuck-param block)
- `sdcm/sct_provision/azure/azure_region_definition_builder.py` — same pass-through
- `docs/configuration_options.md` — regenerated (hook)
- `unit_tests/unit/provisioner/test_stuck_vm_recreate.py` — adapt suite (flat pytest
  functions)
- `unit_tests/unit/provisioner/fake_azure_service.py` — drop redeploy hooks
- `sdcm/utils/decorators.py` — `ProvisionUnrecoverableError` branches in both
  capacity-issue decorators (deferred import to avoid the circular import)
- `unit_tests/unit/test_utils_decorators.py` — capacity-decorator tests for the
  stuck-VM give-up (nemesis skip, critical event)

## Verification

- [ ] Unit tests pass: `uv run python -m pytest unit_tests/unit/provisioner/ -v`
- [ ] New tests cover: two stuck VMs recovered within one shared-deadline round;
      budget pre-check gives up with attempts remaining (`ProvisionUnrecoverableError`,
      budget-named message); in-round wait capped by total deadline; transient
      `AzureError` on one poll does not abort the batch; spot 404 → immediate
      `OperationPreemptedError`; recreate-succeeds-on-attempt-N unchanged
- [ ] `grep -rn "redeploy" sdcm/provision/azure/` returns nothing
- [ ] `uv run sct.py pre-commit` passes (includes `update-conf-docs` regeneration)
- [ ] Staging run of `longevity-10gb-3h-azure-test` (eastus) provisions green; if a
      stuck VM occurs, console shows a single shared wait per round and no redeploy
      attempts
