Skip to content

DR Scenario: Full Primary-Region Loss

Worst case: the entire primary region is unavailable (extended Azure regional outage, or it must be treated as lost). This runbook combines the failover scenarios with a rebuild-from-IaC path for anything that was not replicated or backed up cross-region. Copy per workload and fill every <PLACEHOLDER>. This is the scenario the monthly Automated Restore Drill (L4) cannot fully rehearse — walk it as a tabletop at least quarterly.

Field Value
Workload <WORKLOAD>
Lost (primary) region <PRIMARY_REGION> (e.g. eastus)
Recovery (secondary) region <SECONDARY_REGION> (e.g. westus2)
Env <dev / staging / prod>

Objective

Restore the full workload in <SECONDARY_REGION> when <PRIMARY_REGION> is gone: fail over the data tier (warm, via the Cross-Region Replication Module L2), rebuild the stateless/compute tier from IaC (it may not exist in the DR region yet), restore anything region-local from backups (L1), and cut the edge over. Then plan an orderly return once <PRIMARY_REGION> is restored.

RTO / RPO target

  • RTO: <≤ 4h (prod) / ≤ 8h (staging)> — longer than a plain region failover because compute may need to be built, not just promoted.
  • RPO: data tier as per L2/L1 (<≤ 15 min SQL / ≤ 1h blob / daily for vault-only data>); anything region-local with no cross-region copy is at risk — enumerate it in step 2.

Preconditions

  • Paired/secondary region selected and within the same data-residency boundary (Data Residency Policy Module M6).
  • L2 replication healthy into <SECONDARY_REGION> (SQL failover group + blob object replication).
  • L1 vaults are GeoRedundant with cross_region_restore_enabled so vault data is restorable from the paired region.
  • IaC is reachable and deployable: the Terragrunt Live-Infra Reference (F7) live/ units, with a <SECONDARY_REGION> unit (or one you can stamp out), and module versions pinned via the Module Registry & Versioning (F11).
  • OIDC deploy identity (Azure Client Bootstrap Module B2) and the Terraform state backend (F6) are reachable — state must not live only in the lost region. If it did, this is why F6 uses RA-GZRS; recover the state account first (step 3).
  • You have edge control (Azure Front Door / DNS) outside the lost region.
DR_RG="<rg-secondary>"
SUB="<subscription-id>"
SECONDARY_REGION="<westus2>"
az account set --subscription "$SUB"

Recovery steps

  1. Declare a major DR event & start the clock. Page on-call (On-Call Integration K2), open the war room, assign an Incident Commander. Record declaration time (dr-plan-template.md §5). If the cause is security, not infrastructure, also open the matching Incident Response Runbook (K1).

  2. Triage what is recoverable vs at-risk. From the architecture diagram (Diagram Generator V2) and operational runbooks (Runbook Generator V3), list each component and its DR posture: | Component | Cross-region posture | Recovery path | |---|---|---| | SQL | L2 failover group (warm) | step 4 | | Blob/Files | L2 object replication / L1 GRS vault | steps 4–5 | | AKS / compute | usually region-local | rebuild from IaC, step 6 | | Key Vault (F5) secrets | per-vault — confirm | step 6 (recreate + reseed) | | VMs / region-local data | L1 GeoRedundant vault | step 5 | Anything with no cross-region copy is a known data-loss item — record it.

  3. Confirm state & identity survive the region. Verify the F6 Terraform state account is reachable (RA-GZRS read access or failover) and the OIDC app (B2) authenticates. Without these you cannot deploy IaC — recover them first.

  4. Fail over the data tier (warm half). Follow scenario-sql-failover.md (forced if the primary is truly gone) and verify storage object replication on the DR account per scenario-region-failover.md step 5.

  5. Restore region-local data that had no live replica from the L1 GeoRedundant vault into <SECONDARY_REGION> (cross-region restore) per scenario-data-restore.md.

  6. Rebuild compute & region-local infra from IaC in <SECONDARY_REGION>. Re-apply the F-module chain via F7 Terragrunt in dependency order (mirrors SEQUENCING.md Phase 4): baseline (F1) → network-hub (F2) / key-vault (F5) / acr (F4) → AKS (F3) → ArgoCD app-of-apps (F8): 

    cd live/<SECONDARY_REGION-or-dr>/<env>
terragrunt run-all plan   --terragrunt-non-interactive
terragrunt run-all apply  --terragrunt-non-interactive
    Reseed secrets into the rebuilt Key Vault (F5) from your secret system of record (never from the lost region). Point workloads at the now-primary SQL read-write listener and the DR storage account.

  7. Cut traffic over at the edge (Front Door origin / DNS) to <SECONDARY_REGION> — same commands as scenario-region-failover.md step 7.

  8. Re-establish guardrails in the recovery region. Confirm policy/Defender (F1), NSG baseline (N6), private-endpoint posture (N5), budget alerts (U1), alert rule pack (J4), and synthetic monitoring (X8) are active in <SECONDARY_REGION> — a rebuilt region must not be less hardened than the original (Day-Zero Hardening).

  9. Update status per the comms plan; declare service restored.

Validation checklist

  • SQL primary is in <SECONDARY_REGION>; app reads and writes.
  • Critical blob/file/VM data present and consistent (spot-check newest).
  • AKS workload (F3) Healthy; smoke test of the critical path passes.
  • Secrets reseeded; no Key Vault (F5) access errors.
  • Edge routes to <SECONDARY_REGION>; synthetic monitoring (X8) green.
  • Guardrails active (policy, Defender, NSG, budget, alerts) in the recovery region.
  • Known data-loss items from step 2 documented and communicated.
  • Measured RTO / RPO recorded vs target.

Return to the primary region (fail-back)

Only after <PRIMARY_REGION> is fully restored. Treat as a planned migration, not an emergency — schedule a window.

  1. Re-deploy / verify infra in <PRIMARY_REGION> from IaC (F7), at parity with the recovery region.
  2. Re-establish replication back toward <PRIMARY_REGION> (re-seed L2; confirm direction) and let it catch up.
  3. Planned SQL failover back (scenario-sql-failover.md, no --allow-data-loss).
  4. Restore edge routing to <PRIMARY_REGION>; re-run validation.
  5. Decommission temporary DR-only resources; confirm no orphaned RGs (Sandbox Cleanup X7 only handles ephemeral=true).
  6. File the post-DR review (dr-plan-template.md §8) and feed lessons back into L1/L2 config and the L4 drill scope.

Sign-off

Field Value
DR event ref <id>
Scenario Full primary-region loss
Lost region / recovery region <PRIMARY_REGION> / <SECONDARY_REGION>
Direction <recover / return-to-primary>
Date / time (UTC) <YYYY-MM-DD HH:MM>
Incident Commander <NAME>
Operator(s) <NAMES>
Components rebuilt from IaC <list>
Known data-loss items <list / none>
Measured RTO / RPO <actual> / <actual>
Validation passed ☐ yes ☐ no — notes: <…>
Reviewed by <NAME>