Skip to content

DR Runbook Templates (L3)

Disaster-recovery runbook templates for SnowOps client engagements. These are blanks you copy per client/workload and fill in — they are not a live DR plan on their own. They turn the existing DR machinery into something an on-call engineer can execute under pressure.

Code → name lookup: docs/context/00-asset-glossary.md. Full catalog: docs/context/03-asset-catalog.md. Bring-up/teardown order: SEQUENCING.md (Phase 5 DR operations, Phase 8 teardown).

What this covers (and what builds it)

DR in SnowOps is three real capabilities plus this template layer:

Code Capability Role in DR Where
L1 Azure Backup Policy Module Recoverability half — GeoRedundant Recovery Services vault (VM / Azure Files / SQL-in-VM) + Data Protection vault (AKS), per-env retention, soft-delete + immutability. modules/azure/backup-policy/
L2 Cross-Region Replication Module Active-replication half — blob object replication (source→DR account) + geo-redundant SQL failover group, per-env failover posture. modules/azure/cross-region-replication/
L4 Automated Restore Drill The proof — monthly restore/failover into an ephemeral sandbox RG, validates, tears down, emits a RestoreDrillReport; feeds the Compliance Dashboard (S2) DR panel. apps/restore-drill/
L3 DR Runbook Templates (this dir) The human procedures — declare, fail over, restore, fail back. docs/runbooks/dr/

L1 makes recovery possible, L2 keeps a warm copy ready, L4 rehearses the procedures below every month, and these templates are what a human follows during a real event. When a DR event overlaps a security incident (ransomware, data leak), pair these with the Incident Response Runbooks (K1) under docs/runbooks/incident/.

The templates

File Use it for
dr-plan-template.md The master DR plan for a client/workload — scope, contacts, objectives, declaration, comms, links to the scenarios below. Start here.
scenario-region-failover.md Failing the whole workload over to the secondary region (storage + SQL + DNS/Front Door cutover) and back.
scenario-data-restore.md Point-in-time restore of VM / Azure Files / AKS / blob data from the Azure Backup Policy Module (L1) into a recovery RG.
scenario-sql-failover.md SQL failover-group-specific procedure — planned vs forced failover, grace period, listener, reseed.
scenario-full-region-loss.md Worst case: total loss of the primary region, combining the above plus a rebuild-from-IaC path.

How to use the templates

  1. Copy, don't edit in place. For a client/workload, copy the relevant files into the client repo (e.g. docs/dr/<workload>/). Keep the originals here as the canonical blanks.
  2. Fill every <PLACEHOLDER>. Anything in <ANGLE_BRACKETS> is a value you must supply (resource IDs, regions, contacts). A template with unfilled placeholders is not an approved plan.
  3. Pin the real resource IDs. Pull them from terraform output on the L1 (recovery_services_vault_id, *_policy_id) and L2 (sql_failover_group_id, object_replication_id) modules, or the Terragrunt Live-Infra Reference (F7) live/ units. Don't hand-type ARM IDs.
  4. Rehearse with L4. The Automated Restore Drill exercises the data restore and SQL failover scenarios monthly into an ephemeral sandbox RG and records a RestoreDrillReport. A scenario you have never seen pass in L4 is a hope, not a plan. The drill outcome surfaces on the Compliance Dashboard (S2) DR panel.
  5. Sign off after every real or rehearsed run. Fill the sign-off block at the bottom of each runbook so there is an auditable record.

RTO / RPO summary by environment

These targets are defaults consistent with what L1 retention and L2 failover posture actually deliver. Override per client SLA in the plan template, but keep them coherent with the underlying module config — a 15-minute RTO is not credible against a Manual failover policy.

Env RPO (data loss budget) RTO (time to recover) Backup retention (L1) SQL failover posture (L2)
dev Best-effort (≤ 24h) Best-effort (next business day) daily 7d Manual failover (no auto grace)
staging ≤ 1h (object replication is async; SQL geo-replica is near-sync) ≤ 4h daily 14d + weekly 5w Automatic, 60-min grace
prod ≤ 15 min for SQL (geo-replica) / ≤ 1h for blob (async object replication) ≤ 2h for failover · ≤ 4h for full restore daily 30d + weekly 12w + monthly 12m + yearly 7y Automatic, 120-min grace

Notes: - RPO is bounded by replication lag, not backup frequency. SQL failover groups replicate continuously (small RPO); blob object replication is async, so its RPO is the replication lag (typically minutes, budget ≤ 1h). Point-in-time restore RPO is bounded by the daily backup schedule (L1 runs daily at the configured backup_time, default 23:00 UTC). - RTO for a failover (warm secondary already replicating) is far shorter than a restore (rehydrating from a vault). Region failover leans on L2; data restore leans on L1. Pick the path that meets the env's RTO. - The prod failover grace of 120 min means an Automatic failover only fires after the primary has been unreachable that long — for a faster planned cutover, use the forced/planned path in scenario-sql-failover.md.

Cross-references

  • Recoverability / restore → Azure Backup Policy Module (L1), modules/azure/backup-policy/README.md.
  • Replication / failover → Cross-Region Replication Module (L2), modules/azure/cross-region-replication/README.md.
  • Rehearsal / proof → Automated Restore Drill (L4), apps/restore-drill/, runbook docs/runbooks/test/L4.md; outcomes on the Compliance Dashboard (S2) DR panel.
  • Rebuild from IaC → Terragrunt Live-Infra Reference (F7), live/.
  • Security overlap → Incident Response Runbooks (K1), docs/runbooks/incident/ (ransomware, data-leak in particular).
  • Ordering → SEQUENCING.md Phase 5 (DR ops) and Phase 8 (teardown).