DR Scenario: Secondary-Region Failover¶

Fail the whole workload over from the primary region to the secondary/paired region using the active-replication half of DR: the Cross-Region Replication Module (L2) (blob object replication + SQL failover group), then cut traffic over at the edge. Copy per workload and fill every <PLACEHOLDER>. Pin real IDs from terraform output on L2 / the Terragrunt Live-Infra Reference (F7), not by hand.

Field	Value
Workload	`<WORKLOAD>`
Primary region	`<PRIMARY_REGION>` (e.g. `eastus`)
Secondary region	`<SECONDARY_REGION>` (e.g. `westus2`)
Env	`<dev / staging / prod>`

Objective¶

Restore service in <SECONDARY_REGION> by failing over the SQL failover group and pointing the edge (DNS / Azure Front Door) at the secondary backend, with the storage tier already replicated by L2 object replication. Then fail back to <PRIMARY_REGION> once it is healthy.

RTO / RPO target¶

RTO: <≤ 2h (prod) / ≤ 4h (staging)> — this is a failover, not a restore; the secondary is already warm.
RPO: <≤ 15 min for SQL (continuous geo-replication) / ≤ 1h for blob (object replication is async — RPO = replication lag)>.

Preconditions¶

L2 SQL failover group exists and shows the secondary server as a healthy replica. Env failover posture: <dev Manual / staging Automatic 60m / prod Automatic 120m>.
L2 blob object replication is configured source→DR account and showing a healthy/recent replication time.
DR-region compute (AKS via Secure AKS Module F3, or VMs) exists and can serve traffic, or is part of scenario-full-region-loss.md if it must be rebuilt.
Front Door / Traffic Manager profile has the secondary backend configured (priority/weight), even if disabled.
You have the resource IDs:

PRIMARY_RG="<rg-primary>"
DR_RG="<rg-secondary>"
FG_NAME="<sql-failover-group-name>"          # az ... --name
PRIMARY_SQL_SERVER="<primary-sql-server>"
PARTNER_SQL_SERVER="<partner-sql-server>"
DR_STORAGE="<dr-storage-account>"
FRONT_DOOR_PROFILE="<afd-profile>"
SUB="<subscription-id>"
az account set --subscription "$SUB"

Recovery steps¶

Declare & start the clock. Record declaration time per the DR plan (dr-plan-template.md §5). Open the war room; notify per §6.

Confirm primary is actually down (avoid a needless failover). Check synthetic monitoring (X8) and Service Health:

az rest --method get \
  --url "https://management.azure.com/subscriptions/$SUB/providers/Microsoft.ResourceHealth/availabilityStatuses?api-version=2022-10-01" \
  --query "value[?contains(id, '$PRIMARY_SQL_SERVER')].properties.availabilityState"

Check SQL replication health / lag before failing over (the lag bounds your actual data loss):

az sql failover-group show \
  --name "$FG_NAME" \
  --server "$PRIMARY_SQL_SERVER" \
  --resource-group "$PRIMARY_RG" \
  --query "{role:replicationRole, state:replicationState, partner:partnerServers}"

Fail over SQL (see scenario-sql-failover.md for the planned-vs-forced detail). Run the command against the secondary server so it becomes primary:

Planned (primary reachable, zero data loss — preferred):

az sql failover-group set-primary \
  --name "$FG_NAME" \
  --server "$PARTNER_SQL_SERVER" \
  --resource-group "$DR_RG"

Forced (primary unreachable, accept replication-lag data loss):
```
az sql failover-group set-primary \
  --name "$FG_NAME" \
  --server "$PARTNER_SQL_SERVER" \
  --resource-group "$DR_RG" \
  --allow-data-loss
```
PowerShell equivalent: Switch-AzSqlDatabaseFailoverGroup -ResourceGroupName $DR_RG -ServerName $PARTNER_SQL_SERVER -FailoverGroupName $FG_NAME [-AllowDataLoss].
Verify storage replication state on the DR account. Object replication is async, so confirm the latest replicated content is acceptable:
```
az storage account or-policy list \
  --account-name "$DR_STORAGE" \
  --resource-group "$DR_RG" \
  --query "[].{rule:rules[].sourceContainer, dest:rules[].destinationContainer}"
```
If a container is critically stale, fall back to a point-in-time restore from L1 (scenario-data-restore.md) for that data set.
Bring up / confirm DR compute. Ensure the AKS workload (F3) in <SECONDARY_REGION> is Healthy and pointing at the now-primary SQL read-write listener <fg-name>.database.windows.net and the DR storage account. Roll/restart pods if they cached the old endpoint:
```
az aks get-credentials --resource-group "$DR_RG" --name "<dr-aks-name>"
kubectl rollout restart deployment/<app> -n <namespace>
kubectl rollout status  deployment/<app> -n <namespace>
```

Cut traffic over at the edge (Azure Front Door). Disable the primary origin / raise the secondary origin priority so the edge routes to <SECONDARY_REGION>:

az afd origin update \
  --profile-name "$FRONT_DOOR_PROFILE" \
  --resource-group "$DR_RG" \
  --origin-group-name "<origin-group>" \
  --origin-name "<primary-origin>" \
  --enabled-state Disabled

az afd origin update \
  --profile-name "$FRONT_DOOR_PROFILE" \
  --resource-group "$DR_RG" \
  --origin-group-name "<origin-group>" \
  --origin-name "<secondary-origin>" \
  --priority 1 --enabled-state Enabled

If routing via DNS instead, update the record to the DR endpoint:

az network dns record-set cname set-record \
  --resource-group "<dns-rg>" --zone-name "<zone>" \
  --record-set-name "<host>" --cname "<dr-endpoint>"

Note TTL — propagation is bounded by the record's TTL.

Update status per the comms plan; declare partial restore.

Validation checklist¶

az sql failover-group show reports the secondary server as replicationRole: Primary.
App can read and write to SQL via the read-write listener.
Critical blobs/files present in the DR storage account (spot-check newest objects against expectation).
Synthetic monitoring (X8) for the public endpoint is green from the edge.
End-to-end smoke test of the workload's critical path passes.
No NSG/Private DNS/Key Vault (F5) access errors in the DR region.
Measured RTO recorded vs target; measured RPO (SQL lag + blob lag) recorded.

Fail-back / cleanup¶

Fail back only after <PRIMARY_REGION> is confirmed healthy and has caught up. Treat fail-back as a planned failover (schedule a window) to get zero data loss.

Confirm primary region health (Service Health) and that the old primary SQL server has re-synced as a secondary replica:

az sql failover-group show --name "$FG_NAME" \
  --server "$PRIMARY_SQL_SERVER" --resource-group "$PRIMARY_RG" \
  --query "{role:replicationRole, state:replicationState}"

Re-seed / verify blob object replication is flowing back the intended direction (or re-point the L2 policy if it is unidirectional — confirm the module config before assuming bidirectionality).

Planned failover back to the primary (no --allow-data-loss):

az sql failover-group set-primary \
  --name "$FG_NAME" --server "$PRIMARY_SQL_SERVER" \
  --resource-group "$PRIMARY_RG"

Re-enable the primary origin and restore original Front Door priorities.
Restart/verify primary-region compute; re-run the validation checklist.
Decommission any temporary DR-only resources; confirm no orphaned ephemeral RGs (Sandbox Cleanup X7 backstops only ephemeral=true RGs — clean others manually).
File the post-DR review (dr-plan-template.md §8).

Sign-off¶

Field	Value
DR event / drill ref	`<id>`
Scenario	Secondary-region failover
Direction	`<failover / fail-back>`
Date / time (UTC)	`<YYYY-MM-DD HH:MM>`
Operator	`<NAME>`
SQL failover type	`<planned / forced (data-loss accepted)>`
Measured RTO / RPO	`<actual>` / `<actual>`
Validation passed	☐ yes ☐ no — notes: `<…>`
Reviewed by (IC)	`<NAME>`