Manual Test Runbook — R3: Emergency Change Workflow (break-glass)¶

Owner: Sagar | Time: ~5 min (Parts A + B offline) · +5 min (optional Part C live) | Cloud: none · $0

Promotes R3 (.github/workflows/emergency-change.yml + tests/emergency-change/) from 🟦 Code Complete → 🟩 Shipped. Parts A/B are offline ($0). Part C opens a labelled PR and confirms the workflow files an expedited CAB ticket + posts the checklist.

Prerequisites¶

Local tooling: python3 (Part A), node >= 20 + npm (Part C build), optionally gh (Part C)
(Part C only) apps/cab-automation + apps/ticket-platform buildable; a test repo to open a PR against
Working directory: repo root

Steps¶

Part A — validator unit tests + both fixture paths (offline, ~2 min)¶

Run the offline gate (unit tests, then prove both fixtures):

tests/emergency-change/validate.sh

Expected: the unit tests pass; the valid fixture exits 0 (OK: Emergency Change Record is complete …); the broken fixture exits 1 (FAIL: … missing required section: Retrospective review …) and validate.sh reports that the non-zero exit was expected. Final line: ==> OK: R3 offline gate passed.

(Optional) Run the validator by hand against each fixture to see the messages:

python3 tests/emergency-change/validate_emergency_change.py \
  tests/emergency-change/fixtures/emergency-pr-body.valid.md; echo "exit=$?"
python3 tests/emergency-change/validate_emergency_change.py \
  tests/emergency-change/fixtures/emergency-pr-body.missing-retro.md; echo "exit=$?"

Expected: exit=0 for the valid body, exit=1 for the broken one.

Part B — the emergency-change process + a sample PR body (~2 min, $0)¶

Read the process doc: docs/runbooks/change/emergency-change.md — when break-glass is justified, the inline record requirements, the expedited approval, and the retroactive CAB review within the SLA.
The PR body to use when opening an emergency-change PR is the bundled valid fixture. Paste this as the PR description (it satisfies the R3 validator):

Sample emergency PR body (also at tests/emergency-change/fixtures/emergency-pr-body.valid.md)

## Summary

Production Azure Policy was denying a legitimate, approved customer region and
blocking a revenue-critical prod deploy during a P1 incident (INC-4821). Merged
under break-glass to unblock; CAB pre-approval would have exceeded the incident
SLA.

## Scope of change

- `policy/azure-policy/allowed-locations.json` — add `westeurope` to the
  allow-list. One line; no resource changes.

## Emergency Change Record

### Justification

The `allowed-locations` policy denied the customer's contracted region during a
P1, blocking the deploy that resolves the incident. Normal CAB pre-approval has
a 1-business-day turnaround; the live customer impact warranted break-glass.

### Impact / blast radius

Touches `policy/azure-policy/allowed-locations.json` only — widens a deny
allow-list by a single region. Blast radius is one subscription's policy
assignment; no compute, data, or network resources are mutated. Reversible.

### Authorization

Break-glass authorized by @secops-lead (acting incident commander) at
18:02 UTC, recorded on the INC-4821 incident bridge and in the on-call log.

### Rollback plan

Revert this PR. The policy assignment re-applies the previous deny on the next
GitOps sync (~5 minutes). No data migration or stateful change is involved.

### Retrospective review

Retroactive CAB review owner: @change-manager. Committed review date:
2026-06-16 — within the 5-business-day emergency-change SLA.

(Optional) Dry-run R4 in emergency mode locally, the same way the workflow does, to see the expedited ticket body it would file:

( cd apps/cab-automation && npm install && npm run build )
printf '[{"number":901,"title":"fix(policy): break-glass region unblock",\
"url":"https://example/pr/901","state":"open","author":{"login":"secops"},\
"body":"break-glass","labels":[{"name":"emergency-change"}],"files":[]}]' > /tmp/r3-changes.json
node apps/cab-automation/dist/index.js \
  --input /tmp/r3-changes.json --emergency-labels emergency-change \
  --platform dry-run --out-dir /tmp/r3-out
cat /tmp/r3-out/cab-901.md

Expected: a [emergency change] … ticket body framed as a retroactive CAB review; cab-records.json shows emergency: true. Files nothing (dry-run).

Part C — live labelled PR (~5 min)¶

On a test repo with this workflow installed, create the emergency-change label, then open a PR using the Part B body and apply the label:

gh label create emergency-change --color B60205 --description "Break-glass change (R3)" || true
git checkout -b test/r3-emergency
# make a trivial change, commit, push, then:
gh pr create --title "fix(policy): break-glass region unblock" \
  --body-file tests/emergency-change/fixtures/emergency-pr-body.valid.md
gh pr edit --add-label emergency-change

Watch the emergency-change workflow run:

gh run watch

Expected: - the Validate Emergency Change Record step passes (complete record); - R4 runs in emergency mode and either dry-runs the ticket (no vars.EMERGENCY_CHANGE_TRACKER) or files an expedited CAB ticket via E7; - a PR comment appears: "🚨 Emergency change (break-glass) — mandatory retrospective" with the checklist + SLA + audit-trail line.

Negative check — edit the PR body to delete the Retrospective review section and push/edit. Expected: the workflow re-runs (on synchronize / edited) and the validate step fails with a missing-section error.

Pass criteria¶

Part A — validate.sh passes; valid fixture exit 0, broken fixture exit 1
Part B — process doc reviewed; sample PR body validates; (optional) R4 emergency dry-run renders a retroactive-review ticket
Part C — labelled PR triggers the workflow; record validated; ticket filed/dry-run; checklist comment posted; removing a required section fails the gate

Failure mode¶

If the PR body lacks a complete Emergency Change Record, the workflow fails the PR by design — fill in the missing section(s) (see docs/runbooks/change/emergency-change.md §3). If the workflow can't build R4/E7 or no tracker is configured, it honestly dry-runs (writes the ticket body to the run artifact, files nothing) rather than failing.

Cost impact¶

$0 — pure orchestration; read-only against the repo. A filed CAB ticket lives in the tracker and is managed there.

Removal path¶

Delete .github/workflows/emergency-change.yml, tests/emergency-change/, and docs/runbooks/change/emergency-change.md. Nothing external is provisioned.

Sign-Off¶

Field	Value
Part A (validator + fixtures)	☐ PASS
Part B (process + sample body + R4 dry-run)	☐ PASS
Part C (live labelled PR)	☐ PASS / ☐ skipped
Tester
Date
Result	☐ PASS