Skip to content

Emergency Change (Break-Glass) — R3

The exception path to CAB pre-approval. Use it only when waiting for the normal Change Advisory Board review would cause unacceptable harm. The speed is bought with a strict after-the-fact discipline: a complete inline record and a mandatory retrospective review.

This is the SOC2 CC8.1 / ISO A.12.1.2 emergency-change control. R3 does not loosen change management — it makes the break-glass case auditable instead of improvised.

  • Owner: CA (Change Authority)
  • Trigger label: emergency-change
  • Automation: .github/workflows/emergency-change.yml
  • Record gate: tests/emergency-change/validate_emergency_change.py
  • Filing: drives R4 (apps/cab-automation/) in emergency mode → E7 (apps/ticket-platform/) ticket bridge.

1. When break-glass is justified

Reach for emergency-change only when both are true:

  1. There is a live, time-critical impact — an active P1/P2 incident, a security containment action, or a customer-facing outage — where the change resolves or materially reduces the harm; and
  2. The normal CAB pre-approval turnaround (typically ≥1 business day) would let that harm continue.

It is not for "we forgot to get approval", schedule pressure, or convenience. If you can wait for CAB, wait for CAB (the normal path: R4 assesses the PR and the high-impact subset is reviewed before merge).

2. The expedited approval

  1. The change author opens the PR and applies the emergency-change label.
  2. A named human authorizes the break-glass — the incident commander or an on-call lead with change authority. This is real authorization, just verbal/synchronous (incident bridge, on-call channel) instead of the CAB queue. Record who and when in the PR body (see §3, Authorization).
  3. With the record present, the change may merge without waiting for CAB.

The emergency-change workflow does not auto-approve or auto-merge — it records, files, and reminds. A human still clicks merge.

3. The inline Emergency Change Record (required)

The PR body must contain a complete Emergency Change Record block. R1 (pr-template-check.yml) already enforces the standard template; R3 adds these break-glass-only sections on top, gated by the label. The emergency-change.yml workflow fails the PR if any are missing or empty:

Section What it must say
Emergency Change Record The umbrella heading framing the block.
Justification Why break-glass was warranted — what broke / the pressure that made CAB pre-approval unacceptable.
Impact / blast radius What the change touches and who/what it affects. Be specific (paths, subscriptions, blast radius).
Authorization The named human who authorized break-glass, and when (link the incident bridge / on-call log).
Rollback plan How to back the change out, and whether it's destructive.
Retrospective review The review owner (@handle or owner: <name>) and a committed date within the SLA.

A copy-paste-ready fixture lives at tests/emergency-change/fixtures/emergency-pr-body.valid.md and is reproduced in the R3 runbook Part B.

4. What the automation does

When the emergency-change label is present, .github/workflows/emergency-change.yml:

  1. Validates the Emergency Change Record in the PR body (validate_emergency_change.py). Incomplete record → the check fails.
  2. Files an expedited / retroactive CAB ticket by driving R4 (apps/cab-automation/) in emergency mode (--emergency-labels emergency-change), which frames the ticket as a retroactive review of an expedited change and files it through the E7 snowops-ticket CLI.
  3. Dry-run by default — no secrets in CI; R4 writes the ticket body to the run artifact and files nothing.
  4. A real ticket is filed only when a tracker is configured (vars.EMERGENCY_CHANGE_TRACKER, e.g. github) and a token is present — the same "honest dry-run unless gated" stance as the other workflows.
  5. Posts/updates a checklist comment on the PR with the retrospective-review items and the SLA.

5. Retroactive CAB review within the SLA

The break-glass merge creates a debt: a retroactive CAB review owed within the emergency-change SLA (5 business days by default; see RETRO_SLA_DAYS in the workflow). The named owner from the record runs it. The review:

  • confirms the justification and blast radius held up post-hoc;
  • decides whether the change should stand, be amended, or be reverted;
  • feeds any process gap back into the CAB policy (R4) or this runbook.

The PR comment checklist tracks completion. The change does not count as "closed out" until the retrospective is done.

6. Audit trail

The break-glass change is fully reconstructable from:

  • R1 — pr-template-check.yml: the standard PR template was filled.
  • R3 — this record + the emergency-change.yml run (logs + artifact).
  • R4 — apps/cab-automation/ / the expedited CAB ticket (retroactive review).
  • E7 — apps/ticket-platform/: the filed ticket in the tracker.
  • R2 — apps/change-log/: the change appears in the production change log.

Together these are the evidence that an emergency change was authorized, recorded, filed, and reviewed — not bypassed.