Skip to content

Manual Test Runbook — H7: Break-Glass Accounts + Sign-In Alert

Owner: Sagar  |  Time: ~5 min (Parts A + B, offline) · +30 min Part C (account ceremony + apply) · +20 min Part D (live sign-in drill)  |  Sandbox: snowops-sandbox-tenant-01

Promotes H7 (modules/azure/break-glass/) from 🟦 Code Complete → 🟩 Shipped. Part C creates the 2 break-glass accounts (manual ceremony — H7 does NOT create accounts or any password), wires them into the role-assignable group, grants permanent Global Administrator, and applies the sign-in alert. Part D proves the alert fires on a live break-glass sign-in. Requires Microsoft Entra ID P1 (role-assignable groups are a P1 feature).

Prerequisites

  • Sandbox AAD tenant access (Privileged Role Administrator + Global Administrator active)
  • Microsoft Entra ID P1 license active on the sandbox tenant
  • A Log Analytics workspace (F1 / J1) the tenant sign-in logs flow into
  • At least one standing Global Administrator OUTSIDE the break-glass group (so a destroy can't strip the only admin path)
  • 2 hardware FIDO2 security keys (for the break-glass account ceremony)
  • Local tooling: terraform >= 1.6, go >= 1.22, az CLI >= 2.50, jq
  • SNOWOPS_SANDBOX_TENANT_ID + SNOWOPS_SANDBOX_SUBSCRIPTION_ID env vars set
  • Working directory: repo root

Steps

Part A — terraform fmt + validate (offline, ~2 min)

  1. Module + example formatting + structural validity:
terraform -chdir=modules/azure/break-glass fmt -recursive -check
terraform -chdir=modules/azure/break-glass init -backend=false -input=false
terraform -chdir=modules/azure/break-glass validate
terraform -chdir=modules/azure/break-glass/examples/basic init -backend=false -input=false
terraform -chdir=modules/azure/break-glass/examples/basic validate

Expected: Success! for module + example.

  1. Run the H7 offline Terratest case:
cd tests/terratest
go test -v -timeout 5m -run 'TestBreakGlassValidate' ./modules/azure/

Expected: 1 top-level test passes.

Part B — full Terratest suite (offline, ~3 min)

  1. Run the whole offline suite:
cd tests/terratest
go test -v -timeout 10m ./...

Expected: 23 top-level tests pass.

Part C — break-glass account ceremony + apply (~30 min)

The accounts are a deliberate manual ceremony. H7 never creates a user or generates a password (Identity > Secrets — no secret in Terraform state). Create them by hand, register hardware FIDO2, store a split-knowledge password in a physical safe, then feed the object IDs to the module.

  1. Create the 2 break-glass accounts (cloud-only, in the verified custom domain — never <tenant>.onmicrosoft.com):
az ad user create --display-name "snowops-break-glass-01" \
  --user-principal-name "break-glass-01@<verified-domain>" \
  --password "$(openssl rand -base64 30)" --force-change-password-next-sign-in false
az ad user create --display-name "snowops-break-glass-02" \
  --user-principal-name "break-glass-02@<verified-domain>" \
  --password "$(openssl rand -base64 30)" --force-change-password-next-sign-in false

Then, in the portal, for EACH account: register a hardware FIDO2 key (Security info → Add → Security key) and write the generated password (split knowledge, two halves to two officers) into the physical safe. These steps cannot be automated — that is the point.

  1. Capture the inputs:
export SNOWOPS_SANDBOX_TENANT_ID="<sandbox-tenant-guid>"
export SNOWOPS_SANDBOX_SUBSCRIPTION_ID="<sandbox-sub-guid>"
export BG_IDS=$(az ad user list \
  --filter "startsWith(displayName,'snowops-break-glass')" \
  --query '[].id' -o json)
export LAW_ID=$(az monitor log-analytics workspace show \
  -g snowops-security-rg -n snowops-law --query id -o tsv)
echo "$BG_IDS"   # confirm exactly 2 object IDs
  1. Ensure tenant sign-in logs reach the workspace (no native TF resource for the Entra-tenant diagnostic setting — this is the one manual wiring step):
az monitor diagnostic-settings create \
  --name snowops-entra-signin \
  --resource "/providers/Microsoft.aadiam/diagnosticSettings" \
  --logs '[{"category":"SignInLogs","enabled":true},{"category":"NonInteractiveUserSignInLogs","enabled":true}]' \
  --workspace "$LAW_ID" 2>/dev/null || \
  echo "Configure via Entra → Diagnostic settings if the CLI form errors for your tenant."
  1. Apply against the sandbox tenant:
cd tests/terratest/fixtures/break-glass
terraform init
terraform apply \
  -var "tenant_id=$SNOWOPS_SANDBOX_TENANT_ID" \
  -var "subscription_id=$SNOWOPS_SANDBOX_SUBSCRIPTION_ID" \
  -var "break_glass_member_object_ids=$BG_IDS" \
  -var "log_analytics_workspace_id=$LAW_ID"
  1. Spot-check:
  2. Entra → Groups → snowops-break-glass-accounts: role-assignable, 2 members.
  3. Entra → Roles → Global Administrator → Assignments → Active: the group listed (permanent — NOT eligible).
  4. terraform output assigned_directory_role_template_ids → {"Global Administrator" = "62e90394-..."}.

  5. terraform output alert_enabled → true; sign_in_alert_rule_id non-null.

  6. Dry-run the alert KQL before the live drill — paste the query into Log Analytics → Logs (expect 0 rows until the drill):

terraform output -raw sign_in_alert_query

Part D — live sign-in drill (~20 min)

  1. From a clean browser session, sign in to https://portal.azure.com as break-glass-01@<verified-domain> (FIDO2). Sign out.

  2. Within the alert window (≤ ~10 min for a PT5M evaluation), confirm:

    • Email arrives at the configured receiver (secops@snowops.example in the fixture — substitute a real inbox).
    • Azure Monitor → Alerts shows a fired alert snowops-break-glass-signin, severity 0, with the break-glass UserPrincipalName in the payload.
    • Re-running the KQL from Step 9 now returns the sign-in row.

  3. (Optional) Repeat for break-glass-02 to confirm both accounts are covered.

Pass criteria

  • Part A — terraform validate passes for module + example
  • Part B — full offline Terratest suite passes (23 top-level tests)
  • Part C Step 8 — role-assignable group with 2 members; group holds PERMANENT (active) Global Administrator
  • Part C Step 8 — alert_enabled = true, alert rule ID non-null
  • Part D Step 11 — break-glass sign-in fires the severity-0 alert AND a receiver is notified within the window
  • A standing Global Administrator outside the break-glass group still exists

Teardown

cd tests/terratest/fixtures/break-glass
terraform destroy \
  -var "tenant_id=$SNOWOPS_SANDBOX_TENANT_ID" \
  -var "subscription_id=$SNOWOPS_SANDBOX_SUBSCRIPTION_ID" \
  -var "break_glass_member_object_ids=$BG_IDS" \
  -var "log_analytics_workspace_id=$LAW_ID"

# Delete the throwaway accounts (NOT created by the module — clean up by hand):
for id in $(echo "$BG_IDS" | jq -r '.[]'); do az ad user delete --id "$id"; done

Break-glass safety check before destroy. Destroy revokes the group's Global Admin (it was granted via the group). Confirm another standing Global Administrator exists BEFORE destroying, or the tenant becomes unmanageable. The Entra diagnostic setting from Step 6 is not reverted by destroy — remove it manually if the workspace is being torn down.

Sign-off

  • Tester: _  |  Date: _  |  Result: PASS / FAIL / N/A
  • Notes: