Skip to content

Manual Test Runbook — S2: Azure Policy Compliance Dashboard

Owner: Sagar  |  Time: ~6 min (offline) / ~15 min (with live collect)  |  Sandbox: snowops-sandbox-01

Overview

S2 renders a history of E0 compliance snapshots into a versioned dashboard (JSON + self-contained HTML + markdown): current posture, latest-vs-previous regression delta, a trend line, and a per-framework rollup. Two pieces:

  • apps/compliance-dashboard/ — the TS tool. Fully offline — it never touches Azure; it only reads E0's versioned artifacts.
  • S2 workflow (.github/workflows/compliance-dashboard.yml) — daily cron that collects a fresh E0 snapshot (read-only OIDC), appends it to the evidence store, renders the dashboard, and uploads it as an artifact (Pages is opt-in).

It builds on E0's substrate: it consumes the ComplianceSnapshot schemaVersion 1.0 shape and vendors E0's diffSnapshots semantics for the regression flag. Parts A + B verify the aggregation, framework mapping, and rendering offline; Part C is the live collect-and-render path.

Part A — Offline (no cloud, ~5 min)

A1. Build + typecheck

cd apps/compliance-dashboard
npm ci
npm run typecheck
npm run build

A2. Unit tests

npm test

Expect: 3 suites, 28 tests pass (dashboard aggregation + diff + framework rollup; name→framework mapping; HTML golden-file + markdown rendering + escaping).

A3. Offline CLI — render the example history

node dist/index.js --snapshots-dir examples/snapshots \
  --out-dir /tmp/dash --now 2026-05-30T00:00:00.000Z --github-output /tmp/dash-out.txt
cat /tmp/dash-out.txt

Confirm:

  • /tmp/dash/ contains dashboard.json, dashboard.html, dashboard.md.
  • dashboard.json has schemaVersion: "1.0", current.compliancePercentage: 90, meta.snapshotCount: 3, and delta.regressed: true.
  • /tmp/dash-out.txt shows compliance_percentage=90, secure_score=76, regressed=true.
  • dashboard.html is self-contained: opens in a browser with no network; the cis_azure rollup row shows 2 mapped assignments (MCSB fans out to it plus the CIS assignment), and the trend line (<polyline>) is present.

A4. Regression gate + single-snapshot path

# Latest snapshot regressed vs previous → exit 2:
node dist/index.js --snapshots-dir examples/snapshots --out-dir /tmp/dash \
  --now 2026-05-30T00:00:00.000Z --fail-on-regression true ; echo "exit=$?"

# A single snapshot has no delta and no trend → exit 0 even with the gate:
node dist/index.js --input examples/snapshots/2026-05-29.json --out-dir /tmp/dash1 \
  --now 2026-05-30T00:00:00.000Z --fail-on-regression true ; echo "exit=$?"
grep -c "## Change since" /tmp/dash1/dashboard.md

Confirm the first run exits 2; the second exits 0 and its dashboard.md has no ## Change since section (count 0).

Part B — Workflow lint (~1 min)

ruby -ryaml -e "YAML.load_file('.github/workflows/compliance-dashboard.yml')"

Confirm the workflow carries: schedule cron + workflow_dispatch (fail_on_regression, publish_pages inputs); id-token: write for the E0 OIDC collect; the Collect fresh compliance snapshot (E0) step landing a timestamped file under compliance/snapshots/; the Render dashboard (S2) step invoking dist/index.js --snapshots-dir; and the opt-in deploy-pages job gated on inputs.publish_pages.

Part C — Live collect + render (~10 min, ~$0)

Prerequisite: the OIDC SP for the sandbox environment has Reader + Security Reader on the sandbox subscription (same as E0).

  1. Trigger the workflow via workflow_dispatch (Actions → compliance-dashboard → Run) with publish_pages left off.
  2. Confirm the Collect fresh compliance snapshot (E0) step writes a compliance/snapshots/<timestamp>.json, and the Render dashboard (S2) step prints the one-line summary (compliance …%; secure score …%; N snapshot(s)).
  3. Download the compliance-dashboard-<run_id> artifact; open dashboard.html locally — verify the cards, framework rollup, and (if ≥2 snapshots in the store) the trend line render.
  4. (Optional) Commit one or more collected snapshots into compliance/snapshots/, re-run, and confirm the trend line now spans multiple points.

Pass criteria

  • npm test green (3 suites, 28 tests)
  • Offline render produces JSON+HTML+MD; regressed=true, compliance 90% (A3)
  • HTML is self-contained (no network) with framework rollup + trend line (A3)
  • --fail-on-regression exits 2; single-snapshot exits 0 with no diff (A4)
  • compliance-dashboard.yml parses with the E0 collect + S2 render steps (B)
  • (Live) workflow collects a snapshot and uploads a renderable dashboard (C)

Teardown

  • Offline: rm -rf /tmp/dash /tmp/dash1 /tmp/dash-out.txt.
  • Live: remove any test snapshot from compliance/snapshots/ if it shouldn't be retained as evidence; artifacts auto-expire (90-day retention).

Sign-off

  • Tester: Sagar Chhabra  |  Date: 3/6/2026  |  Result: PASS
  • Notes: Only offline part done