Manual Test Runbook — E0: Lightweight Compliance Snapshot

Owner: Sagar | Time: ~10 min (offline) / ~20 min (with live apply) | Sandbox: snowops-sandbox-01

Overview

E0 is the Baseline-tier evidence floor: a versioned JSON compliance snapshot (Azure Policy compliance state + Defender secure score) emitted on every C1 apply. Two pieces:

apps/evidence-collector/ — the TS tool (read-only ARM: Reader + Security Reader).
C1 (.github/workflows/terraform-plan-apply.yml) — post-apply step that emits
uploads the snapshot artifact when emit_compliance_snapshot: true (default).

The catalog test criterion: C1 apply in sandbox emits a snapshot artifact; assert required fields present; diff between two runs reflects a seeded policy violation. Parts A + B cover the assert-fields + seeded-violation-diff offline; Part C is the live C1 apply.

Part A — Offline (no cloud, ~5 min)

A1. Build + typecheck

cd apps/evidence-collector
npm ci
npx tsc -p tsconfig.json --noEmit
npx tsc -p tsconfig.test.json --noEmit
npm run build

A2. Unit tests

npm test

Expect: 3 suites, 18 tests pass (snapshot build/validate + collector parsers, diff regression detection, markdown rendering).

A3. Offline CLI — required fields + seeded-violation diff

# First snapshot from the clean sample state:
node dist/index.js --subscription-id SAMPLE \
  --input examples/collected.sample.json --out-dir /tmp/ec-snap1

# Second snapshot from the regressed state, diffed against the first:
node dist/index.js --subscription-id SAMPLE \
  --input examples/collected.regressed.json \
  --baseline /tmp/ec-snap1/snapshot.json --out-dir /tmp/ec-snap2

cat /tmp/ec-snap2/summary.md

Confirm in /tmp/ec-snap2/summary.md:

Required fields present in /tmp/ec-snap1/snapshot.json (schemaVersion, generatedAt, subscriptionId, policyCompliance.*, secureScore.*, meta.*).
## Change since … shows ⚠️ Posture regressed.
### Newly non-compliant assignments lists snowops-tls: 0 → 3 (the seeded violation).

Part B — Workflow lint (~1 min)

ruby -ryaml -e "YAML.load_file('.github/workflows/terraform-plan-apply.yml')"

Confirm the apply job now carries the E0 steps (Emit compliance snapshot (E0) + Upload compliance snapshot artifact (E0)) and the new inputs (emit_compliance_snapshot, evidence_tool_repo, evidence_tool_ref).

Part C — Live C1 apply emits the snapshot (~15 min, ~$0)

Prerequisite: a sandbox repo consuming C1 (per docs/runbooks/test/C1.md) with the B2-provisioned OIDC SP granted Reader + Security Reader on the sandbox sub (in addition to its deploy roles).

Trigger a no-op apply (merge a trivial change to the default branch).
In the run's apply job, confirm the Emit compliance snapshot (E0) step succeeds and the compliance-snapshot artifact is attached to the run.
Download the artifact; open snapshot.json — assert schemaVersion: "1.0", non-null policyCompliance.compliancePercentage, and a secureScore block.
(Optional, the seeded-violation half) Seed a policy violation in the sandbox (e.g. create a storage account that the M3 TLS-deny initiative marks non-compliant), wait for Policy evaluation, run a second apply, and diff the two snapshot.json files with node dist/index.js --input <collected> --baseline <prior snapshot.json> — the new non-compliant assignment appears in the diff.

Pass criteria

npm test green (3 suites, 18 tests)
Offline diff (A3) shows the seeded snowops-tls violation + regressed
C1 workflow YAML parses with the E0 steps present
(Live) C1 apply attaches a compliance-snapshot artifact with all required fields

Teardown

Offline: rm -rf /tmp/ec-snap1 /tmp/ec-snap2.
Live: delete any seeded violation resource; artifacts auto-expire per artifact_retention_days.

Sign-off

Tester: _ | Date: _ | Result: PASS / FAIL / N/A
Notes: