E5 — Evidence Sync: Test Runbook¶

Asset: Evidence Sync (E5) Location: apps/evidence-collector/src/sync.ts + .github/workflows/evidence-sync.yml Owner: SnowOps (SO) Milestone: M4

Overview¶

E5 orchestrates the evidence chain: reads an E0 ComplianceSnapshot + E4 Resource Graph query results, maps them to EvidenceItem[], and pushes via an E1/E2/E3 EvidencePlatform. Scheduled weekly via evidence-sync.yml.

Part A — Offline: unit tests¶

cd apps/evidence-collector
npm ci
npm test -- --testPathPattern="sync"

Expected: All tests pass. Tests cover: - Snapshot-derived items: CC7.1 + A.8.15 policy compliance, CC7.1 secure score - Pass/fail thresholds (default 90% policy, 70% Defender) - null secureScore.percentage → not_applicable - Resource Graph items forwarded correctly - Error isolation (one failing push doesn't abort the rest) - Empty options → 0 pushed, 0 errors

Part B — Offline: dry-run workflow dispatch¶

Trigger the workflow in dry-run mode from GitHub Actions UI or CLI:

gh workflow run evidence-sync.yml \
  -f platform=dry-run \
  -f dry_run=true

Expected: Workflow completes successfully. Check the uploaded artifact compliance-snapshot-* contains snapshot.json and summary.md.

Part C — Live: full sync to Vanta (requires Azure + Vanta credentials)¶

Prerequisites: - Repository secrets set: VANTA_API_KEY, VANTA_ACCOUNT_ID - Repository secrets set: AZURE_CLIENT_ID, AZURE_TENANT_ID, AZURE_SUBSCRIPTION_ID

gh workflow run evidence-sync.yml \
  -f platform=vanta \
  -f dry_run=false

Expected: Workflow completes. Verify in the Vanta dashboard that controls for SOC2 CC7.1, ISO A.8.15, and the 7 Resource Graph queries are updated.

Sign-Off Block¶

Step	Executed by	Date	Result
Part A (unit tests)			⬜
Part B (dry-run dispatch)			⬜
Part C (live Vanta sync)			⬜

Signed off: _____ Date: ___

Failure Modes¶

Failure	Cause	Remediation
`platform vanta requires VANTA_API_KEY`	Secret not set	Add VANTA_API_KEY to repository secrets
Evidence sync exits non-zero	One or more items failed to push	Check error log for `framework:controlId: <error>`
Snapshot not found	E0 collect step failed	Check Azure login and subscription ID