Skip to content

Manual Test Runbook — G1: Discovery collectors

Owner: Sagar  |  Time: ~25 min  |  Sandbox: SnowOps sandbox subscription

Purpose

Validate each collector against the sandbox sub, seeded with known good + known bad resources, and confirm finding counts match expectations.

Prerequisites

  • G0 runbook passed; SP exists with Reader + Security Reader
  • X1 sandbox subscription provisioned per X1.md
  • Node 20+, npm
  • az login as the SP (use az login --service-principal --federated-token from a GHA run, or temporary cert auth locally)

Steps

1. Install + build

cd apps/discovery-auditor
npm ci
npm run build
  • tsc exits 0
  • dist/index.js exists and is executable via Node

2. Seed sandbox with known fixtures

Use the policies already in X1 plus one each of:

  • One storage account with publicNetworkAccess=Enabled (will trigger NET-001)
  • One storage account with minimumTlsVersion=TLS1_0 (will trigger ENC-001)
  • One Key Vault with enablePurgeProtection=false (will trigger ENC-003)
  • Verify Defender free-tier on at least 4 plans (will trigger LOG-001 in default config)

3. Run collectors-only (no PDF, no audit log)

node dist/index.js \
  --tenant-id $(az account show --query tenantId -o tsv) \
  --subscription-id $(az account show --query id -o tsv) \
  --out-dir ./out \
  --no-pdf
  • Exit code 0
  • out/run-*.json exists
  • Open JSON: each collector has rows.length > 0 (resource-graph + defender + policy at minimum)
  • errors[] on aad-audit is acceptable if tenant lacks AAD P1; record it

4. Verify finding correctness

In the JSON, find each fixture in findings:

  • NET-001 status=fail with the seeded storage account in offenders[]
  • ENC-001 status=fail with the seeded TLS1_0 account
  • ENC-003 status=fail with the seeded vault
  • At least one rule status=pass (sanity)

5. Performance sanity

  • Total runtime < 90s for a sandbox sub
  • Resource-graph pagination is followed (no $skipToken in last page)

Pass criteria

  • Every seeded violation surfaces as the matching rule's failure
  • No collector throws an unhandled exception
  • JSON artifact is well-formed and roundtrips through JSON.parse

Failure modes & escalation

Symptom Likely cause Action
403 on Graph (AAD audit) Tenant lacks P1 license Acceptable; verify it's recorded in errors[], not thrown
429 on Cost Mgmt Concurrent runs against same sub Re-run; concurrency: group in G4 prevents this in production
Empty secureScores Defender not initialized on a brand-new sub Wait 24h after first resource is created

Sign-off

  • Tester: ___  |  Date: _  |  Result: PASS / FAIL / N/A
  • Notes: