Skip to content

Manual Test Runbook — G2: YAML rule pack + engine

Owner: Sagar  |  Time: ~15 min  |  Sandbox: none required — pure unit test

Purpose

Validate that every rule in the pack has a pass + fail fixture (the auditor's analogue of D3's conftest verify pattern), that every remediation_asset_id resolves to a real entry in claude.md §4, and that the engine reports the right status on each fixture.

Prerequisites

  • Node 20+, npm
  • Repo checked out

Steps

1. Run the engine test suite

cd apps/discovery-auditor
npm ci
npm test -- --testPathPattern engine.test
  • All tests pass (every rule's fail fixture → status=fail, pass fixture → status=pass)
  • "every rule in the pack has a fixture pair (coverage check)" test passes

2. Add a new rule + fixtures (smoke test for the dev loop)

Create a throwaway rule in rules/network.yaml:

  - id: NET-999
    title: TEST — every resource has tags
    severity: low
    target: { collector: resource-graph }
    check:
      kind: count_gt
      where: [{ path: tags, op: not_exists }]
      threshold: 0
    evidence: smoke test
    remediation: smoke test
    remediation_asset_id: D3
    effort_hours: 1
    frameworks: { soc2_cc: [] }
  • npm test fails with the coverage assertion ("rule NET-999 has no fixture pair") — this is what we want; it proves the safety net works
  • Add a fixture pair to engine.test.ts and re-run; passes
  • Revert both changes

3. Verify every remediation_asset_id resolves

grep -hE '^[[:space:]]+remediation_asset_id:' apps/discovery-auditor/rules/*.yaml \
  | awk '{print $2}' | sort -u | while read -r id; do
  grep -qE "\\*\\*${id}\\*\\*" claude.md || echo "MISSING: ${id}"
done
  • No MISSING: lines printed

4. YAML structural sanity

for f in apps/discovery-auditor/rules/*.yaml; do
  python3 -c "import yaml,sys; yaml.safe_load(open('$f'))" || echo "BAD: $f"
done
  • No BAD: lines

Pass criteria

  • Every rule has a fixture pair and produces correct status
  • Coverage assertion catches missing fixtures (proven via step 2)
  • Every remediation_asset_id resolves to a real claude.md §4 entry
  • All YAML parses cleanly

Failure modes & escalation

Symptom Likely cause Action
Coverage assertion fails New rule added without fixture Add fail + pass fixtures to engine.test.ts
remediation_asset_id missing in claude.md Rule references a future module Either bring the module forward or change to the closest existing asset and add a tech-debt note in §7

Sign-off

  • Tester: ___  |  Date: _  |  Result: PASS / FAIL / N/A
  • Notes: