Skip to content

SnowOps — Testing Strategy, Definition of Done & Update Protocol

Testing Layers

Layer Tool Scope Runs When
Unit (IaC) terraform validate, tflint Per module pre-commit + PR
Policy unit conftest verify, kyverno test OPA/Kyverno rules pre-commit + PR
Static security checkov, tfsec, trivy fs, gitleaks All IaC + repo pre-commit + PR
Module integration Terratest (Go) Real apply to X1 sandbox PR (matrix per module) + nightly
Import validation Terratest import path F12 brownfield import blocks PR per module
Workflow integration "Test consumer repo" exercising reusable workflow end-to-end All pipelines/*.yml Nightly
App unit Jest (TS apps) apps/ PR
App integration Mock external APIs + sandbox cloud calls apps/ PR + nightly
Synthetic Azure Monitor synthetic tests Reference architectures in sandbox Continuous
Manual runbook (X6) Markdown step-by-step in docs/runbooks/test/ Per asset, before first client use Human-gated
Compliance attestation Vanta dev account in dry-run mode Asset E flow end-to-end Pre-release
DR drill L4 automated restore Backup → restore → validate → destroy Monthly in sandbox
Tabletop IR scenarios from K5 Team exercise Quarterly

Manual Test Runbook Template

Every asset ships with one at docs/runbooks/test/<asset_id>.md.

# Manual Test Runbook — <Asset ID>: <Name>
**Owner:** Sagar | **Time:** ~XX min | **Sandbox:** snowops-sandbox-01

## Prerequisites
- [ ] Sandbox sub access via PIM activated
- [ ] Local tooling: ...
- [ ] Feature flag / env vars: ...

## Steps
1. ...
2. ...

## Pass criteria
- [ ] Observable A: <expected state>
- [ ] Observable B: <expected state>
- [ ] No errors in <log location>

## Teardown
- [ ] `terraform destroy` / `kubectl delete` / cleanup workflow run

## Sign-off
- Tester: ____ | Date: ____ | Result: PASS / FAIL / N/A
- Notes:

Test Data & Fixtures

  • All fixtures in tests/fixtures/ — versioned, no real client data ever.
  • Synthetic identities, synthetic PII (e.g., Faker generators), synthetic findings.
  • Vanta + HubSpot use developer/sandbox accounts; never production.

Sandbox Guardrails

  • Budget cap per month (Azure Cost Mgmt budget; alerts at 50/80/100%).
  • Auto-destroy nightly for ephemeral=true (X7).
  • Network isolated from any client tenant.
  • Any resource without owner tag → flagged within 24h.

Definition of Done (🟩 Shipped)

An asset moves to 🟩 Shipped only when all ten of the following are true:

  1. Code complete — implementation merged to main.
  2. Automated test exists and passes — per testing layer appropriate to the asset (above).
  3. Manual runbook exists — docs/runbooks/test/<asset_id>.md per template above.
  4. Runbook executed by a human — sign-off block filled in, result PASS.
  5. Docs updated — module README, ADR if architectural, CLAUDE.md state block.
  6. Failure mode known — what breaks, how it surfaces, who pages.
  7. Cost impact known — sandbox + estimated prod cost noted in module README.
  8. Removal path known — clean terraform destroy / uninstall verified in sandbox.
  9. Brownfield path documented (F-modules only) — either F12 import block exists or "greenfield only" is explicitly noted with M3 tracking.
  10. Client-facing guide updated — the appropriate docs/client-guides/*.md reflects the asset.

No 🟩 without all ten. Anything less is 🟦 (Code Complete) or 🧪 (Tested but not signed off).

Update Protocol

At the end of every implementation milestone or significant drop, complete ALL of these steps before declaring the phase done.

Steps

  1. Update asset status icons in docs/context/04-asset-status.md.
  2. Update full asset descriptions in docs/context/03-asset-catalog.md (add → Test: results, key design notes).
  3. Update docs/context/06-project-state.md (§ In Flight, § Runbook Backlog, § M2b core progress).
  4. Update CLAUDE.md § Machine State block (version, updated, assets map, m2b_core_shipped/remaining).
  5. Add a row to docs/context/07-decisions.md if a new architectural decision was made.
  6. Add an entry to docs/context/05-history.md (the vX.X completed-work log).
  7. Bump version + Last Updated at the top of CLAUDE.md.
  8. Commit: chore(context): bump claude.md to vX.Y.

Phase Handoff (Mandatory)

Every phase entry in docs/context/05-history.md must include both subsections:

(a) Verification block — concrete, copy-pasteable commands or runbook pointers the user can run locally to verify the work that just landed. Distinguish "verify locally" (no cloud auth) from "end-to-end" (requires sandbox subscription).

(b) Next phase suggestion — explicitly call out what Claude proposes for the next phase. Name any prerequisites a human must satisfy (Azure credentials, GitHub repo settings, etc.).

A phase that lands code but skips (a) and (b) is NOT done.