Skip to content

Manual Test Runbook — G6: Immutable run audit log

Owner: Sagar  |  Time: ~25 min  |  Sandbox: SnowOps sandbox subscription

Purpose

Validate that the WORM audit log:

  1. accepts new runs (append),
  2. rejects in-place mutation of an existing record (storage-side WORM),
  3. detects tampering via verifyChain if anyone manages to bypass WORM.

Prerequisites

  • G0 + G1 + G4 runbooks passed
  • Storage account exists in the sandbox sub with:
  • Container discovery-audit
  • Versioning enabled
  • Immutability policy on discovery-audit (time-based, ≥1 year, "locked" if you want the strongest guarantee — otherwise leave unlocked for test reusability)
  • SnowOps SP has Storage Blob Data Contributor on the container ONLY (not on the account)
  • SNOWOPS_AUDIT_ACCOUNT_URL + SNOWOPS_AUDIT_CONTAINER configured

Steps

1. Unit tests

cd apps/discovery-auditor
npm ci
npm test -- --testPathPattern audit/log.test
  • Canonical-JSON test passes (key-order independence)
  • 3-record chain verifies clean
  • Tampered field detected
  • Missing record detected

2. Append three real runs

Run G4 three times against the sandbox sub. Capture each run's record_hash from the workflow log.

  • Three blobs appear under runs/<yyyy>/<mm>/
  • Each blob is application/json
  • Each blob's prev_hash matches the prior blob's SHA-256

3. Storage-side WORM enforcement

az storage blob upload \
  --account-name <SA_NAME> --container-name discovery-audit \
  --name runs/2026/05/<existing-run-id>.json \
  --file /etc/hostname --overwrite \
  --auth-mode login
  • Fails with ImmutableBlob or BlobIsImmutable
  • Existing blob content unchanged

4. Local chain verification utility

az storage blob download-batch -d /tmp/audit -s discovery-audit --auth-mode login
node -e "
import('./dist/audit/log.js').then(({ verifyChain }) => {
  const fs = require('fs');
  const files = fs.readdirSync('/tmp/audit/runs', { recursive: true })
    .filter((f) => f.endsWith('.json'))
    .sort();
  const records = files.map((f) => JSON.parse(fs.readFileSync('/tmp/audit/runs/' + f,'utf8')));
  const r = verifyChain(records);
  if (r) { console.error('CHAIN BROKEN:', r); process.exit(1); }
  console.log('chain OK, ' + records.length + ' records');
});
"
  • Prints chain OK, 3 records

5. Tamper simulation (synthetic — does NOT touch WORM)

Copy a record locally, mutate operator, re-run verifyChain:

jq '.operator = "attacker"' /tmp/audit/runs/<yyyy>/<mm>/<file> > /tmp/tampered.json
# replace the file locally only
  • verifyChain returns record_hash mismatch

Pass criteria

  • Three real runs append cleanly
  • WORM blocks overwrite
  • Chain verifies on a clean download
  • Synthetic tamper is detected

Failure modes & escalation

Symptom Likely cause Action
AuthorizationPermissionMismatch on append SP missing Storage Blob Data Contributor on container Re-grant scoped to the container, not the account
Overwrite succeeded Immutability policy not applied or still in disabled state Re-apply via az storage container immutability-policy create
Chain breaks unexpectedly Two concurrent runs raced and computed the same prev_hash Add lease lock to the BlobAuditWriter (open issue if you hit this — current MVP relies on concurrency: in G4)

Sign-off

  • Tester: ___  |  Date: _  |  Result: PASS / FAIL / N/A
  • Notes: