Skip to content

Manual Test Runbook — S4: Compliance Scorecard Generator

Owner: Sagar  |  Time: ~5 min (offline only — S4 is a pure scorecard generator, no live phase)

Overview

S4 turns one E0 ComplianceSnapshot (+ optionally an S2 ComplianceDashboard for framework rollups and trend) into a versioned, branded one-page scorecard — markdown + a self-contained HTML page (+ optional Pandoc PDF). It is the sales/client-facing counterpart to S2's detailed technical dashboard.

S4 measures nothing and re-derives nothing from Azure: every figure is a re-presentation of an E0/S2 number, with one transparent extra — a documented A–F grading band, with its derivation printed inline (the same "never fabricate, show your work" stance as L5/S3). Pure adapt → render + jest, mirroring E0/S1/S2/L4/L5 — there is no live Azure phase to test (the tool only reads JSON and writes markdown/HTML/PDF).

Part A — Build, typecheck, unit tests (~3 min)

cd apps/compliance-scorecard-generator
npm ci
npm run typecheck
npm run build
npm test

Expect: 2 suites, 24 tests pass — grading-band derivation (incl. the "—" not-yet-measured case and the never-fabricate guarantees), branding resolution + neutral defaults, framework-badge 1:1 re-presentation, trend-direction derivation purely from delta sign (incl. the conservative "any decline reads as declining" rule and the regressed-flag override), highlight generation, the always-present methodology section, markdown section ordering, HTML self-containment + escaping + branding application.

Part B — Offline CLI demo (~2 min)

B1. Generate a full scorecard (snapshot + dashboard + branding)

node dist/index.js \
  --snapshot examples/snapshot.json \
  --dashboard examples/dashboard.json \
  --branding examples/branding.json \
  --output /tmp/s4out
cat /tmp/s4out/scorecard.md

Expect: stdout wrote /tmp/s4out/scorecard.md, .../scorecard.html, .../model.json (grade B for Contoso Logistics). The markdown contains, in order:

  • A branded header — "Q2 2026 Compliance & Security Posture Scorecard", "Prepared for: Contoso Logistics", "Prepared by: SnowOps Cloud".
  • ## Overall posture: B — 88.2% with a measurement table (policy compliance 95.3%, Secure Score 81%) and an inline derivation blockquote: "Overall 88.2% = mean of policy compliance 95.3% and Secure Score 81%, banded to grade 'B' (Good) per the documented grading scale."
  • ## Frameworks — a 4-row table (SOC 2 ✅, ISO 27001 ⚠️, CIS Azure ✅, HIPAA ⚠️) matching the fixture's S2 rollup exactly.
  • ## Trend — ↑ Improving, quoting the real deltas verbatim ("Secure Score +3, policy compliance +1.2%, non-compliant resources -5").
  • ## Highlights — bullets naming the highest-impact framework needing attention (ISO 27001) by non-compliant-resource count.
  • ## Methodology — always present; restates the grading band table and explains the framework-badge / trend derivations.
open /tmp/s4out/scorecard.html   # or: python3 -m http.server --directory /tmp/s4out
cat /tmp/s4out/model.json | head -5   # versioned model.json (schema v1.0)

Expect the HTML: a self-contained styled page (inline CSS only, no external script/stylesheet refs), the B grade tile in green, the client's accent color (#0b5fff) applied to headings/borders, a text wordmark "Contoso Logistics" (the fixture has no logoUrl).

B2. Headline-only run (no dashboard — honest "not supplied" sections)

node dist/index.js --snapshot examples/snapshot.json --client-name "Solo Run Inc" --output /tmp/s4solo
grep -A1 "## Frameworks" /tmp/s4solo/scorecard.md
grep -A1 "## Trend" /tmp/s4solo/scorecard.md

Expect: both sections render and state plainly that no S2 rollup/delta was supplied — they are not silently omitted. The methodology section omits the framework-badge and trend-derivation paragraphs (nothing to explain).

B3. Optional PDF (best-effort)

node dist/index.js --snapshot examples/snapshot.json --branding examples/branding.json --output /tmp/s4pdf --pdf
ls /tmp/s4pdf/

Expect: scorecard.pdf present if Pandoc/TeX Live is installed; otherwise a PDF render failed (continuing — markdown/HTML are still written) message on stderr and a successful exit (the markdown/HTML are still written either way — PDF is opt-in and best-effort, same stance as G3).

B4. CLI help / required-flag guards

node dist/index.js --help
node dist/index.js --output /tmp/x          # missing --snapshot
node dist/index.js --snapshot examples/snapshot.json --output /tmp/x   # missing clientName

Expect: usage text on --help; clear error: --snapshot <file> is required and error: branding.clientName is required … messages (exit 1) on the other two.

Pass criteria

  • Part A — builds, typechecks; 24 tests pass
  • Part B1 — full-input run renders the branded header, grade B with visible derivation, all four framework rows, the improving-trend callout with real numbers, and a methodology section; HTML is self-contained and branded; model.json (schema v1.0) written
  • Part B2 — missing-dashboard sections are stated honestly, not omitted
  • Part B3 — PDF render attempted; failure (if Pandoc absent) does not fail the run
  • Part B4 — --help and required-flag error messages behave as documented

Sign-off

  • Tester: _  |  Date: _  |  Result: PASS / FAIL / N/A
  • Notes: