Skip to content

Manual Test Runbook — I6: Patch Compliance Report

Owner: Sagar  |  Time: ~5 min (mostly offline — I6 is a pure report generator; the only live phase is an optional az export)

Overview

I6 turns one Azure Update Manager assessment export (a set of machines, each carrying the OS updates Update Manager found pending) into a versioned, optionally-branded patch-compliance report — markdown + a self-contained HTML page (+ optional Pandoc PDF): overall patch coverage %, the machines out of compliance, the count of pending Critical/Security updates, and SLA aging.

I6 scans nothing and re-derives nothing from Azure: every figure is a re-presentation of what Update Manager already assessed, with two documented extras — a compliant rule (zero pending Critical/Security) and a configurable SLA aging window, each with its basis shown inline (the same "never fabricate, show your work" stance as S4/L5). Pure adapt → render + jest, mirroring E0/S2/S4/L4/L5.

Part A — Build, typecheck, unit tests (~3 min, offline)

cd apps/patch-compliance
npm ci
npm run typecheck
npm run build
npm test

Expect: 2 suites pass — classification normalization (Critical/Security case-insensitive, everything else → Other), the compliant rule (incl. the never-assessed honesty case), coverage math (incl. the null-coverage empty fleet), pending-by-classification tally, SLA aging + breach detection (oldest publishedAt basis, lastAssessedAt fallback, within-window non-breach, configurable --sla-days, future-date clamp, oldest-first sort, never-fabricate null age), highlights + the always-present methodology section, meta/schema stamping, determinism, and the renderer (section ordering, machine rows, honest empty/no-breach states, HTML self-containment + escaping + accent coloring).

Part B — Offline CLI demo (~2 min)

B1. Generate a report from the bundled fixture (unbranded)

node dist/index.js --assessment examples/assessment.json --output /tmp/i6out --now 2026-06-09T12:00:00.000Z
cat /tmp/i6out/patch-compliance.md

Expect: stdout wrote /tmp/i6out/patch-compliance.md, .../patch-compliance.html, .../model.json (coverage 33.3%, 2 SLA breach(es)). The markdown contains, in order:

  • A header — "Patch Compliance Report", "Prepared by: SnowOps Cloud", "As of: 2026-06-09", subscription 11111111-... (no "Prepared for" line — the run is unbranded).
  • ## Patch coverage: 33.3% with a metric table: 6 total machines, 2 compliant, 3 non-compliant, 1 never assessed, 2 pending Critical, 2 pending Security, 2 pending Other.
  • ## Machines — a 6-row table: web-prod-01 ✅, web-prod-02 ⚠️ (1 Crit, aged 100d (since publish) ⛔), api-stg-01 ⚠️ (1 Sec, 8d, no breach), db-prod-01 ⚠️ (1 Crit, aged 86d (since assessment) ⛔ — no publishedAt, so aged from lastAssessedAt), cache-prod-01 ✅ (only Other updates — they don't gate compliance), new-stg-01 ❓ Never assessed (no lastAssessedAt).
  • ## SLA breaches — 2 rows, oldest first: web-prod-02 (100 days, aged from publishedAt), db-prod-01 (86 days, aged from lastAssessedAt).
  • ## Highlights — coverage line, out-of-compliance count, a never-assessed callout, and the SLA-breach line naming web-prod-02 as the oldest.
  • ## Methodology — always present; states the compliant rule, the coverage math, and the 30-day SLA aging basis.
grep -c http /tmp/i6out/patch-compliance.html   # expect 0 — fully self-contained, no network refs
cat /tmp/i6out/model.json | head -5             # versioned model.json (schema v1.0)

Open the HTML — a self-contained styled page (inline CSS only, no external script/stylesheet refs), a 33.3% coverage tile in red (below the 80% band), the default SnowOps accent #1f6feb on headings/borders, and a SnowOps Cloud text wordmark (unbranded run, no logoUrl).

B2. Branded run + custom SLA window

node dist/index.js --assessment examples/assessment.json --branding examples/branding.json \
  --sla-days 90 --output /tmp/i6branded --now 2026-06-09T12:00:00.000Z
grep -A1 "## SLA breaches" /tmp/i6branded/patch-compliance.md

Expect: the title becomes "June 2026 Patch Compliance Report", a "Prepared for: Contoso Logistics" line appears, the accent color is #0b5fff. With a 90-day SLA, only web-prod-02 (100 days) still breaches — db-prod-01 (86 days) drops below the window. This demonstrates --sla-days driving breach detection.

B3. Honest empty / never-assessed states

echo '{"machines":[]}' > /tmp/empty.json
node dist/index.js --assessment /tmp/empty.json --output /tmp/i6empty
grep -A1 "## Machines" /tmp/i6empty/patch-compliance.md
grep "## Patch coverage" /tmp/i6empty/patch-compliance.md

Expect: coverage — (not a fabricated 0/100), and the Machines section plainly states "No machines were present in the assessment export" — I6 never invents a fleet it wasn't given. (The fixture's new-stg-01 already exercises the never-assessed honesty path in B1.)

B4. Optional PDF (best-effort)

node dist/index.js --assessment examples/assessment.json --output /tmp/i6pdf --pdf
ls /tmp/i6pdf/

Expect: patch-compliance.pdf present if Pandoc/TeX Live is installed; otherwise a PDF render failed (continuing — markdown/HTML are still written) message on stderr and a successful exit — PDF is opt-in and best-effort (same stance as G3/S4).

B5. CLI help / required-flag guards

node dist/index.js --help
node dist/index.js --output /tmp/x                       # missing --assessment
node dist/index.js --assessment examples/assessment.json # missing --output

Expect: usage text on --help; clear error: --assessment <file> is required and error: --output <dir> is required messages (exit 1).

Part C — Live Update Manager export (optional, ~5 min)

I6 reads any JSON in the { machines: [...] } shape, so the only "live" step is producing that JSON from Azure. Update Manager assessment results live under patchassessmentresources in Resource Graph; a query like the following exports the per-machine pending-update inventory:

az graph query -q "
  patchassessmentresources
  | where type =~ 'microsoft.compute/virtualmachines/patchassessmentresults/softwarepatches'
  | project id, name, classification=properties.classifications, publishedAt=properties.publishedDateTime
" -o json > /tmp/raw-assessment.json

Reshape raw-assessment.json into the UpdateManagerAssessment contract (group software-patch rows under their parent machine's pendingUpdates, attach each machine's lastAssessedAt from its patchassessmentresults parent), then:

node dist/index.js --assessment /tmp/live-assessment.json --output /tmp/i6live --client-name "Acme Corp"

Expect: a report whose coverage %, non-compliant machines, and SLA breaches reconcile with what the Azure Portal's Update Manager blade shows for the same subscription. I6 only re-presents — the numbers should match the portal's assessment, not diverge from it.

Pass criteria

  • Part A — builds, typechecks; both suites pass
  • Part B1 — fixture run reports 33.3% coverage, the 6-machine table with correct statuses (incl. cache-prod-01 compliant on Other-only updates and new-stg-01 never-assessed), 2 SLA breaches oldest-first with the correct aging basis each; HTML self-contained (grep -c http returns 0); model.json (schema v1.0) written
  • Part B2 — branding applied; --sla-days 90 drops db-prod-01 from the breach list
  • Part B3 — empty fleet renders — coverage and an honest "no machines" state, not a fabricated number
  • Part B4 — PDF render attempted; failure (if Pandoc absent) does not fail the run
  • Part B5 — --help and required-flag error messages behave as documented
  • Part C (optional) — a live Update Manager export reconciles with the portal

Sign-off

  • Tester: _  |  Date: _  |  Result: PASS / FAIL / N/A
  • Notes: