Tabletop Scenario: Vendor / Supply-Chain Breach¶
A SnowOps incident-response tabletop (
K5) — discussion-based, no real consoles. Facilitation method:facilitation-guide.md. Pack overview:README.md.
Exercises runbook: ../incident/vendor-breach.md — Vendor / Supply-Chain Breach.
| Field | Value |
|---|---|
| Scenario | CI/CD vendor breach → OIDC/SP access into the tenant |
Incident class (K1) |
vendor-breach |
| Duration | 60 min |
| Participants | 4–8 |
| Difficulty | standard (+1 curveball) |
| Last run | <YYYY-MM-DD> |
Objective¶
Validate that the team can scope, contain, and recover from a third-party CI/CD
vendor breach that has a credentialed path into the Azure tenant, using
../incident/vendor-breach.md — surfacing gaps in
the runbook, the vendor-integration inventory, and least-privilege posture.
Success looks like: the team quickly maps the vendor's integration (SP / OIDC federation), revokes access, checks for abuse in the logs, and re-establishes trust only after the vendor confirms remediation — citing the runbook at each phase.
Scope & assumptions¶
- In scope: the CI/CD vendor
<BuildCloud>integrated for deployments; it holds an OIDC federated credential on the SPsp-cicd-deploy(Contributor on the prod sub) and an inbound webhook. - Out of scope: unrelated SaaS vendors with no tenant credentials.
- Assumptions (state at kickoff):
K2paging is live; Sentinel (J3) +SigninLogs/AuditLogsflowing; the stale-secret inventory script (H5,STALE.md) is available; OIDC federation is configured for the vendor. - Standing rule: discussion only — no real credential revocation, no real vendor contact.
Roles¶
| Role | Played by | Responsibility |
|---|---|---|
| Incident Commander (IC) | <name> |
Declares severity, owns the response. |
| On-call responder | <name> |
Drives vendor-breach.md. |
| Identity/CICD SME | <name> |
SPs, OIDC federation, RBAC, webhooks. |
| Vendor liaison | <name> |
Owns vendor comms + remediation confirmation. |
| Communications | <name> |
Internal + client updates. |
| Scribe (optional) | <name> |
Timeline + decisions. |
Injects (timeline)¶
| # | T+ | Inject (read to the room) | Maps to K1 phase |
Facilitator notes (don't read aloud) |
|---|---|---|---|---|
| 1 | 0 min | Public disclosure: CI/CD vendor <BuildCloud> announces a breach of their build infrastructure; customer OIDC trust configs and pipeline secrets may be exposed. SnowOps uses them for prod deploys. |
Identification | If asked: the team must determine what <BuildCloud> actually holds — they may not know off-hand (that's a finding). |
| 2 | ~8 min | Inventory shows <BuildCloud> authenticates via an OIDC federated credential on sp-cicd-deploy, which has Contributor on the prod subscription (over-privileged). There's also an inbound deploy webhook. |
Containment | Federation trust + over-broad RBAC. Containment = remove the federated credential trust + block the webhook. |
| 3 | ~16 min | SigninLogs filtered to the vendor's AppId show 3 token issuances in the last 14 days from an unfamiliar IP range, two of them outside the normal pipeline schedule (02:30 weekend). |
Eradication & Investigation | Possible abuse. Pull them to the runbook's SigninLogs/AuditLogs queries + resource-creation hunt. |
| 4 | ~26 min | AuditLogs/AzureActivity show one of those sessions created a new role assignment (User Access Administrator) on a resource group during the off-hours window. |
Recovery | Persistence via RBAC. Recovery must revert this before re-enabling anything. |
| 5 | ~34 min | Federation trust removed, webhook blocked, rogue role assignment reverted, vendor confirms remediation. New short-lived OIDC trust + least-privilege RBAC re-established. No abuse for 30 min. | Post-Incident | Drive Lessons-Learned: least privilege, long-lived secrets → OIDC. |
| 6 | ~40 min | Curveball: deploys are now blocked (you revoked the CI path). A prod hotfix is waiting. How do you ship safely without re-opening the breach? | any | Hold for stretch; tests the recovery/comms tradeoff. |
Discussion Questions (by K1 phase)¶
Push the team into
../incident/vendor-breach.md.
Phase 1 — Identification¶
- The runbook's Initial Triage says determine the scope of the vendor's integration — which credentials/tokens/data were shared. Can you answer that now? Where's the vendor-integration inventory? (If you can't, that's the first finding.)
- How do you confirm this affects your tenant vs just being industry news?
Phase 2 — Containment¶
- Walk the Containment steps: revoke the vendor's active API key / SP credential, and for federated identity (OIDC), remove the federation trust to block token issuance; then block inbound/outbound webhooks/API calls via NSG/WAF. What's the order, and what breaks when you do it (deploys)?
- Would you also suspend
sp-cicd-deployentirely, or just cut the federation?
Phase 3 — Eradication & Investigation¶
- Run the runbook's
SigninLogsKQL filtered to the vendor'sAppIdover 14 days. What are you looking for (unfamiliar IPs, off-schedule issuance)? ThenAuditLogs— what unexpected actions (role assignments, resource creation, data access) do you hunt for? (You'd find the rogue RBAC change.) - The runbook says review the
STALE.mdinventory (viaH5) for long-lived secrets the vendor might hold. What does that surface here?
Phase 4 — Recovery¶
- Per the runbook: re-establish credentials / federated trust only after the vendor confirms remediation, re-enable network/webhooks, monitor for 7 days. What do you revert first (the rogue role assignment), and how do you verify the vendor's fix before trusting them again?
- Curveball: a prod hotfix is blocked — what's the safe interim deploy path that doesn't re-open the breach?
Phase 5 — Lessons Learned¶
- The runbook says enforce least privilege for third-party integrations (Reader
vs Contributor) and migrate long-lived secrets → OIDC federated credentials.
sp-cicd-deployhad Contributor on prod — what should it have? Who owns the RBAC reduction? - Which finding becomes a tracked
K4action item (e.g. build/maintain a vendor blast-radius inventory; scope CI SP to specific RGs), with owner and due date?
Evaluation Rubric¶
Score 1–4 per dimension; total /20.
| Dimension | 1 — Ad hoc | 2 — Partial | 3 — Solid | 4 — Strong | Score |
|---|---|---|---|---|---|
| Detection & triage | Treated as news | Acknowledged late | Scoped the integration | Fast + knew the blast radius immediately | |
| Runbook fluency | Didn't open it | Improvised | Followed vendor-breach.md |
Followed + caught a gap (no inventory) | |
| Containment decisions | Left trust active | Blocked webhook only | Removed OIDC trust + webhook | All that + suspended SP, scoped impact | |
| Roles & comms | Role confusion | Vendor contacted late | Clear IC + vendor liaison | Crisp roles + client comms + vendor remediation gate | |
| Recovery & learning | None | Re-trusted blind | Reverted RBAC + re-established least-priv | Validated vendor fix + safe interim deploy + K4 items |
|
| Total | /20 |
After-Action¶
Fill live; expand into
after-action-template.mdand route findings toK4(apps/post-incident-review/).
- Date / facilitator / attendees:
<…> - Rubric total:
<n>/20 - What went well:
<…> - Top gaps found (cite runbook step):
<e.g. vendor-breach.md assumes you have a vendor-integration inventory; SnowOps doesn't maintain one; CI SP over-privileged> - Improvement items:
<item — owner — due date — destination>