Tabletop Scenario: Distributed Denial of Service (DDoS)¶
A SnowOps incident-response tabletop (
K5) — discussion-based, no real consoles. Facilitation method:facilitation-guide.md. Pack overview:README.md.
Exercises runbook: ../incident/ddos.md — Distributed Denial of Service.
| Field | Value |
|---|---|
| Scenario | Layer-7 application-layer DDoS against the prod Front Door endpoint |
Incident class (K1) |
ddos |
| Duration | 60 min |
| Participants | 4–8 |
| Difficulty | standard (+1 curveball) |
| Last run | <YYYY-MM-DD> |
Objective¶
Validate that the team can identify the attack layer, mitigate it without breaking
legitimate users, and recover using ../incident/ddos.md —
surfacing gaps in the runbook, the WAF rule posture, and auto-scaling assumptions.
Success looks like: the team correctly classifies the attack as Layer 7, applies targeted WAF rate-limiting/blocking, confirms backend health, and tunes out false positives on recovery — citing the runbook at each phase.
Scope & assumptions¶
- In scope: the prod public endpoint behind Azure Front Door + WAF;
backend is the AKS workload (
F3). - Out of scope: internal-only services, the corp VPN.
- Assumptions (state at kickoff):
K2paging is live; Azure DDoS Network Protection is enabled on the relevant VNet; Front Door WAF in Prevention mode; synthetic monitoring (X8) + WAF diagnostic logs flowing to Log Analytics. - Standing rule: discussion only — no real WAF rule changes, no real scaling.
Roles¶
| Role | Played by | Responsibility |
|---|---|---|
| Incident Commander (IC) | <name> |
Declares severity, owns the response. |
| On-call responder | <name> |
Drives ddos.md. |
| Network/Edge SME | <name> |
Front Door, WAF, DDoS Protection. |
| App/Platform SME | <name> |
Backend health, AKS auto-scaling. |
| Communications | <name> |
Status page + client updates. |
| Scribe (optional) | <name> |
Timeline + decisions. |
Injects (timeline)¶
| # | T+ | Inject (read to the room) | Maps to K1 phase |
Facilitator notes (don't read aloud) |
|---|---|---|---|---|
| 1 | 0 min | Synthetic monitoring (X8) goes red: the prod endpoint p95 latency jumps to 8s and 503s climb. Two client tickets: "site is timing out". Front Door WAF alert fires. |
Identification | If asked: DDoS Network Protection shows no L3/4 volumetric spike — bandwidth is normal. So it's Layer 7. |
| 2 | ~8 min | WAF logs show a flood of requests to /api/search?q=... from ~3,000 IPs across many ASNs, all with the same odd User-Agent Mozilla/5.0 (compatible; ZmEu), ~50x normal RPS. |
Containment | Application-layer. The pattern (UA + path) is the handle for a custom WAF rule / rate-limit. |
| 3 | ~16 min | Backend AKS pods are CPU-saturated; HPA scaled to max replicas and is still saturated. DB connection pool exhausted. | Eradication & Investigation | Scaling alone won't win — must filter at the edge. The search endpoint is expensive (no cache). |
| 4 | ~26 min | A custom WAF rate-limit on the abused path + UA block is applied; 503s fall, p95 recovers toward normal. But some real users on the same path now get 429s. | Recovery | False positives — tests tuning, not just blunt blocking. |
| 5 | ~34 min | Traffic normalises; rate-limit relaxed slightly; synthetic + client endpoints green for 20 min. | Post-Incident | Drive Lessons-Learned: edge caching, auto-scale policy, was this a smokescreen? |
| 6 | ~40 min | Curveball: during the DDoS, a separate low-and-slow auth-bruteforce alert fired and was ignored in the noise. Was the DDoS a distraction? | any | Hold for stretch; DDoS as cover for another attack class. |
Discussion Questions (by K1 phase)¶
Push the team into
../incident/ddos.md.
Phase 1 — Identification¶
- The runbook's Initial Triage says determine volumetric (L3/4) vs application-layer (L7). What evidence tells you which (DDoS Protection metrics vs WAF logs), and why does it change the mitigation?
- Is this definitely an attack vs a legitimate traffic surge / bad deploy? How do you confirm?
Phase 2 — Containment (Mitigation)¶
- For L7, the runbook says review WAF logs for attack patterns (user-agents, IPs,
URIs) and apply custom WAF rules to rate-limit/block. What's your rule here
(the
ZmEuUA +/api/searchpath), andRateLimitvs outright block? - For L3/4 the runbook says ensure DDoS Protection Standard is enabled. It is — what does that buy you, and why isn't it enough for this attack?
Phase 3 — Eradication & Investigation¶
- Run the runbook's WAF-log KQL (
ApplicationGatewayFirewallLog/FrontDoorWebApplicationFirewallLog, summarise byclientIp/ruleId) to tune filtering. What does it show, and how do you avoid blocking legit traffic? - The runbook says monitor backend health/CPU continuously. HPA is maxed and still saturated — what does that tell you about edge filtering vs scaling?
Phase 4 — Recovery¶
- Per the runbook: once traffic normalises, slowly adjust rate-limiting to
avoid false positives (real users are getting 429s). How do you tune the
threshold, and how do you confirm via synthetic (
X8) + client endpoints?
Phase 5 — Lessons Learned¶
- The runbook suggests edge caching (CDN/Front Door) to absorb L7 hits and
reviewing auto-scaling policies. Should the expensive
/api/searchbe cached / rate-limited by default? Is HPA max too low? - Curveball: was the DDoS a smokescreen for the auth bruteforce — does this also
run
compromise.md? Which finding becomes a trackedK4action item, with owner and due date?
Evaluation Rubric¶
Score 1–4 per dimension; total /20.
| Dimension | 1 — Ad hoc | 2 — Partial | 3 — Solid | 4 — Strong | Score |
|---|---|---|---|---|---|
| Detection & triage | Assumed outage | Spotted slowness | Identified, classified L7 vs L3/4 | Fast + ruled out volumetric via DDoS metrics | |
| Runbook fluency | Didn't open it | Improvised | Followed ddos.md |
Followed + caught a gap | |
| Containment decisions | Only scaled up | Blunt block of all traffic | Targeted WAF rule on UA+path | Targeted + minimal-collateral rate-limit | |
| Roles & comms | Role confusion | Status page late | Clear IC + status page | Crisp roles + client comms + correct cadence | |
| Recovery & learning | None | Left blunt block on | Tuned false positives | Tuned + caching/HPA fix + smokescreen check + K4 items |
|
| Total | /20 |
After-Action¶
Fill live; expand into
after-action-template.mdand route findings toK4(apps/post-incident-review/).
- Date / facilitator / attendees:
<…> - Rubric total:
<n>/20 - What went well:
<…> - Top gaps found (cite runbook step):
<e.g. ddos.md has no guidance on tuning rate-limits to minimise false positives; no "DDoS as smokescreen" prompt> - Improvement items:
<item — owner — due date — destination>