Skip to content

Manual Test Runbook — U5: Cost Anomaly Detection

Owner: CO  |  Time: ~6 min (Parts A + B offline) · +10 min (optional Part C live apply)  |  Sandbox: snowops-sandbox-01

Promotes U5 (modules/azure/cost-anomaly-alert/) from 🟦 Code Complete → 🟩 Shipped. Parts A + B are offline ($0). Part C creates a REAL azurerm_cost_anomaly_alert on the sandbox subscription and confirms it exists via az costmanagement / the portal. U5 is the detection complement to U1 budget alerts: U1 = "you crossed a threshold", U5 = "your spend pattern changed unexpectedly". Anomaly alerts are free Cost Management objects (~$0).

Prerequisites

  • Sandbox subscription access active (PIM activated if required)
  • az login done; sandbox subscription selected
  • Identity has Cost Management Contributor (or Contributor) on the subscription
  • SNOWOPS_SANDBOX_SUBSCRIPTION_ID + SNOWOPS_SANDBOX_TENANT_ID exported
  • Local tooling: terraform >= 1.6, go >= 1.22, az CLI >= 2.50, jq
  • Working directory: repo root

Steps

Part A — terraform fmt + validate + go vet (offline, ~3 min)

  1. Module + example formatting and validate:
terraform -chdir=modules/azure/cost-anomaly-alert fmt -recursive -check
terraform -chdir=modules/azure/cost-anomaly-alert init -backend=false -input=false
terraform -chdir=modules/azure/cost-anomaly-alert validate

terraform -chdir=modules/azure/cost-anomaly-alert/examples/basic init -backend=false -input=false
terraform -chdir=modules/azure/cost-anomaly-alert/examples/basic validate

Expected: fmt -check clean (exit 0); Success! for both validate calls. The example uses a synthetic subscription_id/tenant_id, so validate runs with no cloud creds.

Sandbox note: the Terraform registry network is blocked in the CI sandbox, so init (which downloads the azurerm provider) fails there identically for every module — confirm against any existing sibling fixture (e.g. terraform -chdir=tests/terratest/fixtures/budget-alert init) and treat it as environmental, not a U5 defect. fmt -check and go vet run fully offline and ARE the offline gate.

  1. The test compiles (go vet):
cd tests/terratest && go vet ./modules/azure/

Expected: no output, exit 0 — TestCostAnomalyAlertValidate compiles.

  1. Offline Terratest case (where registry access is available):
cd tests/terratest
go test -v -timeout 5m ./modules/azure/... -run TestCostAnomalyAlertValidate

Expected: PASS — exercises the for_each over two alerts, the module-wide vs per-alert sender/scope fallbacks, the message field, and the variable validations + preconditions, offline.

To eyeball a rejection, temporarily set an email_subject longer than 70 chars in the fixture and re-run validate — it must fail with the "email_subject must be 1–70 characters" message. Revert after.

Part B — full Terratest suite (offline, ~3 min)

  1. bash cd tests/terratest && go test -count=1 -timeout 15m ./...

Expected: the full suite green (the new TestCostAnomalyAlertValidate included).

Part C — live apply against the sandbox subscription (sandbox, ~10 min, ~$0)

The point of Part C is to prove a real azurerm_cost_anomaly_alert is created and recognised by Cost Management. There is no per-resource charge — anomaly alerts are free, so this window costs effectively $0.

  1. Apply the example against the sandbox subscription:
cd modules/azure/cost-anomaly-alert/examples/basic
terraform init -input=false
terraform apply -auto-approve \
  -var "subscription_id=$SNOWOPS_SANDBOX_SUBSCRIPTION_ID" \
  -var "tenant_id=$SNOWOPS_SANDBOX_TENANT_ID" \
  -var 'email_addresses=["your.name@example.com"]'

Expected: applies clean. terraform output cost_anomaly_alert_ids shows the ARM ID of the created alert; terraform output email_recipients echoes your recipient list.

  1. Confirm the anomaly alert exists. The azurerm_cost_anomaly_alert is a Cost Management scheduled-action of kind InsightAlert. List it via the REST API (the az costmanagement extension does not expose anomaly alerts directly):
SUB="$SNOWOPS_SANDBOX_SUBSCRIPTION_ID"
az rest --method get \
  --url "https://management.azure.com/subscriptions/$SUB/providers/Microsoft.CostManagement/scheduledActions?api-version=2023-11-01&\$filter=properties/scope eq 'subscriptions/$SUB'" \
  --query "value[?kind=='InsightAlert'].{name:name, displayName:properties.displayName, status:properties.status}" -o jsonc

Expected: the snowops-example-anomaly alert is listed with its displayName and an enabled status.

Portal alternative: Cost Management > Cost alerts > Anomaly alerts (or Cost analysis > … > Anomaly detection settings) shows the created alert and its recipients.

  1. Detection latency note (expected, not a failure): Azure needs roughly 10 days of cost history to model the subscription's normal spend before it can flag anomalies. A freshly created alert (or a new subscription) stays quiet until the model warms up — do not expect a test email on apply. Confirming the alert object exists (step 6) is the Part C pass criterion; actual firing is validated organically once history accrues.

  2. Destroy:

cd modules/azure/cost-anomaly-alert/examples/basic
terraform destroy -auto-approve \
  -var "subscription_id=$SNOWOPS_SANDBOX_SUBSCRIPTION_ID" \
  -var "tenant_id=$SNOWOPS_SANDBOX_TENANT_ID"

Expected: clean destroy — the anomaly alert is removed. Nothing else on the subscription is touched (detection is stateless config; no data to clean up).

Pass criteria

  • Part A — fmt -check clean; module + example validate (or registry-block noted as environmental); go vet ./modules/azure/ clean; TestCostAnomalyAlertValidate passes
  • Part B — full offline suite passes
  • (Part C) apply creates the alert; az rest … scheduledActions (or portal) confirms the InsightAlert; destroy removes it cleanly
  • All test changes reverted

Failure modes

  • email_subject rejected at plan — over 70 chars. Variable validation + precondition, by design (Azure caps the subject at 70).
  • email_addresses rejected at plan — 0 or > 20 entries. Azure requires 1–20 recipients; variable validation + precondition enforce it.
  • message rejected at plan — over 250 chars (Azure limit).
  • No anomaly email after apply — expected for the first ~10 days while Azure builds cost history. Not a U5 defect; documented in the README.
  • init fails downloading the provider in CI — registry network is blocked in the sandbox; environmental, identical for every module. fmt -check + go vet are the offline gate.

Cost impact

The cost-anomaly alert is a free Cost Management object — no consumption charge. The only spend it generates is the notification emails on detection. Part C costs effectively $0. No dollar figure is asserted.

Removal path

terraform destroy (Part C step 8) removes each azurerm_cost_anomaly_alert. Detection is stateless subscription config — no orphaned resources and no data to purge. Verified clean in Part C.

Sign-Off

Field Value
Part A (fmt + validate + go vet + Terratest case) ☐ PASS
Part B (offline suite) ☐ PASS
Part C (live apply + alert confirmed + destroy) ☐ PASS / ☐ skipped
Tester
Date
Result ☐ PASS