Skip to content

Manual Test Runbook — J5: Managed Grafana Dashboards-as-Code

Owner: Sagar  |  Time: ~8 min (Parts A + B offline) · +25 min (optional Part C integration apply)  |  Sandbox: snowops-sandbox-01

Promotes J5 (modules/azure/grafana-dashboards/) from 🟦 Code Complete → 🟩 Shipped. Parts A + B are offline ($0). Part C stands up a real Managed Grafana instance (~$9/mo prorated — a few cents for the test window), deploys the dashboards, confirms panels query the J1 workspace, and destroys.

Prerequisites

  • Sandbox subscription access active (PIM activated if required)
  • az login done; sandbox subscription selected; az extension add -n amg
  • Identity has Contributor + Owner/User Access Administrator on the sandbox sub (RBAC grants)
  • SNOWOPS_SANDBOX_SUBSCRIPTION_ID + SNOWOPS_SANDBOX_TENANT_ID exported
  • Local tooling: terraform >= 1.6, go >= 1.22, az CLI >= 2.50, python3, jq
  • Working directory: repo root

Steps

Part A — terraform fmt + validate + dashboards gate (offline, ~4 min)

  1. Module + example:
terraform -chdir=modules/azure/grafana-dashboards fmt -recursive -check
terraform -chdir=modules/azure/grafana-dashboards init -backend=false -input=false
terraform -chdir=modules/azure/grafana-dashboards validate

terraform -chdir=modules/azure/grafana-dashboards/examples/basic init -backend=false -input=false
terraform -chdir=modules/azure/grafana-dashboards/examples/basic validate

Expected: Success! for both.

  1. Dashboards-as-code offline gate:
./modules/azure/grafana-dashboards/scripts/validate.sh

Expected: All 2 dashboard(s) valid: snowops-reliability-cost.json, snowops-security-posture.json.

  1. Offline Terratest case:
cd tests/terratest
go test -v -timeout 5m ./modules/azure/... -run TestGrafanaDashboardsValidate

Expected: PASS — instance + Monitoring Reader (subscription + workspace scopes) + Grafana Admin/Editor/Viewer RBAC fan-out, offline.

Part B — full Terratest suite (offline, ~5 min)

  1. bash cd tests/terratest && go test -count=1 -timeout 15m ./...

Expected: the full suite green (the new TestGrafanaDashboardsValidate included).

Part C — integration apply + dashboard deploy (sandbox, ~25 min, ~$0)

  1. Apply the example (or fixture) against a real workspace. Point monitoring_reader_scopes at a REAL J1 workspace ARM ID + the sandbox sub:
cd modules/azure/grafana-dashboards/examples/basic
terraform init -input=false
terraform apply -auto-approve \
  -var "subscription_scope=/subscriptions/$SNOWOPS_SANDBOX_SUBSCRIPTION_ID" \
  -var "workspace_id=<REAL J1 workspace ARM ID>" \
  -var 'grafana_admin_object_ids=["<your object id>"]'
NAME=$(terraform output -raw grafana_name)
  1. Deploy the dashboards-as-code and confirm:
cd ../..
./scripts/deploy-dashboards.sh "$NAME" snowops-grafana-example-rg "<REAL J1 workspace ARM ID>"

# Confirm both dashboards landed:
az grafana dashboard list --name "$NAME" --resource-group snowops-grafana-example-rg \
  --query "[].{title:title, uid:uid}" -o table

Expected: SnowOps — Security Posture and SnowOps — Reliability & Cost present. Open the endpoint (terraform output grafana_endpoint), pick the Azure Monitor datasource, and confirm at least one panel returns data from the J1 workspace (e.g. failed sign-ins / activity).

  1. Destroy:
cd examples/basic
terraform destroy -auto-approve \
  -var "subscription_scope=/subscriptions/$SNOWOPS_SANDBOX_SUBSCRIPTION_ID" \
  -var "workspace_id=<REAL J1 workspace ARM ID>" \
  -var 'grafana_admin_object_ids=["<your object id>"]'

Expected: clean destroy — the instance + every role assignment removed. The J1 workspace (owned elsewhere) is untouched.

Pass criteria

  • Part A — module + example validate; dashboards gate passes; TestGrafanaDashboardsValidate passes
  • Part B — full offline suite passes
  • (Part C) instance applies; both dashboards deploy; a panel queries the J1 workspace; destroys clean
  • All test resources removed

Failure mode

Panels return Unauthorized — the instance identity wasn't granted Monitoring Reader on the queried scope. Add the scope to monitoring_reader_scopes. A dashboard renders empty when the queried table isn't flowing into the workspace yet (e.g. SecurityIncident needs J3 onboarded) — the offline gate validates structure, not data presence. Both documented in the module README.

Cost impact

Managed Grafana (Standard) ≈ $9/instance/month base + per-active-user, prorated — a few cents for the Part C window. No per-dashboard cost. Log Analytics query cost is the usual per-GB on the J1 workspace.

Removal path

terraform destroy (Part C step 7) removes the instance and all role assignments J5 created; dashboards are destroyed with the instance. The committed JSON under dashboards/ is the source of truth and is untouched. Verified clean in Part C.

Sign-Off

Field Value
Part A (validate + dashboards gate) ☐ PASS
Part B (offline suite) ☐ PASS
Part C (integration apply + deploy) ☐ PASS / ☐ skipped
Tester
Date
Result ☐ PASS