Skip to content

Manual Test Runbook — M2: Customer-Managed Keys (HSM-backed, auto-rotation)

Owner: Sagar  |  Time: ~10 min (Parts A + B) · +10 min (optional Part C integration apply) · +15 min (optional Part D rotation drill)  |  Sandbox: snowops-sandbox-01

Promotes M2 (modules/azure/cmk/) from 🟦 Code Complete → 🟩 Shipped. Part C cost ~$1 prorated (a Premium vault + one HSM key). M2 has a build-tagged integration test (Part C); a real key ROTATION is time-delayed, so the rotate-now drill is the manual Part D.

Prerequisites

  • Sandbox subscription access active (PIM activated if required)
  • az login done; sandbox subscription selected
  • Identity has Owner OR (Contributor + User Access Administrator) on the sandbox sub — UAA is needed because the fixture grants the deployer Crypto Officer on the vault it creates
  • SNOWOPS_SANDBOX_SUBSCRIPTION_ID + SNOWOPS_SANDBOX_TENANT_ID exported
  • Local tooling: terraform >= 1.6, go >= 1.22, az CLI >= 2.50
  • Working directory: repo root

Steps

Part A — terraform fmt + validate (offline, ~3 min)

  1. Module + example:
terraform -chdir=modules/azure/cmk fmt -recursive -check
terraform -chdir=modules/azure/cmk init -backend=false -input=false
terraform -chdir=modules/azure/cmk validate

terraform -chdir=modules/azure/cmk/examples/basic init -backend=false -input=false
terraform -chdir=modules/azure/cmk/examples/basic validate

Expected: Success! for both.

  1. Offline Terratest case:
cd tests/terratest
go test -v -timeout 5m ./modules/azure/... -run TestCMKValidate

Expected: PASS (exercises the HSM key + rotation_policy + preconditions + Crypto role grants offline).

Part B — full Terratest suite (offline, ~3 min)

  1. bash cd tests/terratest && go test -count=1 -timeout 15m ./...

Expected: 31 top-level tests green.

Part C — integration apply (sandbox, ~10 min, ~$1)

  1. Run the build-tagged integration test — creates a Premium vault, grants the deployer Crypto Officer, provisions an HSM-backed RSA-HSM CMK with auto-rotation, asserts the key + versionless ID + rotation policy, then destroys:
cd tests/terratest
go test -v -tags integration -timeout 30m ./modules/azure/... -run TestCMKModule

Expected: PASS.

Known flake: the data-plane key create can return 403 Forbidden if the deployer's Crypto Officer grant has not propagated yet. The fixture orders the grant before the key via depends_on, but RBAC propagation is eventual. If a run 403s on azurerm_key_vault_key.cmk, re-run — the second apply picks up the propagated grant.

Part D — rotate-now drill (optional, ~15 min)

Proves the catalog criterion: "rotation event triggers; new key version used." The policy auto-rotates ~30d before a 365-day expiry, so force a rotation now.

  1. Apply the cmk fixture to stand up a vault + key you can rotate by hand (the fixture is self-contained — it creates the Premium vault + grants the deployer Crypto Officer; the example consumes an existing F5 vault instead):
cd tests/terratest/fixtures/cmk
terraform init -input=false
terraform apply -auto-approve \
  -var "subscription_id=$SNOWOPS_SANDBOX_SUBSCRIPTION_ID" \
  -var "tenant_id=$SNOWOPS_SANDBOX_TENANT_ID" \
  -var "resource_group_name=m2-drill-rg" \
  -var "key_vault_name=snowops-cmk-$RANDOM"
  1. Capture the current version, then force a rotation and confirm a NEW version appears (and that the versionless ID still resolves to the latest):
VAULT=$(terraform output -raw key_versionless_id | sed -E 's#https://([^.]+)\..*#\1#')
BEFORE=$(terraform output -raw key_version)
echo "vault=$VAULT before=$BEFORE"

az keyvault key rotate --vault-name "$VAULT" --name snowops-cmk

az keyvault key show --vault-name "$VAULT" --name snowops-cmk \
  --query "{current:key.kid}" -o json

Expected: the current key ID (key.kid) ends in a DIFFERENT version GUID than $BEFORE — a new version was generated. Consumers wired to the versionless ID transparently move to it.

  1. Confirm the rotation policy is what the module configured:
az keyvault key rotation-policy show --vault-name "$VAULT" --name snowops-cmk \
  --query "{expiry:expiryTime, actions:lifetimeActions}" -o json

Expected: expiryTime = P365D; a rotate lifetime action timed timeBeforeExpiry = P30D.

Pass criteria

  • Part A — module + example validate; TestCMKValidate passes
  • Part B — full offline suite passes (31 top-level)
  • (Part C) TestCMKModule provisions the HSM key + rotation policy and asserts shape, then destroys clean
  • (Part D) a forced rotation produces a new key version; rotation policy shows P365D / P30D
  • All test resources removed; vault purged (provider purges soft-delete on destroy)

Teardown

cd tests/terratest/fixtures/cmk   # if Part D ran
terraform destroy -auto-approve \
  -var "subscription_id=$SNOWOPS_SANDBOX_SUBSCRIPTION_ID" \
  -var "tenant_id=$SNOWOPS_SANDBOX_TENANT_ID" \
  -var "resource_group_name=m2-drill-rg" \
  -var "key_vault_name=<the-name-you-used>"

The fixture provider sets purge_soft_delete_on_destroy = true so the globally-unique vault name + key are reclaimable. Production CMK vaults must NOT set that — a destroyed CMK breaks any consumer still encrypting with it; detach consumers first.

Sign-off

  • Tester: _  |  Date: _  |  Result: PASS / FAIL / N/A
  • Notes: