Skip to content

Manual Test Runbook — F2: Network Hub

Owner: Sagar  |  Time: ~45 min  |  Sandbox: snowops-sandbox-01

Promotes F2 (modules/azure/network-hub/) from 🟦 Code Complete → 🟩 Shipped. Includes one part that costs ~$5 (Azure Firewall hourly). Skip Part C / D if not iterating on the firewall path.

Prerequisites

  • Sandbox subscription access active (PIM activated if required)
  • az login done; az account show confirms the sandbox subscription is selected
  • Identity has Network Contributor on the sandbox subscription
  • Local tooling: terraform >= 1.6, go >= 1.22, az CLI >= 2.50
  • SNOWOPS_SANDBOX_SUBSCRIPTION_ID and SNOWOPS_SANDBOX_TENANT_ID env vars set
  • Working directory: repo root
  • Azure Firewall quota: at least 1 firewall available in the region (az network firewall list-skus confirms availability; sandbox subs ship with default quota)

Steps

Part A — terraform fmt + validate (offline, ~1 min)

  1. Confirm formatting + structural validity of the module on its own:
terraform -chdir=modules/azure/network-hub fmt -check
terraform -chdir=modules/azure/network-hub init -backend=false -input=false
terraform -chdir=modules/azure/network-hub validate

Expected: Success! The configuration is valid.

  1. Confirm the offline Terratest validate suite still passes for the whole azure/ tree (catches accidental fixture drift):
cd tests/terratest
go build ./...
go vet ./...
go test -v -timeout 5m ./modules/azure/... -run 'TestNetworkHubValidate|TestF2NetworkContractConformance|TestContractsRejectBadLiterals'

Expected: 3 top-level tests pass; the network-empty-private-subnets sub-test under TestContractsRejectBadLiterals is the F2-relevant negative case.

Part B — full Terratest suite (offline, ~3 min)

  1. Run the entire offline suite to make sure F2 hasn't regressed F1, F6, or X1:
cd tests/terratest
go test -v -timeout 10m ./...

Expected: ~9 top-level tests pass (TestNoopHarness, TestBaselineValidate, TestStateBackendValidate, TestSandboxValidate, TestF1ContractConformance, TestF6ObjectStoreContractConformance, TestF2NetworkContractConformance, TestNetworkHubValidate, TestContractsRejectBadLiterals with 5 sub-tests).

Part C — integration test (real Azure apply + destroy, ~35 min, ~$5)

Skip if iterating on offline changes only. The Azure Firewall takes ~10 min to create and ~10 min to destroy.

  1. Export sandbox env vars (same as X2 / F1 / F6):
export SNOWOPS_SANDBOX_SUBSCRIPTION_ID="<sandbox-subscription-guid>"
export SNOWOPS_SANDBOX_TENANT_ID="<sandbox-tenant-guid>"
  1. Run the F2 integration test (build tag integration):
cd tests/terratest
go test -v -tags integration -timeout 60m ./modules/azure/... -run TestNetworkHubModule
  1. Watch for these key milestones in the output:
  2. Plan: ~30 to add, 0 to change, 0 to destroy. — sanity check on plan size (hub vNet + 3 hub subnets + 2 spoke vNets + 4 spoke subnets + 5 NSGs + 5 NSG associations + 2 spoke route tables + 2 routes + 4 route-table associations + 2 peerings + firewall + firewall policy + public IP + 2 private DNS zones + 6 vNet links + RG = ~38 resources).
  3. azurerm_firewall.this[0]: Still creating... — expect this for ~10 min; the firewall is the slow path.
  4. All output assertions PASS.
  5. Destroy complete! — clean teardown.

Part D — manual spot-check (optional, ~5 min)

Run while Part C is between InitAndApply and Destroy if you want to confirm live routing / peering / DNS state in the portal or via CLI.

  1. List the hub vNet's subnets and confirm AzureFirewallSubnet exists:
az network vnet subnet list \
  --resource-group "snowops-f2-test-<suffix>-rg" \
  --vnet-name "snowops-f2-test-<suffix>-hub-vnet" \
  --query "[].{name:name, prefix:addressPrefix, nsg:networkSecurityGroup.id}" \
  --output table

Expected: 3 rows (AzureFirewallSubnet, GatewaySubnet, management). Only management has a non-null nsg — Azure forbids NSG attachment to AzureFirewallSubnet and GatewaySubnet.

  1. Confirm bidirectional peerings on the hub:
az network vnet peering list \
  --resource-group "snowops-f2-test-<suffix>-rg" \
  --vnet-name "snowops-f2-test-<suffix>-hub-vnet" \
  --query "[].{name:name, state:peeringState, remote:remoteVirtualNetwork.id}" \
  --output table

Expected: 2 rows (peer-hub-to-apps, peer-hub-to-data), both Connected.

  1. Confirm the spoke route table forces 0.0.0.0/0 to the firewall private IP:
az network route-table route list \
  --resource-group "snowops-f2-test-<suffix>-rg" \
  --route-table-name "rt-snowops-f2-test-<suffix>-apps-vnet-to-fw" \
  --query "[].{name:name, prefix:addressPrefix, nextHop:nextHopType, nextHopIp:nextHopIpAddress}" \
  --output table

Expected: one row, addressPrefix = 0.0.0.0/0, nextHopType = VirtualAppliance, nextHopIpAddress matches the firewall's private IP from the Terraform output.

  1. Confirm Private DNS zone vNet links exist for both spokes:

    az network private-dns link vnet list \
  --resource-group "snowops-f2-test-<suffix>-rg" \
  --zone-name "privatelink.blob.core.windows.net" \
  --query "[].{name:name, vnet:virtualNetwork.id, regState:virtualNetworkLinkState}" \
  --output table

    Expected: 3 rows (link-hub-*, link-apps-*, link-data-*), all Completed.

Part E — synthetic egress probe (optional, ~5 min)

This is the §4.F2 stated test ("synthetic ping from spoke to allowed endpoint"). Skip if Parts A–D already gave you confidence. Requires a temp VM in a spoke and default_allow_egress = true (or a manual firewall rule permitting outbound 53/UDP).

  1. After Part C apply but before destroy, edit tests/terratest/fixtures/network-hub/main.tf locally to set firewall.default_allow_egress = true, then terraform apply in the fixture directory to push the temporary allow-all rule.

  2. Deploy a tiny VM into apps/workload:

    az vm create --resource-group "snowops-f2-test-<suffix>-rg" \
  --name "f2-probe-vm" \
  --image "Ubuntu2204" \
  --vnet-name "snowops-f2-test-<suffix>-apps-vnet" \
  --subnet "workload" \
  --public-ip-address "" \
  --admin-username "snowops" \
  --generate-ssh-keys \
  --size "Standard_B1ls"

  3. az vm run-command invoke to ping 8.8.8.8 and confirm egress:

    az vm run-command invoke \
  --resource-group "snowops-f2-test-<suffix>-rg" \
  --name "f2-probe-vm" \
  --command-id "RunShellScript" \
  --scripts "curl -s --max-time 5 https://api.ipify.org && echo"

    Expected: the returned public IP matches one of firewall_public_ips from the Terraform output. That is the proof that spoke traffic egresses via the firewall.

  4. Delete the probe VM, then terraform destroy will tear down everything else.

Pass criteria

  • Part A — terraform validate passes for the module
  • Part B — full offline Terratest suite passes (9 top-level tests)
  • Part C — TestNetworkHubModule integration test passes end-to-end
  • Hub vNet has 3 subnets including AzureFirewallSubnet + GatewaySubnet
  • Both spokes peered to the hub, both Connected
  • Each spoke has a route table with 0.0.0.0/0 → VirtualAppliance(<fw-ip>)
  • Private DNS zones linked to hub + both spokes
  • Azure Firewall reachable; private IP non-empty; public IP non-empty
  • All Destroy calls complete without error
  • No orphaned resource groups remain (verify with az group list -o table)
  • All test resources tagged ephemeral = true (X7 cleanup safety net)
  • (Part E only) synthetic egress probe returns the firewall public IP

Teardown

The integration test runs terraform destroy automatically. If a failure mid-run orphans resources, clean up manually:

# RG holds everything F2 creates
az group delete --name "snowops-f2-test-<suffix>-rg" --yes --no-wait

Azure Firewall destroy takes 10+ minutes — let it finish before re-running the test, otherwise the next apply may collide on the public IP allocation.

Sign-off

  • Tester: _  |  Date: _  |  Result: PASS / FAIL / N/A
  • Notes: