Skip to content

Manual Test Runbook — F8: K8s Reference Manifests Bundle (ArgoCD app-of-apps)

Owner: Sagar  |  Time: ~5 min (Parts A + B) · +35 min (optional Part C kind bootstrap)  |  Sandbox: kind (local) or snowops-sandbox-01 AKS

Promotes F8 (gitops/) from 🟦 Code Complete → 🟩 Shipped. Parts A + B are 100% offline (no cluster). Part C bootstraps the bundle on a local kind cluster (~$0) and confirms all controllers reach Ready + a synthetic app deploys + the SnowOps Kyverno policies admit/reject correctly.

Prerequisites

  • Local tooling: ruby (for validate.sh), pre-commit
  • (Part C only) kind ≥ 0.23, kubectl ≥ 1.28, helm ≥ 3.14, argocd CLI ≥ 2.11, Docker/Podman running
  • (Part C only) a fork/branch of snowops-automation you can push to (ArgoCD pulls manifests from a git repo, not your working tree) — or use the public repo read-only if the paths are already on main
  • Working directory: repo root

Steps

Part A — offline structural validation (~2 min)

  1. Run the bundle validator (YAML validity + kustomization refs + Application spec shape):
./gitops/validate.sh

Expected: parsed 13 YAML files, 7 ArgoCD Applications then OK: bundle is structurally valid.

  1. Lint the shell + YAML:
shellcheck gitops/bootstrap/bootstrap.sh gitops/validate.sh
ruby -ryaml -e "Dir.glob('gitops/**/*.yaml').each { |f| YAML.load_stream(File.read(f)) }; puts 'all gitops YAML parses'"

Expected: shellcheck clean; all gitops YAML parses.

Part B — pre-commit + policy cross-check (~3 min)

  1. Run pre-commit on the bundle (check-yaml covers gitops/**):
pre-commit run --files \
  gitops/README.md gitops/validate.sh \
  gitops/bootstrap/argocd-namespace.yaml gitops/bootstrap/argocd-values.yaml \
  gitops/bootstrap/root-app.yaml gitops/bootstrap/bootstrap.sh gitops/bootstrap/README.md \
  gitops/apps/kustomization.yaml gitops/apps/cert-manager.yaml gitops/apps/kyverno.yaml \
  gitops/apps/external-secrets.yaml gitops/apps/ingress-nginx.yaml \
  gitops/apps/snowops-policies.yaml gitops/apps/external-secrets-store.yaml \
  gitops/components/external-secrets/serviceaccount.yaml \
  gitops/components/external-secrets/cluster-secret-store.yaml \
  gitops/components/external-secrets/kustomization.yaml \
  docs/runbooks/test/F8.md

Expected: every hook PASS.

  1. Confirm the D4 policies F8 pre-applies still pass kyverno test (F8 references them, doesn't fork them):
policy/kyverno/tests/run-tests.sh

Expected: 5/5 suites pass.

Part C — kind cluster bootstrap (optional, ~35 min, ~$0)

kind has no Azure, so two things behave differently than on AKS and are expected: (a) the ingress-nginx internal-LB Service stays <pending> (no Azure LB to fulfil it), and (b) the ESO ClusterSecretStore is SecretSyncedError until you supply a real AKV + Workload Identity. The kind test validates the controllers, the policy floor, and the sync graph — not the Azure-bound data path (that's the F3 + F5 sandbox runbooks).

  1. Create a kind cluster:
kind create cluster --name snowops-f8
kubectl cluster-info --context kind-snowops-f8
  1. Push the bundle to your fork + repoint the Applications:
# In gitops/bootstrap/root-app.yaml, gitops/apps/snowops-policies.yaml,
# gitops/apps/external-secrets-store.yaml: replace SNOWOPS_ORG + targetRevision
# with your fork + branch, commit, push.
  1. Bootstrap ArgoCD + apply the root:
./gitops/bootstrap/bootstrap.sh --apply-root

  • ArgoCD pods Ready (kubectl -n argocd get pods).

  • Watch the app-of-apps converge:

kubectl -n argocd port-forward svc/argocd-server 8080:443 &
admin_pw=$(kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath='{.data.password}' | base64 -d)
argocd login localhost:8080 --username admin --password "$admin_pw" --insecure --grpc-web
argocd app get snowops-platform
argocd app list
  • snowops-platform (root) shows 6 child Applications.
  • cert-manager, kyverno, external-secrets (wave 0) reach Synced / Healthy.
  • ingress-nginx (wave 1) Synced; controller pod Ready (Service <pending> is OK on kind).
  • snowops-policies (wave 2) Synced; kubectl get clusterpolicy lists all 5 D4 policies.

  • external-secrets-store (wave 2) Synced; kubectl get clustersecretstore snowops-akv exists (status error on kind is expected — no real AKV).

  • Synthetic-app admission test (proves the Kyverno floor is live):

# A compliant pod is admitted:
kubectl create namespace f8-probe
kubectl -n f8-probe run good --image=nginx:1.27-alpine \
  --labels="app.kubernetes.io/name=good,app.kubernetes.io/version=1,app.kubernetes.io/part-of=f8,snowops.io/owner=sagar" \
  --restart=Never --dry-run=server
# A :latest-tagged pod is REJECTED by disallow-latest-tag:
kubectl -n f8-probe run bad --image=nginx:latest --restart=Never --dry-run=server || echo "rejected as expected"
  • compliant pod passes server-side admission.
  • :latest pod is rejected with a Kyverno disallow-latest-tag message.

Pass criteria

  • Part A — validate.sh reports the bundle structurally valid; shellcheck clean
  • Part B — pre-commit all green; kyverno test 5/5
  • Part C — ArgoCD bootstraps; snowops-platform renders 6 child apps
  • Part C — cert-manager / kyverno / external-secrets / ingress-nginx controllers Ready
  • Part C — all 5 D4 ClusterPolicies present; ClusterSecretStore snowops-akv created
  • Part C — compliant pod admitted; :latest pod rejected by Kyverno

Teardown

kind delete cluster --name snowops-f8
# or on AKS:
kubectl delete -f gitops/bootstrap/root-app.yaml
helm uninstall argocd -n argocd && kubectl delete namespace argocd

Sign-off

  • Tester: _  |  Date: _  |  Result: PASS / FAIL / N/A
  • Notes: