Manual Test Runbook — C2: container-build-sign reusable workflow

Owner: Sagar | Time: ~5 min (Parts A + B offline) · ~40 min (Parts C + D + E in sandbox) | Sandbox: X1

Promotes C2 (.github/workflows/container-build-sign.yml) from 🟦 Code Complete → 🟩 Shipped. Parts C–E require a sandbox ACR (F4) and a sandbox Key Vault (F5) holding a self-signed Notation signing cert. Sandbox cost: < $1 (ACR pulls + ephemeral repository).

Purpose

Validate that the C2 reusable workflow correctly:

Parses cleanly and exposes the documented input/secret/output contract.
Builds + pushes a container image to ACR by tag and by digest.
Signs the pushed image with Notation v2 using an AKV-backed signing key.
Emits an SPDX SBOM artifact via Syft.
Fails the build when Grype finds a CVE at or above the configured cutoff.
Succeeds and emits a verifiable signature on a clean image.

Prerequisites

Sandbox subscription access (PIM activated).
Sandbox F4 ACR provisioned (e.g. snowopssandboxacr01). Login server in env var SANDBOX_ACR_LOGIN_SERVER.
Sandbox F5 Key Vault provisioned. Vault name in env var SANDBOX_KV_NAME.
A Notation-compatible signing certificate present in the AKV. Quick path:

az keyvault certificate create \
  --vault-name "${SANDBOX_KV_NAME}" \
  --name "notation-signing-c2" \
  --policy "$(az keyvault certificate get-default-policy | \
      jq '.x509CertificateProperties.subject="CN=snowops-c2-test" |
          .keyProperties.exportable=false |
          .keyProperties.keyType="RSA" |
          .keyProperties.keySize=3072 |
          .x509CertificateProperties.keyUsage=["digitalSignature"] |
          .x509CertificateProperties.ekus=["1.3.6.1.5.5.7.3.3"]')"
AKV_CERT_URL="$(az keyvault certificate show \
  --vault-name "${SANDBOX_KV_NAME}" \
  --name notation-signing-c2 \
  --query id -o tsv)"
echo "${AKV_CERT_URL}"   # https://<vault>.vault.azure.net/certificates/notation-signing-c2/<version>

GitHub Actions repository secrets AZURE_CLIENT_ID, AZURE_TENANT_ID, AZURE_SUBSCRIPTION_ID set on the test repo.
The OIDC app behind those secrets holds:
AcrPush on the sandbox ACR
Key Vault Crypto User AND Key Vault Certificate User on the sandbox AKV
Federated credential subject repo:<org>/<repo>:ref:refs/heads/main (or :pull_request if running Part C from a PR branch).
Local tooling: gh, docker, notation >= 1.2.0, notation-azure-kv >= 1.2.0 (Part E only).
Working directory: repo root.

Steps

Part A — YAML + workflow lint (offline, ~2 min)

Confirm the reusable workflow + caller template parse:

ruby -ryaml -e "YAML.load_file('.github/workflows/container-build-sign.yml')"
ruby -ryaml -e "YAML.load_file('templates/client-repo/.github/workflows/container-build-sign.yml')"

Both commands exit 0 with no output.
If actionlint is installed locally, run it for a deeper check:

actionlint .github/workflows/container-build-sign.yml \
           templates/client-repo/.github/workflows/container-build-sign.yml

No error or warning. (actionlint is optional — skip if not on PATH.)
Confirm pre-commit gates still pass on the touched files:

pre-commit run --files \
  .github/workflows/container-build-sign.yml \
  templates/client-repo/.github/workflows/container-build-sign.yml \
  pipelines/README.md \
  tests/pipeline-integration/container-build-sign/README.md \
  tests/pipeline-integration/container-build-sign/Dockerfile.clean \
  tests/pipeline-integration/container-build-sign/Dockerfile.vulnerable \
  docs/runbooks/test/C2.md

All hooks PASS.

Part B — contract self-check (offline, ~3 min)

Re-read the C2 contract section and confirm:
Every documented input is present in .github/workflows/container-build-sign.yml's on.workflow_call.inputs block.
image_digest + image_reference outputs are declared at the workflow level and wired to the build-sign-scan job outputs.
No secret is referenced in the workflow body via ${{ env.X }} other than the three OIDC inputs (i.e. nothing leaks via env).
notation sign uses --key snowops-signing — the key alias registered one step earlier.

Part C — clean image happy path (sandbox, ~20 min, ~$0.10)

Create a throwaway repo (or a test branch on this one) that consumes the workflow.

# On a feature branch in any test repo:
mkdir -p .github/workflows app
cp <repo-root>/tests/pipeline-integration/container-build-sign/Dockerfile.clean app/Dockerfile
cat > .github/workflows/build.yml <<'EOF'
name: build
on:
  push:
    branches: [main]
  workflow_dispatch:
permissions:
  id-token: write
  contents: read
jobs:
  build:
    uses: <org>/snowops-automation/.github/workflows/container-build-sign.yml@main
    with:
      acr_login_server: "${{ vars.SANDBOX_ACR_LOGIN_SERVER }}"
      image_name: "tests/c2/clean"
      notation_certificate_key_id: "${{ vars.SANDBOX_AKV_CERT_URL }}"
      dockerfile_path: "Dockerfile"
      build_context: "app"
      grype_severity_cutoff: "critical"
    secrets:
      AZURE_CLIENT_ID: ${{ secrets.AZURE_CLIENT_ID }}
      AZURE_TENANT_ID: ${{ secrets.AZURE_TENANT_ID }}
      AZURE_SUBSCRIPTION_ID: ${{ secrets.AZURE_SUBSCRIPTION_ID }}
EOF

Trigger the workflow:

gh workflow run build.yml
gh run watch

Compute tags step emits one line: <server>/tests/c2/clean:<sha>.
Build and push by digest succeeds. The step's digest output starts with sha256:.
Sign image succeeds; notation inspect step output shows one signature with the cert subject CN=snowops-c2-test.
Generate SBOM succeeds. An artifact named sbom-<run_id>.spdx.json is attached to the run.
Scan image with Grype succeeds. Either no findings, or findings only below critical. The job is green.
An artifact named grype-<run_id>.sarif is attached.
The job summary table renders cleanly with digest + cutoff + fail-on-findings.
Verify the ACR side:

az acr repository show-tags --name "${SANDBOX_ACR_NAME}" \
  --repository tests/c2/clean -o tsv
az acr manifest list-referrers --registry "${SANDBOX_ACR_NAME}" \
  --name "tests/c2/clean@<digest>" -o table

The pushed tag appears.
At least one referrer with artifactType=application/vnd.cncf.notary.signature appears.

Part D — planted-CVE failure path (sandbox, ~15 min)

Repoint the consumer workflow at the vulnerable Dockerfile:

cp <repo-root>/tests/pipeline-integration/container-build-sign/Dockerfile.vulnerable app/Dockerfile
# In .github/workflows/build.yml, swap image_name: → "tests/c2/vulnerable".
git commit -am "test: c2 failure-mode probe"
git push

Re-trigger and watch:

gh workflow run build.yml
gh run watch

Build and push by digest succeeds (vulnerable image is still a valid OCI artifact).
Sign image succeeds — signing is independent of vulnerability state.
Generate SBOM succeeds; SBOM artifact still uploaded.
Scan image with Grype fails, with at least one finding at critical.
The Upload Grype SARIF as artifact step still runs (because if: always() is set) and the SARIF artifact is attached for forensic review.
Overall job status: failed.
Optional negative-of-negative: bump grype_severity_cutoff to high on the same vulnerable image — confirm the job still fails (Buster has criticals AND highs).
Optional brownfield-mode check: set fail_on_scan_findings: false and re-run — job succeeds, artifacts still attached.

Part E — signature round-trip (sandbox, ~10 min)

From a workstation with access to the sandbox ACR (PIM activated), pull the signed clean image's digest from Part C and verify it:

az acr login --name "${SANDBOX_ACR_NAME}"
notation cert add --type ca --store snowops \
  "$(az keyvault certificate download \
      --vault-name "${SANDBOX_KV_NAME}" \
      --name notation-signing-c2 \
      --file /tmp/notation-c2.crt --encoding PEM && echo /tmp/notation-c2.crt)"
cat > /tmp/trustpolicy.json <<EOF
{
  "version": "1.0",
  "trustPolicies": [
    {
      "name": "snowops-sandbox",
      "registryScopes": ["${SANDBOX_ACR_LOGIN_SERVER}/tests/c2/clean"],
      "signatureVerification": { "level": "strict" },
      "trustStores": ["ca:snowops"],
      "trustedIdentities": ["x509.subject: CN=snowops-c2-test"]
    }
  ]
}
EOF
notation policy import /tmp/trustpolicy.json
notation verify "${SANDBOX_ACR_LOGIN_SERVER}/tests/c2/clean@<digest-from-part-c>"

notation verify exits 0 and prints Successfully verified signature.

Negative case — verify against the vulnerable image's digest (also signed). Verification should still succeed because the signature is valid regardless of CVE state — D4's Kyverno rule enforces signed-ness, not vulnerability state. Vulnerability gating belongs to C2 itself + S-series drift detection.
- notation verify exits 0 for the vulnerable image too.

Part F — cleanup

Remove the test repositories from sandbox ACR:

az acr repository delete --name "${SANDBOX_ACR_NAME}" --repository tests/c2/clean --yes
az acr repository delete --name "${SANDBOX_ACR_NAME}" --repository tests/c2/vulnerable --yes

Optional: revoke the throwaway signing cert (it's worthless outside the test, but tidy):

az keyvault certificate delete --vault-name "${SANDBOX_KV_NAME}" --name notation-signing-c2

Pass criteria

Parts A + B pass cleanly (YAML, actionlint if installed, pre-commit, contract self-check).
Part C: clean image is built, pushed by digest, signed, SBOM uploaded, Grype scan green.
Part D: vulnerable image build is failed by Grype at the critical cutoff; SARIF artifact still attached.
Part E: notation verify succeeds against the signed image using the AKV-derived cert in the trust store.
No signing material (private key, cert PFX) ever lands on the runner — confirmed by Register signing key step's log only showing notation key add … snowops-signing (no PEM dump).

Sign-off

Tester: ____ | Date: ____ | Result: PASS / FAIL / N/A
Notes: