Skip to content

Manual Test Runbook — D2: PR-Blocking Quality Gates

Owner: Sagar  |  Time: ~25 min  |  Sandbox: none (GitHub-only)

Prerequisites

  • snowops-automation pushed to GitHub with branch protection on main.
  • Repository Settings → Code security has "GitHub Advanced Security" and "Code scanning" enabled so SARIF uploads land on the PR diff (skip if running against a personal repo — checks still report pass/fail).
  • Required status checks configured on main to include every job below:
    • quality-gates / pre-commit (fmt + validate + tflint)
    • quality-gates / checkov (terraform)
    • quality-gates / tfsec (terraform)
    • quality-gates / gitleaks (secrets)
    • quality-gates / trivy (filesystem)
  • Local git configured with push access to the repo.
  • D1 hooks installed locally (pre-commit install --install-hooks) so planted violations have to be force-bypassed past the local gate.

Steps

Part A — clean PR baseline (~5 min)

  1. Branch off main, make a benign edit, push:
git checkout -b test/d2-clean
echo "" >> docs/README.md
git commit -am "chore: D2 baseline smoke test"
git push -u origin test/d2-clean

  1. Open the PR. Wait ~3–5 min for quality-gates to complete.

  2. Expected: every gate job reports ✅. PR is mergeable.

  3. Close the PR without merging; delete the branch.

Part B — planted bad PR (~12 min)

  1. Branch and plant a multi-violation Terraform file:
git checkout main && git pull
git checkout -b test/d2-bad
mkdir -p tests/scratch && cat > tests/scratch/bad_module.tf <<'EOF'
resource "azurerm_storage_account" "leaky" {
  name                            = "snowopsd2demo"
  resource_group_name             = "rg-demo"
  location                        = "eastus"
  account_tier                    = "Standard"
  account_replication_type        = "LRS"
  allow_nested_items_to_be_public = true
  public_network_access_enabled   = true
  min_tls_version                 = "TLS1_0"
}

# NOTE: do not commit real keys — these are gitleaks-rule fixtures.
variable "leaked_key" {
  type    = string
  default = "AKIAIOSFODNN7EXAMPLEAAAA"
}
EOF
  1. Commit-bypass the local hooks so we actually exercise the remote gate (omit --no-verify if you want to confirm D1 also blocks):
git add tests/scratch/bad_module.tf
git commit --no-verify -m "test(d2): planted gate violations"
git push -u origin test/d2-bad

  1. Open the PR. Wait for the gate run.

  2. Expected on the PR Checks tab:

  3. pre-commit (fmt + validate + tflint) — FAIL (terraform_validate may pass if isolated; tflint flags style).
  4. checkov (terraform) — FAIL, HIGH/CRITICAL findings on the storage account (CKV_AZURE_*).
  5. tfsec (terraform) — FAIL, HIGH findings on TLS + public access.
  6. gitleaks (secrets) — FAIL with the fixture AWS access key.

  7. trivy (filesystem) — FAIL on the same secret + misconfigurations.

  8. Expected on the PR diff: SARIF-backed comments on the offending lines (where Code Scanning is enabled).

  9. Close the PR without merging; delete the remote branch.

Part C — bypass attempt (~3 min)

  1. Confirm main is protected against direct push and against merge with failing checks:

    git checkout main
echo "" >> docs/README.md
git commit -am "chore: should be rejected"
git push origin main

    Expected: push rejected with "protected branch" error from GitHub.

  2. Reset your local branch:

    git reset --hard origin/main

Part D — required-check enforcement (~5 min)

  1. Repeat Part B but only fix the gitleaks finding (leave the storage misconfig). Open a PR.

  2. Expected: gitleaks passes, but checkov + tfsec still fail. The PR must remain non-mergeable. The "Merge" button on GitHub is greyed out until every required check is green.

Pass criteria

  • Clean PR (Part A) — all 5 gate jobs pass; PR mergeable.
  • Bad PR (Part B) — every relevant gate fails on the right rule(s).
  • SARIF findings appear inline on the PR diff (where Code Scanning is on).
  • Direct push to main rejected by branch protection (Part C).
  • Mixed-failure PR (Part D) is non-mergeable until every check is green.
  • No false positives surfaced against unrelated repo content.

Teardown

D2 has no cloud side-effects. Local + remote cleanup:

git checkout main
git branch -D test/d2-clean test/d2-bad 2>/dev/null || true
git push origin --delete test/d2-clean test/d2-bad 2>/dev/null || true
rm -rf tests/scratch

Sign-off

  • Tester: _  |  Date: _  |  Result: PASS / FAIL / N/A
  • Notes: