Manual Test Runbook — B1: snowops-onboarder GitHub App

Owner: Sagar | Time: ~60 min | Requires: test GitHub org + sandbox Azure tenant + Azure AD app reg from B2

Purpose

Verify that the B1 onboarder, given an A4-shaped webhook payload, provisions a brand-new client repo to SnowOps standards: repo created from the C4 template, branch protection applied, GitHub Environments with reviewers, Azure OIDC variables seeded, and federated identity credentials attached to the Azure AD App Registration.

Prerequisites

Test GitHub org (e.g. snowops-sandbox) where you have admin access.
A snowops-onboarder GitHub App created with permissions:
administration:write, contents:write, environments:write, secrets:write, actions:read.
The App is installed on the test org.
Sandbox Azure tenant (X1 lineage) with an App Registration created via B2 — note its object ID and client ID.
Sandbox subscription IDs handy for at least one environment.
Local Node 20.10+ and az CLI.
A templates/client-repo/ tree present at the repo root (it is — shipped with C4).

Steps

1. Unit test suite

cd apps/github-onboarder
npm ci
npm run build
npm test

Build succeeds (dist/ populated).
All 6 test suites, 38 tests pass:
template.test.ts (placeholder substitution)
azure-graph.test.ts (federated cred upsert)
octokit-client.test.ts (GitHub API request shapes)
onboarder.test.ts (orchestrator happy path, idempotency, validation)
webhook.test.ts (payload parse + dispatch)
load-template.test.ts (real disk tree loads cleanly)

2. Acquire a GitHub App installation token

# Generate JWT with your App ID + private key, then exchange for installation token.
# Easiest: use `gh` with App authentication, or a one-off node script using @octokit/app.
node -e '
import("@octokit/app").then(async ({ App }) => {
  const app = new App({ appId: process.env.GH_APP_ID, privateKey: process.env.GH_APP_PRIVATE_KEY });
  const installation = await app.octokit.request("GET /orgs/{org}/installation", { org: process.env.GH_ORG });
  const inst = await app.getInstallationOctokit(installation.data.id);
  const tok = await inst.auth({ type: "installation" });
  console.log(tok.token);
});'

Token printed (starts with ghs_). Note it.

3. Acquire a Microsoft Graph token

az login --tenant <client-tenant-id>
az account get-access-token --resource https://graph.microsoft.com --query accessToken -o tsv

Token printed.

4. Run the onboarder against the sandbox

Create /tmp/spec.json:

{
  "client": {
    "slug": "b1-smoke",
    "displayName": "B1 Smoke Client",
    "githubOrg": "snowops-sandbox",
    "platformTeam": "snowops-sandbox/platform",
    "securityTeam": "snowops-sandbox/security",
    "snowopsDeliveryTeam": "snowops/delivery"
  },
  "azure": {
    "tenantId": "<sandbox tenant id>",
    "applicationObjectId": "<from B2>",
    "applicationClientId": "<from B2>"
  },
  "environments": [
    {
      "name": "sandbox",
      "subscriptionId": "<sandbox sub id>",
      "subscriptionDisplayName": "B1 Smoke Sandbox"
    }
  ],
  "hubspot_deal_id": "0"
}

Then run from apps/github-onboarder:

GH_TOKEN=<installation token from step 2> \
GRAPH_TOKEN=<graph token from step 3> \
node --input-type=module -e '
import { OctokitGitHubClient } from "./dist/octokit-client.js";
import { GraphFederatedCredClient } from "./dist/azure-graph.js";
import { handleOnboardWebhook, loadTemplateFromDisk } from "./dist/webhook.js";
import { Octokit } from "@octokit/rest";
import fs from "node:fs";

const payload = JSON.parse(fs.readFileSync("/tmp/spec.json", "utf8"));
const oct = new Octokit({ auth: process.env.GH_TOKEN });
const github = new OctokitGitHubClient(oct.request);
const graph = new GraphFederatedCredClient(process.env.GRAPH_TOKEN);
const result = await handleOnboardWebhook(payload, {
  github, graph,
  loadTemplate: (dir) => loadTemplateFromDisk("../../" + dir),
});
console.log(JSON.stringify(result, null, 2));
'

Script completes with "status": "created".
repoFullName is snowops-sandbox/b1-smoke-infra.
filesPushed lists all 8 template files.

5. Verify the repo via GitHub UI

Browse https://github.com/snowops-sandbox/b1-smoke-infra.
Default branch is main.
All 8 template files are present.
.github/CODEOWNERS has {{ORG}} replaced with snowops-sandbox.
README.md has {{CLIENT_DISPLAY_NAME}} replaced with B1 Smoke Client.

6. Verify branch protection on `main`

GitHub → Settings → Branches → main:

7. Verify Environments

GitHub → Settings → Environments → sandbox:

Environment exists.
Variables visible:
AZURE_TENANT_ID = your sandbox tenant GUID
AZURE_CLIENT_ID = the B2 app client ID
AZURE_SUBSCRIPTION_ID = your sandbox subscription GUID
AZURE_SUBSCRIPTION_NAME = B1 Smoke Sandbox
No secrets (you didn't pass any extraSecrets).

8. Verify federated identity credential in Azure AD

az ad app federated-credential list --id <applicationObjectId> -o table

One credential named gh-actions-b1-smoke-infra-sandbox exists.
Subject is repo:snowops-sandbox/b1-smoke-infra:environment:sandbox.
Issuer is https://token.actions.githubusercontent.com.
Audience is api://AzureADTokenExchange.

9. Idempotency check

Re-run the script from step 4 verbatim.

Exit code 0.
"status": "already_provisioned" (not created).
filesPushed is empty.
Federated credential count in Azure AD is still one (upserted, not duplicated).
No new branch protection diff; existing rules preserved.

10. End-to-end OIDC smoke test (optional, ~10 min extra)

In the new repo, push a trivial workflow that exchanges OIDC and runs az account show:

# .github/workflows/oidc-smoke.yml
on: { workflow_dispatch: {} }
permissions: { id-token: write, contents: read }
jobs:
  smoke:
    runs-on: ubuntu-latest
    environment: sandbox
    steps:
      - uses: azure/login@v2
        with:
          tenant-id: ${{ vars.AZURE_TENANT_ID }}
          client-id: ${{ vars.AZURE_CLIENT_ID }}
          subscription-id: ${{ vars.AZURE_SUBSCRIPTION_ID }}
      - run: az account show

Manually dispatch the workflow.
Azure login step succeeds (means the federated credential worked).
az account show returns the sandbox subscription.

11. Teardown

# Delete federated credentials
for id in $(az ad app federated-credential list --id <applicationObjectId> --query "[?starts_with(name, 'gh-actions-b1-smoke')].id" -o tsv); do
  az ad app federated-credential delete --id <applicationObjectId> --federated-credential-id "$id"
done

# Delete the test repo
gh repo delete snowops-sandbox/b1-smoke-infra --yes

Federated creds removed.
Repo deleted.

Pass criteria

All 38 unit tests pass.
Step 4 returns status: "created" and pushes all 8 template files.
Branch protection matches §6 of this runbook exactly.
Environment variables visible with correct GUIDs.
Federated credential exists with subject repo:<org>/<repo>:environment:sandbox.
Re-running is a no-op: status: "already_provisioned", no duplicates.
(Optional) OIDC smoke workflow logs in to Azure successfully.

Failure modes & escalation

Symptom	Action
`404` creating repo	App not installed on org or missing `administration:write`.
`403` setting environment	Missing `environments:write` permission on the App.
Graph `403` on PATCH/POST	Token lacks `Application.ReadWrite.OwnedBy` consented in client tenant.
Federated cred created but Azure login fails	Subject string mismatch — check the env name in workflow matches what was provisioned.
Branch protection rejected	Required check names not registered yet — push at least one commit so GH has seen the check contexts.
`libsodium` import fails at runtime	Make sure `dist/` was built; `npm run build` updates the lazy-imported wrapper.

Sign-off

Tester: ___ | Date: _ | Result: PASS / FAIL / N/A
Notes: