TD6 — Release an exact artifact and practise rollback

Step 1 — Record the known-good release and finish safe runbooks · 25 minutes

Why: Rollback needs a specific previous digest P and a script that cannot accept a mutable tag. Capture the currently running API reference before any candidate is deployed.

set -euo pipefail
. scripts/course-env.sh
set -a; . ./release.env; set +a
mkdir -p evidence/TD06
previous_image="$(docker inspect --format '{{.Config.Image}}' \
  "$(docker compose -p docker-delivery ps -q delivery-api)")"
case "$previous_image" in ghcr.io/*@sha256:*) ;; *) echo 'BLOCKED: P is not a GHCR digest'; exit 1;; esac
printf 'captured_at_utc=%s\nprevious_image=%s\n' \
  "$(date -u +%FT%TZ)" "$previous_image" > evidence/TD06/previous-release.txt
bash scripts/smoke.sh | tee evidence/TD06/before-smoke.txt
bash scripts/regression.sh | tee evidence/TD06/before-regression.txt
bash -n scripts/deploy.sh scripts/rollback.sh
set +e
bash scripts/deploy.sh delivery-api:mutable-tag > evidence/TD06/mutable-rejected.txt 2>&1
mutable_status=$?
set -e
printf 'exit_code=%s\n' "$mutable_status" >> evidence/TD06/mutable-rejected.txt
test "$mutable_status" -ne 0

Complete your starter scripts/deploy.sh and scripts/rollback.sh yourself. Their contract is: accept only an immutable GHCR digest, record current/previous/pending state on the deployment host, pull instead of build, verify the activated image and service behavior, recover to P after a failed candidate where possible, and preserve the database volume. An interrupted operation must leave a reviewable marker. Test rejection of a mutable tag before running a real deployment.

Expected and verify: P is a lowercase GHCR digest, baseline smoke/regression pass, and the tag rejection exits nonzero without changing the running image. Run docker inspect --format '{{.Config.Image}}' "$(docker compose -p docker-delivery ps -q delivery-api)" again to compare with P.

If it fails: Preserve the error and stop before release. A missing P or failed baseline cannot be repaired by guessing a tag.

Step 2 — Make CI validate and publish one candidate · 35 minutes

Why: A release digest must come from one known source revision. Complete your starter GitHub Actions workflow. Pull requests validate without package publication. A release-branch run validates source, builds/pushes one API candidate, then creates an SBOM, vulnerability result, digest and source-linked artifact. Give the publishing job package-write permission only where needed, pin third-party actions, and keep SSH/host deployment credentials out of CI.

Run local syntax and supplied-application checks before pushing your own completed workflow. These checks do not certify your delivery definitions: implement the workflow validation against the public safety requirements, and demonstrate that a deliberately violated requirement is rejected. No private grading validator is included:

python3 -m venv .venv
lab_python=.venv/bin/python
"$lab_python" -m pip install --require-hashes --requirement requirements-dev.txt
bash -n scripts/deploy.sh scripts/rollback.sh scripts/regression.sh scripts/verify_runtime_contract.sh
"$lab_python" -m compileall -q app tests
"$lab_python" -m pytest -q
docker compose --env-file .env.example -p docker-delivery config --quiet
git diff --check
git add .github/workflows/container-delivery.yml scripts/deploy.sh scripts/rollback.sh
git diff --cached --check
git commit -m 'td6: define immutable release workflow'
release_sha="$(git rev-parse HEAD)"
printf 'release_sha=%s
' "$release_sha" > evidence/TD06/release-source.txt
git push origin HEAD

Inspect the Actions run for this full source SHA. If a push does not start the workflow, check branch and path filters; if you change them, commit a new revision and use its new SHA. A green badge for another commit is not evidence. If your starter lacks a requirement or validator, record BLOCKED and use the published setup/help route; do not copy a private solution.

Step 3 — Inspect the CI artifact and obtain digest-specific approval · 35 minutes

Why: The reviewer approves what CI actually built, not a tag that may move. Audit one run ID and download its exact release artifact. Install/authenticate gh on a trusted terminal for your private repository; no token belongs in evidence.

run_id="$(gh run list --workflow container-delivery.yml --commit "$release_sha" \
  --limit 20 --json databaseId,headSha --jq '.[0].databaseId')"
test -n "$run_id"
gh run watch "$run_id" --exit-status
gh run view "$run_id" --json url,headSha,event,status,conclusion \
  > evidence/TD06/ci-run.json
python3 - "$release_sha" <<'PY'
import json, pathlib, sys
run = json.loads(pathlib.Path('evidence/TD06/ci-run.json').read_text())
if run.get('headSha') != sys.argv[1] or run.get('conclusion') != 'success':
    raise SystemExit('BLOCKED: wrong or unsuccessful CI run')
PY
mkdir -p evidence/TD06/ci
gh run download "$run_id" --name release --dir evidence/TD06/ci
(cd evidence/TD06/ci && sha256sum --check artifact-checksums.sha256)
python3 -m json.tool evidence/TD06/ci/sbom.cdx.json >/dev/null
python3 -m json.tool evidence/TD06/ci/vulnerabilities.json >/dev/null
candidate_image="$(sed -n 's/^image=//p' evidence/TD06/ci/release.env)"
artifact_sha="$(sed -n 's/^git_sha=//p' evidence/TD06/ci/release.env)"
case "$candidate_image" in ghcr.io/*@sha256:*) ;; *) echo 'BLOCKED: D is not a GHCR digest'; exit 1;; esac
test "$artifact_sha" = "$release_sha"
test "$candidate_image" != "$previous_image"
printf 'candidate=%s\nsource=%s\nrun_id=%s\n' \
  "$candidate_image" "$release_sha" "$run_id" > evidence/TD06/candidate.txt

Review the workflow URL in ci-run.json, job conclusions, scan policy result, CycloneDX file, vulnerability JSON, checksums, full source SHA and exact D. The workflow's vulnerability JSON is intentionally filtered to fixed High/Critical candidates; retain TD5's broader scan for context. A zero row in the filtered file is not proof of zero risk. If possible, independently inspect D with docker buildx imagetools inspect "$candidate_image" after registry authentication.

Ask an authorized reviewer other than the active deployer to record the decision for this exact D in evidence/TD06/APPROVAL.md:

candidate=<exact D>
source_sha=<40-character release SHA>
run_url=<actual Actions run URL>
previous=<exact P>
reviewer=<reviewer alias>
decided_at_utc=<actual UTC time>
decision=APPROVED or REJECTED
scan_disposition=<actual finding/review decision>
rollback_trigger=<failed smoke, version, critical CRUD/persistence test, or runtime contract>
reason=<specific reason>

Record the actual decision; the template is not an approval. If CI, GHCR, artifact or reviewer access is absent, write BLOCKED and stop before deployment.

Expected and verify: The run SHA equals release_sha, the artifact SHA equals it, checksums pass, and the approval names D and P exactly. Any mismatch or REJECTED decision blocks promotion.

If it fails: Reconcile the source, run and digest. Do not edit a downloaded artifact to make identities appear to match.

Step 4 — Deploy that digest and test the live service · 35 minutes

Why: The target host must pull the approved D. It must not rebuild. The same commands run on My Laptop's Docker host or your own cloud Docker host; only SSH entry and the browser tunnel differ.

First check the approval and configure temporary private-package access on the Docker host. Run the block in a private terminal with shell tracing off. The temporary Docker credential directory is removed when the shell exits; do not put a token in a file, log or command argument.

grep -Fx "candidate=$candidate_image" evidence/TD06/APPROVAL.md
grep -Fx "source_sha=$release_sha" evidence/TD06/APPROVAL.md
grep -Fx "previous=$previous_image" evidence/TD06/APPROVAL.md
grep -Fx 'decision=APPROVED' evidence/TD06/APPROVAL.md
set +x
ghcr_config="$(mktemp -d)"
chmod 700 "$ghcr_config"
export DOCKER_CONFIG="$ghcr_config"
trap 'docker logout ghcr.io >/dev/null 2>&1 || true; rm -f -- "$DOCKER_CONFIG/config.json"; rmdir -- "$DOCKER_CONFIG"' EXIT
read -r -p 'GitHub username with package access: ' GHCR_USERNAME
read -r -s -p 'GHCR read:packages token: ' ghcr_token
printf '\n'
printf '%s' "$ghcr_token" | docker login ghcr.io --username "$GHCR_USERNAME" --password-stdin
unset ghcr_token

If the package is public, a registry login may be unnecessary. For a private package, repository access alone is not package access. Stop as BLOCKED if login or pull is denied. Keep this terminal open through Step 5 so rollback can pull P if it is not cached.

If deploy/state/current-image does not yet exist, first establish P using --initialize after its TD5 checks and reviewer approval. This is a one-time state transition. If it exists, assert it already names P. Then deploy D:

if test -f deploy/state/current-image; then
  test "$(cat deploy/state/current-image)" = "$previous_image"
else
  bash scripts/deploy.sh --initialize "$previous_image" | tee evidence/TD06/initialize-previous.txt
fi
bash scripts/deploy.sh "$candidate_image" | tee evidence/TD06/deploy.txt
test "$(cat deploy/state/current-image)" = "$candidate_image"
active_image="$(docker inspect --format '{{.Config.Image}}' \
  "$(docker compose -p docker-delivery ps -q delivery-api)")"
test "$active_image" = "$candidate_image"
EXPECTED_GIT_SHA="$release_sha" bash scripts/smoke.sh | tee evidence/TD06/after-smoke.txt
DELIVERY_API_IMAGE="$candidate_image" EXPECTED_GIT_SHA="$release_sha" \
  bash scripts/regression.sh | tee evidence/TD06/after-regression.txt
test "$(docker inspect --format '{{.Config.Image}}' \
  "$(docker compose -p docker-delivery ps -q delivery-api)")" = "$candidate_image"
printf 'active_image=%s\n' "$candidate_image" > evidence/TD06/active-release.txt

The explicit DELIVERY_API_IMAGE=D on regression matters because that test recreates the API container. It prevents a stale release.env or shell variable from silently recreating P. The deploy script checks /version against D's OCI full revision and the target's host publication contract. Regression also creates a real item and verifies it survives API recreation. The PostgreSQL data and secret volumes stay in place.

Expected and verify: P is recorded as previous, D as current, the running container reports D, /version.git_sha is release_sha, and smoke plus critical CRUD/persistence regression pass. If any test fails, the runbook attempts bounded recovery to P and retains truthful state.

If it fails: Preserve the deploy log, active image and error. Check whether deploy/state/pending-operation exists before retrying. Never delete the database volume or manually call a candidate a success.

Step 5 — Inject a safe regression and execute rollback · 30 minutes

Why: The same committed test must fail while D is active and pass once P is active. The safe fixture changes the expected Git SHA, not application data or host exposure.

previous_sha="$(docker image inspect --format '{{index .Config.Labels "org.opencontainers.image.revision"}}' "$previous_image")"
case "$previous_sha" in ????????????????????????????????????????) ;; *) echo 'BLOCKED: P lacks full OCI revision'; exit 1;; esac
test "$previous_sha" != "$release_sha"
set +e
DELIVERY_API_IMAGE="$candidate_image" EXPECTED_GIT_SHA="$previous_sha" \
  bash scripts/regression.sh > evidence/TD06/regression-before-rollback.txt 2>&1
regression_status=$?
set -e
printf 'exit_code=%s\n' "$regression_status" >> evidence/TD06/regression-before-rollback.txt
test "$regression_status" -ne 0
test "$(docker inspect --format '{{.Config.Image}}' \
  "$(docker compose -p docker-delivery ps -q delivery-api)")" = "$candidate_image"
bash scripts/rollback.sh | tee evidence/TD06/rollback.txt
test "$(cat deploy/state/current-image)" = "$previous_image"
EXPECTED_GIT_SHA="$previous_sha" bash scripts/smoke.sh | tee evidence/TD06/after-rollback-smoke.txt
DELIVERY_API_IMAGE="$previous_image" EXPECTED_GIT_SHA="$previous_sha" \
  bash scripts/regression.sh | tee evidence/TD06/regression-after-rollback.txt
rolled_back_image="$(docker inspect --format '{{.Config.Image}}' \
  "$(docker compose -p docker-delivery ps -q delivery-api)")"
test "$rolled_back_image" = "$previous_image"
printf 'rolled_back_image=%s\n' "$rolled_back_image" > evidence/TD06/rolled-back-release.txt

regression-before-rollback.txt must show the expected/observed SHA mismatch and nonzero exit code. regression-after-rollback.txt must show the same suite passing after P returns; rolled-back-release.txt must contain P. A restart of D is not rollback. Application image rollback does not restore PostgreSQL data; a real incompatible schema migration needs its own reviewed restore/migration plan and backup. Do not perform a database restore for this safe exercise.

Write evidence/TD06/OBSERVATIONS.md with actual UTC time, Docker version, release SHA, CI run URL and result, D, P, scan/SBOM checksums, reviewer decision, active image before/after, smoke and regression outputs and exit codes, rollback trigger, and remaining limitation. Use BLOCKED for any unexecuted gate. The active image after rollback has P's older OCI source SHA; the operational definition and evidence commit can have a newer SHA. Record both honestly.

If it fails: Keep both test outputs, state files and active image coordinate. Stop for diagnosis rather than trying a different digest or changing database data.