TD5 — Harden and inspect the same application

Step 1 — Name the threat and exact artifact · 30 minutes

Why: A security control needs an abuse case and a test. A scanner count alone is not a security decision.

Run this from your project root:

set -euo pipefail
. scripts/course-env.sh
set -a; . ./release.env; set +a
case "$DELIVERY_API_IMAGE" in ghcr.io/*@sha256:*) ;; *) echo 'BLOCKED: exact GHCR digest required'; exit 1;; esac
mkdir -p evidence/TD05
printf 'captured_at_utc=%s\nimage=%s\n' "$(date -u +%FT%TZ)" "$DELIVERY_API_IMAGE" > evidence/TD05/image-under-review.txt
docker compose -p docker-delivery ps | tee evidence/TD05/before-ps.txt
bash scripts/smoke.sh | tee evidence/TD05/before-smoke.txt
bash scripts/regression.sh | tee evidence/TD05/before-regression.txt

Write evidence/TD05/THREAT_MODEL.md. For each asset, identify an attacker path, likely abuse, a specific control and a test that could disprove your claim. Cover the Docker daemon, database network, password file, API/proxy filesystem, image identity and one runtime dependency. For one non-assessed example: a public documentation page should have an owner and review date; test that the link still resolves. Add your actual host observations and justify at least one tradeoff.

Expected and verify: Baseline smoke and persistence regression pass, and the scanned image coordinate matches the running container. If they differ, stop before scanning.

Step 2 — Produce an SBOM and scan without a daemon socket · 30 minutes

Why: The SBOM and findings must describe exactly the immutable image from Step 1. Trivy receives a read-only exported image tar, never the daemon socket.

scan_dir="$(mktemp -d)"
trap 'rm -f -- "$scan_dir/api.tar"; rmdir -- "$scan_dir"' EXIT
docker image save --output "$scan_dir/api.tar" "$DELIVERY_API_IMAGE"
test -s "$scan_dir/api.tar"
docker run --rm --platform "$COURSE_IMAGE_PLATFORM" \
  --mount "type=bind,source=$scan_dir,target=/scan,readonly" \
  "$TRIVY_IMAGE" image --input /scan/api.tar --format cyclonedx \
  > evidence/TD05/sbom.cdx.json
docker run --rm --platform "$COURSE_IMAGE_PLATFORM" \
  --mount "type=bind,source=$scan_dir,target=/scan,readonly" \
  "$TRIVY_IMAGE" image --input /scan/api.tar --scanners vuln --format json \
  > evidence/TD05/vulnerabilities.json
python3 -m json.tool evidence/TD05/sbom.cdx.json >/dev/null
python3 -m json.tool evidence/TD05/vulnerabilities.json >/dev/null
printf '%s\n' "$DELIVERY_API_IMAGE" > evidence/TD05/scanned-digest.txt
sha256sum evidence/TD05/sbom.cdx.json evidence/TD05/vulnerabilities.json \
  > evidence/TD05/scanner-artifacts.sha256
rm -f -- "$scan_dir/api.tar"
rmdir -- "$scan_dir"
trap - EXIT

The scanner may download or refresh its vulnerability database. Record scanner image version, UTC time, database/update status shown in stderr, and any offline or update failure. A JSON parse alone does not prove database freshness. Keep the unfiltered JSON for review; TD6's release gate applies its own CRITICAL,HIGH and fixed-vulnerability policy. Do not silently reinterpret that filtered CI file as a complete inventory.

Expected and verify: Both JSON files are nonempty and parse; scanned-digest.txt equals the Step 1 digest; sha256sum -c evidence/TD05/scanner-artifacts.sha256 passes. Inspect sbom.cdx.json for bomFormat: CycloneDX and a components array. Finding counts vary with the actual image and database. Record what you saw, never a prewritten PASS or CVE count.

If it fails: Record BLOCKED: scan/database unavailable and the exact scanner error. Correct network/cache availability or rerun the pinned scanner. Do not mount /var/run/docker.sock to work around the failure.

Step 3 — Build your hardening and test the permitted path · 45 minutes

Why: Least privilege matters only when the real API, proxy and database path still works. Edit your own Dockerfile, compose.yaml and Nginx definition. Run the API as a non-root identity, drop unneeded capabilities, prevent privilege gain, make API/proxy roots read-only with only narrow writable temporary paths, and bound resources. Keep the proxy loopback-only and API/database private. PostgreSQL requires its writable named data volume and a documented runtime exception. Mount the password file read-only and give only the processes that need it read access; do not put its value in YAML, environment, image history or evidence.

docker compose -p docker-delivery config --quiet
git diff --check
docker compose -p docker-delivery up --detach --no-build --force-recreate --wait --wait-timeout 120
bash scripts/smoke.sh | tee evidence/TD05/smoke-hardened.txt
bash scripts/regression.sh | tee evidence/TD05/regression-hardened.txt
docker compose -p docker-delivery ps | tee evidence/TD05/hardened-ps.txt

Check healthz, readyz, version, persistence and publication. On your own cloud host, keep the provider firewall closed to application and database ports and use the SSH tunnel for browser access. If a service fails, inspect its bounded logs and the specific permission error; do not switch to root or use chmod 777 as a blanket repair.

Step 4 — Prove a forbidden action fails for the intended reason · 25 minutes

Why: Pair allowed behavior with a denied action, then inspect the control that caused the denial.

api_id="$(docker compose -p docker-delivery ps -q delivery-api)"
proxy_id="$(docker compose -p docker-delivery ps -q proxy)"
postgres_id="$(docker compose -p docker-delivery ps -q postgres)"
for id in "$api_id" "$proxy_id" "$postgres_id"; do
  docker inspect --format 'name={{.Name}} user={{.Config.User}} readonly={{.HostConfig.ReadonlyRootfs}} capdrop={{json .HostConfig.CapDrop}} security={{json .HostConfig.SecurityOpt}} mounts={{json .Mounts}}' "$id"
done | tee evidence/TD05/runtime-controls.txt
docker exec "$api_id" sh -c 'id; grep -E "^(CapEff|NoNewPrivs):" /proc/1/status; test -r /run/secrets/postgres_password.txt' \
  | tee evidence/TD05/api-identity.txt
docker exec "$proxy_id" sh -c 'id; grep -E "^(CapEff|NoNewPrivs):" /proc/1/status' \
  | tee evidence/TD05/proxy-identity.txt
docker top "$postgres_id" -eo user,pid,ppid,args | tee evidence/TD05/postgres-process.txt
set +e
docker exec --user 0 "$api_id" sh -c 'touch /app/td05-denied' > evidence/TD05/root-write-denied.txt 2>&1
write_status=$?
set -e
printf 'exit_code=%s\n' "$write_status" >> evidence/TD05/root-write-denied.txt
test "$write_status" -ne 0
test "$(docker inspect --format '{{.HostConfig.ReadonlyRootfs}}' "$api_id")" = true
bash scripts/smoke.sh | tee evidence/TD05/allowed-after-denial.txt
docker run --rm --platform "$COURSE_IMAGE_PLATFORM" --network none --read-only \
  --user "$(id -u):$(id -g)" --group-add 20001 --cap-drop ALL \
  --mount type=volume,source=docker-delivery_course-secret,target=/run/secrets,readonly \
  --mount "type=bind,source=$PWD/evidence,target=/evidence,readonly" \
  "$PYTHON_IMAGE" python -c '
from pathlib import Path
secret = Path("/run/secrets/postgres_password.txt").read_bytes().strip()
if not secret:
    raise SystemExit("STOP: protected password file is empty")
leaks = [str(p.relative_to("/evidence")) for p in Path("/evidence").rglob("*")
         if p.is_file() and secret in p.read_bytes()]
if leaks:
    print("STOP: remove the password from these evidence files:", *leaks, sep="\n")
    raise SystemExit(1)
print("PASS: the current password value was not found in local evidence files")'

--user 0 is a one-off diagnostic: root must still receive a read-only file system error at /app. The running API process remains UID 10001. A generic “Permission denied” could be file ownership; it does not by itself prove a read-only mount. Check CapEff: 0000000000000000 and NoNewPrivs: 1 for API/proxy. The secret-file test checks readability only; never cat its value into evidence. Check the secret's name, not its value, with docker inspect ... .Config.Env if investigating a leak.

Expected and verify: Denied touch has a nonzero exit code and the read-only reason; API/proxy roots are read-only with ALL dropped; ordinary endpoint tests still pass. PostgreSQL's steady-state server process is non-root, with its data path writable.

If it fails: Preserve outputs, inspect the observed mount and process identity, then fix the owned definition and replay both allowed and denied tests. Do not report a denial caused only by an unrelated missing file or wrong path.

Step 5 — Triage, verify recovery and save · 50 minutes

Why: A scan is an input to a decision. A security claim must include its limits and show that recovery is still possible.

Create evidence/TD05/TRIAGE.md. For each High/Critical finding in your observed scan, record component/version, fix version or no fix, whether this workload reaches it, attacker path, disposition (rebuild/mitigate/time-bounded accept), owner, review date and residual risk. If there are no High/Critical findings, pick two Medium findings; if there are none, choose two significant SBOM components and explain why no selected CVE appeared. A representative decision is: if a vulnerable package has a fix and is reachable on the API request path, rebuild on a fixed pinned base, scan the new digest, and rerun tests before promotion. If no fix exists and the component is unreachable under this architecture, a dated, evidence-backed exception can be considered. Neither decision asserts that a specific CVE exists in your scan.

Replay the known-good TD4 definition from the recorded SHA, then return to your current definition. Compare the saved known-good definition with your hardened definition and explain any difference. This tests definition replay, not an image rollback. TD6 performs the actual digest rollback. Do not remove named data or secret volumes.

known_good_sha="$(cat evidence/TD04/known-good-sha.txt)"
test -n "$known_good_sha"
rollback_dir="$(mktemp -d)"
rmdir "$rollback_dir"
git worktree add --detach "$rollback_dir" "$known_good_sha"
docker compose -p docker-delivery -f "$rollback_dir/compose.yaml" config --quiet
docker compose -p docker-delivery -f "$rollback_dir/compose.yaml" up \
  --detach --no-build --force-recreate --wait --wait-timeout 120
bash scripts/smoke.sh | tee evidence/TD05/known-good-smoke.txt
docker compose -p docker-delivery up \
  --detach --no-build --force-recreate --wait --wait-timeout 120
bash scripts/smoke.sh | tee evidence/TD05/returned-hardened-smoke.txt
bash scripts/regression.sh | tee evidence/TD05/returned-hardened-regression.txt
git worktree remove "$rollback_dir"

Write evidence/TD05/OBSERVATIONS.md with the actual UTC time, Docker version, scanned/running digest, scanner database status, SBOM/scan hashes, one triage decision and expiry, API/proxy controls, allowed and denied outputs with exit codes, recovery outputs, and one remaining limitation. Use BLOCKED for any step not run. For a rootful Docker host, daemon-level authority remains a residual risk. Verify sha256sum -c evidence/TD05/scanner-artifacts.sha256 after copying evidence.

Expected and verify: The hardened definition is active again; smoke and persistence regression pass. git status --short shows only your intended files. A new base image or rebuild produces a new digest and requires a new scan and approval.

If it fails: Keep the last known-good coordinate and failure output. Restore the original definition and diagnose before claiming completion. Do not delete PostgreSQL data.