Lab — from source code to AKS operations

▶ Do

1. Run the supplied unit and integration tests for the code-complete Skybridge release.
2. Confirm Git and the supported container engine are installed and the daemon is available.
3. Pull and run a known image, then build a minimal local image to verify the complete tool path.

./scripts/test-release.sh
docker version
docker run --rm hello-world
docker run --name workshop-nginx -d -p 8080:80 nginx:alpine
curl --fail http://localhost:8080/
docker rm -f workshop-nginx
echo 'FROM alpine' > Dockerfile.test
docker build -t aks-workshop-check:1 -f Dockerfile.test .
rm Dockerfile.test

▶ Do

1. Review the supplied release manifest and the dev, test, UAT, and production environment profiles.
2. In PROMOTION-BASELINE.md, classify what currently travels with the release and what each environment supplies.
3. Write the proposed invariant: one immutable image reference plus injected environment configuration produces each stage deployment.

./scripts/show-release-manifest.sh
diff environments/dev.env environments/test.env
git rev-parse HEAD

▶ Do

1. Inspect the release manifest, launch scripts, linked libraries, and existing environment documentation.
2. Record build-time dependencies separately from runtime dependencies for the Java gateway and C++ parser.
3. Define the entrypoint, working directory, user, writable paths, ports, and termination behavior in CONTAINER-CONTRACT.md.

./scripts/inspect-runtime.sh gateway-java
./scripts/inspect-runtime.sh parser-cpp
ldd build/parser-cpp/skybridge-parser

▶ Do

1. Classify every setting as invariant, stage configuration, or secret; define its name, format, default policy, and missing-value behavior.
2. Define startup, readiness, and liveness signals plus the structured fields required in logs.
3. Add configuration, secret, health, and telemetry sections to CONTAINER-CONTRACT.md.

./scripts/inventory-config.sh
./scripts/validate-contract.sh CONTAINER-CONTRACT.md

▶ Do

1. Record initial CPU and memory requests, limits, expected concurrency, and shutdown grace period.
2. Require a non-root user, read-only runtime where possible, minimal capabilities, and no embedded credentials.
3. Define test, scan, health, performance, provenance, and rollback gates for promotion of one immutable digest.

▶ Do

1. Grep the three components and the runbook for every tunable: ports, hostnames, DB DSN, timeouts, thread pools, credentials.
2. Sort each into per-environment vs constant, and flag any secret.
3. Capture the result as a config table in CONFIG.md.

git grep -nEI 'password|secret|host|port|timeout|url|dsn' -- apps legacy

▶ Do

1. Refactor each component to read every per-environment setting from an environment variable, with a safe default only where it's truly universal.
2. Add a committed .env.example documenting each variable; keep the real .env git-ignored.
3. Run the app from the same build with the supplied dev, test, and UAT profiles and observe only the intended configuration differences.

./scripts/show-release-manifest.sh > workshop-notes/release-before.txt
./scripts/run-release.sh --env environments/dev.env
./scripts/run-release.sh --env environments/test.env
./scripts/run-release.sh --env environments/uat.env
./scripts/show-release-manifest.sh > workshop-notes/release-after.txt
diff workshop-notes/release-before.txt workshop-notes/release-after.txt
git diff --exit-code -- apps               # no source edit

▶ Do

1. Remove every credential from source and config files; replace with an env var reference.
2. Provide secrets at run time only (an injected env var or a mounted file), never in the build.
3. Add a pre-commit secret scan so a credential can't slip back in.

git grep -nEI 'password=|secret=|BEGIN [A-Z ]*PRIVATE KEY' -- . \
  && echo 'Review every match above' \
  || echo 'No matching tracked content found'

▶ Do

1. Write a multi-stage Dockerfile: a build stage that compiles the gateway, a slim runtime stage with only the JRE and the artifact.
2. Copy dependencies before source so dependency layers cache across code changes.
3. Set a non-root user and an explicit entrypoint that reads config from the environment.

docker build -t gateway-java:0.1.0 apps/gateway-java/
docker run --rm -p 4561:4561 \
  --add-host host.docker.internal:host-gateway \
  -e GATEWAY_PORT=4561 -e PARSER_URL=http://host.docker.internal:8080 \
  gateway-java:0.1.0

▶ Do

1. Write a multi-stage Dockerfile: a build stage with the C++ toolchain, a minimal runtime stage with just the binary and its required libs.
2. Verify the binary runs on the slim base (link errors appear here, not in the build stage).
3. Expose the parser's HTTP port and read its config from the environment.

docker build -t parser-cpp:0.1.0 apps/parser-cpp/
docker run --name parser-local -d -p 8080:8080 parser-cpp:0.1.0
curl --fail http://localhost:8080/healthz
docker rm -f parser-local

▶ Do

1. Multi-stage: build the console assets, then serve them from a minimal static server image.
2. Use the supplied startup script to render a browser-readable config.js from API_BASE before the web server starts. A static nginx image does not substitute environment variables into already-built JavaScript by itself.
3. Run it and load the console in a browser.

docker build -t ops-console:0.1.0 apps/ops-console/
docker run --name ops-console-local -d -p 3000:80 \
  -e API_BASE=http://localhost:8080 ops-console:0.1.0
curl --fail http://localhost:3000/config.js
docker rm -f ops-console-local

▶ Do

1. Add a .dockerignore to each component so build context excludes VCS, local env, and build junk.
2. Scan each image for known CVEs and trim obvious bloat from the runtime stages.
3. Run the gateway under controlled CPU and memory limits; inspect its logs, process list, network, mounts, and live resource use.

docker scout cves gateway-java:0.1.0
docker run -d --name gateway-limited --cpus 1 --memory 512m \
  -e GATEWAY_PORT=4561 -e PARSER_URL=http://parser.invalid:8080 \
  gateway-java:0.1.0
docker logs gateway-limited
docker stats --no-stream gateway-limited
docker inspect gateway-limited
docker rm -f gateway-limited

▶ Do

1. Write compose.yaml with services for gateway, parser, ops-console, and a Postgres instance.
2. Wire them by service name (e.g. PARSER_URL=http://parser:8080, DB_DSN=…@db:5432/…) using the env vars from Phase 2.
3. Add a named Postgres volume, database initialization, health checks, and depends_on conditions for dependencies that must be healthy; then render and start the stack.

docker compose config --quiet
docker compose up -d --wait
docker compose ps
./scripts/smoke.sh 127.0.0.1 4561    # end-to-end through the composed stack

▶ Do

1. Generate controlled traffic and record baseline latency, throughput, CPU, memory, health, and log error counts.
2. Inject one broken service hostname, use Compose state, logs, and network inspection to diagnose it, then restore the configuration.
3. Apply a CPU or memory constraint to one component and explain the resulting runtime signal.

./scripts/load.sh --duration 60s
docker compose ps
docker compose logs --since 5m gateway parser
docker compose config --networks
docker inspect "$(docker compose ps -q gateway)" --format '{{json .NetworkSettings.Networks}}'
docker stats --no-stream

▶ Do

1. Parameterize Compose with the supplied dev, test, and UAT environment profiles.
2. Run the end-to-end test in each profile without invoking an image build.
3. Record each container's image ID and verify it remains unchanged across the three runs.

for STAGE in dev test uat; do
  docker compose --env-file "environments/$STAGE.env" up -d --no-build --wait
  ./scripts/smoke.sh 127.0.0.1 4561
  docker compose images --quiet | sort > "workshop-notes/images-$STAGE.txt"
  docker compose down
done
diff workshop-notes/images-dev.txt workshop-notes/images-test.txt
diff workshop-notes/images-dev.txt workshop-notes/images-uat.txt

▶ Do

1. Complete the supplied CI workflow template so a push builds each image, runs unit tests, scans for CVEs, and pushes sha-<short> only on success. Do not start from a non-executable sketch.
2. Open a trivial PR that changes one line, and watch the pipeline run.
3. Confirm the new SHA tag lands in the registry only when checks are green.

./scripts/validate-ci-workflow.sh .github/workflows/build.yml
git commit -am "chore: trivial change" && git push

▶ Do

1. Add OCI source/revision/version labels, tag each CI-built image by semantic version and commit SHA, and generate an SBOM.
2. Apply the vulnerability policy, push passing images, and record the registry digest returned for each component.
3. Pull by digest into a clean environment and run the end-to-end smoke test without rebuilding.

▶ Do

1. Update CONTAINER-CONTRACT.md with each component's registry digest, ports, required env vars, secret inputs, health endpoints, resource baseline, and shutdown behavior.
2. Record test, scan, SBOM, provenance, and promotion-gate evidence in RELEASE-EVIDENCE.md.
3. Map each runtime field to the Kubernetes object or container-spec field that will satisfy it next.

▶ Do

1. Create a throwaway local cluster with kind.
2. Load the exact SHA-tagged local application images into the cluster; a private registry is not automatically authenticated to a local kind cluster.
3. Confirm the node is Ready and inspect your current context.

kind create cluster --name z2h
SHA=$(git rev-parse --short HEAD)
kind load docker-image gateway-java:sha-$SHA parser-cpp:sha-$SHA ops-console:sha-$SHA --name z2h
kubectl config current-context
kubectl get nodes -o wide

▶ Do

1. Write a Deployment + Service for the gateway and parser, plus a dev-grade StatefulSet + Service + persistent volume claim for Postgres.
2. For kind, reference the exact loaded sha-<short> tags with imagePullPolicy: IfNotPresent. Implement startup/readiness/liveness probes, CPU/memory requests and limits, graceful termination, and a non-root security context. Phase 7 replaces these local references with ACR digests.
3. Apply everything, watch the Pods become Ready, and smoke-test through in-cluster Service DNS.

./scripts/validate-k8s-manifests.sh k8s/
kubectl apply -f k8s/
kubectl get pods,svc
kubectl port-forward svc/gateway 4561:4561

▶ Do

1. Move per-environment settings into a ConfigMap and credentials into a Secret.
2. Reference them from the Deployments via envFrom / valueFrom.
3. Change one value and roll the Deployment — no image rebuild.

kubectl create configmap gateway-config --from-env-file=environments/dev.env \
  --dry-run=client -o yaml | kubectl apply -f -
kubectl create secret generic skybridge-secrets --from-env-file=.env.local \
  --dry-run=client -o yaml | kubectl apply -f -
kubectl rollout restart deploy/gateway deploy/parser
kubectl rollout status deploy/gateway

▶ Do

1. Delete a parser Pod and watch the Deployment replace it, then scale the parser to 3 replicas.
2. Inject a broken image reference or readiness path and diagnose it using workload state, events, description, logs, and metrics.
3. Restore the current image, verify readiness, then load and roll out another tested SHA-tagged image and undo the rollout. Registry digests replace these local tags in Phase 7.

kubectl delete pod -l app=parser          # watch the Deployment replace it
kubectl scale deploy/parser --replicas=3
./scripts/inject-k8s-failure.sh bad-readiness
kubectl get pods
kubectl describe pod -l app=parser
kubectl get events --sort-by=.lastTimestamp
kubectl logs -l app=parser --all-containers --tail=100
./scripts/reset-k8s-failure.sh
NEXT_SHA=<another-tested-short-sha>
kind load docker-image parser-cpp:sha-$NEXT_SHA --name z2h
kubectl set image deploy/parser parser=parser-cpp:sha-$NEXT_SHA
kubectl rollout status deploy/parser
kubectl rollout undo deploy/parser

▶ Do

1. Set the instructor-approved region, verify Azure sign-in, active subscription, regional compute usage, and required permissions; then create a resource group and disposable training AKS cluster.
2. Merge its credentials into your kubeconfig.
3. Confirm the nodes are Ready and your context now points at AKS.

LOCATION=<approved-region>
az account show --query '{subscription:name,tenant:tenantId}' -o table
az provider show --namespace Microsoft.ContainerService --query registrationState -o tsv
az vm list-usage --location "$LOCATION" -o table
az group create --name rg-z2h --location "$LOCATION" \
  --tags purpose=training expires=<YYYY-MM-DD>
az aks create --resource-group rg-z2h --name aks-z2h --location "$LOCATION" \
  --node-count 1 --tier free --generate-ssh-keys
az aks get-credentials --resource-group rg-z2h --name aks-z2h --overwrite-existing
kubectl config current-context
kubectl get nodes -o wide

▶ Do

1. Create an ACR using a globally unique name.
2. Tag and push the already-tested local SHA images. Do not rebuild: a new az acr build would create different artifacts and break the promotion proof.
3. Attach the ACR to the cluster so nodes can pull without a pull secret. This operation creates a role assignment and therefore requires permission such as Owner or User Access Administrator at the applicable scope.

ACR_NAME="acrz2h${RANDOM}${RANDOM}"
SHA=$(git rev-parse --short HEAD)
az acr create --resource-group rg-z2h --name "$ACR_NAME" --sku Basic \
  --role-assignment-mode rbac
LOGIN_SERVER=$(az acr show --name "$ACR_NAME" --query loginServer -o tsv)
az acr login --name "$ACR_NAME"
for IMAGE in gateway-java parser-cpp ops-console; do
  docker image inspect "$IMAGE:sha-$SHA" >/dev/null
  docker tag "$IMAGE:sha-$SHA" "$LOGIN_SERVER/$IMAGE:sha-$SHA"
  docker push "$LOGIN_SERVER/$IMAGE:sha-$SHA"
done
az aks update --resource-group rg-z2h --name aks-z2h --attach-acr "$ACR_NAME"
./scripts/set-aks-images.sh "$ACR_NAME" "$SHA" k8s/aks/

▶ Do

1. Inspect the generated AKS overlay and confirm all three images use fully qualified ACR references pinned by @sha256: digest.
2. Apply them to AKS and expose the gateway with a LoadBalancer Service.
3. Get the external IP and smoke-test the cluster.

grep -R '@sha256:' k8s/aks/
kubectl apply -f k8s/aks/
kubectl rollout status deploy/gateway
kubectl get svc gateway -w        # press Ctrl+C after EXTERNAL-IP appears
IP=$(kubectl get svc gateway -o jsonpath='{.status.loadBalancer.ingress[0].ip}')
test -n "$IP"
./scripts/smoke.sh "$IP" 4561

▶ Do

1. Capture cluster, node-pool, ACR, image-digest, Service, and identity facts in AKS-BASELINE.md.
2. Map the managed control plane, node pool, Azure Load Balancer, ACR authorization, and workload objects to their owners.
3. Keep the cluster running for Phase 8; use the Azure Pricing Calculator to record a dated estimate and its assumptions. Also record the cleanup command.

▶ Do

1. Run the supplied load profile against the AKS gateway and record request rate, latency percentiles, errors, CPU, memory, restarts, and replica readiness.
2. Correlate application logs with Pod identity and immutable image revision; inspect workload state, events, and node placement.
3. Save the load parameters and observations in AKS-OPERATIONS-REPORT.md so the exact test can be repeated.

./scripts/load.sh --target "$IP:4561" --profile baseline --duration 5m
kubectl get pods -o wide
kubectl top pods
kubectl get events --sort-by=.lastTimestamp
kubectl logs -l app=gateway --all-containers --since=10m --prefix

▶ Do

1. Inject the assigned failure: missing configuration, failed readiness, or memory pressure.
2. Follow the diagnostic sequence without skipping steps: get state → describe → events → previous/current logs → metrics → Azure resource health.
3. State one falsifiable hypothesis, restore the workload, rerun the load profile, and record the evidence that confirmed or rejected the hypothesis.

./scripts/inject-aks-failure.sh assigned
kubectl get pods
kubectl describe pod -l app=gateway
kubectl get events --sort-by=.lastTimestamp
kubectl logs deploy/gateway --all-pods --previous
kubectl top pods
az aks show -g rg-z2h -n aks-z2h --query provisioningState -o tsv
az aks nodepool show -g rg-z2h --cluster-name aks-z2h -n nodepool1 \
  --query '{provisioningState:provisioningState,powerState:powerState.code}' -o table
./scripts/reset-aks-failure.sh

▶ Do

1. Choose one evidence-backed change: resource request/limit, JVM container setting, parser concurrency, probe threshold, or replica count.
2. Apply only that change, wait for a healthy rollout, and repeat the exact baseline load profile.
3. Compare before/after latency, throughput, errors, CPU, memory, and cost implication; update the container contract and clean up the Azure resource group.

kubectl apply -f k8s/
kubectl rollout status deploy/gateway
./scripts/load.sh --target "$IP:4561" --profile baseline --duration 5m
./scripts/compare-runs.sh baseline optimized
az group delete -n rg-z2h --yes --no-wait
az group wait --name rg-z2h --deleted
az group exists --name rg-z2h             # must return false