Console release defences, Plan 4 of 4: D4 (the release carries a console; the roll goes dev first) and the rollout

> For agentic workers: REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (- [ ]) syntax for tracking.

Goal: Each public release publishes a signed console artifact. The flagship console is rolled with exactly those bytes, through dev first, carrying dev's proof. The four defences are switched on in the order spec §6 requires.

Architecture: Three repos. The platform README's release recipe gains the console build and a console roll step. carloku-infrastructure/scripts/roll-console.sh gains:

• a dev target;
• --artifact and a single -trimpath build;
• a binding check;
• the dev-proof carry, with a one-time --bootstrap.

The rollout is a runbook section in carloku-infrastructure's README, run by an operator over SSM. It is not code.

Tech Stack: bash, the carlos CLI (with Plans 1–3 built), AWS CLI with SSM, jq.

Spec: docs/superpowers/specs/2026-10-06-console-release-defences-design.md (rev 5): §5 (D4), §6 (rollout). Depends on Plans 1–3 being merged and in a release. The script calls carlos proof get|put (Plan 3) and relies on the box's settle check (Plan 1).

Global Constraints

• The release builds console-linux-arm64 and console-linux-amd64 with go build -trimpath -ldflags "-s -w -X github.com/carlosframework/platform/internal/version.Build=vX.Y.Z" ./cmd/console and lists them in the signed SHA256SUMS.txt (spec §5.1).
• Dev coordinates: profile kass, bucket carlos-dev-640012953448, console box i-05498eada6f0afd23, origin platform.dev.carlosframework.com, region eu-west-1 (spec §5.2).
• Flagship coordinates: profile keymail, bucket carlos-flagship-271376211898, console box i-092c0c1eea75723cb, origin platform.carlosframework.com.
• The script refuses a flagship roll without a dev proof, except with --bootstrap, which prints that it is skipping (spec §5.2).
• The script keeps its final systemctl restart on any box whose binding is wrong (spec §5.2, §2.2).
• The public carlos-linux-* binary is never a console artifact, and the recipe says so (spec §5.2).
• README prose in both repos goes through the copy-review skill before it is committed (Paul's standing rule for READMEs). Plans 1 and 2's README notes need the same gate. The copy rule: no em dashes in new prose ([[copy-voice-no-em-dashes]]).
• Tito is a third party: the script's tito target is left as it is and never run by this plan.
• Plan decision: the console routes' readiness uses the TCP-connect fallback (spec §2.4 rule 2) on 127.0.0.1:8090, not a configured probe. The console's /healthz answers as soon as its listener is up, so a probe would check nothing more than the connect does. No box-local CLI sets registry.Route.Health on a console row today; adding one is a follow-up, needed only if that equivalence ever stops holding.

Review Focus

1. The public release's SHA256SUMS.txt lacks a console line (an older release, or a recipe step skipped). Expect: --artifact refuses an unverifiable file, never ships it. Covered in Task 2 (verify_artifact).
2. Dev has the build but its proof isn't minted yet (inside the 15 minutes). Expect: the flagship target exits non-zero naming how long dev has had it, with nothing shipped to flagship. Task 2's dry-run check.
3. The binding on a box is wrong. Expect: the script says so, keeps its final restart, and still verifies /proc/<pid>/exe. Task 2's dry-run check against the flagship box today (stamped on console.carloku.com, while ExecStart names platform.carlosframework.com; confirm on the box).
4. An operator rolls a hotfix sha, not a release. Expect: one -trimpath build reused for dev and flagship, so the proof's hash matches. Task 2's build-once rule.
5. Someone runs the old two-repo habit (platform-infrastructure's copy of the script for flagship). Expect: that copy says it is retired and points here. Task 3.

---

Task 1: The release recipe publishes a console and rolls it dev first

Files:

• Modify: README.md (platform repo): "Releasing the platform" steps 2, 4 and 6, and a new console step after "Last, roll it"

• [ ] Step 1: Draft the recipe changes

Step 2, append:

   Build the console twice, from the same clean checkout:
   `CGO_ENABLED=0 GOOS=linux GOARCH=<arch> go build -trimpath -ldflags "-s -w -X github.com/carlosframework/platform/internal/version.Build=vX.Y.Z" -o console-linux-<arch> ./cmd/console`
   for arm64 and amd64. The console is `./cmd/console`. The `carlos` binary is
   never a console artifact; shipping it as one took the flagship console down
   on 2026-10-06.

Step 4: "over the eight files" becomes "over the ten files". Step 6: "with all ten files" becomes "with all twelve files".

After the "Last, roll it" paragraph, add:

**Roll the console.** After the boxes are current, roll the release's own
console bytes, dev first:

    curl -sSLO $rel/console-linux-arm64      # verified against SHA256SUMS.txt like the CLI above
    carloku-infrastructure/scripts/roll-console.sh --artifact console-linux-arm64 --version vX.Y.Z - dev
    # wait for dev's proof: 15 minutes of the build running healthily
    carloku-infrastructure/scripts/roll-console.sh --artifact console-linux-arm64 --version vX.Y.Z - flagship

The flagship roll refuses until dev has proven those exact bytes.

• [ ] Step 2: Put the new prose through copy review

Invoke the copy-review skill on the three new passages above. Apply Paul's edits verbatim.

• [ ] Step 3: Commit (platform repo, implementation branch)

git add README.md
git commit -m "README: the release builds and rolls its console, dev first (console defences D4)"

---

Task 2: roll-console.sh: dev target, one artifact, binding check, proof carry

Files:

• Modify: ~/github.com/carlosframework/carloku-infrastructure/scripts/roll-console.sh (whole file below)
• Create the work on a branch in that repo, console-release-defences. It is a GitHub repo: commit on a branch and open a PR for Paul. Do not merge.

Interfaces:

• Consumes: ship-app.sh <profile> <bucket> <app> <ver> <channel> <kind> <artifact> (same directory, unchanged); carlos proof get --app console --sha256 <S> and carlos proof put --app console <file> (Plan 3); carlos version (prints carlos <build>).
• Produces: roll-console.sh [--artifact FILE --version V] [--bootstrap] <platform-checkout|-> dev|flagship|tito.

• [ ] Step 1: Replace the script

#!/bin/bash
# 🤖 Roll the console: build (or take) ONE artifact, ship + promote it,
# verify the running binary. Console release defences D4 (platform spec
# 2026-10-06-console-release-defences-design.md §5.2).
#
#   roll-console.sh [--artifact FILE --version V] [--bootstrap] <platform-checkout|-> <dev|flagship|tito>
#
# --artifact rolls exactly FILE (a release's public console-linux-arm64,
# checked against that release's signed SHA256SUMS.txt first), so dev and
# flagship run the bytes the public got. Without it the script builds
# ./cmd/console ONCE at the checkout's HEAD with -trimpath and keeps the file
# in $CONSOLE_BUILD_DIR (default ~/.cache/roll-console/<sha>) for the next
# target: a second build would be different bytes, and flagship's dev proof
# names bytes.
#
# flagship refuses unless dev has proven those bytes (`carlos proof get`
# against dev). --bootstrap skips that once, for the roll that installs D3.
#
# The box adopts and restarts the console itself; its settle check rolls back
# a build that does not come up. The old adopt + restart SSM steps run only on
# a box whose console.service ExecStart does not name the unit-stamped route's
# live link, where the settle check cannot judge the restart.
set -euo pipefail

artifact= version= bootstrap=0
while [ $# -gt 2 ]; do
  case $1 in
    --artifact) artifact=$2; shift 2;;
    --version) version=$2; shift 2;;
    --bootstrap) bootstrap=1; shift;;
    *) echo "unknown flag $1" >&2; exit 2;;
  esac
done
repo=${1:?usage: roll-console.sh [--artifact FILE --version V] [--bootstrap] <platform-checkout|-> <dev|flagship|tito>}
target=${2:?target: dev, flagship or tito}
scripts=$(cd "$(dirname "$0")" && pwd)

# name -> "aws-profile bucket instance-id console-origin"
coords() {
  case $1 in
    dev)      echo "kass carlos-dev-640012953448 i-05498eada6f0afd23 platform.dev.carlosframework.com";;
    flagship) echo "keymail carlos-flagship-271376211898 i-092c0c1eea75723cb platform.carlosframework.com";;
    tito)     echo "titogo carlos-tito-platform-665106592142 i-025a0b70f3d93304d carlos.tito.io";;
    *) echo "unknown deployment $1: dev, flagship or tito" >&2; exit 2;;
  esac
}
read -r profile bucket instance origin <<<"$(coords "$target")"

# A native carlos to drive ship/promote/proof. From the checkout when one is
# given, else the one on PATH (it must have `proof`, i.e. Plans 1-3).
tmp=$(mktemp -d); trap 'rm -rf "$tmp"' EXIT
if [ "$repo" != - ]; then
  if [ -n "$(git -C "$repo" status --porcelain)" ]; then
    echo "refusing to build: $repo has uncommitted changes" >&2; exit 1
  fi
  (cd "$repo" && CGO_ENABLED=0 go build -trimpath -ldflags "-s -w -X github.com/carlosframework/platform/internal/version.Build=$(git -C "$repo" rev-parse --short HEAD)" -o "$tmp/carlos" .)
  carlos=$tmp/carlos
else
  carlos=$(command -v carlos)
fi
"$carlos" proof -h >/dev/null 2>&1 || { echo "$carlos has no 'proof' verb: build from a checkout with the console release defences" >&2; exit 1; }

if [ -n "$artifact" ]; then
  [ -n "$version" ] || { echo "--artifact needs --version (the release tag)" >&2; exit 2; }
  sums=$(dirname "$artifact")/SHA256SUMS.txt
  [ -f "$sums" ] || { echo "no $sums next to $artifact: download the release's SHA256SUMS.txt and .sig and verify them first (README, Releasing the platform)" >&2; exit 1; }
  want=$(awk -v f="$(basename "$artifact")" '$2==f {print $1}' "$sums")
  [ -n "$want" ] || { echo "$sums has no line for $(basename "$artifact"): this release did not publish a console" >&2; exit 1; }
  got=$(sha256sum "$artifact" | cut -d' ' -f1)
  [ "$got" = "$want" ] || { echo "$artifact sha256 $got does not match $sums ($want)" >&2; exit 1; }
  file=$artifact
else
  [ "$repo" != - ] || { echo "no --artifact and no checkout to build from" >&2; exit 2; }
  version=$(git -C "$repo" rev-parse --short HEAD)
  dir=${CONSOLE_BUILD_DIR:-$HOME/.cache/roll-console/$version}
  file=$dir/console-linux-arm64
  if [ ! -f "$file" ]; then
    mkdir -p "$dir"
    (cd "$repo" && CGO_ENABLED=0 GOOS=linux GOARCH=arm64 go build -trimpath \
      -ldflags "-s -w -X github.com/carlosframework/platform/internal/version.Build=$version" -o "$file" ./cmd/console)
    echo "built $file (kept for the next target)"
  fi
fi
sha=$(sha256sum "$file" | cut -d' ' -f1)
echo "== console $version sha256 $sha -> $target"

# ssm_run <command>: one command on this target's console box (jq-built
# parameters; see the 2026-08-09 note in git history for why).
ssm_run() {
  local params id status
  params=$(jq -n --arg c "$1" '{commands: [$c]}')
  id=$(aws --profile "$profile" --region eu-west-1 ssm send-command --instance-ids "$instance" \
    --document-name AWS-RunShellScript --parameters "$params" --query Command.CommandId --output text)
  aws --profile "$profile" --region eu-west-1 ssm wait command-executed --command-id "$id" --instance-id "$instance" || true
  status=$(aws --profile "$profile" --region eu-west-1 ssm list-command-invocations --command-id "$id" --details \
    --query 'CommandInvocations[0].CommandPlugins[0].Status' --output text)
  aws --profile "$profile" --region eu-west-1 ssm list-command-invocations --command-id "$id" --details \
    --query 'CommandInvocations[0].CommandPlugins[0].Output' --output text
  [ "$status" = Success ]
}

# The binding: which live link console.service runs, and which route carries
# the unit stamp. Equal means the box's settle check judges the restart.
binding=$(ssm_run 'exec=$(systemctl show -p ExecStart console.service | sed -n "s/.*path=\([^ ;]*\).*/\1/p"); \
  stamped=$(sqlite3 -readonly /data/registry.db "select host from routes where unit=\"console.service\"" | tr "\n" " "); \
  echo "$exec|$stamped"')
exec_link=${binding%%|*}; stamped=${binding#*|}
bound=0
[ "$exec_link" = "/opt/carlos/live/${stamped% }" ] && bound=1
if [ $bound = 1 ]; then
  echo "   binding ok: console.service runs $exec_link, stamped on ${stamped% }"
else
  echo "   binding WRONG: console.service runs ${exec_link:-?}, unit stamp on '${stamped% }' — keeping the final restart (fix: platform spec §6 step 1)"
fi

if [ "$target" = flagship ]; then
  proof=$tmp/proof.json
  if AWS_PROFILE=kass AWS_REGION=eu-west-1 CARLOS_DEPLOYMENT_BUCKET=carlos-dev-640012953448 \
       "$carlos" proof get --app console --sha256 "$sha" >"$proof"; then
    echo "   dev proof found"
  elif [ $bootstrap = 1 ]; then
    echo "   --bootstrap: NOT checking for a dev proof (only for the roll that installs D3)"
    proof=
  else
    echo "flagship: dev has not proven $sha (above); roll it to dev and wait, or --bootstrap once" >&2
    exit 1
  fi
fi

CARLOS_BIN=$carlos "$scripts/ship-app.sh" "$profile" "$bucket" console "$version" canary/console binary "$file"
if [ "$target" = flagship ] && [ -n "${proof:-}" ]; then
  AWS_PROFILE=keymail AWS_REGION=eu-west-1 CARLOS_DEPLOYMENT_BUCKET=carlos-flagship-271376211898 \
    "$carlos" proof put --app console "$proof"
fi

if [ $bound = 0 ]; then
  ssm_run "bash -c 'set -a; source /etc/carlos/host.env; /opt/carlos/carlos adopt' 2>&1 | tail -8"
  ssm_run "systemctl restart console && sleep 2 && systemctl is-active console"
fi

echo "== $target: waiting for the running binary to be this build"
for i in $(seq 1 24); do
  exe=$(ssm_run 'readlink /proc/$(systemctl show -p MainPID --value console)/exe' || true)
  case $exe in */versions/$version/console*) break;; esac
  sleep 5
done
case $exe in
  */versions/$version/console*) ;;
  *) echo "$target: console is running ${exe:-nothing}, not versions/$version/console — check the box's adopt journal for a settle rollback" >&2; exit 1;;
esac
curl -fsS -o /dev/null "https://$origin/"
echo "== $target: console $version live at https://$origin"

Checked on the flagship console box on 2026-10-06: /usr/bin/sqlite3 is present, and ExecStart is /opt/carlos/live/platform.carlosframework.com, while carlos instances shows the console.service stamp on console.carloku.com. That is the wrong binding Review Focus 3 expects. Run ssm_run 'command -v sqlite3' against dev once as well. If dev lacks it, install it there (apt-get install -y sqlite3) rather than branch the script.

• [ ] Step 2: Static checks

Run: bash -n scripts/roll-console.sh && shellcheck -S warning scripts/roll-console.sh Expected: no output. Fix every warning except SC2016 on the single-quoted SSM command strings, which are expanded on the box on purpose.

• [ ] Step 3: Read-only check against dev and flagship

Temporarily add [ "${ROLL_DRY:-0}" = 1 ] && exit 0 immediately after the binding report. Run ROLL_DRY=1 scripts/roll-console.sh - dev and ROLL_DRY=1 scripts/roll-console.sh - flagship with the sandbox off (assumed-role profiles: [[aws-from-the-sandbox]]).

Expected:

• dev prints its binding line;
• flagship prints binding WRONG, unless someone already restamped it, which rollout step 1 does;
• nothing is shipped.

Then remove the line.

• [ ] Step 4: Commit and open the PR (carloku-infrastructure)

cd ~/github.com/carlosframework/carloku-infrastructure
git checkout -b console-release-defences
git add scripts/roll-console.sh
git commit -m "roll-console: dev target, one artifact for every target, binding check, dev proof before flagship"
git push -u origin console-release-defences
gh pr create --title "roll-console: dev first, one artifact, dev proof (console release defences D4)" \
  --body "Implements platform spec 2026-10-06-console-release-defences-design.md §5.2. Not to be run until the platform release carrying Plans 1-3 is on the boxes.

🤖 Generated with [Claude Code](https://claude.com/claude-code)"

This repo lives on GitHub, not amadan, so a PR is the right door here. Leave it for Paul to merge.

---

Task 3: Retire platform-infrastructure's copy of the script

Files:

• Modify: ~/amadan.net/carlos/platform-infrastructure/scripts/roll-console.sh: replace the body with a pointer

• [ ] Step 1: Replace the body

#!/bin/bash
# Retired 2026-10: the console roll lives in carloku-infrastructure
# (scripts/roll-console.sh), which rolls dev first and carries dev's proof
# to flagship. Platform spec 2026-10-06-console-release-defences-design.md §5.2.
echo "roll-console.sh moved to carloku-infrastructure/scripts/roll-console.sh (dev, flagship, tito)" >&2
exit 2

• [ ] Step 2: Commit on a branch in that repo

Read that repo's own instructions first (its README or AGENTS.md) for where and how changes land. Then commit: git commit -m "roll-console: retired, see carloku-infrastructure".

---

Task 4: The rollout runbook

Files:

• Modify: ~/github.com/carlosframework/carloku-infrastructure/README.md: add a section "Console release defences: rollout", on the Task 2 branch

• [ ] Step 1: Write the section

The commands below are the spec §6 steps, made concrete. Each SSM command is run with .claude/ssm.sh <profile> <region> <instance> '<cmd>' or the script's ssm_run pattern.

````markdown

Console release defences: rollout

Run once, after the platform release that carries the defences is published. In order:

1. Fix the bindings. On each console box (dev i-05498eada6f0afd23, flagship i-092c0c1eea75723cb), the one route stamped with console.service must be the host its ExecStart names:

systemctl show -p ExecStart console.service # path=/opt/carlos/live/<host> /opt/carlos/carlos route --host <that host> --unit console.service /opt/carlos/carlos route --host <any other stamped host> --unit ""

On flagship today that means stamping platform.carlosframework.com and clearing console.carloku.com.

1. Roll the boxes with carlos system update --console <origin> --version vX.Y.Z on dev and flagship, then carlos system status.
2. Dev proving. On dev's console box, as root:

/opt/carlos/carlos proof keygen --out /etc/carlos/proving.key # prints the public key cat > /etc/carlos/proof-policy.json <<'EOF' {"schema":1,"prove":{"apps":["console"],"prover":"dev","after":"15m","key_file":"/etc/carlos/proving.key"}} EOF chmod 0644 /etc/carlos/proof-policy.json

Record the public key here: dev proving key: <base64>.

1. Roll the release's console to dev: scripts/roll-console.sh --artifact console-linux-arm64 --version vX.Y.Z - dev. Watch the box journal for settled on.
2. Rehearse on dev, then delete each rehearsal release from dev's bucket (apps/<acct>/console/releases/<ver>/) and its staged copy on the box: - ship the public carlos-linux-arm64 as console rehearse-cli with a v0.24.1 carlos, direct-bucket mode (CARLOS_DEPLOYMENT_BUCKET set). Promote it, and expect the box journal to show not pointed live with no restart; - ship a console built from ./cmd/console with os.Exit(2) as the first line of main, as rehearse-exit. Expect a rollback within 30 s, a refusal, and an alert email. A plain re-promote stays refused, and carlos promote --retry rehearse-exit canary/console retries; - ship a console whose serve sleeps forever before listening, as rehearse-hang. Expect a deadline rollback that replaces the process.
3. Wait for dev's proof of step 4's build: carlos proof get --app console --sha256 <sha> against dev.
4. Roll it to flagship: scripts/roll-console.sh --artifact console-linux-arm64 --version vX.Y.Z - flagship.
5. Switch flagship's requirement on. On flagship's console box, as root:

cat > /etc/carlos/proof-policy.json <<'EOF' {"schema":1,"require":[{"app":"console","prover":"dev","key":"<dev proving key>"}]} EOF chmod 0644 /etc/carlos/proof-policy.json

The live build is unaffected. It enters the box's healthy history after 15 minutes.

Break glass: on flagship, rm /etc/carlos/proof-policy.json, then /opt/carlos/carlos adopt --stage-only. The box logs hourly that the requirement is off until the file is back.

The policy files are not in tofu user_data on purpose: a whole-root apply restarts every box. ````

--out is Plan 3 Task 9's flag; its default is /etc/carlos/proving.key.

• [ ] Step 2: Put the section through copy review

Invoke the copy-review skill on the section's prose (not the commands). Apply Paul's edits verbatim.

• [ ] Step 3: Commit on the Task 2 branch and push

git add README.md
git commit -m "README: console release defences rollout runbook"
git push

---

Task 5: Run the rollout

This task is operational and outward-facing: it changes production boxes. Confirm with Paul before each of steps 1, 7 and 8 of the runbook. Steps 2–6 touch only dev or are read-only on flagship, so they need his go-ahead once, at the start.

• [ ] Step 1: Run runbook steps 1–8 in order, pasting each command's output into the branch's amadan discussion (amadan discuss new carlos/platform -branch <impl branch> -title "Console release defences rollout").
• [ ] Step 2: After step 8, check carlos status --live against flagship shows no refusals and curl -sI https://console.carloku.com/ | grep -i x-carlos-version names vX.Y.Z.
• [ ] Step 3: Update the [[console-release-defences]] memory with what was rolled and when.