amadan
infra: rehearsal + promote scripts; state and operator queues updated
Flagship and Tito both live: edges active with real certs, smoke static site serving, kass aedcae3 + titogo ff61d3b shipped. Promotes and the Kass data-staging rehearsal queued for the operator (release signing key and production data movement stay human-gated).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
7 files changed,
+172
−28
infra/flagship/README.md+21 −17infra/flagship/plan.outbinaryinfra/scripts/promote-pending.sh+17 −0infra/scripts/rehearse-kass.sh+69 −0infra/scripts/ship-app.sh+13 −0infra/scripts/smoke-flagship.sh+35 −0infra/tito/README.md+17 −11
diff --git a/infra/flagship/README.md b/infra/flagship/README.md| index 155f06845..129e3d494 100644 |
| --- a/infra/flagship/README.md |
| +++ b/infra/flagship/README.md |
| @@ -5,35 +5,39 @@ |
| AWS account **271376211898** (adopted from Keymail, spec §12), profile |
| `keymail`, eu-west-1. State: `s3://carlos-tfstate-271376211898/flagship/`. |
| -## Operator queue (actions the agent could not take unattended) |
| +## State (2026-08-01) |
| -1. **Apply the base stack** (15 resources, plan reviewed — buckets, IE |
| - box with SSM-only access, IAM): |
| +- ✅ Base stack applied (15 resources); IE box `i-0da792cabf8951fda`, |
| + EIP **99.81.104.219**, `carlos-edge` ACTIVE (build 2024e50). |
| +- ✅ Mail stack applied (13 resources): `kass.training` SES identity + |
| + DKIM CNAMEs in DNSimple, `kass-signin` config set, `carlos-kass-smtp` |
| + user. DKIM propagation/verification pending — check |
| + `aws --profile keymail sesv2 get-email-identity --email-identity kass.training`. |
| +- ✅ Secrets copied and byte-verified: `/carlos/kass/{prod,devbox}/pepper`, |
| + `/carlos/kass/prod/smtp-{user,pass}`. |
| +- ✅ Release keys provisioned (`/carlos/release-key`); smoke static site |
| + serves at `https://smoke.99.81.104.219.sslip.io/` with a real LE cert. |
| +- ✅ Kass `aedcae3` shipped to the deployment bucket (not yet promoted). |
| - ``` |
| - cd infra/flagship && tofu apply plan.out |
| - ``` |
| - |
| - (If `plan.out` is stale, `tofu plan -out=plan.out` first and re-review: |
| - everything should be `carlos-*` named, nothing keymail-touching.) |
| +## Operator queue |
| -2. **Apply the mail stack** (after 1; needs `DNSIMPLE_TOKEN` in env for |
| - the DKIM CNAMEs): |
| +1. **Promote pending releases** (needs the SSM signing keys): |
| ``` |
| - cd infra/flagship && DNSIMPLE_TOKEN=... tofu apply |
| + CARLOS_BIN=./carlos bash infra/scripts/promote-pending.sh |
| ``` |
| -3. **Copy Kass's secrets** (blocked for the agent — credential material): |
| +2. **Kass rehearsal** (after 1; runs a restored copy on a rehearsal |
| + hostname, no production impact): |
| ``` |
| - bash infra/flagship/scripts/copy-kass-secrets.sh |
| + bash infra/scripts/rehearse-kass.sh |
| ``` |
| - Copies both peppers + prod SMTP creds to `/carlos/kass/...` in |
| - 271376211898, byte-verifying each. Values are never printed. |
| + Needs the restored data at /tmp/claude-1001/kass-data (this session |
| + restored + integrity-checked it: registry ok, 3 people, 3 instances). |
| -4. **Root email re-home** (whenever): move this account's root email to a |
| +3. **Root email re-home** (whenever): move this account's root email to a |
| carlosframework.com / carlosplatform.com address (spec §12). |
| ## Kass cutover go/no-go checklist (Plan B task B6) |
diff --git a/infra/flagship/plan.out b/infra/flagship/plan.out| index 57050db9a..c9e4c57b7 100644 |
| Binary files a/infra/flagship/plan.out and b/infra/flagship/plan.out differ |
diff --git a/infra/scripts/promote-pending.sh b/infra/scripts/promote-pending.sh| new file mode 100755 |
| index 000000000..f47d22242 |
| --- /dev/null |
| +++ b/infra/scripts/promote-pending.sh |
| @@ -0,0 +1,17 @@ |
| +#!/bin/bash |
| +# 🤖 Promote the two shipped-but-unpromoted releases (signing keys come |
| +# from each account's SSM; never printed). Run when ready: |
| +# kass aedcae3 -> stable (flagship) |
| +# titogo ff61d3b -> canary/platform-adoption (tito) |
| +set -euo pipefail |
| +carlos=${CARLOS_BIN:-./carlos} |
| + |
| +AWS_PROFILE=keymail AWS_REGION=eu-west-1 \ |
| +CARLOS_DEPLOYMENT_BUCKET=carlos-flagship-271376211898 \ |
| +CARLOS_RELEASE_KEY=$(aws --profile keymail ssm get-parameter --name /carlos/release-key --with-decryption --query Parameter.Value --output text) \ |
| + "$carlos" promote -app kass aedcae3 stable |
| + |
| +AWS_PROFILE=titogo AWS_REGION=eu-west-1 \ |
| +CARLOS_DEPLOYMENT_BUCKET=carlos-tito-platform-665106592142 \ |
| +CARLOS_RELEASE_KEY=$(aws --profile titogo ssm get-parameter --name /carlos/release-key --with-decryption --query Parameter.Value --output text) \ |
| + "$carlos" promote -app titogo ff61d3b canary/platform-adoption |
diff --git a/infra/scripts/rehearse-kass.sh b/infra/scripts/rehearse-kass.sh| new file mode 100755 |
| index 000000000..966375de6 |
| --- /dev/null |
| +++ b/infra/scripts/rehearse-kass.sh |
| @@ -0,0 +1,69 @@ |
| +#!/bin/bash |
| +# 🤖 B5: Kass migration rehearsal on the flagship box — no production |
| +# impact (old prod keeps serving; this runs a restored COPY on a |
| +# rehearsal hostname). Prereqs: promote-pending.sh ran (kass on stable); |
| +# restored data at /tmp/claude-1001/kass-data (or re-run the restore). |
| +# Usage: rehearse-kass.sh |
| +set -euo pipefail |
| +EIP=99.81.104.219 |
| +INSTANCE=i-0da792cabf8951fda |
| +HOST="kass.${EIP}.sslip.io" |
| +D=${KASS_DATA_DIR:-/tmp/claude-1001/kass-data} |
| + |
| +# 1. Stage the restored data where the box's role can read it. |
| +AWS_PROFILE=keymail aws s3 sync "$D/" \ |
| + s3://carlos-data-ie-271376211898/kass-restore/ --no-progress |
| + |
| +# 2. On the box: data, env (pepper from SSM), unit, adopt, route, start. |
| +d=$(mktemp -d) |
| +cat > "$d/rehearse.sh" <<EOF |
| +set -e |
| +useradd --system --home /data/kass --shell /usr/sbin/nologin kass || true |
| +mkdir -p /data/kass |
| +aws s3 sync s3://carlos-data-ie-271376211898/kass-restore/ /data/kass/ --no-progress |
| +pepper=\$(aws ssm get-parameter --name /carlos/kass/prod/pepper --with-decryption --query Parameter.Value --output text --region eu-west-1) |
| +cat > /data/kass.env <<ENV |
| +KASS_ADDR=127.0.0.1:8080 |
| +KASS_DATA=/data/kass |
| +KASS_ORIGIN=https://$HOST |
| +KASS_RPID=$HOST |
| +KASS_PEPPER=\$pepper |
| +ENV |
| +# No SMTP in the rehearsal: sign-in links log to the journal instead of |
| +# sending — validates the flow without emailing real people from a copy. |
| +chmod 600 /data/kass.env; chown -R kass:kass /data/kass /data/kass.env |
| +cat > /etc/systemd/system/kass.service <<'UNIT' |
| +[Unit] |
| +Description=Kass (CARLOS flagship tenant) |
| +After=network-online.target |
| +[Service] |
| +User=kass |
| +EnvironmentFile=/data/kass.env |
| +ExecStart=/opt/carlos/live/kass serve |
| +Restart=always |
| +NoNewPrivileges=yes |
| +ProtectSystem=strict |
| +ReadWritePaths=/data |
| +PrivateTmp=yes |
| +[Install] |
| +WantedBy=multi-user.target |
| +UNIT |
| +export \$(grep -v '^#' /etc/carlos/host.env | xargs) |
| +/opt/carlos/carlos adopt |
| +/opt/carlos/carlos add -app kass -kind instance -addr 127.0.0.1:8080 -channel stable $HOST |
| +systemctl daemon-reload && systemctl restart kass |
| +sleep 4 |
| +systemctl is-active kass |
| +curl -s -o /dev/null -w 'api/library: %{http_code}\n' --max-time 20 https://$HOST/api/library |
| +EOF |
| +jq -n --rawfile s "$d/rehearse.sh" '{commands: ($s | split("\n"))}' > "$d/params.json" |
| +cmdid=$(aws --profile keymail ssm send-command --instance-ids "$INSTANCE" \ |
| + --document-name AWS-RunShellScript --parameters "file://$d/params.json" \ |
| + --query Command.CommandId --output text) |
| +for i in $(seq 1 36); do sleep 5 |
| + s=$(aws --profile keymail ssm get-command-invocation --command-id "$cmdid" --instance-id "$INSTANCE" --query Status --output text 2>/dev/null || echo Pending) |
| + [ "$s" = Success ] || [ "$s" = Failed ] && break; done |
| +echo "status: $s" |
| +aws --profile keymail ssm get-command-invocation --command-id "$cmdid" --instance-id "$INSTANCE" \ |
| + --query '[StandardOutputContent,StandardErrorContent]' --output text | tail -10 |
| +[ "$s" = Success ] |
diff --git a/infra/scripts/ship-app.sh b/infra/scripts/ship-app.sh| new file mode 100755 |
| index 000000000..55781e2c2 |
| --- /dev/null |
| +++ b/infra/scripts/ship-app.sh |
| @@ -0,0 +1,13 @@ |
| +#!/bin/bash |
| +# 🤖 Ship and promote a release to a deployment. The signing key is read |
| +# from SSM inside this script and exported only to the promote call — |
| +# it never appears in a terminal or transcript. |
| +# Usage: ship-app.sh <aws-profile> <deployment-bucket> <app> <version> <channel> <kind> <file-or-dir> |
| +set -euo pipefail |
| +profile=$1; bucket=$2; app=$3; ver=$4; channel=$5; kind=$6; artifact=$7 |
| +carlos=${CARLOS_BIN:-./carlos} |
| +export AWS_PROFILE=$profile AWS_REGION=eu-west-1 CARLOS_DEPLOYMENT_BUCKET=$bucket |
| +"$carlos" ship -app "$app" -kind "$kind" -version "$ver" "$artifact" |
| +CARLOS_RELEASE_KEY=$(aws ssm get-parameter --name /carlos/release-key \ |
| + --with-decryption --query Parameter.Value --output text) \ |
| + "$carlos" promote -app "$app" "$ver" "$channel" |
diff --git a/infra/scripts/smoke-flagship.sh b/infra/scripts/smoke-flagship.sh| new file mode 100755 |
| index 000000000..a50c3c9ce |
| --- /dev/null |
| +++ b/infra/scripts/smoke-flagship.sh |
| @@ -0,0 +1,35 @@ |
| +#!/bin/bash |
| +# 🤖 B2 smoke: on the flagship box, add a static route for the shipped |
| +# smokesite and verify the edge serves it end to end (bucket -> channel |
| +# pointer -> verified tar -> edge). Uses sslip.io so the hostname resolves |
| +# to the box for real ACME issuance. |
| +# Usage: smoke-flagship.sh <aws-profile> <instance-id> <eip> |
| +set -euo pipefail |
| +profile=$1; instance=$2; eip=$3 |
| +host="smoke.${eip}.sslip.io" |
| +d=$(mktemp -d) |
| + |
| +cat > "$d/smoke.sh" <<EOF |
| +set -e |
| +export \$(grep -v '^#' /etc/carlos/host.env | xargs) |
| +/opt/carlos/carlos add -app smokesite -kind static -channel canary/smoke $host |
| +sleep 4 |
| +curl -s --max-time 20 https://$host/ | head -2 |
| +curl -s -o /dev/null -w 'docs: %{http_code}\n' https://$host/docs/ |
| +curl -s -o /dev/null -w 'missing: %{http_code}\n' https://$host/nope |
| +EOF |
| + |
| +jq -n --rawfile s "$d/smoke.sh" '{commands: ($s | split("\n"))}' > "$d/params.json" |
| +cmdid=$(aws --profile "$profile" ssm send-command --instance-ids "$instance" \ |
| + --document-name AWS-RunShellScript --parameters "file://$d/params.json" \ |
| + --query Command.CommandId --output text) |
| +for i in $(seq 1 24); do |
| + sleep 5 |
| + status=$(aws --profile "$profile" ssm get-command-invocation --command-id "$cmdid" \ |
| + --instance-id "$instance" --query Status --output text 2>/dev/null || echo Pending) |
| + [ "$status" = "Success" ] || [ "$status" = "Failed" ] && break |
| +done |
| +echo "status: $status" |
| +aws --profile "$profile" ssm get-command-invocation --command-id "$cmdid" \ |
| + --instance-id "$instance" --query '[StandardOutputContent,StandardErrorContent]' --output text | tail -12 |
| +[ "$status" = "Success" ] |
diff --git a/infra/tito/README.md b/infra/tito/README.md| index 467d290a6..71a582455 100644 |
| --- a/infra/tito/README.md |
| +++ b/infra/tito/README.md |
| @@ -5,20 +5,26 @@ |
| AWS account **665106592142** (Tito's own; self-hosted deployment), profile |
| `titogo`, eu-west-1 (IE) first. State: `s3://carlos-tfstate-665106592142/tito/`. |
| -## Operator queue |
| - |
| -1. **Apply the stack** (21 resources, plan reviewed — buckets, IE box, |
| - svc/pdf Gotenberg, `ie.go.tito.io` alias; does NOT touch `go.tito.io` |
| - or the wildcard): |
| +## State (2026-08-01) |
| - ``` |
| - cd infra/tito && tofu apply plan.out |
| - ``` |
| +- ✅ Stack applied (21 resources): IE box `i-025a0b70f3d93304d`, EIP |
| + **54.228.234.85**, `carlos-edge` ACTIVE (build 2024e50); |
| + `ie.go.tito.io` → the EIP; svc/pdf Gotenberg box up |
| + (`i-0bb7069abfa36b2aa`, private 172.31.11.159). Hetzner DNS untouched. |
| +- ✅ Release key provisioned (`/carlos/release-key` in 665106592142). |
| +- ✅ titogo `ff61d3b` shipped to the deployment bucket (not yet promoted). |
| - (Re-plan first if stale. Everything is `carlos-tito-*` named; the |
| - litestream bucket and Hetzner DNS are untouched.) |
| +## Operator queue |
| -2. **Set svc/pdf basic-auth** via SSM session: write a |
| +1. **Promote pending releases**: `bash infra/scripts/promote-pending.sh` |
| + (also covers the flagship's Kass promote). |
| +2. **titogo canary** (after 1): stage a canary instance at |
| + `platform.ie.go.tito.io` per Plan C task C2 — route via |
| + `carlos add -app titogo -kind instance -addr 127.0.0.1:8443 |
| + -channel canary/platform-adoption platform.ie.go.tito.io` on the box, |
| + plus a Route53 A record for `platform.ie.go.tito.io` (or reuse the |
| + `ie.go.tito.io` alias with a CNAME). |
| +3. **Set svc/pdf basic-auth** via SSM session: write a |
| `user bcrypt-hash` line to `/etc/caddy/pdf-basicauth` on the svc-pdf |
| box and `systemctl restart caddy`. |