eleven / messenger Public

deploy+infra: pin the dev-mesh AMI, teach rebuild --adopt, encode the dev home stand-up (#1301)

An AMI-drift apply destroyed+recreated the Dublin box today (/data lost).
lifecycle ignore_changes=[ami] makes that impossible via routine applies
(-replace still works for deliberate rebuilds); rebuild --adopt provisions a
box something else already replaced (no tofu, no creds needed); the dev home
server + control.env extras are now encoded in the verb per its own note.

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>

Paul Campbell 3 months ago 74a32997fc569e4cc0889343c0d8de3605986a99 parent 3170117

2 files changed, +90 −16

deploy/ops-dev.sh +83 −16
infra/dev-mesh/compute.tf +7 −0

diff --git a/deploy/ops-dev.sh b/deploy/ops-dev.sh

index 585b046..2799331 100755

--- a/deploy/ops-dev.sh

+++ b/deploy/ops-dev.sh

@@ -8,6 +8,8 @@

# ./deploy/ops dev deploy --home # ALSO swap the home service binary (lchat-home)

# ./deploy/ops dev provision [ie|fi] # refresh systemd units + route the short-link apex

# ./deploy/ops dev rebuild --yes # destroy + recreate the Dublin box (new keys)

+# ./deploy/ops dev rebuild --yes --adopt # provision a Dublin box something ELSE already

+# # replaced (no tofu touch) — same bootstrap

# What this encodes that prose used to carry:

# - The dev mesh is NOT a release channel. Nothing here touches the signed

@@ -539,8 +541,8 @@ rebuild)

# replace the instance, and it boots trusting only you. The EIP, S3 bucket

# (hibernated spaces, mesh membership, the id ledger), IAM, and the Route53

# zone are separate resources and survive; the box's /data does not.

- confirm=no

- for a in "$@"; do case "$a" in --yes) confirm=yes ;; *) usage ;; esac; done

+ confirm=no; adopt=no

+ for a in "$@"; do case "$a" in --yes) confirm=yes ;; --adopt) adopt=yes ;; *) usage ;; esac; done

ct="$root/infra/dev-mesh/compute.tf"

# A fresh box trusts exactly one key — make sure it will be one you hold.

@@ -586,6 +588,8 @@ ops dev rebuild — this DESTROYS and recreates the Dublin box.

spaces, mesh membership, the id ledger), IAM, the Route53 zone

needs: LCHAT_MESH_TOKEN re-paired with Helsinki afterwards, and the env

files rebuilt (the runbook's Step 3 blocks)

+With --adopt: no destroy — the box was ALREADY replaced (an AMI-drift apply,

+a console rebuild); this just runs the same provisioning against it.

Coordinate first — this tears down the live half of the dev mesh until

re-provisioned. Re-run with --yes to proceed (tofu will still show its plan

and ask for approval).

@@ -593,16 +597,29 @@ EOF

exit 1

- cd "$root/infra/dev-mesh"

- ensure_creds

- [ -d .terraform ] || run_tofu init -input=false

- # tofu's own interactive approve is the second gate — read the plan: it

- # should replace aws_instance.ie_node (and the key pair), nothing else

- # destructive.

- run_tofu apply -replace=aws_instance.ie_node

- ip="$(run_tofu output -raw ie_node_ip)"

- bucket="$(run_tofu output -raw mesh_bucket)"

+ if [ "$adopt" = yes ]; then

+ # Adopt: the instance was already replaced outside this verb, so there is

+ # nothing for tofu to do — which also means no AWS/DNSimple credentials

+ # are needed. The facts tofu would have output come from elsewhere: the

+ # EIP survives replacement (it IS the committed BOX_IE address), and the

+ # bucket is shared with Helsinki, which survived and knows it.

+ ip="${BOX_IE#*@}"

+ bucket="$(ssh -o ConnectTimeout=10 "$BOX_FI" \

+ "sudo grep '^LCHAT_HIBERNATE_BUCKET=' /data/router.env | cut -d= -f2-" 2>/dev/null || true)"

+ [ -n "$bucket" ] || { echo "ops dev: couldn't read the mesh bucket from Helsinki's router.env — is $BOX_FI up?" >&2; exit 1; }

+ else

+ cd "$root/infra/dev-mesh"

+ ensure_creds

+ [ -d .terraform ] || run_tofu init -input=false

+ # tofu's own interactive approve is the second gate — read the plan: it

+ # should replace aws_instance.ie_node (and the key pair), nothing else

+ # destructive. (-replace overrides the instance's ignore_changes=[ami],

+ # so pinning the AMI does not break deliberate rebuilds.)

+ run_tofu apply -replace=aws_instance.ie_node

+ ip="$(run_tofu output -raw ie_node_ip)"

+ bucket="$(run_tofu output -raw mesh_bucket)"

+ fi

# Same EIP, new host key — the stale known_hosts entry would make every ssh

# fail with "REMOTE HOST IDENTIFICATION HAS CHANGED", so drop it first.

ssh-keygen -R "$ip" >/dev/null 2>&1 || true

@@ -669,14 +686,64 @@ EOF

|| echo "ops dev: WARNING: mesh-seed failed — rerun it on Helsinki when the router is up" >&2

+ # controld extras: without these a rebuilt Dublin silently provisions

+ # non-hibernating, uncapped spaces — the opposite of the dev mesh's point.

+ echo "ops dev: writing /data/control.env"

+ {

+ printf 'LCHAT_CONTROL_HIBERNATE_CREATES=1\n'

+ printf 'LCHAT_MAX_INSTANCES=40\n'

+ printf 'LCHAT_HIBERNATE_BUCKET=%s\n' "$bucket"

+ printf 'LCHAT_HIBERNATE_REGION=eu-west-1\n'

+ } | ssh_ie "ubuntu@$ip" 'sudo tee /data/control.env >/dev/null && sudo chmod 600 /data/control.env && sudo systemctl restart lchat-control'

+ # The dev home server (encoded here per the old completion note). bootstrap

+ # already wrote its unit; home is its own binary. The env carries the

+ # WebAuthn RP origin — it MUST stay home.dev.elevenmessenger.com or every

+ # previously-enrolled dev passkey orphans — and one caller token per region,

+ # minted from each cloud's admin API over its local socket. Token values

+ # move through pipes and shell variables only; nothing prints them.

+ echo "ops dev: building + standing up the dev home server"

+ GOOS=linux GOARCH=arm64 CGO_ENABLED=0 go build -o "$tmp/home" ./home

+ scp_ie -q "$tmp/home" "ubuntu@$ip:/tmp/home"

+ ssh_ie "ubuntu@$ip" 'sudo mv /tmp/home /opt/lchat/home && sudo chmod +x /opt/lchat/home && sudo chown root:root /opt/lchat/home'

+ tok_of() { sed -n 's/.*"token":"$[^"]*$".*/\1/p'; }

+ ie_caller="$(ssh_ie "ubuntu@$ip" 'sudo sh' <<EOS | tok_of

+curl -s --unix-socket /run/lchat/cloud.sock -X POST http://cloud/api/admin/tokens \

+ -H "Authorization: Bearer $admin_token" -d '{"label":"home.dev"}'

+EOS

+)"

+ fi_caller="$(ssh -o ConnectTimeout=10 "$BOX_FI" 'sudo sh' <<'EOS' | tok_of

+ADM="$(grep '^LCHAT_ADMIN_TOKEN=' /data/cloud.env | cut -d= -f2-)"

+curl -s --unix-socket /run/lchat/cloud.sock -X POST http://cloud/api/admin/tokens \

+ -H "Authorization: Bearer $ADM" -d '{"label":"home.dev"}'

+EOS

+)"

+ [ -n "$ie_caller" ] || echo "ops dev: WARNING: no ie caller token minted — home's ie region will 502 until one is set in /data/home.env" >&2

+ [ -n "$fi_caller" ] || echo "ops dev: WARNING: no fi caller token minted — home's fi region will 502 until one is set in /data/home.env" >&2

+ {

+ printf 'LCHAT_HOME_ORIGIN=https://home.dev.elevenmessenger.com\n'

+ printf 'LCHAT_CLOUD_REGIONS=[{"code":"ie","name":"Ireland","url":"https://cloud.ie.dev.11ch.at","token":"%s"},{"code":"fi","name":"Finland","url":"https://cloud.fi.dev.11ch.at","token":"%s"}]\n' "$ie_caller" "$fi_caller"

+ printf 'LCHAT_CLOUD_DEFAULT_REGION=ie\n'

+ } | ssh_ie "ubuntu@$ip" 'sudo tee /data/home.env >/dev/null && sudo chmod 600 /data/home.env'

+ ssh_ie "ubuntu@$ip" 'sudo systemctl enable --now lchat-home'

+ # Service routes are local registry rows mesh-seed doesn't carry; re-adding

+ # an existing row fails harmlessly on a re-run.

+ ssh_ie "ubuntu@$ip" 'sudo /opt/lchat/lchat add-service home.dev.elevenmessenger.com --socket /run/lchat/home.sock' \

+ || echo "ops dev: (home route already present)"

+ ssh_ie "ubuntu@$ip" 'sudo /opt/lchat/lchat add-service cloud.ie.dev.11ch.at --socket /run/lchat/cloud.sock' \

+ || echo "ops dev: (cloud route already present)"

cat <<EOF

Dublin rebuilt on $ip (same EIP — DNS untouched), running $ver, trusting only

-your key: env files written, units restarted, routes seeded from Helsinki.

-Not yet encoded in this verb — EXTEND IT here when next needed, don't reach

-for a runbook: the dev home server (/data/home.env + its unit) and the

-cloud.env demo/pool extras.

+your key: env files written (router/cloud/control/home), units restarted,

+routes seeded from Helsinki, home + cloud service routes added.

+Still not encoded here — EXTEND THIS VERB when next needed, don't reach for a

+runbook: the dev connect broker (lchat-connect-dev needs the vault's sandbox

+Stripe pair) and the cloud.env demo/pool/billing extras.

verify: ssh ubuntu@$ip journalctl -u lchat-router -n 40 --no-pager | grep -i wildcard

+ curl -s https://home.dev.elevenmessenger.com/api/features

then: land the compute.tf PR if it changed for this rebuild, and

./deploy/ops ledger "rebuilt the dev-mesh Dublin box"

EOF

diff --git a/infra/dev-mesh/compute.tf b/infra/dev-mesh/compute.tf

index 4885db9..a9bfa64 100644

--- a/infra/dev-mesh/compute.tf

+++ b/infra/dev-mesh/compute.tf

@@ -108,6 +108,13 @@ resource "aws_instance" "ie_node" {

encrypted = true

}

+ # A newer Canonical AMI must never replace this box via a routine apply —

+ # that destroys /data (it did, 2026-07-23). Deliberate rebuilds still work:

+ # `ops dev rebuild` uses -replace, which overrides ignore_changes.

+ lifecycle {

+ ignore_changes = [ami]

+ }

tags = { Name = "eleven-dev-mesh-ie" }

}