amadan
deploy: dev plan guard stops on the destroy COUNT and applies the saved plan (#1303)
The wording grep missed a live AMI replacement on 2026-07-23 (bare approve prompt, Dublin box destroyed). The guard now refuses when the plan summary counts any destroys (phrasing-independent), plans with -no-color, and apply executes the exact saved plan the guard inspected behind its own explicit confirm.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
1 file changed,
+42
−19
deploy/ops-dev.sh+42 −19
diff --git a/deploy/ops-dev.sh b/deploy/ops-dev.sh| index 2799331..10210b4 100755 |
| --- a/deploy/ops-dev.sh |
| +++ b/deploy/ops-dev.sh |
| @@ -84,32 +84,47 @@ usage() { |
| exit 2 |
| } |
| -# plan_guard [tofu args...]: run a plan, show it, and hard-stop when it would |
| -# REPLACE a resource. Shared by `plan` (whose whole job this is) and `apply` |
| -# (so an interactive "yes" can't sleepwalk into the AMI rebuild). Extra args |
| -# pass through, so a targeted apply guards exactly the graph it will touch. |
| +# plan_guard <planfile|-> [tofu args...]: run a plan, show it, and hard-stop |
| +# when it would DESTROY anything. Shared by `plan` (whose whole job this is) |
| +# and `apply` (so an interactive "yes" can't sleepwalk into the AMI rebuild). |
| +# Extra args pass through, so a targeted apply guards exactly the graph it |
| +# will touch. With a planfile (not "-"), the plan is also SAVED so apply can |
| +# execute exactly what was inspected — nothing can change between the guarded |
| +# plan and the applied one. |
| +# |
| +# The stop condition is the plan summary's own destroy COUNT, not resource- |
| +# header wording: the original grep for "must be replaced" demonstrably missed |
| +# a live AMI replacement on 2026-07-23 (the operator got a bare approve prompt |
| +# and the Dublin box was destroyed), and a count can't be missed by phrasing |
| +# or formatting drift. The wording grep stays as a second net. |
| plan_guard() { |
| + pf="$1"; shift |
| planout="$(mktemp -t lchat-dev-plan.XXXXXX)" |
| + outflag="" |
| + [ "$pf" = - ] || outflag="-out=$pf" |
| # -detailed-exitcode: 0 no changes, 2 changes present, 1 error. Changes are |
| - # fine; what we refuse to shrug at is a pending instance REPLACEMENT. |
| + # fine; what we refuse to shrug at is anything pending DESTRUCTION. |
| + # -no-color so the detection greps see the same bytes a human does. |
| # (Captured to a file, not piped through tee — a pipeline would report tee's |
| # exit status and silently lose tofu's.) |
| rc=0 |
| - run_tofu plan -input=false -detailed-exitcode "$@" >"$planout" 2>&1 || rc=$? |
| + # shellcheck disable=SC2086 # outflag is deliberately word-split (empty or one flag) |
| + run_tofu plan -input=false -no-color -detailed-exitcode $outflag "$@" >"$planout" 2>&1 || rc=$? |
| cat "$planout" |
| if [ "$rc" -eq 1 ]; then rm -f "$planout"; exit 1; fi |
| - if grep -q "must be replaced" "$planout"; then |
| + destroys="$(sed -n 's/^Plan: .* \([0-9][0-9]*\) to destroy.*/\1/p' "$planout" | tail -1)" |
| + if [ "${destroys:-0}" -gt 0 ] || grep -Eq "must be replaced|forces replacement" "$planout"; then |
| rm -f "$planout" |
| cat >&2 <<'EOF' |
| - ██ STOP — this plan REPLACES a resource (see "must be replaced" above). |
| - If it is aws_instance.ie_node, the cause is usually a newer Canonical AMI |
| - (compute.tf pins nothing: most_recent = true, no lifecycle ignore_changes). |
| - A plain apply would DESTROY the Dublin box — its /data dies with it; only |
| + ██ STOP — this plan DESTROYS or REPLACES something (see the plan above). |
| + If it is aws_instance.ie_node, the cause is usually a newer Canonical AMI — |
| + a plain apply would DESTROY the Dublin box; its /data dies with it, only |
| the EIP (and DNS) survive. Your options: |
| - apply just the safe resource: ops dev apply -target=<addr> |
| - make the rebuild deliberate: ops dev rebuild (new keys, guided) |
| - - or add lifecycle { ignore_changes = [ami] } to the instance and re-plan |
| + - a genuinely wanted destroy: run tofu by hand in infra/dev-mesh, |
| + eyes open — this verb will not do it for you. |
| EOF |
| exit 1 |
| fi |
| @@ -332,19 +347,27 @@ plan) |
| cd "$root/infra/dev-mesh" |
| ensure_creds |
| [ -d .terraform ] || run_tofu init -input=false |
| - plan_guard "$@" |
| + plan_guard - "$@" |
| ;; |
| apply) |
| - # The self-serve apply: same credential fetch, same replacement guard — |
| - # tofu's own interactive approve remains the final gate. Extra args pass |
| - # through (e.g. `ops dev apply -target=aws_security_group.ie_node` to land |
| - # a security-group change while an AMI replacement is pending). |
| + # The self-serve apply: same credential fetch, same destroy guard. What |
| + # applies is the SAVED plan the guard inspected — never a fresh one — so a |
| + # refresh between guard and apply can't smuggle a replacement in. The verb |
| + # keeps its own explicit confirm (a saved-plan apply skips tofu's). Extra |
| + # args pass through (e.g. `ops dev apply |
| + # -target=aws_security_group.ie_node` to land a security-group change |
| + # while something destructive is pending). |
| cd "$root/infra/dev-mesh" |
| ensure_creds |
| [ -d .terraform ] || run_tofu init -input=false |
| - plan_guard "$@" |
| - run_tofu apply "$@" |
| + planfile="$(mktemp -t lchat-dev-planfile.XXXXXX)" |
| + trap 'rm -f "$planfile"' EXIT |
| + plan_guard "$planfile" "$@" |
| + printf 'apply exactly this plan? (type yes): ' |
| + read -r answer || answer="" # EOF (no tty) refuses below instead of dying silently |
| + [ "$answer" = yes ] || { echo "ops dev: not applied"; exit 1; } |
| + run_tofu apply -input=false "$planfile" |
| ;; |
| deploy) |