carlos / platform Public

docs: draft the one-off apps/bab// cleanup for the dev bucket

Dry run by default. --apply deletes each current object under exactly
apps/bab// in carlos-dev-640012953448 by exact key, one delete-object
call per key (leaving delete markers); --apply --versions deletes every
listed version and delete marker by version id, one call each. Fixed
bucket, prefix, profile and region; the account is checked, every key
is checked against the prefix, no --recursive and no prefix delete.
Counts before and after. Not run.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

claude 1 week ago 90ae63b0c50b699e900f8f6304945ae679c6f8cc parent 42f6098

1 file changed, +145 −0

docs/superpowers/notes/2026-10-16-empty-app-key-cleanup.bash +145 −0

CI passed — run details

CI log

=== gate ===
cd launchctx && go vet ./... && go test ./... -count=1
ok  	amadan.net/carlos/platform.git/launchctx	0.011s
?   	amadan.net/carlos/platform.git/launchctx/controlname	[no test files]
?   	amadan.net/carlos/platform.git/launchctx/wireid	[no test files]
go vet ./...
gofmt -l .: clean
go test ./... -count=1 -timeout 30m
ok  	github.com/carlosframework/platform	300.698s
?   	github.com/carlosframework/platform/cmd/changelog	[no test files]
ok  	github.com/carlosframework/platform/cmd/console	2.567s
ok  	github.com/carlosframework/platform/cmd/moveprobe	92.265s
ok  	github.com/carlosframework/platform/cmd/zddprobe	0.703s
ok  	github.com/carlosframework/platform/internal/activator	249.808s
ok  	github.com/carlosframework/platform/internal/agentsock	0.016s
ok  	github.com/carlosframework/platform/internal/alerting	0.003s
ok  	github.com/carlosframework/platform/internal/analytics	0.018s
ok  	github.com/carlosframework/platform/internal/appconfig	2.440s
ok  	github.com/carlosframework/platform/internal/apperr	0.075s
ok  	github.com/carlosframework/platform/internal/applog	0.124s
ok  	github.com/carlosframework/platform/internal/awsinv	0.004s
ok  	github.com/carlosframework/platform/internal/boxchoice	0.004s
ok  	github.com/carlosframework/platform/internal/boxid	0.948s
?   	github.com/carlosframework/platform/internal/boxid/boxidtest	[no test files]
?   	github.com/carlosframework/platform/internal/boxid/internal/signtest	[no test files]
ok  	github.com/carlosframework/platform/internal/boxvolumes	0.008s
ok  	github.com/carlosframework/platform/internal/caps	0.011s
ok  	github.com/carlosframework/platform/internal/certwarm	0.002s
ok  	github.com/carlosframework/platform/internal/cgroup	0.696s
ok  	github.com/carlosframework/platform/internal/changelog	0.310s
ok  	github.com/carlosframework/platform/internal/clamd	0.055s
?   	github.com/carlosframework/platform/internal/cliflag	[no test files]
ok  	github.com/carlosframework/platform/internal/console	394.061s
?   	github.com/carlosframework/platform/internal/controlname	[no test files]
?   	github.com/carlosframework/platform/internal/controlsock	[no test files]
ok  	github.com/carlosframework/platform/internal/dbdoor	6.040s
ok  	github.com/carlosframework/platform/internal/dbdoor/dbdoortest	0.009s
ok  	github.com/carlosframework/platform/internal/dbverify	0.004s
ok  	github.com/carlosframework/platform/internal/dnsrec	0.002s
ok  	github.com/carlosframework/platform/internal/docsite	0.005s
ok  	github.com/carlosframework/platform/internal/domains	2.379s
ok  	github.com/carlosframework/platform/internal/doorkey	0.006s
ok  	github.com/carlosframework/platform/internal/economics	0.003s
ok  	github.com/carlosframework/platform/internal/edge	26.549s
?   	github.com/carlosframework/platform/internal/edgeproof	[no test files]
ok  	github.com/carlosframework/platform/internal/edgesfile	0.007s
ok  	github.com/carlosframework/platform/internal/fleetapi	0.129s
ok  	github.com/carlosframework/platform/internal/gate	1.655s
ok  	github.com/carlosframework/platform/internal/genselect	0.014s
ok  	github.com/carlosframework/platform/internal/geo	0.016s
ok  	github.com/carlosframework/platform/internal/geoip	0.029s
ok  	github.com/carlosframework/platform/internal/host	161.738s
ok  	github.com/carlosframework/platform/internal/hostactors	0.368s
ok  	github.com/carlosframework/platform/internal/hostbox	2.493s
ok  	github.com/carlosframework/platform/internal/hostclaim	0.510s
ok  	github.com/carlosframework/platform/internal/hostevidence	0.013s
ok  	github.com/carlosframework/platform/internal/hostexec	4.869s
ok  	github.com/carlosframework/platform/internal/hostfence	0.032s
ok  	github.com/carlosframework/platform/internal/hostfiles	10.529s
ok  	github.com/carlosframework/platform/internal/hostfloor	0.012s
ok  	github.com/carlosframework/platform/internal/hostguard	0.966s
ok  	github.com/carlosframework/platform/internal/hostops	1.598s
?   	github.com/carlosframework/platform/internal/hostops/fencerelease	[no test files]
ok  	github.com/carlosframework/platform/internal/hostops/internal/fencerelease	0.013s
ok  	github.com/carlosframework/platform/internal/hostquarantine	0.013s
ok  	github.com/carlosframework/platform/internal/hostrelocate	0.433s
ok  	github.com/carlosframework/platform/internal/hostresolve	0.130s
ok  	github.com/carlosframework/platform/internal/hostwrite	0.013s
ok  	github.com/carlosframework/platform/internal/infraview	0.012s
ok  	github.com/carlosframework/platform/internal/instancectl	4.175s
ok  	github.com/carlosframework/platform/internal/instanceexec	2.715s
ok  	github.com/carlosframework/platform/internal/instancefs	0.012s
ok  	github.com/carlosframework/platform/internal/instancekey	0.014s
ok  	github.com/carlosframework/platform/internal/instanceupdates	0.013s
ok  	github.com/carlosframework/platform/internal/layout	1.497s
ok  	github.com/carlosframework/platform/internal/lease	9.803s
ok  	github.com/carlosframework/platform/internal/lease/leasetest	0.018s
ok  	github.com/carlosframework/platform/internal/ledger	0.011s
ok  	github.com/carlosframework/platform/internal/mail	0.175s
ok  	github.com/carlosframework/platform/internal/maintenance	0.084s
ok  	github.com/carlosframework/platform/internal/maintenanceexec	5.501s
ok  	github.com/carlosframework/platform/internal/maintenanceguard	0.077s
ok  	github.com/carlosframework/platform/internal/metrics	0.005s
ok  	github.com/carlosframework/platform/internal/moves	20.970s
ok  	github.com/carlosframework/platform/internal/moves/archive	0.773s
ok  	github.com/carlosframework/platform/internal/objstore	0.174s
ok  	github.com/carlosframework/platform/internal/ops	0.191s
ok  	github.com/carlosframework/platform/internal/park	0.204s
ok  	github.com/carlosframework/platform/internal/privatefile	0.004s
ok  	github.com/carlosframework/platform/internal/provision	16.893s
ok  	github.com/carlosframework/platform/internal/publicrelease	0.006s
?   	github.com/carlosframework/platform/internal/regionname	[no test files]
ok  	github.com/carlosframework/platform/internal/registry	4.906s
ok  	github.com/carlosframework/platform/internal/relay	7.233s
ok  	github.com/carlosframework/platform/internal/release	20.097s
ok  	github.com/carlosframework/platform/internal/rollout	3.798s
ok  	github.com/carlosframework/platform/internal/rolloutwords	0.003s
ok  	github.com/carlosframework/platform/internal/rootaws	0.006s
ok  	github.com/carlosframework/platform/internal/rootparticipation	0.007s
ok  	github.com/carlosframework/platform/internal/rootstate	0.024s
ok  	github.com/carlosframework/platform/internal/runs	0.573s
ok  	github.com/carlosframework/platform/internal/s3compat	0.005s
ok  	github.com/carlosframework/platform/internal/safefile	0.004s
ok  	github.com/carlosframework/platform/internal/schedule	2.484s
ok  	github.com/carlosframework/platform/internal/seeds	0.012s
ok  	github.com/carlosframework/platform/internal/sim/hostlife	9.423s
ok  	github.com/carlosframework/platform/internal/sim/litestreamfacts	4.106s
ok  	github.com/carlosframework/platform/internal/status	0.039s
ok  	github.com/carlosframework/platform/internal/steering	0.005s
ok  	github.com/carlosframework/platform/internal/stopevidence	0.169s
ok  	github.com/carlosframework/platform/internal/svcrole	0.002s
ok  	github.com/carlosframework/platform/internal/telemetry	0.004s
ok  	github.com/carlosframework/platform/internal/termfence	0.004s
ok  	github.com/carlosframework/platform/internal/tunnelclient	0.004s
ok  	github.com/carlosframework/platform/internal/unixdial	0.003s
ok  	github.com/carlosframework/platform/internal/updatewords	0.006s
ok  	github.com/carlosframework/platform/internal/usage	0.005s
ok  	github.com/carlosframework/platform/internal/version	0.002s
ok  	github.com/carlosframework/platform/internal/vet	1.557s
=== cross ===
--- shard 0 of 5 ---
cross-compiling darwin/arm64
--- shard 1 of 5 ---
cross-compiling darwin/amd64
--- shard 2 of 5 ---
cross-compiling linux/arm64
linux/arm64 carlos binary: 47120546 bytes (ceiling 52000000)
--- shard 3 of 5 ---
cross-compiling windows/amd64
--- shard 4 of 5 ---
cross-compiling windows/arm64
=== minio-contract ===
minio contract: arm64, minio RELEASE.2025-09-07T16-13-09Z, mc RELEASE.2025-08-13T08-35-41Z
  minio: downloading https://github.com/minio/minio/releases/download/RELEASE.2025-09-07T16-13-09Z/minio.linux-arm64.RELEASE.2025-09-07T16-13-09Z
  minio: downloaded, digest verified
  mc: downloading https://github.com/minio/mc/releases/download/RELEASE.2025-08-13T08-35-41Z/mc.linux-arm64.RELEASE.2025-08-13T08-35-41Z
  mc: downloaded, digest verified
  minio healthy on 127.0.0.1:20999
Added `local` successfully.
Bucket created successfully `local/contract-bucket`.
Bucket created successfully `local/denied-bucket`.
Created policy `contract-only` successfully.
Added user `scopeduser` successfully.
Attached Policies: [contract-only]
To User: scopeduser
  buckets and scoped user ready
=== RUN   TestDeliveredContractMinio
=== RUN   TestDeliveredContractMinio/put
=== RUN   TestDeliveredContractMinio/get_round-trips_the_body
=== RUN   TestDeliveredContractMinio/list_shows_the_key
=== RUN   TestDeliveredContractMinio/head_and_list_agree_on_the_version
=== RUN   TestDeliveredContractMinio/the_app_cannot_tag_its_own_object
=== RUN   TestDeliveredContractMinio/the_app_cannot_tag_its_own_object/not_by_PutObjectTagging
=== RUN   TestDeliveredContractMinio/the_app_cannot_tag_its_own_object/and_not_by_x-amz-tagging_on_upload
=== RUN   TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route
=== RUN   TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/not_by_CreateMultipartUpload
=== RUN   TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/not_by_CopyObject_with_a_REPLACE_tagging_directive
=== RUN   TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/but_multipart_itself_is_allowed
=== RUN   TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/and_copy_itself_is_allowed
=== RUN   TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/a_POST-policy_forged_tag_with_a_matching_version_is_still_not_a_verdict
=== RUN   TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/and_an_ordinary_tagged-with-something-else_upload_is_allowed
=== RUN   TestDeliveredContractMinio/delete_removes_it
=== RUN   TestDeliveredContractMinio/cross-bucket_denial
--- PASS: TestDeliveredContractMinio (0.02s)
    --- PASS: TestDeliveredContractMinio/put (0.00s)
    --- PASS: TestDeliveredContractMinio/get_round-trips_the_body (0.00s)
    --- PASS: TestDeliveredContractMinio/list_shows_the_key (0.00s)
    --- PASS: TestDeliveredContractMinio/head_and_list_agree_on_the_version (0.00s)
    --- PASS: TestDeliveredContractMinio/the_app_cannot_tag_its_own_object (0.00s)
        --- PASS: TestDeliveredContractMinio/the_app_cannot_tag_its_own_object/not_by_PutObjectTagging (0.00s)
        --- PASS: TestDeliveredContractMinio/the_app_cannot_tag_its_own_object/and_not_by_x-amz-tagging_on_upload (0.00s)
    --- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route (0.01s)
        --- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/not_by_CreateMultipartUpload (0.00s)
        --- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/not_by_CopyObject_with_a_REPLACE_tagging_directive (0.00s)
        --- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/but_multipart_itself_is_allowed (0.00s)
        --- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/and_copy_itself_is_allowed (0.00s)
        --- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/a_POST-policy_forged_tag_with_a_matching_version_is_still_not_a_verdict (0.00s)
        --- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/and_an_ordinary_tagged-with-something-else_upload_is_allowed (0.00s)
    --- PASS: TestDeliveredContractMinio/delete_removes_it (0.00s)
    --- PASS: TestDeliveredContractMinio/cross-bucket_denial (0.00s)
PASS
ok  	github.com/carlosframework/platform/internal/objstore	0.022s
=== RUN   TestT4StorageProbeMinio
2026/10/16 08:13:18 moves: T4: STORE: list: operation error S3: ListObjectsV2, https response error StatusCode: 403, RequestID: 18DEF460ECB7EC2F, HostID: 7c420053e0fbcd7f41653fdd92ffa2802c903d05303b1aa15adc2e8bb65d183f, api error SignatureDoesNotMatch: The request signature we calculated does not match the signature you provided. Check your key and signing method.
--- PASS: TestT4StorageProbeMinio (0.00s)
PASS
ok  	github.com/carlosframework/platform/internal/host	0.020s
minio contract: PASS

diff --git a/docs/superpowers/notes/2026-10-16-empty-app-key-cleanup.bash b/docs/superpowers/notes/2026-10-16-empty-app-key-cleanup.bash

new file mode 100755

index 000000000..b0f880b0c

--- /dev/null

+++ b/docs/superpowers/notes/2026-10-16-empty-app-key-cleanup.bash

@@ -0,0 +1,145 @@

+#!/usr/bin/env bash

+# One-off cleanup for the empty-app errors chunks in the dev deployment bucket

+# (branch fix-empty-app-key). Run by hand, by Paul, from a laptop. Nothing in

+# the platform runs this, and it never runs on a box.

+# What it cleans: objects under exactly apps/bab// in carlos-dev-640012953448

+# (eu-west-1). An app-less route on leo-climbing (proof.dev.carlosframework.com:

+# account bab, no app, an Addr upstream nobody listens on) had its refused 502s

+# filed under (bab, "") and shipped as apps/bab//errors/leo-climbing/... .

+# hostops.Apps refuses to walk a bucket holding that prefix. The writer fix

+# stops new ones, but the box's pruner no longer touches the prefix either, so

+# the ones already there stay until they are deleted.

+# Usage:

+# 2026-10-16-empty-app-key-cleanup.bash dry run: list and count, delete nothing

+# 2026-10-16-empty-app-key-cleanup.bash --apply delete each current object by exact key

+# 2026-10-16-empty-app-key-cleanup.bash --apply --versions also delete every version and delete marker under the prefix, by key and version id

+# The bucket is versioned. --apply alone deletes each current object with one

+# delete-object call per key, which leaves a delete marker: reversible, and

+# enough for hostops.Apps, which lists current objects only. --versions

+# deletes each listed version and delete marker by its version id, one call

+# each. That is permanent.

+# Safety: the bucket, prefix, profile and region are fixed below and cannot

+# be overridden. The account is checked before anything else. Every key is

+# checked to start with exactly apps/bab// before it is acted on. There is no

+# --recursive and no prefix delete: each call names one key (and, with

+# --versions, one version id). The first failed call stops the run.

+# Run it after the fixed agent has rolled to every box that uses this bucket,

+# or leo-climbing will write new ones (the old agent was still writing at

+# 2026-10-16 06:46Z). The listing and a log of every call go to a run

+# directory printed at the start.

+set -euo pipefail

+readonly BUCKET="carlos-dev-640012953448"

+readonly ACCOUNT="640012953448"

+readonly PREFIX="apps/bab//"

+readonly PROFILE="carloku-staging"

+readonly REGION="eu-west-1"

+apply=0

+versions=0

+for arg in "$@"; do

+ case "$arg" in

+ --apply) apply=1 ;;

+ --versions) versions=1 ;;

+ *)

+ echo "usage: $0 [--apply [--versions]]" >&2

+ exit 2

+ ;;

+ esac

+done

+if [ "$versions" = 1 ] && [ "$apply" = 0 ]; then

+ echo "--versions only means something with --apply; a dry run already lists every version" >&2

+ exit 2

+fi

+aws_() { aws --profile "$PROFILE" --region "$REGION" "$@"; }

+got_account="$(aws_ sts get-caller-identity --query Account --output text)"

+if [ "$got_account" != "$ACCOUNT" ]; then

+ echo "profile $PROFILE is account $got_account, want $ACCOUNT; stopping" >&2

+ exit 1

+fi

+run="$(mktemp -d "${TMPDIR:-/tmp}/empty-app-cleanup.XXXXXX")"

+echo "run directory: $run"

+# count prints "current versions markers" for the prefix.

+count() {

+ local cur ver

+ cur="$(aws_ s3api list-objects-v2 --bucket "$BUCKET" --prefix "$PREFIX" --output json |

+ jq '[.Contents[]?] | length')"

+ ver="$(aws_ s3api list-object-versions --bucket "$BUCKET" --prefix "$PREFIX" --output json |

+ jq -r '"\([.Versions[]?] | length) \([.DeleteMarkers[]?] | length)"')"

+ echo "$cur $ver"

+# list writes the work list: one line per call, tab-separated

+# kind (object|version|marker), key, version id ("" for object).

+list() {

+ aws_ s3api list-objects-v2 --bucket "$BUCKET" --prefix "$PREFIX" --output json |

+ jq -r '.Contents[]? | ["object", .Key, ""] | @tsv' >"$run/objects.tsv"

+ aws_ s3api list-object-versions --bucket "$BUCKET" --prefix "$PREFIX" --output json |

+ jq -r '(.Versions[]? | ["version", .Key, .VersionId]), (.DeleteMarkers[]? | ["marker", .Key, .VersionId]) | @tsv' >"$run/versions.tsv"

+# check refuses the whole run if any listed key is outside the prefix, or

+# carries a tab, newline or backslash (jq's @tsv escapes those with a

+# backslash, so the key read back would not be the key listed).

+check() {

+ local kind key vid

+ while IFS=$'\t' read -r kind key vid; do

+ if [[ "$key" == *\\* ]]; then

+ echo "listed key with an escaped character: $key; stopping, nothing deleted" >&2

+ exit 1

+ fi

+ if [[ "$key" != "$PREFIX"* ]]; then

+ echo "listed key outside $PREFIX: $key; stopping, nothing deleted" >&2

+ exit 1

+ fi

+ case "$kind" in object | version | marker) ;; *)

+ echo "unexpected row kind $kind for $key; stopping, nothing deleted" >&2

+ exit 1

+ ;;

+ esac

+ done <"$1"

+read -r before_cur before_ver before_mark < <(count)

+echo "before: $before_cur current objects, $before_ver versions, $before_mark delete markers under $PREFIX" | tee "$run/counts.txt"

+list

+check "$run/objects.tsv"

+check "$run/versions.tsv"

+echo "listed: $(wc -l <"$run/objects.tsv") current objects ($run/objects.tsv), $(wc -l <"$run/versions.tsv") versions and markers ($run/versions.tsv)"

+if [ "$apply" = 0 ]; then

+ echo "dry run: nothing deleted. First rows of each list:"

+ head -5 "$run/objects.tsv"

+ head -5 "$run/versions.tsv"

+ echo "re-run with --apply (and --versions to remove every version too)"

+ exit 0

+fi

+work="$run/objects.tsv"

+if [ "$versions" = 1 ]; then

+ work="$run/versions.tsv"

+fi

+n=0

+while IFS=$'\t' read -r kind key vid; do

+ if [ "$kind" = object ]; then

+ aws_ s3api delete-object --bucket "$BUCKET" --key "$key" --output json >>"$run/calls.log"

+ else

+ aws_ s3api delete-object --bucket "$BUCKET" --key "$key" --version-id "$vid" --output json >>"$run/calls.log"

+ fi

+ printf '%s\t%s\t%s\n' "$kind" "$key" "$vid" >>"$run/deleted.tsv"

+ n=$((n + 1))

+done <"$work"

+echo "deleted: $n calls ($run/deleted.tsv)"

+read -r after_cur after_ver after_mark < <(count)

+echo "after: $after_cur current objects, $after_ver versions, $after_mark delete markers under $PREFIX" | tee -a "$run/counts.txt"