amadan

carlos / platform Public

← Discussions

S3 request bill grows with apps × boxes: every box re-walks the whole bucket on each pass (~$3/app/month) Issue open

Claimed by claude~paul@keymail.dev 3 days ago

claude driven by paul@keymail.dev session 882e851a · claude-code/2.1.292 · DOCS-T1

Problem

Flagship S3 was $211 for October by the 20th and is running at about $16/day (~$480/month). About $2/day in early September. Nearly all of it is request charges from polling, not storage and not writes:

| Oct 1–19 (Cost Explorer) | Requests | Cost | |---|---|---| | ListBucket | 26.8M | $146 | | GetObject | 118.9M | $51 | | Storage (~690 GB-month) | | $16 | | PutObject + multipart | 1.8M | $9 |

The bill steps up each time apps are added (+~0.75M LISTs/day on 2026-10-06 and again on 2026-10-18). It scales with apps × boxes, not with usage: about $3 per app per month in polling. Adding a box or a region raises every app's cost. Plan tiering does not help here; litestream (the carlos-data-ie bucket) is ~$26/month of it.

Evidence

S3 server access logs on the flagship bucket and all eu-west-1 store buckets, 2026-10-20 21:14Z to 2026-10-21 11:00Z (5.5M requests; rates match Cost Explorer). Logging has been turned off and the log bucket deleted. Everything that matters is in the flagship deployment bucket, made by the box roles (carlos-flagship-host[-au|-uk|-us]). Tenant store buckets are negligible. The IE box makes ~3.5x the LISTs of each other box because the console shares its role.

| Family (per day) | Count | ≈ /month | |---|---|---| | LIST apps/<acct>/<app>/instances/ | 1.17M | $175 | | LIST apps/<acct>/ (delimiter, one per account) | 0.49M | $73 | | LIST apps/<acct>/<app>/sidecars/ | 0.34M | $52 | | GET control/certs/* | 1.55M | $19 | | GET accounts/<sqid>/* | 1.41M | $17 | | GET apps/*/*/config-gen.json | 1.23M | $15 | | GET apps/*/*/channels/canary | 0.60M | $7 |

Causes (traced at v0.28.6)

  • The walk. provision.listAppSubtree (internal/provision/reconcile.go:2022) does ListDirs("apps/"), a ListDirs per account, then a List of instances/ or sidecars/ for every app in the bucket, not only the ones the box hosts. knownAccounts (:1760) lists accounts/ and re-GETs every account.json each time.
  • Three walks per full pass per box: provision.Reconcile and ReconcileSidecars from reconcileSerializedWith (internal/host/agent.go:1605-1606), plus ListAllRecordsStrict in the moves pass (internal/host/movespass.go:1079).
  • Full passes fire too often. Any bump of the global control/serial, by any tenant, makes all four boxes walk everything (fastTick → fullPass, agent.go:2557/2787). There is also the ungated 5–6 min instancesLoop (agent.go:3380-3397). Fitted: ~570 full passes per box per day.
  • Console run sweep: StartRunSweep (internal/console/runsweep.go:17, 30s) runs ListAllRecordsTolerant across every app, ~2,880 walks/day on IE. seedsweep.go adds one every 5 min.
  • Ungated GET loops: configGenLoop (agent.go:1979, every 3–4s) fully GETs each config-gen.json just to compare ETags. fastMembers (agent.go:2214/2242) GETs every canary channel pointer every 2s tick. The edge cert mirror (internal/edge/certsync.go:410, 30s) GETs every cert (140) on every edge box. telemetryPass reads accounts/<sqid>/telemetry.json per hosted app per pass.

The counts fit roughly 150 apps, 48 accounts with apps and ~140 account manifests. These are fitted, not counted; check against a listing before relying on the per-family split.

Proposed fixes, by saving

  1. One walk per full pass, and walk only what the box needs (~$100–200/month). Share one listAppSubtree result across Reconcile, ReconcileSidecars and the moves pass. Then restrict box-side walks to hosted apps where the full set is not needed, or move to per-app change signals so a tenant's console action does not make every box re-list every app.
  2. Console run sweep (~$90/month). List only apps with active runs, or gate it on the serial instead of 30s unconditional.
  3. knownAccounts (~$17/month). Compute once per pass, and use ListMod + reuse-unchanged as in internal/domains/claimcache.go.
  4. GET loops (~$40/month). Conditional GET (If-None-Match) or ListMod over the prefix with reuse-unchanged, for config-gen, canary channel pointers and the cert mirror. The cert mirror change also shortens the boot refresh that bit the AU box (#118).

All together this should take flagship S3 from ~$430/month to well under $100, and stop each new app adding ~$3/month. (1) and (2) matter before paid plans launch.

LLM

Access-log attribution of the flagship S3 bill to box-side whole-bucket walks, the 30s console run sweep and ungated GET loops, with fixes ranked by saving.

claude driven by paul@keymail.dev session 882e851a · claude-code/2.1.292 · DOCS-T1

Decision (Paul, 2026-10-21): stay on S3, fix the change detection

Alternatives weighed:

  • DynamoDB (eu-west-1 $0.1415 per million read units, AWS Price List API): today's ~9M reads/day, polling unchanged, would cost ~$20/month on-demand against ~$430 on S3. It fits behind release.Store (conditional writes give the CAS), and it would be an AWS-only adapter, which is in keeping with the provider-neutral rule since MinIO has no per-request cost. Rejected for now: moving every control record, keeping ETag/LastModified/ListDirs/ListMod semantics and the root-reads census intact, the 400 KB item cap, and a cutover across boxes and console make it a bigger and riskier job than fixing the polling. It stays the escape hatch if request cost matters again at much larger scale.
  • A cache (Valkey/Redis, or a per-box cache): rejected. A cache is only as good as its invalidation, and invalidation needs the same change signal the boxes lack today. A TTL means stale state or the same polling, and a shared cache is a new component per region and for every self-hoster.
  • Why fix the polling regardless: the work still grows as apps × boxes × passes on any backend (box CPU, pass latency), and with the polling fixed S3 is expected to drop to ~$50/month.

Direction: a change index in the bucket. Writers bump an app's generation, boxes poll one small object with If-None-Match and re-read only the apps that changed, and a slow full walk is kept as a backstop for direct-bucket writers. Plus the console run sweep, the knownAccounts reuse and the GET loops from the list above. A spec comes first, then review, then build.

claude driven by paul@keymail.dev session 882e851a · claude-code/2.1.292 · DOCS-T1

Merged: 26fbd04 (2026-10-22)

The change index (spec docs/superpowers/specs/2026-10-21-s3-change-index-design.md, plan in docs/superpowers/plans/) is on main, dormant until CARLOS_CHANGE_INDEX=read. Writers always bump control/changes.json.

  • Reviews: spec Astra R1 reject → R2 accept; code reviewed per task, then Astra whole-branch R1 reject (4 P1) → one fix wave → R2 accept.
  • Open, accepted for now (Paul merged without them): P2 a failed-index tick disables the whole canary pointer gate including its read callback, so a newly observed head can be forgotten after a rollback (retry waits ≤ 30 min); P2 with the feature off, the moves pass enumerates .moves at the end of each pass (a filesystem read main did not do). Both small follow-ups.
  • Census ruling for the overlays asked on #280 (amended since: runAgent re-pinned after db-region-forward-1).
  • Still owed: release, dev-first roll, arm dev then flagship, and the measurement against the 2026-10-19 baseline (2.26M LIST, 7.5M GET per day). Operator notes are in the README.

claude operated by paul@keymail.dev session 882e851a · claude-code/2.1.292 · DOCS-T1

Released and armed (2026-10-22)

  • v0.28.9 (tag → 26fbd042) published; notes Luna-reviewed, the change-index operator text is Paul's approved README copy.
  • Dev: rolled and armed (leo, dennis, sophie + console). wakecheck env change converged in ~2 s with the index armed.
  • Flagship: boxes rolled 22:59–23:05Z, console 23:06Z; armed iris 23:08:59Z, adi 23:09:46Z, barbara 23:10:29Z, annie 23:11:42Z, console 23:13:54Z. Customer health identical before and after. control/changes.json live.
  • Measurement: Cost Explorer daily ListBucket/GetObject against the 2026-10-19 baseline (2.26M LIST, 7.5M GET/day); 2026-10-23 is the first full armed day.
  • Note: v0.28.9 also turns on the db-read worker by default (~5.8k LIST/day per hibernating box; CARLOS_DB_READS=0 opts out). Left as is.