carlos / platform Public

Fixes items 1 and 2 of discussion #111 — the flagship filling its 20 GB root on 2026-09-06 with 369 staged version directories (9.3 GB) and 1.2 GB of uncapped journal, which failed every push to amadan.net.

Reviewed as a design by Fable and by GPT via codex before any code was written; the placement below is Fable's correction to my first draft, and it is the difference between a fix and a fix that cannot run.

1. Sweep the staged versions nothing on the box still names

/opt/carlos/versions/<ver> accumulates every version the box is ever asked to run. Retention (#136) prunes the bucket; the box's copy is a cache nothing evicts. This is eviction, not retention: ensureStaged re-downloads a missing version before adopting it, so removing one costs a download, never a deploy.

Where it runs. Step 0 of the root carlos adopt pass, right after the registry opens and before openDeployStore, the release pubkey parse and release.Pass. All three are fatal, and release.Pass writes its bake clock with CreateTemp under /data — so on a box with 1.2 MB free, a sweep placed after any of them never runs and the box cannot free itself. restartPass already sits above release.Pass for exactly this reason and says so in its comment.

The discussion proposed "the agent's housekeeping pass". That is not possible: carlos-edge.service is ProtectSystem=strict with ReadWritePaths=/data /run/carlos, which is why staging is already relayed to root through carlos-stage.path. The housekeeper lease is also wrong — it elects one holder per deployment, while /opt/carlos/versions is per-box, so a box that never won would never sweep.

What it keeps. A version directory is pinned by:

• any symlink under /opt/carlos/live, read from the links and not from the registry's Version column. The two agree on a healthy box; the case where they do not is the incident's own — PointLive succeeded, reg.SetVersions then failed on a full /data, and the link is the only record of what is running. .tmp leftovers from a crashed swap pin too.
• any route's Version or PrevVersion, unfiltered by adoptability, wakefulness or canary TTL. PrevVersion carries no grace: a rollback is wanted precisely when the new build has run long enough for one to expire.

Everything else survives 6 hours from its newest file mtime (CARLOS_VERSIONS_GRACE overrides). Files rather than directories because removing entries bumps a directory's mtime, so a half-swept tree would renew its own grace every period, forever. Time-driven rather than size-driven because a size trigger fires only once the disk is already tight, which is the state in which everything else has begun to fail.

Entries that are not directories, or whose names ValidVersion rejects, are somebody else's and are left alone. An unreadable live dir or an unlistable registry means an incomplete pin set — and an incomplete pin set is how a running app loses its binary — so the sweep declines to delete and logs. Blast radius if a pin were missed: a sleeping tenant 503s on wake with "the staged version was removed" (materialize.go).

One lock for the whole pass, in every mode. Four units exec carlos adopt and systemd will start them in the same second. That was tolerable while every write in a pass was a rename or a symlink swap; os.RemoveAll is the first operation that is not idempotent against a concurrent pass. The window is real: the sweep reads the live links, a stage pass repoints one at a version the sweep has just judged unpinned, the sweep deletes the tree that link now names. Rename-to-a-trash-dir does not close it either, because PointLive stats the target and then symlinks. A lock that cannot be taken drops the sweep and runs the pass exactly as before.

2. Cap the journal in bootstrap

/etc/systemd/journald.conf.d/50-cap.conf with SystemMaxUse=256M and RuntimeMaxUse=64M. The filename matches the one the 2026-09-06 relief placed by hand, so this file supersedes it rather than sorting beside it. Three things the first draft missed and review caught: bootstrap's dirs list had no etc/systemd/journald.conf.d, so the write would have failed ENOENT on a fresh box; the drop-in is inert until journald is restarted, which is now done on the live path; and the self-update apply copies named files out of a staged root, so the drop-in is copied there too — that relay is how boxes predating the cap actually get it.

Note for whoever owns pinfra: scripts/update-platform-binary.sh copies only units, sysusers and tmpfiles out of the staged tree, so a box updated by that script alone gets the cap on its next self-update or bootstrap rather than on that script's next run.

Not in this branch

Item 3 (root free bytes on the metrics loop) turns on a fact I have not been able to check: whether /data is a separate volume on the flagship. If it is not, DiskFrac was already ~1.0 during the incident and the 0.90 rule in internal/metrics/alert.go should have fired — which makes it an alert delivery bug, not a probe gap, and a different fix. If it is separate, the fix is to make readDiskFrac/readDiskFreeBytes report the worst-of dataDir() and release.OptDir(), which covers the alert rule, the status band and the console with no wire change.

Item 4 (amadan on its own fleet box) stays on the discussion.

Follow-ups

• Once this is on the flagship, remove the interim guard: /usr/local/sbin/carlos-versions-sweep and carlos-versions-sweep.timer.
• Registry.Delete does not remove live/<host>, so a stale link pins a version forever. The sweep is safe either way (it over-keeps), but the link should be removed on route delete.

Tests

internal/release/sweep_test.go and internal/host/versionsweep_test.go: pins from live links beat a stale registry, PrevVersion pinned without grace, a fresh directory mtime does not renew the grace, an emptied husk is still collectable, foreign entries survive, an unreadable registry or live dir deletes nothing, the pass lock actually excludes and does not deadlock against the restart-marks lock. Plus a bootstrap test for the drop-in.

go build, go vet, gofmt and go test ./... clean, and GOOS=darwin / GOOS=windows both compile.

🤖 Generated with Claude Code

https://claude.ai/code/session_01C4uAcc4PRTkETZo5NJJ67Y