carlos / platform Public

Picked this up. Items 1 and 2 are on branch sweep-staged-versions (amadan branch describe carries the full reasoning); item 3 is blocked on a fact I could not check, and item 4 stays here.

The design was reviewed by Fable and by GPT via codex before any code was written. Two of the three findings changed the shape of the fix.

Item 1 could not go where this issue proposed, and the placement is the fix

"In the agent's housekeeping pass" is not available: carlos-edge.service runs the agent as an unprivileged user under ProtectSystem=strict with ReadWritePaths=/data /run/carlos. It cannot write /opt at all — which is precisely why staging is already relayed to root through carlos-stage.path. The housekeeper lease is wrong for a second reason: it elects one holder per deployment, while /opt/carlos/versions is per-box, so a box that never won the lease would never sweep.

So it belongs in the root carlos adopt pass. But where in that pass is the whole fix, and my first draft had it wrong. Fable caught it: I had the sweep after release.Pass, and release.Pass writes its bake clock with CreateTemp under /data and returns a hard error when it cannot — on which RunAdopt calls log.Fatalf. On this box, with 1.2 MB free, a sweep placed there never runs, and the box has no way to free itself. Exactly the state the sweep exists for is the state it would not survive.

It is now step 0: right after the registry opens, before openDeployStore, before the pubkey parse, before release.Pass — all three fatal. There is a precedent for that reasoning already in the file: restartPass sits above release.Pass for the same reason and says so.

What it keeps

A version directory survives if any live symlink names it, or if any route's Version or PrevVersion does, unfiltered by adoptability, wakefulness or canary TTL. Otherwise it survives 6 hours from its newest file mtime (CARLOS_VERSIONS_GRACE overrides).

Three details worth recording, because each is a way this could have deleted a running binary:

• Live-link pins are read from the links, not from the registry's Version column. The two agree on a healthy box. The case where they disagree is this incident's own: PointLive succeeded, reg.SetVersions then failed on a full /data, and the link is the only record of what is running. Believing the registry there deletes a live binary.
• PrevVersion is pinned with no grace. A rollback is wanted precisely when the new build has been running long enough for a grace to expire.
• *Newest file mtime, not directory mtime.* Deleting entries bumps a directory's mtime, so a tree a failed sweep half-removed would look freshly written and renew its own grace every period, forever.

Time-driven rather than size-driven, deliberately: a size trigger only fires once the disk is already tight, which is the state in which everything else on the box has begun to fail.

An unreadable live dir or an unlistable registry means an incomplete pin set, and an incomplete pin set is how a sleeping tenant 503s on wake with "the staged version was removed". Both cases decline to delete and log.

The race nobody had to think about before

Four units exec carlos adopt — the hourly timer, carlos-stage.service, carlos-restart.service, carlos-config-sync.service — and systemd will start them in the same second. Nothing serialises them, which was fine while every write in a pass was a temp-file-rename or a symlink swap: two passes doing the same work converge. os.RemoveAll is the first operation in a pass that is not idempotent against a concurrent one, and the window is real — the sweep reads the live links, a stage pass repoints one at a version the sweep just judged unpinned, the sweep deletes the tree that link now names.

The branch takes one exclusive flock across the whole of RunAdopt in every mode (the shape lockRestartMarks already uses). Rename-to-a-trash- directory does not close this, incidentally — PointLive stats the target and then symlinks, so a rename between those two still leaves a dangling link and a unit that will not start.

A lock that cannot be taken drops the sweep and runs the pass exactly as it ran before the sweep existed.

Item 2, with three things the drop-in needed

/etc/systemd/journald.conf.d/50-cap.conf, same filename as the one placed by hand here on 2026-09-06 so this supersedes it rather than sorting beside it, SystemMaxUse=256M and RuntimeMaxUse=64M.

Review caught three ways the naive version would have done nothing: bootstrap's dirs list had no etc/systemd/journald.conf.d, so the write would fail ENOENT on a fresh box; a drop-in is inert until journald is restarted (reload does not re-read it); and the self-update apply copies named files out of a staged root, so the drop-in has to be named there too — that relay is how every box that predates the cap actually gets it.

One thing for whoever owns pinfra: scripts/update-platform-binary.sh copies only units, sysusers and tmpfiles out of the staged tree. A box updated by that script alone picks the cap up on its next self-update or bootstrap, not on that script's next run.

Item 3 is blocked on one fact

Both reviewers flagged that my framing assumed /data is a separate volume from root. If it is not, then DiskFrac was already ~1.0 while this box was filling, the 0.90 rule in internal/metrics/alert.go should have fired, and the bug is alert delivery — a different fix from the one this issue proposes. If it is separate, the fix is smaller than I first designed: make readDiskFrac/readDiskFreeBytes report the worst-of dataDir() and release.OptDir(), and the existing 0.90 rule, the status Capacity band and the console are all covered with no wire change and no new field.

Answering it needs the flagship's status/boxes/<label>.json (disk_free_bytes, and whether an alert key was ever written), which needs CARLOS_STATUS_BUCKET or CARLOS_DEPLOYMENT_BUCKET set. I have not been able to read it. Item 3 is not in the branch.

Follow-ups

• Once the branch is on this box, remove the interim guard: /usr/local/sbin/carlos-versions-sweep and carlos-versions-sweep.timer.
• Registry.Delete does not remove live/<host>, so a stale live link pins its version forever. The sweep is safe either way — it over-keeps — but the link should be removed when a route is deleted. Worth its own discussion if nobody picks it up with this.

Item 4 (amadan on its own fleet box) is untouched and stays here.

Items 1 and 2 landed on forge main as caf60d1, green on the branch and green on main after the merge. GitHub mirror fast-forwarded to match.

Main had moved 157 commits while the branch sat, four of them in files it touches, so it was merged forward and re-gated rather than landed on a two-day-old base. One conflict, in bootstrap.go: main rewrote the tail of systemctlPlan's doc comment for carlos-relay.service where this branch had inserted journaldReload. Main's paragraph stays attached to the function it documents; the var moved below it. The ordering the fix depends on — pass lock, registry, sweep, then the three fatal calls — survived the merge intact.

#122 was the same gap found independently on annie-welding two days later. Replied there and closed it as a duplicate of this one.

Nothing sweeps until the boxes are rolled onto caf60d1. The hand cleanup and the interim carlos-versions-sweep.timer on the flagship are still what is holding the line until then; remove them on the box once the roll is done and journalctl -u carlos-adopt shows the hourly sweep line.

Still open on this issue, unchanged:

• Item 3, root free bytes, blocked on whether /data is a separate volume on the flagship. If it is not, DiskFrac was already ~1.0 during the incident and the DiskFrac > 0.90 rule should have fired — which makes it an alert delivery bug and a different fix from the one proposed here.
• Item 4, amadan on its own fleet box.
• The follow-up noted earlier: Registry.Delete does not remove live/<host>, so a stale live link pins its version forever. The sweep is safe either way — it over-keeps — but the link should go when the route does.