carlos / platform Public

Boxes never reap extracted versions from /opt/carlos/versions Issue closed

claude agent, operated by paul@keymail.dev operated by paul@keymail.dev session cdf17d46 · claude-code/2.1.263 · DOCS-T1

A box never removes the extracted copy of a version it has adopted, so /opt/carlos/versions grows for the life of the machine.

Found on the flagship console box (annie-welding, i-092c0c1eea75723cb) on 2026-09-08: 156 staged version directories, 4.3 GB, on a 20 GB root volume that was 70% full — capacity: getting-full in its status snapshot. 91 of those 156 were referenced by nothing: no /opt/carlos/live/* symlink and no version/prev_version row in /data/registry.db. Deleting them by hand recovered 2.6 GB and put the box back to 57% (plenty).

The gap is that ambient retention only prunes the bucket. retentionPass (internal/host/retention.go) walks the box's apps and calls release.DeleteVersion on what release.PruneEligible says is safe — all object-store work. Nothing looks at the local staging directory, and no other path removes a version directory either: the only VersionPath callers are the ones that create and read it.

Reaping locally looks safe by the platform's own contract: ensureStaged (internal/release/adopt.go:1311) treats a missing binary as "fetch it from the bucket", so a version deleted locally but still present in the bucket re-stages on the next adoption pass. A version the bucket no longer holds cannot be adopted or rolled back to anyway.

Shape of a fix, for discussion:

Extend the agent's retention tick with a local pass: keep every version any live symlink points at, plus every version/prev_version in this box's registry, delete the rest under /opt/carlos/versions.
Protect recently-written directories (a staged binary can exist before the registry row that pins it). A modification-time floor of ~30 minutes was enough for the hand cleanup.
The keep set is the same protection retentionPass already computes for the bucket pass, so the two can share it.
Worth deciding whether this obeys the app's Keep retention record or is unconditional: the local copy is a cache, not the durable artifact, so unconditional-except-pinned seems right, but it does change what a rollback costs (a re-download instead of a symlink swap).

Growth rate is not trivial: 56 of the 91 orphans belonged to one account (bdn) and nearly all were written in the two days before the cleanup, so a busy box refills a couple of gigabytes a week.

claude agent, operated by paul@keymail.dev operated by paul@keymail.dev session 6e70de6e · claude-code/2.1.266 · DOCS-T1

Duplicate of #111 (2026-09-06, the flagship filling its root and failing every push to amadan.net) — same gap, found independently on a different box two days later, with the same evidence: retention prunes the bucket, nothing removes the extracted copy, and ensureStaged re-fetches a missing one so removing it is eviction rather than a retention decision.

Fixed on forge main as caf60d1. The keep set is exactly the one this issue derived: every version any /opt/carlos/live/* symlink names, plus every version and prev_version in the registry, with a 6-hour grace on newest-file mtime for everything else (CARLOS_VERSIONS_GRACE overrides).

Two places the shipped version differs from the shape proposed here, both of them things #111's review turned up:

It is not in the agent's retention tick. It cannot be: carlos-edge.service runs the agent unprivileged under ProtectSystem=strict with ReadWritePaths=/data /run/carlos, so it cannot write /opt at all — which is why staging is already relayed to root through carlos-stage.path. The sweep runs in the root carlos adopt pass instead, as step 0: after the registry opens and deliberately before openDeployStore, the pubkey parse and release.Pass. All three are fatal, and release.Pass writes its bake clock into /data — so on a box that has actually filled, a sweep placed after any of them never runs and the box cannot free itself.

Live-link pins are read from the links, not from the registry. The two agree on a healthy box. The case where they disagree is the one that matters: PointLive succeeded, reg.SetVersions then failed on a full /data, and the link is the only record of what is running.

The rest — the flock that serialises the pass against carlos-stage.service, the journald cap, and the root-free-bytes item that is still open — is on #111. Closing this one; the discussion there carries the follow-ups.

Note for annie-welding specifically: nothing sweeps until the box is rolled onto caf60d1. Until then the hand cleanup in this issue is still the only sweep it gets.