carlos / platform Public

carlos adopt runs as root and carlos agent as carlos, on one SQLite registry — a fresh box can end up unstartable Issue open

Found bringing up three fresh region boxes on 2026-08-31 (tito, carlos c3f021d). All three were left unstartable by the same cause, and two needed a second manual repair because it is a race.

The mechanism

carlos-edge.service runs the agent as the carlos user:

ExecStart=/opt/carlos/carlos agent
User=carlos

carlos-adopt.service has no User=, so it runs as root:

ExecStart=/opt/carlos/carlos adopt

Both open /data/registry.db. carlos bootstrap arms adopt's .path and .timer units, so on a fresh box adopt can win the race, create the registry and its -wal/-shm as root, and then the edge can never start:

agent: open registry /data/registry.db: attempt to write a readonly database (8)

systemd gives up after five restarts in three seconds, so the box sits failed with no further attempts.

Why it is not seen on established boxes

Where the registry was created by the edge first, it is carlos-owned, and root writing to a carlos-owned file is fine. The tito uk and se boxes (bootstrapped in August) are unaffected; all three boxes brought up on 2026-08-31 hit it. So this reads as a regression, or at least a latent ordering bug that only a fresh bootstrap exposes.

Also a race, not just an ordering problem

Chowning the registry to carlos and then starting the edge is not reliably enough: adopt's path/timer units are already armed, so it can fire again in between and re-root the files. Two of the three boxes needed the chown twice. Our consumer-side script now retries once, which is a workaround, not a fix.

Suggested fix

Give carlos-adopt.service (and any other unit that opens the registry) the same User=carlos the edge has. If adopt genuinely needs root for part of its work, the registry access should be split out from the privileged part rather than the whole pass running as root.

A cheap fence regardless: have the agent fail with a message that names the cause. "attempt to write a readonly database (8)" is SQLite's words, and it sent us looking at disk space and mount options before file ownership.

Filed from a Claude Code session; the consumer-side workaround is teamtito/carlos-infrastructure fix-bootstrap-region.

Fixed on branch fix-adopt-registry-ownership. CI green.

You had the mechanism exactly right, including that it is a race and not just an ordering problem. The one thing I did differently is the fix itself, and it is worth saying why.

User=carlos on carlos-adopt.service is not available. The pass really does need root: it writes /opt/carlos (the version store) and cycles units via systemctl. The unit says so in its own header, and splitting the registry access out from the privileged part would mean a second process or a second unit for one SQLite handle. So the ownership is converged at both ends of the race instead:

  • carlos bootstrap creates /data/registry.db itself, before it arms anything that could race for it, and inside the chown -R carlos:carlos /data that privilegedPlan already runs. An empty file is a valid empty SQLite database, so the first real opener migrates into it. An existing registry is left completely alone — bootstrap re-runs over live boxes whose edge holds the file open, and that must not touch a schema.
  • Every root adopt pass chowns the database and both sidecars back. This is the half your report is really about: once by hand is not enough because the armed .path/.timer units fire again and re-create -wal/-shm as root in between, which is why two of your three boxes needed the chown twice. A carlos-owned database with a root-owned -wal is exactly as unwritable. It runs on every pass rather than at create, so a box that ends up wrong self-corrects within the hour instead of waiting for a person.

One deliberate choice: a chown failure is logged, never fatal. An adoption pass that can read the registry has releases to stage and units to cycle, and refusing all of that because a chown failed would turn a cosmetic problem into an outage.

The cheap fence is in too, and it covers all four root units plus the edge, since they all reach the registry through the same two openers. Instead of SQLite's words:

agent: open registry /data/registry.db: attempt to write a readonly database (8) agent: the registry is not writable by this process (uid 989). /data/registry.db is owned by root (uid 0, gid 0) — it and its -wal/-shm sidecars must be owned by carlos, the user carlos-edge.service runs as. A root carlos adopt pass that reached the file before the edge did is the usual cause. Fix: chown carlos:carlos /data/registry.db*

Other SQLite errors still read as themselves, so a genuinely full disk is not mislabelled as an ownership problem.

Tests pin the bootstrap ordering, that a re-run does not lose an existing registry's contents, the sidecar coverage, and the hint's text. The ownership-fixing helper is a no-op off root, so it does nothing on your laptop and nothing in CI.

Not merged — five branches went up tonight and I would rather you took them in one pass than have me land them one at a time. Nothing here changes behaviour on an established box.

Triage, 2026-09-05. fix-adopt-registry-ownership reviewed against this report. CI green. Verdict: merge with fixes — the two structural fences are correct, the friendly error is not.

Checked hardest, and clean:

  • An existing registry is genuinely untouched. Stat + O_CREATE|O_WRONLY with no O_TRUNC, and no registry.Open, so a bootstrap re-run over a live box touches neither bytes nor schema. TestBootstrapKeepsAnExistingRegistry pins it. The Stat-to-OpenFile gap has no TOCTOU cost precisely because nothing truncates.
  • The ordering does beat the armed units. Create → unit files → the existing chown -R carlos:carlos /data → systemctl enable/start. On a fresh box nothing is armed before the chown.
  • Sidecar coverage is complete. registry.Open forces journal_mode=WAL, so -wal/-shm are the only sidecars a rollback journal never appears. chownRegistryToService runs before every return in RunAdopt, so all four root units converge on every pass — which is the half your report is really about. Non-root is a real no-op via Geteuid().

The problem is the cheap fence, and it is your own complaint in reverse.

You asked for a message that names the cause, because SQLite's "attempt to write a readonly database (8)" sent you looking at disk space and mount options. The new hint can send the next person somewhere just as wrong, in two ways:

  1. As root it cannot be ownership, and the hint says it is. All four root units call it, and root holds CAP_DAC_OVERRIDE (the units drop no capabilities). A root "readonly database" is therefore never a permissions problem — it is a read-only remount after a filesystem error, an immutable attribute, a hot journal, or a MAC denial. The hint prints "owned by root (uid 0) ... Fix: chown carlos:carlos", which is the one thing that cannot help.
  2. It stats only the database, not the sidecars — so your second-failure shape, a carlos-owned db with a root-owned -wal, prints "owned by carlos ... must be owned by carlos". A self-contradiction, on the exact case that made two of your three boxes need the chown twice.

Fix is small: gate the ownership wording on os.Geteuid() != 0, stat all three files rather than one, and when every existing file already matches the process euid, say ownership is right and name mount or sandbox instead.

Two smaller ones. A -root /tmp/carlos-unitstage run now leaves a 0-byte data/registry.db in the staging tree — harmless if pinfra's update-platform-binary.sh copies only etc/, but that script is in the private repo and I could not read it. Worth confirming before landing, since a wholesale cp -r of the staging tree would truncate a live registry. And the commit message asserts more than the tests pin: nothing pins that the chown precedes the -restart-only early return, and the "logged, never fatal" choice is untested.

Not merging. It is root-side and fleet-wide, and the hint fix should land with it rather than after it.