carlos / platform Public

Migrated from github.com/carlosframework/platform issue #187, opened 2026-08-14. GitHub issues are closed for this repo; see AGENTS.md, "Work is tracked on amadan, not GitHub".

Still outstanding: A permission-denied try-restart is swallowed as a no-op, so an unprivileged agent consumes a version adoption the root pass never completes.

---

Summary

A -backing unit route adopts a new version correctly — staged, live/<host> symlink repointed, registry ready — and then keeps serving the old binary indefinitely. It is not a convergence window: the route never recovers on its own, because the pass that repointed it cannot restart the unit, and the pass that could restart it no longer has anything to adopt.

Observed on two independent routes, a week apart, on the same box.

Environment

• Agent: c42ec0c (6 commits behind head at time of writing)
• Both affected routes: -kind instance -backing unit, channel canary/deploy, no named unit in the registry
• carlos-adopt.timer armed and firing every 2 minutes (local canary-cadence.conf drop-in overrides the shipped OnUnitActiveSec=1h)
• Adopt passes complete normally: carlos-adopt.service: Deactivated successfully

Evidence

| | route A | route B | |---|---|---| | promoted | 2026-08-07T15:52:22Z (6119842) | 2026-08-14T08:34:49Z (5b503a5) | | /opt/carlos/live/<host> repointed | Aug 7 15:52 → versions/6119842/… | Aug 14 08:34 → versions/5b503a5/… | | unit ActiveEnterTimestamp | Aug 7 15:02:17 | Aug 8 (a previous hand restart) | | /api/version actually served | bab0ada (the predecessor) | 2634bb2 (the predecessor) |

Route A sat in this state for seven days and roughly 5,000 adopt passes. ActiveEnterTimestamp never moved.

Mechanism

The restart is attempted, three seconds after the repoint, and systemd refuses it:

Aug 14 08:34:52 carlos[605184]: adopt: try-restart carlos-app@<host>:
  exit status 1: Failed to try-restart carlos-app@<host>.service:
  Interactive authentication required.

Interactive authentication required is polkit refusing a non-root caller. systemctl status 605184 identifies the process:

● carlos-edge.service - carlos edge (TLS terminator, host router, activator, release adoption)

User=carlos. So the unprivileged agent is doing the adoption — which its own unit description lists as part of its job — and it cannot perform the restart that adoption implies.

systemctlTryRestart (internal/host/agent.go) logs and continues, per its best-effort contract:

> "cycle the route's carlos-app@ unit iff it's running, best-effort. A missing or stopped unit is not this pass's problem — try-restart is by definition a no-op on one."

That contract is reasonable for a stopped unit. The problem is that the function cannot distinguish "nothing was running" from "I am not permitted" — both arrive as a non-zero exit — so a permission denial is swallowed as routine and the pass reports success.

The root pass (carlos-adopt.service, running as root, every 2 minutes here) would have restarted it. But by the time it runs, the live symlink already matches the channel pointer, so it finds nothing to adopt and returns before reaching the restart arm. The unprivileged pass consumed the version change; the privileged pass inherited no trigger. Every subsequent pass repeats that, forever.

The unit was active throughout, so try-restart would have succeeded had it been permitted. A manual systemctl restart carlos-app@<host>.service fixes it immediately and permanently — until the next promote.

Ruled out

• Agent version — box is 6 commits behind head, not an old build.
• Timer not armed — is-active → active; list-timers shows it firing on schedule.
• Wrong channel — carlos instances shows both routes on canary/deploy with the new sha adopted.
• Wrong unit name — activator.UnitName produces carlos-app@<host>.service, which is exactly the unit a manual restart cycles successfully.
• Unit inactive / socket-idle — is-active → active.
• Staging or repoint failure — both succeeded, within the same minute as the promote.

Suggested fix

1. systemctlTryRestart should treat a permission denial as an error, not a no-op. Silently reading "not permitted" as "nothing to do" is what makes this both permanent and invisible. A pass that cannot restart should either fail loudly or leave the version unconsumed so the root pass still has work to do.

1. The agent should not be completing an adoption it cannot finish. rootRestartRoute's comment describes the intended design:

> "carlos-edge.service runs as User=carlos with ProtectSystem=strict and ReadWritePaths=/data /run/carlos, so the agent cannot write /opt/carlos: on a real box its pass reports the stage as blocked and relays to this verb (carlos-stage.path), which does the staging, the repoint, the restart and the version stamp."

On this box the agent evidently can write /opt/carlos, so it never reports blocked, never relays, and performs only the half of the job it is permitted to do. Whether that is a drop-in widening ReadWritePaths locally or a broader change, the failure mode suggests the relay should be triggered by "can I complete this adoption?" rather than by "can I write /opt?" alone — write access and restart authority are separate permissions, and today only the first one gates the decision.

Note on diagnosability

This took a long time to find, for a reason worth fixing separately: the adopt log's per-route lines only cover exec-backed routes, so neither affected route appeared anywhere in carlos-adopt.service's journal — present, absent, healthy or refused. The denial was only visible in an unfiltered journalctl window around the promote minute. carlos instances shows the adopted sha but not what the unit is actually running, so it reports the route as healthy too.

From the outside, every available signal said the deploy had succeeded: registry ready, correct sha, adopt passes completing, timer armed, service active. Only /api/version on the live host disagreed.