carlos / platform Public

Fixes for the v0.25.1 flagship roll incident (#256): four commits, A to D, one per break, plus a review commit on B.

What changes

A. carlos system apply restarts carlos-instance-control.service, and a stale one is never ready. The install restarts the control service after the new units are installed and daemon-reload has run, and BEFORE the edge. Rollback and runtime recovery restart it onto the previous binary the same way; boot recovery leaves it alone (socket-activated, not yet running, and a probe before the edge would record a false failure for five minutes). A candidate with no control unit (a downgrade below v0.20) restarts nothing. A restart that fails during install rolls back.

Two checks make a forgotten restart loud instead of silent. The service's GET /ready compares /proc/self/exe with /opt/carlos/carlos by device and inode on every call and fails a binary line while they differ. And it answers X-Carlos-Build: <version.Build>; the agent compares that with its own build and refuses nested wakes when the header names another build or is absent (an older service). Both reasons end with restart carlos-instance-control.service.

B. The layout pass moves the app's own entries with the host, or refuses. Under the fence, inside the serialized section with the journal write, a move scans the data directory for every entry the flat naming rules make the host's (<host>, <host>.*, <host>-*, .activator-<host>.*) and moves each into the instance directory at the same name; a revert brings them back. Ownership is the longest-matching host among the box's rows AND every host the directory's own platform-shaped files are evidence of (X.env, X.db, X.db-wal, X.db-shm, X.config.env, X-sidecar.env, X.db.corrupt, .activator-X.*): a deleted or moved-away app.example.com.au or <host>-staging still owns its files and the app entries beside them, which are never taken and are reported (named for another host: X.env (X's, which has no row on this box)). The entries are recorded in the journal by name (additive field; older journals have none), re-derived on recovery, re-checked for shape, and re-checked for ownership against the box's rows, so a forged journal cannot move a neighbour's directory. Same protections as the platform names: a symlink, a special file, or a hard link with a name outside the entry refuses the move (inode and find -samefile in the reason) and the host is held; hard links wholly inside the entry (an app's own dedup) move together. A taken destination refuses, naming both. What no host's rule names (/data/blobs) is not moved and is logged once per pass that moved something. A nested host that still has entries under its name in /data (the hosts migrated before this) gets a log line and a status note on every pass, and nothing moved. The link walk over an app directory logs before it starts.

C. CARLOS_HOST. Every exec child, flat and nested, and every seed, gets CARLOS_HOST=<row host>, agent-owned (a tenant's env file cannot override it). The socket contract is unchanged. docs/site/cli.md gains "What an instance is started with".

D. WorkingDirectory=/data/instances/%I in the run, seed and sidecar templates, pinned by the unit test (one line each, none in the probe). The IE drop-in 50-workdir-incident-256.conf carries this exact value, so it can be removed once this release is on the box.

What this changes on a box

• On the next carlos system update to a release carrying this: the apply's systemctl sequence becomes daemon-reload, reset-failed carlos-instance-control.service, restart carlos-instance-control.service, reset-failed carlos-edge, restart carlos-edge, relay. Any /start in flight at the control restart is dropped; it would have been stopped by the edge restart anyway (BindsTo), the agent reads it as a failed wake and sends /stop, which the socket unit queues. Rollbacks and runtime recoveries add the same restart before the edge's.
• Until that restart happens on a box where the service is stale, nested wakes are refused with the restart named, instead of failing at NAMESPACE with /opt/carlos/app: No such file or directory.
• Already-nested hosts (the 104 on IE) are not re-migrated, so entries the first migration left behind (/data/<host>/, /data/<host>-blobs, a <host>.env minted flat since) are NOT moved by this. Each such host gets, on every pass, layout: <host>: still in /data from before its move, where the instance cannot see them: ... move what the app needs into /data/.instances/<iid> by hand, and the same text as its note in carlos status --layout. That is how the IE operator finds anything else like the ips keys. A box still flat (or a reverted one) migrates with its entries from now on.
• A revert brings back the entries named for the host. What an app wrote relative to its new working directory while nested (data/blobs under its instance directory) stays in the instance's raw directory: the flat app does not look there, but nothing is lost, and the next migration finds it in place.
• Each layout: log line for a move now lists what moved with it; one more line per pass names /data entries no host owns, and one per scan names files left for a longer or departed host. On IE the first will name /data/blobs (7,766 blobs, 1.1 GB, owner unknown). An operator who establishes the owner moves it into that app's instance directory by hand; the pass carries it on later moves.
• Every nested unit's cwd becomes its data directory; ./data/blobs then lands in /data/instances/<iid>/data/blobs, where only that instance sees it. After the roll, on IE: rm /etc/systemd/system/carlos-instance@sharingangels.bdn.oncarlos.com.service.d/50-workdir-incident-256.conf && systemctl daemon-reload (same value, so the unit's effective configuration does not change).
• Every instance and seed gets CARLOS_HOST. The IPS_ORIGIN hand-set on three environments can stay or go once the app reads CARLOS_HOST.

pinfra: update-platform-binary.sh

Not edited here (other repo). It needs, before systemctl restart carlos-edge:

if [ -f /etc/systemd/system/carlos-instance-control.service ]; then
  systemctl enable --now carlos-instance-control.socket
  systemctl reset-failed carlos-instance-control.service || true
  systemctl restart carlos-instance-control.service
fi

and in the post-conditions:

if [ -f /etc/systemd/system/carlos-instance-control.service ]; then
  systemctl is-active carlos-instance-control.socket
  pid=$(systemctl show -p MainPID --value carlos-instance-control.service)
  [ "$pid" = 0 ] || [ "$(stat -Lc %d:%i /proc/$pid/exe)" = "$(stat -Lc %d:%i /opt/carlos/carlos)" ]
fi

How it was tested

Each fix has tests that fail on main (verified by reverting the implementation file and running them):

• A: TestApplyRestartsTheInstanceControlServiceBeforeTheEdge, TestApplyInstanceControlRestartFailureRollsBack, TestApplyRollbackRestartsTheInstanceControlService, TestApplyDoesNotRestartAControlServiceTheCandidateLacks, the runtime/boot halves of TestRecoveryRollsBackFilesOnlyAtBootAndRestartsAtRuntime (internal/host); TestReadyRefusesWhenTheRunningBinaryIsNotTheInstalledOne, TestClientReadyRefusesAControlServiceOfAnotherBuild (internal/instancectl, over a real unix socket).
• B: TestConvergeMovesAppCreatedEntriesWithTheHost (migrate, decoy host, shared dir reported, revert), TestConvergeRefusesAnAppEntryWithANameOutsideIt, TestConvergeRefusesASymlinkInsideAnAppEntry, TestConvergeRefusesAnAppEntryWhoseDestinationIsTaken, TestRecoverCompletesAMoveWithAppEntries (three crash points), TestRecoverRefusesAJournalNamingAnotherHostsEntry, TestExtraOwnershipIsTheLongestMatchingHost, TestValidateJudgesTheJournalsExtraNames, TestUnattributedEntriesSkipsHostsAndPlatformNames; review: TestScanExtrasLeavesAnotherHostsPlatformFilesAlone (every platform shape, dotted and dashed extension, both directions), TestFixedShapeHostReadsEveryPlatformShape, TestConvergeReportsFlatLeftoversOfANestedHost (internal/layout).
• C: TestExecBackendStartChildHandsTheHost (a real child's environment, tenant forgery overridden), TestLaunchEnvironmentComposesTheSpawnEnvironment, and the two seed env tests assert CARLOS_HOST.
• D: TestInstanceTemplatesShareOneNamespace pins the line in the three templates and its absence from the probe.

Gate: gofmt -l . clean, go vet ./... clean, full make test: every package ok except the two known environment failures (root package reads ~/.carlos/config; internal/changelog needs release tags this clone lacks).

Deliberately not done

• No automatic move of already-nested hosts' flat leftovers: they are reported on every pass, and an operator decides per entry (the app may have minted a newer copy since, as the ips keys show).
• Shared, unattributable directories (/data/blobs) are reported, not moved.
• update-platform-binary.sh is in platform-infrastructure; the lines it needs are above.
• The in-repo copy of the isolated-exec spec is not amended (specs are owned in platform-infrastructure).