amadan
WIP claim Plan 14 batch 2 checkpoint (not for merge): B3 canonical binding and one-attempt rollback capability; partial J1, C2, C3 and C4; stopped on the C3 inventory lock-ordering question
Co-Authored-By: Codex <noreply@openai.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
48 files changed,
+2747
−185
internal/host/agent.go+22 −15internal/host/agent_test.go+7 −32internal/host/claimconstruction.go+21 −1internal/host/claimjournal_transport.go+4 −0internal/host/claimproduction.go+1 −1internal/host/movesfinal.go+1 −1internal/host/plan14_canonical_restart_test.go+276 −0internal/host/plan14_release_lock_test.go+58 −0internal/host/proofbinding.go+259 −0internal/host/proofbinding_test.go+284 −0internal/host/proofobserve.go+59 −34internal/host/proofobserve_test.go+36 −11internal/host/proofroot.go+20 −6internal/host/proofroot_test.go+8 −5internal/host/rootfenceview.go+42 −0internal/host/rootreleaselock.go+25 −0internal/host/rootrestart.go+286 −0internal/host/rootrollback.go+166 −0internal/host/rootrollback_test.go+180 −0internal/host/rootroute.go+96 −0internal/host/unitsettle.go+42 −9internal/host/unitsettle_test.go+11 −9internal/hostbox/journalservice.go+12 −11internal/hostbox/reconcile.go+1 −1internal/hostbox/reconcile_adapter.go+1 −1internal/hostbox/work.go+1 −1internal/hostguard/coldhandover.go+5 −1internal/hostguard/fence.go+1 −1internal/hostguard/guard.go+50 −1internal/hostguard/plan14_root_fences_test.go+165 −0internal/hostguard/record.go+5 −1internal/hostguard/record_other.go+4 −1internal/hostguard/record_unix.go+29 −2internal/hostguard/recoveryfence.go+26 −6internal/hostguard/root.go+120 −0internal/hostguard/rootview.go+89 −0internal/hostguard/writerstop.go+1 −1internal/provision/plan9_b8_census_test.go+4 −0internal/provision/rootrecord.go+50 −0internal/provision/rootrecord_test.go+52 −0internal/provision/testdata/plan14-root-reads.tsv+61 −33internal/release/adopt.go+30 −0internal/rootstate/file_linux.go+38 −0internal/rootstate/file_linux_test.go+42 −0internal/rootstate/file_other.go+9 −0internal/rootstate/state.go+6 −0internal/rootstate/state_linux.go+16 −0internal/rootstate/state_linux_test.go+25 −0
CI failed — run details
CI log
amadan: log truncated — showing the last 209494 bytes of 1048755
2026/10/12 22:57:20 moves: a.go.example.com: mv_abcdefghijklmnopqrstuvwxyz published (landing-published)
2026/10/12 22:57:20 moves: a.go.example.com: landing mv_abcdefghijklmnopqrstuvwxyz is published
2026/10/12 22:57:20 moves: a.go.example.com: mv_abcdefghijklmnopqrstuvwxyz local (landing-local)
2026/10/12 22:57:20 moves: a.go.example.com: landing mv_abcdefghijklmnopqrstuvwxyz is local
2026/10/12 22:57:20 moves: a.go.example.com: skipped: the prepared files manifest needs C8 (files.BuildPrepared)
2026/10/12 22:57:20 moves: a.go.example.com: files, environment, timers and columns installed; .files-gen written
2026/10/12 22:57:20 moves: a.go.example.com: mv_abcdefghijklmnopqrstuvwxyz configured (landing-configured)
2026/10/12 22:57:20 moves: a.go.example.com: landing mv_abcdefghijklmnopqrstuvwxyz is configured
2026/10/12 22:57:20 moves: a.go.example.com: database installed at /tmp/TestNoArmRunsWithoutTheThreeParkHooksrecovery900611645/005/a.go.example.com.db (sha256:ae0661eb68be3f7cf598dba7530492d53b78f5082395b4a1ac8e1f86ea82228d), generation mv_abcdefghijklmnopqrstuvwxyz
2026/10/12 22:57:20 moves: a.go.example.com: mv_abcdefghijklmnopqrstuvwxyz restored (landing-restored)
2026/10/12 22:57:20 moves: a.go.example.com: landing mv_abcdefghijklmnopqrstuvwxyz is restored
2026/10/12 22:57:20 moves: a.go.example.com: landing mv_abcdefghijklmnopqrstuvwxyz waits at restored: this box has no probe wake (claim Plan 11's launcher and Plan 5 Task 9's arming are needed)
2026/10/12 22:57:20 moves: a.go.example.com: mv_abcdefghijklmnopqrstuvwxyz requested (requested)
2026/10/12 22:57:20 moves: a.go.example.com: staged the files archive sha256:c9229ef8e7256a1dcd5c2715b11ac3016951e8fee8e0bfa823aa2460f0d331c8 (2 files, 19 bytes)
2026/10/12 22:57:20 moves: a.go.example.com: not ready: the target has not published a readiness report yet
2026/10/12 22:57:20 moves: a.go.example.com: mv_abcdefghijklmnopqrstuvwxyz ready (ready)
2026/10/12 22:57:20 moves: a.go.example.com: move mv_abcdefghijklmnopqrstuvwxyz is ready (last round moved 1815 bytes; bar 16777216)
2026/10/12 22:57:20 moves: a.go.example.com: mv_abcdefghijklmnopqrstuvwxyz cut_requested (cutting-requested)
2026/10/12 22:57:20 moves: a.go.example.com: mv_abcdefghijklmnopqrstuvwxyz admitted (cutting-admitted)
2026/10/12 22:57:20 moves: a.go.example.com: mv_abcdefghijklmnopqrstuvwxyz stopping (cutting-stopping)
2026/10/12 22:57:20 moves: a.go.example.com: cut mv_abcdefghijklmnopqrstuvwxyz is stopping
2026/10/12 22:57:20 moves: a.go.example.com: mv_abcdefghijklmnopqrstuvwxyz stopped (cutting-stopped)
2026/10/12 22:57:20 moves: a.go.example.com: cut mv_abcdefghijklmnopqrstuvwxyz is stopped
2026/10/12 22:57:20 moves: a.go.example.com: stopped at TXID 3; execution given up, citing moves/bab/app/a.go.example.com/mv_abcdefghijklmnopqrstuvwxyz/stopped-source.json
2026/10/12 22:57:20 moves: a.go.example.com: sealed 4 replica objects at TXID 3
2026/10/12 22:57:20 moves: a.go.example.com: mv_abcdefghijklmnopqrstuvwxyz staged (cutting-staged)
2026/10/12 22:57:20 moves: a.go.example.com: cut mv_abcdefghijklmnopqrstuvwxyz is staged
2026/10/12 22:57:20 moves: a.go.example.com: mv_abcdefghijklmnopqrstuvwxyz archived (cutting-archived)
2026/10/12 22:57:20 moves: a.go.example.com: cut mv_abcdefghijklmnopqrstuvwxyz is archived
2026/10/12 22:57:20 moves: a.go.example.com: mv_abcdefghijklmnopqrstuvwxyz flipped (cutting-flipped)
2026/10/12 22:57:20 moves: a.go.example.com: flipped to uk
2026/10/12 22:57:20 moves: a.go.example.com: mv_abcdefghijklmnopqrstuvwxyz parking (landing-parking)
2026/10/12 22:57:20 moves: a.go.example.com: landing mv_abcdefghijklmnopqrstuvwxyz is parking
2026/10/12 22:57:20 moves: a.go.example.com: mv_abcdefghijklmnopqrstuvwxyz building (landing-building)
2026/10/12 22:57:20 moves: a.go.example.com: landing mv_abcdefghijklmnopqrstuvwxyz is building
2026/10/12 22:57:20 moves: a.go.example.com: generation mv_abcdefghijklmnopqrstuvwxyz built at TXID 3
2026/10/12 22:57:20 moves: a.go.example.com: mv_abcdefghijklmnopqrstuvwxyz published (landing-published)
2026/10/12 22:57:20 moves: a.go.example.com: landing mv_abcdefghijklmnopqrstuvwxyz is published
2026/10/12 22:57:20 moves: a.go.example.com: mv_abcdefghijklmnopqrstuvwxyz local (landing-local)
2026/10/12 22:57:20 moves: a.go.example.com: landing mv_abcdefghijklmnopqrstuvwxyz is local
2026/10/12 22:57:20 moves: a.go.example.com: skipped: the prepared files manifest needs C8 (files.BuildPrepared)
2026/10/12 22:57:20 moves: a.go.example.com: files, environment, timers and columns installed; .files-gen written
2026/10/12 22:57:20 moves: a.go.example.com: mv_abcdefghijklmnopqrstuvwxyz configured (landing-configured)
2026/10/12 22:57:20 moves: a.go.example.com: landing mv_abcdefghijklmnopqrstuvwxyz is configured
2026/10/12 22:57:20 moves: a.go.example.com: database installed at /tmp/TestNoArmRunsWithoutTheThreeParkHookssource_finalization2475589902/005/a.go.example.com.db (sha256:ae0661eb68be3f7cf598dba7530492d53b78f5082395b4a1ac8e1f86ea82228d), generation mv_abcdefghijklmnopqrstuvwxyz
2026/10/12 22:57:20 moves: a.go.example.com: mv_abcdefghijklmnopqrstuvwxyz restored (landing-restored)
2026/10/12 22:57:20 moves: a.go.example.com: landing mv_abcdefghijklmnopqrstuvwxyz is restored
2026/10/12 22:57:20 moves: a.go.example.com: landing mv_abcdefghijklmnopqrstuvwxyz waits at restored: this box has no probe wake (claim Plan 11's launcher and Plan 5 Task 9's arming are needed)
2026/10/12 22:57:20 moves: a.go.example.com: mv_abcdefghijklmnopqrstuvwxyz requested (requested)
2026/10/12 22:57:20 moves: a.go.example.com: staged the files archive sha256:c9229ef8e7256a1dcd5c2715b11ac3016951e8fee8e0bfa823aa2460f0d331c8 (2 files, 19 bytes)
2026/10/12 22:57:20 moves: a.go.example.com: not ready: the target has not published a readiness report yet
a.go.example.com: v7 (sha256 3fab5c181bd28a09b64397df76ae2bfaf1eac182979b5fdb7a342858004f36af) is not proven on dev; staying on nothing (no proof from dev)
2026/10/12 22:57:20 moves: a.go.example.com: T7: stage v7: release: the held release is not proven for this box: a.go.example.com at v7
2026/10/12 22:57:20 moves: a.go.example.com: mv_abcdefghijklmnopqrstuvwxyz requested (requested)
2026/10/12 22:57:20 moves: a.go.example.com: staged the files archive sha256:c9229ef8e7256a1dcd5c2715b11ac3016951e8fee8e0bfa823aa2460f0d331c8 (2 files, 19 bytes)
2026/10/12 22:57:20 moves: a.go.example.com: not ready: the target has not published a readiness report yet
2026/10/12 22:57:20 moves: a.go.example.com: mv_abcdefghijklmnopqrstuvwxyz requested (requested)
2026/10/12 22:57:20 moves: a.go.example.com: staged the files archive sha256:c9229ef8e7256a1dcd5c2715b11ac3016951e8fee8e0bfa823aa2460f0d331c8 (2 files, 19 bytes)
2026/10/12 22:57:20 moves: a.go.example.com: not ready: the target has not published a readiness report yet
2026/10/12 22:57:20 moves: other.example.com: retired the held refusal of a move that is over
2026/10/12 22:57:20 moves: held.example.com: retired the held refusal of a move that is over
2026/10/12 22:57:20 adopt: other.example.com: retired the held refusal of a move that is over
2026/10/12 22:57:20 adopt: held.example.com: retired the held refusal of a move that is over
2026/10/12 22:57:20 moves: a.go.example.com: mv_abcdefghijklmnopqrstuvwxyz requested (requested)
2026/10/12 22:57:20 moves: a.go.example.com: staged the files archive sha256:c9229ef8e7256a1dcd5c2715b11ac3016951e8fee8e0bfa823aa2460f0d331c8 (2 files, 19 bytes)
2026/10/12 22:57:20 moves: a.go.example.com: not ready: the target has not published a readiness report yet
a.go.example.com: v7: the staged bytes do not match the proof (staged sha256 15e53968bd699ace92f6cd47eeb3396484b0e656851d58d1640477612ea1b681, dev proved 3fab5c181bd28a09b64397df76ae2bfaf1eac182979b5fdb7a342858004f36af); not pointed live
2026/10/12 22:57:20 moves: a.go.example.com: T7: stage v7: release: the held release is not proven for this box: a.go.example.com at v7: the bytes live/a.go.example.com names are not the proven ones
a.go.example.com: v7 (sha256 3fab5c181bd28a09b64397df76ae2bfaf1eac182979b5fdb7a342858004f36af) is not proven on dev; staying on v7 (proof from dev: proof signature does not verify against the configured key)
2026/10/12 22:57:20 moves: a.go.example.com: T7: stage v7: release: the held release is not proven for this box: a.go.example.com at v7
2026/10/12 22:57:20 moves: a.go.example.com: mv_abcdefghijklmnopqrstuvwxyz requested (requested)
2026/10/12 22:57:20 moves: a.go.example.com: staged the files archive sha256:c9229ef8e7256a1dcd5c2715b11ac3016951e8fee8e0bfa823aa2460f0d331c8 (2 files, 19 bytes)
2026/10/12 22:57:20 moves: a.go.example.com: not ready: the target has not published a readiness report yet
2026/10/12 22:57:20 agent: a.go.example.com needs a root pass for v7 (a build only root can stage or install, or an install request only root resolves); asked for one
2026/10/12 22:57:20 moves: a.go.example.com: mv_abcdefghijklmnopqrstuvwxyz requested (requested)
2026/10/12 22:57:20 moves: a.go.example.com: staged the files archive sha256:c9229ef8e7256a1dcd5c2715b11ac3016951e8fee8e0bfa823aa2460f0d331c8 (2 files, 19 bytes)
2026/10/12 22:57:20 moves: a.go.example.com: not ready: the target has not published a readiness report yet
2026/10/12 22:57:20 moves: a.go.example.com: mv_abcdefghijklmnopqrstuvwxyz ready (ready)
2026/10/12 22:57:20 moves: a.go.example.com: move mv_abcdefghijklmnopqrstuvwxyz is ready (last round moved 1815 bytes; bar 16777216)
2026/10/12 22:57:20 moves: a.go.example.com: mv_abcdefghijklmnopqrstuvwxyz cut_requested (cutting-requested)
2026/10/12 22:57:20 moves: a.go.example.com: mv_abcdefghijklmnopqrstuvwxyz admitted (cutting-admitted)
2026/10/12 22:57:20 moves: a.go.example.com: mv_abcdefghijklmnopqrstuvwxyz stopping (cutting-stopping)
2026/10/12 22:57:20 moves: a.go.example.com: cut mv_abcdefghijklmnopqrstuvwxyz is stopping
2026/10/12 22:57:20 moves: a.go.example.com: mv_abcdefghijklmnopqrstuvwxyz stopped (cutting-stopped)
2026/10/12 22:57:20 moves: a.go.example.com: cut mv_abcdefghijklmnopqrstuvwxyz is stopped
2026/10/12 22:57:20 moves: a.go.example.com: stopped at TXID 3; execution given up, citing moves/bab/app/a.go.example.com/mv_abcdefghijklmnopqrstuvwxyz/stopped-source.json
2026/10/12 22:57:20 moves: a.go.example.com: sealed 4 replica objects at TXID 3
2026/10/12 22:57:20 moves: a.go.example.com: mv_abcdefghijklmnopqrstuvwxyz staged (cutting-staged)
2026/10/12 22:57:20 moves: a.go.example.com: cut mv_abcdefghijklmnopqrstuvwxyz is staged
2026/10/12 22:57:20 moves: a.go.example.com: mv_abcdefghijklmnopqrstuvwxyz archived (cutting-archived)
2026/10/12 22:57:20 moves: a.go.example.com: cut mv_abcdefghijklmnopqrstuvwxyz is archived
2026/10/12 22:57:20 moves: a.go.example.com: mv_abcdefghijklmnopqrstuvwxyz flipped (cutting-flipped)
2026/10/12 22:57:20 moves: a.go.example.com: flipped to uk
2026/10/12 22:57:20 moves: a.go.example.com: mv_abcdefghijklmnopqrstuvwxyz parking (landing-parking)
2026/10/12 22:57:20 moves: a.go.example.com: landing mv_abcdefghijklmnopqrstuvwxyz is parking
2026/10/12 22:57:20 moves: a.go.example.com: mv_abcdefghijklmnopqrstuvwxyz building (landing-building)
2026/10/12 22:57:20 moves: a.go.example.com: landing mv_abcdefghijklmnopqrstuvwxyz is building
2026/10/12 22:57:20 moves: a.go.example.com: generation mv_abcdefghijklmnopqrstuvwxyz built at TXID 3
2026/10/12 22:57:20 moves: a.go.example.com: mv_abcdefghijklmnopqrstuvwxyz published (landing-published)
2026/10/12 22:57:20 moves: a.go.example.com: landing mv_abcdefghijklmnopqrstuvwxyz is published
2026/10/12 22:57:20 moves: a.go.example.com: mv_abcdefghijklmnopqrstuvwxyz local (landing-local)
2026/10/12 22:57:20 moves: a.go.example.com: landing mv_abcdefghijklmnopqrstuvwxyz is local
2026/10/12 22:57:20 moves: a.go.example.com: skipped: the prepared files manifest needs C8 (files.BuildPrepared)
2026/10/12 22:57:20 moves: a.go.example.com: files, environment, timers and columns installed; .files-gen written
2026/10/12 22:57:20 moves: a.go.example.com: mv_abcdefghijklmnopqrstuvwxyz configured (landing-configured)
2026/10/12 22:57:20 moves: a.go.example.com: landing mv_abcdefghijklmnopqrstuvwxyz is configured
2026/10/12 22:57:20 moves: a.go.example.com: database installed at /tmp/TestStuckMarksPageAnOperatorthe_targets_return717882244/005/a.go.example.com.db (sha256:ae0661eb68be3f7cf598dba7530492d53b78f5082395b4a1ac8e1f86ea82228d), generation mv_abcdefghijklmnopqrstuvwxyz
2026/10/12 22:57:20 moves: a.go.example.com: mv_abcdefghijklmnopqrstuvwxyz restored (landing-restored)
2026/10/12 22:57:20 moves: a.go.example.com: landing mv_abcdefghijklmnopqrstuvwxyz is restored
2026/10/12 22:57:20 moves: a.go.example.com: landing mv_abcdefghijklmnopqrstuvwxyz waits at restored: this box has no probe wake (claim Plan 11's launcher and Plan 5 Task 9's arming are needed)
2026/10/12 22:57:20 moves: a.go.example.com: landing mv_abcdefghijklmnopqrstuvwxyz is past its cut_deadline (2026-10-07T09:01:55Z)
2026/10/12 22:57:20 moves: a.go.example.com: landing mv_abcdefghijklmnopqrstuvwxyz rolls back as rb_cenksphcgz5quwohq3j7boagmu: the landing ran past its deadline
2026/10/12 22:57:20 moves: a.go.example.com: the claim follows rollback rb_cenksphcgz5quwohq3j7boagmu
2026/10/12 22:57:20 moves: a.go.example.com: the target of rollback rb_cenksphcgz5quwohq3j7boagmu stopped at TXID 3 (clean false, src_covered true)
2026/10/12 22:57:20 moves: a.go.example.com: the target of rollback rb_cenksphcgz5quwohq3j7boagmu is released
2026/10/12 22:57:20 moves: a.go.example.com: the target gave execution up for rollback rb_cenksphcgz5quwohq3j7boagmu, citing moves/bab/app/a.go.example.com/mv_abcdefghijklmnopqrstuvwxyz/stopped-target.json
2026/10/12 22:57:20 moves: a.go.example.com: the target of rollback rb_cenksphcgz5quwohq3j7boagmu is returning
2026/10/12 22:57:20 moves: a.go.example.com: the target's half of rb_cenksphcgz5quwohq3j7boagmu cannot go on (the target's stopped database: open /tmp/TestStuckMarksPageAnOperatorthe_targets_return717882244/005/a.go.example.com.db: no such file or directory); it stops for an operator: the target's stopped database is missing
2026/10/12 22:57:20 moves: a.go.example.com: mv_abcdefghijklmnopqrstuvwxyz requested (requested)
2026/10/12 22:57:20 moves: a.go.example.com: staged the files archive sha256:c9229ef8e7256a1dcd5c2715b11ac3016951e8fee8e0bfa823aa2460f0d331c8 (2 files, 19 bytes)
2026/10/12 22:57:20 moves: a.go.example.com: not ready: the target has not published a readiness report yet
2026/10/12 22:57:20 moves: a.go.example.com: mv_abcdefghijklmnopqrstuvwxyz ready (ready)
2026/10/12 22:57:20 moves: a.go.example.com: move mv_abcdefghijklmnopqrstuvwxyz is ready (last round moved 1815 bytes; bar 16777216)
2026/10/12 22:57:20 moves: a.go.example.com: mv_abcdefghijklmnopqrstuvwxyz cut_requested (cutting-requested)
2026/10/12 22:57:20 moves: a.go.example.com: mv_abcdefghijklmnopqrstuvwxyz admitted (cutting-admitted)
2026/10/12 22:57:20 moves: a.go.example.com: mv_abcdefghijklmnopqrstuvwxyz stopping (cutting-stopping)
2026/10/12 22:57:20 moves: a.go.example.com: cut mv_abcdefghijklmnopqrstuvwxyz is stopping
2026/10/12 22:57:20 moves: a.go.example.com: mv_abcdefghijklmnopqrstuvwxyz stopped (cutting-stopped)
2026/10/12 22:57:20 moves: a.go.example.com: cut mv_abcdefghijklmnopqrstuvwxyz is stopped
2026/10/12 22:57:20 moves: a.go.example.com: stopped at TXID 3; execution given up, citing moves/bab/app/a.go.example.com/mv_abcdefghijklmnopqrstuvwxyz/stopped-source.json
2026/10/12 22:57:20 moves: a.go.example.com: sealed 4 replica objects at TXID 3
2026/10/12 22:57:20 moves: a.go.example.com: mv_abcdefghijklmnopqrstuvwxyz staged (cutting-staged)
2026/10/12 22:57:20 moves: a.go.example.com: cut mv_abcdefghijklmnopqrstuvwxyz is staged
2026/10/12 22:57:20 moves: a.go.example.com: mv_abcdefghijklmnopqrstuvwxyz archived (cutting-archived)
2026/10/12 22:57:20 moves: a.go.example.com: cut mv_abcdefghijklmnopqrstuvwxyz is archived
2026/10/12 22:57:20 moves: a.go.example.com: mv_abcdefghijklmnopqrstuvwxyz flipped (cutting-flipped)
2026/10/12 22:57:20 moves: a.go.example.com: flipped to uk
2026/10/12 22:57:20 moves: a.go.example.com: mv_abcdefghijklmnopqrstuvwxyz parking (landing-parking)
2026/10/12 22:57:20 moves: a.go.example.com: landing mv_abcdefghijklmnopqrstuvwxyz is parking
2026/10/12 22:57:20 moves: a.go.example.com: mv_abcdefghijklmnopqrstuvwxyz building (landing-building)
2026/10/12 22:57:20 moves: a.go.example.com: landing mv_abcdefghijklmnopqrstuvwxyz is building
2026/10/12 22:57:20 moves: a.go.example.com: generation mv_abcdefghijklmnopqrstuvwxyz built at TXID 3
2026/10/12 22:57:20 moves: a.go.example.com: mv_abcdefghijklmnopqrstuvwxyz published (landing-published)
2026/10/12 22:57:20 moves: a.go.example.com: landing mv_abcdefghijklmnopqrstuvwxyz is published
2026/10/12 22:57:20 moves: a.go.example.com: mv_abcdefghijklmnopqrstuvwxyz local (landing-local)
2026/10/12 22:57:20 moves: a.go.example.com: landing mv_abcdefghijklmnopqrstuvwxyz is local
2026/10/12 22:57:20 moves: a.go.example.com: skipped: the prepared files manifest needs C8 (files.BuildPrepared)
2026/10/12 22:57:20 moves: a.go.example.com: files, environment, timers and columns installed; .files-gen written
2026/10/12 22:57:20 moves: a.go.example.com: mv_abcdefghijklmnopqrstuvwxyz configured (landing-configured)
2026/10/12 22:57:20 moves: a.go.example.com: landing mv_abcdefghijklmnopqrstuvwxyz is configured
2026/10/12 22:57:20 moves: a.go.example.com: database installed at /tmp/TestStuckMarksPageAnOperatorthe_sources_recovery4012879393/005/a.go.example.com.db (sha256:ae0661eb68be3f7cf598dba7530492d53b78f5082395b4a1ac8e1f86ea82228d), generation mv_abcdefghijklmnopqrstuvwxyz
2026/10/12 22:57:20 moves: a.go.example.com: mv_abcdefghijklmnopqrstuvwxyz restored (landing-restored)
2026/10/12 22:57:20 moves: a.go.example.com: landing mv_abcdefghijklmnopqrstuvwxyz is restored
2026/10/12 22:57:20 moves: a.go.example.com: landing mv_abcdefghijklmnopqrstuvwxyz waits at restored: this box has no probe wake (claim Plan 11's launcher and Plan 5 Task 9's arming are needed)
2026/10/12 22:57:20 moves: a.go.example.com: landing mv_abcdefghijklmnopqrstuvwxyz is past its cut_deadline (2026-10-07T09:01:55Z)
2026/10/12 22:57:20 moves: a.go.example.com: landing mv_abcdefghijklmnopqrstuvwxyz rolls back as rb_4q3lp6ez3n7xbdjudbz5kcksni: the landing ran past its deadline
2026/10/12 22:57:20 moves: a.go.example.com: the claim follows rollback rb_4q3lp6ez3n7xbdjudbz5kcksni
2026/10/12 22:57:20 moves: a.go.example.com: the target of rollback rb_4q3lp6ez3n7xbdjudbz5kcksni stopped at TXID 3 (clean true, src_covered true)
2026/10/12 22:57:20 moves: a.go.example.com: the target of rollback rb_4q3lp6ez3n7xbdjudbz5kcksni is released
2026/10/12 22:57:20 moves: a.go.example.com: the target gave execution up for rollback rb_4q3lp6ez3n7xbdjudbz5kcksni, citing moves/bab/app/a.go.example.com/mv_abcdefghijklmnopqrstuvwxyz/stopped-target.json
2026/10/12 22:57:20 moves: a.go.example.com: the target keeps what it holds for rollback rb_4q3lp6ez3n7xbdjudbz5kcksni until the source has recovered
2026/10/12 22:57:20 moves: a.go.example.com: the source of rollback rb_4q3lp6ez3n7xbdjudbz5kcksni is claimed
2026/10/12 22:57:20 moves: a.go.example.com: the source executes the host again for rollback rb_4q3lp6ez3n7xbdjudbz5kcksni (clean true)
2026/10/12 22:57:20 moves: a.go.example.com: the source of rollback rb_4q3lp6ez3n7xbdjudbz5kcksni is restoring
2026/10/12 22:57:20 moves: a.go.example.com: the source of rollback rb_4q3lp6ez3n7xbdjudbz5kcksni is verified
2026/10/12 22:57:20 moves: a.go.example.com: the recovery of rb_4q3lp6ez3n7xbdjudbz5kcksni stops for an operator (the files are sha256:d68fb23e5d6334f2d10a29ab126dba3aa01d59b5152875017d5fb47de355967c, recorded sha256:c9229ef8e7256a1dcd5c2715b11ac3016951e8fee8e0bfa823aa2460f0d331c8): the source no longer matches what it recorded when it stopped
2026/10/12 22:57:20 moves: a.go.example.com: mv_abcdefghijklmnopqrstuvwxyz requested (requested)
2026/10/12 22:57:20 moves: a.go.example.com: staged the files archive sha256:c9229ef8e7256a1dcd5c2715b11ac3016951e8fee8e0bfa823aa2460f0d331c8 (2 files, 19 bytes)
2026/10/12 22:57:20 moves: a.go.example.com: not ready: the target has not published a readiness report yet
2026/10/12 22:57:20 moves: a.go.example.com: mv_abcdefghijklmnopqrstuvwxyz ready (ready)
2026/10/12 22:57:20 moves: a.go.example.com: move mv_abcdefghijklmnopqrstuvwxyz is ready (last round moved 1815 bytes; bar 16777216)
2026/10/12 22:57:20 moves: a.go.example.com: mv_abcdefghijklmnopqrstuvwxyz cut_requested (cutting-requested)
2026/10/12 22:57:20 moves: a.go.example.com: mv_abcdefghijklmnopqrstuvwxyz admitted (cutting-admitted)
2026/10/12 22:57:20 moves: a.go.example.com: mv_abcdefghijklmnopqrstuvwxyz stopping (cutting-stopping)
2026/10/12 22:57:20 moves: a.go.example.com: cut mv_abcdefghijklmnopqrstuvwxyz is stopping
2026/10/12 22:57:20 moves: a.go.example.com: mv_abcdefghijklmnopqrstuvwxyz stopped (cutting-stopped)
2026/10/12 22:57:20 moves: a.go.example.com: cut mv_abcdefghijklmnopqrstuvwxyz is stopped
2026/10/12 22:57:20 moves: a.go.example.com: stopped at TXID 3; execution given up, citing moves/bab/app/a.go.example.com/mv_abcdefghijklmnopqrstuvwxyz/stopped-source.json
2026/10/12 22:57:20 moves: a.go.example.com: sealed 4 replica objects at TXID 3
2026/10/12 22:57:20 moves: a.go.example.com: mv_abcdefghijklmnopqrstuvwxyz staged (cutting-staged)
2026/10/12 22:57:20 moves: a.go.example.com: cut mv_abcdefghijklmnopqrstuvwxyz is staged
2026/10/12 22:57:20 moves: a.go.example.com: mv_abcdefghijklmnopqrstuvwxyz archived (cutting-archived)
2026/10/12 22:57:20 moves: a.go.example.com: cut mv_abcdefghijklmnopqrstuvwxyz is archived
2026/10/12 22:57:20 moves: a.go.example.com: mv_abcdefghijklmnopqrstuvwxyz flipped (cutting-flipped)
2026/10/12 22:57:20 moves: a.go.example.com: flipped to uk
2026/10/12 22:57:20 moves: a.go.example.com: mv_abcdefghijklmnopqrstuvwxyz parking (landing-parking)
2026/10/12 22:57:20 moves: a.go.example.com: landing mv_abcdefghijklmnopqrstuvwxyz is parking
2026/10/12 22:57:20 moves: a.go.example.com: mv_abcdefghijklmnopqrstuvwxyz building (landing-building)
2026/10/12 22:57:20 moves: a.go.example.com: landing mv_abcdefghijklmnopqrstuvwxyz is building
2026/10/12 22:57:20 moves: a.go.example.com: generation mv_abcdefghijklmnopqrstuvwxyz built at TXID 3
2026/10/12 22:57:20 moves: a.go.example.com: mv_abcdefghijklmnopqrstuvwxyz published (landing-published)
2026/10/12 22:57:20 moves: a.go.example.com: landing mv_abcdefghijklmnopqrstuvwxyz is published
2026/10/12 22:57:20 moves: a.go.example.com: mv_abcdefghijklmnopqrstuvwxyz local (landing-local)
2026/10/12 22:57:20 moves: a.go.example.com: landing mv_abcdefghijklmnopqrstuvwxyz is local
2026/10/12 22:57:20 moves: a.go.example.com: skipped: the prepared files manifest needs C8 (files.BuildPrepared)
2026/10/12 22:57:20 moves: a.go.example.com: files, environment, timers and columns installed; .files-gen written
2026/10/12 22:57:20 moves: a.go.example.com: mv_abcdefghijklmnopqrstuvwxyz configured (landing-configured)
2026/10/12 22:57:20 moves: a.go.example.com: landing mv_abcdefghijklmnopqrstuvwxyz is configured
2026/10/12 22:57:20 moves: a.go.example.com: database installed at /tmp/TestStuckMarksPageAnOperatorthe_sources_dirty_return_install3554612506/005/a.go.example.com.db (sha256:ae0661eb68be3f7cf598dba7530492d53b78f5082395b4a1ac8e1f86ea82228d), generation mv_abcdefghijklmnopqrstuvwxyz
2026/10/12 22:57:20 moves: a.go.example.com: mv_abcdefghijklmnopqrstuvwxyz restored (landing-restored)
2026/10/12 22:57:20 moves: a.go.example.com: landing mv_abcdefghijklmnopqrstuvwxyz is restored
2026/10/12 22:57:20 moves: a.go.example.com: landing mv_abcdefghijklmnopqrstuvwxyz waits at restored: this box has no probe wake (claim Plan 11's launcher and Plan 5 Task 9's arming are needed)
2026/10/12 22:57:20 moves: a.go.example.com: landing mv_abcdefghijklmnopqrstuvwxyz is past its cut_deadline (2026-10-07T09:01:55Z)
2026/10/12 22:57:20 moves: a.go.example.com: landing mv_abcdefghijklmnopqrstuvwxyz rolls back as rb_5jqeoar6qxvyxe56wehfc32nyq: the landing ran past its deadline
2026/10/12 22:57:20 moves: a.go.example.com: the claim follows rollback rb_5jqeoar6qxvyxe56wehfc32nyq
2026/10/12 22:57:20 moves: a.go.example.com: the target of rollback rb_5jqeoar6qxvyxe56wehfc32nyq stopped at TXID 4 (clean false, src_covered true)
2026/10/12 22:57:20 moves: a.go.example.com: the target of rollback rb_5jqeoar6qxvyxe56wehfc32nyq is released
2026/10/12 22:57:20 moves: a.go.example.com: the target gave execution up for rollback rb_5jqeoar6qxvyxe56wehfc32nyq, citing moves/bab/app/a.go.example.com/mv_abcdefghijklmnopqrstuvwxyz/stopped-target.json
2026/10/12 22:57:20 moves: a.go.example.com: the target of rollback rb_5jqeoar6qxvyxe56wehfc32nyq is returning
2026/10/12 22:57:20 moves: a.go.example.com: the return lane of rb_5jqeoar6qxvyxe56wehfc32nyq is sealed: 1 replica objects at TXID 4
2026/10/12 22:57:20 moves: a.go.example.com: the target of rollback rb_5jqeoar6qxvyxe56wehfc32nyq is returned
2026/10/12 22:57:20 moves: a.go.example.com: the target keeps what it holds for rollback rb_5jqeoar6qxvyxe56wehfc32nyq until the source has recovered
2026/10/12 22:57:20 moves: a.go.example.com: the source of rollback rb_5jqeoar6qxvyxe56wehfc32nyq is claimed
2026/10/12 22:57:20 moves: a.go.example.com: the source executes the host again for rollback rb_5jqeoar6qxvyxe56wehfc32nyq (clean false)
2026/10/12 22:57:20 moves: a.go.example.com: the source of rollback rb_5jqeoar6qxvyxe56wehfc32nyq is restoring
2026/10/12 22:57:20 moves: a.go.example.com: the source of rollback rb_5jqeoar6qxvyxe56wehfc32nyq is parking
2026/10/12 22:57:20 moves: a.go.example.com: the return parks nothing: the target never admitted move mv_abcdefghijklmnopqrstuvwxyz: the claim holds no admission of its generation; the pre-move generation stays the source's live state
2026/10/12 22:57:20 moves: a.go.example.com: return mv_abcdefghijklmnopqrstuvwxyz is building
2026/10/12 22:57:20 moves: a.go.example.com: generation rb_5jqeoar6qxvyxe56wehfc32nyq built at TXID 4
2026/10/12 22:57:20 moves: a.go.example.com: return mv_abcdefghijklmnopqrstuvwxyz is published
2026/10/12 22:57:20 moves: a.go.example.com: return mv_abcdefghijklmnopqrstuvwxyz is local
2026/10/12 22:57:20 moves: a.go.example.com: the local artifacts of rb_5jqeoar6qxvyxe56wehfc32nyq are parked in /tmp/TestStuckMarksPageAnOperatorthe_sources_dirty_return_install3554612506/002/.parked/cc4fe64fa6da6232556d039690a7c19690555ce5d542faba1e57a3d3c896c3bb/a.go.example.com/47d8d0f4c296372f581cc4a722e5d8f6
2026/10/12 22:57:20 moves: a.go.example.com: the files paths hold what this install did not write: /tmp/TestStuckMarksPageAnOperatorthe_sources_dirty_return_install3554612506/002/a.go.example.com holds sentinel.txt; nothing was moved or removed
2026/10/12 22:57:20 moves: a.go.example.com: the return rb_5jqeoar6qxvyxe56wehfc32nyq stops for an operator: the files directory holds files the return did not write
2026/10/12 22:57:20 moves: a.go.example.com: the return of rb_5jqeoar6qxvyxe56wehfc32nyq stopped for an operator: the files directory holds files the return did not write
2026/10/12 22:57:20 moves: a.go.example.com: mv_abcdefghijklmnopqrstuvwxyz requested (requested)
2026/10/12 22:57:20 moves: a.go.example.com: staged the files archive sha256:c9229ef8e7256a1dcd5c2715b11ac3016951e8fee8e0bfa823aa2460f0d331c8 (2 files, 19 bytes)
2026/10/12 22:57:20 moves: a.go.example.com: not ready: the target has not published a readiness report yet
2026/10/12 22:57:20 moves: a.go.example.com: mv_abcdefghijklmnopqrstuvwxyz ready (ready)
2026/10/12 22:57:20 moves: a.go.example.com: move mv_abcdefghijklmnopqrstuvwxyz is ready (last round moved 2101248 bytes; bar 16777216)
2026/10/12 22:57:20 moves: a.go.example.com: mv_abcdefghijklmnopqrstuvwxyz cut_requested (cutting-requested)
2026/10/12 22:57:20 moves: a.go.example.com: mv_abcdefghijklmnopqrstuvwxyz admitted (cutting-admitted)
2026/10/12 22:57:20 moves: a.go.example.com: mv_abcdefghijklmnopqrstuvwxyz stopping (cutting-stopping)
2026/10/12 22:57:20 moves: a.go.example.com: cut mv_abcdefghijklmnopqrstuvwxyz is stopping
2026/10/12 22:57:20 moves: a.go.example.com: mv_abcdefghijklmnopqrstuvwxyz stopped (cutting-stopped)
2026/10/12 22:57:20 moves: a.go.example.com: cut mv_abcdefghijklmnopqrstuvwxyz is stopped
2026/10/12 22:57:20 moves: a.go.example.com: stopped at TXID 3; execution given up, citing moves/bab/app/a.go.example.com/mv_abcdefghijklmnopqrstuvwxyz/stopped-source.json
2026/10/12 22:57:20 moves: a.go.example.com: injected crash
2026/10/12 22:57:20 moves: a.go.example.com: mv_abcdefghijklmnopqrstuvwxyz reverting (reverting)
2026/10/12 22:57:20 moves: a.go.example.com: cut mv_abcdefghijklmnopqrstuvwxyz reverts: the cut ran past its deadline
2026/10/12 22:57:20 moves: a.go.example.com: the claim reverts to this box; the staging lane in uk is queued for its park
2026/10/12 22:57:20 moves: a.go.example.com: the revert of mv_abcdefghijklmnopqrstuvwxyz is at verify
2026/10/12 22:57:20 moves: a.go.example.com: the revert of mv_abcdefghijklmnopqrstuvwxyz stops for an operator (the files are sha256:1781ba4522d139763ba2e65325bb810ecf425689c07b611c358ea98a487d2caa, recorded sha256:c9229ef8e7256a1dcd5c2715b11ac3016951e8fee8e0bfa823aa2460f0d331c8): the source no longer matches what it recorded when it stopped
created /tmp/TestBootstrapOffcloudAcceptsRepairedCredentials2755784404/001/data/registry.db
wrote /tmp/TestBootstrapOffcloudAcceptsRepairedCredentials2755784404/001/etc/systemd/system/carlos-adopt.path
wrote /tmp/TestBootstrapOffcloudAcceptsRepairedCredentials2755784404/001/etc/systemd/system/carlos-adopt.service
wrote /tmp/TestBootstrapOffcloudAcceptsRepairedCredentials2755784404/001/etc/systemd/system/carlos-adopt.timer
wrote /tmp/TestBootstrapOffcloudAcceptsRepairedCredentials2755784404/001/etc/systemd/system/carlos-app@.service
wrote /tmp/TestBootstrapOffcloudAcceptsRepairedCredentials2755784404/001/etc/systemd/system/carlos-app@.socket
wrote /tmp/TestBootstrapOffcloudAcceptsRepairedCredentials2755784404/001/etc/systemd/system/carlos-config-sync.service
wrote /tmp/TestBootstrapOffcloudAcceptsRepairedCredentials2755784404/001/etc/systemd/system/carlos-edge.service
wrote /tmp/TestBootstrapOffcloudAcceptsRepairedCredentials2755784404/001/etc/systemd/system/carlos-hibernate-verify.service
wrote /tmp/TestBootstrapOffcloudAcceptsRepairedCredentials2755784404/001/etc/systemd/system/carlos-hibernate-verify.timer
wrote /tmp/TestBootstrapOffcloudAcceptsRepairedCredentials2755784404/001/etc/systemd/system/carlos-instance-control.service
wrote /tmp/TestBootstrapOffcloudAcceptsRepairedCredentials2755784404/001/etc/systemd/system/carlos-instance-control.socket
wrote /tmp/TestBootstrapOffcloudAcceptsRepairedCredentials2755784404/001/etc/systemd/system/carlos-instance-gen@.service
wrote /tmp/TestBootstrapOffcloudAcceptsRepairedCredentials2755784404/001/etc/systemd/system/carlos-instance-probe.service
wrote /tmp/TestBootstrapOffcloudAcceptsRepairedCredentials2755784404/001/etc/systemd/system/carlos-instance-seed@.service
wrote /tmp/TestBootstrapOffcloudAcceptsRepairedCredentials2755784404/001/etc/systemd/system/carlos-instance-sidecar@.service
wrote /tmp/TestBootstrapOffcloudAcceptsRepairedCredentials2755784404/001/etc/systemd/system/carlos-instance@.service
wrote /tmp/TestBootstrapOffcloudAcceptsRepairedCredentials2755784404/001/etc/systemd/system/carlos-instances.slice
wrote /tmp/TestBootstrapOffcloudAcceptsRepairedCredentials2755784404/001/etc/systemd/system/carlos-litestream.service
wrote /tmp/TestBootstrapOffcloudAcceptsRepairedCredentials2755784404/001/etc/systemd/system/carlos-relay.service
wrote /tmp/TestBootstrapOffcloudAcceptsRepairedCredentials2755784404/001/etc/systemd/system/carlos-restart.path
wrote /tmp/TestBootstrapOffcloudAcceptsRepairedCredentials2755784404/001/etc/systemd/system/carlos-restart.service
wrote /tmp/TestBootstrapOffcloudAcceptsRepairedCredentials2755784404/001/etc/systemd/system/carlos-restore-verify.service
wrote /tmp/TestBootstrapOffcloudAcceptsRepairedCredentials2755784404/001/etc/systemd/system/carlos-restore-verify.timer
wrote /tmp/TestBootstrapOffcloudAcceptsRepairedCredentials2755784404/001/etc/systemd/system/carlos-stage.path
wrote /tmp/TestBootstrapOffcloudAcceptsRepairedCredentials2755784404/001/etc/systemd/system/carlos-stage.service
wrote /tmp/TestBootstrapOffcloudAcceptsRepairedCredentials2755784404/001/etc/systemd/system/carlos-system-recover.service
wrote /tmp/TestBootstrapOffcloudAcceptsRepairedCredentials2755784404/001/etc/systemd/system/carlos-system-update.path
wrote /tmp/TestBootstrapOffcloudAcceptsRepairedCredentials2755784404/001/etc/systemd/system/carlos-system-update.service
wrote /tmp/TestBootstrapOffcloudAcceptsRepairedCredentials2755784404/001/etc/systemd/system/carlos.slice
wrote /tmp/TestBootstrapOffcloudAcceptsRepairedCredentials2755784404/001/etc/systemd/system/data-instances.mount
wrote /tmp/TestBootstrapOffcloudAcceptsRepairedCredentials2755784404/001/etc/systemd/system/run-carlos-instances.mount
wrote /tmp/TestBootstrapOffcloudAcceptsRepairedCredentials2755784404/001/etc/systemd/system/run-carlos-logs.mount
wrote /tmp/TestBootstrapOffcloudAcceptsRepairedCredentials2755784404/001/etc/sysusers.d/carlos.conf
wrote /tmp/TestBootstrapOffcloudAcceptsRepairedCredentials2755784404/001/etc/systemd/journald.conf.d/50-cap.conf
wrote /tmp/TestBootstrapOffcloudAcceptsRepairedCredentials2755784404/001/etc/tmpfiles.d/carlos.conf
bootstrap: staged under /tmp/TestBootstrapOffcloudAcceptsRepairedCredentials2755784404/001 — on a real box (--root /) it would also:
useradd --system --user-group --no-create-home --shell /usr/sbin/nologin carlos
groupadd --system carlos-sockets
chown -R carlos:carlos /data
chown -R carlos:carlos-sockets /run/carlos
chown -R carlos:carlos /run/carlos/apps
chmod 2750 /run/carlos
systemctl daemon-reload
systemctl enable carlos-edge.service carlos-relay.service carlos-adopt.timer carlos-adopt.path carlos-stage.path carlos-restart.path carlos-system-update.path carlos-system-recover.service data-instances.mount run-carlos-instances.mount run-carlos-logs.mount carlos-instance-control.socket
systemctl start carlos-relay.service carlos-adopt.timer carlos-adopt.path carlos-stage.path carlos-restart.path carlos-system-update.path data-instances.mount run-carlos-instances.mount run-carlos-logs.mount carlos-instance-control.socket
systemctl restart systemd-journald
offcloud bootstrap complete.
ensure /etc/carlos/host.env sets:
AWS_SHARED_CREDENTIALS_FILE=/etc/carlos/credentials
then stage /opt/carlos/carlos and start carlos-edge.service (see infra/remote-host.md)
2026/10/12 22:57:21 moves: a.go.example.com: mv_abcdefghijklmnopqrstuvwxyz requested (requested)
2026/10/12 22:57:21 moves: a.go.example.com: staged the files archive sha256:c9229ef8e7256a1dcd5c2715b11ac3016951e8fee8e0bfa823aa2460f0d331c8 (2 files, 19 bytes)
2026/10/12 22:57:21 moves: a.go.example.com: not ready: the target has not published a readiness report yet
2026/10/12 22:57:21 moves: a.go.example.com: mv_abcdefghijklmnopqrstuvwxyz ready (ready)
2026/10/12 22:57:21 moves: a.go.example.com: move mv_abcdefghijklmnopqrstuvwxyz is ready (last round moved 1815 bytes; bar 16777216)
2026/10/12 22:57:21 moves: a.go.example.com: mv_abcdefghijklmnopqrstuvwxyz cut_requested (cutting-requested)
2026/10/12 22:57:21 moves: a.go.example.com: mv_abcdefghijklmnopqrstuvwxyz admitted (cutting-admitted)
2026/10/12 22:57:21 moves: a.go.example.com: mv_abcdefghijklmnopqrstuvwxyz stopping (cutting-stopping)
2026/10/12 22:57:21 moves: a.go.example.com: cut mv_abcdefghijklmnopqrstuvwxyz is stopping
2026/10/12 22:57:21 moves: a.go.example.com: mv_abcdefghijklmnopqrstuvwxyz stopped (cutting-stopped)
2026/10/12 22:57:21 moves: a.go.example.com: cut mv_abcdefghijklmnopqrstuvwxyz is stopped
2026/10/12 22:57:21 moves: a.go.example.com: stopped at TXID 3; execution given up, citing moves/bab/app/a.go.example.com/mv_abcdefghijklmnopqrstuvwxyz/stopped-source.json
2026/10/12 22:57:21 moves: a.go.example.com: sealed 4 replica objects at TXID 3
2026/10/12 22:57:21 moves: a.go.example.com: mv_abcdefghijklmnopqrstuvwxyz staged (cutting-staged)
2026/10/12 22:57:21 moves: a.go.example.com: cut mv_abcdefghijklmnopqrstuvwxyz is staged
2026/10/12 22:57:21 moves: a.go.example.com: mv_abcdefghijklmnopqrstuvwxyz archived (cutting-archived)
2026/10/12 22:57:21 moves: a.go.example.com: cut mv_abcdefghijklmnopqrstuvwxyz is archived
2026/10/12 22:57:21 moves: a.go.example.com: mv_abcdefghijklmnopqrstuvwxyz flipped (cutting-flipped)
2026/10/12 22:57:21 moves: a.go.example.com: flipped to uk
2026/10/12 22:57:21 moves: a.go.example.com: mv_abcdefghijklmnopqrstuvwxyz parking (landing-parking)
2026/10/12 22:57:21 moves: a.go.example.com: landing mv_abcdefghijklmnopqrstuvwxyz is parking
2026/10/12 22:57:21 moves: a.go.example.com: mv_abcdefghijklmnopqrstuvwxyz building (landing-building)
2026/10/12 22:57:21 moves: a.go.example.com: landing mv_abcdefghijklmnopqrstuvwxyz is building
2026/10/12 22:57:21 moves: a.go.example.com: generation mv_abcdefghijklmnopqrstuvwxyz built at TXID 3
2026/10/12 22:57:21 moves: a.go.example.com: mv_abcdefghijklmnopqrstuvwxyz published (landing-published)
2026/10/12 22:57:21 moves: a.go.example.com: landing mv_abcdefghijklmnopqrstuvwxyz is published
2026/10/12 22:57:21 moves: a.go.example.com: mv_abcdefghijklmnopqrstuvwxyz local (landing-local)
2026/10/12 22:57:21 moves: a.go.example.com: landing mv_abcdefghijklmnopqrstuvwxyz is local
2026/10/12 22:57:21 moves: a.go.example.com: skipped: the prepared files manifest needs C8 (files.BuildPrepared)
2026/10/12 22:57:21 moves: a.go.example.com: files, environment, timers and columns installed; .files-gen written
2026/10/12 22:57:21 moves: a.go.example.com: mv_abcdefghijklmnopqrstuvwxyz configured (landing-configured)
2026/10/12 22:57:21 moves: a.go.example.com: landing mv_abcdefghijklmnopqrstuvwxyz is configured
2026/10/12 22:57:21 moves: a.go.example.com: database installed at /tmp/TestPlan10J2MissingWakeKeepsRecoveryAtProbing561374562/005/a.go.example.com.db (sha256:ae0661eb68be3f7cf598dba7530492d53b78f5082395b4a1ac8e1f86ea82228d), generation mv_abcdefghijklmnopqrstuvwxyz
2026/10/12 22:57:21 moves: a.go.example.com: mv_abcdefghijklmnopqrstuvwxyz restored (landing-restored)
2026/10/12 22:57:21 moves: a.go.example.com: landing mv_abcdefghijklmnopqrstuvwxyz is restored
2026/10/12 22:57:21 moves: a.go.example.com: landing mv_abcdefghijklmnopqrstuvwxyz waits at restored: this box has no probe wake (claim Plan 11's launcher and Plan 5 Task 9's arming are needed)
2026/10/12 22:57:21 moves: a.go.example.com: landing mv_abcdefghijklmnopqrstuvwxyz is past its cut_deadline (2026-10-07T09:01:55Z)
2026/10/12 22:57:21 moves: a.go.example.com: landing mv_abcdefghijklmnopqrstuvwxyz rolls back as rb_lxs3lh4ytrjw5cbzixl7zwpfte: the landing ran past its deadline
2026/10/12 22:57:21 moves: a.go.example.com: the claim follows rollback rb_lxs3lh4ytrjw5cbzixl7zwpfte
2026/10/12 22:57:21 moves: a.go.example.com: the target of rollback rb_lxs3lh4ytrjw5cbzixl7zwpfte stopped at TXID 3 (clean true, src_covered true)
2026/10/12 22:57:21 moves: a.go.example.com: the target of rollback rb_lxs3lh4ytrjw5cbzixl7zwpfte is released
2026/10/12 22:57:21 moves: a.go.example.com: the target gave execution up for rollback rb_lxs3lh4ytrjw5cbzixl7zwpfte, citing moves/bab/app/a.go.example.com/mv_abcdefghijklmnopqrstuvwxyz/stopped-target.json
2026/10/12 22:57:21 moves: a.go.example.com: the target keeps what it holds for rollback rb_lxs3lh4ytrjw5cbzixl7zwpfte until the source has recovered
2026/10/12 22:57:21 moves: a.go.example.com: the source of rollback rb_lxs3lh4ytrjw5cbzixl7zwpfte is claimed
2026/10/12 22:57:21 moves: a.go.example.com: the source executes the host again for rollback rb_lxs3lh4ytrjw5cbzixl7zwpfte (clean true)
2026/10/12 22:57:21 moves: a.go.example.com: the source of rollback rb_lxs3lh4ytrjw5cbzixl7zwpfte is restoring
2026/10/12 22:57:21 moves: a.go.example.com: the source of rollback rb_lxs3lh4ytrjw5cbzixl7zwpfte is verified
2026/10/12 22:57:21 moves: a.go.example.com: the source still matches what it recorded when it stopped
2026/10/12 22:57:21 moves: a.go.example.com: the source of rollback rb_lxs3lh4ytrjw5cbzixl7zwpfte is probing
2026/10/12 22:57:21 moves: a.go.example.com: the recovery probe of rollback rb_lxs3lh4ytrjw5cbzixl7zwpfte waits: host claims: missing provider: activator recovery probe entry
2026/10/12 22:57:21 moves: a.go.example.com: mv_abcdefghijklmnopqrstuvwxyz requested (requested)
2026/10/12 22:57:21 moves: a.go.example.com: staged the files archive sha256:c9229ef8e7256a1dcd5c2715b11ac3016951e8fee8e0bfa823aa2460f0d331c8 (2 files, 19 bytes)
2026/10/12 22:57:21 moves: a.go.example.com: not ready: the target has not published a readiness report yet
2026/10/12 22:57:21 moves: a.go.example.com: mv_abcdefghijklmnopqrstuvwxyz ready (ready)
2026/10/12 22:57:21 moves: a.go.example.com: move mv_abcdefghijklmnopqrstuvwxyz is ready (last round moved 1815 bytes; bar 16777216)
2026/10/12 22:57:21 moves: a.go.example.com: mv_abcdefghijklmnopqrstuvwxyz cut_requested (cutting-requested)
2026/10/12 22:57:21 moves: a.go.example.com: mv_abcdefghijklmnopqrstuvwxyz admitted (cutting-admitted)
2026/10/12 22:57:21 moves: a.go.example.com: mv_abcdefghijklmnopqrstuvwxyz stopping (cutting-stopping)
2026/10/12 22:57:21 moves: a.go.example.com: cut mv_abcdefghijklmnopqrstuvwxyz is stopping
2026/10/12 22:57:21 moves: a.go.example.com: mv_abcdefghijklmnopqrstuvwxyz stopped (cutting-stopped)
2026/10/12 22:57:21 moves: a.go.example.com: cut mv_abcdefghijklmnopqrstuvwxyz is stopped
2026/10/12 22:57:21 moves: a.go.example.com: stopped at TXID 3; execution given up, citing moves/bab/app/a.go.example.com/mv_abcdefghijklmnopqrstuvwxyz/stopped-source.json
2026/10/12 22:57:21 moves: a.go.example.com: sealed 4 replica objects at TXID 3
2026/10/12 22:57:21 moves: a.go.example.com: mv_abcdefghijklmnopqrstuvwxyz staged (cutting-staged)
2026/10/12 22:57:21 moves: a.go.example.com: cut mv_abcdefghijklmnopqrstuvwxyz is staged
2026/10/12 22:57:21 moves: a.go.example.com: mv_abcdefghijklmnopqrstuvwxyz archived (cutting-archived)
2026/10/12 22:57:21 moves: a.go.example.com: cut mv_abcdefghijklmnopqrstuvwxyz is archived
2026/10/12 22:57:21 moves: a.go.example.com: mv_abcdefghijklmnopqrstuvwxyz flipped (cutting-flipped)
2026/10/12 22:57:21 moves: a.go.example.com: flipped to uk
2026/10/12 22:57:21 moves: a.go.example.com: mv_abcdefghijklmnopqrstuvwxyz parking (landing-parking)
2026/10/12 22:57:21 moves: a.go.example.com: landing mv_abcdefghijklmnopqrstuvwxyz is parking
2026/10/12 22:57:21 moves: a.go.example.com: mv_abcdefghijklmnopqrstuvwxyz building (landing-building)
2026/10/12 22:57:21 moves: a.go.example.com: landing mv_abcdefghijklmnopqrstuvwxyz is building
2026/10/12 22:57:21 moves: a.go.example.com: generation mv_abcdefghijklmnopqrstuvwxyz built at TXID 3
2026/10/12 22:57:21 moves: a.go.example.com: mv_abcdefghijklmnopqrstuvwxyz published (landing-published)
2026/10/12 22:57:21 moves: a.go.example.com: landing mv_abcdefghijklmnopqrstuvwxyz is published
2026/10/12 22:57:21 moves: a.go.example.com: mv_abcdefghijklmnopqrstuvwxyz local (landing-local)
2026/10/12 22:57:21 moves: a.go.example.com: landing mv_abcdefghijklmnopqrstuvwxyz is local
2026/10/12 22:57:21 moves: a.go.example.com: skipped: the prepared files manifest needs C8 (files.BuildPrepared)
2026/10/12 22:57:21 moves: a.go.example.com: files, environment, timers and columns installed; .files-gen written
2026/10/12 22:57:21 moves: a.go.example.com: mv_abcdefghijklmnopqrstuvwxyz configured (landing-configured)
2026/10/12 22:57:21 moves: a.go.example.com: landing mv_abcdefghijklmnopqrstuvwxyz is configured
2026/10/12 22:57:21 moves: a.go.example.com: database installed at /tmp/TestPlan10J2AdapterDrivesTheRecoveryPassdirty=false3808229842/005/a.go.example.com.db (sha256:ae0661eb68be3f7cf598dba7530492d53b78f5082395b4a1ac8e1f86ea82228d), generation mv_abcdefghijklmnopqrstuvwxyz
2026/10/12 22:57:21 moves: a.go.example.com: mv_abcdefghijklmnopqrstuvwxyz restored (landing-restored)
2026/10/12 22:57:21 moves: a.go.example.com: landing mv_abcdefghijklmnopqrstuvwxyz is restored
2026/10/12 22:57:21 moves: a.go.example.com: landing mv_abcdefghijklmnopqrstuvwxyz waits at restored: this box has no probe wake (claim Plan 11's launcher and Plan 5 Task 9's arming are needed)
2026/10/12 22:57:21 moves: a.go.example.com: landing mv_abcdefghijklmnopqrstuvwxyz is past its cut_deadline (2026-10-07T09:01:55Z)
2026/10/12 22:57:21 moves: a.go.example.com: landing mv_abcdefghijklmnopqrstuvwxyz rolls back as rb_uqhh75doxt6uonjlg2ny3nnttu: the landing ran past its deadline
2026/10/12 22:57:21 moves: a.go.example.com: the claim follows rollback rb_uqhh75doxt6uonjlg2ny3nnttu
2026/10/12 22:57:21 moves: a.go.example.com: the target of rollback rb_uqhh75doxt6uonjlg2ny3nnttu stopped at TXID 3 (clean true, src_covered true)
2026/10/12 22:57:21 moves: a.go.example.com: the target of rollback rb_uqhh75doxt6uonjlg2ny3nnttu is released
2026/10/12 22:57:21 moves: a.go.example.com: the target gave execution up for rollback rb_uqhh75doxt6uonjlg2ny3nnttu, citing moves/bab/app/a.go.example.com/mv_abcdefghijklmnopqrstuvwxyz/stopped-target.json
2026/10/12 22:57:21 moves: a.go.example.com: the target keeps what it holds for rollback rb_uqhh75doxt6uonjlg2ny3nnttu until the source has recovered
2026/10/12 22:57:21 moves: a.go.example.com: the source of rollback rb_uqhh75doxt6uonjlg2ny3nnttu is claimed
2026/10/12 22:57:21 moves: a.go.example.com: the source executes the host again for rollback rb_uqhh75doxt6uonjlg2ny3nnttu (clean true)
2026/10/12 22:57:21 moves: a.go.example.com: the source of rollback rb_uqhh75doxt6uonjlg2ny3nnttu is restoring
2026/10/12 22:57:21 moves: a.go.example.com: the source of rollback rb_uqhh75doxt6uonjlg2ny3nnttu is verified
2026/10/12 22:57:21 moves: a.go.example.com: the source still matches what it recorded when it stopped
2026/10/12 22:57:21 moves: a.go.example.com: the source of rollback rb_uqhh75doxt6uonjlg2ny3nnttu is probing
2026/10/12 22:57:21 moves: a.go.example.com: the source of rollback rb_uqhh75doxt6uonjlg2ny3nnttu is probed
2026/10/12 22:57:21 moves: a.go.example.com: the source of rollback rb_uqhh75doxt6uonjlg2ny3nnttu is recovered
2026/10/12 22:57:21 moves: a.go.example.com: the source of rollback rb_uqhh75doxt6uonjlg2ny3nnttu is opened
2026/10/12 22:57:21 moves: a.go.example.com: rollback rb_uqhh75doxt6uonjlg2ny3nnttu waits for the target's park before it closes
2026/10/12 22:57:21 moves: a.go.example.com: mv_abcdefghijklmnopqrstuvwxyz requested (requested)
2026/10/12 22:57:21 moves: a.go.example.com: staged the files archive sha256:c9229ef8e7256a1dcd5c2715b11ac3016951e8fee8e0bfa823aa2460f0d331c8 (2 files, 19 bytes)
2026/10/12 22:57:21 moves: a.go.example.com: not ready: the target has not published a readiness report yet
2026/10/12 22:57:21 moves: a.go.example.com: mv_abcdefghijklmnopqrstuvwxyz ready (ready)
2026/10/12 22:57:21 moves: a.go.example.com: move mv_abcdefghijklmnopqrstuvwxyz is ready (last round moved 1815 bytes; bar 16777216)
2026/10/12 22:57:21 moves: a.go.example.com: mv_abcdefghijklmnopqrstuvwxyz cut_requested (cutting-requested)
2026/10/12 22:57:21 moves: a.go.example.com: mv_abcdefghijklmnopqrstuvwxyz admitted (cutting-admitted)
2026/10/12 22:57:21 moves: a.go.example.com: mv_abcdefghijklmnopqrstuvwxyz stopping (cutting-stopping)
2026/10/12 22:57:21 moves: a.go.example.com: cut mv_abcdefghijklmnopqrstuvwxyz is stopping
2026/10/12 22:57:21 moves: a.go.example.com: mv_abcdefghijklmnopqrstuvwxyz stopped (cutting-stopped)
2026/10/12 22:57:21 moves: a.go.example.com: cut mv_abcdefghijklmnopqrstuvwxyz is stopped
2026/10/12 22:57:21 moves: a.go.example.com: stopped at TXID 3; execution given up, citing moves/bab/app/a.go.example.com/mv_abcdefghijklmnopqrstuvwxyz/stopped-source.json
2026/10/12 22:57:21 moves: a.go.example.com: sealed 4 replica objects at TXID 3
2026/10/12 22:57:21 moves: a.go.example.com: mv_abcdefghijklmnopqrstuvwxyz staged (cutting-staged)
2026/10/12 22:57:21 moves: a.go.example.com: cut mv_abcdefghijklmnopqrstuvwxyz is staged
2026/10/12 22:57:21 moves: a.go.example.com: mv_abcdefghijklmnopqrstuvwxyz archived (cutting-archived)
2026/10/12 22:57:21 moves: a.go.example.com: cut mv_abcdefghijklmnopqrstuvwxyz is archived
2026/10/12 22:57:21 moves: a.go.example.com: mv_abcdefghijklmnopqrstuvwxyz flipped (cutting-flipped)
2026/10/12 22:57:21 moves: a.go.example.com: flipped to uk
2026/10/12 22:57:21 moves: a.go.example.com: mv_abcdefghijklmnopqrstuvwxyz parking (landing-parking)
2026/10/12 22:57:21 moves: a.go.example.com: landing mv_abcdefghijklmnopqrstuvwxyz is parking
2026/10/12 22:57:21 moves: a.go.example.com: mv_abcdefghijklmnopqrstuvwxyz building (landing-building)
2026/10/12 22:57:21 moves: a.go.example.com: landing mv_abcdefghijklmnopqrstuvwxyz is building
2026/10/12 22:57:21 moves: a.go.example.com: generation mv_abcdefghijklmnopqrstuvwxyz built at TXID 3
2026/10/12 22:57:21 moves: a.go.example.com: mv_abcdefghijklmnopqrstuvwxyz published (landing-published)
2026/10/12 22:57:21 moves: a.go.example.com: landing mv_abcdefghijklmnopqrstuvwxyz is published
2026/10/12 22:57:21 moves: a.go.example.com: mv_abcdefghijklmnopqrstuvwxyz local (landing-local)
2026/10/12 22:57:21 moves: a.go.example.com: landing mv_abcdefghijklmnopqrstuvwxyz is local
2026/10/12 22:57:21 moves: a.go.example.com: skipped: the prepared files manifest needs C8 (files.BuildPrepared)
2026/10/12 22:57:21 moves: a.go.example.com: files, environment, timers and columns installed; .files-gen written
2026/10/12 22:57:21 moves: a.go.example.com: mv_abcdefghijklmnopqrstuvwxyz configured (landing-configured)
2026/10/12 22:57:21 moves: a.go.example.com: landing mv_abcdefghijklmnopqrstuvwxyz is configured
2026/10/12 22:57:21 moves: a.go.example.com: database installed at /tmp/TestPlan10J2AdapterDrivesTheRecoveryPassdirty=true1569310574/005/a.go.example.com.db (sha256:ae0661eb68be3f7cf598dba7530492d53b78f5082395b4a1ac8e1f86ea82228d), generation mv_abcdefghijklmnopqrstuvwxyz
2026/10/12 22:57:21 moves: a.go.example.com: mv_abcdefghijklmnopqrstuvwxyz restored (landing-restored)
2026/10/12 22:57:21 moves: a.go.example.com: landing mv_abcdefghijklmnopqrstuvwxyz is restored
2026/10/12 22:57:21 moves: a.go.example.com: landing mv_abcdefghijklmnopqrstuvwxyz waits at restored: this box has no probe wake (claim Plan 11's launcher and Plan 5 Task 9's arming are needed)
2026/10/12 22:57:21 moves: a.go.example.com: landing mv_abcdefghijklmnopqrstuvwxyz is past its cut_deadline (2026-10-07T09:01:55Z)
2026/10/12 22:57:21 moves: a.go.example.com: landing mv_abcdefghijklmnopqrstuvwxyz rolls back as rb_6qthmvkrtdg2ybs3utonkaehvu: the landing ran past its deadline
2026/10/12 22:57:21 moves: a.go.example.com: the claim follows rollback rb_6qthmvkrtdg2ybs3utonkaehvu
2026/10/12 22:57:21 moves: a.go.example.com: the target of rollback rb_6qthmvkrtdg2ybs3utonkaehvu stopped at TXID 4 (clean false, src_covered true)
2026/10/12 22:57:21 moves: a.go.example.com: the target of rollback rb_6qthmvkrtdg2ybs3utonkaehvu is released
2026/10/12 22:57:21 moves: a.go.example.com: the target gave execution up for rollback rb_6qthmvkrtdg2ybs3utonkaehvu, citing moves/bab/app/a.go.example.com/mv_abcdefghijklmnopqrstuvwxyz/stopped-target.json
2026/10/12 22:57:21 moves: a.go.example.com: the target of rollback rb_6qthmvkrtdg2ybs3utonkaehvu is returning
2026/10/12 22:57:21 moves: a.go.example.com: the return lane of rb_6qthmvkrtdg2ybs3utonkaehvu is sealed: 1 replica objects at TXID 4
2026/10/12 22:57:21 moves: a.go.example.com: the target of rollback rb_6qthmvkrtdg2ybs3utonkaehvu is returned
2026/10/12 22:57:21 moves: a.go.example.com: the target keeps what it holds for rollback rb_6qthmvkrtdg2ybs3utonkaehvu until the source has recovered
2026/10/12 22:57:21 moves: a.go.example.com: the source of rollback rb_6qthmvkrtdg2ybs3utonkaehvu is claimed
2026/10/12 22:57:21 moves: a.go.example.com: the source executes the host again for rollback rb_6qthmvkrtdg2ybs3utonkaehvu (clean false)
2026/10/12 22:57:21 moves: a.go.example.com: the source of rollback rb_6qthmvkrtdg2ybs3utonkaehvu is restoring
2026/10/12 22:57:21 moves: a.go.example.com: the source of rollback rb_6qthmvkrtdg2ybs3utonkaehvu is parking
2026/10/12 22:57:21 moves: a.go.example.com: the return parks nothing: the target never admitted move mv_abcdefghijklmnopqrstuvwxyz: the claim holds no admission of its generation; the pre-move generation stays the source's live state
2026/10/12 22:57:21 moves: a.go.example.com: return mv_abcdefghijklmnopqrstuvwxyz is building
2026/10/12 22:57:21 moves: a.go.example.com: generation rb_6qthmvkrtdg2ybs3utonkaehvu built at TXID 4
2026/10/12 22:57:21 moves: a.go.example.com: return mv_abcdefghijklmnopqrstuvwxyz is published
2026/10/12 22:57:21 moves: a.go.example.com: return mv_abcdefghijklmnopqrstuvwxyz is local
2026/10/12 22:57:21 moves: a.go.example.com: the local artifacts of rb_6qthmvkrtdg2ybs3utonkaehvu are parked in /tmp/TestPlan10J2AdapterDrivesTheRecoveryPassdirty=true1569310574/002/.parked/cc4fe64fa6da6232556d039690a7c19690555ce5d542faba1e57a3d3c896c3bb/a.go.example.com/8ae65dc790dca81ebcf58b33666f6326
2026/10/12 22:57:21 moves: a.go.example.com: skipped: the prepared files manifest needs C8 (files.BuildPrepared)
2026/10/12 22:57:21 moves: a.go.example.com: files, environment, timers and columns installed; .files-gen written
2026/10/12 22:57:21 moves: a.go.example.com: return mv_abcdefghijklmnopqrstuvwxyz is configured
2026/10/12 22:57:21 moves: a.go.example.com: database installed at /tmp/TestPlan10J2AdapterDrivesTheRecoveryPassdirty=true1569310574/002/a.go.example.com.db (sha256:36961681634ab864688d66670609ccc9ab1264cadf9267d3a8110bdd623988ba), generation rb_6qthmvkrtdg2ybs3utonkaehvu
2026/10/12 22:57:21 moves: a.go.example.com: return mv_abcdefghijklmnopqrstuvwxyz is restored
2026/10/12 22:57:21 moves: a.go.example.com: the source of rollback rb_6qthmvkrtdg2ybs3utonkaehvu is probing
2026/10/12 22:57:21 moves: a.go.example.com: the source of rollback rb_6qthmvkrtdg2ybs3utonkaehvu is probed
2026/10/12 22:57:21 moves: a.go.example.com: the source of rollback rb_6qthmvkrtdg2ybs3utonkaehvu is recovered
2026/10/12 22:57:21 moves: a.go.example.com: the source of rollback rb_6qthmvkrtdg2ybs3utonkaehvu is opened
2026/10/12 22:57:21 moves: a.go.example.com: rollback rb_6qthmvkrtdg2ybs3utonkaehvu waits for the target's park before it closes
2026/10/12 22:57:21 moves: a.go.example.com: mv_abcdefghijklmnopqrstuvwxyz requested (requested)
2026/10/12 22:57:21 moves: a.go.example.com: staged the files archive sha256:c9229ef8e7256a1dcd5c2715b11ac3016951e8fee8e0bfa823aa2460f0d331c8 (2 files, 19 bytes)
2026/10/12 22:57:21 moves: a.go.example.com: not ready: the target has not published a readiness report yet
2026/10/12 22:57:21 moves: a.go.example.com: mv_abcdefghijklmnopqrstuvwxyz ready (ready)
2026/10/12 22:57:21 moves: a.go.example.com: move mv_abcdefghijklmnopqrstuvwxyz is ready (last round moved 2101248 bytes; bar 16777216)
2026/10/12 22:57:21 moves: a.go.example.com: mv_abcdefghijklmnopqrstuvwxyz cut_requested (cutting-requested)
2026/10/12 22:57:21 moves: a.go.example.com: mv_abcdefghijklmnopqrstuvwxyz admitted (cutting-admitted)
2026/10/12 22:57:21 moves: a.go.example.com: mv_abcdefghijklmnopqrstuvwxyz stopping (cutting-stopping)
2026/10/12 22:57:21 moves: a.go.example.com: cut mv_abcdefghijklmnopqrstuvwxyz is stopping
2026/10/12 22:57:21 moves: a.go.example.com: mv_abcdefghijklmnopqrstuvwxyz stopped (cutting-stopped)
2026/10/12 22:57:21 moves: a.go.example.com: cut mv_abcdefghijklmnopqrstuvwxyz is stopped
2026/10/12 22:57:21 moves: a.go.example.com: stopped at TXID 3; execution given up, citing moves/bab/app/a.go.example.com/mv_abcdefghijklmnopqrstuvwxyz/stopped-source.json
2026/10/12 22:57:21 moves: a.go.example.com: injected crash
2026/10/12 22:57:21 moves: a.go.example.com: mv_abcdefghijklmnopqrstuvwxyz reverting (reverting)
2026/10/12 22:57:21 moves: a.go.example.com: cut mv_abcdefghijklmnopqrstuvwxyz reverts: the cut ran past its deadline
2026/10/12 22:57:21 moves: a.go.example.com: the claim reverts to this box; the staging lane in uk is queued for its park
2026/10/12 22:57:21 moves: a.go.example.com: the revert of mv_abcdefghijklmnopqrstuvwxyz is at verify
2026/10/12 22:57:21 moves: a.go.example.com: the source still matches what it recorded when it stopped
2026/10/12 22:57:21 moves: a.go.example.com: the source answered its probe; the revert of mv_abcdefghijklmnopqrstuvwxyz is probed
2026/10/12 22:57:21 moves: a.go.example.com: the source serves again; the revert of mv_abcdefghijklmnopqrstuvwxyz waits for its lane's park
2026/10/12 22:57:21 moves: a.go.example.com: the revert of mv_abcdefghijklmnopqrstuvwxyz waits for 1 chore(s) of the move, its lane's park, to be acknowledged
2026/10/12 22:57:21 moves: a.go.example.com: mv_abcdefghijklmnopqrstuvwxyz requested (requested)
2026/10/12 22:57:21 moves: a.go.example.com: staged the files archive sha256:c9229ef8e7256a1dcd5c2715b11ac3016951e8fee8e0bfa823aa2460f0d331c8 (2 files, 19 bytes)
2026/10/12 22:57:21 moves: a.go.example.com: not ready: the target has not published a readiness report yet
2026/10/12 22:57:21 moves: a.go.example.com: mv_abcdefghijklmnopqrstuvwxyz ready (ready)
2026/10/12 22:57:21 moves: a.go.example.com: move mv_abcdefghijklmnopqrstuvwxyz is ready (last round moved 2101248 bytes; bar 16777216)
2026/10/12 22:57:21 moves: a.go.example.com: mv_abcdefghijklmnopqrstuvwxyz cut_requested (cutting-requested)
2026/10/12 22:57:21 moves: a.go.example.com: mv_abcdefghijklmnopqrstuvwxyz admitted (cutting-admitted)
2026/10/12 22:57:21 moves: a.go.example.com: mv_abcdefghijklmnopqrstuvwxyz stopping (cutting-stopping)
2026/10/12 22:57:21 moves: a.go.example.com: cut mv_abcdefghijklmnopqrstuvwxyz is stopping
2026/10/12 22:57:21 moves: a.go.example.com: mv_abcdefghijklmnopqrstuvwxyz stopped (cutting-stopped)
2026/10/12 22:57:21 moves: a.go.example.com: cut mv_abcdefghijklmnopqrstuvwxyz is stopped
2026/10/12 22:57:21 moves: a.go.example.com: stopped at TXID 3; execution given up, citing moves/bab/app/a.go.example.com/mv_abcdefghijklmnopqrstuvwxyz/stopped-source.json
2026/10/12 22:57:21 moves: a.go.example.com: injected crash
2026/10/12 22:57:21 moves: a.go.example.com: mv_abcdefghijklmnopqrstuvwxyz reverting (reverting)
2026/10/12 22:57:21 moves: a.go.example.com: cut mv_abcdefghijklmnopqrstuvwxyz reverts: the cut ran past its deadline
2026/10/12 22:57:21 moves: a.go.example.com: the claim reverts to this box; the staging lane in uk is queued for its park
2026/10/12 22:57:21 moves: a.go.example.com: the revert of mv_abcdefghijklmnopqrstuvwxyz is at verify
2026/10/12 22:57:21 moves: a.go.example.com: the source still matches what it recorded when it stopped
2026/10/12 22:57:21 moves: a.go.example.com: mv_abcdefghijklmnopqrstuvwxyz requested (requested)
2026/10/12 22:57:21 moves: a.go.example.com: staged the files archive sha256:c9229ef8e7256a1dcd5c2715b11ac3016951e8fee8e0bfa823aa2460f0d331c8 (2 files, 19 bytes)
2026/10/12 22:57:21 moves: a.go.example.com: not ready: the target has not published a readiness report yet
2026/10/12 22:57:21 moves: a.go.example.com: mv_abcdefghijklmnopqrstuvwxyz ready (ready)
2026/10/12 22:57:21 moves: a.go.example.com: move mv_abcdefghijklmnopqrstuvwxyz is ready (last round moved 2101248 bytes; bar 16777216)
2026/10/12 22:57:21 moves: a.go.example.com: mv_abcdefghijklmnopqrstuvwxyz cut_requested (cutting-requested)
2026/10/12 22:57:21 moves: a.go.example.com: mv_abcdefghijklmnopqrstuvwxyz admitted (cutting-admitted)
2026/10/12 22:57:21 moves: a.go.example.com: mv_abcdefghijklmnopqrstuvwxyz stopping (cutting-stopping)
2026/10/12 22:57:21 moves: a.go.example.com: cut mv_abcdefghijklmnopqrstuvwxyz is stopping
2026/10/12 22:57:21 moves: a.go.example.com: mv_abcdefghijklmnopqrstuvwxyz stopped (cutting-stopped)
2026/10/12 22:57:21 moves: a.go.example.com: cut mv_abcdefghijklmnopqrstuvwxyz is stopped
2026/10/12 22:57:21 moves: a.go.example.com: stopped at TXID 3; execution given up, citing moves/bab/app/a.go.example.com/mv_abcdefghijklmnopqrstuvwxyz/stopped-source.json
2026/10/12 22:57:21 moves: a.go.example.com: injected crash
2026/10/12 22:57:21 moves: a.go.example.com: mv_abcdefghijklmnopqrstuvwxyz reverting (reverting)
2026/10/12 22:57:21 moves: a.go.example.com: cut mv_abcdefghijklmnopqrstuvwxyz reverts: the cut ran past its deadline
2026/10/12 22:57:21 moves: a.go.example.com: the claim reverts to this box; the staging lane in uk is queued for its park
2026/10/12 22:57:21 moves: a.go.example.com: the revert of mv_abcdefghijklmnopqrstuvwxyz is at verify
2026/10/12 22:57:21 moves: a.go.example.com: the source still matches what it recorded when it stopped
2026/10/12 22:57:21 moves: a.go.example.com: the revert's probe of mv_abcdefghijklmnopqrstuvwxyz has not answered yet
2026/10/12 22:57:21 moves: a.go.example.com: the revert of mv_abcdefghijklmnopqrstuvwxyz stops for an operator (9 attempts): the recovery probe ran out of attempts
2026/10/12 22:57:21 moves: a.go.example.com: WARNING: no operator pager is wired on this box, so the stuck-0 page of move mv_abcdefghijklmnopqrstuvwxyz is not sent; the move goes on (Plan 5 Task 9b wires the pager)
2026/10/12 22:57:28 moves: a.go.example.com: mv_abcdefghijklmnopqrstuvwxyz requested (requested)
2026/10/12 22:57:28 moves: a.go.example.com: staged the files archive sha256:c9229ef8e7256a1dcd5c2715b11ac3016951e8fee8e0bfa823aa2460f0d331c8 (2 files, 19 bytes)
2026/10/12 22:57:28 moves: a.go.example.com: not ready: the target has not published a readiness report yet
2026/10/12 22:57:28 moves: a.go.example.com: mv_abcdefghijklmnopqrstuvwxyz aborting (aborting)
2026/10/12 22:57:28 moves: a.go.example.com: the claim aborts move mv_abcdefghijklmnopqrstuvwxyz; the staging lane in uk is queued for its park there
2026/10/12 22:57:28 moves: a.go.example.com: aborting move mv_abcdefghijklmnopqrstuvwxyz waits for uk to park the staging lane (park-replica uk)
2026/10/12 22:57:28 moves: a.go.example.com: move mv_abcdefghijklmnopqrstuvwxyz's lane in is parked
platform.example.com: update to v2 failed: console.service did not come up: unit entered failed state — rolled the symlink back
platform.example.com: update to v2 failed: permission denied — rolled the symlink back
platform.example.com: update to v2 failed: console.service did not come up: not ready after 5s: permission denied — rolled the symlink back
platform.example.com: rolled back to v1 but could not start it: console.service did not come up: not ready after 5s: permission denied — this host may be serving nothing
platform.example.com: update to v2 failed: permission denied — rolled the symlink back
platform.example.com: update to v2 failed: permission denied — rolled the symlink back
platform.example.com: update to v2 failed: permission denied — rolled the symlink back
platform.example.com: update to v2 failed: permission denied — rolled the symlink back
platform.example.com: update to v2 failed: console.service did not come up: not ready after 5s: permission denied — rolled the symlink back
platform.example.com: rolled back to v1 but could not start it: console.service did not come up: not ready after 5s: permission denied — this host may be serving nothing
platform.example.com: update to v2 failed: permission denied — rolled the symlink back
platform.example.com: update to v2 failed: console.service did not come up: settled witness
settled witness — rolled the symlink back
platform.example.com: updated v1 -> v2 (canary/console)
platform.example.com: update to v2 failed: console.service did not come up: not ready after 5s: permission denied — rolled the symlink back
platform.example.com: rolled back to v1 but could not start it: console.service did not come up: not ready after 5s: permission denied — this host may be serving nothing
platform.example.com: update to v2 failed: console.service did not come up: not ready after 5s: permission denied — rolled the symlink back
platform.example.com: update to v2 failed: console.service did not come up: not ready after 5s: permission denied — rolled the symlink back
platform.example.com: rolled back to v1 but could not start it: console.service did not come up: not ready after 5s: permission denied — this host may be serving nothing
platform.example.com: update to v2 failed: console.service did not come up: not ready after 5s: permission denied — rolled the symlink back
platform.example.com: update to v2 failed: console.service did not come up: not ready after 5s: permission denied — rolled the symlink back
platform.example.com: rolled back to v1 but could not start it: permission denied — this host may be serving nothing
platform.example.com: update to v2 failed: console.service did not come up: unit entered failed state — rolled the symlink back
platform.example.com: rolled back to v1 but could not start it: console.service did not come up: systemd restarted it (new invocation) — this host may be serving nothing
created /tmp/TestPlan9AApplyKeepsEnabledControlUnitOnHostEnvErrorCARLOS_HOST4020487760/005/data/registry.db
wrote /tmp/TestPlan9AApplyKeepsEnabledControlUnitOnHostEnvErrorCARLOS_HOST4020487760/005/etc/systemd/system/carlos-adopt.path
wrote /tmp/TestPlan9AApplyKeepsEnabledControlUnitOnHostEnvErrorCARLOS_HOST4020487760/005/etc/systemd/system/carlos-adopt.service
wrote /tmp/TestPlan9AApplyKeepsEnabledControlUnitOnHostEnvErrorCARLOS_HOST4020487760/005/etc/systemd/system/carlos-adopt.timer
wrote /tmp/TestPlan9AApplyKeepsEnabledControlUnitOnHostEnvErrorCARLOS_HOST4020487760/005/etc/systemd/system/carlos-app@.service
wrote /tmp/TestPlan9AApplyKeepsEnabledControlUnitOnHostEnvErrorCARLOS_HOST4020487760/005/etc/systemd/system/carlos-app@.socket
wrote /tmp/TestPlan9AApplyKeepsEnabledControlUnitOnHostEnvErrorCARLOS_HOST4020487760/005/etc/systemd/system/carlos-config-sync.service
wrote /tmp/TestPlan9AApplyKeepsEnabledControlUnitOnHostEnvErrorCARLOS_HOST4020487760/005/etc/systemd/system/carlos-edge.service
wrote /tmp/TestPlan9AApplyKeepsEnabledControlUnitOnHostEnvErrorCARLOS_HOST4020487760/005/etc/systemd/system/carlos-hibernate-verify.service
wrote /tmp/TestPlan9AApplyKeepsEnabledControlUnitOnHostEnvErrorCARLOS_HOST4020487760/005/etc/systemd/system/carlos-hibernate-verify.timer
wrote /tmp/TestPlan9AApplyKeepsEnabledControlUnitOnHostEnvErrorCARLOS_HOST4020487760/005/etc/systemd/system/carlos-instance-control.service
wrote /tmp/TestPlan9AApplyKeepsEnabledControlUnitOnHostEnvErrorCARLOS_HOST4020487760/005/etc/systemd/system/carlos-instance-control.socket
wrote /tmp/TestPlan9AApplyKeepsEnabledControlUnitOnHostEnvErrorCARLOS_HOST4020487760/005/etc/systemd/system/carlos-instance-gen@.service
wrote /tmp/TestPlan9AApplyKeepsEnabledControlUnitOnHostEnvErrorCARLOS_HOST4020487760/005/etc/systemd/system/carlos-instance-probe.service
wrote /tmp/TestPlan9AApplyKeepsEnabledControlUnitOnHostEnvErrorCARLOS_HOST4020487760/005/etc/systemd/system/carlos-instance-seed@.service
wrote /tmp/TestPlan9AApplyKeepsEnabledControlUnitOnHostEnvErrorCARLOS_HOST4020487760/005/etc/systemd/system/carlos-instance-sidecar@.service
wrote /tmp/TestPlan9AApplyKeepsEnabledControlUnitOnHostEnvErrorCARLOS_HOST4020487760/005/etc/systemd/system/carlos-instance@.service
wrote /tmp/TestPlan9AApplyKeepsEnabledControlUnitOnHostEnvErrorCARLOS_HOST4020487760/005/etc/systemd/system/carlos-instances.slice
wrote /tmp/TestPlan9AApplyKeepsEnabledControlUnitOnHostEnvErrorCARLOS_HOST4020487760/005/etc/systemd/system/carlos-litestream.service
wrote /tmp/TestPlan9AApplyKeepsEnabledControlUnitOnHostEnvErrorCARLOS_HOST4020487760/005/etc/systemd/system/carlos-relay.service
wrote /tmp/TestPlan9AApplyKeepsEnabledControlUnitOnHostEnvErrorCARLOS_HOST4020487760/005/etc/systemd/system/carlos-restart.path
wrote /tmp/TestPlan9AApplyKeepsEnabledControlUnitOnHostEnvErrorCARLOS_HOST4020487760/005/etc/systemd/system/carlos-restart.service
wrote /tmp/TestPlan9AApplyKeepsEnabledControlUnitOnHostEnvErrorCARLOS_HOST4020487760/005/etc/systemd/system/carlos-restore-verify.service
wrote /tmp/TestPlan9AApplyKeepsEnabledControlUnitOnHostEnvErrorCARLOS_HOST4020487760/005/etc/systemd/system/carlos-restore-verify.timer
wrote /tmp/TestPlan9AApplyKeepsEnabledControlUnitOnHostEnvErrorCARLOS_HOST4020487760/005/etc/systemd/system/carlos-stage.path
wrote /tmp/TestPlan9AApplyKeepsEnabledControlUnitOnHostEnvErrorCARLOS_HOST4020487760/005/etc/systemd/system/carlos-stage.service
wrote /tmp/TestPlan9AApplyKeepsEnabledControlUnitOnHostEnvErrorCARLOS_HOST4020487760/005/etc/systemd/system/carlos-system-recover.service
wrote /tmp/TestPlan9AApplyKeepsEnabledControlUnitOnHostEnvErrorCARLOS_HOST4020487760/005/etc/systemd/system/carlos-system-update.path
wrote /tmp/TestPlan9AApplyKeepsEnabledControlUnitOnHostEnvErrorCARLOS_HOST4020487760/005/etc/systemd/system/carlos-system-update.service
wrote /tmp/TestPlan9AApplyKeepsEnabledControlUnitOnHostEnvErrorCARLOS_HOST4020487760/005/etc/systemd/system/carlos.slice
wrote /tmp/TestPlan9AApplyKeepsEnabledControlUnitOnHostEnvErrorCARLOS_HOST4020487760/005/etc/systemd/system/data-instances.mount
wrote /tmp/TestPlan9AApplyKeepsEnabledControlUnitOnHostEnvErrorCARLOS_HOST4020487760/005/etc/systemd/system/run-carlos-instances.mount
wrote /tmp/TestPlan9AApplyKeepsEnabledControlUnitOnHostEnvErrorCARLOS_HOST4020487760/005/etc/systemd/system/run-carlos-logs.mount
wrote /tmp/TestPlan9AApplyKeepsEnabledControlUnitOnHostEnvErrorCARLOS_HOST4020487760/005/etc/sysusers.d/carlos.conf
wrote /tmp/TestPlan9AApplyKeepsEnabledControlUnitOnHostEnvErrorCARLOS_HOST4020487760/005/etc/systemd/journald.conf.d/50-cap.conf
wrote /tmp/TestPlan9AApplyKeepsEnabledControlUnitOnHostEnvErrorCARLOS_HOST4020487760/005/etc/tmpfiles.d/carlos.conf
bootstrap: staged under /tmp/TestPlan9AApplyKeepsEnabledControlUnitOnHostEnvErrorCARLOS_HOST4020487760/005 — on a real box (--root /) it would also:
useradd --system --user-group --no-create-home --shell /usr/sbin/nologin carlos
groupadd --system carlos-sockets
chown -R carlos:carlos /data
chown -R carlos:carlos-sockets /run/carlos
chown -R carlos:carlos /run/carlos/apps
chmod 2750 /run/carlos
systemctl daemon-reload
systemctl enable carlos-edge.service carlos-relay.service carlos-adopt.timer carlos-adopt.path carlos-stage.path carlos-restart.path carlos-system-update.path carlos-system-recover.service data-instances.mount run-carlos-instances.mount run-carlos-logs.mount carlos-instance-control.socket
systemctl start carlos-relay.service carlos-adopt.timer carlos-adopt.path carlos-stage.path carlos-restart.path carlos-system-update.path data-instances.mount run-carlos-instances.mount run-carlos-logs.mount carlos-instance-control.socket
systemctl restart systemd-journald
created /tmp/TestPlan9AApplyKeepsEnabledControlUnitOnHostEnvErrorCARLOS_HOST2046518160/005/data/registry.db
wrote /tmp/TestPlan9AApplyKeepsEnabledControlUnitOnHostEnvErrorCARLOS_HOST2046518160/005/etc/systemd/system/carlos-adopt.path
wrote /tmp/TestPlan9AApplyKeepsEnabledControlUnitOnHostEnvErrorCARLOS_HOST2046518160/005/etc/systemd/system/carlos-adopt.service
wrote /tmp/TestPlan9AApplyKeepsEnabledControlUnitOnHostEnvErrorCARLOS_HOST2046518160/005/etc/systemd/system/carlos-adopt.timer
wrote /tmp/TestPlan9AApplyKeepsEnabledControlUnitOnHostEnvErrorCARLOS_HOST2046518160/005/etc/systemd/system/carlos-app@.service
wrote /tmp/TestPlan9AApplyKeepsEnabledControlUnitOnHostEnvErrorCARLOS_HOST2046518160/005/etc/systemd/system/carlos-app@.socket
wrote /tmp/TestPlan9AApplyKeepsEnabledControlUnitOnHostEnvErrorCARLOS_HOST2046518160/005/etc/systemd/system/carlos-config-sync.service
wrote /tmp/TestPlan9AApplyKeepsEnabledControlUnitOnHostEnvErrorCARLOS_HOST2046518160/005/etc/systemd/system/carlos-edge.service
wrote /tmp/TestPlan9AApplyKeepsEnabledControlUnitOnHostEnvErrorCARLOS_HOST2046518160/005/etc/systemd/system/carlos-hibernate-verify.service
wrote /tmp/TestPlan9AApplyKeepsEnabledControlUnitOnHostEnvErrorCARLOS_HOST2046518160/005/etc/systemd/system/carlos-hibernate-verify.timer
wrote /tmp/TestPlan9AApplyKeepsEnabledControlUnitOnHostEnvErrorCARLOS_HOST2046518160/005/etc/systemd/system/carlos-instance-control.service
wrote /tmp/TestPlan9AApplyKeepsEnabledControlUnitOnHostEnvErrorCARLOS_HOST2046518160/005/etc/systemd/system/carlos-instance-control.socket
wrote /tmp/TestPlan9AApplyKeepsEnabledControlUnitOnHostEnvErrorCARLOS_HOST2046518160/005/etc/systemd/system/carlos-instance-gen@.service
wrote /tmp/TestPlan9AApplyKeepsEnabledControlUnitOnHostEnvErrorCARLOS_HOST2046518160/005/etc/systemd/system/carlos-instance-probe.service
wrote /tmp/TestPlan9AApplyKeepsEnabledControlUnitOnHostEnvErrorCARLOS_HOST2046518160/005/etc/systemd/system/carlos-instance-seed@.service
wrote /tmp/TestPlan9AApplyKeepsEnabledControlUnitOnHostEnvErrorCARLOS_HOST2046518160/005/etc/systemd/system/carlos-instance-sidecar@.service
wrote /tmp/TestPlan9AApplyKeepsEnabledControlUnitOnHostEnvErrorCARLOS_HOST2046518160/005/etc/systemd/system/carlos-instance@.service
wrote /tmp/TestPlan9AApplyKeepsEnabledControlUnitOnHostEnvErrorCARLOS_HOST2046518160/005/etc/systemd/system/carlos-instances.slice
wrote /tmp/TestPlan9AApplyKeepsEnabledControlUnitOnHostEnvErrorCARLOS_HOST2046518160/005/etc/systemd/system/carlos-litestream.service
wrote /tmp/TestPlan9AApplyKeepsEnabledControlUnitOnHostEnvErrorCARLOS_HOST2046518160/005/etc/systemd/system/carlos-relay.service
wrote /tmp/TestPlan9AApplyKeepsEnabledControlUnitOnHostEnvErrorCARLOS_HOST2046518160/005/etc/systemd/system/carlos-restart.path
wrote /tmp/TestPlan9AApplyKeepsEnabledControlUnitOnHostEnvErrorCARLOS_HOST2046518160/005/etc/systemd/system/carlos-restart.service
wrote /tmp/TestPlan9AApplyKeepsEnabledControlUnitOnHostEnvErrorCARLOS_HOST2046518160/005/etc/systemd/system/carlos-restore-verify.service
wrote /tmp/TestPlan9AApplyKeepsEnabledControlUnitOnHostEnvErrorCARLOS_HOST2046518160/005/etc/systemd/system/carlos-restore-verify.timer
wrote /tmp/TestPlan9AApplyKeepsEnabledControlUnitOnHostEnvErrorCARLOS_HOST2046518160/005/etc/systemd/system/carlos-stage.path
wrote /tmp/TestPlan9AApplyKeepsEnabledControlUnitOnHostEnvErrorCARLOS_HOST2046518160/005/etc/systemd/system/carlos-stage.service
wrote /tmp/TestPlan9AApplyKeepsEnabledControlUnitOnHostEnvErrorCARLOS_HOST2046518160/005/etc/systemd/system/carlos-system-recover.service
wrote /tmp/TestPlan9AApplyKeepsEnabledControlUnitOnHostEnvErrorCARLOS_HOST2046518160/005/etc/systemd/system/carlos-system-update.path
wrote /tmp/TestPlan9AApplyKeepsEnabledControlUnitOnHostEnvErrorCARLOS_HOST2046518160/005/etc/systemd/system/carlos-system-update.service
wrote /tmp/TestPlan9AApplyKeepsEnabledControlUnitOnHostEnvErrorCARLOS_HOST2046518160/005/etc/systemd/system/carlos.slice
wrote /tmp/TestPlan9AApplyKeepsEnabledControlUnitOnHostEnvErrorCARLOS_HOST2046518160/005/etc/systemd/system/data-instances.mount
wrote /tmp/TestPlan9AApplyKeepsEnabledControlUnitOnHostEnvErrorCARLOS_HOST2046518160/005/etc/systemd/system/run-carlos-instances.mount
wrote /tmp/TestPlan9AApplyKeepsEnabledControlUnitOnHostEnvErrorCARLOS_HOST2046518160/005/etc/systemd/system/run-carlos-logs.mount
wrote /tmp/TestPlan9AApplyKeepsEnabledControlUnitOnHostEnvErrorCARLOS_HOST2046518160/005/etc/sysusers.d/carlos.conf
wrote /tmp/TestPlan9AApplyKeepsEnabledControlUnitOnHostEnvErrorCARLOS_HOST2046518160/005/etc/systemd/journald.conf.d/50-cap.conf
wrote /tmp/TestPlan9AApplyKeepsEnabledControlUnitOnHostEnvErrorCARLOS_HOST2046518160/005/etc/tmpfiles.d/carlos.conf
bootstrap: staged under /tmp/TestPlan9AApplyKeepsEnabledControlUnitOnHostEnvErrorCARLOS_HOST2046518160/005 — on a real box (--root /) it would also:
useradd --system --user-group --no-create-home --shell /usr/sbin/nologin carlos
groupadd --system carlos-sockets
chown -R carlos:carlos /data
chown -R carlos:carlos-sockets /run/carlos
chown -R carlos:carlos /run/carlos/apps
chmod 2750 /run/carlos
systemctl daemon-reload
systemctl enable carlos-edge.service carlos-relay.service carlos-adopt.timer carlos-adopt.path carlos-stage.path carlos-restart.path carlos-system-update.path carlos-system-recover.service data-instances.mount run-carlos-instances.mount run-carlos-logs.mount carlos-instance-control.socket
systemctl start carlos-relay.service carlos-adopt.timer carlos-adopt.path carlos-stage.path carlos-restart.path carlos-system-update.path data-instances.mount run-carlos-instances.mount run-carlos-logs.mount carlos-instance-control.socket
systemctl restart systemd-journald
created /tmp/TestPlan9AApplyKeepsEnabledControlUnitOnHostEnvErrorCARLOS_HOST3370320212/005/data/registry.db
wrote /tmp/TestPlan9AApplyKeepsEnabledControlUnitOnHostEnvErrorCARLOS_HOST3370320212/005/etc/systemd/system/carlos-adopt.path
wrote /tmp/TestPlan9AApplyKeepsEnabledControlUnitOnHostEnvErrorCARLOS_HOST3370320212/005/etc/systemd/system/carlos-adopt.service
wrote /tmp/TestPlan9AApplyKeepsEnabledControlUnitOnHostEnvErrorCARLOS_HOST3370320212/005/etc/systemd/system/carlos-adopt.timer
wrote /tmp/TestPlan9AApplyKeepsEnabledControlUnitOnHostEnvErrorCARLOS_HOST3370320212/005/etc/systemd/system/carlos-app@.service
wrote /tmp/TestPlan9AApplyKeepsEnabledControlUnitOnHostEnvErrorCARLOS_HOST3370320212/005/etc/systemd/system/carlos-app@.socket
wrote /tmp/TestPlan9AApplyKeepsEnabledControlUnitOnHostEnvErrorCARLOS_HOST3370320212/005/etc/systemd/system/carlos-config-sync.service
wrote /tmp/TestPlan9AApplyKeepsEnabledControlUnitOnHostEnvErrorCARLOS_HOST3370320212/005/etc/systemd/system/carlos-edge.service
wrote /tmp/TestPlan9AApplyKeepsEnabledControlUnitOnHostEnvErrorCARLOS_HOST3370320212/005/etc/systemd/system/carlos-hibernate-verify.service
wrote /tmp/TestPlan9AApplyKeepsEnabledControlUnitOnHostEnvErrorCARLOS_HOST3370320212/005/etc/systemd/system/carlos-hibernate-verify.timer
wrote /tmp/TestPlan9AApplyKeepsEnabledControlUnitOnHostEnvErrorCARLOS_HOST3370320212/005/etc/systemd/system/carlos-instance-control.service
wrote /tmp/TestPlan9AApplyKeepsEnabledControlUnitOnHostEnvErrorCARLOS_HOST3370320212/005/etc/systemd/system/carlos-instance-control.socket
wrote /tmp/TestPlan9AApplyKeepsEnabledControlUnitOnHostEnvErrorCARLOS_HOST3370320212/005/etc/systemd/system/carlos-instance-gen@.service
wrote /tmp/TestPlan9AApplyKeepsEnabledControlUnitOnHostEnvErrorCARLOS_HOST3370320212/005/etc/systemd/system/carlos-instance-probe.service
wrote /tmp/TestPlan9AApplyKeepsEnabledControlUnitOnHostEnvErrorCARLOS_HOST3370320212/005/etc/systemd/system/carlos-instance-seed@.service
wrote /tmp/TestPlan9AApplyKeepsEnabledControlUnitOnHostEnvErrorCARLOS_HOST3370320212/005/etc/systemd/system/carlos-instance-sidecar@.service
wrote /tmp/TestPlan9AApplyKeepsEnabledControlUnitOnHostEnvErrorCARLOS_HOST3370320212/005/etc/systemd/system/carlos-instance@.service
wrote /tmp/TestPlan9AApplyKeepsEnabledControlUnitOnHostEnvErrorCARLOS_HOST3370320212/005/etc/systemd/system/carlos-instances.slice
wrote /tmp/TestPlan9AApplyKeepsEnabledControlUnitOnHostEnvErrorCARLOS_HOST3370320212/005/etc/systemd/system/carlos-litestream.service
wrote /tmp/TestPlan9AApplyKeepsEnabledControlUnitOnHostEnvErrorCARLOS_HOST3370320212/005/etc/systemd/system/carlos-relay.service
wrote /tmp/TestPlan9AApplyKeepsEnabledControlUnitOnHostEnvErrorCARLOS_HOST3370320212/005/etc/systemd/system/carlos-restart.path
wrote /tmp/TestPlan9AApplyKeepsEnabledControlUnitOnHostEnvErrorCARLOS_HOST3370320212/005/etc/systemd/system/carlos-restart.service
wrote /tmp/TestPlan9AApplyKeepsEnabledControlUnitOnHostEnvErrorCARLOS_HOST3370320212/005/etc/systemd/system/carlos-restore-verify.service
wrote /tmp/TestPlan9AApplyKeepsEnabledControlUnitOnHostEnvErrorCARLOS_HOST3370320212/005/etc/systemd/system/carlos-restore-verify.timer
wrote /tmp/TestPlan9AApplyKeepsEnabledControlUnitOnHostEnvErrorCARLOS_HOST3370320212/005/etc/systemd/system/carlos-stage.path
wrote /tmp/TestPlan9AApplyKeepsEnabledControlUnitOnHostEnvErrorCARLOS_HOST3370320212/005/etc/systemd/system/carlos-stage.service
wrote /tmp/TestPlan9AApplyKeepsEnabledControlUnitOnHostEnvErrorCARLOS_HOST3370320212/005/etc/systemd/system/carlos-system-recover.service
wrote /tmp/TestPlan9AApplyKeepsEnabledControlUnitOnHostEnvErrorCARLOS_HOST3370320212/005/etc/systemd/system/carlos-system-update.path
wrote /tmp/TestPlan9AApplyKeepsEnabledControlUnitOnHostEnvErrorCARLOS_HOST3370320212/005/etc/systemd/system/carlos-system-update.service
wrote /tmp/TestPlan9AApplyKeepsEnabledControlUnitOnHostEnvErrorCARLOS_HOST3370320212/005/etc/systemd/system/carlos.slice
wrote /tmp/TestPlan9AApplyKeepsEnabledControlUnitOnHostEnvErrorCARLOS_HOST3370320212/005/etc/systemd/system/data-instances.mount
wrote /tmp/TestPlan9AApplyKeepsEnabledControlUnitOnHostEnvErrorCARLOS_HOST3370320212/005/etc/systemd/system/run-carlos-instances.mount
wrote /tmp/TestPlan9AApplyKeepsEnabledControlUnitOnHostEnvErrorCARLOS_HOST3370320212/005/etc/systemd/system/run-carlos-logs.mount
wrote /tmp/TestPlan9AApplyKeepsEnabledControlUnitOnHostEnvErrorCARLOS_HOST3370320212/005/etc/sysusers.d/carlos.conf
wrote /tmp/TestPlan9AApplyKeepsEnabledControlUnitOnHostEnvErrorCARLOS_HOST3370320212/005/etc/systemd/journald.conf.d/50-cap.conf
wrote /tmp/TestPlan9AApplyKeepsEnabledControlUnitOnHostEnvErrorCARLOS_HOST3370320212/005/etc/tmpfiles.d/carlos.conf
bootstrap: staged under /tmp/TestPlan9AApplyKeepsEnabledControlUnitOnHostEnvErrorCARLOS_HOST3370320212/005 — on a real box (--root /) it would also:
useradd --system --user-group --no-create-home --shell /usr/sbin/nologin carlos
groupadd --system carlos-sockets
chown -R carlos:carlos /data
chown -R carlos:carlos-sockets /run/carlos
chown -R carlos:carlos /run/carlos/apps
chmod 2750 /run/carlos
systemctl daemon-reload
systemctl enable carlos-edge.service carlos-relay.service carlos-adopt.timer carlos-adopt.path carlos-stage.path carlos-restart.path carlos-system-update.path carlos-system-recover.service data-instances.mount run-carlos-instances.mount run-carlos-logs.mount carlos-instance-control.socket
systemctl start carlos-relay.service carlos-adopt.timer carlos-adopt.path carlos-stage.path carlos-restart.path carlos-system-update.path data-instances.mount run-carlos-instances.mount run-carlos-logs.mount carlos-instance-control.socket
systemctl restart systemd-journald
bootstrap: instance-control unit settings: unsupported multiline host unit setting
created /tmp/TestPlan9ABootstrapKeepsEnabledControlUnitOnHostEnvErrorCARLOS_3634609253/001/data/registry.db
wrote /tmp/TestPlan9ABootstrapKeepsEnabledControlUnitOnHostEnvErrorCARLOS_3634609253/001/etc/systemd/system/carlos-adopt.path
wrote /tmp/TestPlan9ABootstrapKeepsEnabledControlUnitOnHostEnvErrorCARLOS_3634609253/001/etc/systemd/system/carlos-adopt.service
wrote /tmp/TestPlan9ABootstrapKeepsEnabledControlUnitOnHostEnvErrorCARLOS_3634609253/001/etc/systemd/system/carlos-adopt.timer
wrote /tmp/TestPlan9ABootstrapKeepsEnabledControlUnitOnHostEnvErrorCARLOS_3634609253/001/etc/systemd/system/carlos-app@.service
wrote /tmp/TestPlan9ABootstrapKeepsEnabledControlUnitOnHostEnvErrorCARLOS_3634609253/001/etc/systemd/system/carlos-app@.socket
wrote /tmp/TestPlan9ABootstrapKeepsEnabledControlUnitOnHostEnvErrorCARLOS_3634609253/001/etc/systemd/system/carlos-config-sync.service
wrote /tmp/TestPlan9ABootstrapKeepsEnabledControlUnitOnHostEnvErrorCARLOS_3634609253/001/etc/systemd/system/carlos-edge.service
wrote /tmp/TestPlan9ABootstrapKeepsEnabledControlUnitOnHostEnvErrorCARLOS_3634609253/001/etc/systemd/system/carlos-hibernate-verify.service
wrote /tmp/TestPlan9ABootstrapKeepsEnabledControlUnitOnHostEnvErrorCARLOS_3634609253/001/etc/systemd/system/carlos-hibernate-verify.timer
kept /tmp/TestPlan9ABootstrapKeepsEnabledControlUnitOnHostEnvErrorCARLOS_3634609253/001/etc/systemd/system/carlos-instance-control.service (host.env cannot be parsed)
wrote /tmp/TestPlan9ABootstrapKeepsEnabledControlUnitOnHostEnvErrorCARLOS_3634609253/001/etc/systemd/system/carlos-instance-control.socket
wrote /tmp/TestPlan9ABootstrapKeepsEnabledControlUnitOnHostEnvErrorCARLOS_3634609253/001/etc/systemd/system/carlos-instance-gen@.service
wrote /tmp/TestPlan9ABootstrapKeepsEnabledControlUnitOnHostEnvErrorCARLOS_3634609253/001/etc/systemd/system/carlos-instance-probe.service
wrote /tmp/TestPlan9ABootstrapKeepsEnabledControlUnitOnHostEnvErrorCARLOS_3634609253/001/etc/systemd/system/carlos-instance-seed@.service
wrote /tmp/TestPlan9ABootstrapKeepsEnabledControlUnitOnHostEnvErrorCARLOS_3634609253/001/etc/systemd/system/carlos-instance-sidecar@.service
wrote /tmp/TestPlan9ABootstrapKeepsEnabledControlUnitOnHostEnvErrorCARLOS_3634609253/001/etc/systemd/system/carlos-instance@.service
wrote /tmp/TestPlan9ABootstrapKeepsEnabledControlUnitOnHostEnvErrorCARLOS_3634609253/001/etc/systemd/system/carlos-instances.slice
wrote /tmp/TestPlan9ABootstrapKeepsEnabledControlUnitOnHostEnvErrorCARLOS_3634609253/001/etc/systemd/system/carlos-litestream.service
wrote /tmp/TestPlan9ABootstrapKeepsEnabledControlUnitOnHostEnvErrorCARLOS_3634609253/001/etc/systemd/system/carlos-relay.service
wrote /tmp/TestPlan9ABootstrapKeepsEnabledControlUnitOnHostEnvErrorCARLOS_3634609253/001/etc/systemd/system/carlos-restart.path
wrote /tmp/TestPlan9ABootstrapKeepsEnabledControlUnitOnHostEnvErrorCARLOS_3634609253/001/etc/systemd/system/carlos-restart.service
wrote /tmp/TestPlan9ABootstrapKeepsEnabledControlUnitOnHostEnvErrorCARLOS_3634609253/001/etc/systemd/system/carlos-restore-verify.service
wrote /tmp/TestPlan9ABootstrapKeepsEnabledControlUnitOnHostEnvErrorCARLOS_3634609253/001/etc/systemd/system/carlos-restore-verify.timer
wrote /tmp/TestPlan9ABootstrapKeepsEnabledControlUnitOnHostEnvErrorCARLOS_3634609253/001/etc/systemd/system/carlos-stage.path
wrote /tmp/TestPlan9ABootstrapKeepsEnabledControlUnitOnHostEnvErrorCARLOS_3634609253/001/etc/systemd/system/carlos-stage.service
wrote /tmp/TestPlan9ABootstrapKeepsEnabledControlUnitOnHostEnvErrorCARLOS_3634609253/001/etc/systemd/system/carlos-system-recover.service
wrote /tmp/TestPlan9ABootstrapKeepsEnabledControlUnitOnHostEnvErrorCARLOS_3634609253/001/etc/systemd/system/carlos-system-update.path
wrote /tmp/TestPlan9ABootstrapKeepsEnabledControlUnitOnHostEnvErrorCARLOS_3634609253/001/etc/systemd/system/carlos-system-update.service
wrote /tmp/TestPlan9ABootstrapKeepsEnabledControlUnitOnHostEnvErrorCARLOS_3634609253/001/etc/systemd/system/carlos.slice
wrote /tmp/TestPlan9ABootstrapKeepsEnabledControlUnitOnHostEnvErrorCARLOS_3634609253/001/etc/systemd/system/data-instances.mount
wrote /tmp/TestPlan9ABootstrapKeepsEnabledControlUnitOnHostEnvErrorCARLOS_3634609253/001/etc/systemd/system/run-carlos-instances.mount
wrote /tmp/TestPlan9ABootstrapKeepsEnabledControlUnitOnHostEnvErrorCARLOS_3634609253/001/etc/systemd/system/run-carlos-logs.mount
wrote /tmp/TestPlan9ABootstrapKeepsEnabledControlUnitOnHostEnvErrorCARLOS_3634609253/001/etc/sysusers.d/carlos.conf
wrote /tmp/TestPlan9ABootstrapKeepsEnabledControlUnitOnHostEnvErrorCARLOS_3634609253/001/etc/systemd/journald.conf.d/50-cap.conf
wrote /tmp/TestPlan9ABootstrapKeepsEnabledControlUnitOnHostEnvErrorCARLOS_3634609253/001/etc/tmpfiles.d/carlos.conf
bootstrap: staged under /tmp/TestPlan9ABootstrapKeepsEnabledControlUnitOnHostEnvErrorCARLOS_3634609253/001 — on a real box (--root /) it would also:
useradd --system --user-group --no-create-home --shell /usr/sbin/nologin carlos
groupadd --system carlos-sockets
chown -R carlos:carlos /data
chown -R carlos:carlos-sockets /run/carlos
chown -R carlos:carlos /run/carlos/apps
chmod 2750 /run/carlos
systemctl daemon-reload
systemctl enable carlos-edge.service carlos-relay.service carlos-adopt.timer carlos-adopt.path carlos-stage.path carlos-restart.path carlos-system-update.path carlos-system-recover.service data-instances.mount run-carlos-instances.mount run-carlos-logs.mount carlos-instance-control.socket
systemctl start carlos-relay.service carlos-adopt.timer carlos-adopt.path carlos-stage.path carlos-restart.path carlos-system-update.path data-instances.mount run-carlos-instances.mount run-carlos-logs.mount carlos-instance-control.socket
systemctl restart systemd-journald
bootstrap: instance-control unit settings: unsupported multiline host unit setting
created /tmp/TestPlan9ABootstrapKeepsEnabledControlUnitOnHostEnvErrorCARLOS_2778061932/001/data/registry.db
wrote /tmp/TestPlan9ABootstrapKeepsEnabledControlUnitOnHostEnvErrorCARLOS_2778061932/001/etc/systemd/system/carlos-adopt.path
wrote /tmp/TestPlan9ABootstrapKeepsEnabledControlUnitOnHostEnvErrorCARLOS_2778061932/001/etc/systemd/system/carlos-adopt.service
wrote /tmp/TestPlan9ABootstrapKeepsEnabledControlUnitOnHostEnvErrorCARLOS_2778061932/001/etc/systemd/system/carlos-adopt.timer
wrote /tmp/TestPlan9ABootstrapKeepsEnabledControlUnitOnHostEnvErrorCARLOS_2778061932/001/etc/systemd/system/carlos-app@.service
wrote /tmp/TestPlan9ABootstrapKeepsEnabledControlUnitOnHostEnvErrorCARLOS_2778061932/001/etc/systemd/system/carlos-app@.socket
wrote /tmp/TestPlan9ABootstrapKeepsEnabledControlUnitOnHostEnvErrorCARLOS_2778061932/001/etc/systemd/system/carlos-config-sync.service
wrote /tmp/TestPlan9ABootstrapKeepsEnabledControlUnitOnHostEnvErrorCARLOS_2778061932/001/etc/systemd/system/carlos-edge.service
wrote /tmp/TestPlan9ABootstrapKeepsEnabledControlUnitOnHostEnvErrorCARLOS_2778061932/001/etc/systemd/system/carlos-hibernate-verify.service
wrote /tmp/TestPlan9ABootstrapKeepsEnabledControlUnitOnHostEnvErrorCARLOS_2778061932/001/etc/systemd/system/carlos-hibernate-verify.timer
kept /tmp/TestPlan9ABootstrapKeepsEnabledControlUnitOnHostEnvErrorCARLOS_2778061932/001/etc/systemd/system/carlos-instance-control.service (host.env cannot be parsed)
wrote /tmp/TestPlan9ABootstrapKeepsEnabledControlUnitOnHostEnvErrorCARLOS_2778061932/001/etc/systemd/system/carlos-instance-control.socket
wrote /tmp/TestPlan9ABootstrapKeepsEnabledControlUnitOnHostEnvErrorCARLOS_2778061932/001/etc/systemd/system/carlos-instance-gen@.service
wrote /tmp/TestPlan9ABootstrapKeepsEnabledControlUnitOnHostEnvErrorCARLOS_2778061932/001/etc/systemd/system/carlos-instance-probe.service
wrote /tmp/TestPlan9ABootstrapKeepsEnabledControlUnitOnHostEnvErrorCARLOS_2778061932/001/etc/systemd/system/carlos-instance-seed@.service
wrote /tmp/TestPlan9ABootstrapKeepsEnabledControlUnitOnHostEnvErrorCARLOS_2778061932/001/etc/systemd/system/carlos-instance-sidecar@.service
wrote /tmp/TestPlan9ABootstrapKeepsEnabledControlUnitOnHostEnvErrorCARLOS_2778061932/001/etc/systemd/system/carlos-instance@.service
wrote /tmp/TestPlan9ABootstrapKeepsEnabledControlUnitOnHostEnvErrorCARLOS_2778061932/001/etc/systemd/system/carlos-instances.slice
wrote /tmp/TestPlan9ABootstrapKeepsEnabledControlUnitOnHostEnvErrorCARLOS_2778061932/001/etc/systemd/system/carlos-litestream.service
wrote /tmp/TestPlan9ABootstrapKeepsEnabledControlUnitOnHostEnvErrorCARLOS_2778061932/001/etc/systemd/system/carlos-relay.service
wrote /tmp/TestPlan9ABootstrapKeepsEnabledControlUnitOnHostEnvErrorCARLOS_2778061932/001/etc/systemd/system/carlos-restart.path
wrote /tmp/TestPlan9ABootstrapKeepsEnabledControlUnitOnHostEnvErrorCARLOS_2778061932/001/etc/systemd/system/carlos-restart.service
wrote /tmp/TestPlan9ABootstrapKeepsEnabledControlUnitOnHostEnvErrorCARLOS_2778061932/001/etc/systemd/system/carlos-restore-verify.service
wrote /tmp/TestPlan9ABootstrapKeepsEnabledControlUnitOnHostEnvErrorCARLOS_2778061932/001/etc/systemd/system/carlos-restore-verify.timer
wrote /tmp/TestPlan9ABootstrapKeepsEnabledControlUnitOnHostEnvErrorCARLOS_2778061932/001/etc/systemd/system/carlos-stage.path
wrote /tmp/TestPlan9ABootstrapKeepsEnabledControlUnitOnHostEnvErrorCARLOS_2778061932/001/etc/systemd/system/carlos-stage.service
wrote /tmp/TestPlan9ABootstrapKeepsEnabledControlUnitOnHostEnvErrorCARLOS_2778061932/001/etc/systemd/system/carlos-system-recover.service
wrote /tmp/TestPlan9ABootstrapKeepsEnabledControlUnitOnHostEnvErrorCARLOS_2778061932/001/etc/systemd/system/carlos-system-update.path
wrote /tmp/TestPlan9ABootstrapKeepsEnabledControlUnitOnHostEnvErrorCARLOS_2778061932/001/etc/systemd/system/carlos-system-update.service
wrote /tmp/TestPlan9ABootstrapKeepsEnabledControlUnitOnHostEnvErrorCARLOS_2778061932/001/etc/systemd/system/carlos.slice
wrote /tmp/TestPlan9ABootstrapKeepsEnabledControlUnitOnHostEnvErrorCARLOS_2778061932/001/etc/systemd/system/data-instances.mount
wrote /tmp/TestPlan9ABootstrapKeepsEnabledControlUnitOnHostEnvErrorCARLOS_2778061932/001/etc/systemd/system/run-carlos-instances.mount
wrote /tmp/TestPlan9ABootstrapKeepsEnabledControlUnitOnHostEnvErrorCARLOS_2778061932/001/etc/systemd/system/run-carlos-logs.mount
wrote /tmp/TestPlan9ABootstrapKeepsEnabledControlUnitOnHostEnvErrorCARLOS_2778061932/001/etc/sysusers.d/carlos.conf
wrote /tmp/TestPlan9ABootstrapKeepsEnabledControlUnitOnHostEnvErrorCARLOS_2778061932/001/etc/systemd/journald.conf.d/50-cap.conf
wrote /tmp/TestPlan9ABootstrapKeepsEnabledControlUnitOnHostEnvErrorCARLOS_2778061932/001/etc/tmpfiles.d/carlos.conf
bootstrap: staged under /tmp/TestPlan9ABootstrapKeepsEnabledControlUnitOnHostEnvErrorCARLOS_2778061932/001 — on a real box (--root /) it would also:
useradd --system --user-group --no-create-home --shell /usr/sbin/nologin carlos
groupadd --system carlos-sockets
chown -R carlos:carlos /data
chown -R carlos:carlos-sockets /run/carlos
chown -R carlos:carlos /run/carlos/apps
chmod 2750 /run/carlos
systemctl daemon-reload
systemctl enable carlos-edge.service carlos-relay.service carlos-adopt.timer carlos-adopt.path carlos-stage.path carlos-restart.path carlos-system-update.path carlos-system-recover.service data-instances.mount run-carlos-instances.mount run-carlos-logs.mount carlos-instance-control.socket
systemctl start carlos-relay.service carlos-adopt.timer carlos-adopt.path carlos-stage.path carlos-restart.path carlos-system-update.path data-instances.mount run-carlos-instances.mount run-carlos-logs.mount carlos-instance-control.socket
systemctl restart systemd-journald
bootstrap: instance-control unit settings: unsupported multiline host unit setting
created /tmp/TestPlan9ABootstrapKeepsEnabledControlUnitOnHostEnvErrorCARLOS_642766536/001/data/registry.db
wrote /tmp/TestPlan9ABootstrapKeepsEnabledControlUnitOnHostEnvErrorCARLOS_642766536/001/etc/systemd/system/carlos-adopt.path
wrote /tmp/TestPlan9ABootstrapKeepsEnabledControlUnitOnHostEnvErrorCARLOS_642766536/001/etc/systemd/system/carlos-adopt.service
wrote /tmp/TestPlan9ABootstrapKeepsEnabledControlUnitOnHostEnvErrorCARLOS_642766536/001/etc/systemd/system/carlos-adopt.timer
wrote /tmp/TestPlan9ABootstrapKeepsEnabledControlUnitOnHostEnvErrorCARLOS_642766536/001/etc/systemd/system/carlos-app@.service
wrote /tmp/TestPlan9ABootstrapKeepsEnabledControlUnitOnHostEnvErrorCARLOS_642766536/001/etc/systemd/system/carlos-app@.socket
wrote /tmp/TestPlan9ABootstrapKeepsEnabledControlUnitOnHostEnvErrorCARLOS_642766536/001/etc/systemd/system/carlos-config-sync.service
wrote /tmp/TestPlan9ABootstrapKeepsEnabledControlUnitOnHostEnvErrorCARLOS_642766536/001/etc/systemd/system/carlos-edge.service
wrote /tmp/TestPlan9ABootstrapKeepsEnabledControlUnitOnHostEnvErrorCARLOS_642766536/001/etc/systemd/system/carlos-hibernate-verify.service
wrote /tmp/TestPlan9ABootstrapKeepsEnabledControlUnitOnHostEnvErrorCARLOS_642766536/001/etc/systemd/system/carlos-hibernate-verify.timer
kept /tmp/TestPlan9ABootstrapKeepsEnabledControlUnitOnHostEnvErrorCARLOS_642766536/001/etc/systemd/system/carlos-instance-control.service (host.env cannot be parsed)
wrote /tmp/TestPlan9ABootstrapKeepsEnabledControlUnitOnHostEnvErrorCARLOS_642766536/001/etc/systemd/system/carlos-instance-control.socket
wrote /tmp/TestPlan9ABootstrapKeepsEnabledControlUnitOnHostEnvErrorCARLOS_642766536/001/etc/systemd/system/carlos-instance-gen@.service
wrote /tmp/TestPlan9ABootstrapKeepsEnabledControlUnitOnHostEnvErrorCARLOS_642766536/001/etc/systemd/system/carlos-instance-probe.service
wrote /tmp/TestPlan9ABootstrapKeepsEnabledControlUnitOnHostEnvErrorCARLOS_642766536/001/etc/systemd/system/carlos-instance-seed@.service
wrote /tmp/TestPlan9ABootstrapKeepsEnabledControlUnitOnHostEnvErrorCARLOS_642766536/001/etc/systemd/system/carlos-instance-sidecar@.service
wrote /tmp/TestPlan9ABootstrapKeepsEnabledControlUnitOnHostEnvErrorCARLOS_642766536/001/etc/systemd/system/carlos-instance@.service
wrote /tmp/TestPlan9ABootstrapKeepsEnabledControlUnitOnHostEnvErrorCARLOS_642766536/001/etc/systemd/system/carlos-instances.slice
wrote /tmp/TestPlan9ABootstrapKeepsEnabledControlUnitOnHostEnvErrorCARLOS_642766536/001/etc/systemd/system/carlos-litestream.service
wrote /tmp/TestPlan9ABootstrapKeepsEnabledControlUnitOnHostEnvErrorCARLOS_642766536/001/etc/systemd/system/carlos-relay.service
wrote /tmp/TestPlan9ABootstrapKeepsEnabledControlUnitOnHostEnvErrorCARLOS_642766536/001/etc/systemd/system/carlos-restart.path
wrote /tmp/TestPlan9ABootstrapKeepsEnabledControlUnitOnHostEnvErrorCARLOS_642766536/001/etc/systemd/system/carlos-restart.service
wrote /tmp/TestPlan9ABootstrapKeepsEnabledControlUnitOnHostEnvErrorCARLOS_642766536/001/etc/systemd/system/carlos-restore-verify.service
wrote /tmp/TestPlan9ABootstrapKeepsEnabledControlUnitOnHostEnvErrorCARLOS_642766536/001/etc/systemd/system/carlos-restore-verify.timer
wrote /tmp/TestPlan9ABootstrapKeepsEnabledControlUnitOnHostEnvErrorCARLOS_642766536/001/etc/systemd/system/carlos-stage.path
wrote /tmp/TestPlan9ABootstrapKeepsEnabledControlUnitOnHostEnvErrorCARLOS_642766536/001/etc/systemd/system/carlos-stage.service
wrote /tmp/TestPlan9ABootstrapKeepsEnabledControlUnitOnHostEnvErrorCARLOS_642766536/001/etc/systemd/system/carlos-system-recover.service
wrote /tmp/TestPlan9ABootstrapKeepsEnabledControlUnitOnHostEnvErrorCARLOS_642766536/001/etc/systemd/system/carlos-system-update.path
wrote /tmp/TestPlan9ABootstrapKeepsEnabledControlUnitOnHostEnvErrorCARLOS_642766536/001/etc/systemd/system/carlos-system-update.service
wrote /tmp/TestPlan9ABootstrapKeepsEnabledControlUnitOnHostEnvErrorCARLOS_642766536/001/etc/systemd/system/carlos.slice
wrote /tmp/TestPlan9ABootstrapKeepsEnabledControlUnitOnHostEnvErrorCARLOS_642766536/001/etc/systemd/system/data-instances.mount
wrote /tmp/TestPlan9ABootstrapKeepsEnabledControlUnitOnHostEnvErrorCARLOS_642766536/001/etc/systemd/system/run-carlos-instances.mount
wrote /tmp/TestPlan9ABootstrapKeepsEnabledControlUnitOnHostEnvErrorCARLOS_642766536/001/etc/systemd/system/run-carlos-logs.mount
wrote /tmp/TestPlan9ABootstrapKeepsEnabledControlUnitOnHostEnvErrorCARLOS_642766536/001/etc/sysusers.d/carlos.conf
wrote /tmp/TestPlan9ABootstrapKeepsEnabledControlUnitOnHostEnvErrorCARLOS_642766536/001/etc/systemd/journald.conf.d/50-cap.conf
wrote /tmp/TestPlan9ABootstrapKeepsEnabledControlUnitOnHostEnvErrorCARLOS_642766536/001/etc/tmpfiles.d/carlos.conf
bootstrap: staged under /tmp/TestPlan9ABootstrapKeepsEnabledControlUnitOnHostEnvErrorCARLOS_642766536/001 — on a real box (--root /) it would also:
useradd --system --user-group --no-create-home --shell /usr/sbin/nologin carlos
groupadd --system carlos-sockets
chown -R carlos:carlos /data
chown -R carlos:carlos-sockets /run/carlos
chown -R carlos:carlos /run/carlos/apps
chmod 2750 /run/carlos
systemctl daemon-reload
systemctl enable carlos-edge.service carlos-relay.service carlos-adopt.timer carlos-adopt.path carlos-stage.path carlos-restart.path carlos-system-update.path carlos-system-recover.service data-instances.mount run-carlos-instances.mount run-carlos-logs.mount carlos-instance-control.socket
systemctl start carlos-relay.service carlos-adopt.timer carlos-adopt.path carlos-stage.path carlos-restart.path carlos-system-update.path data-instances.mount run-carlos-instances.mount run-carlos-logs.mount carlos-instance-control.socket
systemctl restart systemd-journald
created /tmp/TestPlan9AFlagOffControlUnitUnchangedOnHostEnvErrorabsent1308337588/005/data/registry.db
wrote /tmp/TestPlan9AFlagOffControlUnitUnchangedOnHostEnvErrorabsent1308337588/005/etc/systemd/system/carlos-adopt.path
wrote /tmp/TestPlan9AFlagOffControlUnitUnchangedOnHostEnvErrorabsent1308337588/005/etc/systemd/system/carlos-adopt.service
wrote /tmp/TestPlan9AFlagOffControlUnitUnchangedOnHostEnvErrorabsent1308337588/005/etc/systemd/system/carlos-adopt.timer
wrote /tmp/TestPlan9AFlagOffControlUnitUnchangedOnHostEnvErrorabsent1308337588/005/etc/systemd/system/carlos-app@.service
wrote /tmp/TestPlan9AFlagOffControlUnitUnchangedOnHostEnvErrorabsent1308337588/005/etc/systemd/system/carlos-app@.socket
wrote /tmp/TestPlan9AFlagOffControlUnitUnchangedOnHostEnvErrorabsent1308337588/005/etc/systemd/system/carlos-config-sync.service
wrote /tmp/TestPlan9AFlagOffControlUnitUnchangedOnHostEnvErrorabsent1308337588/005/etc/systemd/system/carlos-edge.service
wrote /tmp/TestPlan9AFlagOffControlUnitUnchangedOnHostEnvErrorabsent1308337588/005/etc/systemd/system/carlos-hibernate-verify.service
wrote /tmp/TestPlan9AFlagOffControlUnitUnchangedOnHostEnvErrorabsent1308337588/005/etc/systemd/system/carlos-hibernate-verify.timer
wrote /tmp/TestPlan9AFlagOffControlUnitUnchangedOnHostEnvErrorabsent1308337588/005/etc/systemd/system/carlos-instance-control.service
wrote /tmp/TestPlan9AFlagOffControlUnitUnchangedOnHostEnvErrorabsent1308337588/005/etc/systemd/system/carlos-instance-control.socket
wrote /tmp/TestPlan9AFlagOffControlUnitUnchangedOnHostEnvErrorabsent1308337588/005/etc/systemd/system/carlos-instance-gen@.service
wrote /tmp/TestPlan9AFlagOffControlUnitUnchangedOnHostEnvErrorabsent1308337588/005/etc/systemd/system/carlos-instance-probe.service
wrote /tmp/TestPlan9AFlagOffControlUnitUnchangedOnHostEnvErrorabsent1308337588/005/etc/systemd/system/carlos-instance-seed@.service
wrote /tmp/TestPlan9AFlagOffControlUnitUnchangedOnHostEnvErrorabsent1308337588/005/etc/systemd/system/carlos-instance-sidecar@.service
wrote /tmp/TestPlan9AFlagOffControlUnitUnchangedOnHostEnvErrorabsent1308337588/005/etc/systemd/system/carlos-instance@.service
wrote /tmp/TestPlan9AFlagOffControlUnitUnchangedOnHostEnvErrorabsent1308337588/005/etc/systemd/system/carlos-instances.slice
wrote /tmp/TestPlan9AFlagOffControlUnitUnchangedOnHostEnvErrorabsent1308337588/005/etc/systemd/system/carlos-litestream.service
wrote /tmp/TestPlan9AFlagOffControlUnitUnchangedOnHostEnvErrorabsent1308337588/005/etc/systemd/system/carlos-relay.service
wrote /tmp/TestPlan9AFlagOffControlUnitUnchangedOnHostEnvErrorabsent1308337588/005/etc/systemd/system/carlos-restart.path
wrote /tmp/TestPlan9AFlagOffControlUnitUnchangedOnHostEnvErrorabsent1308337588/005/etc/systemd/system/carlos-restart.service
wrote /tmp/TestPlan9AFlagOffControlUnitUnchangedOnHostEnvErrorabsent1308337588/005/etc/systemd/system/carlos-restore-verify.service
wrote /tmp/TestPlan9AFlagOffControlUnitUnchangedOnHostEnvErrorabsent1308337588/005/etc/systemd/system/carlos-restore-verify.timer
wrote /tmp/TestPlan9AFlagOffControlUnitUnchangedOnHostEnvErrorabsent1308337588/005/etc/systemd/system/carlos-stage.path
wrote /tmp/TestPlan9AFlagOffControlUnitUnchangedOnHostEnvErrorabsent1308337588/005/etc/systemd/system/carlos-stage.service
wrote /tmp/TestPlan9AFlagOffControlUnitUnchangedOnHostEnvErrorabsent1308337588/005/etc/systemd/system/carlos-system-recover.service
wrote /tmp/TestPlan9AFlagOffControlUnitUnchangedOnHostEnvErrorabsent1308337588/005/etc/systemd/system/carlos-system-update.path
wrote /tmp/TestPlan9AFlagOffControlUnitUnchangedOnHostEnvErrorabsent1308337588/005/etc/systemd/system/carlos-system-update.service
wrote /tmp/TestPlan9AFlagOffControlUnitUnchangedOnHostEnvErrorabsent1308337588/005/etc/systemd/system/carlos.slice
wrote /tmp/TestPlan9AFlagOffControlUnitUnchangedOnHostEnvErrorabsent1308337588/005/etc/systemd/system/data-instances.mount
wrote /tmp/TestPlan9AFlagOffControlUnitUnchangedOnHostEnvErrorabsent1308337588/005/etc/systemd/system/run-carlos-instances.mount
wrote /tmp/TestPlan9AFlagOffControlUnitUnchangedOnHostEnvErrorabsent1308337588/005/etc/systemd/system/run-carlos-logs.mount
wrote /tmp/TestPlan9AFlagOffControlUnitUnchangedOnHostEnvErrorabsent1308337588/005/etc/sysusers.d/carlos.conf
wrote /tmp/TestPlan9AFlagOffControlUnitUnchangedOnHostEnvErrorabsent1308337588/005/etc/systemd/journald.conf.d/50-cap.conf
wrote /tmp/TestPlan9AFlagOffControlUnitUnchangedOnHostEnvErrorabsent1308337588/005/etc/tmpfiles.d/carlos.conf
bootstrap: staged under /tmp/TestPlan9AFlagOffControlUnitUnchangedOnHostEnvErrorabsent1308337588/005 — on a real box (--root /) it would also:
useradd --system --user-group --no-create-home --shell /usr/sbin/nologin carlos
groupadd --system carlos-sockets
chown -R carlos:carlos /data
chown -R carlos:carlos-sockets /run/carlos
chown -R carlos:carlos /run/carlos/apps
chmod 2750 /run/carlos
systemctl daemon-reload
systemctl enable carlos-edge.service carlos-relay.service carlos-adopt.timer carlos-adopt.path carlos-stage.path carlos-restart.path carlos-system-update.path carlos-system-recover.service data-instances.mount run-carlos-instances.mount run-carlos-logs.mount carlos-instance-control.socket
systemctl start carlos-relay.service carlos-adopt.timer carlos-adopt.path carlos-stage.path carlos-restart.path carlos-system-update.path data-instances.mount run-carlos-instances.mount run-carlos-logs.mount carlos-instance-control.socket
systemctl restart systemd-journald
bootstrap: instance-control unit settings: unsupported multiline host unit setting
created /tmp/TestPlan9AFlagOffControlUnitUnchangedOnHostEnvErrorabsent1308337588/006/data/registry.db
wrote /tmp/TestPlan9AFlagOffControlUnitUnchangedOnHostEnvErrorabsent1308337588/006/etc/systemd/system/carlos-adopt.path
wrote /tmp/TestPlan9AFlagOffControlUnitUnchangedOnHostEnvErrorabsent1308337588/006/etc/systemd/system/carlos-adopt.service
wrote /tmp/TestPlan9AFlagOffControlUnitUnchangedOnHostEnvErrorabsent1308337588/006/etc/systemd/system/carlos-adopt.timer
wrote /tmp/TestPlan9AFlagOffControlUnitUnchangedOnHostEnvErrorabsent1308337588/006/etc/systemd/system/carlos-app@.service
wrote /tmp/TestPlan9AFlagOffControlUnitUnchangedOnHostEnvErrorabsent1308337588/006/etc/systemd/system/carlos-app@.socket
wrote /tmp/TestPlan9AFlagOffControlUnitUnchangedOnHostEnvErrorabsent1308337588/006/etc/systemd/system/carlos-config-sync.service
wrote /tmp/TestPlan9AFlagOffControlUnitUnchangedOnHostEnvErrorabsent1308337588/006/etc/systemd/system/carlos-edge.service
wrote /tmp/TestPlan9AFlagOffControlUnitUnchangedOnHostEnvErrorabsent1308337588/006/etc/systemd/system/carlos-hibernate-verify.service
wrote /tmp/TestPlan9AFlagOffControlUnitUnchangedOnHostEnvErrorabsent1308337588/006/etc/systemd/system/carlos-hibernate-verify.timer
wrote /tmp/TestPlan9AFlagOffControlUnitUnchangedOnHostEnvErrorabsent1308337588/006/etc/systemd/system/carlos-instance-control.service
wrote /tmp/TestPlan9AFlagOffControlUnitUnchangedOnHostEnvErrorabsent1308337588/006/etc/systemd/system/carlos-instance-control.socket
wrote /tmp/TestPlan9AFlagOffControlUnitUnchangedOnHostEnvErrorabsent1308337588/006/etc/systemd/system/carlos-instance-gen@.service
wrote /tmp/TestPlan9AFlagOffControlUnitUnchangedOnHostEnvErrorabsent1308337588/006/etc/systemd/system/carlos-instance-probe.service
wrote /tmp/TestPlan9AFlagOffControlUnitUnchangedOnHostEnvErrorabsent1308337588/006/etc/systemd/system/carlos-instance-seed@.service
wrote /tmp/TestPlan9AFlagOffControlUnitUnchangedOnHostEnvErrorabsent1308337588/006/etc/systemd/system/carlos-instance-sidecar@.service
wrote /tmp/TestPlan9AFlagOffControlUnitUnchangedOnHostEnvErrorabsent1308337588/006/etc/systemd/system/carlos-instance@.service
wrote /tmp/TestPlan9AFlagOffControlUnitUnchangedOnHostEnvErrorabsent1308337588/006/etc/systemd/system/carlos-instances.slice
wrote /tmp/TestPlan9AFlagOffControlUnitUnchangedOnHostEnvErrorabsent1308337588/006/etc/systemd/system/carlos-litestream.service
wrote /tmp/TestPlan9AFlagOffControlUnitUnchangedOnHostEnvErrorabsent1308337588/006/etc/systemd/system/carlos-relay.service
wrote /tmp/TestPlan9AFlagOffControlUnitUnchangedOnHostEnvErrorabsent1308337588/006/etc/systemd/system/carlos-restart.path
wrote /tmp/TestPlan9AFlagOffControlUnitUnchangedOnHostEnvErrorabsent1308337588/006/etc/systemd/system/carlos-restart.service
wrote /tmp/TestPlan9AFlagOffControlUnitUnchangedOnHostEnvErrorabsent1308337588/006/etc/systemd/system/carlos-restore-verify.service
wrote /tmp/TestPlan9AFlagOffControlUnitUnchangedOnHostEnvErrorabsent1308337588/006/etc/systemd/system/carlos-restore-verify.timer
wrote /tmp/TestPlan9AFlagOffControlUnitUnchangedOnHostEnvErrorabsent1308337588/006/etc/systemd/system/carlos-stage.path
wrote /tmp/TestPlan9AFlagOffControlUnitUnchangedOnHostEnvErrorabsent1308337588/006/etc/systemd/system/carlos-stage.service
wrote /tmp/TestPlan9AFlagOffControlUnitUnchangedOnHostEnvErrorabsent1308337588/006/etc/systemd/system/carlos-system-recover.service
wrote /tmp/TestPlan9AFlagOffControlUnitUnchangedOnHostEnvErrorabsent1308337588/006/etc/systemd/system/carlos-system-update.path
wrote /tmp/TestPlan9AFlagOffControlUnitUnchangedOnHostEnvErrorabsent1308337588/006/etc/systemd/system/carlos-system-update.service
wrote /tmp/TestPlan9AFlagOffControlUnitUnchangedOnHostEnvErrorabsent1308337588/006/etc/systemd/system/carlos.slice
wrote /tmp/TestPlan9AFlagOffControlUnitUnchangedOnHostEnvErrorabsent1308337588/006/etc/systemd/system/data-instances.mount
wrote /tmp/TestPlan9AFlagOffControlUnitUnchangedOnHostEnvErrorabsent1308337588/006/etc/systemd/system/run-carlos-instances.mount
wrote /tmp/TestPlan9AFlagOffControlUnitUnchangedOnHostEnvErrorabsent1308337588/006/etc/systemd/system/run-carlos-logs.mount
wrote /tmp/TestPlan9AFlagOffControlUnitUnchangedOnHostEnvErrorabsent1308337588/006/etc/sysusers.d/carlos.conf
wrote /tmp/TestPlan9AFlagOffControlUnitUnchangedOnHostEnvErrorabsent1308337588/006/etc/systemd/journald.conf.d/50-cap.conf
wrote /tmp/TestPlan9AFlagOffControlUnitUnchangedOnHostEnvErrorabsent1308337588/006/etc/tmpfiles.d/carlos.conf
bootstrap: staged under /tmp/TestPlan9AFlagOffControlUnitUnchangedOnHostEnvErrorabsent1308337588/006 — on a real box (--root /) it would also:
useradd --system --user-group --no-create-home --shell /usr/sbin/nologin carlos
groupadd --system carlos-sockets
chown -R carlos:carlos /data
chown -R carlos:carlos-sockets /run/carlos
chown -R carlos:carlos /run/carlos/apps
chmod 2750 /run/carlos
systemctl daemon-reload
systemctl enable carlos-edge.service carlos-relay.service carlos-adopt.timer carlos-adopt.path carlos-stage.path carlos-restart.path carlos-system-update.path carlos-system-recover.service data-instances.mount run-carlos-instances.mount run-carlos-logs.mount carlos-instance-control.socket
systemctl start carlos-relay.service carlos-adopt.timer carlos-adopt.path carlos-stage.path carlos-restart.path carlos-system-update.path data-instances.mount run-carlos-instances.mount run-carlos-logs.mount carlos-instance-control.socket
systemctl restart systemd-journald
created /tmp/TestPlan9AFlagOffControlUnitUnchangedOnHostEnvErrortemplate137490636/005/data/registry.db
wrote /tmp/TestPlan9AFlagOffControlUnitUnchangedOnHostEnvErrortemplate137490636/005/etc/systemd/system/carlos-adopt.path
wrote /tmp/TestPlan9AFlagOffControlUnitUnchangedOnHostEnvErrortemplate137490636/005/etc/systemd/system/carlos-adopt.service
wrote /tmp/TestPlan9AFlagOffControlUnitUnchangedOnHostEnvErrortemplate137490636/005/etc/systemd/system/carlos-adopt.timer
wrote /tmp/TestPlan9AFlagOffControlUnitUnchangedOnHostEnvErrortemplate137490636/005/etc/systemd/system/carlos-app@.service
wrote /tmp/TestPlan9AFlagOffControlUnitUnchangedOnHostEnvErrortemplate137490636/005/etc/systemd/system/carlos-app@.socket
wrote /tmp/TestPlan9AFlagOffControlUnitUnchangedOnHostEnvErrortemplate137490636/005/etc/systemd/system/carlos-config-sync.service
wrote /tmp/TestPlan9AFlagOffControlUnitUnchangedOnHostEnvErrortemplate137490636/005/etc/systemd/system/carlos-edge.service
wrote /tmp/TestPlan9AFlagOffControlUnitUnchangedOnHostEnvErrortemplate137490636/005/etc/systemd/system/carlos-hibernate-verify.service
wrote /tmp/TestPlan9AFlagOffControlUnitUnchangedOnHostEnvErrortemplate137490636/005/etc/systemd/system/carlos-hibernate-verify.timer
wrote /tmp/TestPlan9AFlagOffControlUnitUnchangedOnHostEnvErrortemplate137490636/005/etc/systemd/system/carlos-instance-control.service
wrote /tmp/TestPlan9AFlagOffControlUnitUnchangedOnHostEnvErrortemplate137490636/005/etc/systemd/system/carlos-instance-control.socket
wrote /tmp/TestPlan9AFlagOffControlUnitUnchangedOnHostEnvErrortemplate137490636/005/etc/systemd/system/carlos-instance-gen@.service
wrote /tmp/TestPlan9AFlagOffControlUnitUnchangedOnHostEnvErrortemplate137490636/005/etc/systemd/system/carlos-instance-probe.service
wrote /tmp/TestPlan9AFlagOffControlUnitUnchangedOnHostEnvErrortemplate137490636/005/etc/systemd/system/carlos-instance-seed@.service
wrote /tmp/TestPlan9AFlagOffControlUnitUnchangedOnHostEnvErrortemplate137490636/005/etc/systemd/system/carlos-instance-sidecar@.service
wrote /tmp/TestPlan9AFlagOffControlUnitUnchangedOnHostEnvErrortemplate137490636/005/etc/systemd/system/carlos-instance@.service
wrote /tmp/TestPlan9AFlagOffControlUnitUnchangedOnHostEnvErrortemplate137490636/005/etc/systemd/system/carlos-instances.slice
wrote /tmp/TestPlan9AFlagOffControlUnitUnchangedOnHostEnvErrortemplate137490636/005/etc/systemd/system/carlos-litestream.service
wrote /tmp/TestPlan9AFlagOffControlUnitUnchangedOnHostEnvErrortemplate137490636/005/etc/systemd/system/carlos-relay.service
wrote /tmp/TestPlan9AFlagOffControlUnitUnchangedOnHostEnvErrortemplate137490636/005/etc/systemd/system/carlos-restart.path
wrote /tmp/TestPlan9AFlagOffControlUnitUnchangedOnHostEnvErrortemplate137490636/005/etc/systemd/system/carlos-restart.service
wrote /tmp/TestPlan9AFlagOffControlUnitUnchangedOnHostEnvErrortemplate137490636/005/etc/systemd/system/carlos-restore-verify.service
wrote /tmp/TestPlan9AFlagOffControlUnitUnchangedOnHostEnvErrortemplate137490636/005/etc/systemd/system/carlos-restore-verify.timer
wrote /tmp/TestPlan9AFlagOffControlUnitUnchangedOnHostEnvErrortemplate137490636/005/etc/systemd/system/carlos-stage.path
wrote /tmp/TestPlan9AFlagOffControlUnitUnchangedOnHostEnvErrortemplate137490636/005/etc/systemd/system/carlos-stage.service
wrote /tmp/TestPlan9AFlagOffControlUnitUnchangedOnHostEnvErrortemplate137490636/005/etc/systemd/system/carlos-system-recover.service
wrote /tmp/TestPlan9AFlagOffControlUnitUnchangedOnHostEnvErrortemplate137490636/005/etc/systemd/system/carlos-system-update.path
wrote /tmp/TestPlan9AFlagOffControlUnitUnchangedOnHostEnvErrortemplate137490636/005/etc/systemd/system/carlos-system-update.service
wrote /tmp/TestPlan9AFlagOffControlUnitUnchangedOnHostEnvErrortemplate137490636/005/etc/systemd/system/carlos.slice
wrote /tmp/TestPlan9AFlagOffControlUnitUnchangedOnHostEnvErrortemplate137490636/005/etc/systemd/system/data-instances.mount
wrote /tmp/TestPlan9AFlagOffControlUnitUnchangedOnHostEnvErrortemplate137490636/005/etc/systemd/system/run-carlos-instances.mount
wrote /tmp/TestPlan9AFlagOffControlUnitUnchangedOnHostEnvErrortemplate137490636/005/etc/systemd/system/run-carlos-logs.mount
wrote /tmp/TestPlan9AFlagOffControlUnitUnchangedOnHostEnvErrortemplate137490636/005/etc/sysusers.d/carlos.conf
wrote /tmp/TestPlan9AFlagOffControlUnitUnchangedOnHostEnvErrortemplate137490636/005/etc/systemd/journald.conf.d/50-cap.conf
wrote /tmp/TestPlan9AFlagOffControlUnitUnchangedOnHostEnvErrortemplate137490636/005/etc/tmpfiles.d/carlos.conf
bootstrap: staged under /tmp/TestPlan9AFlagOffControlUnitUnchangedOnHostEnvErrortemplate137490636/005 — on a real box (--root /) it would also:
useradd --system --user-group --no-create-home --shell /usr/sbin/nologin carlos
groupadd --system carlos-sockets
chown -R carlos:carlos /data
chown -R carlos:carlos-sockets /run/carlos
chown -R carlos:carlos /run/carlos/apps
chmod 2750 /run/carlos
systemctl daemon-reload
systemctl enable carlos-edge.service carlos-relay.service carlos-adopt.timer carlos-adopt.path carlos-stage.path carlos-restart.path carlos-system-update.path carlos-system-recover.service data-instances.mount run-carlos-instances.mount run-carlos-logs.mount carlos-instance-control.socket
systemctl start carlos-relay.service carlos-adopt.timer carlos-adopt.path carlos-stage.path carlos-restart.path carlos-system-update.path data-instances.mount run-carlos-instances.mount run-carlos-logs.mount carlos-instance-control.socket
systemctl restart systemd-journald
bootstrap: instance-control unit settings: unsupported multiline host unit setting
created /tmp/TestPlan9AFlagOffControlUnitUnchangedOnHostEnvErrortemplate137490636/006/data/registry.db
wrote /tmp/TestPlan9AFlagOffControlUnitUnchangedOnHostEnvErrortemplate137490636/006/etc/systemd/system/carlos-adopt.path
wrote /tmp/TestPlan9AFlagOffControlUnitUnchangedOnHostEnvErrortemplate137490636/006/etc/systemd/system/carlos-adopt.service
wrote /tmp/TestPlan9AFlagOffControlUnitUnchangedOnHostEnvErrortemplate137490636/006/etc/systemd/system/carlos-adopt.timer
wrote /tmp/TestPlan9AFlagOffControlUnitUnchangedOnHostEnvErrortemplate137490636/006/etc/systemd/system/carlos-app@.service
wrote /tmp/TestPlan9AFlagOffControlUnitUnchangedOnHostEnvErrortemplate137490636/006/etc/systemd/system/carlos-app@.socket
wrote /tmp/TestPlan9AFlagOffControlUnitUnchangedOnHostEnvErrortemplate137490636/006/etc/systemd/system/carlos-config-sync.service
wrote /tmp/TestPlan9AFlagOffControlUnitUnchangedOnHostEnvErrortemplate137490636/006/etc/systemd/system/carlos-edge.service
wrote /tmp/TestPlan9AFlagOffControlUnitUnchangedOnHostEnvErrortemplate137490636/006/etc/systemd/system/carlos-hibernate-verify.service
wrote /tmp/TestPlan9AFlagOffControlUnitUnchangedOnHostEnvErrortemplate137490636/006/etc/systemd/system/carlos-hibernate-verify.timer
wrote /tmp/TestPlan9AFlagOffControlUnitUnchangedOnHostEnvErrortemplate137490636/006/etc/systemd/system/carlos-instance-control.service
wrote /tmp/TestPlan9AFlagOffControlUnitUnchangedOnHostEnvErrortemplate137490636/006/etc/systemd/system/carlos-instance-control.socket
wrote /tmp/TestPlan9AFlagOffControlUnitUnchangedOnHostEnvErrortemplate137490636/006/etc/systemd/system/carlos-instance-gen@.service
wrote /tmp/TestPlan9AFlagOffControlUnitUnchangedOnHostEnvErrortemplate137490636/006/etc/systemd/system/carlos-instance-probe.service
wrote /tmp/TestPlan9AFlagOffControlUnitUnchangedOnHostEnvErrortemplate137490636/006/etc/systemd/system/carlos-instance-seed@.service
wrote /tmp/TestPlan9AFlagOffControlUnitUnchangedOnHostEnvErrortemplate137490636/006/etc/systemd/system/carlos-instance-sidecar@.service
wrote /tmp/TestPlan9AFlagOffControlUnitUnchangedOnHostEnvErrortemplate137490636/006/etc/systemd/system/carlos-instance@.service
wrote /tmp/TestPlan9AFlagOffControlUnitUnchangedOnHostEnvErrortemplate137490636/006/etc/systemd/system/carlos-instances.slice
wrote /tmp/TestPlan9AFlagOffControlUnitUnchangedOnHostEnvErrortemplate137490636/006/etc/systemd/system/carlos-litestream.service
wrote /tmp/TestPlan9AFlagOffControlUnitUnchangedOnHostEnvErrortemplate137490636/006/etc/systemd/system/carlos-relay.service
wrote /tmp/TestPlan9AFlagOffControlUnitUnchangedOnHostEnvErrortemplate137490636/006/etc/systemd/system/carlos-restart.path
wrote /tmp/TestPlan9AFlagOffControlUnitUnchangedOnHostEnvErrortemplate137490636/006/etc/systemd/system/carlos-restart.service
wrote /tmp/TestPlan9AFlagOffControlUnitUnchangedOnHostEnvErrortemplate137490636/006/etc/systemd/system/carlos-restore-verify.service
wrote /tmp/TestPlan9AFlagOffControlUnitUnchangedOnHostEnvErrortemplate137490636/006/etc/systemd/system/carlos-restore-verify.timer
wrote /tmp/TestPlan9AFlagOffControlUnitUnchangedOnHostEnvErrortemplate137490636/006/etc/systemd/system/carlos-stage.path
wrote /tmp/TestPlan9AFlagOffControlUnitUnchangedOnHostEnvErrortemplate137490636/006/etc/systemd/system/carlos-stage.service
wrote /tmp/TestPlan9AFlagOffControlUnitUnchangedOnHostEnvErrortemplate137490636/006/etc/systemd/system/carlos-system-recover.service
wrote /tmp/TestPlan9AFlagOffControlUnitUnchangedOnHostEnvErrortemplate137490636/006/etc/systemd/system/carlos-system-update.path
wrote /tmp/TestPlan9AFlagOffControlUnitUnchangedOnHostEnvErrortemplate137490636/006/etc/systemd/system/carlos-system-update.service
wrote /tmp/TestPlan9AFlagOffControlUnitUnchangedOnHostEnvErrortemplate137490636/006/etc/systemd/system/carlos.slice
wrote /tmp/TestPlan9AFlagOffControlUnitUnchangedOnHostEnvErrortemplate137490636/006/etc/systemd/system/data-instances.mount
wrote /tmp/TestPlan9AFlagOffControlUnitUnchangedOnHostEnvErrortemplate137490636/006/etc/systemd/system/run-carlos-instances.mount
wrote /tmp/TestPlan9AFlagOffControlUnitUnchangedOnHostEnvErrortemplate137490636/006/etc/systemd/system/run-carlos-logs.mount
wrote /tmp/TestPlan9AFlagOffControlUnitUnchangedOnHostEnvErrortemplate137490636/006/etc/sysusers.d/carlos.conf
wrote /tmp/TestPlan9AFlagOffControlUnitUnchangedOnHostEnvErrortemplate137490636/006/etc/systemd/journald.conf.d/50-cap.conf
wrote /tmp/TestPlan9AFlagOffControlUnitUnchangedOnHostEnvErrortemplate137490636/006/etc/tmpfiles.d/carlos.conf
bootstrap: staged under /tmp/TestPlan9AFlagOffControlUnitUnchangedOnHostEnvErrortemplate137490636/006 — on a real box (--root /) it would also:
useradd --system --user-group --no-create-home --shell /usr/sbin/nologin carlos
groupadd --system carlos-sockets
chown -R carlos:carlos /data
chown -R carlos:carlos-sockets /run/carlos
chown -R carlos:carlos /run/carlos/apps
chmod 2750 /run/carlos
systemctl daemon-reload
systemctl enable carlos-edge.service carlos-relay.service carlos-adopt.timer carlos-adopt.path carlos-stage.path carlos-restart.path carlos-system-update.path carlos-system-recover.service data-instances.mount run-carlos-instances.mount run-carlos-logs.mount carlos-instance-control.socket
systemctl start carlos-relay.service carlos-adopt.timer carlos-adopt.path carlos-stage.path carlos-restart.path carlos-system-update.path data-instances.mount run-carlos-instances.mount run-carlos-logs.mount carlos-instance-control.socket
systemctl restart systemd-journald
created /tmp/TestPlan9AFlagOffControlUnitUnchangedOnHostEnvErrorolder45087982/005/data/registry.db
wrote /tmp/TestPlan9AFlagOffControlUnitUnchangedOnHostEnvErrorolder45087982/005/etc/systemd/system/carlos-adopt.path
wrote /tmp/TestPlan9AFlagOffControlUnitUnchangedOnHostEnvErrorolder45087982/005/etc/systemd/system/carlos-adopt.service
wrote /tmp/TestPlan9AFlagOffControlUnitUnchangedOnHostEnvErrorolder45087982/005/etc/systemd/system/carlos-adopt.timer
wrote /tmp/TestPlan9AFlagOffControlUnitUnchangedOnHostEnvErrorolder45087982/005/etc/systemd/system/carlos-app@.service
wrote /tmp/TestPlan9AFlagOffControlUnitUnchangedOnHostEnvErrorolder45087982/005/etc/systemd/system/carlos-app@.socket
wrote /tmp/TestPlan9AFlagOffControlUnitUnchangedOnHostEnvErrorolder45087982/005/etc/systemd/system/carlos-config-sync.service
wrote /tmp/TestPlan9AFlagOffControlUnitUnchangedOnHostEnvErrorolder45087982/005/etc/systemd/system/carlos-edge.service
wrote /tmp/TestPlan9AFlagOffControlUnitUnchangedOnHostEnvErrorolder45087982/005/etc/systemd/system/carlos-hibernate-verify.service
wrote /tmp/TestPlan9AFlagOffControlUnitUnchangedOnHostEnvErrorolder45087982/005/etc/systemd/system/carlos-hibernate-verify.timer
wrote /tmp/TestPlan9AFlagOffControlUnitUnchangedOnHostEnvErrorolder45087982/005/etc/systemd/system/carlos-instance-control.service
wrote /tmp/TestPlan9AFlagOffControlUnitUnchangedOnHostEnvErrorolder45087982/005/etc/systemd/system/carlos-instance-control.socket
wrote /tmp/TestPlan9AFlagOffControlUnitUnchangedOnHostEnvErrorolder45087982/005/etc/systemd/system/carlos-instance-gen@.service
wrote /tmp/TestPlan9AFlagOffControlUnitUnchangedOnHostEnvErrorolder45087982/005/etc/systemd/system/carlos-instance-probe.service
wrote /tmp/TestPlan9AFlagOffControlUnitUnchangedOnHostEnvErrorolder45087982/005/etc/systemd/system/carlos-instance-seed@.service
wrote /tmp/TestPlan9AFlagOffControlUnitUnchangedOnHostEnvErrorolder45087982/005/etc/systemd/system/carlos-instance-sidecar@.service
wrote /tmp/TestPlan9AFlagOffControlUnitUnchangedOnHostEnvErrorolder45087982/005/etc/systemd/system/carlos-instance@.service
wrote /tmp/TestPlan9AFlagOffControlUnitUnchangedOnHostEnvErrorolder45087982/005/etc/systemd/system/carlos-instances.slice
wrote /tmp/TestPlan9AFlagOffControlUnitUnchangedOnHostEnvErrorolder45087982/005/etc/systemd/system/carlos-litestream.service
wrote /tmp/TestPlan9AFlagOffControlUnitUnchangedOnHostEnvErrorolder45087982/005/etc/systemd/system/carlos-relay.service
wrote /tmp/TestPlan9AFlagOffControlUnitUnchangedOnHostEnvErrorolder45087982/005/etc/systemd/system/carlos-restart.path
wrote /tmp/TestPlan9AFlagOffControlUnitUnchangedOnHostEnvErrorolder45087982/005/etc/systemd/system/carlos-restart.service
wrote /tmp/TestPlan9AFlagOffControlUnitUnchangedOnHostEnvErrorolder45087982/005/etc/systemd/system/carlos-restore-verify.service
wrote /tmp/TestPlan9AFlagOffControlUnitUnchangedOnHostEnvErrorolder45087982/005/etc/systemd/system/carlos-restore-verify.timer
wrote /tmp/TestPlan9AFlagOffControlUnitUnchangedOnHostEnvErrorolder45087982/005/etc/systemd/system/carlos-stage.path
wrote /tmp/TestPlan9AFlagOffControlUnitUnchangedOnHostEnvErrorolder45087982/005/etc/systemd/system/carlos-stage.service
wrote /tmp/TestPlan9AFlagOffControlUnitUnchangedOnHostEnvErrorolder45087982/005/etc/systemd/system/carlos-system-recover.service
wrote /tmp/TestPlan9AFlagOffControlUnitUnchangedOnHostEnvErrorolder45087982/005/etc/systemd/system/carlos-system-update.path
wrote /tmp/TestPlan9AFlagOffControlUnitUnchangedOnHostEnvErrorolder45087982/005/etc/systemd/system/carlos-system-update.service
wrote /tmp/TestPlan9AFlagOffControlUnitUnchangedOnHostEnvErrorolder45087982/005/etc/systemd/system/carlos.slice
wrote /tmp/TestPlan9AFlagOffControlUnitUnchangedOnHostEnvErrorolder45087982/005/etc/systemd/system/data-instances.mount
wrote /tmp/TestPlan9AFlagOffControlUnitUnchangedOnHostEnvErrorolder45087982/005/etc/systemd/system/run-carlos-instances.mount
wrote /tmp/TestPlan9AFlagOffControlUnitUnchangedOnHostEnvErrorolder45087982/005/etc/systemd/system/run-carlos-logs.mount
wrote /tmp/TestPlan9AFlagOffControlUnitUnchangedOnHostEnvErrorolder45087982/005/etc/sysusers.d/carlos.conf
wrote /tmp/TestPlan9AFlagOffControlUnitUnchangedOnHostEnvErrorolder45087982/005/etc/systemd/journald.conf.d/50-cap.conf
wrote /tmp/TestPlan9AFlagOffControlUnitUnchangedOnHostEnvErrorolder45087982/005/etc/tmpfiles.d/carlos.conf
bootstrap: staged under /tmp/TestPlan9AFlagOffControlUnitUnchangedOnHostEnvErrorolder45087982/005 — on a real box (--root /) it would also:
useradd --system --user-group --no-create-home --shell /usr/sbin/nologin carlos
groupadd --system carlos-sockets
chown -R carlos:carlos /data
chown -R carlos:carlos-sockets /run/carlos
chown -R carlos:carlos /run/carlos/apps
chmod 2750 /run/carlos
systemctl daemon-reload
systemctl enable carlos-edge.service carlos-relay.service carlos-adopt.timer carlos-adopt.path carlos-stage.path carlos-restart.path carlos-system-update.path carlos-system-recover.service data-instances.mount run-carlos-instances.mount run-carlos-logs.mount carlos-instance-control.socket
systemctl start carlos-relay.service carlos-adopt.timer carlos-adopt.path carlos-stage.path carlos-restart.path carlos-system-update.path data-instances.mount run-carlos-instances.mount run-carlos-logs.mount carlos-instance-control.socket
systemctl restart systemd-journald
bootstrap: instance-control unit settings: unsupported multiline host unit setting
created /tmp/TestPlan9AFlagOffControlUnitUnchangedOnHostEnvErrorolder45087982/006/data/registry.db
wrote /tmp/TestPlan9AFlagOffControlUnitUnchangedOnHostEnvErrorolder45087982/006/etc/systemd/system/carlos-adopt.path
wrote /tmp/TestPlan9AFlagOffControlUnitUnchangedOnHostEnvErrorolder45087982/006/etc/systemd/system/carlos-adopt.service
wrote /tmp/TestPlan9AFlagOffControlUnitUnchangedOnHostEnvErrorolder45087982/006/etc/systemd/system/carlos-adopt.timer
wrote /tmp/TestPlan9AFlagOffControlUnitUnchangedOnHostEnvErrorolder45087982/006/etc/systemd/system/carlos-app@.service
wrote /tmp/TestPlan9AFlagOffControlUnitUnchangedOnHostEnvErrorolder45087982/006/etc/systemd/system/carlos-app@.socket
wrote /tmp/TestPlan9AFlagOffControlUnitUnchangedOnHostEnvErrorolder45087982/006/etc/systemd/system/carlos-config-sync.service
wrote /tmp/TestPlan9AFlagOffControlUnitUnchangedOnHostEnvErrorolder45087982/006/etc/systemd/system/carlos-edge.service
wrote /tmp/TestPlan9AFlagOffControlUnitUnchangedOnHostEnvErrorolder45087982/006/etc/systemd/system/carlos-hibernate-verify.service
wrote /tmp/TestPlan9AFlagOffControlUnitUnchangedOnHostEnvErrorolder45087982/006/etc/systemd/system/carlos-hibernate-verify.timer
wrote /tmp/TestPlan9AFlagOffControlUnitUnchangedOnHostEnvErrorolder45087982/006/etc/systemd/system/carlos-instance-control.service
wrote /tmp/TestPlan9AFlagOffControlUnitUnchangedOnHostEnvErrorolder45087982/006/etc/systemd/system/carlos-instance-control.socket
wrote /tmp/TestPlan9AFlagOffControlUnitUnchangedOnHostEnvErrorolder45087982/006/etc/systemd/system/carlos-instance-gen@.service
wrote /tmp/TestPlan9AFlagOffControlUnitUnchangedOnHostEnvErrorolder45087982/006/etc/systemd/system/carlos-instance-probe.service
wrote /tmp/TestPlan9AFlagOffControlUnitUnchangedOnHostEnvErrorolder45087982/006/etc/systemd/system/carlos-instance-seed@.service
wrote /tmp/TestPlan9AFlagOffControlUnitUnchangedOnHostEnvErrorolder45087982/006/etc/systemd/system/carlos-instance-sidecar@.service
wrote /tmp/TestPlan9AFlagOffControlUnitUnchangedOnHostEnvErrorolder45087982/006/etc/systemd/system/carlos-instance@.service
wrote /tmp/TestPlan9AFlagOffControlUnitUnchangedOnHostEnvErrorolder45087982/006/etc/systemd/system/carlos-instances.slice
wrote /tmp/TestPlan9AFlagOffControlUnitUnchangedOnHostEnvErrorolder45087982/006/etc/systemd/system/carlos-litestream.service
wrote /tmp/TestPlan9AFlagOffControlUnitUnchangedOnHostEnvErrorolder45087982/006/etc/systemd/system/carlos-relay.service
wrote /tmp/TestPlan9AFlagOffControlUnitUnchangedOnHostEnvErrorolder45087982/006/etc/systemd/system/carlos-restart.path
wrote /tmp/TestPlan9AFlagOffControlUnitUnchangedOnHostEnvErrorolder45087982/006/etc/systemd/system/carlos-restart.service
wrote /tmp/TestPlan9AFlagOffControlUnitUnchangedOnHostEnvErrorolder45087982/006/etc/systemd/system/carlos-restore-verify.service
wrote /tmp/TestPlan9AFlagOffControlUnitUnchangedOnHostEnvErrorolder45087982/006/etc/systemd/system/carlos-restore-verify.timer
wrote /tmp/TestPlan9AFlagOffControlUnitUnchangedOnHostEnvErrorolder45087982/006/etc/systemd/system/carlos-stage.path
wrote /tmp/TestPlan9AFlagOffControlUnitUnchangedOnHostEnvErrorolder45087982/006/etc/systemd/system/carlos-stage.service
wrote /tmp/TestPlan9AFlagOffControlUnitUnchangedOnHostEnvErrorolder45087982/006/etc/systemd/system/carlos-system-recover.service
wrote /tmp/TestPlan9AFlagOffControlUnitUnchangedOnHostEnvErrorolder45087982/006/etc/systemd/system/carlos-system-update.path
wrote /tmp/TestPlan9AFlagOffControlUnitUnchangedOnHostEnvErrorolder45087982/006/etc/systemd/system/carlos-system-update.service
wrote /tmp/TestPlan9AFlagOffControlUnitUnchangedOnHostEnvErrorolder45087982/006/etc/systemd/system/carlos.slice
wrote /tmp/TestPlan9AFlagOffControlUnitUnchangedOnHostEnvErrorolder45087982/006/etc/systemd/system/data-instances.mount
wrote /tmp/TestPlan9AFlagOffControlUnitUnchangedOnHostEnvErrorolder45087982/006/etc/systemd/system/run-carlos-instances.mount
wrote /tmp/TestPlan9AFlagOffControlUnitUnchangedOnHostEnvErrorolder45087982/006/etc/systemd/system/run-carlos-logs.mount
wrote /tmp/TestPlan9AFlagOffControlUnitUnchangedOnHostEnvErrorolder45087982/006/etc/sysusers.d/carlos.conf
wrote /tmp/TestPlan9AFlagOffControlUnitUnchangedOnHostEnvErrorolder45087982/006/etc/systemd/journald.conf.d/50-cap.conf
wrote /tmp/TestPlan9AFlagOffControlUnitUnchangedOnHostEnvErrorolder45087982/006/etc/tmpfiles.d/carlos.conf
bootstrap: staged under /tmp/TestPlan9AFlagOffControlUnitUnchangedOnHostEnvErrorolder45087982/006 — on a real box (--root /) it would also:
useradd --system --user-group --no-create-home --shell /usr/sbin/nologin carlos
groupadd --system carlos-sockets
chown -R carlos:carlos /data
chown -R carlos:carlos-sockets /run/carlos
chown -R carlos:carlos /run/carlos/apps
chmod 2750 /run/carlos
systemctl daemon-reload
systemctl enable carlos-edge.service carlos-relay.service carlos-adopt.timer carlos-adopt.path carlos-stage.path carlos-restart.path carlos-system-update.path carlos-system-recover.service data-instances.mount run-carlos-instances.mount run-carlos-logs.mount carlos-instance-control.socket
systemctl start carlos-relay.service carlos-adopt.timer carlos-adopt.path carlos-stage.path carlos-restart.path carlos-system-update.path data-instances.mount run-carlos-instances.mount run-carlos-logs.mount carlos-instance-control.socket
systemctl restart systemd-journald
2026/10/12 22:57:29 moves: a.go.example.com: abort complete — staged copy deleted, instance stays in "ie"
2026/10/12 22:57:29 moves: a.go.example.com: abort complete — staged copy deleted, instance stays in "ie"
2026/10/12 22:57:34 host fence pass: credentials unavailable
2026/10/12 22:57:34 host fence pass: credentials unavailable
2026/10/12 22:57:37 moves: a.go.example.com: mv_abcdefghijklmnopqrstuvwxyz requested (requested)
2026/10/12 22:57:37 moves: a.go.example.com: staged the files archive sha256:c9229ef8e7256a1dcd5c2715b11ac3016951e8fee8e0bfa823aa2460f0d331c8 (2 files, 19 bytes)
2026/10/12 22:57:37 moves: a.go.example.com: not ready: the target has not published a readiness report yet
2026/10/12 22:57:37 moves: a.go.example.com: mv_abcdefghijklmnopqrstuvwxyz ready (ready)
2026/10/12 22:57:37 moves: a.go.example.com: move mv_abcdefghijklmnopqrstuvwxyz is ready (last round moved 1815 bytes; bar 16777216)
2026/10/12 22:57:37 moves: a.go.example.com: mv_abcdefghijklmnopqrstuvwxyz cut_requested (cutting-requested)
2026/10/12 22:57:37 moves: a.go.example.com: mv_abcdefghijklmnopqrstuvwxyz admitted (cutting-admitted)
2026/10/12 22:57:37 moves: a.go.example.com: mv_abcdefghijklmnopqrstuvwxyz stopping (cutting-stopping)
2026/10/12 22:57:37 moves: a.go.example.com: cut mv_abcdefghijklmnopqrstuvwxyz is stopping
2026/10/12 22:57:37 moves: a.go.example.com: mv_abcdefghijklmnopqrstuvwxyz stopped (cutting-stopped)
2026/10/12 22:57:37 moves: a.go.example.com: cut mv_abcdefghijklmnopqrstuvwxyz is stopped
2026/10/12 22:57:37 moves: a.go.example.com: stopped at TXID 3; execution given up, citing moves/bab/app/a.go.example.com/mv_abcdefghijklmnopqrstuvwxyz/stopped-source.json
2026/10/12 22:57:37 moves: a.go.example.com: sealed 4 replica objects at TXID 3
2026/10/12 22:57:37 moves: a.go.example.com: mv_abcdefghijklmnopqrstuvwxyz staged (cutting-staged)
2026/10/12 22:57:37 moves: a.go.example.com: cut mv_abcdefghijklmnopqrstuvwxyz is staged
2026/10/12 22:57:37 moves: a.go.example.com: mv_abcdefghijklmnopqrstuvwxyz archived (cutting-archived)
2026/10/12 22:57:37 moves: a.go.example.com: cut mv_abcdefghijklmnopqrstuvwxyz is archived
2026/10/12 22:57:37 moves: a.go.example.com: mv_abcdefghijklmnopqrstuvwxyz flipped (cutting-flipped)
2026/10/12 22:57:37 moves: a.go.example.com: flipped to uk
2026/10/12 22:57:37 moves: a.go.example.com: the claim's landing phase caught up from "checking" to "probed"
--- FAIL: TestPlan9Fix4RootAssemblyIgnoresReplacedLegacyDirectory (1.04s)
--- FAIL: TestPlan9Fix4RootAssemblyIgnoresReplacedLegacyDirectory/instance-control/missing (0.21s)
plan9fix4_claimjournal_migration_linux_test.go:140: agent-controlled legacy directory prevented root startup: permission denied
--- FAIL: TestPlan9Fix4RootAssemblyIgnoresReplacedLegacyDirectory/instance-control/renamed (0.06s)
plan9fix4_claimjournal_migration_linux_test.go:140: agent-controlled legacy directory prevented root startup: permission denied
--- FAIL: TestPlan9Fix4RootAssemblyIgnoresReplacedLegacyDirectory/instance-control/symlink (0.11s)
plan9fix4_claimjournal_migration_linux_test.go:140: agent-controlled legacy directory prevented root startup: permission denied
--- FAIL: TestPlan9Fix4RootAssemblyIgnoresReplacedLegacyDirectory/instance-control/file (0.17s)
plan9fix4_claimjournal_migration_linux_test.go:140: agent-controlled legacy directory prevented root startup: permission denied
--- FAIL: TestPlan9Fix4RootAssemblyIgnoresReplacedLegacyDirectory/adopt/missing (0.05s)
plan9fix4_claimjournal_migration_linux_test.go:140: agent-controlled legacy directory prevented root startup: permission denied
--- FAIL: TestPlan9Fix4RootAssemblyIgnoresReplacedLegacyDirectory/adopt/renamed (0.10s)
plan9fix4_claimjournal_migration_linux_test.go:140: agent-controlled legacy directory prevented root startup: permission denied
--- FAIL: TestPlan9Fix4RootAssemblyIgnoresReplacedLegacyDirectory/adopt/symlink (0.19s)
plan9fix4_claimjournal_migration_linux_test.go:140: agent-controlled legacy directory prevented root startup: permission denied
--- FAIL: TestPlan9Fix4RootAssemblyIgnoresReplacedLegacyDirectory/adopt/file (0.13s)
plan9fix4_claimjournal_migration_linux_test.go:140: agent-controlled legacy directory prevented root startup: permission denied
2026/10/12 22:57:39 release: pointer digest ratchet: open directory /tmp/TestOneRatchetPerProcessSurvivesAFailedPersist2620164437/002/pointer-digests/bab/km: permission denied
bab/km/canary/x: channel pointer refused: release: the pointer digest mark could not be persisted: open directory /tmp/TestOneRatchetPerProcessSurvivesAFailedPersist2620164437/002/pointer-digests/bab/km: permission denied
bab/km/canary/x: channel pointer refused: release: pointer carries no digests: bab/km/canary/x, and this box has already verified one with digests for that channel
a.example.com: updated -> v1 (canary/x)
a.example.com: updated v1 -> v2 (canary/x)
bab/km/canary/x: channel pointer refused: release: pointer carries no digests: bab/km/canary/x, and this box has already verified one with digests for that channel
a.example.com: updated -> v1 (canary/x)
bab/km/canary/x: channel pointer refused: release: pointer carries no digests: bab/km/canary/x, and CARLOS_REQUIRE_POINTER_DIGEST is on
2026/10/12 22:57:39 agent: platform.example.com has run v1 for 15m0s with no proof in the bucket; asking for a root pass to mint one
2026/10/12 22:57:39 agent: platform.example.com has run v1 for 15m0s with no proof in the bucket; asking for a root pass to mint one
2026/10/12 22:57:39 agent: platform.example.com has run v1 for 15m0s with no proof in the bucket; asking for a root pass to mint one
2026/10/12 22:57:39 agent: platform.example.com has run v1 for 15m0s with no proof in the bucket; asking for a root pass to mint one
2026/10/12 22:57:39 agent: platform.example.com has run v1 for 15m0s with no proof in the bucket; asking for a root pass to mint one
edge.example.com: v2 (sha256 e45688ace7983a0b303d1c20bf8426ef0207abba479ea6dd1739816258df89d4) is not proven on dev; staying on v1 (no proof from dev)
edge.example.com: updated v1 -> v2 (edge)
new.example.com: v2 (sha256 e45688ace7983a0b303d1c20bf8426ef0207abba479ea6dd1739816258df89d4) is not proven on dev; staying on nothing (no proof from dev)
edge.example.com: back-fill manifest for v1: dirstore: apps/bab/fastapp/releases/v1/manifest.json: file does not exist — if this deployment predates account-scoped bucket keys its objects are still at apps/fastapp/, which the runtime no longer reads; move them once with: carlos accounts migrate --app fastapp --from legacy --to bab
edge.example.com: v2 (sha256 e45688ace7983a0b303d1c20bf8426ef0207abba479ea6dd1739816258df89d4) is not proven on dev; staying on v1 (no proof from dev)
bab/fastapp/nowhere: channel has no release yet
2026/10/12 22:57:39 adopt: platform.example.com: console.service settled on /tmp/TestSettledRestartRecordsTheRunningExe411563153/001/versions/new
2026/10/12 22:57:39 adopt: binding: console.service runs /tmp/TestUnsettledRestartRecordsNothing808007796/001/live/console.example.com, but the unit stamp is on platform.example.com; restart not checked
2026/10/12 22:57:39 adopt: platform.example.com: console.service was stopped during settle; not judged, not restarted
platform.example.com: updated a -> b (canary/console)
--- FAIL: TestPrivilegedRegistryEntryPointsUseServiceBoundary (0.01s)
--- FAIL: TestPrivilegedRegistryEntryPointsUseServiceBoundary/runAdoptWithEnsureSystemUnits (0.00s)
registry_privilege_test.go:132: privileged caller does not cross the service registry boundary
created /tmp/TestBootstrapCreatesRegistryBeforeArmingUnits1933200481/001/data/registry.db
wrote /tmp/TestBootstrapCreatesRegistryBeforeArmingUnits1933200481/001/etc/systemd/system/carlos-adopt.path
wrote /tmp/TestBootstrapCreatesRegistryBeforeArmingUnits1933200481/001/etc/systemd/system/carlos-adopt.service
wrote /tmp/TestBootstrapCreatesRegistryBeforeArmingUnits1933200481/001/etc/systemd/system/carlos-adopt.timer
wrote /tmp/TestBootstrapCreatesRegistryBeforeArmingUnits1933200481/001/etc/systemd/system/carlos-app@.service
wrote /tmp/TestBootstrapCreatesRegistryBeforeArmingUnits1933200481/001/etc/systemd/system/carlos-app@.socket
wrote /tmp/TestBootstrapCreatesRegistryBeforeArmingUnits1933200481/001/etc/systemd/system/carlos-config-sync.service
wrote /tmp/TestBootstrapCreatesRegistryBeforeArmingUnits1933200481/001/etc/systemd/system/carlos-edge.service
wrote /tmp/TestBootstrapCreatesRegistryBeforeArmingUnits1933200481/001/etc/systemd/system/carlos-hibernate-verify.service
wrote /tmp/TestBootstrapCreatesRegistryBeforeArmingUnits1933200481/001/etc/systemd/system/carlos-hibernate-verify.timer
wrote /tmp/TestBootstrapCreatesRegistryBeforeArmingUnits1933200481/001/etc/systemd/system/carlos-instance-control.service
wrote /tmp/TestBootstrapCreatesRegistryBeforeArmingUnits1933200481/001/etc/systemd/system/carlos-instance-control.socket
wrote /tmp/TestBootstrapCreatesRegistryBeforeArmingUnits1933200481/001/etc/systemd/system/carlos-instance-gen@.service
wrote /tmp/TestBootstrapCreatesRegistryBeforeArmingUnits1933200481/001/etc/systemd/system/carlos-instance-probe.service
wrote /tmp/TestBootstrapCreatesRegistryBeforeArmingUnits1933200481/001/etc/systemd/system/carlos-instance-seed@.service
wrote /tmp/TestBootstrapCreatesRegistryBeforeArmingUnits1933200481/001/etc/systemd/system/carlos-instance-sidecar@.service
wrote /tmp/TestBootstrapCreatesRegistryBeforeArmingUnits1933200481/001/etc/systemd/system/carlos-instance@.service
wrote /tmp/TestBootstrapCreatesRegistryBeforeArmingUnits1933200481/001/etc/systemd/system/carlos-instances.slice
wrote /tmp/TestBootstrapCreatesRegistryBeforeArmingUnits1933200481/001/etc/systemd/system/carlos-litestream.service
wrote /tmp/TestBootstrapCreatesRegistryBeforeArmingUnits1933200481/001/etc/systemd/system/carlos-relay.service
wrote /tmp/TestBootstrapCreatesRegistryBeforeArmingUnits1933200481/001/etc/systemd/system/carlos-restart.path
wrote /tmp/TestBootstrapCreatesRegistryBeforeArmingUnits1933200481/001/etc/systemd/system/carlos-restart.service
wrote /tmp/TestBootstrapCreatesRegistryBeforeArmingUnits1933200481/001/etc/systemd/system/carlos-restore-verify.service
wrote /tmp/TestBootstrapCreatesRegistryBeforeArmingUnits1933200481/001/etc/systemd/system/carlos-restore-verify.timer
wrote /tmp/TestBootstrapCreatesRegistryBeforeArmingUnits1933200481/001/etc/systemd/system/carlos-stage.path
wrote /tmp/TestBootstrapCreatesRegistryBeforeArmingUnits1933200481/001/etc/systemd/system/carlos-stage.service
wrote /tmp/TestBootstrapCreatesRegistryBeforeArmingUnits1933200481/001/etc/systemd/system/carlos-system-recover.service
wrote /tmp/TestBootstrapCreatesRegistryBeforeArmingUnits1933200481/001/etc/systemd/system/carlos-system-update.path
wrote /tmp/TestBootstrapCreatesRegistryBeforeArmingUnits1933200481/001/etc/systemd/system/carlos-system-update.service
wrote /tmp/TestBootstrapCreatesRegistryBeforeArmingUnits1933200481/001/etc/systemd/system/carlos.slice
wrote /tmp/TestBootstrapCreatesRegistryBeforeArmingUnits1933200481/001/etc/systemd/system/data-instances.mount
wrote /tmp/TestBootstrapCreatesRegistryBeforeArmingUnits1933200481/001/etc/systemd/system/run-carlos-instances.mount
wrote /tmp/TestBootstrapCreatesRegistryBeforeArmingUnits1933200481/001/etc/systemd/system/run-carlos-logs.mount
wrote /tmp/TestBootstrapCreatesRegistryBeforeArmingUnits1933200481/001/etc/sysusers.d/carlos.conf
wrote /tmp/TestBootstrapCreatesRegistryBeforeArmingUnits1933200481/001/etc/systemd/journald.conf.d/50-cap.conf
wrote /tmp/TestBootstrapCreatesRegistryBeforeArmingUnits1933200481/001/etc/tmpfiles.d/carlos.conf
bootstrap: staged under /tmp/TestBootstrapCreatesRegistryBeforeArmingUnits1933200481/001 — on a real box (--root /) it would also:
useradd --system --user-group --no-create-home --shell /usr/sbin/nologin carlos
groupadd --system carlos-sockets
chown -R carlos:carlos /data
chown -R carlos:carlos-sockets /run/carlos
chown -R carlos:carlos /run/carlos/apps
chmod 2750 /run/carlos
systemctl daemon-reload
systemctl enable carlos-edge.service carlos-relay.service carlos-adopt.timer carlos-adopt.path carlos-stage.path carlos-restart.path carlos-system-update.path carlos-system-recover.service data-instances.mount run-carlos-instances.mount run-carlos-logs.mount carlos-instance-control.socket
systemctl start carlos-relay.service carlos-adopt.timer carlos-adopt.path carlos-stage.path carlos-restart.path carlos-system-update.path data-instances.mount run-carlos-instances.mount run-carlos-logs.mount carlos-instance-control.socket
systemctl restart systemd-journald
created /tmp/TestBootstrapRegistryIsAnOpenableRegistry255954588/001/data/registry.db
wrote /tmp/TestBootstrapRegistryIsAnOpenableRegistry255954588/001/etc/systemd/system/carlos-adopt.path
wrote /tmp/TestBootstrapRegistryIsAnOpenableRegistry255954588/001/etc/systemd/system/carlos-adopt.service
wrote /tmp/TestBootstrapRegistryIsAnOpenableRegistry255954588/001/etc/systemd/system/carlos-adopt.timer
wrote /tmp/TestBootstrapRegistryIsAnOpenableRegistry255954588/001/etc/systemd/system/carlos-app@.service
wrote /tmp/TestBootstrapRegistryIsAnOpenableRegistry255954588/001/etc/systemd/system/carlos-app@.socket
wrote /tmp/TestBootstrapRegistryIsAnOpenableRegistry255954588/001/etc/systemd/system/carlos-config-sync.service
wrote /tmp/TestBootstrapRegistryIsAnOpenableRegistry255954588/001/etc/systemd/system/carlos-edge.service
wrote /tmp/TestBootstrapRegistryIsAnOpenableRegistry255954588/001/etc/systemd/system/carlos-hibernate-verify.service
wrote /tmp/TestBootstrapRegistryIsAnOpenableRegistry255954588/001/etc/systemd/system/carlos-hibernate-verify.timer
wrote /tmp/TestBootstrapRegistryIsAnOpenableRegistry255954588/001/etc/systemd/system/carlos-instance-control.service
wrote /tmp/TestBootstrapRegistryIsAnOpenableRegistry255954588/001/etc/systemd/system/carlos-instance-control.socket
wrote /tmp/TestBootstrapRegistryIsAnOpenableRegistry255954588/001/etc/systemd/system/carlos-instance-gen@.service
wrote /tmp/TestBootstrapRegistryIsAnOpenableRegistry255954588/001/etc/systemd/system/carlos-instance-probe.service
wrote /tmp/TestBootstrapRegistryIsAnOpenableRegistry255954588/001/etc/systemd/system/carlos-instance-seed@.service
wrote /tmp/TestBootstrapRegistryIsAnOpenableRegistry255954588/001/etc/systemd/system/carlos-instance-sidecar@.service
wrote /tmp/TestBootstrapRegistryIsAnOpenableRegistry255954588/001/etc/systemd/system/carlos-instance@.service
wrote /tmp/TestBootstrapRegistryIsAnOpenableRegistry255954588/001/etc/systemd/system/carlos-instances.slice
wrote /tmp/TestBootstrapRegistryIsAnOpenableRegistry255954588/001/etc/systemd/system/carlos-litestream.service
wrote /tmp/TestBootstrapRegistryIsAnOpenableRegistry255954588/001/etc/systemd/system/carlos-relay.service
wrote /tmp/TestBootstrapRegistryIsAnOpenableRegistry255954588/001/etc/systemd/system/carlos-restart.path
wrote /tmp/TestBootstrapRegistryIsAnOpenableRegistry255954588/001/etc/systemd/system/carlos-restart.service
wrote /tmp/TestBootstrapRegistryIsAnOpenableRegistry255954588/001/etc/systemd/system/carlos-restore-verify.service
wrote /tmp/TestBootstrapRegistryIsAnOpenableRegistry255954588/001/etc/systemd/system/carlos-restore-verify.timer
wrote /tmp/TestBootstrapRegistryIsAnOpenableRegistry255954588/001/etc/systemd/system/carlos-stage.path
wrote /tmp/TestBootstrapRegistryIsAnOpenableRegistry255954588/001/etc/systemd/system/carlos-stage.service
wrote /tmp/TestBootstrapRegistryIsAnOpenableRegistry255954588/001/etc/systemd/system/carlos-system-recover.service
wrote /tmp/TestBootstrapRegistryIsAnOpenableRegistry255954588/001/etc/systemd/system/carlos-system-update.path
wrote /tmp/TestBootstrapRegistryIsAnOpenableRegistry255954588/001/etc/systemd/system/carlos-system-update.service
wrote /tmp/TestBootstrapRegistryIsAnOpenableRegistry255954588/001/etc/systemd/system/carlos.slice
wrote /tmp/TestBootstrapRegistryIsAnOpenableRegistry255954588/001/etc/systemd/system/data-instances.mount
wrote /tmp/TestBootstrapRegistryIsAnOpenableRegistry255954588/001/etc/systemd/system/run-carlos-instances.mount
wrote /tmp/TestBootstrapRegistryIsAnOpenableRegistry255954588/001/etc/systemd/system/run-carlos-logs.mount
wrote /tmp/TestBootstrapRegistryIsAnOpenableRegistry255954588/001/etc/sysusers.d/carlos.conf
wrote /tmp/TestBootstrapRegistryIsAnOpenableRegistry255954588/001/etc/systemd/journald.conf.d/50-cap.conf
wrote /tmp/TestBootstrapRegistryIsAnOpenableRegistry255954588/001/etc/tmpfiles.d/carlos.conf
bootstrap: staged under /tmp/TestBootstrapRegistryIsAnOpenableRegistry255954588/001 — on a real box (--root /) it would also:
useradd --system --user-group --no-create-home --shell /usr/sbin/nologin carlos
groupadd --system carlos-sockets
chown -R carlos:carlos /data
chown -R carlos:carlos-sockets /run/carlos
chown -R carlos:carlos /run/carlos/apps
chmod 2750 /run/carlos
systemctl daemon-reload
systemctl enable carlos-edge.service carlos-relay.service carlos-adopt.timer carlos-adopt.path carlos-stage.path carlos-restart.path carlos-system-update.path carlos-system-recover.service data-instances.mount run-carlos-instances.mount run-carlos-logs.mount carlos-instance-control.socket
systemctl start carlos-relay.service carlos-adopt.timer carlos-adopt.path carlos-stage.path carlos-restart.path carlos-system-update.path data-instances.mount run-carlos-instances.mount run-carlos-logs.mount carlos-instance-control.socket
systemctl restart systemd-journald
created /tmp/TestBootstrapKeepsAnExistingRegistry2582281630/001/data/registry.db
wrote /tmp/TestBootstrapKeepsAnExistingRegistry2582281630/001/etc/systemd/system/carlos-adopt.path
wrote /tmp/TestBootstrapKeepsAnExistingRegistry2582281630/001/etc/systemd/system/carlos-adopt.service
wrote /tmp/TestBootstrapKeepsAnExistingRegistry2582281630/001/etc/systemd/system/carlos-adopt.timer
wrote /tmp/TestBootstrapKeepsAnExistingRegistry2582281630/001/etc/systemd/system/carlos-app@.service
wrote /tmp/TestBootstrapKeepsAnExistingRegistry2582281630/001/etc/systemd/system/carlos-app@.socket
wrote /tmp/TestBootstrapKeepsAnExistingRegistry2582281630/001/etc/systemd/system/carlos-config-sync.service
wrote /tmp/TestBootstrapKeepsAnExistingRegistry2582281630/001/etc/systemd/system/carlos-edge.service
wrote /tmp/TestBootstrapKeepsAnExistingRegistry2582281630/001/etc/systemd/system/carlos-hibernate-verify.service
wrote /tmp/TestBootstrapKeepsAnExistingRegistry2582281630/001/etc/systemd/system/carlos-hibernate-verify.timer
wrote /tmp/TestBootstrapKeepsAnExistingRegistry2582281630/001/etc/systemd/system/carlos-instance-control.service
wrote /tmp/TestBootstrapKeepsAnExistingRegistry2582281630/001/etc/systemd/system/carlos-instance-control.socket
wrote /tmp/TestBootstrapKeepsAnExistingRegistry2582281630/001/etc/systemd/system/carlos-instance-gen@.service
wrote /tmp/TestBootstrapKeepsAnExistingRegistry2582281630/001/etc/systemd/system/carlos-instance-probe.service
wrote /tmp/TestBootstrapKeepsAnExistingRegistry2582281630/001/etc/systemd/system/carlos-instance-seed@.service
wrote /tmp/TestBootstrapKeepsAnExistingRegistry2582281630/001/etc/systemd/system/carlos-instance-sidecar@.service
wrote /tmp/TestBootstrapKeepsAnExistingRegistry2582281630/001/etc/systemd/system/carlos-instance@.service
wrote /tmp/TestBootstrapKeepsAnExistingRegistry2582281630/001/etc/systemd/system/carlos-instances.slice
wrote /tmp/TestBootstrapKeepsAnExistingRegistry2582281630/001/etc/systemd/system/carlos-litestream.service
wrote /tmp/TestBootstrapKeepsAnExistingRegistry2582281630/001/etc/systemd/system/carlos-relay.service
wrote /tmp/TestBootstrapKeepsAnExistingRegistry2582281630/001/etc/systemd/system/carlos-restart.path
wrote /tmp/TestBootstrapKeepsAnExistingRegistry2582281630/001/etc/systemd/system/carlos-restart.service
wrote /tmp/TestBootstrapKeepsAnExistingRegistry2582281630/001/etc/systemd/system/carlos-restore-verify.service
wrote /tmp/TestBootstrapKeepsAnExistingRegistry2582281630/001/etc/systemd/system/carlos-restore-verify.timer
wrote /tmp/TestBootstrapKeepsAnExistingRegistry2582281630/001/etc/systemd/system/carlos-stage.path
wrote /tmp/TestBootstrapKeepsAnExistingRegistry2582281630/001/etc/systemd/system/carlos-stage.service
wrote /tmp/TestBootstrapKeepsAnExistingRegistry2582281630/001/etc/systemd/system/carlos-system-recover.service
wrote /tmp/TestBootstrapKeepsAnExistingRegistry2582281630/001/etc/systemd/system/carlos-system-update.path
wrote /tmp/TestBootstrapKeepsAnExistingRegistry2582281630/001/etc/systemd/system/carlos-system-update.service
wrote /tmp/TestBootstrapKeepsAnExistingRegistry2582281630/001/etc/systemd/system/carlos.slice
wrote /tmp/TestBootstrapKeepsAnExistingRegistry2582281630/001/etc/systemd/system/data-instances.mount
wrote /tmp/TestBootstrapKeepsAnExistingRegistry2582281630/001/etc/systemd/system/run-carlos-instances.mount
wrote /tmp/TestBootstrapKeepsAnExistingRegistry2582281630/001/etc/systemd/system/run-carlos-logs.mount
wrote /tmp/TestBootstrapKeepsAnExistingRegistry2582281630/001/etc/sysusers.d/carlos.conf
wrote /tmp/TestBootstrapKeepsAnExistingRegistry2582281630/001/etc/systemd/journald.conf.d/50-cap.conf
wrote /tmp/TestBootstrapKeepsAnExistingRegistry2582281630/001/etc/tmpfiles.d/carlos.conf
bootstrap: staged under /tmp/TestBootstrapKeepsAnExistingRegistry2582281630/001 — on a real box (--root /) it would also:
useradd --system --user-group --no-create-home --shell /usr/sbin/nologin carlos
groupadd --system carlos-sockets
chown -R carlos:carlos /data
chown -R carlos:carlos-sockets /run/carlos
chown -R carlos:carlos /run/carlos/apps
chmod 2750 /run/carlos
systemctl daemon-reload
systemctl enable carlos-edge.service carlos-relay.service carlos-adopt.timer carlos-adopt.path carlos-stage.path carlos-restart.path carlos-system-update.path carlos-system-recover.service data-instances.mount run-carlos-instances.mount run-carlos-logs.mount carlos-instance-control.socket
systemctl start carlos-relay.service carlos-adopt.timer carlos-adopt.path carlos-stage.path carlos-restart.path carlos-system-update.path data-instances.mount run-carlos-instances.mount run-carlos-logs.mount carlos-instance-control.socket
systemctl restart systemd-journald
wrote /tmp/TestBootstrapKeepsAnExistingRegistry2582281630/001/etc/systemd/system/carlos-adopt.path
wrote /tmp/TestBootstrapKeepsAnExistingRegistry2582281630/001/etc/systemd/system/carlos-adopt.service
wrote /tmp/TestBootstrapKeepsAnExistingRegistry2582281630/001/etc/systemd/system/carlos-adopt.timer
wrote /tmp/TestBootstrapKeepsAnExistingRegistry2582281630/001/etc/systemd/system/carlos-app@.service
wrote /tmp/TestBootstrapKeepsAnExistingRegistry2582281630/001/etc/systemd/system/carlos-app@.socket
wrote /tmp/TestBootstrapKeepsAnExistingRegistry2582281630/001/etc/systemd/system/carlos-config-sync.service
wrote /tmp/TestBootstrapKeepsAnExistingRegistry2582281630/001/etc/systemd/system/carlos-edge.service
wrote /tmp/TestBootstrapKeepsAnExistingRegistry2582281630/001/etc/systemd/system/carlos-hibernate-verify.service
wrote /tmp/TestBootstrapKeepsAnExistingRegistry2582281630/001/etc/systemd/system/carlos-hibernate-verify.timer
wrote /tmp/TestBootstrapKeepsAnExistingRegistry2582281630/001/etc/systemd/system/carlos-instance-control.service
wrote /tmp/TestBootstrapKeepsAnExistingRegistry2582281630/001/etc/systemd/system/carlos-instance-control.socket
wrote /tmp/TestBootstrapKeepsAnExistingRegistry2582281630/001/etc/systemd/system/carlos-instance-gen@.service
wrote /tmp/TestBootstrapKeepsAnExistingRegistry2582281630/001/etc/systemd/system/carlos-instance-probe.service
wrote /tmp/TestBootstrapKeepsAnExistingRegistry2582281630/001/etc/systemd/system/carlos-instance-seed@.service
wrote /tmp/TestBootstrapKeepsAnExistingRegistry2582281630/001/etc/systemd/system/carlos-instance-sidecar@.service
wrote /tmp/TestBootstrapKeepsAnExistingRegistry2582281630/001/etc/systemd/system/carlos-instance@.service
wrote /tmp/TestBootstrapKeepsAnExistingRegistry2582281630/001/etc/systemd/system/carlos-instances.slice
wrote /tmp/TestBootstrapKeepsAnExistingRegistry2582281630/001/etc/systemd/system/carlos-litestream.service
wrote /tmp/TestBootstrapKeepsAnExistingRegistry2582281630/001/etc/systemd/system/carlos-relay.service
wrote /tmp/TestBootstrapKeepsAnExistingRegistry2582281630/001/etc/systemd/system/carlos-restart.path
wrote /tmp/TestBootstrapKeepsAnExistingRegistry2582281630/001/etc/systemd/system/carlos-restart.service
wrote /tmp/TestBootstrapKeepsAnExistingRegistry2582281630/001/etc/systemd/system/carlos-restore-verify.service
wrote /tmp/TestBootstrapKeepsAnExistingRegistry2582281630/001/etc/systemd/system/carlos-restore-verify.timer
wrote /tmp/TestBootstrapKeepsAnExistingRegistry2582281630/001/etc/systemd/system/carlos-stage.path
wrote /tmp/TestBootstrapKeepsAnExistingRegistry2582281630/001/etc/systemd/system/carlos-stage.service
wrote /tmp/TestBootstrapKeepsAnExistingRegistry2582281630/001/etc/systemd/system/carlos-system-recover.service
wrote /tmp/TestBootstrapKeepsAnExistingRegistry2582281630/001/etc/systemd/system/carlos-system-update.path
wrote /tmp/TestBootstrapKeepsAnExistingRegistry2582281630/001/etc/systemd/system/carlos-system-update.service
wrote /tmp/TestBootstrapKeepsAnExistingRegistry2582281630/001/etc/systemd/system/carlos.slice
wrote /tmp/TestBootstrapKeepsAnExistingRegistry2582281630/001/etc/systemd/system/data-instances.mount
wrote /tmp/TestBootstrapKeepsAnExistingRegistry2582281630/001/etc/systemd/system/run-carlos-instances.mount
wrote /tmp/TestBootstrapKeepsAnExistingRegistry2582281630/001/etc/systemd/system/run-carlos-logs.mount
wrote /tmp/TestBootstrapKeepsAnExistingRegistry2582281630/001/etc/sysusers.d/carlos.conf
wrote /tmp/TestBootstrapKeepsAnExistingRegistry2582281630/001/etc/systemd/journald.conf.d/50-cap.conf
wrote /tmp/TestBootstrapKeepsAnExistingRegistry2582281630/001/etc/tmpfiles.d/carlos.conf
bootstrap: staged under /tmp/TestBootstrapKeepsAnExistingRegistry2582281630/001 — on a real box (--root /) it would also:
useradd --system --user-group --no-create-home --shell /usr/sbin/nologin carlos
groupadd --system carlos-sockets
chown -R carlos:carlos /data
chown -R carlos:carlos-sockets /run/carlos
chown -R carlos:carlos /run/carlos/apps
chmod 2750 /run/carlos
systemctl daemon-reload
systemctl enable carlos-edge.service carlos-relay.service carlos-adopt.timer carlos-adopt.path carlos-stage.path carlos-restart.path carlos-system-update.path carlos-system-recover.service data-instances.mount run-carlos-instances.mount run-carlos-logs.mount carlos-instance-control.socket
systemctl start carlos-relay.service carlos-adopt.timer carlos-adopt.path carlos-stage.path carlos-restart.path carlos-system-update.path data-instances.mount run-carlos-instances.mount run-carlos-logs.mount carlos-instance-control.socket
systemctl restart systemd-journald
--- FAIL: TestRunAdoptOpensTheRegistryThroughTheHelper (0.00s)
registryowner_test.go:328: runAdoptWithEnsureSystemUnits no longer calls openServiceRegistry( the root pass has stopped converging ownership
2026/10/12 22:57:39 agent: restart requested: bounced exec instance exec.example.com
2026/10/12 22:57:39 agent: restart requested: bounced exec instance warm.example.com
2026/10/12 22:57:39 agent: restart requested: bounced exec instance exec.example.com
2026/10/12 22:57:39 agent: restart requested for 1 app(s) and narrowed for 0 on this box, asked for a root restart pass
2026/10/12 22:57:39 adopt: restart requested for bab/eleven: sleep.example.com is hibernating — left asleep (it execs fresh on its next wake)
2026/10/12 22:57:39 adopt: restart requested for bab/eleven: dispatched to 2 route(s)
2026/10/12 22:57:39 adopt: restart requested for bab/eleven: dispatched to 1 route(s)
2026/10/12 22:57:39 adopt: restart requested for bab/titogo: dispatched to 1 route(s)
2026/10/12 22:57:39 adopt: restart generation bab/eleven: 503 slow down (skipped this pass)
2026/10/12 22:57:39 adopt: restart pass: baselining deferred — some apps could not be read this pass
2026/10/12 22:57:39 adopt: restart pass: cannot lock /tmp/TestRestartPassRefusesWithoutALock2799332782/002/restart-marks.json.lock (restart-marks.json.lock: not a regular file with one link) — nothing cycled this pass
2026/10/12 22:57:39 adopt: restart requested for bab/eleven: dispatched to 1 route(s)
2026/10/12 22:57:39 adopt: restart requested for bab/eleven: console.example.com did not cycle: Unit wrong.service not loaded
2026/10/12 22:57:39 adopt: restart requested for bab/eleven: dispatched to 0 route(s)
2026/10/12 22:57:39 adopt: restart marks /tmp/TestRestartPassUnreadableMarksReBaseline781347771/002/restart-marks.json do not parse (invalid character 'n' looking for beginning of object key string) — re-baselining, no restarts this pass
2026/10/12 22:57:39 adopt: restart requested for bab/eleven: dispatched to 1 route(s)
2026/10/12 22:57:39 agent: restart requested: bounced sidecar bot.x.sidecar.internal
2026/10/12 22:57:39 agent: restart requested: bounced sidecar bot.x.sidecar.internal
2026/10/12 22:57:39 agent: restart requested: bounced sidecar bot.a.sidecar.internal
2026/10/12 22:57:39 agent: restart requested: bounced sidecar bot.b.sidecar.internal
2026/10/12 22:57:39 agent: restart requested: bounced sidecar bot.c.sidecar.internal
2026/10/12 22:57:39 agent: restart requested: bounced sidecar bot.bad.sidecar.internal
2026/10/12 22:57:39 agent: restart requested: bounced sidecar bot.good.sidecar.internal
2026/10/12 22:57:39 agent: restart requested: re-wake sidecar bot.bad.sidecar.internal: boom
2026/10/12 22:57:39 agent: restart requested: bounced sidecar bot.eleven.sidecar.internal
2026/10/12 22:57:39 agent: restart requested: bounced sidecar s0.x.sidecar.internal
2026/10/12 22:57:39 agent: restart requested: bounced sidecar s1.x.sidecar.internal
2026/10/12 22:57:39 agent: restart requested: bounced sidecar s2.x.sidecar.internal
2026/10/12 22:57:39 agent: restart requested: bounced sidecar s3.x.sidecar.internal
2026/10/12 22:57:39 agent: restart requested: bounced sidecar s4.x.sidecar.internal
2026/10/12 22:57:39 agent: restart requested: bounced sidecar s5.x.sidecar.internal
2026/10/12 22:57:39 agent: restart requested: bounced sidecar s6.x.sidecar.internal
2026/10/12 22:57:39 agent: restart requested: bounced sidecar s7.x.sidecar.internal
2026/10/12 22:57:39 agent: restart requested: bounced sidecar s8.x.sidecar.internal
2026/10/12 22:57:39 agent: restart requested: bounced sidecar s9.x.sidecar.internal
2026/10/12 22:57:39 agent: restart requested: bounced sidecar s10.x.sidecar.internal
2026/10/12 22:57:39 agent: restart requested: bounced sidecar s11.x.sidecar.internal
2026/10/12 22:57:39 agent: restart requested: bounced sidecar s12.x.sidecar.internal
2026/10/12 22:57:39 agent: narrowed restarts for bab/titogo: this box's position is no longer in the log — re-baselined, nothing cycled (run the verb again if something is still waiting)
2026/10/12 22:57:39 restart: narrowed request for bab/titogo names selector "region", which this build does not understand — nothing cycled for it (upgrade carlos on this box)
2026/10/12 22:57:39 agent: narrowed restart requested: bounced exec instance exec.example.com
2026/10/12 22:57:39 agent: narrowed restart requested: bounced exec instance b.example.com
2026/10/12 22:57:39 agent: restart requested for 0 app(s) and narrowed for 1 on this box, asked for a root restart pass
2026/10/12 22:57:39 adopt: narrowed restart for bab/titogo: 1 request(s), dispatched to 2 route(s)
2026/10/12 22:57:39 adopt: narrowed restart for bab/titogo: sleeper.example.com is hibernating — left asleep (it execs fresh on its next wake)
2026/10/12 22:57:39 adopt: narrowed restart for bab/titogo: 1 request(s), dispatched to 2 route(s)
2026/10/12 22:57:39 adopt: restart requested for bab/titogo: dispatched to 4 route(s)
2026/10/12 22:57:39 adopt: narrowed restart for bab/titogo: 1 request(s), dispatched to 0 route(s)
2026/10/12 22:57:39 restart: narrowed request for bab/titogo names selector "region", which this build does not understand — nothing cycled for it (upgrade carlos on this box)
2026/10/12 22:57:39 adopt: narrowed restart for bab/titogo: 1 request(s), dispatched to 0 route(s)
2026/10/12 22:57:39 adopt: narrowed restarts for bab/titogo: this box's position is no longer in the log — re-baselined, nothing cycled
2026/10/12 22:57:39 adopt: narrowed restart for bab/titogo: 1 request(s), dispatched to 1 route(s)
2026/10/12 22:57:39 adopt: narrowed restart for bab/titogo: 1 request(s), dispatched to 1 route(s)
2026/10/12 22:57:39 agent: restart baseline: the root pass has no narrowed-restart mark for at least one app on this box (a marks file written before narrowed restarts existed) — asked for one root pass to record it
2026/10/12 22:57:39 adopt: narrowed restart for bab/titogo: bill.example.com did not cycle: permission denied
2026/10/12 22:57:39 adopt: narrowed restart for bab/titogo: 1 request(s), dispatched to 0 route(s)
2026/10/12 22:57:39 agent: retry requested for b.example.com: its failure hold on v2 is cleared and it starts the build as today
2026/10/12 22:57:39 agent: retry for a.example.com: nothing held to clear for v2 at epoch 1048577 of book "1a8f833d121dcf2bc8e040340b108a93" (cleared already, a newer failure since the grant, or another book's grant)
2026/10/12 22:57:39 agent: retry for a.example.com: nothing held to clear for v9 at epoch 1048578 of book "1a8f833d121dcf2bc8e040340b108a93" (cleared already, a newer failure since the grant, or another book's grant)
2026/10/12 22:57:39 agent: retry for a.example.com: nothing held to clear for v2 at epoch 1048578 of book "another-book" (cleared already, a newer failure since the grant, or another book's grant)
2026/10/12 22:57:39 agent: retry for a.example.com: nothing held to clear for v2 at epoch 1048578 of book "" (cleared already, a newer failure since the grant, or another book's grant)
2026/10/12 22:57:39 agent: retry requested for a.example.com: its failure hold on v2 is cleared and it starts the build as today
2026/10/12 22:57:39 agent: retry for a.example.com: nothing held to clear for v2 at epoch 1048578 of book "1a8f833d121dcf2bc8e040340b108a93" (cleared already, a newer failure since the grant, or another book's grant)
2026/10/12 22:57:39 agent: retry for a.example.com: nothing held to clear for v2 at epoch 1048577 of book "1a8f833d121dcf2bc8e040340b108a93" (cleared already, a newer failure since the grant, or another book's grant)
2026/10/12 22:57:39 agent: retry requested for a.example.com: its failure hold on v2 is cleared and it starts the build as today
2026/10/12 22:57:39 agent: narrowed restart requested: bounced exec instance b.example.com
2026/10/12 22:57:39 agent: restart requested for 0 app(s) and narrowed for 1 on this box, asked for a root restart pass
2026/10/12 22:57:39 status: fleet: skipping status/boxes/beta-1.json: corrupt snapshot: unexpected end of JSON input
2026/10/12 22:57:39 agent: retention: listing fleets: AccessDenied
2026/10/12 22:57:39 agent: retention: pruned 1 of 1 eligible versions for bab/kass (keep 20)
2026/10/12 22:57:39 agent: retention bab/kass: pruning nothing this pass: no box report names the build state of kass.example.com
2026/10/12 22:57:40 agent: retention: pruned 1 of 1 eligible versions for bab/kass (keep 20)
2026/10/12 22:57:40 agent: retention bab/kass: pruning nothing this pass: this box (sole housekeeper) could not collect its build states: database is locked
2026/10/12 22:57:40 agent: retention bab/kass: pruning nothing this pass: no box report names the build state of kass.example.com
2026/10/12 22:57:40 agent: retention: listing fleets: AccessDenied
2026/10/12 22:57:40 status: fleet: skipping status/boxes/door-1.json: read failed: SlowDown
2026/10/12 22:57:40 agent: retention: listing fleets: AccessDenied
2026/10/12 22:57:40 agent: retention: pruned 3 of 3 eligible versions for bab/kass (keep 20)
2026/10/12 22:57:40 agent: retention: pruned 3 of 3 eligible versions for bab/kass (keep 20)
2026/10/12 22:57:40 agent: retention: pruned 2 of 2 eligible versions for bab/kass (keep 20)
2026/10/12 22:57:40 agent: restart requested: bounced exec instance exec.example.com
2026/10/12 22:57:40 agent: restart requested for 1 app(s) and narrowed for 0 on this box, asked for a root restart pass
2026/10/12 22:57:40 agent: overlap swaps armed (CARLOS_SWAP=overlap)
2026/10/12 22:57:40 agent: CARLOS_SWAP=overlap: deploy store: no deployment store
2026/10/12 22:57:40 agent: CARLOS_SWAP=overlap but no-deploy-store: refusing every managed start, and every stop and move onto a new build, until the agent restarts with it readable
2026/10/12 22:57:40 agent: restart requested for 1 app(s) and narrowed for 0 on this box, asked for a root restart pass
2026/10/12 22:57:40 agent: CARLOS_SWAP=overlap: release key: CARLOS_RELEASE_PUBKEY: expected a base64 32-byte Ed25519 public key
2026/10/12 22:57:40 agent: CARLOS_SWAP=overlap but no-release-key: refusing every managed start, and every stop and move onto a new build, until the agent restarts with it readable
2026/10/12 22:57:40 agent: restart requested for 1 app(s) and narrowed for 0 on this box, asked for a root restart pass
2026/10/12 22:57:40 agent: CARLOS_SWAP=overlap: failure book: /tmp/TestUnavailableStateRefusesManagedStartsunreadable_book3285706033/001/swap-failures.json: invalid character 'n' looking for beginning of object key string
2026/10/12 22:57:40 agent: CARLOS_SWAP=overlap but failure-book-unreadable: refusing every managed start, and every stop and move onto a new build, until the agent restarts with it readable
2026/10/12 22:57:40 agent: restart requested for 1 app(s) and narrowed for 0 on this box, asked for a root restart pass
2026/10/12 22:57:40 agent: CARLOS_SWAP=overlap: failure book: /tmp/TestUnavailableStateRefusesManagedStartsbook_with_a_bad_schema4114873496/001/swap-failures.json: unknown schema 9
2026/10/12 22:57:40 agent: CARLOS_SWAP=overlap but failure-book-unreadable: refusing every managed start, and every stop and move onto a new build, until the agent restarts with it readable
2026/10/12 22:57:40 agent: restart requested for 1 app(s) and narrowed for 0 on this box, asked for a root restart pass
2026/10/12 22:57:40 agent: CARLOS_SWAP=overlap: 1 execution(s) an earlier agent left unrecorded are now failure records
2026/10/12 22:57:40 rollout: bab/blog channel stable: dirstore: apps/bab/blog/channels/stable.json: file does not exist — if this deployment predates account-scoped bucket keys its objects are still at apps/blog/, which the runtime no longer reads; move them once with: carlos accounts migrate --app blog --from legacy --to bab (retry not checked this pass)
2026/10/12 22:57:40 agent: overlap swaps armed (CARLOS_SWAP=overlap)
2026/10/12 22:57:40 agent: CARLOS_SWAP=overlap: 1 execution(s) an earlier agent left unrecorded are now failure records
2026/10/12 22:57:40 agent: CARLOS_SWAP=overlap: save the failure book after opening it: permission denied
2026/10/12 22:57:40 agent: CARLOS_SWAP=overlap but failure-book-unreadable: refusing every managed start, and every stop and move onto a new build, until the agent restarts with it readable
2026/10/12 22:57:40 agent: CARLOS_SWAP=overlap: save the failure book after opening it: permission denied
2026/10/12 22:57:40 agent: CARLOS_SWAP=overlap but failure-book-unreadable: refusing every managed start, and every stop and move onto a new build, until the agent restarts with it readable
2026/10/12 22:57:40 agent: overlap swaps armed (CARLOS_SWAP=overlap)
2026/10/12 22:57:40 agent: REFUSING CARLOS_SWAP=overlap (no-cgroup-delegation): generation containment needs the agent's delegated cgroup subtree and the control channel's generation units, so this box stays on today's stop-then-wake paths
2026/10/12 22:57:40 agent: restart requested: bounced exec instance exec.example.com
2026/10/12 22:57:40 agent: restart requested for 1 app(s) and narrowed for 0 on this box, asked for a root restart pass
2026/10/12 22:57:40 agent: REFUSING CARLOS_SWAP=overlap (no-generation-control): generation containment needs the agent's delegated cgroup subtree and the control channel's generation units, so this box stays on today's stop-then-wake paths
2026/10/12 22:57:40 agent: CARLOS_SWAP=overlap: deploy store: no deployment store: secret detail
2026/10/12 22:57:40 agent: CARLOS_SWAP=overlap but no-deploy-store: refusing every managed start, and every stop and move onto a new build, until the agent restarts with it readable
asleep.example.com: config updated
asleep.example.com: config updated, asleep: delivered on wake
awake.example.com: config updated
awake.example.com: bounce after config change: restart refused: this box cannot restart instances right now
plain.example.com: config updated
2026/10/12 22:57:40 agent: overlap swaps armed (CARLOS_SWAP=overlap)
bot.eleven.bab.sidecar.internal: config updated
m.example.com: config updated
plain.example.com: config updated
2026/10/12 22:57:40 agent: restart requested: bounced exec instance exec.example.com
2026/10/12 22:57:40 agent: restart requested for 1 app(s) and narrowed for 0 on this box, asked for a root restart pass
2026/10/12 22:57:40 agent: restart requested: bounced exec instance exec.example.com
2026/10/12 22:57:40 agent: restart requested for 1 app(s) and narrowed for 0 on this box, asked for a root restart pass
2026/10/12 22:57:40 agent: CARLOS_SWAP=overlap declined, but 1 failure record(s) and 0 hold-ran marker(s) from an armed run are honoured: no managed instance wakes, starts, or is stopped, onto a build they hold for it
2026/10/12 22:57:40 agent: CARLOS_SWAP=overlap declined, but 1 failure record(s) and 0 hold-ran marker(s) from an armed run are honoured: no managed instance wakes, starts, or is stopped, onto a build they hold for it
2026/10/12 22:57:40 agent: restart: leaving held.example.com running: a failure record or hold-ran marker from an armed run holds build v2 for it, and CARLOS_SWAP=overlap is declined on this box
2026/10/12 22:57:40 agent: restart requested: bounced exec instance free.example.com
2026/10/12 22:57:40 agent: restart requested: bounced exec instance nogood.example.com
2026/10/12 22:57:40 agent: restart requested for 1 app(s) and narrowed for 0 on this box, asked for a root restart pass
free.example.com: config updated
held.example.com: config updated
2026/10/12 22:57:40 agent: config sync: leaving held.example.com running: a failure record or hold-ran marker from an armed run holds build v2 for it, and CARLOS_SWAP=overlap is declined on this box
held.example.com: bounce after config change: restart refused: this box cannot restart instances right now
nogood.example.com: config updated
sleeper.example.com: config updated
2026/10/12 22:57:40 agent: config sync: leaving sleeper.example.com running: a failure record or hold-ran marker from an armed run holds build v2 for it, and CARLOS_SWAP=overlap is declined on this box
sleeper.example.com: config updated, asleep: delivered on wake
2026/10/12 22:57:40 agent: CARLOS_SWAP=overlap declined, but 1 failure record(s) and 0 hold-ran marker(s) from an armed run are honoured: no managed instance wakes, starts, or is stopped, onto a build they hold for it
2026/10/12 22:57:40 agent: converge: leaving warm.test running: a failure record or hold-ran marker from an armed run holds build v2 for it, and CARLOS_SWAP=overlap is declined on this box
2026/10/12 22:57:40 agent: adopt bounce: leaving warm.test running: a failure record or hold-ran marker from an armed run holds build v2 for it, and CARLOS_SWAP=overlap is declined on this box
2026/10/12 22:57:40 activator: warm.test hibernated
2026/10/12 22:57:40 activator: wake warm.test: warm.test: refusing to wake: this box cannot run two builds side by side, which this instance's build needs
2026/10/12 22:57:40 agent: CARLOS_SWAP=overlap declined: 1 execution(s) an armed run left unrecorded are now failure records
2026/10/12 22:57:40 agent: CARLOS_SWAP=overlap declined, but 1 failure record(s) and 1 hold-ran marker(s) from an armed run are honoured: no managed instance wakes, starts, or is stopped, onto a build they hold for it
2026/10/12 22:57:40 agent: CARLOS_SWAP=overlap declined: 1 execution(s) an armed run left unrecorded are now failure records
2026/10/12 22:57:40 agent: CARLOS_SWAP=overlap declined: save the failure book after reconciling it: permission denied
2026/10/12 22:57:40 agent: CARLOS_SWAP=overlap declined, but 1 failure record(s) and 1 hold-ran marker(s) from an armed run are honoured: no managed instance wakes, starts, or is stopped, onto a build they hold for it
2026/10/12 22:57:40 agent: CARLOS_SWAP=overlap declined, and the failure book is unreadable: /tmp/TestDeclinedBoxWithAnUnreadableBookFailsClosednot_json3611535387/002/swap-failures.json: invalid character 'n' looking for beginning of object key string
2026/10/12 22:57:40 agent: CARLOS_SWAP=overlap but failure-book-unreadable: refusing every managed start, and every stop and move onto a new build, until the agent restarts with it readable
2026/10/12 22:57:40 agent: CARLOS_SWAP=overlap declined, and the failure book is unreadable: /tmp/TestDeclinedBoxWithAnUnreadableBookFailsClosedbad_schema2083617541/002/swap-failures.json: unknown schema 9
2026/10/12 22:57:40 agent: CARLOS_SWAP=overlap but failure-book-unreadable: refusing every managed start, and every stop and move onto a new build, until the agent restarts with it readable
2026/10/12 22:57:40 agent: CARLOS_SWAP=overlap declined, but 1 failure record(s) and 0 hold-ran marker(s) from an armed run are honoured: no managed instance wakes, starts, or is stopped, onto a build they hold for it
2026/10/12 22:57:40 activator: wake warm.test: start warm.test: this box cannot run two builds side by side, which this instance's build needs
2026/10/12 22:57:40 agent: CARLOS_SWAP=overlap declined, but 0 failure record(s) and 3 hold-ran marker(s) from an armed run are honoured: no managed instance wakes, starts, or is stopped, onto a build they hold for it
2026/10/12 22:57:40 agent: CARLOS_SWAP=overlap declined, but 4 failure record(s) and 2 hold-ran marker(s) from an armed run are honoured: no managed instance wakes, starts, or is stopped, onto a build they hold for it
2026/10/12 22:57:40 agent: CARLOS_SWAP=overlap declined, but 1 failure record(s) and 0 hold-ran marker(s) from an armed run are honoured: no managed instance wakes, starts, or is stopped, onto a build they hold for it
2026/10/12 22:57:40 agent: restart requested: bounced exec instance exec.example.com
2026/10/12 22:57:40 agent: restart requested for 1 app(s) and narrowed for 0 on this box, asked for a root restart pass
2026/10/12 22:57:40 agent: restart requested: bounced exec instance exec.example.com
2026/10/12 22:57:40 agent: restart requested for 1 app(s) and narrowed for 0 on this box, asked for a root restart pass
2026/10/12 22:57:40 agent: CARLOS_SWAP=overlap declined, but 1 failure record(s) and 0 hold-ran marker(s) from an armed run are honoured: no managed instance wakes, starts, or is stopped, onto a build they hold for it
2026/10/12 22:57:40 agent: CARLOS_SWAP=overlap declined, but 1 failure record(s) and 1 hold-ran marker(s) from an armed run are honoured: no managed instance wakes, starts, or is stopped, onto a build they hold for it
2026/10/12 22:57:40 agent: CARLOS_SWAP=overlap declined, but 1 failure record(s) and 0 hold-ran marker(s) from an armed run are honoured: no managed instance wakes, starts, or is stopped, onto a build they hold for it
2026/10/12 22:57:40 agent: CARLOS_SWAP=overlap declined, but 1 failure record(s) and 0 hold-ran marker(s) from an armed run are honoured: no managed instance wakes, starts, or is stopped, onto a build they hold for it
2026/10/12 22:57:40 agent: retry v9 on h.example.com: the grant is for epoch 2097154, the restriction's is 0, so nothing was cleared
2026/10/12 22:57:40 agent: retry for h.example.com: nothing held to clear for v9 at epoch 2097154 of book "0eb793966d6fdf8479c7d7d0260690cd" (cleared already, a newer failure since the grant, or another book's grant)
2026/10/12 22:57:40 agent: retry v2 on h.example.com: the grant is for epoch 1048577, the restriction's is 2097154, so nothing was cleared
2026/10/12 22:57:40 agent: retry for h.example.com: nothing held to clear for v2 at epoch 1048577 of book "0eb793966d6fdf8479c7d7d0260690cd" (cleared already, a newer failure since the grant, or another book's grant)
2026/10/12 22:57:40 agent: retry requested for h.example.com: its failure hold on v2 is cleared and it starts the build as today
2026/10/12 22:57:40 agent: CARLOS_SWAP=overlap declined, but 1 failure record(s) and 0 hold-ran marker(s) from an armed run are honoured: no managed instance wakes, starts, or is stopped, onto a build they hold for it
2026/10/12 22:57:40 rollout: bab/blog channel stable: bucket blip (retry not checked this pass)
2026/10/12 22:57:40 rollout: bab/blog: a signed pointer re-promoted v2, so its failure record is cleared
2026/10/12 22:57:40 agent: CARLOS_SWAP=overlap declined, but 0 failure record(s) and 2 hold-ran marker(s) from an armed run are honoured: no managed instance wakes, starts, or is stopped, onto a build they hold for it
2026/10/12 22:57:40 rollout: m.example.com: a signed rollback to v1 (pointer of 2026-10-01T12:01:00Z) cleared its hold-ran marker
2026/10/12 22:57:40 agent: CARLOS_SWAP=overlap declined, but 1 failure record(s) and 0 hold-ran marker(s) from an armed run are honoured: no managed instance wakes, starts, or is stopped, onto a build they hold for it
2026/10/12 22:57:40 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/12 22:57:40 agent: retry requested for h.blog.example.com: its failure hold on v2 is cleared and it starts the build as today
platform.example.com: update to v2 failed: console.service did not come up: unit entered failed state — rolled the symlink back
platform.example.com: update to v2 failed: console.service did not come up: unit entered failed state — rolled the symlink back
platform.example.com: rolled back to v1 but could not start it: permission denied — this host may be serving nothing
platform.example.com: update to v2 failed: console.service did not come up: unit entered failed state — rolled the symlink back
platform.example.com: rolled back to v1 but could not start it: permission denied — this host may be serving nothing
platform.example.com: update to v2 failed: console.service did not come up: unit entered failed state — rolled the symlink back
platform.example.com: rolled back to v1 but could not start it: file does not exist — this host may be serving nothing
platform.example.com: update to v2 failed: console.service did not come up: unit entered failed state — rolled the symlink back
platform.example.com: rolled back to v1 but could not start it: permission denied — this host may be serving nothing
platform.example.com: update to v2 failed: console.service did not come up: unit entered failed state — rolled the symlink back
platform.example.com: update to v2 failed: console.service did not come up: unit entered failed state — rolled the symlink back
platform.example.com: rolled back to v1 but could not start it: permission denied — this host may be serving nothing
platform.example.com: update to v2 failed: console.service did not come up: unit entered failed state — rolled the symlink back
platform.example.com: rolled back to v1 but could not start it: unit was stopped during settle — this host may be serving nothing
platform.example.com: update to v2 failed: console.service did not come up: unit entered failed state — rolled the symlink back
platform.example.com: rolled back to v1 but could not start it: permission denied — this host may be serving nothing
platform.example.com: update to v2 failed: console.service did not come up: unit entered failed state — rolled the symlink back
platform.example.com: rolled back to v1 but could not start it: console.service did not come up: running /tmp/TestPlan14RollbackCapabilitybad_executable2502171697/001/versions/v2/console, not the new build in /tmp/TestPlan14RollbackCapabilitybad_executable2502171697/001/versions/v1 — this host may be serving nothing
platform.example.com: update to v2 failed: console.service did not come up: permission denied
permission denied — rolled the symlink back
platform.example.com: update to v2 failed: console.service did not come up: unit entered failed state — rolled the symlink back
platform.example.com: rolled back to v1 but could not start it: file does not exist — this host may be serving nothing
platform.example.com: update to v2 failed: console.service did not come up: unit entered failed state — rolled the symlink back
platform.example.com: rolled back to v1 but could not start it: permission denied — this host may be serving nothing
platform.example.com: update to v2 failed: console.service did not come up: unit entered failed state — rolled the symlink back
platform.example.com: rolled back to v1 but could not start it: permission denied — this host may be serving nothing
platform.example.com: update to v2 failed: permission denied — rolled the symlink back
platform.example.com: updated v1 -> v2 (canary/console)
platform.example.com: update to v2 failed: console.service did not come up: unit entered failed state — rolled the symlink back
2026/10/12 22:57:41 activator: rearm.bab.oncarlos.com idle 0s — hibernating
2026/10/12 22:57:41 activator: rearm.bab.oncarlos.com hibernated
2026/10/12 22:57:41 activator: rearm.bab.oncarlos.com hibernated
2026/10/12 22:57:43 activator: sleepy.bab.oncarlos.com idle 1s — hibernating
2026/10/12 22:57:43 activator: sleepy.bab.oncarlos.com hibernated
2026/10/12 22:57:43 agent: schedules from the bucket are off (no deploy store on this box); once-timers registered by instances still run (runner every 15s)
2026/10/12 22:57:43 agent: scheduled work armed (converge on serial + hourly; runner every 15s)
2026/10/12 22:57:43 agent: scheduled work disabled: no deploy store
2026/10/12 22:57:43 activator: storeless.bab.oncarlos.com hibernated
2026/10/12 22:57:43 http: proxy error: context deadline exceeded
2026/10/12 22:57:43 activator: ticky.bab.oncarlos.com idle 0s — hibernating
2026/10/12 22:57:43 activator: ticky.bab.oncarlos.com hibernated
2026/10/12 22:57:44 activator: ticky.bab.oncarlos.com hibernated
2026/10/12 22:57:44 activator: hdr.bab.oncarlos.com hibernated
2026/10/12 22:57:44 agent: sidecar wake requested for 1 sidecar(s) on this box
2026/10/12 22:57:44 agent: sidecar wake requested: woke bot.app.acct.sidecar.internal
2026/10/12 22:57:44 agent: sidecar wake requested: woke bot.b.sidecar.internal
2026/10/12 22:57:44 agent: sidecar wake requested: woke bot.a.sidecar.internal
2026/10/12 22:57:44 agent: sidecar wake requested: woke bot.eleven.sidecar.internal
2026/10/12 22:57:44 agent: sidecar wake requested: woke sc.x.sidecar.internal
2026/10/12 22:57:44 agent: sidecar wake requested: woke sd.x.sidecar.internal
2026/10/12 22:57:44 agent: sidecar wake requested: woke si.x.sidecar.internal
2026/10/12 22:57:44 agent: sidecar wake requested: woke sg.x.sidecar.internal
2026/10/12 22:57:44 agent: sidecar wake requested: woke sf.x.sidecar.internal
2026/10/12 22:57:44 agent: sidecar wake requested: woke sk.x.sidecar.internal
2026/10/12 22:57:44 agent: sidecar wake requested: woke sh.x.sidecar.internal
2026/10/12 22:57:44 agent: sidecar wake requested: woke sb.x.sidecar.internal
2026/10/12 22:57:44 agent: sidecar wake requested: woke sa.x.sidecar.internal
2026/10/12 22:57:44 agent: sidecar wake requested: woke se.x.sidecar.internal
2026/10/12 22:57:44 agent: sidecar wake requested: woke sj.x.sidecar.internal
2026/10/12 22:57:44 agent: sidecar wake requested for 1 sidecar(s) on this box
2026/10/12 22:57:44 agent: sidecar wake requested: woke bot.eleven.sidecar.internal
2026/10/12 22:57:44 agent: sidecar wake requested for 1 sidecar(s) on this box
2026/10/12 22:57:44 agent: sidecar wake requested: woke bot.eleven.sidecar.internal
2026/10/12 22:57:44 agent: sidecar wake requested for 1 sidecar(s) on this box
2026/10/12 22:57:44 agent: sidecar wake requested: woke bot.eleven.sidecar.internal
2026/10/12 22:57:44 agent: sidecar wake requested: woke s0.x.sidecar.internal
2026/10/12 22:57:44 agent: sidecar wake requested: woke s6.x.sidecar.internal
2026/10/12 22:57:44 agent: sidecar wake requested: woke s8.x.sidecar.internal
2026/10/12 22:57:44 agent: sidecar wake requested: woke s4.x.sidecar.internal
2026/10/12 22:57:44 agent: sidecar wake requested: woke s2.x.sidecar.internal
2026/10/12 22:57:44 agent: sidecar wake requested: woke s7.x.sidecar.internal
2026/10/12 22:57:44 agent: sidecar wake requested: woke s12.x.sidecar.internal
2026/10/12 22:57:44 agent: sidecar wake requested: woke s1.x.sidecar.internal
2026/10/12 22:57:44 agent: sidecar wake requested: woke s5.x.sidecar.internal
2026/10/12 22:57:44 agent: sidecar wake requested: woke s3.x.sidecar.internal
2026/10/12 22:57:44 agent: sidecar wake requested: woke s9.x.sidecar.internal
2026/10/12 22:57:44 agent: sidecar wake requested: woke s10.x.sidecar.internal
2026/10/12 22:57:44 agent: sidecar wake requested: woke s11.x.sidecar.internal
2026/10/12 22:57:44 agent: echo.example.com needs a root pass for abc1234 (a build only root can stage or install, or an install request only root resolves); asked for one
2026/10/12 22:57:44 agent: echo.example.com needs a root pass for abc1234 (a build only root can stage or install, or an install request only root resolves); asked for one
2026/10/12 22:57:44 agent: a.example.com needs a root pass for v1 (a build only root can stage or install, or an install request only root resolves); asked for one
2026/10/12 22:57:44 agent: b.example.com needs a root pass for v1 (a build only root can stage or install, or an install request only root resolves); asked for one
2026/10/12 22:57:44 agent: a needs a root pass for v2 (a build only root can stage or install, or an install request only root resolves); asked for one
2026/10/12 22:57:44 agent: a needs a root pass for v2 (a build only root can stage or install, or an install request only root resolves); asked for one
2026/10/12 22:57:44 agent: a needs a root pass for v2 (a build only root can stage or install, or an install request only root resolves); asked for one
2026/10/12 22:57:49 agent: system pointer stable names v2 with no artifact for linux-amd64 — not for this box
2026/10/12 22:57:49 agent: system update v2: fetch bootstrap/router/v2/carlos-linux-arm64: sha256 b5c1fb2efc6d6b4674c2fdcc48ce01b43a3b7c03763c0c3355de0099ee0f8c73 does not match the signed 770e607624d689265ca6c44884d0807d9b054d23c473c106c72be9de08b7376c
2026/10/12 22:57:49 agent: system update v2: store unreachable
2026/10/12 22:57:49 agent: system update v2: store unreachable
2026/10/12 22:57:49 agent: system update v2: store unreachable
2026/10/12 22:57:50 agent: tunnel loop armed (http://127.0.0.1:34363 as "pi-box")
2026/10/12 22:57:50 agent: tunnel connected as "pi-box"
2026/10/12 22:57:50 http2: server: error reading preface from client 127.0.0.1:34363: read tcp 127.0.0.1:52390->127.0.0.1:34363: read: connection reset by peer
2026/10/12 22:57:50 agent: tunnel lost — re-dialing
2026/10/12 22:57:50 agent: tunnel retry in 657.88633ms
2026/10/12 22:57:50 agent: tunnel loop disabled: need all of CARLOS_TUNNEL_URL, CARLOS_TUNNEL_TOKEN, CARLOS_TUNNEL_NAME
2026/10/12 22:57:50 agent: tunnel loop disabled: need all of CARLOS_TUNNEL_URL, CARLOS_TUNNEL_TOKEN, CARLOS_TUNNEL_NAME
2026/10/12 22:57:50 agent: tunnel loop disabled: need all of CARLOS_TUNNEL_URL, CARLOS_TUNNEL_TOKEN, CARLOS_TUNNEL_NAME
2026/10/12 22:57:50 agent: tunnel loop disabled: need all of CARLOS_TUNNEL_URL, CARLOS_TUNNEL_TOKEN, CARLOS_TUNNEL_NAME
2026/10/12 22:57:50 agent: tunnel loop disabled: need all of CARLOS_TUNNEL_URL, CARLOS_TUNNEL_TOKEN, CARLOS_TUNNEL_NAME
2026/10/12 22:57:50 agent: tunnel loop disabled: need all of CARLOS_TUNNEL_URL, CARLOS_TUNNEL_TOKEN, CARLOS_TUNNEL_NAME
created /tmp/TestBootstrapCreatesTheInstanceDirectories162423356/001/data/registry.db
wrote /tmp/TestBootstrapCreatesTheInstanceDirectories162423356/001/etc/systemd/system/carlos-adopt.path
wrote /tmp/TestBootstrapCreatesTheInstanceDirectories162423356/001/etc/systemd/system/carlos-adopt.service
wrote /tmp/TestBootstrapCreatesTheInstanceDirectories162423356/001/etc/systemd/system/carlos-adopt.timer
wrote /tmp/TestBootstrapCreatesTheInstanceDirectories162423356/001/etc/systemd/system/carlos-app@.service
wrote /tmp/TestBootstrapCreatesTheInstanceDirectories162423356/001/etc/systemd/system/carlos-app@.socket
wrote /tmp/TestBootstrapCreatesTheInstanceDirectories162423356/001/etc/systemd/system/carlos-config-sync.service
wrote /tmp/TestBootstrapCreatesTheInstanceDirectories162423356/001/etc/systemd/system/carlos-edge.service
wrote /tmp/TestBootstrapCreatesTheInstanceDirectories162423356/001/etc/systemd/system/carlos-hibernate-verify.service
wrote /tmp/TestBootstrapCreatesTheInstanceDirectories162423356/001/etc/systemd/system/carlos-hibernate-verify.timer
wrote /tmp/TestBootstrapCreatesTheInstanceDirectories162423356/001/etc/systemd/system/carlos-instance-control.service
wrote /tmp/TestBootstrapCreatesTheInstanceDirectories162423356/001/etc/systemd/system/carlos-instance-control.socket
wrote /tmp/TestBootstrapCreatesTheInstanceDirectories162423356/001/etc/systemd/system/carlos-instance-gen@.service
wrote /tmp/TestBootstrapCreatesTheInstanceDirectories162423356/001/etc/systemd/system/carlos-instance-probe.service
wrote /tmp/TestBootstrapCreatesTheInstanceDirectories162423356/001/etc/systemd/system/carlos-instance-seed@.service
wrote /tmp/TestBootstrapCreatesTheInstanceDirectories162423356/001/etc/systemd/system/carlos-instance-sidecar@.service
wrote /tmp/TestBootstrapCreatesTheInstanceDirectories162423356/001/etc/systemd/system/carlos-instance@.service
wrote /tmp/TestBootstrapCreatesTheInstanceDirectories162423356/001/etc/systemd/system/carlos-instances.slice
wrote /tmp/TestBootstrapCreatesTheInstanceDirectories162423356/001/etc/systemd/system/carlos-litestream.service
wrote /tmp/TestBootstrapCreatesTheInstanceDirectories162423356/001/etc/systemd/system/carlos-relay.service
wrote /tmp/TestBootstrapCreatesTheInstanceDirectories162423356/001/etc/systemd/system/carlos-restart.path
wrote /tmp/TestBootstrapCreatesTheInstanceDirectories162423356/001/etc/systemd/system/carlos-restart.service
wrote /tmp/TestBootstrapCreatesTheInstanceDirectories162423356/001/etc/systemd/system/carlos-restore-verify.service
wrote /tmp/TestBootstrapCreatesTheInstanceDirectories162423356/001/etc/systemd/system/carlos-restore-verify.timer
wrote /tmp/TestBootstrapCreatesTheInstanceDirectories162423356/001/etc/systemd/system/carlos-stage.path
wrote /tmp/TestBootstrapCreatesTheInstanceDirectories162423356/001/etc/systemd/system/carlos-stage.service
wrote /tmp/TestBootstrapCreatesTheInstanceDirectories162423356/001/etc/systemd/system/carlos-system-recover.service
wrote /tmp/TestBootstrapCreatesTheInstanceDirectories162423356/001/etc/systemd/system/carlos-system-update.path
wrote /tmp/TestBootstrapCreatesTheInstanceDirectories162423356/001/etc/systemd/system/carlos-system-update.service
wrote /tmp/TestBootstrapCreatesTheInstanceDirectories162423356/001/etc/systemd/system/carlos.slice
wrote /tmp/TestBootstrapCreatesTheInstanceDirectories162423356/001/etc/systemd/system/data-instances.mount
wrote /tmp/TestBootstrapCreatesTheInstanceDirectories162423356/001/etc/systemd/system/run-carlos-instances.mount
wrote /tmp/TestBootstrapCreatesTheInstanceDirectories162423356/001/etc/systemd/system/run-carlos-logs.mount
wrote /tmp/TestBootstrapCreatesTheInstanceDirectories162423356/001/etc/sysusers.d/carlos.conf
wrote /tmp/TestBootstrapCreatesTheInstanceDirectories162423356/001/etc/systemd/journald.conf.d/50-cap.conf
wrote /tmp/TestBootstrapCreatesTheInstanceDirectories162423356/001/etc/tmpfiles.d/carlos.conf
bootstrap: staged under /tmp/TestBootstrapCreatesTheInstanceDirectories162423356/001 — on a real box (--root /) it would also:
useradd --system --user-group --no-create-home --shell /usr/sbin/nologin carlos
groupadd --system carlos-sockets
chown -R carlos:carlos /data
chown -R carlos:carlos-sockets /run/carlos
chown -R carlos:carlos /run/carlos/apps
chmod 2750 /run/carlos
systemctl daemon-reload
systemctl enable carlos-edge.service carlos-relay.service carlos-adopt.timer carlos-adopt.path carlos-stage.path carlos-restart.path carlos-system-update.path carlos-system-recover.service data-instances.mount run-carlos-instances.mount run-carlos-logs.mount carlos-instance-control.socket
systemctl start carlos-relay.service carlos-adopt.timer carlos-adopt.path carlos-stage.path carlos-restart.path carlos-system-update.path data-instances.mount run-carlos-instances.mount run-carlos-logs.mount carlos-instance-control.socket
systemctl restart systemd-journald
2026/10/12 22:57:50 adopt: console.service: cannot read /proc/7/exe (readlink /proc/7/exe: permission denied); binary check skipped
2026/10/12 22:57:50 adopt: binding: console.service runs /tmp/TestNamedRestartBindingMismatchKeepsTheError1559399066/001/live/console.example.com, but the unit stamp is on platform.example.com; restart not checked
2026/10/12 22:57:50 adopt: platform.example.com: cannot read console.service (Access denied); restart not checked
2026/10/12 22:57:50 agent: tunnel dial: dial tcp 127.0.0.1:34363: connect: connection refused (retry in 632.917474ms)
2026/10/12 22:57:51 agent: tunnel dial: dial tcp 127.0.0.1:34363: connect: connection refused (retry in 1.015812624s)
2026/10/12 22:57:52 agent: tunnel dial: dial tcp 127.0.0.1:34363: connect: connection refused (retry in 2.261568499s)
2026/10/12 22:57:54 agent: tunnel dial: dial tcp 127.0.0.1:34363: connect: connection refused (retry in 2.22036128s)
2026/10/12 22:57:55 adopt: platform.example.com: console.service settled on /tmp/TestNamedRestartSettlesAHealthyForwardRestart1931537072/001/versions/new
2026/10/12 22:57:55 adopt: platform.example.com: console.service was stopped during settle; not judged, not restarted
2026/10/12 22:57:55 adopt: platform.example.com: console.service was stopped during the rollback's settle; not judged, not restarted
first.example.com: updated -> v1 (canary/x)
blocked.example.com: failed (v1): nothing installed
first.example.com: failed (v1): nothing installed
held.example.com: failed (v1): nothing installed
retried.example.com: updated -> v1 (canary/x)
served.example.com: updated -> v1 (canary/x)
2026/10/12 22:57:55 agent: analytics flush bab/web 2026-09-07: context deadline exceeded (re-queued)
2026/10/12 22:57:55 adopt: staged-version sweep: removed 1 of 3 staged versions (6 bytes), 2 pinned (0 live), 0 within the 6h0m0s grace: [v1]
2026/10/12 22:57:55 adopt: staged-version sweep: removed 1 of 1 staged versions (6 bytes), 0 pinned (0 live), 0 within the 6h0m0s grace: [v1]
2026/10/12 22:57:55 adopt: staged-version sweep skipped: list routes: database is locked
2026/10/12 22:57:55 adopt: staged-version sweep skipped: read live link /tmp/TestSweepStagedVersionsDeletesNothingWhenLivePinsFail2483361686/001/live/stray: readlink /tmp/TestSweepStagedVersionsDeletesNothingWhenLivePinsFail2483361686/001/live/stray: invalid argument
2026/10/12 22:57:55 adopt: staged-version sweep: 1 staged, 1 pinned (1 live), 0 within the 6h0m0s grace, none removed
2026/10/12 22:57:55 adopt: staged-version sweep: removed 1 of 4 staged versions (6 bytes), 3 pinned (0 live), 0 within the 6h0m0s grace: [v0]
2026/10/12 22:57:55 adopt: staged-version sweep skipped: generations: generations file /tmp/TestSweepStagedVersionsDeletesNothingWhenTheGenerationsFileIsInc3444517605/002/zdd/generations.json: unexpected end of JSON input
2026/10/12 22:57:55 adopt: staged-version sweep skipped: generations: generations file /tmp/TestSweepStagedVersionsDeletesNothingWhenTheGenerationsFileIsInc98678147/002/zdd/generations.json: publication 1 (complete true) while /tmp/TestSweepStagedVersionsDeletesNothingWhenTheGenerationsFileIsInc98678147/002/zdd/generations.seq names publication 2: incomplete
2026/10/12 22:57:55 adopt: CARLOS_VERSIONS_GRACE="soon" is not a non-negative duration — sweeping with the default 6h0m0s
2026/10/12 22:57:55 adopt: CARLOS_VERSIONS_GRACE="-1h" is not a non-negative duration — sweeping with the default 6h0m0s
2026/10/12 22:57:55 adopt: staged-version sweep: removed 1 of 2 staged versions (6 bytes), 1 pinned (0 live), 0 within the 6h0m0s grace: [gone]
2026/10/12 22:57:55 adopt: staged-version sweep: removed 1 of 5 staged versions (6 bytes), 4 pinned (0 live), 0 within the 6h0m0s grace: [v4]
2026/10/12 22:57:55 adopt: staged-version sweep: 1 staged, 1 pinned (0 live), 0 within the 6h0m0s grace, none removed
2026/10/12 22:57:55 adopt: staged-version sweep skipped: the build state of held a.example.com could not be read at its last refresh (s3: timeout)
2026/10/12 22:57:55 adopt: staged-version sweep: removed 1 of 3 staged versions (6 bytes), 2 pinned (0 live), 0 within the 6h0m0s grace: [B]
2026/10/12 22:57:55 adopt: staged-version sweep skipped: the build state of held a.example.com is not read yet
2026/10/12 22:57:55 adopt: staged-version sweep skipped: build state of a.example.com: build state copy: unexpected end of JSON input
2026/10/12 22:57:55 adopt: staged-version sweep skipped: update settings of a.example.com: update settings: unexpected end of JSON input
2026/10/12 22:57:55 adopt: staged-version sweep skipped: the build state of held a.example.com is not read yet
2026/10/12 22:57:55 adopt: staged-version sweep: removed 1 of 1 staged versions (6 bytes), 0 pinned (0 live), 0 within the 6h0m0s grace: [v1]
2026/10/12 22:57:55 adopt: staged-version sweep: removed 1 of 1 staged versions (6 bytes), 0 pinned (0 live), 0 within the 6h0m0s grace: [v1]
2026/10/12 22:57:55 adopt: staged-version sweep: removed 1 of 1 staged versions (6 bytes), 0 pinned (0 live), 0 within the 6h0m0s grace: [v1]
2026/10/12 22:57:55 adopt: staged-version sweep: removed 1 of 1 staged versions (6 bytes), 0 pinned (0 live), 0 within the 6h0m0s grace: [v1]
FAIL
FAIL github.com/carlosframework/platform/internal/host 60.807s
ok github.com/carlosframework/platform/internal/hostactors 0.490s
ok github.com/carlosframework/platform/internal/hostbox 1.320s
ok github.com/carlosframework/platform/internal/hostclaim 0.262s
ok github.com/carlosframework/platform/internal/hostevidence 0.006s
ok github.com/carlosframework/platform/internal/hostexec 0.313s
ok github.com/carlosframework/platform/internal/hostfence 0.033s
ok github.com/carlosframework/platform/internal/hostfiles 4.126s
ok github.com/carlosframework/platform/internal/hostfloor 0.011s
ok github.com/carlosframework/platform/internal/hostguard 0.248s
ok github.com/carlosframework/platform/internal/hostops 1.421s
? github.com/carlosframework/platform/internal/hostops/fencerelease [no test files]
ok github.com/carlosframework/platform/internal/hostops/internal/fencerelease 0.002s
ok github.com/carlosframework/platform/internal/hostquarantine 0.002s
ok github.com/carlosframework/platform/internal/hostresolve 0.131s
ok github.com/carlosframework/platform/internal/hostwrite 0.003s
ok github.com/carlosframework/platform/internal/infraview 0.002s
ok github.com/carlosframework/platform/internal/instancectl 1.711s
ok github.com/carlosframework/platform/internal/instanceexec 2.732s
ok github.com/carlosframework/platform/internal/instancefs 0.012s
ok github.com/carlosframework/platform/internal/instancekey 0.004s
ok github.com/carlosframework/platform/internal/instanceupdates 0.010s
ok github.com/carlosframework/platform/internal/layout 1.402s
ok github.com/carlosframework/platform/internal/lease 9.454s
ok github.com/carlosframework/platform/internal/lease/leasetest 0.006s
ok github.com/carlosframework/platform/internal/ledger 0.015s
ok github.com/carlosframework/platform/internal/mail 0.154s
ok github.com/carlosframework/platform/internal/maintenance 0.084s
ok github.com/carlosframework/platform/internal/maintenanceexec 5.602s
ok github.com/carlosframework/platform/internal/maintenanceguard 0.069s
ok github.com/carlosframework/platform/internal/metrics 0.005s
ok github.com/carlosframework/platform/internal/moves 21.230s
ok github.com/carlosframework/platform/internal/moves/archive 0.724s
ok github.com/carlosframework/platform/internal/objstore 0.171s
ok github.com/carlosframework/platform/internal/ops 0.168s
ok github.com/carlosframework/platform/internal/park 0.197s
ok github.com/carlosframework/platform/internal/privatefile 0.005s
ok github.com/carlosframework/platform/internal/provision 12.668s
ok github.com/carlosframework/platform/internal/publicrelease 0.008s
? github.com/carlosframework/platform/internal/regionname [no test files]
ok github.com/carlosframework/platform/internal/registry 3.909s
ok github.com/carlosframework/platform/internal/relay 7.245s
ok github.com/carlosframework/platform/internal/release 20.085s
ok github.com/carlosframework/platform/internal/rollout 3.338s
ok github.com/carlosframework/platform/internal/rolloutwords 0.004s
ok github.com/carlosframework/platform/internal/rootaws 0.007s
ok github.com/carlosframework/platform/internal/rootstate 0.025s
ok github.com/carlosframework/platform/internal/s3compat 0.005s
ok github.com/carlosframework/platform/internal/safefile 0.004s
ok github.com/carlosframework/platform/internal/schedule 2.281s
ok github.com/carlosframework/platform/internal/sim/hostlife 2.679s
ok github.com/carlosframework/platform/internal/sim/litestreamfacts 3.764s
ok github.com/carlosframework/platform/internal/status 0.043s
ok github.com/carlosframework/platform/internal/steering 0.006s
ok github.com/carlosframework/platform/internal/stopevidence 0.082s
ok github.com/carlosframework/platform/internal/svcrole 0.002s
ok github.com/carlosframework/platform/internal/telemetry 0.004s
ok github.com/carlosframework/platform/internal/termfence 0.004s
ok github.com/carlosframework/platform/internal/tunnelclient 0.004s
ok github.com/carlosframework/platform/internal/unixdial 0.003s
ok github.com/carlosframework/platform/internal/updatewords 0.002s
ok github.com/carlosframework/platform/internal/usage 0.004s
ok github.com/carlosframework/platform/internal/version 0.002s
ok github.com/carlosframework/platform/internal/vet 1.549s
FAIL
make: *** [Makefile:18: test] Error 1
amadan: step gate failed: exit status 2
amadan: job failed at step gate — its output is above, not at the tail of this log
diff --git a/internal/host/agent.go b/internal/host/agent.go| index 1ad05322e..e87d15ee5 100644 |
| --- a/internal/host/agent.go |
| +++ b/internal/host/agent.go |
| @@ -191,6 +191,7 @@ import ( |
| "github.com/carlosframework/platform/internal/provision" |
| "github.com/carlosframework/platform/internal/registry" |
| "github.com/carlosframework/platform/internal/release" |
| + "github.com/carlosframework/platform/internal/rootstate" |
| "github.com/carlosframework/platform/internal/status" |
| "github.com/carlosframework/platform/internal/version" |
| ) |
| @@ -1082,7 +1083,7 @@ func adoptLoop(reg *registry.Registry, svc *activator.Service, sd *scheduleDeps, |
| return |
| } |
| optDir := release.OptDir() |
| - restart := makeRestartRoute(reg, systemctlTryRestart, checkedUnitRestart(systemdUnits{}, optDir, os.Getenv, nil), adoptBounceFn(svc)) |
| + restart := makeRestartRoute(reg, systemctlTryRestart, agentCheckedUnitRestart(systemdUnits{}, optDir, os.Getenv, nil), adoptBounceFn(svc)) |
| log.Printf("agent: adopt loop armed (hourly + jitter)") |
| // First pass shortly after boot (Eleven's OnBootSec=5min, compressed — |
| // the edge is already up by the time this fires), then hourly + jitter. |
| @@ -1134,7 +1135,7 @@ func adoptLoop(reg *registry.Registry, svc *activator.Service, sd *scheduleDeps, |
| release.BakeDir(dataDir()), release.OnStageBlocked(stageRequester(dataDir())), |
| release.RestartHibernating(restart), |
| release.WithStartUnit(makeStartRoute(reg, systemctlStartUnit)), |
| - release.WithRollbackRestart(rollbackUnitRestart(reg, systemdUnits{}, optDir, os.Getenv)), |
| + release.WithRollbackRestart(agentRollbackUnitRestart(reg, systemdUnits{}, optDir, os.Getenv)), |
| release.WithProofGate(agentProofGate()), release.WithHostWrites(writes), |
| release.WithDigestRatchet(pointerRatchet(false, optDir, dataDir(), os.Getenv)), |
| updatesPassOption(dataDir(), false)); err != nil { |
| @@ -2367,13 +2368,13 @@ func fastConvergeLoop(reg *registry.Registry, svc *activator.Service, sd *schedu |
| // sweep instead of the sweep re-cycling the service every tick. The |
| // remaining fields are filled in right below. |
| d := &fastDeps{reg: reg, store: store, pub: pub, optDir: optDir, serialNotify: serialNotify} |
| - restart := d.observeRestart(makeRestartRoute(reg, systemctlTryRestart, checkedUnitRestart(systemdUnits{}, optDir, os.Getenv, nil), adoptBounceFn(svc))) |
| + restart := d.observeRestart(makeRestartRoute(reg, systemctlTryRestart, agentCheckedUnitRestart(systemdUnits{}, optDir, os.Getenv, nil), adoptBounceFn(svc))) |
| // Built once, used by both pass shapes below — see fastPassOpts for the |
| // DO-NOT list and why the stage requester is per-loop, not per-tick. |
| opts := append(fastPassOpts(dd, stageRequester(dd), restart, makeStartRoute(reg, systemctlStartUnit)), release.WithHostWrites(writes)) |
| // The rollback's restart (D1): checked like the forward one, and a replace |
| // rather than a start, so a rejected build left up is cycled out. |
| - opts = append(opts, release.WithRollbackRestart(rollbackUnitRestart(reg, systemdUnits{}, optDir, os.Getenv))) |
| + opts = append(opts, release.WithRollbackRestart(agentRollbackUnitRestart(reg, systemdUnits{}, optDir, os.Getenv))) |
| // D3: the process's one proof gate rides every pass this loop runs, and |
| // fastMembers reads its waiting list. The mint asker has its own |
| // requester, so its 2-minute asks never spend the stage budget and its |
| @@ -3799,7 +3800,7 @@ func unitOnly(f func(unit string) error) func(rt registry.Route) error { |
| // during it, a failed settle or an unchecked fallback). The agent loops pass |
| // nil: the sandboxed agent cannot hash another unit's executable, and spec |
| // §4.5 has root record the healthy history. |
| -func checkedUnitRestart(sys unitSystem, optDir string, getenv func(string) string, settled func(rt registry.Route)) func(rt registry.Route) error { |
| +func agentCheckedUnitRestart(sys unitSystem, optDir string, getenv func(string) string, settled func(rt registry.Route) error) func(rt registry.Route) error { |
| return func(rt registry.Route) error { |
| before, err := sys.Show(rt.Unit) |
| if err != nil { |
| @@ -3838,7 +3839,7 @@ func checkedUnitRestart(sys unitSystem, optDir string, getenv func(string) strin |
| // D3 (proofobserve.go): a build that passed the settle check is a |
| // build this box has run well. |
| if settled != nil { |
| - settled(rt) |
| + err = settled(rt) |
| } |
| } |
| return err |
| @@ -3849,7 +3850,7 @@ func checkedUnitRestart(sys unitSystem, optDir string, getenv func(string) strin |
| // the live link already names the restored build; replace whatever is |
| // running (a deadline failure can leave the rejected build up) and check |
| // the restored build comes up too. |
| -func rollbackUnitRestart(reg *registry.Registry, sys unitSystem, optDir string, getenv func(string) string) func(host string) error { |
| +func agentRollbackUnitRestart(reg *registry.Registry, sys unitSystem, optDir string, getenv func(string) string) func(host string) error { |
| return func(host string) error { |
| rt, err := reg.Get(host) |
| if err != nil || rt == nil || rt.Unit == "" { |
| @@ -3925,10 +3926,6 @@ func systemctlStartUnit(unit string) error { |
| // settled is checkedUnitRestart's hook for a named unit whose settle check |
| // passed (D3: recordSettled, which adds the build to the healthy history); |
| // nil records nothing. |
| -func rootRestartRoute(reg *registry.Registry, optDir string, settled func(rt registry.Route)) func(host string) error { |
| - return makeRestartRoute(reg, systemctlTryRestart, checkedUnitRestart(systemdUnits{}, optDir, os.Getenv, settled), nil) |
| -} |
| - |
| // bounceFn adapts an activator to makeRestartRoute's bounce parameter, |
| // nil-safe: a box with no activator armed gets a nil callback, which |
| // makeRestartRoute reads as "nothing running to cycle" rather than falling |
| @@ -4161,6 +4158,16 @@ func runAdopt(args []string, buildClaims ClaimFactory) { |
| // runAdoptWithEnsureSystemUnits keeps the root-only unit effect injectable; |
| // the rest of adopt startup, including its fatal registry diagnostics, is real. |
| func runAdoptWithEnsureSystemUnits(args []string, buildClaims ClaimFactory, ensureUnits func()) { |
| + err := withClaimRootRelease(context.Background(), hostwrite.Enabled(os.Getenv), func() (rootstate.Store, error) { return rootstate.Open(rootReleasePrivate, true, false) }, func() error { |
| + runAdoptUnderRootRelease(args, buildClaims, ensureUnits) |
| + return nil |
| + }) |
| + if err != nil { |
| + log.Fatalf("adopt: %v", err) |
| + } |
| +} |
| + |
| +func runAdoptUnderRootRelease(args []string, buildClaims ClaimFactory, ensureUnits func()) { |
| reapStagingWrites(release.OptDir(), dataDir(), "/etc", log.Printf) |
| if os.Geteuid() == 0 { |
| if err := repairOffcloudCredentials("/etc/carlos/credentials"); err != nil { |
| @@ -4296,8 +4303,8 @@ func runAdoptWithEnsureSystemUnits(args []string, buildClaims ClaimFactory, ensu |
| // settled: D1's settle passing records the build in the healthy history |
| // (D3, proofobserve.go). Root is the process that can hash the unit's |
| // executable. |
| - restart := rootRestartRoute(reg, release.OptDir(), |
| - recordSettled(systemdUnits{}, procExePath, release.OptDir(), release.RootHealthyDir, time.Now)) |
| + restartChecks := productionRootRestartChecks(systemdUnits{}, release.OptDir()) |
| + restart := rootRestartRouteWith(reg, systemdUnits{}, release.OptDir(), os.Getenv, recordSettled(systemdUnits{}, procExePath, release.OptDir(), release.RootHealthyDir, time.Now), restartChecks) |
| // The restart-request half, and the ONE place it dispatches from — |
| // including for exec-backed routes, whose children belong to the agent: |
| // rootRestartRoute's try-restart arm reaches a carlos-app@ unit, and an |
| @@ -4421,8 +4428,8 @@ func runAdoptWithEnsureSystemUnits(args []string, buildClaims ClaimFactory, ensu |
| // no-ops on the stopped unit, exits 0, and stamps a build that has never |
| // run). -stage-only keeps it for the same reason it keeps the converge: |
| // a relayed stage is exactly the request that performs such a restart. |
| - passOpts := []release.PassOption{release.WithCapabilities(caps.Ops{}), release.BakeDir(dataDir()), |
| - release.WithStartUnit(makeStartRoute(reg, systemctlStartUnit)), release.WithHostWrites(claims.Writes), |
| + passOpts := []release.PassOption{release.WithCapabilities(caps.Ops{}), release.BakeDir(dataDir()), release.WithRestartFailurePropagation(), release.WithRestartAttemptSettled(restartChecks.finish), |
| + release.WithStartUnit(rollbackUnitRestart(reg, systemdUnits{}, release.OptDir(), os.Getenv)), release.WithHostWrites(claims.Writes), |
| release.WithDigestRatchet(pointerRatchet(true, release.OptDir(), dataDir(), os.Getenv)), |
| release.WithRollbackRestart(rollbackUnitRestart(reg, systemdUnits{}, release.OptDir(), os.Getenv)), |
| // D3's gate: a fresh gate per root pass, so the policy file is read |
diff --git a/internal/host/agent_test.go b/internal/host/agent_test.go| index 879e75278..bd97d0570 100644 |
| --- a/internal/host/agent_test.go |
| +++ b/internal/host/agent_test.go |
| @@ -696,38 +696,13 @@ func TestMakeRestartRouteUnitNameBeatsBacking(t *testing.T) { |
| // and cannot write /opt/carlos, so its stage is blocked and relayed here. |
| // Everything else about it is unchanged — no activator, so every route it |
| // used to try-restart by host still is. |
| -func TestRootRestartRouteCarriesTheNamedUnitArm(t *testing.T) { |
| +func TestRootRestartRouteRefusesMissingDeploymentBinding(t *testing.T) { |
| reg := restartTestRegistry(t) |
| - if err := reg.Upsert(registry.Route{ |
| - Host: "console.carloku.com", Kind: "instance", App: "console", |
| - Addr: "127.0.0.1:8090", Status: "ready", Unit: "console.service", |
| - }); err != nil { |
| - t.Fatalf("upsert: %v", err) |
| - } |
| - // A fake systemctl on PATH records the real argv this callback builds — |
| - // the same recording trick TestBootstrapStagingRunsNoSystemctl uses — so |
| - // this covers the production path (systemctlTryRestartUnit's exec.Command) |
| - // and not just the constructor. |
| - bin := t.TempDir() |
| - callLog := filepath.Join(t.TempDir(), "systemctl-calls") |
| - if err := os.WriteFile(filepath.Join(bin, "systemctl"), |
| - []byte("#!/bin/sh\necho \"$@\" >> "+callLog+"\n"), 0o755); err != nil { |
| + if err := reg.Upsert(registry.Route{Host: "console.carloku.com", Kind: "instance", App: "console", Addr: "127.0.0.1:8090", Status: "ready", Unit: "console.service"}); err != nil { |
| t.Fatal(err) |
| } |
| - t.Setenv("PATH", bin+string(os.PathListSeparator)+os.Getenv("PATH")) |
| - |
| - if err := rootRestartRoute(reg, t.TempDir(), nil)("console.carloku.com"); err != nil { |
| - t.Fatalf("root restart: %v", err) |
| - } |
| - got, err := os.ReadFile(callLog) |
| - if err != nil { |
| - t.Fatalf("the root pass ran no systemctl at all: %v", err) |
| - } |
| - // The checked arm reads the unit first (show), finds no binding it can |
| - // verify from the fake's empty answer, and falls back to the plain |
| - // try-restart: the argv this test pins is the last call. |
| - if want := "show console.service -p ActiveState -p SubState -p InvocationID -p MainPID -p NRestarts -p ExecStart -p ActiveEnterTimestamp\ntry-restart console.service\n"; string(got) != want { |
| - t.Fatalf("systemctl calls = %q, want %q", got, want) |
| + if err := rootRestartRoute(reg, t.TempDir(), nil)("console.carloku.com"); err == nil { |
| + t.Fatal("restarted without deployment binding") |
| } |
| } |
| @@ -837,7 +812,7 @@ func TestAdoptPassRefusesToStampAVersionAWrongUnitDidNotStart(t *testing.T) { |
| log.SetOutput(&journal) |
| t.Cleanup(func() { log.SetOutput(os.Stderr) }) |
| - restart := rootRestartRoute(reg, optDir, nil) |
| + restart := makeRestartRoute(reg, systemctlTryRestart, agentCheckedUnitRestart(systemdUnits{}, optDir, os.Getenv, nil), nil) |
| if err := release.Pass(ctx, reg, s, pub, optDir, restart); err != nil { |
| t.Fatalf("Pass: %v", err) |
| } |
| @@ -964,7 +939,7 @@ func TestAdoptPassStartsTheNamedUnitAgainAfterAFailedRestart(t *testing.T) { |
| log.SetOutput(&journal) |
| t.Cleanup(func() { log.SetOutput(os.Stderr) }) |
| - if err := release.Pass(ctx, reg, s, pub, optDir, rootRestartRoute(reg, optDir, nil), |
| + if err := release.Pass(ctx, reg, s, pub, optDir, makeRestartRoute(reg, systemctlTryRestart, agentCheckedUnitRestart(systemdUnits{}, optDir, os.Getenv, nil), nil), |
| release.WithStartUnit(makeStartRoute(reg, systemctlStartUnit))); err != nil { |
| t.Fatalf("Pass: %v", err) |
| } |
| @@ -975,7 +950,7 @@ func TestAdoptPassStartsTheNamedUnitAgainAfterAFailedRestart(t *testing.T) { |
| } |
| // The checked arm's show comes first (its empty fake answer verifies no |
| // binding, so the restart falls back to the plain try-restart). |
| - if want := "show console.service -p ActiveState -p SubState -p InvocationID -p MainPID -p NRestarts -p ExecStart -p ActiveEnterTimestamp\ntry-restart console.service\nstart console.service\n"; string(got) != want { |
| + if want := "show console.service -p Id -p FragmentPath -p DropInPaths -p ActiveState -p SubState -p InvocationID -p MainPID -p NRestarts -p ExecStart -p ActiveEnterTimestamp\ntry-restart console.service\nstart console.service\n"; string(got) != want { |
| t.Fatalf("systemctl calls = %q, want %q — the rolled-back unit must be STARTED (try-restart would no-op on it)", got, want) |
| } |
| if v, err := release.LiveVersion(optDir, "console.test"); err != nil || v != "old" { |
diff --git a/internal/host/claimconstruction.go b/internal/host/claimconstruction.go| index 3a7330acc..954aa88ed 100644 |
| --- a/internal/host/claimconstruction.go |
| +++ b/internal/host/claimconstruction.go |
| @@ -30,6 +30,7 @@ import ( |
| "github.com/carlosframework/platform/internal/park" |
| "github.com/carlosframework/platform/internal/registry" |
| "github.com/carlosframework/platform/internal/release" |
| + "github.com/carlosframework/platform/internal/rootstate" |
| ) |
| type ClaimRuntimeResources struct { |
| @@ -252,6 +253,21 @@ func OpenProductionClaimAssembly(ctx context.Context, store release.Store, r Cla |
| } |
| roots := map[string]string{"data": r.DataDir, "opt": r.OptDir, "units": r.UnitsDir} |
| guard := hostguard.Guard{DataDir: r.DataDir, Claims: hostguard.StoreReader{Store: store}} |
| + var authority rootstate.Store |
| + if r.Mode == "agent" { |
| + guard.FenceView = rootFenceViewClient(rootJournalClient(controlsock.Path)) |
| + } else { |
| + authority, err = rootstate.Open("/var/lib/carlos-claims/authority/fences", true, false) |
| + if err != nil { |
| + return nil, err |
| + } |
| + defer func() { |
| + if resultErr != nil { |
| + _ = authority.Close() |
| + } |
| + }() |
| + guard.Root = hostguard.NewRootAuthority(authority) |
| + } |
| producers := &ClaimProducerSet{CloseWriters: launches.CloseWriters} |
| inputs, admissionEvidence, publicationEvidence := owners.Inputs, owners.AdmissionEvidence, owners.PublicationEvidence |
| var serving *claimServingOwners |
| @@ -360,7 +376,11 @@ func OpenProductionClaimAssembly(ctx context.Context, store release.Store, r Cla |
| return err |
| } |
| closed = true |
| - return errors.Join(inventory.Close(), launches.Close()) |
| + var rootErr error |
| + if authority != nil { |
| + rootErr = authority.Close() |
| + } |
| + return errors.Join(rootErr, inventory.Close(), launches.Close()) |
| } |
| return assembly, nil |
| } |
diff --git a/internal/host/claimjournal_transport.go b/internal/host/claimjournal_transport.go| index f48ade995..1e1225a22 100644 |
| --- a/internal/host/claimjournal_transport.go |
| +++ b/internal/host/claimjournal_transport.go |
| @@ -87,6 +87,10 @@ func (h ClaimRootHTTP) serveJournal(w http.ResponseWriter, r *http.Request, peer |
| w.WriteHeader(http.StatusBadRequest) |
| return |
| } |
| + if q.Op == "fence-view" { |
| + h.serveFenceView(w, r, q) |
| + return |
| + } |
| owner := h.Reconcile.Box + "/" + h.Reconcile.Enrollment |
| if out, found, err := h.Journals.Replay(r.Context(), owner, q); err != nil { |
| // Only a proven mismatch (other owner or bytes under a stored ID) is a |
diff --git a/internal/host/claimproduction.go b/internal/host/claimproduction.go| index a921b4e07..911396989 100644 |
| --- a/internal/host/claimproduction.go |
| +++ b/internal/host/claimproduction.go |
| @@ -481,7 +481,7 @@ func (t claimAdminTokens) With(ctx context.Context, host string, effect func() e |
| } |
| return t.Policy(ctx, c, sel, hostwrite.Identity{}) |
| } |
| - captured, err := hostguard.Capture(t.Guard.DataDir, host) |
| + captured, err := t.Guard.Capture(host) |
| if errors.Is(err, hostguard.ErrNoRecord) { |
| return t.Guard.WithNoInstallation(ctx, host, sel.Captured.Incarnation, check, effect) |
| } |
diff --git a/internal/host/movesfinal.go b/internal/host/movesfinal.go| index 151ff86f9..9e4ba99dd 100644 |
| --- a/internal/host/movesfinal.go |
| +++ b/internal/host/movesfinal.go |
| @@ -582,7 +582,7 @@ func (f *sourceFinal) parkLocal(ctx context.Context, holder hostclaim.Holder, c |
| return err |
| } |
| } |
| - captured, err := hostguard.Capture(guard.DataDir, host) |
| + captured, err := guard.Capture(host) |
| switch { |
| case errors.Is(err, hostguard.ErrNoRecord): |
| return guard.WithNoInstallation(ctx, host, f.g.Incarnation, check, work) |
diff --git a/internal/host/plan14_canonical_restart_test.go b/internal/host/plan14_canonical_restart_test.go| new file mode 100644 |
| index 000000000..f1135887c |
| --- /dev/null |
| +++ b/internal/host/plan14_canonical_restart_test.go |
| @@ -0,0 +1,276 @@ |
| +package host |
| + |
| +import ( |
| + "context" |
| + "crypto/ed25519" |
| + "errors" |
| + "os" |
| + "path/filepath" |
| + "strconv" |
| + "sync" |
| + "testing" |
| + "time" |
| + |
| + "github.com/carlosframework/platform/internal/provision" |
| + "github.com/carlosframework/platform/internal/registry" |
| + "github.com/carlosframework/platform/internal/release" |
| + "github.com/carlosframework/platform/internal/rootstate" |
| +) |
| + |
| +type plan14RestartUnits struct { |
| + props unitProps |
| + exe string |
| + opt string |
| + effects []string |
| + crashNew bool |
| + badRollback bool |
| + failNew bool |
| + afterRestart func() |
| +} |
| + |
| +func (u *plan14RestartUnits) StartTime(pid int) (string, error) { return strconv.Itoa(pid), nil } |
| + |
| +func (u *plan14RestartUnits) Show(string) (unitProps, error) { return u.props, nil } |
| +func (u *plan14RestartUnits) Exe(int) (string, error) { return u.exe, nil } |
| +func (u *plan14RestartUnits) JournalTail(string, int) string { return "" } |
| +func (u *plan14RestartUnits) ResetFailed(unit string) error { |
| + u.effects = append(u.effects, "reset "+unit) |
| + return nil |
| +} |
| +func (u *plan14RestartUnits) TryRestart(unit string) error { return u.Restart(unit) } |
| +func (u *plan14RestartUnits) Restart(unit string) error { |
| + u.effects = append(u.effects, "restart "+unit) |
| + if u.afterRestart != nil { |
| + u.afterRestart() |
| + } |
| + u.props.InvocationID += "x" |
| + u.props.MainPID++ |
| + u.props.ActiveState, u.props.SubState = "active", "running" |
| + var err error |
| + u.exe, err = filepath.EvalSymlinks(release.LiveLink(u.opt, observedHost)) |
| + if u.crashNew && u.exe == release.VersionPath(u.opt, "bab", "v2", "console") { |
| + u.props.ActiveState, u.props.SubState, u.props.MainPID = "failed", "failed", 0 |
| + } |
| + if u.badRollback && u.exe == release.VersionPath(u.opt, "bab", "v1", "console") { |
| + u.exe = release.VersionPath(u.opt, "bab", "v2", "console") |
| + } |
| + if u.failNew && u.exe == release.VersionPath(u.opt, "bab", "v2", "console") { |
| + return os.ErrPermission |
| + } |
| + return err |
| +} |
| + |
| +func TestPlan14CrashedNamedUnitRollsBack(t *testing.T) { |
| + e, _, units, c, pub := plan14AdoptionFixture(t) |
| + units.crashNew = true |
| + forward := rootRestartRouteWith(e.root.reg, units, units.opt, func(string) string { return "5s" }, e.root.obs.recordSettled(e.bake), c) |
| + var rollbackErr error |
| + rollback := rollbackUnitRestart(e.root.reg, units, units.opt, func(string) string { return "5s" }, c) |
| + err := release.Pass(context.Background(), e.root.reg, e.store, pub, units.opt, forward, |
| + release.BakeDir(e.bake), release.WithRestartFailurePropagation(), |
| + release.WithRollbackRestart(func(host string) error { rollbackErr = rollback(host); return rollbackErr })) |
| + if rollbackErr != nil || err == nil { |
| + t.Fatalf("crash recovery: %v / %v", err, rollbackErr) |
| + } |
| + if len(units.effects) != 3 || units.props.MainPID <= 0 || units.exe != release.VersionPath(units.opt, "bab", "v1", "console") { |
| + t.Fatalf("rollback was not verified: %+v", units) |
| + } |
| + if version, err := release.LiveVersion(units.opt, observedHost); err != nil || version != "v1" { |
| + t.Fatalf("restored link = %q, %v", version, err) |
| + } |
| + if got := release.ReadHealthy(e.bake, "bab", "console"); len(got) != 1 { |
| + t.Fatalf("crash recorded health: %+v", got) |
| + } |
| +} |
| + |
| +func plan14AdoptionFixture(t *testing.T) (*proofRootEnv, registry.Route, *plan14RestartUnits, rootRestartChecks, ed25519.PublicKey) { |
| + t.Helper() |
| + e, rt, rec, source := plan14ProofFixture(t) |
| + if got := e.root.observeAndRecord(rt, 15*time.Minute); got.err != nil { |
| + t.Fatal(got.err) |
| + } |
| + source.healthy = func(acct, app, version, digest string) *release.HealthyBuild { |
| + for _, b := range release.ReadHealthy(e.bake, acct, app) { |
| + if b.Version == version && b.SHA256 == digest { |
| + return &b |
| + } |
| + } |
| + return nil |
| + } |
| + u := &plan14RestartUnits{props: source.sys.(*fakeUnits).shows[0], exe: release.VersionPath(source.optDir, "bab", "v1", "console"), opt: source.optDir} |
| + source.sys, e.root.obs.sys = u, u |
| + e.root.obs.procExe = func(int) string { return u.exe } |
| + rec.Pin = "v2" |
| + plan14PutRecord(t, e.store, rec) |
| + ctx := context.Background() |
| + pub, priv, err := ed25519.GenerateKey(nil) |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + for ver, body := range map[string]string{"v1": "x", "v2": "new build"} { |
| + m := release.Manifest{App: "console", Version: ver, Kind: "binary", Artifacts: map[string]release.Artifact{release.AdoptPlatform(): {}}} |
| + if err := release.Ship(ctx, e.store, "bab", m, map[string][]byte{release.AdoptPlatform(): []byte(body)}); err != nil { |
| + t.Fatal(err) |
| + } |
| + } |
| + if err := release.Promote(ctx, e.store, priv, "bab", "console", rt.Channel, "v2"); err != nil { |
| + t.Fatal(err) |
| + } |
| + now, sleep := settleClock() |
| + mem := &rollbackMemory{files: map[string][]byte{}} |
| + c := rootRestartChecks{started: &sync.Map{}, attempts: func() (rootstate.Store, error) { return mem, nil }, rollback: source.rollback, resolve: source.resolve, host: source.forHost, ready: e.root.obs.ready, now: now, sleep: sleep} |
| + return e, rt, u, c, pub |
| +} |
| + |
| +func TestPlan14AdoptionCanonicalReadiness(t *testing.T) { |
| + mutations := map[string]func(*registry.Route){ |
| + "unchanged": func(*registry.Route) {}, |
| + "socket": func(r *registry.Route) { r.Socket = "/run/forged.sock" }, |
| + "addr": func(r *registry.Route) { r.Addr = "" }, |
| + "addrs": func(r *registry.Route) { r.Addrs = "127.0.0.1:1" }, |
| + "health": func(r *registry.Route) { r.Health = "" }, |
| + "unit": func(r *registry.Route) { r.Unit = "" }, |
| + "backing": func(r *registry.Route) { r.Backing = "exec" }, |
| + "empty upstream": func(r *registry.Route) { r.Socket, r.Addr, r.Addrs, r.Health = "", "", "", "" }, |
| + } |
| + for name, mutate := range mutations { |
| + t.Run(name, func(t *testing.T) { |
| + e, rt, units, c, pub := plan14AdoptionFixture(t) |
| + mutate(&rt) |
| + if err := e.root.reg.Upsert(rt); err != nil { |
| + t.Fatal(err) |
| + } |
| + c.ready = func(rt registry.Route) func() error { |
| + if rt.Addr == "" && rt.Socket == "" { |
| + return nil |
| + } |
| + return func() error { return os.ErrPermission } |
| + } |
| + e.root.obs.ready = c.ready |
| + restart := rootRestartRouteWith(e.root.reg, units, units.opt, func(string) string { return "5s" }, e.root.obs.recordSettled(e.bake), c) |
| + err := release.Pass(context.Background(), e.root.reg, e.store, pub, units.opt, restart, |
| + release.BakeDir(e.bake), release.WithRestartFailurePropagation(), |
| + release.WithRollbackRestart(rollbackUnitRestart(e.root.reg, units, units.opt, func(string) string { return "5s" }, c))) |
| + if err == nil { |
| + t.Fatal("adoption swallowed the canonical readiness refusal") |
| + } |
| + got, getErr := e.root.reg.Get(rt.Host) |
| + if getErr != nil || got.Version == "v2" { |
| + t.Fatalf("stamped adoption: %+v, %v", got, getErr) |
| + } |
| + if history := release.ReadHealthy(e.bake, "bab", "console"); len(history) != 1 || history[0].Version != "v1" { |
| + t.Fatalf("new health: %+v", history) |
| + } |
| + }) |
| + } |
| + t.Run("settled refusal propagates", func(t *testing.T) { |
| + e, _, units, c, pub := plan14AdoptionFixture(t) |
| + sentinel := errors.New("settled witness") |
| + restart := rootRestartRouteWith(e.root.reg, units, units.opt, func(string) string { return "5s" }, func(registry.Route) error { return sentinel }, c) |
| + err := release.Pass(context.Background(), e.root.reg, e.store, pub, units.opt, restart, release.WithRestartFailurePropagation()) |
| + if !errors.Is(err, sentinel) { |
| + t.Fatalf("lost settled error: %v", err) |
| + } |
| + }) |
| + t.Run("positive adoption", func(t *testing.T) { |
| + e, _, units, c, pub := plan14AdoptionFixture(t) |
| + restart := rootRestartRouteWith(e.root.reg, units, units.opt, func(string) string { return "5s" }, e.root.obs.recordSettled(e.bake), c) |
| + if err := release.Pass(context.Background(), e.root.reg, e.store, pub, units.opt, restart, release.WithRestartFailurePropagation()); err != nil { |
| + t.Fatal(err) |
| + } |
| + rt, err := e.root.reg.Get(observedHost) |
| + if err != nil || rt.Version != "v2" || len(release.ReadHealthy(e.bake, "bab", "console")) != 2 { |
| + t.Fatalf("canonical adoption failed: %+v, %v", rt, err) |
| + } |
| + }) |
| +} |
| + |
| +func TestPlan14RollbackCanonicalReadiness(t *testing.T) { |
| + for _, missing := range []bool{false, true} { |
| + for _, ready := range []bool{false, true} { |
| + name := strconv.FormatBool(missing) + "/" + strconv.FormatBool(ready) |
| + t.Run(name, func(t *testing.T) { |
| + e, rt, units, c, pub := plan14AdoptionFixture(t) |
| + c.ready = func(r registry.Route) func() error { |
| + if r.Addr != "127.0.0.1:4000" || r.Health != rt.Health { |
| + t.Fatalf("noncanonical rollback readiness: %+v", r) |
| + } |
| + return func() error { |
| + if !ready || units.exe == release.VersionPath(units.opt, "bab", "v2", "console") { |
| + return os.ErrPermission |
| + } |
| + return nil |
| + } |
| + } |
| + forward := rootRestartRouteWith(e.root.reg, units, units.opt, func(string) string { return "5s" }, e.root.obs.recordSettled(e.bake), c) |
| + restart := func(host string) error { |
| + err := forward(host) |
| + if missing { |
| + if e := e.root.reg.Delete(host); e != nil { |
| + t.Fatal(e) |
| + } |
| + } else { |
| + forged := rt |
| + forged.Unit, forged.Backing, forged.Socket, forged.Addr, forged.Addrs, forged.Health = "", "", "", "", "", "" |
| + if e := e.root.reg.Upsert(forged); e != nil { |
| + t.Fatal(e) |
| + } |
| + } |
| + return err |
| + } |
| + var rollbackErr error |
| + rollback := rollbackUnitRestart(e.root.reg, units, units.opt, func(string) string { return "5s" }, c) |
| + err := release.Pass(context.Background(), e.root.reg, e.store, pub, units.opt, restart, |
| + release.BakeDir(e.bake), release.WithRestartFailurePropagation(), |
| + release.WithRollbackRestart(func(host string) error { rollbackErr = rollback(host); return rollbackErr })) |
| + if err == nil { |
| + t.Fatal("forward readiness failure disappeared") |
| + } |
| + if ready != (rollbackErr == nil) { |
| + t.Fatalf("rollback error = %v, readiness = %v", rollbackErr, ready) |
| + } |
| + if len(units.effects) != 3 { |
| + t.Fatalf("rollback did not use the authoritative unit: %v", units.effects) |
| + } |
| + if !ready && !errors.Is(err, rollbackErr) { |
| + t.Fatalf("rollback failure not reported: %v / %v", err, rollbackErr) |
| + } |
| + if history := release.ReadHealthy(e.bake, "bab", "console"); len(history) != 1 { |
| + t.Fatalf("rollback wrote health: %+v", history) |
| + } |
| + }) |
| + } |
| + } |
| +} |
| + |
| +func TestPlan14BindingChangesDuringRestart(t *testing.T) { |
| + e, _, u, c, pub := plan14AdoptionFixture(t) |
| + u.afterRestart = func() { |
| + rec, _, err := provision.FindRecordByHost(context.Background(), e.store, observedHost) |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + rec.RecordSeq++ |
| + plan14PutRecord(t, e.store, *rec) |
| + } |
| + forward := rootRestartRouteWith(e.root.reg, u, u.opt, func(string) string { return "5s" }, e.root.obs.recordSettled(e.bake), c) |
| + err := release.Pass(context.Background(), e.root.reg, e.store, pub, u.opt, forward, release.BakeDir(e.bake), release.WithRestartFailurePropagation(), release.WithRollbackRestart(rollbackUnitRestart(e.root.reg, u, u.opt, func(string) string { return "5s" }, c))) |
| + if err == nil || len(u.effects) != 1 || len(release.ReadHealthy(e.bake, "bab", "console")) != 1 { |
| + t.Fatalf("changed binding settled: %v, %v", err, u.effects) |
| + } |
| +} |
| + |
| +func TestPlan14RollbackInvocationChangesInProbe(t *testing.T) { |
| + e, _, u, c, pub := plan14AdoptionFixture(t) |
| + u.crashNew = true |
| + c.ready = func(registry.Route) func() error { return func() error { u.props.InvocationID += "raced"; return nil } } |
| + forward := rootRestartRouteWith(e.root.reg, u, u.opt, func(string) string { return "5s" }, nil, c) |
| + var recovered error |
| + err := release.Pass(context.Background(), e.root.reg, e.store, pub, u.opt, forward, release.BakeDir(e.bake), release.WithRestartFailurePropagation(), release.WithRollbackRestart(func(host string) error { |
| + recovered = rollbackUnitRestart(e.root.reg, u, u.opt, func(string) string { return "5s" }, c)(host) |
| + return recovered |
| + })) |
| + if err == nil || recovered == nil { |
| + t.Fatal("changed invocation accepted as recovery") |
| + } |
| +} |
diff --git a/internal/host/plan14_release_lock_test.go b/internal/host/plan14_release_lock_test.go| new file mode 100644 |
| index 000000000..65ed7a050 |
| --- /dev/null |
| +++ b/internal/host/plan14_release_lock_test.go |
| @@ -0,0 +1,58 @@ |
| +package host |
| + |
| +import ( |
| + "context" |
| + "errors" |
| + "os" |
| + "sync" |
| + "testing" |
| + |
| + "github.com/carlosframework/platform/internal/rootstate" |
| +) |
| + |
| +type plan14ReleaseStore struct { |
| + rootstate.Store |
| + mu sync.Mutex |
| + held bool |
| + fail error |
| +} |
| + |
| +func (s *plan14ReleaseStore) Lock(ctx context.Context, name string) (func(), error) { |
| + if name != "release.lock" { |
| + return nil, os.ErrPermission |
| + } |
| + if s.fail != nil { |
| + return nil, s.fail |
| + } |
| + s.mu.Lock() |
| + s.held = true |
| + return func() { s.held = false; s.mu.Unlock() }, nil |
| +} |
| +func (s *plan14ReleaseStore) Close() error { return nil } |
| + |
| +func TestPlan14AdoptReleaseLock(t *testing.T) { |
| + s := &plan14ReleaseStore{} |
| + called := 0 |
| + open := func() (rootstate.Store, error) { called++; return s, nil } |
| + effects := 0 |
| + run := func() error { |
| + if !s.held { |
| + t.Fatal("effect outside release lock") |
| + } |
| + effects++ |
| + return nil |
| + } |
| + if err := withClaimRootRelease(context.Background(), true, open, run); err != nil { |
| + t.Fatal(err) |
| + } |
| + if effects != 1 || called != 1 { |
| + t.Fatal(effects, called) |
| + } |
| + s.fail = os.ErrPermission |
| + if err := withClaimRootRelease(context.Background(), true, open, run); !errors.Is(err, os.ErrPermission) || effects != 1 { |
| + t.Fatal("lock failure admitted effects", err, effects) |
| + } |
| + if err := withClaimRootRelease(context.Background(), false, func() (rootstate.Store, error) { t.Fatal("flag-off opened root state"); return nil, nil }, func() error { effects++; return nil }); err != nil || effects != 2 { |
| + t.Fatal(err, effects) |
| + } |
| +} |
diff --git a/internal/host/proofbinding.go b/internal/host/proofbinding.go| new file mode 100644 |
| index 000000000..026dfd869 |
| --- /dev/null |
| +++ b/internal/host/proofbinding.go |
| @@ -0,0 +1,259 @@ |
| +package host |
| + |
| +import ( |
| + "context" |
| + "crypto/sha256" |
| + "encoding/hex" |
| + "encoding/json" |
| + "io" |
| + "os" |
| + "path/filepath" |
| + "strings" |
| + "time" |
| + |
| + "github.com/carlosframework/platform/internal/hostclaim" |
| + "github.com/carlosframework/platform/internal/hostwrite" |
| + "github.com/carlosframework/platform/internal/provision" |
| + "github.com/carlosframework/platform/internal/registry" |
| + "github.com/carlosframework/platform/internal/release" |
| + "github.com/carlosframework/platform/internal/rootstate" |
| +) |
| + |
| +type proofBinding struct { |
| + Route registry.Route |
| + Source string |
| + Network bool |
| + Version string |
| + Digest string |
| + Historical bool |
| + History *release.HealthyBuild |
| + Invocation string |
| + PID int |
| + Start string |
| + RecordKey string |
| + RecordETag string |
| + CandidateUnit string |
| + Files []proofUnitFile |
| +} |
| + |
| +type proofUnitFile struct { |
| + Path string |
| + Identity rootstate.FileIdentity |
| +} |
| + |
| +type proofBindingResolver func(context.Context, registry.Route) (proofBinding, error) |
| + |
| +type proofBindingSource struct { |
| + store release.Store |
| + claim func(context.Context, string) (hostclaim.Observed, error) |
| + box func(context.Context) (string, error) |
| + sys unitSystem |
| + optDir string |
| + openFile func(string) (*os.File, error) |
| + unitDirs []string |
| + healthy func(string, string, string, string) *release.HealthyBuild |
| +} |
| + |
| +func proofRouteReady(rt registry.Route, timeout time.Duration) func() error { |
| + if rt.Addrs == "" { |
| + return routeReady(rt, timeout) |
| + } |
| + var probes []func() error |
| + for _, addr := range strings.Split(rt.Addrs, ",") { |
| + probe := rt |
| + probe.Addr, probe.Addrs, probe.Socket = addr, "", "" |
| + ready := routeReady(probe, timeout) |
| + if ready == nil { |
| + return func() error { return os.ErrPermission } |
| + } |
| + probes = append(probes, ready) |
| + } |
| + return func() error { |
| + for _, probe := range probes { |
| + if err := probe(); err != nil { |
| + return err |
| + } |
| + } |
| + return nil |
| + } |
| +} |
| + |
| +func productionProofBindingSource(store release.Store, sys unitSystem, optDir string) *proofBindingSource { |
| + source := &proofBindingSource{store: store, sys: sys, optDir: optDir, openFile: rootstate.OpenTrustedFile, |
| + unitDirs: []string{"/etc/systemd/system", "/run/systemd/system", "/usr/lib/systemd/system"}, |
| + healthy: func(acct, app, version, digest string) *release.HealthyBuild { |
| + for _, b := range release.ReadHealthy(release.RootHealthyDir, acct, app) { |
| + if b.Version == version && b.SHA256 == digest { |
| + return &b |
| + } |
| + } |
| + return nil |
| + }} |
| + if hostwrite.Enabled(os.Getenv) { |
| + source.claim = func(ctx context.Context, host string) (hostclaim.Observed, error) { |
| + return hostclaim.Read(ctx, store, host) |
| + } |
| + source.box = func(ctx context.Context) (string, error) { |
| + r, err := readProductionRosterRow(ctx, store, os.Getenv) |
| + if err != nil { |
| + return "", err |
| + } |
| + return r.row.BoxID, nil |
| + } |
| + } |
| + return source |
| +} |
| + |
| +func (s *proofBindingSource) resolve(ctx context.Context, candidate registry.Route) (proofBinding, error) { |
| + b, err := s.forHost(ctx, candidate.Host) |
| + if err != nil { |
| + return proofBinding{}, err |
| + } |
| + if !sameProofCandidate(candidate, b.Route, b.CandidateUnit) { |
| + return proofBinding{}, os.ErrPermission |
| + } |
| + return b, nil |
| +} |
| + |
| +func (s *proofBindingSource) binding(ctx context.Context, host string, rollback *proofBinding) (proofBinding, error) { |
| + if s.store == nil || s.sys == nil || s.openFile == nil { |
| + return proofBinding{}, os.ErrPermission |
| + } |
| + rec, etag, err := provision.FindRecordByHost(ctx, s.store, host) |
| + if err != nil { |
| + return proofBinding{}, err |
| + } |
| + rt, err := rootRecordRoute(rec) |
| + if err != nil { |
| + return proofBinding{}, err |
| + } |
| + if rt.Unit == "" { |
| + return proofBinding{}, os.ErrPermission |
| + } |
| + p, err := s.sys.Show(rt.Unit) |
| + if err != nil { |
| + return proofBinding{}, err |
| + } |
| + if p.Name != rt.Unit || p.ExecPath != release.LiveLink(s.optDir, host) { |
| + return proofBinding{}, os.ErrPermission |
| + } |
| + definition := sha256.New() |
| + var files []proofUnitFile |
| + for _, path := range append([]string{p.FragmentPath}, p.DropInPaths...) { |
| + allowed := false |
| + for _, dir := range s.unitDirs { |
| + if strings.HasPrefix(path, dir+"/") { |
| + allowed = true |
| + } |
| + } |
| + if !allowed { |
| + return proofBinding{}, os.ErrPermission |
| + } |
| + f, err := s.openFile(path) |
| + if err != nil { |
| + return proofBinding{}, err |
| + } |
| + identity, err := rootstate.Identity(f) |
| + if err != nil { |
| + f.Close() |
| + return proofBinding{}, err |
| + } |
| + files = append(files, proofUnitFile{Path: path, Identity: identity}) |
| + encoded, _ := json.Marshal(identity) |
| + definition.Write(encoded) |
| + definition.Write([]byte(path + "\x00")) |
| + n, copyErr := io.Copy(definition, io.LimitReader(f, rootstate.MaxRecord+1)) |
| + f.Close() |
| + if copyErr != nil { |
| + return proofBinding{}, copyErr |
| + } |
| + if n > rootstate.MaxRecord { |
| + return proofBinding{}, os.ErrPermission |
| + } |
| + } |
| + claimID, err := s.claimIdentity(ctx, rec) |
| + if err != nil { |
| + return proofBinding{}, err |
| + } |
| + source := claimID + provision.RecordKey(rec.Account, rec.App, host) + "@" + etag + "@" + hex.EncodeToString(definition.Sum(nil)) |
| + var exe string |
| + if rollback != nil { |
| + if rollback.Source != source { |
| + return proofBinding{}, os.ErrPermission |
| + } |
| + exe, err = filepath.EvalSymlinks(release.LiveLink(s.optDir, host)) |
| + } else { |
| + if p.MainPID <= 0 || p.InvocationID == "" { |
| + return proofBinding{}, os.ErrPermission |
| + } |
| + exe, err = s.sys.Exe(p.MainPID) |
| + } |
| + if err != nil { |
| + return proofBinding{}, err |
| + } |
| + rel, err := filepath.Rel(filepath.Join(s.optDir, "versions"), exe) |
| + if err != nil { |
| + return proofBinding{}, err |
| + } |
| + version := strings.Split(rel, string(filepath.Separator))[0] |
| + if !release.ValidVersion(version) || exe != release.VersionPath(s.optDir, rec.Account, version, rec.App) { |
| + return proofBinding{}, os.ErrPermission |
| + } |
| + f, err := s.openFile(exe) |
| + if err != nil { |
| + return proofBinding{}, err |
| + } |
| + defer f.Close() |
| + hash := sha256.New() |
| + if _, err := io.Copy(hash, f); err != nil { |
| + return proofBinding{}, err |
| + } |
| + digest := hex.EncodeToString(hash.Sum(nil)) |
| + var history *release.HealthyBuild |
| + if s.healthy != nil { |
| + history = s.healthy(rec.Account, rec.App, version, digest) |
| + } |
| + historical := history != nil |
| + if rollback != nil && (!historical || !rollback.Historical || rollback.Version != version || rollback.Digest != digest) { |
| + return proofBinding{}, os.ErrPermission |
| + } |
| + if rec.Pin != version && !historical { |
| + if rec.Pin != "" { |
| + return proofBinding{}, os.ErrPermission |
| + } |
| + pub, err := release.ParsePublicKey(os.Getenv("CARLOS_RELEASE_PUBKEY"), "CARLOS_RELEASE_PUBKEY") |
| + if err != nil { |
| + return proofBinding{}, err |
| + } |
| + pointer, err := release.Channel(ctx, s.store, pub, rec.Account, rec.App, rec.Channel) |
| + if err != nil { |
| + return proofBinding{}, err |
| + } |
| + if pointer.Version != version { |
| + return proofBinding{}, os.ErrPermission |
| + } |
| + } |
| + start := "" |
| + if rollback == nil { |
| + start, err = s.sys.StartTime(p.MainPID) |
| + if err != nil || start == "" { |
| + return proofBinding{}, os.ErrPermission |
| + } |
| + again, err := s.sys.Show(rt.Unit) |
| + if err != nil || again.MainPID != p.MainPID || again.InvocationID != p.InvocationID || again.FragmentPath != p.FragmentPath { |
| + return proofBinding{}, os.ErrPermission |
| + } |
| + againExe, err := s.sys.Exe(again.MainPID) |
| + if err != nil || againExe != exe { |
| + return proofBinding{}, os.ErrPermission |
| + } |
| + } |
| + return proofBinding{CandidateUnit: rec.Unit, History: history, Invocation: p.InvocationID, PID: p.MainPID, Start: start, Route: rt, Source: source, Network: true, Version: version, Digest: digest, Historical: historical, RecordKey: provision.RecordKey(rec.Account, rec.App, host), RecordETag: etag, Files: files}, nil |
| +} |
| + |
| +func (s *proofBindingSource) forHost(ctx context.Context, host string) (proofBinding, error) { |
| + return s.binding(ctx, host, nil) |
| +} |
| +func (s *proofBindingSource) rollback(ctx context.Context, b proofBinding) (proofBinding, error) { |
| + return s.binding(ctx, b.Route.Host, &b) |
| +} |
diff --git a/internal/host/proofbinding_test.go b/internal/host/proofbinding_test.go| new file mode 100644 |
| index 000000000..2a104010a |
| --- /dev/null |
| +++ b/internal/host/proofbinding_test.go |
| @@ -0,0 +1,284 @@ |
| +package host |
| + |
| +import ( |
| + "context" |
| + "encoding/json" |
| + "errors" |
| + "os" |
| + "path/filepath" |
| + "testing" |
| + "time" |
| + |
| + "github.com/carlosframework/platform/internal/activator" |
| + "github.com/carlosframework/platform/internal/hostclaim" |
| + "github.com/carlosframework/platform/internal/provision" |
| + "github.com/carlosframework/platform/internal/registry" |
| + "github.com/carlosframework/platform/internal/release" |
| + "github.com/carlosframework/platform/internal/rootstate" |
| +) |
| + |
| +func plan14ProofFixture(t *testing.T) (*proofRootEnv, registry.Route, provision.Record, *proofBindingSource) { |
| + t.Helper() |
| + e := newProofRootEnv(t, provePolicy) |
| + rec := provision.Record{Schema: 1, Host: observedHost, Account: "bab", App: "console", Kind: "instance", |
| + Unit: "console.service", Addr: "127.0.0.1:4000", Channel: "canary/console", Pin: "v1", Health: &provision.Health{Path: "/ready"}} |
| + plan14PutRecord(t, e.store, rec) |
| + health, err := json.Marshal(rec.Health.WithDefaults()) |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + rt := registry.Route{Host: rec.Host, Account: rec.Account, App: rec.App, Kind: rec.Kind, Unit: rec.Unit, |
| + Addr: rec.Addr, Health: string(health), Channel: rec.Channel, Status: "ready"} |
| + if err := e.root.reg.Upsert(rt); err != nil { |
| + t.Fatal(err) |
| + } |
| + unitsDir := t.TempDir() |
| + fragment := filepath.Join(unitsDir, rec.Unit) |
| + if err := os.WriteFile(fragment, []byte("[Service]\nExecStart="+release.LiveLink(e.root.obs.optDir, rec.Host)+"\n"), 0644); err != nil { |
| + t.Fatal(err) |
| + } |
| + f := e.root.obs.sys.(*fakeUnits) |
| + f.shows[0].Name, f.shows[0].FragmentPath = rec.Unit, fragment |
| + f.exe = map[int]string{7: release.VersionPath(e.root.obs.optDir, rec.Account, rec.Pin, rec.App)} |
| + source := &proofBindingSource{store: e.store, sys: f, optDir: e.root.obs.optDir, openFile: os.Open, unitDirs: []string{unitsDir}} |
| + e.root.obs.resolve = source.resolve |
| + e.root.obs.ready = func(registry.Route) func() error { return func() error { return nil } } |
| + return e, rt, rec, source |
| +} |
| + |
| +func plan14PutRecord(t *testing.T, s release.Store, rec provision.Record) { |
| + t.Helper() |
| + key := provision.RecordKey(rec.Account, rec.App, rec.Host) |
| + _, etag, err := s.Get(context.Background(), key) |
| + if err != nil && !errors.Is(err, os.ErrNotExist) { |
| + t.Fatal(err) |
| + } |
| + body, err := json.Marshal(rec) |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + if _, err := s.Put(context.Background(), key, body, etag); err != nil { |
| + t.Fatal(err) |
| + } |
| +} |
| + |
| +func TestPlan14ProofObservationRejectsForgedRegistry(t *testing.T) { |
| + mutations := map[string]func(*registry.Route){ |
| + "account": func(r *registry.Route) { r.Account = "bap" }, |
| + "app": func(r *registry.Route) { r.App = "other" }, |
| + "host": func(r *registry.Route) { r.Host = "other.example.com" }, |
| + "unit": func(r *registry.Route) { r.Unit = "other.service" }, |
| + "socket": func(r *registry.Route) { r.Socket = "/run/forged.sock" }, |
| + "addr": func(r *registry.Route) { r.Addr = "" }, |
| + "addrs": func(r *registry.Route) { r.Addrs = "127.0.0.1:1" }, |
| + "health nil": func(r *registry.Route) { r.Health = "" }, |
| + "health disabled": func(r *registry.Route) { r.Health = "null" }, |
| + "health alternate": func(r *registry.Route) { r.Health = `{"path":"/forged"}` }, |
| + "all upstreams empty": func(r *registry.Route) { r.Addr, r.Addrs, r.Socket, r.Health = "", "", "", "" }, |
| + } |
| + for name, mutate := range mutations { |
| + t.Run(name, func(t *testing.T) { |
| + e, rt, _, _ := plan14ProofFixture(t) |
| + mutate(&rt) |
| + if err := e.root.reg.Delete(observedHost); err != nil { |
| + t.Fatal(err) |
| + } |
| + if err := e.root.reg.Upsert(rt); err != nil { |
| + t.Fatal(err) |
| + } |
| + if got := e.root.observeAndRecord(rt, 15*time.Minute); got.err == nil { |
| + t.Fatal("forged route recorded as healthy") |
| + } |
| + e.root.run(context.Background()) |
| + if err := e.root.obs.recordSettled(e.bake)(rt); err == nil { |
| + t.Fatal("forged settled route accepted") |
| + } |
| + keys, err := e.store.List(context.Background(), "apps/") |
| + if err != nil || len(keys) != 1 { |
| + t.Fatalf("unexpected proof writes: %v, %v", keys, err) |
| + } |
| + for _, acct := range []string{"bab", "bap"} { |
| + for _, app := range []string{"console", "other"} { |
| + if got := release.ReadHealthy(e.bake, acct, app); len(got) != 0 { |
| + t.Fatalf("forged health %s/%s: %+v", acct, app, got) |
| + } |
| + } |
| + } |
| + }) |
| + } |
| + t.Run("canonical failing readiness", func(t *testing.T) { |
| + e, rt, _, _ := plan14ProofFixture(t) |
| + e.root.obs.ready = func(r registry.Route) func() error { |
| + if r.Addr == "" { |
| + return nil |
| + } |
| + return func() error { return os.ErrPermission } |
| + } |
| + e.root.run(context.Background()) |
| + if e.root.obs.recordSettled(e.bake)(rt) == nil || len(release.ReadHealthy(e.bake, "bab", "console")) != 0 || e.proof(t) != nil { |
| + t.Fatal("failed canonical readiness authorized health") |
| + } |
| + }) |
| + t.Run("canonical success", func(t *testing.T) { |
| + e, rt, _, _ := plan14ProofFixture(t) |
| + e.root.run(context.Background()) |
| + if err := e.root.obs.recordSettled(e.bake)(rt); err != nil { |
| + t.Fatal(err) |
| + } |
| + if len(release.ReadHealthy(e.bake, "bab", "console")) != 1 || e.proof(t) == nil { |
| + t.Fatal("canonical healthy build was not recorded and proved") |
| + } |
| + }) |
| +} |
| + |
| +func TestPlan14ProofBindingChangesAfterObservation(t *testing.T) { |
| + e, _, rec, _ := plan14ProofFixture(t) |
| + obs := e.root.recordLive(context.Background()) |
| + rec.Health.Path = "/changed" |
| + plan14PutRecord(t, e.store, rec) |
| + e.root.mintAndLog(context.Background(), obs) |
| + if e.proof(t) != nil { |
| + t.Fatal("stale binding minted a proof") |
| + } |
| + rt, err := e.root.reg.Get(rec.Host) |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + health, err := json.Marshal(rec.Health.WithDefaults()) |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + rt.Health = string(health) |
| + if err := e.root.reg.Upsert(*rt); err != nil { |
| + t.Fatal(err) |
| + } |
| + e.root.run(context.Background()) |
| + if e.proof(t) == nil { |
| + t.Fatal("new successful observation did not mint") |
| + } |
| + t.Run("etag only", func(t *testing.T) { |
| + e, _, rec, _ := plan14ProofFixture(t) |
| + obs := e.root.recordLive(context.Background()) |
| + rec.RecordSeq++ |
| + plan14PutRecord(t, e.store, rec) |
| + e.root.obs.ready = func(registry.Route) func() error { return func() error { return os.ErrPermission } } |
| + e.root.mintAndLog(context.Background(), obs) |
| + if e.proof(t) != nil { |
| + t.Fatal("changed ETag reused cached observation") |
| + } |
| + }) |
| +} |
| + |
| +func TestPlan14ProofBindingProvenance(t *testing.T) { |
| + for _, attack := range []string{"nonroot owner", "writable parent", "writable file", "symlink", "rename"} { |
| + t.Run(attack, func(t *testing.T) { |
| + e, rt, _, source := plan14ProofFixture(t) |
| + path := source.sys.(*fakeUnits).shows[0].FragmentPath |
| + switch attack { |
| + case "writable parent": |
| + if err := os.Chmod(filepath.Dir(path), 0777); err != nil { |
| + t.Fatal(err) |
| + } |
| + case "writable file": |
| + if err := os.Chmod(path, 0666); err != nil { |
| + t.Fatal(err) |
| + } |
| + case "symlink": |
| + if err := os.Rename(path, path+".real"); err != nil { |
| + t.Fatal(err) |
| + } |
| + if err := os.Symlink(path+".real", path); err != nil { |
| + t.Fatal(err) |
| + } |
| + case "rename": |
| + parent := filepath.Dir(path) |
| + if err := os.Rename(parent, parent+".old"); err != nil { |
| + t.Fatal(err) |
| + } |
| + t.Cleanup(func() { os.Remove(path); os.Remove(parent); os.Rename(parent+".old", parent) }) |
| + if err := os.Mkdir(parent, 0755); err != nil { |
| + t.Fatal(err) |
| + } |
| + if err := os.WriteFile(path, []byte("forged"), 0644); err != nil { |
| + t.Fatal(err) |
| + } |
| + } |
| + source.openFile = rootstate.OpenTrustedFile |
| + if e.root.observeAndRecord(rt, 0).err == nil { |
| + t.Fatal("untrusted unit metadata accepted") |
| + } |
| + if got := release.ReadHealthy(e.bake, "bab", "console"); len(got) != 0 { |
| + t.Fatalf("untrusted metadata wrote health: %+v", got) |
| + } |
| + }) |
| + } |
| +} |
| + |
| +func TestPlan14OrdinaryInstanceBinding(t *testing.T) { |
| + e, rt, rec, source := plan14ProofFixture(t) |
| + rec.Addr, rec.Unit = "", "" |
| + plan14PutRecord(t, e.store, rec) |
| + rt.Addr, rt.Unit, rt.Socket, rt.Backing = "", "", provision.DefaultSocket(rec.Host), registry.BackingUnit |
| + if err := e.root.reg.Upsert(rt); err != nil { |
| + t.Fatal(err) |
| + } |
| + u := source.sys.(*fakeUnits) |
| + u.shows[0].Name = activator.UnitName(rec.Host) |
| + if got := e.root.observeAndRecord(rt, 0); got.err != nil { |
| + t.Fatal(got.err) |
| + } |
| + rt.Unit = "other.service" |
| + if got := e.root.observeAndRecord(rt, 0); got.err == nil { |
| + t.Fatal("forged ordinary unit accepted") |
| + } |
| + rec.Unit = "other.service" |
| + plan14PutRecord(t, e.store, rec) |
| + if _, err := source.forHost(context.Background(), rt.Host); err == nil { |
| + t.Fatal("record overrides ordinary naming") |
| + } |
| +} |
| + |
| +func TestPlan14RootRestartDispatchFromRecord(t *testing.T) { |
| + e, rt, rec, source := plan14ProofFixture(t) |
| + rec.Unit = "" |
| + rt.Unit = "" |
| + plan14PutRecord(t, e.store, rec) |
| + needs, err := source.needsRestart(context.Background(), rt) |
| + if err != nil || needs { |
| + t.Fatalf("unmanaged proxy restart: %v %v", needs, err) |
| + } |
| + rec.Unit = "console.service" |
| + plan14PutRecord(t, e.store, rec) |
| + if _, err := source.needsRestart(context.Background(), rt); err == nil { |
| + t.Fatal("cleared candidate unit suppressed authoritative restart") |
| + } |
| + rt.Unit = rec.Unit |
| + needs, err = source.needsRestart(context.Background(), rt) |
| + if err != nil || !needs { |
| + t.Fatalf("named service skipped: %v %v", needs, err) |
| + } |
| +} |
| + |
| +func TestPlan14ProofClaimBinding(t *testing.T) { |
| + e, rt, rec, s := plan14ProofFixture(t) |
| + observed := hostclaim.Observed{Host: rec.Host, ETag: "claim-one", Claim: &hostclaim.Claim{Host: rec.Host, Holder: &hostclaim.Holder{Key: provision.RecordKey(rec.Account, rec.App, rec.Host), Account: rec.Account, App: rec.App, Incarnation: "inc", Stamp: 1, Session: 1, WriterBox: "box"}, Op: hostclaim.Op{Kind: hostclaim.KindServing, Exec: "box"}}} |
| + s.claim = func(context.Context, string) (hostclaim.Observed, error) { return observed, nil } |
| + s.box = func(context.Context) (string, error) { return "box", nil } |
| + cached := e.root.recordLive(context.Background()) |
| + if cached.byHost[rt.Host].err != nil { |
| + t.Fatal(cached.byHost[rt.Host].err) |
| + } |
| + observed.Claim.Holder.App = "other" |
| + e.root.mintAndLog(context.Background(), cached) |
| + if e.proof(t) != nil { |
| + t.Fatal("changed holder minted proof") |
| + } |
| + if _, err := s.forHost(context.Background(), rt.Host); err == nil { |
| + t.Fatal("cross-app holder accepted") |
| + } |
| + observed.Claim.Holder.App = rec.App |
| + observed.Claim.Holder.WriterBox = "other-box" |
| + if _, err := s.forHost(context.Background(), rt.Host); err == nil { |
| + t.Fatal("foreign writer accepted") |
| + } |
| +} |
diff --git a/internal/host/proofobserve.go b/internal/host/proofobserve.go| index 715233b2d..054f6b2d3 100644 |
| --- a/internal/host/proofobserve.go |
| +++ b/internal/host/proofobserve.go |
| @@ -15,8 +15,9 @@ |
| package host |
| import ( |
| + "context" |
| "fmt" |
| - "log" |
| + "os" |
| "time" |
| "github.com/carlosframework/platform/internal/registry" |
| @@ -24,15 +25,18 @@ import ( |
| ) |
| type liveBuild struct { |
| + binding proofBinding |
| SHA256 string |
| Version string |
| ActiveSince time.Time |
| // The process observed, so a later look can tell it is the same one. |
| invocation string |
| pid int |
| + start string |
| } |
| type proofObserver struct { |
| + resolve proofBindingResolver |
| sys unitSystem |
| procExe func(pid int) string // "/proc/<pid>/exe" in production |
| optDir string |
| @@ -42,12 +46,30 @@ type proofObserver struct { |
| func productionProofObserver(optDir string) proofObserver { |
| return proofObserver{sys: systemdUnits{}, procExe: procExePath, optDir: optDir, now: time.Now, |
| - ready: func(rt registry.Route) func() error { return routeReady(rt, 2*time.Second) }} |
| + resolve: func(ctx context.Context, rt registry.Route) (proofBinding, error) { |
| + store, err := openDeployStore(ctx) |
| + if err != nil { |
| + return proofBinding{}, err |
| + } |
| + return productionProofBindingSource(store, systemdUnits{}, optDir).resolve(ctx, rt) |
| + }, |
| + ready: func(rt registry.Route) func() error { return proofRouteReady(rt, 2*time.Second) }} |
| } |
| func procExePath(pid int) string { return fmt.Sprintf("/proc/%d/exe", pid) } |
| func (o proofObserver) observe(rt registry.Route, after time.Duration) (liveBuild, error) { |
| + if o.resolve == nil { |
| + return liveBuild{}, os.ErrPermission |
| + } |
| + binding, err := o.resolve(context.Background(), rt) |
| + if err != nil { |
| + return liveBuild{}, err |
| + } |
| + if binding.Source == "" { |
| + return liveBuild{}, os.ErrPermission |
| + } |
| + rt = binding.Route |
| link := release.LiveLink(o.optDir, rt.Host) |
| before, err := o.sys.Show(rt.Unit) |
| if err != nil { |
| @@ -62,6 +84,10 @@ func (o proofObserver) observe(rt registry.Route, after time.Duration) (liveBuil |
| if before.ActiveEnterTimestamp.IsZero() { |
| return liveBuild{}, fmt.Errorf("%s reports no ActiveEnterTimestamp", rt.Unit) |
| } |
| + start, err := o.sys.StartTime(before.MainPID) |
| + if err != nil || start == "" { |
| + return liveBuild{}, os.ErrPermission |
| + } |
| if up := o.now().Sub(before.ActiveEnterTimestamp); up < after { |
| return liveBuild{}, fmt.Errorf("%s has been active for %s, needs %s", rt.Unit, up.Round(time.Second), after) |
| } |
| @@ -76,20 +102,37 @@ func (o proofObserver) observe(rt registry.Route, after time.Duration) (liveBuil |
| if exe != live { |
| return liveBuild{}, fmt.Errorf("the running executable (sha256 %s) is not the live link's build (sha256 %s)", exe, live) |
| } |
| - if o.ready != nil { |
| + if binding.Network { |
| + if o.ready == nil { |
| + return liveBuild{}, os.ErrPermission |
| + } |
| if ready := o.ready(rt); ready != nil { |
| if err := ready(); err != nil { |
| return liveBuild{}, fmt.Errorf("not ready: %v", err) |
| } |
| + } else { |
| + return liveBuild{}, os.ErrPermission |
| } |
| } |
| again, err := o.sys.Show(rt.Unit) |
| if err != nil || again.ActiveState != "active" || again.InvocationID != before.InvocationID || again.MainPID != before.MainPID { |
| return liveBuild{}, fmt.Errorf("%s restarted during the check", rt.Unit) |
| } |
| + current, err := o.resolve(context.Background(), rt) |
| + if err != nil || current.Source != binding.Source { |
| + return liveBuild{}, os.ErrPermission |
| + } |
| + againExe, err := release.FileSHA256(o.procExe(again.MainPID)) |
| + if err != nil || againExe != exe || again.ExecPath != before.ExecPath { |
| + return liveBuild{}, os.ErrPermission |
| + } |
| + againStart, err := o.sys.StartTime(again.MainPID) |
| + if err != nil || againStart != start { |
| + return liveBuild{}, os.ErrPermission |
| + } |
| ver, _ := release.LiveVersion(o.optDir, rt.Host) |
| return liveBuild{SHA256: exe, Version: ver, ActiveSince: before.ActiveEnterTimestamp, |
| - invocation: before.InvocationID, pid: before.MainPID}, nil |
| + invocation: before.InvocationID, pid: before.MainPID, start: start, binding: binding}, nil |
| } |
| // recordSettled is checkedUnitRestart's settled hook on the root pass: hash |
| @@ -104,37 +147,19 @@ func (o proofObserver) observe(rt registry.Route, after time.Duration) (liveBuil |
| // the binary; without both checks such a process's bytes could enter the |
| // history. A build missed here is recorded later by the root pass's "live for |
| // after" rule. |
| -func recordSettled(sys unitSystem, procExe func(pid int) string, optDir, bakeDir string, now func() time.Time) func(rt registry.Route) { |
| - return func(rt registry.Route) { |
| - p, err := sys.Show(rt.Unit) |
| - if err != nil || p.ActiveState != "active" || p.MainPID <= 0 { |
| - log.Printf("adopt: %s: settled, but cannot read %s's process for the healthy history (%v)", rt.Host, rt.Unit, err) |
| - return |
| - } |
| - link := release.LiveLink(optDir, rt.Host) |
| - if p.ExecPath != link { |
| - log.Printf("adopt: %s: settled, but %s now runs %s, not %s; not recorded as healthy", rt.Host, rt.Unit, p.ExecPath, link) |
| - return |
| - } |
| - sha, err := release.FileSHA256(procExe(p.MainPID)) |
| - if err != nil { |
| - log.Printf("adopt: %s: settled, but hashing its executable failed: %v", rt.Host, err) |
| - return |
| - } |
| - live, err := release.FileSHA256(link) |
| +func recordSettled(sys unitSystem, procExe func(pid int) string, optDir, bakeDir string, now func() time.Time) func(rt registry.Route) error { |
| + o := productionProofObserver(optDir) |
| + o.sys, o.procExe, o.now = sys, procExe, now |
| + return o.recordSettled(bakeDir) |
| +} |
| + |
| +func (o proofObserver) recordSettled(history string) func(registry.Route) error { |
| + return func(rt registry.Route) error { |
| + b, err := o.observe(rt, 0) |
| if err != nil { |
| - log.Printf("adopt: %s: settled, but hashing the live link's build failed: %v", rt.Host, err) |
| - return |
| - } |
| - if sha != live { |
| - log.Printf("adopt: %s: settled, but the running executable (sha256 %s) is not the live link's build (sha256 %s); not recorded as healthy", |
| - rt.Host, sha, live) |
| - return |
| - } |
| - ver, _ := release.LiveVersion(optDir, rt.Host) |
| - if err := release.RecordHealthy(bakeDir, release.RouteAccount(rt), rt.App, |
| - release.HealthyBuild{SHA256: sha, Version: ver, How: "settle", At: now().UTC()}); err != nil { |
| - log.Printf("adopt: %s: recording the healthy build: %v", rt.Host, err) |
| + return err |
| } |
| + return release.RecordHealthy(history, b.binding.Route.Account, b.binding.Route.App, |
| + release.HealthyBuild{SHA256: b.SHA256, Version: b.Version, How: "settle", At: o.now().UTC()}) |
| } |
| } |
diff --git a/internal/host/proofobserve_test.go b/internal/host/proofobserve_test.go| index 90dbb46ae..408c10bb7 100644 |
| --- a/internal/host/proofobserve_test.go |
| +++ b/internal/host/proofobserve_test.go |
| @@ -1,6 +1,7 @@ |
| package host |
| import ( |
| + "context" |
| "errors" |
| "net" |
| "os" |
| @@ -42,9 +43,10 @@ func observeFixture(t *testing.T) (proofObserver, registry.Route, string, time.T |
| procExe: func(int) string { return staged }, |
| optDir: opt, |
| now: func() time.Time { return now }, |
| - ready: func(registry.Route) func() error { return nil }, |
| + ready: func(registry.Route) func() error { return func() error { return nil } }, |
| } |
| rt := registry.Route{Host: observedHost, App: "console", Unit: "console.service"} |
| + o.resolve = proofFixtureResolver(rt) |
| return o, rt, staged, now |
| } |
| @@ -121,8 +123,8 @@ func TestSettledRestartRecordsTheRunningExe(t *testing.T) { |
| {ActiveState: "active", InvocationID: "old", ExecPath: filepath.Join(opt, "live", observedHost)}, |
| {ActiveState: "active", InvocationID: "new", MainPID: 7, ExecPath: filepath.Join(opt, "live", observedHost)}, |
| }, exe: map[int]string{7: staged}} |
| - settled := recordSettled(f, func(int) string { return staged }, opt, bake, time.Now) |
| - if err := checkedUnitRestart(f, opt, func(string) string { return "" }, settled)(rt); err != nil { |
| + settled := proofFixtureSettled(f, func(int) string { return staged }, opt, bake, time.Now, rt) |
| + if err := agentCheckedUnitRestart(f, opt, func(string) string { return "" }, settled)(rt); err != nil { |
| t.Fatalf("restart: %v", err) |
| } |
| sha, _ := release.FileSHA256(staged) |
| @@ -163,7 +165,7 @@ func TestRecordSettledRecordsNothingForAnotherExe(t *testing.T) { |
| rt := registry.Route{Host: observedHost, App: "console", Unit: "console.service"} |
| f := &fakeUnits{shows: []unitProps{{ActiveState: "active", InvocationID: "new", MainPID: 7, |
| ExecPath: c.execPath(opt)}}} |
| - recordSettled(f, func(int) string { return exe }, opt, bake, time.Now)(rt) |
| + proofFixtureSettled(f, func(int) string { return exe }, opt, bake, time.Now, rt)(rt) |
| if h := release.ReadHealthy(bake, "bab", "console"); len(h) != 0 { |
| t.Fatalf("recorded %+v for a process that is not the live link's build", h) |
| } |
| @@ -176,7 +178,7 @@ func TestRecordSettledRecordsNothingForAnotherExe(t *testing.T) { |
| f := &fakeUnits{shows: []unitProps{{ActiveState: "active", InvocationID: "new", MainPID: 7, |
| ExecPath: release.LiveLink(opt, observedHost)}}} |
| staged := filepath.Join(opt, "versions", "new", "console") |
| - recordSettled(f, func(int) string { return staged }, opt, bake, time.Now)(rt) |
| + proofFixtureSettled(f, func(int) string { return staged }, opt, bake, time.Now, rt)(rt) |
| if h := release.ReadHealthy(bake, "bab", "console"); len(h) != 1 || h[0].How != "settle" { |
| t.Fatalf("history = %+v, want the settled build", h) |
| } |
| @@ -188,8 +190,8 @@ func TestUnsettledRestartRecordsNothing(t *testing.T) { |
| rt := registry.Route{Host: observedHost, App: "console", Unit: "console.service"} |
| f := &fakeUnits{shows: []unitProps{{ActiveState: "active", InvocationID: "old", |
| ExecPath: filepath.Join(opt, "live", "console.example.com")}}} |
| - settled := recordSettled(f, func(int) string { return "/nonexistent" }, opt, bake, time.Now) |
| - if err := checkedUnitRestart(f, opt, func(string) string { return "" }, settled)(rt); err != nil { |
| + settled := proofFixtureSettled(f, func(int) string { return "/nonexistent" }, opt, bake, time.Now, rt) |
| + if err := agentCheckedUnitRestart(f, opt, func(string) string { return "" }, settled)(rt); err != nil { |
| t.Fatalf("restart: %v", err) |
| } |
| if h := release.ReadHealthy(bake, "bab", "console"); len(h) != 0 { |
| @@ -209,8 +211,8 @@ func TestStoppedDuringSettleRecordsNothing(t *testing.T) { |
| ExecPath: filepath.Join(opt, "live", observedHost)}, |
| {ActiveState: "inactive", SubState: "dead", InvocationID: "old"}, |
| }} |
| - settled := recordSettled(f, func(int) string { return staged }, opt, bake, time.Now) |
| - if err := checkedUnitRestart(f, opt, func(string) string { return "" }, settled)(rt); err != nil { |
| + settled := proofFixtureSettled(f, func(int) string { return staged }, opt, bake, time.Now, rt) |
| + if err := agentCheckedUnitRestart(f, opt, func(string) string { return "" }, settled)(rt); err != nil { |
| t.Fatalf("restart: %v", err) |
| } |
| if h := release.ReadHealthy(bake, "bab", "console"); len(h) != 0 { |
| @@ -229,11 +231,34 @@ func TestFailedSettleRecordsNothing(t *testing.T) { |
| ExecPath: filepath.Join(opt, "live", observedHost)}, |
| {ActiveState: "failed", SubState: "failed", InvocationID: "new"}, |
| }} |
| - settled := recordSettled(f, func(int) string { return staged }, opt, bake, time.Now) |
| - if err := checkedUnitRestart(f, opt, func(string) string { return "" }, settled)(rt); err == nil { |
| + settled := proofFixtureSettled(f, func(int) string { return staged }, opt, bake, time.Now, rt) |
| + if err := agentCheckedUnitRestart(f, opt, func(string) string { return "" }, settled)(rt); err == nil { |
| t.Fatal("a failed unit settled") |
| } |
| if h := release.ReadHealthy(bake, "bab", "console"); len(h) != 0 { |
| t.Fatalf("a failed build recorded %+v", h) |
| } |
| } |
| + |
| +func proofFixtureResolver(rt registry.Route) proofBindingResolver { |
| + rt.Account = release.RouteAccount(rt) |
| + return func(_ context.Context, candidate registry.Route) (proofBinding, error) { |
| + if candidate.Host != rt.Host || candidate.App != rt.App || candidate.Unit != rt.Unit { |
| + return proofBinding{}, os.ErrPermission |
| + } |
| + return proofBinding{Route: rt, Source: "test-installed-binding", Network: true}, nil |
| + } |
| +} |
| + |
| +func proofFixtureSettled(sys unitSystem, procExe func(int) string, opt, history string, now func() time.Time, rt registry.Route) func(registry.Route) error { |
| + if f, ok := sys.(*fakeUnits); ok { |
| + for i := range f.shows { |
| + if f.shows[i].ActiveEnterTimestamp.IsZero() { |
| + f.shows[i].ActiveEnterTimestamp = now().Add(-time.Minute) |
| + } |
| + } |
| + } |
| + o := proofObserver{sys: sys, procExe: procExe, optDir: opt, now: now, resolve: proofFixtureResolver(rt), |
| + ready: func(registry.Route) func() error { return func() error { return nil } }} |
| + return o.recordSettled(history) |
| +} |
diff --git a/internal/host/proofroot.go b/internal/host/proofroot.go| index 099f55ffe..c29dc25d6 100644 |
| --- a/internal/host/proofroot.go |
| +++ b/internal/host/proofroot.go |
| @@ -106,7 +106,7 @@ func (r proofRoot) recordLive(ctx context.Context) proofObservations { |
| return obs |
| } |
| for _, rt := range routes { |
| - if !release.Adoptable(rt) || rt.Unit == "" { |
| + if !release.Adoptable(rt) { |
| continue |
| } |
| obs.byHost[rt.Host] = r.observeAndRecord(rt, pol.ProveAfter()) |
| @@ -123,9 +123,10 @@ func (r proofRoot) observeAndRecord(rt registry.Route, after time.Duration) live |
| if err != nil { |
| return o |
| } |
| - if err := release.RecordHealthy(r.historyDir(), release.RouteAccount(rt), rt.App, release.HealthyBuild{ |
| + if err := release.RecordHealthy(r.historyDir(), b.binding.Route.Account, b.binding.Route.App, release.HealthyBuild{ |
| SHA256: b.SHA256, Version: b.Version, How: "live", At: r.obs.now().UTC()}); err != nil { |
| r.logf("proof: %s: recording the healthy build: %v", rt.Host, err) |
| + o.err = err |
| } |
| return o |
| } |
| @@ -135,6 +136,14 @@ func (r proofRoot) observeAndRecord(rt registry.Route, after time.Duration) live |
| // the unit is the same process (a rollback can put the link back where it |
| // was and restart the unit; that process has not run for after). |
| func (r proofRoot) stillLive(rt registry.Route, o liveObservation) bool { |
| + if r.obs.resolve == nil { |
| + return false |
| + } |
| + binding, err := r.obs.resolve(context.Background(), rt) |
| + if err != nil || binding.Source != o.build.binding.Source { |
| + return false |
| + } |
| + rt = binding.Route |
| target, err := os.Readlink(release.LiveLink(r.obs.optDir, rt.Host)) |
| if err != nil || target != o.target { |
| return false |
| @@ -143,7 +152,11 @@ func (r proofRoot) stillLive(rt registry.Route, o liveObservation) bool { |
| return true |
| } |
| p, err := r.obs.sys.Show(rt.Unit) |
| - return err == nil && p.ActiveState == "active" && p.InvocationID == o.build.invocation && p.MainPID == o.build.pid |
| + if err != nil || p.ActiveState != "active" || p.InvocationID != o.build.invocation || p.MainPID != o.build.pid { |
| + return false |
| + } |
| + start, err := r.obs.sys.StartTime(p.MainPID) |
| + return err == nil && start == o.build.start |
| } |
| // mintAndLog is the second half: mint on a proving box, leave the |
| @@ -161,7 +174,7 @@ func (r proofRoot) mintAndLog(ctx context.Context, obs proofObservations) { |
| type acctApp struct{ acct, app string } |
| seen := map[acctApp]bool{} |
| for _, rt := range routes { |
| - if !release.Adoptable(rt) || rt.Unit == "" { |
| + if !release.Adoptable(rt) { |
| continue |
| } |
| acct := release.RouteAccount(rt) |
| @@ -177,7 +190,7 @@ func (r proofRoot) mintAndLog(ctx context.Context, obs proofObservations) { |
| continue |
| } |
| if perr == nil && pol.Proves(rt.App) { |
| - r.mint(ctx, pol, rt, acct, o.build) |
| + r.mint(ctx, pol, o.build.binding, o.build) |
| } |
| } |
| if perr != nil { |
| @@ -234,7 +247,8 @@ func (r proofRoot) markRequired(app string) { |
| } |
| } |
| -func (r proofRoot) mint(ctx context.Context, pol *release.ProofPolicy, rt registry.Route, acct string, b liveBuild) { |
| +func (r proofRoot) mint(ctx context.Context, pol *release.ProofPolicy, binding proofBinding, b liveBuild) { |
| + rt, acct := binding.Route, binding.Route.Account |
| refusals, err := release.ListRefusals(r.bakeDir) |
| if err != nil { |
| r.logf("proof: %s: reading refusal records: %v; not minting", rt.Host, err) |
diff --git a/internal/host/proofroot_test.go b/internal/host/proofroot_test.go| index 1baa41cf6..57c80c8b6 100644 |
| --- a/internal/host/proofroot_test.go |
| +++ b/internal/host/proofroot_test.go |
| @@ -40,8 +40,9 @@ func newProofRootEnv(t *testing.T, policy func(keyFile string) string) *proofRoo |
| opt, bake, etc := t.TempDir(), t.TempDir(), t.TempDir() |
| liveTo(t, opt, observedHost, "v1") |
| reg := restartTestRegistry(t) |
| - if err := reg.Upsert(registry.Route{Host: observedHost, Kind: "instance", App: "console", |
| - Channel: "canary/console", Unit: "console.service", Status: "ready"}); err != nil { |
| + rt := registry.Route{Host: observedHost, Kind: "instance", App: "console", |
| + Channel: "canary/console", Unit: "console.service", Status: "ready"} |
| + if err := reg.Upsert(rt); err != nil { |
| t.Fatal(err) |
| } |
| seed, pubB64, err := release.GenerateKey() |
| @@ -73,7 +74,7 @@ func newProofRootEnv(t *testing.T, policy func(keyFile string) string) *proofRoo |
| ExecPath: release.LiveLink(opt, observedHost), ActiveEnterTimestamp: e.now.Add(-20 * time.Minute)}}} |
| e.root = proofRoot{ |
| obs: proofObserver{sys: units, procExe: func(int) string { return staged }, optDir: opt, |
| - now: func() time.Time { return e.now }, ready: func(registry.Route) func() error { return nil }}, |
| + now: func() time.Time { return e.now }, resolve: proofFixtureResolver(rt), ready: func(registry.Route) func() error { return func() error { return nil } }}, |
| reg: reg, store: e.store, bakeDir: bake, policyPath: policyPath, readKey: readProvingKey, |
| logf: func(f string, a ...any) { e.logs = append(e.logs, fmt.Sprintf(f, a...)) }, |
| } |
| @@ -391,7 +392,7 @@ func TestProofRootRecordsTheReplacedBuildBeforeAdoption(t *testing.T) { |
| clock := func() time.Time { return now } |
| root := proofRoot{ |
| obs: proofObserver{sys: units, procExe: procExe, optDir: opt, now: clock, |
| - ready: func(registry.Route) func() error { return nil }}, |
| + resolve: proofFixtureResolver(rt), ready: func(registry.Route) func() error { return func() error { return nil } }}, |
| reg: reg, store: s, bakeDir: bake, policyPath: policyPath, readKey: readProvingKey, |
| logf: func(f string, a ...any) { t.Logf(f, a...) }, |
| } |
| @@ -400,7 +401,9 @@ func TestProofRootRecordsTheReplacedBuildBeforeAdoption(t *testing.T) { |
| restart := func(host string) error { |
| units.shows, units.n = []unitProps{{ActiveState: "active", InvocationID: "b1", MainPID: 8, |
| ExecPath: link, ActiveEnterTimestamp: now}}, 0 |
| - recordSettled(units, procExe, opt, bake, clock)(rt) |
| + if err := root.obs.recordSettled(bake)(rt); err != nil { |
| + return err |
| + } |
| return nil |
| } |
| if err := root.around(ctx, func() error { |
diff --git a/internal/host/rootfenceview.go b/internal/host/rootfenceview.go| new file mode 100644 |
| index 000000000..c62f823ea |
| --- /dev/null |
| +++ b/internal/host/rootfenceview.go |
| @@ -0,0 +1,42 @@ |
| +package host |
| + |
| +import ( |
| + "context" |
| + "encoding/json" |
| + "net/http" |
| + "os" |
| + |
| + "github.com/carlosframework/platform/internal/hostbox" |
| + "github.com/carlosframework/platform/internal/hostguard" |
| +) |
| + |
| +func rootFenceViewClient(call hostbox.JournalCall) hostguard.FenceViewProvider { |
| + return func(ctx context.Context, host string, c hostguard.Captured) (hostguard.FenceView, error) { |
| + if call == nil { |
| + return hostguard.FenceView{}, os.ErrPermission |
| + } |
| + q := hostbox.JournalRequest{V: 1, Op: "fence-view", Target: hostguard.FenceTarget{Host: host, Incarnation: c.Incarnation, Stamp: c.Stamp}} |
| + out, err := call(ctx, q) |
| + if err != nil { |
| + return hostguard.FenceView{}, err |
| + } |
| + if out.V != 1 || out.Error != "" || out.FenceView == nil || out.FenceView.Schema != 1 || out.FenceView.Host != host || out.FenceView.Captured != c { |
| + return hostguard.FenceView{}, os.ErrPermission |
| + } |
| + return *out.FenceView, nil |
| + } |
| +} |
| +func (h ClaimRootHTTP) serveFenceView(w http.ResponseWriter, r *http.Request, q hostbox.JournalRequest) { |
| + root := h.Stop.Coordinator.Guard.Root |
| + if q.V != 1 || root == nil || q.Target.Incarnation == "" || q.Target.Stamp == 0 { |
| + w.WriteHeader(http.StatusForbidden) |
| + return |
| + } |
| + v, err := root.View(r.Context(), q.Target.Host, hostguard.Captured{Incarnation: q.Target.Incarnation, Stamp: q.Target.Stamp}) |
| + if err != nil { |
| + w.WriteHeader(http.StatusConflict) |
| + return |
| + } |
| + w.Header().Set("Content-Type", "application/json") |
| + _ = json.NewEncoder(w).Encode(hostbox.JournalResponse{V: 1, ID: q.ID, FenceView: &v}) |
| +} |
diff --git a/internal/host/rootreleaselock.go b/internal/host/rootreleaselock.go| new file mode 100644 |
| index 000000000..ca4f94b84 |
| --- /dev/null |
| +++ b/internal/host/rootreleaselock.go |
| @@ -0,0 +1,25 @@ |
| +package host |
| + |
| +import ( |
| + "context" |
| + "os" |
| + |
| + "github.com/carlosframework/platform/internal/rootstate" |
| +) |
| + |
| +const rootReleasePrivate = "/var/lib/carlos-release/private" |
| + |
| +func withClaimRootRelease(ctx context.Context, enabled bool, open func() (rootstate.Store, error), effect func() error) error { |
| + if !enabled { |
| + return effect() |
| + } |
| + if open == nil || effect == nil { |
| + return os.ErrPermission |
| + } |
| + s, err := open() |
| + if err != nil { |
| + return err |
| + } |
| + defer s.Close() |
| + return rootstate.WithRelease(ctx, s, effect) |
| +} |
diff --git a/internal/host/rootrestart.go b/internal/host/rootrestart.go| new file mode 100644 |
| index 000000000..0eada410c |
| --- /dev/null |
| +++ b/internal/host/rootrestart.go |
| @@ -0,0 +1,286 @@ |
| +package host |
| + |
| +import ( |
| + "context" |
| + "errors" |
| + "os" |
| + "path/filepath" |
| + "sync" |
| + "time" |
| + |
| + "github.com/carlosframework/platform/internal/registry" |
| + "github.com/carlosframework/platform/internal/release" |
| + "github.com/carlosframework/platform/internal/rootstate" |
| +) |
| + |
| +type rootRestartChecks struct { |
| + dispatch func(context.Context, registry.Route) (bool, error) |
| + resolve proofBindingResolver |
| + started *sync.Map |
| + attempts func() (rootstate.Store, error) |
| + rollback func(context.Context, proofBinding) (proofBinding, error) |
| + host func(context.Context, string) (proofBinding, error) |
| + ready func(registry.Route) func() error |
| + now func() time.Time |
| + sleep func(time.Duration) |
| +} |
| + |
| +func productionRootRestartChecks(sys unitSystem, optDir string) rootRestartChecks { |
| + source := func(ctx context.Context) (*proofBindingSource, error) { |
| + s, err := openDeployStore(ctx) |
| + return productionProofBindingSource(s, sys, optDir), err |
| + } |
| + return rootRestartChecks{started: &sync.Map{}, |
| + dispatch: func(ctx context.Context, rt registry.Route) (bool, error) { |
| + s, err := source(ctx) |
| + if err != nil { |
| + return false, err |
| + } |
| + return s.needsRestart(ctx, rt) |
| + }, |
| + attempts: func() (rootstate.Store, error) { return rootstate.Open(rootRestartState, true, false) }, |
| + rollback: func(ctx context.Context, b proofBinding) (proofBinding, error) { |
| + s, err := source(ctx) |
| + if err != nil { |
| + return proofBinding{}, err |
| + } |
| + return s.rollback(ctx, b) |
| + }, |
| + resolve: func(ctx context.Context, rt registry.Route) (proofBinding, error) { |
| + s, err := source(ctx) |
| + if err != nil { |
| + return proofBinding{}, err |
| + } |
| + return s.resolve(ctx, rt) |
| + }, |
| + host: func(ctx context.Context, host string) (proofBinding, error) { |
| + s, err := source(ctx) |
| + if err != nil { |
| + return proofBinding{}, err |
| + } |
| + return s.forHost(ctx, host) |
| + }, |
| + ready: func(rt registry.Route) func() error { return proofRouteReady(rt, 2*time.Second) }, |
| + now: time.Now, sleep: time.Sleep, |
| + } |
| +} |
| + |
| +func checkedUnitRestart(sys unitSystem, optDir string, getenv func(string) string, settled func(registry.Route) error, checks ...rootRestartChecks) func(registry.Route) error { |
| + c := productionRootRestartChecks(sys, optDir) |
| + if len(checks) == 1 { |
| + c = checks[0] |
| + } |
| + return func(candidate registry.Route) error { |
| + if c.attempts == nil { |
| + return os.ErrPermission |
| + } |
| + s, err := c.attempts() |
| + if err != nil { |
| + return err |
| + } |
| + defer s.Close() |
| + unlock, err := s.Lock(context.Background(), rollbackHostKey(candidate.Host)+".lock") |
| + if err != nil { |
| + return err |
| + } |
| + defer unlock() |
| + binding, err := c.resolve(context.Background(), candidate) |
| + if err != nil { |
| + return err |
| + } |
| + before, err := sys.Show(binding.Route.Unit) |
| + if err != nil { |
| + return err |
| + } |
| + if before.ActiveState == "inactive" { |
| + return errUnitStopped |
| + } |
| + if before.ActiveState != "active" || before.MainPID <= 0 || before.MainPID != binding.PID || before.InvocationID != binding.Invocation { |
| + return os.ErrPermission |
| + } |
| + attempt, err := captureRollback(s, binding, c.now(), unitSettleTimeout(getenv)) |
| + if err != nil { |
| + return err |
| + } |
| + if c.started != nil { |
| + c.started.Store(candidate.Host, attempt.ID) |
| + } |
| + err = restartBoundUnit(sys, optDir, getenv, binding, false, settled, c) |
| + if err == nil { |
| + return settleRollbackAttempt(s, attempt) |
| + } |
| + if errors.Is(err, errUnitStopped) { |
| + return errors.Join(err, settleRollbackAttempt(s, attempt)) |
| + } |
| + attempt.Failed = true |
| + var notUp *release.UnitNotUpError |
| + if !errors.As(err, ¬Up) { |
| + err = errors.Join(&release.UnitNotUpError{Unit: binding.Route.Unit, Reason: err.Error()}, err) |
| + } |
| + return errors.Join(err, writeRollbackState(s, rollbackJournalKey(candidate.Host), attempt)) |
| + } |
| +} |
| + |
| +func restartBoundUnit(sys unitSystem, optDir string, getenv func(string) string, binding proofBinding, rollback bool, settled func(registry.Route) error, c rootRestartChecks) error { |
| + rt := binding.Route |
| + if binding.Source == "" || !binding.Network || c.ready == nil { |
| + return os.ErrPermission |
| + } |
| + ready := c.ready(rt) |
| + if ready == nil { |
| + return os.ErrPermission |
| + } |
| + before, err := sys.Show(rt.Unit) |
| + if err != nil { |
| + return err |
| + } |
| + if before.ExecPath != release.LiveLink(optDir, rt.Host) { |
| + return os.ErrPermission |
| + } |
| + if before.ActiveState == "inactive" && before.SubState == "dead" { |
| + return errUnitStopped |
| + } |
| + want, err := filepath.EvalSymlinks(release.LiveLink(optDir, rt.Host)) |
| + if err != nil { |
| + return err |
| + } |
| + if rollback { |
| + if err := sys.ResetFailed(rt.Unit); err != nil { |
| + return err |
| + } |
| + if err := sys.Restart(rt.Unit); err != nil { |
| + return err |
| + } |
| + } else { |
| + if before.ActiveState != "active" || before.MainPID != binding.PID || before.InvocationID != binding.Invocation { |
| + return os.ErrPermission |
| + } |
| + start, err := sys.StartTime(before.MainPID) |
| + if err != nil || start != binding.Start { |
| + return os.ErrPermission |
| + } |
| + current, err := c.host(context.Background(), rt.Host) |
| + if err != nil || current.Source != binding.Source || current.Version != binding.Version || current.Digest != binding.Digest || current.PID != binding.PID || current.Invocation != binding.Invocation { |
| + return os.ErrPermission |
| + } |
| + if err := sys.TryRestart(rt.Unit); err != nil { |
| + return err |
| + } |
| + } |
| + var observed proofBinding |
| + checkedReady := func() error { |
| + first, err := c.host(context.Background(), rt.Host) |
| + if err != nil || first.Source != binding.Source || want != release.VersionPath(optDir, first.Route.Account, first.Version, first.Route.App) { |
| + return os.ErrPermission |
| + } |
| + if rollback && (first.Version != binding.Version || first.Digest != binding.Digest) { |
| + return os.ErrPermission |
| + } |
| + if err := ready(); err != nil { |
| + return err |
| + } |
| + last, err := c.host(context.Background(), rt.Host) |
| + if err != nil || last.Source != first.Source || last.Version != first.Version || last.Digest != first.Digest || last.Invocation != first.Invocation || last.PID != first.PID || last.Start != first.Start { |
| + return os.ErrPermission |
| + } |
| + observed = last |
| + return nil |
| + } |
| + err = settleUnit(sys, rt.Unit, before.InvocationID, settleOpts{ |
| + Timeout: unitSettleTimeout(getenv), Poll: 500 * time.Millisecond, Hold: 5 * time.Second, |
| + WantDir: filepath.Dir(want), Ready: checkedReady, Now: c.now, Sleep: c.sleep, RequireExe: true, |
| + }) |
| + if err != nil { |
| + return err |
| + } |
| + current, err := c.host(context.Background(), rt.Host) |
| + if err != nil { |
| + return err |
| + } |
| + if current.Invocation != observed.Invocation || current.PID != observed.PID || current.Start != observed.Start || current.Source != binding.Source || want != release.VersionPath(optDir, current.Route.Account, current.Version, current.Route.App) || (rollback && (current.Version != binding.Version || current.Digest != binding.Digest)) { |
| + return os.ErrPermission |
| + } |
| + if settled != nil { |
| + if err := settled(binding.Route); err != nil { |
| + return errors.Join(&release.UnitNotUpError{Unit: rt.Unit, Reason: err.Error()}, err) |
| + } |
| + } |
| + return nil |
| +} |
| + |
| +func rollbackUnitRestart(_ *registry.Registry, sys unitSystem, optDir string, getenv func(string) string, checks ...rootRestartChecks) func(string) error { |
| + c := productionRootRestartChecks(sys, optDir) |
| + if len(checks) == 1 { |
| + c = checks[0] |
| + } |
| + |
| + return func(host string) (result error) { |
| + if c.attempts == nil || c.rollback == nil { |
| + return os.ErrPermission |
| + } |
| + s, err := c.attempts() |
| + if err != nil { |
| + return err |
| + } |
| + defer s.Close() |
| + unlock, err := s.Lock(context.Background(), rollbackHostKey(host)+".lock") |
| + if err != nil { |
| + return err |
| + } |
| + defer unlock() |
| + attempt, binding, err := loadRollback(s, host, c.now()) |
| + if attempt.ID != "" { |
| + defer func() { result = errors.Join(result, settleRollbackAttempt(s, attempt)) }() |
| + } |
| + if err != nil { |
| + return err |
| + } |
| + binding, err = c.rollback(context.Background(), binding) |
| + if err != nil { |
| + return err |
| + } |
| + before, err := sys.Show(binding.Route.Unit) |
| + if err != nil { |
| + return err |
| + } |
| + if before.ActiveState == "inactive" { |
| + return errUnitStopped |
| + } |
| + if before.ActiveState != "failed" && before.MainPID != 0 && !attempt.Failed { |
| + return os.ErrPermission |
| + } |
| + attempt.Recovering = true |
| + if err := writeRollbackState(s, rollbackJournalKey(host), attempt); err != nil { |
| + return err |
| + } |
| + return restartBoundUnit(sys, optDir, getenv, binding, true, nil, c) |
| + } |
| + |
| +} |
| + |
| +func rootRestartRoute(reg *registry.Registry, optDir string, settled func(registry.Route) error) func(string) error { |
| + return rootRestartRouteWith(reg, systemdUnits{}, optDir, os.Getenv, settled, productionRootRestartChecks(systemdUnits{}, optDir)) |
| +} |
| + |
| +func rootRestartRouteWith(reg *registry.Registry, sys unitSystem, optDir string, getenv func(string) string, settled func(registry.Route) error, checks rootRestartChecks) func(string) error { |
| + restart := checkedUnitRestart(sys, optDir, getenv, settled, checks) |
| + return func(host string) error { |
| + rt, err := reg.Get(host) |
| + if err != nil { |
| + return err |
| + } |
| + if rt == nil { |
| + return os.ErrPermission |
| + } |
| + if checks.dispatch != nil { |
| + need, err := checks.dispatch(context.Background(), *rt) |
| + if err != nil { |
| + return err |
| + } |
| + if !need { |
| + return nil |
| + } |
| + } |
| + return restart(*rt) |
| + } |
| +} |
diff --git a/internal/host/rootrollback.go b/internal/host/rootrollback.go| new file mode 100644 |
| index 000000000..d7f996314 |
| --- /dev/null |
| +++ b/internal/host/rootrollback.go |
| @@ -0,0 +1,166 @@ |
| +package host |
| + |
| +import ( |
| + "context" |
| + "crypto/rand" |
| + "crypto/sha256" |
| + "encoding/hex" |
| + "encoding/json" |
| + "errors" |
| + "os" |
| + "strings" |
| + "time" |
| + |
| + "github.com/carlosframework/platform/internal/rootstate" |
| +) |
| + |
| +const rollbackMargin = 30 * time.Second |
| +const rootRestartState = "/var/lib/carlos-release/private/restarts" |
| + |
| +type rollbackAttempt struct { |
| + ID string |
| + Host string |
| + Created time.Time |
| + Deadline time.Time |
| + Failed bool |
| + Recovering bool |
| + Settled bool |
| +} |
| + |
| +type rollbackCapability struct { |
| + Attempt rollbackAttempt |
| + Binding proofBinding |
| +} |
| + |
| +func rollbackHostKey(host string) string { |
| + h := sha256.Sum256([]byte(host)) |
| + return hex.EncodeToString(h[:]) |
| +} |
| +func rollbackJournalKey(host string) string { return rollbackHostKey(host) + ".attempt" } |
| +func rollbackCapabilityKey(host, id string) string { return rollbackHostKey(host) + "." + id + ".cap" } |
| + |
| +func writeRollbackState(s rootstate.Store, key string, v any) error { |
| + b, err := json.Marshal(v) |
| + if err != nil { |
| + return err |
| + } |
| + return s.Write(key, b) |
| +} |
| + |
| +func readRollbackAttempt(s rootstate.Store, host string) (rollbackAttempt, error) { |
| + var a rollbackAttempt |
| + b, err := s.Read(rollbackJournalKey(host)) |
| + if err != nil { |
| + return a, err |
| + } |
| + if json.Unmarshal(b, &a) != nil || a.Host != host || len(a.ID) != 32 { |
| + return a, os.ErrPermission |
| + } |
| + if _, err := hex.DecodeString(a.ID); err != nil { |
| + return a, os.ErrPermission |
| + } |
| + return a, nil |
| +} |
| + |
| +func settleRollbackAttempt(s rootstate.Store, a rollbackAttempt) error { |
| + a.Settled = true |
| + if err := writeRollbackState(s, rollbackJournalKey(a.Host), a); err != nil { |
| + return err |
| + } |
| + return removeRollbackCapabilities(s, a.Host) |
| +} |
| + |
| +func captureRollback(s rootstate.Store, b proofBinding, now time.Time, timeout time.Duration) (rollbackAttempt, error) { |
| + if b.Source == "" || !b.Historical || b.Version == "" || b.Digest == "" || b.History == nil || b.History.Version != b.Version || b.History.SHA256 != b.Digest { |
| + return rollbackAttempt{}, os.ErrPermission |
| + } |
| + if prior, err := readRollbackAttempt(s, b.Route.Host); err == nil { |
| + if !prior.Settled && !now.After(prior.Deadline.Add(rollbackMargin)) { |
| + return rollbackAttempt{}, os.ErrPermission |
| + } |
| + if err := settleRollbackAttempt(s, prior); err != nil { |
| + return rollbackAttempt{}, err |
| + } |
| + } else if !errors.Is(err, os.ErrNotExist) { |
| + return rollbackAttempt{}, err |
| + } |
| + id := make([]byte, 16) |
| + if _, err := rand.Read(id); err != nil { |
| + return rollbackAttempt{}, err |
| + } |
| + a := rollbackAttempt{ID: hex.EncodeToString(id), Host: b.Route.Host, Created: now, Deadline: now.Add(2 * timeout)} |
| + if err := writeRollbackState(s, rollbackCapabilityKey(a.Host, a.ID), rollbackCapability{Attempt: a, Binding: b}); err != nil { |
| + return rollbackAttempt{}, err |
| + } |
| + if err := writeRollbackState(s, rollbackJournalKey(a.Host), a); err != nil { |
| + return rollbackAttempt{}, errors.Join(err, s.Remove(rollbackCapabilityKey(a.Host, a.ID))) |
| + } |
| + return a, nil |
| +} |
| + |
| +func loadRollback(s rootstate.Store, host string, now time.Time) (rollbackAttempt, proofBinding, error) { |
| + a, err := readRollbackAttempt(s, host) |
| + if err != nil { |
| + return rollbackAttempt{}, proofBinding{}, errors.Join(err, removeRollbackCapabilities(s, host)) |
| + } |
| + if a.Settled || a.Recovering || now.Before(a.Created) || now.After(a.Deadline.Add(rollbackMargin)) { |
| + return a, proofBinding{}, errors.Join(os.ErrPermission, settleRollbackAttempt(s, a)) |
| + } |
| + b, err := s.Read(rollbackCapabilityKey(host, a.ID)) |
| + if err != nil { |
| + return a, proofBinding{}, err |
| + } |
| + var cap rollbackCapability |
| + if json.Unmarshal(b, &cap) != nil || cap.Attempt.ID != a.ID || cap.Attempt.Host != host || !cap.Attempt.Deadline.Equal(a.Deadline) || !cap.Attempt.Created.Equal(a.Created) || cap.Binding.Route.Host != host { |
| + return a, proofBinding{}, os.ErrPermission |
| + } |
| + return a, cap.Binding, nil |
| +} |
| + |
| +func (c rootRestartChecks) finish(host string) error { |
| + if c.attempts == nil || c.started == nil { |
| + return os.ErrPermission |
| + } |
| + expected, ok := c.started.Load(host) |
| + if !ok { |
| + return nil |
| + } |
| + defer c.started.Delete(host) |
| + s, err := c.attempts() |
| + if err != nil { |
| + return err |
| + } |
| + defer s.Close() |
| + unlock, err := s.Lock(context.Background(), rollbackHostKey(host)+".lock") |
| + if err != nil { |
| + return err |
| + } |
| + defer unlock() |
| + a, err := readRollbackAttempt(s, host) |
| + if errors.Is(err, os.ErrNotExist) { |
| + return nil |
| + } |
| + if err != nil { |
| + return err |
| + } |
| + if a.ID != expected { |
| + return os.ErrPermission |
| + } |
| + return settleRollbackAttempt(s, a) |
| +} |
| + |
| +func removeRollbackCapabilities(s rootstate.Store, host string) error { |
| + names, err := s.Names() |
| + if err != nil { |
| + return err |
| + } |
| + for _, name := range names { |
| + if !strings.HasPrefix(name, rollbackHostKey(host)+".") || !strings.HasSuffix(name, ".cap") { |
| + continue |
| + } |
| + if err := s.Remove(name); err != nil { |
| + return err |
| + } |
| + } |
| + return nil |
| +} |
diff --git a/internal/host/rootrollback_test.go b/internal/host/rootrollback_test.go| new file mode 100644 |
| index 000000000..31f7eb33e |
| --- /dev/null |
| +++ b/internal/host/rootrollback_test.go |
| @@ -0,0 +1,180 @@ |
| +package host |
| + |
| +import ( |
| + "context" |
| + "encoding/json" |
| + "errors" |
| + "os" |
| + "testing" |
| + "time" |
| + |
| + "github.com/carlosframework/platform/internal/provision" |
| + "github.com/carlosframework/platform/internal/release" |
| + "github.com/carlosframework/platform/internal/rootstate" |
| +) |
| + |
| +type rollbackMemory struct { |
| + rootstate.Store |
| + files map[string][]byte |
| + failWrite bool |
| +} |
| + |
| +func (m *rollbackMemory) Read(k string) ([]byte, error) { |
| + b, ok := m.files[k] |
| + if !ok { |
| + return nil, os.ErrNotExist |
| + } |
| + return append([]byte(nil), b...), nil |
| +} |
| +func (m *rollbackMemory) Write(k string, b []byte) error { |
| + if m.failWrite { |
| + return os.ErrPermission |
| + } |
| + m.files[k] = append([]byte(nil), b...) |
| + return nil |
| +} |
| +func (m *rollbackMemory) Remove(k string) error { delete(m.files, k); return nil } |
| +func (m *rollbackMemory) Lock(context.Context, string) (func(), error) { return func() {}, nil } |
| +func (m *rollbackMemory) Close() error { return nil } |
| + |
| +func TestPlan14RollbackCapability(t *testing.T) { |
| + for _, attack := range []string{"crash", "unit identity", "etag", "missing", "stale", "later process", "settled", "operator stopped", "wrong link", "bad executable", "failed restart result", "missing journal", "different attempt", "spent", "capture write"} { |
| + t.Run(attack, func(t *testing.T) { |
| + e, rt, u, c, pub := plan14AdoptionFixture(t) |
| + u.crashNew = true |
| + mem := &rollbackMemory{files: map[string][]byte{}} |
| + c.attempts = func() (rootstate.Store, error) { return mem, nil } |
| + if attack == "capture write" { |
| + mem.failWrite = true |
| + } |
| + forward := rootRestartRouteWith(e.root.reg, u, u.opt, func(string) string { return "5s" }, e.root.obs.recordSettled(e.bake), c) |
| + restart := func(host string) error { |
| + err := forward(host) |
| + key := rollbackJournalKey(host) |
| + var a rollbackAttempt |
| + b, _ := mem.Read(key) |
| + _ = json.Unmarshal(b, &a) |
| + switch attack { |
| + case "unit identity": |
| + b, err := os.ReadFile(u.props.FragmentPath) |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + if err = os.Rename(u.props.FragmentPath, u.props.FragmentPath+".old"); err != nil { |
| + t.Fatal(err) |
| + } |
| + if err = os.WriteFile(u.props.FragmentPath, b, 0644); err != nil { |
| + t.Fatal(err) |
| + } |
| + case "etag": |
| + rec, _, err := provision.FindRecordByHost(context.Background(), e.store, host) |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + rec.RecordSeq++ |
| + plan14PutRecord(t, e.store, *rec) |
| + case "missing": |
| + delete(mem.files, rollbackCapabilityKey(host, a.ID)) |
| + case "stale": |
| + c.now = func() time.Time { return a.Deadline.Add(rollbackMargin + time.Second) } |
| + case "settled": |
| + a.Settled = true |
| + case "operator stopped": |
| + u.props.ActiveState, u.props.SubState = "inactive", "dead" |
| + case "bad executable": |
| + u.badRollback = true |
| + case "failed restart result": |
| + u.props.ActiveState, u.props.SubState = "active", "running" |
| + u.failNew = true |
| + case "missing journal": |
| + delete(mem.files, key) |
| + case "different attempt": |
| + a.ID = "different" |
| + case "spent": |
| + a.Recovering = true |
| + } |
| + if attack == "settled" || attack == "different attempt" || attack == "spent" { |
| + b, _ := json.Marshal(a) |
| + _ = mem.Write(key, b) |
| + } |
| + return err |
| + } |
| + if attack == "failed restart result" { |
| + u.failNew = true |
| + u.crashNew = false |
| + } |
| + var rollbackErr error |
| + err := release.Pass(context.Background(), e.root.reg, e.store, pub, u.opt, restart, release.BakeDir(e.bake), release.WithRestartFailurePropagation(), release.WithRollbackRestart(func(host string) error { |
| + if attack == "wrong link" { |
| + if err := release.PointLive(u.opt, host, "bab", "v2", "console"); err != nil { |
| + t.Fatal(err) |
| + } |
| + } |
| + // Reconstruct the callback after the forward operation, as another root process does. |
| + rollbackErr = rollbackUnitRestart(e.root.reg, u, u.opt, func(string) string { return "5s" }, c)(host) |
| + return rollbackErr |
| + })) |
| + positive := attack == "crash" || attack == "later process" || attack == "failed restart result" |
| + if err == nil { |
| + t.Fatal("forward failure disappeared") |
| + } |
| + if positive { |
| + if rollbackErr != nil || len(u.effects) != 3 || u.props.MainPID <= 0 { |
| + t.Fatalf("recovery: %v, %+v", rollbackErr, u) |
| + } |
| + } else if rollbackErr == nil && attack != "capture write" { |
| + t.Fatalf("%s accepted", attack) |
| + } |
| + if !positive && attack != "bad executable" && len(u.effects) > 1 { |
| + t.Fatalf("unauthorized restart: %v", u.effects) |
| + } |
| + if attack == "capture write" && len(u.effects) != 0 { |
| + t.Fatal("restart without durable capture") |
| + } |
| + if len(release.ReadHealthy(e.bake, rt.Account, rt.App)) != 1 { |
| + t.Fatal("recorded failed build") |
| + } |
| + if errors.Is(rollbackErr, os.ErrNotExist) { |
| + t.Log("absent capability refused") |
| + } |
| + for k := range mem.files { |
| + if k != rollbackJournalKey(rt.Host) { |
| + t.Fatalf("capability survived settlement: %s", k) |
| + } |
| + } |
| + }) |
| + } |
| +} |
| + |
| +func (m *rollbackMemory) Names() ([]string, error) { |
| + var names []string |
| + for k := range m.files { |
| + names = append(names, k) |
| + } |
| + return names, nil |
| +} |
| + |
| +func TestPlan14RollbackAttemptSettles(t *testing.T) { |
| + for _, failure := range []bool{false, true} { |
| + t.Run(map[bool]string{false: "success", true: "terminal"}[failure], func(t *testing.T) { |
| + e, rt, u, c, pub := plan14AdoptionFixture(t) |
| + mem := &rollbackMemory{files: map[string][]byte{}} |
| + c.attempts = func() (rootstate.Store, error) { return mem, nil } |
| + forward := rootRestartRouteWith(e.root.reg, u, u.opt, func(string) string { return "5s" }, e.root.obs.recordSettled(e.bake), c) |
| + if failure { |
| + u.crashNew = true |
| + } |
| + err := release.Pass(context.Background(), e.root.reg, e.store, pub, u.opt, forward, release.BakeDir(e.bake), release.WithRestartFailurePropagation(), release.WithRestartAttemptSettled(c.finish)) |
| + if (err != nil) != failure { |
| + t.Fatalf("pass: %v", err) |
| + } |
| + a, err := readRollbackAttempt(mem, rt.Host) |
| + if err != nil || !a.Settled { |
| + t.Fatalf("unsettled: %+v %v", a, err) |
| + } |
| + if _, err := mem.Read(rollbackCapabilityKey(rt.Host, a.ID)); !errors.Is(err, os.ErrNotExist) { |
| + t.Fatal("capability retained") |
| + } |
| + }) |
| + } |
| +} |
diff --git a/internal/host/rootroute.go b/internal/host/rootroute.go| new file mode 100644 |
| index 000000000..617e2a5f0 |
| --- /dev/null |
| +++ b/internal/host/rootroute.go |
| @@ -0,0 +1,96 @@ |
| +package host |
| + |
| +import ( |
| + "context" |
| + "crypto/sha256" |
| + "encoding/hex" |
| + "encoding/json" |
| + "github.com/carlosframework/platform/internal/activator" |
| + "github.com/carlosframework/platform/internal/provision" |
| + "github.com/carlosframework/platform/internal/registry" |
| + "os" |
| + "strings" |
| +) |
| + |
| +func rootRecordRoute(rec *provision.Record) (registry.Route, error) { |
| + rt := registry.Route{Host: rec.Host, Account: rec.Account, App: rec.App, Kind: rec.KindOrDefault(), Unit: rec.Unit, |
| + Addr: rec.Addr, Addrs: strings.Join(rec.Addrs, ","), Channel: rec.Channel, Region: rec.Region, Pool: rec.Pool} |
| + if rec.DerivesPaths() { |
| + unit := activator.UnitName(rec.Host) |
| + if rec.Unit != "" && rec.Unit != unit { |
| + return registry.Route{}, os.ErrPermission |
| + } |
| + rt.Unit, rt.Socket, rt.Backing = unit, provision.DefaultSocket(rec.Host), registry.BackingUnit |
| + } |
| + if (rt.Unit != "" && !registry.ValidUnitName(rt.Unit)) || (rt.Kind != "instance" && rt.Kind != "service") { |
| + return registry.Route{}, os.ErrPermission |
| + } |
| + if h := rec.HealthOrDefault(); h != nil { |
| + body, err := json.Marshal(h.WithDefaults()) |
| + if err != nil { |
| + return registry.Route{}, err |
| + } |
| + rt.Health = string(body) |
| + } |
| + if rt.Socket == "" && len(rec.AddrsOrDefault()) == 0 { |
| + return registry.Route{}, os.ErrPermission |
| + } |
| + |
| + return rt, nil |
| +} |
| + |
| +func sameProofCandidate(candidate, r registry.Route, declaredUnit string) bool { |
| + return candidate.Account == r.Account && candidate.App == r.App && candidate.Host == r.Host && (candidate.Unit == declaredUnit || candidate.Unit == r.Unit) && candidate.Socket == r.Socket && candidate.Addr == r.Addr && candidate.Addrs == r.Addrs && candidate.Health == r.Health && candidate.Kind == r.Kind && candidate.Backing == r.Backing |
| +} |
| +func (s *proofBindingSource) needsRestart(ctx context.Context, candidate registry.Route) (bool, error) { |
| + rec, _, err := provision.FindRecordByHost(ctx, s.store, candidate.Host) |
| + if err != nil { |
| + return false, err |
| + } |
| + if _, err := s.claimIdentity(ctx, rec); err != nil { |
| + return false, err |
| + } |
| + rt, err := rootRecordRoute(rec) |
| + if err != nil { |
| + return false, err |
| + } |
| + if !sameProofCandidate(candidate, rt, rec.Unit) { |
| + return false, os.ErrPermission |
| + } |
| + return rt.Unit != "", nil |
| +} |
| + |
| +func (s *proofBindingSource) claimIdentity(ctx context.Context, rec *provision.Record) (string, error) { |
| + if s.claim == nil && s.box == nil { |
| + return "", nil |
| + } |
| + if s.claim == nil || s.box == nil { |
| + return "", os.ErrPermission |
| + } |
| + box, err := s.box(ctx) |
| + if err != nil { |
| + return "", err |
| + } |
| + o, err := s.claim(ctx, rec.Host) |
| + if err != nil { |
| + return "", err |
| + } |
| + if o.Absent || o.ETag == "" || o.Host != rec.Host || o.Claim == nil || o.Claim.Host != rec.Host || o.Claim.Holder == nil { |
| + return "", os.ErrPermission |
| + } |
| + h := o.Claim.Holder |
| + if h.Key != provision.RecordKey(rec.Account, rec.App, rec.Host) || h.Account != rec.Account || h.App != rec.App || h.Incarnation == "" || h.Stamp == 0 || h.Session == 0 || box == "" || h.WriterBox != box || o.Claim.Op.Exec != box { |
| + return "", os.ErrPermission |
| + } |
| + for _, lost := range o.Claim.Lost { |
| + if lost == box { |
| + return "", os.ErrPermission |
| + } |
| + } |
| + b, err := json.Marshal(o.Claim) |
| + if err != nil { |
| + return "", err |
| + } |
| + sum := sha256.Sum256(b) |
| + return o.ETag + "@" + hex.EncodeToString(sum[:]) + "@", nil |
| +} |
diff --git a/internal/host/unitsettle.go b/internal/host/unitsettle.go| index 918ec0bae..54767a4b1 100644 |
| --- a/internal/host/unitsettle.go |
| +++ b/internal/host/unitsettle.go |
| @@ -32,6 +32,9 @@ import ( |
| ) |
| type unitProps struct { |
| + Name string |
| + FragmentPath string |
| + DropInPaths []string |
| ActiveState string |
| SubState string // "dead" with ActiveState inactive is a stop, not a crash |
| InvocationID string |
| @@ -49,13 +52,14 @@ type unitSystem interface { |
| ResetFailed(unit string) error |
| JournalTail(unit string, n int) string |
| Exe(pid int) (string, error) |
| + StartTime(pid int) (string, error) |
| } |
| type systemdUnits struct{} |
| func (systemdUnits) Show(unit string) (unitProps, error) { |
| out, err := exec.Command("systemctl", "show", unit, |
| - "-p", "ActiveState", "-p", "SubState", "-p", "InvocationID", "-p", "MainPID", "-p", "NRestarts", "-p", "ExecStart", "-p", "ActiveEnterTimestamp").Output() |
| + "-p", "Id", "-p", "FragmentPath", "-p", "DropInPaths", "-p", "ActiveState", "-p", "SubState", "-p", "InvocationID", "-p", "MainPID", "-p", "NRestarts", "-p", "ExecStart", "-p", "ActiveEnterTimestamp").Output() |
| if err != nil { |
| return unitProps{}, fmt.Errorf("systemctl show %s: %w", unit, err) |
| } |
| @@ -75,6 +79,28 @@ func (systemdUnits) Exe(pid int) (string, error) { |
| return os.Readlink(fmt.Sprintf("/proc/%d/exe", pid)) |
| } |
| +func (systemdUnits) StartTime(pid int) (string, error) { |
| + if pid <= 0 { |
| + return "", os.ErrPermission |
| + } |
| + body, err := os.ReadFile(fmt.Sprintf("/proc/%d/stat", pid)) |
| + if err != nil { |
| + return "", err |
| + } |
| + i := strings.LastIndexByte(string(body), ')') |
| + if i < 0 { |
| + return "", os.ErrPermission |
| + } |
| + fields := strings.Fields(string(body[i+1:])) |
| + if len(fields) <= 19 { |
| + return "", os.ErrPermission |
| + } |
| + if n, err := strconv.ParseUint(fields[19], 10, 64); err != nil || n == 0 { |
| + return "", os.ErrPermission |
| + } |
| + return fields[19], nil |
| +} |
| + |
| func systemctlVerb(verb, unit string) error { |
| out, err := exec.Command("systemctl", verb, unit).CombinedOutput() |
| if err != nil { |
| @@ -93,6 +119,12 @@ func parseShow(out string) unitProps { |
| continue |
| } |
| switch k { |
| + case "Id": |
| + p.Name = v |
| + case "FragmentPath": |
| + p.FragmentPath = v |
| + case "DropInPaths": |
| + p.DropInPaths = strings.Fields(v) |
| case "ActiveState": |
| p.ActiveState = v |
| case "SubState": |
| @@ -126,13 +158,14 @@ func parseSystemdTime(v string) time.Time { |
| } |
| type settleOpts struct { |
| - Timeout time.Duration // the whole window (CARLOS_UNIT_SETTLE) |
| - Poll time.Duration // 500ms |
| - Hold time.Duration // no-probe readiness: active this long in one invocation |
| - WantDir string // the new version's directory; MainPID's exe must be under it |
| - Ready func() error // nil = no upstream to probe; Hold applies |
| - Now func() time.Time |
| - Sleep func(time.Duration) |
| + RequireExe bool |
| + Timeout time.Duration // the whole window (CARLOS_UNIT_SETTLE) |
| + Poll time.Duration // 500ms |
| + Hold time.Duration // no-probe readiness: active this long in one invocation |
| + WantDir string // the new version's directory; MainPID's exe must be under it |
| + Ready func() error // nil = no upstream to probe; Hold applies |
| + Now func() time.Time |
| + Sleep func(time.Duration) |
| } |
| // errUnitStopped is settleUnit's answer for a unit that was stopped while it |
| @@ -206,7 +239,7 @@ func settleUnit(sys unitSystem, unit, beforeInvocation string, o settleOpts) err |
| if !binCheckSkipped { |
| exe, err := sys.Exe(p.MainPID) |
| if err != nil { |
| - if errors.Is(err, os.ErrPermission) { |
| + if errors.Is(err, os.ErrPermission) && !o.RequireExe { |
| // Permission error; log once and skip binary check for rest |
| log.Printf("adopt: %s: cannot read /proc/%d/exe (%v); binary check skipped", unit, p.MainPID, err) |
| binCheckSkipped = true |
diff --git a/internal/host/unitsettle_test.go b/internal/host/unitsettle_test.go| index a949ee2ef..bb0ce3561 100644 |
| --- a/internal/host/unitsettle_test.go |
| +++ b/internal/host/unitsettle_test.go |
| @@ -8,6 +8,7 @@ import ( |
| "net/http/httptest" |
| "os" |
| "path/filepath" |
| + "strconv" |
| "strings" |
| "syscall" |
| "testing" |
| @@ -50,7 +51,8 @@ func (f *fakeUnits) ResetFailed(u string) error { |
| f.restarts = append(f.restarts, "reset-failed "+u) |
| return nil |
| } |
| -func (f *fakeUnits) JournalTail(string, int) string { return f.journal } |
| +func (f *fakeUnits) JournalTail(string, int) string { return f.journal } |
| +func (f *fakeUnits) StartTime(pid int) (string, error) { return strconv.Itoa(pid), nil } |
| func (f *fakeUnits) Exe(pid int) (string, error) { |
| if err, ok := f.exeErr[pid]; ok { |
| return "", err |
| @@ -386,7 +388,7 @@ func TestNamedRestartBindingMismatchKeepsTheError(t *testing.T) { |
| f := &fakeUnits{shows: []unitProps{{ActiveState: "active", InvocationID: "old", |
| ExecPath: filepath.Join(opt, "live", "console.example.com")}}} |
| failing := &failingTry{fakeUnits: f} |
| - err := checkedUnitRestart(failing, opt, func(string) string { return "" }, nil)(boundRoute()) |
| + err := agentCheckedUnitRestart(failing, opt, func(string) string { return "" }, nil)(boundRoute()) |
| if err == nil || !strings.Contains(err.Error(), "try-restart failed") { |
| t.Fatalf("err = %v, want the try-restart error propagated (codex 12)", err) |
| } |
| @@ -399,7 +401,7 @@ func (failingTry) TryRestart(string) error { return errors.New("try-restart fail |
| func TestNamedRestartShowFailsFallsBack(t *testing.T) { |
| opt := t.TempDir() |
| sys := &showFails{fakeUnits: &fakeUnits{}} |
| - if err := checkedUnitRestart(sys, opt, func(string) string { return "" }, nil)(boundRoute()); err != nil { |
| + if err := agentCheckedUnitRestart(sys, opt, func(string) string { return "" }, nil)(boundRoute()); err != nil { |
| t.Fatalf("err = %v, want today's behaviour (plain try-restart, its nil)", err) |
| } |
| if len(sys.restarts) != 1 || sys.restarts[0] != "try-restart console.service" { |
| @@ -415,7 +417,7 @@ func TestNamedRestartStoppedUnitIsNotChecked(t *testing.T) { |
| opt := t.TempDir() |
| liveTo(t, opt, "platform.example.com", "new") |
| f := &fakeUnits{shows: []unitProps{{ActiveState: "inactive", ExecPath: filepath.Join(opt, "live", "platform.example.com")}}} |
| - if err := checkedUnitRestart(f, opt, func(string) string { return "" }, nil)(boundRoute()); err != nil { |
| + if err := agentCheckedUnitRestart(f, opt, func(string) string { return "" }, nil)(boundRoute()); err != nil { |
| t.Fatalf("err = %v, want nil for an operator-stopped unit", err) |
| } |
| } |
| @@ -433,7 +435,7 @@ func TestRollbackRestartResetsThenRestarts(t *testing.T) { |
| {ActiveState: "active", InvocationID: "bad"}, |
| {ActiveState: "active", InvocationID: "restored", MainPID: 9}, |
| }, exe: map[int]string{9: filepath.Join(opt, "versions", "old", "console")}} |
| - rb := rollbackUnitRestart(reg, f, opt, func(string) string { return "" }) |
| + rb := agentRollbackUnitRestart(reg, f, opt, func(string) string { return "" }) |
| if err := rb("platform.example.com"); err != nil { |
| t.Fatalf("rollback = %v", err) |
| } |
| @@ -472,7 +474,7 @@ func TestNamedRestartSettlesAHealthyForwardRestart(t *testing.T) { |
| ExecPath: filepath.Join(opt, "live", "platform.example.com")}, |
| {ActiveState: "active", SubState: "running", InvocationID: "new", MainPID: 7}, |
| }, exe: map[int]string{7: filepath.Join(newBuildDir(t, opt, "platform.example.com"), "console")}} |
| - if err := checkedUnitRestart(f, opt, func(string) string { return "" }, nil)(rt); err != nil { |
| + if err := agentCheckedUnitRestart(f, opt, func(string) string { return "" }, nil)(rt); err != nil { |
| t.Fatalf("forward restart = %v, want settled", err) |
| } |
| if strings.Join(f.restarts, ",") != "try-restart console.service" { |
| @@ -492,7 +494,7 @@ func TestNamedRestartStopDuringSettleIsNotJudged(t *testing.T) { |
| ExecPath: filepath.Join(opt, "live", "platform.example.com")}, |
| {ActiveState: "inactive", SubState: "dead", InvocationID: "old"}, |
| }} |
| - if err := checkedUnitRestart(f, opt, func(string) string { return "" }, nil)(boundRoute()); err != nil { |
| + if err := agentCheckedUnitRestart(f, opt, func(string) string { return "" }, nil)(boundRoute()); err != nil { |
| t.Fatalf("err = %v, want nil: a stop is not a verdict on the build", err) |
| } |
| if strings.Join(f.restarts, ",") != "try-restart console.service" { |
| @@ -510,7 +512,7 @@ func TestNamedRestartFailedStillFails(t *testing.T) { |
| {ActiveState: "failed", SubState: "failed", InvocationID: "new"}, |
| }} |
| var nu *release.UnitNotUpError |
| - if err := checkedUnitRestart(f, opt, func(string) string { return "" }, nil)(boundRoute()); !errors.As(err, &nu) { |
| + if err := agentCheckedUnitRestart(f, opt, func(string) string { return "" }, nil)(boundRoute()); !errors.As(err, &nu) { |
| t.Fatalf("err = %v, want UnitNotUpError for a failed unit", err) |
| } |
| } |
| @@ -530,7 +532,7 @@ func TestRollbackRestartStopDuringSettleIsNotJudged(t *testing.T) { |
| {ActiveState: "active", InvocationID: "bad"}, |
| {ActiveState: "inactive", SubState: "dead", InvocationID: "bad"}, |
| }} |
| - if err := rollbackUnitRestart(reg, f, opt, func(string) string { return "" })("platform.example.com"); err != nil { |
| + if err := agentRollbackUnitRestart(reg, f, opt, func(string) string { return "" })("platform.example.com"); err != nil { |
| t.Fatalf("rollback = %v, want nil for a stop during its settle", err) |
| } |
| if strings.Join(f.restarts, ",") != "reset-failed console.service,restart console.service" { |
diff --git a/internal/hostbox/journalservice.go b/internal/hostbox/journalservice.go| index 4c02c6737..22ad04408 100644 |
| --- a/internal/hostbox/journalservice.go |
| +++ b/internal/hostbox/journalservice.go |
| @@ -78,16 +78,17 @@ type JournalRequest struct { |
| IssuedAt int64 `json:"issued_at,omitempty"` |
| } |
| type JournalResponse struct { |
| - Revision uint64 `json:"inventory_revision"` |
| - LastRequest string `json:"last_request"` |
| - V int `json:"v"` |
| - ID string `json:"request_id"` |
| - Launch LaunchInventory `json:"launch"` |
| - Launches []LaunchInventory `json:"launches"` |
| - Inventory StopInventory `json:"inventory"` |
| - Completion ProducerCompletion `json:"completion"` |
| - Found bool `json:"found"` |
| - Error string `json:"error"` |
| + FenceView *hostguard.FenceView `json:"fence_view,omitempty"` |
| + Revision uint64 `json:"inventory_revision"` |
| + LastRequest string `json:"last_request"` |
| + V int `json:"v"` |
| + ID string `json:"request_id"` |
| + Launch LaunchInventory `json:"launch"` |
| + Launches []LaunchInventory `json:"launches"` |
| + Inventory StopInventory `json:"inventory"` |
| + Completion ProducerCompletion `json:"completion"` |
| + Found bool `json:"found"` |
| + Error string `json:"error"` |
| } |
| type JournalCall func(context.Context, JournalRequest) (JournalResponse, error) |
| @@ -159,7 +160,7 @@ func JournalReadOrSettle(op string) bool { |
| } |
| func journalRead(op string) bool { |
| - return op == "read" || op == "session" || op == "has-host" || op == "has-effects" || op == "inventory-read" || op == "inventory-exact" || op == "inventory-version" |
| + return op == "fence-view" || op == "read" || op == "session" || op == "has-host" || op == "has-effects" || op == "inventory-read" || op == "inventory-exact" || op == "inventory-version" |
| } |
| // journalRequestV1 freezes every field, including nested inventories, in an |
diff --git a/internal/hostbox/reconcile.go b/internal/hostbox/reconcile.go| index 0081ca0ab..5d5ed5b96 100644 |
| --- a/internal/hostbox/reconcile.go |
| +++ b/internal/hostbox/reconcile.go |
| @@ -108,7 +108,7 @@ func (r Reconciler) Install(ctx context.Context, selected RecordSelection, write |
| if err := guard.Activate(ctx, selected.Host, r.Box, nil); err != nil { |
| return err |
| } |
| - captured, err := hostguard.Capture(guard.DataDir, selected.Host) |
| + captured, err := guard.Capture(selected.Host) |
| if err != nil { |
| return err |
| } |
diff --git a/internal/hostbox/reconcile_adapter.go b/internal/hostbox/reconcile_adapter.go| index ceb403ebc..3fe755848 100644 |
| --- a/internal/hostbox/reconcile_adapter.go |
| +++ b/internal/hostbox/reconcile_adapter.go |
| @@ -79,7 +79,7 @@ func (a ReconcileAdapter) Install(ctx context.Context, rec provision.ClaimRecord |
| return nil |
| } |
| return reconciler.Install(ctx, selected, func() error { |
| - captured, err := hostguard.Capture(a.Reconciler.Guard.DataDir, rec.Host) |
| + captured, err := a.Reconciler.Guard.Capture(rec.Host) |
| if err != nil { |
| return err |
| } |
diff --git a/internal/hostbox/work.go b/internal/hostbox/work.go| index 14831fbf9..9063c5c28 100644 |
| --- a/internal/hostbox/work.go |
| +++ b/internal/hostbox/work.go |
| @@ -58,7 +58,7 @@ func (s WorkSource) Select(ctx context.Context, host, action string) (hostwrite. |
| if ctx == nil || s.Box == "" || s.Region == "" || action == "" || s.Guard.Claims == nil || s.Policy == nil || s.Local == nil || s.Now == nil { |
| return nil, ErrUnsupported |
| } |
| - captured, err := hostguard.Capture(s.Guard.DataDir, host) |
| + captured, err := s.Guard.Capture(host) |
| if err != nil { |
| return nil, err |
| } |
diff --git a/internal/hostguard/coldhandover.go b/internal/hostguard/coldhandover.go| index 23d392925..df473c2bc 100644 |
| --- a/internal/hostguard/coldhandover.go |
| +++ b/internal/hostguard/coldhandover.go |
| @@ -44,6 +44,9 @@ func (g Guard) BeginColdHandover(ctx context.Context, prior, barrier FenceTarget |
| } |
| copied := barrier |
| r.ActiveFence = &copied |
| + if r.root { |
| + r.barrier = &copied |
| + } |
| return d.write(r) |
| }) |
| } |
| @@ -77,13 +80,14 @@ func (g Guard) CompleteColdHandover(ctx context.Context, barrier FenceTarget, al |
| r.ActiveStamp = barrier.Stamp |
| } |
| r.ActiveFence = nil |
| + r.barrier = nil |
| return d.write(r) |
| }) |
| } |
| func (g Guard) CompletedColdPredecessor(ctx context.Context, host string, captured Captured, session uint64, op, phase string) (FenceReceipt, error) { |
| var found FenceReceipt |
| - err := g.locked(ctx, host, func(d *recordDir) error { |
| + err := g.fenceLocked(ctx, host, captured, func(d *recordDir) error { |
| r, err := d.read(host) |
| if err != nil { |
| return err |
diff --git a/internal/hostguard/fence.go b/internal/hostguard/fence.go| index 7e9f3baf3..29ec84e76 100644 |
| --- a/internal/hostguard/fence.go |
| +++ b/internal/hostguard/fence.go |
| @@ -134,7 +134,7 @@ func (g Guard) BeginFence(ctx context.Context, target FenceTarget) (FenceTarget, |
| // It does not consult the claim and is safe for transport recovery. |
| func (g Guard) FenceReceiptFor(ctx context.Context, target FenceTarget) (FenceReceipt, error) { |
| var found FenceReceipt |
| - err := g.locked(ctx, target.Host, func(d *recordDir) error { |
| + err := g.fenceLocked(ctx, target.Host, Captured{Incarnation: target.Incarnation, Stamp: target.Stamp}, func(d *recordDir) error { |
| r, err := d.read(target.Host) |
| if err != nil { |
| return err |
diff --git a/internal/hostguard/guard.go b/internal/hostguard/guard.go| index 04b4be015..0c6bfe101 100644 |
| --- a/internal/hostguard/guard.go |
| +++ b/internal/hostguard/guard.go |
| @@ -39,6 +39,8 @@ func (r StoreReader) Read(ctx context.Context, host string) (hostclaim.Observed, |
| } |
| type Guard struct { |
| + Root *RootAuthority |
| + FenceView FenceViewProvider |
| DataDir string |
| LockDir string |
| Claims ClaimReader |
| @@ -52,6 +54,16 @@ func (g Guard) locked(ctx context.Context, host string, fn func(*recordDir) erro |
| if g.DataDir == "" || ctx == nil { |
| return ErrInvalidInput |
| } |
| + if g.Root != nil { |
| + if g.Root.store == nil { |
| + return ErrInvalidInput |
| + } |
| + unlock, err := g.Root.store.Lock(ctx, host+".lock") |
| + if err != nil { |
| + return err |
| + } |
| + defer unlock() |
| + } |
| dir := g.LockDir |
| if dir == "" { |
| dir = "/run/carlos/hostguard" |
| @@ -61,6 +73,9 @@ func (g Guard) locked(ctx context.Context, host string, fn func(*recordDir) erro |
| return err |
| } |
| defer unlock() |
| + if g.Root != nil { |
| + return fn(&recordDir{root: g.Root}) |
| + } |
| open := g.openRecords |
| if open == nil { |
| open = openRecordDir |
| @@ -119,6 +134,15 @@ func (g Guard) With(ctx context.Context, host string, captured Captured, check f |
| if err != nil { |
| return err |
| } |
| + if g.FenceView != nil { |
| + v, err := g.FenceView(ctx, host, captured) |
| + if err != nil { |
| + return err |
| + } |
| + if err := v.admit(host, captured, c.Holder.Session); err != nil { |
| + return err |
| + } |
| + } |
| if c.Holder.Incarnation != captured.Incarnation { |
| return ErrClaimIncarnation |
| } |
| @@ -132,6 +156,9 @@ func (g Guard) With(ctx context.Context, host string, captured Captured, check f |
| if err != nil { |
| return err |
| } |
| + if r.root && (r.installation.Account != c.Holder.Account || r.installation.App != c.Holder.App) { |
| + return ErrRecordIncarnation |
| + } |
| if r.Incarnation != captured.Incarnation { |
| return ErrRecordIncarnation |
| } |
| @@ -141,6 +168,9 @@ func (g Guard) With(ctx context.Context, host string, captured Captured, check f |
| if captured.Stamp != r.ActiveStamp { |
| return ErrStampMismatch |
| } |
| + if r.barrier != nil { |
| + return ErrFenced |
| + } |
| if r.ActiveFence != nil && r.ActiveFence.matchesSession(c, captured) { |
| return ErrFenced |
| } |
| @@ -163,14 +193,33 @@ func (r *record) retire() { |
| if r.ActiveStamp == 0 { |
| return |
| } |
| + if r.root { |
| + r.rootRetired = append(r.rootRetired, Captured{Incarnation: r.Incarnation, Stamp: r.ActiveStamp}) |
| + } |
| r.RetiredStamps = append(r.RetiredStamps, r.ActiveStamp) |
| - if len(r.RetiredStamps) > 32 { |
| + if !r.root && len(r.RetiredStamps) > 32 { |
| r.RetiredStamps = r.RetiredStamps[len(r.RetiredStamps)-32:] |
| } |
| r.ActiveStamp = 0 |
| } |
| func (g Guard) Activate(ctx context.Context, host, box string, participant func(*hostclaim.Claim) bool) error { |
| + if g.Root != nil { |
| + return g.locked(ctx, host, func(d *recordDir) error { |
| + c, err := g.fresh(ctx, host) |
| + if err != nil { |
| + return err |
| + } |
| + r, err := d.read(host) |
| + if err != nil { |
| + return err |
| + } |
| + if r.barrier != nil || r.Incarnation != c.Holder.Incarnation || r.ActiveStamp != c.Holder.Stamp || r.ActiveStamp == 0 || r.installation.Box != box { |
| + return ErrFenced |
| + } |
| + return d.sync(host) |
| + }) |
| + } |
| return g.locked(ctx, host, func(d *recordDir) error { |
| c, err := g.fresh(ctx, host) |
| if err != nil { |
diff --git a/internal/hostguard/plan14_root_fences_test.go b/internal/hostguard/plan14_root_fences_test.go| new file mode 100644 |
| index 000000000..97b60de5d |
| --- /dev/null |
| +++ b/internal/hostguard/plan14_root_fences_test.go |
| @@ -0,0 +1,165 @@ |
| +//go:build unix |
| + |
| +package hostguard |
| + |
| +import ( |
| + "context" |
| + "errors" |
| + "os" |
| + "path/filepath" |
| + "sync" |
| + "testing" |
| + "time" |
| + |
| + "github.com/carlosframework/platform/internal/rootstate" |
| +) |
| + |
| +type rootTestStore struct { |
| + rootstate.Store |
| + dir string |
| + mu sync.Mutex |
| +} |
| + |
| +func (s *rootTestStore) Read(n string) ([]byte, error) { return os.ReadFile(filepath.Join(s.dir, n)) } |
| +func (s *rootTestStore) Write(n string, b []byte) error { |
| + return os.WriteFile(filepath.Join(s.dir, n), b, 0600) |
| +} |
| +func (s *rootTestStore) Names() ([]string, error) { |
| + entries, e := os.ReadDir(s.dir) |
| + var names []string |
| + for _, n := range entries { |
| + names = append(names, n.Name()) |
| + } |
| + return names, e |
| +} |
| +func (s *rootTestStore) Lock(context.Context, string) (func(), error) { |
| + s.mu.Lock() |
| + return s.mu.Unlock, nil |
| +} |
| +func (s *rootTestStore) Close() error { return nil } |
| + |
| +func TestPlan14RootFences(t *testing.T) { |
| + for _, attack := range []string{"delete", "rename", "replace lock", "forged receipt"} { |
| + t.Run(attack, func(t *testing.T) { |
| + f, target, captured := fenceFixture(t) |
| + store := &rootTestStore{dir: t.TempDir()} |
| + root := NewRootAuthority(store) |
| + f.g.Root = root |
| + f.obs.Claim.Holder.Account, f.obs.Claim.Holder.App = "bab", "console" |
| + installation := RootInstallation{Host: target.Host, Incarnation: captured.Incarnation, Stamp: captured.Stamp, Account: "bab", App: "console", Record: "record@etag", Box: target.Box, Enrollment: target.Enrollment, Migration: 1} |
| + if err := f.g.EstablishRoot(t.Context(), installation, allow); err != nil { |
| + t.Fatal(err) |
| + } |
| + if err := f.g.With(t.Context(), target.Host, captured, allow, func() error { return nil }); err != nil { |
| + t.Fatal(err) |
| + } |
| + if _, err := f.g.BeginFence(t.Context(), target); err != nil { |
| + t.Fatal(err) |
| + } |
| + before, err := store.Read(target.Host + ".json") |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + path := filepath.Join(f.g.DataDir, "hostguard", target.Host+".json") |
| + switch attack { |
| + case "delete": |
| + if err := os.Remove(path); err != nil { |
| + t.Fatal(err) |
| + } |
| + case "rename": |
| + if err := os.Rename(filepath.Dir(path), filepath.Dir(path)+".old"); err != nil { |
| + t.Fatal(err) |
| + } |
| + case "replace lock": |
| + if err := os.Rename(f.g.LockDir, f.g.LockDir+".old"); err != nil { |
| + t.Fatal(err) |
| + } |
| + if err := os.Mkdir(f.g.LockDir, 0700); err != nil { |
| + t.Fatal(err) |
| + } |
| + case "forged receipt": |
| + if err := os.WriteFile(path, []byte(`{"v":1,"host":"shop.example","incarnation":"inc","active_stamp":100}`), 0660); err != nil { |
| + t.Fatal(err) |
| + } |
| + } |
| + effects := 0 |
| + if err := f.g.With(t.Context(), target.Host, captured, allow, func() error { effects++; return nil }); !errors.Is(err, ErrFenced) || effects != 0 { |
| + t.Fatalf("fence bypass: %v %d", err, effects) |
| + } |
| + pending, err := f.g.PendingFences(t.Context()) |
| + if err != nil || len(pending) != 1 || pending[0] != target { |
| + t.Fatalf("lost recovery: %+v %v", pending, err) |
| + } |
| + after, err := store.Read(target.Host + ".json") |
| + if err != nil || string(before) != string(after) { |
| + t.Fatal("root evidence changed", err) |
| + } |
| + got, err := f.g.Capture(target.Host) |
| + if err != nil || got != captured { |
| + t.Fatal("capture used agent state", got, err) |
| + } |
| + }) |
| + } |
| +} |
| + |
| +func TestPlan14RootFenceView(t *testing.T) { |
| + f, target, captured := fenceFixture(t) |
| + store := &rootTestStore{dir: t.TempDir()} |
| + f.g.Root = NewRootAuthority(store) |
| + f.obs.Claim.Holder.Account, f.obs.Claim.Holder.App = "bab", "console" |
| + in := RootInstallation{Host: target.Host, Incarnation: captured.Incarnation, Stamp: captured.Stamp, Account: "bab", App: "console", Record: "record@etag", Box: target.Box, Enrollment: target.Enrollment, Migration: 1} |
| + if err := f.g.EstablishRoot(t.Context(), in, allow); err != nil { |
| + t.Fatal(err) |
| + } |
| + if _, err := f.g.BeginFence(t.Context(), target); err != nil { |
| + t.Fatal(err) |
| + } |
| + view, err := f.g.Root.View(t.Context(), target.Host, captured) |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + view.Pending.FenceID = "forged" |
| + again, err := f.g.Root.View(t.Context(), target.Host, captured) |
| + if err != nil || again.Pending.FenceID != target.FenceID { |
| + t.Fatal("mutable view", err) |
| + } |
| + agent := f.g |
| + agent.Root = nil |
| + agent.FenceView = f.g.Root.View |
| + effects := 0 |
| + if err := agent.With(t.Context(), target.Host, captured, allow, func() error { effects++; return nil }); !errors.Is(err, ErrFenced) || effects != 0 { |
| + t.Fatal("agent bypassed root view", err, effects) |
| + } |
| + if _, err := f.g.Root.View(t.Context(), target.Host, Captured{Incarnation: "wrong", Stamp: 1}); err == nil { |
| + t.Fatal("wrong tuple accepted") |
| + } |
| + agent.FenceView = func(context.Context, string, Captured) (FenceView, error) { return FenceView{}, nil } |
| + if err := agent.With(t.Context(), target.Host, captured, allow, func() error { effects++; return nil }); err == nil || effects != 0 { |
| + t.Fatal("old server granted work") |
| + } |
| +} |
| + |
| +func TestPlan14InventoryRegistrationUnderAgentGuardDeadlocks(t *testing.T) { |
| + f, target, captured := fenceFixture(t) |
| + store := &rootTestStore{dir: t.TempDir()} |
| + root := f.g |
| + root.Root = NewRootAuthority(store) |
| + f.obs.Claim.Holder.Account, f.obs.Claim.Holder.App = "bab", "console" |
| + in := RootInstallation{Host: target.Host, Incarnation: captured.Incarnation, Stamp: captured.Stamp, Account: "bab", App: "console", Record: "record@etag", Box: target.Box, Enrollment: target.Enrollment, Migration: 1} |
| + if err := root.EstablishRoot(t.Context(), in, allow); err != nil { |
| + t.Fatal(err) |
| + } |
| + // Match the inventory-write call inside ReconcileAdapter.Install's work.With. |
| + agent := f.g |
| + agent.FenceView = root.Root.View |
| + ctx, cancel := context.WithTimeout(t.Context(), 100*time.Millisecond) |
| + defer cancel() |
| + requested := false |
| + err := agent.With(ctx, target.Host, captured, allow, func() error { |
| + requested = true |
| + return root.EstablishRoot(ctx, in, allow) |
| + }) |
| + if !requested || !errors.Is(err, context.DeadlineExceeded) { |
| + t.Fatalf("expected the root registration to wait for the agent's held legacy lock: %v, requested=%v", err, requested) |
| + } |
| +} |
diff --git a/internal/hostguard/record.go b/internal/hostguard/record.go| index 7de7237f8..0254e13d9 100644 |
| --- a/internal/hostguard/record.go |
| +++ b/internal/hostguard/record.go |
| @@ -26,6 +26,10 @@ type Captured struct { |
| } |
| type record struct { |
| + root bool |
| + installation RootInstallation |
| + rootRetired []Captured |
| + barrier *FenceTarget |
| V int `json:"v"` |
| Host string `json:"host"` |
| Incarnation string `json:"incarnation"` |
| @@ -36,7 +40,7 @@ type record struct { |
| } |
| func (r record) validate(host string) error { |
| - if r.V != 1 || r.Host != host || !controlname.ValidHostShape(host) || r.Incarnation == "" || len(r.RetiredStamps) > 32 { |
| + if r.V != 1 || r.Host != host || !controlname.ValidHostShape(host) || r.Incarnation == "" || (!r.root && len(r.RetiredStamps) > 32) { |
| return ErrMalformedRecord |
| } |
| var previous uint64 |
diff --git a/internal/hostguard/record_other.go b/internal/hostguard/record_other.go| index f65922179..f39623769 100644 |
| --- a/internal/hostguard/record_other.go |
| +++ b/internal/hostguard/record_other.go |
| @@ -2,7 +2,10 @@ |
| package hostguard |
| -type recordDir struct{} |
| +type recordDir struct { |
| + root *RootAuthority |
| + snapshot *record |
| +} |
| func openRecordDir(string) (*recordDir, error) { return nil, ErrUnsupported } |
| func (*recordDir) close() {} |
diff --git a/internal/hostguard/record_unix.go b/internal/hostguard/record_unix.go| index ab71019c8..6fab4ae57 100644 |
| --- a/internal/hostguard/record_unix.go |
| +++ b/internal/hostguard/record_unix.go |
| @@ -16,8 +16,10 @@ import ( |
| ) |
| type recordDir struct { |
| - fd int |
| - fsync func(int) error |
| + snapshot *record |
| + root *RootAuthority |
| + fd int |
| + fsync func(int) error |
| } |
| func openDirectory(path string) (int, error) { |
| @@ -76,6 +78,15 @@ func (d *recordDir) openRecord(host string) (*os.File, error) { |
| } |
| func (d *recordDir) read(host string) (record, error) { |
| + if d.snapshot != nil { |
| + if d.snapshot.Host != host { |
| + return record{}, ErrInvalidHost |
| + } |
| + return *d.snapshot, nil |
| + } |
| + if d.root != nil { |
| + return d.root.read(host) |
| + } |
| f, err := d.openRecord(host) |
| if err != nil { |
| return record{}, err |
| @@ -89,6 +100,16 @@ func (d *recordDir) read(host string) (record, error) { |
| } |
| func (d *recordDir) sync(host string) error { |
| + if d.snapshot != nil { |
| + return ErrFenced |
| + } |
| + if d.root != nil { |
| + r, err := d.root.read(host) |
| + if err != nil { |
| + return err |
| + } |
| + return d.root.write(r) |
| + } |
| f, err := d.openRecord(host) |
| if err != nil { |
| return err |
| @@ -101,6 +122,12 @@ func (d *recordDir) sync(host string) error { |
| } |
| func (d *recordDir) write(r record) error { |
| + if d.snapshot != nil { |
| + return ErrFenced |
| + } |
| + if d.root != nil { |
| + return d.root.write(r) |
| + } |
| var nonce [16]byte |
| if _, err := rand.Read(nonce[:]); err != nil { |
| return err |
diff --git a/internal/hostguard/recoveryfence.go b/internal/hostguard/recoveryfence.go| index c4f0e6633..9ecfe29c5 100644 |
| --- a/internal/hostguard/recoveryfence.go |
| +++ b/internal/hostguard/recoveryfence.go |
| @@ -8,16 +8,19 @@ import ( |
| ) |
| func (g Guard) PendingFences(ctx context.Context) ([]FenceTarget, error) { |
| - entries, err := os.ReadDir(filepath.Join(g.DataDir, "hostguard")) |
| + if g.FenceView != nil { |
| + return nil, ErrUnsupported |
| + } |
| + names, err := g.recordNames() |
| if err != nil { |
| return nil, err |
| } |
| var out []FenceTarget |
| - for _, entry := range entries { |
| - if entry.IsDir() || !strings.HasSuffix(entry.Name(), ".json") { |
| + for _, name := range names { |
| + if !strings.HasSuffix(name, ".json") { |
| continue |
| } |
| - host := strings.TrimSuffix(entry.Name(), ".json") |
| + host := strings.TrimSuffix(name, ".json") |
| err := g.locked(ctx, host, func(d *recordDir) error { |
| r, err := d.read(host) |
| if err != nil { |
| @@ -41,7 +44,7 @@ func (g Guard) PendingFences(ctx context.Context) ([]FenceTarget, error) { |
| return out, nil |
| } |
| func (g Guard) ResumeFence(ctx context.Context, target FenceTarget) error { |
| - return g.locked(ctx, target.Host, func(d *recordDir) error { |
| + return g.fenceLocked(ctx, target.Host, Captured{Incarnation: target.Incarnation, Stamp: target.Stamp}, func(d *recordDir) error { |
| r, err := d.read(target.Host) |
| if err != nil { |
| return err |
| @@ -55,7 +58,7 @@ func (g Guard) ResumeFence(ctx context.Context, target FenceTarget) error { |
| func (g Guard) ExistingFence(ctx context.Context, want FenceTarget) (FenceTarget, error) { |
| var found FenceTarget |
| - err := g.locked(ctx, want.Host, func(d *recordDir) error { |
| + err := g.fenceLocked(ctx, want.Host, Captured{Incarnation: want.Incarnation, Stamp: want.Stamp}, func(d *recordDir) error { |
| r, err := d.read(want.Host) |
| if err != nil { |
| return err |
| @@ -68,3 +71,20 @@ func (g Guard) ExistingFence(ctx context.Context, want FenceTarget) (FenceTarget |
| }) |
| return found, err |
| } |
| + |
| +func (g Guard) recordNames() ([]string, error) { |
| + if g.Root != nil { |
| + return g.Root.store.Names() |
| + } |
| + entries, err := os.ReadDir(filepath.Join(g.DataDir, "hostguard")) |
| + if err != nil { |
| + return nil, err |
| + } |
| + var names []string |
| + for _, entry := range entries { |
| + if !entry.IsDir() { |
| + names = append(names, entry.Name()) |
| + } |
| + } |
| + return names, nil |
| +} |
diff --git a/internal/hostguard/root.go b/internal/hostguard/root.go| new file mode 100644 |
| index 000000000..0597fd949 |
| --- /dev/null |
| +++ b/internal/hostguard/root.go |
| @@ -0,0 +1,120 @@ |
| +package hostguard |
| + |
| +import ( |
| + "bytes" |
| + "context" |
| + "encoding/json" |
| + "errors" |
| + "io" |
| + "os" |
| + |
| + "github.com/carlosframework/platform/internal/hostclaim" |
| + "github.com/carlosframework/platform/internal/rootstate" |
| +) |
| + |
| +type RootInstallation struct { |
| + Host, Incarnation, Account, App, Record, Box, Enrollment string |
| + Stamp, Migration uint64 |
| +} |
| +type RootAuthority struct{ store rootstate.Store } |
| + |
| +func NewRootAuthority(store rootstate.Store) *RootAuthority { return &RootAuthority{store: store} } |
| + |
| +type rootRecord struct { |
| + Schema int |
| + Installation RootInstallation |
| + Retired []Captured |
| + Barrier *FenceTarget |
| + State record |
| +} |
| + |
| +func (a *RootAuthority) read(host string) (record, error) { |
| + if a == nil || a.store == nil || validateFileHost(host) != nil { |
| + return record{}, ErrInvalidInput |
| + } |
| + b, err := a.store.Read(host + ".json") |
| + if errors.Is(err, os.ErrNotExist) { |
| + return record{}, ErrNoRecord |
| + } |
| + if err != nil { |
| + return record{}, err |
| + } |
| + var saved rootRecord |
| + dec := json.NewDecoder(bytes.NewReader(b)) |
| + dec.DisallowUnknownFields() |
| + if dec.Decode(&saved) != nil || dec.Decode(new(any)) != io.EOF || saved.Schema != 1 || saved.Installation.Host != host || saved.Installation.Incarnation != saved.State.Incarnation || saved.Installation.Stamp != saved.State.ActiveStamp || saved.Installation.Migration == 0 { |
| + return record{}, ErrMalformedRecord |
| + } |
| + r := saved.State |
| + r.root = true |
| + r.installation = saved.Installation |
| + r.rootRetired = saved.Retired |
| + r.barrier = saved.Barrier |
| + if err := r.validate(host); err != nil { |
| + return record{}, err |
| + } |
| + return r, nil |
| +} |
| +func (a *RootAuthority) write(r record) error { |
| + if a == nil || a.store == nil || !r.root || r.validate(r.Host) != nil { |
| + return ErrMalformedRecord |
| + } |
| + r.installation.Host, r.installation.Incarnation, r.installation.Stamp = r.Host, r.Incarnation, r.ActiveStamp |
| + b, err := json.Marshal(rootRecord{Schema: 1, Installation: r.installation, Retired: r.rootRetired, Barrier: r.barrier, State: r}) |
| + if err != nil { |
| + return err |
| + } |
| + return a.store.Write(r.Host+".json", b) |
| +} |
| +func (g Guard) Capture(host string) (Captured, error) { |
| + if g.Root == nil { |
| + return Capture(g.DataDir, host) |
| + } |
| + r, err := g.Root.read(host) |
| + if err != nil { |
| + return Captured{}, err |
| + } |
| + return Captured{Incarnation: r.Incarnation, Stamp: r.ActiveStamp}, nil |
| +} |
| +func (g Guard) EstablishRoot(ctx context.Context, in RootInstallation, check func(*hostclaim.Claim) error) error { |
| + if g.Root == nil || check == nil || in.Host == "" || in.Incarnation == "" || in.Stamp == 0 || in.Migration == 0 || in.Account == "" || in.App == "" || in.Record == "" || in.Box == "" || in.Enrollment == "" { |
| + return ErrInvalidInput |
| + } |
| + return g.locked(ctx, in.Host, func(d *recordDir) error { |
| + c, err := g.fresh(ctx, in.Host) |
| + if err != nil { |
| + return err |
| + } |
| + if err := check(c); err != nil { |
| + return err |
| + } |
| + if c.Holder.Incarnation != in.Incarnation || c.Holder.Stamp != in.Stamp || c.Holder.Account != in.Account || c.Holder.App != in.App { |
| + return ErrClaimIncarnation |
| + } |
| + r, err := d.read(in.Host) |
| + if err != nil && !errors.Is(err, ErrNoRecord) { |
| + return err |
| + } |
| + if errors.Is(err, ErrNoRecord) { |
| + r = record{V: 1, Host: in.Host, Incarnation: in.Incarnation, root: true} |
| + } |
| + want := Captured{Incarnation: in.Incarnation, Stamp: in.Stamp} |
| + for _, old := range r.rootRetired { |
| + if old == want { |
| + return ErrRetiredStamp |
| + } |
| + } |
| + if r.barrier != nil { |
| + return ErrFenced |
| + } |
| + if r.Incarnation == in.Incarnation && r.ActiveStamp > in.Stamp { |
| + return ErrOldStamp |
| + } |
| + if r.Incarnation != in.Incarnation || r.ActiveStamp != in.Stamp { |
| + r.retire() |
| + r.RetiredStamps = nil |
| + } |
| + r.Incarnation, r.ActiveStamp, r.installation = in.Incarnation, in.Stamp, in |
| + return d.write(r) |
| + }) |
| +} |
diff --git a/internal/hostguard/rootview.go b/internal/hostguard/rootview.go| new file mode 100644 |
| index 000000000..de1c027ef |
| --- /dev/null |
| +++ b/internal/hostguard/rootview.go |
| @@ -0,0 +1,89 @@ |
| +package hostguard |
| + |
| +import "context" |
| + |
| +type FenceView struct { |
| + Schema int |
| + Host string |
| + Captured Captured |
| + Installation RootInstallation |
| + Retired []Captured |
| + Pending *FenceTarget |
| + Receipts []FenceReceipt |
| + Barrier *FenceTarget |
| +} |
| +type FenceViewProvider func(context.Context, string, Captured) (FenceView, error) |
| + |
| +func (a *RootAuthority) View(ctx context.Context, host string, captured Captured) (FenceView, error) { |
| + if err := ctx.Err(); err != nil { |
| + return FenceView{}, err |
| + } |
| + r, err := a.read(host) |
| + if err != nil { |
| + return FenceView{}, err |
| + } |
| + known := r.Incarnation == captured.Incarnation && r.ActiveStamp == captured.Stamp && captured.Stamp != 0 |
| + for _, old := range r.rootRetired { |
| + if old == captured { |
| + known = true |
| + } |
| + } |
| + if !known { |
| + return FenceView{}, ErrStampMismatch |
| + } |
| + return FenceView{Schema: 1, Host: host, Captured: captured, Installation: r.installation, Retired: r.rootRetired, Pending: r.ActiveFence, Receipts: r.FenceJournal, Barrier: r.barrier}, nil |
| +} |
| +func (v FenceView) validate(host string, c Captured) error { |
| + if v.Schema != 1 || v.Host != host || v.Installation.Host != host || v.Installation.Migration == 0 || v.Captured != c { |
| + return ErrMalformedRecord |
| + } |
| + if v.Pending != nil && v.Pending.validate(host) != nil { |
| + return ErrMalformedRecord |
| + } |
| + if v.Barrier != nil && v.Barrier.validate(host) != nil { |
| + return ErrMalformedRecord |
| + } |
| + for _, r := range v.Receipts { |
| + if r.validate(host) != nil { |
| + return ErrMalformedRecord |
| + } |
| + } |
| + return nil |
| +} |
| +func (v FenceView) admit(host string, c Captured, claimSession uint64) error { |
| + if err := v.validate(host, c); err != nil { |
| + return err |
| + } |
| + if v.Installation.Incarnation != c.Incarnation || v.Installation.Stamp != c.Stamp || c.Stamp == 0 { |
| + return ErrInactive |
| + } |
| + if v.Barrier != nil { |
| + return ErrFenced |
| + } |
| + denied := func(t FenceTarget) bool { |
| + return t.Host == host && t.Incarnation == c.Incarnation && t.Stamp == c.Stamp && t.Session == claimSession |
| + } |
| + if v.Pending != nil && denied(*v.Pending) { |
| + return ErrFenced |
| + } |
| + for _, r := range v.Receipts { |
| + if denied(r.Target) { |
| + return ErrFenced |
| + } |
| + } |
| + return nil |
| +} |
| +func (g Guard) fenceLocked(ctx context.Context, host string, c Captured, fn func(*recordDir) error) error { |
| + if g.FenceView == nil { |
| + return g.locked(ctx, host, fn) |
| + } |
| + view, err := g.FenceView(ctx, host, c) |
| + if err != nil { |
| + return err |
| + } |
| + if err := view.validate(host, c); err != nil { |
| + return err |
| + } |
| + r := record{V: 1, Host: host, Incarnation: view.Installation.Incarnation, ActiveStamp: view.Installation.Stamp, ActiveFence: view.Pending, FenceJournal: view.Receipts, root: true, installation: view.Installation, rootRetired: view.Retired, barrier: view.Barrier} |
| + return fn(&recordDir{snapshot: &r}) |
| +} |
diff --git a/internal/hostguard/writerstop.go b/internal/hostguard/writerstop.go| index 86d0c32ec..59ceb0891 100644 |
| --- a/internal/hostguard/writerstop.go |
| +++ b/internal/hostguard/writerstop.go |
| @@ -95,7 +95,7 @@ func (g Guard) CompletedWriterStops(ctx context.Context, host string, captured C |
| return nil, ErrInvalidInput |
| } |
| var out []FenceReceipt |
| - err := g.locked(ctx, host, func(d *recordDir) error { |
| + err := g.fenceLocked(ctx, host, captured, func(d *recordDir) error { |
| r, err := d.read(host) |
| if err != nil { |
| return err |
diff --git a/internal/provision/plan9_b8_census_test.go b/internal/provision/plan9_b8_census_test.go| index a290e32e3..18846db42 100644 |
| --- a/internal/provision/plan9_b8_census_test.go |
| +++ b/internal/provision/plan9_b8_census_test.go |
| @@ -472,6 +472,10 @@ func Reviewed(ctx context.Context, s Store, k string) error { |
| // BEGIN REVIEWED INVENTORY |
| var plan9B8CensusInventory = []censusEntry{ |
| + {"internal/host/proofbinding.go", "proofBindingSource.binding", "key-site", "key:RecordKey", "6847b909af286f4174197073a630c835edfeb1bd3aef864b2d19d36ae4dbb5c0"}, |
| + {"internal/host/proofbinding.go", "type proofBinding", "key-site", "key:RecordKey", "b5455029fb4bb12ca54c0c1f2594fb1633a7717e7796a8ba5ec37794954f8e26"}, |
| + {"internal/host/rootroute.go", "proofBindingSource.claimIdentity", "key-site", "key:RecordKey", "ad6e0fc3f2cd360473a5d0a2267d2dc7944b45d80947613f791db8d573f8fee6"}, |
| + {"internal/provision/rootrecord.go", "FindRecordByHost", "key-site", "key:ParseRecordKey,lit:instances", "10c8254cf896be034334d634151d169760a909ac1ba4c1fc334785e86e4b84d4"}, |
| // Read-only discovery of signed loss records; no record write or lifecycle authority. |
| {"internal/console/claim_loss_pages.go", "Server.pageClaimLossRecord", "key-site", "key:LossRecordKey", "d21e2450f540edf6645c457eb08c72f470969f5ea4fcbc597157bb7dbeb27287"}, |
| {"internal/hostbox/parkleases.go", "ReplicaParkLeaseSet", "key-site", "lit:instances", "0ca8d9b743402fda9d15cc215464423b35e21c1c2b47f91294d76188b7d43fb6"}, |
diff --git a/internal/provision/rootrecord.go b/internal/provision/rootrecord.go| new file mode 100644 |
| index 000000000..6e7e30e21 |
| --- /dev/null |
| +++ b/internal/provision/rootrecord.go |
| @@ -0,0 +1,50 @@ |
| +package provision |
| + |
| +import ( |
| + "context" |
| + "os" |
| + |
| + "github.com/carlosframework/platform/internal/release" |
| +) |
| + |
| +func FindRecordByHost(ctx context.Context, s release.Store, host string) (*Record, string, error) { |
| + known, partial, err := knownAccounts(ctx, s) |
| + if err != nil { |
| + return nil, "", err |
| + } |
| + if partial { |
| + return nil, "", os.ErrPermission |
| + } |
| + keys, partial, err := listAppSubtree(ctx, s, known, "instances/") |
| + if err != nil { |
| + return nil, "", err |
| + } |
| + if partial { |
| + return nil, "", os.ErrPermission |
| + } |
| + var account, app string |
| + for _, key := range keys { |
| + a, p, h, ok := ParseRecordKey(key, known) |
| + if !ok || h != host { |
| + continue |
| + } |
| + if account != "" { |
| + return nil, "", os.ErrPermission |
| + } |
| + account, app = a, p |
| + } |
| + if account == "" { |
| + return nil, "", os.ErrNotExist |
| + } |
| + rec, etag, err := GetRecordETag(ctx, s, account, app, host) |
| + if err != nil { |
| + return nil, "", err |
| + } |
| + if etag == "" || rec.Schema != 1 || !release.ValidAccount(account) || !release.ValidApp(app) { |
| + return nil, "", os.ErrPermission |
| + } |
| + if err := ValidateShape(*rec); err != nil { |
| + return nil, "", err |
| + } |
| + return rec, etag, nil |
| +} |
diff --git a/internal/provision/rootrecord_test.go b/internal/provision/rootrecord_test.go| new file mode 100644 |
| index 000000000..cc28461fe |
| --- /dev/null |
| +++ b/internal/provision/rootrecord_test.go |
| @@ -0,0 +1,52 @@ |
| +package provision |
| + |
| +import ( |
| + "context" |
| + "encoding/json" |
| + "testing" |
| + |
| + "github.com/carlosframework/platform/internal/release" |
| +) |
| + |
| +func TestPlan14ProofRecordLookup(t *testing.T) { |
| + ctx := context.Background() |
| + s := release.NewDirStore(t.TempDir()) |
| + rec := Record{Schema: 1, Host: "console.example.com", Account: "bab", App: "console", Addr: "127.0.0.1:4000", Unit: "console.service"} |
| + raw, err := json.Marshal(rec) |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + key := RecordKey(rec.Account, rec.App, rec.Host) |
| + etag, err := s.Put(ctx, key, raw, "") |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + got, tag, err := FindRecordByHost(ctx, s, rec.Host) |
| + if err != nil || got.Account != rec.Account || got.App != rec.App || tag != etag { |
| + t.Fatalf("lookup: %+v, %q, %v", got, tag, err) |
| + } |
| + if _, _, err := FindRecordByHost(ctx, s, "missing.example.com"); err == nil { |
| + t.Fatal("missing record bound") |
| + } |
| + duplicate := RecordKey("bab", "other", rec.Host) |
| + if _, err := s.Put(ctx, duplicate, []byte("malformed"), ""); err != nil { |
| + t.Fatal(err) |
| + } |
| + if _, _, err := FindRecordByHost(ctx, s, rec.Host); err == nil { |
| + t.Fatal("malformed competing key was ignored") |
| + } |
| + if err := s.Delete(ctx, duplicate); err != nil { |
| + t.Fatal(err) |
| + } |
| + rec.Account = "bap" |
| + raw, err = json.Marshal(rec) |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + if _, err := s.Put(ctx, key, raw, etag); err != nil { |
| + t.Fatal(err) |
| + } |
| + if _, _, err := FindRecordByHost(ctx, s, rec.Host); err == nil { |
| + t.Fatal("key/body mismatch bound") |
| + } |
| +} |
diff --git a/internal/provision/testdata/plan14-root-reads.tsv b/internal/provision/testdata/plan14-root-reads.tsv| index 7f37aa64a..a4702fdbb 100644 |
| --- a/internal/provision/testdata/plan14-root-reads.tsv |
| +++ b/internal/provision/testdata/plan14-root-reads.tsv |
| @@ -51,17 +51,17 @@ |
| @coverage internal/geo 5886c699c4902a1162ee9640471b8d5e3d2fc18f1dd0d43b70b0f8a459d66b02 |
| @coverage internal/geo/data 27dcfc47493fe7ae61cffa81a700a9ac09e8253fdc88e20da486cc651805af68 |
| @coverage internal/geoip 14994dadea58daebdb9772d8591e916353803fb495de7443a4167682c84cbd37 |
| -@coverage internal/host fe2fb96c1f010207c29b476925325e7c823f8d5ce3a6b845fccebb6b57388906 |
| +@coverage internal/host ce0e6ccf69d5650d42cc5d892247d23e8bd58e81f9db6928d4e8ed6ca8821a72 |
| @coverage internal/host/units d45d8d20dc5c3fc7d00e6106d2a0f3475ef04c6cd988fa0b0717fa04b0dd119f |
| @coverage internal/hostactors e5e2e7daed861dba7b57151a6ba023d6e545dfc53d8c6f894d3217cf320bcaf7 |
| -@coverage internal/hostbox a75f9f3e7679d4048a3b056ac492a4ac29ce6789fe9dbca9aa28cf5b81fd8205 |
| +@coverage internal/hostbox 1d39f4bed60bb3e36ec7d9e9420f9caa8e183ead87b7a2177d496756a7809c9e |
| @coverage internal/hostclaim 242add93e1bc7c1d409710db0b0bbb4cccb3dd0ee85e89f217549f1aacfb41ce |
| @coverage internal/hostevidence d85c7894b2c959aaac24176846e0a5b6a28861003d1207d830827fd90b6c4d78 |
| @coverage internal/hostexec 94f7eae800d6f985552854757c6f011501c17696eeb564af1c15e2dc145d8c73 |
| @coverage internal/hostfence 2705d961a8ffd650e1096c9a87085eef267c2de9c223c1993f249b7711a6c2b2 |
| @coverage internal/hostfiles 08ab5139a8b38d1baee773a7a1056e0b6fb9fbb4e3b5cbb2be0e968cc8bea86e |
| @coverage internal/hostfloor 5bf88a24c5671c612befb8bd1f8b719aa2037e812a34488853b906a95d1650c7 |
| -@coverage internal/hostguard cbd3d4d8514845027ea766c77d8ec41f7d42489442bcae55707c5f4ab4415a09 |
| +@coverage internal/hostguard 229f8f4d704e84ae611f7b655f3258354e5b636d449c7cb7841962f5f14f487f |
| @coverage internal/hostops 36b1499a44f3367ebbcc4cb4f4129978af3cc1a6e2b9e27e55ae18f17afec25c |
| @coverage internal/hostops/fencerelease b8cf08bdb7dfdc622f4f91d089570e0994141e03beef68a070202d985c423513 |
| @coverage internal/hostops/internal/fencerelease 36ec37feda681a0caf19d6b9452f27eb6a45fd8323154920c8fd29fafbac68ca |
| @@ -89,16 +89,16 @@ |
| @coverage internal/ops db33a5703fc8621e440810eb6917f45625bfb306e4378760c39e83c5f7a384cd |
| @coverage internal/park 57a5bba1298f6ff0e4016cec186dfdd39a7de86994b79de2b2aa479d0c3cc424 |
| @coverage internal/privatefile 5cc645be05efdd9ed8573e9ef355138c0f63c273d4aa1e47a6307c629bb58694 |
| -@coverage internal/provision a09ef135a84446f60b251141e2ba6cd2af096e51a459e2d5c896453b8e8c9a91 |
| +@coverage internal/provision a98debe6ab33a3693e8e56f15c1b75b604a4e4ba4574a99cd34a0af5dbd7fa90 |
| @coverage internal/publicrelease 5eb4e6cab00a189bddee90afe761dccda583e773cf276a872a66e2b195f7a7dc |
| @coverage internal/regionname e718465fc377a1b94dcba0d3dd22916633763b004f850b83fc4d1924e21871d0 |
| @coverage internal/registry 8545e5941aa41013e8c3cfd0c50a8a66a75f70133523cf9398c3c631c1c1d7e5 |
| @coverage internal/relay 56937dacffe2b0752e1177e742cc8f8b50cadcbaeaa4fc858a32cef1d9325f1d |
| -@coverage internal/release 0e7340191d2e2af8708673e9bbb6cf13d5eb02927917cf225ca084ab834815bc |
| +@coverage internal/release 5c4babcd09ba18de65e30958a3746428e4b24b8371416bd446bb510a7e3dba57 |
| @coverage internal/rollout 061226d57365ebe663b920a533146130bb6ec63a3a01a5c25b78ae9536da866b |
| @coverage internal/rolloutwords e3bdc32c3181f89f69e45b8ebfbce32c69f85258d496d54caa90cfc8ac883abf |
| @coverage internal/rootaws ac1183d58dcc919329526006fc5e6cddf4f1e4882d6657428e35d50b4c60b461 |
| -@coverage internal/rootstate 92dfdd5119f03dde4fbcbc4ba8833954c9b2a900056bd8138af3c7307ae35725 |
| +@coverage internal/rootstate 2e2ba85261ed01dc0bc041082702b9f6f3fce0dfad61c30f2bd167750175cbc2 |
| @coverage internal/s3compat 63bc6cff61150aaa3dc5198178ff48e2478d60d977c05112eabb290695ce5955 |
| @coverage internal/safefile 8e7a73fe7fe07f9ebc5510bbbe71854e888555bcde14110c0b4deb078b02fb27 |
| @coverage internal/schedule 2dcfa9601b1b34dc172735900a026467f6d2436126442509e33ed7e44ea0b319 |
| @@ -1009,13 +1009,12 @@ internal/host/adopt_advice.go nftGateUIDs diagnostic H3 no-effect file:Command@5 |
| internal/host/adopt_advice.go unitsAreActive diagnostic H3 no-effect file:Command@176 4be247ee29a60420b2ec0c41b036482c9418ac38879ad608710e32a9d84e3a88 |
| internal/host/adopt_advice.go warnBackingMismatch diagnostic H3 no-effect registry:Backing@210,registry:Host@297,registry:Kind@170,registry:Platform@195,whole:Route@156 7f451d63f30ab365a1c4cf9975010a1f4cd0ddda4c9d7c79fdf64299fa1fe1f3 |
| internal/host/adoptlock.go lockExclusive authority C,E1,J4 root-input;kept=C,E1;scope=protected-decision-only file:OpenFile@72 257b9a621df9e85a330de958d2b8d78673a1fde01b325655ae8042e1af871490 |
| -internal/host/agent.go checkedUnitRestart authority B3,J1,J4 route-and-input file:EvalSymlinks@750,registry:Host@1510,registry:Host@1609,registry:Host@310,registry:Host@411,registry:Host@571,registry:Unit@1519,registry:Unit@1618,registry:Unit@214,registry:Unit@319,registry:Unit@358,registry:Unit@545,registry:Unit@605,registry:Unit@681,registry:Unit@821,registry:Unit@861,registry:Unit@965,whole:Route@1131,whole:Route@1787 9f10a97ed2d0e290be68b867fde792c761df5e66d76b35c87a7951b032e4bf8f |
| internal/host/agent.go configGenLoop legacy accepted-legacy accepted-legacy;rev3=J1;basis=root-route whole:Route@1301 5d8aa5e26ea69142eba3eabd0ccb52ab853ea33bf006179595ad2fb8b4835bbb |
| internal/host/agent.go convergeAdopted legacy accepted-legacy accepted-legacy;rev3=J1;basis=root-route registry:Version@481,whole:Route@468,whole:Route@520 131ca488f8904f4fd0b303cb0f2f921a792d0450371476afb7525fe55be8decf |
| internal/host/agent.go countExpiredCanaries legacy accepted-legacy accepted-legacy;rev3=J1;basis=root-route registry:App@500,registry:Channel@508,registry:Platform@270,whole:Route@256,whole:Route@300,whole:Route@495 bb9ffaec6dcfa0df2bffa45100a7c40ec48b28cbb9c24d71aadb0cfe8f91037b |
| internal/host/agent.go dataDir legacy accepted-legacy accepted-legacy;rev3=J4;basis=root-input data-literal@43,data-literal@95 00e671cc828ca4179bf7e0cea464be8a697a2a6b2f229bb410df3d3a4fbf5086 |
| internal/host/agent.go domainsTickConfig legacy accepted-legacy accepted-legacy;rev3=J4;basis=root-input data-literal@469,data-literal@510 48bcacda59779124df3bfaa743c57c78674d84ff23d6f6492b5dd03d4daa8a13 |
| -internal/host/agent.go fastConvergeLoop legacy accepted-legacy accepted-legacy;rev3=J1;basis=root-route whole:Route@8292,whole:Route@8855 9a5d73c716e621765fd71566eba0d62117618efeb174bb5a809a23f26db4e448 |
| +internal/host/agent.go fastConvergeLoop authority B3 kept=B3;scope=agent-restart-helper-selection-only;legacy-loop-behavior-preserved whole:Route@8302,whole:Route@8865 79dcfc65a593861fa5927d257db8093f56bf0733941f3b8891983c5ebf191577 |
| internal/host/agent.go fastDeps.fastMembers legacy accepted-legacy accepted-legacy;rev3=J1;basis=root-route registry:Host@1222,registry:Host@1287,registry:Host@1332,registry:Host@1778,registry:Host@1817,registry:Host@1876,registry:Host@1921,registry:Host@2132,registry:Host@2256,registry:Host@2355,registry:Host@2399,registry:Host@507,whole:Route@1273,whole:Route@188,whole:Route@495 b08fbffe851b1976996a60780d2685dfc525894d724cc24d3e95f93095fc538b |
| internal/host/agent.go fastTick legacy accepted-legacy accepted-legacy;rev3=J1;basis=root-route registry:Host@3643 c5a53d97531f6059b174005b8214333642019dee25436d130289c46790efb295 |
| internal/host/agent.go genPollPass legacy accepted-legacy accepted-legacy;rev3=J1;basis=root-route whole:Route@399,whole:Route@639,whole:Route@685 e53a6517dcd1cb54c3a5b0ced09b29281cf245d992fd6edd5cc6fd98f339eb87 |
| @@ -1026,8 +1025,7 @@ internal/host/agent.go makeRestartRoute authority B3,J1 root-route;kept=B3;scope |
| internal/host/agent.go makeStartRoute authority B3,J1 root-route;kept=B3;scope=protected-decision-only registry:Hibernate@198,registry:Unit@214,registry:Unit@267,whole:Route@185 6cf9d7010bdda2f7192bf19c0e443e9b4e3d3a93b65d9ec78236260cb2612c65 |
| internal/host/agent.go nestedGate legacy accepted-legacy accepted-legacy;rev3=J4;basis=root-input data-literal@223 01fcf4126321fb7bf0f95f13e0d288cbb4adebc006af034784bbe5545da6240a |
| internal/host/agent.go provisionTickConfig legacy accepted-legacy accepted-legacy;rev3=J4;basis=root-input data-literal@309,data-literal@350 6aff92fb17d871a74f613a0111242a4156e35f792f736fd35ee2ffc853ffbc41 |
| -internal/host/agent.go rollbackUnitRestart authority B3,J1,J4 route-and-input file:EvalSymlinks@510,registry:Unit@1153,registry:Unit@227,registry:Unit@285,registry:Unit@322,registry:Unit@412,registry:Unit@454,registry:Unit@629,whole:Route@214,whole:Route@814 124eed950b5917aef73bb2419a300c50f84951a1a848ab7501b8989994de654b |
| -internal/host/agent.go runAdoptWithEnsureSystemUnits authority B3,C,D,E1,E2,E3,E4,F,J1 root-route;kept=B3,C,D,E1,E2,E3,E4,F;scope=protected-decision-only whole:Route@10115,whole:Route@16287,whole:Route@16323,whole:Route@7543 1fafff7c0f3700a6094bad44a9762554246fe88ac8fbb5d7ac1378f80dffda17 |
| +internal/host/agent.go runAdoptWithEnsureSystemUnits authority B3,C,D,E1,E2,E3,E4,F,J1 main adoption body under outer root release serialization; existing authorities retained; draft incomplete file:Open@220 94eff5977a4f7dc357e2b6aab6d00afc569be26712693c58cef990718037003b |
| internal/host/agent.go runAgent authority B,C,D,F,J1,J4 route-and-input;kept=B,C,D,F;scope=protected-decision-only file:Exec@12092,file:Open@569,registry:App@8483,whole:Route@19389,whole:Route@8475 c132bf08e6e6d1d0596391a7eb2f361b7afcb47923c82d753b52f75c002fe260 |
| internal/host/agent.go runWorkerBootPass legacy accepted-legacy accepted-legacy;rev3=J1;basis=root-route registry:Host@453,whole:Route@110,whole:Route@126,whole:Route@248,whole:Route@266,whole:Route@380 b6098cea4024597d0b6590dc0d02425fd667b73a955f3beaf0c243323f011e83 |
| internal/host/agent.go stampedUnitNames legacy accepted-legacy accepted-legacy;rev3=J1;basis=root-route registry:Unit@273,registry:Unit@295,registry:Unit@329,registry:Unit@365,whole:Route@212,whole:Route@259 cc329e3e91ec6aa6679f395df3fce6d3a6f6834d676a4914653dadaf6f1e181b |
| @@ -1077,7 +1075,7 @@ internal/host/claimbackfill.go newClaimBackfill authority C,G,J4 root-input;kept |
| internal/host/claimcensus.go claimUnitCensus authority C,G,J4 root-input;kept=C,G;scope=protected-decision-only file:ReadFile@569 82ed33ee9375b7d41d68637cedece421876874f535ebcf801b812f9e3b1bd10e |
| internal/host/claimcleanup_identity.go claimCleanupEnrollment.configureSigner authority C,G,J4 root-input;kept=C,G;scope=protected-decision-only file:Read@268 2a4b3bd51f544a5bdd11033389f9d6585785567555d3c12695e335799c756cd6 |
| internal/host/claimcleanup_identity.go verifyClaimCleanupEnrollment authority C,G,J4 root-input;kept=C,G;scope=protected-decision-only file:Read@173 7a48c65538d68dd362442152e099d8542890ba3241afbbbff01a9c153379ca14 |
| -internal/host/claimconstruction.go OpenProductionClaimAssembly authority C,G,J1,J4 route-and-input;kept=C,G;scope=protected-decision-only file:Read@4059,file:Read@8354,file:ReadFile@3603,registry:Host@8381,whole:Route@8469 a7dfd3a63b259d471e1dfc01667fa6cf5cc3c055c7400e6c931c0aa346093099 |
| +internal/host/claimconstruction.go OpenProductionClaimAssembly authority C,G,J1,J4 root installation/fence backend and lock-free agent projection; establishment blocked; draft incomplete file:Open@4480,file:Read@4059,file:Read@8762,file:ReadFile@3603,registry:Host@8789,whole:Route@8877 3505005d000744ff292e7dd70cf4323a5837793d9c00b0a08f52e4997cbfcc62 |
| internal/host/claimconstruction.go claimConstructionProcessStart authority C,G,J4 root-input;kept=C,G;scope=protected-decision-only file:ReadFile@68 0681ee9abf64de4aca4b4ba035e1732ed2560ca41564eea6cb854a76bee99f45 |
| internal/host/claimconstruction.go refusingRootWake authority C,G,J4 root-input;kept=C,G;scope=protected-decision-only file:Prepare@641,file:Read@123,file:Read@202 20f7779f6e5cb16b383707ff2c16f3ca0318673e48d20245aed65bfba52ba453 |
| internal/host/claimhandover_crossbox.go claimCrossBoxHandover.After authority C,G,J4 root-input;kept=C,G;scope=protected-decision-only file:Exec@2009,file:Exec@716 a0fb9c27f5be5fce8204611f6421522529754474cf52f980582f7afab141d3e2 |
| @@ -1097,7 +1095,7 @@ internal/host/claimjournal_migration_linux.go quiesceLegacyClaimJournals authori |
| internal/host/claimjournal_migration_linux.go recoverLegacyClaimEdgeMask authority C,G,J4 root-input;kept=C,G;scope=protected-decision-only file:ReadFile@215,file:ReadFile@77 4c7bc64a8133109837f42d0a7ffc687e6f6b357b3eaf0951bf370a49984d2a21 |
| internal/host/claimjournal_migration_linux.go var claimSystemctl authority C,G,J4 root-input;kept=C,G;scope=protected-decision-only file:CommandContext@89 b86a40d4a1d0bcd00d410afe2aac4d2de0c409dffb50b3fc8313cf65faef9b76 |
| internal/host/claimjournal_proof.go legacyJournalProof authority C,G,J4 root-input;kept=C,G;scope=protected-decision-only file:Read@817 c5a5935a1a60cc5ab6a0a2754e8f1d5f3c3cfb9d10812062c88826c32633cc9a |
| -internal/host/claimjournal_transport.go ClaimRootHTTP.serveJournal authority C,G,J1,J4 route-and-input;kept=C,G;scope=protected-decision-only file:Read@2829,whole:Route@3983 08e461c315ac2a13b4a1f588b6188aabe7e1c6cc6fcd8364758807eec53da88d |
| +internal/host/claimjournal_transport.go ClaimRootHTTP.serveJournal authority C,G,J1,J4 root installation/fence backend and lock-free agent projection; establishment blocked; draft incomplete file:Read@2896,whole:Route@4050 735c1af3378975ac37c45af29f67a21e409fff8132c0b0a0e2f9f60a6fe7ab4c |
| internal/host/claimlaunch_execpermit.go ClaimLaunchRoot.execPermit authority C,G,J4 root-input;kept=C,G;scope=protected-decision-only file:Read@157 230b6384c7630684301d13c193b7326ae2506b38bd37f0b41edaa1ba22a89d8d |
| internal/host/claimlaunch_kernel_linux.go claimLaunchProcessIdentity authority C,G,J4 root-input;kept=C,G;scope=protected-decision-only file:ReadFile@94 fffe76e36b957979be55a4a713e2d7caefa67d807d91b33f7f579971f7e0a781 |
| internal/host/claimlaunch_kernel_linux.go linuxClaimLaunchKernel.Observe authority C,G,J4 root-input;kept=C,G;scope=protected-decision-only file:Lstat@430,file:Stat@1515,file:Stat@1611,file:Stat@1710 8b495df60193fd107461f8b62b7f08824635865d962a8e87aea7452d3678c128 |
| @@ -1130,7 +1128,7 @@ internal/host/claimparticipation.go productionClaimLocal.prepareWith authority D |
| internal/host/claimparticipation_stale.go participationCleanupRead.Read authority D,J4 root-input file:Read@34 d133a86593d56eafb9817341f4d3f1c148e593849633ea9cb86dd368313a6c9c |
| internal/host/claimparticipation_stale.go productionClaimLocal.clearUntracked authority D,J4 root-input file:Read@989 65d29ab57365fb515d688847c8f8e56941e9a550d83b14b8f622e75508273b9d |
| internal/host/claimproduction.go ProductionClaimOwnerProvider.OpenClaimOwners authority C,G,J4 root-input;kept=C,G;scope=protected-decision-only file:Open@4191 53a344a95133f73cf730a62e3f44ce7d9a9a25f0a6bcd8b9719ab0f1853f3287 |
| -internal/host/claimproduction.go claimAdminTokens.With authority C,G,J4 root-input;kept=C,G;scope=protected-decision-only file:Read@317 fc2d78a9c06e996e365baa1ac02584784459f8280e2e7aa4913eff0b15d202bc |
| +internal/host/claimproduction.go claimAdminTokens.With authority C,G,J4,J1 root installation/fence backend and lock-free agent projection; establishment blocked; draft incomplete file:Read@317 5fe40c6b6588e7a09fa0b4b39c1c7018efaf1eb7117712db7cc376bd13189182 |
| internal/host/claimproduction.go claimInputPolicy authority C,G,J1 root-route;kept=C,G;scope=protected-decision-only whole:Route@1096,whole:Route@1161,whole:Route@618,whole:Route@979 46ee6b133f701b6e46e55f7d9a3df6dd0254a7a2b0c2e150cbc85a899350f4f2 |
| internal/host/claimproduction.go claimServingLaunchSource authority C,G,J1 root-route;kept=C,G;scope=protected-decision-only registry:Host@368,whole:Route@285,whole:Route@492,whole:Route@810 6319f598905301a59898aa40443a13bb7bae0f3126b204050b10d581bd1266c0 |
| internal/host/claimproduction.go declinedClaimGate authority C,G,J1 root-route;kept=C,G;scope=protected-decision-only registry:Host@187,whole:Route@348 6e5d03161f0d020aa22662ee345571a3373fd84105ebf184cc20deff3f55134c |
| @@ -1268,7 +1266,7 @@ internal/host/movescut.go sealStopped legacy accepted-legacy accepted-legacy;rev |
| internal/host/movesfinal.go moveJournalPresent legacy accepted-legacy accepted-legacy;rev3=J4;basis=root-input file:Lstat@159 b11f2c352d97cc64441db8901ee7a390a7db374b3bd8a8f7e3b58939b09205f0 |
| internal/host/movesfinal.go movesWorker.sweepFinalJournals legacy accepted-legacy accepted-legacy;rev3=J4;basis=root-input file:ReadDir@131,file:ReadDir@604 71cb4821b87f035e8ae6d372ebf97fca5b53b9aa4008757cc6e2acfa556a252b |
| internal/host/movesfinal.go sourceFinal.installation legacy accepted-legacy accepted-legacy;rev3=J4;basis=root-input file:Read@135 fb379244205bc7f49a573e6b99727b6a2142d6a7b94ec55555c36be0b03b2f65 |
| -internal/host/movesfinal.go sourceFinal.parkLocal legacy accepted-legacy accepted-legacy;rev3=J4;basis=root-input file:Exec@455 ddf6984ed7366a0ab1d97996d2026bcd040fc5630709bd82d027a23f48bd738e |
| +internal/host/movesfinal.go sourceFinal.parkLocal authority C,J1,J4 root installation/fence backend and lock-free agent projection; establishment blocked; draft incomplete file:Exec@455 a82ea707c77ca8aeba8af079250954aa07581be809624f86f892936c9dbecb42 |
| internal/host/movesfinal.go sourceFinal.run legacy accepted-legacy accepted-legacy;rev3=J1,J4;basis=route-and-input file:Read@1107,registry:DBPath@541,registry:DBPath@560,registry:DBPath@700,registry:HoldOp@319,whole:Route@306 82d7de89bb9cd36c9dfdb1d58a3f6b2feefeaca47cc8b7ea85043bc34e8fe041 |
| internal/host/movesfloor.go judgePrevWith authority F,J4 route-and-input file:Stat@315,whole:Route@1212 bd541199bdf85a56014e013c2c80e58edcee4506489f92a92899f7af84670924 |
| internal/host/movesfloor.go readBeneathLine authority F,J4 root-input file:Stat@202 e971fcdd0330a2b634602711ca6a32580eb0494355bf44f7892a15204dd96440 |
| @@ -1411,15 +1409,15 @@ internal/host/probes.go systemUpdatePathActive legacy accepted-legacy accepted-l |
| internal/host/probes.go unitBackedSockets legacy accepted-legacy accepted-legacy;rev3=J1;basis=root-route registry:Backing@125,registry:Socket@168,whole:Route@111 708cb874cf691a31d84a8f1eda7672cf8f76fd0656abbad9bcdfad1e051d41ab |
| internal/host/proofask.go mintAsker.stagedSHA authority B3,J4 root-input;kept=B3;scope=protected-decision-only file:Stat@441,file:Stat@75 44cc98cf53c741fe20c4dc0cf102f71948bf5530eb78965ce262c44746714fac |
| internal/host/proofask.go mintAsker.tick authority B3,J1,J4 route-and-input;kept=B3;scope=protected-decision-only file:EvalSymlinks@983,registry:App@1203,registry:App@1573,registry:App@2015,registry:App@829,registry:Host@1032,registry:Host@1302,registry:Host@1470,registry:Host@1817,registry:Host@1861,registry:Host@2207,registry:Host@2350,registry:Host@2383,registry:Host@879,registry:Host@951,registry:Unit@1337,registry:Unit@800,whole:Route@1175,whole:Route@760,whole:Route@793 857e70b24c0b245f95945e88e39f6141be25c8053ef1012cb03e1294107f66e0 |
| -internal/host/proofobserve.go productionProofObserver authority B3,J1,J4 root-route whole:Route@222 8ddfc09d2ae152be2646e2cdf3838118d1616725ec05cd12505f041ef9ba7ab5 |
| -internal/host/proofobserve.go proofObserver.observe authority B3,J1,J4 root-route registry:Host@128,registry:Host@1717,registry:Unit@1459,registry:Unit@164,registry:Unit@1664,registry:Unit@245,registry:Unit@356,registry:Unit@496,registry:Unit@642,registry:Unit@791,whole:Route@1309 ec45edb6123e82a3bc6fae089ddb1350c3762b516d12eda1f6ffd417d91248ab |
| -internal/host/proofobserve.go recordSettled authority B3,J1,J4 root-route registry:App@1266,registry:Host@1114,registry:Host@1188,registry:Host@1432,registry:Host@361,registry:Host@433,registry:Host@557,registry:Host@753,registry:Host@921,registry:Unit@192,registry:Unit@370,registry:Unit@566,whole:Route@1261 e30fb3508a13efb76e3c9c103844674dbde2b7c2e1b2939d569782e529b6f0ea |
| +internal/host/proofobserve.go productionProofObserver authority B3,J1,J4 root-route whole:Route@412,whole:Route@492 d6f13ea3361b311d24a7ae794526bed5ce1753482e6532295e653fe4b01c3701 |
| +internal/host/proofobserve.go proofObserver.observe authority B3,J1,J4 root-route registry:Host@2668,registry:Host@381,registry:Unit@1165,registry:Unit@1951,registry:Unit@2156,registry:Unit@417,registry:Unit@498,registry:Unit@609,registry:Unit@749,registry:Unit@895,whole:Route@1750,whole:Route@206,whole:Route@2217,whole:Route@326,whole:Route@339 f9ed26c9b7aa1d4880c392f3b8b6ca3af0dcbb6e16ce7b51bd78ee5111a20f6d |
| +internal/host/proofobserve.go recordSettled authority B3,J1,J4 root-route 7934e49134d055e4a032e5fd04f51f85a1c9f0205b51872b73766dbfc07e3cb0 |
| internal/host/proofroot.go proofRoot.markRequired diagnostic H3 no-effect file:Stat@88 ae9121c0dde0beee0fcff30dd64f653fc66a163b5b06d92652729761238053c9 |
| -internal/host/proofroot.go proofRoot.mint authority B3,J1,J4 root-route registry:App@1118,registry:App@1740,registry:App@1855,registry:App@2059,registry:App@714,registry:Host@1039,registry:Host@1379,registry:Host@1846,registry:Host@1969,registry:Host@249,registry:Host@402,registry:Host@915 c5c5eccf16a33f427012a0ae7042373ef115dffcc28b83af828b21c6c0e44978 |
| -internal/host/proofroot.go proofRoot.mintAndLog authority B3,J1,J4 route-and-input file:Stat@1584,registry:App@450,registry:App@609,registry:App@732,registry:Host@488,registry:Host@664,registry:Unit@362,whole:Route@322,whole:Route@355,whole:Route@425,whole:Route@522,whole:Route@557,whole:Route@762 88110bedb9c021e2329661900f748e1bb42652754e68760c4cd7fbbe466419c3 |
| -internal/host/proofroot.go proofRoot.observeAndRecord authority B3,J1,J4 route-and-input file:Readlink@108,registry:App@361,registry:Host@151,registry:Host@537,whole:Route@186,whole:Route@356 fadb09b9406f52b578d1aa3adebb035845bfcf04dd8c938ca84084d4c235803a |
| -internal/host/proofroot.go proofRoot.recordLive authority B3,J1,J4 root-route registry:Host@500,registry:Unit@455,whole:Route@415,whole:Route@448,whole:Route@530 f9f23ef7a1c9ce2e19778367aa8d2837a17b502b848e9a25dc760f8d31c12e5a |
| -internal/host/proofroot.go proofRoot.stillLive authority B3,J1,J4 route-and-input file:Readlink@90,registry:Host@133,registry:Unit@262 2b43fc1dfa9fb8e2590a2d5fcbbdbc70652fb5d4562f6be0de68012b35e65abc |
| +internal/host/proofroot.go proofRoot.mint authority B3,J1,J4 root-route registry:Account@136,registry:App@1158,registry:App@1780,registry:App@1895,registry:App@2099,registry:App@754,registry:Host@1079,registry:Host@1419,registry:Host@1886,registry:Host@2009,registry:Host@289,registry:Host@442,registry:Host@955,whole:Route@129,whole:Route@144 194c34520aed1df0fa619201564528c9a122394b378b0e12abe2d2a2bf1249f2 |
| +internal/host/proofroot.go proofRoot.mintAndLog authority B3,J1,J4 canonical record/unit/executable authority and one-attempt rollback; draft incomplete file:Stat@1574,registry:App@433,registry:App@592,registry:App@715,registry:Host@471,registry:Host@647,whole:Route@322,whole:Route@355,whole:Route@408,whole:Route@505,whole:Route@540 e2a422e0381888600cdf3d26003a07c3c217345a2647065f6b8178f3c27d791d |
| +internal/host/proofroot.go proofRoot.observeAndRecord authority B3,J1,J4 route-and-input file:Readlink@108,registry:Account@335,registry:App@360,registry:Host@151,registry:Host@549,whole:Route@186,whole:Route@345,whole:Route@370 9739fbef16b4aa5894f0f00bea4012a592aea4efa9ef07202de45ef588f935fb |
| +internal/host/proofroot.go proofRoot.recordLive authority B3,J1,J4 canonical record/unit/executable authority and one-attempt rollback; draft incomplete registry:Host@483,whole:Route@415,whole:Route@448,whole:Route@513 d2919a2548d64a18c2426179041ff928bdc6981e571053060e67fa30cb45bdca |
| +internal/host/proofroot.go proofRoot.stillLive authority B3,J1,J4 route-and-input file:Readlink@291,registry:Host@334,registry:Unit@463,whole:Route@172,whole:Route@256,whole:Route@269 24c1c38c37e4116a28adefdc698a5dc268fa713adabe1ac1e97a5a1f24e50387 |
| internal/host/proofroot.go readProvingKey authority B3,J1,J4 root-input file:ReadFile@329,file:Stat@75 55656de233a53af669ebaf7319c705008595e24f602f0a27bd87b54cba8d6421 |
| internal/host/registrychild_exec_linux.go registryChildExec legacy accepted-legacy accepted-legacy;rev3=J4;basis=root-input file:Command@157 53afa6b0ae4cca751945a4c4c0f4c58e31cb48272ecfe77dc33c75728ec1936f |
| internal/host/registrychild_exec_linux.go var registryExecutable legacy accepted-legacy accepted-legacy;rev3=J4;basis=root-input file:Open@50 cb2c30ef23dc62d93532a9d3c8cd0e386614136a0f016660d7f83bef746a1d8b |
| @@ -1553,7 +1551,7 @@ internal/host/unitsettle.go routeReady authority B3,J1,J4 root-route registry:Ad |
| internal/host/unitsettle.go systemctlVerb authority B3,J1,J4 root-input file:Command@59 ca18cc4561d686e6be1606ea864b583569e94765740ca053f963505a1786e443 |
| internal/host/unitsettle.go systemdUnits.Exe authority B3,J1,J4 root-input file:Readlink@59 fdf47aad07111d423c2e928de746b26a4cbdead9e6a11a9a02192cc44d5822d4 |
| internal/host/unitsettle.go systemdUnits.JournalTail authority B3,J1,J4 root-input file:Command@72 9c7916578ce243bede29e9d8c16e3571172449afd98d159594668c3279cb2305 |
| -internal/host/unitsettle.go systemdUnits.Show authority B3,J1,J4 root-input file:Command@72 5f07ba007b89aa0d29c71303cb41325323cb57b26ec53a5466308d429929a824 |
| +internal/host/unitsettle.go systemdUnits.Show authority B3,J1,J4 root-input file:Command@72 aaba4cf699092360ac7be3fd98dfe226760f05290f003f0e7665424f948a59b5 |
| internal/host/updatesreport.go updatesReport diagnostic H3 no-effect registry:BuildState@482,registry:Host@1353,registry:Host@325,registry:UpdateSettings@267,whole:Route@210 4f907d59ea76f17e5d5b7f39bb7c711db1101f45427cf2bcda7b97be1b31df90 |
| internal/host/versionsweep.go settingsPins authority E2,J3,J4 root-route whole:Route@115,whole:Route@160 ba7a6ca5a9bf5c4ef9db3b84e1703f567d735fff61afc7d55b5c2e2c1a0702b2 |
| internal/host/versionsweep.go sweepStagedVersions authority E2,J3,J4 root-route registry:GoodVersion@868,registry:GoodVersion@900,registry:PrevVersion@681,registry:PrevVersion@713,registry:Version@626,registry:Version@654,whole:Route@1201,whole:Route@405 5ebcf5537c60166fcee7c82864ff5c736c1f89fcce13a4642a132ef40d2549d4 |
| @@ -1619,9 +1617,9 @@ internal/hostbox/launchjournal.go LaunchJournal.SessionLaunches authority C,J2,J |
| internal/hostbox/launchjournal.go LaunchJournal.mutate authority C,J2,J4 root-input file:QueryRowContext@336 7ce6fc570958de43ef0666e4fca26249c255f5f96849f309faf46f2cf060c53b |
| internal/hostbox/launchjournal.go OpenLaunchJournal authority C,J2,J4 root-input file:Exec@582,file:Lstat@665,file:Open@203 365f4535365ab3942ef14f0326f808bb5b91d54770b97d709f3e75afd098dd41 |
| internal/hostbox/park.go Regional.observe authority C,J2,J4 root-input file:Read@353 a75a92dcaff677ca3d7826b6c0bea3963e9a4dfc5e6c70d7349eb2da5f8e6b73 |
| -internal/hostbox/reconcile.go Reconciler.Install authority C,J2,J4 root-input file:Read@266 2a440c2769a12e5b1753c157b65e8185f6006b526f22f351896ddf2e4d1ced47 |
| +internal/hostbox/reconcile.go Reconciler.Install authority C,J2,J4,J1 root installation/fence backend and lock-free agent projection; establishment blocked; draft incomplete file:Read@266 c6013e40767c232b8b5edaea49cf543b522375c5eb1e34d15a7aff2267bab6eb |
| internal/hostbox/reconcile.go activationReader.Read authority C,J2,J4 root-input file:Read@183,file:Read@26 5451ef2cfc1cad30fd9fde526b28749580de2a169867fd5cabe2e95bf7fa89e0 |
| -internal/hostbox/reconcile_adapter.go ReconcileAdapter.Install authority C,J2,J4 route-and-input file:Read@1980,file:Read@3196,registry:ClaimOwner@3007,registry:ClaimOwner@3512,registry:ClaimOwnership@3849,registry:ClaimOwnership@4069,registry:ClaimRowID@3103,registry:ClaimRowID@3250,registry:ClaimRowID@3403,registry:ClaimRowID@3483,whole:ClaimOwnership@3853,whole:ClaimOwnership@3871,whole:ClaimOwnership@3937,whole:ClaimOwnership@4073,whole:ClaimOwnership@4091,whole:Route@3730,whole:Route@3835,whole:Route@3978,whole:Route@4055,whole:Route@988 faaeaa6b98f14e0c3d70dc9ddc6fcab930b6902fe7133650f9792556d9f9fc39 |
| +internal/hostbox/reconcile_adapter.go ReconcileAdapter.Install authority C,J2,J4,J1 root installation/fence backend and lock-free agent projection; establishment blocked; draft incomplete file:Read@1961,file:Read@3177,registry:ClaimOwner@2988,registry:ClaimOwner@3493,registry:ClaimOwnership@3830,registry:ClaimOwnership@4050,registry:ClaimRowID@3084,registry:ClaimRowID@3231,registry:ClaimRowID@3384,registry:ClaimRowID@3464,whole:ClaimOwnership@3834,whole:ClaimOwnership@3852,whole:ClaimOwnership@3918,whole:ClaimOwnership@4054,whole:ClaimOwnership@4072,whole:Route@3711,whole:Route@3816,whole:Route@3959,whole:Route@4036,whole:Route@988 eda5a243703517f6b765a5ae81e1e373002a2b352cc7253bda22da2f3adf8f56 |
| internal/hostbox/reconcile_adapter.go ReconcileAdapter.Retire authority C,J2,J4 root-route registry:ClaimIncarnation@1309,registry:ClaimIncarnation@322,registry:ClaimIncarnation@3471,registry:ClaimIncarnation@3493,registry:ClaimOwner@1366,registry:ClaimOwner@1434,registry:ClaimOwner@2190,registry:ClaimOwner@2596,registry:ClaimOwner@3551,registry:ClaimOwner@3567,registry:ClaimOwner@3803,registry:ClaimOwnership@147,registry:ClaimRowID@3585,registry:ClaimRowID@3826,registry:ClaimStamp@1338,registry:ClaimStamp@3517,registry:ClaimStamp@3533,registry:ClaimStamp@362,registry:Host@1257,registry:Host@2095,registry:Host@2641,registry:Host@2698,registry:Host@288,registry:Host@3390,registry:Host@3771,registry:Name@3788,registry:Registration@3607,registry:Registration@3847,whole:ClaimOwnership@151,whole:Route@1069,whole:Route@1152,whole:Route@231,whole:Timer@3456 1c66bdfafb81fc5a563de6a2e3b19be56137b50cfc1bb5804bce2caa25fb840e |
| internal/hostbox/regionalworker.go RegionalWorker.Census authority C,J2,J4 root-input file:Read@1361 477bcb896b5ae8ec80af592d68eceb7992ad8a23cd6cff6043ad03b3ef2a8085 |
| internal/hostbox/regionalworker.go RegionalWorker.runHost authority C,J2,J4 root-input file:Exec@2481,file:Read@2264,file:Read@227,file:Read@3122,file:Read@513 ac4a0f827f5096b35870f9c4adc4199a43c64bc60aea29b7b7bc9ab36b80710c |
| @@ -1654,7 +1652,7 @@ internal/hostbox/wakepolicy.go Policy.check authority C,J2,J4 root-input file:Ex |
| internal/hostbox/wakepolicy.go WakePhase authority C,J2,J4 root-input file:Exec@115 cab124d9d5af2206733aa5d5a1fcb85fc387cdaf88f5ad1b14dfd6fa8c6da895 |
| internal/hostbox/wakepolicy.go relocationProbe authority C,J2,J4 root-input file:Exec@154 90b1ed488226cdbc739898c69ee700451d896613ed7c5f15d792997642fc79c4 |
| internal/hostbox/work.go Serving authority C,J2,J4 root-input file:Exec@159 e196fc36950ecf0b1b359448ab58a1fa8922f1fe2418473bd39f03bd4cfc44a6 |
| -internal/hostbox/work.go WorkSource.Select authority C,J2,J4 root-input file:Prepare@592,file:Prepare@623,file:Read@381 80a13708c298cfef83165346fd1a72cc41bc20a1b131d464d4f69ee2ebbe062c |
| +internal/hostbox/work.go WorkSource.Select authority C,J2,J4,J1 root installation/fence backend and lock-free agent projection; establishment blocked; draft incomplete file:Prepare@573,file:Prepare@604,file:Read@362 9ec241046c53755d578cd81f49e4b1af826beb5ced9c8bfaec9a5e253ee13c85 |
| internal/hostbox/work.go type WorkSource diagnostic H3 schema-only file:Prepare@74 452e36e59b9b8b3526f4c3ba656075a45334d80f95a021af2f03a57d3e89f360 |
| internal/hostbox/writer.go Writer.Published authority C,J2,J4 root-input file:Exec@288 9ce6abccd517c52ad1781fba4e598261ce7ce52ce45ecbd5189f89b489f562e9 |
| internal/hostclaim/adopt.go Adopt legacy accepted-legacy accepted-legacy;rev3=J4;basis=root-input file:Exec@868 24c06b99cce6c638acb19826985cfadd3c736fd0ace7289bcf7ac3dfe784f05a |
| @@ -1745,16 +1743,16 @@ internal/hostfloor/report.go Disk.Running authority F,J4 root-input file:ReadDir |
| internal/hostfloor/report.go ReadEnrollment authority F,J4 root-input file:ReadFile@68 c4107af2612a15093d249ab8a3dee553a9b28c172b9548921425f10637edd8a7 |
| internal/hostguard/fence.go StopAuthorityMatches authority C,J2,J4 root-input file:Exec@1240,file:Exec@1402,file:Exec@1421,file:Exec@289 a2deddcbe8f4f55931510c26c7c3c3e94a35dde3b52620bfd56412703ed0d120 |
| internal/hostguard/flock_unix.go acquire authority C,J2,J4 root-input file:Fstat@805,file:Openat@444,file:Openat@571 6a6c87b8137458147e712d281e6e2dedd1c8a8bc7443303bda85ba6ebc1b07d6 |
| -internal/hostguard/guard.go Guard.Activate authority C,J2,J4 root-input file:Exec@253 467debf11dafc9986cd611f208563111cccb8ad48507582b66e692d6ff07d6fc |
| +internal/hostguard/guard.go Guard.Activate authority C,J2,J4,J1 root installation/fence backend and lock-free agent projection; establishment blocked; draft incomplete file:Exec@674 5cbb74a1fe791bb4226a037f1c82493d7f158cf89a77fa79567cf4b7f0d03977 |
| internal/hostguard/guard.go Guard.freshObserved authority C,J2,J4 root-input file:Read@247 01427b03818f6ffa23ad9c5469159102efa000e6440b1704b00ab319c189c85d |
| internal/hostguard/guard.go StoreReader.Read authority C,J2,J4 root-input file:Read@165,file:Read@21 13817e03f19e8af05801ddbde8408752f0c065a6cc85ed43c7e0296e3ea019a3 |
| internal/hostguard/guard.go type ClaimReader diagnostic H3 schema-only file:Read@30 53af4022a018dac954180f0b150e95e17b7022d10dfb35f67e3d17b4a0ea6a70 |
| internal/hostguard/record_unix.go openDirectory authority C,J2,J4 root-input file:Fstat@305,file:Open@59 6d2b2c8d8276bb03839cfc2b8a69b841e2297e9b134f94118479bc56dc46d1fd |
| internal/hostguard/record_unix.go recordDir.openRecord authority C,J2,J4 root-input file:Fstat@496,file:Openat@223 27588197380c52ddfcec3910eadb88de91d4279f6be575216d29d383adcd2dc7 |
| -internal/hostguard/record_unix.go recordDir.read authority C,J2,J4 root-input file:ReadAll@157 e767a5b3f4324a87991aebdb0ac6aeb5578d4fc67af239ec5d36d27322e8ec7a |
| -internal/hostguard/record_unix.go recordDir.write authority C,J2,J4 root-input file:Read@78 7630afb50d0e1f7c49e799c51934282ffcab78708aa693fb9e187b6f94950660 |
| +internal/hostguard/record_unix.go recordDir.read authority C,J2,J4,J1 root installation/fence backend and lock-free agent projection; establishment blocked; draft incomplete file:ReadAll@330 d7bc81639d9879821e03102c845118671df16d992dd7d956b36848285a8fce4e |
| +internal/hostguard/record_unix.go recordDir.write authority C,J2,J4,J1 root installation/fence backend and lock-free agent projection; establishment blocked; draft incomplete file:Read@172 68466bef0002f32731340f43b96bb07bd2184a5fe4d8797959d34b9d089fd759 |
| internal/hostguard/record_unix.go recordDir.writeUsing authority C,J2,J4 root-input file:Openat@670 b5236ba5327d37c85bc9acb620cde1c537609a51e04c5d2fefda13dfa23e5f4a |
| -internal/hostguard/recoveryfence.go Guard.PendingFences authority C,J2,J4 root-input file:ReadDir@92 8c6219c99edb2c70345197a873e53814a89dc5095ae717ed7faea69a4c0e4aff |
| +internal/hostguard/recoveryfence.go Guard.PendingFences authority C,J2,J4,J1 root installation/fence backend and lock-free agent projection; establishment blocked; draft incomplete ea07377f384ebbcf8790de9909eb79f4df16fb04bf81d4387c15a67f024e412b |
| internal/hostguard/writerstop.go RecoveryWriterStop authority C,J2,J4 root-input file:Exec@284 ec907a4a5c68831b14575860ef570fe2e77c0fed309175bfa75c5ce61fffb109 |
| internal/hostguard/writerstop.go operationWriter authority C,J2,J4 root-input file:Exec@271 00633d47bc43322edb97133b520df66bf4dda1bc22d85c32322e129874ccbfd3 |
| internal/hostops/app_mutation_runner.go mutationBoundary.RunAppMutation legacy accepted-legacy accepted-legacy;rev3=J4;basis=root-input file:Read@1057 6bdf9d56ccb33b590eb6373bcd08ae53feb75e9f827a64c6a18606d82bebed2f |
| @@ -2184,10 +2182,10 @@ internal/relay/serve.go readPSI legacy accepted-legacy accepted-legacy;rev3=J4;b |
| internal/relay/server.go readCounter.ReadFrom legacy accepted-legacy accepted-legacy;rev3=J4;basis=root-input file:ReadFrom@22,file:ReadFrom@83 ea3eeed88cecde61e2c4774ddc1e9e471ddfe904ec703d6551c67a0d04651648 |
| internal/release/adopt.go Adoptable authority C,E3,J1 root-route;kept=C,E3;scope=protected-decision-only registry:App@74,registry:Kind@49,registry:Platform@91,registry:Status@109,registry:Status@128 1756c9b3f31f6d168ee5e90ff06c0f848cf7e95c61f4924611cdfd4c1dc8d181 |
| internal/release/adopt.go CapabilityRoute authority E4,J1 root-route;kept=E4;scope=protected-decision-only registry:Backing@72,whole:Route@65 a42e07c9b88999682eff9750702117dee2753d88bb0de6b9236c7da43528c3a1 |
| -internal/release/adopt.go PassFilter authority E3,J1 root-route;kept=E3;scope=protected-decision-only registry:App@24283,registry:App@26740,registry:App@27232,registry:App@27469,registry:App@28876,registry:App@4401,registry:App@7426,registry:App@7567,registry:Hibernate@27115,registry:Host@1239,registry:Host@1335,registry:Host@1370,registry:Host@27285,registry:Host@28201,registry:Host@28929,registry:UpdateSettings@7304,whole:Route@1006,whole:Route@1030,whole:Route@1061,whole:Route@1118,whole:Route@1147,whole:Route@1169,whole:Route@2091,whole:Route@24278,whole:Route@26735,whole:Route@26993,whole:Route@27015,whole:Route@27062,whole:Route@27080,whole:Route@27090,whole:Route@27133,whole:Route@27154,whole:Route@27167,whole:Route@27227,whole:Route@27253,whole:Route@27327,whole:Route@27366,whole:Route@2738,whole:Route@27382,whole:Route@27410,whole:Route@27464,whole:Route@27490,whole:Route@28243,whole:Route@28297,whole:Route@28313,whole:Route@28820,whole:Route@28871,whole:Route@28897,whole:Route@28971,whole:Route@29006,whole:Route@29022,whole:Route@4323,whole:Route@4345,whole:Route@4396,whole:Route@4422,whole:Route@7255,whole:Route@7369,whole:Route@7414,whole:Route@7562 9d5b17141ccbaca3b7e520c7d1ed8f074f375921c754f3017281b549b68ff4ee |
| +internal/release/adopt.go PassFilter authority B3,E3,J1 root-route;kept=B3,E3;scope=protected-decision-only registry:App@24349,registry:App@26806,registry:App@27298,registry:App@27535,registry:App@28942,registry:App@4467,registry:App@7492,registry:App@7633,registry:Hibernate@27181,registry:Host@1305,registry:Host@1401,registry:Host@1436,registry:Host@27351,registry:Host@28267,registry:Host@28995,registry:UpdateSettings@7370,whole:Route@1072,whole:Route@1096,whole:Route@1127,whole:Route@1184,whole:Route@1213,whole:Route@1235,whole:Route@2157,whole:Route@24344,whole:Route@26801,whole:Route@27059,whole:Route@27081,whole:Route@27128,whole:Route@27146,whole:Route@27156,whole:Route@27199,whole:Route@27220,whole:Route@27233,whole:Route@27293,whole:Route@27319,whole:Route@27393,whole:Route@27432,whole:Route@27448,whole:Route@27476,whole:Route@27530,whole:Route@27556,whole:Route@2804,whole:Route@28309,whole:Route@28363,whole:Route@28379,whole:Route@28886,whole:Route@28937,whole:Route@28963,whole:Route@29037,whole:Route@29072,whole:Route@29088,whole:Route@4389,whole:Route@4411,whole:Route@4462,whole:Route@4488,whole:Route@7321,whole:Route@7435,whole:Route@7480,whole:Route@7628 3a87b6584a8f83c6bf0d5367b91b8dac8109d5db66e19d79d8b22c4629be3ba7 |
| internal/release/adopt.go RouteAccount authority B3,E3,E4,J1 root-route;kept=B3,E3,E4;scope=protected-decision-only registry:Account@63,registry:Account@86 2dcd516f6e5d6144c8d9aac56569ba964ee49441123ac1fd57af621d6553d627 |
| internal/release/adopt.go SidecarAdoptable legacy accepted-legacy accepted-legacy;rev3=J1;basis=root-route registry:App@74,registry:Platform@91,registry:Status@109,registry:Status@128 aeb1f3d46b0393ca131b213cb9974f0f414e733fac0e033e7bfbfff0dda3d630 |
| -internal/release/adopt.go adoptHost authority B3,E1,E2,E3,E4,J1 root-route;kept=B3,E1,E2,E3,E4;scope=protected-decision-only registry:App@10271,registry:App@10431,registry:App@11460,registry:App@11917,registry:App@13283,registry:App@17583,registry:App@22426,registry:App@29931,registry:App@3114,registry:App@6423,registry:App@7697,registry:App@7891,registry:Hibernate@30610,registry:Host@10151,registry:Host@10184,registry:Host@10284,registry:Host@10444,registry:Host@10546,registry:Host@10578,registry:Host@10643,registry:Host@11093,registry:Host@11226,registry:Host@11546,registry:Host@12007,registry:Host@12755,registry:Host@13056,registry:Host@13269,registry:Host@13585,registry:Host@17305,registry:Host@17413,registry:Host@17551,registry:Host@18250,registry:Host@18408,registry:Host@18870,registry:Host@18922,registry:Host@19042,registry:Host@19129,registry:Host@20235,registry:Host@20430,registry:Host@2075,registry:Host@21251,registry:Host@21392,registry:Host@22394,registry:Host@22521,registry:Host@2264,registry:Host@2345,registry:Host@23863,registry:Host@24513,registry:Host@27746,registry:Host@28364,registry:Host@28689,registry:Host@28763,registry:Host@28914,registry:Host@30088,registry:Host@30195,registry:Host@30423,registry:Host@30566,registry:Host@3127,registry:Host@31311,registry:Host@31437,registry:Host@31525,registry:Host@4248,registry:Host@5211,registry:Host@5316,registry:Host@6535,registry:Host@6583,registry:Host@6700,registry:Host@6732,registry:Host@7583,registry:Host@7751,registry:Host@7877,registry:Host@7996,registry:Host@8474,registry:Host@8861,registry:Host@9087,registry:Host@9183,registry:Kind@552,registry:Unit@577,whole:Route@10266,whole:Route@10426,whole:Route@11450,whole:Route@11907,whole:Route@13786,whole:Route@14878,whole:Route@15181,whole:Route@17573,whole:Route@19964,whole:Route@2176,whole:Route@22416,whole:Route@27317,whole:Route@3109,whole:Route@3554,whole:Route@6418,whole:Route@9408 6d1bcf629b1636721e1ef347abd2039024fed8b6902f86dca85ea9035537d1a6 |
| +internal/release/adopt.go adoptHost authority B3,E1,E2,E3,E4,J1 root adoption attempt terminal settlement; existing adoption authorities retained; draft incomplete registry:App@10271,registry:App@10431,registry:App@11460,registry:App@11917,registry:App@13283,registry:App@17583,registry:App@22746,registry:App@30353,registry:App@3114,registry:App@6423,registry:App@7697,registry:App@7891,registry:Hibernate@31032,registry:Host@10151,registry:Host@10184,registry:Host@10284,registry:Host@10444,registry:Host@10546,registry:Host@10578,registry:Host@10643,registry:Host@11093,registry:Host@11226,registry:Host@11546,registry:Host@12007,registry:Host@12755,registry:Host@13056,registry:Host@13269,registry:Host@13585,registry:Host@17305,registry:Host@17413,registry:Host@17551,registry:Host@18250,registry:Host@18408,registry:Host@18870,registry:Host@18922,registry:Host@19042,registry:Host@19129,registry:Host@20235,registry:Host@20430,registry:Host@2075,registry:Host@21090,registry:Host@21475,registry:Host@21616,registry:Host@2264,registry:Host@22714,registry:Host@22841,registry:Host@2345,registry:Host@24183,registry:Host@24833,registry:Host@28066,registry:Host@28684,registry:Host@29009,registry:Host@29083,registry:Host@29336,registry:Host@30510,registry:Host@30617,registry:Host@30845,registry:Host@30988,registry:Host@3127,registry:Host@31733,registry:Host@31859,registry:Host@31947,registry:Host@4248,registry:Host@5211,registry:Host@5316,registry:Host@6535,registry:Host@6583,registry:Host@6700,registry:Host@6732,registry:Host@7583,registry:Host@7751,registry:Host@7877,registry:Host@7996,registry:Host@8474,registry:Host@8861,registry:Host@9087,registry:Host@9183,registry:Kind@552,registry:Unit@577,whole:Route@10266,whole:Route@10426,whole:Route@11450,whole:Route@11907,whole:Route@13786,whole:Route@14878,whole:Route@15181,whole:Route@17573,whole:Route@19964,whole:Route@2176,whole:Route@22736,whole:Route@27637,whole:Route@3109,whole:Route@3554,whole:Route@6418,whole:Route@9408 f6a3caf9e79f45ff6e13507bdfeb63911ec771e7ce3ae0355f709e7b2d8e7575 |
| internal/release/adopt.go backfillManifest legacy accepted-legacy accepted-legacy;rev3=J4;basis=root-input file:Stat@105 30982c04c6adcea43f44a51960808368017b860cf50eebb282aa53871c487860 |
| internal/release/adopt.go backfillManifestDigest legacy accepted-legacy accepted-legacy;rev3=J4;basis=root-input file:Stat@203 63f6faa81e0d54b60862bdda8cf6ac128d8a723d1987d187eecfb5c3924b254d |
| internal/release/adopt.go capabilityPlan authority E4,J1,J4 root-route registry:App@503,registry:Caps@261,registry:Caps@348,registry:Hibernate@567,registry:Host@478 ca4730fc93cecaa05477e9e5ba795f4088f3c1b557c021f66a12a81180ffb32f |
| @@ -2329,7 +2327,7 @@ internal/rollout/worker.go scheduler.executeWorker legacy accepted-legacy accept |
| internal/rootaws/credentials_unix.go readStaticCredentials legacy accepted-legacy accepted-legacy;rev3=J4;basis=root-input file:Fstat@200,file:ReadAll@518 b312c4254cf614504a57071dd1dc976fc29288f9f24dfe6ae9b9cefe0cdb9ff3 |
| internal/rootaws/open_darwin.go openCredentials legacy accepted-legacy accepted-legacy;rev3=J4;basis=root-input file:OpenFile@62 afdf951a528427a49f498f6dcedb30743663b13ccfe0afb09eafb1086567277f |
| internal/rootaws/open_linux.go openCredentials legacy accepted-legacy accepted-legacy;rev3=J4;basis=root-input file:OpenFile@212 21b7d8dbdcee31d9eb41933dee07071c01b3a704a12e1425516777cd98ea6cf9 |
| -internal/rootstate/state.go type Store diagnostic H3 schema-only file:Read@24 a35e15e9f589daa85088d065907114713216a0e255ca68f0c8edb77e4c948aaa |
| +internal/rootstate/state.go type Store authority A,B3 descriptor-checked root rollback capability identity/removal file:Read@24 c55cd04c708d1641eeb748500b34d2d0ebe401a4741f521a1cf1f4d8cce800b8 |
| internal/rootstate/state_linux.go Open authority A root-input file:Open@214,file:Open@5 4c77898a23a680ac70abebf3efc6cf6d47c5be9aacb6c0f907d532b074ae0846 |
| internal/rootstate/state_linux.go descendSync authority A root-input file:Fstat@1028,file:Fstat@1497,file:Openat@524,file:Openat@772 db80107ad25c50814098cbb6d60e6683ae65beb2bb0eff2d6e3abb8f9cfee1e7 |
| internal/rootstate/state_linux.go directory.Names authority A root-input file:Openat@182,file:ReadDir@382 42f26950076aafdb5d7db012a741f85f74046b17e84211bc3c182b76535d45ec |
| @@ -4193,3 +4191,33 @@ update.go selfReplace legacy accepted-legacy accepted-legacy;rev3=J4;basis=root- |
| @legacy 39d7472e2fdf0194ddba4874a412f7f1d0e9231f0639f44353766391d758af85 update.go maybeUpdateHint legacy accepted-legacy accepted-legacy;rev3=J4;basis=root-input file:ReadFile@487 6e60a27646c0bf34bb9cd13c523048ba3811119c4dfdd8c3faf076a1853047b1 |
| @legacy 8d60f6ba00bb044769025b98ca8b5061309922e8fe8abc28bb309991d73cac4e update.go runUpdate legacy accepted-legacy accepted-legacy;rev3=J4;basis=root-input file:EvalSymlinks@1057 9f2474b62b2960355ac3c24c7985a0e9b253094f66a85fd17cfdb7b94f8d79e1 |
| @legacy e97d9fba4abbfbc087a9e8a196aac06b73fb83d5f3c0ce1270fa32714eebb091 update.go selfReplace legacy accepted-legacy accepted-legacy;rev3=J4;basis=root-input file:Command@843,file:CreateTemp@244 c97bcb69dde01d481ddc7c645cbb44e4ab61af7ddea89fac51760106525c7324 |
| +internal/host/agent.go agentCheckedUnitRestart authority B3,J1 kept=B3;scope=agent-only-legacy-behavior-preservation;not-root-authority file:EvalSymlinks@761,registry:Host@1521,registry:Host@1620,registry:Host@321,registry:Host@422,registry:Host@582,registry:Unit@1530,registry:Unit@1629,registry:Unit@225,registry:Unit@330,registry:Unit@369,registry:Unit@556,registry:Unit@616,registry:Unit@692,registry:Unit@832,registry:Unit@872,registry:Unit@976,whole:Route@1142,whole:Route@1804 b475d7624701ba8fb18e68774a5fe35066c4d94ceb580d6a5eb4d449175b76e2 |
| +internal/host/agent.go agentRollbackUnitRestart authority B3,J1 kept=B3;scope=agent-only-legacy-behavior-preservation;not-root-authority file:EvalSymlinks@515,registry:Unit@1158,registry:Unit@232,registry:Unit@290,registry:Unit@327,registry:Unit@417,registry:Unit@459,registry:Unit@634,whole:Route@219,whole:Route@819 ca7c2a6c2cb0a4957f74c2767cc1d120c8ccb8a1d79b8a05760042e891b1d895 |
| +internal/host/proofbinding.go proofBindingSource.forHost authority B3,J1 canonical record/unit/executable authority and one-attempt rollback; draft incomplete 3561b1aa7330658fa7e761ce53d599a85584c8cd9b2d15b82d1cee296589ec1a |
| +internal/host/proofbinding.go proofBindingSource.resolve authority B3,J1 canonical record/unit/executable authority and one-attempt rollback; draft incomplete registry:Host@134,whole:Route@223,whole:Route@236 20b927104656ee5b93d0ce9b9fe6eae7dc0a0b19dfa77e6e73e4b960e87f831c |
| +internal/host/proofbinding.go proofRouteReady authority B3,J1 kept=B3;scope=canonical-binding-and-readiness;implementation-incomplete registry:Addr@231,registry:Addrs@198,registry:Addrs@243,registry:Addrs@82,registry:Socket@256,whole:Route@119,whole:Route@226,whole:Route@306 b20c7bd1b2848a8f86457f9cf6153880fca078ca4bb4c335c7da042e306a830b |
| +internal/host/proofobserve.go proofObserver.recordSettled authority B3,J1 kept=B3;scope=canonical-binding-and-readiness;implementation-incomplete registry:Account@227,registry:App@252,whole:Route@144,whole:Route@237,whole:Route@262 e0676272c20544b61b4150ce08d06032a9948b9ccc924adf0d63a7d92fec46fd |
| +internal/host/rootrestart.go checkedUnitRestart authority B3,J1 canonical record/unit/executable authority and one-attempt rollback; draft incomplete registry:Host@1167,registry:Host@1703,registry:Host@510,registry:Unit@1585,registry:Unit@711,whole:Route@1593,whole:Route@638,whole:Route@719 a2355ed66e0ec7876b68062f9b8488afcc7ad0845cc8a163e5572163ebb12194 |
| +internal/host/rootrestart.go productionRootRestartChecks authority B3,J1 canonical record/unit/executable authority and one-attempt rollback; draft incomplete file:Open@531,whole:Route@1224,whole:Route@469,whole:Route@957 2334f2179699968016be1342722b66623212da70006750eedb690bbc42e3a8c1 |
| +internal/host/rootrestart.go restartBoundUnit authority B3,J1 canonical record/unit/executable authority and one-attempt rollback; draft incomplete file:EvalSymlinks@632,registry:Account@1711,registry:Account@2858,registry:App@1747,registry:App@2898,registry:Host@1196,registry:Host@1613,registry:Host@2012,registry:Host@2629,registry:Host@483,registry:Host@679,registry:Unit@1465,registry:Unit@2315,registry:Unit@3160,registry:Unit@393,registry:Unit@765,registry:Unit@830,whole:Route@1717,whole:Route@1753,whole:Route@198,whole:Route@2866,whole:Route@2906,whole:Route@3087,whole:Route@316 28559399e6e4ac594de69e3c4c305b9985550b08fb0989eb9a68f005be7d3e4d |
| +internal/host/rootrestart.go rootRestartRouteWith authority B3,J1 canonical record/unit/executable authority and one-attempt rollback; draft incomplete whole:Route@363,whole:Route@491,whole:Route@590 d93f43a503ec8b3da6e7d803d0f4c620b112855e144757096f5015f1b9e44c1f |
| +internal/host/unitsettle.go systemdUnits.StartTime authority B3,J1 kept=B3;scope=canonical-binding-and-readiness;implementation-incomplete file:ReadFile@119 8806107d8806e1162ca493d8da84ae79cd17c32454be577517dc421c18858c20 |
| +internal/rootstate/file_linux.go OpenTrustedFile authority A,B3 kept=B3;scope=canonical-binding-and-readiness;implementation-incomplete file:Open@174 8439fc9a19ef3b4b6b70fb0c82cdb2b98da20aaa7c1b14a2f47dd80e9424f262 |
| +internal/provision/rootrecord.go FindRecordByHost authority B3,J1 kept=B3;scope=canonical-binding-and-readiness;implementation-incomplete 10c8254cf896be034334d634151d169760a909ac1ba4c1fc334785e86e4b84d4 |
| +internal/host/rootrestart.go rollbackUnitRestart authority B3,J1 canonical record/unit/executable authority and one-attempt rollback; draft incomplete registry:Unit@904,whole:Route@912 80241c74e4bc690ce6223e1ae879e5e55d0a3875594f3f18412259ae106f0b84 |
| +internal/host/rootrestart.go rootRestartRoute authority B3,J1 kept=B3;scope=canonical-binding-and-readiness;implementation-incomplete e053df9a9fdf7860e92ff9938632ac2de2593c6b1d4e044c9ccbd93d71c123d9 |
| +internal/host/agent.go runAdoptUnderRootRelease authority B3,C,D,E1,E2,E3,E4,F,J1 main adoption body under outer root release serialization; existing authorities retained; draft incomplete whole:Route@10234,whole:Route@16534,whole:Route@16570,whole:Route@7662 3152a11fb471e90388becbbda2b1b451d4dab48592190ef3c7e59b638e0da471 |
| +internal/host/proofbinding.go productionProofBindingSource authority B3,J1 canonical record/unit/executable authority and one-attempt rollback; draft incomplete file:Read@692 e8a5f225e2286be5ff7dc8224c673d94d218ab365c13432f11c4d81f646c8eca |
| +internal/host/proofbinding.go proofBindingSource.binding authority B3,J1 canonical record/unit/executable authority and one-attempt rollback; draft incomplete file:EvalSymlinks@1950,registry:Unit@3750,registry:Unit@423,registry:Unit@506,registry:Unit@578,whole:Route@4187,whole:Route@4194 6847b909af286f4174197073a630c835edfeb1bd3aef864b2d19d36ae4dbb5c0 |
| +internal/host/proofbinding.go proofBindingSource.rollback authority B3,J1 canonical record/unit/executable authority and one-attempt rollback; draft incomplete registry:Host@122,whole:Route@124 43b49f858f0b360e5ccc1e8342eb8979b02832f5055b5a267c1499e70b5dd85a |
| +internal/host/rootrollback.go captureRollback authority B3,J1 canonical record/unit/executable authority and one-attempt rollback; draft incomplete file:Read@734,registry:Host@371,registry:Host@853,whole:Route@373,whole:Route@855 924ed7ca528c1f9f9eb53f9872c8a20f2a0d400cbe565d58b3cc25ea6a7741b0 |
| +internal/host/rootrollback.go loadRollback authority B3,J1 canonical record/unit/executable authority and one-attempt rollback; draft incomplete file:Read@467,registry:Host@765,whole:Route@777 e7c0dbfde8be208b5f6558389e2d16e8ee614e0b5cfe1a98b8d13f2100a265c5 |
| +internal/host/rootrollback.go readRollbackAttempt authority B3,J1 canonical record/unit/executable authority and one-attempt rollback; draft incomplete file:Read@118 3a41d88c6ce8e92d45bf08b25017698538a59d8cadef5ab81662ed63786d95fb |
| +internal/host/rootroute.go proofBindingSource.claimIdentity authority B3,J1 canonical record/unit/executable authority and one-attempt rollback; draft incomplete file:Exec@750 ad6e0fc3f2cd360473a5d0a2267d2dc7944b45d80947613f791db8d573f8fee6 |
| +internal/host/rootroute.go proofBindingSource.needsRestart authority B3,J1 canonical record/unit/executable authority and one-attempt rollback; draft incomplete registry:Host@162,registry:Unit@463,whole:Route@392,whole:Route@403 a7e59e27e4e737abd193405d209ed5cf1321576b487a66458b0831d598219192 |
| +internal/host/rootroute.go rootRecordRoute authority B3,J1 canonical record/unit/executable authority and one-attempt rollback; draft incomplete registry:Backing@477,registry:Health@856,registry:Kind@615,registry:Kind@640,registry:Socket@466,registry:Socket@888,registry:Unit@457,registry:Unit@560,registry:Unit@601,whole:Route@996 fa0904367baf3e9db943b2be50d147516c1e220df787e32c796f9cd6913dbcf4 |
| +internal/host/rootroute.go sameProofCandidate authority B3,J1 canonical record/unit/executable authority and one-attempt rollback; draft incomplete registry:Account@110,registry:Account@89,registry:Addr@273,registry:Addr@291,registry:Addrs@301,registry:Addrs@320,registry:App@123,registry:App@140,registry:Backing@391,registry:Backing@412,registry:Health@331,registry:Health@351,registry:Host@149,registry:Host@167,registry:Kind@363,registry:Kind@381,registry:Socket@241,registry:Socket@261,registry:Unit@178,registry:Unit@212,registry:Unit@230 a817b55fd4f4c480d0230399bc1a161833150f4d5844de4a96e5a2b981c0320e |
| +internal/hostguard/recoveryfence.go Guard.recordNames authority C,J1,J4 root installation/fence backend and lock-free agent projection; establishment blocked; draft incomplete file:ReadDir@119 e44562b3d64921be270d315e0434c60ebef5dda5087f48344fe70a35fed4ae91 |
| +internal/hostguard/root.go RootAuthority.read authority C,J1,J4 root installation/fence backend and lock-free agent projection; establishment blocked; draft incomplete file:Read@175 04404ade39ee41e19d51c7b48c51f7b4cadb7e3ef0ab16ef23efa338d374a594 |
| +internal/rootstate/file_linux.go Identity authority A,B3 descriptor-checked root rollback capability identity/removal file:Fstat@81 eb8b39ca4c1fff35019f0e9cb93a33378173c543060d24517bf1e9ab607df2a5 |
| +internal/rootstate/state_linux.go directory.Remove authority A,B3 descriptor-checked root rollback capability identity/removal file:Read@200 884759a5ab53bca8c61f5942a458fc6c4bd4431f0697bff34041d806ad4aeed8 |
diff --git a/internal/release/adopt.go b/internal/release/adopt.go| index 6c7d515a7..4d75861c8 100644 |
| --- a/internal/release/adopt.go |
| +++ b/internal/release/adopt.go |
| @@ -37,6 +37,9 @@ var adoptPlatform = runtime.GOOS + "-" + runtime.GOARCH |
| type PassOption func(*passConfig) |
| type passConfig struct { |
| + propagateRestarts bool |
| + restartAttemptSettled func(string) error |
| + restartErrors *[]error |
| hostWrites hostwrite.Config |
| selectedRestart, selectedStart hostwrite.Callback |
| // selected is this pass's release work per adoptable host, captured once |
| @@ -317,6 +320,14 @@ func WithStartUnit(start func(host string) error) PassOption { |
| } |
| // WithRollbackRestart sets D1's rollback restart (see passConfig). |
| +func WithRestartAttemptSettled(fn func(string) error) PassOption { |
| + return func(c *passConfig) { c.restartAttemptSettled = fn } |
| +} |
| + |
| +func WithRestartFailurePropagation() PassOption { |
| + return func(c *passConfig) { c.propagateRestarts = true } |
| +} |
| + |
| func WithRollbackRestart(fn func(host string) error) PassOption { |
| return func(c *passConfig) { c.rollbackRestart = fn } |
| } |
| @@ -426,6 +437,9 @@ func PassFilter(ctx context.Context, reg *registry.Registry, s Store, pub ed2551 |
| for _, opt := range opts { |
| opt(&cfg) |
| } |
| + if cfg.propagateRestarts { |
| + cfg.restartErrors = new([]error) |
| + } |
| cfg.pub = pub |
| cfg.route = reg.Get |
| // Settings spec §4.3: the failure book is read fresh for every pass, so a |
| @@ -1039,6 +1053,9 @@ func PassFilter(ctx context.Context, reg *registry.Registry, s Store, pub ed2551 |
| ptr := pointers[groupKey{RouteAccount(rt), rt.App, routeChannel(rt)}] |
| adoptHost(ctx, selected[rt.Host], reg, s, optDir, bake, poisoned, rt, ptr, nil, now, cfg, routeHolds(rt), bakeIgnore(rt)) |
| } |
| + if cfg.restartErrors != nil { |
| + return errors.Join(*cfg.restartErrors...) |
| + } |
| return nil |
| } |
| @@ -1566,6 +1583,13 @@ func adoptHost(ctx context.Context, work hostwrite.Work, reg *registry.Registry, |
| // plan.Set is non-nil whenever converged is true (capabilityPlan |
| // guarantees it); the nil check is here anyway, because a nil deref |
| // on this path would panic the root adopt pass for the whole box. |
| + if cfg.restartAttemptSettled != nil { |
| + defer func() { |
| + if err := cfg.restartAttemptSettled(rt.Host); err != nil && cfg.restartErrors != nil { |
| + *cfg.restartErrors = append(*cfg.restartErrors, err) |
| + } |
| + }() |
| + } |
| var rerr error |
| daemonRestart := converged && plan.Set != nil && plan.Set.Daemon |
| if cfg.hostWrites.Enabled { |
| @@ -1580,6 +1604,9 @@ func adoptHost(ctx context.Context, work hostwrite.Work, reg *registry.Registry, |
| rerr = restart(rt.Host) |
| } |
| if rerr != nil { |
| + if cfg.restartErrors != nil { |
| + *cfg.restartErrors = append(*cfg.restartErrors, rerr) |
| + } |
| // Poison the build for the rest of the pass and repoint back — |
| // the host keeps running what it ran. The symlink goes back |
| // FIRST: caps.Restore may start the service, and it must start |
| @@ -1734,6 +1761,9 @@ func adoptHost(ctx context.Context, work hostwrite.Work, reg *registry.Registry, |
| serr = start(rt.Host) |
| } |
| if serr != nil { |
| + if cfg.restartErrors != nil { |
| + *cfg.restartErrors = append(*cfg.restartErrors, serr) |
| + } |
| fmt.Printf(" %s: rolled back to %s but could not start it: %v — this host may be serving nothing\n", |
| rt.Host, cur, serr) |
| } |
diff --git a/internal/rootstate/file_linux.go b/internal/rootstate/file_linux.go| new file mode 100644 |
| index 000000000..a7ed304d2 |
| --- /dev/null |
| +++ b/internal/rootstate/file_linux.go |
| @@ -0,0 +1,38 @@ |
| +package rootstate |
| + |
| +import ( |
| + "os" |
| + "path/filepath" |
| + |
| + "github.com/carlosframework/platform/internal/safefile" |
| + "golang.org/x/sys/unix" |
| +) |
| + |
| +func OpenTrustedFile(path string) (*os.File, error) { |
| + if !filepath.IsAbs(path) || filepath.Clean(path) != path || path == "/" { |
| + return nil, os.ErrPermission |
| + } |
| + s, err := Open(filepath.Dir(path), false, true) |
| + if err != nil { |
| + return nil, err |
| + } |
| + defer s.Close() |
| + d := s.(*directory) |
| + f, err := safefile.OpenFileAt(int(d.file.Fd()), filepath.Base(path), os.O_RDONLY, 0) |
| + if err != nil { |
| + return nil, err |
| + } |
| + if err := trusted(int(f.Fd()), 0, unix.S_IFREG); err != nil { |
| + f.Close() |
| + return nil, err |
| + } |
| + return f, nil |
| +} |
| + |
| +func Identity(f *os.File) (FileIdentity, error) { |
| + var st unix.Stat_t |
| + if err := unix.Fstat(int(f.Fd()), &st); err != nil { |
| + return FileIdentity{}, err |
| + } |
| + return FileIdentity{Device: uint64(st.Dev), Inode: st.Ino}, nil |
| +} |
diff --git a/internal/rootstate/file_linux_test.go b/internal/rootstate/file_linux_test.go| new file mode 100644 |
| index 000000000..c1e0ea532 |
| --- /dev/null |
| +++ b/internal/rootstate/file_linux_test.go |
| @@ -0,0 +1,42 @@ |
| +package rootstate |
| + |
| +import ( |
| + "errors" |
| + "os" |
| + "path/filepath" |
| + "testing" |
| + |
| + "golang.org/x/sys/unix" |
| +) |
| + |
| +func TestPlan14ProofBindingFileProvenance(t *testing.T) { |
| + t.Run("root-owned regular file", func(t *testing.T) { |
| + for _, path := range []string{"/", "/usr", "/usr/bin", "/usr/bin/true"} { |
| + var st unix.Stat_t |
| + if err := unix.Lstat(path, &st); err != nil { |
| + t.Fatal(err) |
| + } |
| + if st.Uid != 0 { |
| + t.Skipf("root-owned fixture unavailable: %s uid=%d", path, st.Uid) |
| + } |
| + } |
| + f, err := OpenTrustedFile("/usr/bin/true") |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + f.Close() |
| + }) |
| + path := filepath.Join(t.TempDir(), "unit.service") |
| + if err := os.WriteFile(path, []byte("[Service]\n"), 0644); err != nil { |
| + t.Fatal(err) |
| + } |
| + if f, err := OpenTrustedFile(path); err == nil { |
| + f.Close() |
| + t.Fatal("accepted a unit under a writable/nonroot ancestor") |
| + } |
| + for _, path := range []string{"relative", "/etc/../etc/passwd", "/"} { |
| + if _, err := OpenTrustedFile(path); !errors.Is(err, os.ErrPermission) { |
| + t.Fatalf("%q: %v", path, err) |
| + } |
| + } |
| +} |
diff --git a/internal/rootstate/file_other.go b/internal/rootstate/file_other.go| new file mode 100644 |
| index 000000000..b36153918 |
| --- /dev/null |
| +++ b/internal/rootstate/file_other.go |
| @@ -0,0 +1,9 @@ |
| +//go:build !linux |
| + |
| +package rootstate |
| + |
| +import "os" |
| + |
| +func OpenTrustedFile(string) (*os.File, error) { return nil, os.ErrPermission } |
| + |
| +func Identity(*os.File) (FileIdentity, error) { return FileIdentity{}, os.ErrPermission } |
diff --git a/internal/rootstate/state.go b/internal/rootstate/state.go| index e2a5adf0d..24ca9aa22 100644 |
| --- a/internal/rootstate/state.go |
| +++ b/internal/rootstate/state.go |
| @@ -12,6 +12,7 @@ const MaxRecord = 1 << 20 |
| type Store interface { |
| Read(string) ([]byte, error) |
| Write(string, []byte) error |
| + Remove(string) error |
| Names() ([]string, error) |
| Lock(context.Context, string) (func(), error) |
| Close() error |
| @@ -35,3 +36,8 @@ func fileMode(public bool) os.FileMode { |
| } |
| return 0600 |
| } |
| + |
| +type FileIdentity struct { |
| + Device uint64 |
| + Inode uint64 |
| +} |
diff --git a/internal/rootstate/state_linux.go b/internal/rootstate/state_linux.go| index 05611436c..aa5701f8f 100644 |
| --- a/internal/rootstate/state_linux.go |
| +++ b/internal/rootstate/state_linux.go |
| @@ -220,3 +220,19 @@ func (d *directory) Lock(ctx context.Context, name string) (func(), error) { |
| } |
| } |
| } |
| + |
| +func (d *directory) Remove(name string) error { |
| + if !component(name) || uint32(os.Geteuid()) != d.owner { |
| + return os.ErrPermission |
| + } |
| + if err := d.check(); err != nil { |
| + return err |
| + } |
| + if _, err := d.Read(name); err != nil && !errors.Is(err, os.ErrNotExist) { |
| + return err |
| + } |
| + if err := unix.Unlinkat(int(d.file.Fd()), name, 0); err != nil && !errors.Is(err, unix.ENOENT) { |
| + return err |
| + } |
| + return unix.Fsync(int(d.file.Fd())) |
| +} |
diff --git a/internal/rootstate/state_linux_test.go b/internal/rootstate/state_linux_test.go| index 7fdb68c73..9e52554b0 100644 |
| --- a/internal/rootstate/state_linux_test.go |
| +++ b/internal/rootstate/state_linux_test.go |
| @@ -241,3 +241,28 @@ func TestPlan14DescendRetriesEveryAncestorSync(t *testing.T) { |
| }) |
| } |
| } |
| + |
| +func TestPlan14RemoveCapability(t *testing.T) { |
| + path, s := fixture(t) |
| + if err := s.Write("capability", []byte("secret")); err != nil { |
| + t.Fatal(err) |
| + } |
| + if err := s.Remove("capability"); err != nil { |
| + t.Fatal(err) |
| + } |
| + if _, err := s.Read("capability"); !errors.Is(err, os.ErrNotExist) { |
| + t.Fatal(err) |
| + } |
| + if err := s.Remove("capability"); err != nil { |
| + t.Fatal(err) |
| + } |
| + if err := s.Remove("../escape"); err == nil { |
| + t.Fatal("escaped") |
| + } |
| + if err := os.Symlink("outside", filepath.Join(path, "capability")); err != nil { |
| + t.Fatal(err) |
| + } |
| + if err := s.Remove("capability"); err == nil { |
| + t.Fatal("removed untrusted entry") |
| + } |
| +} |