amadan

carlos / platform Public

host: relay a blocked stage to a root pass, as config already does

A promotion the agent notices in seconds could take an hour to serve. The
adopt loops run inside carlos-edge.service (ProtectSystem=strict, no /opt
write), so canaryAdoptLoop resolves a freshly promoted pointer within 2-3
seconds and can stage none of it — the only process that can write the
version store is the hourly root pass on carlos-adopt.timer. The fast loop's
whole purpose is defeated by the one step it is forbidden.

Config already solved this: configGenLoop touches /data/config-sync-requested,
carlos-adopt.path relays it into carlos-config-sync.service, and the root pass
does the half the agent cannot. This is the same relay for releases.

- release.OnStageBlocked — a PassOption reporting a version that could not be
  staged BECAUSE THIS UID CANNOT WRITE (EROFS/EPERM/EACCES only). Deliberately
  narrow: a bucket error, a bad signature or a poisoned version fails
  identically as root, and firing on those would relay an outage into a root
  pass every tick. Unset for every existing caller, so behaviour is unchanged
  wherever it is not passed.
- stageRequester touches /data/stage-requested, throttled to one request a
  minute per loop. The throttle is load-bearing in a way the config marker's
  is not: a config etag moves rarely, but a blocked stage repeats on EVERY
  tick of a 2-3 second loop, so an untouched relay would ask systemd for ~30
  root passes a minute, each re-blocking behind the last.
- carlos-stage.path -> carlos-stage.service, running `carlos adopt
  -stage-only`. Not carlos-adopt.service, for exactly the reason
  carlos-adopt.path does not point there either: that unit prints the
  canary-TTL nag, an HOURLY visibility feature, and this trigger fires per
  request. Everything else the root pass does is kept — config sync and the
  capability converge are also root-only, and a version staged now wants its
  unit shape and env file converged with it, not an hour later.

The hourly timer remains the floor. A dropped or throttled request costs
latency, never correctness.

Found by deploying an app through the documented `carlos instances enable` +
`carlos instances create` flow: the route was minted automatically 45 seconds
after the record, the activator armed and tried to wake it, and it 503'd until
someone SSHed in and ran `systemctl start carlos-adopt.service` by hand.

Build, GOOS=darwin cross-compile, vet and all 20 test packages clean.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
James Adam ecc940b60fba34885c1f6a7c3c5a393997e44e3b parent 6a52ae0
8 files changed, +386 −7
  • internal/host/agent.go +77 −3
  • internal/host/bootstrap.go +5 −4
  • internal/host/host_test.go +43 −0
  • internal/host/stage_relay_test.go +79 −0
  • internal/host/units/carlos-stage.path +26 −0
  • internal/host/units/carlos-stage.service +52 −0
  • internal/release/adopt.go +44 −0
  • internal/release/stage_blocked_test.go +60 −0
diff --git a/internal/host/agent.go b/internal/host/agent.go
index 53743493c..9ad7091d8 100644
--- a/internal/host/agent.go
+++ b/internal/host/agent.go
@@ -243,7 +243,12 @@ func adoptLoop(reg *registry.Registry) {
// It could not converge a drop-in if it tried; all it would produce is
// a permission-denied line every hour. The capability converge belongs
// to RunAdopt (carlos-adopt.service, root) alone — see below.
- if err := release.Pass(ctx, reg, store, pub, optDir, restartRoute, release.BakeDir(dataDir())); err != nil {
+ //
+ // OnStageBlocked, by contrast, is exactly right here: it fires only
+ // when this sandboxed process cannot write the version store, and all
+ // it does is ask the root pass to do the part this one cannot.
+ if err := release.Pass(ctx, reg, store, pub, optDir, restartRoute,
+ release.BakeDir(dataDir()), release.OnStageBlocked(stageRequester(dataDir()))); err != nil {
log.Printf("agent: adopt pass: %v", err)
}
}
@@ -449,6 +454,53 @@ func requestRootSync(dd string) error {
[]byte(time.Now().UTC().Format(time.RFC3339)+"\n"), 0o644)
}
+// stageRequestMarker is the release half of configSyncMarker. The agent's
+// adopt loops run inside carlos-edge.service (ProtectSystem=strict, no /opt
+// write), so they can SEE a promotion within seconds and stage nothing; the
+// only process that can write the version store is the root pass on
+// carlos-adopt.timer, an hour away. Touching this file relays "there is a
+// version here I cannot stage" to carlos-stage.path, exactly as the config
+// half relays to carlos-adopt.path. Lives in the data dir beside its config
+// twin, outside the *.config.env namespace reapConfigEnv sweeps.
+const stageRequestMarker = "stage-requested"
+
+// stageRequestEvery throttles those requests. Unlike the config marker —
+// touched only when a generation etag MOVES, which is naturally rare — a
+// blocked stage repeats on every tick of a loop that runs every 2-3 seconds,
+// so an untouched throttle would ask systemd to start a root pass ~30 times a
+// minute and each pass would re-block behind the one before it. One request a
+// minute is far below the hourly floor it improves on and far above the rate a
+// pass takes to finish.
+const stageRequestEvery = time.Minute
+
+// stageRequester returns a release.OnStageBlocked callback that touches the
+// marker at most once per stageRequestEvery. The closure holds the clock, so
+// each loop gets its own budget: the canary loop asking does not silence the
+// hourly one, and neither can starve the other.
+//
+// Failures are logged and swallowed, like requestRootSync's: a request that
+// cannot be written costs latency, never correctness, because
+// carlos-adopt.timer remains the floor underneath it.
+func stageRequester(dd string) func(host, ver string, err error) {
+ var mu sync.Mutex
+ var last time.Time
+ return func(host, ver string, _ error) {
+ mu.Lock()
+ if !last.IsZero() && time.Since(last) < stageRequestEvery {
+ mu.Unlock()
+ return
+ }
+ last = time.Now()
+ mu.Unlock()
+ if err := os.WriteFile(filepath.Join(dd, stageRequestMarker),
+ []byte(time.Now().UTC().Format(time.RFC3339)+"\n"), 0o644); err != nil {
+ log.Printf("agent: cannot request root stage for %s %s: %v", host, ver, err)
+ return
+ }
+ log.Printf("agent: %s needs %s staged and this process cannot write the version store — asked for a root pass", host, ver)
+ }
+}
+
// configGenLoop is canaryAdoptLoop's shape applied to issue #44: a config
// set should converge like a canary adopts — seconds, not the next hour.
// Every ~3s it reads one small object per config-carrying app (the
@@ -605,7 +657,14 @@ func canaryAdoptLoop(reg *registry.Registry) {
// version left until the hourly root pass converges it — up to an
// hour. release.adoptHost prints one line per such repoint so the
// window is visible in the journal rather than silent.
- if err := release.PassFilter(ctx, reg, store, pub, optDir, restartRoute, isCanaryRoute, release.BakeDir(dataDir())); err != nil {
+ //
+ // OnStageBlocked IS wired here, and this is the loop it matters most
+ // for: a canary is promoted to be picked up in seconds, and staging is
+ // the one step of that pickup this process cannot perform. Throttled
+ // inside stageRequester — this tick is 2-3 seconds, the request is at
+ // most one a minute.
+ if err := release.PassFilter(ctx, reg, store, pub, optDir, restartRoute, isCanaryRoute,
+ release.BakeDir(dataDir()), release.OnStageBlocked(stageRequester(dataDir()))); err != nil {
log.Printf("agent: canary adopt pass: %v", err)
}
}
@@ -1182,6 +1241,7 @@ func RunAdopt(args []string) {
fs := flag.NewFlagSet("adopt", flag.ExitOnError)
regPath := fs.String("registry", filepath.Join(dataDir(), "registry.db"), "registry database")
configOnly := fs.Bool("config-only", false, "sync unit-backed config and restart changed routes only: no release adoption, no canary nag")
+ stageOnly := fs.Bool("stage-only", false, "adopt releases without the canary nag: the agent-requested pass (carlos-stage.service)")
_ = fs.Parse(args)
reg, err := registry.Open(*regPath)
if err != nil {
@@ -1259,7 +1319,21 @@ func RunAdopt(args []string) {
// ProtectSystem=strict sandbox and cannot write unit configuration at all,
// which is why they call Pass/PassFilter with BakeDir only and why both
// carry a comment saying never to add this one.
- if err := release.Pass(ctx, reg, store, pub, release.OptDir(), restartRoute, release.NagExpiredCanaries(os.Stdout), release.WithCapabilities(caps.Ops{}), release.BakeDir(dataDir())); err != nil {
+ //
+ // -stage-only drops the nag and keeps everything else. It is what
+ // carlos-stage.service runs when the agent relays a version it could not
+ // stage: that request arrives at most once a minute per loop, and the nag
+ // is an HOURLY visibility feature — firing it per request would flood the
+ // journal with the same expired-canary lines, which is precisely what the
+ // option's own doc comment says never to do. The capability converge and
+ // config sync stay: both are jobs only this root pass can do, and a
+ // newly staged version wants its unit shape and env file converged with
+ // it, not an hour later.
+ passOpts := []release.PassOption{release.WithCapabilities(caps.Ops{}), release.BakeDir(dataDir())}
+ if !*stageOnly {
+ passOpts = append(passOpts, release.NagExpiredCanaries(os.Stdout))
+ }
+ if err := release.Pass(ctx, reg, store, pub, release.OptDir(), restartRoute, passOpts...); err != nil {
log.Fatalf("adopt: %v", err)
}
diff --git a/internal/host/bootstrap.go b/internal/host/bootstrap.go
index 283fb3d10..039a92c22 100644
--- a/internal/host/bootstrap.go
+++ b/internal/host/bootstrap.go
@@ -43,13 +43,14 @@ const tmpfilesConf = "d /run/carlos 0750 " + serviceUser + " " + serviceUser + "
"d /run/carlos/apps 0750 " + serviceUser + " " + serviceUser + " -\n"
// enabledUnits are what a live bootstrap turns on: the always-on front door,
-// the hourly adoption timer, and the path unit that lets the agent ask for an
-// adoption pass NOW (carlos-adopt.path — the unit-backed half of config fast
-// convergence, issue #44). The litestream + restore-verify units install
+// the hourly adoption timer, and the two path units that let the sandboxed
+// agent ask a root pass for the halves it cannot do itself: carlos-adopt.path
+// (config fast convergence, issue #44) and carlos-stage.path (release staging
+// — the agent sees a promotion in seconds but cannot write /opt). The litestream + restore-verify units install
// everywhere but stay inert until an operator stages /data/litestream.env and
// generates /etc/litestream.yml (`carlos ops litestream`) — a box with no
// replication bucket is unaffected.
-var enabledUnits = []string{"carlos-edge.service", "carlos-adopt.timer", "carlos-adopt.path"}
+var enabledUnits = []string{"carlos-edge.service", "carlos-adopt.timer", "carlos-adopt.path", "carlos-stage.path"}
// systemctlPlan is the exact systemctl sequence a live bootstrap runs — and
// the same sequence a staging run (-root <dir>) PRINTS instead of running, so
diff --git a/internal/host/host_test.go b/internal/host/host_test.go
index fcca235b5..960f7df3a 100644
--- a/internal/host/host_test.go
+++ b/internal/host/host_test.go
@@ -35,6 +35,8 @@ var expectedUnits = []string{
"carlos-litestream.service",
"carlos-restore-verify.service",
"carlos-restore-verify.timer",
+ "carlos-stage.path",
+ "carlos-stage.service",
}
// requireUnixSockets skips a test when the sandbox forbids unix-socket
@@ -158,6 +160,47 @@ func TestBootstrapWritesUnitsToStagingDir(t *testing.T) {
t.Error("carlos-adopt.path must not start carlos-adopt.service — that unit adopts releases and nags on expired canaries")
}
+ // The release fast-convergence relay, and the same invariant: which unit
+ // the path starts. carlos-adopt.service prints the canary-TTL nag, an
+ // HOURLY visibility feature; this trigger fires on agent request, so
+ // aiming it there would repeat the same expired-canary lines every time a
+ // promotion could not be staged.
+ stagePath, err := os.ReadFile(filepath.Join(unitDir, "carlos-stage.path"))
+ if err != nil {
+ t.Fatal(err)
+ }
+ for _, want := range []string{
+ "PathChanged=/data/stage-requested",
+ "Unit=carlos-stage.service",
+ "WantedBy=multi-user.target",
+ } {
+ if !strings.Contains(string(stagePath), want) {
+ t.Errorf("carlos-stage.path: missing %q", want)
+ }
+ }
+ if strings.Contains(string(stagePath), "Unit=carlos-adopt.service") {
+ t.Error("carlos-stage.path must not start carlos-adopt.service — that unit nags on expired canaries, and this trigger is per-request")
+ }
+
+ // The staging pass must run -stage-only, and must NOT be startable by
+ // anything but its path unit.
+ stageSvc, err := os.ReadFile(filepath.Join(unitDir, "carlos-stage.service"))
+ if err != nil {
+ t.Fatal(err)
+ }
+ if !strings.Contains(string(stageSvc), "carlos adopt -stage-only") {
+ t.Error("carlos-stage.service must run `carlos adopt -stage-only` — the full pass would nag per request")
+ }
+ // A SECTION header, not any mention: the unit's own comment explains why
+ // it has no [Install], and a substring match would fail on the
+ // explanation. Same shape as carlos-config-sync.service, which documents
+ // the same choice the same way.
+ for _, line := range strings.Split(string(stageSvc), "\n") {
+ if strings.TrimSpace(line) == "[Install]" {
+ t.Error("carlos-stage.service must have no [Install] section: only carlos-stage.path should ever start it")
+ }
+ }
+
syncUnit, err := os.ReadFile(filepath.Join(unitDir, "carlos-config-sync.service"))
if err != nil {
t.Fatal(err)
diff --git a/internal/host/stage_relay_test.go b/internal/host/stage_relay_test.go
new file mode 100644
index 000000000..66a3ab4a1
--- /dev/null
+++ b/internal/host/stage_relay_test.go
@@ -0,0 +1,79 @@
+package host
+
+import (
+ "errors"
+ "os"
+ "path/filepath"
+ "strings"
+ "syscall"
+ "testing"
+ "time"
+)
+
+// The relay exists because the agent's adopt loops run inside
+// carlos-edge.service (ProtectSystem=strict, no /opt write): they resolve a
+// promotion in seconds and can stage none of it, while the only process that
+// can runs hourly. These tests pin the two halves that make the relay safe —
+// it fires, and it does not fire too often.
+
+func TestStageRequesterWritesMarker(t *testing.T) {
+ dd := t.TempDir()
+ req := stageRequester(dd)
+ req("echo.example.com", "abc1234", syscall.EROFS)
+
+ b, err := os.ReadFile(filepath.Join(dd, stageRequestMarker))
+ if err != nil {
+ t.Fatalf("marker not written: %v", err)
+ }
+ // A timestamp, like requestRootSync's: systemd watches the path, not the
+ // bytes, but an operator reading /data should see when the last request
+ // went out.
+ if _, err := time.Parse(time.RFC3339, strings.TrimSpace(string(b))); err != nil {
+ t.Errorf("marker content %q is not an RFC3339 timestamp: %v", b, err)
+ }
+}
+
+// The throttle is the whole reason this is safe to wire into a loop that ticks
+// every 2-3 seconds. Without it a blocked stage — which repeats every tick,
+// unlike a config etag moving — would ask systemd to start a root pass ~30
+// times a minute, each one re-blocking behind the last.
+func TestStageRequesterThrottles(t *testing.T) {
+ dd := t.TempDir()
+ req := stageRequester(dd)
+ marker := filepath.Join(dd, stageRequestMarker)
+
+ req("echo.example.com", "abc1234", syscall.EROFS)
+ first, err := os.Stat(marker)
+ if err != nil {
+ t.Fatalf("first request wrote nothing: %v", err)
+ }
+ if err := os.Remove(marker); err != nil {
+ t.Fatal(err)
+ }
+ // Every subsequent request inside the window must be dropped — proven by
+ // the marker not coming back after we removed it.
+ for i := 0; i < 20; i++ {
+ req("echo.example.com", "abc1234", syscall.EROFS)
+ }
+ if _, err := os.Stat(marker); !errors.Is(err, os.ErrNotExist) {
+ t.Errorf("throttle let a second request through inside %v (marker recreated)", stageRequestEvery)
+ }
+ _ = first
+}
+
+// Each loop gets its own budget: the canary loop asking must not silence the
+// hourly one. stageRequester returns a fresh closure per call, so this is a
+// property of the API shape rather than of the clock.
+func TestStageRequesterBudgetIsPerRequester(t *testing.T) {
+ dd := t.TempDir()
+ marker := filepath.Join(dd, stageRequestMarker)
+
+ stageRequester(dd)("a.example.com", "v1", syscall.EROFS)
+ if err := os.Remove(marker); err != nil {
+ t.Fatal(err)
+ }
+ stageRequester(dd)("b.example.com", "v1", syscall.EROFS)
+ if _, err := os.Stat(marker); err != nil {
+ t.Errorf("a second requester was throttled by the first's clock: %v", err)
+ }
+}
diff --git a/internal/host/units/carlos-stage.path b/internal/host/units/carlos-stage.path
new file mode 100644
index 000000000..f13cc0c19
--- /dev/null
+++ b/internal/host/units/carlos-stage.path
@@ -0,0 +1,26 @@
+# Fast release convergence — carlos-adopt.path's twin for the OTHER half the
+# sandboxed agent cannot do itself. The agent's adoptLoop/canaryAdoptLoop see a
+# promotion within seconds but run inside carlos-edge.service
+# (ProtectSystem=strict, no /opt write), so they can resolve the new pointer
+# and stage nothing; release.OnStageBlocked touches this file when that
+# happens. Without it a promote is invisible for up to an hour on a box that
+# noticed it immediately — the canary loop's 2-second cadence buys nothing,
+# because the step it is racing to perform is the one it is forbidden.
+#
+# It starts carlos-stage.service — `carlos adopt -stage-only` — deliberately
+# NOT carlos-adopt.service: that unit also prints the canary-TTL nag, an
+# HOURLY visibility feature, and this trigger fires per agent request. Same
+# split, and the same reason, as carlos-adopt.path pointing at
+# carlos-config-sync.service rather than the full pass.
+#
+# /data is hardcoded like every path in these unit files — a box with a
+# nonstandard data dir simply never fast-triggers and keeps the hourly floor.
+[Unit]
+Description=CARLOS release staging on agent request (fast release convergence)
+
+[Path]
+PathChanged=/data/stage-requested
+Unit=carlos-stage.service
+
+[Install]
+WantedBy=multi-user.target
diff --git a/internal/host/units/carlos-stage.service b/internal/host/units/carlos-stage.service
new file mode 100644
index 000000000..8ec23d0de
--- /dev/null
+++ b/internal/host/units/carlos-stage.service
@@ -0,0 +1,52 @@
+# One release-adoption pass WITHOUT the canary-TTL nag — the privileged half
+# of fast release convergence. Started only by carlos-stage.path, which fires
+# when the agent touches /data/stage-requested; it has no [Install] section
+# because nothing else should ever want it. Same root + hardening shape as
+# carlos-adopt.service, which is what writing /opt/carlos/versions, moving
+# live/<host> and restarting units requires.
+#
+# Deliberately NOT carlos-adopt.service, and the reason is the nag alone: that
+# unit prints one line per expired canary/* route, an HOURLY visibility
+# feature, and this trigger fires on agent request (throttled to one a minute
+# per loop, not once an hour). Pointing the relay at the full pass would flood
+# the journal with the same expired-canary lines — exactly what
+# NagExpiredCanaries' own doc comment says never to do. Everything else the
+# hourly pass does is KEPT: config sync and the capability converge are both
+# jobs only a root pass can perform, and a version staged now wants its unit
+# shape and env file converged with it rather than an hour later.
+[Unit]
+Description=carlos release staging (on agent request, no canary nag)
+After=network-online.target
+Wants=network-online.target
+# A path unit re-triggers as often as the agent asks; the default start rate
+# limit would latch this unit into start-limit-hit after a burst and then
+# ignore every later trigger until an operator reset it. ([Unit], not
+# [Service] — systemd moved these two keys in v230.)
+StartLimitIntervalSec=0
+
+[Service]
+Type=oneshot
+# CARLOS_DEPLOYMENT_BUCKET/_DIR + CARLOS_RELEASE_PUBKEY live here.
+EnvironmentFile=-/etc/carlos/host.env
+ExecStart=/opt/carlos/carlos adopt -stage-only
+
+# --- hardening ---
+# Runs as ROOT deliberately, exactly like carlos-adopt.service: a pass writes
+# the version store (/opt/carlos) and restarts instance units via systemctl.
+# NoNewPrivileges and the syscall filter are omitted for the systemctl child;
+# the rest applies.
+ProtectHome=yes
+PrivateTmp=yes
+PrivateDevices=yes
+ProtectClock=yes
+ProtectHostname=yes
+ProtectKernelTunables=yes
+ProtectKernelModules=yes
+ProtectKernelLogs=yes
+RestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX
+RestrictNamespaces=yes
+RestrictRealtime=yes
+LockPersonality=yes
+MemoryDenyWriteExecute=yes
+MemoryMax=256M
+TasksMax=64
diff --git a/internal/release/adopt.go b/internal/release/adopt.go
index f76438e33..3cac99d93 100644
--- a/internal/release/adopt.go
+++ b/internal/release/adopt.go
@@ -13,6 +13,7 @@ import (
"io"
"os"
"runtime"
+ "syscall"
"time"
"github.com/carlosframework/platform/internal/caps"
@@ -44,6 +45,12 @@ type passConfig struct {
// bakeDir, when non-empty, is the directory holding the bake clock
// (BakeDir option); empty keeps the legacy /opt-dir location.
bakeDir string
+
+ // stageBlocked is called when staging fails because this process cannot
+ // write the version store (OnStageBlocked). nil — every existing caller
+ // and every existing test — means the pass reports such a failure exactly
+ // as it always has and nothing else happens.
+ stageBlocked func(host, ver string, err error)
}
// NagExpiredCanaries opts a Pass/PassFilter call into spec §5's nag: one
@@ -119,6 +126,37 @@ func BakeDir(dir string) PassOption {
return func(c *passConfig) { c.bakeDir = dir }
}
+// OnStageBlocked reports a version this pass could not stage because the
+// process cannot WRITE the version store — never because the bucket, the
+// signature or the manifest was bad. It is the release half of the relay
+// carlos-adopt.path already runs for config: the agent's in-process loops are
+// sandboxed (carlos-edge.service sets ProtectSystem=strict with no /opt write),
+// so they can see a promotion within seconds and stage nothing, while the only
+// process that CAN stage runs hourly off carlos-adopt.timer. Without a relay a
+// promote is invisible for up to an hour on a box that noticed it immediately.
+//
+// Deliberately narrow. A read failure, a bad signature or a poisoned version
+// must NOT fire this: none of them is fixed by re-running as root, and a
+// callback that fired on any staging error would relay a bucket outage into a
+// root pass every tick. Only EROFS/EPERM/EACCES — "this uid cannot write
+// there" — describe a job a privileged pass would actually finish.
+//
+// Never fatal, and never a substitute for the timer: the hourly pass remains
+// the floor, exactly as it is for config. A dropped or throttled request costs
+// latency, not correctness.
+func OnStageBlocked(fn func(host, ver string, err error)) PassOption {
+ return func(c *passConfig) { c.stageBlocked = fn }
+}
+
+// stageBlocked reports whether err is the version store refusing a write to
+// THIS uid, rather than any of the many other ways staging fails. errors.Is
+// unwraps the *os.PathError that mkdir/openat return.
+func stageBlocked(err error) bool {
+ return errors.Is(err, syscall.EROFS) ||
+ errors.Is(err, syscall.EPERM) ||
+ errors.Is(err, syscall.EACCES)
+}
+
// Pass is one full adoption sweep: group instance routes by (App, Channel),
// resolve each channel pointer once, verify signatures and artifact sha256s,
// download/stage missing versions, and repoint live/<host> symlinks.
@@ -532,6 +570,12 @@ func adoptHost(ctx context.Context, reg *registry.Registry, s Store, optDir stri
m, err := ensureStaged(ctx, s, optDir, RouteAccount(rt), rt.App, ver, rt.Host)
if err != nil {
fmt.Printf(" %s: %v\n", rt.Host, err)
+ // A version this uid cannot write is the one staging failure a
+ // privileged pass would finish; everything else (bucket, signature,
+ // manifest) fails the same way as root. See OnStageBlocked.
+ if cfg.stageBlocked != nil && stageBlocked(err) {
+ cfg.stageBlocked(rt.Host, ver, err)
+ }
return
}
diff --git a/internal/release/stage_blocked_test.go b/internal/release/stage_blocked_test.go
new file mode 100644
index 000000000..2ef086258
--- /dev/null
+++ b/internal/release/stage_blocked_test.go
@@ -0,0 +1,60 @@
+package release
+
+import (
+ "errors"
+ "fmt"
+ "io/fs"
+ "os"
+ "syscall"
+ "testing"
+)
+
+// stageBlocked is deliberately narrow: it must match "this uid cannot write
+// the version store" and nothing else. A callback that fired on any staging
+// error would relay a bucket outage into a root pass on every tick — and none
+// of those failures is fixed by re-running as root.
+func TestStageBlockedMatchesOnlyWriteRefusals(t *testing.T) {
+ for _, tc := range []struct {
+ name string
+ err error
+ want bool
+ }{
+ // The production shape: mkdir under a ProtectSystem=strict sandbox.
+ {"EROFS via PathError", &os.PathError{Op: "mkdir", Path: "/opt/carlos/versions/abc", Err: syscall.EROFS}, true},
+ {"EACCES via PathError", &os.PathError{Op: "openat", Path: "/opt/carlos/versions/abc", Err: syscall.EACCES}, true},
+ {"EPERM via PathError", &os.PathError{Op: "mkdir", Path: "/opt/carlos/versions/abc", Err: syscall.EPERM}, true},
+ // Wrapped, because the pass wraps before it reports.
+ {"wrapped EROFS", fmt.Errorf("stage abc: %w", &os.PathError{Op: "mkdir", Err: syscall.EROFS}), true},
+
+ // Everything a root pass would fail at identically.
+ {"not exist", fs.ErrNotExist, false},
+ {"bad signature", errors.New("manifest signature does not verify"), false},
+ {"sha mismatch", errors.New("artifact sha256 mismatch"), false},
+ {"bucket 403", errors.New("operation error S3: GetObject, api error AccessDenied"), false},
+ {"nil", nil, false},
+ } {
+ t.Run(tc.name, func(t *testing.T) {
+ if got := stageBlocked(tc.err); got != tc.want {
+ t.Errorf("stageBlocked(%v) = %v, want %v", tc.err, got, tc.want)
+ }
+ })
+ }
+}
+
+// The option must be opt-in: every existing caller and test passes no
+// OnStageBlocked, and those calls must behave exactly as they always have.
+func TestOnStageBlockedUnsetByDefault(t *testing.T) {
+ var c passConfig
+ if c.stageBlocked != nil {
+ t.Fatal("passConfig.stageBlocked must default to nil")
+ }
+ called := false
+ OnStageBlocked(func(string, string, error) { called = true })(&c)
+ if c.stageBlocked == nil {
+ t.Fatal("OnStageBlocked did not set the hook")
+ }
+ c.stageBlocked("h", "v", syscall.EROFS)
+ if !called {
+ t.Error("hook not invoked")
+ }
+}