amadan

carlos / platform Public

plan: claim Plan 6b rev 2, answers review round 1 (four P2) and rebases on main

Co-Authored-By: Codex <noreply@openai.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
codex pushed by claude 5cab972d3bfff8f558bddb7c5d5253f63366d9e1 parent 2d67ec4
1 file changed, +3042 −380
  • docs/superpowers/plans/2026-10-06-host-lifecycle-claim-6b-box-assembly.md +3042 −380

CI passed — run details

diff --git a/docs/superpowers/plans/2026-10-06-host-lifecycle-claim-6b-box-assembly.md b/docs/superpowers/plans/2026-10-06-host-lifecycle-claim-6b-box-assembly.md
index 1ac156d80..c9a14d847 100644
--- a/docs/superpowers/plans/2026-10-06-host-lifecycle-claim-6b-box-assembly.md
+++ b/docs/superpowers/plans/2026-10-06-host-lifecycle-claim-6b-box-assembly.md
@@ -1,10 +1,12 @@
# Host lifecycle claim, Plan 6b: production box assembly
-Implementation plan, 2026-10-06, rev 1. Build on `feat/hostclaim-box-assembly`, cut from `feat/hostclaim-box` at **7cc781600f69e25b61ad2ba02ccf2a880b0266de**. That parent contains `b4910ff2`, the fix round in `3b8539a5`, the subsequent main merge, and the final flag-off fixes in `7cc78160`. This document supplies the next implementation steps. Authoring it stages nothing, commits nothing and creates no implementation branch.
+Implementation plan, 2026-10-07, rev 2. Build on `feat/hostclaim-box-assembly`, cut from `origin/main` at **2b2cc879a669b850bf8ec410b13d20673709477d**. Plan 6 landed as `d649d8d4`; this base includes `84c80eba`, region HA plan 1b and the subsequent main changes. This document supplies the next implementation steps. Authoring it stages nothing, commits nothing and creates no implementation branch.
-> For agentic workers: apply the tasks in order, tests before implementation. Keep `CARLOS_HOST_CLAIM` unset. Do not execute branch creation or deployment commands in the design worktree. The code blocks below are complete new files or exact patches against the pinned parent. Use the extractor below to apply patches. Missing authority refuses the dependent operation before an effect.
+Rev 2 answers all four round-1 findings: P2-1 moves inbound, config-gate and co-spawn refusals before admission effects (Task 4); P2-2 retains the persisted attempt's boot/controller tuple and routes restart recovery through root's dead-controller proof (Task 7); P2-3 journals recoverable pre-launch state before restore or materialization (Tasks 3, 5 and 7); P2-4 rebases every block onto current main and retains `boxid.Verifier.Enroll` and main's region HA readers (Tasks 1 and 6). Task 2 supplies regression tests; the authoring verification records the executed failures and gates.
-The work covers review findings 1 through 4 in `.superpowers/sdd/2026-10-05-host-lifecycle-claim-6-box-wiring/review-610.md` from the `box` worktree: startup composition, durable wake retries, reconcile admission and retirement, and root evidence and role launches. It uses claim **rev 6.30**, especially section 1.1 and the ruled P6-D1 through P6-D8 in section 6.6. Plan 6 rev 4 remains the implementation history of the parent. Do not replay it.
+> For agentic workers: apply the tasks in order, tests before implementation. Keep `CARLOS_HOST_CLAIM` unset. Do not execute branch creation or deployment commands in the design worktree. The code blocks below are complete new files or exact patches against the pinned main base. Use the extractor below to apply patches. Missing authority refuses the dependent operation before an effect.
+
+The work covers review findings 1 through 4 in `.superpowers/sdd/2026-10-05-host-lifecycle-claim-6-box-wiring/review-610.md` from the `box` worktree: startup composition, durable wake retries, reconcile admission and retirement, and root evidence and role launches. It uses claim **rev 6.30**, especially section 1.1 and the ruled P6-D1 through P6-D8 in section 6.6. Plan 6 rev 4 remains implementation history; its landed code is already on main. Do not replay it.
## Boundaries and required providers
@@ -40,12 +42,13 @@ Run these commands only in the later implementation checkout:
```bash
git status --short
-git rev-parse feat/hostclaim-box
-git merge-base --is-ancestor 7cc781600f69e25b61ad2ba02ccf2a880b0266de feat/hostclaim-box
-git switch -c feat/hostclaim-box-assembly feat/hostclaim-box
+git fetch origin main
+git rev-parse origin/main
+git merge-base --is-ancestor 84c80eba origin/main
+git switch -c feat/hostclaim-box-assembly origin/main
```
-Expected: a clean implementation checkout, the fix round included, and the new assembly branch. If the parent has moved, compare its changes before applying any patch. Preserve the captured callback, enabled config-error and cleanup-latch fixes in `3b8539a5`, the layout integration from the main merge, and the flag-off schedule and host fixes in `7cc78160`.
+Expected: a clean implementation checkout and the new assembly branch from the latest fetched main. These blocks were verified at `2b2cc879a669b850bf8ec410b13d20673709477d`. If main has moved, compare it with that base and refresh the patches before applying them. Preserve main's `boxid.Verifier.Enroll`, `termfence`, `systemparticipation.go`, `boxrecord`, `edgesclaims` and `boxvolumes`; do not replay region HA plan 1b or reintroduce its superseded readers. Preserve the existing flag-off and hostbox fixes, including main's nested-default layout and startup layout recovery. The enabled claim path remains limited to its explicitly supported launch shapes.
Read these exact local sources before changing code:
@@ -62,7 +65,7 @@ The source paths, not these line numbers, are authoritative if another documenta
## Task 2: install the regression tests first
-The extractor applies the exact file blocks in this document. Run it once with `tests`, observe the failure, then run it with `implementation` after reading Tasks 3 through 8. New files are complete. Existing files use exact zero-context patches against the pinned parent. A literal `\u2014` in a patch encodes existing parent punctuation; the extractor restores that character without changing the original string or comment. This is not an invitation to rewrite product copy.
+The extractor applies the exact file blocks in this document. Run it once with `tests`, observe the failure, then run it with `implementation` after reading Tasks 3 through 8. New files are complete. Existing files use exact zero-context patches against the pinned main base. A literal `\u2014` in a patch encodes existing parent punctuation; the extractor restores that character without changing the original string or comment. This is not an invitation to rewrite product copy.
Save this extractor as `$TMPDIR/claim6b-apply.py` in the implementation session, using an on-disk `TMPDIR`. It does not stage or commit anything.
@@ -99,7 +102,7 @@ for kind, name, body in blocks:
```bash
python3 "$TMPDIR/claim6b-apply.py" docs/superpowers/plans/2026-10-06-host-lifecycle-claim-6b-box-assembly.md tests
-go test ./internal/registry ./internal/hostbox ./internal/activator ./internal/host ./internal/provision ./internal/console ./internal/instancectl ./internal/schedule -run '^Test(Claim|ServingRetries|Renewing|AggregateContext|IdentitySeparates|EmptyCensus|RegistryRows|ColdHandover|HostWakeReset|RootRetirement|BeginFence|Cleanup|Fence|StartAdmitted|SameAdmission|SetupAt)' -count=1
+nice -n 10 go test -p 1 ./internal/registry ./internal/hostbox ./internal/activator ./internal/host ./internal/provision ./internal/console ./internal/instancectl ./internal/schedule -run '^Test(Claim|ServingRetries|Renewing|AggregateContext|Identity|AdapterSelectionLayout|Launch|Prelaunch|PreLaunch|Recover|EmptyCensus|RegistryRows|ColdHandover|HostWakeReset|RootRetirement|BeginFence|Cleanup|Fence|StartAdmitted|SameAdmission|SetupAt)' -count=1
```
Expected red: missing assembly, retry, journal, registry-ownership, delivery-admission and handover APIs. These tests must be present before the implementations below. No passing test that used an unavailable cgroup delegation counts as a kernel witness.
@@ -1474,7 +1477,19 @@ func TestEmptyCensusRefusesUnclassifiedArtifactsAndNeverWritesInPrecheck(t *test
```go
package hostbox
-import "testing"
+import (
+ "context"
+ "crypto"
+ "crypto/rand"
+ "crypto/rsa"
+ "crypto/sha256"
+ "crypto/x509"
+ "encoding/base64"
+ "encoding/json"
+ "errors"
+ "github.com/carlosframework/platform/internal/boxid"
+ "testing"
+)
func TestIdentitySeparatesPlacementAndCloudRegion(t *testing.T) {
p := IdentityProvider{Account: "123", Box: "box", Region: "ie", AWSRegion: "eu-west-1", Bucket: "data", Namespace: "platform"}
@@ -1487,6 +1502,66 @@ func TestIdentitySeparatesPlacementAndCloudRegion(t *testing.T) {
t.Fatal("foreign cloud region admitted")
}
}
+
+type enrollmentRoster struct{ actor Actor }
+
+func (r enrollmentRoster) Actor(context.Context, string) (Actor, error) { return r.actor, nil }
+
+type enrollmentEvidence []byte
+
+func (e enrollmentEvidence) Load(context.Context, string) ([]byte, error) { return e, nil }
+
+func TestIdentityEnrollmentBindsSignedCloudAndPlacement(t *testing.T) {
+ key, err := rsa.GenerateKey(rand.Reader, 2048)
+ if err != nil {
+ t.Fatal(err)
+ }
+ cert := &x509.Certificate{PublicKey: &key.PublicKey, PublicKeyAlgorithm: x509.RSA}
+ actor := Actor{Box: "annie-welding", Enrollment: "en", Revision: 1, Region: "ie", Bucket: "data", Namespace: "platform", Eligible: true, Cloud: CloudIdentity{Provider: "aws", Account: "111122223333", Region: "eu-west-1", InstanceID: "i-0abc"}}
+ p := IdentityProvider{Account: actor.Cloud.Account, Box: actor.Box, Region: actor.Region, AWSRegion: actor.Cloud.Region, Bucket: actor.Bucket, Namespace: actor.Namespace, Roster: enrollmentRoster{actor}, Certs: map[string]*x509.Certificate{"eu-west-1": cert}}
+ record := func(box, account, region, instance string) []byte {
+ t.Helper()
+ doc, err := json.Marshal(map[string]string{"accountId": account, "region": region, "instanceId": instance})
+ if err != nil {
+ t.Fatal(err)
+ }
+ sum := sha256.Sum256(doc)
+ sig, err := rsa.SignPKCS1v15(rand.Reader, key, crypto.SHA256, sum[:])
+ if err != nil {
+ t.Fatal(err)
+ }
+ raw, err := json.Marshal(boxid.Record{V: 1, Box: box, IdentityDocument: base64.StdEncoding.EncodeToString(doc), IdentitySignature: base64.StdEncoding.EncodeToString(sig)})
+ if err != nil {
+ t.Fatal(err)
+ }
+ return raw
+ }
+ good := record(actor.Box, actor.Cloud.Account, actor.Cloud.Region, actor.Cloud.InstanceID)
+ p.Evidence = enrollmentEvidence(good)
+ if got, err := p.Verify(t.Context()); err != nil || got != actor {
+ t.Fatalf("enrollment: got=%+v err=%v", got, err)
+ }
+ for name, raw := range map[string][]byte{
+ "other instance": record(actor.Box, actor.Cloud.Account, actor.Cloud.Region, "i-other"),
+ "other box": record("bert-sawing", actor.Cloud.Account, actor.Cloud.Region, actor.Cloud.InstanceID),
+ "other account": record(actor.Box, "999999999999", actor.Cloud.Region, actor.Cloud.InstanceID),
+ "other region": record(actor.Box, actor.Cloud.Account, "eu-west-2", actor.Cloud.InstanceID),
+ "malformed": []byte("{}"),
+ } {
+ t.Run(name, func(t *testing.T) {
+ q := p
+ q.Evidence = enrollmentEvidence(raw)
+ if _, err := q.Verify(t.Context()); !errors.Is(err, boxid.ErrUnverified) {
+ t.Fatalf("unbound evidence accepted: %v", err)
+ }
+ })
+ }
+ q := p
+ q.AWSRegion = "eu-west-2"
+ if _, err := q.Verify(t.Context()); !errors.Is(err, ErrChanged) {
+ t.Fatalf("wrong cloud placement accepted: %v", err)
+ }
+}
```
#### New file: `internal/hostbox/launchjournal_test.go`
@@ -1548,6 +1623,72 @@ func TestLaunchJournalDurableExactReservations(t *testing.T) {
t.Fatal("successor read predecessor inventory")
}
}
+
+func TestLaunchJournalPrelaunchRecordIsRecoverableWithoutEffects(t *testing.T) {
+ ctx := context.Background()
+ file := filepath.Join(t.TempDir(), "launch.db")
+ j, err := OpenLaunchJournal(file)
+ if err != nil {
+ t.Fatal(err)
+ }
+ former := hostguard.FenceTarget{Host: "app.example.test", Incarnation: "inc", Stamp: 7, Session: 7, Box: "box", Enrollment: "enroll", BootID: "boot-1", ControllerID: "controller-1"}
+ parent := LaunchGroup{RelativePath: "carlos/claim-parent", Device: 1, Inode: 9}
+ for _, bad := range []LaunchInventory{
+ {ControllerPID: 41, ProcessStart: "start-1", Target: former, Groups: []LaunchGroup{parent}, Files: []LaunchFile{{Device: 1, Inode: 2}}},
+ {ControllerPID: 41, ProcessStart: "start-1", Target: former, Groups: []LaunchGroup{parent}, NoState: true, ProducerIDs: []string{"replicator"}},
+ {ControllerPID: 41, ProcessStart: "start-1", Target: former, Groups: []LaunchGroup{parent, {RelativePath: "carlos/claim-parent/role", Device: 1, Inode: 10}}, NoState: true},
+ {ControllerPID: 41, ProcessStart: "start-1", Target: former, NoState: true},
+ } {
+ if err = j.Prelaunch(ctx, bad); err == nil {
+ t.Fatalf("pre-launch record accepted effects or missing containment: %+v", bad)
+ }
+ }
+ if err = j.Prelaunch(ctx, LaunchInventory{ControllerPID: 41, ProcessStart: "start-1", Target: former, Groups: []LaunchGroup{parent}, NoState: true}); err != nil {
+ t.Fatal(err)
+ }
+ successor := former
+ successor.Session, successor.Stamp, successor.BootID, successor.ControllerID = 8, 8, "boot-2", "controller-2"
+ if err = j.Prelaunch(ctx, LaunchInventory{ControllerPID: 42, ProcessStart: "start-2", Target: successor, Groups: []LaunchGroup{{RelativePath: "carlos/claim-successor", Device: 1, Inode: 11}}, NoState: true}); err != nil {
+ t.Fatal(err)
+ }
+ if err = j.Close(); err != nil {
+ t.Fatal(err)
+ }
+ // A restarted controller reopens the journal and finds the former tuple.
+ j, err = OpenLaunchJournal(file)
+ if err != nil {
+ t.Fatal(err)
+ }
+ defer j.Close()
+ found, err := j.SessionLaunches(ctx, former.Host, former.Incarnation, former.Stamp, former.Session, former.Box, former.Enrollment)
+ if err != nil || len(found) != 1 || !SameLaunchOwner(found[0].Target, former) || found[0].ControllerPID != 41 || found[0].ProcessStart != "start-1" {
+ t.Fatalf("session records = %+v, %v", found, err)
+ }
+ stop, err := j.ReadStopKernelJournal(ctx, former)
+ if err != nil || !stop.Settled || len(stop.Cgroups) != 1 || len(stop.Files) != 0 || len(stop.ProducerIDs) != 0 {
+ t.Fatalf("pre-launch stop journal = %+v, %v", stop, err)
+ }
+ // The record is not an installation effect, so cold handover still accepts it.
+ if effects, err := j.HasInstallationEffects(ctx, former.Host, former.Incarnation, former.Stamp); err != nil || effects {
+ t.Fatalf("pre-launch record counted as installation effect: %v %v", effects, err)
+ }
+ if err = j.SettleRecovered(ctx, former); err != nil {
+ t.Fatal(err)
+ }
+ other, err := j.Read(ctx, successor)
+ if err != nil || len(other.Reservations) != 0 || !other.NoState {
+ t.Fatalf("recovering the former tuple changed its successor: %+v %v", other, err)
+ }
+ if found, err = j.SessionLaunches(ctx, former.Host, former.Incarnation, former.Stamp, 99, former.Box, former.Enrollment); err != nil || len(found) != 0 {
+ t.Fatalf("unrelated session matched: %+v %v", found, err)
+ }
+ if err = j.RecordFile(ctx, former, LaunchFile{Device: 1, Inode: 12}); err != nil {
+ t.Fatal(err)
+ }
+ if got, err := j.Read(ctx, former); err != nil || got.NoState || len(got.Files) != 1 {
+ t.Fatalf("launch files after pre-launch = %+v %v", got, err)
+ }
+}
```
#### New file: `internal/hostbox/registryrows_test.go`
@@ -1806,8 +1947,10 @@ package provision
import (
"context"
+ "errors"
"github.com/carlosframework/platform/internal/registry"
"github.com/carlosframework/platform/internal/release"
+ "os"
"path/filepath"
"testing"
)
@@ -1924,6 +2067,493 @@ func TestClaimInstallGuardsMintAndKeepInBothPasses(t *testing.T) {
})
}
}
+
+// claimEffects stands in for the production ClaimInstall hook's durable
+// effects: activating the host guard and writing the inventory. Each call
+// leaves a sentinel file for both, so a pass that reached the hook cannot
+// look untouched, whatever the hook went on to decide.
+type claimEffects struct {
+ dir string
+ owner registry.ClaimOwnership
+ calls int
+}
+
+func newClaimEffects(t *testing.T) *claimEffects {
+ t.Helper()
+ return &claimEffects{dir: t.TempDir(), owner: registry.ClaimOwnership{ClaimIncarnation: "inc", ClaimStamp: 1, ClaimOwner: "owner", ClaimRowID: "row"}}
+}
+
+func (e *claimEffects) hook(t *testing.T) func(context.Context, ClaimRecord, func(registry.ClaimOwnership) error) error {
+ return func(_ context.Context, rec ClaimRecord, write func(registry.ClaimOwnership) error) error {
+ e.calls++
+ for _, name := range []string{"guard.active", "inventory." + rec.Host} {
+ if err := os.WriteFile(filepath.Join(e.dir, name), []byte(rec.Key), 0o600); err != nil {
+ t.Fatal(err)
+ }
+ }
+ return write(e.owner)
+ }
+}
+
+// untouched fails unless the hook never ran: no call, no guard, no inventory.
+func (e *claimEffects) untouched(t *testing.T, why string) {
+ t.Helper()
+ left, err := os.ReadDir(e.dir)
+ if err != nil {
+ t.Fatal(err)
+ }
+ if e.calls != 0 || len(left) != 0 {
+ t.Fatalf("%s: claim hook ran before refusal: calls=%d sentinels=%d", why, e.calls, len(left))
+ }
+}
+
+// reached fails unless the hook ran exactly n times and left its sentinels,
+// proving the harness can observe the effect untouched() denies.
+func (e *claimEffects) reached(t *testing.T, n int, why string) {
+ t.Helper()
+ left, err := os.ReadDir(e.dir)
+ if err != nil {
+ t.Fatal(err)
+ }
+ if e.calls != n || len(left) == 0 {
+ t.Fatalf("%s: claim hook calls=%d (want %d) sentinels=%d", why, e.calls, n, len(left))
+ }
+}
+
+func instanceClaimFixture(t *testing.T) (release.Store, *registry.Registry, string, ReconcileConfig) {
+ t.Helper()
+ store, sqid := setup(t)
+ record := acmeRecord(sqid)
+ if err := CreateRecord(ctx, store, record); err != nil {
+ t.Fatal(err)
+ }
+ reg := regWith(t)
+ t.Cleanup(func() { reg.Close() })
+ return store, reg, record.Host, ReconcileConfig{ClaimEnabled: true, DataDir: t.TempDir(), Region: "ie", Pools: []string{"tito"}}
+}
+
+func sidecarClaimFixture(t *testing.T) (*sidecarFixture, string, ReconcileConfig) {
+ t.Helper()
+ f := sidecarSetup(t)
+ t.Cleanup(func() { f.reg.Close() })
+ f.putManifest(t, "eleven", SidecarManifest{Schema: 1, App: "eleven", Args: []string{"run"}, Channel: "stable"})
+ f.putRecord(t, SidecarRecord{Schema: 1, Name: "bot", App: "eleven", Channel: "stable"})
+ return f, f.host("bot", "eleven"), ReconcileConfig{ClaimEnabled: true, DataDir: f.dir}
+}
+
+func TestClaimRefusalsPrecedeInstallHookInstanceMint(t *testing.T) {
+ refuse := errors.New("no sealed config")
+ cases := []struct {
+ name string
+ apply func(*ReconcileConfig, *int)
+ gated bool
+ }{
+ {"inbound none", func(cfg *ReconcileConfig, _ *int) { cfg.Inbound = InboundNone }, false},
+ {"config gate", func(cfg *ReconcileConfig, gates *int) {
+ cfg.ConfigGate = func(context.Context, registry.Route) error { *gates++; return refuse }
+ }, true},
+ {"layout unreadable", func(cfg *ReconcileConfig, _ *int) {
+ cfg.Layout = func(context.Context, release.Store) (string, error) { return "", refuse }
+ }, false},
+ }
+ for _, tc := range cases {
+ t.Run(tc.name, func(t *testing.T) {
+ s, reg, host, cfg := instanceClaimFixture(t)
+ fx := newClaimEffects(t)
+ cfg.ClaimInstall = fx.hook(t)
+ gates := 0
+ tc.apply(&cfg, &gates)
+ if err := Reconcile(ctx, s, reg, cfg); err != nil {
+ t.Fatal(err)
+ }
+ fx.untouched(t, tc.name)
+ if got, err := reg.Get(host); err != nil || got != nil {
+ t.Fatalf("refused mint wrote a route: %v", err)
+ }
+ if tc.gated && gates != 1 {
+ t.Fatalf("config gate asked %d times, want once", gates)
+ }
+ })
+ }
+ t.Run("accepted mint reaches the hook once", func(t *testing.T) {
+ s, reg, host, cfg := instanceClaimFixture(t)
+ fx := newClaimEffects(t)
+ cfg.ClaimInstall = fx.hook(t)
+ cfg.ConfigGate = func(context.Context, registry.Route) error { return nil }
+ if err := Reconcile(ctx, s, reg, cfg); err != nil {
+ t.Fatal(err)
+ }
+ fx.reached(t, 1, "accepted")
+ if got, err := reg.Get(host); err != nil || got == nil || got.ClaimOwnership != fx.owner {
+ t.Fatalf("accepted mint lost ownership: %v", err)
+ }
+ })
+}
+
+// Both refusals are mint-time rules. A kept row converges as before, and its
+// claim hook still runs, whatever the box's inbound posture or config gate.
+func TestClaimKeepStillReachesInstallHookInstance(t *testing.T) {
+ s, reg, host, cfg := instanceClaimFixture(t)
+ fx := newClaimEffects(t)
+ cfg.ClaimInstall = fx.hook(t)
+ if err := Reconcile(ctx, s, reg, cfg); err != nil {
+ t.Fatal(err)
+ }
+ fx.reached(t, 1, "mint")
+ minted, err := reg.Get(host)
+ if err != nil || minted == nil {
+ t.Fatalf("mint: %v", err)
+ }
+ gates := 0
+ cfg.Inbound = InboundNone
+ cfg.ConfigGate = func(context.Context, registry.Route) error { gates++; return errors.New("refuse") }
+ if err := Reconcile(ctx, s, reg, cfg); err != nil {
+ t.Fatal(err)
+ }
+ fx.reached(t, 2, "keep")
+ if gates != 0 {
+ t.Fatalf("config gate asked %d times for a kept row", gates)
+ }
+ if got, err := reg.Get(host); err != nil || got == nil || *got != *minted {
+ t.Fatalf("keep changed the row: %v", err)
+ }
+}
+
+func TestClaimRefusalsPrecedeInstallHookSidecarMint(t *testing.T) {
+ refuse := errors.New("no sealed config")
+ cases := []struct {
+ name string
+ apply func(*testing.T, string, *ReconcileConfig, *int)
+ }{
+ {"co-spawn staged", func(t *testing.T, host string, cfg *ReconcileConfig, _ *int) {
+ if err := os.WriteFile(filepath.Join(cfg.DataDir, host+sidecarCoSpawnSuffix), []byte("A=1\n"), 0o600); err != nil {
+ t.Fatal(err)
+ }
+ }},
+ {"config gate", func(_ *testing.T, _ string, cfg *ReconcileConfig, gates *int) {
+ cfg.ConfigGate = func(context.Context, registry.Route) error { *gates++; return refuse }
+ }},
+ }
+ for _, tc := range cases {
+ t.Run(tc.name, func(t *testing.T) {
+ f, host, cfg := sidecarClaimFixture(t)
+ fx := newClaimEffects(t)
+ cfg.ClaimInstall = fx.hook(t)
+ gates := 0
+ tc.apply(t, host, &cfg, &gates)
+ f.reconcile(t, cfg)
+ fx.untouched(t, tc.name)
+ if got, err := f.reg.Get(host); err != nil || got != nil {
+ t.Fatalf("refused mint wrote a route: %v", err)
+ }
+ if tc.name == "config gate" && gates != 1 {
+ t.Fatalf("config gate asked %d times, want once", gates)
+ }
+ })
+ }
+ t.Run("accepted mint reaches the hook once", func(t *testing.T) {
+ f, host, cfg := sidecarClaimFixture(t)
+ fx := newClaimEffects(t)
+ cfg.ClaimInstall = fx.hook(t)
+ cfg.ConfigGate = func(context.Context, registry.Route) error { return nil }
+ f.reconcile(t, cfg)
+ fx.reached(t, 1, "accepted")
+ if got, err := f.reg.Get(host); err != nil || got == nil || got.ClaimOwnership != fx.owner {
+ t.Fatalf("accepted mint lost ownership: %v", err)
+ }
+ })
+}
+
+// The co-spawn rule is mint-time only: a file staged after the row exists
+// must not freeze the row's convergence, nor stop its claim hook.
+func TestClaimKeepStillReachesInstallHookSidecar(t *testing.T) {
+ f, host, cfg := sidecarClaimFixture(t)
+ fx := newClaimEffects(t)
+ cfg.ClaimInstall = fx.hook(t)
+ f.reconcile(t, cfg)
+ fx.reached(t, 1, "mint")
+ minted, err := f.reg.Get(host)
+ if err != nil || minted == nil {
+ t.Fatalf("mint: %v", err)
+ }
+ gates := 0
+ cfg.ConfigGate = func(context.Context, registry.Route) error { gates++; return errors.New("refuse") }
+ if err := os.WriteFile(filepath.Join(cfg.DataDir, host+sidecarCoSpawnSuffix), []byte("A=1\n"), 0o600); err != nil {
+ t.Fatal(err)
+ }
+ f.reconcile(t, cfg)
+ fx.reached(t, 2, "keep")
+ if gates != 0 {
+ t.Fatalf("config gate asked %d times for a kept row", gates)
+ }
+ if got, err := f.reg.Get(host); err != nil || got == nil || *got != *minted {
+ t.Fatalf("keep changed the row: %v", err)
+ }
+}
+
+// With the flag off the claim hook is never consulted, and the same refusals
+// still decide the mint exactly as before.
+func TestClaimFlagOffRefusalsUnchanged(t *testing.T) {
+ refuse := func(context.Context, registry.Route) error { return errors.New("refuse") }
+ hookRan := false
+ hook := func(context.Context, ClaimRecord, func(registry.ClaimOwnership) error) error {
+ hookRan = true
+ return ErrClaimAdmission
+ }
+ t.Run("instance", func(t *testing.T) {
+ for _, tc := range []struct {
+ name string
+ apply func(*ReconcileConfig)
+ mint bool
+ }{
+ {"inbound none", func(c *ReconcileConfig) { c.Inbound = InboundNone }, false},
+ {"config gate", func(c *ReconcileConfig) { c.ConfigGate = refuse }, false},
+ {"accepted", func(c *ReconcileConfig) {}, true},
+ } {
+ s, reg, host, cfg := instanceClaimFixture(t)
+ cfg.ClaimEnabled, cfg.ClaimInstall = false, hook
+ tc.apply(&cfg)
+ if err := Reconcile(ctx, s, reg, cfg); err != nil {
+ t.Fatal(err)
+ }
+ got, err := reg.Get(host)
+ if err != nil || (got != nil) != tc.mint {
+ t.Fatalf("%s: minted=%v err=%v, want minted=%v", tc.name, got != nil, err, tc.mint)
+ }
+ if got != nil && got.ClaimOwnership != (registry.ClaimOwnership{}) {
+ t.Fatalf("%s: flag-off mint carries ownership", tc.name)
+ }
+ }
+ })
+ t.Run("sidecar", func(t *testing.T) {
+ for _, tc := range []struct {
+ name string
+ apply func(*testing.T, string, *ReconcileConfig)
+ mint bool
+ }{
+ {"co-spawn", func(t *testing.T, host string, c *ReconcileConfig) {
+ if err := os.WriteFile(filepath.Join(c.DataDir, host+sidecarCoSpawnSuffix), nil, 0o600); err != nil {
+ t.Fatal(err)
+ }
+ }, false},
+ {"config gate", func(_ *testing.T, _ string, c *ReconcileConfig) { c.ConfigGate = refuse }, false},
+ {"accepted", func(*testing.T, string, *ReconcileConfig) {}, true},
+ } {
+ f, host, cfg := sidecarClaimFixture(t)
+ cfg.ClaimEnabled, cfg.ClaimInstall = false, hook
+ tc.apply(t, host, &cfg)
+ f.reconcile(t, cfg)
+ got, err := f.reg.Get(host)
+ if err != nil || (got != nil) != tc.mint {
+ t.Fatalf("%s: minted=%v err=%v, want minted=%v", tc.name, got != nil, err, tc.mint)
+ }
+ }
+ })
+ if hookRan {
+ t.Fatal("flag-off pass consulted the claim hook")
+ }
+}
+
+func TestClaimInstallWrapperRefusesUnownedExistingBeforeHook(t *testing.T) {
+ owner := registry.ClaimOwnership{ClaimIncarnation: "inc", ClaimStamp: 1, ClaimOwner: "owner", ClaimRowID: "row"}
+ calls, writes := 0, 0
+ hook := func(_ context.Context, _ ClaimRecord, write func(registry.ClaimOwnership) error) error {
+ calls++
+ return write(owner)
+ }
+ write := func(registry.ClaimOwnership) error { writes++; return nil }
+ cfg := ReconcileConfig{ClaimEnabled: true, ClaimInstall: hook}
+
+ if err := cfg.claimInstall(ctx, ClaimRecord{}, &registry.Route{}, write); !errors.Is(err, registry.ErrClaimOwnership) {
+ t.Fatalf("unowned existing row: err = %v, want ErrClaimOwnership", err)
+ }
+ if calls != 0 || writes != 0 {
+ t.Fatalf("unowned existing row reached the hook: calls=%d writes=%d", calls, writes)
+ }
+ // Valid owned rows still keep: the hook runs and its write goes through.
+ if err := cfg.claimInstall(ctx, ClaimRecord{}, &registry.Route{ClaimOwnership: owner}, write); err != nil {
+ t.Fatal(err)
+ }
+ if calls != 1 || writes != 1 {
+ t.Fatalf("owned existing row: calls=%d writes=%d, want 1 and 1", calls, writes)
+ }
+ // A different valid owner is still refused, by the write, after the hook.
+ other := owner
+ other.ClaimStamp++
+ if err := cfg.claimInstall(ctx, ClaimRecord{}, &registry.Route{ClaimOwnership: other}, write); !errors.Is(err, registry.ErrClaimOwnership) {
+ t.Fatalf("foreign owner: err = %v, want ErrClaimOwnership", err)
+ }
+ if calls != 2 || writes != 1 {
+ t.Fatalf("foreign owner: calls=%d writes=%d, want 2 and 1", calls, writes)
+ }
+ // A mint has no existing row and is unaffected.
+ if err := cfg.claimInstall(ctx, ClaimRecord{}, nil, write); err != nil {
+ t.Fatal(err)
+ }
+ if calls != 3 || writes != 2 {
+ t.Fatalf("mint: calls=%d writes=%d, want 3 and 2", calls, writes)
+ }
+ // Flag off: the hook is not consulted and an unowned row is written as before.
+ off := ReconcileConfig{ClaimInstall: hook}
+ if err := off.claimInstall(ctx, ClaimRecord{}, &registry.Route{}, write); err != nil {
+ t.Fatal(err)
+ }
+ if calls != 3 || writes != 3 {
+ t.Fatalf("flag off: calls=%d writes=%d, want 3 and 3", calls, writes)
+ }
+}
+
+// Existing rows this pass will not keep are refused before the claim hook, so
+// the guard activation and inventory write it stands for never happen. Each
+// row's refusal is isolated: the hand-added rows carry a VALID owner, so the
+// wrapper's ownership check cannot be what stops them.
+func TestClaimKnownExistingRowRefusalsPrecedeInstallHookInstance(t *testing.T) {
+ cases := []struct {
+ name string
+ setup func(*testing.T, release.Store, *registry.Registry, string, *claimEffects, ReconcileConfig)
+ }{
+ {"shape mismatch", func(t *testing.T, _ release.Store, reg *registry.Registry, host string, fx *claimEffects, _ ReconcileConfig) {
+ if err := reg.Upsert(registry.Route{Host: host, Kind: "instance", App: "other", ClaimOwnership: fx.owner}); err != nil {
+ t.Fatal(err)
+ }
+ }},
+ {"legacy owned row without claim owner", func(t *testing.T, s release.Store, reg *registry.Registry, host string, _ *claimEffects, cfg ReconcileConfig) {
+ cfg.ClaimEnabled = false
+ if err := Reconcile(ctx, s, reg, cfg); err != nil {
+ t.Fatal(err)
+ }
+ if got, err := reg.Get(host); err != nil || got == nil || !got.Provisioned {
+ t.Fatalf("legacy mint: %v", err)
+ }
+ }},
+ {"legacy adoptable row without claim owner", func(t *testing.T, s release.Store, reg *registry.Registry, host string, _ *claimEffects, cfg ReconcileConfig) {
+ cfg.ClaimEnabled = false
+ if err := Reconcile(ctx, s, reg, cfg); err != nil {
+ t.Fatal(err)
+ }
+ if err := reg.SetProvisioned(host, false); err != nil {
+ t.Fatal(err)
+ }
+ }},
+ }
+ for _, tc := range cases {
+ t.Run(tc.name, func(t *testing.T) {
+ s, reg, host, cfg := instanceClaimFixture(t)
+ fx := newClaimEffects(t)
+ tc.setup(t, s, reg, host, fx, cfg)
+ before, err := reg.Get(host)
+ if err != nil || before == nil {
+ t.Fatalf("setup left no row: %v", err)
+ }
+ gates := 0
+ cfg.ClaimInstall = fx.hook(t)
+ cfg.ConfigGate = func(context.Context, registry.Route) error { gates++; return nil }
+ if err := Reconcile(ctx, s, reg, cfg); err != nil {
+ t.Fatal(err)
+ }
+ fx.untouched(t, tc.name)
+ if gates != 0 {
+ t.Fatalf("config gate asked %d times for an existing row", gates)
+ }
+ if got, err := reg.Get(host); err != nil || got == nil || *got != *before {
+ t.Fatalf("refused existing row changed: %v", err)
+ }
+ })
+ }
+}
+
+func TestClaimKnownExistingRowRefusalsPrecedeInstallHookSidecar(t *testing.T) {
+ cases := []struct {
+ name string
+ setup func(*testing.T, *sidecarFixture, string, *claimEffects, ReconcileConfig)
+ }{
+ {"hand-added row", func(t *testing.T, f *sidecarFixture, host string, fx *claimEffects, _ ReconcileConfig) {
+ f.mustUpsert(t, registry.Route{Host: host, Kind: SidecarKind, Slug: "bot", App: "eleven", Account: f.sqid, ClaimOwnership: fx.owner})
+ }},
+ {"legacy owned row without claim owner", func(t *testing.T, f *sidecarFixture, host string, _ *claimEffects, cfg ReconcileConfig) {
+ cfg.ClaimEnabled = false
+ f.reconcile(t, cfg)
+ if got, err := f.reg.Get(host); err != nil || got == nil || !got.Provisioned {
+ t.Fatalf("legacy mint: %v", err)
+ }
+ }},
+ }
+ for _, tc := range cases {
+ t.Run(tc.name, func(t *testing.T) {
+ f, host, cfg := sidecarClaimFixture(t)
+ fx := newClaimEffects(t)
+ tc.setup(t, f, host, fx, cfg)
+ before, err := f.reg.Get(host)
+ if err != nil || before == nil {
+ t.Fatalf("setup left no row: %v", err)
+ }
+ gates := 0
+ cfg.ClaimInstall = fx.hook(t)
+ cfg.ConfigGate = func(context.Context, registry.Route) error { gates++; return nil }
+ f.reconcile(t, cfg)
+ fx.untouched(t, tc.name)
+ if gates != 0 {
+ t.Fatalf("config gate asked %d times for an existing row", gates)
+ }
+ if got, err := f.reg.Get(host); err != nil || got == nil || *got != *before {
+ t.Fatalf("refused existing row changed: %v", err)
+ }
+ })
+ }
+}
+
+// Flag off, the wrapper never consults the hook: a shape mismatch is still
+// left alone by the write, and an adoptable unowned row is still adopted.
+func TestClaimFlagOffExistingRowsUnchanged(t *testing.T) {
+ hookRan := false
+ hook := func(context.Context, ClaimRecord, func(registry.ClaimOwnership) error) error {
+ hookRan = true
+ return ErrClaimAdmission
+ }
+ t.Run("shape mismatch left alone", func(t *testing.T) {
+ s, reg, host, cfg := instanceClaimFixture(t)
+ cfg.ClaimEnabled, cfg.ClaimInstall = false, hook
+ if err := reg.Upsert(registry.Route{Host: host, Kind: "instance", App: "other"}); err != nil {
+ t.Fatal(err)
+ }
+ before := mustGet(t, reg, host)
+ if err := Reconcile(ctx, s, reg, cfg); err != nil {
+ t.Fatal(err)
+ }
+ if got := mustGet(t, reg, host); *got != *before {
+ t.Fatal("flag-off pass changed a mismatched row")
+ }
+ })
+ t.Run("adoptable unowned row adopted", func(t *testing.T) {
+ s, reg, host, cfg := instanceClaimFixture(t)
+ cfg.ClaimEnabled, cfg.ClaimInstall = false, hook
+ if err := Reconcile(ctx, s, reg, cfg); err != nil {
+ t.Fatal(err)
+ }
+ if err := reg.SetProvisioned(host, false); err != nil {
+ t.Fatal(err)
+ }
+ if err := Reconcile(ctx, s, reg, cfg); err != nil {
+ t.Fatal(err)
+ }
+ if got := mustGet(t, reg, host); !got.Provisioned || got.ClaimOwnership != (registry.ClaimOwnership{}) {
+ t.Fatalf("flag-off adoption: provisioned=%v owner=%+v", got.Provisioned, got.ClaimOwnership)
+ }
+ })
+ t.Run("sidecar hand-added row left alone", func(t *testing.T) {
+ f, host, cfg := sidecarClaimFixture(t)
+ cfg.ClaimEnabled, cfg.ClaimInstall = false, hook
+ f.mustUpsert(t, registry.Route{Host: host, Kind: SidecarKind, Slug: "bot", App: "eleven", Account: f.sqid})
+ before := mustGet(t, f.reg, host)
+ f.reconcile(t, cfg)
+ if got := mustGet(t, f.reg, host); *got != *before {
+ t.Fatal("flag-off pass changed a hand-added sidecar row")
+ }
+ })
+ if hookRan {
+ t.Fatal("flag-off pass consulted the claim hook")
+ }
+}
```
#### New file: `internal/registry/claimhold_test.go`
@@ -2231,31 +2861,1488 @@ func TestClaimTimerDeliveryDrainsBeforeBookkeeping(t *testing.T) {
}
```
-#### Patch: `internal/schedule/claimintegration_test.go`
-
-```diff
---- a/internal/schedule/claimintegration_test.go
-+++ b/internal/schedule/claimintegration_test.go
-@@ -63,0 +64,4 @@
-+ StartDelivery: func(ctx context.Context, w hostwrite.Work, _ registry.Route, _ registry.Timer) (context.Context, func(), error) {
-+ err := w.With(ctx, hostwrite.Identity{}, func() error { return nil })
-+ return ctx, func() {}, err
-+ },
-@@ -224,0 +229,9 @@
-+
-+func (d pausedDelivery) DeliverClaim(ctx context.Context, rt registry.Route, req *http.Request, admit func(context.Context) (context.Context, func(), error)) (int, error) {
-+ child, finish, err := admit(ctx)
-+ if err != nil {
-+ return 0, err
-+ }
-+ defer finish()
-+ return d.Deliver(child, rt, req)
-+}
-```
-
+#### Patch: `internal/schedule/claimintegration_test.go`
+
+```diff
+--- a/internal/schedule/claimintegration_test.go
++++ b/internal/schedule/claimintegration_test.go
+@@ -63,0 +64,4 @@
++ StartDelivery: func(ctx context.Context, w hostwrite.Work, _ registry.Route, _ registry.Timer) (context.Context, func(), error) {
++ err := w.With(ctx, hostwrite.Identity{}, func() error { return nil })
++ return ctx, func() {}, err
++ },
+@@ -224,0 +229,9 @@
++
++func (d pausedDelivery) DeliverClaim(ctx context.Context, rt registry.Route, req *http.Request, admit func(context.Context) (context.Context, func(), error)) (int, error) {
++ child, finish, err := admit(ctx)
++ if err != nil {
++ return 0, err
++ }
++ defer finish()
++ return d.Deliver(child, rt, req)
++}
+```
+
+#### New file: `internal/host/claimretryrecovery_test.go`
+
+```go
+package host
+
+import (
+ "context"
+ "encoding/json"
+ "errors"
+ "fmt"
+ "os"
+ "path/filepath"
+ "strings"
+ "testing"
+ "time"
+
+ "github.com/carlosframework/platform/internal/activator"
+ "github.com/carlosframework/platform/internal/cgroup"
+ "github.com/carlosframework/platform/internal/genselect"
+ "github.com/carlosframework/platform/internal/hostbox"
+ "github.com/carlosframework/platform/internal/hostclaim"
+ "github.com/carlosframework/platform/internal/hostguard"
+ "github.com/carlosframework/platform/internal/hostwrite"
+ "github.com/carlosframework/platform/internal/lease"
+ "github.com/carlosframework/platform/internal/registry"
+ "github.com/carlosframework/platform/internal/release"
+)
+
+// These tests drive the real wake assembly, serving retries, launch journal,
+// guard and stop coordinator. The kernel below is a fixture that reads the
+// real launch journal; it is not privileged cgroup or /proc verification.
+
+const recoveryHost = "shop.bab.oncarlos.com"
+
+var recoveryPolicy = activator.StartPolicy{MaxStarts: 5, Window: 30 * time.Second}
+
+type recoveryHeld struct{}
+
+func (recoveryHeld) Err() error { return nil }
+func (recoveryHeld) Lost() <-chan struct{} { return nil }
+func (recoveryHeld) Release(context.Context) error { return nil }
+func (recoveryHeld) BindGeneration(string, func(context.Context) error) error {
+ return hostwrite.ErrUnavailable
+}
+func (recoveryHeld) Replicate(context.Context, string, time.Duration) (*lease.Replicator, error) {
+ return nil, hostwrite.ErrUnavailable
+}
+
+type recoveryInputs struct{ f *recoveryFixture }
+
+func (r recoveryInputs) Select(_ context.Context, _ registry.Route, o hostclaim.Observed, _ activator.StartPolicy) (ClaimLaunchInputs, error) {
+ h := o.Claim.Holder
+ r.f.seq++
+ return ClaimLaunchInputs{Roles: []string{"main"}, Identity: hostwrite.Identity{Config: "config", Release: "release"}, Admission: hostclaim.Admission{ObservedSession: h.Session, ObservedGeneration: h.Generation, Box: "box-1", Mode: "cold", NewGeneration: fmt.Sprintf("gen-%d", r.f.seq), Region: "ie", PredecessorRegion: "ie"}, Executable: "/usr/bin/true", ExecutableSHA256: strings.Repeat("a", 64), DBPath: filepath.Join(r.f.guard.DataDir, "state", "data.db"), Empty: true}, nil
+}
+func (r recoveryInputs) Materialize(ctx context.Context, o hostclaim.Observed, _ ClaimLaunchInputs, _ hostwrite.Work) error {
+ if r.f.materialize == nil {
+ return nil
+ }
+ return r.f.materialize(ctx, o)
+}
+func (r recoveryInputs) EmptyAuthority(context.Context, hostclaim.Observed, ClaimLaunchInputs) error {
+ return r.f.restoreFailure
+}
+func (recoveryInputs) Publish(context.Context, hostclaim.Observed, ClaimLaunchInputs) error {
+ return nil
+}
+
+type recoveryKernel struct {
+ journal *hostbox.LaunchJournal
+ alive bool
+ verified []hostguard.FenceTarget
+ recovered []hostguard.FenceTarget
+}
+
+func (k *recoveryKernel) evidence(inv StopInventory, j hostbox.StopKernelJournal) StopKernelEvidence {
+ groups := []string{}
+ for _, g := range j.Cgroups {
+ groups = append(groups, fmt.Sprintf("%s:%d:%d", g.RelativePath, g.Device, g.Inode))
+ }
+ return StopKernelEvidence{ObservedBootID: j.Target.BootID, InventorySHA256: digestStop(inv), LocalJournalSHA256: digestStop(j), CgroupIdentities: groups, UnitInvocationIDs: []string{}}
+}
+func (k *recoveryKernel) VerifyStopped(ctx context.Context, t hostguard.FenceTarget, inv StopInventory, ack DrainAcknowledgement) (StopKernelEvidence, error) {
+ k.verified = append(k.verified, t)
+ j, err := k.journal.ReadStopKernelJournal(ctx, t)
+ if err != nil {
+ return StopKernelEvidence{}, err
+ }
+ if !j.Settled {
+ return StopKernelEvidence{}, ErrClaimStopInventory
+ }
+ if err := validateClaimDrainProducers(j.ProducerIDs, ack); err != nil {
+ return StopKernelEvidence{}, err
+ }
+ return k.evidence(inv, j), nil
+}
+func (k *recoveryKernel) RecoverStopped(ctx context.Context, t hostguard.FenceTarget, inv StopInventory) (StopKernelEvidence, DrainAcknowledgement, error) {
+ k.recovered = append(k.recovered, t)
+ j, err := k.journal.ReadStopKernelJournal(ctx, t)
+ if err != nil {
+ return StopKernelEvidence{}, DrainAcknowledgement{}, err
+ }
+ if k.alive {
+ return StopKernelEvidence{}, DrainAcknowledgement{}, ErrClaimStopControllerAlive
+ }
+ if err := k.journal.SettleRecovered(ctx, t); err != nil {
+ return StopKernelEvidence{}, DrainAcknowledgement{}, err
+ }
+ j.Settled = true
+ return k.evidence(inv, j), DrainClaimAcknowledgement(t.FenceID, t.ControllerID, j.ProcessStart, j.ProducerIDs), nil
+}
+
+type recoveryFixture struct {
+ t *testing.T
+ claims release.Store
+ evidence release.Store
+ guard hostguard.Guard
+ journal *hostbox.LaunchJournal
+ kernel *recoveryKernel
+ now time.Time
+ seq int
+ peer StopPeer
+ requests []StopRequest
+ materialize func(context.Context, hostclaim.Observed) error
+ restoreFailure error
+ recordPrelaunch bool
+ revision uint64
+}
+
+type recoveryController struct {
+ local claimStopLocal
+ wake *ClaimWakeAssembly
+ runtime *activator.ClaimRuntime
+ assembly ClaimAssembly
+}
+
+func newRecoveryFixture(t *testing.T) *recoveryFixture {
+ t.Helper()
+ ctx := t.Context()
+ st := release.NewDirStore(t.TempDir())
+ now := time.Date(2026, 10, 6, 12, 0, 0, 0, time.UTC)
+ in := hostclaim.NewHolder{Family: "instance", Key: "apps/bab/jam/instances/" + recoveryHost + ".json", Account: "bab", App: "jam", Host: recoveryHost, Region: "ie", RecordRegion: "ie", CreatedAt: now}
+ body := []byte(`{"schema":1,"host":"shop.bab.oncarlos.com","account":"bab","app":"jam","region":"ie","created_at":"2026-10-06T12:00:00Z"}`)
+ absent, err := hostclaim.Read(ctx, st, recoveryHost)
+ if err != nil {
+ t.Fatal(err)
+ }
+ o, err := hostclaim.Update(ctx, st, absent, hostclaim.BeginCreate(in, body, "create", "console", now))
+ if err != nil {
+ t.Fatal(err)
+ }
+ if _, err = hostclaim.Update(ctx, st, o, hostclaim.CompleteCreate("create", "box-1")); err != nil {
+ t.Fatal(err)
+ }
+ data := t.TempDir()
+ if err = os.Mkdir(filepath.Join(data, "hostguard"), 0770); err != nil {
+ t.Fatal(err)
+ }
+ journal, err := hostbox.OpenLaunchJournal(filepath.Join(t.TempDir(), "launches.db"))
+ if err != nil {
+ t.Fatal(err)
+ }
+ t.Cleanup(func() { journal.Close() })
+ return &recoveryFixture{t: t, claims: st, evidence: release.NewDirStore(t.TempDir()), guard: hostguard.Guard{DataDir: data, LockDir: t.TempDir(), Claims: hostguard.StoreReader{Store: st}}, journal: journal, kernel: &recoveryKernel{journal: journal}, now: now, recordPrelaunch: true}
+}
+
+func (f *recoveryFixture) newID() string { f.seq++; return fmt.Sprintf("id-%d", f.seq) }
+
+func (f *recoveryFixture) read() hostclaim.Observed {
+ f.t.Helper()
+ o, err := hostclaim.Read(f.t.Context(), f.claims, recoveryHost)
+ if err != nil {
+ f.t.Fatal(err)
+ }
+ return o
+}
+
+func (f *recoveryFixture) retry() hostclaim.WakeRetry {
+ f.t.Helper()
+ o := f.read()
+ if o.Claim.Holder.WakeRetry == nil {
+ f.t.Fatal("retry episode missing")
+ }
+ return *o.Claim.Holder.WakeRetry
+}
+
+func recoveryTuple(o hostclaim.Observed, l claimStopLocal) hostguard.FenceTarget {
+ h := o.Claim.Holder
+ return hostguard.FenceTarget{Region: l.Actor.Region, Host: o.Host, Incarnation: h.Incarnation, Stamp: h.Stamp, Session: h.Session, Box: l.Actor.Box, Enrollment: l.Actor.Enrollment, BootID: l.BootID, ControllerID: l.ControllerID, ClaimETag: o.ETag, DirectoryRevision: fmt.Sprint(l.Actor.Revision)}
+}
+
+// root is instance-control's coordinator for one request. The live peer is
+// whichever controller process the fixture most recently started.
+func (f *recoveryFixture) root(drain StopDrainer) ClaimStopCoordinator {
+ return ClaimStopCoordinator{Guard: f.guard, Claims: hostguard.StoreReader{Store: f.claims}, CarlosUID: 1001, Evidence: f.evidence, Drainer: drain, Kernel: f.kernel, NewID: f.newID, Now: func() time.Time { return f.now },
+ Peers: stopPeerFunc(func(context.Context) (StopPeer, error) { return f.peer, nil }),
+ Inventory: stopInventoryFunc(func(ctx context.Context, host string) (StopInventory, error) {
+ o, err := hostclaim.Read(ctx, f.claims, host)
+ if err != nil {
+ return StopInventory{}, err
+ }
+ h := o.Claim.Holder
+ return StopInventory{Complete: true, Host: host, Incarnation: h.Incarnation, InstallationStamp: h.Stamp, Box: "box-1", Enrollment: "enrollment-1", DirectoryRevision: "7"}, nil
+ })}
+}
+
+// controller starts one agent process: its own wake assembly, retries and stop
+// path, sharing the durable claim, guard and launch journal of the box.
+func (f *recoveryFixture) controller(boot, controller string, pid int, start string) *recoveryController {
+ t := f.t
+ revision := f.revision
+ if revision == 0 {
+ revision = 7
+ }
+ local := claimStopLocal{Actor: hostbox.Actor{Box: "box-1", Enrollment: "enrollment-1", Revision: revision, Region: "ie"}, BootID: boot, ControllerID: controller, PID: pid, ProcessStart: start}
+ f.peer = StopPeer{UID: 1001, PID: pid, ProcessStart: start, BootID: boot, ControllerID: controller}
+ producers := &ClaimProducerSet{}
+ w := &ClaimWakeAssembly{LeaseRegion: "eu-west-1", Inputs: recoveryInputs{f}, Manager: new(cgroup.Manager)}
+ w.AfterAdmission = func(context.Context, hostclaim.Observed, hostclaim.Observed) error { return nil }
+ // The unprivileged state-directory fixture runs under the real selected
+ // work; ColdRestore and its generation-marker writes remain unchanged.
+ w.PrepareState = func(ctx context.Context, _ hostclaim.Observed, work hostwrite.Work, in ClaimLaunchInputs) error {
+ return work.With(ctx, in.Identity, func() error { return os.MkdirAll(filepath.Dir(in.DBPath), 0700) })
+ }
+ w.RetireArtifacts = func(context.Context, hostclaim.Observed, string) error { return hostbox.ErrChanged }
+ w.Retain = func(hostclaim.Observed, activator.Proc, activator.Rep, error) {}
+ w.ObserveLoss = func(hostclaim.Observed, error) {}
+ w.Producers = func(_ context.Context, o hostclaim.Observed) (*ClaimProducers, error) {
+ return producers.Ensure(recoveryTuple(o, local), start, f.journal.RecordProducer)
+ }
+ w.Launch = func(context.Context, hostclaim.Observed, hostwrite.Work, ClaimLaunchInputs, *activator.RoleLaunch) (*activator.RoleLaunch, error) {
+ return nil, hostwrite.ErrUnavailable
+ }
+ w.Prelaunch = func(ctx context.Context, o hostclaim.Observed, work hostwrite.Work, in ClaimLaunchInputs) (*activator.RoleLaunch, error) {
+ target := recoveryTuple(o, local)
+ if !f.recordPrelaunch {
+ // rev1 placement: nothing is journaled until the launch itself.
+ return &activator.RoleLaunch{Target: target, Journal: f.journal}, nil
+ }
+ err := work.With(ctx, in.Identity, func() error {
+ return f.journal.Prelaunch(ctx, hostbox.LaunchInventory{ControllerPID: pid, ProcessStart: start, Target: target, Groups: []hostbox.LaunchGroup{{RelativePath: "carlos/" + claimContainmentName(target), Device: 1, Inode: target.Session}}, NoState: true})
+ })
+ if err != nil {
+ return nil, err
+ }
+ return &activator.RoleLaunch{Target: target, Journal: f.journal}, nil
+ }
+ w.Retries = activator.ServingRetries{Claims: f.claims, Box: "box-1", Enrollment: "enrollment-1", BootID: boot, ControllerID: controller, Now: func() time.Time { return f.now }, NewID: f.newID, Inputs: func(context.Context, registry.Route) (string, error) { return "inputs", nil }}
+ a := ClaimAssembly{Claims: f.claims, Launches: f.journal, Wake: w, Producers: producers, stopLocal: func(context.Context) (claimStopLocal, error) { return local, nil }}
+ a.collectStop = func(ctx context.Context, req StopRequest, drain StopDrainer) (StopResult, error) {
+ f.requests = append(f.requests, req)
+ result, err := f.root(drain).FenceAndStop(ctx, req)
+ if err != nil {
+ return result, err
+ }
+ return validateTransportStop(req, result)
+ }
+ a.bindClaimWakeStop()
+ source := hostbox.WorkSource{Guard: f.guard, Box: "box-1", Region: "ie", Now: func() time.Time { return f.now }, Policy: func(context.Context, *hostclaim.Claim, hostbox.Selection, hostwrite.Identity) error { return nil }, Local: func(context.Context, *hostclaim.Claim, hostbox.Selection) error { return nil }}
+ writer := hostbox.Writer{Claims: f.claims, Box: "box-1", Region: "ie", AdmissionEvidence: func(context.Context, hostclaim.Observed, hostclaim.Admission, *genselect.Plan) error { return nil }}
+ runtime, err := w.Build(t.Context(), source, writer)
+ if err != nil {
+ t.Fatal(err)
+ }
+ return &recoveryController{local: local, wake: w, runtime: runtime, assembly: a}
+}
+
+// admit stands in for Prerequisites' S3 lease, then reserves, admits, journals
+// the pre-launch record and binds the attempt through the real runtime.
+func (c *recoveryController) admit(t *testing.T, f *recoveryFixture) hostclaim.Observed {
+ t.Helper()
+ live, cancel := context.WithCancel(context.Background())
+ t.Cleanup(cancel)
+ done := make(chan struct{})
+ go func() { <-live.Done(); close(done) }()
+ o := f.read()
+ c.wake.mu.Lock()
+ c.wake.pending[recoveryHost] = &claimWakeAttempt{observed: o, held: recoveryHeld{}, ctx: live, cancel: cancel, done: done}
+ c.wake.mu.Unlock()
+ admitted, err := c.runtime.Admit(t.Context(), registry.Route{Host: recoveryHost}, o, recoveryPolicy)
+ if err != nil {
+ t.Fatal(err)
+ }
+ if r := admitted.Claim.Holder.WakeRetry; r == nil || r.Attempt == nil || r.Attempt.Session != admitted.Claim.Holder.Session || r.Attempt.ControllerID != c.local.ControllerID {
+ t.Fatalf("attempt not bound to admitted session: %+v", r)
+ }
+ return admitted
+}
+
+func (f *recoveryFixture) receipts() []StopReceipt {
+ f.t.Helper()
+ keys, err := f.evidence.List(f.t.Context(), "control/hosts/"+recoveryHost+"/serving-evidence/")
+ if err != nil {
+ f.t.Fatal(err)
+ }
+ out := []StopReceipt{}
+ for _, key := range keys {
+ body, _, err := f.evidence.Get(f.t.Context(), key)
+ if err != nil {
+ f.t.Fatal(err)
+ }
+ var r StopReceipt
+ if err = json.Unmarshal(body, &r); err != nil {
+ f.t.Fatal(err)
+ }
+ out = append(out, r)
+ }
+ return out
+}
+
+func TestClaimWakeRestoreFailureSettlesFromPrelaunchRecord(t *testing.T) {
+ f := newRecoveryFixture(t)
+ ctx := t.Context()
+ c := f.controller("boot-1", "controller-1", 4101, "start-1")
+ admitted := c.admit(t, f)
+ // The record exists before Prepare can hand out a Restore closure.
+ record, err := f.journal.Read(ctx, recoveryTuple(admitted, c.local))
+ if err != nil || record.ControllerPID != 4101 || record.ProcessStart != "start-1" || len(record.Groups) != 1 || len(record.Reservations) != 0 || !record.NoState {
+ t.Fatalf("no recoverable pre-launch record before restore: %+v %v", record, err)
+ }
+ s, err := c.runtime.Prepare(ctx, registry.Route{Host: recoveryHost}, admitted, recoveryPolicy)
+ if err != nil {
+ t.Fatal(err)
+ }
+ // Fail inside the production restore closure after PrepareState, at
+ // ColdRestore's empty-authority check. Do not replace that closure.
+ failure := errors.New("initial-empty authority unavailable")
+ f.restoreFailure = failure
+ restoreErr := s.Restore(ctx)
+ if !errors.Is(restoreErr, failure) {
+ t.Fatalf("restore failure = %v", restoreErr)
+ }
+ if _, err := os.Stat(filepath.Join(f.guard.DataDir, "state")); err != nil {
+ t.Fatalf("PrepareState did not run: %v", err)
+ }
+ if err = c.runtime.FailedAdmission(ctx, admitted, restoreErr); err != nil {
+ t.Fatalf("restore failure stranded the attempt: %v", err)
+ }
+ retry := f.retry()
+ if retry.State != "open" || retry.AttemptsReserved != 1 || retry.Attempt != nil {
+ t.Fatalf("restore failure not settled without refund: %+v", retry)
+ }
+ if c.wake.get(recoveryHost) != nil {
+ t.Fatal("failed attempt still pending")
+ }
+ if len(f.kernel.verified) == 0 || len(f.kernel.recovered) != 0 {
+ t.Fatalf("verified=%d recovered=%d", len(f.kernel.verified), len(f.kernel.recovered))
+ }
+ for _, req := range f.requests {
+ if req.Recover || req.ControllerID != "controller-1" || req.WriterSession != admitted.Claim.Holder.Session {
+ t.Fatalf("live controller stop used another tuple: %+v", req)
+ }
+ }
+ if got := f.receipts(); len(got) != 1 || got[0].ControllerID != "controller-1" || got[0].WriterSession != admitted.Claim.Holder.Session {
+ t.Fatalf("receipts = %+v", got)
+ }
+}
+
+// Control for the test above: with rev1's placement, nothing is journaled
+// before launch, root cannot verify the stop and the attempt stays reserved.
+func TestClaimWakeRestoreFailureWithoutPrelaunchRecordStrands(t *testing.T) {
+ f := newRecoveryFixture(t)
+ f.restoreFailure = errors.New("initial-empty authority unavailable")
+ ctx := t.Context()
+ f.recordPrelaunch = false
+ c := f.controller("boot-1", "controller-1", 4101, "start-1")
+ admitted := c.admit(t, f)
+ s, err := c.runtime.Prepare(ctx, registry.Route{Host: recoveryHost}, admitted, recoveryPolicy)
+ if err != nil {
+ t.Fatal(err)
+ }
+ restoreErr := s.Restore(ctx)
+ if err = c.runtime.FailedAdmission(ctx, admitted, restoreErr); err == nil {
+ t.Fatal("stop verified without a launch record")
+ }
+ if retry := f.retry(); retry.State != "attempting" || retry.AttemptsReserved != 1 {
+ t.Fatalf("retry = %+v", retry)
+ }
+ if got := f.receipts(); len(got) != 0 {
+ t.Fatalf("evidence without a launch record: %+v", got)
+ }
+}
+
+func TestClaimWakeMaterializeFailureSettlesFromPrelaunchRecord(t *testing.T) {
+ f := newRecoveryFixture(t)
+ ctx := t.Context()
+ c := f.controller("boot-1", "controller-1", 4101, "start-1")
+ admitted := c.admit(t, f)
+ failure := errors.New("materialization failed")
+ recorded := false
+ f.materialize = func(ctx context.Context, o hostclaim.Observed) error {
+ _, err := f.journal.Read(ctx, recoveryTuple(o, c.local))
+ recorded = err == nil
+ return failure
+ }
+ s, err := c.runtime.Prepare(ctx, registry.Route{Host: recoveryHost}, admitted, recoveryPolicy)
+ if err != nil {
+ t.Fatal(err)
+ }
+ if err := s.Restore(ctx); err != nil {
+ t.Fatal(err)
+ }
+ // startAdmitted's loop: validate the reservation, materialize, then Failed.
+ if err = s.Reserve(ctx, 0); err != nil {
+ t.Fatal(err)
+ }
+ materializeErr := s.Materialize(ctx)
+ if !errors.Is(materializeErr, failure) || !recorded {
+ t.Fatalf("materialization ran without the pre-launch record: recorded=%v err=%v", recorded, materializeErr)
+ }
+ if err = s.Failed(ctx, 0); err != nil {
+ t.Fatalf("materialization failure stranded the attempt: %v", err)
+ }
+ if err = c.runtime.FailedAdmission(ctx, admitted, materializeErr); err != nil {
+ t.Fatal(err)
+ }
+ retry := f.retry()
+ if retry.State != "open" || retry.AttemptsReserved != 1 || retry.Attempt != nil {
+ t.Fatalf("materialization failure not settled without refund: %+v", retry)
+ }
+ if len(f.kernel.verified) != 1 {
+ t.Fatalf("the repeated stop recollected evidence: %d", len(f.kernel.verified))
+ }
+}
+
+func TestClaimRetryCrashBeforeLaunchRecoversOriginalTupleAfterRestart(t *testing.T) {
+ f := newRecoveryFixture(t)
+ ctx := t.Context()
+ former := f.controller("boot-1", "controller-1", 4101, "start-1")
+ admitted := former.admit(t, f)
+ prepared, err := former.runtime.Prepare(ctx, registry.Route{Host: recoveryHost}, admitted, recoveryPolicy)
+ if err != nil {
+ t.Fatal(err)
+ }
+ // Run the assembly's real ColdRestore.InstallEmpty, then discard the
+ // ClaimStart before materialization or launch.
+ materializations := 0
+ f.materialize = func(context.Context, hostclaim.Observed) error { materializations++; return nil }
+ if err := prepared.Restore(ctx); err != nil {
+ t.Fatal(err)
+ }
+ marker := filepath.Join(f.guard.DataDir, "state", "data.db.gen")
+ if raw, err := os.ReadFile(marker); err != nil || strings.TrimSpace(string(raw)) != admitted.Claim.Holder.Generation {
+ t.Fatalf("real restore boundary: %q %v", raw, err)
+ }
+ t.Cleanup(func() {
+ if materializations != 0 {
+ t.Fatalf("materialization ran across the crash: %d", materializations)
+ }
+ })
+ attempt := *admitted.Claim.Holder.WakeRetry.Attempt
+ // The controller crashes after restore and before materialization. A later record
+ // for another session must not be touched by recovering this one.
+ successor := recoveryTuple(admitted, claimStopLocal{Actor: former.local.Actor, BootID: "boot-1", ControllerID: "controller-2"})
+ successor.Session++
+ successor.Stamp++
+ if err := f.journal.Prelaunch(ctx, hostbox.LaunchInventory{ControllerPID: 4202, ProcessStart: "start-2", Target: successor, Groups: []hostbox.LaunchGroup{{RelativePath: "carlos/successor", Device: 1, Inode: 99}}, NoState: true}); err != nil {
+ t.Fatal(err)
+ }
+ reservation := hostbox.LaunchReservation{ID: "successor:main", Role: "main", InvocationID: "successor-invocation", Inputs: "inputs", Target: successor, Group: hostbox.LaunchGroup{RelativePath: "carlos/successor/role", Device: 1, Inode: 100, InvocationID: "successor-invocation"}, Executable: hostbox.LaunchFile{Device: 1, Inode: 101}}
+ if err := f.journal.Reserve(ctx, reservation); err != nil {
+ t.Fatal(err)
+ }
+
+ restarted := f.controller("boot-1", "controller-2", 4202, "start-2")
+ rt := registry.Route{Host: recoveryHost}
+ f.kernel.alive = true
+ err = restarted.runtime.Prerequisites(ctx, rt, f.read(), recoveryPolicy)
+ if !errors.Is(err, ErrClaimStopControllerAlive) {
+ t.Fatalf("live former controller = %v", err)
+ }
+ if retry := f.retry(); retry.State != "attempting" || retry.Attempt == nil || *retry.Attempt != attempt || retry.AttemptsReserved != 1 {
+ t.Fatalf("settled before the former controller was proved dead: %+v", retry)
+ }
+ if got := f.receipts(); len(got) != 0 {
+ t.Fatalf("evidence before proof: %+v", got)
+ }
+
+ f.kernel.alive = false
+ if err = restarted.runtime.Prerequisites(ctx, rt, f.read(), recoveryPolicy); !errors.Is(err, hostbox.ErrChanged) {
+ t.Fatalf("settlement after restart = %v", err)
+ }
+ for _, req := range f.requests {
+ if !req.Recover || req.BootID != attempt.BootID || req.ControllerID != attempt.ControllerID || req.WriterSession != attempt.Session {
+ t.Fatalf("restarted controller substituted its own tuple: %+v", req)
+ }
+ }
+ if len(f.kernel.verified) != 0 || len(f.kernel.recovered) != 2 || f.kernel.recovered[0].FenceID != f.kernel.recovered[1].FenceID {
+ t.Fatalf("verified=%v recovered=%v", f.kernel.verified, f.kernel.recovered)
+ }
+ for _, target := range f.kernel.recovered {
+ if target.ControllerID != "controller-1" || target.BootID != "boot-1" || target.Session != attempt.Session {
+ t.Fatalf("recovery target = %+v", target)
+ }
+ }
+ retry := f.retry()
+ if retry.State != "open" || retry.AttemptsReserved != 1 || retry.Attempt != nil {
+ t.Fatalf("crash attempt not settled without refund: %+v", retry)
+ }
+ if got := f.receipts(); len(got) != 1 || got[0].ControllerID != "controller-1" || got[0].BootID != "boot-1" {
+ t.Fatalf("receipts = %+v", got)
+ }
+ other, err := f.journal.Read(ctx, successor)
+ if err != nil || len(other.Reservations) != 1 || other.Reservations[0].State != "reserved" {
+ t.Fatalf("recovery changed another session's record: %+v %v", other, err)
+ }
+ // The next wake stops its predecessor through the same journaled tuple and
+ // reuses the durable evidence instead of being refused for controller-2.
+ if err = restarted.wake.Stop(ctx, f.read()); err != nil {
+ t.Fatalf("predecessor stop after restart = %v", err)
+ }
+ if len(f.kernel.recovered) != 2 || len(f.kernel.verified) != 0 {
+ t.Fatalf("predecessor stop recollected: recovered=%d verified=%d", len(f.kernel.recovered), len(f.kernel.verified))
+ }
+}
+
+func TestClaimRetryRecoveryRefusesMissingLaunchRecord(t *testing.T) {
+ f := newRecoveryFixture(t)
+ ctx := t.Context()
+ f.recordPrelaunch = false
+ former := f.controller("boot-1", "controller-1", 4101, "start-1")
+ admitted := former.admit(t, f)
+ attempt := *admitted.Claim.Holder.WakeRetry.Attempt
+ restarted := f.controller("boot-2", "controller-2", 4202, "start-2")
+ err := restarted.runtime.Prerequisites(ctx, registry.Route{Host: recoveryHost}, f.read(), recoveryPolicy)
+ if err == nil || errors.Is(err, hostbox.ErrChanged) {
+ t.Fatalf("missing launch record settled: %v", err)
+ }
+ if len(f.requests) != 1 || !f.requests[0].Recover || f.requests[0].ControllerID != "controller-1" || f.requests[0].BootID != "boot-1" {
+ t.Fatalf("stop requests = %+v", f.requests)
+ }
+ if retry := f.retry(); retry.State != "attempting" || retry.Attempt == nil || *retry.Attempt != attempt || retry.AttemptsReserved != 1 {
+ t.Fatalf("retry = %+v", retry)
+ }
+ if got := f.receipts(); len(got) != 0 {
+ t.Fatalf("evidence without a launch record: %+v", got)
+ }
+}
+
+type recoveryOnlyKernel struct {
+ stopKernelFunc
+ recover func(context.Context, hostguard.FenceTarget, StopInventory) (StopKernelEvidence, DrainAcknowledgement, error)
+}
+
+func (k recoveryOnlyKernel) RecoverStopped(ctx context.Context, t hostguard.FenceTarget, in StopInventory) (StopKernelEvidence, DrainAcknowledgement, error) {
+ return k.recover(ctx, t, in)
+}
+
+func TestClaimStopRecoveryModeRefusesWithoutRootProof(t *testing.T) {
+ c, req, _ := claimStopFixture(t)
+ seq := 0
+ c.NewID = func() string { seq++; return fmt.Sprintf("id-%d", seq) }
+ drained := false
+ c.Drainer = stopDrainFunc(func(context.Context, StopRequest, string) (DrainAcknowledgement, error) {
+ drained = true
+ return DrainAcknowledgement{}, nil
+ })
+ c.Peers = stopPeerFunc(func(context.Context) (StopPeer, error) {
+ return StopPeer{UID: 1001, PID: 43, ProcessStart: "start-43", BootID: "boot-1", ControllerID: "controller-2"}, nil
+ })
+ if _, err := c.FenceAndStop(t.Context(), req); !errors.Is(err, ErrClaimStopIdentity) {
+ t.Fatalf("another tuple without recovery = %v", err)
+ }
+ req.Recover = true
+ if _, err := c.FenceAndStop(t.Context(), req); !errors.Is(err, ErrClaimStopUnavailable) || drained || seq != 0 {
+ t.Fatalf("recovery without a recovery kernel = %v drained=%v ids=%d", err, drained, seq)
+ }
+ if pending, err := c.Guard.PendingFences(t.Context()); err != nil || len(pending) != 0 {
+ t.Fatalf("refused recovery installed a fence: %+v %v", pending, err)
+ }
+ cleanup := req
+ cleanup.Cleanup = true
+ if _, err := c.FenceAndStop(t.Context(), cleanup); !errors.Is(err, ErrClaimStopUnavailable) {
+ t.Fatalf("recovery combined with cleanup = %v", err)
+ }
+ evidence := StopKernelEvidence{InventorySHA256: strings.Repeat("b", 64), LocalJournalSHA256: strings.Repeat("c", 64), CgroupIdentities: []string{"cg-1"}}
+ c.Kernel = recoveryOnlyKernel{recover: func(_ context.Context, target hostguard.FenceTarget, _ StopInventory) (StopKernelEvidence, DrainAcknowledgement, error) {
+ return StopKernelEvidence{}, DrainAcknowledgement{}, ErrClaimStopControllerAlive
+ }}
+ if _, err := c.FenceAndStop(t.Context(), req); !errors.Is(err, ErrClaimStopControllerAlive) {
+ t.Fatalf("live former controller = %v", err)
+ }
+ // The same IDs and process start as the live peer are that peer, not a
+ // former controller, whatever the recovery kernel reports.
+ c.Peers = stopPeerFunc(func(context.Context) (StopPeer, error) {
+ return StopPeer{UID: 1001, PID: 42, ProcessStart: "start-42", BootID: req.BootID, ControllerID: req.ControllerID}, nil
+ })
+ c.Kernel = recoveryOnlyKernel{recover: func(_ context.Context, target hostguard.FenceTarget, _ StopInventory) (StopKernelEvidence, DrainAcknowledgement, error) {
+ return evidence, DrainClaimAcknowledgement(target.FenceID, target.ControllerID, "start-42", nil), nil
+ }}
+ if _, err := c.FenceAndStop(t.Context(), req); !errors.Is(err, ErrClaimStopIdentity) {
+ t.Fatalf("live peer recovered as former controller = %v", err)
+ }
+ c.Kernel = recoveryOnlyKernel{recover: func(_ context.Context, target hostguard.FenceTarget, _ StopInventory) (StopKernelEvidence, DrainAcknowledgement, error) {
+ return evidence, DrainClaimAcknowledgement(target.FenceID, target.ControllerID, "start-41", nil), nil
+ }}
+ result, err := c.FenceAndStop(t.Context(), req)
+ if err != nil || drained {
+ t.Fatalf("restarted controller recovery = %v drained=%v", err, drained)
+ }
+ if result.Target.ControllerID != req.ControllerID || result.Target.BootID != req.BootID {
+ t.Fatalf("recovery receipt names %+v", result.Target)
+ }
+}
+
+func TestClaimRetryRecoveryRetainsRecordedDirectoryRevision(t *testing.T) {
+ f := newRecoveryFixture(t)
+ former := f.controller("boot-1", "controller-1", 4101, "start-1")
+ former.admit(t, f)
+ f.revision = 8
+ restarted := f.controller("boot-1", "controller-2", 4202, "start-2")
+ err := restarted.runtime.Prerequisites(t.Context(), registry.Route{Host: recoveryHost}, f.read(), recoveryPolicy)
+ if !errors.Is(err, hostbox.ErrChanged) {
+ t.Fatalf("same-enrollment roster change stranded recovery: %v", err)
+ }
+ if len(f.requests) != 1 || f.requests[0].ExpectedDirectoryRevision != "7" {
+ t.Fatalf("recovery substituted current directory: %+v", f.requests)
+ }
+ if retry := f.retry(); retry.State != "open" || retry.Attempt != nil || retry.AttemptsReserved != 1 {
+ t.Fatalf("retry = %+v", retry)
+ }
+}
+```
+
+#### New file: `internal/host/claimstop_drain_test.go`
+
+```go
+package host
+
+import (
+ "encoding/json"
+ "errors"
+ "testing"
+)
+
+func TestClaimEmptyProducerDrainSurvivesTransport(t *testing.T) {
+ ack := DrainClaimAcknowledgement("fence", "controller", "start", nil)
+ raw, err := json.Marshal(ack)
+ if err != nil {
+ t.Fatal(err)
+ }
+ var wire DrainAcknowledgement
+ if err := json.Unmarshal(raw, &wire); err != nil {
+ t.Fatal(err)
+ }
+ for _, producers := range [][]string{nil, {}} {
+ if err := validateClaimDrainProducers(producers, wire); err != nil {
+ t.Fatalf("zero-producer pre-launch stop refused after JSON: %v", err)
+ }
+ }
+ for _, ids := range [][]string{{"unrecorded"}, {"unrecorded", "unrecorded"}} {
+ forged := DrainClaimAcknowledgement("fence", "controller", "start", ids)
+ if err := validateClaimDrainProducers(nil, forged); !errors.Is(err, ErrClaimStopIdentity) {
+ t.Fatalf("unrecorded producers accepted: %v", err)
+ }
+ }
+ wire.Digest = "bad"
+ if err := validateClaimDrainProducers(nil, wire); !errors.Is(err, ErrClaimStopIdentity) {
+ t.Fatalf("bad digest accepted: %v", err)
+ }
+}
+```
+
+#### New file: `internal/hostbox/admission_refusal_test.go`
+
+```go
+//go:build unix
+
+package hostbox
+
+import (
+ "context"
+ "database/sql"
+ "encoding/json"
+ "errors"
+ "os"
+ "path/filepath"
+ "testing"
+ "time"
+
+ "github.com/carlosframework/platform/internal/domains"
+ "github.com/carlosframework/platform/internal/hostclaim"
+ "github.com/carlosframework/platform/internal/hostguard"
+ "github.com/carlosframework/platform/internal/provision"
+ "github.com/carlosframework/platform/internal/registry"
+ "github.com/carlosframework/platform/internal/release"
+)
+
+// admissionFixture is a bucket holding one record (instance or sidecar) and
+// the adopted hostclaim for its host, plus the identity a claim hook would
+// stamp on the route.
+type admissionFixture struct {
+ store release.Store
+ data string
+ account, host, key string
+ family, name string
+ holder hostclaim.Holder
+ owner registry.ClaimOwnership
+ pass func(context.Context, release.Store, *registry.Registry, provision.ReconcileConfig) error
+}
+
+func newAdmissionFixture(t *testing.T, sidecar bool) admissionFixture {
+ t.Helper()
+ ctx := t.Context()
+ store := release.NewDirStore(t.TempDir())
+ account := domains.Sqid(2)
+ raw, err := json.Marshal(map[string]any{"id": 2, "sqid": account, "name": "fixture", "created_at": "2026-08-02T09:00:00Z"})
+ if err != nil {
+ t.Fatal(err)
+ }
+ if _, err := store.Put(ctx, "accounts/"+account+"/account.json", raw, ""); err != nil {
+ t.Fatal(err)
+ }
+ data := t.TempDir()
+ if err := os.Mkdir(filepath.Join(data, "hostguard"), 0770); err != nil {
+ t.Fatal(err)
+ }
+ created := time.Date(2026, 10, 7, 0, 0, 0, 0, time.UTC)
+ f := admissionFixture{store: store, data: data, account: account, host: "shop.example", family: "instance"}
+ f.key = provision.RecordKey(account, "app", f.host)
+ f.pass = provision.Reconcile
+ if sidecar {
+ f.host = provision.SidecarHost("worker", "app", account)
+ f.family, f.name = "sidecar", "worker"
+ f.key = provision.SidecarRecordKey(account, "app", "worker")
+ f.pass = provision.ReconcileSidecars
+ if err := provision.PutSidecarManifest(ctx, store, account, "app", provision.SidecarManifest{Schema: 1, App: "app", Args: []string{"run"}, Channel: "stable"}); err != nil {
+ t.Fatal(err)
+ }
+ if err := provision.CreateSidecarRecord(ctx, store, provision.SidecarRecord{Schema: 1, Name: "worker", App: "app", Account: account, Host: f.host, Region: "ie", Channel: "stable", CreatedAt: created}); err != nil {
+ t.Fatal(err)
+ }
+ } else {
+ if err := provision.PutManifest(ctx, store, account, "app", provision.Manifest{Schema: 1, App: "app", Domain: "example"}); err != nil {
+ t.Fatal(err)
+ }
+ if err := provision.CreateRecord(ctx, store, provision.Record{Schema: 1, Host: f.host, Slug: "shop", App: "app", Account: account, Region: "ie", Channel: "stable", CreatedAt: created}); err != nil {
+ t.Fatal(err)
+ }
+ }
+ o, err := hostclaim.Read(ctx, store, f.host)
+ if err != nil {
+ t.Fatal(err)
+ }
+ o, err = hostclaim.Update(ctx, store, o, hostclaim.Adopt(hostclaim.NewHolder{Family: f.family, Name: f.name, Key: f.key, Account: account, App: "app", Host: f.host, Region: "ie", CreatedAt: created}, "box", nil, nil, "adopt", func() string { return "id" }))
+ if err != nil {
+ t.Fatal(err)
+ }
+ f.holder = *o.Claim.Holder
+ f.owner = registry.ClaimOwnership{ClaimIncarnation: f.holder.Incarnation, ClaimStamp: f.holder.Stamp, ClaimOwner: "owner", ClaimRowID: "row"}
+ return f
+}
+
+func TestClaimRefusalPreservesDurableGuardAndInventory(t *testing.T) {
+ type env struct {
+ cfg *provision.ReconcileConfig
+ reg *registry.Registry
+ f admissionFixture
+ }
+ allow := func(gates *int) func(context.Context, registry.Route) error {
+ return func(context.Context, registry.Route) error { *gates++; return nil }
+ }
+ refuse := func(gates *int) func(context.Context, registry.Route) error {
+ return func(context.Context, registry.Route) error { *gates++; return errors.New("config refused") }
+ }
+ upsert := func(t *testing.T, e env, rt registry.Route) {
+ t.Helper()
+ if err := e.reg.Upsert(rt); err != nil {
+ t.Fatal(err)
+ }
+ }
+ cases := []struct {
+ name string
+ sidecar bool
+ reached bool // the claim hook ran: guard and inventory written
+ gates int // ConfigGate calls this pass makes
+ apply func(*testing.T, env, *int)
+ }{
+ {name: "inbound", apply: func(_ *testing.T, e env, _ *int) { e.cfg.Inbound = provision.InboundNone }},
+ {name: "instance config", gates: 1, apply: func(_ *testing.T, e env, gates *int) { e.cfg.ConfigGate = refuse(gates) }},
+ {name: "sidecar config", sidecar: true, gates: 1, apply: func(_ *testing.T, e env, gates *int) { e.cfg.ConfigGate = refuse(gates) }},
+ {name: "co-spawn", sidecar: true, apply: func(t *testing.T, e env, _ *int) {
+ if err := os.WriteFile(filepath.Join(e.f.data, e.f.host+"-sidecar.env"), []byte("A=1\n"), 0600); err != nil {
+ t.Fatal(err)
+ }
+ }},
+ {name: "instance shape mismatch", apply: func(t *testing.T, e env, gates *int) {
+ e.cfg.ConfigGate = allow(gates)
+ upsert(t, e, registry.Route{Host: e.f.host, Kind: "instance", App: "other", ClaimOwnership: e.f.owner})
+ }},
+ {name: "instance row without claim owner", apply: func(t *testing.T, e env, gates *int) {
+ e.cfg.ConfigGate = allow(gates)
+ upsert(t, e, registry.Route{Host: e.f.host, Kind: "instance", App: "app", Account: e.f.account, Provisioned: true})
+ }},
+ {name: "sidecar row not owned by this pass", sidecar: true, apply: func(t *testing.T, e env, gates *int) {
+ e.cfg.ConfigGate = allow(gates)
+ upsert(t, e, registry.Route{Host: e.f.host, Kind: provision.SidecarKind, Slug: "worker", App: "app", Account: e.f.account, ClaimOwnership: e.f.owner})
+ }},
+ {name: "sidecar row without claim owner", sidecar: true, apply: func(t *testing.T, e env, gates *int) {
+ e.cfg.ConfigGate = allow(gates)
+ upsert(t, e, registry.Route{Host: e.f.host, Kind: provision.SidecarKind, Slug: "worker", App: "app", Account: e.f.account, Provisioned: true})
+ }},
+ // Positive controls: the same fixtures reach the hook when nothing refuses.
+ {name: "instance accepted", reached: true, gates: 1, apply: func(_ *testing.T, e env, gates *int) { e.cfg.ConfigGate = allow(gates) }},
+ {name: "sidecar accepted", sidecar: true, reached: true, gates: 1, apply: func(_ *testing.T, e env, gates *int) { e.cfg.ConfigGate = allow(gates) }},
+ {name: "instance owned row kept", reached: true, apply: func(t *testing.T, e env, gates *int) {
+ e.cfg.ConfigGate = allow(gates)
+ upsert(t, e, registry.Route{Host: e.f.host, Kind: "instance", App: "app", Account: e.f.account, Provisioned: true, ClaimOwnership: e.f.owner})
+ }},
+ {name: "sidecar owned row kept", sidecar: true, reached: true, apply: func(t *testing.T, e env, gates *int) {
+ e.cfg.ConfigGate = allow(gates)
+ upsert(t, e, registry.Route{Host: e.f.host, Kind: provision.SidecarKind, Slug: "worker", App: "app", Account: e.f.account, Provisioned: true, ClaimOwnership: e.f.owner})
+ }},
+ }
+ for _, tc := range cases {
+ t.Run(tc.name, func(t *testing.T) {
+ ctx := t.Context()
+ f := newAdmissionFixture(t, tc.sidecar)
+ reg, err := registry.Open(filepath.Join(t.TempDir(), "registry.db"))
+ if err != nil {
+ t.Fatal(err)
+ }
+ defer reg.Close()
+ inventory, err := OpenStopInventoryDB(filepath.Join(t.TempDir(), "inventory.db"))
+ if err != nil {
+ t.Fatal(err)
+ }
+ defer inventory.Close()
+ guard := hostguard.Guard{DataDir: f.data, LockDir: t.TempDir(), Claims: hostguard.StoreReader{Store: f.store}}
+ h := f.holder
+ in := StopInventory{Host: f.host, Incarnation: h.Incarnation, InstallationStamp: h.Stamp, Box: "box", Enrollment: "en", DirectoryRevision: "1", Complete: true}
+ calls, gates := 0, 0
+ cfg := provision.ReconcileConfig{DataDir: f.data, ClaimEnabled: true, Region: "ie"}
+ cfg.ClaimInstall = func(ctx context.Context, selected provision.ClaimRecord, write func(registry.ClaimOwnership) error) error {
+ calls++
+ if err := guard.Activate(ctx, selected.Host, "box", nil); err != nil {
+ return err
+ }
+ if err := inventory.WriteStopInventory(ctx, in); err != nil {
+ return err
+ }
+ return write(f.owner)
+ }
+ tc.apply(t, env{cfg: &cfg, reg: reg, f: f}, &gates)
+ before, err := reg.Get(f.host)
+ if err != nil {
+ t.Fatal(err)
+ }
+ if err := f.pass(ctx, f.store, reg, cfg); err != nil {
+ t.Fatal(err)
+ }
+ if gates != tc.gates {
+ t.Fatalf("config gate asked %d times, want %d", gates, tc.gates)
+ }
+ _, guardErr := os.ReadFile(filepath.Join(f.data, "hostguard", f.host+".json"))
+ _, inventoryErr := inventory.ReadInstallation(ctx, f.host, h.Incarnation, h.Stamp)
+ after, err := reg.Get(f.host)
+ if err != nil {
+ t.Fatal(err)
+ }
+ if tc.reached {
+ if calls != 1 || guardErr != nil || inventoryErr != nil {
+ t.Fatalf("positive control: calls=%d guard=%v inventory=%v", calls, guardErr, inventoryErr)
+ }
+ if after == nil || after.ClaimOwnership != f.owner {
+ t.Fatalf("positive control lost ownership: %+v", after)
+ }
+ return
+ }
+ if calls != 0 || !errors.Is(guardErr, os.ErrNotExist) || !errors.Is(inventoryErr, sql.ErrNoRows) {
+ t.Fatalf("refusal changed durable state: calls=%d guard=%v inventory=%v", calls, guardErr, inventoryErr)
+ }
+ if (before == nil) != (after == nil) || (before != nil && *before != *after) {
+ t.Fatalf("refusal changed the registry: before=%+v after=%+v", before, after)
+ }
+ })
+ }
+}
+```
+
+#### New file: `internal/hostbox/intended_recovery_test.go`
+
+```go
+//go:build linux
+
+package hostbox
+
+import (
+ "bytes"
+ "context"
+ "errors"
+ "os"
+ "path/filepath"
+ "testing"
+ "time"
+
+ "golang.org/x/sys/unix"
+
+ "github.com/carlosframework/platform/internal/hostclaim"
+ "github.com/carlosframework/platform/internal/hostguard"
+ "github.com/carlosframework/platform/internal/registry"
+)
+
+// intendedFixture is a prior installation whose PrepareState persisted the
+// state-dir intent and then failed or crashed before its final inventory write.
+type intendedFixture struct {
+ handover ColdHandover
+ prior hostguard.FenceTarget
+ next ColdSuccessor
+ inventory *StopInventoryDB
+ reg *registry.Registry
+ data string
+ dbPath string
+ state string
+ id string
+}
+
+func newIntendedFixture(t *testing.T, edit func(*StopInventoryEntry)) intendedFixture {
+ t.Helper()
+ source, store, initial := workFixture(t)
+ ctx := t.Context()
+ admit := func(old hostclaim.Observed, generation string) hostclaim.Observed {
+ next, err := hostclaim.Update(ctx, store, old, hostclaim.AdmitWriter(hostclaim.Admission{ObservedOpID: old.Claim.Op.ID, ObservedPhase: old.Claim.Op.Phase, ObservedSession: old.Claim.Holder.Session, ObservedGeneration: old.Claim.Holder.Generation, Box: "box", Region: "ie", Mode: "cold", NewGeneration: generation}))
+ if err != nil {
+ t.Fatal(err)
+ }
+ return next
+ }
+ old := admit(initial, "first")
+ if err := source.Guard.Activate(ctx, old.Host, "box", nil); err != nil {
+ t.Fatal(err)
+ }
+ prior := hostguard.FenceTarget{FenceID: "fence", Region: "ie", Host: old.Host, Incarnation: old.Claim.Holder.Incarnation, Stamp: old.Claim.Holder.Stamp, Session: old.Claim.Holder.Session, Box: "box", Enrollment: "enrollment", BootID: "boot", ControllerID: "controller", ClaimETag: old.ETag, DirectoryRevision: "1", Op: old.Claim.Op.ID, Phase: old.Claim.Op.Phase}
+ if _, err := source.Guard.BeginFence(ctx, prior); err != nil {
+ t.Fatal(err)
+ }
+ if err := source.Guard.CompleteFence(ctx, prior, []byte("verified-root-receipt"), time.Now().UTC(), func(context.Context, hostguard.FenceTarget, []byte, time.Time) error { return nil }); err != nil {
+ t.Fatal(err)
+ }
+ base := t.TempDir()
+ reg, err := registry.Open(filepath.Join(base, "registry.db"))
+ if err != nil {
+ t.Fatal(err)
+ }
+ t.Cleanup(func() { reg.Close() })
+ inventory, err := OpenStopInventoryDB(filepath.Join(base, "inventory.db"))
+ if err != nil {
+ t.Fatal(err)
+ }
+ t.Cleanup(func() { inventory.Close() })
+ launches, err := OpenLaunchJournal(filepath.Join(base, "launches.db"))
+ if err != nil {
+ t.Fatal(err)
+ }
+ t.Cleanup(func() { launches.Close() })
+ owner := registry.ClaimOwnership{ClaimIncarnation: prior.Incarnation, ClaimStamp: prior.Stamp, ClaimOwner: "old-owner", ClaimRowID: "old-route"}
+ data := source.Guard.DataDir
+ dbPath := filepath.Join(data, old.Host+".db")
+ if err = os.WriteFile(dbPath, []byte("untrusted-local-database"), 0600); err != nil {
+ t.Fatal(err)
+ }
+ if err = reg.Upsert(registry.Route{Host: old.Host, Kind: "instance", Socket: "/run/example.sock", DBPath: dbPath, ClaimOwnership: owner}); err != nil {
+ t.Fatal(err)
+ }
+ if err = reg.UpsertTimer(registry.Timer{Host: old.Host, Name: "daily", Kind: "cron", Spec: "0 0 * * *", NextAt: 123, Attempts: 2}); err != nil {
+ t.Fatal(err)
+ }
+ parker := TypedParker{Roots: map[string]string{"data": data}, ParkedRoot: data, Incarnation: prior.Incarnation, Host: prior.Host, Retirement: "capture"}
+ artifact, err := parker.CaptureArtifact("data", old.Host+".db", owner.ClaimOwner, ArtifactPath)
+ if err != nil {
+ t.Fatal(err)
+ }
+ // The intent PrepareState writes before Mkdirat: typed, no identity.
+ state := StopInventoryEntry{Kind: "path", ArtifactID: "state-dir", ManagedRoot: "data", RelativePath: old.Host, Ownership: owner.ClaimOwner, Materialization: "intended", Type: unix.S_IFDIR}
+ if edit != nil {
+ edit(&state)
+ }
+ in := StopInventory{Complete: true, Host: old.Host, Incarnation: prior.Incarnation, InstallationStamp: prior.Stamp, Box: "box", Enrollment: "enrollment", DirectoryRevision: "1", Entries: []StopInventoryEntry{
+ {Kind: "route", ArtifactID: owner.ClaimRowID, ManagedRoot: "registry", RelativePath: old.Host, RowIdentity: owner.ClaimRowID, Ownership: owner.ClaimOwner, Materialization: "materialized"},
+ {Kind: "path", ArtifactID: "database", ManagedRoot: "data", RelativePath: artifact.Name, Ownership: artifact.Owner, Materialization: "materialized", Device: artifact.Dev, Inode: artifact.Ino, Type: artifact.Type},
+ state,
+ }}
+ if err = inventory.WriteStopInventory(ctx, in); err != nil {
+ t.Fatal(err)
+ }
+ successor := admit(old, "second")
+ next := ColdSuccessor{Host: old.Host, Incarnation: prior.Incarnation, Stamp: successor.Claim.Holder.Stamp, Session: successor.Claim.Holder.Session, Generation: successor.Claim.Holder.Generation, Box: "box", Enrollment: "enrollment", DirectoryRevision: "1", ETag: successor.ETag, Op: successor.Claim.Op.ID, Phase: successor.Claim.Op.Phase}
+ handover := ColdHandover{Guard: source.Guard, Claims: store, Registry: reg, Inventory: inventory, Launches: launches, Roots: map[string]string{"data": data}, JournalDir: filepath.Join(base, "handover"), VerifyReceipt: func(context.Context, hostguard.FenceReceipt) error { return nil }}
+ return intendedFixture{handover: handover, prior: prior, next: next, inventory: inventory, reg: reg, data: data, dbPath: dbPath, state: filepath.Join(data, old.Host), id: "handover-" + ownershipFor(next.Host, next.Incarnation, next.Stamp, next.Box)}
+}
+
+func (f intendedFixture) parked(retirement, name string) string {
+ return filepath.Join(f.data, ".parked", f.prior.Incarnation, f.prior.Host, f.id+retirement, "data", name)
+}
+
+func (f intendedFixture) intendedJournal() string {
+ return filepath.Join(f.handover.JournalDir, "retired", f.id+"-intended.json")
+}
+
+func (f intendedFixture) requireUnmoved(t *testing.T) {
+ t.Helper()
+ if body, err := os.ReadFile(f.dbPath); err != nil || string(body) != "untrusted-local-database" {
+ t.Fatalf("refusal moved prior database: %v", err)
+ }
+ if route, err := f.reg.Get(f.prior.Host); err != nil || route == nil || route.ClaimStamp != f.prior.Stamp || route.ClaimOwner != "old-owner" {
+ t.Fatalf("refusal changed prior route: %+v %v", route, err)
+ }
+ if _, err := os.Lstat(f.intendedJournal()); !errors.Is(err, os.ErrNotExist) {
+ t.Fatalf("refusal recorded an intended capture: %v", err)
+ }
+}
+
+func (f intendedFixture) requireComplete(t *testing.T) {
+ t.Helper()
+ if _, err := os.Lstat(f.dbPath); !errors.Is(err, os.ErrNotExist) {
+ t.Fatalf("prior database not parked: %v", err)
+ }
+ if _, err := os.Lstat(f.state); !errors.Is(err, os.ErrNotExist) {
+ t.Fatalf("intended state dir remains: %v", err)
+ }
+ if route, err := f.reg.Get(f.prior.Host); err != nil || route == nil || route.ClaimStamp != f.next.Stamp {
+ t.Fatalf("route not handed to successor: %+v %v", route, err)
+ }
+}
+
+// Mkdirat failed (ENOSPC/EACCES) after the intent was durable: nothing exists
+// to move, so the next admission's handover must complete.
+func TestColdHandoverResumesAbsentIntendedStateDir(t *testing.T) {
+ for name, typ := range map[string]uint32{"typed": unix.S_IFDIR, "untyped": 0} {
+ t.Run(name, func(t *testing.T) {
+ f := newIntendedFixture(t, func(e *StopInventoryEntry) { e.Type = typ })
+ ctx := t.Context()
+ if err := f.handover.Run(ctx, f.prior, f.next); err != nil {
+ t.Fatalf("absent intended path stranded handover: %v", err)
+ }
+ f.requireComplete(t)
+ if _, err := os.Lstat(f.intendedJournal()); !errors.Is(err, os.ErrNotExist) {
+ t.Fatalf("absent intent captured an artifact: %v", err)
+ }
+ if _, err := os.Lstat(f.parked("-intended", f.prior.Host)); !errors.Is(err, os.ErrNotExist) {
+ t.Fatalf("absent intent parked something: %v", err)
+ }
+ if err := f.handover.Run(ctx, f.prior, f.next); err != nil {
+ t.Fatal(err)
+ }
+ if err := f.handover.Recover(ctx); err != nil {
+ t.Fatal(err)
+ }
+ f.requireComplete(t)
+ })
+ }
+}
+
+// Crash after Mkdirat and before the materialized write: the empty directory
+// is parked by its captured inode, never unlinked, and replay never recaptures.
+func TestColdHandoverParksTypedEmptyIntendedStateDirAndReplays(t *testing.T) {
+ f := newIntendedFixture(t, nil)
+ ctx := t.Context()
+ if err := os.Mkdir(f.state, 0700); err != nil {
+ t.Fatal(err)
+ }
+ created, err := os.Lstat(f.state)
+ if err != nil {
+ t.Fatal(err)
+ }
+ // Fail the main retirement after the intended park, as an interrupted run.
+ blocked := f.parked("-old", f.prior.Host+".db")
+ if err = os.MkdirAll(filepath.Dir(blocked), 0700); err != nil {
+ t.Fatal(err)
+ }
+ if err = os.WriteFile(blocked, []byte("obstruction"), 0600); err != nil {
+ t.Fatal(err)
+ }
+ if err = f.handover.Run(ctx, f.prior, f.next); !errors.Is(err, ErrChanged) {
+ t.Fatalf("obstructed main retirement: %v", err)
+ }
+ parked := f.parked("-intended", f.prior.Host)
+ got, err := os.Lstat(parked)
+ if err != nil || !os.SameFile(created, got) {
+ t.Fatalf("intended dir not parked by inode: %v", err)
+ }
+ if _, err = os.Lstat(f.state); !errors.Is(err, os.ErrNotExist) {
+ t.Fatalf("intended dir still at source: %v", err)
+ }
+ if _, err = os.Lstat(f.dbPath); err != nil {
+ t.Fatalf("main retirement ran past obstruction: %v", err)
+ }
+ body, err := os.ReadFile(f.intendedJournal())
+ if err != nil {
+ t.Fatal(err)
+ }
+ var journal RetirementJournal
+ if err = decodeStrict(body, &journal); err != nil || !journal.Complete || len(journal.Inventory.Artifacts) != 1 || journal.Target == nil || *journal.Target != f.prior {
+ t.Fatalf("intended capture not journalled under the exact prior: %+v %v", journal, err)
+ }
+
+ // A same-named directory appearing after the durable decision is unknown.
+ if err = os.Mkdir(f.state, 0700); err != nil {
+ t.Fatal(err)
+ }
+ if err = os.Remove(blocked); err != nil {
+ t.Fatal(err)
+ }
+ if err = f.handover.Run(ctx, f.prior, f.next); !errors.Is(err, ErrChanged) {
+ t.Fatalf("late same-named directory accepted: %v", err)
+ }
+ if _, err = os.Lstat(f.state); err != nil {
+ t.Fatalf("late directory touched: %v", err)
+ }
+ if _, err = os.Lstat(f.dbPath); err != nil {
+ t.Fatalf("refusal moved prior database: %v", err)
+ }
+ if err = os.Remove(f.state); err != nil {
+ t.Fatal(err)
+ }
+
+ if err = f.handover.Recover(ctx); err != nil {
+ t.Fatal(err)
+ }
+ f.requireComplete(t)
+ if got, err = os.Lstat(parked); err != nil || !os.SameFile(created, got) {
+ t.Fatalf("parked intended dir not preserved: %v", err)
+ }
+ if again, err := os.ReadFile(f.intendedJournal()); err != nil || !bytes.Equal(again, body) {
+ t.Fatalf("replay rewrote intended capture: %v", err)
+ }
+ mainBody, err := os.ReadFile(filepath.Join(f.handover.JournalDir, "retired", f.id+"-old.json"))
+ if err != nil {
+ t.Fatal(err)
+ }
+ var rows RetirementJournal
+ if err = decodeStrict(mainBody, &rows); err != nil || !rows.Complete || len(rows.Archives) == 0 {
+ t.Fatalf("row archives missing: %v", err)
+ }
+ if _, err = os.Lstat(f.parked("-old", f.prior.Host+".db")); err != nil {
+ t.Fatalf("parked prior database missing: %v", err)
+ }
+
+ for range 2 {
+ if err = f.handover.Run(ctx, f.prior, f.next); err != nil {
+ t.Fatal(err)
+ }
+ if err = f.handover.Recover(ctx); err != nil {
+ t.Fatal(err)
+ }
+ }
+ if got, err = os.Lstat(parked); err != nil || !os.SameFile(created, got) {
+ t.Fatalf("replay disturbed parked intended dir: %v", err)
+ }
+ if again, err := os.ReadFile(filepath.Join(f.handover.JournalDir, "retired", f.id+"-old.json")); err != nil || !bytes.Equal(again, mainBody) {
+ t.Fatalf("replay rewrote row archives: %v", err)
+ }
+}
+
+// Ownership of a present intended name is only provable for an empty
+// directory of the recorded type with no successor claim on it. Every other
+// shape refuses before any artifact moves, and keeps refusing.
+func TestColdHandoverRefusesUnprovableIntendedStateDir(t *testing.T) {
+ cases := []struct {
+ name string
+ edit func(*StopInventoryEntry)
+ setup func(*testing.T, intendedFixture)
+ want error
+ check func(*testing.T, intendedFixture)
+ }{
+ {
+ name: "nonempty",
+ setup: func(t *testing.T, f intendedFixture) {
+ if err := os.Mkdir(f.state, 0700); err != nil {
+ t.Fatal(err)
+ }
+ if err := os.WriteFile(filepath.Join(f.state, "data.db"), []byte("unknown"), 0600); err != nil {
+ t.Fatal(err)
+ }
+ },
+ want: ErrChanged,
+ check: func(t *testing.T, f intendedFixture) {
+ if body, err := os.ReadFile(filepath.Join(f.state, "data.db")); err != nil || string(body) != "unknown" {
+ t.Fatalf("nonempty intended dir touched: %v", err)
+ }
+ },
+ },
+ {
+ name: "symlink",
+ setup: func(t *testing.T, f intendedFixture) {
+ if err := os.Mkdir(filepath.Join(f.data, "elsewhere"), 0700); err != nil {
+ t.Fatal(err)
+ }
+ if err := os.Symlink("elsewhere", f.state); err != nil {
+ t.Fatal(err)
+ }
+ },
+ want: ErrChanged,
+ check: func(t *testing.T, f intendedFixture) {
+ if info, err := os.Lstat(f.state); err != nil || info.Mode()&os.ModeSymlink == 0 {
+ t.Fatalf("symlink touched: %v", err)
+ }
+ if _, err := os.Lstat(filepath.Join(f.data, "elsewhere")); err != nil {
+ t.Fatalf("symlink target touched: %v", err)
+ }
+ },
+ },
+ {
+ name: "regular file",
+ setup: func(t *testing.T, f intendedFixture) {
+ if err := os.WriteFile(f.state, []byte("unknown"), 0600); err != nil {
+ t.Fatal(err)
+ }
+ },
+ want: ErrChanged,
+ check: func(t *testing.T, f intendedFixture) {
+ if body, err := os.ReadFile(f.state); err != nil || string(body) != "unknown" {
+ t.Fatalf("regular file touched: %v", err)
+ }
+ },
+ },
+ {
+ name: "untyped intent",
+ edit: func(e *StopInventoryEntry) { e.Type = 0 },
+ want: ErrChanged,
+ },
+ {
+ name: "wrong recorded type",
+ edit: func(e *StopInventoryEntry) { e.Type = unix.S_IFREG },
+ want: ErrChanged,
+ },
+ {
+ name: "intent with identity",
+ edit: func(e *StopInventoryEntry) { e.Device, e.Inode = 1, 1 },
+ want: ErrTypedInventory,
+ },
+ {
+ name: "successor state",
+ setup: func(t *testing.T, f intendedFixture) {
+ if err := os.Mkdir(f.state, 0700); err != nil {
+ t.Fatal(err)
+ }
+ successor := StopInventory{Complete: true, Host: f.next.Host, Incarnation: f.next.Incarnation, InstallationStamp: f.next.Stamp, Box: f.next.Box, Enrollment: f.next.Enrollment, DirectoryRevision: f.next.DirectoryRevision, Entries: []StopInventoryEntry{{Kind: "path", ArtifactID: "successor-state", ManagedRoot: "data", RelativePath: f.prior.Host, Ownership: "successor-owner", Materialization: "intended", Type: unix.S_IFDIR}}}
+ if err := f.inventory.WriteStopInventory(t.Context(), successor); err != nil {
+ t.Fatal(err)
+ }
+ },
+ want: ErrChanged,
+ },
+ }
+ for _, tc := range cases {
+ t.Run(tc.name, func(t *testing.T) {
+ f := newIntendedFixture(t, tc.edit)
+ setup := tc.setup
+ if setup == nil {
+ setup = func(t *testing.T, f intendedFixture) {
+ if err := os.Mkdir(f.state, 0700); err != nil {
+ t.Fatal(err)
+ }
+ }
+ }
+ setup(t, f)
+ before, err := os.Lstat(f.state)
+ if err != nil {
+ t.Fatal(err)
+ }
+ for range 2 {
+ if err = f.handover.Run(t.Context(), f.prior, f.next); !errors.Is(err, tc.want) {
+ t.Fatalf("got %v, want %v", err, tc.want)
+ }
+ f.requireUnmoved(t)
+ if after, err := os.Lstat(f.state); err != nil || !os.SameFile(before, after) {
+ t.Fatalf("refused intended name changed: %v", err)
+ }
+ if _, err = os.Lstat(f.parked("-intended", f.prior.Host)); !errors.Is(err, os.ErrNotExist) {
+ t.Fatalf("refused intended name parked: %v", err)
+ }
+ }
+ if tc.check != nil {
+ tc.check(t, f)
+ }
+ })
+ }
+}
+
+func TestColdHandoverAbsentIntentionDecisionSurvivesInterruptedRetirement(t *testing.T) {
+ f := newIntendedFixture(t, nil)
+ ctx := t.Context()
+ blocked := f.parked("-old", f.prior.Host+".db")
+ if err := os.MkdirAll(filepath.Dir(blocked), 0700); err != nil {
+ t.Fatal(err)
+ }
+ if err := os.WriteFile(blocked, []byte("obstruction"), 0600); err != nil {
+ t.Fatal(err)
+ }
+ if err := f.handover.Run(ctx, f.prior, f.next); !errors.Is(err, ErrChanged) {
+ t.Fatalf("first retirement: %v", err)
+ }
+ if err := os.Remove(blocked); err != nil {
+ t.Fatal(err)
+ }
+ if err := os.Mkdir(f.state, 0700); err != nil {
+ t.Fatal(err)
+ }
+ unknown, err := os.Lstat(f.state)
+ if err != nil {
+ t.Fatal(err)
+ }
+ if err := f.handover.Recover(ctx); !errors.Is(err, ErrChanged) {
+ t.Fatalf("accepted newly appearing intended path: %v", err)
+ }
+ got, err := os.Lstat(f.state)
+ if err != nil || !os.SameFile(unknown, got) {
+ t.Fatalf("unknown path moved: %v", err)
+ }
+ f.requireUnmoved(t)
+ if err := os.Remove(f.state); err != nil {
+ t.Fatal(err)
+ }
+ if err := f.handover.Recover(ctx); err != nil {
+ t.Fatal(err)
+ }
+ f.requireComplete(t)
+}
+
+func TestColdHandoverEmptyDirectoryUsesCapturedIdentity(t *testing.T) {
+ root := t.TempDir()
+ p := TypedParker{Roots: map[string]string{"data": root}, ParkedRoot: root, Host: "host", Incarnation: "inc", Retirement: "retire"}
+ path := filepath.Join(root, "state")
+ if err := os.Mkdir(path, 0700); err != nil {
+ t.Fatal(err)
+ }
+ a, err := p.CaptureArtifact("data", "state", "owner", ArtifactPath)
+ if err != nil {
+ t.Fatal(err)
+ }
+ if err := p.VerifyEmptyDirectory(a); err != nil {
+ t.Fatal(err)
+ }
+ if err := os.Rename(path, filepath.Join(root, "captured")); err != nil {
+ t.Fatal(err)
+ }
+ if err := os.Mkdir(path, 0700); err != nil {
+ t.Fatal(err)
+ }
+ if err := p.VerifyEmptyDirectory(a); !errors.Is(err, ErrChanged) {
+ t.Fatalf("substitute empty directory accepted: %v", err)
+ }
+ if err := os.Remove(path); err != nil {
+ t.Fatal(err)
+ }
+ if err := os.Symlink("captured", path); err != nil {
+ t.Fatal(err)
+ }
+ if err := p.VerifyEmptyDirectory(a); err == nil {
+ t.Fatal("followed symlink to captured directory")
+ }
+ if err := os.Remove(path); err != nil {
+ t.Fatal(err)
+ }
+ if err := os.Rename(filepath.Join(root, "captured"), path); err != nil {
+ t.Fatal(err)
+ }
+ if err := os.WriteFile(filepath.Join(path, "unknown"), nil, 0600); err != nil {
+ t.Fatal(err)
+ }
+ if err := p.VerifyEmptyDirectory(a); !errors.Is(err, ErrChanged) {
+ t.Fatalf("nonempty directory accepted: %v", err)
+ }
+}
+```
+
+#### New file: `internal/hostbox/reconcile_adapter_layout_test.go`
+
+```go
+//go:build unix
+
+package hostbox
+
+import (
+ "context"
+ "errors"
+ "os"
+ "path/filepath"
+ "testing"
+ "time"
+
+ "github.com/carlosframework/platform/internal/hostclaim"
+ "github.com/carlosframework/platform/internal/hostguard"
+ "github.com/carlosframework/platform/internal/hostwrite"
+ "github.com/carlosframework/platform/internal/provision"
+ "github.com/carlosframework/platform/internal/registry"
+ "github.com/carlosframework/platform/internal/release"
+)
+
+// The production adapter, driven by the real reconcile passes, hands the
+// RecordSelection it derived to the record check. That check aborts the
+// install, so nothing is written; what it saw is the layout the claim
+// inventory would have been stamped with.
+func TestAdapterSelectionLayoutFollowsRoutePaths(t *testing.T) {
+ stop := errors.New("selection captured")
+ for _, tc := range []struct {
+ name string
+ sidecar bool
+ mode string
+ want string
+ }{
+ {"instance minted nested", false, release.LayoutModeNested, "nested"},
+ {"instance minted flat", false, release.LayoutModeFlat, "flat"},
+ {"sidecar stays flat on a nested box", true, release.LayoutModeNested, "flat"},
+ } {
+ t.Run(tc.name, func(t *testing.T) {
+ ctx := t.Context()
+ f := newAdmissionFixture(t, tc.sidecar)
+ reg, err := registry.Open(filepath.Join(t.TempDir(), "registry.db"))
+ if err != nil {
+ t.Fatal(err)
+ }
+ defer reg.Close()
+ inventory, err := OpenStopInventoryDB(filepath.Join(t.TempDir(), "inventory.db"))
+ if err != nil {
+ t.Fatal(err)
+ }
+ defer inventory.Close()
+ var seen []RecordSelection
+ adapter := ReconcileAdapter{
+ Registry: reg, Inventory: inventory, Box: "box", Enrollment: "en", DirectoryRevision: "1",
+ Reconciler: Reconciler{
+ Claims: f.store, Box: "box", Region: "ie",
+ Guard: hostguard.Guard{DataDir: f.data, LockDir: t.TempDir(), Claims: hostguard.StoreReader{Store: f.store}},
+ Writes: hostwrite.Config{Enabled: true},
+ Now: func() time.Time { return time.Unix(100, 0) },
+ CheckRecord: func(_ context.Context, s RecordSelection) error { seen = append(seen, s); return stop },
+ CheckLocal: func(context.Context, hostclaim.Observed, RecordSelection) error { return nil },
+ },
+ }
+ cfg := provision.ReconcileConfig{DataDir: f.data, Region: "ie"}
+ cfg.Layout = func(context.Context, release.Store) (string, error) { return tc.mode, nil }
+ adapter.Configure(&cfg)
+ if !cfg.ClaimEnabled {
+ t.Fatal("adapter did not enable claim admission")
+ }
+ if err := f.pass(ctx, f.store, reg, cfg); err != nil {
+ t.Fatal(err)
+ }
+ if len(seen) != 1 {
+ t.Fatalf("record check ran %d times, want 1", len(seen))
+ }
+ if seen[0].Layout != tc.want || seen[0].Host != f.host || seen[0].Family != f.family {
+ t.Fatalf("selection = %+v, want layout %q for %s %s", seen[0], tc.want, f.family, f.host)
+ }
+ if row, err := reg.Get(f.host); err != nil || row != nil {
+ t.Fatalf("aborted install wrote a route: %v", err)
+ }
+ if _, err := os.Stat(filepath.Join(f.data, "hostguard", f.host+".json")); !errors.Is(err, os.ErrNotExist) {
+ t.Fatalf("aborted install activated the guard: %v", err)
+ }
+ })
+ }
+}
+```
+
## Task 3: persist containment, inventory and guard recovery
-Persist reservations before spawn and intentions before materialization. Store controller PID and kernel process-start identity with each reservation. A complete inventory is an explicit validated record. Recovery never turns missing inventory into proof of emptiness. The guard's durable fence remains authoritative after agent restart, root restart and reboot. Cold handover adds a successor barrier before parking predecessor artifacts; only the exact completed handover can clear it.
+Persist reservations before spawn and intentions before materialization. Record the admitted session's boot ID, controller ID, PID, process-start identity and directory revision with its empty parent before restore. Append executable/state file identities and role/producer reservations to that same record when they become available. Store controller PID and kernel process-start identity with each reservation. A complete inventory is an explicit validated record. Recovery never turns missing inventory into proof of emptiness. The guard's durable fence remains authoritative after agent restart, root restart and reboot. Cold handover adds a successor barrier before parking predecessor artifacts; only the exact completed handover can clear it.
#### Patch: `internal/cgroup/cgroup.go`
@@ -2364,6 +4451,53 @@ func (j *LaunchJournal) Create(ctx context.Context, in LaunchInventory) error {
_, err = j.db.ExecContext(ctx, "INSERT INTO launches(identity,body) VALUES(?,?)", launchKey(in.Target), body)
return err
}
+
+// Prelaunch persists an admitted session's recoverable record before restore
+// or materialization. It names this controller and the session's empty parent
+// group only. State directories are inventoried before restore; file identities
+// and producer reservations are added later. A failure or crash from here on
+// leaves the exact tuple root stops and settles.
+func (j *LaunchJournal) Prelaunch(ctx context.Context, in LaunchInventory) error {
+ if len(in.Groups) != 1 || in.Groups[0].RelativePath == "" || in.Groups[0].Device == 0 || in.Groups[0].Inode == 0 || in.Groups[0].InvocationID != "" || len(in.Files) != 0 || !in.NoState || len(in.ProducerIDs) != 0 {
+ return ErrUnsupported
+ }
+ in.ProducerIDs = []string{}
+ return j.Create(ctx, in)
+}
+
+// SessionLaunches returns every record journaled for one writer session of
+// this box, whichever boot or controller wrote it. A restarted controller uses
+// it to name the original tuple. An empty result never proves emptiness.
+func (j *LaunchJournal) SessionLaunches(ctx context.Context, host, incarnation string, stamp, session uint64, box, enrollment string) ([]LaunchInventory, error) {
+ if j == nil || j.db == nil || !component(host) || incarnation == "" || stamp == 0 || session == 0 || box == "" || enrollment == "" {
+ return nil, ErrUnsupported
+ }
+ rows, err := j.db.QueryContext(ctx, "SELECT identity, body FROM launches")
+ if err != nil {
+ return nil, err
+ }
+ defer rows.Close()
+ var out []LaunchInventory
+ for rows.Next() {
+ var key string
+ var body []byte
+ if err = rows.Scan(&key, &body); err != nil {
+ return nil, err
+ }
+ var in LaunchInventory
+ if err = json.Unmarshal(body, &in); err != nil {
+ return nil, err
+ }
+ if !validLaunchOwner(in.Target) || launchKey(in.Target) != key {
+ return nil, ErrTypedInventory
+ }
+ t := in.Target
+ if t.Host == host && t.Incarnation == incarnation && t.Stamp == stamp && t.Session == session && t.Box == box && t.Enrollment == enrollment {
+ out = append(out, in)
+ }
+ }
+ return out, rows.Err()
+}
func (j *LaunchJournal) Read(ctx context.Context, t hostguard.FenceTarget) (LaunchInventory, error) {
var in LaunchInventory
var raw []byte
@@ -2919,12 +5053,69 @@ func (g Guard) ExistingFence(ctx context.Context, want FenceTarget) (FenceTarget
}
```
+#### Patch: `internal/hostbox/typedpark_linux.go`
+
+```diff
+--- a/internal/hostbox/typedpark_linux.go
++++ b/internal/hostbox/typedpark_linux.go
+@@ -8,0 +9,2 @@
++ "io"
++ "os"
+@@ -200,0 +203,34 @@
++
++// VerifyEmptyDirectory reads the captured directory itself, with no symlink
++// traversal. The caller still parks by the same recorded inode under cleanup.
++func (p TypedParker) VerifyEmptyDirectory(a Artifact) error {
++ if err := p.validate(); err != nil {
++ return err
++ }
++ root, ok := p.Roots[a.Root]
++ if !ok || !managedName(a.Name) || a.Type != unix.S_IFDIR {
++ return ErrTypedInventory
++ }
++ parent, err := openParent(root, a.Name, false)
++ if err != nil {
++ return err
++ }
++ defer unix.Close(parent.dir)
++ fd, err := unix.Openat(parent.dir, parent.base, unix.O_RDONLY|unix.O_DIRECTORY|unix.O_NOFOLLOW|unix.O_CLOEXEC, 0)
++ if err != nil {
++ return err
++ }
++ f := os.NewFile(uintptr(fd), a.Name)
++ defer f.Close()
++ var st unix.Stat_t
++ if err := unix.Fstat(fd, &st); err != nil {
++ return err
++ }
++ if !sameArtifact(a, &st) {
++ return ErrChanged
++ }
++ if names, err := f.Readdirnames(1); len(names) != 0 || !errors.Is(err, io.EOF) {
++ return ErrChanged
++ }
++ return nil
++}
+```
+
+#### Patch: `internal/hostbox/typedpark_other.go`
+
+```diff
+--- a/internal/hostbox/typedpark_other.go
++++ b/internal/hostbox/typedpark_other.go
+@@ -18,0 +19,2 @@
++
++func (TypedParker) VerifyEmptyDirectory(Artifact) error { return ErrTypedParkUnsupported }
+```
+
## Task 4: condition registry ownership, admission and retirement
-The create and keep callbacks run inside Reconciler.Install. Both removal families call typed retirement under the flag. Archive the complete selected SQL rows before deletion, then commit the archive and conditional deletes atomically. A replay may prove the predecessor was removed; it cannot delete a replacement. Ordinary upsert preserves a hold. A hold transition compares the captured ownership and observed hold_op, and does not clear failed-host state.
+Check InboundNone, ConfigGate and co-spawn refusals before calling claimInstall. Refusal must leave the guard and inventory byte-for-byte unchanged. Known incompatible or unowned existing rows also refuse before the hook. Only an accepted record reaches Reconciler.Install and its create or keep callback. Derive the inventory layout from the selected route, preserving main's nested paths and flat sidecar layout. Both removal families call typed retirement under the flag. Archive the complete selected SQL rows before deletion, then commit the archive and conditional deletes atomically. A replay may prove the predecessor was removed; it cannot delete a replacement. Ordinary upsert preserves a hold. A hold transition compares the captured ownership and observed hold_op, and does not clear failed-host state.
Cold handover deliberately parks the full old installation. It takes its route/timer snapshot only after positive drain, retains the archived bytes, assigns successor ownership to the restored rows, preserves timer registrations and scheduling bookkeeping, and performs a fresh filesystem restore. It never relabels a live database as the successor's. A completed handover cannot restore its original snapshot over a later registration.
+PrepareState records a typed directory intention before mkdir and captures its inode before restore. On recovery, an absent intended path moves nothing; if all intended paths are absent, persist that decision before the main retirement so replay refuses a newly appearing path. A present intended path is recoverable only as an empty directory of the recorded type, under the exact prior cleanup authority and successor barrier, with no successor effects or path inventory. Verify emptiness on the captured no-follow descriptor, persist that identity once, then park it through TypedRetirement. Never unlink it or recapture on replay. Nonempty, symlinked, untyped, mismatched or successor-owned paths refuse before any artifact moves. The barrier excludes legitimate writers; out-of-band privileged mutation remains outside this protocol, as in the existing typed parker.
+
#### New file: `internal/hostbox/cleanup_authority.go`
```go
@@ -2987,14 +5178,15 @@ type ColdHandover struct {
VerifyReceipt hostguard.CleanupVerify
}
type coldJournal struct {
- V int
- ID, Phase string
- Prior hostguard.FenceTarget
- Barrier hostguard.FenceTarget
- Successor ColdSuccessor
- Inventory StopInventory
- Receipt hostguard.FenceReceipt
- Before, After registry.ColdRows
+ V int
+ IntendedAbsent bool
+ ID, Phase string
+ Prior hostguard.FenceTarget
+ Barrier hostguard.FenceTarget
+ Successor ColdSuccessor
+ Inventory StopInventory
+ Receipt hostguard.FenceReceipt
+ Before, After registry.ColdRows
}
func ownershipFor(host, incarnation string, stamp uint64, box string) string {
@@ -3136,8 +5328,26 @@ func (h ColdHandover) Run(ctx context.Context, prior hostguard.FenceTarget, next
}
if j.Phase == "prepared" {
inventory := RetirementInventory{Host: prior.Host, Captured: hostguard.Captured{Incarnation: prior.Incarnation, Stamp: prior.Stamp}, Ownership: j.Before.Route.ClaimOwner}
+ var intended []StopInventoryEntry
+ names := map[string]int{}
for _, entry := range j.Inventory.Entries {
- if entry.Ownership != inventory.Ownership || entry.Materialization != "materialized" || entry.Kind == "unit-state" {
+ if entry.Kind != "route" && entry.Kind != "timer" {
+ names[entry.ManagedRoot+"\x00"+entry.RelativePath]++
+ }
+ if entry.Ownership != inventory.Ownership || entry.Kind == "unit-state" {
+ return ErrTypedInventory
+ }
+ // An intended path is an intent PrepareState persisted before its
+ // mkdir and never finalized. It carries no captured identity, so it
+ // is handled only by retireIntended below, never as an artifact.
+ if entry.Materialization == "intended" {
+ if entry.Kind != "path" || entry.Device != 0 || entry.Inode != 0 || !managedName(entry.RelativePath) || h.Roots[entry.ManagedRoot] == "" {
+ return ErrTypedInventory
+ }
+ intended = append(intended, entry)
+ continue
+ }
+ if entry.Materialization != "materialized" {
return ErrTypedInventory
}
if entry.Kind == "timer" {
@@ -3148,12 +5358,20 @@ func (h ColdHandover) Run(ctx context.Context, prior hostguard.FenceTarget, next
for _, timer := range j.Before.Timers {
inventory.Artifacts = append(inventory.Artifacts, Artifact{Kind: ArtifactTimer, Root: "registry", Name: timer.Host + "/" + timer.Name, Owner: timer.ClaimOwner, RowID: timer.ClaimRowID + "~" + timer.Registration})
}
+ for _, entry := range intended {
+ if names[entry.ManagedRoot+"\x00"+entry.RelativePath] != 1 {
+ return ErrTypedInventory
+ }
+ }
intent, err := NewRetirementJournal(j.ID+"-old", inventory)
if err != nil {
return err
}
intent.Target = &j.Prior
intent.StopReceipt = append([]byte(nil), j.Receipt.Bytes...)
+ if err = h.retireIntended(ctx, j, intended); err != nil {
+ return err
+ }
retire := TypedRetirement{JournalDir: filepath.Join(h.JournalDir, "retired"), Authority: InstallationCleanup{Guard: h.Guard, Verify: h.VerifyReceipt}, Filesystem: TypedParker{Roots: h.Roots, ParkedRoots: h.Roots, Incarnation: prior.Incarnation, Host: prior.Host, Retirement: intent.ID}, Rows: RegistryRows{Registry: h.Registry, Inventory: inventory}, Ack: func(context.Context, RetirementJournal) error { return nil }}
if err = retire.Run(ctx, prior, intent); err != nil {
return err
@@ -3250,6 +5468,140 @@ func (h ColdHandover) Run(ctx context.Context, prior hostguard.FenceTarget, next
return writeColdJournal(path, j)
})
}
+
+// retireIntended settles prior state paths whose intent is durable but whose
+// materialization is not. PrepareState verifies absence before persisting the
+// intent and refuses an intended name it finds present, so with the prior
+// controller fenced dead, the successor barred and without effects or path
+// inventory, the only writer of such a name is the prior's own mkdir. An absent
+// name moves nothing. A present one must still be an empty directory of the
+// recorded type; it is captured into a typed retirement journal and parked by
+// inode under exact cleanup authority, so it is preserved, never unlinked.
+// Anything else refuses before any artifact moves.
+func (h ColdHandover) retireIntended(ctx context.Context, j coldJournal, intended []StopInventoryEntry) error {
+ if len(intended) == 0 {
+ return nil
+ }
+ if successor, err := h.Inventory.ReadInstallation(ctx, j.Successor.Host, j.Successor.Incarnation, j.Successor.Stamp); err == nil {
+ for _, entry := range successor.Entries {
+ if entry.Kind != "route" {
+ return ErrChanged
+ }
+ }
+ } else if !errors.Is(err, sql.ErrNoRows) {
+ return err
+ }
+ id := j.ID + "-intended"
+ dir := filepath.Join(h.JournalDir, "retired")
+ parker := TypedParker{Roots: h.Roots, ParkedRoots: h.Roots, Incarnation: j.Prior.Incarnation, Host: j.Prior.Host, Retirement: id}
+ want := RetirementInventory{Host: j.Prior.Host, Captured: hostguard.Captured{Incarnation: j.Prior.Incarnation, Stamp: j.Prior.Stamp}, Ownership: j.Before.Route.ClaimOwner}
+ body, err := os.ReadFile(filepath.Join(dir, id+".json"))
+ resumed := err == nil
+ if resumed {
+ // The captured identities were decided once, durably, by this
+ // controller. Never recapture: the parked inode must be the journalled one.
+ var saved RetirementJournal
+ if decodeStrict(body, &saved) != nil || saved.ID != id || saved.Inventory.Host != want.Host || saved.Inventory.Captured != want.Captured || saved.Inventory.Ownership != want.Ownership {
+ return ErrTypedJournal
+ }
+ want.Artifacts = saved.Inventory.Artifacts
+ for _, a := range want.Artifacts {
+ found := 0
+ for _, entry := range intended {
+ if a.Root == entry.ManagedRoot && a.Name == entry.RelativePath {
+ found++
+ }
+ }
+ if found != 1 {
+ return ErrTypedJournal
+ }
+ }
+ } else if !errors.Is(err, os.ErrNotExist) {
+ return err
+ }
+ resumed = resumed || j.IntendedAbsent
+ for _, entry := range intended {
+ journalled := false
+ for _, a := range want.Artifacts {
+ if a.Root == entry.ManagedRoot && a.Name == entry.RelativePath {
+ if a.Kind != ArtifactPath || a.Owner != entry.Ownership || entry.Type == 0 || a.Type != entry.Type {
+ return ErrTypedJournal
+ }
+ journalled = true
+ }
+ }
+ if journalled {
+ continue
+ }
+ // Absence is read below the configured root only; a missing root is
+ // not evidence that the name was never materialized.
+ if info, err := os.Lstat(h.Roots[entry.ManagedRoot]); err != nil || !info.IsDir() {
+ return ErrTypedInventory
+ }
+ a, err := parker.CaptureArtifact(entry.ManagedRoot, entry.RelativePath, entry.Ownership, ArtifactPath)
+ if errors.Is(err, fs.ErrNotExist) {
+ continue
+ }
+ if err != nil {
+ return err
+ }
+ // A name that was absent when the journal was written has no recorded
+ // decision; its later appearance is unknown ownership.
+ if resumed || entry.Type == 0 || a.Type != entry.Type {
+ return ErrChanged
+ }
+ if err = parker.VerifyEmptyDirectory(a); err != nil {
+ return err
+ }
+ again, err := parker.CaptureArtifact(entry.ManagedRoot, entry.RelativePath, entry.Ownership, ArtifactPath)
+ if err != nil || again != a {
+ return ErrChanged
+ }
+ want.Artifacts = append(want.Artifacts, a)
+ }
+ if len(want.Artifacts) == 0 {
+ if j.IntendedAbsent {
+ return nil
+ }
+ // Persist the all-absent decision before the main retirement can move
+ // anything. Replay must not attribute a newly appearing name to prior.
+ return h.Guard.CleanupWith(ctx, j.Prior, h.VerifyReceipt, func() error {
+ path := filepath.Join(h.JournalDir, j.ID+".json")
+ body, err := os.ReadFile(path)
+ if err != nil {
+ return err
+ }
+ var saved coldJournal
+ if decodeStrict(body, &saved) != nil || saved.ID != j.ID || saved.Prior != j.Prior || saved.Successor != j.Successor || saved.Phase != "prepared" {
+ return ErrTypedJournal
+ }
+ saved.IntendedAbsent = true
+ return writeColdJournal(path, saved)
+ })
+ }
+ intent, err := NewRetirementJournal(id, want)
+ if err != nil {
+ return err
+ }
+ target := j.Prior
+ intent.Target = &target
+ intent.StopReceipt = append([]byte(nil), j.Receipt.Bytes...)
+ retire := TypedRetirement{JournalDir: dir, Authority: InstallationCleanup{Guard: h.Guard, Verify: h.VerifyReceipt}, Filesystem: parker, Rows: RegistryRows{Registry: h.Registry, Inventory: want}, Ack: func(context.Context, RetirementJournal) error { return nil }}
+ if err = retire.Run(ctx, j.Prior, intent); err != nil {
+ return err
+ }
+ // Every captured name is parked now; one present again is not the prior's.
+ for _, a := range want.Artifacts {
+ if _, err := parker.CaptureArtifact(a.Root, a.Name, a.Owner, ArtifactPath); !errors.Is(err, fs.ErrNotExist) {
+ if err != nil {
+ return err
+ }
+ return ErrChanged
+ }
+ }
+ return nil
+}
+
func writeColdJournal(path string, j coldJournal) error {
if err := os.MkdirAll(filepath.Dir(path), 0700); err != nil {
return err
@@ -3684,7 +6036,13 @@ func (a ReconcileAdapter) Install(ctx context.Context, rec provision.ClaimRecord
if err != nil {
return err
}
- selected := RecordSelection{Host: rec.Host, Key: rec.Key, ETag: etag, Incarnation: inc, Family: rec.Family, Layout: "flat", DryRun: rec.DryRun}
+ // The claim inventory records where this box's files are: the route's own
+ // paths say so (sidecars are always flat).
+ layout := "flat"
+ if rec.Family != "sidecar" && provision.NestedPaths(a.Reconciler.Guard.DataDir, rec.Route) {
+ layout = "nested"
+ }
+ selected := RecordSelection{Host: rec.Host, Key: rec.Key, ETag: etag, Incarnation: inc, Family: rec.Family, Layout: layout, DryRun: rec.DryRun}
reconciler := a.Reconciler
priorCheck := reconciler.CheckRecord
reconciler.CheckRecord = func(ctx context.Context, selected RecordSelection) error {
@@ -3995,6 +6353,12 @@ func (cfg ReconcileConfig) claimInstall(ctx context.Context, selected ClaimRecor
if cfg.ClaimInstall == nil {
return ErrClaimAdmission
}
+ // An existing row without a valid claim owner can never be kept: the
+ // write below would refuse it. Decide that before the hook, whose guard
+ // and inventory effects a refused row must not cause.
+ if existing != nil && !existing.ClaimOwnership.Valid() {
+ return registry.ErrClaimOwnership
+ }
return cfg.ClaimInstall(ctx, selected, func(owner registry.ClaimOwnership) error {
if !owner.Valid() || (existing != nil && existing.ClaimOwnership != owner) {
return registry.ErrClaimOwnership
@@ -4023,31 +6387,34 @@ func (cfg ReconcileConfig) claimRetire(ctx context.Context, reg *registry.Regist
+ ClaimInstall func(context.Context, ClaimRecord, func(registry.ClaimOwnership) error) error
+ ClaimRetire func(context.Context, registry.Route) error
+
-@@ -1002,33 +1006,23 @@
+@@ -1051,2 +1055,11 @@
- switch {
- case !ok:
-- // A box that declared CARLOS_INBOUND=none (design §8) has told
-- // this pass what work it will ACCEPT, not made a claim about its
-- // network \u2014 fleet boxes are already outbound-only dial in this
-- // estate. Ordinary instance records carry no placement
-- // vocabulary of their own (that is ReconcileSidecars' gate,
-- // Task 9, over SidecarRecord.Placement); every routed instance
-- // this pass mints needs inbound service, so a no-inbound box
-- // must refuse ALL of them outright, loudly, rather than mint
-- // something it can never serve \u2014 a misfiled record becomes a
-- // log line here, never a silent outage.
-- if cfg.Inbound == InboundNone {
-- log.Printf("provision: refusing to mint route %s (app %q acct %q): this box declares "+
-- "CARLOS_INBOUND=none (no-inbound) and cannot serve routed instance traffic", rec.Host, rec.App, rec.Account)
-- continue
-- }
++
++ // Refusals that need no claim are decided BEFORE cfg.claimInstall: the
++ // hook activates the host guard and writes inventory, effects that a
++ // record this pass will not act on must never cause. Only work that
++ // is going to touch the registry reaches it.
++ var prior *registry.Route
++ nested := false
++ if ok {
++ prior = &existing
++ nested = NestedPaths(cfg.DataDir, existing)
++ } else {
+@@ -1068,6 +1080,0 @@
- // The route's whole shape now comes from the record (routeFor):
- // Kind, Addr, Unit, Environment, StatusNamed, Hibernate and the
- // path derivation. A record carrying none of them still mints the
- // exact route this branch hardcoded before they existed \u2014 that is
- // the additivity rule, pinned by TestRecordWireShapeIsAdditive and
- // TestReconcileMints.
-- nr := routeFor(rec, cfg.DataDir, seed)
+@@ -1078,2 +1085,2 @@
+- nested, lerr := mintLayout()
+- if lerr != nil {
++ var lerr error
++ if nested, lerr = mintLayout(); lerr != nil {
+@@ -1083,10 +1090,39 @@
+- nr := routeFor(rec, cfg.DataDir, seed, nested)
- // A box that cannot provide this instance's config must not
- // admit it: it would start without its secrets, and a keyed
- // sibling would be left with a route that lost the race to one
@@ -4057,30 +6424,46 @@ func (cfg ReconcileConfig) claimRetire(ctx context.Context, reg *registry.Regist
- if cfg.ConfigGate != nil {
- if err := cfg.ConfigGate(ctx, nr); err != nil {
- continue
-+
-+ var prior *registry.Route
-+ if ok {
-+ prior = &existing
+ }
-+ selected := ClaimRecord{Host: rec.Host, Key: RecordKey(rec.Account, rec.App, rec.Host), Family: "instance", Account: rec.Account, App: rec.App, CreatedAt: rec.CreatedAt, Value: rec, Route: routeFor(rec, cfg.DataDir, seed), DryRun: cfg.DryRun}
++ // The route's whole shape now comes from the record (routeFor):
++ // Kind, Addr, Unit, Environment, StatusNamed, Hibernate and the
++ // path derivation. A record carrying none of them still mints the
++ // exact route this branch hardcoded before they existed \u2014 that is
++ // the additivity rule, pinned by TestRecordWireShapeIsAdditive and
++ // TestReconcileMints.
++ nr := routeFor(rec, cfg.DataDir, seed, nested)
++ // A box that cannot provide this instance's config must not
++ // admit it: it would start without its secrets, and a keyed
++ // sibling would be left with a route that lost the race to one
++ // that was never going to work. Refused before the claim hook, the
++ // write and the dry-run line, so a dry run reports what a real pass
++ // would do.
++ if !ok && cfg.ConfigGate != nil {
++ if err := cfg.ConfigGate(ctx, nr); err != nil {
++ continue
++ }
++ }
++ // An existing row this record neither owns nor may adopt is known to be
++ // left alone before any claim work: with claims on, the hook would
++ // activate the guard and write inventory only to be refused after.
++ // (Flag off, the same decision is made inside the write below.)
++ if ok && cfg.ClaimEnabled && !(existing.Provisioned && existing.App == rec.App && existing.Account == rec.Account) {
++ if reason := shapeMismatch(existing, nr); reason != "" {
++ log.Printf("provision: %s exists but is not the route this record describes (%s) \u2014 leaving it alone",
++ rec.Host, reason)
++ continue
++ }
++ }
++ selected := ClaimRecord{Host: rec.Host, Key: RecordKey(rec.Account, rec.App, rec.Host), Family: "instance", Account: rec.Account, App: rec.App, CreatedAt: rec.CreatedAt, Value: rec, Route: nr, DryRun: cfg.DryRun}
+ if err := cfg.claimInstall(ctx, selected, prior, func(owner registry.ClaimOwnership) error {
+ switch {
+ case !ok:
-+ // A box that declared CARLOS_INBOUND=none (design §8) has told
-+ // this pass what work it will ACCEPT, not made a claim about its
-+ // network \u2014 fleet boxes are already outbound-only dial in this
-+ // estate. Ordinary instance records carry no placement
-+ // vocabulary of their own (that is ReconcileSidecars' gate,
-+ // Task 9, over SidecarRecord.Placement); every routed instance
-+ // this pass mints needs inbound service, so a no-inbound box
-+ // must refuse ALL of them outright, loudly, rather than mint
-+ // something it can never serve \u2014 a misfiled record becomes a
-+ // log line here, never a silent outage.
-+ if cfg.Inbound == InboundNone {
-+ log.Printf("provision: refusing to mint route %s (app %q acct %q): this box declares "+
-+ "CARLOS_INBOUND=none (no-inbound) and cannot serve routed instance traffic", rec.Host, rec.App, rec.Account)
++ nr.ClaimOwnership = owner
++ if cfg.DryRun {
++ log.Printf("provision: dry-run: would mint %s route %s (app %q acct %q channel %q)",
++ nr.Kind, rec.Host, rec.App, rec.Account, rec.Channel)
+ return nil
-@@ -1036,45 +1030,18 @@
+@@ -1094,45 +1130,117 @@
- }
- if cfg.DryRun {
- log.Printf("provision: dry-run: would mint %s route %s (app %q acct %q channel %q)",
@@ -4126,36 +6509,6 @@ func (cfg ReconcileConfig) claimRetire(ctx context.Context, reg *registry.Regist
- dropped := ""
- if !*rec.Hibernate && existing.Replication != "" {
- dropped = fmt.Sprintf(" (replication %q cleared: only a hibernating route's rung is read)", existing.Replication)
-+ // The route's whole shape now comes from the record (routeFor):
-+ // Kind, Addr, Unit, Environment, StatusNamed, Hibernate and the
-+ // path derivation. A record carrying none of them still mints the
-+ // exact route this branch hardcoded before they existed \u2014 that is
-+ // the additivity rule, pinned by TestRecordWireShapeIsAdditive and
-+ // TestReconcileMints.
-+ nr := routeFor(rec, cfg.DataDir, seed)
-+ nr.ClaimOwnership = owner
-+ // A box that cannot provide this instance's config must not
-+ // admit it: it would start without its secrets, and a keyed
-+ // sibling would be left with a route that lost the race to one
-+ // that was never going to work. Refused before the write, and
-+ // before the dry-run line, so a dry run reports what a real pass
-+ // would do.
-+ if cfg.ConfigGate != nil {
-+ if err := cfg.ConfigGate(ctx, nr); err != nil {
-+ return nil
-+ }
-@@ -1083,5 +1050,3 @@
-- log.Printf("provision: dry-run: would converge %s hibernate %v -> %v%s", rec.Host, existing.Hibernate, *rec.Hibernate, dropped)
-- } else if err := reg.SetHibernate(rec.Host, *rec.Hibernate); err != nil {
-- log.Printf("provision: converge hibernate on %s: %v", rec.Host, err)
-- } else {
-- log.Printf("provision: converged %s hibernate %v -> %v%s", rec.Host, existing.Hibernate, *rec.Hibernate, dropped)
-+ log.Printf("provision: dry-run: would mint %s route %s (app %q acct %q channel %q)",
-+ nr.Kind, rec.Host, rec.App, rec.Account, rec.Channel)
-+ return nil
-@@ -1089,2 +1054,115 @@
-- }
-- if existing.Gate != rec.Gate {
+ if err := reg.Upsert(nr); err != nil {
+ log.Printf("provision: mint %s: %v", rec.Host, err)
+ return nil
@@ -4241,7 +6594,7 @@ func (cfg ReconcileConfig) claimRetire(ctx context.Context, reg *registry.Regist
+ // by design (the layout pass moved them), so when that is the
+ // only difference the line says so, rather than pointing an
+ // operator at `carlos remove`.
-+ want := routeFor(rec, cfg.DataDir, seed)
++ want := routeFor(rec, cfg.DataDir, seed, false)
+ unitDrift := rec.Unit != "" && want.Unit != existing.Unit
+ if want.Kind != existing.Kind || unitDrift ||
+ want.Socket != existing.Socket || want.DBPath != existing.DBPath {
@@ -4265,18 +6618,20 @@ func (cfg ReconcileConfig) claimRetire(ctx context.Context, reg *registry.Regist
+ // as before: adoption grants this pass's removal phase authority
+ // over the row, so the match test is the only thing standing
+ // between a backfill and a retired production route.
-+ want := routeFor(rec, cfg.DataDir, seed)
++ // A hand-added row on a nested box (`carlos add` there mints
++ // nested too) matches the record at the nested paths; the row's
++ // own shape says which derivation to compare against.
++ want := routeFor(rec, cfg.DataDir, seed, NestedPaths(cfg.DataDir, existing))
+ if reason := shapeMismatch(existing, want); reason != "" {
+ log.Printf("provision: %s exists but is not the route this record describes (%s) \u2014 leaving it alone",
+ rec.Host, reason)
+ return nil
-+ }
-@@ -1092,5 +1170,12 @@
-- log.Printf("provision: dry-run: would converge %s gate %s -> %s", rec.Host, gateWord(existing.Gate), gateWord(rec.Gate))
-- } else if err := reg.SetGate(rec.Host, rec.Gate); err != nil {
-- log.Printf("provision: converge gate on %s: %v", rec.Host, err)
+@@ -1141,5 +1249,12 @@
+- log.Printf("provision: dry-run: would converge %s hibernate %v -> %v%s", rec.Host, existing.Hibernate, *rec.Hibernate, dropped)
+- } else if err := reg.SetHibernate(rec.Host, *rec.Hibernate); err != nil {
+- log.Printf("provision: converge hibernate on %s: %v", rec.Host, err)
- } else {
-- log.Printf("provision: converged %s gate %s -> %s", rec.Host, gateWord(existing.Gate), gateWord(rec.Gate))
+- log.Printf("provision: converged %s hibernate %v -> %v%s", rec.Host, existing.Hibernate, *rec.Hibernate, dropped)
+ log.Printf("provision: dry-run: would adopt existing route %s (app %q acct %q) \u2014 this record would own its channel and environment",
+ rec.Host, rec.App, rec.Account)
+ // convergeOwned reports channel and environment together,
@@ -4289,7 +6644,19 @@ func (cfg ReconcileConfig) claimRetire(ctx context.Context, reg *registry.Regist
+ // writes nothing.
+ convergeOwned(reg, existing, rec, true)
+ return nil
-@@ -1098,8 +1183,3 @@
+@@ -1147,8 +1262,3 @@
+- }
+- if existing.Gate != rec.Gate {
+- if cfg.DryRun {
+- log.Printf("provision: dry-run: would converge %s gate %s -> %s", rec.Host, gateWord(existing.Gate), gateWord(rec.Gate))
+- } else if err := reg.SetGate(rec.Host, rec.Gate); err != nil {
+- log.Printf("provision: converge gate on %s: %v", rec.Host, err)
+- } else {
+- log.Printf("provision: converged %s gate %s -> %s", rec.Host, gateWord(existing.Gate), gateWord(rec.Gate))
++ if err := reg.SetProvisioned(rec.Host, true); err != nil {
++ log.Printf("provision: adopt %s: %v", rec.Host, err)
++ return nil
+@@ -1156,64 +1266 @@
- }
- // seedIsKnown guards against stamping a transient manifest-read
- // failure's "" fallback over a working seed \u2014 see seedFor.
@@ -4298,10 +6665,7 @@ func (cfg ReconcileConfig) claimRetire(ctx context.Context, reg *registry.Regist
- log.Printf("provision: dry-run: would converge %s seed", rec.Host)
- } else if err := reg.SetSeed(rec.Host, seed); err != nil {
- log.Printf("provision: converge seed on %s: %v", rec.Host, err)
-+ if err := reg.SetProvisioned(rec.Host, true); err != nil {
-+ log.Printf("provision: adopt %s: %v", rec.Host, err)
-+ return nil
-@@ -1107,52 +1187 @@
+- }
- }
- // Unit is compared only when the RECORD states one: `carlos
- // route -unit` (issue #110) is the supported way to stamp the
@@ -4322,7 +6686,7 @@ func (cfg ReconcileConfig) claimRetire(ctx context.Context, reg *registry.Regist
- // by design (the layout pass moved them), so when that is the
- // only difference the line says so, rather than pointing an
- // operator at `carlos remove`.
-- want := routeFor(rec, cfg.DataDir, seed)
+- want := routeFor(rec, cfg.DataDir, seed, false)
- unitDrift := rec.Unit != "" && want.Unit != existing.Unit
- if want.Kind != existing.Kind || unitDrift ||
- want.Socket != existing.Socket || want.DBPath != existing.DBPath {
@@ -4346,7 +6710,10 @@ func (cfg ReconcileConfig) claimRetire(ctx context.Context, reg *registry.Regist
- // as before: adoption grants this pass's removal phase authority
- // over the row, so the match test is the only thing standing
- // between a backfill and a retired production route.
-- want := routeFor(rec, cfg.DataDir, seed)
+- // A hand-added row on a nested box (`carlos add` there mints
+- // nested too) matches the record at the nested paths; the row's
+- // own shape says which derivation to compare against.
+- want := routeFor(rec, cfg.DataDir, seed, NestedPaths(cfg.DataDir, existing))
- if reason := shapeMismatch(existing, want); reason != "" {
- log.Printf("provision: %s exists but is not the route this record describes (%s) \u2014 leaving it alone",
- rec.Host, reason)
@@ -4355,7 +6722,7 @@ func (cfg ReconcileConfig) claimRetire(ctx context.Context, reg *registry.Regist
- if cfg.DryRun {
- log.Printf("provision: dry-run: would adopt existing route %s (app %q acct %q) \u2014 this record would own its channel and environment",
+ log.Printf("provision: adopted existing route %s (app %q acct %q) \u2014 this record now owns its channel and environment",
-@@ -1160,24 +1189,11 @@
+@@ -1221,24 +1268,11 @@
- // convergeOwned reports channel and environment together,
- // and only the columns that actually differ (design §4.1) \u2014
- // a fixed "converging channel X -> Y" line here would both
@@ -4391,7 +6758,7 @@ func (cfg ReconcileConfig) claimRetire(ctx context.Context, reg *registry.Regist
+ return nil
+ }); err != nil {
+ log.Printf("provision: claim install %s: %v", rec.Host, err)
-@@ -1276 +1292 @@
+@@ -1337 +1371 @@
- if err := reg.Delete(rt.Host); err != nil {
+ if err := cfg.claimRetire(ctx, reg, rt); err != nil {
```
@@ -4401,68 +6768,22 @@ func (cfg ReconcileConfig) claimRetire(ctx context.Context, reg *registry.Regist
```diff
--- a/internal/provision/sidecar_reconcile.go
+++ b/internal/provision/sidecar_reconcile.go
-@@ -375,30 +375,29 @@
+@@ -375,2 +375,10 @@
- switch {
- case !ok:
-- // One host, one mechanism: a host already carrying the legacy
-- // file-staged co-spawn must not ALSO get a native sidecar row, or
-- // the same workload runs twice under two owners that know nothing
-- // of each other.
-- //
-- // This is a MINT-TIME check and nothing more. A -sidecar.env
-- // staged AFTER this row exists is not detected, and that host then
-- // carries both mechanisms with no complaint from this pass. It is
-- // left that way deliberately: the reverse \u2014 a converge-time
-- // refusal \u2014 would mean a file dropped by hand could stop an
-- // already-provisioned sidecar converging, which hands anything
-- // that can write the data dir a way to freeze a member's row. The
-- // direction that matters is the one guarded: the file came first,
-- // so the file keeps the host.
-- if coSpawnStaged(cfg.DataDir, rec.Host) {
-- log.Printf("provision: REFUSING to mint sidecar %s: %s%s is staged on this box \u2014 "+
-- "the legacy file co-spawn and a native sidecar row are two mechanisms for one host; "+
-- "remove the file to migrate this host onto sidecar records",
-- rec.Host, rec.Host, sidecarCoSpawnSuffix)
-- continue
-- }
-- nr := sidecarRouteFor(rec, m, cfg.DataDir)
-- // Same admission rule as the instance pass (amadan #213): a
-- // sidecar that would start without its sealed config is not
-- // minted here. See ReconcileConfig.ConfigGate.
-- if cfg.ConfigGate != nil {
-- if err := cfg.ConfigGate(ctx, nr); err != nil {
-- continue
+
++ // Refusals that need no claim are decided BEFORE cfg.claimInstall: the
++ // hook activates the host guard and writes inventory, effects that a
++ // record this pass will not act on must never cause. Only work that
++ // is going to touch the registry reaches it.
+ var prior *registry.Route
++ nr := sidecarRouteFor(rec, m, cfg.DataDir)
+ if ok {
+ prior = &existing
-+ }
-+ selected := ClaimRecord{Host: rec.Host, Key: SidecarRecordKey(rec.Account, rec.App, rec.Name), Family: "sidecar", Account: rec.Account, App: rec.App, CreatedAt: rec.CreatedAt, Value: rec, Route: sidecarRouteFor(rec, m, cfg.DataDir), DryRun: cfg.DryRun}
-+ if err := cfg.claimInstall(ctx, selected, prior, func(owner registry.ClaimOwnership) error {
-+ switch {
-+ case !ok:
-+ // One host, one mechanism: a host already carrying the legacy
-+ // file-staged co-spawn must not ALSO get a native sidecar row, or
-+ // the same workload runs twice under two owners that know nothing
-+ // of each other.
-+ //
-+ // This is a MINT-TIME check and nothing more. A -sidecar.env
-+ // staged AFTER this row exists is not detected, and that host then
-+ // carries both mechanisms with no complaint from this pass. It is
-+ // left that way deliberately: the reverse \u2014 a converge-time
-+ // refusal \u2014 would mean a file dropped by hand could stop an
-+ // already-provisioned sidecar converging, which hands anything
-+ // that can write the data dir a way to freeze a member's row. The
-+ // direction that matters is the one guarded: the file came first,
-+ // so the file keeps the host.
-+ if coSpawnStaged(cfg.DataDir, rec.Host) {
-+ log.Printf("provision: REFUSING to mint sidecar %s: %s%s is staged on this box \u2014 "+
-+ "the legacy file co-spawn and a native sidecar row are two mechanisms for one host; "+
-+ "remove the file to migrate this host onto sidecar records",
-+ rec.Host, rec.Host, sidecarCoSpawnSuffix)
-+ return nil
-@@ -406,24 +405,37 @@
-- }
++ } else {
+@@ -398 +405,0 @@
+- nr := sidecarRouteFor(rec, m, cfg.DataDir)
+@@ -407,21 +414,7 @@
- if cfg.DryRun {
- log.Printf("provision: dry-run: would mint sidecar route %s (app %q acct %q channel %q args %s)",
- rec.Host, rec.App, rec.Account, nr.Channel, nr.Args)
@@ -4484,18 +6805,21 @@ func (cfg ReconcileConfig) claimRetire(ctx context.Context, reg *registry.Regist
- // backfill can legitimately exist \u2014 and adoption is an authority
- // TRANSFER (it is what grants the removal phase power over a row),
- // never a shape coincidence.
-- log.Printf("provision: %s exists but is not a sidecar row this pass owns (kind %q provisioned %v app %q acct %q) \u2014 leaving it alone",
-- rec.Host, existing.Kind, existing.Provisioned, existing.App, existing.Account)
-+ nr := sidecarRouteFor(rec, m, cfg.DataDir)
++ }
++ // A row at this host that this pass does not own is left alone before
++ // any claim work, so the hook never activates the guard or writes
++ // inventory for it. (Flag off, the same decision is made inside the
++ // write below.)
++ if ok && cfg.ClaimEnabled && !(existing.Kind == SidecarKind && existing.Provisioned &&
++ existing.App == rec.App && existing.Account == rec.Account) {
+@@ -429,0 +423,34 @@
++ continue
++ }
++ selected := ClaimRecord{Host: rec.Host, Key: SidecarRecordKey(rec.Account, rec.App, rec.Name), Family: "sidecar", Account: rec.Account, App: rec.App, CreatedAt: rec.CreatedAt, Value: rec, Route: nr, DryRun: cfg.DryRun}
++ if err := cfg.claimInstall(ctx, selected, prior, func(owner registry.ClaimOwnership) error {
++ switch {
++ case !ok:
+ nr.ClaimOwnership = owner
-+ // Same admission rule as the instance pass (amadan #213): a
-+ // sidecar that would start without its sealed config is not
-+ // minted here. See ReconcileConfig.ConfigGate.
-+ if cfg.ConfigGate != nil {
-+ if err := cfg.ConfigGate(ctx, nr); err != nil {
-+ return nil
-+ }
-+ }
+ if cfg.DryRun {
+ log.Printf("provision: dry-run: would mint sidecar route %s (app %q acct %q channel %q args %s)",
+ rec.Host, rec.App, rec.Account, nr.Channel, nr.Args)
@@ -4523,7 +6847,7 @@ func (cfg ReconcileConfig) claimRetire(ctx context.Context, reg *registry.Regist
+ return nil
+ }); err != nil {
+ log.Printf("provision: claim install %s: %v", rec.Host, err)
-@@ -459 +471 @@
+@@ -459 +486 @@
- if err := reg.Delete(rt.Host); err != nil {
+ if err := cfg.claimRetire(ctx, reg, rt); err != nil {
```
@@ -5138,7 +7462,7 @@ func (r *ColdRows) UnmarshalJSON(body []byte) error {
## Task 5: reserve serving retries and role launches
-Read-only prerequisite failures spend no attempt. Reserve before writer admission, bind the new session before any spawn, and settle every failure through the exact root stop target. A failed admission also releases its held lease; it does not refund retry authority. Run restore and materialization before preparing the concrete per-role launch. Pin a verified executable descriptor, register the application producer, journal the role reservation, and run final admission and Start under the captured guard.
+Read-only prerequisite failures spend no attempt. Reserve before writer admission, bind the new session before any spawn, and settle every failure through the exact root stop target. A failed admission also releases its held lease; it does not refund retry authority. Recovery uses the persisted attempt, including its boot and controller identity, and refuses while its former controller can still run. Persist the exact pre-launch attempt and controller identity before restore or materialization. Restore failure, materialization failure and a crash between them must remain discoverable by stop and settlement. Prepare the concrete per-role launch only after those steps complete. Pin a verified executable descriptor, register the application producer, journal the role reservation, and run final admission and Start under the captured guard.
Mount reset-wake-retry on POST /api/cli/ops/hosts/{host}/resolve with human operator authentication, flag gating, strict bounded JSON, conditional incarnation/episode checks and durable operator-request identity. The Plan 8b host-resolution dispatcher owns the other actions. When the branches meet, put this action in that dispatcher's action switch and retain its declare-lost/readmit routes and checks; do not register a second handler for the same pattern. This reset grants eligibility only.
@@ -5896,17 +8220,15 @@ Keep placement region separate from the AWS region used to sign regional lease r
```diff
--- a/internal/hostbox/identity.go
+++ b/internal/hostbox/identity.go
-@@ -30,0 +31 @@
+@@ -32,0 +33 @@
+ AWSRegion string
-@@ -38 +39 @@
+@@ -40 +41 @@
- if p.Roster == nil || p.Evidence == nil || p.Account == "" || p.Box == "" || p.Region == "" || p.Bucket == "" || p.Namespace == "" || len(p.Certs) == 0 {
+ if p.Roster == nil || p.Evidence == nil || p.Account == "" || p.Box == "" || p.Region == "" || p.AWSRegion == "" || p.Bucket == "" || p.Namespace == "" || len(p.Certs) == 0 {
-@@ -45 +46 @@
+@@ -47 +48 @@
- if a.Box != p.Box || a.Enrollment == "" || a.Revision == 0 || !a.Eligible || a.Quarantined || a.Region != p.Region || a.Bucket != p.Bucket || a.Namespace != p.Namespace || a.Cloud.Provider != "aws" || a.Cloud.Account != p.Account || a.Cloud.Region != p.Region || a.Cloud.InstanceID == "" {
+ if !p.actorMatches(a) {
-@@ -47,0 +49 @@
-+
-@@ -62,0 +65,4 @@
+@@ -68,0 +70,4 @@
+}
+
+func (p IdentityProvider) actorMatches(a Actor) bool {
@@ -6047,6 +8369,10 @@ The actual agent, root adoption and instance-control entrypoints create the depl
Use the existing Unix control socket. The kernel authenticates carlos UID and current edge MainPID, plus process-start, boot and controller identity. Root writes the exact durable fence, releases hostguard, challenges the agent to cancel and join registered producers, checks the complete containment tree and privileged descriptor inventory, then persists and publishes the original receipt bytes. No client supplies paths, arbitrary PIDs or units. Serving stop has no operation obligation and cannot acknowledge one. Existing legacy mutation verbs refuse under the flag.
+Restarted controllers authenticate as themselves but request root recovery of the persisted attempt's boot, controller, session and directory revision. Root loads that original journal, proves the former controller cannot run, checks containment and descriptors, and only then publishes evidence and settles the retry. The new controller cannot drain on the old one's behalf. Missing launch records refuse; they never become positive empty evidence. A zero-producer pre-launch record is valid: compare producer membership independently of whether JSON encoded an empty slice as null or an empty array, while still validating the acknowledgement digest and rejecting extra producers.
+
+The pre-launch record is durable before the retry bind, restore and materialization. The interval between remote admission and that commit still depends on the existing cold-handover barrier and Plan 5 positive-empty/backfill authority. A failure in handover, guard activation or pre-launch persistence must refuse without fabricated evidence or a retry-budget refund. This plan does not claim to resolve that separate admission interval by treating a missing journal as empty.
+
Timer delivery registers after any cold wake and immediately before backend HTTP, using its original captured selection. Registering before the wake would make the wake wait for its own timer during predecessor drain. A changed session therefore refuses that selected delivery without bookkeeping; a later tick can select the successor normally.
#### Patch: `internal/host/agent.go`
@@ -6054,9 +8380,9 @@ Timer delivery registers after any cold wake and immediately before backend HTTP
```diff
--- a/internal/host/agent.go
+++ b/internal/host/agent.go
-@@ -179,0 +180 @@
+@@ -183,0 +184 @@
+ "github.com/carlosframework/platform/internal/hostbox"
-@@ -206,7 +206,0 @@
+@@ -210,7 +210,0 @@
-// productionClaimFactory is the fail-closed production seam until Plan 8
-// supplies verified roster, evidence, lease, and move providers. Startup does
-// not call it while CARLOS_HOST_CLAIM is disabled.
@@ -6064,10 +8390,10 @@ Timer delivery registers after any cold wake and immediately before backend HTTP
- return ClaimProviders{}, hostwrite.ErrUnavailable
-}
-
-@@ -216 +210 @@
+@@ -220 +214 @@
- claims, err := newClaimWiring(agentCtx, os.Getenv, buildClaims)
+ claimCtx, err := claimEntryContext(agentCtx, os.Getenv, openDeployStore)
-@@ -244 +238,6 @@
+@@ -248 +242,6 @@
- cgm := cgroup.Setup(log.Printf)
+ var cgm *cgroup.Manager
+ if hostwrite.Enabled(claimEntryEnv(claimCtx)) {
@@ -6075,13 +8401,13 @@ Timer delivery registers after any cold wake and immediately before backend HTTP
+ } else {
+ cgm = cgroup.Setup(log.Printf)
+ }
-@@ -266,0 +266,5 @@
+@@ -270,0 +270,5 @@
+ }
+ claimCtx = withClaimRuntimeResources(claimCtx, ClaimRuntimeResources{Mode: "agent", Registry: reg, Manager: cgm, DataDir: dd, OptDir: release.OptDir(), UnitsDir: "/etc/systemd/system"})
+ claims, err := newClaimWiring(claimCtx, claimEntryEnv(claimCtx), buildClaims)
+ if err != nil {
+ log.Fatalf("agent: host claims: %v", err)
-@@ -409,23 +413,27 @@
+@@ -413,23 +417,27 @@
- //
- // Then layout recovery (design §8.2, "Recovery is the journal"):
- // every journal a previous agent left is completed, and the host
@@ -6132,30 +8458,30 @@ Timer delivery registers after any cold wake and immediately before backend HTTP
+ },
+ svc.Start)
+ }
-@@ -537 +545 @@
+@@ -541 +549 @@
- go fastConvergeLoop(reg, svc, sd, claims.Writes, serialClaims)
+ go fastConvergeLoop(reg, svc, sd, claims.Writes, serialClaims, claims.Providers.Provision)
-@@ -549 +557 @@
+@@ -553 +561 @@
- go instancesLoop(reg)
+ go instancesLoop(reg, claims.Providers.Provision)
-@@ -694,0 +703 @@
+@@ -701,0 +710 @@
+ sd.runner.StartDelivery = w.Providers.Delivery
-@@ -2030 +2039 @@
+@@ -2047 +2056 @@
-func fastConvergeLoop(reg *registry.Registry, svc *activator.Service, sd *scheduleDeps, writes hostwrite.Config, serialNotify chan<- struct{}) {
+func fastConvergeLoop(reg *registry.Registry, svc *activator.Service, sd *scheduleDeps, writes hostwrite.Config, serialNotify chan<- struct{}, configure ...func(*provision.ReconcileConfig)) {
-@@ -2107 +2116 @@
+@@ -2124 +2133 @@
- if err := reconcileSerialized(ctx, store, reg, provisionTickConfig(os.Getenv)); err != nil {
+ if err := reconcileSerialized(ctx, store, reg, configuredClaimProvision(provisionTickConfig(os.Getenv), configure)); err != nil {
-@@ -2662 +2671 @@
+@@ -2674 +2683 @@
-func instancesLoop(reg *registry.Registry) {
+func instancesLoop(reg *registry.Registry, configure ...func(*provision.ReconcileConfig)) {
-@@ -2669 +2678 @@
+@@ -2681 +2690 @@
- cfg := provisionTickConfig(os.Getenv)
+ cfg := configuredClaimProvision(provisionTickConfig(os.Getenv), configure)
-@@ -3596 +3605 @@
+@@ -3609 +3618 @@
- claims, err := newClaimWiring(context.Background(), os.Getenv, buildClaims)
+ claimCtx, err := claimEntryContext(context.Background(), os.Getenv, openDeployStore)
-@@ -3606,0 +3616,21 @@
+@@ -3619,0 +3629,21 @@
+ reg, err := openServiceRegistry(*regPath)
+ if err != nil {
+ if hint := registryOpenHint(*regPath, err); hint != "" {
@@ -6177,7 +8503,7 @@ Timer delivery registers after any cold wake and immediately before backend HTTP
+ }
+ }()
+
-@@ -3634,7 +3663,0 @@
+@@ -3647,7 +3676,0 @@
- reg, err := openServiceRegistry(*regPath)
- if err != nil {
- if hint := registryOpenHint(*regPath, err); hint != "" {
@@ -6307,6 +8633,10 @@ type ClaimAssembly struct {
NewID func() string
Provision func(*provision.ReconcileConfig)
Close func(context.Context) error
+ // Test seams for the local stop identity and the root transport. Nil uses
+ // the verified identity and the instance-control socket.
+ stopLocal func(context.Context) (claimStopLocal, error)
+ collectStop func(context.Context, StopRequest, StopDrainer) (StopResult, error)
}
func (a ClaimAssembly) Build(ctx context.Context) (ClaimProviders, error) {
@@ -6752,28 +9082,8 @@ func OpenProductionClaimAssembly(ctx context.Context, store release.Store, r Cla
digest := sha256.Sum256(raw)
return hex.EncodeToString(digest[:]), nil
}
- wake.Retries.StopAttempt = func(ctx context.Context, o hostclaim.Observed, attempt hostclaim.WakeRetryAttempt) error {
- pending := wake.get(o.Host)
- if pending != nil {
- pending.mu.Lock()
- launch := pending.launch
- pending.mu.Unlock()
- if launch != nil && launch.Target.Session == attempt.Session {
- journal, err := launches.Read(ctx, launch.Target)
- if err == nil {
- for _, r := range journal.Reservations {
- if r.ID == attempt.ID+":main" && r.InvocationID == attempt.InvocationID && r.Inputs == attempt.LaunchInputs && r.State == "settled" {
- return nil
- }
- }
- }
- }
- }
- if wake.Stop == nil {
- return hostwrite.ErrUnavailable
- }
- return wake.Stop(ctx, o)
- }
+ // ClaimAssembly.Build binds Retries.StopAttempt to the attempt's persisted
+ // boot and controller tuple; a wake runtime cannot be wrapped without it.
reconcile := &hostbox.ReconcileAdapter{Registry: r.Registry, Inventory: inventory, Retirement: hostbox.TypedRetirement{JournalDir: filepath.Join(local, "retire")}}
stop := &ClaimStopCoordinator{Guard: guard, Claims: hostguard.StoreReader{Store: store}, Inventory: inventory, Evidence: store, NewID: newID, Now: time.Now}
assembly := &ClaimAssembly{SharedPaths: claimGlobalSharedPaths(r), Mode: r.Mode, Producers: producers, Launches: launches, ManagedRoots: roots, Reconcile: reconcile, RootJournal: launches, Controllers: owners.Controllers, Claims: store, Guard: guard, Identity: owners.Identity, Local: owners.Local, Policy: hostbox.Policy{Inputs: owners.InputPolicy, Moves: owners.MovesPolicy, Held: hostbox.RegistryHoldPolicy(r.Registry)}, Data: owners.Data, Wake: wake, Stop: stop, Evidence: owners.Evidence, LeaseSet: owners.LeaseSet, View: owners.View, Authorize: owners.Authorize, AdmissionEvidence: owners.AdmissionEvidence, PublicationEvidence: owners.PublicationEvidence, NewID: newID}
@@ -7197,12 +9507,17 @@ func RetireClaimOverSocket(ctx context.Context, socket string, target hostguard.
package host
import (
+ "context"
"crypto/sha256"
"encoding/hex"
"encoding/json"
+ "strconv"
+
"github.com/carlosframework/platform/internal/cgroup"
"github.com/carlosframework/platform/internal/hostbox"
+ "github.com/carlosframework/platform/internal/hostclaim"
"github.com/carlosframework/platform/internal/hostguard"
+ "github.com/carlosframework/platform/internal/hostwrite"
)
type ClaimRoleAssembly struct {
@@ -7224,6 +9539,19 @@ func claimContainmentName(target hostguard.FenceTarget) string {
sum := sha256.Sum256(raw)
return "claim-" + hex.EncodeToString(sum[:24]) + ".local"
}
+
+// launchTarget binds the observed session to this controller's verified tuple.
+func (a *ClaimRoleAssembly) launchTarget(ctx context.Context, o hostclaim.Observed) (hostguard.FenceTarget, error) {
+ if a == nil || a.BootID == "" || a.ControllerID == "" || o.Claim == nil || o.Claim.Holder == nil {
+ return hostguard.FenceTarget{}, hostwrite.ErrUnavailable
+ }
+ actor, err := a.Identity.Verify(ctx)
+ if err != nil {
+ return hostguard.FenceTarget{}, err
+ }
+ h := o.Claim.Holder
+ return hostguard.FenceTarget{Region: actor.Region, Host: o.Host, Incarnation: h.Incarnation, Stamp: h.Stamp, Session: h.Session, Op: o.Claim.Op.ID, Phase: o.Claim.Op.Phase, Box: actor.Box, Enrollment: actor.Enrollment, BootID: a.BootID, ControllerID: a.ControllerID, ClaimETag: o.ETag, DirectoryRevision: strconv.FormatUint(actor.Revision, 10)}, nil
+}
```
#### New file: `internal/host/claimroleassembly_linux.go`
@@ -7252,21 +9580,84 @@ import (
"github.com/carlosframework/platform/internal/cgroup"
"github.com/carlosframework/platform/internal/hostbox"
"github.com/carlosframework/platform/internal/hostclaim"
- "github.com/carlosframework/platform/internal/hostguard"
"github.com/carlosframework/platform/internal/hostwrite"
"golang.org/x/sys/unix"
)
-func (a *ClaimRoleAssembly) Launch(ctx context.Context, o hostclaim.Observed, work hostwrite.Work, in ClaimLaunchInputs) (*activator.RoleLaunch, error) {
- if a == nil || a.Manager == nil || a.Journal == nil || a.Inventory == nil || work == nil || a.BootID == "" || a.ControllerID == "" || o.Claim == nil || o.Claim.Holder == nil || !filepath.IsAbs(in.DBPath) {
+// Prelaunch prepares the session's empty parent group and journals it with
+// this controller's PID and process start before restore or materialization.
+// A failure or crash after it returns leaves the exact record root needs.
+func (a *ClaimRoleAssembly) Prelaunch(ctx context.Context, o hostclaim.Observed, work hostwrite.Work, in ClaimLaunchInputs) (*activator.RoleLaunch, error) {
+ if a == nil || a.Manager == nil || a.Journal == nil || a.Inventory == nil || work == nil || o.Claim == nil || o.Claim.Holder == nil {
return nil, hostwrite.ErrUnavailable
}
- actor, err := a.Identity.Verify(ctx)
+ target, err := a.launchTarget(ctx, o)
if err != nil {
return nil, err
}
- h := o.Claim.Holder
- target := hostguard.FenceTarget{Region: actor.Region, Host: o.Host, Incarnation: h.Incarnation, Stamp: h.Stamp, Session: h.Session, Op: o.Claim.Op.ID, Phase: o.Claim.Op.Phase, Box: actor.Box, Enrollment: actor.Enrollment, BootID: a.BootID, ControllerID: a.ControllerID, ClaimETag: o.ETag, DirectoryRevision: strconv.FormatUint(actor.Revision, 10)}
+ // The installation inventory is what root verifies a stop against. Without
+ // it the record could never be settled, so refuse before any effect.
+ inventory, err := a.Inventory.ReadExactStopInventory(ctx, target)
+ if err != nil {
+ return nil, err
+ }
+ if !inventory.Complete || inventory.Host != target.Host || inventory.Incarnation != target.Incarnation || inventory.InstallationStamp != target.Stamp || inventory.Box != target.Box || inventory.Enrollment != target.Enrollment || inventory.DirectoryRevision != target.DirectoryRevision {
+ return nil, hostbox.ErrTypedInventory
+ }
+ start, err := claimConstructionProcessStart()
+ if err != nil {
+ return nil, err
+ }
+ parentName := claimContainmentName(target)
+ parentPath, err := filepath.Rel("/sys/fs/cgroup", filepath.Join(a.Manager.Root(), cgroup.ExecName(parentName)))
+ if err != nil || !filepath.IsLocal(parentPath) {
+ return nil, hostwrite.ErrUnavailable
+ }
+ var parent *cgroup.Prepared
+ err = work.With(ctx, in.Identity, func() error {
+ var e error
+ if parent, e = a.Manager.PrepareStrict(parentName, in.Limit); e != nil {
+ return e
+ }
+ id, e := parent.Identity()
+ if e != nil {
+ return e
+ }
+ return a.Journal.Prelaunch(ctx, hostbox.LaunchInventory{ControllerPID: os.Getpid(), ProcessStart: start, Target: target, Groups: []hostbox.LaunchGroup{{RelativePath: parentPath, Device: id.Device, Inode: id.Inode}}, NoState: true})
+ })
+ if err != nil {
+ if parent != nil {
+ err = errors.Join(err, parent.Close())
+ }
+ return nil, err
+ }
+ return &activator.RoleLaunch{Parent: parent, ParentPath: parentPath, CgroupRoot: "/sys/fs/cgroup", Target: target, Work: work, Identity: in.Identity, Journal: a.Journal}, nil
+}
+
+// Launch completes the pre-launch record after restore and materialization.
+// It reuses that exact parent group; it never creates a second record.
+func (a *ClaimRoleAssembly) Launch(ctx context.Context, o hostclaim.Observed, work hostwrite.Work, in ClaimLaunchInputs, pre *activator.RoleLaunch) (*activator.RoleLaunch, error) {
+ if a == nil || a.Manager == nil || a.Journal == nil || a.Inventory == nil || work == nil || pre == nil || pre.Parent == nil || pre.Journal != a.Journal || o.Claim == nil || o.Claim.Holder == nil || !filepath.IsAbs(in.DBPath) {
+ return nil, hostwrite.ErrUnavailable
+ }
+ target, err := a.launchTarget(ctx, o)
+ if err != nil {
+ return nil, err
+ }
+ if !hostbox.SameLaunchOwner(pre.Target, target) {
+ return nil, hostbox.ErrChanged
+ }
+ parentID, err := pre.Parent.Identity()
+ if err != nil {
+ return nil, err
+ }
+ recorded, err := a.Journal.Read(ctx, target)
+ if err != nil {
+ return nil, err
+ }
+ if recorded.ControllerPID != os.Getpid() || len(recorded.Groups) == 0 || recorded.Groups[0] != (hostbox.LaunchGroup{RelativePath: pre.ParentPath, Device: parentID.Device, Inode: parentID.Inode}) {
+ return nil, hostbox.ErrChanged
+ }
inventory, err := a.Inventory.ReadExactStopInventory(ctx, target)
if err != nil {
return nil, err
@@ -7302,39 +9693,9 @@ func (a *ClaimRoleAssembly) Launch(ctx context.Context, o hostclaim.Observed, wo
if !owned {
return nil, hostwrite.ErrUnavailable
}
- parentName := claimContainmentName(target)
- var parent *cgroup.Prepared
- if err = work.With(ctx, in.Identity, func() error { var e error; parent, e = a.Manager.PrepareStrict(parentName, in.Limit); return e }); err != nil {
- return nil, err
- }
- parentID, err := parent.Identity()
- if err != nil {
- _ = parent.Close()
- return nil, err
- }
- raw, err := os.ReadFile("/proc/self/stat")
- if err != nil {
- _ = parent.Close()
- return nil, err
- }
- end := strings.LastIndexByte(string(raw), ')')
- if end < 0 {
- _ = parent.Close()
- return nil, hostwrite.ErrUnavailable
- }
- fields := strings.Fields(string(raw)[end+1:])
- if len(fields) < 20 {
- _ = parent.Close()
- return nil, hostwrite.ErrUnavailable
- }
- parentPath, err := filepath.Rel("/sys/fs/cgroup", filepath.Join(a.Manager.Root(), cgroup.ExecName(parentName)))
- if err != nil || !filepath.IsLocal(parentPath) {
- _ = parent.Close()
- return nil, hostwrite.ErrUnavailable
- }
- launch := &activator.RoleLaunch{Parent: parent, ParentPath: parentPath, CgroupRoot: "/sys/fs/cgroup", Target: target, Work: work, Identity: in.Identity, Journal: a.Journal}
+ launch := *pre
+ launch.Target, launch.Work, launch.Identity = target, work, in.Identity
var mu sync.Mutex
- initialized := false
launch.Initialize = func(ctx context.Context) error {
mu.Lock()
defer mu.Unlock()
@@ -7371,13 +9732,7 @@ func (a *ClaimRoleAssembly) Launch(ctx context.Context, o hostclaim.Observed, wo
if len(files) == 0 {
return hostwrite.ErrUnavailable
}
- if !initialized {
- if err := a.Journal.Create(ctx, hostbox.LaunchInventory{ControllerPID: os.Getpid(), ProcessStart: fields[19], Target: target, Groups: []hostbox.LaunchGroup{{RelativePath: parentPath, Device: parentID.Device, Inode: parentID.Inode}}, Files: files, ProducerIDs: []string{}}); err != nil {
- return err
- }
- initialized = true
- return nil
- }
+ // The pre-launch record already exists; files join it before Start.
for _, f := range files {
if err := a.Journal.RecordFile(ctx, target, f); err != nil {
return err
@@ -7388,7 +9743,7 @@ func (a *ClaimRoleAssembly) Launch(ctx context.Context, o hostclaim.Observed, wo
launch.Executable = func(ctx context.Context, cmd *exec.Cmd) (*os.File, error) {
return verifiedClaimExecutable(ctx, cmd, in.Executable, in.ExecutableSHA256)
}
- return launch, nil
+ return &launch, nil
}
func verifiedClaimExecutable(ctx context.Context, cmd *exec.Cmd, path, digest string) (*os.File, error) {
@@ -7573,7 +9928,7 @@ func (a *ClaimRoleAssembly) PrepareState(ctx context.Context, o hostclaim.Observ
}
if index < 0 {
sum := sha256.Sum256([]byte(owner + "\x00state-dir\x00" + relative))
- inventory.Entries = append(inventory.Entries, hostbox.StopInventoryEntry{Kind: "path", ArtifactID: hex.EncodeToString(sum[:]), ManagedRoot: "data", RelativePath: relative, Ownership: owner, Materialization: "intended"})
+ inventory.Entries = append(inventory.Entries, hostbox.StopInventoryEntry{Kind: "path", ArtifactID: hex.EncodeToString(sum[:]), ManagedRoot: "data", RelativePath: relative, Ownership: owner, Type: unix.S_IFDIR, Materialization: "intended"})
index = len(inventory.Entries) - 1
if err = a.Inventory.WriteStopInventory(ctx, inventory); err != nil {
return err
@@ -7613,7 +9968,11 @@ import (
"net"
)
-func (*ClaimRoleAssembly) Launch(context.Context, hostclaim.Observed, hostwrite.Work, ClaimLaunchInputs) (*activator.RoleLaunch, error) {
+func (*ClaimRoleAssembly) Prelaunch(context.Context, hostclaim.Observed, hostwrite.Work, ClaimLaunchInputs) (*activator.RoleLaunch, error) {
+ return nil, hostwrite.ErrUnavailable
+}
+
+func (*ClaimRoleAssembly) Launch(context.Context, hostclaim.Observed, hostwrite.Work, ClaimLaunchInputs, *activator.RoleLaunch) (*activator.RoleLaunch, error) {
return nil, hostwrite.ErrUnavailable
}
@@ -7775,27 +10134,43 @@ func claimStateDatabaseOwner(string) error { return hostwrite.ErrUnavailable
-}
+type StopInventoryEntry = hostbox.StopInventoryEntry
+type StopInventory = hostbox.StopInventory
-@@ -62,0 +47,2 @@
-+ Region string
-+ Cleanup bool
-@@ -76,0 +63 @@
+@@ -62,0 +47,5 @@
++ Region string
++ Cleanup bool
++ // Recover names a launch tuple journaled by an earlier controller process
++ // or boot. The live controller asks; root proves that controller dead.
++ Recover bool
+@@ -76,0 +66 @@
+ ProducerIDs []string
-@@ -87,0 +75 @@
+@@ -87,0 +78 @@
+ ObservedBootID string
-@@ -98,0 +87,3 @@
+@@ -98,0 +90,3 @@
+ Region string `json:"region,omitempty"`
+ ClaimETag string `json:"claim_etag,omitempty"`
+ ObservedBootID string `json:"observed_boot_id,omitempty"`
-@@ -124,0 +116 @@
+@@ -124,0 +119 @@
+ Target hostguard.FenceTarget
-@@ -158 +150 @@
+@@ -158 +153 @@
- !stopComponent(r.Host) || !stopComponent(r.OperationID) || !stopComponent(r.Phase) || !stopComponent(r.ObligationID) || !stopComponent(r.Box) ||
+ !stopComponent(r.Host) || !stopOperationTuple(r.OperationID, r.Phase, r.ObligationID) || !stopComponent(r.Box) ||
-@@ -166,0 +159,3 @@
+@@ -167 +162,4 @@
+- if ctx == nil || c.validateRequest(req) != nil {
+ if req.WriterSession == 0 {
+ return StopResult{}, &ClaimDependencyError{Contracts: []string{"Plan 5 positive complete no-process witness for an installation with writer_session=0"}}
+ }
-@@ -177,11 +172,32 @@
++ if ctx == nil || c.validateRequest(req) != nil || (req.Cleanup && req.Recover) {
+@@ -169,0 +168,7 @@
++ var recovery stopRecoveryKernel
++ if req.Recover {
++ var ok bool
++ if recovery, ok = c.Kernel.(stopRecoveryKernel); !ok {
++ return StopResult{}, ErrClaimStopUnavailable
++ }
++ }
+@@ -174 +179 @@
+- if peer.UID != c.CarlosUID || peer.PID <= 0 || peer.ProcessStart == "" || peer.BootID != req.BootID || peer.ControllerID != req.ControllerID {
++ if peer.UID != c.CarlosUID || peer.PID <= 0 || peer.ProcessStart == "" || peer.BootID == "" || peer.ControllerID == "" {
+@@ -177,8 +182,2 @@
- observed, err := c.Claims.Read(ctx, req.Host)
- if err != nil {
- return StopResult{}, err
@@ -7804,8 +10179,9 @@ func claimStateDatabaseOwner(string) error { return hostwrite.ErrUnavailable
- observed.Claim.Host != req.Host || observed.Claim.Holder.Incarnation != req.Incarnation ||
- observed.Claim.Holder.Stamp != req.InstallationStamp || observed.Claim.Holder.Session != req.WriterSession ||
- observed.Claim.Op.ID != req.OperationID || observed.Claim.Op.Phase != req.Phase || observed.Claim.Op.Exec != req.Box {
-- return StopResult{}, ErrClaimStopIdentity
-- }
++ // Only recovery may name another tuple, and only for this box's session.
++ if !req.Recover && (peer.BootID != req.BootID || peer.ControllerID != req.ControllerID) {
+@@ -187 +186,32 @@
- inv, err := c.Inventory.ReadStopInventory(ctx, req.Host)
+ var observed hostclaim.Observed
+ var cleanupTarget *hostguard.FenceTarget
@@ -7839,14 +10215,14 @@ func claimStateDatabaseOwner(string) error { return hostwrite.ErrUnavailable
+ } else {
+ inv, err = c.Inventory.ReadStopInventory(ctx, req.Host)
+ }
-@@ -199 +215,5 @@
+@@ -199 +229,5 @@
- target := hostguard.FenceTarget{FenceID: fenceID, Host: req.Host, Incarnation: req.Incarnation, Stamp: req.InstallationStamp,
+ gen := uint64(0)
+ if observed.Claim != nil {
+ gen = observed.Claim.Gen
+ }
+ target := hostguard.FenceTarget{Region: req.Region, ClaimGen: gen, ObligationID: req.ObligationID, FenceID: fenceID, Host: req.Host, Incarnation: req.Incarnation, Stamp: req.InstallationStamp,
-@@ -202 +222,6 @@
+@@ -202 +236,6 @@
- target, err = c.Guard.BeginFence(ctx, target)
+ if cleanupTarget != nil {
+ target = *cleanupTarget
@@ -7854,23 +10230,60 @@ func claimStateDatabaseOwner(string) error { return hostwrite.ErrUnavailable
+ } else {
+ target, err = c.Guard.BeginFence(ctx, target)
+ }
-@@ -212,2 +237,4 @@
+@@ -212,2 +251,4 @@
- if err := c.Publish(ctx, result); err != nil {
- return result, err
+ if !req.Cleanup {
+ if err := c.Publish(ctx, result); err != nil {
+ return result, err
+ }
-@@ -234 +261 @@
+@@ -219,10 +260,26 @@
+- ack, err := c.Drainer.DrainClaim(ctx, req, fenceID)
+- if err != nil {
+- return StopResult{}, err
+- }
+- if ack.FenceID != fenceID || ack.ControllerID != req.ControllerID || ack.ProcessStart != peer.ProcessStart || !stopDigest(ack.Digest) {
+- return StopResult{}, ErrClaimStopIdentity
+- }
+- kernel, err := c.Kernel.VerifyStopped(ctx, target, inv, ack)
+- if err != nil {
+- return StopResult{}, err
++ var ack DrainAcknowledgement
++ var kernel StopKernelEvidence
++ if req.Recover {
++ // Root reads the tuple's own launch record and proves its controller
++ // dead, or the boot gone, before it kills, verifies and settles. A
++ // missing record or a live controller refuses; the fence stays pending.
++ kernel, ack, err = recovery.RecoverStopped(ctx, target, inv)
++ if err != nil {
++ return StopResult{}, err
++ }
++ if ack.FenceID != fenceID || ack.ControllerID != req.ControllerID || ack.ProcessStart == "" || !stopDigest(ack.Digest) ||
++ (req.BootID == peer.BootID && req.ControllerID == peer.ControllerID && ack.ProcessStart == peer.ProcessStart) {
++ return StopResult{}, ErrClaimStopIdentity
++ }
++ } else {
++ ack, err = c.Drainer.DrainClaim(ctx, req, fenceID)
++ if err != nil {
++ return StopResult{}, err
++ }
++ if ack.FenceID != fenceID || ack.ControllerID != req.ControllerID || ack.ProcessStart != peer.ProcessStart || !stopDigest(ack.Digest) {
++ return StopResult{}, ErrClaimStopIdentity
++ }
++ kernel, err = c.Kernel.VerifyStopped(ctx, target, inv, ack)
++ if err != nil {
++ return StopResult{}, err
++ }
+@@ -234 +291 @@
- receipt := StopReceipt{V: 1, ID: receiptID, Host: req.Host, Incarnation: req.Incarnation,
+ receipt := StopReceipt{Region: req.Region, ClaimETag: target.ClaimETag, ObservedBootID: kernel.ObservedBootID, V: 1, ID: receiptID, Host: req.Host, Incarnation: req.Incarnation,
-@@ -237 +264 @@
+@@ -237 +294 @@
- BootID: req.BootID, ControllerID: req.ControllerID, ClaimGen: observed.Claim.Gen,
+ BootID: req.BootID, ControllerID: req.ControllerID, ClaimGen: target.ClaimGen,
-@@ -255 +282 @@
+@@ -255 +312 @@
- key := fmt.Sprintf("control/hosts/%s/evidence/%s/%s/%s.json", req.Host, req.OperationID, req.Box, receiptID)
+ key := stopEvidenceKey(req.Host, req.OperationID, req.Box, receiptID)
-@@ -257,3 +284,5 @@
+@@ -257,3 +314,5 @@
- result := StopResult{Key: key, Digest: hex.EncodeToString(sum[:]), Bytes: append([]byte(nil), body...), EvidenceAt: evidenceAt}
- if err := c.Publish(ctx, result); err != nil {
- return result, err
@@ -7879,19 +10292,19 @@ func claimStateDatabaseOwner(string) error { return hostwrite.ErrUnavailable
+ if err := c.Publish(ctx, result); err != nil {
+ return result, err
+ }
-@@ -269 +298 @@
+@@ -269 +328 @@
- if receipt.V != 1 || receipt.ID == "" || receipt.Host != req.Host || receipt.Incarnation != req.Incarnation ||
+ if receipt.Region != req.Region || receipt.V != 1 || receipt.ID == "" || receipt.Host != req.Host || receipt.Incarnation != req.Incarnation ||
-@@ -277 +306 @@
+@@ -277 +336 @@
- key := fmt.Sprintf("control/hosts/%s/evidence/%s/%s/%s.json", req.Host, req.OperationID, req.Box, receipt.ID)
+ key := stopEvidenceKey(req.Host, req.OperationID, req.Box, receipt.ID)
-@@ -279 +308 @@
+@@ -279 +338 @@
- return StopResult{Key: key, Digest: hex.EncodeToString(sum[:]), Bytes: append([]byte(nil), durable.Bytes...), EvidenceAt: durable.EvidenceAt}, nil
+ return StopResult{Target: durable.Target, Key: key, Digest: hex.EncodeToString(sum[:]), Bytes: append([]byte(nil), durable.Bytes...), EvidenceAt: durable.EvidenceAt}, nil
-@@ -296 +325 @@
+@@ -296 +355 @@
- wantKey := fmt.Sprintf("control/hosts/%s/evidence/%s/%s/%s.json", receipt.Host, receipt.OperationID, receipt.Box, receipt.ID)
+ wantKey := stopEvidenceKey(receipt.Host, receipt.OperationID, receipt.Box, receipt.ID)
-@@ -315,0 +345,13 @@
+@@ -315,0 +375,13 @@
+
+func stopOperationTuple(op, phase, obligation string) bool {
+ if op == "" || phase == "" || obligation == "" {
@@ -7955,13 +10368,8 @@ func (k LinuxStopKernel) VerifyStopped(ctx context.Context, t hostguard.FenceTar
if !inv.Complete || j.V != 1 || !j.Settled || !sameStopSession(j.Target, t) || len(j.Cgroups) == 0 || ack.FenceID != t.FenceID || ack.ControllerID != t.ControllerID {
return StopKernelEvidence{}, ErrClaimStopInventory
}
- ids := append([]string(nil), j.ProducerIDs...)
- sort.Strings(ids)
- if !reflect.DeepEqual(ids, ack.ProducerIDs) || ack.Digest != digestStop(struct {
- FenceID, ControllerID, ProcessStart string
- ProducerIDs []string
- }{ack.FenceID, ack.ControllerID, ack.ProcessStart, ack.ProducerIDs}) {
- return StopKernelEvidence{}, ErrClaimStopIdentity
+ if err := validateClaimDrainProducers(j.ProducerIDs, ack); err != nil {
+ return StopKernelEvidence{}, err
}
groups, err := scanStopCgroups(ctx, k.CgroupRoot, j.Cgroups)
if err != nil {
@@ -8810,6 +11218,7 @@ import (
"io"
"net"
"net/http"
+ "slices"
"sort"
"time"
@@ -9001,6 +11410,19 @@ func validateTransportStop(req StopRequest, result StopResult) (StopResult, erro
}
return result, nil
}
+
+// validateClaimDrainProducers is shared by the kernel verifier and fixtures.
+func validateClaimDrainProducers(producers []string, ack DrainAcknowledgement) error {
+ ids := append([]string(nil), producers...)
+ sort.Strings(ids)
+ if !slices.Equal(ids, ack.ProducerIDs) || ack.Digest != digestStop(struct {
+ FenceID, ControllerID, ProcessStart string
+ ProducerIDs []string
+ }{ack.FenceID, ack.ControllerID, ack.ProcessStart, ack.ProducerIDs}) {
+ return ErrClaimStopIdentity
+ }
+ return nil
+}
```
#### New file: `internal/host/claimtimer.go`
@@ -9123,22 +11545,27 @@ type ClaimWakeAssembly struct {
Retries activator.ServingRetries
Stop func(context.Context, hostclaim.Observed) error
RetireArtifacts func(context.Context, hostclaim.Observed, string) error
- Launch func(context.Context, hostclaim.Observed, hostwrite.Work, ClaimLaunchInputs) (*activator.RoleLaunch, error)
- Ready func(context.Context, *exec.Cmd, registry.Route, ClaimLaunchInputs) error
- Retain func(hostclaim.Observed, activator.Proc, activator.Rep, error)
- ObserveLoss func(hostclaim.Observed, error)
- mu sync.Mutex
- pending map[string]*claimWakeAttempt
+ // Prelaunch journals the admitted session's recoverable record before
+ // restore or materialization; Launch later completes that same record.
+ PrepareState func(context.Context, hostclaim.Observed, hostwrite.Work, ClaimLaunchInputs) error
+ Prelaunch func(context.Context, hostclaim.Observed, hostwrite.Work, ClaimLaunchInputs) (*activator.RoleLaunch, error)
+ Launch func(context.Context, hostclaim.Observed, hostwrite.Work, ClaimLaunchInputs, *activator.RoleLaunch) (*activator.RoleLaunch, error)
+ Ready func(context.Context, *exec.Cmd, registry.Route, ClaimLaunchInputs) error
+ Retain func(hostclaim.Observed, activator.Proc, activator.Rep, error)
+ ObserveLoss func(hostclaim.Observed, error)
+ mu sync.Mutex
+ pending map[string]*claimWakeAttempt
}
type claimWakeAttempt struct {
releaseMu sync.Mutex
+ prelaunch *activator.RoleLaunch
launch *activator.RoleLaunch
repDone chan struct{}
repErr error
released bool
observed hostclaim.Observed
inputs ClaimLaunchInputs
- held *lease.Held
+ held claimHeldLease
rep *lease.Replicator
ctx context.Context
cancel context.CancelFunc
@@ -9150,11 +11577,19 @@ func (w *ClaimWakeAssembly) Build(ctx context.Context, source hostbox.WorkSource
if w == nil || w.LeaseRegion == "" || w.AfterAdmission == nil || w.Inputs == nil || w.Manager == nil || w.Stop == nil || w.RetireArtifacts == nil || w.Retain == nil || w.ObserveLoss == nil || w.Producers == nil {
return nil, hostwrite.ErrUnavailable
}
- if w.Launch == nil {
+ if w.Launch == nil || w.Prelaunch == nil || w.PrepareState == nil {
if w.Roles == nil {
return nil, hostwrite.ErrUnavailable
}
- w.Launch = w.Roles.Launch
+ if w.PrepareState == nil {
+ w.PrepareState = w.Roles.PrepareState
+ }
+ if w.Launch == nil {
+ w.Launch = w.Roles.Launch
+ }
+ if w.Prelaunch == nil {
+ w.Prelaunch = w.Roles.Prelaunch
+ }
}
if w.Retries.ExactHealth == nil {
w.Retries.ExactHealth = w.exactHealth
@@ -9269,6 +11704,11 @@ func (w *ClaimWakeAssembly) Build(ctx context.Context, source hostbox.WorkSource
if err := source.Guard.Activate(call, rt.Host, writer.Box, nil); err != nil {
return admitted, err
}
+ // The launch record precedes the retry bind, restore and materialization.
+ // Any later failure or crash leaves the exact tuple root stops and settles.
+ if err := w.persistPrelaunch(call, a, source, admitted, in); err != nil {
+ return admitted, err
+ }
return admitted, nil
}
runtime.Prepare = func(call context.Context, rt registry.Route, o hostclaim.Observed, p activator.StartPolicy) (activator.ClaimStart, error) {
@@ -9282,7 +11722,12 @@ func (w *ClaimWakeAssembly) Build(ctx context.Context, source hostbox.WorkSource
}
a.mu.Lock()
a.observed = o
+ pre := a.prelaunch
a.mu.Unlock()
+ // No restore or materialization runs without the pre-launch record.
+ if pre == nil || pre.Target.Host != o.Host || pre.Target.Incarnation != o.Claim.Holder.Incarnation || pre.Target.Stamp != o.Claim.Holder.Stamp || pre.Target.Session != o.Claim.Holder.Session {
+ return activator.ClaimStart{}, hostbox.ErrChanged
+ }
selected, err := source.SelectAt(call, rt.Host, "spawn", hostguard.Captured{Incarnation: o.Claim.Holder.Incarnation, Stamp: o.Claim.Holder.Stamp})
if err != nil {
return activator.ClaimStart{}, err
@@ -9292,10 +11737,7 @@ func (w *ClaimWakeAssembly) Build(ctx context.Context, source hostbox.WorkSource
restore := hostbox.ColdRestore{Reader: writer.Data, Bucket: writer.Bucket, Generation: o.Claim.Holder.Generation, Work: selected, Identity: in.Identity, RetireArtifacts: func(ctx context.Context, g string) error { return w.RetireArtifacts(ctx, o, g) }}
s := activator.ClaimStart{Live: a.ctx, Work: selected, Identity: in.Identity, Manager: w.Manager, Host: rt.Host, Limit: in.Limit, Version: in.Version}
s.Restore = func(call context.Context) error {
- if w.Roles == nil {
- return hostwrite.ErrUnavailable
- }
- if err := w.Roles.PrepareState(call, o, selected, in); err != nil {
+ if err := w.PrepareState(call, o, selected, in); err != nil {
return err
}
if err := a.leaseErr(); err != nil {
@@ -9314,7 +11756,7 @@ func (w *ClaimWakeAssembly) Build(ctx context.Context, source hostbox.WorkSource
}
s.Ready = func(ctx context.Context, cmd *exec.Cmd) error { return w.Ready(ctx, cmd, rt, in) }
s.PrepareLaunch = func(ctx context.Context) (*activator.RoleLaunch, error) {
- launch, err := w.Launch(ctx, o, selected, in)
+ launch, err := w.Launch(ctx, o, selected, in, pre)
if err != nil {
return nil, err
}
@@ -9403,6 +11845,48 @@ func (w *ClaimWakeAssembly) Build(ctx context.Context, source hostbox.WorkSource
}
return w.Retries.Wrap(runtime)
}
+
+// claimHeldLease is the part of lease.Held one admitted attempt uses.
+type claimHeldLease interface {
+ Err() error
+ Lost() <-chan struct{}
+ Release(context.Context) error
+ BindGeneration(string, func(context.Context) error) error
+ Replicate(context.Context, string, time.Duration) (*lease.Replicator, error)
+}
+
+// persistPrelaunch records the admitted session under the captured guard. It
+// runs once per attempt and binds exactly the admitted writer session.
+func (w *ClaimWakeAssembly) persistPrelaunch(ctx context.Context, a *claimWakeAttempt, source hostbox.WorkSource, admitted hostclaim.Observed, in ClaimLaunchInputs) error {
+ if w.Prelaunch == nil || admitted.Claim == nil || admitted.Claim.Holder == nil {
+ return hostwrite.ErrUnavailable
+ }
+ a.mu.Lock()
+ existing := a.prelaunch
+ a.mu.Unlock()
+ if existing != nil {
+ return hostbox.ErrChanged
+ }
+ h := admitted.Claim.Holder
+ selected, err := source.SelectAt(ctx, admitted.Host, "spawn", hostguard.Captured{Incarnation: h.Incarnation, Stamp: h.Stamp})
+ if err != nil {
+ return err
+ }
+ pre, err := w.Prelaunch(ctx, admitted, claimLeaseWork{Work: selected, Alive: a.ctx, Check: a.leaseErr}, in)
+ if err != nil {
+ return err
+ }
+ if pre == nil || pre.Journal == nil || pre.Target.Host != admitted.Host || pre.Target.Incarnation != h.Incarnation || pre.Target.Stamp != h.Stamp || pre.Target.Session != h.Session {
+ if pre != nil && pre.Parent != nil {
+ return errors.Join(hostbox.ErrChanged, pre.Parent.Close())
+ }
+ return hostbox.ErrChanged
+ }
+ a.mu.Lock()
+ a.prelaunch = pre
+ a.mu.Unlock()
+ return nil
+}
func (w *ClaimWakeAssembly) get(host string) *claimWakeAttempt {
w.mu.Lock()
defer w.mu.Unlock()
@@ -9440,10 +11924,15 @@ func (w *ClaimWakeAssembly) release(ctx context.Context, host string, a *claimWa
}
if err == nil {
a.mu.Lock()
- launch := a.launch
+ parent := (*cgroup.Prepared)(nil)
+ if a.launch != nil {
+ parent = a.launch.Parent
+ } else if a.prelaunch != nil {
+ parent = a.prelaunch.Parent
+ }
a.mu.Unlock()
- if launch != nil && launch.Parent != nil {
- err = launch.Parent.Close()
+ if parent != nil {
+ err = parent.Close()
}
}
a.mu.Lock()
@@ -9613,6 +12102,8 @@ package host
import (
"context"
+ "database/sql"
+ "errors"
"fmt"
"os"
"strings"
@@ -9621,12 +12112,51 @@ import (
"github.com/carlosframework/platform/internal/hostbox"
"github.com/carlosframework/platform/internal/hostclaim"
"github.com/carlosframework/platform/internal/hostguard"
+ "github.com/carlosframework/platform/internal/hostwrite"
)
+// claimStopLocal is this controller's verified identity when it asks for a stop.
+type claimStopLocal struct {
+ Actor hostbox.Actor
+ BootID string
+ ControllerID string
+ PID int
+ ProcessStart string
+}
+
+// claimLaunchOwner is the tuple that journaled a launch. PID and ProcessStart
+// are zero when no record was found; root then refuses the stop.
+type claimLaunchOwner struct {
+ BootID, ControllerID string
+ DirectoryRevision string
+ PID int
+ ProcessStart string
+}
+
+// local reports whether this very process wrote the record. The same IDs in a
+// restarted process are not local: its producers ended with the old process.
+func (o claimLaunchOwner) local(l claimStopLocal) bool {
+ if o.BootID != l.BootID || o.ControllerID != l.ControllerID {
+ return false
+ }
+ return (o.PID == 0 && o.ProcessStart == "") || (o.PID == l.PID && o.ProcessStart == l.ProcessStart)
+}
+
+func ownerOf(in hostbox.LaunchInventory) claimLaunchOwner {
+ return claimLaunchOwner{BootID: in.Target.BootID, ControllerID: in.Target.ControllerID, DirectoryRevision: in.Target.DirectoryRevision, PID: in.ControllerPID, ProcessStart: in.ProcessStart}
+}
+
+// recoveryStopDrain refuses. A recovered controller's producers ended with it,
+// and this controller's producers are never offered as that controller's drain.
+type recoveryStopDrain struct{}
+
+func (recoveryStopDrain) DrainClaim(context.Context, StopRequest, string) (DrainAcknowledgement, error) {
+ return DrainAcknowledgement{}, ErrClaimStopIdentity
+}
+
func (a ClaimAssembly) bindClaimWakeStop() {
a.Wake.Stop = func(ctx context.Context, captured hostclaim.Observed) error {
- _, err := a.collectCapturedStop(ctx, captured)
- if err != nil {
+ if err := a.stopCapturedSession(ctx, captured); err != nil {
return err
}
pending := a.Wake.get(captured.Host)
@@ -9638,14 +12168,142 @@ func (a ClaimAssembly) bindClaimWakeStop() {
}
return nil
}
+ a.Wake.Retries.StopAttempt = a.stopRetryAttempt
+}
+
+func (a ClaimAssembly) localStopIdentity(ctx context.Context) (claimStopLocal, error) {
+ if a.stopLocal != nil {
+ return a.stopLocal(ctx)
+ }
+ if a.Controllers == nil {
+ return claimStopLocal{}, hostwrite.ErrUnavailable
+ }
+ actor, err := a.Identity.Verify(ctx)
+ if err != nil {
+ return claimStopLocal{}, err
+ }
+ controller, err := a.Controllers.VerifiedController(ctx, os.Getpid())
+ if err != nil {
+ return claimStopLocal{}, err
+ }
+ boot, err := os.ReadFile("/proc/sys/kernel/random/boot_id")
+ if err != nil {
+ return claimStopLocal{}, err
+ }
+ start, err := claimConstructionProcessStart()
+ if err != nil {
+ return claimStopLocal{}, err
+ }
+ local := claimStopLocal{Actor: actor, BootID: strings.TrimSpace(string(boot)), ControllerID: controller, PID: os.Getpid(), ProcessStart: start}
+ if local.BootID == "" || local.ControllerID == "" || local.ProcessStart == "" {
+ return claimStopLocal{}, hostwrite.ErrUnavailable
+ }
+ return local, nil
+}
+
+// stopRetryAttempt stops the exact tuple persisted in a serving retry attempt.
+// After a controller restart or reboot that tuple is not this controller's.
+// It is requested in recovery mode, and root proves its controller dead.
+func (a ClaimAssembly) stopRetryAttempt(ctx context.Context, o hostclaim.Observed, attempt hostclaim.WakeRetryAttempt) error {
+ if o.Claim == nil || o.Claim.Holder == nil || attempt.ID == "" || attempt.BootID == "" || attempt.ControllerID == "" {
+ return hostbox.ErrChanged
+ }
+ if a.Wake == nil || a.Launches == nil {
+ return hostwrite.ErrUnavailable
+ }
+ if attempt.Session == 0 {
+ // An unbound attempt cannot reach Prepare, so it never launched. Its
+ // holder session is stopped from that session's own launch records.
+ if a.Wake.Stop == nil {
+ return hostwrite.ErrUnavailable
+ }
+ return a.Wake.Stop(ctx, o)
+ }
+ h := o.Claim.Holder
+ if h.Session != attempt.Session {
+ return hostbox.ErrChanged
+ }
+ local, err := a.localStopIdentity(ctx)
+ if err != nil {
+ return err
+ }
+ if attempt.Box != local.Actor.Box || attempt.Enrollment != local.Actor.Enrollment {
+ return hostbox.ErrChanged
+ }
+ owner := claimLaunchOwner{BootID: attempt.BootID, ControllerID: attempt.ControllerID}
+ recorded, err := a.Launches.Read(ctx, hostguard.FenceTarget{Host: o.Host, Incarnation: h.Incarnation, Stamp: h.Stamp, Session: attempt.Session, Box: attempt.Box, Enrollment: attempt.Enrollment, BootID: attempt.BootID, ControllerID: attempt.ControllerID})
+ switch {
+ case err == nil:
+ owner = ownerOf(recorded)
+ // This process already stopped and settled the attempt's own launch
+ // before failing it; fencing the session would end its later retries.
+ if owner.local(local) && attemptSettled(recorded, attempt) {
+ return nil
+ }
+ case errors.Is(err, sql.ErrNoRows):
+ // Keep the persisted tuple. Root refuses it without a launch record;
+ // this controller's tuple is never substituted.
+ default:
+ return err
+ }
+ _, err = a.collectCapturedStop(ctx, o, local, owner)
+ return err
+}
+
+func attemptSettled(in hostbox.LaunchInventory, attempt hostclaim.WakeRetryAttempt) bool {
+ for _, r := range in.Reservations {
+ if r.ID == attempt.ID+":main" && r.InvocationID == attempt.InvocationID && r.Inputs == attempt.LaunchInputs && r.State == "settled" {
+ return true
+ }
+ }
+ return false
+}
+
+// stopCapturedSession stops each journaled launch of the captured session. A
+// record left by an earlier controller or boot keeps its original tuple.
+func (a ClaimAssembly) stopCapturedSession(ctx context.Context, captured hostclaim.Observed) error {
+ if captured.Claim == nil || captured.Claim.Holder == nil {
+ return hostbox.ErrChanged
+ }
+ h := captured.Claim.Holder
+ if h.Session == 0 {
+ return &ClaimDependencyError{Contracts: []string{"Plan 5 positive complete no-process witness for an installation with writer_session=0"}}
+ }
+ if a.Launches == nil {
+ return hostwrite.ErrUnavailable
+ }
+ local, err := a.localStopIdentity(ctx)
+ if err != nil {
+ return err
+ }
+ launches, err := a.Launches.SessionLaunches(ctx, captured.Host, h.Incarnation, h.Stamp, h.Session, local.Actor.Box, local.Actor.Enrollment)
+ if err != nil {
+ return err
+ }
+ if len(launches) == 0 {
+ // Nothing is journaled. Request this controller's tuple; root refuses it
+ // unless its own records prove the stop. Absence is never emptiness.
+ _, err = a.collectCapturedStop(ctx, captured, local, claimLaunchOwner{BootID: local.BootID, ControllerID: local.ControllerID})
+ return err
+ }
+ for _, in := range launches {
+ if _, err := a.collectCapturedStop(ctx, captured, local, ownerOf(in)); err != nil {
+ return err
+ }
+ }
+ return nil
}
-func (a ClaimAssembly) collectCapturedStop(ctx context.Context, captured hostclaim.Observed) (StopResult, error) {
+
+func (a ClaimAssembly) collectCapturedStop(ctx context.Context, captured hostclaim.Observed, local claimStopLocal, owner claimLaunchOwner) (StopResult, error) {
if captured.Claim == nil || captured.Claim.Holder == nil {
return StopResult{}, hostbox.ErrChanged
}
if captured.Claim.Holder.Session == 0 {
return StopResult{}, &ClaimDependencyError{Contracts: []string{"Plan 5 positive complete no-process witness for an installation with writer_session=0"}}
}
+ if owner.BootID == "" || owner.ControllerID == "" {
+ return StopResult{}, hostwrite.ErrUnavailable
+ }
observed, err := hostclaim.Read(ctx, a.Claims, captured.Host)
if err != nil {
return StopResult{}, err
@@ -9653,18 +12311,7 @@ func (a ClaimAssembly) collectCapturedStop(ctx context.Context, captured hostcla
if !sameStopWriter(captured, observed) {
return StopResult{}, hostbox.ErrChanged
}
- actor, err := a.Identity.Verify(ctx)
- if err != nil {
- return StopResult{}, err
- }
- controller, err := a.Controllers.VerifiedController(ctx, os.Getpid())
- if err != nil {
- return StopResult{}, err
- }
- boot, err := os.ReadFile("/proc/sys/kernel/random/boot_id")
- if err != nil {
- return StopResult{}, err
- }
+ actor := local.Actor
claim, holder := observed.Claim, observed.Claim.Holder
obligation := ""
if claim.Op.Kind != hostclaim.KindServing {
@@ -9680,14 +12327,27 @@ func (a ClaimAssembly) collectCapturedStop(ctx context.Context, captured hostcla
return StopResult{}, hostbox.ErrChanged
}
}
- req := StopRequest{Region: actor.Region, Host: observed.Host, Incarnation: holder.Incarnation, InstallationStamp: holder.Stamp, WriterSession: holder.Session, OperationID: claim.Op.ID, Phase: claim.Op.Phase, ObligationID: obligation, Box: actor.Box, Enrollment: actor.Enrollment, BootID: strings.TrimSpace(string(boot)), ControllerID: controller, ExpectedClaimETag: observed.ETag, ExpectedDirectoryRevision: fmt.Sprint(actor.Revision)}
+ recovering := !owner.local(local)
+ directoryRevision := owner.DirectoryRevision
+ if directoryRevision == "" {
+ // No record was found. Root still requires exact journal evidence.
+ directoryRevision = fmt.Sprint(actor.Revision)
+ }
+ req := StopRequest{Region: actor.Region, Recover: recovering, Host: observed.Host, Incarnation: holder.Incarnation, InstallationStamp: holder.Stamp, WriterSession: holder.Session, OperationID: claim.Op.ID, Phase: claim.Op.Phase, ObligationID: obligation, Box: actor.Box, Enrollment: actor.Enrollment, BootID: owner.BootID, ControllerID: owner.ControllerID, ExpectedClaimETag: observed.ETag, ExpectedDirectoryRevision: directoryRevision}
target := hostguard.FenceTarget{Region: req.Region, Host: req.Host, Incarnation: req.Incarnation, Stamp: req.InstallationStamp, Session: req.WriterSession, Op: req.OperationID, Phase: req.Phase, ObligationID: req.ObligationID, Box: req.Box}
if !hostguard.StopAuthorityMatches(claim, target) {
return StopResult{}, hostbox.ErrChanged
}
- drain, err := a.Wake.Producers(ctx, observed)
- if err != nil {
- return StopResult{}, err
+ var drain StopDrainer = recoveryStopDrain{}
+ if !recovering {
+ producers, err := a.Wake.Producers(ctx, observed)
+ if err != nil {
+ return StopResult{}, err
+ }
+ drain = producers
+ }
+ if a.collectStop != nil {
+ return a.collectStop(ctx, req, drain)
}
return CollectClaimStop(ctx, controlsock.Path, req, drain)
}
@@ -9792,17 +12452,17 @@ func RunWithClaim(args []string, handler http.Handler) { runWithClaim(args, hand
+++ b/internal/instancectl/server.go
@@ -33,0 +34 @@
+ ClaimStop http.Handler
-@@ -84 +85,3 @@
+@@ -95 +96,3 @@
-func Run(args []string) {
+func Run(args []string) { runWithClaim(args, nil) }
+
+func runWithClaim(args []string, claim http.Handler) {
-@@ -91,0 +95 @@
+@@ -102,0 +106 @@
+ ClaimStop: claim,
-@@ -135 +139 @@
+@@ -151 +155 @@
- srv := &http.Server{Handler: s.Handler(), ReadHeaderTimeout: 5 * time.Second}
+ srv := &http.Server{Handler: s.Handler(), ReadHeaderTimeout: 5 * time.Second, ConnContext: requestConnContext}
-@@ -209,5 +213,10 @@
+@@ -225,5 +229,10 @@
- mux.HandleFunc("POST /start", s.handleStart)
- mux.HandleFunc("POST /seed", s.handleSeed)
- mux.HandleFunc("POST /stop", s.handleStop)
@@ -9818,7 +12478,7 @@ func RunWithClaim(args []string, handler http.Handler) { runWithClaim(args, hand
+ mux.HandleFunc("POST /stop", s.claimAwareLegacy(s.handleStop))
+ mux.HandleFunc("POST /reset", s.claimAwareLegacy(s.handleReset))
+ mux.HandleFunc("POST /sweep", s.claimAwareLegacy(s.handleSweep))
-@@ -667,0 +677,10 @@
+@@ -763,0 +773,10 @@
+
+func (s *Server) claimAwareLegacy(next http.HandlerFunc) http.HandlerFunc {
+ return func(w http.ResponseWriter, r *http.Request) {
@@ -9929,16 +12589,16 @@ Apply all implementation blocks after the tests-first failure. These tasks share
```bash
python3 "$TMPDIR/claim6b-apply.py" docs/superpowers/plans/2026-10-06-host-lifecycle-claim-6b-box-assembly.md implementation
gofmt -l .
-go test ./internal/registry ./internal/hostbox ./internal/activator ./internal/host ./internal/provision ./internal/console ./internal/instancectl ./internal/schedule -run '^Test(Claim|ServingRetries|Renewing|AggregateContext|IdentitySeparates|EmptyCensus|RegistryRows|ColdHandover|HostWakeReset|RootRetirement|BeginFence|Cleanup|Fence|StartAdmitted|SameAdmission|SetupAt)' -count=1
-go build ./...
-go vet ./...
-go test ./... -count=1
-GOOS=darwin GOARCH=amd64 go build ./...
-GOOS=windows GOARCH=amd64 go build ./...
-GOOS=linux GOARCH=arm64 go build ./...
+nice -n 10 go test -p 1 ./internal/registry ./internal/hostbox ./internal/activator ./internal/host ./internal/provision ./internal/console ./internal/instancectl ./internal/schedule -run '^Test(Claim|ServingRetries|Renewing|AggregateContext|Identity|AdapterSelectionLayout|Launch|Prelaunch|PreLaunch|Recover|EmptyCensus|RegistryRows|ColdHandover|HostWakeReset|RootRetirement|BeginFence|Cleanup|Fence|StartAdmitted|SameAdmission|SetupAt)' -count=1
+nice -n 10 go build -p 1 ./...
+nice -n 10 go vet -p 1 ./...
+nice -n 10 go test -p 1 ./... -count=1
+GOOS=darwin GOARCH=amd64 nice -n 10 go build -p 1 ./...
+GOOS=windows GOARCH=amd64 nice -n 10 go build -p 1 ./...
+GOOS=linux GOARCH=arm64 nice -n 10 go build -p 1 ./...
```
-Use the default Go build cache. If this environment denies writes to it, use only the existing shared on-disk cache that the machine provides; do not allocate a new task cache or use the RAM-backed /tmp. An exported source tree without Git metadata needs `-buildvcs=false`; that flag changes build stamping, not the code being checked.
+Use the default Go build cache. For this authoring session only, the user authorized `GOCACHE=/home/paulca/.cache/tmp/claude-1001/own/gocache-6b` if the default is read-only; remove that fallback cache when done. Put scratch under `/home/paulca/.cache/tmp/claude-1001/` and remove it after verification. Run Go gates serially with `nice -n 10` and `-p 1`. An exported source tree without Git metadata needs `-buildvcs=false`; that flag changes build stamping, not the code being checked.
Review the resulting diff against the parent, including actual callers. The repository requires authority/production review by Opus before implementation lands. Plan authorship and local compilation are not that review. Land the eventual implementation only through the reviewed amadan branch and its green exact-tip gate. This document does not authorize a production rollout or infrastructure change.
@@ -9949,7 +12609,7 @@ Keep CARLOS_HOST_CLAIM unset until the owner adapters and these checks pass in t
1. Verify flag-off startup, ordinary reconcile, timers and root verbs against the same binary. Compare routes and timer bookkeeping before and after a serial bump. No claim provider opens or worker starts from an unset flag.
2. Run the strict containment tests in a delegated system service with the real required controllers, including io. The authoring machine's user service lacks that controller. Confirm every enabled supported launch uses the recorded session parent and a fresh role child, with no host-level fallback.
3. Stop a main process with a forked child holding SQLite, WAL and SHM descriptors. Include renamed and deleted files, a late-created WAL, a producer issuing bounded requests, and a timer HTTP request paused after admission. Root must refuse until every producer and child is joined and the complete descriptor scan is clear. Unreadable proc entries or incomplete inventory must produce no receipt.
-4. Kill the agent after reservation, after Start and during readiness. Retry must stop the exact attempt and retain the spent budget. Test all five failed starts, agent restart, changed release and changed stamp. Only exact backend health or the authenticated conditional operator reset grants eligibility. A service credential and a non-operator cannot reset.
+4. Kill the agent after reservation, after pre-launch journal commit, after restore but before materialization, after Start and during readiness. Inject restore and materialization failures. Restart with a new controller identity and a newer roster revision in the same enrollment; root must stop the persisted old tuple, refuse while its old process is alive, and accept an empty producer set only with valid evidence. Retry must stop the exact attempt and retain the spent budget. Test all five failed starts, agent restart, changed release and changed stamp. Only exact backend health or the authenticated conditional operator reset grants eligibility. A service credential and a non-operator cannot reset.
5. Stop root after fencing, after drain, after durable receipt, after an archival fsync and after SQL deletion. Resume each point. Publication retries the original bytes and evidence time. A failed acknowledgement must be replayable even after the route disappeared. A replacement row or path must survive predecessor recovery.
6. Reboot with a pending stop. Prove the old controller cannot exist using the real kernel boot ID. Reconstruct positive evidence from the retained inventory and journal; missing old paths on the same boot or a reused path on the new boot must refuse. A persisted fence must deny returning work before provider refresh.
7. Exercise two consecutive cold admissions. Hold the old session stopped while the successor barrier is durable. Crash at every park/row-restore phase, resume, then create a newer timer registration and replay the completed handover. The registration and current bookkeeping must remain unchanged. No old database bytes may be reused as the new writable database.
@@ -9965,7 +12625,7 @@ Do not substitute passing filesystem fixtures for these kernel witnesses. The su
- Writer retry state is durable. Lease cleanup and failure settlement do not refund attempts. Probe authority cannot clear serving failure.
- The role launcher pins the executable bytes selected by the authenticated release manifest, not a checksum supplied by an app. Readiness uses the direct backend socket and exact process identity, then rechecks the journal's attempt/session/input binding.
- Cold ownership handover parks the complete predecessor and preserves its archive. Restored rows receive new ownership while timer registration and scheduling state stay exact. A guard barrier closes the crash interval between remote admission and local completion.
-- The inventory includes managed directories so late-created WAL/SHM names remain in descriptor scope. Missing inventory, unresolved materialization and reused identities refuse. Shared directories and the whole registry database are excluded from host retirement.
+- The inventory includes managed directories so late-created WAL/SHM names remain in descriptor scope. Missing inventory and reused identities refuse. Unresolved materialization follows only the explicit typed intention recovery below; it never licenses arbitrary artifact capture. Shared directories and the whole registry database are excluded from host retirement.
- Root authenticates kernel peer identity and validates evidence independently. Serving stop and operation evidence cannot be confused. Durable cleanup permits old exact-target completion, never new publication authority or successor deletion.
- Timer admission happens after cold wake. Cancellation joins HTTP before drain acknowledgement. Post-delivery bookkeeping retains the original selection and registration.
- Registry archival is durable before deletion. Ownership, row ID, registration and observed hold are checked in SQL. Clearing a stale hold cannot erase failed-host, retry, quarantine or execution-loss state.
@@ -9977,18 +12637,20 @@ These are implementation dependencies, not undecided design choices: Plan 8's ve
### Authoring verification
-The code was assembled and tested in disposable on-disk exports of `feat/hostclaim-box` at `7cc781600f69e25b61ad2ba02ccf2a880b0266de`. Only this plan is written to the design worktree. No implementation branch, staged files or commit was created.
+Every code block was replayed into a fresh on-disk export of `origin/main` at `2b2cc879a669b850bf8ec410b13d20673709477d`. That was the latest remote main checked for the final replay, after incorporating the commits that arrived during authoring. Only this plan was written to the design worktree; no branch, index or commit was changed. Plan 5's console-ops document was not touched.
+
+The document's exact extractor applied the tests first; test compilation failed on the missing assembly APIs as expected. Applying every implementation block then reproduced the authored overlay byte for byte. Whole-tree `gofmt -l .` produced no output.
+
+Final authoring gates ran serially with `nice -n 10`, `-p 1` and `-buildvcs=false`:
-The document extractor was exercised against a fresh parent export. The tests-only stage failed on the missing new APIs. Applying its implementation blocks then reproduced every overlay file byte for byte, and that replayed tree passed `go build -buildvcs=false ./...`.
+- `go build ./...`, `go vet ./...` and `go test ./... -run '^$'` passed. The last command compiles all tests; it does not execute the full suite.
+- Darwin/amd64, Windows/amd64 and Linux/arm64 builds passed.
+- Focused host, hostguard, activator, hostbox, registry, provision, instancectl, cgroup and schedule tests passed, covering admission refusal and accepted create/keep, nested layout, signed enrollment, journal recovery, durable retry settlement, stop evidence and typed retirement. The console `TestHostWakeReset` cases passed separately. Main's layout/default/startup integration cases also passed after the base update.
-The final overlay also passed:
+Behavioral failing-first checks complemented the missing-API compile failure. Restoring the old admission ordering made refusal tests observe guard/inventory changes; removing the existing-row refusal and nested-layout fixes failed their regressions. Substituting the restarted controller's tuple failed restart recovery, and substituting its directory revision failed recovery after a same-enrollment roster update. Removing the pre-launch journal write failed restore/materialization/crash recovery tests. The old nil-versus-empty producer comparison failed after acknowledgement JSON round-trip. Comparing signed cloud region with placement region failed enrollment validation. The old handover failed absent and empty intended-directory recovery; the interrupted all-absent case also failed until its decision became durable. Each corrected implementation passed its regression afterward.
-- `go vet -buildvcs=false ./...` and `go test -buildvcs=false ./... -run '^$'`, compiling every test package without claiming those tests executed.
-- Whole-tree `gofmt -l .`, with no output.
-- Darwin/amd64, Windows/amd64 and Linux/arm64 `go build -buildvcs=false ./...` cross-builds.
-- Focused host, hostguard, activator, hostbox, registry, provision, instancectl and cgroup tests covering the new and changed paths. The exact reset-door tests passed separately with `go test ./internal/console -run TestHostWakeReset -count=1`.
-- The complete schedule and hostguard package suites. The broader registry, provision and hostbox suites also passed before the last parent-base update; the final base was checked with the focused cases above.
+The recovery fixture drives the real wake assembly, serving retry state, disk-backed launch journal, guard, stop coordinator and `ColdRestore.InstallEmpty`; it injects the lease, unprivileged directory creation and kernel observations. It does not replace the restore closure. The crash case verifies the real generation marker before discarding the old controller and confirms materialization never ran. These tests demonstrate assembly and persistence behavior, not privileged containment or descriptor evidence.
-A wider test selection reached existing socket-based tests and failed under this sandbox: activator control-socket setup returned `setsockopt: operation not permitted`, and console fixtures could not listen on `[::1]:0`. Those are not green tests and were not bypassed by changing application behavior. A full `go test ./...` run and the delegated kernel witnesses remain required in a suitable test service. No Opus review or live deployment was performed.
+Opus reviewed admission and recovery authority. The resulting fixes include incompatible-row refusal, nested inventory layout, persisted directory revision, zero-producer validation and typed intention recovery, including descriptor-based emptiness and a durable all-absent decision. The eventual implementation still needs the repository's reviewed-branch gate. Full-suite execution and the delegated kernel witnesses above remain required before rollout; no live deployment was performed.
-The default cache was not writable in this sandbox. Verification used the existing shared disk cache at `/home/paulca/.cache/tmp/gocache` with `GOPROXY=off`; it did not create a per-task Go cache. Disposable exports and authoring scratch are removed when this plan is handed back.
+The default Go cache was read-only in this sandbox. The user's authorized fallback was `/home/paulca/.cache/tmp/claude-1001/own/gocache-6b`. That cache and all scratch created for this revision were removed after verification.