carlos / platform Public

plan: claim Plan 8b rev 3, answers review round 2 (four P2)

Co-Authored-By: Codex <noreply@openai.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
codex agent, operated by paul@keymail.dev pushed by claude agent, operated by paul@keymail.dev f610d78e1659453930a22750494d713f9e206c62 parent 408bcb3
1 file changed, +609 −83
  • docs/superpowers/plans/2026-10-06-host-lifecycle-claim-8b-floor-roster.md +609 −83

CI passed — run details

diff --git a/docs/superpowers/plans/2026-10-06-host-lifecycle-claim-8b-floor-roster.md b/docs/superpowers/plans/2026-10-06-host-lifecycle-claim-8b-floor-roster.md
index 81c0162ba..7bba1a402 100644
--- a/docs/superpowers/plans/2026-10-06-host-lifecycle-claim-8b-floor-roster.md
+++ b/docs/superpowers/plans/2026-10-06-host-lifecycle-claim-8b-floor-roster.md
@@ -1,6 +1,6 @@
# Host lifecycle claim, Plan 8b: the floor, the roster and live fencing
-Rev 2 answers the review (one P1, six P2) and aligns with claim rev 6.30.
+Rev 3 answers review round 2 (four P2).
Implementation plan, 2026-10-06. Branch to implement: `feat/hostclaim-floor`,
from `feat/hostclaim-leases` at
@@ -87,8 +87,10 @@ nothing about disks.
bypass that check. Cloud identity and enrollment never change in place.
3. A signed invitation supplies the random enrollment nonce before a new box
becomes a member. That box can arm its root guard and report readiness
- while admission still refuses it. This breaks the post-floor enrollment
- cycle without creating provisional execution authority.
+ while admission still refuses it. Fleet invitations generate enrollment
+ first, then derive `fleet-<enrollment>` and bind the identity evidence to
+ that reference. This breaks the post-floor enrollment cycle without
+ creating provisional execution authority.
4. The roster has additive `regions`, `publication` and `floor_sha256` fields.
The initial publication CAS freezes membership at `preparing`; failed
readiness checks leave it frozen. Publication writes the authentication
@@ -117,7 +119,8 @@ nothing about disks.
exact enrollment references and authenticated hashes. Neither expires.
An uncertain start is cancelled by first invalidating its expected claim
ETag, including an absent claim's create-only position. The start CAS
- writes `actors_hold`. A first-phase participant checks the hold before
+ writes `actors_hold` after rereading actor eligibility and termination
+ fences. A first-phase participant checks the hold before
its acknowledgement. No lease PUT or effect may substitute for that check.
9. The root checker, policy, identities and unit drop-ins live outside
`system/router`'s restore set. `armed` already enforces the prospective
@@ -126,13 +129,16 @@ nothing about disks.
Neither a shell fallback nor `version` inspection can run unverified bytes.
10. Root writes update-begun before inspecting the complete controller
directory. Any entry, including `lost` or malformed state, refuses the
- update. Controllers persist `pending` before checking root's marker and
- journal. Only D6's completed cleanup clears controller exclusion.
+ update. Recovery repeats that complete scan before any recovery effect,
+ including completion of a transaction without a journal. Controllers
+ persist `pending` before checking root's marker and journal. Only D6's
+ completed cleanup clears controller exclusion.
11. Operator loss resolution still requires a person and `Store.IsOperator`.
Fresh inspection binds exact claim, record, enrollment, operation and
consequences. Compute is observed by the operator CLI, not the console.
- Global quarantine precedes claim mutation, partial failure keeps it, and
- readmission clears it last by the original roster ETag. Unknown timers
+ Signed acceptance evidence is durable before quarantine records the loss
+ ID. Global quarantine precedes claim mutation, partial failure keeps it,
+ and readmission clears it last by the original roster ETag. Unknown timers
remain unknown. A successful retry retains the original accepted fence.
12. Only operation owners present on the parent are wired: delete/parking and
serving's exact destroyed-local-only case. Moving, landing, reverting,
@@ -537,16 +543,26 @@ func (d Disk) Begin(id, dataDir string) error {
if err := Durable(marker, raw, 0644); err != nil {
return err
}
- entries, err := os.ReadDir(filepath.Join(dataDir, ".hostexec"))
- if err != nil || len(entries) != 0 {
+ if err := CheckControllersIdle(dataDir); err != nil {
removeErr := RemoveDurable(marker)
- if err == nil {
- err = ErrFloor
- }
return errors.Join(err, removeErr)
}
return nil
}
+
+// CheckControllersIdle refuses while any controller state is present. The
+// directory must also be readable: a missing or unreadable directory cannot
+// prove that no controller is active.
+func CheckControllersIdle(dataDir string) error {
+ entries, err := os.ReadDir(filepath.Join(dataDir, ".hostexec"))
+ if err != nil {
+ return err
+ }
+ if len(entries) != 0 {
+ return ErrFloor
+ }
+ return nil
+}
func RemoveDurable(path string) error {
if err := os.Remove(path); err != nil && !errors.Is(err, os.ErrNotExist) {
return err
@@ -1199,6 +1215,63 @@ func TestEnrollmentUsesSignedCloudAndRefusesRebinding(t *testing.T) {
t.Fatal("reference rebound")
}
}
+
+func TestFleetInvitationBindsReferenceToEnrollment(t *testing.T) {
+ d, sg, o := fixture(t)
+ key, err := rsa.GenerateKey(rand.Reader, 2048)
+ if err != nil {
+ t.Fatal(err)
+ }
+ cert := &x509.Certificate{PublicKey: &key.PublicKey, PublicKeyAlgorithm: x509.RSA}
+ document := []byte(`{"accountId":"123","region":"eu-west-1","instanceId":"i-fleet"}`)
+ sum := sha256.Sum256(document)
+ signature, err := rsa.SignPKCS1v15(rand.Reader, key, crypto.SHA256, sum[:])
+ if err != nil {
+ t.Fatal(err)
+ }
+ rec := boxid.Record{V: 1, IdentityDocument: base64.StdEncoding.EncodeToString(document), IdentitySignature: base64.StdEncoding.EncodeToString(signature)}
+ raw, _ := json.Marshal(rec)
+ invite, err := d.Invite(context.Background(), sg, "", raw)
+ if err != nil {
+ t.Fatal(err)
+ }
+ if invite.Reference != "fleet-"+invite.Enrollment {
+ t.Fatalf("reference = %q, enrollment = %q", invite.Reference, invite.Enrollment)
+ }
+ rec.Box = invite.Reference
+ bound, _ := json.Marshal(rec)
+ retry, err := d.Invite(context.Background(), sg, invite.Reference, bound)
+ if err != nil || retry != invite {
+ t.Fatalf("retry = %+v, %v", retry, err)
+ }
+ if _, err := d.Invite(context.Background(), sg, "fleet-00000000000000000000000000000000", bound); err == nil {
+ t.Fatal("caller-chosen fleet reference accepted")
+ }
+ v := boxid.Verifier{Account: "123", Certs: map[string]*x509.Certificate{"eu-west-1": cert}}
+ ordinary := rec
+ ordinary.Box = "ordinary-label"
+ ordinaryRaw, _ := json.Marshal(ordinary)
+ if _, err := d.Invite(context.Background(), sg, ordinary.Box, ordinaryRaw); err != nil {
+ t.Fatal(err)
+ }
+ badFleet := EnrollmentRequest{Reference: ordinary.Box, Namespace: "fleet:workers", Label: "fleet", Region: "ie", Pool: "default", AWSRegion: "eu-west-1", Operator: "operator"}
+ if _, err := d.Enroll(context.Background(), sg, o, v, badFleet, ordinaryRaw); err == nil {
+ t.Fatal("fleet namespace accepted arbitrary reference")
+ }
+ in := EnrollmentRequest{Reference: invite.Reference, Namespace: "fleet:workers", Label: "fleet", Region: "ie", Pool: "default", AWSRegion: "eu-west-1", Operator: "operator"}
+ next, err := d.Enroll(context.Background(), sg, o, v, in, bound)
+ if err != nil {
+ t.Fatal(err)
+ }
+ if got := next.Roster.Boxes[invite.Reference]; got.Enrollment != invite.Enrollment || got.BoxID != invite.Reference {
+ t.Fatal(got)
+ }
+ rec.Box = "fleet-ffffffffffffffffffffffffffffffff"
+ mismatched, _ := json.Marshal(rec)
+ if _, err := d.Enroll(context.Background(), sg, o, v, in, mismatched); err == nil {
+ t.Fatal("mismatched fleet evidence accepted")
+ }
+}
```
### Create `internal/hostactors/roster_test.go`
@@ -1293,6 +1366,8 @@ import (
"context"
"encoding/json"
"errors"
+ "github.com/carlosframework/platform/internal/hostclaim"
+ "github.com/carlosframework/platform/internal/hostfloor"
"github.com/carlosframework/platform/internal/release"
"strings"
"testing"
@@ -1507,6 +1582,69 @@ func TestFenceAuthorityErrorsAndMismatchesRefuse(t *testing.T) {
})
}
}
+
+func TestStartRevalidatesReservedActorEligibility(t *testing.T) {
+ for _, fault := range []string{"good", "quarantined", "retired", "fenced", "malformed", "unreadable", "console-quarantined", "console-fenced"} {
+ t.Run(fault, func(t *testing.T) {
+ ctx := t.Context()
+ d, sg, o := fixture(t)
+ floor := hostfloor.Floor{Min: 1, MinRelease: "v1.2.3"}
+ if err := hostfloor.Publish(ctx, d.Store, sg, d.Bucket, floor); err != nil {
+ t.Fatal(err)
+ }
+ f, err := hostfloor.Read(ctx, d.Store, d.PublicKey, d.Bucket)
+ if err != nil {
+ t.Fatal(err)
+ }
+ r := Clone(o.Roster)
+ r.Publication, r.FloorSHA256 = "published", f.Digest
+ actor, enrollment := "box:box", r.Boxes["box"].Enrollment
+ if strings.HasPrefix(fault, "console-") {
+ actor, enrollment = "console:console-1", strings.Repeat("c", 32)
+ r.Consoles["console-1"] = Console{Enrollment: enrollment, InstanceID: "console-1", Namespaces: []string{"platform"}, DeploymentBucket: d.Bucket, Compute: &ComputeBinding{Box: "box", Enrollment: r.Boxes["box"].Enrollment}}
+ }
+ o, err = d.Replace(ctx, sg, o, r)
+ if err != nil {
+ t.Fatal(err)
+ }
+ before, err := hostclaim.Read(ctx, d.Store, "shop.example.com")
+ if err != nil {
+ t.Fatal(err)
+ }
+ h := Reservation{ID: "hold", Host: before.Host, OpID: "operation", ExpectedClaimETag: before.ETag, FloorSHA256: f.Digest, Actors: map[string]string{actor: enrollment}, Releases: map[string]Report{actor: {Enrollment: enrollment, GeneratedAt: time.Now()}}}
+ o, err = d.Reserve(ctx, sg, o, h, false)
+ if err != nil {
+ t.Fatal(err)
+ }
+ switch fault {
+ case "quarantined", "console-quarantined":
+ _, err = d.Quarantine(ctx, sg, o, "box", o.Roster.Boxes["box"].Enrollment, "loss")
+ case "retired":
+ r = Clone(o.Roster)
+ b := r.Boxes["box"]
+ b.State = "retired"
+ r.Boxes["box"] = b
+ _, err = d.Replace(ctx, sg, o, r)
+ case "fenced", "console-fenced", "malformed":
+ writeFence(t, d, o, "box", time.Now(), fault == "malformed")
+ case "unreadable":
+ d.Store = brokenFenceStore{d.Store}
+ }
+ if err != nil {
+ t.Fatal(err)
+ }
+ called := false
+ _, err = d.Start(ctx, before, h.ID, func(*hostclaim.Claim) error { called = true; return errors.New("stop before writing") })
+ if err == nil || called != (fault == "good") {
+ t.Fatalf("transition called=%v, error=%v", called, err)
+ }
+ after, err := hostclaim.Read(ctx, d.Store, before.Host)
+ if err != nil || before.ETag != after.ETag || before.Absent != after.Absent {
+ t.Fatal("refusal changed claim", err)
+ }
+ })
+ }
+}
```
### Create `internal/hostactors/attempt_regression_test.go`
@@ -2403,10 +2541,14 @@ import (
"context"
"crypto/rand"
"encoding/hex"
+ "encoding/json"
"errors"
+ "os"
+ "strings"
+
+ "github.com/carlosframework/platform/internal/boxid"
"github.com/carlosframework/platform/internal/hostfloor"
"github.com/carlosframework/platform/internal/release"
- "os"
)
type Invitation struct {
@@ -2419,31 +2561,57 @@ type Invitation struct {
// Invite grants no membership. It gives an enrolled-identity candidate a
// stable nonce with which to install its root guard and publish readiness.
+// A fleet caller passes an empty reference, then changes only boxid.Record.Box
+// to the returned Reference and marshals it again before publishing or Enroll.
+// That field is outside the AWS-signed identity document and signature.
func (d Directory) Invite(ctx context.Context, sg release.Signer, reference string, evidence []byte) (Invitation, error) {
- if !segment(reference) {
+ fleet := reference == ""
+ if !fleet && !segment(reference) {
return Invitation{}, ErrRoster
}
- key := "control/host-actor-invitations/" + reference + ".json"
- raw, _, err := d.Store.Get(ctx, key)
- if err == nil {
- var invite Invitation
- if release.VerifyRootEnvelope(d.PublicKey, raw, &invite) != nil || invite.V != 1 || invite.Bucket != d.Bucket || invite.Reference != reference || invite.EvidenceSHA256 != hostfloor.Digest(evidence) {
+ if !fleet {
+ key := "control/host-actor-invitations/" + reference + ".json"
+ raw, _, err := d.Store.Get(ctx, key)
+ if err == nil {
+ var invite Invitation
+ if release.VerifyRootEnvelope(d.PublicKey, raw, &invite) != nil || invite.V != 1 || invite.Bucket != d.Bucket || invite.Reference != reference || !token.MatchString(invite.Enrollment) || invite.EvidenceSHA256 != hostfloor.Digest(evidence) || (strings.HasPrefix(reference, "fleet-") && reference != "fleet-"+invite.Enrollment) {
+ return Invitation{}, ErrRoster
+ }
+ return invite, nil
+ }
+ if !errors.Is(err, os.ErrNotExist) {
+ return Invitation{}, err
+ }
+ // Fleet references are minted from their enrollment nonce. A caller may
+ // retry one returned by this function, but cannot choose a fresh one.
+ if strings.HasPrefix(reference, "fleet-") {
return Invitation{}, ErrRoster
}
- return invite, nil
- }
- if !errors.Is(err, os.ErrNotExist) {
- return Invitation{}, err
}
var id [16]byte
- if _, err = rand.Read(id[:]); err != nil {
+ if _, err := rand.Read(id[:]); err != nil {
return Invitation{}, err
}
- invite := Invitation{1, d.Bucket, reference, hex.EncodeToString(id[:]), hostfloor.Digest(evidence)}
- raw, err = release.SignRootEnvelope(sg, invite)
+ enrollment := hex.EncodeToString(id[:])
+ if fleet {
+ reference = "fleet-" + enrollment
+ var rec boxid.Record
+ if json.Unmarshal(evidence, &rec) != nil || rec.V != 1 {
+ return Invitation{}, ErrRoster
+ }
+ rec.Box = reference
+ var err error
+ evidence, err = json.Marshal(rec)
+ if err != nil {
+ return Invitation{}, err
+ }
+ }
+ invite := Invitation{1, d.Bucket, reference, enrollment, hostfloor.Digest(evidence)}
+ raw, err := release.SignRootEnvelope(sg, invite)
if err != nil {
return Invitation{}, err
}
+ key := "control/host-actor-invitations/" + reference + ".json"
if _, err = d.Store.Put(ctx, key, raw, ""); err != nil {
return Invitation{}, err
}
@@ -2456,7 +2624,7 @@ func (d Directory) enrollmentNonce(ctx context.Context, in EnrollmentRequest, ev
return "", err
}
var invite Invitation
- if release.VerifyRootEnvelope(d.PublicKey, raw, &invite) != nil || invite.V != 1 || invite.Bucket != d.Bucket || invite.Reference != in.Reference || !token.MatchString(invite.Enrollment) || invite.EvidenceSHA256 != hostfloor.Digest(evidence) {
+ if release.VerifyRootEnvelope(d.PublicKey, raw, &invite) != nil || invite.V != 1 || invite.Bucket != d.Bucket || invite.Reference != in.Reference || !token.MatchString(invite.Enrollment) || ((strings.HasPrefix(in.Namespace, "fleet:") || strings.HasPrefix(in.Reference, "fleet-")) && in.Reference != "fleet-"+invite.Enrollment) || invite.EvidenceSHA256 != hostfloor.Digest(evidence) {
return "", ErrRoster
}
return invite.Enrollment, nil
@@ -2795,6 +2963,7 @@ import (
"github.com/carlosframework/platform/internal/hostfloor"
"github.com/carlosframework/platform/internal/release"
"reflect"
+ "strings"
"time"
)
@@ -2805,7 +2974,7 @@ func (d Directory) Start(ctx context.Context, o hostclaim.Observed, id string, t
return hostclaim.Observed{}, err
}
h, ok := roster.Roster.Holds[id]
- if !ok || h.Host != o.Host || h.ExpectedClaimETag != o.ETag {
+ if !ok || h.Host != o.Host || h.ExpectedClaimETag != o.ETag || roster.Roster.Publication != "published" || roster.Roster.FloorSHA256 != h.FloorSHA256 {
return hostclaim.Observed{}, ErrRoster
}
floor, err := hostfloor.Read(ctx, d.Store, d.PublicKey, d.Bucket)
@@ -2817,6 +2986,29 @@ func (d Directory) Start(ctx context.Context, o hostclaim.Observed, id string, t
return hostclaim.Observed{}, ErrRoster
}
}
+ // Reservations pin identity, but never preserve an actor's eligibility.
+ for actor, enrollment := range h.Actors {
+ ref, boxActor := strings.CutPrefix(actor, "box:")
+ if !boxActor {
+ id, consoleActor := strings.CutPrefix(actor, "console:")
+ console, exists := roster.Roster.Consoles[id]
+ if !consoleActor || !exists || console.Enrollment != enrollment {
+ return hostclaim.Observed{}, ErrRoster
+ }
+ if console.Compute == nil {
+ continue
+ }
+ ref, enrollment = console.Compute.Box, console.Compute.Enrollment
+ }
+ box, exists := roster.Roster.Boxes[ref]
+ if !exists || box.Enrollment != enrollment || box.State != "active" {
+ return hostclaim.Observed{}, ErrRoster
+ }
+ fenced, err := d.TerminationFence(ctx, roster, ref, time.Now())
+ if err != nil || fenced {
+ return hostclaim.Observed{}, ErrRoster
+ }
+ }
return hostclaim.Update(ctx, d.Store, o, func(c *hostclaim.Claim) error {
if err := tr(c); err != nil {
return err
@@ -3314,7 +3506,9 @@ The update lock owns recovery of both durable records. If activation was saved
before the marker, or the marker survives without a journal, verify the signed
transaction and selected current image and close that transaction. A journal
requires its matching marker and verifies the selected recovery image before
-any executable action. Journal removal is checked and synced. Completion is
+any executable action. Every unfinished recovery first scans the complete
+controller directory; pending, lost, malformed or unreadable state refuses
+before executables, journal, marker or completion evidence can change. Journal removal is checked and synced. Completion is
written and synced before marker removal; recovery retries either write. Report
completion only when both marker and journal are absent. A failed staging or
no-op return runs the same no-journal cleanup before releasing the update lock.
@@ -3536,6 +3730,102 @@ func TestGuardRecoveryRetainsMarkerOnInvalidImageOrMarker(t *testing.T) {
})
}
}
+
+func TestGuardRecoveryRefusesControllersAfterBegin(t *testing.T) {
+ states := []struct {
+ name string
+ malformed bool
+ unreadable bool
+ }{
+ {name: "pending"},
+ {name: "active"},
+ {name: "lost"},
+ {name: "malformed", malformed: true},
+ {name: "unreadable", unreadable: true},
+ }
+ paths := []struct {
+ name string
+ journal bool
+ marker bool
+ }{
+ {name: "journal-backing-up", journal: true, marker: true},
+ {name: "terminal-marker", marker: true},
+ {name: "terminal-no-marker"},
+ }
+ for _, state := range states {
+ for _, path := range paths {
+ t.Run(state.name+"-"+path.name, func(t *testing.T) {
+ f, id := guardRecoveryFixture(t)
+ a := f.a
+ if err := a.floor.Begin(id, a.dataDir); err != nil {
+ t.Fatal(err)
+ }
+ marker := filepath.Join(a.floor.Dir, "update-begun.json")
+ binary := systemBinaryPath(a.optDir)
+ before, err := os.ReadFile(binary)
+ if err != nil {
+ t.Fatal(err)
+ }
+ if path.journal {
+ if err := a.journal(systemJournal{Phase: systemPhaseBackingUp, Target: "new-build", PrevBuild: "old-build", StartedAt: f.now}); err != nil {
+ t.Fatal(err)
+ }
+ }
+ controllers := filepath.Join(a.dataDir, ".hostexec")
+ if state.unreadable {
+ if err := os.RemoveAll(controllers); err != nil {
+ t.Fatal(err)
+ }
+ if err := os.WriteFile(controllers, []byte("not a directory"), 0644); err != nil {
+ t.Fatal(err)
+ }
+ } else if state.malformed {
+ if err := os.WriteFile(filepath.Join(controllers, "host.json"), []byte("{"), 0600); err != nil {
+ t.Fatal(err)
+ }
+ } else {
+ raw, err := json.Marshal(map[string]string{"state": state.name})
+ if err != nil {
+ t.Fatal(err)
+ }
+ if err := os.WriteFile(filepath.Join(controllers, "host.json"), raw, 0600); err != nil {
+ t.Fatal(err)
+ }
+ }
+ if !path.marker {
+ if err := hostfloor.RemoveDurable(marker); err != nil {
+ t.Fatal(err)
+ }
+ }
+ if err := a.Run(t.Context(), true); err == nil {
+ t.Fatal("unsafe recovery succeeded")
+ }
+ if path.marker {
+ if _, err := os.Stat(marker); err != nil {
+ t.Fatal("marker discarded", err)
+ }
+ } else if _, err := os.Stat(marker); !os.IsNotExist(err) {
+ t.Fatal("marker unexpectedly present", err)
+ }
+ if path.journal {
+ if _, err := os.Stat(a.file("journal.json")); err != nil {
+ t.Fatal("journal discarded", err)
+ }
+ }
+ if _, err := os.Stat(filepath.Join(a.floor.Dir, "completed.json")); !os.IsNotExist(err) {
+ t.Fatal("completion written", err)
+ }
+ after, err := os.ReadFile(binary)
+ if err != nil || string(after) != string(before) {
+ t.Fatal("binary changed", err)
+ }
+ if len(f.calls) != 0 {
+ t.Fatal("recovery executed a binary", f.calls)
+ }
+ })
+ }
+ }
+}
```
### Create `internal/hostexec/updateidentity_test.go`
@@ -3916,11 +4206,17 @@ func (a *systemApplier) checkGuardRecovery() error {
if readErr != nil && !errors.Is(readErr, os.ErrNotExist) {
return readErr
}
+ if err := hostfloor.CheckControllersIdle(a.dataDir); err != nil {
+ return err
+ }
return a.endGuard()
}
if err != nil {
return err
}
+ if err := hostfloor.CheckControllersIdle(a.dataDir); err != nil {
+ return err
+ }
tx, err := a.guardTransaction()
if err != nil {
return err
@@ -5086,6 +5382,17 @@ go test ./internal/hostfence -count=1
Replace both placeholder HTTP handlers and the CLI's impossible-success branch. Inspection returns a signed challenge with readable consequences. The operator observes the enrolled instance after inspection, explicitly confirms the attestation and submits the same envelope. --yes never manufactures an attestation about compute or disk destruction. The console has no EC2 client. Accepted losses are durable before their claim CAS; a claim-write failure leaves global quarantine in place. Readmission validates the fresh private report and exact retirement receipt, all retained declarations and complete claim inventory before clearing the global deny. Report publishing runs at startup and five-minute intervals outside NewServer. System promote and rollback reserve exact affected actors before changing a pointer; rollback also binds the pointer ETag read before reservation.
+Persist root-signed acceptance at
+`control/host-loss-acceptances/<declaration-id>.json` before the roster CAS
+records that ID. It contains the prepared loss record, original acceptance
+and fence times, enrollment, consequences, expected quarantine revision and
+observed roster ETag. Retry reads it before checking challenge expiry,
+materializes the loss record after quarantine, and preserves those times.
+An unrelated roster change uses a fresh CAS with the same box identity and
+prior quarantine revision. A competing declaration for that box defeats the
+prepared revision and requires new inspection; it adds no orphan loss ID.
+Readmission still requires every quarantined declaration to be committed.
+
**Step 1: add the failing tests.**
### Create `internal/hostresolve/service_test.go`
@@ -5097,12 +5404,16 @@ import (
"context"
"crypto/ed25519"
"crypto/rand"
+ "encoding/json"
"errors"
"fmt"
"github.com/carlosframework/platform/internal/hostactors"
"github.com/carlosframework/platform/internal/hostclaim"
"github.com/carlosframework/platform/internal/hostfence"
+ "github.com/carlosframework/platform/internal/hostfloor"
+ "github.com/carlosframework/platform/internal/hostquarantine"
"github.com/carlosframework/platform/internal/release"
+ "github.com/carlosframework/platform/internal/status"
"strings"
"testing"
"time"
@@ -5110,14 +5421,22 @@ import (
type failClaimStore struct {
release.Store
- fail bool
+ fail bool
+ crash string
}
func (s *failClaimStore) Put(ctx context.Context, key string, raw []byte, etag string) (string, error) {
if s.fail && strings.HasPrefix(key, "control/hosts/") {
return "", errors.New("claim write unavailable")
}
- return s.Store.Put(ctx, key, raw, etag)
+ if s.crash == "acceptance" && strings.HasPrefix(key, "control/host-loss-acceptances/") || s.crash == "before-quarantine" && key == hostactors.Key || s.crash == "loss-create" && strings.HasPrefix(key, "control/host-loss/") {
+ return "", errors.New("crash before durable write")
+ }
+ next, err := s.Store.Put(ctx, key, raw, etag)
+ if err == nil && s.crash == "after-quarantine" && key == hostactors.Key {
+ return "", errors.New("crash after durable quarantine")
+ }
+ return next, err
}
func resolveFixture(t *testing.T) (Service, *failClaimStore, *time.Time, string) {
t.Helper()
@@ -5202,6 +5521,135 @@ func TestFailedClaimWriteLeavesGlobalQuarantine(t *testing.T) {
t.Fatal(err)
}
}
+
+func TestDeclareCrashAndChallengeExpiryAllowRetryAndReadmission(t *testing.T) {
+ for _, crash := range []string{"acceptance", "before-quarantine", "after-quarantine", "loss-create"} {
+ t.Run(crash, func(t *testing.T) {
+ ctx := t.Context()
+ s, store, now, host := resolveFixture(t)
+ in := submitted(t, s, now, host)
+ acceptedAt := *now
+ before, err := hostclaim.Read(ctx, store, host)
+ if err != nil {
+ t.Fatal(err)
+ }
+ store.crash = crash
+ if _, err := s.Declare(ctx, "operator", host, in); err == nil {
+ t.Fatal("crash not reached")
+ }
+ unchanged, err := hostclaim.Read(ctx, store, host)
+ if err != nil || unchanged.ETag != before.ETag {
+ t.Fatal("claim changed before quarantine completion", err)
+ }
+ roster, err := s.Directory.Read(ctx)
+ if err != nil {
+ t.Fatal(err)
+ }
+ b := roster.Roster.Boxes["dead"]
+ if crash == "acceptance" || crash == "before-quarantine" {
+ if b.State != "active" || len(b.LossIDs) != 0 {
+ t.Fatal("quarantined before acceptance", b)
+ }
+ } else if b.State != "quarantined" || len(b.LossIDs) != 1 || b.LossIDs[0] != in.Confirmation {
+ t.Fatal("deny not retained", b)
+ }
+ if _, err := hostclaim.ReadLossRecord(ctx, store, s.Directory.PublicKey, in.Confirmation); !errors.Is(err, hostclaim.ErrLossRecordAbsent) {
+ t.Fatal("loss unexpectedly materialized", err)
+ }
+ if crash != "acceptance" {
+ if _, err := s.readAcceptance(ctx, in.Confirmation); err != nil {
+ t.Fatal("acceptance missing", err)
+ }
+ }
+ store.crash = ""
+ if crash == "before-quarantine" {
+ // An unrelated roster CAS must not strand accepted evidence.
+ if _, err := s.Directory.Replace(ctx, s.Signer, roster, hostactors.Clone(roster.Roster)); err != nil {
+ t.Fatal(err)
+ }
+ }
+ *now = now.Add(time.Hour)
+ if _, err := hostfence.VerifyChallenge(s.Directory.PublicKey, in.Challenge, *now); err == nil {
+ t.Fatal("challenge did not expire")
+ }
+ if crash == "acceptance" {
+ if _, err := s.Declare(ctx, "operator", host, in); err == nil {
+ t.Fatal("unaccepted expired submission allowed")
+ }
+ return
+ }
+ if _, err := s.Declare(ctx, "operator", host, in); err != nil {
+ t.Fatal("accepted retry expired", err)
+ }
+ loss, err := hostclaim.ReadLossRecord(ctx, store, s.Directory.PublicKey, in.Confirmation)
+ if err != nil || loss.Record.State != "committed" || !loss.Record.AcceptedAt.Equal(acceptedAt) || !loss.Record.FencedAt.Equal(acceptedAt) {
+ t.Fatal("acceptance changed", loss, err)
+ }
+ prepareReadmission(t, s, *now, loss.Record)
+ roster, err = s.Directory.Read(ctx)
+ if err != nil {
+ t.Fatal(err)
+ }
+ b = roster.Roster.Boxes["dead"]
+ if err := s.Readmit(ctx, "dead", b.Enrollment, b.QuarantineRevision, b.LossIDs); err != nil {
+ t.Fatal("orphan blocked readmission", err)
+ }
+ actor, err := s.Directory.Lookup(ctx, "dead")
+ if err != nil || actor.State != "active" {
+ t.Fatal("readmission failed", actor, err)
+ }
+ })
+ }
+}
+
+func prepareReadmission(t *testing.T, s Service, now time.Time, loss hostclaim.LossRecord) {
+ t.Helper()
+ ctx := t.Context()
+ store := s.Directory.Store
+ floor := hostfloor.Floor{Min: 1, MinRelease: "v1.2.3"}
+ if err := hostfloor.Publish(ctx, store, s.Signer, s.Directory.Bucket, floor); err != nil {
+ t.Fatal(err)
+ }
+ f, err := hostfloor.Read(ctx, store, s.Directory.PublicKey, s.Directory.Bucket)
+ if err != nil {
+ t.Fatal(err)
+ }
+ hash := strings.Repeat("d", 64)
+ rel := release.SystemRelease{Version: "build123", ReleaseTag: floor.MinRelease, MovesProtocol: 1, Artifacts: map[string]release.SystemArtifact{"linux-amd64": {Key: release.SystemArtifactKey("build123", "linux-amd64"), SHA256: hash}}}
+ if _, err := release.PromoteSystem(ctx, store, s.Signer, "router", "stable", rel, release.SystemPromoteOptions{}); err != nil {
+ t.Fatal(err)
+ }
+ wire, _, err := store.Get(ctx, release.SystemPointerKey("router", "stable"))
+ if err != nil {
+ t.Fatal(err)
+ }
+ if err := release.PublishSystemIdentity(ctx, store, s.Directory.PublicKey, wire); err != nil {
+ t.Fatal(err)
+ }
+ receipt := hostquarantine.RetirementCompletion{V: 1, Box: loss.Box, Enrollment: loss.Enrollment, InstanceID: loss.CloudObservation.InstanceID, QuarantineRevision: loss.QuarantineRevision, BootID: "new-boot", CompletedAt: now, Complete: true, InventorySHA256: strings.Repeat("e", 64), Entries: []hostquarantine.RetirementEntry{{Host: loss.Host, Incarnation: loss.HolderIncarnation, Stamp: loss.HolderStamp, RetirementID: "retirement", JournalSHA256: strings.Repeat("f", 64)}}}
+ if err := receipt.Validate(); err != nil {
+ t.Fatal(err)
+ }
+ raw, err := json.Marshal(receipt)
+ if err != nil {
+ t.Fatal(err)
+ }
+ key, err := hostquarantine.RetirementKey(loss.Box, loss.Enrollment, loss.QuarantineRevision)
+ if err != nil {
+ t.Fatal(err)
+ }
+ if err := release.CreateExact(ctx, store, key, raw); err != nil {
+ t.Fatal(err)
+ }
+ report := status.Box{GeneratedAt: now, Label: "dead-label", Metrics: &status.BoxMetrics{Enrollment: loss.Enrollment, InstanceID: loss.CloudObservation.InstanceID, Release: rel.Version, ReleaseTag: rel.ReleaseTag, Platform: "linux-amd64", BinarySHA256: hash, MovesProtocol: 1, BootID: receipt.BootID, FloorSHA256: f.Digest, FloorGuardReady: true, HostClaimEnabled: true, RetirementReceiptSHA256: hostfloor.Digest(raw)}}
+ raw, err = json.Marshal(report)
+ if err != nil {
+ t.Fatal(err)
+ }
+ if err := release.CreateExact(ctx, store, "status/boxes/"+loss.Box+".json", raw); err != nil {
+ t.Fatal(err)
+ }
+}
```
Run this now and expect failure before the implementation is present:
@@ -5229,6 +5677,7 @@ import (
"github.com/carlosframework/platform/internal/hostclaim"
"github.com/carlosframework/platform/internal/hostfence"
"github.com/carlosframework/platform/internal/release"
+ "math"
"os"
"reflect"
"slices"
@@ -5381,6 +5830,17 @@ func (s Service) Declare(ctx context.Context, actor, host string, in hostfence.S
if !errors.Is(readErr, hostclaim.ErrLossRecordAbsent) {
return hostclaim.Observed{}, readErr
}
+ // Acceptance survives a crash before either quarantine or loss materialization.
+ acceptance, readErr := s.readAcceptance(ctx, c.ID)
+ if readErr == nil {
+ if acceptance.Loss.Actor != actor || acceptance.Loss.Host != host || acceptance.Loss.Enrollment != c.Enrollment || acceptance.Loss.Box != c.Box || acceptance.Loss.ExpectedClaimETag != c.ClaimETag {
+ return hostclaim.Observed{}, ErrResolve
+ }
+ return s.accept(ctx, acceptance)
+ }
+ if !errors.Is(readErr, os.ErrNotExist) {
+ return hostclaim.Observed{}, readErr
+ }
if _, err = hostfence.VerifyChallenge(s.Directory.PublicKey, in.Challenge, now); err != nil {
return hostclaim.Observed{}, err
}
@@ -5410,21 +5870,88 @@ func (s Service) Declare(ctx context.Context, actor, host string, in hostfence.S
if err != nil || !reflect.DeepEqual(shown, c.Consequences) {
return hostclaim.Observed{}, ErrResolve
}
- quarantined, err := s.Directory.Quarantine(ctx, s.Signer, roster, c.Box, c.Enrollment, c.ID)
- if err != nil {
- return hostclaim.Observed{}, err
+ if b.QuarantineRevision == math.MaxUint64 || slices.Contains(b.LossIDs, c.ID) {
+ return hostclaim.Observed{}, ErrResolve
}
- // A partial declaration remains denied. Never undo this CAS on error.
- q := quarantined.Roster.Boxes[c.Box]
- loss := hostclaim.LossRecord{V: 1, ID: c.ID, State: "prepared", Actor: actor, AcceptedAt: now, Host: host, HolderIncarnation: c.HolderIncarnation, HolderStamp: c.HolderStamp, HolderSession: c.HolderSession, ExpectedClaimGen: c.ClaimGen, ExpectedClaimETag: c.ClaimETag, OpID: c.Op.ID, OpKind: c.Op.Kind, OpPhase: c.Op.Phase, Box: c.Box, Enrollment: c.Enrollment, QuarantineRevision: q.QuarantineRevision, FencedAt: now, ConfirmedConsequences: actual, Timers: hostclaim.TimerLoss{Knowledge: hostclaim.TimersUnknown}}
+ loss := hostclaim.LossRecord{V: 1, ID: c.ID, State: "prepared", Actor: actor, AcceptedAt: now, Host: host, HolderIncarnation: c.HolderIncarnation, HolderStamp: c.HolderStamp, HolderSession: c.HolderSession, ExpectedClaimGen: c.ClaimGen, ExpectedClaimETag: c.ClaimETag, OpID: c.Op.ID, OpKind: c.Op.Kind, OpPhase: c.Op.Phase, Box: c.Box, Enrollment: c.Enrollment, QuarantineRevision: b.QuarantineRevision + 1, FencedAt: now, ConfirmedConsequences: actual, Timers: hostclaim.TimerLoss{Knowledge: hostclaim.TimersUnknown}}
loss.CloudObservation = hostclaim.LossCloudObservation{ObservedAt: in.Observation.ObservedAt, Account: in.Observation.Account, Region: in.Observation.Region, InstanceID: in.Observation.InstanceID, State: in.Observation.State, RequestID: in.Observation.RequestID}
loss.OperatorAttestation = hostclaim.LossOperatorAttestation{MaintainComputeFence: true, ColdQuarantineStartOnly: true, DestroyedStorage: in.Attestation.Disks}
+ acceptance = lossAcceptance{V: 1, Bucket: s.Directory.Bucket, RosterETag: roster.ETag, Loss: loss}
+ wire, err := release.SignRootEnvelope(s.Signer, acceptance)
+ if err != nil {
+ return hostclaim.Observed{}, err
+ }
+ key, err := acceptanceKey(c.ID)
+ if err != nil {
+ return hostclaim.Observed{}, err
+ }
+ if err := release.CreateExact(ctx, s.Directory.Store, key, wire); err != nil {
+ return hostclaim.Observed{}, err
+ }
+ return s.accept(ctx, acceptance)
+}
+
+type lossAcceptance struct {
+ V int `json:"v"`
+ Bucket string `json:"bucket"`
+ RosterETag string `json:"roster_etag"`
+ Loss hostclaim.LossRecord `json:"loss"`
+}
+
+func acceptanceKey(id string) (string, error) {
+ if _, err := hostclaim.LossRecordKey(id); err != nil {
+ return "", err
+ }
+ return "control/host-loss-acceptances/" + id + ".json", nil
+}
+func (s Service) readAcceptance(ctx context.Context, id string) (lossAcceptance, error) {
+ key, err := acceptanceKey(id)
+ if err != nil {
+ return lossAcceptance{}, err
+ }
+ raw, _, err := s.Directory.Store.Get(ctx, key)
+ if err != nil {
+ return lossAcceptance{}, err
+ }
+ var a lossAcceptance
+ if release.VerifyRootEnvelope(s.Directory.PublicKey, raw, &a) != nil || a.V != 1 || a.Bucket != s.Directory.Bucket || a.RosterETag == "" || a.Loss.ID != id || a.Loss.State != "prepared" || a.Loss.Validate() != nil {
+ return lossAcceptance{}, ErrResolve
+ }
+ return a, nil
+}
+func (s Service) accept(ctx context.Context, a lossAcceptance) (hostclaim.Observed, error) {
+ roster, err := s.Directory.Read(ctx)
+ if err != nil {
+ return hostclaim.Observed{}, err
+ }
+ loss := a.Loss
+ b, ok := roster.Roster.Boxes[loss.Box]
+ if !ok || b.Enrollment != loss.Enrollment || b.State == "retired" || b.Cloud.Account != loss.CloudObservation.Account || b.Cloud.Region != loss.CloudObservation.Region || b.Cloud.InstanceID != loss.CloudObservation.InstanceID {
+ return hostclaim.Observed{}, ErrResolve
+ }
+ if !slices.Contains(b.LossIDs, loss.ID) {
+ // Retry unrelated roster changes with a fresh CAS. A declaration for
+ // this same box defeats the accepted revision and needs new inspection.
+ if b.QuarantineRevision == math.MaxUint64 || b.QuarantineRevision+1 != loss.QuarantineRevision {
+ return hostclaim.Observed{}, ErrResolve
+ }
+ roster, err = s.Directory.Quarantine(ctx, s.Signer, roster, loss.Box, loss.Enrollment, loss.ID)
+ if err != nil {
+ return hostclaim.Observed{}, err
+ }
+ b = roster.Roster.Boxes[loss.Box]
+ }
+ if b.State != "quarantined" || b.QuarantineRevision < loss.QuarantineRevision {
+ return hostclaim.Observed{}, ErrResolve
+ }
+ // The global deny remains in force through every subsequent failure.
prepared, err := hostclaim.CreateLossRecord(ctx, s.Directory.Store, s.Signer, loss)
if err != nil {
return hostclaim.Observed{}, err
}
return s.finish(ctx, prepared)
}
+
func (s Service) finish(ctx context.Context, loss hostclaim.ObservedLossRecord) (hostclaim.Observed, error) {
observed, err := hostclaim.Read(ctx, s.Directory.Store, loss.Record.Host)
if err != nil {
@@ -6318,8 +6845,8 @@ or cloud resource was changed while writing this plan.
## Authoring verification
-Rev 2 was reconstructed from this document's 42 complete Go files and five
-patches covering 12 existing files in a fresh disposable export of
+Rev 3 was reconstructed from this document's 42 complete Go files and five
+patches covering 12 existing files in a disposable export of
`ca5a79d8a3821172f5829231014969f106bc33e9`. Every printed Go file matched
its compiled source byte for byte, and every patch applied to that parent.
No implementation files or branch were created in the design worktree.
@@ -6329,52 +6856,51 @@ These checks passed on the final reconstruction:
```sh
go build -buildvcs=false ./...
go test -buildvcs=false -exec=/bin/true ./...
-go vet -buildvcs=false ./...
-go test -buildvcs=false -race -count=1 ./internal/hostactors ./internal/hostfloor ./internal/hostfence ./internal/hostresolve ./internal/hostexec
-go test -buildvcs=false ./internal/host -run 'TestFloorRefusal|TestGuardRecovery|TestUnpublishedCompletion|TestApply|TestRecovery' -count=1
-go test -buildvcs=false ./internal/console -run '^TestHostClaimRoutesArePersonOperatorFlagGatedAndInert$' -count=1
-go test -buildvcs=false . -run 'TestHostClaimFlag|TestParseHostResolve' -count=1
-GOOS=darwin GOARCH=amd64 go build -buildvcs=false ./...
-GOOS=windows GOARCH=amd64 go build -buildvcs=false ./...
-GOOS=linux GOARCH=arm64 go build -buildvcs=false ./...
+go test -buildvcs=false ./internal/hostactors ./internal/hostfloor ./internal/hostresolve ./internal/hostfence ./internal/hostexec -count=1
+go test -buildvcs=false ./internal/host -run 'TestFloorRefusal|TestGuardRecovery|TestUnpublishedCompletion' -count=1
gofmt -l .
```
`-exec=/bin/true` compiles every test package without executing its tests.
-The runtime checks above are targeted; the full repository runtime gate
-remains an implementation-branch requirement. The revised Markdown also
-passes `git diff --check`, has balanced code fences and contains no em dashes.
-
-Regression tests cover active-roster fence exclusion from census and
-eligibility, invalid or unreadable fence authority, exact console process
-and compute binding, unknown placement scope, expired and superseded
-confirmation refresh, replay after consumption, every retained update-guard
-crash state, unpublished completion, and a controller resumed after an
-update with the wrong running image or floor. L20 tests include the newest
-stale console report, changed floor, missing guard, incompatible image,
-malformed newest evidence, future timestamps and ordinary-start refusal.
-The fence-reference test checks enrollment, exact key and byte digest.
-
-Deliberate regressions were watched fail their behavioral assertions:
-ignoring a valid termination fence, omitting placement-scope validation,
-omitting enrollment from the returned evidence reference, restoring the old
-update recovery/completion bodies, and restoring marker-only controller
-checks. Each mutation was removed before final verification. The paired
-journal and automatic-loss persistence tests remain explicit requirements
-of the operation-owner integration described above.
+The runtime checks above are targeted. The full repository runtime gate,
+race checks and cross-platform builds remain implementation-branch gates;
+this revision does not claim to have rerun rev 2's broader checks.
+
+The round 2 regressions are:
+
+- `TestStartRevalidatesReservedActorEligibility`: reserve, then quarantine,
+ retire or terminate a participant or bound console compute box. Start
+ refuses before invoking the transition and leaves the claim unchanged.
+ Malformed and unreadable fence authority also refuse.
+- `TestFleetInvitationBindsReferenceToEnrollment`: generate enrollment,
+ derive the fleet reference, bind the unchanged signed cloud evidence,
+ retry the invitation and complete enrollment. Arbitrary references and
+ mismatched fleet evidence refuse.
+- `TestGuardRecoveryRefusesControllersAfterBegin`: pending, active, lost,
+ malformed and unreadable controller state after the initial update scan
+ refuse journaled recovery and both unfinished no-journal paths. No binary,
+ journal, marker or completion change occurs.
+- `TestDeclareCrashAndChallengeExpiryAllowRetryAndReadmission`: interrupt
+ before or after quarantine and before loss creation, expire the challenge,
+ retry from durable acceptance, then complete readmission. The original
+ acceptance and fence times survive. Failure to persist acceptance leaves
+ no quarantine, and an unaccepted expired submission still refuses. An
+ unrelated roster CAS does not strand accepted evidence.
+
+Removing each corresponding fix was watched fail its behavioral regression.
+Every mutation was removed before the final build and runtime checks. The
+Markdown passes `git diff --check`, has balanced fences and no em dashes.
The default Go cache was attempted and was read-only in this sandbox.
-Verification reused the existing shared on-disk cache by setting
-`GOCACHE=/home/paulca/.cache/tmp/gocache`; it created no per-task Go cache
-and used no RAM-backed `/tmp` scratch. Disposable exports and authoring
-scratch were removed after verification. The shared cache was retained.
-
-An independent GPT-6 Astra review checked the revised roster, update guard
-and automatic-fence integration contracts. Its console report identity
-finding was fixed and covered by regression tests; no findings remained in
-that reviewed scope. Opus was unavailable in this session, so this is not an
-Opus review. The repository's required authority review before implementation
-lands, and the named cross-branch activation dependencies, remain gates.
-This task revised only the requested Markdown plan, staged nothing and
-committed nothing. No production resource, floor, feature flag or service
-was changed.
+Verification reused the existing shared on-disk cache with
+`GOCACHE=/home/paulca/.cache/tmp/gocache`, created no per-task Go cache and
+used no RAM-backed `/tmp` scratch. Disposable exports and authoring scratch
+were removed after verification. The shared cache was retained.
+
+An independent GPT-6 Astra review checked the four authority-related fixes.
+Its finding about acceptance retry after an unrelated roster CAS was fixed
+and covered by the crash regression. Opus was unavailable through the
+collaboration tool, so this does not replace the repository's required Opus
+review before implementation lands. Cross-branch activation dependencies
+remain gates. Only this Markdown plan changed; nothing was staged or
+committed, and no production resource, floor, flag or service changed.