amadan
plan: claim Plan 8 rev 2, answers the independent review (one P1, two P2)
Co-Authored-By: Codex <noreply@openai.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
1 file changed,
+670
−58
docs/superpowers/plans/2026-10-05-host-lifecycle-claim-8-leases-floor.md+670 −58
CI passed — run details
diff --git a/docs/superpowers/plans/2026-10-05-host-lifecycle-claim-8-leases-floor.md b/docs/superpowers/plans/2026-10-05-host-lifecycle-claim-8-leases-floor.md| index 3ad056e76..c7d2ff41c 100644 |
| --- a/docs/superpowers/plans/2026-10-05-host-lifecycle-claim-8-leases-floor.md |
| +++ b/docs/superpowers/plans/2026-10-05-host-lifecycle-claim-8-leases-floor.md |
| @@ -1,6 +1,8 @@ |
| # Host Lifecycle Claim, Plan 8: inert lease and lost-box contracts |
| -Rev 1 follows claim spec **rev 6.17**, §7.3 D2–D6. These contracts remain **PROPOSED**. Paul has not ruled on P1–P3 in §10.1 and amadan discussion **#236**. Landing these libraries does not authorize activation. |
| +Rev 2 answers the independent review (one P1, two P2). |
| + |
| +This plan follows claim spec **rev 6.20**, §7.3 D2–D6. These contracts remain **PROPOSED**. Paul has not ruled on P1–P3 in §10.1 and amadan discussion **#236**. Landing these libraries does not authorize activation. |
| > **For agentic workers:** implement the checked tasks in order. This document contains complete new files and exact patches for existing files. The branch and gate commands belong to the later implementation session. Authoring this plan changes only this Markdown file: no branch change, staging, commit or production action. |
| @@ -19,7 +21,7 @@ Use the repository's reviewed-branch workflow when the implementation is ready. |
| ## Sources and boundaries |
| -- `docs/superpowers/specs/2026-10-02-host-lifecycle-claim-design.md`, rev 6.17: §2, C1–C6, §6.1, §6.3, operative §6.4a, §7.1–§7.3 and §10–§10.1. §7.3 supplies the proposed claim contracts; §10.1 records the still-open approval boundaries. |
| +- `docs/superpowers/specs/2026-10-02-host-lifecycle-claim-design.md`, rev 6.20: §2, C1–C6, §6.1, §6.3, operative §6.4a, §7.1–§7.3 and §10–§10.1. §7.3 supplies the proposed claim contracts; §10.1 records the still-open approval boundaries. |
| - Plans 2, 3, 4, 7 and 7b in this directory supply the storage, local guard, retirement and consumer conventions. Plan 3's R9 applies to every local lease file and lock. Plan 4 owns immutable parking scopes and the late-completion horizon; this plan preserves those journals and retirement identities. |
| - Compile baseline: `feat/hostclaim-boxguard` at `8c40aca61d6b25ea5e936ffab171d237f1b13483`, plus the disjoint `origin/main` changes at `562f3dfcb4934c0dec027f730e34bbff08bbbd15`. Existing `hostclaim.Update` clones the observation, checks its ETag and assigns the next claim generation. `CompleteObligation` handles discovery completion using the chore's original operation ID. |
| - `internal/lease/{lease,health,endpoint}.go`: replica leases use a different object and 15-minute TTL. Reuse their configured S3 client, not their wire encoding or TTL. `internal/release/{store,conditional_delete}.go` supplies create-only PUT, conditional replacement and conditional deletion. |
| @@ -30,13 +32,15 @@ Use the repository's reviewed-branch workflow when the implementation is ready. |
| 1. **R1:** with `CARLOS_HOST_CLAIM` unset, existing production paths stay unchanged. The CLI and console snapshot trimmed nonempty flag semantics; `0` is enabled, whitespace is disabled. The only new live route returns `awaiting Paul's ruling P2/P3 (amadan #236)` after person/operator authorization. It never creates a challenge, changes a claim, writes a loss record or invokes EC2. Libraries have no production callers here. |
| 2. **D6:** the remote object is exactly `control/hosts/<host>/exec/<box>.json` with `op` and `expires`. TTL is 90 seconds, renewal 20 seconds, denial margin five seconds. The deadline starts with the request, includes suspension through Linux `CLOCK_BOOTTIME`, and never extends from an ambiguous response or wall-clock rollback. A lost handle never revives. |
| -3. The lifetime lock is `/run/carlos/hostexec/<host>.lock`; local state is `<DataDir>/.hostexec/<host>.json`. The host guard serializes record changes. Libraries never create directories or change pathname ownership/mode. Provision the directories with the same setgid `2770 carlos carlos` boundary as Plan 3 before connecting callers. Tests provision only their scratch directories. Darwin and Windows compile; an unsupported suspend-aware clock refuses controller startup. |
| -4. Lease network requests and claim acknowledgements never hold the host guard. Denial and cleanup run independently of renewal and persistence. Cleanup must terminate the captured host processes without waiting for that guard. An uncertain active-file write retains the guard until a durable lost record is written; a timed-out `Close` retains controller exclusion. Independent writers use `CheckLocal` **inside** `Guard.With`, alongside fresh claim/quarantine checks and their separate cleanup-denial state. A file left by a dead controller cannot authorize work. |
| +3. The lifetime lock is `/run/carlos/hostexec/<host>.lock`; a separate `/run/carlos/hostexec/<host>.live` lock is held only while the controller has not lost its lease. Local state is `<DataDir>/.hostexec/<host>.json`. The host guard serializes record changes. Libraries never create directories or change pathname ownership/mode. Provision the directories with the same setgid `2770 carlos carlos` boundary as Plan 3 before connecting callers. Tests provision only their scratch directories. Darwin and Windows compile; an unsupported suspend-aware clock refuses controller startup. |
| +4. Lease network requests and claim acknowledgements never hold the host guard. Denial and cleanup run independently of renewal and persistence. Cleanup must terminate the captured host processes without waiting for that guard. Ambiguity revokes the live lock before signalling loss, so `CheckLocal` refuses even when an older `active` file still has a future deadline. Both an uncertain active-file write and a failed lost-state write retain the guard and retry until denial is durable; a timed-out `Close` retains controller exclusion. Independent writers use `CheckLocal` **inside** `Guard.With`, alongside fresh claim/quarantine checks and their separate cleanup-denial state. A file left by a dead controller cannot authorize work. |
| 5. CC9 clears only the matching local lease record after a successful fresh claim read proves serving, free or exclusion of that box. It preserves quarantine, retired stamps and cleanup failures. `Ready` returns a coherent durable acknowledgement tuple. The `exec_ack.<box>` journal entry is historical evidence; it is not permission to keep working after lease loss. |
| -6. **D4/D5:** a loss record is prepared durably before a claim CAS. A declaration binds holder, operation, phase, enrolled box, claim revision and confirmed effects. The transition revokes only that box, sets its executor to `none` while preserving `holder.stamp`, retains remote obligations, and uses `CompleteObligation` only for exact destroyed-local scopes. Stopped or terminated compute alone never proves that disks were destroyed. Lost revisions and declaration references survive free/reuse; readmission removes only the matching revision and grants no writer. |
| +6. **D4/D5:** a loss record is prepared durably before a claim CAS. A declaration binds holder, operation, phase, enrolled box, claim revision and confirmed effects. The transition revokes only that box, sets its executor to `none` while preserving `holder.stamp`, retains remote obligations, and uses `CompleteObligation` only for exact destroyed-local scopes. Stopped or terminated compute alone never proves that disks were destroyed. A freshly bound second declaration must have a strictly newer quarantine revision; its consequences do not change the first declaration. Lost revisions and declaration references survive free/reuse; readmission removes only the matching revision and grants no writer. A serving claim with empty `op.id` accepts only destroyed-local-only consequences bound to the current claim generation/ETag and each chore’s original operation and retirement identity. This path preserves the entire serving operation and holder, including executor, stamp and sessions. Its only claim effects are exact local-chore discharge, normal discovery completion and required loss/quarantine bookkeeping; it never writes a serving-operation recovery journal or reassigns regional work. |
| 7. Missing or unreadable evidence refuses. Timers explicitly distinguish `complete`, `partial` and `unknown`; unknown never becomes an empty complete inventory. Additional write loss belongs to the same admission, with the original source-cut session and validated baseline position. Durable archival precedes removal of loss-bearing admission evidence. |
| 8. **D2/D3:** admission checks freshly consult the directory interface on every call. The real root-signed directory, startup retirement execution, global quarantine CAS and readmission coordinator are outside this scope. Cloud verification accepts only one exact enrolled instance, a matching reservation owner, and `stopped` or `terminated`. Missing, paginated/partial, ambiguous and failed responses refuse. The challenge expires in 120 seconds; observations start after issuance, are at most 60 seconds old and at most five seconds in the console's future. The signed challenge authenticates console intent, not AWS evidence. |
| +The claim spec’s proposed **§7.3 D3/D5** text must state the same rules: strictly newer, freshly bound successive declarations; matching-revision readmission; the serving destroyed-local-only exception; and durable lease denial before releasing the guard, with independent cleanup. Rev 6.20 already contains the review’s additions. This plan does not edit the spec or turn the proposals into rulings. |
| + |
| Each task below includes its implementation and tests. Save each existing-file patch and apply it with `git apply --unidiff-zero` against the stated baseline; context is omitted to keep Go tabs intact inside Markdown. Apply patches exactly; do not substitute prose for omitted code. Source filenames above are baseline references; the complete additions and changes follow. |
| ## Task 1: Signed loss records, additive claim state and bound transitions |
| @@ -587,10 +591,18 @@ func safeLossID(id string) bool { |
| func (r LossRecord) Validate() error { |
| if r.V != lossRecordVersion || !safeLossID(r.ID) || (r.State != "prepared" && r.State != "committed" && r.State != "conflicted") || |
| r.Actor == "" || r.AcceptedAt.IsZero() || r.Host == "" || !isHex64(r.HolderIncarnation) || r.HolderStamp == 0 || r.ExpectedClaimGen == 0 || |
| - r.ExpectedClaimETag == "" || r.OpID == "" || !knownKinds[r.OpKind] || r.Box == "" || r.Enrollment == "" || |
| + r.ExpectedClaimETag == "" || !knownKinds[r.OpKind] || r.Box == "" || r.Enrollment == "" || |
| r.QuarantineRevision == 0 || r.FencedAt.IsZero() { |
| return ErrLossRecord |
| } |
| + if r.OpKind == KindServing { |
| + if r.OpID != "" || r.OpPhase != "" || !destroyedLocalOnly(r.ConfirmedConsequences) || |
| + len(r.WriteSources) != 0 || len(r.ArchivedAdmissionLosses) != 0 { |
| + return ErrLossRecord |
| + } |
| + } else if r.OpID == "" { |
| + return ErrLossRecord |
| + } |
| if _, err := Key(r.Host); err != nil { |
| return ErrLossRecord |
| } |
| @@ -630,6 +642,29 @@ func (r LossRecord) Validate() error { |
| return nil |
| } |
| +func destroyedLocalOnly(c LossConsequences) bool { |
| + if !validConsequences(c) || c.RevokeExecution || c.RemoveParticipation || len(c.ReassignRegional) != 0 || |
| + c.WritesPotentiallyLost || c.TimersPotentiallyLost || len(c.DischargeLocal) == 0 || |
| + len(c.DestroyedStorage) != len(c.DischargeLocal) { |
| + return false |
| + } |
| + intent, err := canonicalRaw(c.RecoveryIntent) |
| + if err != nil || intent != `{"kind":"destroyed-local-only"}` { |
| + return false |
| + } |
| + want := map[string]bool{} |
| + for _, ref := range c.DischargeLocal { |
| + want[ref.Op+"/"+ref.ID+"/"+ref.Retirement] = true |
| + } |
| + for _, assertion := range c.DestroyedStorage { |
| + if !want[assertion] { |
| + return false |
| + } |
| + delete(want, assertion) |
| + } |
| + return len(want) == 0 |
| +} |
| + |
| func validConsequences(c LossConsequences) bool { |
| if len(c.RecoveryIntent) == 0 || !json.Valid(c.RecoveryIntent) { |
| return false |
| @@ -1223,6 +1258,9 @@ func declareLostTransition(in LostDeclaration, owner LossOperationOwner) func(*C |
| c.Op.Kind != in.OpKind || c.Op.Phase != in.OpPhase { |
| return fmt.Errorf("%w: loss declaration no longer matches the holder and operation", ErrStaleOp) |
| } |
| + if old := c.LostRevisions[in.Box]; old != 0 && in.QuarantineRevision <= old { |
| + return fmt.Errorf("%w: declaration requires a strictly newer quarantine revision", ErrWrongState) |
| + } |
| if lossRefPresent(c, in.LossID) { |
| return ErrLossAlreadyApplied |
| } |
| @@ -1243,6 +1281,23 @@ func declareLostTransition(in LostDeclaration, owner LossOperationOwner) func(*C |
| return err |
| } |
| + if c.Op.Kind == KindServing { |
| + if in.OpID != "" || in.OpPhase != "" || !destroyedLocalOnly(in.Consequences) { |
| + return ErrLossRecord |
| + } |
| + // No operation journal, authority, regional reassignment or holder |
| + // mutation is permitted in this branch. Original chore identities |
| + // still drive discovery completion after adoption cleared op.id. |
| + if err := dischargeDestroyedLocal(c, in); err != nil { |
| + return err |
| + } |
| + recordLost(c, in) |
| + return nil |
| + } |
| + if in.OpID == "" { |
| + return ErrLossRecord |
| + } |
| + |
| isExec := c.Op.Exec == in.Box |
| isParticipant := stringPresent(c.Op.Participants, in.Box) |
| if isExec != in.Consequences.RevokeExecution || isParticipant != in.Consequences.RemoveParticipation { |
| @@ -1290,31 +1345,39 @@ func declareLostTransition(in LostDeclaration, owner LossOperationOwner) func(*C |
| } |
| o.Owner = "region:" + o.Region |
| } |
| - for _, ref := range in.Consequences.DischargeLocal { |
| - o, ok := exactObligation(c, ref) |
| - if !ok || o.Kind != "park-local" || o.Box != in.Box || !destroyedStorageCovers(in, ref) { |
| - return fmt.Errorf("%w: local obligation %q lacks exact destroyed-storage evidence", ErrWrongState, ref.ID) |
| - } |
| - if err := CompleteObligation(ref.Op, ref.ID)(c); err != nil { |
| - return err |
| - } |
| + if err := dischargeDestroyedLocal(c, in); err != nil { |
| + return err |
| } |
| if c.Holder.Stamp != stamp || c.Holder.Session != session { |
| return fmt.Errorf("%w: declare-lost changed installation authority", ErrMalformed) |
| } |
| - if !stringPresent(c.Lost, in.Box) { |
| - c.Lost = append(c.Lost, in.Box) |
| - } |
| - if c.LostRevisions == nil { |
| - c.LostRevisions = map[string]uint64{} |
| + recordLost(c, in) |
| + return nil |
| + } |
| +} |
| + |
| +func recordLost(c *Claim, in LostDeclaration) { |
| + if !stringPresent(c.Lost, in.Box) { |
| + c.Lost = append(c.Lost, in.Box) |
| + } |
| + if c.LostRevisions == nil { |
| + c.LostRevisions = map[string]uint64{} |
| + } |
| + c.LostRevisions[in.Box] = in.QuarantineRevision |
| + c.LossRefs = append(c.LossRefs, in.LossID) |
| +} |
| + |
| +func dischargeDestroyedLocal(c *Claim, in LostDeclaration) error { |
| + for _, ref := range in.Consequences.DischargeLocal { |
| + o, ok := exactObligation(c, ref) |
| + if !ok || o.Kind != "park-local" || o.Box != in.Box || !destroyedStorageCovers(in, ref) { |
| + return fmt.Errorf("%w: local obligation %q lacks exact destroyed-storage evidence", ErrWrongState, ref.ID) |
| } |
| - if old := c.LostRevisions[in.Box]; old != 0 && old != in.QuarantineRevision { |
| - return fmt.Errorf("%w: box has a different quarantine revision", ErrWrongState) |
| + if err := CompleteObligation(ref.Op, ref.ID)(c); err != nil { |
| + return err |
| } |
| - c.LostRevisions[in.Box] = in.QuarantineRevision |
| - c.LossRefs = append(c.LossRefs, in.LossID) |
| - return nil |
| } |
| + return nil |
| } |
| func destroyedStorageCovers(in LostDeclaration, ref ObligationRef) bool { |
| @@ -1474,9 +1537,13 @@ func archiveAdmissionLossesTransition(lossID string, archived []ArchivedAdmissio |
| package hostclaim |
| import ( |
| + "bytes" |
| "context" |
| "crypto/ed25519" |
| + "encoding/json" |
| "errors" |
| + "fmt" |
| + "reflect" |
| "testing" |
| "time" |
| @@ -1629,6 +1696,251 @@ func TestDeclareLostFailsClosedWithoutOperationOwnerOrExactDiskProof(t *testing. |
| t.Fatalf("refusal mutated claim: %+v", fresh.Claim) |
| } |
| } |
| + |
| +func lossForObservation(t *testing.T, obs Observed, id string, revision uint64, box string, effects LossConsequences) LossRecord { |
| + t.Helper() |
| + r := validLossRecord(t) |
| + r.ID, r.Host = id, obs.Host |
| + r.HolderIncarnation, r.HolderStamp, r.HolderSession = obs.Claim.Holder.Incarnation, obs.Claim.Holder.Stamp, obs.Claim.Holder.Session |
| + r.ExpectedClaimGen, r.ExpectedClaimETag = obs.Claim.Gen, obs.ETag |
| + r.OpID, r.OpKind, r.OpPhase = obs.Claim.Op.ID, obs.Claim.Op.Kind, obs.Claim.Op.Phase |
| + r.Box, r.QuarantineRevision = box, revision |
| + r.ConfirmedConsequences = effects |
| + r.WriteSources = nil |
| + r.OperatorAttestation.DestroyedStorage = append([]string(nil), effects.DestroyedStorage...) |
| + return r |
| +} |
| + |
| +func TestDeclareLostNewerRevisionAppliesSecondConsequences(t *testing.T) { |
| + ctx := t.Context() |
| + s := release.NewDirStore(t.TempDir()) |
| + pub, priv, err := ed25519.GenerateKey(nil) |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + obs := servingClaim(t, ctx, s) |
| + obs, err = Update(ctx, s, obs, func(c *Claim) error { |
| + c.Op = Op{Kind: KindDeleting, ID: "delete", Phase: "parking", Exec: "box-a", Participants: []string{"box-a"}} |
| + c.Obligations = []Obligation{{ID: "local", Op: "delete", Kind: "park-local", Box: "box-a", Retirement: "ret", Owner: "box:box-a"}} |
| + return nil |
| + }) |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + first := lossForObservation(t, obs, "stopped", 9, "box-a", LossConsequences{ |
| + RevokeExecution: true, RemoveParticipation: true, RecoveryIntent: []byte(`{"kind":"wait-for-storage"}`), |
| + }) |
| + if _, err := CreateLossRecord(ctx, s, release.RootSigner(priv), first); err != nil { |
| + t.Fatal(err) |
| + } |
| + stopped, err := DeclareLost(ctx, s, pub, obs, first.ID, testLossOwner{}) |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + if len(stopped.Claim.Obligations) != 1 { |
| + t.Fatal("compute stop discharged local storage") |
| + } |
| + second := lossForObservation(t, stopped, "destroyed", 10, "box-a", LossConsequences{ |
| + DischargeLocal: []ObligationRef{{ID: "local", Op: "delete", Retirement: "ret"}}, |
| + DestroyedStorage: []string{"delete/local/ret"}, RecoveryIntent: []byte(`{"kind":"destroyed-storage"}`), |
| + }) |
| + // A new revision does not excuse stale claim generation or ETag bindings. |
| + stale := second |
| + stale.ID, stale.ExpectedClaimGen, stale.ExpectedClaimETag = "stale", obs.Claim.Gen, obs.ETag |
| + if _, err := CreateLossRecord(ctx, s, release.RootSigner(priv), stale); err != nil { |
| + t.Fatal(err) |
| + } |
| + if _, err := DeclareLost(ctx, s, pub, stopped, stale.ID, testLossOwner{}); !errors.Is(err, ErrConflict) { |
| + t.Fatalf("stale second binding = %v", err) |
| + } |
| + if _, err := CreateLossRecord(ctx, s, release.RootSigner(priv), second); err != nil { |
| + t.Fatal(err) |
| + } |
| + destroyed, err := DeclareLost(ctx, s, pub, stopped, second.ID, testLossOwner{}) |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + if len(destroyed.Claim.Obligations) != 0 || destroyed.Claim.LostRevisions["box-a"] != 10 || |
| + !reflect.DeepEqual(destroyed.Claim.LossRefs, []string{"stopped", "destroyed"}) { |
| + t.Fatalf("second declaration not applied: %+v", destroyed.Claim) |
| + } |
| + for _, rev := range []uint64{8, 9, 10} { |
| + r := lossForObservation(t, destroyed, fmt.Sprintf("older_%d", rev), rev, "box-a", LossConsequences{RecoveryIntent: []byte(`{}`)}) |
| + if _, err := CreateLossRecord(ctx, s, release.RootSigner(priv), r); err != nil { |
| + t.Fatal(err) |
| + } |
| + if _, err := DeclareLost(ctx, s, pub, destroyed, r.ID, testLossOwner{}); !errors.Is(err, ErrWrongState) { |
| + t.Fatalf("revision %d accepted: %v", rev, err) |
| + } |
| + } |
| + if _, err := Update(ctx, s, destroyed, Readmit("box-a", 9)); !errors.Is(err, ErrWrongState) { |
| + t.Fatalf("old readmission cleared newer quarantine: %v", err) |
| + } |
| + for _, id := range []string{first.ID, second.ID} { |
| + retry, err := DeclareLost(ctx, s, pub, destroyed, id, testLossOwner{}) |
| + if err != nil || retry.ETag != destroyed.ETag { |
| + t.Fatalf("replay %s = %v", id, err) |
| + } |
| + } |
| + unchanged, err := ReadLossRecord(ctx, s, pub, first.ID) |
| + if err != nil || !reflect.DeepEqual(unchanged.Record, first) { |
| + t.Fatal("first declaration was rewritten", err) |
| + } |
| + fresh, err := Read(ctx, s, obs.Host) |
| + if err != nil || fresh.ETag != destroyed.ETag { |
| + t.Fatal("refusal or replay mutated claim", err) |
| + } |
| + readmitted, err := Update(ctx, s, fresh, Readmit("box-a", 10)) |
| + if err != nil || len(readmitted.Claim.Lost) != 0 || len(readmitted.Claim.LossRefs) != 2 { |
| + t.Fatal("matching readmission", err) |
| + } |
| +} |
| + |
| +func TestServingDestroyedLocalOnlyPreservesAuthorityAndOtherChores(t *testing.T) { |
| + for _, otherChores := range []bool{false, true} { |
| + t.Run(fmt.Sprint(otherChores), func(t *testing.T) { |
| + ctx := t.Context() |
| + s := release.NewDirStore(t.TempDir()) |
| + pub, priv, err := ed25519.GenerateKey(nil) |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + obs, err := Read(ctx, s, testHost) |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + var regions []string |
| + boxes := []string{"box-a"} |
| + if otherChores { |
| + regions = []string{"uk"} |
| + boxes = append(boxes, "other-box") |
| + } |
| + obs, err = Update(ctx, s, obs, Adopt(newHolder(), "serving-box", regions, boxes, "discovery", counter())) |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + if obs.Claim.Op.Kind != KindServing || obs.Claim.Op.ID != "" { |
| + t.Fatal("fixture is not an adopted serving claim") |
| + } |
| + var ref ObligationRef |
| + for _, o := range obs.Claim.Obligations { |
| + if o.Kind == "park-local" && o.Box == "box-a" { |
| + ref = ObligationRef{ID: o.ID, Op: o.Op, Retirement: o.Retirement} |
| + } |
| + } |
| + effects := LossConsequences{DischargeLocal: []ObligationRef{ref}, DestroyedStorage: []string{ref.Op + "/" + ref.ID + "/" + ref.Retirement}, RecoveryIntent: []byte(`{"kind":"destroyed-local-only"}`)} |
| + r := lossForObservation(t, obs, "local_only", 11, "box-a", effects) |
| + if _, err := CreateLossRecord(ctx, s, release.RootSigner(priv), r); err != nil { |
| + t.Fatal(err) |
| + } |
| + got, err := DeclareLost(ctx, s, pub, obs, r.ID, testLossOwner{mutate: true}) |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + want, err := cloneClaim(obs.Claim) |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + remaining := make([]Obligation, 0, len(want.Obligations)-1) |
| + for _, o := range want.Obligations { |
| + if o.ID != ref.ID { |
| + remaining = append(remaining, o) |
| + } |
| + } |
| + want.Obligations = remaining |
| + if !otherChores { |
| + want.Touched.Complete, want.Touched.DiscoveryOp = true, "" |
| + } |
| + want.Gen = got.Claim.Gen |
| + want.Lost, want.LostRevisions, want.LossRefs = []string{"box-a"}, map[string]uint64{"box-a": 11}, []string{r.ID} |
| + // Compare serialized claims so nil/empty omitted arrays are equivalent. |
| + wantBytes, _ := json.Marshal(want) |
| + gotBytes, _ := json.Marshal(got.Claim) |
| + if !bytes.Equal(wantBytes, gotBytes) { |
| + t.Fatalf("local-only changed unrelated state:\nwant %s\ngot %s", wantBytes, gotBytes) |
| + } |
| + if retry, err := DeclareLost(ctx, s, pub, got, r.ID, testLossOwner{}); err != nil || retry.ETag != got.ETag { |
| + t.Fatal("local-only replay", err) |
| + } |
| + }) |
| + } |
| +} |
| + |
| +func TestServingDestroyedLocalOnlyRefusesWideningAndStaleBindings(t *testing.T) { |
| + ctx := t.Context() |
| + s := release.NewDirStore(t.TempDir()) |
| + pub, priv, err := ed25519.GenerateKey(nil) |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + obs, err := Read(ctx, s, testHost) |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + obs, err = Update(ctx, s, obs, Adopt(newHolder(), "serving-box", nil, []string{"box-a"}, "discovery", counter())) |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + o := obs.Claim.Obligations[0] |
| + ref := ObligationRef{ID: o.ID, Op: o.Op, Retirement: o.Retirement} |
| + base := lossForObservation(t, obs, "local", 1, "box-a", LossConsequences{DischargeLocal: []ObligationRef{ref}, DestroyedStorage: []string{ref.Op + "/" + ref.ID + "/" + ref.Retirement}, RecoveryIntent: []byte(`{"kind":"destroyed-local-only"}`)}) |
| + for name, mutate := range map[string]func(*LossRecord){ |
| + "execution": func(r *LossRecord) { r.ConfirmedConsequences.RevokeExecution = true }, |
| + "participation": func(r *LossRecord) { r.ConfirmedConsequences.RemoveParticipation = true }, |
| + "regional": func(r *LossRecord) { r.ConfirmedConsequences.ReassignRegional = []ObligationRef{ref} }, |
| + "writes": func(r *LossRecord) { r.ConfirmedConsequences.WritesPotentiallyLost = true }, |
| + "timers": func(r *LossRecord) { r.ConfirmedConsequences.TimersPotentiallyLost = true }, |
| + "recovery": func(r *LossRecord) { |
| + r.ConfirmedConsequences.RecoveryIntent = []byte(`{"kind":"destroyed-local-only","destination":"other"}`) |
| + }, |
| + "empty": func(r *LossRecord) { r.ConfirmedConsequences.DischargeLocal = nil }, |
| + "missing-proof": func(r *LossRecord) { |
| + r.ConfirmedConsequences.DestroyedStorage = nil |
| + r.OperatorAttestation.DestroyedStorage = nil |
| + }, |
| + "op": func(r *LossRecord) { r.OpID = "discovery" }, |
| + "phase": func(r *LossRecord) { r.OpPhase = "recover" }, |
| + "non-serving-empty-op": func(r *LossRecord) { r.OpKind = KindDeleting }, |
| + } { |
| + t.Run(name, func(t *testing.T) { |
| + r := base |
| + r.ID = name |
| + mutate(&r) |
| + if _, err := CreateLossRecord(ctx, s, release.RootSigner(priv), r); !errors.Is(err, ErrLossRecord) { |
| + t.Fatalf("widened serving declaration = %v", err) |
| + } |
| + }) |
| + } |
| + for name, mutate := range map[string]func(*LossRecord){ |
| + "claim-gen": func(r *LossRecord) { r.ExpectedClaimGen++ }, |
| + "claim-etag": func(r *LossRecord) { r.ExpectedClaimETag = "stale" }, |
| + "stamp": func(r *LossRecord) { r.HolderStamp++ }, |
| + "session": func(r *LossRecord) { r.HolderSession++ }, |
| + "box": func(r *LossRecord) { r.Box = "other-box" }, |
| + "original-op": func(r *LossRecord) { r.ConfirmedConsequences.DischargeLocal[0].Op = "other-op" }, |
| + "retirement": func(r *LossRecord) { r.ConfirmedConsequences.DischargeLocal[0].Retirement = "other-retirement" }, |
| + } { |
| + t.Run(name, func(t *testing.T) { |
| + r := base |
| + r.ID = name |
| + r.ConfirmedConsequences.DischargeLocal = append([]ObligationRef(nil), base.ConfirmedConsequences.DischargeLocal...) |
| + mutate(&r) |
| + ref := r.ConfirmedConsequences.DischargeLocal[0] |
| + r.ConfirmedConsequences.DestroyedStorage = []string{ref.Op + "/" + ref.ID + "/" + ref.Retirement} |
| + r.OperatorAttestation.DestroyedStorage = r.ConfirmedConsequences.DestroyedStorage |
| + if _, err := CreateLossRecord(ctx, s, release.RootSigner(priv), r); err != nil { |
| + t.Fatal(err) |
| + } |
| + if _, err := DeclareLost(ctx, s, pub, obs, r.ID, testLossOwner{}); err == nil { |
| + t.Fatal("stale or wrong scope was accepted") |
| + } |
| + }) |
| + } |
| + fresh, err := Read(ctx, s, obs.Host) |
| + if err != nil || fresh.ETag != obs.ETag { |
| + t.Fatal("refusal changed serving claim", err) |
| + } |
| +} |
| ``` |
| - [ ] **Run the task tests.** |
| @@ -1640,7 +1952,7 @@ go test ./internal/release -run RootEnvelope -count=1 |
| ## Task 2: Quarantine interface, retirement receipt and exact compute observation |
| -**Produces:** `ActorDirectory`, a fresh admission gate, the returning-machine completion type, the read-only EC2 verifier and signed challenge types. **Consumes:** Task 1’s existing-envelope helpers and the existing `awsinv` EC2 read seam. The tests supply the only directory implementation. The challenge functions authenticate and validate the bound data; the live console still cannot issue or accept a declaration. |
| +**Produces:** `ActorDirectory`, a fresh admission gate, the returning-machine completion type, the read-only EC2 verifier and signed challenge types. **Consumes:** Task 1’s existing-envelope helpers and the existing `awsinv` EC2 read seam. The tests supply the only directory implementation. The challenge functions authenticate and validate the bound data; the live console still cannot issue or accept a declaration. A serving destroyed-local-only challenge uses structured `discharge_local` references, each carrying the original chore ID, operation and retirement identity. It binds the claim revision while leaving the current operation ID empty; generic chore strings and other recovery effects are refused for that case. |
| - [ ] **Create `internal/hostquarantine/quarantine.go` with this complete content.** |
| @@ -2051,6 +2363,7 @@ import ( |
| "time" |
| "github.com/carlosframework/platform/internal/awsinv" |
| + "github.com/carlosframework/platform/internal/hostclaim" |
| "github.com/carlosframework/platform/internal/release" |
| ) |
| @@ -2087,11 +2400,12 @@ type RecordObservation struct { |
| // Consequences is signed by the console so the confirmation cannot be |
| // replayed for a different transition or a different set of claimed losses. |
| type Consequences struct { |
| - Transition string `json:"transition"` |
| - Chores []string `json:"chores,omitempty"` |
| - PossibleWriteLoss bool `json:"possible_write_loss,omitempty"` |
| - PossibleTimerLoss bool `json:"possible_timer_loss,omitempty"` |
| - DestroyedStorageAssertion []string `json:"destroyed_storage_assertion,omitempty"` |
| + Transition string `json:"transition"` |
| + DischargeLocal []hostclaim.ObligationRef `json:"discharge_local,omitempty"` |
| + Chores []string `json:"chores,omitempty"` |
| + PossibleWriteLoss bool `json:"possible_write_loss,omitempty"` |
| + PossibleTimerLoss bool `json:"possible_timer_loss,omitempty"` |
| + DestroyedStorageAssertion []string `json:"destroyed_storage_assertion,omitempty"` |
| } |
| // Challenge is the console's exact, short-lived snapshot of the transition |
| @@ -2150,13 +2464,41 @@ func PrepareChallenge(now time.Time, c Challenge) (Challenge, error) { |
| func validateChallenge(c Challenge) error { |
| if c.ID == "" || c.Actor == "" || c.Host == "" || c.HolderIncarnation == "" || |
| - c.ClaimGen == 0 || c.ClaimETag == "" || c.Op.ID == "" || c.Op.Kind == "" || |
| + c.ClaimGen == 0 || c.ClaimETag == "" || c.Op.Kind == "" || |
| c.Record.Key == "" || c.Record.CommitState == "" || c.Box == "" || c.Enrollment == "" || |
| c.Cloud.Provider != "aws" || c.Cloud.Account == "" || c.Cloud.Region == "" || c.Cloud.InstanceID == "" || |
| c.Consequences.Transition == "" || c.IssuedAt.IsZero() || c.ExpiresAt.IsZero() || |
| !c.ExpiresAt.Equal(c.IssuedAt.Add(ChallengeTTL)) { |
| return fmt.Errorf("%w: incomplete or incorrectly timed challenge", ErrChallenge) |
| } |
| + if c.Op.Kind == hostclaim.KindServing || c.Consequences.Transition == "destroyed-local-only" { |
| + if c.Op.Kind != hostclaim.KindServing || c.Op.ID != "" || c.Op.Phase != "" || |
| + c.Consequences.Transition != "destroyed-local-only" || c.Consequences.PossibleWriteLoss || |
| + c.Consequences.PossibleTimerLoss || len(c.Consequences.Chores) != 0 || |
| + len(c.Consequences.DischargeLocal) == 0 || |
| + len(c.Consequences.DestroyedStorageAssertion) != len(c.Consequences.DischargeLocal) { |
| + return ErrChallenge |
| + } |
| + want := map[string]bool{} |
| + for _, ref := range c.Consequences.DischargeLocal { |
| + key := ref.Op + "/" + ref.ID + "/" + ref.Retirement |
| + if ref.ID == "" || ref.Op == "" || ref.Retirement == "" || want[key] { |
| + return ErrChallenge |
| + } |
| + want[key] = true |
| + } |
| + for _, assertion := range c.Consequences.DestroyedStorageAssertion { |
| + if !want[assertion] { |
| + return ErrChallenge |
| + } |
| + delete(want, assertion) |
| + } |
| + if len(want) != 0 { |
| + return ErrChallenge |
| + } |
| + } else if c.Op.ID == "" || len(c.Consequences.DischargeLocal) != 0 { |
| + return ErrChallenge |
| + } |
| if (c.Record.CommitState == "absent") != (c.Record.ETag == "") { |
| return fmt.Errorf("%w: record absence and ETag are inconsistent", ErrChallenge) |
| } |
| @@ -2253,7 +2595,11 @@ package hostfence |
| import ( |
| "context" |
| "crypto/ed25519" |
| + "encoding/base64" |
| + "encoding/json" |
| "errors" |
| + "github.com/carlosframework/platform/internal/hostclaim" |
| + "reflect" |
| "testing" |
| "time" |
| @@ -2396,6 +2742,88 @@ func TestValidateObservationFreshnessAndBinding(t *testing.T) { |
| } |
| } |
| } |
| + |
| +func TestServingDestroyedLocalOnlyChallengeBindsOriginalChore(t *testing.T) { |
| + now := time.Date(2026, 10, 5, 12, 0, 0, 0, time.UTC) |
| + pub, priv, err := ed25519.GenerateKey(nil) |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + c := challengeFixture(t, now) |
| + c.Op = Operation{Kind: hostclaim.KindServing} |
| + c.Consequences = Consequences{ |
| + Transition: "destroyed-local-only", |
| + DischargeLocal: []hostclaim.ObligationRef{{ID: "local", Op: "discovery", Retirement: "retired-installation"}}, |
| + DestroyedStorageAssertion: []string{"discovery/local/retired-installation"}, |
| + } |
| + wire, err := SignChallenge(release.RootSigner(priv), c) |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + got, err := VerifyChallenge(pub, wire, now.Add(time.Second)) |
| + if err != nil || !reflect.DeepEqual(got, c) { |
| + t.Fatalf("serving binding round trip = %+v %v", got, err) |
| + } |
| + for name, mutate := range map[string]func(*Challenge){ |
| + "generic": func(c *Challenge) { c.Consequences.Transition = "declare-lost" }, |
| + "non-serving": func(c *Challenge) { c.Op.Kind = "moving" }, |
| + "current-op": func(c *Challenge) { c.Op.ID = "discovery" }, |
| + "phase": func(c *Challenge) { c.Op.Phase = "recover" }, |
| + "claim-gen": func(c *Challenge) { c.ClaimGen = 0 }, |
| + "claim-etag": func(c *Challenge) { c.ClaimETag = "" }, |
| + "write-loss": func(c *Challenge) { c.Consequences.PossibleWriteLoss = true }, |
| + "timer-loss": func(c *Challenge) { c.Consequences.PossibleTimerLoss = true }, |
| + "unbound-chore": func(c *Challenge) { c.Consequences.Chores = []string{"extra"} }, |
| + "empty": func(c *Challenge) { c.Consequences.DischargeLocal = nil }, |
| + "original-op": func(c *Challenge) { c.Consequences.DischargeLocal[0].Op = "" }, |
| + "retirement": func(c *Challenge) { c.Consequences.DischargeLocal[0].Retirement = "" }, |
| + "proof": func(c *Challenge) { |
| + c.Consequences.DestroyedStorageAssertion = []string{"other/local/retired-installation"} |
| + }, |
| + } { |
| + t.Run(name, func(t *testing.T) { |
| + bad := c |
| + bad.Consequences.DischargeLocal = append([]hostclaim.ObligationRef(nil), c.Consequences.DischargeLocal...) |
| + mutate(&bad) |
| + if _, err := SignChallenge(release.RootSigner(priv), bad); !errors.Is(err, ErrChallenge) { |
| + t.Fatalf("widened challenge = %v", err) |
| + } |
| + }) |
| + } |
| + // The signature covers the exact revision and each original chore identity. |
| + for _, field := range []string{"claim_gen", "claim_etag", "op", "retirement"} { |
| + var envelope struct { |
| + Payload string `json:"payload"` |
| + Sig string `json:"sig"` |
| + } |
| + if err := json.Unmarshal(wire, &envelope); err != nil { |
| + t.Fatal(err) |
| + } |
| + var payload map[string]any |
| + raw, err := base64.StdEncoding.DecodeString(envelope.Payload) |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + if err := json.Unmarshal(raw, &payload); err != nil { |
| + t.Fatal(err) |
| + } |
| + switch field { |
| + case "claim_gen": |
| + payload[field] = float64(c.ClaimGen + 1) |
| + case "claim_etag": |
| + payload[field] = "other-etag" |
| + default: |
| + refs := payload["consequences"].(map[string]any)["discharge_local"].([]any) |
| + refs[0].(map[string]any)[field] = "other-identity" |
| + } |
| + raw, _ = json.Marshal(payload) |
| + envelope.Payload = base64.StdEncoding.EncodeToString(raw) |
| + tampered, _ := json.Marshal(envelope) |
| + if _, err := VerifyChallenge(pub, tampered, now); !errors.Is(err, ErrChallenge) { |
| + t.Fatalf("changed %s retained signature: %v", field, err) |
| + } |
| + } |
| +} |
| ``` |
| - [ ] **Run the task tests.** |
| @@ -2658,7 +3086,8 @@ type Config struct { |
| // It must not wait for the host guard: uncertain disk writes retain that |
| // guard to deny other processes, while termination must proceed independently. |
| // Its failure remains a separate denial owned by the caller, even after CC9. |
| - Cleanup func(context.Context) error |
| + Cleanup func(context.Context) error |
| + writeLocal func(*localDir, State) error // test seam; nil uses durable descriptor-relative I/O |
| } |
| type Controller struct { |
| @@ -2678,6 +3107,7 @@ type Controller struct { |
| done chan struct{} |
| watchDone chan struct{} |
| unlock func() |
| + unlive func() |
| closeMu sync.Mutex |
| closed bool |
| } |
| @@ -2718,6 +3148,10 @@ func Open(ctx context.Context, cfg Config) (*Controller, error) { |
| pending := false |
| defer func() { |
| if !pending { |
| + if c.unlive != nil { |
| + c.unlive() |
| + c.unlive = nil |
| + } |
| c.unlock() |
| c.unlock = nil |
| } |
| @@ -2729,9 +3163,12 @@ func Open(ctx context.Context, cfg Config) (*Controller, error) { |
| return openErr |
| } |
| defer d.close() |
| - writeErr := d.write(c.state) |
| - pending = writeErr == nil |
| - return writeErr |
| + if writeErr := c.write(d, c.state); writeErr != nil { |
| + return writeErr |
| + } |
| + c.unlive, lockErr = tryAcquireLive(cfg.LockDir, cfg.Binding.Host) |
| + pending = lockErr == nil |
| + return lockErr |
| }) |
| if !errors.Is(err, ErrControllerBusy) { |
| break |
| @@ -2752,6 +3189,10 @@ func Open(ctx context.Context, cfg Config) (*Controller, error) { |
| } |
| if err != nil { |
| c.state.State = "lost" |
| + if c.unlive != nil { |
| + c.unlive() |
| + c.unlive = nil |
| + } |
| if c.unlock != nil { |
| _ = c.save(c.state) |
| c.unlock() |
| @@ -2764,14 +3205,28 @@ func Open(ctx context.Context, cfg Config) (*Controller, error) { |
| return c, nil |
| } |
| +func (c *Controller) write(d *localDir, s State) error { |
| + if c.cfg.writeLocal != nil { |
| + return c.cfg.writeLocal(d, s) |
| + } |
| + return d.write(s) |
| +} |
| + |
| +// save retains the guard until lost state is durable, including failures before |
| +// rename. The revoked live lock denies CheckLocal while the old file survives. |
| func (c *Controller) save(s State) error { |
| return c.cfg.Guard.WithLocal(context.Background(), s.Host, func() error { |
| - d, err := openLocal(c.cfg.Guard.DataDir) |
| - if err != nil { |
| - return err |
| + for { |
| + d, err := openLocal(c.cfg.Guard.DataDir) |
| + if err == nil { |
| + err = c.write(d, s) |
| + d.close() |
| + } |
| + if err == nil || s.State != "lost" { |
| + return err |
| + } |
| + time.Sleep(100 * time.Millisecond) |
| } |
| - defer d.close() |
| - return d.write(s) |
| }) |
| } |
| func (c *Controller) authority(ctx context.Context) error { |
| @@ -2801,6 +3256,12 @@ func (c *Controller) loseLocked(err error) { |
| } |
| c.err = err |
| c.state.State = "lost" |
| + // Revoke cross-process admission before publishing loss or starting I/O. |
| + // This close needs no disk space. The lifetime lock remains held. |
| + if c.unlive != nil { |
| + c.unlive() |
| + c.unlive = nil |
| + } |
| close(c.lost) |
| c.cancel() |
| c.cfg.CloseAdmission() |
| @@ -3009,7 +3470,7 @@ func (c *Controller) publish(next State) error { |
| if err != nil { |
| return err |
| } |
| - writeErr := d.write(next) |
| + writeErr := c.write(d, next) |
| c.mu.Lock() |
| if writeErr != nil { |
| c.loseLocked(writeErr) |
| @@ -3034,7 +3495,7 @@ func (c *Controller) publish(next State) error { |
| // Cancellation does not release the guard over uncertain active bytes. |
| // Close may time out; it keeps the lifetime lock until this join completes. |
| for { |
| - if persistErr := d.write(denied); persistErr == nil { |
| + if persistErr := c.write(d, denied); persistErr == nil { |
| return err |
| } |
| time.Sleep(100 * time.Millisecond) |
| @@ -3082,13 +3543,17 @@ func CheckLocal(dataDir string, expected Binding, clock Clock, lockDir string) e |
| if lockDir == "" { |
| lockDir = "/run/carlos/hostexec" |
| } |
| - unlock, err := tryAcquire(lockDir, expected.Host) |
| - if err == nil { |
| - unlock() |
| - return ErrLost |
| - } |
| - if !errors.Is(err, ErrControllerBusy) { |
| - return err |
| + // Guard.With serializes these probes with other writers and Open. A free |
| + // live lock denies even if ENOSPC left a future-dated active file behind. |
| + for _, probe := range []func(string, string) (func(), error){tryAcquire, tryAcquireLive} { |
| + unlock, err := probe(lockDir, expected.Host) |
| + if err == nil { |
| + unlock() |
| + return ErrLost |
| + } |
| + if !errors.Is(err, ErrControllerBusy) { |
| + return err |
| + } |
| } |
| d, err := openLocal(dataDir) |
| if err != nil { |
| @@ -3677,7 +4142,8 @@ func (*localDir) write(State) error { return ErrUnsupp |
| func (*localDir) remove(string) error { return ErrUnsupported } |
| func acquire(context.Context, string, string) (func(), error) { return nil, ErrUnsupported } |
| -func tryAcquire(string, string) (func(), error) { return nil, ErrUnsupported } |
| +func tryAcquire(string, string) (func(), error) { return nil, ErrUnsupported } |
| +func tryAcquireLive(string, string) (func(), error) { return nil, ErrUnsupported } |
| ``` |
| - [ ] **Create `internal/hostexec/local_unix.go` with this complete content.** |
| @@ -3817,6 +4283,14 @@ func (d *localDir) remove(host string) error { |
| return d.sync(d.fd) |
| } |
| func tryAcquire(dir, host string) (func(), error) { |
| + return tryAcquireNamed(dir, host, ".lock") |
| +} |
| + |
| +func tryAcquireLive(dir, host string) (func(), error) { |
| + return tryAcquireNamed(dir, host, ".live") |
| +} |
| + |
| +func tryAcquireNamed(dir, host, suffix string) (func(), error) { |
| if _, err := Key(host, "check"); err != nil { |
| return nil, err |
| } |
| @@ -3826,10 +4300,10 @@ func tryAcquire(dir, host string) (func(), error) { |
| } |
| defer unix.Close(fd) |
| flags := unix.O_RDWR | unix.O_CREAT | unix.O_NOFOLLOW | unix.O_CLOEXEC | unix.O_NONBLOCK |
| - lock, err := unix.Openat(fd, host+".lock", flags|unix.O_EXCL, 0660) |
| + lock, err := unix.Openat(fd, host+suffix, flags|unix.O_EXCL, 0660) |
| created := err == nil |
| if errors.Is(err, unix.EEXIST) { |
| - lock, err = unix.Openat(fd, host+".lock", flags, 0660) |
| + lock, err = unix.Openat(fd, host+suffix, flags, 0660) |
| } |
| if err != nil { |
| return nil, err |
| @@ -3985,8 +4459,12 @@ package hostexec |
| import ( |
| "context" |
| + "encoding/json" |
| "errors" |
| + "golang.org/x/sys/unix" |
| "os" |
| + "os/exec" |
| + "sync/atomic" |
| "testing" |
| "time" |
| @@ -4165,6 +4643,127 @@ func TestCC9SuccessorExclusionClearsOnlyExpectedOperation(t *testing.T) { |
| t.Fatalf("unrelated fence changed: %v, %v", still, err) |
| } |
| } |
| + |
| +func TestAmbiguousRenewalENOSPCBeforeRenameDeniesImmediately(t *testing.T) { |
| + cfg, _, clock, store, cleaned := fixture(t) |
| + store.failAt = 2 |
| + var noSpace atomic.Bool |
| + noSpace.Store(true) |
| + var failures atomic.Int32 |
| + cfg.writeLocal = func(d *localDir, s State) error { |
| + if s.State != "lost" || !noSpace.Load() { |
| + return d.write(s) |
| + } |
| + original := d.sync |
| + d.sync = func(int) error { return unix.ENOSPC } |
| + defer func() { d.sync = original }() |
| + err := d.write(s) // file fsync fails before rename; old active bytes remain |
| + if errors.Is(err, unix.ENOSPC) { |
| + failures.Add(1) |
| + } |
| + return err |
| + } |
| + c, err := Open(t.Context(), cfg) |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + defer func() { noSpace.Store(false); closeController(t, c) }() |
| + active := ready(t, c) |
| + until(t, func() bool { |
| + clock.mu.Lock() |
| + defer clock.mu.Unlock() |
| + for _, a := range clock.alarms { |
| + if a.at == clock.boot+int64(Renewal) { |
| + return true |
| + } |
| + } |
| + return false |
| + }) |
| + clock.advance(Renewal) |
| + select { |
| + case <-c.Lost(): |
| + case <-time.After(2 * time.Second): |
| + t.Fatal("ambiguous renewal did not latch loss") |
| + } |
| + // Check from the loss signal onward, before waiting for any persistence. |
| + if err := CheckLocal(cfg.Guard.DataDir, active.Binding, clock, cfg.LockDir); !errors.Is(err, ErrLost) { |
| + t.Fatalf("admitted immediately after ambiguity: %v", err) |
| + } |
| + until(t, func() bool { return failures.Load() >= 2 && cleaned.Load() >= 2 }) |
| + if _, err := c.Ready(t.Context()); err == nil { |
| + t.Fatal("in-memory admission survived loss") |
| + } |
| + d, err := openLocal(cfg.Guard.DataDir) |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + defer d.close() |
| + old, err := d.read(active.Host) |
| + if err != nil || old != active { |
| + t.Fatalf("test did not leave the old active file: %+v %v", old, err) |
| + } |
| + _, boot, _ := clock.Now() |
| + if boot >= active.SafeUntil { |
| + t.Fatal("test reached expiry instead of exercising immediate denial") |
| + } |
| + binding, _ := json.Marshal(active.Binding) |
| + cmd := exec.CommandContext(t.Context(), os.Args[0], "-test.run=^TestCheckLocalRefusesLostLiveLock$") |
| + cmd.Env = append(os.Environ(), "HOSTEXEC_PROBE_BINDING="+string(binding), "HOSTEXEC_PROBE_DATA="+cfg.Guard.DataDir, "HOSTEXEC_PROBE_LOCKS="+cfg.LockDir) |
| + if output, err := cmd.CombinedOutput(); err != nil { |
| + t.Fatalf("independent process admitted old active state: %v\n%s", err, output) |
| + } |
| + ctx, cancel := context.WithTimeout(t.Context(), 30*time.Millisecond) |
| + defer cancel() |
| + entered := false |
| + err = cfg.Guard.WithLocal(ctx, active.Host, func() error { entered = true; return nil }) |
| + if !errors.Is(err, context.DeadlineExceeded) || entered { |
| + t.Fatalf("guard released while denial write failed: %v entered=%v", err, entered) |
| + } |
| + if unlock, err := tryAcquire(cfg.LockDir, active.Host); !errors.Is(err, ErrControllerBusy) { |
| + if unlock != nil { |
| + unlock() |
| + } |
| + t.Fatalf("controller exclusion released: %v", err) |
| + } |
| + ctxClose, cancelClose := context.WithTimeout(t.Context(), 30*time.Millisecond) |
| + defer cancelClose() |
| + if err := c.Close(ctxClose); !errors.Is(err, context.DeadlineExceeded) { |
| + t.Fatalf("Close did not retain the unfinished controller: %v", err) |
| + } |
| + noSpace.Store(false) |
| + select { |
| + case <-c.cleanupDone: |
| + case <-time.After(2 * time.Second): |
| + t.Fatal("lost state was not retried after disk recovery") |
| + } |
| + lost, err := d.read(active.Host) |
| + if err != nil || lost.State != "lost" { |
| + t.Fatalf("denial not durable: %+v %v", lost, err) |
| + } |
| + if err := cfg.Guard.WithLocal(t.Context(), active.Host, func() error { |
| + if err := CheckLocal(cfg.Guard.DataDir, active.Binding, clock, cfg.LockDir); !errors.Is(err, ErrLost) { |
| + t.Fatalf("disk recovery revived admission: %v", err) |
| + } |
| + return nil |
| + }); err != nil { |
| + t.Fatal(err) |
| + } |
| +} |
| + |
| +func TestCheckLocalRefusesLostLiveLock(t *testing.T) { |
| + raw := os.Getenv("HOSTEXEC_PROBE_BINDING") |
| + if raw == "" { |
| + t.Skip("subprocess witness") |
| + } |
| + var binding Binding |
| + if err := json.Unmarshal([]byte(raw), &binding); err != nil { |
| + t.Fatal(err) |
| + } |
| + clock := &testClock{boot: int64(time.Hour + Renewal)} |
| + if err := CheckLocal(os.Getenv("HOSTEXEC_PROBE_DATA"), binding, clock, os.Getenv("HOSTEXEC_PROBE_LOCKS")); !errors.Is(err, ErrLost) { |
| + t.Fatalf("old active record authorized an independent process: %v", err) |
| + } |
| +} |
| ``` |
| - [ ] **Run the task tests.** |
| @@ -4587,16 +5186,29 @@ The proposed answers live in **§7.3/§10.1** and **amadan #236**. They are not |
| 6. **Verify within approved scope.** Use deterministic local witnesses first. Any live compute fence or production infrastructure change needs the applicable scoped authorization. Verify stopped/terminated instance identity, unknown-timer reporting, quarantine across another host, complete returning-box retirement and matching-revision readmission. Verify every changed executable restore/boot route against below-floor candidates; lease expiry is never fencing evidence. |
| 7. **Recover without downgrading authority.** Pause new lifecycle starts if rollout fails. Preserve claims, quarantine, loss history and armed floor policy. Use a compatible fixed release or an authorized above-floor rollback. Removing the flag, deleting a floor object or invoking an old recovery executable cannot cancel the persistent floor. |
| +## Self-review |
| + |
| +Rev 2 addresses each independent finding with code and regression tests: |
| + |
| +- **P1 — denial after ambiguous renewal:** `loseLocked` revokes the process-held live lock before notifying callers. `save` holds the host guard across failed lost-state writes and retries until the file and directory are synced. The lifetime lock remains held until all work joins; cleanup runs independently. `TestAmbiguousRenewalENOSPCBeforeRenameDeniesImmediately` fails the temporary file’s sync with `ENOSPC` before rename, proves that the old active file and future deadline remain, and checks immediate local and separate-process denial, guard retention, independent cleanup, timed-out close and eventual durable denial. The live lock also closes the interval while persistence is waiting to acquire the guard. A crash drops both locks, so surviving active bytes still cannot authorize work. |
| +- **P2 — successive declarations:** `TestDeclareLostNewerRevisionAppliesSecondConsequences` first records a stopped box while retaining its local chore, then accepts a freshly bound higher revision with destroyed-storage evidence. Older and equal revisions under new declaration IDs refuse; a stale claim binding refuses even with a higher revision. Replays retain both immutable declarations, and revision 9 readmission cannot clear revision 10. |
| +- **P2 — serving local-only:** the record, transition and signed challenge permit an empty current operation ID only for the constrained serving case. `TestServingDestroyedLocalOnlyPreservesAuthorityAndOtherChores` starts with real `Adopt` output and compares the whole resulting claim, allowing only exact local discharge, discovery completion, generation and loss/quarantine bookkeeping. `TestServingDestroyedLocalOnlyRefusesWideningAndStaleBindings` rejects authority changes, regional work, write/timer-loss effects, broader recovery intent, stale claim bindings and wrong chore identities. `TestServingDestroyedLocalOnlyChallengeBindsOriginalChore` checks the signed round trip, narrowing rules and tampered claim/chore bindings. |
| + |
| +The live-lock probe must run inside `Guard.With`, as does the lifetime-lock probe: the host guard serializes probes and controller startup across processes. Neither lock grants claim or quarantine authority. The extra live lock uses the same descriptor-relative I/O and provisioned directory as the lifetime lock, and no network request holds the guard. All production integrations remain deferred to Plan 8b. |
| + |
| ## Authoring verification |
| -The complete code was checked in disposable on-disk source exports outside this repository, against the pinned merge baseline above. A second fresh export was reconstructed from this document's **33 complete Go files and two exact patches**; every Go source matched the tested export, apart from the follow-on-plan wording in comments. Archives use `-buildvcs=false` because they are not Git checkouts. The default cache was attempted first; the existing shared on-disk cache was used after the sandbox reported it read-only. |
| +Rev 2 was checked in a disposable on-disk source export outside this repository, reconstructed from the pinned merge baseline above plus this document’s **33 complete Go files and two exact patches**. All embedded Go files match the tested source byte for byte; the baseline patches are unchanged. Archives use `-buildvcs=false` because they are not Git checkouts. The default cache was attempted first; the existing shared on-disk `/home/paulca/.cache/tmp/gocache` was used after the sandbox reported the default cache read-only. |
| -Checks on 2026-10-05: |
| +Checks on 2026-10-05 for rev 2: |
| - Whole-repository builds passed for Linux/amd64, Darwin/amd64, Windows/amd64 and Linux/arm64. |
| -- All repository test packages compiled with `go test -buildvcs=false -exec=/bin/true ./...`; this is compilation evidence, not a claim that the whole test suite ran. |
| -- The complete `hostclaim`, `hostexec`, `hostguard`, `hostquarantine`, `hostfence` and `awsinv` suites passed with `-race`. Focused new envelope, transport, console-route, CLI-parser and console-flag tests passed. |
| -- Whole-repository `go vet -buildvcs=false ./...` passed, and `gofmt -l .` was empty. |
| -- Independent GPT reviews found and corrected stale lease publication, orphan-controller admission, CC9 replacement races, premature acknowledgement ETags, unsigned archival acceptance, source-cut provenance, missing destroyed-storage assertions and target-only loss suppression. The explicitly requested Opus CLI review produced no verdict before a 45-second timeout in this sandbox; the required Opus review remains outstanding before implementation lands. |
| +- All repository test packages compiled with `go test -buildvcs=false -exec=/bin/true ./...`. This checks compilation; it does not run the whole test suite. |
| +- The complete `hostclaim`, `hostexec` and `hostfence` suites passed with `-race -count=1`, including all new regression tests. |
| +- Each regression was also watched to fail in scratch when its corresponding restriction or failure was restored: remove the live-lock check, reject different quarantine revisions, or require a nonempty operation ID in the loss record or challenge. The fixes were restored afterward. |
| +- Whole-repository `go vet -buildvcs=false ./...` passed, and `gofmt -l .` was empty. The plan’s `git diff --check` was clean. |
| +- The repository-required Opus review was attempted through the CLI with `--model opus`; it timed out after 45 seconds without a verdict. That review remains outstanding before implementation lands. |
| + |
| +Rev 1’s checks also covered the complete `hostguard`, `hostquarantine` and `awsinv` suites with `-race`, plus focused envelope, transport, console-route, CLI-parser and console-flag tests. Its independent reviews corrected stale lease publication, orphan-controller admission, CC9 replacement races, premature acknowledgement ETags, unsigned archival acceptance, source-cut provenance, missing destroyed-storage assertions and target-only loss suppression. These are historical rev 1 results, not additional rev 2 test runs. |
| Only this Markdown plan was written in the repository. Scratch exports were removed after verification. No implementation files, branches, staging, commits, cloud mutations or feature-flag changes were made. The tested code remains an inert proposal; it does not implement or certify the deferred floor, real directory or live recovery integrations. |