plan: claim Plan 5 rev 0 (draft), console operations
Tasks 1-4 complete (record/transition/recovery primitives, the shared retirement entry point, inert adapters); tasks 5-9 wait on spec contracts P5-D1..D6 listed in the plan. Real retirement call sites mapped.
Co-Authored-By: Codex <noreply@openai.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
1 file changed,
+2020
−0
docs/superpowers/plans/2026-10-05-host-lifecycle-claim-5-console-ops.md+2020 −0
CI passed — run details
diff --git a/docs/superpowers/plans/2026-10-05-host-lifecycle-claim-5-console-ops.md b/docs/superpowers/plans/2026-10-05-host-lifecycle-claim-5-console-ops.md| new file mode 100644 |
| index 000000000..c5fbdd3da |
| --- /dev/null |
| +++ b/docs/superpowers/plans/2026-10-05-host-lifecycle-claim-5-console-ops.md |
| @@ -0,0 +1,2020 @@ |
| +# Host Lifecycle Claim, Plan 5: console operations |
| + |
| +> **For agentic workers:** implement this plan task by task, with a failing test before each behavior change. Do not enable `CARLOS_HOST_CLAIM`. Resolve the decisions below before implementing the dependent authority paths; an interface without its production provider is not completed integration. |
| + |
| +**Goal:** make instance and sidecar creation, every record retirement, app purge and placement follow the host lifecycle claim. Add the console's recovery pass. Keep all current behavior when the startup flag is unset. |
| + |
| +**Architecture:** `internal/hostops` composes the real `hostclaim` transitions and conditional record operations. `provision.RetireRecord` is the shared, explicitly configured retirement entry point. The console supplies authentication, admission registration, authoritative inventory, executor selection and durable evidence. Regional owners use `internal/park`; the console does not acknowledge another machine's local work. Recovery runs from `cmd/console.serve`, not `NewServer`. |
| + |
| +**Tech stack:** Go 1.26, the existing `release.Store` and conditional delete adapter, `hostclaim`, `provision`, `park`, and standard-library JSON/context/time. No new dependency. |
| + |
| +**Spec:** `docs/superpowers/specs/2026-10-02-host-lifecycle-claim-design.md`, **rev 6.16**, §2, C1–C7 including C5a, §4, all of §5, §6.2/6.2a/6.5, §8–§12. Plans 2, 3, 4 rev 2, 7 and 7b supply storage, guard, park and door interfaces. Plan 1's roadmap assigns Plan 5 “create, delete and `RetireRecord` (every call site), admission.json, purge journal and fence, placement, the claim-recovery pass (§5, §8)”, depending on **Plans 1 and 4**. Plan 8 owns the floor and activation. |
| + |
| +**Source baseline:** `origin/main` at `04cd2d78`, the committed claim library `47426cc1`, Plan 7's committed `feat/hostclaim-door` at `87c8ef92`, and the published Plan 4 rev 2 code. The planning checkout is older than main. `edge-replica-placements` at `b5b0a53a` was also inspected. There is no `feat/hostclaim-park` commit at authoring time; do not substitute an uncommitted sibling directory for the branch dependency. |
| + |
| +**Scope of the code below:** complete, compiling files for the transitions, record operations, recovery traversal, parking adapter, retirement dispatch and console pass whose behavior the spec defines. The admission coordinator, full purge coordinator, placement completion and production authority providers depend on D1–D6. Their algorithms, call sites and required tests are specified below, but no fabricated wire format or permissive implementation is supplied for those unresolved contracts. Those tasks remain open. This plan is not evidence that the full console integration can be enabled. |
| + |
| +## Global constraints |
| + |
| +- **Inert:** reuse Plan 7's `Config.HostClaim`. `cmd/console` reads `CARLOS_HOST_CLAIM` once through `hostClaimEnabled(os.Getenv)`: trimmed nonempty enables, including `0`. Never read the environment in handlers, `provision` or recovery. Flag-off takes the existing path before constructing any new provider or reading a claim/admission/journal. Existing tests stay unchanged. |
| +- **C2:** persist intent before effects, including private-store provisioning and first-route attach effects. `Attempt` receives the claim observation made **before** admission registration. Its claim CAS runs exactly once. A 412 restarts the outer create from step 0 with a new registration, operation ID and server-assigned UTC nanosecond creation time. Never reuse the old incarnation or reread the claim inside a generic CAS retry loop. |
| +- **C3/C5a:** `release.DeleteIfMatch` is for exact control-record ETags; `park.Engine` is for exact data-object versions. Never use a bare data-key delete. Never delete `control/hosts/<host>.json`, including a holderless purge fence. Free is a conditional write preserving retirement history, quarantine, quiet times and cancelled-create hints. |
| +- **C6:** only `os.ErrNotExist` establishes absence. Failed/malformed records, claims, admissions, journals, inventory or listings refuse or wait. Discard a partial listing when its call returns an error. Do not use `ListRecords`/`ListSidecarRecords` as an authoritative purge census: they cap answers and skip bad bodies. Use complete listings and individual validated reads. |
| +- **Ownership:** account membership, service roles, CSRF, fresh authentication, control-plane protection, production/trash checks, domain ownership and DNS policy stay in their existing order. A lifecycle claim is distinct from `domains.ClaimKey` and `hostClaimHolder` in `apex.go`; it replaces neither. Preserve `hostHolder` and `hostClaimHolder` from `04cd2d78`. |
| +- **Stop evidence:** `BeginDelete` closes new door admission everywhere through the shared claim. It retains the current executor/stamp. Queuing parking chores is not proof of stop. Plan 3/6 evidence or Plan 8's verified lost-box fence authorizes each park operation; lease expiry, an absent record and an old heartbeat do not. Plan 8's participant acknowledgement barrier precedes departure from an operation's first phase. |
| +- **Chores:** `WriteSet` uses touched regions/boxes when complete, or their union with the complete deployment inventory otherwise. `ToParking` assigns a separate obligation ID and retirement ID to every chore and preserves pre-existing chores. `CompleteObligation` matches the chore's own op ID, including discovery chores that survive adoption. Only `FreeAfterPark` clears an ordinary parked holder after every chore finishes. |
| +- **Portability:** no Linux-specific code in the console coordinator. Darwin and Windows compile, with local retirement refusal supplied by Plan 4. Use the default Go cache. If it is read-only, use the existing shared on-disk `/home/paulca/.cache/tmp/gocache`; never `/tmp` and never a new task-specific cache. |
| +- **Rollout:** no flag, floor, IAM, DNS, deployment or production changes in this plan or its implementation branch. Plans 5 and 7 land inert until Plan 8. Keep below-floor behavior and activation sequencing under Plan 8; never begin a protected operation and then fall through to legacy deletion. |
| + |
| +## Decisions the spec does not finish |
| + |
| +Resolve these in the owning design before completing the affected implementation steps. The refusals in the code below are dependency checks, not substitutes for the missing implementations. |
| + |
| +1. **D1 — admission wire format and lease ownership.** §5.1/§5.3 fixes the key `apps/<acct>/<app>/admission.json`, open/closed state, registration contents `{create id, host, expires}`, renewal until serving/cancelled, and conditional writes. It does not give JSON property names/container shape, a lease duration/renewal cadence, ownership for overlapping same-create renewers, or how a recreated app reopens closed admission without allowing an old renewal/remove to affect it. Approve those details together. Preserve the specified read-claim → register → final admission check → exact saved-claim CAS ordering; do not invent a five-minute registration TTL from the five-minute recovery threshold. Define cleanup of registrations whose creates completed before a console crashed. |
| +2. **D2 — authoritative deployment inventory and executor selection.** A region can contain several boxes; `fleet/<name>`, pools, replicas, unconstrained/static/everywhere records and route-only legacy hosts also exist. §5.1 says “the record region's box”, but no durable unique executor selection or complete eligible inventory API is specified. Existing placement/status joins intentionally drop unreadable/stale entries and are not proof of a complete deployment. Define the source, scope and identity bindings for every region and box, the current legacy writer, and discovery of legacy records across apps. Coordinate with #212 and Plan 8 D7. Never choose the console box or the first live heartbeat as executor. Unsupported shapes must refuse flagged operations explicitly until resolved. |
| +3. **D3 — durable evidence and regional worker adapters.** Plan 4 requires real `Authorize`, `View`, aggregate `Leases`, original `EvidenceAt`, and a positive enforced maximum request duration. Plans 3/6/8 do not yet give the console a complete durable evidence transport/binding contract, regional-worker assignment, or authority for asserting an untouched box has no local state. Define how a console queues work without gaining another region's data privileges, how evidence survives retries and executor loss, and how a discovery chore's evidence is established. A retry's clock is not evidence. `RunOwnedRemote` below is the checked adapter; it is not an evidence provider. |
| +4. **D4 — purge's durable host set, fence ownership and final cleanup.** The journal key and ordered phases are specified, but the persisted host/record/fence list, app identity/generation, per-host progress and cleanup checkpoint schema are not. A crash after draining a registration must not lose the identity of the holderless fence it created. Define behavior when a drained registration's host is currently held by another app or operation: CC21 authorizes fences only from free/absent, so it cannot overwrite that holder. Define replay when a host frees before app cleanup, the exact order of `done`, fence release, admission removal/reopening and journal removal, and retry after the app tombstone was removed. `release.PurgeApp` currently sweeps `purge.json` and `admission.json` with the app prefix; it must not be used unchanged. The journal must remain authoritative until its own completion removes it. |
| +5. **D5 — placement's operation completion and app-wide serialization.** §5.4 defines the start CAS and target intent, but not stop/transfer/install phases, rollback, final executor/stamp, or how one app-level placement change excludes concurrent creates across several host claims. Current placement writes app SQLite state, then routes, then records. Claiming hosts one at a time after the app write is too late. Define a durable app-wide operation/census and its replay owner, including route-only apps and empty apps, plus interaction with `Record.Replicas` and moves. Do not implement `placing → serving` by assigning the target region's first box or issuing an unexplained stamp. The start transition below deliberately has no guessed completion transition. |
| +6. **D6 — create/rollback effects and purge teardown replay.** Private storage is provisioned before `CreateRecord` today; first-route attach writes a separate domain claim before its record; sidecar deletion later removes its wake object; purge tears down store/email estate and console rows. Define durable recovery for those effects under the operation, including ambiguous create writes, a rollback while still `creating`, same-key successor protection for auxiliary objects, and concurrent app restore/rename/move/trash/teardown. A complete create intent must reconstruct every pre-serving effect it permits. App purge needs backend authority on the console; auto-purge boxes lack it and must durably hand work to the same coordinator, without broadening IAM. Define observable pending/accepted outcomes so a successful delete response never claims that parking is finished. Reuse existing copy where accurate; review any new user-facing strings during implementation. |
| + |
| +D1–D6 are concrete missing contracts. Ordinary implementation choices below—two-minute recovery cadence, helper/package names and in-memory batching—are not additional spec gates. |
| + |
| +## Every retirement call site |
| + |
| +`git grep -n RetireRecord origin/main -- '*.go'` and the same search on this worktree and `edge-replica-placements` find **zero** definitions or calls. The brief/spec use the intended future name. Do not claim these paths already call it. Add the dispatch in Task 3 and migrate the following actual callers. Line numbers refer to `origin/main:04cd2d78`. |
| + |
| +| Existing call | Role | Required regression, each with flag off and on | |
| +|---|---|---| |
| +| `internal/console/apiinstances.go:1186` → `DeleteRecord` | Sibling-apex create rollback | `TestClaimRetireCreateRollback`: capture this attempt's incarnation; a successor at the key survives; pending rollback is replayed, not bare-deleted. | |
| +| `internal/console/apiinstances.go:1675` → `DeleteRecord` | Instance delete API | `TestClaimRetireInstanceDelete`: intent before conditional record deletion; parking obligations and door closure; repeated request resumes the same op. | |
| +| `internal/console/apisidecars.go:374` → `DeleteSidecarRecord` | Sidecar delete API | `TestClaimRetireSidecarDelete`: holder family/name/key remain sidecar; all touched regions/boxes covered; wake-object cleanup cannot erase a successor's wake. | |
| +| `internal/console/apidomains.go:855` → `DeleteRecord` | Domain detach API, first route | `TestClaimRetireAPIDomainDetach`: retire route record before releasing domain claim; a blocked retirement leaves the claim attached. Alias-only detach creates no record retirement. | |
| +| `internal/console/domainattach.go:304` → `DeleteRecord` | Shared/browser domain detach, first route | `TestClaimRetireUIDomainDetach`: same order and refusal as API; preserve platform-operator and ownership checks. | |
| +| `internal/console/domainattach.go:492` → `DeleteRecord` | `rollbackAttach` | `TestClaimRetireAttachRollback`: ambiguous own write uses its captured incarnation; known duplicate never retires another attempt; late record remains an orphan. Callers are `:226` and `:236`. | |
| +| `internal/console/ownership.go:442` → `release.PurgeApp` | Owner purge | `TestClaimRetirePurgeApp`: every instance/sidecar is reserved before store/email teardown; retire under purge ID; no transition back to serving. | |
| +| `internal/domains/reconcile.go:232` → `release.PurgeApp` | Box auto-purge | `TestClaimRetireAutoPurge`: same journal/coordinator; box with no backend authority queues/waits and cannot run a raw prefix sweep. | |
| + |
| +The lower-level legacy helpers are `internal/provision/provision.go:1071` and `internal/provision/sidecar.go:398`. `release.PurgeApp` is `internal/release/trash.go:182`; its prefix loop also deletes provisioning records without calling either helper. Migrate the sweep, not only the console's named delete calls. `dnsrec.DeleteRecord` is DNS, not a provisioning-record retirement. Keep legacy primitive tests at `provision_test.go:518,525` and `reconcile_test.go:1025,1062,1970,2434` as flag-off coverage. |
| + |
| +Create callers are `apiinstances.go:1168`, `apisidecars.go:330`, `domainattach.go:192`, and `provision/replica.go:83` (`CreateRecordReplica` delegates to `CreateRecord`). First-route static creation is still a host-ownership path. Both `origin/main` and `edge-replica-placements` carry replica role/placement behavior; the latter is older in other areas, so do not merge its old console files over main. No `PlaceInstance` symbol exists in those refs: actual placement is `placeAppTarget` in `fleets.go`, with `SetAppPlacement` before `reg.SetPlacement` at `:616` and `SetRecordRegion` at `:639` (`provision.go:853`). All three placement doors call this shared function. |
| + |
| +## Review focus |
| + |
| +1. A creator paused before claim CAS loses after purge fences two hosts and writes both free again. Its old absent/ETag observation never becomes usable. |
| +2. Recovery reads the record before the claim when deciding orphanhood, even after the one-hour hint expires; exact conditional deletion cannot erase a replacement. |
| +3. Every old delete door and raw purge sweep reaches the configured retirement coordinator. A nil provider with the flag on refuses; it never selects the legacy path. |
| +4. Adoption uses one discovery implementation, including non-serving regions and boxes. A listing error cannot shorten the chore set. |
| +5. Purge retains its reservation while any host, fence, backend effect or cleanup is unfinished; host free never releases the app reservation. |
| +6. Remote acknowledgements retain original evidence time, update `quiet_after` and preserve discovery completion. Console recovery never treats an expired execution lease as proof to park. |
| +7. Placement claims precede app, route and record writes; no app-wide request partially changes placement before encountering another host's active move. |
| +8. Flag-off tests observe no new claim/admission/park calls. Enabled code remains unactivated until the complete Plan 8 floor and participant protocol are present. |
| + |
| +## File structure |
| + |
| +| File | Responsibility | |
| +|---|---| |
| +| `internal/hostops/record.go` | Validated, ETag-bearing records; stamped create inputs; complete app record lists; orphan order | |
| +| `internal/hostops/operations.go` | Exact saved-observation create attempt, completion/cancellation, shared adoption, delete/parking/free | |
| +| `internal/hostops/retirement.go` | Resume retirement through the shared provisioning interface | |
| +| `internal/hostops/purge.go` | Specified journal envelope/phases, holderless fences, purge intent, placement start | |
| +| `internal/hostops/recovery.go` | Claim discovery, fresh per-host dispatch, uncapped record/app/purge traversal | |
| +| `internal/hostops/remote.go` | Plan 4 scoped regional work adapter | |
| +| `internal/hostops/operations_test.go` | Complete primitive/race/refusal tests below | |
| +| `internal/provision/retirement.go`, `retirement_test.go` | Explicit flag dispatch and mandatory enabled provider | |
| +| `internal/console/hostclaimops.go`, `hostclaimops_test.go` | Server adapters and background pass, inert when disabled | |
| +| `internal/console/server.go`, `cmd/console/main.go` | Config injection and pass startup; reuse Plan 7 flag | |
| +| `internal/hostops/admission.go`, `purge_coordinator.go`, `placement.go` | D1/D4/D5-dependent coordinators; no invented implementation in this revision | |
| +| `internal/console/hostclaimauthority.go` | D2/D3/D6 production providers, D1 admission lease ownership | |
| +| Existing create/delete/purge/placement files in the table | Flag-gated integration and individual door regressions | |
| + |
| +### Task 1: Establish the implementation branch and failing tests |
| + |
| +- [ ] **Step 1: Wait for committed Plan 4, then run these commands in a clean implementation checkout.** They are not commands to run while authoring this Markdown. |
| + |
| +```bash |
| +git fetch origin |
| +git cat-file -e feat/hostclaim-park:internal/park/remote.go |
| +git cat-file -e feat/hostclaim-park:internal/park/local.go |
| +git switch -c feat/hostclaim-console feat/hostclaim-park |
| +git merge --no-edit origin/main |
| +if ! git merge-base --is-ancestor feat/hostclaim-door origin/main |
| +then |
| + git merge --no-edit feat/hostclaim-door |
| +fi |
| +git merge-base --is-ancestor feat/hostclaim-park HEAD |
| +git merge-base --is-ancestor origin/main HEAD |
| +git grep -n 'HostClaim' -- cmd/console/main.go internal/console/server.go |
| +``` |
| + |
| +Expected: both park files exist in a commit, both ancestry checks exit 0, and Plan 7's `Config.HostClaim` and startup flag are present. If Plan 7 is not on main, the merge above supplies them. If that branch itself is not committed, wait rather than introducing a conflicting second flag implementation. Resolve conflicts using main's newer console code and the real library signatures. |
| + |
| +- [ ] **Step 2: Add the complete failing test files below.** The tests use only local `DirStore` instances and injected authorities. Test authority callbacks are intentionally permissive only inside the test fixture. Production must supply D1–D3/D6 providers. |
| + |
| +**`internal/hostops/operations_test.go`** |
| + |
| +```go |
| +package hostops |
| + |
| +import ( |
| + "context" |
| + "errors" |
| + "fmt" |
| + "os" |
| + "testing" |
| + "time" |
| + |
| + "github.com/carlosframework/platform/internal/hostclaim" |
| + "github.com/carlosframework/platform/internal/provision" |
| + "github.com/carlosframework/platform/internal/release" |
| +) |
| + |
| +type authority struct{ err error } |
| + |
| +func (a *authority) Create(context.Context, hostclaim.Observed, Record, string) error { return a.err } |
| +func (a *authority) Executor(context.Context, hostclaim.Observed, Record) (string, error) { |
| + return "box-a", a.err |
| +} |
| +func (a *authority) StartDelete(context.Context, hostclaim.Observed, Record) error { return a.err } |
| +func (a *authority) FirstPhase(context.Context, hostclaim.Observed) error { return a.err } |
| +func (a *authority) Inventory(context.Context, hostclaim.Observed) ([]string, []string, error) { |
| + return []string{"ie", "us"}, []string{"box-a", "box-b"}, a.err |
| +} |
| +func (a *authority) Adoption(context.Context, Record) (string, string, []string, []string, error) { |
| + return "ie", "box-a", []string{"us"}, []string{"box-b"}, a.err |
| +} |
| +func (a *authority) PurgeRetirement(context.Context, hostclaim.Observed) error { return a.err } |
| + |
| +type faultStore struct { |
| + release.Store |
| + failGet string |
| + beforePut func(string) error |
| + beforeDelete func(string, string) error |
| + bareDeletes int |
| +} |
| + |
| +func (s *faultStore) Get(ctx context.Context, key string) ([]byte, string, error) { |
| + if key == s.failGet { |
| + return nil, "", errors.New("read failed") |
| + } |
| + return s.Store.Get(ctx, key) |
| +} |
| +func (s *faultStore) Put(ctx context.Context, key string, b []byte, e string) (string, error) { |
| + if s.beforePut != nil { |
| + if err := s.beforePut(key); err != nil { |
| + return "", err |
| + } |
| + } |
| + return s.Store.Put(ctx, key, b, e) |
| +} |
| +func (s *faultStore) Delete(context.Context, string) error { |
| + s.bareDeletes++ |
| + return errors.New("bare delete forbidden") |
| +} |
| +func (s *faultStore) DeleteIfMatch(ctx context.Context, key, e string) error { |
| + if s.beforeDelete != nil { |
| + if err := s.beforeDelete(key, e); err != nil { |
| + return err |
| + } |
| + } |
| + return release.DeleteIfMatch(ctx, s.Store, key, e) |
| +} |
| + |
| +type fixture struct { |
| + ctx context.Context |
| + store *faultStore |
| + c Controller |
| + auth *authority |
| + now time.Time |
| + serial int |
| +} |
| + |
| +func newFixture(t *testing.T) *fixture { |
| + t.Helper() |
| + f := &fixture{ctx: context.Background(), now: time.Date(2026, 10, 5, 12, 0, 0, 0, time.UTC), auth: &authority{}} |
| + f.store = &faultStore{Store: release.NewDirStore(t.TempDir())} |
| + f.c = Controller{Store: f.store, Authority: f.auth, Now: func() time.Time { return f.now }, NewID: func() string { f.serial++; return fmt.Sprintf("id-%d", f.serial) }} |
| + return f |
| +} |
| +func (f *fixture) record(t *testing.T, host string) Record { |
| + t.Helper() |
| + r, err := PrepareInstance(provision.Record{Schema: 1, Account: "bab", App: "jam", Host: host, Slug: "shop", Channel: "edge", Region: "ie"}, f.now, "ie") |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + return r |
| +} |
| +func (f *fixture) serving(t *testing.T, host string) hostclaim.Observed { |
| + t.Helper() |
| + r := f.record(t, host) |
| + before, err := hostclaim.Read(f.ctx, f.store, host) |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + obs, err := f.c.Attempt(f.ctx, before, r, f.c.NewID(), "test") |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + return obs |
| +} |
| +func TestAttemptUsesPreRegistrationObservation(t *testing.T) { |
| + f := newFixture(t) |
| + for _, host := range []string{"one.bab.example.com", "two.bab.example.com"} { |
| + before, err := hostclaim.Read(f.ctx, f.store, host) |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + fenced, err := hostclaim.Update(f.ctx, f.store, before, FencePurge("purge", f.now)) |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + if _, err = hostclaim.Update(f.ctx, f.store, fenced, ReleasePurgeFence("purge")); err != nil { |
| + t.Fatal(err) |
| + } |
| + r := f.record(t, host) |
| + if _, err = f.c.Attempt(f.ctx, before, r, "paused", "test"); !errors.Is(err, hostclaim.ErrConflict) { |
| + t.Fatalf("late claim: %v", err) |
| + } |
| + if _, _, err = f.store.Get(f.ctx, r.Holder.Key); !errors.Is(err, os.ErrNotExist) { |
| + t.Fatalf("record landed: %v", err) |
| + } |
| + } |
| + if f.store.bareDeletes != 0 { |
| + t.Fatal("deleted claim") |
| + } |
| +} |
| +func TestRecoverCreateAbsentAndLateOrphan(t *testing.T) { |
| + f := newFixture(t) |
| + host := "shop.bab.example.com" |
| + r := f.record(t, host) |
| + before, _ := hostclaim.Read(f.ctx, f.store, host) |
| + if _, err := hostclaim.Update(f.ctx, f.store, before, hostclaim.BeginCreate(r.Holder, r.Body, "create", "test", f.now)); err != nil { |
| + t.Fatal(err) |
| + } |
| + f.now = f.now.Add(5 * time.Minute) |
| + if err := f.c.RecoverCreate(f.ctx, host); !errors.Is(err, ErrPending) { |
| + t.Fatal(err) |
| + } |
| + f.now = f.now.Add(time.Nanosecond) |
| + if err := f.c.RecoverCreate(f.ctx, host); err != nil { |
| + t.Fatal(err) |
| + } |
| + obs, _ := hostclaim.Read(f.ctx, f.store, host) |
| + if !obs.Claim.Free() || len(obs.Claim.CancelledCreates) != 1 || len(obs.Claim.RetiredIncarnations) != 1 { |
| + t.Fatalf("cancel: %+v", obs.Claim) |
| + } |
| + f.now = f.now.Add(2 * time.Hour) |
| + if _, err := f.store.Put(f.ctx, r.Holder.Key, r.Body, ""); err != nil { |
| + t.Fatal(err) |
| + } |
| + late, err := ReadRecord(f.ctx, f.store, r.Holder.Key) |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + removed, err := RemoveOrphan(f.ctx, f.store, late) |
| + if err != nil || !removed { |
| + t.Fatalf("late orphan: %t %v", removed, err) |
| + } |
| +} |
| +func TestCreateRecoveryReadErrorsNeverCancel(t *testing.T) { |
| + f := newFixture(t) |
| + r := f.record(t, "shop.bab.example.com") |
| + before, _ := hostclaim.Read(f.ctx, f.store, r.Holder.Host) |
| + obs, err := hostclaim.Update(f.ctx, f.store, before, hostclaim.BeginCreate(r.Holder, r.Body, "create", "test", f.now)) |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + f.now = f.now.Add(6 * time.Minute) |
| + f.store.failGet = r.Holder.Key |
| + if err := f.c.RecoverCreate(f.ctx, r.Holder.Host); err == nil { |
| + t.Fatal("read error accepted") |
| + } |
| + got, _ := hostclaim.Read(f.ctx, f.store, r.Holder.Host) |
| + if got.ETag != obs.ETag { |
| + t.Fatal("acted after read error") |
| + } |
| + f.store.failGet = "" |
| + if _, err := f.store.Put(f.ctx, r.Holder.Key, r.Body, ""); err != nil { |
| + t.Fatal(err) |
| + } |
| + if err := f.c.RecoverCreate(f.ctx, r.Holder.Host); err != nil { |
| + t.Fatal(err) |
| + } |
| + got, _ = hostclaim.Read(f.ctx, f.store, r.Holder.Host) |
| + if got.Claim.Op.Kind != hostclaim.KindServing || got.Claim.Holder.Stamp != got.Claim.Gen { |
| + t.Fatal("not completed") |
| + } |
| +} |
| +func TestRetireWaitsForExactRecordAndAuthority(t *testing.T) { |
| + f := newFixture(t) |
| + obs := f.serving(t, "shop.bab.example.com") |
| + key := obs.Claim.Holder.Key |
| + deleting, err := f.c.BeginRetire(f.ctx, key, "test", obs.Claim.Holder.Incarnation) |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + f.auth.err = ErrPending |
| + if err := f.c.Retire(f.ctx, obs.Host, deleting.Claim.Op.ID); !errors.Is(err, ErrPending) { |
| + t.Fatal(err) |
| + } |
| + if _, _, err := f.store.Get(f.ctx, key); err != nil { |
| + t.Fatal("record deleted before barrier") |
| + } |
| + f.auth.err = nil |
| + f.store.beforeDelete = func(k, e string) error { return release.ErrCASConflict } |
| + if err := f.c.Retire(f.ctx, obs.Host, deleting.Claim.Op.ID); !errors.Is(err, release.ErrCASConflict) { |
| + t.Fatal(err) |
| + } |
| + current, _ := hostclaim.Read(f.ctx, f.store, obs.Host) |
| + if current.Claim.Op.Kind != hostclaim.KindDeleting { |
| + t.Fatal("parked after 412") |
| + } |
| + f.store.beforeDelete = nil |
| + if err := f.c.Retire(f.ctx, obs.Host, deleting.Claim.Op.ID); err != nil { |
| + t.Fatal(err) |
| + } |
| + current, _ = hostclaim.Read(f.ctx, f.store, obs.Host) |
| + if current.Claim.Op.Kind != hostclaim.KindParking || len(current.Claim.Obligations) != 2 { |
| + t.Fatalf("parking: %+v", current.Claim) |
| + } |
| + if err := f.c.Free(f.ctx, obs.Host, deleting.Claim.Op.ID); err == nil { |
| + t.Fatal("freed with chores") |
| + } |
| + for _, ob := range current.Claim.Obligations { |
| + var err error |
| + current, err = hostclaim.Update(f.ctx, f.store, current, hostclaim.CompleteObligation(ob.Op, ob.ID)) |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + } |
| + if err := f.c.Free(f.ctx, obs.Host, deleting.Claim.Op.ID); err != nil { |
| + t.Fatal(err) |
| + } |
| + if f.store.bareDeletes != 0 { |
| + t.Fatal("bare delete") |
| + } |
| +} |
| +func TestAdoptionIsNotOrphanAndHasDiscovery(t *testing.T) { |
| + f := newFixture(t) |
| + r := f.record(t, "shop.bab.example.com") |
| + if _, err := f.store.Put(f.ctx, r.Holder.Key, r.Body, ""); err != nil { |
| + t.Fatal(err) |
| + } |
| + read, err := ReadRecord(f.ctx, f.store, r.Holder.Key) |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + if removed, err := RemoveOrphan(f.ctx, f.store, read); err != nil || removed { |
| + t.Fatal("legacy treated as orphan") |
| + } |
| + obs, err := f.c.Adopt(f.ctx, read) |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + if obs.Claim.Touched.Complete || len(obs.Claim.Obligations) != 2 || obs.Claim.Holder.Location != "instances/"+r.Holder.Host || obs.Claim.Holder.WriterBox != "box-a" { |
| + t.Fatalf("adopt: %+v", obs.Claim) |
| + } |
| + if _, err := f.c.BeginRetire(f.ctx, r.Holder.Key, "test", obs.Claim.Holder.Incarnation); err == nil { |
| + t.Fatal("delete overlapped discovery") |
| + } |
| +} |
| +func TestPurgeJournalPhasesAndFenceRetention(t *testing.T) { |
| + f := newFixture(t) |
| + p, _, err := ReservePurge(f.ctx, f.store, "bab", "jam", "purge") |
| + if err != nil || p.Phase != "reserved" { |
| + t.Fatal(err) |
| + } |
| + if _, _, err := ReservePurge(f.ctx, f.store, "bab", "jam", "other"); !errors.Is(err, ErrPending) { |
| + t.Fatal(err) |
| + } |
| + if err := RemovePurge(f.ctx, f.store, "bab", "jam", "purge"); !errors.Is(err, ErrPending) { |
| + t.Fatal(err) |
| + } |
| + if err := AdvancePurge(f.ctx, f.store, "bab", "jam", "purge", "reserved", "done"); err == nil { |
| + t.Fatal("skipped phases") |
| + } |
| + for i := 0; i < len(phases)-1; i++ { |
| + if err := AdvancePurge(f.ctx, f.store, "bab", "jam", "purge", phases[i], phases[i+1]); err != nil { |
| + t.Fatal(err) |
| + } |
| + } |
| + if err := RemovePurge(f.ctx, f.store, "bab", "jam", "purge"); err != nil { |
| + t.Fatal(err) |
| + } |
| +} |
| +func TestPurgeRetiresUnderOwnOperation(t *testing.T) { |
| + f := newFixture(t) |
| + obs := f.serving(t, "shop.bab.example.com") |
| + rec, err := ReadRecord(f.ctx, f.store, obs.Claim.Holder.Key) |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + obs, err = hostclaim.Update(f.ctx, f.store, obs, BeginPurge("purge", "test", f.now, []string{"ie"}, []string{"box-a"})) |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + obs, err = hostclaim.Update(f.ctx, f.store, obs, PurgeRecordIntent("purge", rec.Holder.Key, rec.ETag)) |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + f.auth.err = ErrPending |
| + if err := f.c.Retire(f.ctx, obs.Host, "purge"); !errors.Is(err, ErrPending) { |
| + t.Fatal(err) |
| + } |
| + f.auth.err = nil |
| + if err := f.c.Retire(f.ctx, obs.Host, "purge"); err != nil { |
| + t.Fatal(err) |
| + } |
| + got, _ := hostclaim.Read(f.ctx, f.store, obs.Host) |
| + if got.Claim.Op.Kind != hostclaim.KindParking || got.Claim.Op.ID != "purge" { |
| + t.Fatal("purge identity lost") |
| + } |
| +} |
| +func TestPrepareRejectsCallerTimeAndSidecarIdentity(t *testing.T) { |
| + f := newFixture(t) |
| + if _, err := PrepareInstance(provision.Record{CreatedAt: f.now}, f.now, "ie"); err == nil { |
| + t.Fatal("accepted caller time") |
| + } |
| + r, err := PrepareSidecar(provision.SidecarRecord{Account: "bab", App: "jam", Name: "job", Host: provision.SidecarHost("job", "jam", "bab"), Region: "ie"}, f.now, "ie") |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + before, _ := hostclaim.Read(f.ctx, f.store, r.Holder.Host) |
| + obs, err := f.c.Attempt(f.ctx, before, r, "sidecar", "test") |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + if obs.Claim.Holder.Family != "sidecar" { |
| + t.Fatal("sidecar became instance") |
| + } |
| +} |
| +func TestRecoveryFreshReadsAndRequiredHooks(t *testing.T) { |
| + f := newFixture(t) |
| + if err := (Recovery{Controller: f.c}).Pass(f.ctx); !errors.Is(err, ErrContract) { |
| + t.Fatal(err) |
| + } |
| + calls := 0 |
| + r := Recovery{Controller: f.c, Purge: func(context.Context, hostclaim.Observed) error { return nil }, Placement: func(context.Context, hostclaim.Observed) error { return nil }, SweepRecords: func(context.Context) error { calls++; return nil }, SweepPurges: func(context.Context) error { calls++; return nil }, SweepAdmissions: func(context.Context) error { return nil }} |
| + if err := r.Pass(f.ctx); err != nil { |
| + t.Fatal(err) |
| + } |
| + if calls != 2 { |
| + t.Fatal(calls) |
| + } |
| +} |
| + |
| +func TestOrphanConditionalDeleteCannotEraseReplacement(t *testing.T) { |
| + f := newFixture(t) |
| + r := f.record(t, "shop.bab.example.com") |
| + if _, err := f.store.Put(f.ctx, r.Holder.Key, r.Body, ""); err != nil { |
| + t.Fatal(err) |
| + } |
| + old, err := ReadRecord(f.ctx, f.store, r.Holder.Key) |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + absent, _ := hostclaim.Read(f.ctx, f.store, r.Holder.Host) |
| + if _, err := hostclaim.Update(f.ctx, f.store, absent, func(*hostclaim.Claim) error { return nil }); err != nil { |
| + t.Fatal(err) |
| + } |
| + replacement := append([]byte(nil), r.Body...) |
| + replacement = append(replacement, ' ') |
| + f.store.beforeDelete = func(key, etag string) error { |
| + _, err := f.store.Store.Put(f.ctx, key, replacement, etag) |
| + return err |
| + } |
| + if removed, err := RemoveOrphan(f.ctx, f.store, old); removed || !errors.Is(err, release.ErrCASConflict) { |
| + t.Fatalf("removed=%t err=%v", removed, err) |
| + } |
| + if _, _, err := f.store.Get(f.ctx, r.Holder.Key); err != nil { |
| + t.Fatal("replacement lost") |
| + } |
| +} |
| + |
| +func TestPlacementStartKeepsInstallationAndExcludesDelete(t *testing.T) { |
| + f := newFixture(t) |
| + obs := f.serving(t, "shop.bab.example.com") |
| + stamp := obs.Claim.Holder.Stamp |
| + placed, err := hostclaim.Update(f.ctx, f.store, obs, BeginPlacement("place", "test", "fleet/customer", f.now, []string{"ie", "us"}, []string{"box-a", "box-b"})) |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + if placed.Claim.Op.Exec != obs.Claim.Op.Exec || placed.Claim.Holder.Stamp != stamp || placed.Claim.Op.Intent["target"] != "fleet/customer" { |
| + t.Fatal("placement changed authority") |
| + } |
| + rec, err := ReadRecord(f.ctx, f.store, obs.Claim.Holder.Key) |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + if _, err := hostclaim.Update(f.ctx, f.store, placed, hostclaim.BeginDelete("delete", "test", rec.ETag, f.now, nil, nil)); err == nil { |
| + t.Fatal("delete overlapped placement") |
| + } |
| +} |
| + |
| +func TestFenceReleasePreservesHistoryAndRefusesAnotherPurge(t *testing.T) { |
| + f := newFixture(t) |
| + before, _ := hostclaim.Read(f.ctx, f.store, "shop.bab.example.com") |
| + obs, err := hostclaim.Update(f.ctx, f.store, before, func(c *hostclaim.Claim) error { |
| + c.Lost = []string{"lost-box"} |
| + c.QuietAfter = map[string]time.Time{"ie": f.now.Add(time.Hour)} |
| + return FencePurge("purge", f.now)(c) |
| + }) |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + if _, err := hostclaim.Update(f.ctx, f.store, obs, ReleasePurgeFence("other")); err == nil { |
| + t.Fatal("released other's fence") |
| + } |
| + obs, err = hostclaim.Update(f.ctx, f.store, obs, ReleasePurgeFence("purge")) |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + if !obs.Claim.Free() || len(obs.Claim.Lost) != 1 || obs.Claim.QuietAfter["ie"].IsZero() { |
| + t.Fatal("lost history") |
| + } |
| +} |
| +``` |
| + |
| +**`internal/provision/retirement_test.go`** |
| + |
| +```go |
| +package provision |
| + |
| +import ( |
| + "context" |
| + "errors" |
| + "testing" |
| + |
| + "github.com/carlosframework/platform/internal/release" |
| +) |
| + |
| +type retireSpy struct{ calls int } |
| + |
| +func (s *retireSpy) RetireRecord(context.Context, RetireRef) error { |
| + s.calls++ |
| + return errors.New("claim path") |
| +} |
| +func TestRetireRecordFlagOffAndRequiredProvider(t *testing.T) { |
| + ctx := context.Background() |
| + store := release.NewDirStore(t.TempDir()) |
| + spy := &retireSpy{} |
| + for _, family := range []string{"instance", "sidecar"} { |
| + ref := RetireRef{Family: family, Account: "bab", App: "jam", Host: "shop.bab.example.com"} |
| + if family == "sidecar" { |
| + ref.Name = "job" |
| + ref.Host = SidecarHost(ref.Name, ref.App, ref.Account) |
| + } |
| + if err := RetireRecord(ctx, store, ref, RetireOptions{Claims: spy}); err != nil { |
| + t.Fatal(err) |
| + } |
| + if spy.calls != 0 { |
| + t.Fatal("flag off called claims") |
| + } |
| + if err := RetireRecord(ctx, store, ref, RetireOptions{HostClaim: true}); !errors.Is(err, ErrClaimRetirer) { |
| + t.Fatal(err) |
| + } |
| + } |
| + ref := RetireRef{Family: "instance", Account: "bab", App: "jam", Host: "shop.bab.example.com"} |
| + if err := RetireRecord(ctx, store, ref, RetireOptions{HostClaim: true, Claims: spy}); err == nil || spy.calls != 1 { |
| + t.Fatal("claim path not required") |
| + } |
| +} |
| +``` |
| + |
| +**`internal/console/hostclaimops_test.go`** |
| + |
| +```go |
| +package console |
| + |
| +import ( |
| + "context" |
| + "errors" |
| + "testing" |
| + |
| + "github.com/carlosframework/platform/internal/hostops" |
| + "github.com/carlosframework/platform/internal/provision" |
| + "github.com/carlosframework/platform/internal/release" |
| +) |
| + |
| +func TestClaimRecoveryAndRetirementAreInert(t *testing.T) { |
| + s := &Server{cfg: Config{Releases: release.NewDirStore(t.TempDir())}} |
| + ctx := context.Background() |
| + if err := s.claimRecoveryPass(ctx); err != nil { |
| + t.Fatal(err) |
| + } |
| + s.StartClaimRecoveryPass(ctx) // Must return without a ticker or store reads. |
| + ref := provision.RetireRef{Family: "instance", Account: "bab", App: "jam", Host: "shop.bab.example.com"} |
| + if err := s.retireHostRecord(ctx, ref); err != nil { |
| + t.Fatal(err) |
| + } |
| + s.cfg.HostClaim = true |
| + if err := s.claimRecoveryPass(ctx); !errors.Is(err, hostops.ErrContract) { |
| + t.Fatal(err) |
| + } |
| + if err := s.retireHostRecord(ctx, ref); !errors.Is(err, provision.ErrClaimRetirer) { |
| + t.Fatal(err) |
| + } |
| +} |
| +``` |
| + |
| +- [ ] **Step 3: Run the new tests before implementation.** |
| + |
| +```bash |
| +go test ./internal/hostops ./internal/provision ./internal/console -run 'AttemptUses|RecoverCreate|CreateRecovery|RetireRecordFlagOff|RetireWaits|AdoptionIs|PurgeJournal|PurgeRetires|PrepareRejects|RecoveryFresh|OrphanConditional|PlacementStart|FenceRelease|ClaimRecoveryAnd' -count=1 |
| +``` |
| + |
| +Expected: compilation fails on the missing `Controller`, `RetireRecord`, `Recovery` and server helper names. Do not alter the assertions to make an existing unguarded path pass. After the code tasks below, the same command must pass. Add the real door/coordinator tests in Tasks 5–9 before their respective integrations; these primitive tests do not stand in for them. |
| + |
| +### Task 2: Add complete record, transition and recovery primitives |
| + |
| +**Files:** the seven complete files below. The dependency interfaces deliberately return errors rather than grant authority when absent. `Authority` methods are semantic contracts defined by D1–D3/D6; they are not implementations of those contracts. |
| + |
| +- [ ] **Step 1: Add `record.go`.** Record parsing is additive for existing fields, rejects trailing data, and binds the body to the exact key. Preserve the original bytes/ETag rather than reserializing a record before conditional deletion. `DecodeRecord` retains main's maintenance-projection validation. `PrepareInstance` and `PrepareSidecar` take internal unstamped records, not user-provided creation times. |
| + |
| +**`internal/hostops/record.go`** |
| + |
| +```go |
| +package hostops |
| + |
| +import ( |
| + "bytes" |
| + "context" |
| + "encoding/json" |
| + "errors" |
| + "io" |
| + "os" |
| + "strings" |
| + "time" |
| + |
| + "github.com/carlosframework/platform/internal/controlname" |
| + "github.com/carlosframework/platform/internal/hostclaim" |
| + "github.com/carlosframework/platform/internal/provision" |
| + "github.com/carlosframework/platform/internal/release" |
| +) |
| + |
| +var ( |
| + ErrPending = errors.New("hostops: pending") |
| + ErrContract = errors.New("hostops: required contract unavailable") |
| + ErrIdentity = errors.New("hostops: identity mismatch") |
| +) |
| + |
| +type Record struct { |
| + Holder hostclaim.NewHolder |
| + Body []byte |
| + ETag string |
| +} |
| + |
| +func decode(body []byte, out any) error { |
| + d := json.NewDecoder(bytes.NewReader(body)) |
| + if err := d.Decode(out); err != nil { |
| + return err |
| + } |
| + if _, err := d.Token(); err != io.EOF { |
| + return ErrIdentity |
| + } |
| + return nil |
| +} |
| + |
| +func ReadRecord(ctx context.Context, s release.Store, key string) (Record, error) { |
| + parts := strings.Split(key, "/") |
| + if len(parts) != 5 || parts[0] != "apps" || !controlname.ValidAccount(parts[1]) || !controlname.ValidApp(parts[2]) || !strings.HasSuffix(parts[4], ".json") { |
| + return Record{}, ErrIdentity |
| + } |
| + body, etag, err := s.Get(ctx, key) |
| + if err != nil { |
| + return Record{}, err |
| + } |
| + if etag == "" { |
| + return Record{}, ErrIdentity |
| + } |
| + n := hostclaim.NewHolder{Key: key, Account: parts[1], App: parts[2]} |
| + switch parts[3] { |
| + case "instances": |
| + var r provision.Record |
| + if err := decode(body, &r); err != nil { |
| + return Record{}, err |
| + } |
| + if r.Account != n.Account || r.App != n.App || key != provision.RecordKey(r.Account, r.App, r.Host) || !controlname.ValidHostShape(r.Host) { |
| + return Record{}, ErrIdentity |
| + } |
| + // DecodeRecord is also the ordinary instance reader's shape validator. |
| + if _, err := provision.DecodeRecord(body); err != nil { |
| + return Record{}, err |
| + } |
| + n.Family, n.Host, n.RecordRegion, n.CreatedAt = "instance", r.Host, r.Region, r.CreatedAt |
| + case "sidecars": |
| + var r provision.SidecarRecord |
| + if err := decode(body, &r); err != nil { |
| + return Record{}, err |
| + } |
| + if r.Account != n.Account || r.App != n.App || !controlname.ValidLabel(r.Name) || key != provision.SidecarRecordKey(r.Account, r.App, r.Name) || r.Host != provision.SidecarHost(r.Name, r.App, r.Account) { |
| + return Record{}, ErrIdentity |
| + } |
| + n.Family, n.Host, n.Name, n.RecordRegion, n.CreatedAt = "sidecar", r.Host, r.Name, r.Region, r.CreatedAt |
| + default: |
| + return Record{}, ErrIdentity |
| + } |
| + return Record{Holder: n, Body: body, ETag: etag}, nil |
| +} |
| + |
| +func (r Record) Incarnation() (string, error) { |
| + n := r.Holder |
| + return hostclaim.Incarnation(n.Account, n.App, n.Host, n.CreatedAt) |
| +} |
| + |
| +func matches(r Record, h *hostclaim.Holder) bool { |
| + if h == nil { |
| + return false |
| + } |
| + n := r.Holder |
| + inc, err := r.Incarnation() |
| + return err == nil && n.Key == h.Key && n.Family == h.Family && n.Account == h.Account && n.App == h.App && n.Name == h.Name && n.CreatedAt.Equal(h.CreatedAt) && inc == h.Incarnation |
| +} |
| + |
| +// The caller reads the record first. This function performs the later claim read. |
| +func RemoveOrphan(ctx context.Context, s release.Store, r Record) (bool, error) { |
| + inc, err := r.Incarnation() |
| + if err != nil { |
| + return false, err |
| + } |
| + obs, err := hostclaim.Read(ctx, s, r.Holder.Host) |
| + if err != nil { |
| + return false, err |
| + } |
| + if !hostclaim.IsOrphan(inc, obs) { |
| + return false, nil |
| + } |
| + if err := release.DeleteIfMatch(ctx, s, r.Holder.Key, r.ETag); err != nil { |
| + return false, err |
| + } |
| + return true, nil |
| +} |
| + |
| +func Records(ctx context.Context, s release.Store, acct, app string) ([]Record, error) { |
| + if !controlname.ValidAccount(acct) || !controlname.ValidApp(app) { |
| + return nil, ErrIdentity |
| + } |
| + var out []Record |
| + for _, prefix := range []string{provision.RecordsPrefix(acct, app), provision.SidecarsPrefix(acct, app)} { |
| + keys, err := s.List(ctx, prefix) |
| + if err != nil { |
| + return nil, err |
| + } |
| + for _, key := range keys { |
| + tail, ok := strings.CutPrefix(key, prefix) |
| + if !ok { |
| + return nil, ErrIdentity |
| + } |
| + if strings.Contains(tail, "/") || !strings.HasSuffix(tail, ".json") { |
| + continue |
| + } |
| + r, err := ReadRecord(ctx, s, key) |
| + if errors.Is(err, os.ErrNotExist) { |
| + continue |
| + } |
| + if err != nil { |
| + return nil, err |
| + } |
| + out = append(out, r) |
| + } |
| + } |
| + return out, nil |
| +} |
| + |
| +// PrepareInstance accepts only an unstamped internal request, never a caller time. |
| +func PrepareInstance(r provision.Record, now time.Time, region string) (Record, error) { |
| + if !r.CreatedAt.IsZero() || now.IsZero() || region == "" || len(r.Maintenance) != 0 { |
| + return Record{}, ErrIdentity |
| + } |
| + r.CreatedAt = now.UTC() |
| + body, err := json.Marshal(r) |
| + if err != nil { |
| + return Record{}, err |
| + } |
| + n := hostclaim.NewHolder{Family: "instance", Key: provision.RecordKey(r.Account, r.App, r.Host), Account: r.Account, App: r.App, Host: r.Host, RecordRegion: r.Region, Region: region, CreatedAt: r.CreatedAt} |
| + return Record{Holder: n, Body: body}, nil |
| +} |
| + |
| +func PrepareSidecar(r provision.SidecarRecord, now time.Time, region string) (Record, error) { |
| + if !r.CreatedAt.IsZero() || now.IsZero() || region == "" { |
| + return Record{}, ErrIdentity |
| + } |
| + r.CreatedAt = now.UTC() |
| + body, err := json.Marshal(r) |
| + if err != nil { |
| + return Record{}, err |
| + } |
| + n := hostclaim.NewHolder{Family: "sidecar", Key: provision.SidecarRecordKey(r.Account, r.App, r.Name), Account: r.Account, App: r.App, Name: r.Name, Host: r.Host, RecordRegion: r.Region, Region: region, CreatedAt: r.CreatedAt} |
| + return Record{Holder: n, Body: body}, nil |
| +} |
| +``` |
| + |
| +- [ ] **Step 2: Add `operations.go`.** The successful claim CAS precedes the record PUT. `BeginRetire` requires the expected incarnation to prevent a rollback from following a reused key. A post-start failure is returned with the durable operation intact for recovery. `FirstPhase` is Plan 8's acknowledgement check, not a stop-evidence substitute; parking owners still enforce C5. |
| + |
| +**`internal/hostops/operations.go`** |
| + |
| +```go |
| +package hostops |
| + |
| +import ( |
| + "context" |
| + "errors" |
| + "os" |
| + "time" |
| + |
| + "github.com/carlosframework/platform/internal/hostclaim" |
| + "github.com/carlosframework/platform/internal/release" |
| +) |
| + |
| +// These callbacks are authority providers, not optional best-effort hints. |
| +// Production construction waits for the decisions named in this plan. |
| +type Authority interface { |
| + Create(context.Context, hostclaim.Observed, Record, string) error |
| + Executor(context.Context, hostclaim.Observed, Record) (string, error) |
| + StartDelete(context.Context, hostclaim.Observed, Record) error |
| + FirstPhase(context.Context, hostclaim.Observed) error |
| + Inventory(context.Context, hostclaim.Observed) ([]string, []string, error) |
| + Adoption(context.Context, Record) (region, box string, otherRegions, otherBoxes []string, err error) |
| + PurgeRetirement(context.Context, hostclaim.Observed) error |
| +} |
| + |
| +type Controller struct { |
| + Store release.Store |
| + Authority Authority |
| + Now func() time.Time |
| + NewID func() string |
| +} |
| + |
| +func (c Controller) ready() error { |
| + if c.Store == nil || c.Authority == nil || c.Now == nil || c.NewID == nil { |
| + return ErrContract |
| + } |
| + return nil |
| +} |
| + |
| +// Attempt consumes the claim observation taken BEFORE app registration. |
| +// Create's caller owns registration renewal/removal. A failed CAS is never retried here. |
| +func (c Controller) Attempt(ctx context.Context, before hostclaim.Observed, r Record, id, by string) (hostclaim.Observed, error) { |
| + if err := c.ready(); err != nil { |
| + return hostclaim.Observed{}, err |
| + } |
| + if before.Host != r.Holder.Host || r.Holder.CreatedAt.IsZero() || id == "" { |
| + return hostclaim.Observed{}, ErrIdentity |
| + } |
| + // Last pre-CAS read of registration, open admission and purge reservation. |
| + if err := c.Authority.Create(ctx, before, r, id); err != nil { |
| + return hostclaim.Observed{}, err |
| + } |
| + obs, err := hostclaim.Update(ctx, c.Store, before, hostclaim.BeginCreate(r.Holder, r.Body, id, by, c.Now())) |
| + if err != nil { |
| + return hostclaim.Observed{}, err |
| + } |
| + // No raw record effect precedes the durable claim intent. |
| + if _, err = c.Store.Put(ctx, r.Holder.Key, r.Body, ""); err != nil { |
| + if !errors.Is(err, release.ErrCASConflict) { |
| + return obs, err |
| + } |
| + existing, rerr := ReadRecord(ctx, c.Store, r.Holder.Key) |
| + if rerr != nil { |
| + return obs, rerr |
| + } |
| + if !matches(existing, obs.Claim.Holder) { |
| + removed, rerr := RemoveOrphan(ctx, c.Store, existing) |
| + if rerr != nil { |
| + return obs, rerr |
| + } |
| + if !removed { |
| + return obs, ErrPending |
| + } |
| + if _, err = c.Store.Put(ctx, r.Holder.Key, r.Body, ""); err != nil { |
| + return obs, err |
| + } |
| + } |
| + } |
| + return c.Complete(ctx, obs.Host, id) |
| +} |
| + |
| +func (c Controller) Complete(ctx context.Context, host, id string) (hostclaim.Observed, error) { |
| + if err := c.ready(); err != nil { |
| + return hostclaim.Observed{}, err |
| + } |
| + obs, err := hostclaim.Read(ctx, c.Store, host) |
| + if err != nil { |
| + return obs, err |
| + } |
| + if obs.Absent || obs.Claim.Op.Kind != hostclaim.KindCreating || obs.Claim.Op.ID != id { |
| + return obs, ErrPending |
| + } |
| + r, err := ReadRecord(ctx, c.Store, obs.Claim.Holder.Key) |
| + if err != nil { |
| + return obs, err |
| + } |
| + if !matches(r, obs.Claim.Holder) { |
| + return obs, ErrIdentity |
| + } |
| + box, err := c.Authority.Executor(ctx, obs, r) |
| + if err != nil { |
| + return obs, err |
| + } |
| + return hostclaim.Update(ctx, c.Store, obs, hostclaim.CompleteCreate(id, box)) |
| +} |
| + |
| +func (c Controller) RecoverCreate(ctx context.Context, host string) error { |
| + if err := c.ready(); err != nil { |
| + return err |
| + } |
| + obs, err := hostclaim.Read(ctx, c.Store, host) |
| + if err != nil { |
| + return err |
| + } |
| + if obs.Absent || obs.Claim.Op.Kind != hostclaim.KindCreating { |
| + return nil |
| + } |
| + if obs.Claim.Op.StartedAt.IsZero() || !c.Now().After(obs.Claim.Op.StartedAt.Add(5*time.Minute)) { |
| + return ErrPending |
| + } |
| + r, err := ReadRecord(ctx, c.Store, obs.Claim.Holder.Key) |
| + if errors.Is(err, os.ErrNotExist) { |
| + _, err = hostclaim.Update(ctx, c.Store, obs, hostclaim.CancelCreate(obs.Claim.Op.ID, c.Now())) |
| + return err |
| + } |
| + if err != nil { |
| + return err |
| + } |
| + if !matches(r, obs.Claim.Holder) { |
| + return ErrPending |
| + } |
| + _, err = c.Complete(ctx, host, obs.Claim.Op.ID) |
| + return err |
| +} |
| + |
| +// Adopt is the shared IO wrapper around the one hostclaim.Adopt transition. |
| +func (c Controller) Adopt(ctx context.Context, r Record) (hostclaim.Observed, error) { |
| + if err := c.ready(); err != nil { |
| + return hostclaim.Observed{}, err |
| + } |
| + obs, err := hostclaim.Read(ctx, c.Store, r.Holder.Host) |
| + if err != nil { |
| + return obs, err |
| + } |
| + if !obs.Absent { |
| + if !matches(r, obs.Claim.Holder) { |
| + return obs, ErrIdentity |
| + } |
| + return obs, nil |
| + } |
| + region, box, regions, boxes, err := c.Authority.Adoption(ctx, r) |
| + if err != nil { |
| + return obs, err |
| + } |
| + // Re-read the exact record after discovery; never adopt a stale listing body. |
| + fresh, err := ReadRecord(ctx, c.Store, r.Holder.Key) |
| + if err != nil { |
| + return obs, err |
| + } |
| + if fresh.ETag != r.ETag { |
| + return obs, ErrPending |
| + } |
| + n := fresh.Holder |
| + n.Region = region |
| + return hostclaim.Update(ctx, c.Store, obs, hostclaim.Adopt(n, box, regions, boxes, c.NewID(), c.NewID)) |
| +} |
| + |
| +// BeginRetire only starts the claim operation. Its successful return does not |
| +// mean the record is gone or the host is free; the caller must run/retry Retire. |
| +func (c Controller) BeginRetire(ctx context.Context, key, by, expectedInc string) (hostclaim.Observed, error) { |
| + if err := c.ready(); err != nil { |
| + return hostclaim.Observed{}, err |
| + } |
| + r, err := ReadRecord(ctx, c.Store, key) |
| + if err != nil { |
| + return hostclaim.Observed{}, err |
| + } |
| + inc, ierr := r.Incarnation() |
| + if ierr != nil { |
| + return hostclaim.Observed{}, ierr |
| + } |
| + if expectedInc == "" || inc != expectedInc { |
| + return hostclaim.Observed{}, ErrIdentity |
| + } |
| + obs, err := c.Adopt(ctx, r) |
| + if err != nil { |
| + return obs, err |
| + } |
| + if err := c.Authority.StartDelete(ctx, obs, r); err != nil { |
| + return obs, err |
| + } |
| + regions, boxes, err := c.Authority.Inventory(ctx, obs) |
| + if err != nil { |
| + return obs, err |
| + } |
| + return hostclaim.Update(ctx, c.Store, obs, hostclaim.BeginDelete(c.NewID(), by, r.ETag, c.Now(), regions, boxes)) |
| +} |
| + |
| +func (c Controller) Retire(ctx context.Context, host, id string) error { |
| + if err := c.ready(); err != nil { |
| + return err |
| + } |
| + obs, err := hostclaim.Read(ctx, c.Store, host) |
| + if err != nil { |
| + return err |
| + } |
| + if obs.Absent || obs.Claim.Op.ID != id { |
| + return ErrPending |
| + } |
| + op := obs.Claim.Op |
| + if op.Kind != hostclaim.KindDeleting && op.Kind != hostclaim.KindPurging { |
| + return ErrPending |
| + } |
| + if obs.Claim.Holder == nil || op.Intent["key"] != obs.Claim.Holder.Key || op.Intent["etag"] == "" { |
| + return ErrIdentity |
| + } |
| + if err := c.Authority.FirstPhase(ctx, obs); err != nil { |
| + return err |
| + } |
| + if op.Kind == hostclaim.KindPurging { |
| + if err := c.Authority.PurgeRetirement(ctx, obs); err != nil { |
| + return err |
| + } |
| + } |
| + regions, boxes, err := c.Authority.Inventory(ctx, obs) |
| + if err != nil { |
| + return err |
| + } |
| + // Validate record identity before deleting even the stored version. |
| + r, rerr := ReadRecord(ctx, c.Store, op.Intent["key"]) |
| + if rerr != nil && !errors.Is(rerr, os.ErrNotExist) { |
| + return rerr |
| + } |
| + if rerr == nil { |
| + if r.ETag != op.Intent["etag"] || !matches(r, obs.Claim.Holder) { |
| + return ErrPending |
| + } |
| + if err := release.DeleteIfMatch(ctx, c.Store, r.Holder.Key, r.ETag); err != nil { |
| + return err |
| + } |
| + } |
| + // A success/412 alone never certifies absence. Re-read before parking. |
| + _, _, err = c.Store.Get(ctx, op.Intent["key"]) |
| + if !errors.Is(err, os.ErrNotExist) { |
| + if err != nil { |
| + return err |
| + } |
| + return ErrPending |
| + } |
| + // The old observation binds the entire sequence to this operation. |
| + _, err = hostclaim.Update(ctx, c.Store, obs, hostclaim.ToParking(id, regions, boxes, c.NewID)) |
| + return err |
| +} |
| + |
| +func (c Controller) Free(ctx context.Context, host, id string) error { |
| + if err := c.ready(); err != nil { |
| + return err |
| + } |
| + obs, err := hostclaim.Read(ctx, c.Store, host) |
| + if err != nil { |
| + return err |
| + } |
| + if obs.Absent { |
| + return ErrPending |
| + } |
| + _, err = hostclaim.Update(ctx, c.Store, obs, hostclaim.FreeAfterPark(id)) |
| + return err |
| +} |
| +``` |
| + |
| +- [ ] **Step 3: Add `purge.go`.** The pure transitions operate only inside `hostclaim.Update`. The journal envelope and ordered phases are specified; the full D4 coordinator must satisfy each phase precondition before invoking `AdvancePurge`, and verify the matching `done` journal before fence release. `AdvancePurge` preserves approved additive journal fields. No function deletes a claim. Placement starts retain executor/stamp and have no invented completion path. |
| + |
| +**`internal/hostops/purge.go`** |
| + |
| +```go |
| +package hostops |
| + |
| +import ( |
| + "context" |
| + "encoding/json" |
| + "errors" |
| + "os" |
| + "time" |
| + |
| + "github.com/carlosframework/platform/internal/controlname" |
| + "github.com/carlosframework/platform/internal/hostclaim" |
| + "github.com/carlosframework/platform/internal/release" |
| +) |
| + |
| +type Purge struct { |
| + ID string `json:"id"` |
| + Phase string `json:"phase"` |
| +} |
| + |
| +var phases = []string{"reserved", "adopted", "claimed", "torn-down", "retired", "done"} |
| + |
| +func phaseIndex(s string) int { |
| + for i, p := range phases { |
| + if s == p { |
| + return i |
| + } |
| + } |
| + return -1 |
| +} |
| +func PurgeKey(acct, app string) (string, error) { |
| + if !controlname.ValidAccount(acct) || !controlname.ValidApp(app) { |
| + return "", ErrIdentity |
| + } |
| + return "apps/" + acct + "/" + app + "/purge.json", nil |
| +} |
| +func ReadPurge(ctx context.Context, s release.Store, acct, app string) (Purge, string, error) { |
| + key, err := PurgeKey(acct, app) |
| + if err != nil { |
| + return Purge{}, "", err |
| + } |
| + body, etag, err := s.Get(ctx, key) |
| + if err != nil { |
| + return Purge{}, "", err |
| + } |
| + var p Purge |
| + if err := decode(body, &p); err != nil { |
| + return Purge{}, "", err |
| + } |
| + if p.ID == "" || phaseIndex(p.Phase) < 0 || etag == "" { |
| + return Purge{}, "", ErrIdentity |
| + } |
| + return p, etag, nil |
| +} |
| +func ReservePurge(ctx context.Context, s release.Store, acct, app, id string) (Purge, string, error) { |
| + key, err := PurgeKey(acct, app) |
| + if err != nil || id == "" { |
| + return Purge{}, "", ErrIdentity |
| + } |
| + p := Purge{ID: id, Phase: "reserved"} |
| + body, err := json.Marshal(p) |
| + if err != nil { |
| + return Purge{}, "", err |
| + } |
| + etag, err := s.Put(ctx, key, body, "") |
| + if errors.Is(err, release.ErrCASConflict) { |
| + existing, tag, rerr := ReadPurge(ctx, s, acct, app) |
| + if rerr != nil { |
| + return Purge{}, "", rerr |
| + } |
| + if existing.ID != id { |
| + return existing, tag, ErrPending |
| + } |
| + return existing, tag, nil |
| + } |
| + return p, etag, err |
| +} |
| + |
| +// AdvancePurge preserves additional journal fields owned by the final durable |
| +// host-set schema. Preconditions for the next phase belong to the coordinator. |
| +func AdvancePurge(ctx context.Context, s release.Store, acct, app, id, from, to string) error { |
| + if phaseIndex(from) < 0 || phaseIndex(to) != phaseIndex(from)+1 { |
| + return ErrIdentity |
| + } |
| + key, err := PurgeKey(acct, app) |
| + if err != nil { |
| + return err |
| + } |
| + body, etag, err := s.Get(ctx, key) |
| + if err != nil { |
| + return err |
| + } |
| + var p Purge |
| + if err := decode(body, &p); err != nil { |
| + return err |
| + } |
| + if p.ID != id || p.Phase != from || etag == "" { |
| + return ErrPending |
| + } |
| + var fields map[string]json.RawMessage |
| + if err := decode(body, &fields); err != nil { |
| + return err |
| + } |
| + fields["phase"], err = json.Marshal(to) |
| + if err != nil { |
| + return err |
| + } |
| + body, err = json.Marshal(fields) |
| + if err != nil { |
| + return err |
| + } |
| + _, err = s.Put(ctx, key, body, etag) |
| + return err |
| +} |
| + |
| +// Only the completed coordinator may call this, after all fence-release writes. |
| +func RemovePurge(ctx context.Context, s release.Store, acct, app, id string) error { |
| + p, etag, err := ReadPurge(ctx, s, acct, app) |
| + if errors.Is(err, os.ErrNotExist) { |
| + return nil |
| + } |
| + if err != nil { |
| + return err |
| + } |
| + if p.ID != id || p.Phase != "done" { |
| + return ErrPending |
| + } |
| + key, err := PurgeKey(acct, app) |
| + if err != nil { |
| + return err |
| + } |
| + return release.DeleteIfMatch(ctx, s, key, etag) |
| +} |
| + |
| +func FencePurge(id string, now time.Time) func(*hostclaim.Claim) error { |
| + return func(c *hostclaim.Claim) error { |
| + if id == "" || now.IsZero() { |
| + return ErrIdentity |
| + } |
| + if !c.Free() || len(c.Obligations) != 0 { |
| + return ErrPending |
| + } |
| + c.Op = hostclaim.Op{Kind: hostclaim.KindPurging, ID: id, StartedAt: now.UTC()} |
| + return nil |
| + } |
| +} |
| +func ReleasePurgeFence(id string) func(*hostclaim.Claim) error { |
| + return func(c *hostclaim.Claim) error { |
| + if id == "" || c.Holder != nil || c.Op.Kind != hostclaim.KindPurging || c.Op.ID != id || len(c.Obligations) != 0 { |
| + return ErrPending |
| + } |
| + c.Op = hostclaim.Op{} |
| + return nil |
| + } |
| +} |
| +func BeginPurge(id, by string, now time.Time, regions, boxes []string) func(*hostclaim.Claim) error { |
| + return func(c *hostclaim.Claim) error { |
| + if id == "" || now.IsZero() { |
| + return ErrIdentity |
| + } |
| + if err := hostclaim.StartGuard(c, regions, boxes); err != nil { |
| + return err |
| + } |
| + c.Op = hostclaim.Op{Kind: hostclaim.KindPurging, ID: id, By: by, StartedAt: now.UTC(), Exec: c.Op.Exec, Participants: c.Op.Participants} |
| + return nil |
| + } |
| +} |
| +func PurgeRecordIntent(id, key, etag string) func(*hostclaim.Claim) error { |
| + return func(c *hostclaim.Claim) error { |
| + if id == "" || etag == "" || c.Op.Kind != hostclaim.KindPurging || c.Op.ID != id || c.Holder == nil || c.Holder.Key != key { |
| + return ErrIdentity |
| + } |
| + if c.Op.Intent == nil { |
| + c.Op.Intent = map[string]string{} |
| + } |
| + if old := c.Op.Intent["etag"]; old != "" && old != etag { |
| + return ErrPending |
| + } |
| + c.Op.Intent["key"], c.Op.Intent["etag"] = key, etag |
| + return nil |
| + } |
| +} |
| + |
| +// No completion transition is supplied until placement's executor/evidence |
| +// contract is resolved. This start transition alone grants no new execution. |
| +func BeginPlacement(id, by, target string, now time.Time, regions, boxes []string) func(*hostclaim.Claim) error { |
| + return func(c *hostclaim.Claim) error { |
| + if id == "" || now.IsZero() { |
| + return ErrIdentity |
| + } |
| + if err := hostclaim.StartGuard(c, regions, boxes); err != nil { |
| + return err |
| + } |
| + c.Op = hostclaim.Op{Kind: hostclaim.KindPlacing, ID: id, By: by, StartedAt: now.UTC(), Exec: c.Op.Exec, Participants: c.Op.Participants, Intent: map[string]string{"target": target}} |
| + return nil |
| + } |
| +} |
| +``` |
| + |
| +- [ ] **Step 4: Add `recovery.go`.** The pass lists direct host objects for discovery, then freshly reads each claim. A failed list returns no partial authority. Record sweeps cover instance and sidecar keys, including records written after the cancellation hint expires. Purge sweeps find reservations even before any host was claimed. Per-app record collection finishes before actions on that app, so a malformed sibling record cannot be silently omitted from the collected set. |
| + |
| +**`internal/hostops/recovery.go`** |
| + |
| +```go |
| +package hostops |
| + |
| +import ( |
| + "context" |
| + "errors" |
| + "os" |
| + "strings" |
| + "time" |
| + |
| + "github.com/carlosframework/platform/internal/hostclaim" |
| + "github.com/carlosframework/platform/internal/release" |
| +) |
| + |
| +// ListClaims is discovery only. Every action subsequently reads its own key. |
| +func ListClaims(ctx context.Context, s release.Store) ([]string, error) { |
| + keys, err := s.List(ctx, "control/hosts/") |
| + if err != nil { |
| + return nil, err |
| + } |
| + var hosts []string |
| + for _, key := range keys { |
| + tail, ok := strings.CutPrefix(key, "control/hosts/") |
| + if !ok { |
| + return nil, ErrIdentity |
| + } |
| + if strings.Contains(tail, "/") || !strings.HasSuffix(tail, ".json") { |
| + continue |
| + } |
| + host := strings.TrimSuffix(tail, ".json") |
| + if _, err := hostclaim.Key(host); err != nil { |
| + return nil, err |
| + } |
| + hosts = append(hosts, host) |
| + } |
| + return hosts, nil |
| +} |
| + |
| +type Recovery struct { |
| + SweepAdmissions func(context.Context) error |
| + Controller Controller |
| + // These are required integrations, not no-op defaults. |
| + Purge func(context.Context, hostclaim.Observed) error |
| + Placement func(context.Context, hostclaim.Observed) error |
| + SweepRecords func(context.Context) error |
| + SweepPurges func(context.Context) error |
| +} |
| + |
| +func (r Recovery) Pass(ctx context.Context) error { |
| + c := r.Controller |
| + if err := c.ready(); err != nil { |
| + return err |
| + } |
| + if r.Purge == nil || r.Placement == nil || r.SweepRecords == nil || r.SweepPurges == nil || r.SweepAdmissions == nil { |
| + return ErrContract |
| + } |
| + hosts, err := ListClaims(ctx, c.Store) |
| + if err != nil { |
| + return err |
| + } |
| + var all []error |
| + for _, host := range hosts { |
| + if err := ctx.Err(); err != nil { |
| + return errors.Join(append(all, err)...) |
| + } |
| + obs, err := hostclaim.Read(ctx, c.Store, host) |
| + if err != nil { |
| + all = append(all, err) |
| + continue |
| + } |
| + if obs.Absent { |
| + continue |
| + } |
| + op := obs.Claim.Op |
| + // Zero time is unknown age, never old enough. Freeing empty parking is safe |
| + // independently of age; this includes Plan 4's holderless sweep retries. |
| + if op.Kind == hostclaim.KindParking && len(obs.Claim.Obligations) == 0 { |
| + err = c.Free(ctx, host, op.ID) |
| + } else if op.Kind != "" && op.Kind != hostclaim.KindServing && !op.StartedAt.IsZero() && c.Now().After(op.StartedAt.Add(5*time.Minute)) { |
| + switch op.Kind { |
| + case hostclaim.KindCreating: |
| + err = c.RecoverCreate(ctx, host) |
| + case hostclaim.KindDeleting: |
| + err = c.Retire(ctx, host, op.ID) |
| + case hostclaim.KindPurging: |
| + err = r.Purge(ctx, obs) |
| + case hostclaim.KindPlacing: |
| + err = r.Placement(ctx, obs) |
| + } |
| + } |
| + if err != nil && !errors.Is(err, ErrPending) { |
| + all = append(all, err) |
| + } |
| + } |
| + // Records have their own uncapped scan: late orphans outlive the one-hour |
| + // cancellation hint; records without claims need the shared adoption path. |
| + if err := r.SweepRecords(ctx); err != nil { |
| + all = append(all, err) |
| + } |
| + // Reservations can exist without any claim (crash just after reserve). |
| + if err := r.SweepPurges(ctx); err != nil { |
| + all = append(all, err) |
| + } |
| + if err := r.SweepAdmissions(ctx); err != nil { |
| + all = append(all, err) |
| + } |
| + return errors.Join(all...) |
| +} |
| + |
| +type App struct{ Account, Name string } |
| + |
| +func Apps(ctx context.Context, s release.Store) ([]App, error) { |
| + accounts, err := s.ListDirs(ctx, "apps/") |
| + if err != nil { |
| + return nil, err |
| + } |
| + var out []App |
| + for _, prefix := range accounts { |
| + account := strings.TrimSuffix(strings.TrimPrefix(prefix, "apps/"), "/") |
| + if _, err := PurgeKey(account, "valid"); err != nil || prefix != "apps/"+account+"/" { |
| + return nil, ErrIdentity |
| + } |
| + apps, err := s.ListDirs(ctx, prefix) |
| + if err != nil { |
| + return nil, err |
| + } |
| + for _, ap := range apps { |
| + app := strings.TrimSuffix(strings.TrimPrefix(ap, prefix), "/") |
| + if _, err := PurgeKey(account, app); err != nil || ap != prefix+app+"/" { |
| + return nil, ErrIdentity |
| + } |
| + out = append(out, App{Account: account, Name: app}) |
| + } |
| + } |
| + return out, nil |
| +} |
| + |
| +func (c Controller) SweepAllRecords(ctx context.Context) error { |
| + if err := c.ready(); err != nil { |
| + return err |
| + } |
| + apps, err := Apps(ctx, c.Store) |
| + if err != nil { |
| + return err |
| + } |
| + var all []error |
| + for _, app := range apps { |
| + records, err := Records(ctx, c.Store, app.Account, app.Name) |
| + if err != nil { |
| + all = append(all, err) |
| + continue |
| + } |
| + for _, listed := range records { |
| + record, err := ReadRecord(ctx, c.Store, listed.Holder.Key) |
| + if errors.Is(err, os.ErrNotExist) { |
| + continue |
| + } |
| + if err != nil { |
| + all = append(all, err) |
| + continue |
| + } |
| + obs, err := hostclaim.Read(ctx, c.Store, record.Holder.Host) |
| + if err != nil { |
| + all = append(all, err) |
| + continue |
| + } |
| + if obs.Absent { |
| + _, err = c.Adopt(ctx, record) |
| + } else { |
| + _, err = RemoveOrphan(ctx, c.Store, record) |
| + } |
| + if err != nil { |
| + all = append(all, err) |
| + } |
| + } |
| + } |
| + return errors.Join(all...) |
| +} |
| + |
| +func SweepAllPurges(ctx context.Context, s release.Store, replay func(context.Context, App, Purge) error) error { |
| + if replay == nil { |
| + return ErrContract |
| + } |
| + apps, err := Apps(ctx, s) |
| + if err != nil { |
| + return err |
| + } |
| + var all []error |
| + for _, app := range apps { |
| + p, _, err := ReadPurge(ctx, s, app.Account, app.Name) |
| + if errors.Is(err, os.ErrNotExist) { |
| + continue |
| + } |
| + if err != nil { |
| + all = append(all, err) |
| + continue |
| + } |
| + if err := replay(ctx, app, p); err != nil { |
| + all = append(all, err) |
| + } |
| + } |
| + return errors.Join(all...) |
| +} |
| +``` |
| + |
| +- [ ] **Step 5: Add the exact Plan 4 regional adapter.** `RunOwnedRemote` checks the chore's own op and ID, nominated region and owner before binding scope and running the engine. It does not acknowledge anything itself. All destructive storage mechanics remain inside `internal/park`. |
| + |
| +**`internal/hostops/remote.go`** |
| + |
| +```go |
| +package hostops |
| + |
| +import ( |
| + "context" |
| + |
| + "github.com/carlosframework/platform/internal/hostclaim" |
| + "github.com/carlosframework/platform/internal/park" |
| +) |
| + |
| +type ParkScopeProvider func(context.Context, hostclaim.Observed, hostclaim.Obligation) (hostclaim.ParkScope, error) |
| + |
| +// RunOwnedRemote is called only by the appointed regional worker. It never |
| +// takes another region's chore or acknowledges local work on another box. |
| +func RunOwnedRemote(ctx context.Context, e *park.Engine, w park.Work, scope ParkScopeProvider) error { |
| + if e == nil || e.Claims == nil || e.Region == "" || scope == nil || e.Authorize == nil || e.View == nil || e.Leases == nil { |
| + return ErrContract |
| + } |
| + obs, err := hostclaim.Read(ctx, e.Claims, w.Host) |
| + if err != nil { |
| + return err |
| + } |
| + if obs.Absent { |
| + return ErrPending |
| + } |
| + for _, ob := range obs.Claim.Obligations { |
| + if ob.ID != w.ID || ob.Op != w.Op { |
| + continue |
| + } |
| + if ob.Region != e.Region || ob.Owner != "region:"+e.Region || (ob.Kind != "park-replica" && ob.Kind != "park-generation") { |
| + return ErrIdentity |
| + } |
| + if err := e.Authorize(ctx, obs, ob); err != nil { |
| + return err |
| + } |
| + if ob.Park == nil { |
| + p, err := scope(ctx, obs, ob) |
| + if err != nil { |
| + return err |
| + } |
| + if err := e.Bind(ctx, w, p); err != nil { |
| + return err |
| + } |
| + } |
| + return e.Run(ctx, w) |
| + } |
| + return ErrPending |
| +} |
| +``` |
| + |
| +### Task 3: Introduce the shared retirement entry point |
| + |
| +- [ ] **Step 1: Add the explicit provisioning dispatcher.** The flag is a passed startup value. Nil claims with the flag enabled is a refusal, never an implicit legacy mode. Flag-off continues through the unchanged `DeleteRecord`/`DeleteSidecarRecord` primitives. |
| + |
| +**`internal/provision/retirement.go`** |
| + |
| +```go |
| +package provision |
| + |
| +import ( |
| + "context" |
| + "errors" |
| + |
| + "github.com/carlosframework/platform/internal/controlname" |
| + "github.com/carlosframework/platform/internal/release" |
| +) |
| + |
| +type RetireRef struct { |
| + Family, Account, App, Host, Name, By string |
| + // Required by rollbacks: never retire a successor to the failed attempt. |
| + Incarnation string |
| +} |
| +type ClaimRetirer interface { |
| + RetireRecord(context.Context, RetireRef) error |
| +} |
| +type RetireOptions struct { |
| + HostClaim bool |
| + Claims ClaimRetirer |
| +} |
| + |
| +var ErrClaimRetirer = errors.New("provision: claim retirement unavailable") |
| + |
| +func RetireRecord(ctx context.Context, s release.Store, ref RetireRef, opt RetireOptions) error { |
| + if !opt.HostClaim { |
| + switch ref.Family { |
| + case "instance": |
| + return DeleteRecord(ctx, s, ref.Account, ref.App, ref.Host) |
| + case "sidecar": |
| + return DeleteSidecarRecord(ctx, s, ref.Account, ref.App, ref.Name) |
| + default: |
| + return ErrClaimRetirer |
| + } |
| + } |
| + if opt.Claims == nil || !controlname.ValidAccount(ref.Account) || !controlname.ValidApp(ref.App) || !controlname.ValidHostShape(ref.Host) { |
| + return ErrClaimRetirer |
| + } |
| + switch ref.Family { |
| + case "instance": |
| + if ref.Name != "" { |
| + return ErrClaimRetirer |
| + } |
| + case "sidecar": |
| + if !controlname.ValidLabel(ref.Name) || ref.Host != SidecarHost(ref.Name, ref.App, ref.Account) { |
| + return ErrClaimRetirer |
| + } |
| + default: |
| + return ErrClaimRetirer |
| + } |
| + return opt.Claims.RetireRecord(ctx, ref) |
| +} |
| +``` |
| + |
| +- [ ] **Step 2: Implement its claim-backed provider.** A returning delete resumes the current deleting operation. A parking state means record withdrawal has finished, not that the host is reusable. A matching serving holder with no record requires operator recovery under §10. A purge remains under its own journal; this method cannot restart it as a delete. Rollbacks supply `RetireRef.Incarnation`; ordinary authorized delete resolves and binds the current one before its start CAS. |
| + |
| +**`internal/hostops/retirement.go`** |
| + |
| +```go |
| +package hostops |
| + |
| +import ( |
| + "context" |
| + "errors" |
| + "os" |
| + |
| + "github.com/carlosframework/platform/internal/hostclaim" |
| + "github.com/carlosframework/platform/internal/provision" |
| +) |
| + |
| +func (c Controller) RetireRecord(ctx context.Context, ref provision.RetireRef) error { |
| + if err := c.ready(); err != nil { |
| + return err |
| + } |
| + key := provision.RecordKey(ref.Account, ref.App, ref.Host) |
| + if ref.Family == "sidecar" { |
| + key = provision.SidecarRecordKey(ref.Account, ref.App, ref.Name) |
| + } |
| + rec, rerr := ReadRecord(ctx, c.Store, key) |
| + if rerr != nil && !errors.Is(rerr, os.ErrNotExist) { |
| + return rerr |
| + } |
| + obs, err := hostclaim.Read(ctx, c.Store, ref.Host) |
| + if err != nil { |
| + return err |
| + } |
| + if obs.Absent { |
| + if errors.Is(rerr, os.ErrNotExist) { |
| + return nil |
| + } |
| + obs, err = c.Adopt(ctx, rec) |
| + if err != nil { |
| + return err |
| + } |
| + } |
| + h := obs.Claim.Holder |
| + if h == nil { |
| + if rerr == nil { |
| + _, err = RemoveOrphan(ctx, c.Store, rec) |
| + return err |
| + } |
| + if obs.Claim.Free() { |
| + return nil |
| + } |
| + return ErrPending |
| + } |
| + if h.Key != key || h.Family != ref.Family || h.Account != ref.Account || h.App != ref.App || (ref.Incarnation != "" && ref.Incarnation != h.Incarnation) { |
| + return ErrIdentity |
| + } |
| + if rerr == nil && !matches(rec, h) { |
| + return ErrIdentity |
| + } |
| + switch obs.Claim.Op.Kind { |
| + case hostclaim.KindServing: |
| + if rerr != nil { |
| + return ErrPending |
| + } // Old-console deletion needs operator recovery. |
| + obs, err = c.BeginRetire(ctx, key, ref.By, h.Incarnation) |
| + if err != nil { |
| + return err |
| + } |
| + return c.Retire(ctx, ref.Host, obs.Claim.Op.ID) |
| + case hostclaim.KindDeleting: |
| + return c.Retire(ctx, ref.Host, obs.Claim.Op.ID) |
| + case hostclaim.KindParking: |
| + // Record withdrawal completed. Chores still block reuse in the claim. |
| + return nil |
| + default: |
| + return ErrPending // A purge is driven by its journal, never this path. |
| + } |
| +} |
| +``` |
| + |
| +### Task 4: Add inert console adapters and pass construction fields |
| + |
| +- [ ] **Step 1: Add the following two fields to `console.Config`, next to the existing Plan 7 `HostClaim bool`.** Add the `internal/hostops` import to `server.go`. Do not add another `HostClaim` field or environment read. The command below applies the exact additive source edit and refuses duplicate application: |
| + |
| +```bash |
| +python3 - <<'PYCODE' |
| +from pathlib import Path |
| +p = Path('internal/console/server.go') |
| +s = p.read_text() |
| +assert 'HostClaim bool' in s or 'HostClaim bool' in s |
| +assert 'HostClaimOps' not in s |
| +needle = '"github.com/carlosframework/platform/internal/' |
| +pos = s.index(needle) |
| +s = s[:pos] + '"github.com/carlosframework/platform/internal/hostops"\n\t' + s[pos:] |
| +needle = 'type Config struct {' |
| +assert s.count(needle) == 1 |
| +s = s.replace(needle, needle + '\n HostClaimOps *hostops.Controller\n HostClaimRecovery *hostops.Recovery\n', 1) |
| +p.write_text(s) |
| +PYCODE |
| +``` |
| + |
| +- [ ] **Step 2: Add the complete server helper file.** `NewServer` does not start the loop. Nil production providers remain a visible refusal until Tasks 5–9 are complete; do not wire an enabled deployment at this stage. |
| + |
| +**`internal/console/hostclaimops.go`** |
| + |
| +```go |
| +package console |
| + |
| +import ( |
| + "context" |
| + "log" |
| + "time" |
| + |
| + "github.com/carlosframework/platform/internal/hostclaim" |
| + "github.com/carlosframework/platform/internal/hostops" |
| + "github.com/carlosframework/platform/internal/provision" |
| +) |
| + |
| +func (s *Server) retireHostRecord(ctx context.Context, ref provision.RetireRef) error { |
| + var claims provision.ClaimRetirer |
| + if s.cfg.HostClaimOps != nil { |
| + claims = s.cfg.HostClaimOps |
| + } |
| + return provision.RetireRecord(ctx, s.cfg.Releases, ref, provision.RetireOptions{HostClaim: s.cfg.HostClaim, Claims: claims}) |
| +} |
| +func (s *Server) adoptHostRecord(ctx context.Context, key string) (hostclaim.Observed, error) { |
| + if !s.cfg.HostClaim || s.cfg.HostClaimOps == nil { |
| + return hostclaim.Observed{}, hostops.ErrContract |
| + } |
| + rec, err := hostops.ReadRecord(ctx, s.cfg.Releases, key) |
| + if err != nil { |
| + return hostclaim.Observed{}, err |
| + } |
| + return s.cfg.HostClaimOps.Adopt(ctx, rec) |
| +} |
| + |
| +const claimRecoveryInterval = 2 * time.Minute |
| + |
| +// Called by serve, never NewServer. Tests can invoke the one-pass method. |
| +func (s *Server) StartClaimRecoveryPass(ctx context.Context) { |
| + if !s.cfg.HostClaim { |
| + return |
| + } |
| + if s.cfg.HostClaimRecovery == nil { |
| + log.Printf("console: claim recovery: %v", hostops.ErrContract) |
| + return |
| + } |
| + t := time.NewTicker(claimRecoveryInterval) |
| + defer t.Stop() |
| + for { |
| + if err := s.claimRecoveryPass(ctx); err != nil { |
| + log.Printf("console: claim recovery: %v", err) |
| + } |
| + select { |
| + case <-ctx.Done(): |
| + return |
| + case <-t.C: |
| + } |
| + } |
| +} |
| +func (s *Server) claimRecoveryPass(ctx context.Context) error { |
| + if !s.cfg.HostClaim { |
| + return nil |
| + } |
| + if s.cfg.HostClaimRecovery == nil { |
| + return hostops.ErrContract |
| + } |
| + return s.cfg.HostClaimRecovery.Pass(ctx) |
| +} |
| +``` |
| + |
| +- [ ] **Step 3: Format and run all supplied tests.** |
| + |
| +```bash |
| +gofmt -w internal/hostops internal/provision/retirement.go internal/provision/retirement_test.go internal/console/hostclaimops.go internal/console/hostclaimops_test.go internal/console/server.go |
| +go test ./internal/hostops -race -count=1 |
| +go test ./internal/provision -run '^TestRetireRecordFlagOffAndRequiredProvider$' -count=1 |
| +go test ./internal/console -run '^TestClaimRecoveryAndRetirementAreInert$' -count=1 |
| +go build ./... |
| +GOOS=darwin GOARCH=amd64 go build ./... |
| +GOOS=windows GOARCH=amd64 go build ./... |
| +``` |
| + |
| +Expected: the new tests pass and all builds exit 0. This verifies real Go interfaces and the specified primitives. It does not certify D1–D6 providers or the later handler integrations, which are not yet implemented. |
| + |
| +### Task 5: Complete app admission and create integration after D1/D2/D6 |
| + |
| +**Blocked code boundary:** settle D1, D2 and D6 and add their complete coordinator/test files to this plan before implementing the production path. The primitives above do not implement a registration lease. Do not wire `Authority.Create` to an always-success callback, or call `Attempt` before a real registration exists. |
| + |
| +- [ ] **Step 1: Implement the approved admission record and lease tests first.** The exact key is `apps/<acct>/<app>/admission.json`. The approved document must represent `open`/`closed` and registrations containing the create ID, host and expiry. First creation is `Store.Put(..., "")`; every subsequent change uses the exact read ETag. A CAS failure re-reads and re-applies only the intended registration change; it must preserve closure and every other registration. A read error does not create an open document. Do not delete/recreate admission to resolve contention. |
| + |
| +Tests must pause two consoles at every boundary: saved host observation, registration CAS, renewal, final registration read, claim CAS, record PUT and serving CAS. Close admission concurrently. Verify no registration is accepted after closed, failed renewals stop new effects, ambiguous renewals are not accepted as ownership, and completion removes only the matching create registration. Expired-but-claim-named entries cannot drain. A serving/cancelled create whose console died before removal must be reaped by `Recovery.SweepAdmissions`; recheck both admission ETag and fresh claim. |
| + |
| +- [ ] **Step 2: Place the existing create checks around the transaction.** All three console create doors retain their existing authentication, policy, manifest, class/role, fleet/pool, host/alias ownership and duplicate checks. Preserve `hostHolder`/`hostClaimHolder`, including neutral cross-account errors. Under the flag, read the global lifecycle claim before registering. If it is absent, prove the host has no existing record through the approved discovery routine; adopt a legacy record rather than overwrite it. For no claim and no record, retain `leftoverReplicaRefusal` from `fe2088e3` before the first claim attempt. Its locality limitation is unchanged; it is not a complete deployment census. |
| + |
| +For an already-free claim, do not use that legacy host-prefix LTX scan to block an incarnation-scoped successor forever: old locations are isolated and swept. Honor claim obligations and the region's `quiet_after` at installation/first wake through Plan 6. Do not reset `quiet_after` on create. |
| + |
| +- [ ] **Step 3: Assign the record identity in the console.** Flagged request decoding never accepts `created_at`. Build an unstamped `provision.Record` or `SidecarRecord` from validated request fields. Pass it to `PrepareInstance`/`PrepareSidecar` with this attempt's server UTC time and resolved storage region. The input's `CreatedAt` must be zero; the resulting body and `NewHolder` share exactly the assigned time. `SidecarHost` takes **name, app, sqid**. The legacy `CreateRecord` functions stay byte-compatible for flag-off callers; on the flagged path the coordinator is their replacement and rejects caller-supplied/zero timestamps before writing. |
| + |
| +After a lost claim CAS, do not pass the already-stamped record back into another attempt. Rebuild it with a fresh timestamp and registration, including when a fake clock is held constant. `BeginCreate` derives the incarnation and validates the persisted record body. Its intent keys are exactly `key` and `record`, with `record` containing JSON because `Op.Intent` is `map[string]string`. |
| + |
| +- [ ] **Step 4: Complete the cross-object create coordinator.** `Controller.Attempt` covers the record-only transaction: saved observation → `BeginCreate` → create-only record PUT → `CompleteCreate`. The final `Authority.Create` check must read registration/open state and purge reservation immediately before the exact saved-observation CAS. The registration remains renewed until `serving` or cancellation, including during ambiguous storage outcomes. Do not remove it merely because the HTTP request ended. |
| + |
| +D6 must split/persist any pre-record external effect without violating intent-before-effect. In particular, private storage currently precedes `CreateRecord`; moving only that final call behind `Attempt` would leave provisioning outside the claim. First-route attach's separate hostname claim also needs its defined retry/rollback protocol. The read-only variant of `Authority.Create` must never hide a backend mutation. |
| + |
| +On create-only record conflict, read the existing record, then fresh claim. Remove it only through `RemoveOrphan` and retry only the record PUT under the same successful creating claim. A same-incarnation record permits completion. A changed/different legitimate claim refuses. A transport error leaves the claim for recovery; it is not proof that the record is absent. |
| + |
| +- [ ] **Step 5: Wire recovery and adoption to the same coordinator.** Use `RecoverCreate` only when `StartedAt` is known and older than five minutes. Matching record: `Complete`. Authoritatively missing: `CancelCreate`, adding the retired incarnation and one-hour late-write hint. Any other result waits. `SweepAllRecords` handles orphanhood beyond that hour and records absent from `cancelled_creates`. It reads records before claims; an absent claim invokes the shared adoption path, never deletion. |
| + |
| +`Authority.Adoption` performs the shared complete discovery and returns the resolved live region/box plus every other deployment region and box. `Controller.Adopt` calls the library's one `hostclaim.Adopt`; do not write another ad hoc holder constructor in the door, reconciler or purge. Plan 6's reconcile entry point uses the same discovery/coordinator contract. Set `WriterBox` to the pre-release writer, legacy location to `instances/<host>`, and retain discovery chores until their own op ID is fully acknowledged. If discovery cannot prove a unique record/writer or complete inventory, refuse. |
| + |
| +```bash |
| +go test ./internal/hostops ./internal/console -run 'ClaimAdmission|ClaimCreate|ClaimAdopt|Leftover|Apex|Sidecar' -count=1 |
| +go test ./internal/hostops -race -count=1 |
| +``` |
| + |
| +Expected before implementation: new ordering/lease witnesses fail. Expected afterward: all pass, with the old flag-off tests unchanged. A compile-only provider or missing test name is not a pass; inspect `go test -list` for every named witness. |
| + |
| +### Task 6: Migrate every retirement caller and finish the pull-door boundary |
| + |
| +**Depends on:** Task 5's production provider, D2/D3 inventory and participant evidence, D6 rollback/pending outcomes. The shared `RetireRecord` dispatcher is already complete; these caller changes remain gated on those contracts. |
| + |
| +- [ ] **Step 1: Add every table-row regression before changing its caller.** Run the existing API/UI fixtures through their actual routes, with a `DirStore` that records writes and can stop before/after `Put`/`DeleteIfMatch`. Assert the sequence, response, operation ID, exact ETag, preserved domain claim, and post-response claim/record state—not merely that a mock retirement method was invoked. For the enabled variants seed serving claims or exercise approved legacy adoption. The six direct callers plus two purge callers must each have a named test from the table. |
| + |
| +- [ ] **Step 2: Route ordinary deletes through `retireHostRecord`.** Preserve surrounding response, audit and serial behavior once the requested record withdrawal has actually completed. For an instance use this expression in place of `provision.DeleteRecord`: |
| + |
| +```go |
| +s.retireHostRecord(ctx, provision.RetireRef{ |
| + Family: "instance", Account: acct.Sqid, App: app, |
| + Host: host, By: principal, |
| +}) |
| +``` |
| + |
| +For the sidecar API use: |
| + |
| +```go |
| +s.retireHostRecord(ctx, provision.RetireRef{ |
| + Family: "sidecar", Account: acct.Sqid, App: app, |
| + Name: name, Host: provision.SidecarHost(name, app, acct.Sqid), |
| + By: principal, |
| +}) |
| +``` |
| + |
| +These are complete replacement call expressions within their existing `if err := ...; err != nil` blocks. The UI detach's existing actor variable supplies `By`. Audit actor spelling does not grant authority. Keep alias-only detach outside lifecycle retirement: it holds no provisioning record. A pending/error retirement must prevent domain-claim release and downstream success/audit that asserts removal. |
| + |
| +- [ ] **Step 3: Bind both rollback paths to their own incarnation.** The create coordinator returns the assigned incarnation even for an ambiguous record PUT. Pass it as `RetireRef.Incarnation` in the sibling-apex rollback. Extend `rollbackAttach` and both its callers (`domainattach.go:226,236`) to carry that captured identity. A duplicate known to belong to another attempt keeps `dropRecord=false`; an ambiguous own write must never rediscover “whatever is there now” and retire it. D6 must supply the creating-state cancel/retire recovery intent; `Controller.RetireRecord` intentionally returns pending for `creating`, rather than inventing a serving transition as a shortcut. Do not erase that outstanding obligation when the handler returns. |
| + |
| +- [ ] **Step 4: Complete delete recovery through the existing transitions.** `BeginRetire` binds the expected incarnation and record ETag, calls `BeginDelete`, and keeps executor/stamp. `Retire` waits for the first-phase barrier, deletes only the intent's ETag and rereads authoritative absence before `ToParking`. A newer same-incarnation record ETag is not silently adopted into the old delete intent. Treat it as pending/operator-visible until the owning protocol defines a safe refresh; never delete the new ETag just because a 412 happened. Existing-claim/missing-holder-record is not automatic legacy adoption: §10 reserves old-console damage for operator resolution. |
| + |
| +The regional/local owners finish chores. The console's pass calls `FreeAfterPark` only after none remain. Emit a serial bump after a new deleting/parking/free transition through the existing `s.bumpSerial` path; it accelerates observation but is not a substitute for the durable claim. Do not create another serial protocol. |
| + |
| +- [ ] **Step 5: Coordinate with Plan 7's `apidbclaim.go`.** Current `dbClaim` already rejects `deleting`, `parking`, `purging` and holderless claims, and the common request gate runs for listings and file/resume requests. Delete step 1 is therefore the durable, all-console door-close check; no separate local-only flag is needed. Extend that switch to refuse `creating` as well: |
| + |
| +```go |
| +case hostclaim.KindCreating, hostclaim.KindDeleting, hostclaim.KindParking, hostclaim.KindPurging: |
| + return obs, errDBClaimClosed |
| +``` |
| + |
| +Once the shared adoption provider is ready, replace the absent-claim return in `dbClaim` with the following block. Keep it after existing authentication and fresh record lookup. It does not replace the holder identity checks that follow it: |
| + |
| +```go |
| +if obs.Absent { |
| + obs, err = s.adoptHostRecord(ctx, provision.RecordKey(g.acct.Sqid, g.app, g.host)) |
| + if err != nil { |
| + return hostclaim.Observed{}, err |
| + } |
| +} |
| +``` |
| + |
| +Do not add an app-reservation read as a guessed second pull-door rule: §5 closes create admission at reservation and host door admission at its operation CAS. Purge host claims use the same `purging` refusal. Preserve Plan 7's O1/O2 comparison of complete holder/op and signed exact-version handle checks; preserve Plan 7b generation selection when integrated. A stream admitted before delete may finish using its pinned version. A **new** GET/resume with an old handle after delete starts must hit fresh `dbClaim` and refuse before any version open. |
| + |
| +Tests: pause O1 then begin delete before O2; pause file authorization then delete/recreate and verify it only opens the authorized old version; issue new listing, file and Range resume after begin-delete and count zero new opens. Repeat through a second server sharing the store. Test create-state refusal, adoption discovery failure, free fences, and the flag-off legacy protocol. Do not weaken generation-aware refusal to make adoption succeed. |
| + |
| +```bash |
| +go test ./internal/console -run 'ClaimRetire|ClaimDoor|DB|DomainDetach|Attach|Sidecar' -count=1 |
| +go test ./internal/provision ./internal/hostops -count=1 |
| +git grep -n 'provision.DeleteRecord\|provision.DeleteSidecarRecord' -- internal/console |
| +git grep -n 'RetireRecord' -- '*.go' |
| +``` |
| + |
| +Expected: no remaining raw provisioning-delete call in console production code; remaining legacy calls are in the explicit flag-off dispatcher/tests. Every matrix test passes. Purge sweeps are checked separately below; the grep alone does not prove all deletion paths are covered. |
| + |
| +### Task 7: Implement the purge coordinator after D1/D4/D6 |
| + |
| +**Blocked code boundary:** the specified journal envelope and pure fence/host transitions above compile. Add the approved durable host-set schema, coordinator and full tests once D4 is settled; the following is its required phase algorithm, not a claimed complete implementation. |
| + |
| +- [ ] **Step 1: Write restart witnesses for every phase and cross-object boundary.** Use two hosts, at least one sidecar, two console instances sharing the store, a recording store/email backend, and a fake crash after each durable effect. Restart with only the persisted objects. No process-local host list, remembered ETag or in-memory “already tore down” flag may be required. Include an app with no host claim at all to prove reservation discovery does not depend on `control/hosts/`. |
| + |
| +- [ ] **Step 2: Reserve, close and drain.** Run existing owner/control-plane/trash/production/backend-readiness checks first. Reserve `apps/<acct>/<app>/purge.json` create-only as exactly the specified `{id, phase: "reserved"}` envelope plus the approved D4 fields. On conflict, read the existing reservation: retry its existing ID; never overwrite it with a new purge. Close admission by CAS. List all instance and sidecar records, plus all registered create hosts, without a count cap or malformed-record skipping. |
| + |
| +For each registration, fresh claim decides: |
| + |
| +| Registration/claim | Required action | |
| +|---|---| |
| +| Not expired; claim does not name it | Wait; do not remove the registration. | |
| +| Claim names its creating ID/incarnation | Keep it, even expired; create recovery completes/cancels it. Never treat expiry as cancellation. | |
| +| It reached serving | Include that holder and wait for registration completion/removal under D1. | |
| +| Expired and claim does not name it; host is free/absent | Persist the host in the D4 journal; CAS `FencePurge(purgeID)` using that fresh observation; only then remove the registration with the admission ETag. | |
| +| Read/parse failure or another holder/operation prevents the fence | Wait/refuse with the reservation intact; use D4's resolution, never overwrite another holder. | |
| + |
| +A lost fence CAS requires fresh observation and reevaluation; it does not license unconditional registration removal. Test a creator paused after its final registration read while purge fences **two** different hosts and finishes. Both old create CAS attempts still fail after the fences become written free claims. Test both original absence and original free-ETag observations. |
| + |
| +- [ ] **Step 3: Adopt and claim the fixed host set.** Invoke the shared `Controller.Adopt` for every legacy record before deciding which hosts belong to the app. Finish drain and persist the fixed host set in the journal. Advance `reserved → adopted` only after the approved census is durable. For every serving holder, CAS `BeginPurge` with the **same purge ID**, preserving executor/stamp and using `StartGuard` against the writes this purge will make. A host under another operation refuses; previously acquired claims remain reserved for retry. Advance to `claimed` only after every host is secured. Holderless fences are retained separately; never call `ToParking` on them. |
| + |
| +- [ ] **Step 4: Tear down only after all claims.** Reuse `purgeAppStores`, all environment-store handling, mail backend-readiness checks and `mail.Teardown` in their current safe order. Retain `release.PurgeReady`'s protection rather than bypass it when recovery runs. Under D6, replay uses durable app identity and backend idempotency; a failed store teardown never advances the phase or hides remaining mail estate. Advance `claimed → torn-down` only after every required teardown is durably complete. Auto-purge boxes delegate to this coordinator without receiving SES/object-store backend authority. |
| + |
| +- [ ] **Step 5: Retire records under purging, not through deleting.** For each host, read its record with ETag; verify holder identity; persist `PurgeRecordIntent` into the host's **existing** purging op. `Authority.PurgeRetirement` freshly checks the matching app reservation ID and phase `torn-down` or later. `Controller.Retire` replays exact deletion and moves `purging → parking`. It never calls `BeginDelete` on a purging claim and never makes the host serving. If the intent already exists and the record is absent, replay from the intent. An absent record without recorded intent needs the approved D4 recovery rule, not a fabricated ETag. |
| + |
| +The old `release.PurgeApp` loop cannot remove records or coordinator objects in parallel. Extract a claim-aware final sweep entry point or supply an explicit retirement strategy; avoid a `release → provision → release` import cycle. Whichever interface D4 approves must refuse raw deletion of instance/sidecar record keys and skip `admission.json` and `purge.json` until their designated completion steps. Keep other `PurgeApp` callers legacy only when their startup flag is false. |
| + |
| +- [ ] **Step 6: Finish hosts, prefix cleanup and fences durably.** Wait for every host's obligations and ordinary free transition, or the D4-approved durable proof that this purge's old holder is retired. Preserve enough journal identity to avoid touching a host acquired later by another app. Advance through `retired → done` only at the approved checkpoints. Retain final sweep retryability if the app tombstone disappeared. Release each holderless fence with `ReleasePurgeFence` under the matching purge ID and current ETag, never `Delete`. Remove the journal only through its own `done` completion, conditionally; a claim becoming free cannot do that. Finish console row/door-key/relay bookkeeping under D6. Leave the admission object closed until the defined final cleanup/recreation boundary. |
| + |
| +Required witnesses beyond primitive tests: teardown blocked on a single active move; bad record in page two blocks teardown; a disappearing registration remains recoverable from the journal; one host frees while another parks; crash after prefix cleanup before fence release; crash after one fence release; a recreated app races a stale purge worker; failed store/email teardown and absent backend; stale journal ETag; another app's holder prevents fencing; no backend calls at all before all claims. Repeat auto-purge and owner purge with the flag unset to preserve today's behavior. |
| + |
| +```bash |
| +go test ./internal/hostops ./internal/console ./internal/domains ./internal/release -run 'ClaimPurge|ClaimRetirePurge|ClaimRetireAutoPurge|Purge' -count=1 |
| +go test ./internal/hostops -run 'Purge|AttemptUsesPreRegistration' -race -count=1 |
| +``` |
| + |
| +Expected: every phase resumes from persisted state; injected read failures make no dependent effects; no host claim is deleted; flag-off purge tests remain unchanged. Before the coordinator exists, these integration tests must fail rather than be skipped. |
| + |
| +### Task 8: Complete placement and the authority providers after D2/D3/D5 |
| + |
| +- [ ] **Step 1: Write the placement ordering tests before touching `placeAppTarget`.** Cover `handleAPIPlace`, `handleAppPlace`, `handleFleetPlacePage`, a multi-host app with one active move, an app with no records, a route-only app, and a concurrent create inheriting placement. Spy on `SetAppPlacement`, `reg.SetPlacement` and record PUTs: all are zero until the approved app/host claim acquisition barrier is complete. An incomplete inventory is not a smaller request. |
| + |
| +- [ ] **Step 2: Start each placement with the supplied `BeginPlacement`.** Validate the target's fleet ownership using existing neutral 404 behavior, resolve its intended write set with D2, and persist target and D5's reconstruction inputs before app, route or record writes. Keep routes-before-records ordering inside the claimed operation where D5 retains it. The existing `SetRecordRegion` rereads/retries without an operation check; do not call it unguarded after one earlier claim read. Bind record writes to the operation/phase/incarnation and exact record ETag, and let a stale worker refuse after the claim changes. |
| + |
| +Do not equate a `Record.Replicas` entry to execution ownership. Replica records retain their own host/region/role; their data is not covered by retiring only the home instance. `edge-replica-placements` narrows replica reconciliation to the replica's region; preserve that behavior, and preserve sidecar kind separation. Re-run the consumer search for replica materializers when implementing: `CreateRecordReplica` currently delegates to `CreateRecord`, and no named `PlaceInstance` API can be patched instead of these actual paths. |
| + |
| +- [ ] **Step 3: Implement the approved replay/completion protocol.** Recovery's `Placement` callback must reread claim and operation, reconstruct solely from intent/journal, and finish D5's stop, install, executor/stamp and release rules. Any change to `op.exec` obeys R6-4 atomically. Never clear unrelated obligations on release. Pending remote/local work and conflicting moves keep placement pending. There is intentionally no guessed `CompletePlacement` in this plan revision. |
| + |
| +- [ ] **Step 4: Supply every `Authority` method with its actual source.** `Create` reads admission/reservation; `Executor` proves the exact executor for the record; `StartDelete` invokes the shared Plan 8 floor/start check; `FirstPhase` verifies the participant barrier; `Inventory` returns the complete required deployment scope; `Adoption` performs shared discovery; `PurgeRetirement` verifies the current app journal. A failed/missing provider returns a refusal. Finish application authorization before any of these helpers are exposed through a door. |
| + |
| +- [ ] **Step 5: Supply Plan 4 adapters without cross-region authority.** Construct each `park.Engine` for exactly its nominated region/data bucket. Provide moves/Plan 7b's authoritative regional `View`, durable stop/fence checks in `Authorize`, and aggregate renewable leases for every instance/moves/staging location its scope uses. `ParkScopeProvider` returns the original retirement location/incarnation/evidence time and the approved enforced request bound. Existing scoped chores keep their scope on retries. `Engine.Bind` persists previously unscoped chores before work; `Run` does manifests, version deletes, the horizon and `quiet_after`/completion CAS. Do not implement those effects again in console code. |
| + |
| +`park.Work` is `{Host, ID, Op}`. `Engine.Bind(ctx, work, hostclaim.ParkScope)` and `Engine.Run(ctx, work)` are the exact Plan 4 APIs. Local owners use `park.Local.RetireInstallation`, `RetireArtifacts` or `AcknowledgeEmpty` through Plan 6; a console cannot call `CompleteObligation` on their behalf. The console recovery switch intentionally does nothing for parking with open chores. It only frees ordinary parking when all chores are gone. Plan 4 completes a holderless sweep's last chore and frees it in one CAS; the console does not fabricate a holder or start a duplicate sweep. |
| + |
| +Tests must prove the adapter rejects another region, local chore, stale op/obligation, missing provider and wrong evidence; retries preserve serialized scope; a horizon equality or residue keeps the chore open; no invocation crosses a sibling data bucket; discovery completes only after both regional and local acknowledgements. Keep Plan 4's existing full storage fault tests as the mechanism tests. |
| + |
| +```bash |
| +go test ./internal/hostops ./internal/console -run 'ClaimPlacement|ClaimPark|ClaimAuthority' -count=1 |
| +go test ./internal/park ./internal/hostclaim -count=1 |
| +``` |
| + |
| +Expected: rejected placement causes zero app/registry/record effects; all replay tests pass; no new IAM grants are needed for a console to drive the correct regional owner. |
| + |
| +### Task 9: Finish recovery construction, integration review and gates |
| + |
| +- [ ] **Step 1: Build the real `hostops.Recovery`.** Set `Controller` to the same controller the doors use. `SweepRecords` calls `Controller.SweepAllRecords`; `SweepPurges` calls `SweepAllPurges` with the D4 coordinator; `SweepAdmissions` implements D1 cleanup. `Purge` and `Placement` reread and resume their approved durable coordinators. Never use successful empty callbacks in production. `SweepAllPurges` is required because a reservation may be the only object left after a crash. `ListClaims` filters direct host JSON keys; nested `exec/<box>.json` objects must not be parsed as claims. Its snapshots are discovery only; all acting reads are fresh. |
| + |
| +With the flag enabled, start the pass from `serve` alongside `StartEconomicsPass` after constructing its real dependencies: |
| + |
| +```go |
| +go srv.StartClaimRecoveryPass(context.Background()) |
| +``` |
| + |
| +The method itself makes the flag-off call a no-op, consistent with existing pass startup. Prefer the existing process-lifetime cancellation context if `serve` has gained one by implementation time. Do not start goroutines from `NewServer`. Unknown move/push kinds are left to their owners, never “recovered” as delete. Report a per-host failure and continue unrelated hosts; a failed authoritative listing never yields partial retirement authority. |
| + |
| +- [ ] **Step 2: Exercise recovery with actual phase fixtures.** A three-minute create stays creating; exactly five minutes stays creating; older than five completes or cancels according to a fresh record read. Deleting resumes conditional deletion; parking with work waits; parking empty frees; holderless fence delegates to purge; move/landing/reverting/restoring/pushing remain untouched. Claims with unknown age wait. Scan a malformed record, failed second list page, malformed claim, expired cancellation hint with a late orphan, a legacy sidecar and a reservation with no claims. Assert no action based on the failed read. A ticker test cancels cleanly and `NewServer` never launches one. |
| + |
| +- [ ] **Step 3: Check the delivery matrix before calling Plan 5 complete.** |
| + |
| +| Required delivery | Evidence | |
| +|---|---| |
| +| D1–D6 | Decisions recorded; complete reviewed coordinator/provider code; no production placeholder callbacks | |
| +| Flag-off | Existing console/provision/domain/purge tests unchanged and passing; no new claim or admission calls | |
| +| Creates | Instance, sidecar, first-route and replica-producing paths all use the one admission protocol | |
| +| Retirement | Every exact call-site row has its named behavior test, including raw purge sweeps | |
| +| Door | Second-console new list/file/resume refusal after begin-delete; pinned admitted streams retain version safety | |
| +| Purge | Complete census, retained journal, two-host fences and crash replay across teardown/final cleanup | |
| +| Placement | App-wide before-effect barrier and defined executor/stamp completion, including concurrent create | |
| +| Recovery | Constructed production pass with real admission/purge/placement/adoption providers | |
| +| Parking | Nominated regional/local owners, original evidence, complete scope, horizons and persisted acknowledgements | |
| +| Activation | Plan 8 shared floor/participant barriers integrated; flag still unset | |
| + |
| +- [ ] **Step 4: Run the required repository checks.** Use the default shared cache; use only the existing on-disk fallback if the sandbox refuses it. |
| + |
| +```bash |
| +go test ./internal/hostops ./internal/hostclaim ./internal/park -race -count=1 |
| +go test ./internal/console ./internal/provision ./internal/domains ./internal/release -count=1 |
| +go build ./... |
| +go vet ./... |
| +go test ./... -count=1 |
| +gofmt -l . |
| +GOOS=darwin GOARCH=amd64 go build ./... |
| +GOOS=windows GOARCH=amd64 go build ./... |
| +GOOS=linux GOARCH=arm64 go build ./... |
| +``` |
| + |
| +Expected: tests pass, build/vet/cross-build exit 0, `gofmt -l .` prints nothing. Tests that invoke the CLI use `carlosTestEnv` and a scratch `cmd.Dir`; no operator credentials, current project configuration or production traffic. Run the joint model's stated 30k+ seed gate and mutation witnesses for admission fencing, record-before-claim orphan checks, purge teardown ordering and stale placement writes; record the exact model checkout/command when the implementation is integrated. A model pass is additional evidence, not replacement for the real handler/coordinator tests. |
| + |
| +- [ ] **Step 5: Obtain the authority/production-path review required by AGENTS.md.** Use the explicitly named Opus reviewer through the repository's available agent tooling at implementation time; do not label an unavailable review as passed. Review all cross-object races above, D1–D6 resolutions, every old delete door, and the full final branch. This authoring task performs no production action, staging, commit, push or merge. |
| + |
| +- [ ] **Step 6: Land only the completed, reviewed inert implementation through amadan.** Push the branch to the remote that points to amadan, describe its final behavior and evidence, squash to one change commit, wait for green CI on that exact tip, and use `amadan branch merge carlos/platform feat/hostclaim-console -expect <tip> -message "..."`. Check main's CI after landing. Never merge on GitHub or locally to main. If D1–D6 are still open, describe any primitive-only landing as partial and keep the integration tasks open; do not mark Plan 5 complete or enable the flag. |
| + |
| + |
| +## Authoring verification |
| + |
| +Checked on 2026-10-05 in a disposable on-disk source export, outside the planning repository. The source was Plan 7 at `87c8ef92`, including main `04cd2d78` and the claim library. Because Plan 4 was not committed, the export received its rev 2 **exact** `ParkScope`/holderless-sweep edit and its complete `remote.go`/`sweep.go` blocks. It then received the eleven complete Go files above and the additive console Config edit. No repository source file was changed. |
| + |
| +Results: |
| + |
| +- All eleven complete Go blocks exactly matched their compiled export files and were `gofmt` clean. Markdown code fences were balanced. |
| +- All supplied `hostops` tests passed, including a race-detector run. The new provisioning dispatcher and console inertness tests passed with the focused test selection in Task 1. `go vet -buildvcs=false ./internal/hostops` passed. |
| +- Whole-repository `go build -buildvcs=false ./...` passed for Linux/amd64, Darwin/amd64, Windows/amd64 and Linux/arm64 with the final blocks present. These are compile checks; cross-platform test binaries were not executed. |
| +- The default `/home/paulca/.cache/go-build` was attempted and rejected as read-only. Verification used the **existing shared** `/home/paulca/.cache/tmp/gocache`, not a cache under `/tmp` or a new per-task cache. |
| +- During authoring, the door branch advanced to `3d143b23` with ranged version reads and the Plan 7b package merge. The source export's door files were checked against pinned `87c8ef92`; the evidence above is for that explicit baseline. The branch commands intentionally merge the actual committed dependency at implementation time; retain its range-read/generation changes and rerun the gates then. |
| + |
| +The disposable export and authoring scratch files were removed after verification. Only this plan was written in the planning checkout; nothing was staged or committed. There was no live infrastructure test, and no Opus review was claimed. The full handler/coordinator test matrix, joint-model gate and authority review remain implementation work. In particular, successful compilation does **not** complete D1–D6 or make the inert feature ready to enable. |