carlos / platform Public

Neither spec is ready for planning. B rev 3 resolves most of the original round-2 findings, but the combined protocol still permits concurrent writers after declare-lost, cannot safely exclude delayed staging work, and has several crash paths that cannot finish.

I read AGENTS.md, both complete specs, and all five prior findings documents. Code references below refer to 6e4d3a0e unless explicitly qualified. The db-door implementation is absent from this worktree; I checked it at 653ad4b1. This was a static, read-only review: no files changed, tests run, or production actions taken. The requested repository-policy Opus cross-check was unavailable; these are my findings.

1. Moves round-2 dispositions

“Resolved” means the particular design defect is addressed, not that the implementation exists or the composed protocol is approved.

| R2 finding | Verdict | Reason | |---|---|---| | R2-1: surviving flat-exec descendants | Resolved in B | §5.3 requires strict containment, empty process trees and stray-process checks; A must adopt D3. | | R2-2: stop evidence lost with activator memory | Resolved in design | Reconstruction now uses observable state, including never-spawned operations, rather than stopHistory. | | R2-3: dirty source recovery replays installation | Resolved | §7.8 has separate source recovery phases and prohibits reinstalling after probing. Pre-flip recovery has a different gap below. | | R2-4: reconcile retires rollback target | Resolved | Both participants remain held until release and cleanup finish. | | R2-5: rollback leaves target replica live | Resolved as a requirement | Target replica/staging parking is now mandatory before finalization; the park algorithm has new completion failures below. | | R2-6: premature installation markers | Resolved | Bucket park/copy/publication are separated; .files-gen follows durable env, timer and registry installation. | | R2-7: mirror deletes adjacent metadata | Resolved | Mirroring is confined to replica/; archives, timers and operation metadata are separate. | | R2-8: incomplete and unpinned launch environment | Partly | Composition and input versions are covered, but checking at admission/configuration does not exclude delayed config/adoption work or changed inherited environment at the actual spawn. | | R2-9: missing markers and restore verdict | Resolved as inventory | Markers are classified and excluded from content digests; verdict invalidation is specified. | | R2-10: incomplete cut budget | Partly | T11 includes the missing stages, but budgets one probe attempt while §7.7 permits three; no unambiguous end-to-end deadline constrains landing retries. |

The carried-forward timer finding also remains partly resolved: draining delivery and freezing the converger do not cover the runner’s separate pruning path.

2. D1–D8 and I-7

D1 — B is right: admission must identify the box and phase.

A region alone is insufficient. Wake currently refreshes a local row at activator.go:1071; ordinary reconcile mints it ready at reconcile.go:364. Neither expresses the transfer of execution authority.

Exact text:

• A §§4, 8: “For move operations, admission requires the holder incarnation, execution holder, participant identity and phase-specific permission defined by the moves spec. Region equality alone never grants admission. Ordinary wake and recovery probe are distinct permissions.”
• B §19.1/K1: “The admission table governs every spawn, including Seed, probe retries and recovery. The table, current holder incarnation and exec must all authorize the exact box. Check authorization before restore or other local mutation as well as before spawn.”

D2 — A is right; B has correctly withdrawn the disagreement.

Claim revision and installed generation serve different purposes. The existing creation identity is explicitly continuity metadata, not a database generation: types.go:60.

Exact text:

• A §2: “gen is the revision of the whole claim and increases on every successful claim write. It is not an installed-data generation.”
• B §5.1: “Installed generations are identified by operation id and incarnation. Claim gen is used only for claim concurrency and observations; it is never an installed-generation marker.”

D3 — B is right, with a required restart-recovery API distinction.

Best-effort placement can leave descendants outside the host cgroup: cgroup.go:230. Also, PrepareStrict rejects a populated group, so it cannot itself reopen and kill a surviving group after restart: strict_linux.go:206.

Exact text:

• A §6: “All lifecycle-controlled spawns require strict containment. Stop evidence requires a closed admission fence, drained spawn attempts, an empty complete process tree, closed replicators and no remaining database/WAL/SHM holders. An absent host cgroup is insufficient where an earlier spawn could escape containment.”
• B §5.3: “Reconstruction uses a cleanup-only operation that can reopen, identify, kill and drain an existing populated cgroup without granting spawn authority. PrepareStrict remains the empty-group spawn API. A pre-move best-effort child is not certified stopped by ordinary parent exit.”

Apply this to delete/park as well as moves.

D4 — B is right about IAM; its proposed exception is unsafe. Neither exclusion rule works as written.

The grants are precise:

| Principal | Own data bucket | Sibling moves/* | Sibling instances/* | |---|---|---|---| | Region host | Get, Put, Delete; List | Put, Delete; prefix-conditioned List | No grant through sibling staging |

See main.tf:166 and main.tf:217. A source cannot read the destination lock. But two workers for the same operation share its prefix, and MirrorPrefix deletes destination-only keys: mirror.go:222.

Exact text:

• A C3: “Cross-region staging uses a shared staging-work lease in the deployment bucket, keyed by destination region and host, accessible to both participants. Every staging writer, seal, park and abort obeys this lease. Actors acquire it before revalidating operation and phase. Per-operation naming alone is not exclusion.”
• B §19.3 D4: “Retain M17’s push-only sibling grant. Replace the lease exception with the shared staging-work lease. Before sealing or parking staging, drain earlier rounds and their outstanding I/O. Landing consumes a sealed manifest of exact object identities, using conditional copies; no earlier round may mutate the sealed input.”

The deployment-bucket grant already permits this coordination. The late-I/O requirement still needs the correction in N1 below.

D5 — B is right about separating move retirement from incarnation deletion, but operation id alone is insufficient.

The target can park an earlier stay during installation and later park this move’s failed landing during rollback. Both currently use superseded/<move id>/<host>/. Those are different datasets within one operation. A unique operation id does not separate them.

The source/staging coordinates are independently derived in movespass.go:364; copying supports a destination key, so the distinction can be preserved: mirror.go:467.

Exact text:

• A §§2, 5: “A park-superseded obligation identifies operation, retirement event, location, retired generation and artifact class. Its immutable manifest belongs to that exact retirement event, not merely to the incarnation or operation.”
• B §§6.4, 7.6, 7.8, 7.9: “Use distinct destinations beneath superseded/<op>/<host>/<retirement-id>/, retaining separate instances/ and moves/ subtrees. Initial target retirement, failed-target retirement, source retirement and staging retirement never share a manifest.”

D6 — B identifies a real omission, but neither spec supplies independent execution fencing.

Changing exec cannot stop a partitioned process. Existing supervision reacts to the regional replica lease, not a globally enforced execution grant: activator.go:2266. Source and target leases cover different buckets.

Exact text:

• A §4: “Declaring a box lost does not itself establish stop evidence. Before transferring execution, resolve must obtain independently verified execution fencing of the exact box incarnation. Without that proof it retains the hold. Lost-box quarantine survives operation release and prevents readmission until its old state has been retired.”
• B §7.10/D6: “A claim CAS is bookkeeping, not independent fencing. The resolve command must establish verified compute fencing, or an independently enforced execution-lease expiry, before reopening the source. Recover validated durable target data where available, not only an already-published return. Retain M16’s precommit restriction, exact-participant check, source refusal and explicit loss confirmation.”

The command must perform this through platform interfaces; it cannot require a follow-up box command.

D7 — B is right.

The existing MoveComplete combines region change and clearing the old move, provision.go:1053; it cannot be reused unchanged for the proposed multi-stage protocol.

Exact text for both operation tables:

> “Legal move-kind transitions are serving → moving; moving → landing or reverting; landing → restoring or, after committed finalization, serving; reverting/restoring → serving only after their recovery and cleanup conditions hold. Kind transitions do not independently transfer execution authority.”

Also reconcile K4’s “by that participant” with §7.5’s source completing the target handoff after a crash. Name one authorized writer for each transition and its recovery.

D8 — Both rules are compatible; retain B’s stronger operation-specific completion condition.

Existing MoveAbortComplete simply clears the record, provision.go:1043, so builders must not mistake that helper for the new completion contract.

Exact text:

• A §2: “All obligations block free. Each operation additionally defines which obligations block its release to serving; release preserves obligations owned by any other durable lifecycle work.”
• B K8: “A move releases to serving only after every obligation it created has a durable completion acknowledgement. Record cleanup and claim release follow the crash-recoverable finalization protocol, including abort.”

I-7 — B’s explicit flat-file inventory is necessary.

Flat env files are siblings of the database, not entries in the flat files directory: paths.go:50. EnsureAdminToken deliberately preserves an existing token: admintoken.go:104.

Exact text:

• A §5.4: “Local retirement covers both layouts. For flat hosts it includes <DataDir>/<host>/, the database and companion artifacts, <host>.env, <host>.config.env, <host>-sidecar.env, generation/lineage markers and host-bound operational state. Nested inventory uses the resolved instance directory. Persist this inventory independently of the replaceable operation journal.”
• B §17 I-7: “Agreed: A’s local retirement inventory covers flat files and sibling env files as well as nested state. A host cannot become free until every recorded location acknowledges completion, including an empty retirement.”

3. New findings

The following are additional failure traces or failures in the proposed remedies, rather than missing implementation being reported as a defect.

N1 — P1: client cancellation does not establish the storage exclusion assumed by the simulation. A §3.1/C3; B §19.4.

A destructive request is issued before SafeUntil; the client times out, the lease expires, and a successor installs new data. The old server-side request can still complete against the reused key. Checking the deadline before submission and cancelling the client does not supply a server-side ownership condition.

The current delete has no condition at all: mirror.go:493. The lease deadline contract itself warns that notification is not proof: lease.go:182.

Fix: specify conditional/version-bound destructive operations or non-reused physical object identities that make late completion harmless. Keep clock/pause assumptions separate from server-side completion. Model “request submitted, client cancelled, server completes later”; an atomic bucket-call model cannot catch this.

N2 — P1: the D4 exception permits a delayed round to corrupt sealed staging. B §§7.3–7.6, 19 K5/D4; A C3.

Worker W reads the permitted phase, then pauses. Another worker finishes the final round and flips. W resumes its same-operation mirror and overwrites or deletes keys while the target copies them. Operation id and incarnation still match.

The existing pass mutex is process-local, movespass.go:238; destination mutation occurs later in mirror.go:198 and :228.

Fix: adopt D4’s shared staging lease, validation under that lease, explicit round draining and sealed object identities. Exercise two workers for the same operation, not just competing move ids.

N3 — P1: declare-lost can start a second writer immediately. A §4; B §7.10, K6/K11.

The target probe remains alive while partitioned from the control bucket. Resolve changes exec to none, then the source recovers. The target has not read the new claim and continues background writes. A future kill “when it returns” does not repair the interval with two writers.

Regional replication supervision at activator.go:2266 does not fence a process merely because another bucket’s claim changed.

Fix: D6’s independent fence before source admission. Test a live target partitioned only from control-plane access, including one still able to reach its regional bucket and external systems.

N4 — P1: the claim and record have unjournaled transaction gaps. A C2/§5; B §§7.1, 7.5, 7.8–7.11.

Concrete crash points:

• Create claims serving, then crashes before CreateRecord.
• Move claims moving, then crashes before publishing record.move.
• Delete removes the record, then crashes before creating park obligations.
• Abort clears move, then crashes before releasing the claim.

The current worker discovers work through records and skips Move == nil: movespass.go:242. Neither spec defines a complete recovery owner and replay decision for all these states. “Console: stateless” is false for a claim-only accepted request without durable reconstruction inputs.

Fix: persist operation intent and reconstruction inputs before the first cross-object side effect; define discovery and reconciliation of incomplete claims without relying on an existing record. Specify every ambiguous-write recovery and terminal ordering. A stale finalizer must never clear a newer operation.

N5 — P1: lazy legacy adoption does not establish a complete touched inventory. A §§7, 9; B §§0.4, 7.12.

Before rollout, H moved from UK to US. The old move left the UK replica; current code purges local artifacts and flips the record, movespass.go:492, without retiring that source replica.

A door adopts H with only US in touched. Delete parks US and frees the claim. Recreate H in UK; the old replica remains eligible for host-keyed restore, backend_exec.go:300. It has no move-generation marker, so origin-marker absence does not distinguish it from legitimate origin data.

Fix: complete legacy location discovery and retirement acknowledgement before allowing reuse. Unknown historical coverage must block reuse. A first door cannot certify global cleanliness from the current record region or the console’s own bucket.

N6 — P1: parking an active coordination lock makes ordinary retries conflict permanently. A §5.4.3; B §17 I-8.

The park acquires a lease and copies every key, including lock.json, into its immutable manifest. Renewal changes the source lock ETag. A retry acquires another lock body. The manifest comparison now fails against the worker’s own coordination state, leaving the obligation outstanding.

Renewal updates the lease identity at health.go:129. Existing mirroring explicitly excludes lock.json for this reason: mirror.go:174.

Fix: exclude live coordination locks from copied data and manifest completeness checks. Release only the current lease handle, conditionally, after durable completion. A historical lock observation may be recorded as evidence, but cannot be a replayable data entry.

N7 — P1: one move can reuse its own immutable park manifest for different data. B §§6.4, 7.6, 7.8, 7.9; A’s borrowed park helper.

A returning host already has residue on the target. Landing parks that residue into superseded/<move>/<host>/ and installs new data. Its probe writes, then fails. Rollback cleaning parks the new target replica into the same destination.

With A’s immutable-manifest rule, cleaning encounters unlisted or changed keys and cannot finish. Overwriting the manifest instead destroys the earlier retirement record. Replica and staging retirement also need distinct mappings.

Fix: use D5’s retirement-event ids and separate artifact subtrees. Test a move back to a previously used region followed by dirty rollback. Unique move ids alone do not cover this case.

N8 — P1: status filters do not drain config, adoption or placement work already in flight. A C3/§8; B §§7.3–7.6, 9.1–9.2, K9.

A config pass selects A’s ready row and renders its secrets. It stalls. A is deleted and parked; B is created at the same host. The old pass resumes and writes A’s <host>.config.env. B can then launch with A’s credentials.

The write uses the captured route at syncexec.go:1091, culminating in the rename at :1180. Release adoption similarly changes a host-keyed live symlink at adopt.go:1045. Its earlier ready filter does not protect that later mutation.

There is also a separate placement writer: fleets.go:616 changes registry placement before provision.go:853 changes the record.

Fix: make these actors participate in lifecycle exclusion, including draining already-admitted work. Bind host-specific writes to incarnation and operation under the same local mutation guard. Guard placement before any app, registry or record mutation. At actual probe spawn, revalidate the complete composed environment and pinned executable, including inherited agent variables after restart.

N9 — P1: app purge can destroy the move’s store before encountering the host claim. A §5; B §7.12/K7.

The proposed refusal is inside PurgeApp/record retirement. The console destroys app stores before calling that function: ownership.go:404, followed by PurgeApp at :442.

An app with a moving instance can therefore lose its store and credentials before purge reports that the move prevents deletion. Phase 1 permits shared app stores; deferring store migration does not make their destruction safe.

Fix: acquire a durable app-purge reservation and the necessary host lifecycle exclusions before store/email teardown or prefix deletion. Concurrent creation must also be excluded. A read-only “no move currently exists” preflight is insufficient.

N10 — P1: waiting out the db-door cache does not drain authorized reads. A §§5.1, 8; B §19.1.

Console A authorizes an old-incarnation GET, then stalls before opening the object. Console B begins deletion and waits one cache lifetime. It parks, frees and recreates the host. A resumes and opens the reused LTX coordinate.

At 653ad4b1:internal/console/apidb.go:287, authorization precedes :312, the actual OpenLTXFile. Cache expiry does not revoke that admitted request or prove that it finished.

Fix: bind reads to immutable object versions/identities, or use distributed read reservations whose completion/expiry is part of retirement. Stop accepting reads and drain them on every console before namespace reuse. Model a stalled handler between authorization and object open, not only an already-streaming response.

N11 — P1: pre-flip recovery can reject its own newly acknowledged writes after a crash. B §7.4 reverting; contrast §7.8’s corrected recovery journal.

A final staging/archive step fails. The source verifies its stopped digests and starts the recovery probe. That probe changes files and acknowledges background work. The agent crashes before opening and recording aborting.

Recovery re-enters §7.4 and compares the changed files against the original stopped digest. It now declares a mismatch and remains stuck. Reinstalling the baseline would lose the probe’s writes.

The probe runs the real writable application, through backend_exec.go:912.

Fix: give pre-flip recovery durable verified → probing → recovered → opened phases too. After entering probing, never reapply the old baseline check as an admission requirement or restore over recovery writes.

N12 — P1: the version floor omits an automatic path back to an old executable. A §9; B §§11, 14.3, 15.

B names system apply and explicit rollback. Boot recovery independently restores carlos.prev: systemapply.go:901, calling the shared restore path that copies the old binary at systemapply.go:817.

A leftover apply journal can therefore restore a pre-floor executable while a host is landing or restoring unless the floor is enforced in this path too. Old reconcile then knows only the ordinary ready mint.

Additionally, B §15 puts flagship rehearsals before floor publication, while §11 requires publication before the first production move.

Fix: enforce the floor in every executable restoration path, including automatic recovery and boot. Validate .prev and outstanding update journals before activation. Publish the floor before protected production rehearsals. Define authoritative enrollment of all relevant boxes/consoles and exclusion of old writers; a version report cannot constrain an old executable that ignores the claim.

N13 — P2: abort assigns remote parking to an actor without the necessary permission. B §7.11; A C3.

The source is told to park target staging into superseded/. Under M17 it cannot read target staging and cannot write target superseded/; it has only sibling moves/* Put/Delete/List. The existing abort only deletes remote staging, movespass.go:396, which explains why its existing authority was sufficient.

Fix: make target staging retirement a durable obligation executed by the target-region actor. The source parks its own snapshots. Abort retains its claim until both acknowledge completion. Do not broaden M17 to cross-region data reads.

N14 — P2: a second timer-pruning actor bypasses the freeze. B §9.4/K9.

The runner performs its own daily prune at runner.go:260. PruneTimers removes aged completed once-timers even for an alive host: timers.go:188.

A held source can therefore change its timer snapshot after stopped, causing clean recovery’s digest comparison to fail despite no application write. Freezing only declaration convergence and delivery misses this actor.

Fix: make both prune callers respect the operation’s timer freeze, or define a canonical snapshot that explicitly excludes disposable history. Exercise the daily runner sweep during landing and rollback.

N15 — P2: the promised cut window still lacks one enforceable budget. B §§7.3 T11, 7.4, 7.6–7.7.

T11 reserves one 30-second probe plus 20-second settle, while §7.7 allows three attempts and landing has a separate ten-minute deadline. Those rules permit a successful move substantially beyond an admitted 120-second window.

The existing health timeout is a per-start bound, backend_exec.go:36, not an end-to-end move deadline.

Fix: persist an absolute cut deadline, allocate remaining time across installation and probe attempts, and begin recovery when it is exhausted. Keep recovery’s deadline separate. State whether the estimate promises one-attempt success or includes permitted retries.

There are no additional material P3 findings.

4. Verdicts

| Spec | Verdict | Planning blockers | |---|---|---| | A: host lifecycle claim | Not ready | Incomplete legacy location adoption; missing cross-object recovery; lock-bearing park manifests; stale background writers and db reads; insufficient execution/storage fencing. | | B: move-any-app rev 3, Phase 1 | Not ready | Unsafe D4/D6 implementations; colliding park transactions; unjournaled pre-flip recovery; abort IAM mismatch; incomplete integration with purge, background passes and automatic executable recovery. |

A also fails to carry forward the prior R3-12 fresh-incarnation requirement explicitly. Both creation helpers still accept caller-provided timestamps—CreateRecord and CreateSidecarRecord—while A clears the holder on free and retains incarnation-keyed immutable park manifests. Reusing the old creation identity must be prohibited. This is an unresolved prior finding, not counted among the new findings.

The joint simulation should use the real permission split and include late server-side completion, claim-only operations, legacy residue discovery, delayed config/adoption writes, purge teardown, distributed read reservations, automatic boot recovery, and multiple retirement events within one operation. Randomizing only the named move phases will miss these failures.

M1–M18 and the reduced follow-on scope can remain unchanged.

---

FIRST PROMPT: Joint read-only review of the host-lifecycle claim and move-any-app specs against this worktree.

WHAT: Check prior fixes, settle the interfaces, and find composition failures.

STATUS: Review complete. Neither spec is ready for planning. Workspace unchanged.

NEEDED: Nothing for this review.

RECOMMENDED: Revise the shared exclusion and recovery contracts, then run the expanded joint simulation before implementation planning.

SESSION: Safe to close. Nothing is running; no handoff file was written because the review is read-only.