amadan
plan: self-serve moves 2, rev 6 (Astra review 5: predecessor-bound writes, a nonce receipt on every transition, stale-worker fencing, pending = no confirmed request, expired open fenced not replayed, stage book reconciled per complete round, identity by boxid.Key, return handshake bound to instance and incarnation, maintenance restage)
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
1 file changed,
+33
−31
docs/superpowers/plans/2026-10-22-self-serve-moves-2-door-token-job.md+33 −31
CI cancelled — superseded by a newer push — run details
CI log
=== gate === .amadan/test-gate.sh go: downloading github.com/aws/aws-sdk-go-v2/service/cloudwatch v1.68.0 go: downloading github.com/aws/aws-sdk-go-v2/service/costexplorer v1.67.5 go: downloading github.com/aws/smithy-go v1.28.1 go: downloading github.com/aws/aws-sdk-go-v2/service/s3 v1.105.0 go: downloading github.com/aws/aws-sdk-go-v2 v1.44.0 go: downloading github.com/superfly/ltx v0.5.1 go: downloading golang.org/x/net v0.56.0 go: downloading modernc.org/sqlite v1.55.0 go: downloading github.com/benbjohnson/litestream v0.5.14 go: downloading github.com/aws/aws-sdk-go-v2/config v1.32.6 go: downloading github.com/aws/aws-sdk-go-v2/service/route53 v1.64.1 go: downloading github.com/aws/aws-sdk-go-v2/service/autoscaling v1.74.0 go: downloading github.com/aws/aws-sdk-go-v2/service/sts v1.45.6 go: downloading github.com/aws/aws-sdk-go-v2/credentials v1.19.6 go: downloading github.com/aws/aws-sdk-go-v2/service/ec2 v1.324.0 go: downloading github.com/oschwald/maxminddb-golang v1.13.1 go: downloading github.com/aws/aws-sdk-go-v2/service/iam v1.58.2 go: downloading golang.org/x/crypto v0.53.0 go: downloading golang.org/x/sys v0.46.0 go: downloading github.com/aws/aws-sdk-go-v2/service/sns v1.43.0 go: downloading github.com/aws/aws-sdk-go-v2/service/acm v1.45.0 go: downloading github.com/aws/aws-sdk-go-v2/service/resourcegroupstaggingapi v1.37.0 go: downloading github.com/pion/logging v0.2.4 go: downloading github.com/pion/stun/v3 v3.0.1 go: downloading github.com/pion/turn/v4 v4.1.4 go: downloading golang.org/x/net v0.56.0 go: downloading modernc.org/sqlite v1.55.0 go: downloading github.com/benbjohnson/litestream v0.5.14 go: downloading golang.org/x/crypto v0.53.0 go: downloading github.com/aws/aws-sdk-go-v2/service/acm v1.45.0 go: downloading github.com/aws/aws-sdk-go-v2/service/ec2 v1.324.0 go: downloading github.com/aws/aws-sdk-go-v2/service/sesv2 v1.66.6 go: downloading github.com/pierrec/lz4/v4 v4.1.22 go: downloading github.com/aws/aws-sdk-go-v2/internal/configsources v1.4.40 go: downloading github.com/pion/stun/v3 v3.0.1 go: downloading github.com/pierrec/lz4/v4 v4.1.22 go: downloading github.com/aws/aws-sdk-go-v2/internal/v4a v1.4.38 go: downloading github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.13.19 go: downloading github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.13.40 go: downloading github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.18.16 go: downloading github.com/aws/aws-sdk-go-v2/internal/ini v1.8.4 go: downloading github.com/aws/aws-sdk-go-v2/service/signin v1.0.4 go: downloading github.com/aws/aws-sdk-go-v2/service/sso v1.30.8 go: downloading github.com/aws/aws-sdk-go-v2/service/sesv2 v1.66.6 go: downloading github.com/aws/aws-sdk-go-v2/service/ssm v1.74.0 go: downloading github.com/aws/aws-sdk-go-v2/service/ssooidc v1.35.12 go: downloading github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.7.40 go: downloading github.com/pion/randutil v0.1.0 go: downloading github.com/aws/aws-sdk-go-v2/service/ssooidc v1.35.12 go: downloading github.com/pion/transport/v4 v4.0.1 go: downloading github.com/klauspost/compress v1.18.0 go: downloading github.com/pion/transport/v4 v4.0.1 go: downloading github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.7.14 go: downloading github.com/aws/aws-sdk-go-v2/service/internal/s3shared v1.19.31 go: downloading github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.7.14 go: downloading github.com/aws/aws-sdk-go-v2/service/internal/checksum v1.9.23 go: downloading github.com/aws/aws-sdk-go-v2/service/internal/checksum v1.9.23 go: downloading github.com/aws/aws-sdk-go-v2/service/internal/s3shared v1.19.31 go: downloading github.com/wlynxg/anet v0.0.5 go: downloading github.com/wlynxg/anet v0.0.5 go: downloading github.com/prometheus/client_golang v1.17.0 go: downloading github.com/aws/aws-sdk-go-v2/feature/s3/manager v1.20.18 go: downloading golang.org/x/sync v0.21.0 go: downloading github.com/lmittmann/tint v1.1.3 go: downloading github.com/mattn/go-isatty v0.0.20 go: downloading golang.org/x/sync v0.21.0 go: downloading github.com/pion/transport/v3 v3.0.8 go: downloading github.com/pion/dtls/v3 v3.0.7 go: downloading github.com/aws/aws-sdk-go-v2/feature/s3/manager v1.20.18 go: downloading github.com/mattn/go-isatty v0.0.20 go: downloading github.com/lmittmann/tint v1.1.3 go: downloading github.com/prometheus/client_golang v1.17.0 go: downloading github.com/pion/transport/v3 v3.0.8 go: downloading github.com/pion/dtls/v3 v3.0.7 go: downloading github.com/cespare/xxhash/v2 v2.3.0 go: downloading github.com/prometheus/procfs v0.12.0 go: downloading google.golang.org/protobuf v1.36.10 go: downloading github.com/prometheus/common v0.45.0 go: downloading github.com/beorn7/perks v1.0.1 go: downloading github.com/prometheus/client_model v0.5.0 go: downloading github.com/matttproud/golang_protobuf_extensions/v2 v2.0.0 go: downloading github.com/prometheus/client_model v0.5.0 go: downloading google.golang.org/protobuf v1.36.10 go: downloading golang.org/x/text v0.38.0 go: downloading golang.org/x/text v0.38.0 go: downloading modernc.org/libc v1.74.1 go: downloading modernc.org/libc v1.74.1 go: downloading modernc.org/memory v1.11.0 go: downloading modernc.org/mathutil v1.7.1 go: downloading github.com/dustin/go-humanize v1.0.1 go: downloading github.com/google/uuid v1.6.0 go: downloading modernc.org/memory v1.11.0 go: downloading github.com/dustin/go-humanize v1.0.1 go: downloading github.com/google/uuid v1.6.0 go: downloading modernc.org/mathutil v1.7.1 go: downloading github.com/remyoudompheng/bigfft v0.0.0-20230129092748-24d4a6f8daec go: downloading github.com/remyoudompheng/bigfft v0.0.0-20230129092748-24d4a6f8daec === launchctx (5 s) ok amadan.net/carlos/platform.git/launchctx 0.013s ? amadan.net/carlos/platform.git/launchctx/controlname [no test files] ? amadan.net/carlos/platform.git/launchctx/wireid [no test files] === vet (86 s) go: downloading github.com/aws/aws-sdk-go-v2/service/sesv2 v1.66.6 go: downloading golang.org/x/crypto v0.53.0 go: downloading github.com/aws/aws-sdk-go-v2/service/acm v1.45.0 go: downloading github.com/aws/aws-sdk-go-v2/service/ec2 v1.324.0 go: downloading github.com/aws/aws-sdk-go-v2/service/iam v1.58.2 go: downloading github.com/aws/aws-sdk-go-v2/service/resourcegroupstaggingapi v1.37.0 go: downloading github.com/aws/aws-sdk-go-v2/service/route53 v1.64.1 go: downloading github.com/benbjohnson/litestream v0.5.14 go: downloading modernc.org/sqlite v1.55.0 go: downloading golang.org/x/net v0.56.0 go: downloading github.com/aws/aws-sdk-go-v2/service/ssm v1.74.0 go: downloading github.com/klauspost/compress v1.18.0 go: downloading golang.org/x/tools v0.47.0 go: downloading github.com/pion/stun/v3 v3.0.1 go: downloading github.com/pierrec/lz4/v4 v4.1.22 go: downloading github.com/aws/aws-sdk-go-v2/service/ssooidc v1.35.12 go: downloading github.com/aws/aws-sdk-go-v2/service/internal/checksum v1.9.23 go: downloading github.com/wlynxg/anet v0.0.5 go: downloading github.com/pion/transport/v3 v3.0.8 go: downloading golang.org/x/sync v0.21.0 go: downloading github.com/prometheus/client_golang v1.17.0 go: downloading github.com/pion/dtls/v3 v3.0.7 go: downloading github.com/mattn/go-isatty v0.0.20 go: downloading github.com/lmittmann/tint v1.1.3 go: downloading github.com/aws/aws-sdk-go-v2/feature/s3/manager v1.20.18 go: downloading github.com/prometheus/client_model v0.5.0 go: downloading github.com/prometheus/procfs v0.12.0 go: downloading google.golang.org/protobuf v1.36.10 go: downloading github.com/matttproud/golang_protobuf_extensions/v2 v2.0.0 go: downloading golang.org/x/text v0.38.0 go: downloading golang.org/x/mod v0.37.0 go: downloading modernc.org/libc v1.74.1 go: downloading modernc.org/memory v1.11.0 go: downloading github.com/google/uuid v1.6.0 go: downloading modernc.org/mathutil v1.7.1 go: downloading github.com/dustin/go-humanize v1.0.1 go: downloading github.com/remyoudompheng/bigfft v0.0.0-20230129092748-24d4a6f8daec === gofmt (1 s) gofmt -l .: clean === rest (242 s) ? github.com/carlosframework/platform/cmd/changelog [no test files] ok github.com/carlosframework/platform/cmd/console 5.572s ok github.com/carlosframework/platform/cmd/moveprobe 13.563s ok github.com/carlosframework/platform/cmd/zddprobe 0.745s ok github.com/carlosframework/platform/internal/activator 64.308s ok github.com/carlosframework/platform/internal/agentsock 0.077s ok github.com/carlosframework/platform/internal/alerting 0.008s ok github.com/carlosframework/platform/internal/analytics 0.062s ok github.com/carlosframework/platform/internal/appconfig 4.575s ok github.com/carlosframework/platform/internal/apperr 0.314s ok github.com/carlosframework/platform/internal/applog 0.341s ok github.com/carlosframework/platform/internal/awsinv 0.012s ok github.com/carlosframework/platform/internal/boxchoice 0.015s ok github.com/carlosframework/platform/internal/boxid 0.716s ? github.com/carlosframework/platform/internal/boxid/boxidtest [no test files] ? github.com/carlosframework/platform/internal/boxid/internal/signtest [no test files] ok github.com/carlosframework/platform/internal/boxvolumes 0.034s ok github.com/carlosframework/platform/internal/caps 0.066s ok github.com/carlosframework/platform/internal/certwarm 0.014s ok github.com/carlosframework/platform/internal/cgroup 0.940s ok github.com/carlosframework/platform/internal/changelog 1.418s ok github.com/carlosframework/platform/internal/claimactivation 1.908s ok github.com/carlosframework/platform/internal/clamd 0.097s ? github.com/carlosframework/platform/internal/cliflag [no test files] ? github.com/carlosframework/platform/internal/controlname [no test files] ? github.com/carlosframework/platform/internal/controlsock [no test files] ok github.com/carlosframework/platform/internal/dbdoor 6.736s ok github.com/carlosframework/platform/internal/dbdoor/dbdoortest 0.022s ok github.com/carlosframework/platform/internal/dbverify 0.016s ok github.com/carlosframework/platform/internal/dnsrec 0.041s ok github.com/carlosframework/platform/internal/docsite 0.040s ok github.com/carlosframework/platform/internal/domains 5.011s ok github.com/carlosframework/platform/internal/doorkey 0.018s ok github.com/carlosframework/platform/internal/economics 0.010s ok github.com/carlosframework/platform/internal/edge 48.620s ? github.com/carlosframework/platform/internal/edgeproof [no test files] ok github.com/carlosframework/platform/internal/edgesfile 0.014s ok github.com/carlosframework/platform/internal/fleetapi 0.245s ok github.com/carlosframework/platform/internal/gate 1.702s ok github.com/carlosframework/platform/internal/genselect 0.045s ok github.com/carlosframework/platform/internal/geo 0.020s ok github.com/carlosframework/platform/internal/geoip 0.075s ok github.com/carlosframework/platform/internal/historyreadback 0.407s ok github.com/carlosframework/platform/internal/host 148.350s ok github.com/carlosframework/platform/internal/hostactors 6.963s ok github.com/carlosframework/platform/internal/hostactors/awsmembership 0.014s ok github.com/carlosframework/platform/internal/hostbox 4.618s ok github.com/carlosframework/platform/internal/hostclaim 1.451s ok github.com/carlosframework/platform/internal/hostenv 0.015s ok github.com/carlosframework/platform/internal/hostevidence 0.017s ok github.com/carlosframework/platform/internal/hostexec 7.740s ok github.com/carlosframework/platform/internal/hostfence 0.113s ok github.com/carlosframework/platform/internal/hostfiles 10.922s ok github.com/carlosframework/platform/internal/hostfloor 0.054s ok github.com/carlosframework/platform/internal/hostguard 1.226s ok github.com/carlosframework/platform/internal/hostops 3.177s ? github.com/carlosframework/platform/internal/hostops/fencerelease [no test files] ok github.com/carlosframework/platform/internal/hostops/internal/fencerelease 0.014s ok github.com/carlosframework/platform/internal/hostquarantine 0.003s ok github.com/carlosframework/platform/internal/hostrelocate 1.111s ok github.com/carlosframework/platform/internal/hostresolve 0.376s ok github.com/carlosframework/platform/internal/hostwrite 0.004s ok github.com/carlosframework/platform/internal/infraview 0.028s ok github.com/carlosframework/platform/internal/instancectl 5.398s ok github.com/carlosframework/platform/internal/instanceexec 2.983s ok github.com/carlosframework/platform/internal/instancefs 0.008s ok github.com/carlosframework/platform/internal/instancekey 0.007s ok github.com/carlosframework/platform/internal/instanceupdates 0.034s ok github.com/carlosframework/platform/internal/layout 2.673s ok github.com/carlosframework/platform/internal/lbhealth 0.013s ok github.com/carlosframework/platform/internal/lease 10.166s ok github.com/carlosframework/platform/internal/lease/leasetest 0.835s ok github.com/carlosframework/platform/internal/ledger 0.027s ok github.com/carlosframework/platform/internal/mail 0.415s ok github.com/carlosframework/platform/internal/maintenance 0.124s ok github.com/carlosframework/platform/internal/maintenanceexec 5.870s ok github.com/carlosframework/platform/internal/maintenanceguard 0.282s ok github.com/carlosframework/platform/internal/metrics 0.013s ok github.com/carlosframework/platform/internal/moves 34.094s ok github.com/carlosframework/platform/internal/moves/archive 1.507s ok github.com/carlosframework/platform/internal/objstore 0.600s ok github.com/carlosframework/platform/internal/ops 0.459s ok github.com/carlosframework/platform/internal/park 0.499s ok github.com/carlosframework/platform/internal/plans 0.008s ok github.com/carlosframework/platform/internal/privatefile 0.004s ok github.com/carlosframework/platform/internal/provision 56.667s ok github.com/carlosframework/platform/internal/publicrelease 0.012s ? github.com/carlosframework/platform/internal/regionname [no test files] ok github.com/carlosframework/platform/internal/registry 7.049s ok github.com/carlosframework/platform/internal/relay 7.247s ok github.com/carlosframework/platform/internal/release 35.344s ok github.com/carlosframework/platform/internal/rollout 4.932s ok github.com/carlosframework/platform/internal/rolloutwords 0.008s ok github.com/carlosframework/platform/internal/rootaws 0.014s ok github.com/carlosframework/platform/internal/rootparticipation 0.062s ok github.com/carlosframework/platform/internal/rootstate 0.030s ok github.com/carlosframework/platform/internal/runs 0.553s ok github.com/carlosframework/platform/internal/s3compat 0.005s ok github.com/carlosframework/platform/internal/safefile 0.005s ok github.com/carlosframework/platform/internal/schedule 3.506s ok github.com/carlosframework/platform/internal/seeds 0.017s ok github.com/carlosframework/platform/internal/sim/hostlife 20.153s ok github.com/carlosframework/platform/internal/sim/litestreamfacts 4.009s ok github.com/carlosframework/platform/internal/status 0.100s ok github.com/carlosframework/platform/internal/steering 0.009s ok github.com/carlosframework/platform/internal/stopevidence 0.323s ok github.com/carlosframework/platform/internal/svcrole 0.003s ok github.com/carlosframework/platform/internal/telemetry 0.005s ok github.com/carlosframework/platform/internal/termfence 0.008s ok github.com/carlosframework/platform/internal/tunnelclient 0.005s ok github.com/carlosframework/platform/internal/unixdial 0.003s ok github.com/carlosframework/platform/internal/updatewords 0.009s ok github.com/carlosframework/platform/internal/usage 0.006s ok github.com/carlosframework/platform/internal/version 0.002s ok github.com/carlosframework/platform/internal/vet 1.831s === internal_console@0 (90 s) ok github.com/carlosframework/platform/internal/console 87.241s === internal_console@1 (52 s) ok github.com/carlosframework/platform/internal/console 49.245s === internal_console@2 (53 s) ok github.com/carlosframework/platform/internal/console 50.794s === internal_console@3 (50 s) ok github.com/carlosframework/platform/internal/console 46.924s === internal_console@4 (53 s) ok github.com/carlosframework/platform/internal/console 50.772s === internal_console@5 (48 s) ok github.com/carlosframework/platform/internal/console 45.734s === internal_console@6 (51 s) ok github.com/carlosframework/platform/internal/console 47.903s === internal_console@7 (59 s) ok github.com/carlosframework/platform/internal/console 56.459s === root@0 (48 s) ok github.com/carlosframework/platform 44.692s === root@1 (52 s) ok github.com/carlosframework/platform 48.929s === root@2 (51 s) ok github.com/carlosframework/platform 48.577s === cross === --- shard 0 of 5 --- cross-compiling darwin/arm64 amadan: job cancelled — a newer push superseded it --- shard 1 of 5 --- cross-compiling darwin/amd64 --- shard 2 of 5 --- cross-compiling linux/arm64 amadan: job cancelled — a newer push superseded it --- shard 3 of 5 --- cross-compiling windows/amd64 --- shard 4 of 5 --- cross-compiling windows/arm64 amadan: job cancelled — a newer push superseded it
diff --git a/docs/superpowers/plans/2026-10-22-self-serve-moves-2-door-token-job.md b/docs/superpowers/plans/2026-10-22-self-serve-moves-2-door-token-job.md| index ffda239f9..ffaca85b2 100644 |
| --- a/docs/superpowers/plans/2026-10-22-self-serve-moves-2-door-token-job.md |
| +++ b/docs/superpowers/plans/2026-10-22-self-serve-moves-2-door-token-job.md |
| @@ -2,9 +2,9 @@ |
| > **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. |
| -Rev 5, 2026-10-22. Reviews: rev 1 (`after236/selfserve-plan2-review.md`, 20 findings), rev 2 (`selfserve-plan2-review2.md`, 21), rev 3 (`selfserve-plan2-review3.md`, 21 items), rev 4 (`selfserve-plan2-review4.md`, 21 items, 9 closed). Rev 5 is **self-contained** (nothing is "as rev N"; every schema, phase, predicate, sentence and test list is in this document) and answers rev 4's open items: the gate orders authentication, host shape and the capability check before the limiter and the no-bucket rail (item 1); operator moves keep today's admission (nil or stale metrics, no identity record) and only the self-serve path requires fresh identity; `BootID` is dropped (items 4, 8); the evidence reader takes the move and a freshness table is stated (item 5); the staged inventory is a **move-bound book** built from every copy event the mirror reports, including same-ETag and deferred copies, through a new observer hook that leaves ordinary mirroring unchanged (item 6); a participant is identified by `BoxID` **and** `InstanceID` in the resolver, in report validation and at every gate, and abort never requires the target (item 8); every request persists its **nonce, expected sequence and proposed stamp before the write**, confirmation is conditional on the record carrying that exact stamp, replay is idempotent, and consumption and the phase advance are one UPDATE predicated on the nonce (item 10); compatible sets include the predecessor state while a phase's write is pending (item 10); expiry is bound to its phase and cleared on reconciliation into another phase, with the health rule taking precedence (item 12); abandonment is conditional on the fenced observation and persisted before release, and every reclaim fences first, operator reservations included (item 13); own retries are accepted only for a complete expected selection after the whole table validates (item 14); the status view joins rows regardless of the mode (item 17); events carry a per-row sequence and are delivered inside the caller's transaction, the harness is an exported bridge **inside package host** behind the `e2e` tag with the CI step named, the Go declarations are complete, and queue ordering is stated (item 20); the open record is written **before** the open transition with the proposed open time, return requests have unique durable identities, the report path is named, acceptance is bound to the lifetime and the forward's participants, and leftovers are judged by the forward's layout (item 21). |
| +Rev 6, 2026-10-22. Rev 5's review (`selfserve-plan2-review5.md`: items 2, 3, 4, 5, 7, 9, 11, 12, 14, 15, 16, 18, 19 closed) is answered: every job and operator record write is **predecessor-bound** (`provision.SetRecordMoveAt(ctx, s, acct, app, host, expectSeq, mut)`: one CAS write against the record whose `record_seq` equals the persisted expectation, no fresh-read retry), so a voided request can never land and an operator door never rebuilds a write after a reclaim fence (items 10, 13); every self-serve mutation stamps a **nonce receipt** on the record (`SelfServe.LastRequest`, or `LastMove.Request` for a clearing write), so confirmation is exact for every transition and distinct from an operator's reconciliation (item 10); confirm and advance require an unconsumed, unvoided request, so a stale worker cannot confirm a voided one (item 10); "pending" means **no confirmed request for the phase's write**, so the interval before issuance is compatible with the predecessor state (item 10); an expired uncommitted open is fenced and aborted, never replayed (N1); the stage book is reconciled against each complete successful round so deleted keys leave it (item 6); identity records are loaded by `boxid.Key(BoxID)` and `Verified` is narrowed to what `Enroll` proves (item 8); the gate's exception for an authorised malformed hostname and plan 1's outer 503 are declared (item 1); the view joins row id, move id, self-serve id and incarnation (item 17); the open record adopts an existing complete intent, return discovery and acceptance compare `InstanceID`, incarnation is re-checked at the door and in the guarded write, and `Layout` comes from the source's `stopped` report so every operator move has it (item 21); a maintenance restage in `ready` has its own kind and keeps the confirmation deadline (N2); the gate types, `StagedRequest`, failed diagnostics, the request argument to `confirmed`, and the bridge's compile rule are stated (item 20). Reviews: rev 1 (`after236/selfserve-plan2-review.md`, 20 findings), rev 2 (`selfserve-plan2-review2.md`, 21), rev 3 (`selfserve-plan2-review3.md`, 21 items), rev 4 (`selfserve-plan2-review4.md`, 21 items, 9 closed). Rev 5 is **self-contained** (nothing is "as rev N"; every schema, phase, predicate, sentence and test list is in this document) and answers rev 4's open items: the gate orders authentication, host shape and the capability check before the limiter and the no-bucket rail (item 1); operator moves keep today's admission (nil or stale metrics, no identity record) and only the self-serve path requires fresh identity; `BootID` is dropped (items 4, 8); the evidence reader takes the move and a freshness table is stated (item 5); the staged inventory is a **move-bound book** built from every copy event the mirror reports, including same-ETag and deferred copies, through a new observer hook that leaves ordinary mirroring unchanged (item 6); a participant is identified by `BoxID` **and** `InstanceID` in the resolver, in report validation and at every gate, and abort never requires the target (item 8); every request persists its **nonce, expected sequence and proposed stamp before the write**, confirmation is conditional on the record carrying that exact stamp, replay is idempotent, and consumption and the phase advance are one UPDATE predicated on the nonce (item 10); compatible sets include the predecessor state while a phase's write is pending (item 10); expiry is bound to its phase and cleared on reconciliation into another phase, with the health rule taking precedence (item 12); abandonment is conditional on the fenced observation and persisted before release, and every reclaim fences first, operator reservations included (item 13); own retries are accepted only for a complete expected selection after the whole table validates (item 14); the status view joins rows regardless of the mode (item 17); events carry a per-row sequence and are delivered inside the caller's transaction, the harness is an exported bridge **inside package host** behind the `e2e` tag with the CI step named, the Go declarations are complete, and queue ordering is stated (item 20); the open record is written **before** the open transition with the proposed open time, return requests have unique durable identities, the report path is named, acceptance is bound to the lifetime and the forward's participants, and leftovers are judged by the forward's layout (item 21). |
| -Builds on plan 1 (`2026-10-22-self-serve-moves-1-lifetimes-and-order.md`, rev 11): records carry a lifetime, `record_seq` and `move_seq`, rows carry the pair, readers and writers are judged, `fenceRecord(ctx, acct, app, host, op fenceOwner) (fenceResult, Observation, error)` exists, the console runs behind the lifetimes cutover. This plan adds everything a Tito customer's request needs up to a finished move except the holding page, the edge probe mode and cleanup (plans 3 and 4). |
| +Builds on plan 1 (`2026-10-22-self-serve-moves-1-lifetimes-and-order.md`, rev 13): records carry a lifetime, `record_seq` and `move_seq`, rows carry the pair, readers and writers are judged, `fenceRecord(ctx, acct, app, host, op fenceOwner) (fenceResult, Observation, error)` exists, the console runs behind the lifetimes cutover. This plan adds everything a Tito customer's request needs up to a finished move except the holding page, the edge probe mode and cleanup (plans 3 and 4). |
| **Goal:** With `CARLOS_SELF_SERVE_MOVES=offline` on a claims-off, Phase-1-off console, Tito's existing residency calls (prepare, status, cut, cancel) run the offline move as a console job: an owner-delegated move-capable token asks, the door refuses what cannot be offered before anything is written, per-box reservations serialise moves per box, the boxes prove the lane, the target's cleanliness and the source's eligibility before any copy and again before the stop, staging runs until caught up, the cut runs start, seal, flip, open, health, finish with a deadline on every wait, every failure before the committed `open` initiates a lossless abort, and the view tells Tito `preparing`, `ready`, `cutting`, `none` with `last` and the abort's three facts. The operator's return accepts exactly the forward move's leftovers. |
| @@ -16,14 +16,14 @@ Builds on plan 1 (`2026-10-22-self-serve-moves-1-lifetimes-and-order.md`, rev 11 |
| ## Global Constraints |
| -- **Mode.** `selfServeMode()` is a process-level flag: true when `CARLOS_SELF_SERVE_MOVES=offline` (`cfg.SelfServeMoves`) with `!cfg.HostClaim && !cfg.MovesEnabled`, or when the console found an open self-serve row, a pending release, an undelivered event or any `active.json` reservation at start (recovery mode; re-evaluated by the pass every tick and never turned off while any of those exist). In self-serve mode the three mutation doors are the self-serve doors; outside it they are the legacy doors byte for byte. **Admission** (a new prepare) needs the switch on as well. **Rendering never depends on the mode**: the status door joins rows whenever the console is claims-off and Phase 1 is off (Task 5). |
| +- **Mode.** `selfServeMode()` is a process-level flag: true when `CARLOS_SELF_SERVE_MOVES=offline` (`cfg.SelfServeMoves`) with `!cfg.HostClaim && !cfg.MovesEnabled`, or when the console found an open self-serve row, a pending release, an undelivered event or any `active.json` reservation at start (recovery mode; re-evaluated by the pass every tick and never turned off while any of those exist). In self-serve mode the three mutation doors are the self-serve doors; outside it they are the legacy doors byte for byte. **Admission** (a new prepare) needs the switch on as well. **Rendering never depends on the mode**: the status door joins rows whenever the console is claims-off and Phase 1 is off (Task 5). **Plan 1's lifetimes admission (`lifetimesAdmit`) wraps every record route first**: in `unknown` or `draining` its 503 answers before any gate of this plan, authorised or not; the route tests state that precedence. |
| - The operator doors keep their answers byte for byte (snapshots of every refusal, including a malformed body combined with each class refusal, replayed from main). A job tick never calls an HTTP door. |
| - Every wait has a deadline evaluated from the row before any evidence is read; an expiry is **bound to the phase that set it** (`expiry_phase`) and the action is decided by the record (Task 8): while the record is `offline-landed` or earlier → abort; committed open → page; unreadable → `open-uncertain`; the health phase's expiry → warning and `finishing` (precedence over the page rule). |
| -- Every record transition the job writes goes through `selfServeApply` (Task 7) with a **persisted request**: nonce, expected `record_seq` and the exact proposed stamp are written to the row before the record write; the mutation carries the proposed stamp verbatim; confirmation is the record carrying it; replay is the same write again; after every landed write the serial is bumped at least once (`notified_seq`). |
| +- Every record transition the job writes goes through `selfServeApply` (Task 7) with a **persisted request**: nonce, expected `record_seq` and the exact proposed stamp are written to the row before the record write; the write is `provision.SetRecordMoveAt` **bound to that expected sequence** (one CAS attempt, no fresh-read retry); the mutation carries the proposed stamp verbatim **and the nonce as a receipt** (`Move.SelfServe.LastRequest = nonce`, or `LastMove.Request = nonce` for a write that clears the move); confirmation is the record carrying the receipt; replay is the same bound write again; after every landed write the serial is bumped at least once (`notified_seq`). Operator doors use the same bound write with the sequence they observed before reserving (Task 3). |
| - One console runs the pass under `control/offline-moves/console.json` (a 60 s lease, renewed every 20 s during a tick, checked before every write attempt with a fresh clock, and bounding every store write's context). The SQLite journal is the console's; a replacement console takes over by opening the **same** database file (one console host per deployment; the lease guards the day there are two) and recovers from the rows and the records. |
| - Refusals before writes; the record checks run again from the fresh judged record immediately before the queue write and before the start. |
| - **Participants** are pinned at admission as `{BoxID, Label, InstanceID string; Verified bool}` (`BootID` is not pinned: it has no claims-off producer and a reboot does not change the machine's identity). **One resolver** (`moves.ParticipantRole`) decides a box's role in every box path and every console gate from `BoxID` **and** `InstanceID`; **one identity check** (`moves.ParticipantStillPresent`, Task 6) at every gate. A participant pinned with an empty `InstanceID` (the operator path on a box without a fresh report) keeps today's region rule for that participant and forces the sentinel reservation. A move without `Participants` keeps today's region rule everywhere. |
| -- Timestamps in SQLite are Unix nanoseconds; request identities are **proposed** stamps persisted before the write and confirmed by the record. |
| +- Timestamps in SQLite are Unix nanoseconds; request identities are the **nonce** persisted before the write and read back from the record's receipt. |
| - Copy rows pending Paul's review (every sentence in quotes in Tasks 5, 7 and 8). No tenant names: `shop.example.com`, account `bab`, app `shop`, display name "Shop's account". |
| - Census: `internal/provision/testdata/plan14-root-reads-self-serve-moves-2.tsv` loaded by a new `loadRootReadInventory(..., true)` block in `internal/provision/plan14_root_reads_test.go` after every existing overlay and before `compareRootReads`, `rev3=PROPOSED-self-serve-moves-2`, `@fields` for the new record fields (`self_serve`, `participants`, `queued_at`, `precheck_at`, `return_of`), reviewed dispositions for the changed envelopes (`internal/host`, `internal/status`, `internal/moves`, `internal/console`, `internal/lease`); `TestPlan9B8RecordWriteCensus` takes `plan14SelfServeMoves2WriteCensusInventory()` wrapping plan 1's with every added or changed writer: the new transitions, the box's report writers (lane, precheck, ready, staged, reverting, return-ready), the console's `open.json`, the return request writer, the console lease, `active.json`'s table form, the write guard; both green. |
| - Local tests: touched packages with `-run`; CI runs the full suite **and** `go test -tags e2e ./internal/e2e/...` (Task 10 adds the step to `.amadan/test-gate.sh`). |
| @@ -47,11 +47,11 @@ Builds on plan 1 (`2026-10-22-self-serve-moves-1-lifetimes-and-order.md`, rev 11 |
| **Interfaces:** |
| - `func (s *Store) NewAppTokenContext(ctx context.Context, accountID int64, app, mintedBy, label string, moves bool, now time.Time) (string, error)`; `func (s *Store) NewAppToken(accountID int64, app, mintedBy, label string, moves bool, now time.Time) (string, error)`; existing callers pass `false`. |
| -- `func (s *Server) selfServeGate(w http.ResponseWriter, r *http.Request) (p Principal, acct *Account, app, host string, ok bool)`, **in this order**: (1) `apiAccountAppSaying(w, r, s.store.HasAppIn, mainMoveCopy().gate)` (authentication and the account/app binding: today's answers for an invalid or revoked credential; its SQL is **binding SQL**, the only SQL this gate performs); (2) `host := r.PathValue("host")` with `ValidHostShape` (404 as `moveGate`); (3) `moveCapable(p, acct, app)` else `http.NotFound(w, r)` (`moveCapable`: `Kind == PrincipalService && Role == svcrole.Instance && AccountID == acct.ID && App == app && Moves`); (4) the instance gate's **rate limiter** and (5) the **no-deployment-bucket 503**, exactly as `instanceGateSaying` applies them, factored into `instanceRails(w, r, acct)` so both gates share one implementation. An unauthorised caller therefore always gets the 404 even under an exhausted limiter or with no bucket; an authorised caller keeps those rails. No store read and no evidence SQL (no `self_serve_moves`, no records) before the gate returns. |
| +- `func (s *Server) selfServeGate(w http.ResponseWriter, r *http.Request) (p Principal, acct *Account, app, host string, ok bool)`, **in this order**: (1) `apiAccountAppSaying(w, r, s.store.HasAppIn, mainMoveCopy().gate)` (authentication and the account/app binding: today's answers for an invalid or revoked credential; its SQL is **binding SQL**, the only SQL this gate performs); (2) `host := r.PathValue("host")` with `ValidHostShape` (404 as `moveGate`); (3) `moveCapable(p, acct, app)` else `http.NotFound(w, r)` (`moveCapable`: `Kind == PrincipalService && Role == svcrole.Instance && AccountID == acct.ID && App == app && Moves`); (4) the instance gate's **rate limiter** and (5) the **no-deployment-bucket 503**, exactly as `instanceGateSaying` applies them, factored into `instanceRails(w, r, acct, noun string, c gateCopy) bool` (the noun and the copy family `instanceGateSaying` uses, so its sentences are preserved) so both gates share one implementation. `apiAccountAppSaying` returns the audit actor; the principal is `principalFrom(r.Context())`. An unauthorised caller therefore always gets the 404 even under an exhausted limiter or with no bucket; an authorised caller keeps those rails. **Declared exception:** an authorised move-capable token with a malformed hostname gets the 404 before the rails (today's `moveGate` would answer 429/503 first); it is tested as such. No store read and no evidence SQL (no `self_serve_moves`, no records) before the gate returns. |
| - **The dispatcher**: `handleAPIInstanceMoveRequest`, `Cut`, `Abort`: `s.cfg.MovesEnabled` → Phase 1 as today; `s.selfServeMode()` → the self-serve body (Task 5); else the legacy body. `handleAPIInstanceMoveStatus`: `s.cfg.MovesEnabled` → Phase 1 as today; else `moveGate` (today's; an absent record is today's 404), then, whenever `!cfg.HostClaim && !cfg.MovesEnabled` (**not** gated on `selfServeMode`), the row join (Task 5's precedence; a SQL failure → 502 "could not read the move"); a host with no row → today's view. |
| - The CLI (`carlos instance-token --moves`) is a carlos-repo change. Existing tokens have `moves = 0`. #155's host-bound tokens: a comment names the future refusal. |
| -- [ ] **Step 1: Write the failing tests**: `TestOwnerMintsAMoveCapableToken` (both listings, `Principal.Moves`, `Principal.TokenID` equals the listing id, the audit line); `TestMemberCannotMintAMoveCapableToken`; `TestPlainMintIsUnchanged`; `TestMoveCapableGateAtTheRoutes` (a counting store and a counting SQL wrapper that classifies statements as binding (`cli_tokens`, `accounts`, `apps`) or evidence (everything else): in self-serve mode, for each of a plain instance token, an admin token, another app's move-capable token and a member's token, against absent, eligible, ineligible and unreadable records, with admission on and off, **with the limiter exhausted and with no deployment bucket** → the 404 with zero store reads and zero evidence SQL; the owner's move-capable token passes and then meets the limiter's 429 and the no-bucket 503 as the instance gate gives them; a revoked token → today's 401; outside self-serve mode a plain instance token reaches the legacy body exactly as main). |
| +- [ ] **Step 1: Write the failing tests**: `TestOwnerMintsAMoveCapableToken` (both listings, `Principal.Moves`, `Principal.TokenID` equals the listing id, the audit line); `TestMemberCannotMintAMoveCapableToken`; `TestPlainMintIsUnchanged`; `TestMoveCapableGateAtTheRoutes` (plan 1's outer `lifetimesAdmit` in `unknown` and `draining` → 503 before this gate for every caller; then, in `on`: a counting store and a counting SQL wrapper that classifies statements as binding (`cli_tokens`, `accounts`, `apps`) or evidence (everything else): in self-serve mode, for each of a plain instance token, an admin token, another app's move-capable token and a member's token, against absent, eligible, ineligible and unreadable records, with admission on and off, **with the limiter exhausted and with no deployment bucket** → the 404 with zero store reads and zero evidence SQL; the owner's move-capable token passes and then meets the limiter's 429 and the no-bucket 503 as the instance gate gives them; the owner's token with a malformed hostname → 404 before the rails (the declared exception); a revoked token → today's 401; outside self-serve mode a plain instance token reaches the legacy body exactly as main). |
| - [ ] **Step 2: Run to verify they fail**: `go test ./internal/console -run 'MoveCapable|Mint' -count=1`. |
| - [ ] **Step 3: Implement.** |
| - [ ] **Step 4: Run to verify they pass.** |
| @@ -65,13 +65,13 @@ Builds on plan 1 (`2026-10-22-self-serve-moves-1-lifetimes-and-order.md`, rev 11 |
| **Interfaces:** |
| - `const OfflineQueued = "offline-queued"`. |
| -- `type SelfServeMove struct { ID string \`json:"id"\`; Incarnation uint64 \`json:"incarnation"\`; DisplayName string \`json:"display_name,omitempty"\`; RequestedBy string \`json:"requested_by,omitempty"\`; EstimateSeconds int \`json:"estimate_seconds,omitempty"\`; OfflineAt *time.Time \`json:"offline_at,omitempty"\` }`. |
| +- `type SelfServeMove struct { ID string \`json:"id"\`; Incarnation uint64 \`json:"incarnation"\`; DisplayName string \`json:"display_name,omitempty"\`; RequestedBy string \`json:"requested_by,omitempty"\`; EstimateSeconds int \`json:"estimate_seconds,omitempty"\`; OfflineAt *time.Time \`json:"offline_at,omitempty"\`; LastRequest string \`json:"last_request,omitempty"\` }` (**the receipt**: every self-serve mutation sets it to the request's nonce); `LastMove` gains `Request string \`json:"request,omitempty"\`` (the receipt of the clearing write: `OfflineFinish`, `OfflineAbortAt`, the abort path's finish). **Every transition the job writes takes `req provision.Receipt{Nonce string}`** as an additive final parameter (`OfflineQueue`, `OfflineQueuedToStaging`, `OfflinePrecheck`, `OfflineRestage`, `OfflineSetEstimate`, `OfflineStartStaged`, `OfflineSeal`, `OfflineFlip`, `OfflineOpenTransition`, `OfflineFinishingTransition`, `OfflineFinish`, `OfflineAbort`, `OfflineAbortAt`): a non-empty nonce is written to `SelfServe.LastRequest` (or `LastMove.Request`); the operator doors pass an empty receipt, which leaves the field unchanged, so an operator transition never confirms a job request. |
| - `type Participant struct { BoxID string \`json:"box_id"\`; Label string \`json:"label"\`; InstanceID string \`json:"instance_id,omitempty"\`; Verified bool \`json:"verified,omitempty"\` }`; `type Participants struct { Source []Participant \`json:"source"\`; Target Participant \`json:"target"\` }`. |
| - `func OfflineQueue(id, from, to, by string, st StageTarget, p Participants, ss SelfServeMove, at time.Time) func(*Record) error`: `OfflineStage`'s refusals (a move in flight, a seed block, an open run, the same region) plus `rec.Incarnation != ss.Incarnation` → `fmt.Errorf("%w: the instance was recreated since the move was requested", ErrMoveConflict)`; creates the move in `offline-queued` with `Kind: offline`, `From`, `To`, `Stager: p.Source[0].Label`, `StageBucket/StageRegion`, `Participants`, `SelfServe`, `RequestedAt: at`, `QueuedAt: &at`, `StagedAt: &at` (the pre-copy intent exists, so `StartedOffline()` is false until the start), `PrecheckAt: &at`, `Seq: rec.MoveSeq`; `move_seq` unchanged. **`at` is the caller's proposed stamp** and is stored verbatim (no clock read inside the mutation); the same holds for every transition below that takes `at`. |
| -- `func OfflineQueuedToStaging(id string, at time.Time) func(*Record) error` (`StagedAt = &at`); `func OfflinePrecheck(id string, at time.Time) func(*Record) error` (queued or staging; `PrecheckAt = &at`); `OfflineRestage(id, at)` stores `at` verbatim (today's signature; the console passes its proposed stamp); `func OfflineSetEstimate(id string, seconds int) func(*Record) error` (queued or staging; `SelfServe == nil` → `ErrMoveConflict` "not a self-serve move"; equal → `errRecordUnchanged`); `func OfflineAbortAt(id string, at time.Time) func(*Record) error` (from `offline-queued` only: clears the move with `LastMove = {ID, aborted, At: at, OfflineS: 0}`, `move_seq` unchanged; every other state → `OfflineAbort`'s behaviour; a repeated abort on a reverting record stays a no-write success); `OfflineStage` and `OfflineStart` gain `returnOf string` (additive; "" for every existing caller); `OfflineStartStaged(id, at)` sets `SelfServe.OfflineAt = &at` when `SelfServe != nil`; `OfflineOpenTransition(id, at)` sets `OpenedAt = at`. Every other transition refuses `offline-queued` with `ErrMoveConflict` and the existing `offlineStateRefusal` words. |
| -- **The open record** `type moves.OpenRecord struct { MoveID string \`json:"move_id"\`; Host string \`json:"host"\`; Account string \`json:"account"\`; App string \`json:"app"\`; Incarnation uint64 \`json:"incarnation"\`; From string \`json:"from"\`; To string \`json:"to"\`; StageBucket string \`json:"stage_bucket"\`; StageRegion string \`json:"stage_region"\`; Participants *provision.Participants \`json:"participants,omitempty"\`; Layout string \`json:"layout"\` (`flat` or `nested`, from the source precheck's `Layout`); OpenedAt time.Time \`json:"opened_at"\` }` at `OpenRecordKey(host, id) = "control/offline-moves/<host>/<id>/open.json"`, written **create-only before** the open transition with `OpenedAt` = the proposed open stamp, by the operator open door and by the job alike; the open transition then carries the same `at`; a failed open after the record was written leaves an open record for a move whose record region never changed, which the return door rejects (`To != rec.Region`). Because the record exists before the open can commit, no clearing transition can outrun it. |
| +- `func OfflineQueuedToStaging(id string, at time.Time) func(*Record) error` (`StagedAt = &at`); `func OfflinePrecheck(id string, at time.Time) func(*Record) error` (queued or staging; `PrecheckAt = &at`); `OfflineRestage(id, at)` stores `at` verbatim (today's signature; the console passes its proposed stamp); `func OfflineSetEstimate(id string, seconds int) func(*Record) error` (queued or staging; `SelfServe == nil` → `ErrMoveConflict` "not a self-serve move"; equal → `errRecordUnchanged`); `func OfflineAbortAt(id string, at time.Time, req Receipt) func(*Record) error` (from `offline-queued` only: clears the move with `LastMove = {ID, aborted, At: at, OfflineS: 0}`, `move_seq` unchanged; every other state → `OfflineAbort`'s behaviour; a repeated abort on a reverting record stays a no-write success); `OfflineStage` and `OfflineStart` gain `returnOf string` (additive; "" for every existing caller); `OfflineStartStaged(id, at)` sets `SelfServe.OfflineAt = &at` when `SelfServe != nil`; `OfflineOpenTransition(id, at)` sets `OpenedAt = at`. Every other transition refuses `offline-queued` with `ErrMoveConflict` and the existing `offlineStateRefusal` words. |
| +- **The open record** `type moves.OpenRecord struct { MoveID string \`json:"move_id"\`; Host string \`json:"host"\`; Account string \`json:"account"\`; App string \`json:"app"\`; Incarnation uint64 \`json:"incarnation"\`; From string \`json:"from"\`; To string \`json:"to"\`; StageBucket string \`json:"stage_bucket"\`; StageRegion string \`json:"stage_region"\`; Participants *provision.Participants \`json:"participants,omitempty"\`; Layout string \`json:"layout"\` (`flat` or `nested`, from the source precheck's `Layout`); OpenedAt time.Time \`json:"opened_at"\` }` at `OpenRecordKey(host, id) = "control/offline-moves/<host>/<id>/open.json"`, written **create-only before** the open transition, by the operator open door and by the job alike: the writer first reads the key; an existing **complete** record for this `MoveID` is **adopted** (its `OpenedAt` becomes the proposed open stamp of this attempt, so a retry after a crash between the record and the transition opens at the recorded time); absent → written with this attempt's proposed stamp; present for another move id → 409 "another move's open record is in the way" (operator repair). `Layout` comes from the source's `stopped` report (`OfflineReport.Layout`, filled by every source at stop, so a direct operator move has it too). The open transition then carries the same `at`. Proof that the open **committed** is `rec.Move.OpenedAt == open.OpenedAt` while the move is on the record, or `LastMove.ID == open.MoveID && LastMove.Outcome == done` after it cleared; the return door requires the latter (a region comparison proves nothing: `OfflineFlip` changes the region before open). Because the record exists before the open can commit, no clearing transition can outrun it. |
| -- [ ] **Step 1: Write the failing tests**: `TestOfflineQueueThenStaging` (fields; `StartedOffline() == false`; `Seq == MoveSeq`; the stamps are the caller's `at` verbatim); `TestOfflineQueueRefusesARecreatedLifetime`; `TestOfflineQueuedRefusesEveryOtherVerb`; `TestOfflinePrecheckMovesOnlyTheStamp`; `TestOfflineAbortAtClearsAQueuedMove`; `TestOfflineSetEstimateRefusesWithoutSelfServe`; `TestReturnOfCarriedByStageAndStart`; `TestOpenRecordPrecedesTheOpenTransition` (the operator open door and the job write it before the transition with the same `OpenedAt`; a second write is a no-op; a failed transition leaves a record the return door rejects); the allowlist test. |
| +- [ ] **Step 1: Write the failing tests**: `TestOfflineQueueThenStaging` (fields; `StartedOffline() == false`; `Seq == MoveSeq`; the stamps are the caller's `at` verbatim); `TestOfflineQueueRefusesARecreatedLifetime`; `TestOfflineQueuedRefusesEveryOtherVerb`; `TestOfflinePrecheckMovesOnlyTheStamp`; `TestOfflineAbortAtClearsAQueuedMove`; `TestOfflineSetEstimateRefusesWithoutSelfServe`; `TestReturnOfCarriedByStageAndStart`; `TestOpenRecordPrecedesTheOpenTransition` (the operator open door and the job write it before the transition with the same `OpenedAt`; a crash between the record and the transition → the retry adopts the recorded `OpenedAt` and the record agrees; another move's record → 409; a failed open leaves a record the return door rejects because `LastMove` does not say `done`); `TestReceiptsOnEveryTransition` (each listed transition with a nonce writes `LastRequest` or `LastMove.Request`; an empty receipt leaves it unchanged); the allowlist test. |
| - [ ] **Step 2: Run to verify they fail; Step 3: implement; Step 4: run** `go test ./internal/provision -run 'OfflineQueue|OfflinePrecheck|OfflineAbortAt|SetEstimate|ReturnOf|DiffersOnlyWhere' -count=1`, `go test ./internal/console -run 'OpenRecord' -count=1`, then the record-write census. |
| - [ ] **Step 5: Commit** `provision: the offline-queued state, the precheck stamp, the queued abort, caller-proposed stamps, the open record and the self-serve fields`. |
| @@ -87,9 +87,10 @@ Builds on plan 1 (`2026-10-22-self-serve-moves-1-lifetimes-and-order.md`, rev 11 |
| - `func (s *Server) reserveBoxes(ctx context.Context, sel offlineSelection, r moves.OfflineReservation) *offlineRefusal`, in order: (1) read `active.json` (absent = empty); a body with no `Reservations` and `Host != ""` is a **legacy lock** reserving every box for `{move, Op, Host, Account, App, At}`, normalised into the table on the first new write; (2) **validate the whole object**: any malformed entry → `unreadable`; (3) **own retry**: entries with this `(Kind, ID)` exist → accepted only when they cover **exactly** the expected keys (every participant's `BoxID`, or `"*"` for a sentinel selection) with no extra; a partial own reservation → the missing keys are acquired in the same write if free, else `busy`; an own reservation whose keys differ from the expected selection → refuse "the reservation does not match this move's participants" (the job aborts); an accepted retry keeps `ReservedAt`; (4) `cfg.SelfServeMoves != "offline"` → refuse `busy` when any entry of another operation exists (today's deployment-wide semantics); (5) any expected key held by another `(Kind, ID)` → `busy` ("<host> is moving offline. move one instance at a time") unless reclaimable (below); (6) one conditional `Put` with every expected key and the projection (`Host, Op, Account, App, At` of the oldest `move` entry; of the oldest entry of any kind when no move holds); a CAS conflict re-reads once and retries from (2), then `busy`. |
| - `func (s *Server) releaseBoxes(ctx context.Context, kind, id string) error`: removes every entry with that `(Kind, ID)`, rewrites the projection, deletes the object when empty (`DeleteIfMatch`); absence is success. |
| - **Reclaim** (`time.Since(ReservedAt) >= offlineLockGrace`), for `move`: (a) `fenceRecord(ctx, acct, app, host, fenceOwner{})` (plan 1) must return `fenced` or `absent` (an error → not reclaimable); (b) the fenced observation carries no move with `ID`; (c) `SelfServeMove(id)` is absent or terminal with `release_pending = 0`; (d) no row for the host has `request_issued` set. An **operator** reservation (no row) is reclaimed on (a) and (b) alone: the fence makes a delayed operator start or stage PUT fail its CAS, so a negative observation after the fence is conclusive. For `cleanup` (plan 4): the owning box's status report (`status/boxes/<label>.json`, `GeneratedAt` within 2 status intervals, `Metrics.BoxID` equal to the key) lists the id in a terminal state with `RootRequest == ""` and `ReleasePending == false`, or its `Retirements` block is complete (`RetirementsComplete`) and lists no such id. Any read failure, a malformed body, or an absent or incomplete status → not reclaimable. |
| +- **Operator admission is predecessor-bound too:** the start and stage doors observe the record (judged) **before** reserving, reserve with `OfflineReservation.ExpectSeq = rec.RecordSeq` (a new tagged field), then write the record with `provision.SetRecordMoveAt(ctx, ..., ExpectSeq, mut)`; a CAS conflict → 409 "the instance changed while the move was being started. try again" and the door releases its reservation; the door never re-reads and retries. A reclaim's fence advances `record_seq`, so a paused door that resumes after the reclaim cannot write (its bound CAS fails) and cannot rebuild a write. |
| - **Rollout and rollback:** old consoles cannot be reintroduced while any reservation exists that the old console did not create (its release deletes the whole object when the projection names its own move; `TestOldConsoleReleaseErasesATable` documents the hazard); the supported sequence is roll consoles → turn the switch on; rollback is switch off → wait until **every incompatible operation** ends (every self-serve row terminal and released, every cleanup reservation gone, every operator reservation created under the table form released) → the old console. |
| -- [ ] **Step 1: Write the failing tests**: `TestReserveWritesEveryParticipantAndTheProjection`; `TestReserveRefusesAHeldParticipant`; `TestOwnRetryNeedsTheCompleteSelection` (exact → accepted keeping `ReservedAt`; partial with free keys → completed in one write; partial with a held key → busy; a changed selection → the mismatch refusal; a malformed unrelated entry is detected before the retry is accepted); `TestSentinelReservationConflictsBothWays`; `TestDisjointBoxesReserveAtOnceWithTheSwitch`; `TestWithoutTheSwitchOneReservationAtATime`; `TestLegacyLockBodyIsNormalised`; `TestMalformedEntryIsUnreadable`; `TestReclaimOnlyOnEvidence` (each positive case and each refusing case above; a delayed queue PUT that commits after a negative re-read and before the fence → the fence observes it and the reservation is kept; a delayed operator start PUT → the fence fails it and the reservation is reclaimed; a fence error → kept; a SQL failure → kept; a cleanup-only obligation at switch-off → kept and recovery proceeds); `TestReleaseRemovesOnlyItsEntriesAndDeletesWhenEmpty`; `TestOldConsoleAlgorithmsAgainstTheNewBody`; `TestSwitchOffRecoveryReacquiresItsOwnReservation` (two disjoint reservations, each retries its own → accepted); `TestOperatorDoorsUseReservations`. |
| +- [ ] **Step 1: Write the failing tests**: `TestReserveWritesEveryParticipantAndTheProjection`; `TestReserveRefusesAHeldParticipant`; `TestOwnRetryNeedsTheCompleteSelection` (exact → accepted keeping `ReservedAt`; partial with free keys → completed in one write; partial with a held key → busy; a changed selection → the mismatch refusal; a malformed unrelated entry is detected before the retry is accepted); `TestSentinelReservationConflictsBothWays`; `TestDisjointBoxesReserveAtOnceWithTheSwitch`; `TestWithoutTheSwitchOneReservationAtATime`; `TestLegacyLockBodyIsNormalised`; `TestMalformedEntryIsUnreadable`; `TestReclaimOnlyOnEvidence` (each positive case and each refusing case above; a delayed queue PUT that commits after a negative re-read and before the fence → the fence observes it and the reservation is kept; a delayed operator start PUT → the fence fails it and the reservation is reclaimed; **an operator door paused between reserving and its record write** while the reclaim fences and another operation is admitted → the door's bound write fails with 409, nothing starts; a fence error → kept; a SQL failure → kept; a cleanup-only obligation at switch-off → kept and recovery proceeds); `TestReleaseRemovesOnlyItsEntriesAndDeletesWhenEmpty`; `TestOldConsoleAlgorithmsAgainstTheNewBody`; `TestSwitchOffRecoveryReacquiresItsOwnReservation` (two disjoint reservations, each retries its own → accepted); `TestOperatorDoorsUseReservations`. |
| - [ ] **Step 2: Run to verify they fail; Step 3: implement; Step 4: run** `go test ./internal/console -run 'Reserve|Reclaim|Release|OldConsole|Sentinel|OwnRetry|OperatorDoorsUse|Malformed' -count=1`. |
| - [ ] **Step 5: Commit** `console: per-box reservations with a legacy projection, a sentinel, complete own retries and fenced reclaim`. |
| @@ -101,16 +102,16 @@ Builds on plan 1 (`2026-10-22-self-serve-moves-1-lifetimes-and-order.md`, rev 11 |
| - Test: `internal/console/selfserve_store_test.go`, `internal/console/selfserve_pass_test.go` |
| **Interfaces:** |
| -- Table `self_serve_moves (id TEXT PRIMARY KEY, account TEXT, app TEXT, host TEXT, incarnation INTEGER, from_region TEXT, to_region TEXT, participants TEXT, display_name TEXT, requested_by TEXT, phase TEXT, phase_started_at INTEGER, step_deadline INTEGER, window_deadline INTEGER, expiry_pending INTEGER, expiry_phase TEXT, staging_started_at INTEGER, request_issued INTEGER, request_kind TEXT, request_phase TEXT, request_nonce TEXT, request_expect_seq INTEGER, request_proposed INTEGER, request_confirmed INTEGER, request_consumed INTEGER, stage_request INTEGER, precheck_request INTEGER, rounds INTEGER, estimate_seconds INTEGER, estimate_terms TEXT, reason TEXT, health_attempts INTEGER, health_consecutive_ok INTEGER, health_first_ok_at INTEGER, health_last_probe_at INTEGER, health_last_error TEXT, finish_gate TEXT, notified_seq INTEGER, event_seq INTEGER, outcome TEXT, outcome_reason TEXT, warning TEXT, abort_requested_at INTEGER, source_reopened_at INTEGER, abort_finished_at INTEGER, effects_done INTEGER, release_pending INTEGER, open_record_pending INTEGER, created_at INTEGER, updated_at INTEGER)`; `CREATE UNIQUE INDEX IF NOT EXISTS self_serve_moves_open ON self_serve_moves(account, app, host) WHERE outcome = ''`. |
| -- **The request lifecycle** (one request at a time per row): `IssueSelfServeRequest(id, kind, phase string, nonce string, expectSeq uint64, proposed time.Time, now time.Time) (bool, error)` (`WHERE request_consumed <> 0 OR request_issued = 0`: sets `request_issued = now, request_kind, request_phase, request_nonce, request_expect_seq, request_proposed, request_confirmed = 0, request_consumed = 0`); `ConfirmSelfServeRequest(id, nonce string, now time.Time) error` (`WHERE request_nonce = ?`: `request_confirmed = now`; for kind `stage` also `stage_request = request_proposed`, for `precheck` also `precheck_request = request_proposed`); `VoidSelfServeRequest(id, nonce string) error` (`request_consumed = 1` without confirmation; the phase function decides again); `AdvanceSelfServe(id, from, to, reason string, step, window time.Time, nonce string, now time.Time) (bool, error)`: one UPDATE with the predicate `phase = ? AND (abort_requested_at = 0 OR ? IN ('reverting','done','opened','health','finishing')) AND (? = '' OR request_nonce = ?)` that sets the phase, `phase_started_at`, the deadlines, `expiry_pending = 0, expiry_phase = ''` when `to <> expiry_phase`, and `request_consumed = 1` when a nonce is given (**consumption and the advance are one statement**), and inserts the phase event in the same transaction (`event_seq = event_seq + 1`); a stale worker holding another nonce cannot consume. |
| +- Table `self_serve_moves (id TEXT PRIMARY KEY, account TEXT, app TEXT, host TEXT, incarnation INTEGER, from_region TEXT, to_region TEXT, participants TEXT, display_name TEXT, requested_by TEXT, phase TEXT, phase_started_at INTEGER, step_deadline INTEGER, window_deadline INTEGER, expiry_pending INTEGER, expiry_phase TEXT, staging_started_at INTEGER, request_issued INTEGER, request_kind TEXT, request_phase TEXT, request_nonce TEXT, request_expect_seq INTEGER, request_proposed INTEGER, request_confirmed INTEGER, request_consumed INTEGER, request_voided INTEGER, stage_request INTEGER, precheck_request INTEGER, rounds INTEGER, estimate_seconds INTEGER, estimate_terms TEXT, reason TEXT, health_attempts INTEGER, health_consecutive_ok INTEGER, health_first_ok_at INTEGER, health_last_probe_at INTEGER, health_last_error TEXT, finish_gate TEXT, notified_seq INTEGER, event_seq INTEGER, outcome TEXT, outcome_reason TEXT, warning TEXT, abort_requested_at INTEGER, source_reopened_at INTEGER, abort_finished_at INTEGER, effects_done INTEGER, release_pending INTEGER, open_record_pending INTEGER, created_at INTEGER, updated_at INTEGER)`; `CREATE UNIQUE INDEX IF NOT EXISTS self_serve_moves_open ON self_serve_moves(account, app, host) WHERE outcome = ''`. |
| +- **The request lifecycle** (one request at a time per row): `IssueSelfServeRequest(id, kind, phase string, nonce string, expectSeq uint64, proposed time.Time, now time.Time) (bool, error)` (`WHERE request_consumed <> 0 OR request_issued = 0`: sets `request_issued = now, request_kind, request_phase, request_nonce, request_expect_seq, request_proposed, request_confirmed = 0, request_consumed = 0`); `ConfirmSelfServeRequest(id, nonce string, now time.Time) (bool, error)` (`WHERE request_nonce = ? AND request_consumed = 0 AND request_confirmed = 0`: `request_confirmed = now`; false → a voided or already consumed request: the caller stops; for kind `stage` also `stage_request = request_proposed`, for `precheck` also `precheck_request = request_proposed`); `VoidSelfServeRequest(id, nonce string) error` (`WHERE request_nonce = ? AND request_consumed = 0`: `request_consumed = 1, request_voided = 1`; the phase function decides again); `IssueSelfServeRequest` additionally requires `phase = ? AND outcome = ''` (the issuing phase owns the request); `AdvanceSelfServe(id, from, to, reason string, step, window time.Time, nonce string, now time.Time) (bool, error)`: one UPDATE with the predicate `phase = ? AND (abort_requested_at = 0 OR ? IN ('reverting','done','opened','health','finishing')) AND (? = '' OR request_nonce = ?)` that sets the phase, `phase_started_at`, the deadlines, `expiry_pending = 0, expiry_phase = ''` when `to <> expiry_phase`, and, when a nonce is given, additionally requires `request_nonce = ? AND request_confirmed <> 0 AND request_consumed = 0` and sets `request_consumed = 1` (**consumption and the advance are one statement**), and inserts the phase event in the same transaction (`event_seq = event_seq + 1`); a stale worker holding another nonce, or the same nonce after a void, cannot confirm or consume. |
| - Other store methods: `AdmitSelfServeMove(m SelfServeMove) (admitted bool, open *SelfServeMove, err error)`; `SelfServeMove(id string) (*SelfServeMove, error)`; `SelfServeMoveForHost(acct, app, host string) (open, terminal *SelfServeMove, err error)` (the open row, and the newest terminal row); `OpenSelfServeMoves() ([]SelfServeMove, error)` (`outcome = '' OR release_pending = 1 OR effects_done = 0 OR open_record_pending = 1`, **ordered by `created_at, id`**: the pass steps rows in that order; a row waiting on a busy box stays in `reserving` and does not block rows with disjoint participants); `SetSelfServeEstimate`, `RecordSelfServeHealth(id string, probeAt time.Time, ok bool, errText string) error` (`health_attempts + 1`; `consecutive_ok = CASE ok THEN consecutive_ok + 1 ELSE 0`; `first_ok_at` kept once set), `SetSelfServeFinishGate`, `SetSelfServeWarning`, `SetSelfServeNotifiedSeq(id string, seq uint64) error`, `SetSelfServeExpiry(id, phase string, pending bool) error`, `RequestSelfServeAbort(id string, phases []string, now time.Time) (bool, error)` (`WHERE phase IN (...) AND abort_requested_at = 0`), `SetSelfServeReopened`, `SetSelfServeAbortFinished`, `FinishSelfServeMove(id, outcome, reason, warning string, now time.Time) error` (`release_pending = 1`, `effects_done = 0`), `SetSelfServeEffectsDone(id string) error`, `ClearSelfServeRelease(id string) error`, `SetSelfServeOpenRecordPending(id string, pending bool) error`. |
| - Table `self_serve_events (id TEXT, seq INTEGER, phase TEXT, account TEXT, app TEXT, host TEXT, at INTEGER, reason TEXT, detail TEXT, emitted INTEGER, PRIMARY KEY (id, seq))`: `InsertSelfServeEventTx(tx *sql.Tx, e SelfServeEvent) error` (allocates `seq` from the row's `event_seq` in the same transaction; used by `AdvanceSelfServe` and by every refusal, fact decision, lane probe, health probe and timing); `UnemittedSelfServeEvents(limit int) ([]SelfServeEvent, error)`; `DeliverSelfServeEvent(ctx context.Context, e SelfServeEvent, deliver func(tx *sql.Tx) error) error` (one transaction: `deliver(tx)` inserts the audit row through `auditActivityTx(tx, ...)`, then `emitted = 1`; a failure rolls both back). Table `self_serve_timings (id TEXT, mark TEXT, account TEXT, app TEXT, host TEXT, at INTEGER, bytes INTEGER, PRIMARY KEY (id, mark))`. |
| - Phases: `requested`, `reserving`, `queued`, `reports`, `staging`, `prechecks`, `ready`, `restage`, `barrier`, `stopping`, `sealing`, `landing`, `landed`, `open-uncertain`, `opened`, `health`, `finishing`, `reverting`, `done`. |
| -- **Compatible sets** (Task 7's reconciliation; "pending" = the row has an unconfirmed request for that phase's write): `offline-queued` ↔ {queued, reports, staging(pending)}; `offline-staging` ↔ {staging, prechecks, ready, restage, barrier, stopping(pending)}; `offline-stopping` ↔ {stopping, sealing(pending)}; `offline-sealing` ↔ {sealing}; `offline-sealed` ↔ {sealing, landing(pending)}; `offline-landing` ↔ {landing}; `offline-landed` ↔ {landed, open-uncertain, opened(pending)}; `offline-open` ↔ {opened, health, finishing}; `offline-finishing` ↔ {finishing}; `offline-reverting` ↔ {reverting}; cleared with `LastMove.ID == id` ↔ {done, finishing(gate-2 pending), reverting(finish pending)}; no move and no `LastMove` for the id ↔ {requested, reserving, queued(pending)}. Reconciliation moves the row **forward** to the first phase of the record's set only when the row's phase is not in that set. |
| +- **Compatible sets** (Task 7's reconciliation; "pending" = the row has **no confirmed request** for that phase's write, whether none was issued yet, or one is issued and unconfirmed): `offline-queued` ↔ {queued, reports, staging(pending)}; `offline-staging` ↔ {staging, prechecks, ready, restage, barrier, stopping(pending)}; `offline-stopping` ↔ {stopping, sealing(pending)}; `offline-sealing` ↔ {sealing}; `offline-sealed` ↔ {sealing, landing(pending)}; `offline-landing` ↔ {landing}; `offline-landed` ↔ {landed, open-uncertain, opened(pending)}; `offline-open` ↔ {opened, health, finishing}; `offline-finishing` ↔ {finishing}; `offline-reverting` ↔ {reverting}; cleared with `LastMove.ID == id` ↔ {done, finishing(gate-2 pending), reverting(finish pending)}; no move and no `LastMove` for the id ↔ {requested, reserving, queued(pending)}. Reconciliation moves the row **forward** to the first phase of the record's set only when the row's phase is not in that set. |
| - Lease: `const selfServeLeaseKey = "control/offline-moves/console.json"`; `type consoleLease struct { Console string; Until time.Time }`; `acquireSelfServeLease(ctx, now) (*leaseToken, bool, error)`; `leaseToken.renew(ctx, now)` (every 20 s during a tick); `leaseToken.valid(now) bool`; `leaseToken.deadline() time.Time`; `selfServeApply` derives `ctx, cancel := context.WithDeadline(ctx, t.deadline())` for the record write and re-checks `t.valid(now)` before each retry through `provision.WithWriteGuard(ctx, func() error)`, a hook `WriteRecord` calls immediately before every `Put`. A cancelled context does **not** prove the write cannot commit; that is why every write is a persisted request (Task 7). |
| - `StartSelfServeMovesPass`: runs while `selfServeMode()`; a failing startup read of rows, events or `active.json` → the pass runs (fail closed) and retries the reads each tick, logging; `selfServeTick`: renew the lease, the reservation scan, `drainSelfServeEvents`, then one step per open row in order. |
| -- [ ] **Step 1: Write the failing tests**: `TestSelfServeStoreRoundTrip`; `TestAdmitIsAtomic`; `TestRequestLifecycle` (no second issue while unconsumed; confirm only with the nonce; void; advance with the wrong nonce → false; advance with the right nonce consumes in the same statement); `TestAdvanceIsPredicatedOnPhaseAndNoAbort`; `TestAdvanceIntoOpenedIgnoresAnObsoleteAbortFlag`; `TestAdvanceClearsAForeignExpiry`; `TestEventAndAdvanceAreOneTransaction`; `TestEventDeliveryIsTransactional`; `TestEventsHaveAPerRowSequence` (repeated prechecks produce distinct events); `TestOpenRowsOrderedAndIncludePending`; `TestPassStartsAndRetriesOnReadFailure`; `TestConsoleLeaseExcludesASecondConsole`; `TestLeaseBoundsEveryWriteAttempt`; `TestPagingArmedForSelfServe`; `TestClaimStoreOpensWithEveryNewTable`. |
| +- [ ] **Step 1: Write the failing tests**: `TestSelfServeStoreRoundTrip`; `TestAdmitIsAtomic`; `TestRequestLifecycle` (no second issue while unconsumed; issue only from the owning phase; confirm only with the nonce and only once; a voided request cannot be confirmed or consumed even with its own nonce; advance with the wrong nonce → false; advance with the right confirmed nonce consumes in the same statement); `TestAdvanceIsPredicatedOnPhaseAndNoAbort`; `TestAdvanceIntoOpenedIgnoresAnObsoleteAbortFlag`; `TestAdvanceClearsAForeignExpiry`; `TestEventAndAdvanceAreOneTransaction`; `TestEventDeliveryIsTransactional`; `TestEventsHaveAPerRowSequence` (repeated prechecks produce distinct events); `TestOpenRowsOrderedAndIncludePending`; `TestPassStartsAndRetriesOnReadFailure`; `TestConsoleLeaseExcludesASecondConsole`; `TestLeaseBoundsEveryWriteAttempt`; `TestPagingArmedForSelfServe`; `TestClaimStoreOpensWithEveryNewTable`. |
| - [ ] **Step 2: Run to verify they fail; Step 3: implement; Step 4: run** `go test ./internal/console -run 'SelfServe|ConsoleLease|LeaseBounds|RequestLifecycle|EventAnd|EventDelivery|EventsHave|PagingArmed|ClaimStoreOpens' -count=1`. |
| - [ ] **Step 5: Commit** `console: the self-serve journal with persisted requests and nonce-predicated advances, sequenced transactional events, the lease and the pass skeleton`. |
| @@ -124,12 +125,12 @@ Builds on plan 1 (`2026-10-22-self-serve-moves-1-lifetimes-and-order.md`, rev 11 |
| **Interfaces:** |
| - `func recordEligibility(rec *provision.Record, c moveCopy) *offlineRefusal`: `moveGate`'s checks in order (`Kind`, `Addr`/`Addrs`, fleet placement, `Pool`, `!HibernateOrDefault()`; 400s). |
| - `func (s *Server) offlineRecordChecks(ctx, acct, app, host string, rec *provision.Record) *offlineRefusal` (the pre-body half: the seed block, `Role`, `Replicas`, `Steering`/`Unit`, `offlineClaimEvidence`); `func (s *Server) offlineRegionChecks(ctx, acct, app, host string, rec *provision.Record, region string, stage, requireIDs bool, returnOf string) (offlineSelection, *offlineRefusal)` (the post-body half: the region, the manifest, the fleet, the home rule, `offlineSelect`, the stage rules, participant pinning, the return's evidence); `type offlineSelection struct { From, To string; Fleet offlineFleet; Source []provision.Participant; Target provision.Participant; Stage provision.StageTarget; ReturnOf string; Sentinel bool }` (`Sentinel` when any participant's `InstanceID` is empty). |
| -- **Participant pinning.** `BoxID` and `Label` from the edges file. With `requireIDs: false` (**every operator door**): `InstanceID` from `status/boxes/<label>.json` when the report exists, is within 10 minutes and carries `Metrics.BoxID == BoxID`; otherwise `InstanceID = ""`, `Verified = false`, and the move proceeds as today (`offlineSelect` already admits a box with nil metrics); no identity record is read. With `requireIDs: true` (**the self-serve door and the job**): the status report must be present and fresh with a non-empty `InstanceID` (else "the box <label> has not reported recently"); `Verified` when `boxid.Load(ctx, s, label)` finds a record and the verifier (constructed as `ClaimRootHTTP` constructs it: the deployment account and the manifest's AWS region for the box's region) `Enroll`s it to evidence naming that `InstanceID`; a record that exists but does not verify → refuse "box <label> has an identity record that does not verify"; no record → pinned with `Verified: false` (allowed; the status report's `InstanceID` is the identity). |
| +- **Participant pinning.** `BoxID` and `Label` from the edges file. With `requireIDs: false` (**every operator door**): `InstanceID` from `status/boxes/<label>.json` when the report exists, is within 10 minutes and carries `Metrics.BoxID == BoxID`; otherwise `InstanceID = ""`, `Verified = false`, and the move proceeds as today (`offlineSelect` already admits a box with nil metrics); no identity record is read. With `requireIDs: true` (**the self-serve door and the job**): the status report must be present and fresh with a non-empty `InstanceID` (else "the box <label> has not reported recently"); `Verified` when `boxid.Load(ctx, s, BoxID)` (the record key is `boxid.Key(BoxID)`, never the label) finds a record and the verifier (constructed as `ClaimRootHTTP` constructs it) `Enroll`s it with `boxid.Expect{Box: BoxID, AWSRegion: the manifest's AWS region for the box's region, Account: the deployment account}` to evidence naming that `InstanceID`; `Verified` means exactly what `Enroll` proves (a signed document of this deployment naming the instance, consistent with the roster); it does not claim reference ownership beyond the roster rule, and the plan relies on it only to detect a replaced machine; a record that exists but does not verify → refuse "box <label> has an identity record that does not verify"; no record → pinned with `Verified: false` (allowed; the status report's `InstanceID` is the identity). |
| - **The self-serve bodies.** `POST .../move {"region", "display_name"?, "requested_by"?, "start_at"?}`: `selfServeGate`; `s.selfServeAdmits()` else 409 "self-serve moves are not enabled on this console"; `obs := s.observeRecord(...)` (absent or stale → 404; corrupt or unreadable → 502 "could not read the instance"); `recordEligibility`; `rec.ActiveRun.Open()` → the existing run 409; `rec.Move != nil` → 409 `offlineConflict`'s sentence; an open row → the same 409; the body; `start_at` → 400 "scheduled starts are not available yet"; `display_name` over 80 runes or with a control character → 400 "display_name must be at most 80 characters of plain text"; `offlineRecordChecks`; `offlineRegionChecks(..., stage: true, requireIDs: true, "")`; one source box → else 409 "<region> has <n> boxes. a self-serve move needs exactly one box in each region"; the source's status facts (Task 6: `Hosts[host]` with `Incarnation == rec.Incarnation`, `Known`, `CollectedAt` and `ReplicaCollectedAt` within 10 minutes and not more than 60 s in the future, else 409 "the source box has not reported recently. try again in a minute"; `OverCap` → 409 "this account's files are too large to move this way"; `Truncated` → 409 "could not measure the account's files. try again in a minute"); the lane (`Lanes` names the target bucket) → else 409 "no staging lane is configured from <from> to <to>"; the lifetime: `rec.Incarnation == 0` → `withHostname`: `migrateHostLifetimeLocked(ctx, acct.Sqid, app, host, obs.ETag)` (**bound**); `ErrCASConflict` or settling → 409 "another change to this instance is still settling. try again in a minute"; committed → `rec` is the returned record and `recordEligibility`, the run and move checks run again on it; `AdmitSelfServeMove({..., Incarnation: rec.Incarnation, Participants, Phase: requested, RequestedBy: "token:" + p.TokenID + " " + requested_by, DisplayName, EstimateSeconds, EstimateTerms, WindowDeadline: now + 24 h})`; not admitted → 409 the in-flight sentence; `bumpSerial`; 202 with the view. `GET .../move?estimate=1®ion=uk`: `moveGate`; the list through the lane without writing or migrating; 200 `{"estimate_seconds", "files_bytes", "replica_bytes"}` or the refusal. `POST .../move/cut {"id"}`: `selfServeGate`; the row; `row.ID != body.id` or none → 409 "this is not the move you confirmed"; `AdvanceSelfServe(id, ready, restage, ..., "")` → 202; false → re-read: a cut phase → 202; abort requested or `aborted` → 409 "the move was cancelled"; else 409 "the move is not ready to cut yet. it is <phase>". `DELETE .../move {"id"}`: `selfServeGate`; the row and id; `RequestSelfServeAbort(id, [requested, reserving, queued, reports, staging, prechecks, ready], now)` → 202; false → re-read: abort already requested → 202; a cut phase → 409 "the move was confirmed and cannot be cancelled. it completes or returns on its own"; terminal → 409 "no move is in flight". |
| -- **The view** (`selfServeViewOf(rec, open, terminal, now)`, rendered whenever claims-off and Phase 1 off): (1) the record carries an offline move with `SelfServe != nil` **and** `open != nil && open.ID == rec.Move.SelfServe.ID && open.Incarnation == rec.Incarnation` → the open row's view (`preparing` through `ready`, `cutting` from `restage`); (2) the record carries any other move → today's legacy view, no `last`; (3) no record move and an open row with `open.Incarnation == rec.Incarnation` → `preparing`; (4) no record move and a terminal row with `terminal.Incarnation == rec.Incarnation` and `now - UpdatedAt < 7 days` → `none` with `last`; (5) otherwise today's view. `type selfServeMoveView struct { State string \`json:"state"\`; Target string \`json:"target,omitempty"\`; UpdatedAt int64 \`json:"updated_at,omitempty"\`; Kind string \`json:"kind"\`; ID string \`json:"id,omitempty"\`; EstimateSeconds int \`json:"estimate_seconds,omitempty"\`; OfflineAt int64 \`json:"offline_at,omitempty"\`; Reason string \`json:"reason,omitempty"\`; Abort *selfServeAbortView \`json:"abort,omitempty"\`; Last *selfServeLast \`json:"last,omitempty"\` }`; `type selfServeAbortView struct { RequestedAt int64 \`json:"requested_at"\`; SourceReopenedAt int64 \`json:"source_reopened_at,omitempty"\`; FinishedAt int64 \`json:"finished_at,omitempty"\` }`; `type selfServeLast struct { ID string \`json:"id"\`; Outcome string \`json:"outcome"\`; Reason string \`json:"reason,omitempty"\`; Warning string \`json:"warning,omitempty"\`; From string \`json:"from"\`; To string \`json:"to"\`; At int64 \`json:"at"\` }` (unix seconds; regions through `homeRegionOut`). Phase 1's `moveView` is untouched. |
| +- **The view** (`selfServeViewOf(rec, open, terminal, now)`, rendered whenever claims-off and Phase 1 off): (1) the record carries an offline move with `SelfServe != nil` **and** `open != nil && open.ID == rec.Move.ID && open.ID == rec.Move.SelfServe.ID && open.Incarnation == rec.Incarnation` → the open row's view (`preparing` through `ready`, `cutting` from `restage`); (2) the record carries any other move → today's legacy view, no `last`; (3) no record move and an open row with `open.Incarnation == rec.Incarnation` → `preparing`; (4) no record move and a terminal row with `terminal.Incarnation == rec.Incarnation` and `now - UpdatedAt < 7 days` → `none` with `last`; (5) otherwise today's view. `type selfServeMoveView struct { State string \`json:"state"\`; Target string \`json:"target,omitempty"\`; UpdatedAt int64 \`json:"updated_at,omitempty"\`; Kind string \`json:"kind"\`; ID string \`json:"id,omitempty"\`; EstimateSeconds int \`json:"estimate_seconds,omitempty"\`; OfflineAt int64 \`json:"offline_at,omitempty"\`; Reason string \`json:"reason,omitempty"\`; Abort *selfServeAbortView \`json:"abort,omitempty"\`; Last *selfServeLast \`json:"last,omitempty"\` }`; `type selfServeAbortView struct { RequestedAt int64 \`json:"requested_at"\`; SourceReopenedAt int64 \`json:"source_reopened_at,omitempty"\`; FinishedAt int64 \`json:"finished_at,omitempty"\` }`; `type selfServeLast struct { ID string \`json:"id"\`; Outcome string \`json:"outcome"\`; Reason string \`json:"reason,omitempty"\`; Warning string \`json:"warning,omitempty"\`; From string \`json:"from"\`; To string \`json:"to"\`; At int64 \`json:"at"\` }` (unix seconds; regions through `homeRegionOut`). Phase 1's `moveView` is untouched. |
| - Copy rows: "only an account owner can mint a move-capable instance token"; "self-serve moves are not enabled on this console"; "could not read the instance"; "could not read the move"; "scheduled starts are not available yet"; "display_name must be at most 80 characters of plain text"; "<region> has <n> boxes. a self-serve move needs exactly one box in each region"; "the source box has not reported recently. try again in a minute"; "the box <label> has not reported recently"; "box <label> has an identity record that does not verify"; "this account's files are too large to move this way"; "could not measure the account's files. try again in a minute"; "no staging lane is configured from <from> to <to>"; "another change to this instance is still settling. try again in a minute"; "this is not the move you confirmed"; "the move was cancelled"; "the move is not ready to cut yet. it is <phase>"; "the move was confirmed and cannot be cancelled. it completes or returns on its own"; "no move is in flight"; "the reservation does not match this move's participants". |
| -- [ ] **Step 1: Write the failing tests**: `TestOperatorDoorAnswersAreUnchanged` (snapshots from main, incl. operator moves with nil metrics and no identity record **succeeding** as today); `TestSwitchOffIsByteIdentical`; `TestRecordEligibilityIsMoveGatesChecks`; `TestStartValidationSplitPreservesOperatorOrdering`; `TestParticipantPinning` (operator: nil metrics → pinned id-less, sentinel; self-serve: fresh report required; verified with a record; unverified without; a failing record refuses); `TestPrepareRefusesBeforeWriting` (one subtest per refusal); `TestPrepareMigratesALegacyRecordBoundThenAdmits`; `TestPrepareWritesTheRowFirst`; `TestMutationsNameTheMoveAndRace` (barriers both orders; a stale snapshot; a second cut → 202; a stale nonce cannot confirm or consume); `TestViewPrecedence` (the five cases; an operator move over an open row; a recreation; a terminal row after switch-off, a restart and a later poll within 7 days → `last` rendered; older → not); `TestEstimateDoorWritesNothing`; `TestStatusStaysOpenToPlainInstanceTokens`. |
| +- [ ] **Step 1: Write the failing tests**: `TestOperatorDoorAnswersAreUnchanged` (snapshots from main, incl. operator moves with nil metrics and no identity record **succeeding** as today); `TestSwitchOffIsByteIdentical`; `TestRecordEligibilityIsMoveGatesChecks`; `TestStartValidationSplitPreservesOperatorOrdering`; `TestParticipantPinning` (operator: nil metrics → pinned id-less, sentinel; self-serve: fresh report required; verified with a record under `boxid.Key(BoxID)` when the label differs from the id; unverified without; a failing record refuses); `TestPrepareRefusesBeforeWriting` (one subtest per refusal); `TestPrepareMigratesALegacyRecordBoundThenAdmits`; `TestPrepareWritesTheRowFirst`; `TestMutationsNameTheMoveAndRace` (barriers both orders; a stale snapshot; a second cut → 202; a stale nonce cannot confirm or consume); `TestViewPrecedence` (the five cases; an operator move over an open row; a recreation; a terminal row after switch-off, a restart and a later poll within 7 days → `last` rendered; older → not); `TestEstimateDoorWritesNothing`; `TestStatusStaysOpenToPlainInstanceTokens`. |
| - [ ] **Step 2: Run to verify they fail; Step 3: implement; Step 4: run** `go test ./internal/console -run 'SelfServe|SwitchOff|OperatorDoorAnswers|RecordEligibility|StartValidationSplit|ParticipantPinning|Prepare|MutationsName|ViewPrecedence|EstimateDoor' -count=1`. |
| - [ ] **Step 5: Commit** `console: record eligibility and the split start validation shared by the operator doors and the job; participants pinned by instance; the self-serve door and a mode-independent view`. |
| @@ -140,16 +141,16 @@ Builds on plan 1 (`2026-10-22-self-serve-moves-1-lifetimes-and-order.md`, rev 11 |
| - Test: `internal/moves/offline_test.go`, `internal/moves/participants_test.go`, `internal/lease/observed_test.go`, `internal/host/movesoffline_reports_test.go`, `internal/status/snapshot_test.go` |
| **Interfaces:** |
| -- `OfflineReport` gains `BoxID string \`json:"box_id,omitempty"\``, `InstanceID string \`json:"instance_id,omitempty"\``, `Request time.Time \`json:"request,omitzero"\``, `Lane *LaneReport \`json:"lane,omitempty"\``, `Ready *TargetReadyReport \`json:"ready,omitempty"\``, `Precheck *SourcePrecheck \`json:"precheck,omitempty"\``, `StagedKnown bool \`json:"staged_known,omitempty"\``, `Staged map[string]StagedObject \`json:"staged,omitempty"\`` (the book), `StagedBucket string \`json:"staged_bucket,omitempty"\``, `StagedRegion string \`json:"staged_region,omitempty"\``, `StagedRequest time.Time \`json:"staged_request,omitzero"\``, `RoundCopiedBytes int64 \`json:"round_copied_bytes,omitempty"\``, `RoundDeferredBytes int64 \`json:"round_deferred_bytes,omitempty"\``, `RoundDeferredObjects int \`json:"round_deferred_objects,omitempty"\``, `Rounds int \`json:"rounds,omitempty"\``, `TombstoneLifted bool \`json:"tombstone_lifted,omitempty"\``, `TombstoneLiftedAt time.Time \`json:"tombstone_lifted_at,omitzero"\``, `FinalCopiedBytes int64 \`json:"final_copied_bytes,omitempty"\``; roles gain `OfflineRoleLane = "lane"`; phases gain `OfflinePhaseLane = "lane"`, `OfflinePhasePrecheck = "precheck"`, `OfflinePhaseReady = "ready"`, `OfflinePhaseReverting = "reverting"`. |
| +- `OfflineReport` gains `BoxID string \`json:"box_id,omitempty"\``, `InstanceID string \`json:"instance_id,omitempty"\``, `Request time.Time \`json:"request,omitzero"\``, `Lane *LaneReport \`json:"lane,omitempty"\``, `Ready *TargetReadyReport \`json:"ready,omitempty"\``, `Precheck *SourcePrecheck \`json:"precheck,omitempty"\``, `StagedKnown bool \`json:"staged_known,omitempty"\``, `Staged map[string]StagedObject \`json:"staged,omitempty"\`` (the book), `StagedBucket string \`json:"staged_bucket,omitempty"\``, `StagedRegion string \`json:"staged_region,omitempty"\``, `StagedRequest time.Time \`json:"staged_request,omitzero"\`` (**always equal to `StageRequest`** on a stager report; the stager sets both; today's duplicate-round suppression keeps reading `StageRequest`; a report with only one of them fails contextual validation), `Layout string \`json:"layout,omitempty"\`` (every source's `stopped` and `precheck` report: `flat` or `nested`), `RoundCopiedBytes int64 \`json:"round_copied_bytes,omitempty"\``, `RoundDeferredBytes int64 \`json:"round_deferred_bytes,omitempty"\``, `RoundDeferredObjects int \`json:"round_deferred_objects,omitempty"\``, `Rounds int \`json:"rounds,omitempty"\``, `TombstoneLifted bool \`json:"tombstone_lifted,omitempty"\``, `TombstoneLiftedAt time.Time \`json:"tombstone_lifted_at,omitzero"\``, `FinalCopiedBytes int64 \`json:"final_copied_bytes,omitempty"\``; roles gain `OfflineRoleLane = "lane"`; phases gain `OfflinePhaseLane = "lane"`, `OfflinePhasePrecheck = "precheck"`, `OfflinePhaseReady = "ready"`, `OfflinePhaseReverting = "reverting"`. |
| - `type StagedObject struct { Size int64 \`json:"size"\`; ETag string \`json:"etag"\`; VersionID string \`json:"version_id,omitempty"\`; Deferred bool \`json:"deferred,omitempty"\` }` (`Size` keeps today's `int64` for the seal consumers). `type PrefixObject struct { ETag string \`json:"etag"\`; Size int64 \`json:"size"\`; VersionID string \`json:"version_id,omitempty"\`; LastModified time.Time \`json:"last_modified"\` }`; `type PrefixFacts struct { Known bool \`json:"known"\`; Objects map[string]PrefixObject \`json:"objects,omitempty"\` }` (keys **relative to the prefix**; current versions only; delete markers excluded; `lock.json` excluded). |
| - `type Fact struct { Known bool \`json:"known"\`; OK bool \`json:"ok"\`; Detail string \`json:"detail,omitempty"\` }`. `type LaneReport struct { Bucket string \`json:"bucket"\`; Region string \`json:"region"\`; Key string \`json:"key"\`; OK bool \`json:"ok"\`; Error string \`json:"error,omitempty"\` }` (the existing Phase 1 `laneProbe`: put-if-absent, list, delete-version under `moves/_lane/<move id>`). `type TargetReadyReport struct { RouteAbsent Fact \`json:"route_absent"\`; TimersAbsent Fact \`json:"timers_absent"\`; LiveLink Fact \`json:"live_link"\`; ChannelBuild string \`json:"channel_build"\`; ProgramsFile Fact \`json:"programs_file"\`; DataPaths map[string]Fact \`json:"data_paths"\`; ParkDir Fact \`json:"park_dir"\`; DiskFree Fact \`json:"disk_free"\`; MovesPrefix PrefixFacts \`json:"moves_prefix"\`; InstancesPrefix PrefixFacts \`json:"instances_prefix"\`; Lock Fact \`json:"lock"\`; Capabilities []string \`json:"capabilities"\`; SecretsKeyFP string \`json:"secrets_key_fp"\`; StagePathArmed Fact \`json:"stage_path_armed"\`; Leftovers *ReturnLeftovers \`json:"leftovers,omitempty"\` }`. `type StrayEntry struct { Path string \`json:"path"\`; Kind string \`json:"kind"\` (dir, file, symlink, other); Bytes int64 \`json:"bytes"\`; Entries int \`json:"entries"\`; Known bool \`json:"known"\`; Detail string \`json:"detail,omitempty"\` }`; `type StraysFact struct { Known bool \`json:"known"\`; Entries []StrayEntry \`json:"entries,omitempty"\`; Detail string \`json:"detail,omitempty"\` }`; `type SourcePrecheck struct { FilesBytes int64 \`json:"files_bytes"\`; FilesEntries int \`json:"files_entries"\`; Layout string \`json:"layout"\` (flat, nested); ArchiveEligible Fact \`json:"archive_eligible"\`; Strays StraysFact \`json:"strays"\`; Build string \`json:"build"\`; StagePathArmed Fact \`json:"stage_path_armed"\`; StraysRemoved []string \`json:"strays_removed,omitempty"\`; StraysLeft []StrayEntry \`json:"strays_left,omitempty"\` }`: a non-directory stray is a precheck refusal ("<host> has a stray entry beside its files directory: <path>") so the stop never happens; an empty directory is removed at start after quiescence (`ReadDir` → zero entries → `os.Remove`; errors → `StraysLeft` with `Detail`). The dry run is `archive.Build(files, envPath, io.Discard)` over `offlineFilesOf(host, rt)`'s selection. |
| - `type status.HostFacts struct { CollectedAt int64 \`json:"collected_at"\`; Known bool \`json:"known"\`; Err string \`json:"err,omitempty"\`; Incarnation uint64 \`json:"incarnation"\`; ReplicaBytes int64 \`json:"replica_bytes"\`; ReplicaObjects int \`json:"replica_objects"\`; ReplicaCollectedAt int64 \`json:"replica_collected_at"\`; FilesBytes int64 \`json:"files_bytes"\`; FilesEntries int \`json:"files_entries"\`; OverCap bool \`json:"over_cap,omitempty"\`; Truncated bool \`json:"truncated,omitempty"\`; Strays []string \`json:"strays,omitempty"\` }`; `measureFiles(ctx, files, envPath string, budget time.Duration, capBytes int64) (bytes int64, entries int, overCap, truncated bool, err error)` (a bounded walker applying the archive's counting rules without stopping at the cap: records `OverCap` and continues until the budget (60 s) or 2 × the cap, where it records `Truncated`); the replica listing cached per tick with its own `ReplicaCollectedAt`; `Known` only when every part succeeded. `BoxMetrics.Lanes []string` from `CARLOS_MOVE_LANES`; `BoxMetrics.BoxID string` (the edges writer's value); `BoxMetrics.InstanceID` exists. |
| - **Participants:** `func ParticipantRole(m *provision.MoveState, boxID, instanceID, label, placement, home string) (source, target, sealer, stager bool)`: with `Participants`: a participant matches when `p.BoxID == boxID && (p.InstanceID == "" || p.InstanceID == instanceID)`; a box whose `BoxID` matches a participant with a non-empty different `InstanceID` has **no role** (it logs "box <label> is not the pinned instance for move <id>" once per move and does nothing); without `Participants`: today's region and label rules. `func ParticipantStillPresent(p provision.Participant, m *status.BoxMetrics) bool` = `p.InstanceID == "" || (m != nil && m.BoxID == p.BoxID && m.InstanceID == p.InstanceID)` (the console's gate check, on the label's status report within 10 minutes; a missing report with a non-empty pinned id → "box <label> was replaced" after 10 minutes, "box <label> has not reported recently" before). |
| -- **Validation:** `ValidateOfflineReport(r)` is **unchanged** (legacy structure). `func ValidateOfflineReportFor(r OfflineReport, m *provision.MoveState, now time.Time) error` adds the contextual rules: a move with `Participants` → the report's `BoxID` and `InstanceID` equal the pinned participant for its role (`InstanceID` compared only when the pinned one is non-empty); a move with `SelfServe` → role `lane` ↔ phase `lane` with `Lane`; `source`+`precheck` with `Precheck` and `Request`; `target`+`ready` with `Ready` and `Request`; `stager`+`staged` with `StagedKnown`, `StagedBucket/Region` equal to the record's, `StagedRequest`; `sealer`+`reverting` with `TombstoneLifted`; a `failed` report needs only the identity rule (its diagnostic is kept). **Freshness table:** request-bound reports (`lane` by move id; `precheck`, `ready` by `Request`; `staged` by `StagedRequest`) have **no age limit** and at most 60 s of future skew; the phase reports today's gates read (`stopped`, `sealed`, `landed`, `opened`, `reverted`, `reverting`, `failed`) keep `offlineFresh` (2 min, 1 min skew) where today applies it; the seal is immutable and gets only the identity rule; participant identity from status is 10 min. `func (s *Server) readOfflineEvidence(ctx context.Context, host, id string, m *provision.MoveState) (offlineEvidence, error)` applies both validators and drops a report that fails the contextual rule, logging it; every caller (`offlineStepFor`, the seal, flip, open and finish checks, `offlineView`, the job) passes the move it holds (`nil` for a move it has not read: legacy validation only). |
| -- **The stage book (item 6).** `type lease.CopyEvent struct { Rel string; SrcETag string; SrcSize int64; DstETag string; DstVersionID string; Unchanged, Deferred, Deleted bool }`; `func MirrorPrefixKeepObserved(ctx, m MirrorStore, srcBucket, srcPrefix, dstBucket, dstPrefix string, rc Receipts, keep func(string) bool, observe func(CopyEvent)) (MirrorRound, error)`: emits one event per source key the round decided (a copy with the ETag and version the copy returned; `Unchanged` with the listed destination ETag when the round skipped it as equal; `Deferred` when the round deferred it) and `Deleted` per destination key removed; `MirrorPrefixKeep` = `...Observed(..., nil)`, so ordinary mirroring and the receipt ledger are unchanged. The stager keeps `stageBook{MoveID string; Objects map[string]moves.StagedObject}` across rounds: seeded at the first round of a move from its own last `staged` report for that move id when one exists (else empty); every event upserts (`Deleted` removes; `Deferred` marks the existing entry deferred without changing its ETag, or inserts `{Deferred: true}` for a key never copied); the `staged` report carries the whole book with `StagedKnown: true`, `StagedBucket/Region` from the record, `StagedRequest = m.StagedAt`. **Staged ownership** (the console's prechecks and barrier): `StagedKnown`, `StagedBucket/Region` equal to the record's, `StagedRequest == row.StageRequest`; the target's `MovesPrefix.Known`; the key sets are equal (both relative to their prefixes; entries with `Deferred` and no ETag must be **absent** from the target); for every key the target's `ETag` equals the book's, sizes are equal, and version ids are equal when both are present; the barrier additionally requires `RoundDeferredObjects == 0`; a mismatch → "the staging prefix for <host> does not match what this move staged". |
| +- **Validation:** `ValidateOfflineReport(r)` is **unchanged** (legacy structure). `func ValidateOfflineReportFor(r OfflineReport, m *provision.MoveState, now time.Time) error` adds the contextual rules: a move with `Participants` → the report's `BoxID` and `InstanceID` equal the pinned participant for its role (`InstanceID` compared only when the pinned one is non-empty); a move with `SelfServe` → role `lane` ↔ phase `lane` with `Lane`; `source`+`precheck` with `Precheck` and `Request`; `target`+`ready` with `Ready` and `Request`; `stager`+`staged` with `StagedKnown`, `StagedBucket/Region` equal to the record's, `StagedRequest`; `sealer`+`reverting` with `TombstoneLifted`; a `failed` report needs only the identity rule (its diagnostic is kept: the console reads `moves.FailedDetail(r)` = `StageError` when set, else the first failed item's text, else "the box reported a failure without detail"). **Freshness table:** request-bound reports (`lane` by move id; `precheck`, `ready` by `Request`; `staged` by `StagedRequest`) have **no age limit** and at most 60 s of future skew; the phase reports today's gates read (`stopped`, `sealed`, `landed`, `opened`, `reverted`, `reverting`, `failed`) keep `offlineFresh` (2 min, 1 min skew) where today applies it; the seal is immutable and gets only the identity rule; participant identity from status is 10 min. `func (s *Server) readOfflineEvidence(ctx context.Context, host, id string, m *provision.MoveState) (offlineEvidence, error)` applies both validators and drops a report that fails the contextual rule, logging it; every caller (`offlineStepFor`, the seal, flip, open and finish checks, `offlineView`, the job) passes the move it holds (`nil` for a move it has not read: legacy validation only). |
| +- **The stage book (item 6).** `type lease.CopyEvent struct { Rel string; SrcETag string; SrcSize int64; DstETag string; DstVersionID string; Unchanged, Deferred, Deleted bool }`; `func MirrorPrefixKeepObserved(ctx, m MirrorStore, srcBucket, srcPrefix, dstBucket, dstPrefix string, rc Receipts, keep func(string) bool, observe func(CopyEvent)) (MirrorRound, error)`: emits one event per source key the round decided (a copy with the ETag and version the copy returned; `Unchanged` with the listed destination ETag when the round skipped it because the ETags were equal **or because a receipt vouched for the multipart copy**, the receipt's `DstETag`/`DstVersionID` in the latter case; `Deferred` when the round deferred it) and `Deleted` per destination key removed, and finally `Complete: true` once with the round's full source and destination key sets when the round finished without error; `MirrorPrefixKeep` = `...Observed(..., nil)`, so ordinary mirroring and the receipt ledger are unchanged. The stager keeps `stageBook{MoveID string; Objects map[string]moves.StagedObject}` across rounds: seeded at the first round of a move from its own last `staged` report for that move id when one exists (else empty); every event upserts (`Deleted` removes; `Deferred` marks the existing entry deferred without changing its ETag, or inserts `{Deferred: true}` for a key never copied); on `Complete` the book is **reconciled**: every entry whose key is in neither the source nor the destination set is dropped (a deletion a crashed round completed but never reported), and a deferred-never-copied entry whose key left the source is dropped; a round that fails before `Complete` leaves the book as the events left it (the next complete round reconciles); the `staged` report carries the whole book with `StagedKnown: true`, `StagedBucket/Region` from the record, `StagedRequest = m.StagedAt`. **Staged ownership** (the console's prechecks and barrier): `StagedKnown`, `StagedBucket/Region` equal to the record's, `StagedRequest == row.StageRequest`; the target's `MovesPrefix.Known`; the key sets are equal (both relative to their prefixes; entries with `Deferred` and no ETag must be **absent** from the target); for every key the target's `ETag` equals the book's, sizes are equal, and version ids are equal when both are present; the barrier additionally requires `RoundDeferredObjects == 0`; a mismatch → "the staging prefix for <host> does not match what this move staged". |
| - **Reports to write:** the source participant in `offline-queued`: the lane probe once per move → the `lane` report; `offlineSourcePrecheck` for the current `PrecheckAt` → `precheck` with `Request = PrecheckAt`; the target participant: `offlineTargetReady` → `ready` with `Request = PrecheckAt`; in `offline-staging` both re-report when `PrecheckAt` is newer than their last `Request`; the stager's `staged` after each round; the sealer's `reverting` after `LiftTombstone` succeeds and before the prefix cleanup; the final `reverted` keeps both lift fields; the normal seal's final-copy bytes in `FinalCopiedBytes`. |
| -- [ ] **Step 1: Write the failing tests**: `TestParticipantRoleByBoxAndInstance` (a matching pair; an id-less participant matches any instance of the box; a replaced box has no role); `TestParticipantStillPresent`; `TestLegacyReportsStillValidate`; `TestContextualValidationForSelfServeMoves` (the matrix: every role/phase/payload; identity mismatch; a failed report kept with only identity checked); `TestFreshnessTable`; `TestEmptyStagedInventoryRoundTrips`; `TestObservedMirrorEmitsEveryDecision` (copy, unchanged same-ETag, deferred, deleted; the receipt ledger unchanged; `MirrorPrefixKeep` behaviour unchanged against the production `NewS3Mirror` wrapper through `stageDeleteGate`); `TestStageBookAcrossRounds` (seeded from the last report; same-ETag copies present; a deferred then copied key; a deleted key gone; an overwrite between the copy and the report → the copy's ETag; a crash before the report → the next round re-copies and the book converges); `TestStagedOwnershipComparison` (equal; missing; extra; changed ETag; changed size; version mismatch; deferred-and-never-copied present → refuse; wrong request; wrong destination; unknown prefix); `TestLaneReportUsesTheExistingProbe`; `TestTargetReadyReportsEveryFactWithKnown`; `TestSourcePrecheckFacts` (incl. `Layout`); `TestStrayFileRefusesBeforeStop`; `TestEmptyStrayDirsRemovedAtStartOnly`; `TestSealerPublishesTombstoneLiftedBeforeCleanup`; `TestQueuedRoundRunsOnlyLanePrecheckAndReady`; `TestParticipantsReReportOnANewPrecheckStamp`; `TestStatusHostsBlockThroughTheRealPath` (`CollectBox → WriteBox → LoadBoxesComplete → BuildFleet → PublishFleet`: `Hosts`, `Lanes`, `BoxID`, `InstanceID` in the private report, absent from `fleet.json`); `TestHostFactsCarryTheLifetimeAndAges`; `TestVersionedArmedForOffline`. |
| +- [ ] **Step 1: Write the failing tests**: `TestParticipantRoleByBoxAndInstance` (a matching pair; an id-less participant matches any instance of the box; a replaced box has no role); `TestParticipantStillPresent`; `TestLegacyReportsStillValidate`; `TestContextualValidationForSelfServeMoves` (the matrix: every role/phase/payload; identity mismatch; a failed report kept with only identity checked); `TestFreshnessTable`; `TestEmptyStagedInventoryRoundTrips`; `TestObservedMirrorEmitsEveryDecision` (copy, unchanged same-ETag, deferred, deleted; the receipt ledger unchanged; `MirrorPrefixKeep` behaviour unchanged against the production `NewS3Mirror` wrapper through `stageDeleteGate`); `TestStageBookAcrossRounds` (through `NewS3Mirror`'s wrapper and `stageDeleteGate` against a fake store: seeded from the last report; same-ETag copies present; a receipt-backed multipart skip present with the receipt's ETag; a deferred then copied key; a deleted key gone; **a key deleted by a round that crashed before its report is dropped by the next complete round**; a deferred-never-copied key whose source vanished is dropped; an overwrite between the copy and the report → the copy's ETag; a crash before the report → the next round re-copies and the book converges); `TestStagedOwnershipComparison` (equal; missing; extra; changed ETag; changed size; version mismatch; deferred-and-never-copied present → refuse; wrong request; wrong destination; unknown prefix); `TestLaneReportUsesTheExistingProbe`; `TestTargetReadyReportsEveryFactWithKnown`; `TestSourcePrecheckFacts` (incl. `Layout`); `TestStrayFileRefusesBeforeStop`; `TestEmptyStrayDirsRemovedAtStartOnly`; `TestSealerPublishesTombstoneLiftedBeforeCleanup`; `TestQueuedRoundRunsOnlyLanePrecheckAndReady`; `TestParticipantsReReportOnANewPrecheckStamp`; `TestStatusHostsBlockThroughTheRealPath` (`CollectBox → WriteBox → LoadBoxesComplete → BuildFleet → PublishFleet`: `Hosts`, `Lanes`, `BoxID`, `InstanceID` in the private report, absent from `fleet.json`); `TestHostFactsCarryTheLifetimeAndAges`; `TestVersionedArmedForOffline`. |
| - [ ] **Step 2: Run to verify they fail; Step 3: implement; Step 4: run** the touched packages' targeted tests, then the root-read census. |
| - [ ] **Step 5: Commit** `boxes: participant identity by box and instance, contextual report validation with a freshness table, the lane, the source precheck, target-ready, a move-bound stage book from observed copies, and the per-host status block`. |
| @@ -160,8 +161,9 @@ Builds on plan 1 (`2026-10-22-self-serve-moves-1-lifetimes-and-order.md`, rev 11 |
| - Test: `internal/e2e/selfserve/job_test.go` |
| **Interfaces:** |
| -- `func (s *Server) selfServeApply(ctx context.Context, t *leaseToken, row *SelfServeMove, req request, mut func(at time.Time) func(*provision.Record) error, confirmed func(provision.Record) bool) (*provision.Record, error)` with `type request struct { Kind, Phase string }`: (1) the judged record → `expectSeq = rec.RecordSeq`; `proposed = max(now, latest stamp on the record + 1 ns)`; `nonce = random`; `IssueSelfServeRequest(id, kind, phase, nonce, expectSeq, proposed, now)` (false → a request is outstanding: go to (3)); (2) `provision.SetRecordMove(ctx', mut(proposed))` with the lease-bounded context and the write guard; (3) **confirmation** from a fresh judged read, whatever (2) returned: `confirmed(rec)` (the record carries `proposed` in the field this request sets) → `ConfirmSelfServeRequest(id, nonce, now)`; else `rec.RecordSeq == expectSeq` (unchanged) → the write did not land: repeat (2) with the same `proposed` (idempotent replay; bounded by the lease); else (changed by another writer) → `VoidSelfServeRequest(id, nonce)` and return `errRequestVoided` (the phase function decides again next tick); after a confirmed write: `bumpSerialErr` and, on success, `SetSelfServeNotifiedSeq(id, rec.RecordSeq)`; a failed bump leaves `notified_seq` behind and every later tick bumps again while `rec.RecordSeq != notified_seq`. A crash anywhere resumes at (3) on the next tick from the persisted request. |
| -- `selfServeStep`: (1) the row's deadlines: an expired `StepDeadline`/`WindowDeadline` → `SetSelfServeExpiry(id, phase, true)` (no action yet); (2) the judged record; (3) **reconcile forward** by the compatible sets (Task 4): the row's phase not in the record's set → `AdvanceSelfServe` to the set's first phase (without the abort predicate for `opened`), which clears a foreign expiry; an unconfirmed request → step (3) of `selfServeApply` first; (4) the expiry action for an `expiry_pending` whose `expiry_phase == phase`: the record at or before `offline-landed` → abort; committed open → the blocked action (page); unreadable → `open-uncertain`; `health` → warning, page, `finishing`; (5) the abort step when requested and the record at or before `offline-landed`; (6) the phase function. Every wait for a report is for `Request`/`StagedRequest == request_proposed` of the **confirmed** request; the report is consumed by the `AdvanceSelfServe` that leaves the waiting phase, with the nonce. |
| +- `type persistedRequest struct { Kind, Phase, Nonce string; ExpectSeq uint64; Proposed time.Time }`; `func (s *Server) selfServeApply(ctx context.Context, t *leaseToken, row *SelfServeMove, req request, mut func(at time.Time, rc provision.Receipt) func(*provision.Record) error, confirmed func(rec provision.Record, pr persistedRequest) bool) (*provision.Record, error)` with `type request struct { Kind, Phase string }`: (1) the judged record → `pr.ExpectSeq = rec.RecordSeq`; `pr.Proposed = max(now, latest stamp on the record + 1 ns)`; `pr.Nonce = random`; `IssueSelfServeRequest(id, kind, phase, nonce, expectSeq, proposed, now)` (false → a request is outstanding: load it as `pr` and go to (3)); (2) `provision.SetRecordMoveAt(ctx', s, acct, app, host, pr.ExpectSeq, mut(pr.Proposed, Receipt{pr.Nonce}))` with the lease-bounded context and the write guard: **one CAS attempt against the record at `ExpectSeq`; no fresh-read retry inside**; (3) **confirmation** from a fresh judged read, whatever (2) returned: `rec` carries the receipt (`SelfServe.LastRequest == pr.Nonce`, or `LastMove.Request == pr.Nonce` for a clearing write; `confirmed` may add the phase's own check) → `ConfirmSelfServeRequest(id, nonce, now)` (false → voided meanwhile: stop); else `rec.RecordSeq == pr.ExpectSeq` (unchanged) → the write did not land: repeat (2) (idempotent replay of the same bound write; bounded by the lease); else (the record advanced without the receipt: another writer, an operator) → `VoidSelfServeRequest(id, nonce)` and return `errRequestVoided`; the bound write makes this safe: a PUT transmitted against `ExpectSeq` cannot commit on a record that has moved past it; after a confirmed write: `bumpSerialErr` and, on success, `SetSelfServeNotifiedSeq(id, rec.RecordSeq)`; a failed bump leaves `notified_seq` behind and every later tick bumps again while `rec.RecordSeq != notified_seq`. A crash anywhere resumes at (3) on the next tick from the persisted request. |
| +- `provision.SetRecordMoveAt(ctx, s, acct, app, host string, expectSeq uint64, mut) (*provision.Record, error)`: reads the judged record; `rec.RecordSeq != expectSeq` → `ErrCASConflict` without writing; otherwise one `WriteRecord` against the observed ETag; a CAS conflict → `ErrCASConflict` (no retry). `SetRecordMove` (today's, with its retry) is no longer used by any self-serve or operator move writer. |
| +- `selfServeStep`: (1) the row's deadlines: an expired `StepDeadline`/`WindowDeadline` → `SetSelfServeExpiry(id, phase, true)` (no action yet); (2) the judged record; (3) **reconcile forward** by the compatible sets (Task 4): an unconfirmed request → step (3) of `selfServeApply` first, **except** when `expiry_pending` is set for the issuing phase and the record is at or before `offline-landed`: then the record is observed once; the receipt present → confirmed (the write won before the expiry, the expiry is post-write); absent → `fenceRecord` (plan 1; so the outstanding bound PUT can never land), `VoidSelfServeRequest`, and the expiry action runs (abort), **never a replay** (`TestExpiredOpenIsNeverReplayed`); then the row's phase not in the record's set → `AdvanceSelfServe` to the set's first phase (without the abort predicate for `opened`), which clears a foreign expiry; (4) the expiry action for an `expiry_pending` whose `expiry_phase == phase`: the record at or before `offline-landed` → abort; committed open → the blocked action (page); unreadable → `open-uncertain`; `health` → warning, page, `finishing`; (5) the abort step when requested and the record at or before `offline-landed`; (6) the phase function. Every wait for a report is for `Request`/`StagedRequest == request_proposed` of the **confirmed** request; the report is consumed by the `AdvanceSelfServe` that leaves the waiting phase, with the nonce. |
| - **Reconciliation and recovery table** (the record is the truth): |
| | row phase | record | action | |
| @@ -184,10 +186,10 @@ Builds on plan 1 (`2026-10-22-self-serve-moves-1-lifetimes-and-order.md`, rev 11 |
| - `reports`: the `lane` (`OK`), the source `precheck` and the target `ready` with `Request == PrecheckAt`, each from the pinned participant; `targetReadyDecision` and `sourcePrecheckDecision` (the console enumerates the required path set and fact set itself: a missing entry or `!Known` → "could not determine <fact> on <label>"; not OK → the fact's sentence: "the target box already has a route for <host>", "the target box has timers for <host>", "the target box has a stale live link for <host> naming <build>", "the target box's programs file names <build>", "the target box already holds data for <host>: <path>", "the target box's park directory is not clean", "the target box has too little free disk for <host>" (`!Known` or `< 2 × replica`), "the target bucket already holds objects for <host>", "the target bucket holds a lock for <host> (<owner>)", "<host> has a stray entry beside its files directory: <path>", "<host>'s files cannot be archived: <detail>"; before staging both prefixes must be `Known` and empty); both pass → `selfServeApply(stage, OfflineQueuedToStaging, confirmed: StagedAt == proposed)`, `StagingStartedAt = now`, → `staging`; 5 min → abort. |
| - `staging`: the stager report with `StagedRequest == row.StageRequest`: the ready rule `Rounds >= 2 && RoundCopiedBytes + RoundDeferredBytes < 16 MiB` → `prechecks`; else `selfServeApply(stage, OfflineRestage, confirmed: StagedAt == proposed)` → a new wait; a `failed` report → abort with its `Detail`; a round past `max(10 min, 3 × est.Copy)` → abort; `now - StagingStartedAt > 30 min` or `Rounds >= 10` → `prechecks` from the last completed round. |
| - `prechecks`: `selfServeApply(precheck, OfflinePrecheck, confirmed: PrecheckAt == proposed)`; both participants' reports answering it; the staged-ownership comparison (Task 6); `InstancesPrefix` known and empty; both pass → `ready`, `WindowDeadline = now + 24 h`, the estimate refined and `OfflineSetEstimate` (a later disagreement resolves towards the row); 5 min → abort. |
| - - `ready`: every 10 min a restage; `WindowDeadline` → abort "the move was not confirmed in time"; the cut → `restage` (Task 5). |
| + - `ready`: every 10 min a **maintenance restage** (request kind `maintain`: `selfServeApply(maintain, OfflineRestage)`, the matching `staged` report consumed by an advance `ready → ready` with the nonce; the row stays `ready`, `WindowDeadline` is **not** touched, no prechecks run, and the cut predicate is untouched); `WindowDeadline` → abort "the move was not confirmed in time"; the cut → `restage` (Task 5). |
| - The pre-offline abort (requested, reserving, queued, staging): no record move → the finaliser (`aborted`; `source_reopened_at = abort_requested_at`); queued → `selfServeApply(abort, OfflineAbortAt, confirmed: Move == nil && LastMove.ID == row.ID)`; staging → `OfflineAbort` → the stager's `reverted` → `OfflineFinish`; then the finaliser. |
| -- [ ] **Step 1: Write the failing tests** on the bridge of Task 10: `TestEveryBoundaryRecovers` (Review Focus 1: for every write, a crash before the write with the request persisted, a lost response, a crash before confirmation, a crash before the bump, a second crash during recovery, SQLite reopened each time; a fake store that **commits after the context was cancelled**: the replay sees the proposed stamp and confirms, no second write; one request per wait, one bump per landed write; ordinary multi-tick progression through start and flip); `TestReconcileForwardNeverRegresses` (every row of the table; `stopping(pending)` with the record still `offline-staging` stays; `landing(pending)` with `offline-sealed` stays; health and finishing survive ticks at `offline-open`); `TestRequestsAreNeverReissuedWhileUnconsumed`; `TestStaleEvidenceNeverSatisfiesANewRequest` (an issued-but-unwritten restage: the previous round's report does not satisfy it; after the crash the replay lands the proposed stamp and only a report carrying it does); `TestAbandonedAdmissionIsFencedAndConditional` (a queue PUT committing after a negative re-read: the fence observes the move and the job continues as `queued`; a PUT that never lands: the fence succeeds, the row is voided durably before the release); `TestReplacedParticipantAtEveryGate`; `TestReservingWaitsThenAborts`; `TestEligibilityRechecksBeforeQueue`; `TestReportsDecideFromFacts`; `TestStagingReadyRule`; `TestPrechecksCompareTheBook`; `TestReadyRestagesAndExpires`; `TestDeadlinesEvaluateBeforeEvidence`; `TestAbortBeforeOfflineIsLossless`; `TestQueueOrderAndDisjointProgress` (three rows: two on one box in order, a third on disjoint boxes advances while the second waits). |
| +- [ ] **Step 1: Write the failing tests** on the bridge of Task 10: `TestEveryBoundaryRecovers` (Review Focus 1: for every write, a crash before the write with the request persisted, a lost response, a crash before confirmation, a crash before the bump, a second crash during recovery, SQLite reopened each time; a fake store that **commits after the context was cancelled**: the replay sees the proposed stamp and confirms, no second write; one request per wait, one bump per landed write; ordinary multi-tick progression through start and flip); `TestReconcileForwardNeverRegresses` (every row of the table; `stopping` with no request issued yet and the record still `offline-staging` stays; `stopping(pending)` likewise; `landing` with no request yet and `offline-sealed` stays; health and finishing survive ticks at `offline-open`); `TestRequestsAreNeverReissuedWhileUnconsumed`; `TestStaleEvidenceNeverSatisfiesANewRequest` (an issued-but-unwritten restage: the previous round's report does not satisfy it; after the crash the replay lands the proposed stamp and only a report carrying it does); `TestAbandonedAdmissionIsFencedAndConditional` (a queue PUT committing after a negative re-read: the fence observes the move and the job continues as `queued`; a PUT that never lands: the fence succeeds, the row is voided durably before the release); `TestReplacedParticipantAtEveryGate`; `TestReservingWaitsThenAborts`; `TestEligibilityRechecksBeforeQueue`; `TestReportsDecideFromFacts`; `TestStagingReadyRule`; `TestPrechecksCompareTheBook`; `TestReadyRestagesAndExpires` (the maintenance restage keeps the row in `ready`, keeps `WindowDeadline`, never enters `restage`/`barrier`, and the cut still needs its door); `TestExpiredOpenIsNeverReplayed` (an open request persisted, the PUT never sent, a restart after the landed deadline: the record is fenced, the request voided, the move aborts; the same with the PUT committed before the crash → confirmed, the move proceeds as opened); `TestBoundWritesNeverLandAfterAVoid` (another writer advances the record after a request was issued: the bound write fails, the request is voided, no later commit appears); `TestDeadlinesEvaluateBeforeEvidence`; `TestAbortBeforeOfflineIsLossless`; `TestQueueOrderAndDisjointProgress` (three rows: two on one box in order, a third on disjoint boxes advances while the second waits). |
| - [ ] **Step 2: Run to verify they fail; Step 3: implement; Step 4: run** `go test -tags e2e ./internal/e2e/selfserve -run 'EveryBoundary|ReconcileForward|RequestsAreNever|StaleEvidence|AbandonedAdmission|ReplacedParticipant|Reserving|EligibilityRechecks|ReportsDecide|StagingReady|PrechecksCompare|ReadyRestages|DeadlinesEvaluate|AbortBeforeOffline|QueueOrder' -count=1`. |
| - [ ] **Step 5: Commit** `console: the self-serve job from requested to ready, reconciling forward with persisted requests, a conditional fenced abandonment and at-least-once notification`. |
| @@ -201,11 +203,11 @@ Builds on plan 1 (`2026-10-22-self-serve-moves-1-lifetimes-and-order.md`, rev 11 |
| **Interfaces:** |
| - Phases and deadlines (`est` the row's terms): `restage`: `selfServeApply(stage, OfflineRestage)`; the matching `staged`; `max(5 min, 2 × est.Copy)`; expiry → abort → `barrier`. `barrier`: `selfServeApply(precheck, OfflinePrecheck)`; both participants' reports passing the decisions and the book comparison with zero deferrals; `max(5 min, 2 × est.Copy)`; a failed fact or expiry → abort → `stopping`. `stopping`: `offlineRecordChecks` + `offlineRegionChecks(..., stage: false, requireIDs: true, "")` on the fresh record, then `selfServeApply(start, OfflineStartStaged, confirmed: State == offline-stopping && SelfServe.OfflineAt == proposed)`; then the seal check with participants → `selfServeApply(seal, OfflineSeal, confirmed: State == offline-sealing)`; `max(5 min, 2 × est.Stop)`; expiry → abort. `sealing`: the sealer's `sealed`; `max(10 min, 3 × est.Copy)`; expiry → abort. `landing`: `offlineFlipCheck` → `selfServeApply(flip, OfflineFlip, confirmed: State == offline-landing)`; the pinned target's `landed`; `max(10 min, 3 × est.Land)`; expiry → abort. `landed`: `offlineOpenCheck` → write the open record (create-only, `OpenedAt = proposed`; `open_record_pending = 1` until it exists) → `selfServeApply(open, OfflineOpenTransition, confirmed: State == offline-open && OpenedAt == proposed)`; a refusal or `max(3 min, 2 × est.Open)` **while the record is `offline-landed`** → abort; an unconfirmable write (the record unreadable) → `open-uncertain`. `open-uncertain`: re-read each tick: `offline-open` → `opened`; `offline-landed` → `landed` (the deadline continues); 5 min unreadable → page, stay. `opened`: the target's `opened`; `max(3 min, 2 × est.Open)`; expiry → blocked: page, retry. `health`: a probe when `now - HealthLastProbeAt >= 30 s` (`probeTarget`: the target box's `EIP` from its edges file, TLS with the host as SNI and `Host`, `GET <ReadyPathOrDefault>?carlos-probe=<random>`, `Cache-Control: no-cache`, no redirects, 30 s; OK when 2xx and `x-carlos-version` equals the sealer report's `Release`; plan 3 adds `X-Carlos-Probe`/`X-Carlos-Served-By`); `RecordSelfServeHealth`; `HealthConsecutiveOK >= 2` → `finishing`; expiry `max(3 min, 2 × est.Promote) + 30 s` → `SetSelfServeWarning("the account did not answer on <region> within <n> minutes")`, page, `finishing`. `finishing`: gate 1 (`offlineFinishCheck` opened + every pinned source parked) → `selfServeApply(finish1, OfflineFinishingTransition)`, `FinishGate = lifted`; gate 2 (lifted + every pinned source gone) → `selfServeApply(finish, OfflineFinish, confirmed: Move == nil && LastMove.ID == row.ID)`; 10 min each → blocked: page, retry. `reverting`: `OfflineAbort` once (or `OfflineAbortAt` from queued); `SourceReopenedAt` when (no tombstone, or the sealer's `reverting`/`reverted` with `TombstoneLifted`) **and** every pinned source's `reverted`; `AbortFinishedAt` when `offlineStepFor` at `offline-reverting` passes → `OfflineFinish`; **the target is never required**: a target that stopped reporting after landing does not block the source's reopening; the abort's `Detail` records "target <label> did not report; its staged copy is left for cleanup"; 10 min → blocked: page, retry. |
| - **The finaliser** `selfServeFinalise(ctx, t, row, outcome, reason, warning)`: (1) `FinishSelfServeMove` (outcome; `release_pending = 1`, `effects_done = 0`); (2) the terminal effects: the timings (Task 9), the terminal event, `bumpSerialErr`; `SetSelfServeEffectsDone`; (3) `releaseBoxes(move, row.ID)`; (4) `ClearSelfServeRelease`. A crash between any two leaves the row in `OpenSelfServeMoves` and the next tick resumes at the first unfinished step. The operator doors that clear a self-serve move's record route through it. |
| -- **The operator return handshake (§7.5):** (1) `POST .../move/offline/return {"region", "return_of"}` (operator-only, `offlineGate`): the open record at `OpenRecordKey(host, return_of)` must exist with `To == rec.Region`, `From == region`, `Incarnation == rec.Incarnation`, and the record carries no move; else 409 "no completed move <id> from <region> to <rec.Region> for this instance"; writes `control/offline-moves/<host>/<return_of>/return/<requested_at unix nanos>-<console id>.json` `{"requested_at", "region", "console"}` create-only (a new call makes a new request; the **newest** `requested_at` is the current request; older ones are ignored) and answers 202 "return requested; the target box will report its leftovers". (2) The moves pass of every box in `region` lists `control/offline-moves/<host>/*/return/` for hosts whose open record names `From == its region` and, when the open record carries `Participants`, whose `Source` includes its own `BoxID` (an id-less forward falls to the region rule); it reads the newest request, fills `Leftovers` against the open record (`OpenedAt`, `Layout`, `MoveID`) and writes `control/offline-moves/<host>/<return_of>/reports/return-ready-<box id>.json` (`Op = return_of`, role `target`, phase `ready`, `Request = requested_at`, `BoxID/InstanceID`), without touching the record; it re-reports whenever a newer request appears. (3) The operator's `stage` (or `start`) door with `return_of`: `readReturnReport(ctx, host, returnOf)` must find a report with `Request` equal to the newest request's `requested_at`, within 10 minutes, from a box whose `BoxID` is a forward source participant (or in `From` for an id-less forward); then `targetReadyDecision` in return mode accepts **exactly**: `Lock` present with owner `moves.TombstoneOwner(return_of)` and lifted; `InstancesPrefix.Objects` (current versions, `lock.json` excluded) all with `LastModified < OpenedAt` (any newer → `NewerObjects`, refused); `DataPaths["/data/instances/<host>"]` absent or present-and-empty; for `Layout == nested` also `DataPaths["/data/.instances/<host>"]` absent or present-and-empty, and the mount view likewise; for `flat` the nested pair must be absent; park directories: each present one must be `<return_of>/source/` or `<return_of>/sealer/` (absence allowed: cleanup may have removed them; any other → refused); `LiveLink`/`ProgramsFile` naming the channel build; anything else → the ordinary refusal. Then `OfflineStage(..., returnOf)`/`OfflineStart(..., returnOf)` carry it on the record. `type moves.ReturnLeftovers struct { ForwardID string \`json:"forward_id"\`; LockOwner string \`json:"lock_owner"\`; LockLifted bool \`json:"lock_lifted"\`; ParkDirs []string \`json:"park_dirs"\`; EmptyNested []string \`json:"empty_nested"\`; NewerObjects []string \`json:"newer_objects,omitempty"\`; Layout string \`json:"layout"\` }`. |
| +- **The operator return handshake (§7.5):** (1) `POST .../move/offline/return {"region", "return_of"}` (operator-only, `offlineGate`): the open record at `OpenRecordKey(host, return_of)` must exist with `To == rec.Region`, `From == region`, `Incarnation == rec.Incarnation`, and the record carries no move; else 409 "no completed move <id> from <region> to <rec.Region> for this instance"; writes `control/offline-moves/<host>/<return_of>/return/<requested_at unix nanos>-<console id>.json` `{"requested_at", "region", "console"}` create-only (a new call makes a new request; the **newest** `requested_at` is the current request; older ones are ignored) and answers 202 "return requested; the target box will report its leftovers". (2) The moves pass of every box in `region` lists `control/offline-moves/<host>/*/return/` for hosts whose open record names `From == its region` and, when the open record carries `Participants`, whose `Source` includes its own `BoxID` **and** (when pinned) its own `InstanceID` (an id-less forward falls to the region rule); a box whose `BoxID` matches but whose `InstanceID` differs does nothing; it reads the newest request, fills `Leftovers` against the open record (`OpenedAt`, `Layout`, `MoveID`) and writes `control/offline-moves/<host>/<return_of>/reports/return-ready-<box id>.json` (`Op = return_of`, role `target`, phase `ready`, `Request = requested_at`, `BoxID/InstanceID`), without touching the record; it re-reports whenever a newer request appears. (3) The operator's `stage` (or `start`) door with `return_of`: `readReturnReport(ctx, host, returnOf)` must find a report with `Request` equal to the newest request's `requested_at`, within 10 minutes, from a box whose `BoxID` **and `InstanceID`** match a forward source participant (or in `From` for an id-less forward); the open record's `Incarnation` must equal the current record's **again here** and inside the guarded write (`OfflineStage`/`OfflineStart` with `returnOf` refuse with `ErrMoveConflict` "the instance was recreated since that move" when `rec.Incarnation != open.Incarnation`, the open record being passed to the mutation); then `targetReadyDecision` in return mode accepts **exactly**: `Lock` present with owner `moves.TombstoneOwner(return_of)` and lifted; `InstancesPrefix.Objects` (current versions, `lock.json` excluded) all with `LastModified < OpenedAt` (any newer → `NewerObjects`, refused); `DataPaths["/data/instances/<host>"]` absent or present-and-empty; for `Layout == nested` also `DataPaths["/data/.instances/<host>"]` absent or present-and-empty, and the mount view likewise; for `flat` the nested pair must be absent; park directories: each present one must be `<return_of>/source/` or `<return_of>/sealer/` (absence allowed: cleanup may have removed them; any other → refused); `LiveLink`/`ProgramsFile` naming the channel build; anything else → the ordinary refusal. Then `OfflineStage(..., returnOf)`/`OfflineStart(..., returnOf)` carry it on the record. `type moves.ReturnLeftovers struct { ForwardID string \`json:"forward_id"\`; LockOwner string \`json:"lock_owner"\`; LockLifted bool \`json:"lock_lifted"\`; ParkDirs []string \`json:"park_dirs"\`; EmptyNested []string \`json:"empty_nested"\`; NewerObjects []string \`json:"newer_objects,omitempty"\`; Layout string \`json:"layout"\` }`. |
| - Paging: `alerting.Page` through `MovePageAttempt`/`SetMovePageAttempt` (seconds encoding, unchanged) with keys `selfserve:<id>:<condition>` every 10 minutes for: `opened` past its deadline; a health failure; `finishing`/`reverting` past 10 minutes; `open-uncertain` unreadable past 5 minutes; an open record still pending after 5 minutes; a reservation the reclaim rule cannot reclaim older than the grace period (the per-tick scan, also at start); a lane probe failing on a configured pair; the attempt is recorded **before** the page is sent, as the existing paging does. |
| - Copy rows: "the account did not answer on <region> within <n> minutes"; "the move was not started in time"; "the move was not confirmed in time"; "another move is in flight"; "box <label> was replaced"; "the move could not be started"; "the instance was recreated"; "target <label> did not report; its staged copy is left for cleanup"; "no completed move <id> from <region> to <rec.Region> for this instance"; "return requested; the target box will report its leftovers"; the fact refusals of Task 7. |
| -- [ ] **Step 1: Write the failing tests** on the bridge: `TestCutRunsEveryPhaseToDone`; `TestEveryDeadlineBeforeCommittedOpenAborts`; `TestLandedExpiryDefersToTheRecord` (restart after a committed open with an old `landed` expiry → `opened`, no abort; the expiry is cleared and the move runs to `done`); `TestHealthExpiryTakesPrecedence`; `TestOpenUncertainIsReconciledByTheRecord`; `TestOpenRecordPrecedesTheOpen` (a crash between the open record and the transition; a crash after; both recover); `TestAbortIsInitiatedNotCompleted` (the real sealer; Review Focus 3; the target disappears after landing → the source reopens, the detail names the target); `TestOpenedBlockedPagesAndRetries`; `TestHealthFailureFinishesWithAWarningAndPages`; `TestHealthAttemptsPersist`; `TestFinishingNeverBypassesTheGate`; `TestFinaliseResumesAtEveryStep`; `TestOperatorDoorsFinaliseASelfServeMove`; `TestJobYieldsToAnOperatorTransition`; `TestSwitchOffMidFlightCompletesAndRendersAfterRestart`; `TestTwoMovesOnDisjointBoxesRunAtOnce`; `TestTwoMovesOnOneBoxQueueInOrder`; `TestOrphanReservationIsPagedAtBootstrap`; `TestOperatorReturnHandshake` (the return door; the box's report through its real moves pass; a second request supersedes the first and the box re-reports; the stage door accepting exactly the forward's leftovers for a flat and a nested forward, fully and partially cleaned; a newer object, a live lock, a foreign park dir, a non-empty nested dir, a report from a non-participant box, an open record of another lifetime, a `return_of` whose open record is absent, each refuse; a direct start with `return_of` → the same acceptance; restart at every handshake boundary). |
| +- [ ] **Step 1: Write the failing tests** on the bridge: `TestCutRunsEveryPhaseToDone`; `TestEveryDeadlineBeforeCommittedOpenAborts`; `TestLandedExpiryDefersToTheRecord` (restart after a committed open with an old `landed` expiry → `opened`, no abort; the expiry is cleared and the move runs to `done`); `TestHealthExpiryTakesPrecedence`; `TestOpenUncertainIsReconciledByTheRecord`; `TestOpenRecordPrecedesTheOpen` (a crash between the open record and the transition; a crash after; both recover); `TestAbortIsInitiatedNotCompleted` (the real sealer; Review Focus 3; the target disappears after landing → the source reopens, the detail names the target); `TestOpenedBlockedPagesAndRetries`; `TestHealthFailureFinishesWithAWarningAndPages`; `TestHealthAttemptsPersist`; `TestFinishingNeverBypassesTheGate`; `TestFinaliseResumesAtEveryStep`; `TestOperatorDoorsFinaliseASelfServeMove`; `TestJobYieldsToAnOperatorTransition`; `TestSwitchOffMidFlightCompletesAndRendersAfterRestart`; `TestTwoMovesOnDisjointBoxesRunAtOnce`; `TestTwoMovesOnOneBoxQueueInOrder`; `TestOrphanReservationIsPagedAtBootstrap`; `TestOperatorReturnHandshake` (the return door; the box's report through its real moves pass; a second request supersedes the first and the box re-reports; the stage door accepting exactly the forward's leftovers for a flat and a nested forward, fully and partially cleaned; a newer object, a live lock, a foreign park dir, a non-empty nested dir, a report from a non-participant box, a report from the right `BoxID` with another `InstanceID`, an open record of another lifetime at the request door and again at the stage door and inside the write, a `return_of` whose open record is absent, each refuse; a direct start with `return_of` → the same acceptance; restart at every handshake boundary). |
| - [ ] **Step 2: Run to verify they fail; Step 3: implement; Step 4: run** the targeted tests, plus `go test ./internal/console -run 'SelfServe|Offline|Return' -count=1`. |
| - [ ] **Step 5: Commit** `console: the self-serve cut with the no-return boundary at the committed open, the abort facts without the target, one finaliser, durable paging and a recoverable operator return handshake`. |
| @@ -228,7 +230,7 @@ Builds on plan 1 (`2026-10-22-self-serve-moves-1-lifetimes-and-order.md`, rev 11 |
| ### Task 10: The exported bridge and the end-to-end witnesses |
| **Files:** |
| -- Create: `internal/host/e2e_bridge.go` (**in package `host`**, `//go:build e2e`: `type E2E struct { ... }` with `func NewE2E(t testing.TB, store release.Store) *E2E`, `(*E2E).Pass(ctx)` (one moves pass on every box), `(*E2E).Serial(ctx)` (passes until the serial is consumed), `(*E2E).Clock() *fakeClock`, `(*E2E).Fault(point string, f func())` (the fault points the tests name), `(*E2E).Box(label) BoxHandle` exposing the box id, instance id, registry, data bucket and the worker's `armOffline`/`reconcileSerializedWith` through methods; it reaches `movesWorker`, `moveCfg`, `newMovesWorker`, `armOffline`, `reconcileSerializedWith` directly because it is in the package, so nothing private moves and no import cycle arises; the Linux-only fixtures stay behind `//go:build linux && e2e` with a stub that skips), `internal/host/e2e_bridge_compile.go` (no tag: a one-line reference that keeps `go vet` honest about the exported names on every platform), `internal/console/testhooks.go` (`//go:build e2e`: `console.TestHooks{Tick func(now time.Time); Fault func(point string, f func())}`), `internal/e2e/selfserve/*_test.go` (`//go:build e2e`), `.amadan/test-gate.sh` (a new step `go test -tags e2e ./internal/e2e/...` after the ordinary suite, plus `go test -tags e2e -list . ./internal/e2e/selfserve | grep -q '^Test'` so an empty discovery fails the gate) |
| +- Create: `internal/host/e2e_bridge.go` (**in package `host`**, `//go:build e2e`: `type E2E struct { ... }` with `func NewE2E(t testing.TB, store release.Store) *E2E`, `(*E2E).Pass(ctx)` (one moves pass on every box), `(*E2E).Serial(ctx)` (passes until the serial is consumed), `(*E2E).Clock() *fakeClock`, `(*E2E).Fault(point string, f func())` (the fault points the tests name), `(*E2E).Box(label) BoxHandle` exposing the box id, instance id, registry, data bucket and the worker's `armOffline`/`reconcileSerializedWith` through methods; it reaches `movesWorker`, `moveCfg`, `newMovesWorker`, `armOffline`, `reconcileSerializedWith` directly because it is in the package, so nothing private moves and no import cycle arises; the Linux-only fixtures stay behind `//go:build linux && e2e` with a stub that skips), (no untagged companion file: symbols declared under `e2e` cannot be referenced without the tag; instead the gate runs `go vet -tags e2e ./internal/host/ ./internal/e2e/...` so the bridge compiles on every CI run), the fixtures the bridge needs that today live in `movesoffline_e2e_test.go` (the two-box store, registries, fake clock and the pass driver) are **moved** into `e2e_bridge.go` as non-test code under the tag and the test file imports them from there, `internal/console/testhooks.go` (`//go:build e2e`: `console.TestHooks{Tick func(now time.Time); Fault func(point string, f func())}`), `internal/e2e/selfserve/*_test.go` (`//go:build e2e`), `.amadan/test-gate.sh` (new steps `go vet -tags e2e ./internal/host/ ./internal/e2e/...` and `go test -tags e2e ./internal/e2e/...` after the ordinary suite, plus `go test -tags e2e -list . ./internal/e2e/selfserve | grep -q '^Test'` so an empty discovery fails the gate) |
| - Test: the witnesses of Tasks 7 and 8, plus `TestSelfServeForwardMove`; `TestSelfServeCancelInReady`; `TestSelfServeAbortByFailedPrecheck`; `TestSelfServeAbortByStopDeadline`; `TestSelfServeAbortByLandingFailure`; `TestSelfServeAbortByOpenCheckRefusal`; `TestSelfServePostOpenFailureIsDoneWithAWarning`; `TestSelfServeCutNamingAStaleID`; `TestSelfServeRecreatedBetweenRowAndRecord`; `TestSelfServeTwoMovesDisjointAndQueued`; `TestSelfServeReservationReclaimOnlyOnEvidence`; `TestSelfServeOperatorDoorsUnchangedOnTheBridge`; `TestSelfServeThenOperatorReturn`; `TestClaimsOffInstanceIDIsPublished` (the agent's status tick carries `InstanceID` and `BoxID` on a claims-off box). |
| - [ ] **Step 1: Write the bridge and the tests.** **Step 2: Run them with `-tags e2e`; fix what they find within this plan's scope.** **Step 3: Both census tests and the cross-builds**; write `docs/superpowers/notes/census-proposal-self-serve-moves-2.md` (the overlay's loader block, the changed envelopes, the new writers incl. the return request, the open record, the console lease and `active.json`'s table form). **Step 4: Commit** `self-serve moves: an exported two-box bridge inside package host and the end-to-end witnesses, run by the gate`; push. |
| @@ -239,5 +241,5 @@ Builds on plan 1 (`2026-10-22-self-serve-moves-1-lifetimes-and-order.md`, rev 11 |
| - Spec coverage: §5 (T1), §6.1 (T5), §6.2 (T7), §6.3 (T9, T5, T6), §7 admission and recovery (T7's table), §7.1 (T3), §7.2 and the three facts with the sealer's durable lift (T6, T8), §7.3 (T6, T7, T8's barrier), §7.4 without the probe mode (T8), §7.5 the no-return boundary and the operator return (T2's open record, T8), §7.7 (T5), §8 pages, log and member stream (T8, T9), §9 (T3's rollout). Deferred: §6.4 holding page and the probe headers (plan 3); §7.6 cleanup (plan 4), whose status contract the reclaim rule names (`RetirementsComplete`, a terminal state, `RootRequest == ""`, `ReleasePending == false`, `Metrics.BoxID`). |
| - Dependencies for plan 3: `SelfServeMove{DisplayName, OfflineAt, EstimateSeconds}`, `Participants.Target.BoxID`, `moveCfg.BoxID`, `probeTarget`; for plan 4: the status `Retirements` contract and the open record. |
| -- Names used across tasks: `selfServeGate`, `instanceRails`, `moveCapable`, `Principal.Moves/TokenID`, `selfServeMode`, `selfServeAdmits`, `recordEligibility`, `offlineRecordChecks`, `offlineRegionChecks`, `offlineSelection`, `OfflineQueued`, `OfflineQueue`, `OfflineQueuedToStaging`, `OfflinePrecheck`, `OfflineSetEstimate`, `OfflineAbortAt`, `provision.SelfServeMove`, `Participant{BoxID, Label, InstanceID, Verified}`, `Participants`, `moves.OpenRecord{..., Layout}`, `OpenRecordKey`, `moves.ParticipantRole`, `moves.ParticipantStillPresent`, `moves.OfflineReservation`, `reserveBoxes`, `releaseBoxes`, `selfServeTick`, `selfServeStep`, `selfServeApply`, `request`, `errRequestVoided`, `selfServeFinalise`, `leaseToken`, `provision.WithWriteGuard`, `IssueSelfServeRequest`, `ConfirmSelfServeRequest`, `VoidSelfServeRequest`, `AdvanceSelfServe(..., nonce)`, `SetSelfServeExpiry`, `SetSelfServeNotifiedSeq`, `SetSelfServeOpenRecordPending`, `InsertSelfServeEventTx`, `DeliverSelfServeEvent`, `selfServeMoveView`, `selfServeAbortView`, `selfServeLast`, `moves.Fact`, `PrefixFacts`, `PrefixObject`, `StagedObject{Size, ETag, VersionID, Deferred}`, `LaneReport`, `TargetReadyReport`, `SourcePrecheck{..., Layout}`, `StrayEntry`, `StraysFact`, `ReturnLeftovers`, `ValidateOfflineReportFor`, `readOfflineEvidence(ctx, host, id, m)`, `readReturnReport`, `lease.CopyEvent`, `lease.MirrorPrefixKeepObserved`, `stageBook`, `status.HostFacts`, `BoxMetrics.Hosts/Lanes/BoxID/InstanceID`, `measureFiles`, `targetReadyDecision`, `sourcePrecheckDecision`, `probeTarget`, `estimateTerms`, `MarkSelfServeTiming`, `SelfServeEvent`, `host.E2E`/`NewE2E`, `console.TestHooks`. |
| +- Names used across tasks: `selfServeGate`, `instanceRails(w, r, acct, noun, c)`, `moveCapable`, `Principal.Moves/TokenID`, `selfServeMode`, `selfServeAdmits`, `recordEligibility`, `offlineRecordChecks`, `offlineRegionChecks`, `offlineSelection`, `OfflineQueued`, `OfflineQueue`, `OfflineQueuedToStaging`, `OfflinePrecheck`, `OfflineSetEstimate`, `OfflineAbortAt`, `provision.SelfServeMove`, `Participant{BoxID, Label, InstanceID, Verified}`, `Participants`, `moves.OpenRecord{..., Layout}`, `OpenRecordKey`, `moves.ParticipantRole`, `moves.ParticipantStillPresent`, `moves.OfflineReservation`, `reserveBoxes`, `releaseBoxes`, `selfServeTick`, `selfServeStep`, `selfServeApply`, `request`, `persistedRequest`, `provision.Receipt`, `SelfServeMove.LastRequest`, `LastMove.Request`, `provision.SetRecordMoveAt`, `OfflineReservation.ExpectSeq`, `moves.FailedDetail`, `OfflineReport.Layout`, `errRequestVoided`, `selfServeFinalise`, `leaseToken`, `provision.WithWriteGuard`, `IssueSelfServeRequest`, `ConfirmSelfServeRequest`, `VoidSelfServeRequest`, `request_voided`, `AdvanceSelfServe(..., nonce)`, `SetSelfServeExpiry`, `SetSelfServeNotifiedSeq`, `SetSelfServeOpenRecordPending`, `InsertSelfServeEventTx`, `DeliverSelfServeEvent`, `selfServeMoveView`, `selfServeAbortView`, `selfServeLast`, `moves.Fact`, `PrefixFacts`, `PrefixObject`, `StagedObject{Size, ETag, VersionID, Deferred}`, `LaneReport`, `TargetReadyReport`, `SourcePrecheck{..., Layout}`, `StrayEntry`, `StraysFact`, `ReturnLeftovers`, `ValidateOfflineReportFor`, `readOfflineEvidence(ctx, host, id, m)`, `readReturnReport`, `lease.CopyEvent`, `lease.MirrorPrefixKeepObserved`, `stageBook`, `status.HostFacts`, `BoxMetrics.Hosts/Lanes/BoxID/InstanceID`, `measureFiles`, `targetReadyDecision`, `sourcePrecheckDecision`, `probeTarget`, `estimateTerms`, `MarkSelfServeTiming`, `SelfServeEvent`, `host.E2E`/`NewE2E`, `console.TestHooks`. |
| - Review Focus 1 to 5 are pinned by T7, T3, T8, T5, T1. |