carlos / platform Public

console: the lifetimes cutover door, an admission gate that activation waits on, a latched on state and the CLI command

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

claude 2 days ago 6730df5d05d9e492130c4c4096105ff8f2a3f9f4 parent 5ed73da

27 files changed, +1895 −69

cmd/console/main.go +6 −0
docs/site/cli.md +32 −0
docs/superpowers/notes/census-proposal-self-serve-moves-1.md +2 −0
internal/console/apiinstanceimport.go +8 −1
internal/console/apiinstances.go +6 −2
internal/console/apiinstanceseed.go +8 −0
internal/console/authcli.go +4 −0
internal/console/claim_invariant_fixtures_test.go +22 −11
internal/console/claim_invariant_probes_worker_test.go +3 −2
internal/console/claim_invariant_test.go +2 −0
internal/console/claim_workers.go +19 −18
internal/console/expirypass.go +5 −10
internal/console/lifetimes_cutover.go +550 −0
internal/console/lifetimes_cutover_test.go +921 −0
internal/console/regiondrift.go +14 −1
internal/console/runsweep.go +9 −9
internal/console/seedsweep.go +8 −10
internal/console/server.go +29 −2
internal/console/svcroutes_test.go +3 −0
internal/console/testdata/claim-routes.txt +2 −0
internal/provision/plan14_root_reads_test.go +3 −1
internal/provision/self_serve_moves1_census_test.go +25 −0
internal/provision/testdata/plan14-root-reads-self-serve-moves-1.tsv +14 −1
system.go +7 −1
system_lifetimes.go +103 −0
system_lifetimes_test.go +88 −0
system_test.go +2 −0

diff --git a/cmd/console/main.go b/cmd/console/main.go

index bdee22837..ed19a70ec 100644

--- a/cmd/console/main.go

+++ b/cmd/console/main.go

@@ -534,6 +534,12 @@ func serve() {

// rolled-back seeds, deletes snapshots no live seed needs. Same

// placement rule as every pass above.

go srv.RunClaimWorker(context.Background(), "StartSeedsPass")

+ // The lifetimes cutover watch (internal/console/lifetimes_cutover.go):

+ // reads control/lifetimes-cutover.json every 10 seconds, turns instance

+ // lifetimes on once the work admitted before it has drained, and runs the

+ // claims-off app materialisation as its first admitted iteration. Off on

+ // a host-claim console.

+ go srv.RunClaimWorker(context.Background(), "StartLifetimesCutoverWatch")

// The relay's writer lease and its converging reconcile

// (internal/console/relayops.go). Same placement rule as every pass

// above, and it returns immediately when CARLOS_RELAY_REGIONS is empty —

diff --git a/docs/site/cli.md b/docs/site/cli.md

index a964ca5ab..6e15a3688 100644

--- a/docs/site/cli.md

+++ b/docs/site/cli.md

@@ -2326,6 +2326,38 @@ carlos system legacy-routes assign says which account and app the routes an oper

carlos system legacy-routes assign --name shop --to bac/shop

```

+#### Instance lifetimes

+carlos system lifetimes cutover switches a console without host claims onto instance lifetimes. It is done once per deployment, and it cannot be undone. Every box must run a release that knows about lifetimes first.

+Run it in this order:

+1. Stop the old console. An older console must never run beside this one, and must never be started again once the cutover has begun.

+2. Start the new console.

+3. Run the command, and read its output until it says on.

+```sh

+carlos system lifetimes cutover

+```

+The console writes control/lifetimes-cutover.json and stops taking new changes to instances at once. Reads still answer. Work it had already started is allowed to finish, and the command says how much is left:

+```

+instance lifetimes: switching on; 2 unit(s) of earlier work still finishing

+```

+A minute after the last of that work ends, lifetimes are on:

+```

+instance lifetimes: on since 2026-10-22T12:01:30Z

+```

+Until then, a change to an instance is refused with "the console is switching on instance lifetimes. try again in a minute". The command asks again every 5 seconds until lifetimes are on. Running it again is safe: it only reports how far the console has got. With --no-wait it prints the state once and returns.

+A console that restarts after the cutover waits a minute after its own start before it turns lifetimes on again. A marker written by hand closes the console to changes within 10 seconds. If the marker exists but cannot be read, the command says so; repair control/lifetimes-cutover.json by hand. A console with host claims has no lifetimes, and refuses the command.

+Operators only.

#### Box and group roles

carlos system box role says whether new instances may be placed on a box. A box with the default role can take them. A legacy or platform box keeps the instances it has and takes no new ones: no new instance, relocation or move lands on it. carlos system group role sets the role of every box a region's group has enrolled, and of every box it enrolls later. Both work only on a console with region HA on.

diff --git a/docs/superpowers/notes/census-proposal-self-serve-moves-1.md b/docs/superpowers/notes/census-proposal-self-serve-moves-1.md

index b22b0c549..9025d4d0f 100644

--- a/docs/superpowers/notes/census-proposal-self-serve-moves-1.md

+++ b/docs/superpowers/notes/census-proposal-self-serve-moves-1.md

@@ -9,3 +9,5 @@ Task 2 adds 15 console lifetime-store units and re-pins `migrate`, `migrateAppsK

Task 6a adds the provision lifetime judge and judged readers, listings and updates. The root census reported the provision envelope and two stale declaration pins: `IndexRecords` and `RecordIndex.Find`. Both retain their authority classification and B3,C ownership. `IndexRecords` now retains every candidate owner key; `Find` judges current bodies on each lookup, requires exactly one valid candidate, refuses corruption and read failures, and validates the chosen record's schema and shape. Nil-judge duplicate candidates remain ambiguous. The overlay records these reviewed dispositions and refreshes the provision envelope over the judge, context helpers, shared record decoding, update guards and slot-counter read limit. No root-read unit is removed, no legacy fingerprint is added, and the legacy seal and registry-field pin stand.

The write census reported four changed sites and three added sites. `GetRecordRaw` remains a read-only key-site and shares its decoder with `ObserveWith`. `updateRecordSeeding` and `claimRawUpdateRecord` remain record-writers through `WriteRecord`: the former judges before each mutation, including its CAS retry; the latter judges before checking the caller's ETag. `IndexRecords` remains a read-only key-site. `ListRecordsCompleteJudged`, `ListRecordsStrictJudged` and `readAllRecordsJudged` are added as read-only key-sites, with separate reviewed dispositions for their completeness and error behavior. Every listed missing body remains a gap; only a present stale body is silently omitted. All Task 6a entries remain `PROPOSED-self-serve-moves-1`, awaiting Paul's census ruling. The removal loop and host and console consumers remain Task 6b work.

+Task 0 adds the lifetimes cutover: the operator door `POST /api/cli/system/lifetimes/cutover`, the admission gate and its watcher in `internal/console/lifetimes_cutover.go`, and `carlos system lifetimes cutover`. The record-write census reported one unlisted and five changed sites. `Server.handleAPILifetimesCutover` is added as nonrecord: one create-only Put of `control/lifetimes-cutover.json` in the deployment bucket, with no record key and no record literal. `Server.runImportUpload` and `Server.runFrom` remain record-writers through `SetRecordSeed`; each now carries its request's admitted unit and ends it after its last write, with no write site added or moved. `Server.handleAPIInstanceList` remains a read-only key-site and lists through `ListRecordsCompleteJudged` with the request's judge, which is nil before the cutover. `Server.registerCLIRoutes` registers the door. `NewServer` adds the judge, the gate and its first marker read, and moves the claims-off app materialisation to the watch's first admitted iteration; it writes nothing. The root-read census reported four changed accepted-legacy console units (`serve`, `NewServer`, `Server.handleAPIInstanceList`, `Server.runFrom`), two re-pinned declarations (`runSystem`, `systemVerb`, the lifetimes verb at their existing authority) and three envelopes (`.`, `cmd/console`, `internal/console`). The four legacy units are re-pinned with reviewed bases, which adds four fingerprints and moves the seal; no root read, class or task is added and the registry fields stand. Everything here remains `PROPOSED-self-serve-moves-1`, awaiting Paul's census ruling.

diff --git a/internal/console/apiinstanceimport.go b/internal/console/apiinstanceimport.go

index d5716143a..109e236e7 100644

--- a/internal/console/apiinstanceimport.go

+++ b/internal/console/apiinstanceimport.go

@@ -343,6 +343,10 @@ type importUpload struct {

id string

dir, spool string

release func()

+ // unit is the request's admitted unit, handed over before the

+ // goroutine starts: the job ends it after its last write

+ // (lifetimes_cutover.go).

+ unit *admittedUnit

}

// handleAPIImportUpload is spec §5.2, in two halves (preflight M6, a

@@ -436,7 +440,7 @@ func (s *Server) handleAPIImportUpload(w http.ResponseWriter, r *http.Request) {

handedOver = true

s.seedCopies.Add(1)

go s.runImportUpload(&importUpload{ctx: context.WithoutCancel(ctx), acct: acct, app: app, host: host, by: principal,

- id: seed.ID, dir: dir, spool: spool, release: releaseScratch})

+ id: seed.ID, dir: dir, spool: spool, release: releaseScratch, unit: detachUnit(ctx)})

}

// readImportRecord reads host's record under instanceStoreTimeout.

@@ -529,6 +533,9 @@ func (s *Server) spoolImport(w http.ResponseWriter, r *http.Request, spool strin

// the sweep aborts it (Plan 8).

func (s *Server) runImportUpload(j *importUpload) {

defer s.seedCopies.Done()

+ // Registered second, so it runs after every other deferred step: the

+ // abort, the spool's removal and the reservation's release.

+ defer j.unit.done()

defer j.release()

ctx := j.ctx

sqid, app, host := j.acct.Sqid, j.app, j.host

diff --git a/internal/console/apiinstances.go b/internal/console/apiinstances.go

index 1aed8a1d5..dcb75f76f 100644

--- a/internal/console/apiinstances.go

+++ b/internal/console/apiinstances.go

@@ -751,7 +751,9 @@ func (s *Server) handleAPIInstanceList(w http.ResponseWriter, r *http.Request) {

}

ctx, cancel := context.WithTimeout(r.Context(), instanceStoreTimeout)

defer cancel()

- recs, truncated, skipped, err := provision.ListRecordsComplete(ctx, s.cfg.Releases, acct.Sqid, app, instanceListMax)

+ // Judged when the request was admitted with lifetimes on: a record its

+ // slot no longer vouches for is not listed. Unjudged (nil) as main.

+ recs, truncated, skipped, err := provision.ListRecordsCompleteJudged(ctx, s.cfg.Releases, provision.JudgeFrom(ctx), acct.Sqid, app, instanceListMax)

if err != nil {

log.Printf("console: api instances list %s/%s: %v", acct.Sqid, app, err)

jsonError(w, http.StatusBadGateway, "could not list instances")

@@ -873,7 +875,9 @@ func (s *Server) handleAPIInstanceGet(w http.ResponseWriter, r *http.Request) {

}

ctx, cancel := context.WithTimeout(r.Context(), instanceStoreTimeout)

defer cancel()

- rec, err := provision.GetRecord(ctx, s.cfg.Releases, acct.Sqid, app, host)

+ // Judged when the request was admitted with lifetimes on: a stale body

+ // reads as absent. Unjudged (nil) as main.

+ rec, _, err := provision.ReadRecordJudged(ctx, s.cfg.Releases, provision.JudgeFrom(ctx), acct.Sqid, app, host)

if errors.Is(err, os.ErrNotExist) {

http.NotFound(w, r)

return

diff --git a/internal/console/apiinstanceseed.go b/internal/console/apiinstanceseed.go

index eaf271824..a5a91ffe0 100644

--- a/internal/console/apiinstanceseed.go

+++ b/internal/console/apiinstanceseed.go

@@ -204,6 +204,10 @@ type fromCopy struct {

plan dbdoor.RestorePlan

client *dbdoor.Client

release func()

+ // unit is the request's admitted unit, handed over before the

+ // goroutine starts: runFrom ends it after its last write

+ // (lifetimes_cutover.go).

+ unit *admittedUnit

}

// createFrom is --from (spec §4). Every refusal up to the record's

@@ -219,6 +223,7 @@ func (s *Server) createFrom(w http.ResponseWriter, r *http.Request, c seedCreate

held := &heldResponse{header: w.Header()}

job := s.beginFrom(held, r, c, from)

if job != nil {

+ job.unit = detachUnit(r.Context())

s.seedCopies.Add(1)

go s.runFrom(job)

}

@@ -368,6 +373,9 @@ func (s *Server) beginFrom(w http.ResponseWriter, r *http.Request, c seedCreate,

// sweep aborts it (Plan 8).

func (s *Server) runFrom(j *fromCopy) {

defer s.seedCopies.Done()

+ // Registered second, so it runs after every other deferred step: the

+ // abort, the scratch's removal and the reservation's release.

+ defer j.unit.done()

defer j.release()

ctx := j.ctx

c := j.c

diff --git a/internal/console/authcli.go b/internal/console/authcli.go

index c597c3e56..5a42cc6c8 100644

--- a/internal/console/authcli.go

+++ b/internal/console/authcli.go

@@ -1017,6 +1017,10 @@ func (s *Server) registerCLIRoutes() {

s.cliRoute("POST /api/cli/system/seeds-floor", svcrole.Set{}, s.handleAPISystemSeedsFloor)

// Legacy route assignments (seeded-instances spec §6.4): apiSystemRoller.

s.cliRoute("POST /api/cli/system/legacy-routes", svcrole.Set{}, s.handleAPISystemLegacyRoutes)

+ // The lifetimes cutover (lifetimes_cutover.go): an operator's own CLI

+ // token, else the move family's 404; closed to every service

+ // credential. A host-claim console answers its 409.

+ s.cliRoute("POST /api/cli/system/lifetimes/cutover", svcrole.Set{}, s.handleAPILifetimesCutover)

// Pointer digests (#253, spec §4.6): each account's emission marker and

// the channels still without digests. apiSystemRoller, like the update

// doors.

diff --git a/internal/console/claim_invariant_fixtures_test.go b/internal/console/claim_invariant_fixtures_test.go

index 8919640e9..74883ad8b 100644

--- a/internal/console/claim_invariant_fixtures_test.go

+++ b/internal/console/claim_invariant_fixtures_test.go

@@ -1388,6 +1388,16 @@ var claimFixtures = map[string]claimFixture{

// app may import routes filed under the operator account. It waits for

// #250 with the import doors, and refuses before it reads anything; the

// app it names is the one whose imports it would widen.

+ // The lifetimes cutover (lifetimes_cutover.go) is for consoles without

+ // host claims: a claim console answers its 409 after the operator check

+ // and before it reads or writes anything. It names no app; the app here

+ // is the world's, whose foreign run proves nothing app-scoped moved.

+ "route POST /api/cli/system/lifetimes/cutover": {refused: &claimRefusalFixture{

+ apps: claimRefs("bab/titogo"),

+ do: claimRefusedAPI(http.MethodPost, "/api/cli/system/lifetimes/cutover", ""),

+ status: http.StatusConflict,

+ msg: msgLifetimesClaimsOn,

+ }},

"route POST /api/cli/system/legacy-routes": {refused: &claimRefusalFixture{

apps: claimRefs("bab/kass"),

do: claimRefusedAPI(http.MethodPost, "/api/cli/system/legacy-routes", `{"name":"kass","to":"bab/kass"}`),

@@ -1842,17 +1852,18 @@ var claimFixtures = map[string]claimFixture{

}},

// ---- workers ----

- "worker LogPlacementDiscrepancies": claimRead("logs region drift between the registry and instance records; writes nothing"),

- "worker PublishMovesConsole": claimNonApp("the moves console record (control/moves-console.json): the platform's moves protocol and key fingerprint; no app, and nothing with the moves switch off"),

- "worker ReportApexSquats": claimRead("logs apex squats found in the registry; writes nothing"),

- "worker StartRunSweep": claimRead("off on a host-claim console, where every run route refuses: it logs that and writes nothing"),

- "worker StartSeedsPass": claimRead("off on a host-claim console, where every seed door and finishSeed refuse: it logs that and writes nothing"),

- "worker StartAlertWatcher": claimNonApp("box alert state under alerts/boxes and operator mail"),

- "worker StartEconomicsPass": claimNonApp("daily usage census, statements and bills: deployment and account economics"),

- "worker StartGeoipJobs": claimNonApp("GeoIP orphan sweep, D1 expiry and feed status under control/geoip/: the platform database, no app"),

- "worker StartHostReports": claimNonApp("this console's census report (status/consoles) and roster activation release"),

- "worker StartInfraPass": claimNonApp("the deployment's infrastructure inventory snapshot"),

- "worker StartPlanLimitsPass": claimNonApp("account plan rows and their limits records under accounts/<sqid>/limits.json; names no app"),

+ "worker LogPlacementDiscrepancies": claimRead("logs region drift between the registry and instance records; writes nothing"),

+ "worker PublishMovesConsole": claimNonApp("the moves console record (control/moves-console.json): the platform's moves protocol and key fingerprint; no app, and nothing with the moves switch off"),

+ "worker ReportApexSquats": claimRead("logs apex squats found in the registry; writes nothing"),

+ "worker StartRunSweep": claimRead("off on a host-claim console, where every run route refuses: it logs that and writes nothing"),

+ "worker StartSeedsPass": claimRead("off on a host-claim console, where every seed door and finishSeed refuse: it logs that and writes nothing"),

+ "worker StartLifetimesCutoverWatch": claimRead("off on a host-claim console, where lifetimes are not for it: it logs that and reads nothing"),

+ "worker StartAlertWatcher": claimNonApp("box alert state under alerts/boxes and operator mail"),

+ "worker StartEconomicsPass": claimNonApp("daily usage census, statements and bills: deployment and account economics"),

+ "worker StartGeoipJobs": claimNonApp("GeoIP orphan sweep, D1 expiry and feed status under control/geoip/: the platform database, no app"),

+ "worker StartHostReports": claimNonApp("this console's census report (status/consoles) and roster activation release"),

+ "worker StartInfraPass": claimNonApp("the deployment's infrastructure inventory snapshot"),

+ "worker StartPlanLimitsPass": claimNonApp("account plan rows and their limits records under accounts/<sqid>/limits.json; names no app"),

"worker StartClaimRecoveryPass": {door: &claimDoor{

apps: claimRefs("bab/titogo"),

setup: func(t *testing.T, cx *claimCase) { cx.crashedRestore(t) },

diff --git a/internal/console/claim_invariant_probes_worker_test.go b/internal/console/claim_invariant_probes_worker_test.go

index 765f8db9a..605255021 100644

--- a/internal/console/claim_invariant_probes_worker_test.go

+++ b/internal/console/claim_invariant_probes_worker_test.go

@@ -77,8 +77,9 @@ var claimWorkerProbes = map[string]claimProbe{

cx.addRoute(t, registry.Route{Host: "titogo.bab.oncarlos.com", Kind: "instance", Status: "ready",

App: "kass", Account: "bab", Addr: "127.0.0.1:9301"})

}, "apex squat", "titogo.bab.oncarlos.com"),

- "worker StartRunSweep": claimLogWorkerProbe("StartRunSweep", nil, "run sweep", "host-claim console"),

- "worker StartSeedsPass": claimLogWorkerProbe("StartSeedsPass", nil, "seeds pass", "host-claim console"),

+ "worker StartRunSweep": claimLogWorkerProbe("StartRunSweep", nil, "run sweep", "host-claim console"),

+ "worker StartSeedsPass": claimLogWorkerProbe("StartSeedsPass", nil, "seeds pass", "host-claim console"),

+ "worker StartLifetimesCutoverWatch": claimLogWorkerProbe("StartLifetimesCutoverWatch", nil, "lifetimes cutover watch", "host-claim console"),

"worker LogPlacementDiscrepancies": claimLogWorkerProbe("LogPlacementDiscrepancies", claimRegionalApp,

msgPlacementDiscrepancy("drift.go.tito.io", "blog", "uk")),

diff --git a/internal/console/claim_invariant_test.go b/internal/console/claim_invariant_test.go

index 5e438dd64..616feb5a8 100644

--- a/internal/console/claim_invariant_test.go

+++ b/internal/console/claim_invariant_test.go

@@ -114,6 +114,8 @@ var claimUnsupported = map[string]string{

"no operation gates both, so every transfer is refused",

claimMoveResolveUnavailableMsg: "the operator's door out of a stuck move (CARLOS_MOVES_PHASE1) has only its gate until " +

"the Moves resolution body lands; it answers this sentence and writes nothing",

+ msgLifetimesClaimsOn: "instance lifetimes are the claims-off console's protocol (self-serve moves plan 1); " +

+ "the cutover door refuses after its operator check and before it reads or writes anything",

seedClaimConsoleRefusal().Msg: "seeded instances need nested boxes, and the claim-armed path waits for #250 (seeded-instances spec §3.4); " +

"every import door refuses before it reads the record, and the legacy route assignment before it reads anything",

}

diff --git a/internal/console/claim_workers.go b/internal/console/claim_workers.go

index 803e6bbc7..204edae8c 100644

--- a/internal/console/claim_workers.go

+++ b/internal/console/claim_workers.go

@@ -5,24 +5,25 @@ import "context"

func (s *Server) registerClaimWorkers() {

s.claimWorkers = map[string]func(context.Context){}

for name, h := range map[string]func(context.Context){

- "StartHostReports": s.StartHostReports,

- "StartAlertWatcher": s.StartAlertWatcher,

- "StartGeoipJobs": s.StartGeoipJobs,

- "StartClaimRecoveryPass": s.StartClaimRecoveryPass,

- "StartEconomicsPass": s.StartEconomicsPass,

- "StartInfraPass": s.StartInfraPass,

- "StartScanPass": s.StartScanPass,

- "StartScanDrain": s.StartScanDrain,

- "StartExpiryPass": s.StartExpiryPass,

- "StartPlanLimitsPass": s.StartPlanLimitsPass,

- "StartRunSweep": s.StartRunSweep,

- "StartSeedsPass": s.StartSeedsPass,

- "StartRelay": s.StartRelay,

- "StartMailEventPush": s.StartMailEventPush,

- "StartDoorKeyResign": s.StartDoorKeyResign,

- "ReportApexSquats": s.ReportApexSquats,

- "LogPlacementDiscrepancies": s.LogPlacementDiscrepancies,

- "PublishMovesConsole": s.publishMovesConsoleWorker,

+ "StartHostReports": s.StartHostReports,

+ "StartAlertWatcher": s.StartAlertWatcher,

+ "StartGeoipJobs": s.StartGeoipJobs,

+ "StartClaimRecoveryPass": s.StartClaimRecoveryPass,

+ "StartEconomicsPass": s.StartEconomicsPass,

+ "StartInfraPass": s.StartInfraPass,

+ "StartScanPass": s.StartScanPass,

+ "StartScanDrain": s.StartScanDrain,

+ "StartExpiryPass": s.StartExpiryPass,

+ "StartPlanLimitsPass": s.StartPlanLimitsPass,

+ "StartRunSweep": s.StartRunSweep,

+ "StartSeedsPass": s.StartSeedsPass,

+ "StartLifetimesCutoverWatch": s.StartLifetimesCutoverWatch,

+ "StartRelay": s.StartRelay,

+ "StartMailEventPush": s.StartMailEventPush,

+ "StartDoorKeyResign": s.StartDoorKeyResign,

+ "ReportApexSquats": s.ReportApexSquats,

+ "LogPlacementDiscrepancies": s.LogPlacementDiscrepancies,

+ "PublishMovesConsole": s.publishMovesConsoleWorker,

} {

e := claimEntry{Kind: "worker", Pattern: name, Handler: claimHandlerName(h)}

s.claimEntries.add(e)

diff --git a/internal/console/expirypass.go b/internal/console/expirypass.go

index 2f58f1817..adb53b0d2 100644

--- a/internal/console/expirypass.go

+++ b/internal/console/expirypass.go

@@ -28,16 +28,11 @@ func (s *Server) StartExpiryPass(ctx context.Context) {

if s.cfg.Releases == nil {

return

}

- t := time.NewTicker(expiryPassInterval)

- defer t.Stop()

- for {

- s.expiryPass(ctx, time.Now())

- select {

- case <-ctx.Done():

- return

- case <-t.C:

- }

+ // Each pass is a unit of record work (lifetimes_cutover.go): its purges

+ // remove instance records, so it is admitted as a writer.

+ lifetimesWorker(ctx, expiryPassInterval, func(ctx context.Context) {

+ s.lifetimesIteration(ctx, "expiry pass", true, false, func(ctx context.Context) { s.expiryPass(ctx, time.Now()) })

+ })

}

// expiryPass is one pass at the given clock.

diff --git a/internal/console/lifetimes_cutover.go b/internal/console/lifetimes_cutover.go

new file mode 100644

index 000000000..0906d5a54

--- /dev/null

+++ b/internal/console/lifetimes_cutover.go

@@ -0,0 +1,550 @@

+package console

+import (

+ "context"

+ "encoding/json"

+ "errors"

+ "fmt"

+ "log"

+ "net/http"

+ "os"

+ "strings"

+ "sync"

+ "sync/atomic"

+ "time"

+ "github.com/carlosframework/platform/internal/alerting"

+ "github.com/carlosframework/platform/internal/provision"

+ "github.com/carlosframework/platform/internal/release"

+// The lifetimes cutover (self-serve moves plan 1, Task 0; the Global

+// Constraint "Cutover (the late legacy delete)"). A claims-off console turns

+// instance lifetimes on once, for good, behind an admission gate:

+//

+// - every claims-off unit of record work (a request, a detached

+// continuation, one worker iteration) is admitted before it starts and

+// ended when it is over, and runs in the mode it was admitted in;

+// - the marker control/lifetimes-cutover.json, written create-only by the

+// operator door, closes admission to legacy work at once;

+// - lifetimes turn on only after a validated marker, no legacy unit still

+// running, and lifetimeSettle past the latest of the marker's time, this

+// process's start and the end of the last legacy unit, so a legacy write

+// that storage had fully received lands before lifetimes start;

+// - on is latched for the life of the process.

+//

+// The console's cutover is the only one: boxes perform no slot mutation of

+// their own and a claims-off box purges nothing from this release on.

+const (

+ lifetimesCutoverKey = "control/lifetimes-cutover.json"

+ lifetimesCutoverPath = "/api/cli/system/lifetimes/cutover"

+ // lifetimeSettle is how long after the last legacy work a fully received

+ // storage request is assumed to have completed.

+ lifetimeSettle = 60 * time.Second

+ // lifetimesWatchInterval is how often the watcher reads the marker: a

+ // marker written by hand closes admission within this long.

+ lifetimesWatchInterval = 10 * time.Second

+ // lifetimesPageEvery bounds the unreadable-marker page while the console

+ // is still starting.

+ lifetimesPageEvery = time.Hour

+ msgLifetimesDraining = "the console is switching on instance lifetimes. try again in a minute"

+ msgLifetimesStarting = "the console is starting. try again in a minute"

+ msgLifetimesClaimsOn = "this console runs host claims; lifetimes are not for it"

+ msgLifetimesMarkerUnreadable = "the cutover marker exists but cannot be read. repair control/lifetimes-cutover.json by hand"

+ msgLifetimesMarkerUnwritten = "could not write the cutover marker. Try again."

+ pageLifetimesMarkerUnread = "the console cannot read control/lifetimes-cutover.json"

+// lifetimesNow is the gate's clock. A TEST SEAM: NewServer captures it, so a

+// test sets it before it builds the server.

+var lifetimesNow = time.Now

+// lifetimesIterationHook, when set, sees every admitted worker iteration

+// before it runs, with the context it runs under. A TEST SEAM.

+var lifetimesIterationHook func(name string, ctx context.Context, u *admittedUnit)

+// cutoverMarker is the body of control/lifetimes-cutover.json.

+type cutoverMarker struct {

+ At time.Time `json:"at"`

+ By string `json:"by"`

+type cutoverState int

+const (

+ cutoverUnknown cutoverState = iota // the marker not yet read successfully

+ cutoverBefore // the marker read and absent: legacy, as main

+ cutoverDraining // the marker exists (or may): no new legacy unit

+ cutoverOn // lifetimes on; latched

+func (c cutoverState) String() string {

+ switch c {

+ case cutoverBefore:

+ return "before"

+ case cutoverDraining:

+ return "draining"

+ case cutoverOn:

+ return "on"

+ }

+ return "unknown"

+// cutoverStatus is what the door answers and the CLI prints. At is the

+// marker's time while draining and the activation's once on; Pending is the

+// legacy units still running.

+type cutoverStatus struct {

+ State string `json:"state"`

+ At time.Time `json:"at"`

+ Pending int `json:"pending"`

+// cutoverGate is the process's admission gate. The zero gate is not usable;

+// newCutoverGate starts it in unknown.

+type cutoverGate struct {

+ mu sync.Mutex

+ state cutoverState

+ markerSeen bool // a marker was read and decoded (or written by the door)

+ at time.Time // that marker's time

+ started time.Time // this process's start

+ lastDone time.Time // when the last before unit ended

+ admitted int // before units still running

+ drained chan struct{}

+ pageAt time.Time // the last unreadable-marker page

+ onAt time.Time // when lifetimes turned on

+ readFailed bool // the last marker read failed

+ clock func() time.Time

+func newCutoverGate(clock func() time.Time) *cutoverGate {

+ drained := make(chan struct{})

+ close(drained)

+ return &cutoverGate{clock: clock, started: clock(), drained: drained}

+// admittedUnit is one unit of record work. mode is cutoverBefore (legacy, no

+// judge) or cutoverOn (judged). A unit admitted on a console the gate does

+// not govern (host claims, no bucket) is a legacy unit no one counts.

+type admittedUnit struct {

+ g *cutoverGate

+ mode cutoverState

+ counted bool

+ once sync.Once

+ detached atomic.Bool

+// admit admits one unit at now, or refuses it with the sentence to answer.

+func (g *cutoverGate) admit(now time.Time, mutating bool) (*admittedUnit, bool, string) {

+ g.mu.Lock()

+ defer g.mu.Unlock()

+ switch g.state {

+ case cutoverBefore:

+ if g.admitted == 0 {

+ g.drained = make(chan struct{})

+ }

+ g.admitted++

+ return &admittedUnit{g: g, mode: cutoverBefore, counted: true}, true, ""

+ case cutoverDraining:

+ if mutating {

+ return nil, false, msgLifetimesDraining

+ }

+ // A plain read lands no write: it is admitted judged.

+ return &admittedUnit{g: g, mode: cutoverOn}, true, ""

+ case cutoverOn:

+ return &admittedUnit{g: g, mode: cutoverOn}, true, ""

+ }

+ return nil, false, msgLifetimesStarting

+// done ends the unit, exactly once however often it is called. The last

+// before unit to end records when, which the settle counts from.

+func (u *admittedUnit) done() {

+ if u == nil {

+ return

+ }

+ u.once.Do(func() {

+ if !u.counted {

+ return

+ }

+ g := u.g

+ now := g.clock()

+ g.mu.Lock()

+ defer g.mu.Unlock()

+ g.admitted--

+ if now.After(g.lastDone) {

+ g.lastDone = now

+ }

+ if g.admitted == 0 {

+ close(g.drained)

+ }

+ })

+// observe applies one marker read: m is the decoded marker (nil when absent)

+// and readErr a failed read or decode. on is never left.

+func (g *cutoverGate) observe(m *cutoverMarker, readErr error, now time.Time) {

+ g.mu.Lock()

+ defer g.mu.Unlock()

+ if g.state == cutoverOn {

+ return

+ }

+ g.readFailed = readErr != nil

+ switch {

+ case readErr != nil:

+ // Uncertainty closes admission but can never activate: markerSeen

+ // stays as it was, and is false unless a marker was validated.

+ if g.state != cutoverUnknown {

+ g.state = cutoverDraining

+ }

+ case m != nil:

+ g.state, g.markerSeen, g.at = cutoverDraining, true, m.At

+ case g.state == cutoverUnknown, g.state == cutoverDraining && !g.markerSeen:

+ g.state = cutoverBefore

+ }

+// activateWhenDrained turns lifetimes on when they may be, and reports

+// whether they are on.

+func (g *cutoverGate) activateWhenDrained(now time.Time) bool {

+ g.mu.Lock()

+ defer g.mu.Unlock()

+ if g.state == cutoverOn {

+ return true

+ }

+ if g.state != cutoverDraining || !g.markerSeen || g.admitted != 0 {

+ return false

+ }

+ from := g.at

+ for _, t := range []time.Time{g.started, g.lastDone} {

+ if t.After(from) {

+ from = t

+ }

+ if now.Before(from.Add(lifetimeSettle)) {

+ return false

+ }

+ g.state, g.onAt = cutoverOn, now

+ return true

+// pageDue reports whether an unreadable marker should page now: only while

+// the console is still starting, and at most once an hour.

+func (g *cutoverGate) pageDue(now time.Time) bool {

+ g.mu.Lock()

+ defer g.mu.Unlock()

+ if g.state != cutoverUnknown || !g.readFailed {

+ return false

+ }

+ if !g.pageAt.IsZero() && now.Sub(g.pageAt) < lifetimesPageEvery {

+ return false

+ }

+ g.pageAt = now

+ return true

+func (g *cutoverGate) status() cutoverStatus {

+ g.mu.Lock()

+ defer g.mu.Unlock()

+ st := cutoverStatus{State: g.state.String(), Pending: g.admitted}

+ switch {

+ case g.state == cutoverOn:

+ st.At = g.onAt

+ case g.markerSeen:

+ st.At = g.at

+ }

+ return st

+func (g *cutoverGate) current() cutoverState {

+ g.mu.Lock()

+ defer g.mu.Unlock()

+ return g.state

+// drainedCh is closed while no before unit is running.

+func (g *cutoverGate) drainedCh() <-chan struct{} {

+ g.mu.Lock()

+ defer g.mu.Unlock()

+ return g.drained

+// lifetimesGoverned reports whether the gate governs this console: a

+// claims-off console with a deployment bucket.

+func (s *Server) lifetimesGoverned() bool {

+ return !s.cfg.HostClaim && s.cfg.Releases != nil && s.cutover != nil

+// lifetimesAdmit admits one unit of record work. A console the gate does not

+// govern admits every unit as legacy work no one counts.

+func (s *Server) lifetimesAdmit(ctx context.Context, mutating bool) (*admittedUnit, bool, string) {

+ if !s.lifetimesGoverned() {

+ return &admittedUnit{mode: cutoverBefore}, true, ""

+ }

+ return s.cutover.admit(s.cutover.clock(), mutating)

+// recordCtx is the context a unit's record work runs under: the judge and

+// the record sequence for a unit admitted on, ctx unchanged otherwise.

+func (s *Server) recordCtx(ctx context.Context, u *admittedUnit) context.Context {

+ if u == nil || u.mode != cutoverOn {

+ return ctx

+ }

+ return provision.WithLifetimes(ctx, s.judge)

+// lifetimesOn reports whether this console has turned lifetimes on.

+func (s *Server) lifetimesOn() bool {

+ return s.lifetimesGoverned() && s.cutover.current() == cutoverOn

+type admittedUnitKey struct{}

+// unitFrom is the unit a request was admitted as, or nil.

+func unitFrom(ctx context.Context) *admittedUnit {

+ u, _ := ctx.Value(admittedUnitKey{}).(*admittedUnit)

+ return u

+// detachUnit hands the request's unit to a detached continuation: the

+// request's end no longer ends it, and the continuation calls done() exactly

+// once, after its last write. Called before the goroutine starts.

+func detachUnit(ctx context.Context) *admittedUnit {

+ u := unitFrom(ctx)

+ if u != nil {

+ u.detached.Store(true)

+ }

+ return u

+// lifetimesExempt are the paths that never read or write an instance record:

+// health, sign-in, SES ingest, and the cutover door itself, which must answer

+// while admission is closed.

+func lifetimesExempt(path string) bool {

+ switch {

+ case path == "/healthz", path == "/api/ses/events", path == lifetimesCutoverPath:

+ return true

+ case strings.HasPrefix(path, "/auth/"), strings.HasPrefix(path, "/static/"):

+ return true

+ }

+ return false

+// lifetimesMutating is whether a request may write: every method but a read.

+func lifetimesMutating(method string) bool {

+ switch method {

+ case http.MethodGet, http.MethodHead, http.MethodOptions:

+ return false

+ }

+ return true

+// serveAdmitted runs the mux under the request's admitted unit: refused with

+// a 503 when the gate is closed to it, ended when the handler returns unless

+// a detached continuation took it over.

+func (s *Server) serveAdmitted(w http.ResponseWriter, r *http.Request) {

+ if !s.lifetimesGoverned() || lifetimesExempt(r.URL.Path) {

+ s.mux.ServeHTTP(w, r)

+ return

+ }

+ u, ok, why := s.lifetimesAdmit(r.Context(), lifetimesMutating(r.Method))

+ if !ok {

+ if strings.HasPrefix(r.URL.Path, "/api/") {

+ jsonError(w, http.StatusServiceUnavailable, why)

+ } else {

+ http.Error(w, why, http.StatusServiceUnavailable)

+ }

+ return

+ }

+ defer func() {

+ if !u.detached.Load() {

+ u.done()

+ }

+ }()

+ ctx := context.WithValue(s.recordCtx(r.Context(), u), admittedUnitKey{}, u)

+ s.mux.ServeHTTP(w, r.WithContext(ctx))

+// lifetimesIteration is one worker iteration as a unit: admitted (mutating

+// for a pass that writes), skipped when refused or when it needs on and was

+// admitted otherwise, and ended when it returns. It reports whether run ran.

+func (s *Server) lifetimesIteration(ctx context.Context, name string, mutating, onlyOn bool, run func(context.Context)) bool {

+ u, ok, _ := s.lifetimesAdmit(ctx, mutating)

+ if !ok {

+ return false

+ }

+ defer u.done()

+ if onlyOn && u.mode != cutoverOn {

+ return false

+ }

+ rctx := s.recordCtx(ctx, u)

+ if lifetimesIterationHook != nil {

+ lifetimesIterationHook(name, rctx, u)

+ }

+ run(rctx)

+ return true

+// lifetimesWorker runs iterate now and then every interval until ctx is done.

+// The worker lifecycle rule: it never returns because of the gate's state;

+// each iteration admits for itself.

+func lifetimesWorker(ctx context.Context, every time.Duration, iterate func(context.Context)) {

+ t := time.NewTicker(every)

+ defer t.Stop()

+ for {

+ iterate(ctx)

+ select {

+ case <-ctx.Done():

+ return

+ case <-t.C:

+ }

+// readCutoverMarker reads the marker: nil with no error when it is absent; a

+// body that does not decode, or carries no time, is an error.

+func readCutoverMarker(ctx context.Context, st release.Store) (*cutoverMarker, error) {

+ body, _, err := st.Get(ctx, lifetimesCutoverKey)

+ if errors.Is(err, os.ErrNotExist) {

+ return nil, nil

+ }

+ if err != nil {

+ return nil, err

+ }

+ var m cutoverMarker

+ if err := json.Unmarshal(body, &m); err != nil {

+ return nil, fmt.Errorf("%s: %w", lifetimesCutoverKey, err)

+ }

+ if m.At.IsZero() {

+ return nil, fmt.Errorf("%s: no time", lifetimesCutoverKey)

+ }

+ return &m, nil

+// readCutoverAtStart is the first marker read, made by NewServer: a console

+// that can read its bucket starts in before or draining rather than refusing

+// every door until the watcher's first tick. A failed read leaves unknown.

+func (s *Server) readCutoverAtStart(ctx context.Context) {

+ rctx, cancel := context.WithTimeout(ctx, instanceStoreTimeout)

+ m, err := readCutoverMarker(rctx, s.cfg.Releases)

+ cancel()

+ s.cutover.observe(m, err, s.cutover.clock())

+ if err != nil {

+ log.Printf("console: lifetimes: the cutover marker could not be read at start, so every instance door refuses until it can: %v", err)

+ }

+// lifetimesWatchOnce is one read of the marker, applied, and the activation

+// check. It also runs the claims-off app materialisation once, as the first

+// admitted iteration (Task 4's startup rule): never before the gate has a

+// state.

+func (s *Server) lifetimesWatchOnce(ctx context.Context) {

+ if !s.lifetimesGoverned() {

+ return

+ }

+ g := s.cutover

+ rctx, cancel := context.WithTimeout(ctx, instanceStoreTimeout)

+ m, err := readCutoverMarker(rctx, s.cfg.Releases)

+ cancel()

+ now := g.clock()

+ was := g.current()

+ g.observe(m, err, now)

+ if err != nil && g.pageDue(now) {

+ log.Printf("console: lifetimes: %s: %v", pageLifetimesMarkerUnread, err)

+ if perr := alerting.Page(ctx, alerting.LevelIncident, pageLifetimesMarkerUnread, err.Error()); perr != nil {

+ log.Printf("console: lifetimes: page: %v", perr)

+ }

+ g.activateWhenDrained(now)

+ if is := g.current(); is != was {

+ if err != nil {

+ log.Printf("console: lifetimes: %s -> %s (the marker could not be read: %v)", was, is, err)

+ } else {

+ log.Printf("console: lifetimes: %s -> %s", was, is)

+ }

+ s.materializeAppsOnce(ctx)

+// materializeAppsOnce converges app rows onto the bucket in the first

+// admitted iteration, as NewServer did before the gate (once, logged on

+// failure). It waits while the gate is unknown or draining.

+func (s *Server) materializeAppsOnce(ctx context.Context) {

+ if s.appsMaterialized.Load() {

+ return

+ }

+ s.lifetimesIteration(ctx, "materialize apps", true, false, func(ctx context.Context) {

+ s.appsMaterialized.Store(true)

+ if err := MaterializeApps(ctx, s.cfg.Releases, s.store); err != nil {

+ log.Printf("console: could not converge app rows with the bucket: %v", err)

+ }

+ })

+// StartLifetimesCutoverWatch reads the marker now and every 10 seconds for

+// the life of the process. A host-claim console has no lifetimes: it logs

+// that and returns.

+func (s *Server) StartLifetimesCutoverWatch(ctx context.Context) {

+ if s.cfg.HostClaim {

+ log.Printf("console: lifetimes cutover watch: off on a host-claim console, where lifetimes are not for it")

+ return

+ }

+ if !s.lifetimesGoverned() {

+ return

+ }

+ lifetimesWorker(ctx, lifetimesWatchInterval, s.lifetimesWatchOnce)

+// handleAPILifetimesCutover is `carlos system lifetimes cutover`: an

+// operator's own CLI token (else the move family's 404). It writes the marker

+// create-only and closes admission before it answers; run again, it answers

+// the status, so the CLI polls it until on.

+func (s *Server) handleAPILifetimesCutover(w http.ResponseWriter, r *http.Request) {

+ addr, ok := s.moveOperator(r)

+ if !ok || s.cfg.Releases == nil {

+ http.NotFound(w, r)

+ return

+ }

+ if s.cfg.HostClaim {

+ jsonError(w, http.StatusConflict, msgLifetimesClaimsOn)

+ return

+ }

+ g := s.cutover

+ now := g.clock()

+ m := cutoverMarker{At: now.UTC(), By: addr}

+ body, err := json.Marshal(m)

+ if err != nil {

+ jsonError(w, http.StatusInternalServerError, msgLifetimesMarkerUnwritten)

+ return

+ }

+ ctx, cancel := context.WithTimeout(r.Context(), instanceStoreTimeout)

+ defer cancel()

+ _, err = s.cfg.Releases.Put(ctx, lifetimesCutoverKey, body, "")

+ switch {

+ case err == nil:

+ g.observe(&m, nil, now)

+ log.Printf("console: lifetimes: the cutover marker is written (by %s); admission is closed to legacy work", addr)

+ s.auditSystem(addr, "lifetimes-cutover", "switched instance lifetimes on (cli)")

+ case errors.Is(err, release.ErrCASConflict):

+ existing, rerr := readCutoverMarker(ctx, s.cfg.Releases)

+ if rerr != nil || existing == nil {

+ if rerr != nil {

+ g.observe(nil, rerr, now)

+ }

+ log.Printf("console: lifetimes: the cutover marker exists but cannot be read: %v", rerr)

+ jsonError(w, http.StatusBadGateway, msgLifetimesMarkerUnreadable)

+ return

+ }

+ g.observe(existing, nil, now)

+ default:

+ // The write may have landed: admission closes on the uncertainty,

+ // and only a validated read of the marker can activate.

+ g.observe(nil, err, now)

+ log.Printf("console: lifetimes: write the cutover marker: %v", err)

+ jsonError(w, http.StatusBadGateway, msgLifetimesMarkerUnwritten)

+ return

+ }

+ g.activateWhenDrained(now)

+ writeJSON(w, http.StatusOK, g.status())

diff --git a/internal/console/lifetimes_cutover_test.go b/internal/console/lifetimes_cutover_test.go

new file mode 100644

index 000000000..2fa5def86

--- /dev/null

+++ b/internal/console/lifetimes_cutover_test.go

@@ -0,0 +1,921 @@

+package console

+import (

+ "bytes"

+ "context"

+ "encoding/json"

+ "errors"

+ "net/http"

+ "net/http/httptest"

+ "path/filepath"

+ "strings"

+ "sync"

+ "sync/atomic"

+ "testing"

+ "time"

+ "github.com/carlosframework/platform/internal/provision"

+ "github.com/carlosframework/platform/internal/release"

+ "github.com/carlosframework/platform/internal/seeds"

+// ---- fixtures -------------------------------------------------------------

+// lifetimesClock is the gate's controllable clock.

+type lifetimesClock struct {

+ mu sync.Mutex

+ t time.Time

+func (c *lifetimesClock) Now() time.Time {

+ c.mu.Lock()

+ defer c.mu.Unlock()

+ return c.t

+func (c *lifetimesClock) Advance(d time.Duration) time.Time {

+ c.mu.Lock()

+ defer c.mu.Unlock()

+ c.t = c.t.Add(d)

+ return c.t

+// useLifetimesClock points the gate's clock at a fixed time for every server

+// built after it, until the test ends.

+func useLifetimesClock(t *testing.T) *lifetimesClock {

+ t.Helper()

+ c := &lifetimesClock{t: time.Date(2026, 10, 22, 12, 0, 0, 0, time.UTC)}

+ old := lifetimesNow

+ lifetimesNow = c.Now

+ t.Cleanup(func() { lifetimesNow = old })

+ return c

+const (

+ ltApp = "shop"

+ ltOne = "one.shop.example.com"

+ ltOld = "old.shop.example.com"

+ ltFresh = "new.shop.example.com"

+// lifetimesFx is a claims-off console with the app shop, instances enabled,

+// a member's token and an operator's.

+type lifetimesFx struct {

+ fx *consoleFixture

+ clock *lifetimesClock

+ member string

+ operator string

+func newLifetimesFx(t *testing.T) lifetimesFx {

+ t.Helper()

+ clock := useLifetimesClock(t)

+ fx := newConsoleFixture(t)

+ now := time.Now()

+ if _, err := fx.st.CreateAppIn(OpsAccountID, ltApp, "paul@keymail.dev", now); err != nil {

+ t.Fatal(err)

+ }

+ if err := fx.st.AddMember(OpsAccountID, "alice@keymail.dev", RoleMember, now); err != nil {

+ t.Fatal(err)

+ }

+ enableInstances(t, fx, ltApp, provision.Manifest{Schema: 1, App: ltApp, Domain: "shop.example.com", Channel: "stable"})

+ member := cliToken(t, fx, "alice@keymail.dev", "code-lifetimes-member")

+ if err := fx.st.SetOperator("paul@keymail.dev", true); err != nil {

+ t.Fatal(err)

+ }

+ operator := cliToken(t, fx, "paul@keymail.dev", "code-lifetimes-operator")

+ if got := fx.srv.cutover.status().State; got != "before" {

+ t.Fatalf("a console over a bucket with no marker starts %s, want before", got)

+ }

+ return lifetimesFx{fx: fx, clock: clock, member: member, operator: operator}

+// cutover calls the door as token.

+func (l lifetimesFx) cutover(t *testing.T, token string) (int, cutoverStatus, string) {

+ t.Helper()

+ code, _, body := rawReq(t, http.MethodPost, l.fx.ts.URL+lifetimesCutoverPath, token, "")

+ var st cutoverStatus

+ _ = json.Unmarshal([]byte(body), &st)

+ return code, st, body

+// legacyRecord writes a fieldless record of shop for host, as main does.

+func legacyRecord(t *testing.T, st release.Store, host, channel string) {

+ t.Helper()

+ rec := provision.Record{Schema: 1, Host: host, Slug: strings.SplitN(host, ".", 2)[0], App: ltApp,

+ Account: release.OpsAccount, Channel: channel, CreatedAt: time.Now().UTC()}

+ if err := provision.CreateRecord(context.Background(), st, rec); err != nil {

+ t.Fatal(err)

+ }

+// retiredSlot retires host's lifetime: its legacy body is now obsolete.

+func retiredSlot(t *testing.T, st release.Store, host string) {

+ t.Helper()

+ slot := provision.HostIncarnation{Incarnation: 1, State: provision.SlotRetired, Account: release.OpsAccount, App: ltApp}

+ if _, err := provision.WriteHostIncarnation(context.Background(), st, host, slot, ""); err != nil {

+ t.Fatal(err)

+ }

+func rawRecord(t *testing.T, st release.Store, host string) []byte {

+ t.Helper()

+ b, _, err := st.Get(context.Background(), provision.RecordKey(release.OpsAccount, ltApp, host))

+ if err != nil {

+ t.Fatal(err)

+ }

+ return b

+// bucketState is every key in st and its ETag.

+func cutoverBucket(t *testing.T, st release.Store) map[string]string {

+ t.Helper()

+ keys, err := st.List(context.Background(), "")

+ if err != nil {

+ t.Fatal(err)

+ }

+ out := map[string]string{}

+ for _, k := range keys {

+ _, etag, err := st.Get(context.Background(), k)

+ if err != nil {

+ t.Fatal(err)

+ }

+ out[k] = etag

+ }

+ return out

+func cutoverErrorOf(body string) string {

+ var e struct {

+ Error string `json:"error"`

+ }

+ _ = json.Unmarshal([]byte(body), &e)

+ return e.Error

+// markerFaultStore fails every read of the marker while fail is set.

+type markerFaultStore struct {

+ release.Store

+ fail atomic.Bool

+func (m *markerFaultStore) Get(ctx context.Context, key string) ([]byte, string, error) {

+ if key == lifetimesCutoverKey && m.fail.Load() {

+ return nil, "", errors.New("the bucket did not answer")

+ }

+ return m.Store.Get(ctx, key)

+func (m *markerFaultStore) DeleteIfMatch(ctx context.Context, key, etag string) error {

+ return release.DeleteIfMatch(ctx, m.Store, key, etag)

+// putPauses holds the nth Put of a key (1-based) in the fixture's bucket

+// until released, through deletableStore's putHook.

+type putPause struct {

+ nth int32

+ n atomic.Int32

+ entered chan struct{}

+ release chan struct{}

+func pausePuts(t *testing.T, fx *consoleFixture, pauses map[string]*putPause) {

+ t.Helper()

+ ds, ok := fx.rel.(*deletableStore)

+ if !ok {

+ t.Fatal("the fixture's bucket is not a deletableStore")

+ }

+ for _, p := range pauses {

+ p.entered, p.release = make(chan struct{}), make(chan struct{})

+ }

+ ds.putHook = func(_ context.Context, key string) {

+ p := pauses[key]

+ if p == nil || p.n.Add(1) != p.nth {

+ return

+ }

+ close(p.entered)

+ <-p.release

+ }

+ t.Cleanup(func() {

+ for _, p := range pauses {

+ select {

+ case <-p.release:

+ default:

+ close(p.release)

+ }

+ })

+func waitFor(t *testing.T, ch <-chan struct{}, what string) {

+ t.Helper()

+ select {

+ case <-ch:

+ case <-time.After(10 * time.Second):

+ t.Fatalf("timed out waiting for %s", what)

+ }

+// ---- the door -------------------------------------------------------------

+func TestCutoverDoorIsOperatorOnlyAndIdempotent(t *testing.T) {

+ l := newLifetimesFx(t)

+ want := canonicalNotFound(t)

+ status, h, body := rawReq(t, http.MethodPost, l.fx.ts.URL+lifetimesCutoverPath, l.member, "")

+ assertNotFound(t, "the cutover door for a member", want, status, h, body)

+ if _, _, err := l.fx.rel.Get(context.Background(), lifetimesCutoverKey); err == nil {

+ t.Fatal("a member's refused call wrote the marker")

+ }

+ at := l.clock.Now()

+ code, st, body := l.cutover(t, l.operator)

+ if code != http.StatusOK || st.State != "draining" || !st.At.Equal(at) || st.Pending != 0 {

+ t.Fatalf("the operator's cutover = %d %s, want 200 draining at %v", code, body, at)

+ }

+ raw, etag, err := l.fx.rel.Get(context.Background(), lifetimesCutoverKey)

+ if err != nil {

+ t.Fatal(err)

+ }

+ var m cutoverMarker

+ if err := json.Unmarshal(raw, &m); err != nil || !m.At.Equal(at) || m.By != "paul@keymail.dev" {

+ t.Fatalf("marker = %s (%v), want at %v by paul", raw, err, at)

+ }

+ l.clock.Advance(5 * time.Second)

+ code, st, body = l.cutover(t, l.operator)

+ if code != http.StatusOK || st.State != "draining" || !st.At.Equal(at) {

+ t.Fatalf("the second call = %d %s, want the same draining status", code, body)

+ }

+ if _, again, _ := l.fx.rel.Get(context.Background(), lifetimesCutoverKey); again != etag {

+ t.Fatal("the second call rewrote the marker")

+ }

+ events, err := l.fx.st.Events(OpsAccountID, "platform", 50)

+ if err != nil {

+ t.Fatal(err)

+ }

+ var rows []string

+ for _, e := range events {

+ if e.Action == "lifetimes-cutover" {

+ rows = append(rows, e.Detail+" | "+e.Address)

+ }

+ if len(rows) != 1 || rows[0] != "switched instance lifetimes on (cli) | paul@keymail.dev" {

+ t.Fatalf("audit rows = %q, want one", rows)

+ }

+ l.fx.srv.cfg.HostClaim = true

+ code, _, body = l.cutover(t, l.operator)

+ l.fx.srv.cfg.HostClaim = false

+ if code != http.StatusConflict || cutoverErrorOf(body) != msgLifetimesClaimsOn {

+ t.Fatalf("on a host-claim console = %d %s, want 409", code, body)

+ }

+ // A marker that exists and cannot be read is never a success.

+ u := newLifetimesFx(t)

+ if _, err := u.fx.rel.Put(context.Background(), lifetimesCutoverKey, []byte("{not json"), ""); err != nil {

+ t.Fatal(err)

+ }

+ code, _, body = u.cutover(t, u.operator)

+ if code != http.StatusBadGateway || cutoverErrorOf(body) != msgLifetimesMarkerUnreadable {

+ t.Fatalf("an unreadable marker = %d %s, want 502", code, body)

+ }

+ u.clock.Advance(10 * time.Minute)

+ u.fx.srv.lifetimesWatchOnce(context.Background())

+ if got := u.fx.srv.cutover.status().State; got != "draining" {

+ t.Fatalf("with an unreadable marker the state is %s, want draining (and never on)", got)

+ }

+// ---- the states -----------------------------------------------------------

+func TestCutoverStates(t *testing.T) {

+ t.Run("unknown at start refuses every door", func(t *testing.T) {

+ useLifetimesClock(t)

+ st, err := Open(filepath.Join(t.TempDir(), "console.db"))

+ if err != nil {

+ t.Fatal(err)

+ }

+ t.Cleanup(func() { st.Close() })

+ rel := &markerFaultStore{Store: release.NewDirStore(t.TempDir())}

+ rel.fail.Store(true)

+ srv, err := NewServer(Config{Releases: rel}, st)

+ if err != nil {

+ t.Fatal(err)

+ }

+ ts := httptest.NewServer(srv)

+ t.Cleanup(ts.Close)

+ for _, c := range []struct{ method, path string }{

+ {http.MethodGet, "/api/cli/apps/bab/shop/instances"},

+ {http.MethodGet, "/api/cli/apps/bab/shop/instances/" + ltOne},

+ {http.MethodPost, "/api/cli/apps/bab/shop/instances"},

+ {http.MethodDelete, "/api/cli/apps/bab/shop/instances/" + ltOne},

+ } {

+ code, _, body := rawReq(t, c.method, ts.URL+c.path, "", "")

+ if code != http.StatusServiceUnavailable || cutoverErrorOf(body) != msgLifetimesStarting {

+ t.Errorf("%s %s while unknown = %d %s, want 503 starting", c.method, c.path, code, body)

+ }

+ code, _, body := rawReq(t, http.MethodGet, ts.URL+"/a/bab/apps/shop", "", "")

+ if code != http.StatusServiceUnavailable || strings.TrimSpace(body) != msgLifetimesStarting {

+ t.Errorf("a page while unknown = %d %q, want 503 starting", code, body)

+ }

+ srv.lifetimesWatchOnce(context.Background())

+ if got := srv.cutover.status().State; got != "unknown" {

+ t.Fatalf("a failed read leaves %s, want unknown", got)

+ }

+ rel.fail.Store(false)

+ srv.lifetimesWatchOnce(context.Background())

+ if got := srv.cutover.status().State; got != "before" {

+ t.Fatalf("an absent marker read = %s, want before", got)

+ }

+ if code, _, _ := rawReq(t, http.MethodGet, ts.URL+"/api/cli/apps/bab/shop/instances", "", ""); code != http.StatusUnauthorized {

+ t.Fatalf("before, a request reaches its handler: got %d, want its 401", code)

+ }

+ })

+ t.Run("marker, drain, settle, latched", func(t *testing.T) {

+ l := newLifetimesFx(t)

+ at := l.clock.Now()

+ if code, st, body := l.cutover(t, l.operator); code != http.StatusOK || st.State != "draining" {

+ t.Fatalf("cutover = %d %s", code, body)

+ }

+ code, _, body := rawReq(t, http.MethodPost, instancesURL(l.fx, "bab", ltApp), l.member, `{"host":"new.shop.example.com"}`)

+ if code != http.StatusServiceUnavailable || cutoverErrorOf(body) != msgLifetimesDraining {

+ t.Fatalf("a create once the marker is written = %d %s, want 503 draining", code, body)

+ }

+ l.clock.Advance(lifetimeSettle - time.Second)

+ l.fx.srv.lifetimesWatchOnce(context.Background())

+ if got := l.fx.srv.cutover.status().State; got != "draining" {

+ t.Fatalf("59 s after the marker = %s, want draining", got)

+ }

+ l.clock.Advance(time.Second)

+ l.fx.srv.lifetimesWatchOnce(context.Background())

+ st := l.fx.srv.cutover.status()

+ if st.State != "on" || !st.At.Equal(at.Add(lifetimeSettle)) || !l.fx.srv.lifetimesOn() {

+ t.Fatalf("60 s after the marker = %+v, want on", st)

+ }

+ // Later, a failed read and a corrupt marker: on is kept, never left

+ // and re-entered (its time stands).

+ l.clock.Advance(10 * time.Minute)

+ fault := &markerFaultStore{Store: l.fx.rel}

+ fault.fail.Store(true)

+ l.fx.srv.cfg.Releases = fault

+ l.fx.srv.lifetimesWatchOnce(context.Background())

+ l.fx.srv.cfg.Releases = l.fx.rel

+ if _, err := l.fx.rel.Put(context.Background(), lifetimesCutoverKey, []byte("corrupt"), markerETag(t, l.fx.rel, lifetimesCutoverKey)); err != nil {

+ t.Fatal(err)

+ }

+ l.fx.srv.lifetimesWatchOnce(context.Background())

+ if after := l.fx.srv.cutover.status(); after.State != "on" || !after.At.Equal(st.At) {

+ t.Fatalf("after a failed read and a corrupt marker = %+v, want on since %v (latched)", after, st.At)

+ }

+ })

+ t.Run("a restart into an existing marker settles from its own start", func(t *testing.T) {

+ for name, age := range map[string]time.Duration{"older than the settle": 10 * time.Minute, "10 s old": 10 * time.Second} {

+ t.Run(name, func(t *testing.T) {

+ l := newLifetimesFx(t)

+ body, _ := json.Marshal(cutoverMarker{At: l.clock.Now().Add(-age), By: "paul@keymail.dev"})

+ if _, err := l.fx.rel.Put(context.Background(), lifetimesCutoverKey, body, ""); err != nil {

+ t.Fatal(err)

+ }

+ start := l.clock.Now()

+ srv, err := NewServer(l.fx.srv.cfg, l.fx.st)

+ if err != nil {

+ t.Fatal(err)

+ }

+ if st := srv.cutover.status(); st.State != "draining" || st.Pending != 0 {

+ t.Fatalf("a restart into a marker = %+v, want draining with nothing pending", st)

+ }

+ l.clock.Advance(lifetimeSettle - time.Second)

+ srv.lifetimesWatchOnce(context.Background())

+ if got := srv.cutover.status().State; got != "draining" {

+ t.Fatalf("59 s after the start = %s, want draining", got)

+ }

+ l.clock.Advance(time.Second)

+ srv.lifetimesWatchOnce(context.Background())

+ if st := srv.cutover.status(); st.State != "on" || !st.At.Equal(start.Add(lifetimeSettle)) {

+ t.Fatalf("60 s after the start = %+v, want on", st)

+ }

+ })

+ }

+ })

+func markerETag(t *testing.T, st release.Store, key string) string {

+ t.Helper()

+ _, etag, err := st.Get(context.Background(), key)

+ if err != nil {

+ t.Fatal(err)

+ }

+ return etag

+func TestActivationRequiresAValidatedMarker(t *testing.T) {

+ clock := &lifetimesClock{t: time.Date(2026, 10, 22, 12, 0, 0, 0, time.UTC)}

+ g := newCutoverGate(clock.Now)

+ g.observe(nil, nil, clock.Now())

+ g.observe(nil, errors.New("timeout"), clock.Now())

+ if g.current() != cutoverDraining {

+ t.Fatalf("a read failure from before = %s, want draining", g.current())

+ }

+ clock.Advance(2 * lifetimeSettle)

+ if g.activateWhenDrained(clock.Now()) || g.current() != cutoverDraining {

+ t.Fatalf("no units, 120 s, no validated marker = %s, want draining, not on", g.current())

+ }

+ if _, ok, why := g.admit(clock.Now(), true); ok || why != msgLifetimesDraining {

+ t.Fatalf("a mutation while draining by uncertainty = %v %q", ok, why)

+ }

+ g.observe(nil, nil, clock.Now())

+ if g.current() != cutoverBefore {

+ t.Fatalf("a successful absent read = %s, want before", g.current())

+ }

+ at := clock.Now()

+ g.observe(&cutoverMarker{At: at, By: "paul@keymail.dev"}, nil, at)

+ if g.current() != cutoverDraining {

+ t.Fatalf("a present read = %s, want draining", g.current())

+ }

+ clock.Advance(lifetimeSettle - time.Second)

+ if g.activateWhenDrained(clock.Now()) {

+ t.Fatal("on before the settle")

+ }

+ clock.Advance(time.Second)

+ if !g.activateWhenDrained(clock.Now()) {

+ t.Fatalf("not on after the settle: %s", g.current())

+ }

+ // A restart into a 10-minute-old marker: draining, then on 60 s after

+ // the process's start, not at once.

+ r := newCutoverGate(clock.Now)

+ start := clock.Now()

+ r.observe(&cutoverMarker{At: start.Add(-10 * time.Minute)}, nil, start)

+ if r.current() != cutoverDraining || r.activateWhenDrained(start) {

+ t.Fatalf("a restart into an old marker = %s, want draining", r.current())

+ }

+ if r.activateWhenDrained(start.Add(lifetimeSettle - time.Nanosecond)) {

+ t.Fatal("on before 60 s after the start")

+ }

+ if !r.activateWhenDrained(start.Add(lifetimeSettle)) {

+ t.Fatal("not on 60 s after the start")

+ }

+// ---- admission ------------------------------------------------------------

+func TestDrainingRefusesEveryClaimsOffMutation(t *testing.T) {

+ l := newLifetimesFx(t)

+ legacyRecord(t, l.fx.rel, ltOne, "edge")

+ legacyRecord(t, l.fx.rel, ltOld, "edge")

+ retiredSlot(t, l.fx.rel, ltOld)

+ base := instancesURL(l.fx, "bab", ltApp)

+ // Before the cutover a read is legacy: the obsolete body still shows.

+ if code, _, body := rawReq(t, http.MethodGet, base+"/"+ltOld, l.member, ""); code != http.StatusOK {

+ t.Fatalf("before, the legacy read = %d %s, want 200", code, body)

+ }

+ if code, _, body := l.cutover(t, l.operator); code != http.StatusOK {

+ t.Fatalf("cutover = %d %s", code, body)

+ }

+ before := cutoverBucket(t, l.fx.rel)

+ for _, c := range []struct{ name, method, url, body string }{

+ {"create", http.MethodPost, base, `{"host":"new.shop.example.com"}`},

+ {"update", http.MethodPatch, base + "/" + ltOne, `{"upstreams":[]}`},

+ {"delete", http.MethodDelete, base + "/" + ltOne, ""},

+ {"channel", http.MethodPost, base + "/" + ltOne + "/channel", `{"channel":"beta"}`},

+ {"updates", http.MethodPut, base + "/" + ltOne + "/updates", `{"auto":true}`},

+ {"import upload", http.MethodPut, base + "/" + ltOne + "/import?seed=x", "x"},

+ {"run", http.MethodPost, base + "/" + ltOne + "/runs", `{}`},

+ {"trash the app", http.MethodPost, l.fx.ts.URL + "/api/cli/apps/bab/shop/delete", `{}`},

+ {"browser restart", http.MethodPost, l.fx.u("/apps/shop/instances/" + ltOne + "/restart"), ""},

+ {"browser detach", http.MethodPost, l.fx.u("/apps/shop/domains/detach"), ""},

+ } {

+ t.Run(c.name, func(t *testing.T) {

+ code, _, body := rawReq(t, c.method, c.url, l.member, c.body)

+ msg := cutoverErrorOf(body)

+ if !strings.Contains(c.url, "/api/") {

+ msg = strings.TrimSpace(body)

+ }

+ if code != http.StatusServiceUnavailable || msg != msgLifetimesDraining {

+ t.Fatalf("= %d %s, want 503 draining", code, body)

+ }

+ })

+ }

+ if after := cutoverBucket(t, l.fx.rel); len(after) != len(before) {

+ t.Fatalf("the refused doors changed the bucket: %d keys, then %d", len(before), len(after))

+ } else {

+ for k, e := range before {

+ if after[k] != e {

+ t.Fatalf("the refused doors rewrote %s", k)

+ }

+ // GETs answer, judged: the obsolete body is absent, the valid one reads.

+ if code, _, body := rawReq(t, http.MethodGet, base+"/"+ltOne, l.member, ""); code != http.StatusOK {

+ t.Fatalf("GET one = %d %s", code, body)

+ }

+ if code, _, body := rawReq(t, http.MethodGet, base+"/"+ltOld, l.member, ""); code != http.StatusNotFound {

+ t.Fatalf("GET of the obsolete body while draining = %d %s, want 404 (judged)", code, body)

+ }

+ requireListing(t, l, ltOne)

+func requireListing(t *testing.T, l lifetimesFx, hosts ...string) {

+ t.Helper()

+ code, _, body := rawReq(t, http.MethodGet, instancesURL(l.fx, "bab", ltApp), l.member, "")

+ if code != http.StatusOK {

+ t.Fatalf("list = %d %s", code, body)

+ }

+ var list struct {

+ Instances []provision.Record `json:"instances"`

+ }

+ if err := json.Unmarshal([]byte(body), &list); err != nil {

+ t.Fatal(err)

+ }

+ var got []string

+ for _, r := range list.Instances {

+ got = append(got, r.Host)

+ }

+ if strings.Join(got, ",") != strings.Join(hosts, ",") {

+ t.Fatalf("list = %v, want %v", got, hosts)

+ }

+func TestActivationWaitsForAdmittedWork(t *testing.T) {

+ clock := useLifetimesClock(t)

+ f := newImportFixture(t)

+ if err := f.fx.st.SetOperator("paul@keymail.dev", true); err != nil {

+ t.Fatal(err)

+ }

+ l := lifetimesFx{fx: f.fx, clock: clock, member: f.token, operator: cliToken(t, f.fx, "paul@keymail.dev", "code-lifetimes-op")}

+ const chanHost = "chan.shop.example.com"

+ legacyRecord(t, f.fx.rel, chanHost, "edge")

+ body, _ := sqliteBytes(t)

+ begun := f.begun(t, importHost, body)

+ // Two units admitted before the marker: the channel door, held before

+ // its record write, and the import's detached continuation, held in its

+ // check.

+ pauses := map[string]*putPause{provision.RecordKey("bab", fromApp, chanHost): {nth: 1}}

+ pausePuts(t, f.fx, pauses)

+ checking, proceed := make(chan struct{}), make(chan struct{})

+ letCheckRun := sync.OnceFunc(func() { close(proceed) })

+ old := checkSeedDBConsole

+ t.Cleanup(func() { checkSeedDBConsole = old })

+ t.Cleanup(letCheckRun) // a failed assertion must not strand the continuation

+ var once sync.Once

+ checkSeedDBConsole = func(ctx context.Context, path string) error {

+ once.Do(func() { close(checking) })

+ <-proceed

+ return old(ctx, path)

+ }

+ moved := make(chan int, 1)

+ go func() {

+ code, _, _ := rawReq(t, http.MethodPost, channelURL(f.fx, "bab", fromApp, chanHost), f.token, `{"channel":"beta"}`)

+ moved <- code

+ }()

+ waitFor(t, pauses[provision.RecordKey("bab", fromApp, chanHost)].entered, "the channel door's record write")

+ if code, out := uploadImport(t, f, importHost, body); code != http.StatusAccepted {

+ t.Fatalf("upload = %d %v", code, out)

+ }

+ waitFor(t, checking, "the import's check")

+ at := clock.Now()

+ code, st, out := l.cutover(t, l.operator)

+ if code != http.StatusOK || st.State != "draining" || st.Pending != 2 {

+ t.Fatalf("cutover = %d %s, want draining with 2 pending", code, out)

+ }

+ clock.Advance(time.Second)

+ if code, _, out := rawReq(t, http.MethodPost, f.url, f.token, `{"host":"late.shop.example.com"}`); code != http.StatusServiceUnavailable || cutoverErrorOf(out) != msgLifetimesDraining {

+ t.Fatalf("a create at At + 1 s = %d %s, want 503 draining", code, out)

+ }

+ clock.Advance(lifetimeSettle)

+ f.fx.srv.lifetimesWatchOnce(context.Background())

+ if st := f.fx.srv.cutover.status(); st.State != "draining" || st.Pending != 2 || !st.At.Equal(at) {

+ t.Fatalf("61 s after the marker with both running = %+v, want draining, 2 pending", st)

+ }

+ close(pauses[provision.RecordKey("bab", fromApp, chanHost)].release)

+ if code := <-moved; code != http.StatusOK {

+ t.Fatalf("the channel move = %d, want 200", code)

+ }

+ if got := recordChannel(t, f.fx, fromApp, chanHost); got != "beta" {

+ t.Fatalf("channel = %q, want the legacy write to land", got)

+ }

+ if raw := rawRecord(t, f.fx.rel, chanHost); bytes.Contains(raw, []byte("record_seq")) {

+ t.Fatalf("the channel write carries a record sequence; a before unit writes legacy: %s", raw)

+ }

+ if st := f.fx.srv.cutover.status(); st.Pending != 1 {

+ t.Fatalf("after the channel door = %+v, want 1 pending", st)

+ }

+ letCheckRun()

+ f.fx.srv.seedCopies.Wait()

+ rec, err := f.record(t, importHost)

+ if err != nil {

+ t.Fatal(err)

+ }

+ requireImportState(t, rec, begun.Seed.ID, seeds.StateUploaded, "")

+ if raw := rawRecord(t, f.fx.rel, importHost); bytes.Contains(raw, []byte("record_seq")) {

+ t.Fatalf("the import's write carries a record sequence: %s", raw)

+ }

+ last := clock.Now()

+ if st := f.fx.srv.cutover.status(); st.Pending != 0 || st.State != "draining" {

+ t.Fatalf("both finished = %+v, want draining with none pending", st)

+ }

+ clock.Advance(lifetimeSettle - time.Second)

+ f.fx.srv.lifetimesWatchOnce(context.Background())

+ if f.fx.srv.lifetimesOn() {

+ t.Fatal("on before the settle after the last legacy unit")

+ }

+ clock.Advance(time.Second)

+ f.fx.srv.lifetimesWatchOnce(context.Background())

+ if st := f.fx.srv.cutover.status(); st.State != "on" || !st.At.Equal(last.Add(lifetimeSettle)) {

+ t.Fatalf("60 s after the last legacy unit = %+v, want on", st)

+ }

+func TestNoUnitChangesModeMidFlight(t *testing.T) {

+ l := newLifetimesFx(t)

+ s := l.fx.srv

+ ctx := context.Background()

+ early, ok, _ := s.lifetimesAdmit(ctx, true)

+ if !ok || early.mode != cutoverBefore {

+ t.Fatalf("admitted before the marker: %+v %v", early, ok)

+ }

+ earlyCtx := s.recordCtx(ctx, early)

+ if code, _, body := l.cutover(t, l.operator); code != http.StatusOK {

+ t.Fatalf("cutover = %d %s", code, body)

+ }

+ read, ok, _ := s.lifetimesAdmit(ctx, false)

+ if !ok || provision.JudgeFrom(s.recordCtx(ctx, read)) == nil {

+ t.Fatal("a read admitted while draining is not judged")

+ }

+ read.done()

+ l.clock.Advance(10 * time.Minute)

+ s.lifetimesWatchOnce(ctx)

+ if s.lifetimesOn() {

+ t.Fatal("on while a unit admitted before the marker runs")

+ }

+ if provision.JudgeFrom(earlyCtx) != nil || provision.JudgeFrom(s.recordCtx(ctx, early)) != nil {

+ t.Fatal("the unit admitted before the marker gained a judge")

+ }

+ early.done()

+ early.done() // exactly one end, however often called

+ if st := s.cutover.status(); st.Pending != 0 {

+ t.Fatalf("pending = %d after one unit ended twice", st.Pending)

+ }

+ l.clock.Advance(lifetimeSettle)

+ s.lifetimesWatchOnce(ctx)

+ late, ok, _ := s.lifetimesAdmit(ctx, true)

+ if !ok || late.mode != cutoverOn || provision.JudgeFrom(s.recordCtx(ctx, late)) == nil {

+ t.Fatalf("a unit admitted on = %+v %v, want the judge", late, ok)

+ }

+ late.done()

+ if provision.JudgeFrom(s.recordCtx(ctx, early)) != nil {

+ t.Fatal("the before unit's context changed mode after activation")

+ }

+func TestWorkersLoopAcrossTheCutover(t *testing.T) {

+ l := newLifetimesFx(t)

+ s := l.fx.srv

+ ctx, cancel := context.WithCancel(context.Background())

+ type seen struct {

+ judged bool

+ mode cutoverState

+ }

+ var mu sync.Mutex

+ var sweeps []seen

+ sweepHeld, sweepGo := make(chan struct{}), make(chan struct{})

+ lifetimesIterationHook = func(name string, ictx context.Context, u *admittedUnit) {

+ if name != "seeds pass" {

+ return

+ }

+ mu.Lock()

+ sweeps = append(sweeps, seen{provision.JudgeFrom(ictx) != nil, u.mode})

+ first := len(sweeps) == 1

+ mu.Unlock()

+ if first {

+ close(sweepHeld)

+ <-sweepGo

+ }

+ // The recovery and migration passes of Tasks 3 and 5 run only on: two

+ // stand-in passes, started before the marker on the real loop.

+ var iterations atomic.Int32

+ ran := make(chan bool, 64)

+ var wg sync.WaitGroup

+ for _, name := range []string{"recovery pass", "migration pass"} {

+ wg.Add(1)

+ go func() {

+ defer wg.Done()

+ lifetimesWorker(ctx, time.Millisecond, func(ctx context.Context) {

+ iterations.Add(1)

+ s.lifetimesIteration(ctx, name, true, true, func(ctx context.Context) { ran <- provision.JudgeFrom(ctx) != nil })

+ })

+ }()

+ }

+ t.Cleanup(func() {

+ cancel()

+ wg.Wait()

+ lifetimesIterationHook = nil

+ })

+ sweepDone := make(chan struct{})

+ go func() { s.seedsIteration(ctx); close(sweepDone) }()

+ waitFor(t, sweepHeld, "the first seed sweep iteration")

+ if code, st, body := l.cutover(t, l.operator); code != http.StatusOK || st.Pending < 1 {

+ t.Fatalf("cutover = %d %s, want draining with the sweep pending", code, body)

+ }

+ for iterations.Load() < 20 {

+ time.Sleep(time.Millisecond)

+ }

+ select {

+ case <-ran:

+ t.Fatal("a pass that needs on ran before activation")

+ default:

+ }

+ close(sweepGo)

+ waitFor(t, sweepDone, "the first sweep to finish")

+ l.clock.Advance(lifetimeSettle)

+ s.lifetimesWatchOnce(context.Background())

+ if !s.lifetimesOn() {

+ t.Fatalf("not on: %+v", s.cutover.status())

+ }

+ for i := 0; i < 2; i++ {

+ select {

+ case judged := <-ran:

+ if !judged {

+ t.Fatal("a pass ran on without the judge")

+ }

+ case <-time.After(10 * time.Second):

+ t.Fatal("the passes did not run after activation without a restart")

+ }

+ s.seedsIteration(ctx)

+ mu.Lock()

+ defer mu.Unlock()

+ if len(sweeps) != 2 || sweeps[0].judged || sweeps[0].mode != cutoverBefore || !sweeps[1].judged || sweeps[1].mode != cutoverOn {

+ t.Fatalf("seed sweeps = %+v, want a legacy one then a judged one", sweeps)

+ }

+func TestMarkerFailureAfterActivationKeepsJudgedReads(t *testing.T) {

+ l := newLifetimesFx(t)

+ legacyRecord(t, l.fx.rel, ltOne, "edge")

+ legacyRecord(t, l.fx.rel, ltOld, "edge")

+ retiredSlot(t, l.fx.rel, ltOld)

+ if code, _, body := l.cutover(t, l.operator); code != http.StatusOK {

+ t.Fatalf("cutover = %d %s", code, body)

+ }

+ l.clock.Advance(lifetimeSettle)

+ l.fx.srv.lifetimesWatchOnce(context.Background())

+ if !l.fx.srv.lifetimesOn() {

+ t.Fatal("not on")

+ }

+ fault := &markerFaultStore{Store: l.fx.rel}

+ fault.fail.Store(true)

+ l.fx.srv.cfg.Releases = fault

+ l.fx.srv.lifetimesWatchOnce(context.Background())

+ if !l.fx.srv.lifetimesOn() {

+ t.Fatal("a failed marker read left on")

+ }

+ base := instancesURL(l.fx, "bab", ltApp)

+ if code, _, body := rawReq(t, http.MethodGet, base+"/"+ltOld, l.member, ""); code != http.StatusNotFound {

+ t.Fatalf("GET of the obsolete body = %d %s, want 404", code, body)

+ }

+ if code, _, body := rawReq(t, http.MethodGet, base+"/"+ltOne, l.member, ""); code != http.StatusOK {

+ t.Fatalf("GET one = %d %s", code, body)

+ }

+ requireListing(t, l, ltOne)

+// lateDeleteStore answers a Delete with a timeout and applies it 5 s later

+// on the clock: a request storage had fully received when the client gave

+// up.

+type lateDeleteStore struct {

+ release.Store

+ clock *lifetimesClock

+ mu sync.Mutex

+ pending map[string]time.Time

+ landed map[string]time.Time

+func (d *lateDeleteStore) Delete(_ context.Context, key string) error {

+ d.mu.Lock()

+ defer d.mu.Unlock()

+ d.pending[key] = d.clock.Now().Add(5 * time.Second)

+ return context.DeadlineExceeded

+func (d *lateDeleteStore) apply(t *testing.T) {

+ t.Helper()

+ d.mu.Lock()

+ defer d.mu.Unlock()

+ now := d.clock.Now()

+ for key, at := range d.pending {

+ if now.Before(at) {

+ continue

+ }

+ if err := d.Store.Delete(context.Background(), key); err != nil {

+ t.Fatal(err)

+ }

+ d.landed[key] = now

+ delete(d.pending, key)

+ }

+func TestSettleStartsAfterTheLastLegacyUnit(t *testing.T) {

+ l := newLifetimesFx(t)

+ s := l.fx.srv

+ legacyRecord(t, l.fx.rel, ltOne, "edge")

+ late := &lateDeleteStore{Store: l.fx.rel, clock: l.clock, pending: map[string]time.Time{}, landed: map[string]time.Time{}}

+ s.cfg.Releases = late

+ u, ok, _ := s.lifetimesAdmit(context.Background(), true)

+ if !ok || u.mode != cutoverBefore {

+ t.Fatal("the legacy unit was not admitted")

+ }

+ if code, _, body := l.cutover(t, l.operator); code != http.StatusOK {

+ t.Fatalf("cutover = %d %s", code, body)

+ }

+ l.clock.Advance(10 * time.Minute)

+ key := provision.RecordKey(release.OpsAccount, ltApp, ltOne)

+ if err := late.Delete(context.Background(), key); !errors.Is(err, context.DeadlineExceeded) {

+ t.Fatalf("the ambiguous delete = %v", err)

+ }

+ ended := l.clock.Now()

+ u.done()

+ var on time.Time

+ for i := 0; i <= 61 && on.IsZero(); i++ {

+ late.apply(t)

+ s.lifetimesWatchOnce(context.Background())

+ if s.lifetimesOn() {

+ on = l.clock.Now()

+ break

+ }

+ l.clock.Advance(time.Second)

+ }

+ if on.IsZero() || on.Before(ended.Add(lifetimeSettle)) {

+ t.Fatalf("on at %v, want no earlier than %v (the marker is 10 min old)", on, ended.Add(lifetimeSettle))

+ }

+ landed, ok := late.landed[key]

+ if !ok || !landed.Before(on) || len(late.pending) != 0 {

+ t.Fatalf("the late delete landed %v (pending %v), want before activation at %v", landed, late.pending, on)

+ }

+ if _, _, err := l.fx.rel.Get(context.Background(), key); err == nil {

+ t.Fatal("the record is still there")

+ }

+func TestDetachedContinuationsOwnTheirUnit(t *testing.T) {

+ t.Run("import upload", func(t *testing.T) {

+ useLifetimesClock(t)

+ f := newImportFixture(t)

+ body, _ := sqliteBytes(t)

+ begun := f.begun(t, importHost, body)

+ key := provision.RecordKey("bab", fromApp, importHost)

+ pauses := map[string]*putPause{key: {nth: 1}}

+ pausePuts(t, f.fx, pauses)

+ if code, out := uploadImport(t, f, importHost, body); code != http.StatusAccepted {

+ t.Fatalf("upload = %d %v", code, out)

+ }

+ waitFor(t, pauses[key].entered, "the continuation's record write")

+ if st := f.fx.srv.cutover.status(); st.Pending != 1 {

+ t.Fatalf("the handler answered; the continuation still writes: pending = %d, want 1", st.Pending)

+ }

+ close(pauses[key].release)

+ f.fx.srv.seedCopies.Wait()

+ if st := f.fx.srv.cutover.status(); st.Pending != 0 {

+ t.Fatalf("after the job's last write pending = %d, want 0 (exactly one end)", st.Pending)

+ }

+ rec, err := f.record(t, importHost)

+ if err != nil {

+ t.Fatal(err)

+ }

+ requireImportState(t, rec, begun.Seed.ID, seeds.StateUploaded, "")

+ })

+ t.Run("copy from", func(t *testing.T) {

+ useLifetimesClock(t)

+ f := newFromFixture(t, 2)

+ key := provision.RecordKey("bab", fromApp, fromHost)

+ pauses := map[string]*putPause{key: {nth: 2}} // the reservation, then the job's

+ pausePuts(t, f.fx, pauses)

+ code, out := f.create(t, fromBody)

+ if code != http.StatusAccepted {

+ t.Fatalf("copy = %d %v", code, out)

+ }

+ waitFor(t, pauses[key].entered, "the copy's record write")

+ if st := f.fx.srv.cutover.status(); st.Pending != 1 {

+ t.Fatalf("the handler answered; the copy still writes: pending = %d, want 1", st.Pending)

+ }

+ close(pauses[key].release)

+ f.fx.srv.seedCopies.Wait()

+ if st := f.fx.srv.cutover.status(); st.Pending != 0 {

+ t.Fatalf("after the copy's last write pending = %d, want 0", st.Pending)

+ }

+ if rec, err := f.record(t, fromHost); err != nil || rec.Seed == nil {

+ t.Fatalf("record = %+v %v", rec, err)

+ }

+ })

diff --git a/internal/console/regiondrift.go b/internal/console/regiondrift.go

index c226272c5..6fd103735 100644

--- a/internal/console/regiondrift.go

+++ b/internal/console/regiondrift.go

@@ -3,6 +3,7 @@ package console

import (

"context"

"log"

+ "time"

"github.com/carlosframework/platform/internal/provision"

)

@@ -20,8 +21,20 @@ const discrepancyScanCap = 10000

// provenance, so each line is a CANDIDATE for review, worded as one (C29).

// It only logs: an error is reported and skipped, never returned, so the

// check can neither block nor fail console start.

+//

+// It reads instance records, so it runs as a unit of record work

+// (lifetimes_cutover.go): admitted as a read, and retried every

+// lifetimesWatchInterval while the console is still starting.

func (s *Server) LogPlacementDiscrepancies(ctx context.Context) {

- s.logPlacementDiscrepancies(ctx, discrepancyScanCap)

+ for !s.lifetimesIteration(ctx, "placement check", false, false, func(ctx context.Context) {

+ s.logPlacementDiscrepancies(ctx, discrepancyScanCap)

+ }) {

+ select {

+ case <-ctx.Done():

+ return

+ case <-time.After(lifetimesWatchInterval):

+ }

}

func (s *Server) logPlacementDiscrepancies(ctx context.Context, scanCap int) {

diff --git a/internal/console/runsweep.go b/internal/console/runsweep.go

index 5ee0c011d..6cf2249c0 100644

--- a/internal/console/runsweep.go

+++ b/internal/console/runsweep.go

@@ -33,16 +33,16 @@ func (s *Server) StartRunSweep(ctx context.Context) {

log.Printf("console: run sweep: off on a host-claim console, where carlos run is refused")

return

}

- t := time.NewTicker(runSweepInterval)

- defer t.Stop()

- for pass := 0; ; pass++ {

- s.runSweepPass(ctx, time.Now(), pass%runSweepHousekeepingEvery == 0)

- select {

- case <-ctx.Done():

- return

- case <-t.C:

+ pass := 0

+ lifetimesWorker(ctx, runSweepInterval, func(ctx context.Context) {

+ // Each pass is a unit of record work (lifetimes_cutover.go). The

+ // housekeeping count advances only on a pass that ran.

+ if s.lifetimesIteration(ctx, "run sweep", true, false, func(ctx context.Context) {

+ s.runSweepPass(ctx, time.Now(), pass%runSweepHousekeepingEvery == 0)

+ }) {

+ pass++

}

- }

+ })

}

// runSweep is one full pass: every phase.

diff --git a/internal/console/seedsweep.go b/internal/console/seedsweep.go

index b93fa3d10..16ff8fe05 100644

--- a/internal/console/seedsweep.go

+++ b/internal/console/seedsweep.go

@@ -73,16 +73,14 @@ func (s *Server) StartSeedsPass(ctx context.Context) {

if _, ok := s.cfg.SeedData.(release.ModLister); !ok {

log.Printf("console: seeds pass: the data bucket cannot list last-modified times, so this sweep deletes no unreferenced snapshot")

}

- t := time.NewTicker(seedsPassInterval)

- defer t.Stop()

- for {

- s.seedsPass(ctx, time.Now())

- select {

- case <-ctx.Done():

- return

- case <-t.C:

- }

+ lifetimesWorker(ctx, seedsPassInterval, s.seedsIteration)

+// seedsIteration is one sweep as a unit of record work (lifetimes_cutover.go):

+// admitted as a writer, skipped while the gate refuses it, and run in the

+// mode it was admitted in.

+func (s *Server) seedsIteration(ctx context.Context) {

+ s.lifetimesIteration(ctx, "seeds pass", true, false, func(ctx context.Context) { s.seedsPass(ctx, time.Now()) })

}

// seedStale reports a seed stuck where no box moves it: a --from seed

diff --git a/internal/console/server.go b/internal/console/server.go

index 05b4e71a3..cb45d66f7 100644

--- a/internal/console/server.go

+++ b/internal/console/server.go

@@ -864,6 +864,13 @@ type Server struct {

claimBackendObserver func(backendEvent) // fixture recorder only

claimEntries claimInventory

claimWorkers map[string]func(context.Context)

+ // The lifetimes cutover (lifetimes_cutover.go): the judge a claims-off

+ // unit admitted on runs under (nil on a host-claim console), the

+ // admission gate, and whether the claims-off app materialisation has run

+ // in its first admitted iteration.

+ judge *provision.LifetimeJudge

+ cutover *cutoverGate

+ appsMaterialized atomic.Bool

// The console's move pass (movesrecovery.go): its clock (nil is

// time.Now; a TEST SEAM) and when each close was first seen due.

movesClock func() time.Time

@@ -1578,7 +1585,25 @@ func NewServer(cfg Config, st *Store) (*Server, error) {

return nil, err

}

- materializeQuietly(context.Background(), cfg.Releases, st, cfg.HostClaim)

+ // A host-claim console converges its app rows here, as it always has. A

+ // claims-off console converges accounts here and its app rows in the

+ // first admitted iteration of the lifetimes cutover watch, so no listing

+ // runs before the admission gate has a state (lifetimes_cutover.go).

+ if cfg.HostClaim {

+ materializeQuietly(context.Background(), cfg.Releases, st, cfg.HostClaim)

+ } else {

+ if _, err := MaterializeAccounts(context.Background(), cfg.Releases, st); err != nil {

+ log.Printf("console: could not materialize accounts from the bucket: %v", err)

+ }

+ s.judge = provision.NewLifetimeJudge()

+ }

+ // The admission gate starts unknown; the first marker read happens here,

+ // so a console that can read its bucket starts in before or draining.

+ // The watcher repeats it every 10 seconds.

+ s.cutover = newCutoverGate(lifetimesNow)

+ if s.lifetimesGoverned() {

+ s.readCutoverAtStart(context.Background())

+ }

// The pre-accounts backfill: operators become owners of ops, every granted

// app name is claimed there, and non-operator grant holders are left out

// and REPORTED — a per-app grant was never authority over ops. Additive and

@@ -1689,7 +1714,9 @@ func (s *Server) ServeHTTP(w http.ResponseWriter, r *http.Request) {

http.Redirect(w, r, s.cfg.Origin+redirectTarget(r.URL), code)

return

}

- s.mux.ServeHTTP(w, r)

+ // Every claims-off request is a unit of record work, admitted by the

+ // lifetimes cutover gate before the mux sees it (lifetimes_cutover.go).

+ s.serveAdmitted(w, r)

}

// redirectTarget is the path and query to hang off Origin: the request's own

diff --git a/internal/console/svcroutes_test.go b/internal/console/svcroutes_test.go

index 94d5b365f..2dd8504ad 100644

--- a/internal/console/svcroutes_test.go

+++ b/internal/console/svcroutes_test.go

@@ -112,6 +112,9 @@ var closedDoors = []string{

// An assignment lets an import take over routes filed under the

// operator account.

"POST /api/cli/system/legacy-routes",

+ // The lifetimes cutover (self-serve moves plan 1): deployment-wide and

+ // operator-only. It closes the console to legacy record work.

+ "POST /api/cli/system/lifetimes/cutover",

// A credential may never mint a credential, including itself. This is

// the escalation cut of spec §3.

"POST /api/cli/apps/{account}/{app}/instance-token",

diff --git a/internal/console/testdata/claim-routes.txt b/internal/console/testdata/claim-routes.txt

index 932579962..722422057 100644

--- a/internal/console/testdata/claim-routes.txt

+++ b/internal/console/testdata/claim-routes.txt

@@ -253,6 +253,7 @@ route POST /api/cli/system/antivirus

route POST /api/cli/system/antivirus/rotate-webhook-key

route POST /api/cli/system/layout

route POST /api/cli/system/legacy-routes

+route POST /api/cli/system/lifetimes/cutover

route POST /api/cli/system/pointer-digests

route POST /api/cli/system/rollback

route POST /api/cli/system/seeds-floor

@@ -292,6 +293,7 @@ worker StartExpiryPass

worker StartGeoipJobs

worker StartHostReports

worker StartInfraPass

+worker StartLifetimesCutoverWatch

worker StartMailEventPush

worker StartPlanLimitsPass

worker StartRelay

diff --git a/internal/provision/plan14_root_reads_test.go b/internal/provision/plan14_root_reads_test.go

index cd2c3bd60..1c15e4713 100644

--- a/internal/provision/plan14_root_reads_test.go

+++ b/internal/provision/plan14_root_reads_test.go

@@ -1266,7 +1266,9 @@ func TestPlan14RootReadCensus(t *testing.T) {

if err = loadRootReadInventory(filepath.Join(root, "internal/provision/testdata/plan14-root-reads-self-serve-moves-1.tsv"), &inv, true); err != nil {

t.Fatal(err)

}

- seal = "3bfd8a2df46d689def6f0728415fe23c4da3682e43546536c60a4e7c02a55378"

+ // Task 0 (the lifetimes cutover) re-pins four accepted-legacy console

+ // units, which moves the seal; still rev3=PROPOSED-self-serve-moves-1.

+ seal = "d77a71da099f1c0477430fb08a9c6f505319bfd9e72d929938e712754c693f1f"

errs := append(compareRootReads(got, inv), checkRootReadLegacy(inv, seal)...)

errs = append(errs, checkRootReadDeclarations(got, inv)...)

if len(errs) > 0 {

diff --git a/internal/provision/self_serve_moves1_census_test.go b/internal/provision/self_serve_moves1_census_test.go

index 67c0286e8..9e90d1dcb 100644

--- a/internal/provision/self_serve_moves1_census_test.go

+++ b/internal/provision/self_serve_moves1_census_test.go

@@ -31,12 +31,31 @@ package provision

// - readAllRecordsJudged: key-site; discover known-account record keys and

// judge them with bounded parallel reads; strict errors and tolerant

// incompleteness preserve gaps. None of the judged listings writes.

+//

+// Task 0 (the lifetimes cutover), reviewed dispositions (all PROPOSED):

+// - lifetimes_cutover.go:Server.handleAPILifetimesCutover: nonrecord; one

+// create-only Put of control/lifetimes-cutover.json (an operator, a

+// claims-off console); no record key, no record literal.

+// - Server.runImportUpload, Server.runFrom: record-writers as before through

+// SetRecordSeed; each now carries its request's admitted unit and ends it

+// after its last write. No write site added or moved.

+// - Server.handleAPIInstanceList: key-site as before; lists through

+// ListRecordsCompleteJudged with the request's judge (nil before the

+// cutover: ListRecordsComplete). A read; no write.

+// - Server.registerCLIRoutes: key-site as before; registers the cutover door.

+// - NewServer: key-site as before; the judge, the gate and its first marker

+// read; claims-off app materialisation moves to the cutover watch. No write.

func plan14SelfServeMoves1WriteCensusInventory() []censusEntry {

removed := []struct{ file, unit string }{

{"internal/provision/provision.go", "GetRecordETag"},

{"internal/provision/provision.go", "updateRecordSeeding"},

{"internal/provision/provision.go", "claimRawUpdateRecord"},

{"internal/provision/rootrecord.go", "IndexRecords"},

+ {"internal/console/apiinstanceimport.go", "Server.runImportUpload"},

+ {"internal/console/apiinstances.go", "Server.handleAPIInstanceList"},

+ {"internal/console/apiinstanceseed.go", "Server.runFrom"},

+ {"internal/console/authcli.go", "Server.registerCLIRoutes"},

+ {"internal/console/server.go", "NewServer"},

}

added := []censusEntry{

{"internal/provision/provision.go", "updateRecordSeeding", "record-writer", "key:RecordKey,key:WriteRecord", "20e3b0498fb8d410c3d10bc7c788b0eec137cf58aa4dfa2afb42da837ebd71ea"},

@@ -47,6 +66,12 @@ func plan14SelfServeMoves1WriteCensusInventory() []censusEntry {

{"internal/provision/lifetime_judge.go", "readAllRecordsJudged", "key-site", "key:ParseRecordKey,lit:instances", "e56e43bb89e7a013445033b390cd57bf6eda16a539f5a398d1dd26597fe30ca4"},

{"internal/provision/lifetime.go", "GetRecordRaw", "key-site", "key:RecordKey", "1e893cadc5dedac4687a6877457254c8e07c328a91b2c5923ef87fd3c41ff1fc"},

{"internal/provision/lifetime.go", "WriteHostIncarnation", "nonrecord", "sel:Put", "c5f2719a53447c0477706600081a9846444071f39b85759a463d1ee757dc760e"},

+ {"internal/console/lifetimes_cutover.go", "Server.handleAPILifetimesCutover", "nonrecord", "sel:Put", "90e15124c0c7fa2b3902bf68c54d418c16792fa29f88ea2ec906353fc9b82c64"},

+ {"internal/console/apiinstanceimport.go", "Server.runImportUpload", "record-writer", "key:SetRecordSeed", "f602216e89e9a93c330afb5153894621918ab390655897c1dc9bd2435f5cab80"},

+ {"internal/console/apiinstances.go", "Server.handleAPIInstanceList", "key-site", "lit:instances", "ac702aaa6da4c5a6ed7bb69427ba03ec3f2ee7753cce57c58349a978bbdcfadf"},

+ {"internal/console/apiinstanceseed.go", "Server.runFrom", "record-writer", "key:SetRecordSeed", "0f0ed34b5a5c99cc7f8245f2e8d6284c300b4c7314b312af69f96d6d49f0d4c0"},

+ {"internal/console/authcli.go", "Server.registerCLIRoutes", "key-site", "lit:instances,lit:sidecars", "59882a91226ccba287af507482bea1aba49d6cab94fd32de0f2d586276ae2f92"},

+ {"internal/console/server.go", "NewServer", "key-site", "lit:instances", "737b2f91242d1a6750de07b43ea0c72bc0bc24365211e8a6abf7b3307126f5e8"},

}

entries := plan14ClaimRosterBindingWriteCensusInventory()

for _, r := range removed {

diff --git a/internal/provision/testdata/plan14-root-reads-self-serve-moves-1.tsv b/internal/provision/testdata/plan14-root-reads-self-serve-moves-1.tsv

index f4e9db3db..d2e3fc985 100644

--- a/internal/provision/testdata/plan14-root-reads-self-serve-moves-1.tsv

+++ b/internal/provision/testdata/plan14-root-reads-self-serve-moves-1.tsv

@@ -45,7 +45,20 @@ internal/console/store_gate.go claimSQLCensus legacy accepted-legacy accepted-le

@remove internal/console/tempstore.go Store.CreateTemporaryAppInRegion

@removed internal/console/tempstore.go Store.CreateTemporaryAppInRegion H3 census-ruling-PROPOSED-self-serve-moves-1;rev3=PROPOSED-self-serve-moves-1;basis=constructor remains declared but has no direct read; app insertion and lifetime minting moved to insertAppRow, classified above; proposed removal only

# Task 2 envelopes, rev3=PROPOSED-self-serve-moves-1: internal/console covers the lifetime store, app constructors, lifetime projection and table-scoped gate labels; internal/provision (above) adds only ErrLifetimeExhausted beyond Task 1. Registry fields are unchanged.

-@coverage internal/console fdb47efcb501addb7b7b0bfc756247c64494153e6cbc20d50e5873e4e011ac87

+@coverage internal/console c3b410d7e8d1c98c61d3278a7e1a5351baa4284e311dc90294efe4251f1544a9

# Task 6a: rev3=PROPOSED-self-serve-moves-1; reviewed provision envelope now includes bounded record-then-slot observations, judged read/list/update gates and signed-counter exhaustion at slot read. No removal-loop, host or console change; registry fields and the legacy seal stand.

internal/provision/rootrecord.go IndexRecords authority B3,C rev3=PROPOSED-self-serve-moves-1;basis=retain every discovered candidate owner key per host for fresh lifetime judgement at Find; partial discovery still retries and no verdict or body is cached e44ac431865bf38baa4105f66c9cb23014cc3c7ffcd41d5385244cc76fc4c0b9

internal/provision/rootrecord.go RecordIndex.Find authority B3,C rev3=PROPOSED-self-serve-moves-1;basis=observe each candidate through the context judge, refuse corruption and read failures, require one valid owner and validate its schema and shape; nil-judge duplicates retain permission refusal 760eed59cbd6a8ddab1098ce3a5f7a4272ecda81c17cc079dc135aba3a85a8f8

+# Task 0 (the lifetimes cutover), rev3=PROPOSED-self-serve-moves-1: four changed accepted-legacy console units re-pinned with reviewed bases (serve starts the cutover watch; NewServer adds the judge, the gate and its first marker read and moves claims-off app materialisation to the watch; the instance listing reads judged under an admitted-on unit; runFrom ends its handed-over unit); system.go runSystem and systemVerb re-pinned for the lifetimes verb at their existing authority; envelopes for ., cmd/console and internal/console (refreshed above) re-pinned. Four new legacy fingerprints need a new seal, checked after the preceding one. No root read, class or task is added; registry fields stand.

+@legacy d5a330a59973d7a46729154b5b2a1ee79d06622f9d7d9fe08f31bec85ab7c935 cmd/console/main.go serve legacy accepted-legacy accepted-legacy;rev3=PROPOSED-self-serve-moves-1;basis=the console (cmd/console serve under console.service, user carlos-console): Task 0 starts the lifetimes cutover watch beside the other claim workers; the same three data path literals move; no privileged reader is added data-literal@4077,data-literal@5178,data-literal@5415 9cc6c0cbc9da08f9d4466069251d8028710ca2fa85a01be81dae7ce47bd2e4d9

+cmd/console/main.go serve legacy accepted-legacy accepted-legacy;rev3=PROPOSED-self-serve-moves-1;basis=the console (cmd/console serve under console.service, user carlos-console): Task 0 starts the lifetimes cutover watch beside the other claim workers; the same three data path literals move; no privileged reader is added data-literal@4077,data-literal@5178,data-literal@5415 9cc6c0cbc9da08f9d4466069251d8028710ca2fa85a01be81dae7ce47bd2e4d9

+@legacy 0b23119c935fb39171bb2787d8725ca0eb7736c067eab08b07236df0a23c6596 internal/console/apiinstances.go Server.handleAPIInstanceList legacy accepted-legacy accepted-legacy;rev3=PROPOSED-self-serve-moves-1;basis=API listing reads through ListRecordsCompleteJudged with the request judge, which is nil before the lifetimes cutover; the same SQLite query moves; no root settings decision occurs here file:Query@1830 ac702aaa6da4c5a6ed7bb69427ba03ec3f2ee7753cce57c58349a978bbdcfadf

+internal/console/apiinstances.go Server.handleAPIInstanceList legacy accepted-legacy accepted-legacy;rev3=PROPOSED-self-serve-moves-1;basis=API listing reads through ListRecordsCompleteJudged with the request judge, which is nil before the lifetimes cutover; the same SQLite query moves; no root settings decision occurs here file:Query@1830 ac702aaa6da4c5a6ed7bb69427ba03ec3f2ee7753cce57c58349a978bbdcfadf

+@legacy 992541f409e55b7643b6d49ab5e583b87d7bb4ab23c7d86c60e89ebfa07287d6 internal/console/apiinstanceseed.go Server.runFrom legacy accepted-legacy accepted-legacy;rev3=PROPOSED-self-serve-moves-1;basis=the detached half of the --from door, unchanged, now ends the request unit it was handed after its last write; the same scratch Stat moves; every failure still moves the record to aborted file:Stat@3142 0f0ed34b5a5c99cc7f8245f2e8d6284c300b4c7314b312af69f96d6d49f0d4c0

+internal/console/apiinstanceseed.go Server.runFrom legacy accepted-legacy accepted-legacy;rev3=PROPOSED-self-serve-moves-1;basis=the detached half of the --from door, unchanged, now ends the request unit it was handed after its last write; the same scratch Stat moves; every failure still moves the record to aborted file:Stat@3142 0f0ed34b5a5c99cc7f8245f2e8d6284c300b4c7314b312af69f96d6d49f0d4c0

+@legacy 5402211d21797809f0f1c71318484f0dbfb3b6e82b649684cfcb5fae044cde13 internal/console/server.go NewServer legacy accepted-legacy accepted-legacy;rev3=PROPOSED-self-serve-moves-1;basis=console construction adds the lifetime judge, the cutover admission gate and its first marker read; claims-off app materialisation moves to the cutover watch; the existing logs/data route literal moves and is not a filesystem read data-literal@9149 737b2f91242d1a6750de07b43ea0c72bc0bc24365211e8a6abf7b3307126f5e8

+internal/console/server.go NewServer legacy accepted-legacy accepted-legacy;rev3=PROPOSED-self-serve-moves-1;basis=console construction adds the lifetime judge, the cutover admission gate and its first marker read; claims-off app materialisation moves to the cutover watch; the existing logs/data route literal moves and is not a filesystem read data-literal@9149 737b2f91242d1a6750de07b43ea0c72bc0bc24365211e8a6abf7b3307126f5e8

+system.go runSystem authority B,C,D,F,G,J4 rev3=PROPOSED-self-serve-moves-1;identity=explicit operator CLI uid 0;basis=adds the lifetimes family to the system dispatch (carlos system lifetimes cutover, a console door call as the operator); every existing command branch unchanged 8276403c56bc08639e62dee3a348573707c6f5b6909c8b1423bdc21e900390c1

+system.go systemVerb authority B,C,D,F,G,J4 rev3=PROPOSED-self-serve-moves-1;identity=explicit operator CLI uid 0;basis=adds the lifetimes family to the system dispatch (carlos system lifetimes cutover, a console door call as the operator); every existing command branch unchanged 5952f7a80803ed5279dea42b3b2166e2d2d93ddd9fbe2e13ac8b1c00044bdb70

+@coverage . a523e20ce94095ba3023f1abe6f3af29dac8dc8cecca221f2c8a7d827549d10b

+@coverage cmd/console 3d40f88dab5d2cb0eff35c526ef7c2a19fd3fb700dad33678e630e9e0ced10f5

diff --git a/system.go b/system.go

index 0717c9333..cbf3e9370 100644

--- a/system.go

+++ b/system.go

@@ -50,7 +50,7 @@ func systemVerb(args []string) (verb string, rest []string, component string) {

// untouched in rest for runSystemAntivirusVerb to read.

// layout is a family too: migrate|revert|status stay in rest for

// runSystemLayout.

- case "claims", "files", "guard-arm", "guard-exec", "journal-migrate", "floor-allows-prev", "update", "status", "rollback", "cancel-activation", "apply", "antivirus", "layout", "geoip", "seeds-floor", "legacy-routes", "box", "group", "install-channel", "join", "host-env", "help", "-h", "--help":

+ case "claims", "files", "guard-arm", "guard-exec", "journal-migrate", "floor-allows-prev", "update", "status", "rollback", "cancel-activation", "apply", "antivirus", "layout", "geoip", "seeds-floor", "legacy-routes", "lifetimes", "box", "group", "install-channel", "join", "host-env", "help", "-h", "--help":

return args[0], args[1:], component

}

return "", args, component

@@ -93,6 +93,8 @@ func runSystem(args []string) {

runSystemSeedsFloorVerb(rest)

case "legacy-routes":

runSystemLegacyRoutesVerb(rest)

+ case "lifetimes":

+ runSystemLifetimesVerb(rest)

case "box":

runSystemBoxVerb(rest)

case "group":

@@ -140,6 +142,7 @@ func systemUsage() {

carlos system geoip <publish|status|off> [flags]

carlos system seeds-floor publish --min-release <release>

carlos system legacy-routes assign --name <name> --to <account>/<app>

+ carlos system lifetimes cutover [--no-wait]

carlos system box role <box> <default|legacy|platform>

carlos system box publish-record

carlos system group role <region> <default|legacy>

@@ -160,6 +163,9 @@ The platform's own binary (ops account owners and CARLOS_SYSTEM_OPERATORS):

the first release whose boxes can place a seed, such as v0.27.0

legacy-routes assign --name <name> --to <account>/<app>

`+systemLegacyRoutesVerb+`

+ lifetimes cutover [--no-wait]

+ switch a console without host claims onto instance lifetimes,

+ once, after the old console has stopped. Waits until they are on

box role <box> <default|legacy|platform>

whether new instances may be placed on a box. A legacy or

platform box keeps the instances it has and takes no new ones

diff --git a/system_lifetimes.go b/system_lifetimes.go

new file mode 100644

index 000000000..83c139a57

--- /dev/null

+++ b/system_lifetimes.go

@@ -0,0 +1,103 @@

+package main

+import (

+ "context"

+ "encoding/json"

+ "flag"

+ "fmt"

+ "io"

+ "log"

+ "net/http"

+ "os"

+ "time"

+// carlos system lifetimes cutover (self-serve moves plan 1, Task 0): switch a

+// claims-off console onto instance lifetimes, once. The console writes its

+// cutover marker, stops taking legacy work, waits for the work it already

+// took, and turns lifetimes on a minute after the last of it. Operators only.

+// Run it after the old console process has stopped and the new one started.

+const lifetimesCutoverDoor = "/api/cli/system/lifetimes/cutover"

+// lifetimesCutoverPoll is how often the command asks again while the console

+// is still switching. A variable so a test need not wait.

+var lifetimesCutoverPoll = 5 * time.Second

+// lifetimesCutoverStatus is what the door answers.

+type lifetimesCutoverStatus struct {

+ State string `json:"state"`

+ At time.Time `json:"at"`

+ Pending int `json:"pending"`

+// SystemLifetimesCutover asks the console to switch lifetimes on (or, once it

+// has been asked, how far it has got). The door is idempotent.

+func (s *apiSession) SystemLifetimesCutover(ctx context.Context) (lifetimesCutoverStatus, error) {

+ return systemLifetimesCutover(ctx, s.postJSON)

+func systemLifetimesCutover(ctx context.Context, post rolePost) (lifetimesCutoverStatus, error) {

+ status, body, err := post(ctx, lifetimesCutoverDoor, map[string]string{})

+ if err != nil {

+ return lifetimesCutoverStatus{}, err

+ }

+ if status != http.StatusOK {

+ return lifetimesCutoverStatus{}, apiErrorFromBody(status, body)

+ }

+ var out lifetimesCutoverStatus

+ if err := json.Unmarshal(body, &out); err != nil {

+ return lifetimesCutoverStatus{}, fmt.Errorf("unreadable response: %w", err)

+ }

+ return out, nil

+// lifetimesCutoverLine is the line the command prints for a status.

+func lifetimesCutoverLine(st lifetimesCutoverStatus) string {

+ if st.State == "on" {

+ return fmt.Sprintf("instance lifetimes: on since %s", st.At.UTC().Format(time.RFC3339))

+ }

+ return fmt.Sprintf("instance lifetimes: switching on; %d unit(s) of earlier work still finishing", st.Pending)

+// waitLifetimesCutover calls the door, prints its line, and with wait asks

+// again every lifetimesCutoverPoll until lifetimes are on, printing each line

+// that differs from the last.

+func waitLifetimesCutover(ctx context.Context, post rolePost, w io.Writer, wait bool, sleep func(time.Duration)) error {

+ last := ""

+ for {

+ st, err := systemLifetimesCutover(ctx, post)

+ if err != nil {

+ return err

+ }

+ if line := lifetimesCutoverLine(st); line != last {

+ fmt.Fprintln(w, line)

+ last = line

+ }

+ if st.State == "on" || !wait {

+ return nil

+ }

+ sleep(lifetimesCutoverPoll)

+ }

+// runSystemLifetimesVerb is `carlos system lifetimes cutover [--no-wait]`.

+func runSystemLifetimesVerb(rest []string) {

+ if len(rest) == 0 || rest[0] != "cutover" {

+ systemUsage()

+ os.Exit(2)

+ }

+ fs := flag.NewFlagSet("system lifetimes cutover", flag.ExitOnError)

+ consoleFlag := consoleOption(fs, "console origin (default: the only logged-in one)")

+ noWait := fs.Bool("no-wait", false, "print the console's state once and return, rather than waiting until lifetimes are on")

+ fs.Parse(rest[1:])

+ if fs.NArg() != 0 {

+ fs.Usage()

+ os.Exit(2)

+ }

+ sess := requireDeployAPISession(*consoleFlag)

+ printUsing(stdout, sess)

+ if err := waitLifetimesCutover(context.Background(), sess.postJSON, stdout, !*noWait, time.Sleep); err != nil {

+ log.Fatalf("carlos system lifetimes cutover: %v", err)

+ }

diff --git a/system_lifetimes_test.go b/system_lifetimes_test.go

new file mode 100644

index 000000000..da6b3ccb3

--- /dev/null

+++ b/system_lifetimes_test.go

@@ -0,0 +1,88 @@

+package main

+import (

+ "bytes"

+ "context"

+ "net/http"

+ "testing"

+ "time"

+// cutoverReplies answers each door call with the next reply, recording the

+// path and the sleeps between calls.

+type cutoverReplies struct {

+ replies []struct {

+ status int

+ body string

+ }

+ paths []string

+ sleeps []time.Duration

+func (c *cutoverReplies) post(_ context.Context, path string, _ any) (int, []byte, error) {

+ c.paths = append(c.paths, path)

+ r := c.replies[0]

+ if len(c.replies) > 1 {

+ c.replies = c.replies[1:]

+ }

+ return r.status, []byte(r.body), nil

+func (c *cutoverReplies) reply(status int, body string) *cutoverReplies {

+ c.replies = append(c.replies, struct {

+ status int

+ body string

+ }{status, body})

+ return c

+func TestSystemLifetimesCutoverCLI(t *testing.T) {

+ if verb, rest, _ := systemVerb([]string{"lifetimes", "cutover", "--no-wait"}); verb != "lifetimes" || len(rest) != 2 || rest[0] != "cutover" {

+ t.Fatalf("systemVerb = %q %q", verb, rest)

+ }

+ // The client call.

+ c := (&cutoverReplies{}).reply(http.StatusOK, `{"state":"draining","at":"2026-10-22T12:00:00Z","pending":2}`)

+ st, err := systemLifetimesCutover(context.Background(), c.post)

+ if err != nil || st.State != "draining" || st.Pending != 2 || !st.At.Equal(time.Date(2026, 10, 22, 12, 0, 0, 0, time.UTC)) {

+ t.Fatalf("status = %+v, %v", st, err)

+ }

+ if len(c.paths) != 1 || c.paths[0] != "/api/cli/system/lifetimes/cutover" {

+ t.Fatalf("asked %v", c.paths)

+ }

+ c = (&cutoverReplies{}).reply(http.StatusConflict, `{"error":"this console runs host claims; lifetimes are not for it"}`)

+ if _, err := systemLifetimesCutover(context.Background(), c.post); err == nil || err.Error() != "this console runs host claims; lifetimes are not for it" {

+ t.Fatalf("a refusal = %v", err)

+ }

+ // Waiting: the switching line while work finishes, once per change,

+ // then the on line, polling every 5 s.

+ c = (&cutoverReplies{}).

+ reply(http.StatusOK, `{"state":"draining","at":"2026-10-22T12:00:00Z","pending":2}`).

+ reply(http.StatusOK, `{"state":"draining","at":"2026-10-22T12:00:00Z","pending":0}`).

+ reply(http.StatusOK, `{"state":"on","at":"2026-10-22T12:01:30Z","pending":0}`)

+ var out bytes.Buffer

+ if err := waitLifetimesCutover(context.Background(), c.post, &out, true, func(d time.Duration) { c.sleeps = append(c.sleeps, d) }); err != nil {

+ t.Fatal(err)

+ }

+ want := "instance lifetimes: switching on; 2 unit(s) of earlier work still finishing\n" +

+ "instance lifetimes: switching on; 0 unit(s) of earlier work still finishing\n" +

+ "instance lifetimes: on since 2026-10-22T12:01:30Z\n"

+ if out.String() != want {

+ t.Fatalf("output:\n%s\nwant:\n%s", out.String(), want)

+ }

+ if len(c.sleeps) != 3 || c.sleeps[0] != 5*time.Second {

+ t.Fatalf("sleeps = %v, want three of 5 s", c.sleeps)

+ }

+ // --no-wait: one call, one line.

+ c = (&cutoverReplies{}).reply(http.StatusOK, `{"state":"draining","at":"2026-10-22T12:00:00Z","pending":1}`)

+ out.Reset()

+ if err := waitLifetimesCutover(context.Background(), c.post, &out, false, func(time.Duration) { t.Fatal("--no-wait slept") }); err != nil {

+ t.Fatal(err)

+ }

+ if out.String() != "instance lifetimes: switching on; 1 unit(s) of earlier work still finishing\n" || len(c.paths) != 1 {

+ t.Fatalf("--no-wait printed %q after %d calls", out.String(), len(c.paths))

+ }

diff --git a/system_test.go b/system_test.go

index 68d95bc74..94d27a6d6 100644

--- a/system_test.go

+++ b/system_test.go

@@ -36,6 +36,8 @@ func TestSystemVerbAcceptsTheComponentForm(t *testing.T) {

// install-channel take no arguments.

{[]string{"host-env", "write", "--deployment", "flagship", "--region", "ie"}, "host-env", "router", 5},

{[]string{"install-channel"}, "install-channel", "router", 0},

+ // lifetimes is a family: cutover and its flag stay in rest.

+ {[]string{"lifetimes", "cutover", "--no-wait"}, "lifetimes", "router", 2},

{[]string{"join"}, "join", "router", 0},

{[]string{}, "", "router", 0},

{[]string{"router"}, "", "router", 0},