amadan
console: a create cleans up a corrupt or stale body under its own live slot, quarantine first (final review 2, R1)
An owned live slot with a present body it does not vouch for went to the orphan retirement, which retired the slot and deleted the bytes on their ETag with no copy. It now goes to retireStaleLocked with liveOwned, as Task 4's cases (d) and (g): a corrupt body is copied to control/quarantine first, then a cleanup retirement retires the slot before the body leaves, and the create classifies again before it allocates. A failure before the cleanup row (the copy included) touches nothing and answers pending. The absent-body orphan path and the foreign-present refusal are unchanged. Census: internal/console envelope refreshed.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
5 files changed,
+167
−3
docs/superpowers/notes/census-proposal-self-serve-moves-1.md+2 −0internal/console/lifetimes_create.go+22 −2internal/console/lifetimes_create_test.go+132 −0internal/provision/self_serve_moves1_census_test.go+9 −0internal/provision/testdata/plan14-root-reads-self-serve-moves-1.tsv+2 −1
diff --git a/docs/superpowers/notes/census-proposal-self-serve-moves-1.md b/docs/superpowers/notes/census-proposal-self-serve-moves-1.md| index f066343a9..1112e4aa6 100644 |
| --- a/docs/superpowers/notes/census-proposal-self-serve-moves-1.md |
| +++ b/docs/superpowers/notes/census-proposal-self-serve-moves-1.md |
| @@ -44,4 +44,6 @@ Task 9 fix round 1 corrects Task 3's create door. When a delayed create's SQLite |
| Final review fix round. F1 changes the cutover door: it skips the box floor only once a validated marker body has been read or written by the door, never on the gate's state alone. A failed marker read still closes admission. `Server.handleAPILifetimesCutover` stays nonrecord and is re-pinned in place in the record-write inventory. F2 narrows the create's orphan rule: a live slot of another owner is retired only when no record exists under that owner's key, and a present body of any verdict refuses the create and is left untouched. `Server.planCreateLocked` is not a census unit, so only the envelope moves. F3 judges the catch-all label counts. `internal/domains` cannot import `internal/provision`, so it declares an `InstanceObserver` interface and its own context key, and `provision.WithJudge` records the same judge there (nil records none). `domains.InstanceLabelCountsJudged` counts only what the judge vouches for and reports a label with an unjudgeable record as unknown; with no observer it is `InstanceLabelCounts`. `ReconcileAliases` and `deriveInstanceAliases` (accepted legacy, rev3=J1) are re-frozen with reviewed bases: a stale body no longer makes a label shared, and an unknown label keeps its existing alias row and derives none, which adds four read sites on that row (`registry:Account`, `registry:Alias`, `registry:App` and the whole `RouteAlias`). That moves the seal. In the record-write inventory, `InstanceLabelCounts` leaves and `instanceHosts`, the key listing it now shares with the judged counter, takes its place as a key-site. `Server.catchallInstanceName` counts through the judged counter with the request's judge. The `internal/domains` and `internal/provision` envelopes are refreshed in place. F5 makes the migration pass report the migrations it attempted rather than the candidates it listed, so a candidate a pending operation blocks leaves the pass on its ten-minute idle cadence. `Server.migratePreLifetimeRecords` stays a key-site and is re-pinned in place. F6 changes a host test only. F7 bounds two in-process maps: the hostname exclusion is now refcounted, so an entry goes when its last holder or waiter is done, and the hourly lifetime log throttle prunes keys older than two hours when it logs a line. Neither reads or writes the bucket. The `internal/console` envelope is refreshed in place after each finding. No root read, class or task is added. |
| +Final review fix round 2. R1 closes the own-key half of F2. A create over a live slot of its own key with a present corrupt or stale body no longer takes the orphan retirement, which deleted the bytes with no copy. It now takes Task 4's cleanup: `Server.retireStaleLocked` copies a corrupt body to `control/quarantine/` first, then a cleanup retirement retires the slot before the body is deleted on its ETag. The create then classifies again. A failure before the cleanup row, the copy included, touches nothing and answers pending. `Server.planCreateLocked` is not a census unit and `Server.retireStaleLocked` is unchanged, so only the `internal/console` envelope moves. |
| + |
| Taken together, the rows marked `PROPOSED-self-serve-moves-1` are the census of one change. Claims-off instance records gain a lifetime: the host incarnation slot, written only through `WriteHostIncarnation` by the console's create, retirement and migration protocols. Every claims-off consumer judges a record against its slot before it acts on it. Every record write stays on `WriteRecord` with the sequence discipline. The per-host `move_seq` and each row's `(incarnation, serving_seq)` pair order the publications an edge folds. Each row is one of five kinds: a write site added or re-pinned in the record-write inventory (each through `WriteRecord`, a `DeleteIfMatch` on an observed ETag, or a Put of a control key that is not a record); a read made judged at its existing class; a console SQLite unit or a legacy fingerprint with a reviewed basis; the `@fields` schema row, which pins the registry fields Task 7 adds (`Route.Incarnation`, `Route.ServingSeq`, and the remote request's `Pair` and `ExpectPair`); or a source envelope refreshed over the new code. No row adds a raw record write, a root-read class or a task. The two registry read sites added in Task 8 are named in its paragraph. These rows are one proposal, for Paul to rule on once, together, after the plan's final review. Until then every row stays PROPOSED, and a green census means only that the rows match the code, not that they are accepted. |
diff --git a/internal/console/lifetimes_create.go b/internal/console/lifetimes_create.go| index 97463a82c..e34a840be 100644 |
| --- a/internal/console/lifetimes_create.go |
| +++ b/internal/console/lifetimes_create.go |
| @@ -149,10 +149,12 @@ func (s *Server) lifetimeAppCheck(ctx context.Context, sqid, app, lifetime strin |
| } |
| // planCreateLocked is step 2. A valid record under this key is the create's |
| -// conflict. A live slot whose record key has no valid record is an orphan (a |
| +// conflict. A live slot whose record key has no record is an orphan (a |
| // purge deleted the record, or a durable allocation row was lost, which this |
| // plan treats as outside its guarantees): it is retired first and the create |
| -// continues from the retired slot. Admission (step 1) ran under the same |
| +// continues from the retired slot. A live slot of this key with a corrupt or |
| +// stale body is cleaned up first (retireStaleLocked, the corrupt bytes |
| +// quarantined), and the create continues from the retired slot. Admission (step 1) ran under the same |
| // hostname lock, so no operation is pending for the hostname here. A live |
| // slot owned by another key with any record present under that key (valid, |
| // stale-looking or corrupt), or a holder hostHolder finds, refuses the |
| @@ -192,6 +194,24 @@ func (s *Server) planCreateLocked(ctx context.Context, acct *Account, rec provis |
| // again: something outside this console wrote it. |
| return createPlan{}, errLifetimeSettling |
| } |
| + if owned && obs.Present { |
| + // A body this key's live slot does not vouch for (corrupt |
| + // or stale) is no orphan: it is cleaned up as Task 4's |
| + // cases (d) and (g), the corrupt bytes copied aside first, |
| + // then a cleanup retirement (the slot, then the body). The |
| + // create then classifies again. Nothing is durable before |
| + // the cleanup row, so a failure there (the copy among |
| + // them) touches nothing and answers pending. |
| + switch err := s.retireStaleLocked(ctx, rec.Account, rec.App, host, obs, true); { |
| + case err == nil: |
| + case errors.Is(err, errLifetimeSettling), errors.Is(err, errLifetimePending): |
| + return createPlan{}, err |
| + default: |
| + log.Printf("console: lifetimes: create %s: clean up the body under its live slot: %v (pending)", host, err) |
| + return createPlan{}, errLifetimePending |
| + } |
| + continue |
| + } |
| if err := s.retireSlotOrphanLocked(ctx, host, slot, obs.SlotETag, ""); err != nil { |
| return createPlan{}, err |
| } |
diff --git a/internal/console/lifetimes_create_test.go b/internal/console/lifetimes_create_test.go| index eaff06788..eebca02fb 100644 |
| --- a/internal/console/lifetimes_create_test.go |
| +++ b/internal/console/lifetimes_create_test.go |
| @@ -10,6 +10,7 @@ import ( |
| "net/http" |
| "net/url" |
| "os" |
| + "regexp" |
| "runtime" |
| "strconv" |
| "strings" |
| @@ -2554,3 +2555,134 @@ func TestLifetimeLockAndLogStateIsBounded(t *testing.T) { |
| t.Fatal("a key logged within the hour was logged again") |
| } |
| } |
| + |
| +// A live slot of this key with a present body it does not vouch for (corrupt |
| +// or stale) is not an orphan: the create cleans it up as Task 4's cases (d) |
| +// and (g) do, the corrupt bytes copied aside first, the slot retired before |
| +// the body leaves, and only then the successor allocated. A quarantine copy |
| +// that fails touches nothing (final review 2, R1). |
| +func TestCreateCleansUpAPresentBodyUnderItsOwnLiveSlot(t *testing.T) { |
| + ctx := context.Background() |
| + const malformed = `{"schema":1,"host":` |
| + quarantineKey := regexp.MustCompile(`^control/quarantine/bab/` + ltApp + `/` + regexp.QuoteMeta(ltOne) + `\.[0-9]+\.json$`) |
| + arrange := func(t *testing.T, body []byte) (ltOn, string, string) { |
| + t.Helper() |
| + l := newLtOn(t) |
| + slotETag := writeSlot(t, l.fx.rel, ltOne, provision.HostIncarnation{Incarnation: 2, State: provision.SlotLive, Account: "bab", App: ltApp, Intent: "create lost"}) |
| + etag, err := l.fx.rel.Put(ctx, recKey(ltApp, ltOne), body, "") |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + return l, slotETag, etag |
| + } |
| + // atDelete checks, when the body is deleted, that the slot is already |
| + // retired and no allocation has begun. |
| + atDelete := func(t *testing.T, l ltOn, seen *bool) func(string, string) error { |
| + return func(op, key string) error { |
| + if op != "delete-if-match" || key != recKey(ltApp, ltOne) { |
| + return nil |
| + } |
| + *seen = true |
| + slot, _, err := provision.ReadHostIncarnation(ctx, l.fx.rel, ltOne) |
| + if err != nil || slot == nil || slot.State != provision.SlotRetired || slot.Incarnation != 2 || !slotOwnedBy(slot, "bab", ltApp) { |
| + t.Errorf("at the body's delete the slot is %+v (%v), want shop's incarnation 2 retired", slot, err) |
| + } |
| + if rows := allocRows(t, l.fx.st, ltOne); len(rows) != 0 { |
| + t.Errorf("an allocation began before the body left: %+v", rows) |
| + } |
| + return nil |
| + } |
| + } |
| + for name, c := range map[string]struct { |
| + body []byte |
| + quarantine bool |
| + }{ |
| + "a corrupt body": {[]byte(malformed), true}, |
| + "a stale body": {func() []byte { |
| + b, _ := json.Marshal(provision.Record{Schema: 1, Host: ltOne, Slug: "one", App: ltApp, Account: "bab", Channel: "stable", Incarnation: 1, CreatedAt: time.Now().UTC()}) |
| + return b |
| + }(), false}, |
| + } { |
| + t.Run(name, func(t *testing.T) { |
| + l, _, _ := arrange(t, c.body) |
| + var seen bool |
| + l.bucket.setHook(atDelete(t, l, &seen)) |
| + code, out := l.create(t, ltOne) |
| + if code != http.StatusCreated { |
| + t.Fatalf("create = %d %s, want 201", code, out) |
| + } |
| + if !seen { |
| + t.Fatal("the body was never deleted on its ETag") |
| + } |
| + keys, err := l.fx.rel.List(ctx, "control/quarantine/") |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + if !c.quarantine { |
| + if len(keys) != 0 { |
| + t.Fatalf("a stale body was quarantined: %v", keys) |
| + } |
| + } else { |
| + if len(keys) != 1 || !quarantineKey.MatchString(keys[0]) { |
| + t.Fatalf("quarantine keys = %v, want one control/quarantine/bab/%s/%s.<nanos>.json", keys, ltApp, ltOne) |
| + } |
| + if got, _, err := l.fx.rel.Get(ctx, keys[0]); err != nil || string(got) != malformed { |
| + t.Fatalf("quarantined bytes = %q (%v), want the corrupt body exactly", got, err) |
| + } |
| + } |
| + rets := retireRows(t, l.fx.st, ltOne) |
| + if len(rets) != 1 || rets[0].Kind != retireCleanup || rets[0].State != lifetimeCommitted || rets[0].Incarnation != 2 { |
| + t.Fatalf("retirements = %+v, want one committed cleanup at 2", rets) |
| + } |
| + a := oneAlloc(t, l.fx.st, ltOne, lifetimeCommitted) |
| + requireServing(t, l.fx.rel, ltApp, ltOne, 3, a.OpID) |
| + // The order in the bucket: copy aside, slot retired, body |
| + // removed, then the successor's slot. |
| + var order []string |
| + slotPuts := 0 |
| + l.bucket.mu.Lock() |
| + ops := append([]string(nil), l.bucket.ops...) |
| + l.bucket.mu.Unlock() |
| + for _, op := range ops { |
| + switch { |
| + case strings.HasPrefix(op, "put control/quarantine/"): |
| + order = append(order, "quarantine") |
| + case op == "put "+provision.HostIncarnationKey(ltOne): |
| + slotPuts++ |
| + order = append(order, "slot"+strconv.Itoa(slotPuts)) |
| + case op == "delete-if-match "+recKey(ltApp, ltOne): |
| + order = append(order, "delete") |
| + } |
| + } |
| + want := "slot1,delete,slot2" |
| + if c.quarantine { |
| + want = "quarantine," + want |
| + } |
| + if strings.Join(order, ",") != want { |
| + t.Fatalf("bucket order = %v, want %s", order, want) |
| + } |
| + }) |
| + } |
| + t.Run("a quarantine copy that fails", func(t *testing.T) { |
| + l, slotETag, etag := arrange(t, []byte(malformed)) |
| + l.bucket.setHook(func(op, key string) error { |
| + if op == "put" && strings.HasPrefix(key, "control/quarantine/") { |
| + return errBucketDown |
| + } |
| + return nil |
| + }) |
| + code, out := l.create(t, ltOne) |
| + if code != http.StatusBadGateway || ltErr(out) != msgLifetimeCreatePending { |
| + t.Fatalf("create = %d %s, want the pending 502", code, out) |
| + } |
| + if raw, e := rawOf(t, l.fx.rel, ltApp, ltOne); string(raw) != malformed || e != etag { |
| + t.Fatalf("the record after a failed copy = %q %s, want its bytes on %s", raw, e, etag) |
| + } |
| + if _, e := slotOf(t, l.fx.rel, ltOne); e != slotETag { |
| + t.Fatal("the slot moved after a failed copy") |
| + } |
| + if r, a := retireRows(t, l.fx.st, ltOne), allocRows(t, l.fx.st, ltOne); len(r) != 0 || len(a) != 0 { |
| + t.Fatalf("a failed copy left retirements %+v and allocations %+v", r, a) |
| + } |
| + }) |
| +} |
diff --git a/internal/provision/self_serve_moves1_census_test.go b/internal/provision/self_serve_moves1_census_test.go| index 6caf30e9a..180d3c4de 100644 |
| --- a/internal/provision/self_serve_moves1_census_test.go |
| +++ b/internal/provision/self_serve_moves1_census_test.go |
| @@ -241,6 +241,15 @@ package provision |
| // - F5: lifetimes_migrate.go:Server.migratePreLifetimeRecords: key-site as |
| // before, re-pinned in place; it returns the migrations it attempted |
| // rather than the candidates it listed. No write site changes. |
| +// |
| +// Final review fix round 2 (PROPOSED): no write site added, moved or made |
| +// raw. |
| +// - R1: lifetimes_create.go:Server.planCreateLocked (not a census unit) |
| +// sends a present corrupt or stale body under this key's live slot to |
| +// lifetimes_retire.go:Server.retireStaleLocked (key-site as before): the |
| +// quarantineCopy Put of a control key first for a corrupt body, then a |
| +// cleanup retirement whose writes are the dispatcher's |
| +// (WriteHostIncarnation, then DeleteIfMatch on the observed ETag). |
| func plan14SelfServeMoves1WriteCensusInventory() []censusEntry { |
| removed := []struct{ file, unit string }{ |
| {"internal/provision/provision.go", "GetRecordETag"}, |
diff --git a/internal/provision/testdata/plan14-root-reads-self-serve-moves-1.tsv b/internal/provision/testdata/plan14-root-reads-self-serve-moves-1.tsv| index 2509d38db..4896fbd56 100644 |
| --- a/internal/provision/testdata/plan14-root-reads-self-serve-moves-1.tsv |
| +++ b/internal/provision/testdata/plan14-root-reads-self-serve-moves-1.tsv |
| @@ -45,7 +45,7 @@ internal/console/store_gate.go claimSQLCensus legacy accepted-legacy accepted-le |
| @remove internal/console/tempstore.go Store.CreateTemporaryAppInRegion |
| @removed internal/console/tempstore.go Store.CreateTemporaryAppInRegion H3 census-ruling-PROPOSED-self-serve-moves-1;rev3=PROPOSED-self-serve-moves-1;basis=constructor remains declared but has no direct read; app insertion and lifetime minting moved to insertAppRow, classified above; proposed removal only |
| # Task 2 envelopes, rev3=PROPOSED-self-serve-moves-1: internal/console covers the lifetime store, app constructors, lifetime projection and table-scoped gate labels; internal/provision (above) adds only ErrLifetimeExhausted beyond Task 1. Registry fields are unchanged. |
| -@coverage internal/console d982f1babc8f4c803bea0fb0b1b5a53159a9ba0ec8e2005aae12e7b779327f7a |
| +@coverage internal/console 94c344b9d0fd0656bb87a90aa47abd5ff85329f2b07af16b609583513a6f4b98 |
| # Task 6a: rev3=PROPOSED-self-serve-moves-1; reviewed provision envelope now includes bounded record-then-slot observations, judged read/list/update gates and signed-counter exhaustion at slot read. No removal-loop, host or console change; registry fields and the legacy seal stand. |
| internal/provision/rootrecord.go IndexRecords authority B3,C rev3=PROPOSED-self-serve-moves-1;basis=retain every discovered candidate owner key per host for fresh lifetime judgement at Find; partial discovery still retries and no verdict or body is cached e44ac431865bf38baa4105f66c9cb23014cc3c7ffcd41d5385244cc76fc4c0b9 |
| internal/provision/rootrecord.go RecordIndex.Find authority B3,C rev3=PROPOSED-self-serve-moves-1;basis=observe each candidate through the context judge, refuse corruption and read failures, require one valid owner and validate its schema and shape; nil-judge duplicates retain permission refusal 760eed59cbd6a8ddab1098ce3a5f7a4272ecda81c17cc079dc135aba3a85a8f8 |
| @@ -201,3 +201,4 @@ internal/domains/aliases.go ReconcileAliases legacy accepted-legacy accepted-leg |
| internal/domains/aliases.go deriveInstanceAliases legacy accepted-legacy accepted-legacy;rev3=PROPOSED-self-serve-moves-1;basis=root-route; the agent (/opt/carlos/carlos agent, carlos-edge.service, user carlos): rev3=J1's body; a stale body under a retired lifetime no longer makes a label shared, and a label with a record that cannot be judged keeps its existing alias row of the catch-all's own account and app (reading that row's Account, App and Alias) and derives no new one; a report-only conservative keep, never a new route registry:Account@2163,registry:Account@2950,registry:Account@3913,registry:Alias@2225,registry:Alias@2293,registry:Alias@2337,registry:Alias@3006,registry:Alias@557,registry:App@1686,registry:App@2189,registry:App@2976,registry:App@3939,registry:Host@1843,registry:Host@3892,registry:Host@3902,registry:Host@4098,registry:Provisioned@1636,registry:Replica@1672,registry:Slug@1775,whole:Route@1620,whole:Route@1726,whole:Route@3502,whole:RouteAlias@2143,whole:RouteAlias@2920,whole:RouteAlias@3874,whole:RouteAlias@509,whole:RouteAlias@539,whole:RouteAlias@552,whole:RouteAlias@568 888a5a32dbbcd9619b0bf513c0dc746686ad7c4e5681bf82248022e8f13b2512 |
| # Final review fix round F5, rev3=PROPOSED-self-serve-moves-1: the migration pass reports the migrations it attempted, not the candidates it listed, so a blocked candidate leaves it on the idle cadence; no read site added. The internal/console envelope is refreshed in place above; no root read, class, legacy row or seal changes. |
| # Final review fix round F7, rev3=PROPOSED-self-serve-moves-1: the hostname exclusion becomes a refcounted keyedLocks (an entry goes when its last holder or waiter is done) and the hourly lifetime log throttle prunes keys older than two hours when it logs. In-process state only: no read site, class, legacy row or seal changes; the internal/console envelope is refreshed in place above. |
| +# Final review fix round 2, R1, rev3=PROPOSED-self-serve-moves-1: a create over a live slot of its own key with a present corrupt or stale body cleans it up through retireStaleLocked (the corrupt bytes quarantined, then a cleanup retirement) instead of the orphan retirement; no read site added. The internal/console envelope is refreshed in place above; no root read, class, legacy row or seal changes. |