design: host ownership rev 2, answers Astra round 1 (A-1 to A-15)
Aliases acquired in two phases (pending, then committed) with conditional cleanup; the create door verifies before CompleteCreate and runs the lifecycle protocol in full; lazy adoption dropped, backfill marker after the floor; one namespace rule for minted hosts across every door; pending app rows with a sweep; inventory, migrate, privacy, purge and cost corrected.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
1 file changed,
+613
−386
docs/superpowers/specs/2026-10-06-host-ownership-design.md+613 −386
CI passed — run details
diff --git a/docs/superpowers/specs/2026-10-06-host-ownership-design.md b/docs/superpowers/specs/2026-10-06-host-ownership-design.md| index ca5a4e4ac..03f25789f 100644 |
| --- a/docs/superpowers/specs/2026-10-06-host-ownership-design.md |
| +++ b/docs/superpowers/specs/2026-10-06-host-ownership-design.md |
| @@ -1,8 +1,20 @@ |
| # Host-scoped atomic ownership |
| -Design, 2026-10-06, **rev 1**. Closes amadan #232. Design only; nothing |
| +Design, 2026-10-06, **rev 2**. Closes amadan #232. Design only; nothing |
| here is built. |
| +Rev 1 (b70d4d30) went to Astra for a read-only review against the code |
| +(`~/.claude/tmp/host-ownership-review.md` on DOCS-T1, findings cited |
| +`A-1` to `A-15` below). Rev 2 answers all of them: aliases are acquired |
| +in two phases (pending, then committed) with conditional cleanup (A-3, |
| +A-4, A-5, A-8); the create door verifies before it grants serving |
| +authority and follows the lifecycle create protocol in full (A-6, A-9); |
| +lazy adoption is dropped and the backfill marker follows the version |
| +floor (A-1, A-2); minted namespaces get one explicit rule applied to every |
| +door (A-7); app-name writes get a pending state and a sweep (A-10); the |
| +inventory, migrate, privacy, purge and cost sections are corrected |
| +(A-11 to A-15). |
| + |
| Related specs: `2026-10-02-host-lifecycle-claim-design.md` rev 6.14 on |
| `design-replica-ownership` (the lifecycle claim), with its library at |
| `internal/hostclaim` on `feat/hostclaim-library`; `2026-08-12-domain-attach-design.md` |
| @@ -32,23 +44,28 @@ The design: |
| 1. **Record-backed hosts** (instances, replicas, static records, sidecars) |
| are owned by the **lifecycle claim's holder** at |
| - `control/hosts/<host>.json`. The create door's claim CAS is the atomic |
| - gate. There is no second ownership object for these hosts, and this |
| - spec adds no field to that claim. |
| + `control/hosts/<host>.json`. The lifecycle create protocol's claim CAS |
| + is the atomic gate. There is no second ownership object for these |
| + hosts, and this spec adds no field to that claim. |
| 2. **Aliases** (hosts with a claim and no record) are owned by the |
| - **global hostname claim** at `domains/<host>.json`, as today. |
| -3. The two families meet at one rule: **write your own object, then |
| - re-read the other family's, and back off if it is held.** With S3's |
| - strong read-after-write consistency this guarantees at least one of |
| - two racing writers sees the other, so two writers can never both |
| - finish. A writer that backs off leaves nothing behind and answers 409. |
| + **global hostname claim** at `domains/<host>.json`. The claim gains |
| + one additive field, `pending`, so an alias is acquired in two phases: |
| + written pending, verified, then committed. Only committed claims |
| + materialise or count as held by their owner. |
| +3. The two families meet at one rule: **write your own authority first, |
| + re-read the other family's, back off if it is held, and only then |
| + commit.** With S3's strong read-after-write consistency at least one |
| + of two racing writers sees the other. Every back-off deletes only the |
| + exact object version it wrote. |
| 4. **Minted hosts** (apex `<app>.<sqid>.<domain>`, static canary |
| `<name>.<app>.<sqid>.<domain>`) are owned by the app whose name they |
| - carry. That is decided from the name, with the console's unique |
| - (account, app) row as the atomic gate, and the alias door refuses |
| - those names unless the attaching app is the name's app. |
| -5. Legacy hosts get a claim from a backfill verb, and the door adopts |
| - lazily until a completion marker says the backfill is done. |
| + carry. One namespace rule, applied by every door, says so from the |
| + name alone. The console's unique (account, app) row is the atomic gate |
| + for the name, and it too is written then verified. |
| +5. Legacy hosts get a claim from a backfill verb that runs after the |
| + version floor, so no old writer can add an unclaimed record behind it. |
| + The door treats an absent claim as free only once the backfill has |
| + written its completion marker. There is no lazy adoption. |
| ## 1. The problem, in the code |
| @@ -73,39 +90,50 @@ Consequences: |
| handler's 15 s budget; it sees only SQL-listed apps and only this |
| console box's registry; direct-bucket records of apps not in SQL are |
| outside it. |
| +- The checks run before later validation (replica rails at :887, shape |
| + at :1092), so an invalid request answers 409 for a held host and 400 |
| + for a free one: an occupancy oracle. |
| ## 2. Writer inventory |
| -Every path that puts a hostname on the platform or takes one off, with |
| -what it writes today and what gates it. |
| +Every path that puts a hostname on the platform or takes one off. The |
| +**Role** column says how it relates to ownership: *decides* (it is an |
| +authority write), *derives* (it reads an authority and materialises), |
| +*operator* (outside the guarantee, by hand), or *no change* (host and |
| +holder unchanged). |
| -| Writer | Hosts | Writes today | Gate today | Gap | |
| +| Writer | Hosts | Writes today | Role | Gap today | |
| |---|---|---|---|---| |
| -| Instance create (`handleAPIInstanceCreate`) | `<label>.<m.Domain>`; platform `<label>.<sqid>.<apps>` or a custom manifest domain | record, create-only | sibling-label check, `hostHolder`, `hostClaimHolder`, post-write label re-check | check then write; scan cost; local registry only | |
| -| Instance delete (`handleAPIInstanceDelete`, :1675) | same | `DeleteRecord` | app gate | leaves no release of anything host-scoped (nothing to release yet) | |
| -| Replica create (same door, `Role: replica`, `CreateRecordReplica`) | one label + the app's domain | record | same as create | same as create | |
| -| Sidecar create and delete (apisidecars.go:285, :347) | `<name>.<app>.<sqid>.sidecar.internal` | sidecar record at `sidecars/<name>.json` | name validity | none: the host embeds account and app, so no cross-app collision exists; never routable | |
| -| Canary (`carlos canary`, canary.go:231) | `<canary>-<app>.<sqid>.<apps>` | the instance create door | the door's | the door's; flat label makes cross-app collisions easy (#207) | |
| -| Static canary mint (domains/reconcile.go:328) | `<name>.<app>.<sqid>.<apps>` | registry route on each box, no record, no claim | `byHost` (local routes) | an alias claimed at that name is shadowed silently | |
| -| Platform apex mint (domains/reconcile.go:266) | `<app>.<sqid>.<apps>` | registry route, no record, no claim | `byHost`; `hasApex` per (account, app) | defers to whatever route got there first, forever | |
| -| App create, rename, move (server.go:3667, appsettings.go:802, :1305) | the apex | console SQL row | `apexHolder` (the same scan) | check then write against records; scan cost | |
| -| Domain attach, alias branch (apidomains.go:444; domainattach.go:153) | any hostname | `domains/<host>.json`, create-only | `deploymentRouteTaken` (fleet fold + claim), `domainHostPolicy` | does not read records or the lifecycle claim | |
| -| Domain attach, first-route branch (domainattach.go:183-192) | any hostname | claim (Route "") then static record | the same, claim-then-record order | the record half has the instance door's race against another app's record | |
| -| Platform attach (`--platform`, `attachDomainAs`) | names in the deployment's zones | the same two branches | `platformOperator` | the same | |
| -| Domain detach (apidomains.go:853; domainattach.go:304) | | `DeleteRecord` when Route "" then `DetachClaim` | owner match | none new | |
| -| Domains pass (`ReconcileAliases`, aliases.go) | | `route_aliases`, `wildcard_domains`, `route_paths` | routes win; `deploymentRoutes` | derived, not a writer of ownership; the final backstop | |
| -| Route aliases (`route_aliases` table) | | derived from claims only | | not a writer | |
| -| Moves (apiinstancemove.go, `SetRecordMove`) | unchanged host | record fields | lifecycle op (moves spec) | never changes who holds a host | |
| -| Placement (`carlos apps place`, `SetRecordRegion`) and pools | unchanged host | record Region | | never changes who holds a host | |
| -| Account migrate (`carlos accounts migrate`, accounts.go:312) | unchanged host | copies `apps/<src>/<app>/` to `apps/<dst>/<app>/`, create-only; stamps routes' account; deletes nothing | | two records per host until the operator deletes the source; `domains/` claims still name the source account; see §4.8 | |
| -| App purge (`release.PurgeApp`, trash.go:182) | all of an app's hosts | deletes every key under the app prefix | `PurgeReady` | leaves the app's `domains/` claims in place (an existing orphan source) | |
| -| Mail domains (`email/domain-claims/<domain>.json`) | email sending domains | its own create-only claim | its own owner gate | a different namespace: a domain can both serve a site and send mail. Out of scope | |
| - |
| -Two things stand out. Every record-backed writer already funnels through |
| -`provision.CreateRecord` and `DeleteRecord` (or `PurgeApp`), and the |
| -lifecycle claim spec already wraps exactly those two with a per-host CAS. |
| -Every alias writer already funnels through `domains.AttachClaim` and |
| -`DetachClaim`, which are already create-only by host. |
| +| Instance create (`handleAPIInstanceCreate`) | `<label>.<m.Domain>` | record, create-only | decides | check then write; scan cost; local registry; ordering oracle | |
| +| Instance delete (:1675) | same | `DeleteRecord` | decides | nothing host-scoped to release yet | |
| +| Replica create (same door, `CreateRecordReplica`) | one label + the app's domain | record | decides | as create | |
| +| Sidecar create and delete (apisidecars.go:285, :347; `CreateSidecarRecord`, `DeleteSidecarRecord`, sidecar.go:322, :398) | `<name>.<app>.<sqid>.sidecar.internal` | sidecar record at `sidecars/<name>.json`; its own primitives, not `CreateRecord` | decides (lifecycle family `sidecar`) | none for ownership: the host embeds account and app; never routable | |
| +| Canary (`carlos canary`, canary.go:231) | `<canary>-<app>.<sqid>.<apps>` | the create door | decides, via the door | flat label makes collisions easy (#207) | |
| +| Platform apex mint (domains/reconcile.go:266) | `<app>.<sqid>.<apps>` | registry route via `Upsert`; candidates come from routes (`candidateApps`, :757), not SQL | derives from the app name | defers to any route already at the host | |
| +| Static canary mint (domains/reconcile.go:328-337) | `<name>.<app>.<sqid>.<apps>` | registry `Upsert`, **no `byHost` check**; `Upsert` rewrites `app` and `account` on conflict (registry.go:990) | derives from the app name | can overwrite an unrelated route at that host | |
| +| Trash and retired-canary route removal (domains/reconcile.go:207, :356) | the app's routes | `reg.Delete` | derives | none | |
| +| Instance and sidecar route mint and removal (provision/reconcile.go:965, :1178; sidecar_reconcile.go:403, :450) | record hosts | registry rows | derives from records | first-listed record wins a duplicate host | |
| +| App create, rename, move (server.go:3667, :3707; appsettings.go:802, :827, :1305, :1350) | the apex | SQL app row | decides the name | check then write against records; no crash recovery after the row | |
| +| Startup app-row backfill (store.go:676) | the apex | `INSERT OR IGNORE` ops-account rows for owners with no row | decides the name, by hand | an operator-era path; names it takes are checked by nothing | |
| +| Domain attach, alias branch (apidomains.go:444; domainattach.go:153) | any hostname | `domains/<host>.json`, create-only; same-owner `Route` converge in place (claims.go:150) | decides | does not read records or the lifecycle claim; converge can turn a first-route claim into an alias | |
| +| Domain attach, first-route branch (domainattach.go:183-192) | any hostname | claim (`Route ""`) then static record | decides, both families | the record half has the create door's race | |
| +| Attach rollback (`rollbackAttach`, domainattach.go:483-499) | | `DeleteRecord`, `DetachClaim` | cleanup | unconditional deletes: a paused rollback can remove a successor's claim | |
| +| Platform attach (`--platform`, `attachDomainAs`) | names in the deployment's zones | the same two branches | decides | the same | |
| +| Domain detach (apidomains.go:853; domainattach.go:304) | | `DeleteRecord` when `Route ""`, then `DetachClaim` (read, then unconditional delete, claims.go:182) | decides | unconditional delete | |
| +| Claim capabilities and paths (`SetClaimCapabilities`, `SetClaimPaths`, claims.go:302, :398) | | fields of an existing claim, owner-gated CAS | no change to ownership | none | |
| +| Domains pass (`ReconcileAliases`, aliases.go:95-133) | | `route_aliases`, `wildcard_domains`, `route_paths` | derives from claims | never reads the lifecycle claim; routes win only once a route is published | |
| +| Operator routes (`carlos route add`, `carlos remove`; cli.go:255, :278) | any | registry `Upsert`, `Delete` on one box | operator | by hand; covered by the fleet fold, not by any object | |
| +| Moves (apiinstancemove.go, `SetRecordMove`) | unchanged | record fields | no change | | |
| +| Placement (`SetRecordRegion`) and pools | unchanged | record Region | no change | | |
| +| Account migrate (accounts.go:312) | unchanged | copies `apps/<src>/<app>/` to `apps/<dst>/<app>/` create-only; only channel pointers are rebound (:369), records are copied unchanged (:416); stamps routes' account; deletes nothing | operator | the copied record names the source account, which `GetRecord` refuses as disagreeing with its path (provision.go:1145); two records per host; `domains/` claims still name the source | |
| +| App purge (`release.PurgeApp`, trash.go:182-278; console door ownership.go:442; box auto-purge domains/reconcile.go:232) | all of an app's hosts | deletes every key under the app prefix and the marker | decides (release) | leaves the app's `domains/` claims; a new app of the same name inherits them | |
| +| Mail domains (`email/domain-claims/<domain>.json`) | email sending domains | its own create-only claim | separate namespace | out of scope: a domain can serve a site and send mail | |
| + |
| +Every record-backed writer funnels through `CreateRecord`, |
| +`CreateSidecarRecord`, `DeleteRecord`, `DeleteSidecarRecord` or |
| +`PurgeApp`, and the lifecycle claim spec wraps exactly those with a |
| +per-host CAS. Every alias writer funnels through `AttachClaim` and |
| +`DetachClaim`. |
| ## 3. The objects |
| @@ -113,26 +141,35 @@ Every alias writer already funnels through `domains.AttachClaim` and |
| domains.Claim: `{schema, host, account, app, route, created_at, |
| created_by, wildcard, catchall, paths}`. `AttachClaim` reads, then |
| -`Put(key, body, "")` when absent; a lost create re-reads once and judges |
| -the winner; another owner gets `ErrHostUnavailable`, which never names |
| -the holder. Same owner converges `Route`. `DetachClaim` is owner-gated |
| -and quiet on mismatch. Every box folds `domains/` on its fast-converge |
| -leg through `ClaimCache` (one `ListMod` per pass, GETs only for changed |
| -objects), and `carlos domains list` and app Settings list it too. |
| - |
| -It is the right owner for aliases. It is the wrong owner for instance |
| -hosts, for three concrete reasons: |
| - |
| -- the fleet folds this prefix every few seconds on every box, and |
| - `ListObjectsV2` pages at 1000 keys; putting every instance host under |
| - it multiplies that listing by the instance count, forever; |
| -- the detach door's first-route branch deletes the record when |
| - `Route == ""` (apidomains.go:853). An instance ownership claim with no |
| - route would make `carlos domains detach <instance host>` an instance |
| - delete, unless every reader of the prefix learns a new `kind`; |
| -- the lifecycle claim already is a per-host, create-only, holder-naming |
| - object for record-backed hosts, with crash recovery designed and |
| - reviewed. A second one would be the duplicate the brief rules out. |
| +`Put(key, body, "")` when absent; another owner gets |
| +`ErrHostUnavailable`, which never names the holder; the same owner |
| +converges `Route` in place. `DetachClaim` is owner-gated and quiet on |
| +mismatch. Every box folds `domains/` on its fast-converge leg through |
| +`ClaimCache`, and `carlos domains list` and app Settings list it too. |
| + |
| +This spec adds one field, additive and `omitempty`: |
| + |
| +``` |
| +"pending": "<acquisition id>" |
| +``` |
| + |
| +A pending claim is an acquisition that has not been verified. It |
| +reserves the key (nobody else can create there) but it is **not held**: |
| +the alias pass does not materialise it, Settings and `domains list` show |
| +it as "attaching", and `SetClaimCapabilities` and `SetClaimPaths` refuse |
| +it. The acquisition id is random, chosen by the writer, and is how a |
| +retry tells its own acquisition from another's. A claim with no `pending` |
| +field is committed, which is every claim that exists today. |
| + |
| +The field must be declared in `Claim` in the same change that first |
| +writes it (the converge rule in claims.go's own comment). |
| + |
| +It remains the right owner for aliases and the wrong owner for instance |
| +hosts: the fleet folds this prefix every few seconds on every box and |
| +`ListObjectsV2` pages at 1000 keys; the detach door deletes the record |
| +when `Route == ""`; and the lifecycle claim already is a per-host, |
| +create-only, holder-naming object for record-backed hosts, with crash |
| +recovery designed and reviewed. |
| ### 3.2 The lifecycle claim, `control/hosts/<host>.json` |
| @@ -140,31 +177,36 @@ hostclaim.Claim: `{v, host, gen, holder{family, key, account, app, name, |
| incarnation, created_at, location, ...}, op{kind, id, ...}, touched, |
| inventory, obligations, lost, retired_incarnations, quiet_after, |
| cancelled_creates}`. `hostclaim.Update` writes the whole object with |
| -`If-Match` on the ETag read, or `If-None-Match` when it read absence. |
| -The object is never deleted (C5a); free is a written state. |
| -`BeginCreate` refuses unless `c.Free()`. `Key` accepts any valid host |
| -shape, custom domains included. |
| - |
| -Spec §5.1 is the create protocol: read the claim (ETag or absent), |
| -CAS to `{holder: new, op: creating}`, `CreateRecord` create-only, CAS to |
| -`serving`. §5.1 step 5 is the recovery pass for a `creating` op older |
| -than five minutes: record present with this incarnation, complete it; |
| -record authoritatively absent, cancel it (free the claim, retire the |
| -incarnation, list the key under `cancelled_creates`). §5.2 is delete: |
| -CAS `deleting`, `DeleteIfMatch` the record, park, free. |
| - |
| -That CAS is host-scoped and atomic. Two creates of one host from two |
| -apps both read absent, both `If-None-Match`, and exactly one succeeds. |
| +`If-Match` on the ETag read, or `If-None-Match` when it read absence. The |
| +object is never deleted (C5a); free is a written state. `BeginCreate` |
| +refuses unless `c.Free()`. `CompleteCreate` moves `op` to `serving` and |
| +drops the create's op id, after which `CancelCreate` is illegal |
| +(create.go:93, :104). `IsOrphan` treats a record whose incarnation an |
| +existing claim does not name as an orphan (orphan.go:10). `Adopt` writes |
| +a claim only for a host that has never had one (adopt.go). |
| + |
| +Spec §5.1 is the create protocol: step 0 reads the claim and keeps its |
| +ETag or absence, then registers the create in the app's `admission.json` |
| +as a lease; step 1 assigns `CreatedAt`; step 2 CASes the claim to |
| +`{holder: new, op: creating}` conditional on exactly what step 0 read, |
| +and a 412 restarts from step 0 with a new registration and a new |
| +`CreatedAt`; step 3 is `CreateRecord`; step 4 CASes to `serving`; step 5 |
| +is the recovery pass. §5.2 is delete; §5.3 purge; §9 legacy adoption; |
| +§10 the version floor. |
| + |
| **The lifecycle claim's holder is the ownership object for every |
| -record-backed host.** This spec adds nothing to it; it specifies how the |
| -console's doors use it and how aliases and minted hosts compose with it. |
| +record-backed host.** This spec adds nothing to it and changes none of |
| +its invariants. It specifies where the console's doors read the other |
| +family's authority inside that protocol, and asks the lifecycle spec for |
| +two things (§6). |
| ### 3.3 The record and the registry |
| The record stays the provisioning authority: boxes mint routes from |
| -records, and (under the lifecycle spec's C4) admit them only when the |
| +records, and under the lifecycle spec's C4 admit them only when the |
| claim's holder names the record's incarnation. The registry's routes and |
| -`route_aliases` stay derived. Nothing here changes the box side. |
| +`route_aliases` stay derived. The one box-side change this spec asks for |
| +is in the static canary mint (§4.6). |
| ## 4. The design |
| @@ -172,413 +214,598 @@ claim's holder names the record's incarnation. The registry's routes and |
| Each hostname has exactly one **ownership authority** by family: |
| -| Family | Authority | Atomic step | |
| -|---|---|---| |
| -| record-backed (instance, replica, static record, sidecar) | lifecycle holder | `hostclaim.Update(BeginCreate)` from absent or free | |
| -| alias | `domains/<host>.json` | `AttachClaim`'s create-only `Put` | |
| -| minted (apex, static canary host) | the app named in the host | the console's unique (account, app) row, trash included | |
| +| Family | Authority | Held means | Atomic step | |
| +|---|---|---|---| |
| +| record-backed (instance, replica, static record, sidecar) | lifecycle claim | `holder != nil` or `op.kind != ""` | `Update(BeginCreate)` from absent or free | |
| +| alias | `domains/<host>.json` | a claim exists (pending or committed; a pending one is held against everyone but its own acquisition) | `AttachClaim`'s create-only `Put` | |
| +| minted (apex, static canary host) | the app named in the host | the (account, app) row exists, live or trashed | the console's unique row insert | |
| -A host can be in only one family at a time. The cross-family rule: |
| +A host can be in only one family at a time. The cross-family rule, for |
| +every writer: |
| -> **Write your own authority first. Then re-read every other family's |
| -> authority for the host. If any is held, undo your own write and answer |
| -> 409. Only then write anything else (the record, the alias config, the |
| -> SQL row's dependants).** |
| +> **Write your own authority first, unverified. Then re-read every other |
| +> family's authority for the host. If any is held, delete exactly the |
| +> object version you wrote and answer 409. Otherwise commit: the record, |
| +> the committed alias, the verified app row.** |
| Why it is enough: S3 gives strong read-after-write consistency for |
| `PutObject` and `GetObject` on a key. Take two writers W1 (family X) and |
| -W2 (family Y) racing for host H. W1's write of X happens at t1 and W2's |
| -write of Y at t2. If t1 < t2, W2's re-read of X is after t2 > t1 and sees |
| -W1's object. Symmetrically for t2 < t1. So at least one of them backs |
| -off. Both may back off; that is two 409s and no residue, and either |
| -retry then wins. A deterministic tie-break was considered and rejected: |
| -the side that must yield cannot tell whether the other side also saw it, |
| -so any rule that lets one side keep going is unsound in some |
| -interleaving. |
| - |
| -The pre-write check stays as a cheap filter (it answers the common, |
| -sequential case without a write), but it is never what the design relies |
| -on. |
| +W2 (family Y) racing for host H, writing at t1 and t2. If t1 < t2, W2's |
| +re-read of X is after t2 > t1 and sees W1's object. Symmetrically for |
| +t2 < t1. So at least one backs off. Both may back off; that is two 409s |
| +and no residue, and either retry then wins. A deterministic tie-break was |
| +considered and rejected: the side that must yield cannot tell whether |
| +the other side also saw it, so any rule that lets one side keep going is |
| +unsound in some interleaving. |
| + |
| +Three corollaries the rev 1 text lacked: |
| + |
| +- **Nothing is committed before verification.** An unverified authority |
| + (a `creating` lifecycle op, a pending alias claim, a pending app row) |
| + reserves the name against third parties but is not success, is not |
| + materialised, and is not what a same-owner retry may report as done. |
| + A retry that meets its owner's unverified authority finishes the |
| + verification itself or refuses; it never short-circuits (A-3, A-4). |
| +- **Cleanup is conditional.** Every back-off and rollback deletes with |
| + `release.DeleteIfMatch` on the ETag the write returned, and refuses |
| + when the backend cannot enforce it. A paused rollback can never remove |
| + a successor's object, including another account's (A-5). |
| +- **Family does not change in place.** A first-route claim does not |
| + become an alias by converging `Route`, and a record-backed host does |
| + not become an alias while its record stands. Conversion is detach then |
| + attach, each through its own door (A-4). |
| + |
| +The pre-write check stays as a cheap filter for the sequential case, but |
| +the design never relies on it. |
| ### 4.2 Instance create |
| Replaces `hostHolder` and `hostClaimHolder` in `handleAPIInstanceCreate`. |
| Replicas, static records and canary instances go through the same door. |
| - |
| -1. Shape, apex and sibling-label checks, unchanged (name-derived, §4.6). |
| -2. **Pre-check, cheap.** `GET domains/<host>`. A claim for another owner, |
| - or this app's own alias at that host, is 409 (an app cannot hold a host |
| - as both an alias and an instance; detach first). One `registry.Get` on |
| - this box, kept as today for operator-added routes with no record. The |
| - fleet-wide fold `edgesfile.DeploymentRoutes` replaces "this box only": |
| - a host a live box publishes as a real route that no record and no |
| - claim explains is 409 as an operator repair. The fold is best-effort |
| - and permissive on read failure, exactly as the attach door uses it. |
| -3. **Claim.** `hostclaim.Read`, then `Update(BeginCreate(...))`. The |
| - lifecycle spec's own create step 2. `ErrConflict` (lost the |
| - `If-None-Match` or `If-Match`) re-reads once: a holder that is not |
| - this app is 409; a `creating` op by anyone is 409 (never wait inside a |
| - handler); free means retry the CAS once. Other errors fail closed. |
| -4. **Verify.** `GET domains/<host>` again. An alias claim (`Route != ""`) |
| - for anyone: `Update(CancelCreate)`, 409. A first-route claim |
| - (`Route == ""`) is not a reason to back off: that writer has to pass |
| - through this same lifecycle CAS and will lose it (§4.5). |
| -5. `CreateRecord`, create-only. Then `Update(CompleteCreate)`. These are |
| - lifecycle steps 3 and 4 and need no change. |
| -6. The post-write sibling-label re-check stays (apiinstances.go:1180), |
| - and its undo becomes `CancelCreate` plus `DeleteIfMatch` on the |
| - record, so the host does not stay held by a withdrawn create. |
| +The protocol is the lifecycle spec's §5.1 **in full**, with the |
| +cross-family reads inserted at two points. Nothing below replaces or |
| +shortens a lifecycle step (A-9). |
| + |
| +**Placement.** Every ownership read and write sits after every other |
| +validation the door performs (shape, replica rails, region, pool, |
| +environment, the store walk's preconditions), immediately before the |
| +record write, where `leftoverReplicaRefusal` sits today ("last of the |
| +checks"). An invalid request therefore answers its 400 whether or not |
| +the host is held (A-13). |
| + |
| +1. Shape and name-derived checks, unchanged and early: `ValidHost`, the |
| + namespace rule of §4.6 (which subsumes today's sibling-label check). |
| +2. **Pre-check, cheap, after all validation.** `GET domains/<host>`: any |
| + claim, pending or committed, is a refusal (§7 for the sentence). One |
| + `registry.Get` on this box for an operator route with no record. |
| + `edgesfile.DeploymentRoutes` for a route a live box publishes that no |
| + record explains. The fold is best-effort and permissive on read |
| + failure, as the attach door uses it. |
| +3. **Lifecycle step 0 and 1.** `hostclaim.Read`, keep the ETag or |
| + absence; register the create in `admission.json` as the lifecycle spec |
| + says, holding its lease; assign `CreatedAt`. |
| +4. **Lifecycle step 2.** `Update(BeginCreate)` conditional on step 3's |
| + read. On `ErrConflict`: re-read once to classify for the member (held |
| + by another, or a purge fence, or free again), answer 409 in the first |
| + two cases, and in the third **restart from step 3** with a new |
| + registration and a new `CreatedAt`, at most once per request. Never |
| + retry the CAS alone, and never wait inside the handler. |
| +5. **Verify.** `GET domains/<host>` again, and `AppNameState` for the |
| + host's label when the host is in the account's platform namespace. Any |
| + `domains/` object, pending or committed, alias or first-route, or a |
| + live or trashed app at the label: `Update(CancelCreate)` (legal: the |
| + op is still `creating`), release the registration, 409. The rev 1 |
| + exception for `Route == ""` claims is withdrawn: a first-route writer |
| + may converge that claim while this create proceeds, so its presence is |
| + not safe to ignore (A-4). |
| +6. **Lifecycle step 3.** `CreateRecord`, create-only. |
| +7. **Lifecycle step 4.** `Update(CompleteCreate)`. From here the host is |
| + serving and the only way back is delete and park (§4.3). Nothing after |
| + this step may call `CancelCreate` (A-6). |
| + |
| +The sibling-label re-check that today runs after the record write |
| +(apiinstances.go:1180) moves into step 5, before the record exists. The |
| +recovery pass can therefore complete any `creating` claim whose record is |
| +present without repeating door predicates: a record exists only after |
| +every verification passed, and the one write that could still race (an |
| +app row appearing after step 5) is caught on the other side, because |
| +`claimApp` writes its row and then reads this claim (§4.6). |
| The scan of other apps' records is gone. The only thing that can say a |
| -host is held is the host's own claim. |
| +record-backed host is held is the host's own lifecycle claim. |
| ### 4.3 Instance delete |
| Lifecycle §5.2 unchanged: CAS `deleting`, `DeleteIfMatch` the record, |
| park, free. Ownership is released when the claim reaches free. A host |
| -that is `deleting` or `parking` is held for the create door (step 3 |
| -sees a non-free claim) and answers 409 with a sentence that says the |
| -address is being released, not who held it. |
| +that is `deleting` or `parking` is held for the create door (step 4 sees |
| +a non-free claim). The member-facing sentence is §7's. |
| ### 4.4 Alias attach and detach |
| In `handleAPIDomainAttach`'s alias branch and `attachDomainAs`'s alias |
| -branch, around the existing `AttachClaim`: |
| - |
| -1. Policy and `deploymentRouteTaken`, unchanged and in the same order |
| - (the ordering rules from the Task 5 review stand: nothing goes between |
| - `deploymentRouteTaken` and the write). |
| -2. **Pre-check, cheap.** `hostclaim.Read`. A holder, or any op other than |
| - none, is 409 with `hostUnavailableMsg`. Absent and free pass. |
| -3. `AttachClaim`. `created == false` (a re-attach or a no-op) ends here: |
| - ownership was decided on an earlier call. This keeps the steady-state |
| - cost of titogo's reload re-attaches at one GET, as today. |
| -4. **Verify.** `hostclaim.Read` again. A holder or a `creating` op: |
| - `DetachClaim` (own, just created), 409 with the same bytes. |
| -5. Audit and serial, unchanged. |
| - |
| -Detach is unchanged: an alias has no lifecycle claim to release. |
| +branch, around the existing `AttachClaim`. Policy and |
| +`deploymentRouteTaken` stay first and in their existing order; the Task 5 |
| +ordering rules stand, and the new reads sit below them, immediately |
| +around the write. |
| + |
| +1. `GET domains/<host>` (the read `AttachClaim` already does; it returns |
| + the claim and ETag to the door). |
| + - **Committed, own, same route:** done, no write, 200. This is the |
| + titogo reload steady state and costs this one GET. |
| + - **Committed, own, different route, `Route != ""`:** converge as |
| + today (CAS on the ETag), audit, serial. Ownership was decided when |
| + the claim was committed. |
| + - **Committed, own, `Route == ""`:** a first-route claim. Refuse: |
| + "this hostname is this app's own route; detach it before attaching |
| + it as an alias". No in-place conversion (A-4). |
| + - **Pending, own acquisition id:** a retry of this request. Go to |
| + step 4. |
| + - **Pending, own owner, another acquisition id, older than the |
| + acquisition timeout (60 s):** a crashed attempt. Take it over: go to |
| + step 4 with this ETag. |
| + - **Any other pending or committed claim:** 409, `hostUnavailableMsg`. |
| + - **Absent:** continue. |
| +2. **Pre-check, cheap.** `hostclaim.Read`. A holder or any op: 409, |
| + `hostUnavailableMsg`. Absent or free: continue. |
| +3. **Write pending.** `Put(domains/<host>, claim{pending: id}, "")`, |
| + create-only. Keep the returned ETag `E1`. A CAS conflict re-reads and |
| + re-enters step 1 once. |
| +4. **Verify.** `hostclaim.Read` again. A holder or any op: |
| + `DeleteIfMatch(domains/<host>, E1)`, 409 with the same bytes. A failed |
| + or refused conditional delete leaves a pending claim, which the sweep |
| + clears (§5) and which never materialises. |
| +5. **Commit.** `Put(domains/<host>, claim without pending, E1)`. Keep |
| + `E2`. A conflict here means someone else's cleanup or takeover moved |
| + the object: re-read and re-enter step 1 once. |
| +6. Audit and serial, only on a commit. |
| + |
| +Two same-owner requests A1 and A2 racing an instance create B (A-3's |
| +interleaving): A2 now meets A1's pending claim and either waits for A1's |
| +outcome by re-reading (a retry with A1's id) or, as a different request, |
| +answers 409 "attach in progress, retry". A2 can only report success after |
| +a commit it observed or performed, and A1's cleanup is conditional on |
| +`E1`, so a claim A2 committed cannot be removed by A1. |
| + |
| +Detach: `DeleteIfMatch` on the ETag `GetClaim` returned, owner-matched as |
| +today, quiet on mismatch and on a lost condition (A-5). A pending claim |
| +of the caller's own may be detached the same way. |
| ### 4.5 First-route attach |
| The first-route branch writes a claim with `Route == ""` and then a |
| -static record. The record makes the host record-backed, so the lifecycle |
| +static record. The record makes the host record-backed; the lifecycle |
| holder is its authority and the `domains/` claim is the attach surface's |
| -marker (detach reads it to know it owns the record). |
| - |
| -Order: `AttachClaim` (create-only, as today), then the create protocol of |
| -§4.2 steps 3 to 5 for the static record, with the pre-check of step 2 |
| -reduced to the registry and fold reads. If the lifecycle CAS loses, |
| -`DetachClaim` the claim this call created and answer 409. The instance |
| -create's verify step 4 ignores `Route == ""` claims for this reason: the |
| -first-route writer cannot finish without winning the lifecycle CAS. |
| - |
| -The existing idempotent branch (a record already at the key that |
| -`attachWrote` recognises) becomes: the claim's holder is this app's |
| -record, so the attach is already complete. |
| - |
| -Detach's first-route branch becomes lifecycle delete (§4.3) followed by |
| -`DetachClaim`, the mirror of attach. |
| - |
| -### 4.6 Minted hosts: apex and static canary |
| +marker. The claim is pending until the record is serving. |
| + |
| +1. Policy, `attachChannel`, `deploymentRouteTaken` as today. |
| +2. **Write pending** `domains/<host>` with `Route ""` and an acquisition |
| + id, create-only, keeping `E1`. Any existing claim: as §4.4 step 1. |
| +3. The create protocol of §4.2 steps 3 to 7 for the static record. Its |
| + verify (step 5) reads `domains/<host>` and finds this call's own |
| + pending claim, recognised by acquisition id; that is not a conflict. |
| + Any other `domains/` state is. |
| +4. If the lifecycle CAS loses or the verify fails: |
| + `DeleteIfMatch(domains/<host>, E1)`, 409. |
| +5. After `CompleteCreate`: commit the claim, `Put(..., E1)`. |
| +6. Audit and serial. |
| + |
| +`attachWrote` and its shape restriction stay (A-4): the idempotent branch |
| +adopts a record only when it is kind static, no slug, no region, no pool, |
| +**and** the lifecycle holder is this app's record. The holder alone is |
| +not enough, because an instance record of the same app can sit at the |
| +same key under a custom manifest domain, and detach must never gain |
| +authority over it. |
| + |
| +Detach's first-route branch becomes lifecycle delete (§4.3) and then |
| +`DeleteIfMatch` on the claim. |
| + |
| +A crash between step 2 and step 5 leaves a pending first-route claim with |
| +or without a record. The lifecycle recovery pass settles the record half |
| +(§5); the sweep then commits the claim when the holder is this app's |
| +static record, or deletes it when the host is free (§5). |
| + |
| +### 4.6 Minted hosts and the namespace rule |
| The apex `<app>.<sqid>.<apps>` and a static app's canary host |
| `<name>.<app>.<sqid>.<apps>` are minted by boxes from the app's name and |
| channel pointers. They have no record and no claim, and should not get |
| one: the name is the ownership. |
| -- **Atomic gate:** the console's app row. `claimApp` and the rename and |
| - move doors already refuse a name that is live or trashed in the |
| - account. That row is the apex's `If-None-Match`. |
| -- **Cross-family rule applied:** `claimApp` writes the row, then re-reads |
| - the lifecycle claim of the apex host (one GET, replacing `apexHolder`'s |
| - scan). Held by another app: delete the row, refuse. The instance |
| - create door already does the other half: write, then re-read |
| - `AppNameState` for the label, withdraw on a hit (§4.2 step 6). Rename |
| - and move do the same for the new apex. |
| -- **Static canary space.** The alias doors refuse any host of the form |
| - `<x>.<app>.<sqid>.<apps>` unless the attaching app is `app` in account |
| - `sqid`. Decided from the name alone, so it is not an oracle. On a |
| - deployment with `ReservePlatformDomains` the whole apps domain is |
| - already `hostReserved` except the delegated alias accounts; this rule |
| - matters when reservation is off. Instance hosts cannot be in this |
| - space (`ValidHost` is one label under `m.Domain`). |
| -- The domains pass's "routes always win" and `deploymentRoutes` checks |
| - stay as the backstop; they should no longer fire for anything this |
| - design admits. |
| +**The namespace rule**, decided from the name and the account's app |
| +rows alone, so it is never an oracle (A-7): |
| + |
| +| Host shape under `<sqid>.<apps>` | Belongs to | Allowed | |
| +|---|---|---| |
| +| `<l>.<sqid>.<apps>` where `<l>` is a live or trashed app of the account | app `<l>` | an instance of `<l>` (the apex's own record), nothing else: no alias, no first-route, no instance of another app | |
| +| `<l>.<sqid>.<apps>` where `<l>` is no app | nobody yet | an instance of any app of the account (today's canary and slug hosts); an alias only on deployments with reservation off, as today | |
| +| `<x>.<app>.<sqid>.<apps>` | app `<app>` | nothing through any door. Minted only. An alias, a first-route or an instance here is refused whatever the attaching app | |
| +| a manifest `Domain` that is `<app>.<sqid>.<apps>` or deeper | app `<app>` | refused for any other app at the manifest door (`PutManifestIfMatch` callers) and at `ValidHost`'s caller in the create door | |
| + |
| +Applied by: the instance create door (step 1 of §4.2), both attach |
| +branches (with `domainHostPolicy`, before any write), and the manifest |
| +door. On deployments with `ReservePlatformDomains` the whole apps domain |
| +is `hostReserved` for members except the delegated alias accounts; the |
| +rule matters for those accounts and for deployments with reservation off. |
| + |
| +**The apex gate.** The console's app row is the apex's `If-None-Match`, |
| +and it follows the write-then-verify rule: |
| + |
| +1. `claimApp` (and rename, and move, for the new name) inserts the row |
| + with `pending = 1` in its transaction. The unique (account, app) |
| + constraint, trash included, is the atomic step. A pending row reserves |
| + the name against siblings exactly like a live row. |
| +2. Read the lifecycle claim of the apex host and `GET domains/<apex>`. |
| + Either held (a holder, an op, or any claim, pending or committed): |
| + delete the row (or, for rename and move, revert the pending change), |
| + refuse. Rename and move record `{app id, kind, from, to}` in a |
| + `pending_app_ops` table in the same transaction as their change, so |
| + the revert is one row away. |
| +3. Clear `pending` and the `pending_app_ops` row. |
| + |
| +The instance create door does the other half (§4.2 step 5). Rows and |
| +ops left pending by a crash are settled by the app-name sweep (§5): |
| +re-run step 2; held, revert; free, clear. A pending row is not an app a |
| +member can deploy to: the doors treat it as "being created" (A-10). |
| + |
| +**Apex candidates from routes.** The box mints an apex for any (account, |
| +app) that has routes (`candidateApps`), including an app with no SQL row. |
| +Such an app's name is reserved by nothing. The backfill verb (§8.1) |
| +reports every app prefix in the bucket that has no row, and the startup |
| +backfill at store.go:676 is the existing operator path that gives it one; |
| +this spec makes that path go through the same pending-and-verify steps. |
| + |
| +**The static canary mint** must not overwrite a route. The box-side |
| +change: before `reg.Upsert(nr)` at domains/reconcile.go:337, skip and |
| +log when `byHost` has the host and the existing route's app or account |
| +differ from the mint's. The apex mint already checks `byHost`. |
| ### 4.7 Replicas, sidecars, fleets, pools, moves |
| - A replica is a record with its own host, created by the same door: |
| §4.2 applies unchanged. Its lifecycle is the moves spec's. |
| -- Sidecars: the lifecycle claim covers them (family `sidecar`) for |
| - lifecycle. For ownership nothing is needed: the host embeds account and |
| - app, the attach door refuses the suffix, and the edge never routes it. |
| +- Sidecars: the lifecycle claim covers them (family `sidecar`). For |
| + ownership nothing is needed: the host embeds account and app, the |
| + attach door refuses the suffix, and the edge never routes it. |
| - Fleets, pools and placement change `Region` on an existing record and |
| never the host or the holder. |
| - Moves change `op` and `holder.location`, never `holder.account` or |
| - `holder.app`. A host under a move is held; the create door sees a |
| - non-free claim and refuses. |
| + `holder.app`. A host under a move is held; the create door refuses. |
| ### 4.8 Account migrate |
| -`carlos accounts migrate --app A --to <dst>` copies records to the |
| -destination prefix and deletes nothing, so a host has two records until |
| -the operator removes the source. The lifecycle holder names the exact |
| -record key and the incarnation hashes the account, so the holder must |
| -move or the destination record is an orphan by C1 and is never admitted. |
| - |
| -This needs a holder transfer: one CAS from `serving` that rewrites |
| -`holder.key`, `holder.account` and `holder.incarnation` to the |
| -destination record, from a verb that has proved the destination record |
| -exists with the same body. That is a lifecycle transition and belongs to |
| -the lifecycle spec, not here. Until it exists, migrate must refuse an app |
| -whose hosts carry a claim, with the sentence naming this spec. |
| - |
| -Also recorded: migrate does not touch `domains/` claims, so an app's |
| -aliases keep naming the source account and the alias pass refuses them |
| -after the move (`claim says <src>`). A `--rebind-claims` step for the |
| -verb is a follow-up. |
| +`carlos accounts migrate --app A --to <dst>` copies the app's objects to |
| +the destination prefix create-only and deletes nothing. Records are |
| +copied with their body unchanged, so the destination copy names the |
| +source account and `GetRecord` refuses it as disagreeing with its own |
| +path (provision.go:1145). The verb was written for release objects and |
| +channel pointers; for instance records it has never produced a valid |
| +destination record (A-12). |
| + |
| +Under this design a host's lifecycle holder names the source record's |
| +exact key, the incarnation hashes the source account, `holder.location` |
| +is incarnation-scoped, and a sidecar's host embeds the source sqid. A |
| +transfer therefore needs a protocol for record identity, storage |
| +location, execution authority, sidecars and `domains/` claims. That is |
| +not a one-field CAS and it is not designed here. |
| + |
| +**Rule:** `accounts migrate` refuses an app that has any instance or |
| +sidecar record, with a sentence naming this spec. Release objects and |
| +pointers migrate as today. The check reads the app's record prefixes |
| +under the floor, where every record has a claim, so no adoption can race |
| +it. The follow-up thread owns the transfer and the `domains/` rebind. |
| ## 5. Crash recovery |
| -Each crash window, what it leaves, and who clears it. No step below |
| -deletes anything it did not create, and nothing frees a host that has a |
| -record with the holder's incarnation. |
| +Each crash window, what it leaves, and who clears it. No step deletes |
| +anything it did not create or cannot name by exact version, and nothing |
| +frees a host that has a record with the holder's incarnation. |
| | Crash after | Left behind | Cleared by | |
| |---|---|---| |
| -| §4.2 step 3 (claim `creating`, no record) | an orphaned claim | the lifecycle recovery pass (§5.1 step 5): record absent, `CancelCreate`. The host is held for at most the five-minute age plus the pass interval. It never steals: a record present with this incarnation completes instead | |
| -| §4.2 step 5 record PUT, before `CompleteCreate` | claim `creating`, record present | the same pass completes the create | |
| -| §4.2 step 4 back-off, before `CancelCreate` | claim `creating`, an alias claim present | the pass cancels; the alias was the earlier writer and stands | |
| -| §4.4 step 3 (alias claim written, verify not run) | an alias claim that may collide with a lifecycle holder | the alias pass refuses it (routes win), so nothing is served wrongly. A console sweep lists `domains/` claims with `Route != ""` whose host has a lifecycle holder, logs them as conflicts, and leaves removal to detach or an operator: deleting a tenant's claim automatically is not this design's to do | |
| -| §4.5 first-route, after `AttachClaim`, before the lifecycle CAS | a `Route == ""` claim with no record | inert today (nothing materialises from a routeless claim), visible to its owner, cleared by detach or a retry. Unchanged | |
| +| §4.2 step 4 (claim `creating`, no record) | an orphaned claim | the lifecycle recovery pass (§5.1 step 5): record absent, `CancelCreate`, incarnation retired. Held for at most five minutes plus the pass interval. Never steals: a record present with this incarnation completes instead | |
| +| §4.2 step 6, before `CompleteCreate` | claim `creating`, record present | the same pass completes it. Safe without re-running door predicates: the record exists only after step 5 verified | |
| +| §4.2 step 5 back-off, before `CancelCreate` | claim `creating`, a `domains/` claim or an app row present | the pass cancels (record absent). The other writer was earlier and stands | |
| +| §4.4 step 3 (pending alias, verify not run) | a pending claim | never materialised. The **claim sweep** (console, every few minutes, over `ClaimCache`): a pending claim older than the acquisition timeout whose host has a lifecycle holder or op is deleted with `DeleteIfMatch`; one whose host is free is committed on behalf of its owner only if `Route` names a ready route of that owner (the fold says so), otherwise deleted. Deleting a pending claim removes nothing anyone was told succeeded | |
| +| §4.4 step 4 back-off, before `DeleteIfMatch` | the same pending claim | the same | |
| +| §4.4 step 5 (verified, commit PUT lost) | a verified but still pending claim | the sweep, as above; the owner's retry recognises its id and commits | |
| +| §4.5 step 2 to 5 | a pending first-route claim, with or without a record and a lifecycle state | the recovery pass settles the record half; then the sweep commits the claim when the holder is this app's static record (`attachWrote` shape), or deletes it when the host is free | |
| +| §4.6 step 1 (pending app row, unverified) | a name reserved by a row nobody verified | the **app-name sweep** (console, startup and every minute): re-run step 2 for every pending row and `pending_app_ops` row older than a minute; held, revert and log; free, clear | |
| | delete (§4.3) at any step | the lifecycle spec's recovery owners | unchanged | |
| -| app purge | records gone, `domains/` claims of the app left | the same console sweep lists claims whose account and app are purged and removes them. Removal is safe here: the owner no longer exists | |
| +| app purge | records gone; the app's `domains/` claims left | **the console purge door**, which holds the durable evidence (`PurgeReady`, the tombstone), detaches the app's claims with `DeleteIfMatch` **before** `PurgeApp`, listing them through `ClaimCache`. Box auto-purge has no bucket write grant on `domains/` and leaves them; the sweep reports those as "owner purged" and does not delete, because SQL absence is not proof (direct-bucket apps exist) (A-14) | |
| Old-console deletes during the roll (a record gone with a `serving` |
| claim) are the lifecycle spec's §10 case: the host is held until |
| -`carlos ops host resolve`. For the create door that is a 409 naming an |
| -operator repair, which is visible and safe. |
| +`carlos ops host resolve`. For the create door that is a refusal, §7's |
| +sentence, visible and safe. |
| + |
| +**Name reuse after purge.** A `domains/` claim names account and app by |
| +name, so a new app of the same name would inherit a purged app's aliases. |
| +The purge door's detach closes that going forward. Existing inherited |
| +claims are listed by the backfill verb for the operator. |
| ## 6. Composition with the lifecycle claim |
| Ownership and lifecycle are **one object for record-backed hosts** and |
| -**two for aliases**, and the two never both own a host. Concretely: |
| - |
| -- This spec asks nothing new of `internal/hostclaim`: `Read`, `Update`, |
| - `BeginCreate`, `CancelCreate`, `CompleteCreate`, and the recovery pass |
| - are used as specified. The ownership question "who holds H" is |
| - `Read(H).Claim.Holder` and nothing else. |
| -- It asks one thing of the lifecycle spec: the holder transfer for |
| - account migrate (§4.8), as a new transition or a ruling that migrate |
| - is refused for claimed hosts. |
| -- It adds console-side protocol around the lifecycle CAS: the pre-check |
| - and verify reads of `domains/`, and the alias doors' reads of |
| - `control/hosts/`. These are reads, and the lifecycle claim does not |
| - know aliases exist. |
| -- The lifecycle floor (§10 of that spec) gates this design too, since |
| - the create door's atomic step is the lifecycle create. Below the floor |
| - the door keeps today's checks. |
| +**two for aliases**, and the two never both own a host. |
| + |
| +- This spec uses `internal/hostclaim` as specified: `Read`, `Update`, |
| + `BeginCreate`, `CancelCreate` (only while `creating`), `CompleteCreate`, |
| + `Free`, and the recovery pass. "Who holds H" is `Read(H).Claim.Holder` |
| + and the op, nothing else. The create door runs the lifecycle protocol |
| + in full, admission lease included. |
| +- It asks the lifecycle spec for **two changes**, both to §9 adoption |
| + (A-1): |
| + 1. `Adopt` requires deployment-wide evidence: the adopting actor has |
| + listed every `apps/*/*/instances/<host>.json` and |
| + `apps/*/*/sidecars/*.json` for the host and found exactly one |
| + record. A host with two or more records is quarantined: not adopted, |
| + reported, refused to every door. Reconcile, the doors, purge and the |
| + recovery pass, which §9 names as adopting actors, take the same rule |
| + or defer adoption to the backfill verb. |
| + 2. Account migrate is refused for hosts with records until a transfer |
| + transition exists (§4.8). A ruling, not a mechanism. |
| +- It adds console-side protocol around the lifecycle CAS: the two reads |
| + of `domains/`, the alias doors' reads of `control/hosts/`, and the two |
| + sweeps. The lifecycle claim does not know aliases exist. |
| +- The lifecycle floor gates this design. Below the floor the doors keep |
| + today's checks; the pending alias field is written only above it, so |
| + no old box folds a pending claim as committed. |
| The alternative, an `alias` holder family in the lifecycle claim so that |
| -one object owns every routable name, was considered (§12). It is |
| -simpler to reason about but changes a spec that twelve joint rounds and |
| -a simulation have settled, and it puts every custom domain into the |
| -recovery pass's listing. |
| +one object owns every routable name, was considered (§12). |
| ## 7. Privacy |
| -- A refusal caused by another **account** never names it. The lifecycle |
| - holder's account differing from the caller's produces the same |
| - `hostUnavailableMsg` bytes and status as the `domains/` door's |
| - `ErrHostUnavailable`, on every door, including the create door's |
| - back-off in §4.2 step 4. Today's `hostHeldSentence` can say "routed for |
| - account X"; that sentence is retired for cross-account holds and kept |
| - only when the route names the caller's own account. |
| -- Within the caller's account the refusal may name the holding app and |
| - whether it is trashed, as today: a member can already list the |
| - account's apps. |
| -- Ordering stays neutral: the new reads sit where `deploymentRouteTaken` |
| - sits, below every gate that could refuse for another reason, so a |
| - prober cannot tell a held host from a free one by which refusal comes |
| - first. |
| -- The claim objects live in the deployment bucket and are read by |
| - consoles and boxes only. No door returns a claim body to a member. |
| -- The console sweep's log lines name hosts and owners; they go to the |
| - operator log, not to any tenant's activity log. |
| +**Placement.** Every ownership read sits after every other validation |
| +and immediately before the write it protects, on every door (A-13). The |
| +instance door's reads move from :772 to the position of |
| +`leftoverReplicaRefusal`. The alias doors' reads sit between |
| +`deploymentRouteTaken` and the `AttachClaim` write, where the Task 5 |
| +ordering rule puts the write's own refusal. |
| + |
| +**Responses.** One table, for every door: |
| + |
| +| Cause | Instance and app-name doors | Alias doors | |
| +|---|---|---| |
| +| held by another **account**, any family, any op | 409 `hostUnavailableMsg`, byte for byte | 409 `hostUnavailableMsg` | |
| +| held by another **app of the caller's account**, serving | 409 naming the app, as `hostHeldSentence` does today | 409 `hostUnavailableMsg` (the alias door's existing stronger neutrality, apidomains.go:458, is kept) | |
| +| held by another app of the caller's account, non-serving op (`creating`, `deleting`, `moving`...) | 409 naming the app and "is being changed; try again later" | 409 `hostUnavailableMsg` | |
| +| held by the caller's **own** app in the other family | 409 naming the conflict and the detach or delete that frees it | the same | |
| +| operator route, no record, no claim, caller's own account on the route | 409 "operator repair", as today | 409 `hostUnavailableMsg` | |
| +| operator route naming another account | 409 `hostUnavailableMsg` (today's "routed for account X" sentence is retired) | 409 `hostUnavailableMsg` | |
| +| a pending acquisition by another request | 409 "attach in progress, retry" only when the pending owner is the caller's own app; otherwise `hostUnavailableMsg` | the same | |
| + |
| +"Being released" and "operator repair" are therefore never said about a |
| +foreign hold. Nothing a member receives distinguishes a foreign hold from |
| +a foreign alias claim. |
| + |
| +**Logs and sweeps.** The two sweeps and the backfill verb name hosts, |
| +accounts and keys in the operator log and the verb's stdout only. No |
| +tenant activity log receives a line about another account. |
| + |
| +**Buckets.** `control/hosts/` and `domains/` are read by consoles and |
| +boxes. No door returns a claim body to a member. |
| ## 8. Migration |
| ### 8.1 Backfill |
| -A console-side verb, `carlos ops hosts backfill [--dry-run]`: |
| +A console-side verb, `carlos ops hosts backfill [--dry-run]`, runnable |
| +only above the version floor: |
| 1. List the bucket, not SQL: every `apps/*/*/instances/*.json` and |
| - `apps/*/*/sidecars/*.json` (the reconciler's own listing). This covers |
| - direct-bucket records of apps SQL does not list. |
| -2. For each host with no lifecycle claim, `claim.Adopt` with the current |
| - record as holder, as the lifecycle spec's §9 defines (one adoption |
| - routine, discovery obligations included). Create-only; a claim that |
| + `apps/*/*/sidecars/*.json`. This covers direct-bucket records of apps |
| + SQL does not list. |
| +2. For each host with exactly one record and no lifecycle claim, |
| + `claim.Adopt` with that record as holder, as the lifecycle spec's §9 |
| + defines (discovery obligations included). Create-only; a claim that |
| appeared since is left alone. |
| -3. A host with **two or more records** (an existing collision) is not |
| - adopted. It is reported with both keys, and an operator picks the |
| - winner by deleting the loser's record, then re-runs. The door refuses |
| - creates at that host meanwhile (§8.2), which is strictly safer than |
| - today. |
| -4. On a clean pass, write `control/hosts-backfill.json` with the count |
| - and time. Re-running is idempotent. |
| - |
| -Aliases need no backfill: `domains/` already is their authority. |
| - |
| -### 8.2 Lazy adoption at the door |
| - |
| -Until the marker exists, a create that reads an absent claim cannot |
| -treat absence as free: a legacy record under another app may hold the |
| -host. So the door keeps today's `hostHolder` scan **only on the absent |
| -path**, and on a hit it adopts the claim for the record it found and |
| -refuses. On a miss it proceeds with `BeginCreate`. Once the marker is |
| -present the scan is removed and absent means free. A host with a claim |
| -never scans, so the cost falls as the backfill proceeds and goes to zero |
| -at the marker. |
| +3. A host with **two or more records** is quarantined: not adopted, |
| + reported with every key, and refused to the create door (an absent |
| + claim with a quarantine entry is held). The operator picks the winner |
| + by deleting the loser's record, then re-runs. The quarantine list is |
| + `control/hosts-quarantine.json`, rewritten whole by the verb. |
| +4. A record that does not decode, or whose body disagrees with its key, |
| + is reported and blocks completion. Nothing is adopted for it. |
| +5. Report, without acting: app prefixes with no SQL row (§4.6); |
| + `domains/` claims whose owner has no app row; records under accounts |
| + the console does not know. |
| +6. **Completion** needs two consecutive clean passes with the floor in |
| + force: no unclaimed record, no quarantine, no undecodable record, and |
| + the second listing showing no record the first did not. Then the verb |
| + writes `control/hosts-backfill.json` with the count, the floor version |
| + and the time. Re-running is idempotent. |
| + |
| +Aliases need no backfill: `domains/` already is their authority, and |
| +every existing claim is committed by definition. |
| + |
| +### 8.2 The door before and after the marker |
| + |
| +There is no lazy adoption (A-1). Before the marker the create door runs |
| +**today's** path: the scan, `hostClaimHolder`, plain `CreateRecord`, no |
| +claim. An absent claim is not evidence of anything until the backfill |
| +says every record has one. After the marker, an absent claim is free, |
| +and the door runs §4.2. |
| + |
| +The marker cannot go stale (A-2): it is written only above the floor, |
| +where no console or box below this release runs, so nothing can write a |
| +record without a claim after it. If the floor is ever lowered, the |
| +marker is deleted first and the doors fall back. |
| ### 8.3 Mixed-version consoles |
| -An old console creates without a claim. Its record is a legacy host and |
| -is adopted by the first new actor that touches it (§8.2, or reconcile |
| -under the lifecycle spec). Two consoles of different versions racing for |
| -one host remain racy until every console is new; that is the lifecycle |
| -floor's job and the roll is consoles first. An old console's delete |
| -leaves a `serving` claim with no record; §5 covers it. |
| +Below the floor, old and new consoles both run today's path, so the race |
| +remains open until the floor is published; that is the lifecycle spec's |
| +rollout order and the roll is consoles first. An old console's delete |
| +after the floor cannot happen, because the floor refuses old binaries |
| +(lifecycle §10). A `serving` claim whose record is gone can still arise |
| +from a crashed new-console delete, and §5 covers it. |
| ### 8.4 Cross-account |
| -- Backfill adopts whatever record exists, including records under |
| - accounts the console does not know. It reports those but never |
| - rewrites or deletes them. |
| -- The door's refusals follow §7. Lazy adoption on the absent path reads |
| - only the caller's own account's records, as the scan does today, so a |
| - foreign record is not found by it; the foreign host's claim comes from |
| - the backfill or from its own account's next touch, and until then the |
| - box-side `duplicate host` log is what it is today. |
| +- Backfill adopts whatever single record exists, including records under |
| + accounts the console does not know. It reports those and never |
| + rewrites, re-homes or deletes a record. |
| +- Nothing in this design deletes a legacy record: the recovery pass |
| + deletes only orphans established under §5.1 step 5, and under the |
| + marker every legacy record has the claim the backfill gave it, so none |
| + is an orphan. |
| +- Refusals follow §7's table. |
| ## 9. Cost |
| -Per instance create, today: one registry read, two SQL reads, N |
| -sequential `GetRecord` calls (one per other app of the account, live and |
| -trashed), one claim GET, one record PUT. For an account with 50 apps that |
| -is 50 GETs and roughly a second of the 15 s budget. |
| - |
| -After: one `domains/` GET, one fleet fold read (`edgesfile.DeploymentRoutes`, |
| -the same read the attach door already pays), one claim GET, one claim |
| -PUT, one `domains/` GET, one record PUT, one claim PUT. Five bucket calls |
| -plus the lifecycle spec's two, independent of account size. The lifecycle CAS pair is a |
| -cost the lifecycle spec already commits to; this design adds two GETs |
| -to it and removes N. |
| +Per path, bucket calls only. "Today" is the current main. |
| -Per alias attach on the create path: two claim GETs more than today. On |
| -the re-attach path (titogo's reload loop): unchanged, one GET, because |
| -§4.4 stops after `created == false`. |
| - |
| -Boxes: nothing new. `control/hosts/` is folded only by the console's |
| -recovery pass, on the cadence the lifecycle spec sets. |
| +| Path | Today | After | |
| +|---|---|---| |
| +| Instance create, account with N apps | registry read, 2 SQL reads, N sequential GETs, 1 claim GET, 1 record PUT | 1 `domains/` GET, 1 fold read (the attach door's existing fold), lifecycle step 0 (1 GET + the admission lease's PUT), 1 claim PUT, 1 `domains/` GET + 1 SQL read, 1 record PUT, 1 claim PUT. Independent of N. The lifecycle pair and the lease are the lifecycle spec's cost; this spec adds the two `domains/` GETs and removes N | |
| +| Alias re-attach, no-op (titogo's reload loop) | 1 GET | 1 GET, unchanged | |
| +| Alias repoint (same owner, new route) | 1 GET + 1 PUT | 1 GET + 1 PUT, unchanged | |
| +| Alias fresh attach | 1 GET + 1 PUT | 1 GET, 1 lifecycle GET, 1 PUT (pending), 1 lifecycle GET, 1 PUT (commit): two GETs and one PUT more | |
| +| First-route attach | 1 GET + 1 PUT + record PUT | pending PUT, the create path above, commit PUT | |
| +| Detach | 1 GET + 1 DELETE | the same, conditional | |
| +| App create, rename, move | the N-GET scan | 1 lifecycle GET + 1 `domains/` GET | |
| +| Claim sweep | none | one `ListMod` of `domains/` every few minutes (the `ClaimCache` steady state), GETs for changed objects only, writes only for stale pending claims | |
| +| App-name sweep | none | SQL only, plus the two GETs per pending row, which is normally zero rows | |
| +| Boxes | | nothing new; the canary mint check is a map lookup | |
| + |
| +Migrate's refusal check lists the app's two record prefixes once. |
| ## 10. Tests |
| -All against a dirstore with real CAS, in `internal/console`, foreground. |
| +Against a dirstore with real CAS and conditional delete, in |
| +`internal/console`, foreground. Every interruption test runs the |
| +recovery pass and both sweeps afterwards and asserts the residual bucket |
| +state, not only the responses (A-15's test note). |
| 1. **Concurrent create.** Two apps, one host, 2 and then 16 goroutines |
| per app. Exactly one 201; every other response is 409 with the |
| - within-account sentence; exactly one record in the bucket; the claim's |
| - holder names it; `op.kind == serving`. Run at seeds 1..200. |
| -2. **Alias versus instance, both orders and concurrent.** Attach then |
| - create, create then attach, and both in parallel with an injected |
| - delay at every point between steps 2 and 5 of §4.2 and 2 and 4 of |
| - §4.4. Invariant after both return: never both 2xx; the bucket holds |
| - either an alias claim and a free or absent lifecycle claim, or a |
| - record with a matching holder and no alias claim, or neither. |
| -3. **First-route versus instance** under the same harness; the |
| - `Route == ""` claim never causes a create to back off on its own. |
| -4. **Crash between claim and record.** Inject a failure after |
| - `BeginCreate`. Run the recovery pass with the clock advanced past five |
| - minutes: claim free, incarnation retired, `cancelled_creates` lists the |
| - key; a new create of the host by another app succeeds. The negative: |
| - with the record present, the pass completes and never frees. |
| -5. **Crash after alias claim, before verify** with a lifecycle holder |
| - present: the sweep reports it; the alias pass does not materialise it; |
| - detach clears it. |
| -6. **Backfill.** Seed records with no claims across three apps and two |
| - accounts, including one host with two records. The verb adopts the |
| - singletons, refuses the pair and names both keys, writes no marker; |
| - after the loser is deleted it adopts and writes the marker. The door |
| - refuses creates at the collided host throughout. |
| -7. **Lazy adoption.** No marker, a legacy record under app A, a create |
| - for the host under app B: 409, and the claim now names A's record. |
| - With the marker, the same create reads absent and succeeds (the |
| - fixture has no legacy record then). |
| -8. **Privacy.** A cross-account hold on every door answers the same |
| - status and body as a foreign `domains/` claim, byte for byte, and the |
| - body contains no sqid. A table test over the holder kinds. |
| -9. **Apex.** `claimApp` racing an instance create at the label: the |
| - write-then-verify rule yields at most one winner, in both orders. |
| -10. **Alias refusal in canary space** with reservation off. |
| -11. **Migrate refusal** for an app with a claimed host. |
| + within-account sentence; exactly one record; the claim's holder names |
| + it; `op.kind == serving`. Seeds 1..200. |
| +2. **Alias versus instance, deterministic schedules.** Attach then |
| + create, create then attach, and every interleaving of §4.2 steps 2 to 7 |
| + with §4.4 steps 1 to 5 under an injected scheduler. Invariant after |
| + both return and after recovery: never both 2xx; the bucket holds |
| + either one committed alias claim and a free or absent lifecycle claim, |
| + or a record with a matching holder and no `domains/` object, or |
| + neither. No pending claim survives the sweep. |
| +3. **Same-owner alias retries** (A-3's schedule): A1, A2 for the same |
| + owner racing B. Whoever answered 2xx, the object it reported exists |
| + afterwards. A2 never reports success on a claim A1 later removed. |
| +4. **Family conversion** (A-4): a first-route claim, a concurrent alias |
| + attach by the same owner that would converge `Route`, and an instance |
| + create. The converge is refused; never both the record and an alias. |
| +5. **Stale cleanup** (A-5): a rollback paused across a detach and a |
| + different account's attach. The conditional delete fails; the |
| + successor's claim stands. |
| +6. **Crash between claim and record.** Inject a failure after |
| + `BeginCreate`; advance the clock past five minutes; recovery frees the |
| + claim, retires the incarnation, lists the key; a create by another app |
| + succeeds. The negative: record present, recovery completes, never |
| + frees. |
| +7. **Crash after commit point.** Inject a failure after `CompleteCreate` |
| + with the sibling re-check forced to fail: the door does not call |
| + `CancelCreate`; the host stays serving; the test documents that this |
| + path cannot occur because the check moved before the record. |
| +8. **Pending alias never serves** (A-8): a pending claim and a lifecycle |
| + holder whose route is not yet published; `ReconcileAliases` writes no |
| + `route_aliases` row; the sweep deletes the pending claim. |
| +9. **Backfill.** Records with no claims across three apps and two |
| + accounts, one host with two records, one undecodable record. The verb |
| + adopts the singletons, quarantines the pair, reports the bad record, |
| + writes no marker. After repair, two clean passes write it. The door |
| + refuses creates at the quarantined host throughout and treats absent |
| + as not free until the marker. |
| +10. **Legacy writer after the floor** (A-2): with the floor published, |
| + an attempt to run an old-path create is refused by the floor; with |
| + the floor absent, the marker verb refuses to run. |
| +11. **Privacy table.** For every row of §7's table and every door, the |
| + status and body match the table, cross-account bodies are |
| + byte-identical to a foreign `domains/` claim's refusal and contain no |
| + sqid, and an invalid request answers 400 whether or not the host is |
| + held. |
| +12. **Apex.** `claimApp` racing an instance create at the label under |
| + every interleaving: at most one winner; the loser's row or record is |
| + gone after the sweeps; no pending row survives. |
| +13. **Namespace rule.** Each row of §4.6's table against each door, with |
| + reservation on and off, including a manifest `Domain` of |
| + `<app>.<sqid>.<apps>` for another app. |
| +14. **Canary mint does not overwrite.** A route at `<x>.<app>.<sqid>.<apps>` |
| + owned by another app; the mint skips and logs. |
| +15. **Migrate refusal** for an app with a record; success for an app |
| + with releases only. |
| +16. **Purge detaches claims.** The console purge door removes the app's |
| + claims before `PurgeApp`; box auto-purge leaves them and the sweep |
| + reports without deleting. |
| The joint simulation (`internal/sim/hostlife`) should gain the alias |
| -actor and the two-family rule as an invariant, if the moves session |
| -agrees; this spec does not depend on it. |
| +actor, the pending state and the two-family invariant, if the moves |
| +session agrees; this spec does not depend on it. |
| ## 11. Rollout |
| 1. Land `internal/hostclaim` on main (`feat/hostclaim-library`). |
| -2. Ship the backfill verb and the console sweep (§5) with `--dry-run`. |
| - Run dry on flagship, read the collision list, repair, run live, confirm |
| - the marker. |
| -3. Ship the doors: create, delete, attach, first-route, detach, claimApp, |
| - rename, move. Behind the lifecycle floor; below it the doors keep |
| - today's checks. The absent-path scan stays until the marker (§8.2). |
| -4. Roll consoles, then boxes, in the lifecycle spec's order. Publish the |
| +2. Ship the doors, the two sweeps, the backfill verb, the `pending` |
| + field, the namespace rule and the canary-mint check, all **dormant |
| + below the floor**: below it every door runs today's path and the verb |
| + refuses to run. |
| +3. Roll consoles, then boxes, in the lifecycle spec's order. Publish the |
| floor. |
| -5. Remove the absent-path scan and `hostHolder`'s record loop once every |
| - deployment has the marker. `apexHolder` and `ReportApexSquats` read the |
| - claim instead of scanning. |
| -6. Follow-ups filed, not built here: migrate's holder transfer and claim |
| - rebind (§4.8); the purge sweep's claim removal (§5). |
| +4. Run the backfill dry; read the quarantine and the reports; repair; |
| + run live until two clean passes write the marker. |
| +5. The doors read the marker and switch to §4.2 on the next request. |
| + `hostHolder`'s record loop, `apexHolder`'s scan and `ReportApexSquats`' |
| + route walk are removed in the release after. |
| +6. Follow-ups filed, not built here: migrate's transfer and `domains/` |
| + rebind (§4.8); the lifecycle spec's adoption evidence rule (§6). |
| ## 12. Alternatives considered |
| - **`domains/<host>.json` as the ownership object for everything.** |
| - Rejected for the three reasons in §3.1: fleet-wide listing cost, the |
| - detach door's record delete on `Route == ""`, and duplicating the |
| - lifecycle claim's create protocol. |
| + Rejected: fleet-wide listing cost, the detach door's record delete on |
| + `Route == ""`, and duplicating the lifecycle claim's create protocol. |
| - **A new `hosts/<host>.json` ownership-only object.** Rejected: it would |
| duplicate the lifecycle holder for every record-backed host and need |
| its own crash recovery. |
| - **An `alias` holder family in the lifecycle claim.** One object for |
| - every routable name, no two-family rule. Rejected for now: it changes a |
| - settled spec and its validation (`holderKeyMatches`), gives every custom |
| - domain a lifecycle object the recovery pass lists, and gives titogo's |
| - reload loop a second GET per host unless the same `created == false` |
| - short-circuit is added. It remains the cleanest end state if the |
| - two-family rule proves hard to keep as doors are added, and §13 asks. |
| + every routable name, no two-family rule, and the alias pass would read |
| + one authority (A-8's "unify" option). Rejected for now: it changes a |
| + settled spec and `holderKeyMatches`, gives every custom domain a |
| + lifecycle object the recovery pass lists, and still needs the pending |
| + state for same-owner retries. It remains the cleanest end state if the |
| + two-family rule proves hard to keep as doors are added; §13 asks. |
| +- **Box-side alias pass reading the lifecycle claim** (A-8's other |
| + option). Rejected: one GET per alias per box per pass, or a second |
| + `ClaimCache` over `control/hosts/`. The pending state gives the pass a |
| + committed-only rule with no new read. |
| - **Deterministic tie-break instead of both backing off.** Rejected as |
| unsound (§4.1). |
| -- **Lazy adoption only, no backfill.** Rejected: the scan cost would |
| - persist indefinitely and the existing collisions would never be found. |
| +- **Lazy adoption.** Rejected (A-1): the door can only see the caller's |
| + account, so it would create claims over unseen foreign records and the |
| + recovery pass would then delete them as orphans. |
| +- **A one-field holder transfer for migrate.** Rejected (A-12): identity, |
| + location, execution authority, sidecars and aliases all move with the |
| + account. |
| ## 13. Open questions for Paul |
| -1. Two families with the write-then-verify rule (this spec), or one |
| - object with an `alias` family in the lifecycle claim (§12)? The second |
| - needs the lifecycle spec reopened. |
| -2. Account migrate: add a holder-transfer transition to the lifecycle |
| - spec, or refuse migrate for claimed hosts until moves are done? |
| -3. The console sweep: should it remove conflicting alias claims |
| - automatically after a grace period, or only report them (this spec)? |
| -4. Backfill collisions on flagship: operator picks the winner by hand |
| - (this spec), or the verb keeps the record the provisioning pass |
| - currently keeps (first in listing order)? |
| -5. Is the `<x>.<app>.<sqid>.<apps>` alias refusal acceptable on |
| - deployments with reservation off, or should those names stay open? |
| -6. Should an app's own alias at a host it now wants as an instance be |
| - converted in place rather than refused? |
| +1. Two families with pending alias claims (this spec), or one object with |
| + an `alias` family in the lifecycle claim (§12)? The second reopens the |
| + lifecycle spec. |
| +2. The lifecycle spec's §9 adoption rule: accept the deployment-wide |
| + single-record evidence requirement, and defer adoption to the backfill |
| + verb for all actors (§6)? |
| +3. Account migrate: refuse apps with records indefinitely (this spec), or |
| + file the transfer protocol now? |
| +4. The claim sweep commits a stale pending alias on its owner's behalf |
| + when the route is ready. Prefer delete-only, so nothing is ever |
| + attached without a request completing? |
| +5. The namespace rule refuses `<x>.<app>.<sqid>.<apps>` through every |
| + door, and a manifest `Domain` under another app's apex. Acceptable on |
| + deployments with reservation off? |
| +6. Backfill quarantine: operator picks the winner by hand (this spec), or |
| + the verb keeps the record the provisioning pass currently keeps (first |
| + in listing order)? |
| +7. The acquisition timeout of 60 s and the recovery age of five minutes |
| + are guesses. Shorter? |