carlos / platform Public
Instances that start from a database: --from (#260) and the import door (#261)
Status: rev 7, 2026-10-08. Rev 3 answered round 2 (§12), rev 4 round 3 (§13), rev 5 round 4 (§14), rev 6 round 5 (§15), and rev 7 round 6 (§16). Paul approved rev 1 (c66792a3). Rev 2 answers Astra's round 1 (CODEX-FINDINGS-RUN-AND-SEEDED-R1.md; findings 8 to 20 are this spec's). Each change is marked (codex N), and §11 lists the dispositions.
Discussions: amadan carlos/platform #260 and #261. Companion spec: 2026-10-08-carlos-run-design.md (#259).
1. The problem
Two requests from Tito share one mechanism.
- #260, a canary with real data. The commonest ask from Tito's team is "make me a canary with this account's real data". Today that takes root over SSM: copy the file, chown, register, grant.
carlos db pullgives a member a read-only copy on their laptop, but there's no way back in.instances createcan't start from data, andManifest.Seedis a first-boot argv (provision.go:233), not data. - #261, converting controld instances. About 71
titogo@<host>units on carlos-tito-ie run outside the platform, each behind an addr-backed row. Paul ruled on 2026-10-08 that every one of them converts to platform-run, and that nothing new is built for controld instances. - The import door the conversion needs was designed in2026-09-24-instance-moves-for-every-app-design.md§9 (gap G9), on the unmergeddocs-instance-moves-designbranch. - The moves spec that replaced it (2026-10-02-move-any-app-design.md) dropped it. That spec's "adoption" is the host claim taking over legacy platform records, not importing an instance the platform never ran.
Both requests come down to one thing: create an instance whose database starts as a given one.
2. Rulings (Paul, 2026-10-08)
- One mechanism, two sources. A sibling's replica (
--from) and an uploaded snapshot (import) both become one object, which the box places on first wake. - Uploads go through the console, not a presigned URL. This departs from §9 of the 09-24 design, following the console's own rule (F8,
apidb.go:299,internal/edge/storedoor.go): bytes pass through the console, and nothing a request supplies picks a bucket or a key. --fromstays in the source's region. To get to another region, runcarlos moveafterwards, as §9 said for adoption.- The new instance gets its own environment, never the source's per-host one, and an empty private store.
- Who may, for both doors: any person who is a member of the account, plus service credentials with the
instancerole. "This is day-to-day operator stuff ... this needs to be something developers can do." Stricter permissions may come later. - The swap accepts an operator-filed row. Tito's addr rows are filed under the deployment's operator account (G17), so a strict same-account check would refuse all 71. Rev 2 binds this exception to an operator's assignment (§6.4, codex 9).
- Platform-run only, nested only.
3. The shared step: a seed snapshot
3.1 The object
The snapshot lives at seeds/<host>/<id>/snapshot.db in the home region's data bucket, beside the instances/ and moves/ prefixes. <id> is the seed's ID (§3.2). Every write and delete of a snapshot names its seed ID, so cleaning up one seed can never touch another seed of the same host (codex R4-1).
- The console writes it. The owning box reads it and deletes it.
- The console's region is the home region, the only one whose data bucket it may write (
resolveReplica's residency rule,apidb.go:54-71). So both doors refuse a record homed anywhere else, with the existingErrResidencymessage. - There is no lifecycle rule on
seeds/(codex R2-5). A--frominstance that nobody wakes for a week still needs its snapshot. Instead, the console's deadline sweep deletesseeds/<host>/<id>/only in three cases: - no record names it, and the snapshot is more than an hour old. Every door writes its record before the snapshot (§4 step 7, §5.1), so a snapshot without a record is from a door that died between the two writes (codex R3-2); - the record's seed isplacedorcommittedand has another ID; - it belongs to an import stuck inawaiting-uploadoruploadedfor more than 7 days, or to a--fromstuck inreservingfor more than an hour. The sweep moves the record to the terminalabortedby CAS first, and deletes nothing unless that CAS succeeds (§5.6).
3.2 The record
provision.Record gains:
Seed *SeedSource `json:"seed,omitempty"`
type SeedSource struct {
ID string `json:"id"` // random, minted with the seed; never reused
Kind string `json:"kind"` // "replica" | "import"
From string `json:"from,omitempty"` // replica: the source host
TXID string `json:"txid,omitempty"` // replica: the TXID actually restored
SHA256 string `json:"sha256"` // of snapshot.db
Bytes int64 `json:"bytes"`
State string `json:"state"` // §3.2 states
Reason string `json:"reason,omitempty"` // set with "refused"
By string `json:"by"` // the caller
At time.Time `json:"at"`
// import only:
Replaces string `json:"replaces,omitempty"` // the addr the swap replaces
VerifyHash string `json:"verify_hash,omitempty"` // sha256 of the verify token (§5.4)
}States.
| Door | Path through the states | |---|---| | --from | reserving → ready → placed | | import | awaiting-upload → uploaded → ready → swapped → placed → committed |
Either kind can also be abandoned. Its record then goes aborted → rolled-back, and is deleted (§5.6).
An import can also leave the path:
refused: the box would not swap;Reasonsays why.aborted, and then the record is deleted.
What reconcile does with each state (codex 13). It reads the import journal first (§5.7). Then it dispatches on the seed state before any other per-record handling:
| State | Reconcile | |---|---| | aborted | runs the idempotent rollback (§5.6), or, with no journal row, deletes seeds/<host>/<id>/ and CASes the record to rolled-back | | rolled-back | skips the record; the console finishes it (§5.6) | | reserving, awaiting-upload, uploaded, refused | skips the record; nothing is minted | | ready | mints (--from) or swaps (import) | | swapped, placed, committed | ordinary handling, plus the import gate (§5.4) while not committed |
Seed stays on the record after placement. It is the provenance instances list shows.
3.3 The seed floor (codex 16)
An old box, or an old binary a box recovers to, would read a seeded record as an ordinary new instance and mint it empty. A seeded instance that comes up empty looks healthy and is wrong. So seeds get a protocol floor, built on the moves floor's mechanism (internal/moves/floor.go):
- The floor object. An operator publishes
control/seeds-protocol.json. - What a box does. A box running a seed-capable release arms the floor locally. Once armed, it gives a verdict on
carlos.prev, and the recovery units refuse acarlos.prevbelow the floor, exactly as they do for the moves floor (carlos-system-recover.service). - What the console checks. It writes no seed state at all, not even
awaiting-upload, until every box in the home region reports, in its status, the seeds floor armed andcarlos.preveither excluded or seed-capable. Otherwise the door answers 409 and names the boxes.
3.4 Placing the snapshot on first wake (codex 8)
This version: the plain restore path only. On a box with claim control armed, both doors refuse with refused: "claim". The claim's launch path doesn't run tenant code yet (rolelaunch_linux.go:23-28; instancectl/server.go:840), and its restore never trusts a local file (hostbox/restore.go:142, host/claimwakeassembly.go:252). The claim algorithm is set out at the end of this section for #250 to build.
The plain path. One more fallback goes into ExecBackend.Restore (backend_exec.go:611), beside the staging fallback (:633-645). It applies only to a row whose record seed is ready, swapped or placed:
- Is there already data? - If the host has any replica of its own, it has been placed already. Restore normally, and ignore the seed. - A local database counts only with installation evidence (codex R2-1). Step 3 writes
<db>.seed, which names the seed'sID, next to the database. If the local file's marker names the record's current seedID, it is this seed, placed by an interrupted first wake: use it, and go to step 4. - Anything else is purged before step 2, throughinstancefs: - a local database with no marker, or with a marker naming another ID; - its-waland-shmfiles; - litestream's local metadata.
So a database left behind by an earlier, aborted import of the same host can never stand in for a new one.
- Fetch and check. Read
seeds/<host>/<id>/snapshot.dbinto a temp file inside the instance's directory. Check its length and sha256 against the record, then runlease.CheckDBContext(integrity, and at least one table). - Place it. Write
<db>.seednaming the seedID, then rename the temp file onto the row's database path. Both go throughinstancefs, owned by the instance, in the nested tree. The marker is written first, so a database at that path always has its marker. - Replicate. Start replication. The first upload is generation one of the host's own lineage. Nothing of the source's lineage is copied.
- Advance the state. After the first upload succeeds, set the seed to
placedand deleteseeds/<host>/<id>/.
Failures. Any failure in steps 2 to 4 fails the wake. The reason goes to the box log and the carlos logs sink. It never falls through to an empty database and the manifest's seed command. The manifest's seed command is never run for a seeded row. The snapshot is deleted only after the host's own replica exists, so a crash at any point retries from either the placed local file or the snapshot.
The claim algorithm, for #250. This is not built here.
- Select the remote baseline first.
- If no baseline exists and the record still carries an unplaced seed: purge the local state, and install the snapshot as the admitted generation's content, whatever local file exists.
- Publish the generation before the seed is marked
placedand its snapshot deleted.
4. --from: a copy of a sibling (#260)
carlos instances create --app <app> --host <new> --from <host> [--at <txid>] [--environment <env>] [--channel <c>]The gate (codex 17). The door authorizes through cliRoute with the instance role, plus the membership check for people. It does not call dbGate, which accepts only people (apidb.go:110); the db routes' own refusal of every service role (authcli.go:690) stays as it is. The source checks inside dbGate are factored into resolveDBSource:
- the record exists;
- same account and app;
- not pooled, and not on a fleet;
- residency;
- the versioned claim read when claims are on.
Both doors call it.
The console's steps:
- Apply
instances create's existing checks to the new host, and the seed floor (§3.3). - Resolve the source with
resolveDBSource. - Refuse a source and target of different apps or accounts (409). Customer data never leaves its account by this door.
- Plan the restore (codex 18).
RestoreTogains a target TXID, its zero value meaning "latest", and passes it through to litestream'sRestoreOptions.TXID(litestream v0.5.14db.go:2823-2824). It returns the TXID actually restored. An--atolder than the replica's retained history is refused with "history before <txid> is no longer kept".
--at <time> is left out of this version (codex R2-18). A timestamp restore needs each LTX file's transaction time, and the db door's client ignores useMetadata (dbdoor/client.go:78). The console listing asks for false (apidb.go:273), so a time would be matched against S3's modification time instead. Adding times means carrying metadata through both, which is its own change.
- Admit the restore (codex 19, R2-19). Before downloading anything: - Work out the restored size exactly. Read the header of the last planned LTX file, with a ranged GET of its first page. Its
Commit(the database's page count at that TXID) times itsPageSizeis the size of the database the restore will produce. The compressed sizes of the LTX files say nothing about that. - Refuse with 413 if the restored size is over 4 GiB. - Reserve scratch. Take the restored size plus the sum of the planned LTX sizes from the scratch reservation pool.
The scratch reservation pool. One in-process counter in the console, shared by --from restores and import spools (§5.2): - the pool's capacity is the scratch filesystem's free space at console start, less 1 GiB, re-read whenever the pool is empty; - a reservation that doesn't fit is refused with 507; - reservations are released on every exit path.
Restores and spools are also limited together to one at a time per app and two across the deployment, with 429 beyond that. A restore runs under a 30-minute deadline. While it runs, a watcher stats the output every second and cancels the restore if the file grows past the restored size it reserved.
- Restore and check. Restore with
RestoreTointo a private 0700 directory under the console's scratch,<console data>/scratch/seeds/. RunCheckDBContextunder the same deadline. This never reads the live file. - Write (codex R3-2), in this order: 1. Reserve the record. This happens before the restore, right after step 3. Create the record with seed
reserving, kindreplica, its seedIDand the source. Nothing mints areservingrecord, and the sweep leaves its snapshot alone (§3.1). 2. Upload. Writeseeds/<new>/<id>/snapshot.dbwithPutFromfrom the restored file (seekable, under S3's 5 GB single-PUT limit). 3. Publish. CAS the record fromreservingtoready, with the restored TXID, sha256 and bytes.
If the door fails after step 1, it CASes the record to aborted, which the box (or the sweep) cleans up as for an import that never swapped.
Cleanup. The scratch is deleted on every exit path: success, refusal, error, deadline, and client disconnect. The console also empties scratch/seeds/ when it starts.
Environment. The new instance gets its own. With private storage, it is always the host-derived one (see Storage below). Without private storage, it is --environment if given, or the app's default otherwise. Never the source's per-host environment, which carries host-keyed values such as the origin and the cookie domain (the warning in instances_cli.go:235-245).
Storage (codex R5-1). An app with private storage must get a new, empty store, and an ordinary create doesn't guarantee that. It keeps an explicit environment (apiinstances.go:1096-1105), and stores are provisioned per account, app and environment (objstore.GrantIn, instanceprivatestore.go:250). So two instances that name one environment share its store (#408 §5). For an app with private storage, both seed doors (--from here, and import in §5.1) therefore:
- refuse an explicit
--environmentwith 409, "a seeded instance gets its own environment and an empty store". They always use the new host's derived environment (rec.Slug). - refuse if a store already exists for that derived environment, with 409 naming it. A store left behind by a deleted instance of the same host is never inherited. The one exception is a retry of the same seed: the store's grant records the seed
IDthat created it, and a retry with thatIDreuses it. - leave every existing store and binding untouched.
For an app without private storage, --environment works as it does for create. There's no private store to share.
5. The import door (#261)
carlos instances import --app <app> --host <host> --file <snapshot.db> --replaces <addr> [--environment <env>] [--verify <path>]The CLI drives the calls below, and titogo's conversion tool makes the same calls with its service credential. The CLI's --verify <path> makes a GET with the verify token after the swap and commits on a 2xx. Without it, the CLI stops after the swap and prints the token and the commit command.
5.1 Begin
POST /api/cli/apps/{a}/{app}/instances with "import": {"sha256": "...", "bytes": N, "replaces": "<addr>"} plus the ordinary create fields.
The console checks:
- the caller (§2 ruling 5);
- the seed floor (§3.3);
- that the host isn't already a platform record;
- the storage rules in §4 (an app with private storage gets the derived environment and a new, empty store);
bytesis at most 4 GiB.
The console can't see a box's registry, so the box checks the existing row at the swap (§5.4), and the box is the authority on it.
The console mints a verify token (32 random bytes). It answers 201 with the token. This is the only time the token appears; the record keeps only its hash. The record is written with seed awaiting-upload.
5.2 Upload (codex 19, codex 20)
PUT …/instances/{host}/import, streamed. Before reading the body, the console reserves bytes from the scratch reservation pool (§4 step 5), under the same per-app and deployment limits. If the reservation is refused, it answers 507 or 429. Then:
- It spools the body to
<console data>/scratch/seeds/<host>.dbunderhttp.MaxBytesReader(bytes), hashing as it writes. - A length or hash that differs from the record is refused with 422.
- It runs
CheckDBContexton the spooled file, under a 30-minute deadline. Not a database, or failing integrity, is refused with 422. - It writes
seeds/<host>/<id>/snapshot.dbwith onePutFromfrom the spooled file. A seekable file under 5 GB needs no multipart upload, so there's no unfinished multipart upload to clean up and no new IAM action. - It deletes the spool, and the seed becomes
uploaded.
The spool is deleted on every exit path, and on console start.
The snapshot must be consistent, and taken after the app has stopped the instance (VACUUM INTO once its supervisor confirms the process is down). The console can't see that; it can only check the file.
5.3 Stopped
POST …/instances/{host}/import/stopped is the app saying its own process is down. The seed becomes ready, and the console bumps the serial.
Until the swap, the app's own row keeps answering, so the app's router must answer 503 for a stopped instance.
5.4 The swap, the gate, and commit (codex 10, 11, 12, 14, 15; R2-3)
Reconcile on the box finds a record with seed ready and Replaces set, and an existing row for the host. Today such a row is left alone ("exists but is not the route this record describes", reconcile.go:1158, :1287), and that stays the rule for every other case.
The journal comes first. Before any of the predicates below, reconcile checks the box's import journal (§5.7) for the host. A journal row means this box has already swapped, so the legacy-row predicates are never applied to a row the box itself replaced (codex R2-3).
This one case swaps instead, in one registry transaction, only if every condition holds. If any fails, the seed becomes refused with the condition named.
The row's shape (codex 10, R1-10). Registry kinds are instance, service, static and sidecar, and the backing is a separate set of fields (registry.go:31-40). The row must be:
- kind
instanceorservice; - backed by
AddrorAddrs, with an emptySocket; - without
Provisionedfrom any record; - with an addr equal to
Replaces.
The row must also carry none of the columns shapeMismatch refuses adoption over (reconcile.go:598-610): Backing, Tunnel, Caps, Platform or Canary. These are box-scoped or owned by the domains pass, and no import takes them over.
Ownership: the row's app equals the record's app, and the row's account is either:
- the record's account; or
- the deployment's operator account, with an operator assignment (§6.4) that maps that app name to the record's account and app.
Layout (codex 12). The box's layout intent and probe must say nested (mintLayoutReader). The import branch derives the new row's paths with nested = true explicitly. It doesn't use NestedPaths(existing), which would read the addr row as flat (reconcile.go:1103-1107).
The swap transaction (codex 14). In one registry transaction, the box:
- writes a journal row (§5.7): the host, the seed
ID, the complete original row, the record key, stateswapped, and the time; - replaces the row with the exec row the record describes, filed under the record's account, with
ImportGateset to the record'sVerifyHash.
Then it CASes the record's seed from ready to swapped.
Crashes. A crash before the transaction leaves the addr row untouched. A crash after it, but before the record CAS, is resumed on the next pass: the journal row names this seed ID and the record still says ready, so reconcile only redoes the record CAS (codex R2-3).
With claims off, which is the only case built here, there's no claim CAS. The claim-armed refusal (§3.4) covers the claim case until #250.
The import gate (codex 15). While a row has ImportGate set:
- the edge, and the edge door, answer 503 with
Retry-After: 5to every request except one carryingCarlos-Import-Verify: <token>, whose sha256 matches; - the schedule runner sends no ticks to the row;
carlos runrefuses it (run spec §5.1 step 9).
So until commit, only the app's own verify requests reach the instance, and nothing ordinary has been served. The first verify request wakes the instance, and §3.4 places the snapshot.
Commit. POST …/instances/{host}/import/commit is a record CAS that moves the seed from placed to committed. Commit is refused before placed, because nothing has been verified yet. Reconcile then clears ImportGate and deletes the journal row in one transaction.
5.5 Verify
verify (moves spec §6.5) is how titogo knows the adopted instance serves. Titogo sends its verify path through the edge with the verify header. If it passes, it commits. If it fails, it aborts.
5.6 Abort (R2-4, R2-7)
POST …/instances/{host}/import/abort is a record CAS from any state before committed to aborted. It is refused while the record has a non-terminal ActiveRun, although run admission already refuses uncommitted imports. A commit and an abort race on the record's ETag, and exactly one wins. After committed, abort is refused; carlos instances delete is the verb then.
Deleting an uncommitted import is refused (codex R2-7). DELETE …/instances/{host} on a record whose seed is an import that isn't committed answers 409, "abort the import instead". The only exceptions are awaiting-upload, uploaded and refused, where no box has swapped, so the console moves the record straight to rolled-back, deletes seeds/<host>/<id>/, and finishes as below.
The record is kept until everything is clean. An aborted record stays in the bucket while the box rolls back. The box then CASes it to rolled-back, never deleting it. A box that never swapped (there's no journal row for this seed ID) deletes seeds/<host>/<id>/ and does the same CAS.
The console finishes (codex R6-1). For a rolled-back record, the console's abort handler, or failing that its deadline sweep:
- Tears down the private store this seed created, if there is one. The store record for the derived environment is torn down with the existing
objstore.TeardownIn(walk.go:559-566), but only if its grant names this seedID(§4 Storage). A store without thatIDwas there before the seed, and is never touched. - Deletes the record by CAS, if it still names this seed
ID.
The record holds the hostname until the store is gone, so a re-import of the same host can't meet the leftover store that §4 would refuse. The same finish runs for a --from that fails after creating its store, from its aborted record (§4 step 7).
Rollback. Reconcile runs it for any journal row whose import is abandoned, meaning any of these:
- the record is
aborted; - the record's seed
IDno longer matches the journal's; - the record is gone altogether.
Rollback is driven by the journal, not the record, so it doesn't depend on the record still existing (codex R2-7). Each step is idempotent, and the journal's state says where to resume after a restart:
- Close the gate durably. In one transaction: - set the journal row to
rolling-back, with a fence generation; - set the row'sImportGateto a closed sentinel that matches no token.
From here on the edge refuses even verify requests, and that survives an agent restart.
- Fence and stop (codex R2-4, R3 open). This follows the layout pass's handling of the same API (
internal/layout/pass.go:654-680): - Ours already? IfFenceGeneration(host)equals the journal's fence generation, the fence is this rollback's own, from an earlier pass. Carry on with it. - Otherwise, callFenceAndStopIfUnfenced(host, generation). It refuses any fence that already exists, including one from an earlier attempt (maintenance.go:103-106), which is why the check above comes first. -ErrFenceHeldfor a fence that isn't ours means another operation holds the host. Defer to the next pass. -ErrStopUnproven. A cold host has no live entry, and a host never woken has no stop history (maintenance.go:113-121). Proceed only if all of these hold, as the layout pass does: - the box swept its instances at agent start; -Quiesced(host)is true; -StopUncertain(host)is false.
Otherwise lift this attempt's fence and defer.
Deferring is safe. The closed gate from step 1 still refuses every request, ticks are skipped for a gated row, and runs are refused, so nothing can wake the instance before the next attempt.
- Remove the local data through
instancefs: the database,-wal,-shm,<db>.seedand litestream's local metadata (codex R2-1). - Restore the route. In one transaction, put the journal's saved row back exactly as it was, and set the journal row to
cleanup-pending(codex R3-1). - Lift the fence.
- Clean the bucket. Delete the host's
instances/replica, if any, andseeds/<host>/<id>/for this seedIDonly. - Delete the journal row.
- CAS the record to
rolled-back, if it still names this seedID. The console finishes from there (store teardown, then record delete). The record goes last of all (codex R4-1, R6-1). Until then it holds the hostname, andCreateRecord's create-only write refuses a new import or--fromfor that host (provision.go:845). So no second seed can arrive while this one's cleanup is still running. A crash after step 7 leaves anabortedrecord with no journal row, and §3.2's no-journal path (delete this seed's snapshot, then the CAS) finishes the box's part idempotently.
After a restart in cleanup-pending, the box resumes at step 6. Its deletes are scoped to this seed ID, and the record still holds the hostname. It doesn't swap, fence or stop anything: the route is the restored legacy row, and its process is the app's own. Until the journal row is gone, the host's leftover replica is this box's to delete. So a retried import never meets the creation gate's "replica data remains" refusal (apiinstances.go:1146).
Why abort loses nothing. Under the gate, nothing but verify requests reached the instance, so it holds no writes the old process never saw. After step 1, not even those reach it.
5.7 The import journal
The journal is a box-local registry table, import_journal, replacing rev 2's replaced_routes. It has one row per host, with these columns:
- host;
- seed
ID; - the original row as JSON;
- record key;
- state:
swapped,rolling-backorcleanup-pending; - fence generation;
- time.
Reconcile reads it before the bucket records, on every pass and at agent start, and acts on it per §5.4 and §5.6.
A journal row is deleted either in the same transaction that clears the gate at commit, or as the last step of a rollback, after the bucket and the record are clean. So while an import is in flight, the box always knows what it replaced, whatever has happened to the record.
5.8 Moves wait for placement (codex R4-2)
A seed exists only as a snapshot in the home bucket until its first wake places it. A move before then would carry no replica and no snapshot: the legacy move worker treats an empty replica as ready (movespass.go:815, :919). So a move is refused:
- for a
--frominstance until its seed isplaced; - for an import until it is
committed.
The refusal is applied in two places:
- In the move door, under the record CAS. This covers both the legacy door (
apiinstancemove.go:399, throughprovision.go:1146) and the moves series' request door. - In the move worker, before any effect. A record that slips past an old console is still refused.
The message is "wake the instance once before moving it" for a --from instance, and "commit or abort the import first" for an import. Moving a seed before it is placed would mean handing the seed, and an import's rollback, to another region. That isn't offered.
6. Who may, and what is checked
6.1 Callers
- People who are members of the account.
- Service credentials with the
instancerole, checked per route bycliRoute.
The same rule covers --from, every import call, commit and abort.
6.2 Same app, same account
--fromcopies only between instances of one app in one account.- An import creates an instance of the app the route already belongs to, under the rules in §5.4.
6.3 Layout and claims
- At the console. Both doors refuse unless the region's layout report shows the target box nested (
apistatuslayout.go), and the box doesn't report claim control armed. - At the box. It refuses again: in §3.4 unless
ClassifyanswersLayoutNested, and in §5.4 unless the layout intent is nested.
A flat box is ruled out because a seeded instance there would be one uid among every tenant. For Tito's controld instances, it would also lose the 0700 titogo protection that is the only thing keeping them safe today (#126).
6.4 The operator assignment (G17, codex 9)
A direct-mode carlos add files a route under the deployment's operator account, not the app's own. Tito's rows are account=bab, for app bae/titogo. An app name alone doesn't say which account it means: the operator account's titogo row could belong to any account's titogo. So the exception needs an operator to say which.
carlos system legacy-routes assign --name titogo --to bae/titogoWho may. This is an operator-only door (CARLOS_SYSTEM_OPERATORS).
What it writes. An entry in control/legacy-route-owners.json, which maps an app name, as filed under the operator account, to one account and app. A name can have only one entry. Re-assigning it is refused while any import under the old entry is between ready and committed.
How the swap uses it. The swap accepts an operator-filed row only when this file maps the row's app name to the record's account and app. The swap files the new row under the app's account, which fixes G17 for every converted host.
The swap accepts nothing wider:
- no row of another app;
- no row filed under a third account;
- no row that isn't addr-backed;
- no row whose addr has changed since
Replaceswas recorded.
7. What a member sees
In carlos instances list, a seeded instance shows one of:
- "seeded from <host> at txid N";
- "imported <first 12 of sha256>";
- "import awaiting upload", "awaiting stop", "swapped, awaiting verify", or "refused: <reason>".
Audit rows:
instance-seed, naming the source host and TXID (or the sha256 and bytes) and the caller;import-uploaded,import-stopped,import-committedandimport-aborted;legacy-route-assign.
Errors. Each one has its own message:
- other app or other account;
- other region;
- box not nested;
- claim control armed;
- too large;
- no scratch space;
- history no longer kept;
- hash or length mismatch;
- not a database;
- boxes below the seed floor;
- commit before placed;
- abort after commit;
- delete of an uncommitted import (abort it instead);
- move before placement or commit;
- abort while a run is open.
Some come back as the swap's refusal reasons instead:
- no addr row;
- addr changed;
- row of another app;
- operator row with no assignment.
Every CLI and console string goes through copy review before it's written into the code.
8. Security
- Data never leaves its account. Neither door crosses an account or an app.
- Uploads are pinned. An upload is pinned to one record, one length and one hash. The console checks the hash as it spools the file, and the box checks it again before it trusts the file.
- The verify token never sits on any box or bucket. Only its hash does. Without the token, nothing ordinary reaches an un-committed import.
- No standing identity gains a read. The console already reads the home region's replicas for
db pull. Boxes read only their own region's bucket, andseeds/is in it. - A forged record is still the main risk. A seed is a record field, and #171 means any compromised box can forge a record. A forged seed could start a new instance from a snapshot already in the bucket, or a forged assignment file could widen §6.4. Both are the existing record-authority boundary (G13 in the moves series), and #171's fix bounds them.
- The swap only narrows the "leave it alone" rule. It narrows it for a row named by its exact addr, of the same app, under the app's account or an operator-assigned one. Every other mismatch is left alone, as today.
9. Testing
- The floor: - no seed state is written until every box reports the floor armed; - a recovery below the floor is refused.
- Box placement: - each failure (length, hash,
CheckDB) fails the wake, and never falls back to an empty database; - a local database whose<db>.seednames an earlier seedID(abort, then re-import) is purged with its WAL, SHM and metadata, and the new snapshot placed; - a crash after the rename but before the first upload resumes from the local file; - a crash before the rename retries from the snapshot; - the snapshot is deleted only after the first upload; - refusal with claim control armed. - Reconcile: - the swap's conditions, each refused on its own, with its reason; - the operator assignment, present, absent and mapping elsewhere; - nested paths derived for the new row; - the original row stored and restored exactly; - rows carrying
Backing,Tunnel,Caps,PlatformorCanaryrefused; -aborteddispatched before the state filter; - a crash between the swap transaction and the record CAS resumes; - rollback for a record that is aborted, replaced by another seedID, or deleted outright; - the rollback fence: a verify request during rollback is refused, stop evidence is required, and the fence is re-installed after a restart; - an idempotent rollback across a crash between any two of its steps, including between the route restore and the replica delete (cleanup-pendingresumes, and never stops the legacy process); - the fence: resumed when it is ours by generation; deferred when it is another operation's; for a never-woken host, accepted only when the box was swept at start and the host is quiesced; ---from'sreservingrecord: the sweep never deletes its snapshot, and abandons it by CAS only after an hour; - abort A, crash between the journal delete and the record delete, then restart: A finishes, and a second import of the host is refused until A's record is gone; - a cleanup never deletes another seedID's snapshot; - a move refused before placement or commit, in the door (both the legacy and the moves series') and in the worker; - The gate: - the edge and the door answer 503 without the token and pass with it; - no ticks while gated; - commit clears the gate; - the commit and abort race.
- Console: - both doors' gates, including
instanceservice credentials anddb pullstill refusing them; ---fromwith--at <txid>, and history no longer kept; - on an app with private storage,--environmentrefused and an existing store for the derived environment refused, except a retry of the same seedID; the source's store and bindings unchanged; - abort and then re-import with private storage on, both before upload and after the swap. The seed's store is torn down and the re-import succeeds. A store whose grant names another seedID, or none, is never torn down. A crash between the teardown and the record delete resumes; - a failed--fromafter store provisioning: the store is torn down and the host can be created again; - restored size from the last LTX header, and the 413; - the shared reservation pool across restores and spools, its 507, and release on every path; - the watcher cancelling a restore that outgrows its reservation; - the per-app and deployment limits shared by both; - delete refused for an uncommitted import, allowed before upload completes; - the seeds sweep: deletes only unreferenced or superseded snapshots, never one a live--fromrecord still needs; - scratch cleanup on every exit path and at start; - every import state transition; - hash, length andCheckDBrefusal; - the audit rows. - CLI: -
instances create --fromandinstances import, with and without--verify, end to end against a fake console; -system legacy-routes assign. - On the dev box: - a
--fromcanary of a real instance; - an import of a stopped snapshot over an addr row, with a verify, then a commit; - a second import aborted after the swap, with the addr row restored and serving. - Cross-compile: darwin and windows builds stay green.
10. Rollout and #261's schedule
- Build both doors now. They refuse on flat and claim-armed boxes, so they're safe to ship before Tito's boxes are nested.
- Release, then publish the seeds floor on each deployment once its boxes run the release.
- Roll the nested layout to Tito's boxes (#247 steps 2 to 5, with Tito's own approvals).
- An operator assigns
titogotobae/titogo(§6.4). - Titogo builds its half (its design §4 and §7): snapshot, import, carry env, verify with the token, commit or abort, repoint, retire; import mirrors that write through the instance; and controld's
retire. - Convert one canary host, abort it once on purpose, then convert it again.
- Convert the rest, a few at a time, with Paul's go.
If the claim is activated (#250) before the conversion finishes, the remaining imports wait for §3.4's claim algorithm.
11. Round 1 dispositions
| Finding | Disposition | |---|---| | codex 8 (P1) warm-file shortcut vs claim restore | Accepted. §3.4: this version is the plain path only, and claim-armed boxes refuse; the claim algorithm is written down for #250 as Astra gave it | | codex 9 (P1) app name doesn't fix the account | Accepted. §6.4: an operator assignment maps the name to one account and app | | codex 10 (P2) Kind == "addr" doesn't exist | Accepted. §5.4: the shape defined by kind and backing fields | | codex 11 (P2) the claim acquisition at the swap | Accepted by scoping. With claims off the swap is one registry transaction; claim-armed boxes refuse (§3.4) | | codex 12 (P2) existing-row branch derives flat | Accepted. §5.4: the import branch derives nested explicitly | | codex 13 (P2) the state filter makes rollback unreachable | Accepted. §3.2: dispatch on seed state first | | codex 14 (P2) Replaces can't restore the row | Accepted. §5.4: the complete row goes in replaced_routes in the swap's transaction | | codex 15 (P1) abort safety not observable | Accepted. §5.4: the import gate with a verify token, and commit as the durable latch; abort only before commit | | codex 16 (P1) downgrade consumes seed records | Accepted. §3.3: the seeds floor, armed with a carlos.prev verdict, before any seed state is written | | codex 17 (P1) dbGate refuses service credentials | Accepted. §4: resolveDBSource is factored out; the db door's own gate is unchanged | | codex 18 (P2) RestoreTo can't honour --at | Accepted. §4 step 4: a target passed to litestream's TXID/Timestamp options | | codex 19 (P2) scratch isn't bounded | Accepted. §4 step 5 and §5.2: admission from the plan's sizes, free-space checks, deadlines, cleanup on every path and at start | | codex 20 (P2) multipart upload | Accepted, differently. §5.2: spool to scratch (which CheckDB needs anyway), then one PutFrom under the 5 GB single-PUT limit; no multipart, no new IAM action |
12. Round 2 dispositions
| Finding | Disposition | |---|---| | R1-10, still open: protected columns | Accepted. §5.4: a row with Backing, Tunnel, Caps, Platform or Canary is refused, as adoption refuses it | | R1-18, still open: timestamps | Accepted by scope. §4 step 4: --at <txid> only in this version; carrying LTX times through the db door is its own change | | R1-19, still open: scratch isn't bounded | Accepted. §4 step 5: restored size from the last LTX header's page count and page size; one reservation pool for restores and spools; shared limits; a watcher that cancels on overgrowth | | R2-1 (P1) re-import resurrects aborted data | Accepted. §3.4: a seed ID and a <db>.seed marker; a local file without a matching marker is purged; rollback removes the database, WAL, SHM, marker and metadata | | R2-2 (P1) run gets past the import gate | Accepted. Run spec §5.1 step 9 and §5.4 here: runs are refused on uncommitted imports, under the record CAS | | R2-3 (P2) a crash between the swap and the record CAS | Accepted. §5.4 and §5.7: the journal is read first, keyed by seed ID; a swapped-but-ready import resumes with the CAS | | R2-4 (P2) a verify wake during rollback | Accepted. §5.6: the gate is closed durably first, then FenceAndStopIfUnfenced with complete stop evidence; the fence is re-installed after a restart | | R2-5 (P1) the lifecycle rule deletes needed seeds | Accepted. §3.1: no lifecycle rule; a console sweep deletes only unreferenced or superseded snapshots, after a terminal abort for stale imports | | R2-7 (P2) delete bypasses rollback | Accepted. §5.6: delete is refused for an uncommitted import; rollback is driven by the journal, including when the record is gone |
13. Round 3 dispositions
| Finding | Disposition | |---|---| | R2-4, still open: the fence retry can't work | Accepted. §5.6 step 2: the layout pass's pattern; our own fence is recognised by the journal's generation, and a cold host's stop is accepted only when the box was swept at start and the host is quiesced | | R3-1 (P2) rollback forgets replica cleanup | Accepted. §5.6 steps 4 to 8: the journal stays as cleanup-pending until the bucket and record are clean; resuming never swaps or stops | | R3-2 (P2) sweep deletes a snapshot mid-create | Accepted. §4 step 7: the record is reserved before the upload; the sweep never deletes a snapshot named by a live record, and abandons a stale reservation by CAS first |
14. Round 4 dispositions
| Finding | Disposition | |---|---| | R4-1 (P1) a resumed rollback deletes the next import's snapshot | Accepted, both fixes. §3.1: snapshot keys carry the seed ID, and every delete is scoped to it. §5.6: the aborted record is deleted last, so it holds the hostname until cleanup is done | | R4-2 (P2) a move strands an unplaced seed | Accepted. §5.8: moves are refused until a --from seed is placed or an import is committed, in the door and in the worker |
15. Round 5 dispositions
| Finding | Disposition | |---|---| | R5-1 (P1) create doesn't guarantee an empty private store | Accepted. §4 Storage: with private storage, both seed doors refuse an explicit environment, use the derived one, and refuse an existing store unless it was created by the same seed ID |
16. Round 6 dispositions
| Finding | Disposition | |---|---| | R6-1 (P2) abort leaves a store that blocks re-import | Accepted. §5.6: the box ends at rolled-back; the console tears down the store only if its grant names this seed ID, then deletes the record, which holds the hostname until then. The same applies to a failed --from |