amadan
specs rev 2: answer Astra round 1 (20 findings, all accepted)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
3 files changed,
+855
−435
docs/superpowers/specs/2026-10-08-carlos-run-design.md+345 −182docs/superpowers/specs/2026-10-08-seeded-instances-design.md+467 −253docs/superpowers/specs/CODEX-FINDINGS-RUN-AND-SEEDED-R1.md+43 −0
CI passed — run details
diff --git a/docs/superpowers/specs/2026-10-08-carlos-run-design.md b/docs/superpowers/specs/2026-10-08-carlos-run-design.md| index 92d23e413..65a5eb76f 100644 |
| --- a/docs/superpowers/specs/2026-10-08-carlos-run-design.md |
| +++ b/docs/superpowers/specs/2026-10-08-carlos-run-design.md |
| @@ -1,23 +1,29 @@ |
| # `carlos run`: a member runs an app command against one instance (#259) |
| -**Status:** rev 1, 2026-10-08. Design agreed with Paul in chat the same day; |
| -this document is for his review, then Astra's. |
| +**Status:** rev 2, 2026-10-08. Rev 1 (c66792a3) was approved by Paul. |
| +Rev 2 answers Astra's round 1 (`CODEX-FINDINGS-RUN-AND-SEEDED-R1.md`; |
| +findings 1 to 7 are this spec's). Each change is marked **(codex N)**, and |
| +§11 lists the dispositions. |
| **Discussion:** amadan carlos/platform #259. Companion spec: |
| `2026-10-08-seeded-instances-design.md` (#260, #261). |
| ## 1. The problem |
| -Every one-off command against a live instance today goes over |
| -`aws ssm send-command` as root on the box. On 2026-10-06 a titogo session |
| -imported 22,462 check-ins that way (titogo PR #4048): it ran the app's own |
| -binary against the instance's database, read a credential from the app's |
| -config, and staged the input file through presigned URLs. The canary recipe |
| -(`titogo add`, `auth grant`, `seed`) has the same shape. |
| +Every one-off command against a live instance goes over |
| +`aws ssm send-command` as root on the box today. On 2026-10-06 a titogo |
| +session imported 22,462 check-ins that way (titogo PR #4048). It: |
| -Two members of Tito's team can sign in to the console as members of `tito` |
| -but cannot do any of this without root on a box every tenant shares. The |
| -platform has no door for "run this app's command against that instance". |
| +- ran the app's own binary against the instance's database; |
| +- read a credential from the app's config; |
| +- staged the input file through presigned URLs. |
| + |
| +The canary recipe (`titogo add`, `auth grant`, `seed`) has the same shape. |
| + |
| +Two members of Tito's team can sign in to the console as members of `tito`, |
| +but they can't do any of this without root on a box every tenant shares. |
| +The platform has no door for "run this app's command against that |
| +instance". |
| ## 2. Rulings (Paul, 2026-10-08) |
| @@ -26,8 +32,8 @@ platform has no door for "run this app's command against that instance". |
| read every instance's database and publish to non-stable channels, so |
| this grants nothing new. |
| 2. **Output transport:** the run's own chunks in the deployment bucket, |
| - polled by the CLI through the console. No new network path; the console |
| - still never connects to a box. |
| + polled by the CLI through the console. There is no new network path; the |
| + console still never connects to a box. |
| 3. **Platform-run instances only.** controld-run (`titogo@`) instances are |
| converted (#261) rather than served. |
| @@ -35,17 +41,18 @@ platform has no door for "run this app's command against that instance". |
| ``` |
| carlos run --app <app> --instance <host> [--file <path>]... [--timeout 30m] -- <arg>... |
| +carlos run --follow <id> |
| ``` |
| -- Runs the release the instance follows, as `/opt/carlos/live/<host> |
| - <arg>...`. The caller never names the binary. |
| -- Runs as the instance: same uid, same namespace, same bind mounts, same |
| - slice, same environment. Never root, never the agent. |
| -- Streams stdout and stderr to the terminal (about one second behind) and |
| - exits with the command's own status. |
| -- Records who ran what, where and how it ended. |
| +- It runs the release the instance follows, as |
| + `/opt/carlos/live/<host> <arg>...`. The caller never names the binary. |
| +- It runs as the instance: the same uid, namespace, bind mounts, slice and |
| + environment. Never root, and never the agent. |
| +- It streams stdout and stderr to the terminal, about one second behind, |
| + and exits with the command's own status. |
| +- It records who ran what, where, and how it ended. |
| -**Defaults** (agreed in chat): |
| +**Defaults:** |
| | | | |
| |---|---| |
| @@ -57,259 +64,415 @@ carlos run --app <app> --instance <host> [--file <path>]... [--timeout 30m] -- < |
| | `--file` | repeatable; 64 MB in total | |
| | concurrency | one run at a time per instance | |
| | layout | refused unless the instance is nested | |
| +| host lifecycle claim | refused while the box's claim control is armed (§3.1) | |
| **Not in this version:** |
| - `--version`. The trampoline execs only `/opt/carlos/live/<host>` |
| (`internal/instanceexec/exec.go:72`), and the control service binds that |
| - link to the instance's own build. Running another build needs a second |
| - exe link and its own review. |
| + link to the instance's own build. |
| - `--stopped` (stop, run, start). |
| - `--record` (keep the output). |
| -- An interactive shell. `run` is argv only, which keeps every run |
| - auditable. |
| +- An interactive shell. |
| + |
| +### 3.1 With the host lifecycle claim armed (codex 2) |
| + |
| +The claim code is on main but not active anywhere. Once a box arms claim |
| +control, two things stop working: |
| + |
| +- instancectl answers 503 to every legacy start, seed and stop route |
| + (`claimAwareLegacy`, `internal/instancectl/server.go:840`); |
| +- `RoleLaunch.Start` refuses every launch until the isolated root protocol |
| + carries reservations (`internal/activator/rolelaunch_linux.go:23-28`). |
| + |
| +So on such a box `carlos run` refuses with `refused: "claim"`, the same way |
| +it refuses on a flat box. Supporting it there is part of the claim's |
| +activation work (#250), not this spec. That work must add `cmd` to the |
| +reservation protocol beside `main`, `seed` and `sidecar`, and must settle |
| +the cmd unit's producer before the generation is published. #250 gets a |
| +note saying so. |
| ## 4. The command's contract |
| -What the app's binary sees, for the `docs/site/cli.md` section "What an |
| -instance is started with": |
| +This is what the app's binary sees. It goes into the `docs/site/cli.md` |
| +section "What an instance is started with". |
| - **argv:** exactly what followed `--`, after the binary. The platform |
| appends nothing, unlike a seed's `--db` (`seedArgv`). |
| -- **Environment:** the same one a seed gets (`seedVars`, |
| - `backend_exec.go:1359`): the app's config bundle with `<host>.env` on |
| - top, `CARLOS_HOST`, `CARLOS_REGION`, `CARLOS_ADMIN_TOKEN`, and no control |
| - socket. Plus: |
| +- **Environment:** a new composer, `cmdVars` (codex 7). It takes the |
| + instance's file environment (the config bundle with `<host>.env` on |
| + top), then: |
| + - removes `CARLOS_CONTROL_SOCKET` and `CARLOS_PURGE`, as `seedVars` does |
| + (`backend_exec.go:1359`); |
| + - sets or removes `CARLOS_REGION` from the backend's region, exactly as |
| + the serving composer does (`backend_exec.go:1340-1344`); |
| + - sets `CARLOS_HOST`, keeps `CARLOS_ADMIN_TOKEN`, and adds the three |
| + variables below. |
| | Variable | Value | |
| |---|---| |
| -| `CARLOS_DB` | the instance's database path, the one `--db` names when the instance is started | |
| +| `CARLOS_DB` | the instance's database path, the one `--db` names when the instance starts | |
| | `CARLOS_RUN_ID` | this run's identifier | |
| -| `CARLOS_RUN_DIR` | a directory holding the `--file` uploads under their base names. Present even with no files; deleted after the run | |
| +| `CARLOS_RUN_DIR` | a directory holding the `--file` uploads under their base names. Always present, even with no files. Deleted after the run | |
| - **Working directory:** the instance's data directory, as for every |
| template (#256 item D). |
| -- **Signals:** SIGTERM on cancel or timeout, SIGKILL 20 seconds later |
| - (`KillMode=mixed`, `TimeoutStopSec=20`, as the other templates). |
| +- **Signals:** SIGTERM on cancel, timeout or an instance stop (§5.3), then |
| + SIGKILL 20 seconds later (`KillMode=mixed`, `TimeoutStopSec=20`). |
| -Secrets reach the command only through its environment, never in argv the |
| -platform builds, and the platform never prints the environment. |
| +Secrets reach the command only through its environment. The platform never |
| +puts them in argv and never prints the environment. |
| ## 5. The flow |
| -### 5.1 Request (CLI → console) |
| +### 5.1 Request (CLI to console) |
| `POST /api/cli/apps/{account}/{app}/instances/{host}/runs`, multipart: a |
| -`meta` part (`argv`, `timeout`, file names and sizes) and one part per file. |
| -The console streams file parts straight to the bucket (the F8 rule: bytes |
| -go through the console, never a presigned URL) under a request-wide |
| -`http.MaxBytesReader` of 64 MB plus the meta allowance. |
| +`meta` part (`argv`, `timeout`, file names and sizes), then one part per |
| +file. The console streams the file parts straight to the bucket (the F8 |
| +rule: bytes go through the console, never through a presigned URL). The |
| +whole request is capped by `http.MaxBytesReader` at 64 MB plus the meta |
| +allowance. |
| The console refuses, in this order, each with its own status and message: |
| 1. A service credential (`cliRoute` with the zero `svcrole.Set`). |
| 2. Not a member of the account: the byte-identical 404 `apiAccountApp` |
| gives. |
| -3. Budget (429), on the instance verbs' per-token window. |
| +3. Over budget (429), on the instance verbs' per-token window. |
| 4. No record for the host, or a record of another app: 404. |
| -5. The record is not platform-run exec, or is pooled, or is placed on a |
| +5. The record isn't platform-run exec, or is pooled, or is placed on a |
| customer's fleet: 409. |
| 6. A move is in flight (`move` state other than none): 409. |
| 7. A run is already open on the instance: 409, naming its id and caller. |
| -8. argv empty, longer than 256 arguments or 64 KB in total, or holding a |
| - NUL byte; timeout outside 1s to 6h; files over the cap or with names |
| - that are not a plain base name: 400. |
| +8. Bad input, 400: |
| + - argv empty, more than 256 arguments, over 64 KB in total, or |
| + containing a NUL byte; |
| + - a timeout outside 1 second to 6 hours; |
| + - files over the cap, or a file name that isn't a plain base name. |
| -It then writes, under the app's own subtree (`release.AppPrefix`, so the |
| -owner gate maps onto it and app trash and purge sweep it): |
| +If none apply, it writes these under the app's own subtree |
| +(`release.AppPrefix`, so the owner gate maps onto them and app trash and |
| +purge sweep them): |
| - `runs/<id>/in/<name>` for each file; |
| -- `runs/<id>/request.json`: host, argv, timeout, file list with sizes and |
| - sha256, caller, requested-at; |
| -- `runs/index.json`: the app's open runs (id, host, state), updated by |
| - conditional PUT on its ETag; |
| - |
| -bumps `control/serial`, writes the audit row (§6), and answers `202` with |
| -the run id. |
| - |
| -### 5.2 Pickup (box) |
| - |
| -`runs/index.json` follows `carlos restart`'s pattern (`internal/host/ |
| -restart.go`): it is read only when the serial moves, and a box acts only |
| -when the index's ETag differs from the one it last saw. A box with nothing |
| -happening pays nothing beyond its existing serial GET. |
| - |
| -A box acts on a run only for a host whose exec row it owns: the registry |
| -row is on this box and, with the host lifecycle claim on, this box holds |
| -the claim. Every other box ignores it. The owning box: |
| - |
| -1. Writes `runs/<id>/picked.json` (box, time). The console reports |
| - "not picked up" if this is absent 60 seconds after the request. |
| -2. Refuses unless `activator.Classify` answers `LayoutNested` for the row |
| - and `nestedGate` passes. A refusal is a `result.json` with |
| - `refused: "flat"`, never a fallback to running the command as the |
| - agent. |
| -3. Wakes the instance if it is hibernated, through the ordinary wake path, |
| - so the database is restored and replication is running. A wake failure |
| - is a `result.json` with `refused: "wake"` and the wake's error. |
| -4. Takes a **run hold** on the instance: while it is held, hibernation |
| - does not stop the instance and a move request is refused. Without it, |
| - a hibernation during the run would stop replication and lose the |
| - command's last writes. |
| -5. Downloads the files into |
| - `/data/instances/<host>/.carlos-run/<id>/`, checking each against the |
| - size and sha256 in `request.json`, mode 0700, owned by `carlos`. |
| - |
| -### 5.3 The unit |
| - |
| -A fourth template, `carlos-instance-cmd@.service`, copied from |
| -`carlos-instance-seed@.service` with every sandbox line unchanged and a |
| -test that fails if the two ever differ outside the lines below: |
| +- `runs/<id>/request.json`: host, argv, timeout, the file list with sizes |
| + and sha256, the caller, and the time requested; |
| +- `runs/index.json`: the app's open runs (id, host, state, cancel flag), |
| + updated by a conditional PUT on its ETag. |
| + |
| +It then bumps `control/serial`, writes the `run-start` audit row (§6), and |
| +answers `202` with the run id. |
| + |
| +**Run and move are serialized at the console.** A move request for a host |
| +with an open run in `runs/index.json` is refused (409, naming the run). A |
| +run request for a moving host is refused (step 6). Both checks read the |
| +record and the index at their current ETags, and the write that wins |
| +carries the ETag it read. |
| + |
| +### 5.2 Pickup on the box (codex 3) |
| + |
| +Restart's pattern (`internal/host/restart.go`) records the first sight of |
| +an object and never acts on it (`:114`, `:165`). That's right for restart |
| +and wrong here: it would drop runs queued before a box started, or before |
| +it came to own a host. So pickup works like this instead: |
| + |
| +- **When the box looks.** A box reads `runs/index.json` for each app it |
| + hosts at agent start, whenever it newly owns a host's exec row, and |
| + whenever the serial moves and the index's ETag differs from the last one |
| + it read. An idle box still pays nothing beyond its serial GET. |
| +- **Which runs it takes.** It acts on every open run for a host whose exec |
| + row it owns. Every other box ignores the run. |
| +- **Taking a run.** The box writes `runs/<id>/picked.json` (box, agent boot |
| + id, time) as a **create-only** PUT (`release.Store.Put` with the |
| + create-only precondition, `store.go:767-777`). If the create fails, some |
| + box already took the run, and this one leaves it alone. |
| +- **A run this box picked but never finished.** At agent start, the box |
| + checks each open run whose `picked.json` names this box with an earlier |
| + boot id. That run's unit is gone: `BindsTo=carlos-edge.service` stopped |
| + it with the edge. The box writes its `result.json` as |
| + `interrupted: "agent restart"`. **A picked run is never executed |
| + again.** |
| +- **No pickup.** The console reports "not picked up" if `picked.json` is |
| + still missing 60 seconds after the request. |
| + |
| +Once the box has taken the run: |
| + |
| +1. **The layout check.** It refuses unless `activator.Classify` answers |
| + `LayoutNested` for the row and `nestedGate` passes. A refusal is a |
| + `result.json` with `refused: "flat"`, never a fallback to running the |
| + command as the agent. With claim control armed it is |
| + `refused: "claim"` (§3.1). |
| +2. **The wake.** It wakes the instance if it is hibernated, through the |
| + ordinary wake path, so the database is restored and replication is |
| + running. A wake failure is `refused: "wake"`, with the wake's error. |
| +3. **The files.** It downloads the files into |
| + `provision.InstanceDir(dataDir, host)/.carlos-run/<id>/` (codex 6: |
| + long hosts hash, `provision.go:2032-2045`). Each one is checked against |
| + its size and sha256 in `request.json`, written mode 0700 by `carlos` |
| + through `instancefs`, and no symlink is followed. |
| +4. **The launch.** It attaches the run to the instance's activator entry |
| + (§5.3), then starts the unit. |
| + |
| +### 5.3 The run is part of the instance's writer lifecycle (codex 1) |
| + |
| +The command writes the instance's database, so it has to be inside every |
| +rule that protects those writes, not beside them. The activator entry for |
| +the host gains a `cmd` field: the run id, the unit's InvocationID, and a |
| +done channel. Then: |
| + |
| +- **Every way the instance stops** first stops the cmd unit and waits for |
| + it to settle, before replication is closed or the entry is forgotten. |
| + That covers: |
| + - hibernation (`stop`, `activator.go:2108`); |
| + - `Bounce` (`:1425`), which config sync and `carlos restart` use; |
| + - convergence that stops an instance; |
| + - a serving process that fails; |
| + - the move freeze; |
| + - agent shutdown. |
| + |
| + The ordering lives in `finishStop` (`:2355`), so a new stop path can't |
| + skip it. A run stopped this way ends as |
| + `interrupted: "<reason>"`, and everything it committed before the stop |
| + was replicated before replication closed. |
| +- **Hibernation's idle timer doesn't fire** while `cmd` is set. That's |
| + the run hold. The other stops above aren't held off; they interrupt the |
| + run. |
| +- **A restore never runs under the command.** The cmd unit starts only for |
| + an entry that is awake and replicating. While the entry is waking or |
| + stopping it isn't started, and the box waits up to 60 seconds for the |
| + wake. |
| +- **A wake failure, or a stop before launch,** ends the run as |
| + `refused: "wake"` or `interrupted`. It is never queued for later. |
| + |
| +### 5.4 The unit (codex 4) |
| + |
| +A fourth template, `carlos-instance-cmd@.service`, is copied from |
| +`carlos-instance-seed@.service`. Every sandbox line stays the same, and a |
| +test fails if the two ever differ outside these lines: |
| - `Description=carlos exec-backed instance %I (cmd)` |
| - `ExecStart=/opt/carlos/carlos instance-exec cmd %I` and |
| `ExecStopPost=/opt/carlos/carlos instance-exit cmd %I` |
| -- `TimeoutStartSec=6h5m`, a backstop past the 6-hour cap; the agent |
| - enforces the run's own timeout |
| +- `TimeoutStartSec=6h5m`, a backstop past the 6-hour cap. The agent |
| + enforces the run's own timeout. |
| - `StandardOutput=file:/run/carlos/logs/%I.cmd.sock` and |
| `StandardError=file:/run/carlos/logs/%I.cmd-err.sock`, two listeners so |
| - the streams stay separate |
| + the streams stay separate. |
| + |
| +`Type=oneshot` and `RemainAfterExit=yes` stay, so the result is kept until |
| +the agent reads it, as for the seed. |
| + |
| +`instancectl` gains `RoleCmd = "cmd"` and |
| +`UnitPrefixCmd = "carlos-instance-cmd@"` (`internal/instancectl/names.go`; |
| +`run` is already the serving role). The control service starts, stops and |
| +sweeps the new unit like the other three, through two routes: |
| + |
| +- **`POST /cmd`** starts the unit and answers at once with its |
| + InvocationID. (`/seed` blocks for up to 150 seconds; this one doesn't.) |
| +- **`GET /cmd/{host}?invocation=<id>`** is a bounded long-poll, at most 30 |
| + seconds. It answers `{state, InvocationID, Result, ExecMainStatus, |
| + Code}`, where `state` is `running`, or `done` once that invocation's |
| + main process has exited. `done` covers both "active (exited)" and |
| + "failed". |
| + |
| +When `GET` answers `done`, the agent records the result, then calls the |
| +existing `/stop` for the host's cmd role and waits for the unit to settle, |
| +exactly as `/seed` does (`server.go:431-448`). An invocation id that |
| +doesn't match is `gone`, and the run ends `interrupted`. |
| + |
| +The agent writes `launch.cmd.json` through `writeLaunch`, with argv |
| +`[/opt/carlos/live/<host>, <arg>...]` and the environment from §4. |
| -`instancectl` gains `RoleCmd = "cmd"` (`internal/instancectl/names.go`; |
| -`run` is the serving role) and `UnitPrefixCmd = "carlos-instance-cmd@"`, so |
| -the control service starts, stops and sweeps it like the other three. Its |
| -`POST /cmd` starts the unit and answers at once, unlike `/seed`, which |
| -blocks for at most 150 seconds; the agent then waits on the unit with |
| -bounded `GET /cmd/{host}` calls that answer `{active, InvocationID, |
| -Result, ExecMainStatus, Code}`. The agent writes `launch.cmd.json` through |
| -`writeLaunch`, with argv `[/opt/carlos/live/<host>, <arg>...]` and the |
| -environment in §4. |
| +### 5.5 Output (codex 5) |
| -### 5.4 Output |
| +The agent reads the two sockets and writes `runs/<id>/out/<seq>-<stream>`, |
| +where stream is `o` or `e`. The seq is zero-padded and shared across both |
| +streams, so the CLI can interleave them in the order the agent read them. |
| +It writes a chunk every second or every 256 KB, whichever comes first. |
| -The agent reads the two sockets and writes |
| -`runs/<id>/out/<seq>-<stream>` (stream `o` or `e`; seq zero-padded, |
| -shared across both streams so the CLI can interleave them in the order the |
| -agent read them) every second or every 256 KB, whichever comes first. |
| -Total output is capped at 64 MB; past the cap the agent stops writing |
| -chunks and writes one `truncated` marker, and the command keeps running. |
| +Total output is capped at 64 MB. Past the cap the agent stops writing |
| +chunks and records `truncated`; the command keeps running. |
| -When the unit becomes inactive the agent writes `runs/<id>/result.json`: |
| +`result.json` is written **only after every chunk is durably written**, |
| +and it names the last seq: |
| ```json |
| {"exit": 0, "signal": "", "timed_out": false, "cancelled": false, |
| - "started": "...", "ended": "...", "box": "...", "chunks": 17, |
| - "truncated": false, "refused": ""} |
| + "interrupted": "", "refused": "", "started": "...", "ended": "...", |
| + "box": "...", "last_seq": 17, "truncated": false} |
| ``` |
| -then deletes the run directory and releases the run hold. |
| +The agent then deletes the run directory and clears the entry's `cmd`. |
| -### 5.5 Following (CLI ↔ console) |
| +### 5.6 Following (CLI and console) (codex 5) |
| -`GET …/runs/{id}?after=<seq>` answers the chunks after `seq` (bounded per |
| -response) and `result.json` once it exists. The CLI polls it every second, |
| -writes `o` chunks to stdout and `e` chunks to stderr, and exits with: |
| +`GET …/runs/{id}?after=<seq>` answers the chunks after `seq` (a bounded |
| +page) and `result.json` once it exists. The CLI: |
| + |
| +- polls every second; |
| +- writes `o` chunks to stdout and `e` chunks to stderr; |
| +- keeps reading pages until it has every chunk up to `result.last_seq`; |
| +- then sends `POST …/runs/{id}/ack` with `last_seq`. |
| + |
| +The console refuses an ack whose `last_seq` doesn't match the result. The |
| +CLI exits with: |
| | case | exit | |
| |---|---| |
| | command exited | its own status | |
| | command killed by a signal | 128 + the signal number | |
| | timed out | 124 (`timeout(1)`'s) | |
| -| refused by the box (flat, wake) or not picked up in 60 s | 3 | |
| +| interrupted by an instance stop or agent restart | 125 | |
| +| refused by the box (flat, claim, wake) or not picked up in 60 s | 3 | |
| | refused by the console | 2 | |
| -### 5.6 Cancel and timeout |
| +### 5.7 Cancel and timeout |
| + |
| +When you press Ctrl-C: |
| -Ctrl-C makes the CLI send `POST …/runs/{id}/cancel`. The console sets |
| -`cancel` on the run in `runs/index.json` and bumps the serial; the owning |
| -box stops the unit. A second Ctrl-C exits the CLI without waiting; the |
| -cancel still happens. The agent stops the unit itself at the run's |
| -timeout. Both end in an ordinary `result.json`. |
| +1. The CLI sends `POST …/runs/{id}/cancel`. |
| +2. The console sets `cancel` on the run in `runs/index.json` and bumps the |
| + serial. |
| +3. The owning box stops the unit. |
| -If the CLI goes away without cancelling, the run continues to its end or |
| +A second Ctrl-C exits the CLI without waiting; the cancel still happens. |
| +At the run's timeout the agent stops the unit itself. All of these end in |
| +an ordinary `result.json`. |
| + |
| +If the CLI goes away without cancelling, the run carries on to its end or |
| its timeout. `carlos run --follow <id>` attaches to an open or finished run |
| -and prints everything not yet acknowledged. |
| +and prints every chunk not yet acknowledged. |
| + |
| +### 5.8 Cleanup |
| -### 5.7 Cleanup |
| +- **The ack.** After a valid ack (§5.6) the console deletes `runs/<id>/` |
| + and drops the run from the index. |
| +- **The console sweep.** It runs in the existing deadline-sweep loop and: |
| + - deletes any run whose result is more than 24 hours old, whether or not |
| + it was acknowledged; |
| + - closes any run with no result once its timeout has passed by an hour, |
| + writing the audit row's end as `lost`. |
| -- When the CLI has the result it sends `POST …/runs/{id}/ack`; the console |
| - deletes `runs/<id>/` and drops the run from the index. |
| -- A console sweep (the existing deadline-sweep loop) deletes any run whose |
| - result is more than 24 hours old, and closes any run with no result |
| - whose timeout has passed by an hour, writing the audit row's end as |
| - `lost`. This replaces the 24-hour bucket lifecycle rule named in chat: |
| - the runs live under each app's subtree, which a prefix rule cannot |
| - match, and the sweep needs no infrastructure change. |
| -- The box sweeps `.carlos-run/` directories with no open run at agent |
| - start. |
| + The 24-hour bucket lifecycle rule from chat is replaced by this sweep. |
| + The runs live under each app's subtree, which a prefix rule can't match, |
| + and the sweep needs no infrastructure change. |
| +- **The box sweep.** At agent start, the box deletes any `.carlos-run/` |
| + directory with no open run. |
| ## 6. The record |
| -Two audit rows through `Server.auditActivity`, target the instance: |
| +Two audit rows are written through `Server.auditActivity`, with the |
| +instance as the target: |
| -- `run-start`: caller, app, host, argv, file names and sizes, the release |
| - version the instance follows, run id. |
| -- `run-end`: run id, exit, signal, timed out, cancelled, started, ended, |
| - box; or `lost`. Written by the console the first time it reads |
| - `result.json`, whether from the CLI's poll or the sweep. |
| +- `run-start`: the caller, app, host, argv, file names and sizes, the |
| + release version the instance follows, and the run id. |
| +- `run-end`: the run id, exit, signal, timed out, cancelled, interrupted, |
| + refused, started, ended and box; or `lost`. The console writes it the |
| + first time it reads `result.json`, whether from the CLI's poll or from |
| + the sweep. |
| -Output is never in the audit log. argv is: a member who puts a secret in |
| -argv has put it in the log, and `docs/site/cli.md` says so and says to pass |
| -secrets through `--file`. |
| +Output never goes in the audit log, but argv does. A member who puts a |
| +secret in argv has put it in the log, and `docs/site/cli.md` says so and |
| +says to pass secrets through `--file`. |
| ## 7. Security |
| -- **Reach.** The command has exactly the instance's reach, which is why it |
| - is refused on a flat box: there it would have the agent's. |
| +- **Reach.** The command has exactly the instance's reach. That's why it |
| + is refused on a flat box, where it would have the agent's. |
| - **No widening for members.** A member can already pull the database and |
| - publish a build to a non-stable channel that the instance runs; `run` |
| - executes only the build the instance follows. |
| -- **Box-written results are box-forgeable.** Any box can write anywhere in |
| - the deployment bucket today (#171); a compromised box could forge |
| - another box's output or result. That is the existing record-authority |
| - boundary, not a new one, and `run` adds no path from a forged result to |
| - any action. |
| -- **File names** are plain base names checked at the console and again by |
| - the agent, which writes with `instancefs` (no symlink is followed). |
| + publish a build to a non-stable channel that the instance runs. `run` |
| + only executes the build the instance follows. |
| +- **Box-written objects can be forged.** Any box can write anywhere in the |
| + deployment bucket today (#171), so a compromised box could forge another |
| + box's `picked.json`, output or result. That is the existing |
| + record-authority boundary, not a new one, and `run` adds no path from a |
| + forged object to any action. |
| +- **File names** are plain base names, checked at the console and again by |
| + the agent. |
| ## 8. Errors the member sees |
| -Each is its own message, and every one names the instance: |
| +Each has its own message, and every one names the instance: |
| - not a member, or no such instance; |
| - not platform-run, pooled, or on a customer's fleet; |
| -- moving; |
| -- a run already open (with its id and who started it); |
| -- the box is not nested ("this box has not moved to the isolated layout |
| - yet"); |
| +- moving, or (for a move) a run is open; |
| +- a run is already open (with its id and who started it); |
| +- the box isn't nested yet; |
| +- the box has claim control armed; |
| - not picked up within 60 seconds; |
| - the wake failed (with the wake's error); |
| +- interrupted (with the reason); |
| - timed out; |
| -- file too large, or output truncated. |
| +- a file is too large, or the output was truncated. |
| -All CLI and console strings go through copy review before they are written |
| +Every CLI and console string goes through copy review before it is written |
| into the code. |
| ## 9. Testing |
| -- Console door: each refusal in §5.1 in order; the member-only rule; the |
| - audit rows; the index CAS under two concurrent requests; cancel, ack and |
| - the sweep. |
| -- Box: pickup only on the owning box; the flat refusal; the wake; the run |
| - hold against hibernation and a move request; file checks; chunking, |
| - ordering and the output cap; timeout; cancel; result. Against a fake |
| - control service and an in-memory bucket. |
| -- Units: the cmd template equals the seed template outside the named lines. |
| -- `instancectl`: `RoleCmd` parsing, `/cmd` start and status, sweep. |
| -- CLI: interleaving, exit codes, Ctrl-C once and twice, `--follow`. |
| -- On the dev box once built: a real run against a nested instance, |
| - including a write the instance then serves, and a run across a |
| - hibernation timer. |
| -- `GOOS=darwin` and `windows` builds stay green: the unit and control code |
| - is linux-only behind build tags with stubs. |
| +- **Console door:** |
| + - each refusal in §5.1, in order; |
| + - the member-only rule; |
| + - the audit rows; |
| + - the index CAS under two concurrent requests; |
| + - the run/move serialization in both directions; |
| + - cancel, ack (including a wrong `last_seq`) and the sweep. |
| +- **Pickup:** |
| + - only the owning box acts; |
| + - a run queued before agent start is picked up; |
| + - a run queued before the box owned the host is picked up; |
| + - the create-only `picked.json` with two boxes racing; |
| + - a run picked before an agent restart ends `interrupted` and is never |
| + run again. |
| +- **Writer lifecycle:** |
| + - each stop path in §5.3 stops and settles cmd before replication |
| + closes, shown by a write the command commits just before the stop |
| + reaching the replica; |
| + - the idle timer is held while a run is open; |
| + - no start while the entry is waking or stopping. |
| +- **Unit and control:** |
| + - the cmd template equals the seed template outside the named lines; |
| + - `RoleCmd` parsing; |
| + - `/cmd` start, the long-poll for `running`, `done` and `gone`; |
| + - stop after done; |
| + - the sweep; |
| + - refusal while claim control is armed. |
| +- **Output:** |
| + - chunk ordering across both streams; |
| + - the cap; |
| + - `result.json` only after the last chunk; |
| + - several pages of output after the result exists, all delivered before |
| + the ack. |
| +- **CLI:** |
| + - interleaving; |
| + - every exit code; |
| + - Ctrl-C once and twice; |
| + - `--follow`. |
| +- **On the dev box once built:** |
| + - a real run against a nested instance, including a write the instance |
| + then serves; |
| + - a run across a hibernation timer; |
| + - a run interrupted by `carlos restart`. |
| +- **Cross-compile:** `GOOS=darwin` and `windows` builds stay green. The |
| + unit and control code is linux-only, behind build tags with stubs. |
| ## 10. Rollout |
| -Ships in the next release. It does nothing on a box until a member runs it, |
| -and refuses on a flat box, so no switch is needed. Tito's boxes need the |
| -nested layout before Doc and Vicky can use it (#247). |
| +It ships in the next release. It does nothing on a box until a member runs |
| +it, and it refuses on flat boxes and on claim-armed boxes, so it needs no |
| +switch. Tito's boxes need the nested layout before Doc and Vicky can use it |
| +(#247). The claim's activation (#250) carries the work in §3.1. |
| + |
| +## 11. Round 1 dispositions |
| + |
| +| Finding | Disposition | |
| +|---|---| |
| +| codex 1 (P1) the hold misses stop paths | **Accepted.** §5.3: the run joins the instance's writer lifecycle, and every stop settles cmd before replication closes; run and move are serialized at the console (§5.1) | |
| +| codex 2 (P2) claim-armed launch is missing | **Accepted.** §3.1: refused while claim control is armed; the work moves to #250 | |
| +| codex 3 (P2) the ETag pattern drops or replays runs | **Accepted.** §5.2: scan at start and on ownership; create-only `picked.json`; a picked run is never run again | |
| +| codex 4 (P2) oneshot success never becomes inactive | **Accepted.** §5.4: completion is the invocation's main process exiting; the agent records the result, then stops and settles the unit | |
| +| codex 5 (P2) ack can delete unread output | **Accepted.** §5.5 and §5.6: the result comes after the last chunk and names `last_seq`; the ack must match | |
| +| codex 6 (P2) the long-host path | **Accepted.** §5.2 step 3: `provision.InstanceDir` | |
| +| codex 7 (P3) `CARLOS_REGION` | **Accepted.** §4: `cmdVars` | |
diff --git a/docs/superpowers/specs/2026-10-08-seeded-instances-design.md b/docs/superpowers/specs/2026-10-08-seeded-instances-design.md| index 45f21b24f..6790974c3 100644 |
| --- a/docs/superpowers/specs/2026-10-08-seeded-instances-design.md |
| +++ b/docs/superpowers/specs/2026-10-08-seeded-instances-design.md |
| @@ -1,66 +1,76 @@ |
| # Instances that start from a database: `--from` (#260) and the import door (#261) |
| -**Status:** rev 1, 2026-10-08. Design agreed with Paul in chat the same day; |
| -this document is for his review, then Astra's. |
| +**Status:** rev 2, 2026-10-08. Paul approved rev 1 (c66792a3). Rev 2 |
| +answers Astra's round 1 (`CODEX-FINDINGS-RUN-AND-SEEDED-R1.md`; findings |
| +8 to 20 are this spec's). Each change is marked **(codex N)**, and §11 lists |
| +the dispositions. |
| **Discussions:** amadan carlos/platform #260 and #261. Companion spec: |
| `2026-10-08-carlos-run-design.md` (#259). |
| ## 1. The problem |
| -Two requests from Tito, one mechanism. |
| +Two requests from Tito share one mechanism. |
| - **#260, a canary with real data.** The commonest ask from Tito's team is |
| "make me a canary with this account's real data". Today that takes root |
| over SSM: copy the file, chown, register, grant. `carlos db pull` gives a |
| - member a read-only copy on their laptop, but there is no way back in: |
| - `instances create` cannot start from data, and `Manifest.Seed` is a |
| + member a read-only copy on their laptop, but there's no way back in. |
| + `instances create` can't start from data, and `Manifest.Seed` is a |
| first-boot argv (`provision.go:233`), not data. |
| -- **#261, converting controld instances.** About 71 `titogo@<host>` units on |
| - carlos-tito-ie run outside the platform, each behind an addr row. Paul |
| - ruled on 2026-10-08 that every one converts to platform-run and nothing |
| - new is built for controld instances. The import door that conversion needs |
| - was designed in `2026-09-24-instance-moves-for-every-app-design.md` §9 |
| - (gap G9, on the unmerged `docs-instance-moves-design` branch). The moves |
| - spec that replaced it (`2026-10-02-move-any-app-design.md`) dropped it: |
| - its "adoption" is the host claim taking over legacy platform records, not |
| - importing an instance the platform never ran. |
| - |
| -Both are "create an instance whose database starts as this one". |
| +- **#261, converting controld instances.** About 71 `titogo@<host>` units |
| + on carlos-tito-ie run outside the platform, each behind an addr-backed |
| + row. Paul ruled on 2026-10-08 that every one of them converts to |
| + platform-run, and that nothing new is built for controld instances. |
| + - The import door the conversion needs was designed in |
| + `2026-09-24-instance-moves-for-every-app-design.md` §9 (gap G9), on the |
| + unmerged `docs-instance-moves-design` branch. |
| + - The moves spec that replaced it (`2026-10-02-move-any-app-design.md`) |
| + dropped it. That spec's "adoption" is the host claim taking over legacy |
| + platform records, not importing an instance the platform never ran. |
| + |
| +Both requests come down to one thing: create an instance whose database |
| +starts as a given one. |
| ## 2. Rulings (Paul, 2026-10-08) |
| 1. **One mechanism, two sources.** A sibling's replica (`--from`) and an |
| - uploaded snapshot (import) both become one object the box places on |
| - first wake. |
| + uploaded snapshot (import) both become one object, which the box places |
| + on first wake. |
| 2. **Uploads go through the console**, not a presigned URL. This departs |
| - from §9 of the 09-24 design, on the console's own rule (F8, |
| + from §9 of the 09-24 design, following the console's own rule (F8, |
| `apidb.go:299`, `internal/edge/storedoor.go`): bytes pass through the |
| - console, and nothing a request supplies picks a bucket or key. |
| -3. **`--from` stays in the source's region.** Moving to another region is |
| - `carlos move` afterwards, as §9 said for adoption. |
| -4. **The new instance gets its own environment** (never the source's |
| - per-host one) and **an empty private store**. |
| + console, and nothing a request supplies picks a bucket or a key. |
| +3. **`--from` stays in the source's region.** To get to another region, |
| + run `carlos move` afterwards, as §9 said for adoption. |
| +4. **The new instance gets its own environment**, never the source's |
| + per-host one, and **an empty private store**. |
| 5. **Who may, for both doors:** any person who is a member of the account, |
| plus service credentials with the `instance` role. "This is day-to-day |
| operator stuff ... this needs to be something developers can do." |
| Stricter permissions may come later. |
| -6. **The swap accepts an operator-filed row** (§6.4): Tito's addr rows are |
| - filed under the deployment's operator account (G17), so a strict |
| - same-account check would refuse all 71. |
| -7. **Platform-run only, nested only.** Both doors refuse unless the target |
| - box is nested. |
| +6. **The swap accepts an operator-filed row.** Tito's addr rows are filed |
| + under the deployment's operator account (G17), so a strict same-account |
| + check would refuse all 71. Rev 2 binds this exception to an operator's |
| + assignment (§6.4, codex 9). |
| +7. **Platform-run only, nested only.** |
| ## 3. The shared step: a seed snapshot |
| ### 3.1 The object |
| -`seeds/<host>/snapshot.db` in the home region's data bucket, beside the |
| -`instances/` and `moves/` prefixes. The console writes it; the owning box |
| -reads and deletes it. The console's region is the home region, the only |
| -one whose data bucket it may write (`resolveReplica`'s residency rule, |
| -`apidb.go:54`), so both doors refuse a record homed elsewhere with the |
| -existing `ErrResidency` message. |
| +The snapshot lives at `seeds/<host>/snapshot.db` in the home region's data |
| +bucket, beside the `instances/` and `moves/` prefixes. |
| + |
| +- The console writes it. The owning box reads it and deletes it. |
| +- The console's region is the home region, the only one whose data bucket |
| + it may write (`resolveReplica`'s residency rule, `apidb.go:54-71`). So |
| + both doors refuse a record homed anywhere else, with the existing |
| + `ErrResidency` message. |
| +- A bucket lifecycle rule on the `seeds/` prefix expires objects after 7 |
| + days, as a backstop against an import that is abandoned and never |
| + aborted. `seeds/` is a top-level prefix of the data bucket, so a prefix |
| + rule matches it. This is a change in both infrastructure repos. |
| ### 3.2 The record |
| @@ -70,58 +80,110 @@ existing `ErrResidency` message. |
| Seed *SeedSource `json:"seed,omitempty"` |
| type SeedSource struct { |
| - Kind string `json:"kind"` // "replica" | "import" |
| - From string `json:"from,omitempty"` // replica: the source host |
| - TXID string `json:"txid,omitempty"` // replica: the TXID restored |
| - SHA256 string `json:"sha256"` // of snapshot.db |
| - Bytes int64 `json:"bytes"` |
| - State string `json:"state"` // see below |
| - By string `json:"by"` // the caller |
| - At time.Time `json:"at"` |
| + Kind string `json:"kind"` // "replica" | "import" |
| + From string `json:"from,omitempty"` // replica: the source host |
| + TXID string `json:"txid,omitempty"` // replica: the TXID actually restored |
| + SHA256 string `json:"sha256"` // of snapshot.db |
| + Bytes int64 `json:"bytes"` |
| + State string `json:"state"` // §3.2 states |
| + Reason string `json:"reason,omitempty"` // set with "refused" |
| + By string `json:"by"` // the caller |
| + At time.Time `json:"at"` |
| // import only: |
| - Replaces string `json:"replaces,omitempty"` // the addr the swap replaces |
| - Reason string `json:"reason,omitempty"` // set with state refused |
| + Replaces string `json:"replaces,omitempty"` // the addr the swap replaces |
| + VerifyHash string `json:"verify_hash,omitempty"` // sha256 of the verify token (§5.4) |
| } |
| ``` |
| -States: `awaiting-upload` → `uploaded` → `ready` → `placed`, or `aborted`, |
| -or `refused` (the box would not swap; `Reason` says why). |
| -A `--from` record is written straight at `ready`. Reconcile skips a record |
| -whose seed is not `ready` or `placed`. The box sets `placed` after the |
| -first wake succeeds. `Seed` stays on the record once placed: it is the |
| -provenance `instances list` shows. |
| - |
| -Old boxes must not mint a seeded record as an empty instance. The record |
| -is written only when every box in the region reports a release that |
| -understands `Seed` (the protocol-floor check the moves series uses, |
| -`moves_protocol`), otherwise the door answers 409 naming the boxes. |
| - |
| -### 3.3 The box: placing the snapshot on first wake |
| - |
| -One more fallback in `ExecBackend.Restore` (`backend_exec.go:611`), beside |
| -the staging fallback (`:633-645`), and the same step in the claim path's |
| -restore of a cold admission with no baseline (`claimwake.go`, `s.Restore`): |
| - |
| -1. A warm local database, or any replica of the host's own: the seed is |
| - not used. A host with its own replica has already been placed. |
| -2. The record's seed is `ready`: read `seeds/<host>/snapshot.db`, check its |
| - length and sha256 against the record, then `lease.CheckDB` (integrity |
| - and at least one table). |
| -3. Put it at the row's database path through `instancefs`, owned by the |
| - instance, in the nested tree. |
| -4. Start replication. The first upload is generation one of the host's own |
| - lineage: nothing of the source's lineage is copied. |
| -5. After the first upload succeeds: set the record's seed to `placed`, |
| - delete `seeds/<host>/`. |
| - |
| -Any failure in steps 2 to 4 fails the wake, with the reason in the box log |
| -and the `carlos logs` sink. It **never** falls through to an empty |
| -database and the manifest's seed command: a seeded instance that came up |
| -empty would look healthy and be wrong. The manifest's seed command is not |
| -run for a seeded instance at all; the data already exists. |
| - |
| -The snapshot is deleted only after the host's own replica exists, so a |
| -crash anywhere before that retries from the same snapshot. |
| +**States.** |
| + |
| +| Door | Path through the states | |
| +|---|---| |
| +| `--from` | `ready` → `placed` | |
| +| import | `awaiting-upload` → `uploaded` → `ready` → `swapped` → `placed` → `committed` | |
| + |
| +An import can also leave the path: |
| + |
| +- `refused`: the box would not swap; `Reason` says why. |
| +- `aborted`, and then the record is deleted. |
| + |
| +**What reconcile does with each state** (codex 13). It dispatches on the |
| +seed state **before** any other per-record handling: |
| + |
| +| State | Reconcile | |
| +|---|---| |
| +| `aborted` | runs the idempotent rollback (§5.6) | |
| +| `awaiting-upload`, `uploaded`, `refused` | skips the record; nothing is minted | |
| +| `ready` | mints (`--from`) or swaps (import) | |
| +| `swapped`, `placed`, `committed` | ordinary handling, plus the import gate (§5.4) while not `committed` | |
| + |
| +**`Seed` stays on the record after placement.** It is the provenance |
| +`instances list` shows. |
| + |
| +### 3.3 The seed floor (codex 16) |
| + |
| +An old box, or an old binary a box recovers to, would read a seeded record |
| +as an ordinary new instance and mint it empty. A seeded instance that comes |
| +up empty looks healthy and is wrong. So seeds get a protocol floor, built |
| +on the moves floor's mechanism (`internal/moves/floor.go`): |
| + |
| +- **The floor object.** An operator publishes |
| + `control/seeds-protocol.json`. |
| +- **What a box does.** A box running a seed-capable release arms the floor |
| + locally. Once armed, it gives a verdict on `carlos.prev`, and the |
| + recovery units refuse a `carlos.prev` below the floor, exactly as they do |
| + for the moves floor (`carlos-system-recover.service`). |
| +- **What the console checks.** It writes no seed state at all, not even |
| + `awaiting-upload`, until every box in the home region reports, in its |
| + status, the seeds floor armed and `carlos.prev` either excluded or |
| + seed-capable. Otherwise the door answers 409 and names the boxes. |
| + |
| +### 3.4 Placing the snapshot on first wake (codex 8) |
| + |
| +**This version: the plain restore path only.** On a box with claim control |
| +armed, both doors refuse with `refused: "claim"`. The claim's launch path |
| +doesn't run tenant code yet (`rolelaunch_linux.go:23-28`; |
| +`instancectl/server.go:840`), and its restore never trusts a local file |
| +(`hostbox/restore.go:142`, `host/claimwakeassembly.go:252`). The claim |
| +algorithm is set out at the end of this section for #250 to build. |
| + |
| +**The plain path.** One more fallback goes into `ExecBackend.Restore` |
| +(`backend_exec.go:611`), beside the staging fallback (`:633-645`). It |
| +applies only to a row whose record seed is `ready`, `swapped` or `placed`: |
| + |
| +1. **Is there already data?** |
| + - If the host has any replica of its own, it has been placed already. |
| + Restore normally, and ignore the seed. |
| + - If there's a local database, it is the placed seed from an |
| + interrupted first wake, because placement (step 3) is an atomic |
| + rename and nothing else writes that path for a seeded row. Use it, and |
| + go to step 4. |
| +2. **Fetch and check.** Read `seeds/<host>/snapshot.db` into a temp file |
| + inside the instance's directory. Check its length and sha256 against |
| + the record, then run `lease.CheckDBContext` (integrity, and at least one |
| + table). |
| +3. **Place it.** Rename the temp file onto the row's database path through |
| + `instancefs`, owned by the instance, in the nested tree. |
| +4. **Replicate.** Start replication. The first upload is generation one of |
| + the host's own lineage. Nothing of the source's lineage is copied. |
| +5. **Advance the state.** After the first upload succeeds, set the seed to |
| + `placed` and delete `seeds/<host>/`. |
| + |
| +**Failures.** Any failure in steps 2 to 4 fails the wake. The reason goes |
| +to the box log and the `carlos logs` sink. It **never** falls through to an |
| +empty database and the manifest's seed command. The manifest's seed |
| +command is never run for a seeded row. The snapshot is deleted only after |
| +the host's own replica exists, so a crash at any point retries from either |
| +the placed local file or the snapshot. |
| + |
| +**The claim algorithm, for #250.** This is not built here. |
| + |
| +- Select the remote baseline first. |
| +- If no baseline exists and the record still carries an unplaced seed: |
| + purge the local state, and install the snapshot as the admitted |
| + generation's content, whatever local file exists. |
| +- Publish the generation before the seed is marked `placed` and its |
| + snapshot deleted. |
| ## 4. `--from`: a copy of a sibling (#260) |
| @@ -129,24 +191,55 @@ crash anywhere before that retries from the same snapshot. |
| carlos instances create --app <app> --host <new> --from <host> [--at <txid|time>] [--environment <env>] [--channel <c>] |
| ``` |
| -The console: |
| - |
| -1. Applies `instances create`'s existing checks to the new host. |
| -2. Resolves the source exactly as `db pull` does (`dbGate`): same app, same |
| - account, a record, not pooled, not on a fleet, homed in this region, |
| - and, with the host claim on, read through a versioned claim. |
| -3. Refuses a source and a target of different apps or accounts (409): |
| - customer data never leaves its account by this door. |
| -4. Restores the source's replica at its latest TXID, or at `--at`, with |
| - `dbdoor.RestoreTo` into a private 0700 directory under the console's |
| - own scratch, and runs `CheckDB`. This is the code `db pull` runs on the |
| - member's laptop, run here instead; it never reads the live file. |
| -5. Refuses a restored database over 4 GiB (413). The console needs scratch |
| - for one restore at a time per app; a second concurrent `--from` for the |
| - same app waits on a per-app semaphore, and one beyond a deployment-wide |
| - limit of two is refused with 429. |
| -6. Writes `seeds/<new>/snapshot.db`, then the record with seed `ready`, |
| - kind `replica`, the source and the TXID, then deletes its scratch. |
| +**The gate (codex 17).** The door authorizes through `cliRoute` with the |
| +`instance` role, plus the membership check for people. It does **not** |
| +call `dbGate`, which accepts only people (`apidb.go:110`); the db routes' |
| +own refusal of every service role (`authcli.go:690`) stays as it is. The |
| +source checks inside `dbGate` are factored into `resolveDBSource`: |
| + |
| +- the record exists; |
| +- same account and app; |
| +- not pooled, and not on a fleet; |
| +- residency; |
| +- the versioned claim read when claims are on. |
| + |
| +Both doors call it. |
| + |
| +**The console's steps:** |
| + |
| +1. Apply `instances create`'s existing checks to the new host, and the |
| + seed floor (§3.3). |
| +2. Resolve the source with `resolveDBSource`. |
| +3. Refuse a source and target of different apps or accounts (409). |
| + Customer data never leaves its account by this door. |
| +4. **Plan the restore (codex 18).** `RestoreTo` gains a target, its zero |
| + value meaning "latest". Litestream's restore options already carry |
| + `TXID` and `Timestamp` (litestream v0.5.14 `db.go:2823-2828`); the |
| + wrapper passes them through, and returns the TXID actually restored. |
| + An `--at` older than the replica's retained history is refused with |
| + "history before <txid/time> is no longer kept". |
| +5. **Admit the restore (codex 19).** Before downloading anything: |
| + - sum the planned LTX files' sizes from the listing (the plan already |
| + has them); |
| + - refuse with 413 if the sum is over 4 GiB; |
| + - refuse with 507 unless the console's scratch filesystem has free space |
| + of at least twice that sum plus 1 GiB. |
| + |
| + Restores are limited to one at a time per app (a semaphore) and two |
| + across the deployment; more than that is refused with 429. The restore |
| + runs under a 30-minute deadline. |
| +6. **Restore and check.** Restore with `RestoreTo` into a private 0700 |
| + directory under the console's scratch, `<console data>/scratch/seeds/`. |
| + Run `CheckDBContext` under the same deadline. This never reads the live |
| + file. |
| +7. **Write.** Write `seeds/<new>/snapshot.db` with `PutFrom` from the |
| + restored file (seekable, under S3's 5 GB single-PUT limit). Then write |
| + the record with seed `ready`, kind `replica`, the source and the |
| + restored TXID. |
| + |
| +**Cleanup.** The scratch is deleted on every exit path: success, refusal, |
| +error, deadline, and client disconnect. The console also empties |
| +`scratch/seeds/` when it starts. |
| **Environment.** The new instance gets its own: `--environment` if given, |
| otherwise what a plain `create` derives (the app's default, or the |
| @@ -155,236 +248,357 @@ environment, which carries host-keyed values such as the origin and the |
| cookie domain (the warning in `instances_cli.go:235-245`). |
| **Storage.** An app with private storage gets the new host's own empty |
| -store, as any `create` does. Copying the source's store is not offered in |
| -this version. |
| +store, as any `create` does. |
| ## 5. The import door (#261) |
| ``` |
| -carlos instances import --app <app> --host <host> --file <snapshot.db> --replaces <addr> [--environment <env>] |
| +carlos instances import --app <app> --host <host> --file <snapshot.db> --replaces <addr> [--environment <env>] [--verify <path>] |
| ``` |
| -The CLI drives the four calls below. Titogo's conversion tool makes the |
| -same calls with its service credential. |
| +The CLI drives the calls below, and titogo's conversion tool makes the same |
| +calls with its service credential. The CLI's `--verify <path>` makes a GET |
| +with the verify token after the swap and commits on a 2xx. Without it, the |
| +CLI stops after the swap and prints the token and the commit command. |
| ### 5.1 Begin |
| `POST /api/cli/apps/{a}/{app}/instances` with |
| -`"import": {"sha256": "...", "bytes": N, "replaces": "<addr>"}` and the |
| +`"import": {"sha256": "...", "bytes": N, "replaces": "<addr>"}` plus the |
| ordinary create fields. |
| The console checks: |
| - the caller (§2 ruling 5); |
| -- the host is not already a platform record. The console cannot see a |
| - box's registry, so the addr row's existence and shape are checked by the |
| - box at the swap (§5.4), which is the authority; |
| +- the seed floor (§3.3); |
| +- that the host isn't already a platform record; |
| - `bytes` is at most 4 GiB. |
| -It writes the record with seed `awaiting-upload` and answers `201`. No box |
| -mints anything for it yet. |
| +The console can't see a box's registry, so the box checks the existing row |
| +at the swap (§5.4), and the box is the authority on it. |
| + |
| +The console mints a **verify token** (32 random bytes). It answers `201` |
| +with the token. This is the only time the token appears; the record keeps |
| +only its hash. The record is written with seed `awaiting-upload`. |
| + |
| +### 5.2 Upload (codex 19, codex 20) |
| + |
| +`PUT …/instances/{host}/import`, streamed. Before reading the body, the |
| +console checks its scratch filesystem has free space of at least `bytes` |
| +plus 1 GiB; if not, 507. Then: |
| -### 5.2 Upload |
| +1. It spools the body to `<console data>/scratch/seeds/<host>.db` under |
| + `http.MaxBytesReader(bytes)`, hashing as it writes. |
| +2. A length or hash that differs from the record is refused with 422. |
| +3. It runs `CheckDBContext` on the spooled file, under a 30-minute |
| + deadline. Not a database, or failing integrity, is refused with 422. |
| +4. It writes `seeds/<host>/snapshot.db` with one `PutFrom` from the |
| + spooled file. A seekable file under 5 GB needs no multipart upload, so |
| + there's no unfinished multipart upload to clean up and no new IAM |
| + action. |
| +5. It deletes the spool, and the seed becomes `uploaded`. |
| -`PUT …/instances/{host}/import`, streamed. The console hashes as it |
| -writes, in parts of 64 MB, to `seeds/<host>/snapshot.db`. A length or hash |
| -that differs from the record is refused (422), and the partial object is |
| -deleted. On success the seed is `uploaded`. |
| +The spool is deleted on every exit path, and on console start. |
| -The snapshot must be consistent and taken after the app stopped the |
| -instance (`VACUUM INTO` after its supervisor confirms the process is |
| -down). The console cannot see that. It can only check that the file is a |
| -SQLite database that passes `CheckDB`, which it does before marking the |
| -seed `uploaded`. |
| +The snapshot must be consistent, and taken after the app has stopped the |
| +instance (`VACUUM INTO` once its supervisor confirms the process is down). |
| +The console can't see that; it can only check the file. |
| ### 5.3 Stopped |
| `POST …/instances/{host}/import/stopped` is the app saying its own process |
| is down. The seed becomes `ready`, and the console bumps the serial. |
| -Until then the app's own addr row keeps answering, so the app's router |
| -must answer 503 for a stopped instance. For a small database that window |
| -is a minute or two, and it is the whole cost of adopting an instance. |
| +Until the swap, the app's own row keeps answering, so the app's router must |
| +answer 503 for a stopped instance. |
| + |
| +### 5.4 The swap, the gate, and commit (codex 10, 11, 12, 14, 15) |
| + |
| +Reconcile on the box finds a record with seed `ready` and `Replaces` set, |
| +and an existing row for the host. Today such a row is left alone ("exists |
| +but is not the route this record describes", `reconcile.go:1158`, `:1287`), |
| +and that stays the rule for every other case. |
| + |
| +This one case swaps instead, in one registry transaction, only if every |
| +condition holds. If any fails, the seed becomes `refused` with the |
| +condition named. |
| -### 5.4 The swap (reconcile) |
| +**The row's shape** (codex 10). Registry kinds are `instance`, `service`, |
| +`static` and `sidecar`, and the backing is a separate set of fields |
| +(`registry.go:31-40`). The row must be: |
| -On the owning box, reconcile meets a record with seed `ready` and |
| -`Replaces` set, and an existing row for the host. |
| +- kind `instance` or `service`; |
| +- backed by `Addr` or `Addrs`, with an empty `Socket` and no tunnel; |
| +- without `Provisioned` from any record; |
| +- with an addr equal to `Replaces`. |
| -Today an unprovisioned row of another shape is left alone with "exists but |
| -is not the route this record describes" (`reconcile.go:1158`, `:1287`). |
| -That stays the rule for every other case. This one case swaps instead, in |
| -one registry transaction, and only if all of these hold: |
| +**Ownership:** the row's app equals the record's app, and the row's account |
| +is either: |
| -- the row is an addr row (`Kind` addr) and its `Addr` equals `Replaces`; |
| -- the row is not provisioned by another record; |
| -- the row's app equals the record's app; and |
| -- the row's account is either the record's account, or the deployment's |
| - operator account (G17, §6.4). |
| +- the record's account; or |
| +- the deployment's operator account, with an operator assignment (§6.4) |
| + that maps that app name to the record's account and app. |
| -The addr row becomes the exec row this record describes, filed under the |
| -record's account. With the host claim on, the swap is the record's first |
| -claim acquisition: the claim CAS happens before the registry write, as for |
| -any create, and a refused CAS leaves the addr row untouched. |
| +**Layout** (codex 12). The box's layout intent and probe must say nested |
| +(`mintLayoutReader`). The import branch derives the new row's paths with |
| +`nested = true` explicitly. It doesn't use `NestedPaths(existing)`, which |
| +would read the addr row as flat (`reconcile.go:1103-1107`). |
| -If any condition fails, the box leaves the row alone, sets the seed to |
| -`refused` with the condition that failed, and mints nothing. The app sees |
| -that through `GET …/instances/{host}` (or `carlos instances list`), and |
| -abort then deletes the record. A swap that has not happened within five |
| -minutes of `stopped` is shown as pending with the last box that looked. |
| +In the same transaction, the box (codex 14): |
| -The edge picks up the new row on its next reload. The next request wakes |
| -the instance, and §3.3 places the snapshot. |
| +- writes the **complete original row** into a new registry table, |
| + `replaced_routes` (host, the row's full JSON, record key, time); |
| +- replaces the row with the exec row the record describes, filed under the |
| + record's account, with `ImportGate` set to the record's `VerifyHash`. |
| -### 5.5 Abort |
| +Then it sets the seed to `swapped`. With claims off, which is the only case |
| +built here, there's no claim CAS. So the transaction is the whole swap, and |
| +a crash either leaves the addr row untouched or completes the swap (codex |
| +11). The claim-armed refusal (§3.4) covers the claim case until #250. |
| -`POST …/instances/{host}/import/abort` undoes the import, which is how |
| -titogo rolls back automatically when `verify` fails: |
| +**The import gate** (codex 15). While a row has `ImportGate` set: |
| -- **Before the swap:** delete the record and `seeds/<host>/`. |
| -- **After the swap:** mark the seed `aborted`. Reconcile on the owning box |
| - stops the instance and swaps the exec row back to the addr row it |
| - replaced (the record still names it), in one transaction. Then it |
| - releases the claim, deletes the record, and deletes the host's |
| - `instances/` replica and `seeds/<host>/`. |
| +- the edge, and the edge door, answer 503 with `Retry-After: 5` to every |
| + request **except** one carrying `Carlos-Import-Verify: <token>`, whose |
| + sha256 matches; |
| +- the schedule runner sends no ticks to the row. |
| -Abort is refused once the seed is `placed` and the instance has served a |
| -request that was not the platform's own `verify`. After that, the instance |
| -may hold writes the old process never saw, and `carlos instances delete` |
| -is the honest verb. |
| +So until commit, the only requests that reach the instance are the app's |
| +own verify requests, and nothing ordinary has been served. The first verify |
| +request wakes the instance, and §3.4 places the snapshot. |
| -### 5.6 Verify |
| +**Commit.** `POST …/instances/{host}/import/commit` is a record CAS that |
| +moves the seed from `placed` to `committed`. Reconcile then clears |
| +`ImportGate` and deletes the `replaced_routes` row. Commit is refused before |
| +`placed`, because there's nothing verified yet. |
| -`verify` (moves spec §6.5) is how titogo knows the adopted instance |
| -serves. Nothing new is needed here: once the exec row is live, the app |
| -calls its own verify path through the instance, and aborts if it fails. |
| +### 5.5 Verify |
| + |
| +`verify` (moves spec §6.5) is how titogo knows the adopted instance serves. |
| +Titogo sends its verify path through the edge with the verify header. If it |
| +passes, it commits. If it fails, it aborts. |
| + |
| +### 5.6 Abort |
| + |
| +`POST …/instances/{host}/import/abort` is a record CAS from any state |
| +before `committed` to `aborted`. A commit and an abort race on the |
| +record's ETag, and exactly one of them wins. After `committed`, abort is |
| +refused; `carlos instances delete` is the verb then. |
| + |
| +Reconcile's idempotent rollback for `aborted` is: |
| + |
| +1. **If the swap happened** (there's a `replaced_routes` row): |
| + - stop the instance; |
| + - in one transaction, put the saved row back exactly as it was, and |
| + delete the `replaced_routes` row. |
| +2. Delete the host's `instances/` replica, if any, and `seeds/<host>/`. |
| +3. Delete the record. The console does this if the box never swapped. |
| + |
| +**Why abort loses nothing.** Under the gate nothing but verify requests |
| +reached the instance, so it holds no writes the old process never saw. |
| ## 6. Who may, and what is checked |
| ### 6.1 Callers |
| - People who are members of the account. |
| -- Service credentials with the `instance` role (`svcrole`), checked per |
| - route by `cliRoute`. |
| +- Service credentials with the `instance` role, checked per route by |
| + `cliRoute`. |
| -The same rule applies to `--from`, to all four import calls and to abort. |
| -Tighter permissions can come later. |
| +The same rule covers `--from`, every import call, commit and abort. |
| ### 6.2 Same app, same account |
| - `--from` copies only between instances of one app in one account. |
| -- An import creates an instance of the app the route already belongs to. |
| - It cannot claim another app's host. |
| +- An import creates an instance of the app the route already belongs to, |
| + under the rules in §5.4. |
| + |
| +### 6.3 Layout and claims |
| -### 6.3 Layout |
| +- **At the console.** Both doors refuse unless the region's layout report |
| + shows the target box nested (`apistatuslayout.go`), and the box doesn't |
| + report claim control armed. |
| +- **At the box.** It refuses again: in §3.4 unless `Classify` answers |
| + `LayoutNested`, and in §5.4 unless the layout intent is nested. |
| -Both doors refuse at the console unless the region's layout report shows |
| -the target box nested (`apistatuslayout.go`). The box refuses again in |
| -§3.3 unless `Classify` answers `LayoutNested` for the row. On a flat box a |
| -seeded instance would be one uid among every tenant, and for Tito's |
| -controld instances it would lose the `0700 titogo` protection that is the |
| -only thing keeping them safe today (#126). |
| +A flat box is ruled out because a seeded instance there would be one uid |
| +among every tenant. For Tito's controld instances, it would also lose the |
| +`0700 titogo` protection that is the only thing keeping them safe today |
| +(#126). |
| -### 6.4 The operator-account exception (G17) |
| +### 6.4 The operator assignment (G17, codex 9) |
| A direct-mode `carlos add` files a route under the deployment's operator |
| -account, not the app's own. Tito's rows are `account=bab` for app |
| -`bae/titogo`. The swap accepts that row when its app name matches, and |
| -files the new row under the app's account, which fixes G17 for every |
| +account, not the app's own. Tito's rows are `account=bab`, for app |
| +`bae/titogo`. An app name alone doesn't say which account it means: the |
| +operator account's `titogo` row could belong to any account's `titogo`. So |
| +the exception needs an operator to say which. |
| + |
| +``` |
| +carlos system legacy-routes assign --name titogo --to bae/titogo |
| +``` |
| + |
| +**Who may.** This is an operator-only door (`CARLOS_SYSTEM_OPERATORS`). |
| + |
| +**What it writes.** An entry in `control/legacy-route-owners.json`, which |
| +maps an app name, as filed under the operator account, to one account and |
| +app. A name can have only one entry. Re-assigning it is refused while any |
| +import under the old entry is between `ready` and `committed`. |
| + |
| +**How the swap uses it.** The swap accepts an operator-filed row only when |
| +this file maps the row's app name to the record's account and app. The |
| +swap files the new row under the app's account, which fixes G17 for every |
| converted host. |
| -It accepts nothing wider: |
| +The swap accepts nothing wider: |
| -- not a row of another app; |
| -- not a row filed under a third account; |
| -- not a row that is not addr-backed; |
| -- not a row whose addr has changed since `Replaces` was recorded. |
| +- no row of another app; |
| +- no row filed under a third account; |
| +- no row that isn't addr-backed; |
| +- no row whose addr has changed since `Replaces` was recorded. |
| ## 7. What a member sees |
| -- **`carlos instances list`** marks a seeded instance: |
| - - "seeded from <host> at txid N" for `--from`; |
| - - "imported <first 12 of sha256>" for an import; |
| - - "import awaiting upload" or "import awaiting stop" while those are |
| - pending. |
| -- **Audit rows:** |
| - - `instance-seed` names the source host and TXID, or the sha256 and |
| - bytes, and the caller; |
| - - `import-stopped`, `import-swapped` and `import-aborted` each record |
| - their step. |
| -- **Errors** each have their own message: |
| - - other app or other account; |
| - - other region; |
| - - box not nested; |
| - - too large; |
| - - hash or length mismatch; |
| - - not a database; |
| - - no addr row to replace; |
| - - addr changed; |
| - - row belongs to another app; |
| - - boxes below the protocol floor; |
| - - abort refused after serving. |
| - |
| -All CLI and console strings go through copy review before they are written |
| +**In `carlos instances list`**, a seeded instance shows one of: |
| + |
| +- "seeded from <host> at txid N"; |
| +- "imported <first 12 of sha256>"; |
| +- "import awaiting upload", "awaiting stop", "swapped, awaiting verify", |
| + or "refused: <reason>". |
| + |
| +**Audit rows:** |
| + |
| +- `instance-seed`, naming the source host and TXID (or the sha256 and |
| + bytes) and the caller; |
| +- `import-uploaded`, `import-stopped`, `import-committed` and |
| + `import-aborted`; |
| +- `legacy-route-assign`. |
| + |
| +**Errors.** Each one has its own message: |
| + |
| +- other app or other account; |
| +- other region; |
| +- box not nested; |
| +- claim control armed; |
| +- too large; |
| +- no scratch space; |
| +- history no longer kept; |
| +- hash or length mismatch; |
| +- not a database; |
| +- boxes below the seed floor; |
| +- commit before placed; |
| +- abort after commit. |
| + |
| +Some come back as the swap's refusal reasons instead: |
| + |
| +- no addr row; |
| +- addr changed; |
| +- row of another app; |
| +- operator row with no assignment. |
| + |
| +Every CLI and console string goes through copy review before it's written |
| into the code. |
| ## 8. Security |
| - **Data never leaves its account.** Neither door crosses an account or an |
| app. |
| -- **Uploads are pinned.** An import is pinned to one record, one length |
| - and one hash. The console checks the hash as it writes, and the box |
| +- **Uploads are pinned.** An upload is pinned to one record, one length and |
| + one hash. The console checks the hash as it spools the file, and the box |
| checks it again before it trusts the file. |
| -- **No standing identity gains a read.** The console already reads the |
| - home region's replicas for `db pull`. Boxes read only their own |
| - region's bucket, and `seeds/` is in it. |
| -- **A forged record is still the main risk.** A seed is a record field, |
| - and #171 means any compromised box can forge a record. A forged seed |
| - could make a new instance start from data the forger chose, though only |
| - data that is already in that bucket. This is the same exposure the |
| - moves series names as G13, and #171's fix bounds both. |
| -- **The swap only narrows the "leave it alone" rule.** It narrows it for |
| - a row that this record names by its exact addr, that belongs to the |
| - same app, and that is filed under the app's own account or the |
| - operator's. Every other mismatch is left alone, as today. |
| +- **The verify token never sits on any box or bucket.** Only its hash |
| + does. Without the token, nothing ordinary reaches an un-committed import. |
| +- **No standing identity gains a read.** The console already reads the home |
| + region's replicas for `db pull`. Boxes read only their own region's |
| + bucket, and `seeds/` is in it. |
| +- **A forged record is still the main risk.** A seed is a record field, and |
| + #171 means any compromised box can forge a record. A forged seed could |
| + start a new instance from a snapshot already in the bucket, or a forged |
| + assignment file could widen §6.4. Both are the existing record-authority |
| + boundary (G13 in the moves series), and #171's fix bounds them. |
| +- **The swap only narrows the "leave it alone" rule.** It narrows it for a |
| + row named by its exact addr, of the same app, under the app's account or |
| + an operator-assigned one. Every other mismatch is left alone, as today. |
| ## 9. Testing |
| -- **Box:** |
| - - placing the snapshot, and each failure (length, hash, `CheckDB`) failing |
| - the wake without ever falling back to an empty database; |
| - - a crash before the first upload retrying from the snapshot; |
| - - the snapshot deleted only after the first upload; |
| - - in both the plain path and the claim path. |
| +- **The floor:** |
| + - no seed state is written until every box reports the floor armed; |
| + - a recovery below the floor is refused. |
| +- **Box placement:** |
| + - each failure (length, hash, `CheckDB`) fails the wake, and never falls |
| + back to an empty database; |
| + - a crash after the rename but before the first upload resumes from the |
| + local file; |
| + - a crash before the rename retries from the snapshot; |
| + - the snapshot is deleted only after the first upload; |
| + - refusal with claim control armed. |
| - **Reconcile:** |
| - - the swap's four conditions, each refused on its own; |
| - - the operator-account case; |
| - - a refused claim CAS leaving the addr row; |
| - - abort after the swap restoring the addr row exactly. |
| + - the swap's conditions, each refused on its own, with its reason; |
| + - the operator assignment, present, absent and mapping elsewhere; |
| + - nested paths derived for the new row; |
| + - the original row stored and restored exactly; |
| + - `aborted` dispatched before the state filter; |
| + - an idempotent rollback across a crash between its steps. |
| +- **The gate:** |
| + - the edge and the door answer 503 without the token and pass with it; |
| + - no ticks while gated; |
| + - commit clears the gate; |
| + - the commit and abort race. |
| - **Console:** |
| - - both doors' gates; |
| - - `--from` restore, cap, semaphore and scratch cleanup on every path; |
| - - import begin, upload, stopped and abort in every state; |
| - - hash and length refusal; |
| - - the protocol floor; |
| + - both doors' gates, including `instance` service credentials and |
| + `db pull` still refusing them; |
| + - `--from` with `--at` by TXID and by time, and history no longer kept; |
| + - scratch admission, the semaphore and the deadline; |
| + - scratch cleanup on every exit path and at start; |
| + - every import state transition; |
| + - hash, length and `CheckDB` refusal; |
| - the audit rows. |
| -- **CLI:** `instances create --from`, `instances import` end to end against |
| - a fake console. |
| +- **CLI:** |
| + - `instances create --from` and `instances import`, with and without |
| + `--verify`, end to end against a fake console; |
| + - `system legacy-routes assign`. |
| - **On the dev box:** |
| - a `--from` canary of a real instance; |
| - - an import of a stopped snapshot over an addr row; |
| - - an abort after the swap. |
| + - an import of a stopped snapshot over an addr row, with a verify, then |
| + a commit; |
| + - a second import aborted after the swap, with the addr row restored and |
| + serving. |
| - **Cross-compile:** darwin and windows builds stay green. |
| ## 10. Rollout and #261's schedule |
| -1. Build both doors now. They refuse on flat boxes, so they are safe to |
| - ship before Tito's boxes are nested. |
| -2. Roll the nested layout to Tito's boxes (#247 steps 2 to 5, Tito's own |
| - approvals). |
| -3. Titogo builds its half (its design §4 and §7: snapshot, import, carry |
| - env, swap, repoint, retire; import mirrors that write through the |
| - instance; controld's `retire`). |
| -4. Convert one canary host, verify, abort it once on purpose, convert it |
| +1. Build both doors now. They refuse on flat and claim-armed boxes, so |
| + they're safe to ship before Tito's boxes are nested. |
| +2. Release, then publish the seeds floor on each deployment once its boxes |
| + run the release. |
| +3. Roll the nested layout to Tito's boxes (#247 steps 2 to 5, with Tito's |
| + own approvals). |
| +4. An operator assigns `titogo` to `bae/titogo` (§6.4). |
| +5. Titogo builds its half (its design §4 and §7): snapshot, import, carry |
| + env, verify with the token, commit or abort, repoint, retire; import |
| + mirrors that write through the instance; and controld's `retire`. |
| +6. Convert one canary host, abort it once on purpose, then convert it |
| again. |
| -5. Convert the rest, a few at a time, with Paul's go. |
| +7. Convert the rest, a few at a time, with Paul's go. |
| + |
| +If the claim is activated (#250) before the conversion finishes, the |
| +remaining imports wait for §3.4's claim algorithm. |
| + |
| +## 11. Round 1 dispositions |
| + |
| +| Finding | Disposition | |
| +|---|---| |
| +| codex 8 (P1) warm-file shortcut vs claim restore | **Accepted.** §3.4: this version is the plain path only, and claim-armed boxes refuse; the claim algorithm is written down for #250 as Astra gave it | |
| +| codex 9 (P1) app name doesn't fix the account | **Accepted.** §6.4: an operator assignment maps the name to one account and app | |
| +| codex 10 (P2) `Kind == "addr"` doesn't exist | **Accepted.** §5.4: the shape defined by kind and backing fields | |
| +| codex 11 (P2) the claim acquisition at the swap | **Accepted** by scoping. With claims off the swap is one registry transaction; claim-armed boxes refuse (§3.4) | |
| +| codex 12 (P2) existing-row branch derives flat | **Accepted.** §5.4: the import branch derives nested explicitly | |
| +| codex 13 (P2) the state filter makes rollback unreachable | **Accepted.** §3.2: dispatch on seed state first | |
| +| codex 14 (P2) `Replaces` can't restore the row | **Accepted.** §5.4: the complete row goes in `replaced_routes` in the swap's transaction | |
| +| codex 15 (P1) abort safety not observable | **Accepted.** §5.4: the import gate with a verify token, and commit as the durable latch; abort only before commit | |
| +| codex 16 (P1) downgrade consumes seed records | **Accepted.** §3.3: the seeds floor, armed with a `carlos.prev` verdict, before any seed state is written | |
| +| codex 17 (P1) `dbGate` refuses service credentials | **Accepted.** §4: `resolveDBSource` is factored out; the db door's own gate is unchanged | |
| +| codex 18 (P2) `RestoreTo` can't honour `--at` | **Accepted.** §4 step 4: a target passed to litestream's TXID/Timestamp options | |
| +| codex 19 (P2) scratch isn't bounded | **Accepted.** §4 step 5 and §5.2: admission from the plan's sizes, free-space checks, deadlines, cleanup on every path and at start | |
| +| codex 20 (P2) multipart upload | **Accepted, differently.** §5.2: spool to scratch (which `CheckDB` needs anyway), then one `PutFrom` under the 5 GB single-PUT limit; no multipart, no new IAM action | |
diff --git a/docs/superpowers/specs/CODEX-FINDINGS-RUN-AND-SEEDED-R1.md b/docs/superpowers/specs/CODEX-FINDINGS-RUN-AND-SEEDED-R1.md| new file mode 100644 |
| index 000000000..08654d547 |
| --- /dev/null |
| +++ b/docs/superpowers/specs/CODEX-FINDINGS-RUN-AND-SEEDED-R1.md |
| @@ -0,0 +1,43 @@ |
| +1. **P1 — Run §5.2: the hold does not protect against every stop path.** Blocking hibernation and new move requests leaves restart, convergence, process failure and claim handoff uncovered. `Bounce` stops an awake instance without consulting such a hold ([activator.go:1425](/home/paulca/amadan.net/carlos/platform/.claude/worktrees/run-and-seeded/internal/activator/activator.go:1425)); completion closes replication and forgets the entry ([activator.go:2355](/home/paulca/amadan.net/carlos/platform/.claude/worktrees/run-and-seeded/internal/activator/activator.go:2355)). The separate cmd unit could continue writing after that checkpoint. **Fix:** register cmd as part of the instance’s writer lifecycle. Every unavoidable stop must terminate and settle cmd before closing replication, releasing ownership or replacing the database. Serialize run and move admission through the same authority. |
| + |
| +2. **P2 — Run §§5.2–5.3; seeded §3.3: claim-enabled isolated execution is an unfulfilled prerequisite.** Legacy start/seed/stop routes return 503 when claim control is armed ([server.go:840](/home/paulca/amadan.net/carlos/platform/.claude/worktrees/run-and-seeded/internal/instancectl/server.go:840)). The claim launch implementation explicitly refuses execution until the isolated root protocol supports reservations ([rolelaunch_linux.go:23](/home/paulca/amadan.net/carlos/platform/.claude/worktrees/run-and-seeded/internal/activator/rolelaunch_linux.go:23)). Neither a plain `/cmd` nor an ordinary claim wake supplies this missing integration. **Fix:** name and complete the guarded root-launch dependency, including cmd reservations and producer settlement, before claiming either feature works with claims enabled. |
| + |
| +3. **P2 — Run §5.2: restart’s ETag pattern can drop queued work or replay commands.** Restart deliberately ignores first sight and keeps its comparison state only for the agent’s lifetime ([restart.go:114](/home/paulca/amadan.net/carlos/platform/.claude/worktrees/run-and-seeded/internal/host/restart.go:114), [restart.go:165](/home/paulca/amadan.net/carlos/platform/.claude/worktrees/run-and-seeded/internal/host/restart.go:165)). Copying that behavior misses runs already queued when a box boots or acquires ownership. Replaying them instead is unsafe because `picked.json` has no conditional acquisition or recovery semantics. **Fix:** scan outstanding runs at startup and ownership changes; conditionally acquire each run against its instance incarnation. Recover a picked run through its recorded invocation or mark it interrupted; never blindly execute it again. |
| + |
| +4. **P2 — Run §§5.3–5.4: successful commands never reach the specified completion condition.** The permitted template differences retain `Type=oneshot` and `RemainAfterExit=yes` ([seed unit:19](/home/paulca/amadan.net/carlos/platform/.claude/worktrees/run-and-seeded/internal/host/units/carlos-instance-seed@.service:19)). Success therefore remains `active (exited)`, while §5.4 waits for inactivity. The current seed handler explicitly reads the retained result and then stops the unit ([server.go:431](/home/paulca/amadan.net/carlos/platform/.claude/worktrees/run-and-seeded/internal/instancectl/server.go:431)). **Fix:** define asynchronous start and completion explicitly: observe completion of the matching oneshot invocation, capture its result, then stop and settle it. Alternatively, specify different lifecycle directives and an appropriate runtime limit. |
| + |
| +5. **P2 — Run §§5.5, 5.7: acknowledgement can delete unread output.** A bounded response includes the result whenever it exists ([run spec:209](/home/paulca/amadan.net/carlos/platform/.claude/worktrees/run-and-seeded/docs/superpowers/specs/2026-10-08-carlos-run-design.md:209)); receiving that result triggers deletion ([run spec:235](/home/paulca/amadan.net/carlos/platform/.claude/worktrees/run-and-seeded/docs/superpowers/specs/2026-10-08-carlos-run-design.md:235)). A completed command with several pages of output loses everything beyond the first returned page. **Fix:** publish the result only after final chunks are durable, require the CLI to consume through the declared final sequence, and require that sequence on acknowledgement. |
| + |
| +6. **P2 — Run §§5.2, 5.7: the upload directory is outside the instance’s mounted tree for long hostnames.** The spec uses `/data/instances/<host>/`; actual nested directories use `InstanceID(host)`, which hashes names longer than 40 bytes ([provision.go:2032](/home/paulca/amadan.net/carlos/platform/.claude/worktrees/run-and-seeded/internal/provision/provision.go:2032)). **Fix:** derive download, `CARLOS_RUN_DIR` and cleanup paths through `provision.InstanceDir` or the backend’s resolved instance paths. |
| + |
| +7. **P3 — Run §4: `seedVars` does not supply the promised region.** It strips the control variables and sets only `CARLOS_HOST` ([backend_exec.go:1359](/home/paulca/amadan.net/carlos/platform/.claude/worktrees/run-and-seeded/internal/activator/backend_exec.go:1359)). Authoritative region assignment occurs in the serving-process composer ([backend_exec.go:1340](/home/paulca/amadan.net/carlos/platform/.claude/worktrees/run-and-seeded/internal/activator/backend_exec.go:1340)). **Fix:** specify a cmd environment composer that sets or removes `CARLOS_REGION` from the backend’s region before adding the run variables. |
| + |
| +8. **P1 — Seeded §3.3: the shared warm-file shortcut contradicts claim restore.** Consider a crash after installing the seed locally but before its first upload. Step 1 says the warm file suppresses seed installation. Claim recovery, however, never trusts that file: its empty-baseline path purges local state and writes only a generation marker ([claimwakeassembly.go:252](/home/paulca/amadan.net/carlos/platform/.claude/worktrees/run-and-seeded/internal/host/claimwakeassembly.go:252), [restore.go:142](/home/paulca/amadan.net/carlos/platform/.claude/worktrees/run-and-seeded/internal/hostbox/restore.go:142)). Applying the stated shared algorithm therefore removes the only installed copy without reinstalling the seed. **Fix:** separate the two restore algorithms. In claim mode, select the remote baseline first; if initial-seed authority remains and no baseline exists, purge and reinstall the seed under the admitted generation regardless of a local file. Preserve generation publication before retiring seed recovery state. |
| + |
| +9. **P1 — Seeded §§5.4, 6.4: app-name equality does not establish the destination account for an operator-filed row.** App identity is account-scoped ([qualified.go:4](/home/paulca/amadan.net/carlos/platform/.claude/worktrees/run-and-seeded/internal/release/qualified.go:4)). An operator row named `foo` does not distinguish the intended `bae/foo` from another account’s `foo`. The proposed exception removes the account comparison without replacing that missing ownership evidence. **Fix:** retain the operator-account exception, but bind each permitted transfer to an operator-authored authorization naming the exact host, old address, destination account and app. Check that authorization during the swap. |
| + |
| +10. **P2 — Seeded §5.4: `Kind == "addr"` does not describe an existing route class.** Registry kinds are `instance`, `service`, `static` and `sidecar`; TCP backing is represented separately by `Addr`/`Addrs`, and `Socket` takes precedence ([registry.go:31](/home/paulca/amadan.net/carlos/platform/.claude/worktrees/run-and-seeded/internal/registry/registry.go:31)). The literal predicate refuses the intended imports. **Fix:** define the accepted legacy route shape using the actual kind and backing fields, including empty socket, tunnel and incompatible ownership fields. |
| + |
| +11. **P2 — Seeded §§5.1, 5.4: ordinary creation and reconcile cannot perform the described first claim acquisition.** Console creation acquires the claim before record effects ([claim_create.go:99](/home/paulca/amadan.net/carlos/platform/.claude/worktrees/run-and-seeded/internal/console/claim_create.go:99)). Reconcile then rejects an existing unclaimed row before calling its claim-install hook, and requires an existing owner to equal the installed owner ([claimassembly.go:20](/home/paulca/amadan.net/carlos/platform/.claude/worktrees/run-and-seeded/internal/provision/claimassembly.go:20)). **Fix:** specify an import-specific acquisition and replacement transaction, with durable recovery between claim CAS and registry commit. Define how begin remains pending, how the exact legacy row is checked, and how a failed or interrupted swap settles its acquired claim. |
| + |
| +12. **P2 — Seeded §§5.4, 6.3: the existing-row branch derives a flat replacement.** Reconcile chooses `nested` from the existing row’s paths; it reads the box’s mint layout only when no row exists ([reconcile.go:1103](/home/paulca/amadan.net/carlos/platform/.claude/worktrees/run-and-seeded/internal/provision/reconcile.go:1103)). An addr row consequently produces flat exec paths, which the seed’s nested-only wake must refuse. **Fix:** the import branch must validate current nested layout and explicitly derive the replacement’s nested paths, independently of the addr row. |
| + |
| +13. **P2 — Seeded §§3.2, 5.5: the state filter makes rollback unreachable.** Section 3.2 skips every state except `ready` and `placed`; §5.5 first changes the state to `aborted`, then relies on reconcile to restore the old row. Reconcile’s per-record dispatch is where this ordering must be implemented ([reconcile.go:1072](/home/paulca/amadan.net/carlos/platform/.claude/worktrees/run-and-seeded/internal/provision/reconcile.go:1072)). **Fix:** dispatch `aborted` to an idempotent rollback path before applying the ordinary seed-admission filter. |
| + |
| +14. **P2 — Seeded §5.5: `Replaces` cannot restore the original route exactly.** The original route includes account, kind, channel, environment, gate, unit, capabilities and other fields beyond its address ([registry.go:36](/home/paulca/amadan.net/carlos/platform/.claude/worktrees/run-and-seeded/internal/registry/registry.go:36)). The proposed seed record retains none of that original state. **Fix:** durably journal the complete original row before replacement, compare it inside the swap transaction, and restore that exact snapshot on abort. Include ownership and associated state that the transaction changes. |
| + |
| +15. **P1 — Seeded §§5.5–5.6: the abort safety condition is neither authenticated nor durably observable.** Usage accounting aggregates request counts and bytes by host ([usagemeter.go:36](/home/paulca/amadan.net/carlos/platform/.claude/worktrees/run-and-seeded/internal/edge/usagemeter.go:36)); analytics records paths and methods, but no authenticated import-verification identity ([meter.go:19](/home/paulca/amadan.net/carlos/platform/.claude/worktrees/run-and-seeded/internal/analytics/meter.go:19)). An app-defined verify path cannot establish that no ordinary request has served. **Fix:** introduce verification authority bound to the import incarnation and a durable “ordinary traffic admitted” latch. Serialize abort against that latch before forwarding ordinary requests, so an accepted write cannot race replica deletion. |
| + |
| +16. **P1 — Seeded §§3.2, 10: observing upgraded boxes does not prevent a later downgrade from consuming seed records.** The current protocol is version 1 ([floor.go:15](/home/paulca/amadan.net/carlos/platform/.claude/worktrees/run-and-seeded/internal/moves/floor.go:15)). The selector’s rollback checks apply only when a floor exists ([boxselect.go:92](/home/paulca/amadan.net/carlos/platform/.claude/worktrees/run-and-seeded/internal/console/boxselect.go:92)); recovery can execute `carlos.prev` when no local floor is armed ([recovery unit:29](/home/paulca/amadan.net/carlos/platform/.claude/worktrees/run-and-seeded/internal/host/units/carlos-system-recover.service:29)). **Fix:** allocate a seed-capable protocol/release floor, publish it, and require evidence that affected boxes have armed it and excluded incompatible rollback binaries before writing any seed state, including `awaiting-upload`. |
| + |
| +17. **P1 — Seeded §§2, 4: calling `dbGate` rejects the required service credentials.** Its handler explicitly requires `PrincipalUser` ([apidb.go:110](/home/paulca/amadan.net/carlos/platform/.claude/worktrees/run-and-seeded/internal/console/apidb.go:110)); the DB routes independently reject all service roles ([authcli.go:690](/home/paulca/amadan.net/carlos/platform/.claude/worktrees/run-and-seeded/internal/console/authcli.go:690)). **Fix:** authorize the seed door through the instance-role gate, then reuse a factored source resolver containing the record, account/app, residency and versioned-claim checks. Keep the existing DB download door’s authorization unchanged. |
| + |
| +18. **P2 — Seeded §4: `RestoreTo` cannot honor `--at`.** Its API accepts no transaction or time target, and its restore options set only the output path ([restore.go:64](/home/paulca/amadan.net/carlos/platform/.claude/worktrees/run-and-seeded/internal/dbdoor/restore.go:64), [restore.go:132](/home/paulca/amadan.net/carlos/platform/.claude/worktrees/run-and-seeded/internal/dbdoor/restore.go:132)). **Fix:** specify a target-aware restore API and historical planning under the pinned source view, including unavailable-history errors and the actual restored TXID. |
| + |
| +19. **P2 — Seeded §§4, 5.2: the resource limits do not bound restore scratch or explain upload validation.** `RestoreTo` downloads complete planned LTX files before restoration finishes ([restore.go:186](/home/paulca/amadan.net/carlos/platform/.claude/worktrees/run-and-seeded/internal/dbdoor/restore.go:186)). Checking the output against 4 GiB afterward cannot prevent disk exhaustion. Import’s `CheckDB` also requires a local database file and scans its pages ([check.go:42](/home/paulca/amadan.net/carlos/platform/.claude/worktrees/run-and-seeded/internal/lease/check.go:42)), despite the upload being described solely as streaming to the bucket. **Fix:** define bounded scratch for both doors, accounting for LTX files and restored output together, with disk admission, deadlines, cancellable checks and cleanup on failure, disconnect and restart. |
| + |
| +20. **P2 — Seeded §5.2: multipart upload and failure cleanup require additional storage support and permissions.** The existing streaming interface requires a seekable body ([store.go:248](/home/paulca/amadan.net/carlos/platform/.claude/worktrees/run-and-seeded/internal/release/store.go:248)); its S3 implementation makes one `PutObject` call ([store.go:799](/home/paulca/amadan.net/carlos/platform/.claude/worktrees/run-and-seeded/internal/release/store.go:799)). The bucket policy grants object get/put/delete, without multipart-abort authority ([main.tf:163](/home/paulca/amadan.net/carlos/platform/.claude/worktrees/run-and-seeded/infra/modules/region-host/main.tf:163)). **Fix:** specify multipart initiation, part upload, completion and abort, with scoped abort permission and recovery of abandoned upload IDs. Deleting an object is not cleanup of an unfinished multipart upload. |
| + |
| +`carlos run`: **revise**. |
| + |
| +Seeded instances: **revise**. |
| \ No newline at end of file |