| 1 | # Seeded Instances, Plan 5: `carlos instances create --from` Implementation Plan |
| 2 | |
| 3 | > **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. |
| 4 | |
| 5 | **Goal:** A member or an `instance`-role service credential creates a new instance whose database starts as a copy of a sibling's replica, at the latest transaction or a named TXID, inside the same app, account and region, with its own environment and an empty private store. |
| 6 | |
| 7 | **Architecture:** The create door (`handleAPIInstanceCreate`) runs every existing check, then hands a request carrying `"from"` to `createFrom` in a new file. It checks the region's readiness (Plan 3), resolves the source with `resolveDBSource` (factored out of `dbGate`, whose own person-only gate and messages stay), plans the restore through `dbdoor.PlanRestore` (the last planned LTX header's `Commit × PageSize` is the restored size), admits it against the 4 GiB limit and a shared scratch reservation pool with per-app and deployment limits, reserves the record (`reserving`, with the seed `ID`), provisions the private store marked with that seed `ID`, restores with `dbdoor.RestoreToTarget` into private scratch under a 30-minute deadline and a size watcher, uploads the file with one `PutFrom`, and publishes the record `ready`. Any failure after the reservation CASes the record to `aborted`. The console restores through its own in-process door (`replicaDoer`), so the restore is the same code `carlos db pull` runs. |
| 8 | |
| 9 | **Tech Stack:** Go 1.25, litestream v0.5.14 (`CalcRestorePlan`, `RestoreOptions.TXID`), superfly/ltx v0.5.1 (`Header`, `HeaderSize = 100`), `release.PutObjectFrom`, `objstore`. |
| 10 | |
| 11 | **Spec:** `docs/superpowers/specs/2026-10-08-seeded-instances-design.md` rev 7: §4 in full (codex 17, 18, 19, R2-18, R2-19, R3-2, R5-1), §3.1 (the console writes the snapshot; residency), §6.1, §6.2, §7 (`--from`'s list cell, the `instance-seed` audit row). Index: `2026-10-08-seeded-instances-0-index.md`. Copy deck rows SD-N20 to SD-N47, SD-N114. |
| 12 | |
| 13 | ## Global Constraints |
| 14 | |
| 15 | - Every user-facing string (CLI output, errors, console messages, docs prose) is copy-reviewed by Paul before it is written into code; implementation tasks use only strings from the copy deck. Copy voice: no em dashes; full stops and new sentences instead. |
| 16 | - CI cross-compiles darwin and windows: linux-only code needs `//go:build linux` plus a `!linux` stub. |
| 17 | - Local test commands: targeted tests (`go test ./internal/<pkg>/ -run TestX`), not the full suite in a loop. Inside the sandbox the default Go build cache may be read-only; if so use `GOCACHE=$TMPDIR/gocache` re-exported per command. Socket tests need a short TMPDIR (`TMPDIR=/tmp/g`). The local gate may need `GOFLAGS=-buildvcs=false`. |
| 18 | - Heavy loops run under `nice`; no -race full suite until the end of a plan. |
| 19 | - Production use needs nested boxes; the claim-armed path is refused here and left to #250 (spec §3.4). |
| 20 | - Commits end with `Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>`. |
| 21 | - No new file in this series imports `internal/moves`. New sentences go in new files (`apiinstanceseed.go`, `seedscratch.go`), never in a file listed in `movesScanFiles` (`apiinstances.go`, `apidb.go`'s neighbour `authcli.go`, `server.go`, `cmd/console/main.go` are listed; `apiinstances.go`, `authcli.go` and `server.go` are also legacy-exempt, where `TestMovesLegacyExemptionsAreLegacy` fails on any new sentence). |
| 22 | - `TestClaimAuthoritySourceCensus` inventories console and `cmd/console` calls in its `selected` set; add `NewS3StoreIn` to that set and every new site (including any `Open` or `WithAppMutation` this plan writes in `internal/console`) to `internal/console/testdata/claim-authority-sites.txt`. |
| 23 | - The `instance` role reaches this door through `cliRoute`'s existing `svcrole.Allow(svcrole.Instance)` on `POST …/instances` (`authcli.go:660`). The db door's routes stay `svcrole.Set{}` (`authcli.go:690-691`), and `dbGate` keeps refusing every service principal (`apidb.go:110`). |
| 24 | - When a task wires a deck row, append its id to `seededWiredIDs` in `approved_seeded_copy_test.go` in the same commit. |
| 25 | - Write each deck sentence as exactly one string literal or one format string holding the row's text, with `%s` or `%d` at each `<placeholder>` and nothing else: no prefix, no trailing `\n`, no leading spaces. The gate's matcher (`movesRowSpoken`, `approved_moves_copy_test.go:811`) anchors both ends. Print with `fmt.Fprintln(w, sentence)` or a `" %s\n"` format around a separately built sentence. |
| 26 | |
| 27 | ## Review Focus |
| 28 | |
| 29 | 1. A client that disconnects mid-restore must still leave nothing behind: the scratch directory removed, the reservation released, and the reserved record CASed to `aborted` under a context that is not the request's (which is already cancelled). Test in Task 5 (`TestFromDisconnectAbortsTheRecordAndCleansScratch`). |
| 30 | 2. Two concurrent `--from` creates for the same app: the second gets 429 before it reads anything or writes a record; a third for another app proceeds until the deployment's two are busy. Test in Task 3 (`TestScratchLimitsPerAppAndDeployment`). |
| 31 | 3. `--at` with a TXID older than the replica's oldest retained snapshot must answer "history before <txid> is no longer kept", not "nothing replicated" and not a 502. Test in Task 2 (`TestPlanRestoreTargetBeforeHistory`). |
| 32 | 4. An app with private storage: an existing store for the new host's derived environment is refused unless its declared seed is this seed's ID, and no store of the source is read, bound or touched. Test in Task 5 (`TestFromPrivateStorageRules`). |
| 33 | 5. A restore that overruns the size it reserved must be cancelled by the watcher and the record aborted. Test in Task 5 (`TestFromWatcherCancelsAnOvergrownRestore`). |
| 34 | |
| 35 | --- |
| 36 | |
| 37 | ## File structure |
| 38 | |
| 39 | | path | responsibility | |
| 40 | |---|---| |
| 41 | | `internal/console/apidb.go` | `resolveDBSource` factored out of `dbGate` | |
| 42 | | `internal/dbdoor/restore.go` | `RestoreToTarget`, `ErrHistoryGone` | |
| 43 | | `internal/dbdoor/plan.go` | `PlanRestore`, `RestorePlan` | |
| 44 | | `internal/console/seeddoer.go` | `replicaDoer`: the db door's tails served in process | |
| 45 | | `internal/console/seedscratch.go`, `seedscratch_unix.go`, `seedscratch_windows.go` | the scratch reservation pool, the limits, free space, start-up cleanup | |
| 46 | | `internal/console/apiinstanceseed.go` | `createFrom` and its helpers | |
| 47 | | `internal/console/apiinstances.go` | the request's `from` field; the hand-off; the explicit-environment refusal | |
| 48 | | `internal/console/instanceprivatestore.go` | `instanceStoreProvisionFor(…, seedID)` | |
| 49 | | `internal/objstore/objstore.go` | `Declared.Seed` | |
| 50 | | `internal/console/server.go`, `cmd/console/main.go` | `Config.SeedData`, `Config.SeedScratchDir`, the pool at start | |
| 51 | | `instances_cli.go`, `apimode.go` | `--from`, `--at`, the client call, the SEED column | |
| 52 | | `docs/site/cli.md` | SD-N47 | |
| 53 | |
| 54 | --- |
| 55 | |
| 56 | ### Task 1: `resolveDBSource` |
| 57 | |
| 58 | **Files:** |
| 59 | - Modify: `internal/console/apidb.go:110-188` (`dbGate`) |
| 60 | - Test: `internal/console/apidb_source_test.go` |
| 61 | |
| 62 | **Interfaces:** |
| 63 | - Consumes: `resolveReplica`, `dbClassRefusal`, `dbClaim(ctx, g dbRequest, rec provision.Record) (hostclaim.Observed, error)`. |
| 64 | - Produces: `type dbSource struct{ rec provision.Record; prefix string; claim hostclaim.Observed }`; `(s *Server) resolveDBSource(ctx context.Context, acct *Account, actor, app, host string) (dbSource, error)`; sentinel `errSourceClass` (wrapping `dbClassRefusal`'s sentence) and `errSourceClaim` (wrapping the claim read's error). Its errors: `os.ErrNotExist` (no record), `ErrBadHost`, `errSourceClass`, `ErrPooled`, `ErrResidency`, `errSourceClaim`, or a read error. The checks, in order: the record exists; it is this account's and app's (the key path guarantees it, and `GetRecord` refuses a body that disagrees); its class (`dbClassRefusal`: an instance, not addr-backed, hibernating); not pooled and not on a fleet; residency; the versioned claim read when `s.cfg.HostClaim`. |
| 65 | |
| 66 | `dbGate` keeps its order and every answer: person-only, membership, the rate limit, the bucket check, the host shape, then `resolveDBSource`, mapping each error exactly as it maps the same condition today. |
| 67 | |
| 68 | - [ ] **Step 1: Write the failing test** |
| 69 | |
| 70 | ```go |
| 71 | package console |
| 72 | |
| 73 | import ( |
| 74 | "context" |
| 75 | "errors" |
| 76 | "os" |
| 77 | "strings" |
| 78 | "testing" |
| 79 | "time" |
| 80 | |
| 81 | "github.com/carlosframework/platform/internal/provision" |
| 82 | ) |
| 83 | |
| 84 | func TestResolveDBSource(t *testing.T) { |
| 85 | fx, _ := instanceFixture(t, "code-db-source") |
| 86 | m := titoManifest() |
| 87 | m.Pool = "" |
| 88 | enableInstances(t, fx, "titogo", m) |
| 89 | fx.srv.cfg.Region = "ie" |
| 90 | fx.srv.cfg.HibernatePrefix = "instances" |
| 91 | acct, err := fx.st.AccountBySqid("bab") |
| 92 | if err != nil { |
| 93 | t.Fatal(err) |
| 94 | } |
| 95 | ctx := context.Background() |
| 96 | put := func(host string, mut func(*provision.Record)) { |
| 97 | rec := provision.Record{Schema: 1, Host: host, Slug: strings.Split(host, ".")[0], App: "titogo", Account: "bab", |
| 98 | Region: "ie", Channel: "stable", CreatedAt: time.Now().UTC()} |
| 99 | if mut != nil { |
| 100 | mut(&rec) |
| 101 | } |
| 102 | if err := provision.CreateRecord(ctx, fx.rel, rec); err != nil { |
| 103 | t.Fatal(err) |
| 104 | } |
| 105 | } |
| 106 | put("acme.go.tito.io", nil) |
| 107 | put("pooled.go.tito.io", func(r *provision.Record) { r.Pool = "tito" }) |
| 108 | put("far.go.tito.io", func(r *provision.Record) { r.Region = "uk" }) |
| 109 | put("addr.go.tito.io", func(r *provision.Record) { r.Addr = "127.0.0.1:9000" }) |
| 110 | |
| 111 | src, err := fx.srv.resolveDBSource(ctx, acct, "alice@keymail.dev", "titogo", "acme.go.tito.io") |
| 112 | if err != nil || src.prefix != "instances/acme.go.tito.io" || src.rec.Host != "acme.go.tito.io" { |
| 113 | t.Fatalf("acme = %+v, %v", src, err) |
| 114 | } |
| 115 | for host, want := range map[string]error{ |
| 116 | "nope.go.tito.io": os.ErrNotExist, |
| 117 | "pooled.go.tito.io": ErrPooled, |
| 118 | "far.go.tito.io": ErrResidency, |
| 119 | "addr.go.tito.io": errSourceClass, |
| 120 | } { |
| 121 | if _, err := fx.srv.resolveDBSource(ctx, acct, "alice@keymail.dev", "titogo", host); !errors.Is(err, want) { |
| 122 | t.Errorf("%s: err = %v, want %v", host, err, want) |
| 123 | } |
| 124 | } |
| 125 | } |
| 126 | ``` |
| 127 | |
| 128 | `Store.AccountBySqid(sqid) (*Account, error)` is `store.go:991`. |
| 129 | |
| 130 | - [ ] **Step 2: Run it to verify it fails** |
| 131 | |
| 132 | Run: `go test ./internal/console/ -run TestResolveDBSource -count=1` |
| 133 | Expected: FAIL to compile: `fx.srv.resolveDBSource undefined`. |
| 134 | |
| 135 | - [ ] **Step 3: Factor it out** |
| 136 | |
| 137 | In `apidb.go`: |
| 138 | |
| 139 | ```go |
| 140 | var ( |
| 141 | errSourceClass = errors.New("db: not an instance the platform keeps a replica of") |
| 142 | errSourceClaim = errors.New("db: the instance's claim could not be read") |
| 143 | ) |
| 144 | |
| 145 | // dbSource is a replica source both the db door and the --from door may |
| 146 | // read (seeded-instances spec §4, codex 17): a record of this account and |
| 147 | // app, an instance with a replica of its own, homed in this console's |
| 148 | // region, and, with claims on, its versioned claim. |
| 149 | type dbSource struct { |
| 150 | rec provision.Record |
| 151 | prefix string |
| 152 | claim hostclaim.Observed |
| 153 | } |
| 154 | |
| 155 | // resolveDBSource holds the source checks dbGate made inline. It does no |
| 156 | // authorization: dbGate keeps its person-only gate, and the --from door |
| 157 | // authorizes through cliRoute's instance role. |
| 158 | func (s *Server) resolveDBSource(ctx context.Context, acct *Account, actor, app, host string) (dbSource, error) { |
| 159 | if !provision.ValidHostShape(host) { |
| 160 | return dbSource{}, fmt.Errorf("%w: %q", ErrBadHost, host) |
| 161 | } |
| 162 | rec, err := provision.GetRecord(ctx, s.cfg.Releases, acct.Sqid, app, host) |
| 163 | if err != nil { |
| 164 | return dbSource{}, err // os.ErrNotExist passes through as itself |
| 165 | } |
| 166 | if msg := dbClassRefusal(*rec); msg != "" { |
| 167 | return dbSource{rec: *rec}, fmt.Errorf("%w: %s", errSourceClass, msg) |
| 168 | } |
| 169 | prefix, err := s.resolveReplica(*rec, host) |
| 170 | if err != nil { |
| 171 | return dbSource{rec: *rec}, err |
| 172 | } |
| 173 | src := dbSource{rec: *rec, prefix: prefix} |
| 174 | if s.cfg.HostClaim { |
| 175 | if s.cfg.DBVersions == nil { |
| 176 | return src, fmt.Errorf("%w: this console has no versioned database reader", errSourceClaim) |
| 177 | } |
| 178 | g := dbRequest{acct: acct, actor: actor, app: app, host: host} |
| 179 | if src.claim, err = s.dbClaim(ctx, g, *rec); err != nil { |
| 180 | return src, fmt.Errorf("%w: %w", errSourceClaim, err) |
| 181 | } |
| 182 | src.prefix = src.claim.Claim.Holder.Location |
| 183 | } |
| 184 | return src, nil |
| 185 | } |
| 186 | ``` |
| 187 | |
| 188 | Rewrite `dbGate` from the record read on (`apidb.go:135`) to call it and keep every existing answer: |
| 189 | |
| 190 | ```go |
| 191 | ctx := r.Context() |
| 192 | src, err := s.resolveDBSource(ctx, acct, actor, app, host) |
| 193 | switch { |
| 194 | case errors.Is(err, os.ErrNotExist), errors.Is(err, ErrBadHost): |
| 195 | http.NotFound(w, r) |
| 196 | return dbRequest{}, false |
| 197 | case errors.Is(err, errSourceClass): |
| 198 | jsonError(w, http.StatusConflict, dbClassRefusal(src.rec)) |
| 199 | return dbRequest{}, false |
| 200 | case errors.Is(err, ErrPooled): |
| 201 | jsonError(w, http.StatusConflict, fmt.Sprintf( |
| 202 | "%s runs in the pool %s, so its data is on the pool's own server. carlos db cannot read it.", host, src.rec.Pool)) |
| 203 | return dbRequest{}, false |
| 204 | case errors.Is(err, ErrResidency): |
| 205 | jsonError(w, http.StatusConflict, s.dbResidencyRefusal(src.rec)) |
| 206 | return dbRequest{}, false |
| 207 | case errors.Is(err, errSourceClaim): |
| 208 | g := dbRequest{acct: acct, actor: actor, app: app, host: host} |
| 209 | if s.cfg.DBVersions == nil { |
| 210 | jsonError(w, http.StatusServiceUnavailable, "this console has no versioned database reader") |
| 211 | return dbRequest{}, false |
| 212 | } |
| 213 | s.dbClaimFailure(w, r, g, errors.Unwrap(err)) |
| 214 | return dbRequest{}, false |
| 215 | case err != nil: |
| 216 | log.Printf("console: api db %s/%s/%s: read record: %v", acct.Sqid, app, host, err) |
| 217 | jsonError(w, http.StatusBadGateway, "could not read the instance. Try again.") |
| 218 | return dbRequest{}, false |
| 219 | } |
| 220 | if s.cfg.HostClaim { |
| 221 | return dbRequest{claim: src.claim, acct: acct, actor: actor, app: app, host: host, prefix: src.prefix}, true |
| 222 | } |
| 223 | rd, err := s.dbReplica(ctx, src.prefix) |
| 224 | if err != nil { |
| 225 | log.Printf("console: api db %s/%s/%s: replica client: %v", acct.Sqid, app, host, err) |
| 226 | jsonError(w, http.StatusBadGateway, "could not read the replica. Try again.") |
| 227 | return dbRequest{}, false |
| 228 | } |
| 229 | return dbRequest{acct: acct, actor: actor, app: app, host: host, prefix: src.prefix, rd: rd}, true |
| 230 | ``` |
| 231 | |
| 232 | `errors.Unwrap` of a `fmt.Errorf("%w: %w", …)` returns nil (it has two wrapped errors); store the claim error on `dbSource` instead if `dbClaimFailure` needs the original: add `claimErr error` to `dbSource`, set it beside the wrap, and pass `src.claimErr`. Every string above is `main`'s own, moved, not new: `TestMovesLegacyExemptionsAreLegacy` does not scan `apidb.go`. |
| 233 | |
| 234 | - [ ] **Step 4: Run the tests to verify they pass** |
| 235 | |
| 236 | Run: `go test ./internal/console/ -run 'TestResolveDBSource|TestAPIDB|TestDB|TestLeftover' -count=1` |
| 237 | Expected: PASS: every db door test passes unchanged. |
| 238 | |
| 239 | - [ ] **Step 5: Commit** |
| 240 | |
| 241 | ```bash |
| 242 | git add internal/console/apidb.go internal/console/apidb_source_test.go |
| 243 | git commit -m "console: resolveDBSource, the db door's source checks, factored out (Plan 5 Task 1) |
| 244 | |
| 245 | Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>" |
| 246 | ``` |
| 247 | |
| 248 | ### Task 2: Restore to a TXID, and the restore plan's exact size |
| 249 | |
| 250 | **Files:** |
| 251 | - Modify: `internal/dbdoor/restore.go:64` (`RestoreTo`), `:132` (`restoreOnce`) |
| 252 | - Create: `internal/dbdoor/plan.go` |
| 253 | - Test: `internal/dbdoor/plan_test.go`, `internal/dbdoor/restore_test.go` |
| 254 | |
| 255 | **Interfaces:** |
| 256 | - Consumes: `litestream.CalcRestorePlan(ctx, client ReplicaClient, txID ltx.TXID, timestamp time.Time, logger *slog.Logger) ([]*ltx.FileInfo, error)` (v0.5.14 `replica.go:1441`), `ltx.Header.UnmarshalBinary`, `ltx.HeaderSize`. |
| 257 | - Produces: `RestoreToTarget(ctx context.Context, c *Client, parent string, target uint64) (*Restored, error)` (target 0 is latest; `RestoreTo` becomes `RestoreToTarget(ctx, c, parent, 0)`); `ErrHistoryGone`; `type RestorePlan struct{ TXID uint64; LTXBytes int64; RestoredBytes int64; Files int }`; `PlanRestore(ctx context.Context, c *Client, target uint64) (RestorePlan, error)`. |
| 258 | |
| 259 | `PlanRestore` asks litestream for the plan the restore will follow (a target at or past the latest transaction plans the latest copy, `Target` 0), then reads the first `ltx.HeaderSize` (100) bytes of the last planned file with a ranged open: its `Commit` (pages after its last transaction) times its `PageSize` is the restored database's size. `TXID` is the last file's `MaxTXID`, the transaction the restore will actually reach (it can be past the target: litestream applies whole files). With a target, litestream's "not available" errors mean the history before the target is gone (`ErrHistoryGone`); without one, nothing was replicated (`ErrNothingReplicated`). |
| 260 | |
| 261 | - [ ] **Step 1: Write the failing tests** |
| 262 | |
| 263 | `internal/dbdoor/plan_test.go`, on the package's own kit: `dbdoortest.Replicate(t, dir, batches)` (`dbdoortest/replica.go:31`, one L0 file per batch), `newTestDoor(t, src).do` (the in-process door `restore_test.go` uses) and `collect(t, c, level, seek)` (`client_test.go:235`): |
| 264 | |
| 265 | ```go |
| 266 | package dbdoor |
| 267 | |
| 268 | import ( |
| 269 | "context" |
| 270 | "errors" |
| 271 | "os" |
| 272 | "testing" |
| 273 | |
| 274 | "github.com/benbjohnson/litestream" |
| 275 | "github.com/superfly/ltx" |
| 276 | |
| 277 | "github.com/carlosframework/platform/internal/dbdoor/dbdoortest" |
| 278 | ) |
| 279 | |
| 280 | func TestPlanRestoreSizesTheRestoreExactly(t *testing.T) { |
| 281 | src := t.TempDir() |
| 282 | dbdoortest.Replicate(t, src, 4) |
| 283 | plan, err := PlanRestore(context.Background(), NewClient(newTestDoor(t, src).do), 0) |
| 284 | if err != nil { |
| 285 | t.Fatal(err) |
| 286 | } |
| 287 | r, err := RestoreToTarget(context.Background(), NewClient(newTestDoor(t, src).do), t.TempDir(), 0) |
| 288 | if err != nil { |
| 289 | t.Fatal(err) |
| 290 | } |
| 291 | defer os.RemoveAll(r.Dir) |
| 292 | st, err := os.Stat(r.Path) |
| 293 | if err != nil { |
| 294 | t.Fatal(err) |
| 295 | } |
| 296 | if plan.RestoredBytes != st.Size() || plan.TXID != r.TXID || plan.LTXBytes <= 0 || plan.Files == 0 { |
| 297 | t.Fatalf("plan %+v, restored %d bytes at %d; want the plan's size and txid", plan, st.Size(), r.TXID) |
| 298 | } |
| 299 | } |
| 300 | |
| 301 | func TestRestoreToATargetTXID(t *testing.T) { |
| 302 | src := t.TempDir() |
| 303 | dbdoortest.Replicate(t, src, 4) |
| 304 | files := collect(t, NewClient(newTestDoor(t, src).do), 0, 0) |
| 305 | at := uint64(files[0].MaxTXID) |
| 306 | r, err := RestoreToTarget(context.Background(), NewClient(newTestDoor(t, src).do), t.TempDir(), at) |
| 307 | if err != nil { |
| 308 | t.Fatal(err) |
| 309 | } |
| 310 | defer os.RemoveAll(r.Dir) |
| 311 | if r.TXID != at { |
| 312 | t.Fatalf("restored to %d, want the first file's end %d", r.TXID, at) |
| 313 | } |
| 314 | if rows := countRows(t, r.Path); rows != dbdoortest.Rows { |
| 315 | t.Fatalf("rows = %d, want one batch's %d", rows, dbdoortest.Rows) |
| 316 | } |
| 317 | } |
| 318 | |
| 319 | func TestPlanRestoreTargetBeforeHistory(t *testing.T) { |
| 320 | src := t.TempDir() |
| 321 | dbdoortest.Replicate(t, src, 4) |
| 322 | files := collect(t, NewClient(newTestDoor(t, src).do), 0, 0) |
| 323 | first := files[0] |
| 324 | // The oldest file is gone (retention): history before its end is not kept. |
| 325 | if err := os.Remove(litestream.LTXFilePath(src, 0, first.MinTXID, first.MaxTXID)); err != nil { |
| 326 | t.Fatal(err) |
| 327 | } |
| 328 | if _, err := PlanRestore(context.Background(), NewClient(newTestDoor(t, src).do), uint64(first.MinTXID)); !errors.Is(err, ErrHistoryGone) { |
| 329 | t.Fatalf("a target before the oldest kept file: err = %v, want ErrHistoryGone", err) |
| 330 | } |
| 331 | if _, err := PlanRestore(context.Background(), NewClient(newTestDoor(t, t.TempDir()).do), 0); !errors.Is(err, ErrNothingReplicated) { |
| 332 | t.Fatalf("an empty replica: err = %v, want ErrNothingReplicated", err) |
| 333 | } |
| 334 | _ = ltx.HeaderSize |
| 335 | } |
| 336 | ``` |
| 337 | |
| 338 | (`countRows` is `restore_test.go`'s. Drop the `_ = ltx.HeaderSize` line and the `ltx` import if nothing else in the file needs it; `litestream.LTXFilePath(root, level, min, max)` is the file replica's own layout, `file/replica_client.go:90`.) |
| 339 | |
| 340 | - [ ] **Step 2: Run the tests to verify they fail** |
| 341 | |
| 342 | Run: `go test ./internal/dbdoor/ -run 'TestPlanRestore|TestRestoreToATarget' -count=1` |
| 343 | Expected: FAIL to compile: `undefined: PlanRestore`. |
| 344 | |
| 345 | - [ ] **Step 3: Add the target to the restore** |
| 346 | |
| 347 | In `restore.go`: |
| 348 | |
| 349 | ```go |
| 350 | // ErrHistoryGone: a restore target older than the replica's retained |
| 351 | // history (seeded-instances spec §4 step 4). |
| 352 | var ErrHistoryGone = errors.New("dbdoor: history before the target is no longer kept") |
| 353 | |
| 354 | func RestoreTo(ctx context.Context, c *Client, parent string) (*Restored, error) { |
| 355 | return RestoreToTarget(ctx, c, parent, 0) |
| 356 | } |
| 357 | |
| 358 | // RestoreToTarget is RestoreTo at a target transaction: 0 is the latest. |
| 359 | // It restores whole planned files, so the copy can reach past target; |
| 360 | // Restored.TXID says where it ended. |
| 361 | func RestoreToTarget(ctx context.Context, c *Client, parent string, target uint64) (*Restored, error) { |
| 362 | // ... RestoreTo's body as it is, with two changes: |
| 363 | // err = restoreOnce(ctx, rec, out, target) |
| 364 | // case errors.Is(err, litestream.ErrNoSnapshots) || errors.Is(err, litestream.ErrTxNotAvailable): |
| 365 | // if target != 0 { return nil, ErrHistoryGone } |
| 366 | // return nil, ErrNothingReplicated |
| 367 | } |
| 368 | |
| 369 | func restoreOnce(ctx context.Context, client litestream.ReplicaClient, out string, target uint64) error { |
| 370 | opt := litestream.NewRestoreOptions() |
| 371 | opt.OutputPath = out |
| 372 | opt.TXID = ltx.TXID(target) |
| 373 | return litestream.NewReplicaWithClient(nil, client).Restore(ctx, opt) |
| 374 | } |
| 375 | ``` |
| 376 | |
| 377 | Move `RestoreTo`'s whole body into `RestoreToTarget` and make those two edits; nothing else in it changes (`spec §12`'s three conditions still hold: `Follow` stays false, the client is not a `ReplicaClientV3`, `CheckDB` runs after). |
| 378 | |
| 379 | - [ ] **Step 4: Write `internal/dbdoor/plan.go`** |
| 380 | |
| 381 | ```go |
| 382 | package dbdoor |
| 383 | |
| 384 | import ( |
| 385 | "context" |
| 386 | "errors" |
| 387 | "fmt" |
| 388 | "io" |
| 389 | "log/slog" |
| 390 | "time" |
| 391 | |
| 392 | "github.com/benbjohnson/litestream" |
| 393 | "github.com/superfly/ltx" |
| 394 | ) |
| 395 | |
| 396 | // RestorePlan is what a restore will read and produce, known before |
| 397 | // anything is downloaded (seeded-instances spec §4 step 5, codex R2-19). |
| 398 | type RestorePlan struct { |
| 399 | TXID uint64 // the last planned file's MaxTXID: where the restore ends |
| 400 | LTXBytes int64 // the planned files' compressed sizes, summed |
| 401 | RestoredBytes int64 // the restored database's size: Commit pages times PageSize |
| 402 | Files int |
| 403 | // Target is what to ask RestoreToTarget for: the requested target, or |
| 404 | // 0 (latest) when the request is at or past the latest transaction, |
| 405 | // which litestream would otherwise refuse as not available. |
| 406 | Target uint64 |
| 407 | } |
| 408 | |
| 409 | // PlanRestore plans a restore to target (0: latest) through c and reads the |
| 410 | // header of the last planned file with a ranged open of its first |
| 411 | // ltx.HeaderSize bytes. |
| 412 | func PlanRestore(ctx context.Context, c *Client, target uint64) (RestorePlan, error) { |
| 413 | if target != 0 { |
| 414 | latest, err := PlanRestore(ctx, c, 0) |
| 415 | if err != nil { |
| 416 | return RestorePlan{}, err |
| 417 | } |
| 418 | if target >= latest.TXID { |
| 419 | return latest, nil // at or past the end: the latest copy |
| 420 | } |
| 421 | } |
| 422 | infos, err := litestream.CalcRestorePlan(ctx, c, ltx.TXID(target), time.Time{}, slog.Default()) |
| 423 | switch { |
| 424 | case errors.Is(err, litestream.ErrTxNotAvailable) || errors.Is(err, litestream.ErrNoSnapshots): |
| 425 | if target != 0 { |
| 426 | return RestorePlan{}, ErrHistoryGone |
| 427 | } |
| 428 | return RestorePlan{}, ErrNothingReplicated |
| 429 | case err != nil: |
| 430 | return RestorePlan{}, err |
| 431 | case len(infos) == 0: |
| 432 | return RestorePlan{}, ErrNothingReplicated |
| 433 | } |
| 434 | p := RestorePlan{Target: target} |
| 435 | for _, f := range infos { |
| 436 | p.LTXBytes += f.Size |
| 437 | } |
| 438 | p.Files = len(infos) |
| 439 | last := infos[len(infos)-1] |
| 440 | p.TXID = uint64(last.MaxTXID) |
| 441 | rc, err := c.OpenLTXFile(ctx, last.Level, last.MinTXID, last.MaxTXID, 0, ltx.HeaderSize) |
| 442 | if err != nil { |
| 443 | return RestorePlan{}, fmt.Errorf("dbdoor: read the last planned file's header: %w", err) |
| 444 | } |
| 445 | defer rc.Close() |
| 446 | buf := make([]byte, ltx.HeaderSize) |
| 447 | if _, err := io.ReadFull(rc, buf); err != nil { |
| 448 | return RestorePlan{}, fmt.Errorf("dbdoor: read the last planned file's header: %w", err) |
| 449 | } |
| 450 | var h ltx.Header |
| 451 | if err := h.UnmarshalBinary(buf); err != nil { |
| 452 | return RestorePlan{}, fmt.Errorf("dbdoor: the last planned file's header: %w", err) |
| 453 | } |
| 454 | p.RestoredBytes = int64(h.Commit) * int64(h.PageSize) |
| 455 | return p, nil |
| 456 | } |
| 457 | ``` |
| 458 | |
| 459 | - [ ] **Step 5: Run the tests to verify they pass** |
| 460 | |
| 461 | Run: `go test ./internal/dbdoor/ -count=1` |
| 462 | Expected: PASS: the new tests and every existing `RestoreTo` test. |
| 463 | |
| 464 | - [ ] **Step 6: Commit** |
| 465 | |
| 466 | ```bash |
| 467 | git add internal/dbdoor/ |
| 468 | git commit -m "dbdoor: restore to a TXID and plan a restore's exact size (Plan 5 Task 2) |
| 469 | |
| 470 | Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>" |
| 471 | ``` |
| 472 | |
| 473 | ### Task 3: The scratch reservation pool and the shared limits |
| 474 | |
| 475 | **Files:** |
| 476 | - Create: `internal/console/seedscratch.go`, `internal/console/seedscratch_unix.go` (`//go:build !windows`), `internal/console/seedscratch_windows.go` (`//go:build windows`), `internal/console/seedscratch_test.go` |
| 477 | - Modify: `internal/console/server.go` (`Config.SeedScratchDir`, `Server.seedScratch`, `NewServer` clears the directory and builds the pool) |
| 478 | |
| 479 | **Interfaces:** |
| 480 | - Produces: `type seedScratch struct{…}`; `newSeedScratch(dir string, free func(string) (int64, error)) *seedScratch`; `(p *seedScratch) reserve(appKey string, bytes int64) (release func(), ref *doorRefusal)`; `(p *seedScratch) dir() string`; `clearSeedScratch(dir string) error`; `scratchFree(dir string) (int64, error)` (unix: `syscall.Statfs`, `Bavail * Bsize`; windows: an error); `Config.SeedScratchDir string`. |
| 481 | |
| 482 | The pool (spec §4 step 5): capacity is the scratch filesystem's free space less 1 GiB, read at start and re-read whenever nothing is reserved; a reservation that does not fit is 507 (SD-N27); at most one reservation per app (429, SD-N28) and two across the deployment (429, SD-N29); every reservation is released exactly once by the returned func. |
| 483 | |
| 484 | - [ ] **Step 1: Write the failing tests** |
| 485 | |
| 486 | ```go |
| 487 | package console |
| 488 | |
| 489 | import ( |
| 490 | "net/http" |
| 491 | "os" |
| 492 | "path/filepath" |
| 493 | "testing" |
| 494 | ) |
| 495 | |
| 496 | func TestScratchLimitsPerAppAndDeployment(t *testing.T) { |
| 497 | p := newSeedScratch(t.TempDir(), func(string) (int64, error) { return 100 << 30, nil }) |
| 498 | rel1, ref := p.reserve("bab/titogo", 1<<30) |
| 499 | if ref != nil { |
| 500 | t.Fatalf("first: %+v", ref) |
| 501 | } |
| 502 | if _, ref := p.reserve("bab/titogo", 1<<30); ref == nil || ref.Status != http.StatusTooManyRequests { |
| 503 | t.Fatalf("second for the same app: %+v, want 429", ref) |
| 504 | } |
| 505 | rel2, ref := p.reserve("bab/kass", 1<<30) |
| 506 | if ref != nil { |
| 507 | t.Fatalf("another app: %+v", ref) |
| 508 | } |
| 509 | if _, ref := p.reserve("bab/blog", 1<<30); ref == nil || ref.Status != http.StatusTooManyRequests { |
| 510 | t.Fatalf("a third across the deployment: %+v, want 429", ref) |
| 511 | } |
| 512 | rel1() |
| 513 | rel1() // releasing twice is harmless |
| 514 | if rel3, ref := p.reserve("bab/blog", 1<<30); ref != nil { |
| 515 | t.Fatalf("after a release: %+v", ref) |
| 516 | } else { |
| 517 | rel3() |
| 518 | } |
| 519 | rel2() |
| 520 | } |
| 521 | |
| 522 | func TestScratchCapacityIsFreeSpaceLessAGiB(t *testing.T) { |
| 523 | free := int64(5 << 30) |
| 524 | reads := 0 |
| 525 | p := newSeedScratch(t.TempDir(), func(string) (int64, error) { reads++; return free, nil }) |
| 526 | if _, ref := p.reserve("a/x", 4<<30+1); ref == nil || ref.Status != http.StatusInsufficientStorage { |
| 527 | t.Fatalf("over capacity: %+v, want 507", ref) |
| 528 | } |
| 529 | rel, ref := p.reserve("a/x", 4<<30) |
| 530 | if ref != nil { |
| 531 | t.Fatalf("at capacity: %+v", ref) |
| 532 | } |
| 533 | if _, ref := p.reserve("a/y", 1); ref == nil || ref.Status != http.StatusInsufficientStorage { |
| 534 | t.Fatalf("a full pool: %+v, want 507", ref) |
| 535 | } |
| 536 | rel() |
| 537 | free = 2 << 30 // the disk filled from elsewhere; an empty pool re-reads it |
| 538 | if _, ref := p.reserve("a/x", 2<<30); ref == nil || ref.Status != http.StatusInsufficientStorage { |
| 539 | t.Fatalf("after a re-read: %+v, want 507", ref) |
| 540 | } |
| 541 | if reads < 2 { |
| 542 | t.Fatalf("free space read %d times, want a re-read when the pool emptied", reads) |
| 543 | } |
| 544 | } |
| 545 | |
| 546 | func TestClearSeedScratch(t *testing.T) { |
| 547 | dir := filepath.Join(t.TempDir(), "scratch", "seeds") |
| 548 | os.MkdirAll(filepath.Join(dir, "from-123"), 0o700) |
| 549 | os.WriteFile(filepath.Join(dir, "canary.go.tito.io.db"), []byte("spool"), 0o600) |
| 550 | if err := clearSeedScratch(dir); err != nil { |
| 551 | t.Fatal(err) |
| 552 | } |
| 553 | ents, err := os.ReadDir(dir) |
| 554 | if err != nil || len(ents) != 0 { |
| 555 | t.Fatalf("after clear: %v, %v", ents, err) |
| 556 | } |
| 557 | st, _ := os.Stat(dir) |
| 558 | if st.Mode().Perm() != 0o700 { |
| 559 | t.Fatalf("mode %v, want 0700", st.Mode().Perm()) |
| 560 | } |
| 561 | } |
| 562 | ``` |
| 563 | |
| 564 | - [ ] **Step 2: Run the tests to verify they fail** |
| 565 | |
| 566 | Run: `go test ./internal/console/ -run 'TestScratch|TestClearSeedScratch' -count=1` |
| 567 | Expected: FAIL to compile. |
| 568 | |
| 569 | - [ ] **Step 3: Write the pool** |
| 570 | |
| 571 | `internal/console/seedscratch.go`: |
| 572 | |
| 573 | ```go |
| 574 | package console |
| 575 | |
| 576 | import ( |
| 577 | "fmt" |
| 578 | "net/http" |
| 579 | "os" |
| 580 | "sync" |
| 581 | ) |
| 582 | |
| 583 | // seedScratch is the console's one scratch reservation pool, shared by |
| 584 | // --from restores and import spools (seeded-instances spec §4 step 5, |
| 585 | // §5.2): an in-process counter over <console data>/scratch/seeds/, and the |
| 586 | // per-app and deployment limits on how many run at once. |
| 587 | type seedScratch struct { |
| 588 | root string |
| 589 | free func(string) (int64, error) |
| 590 | |
| 591 | mu sync.Mutex |
| 592 | capacity int64 |
| 593 | used int64 |
| 594 | running int |
| 595 | perApp map[string]bool |
| 596 | } |
| 597 | |
| 598 | const ( |
| 599 | seedScratchSlack = 1 << 30 // capacity is free space less this |
| 600 | seedScratchPerApp = 1 |
| 601 | seedScratchAllApps = 2 |
| 602 | ) |
| 603 | |
| 604 | func newSeedScratch(root string, free func(string) (int64, error)) *seedScratch { |
| 605 | p := &seedScratch{root: root, free: free, perApp: map[string]bool{}, capacity: -1} |
| 606 | p.readCapacity() |
| 607 | return p |
| 608 | } |
| 609 | |
| 610 | func (p *seedScratch) dir() string { return p.root } |
| 611 | |
| 612 | // readCapacity is called with p.mu held, or before the pool is shared. |
| 613 | func (p *seedScratch) readCapacity() { |
| 614 | n, err := p.free(p.root) |
| 615 | if err != nil { |
| 616 | p.capacity = 0 |
| 617 | return |
| 618 | } |
| 619 | p.capacity = max(n-seedScratchSlack, 0) |
| 620 | } |
| 621 | |
| 622 | // reserve takes bytes for appKey (account/app), or answers why not. The |
| 623 | // returned release is safe to call more than once. |
| 624 | func (p *seedScratch) reserve(appKey string, bytes int64) (func(), *doorRefusal) { |
| 625 | p.mu.Lock() |
| 626 | defer p.mu.Unlock() |
| 627 | if p.perApp[appKey] { |
| 628 | app := appKey |
| 629 | if i := lastSlash(appKey); i >= 0 { |
| 630 | app = appKey[i+1:] |
| 631 | } |
| 632 | return nil, &doorRefusal{http.StatusTooManyRequests, fmt.Sprintf("another copy or upload is running for %s. Try again when it finishes.", app)} |
| 633 | } |
| 634 | if p.running >= seedScratchAllApps { |
| 635 | return nil, &doorRefusal{http.StatusTooManyRequests, "the console is already running two copies or uploads. Try again in a few minutes."} |
| 636 | } |
| 637 | if p.used == 0 { |
| 638 | p.readCapacity() |
| 639 | } |
| 640 | if bytes > p.capacity-p.used { |
| 641 | return nil, &doorRefusal{http.StatusInsufficientStorage, "the console has no room for this right now. Try again later."} |
| 642 | } |
| 643 | p.used += bytes |
| 644 | p.running++ |
| 645 | p.perApp[appKey] = true |
| 646 | var once sync.Once |
| 647 | return func() { |
| 648 | once.Do(func() { |
| 649 | p.mu.Lock() |
| 650 | defer p.mu.Unlock() |
| 651 | p.used -= bytes |
| 652 | p.running-- |
| 653 | delete(p.perApp, appKey) |
| 654 | }) |
| 655 | }, nil |
| 656 | } |
| 657 | |
| 658 | func lastSlash(s string) int { |
| 659 | for i := len(s) - 1; i >= 0; i-- { |
| 660 | if s[i] == '/' { |
| 661 | return i |
| 662 | } |
| 663 | } |
| 664 | return -1 |
| 665 | } |
| 666 | |
| 667 | // clearSeedScratch empties the scratch directory at console start (spec §4 |
| 668 | // "Cleanup", §5.2): whatever a crashed restore or spool left is gone before |
| 669 | // the pool counts free space. |
| 670 | func clearSeedScratch(dir string) error { |
| 671 | if dir == "" { |
| 672 | return nil |
| 673 | } |
| 674 | if err := os.RemoveAll(dir); err != nil { |
| 675 | return err |
| 676 | } |
| 677 | if err := os.MkdirAll(dir, 0o700); err != nil { |
| 678 | return err |
| 679 | } |
| 680 | return os.Chmod(dir, 0o700) |
| 681 | } |
| 682 | ``` |
| 683 | |
| 684 | Use `strings.LastIndex` instead of `lastSlash` if you prefer; the deck's `<app>` is the app name alone. |
| 685 | |
| 686 | `internal/console/seedscratch_unix.go`: |
| 687 | |
| 688 | ```go |
| 689 | //go:build !windows |
| 690 | |
| 691 | package console |
| 692 | |
| 693 | import "syscall" |
| 694 | |
| 695 | // scratchFree is the bytes available to the console on dir's filesystem. |
| 696 | // Bavail times Bsize, multiplied once here (Bsize is int64 on Linux and |
| 697 | // uint32 on darwin, internal/host/statfs_unix.go's note). |
| 698 | func scratchFree(dir string) (int64, error) { |
| 699 | var st syscall.Statfs_t |
| 700 | if err := syscall.Statfs(dir, &st); err != nil { |
| 701 | return 0, err |
| 702 | } |
| 703 | return int64(uint64(st.Bavail) * uint64(st.Bsize)), nil |
| 704 | } |
| 705 | ``` |
| 706 | |
| 707 | `internal/console/seedscratch_windows.go`: |
| 708 | |
| 709 | ```go |
| 710 | //go:build windows |
| 711 | |
| 712 | package console |
| 713 | |
| 714 | import "errors" |
| 715 | |
| 716 | // scratchFree: the console does not run on Windows; the pool then has no |
| 717 | // capacity and every seed door answers 507. |
| 718 | func scratchFree(string) (int64, error) { return 0, errors.New("no free-space reading on windows") } |
| 719 | ``` |
| 720 | |
| 721 | In `server.go`: add `SeedScratchDir string` to `Config` (doc: "<console data>/scratch/seeds: --from restores and import spools; cleared at start. Empty: the seed doors answer 507."), a `seedScratch *seedScratch` field on `Server`, and in `NewServer`, where it builds its other per-process state: |
| 722 | |
| 723 | ```go |
| 724 | if cfg.SeedScratchDir != "" { |
| 725 | if err := clearSeedScratch(cfg.SeedScratchDir); err != nil { |
| 726 | log.Printf("console: seed scratch %s: %v (the seed doors will refuse)", cfg.SeedScratchDir, err) |
| 727 | } |
| 728 | } |
| 729 | s.seedScratch = newSeedScratch(cfg.SeedScratchDir, func(dir string) (int64, error) { |
| 730 | if dir == "" { |
| 731 | return 0, errors.New("no seed scratch directory") |
| 732 | } |
| 733 | return scratchFree(dir) |
| 734 | }) |
| 735 | ``` |
| 736 | |
| 737 | - [ ] **Step 4: Run the tests to verify they pass** |
| 738 | |
| 739 | Run: `go test ./internal/console/ -run 'TestScratch|TestClearSeedScratch' -count=1 && GOOS=windows go build ./internal/console/ && GOOS=darwin go build ./internal/console/` |
| 740 | Expected: PASS and clean builds. |
| 741 | |
| 742 | - [ ] **Step 5: Wire SD-N27 to SD-N29 and commit** |
| 743 | |
| 744 | Append `"SD-N27", "SD-N28", "SD-N29"`; run `go test . -run TestSeededWiredCopyIsApplied -count=1`. |
| 745 | |
| 746 | ```bash |
| 747 | git add internal/console/seedscratch*.go internal/console/server.go approved_seeded_copy_test.go |
| 748 | git commit -m "console: the seed scratch reservation pool and its limits (Plan 5 Task 3) |
| 749 | |
| 750 | Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>" |
| 751 | ``` |
| 752 | |
| 753 | ### Task 4: The console's own door, and the store marked with the seed |
| 754 | |
| 755 | **Files:** |
| 756 | - Create: `internal/console/seeddoer.go`, `internal/console/seeddoer_test.go` |
| 757 | - Modify: `internal/objstore/objstore.go` (`Declared`), `internal/console/instanceprivatestore.go:176` (`instanceStoreProvision` becomes a wrapper) |
| 758 | - Test: `internal/console/instanceprivatestore_test.go` |
| 759 | |
| 760 | **Interfaces:** |
| 761 | - Consumes: `replicaReader`, `parseDBLevel`, `dbdoor.ListResponse`, `dbdoor.ParseSpan`, `dbdoor.ParseRange`. |
| 762 | - Produces: `replicaDoer(rd replicaReader) dbdoor.Doer`; `objstore.Declared.Seed string` (`json:"seed,omitempty"`); `(s *Server) instanceStoreProvisionFor(ctx, w, acct *Account, app, env, actor, seedID string) (*objstore.Record, bool)`; `seedOwnsStore(rec *objstore.Record, seedID string) bool`. |
| 763 | |
| 764 | The seed `ID` goes on the store record's `Declared` block, which the console itself writes before the grant walk (`instanceprivatestore.go:237-249`), rather than on `GrantBlock`, which `objstore.GrantIn`'s shared walk writes for every grant path. A store "exists" when its record has a `Grant` or `Provisioned` block. A seeded create may use an existing store only when `Declared.Seed` is this seed's ID; a declared-only record (no estate) is re-declared for this seed. |
| 765 | |
| 766 | - [ ] **Step 1: Write the failing tests** |
| 767 | |
| 768 | `internal/console/seeddoer_test.go`: |
| 769 | |
| 770 | ```go |
| 771 | package console |
| 772 | |
| 773 | import ( |
| 774 | "context" |
| 775 | "os" |
| 776 | "path/filepath" |
| 777 | "testing" |
| 778 | |
| 779 | lsfile "github.com/benbjohnson/litestream/file" |
| 780 | |
| 781 | "github.com/carlosframework/platform/internal/dbdoor" |
| 782 | "github.com/carlosframework/platform/internal/dbdoor/dbdoortest" |
| 783 | ) |
| 784 | |
| 785 | func TestReplicaDoerRestoresLikeTheDoor(t *testing.T) { |
| 786 | root := t.TempDir() |
| 787 | dbdoortest.Replicate(t, root, 25) |
| 788 | c := dbdoor.NewClient(replicaDoer(lsfile.NewReplicaClient(root))) |
| 789 | plan, err := dbdoor.PlanRestore(context.Background(), c, 0) |
| 790 | if err != nil { |
| 791 | t.Fatal(err) |
| 792 | } |
| 793 | r, err := dbdoor.RestoreToTarget(context.Background(), c, t.TempDir(), 0) |
| 794 | if err != nil { |
| 795 | t.Fatal(err) |
| 796 | } |
| 797 | defer os.RemoveAll(r.Dir) |
| 798 | st, _ := os.Stat(r.Path) |
| 799 | if st.Size() != plan.RestoredBytes || r.TXID != plan.TXID { |
| 800 | t.Fatalf("restored %d at %d; plan %+v", st.Size(), r.TXID, plan) |
| 801 | } |
| 802 | _ = filepath.Join |
| 803 | } |
| 804 | ``` |
| 805 | |
| 806 | (`dbdoortest.Replicate(t testing.TB, dir string, batches int) string`, `dbdoortest/replica.go:31`.) |
| 807 | |
| 808 | In `instanceprivatestore_test.go`, add: |
| 809 | |
| 810 | ```go |
| 811 | func TestSeedOwnsStore(t *testing.T) { |
| 812 | id := "sd_aaaaaaaaaaaaaaaaaaaaaaaaaa" |
| 813 | for _, c := range []struct { |
| 814 | rec *objstore.Record |
| 815 | want bool |
| 816 | }{ |
| 817 | {nil, true}, // no store: this seed may make one |
| 818 | {&objstore.Record{Declared: &objstore.Declared{}}, true}, // declared only: no estate yet |
| 819 | {&objstore.Record{Declared: &objstore.Declared{Seed: id}, Grant: &objstore.GrantBlock{}}, true}, |
| 820 | {&objstore.Record{Declared: &objstore.Declared{Seed: "sd_bbbbbbbbbbbbbbbbbbbbbbbbbb"}, Grant: &objstore.GrantBlock{}}, false}, |
| 821 | {&objstore.Record{Declared: &objstore.Declared{}, Provisioned: &objstore.Provisioned{}}, false}, |
| 822 | } { |
| 823 | if got := seedOwnsStore(c.rec, id); got != c.want { |
| 824 | t.Errorf("seedOwnsStore(%+v) = %v, want %v", c.rec, got, c.want) |
| 825 | } |
| 826 | } |
| 827 | } |
| 828 | ``` |
| 829 | |
| 830 | - [ ] **Step 2: Run the tests to verify they fail** |
| 831 | |
| 832 | Run: `go test ./internal/console/ -run 'TestReplicaDoerRestoresLikeTheDoor|TestSeedOwnsStore' -count=1` |
| 833 | Expected: FAIL to compile. |
| 834 | |
| 835 | - [ ] **Step 3: Write `seeddoer.go`** |
| 836 | |
| 837 | ```go |
| 838 | package console |
| 839 | |
| 840 | import ( |
| 841 | "bytes" |
| 842 | "context" |
| 843 | "encoding/json" |
| 844 | "errors" |
| 845 | "io" |
| 846 | "net/http" |
| 847 | "os" |
| 848 | "sort" |
| 849 | "strings" |
| 850 | |
| 851 | "github.com/superfly/ltx" |
| 852 | |
| 853 | "github.com/carlosframework/platform/internal/dbdoor" |
| 854 | ) |
| 855 | |
| 856 | // replicaDoer serves the db door's two tails (ListTail, GetTail) from rd in |
| 857 | // process, so the console's own --from restore runs through |
| 858 | // dbdoor.RestoreToTarget: its retry, its private 0700 directory, its |
| 859 | // CheckDB, the same code carlos db pull runs (seeded-instances spec §4 |
| 860 | // step 6). |
| 861 | func replicaDoer(rd replicaReader) dbdoor.Doer { |
| 862 | return func(ctx context.Context, tail string, h http.Header) (*http.Response, error) { |
| 863 | rest, ok := strings.CutPrefix(tail, "/ltx/") |
| 864 | if !ok { |
| 865 | return status(http.StatusNotFound, nil), nil |
| 866 | } |
| 867 | parts := strings.Split(rest, "/") |
| 868 | level, ok := parseDBLevel(parts[0]) |
| 869 | if !ok || len(parts) > 2 { |
| 870 | return status(http.StatusNotFound, nil), nil |
| 871 | } |
| 872 | if len(parts) == 1 { |
| 873 | itr, err := rd.LTXFiles(ctx, level, 0, false) |
| 874 | if err != nil { |
| 875 | return nil, err |
| 876 | } |
| 877 | out := dbdoor.ListResponse{Files: []dbdoor.LTXFile{}} |
| 878 | for itr.Next() { |
| 879 | f := itr.Item() |
| 880 | out.Files = append(out.Files, dbdoor.LTXFile{Level: level, MinTXID: uint64(f.MinTXID), MaxTXID: uint64(f.MaxTXID), Size: f.Size, CreatedAt: f.CreatedAt}) |
| 881 | } |
| 882 | err = errors.Join(itr.Err(), itr.Close()) |
| 883 | if err != nil { |
| 884 | return nil, err |
| 885 | } |
| 886 | sort.Slice(out.Files, func(i, j int) bool { return out.Files[i].MinTXID < out.Files[j].MinTXID }) |
| 887 | body, _ := json.Marshal(out) |
| 888 | return status(http.StatusOK, io.NopCloser(bytes.NewReader(body))), nil |
| 889 | } |
| 890 | lo, hi, err := dbdoor.ParseSpan(parts[1]) |
| 891 | if err != nil { |
| 892 | return status(http.StatusNotFound, nil), nil |
| 893 | } |
| 894 | off, size, err := dbdoor.ParseRange(h.Get("Range")) |
| 895 | if err != nil { |
| 896 | return status(http.StatusRequestedRangeNotSatisfiable, nil), nil |
| 897 | } |
| 898 | rc, err := rd.OpenLTXFile(ctx, level, ltx.TXID(lo), ltx.TXID(hi), off, size) |
| 899 | if errors.Is(err, os.ErrNotExist) { |
| 900 | return status(http.StatusNotFound, nil), nil |
| 901 | } |
| 902 | if err != nil { |
| 903 | return nil, err |
| 904 | } |
| 905 | code := http.StatusOK |
| 906 | if h.Get("Range") != "" { |
| 907 | code = http.StatusPartialContent |
| 908 | } |
| 909 | return status(code, rc), nil |
| 910 | } |
| 911 | } |
| 912 | |
| 913 | func status(code int, body io.ReadCloser) *http.Response { |
| 914 | if body == nil { |
| 915 | body = io.NopCloser(strings.NewReader("")) |
| 916 | } |
| 917 | return &http.Response{StatusCode: code, Header: http.Header{}, Body: body} |
| 918 | } |
| 919 | ``` |
| 920 | |
| 921 | - [ ] **Step 4: Mark the store with the seed** |
| 922 | |
| 923 | In `internal/objstore/objstore.go`, `Declared` gains: |
| 924 | |
| 925 | ```go |
| 926 | // Seed is the seed id that created this store (seeded-instances spec §4 |
| 927 | // Storage, codex R5-1, R6-1): only a retry of that seed may use it, and |
| 928 | // only that seed's abort may tear it down. Empty for every other store. |
| 929 | Seed string `json:"seed,omitempty"` |
| 930 | ``` |
| 931 | |
| 932 | In `instanceprivatestore.go`, rename `instanceStoreProvision`'s body to `instanceStoreProvisionFor(ctx, w, acct, app, env, actor, seedID string)` and keep `instanceStoreProvision(ctx, w, acct, app, env, actor)` as `return s.instanceStoreProvisionFor(ctx, w, acct, app, env, actor, "")`. Inside, replace the `if rec.Declared == nil { … }` block with: |
| 933 | |
| 934 | ```go |
| 935 | redeclare := rec.Declared == nil |
| 936 | if seedID != "" && rec.Declared != nil && rec.Grant == nil && rec.Provisioned == nil && rec.Declared.Seed != seedID { |
| 937 | redeclare = true // a declared-only record holds no estate: this seed takes it |
| 938 | } |
| 939 | if redeclare { |
| 940 | rec.Declared = &objstore.Declared{EnvPrefix: prefix, At: time.Now().UTC(), By: actor, Seed: seedID} |
| 941 | // ... the existing PutIn and its two error answers, unchanged |
| 942 | } |
| 943 | ``` |
| 944 | |
| 945 | and add, beside it: |
| 946 | |
| 947 | ```go |
| 948 | // seedOwnsStore reports whether a seeded create may use rec (spec §4 |
| 949 | // Storage): there is no store, the record holds no estate yet, or the |
| 950 | // estate was made by this very seed. |
| 951 | func seedOwnsStore(rec *objstore.Record, seedID string) bool { |
| 952 | if rec == nil || (rec.Grant == nil && rec.Provisioned == nil) { |
| 953 | return true |
| 954 | } |
| 955 | return rec.Declared != nil && rec.Declared.Seed == seedID |
| 956 | } |
| 957 | ``` |
| 958 | |
| 959 | The refusal sentence for a store that is not this seed's (SD-N34) is spoken by `createFrom` (Task 5), which checks `seedOwnsStore` before it reserves the record. |
| 960 | |
| 961 | - [ ] **Step 5: Run the tests to verify they pass** |
| 962 | |
| 963 | Run: `go test ./internal/console/ -run 'TestReplicaDoer|TestSeedOwnsStore|TestInstanceStore|TestPrivateStore' -count=1 && go test ./internal/objstore/ -count=1` |
| 964 | Expected: PASS. |
| 965 | |
| 966 | - [ ] **Step 6: Commit** |
| 967 | |
| 968 | ```bash |
| 969 | git add internal/console/seeddoer.go internal/console/seeddoer_test.go internal/console/instanceprivatestore.go internal/console/instanceprivatestore_test.go internal/objstore/objstore.go |
| 970 | git commit -m "console: the in-process replica door and stores marked with their seed (Plan 5 Task 4) |
| 971 | |
| 972 | Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>" |
| 973 | ``` |
| 974 | |
| 975 | ### Task 5: `createFrom` |
| 976 | |
| 977 | **Files:** |
| 978 | - Create: `internal/console/apiinstanceseed.go`, `internal/console/apiinstanceseed_test.go` |
| 979 | - Modify: `internal/console/apiinstances.go` (the request struct `:655-699`; the explicit-environment refusal before the `m.PrivateStorage` block `:1096`; the hand-off after `leftoverReplicaRefusal` `:1169`) |
| 980 | - Modify: `internal/console/server.go` (`Config.SeedData release.Store`), `cmd/console/main.go` (`SeedData`, `SeedScratchDir`), `internal/console/testdata/claim-authority-sites.txt`, `internal/console/claim_authority_census_test.go` (`"NewS3StoreIn": true`) |
| 981 | |
| 982 | **Interfaces:** |
| 983 | - Consumes: Tasks 1 to 4; `seedRegionReady(ctx, region, false)` (Plan 3); `provision.CreateRecord`, `SetRecordSeed`, `SeedPublishReplica`, `SeedAbort`; `seeds.NewID`, `seeds.SnapshotKey`, `seeds.MaxBytes`; `release.PutObjectFrom`; `objstore.GetIn`; `storeBucketName`; `s.dbReplica`; `s.auditActivity`; `s.bumpSerial`; `recordView`. |
| 984 | - Produces: the create body's `"from": {"host": "<source>", "txid": N}`; `type seedFromReq struct{ Host string; TXID uint64 }`; `type seedCreate struct{ acct *Account; app, principal, message string; m *provision.Manifest; rec provision.Record; placed, regionFrom string }`; `(s *Server) createFrom(w, r, c seedCreate, from seedFromReq)`; `seedStoreRefusal(ctx, c seedCreate, seedID string) *doorRefusal` (Plan 6 reuses it); `seedCommonRefusal(req …) *doorRefusal` (kind, addrs, role); `restoreWatch` (the size watcher); `Config.SeedData release.Store`. |
| 985 | |
| 986 | The door's steps, in order (spec §4): |
| 987 | |
| 988 | 1. In `handleAPIInstanceCreate`: the body's `from` (and Plan 6's `import`) are seed doors. Before the shape fields are used, `seedCommonRefusal` refuses `kind` other than `""`/`instance`, any `addrs`, any `role` or `replicas` (400, SD-N36). Before the `m.PrivateStorage` block, a seed door on an app with private storage and a non-empty `environment` is refused (409, SD-N33). After `leftoverReplicaRefusal` (so every ordinary check has run), the handler calls `createFrom` and returns. |
| 989 | 2. `createFrom`: `placed != ""` or `m.Pool != ""` → 409 SD-N37. `s.cfg.SeedData == nil` or `s.cfg.HibernateBucket == ""` → 503 SD-N38. A `txid` of 0 on the wire means latest; the CLI refuses a non-positive `--at` (SD-N40). |
| 990 | 3. `seedRegionReady(ctx, rec.Region, false)`. |
| 991 | 4. `resolveDBSource`: not found → 409 SD-N20; class → 409 SD-N23; pooled → 409 SD-N21; fleet (residency with a fleet region) → 409 SD-N22; residency → 409 with `s.dbResidencyRefusal(src.rec)` (the existing sentence); claim → 409 SD-N8 (unreachable, `seedRegionReady` refused first); other → 502 SD-N39. |
| 992 | 5. The source's region must equal the new record's: otherwise 409 SD-N35. |
| 993 | 6. `client := dbdoor.NewClient(replicaDoer(rd))`; `plan := dbdoor.PlanRestore(ctx, client, from.TXID)`: `ErrHistoryGone` → 409 SD-N24 (with the txid); `ErrNothingReplicated` → 409 SD-N25; other → 502 SD-N39. |
| 994 | 7. `plan.RestoredBytes > seeds.MaxBytes` → 413 SD-N26. Reserve `plan.RestoredBytes + plan.LTXBytes` from the pool under `acct.Sqid+"/"+app`; refusal as the pool answers; release on every exit (`defer`). |
| 995 | 8. Mint the seed `ID`. With private storage: `objstore.GetIn(ctx, s.cfg.Releases, acct.Sqid, app, rec.Environment)`; `!seedOwnsStore` → 409 SD-N34 naming `rec.Environment` and `storeBucketName`. |
| 996 | 9. Reserve the record: `rec.Seed = &provision.SeedSource{ID, Kind: replica, From: source, State: reserving, By: principal, At: now}`; `provision.CreateRecord`; CAS conflict → 409 with the existing "instance <host> already exists". |
| 997 | 10. From here every failure aborts: `abort := func() { provision.SetRecordSeed(context.WithoutCancel(r.Context()), …, provision.SeedAbort(id)) }`. |
| 998 | 11. With private storage: `s.instanceStoreProvisionFor(ctx, w, acct, app, rec.Environment, principal, id)`; on refusal (it has answered) abort and return. |
| 999 | 12. Restore: a fresh `os.MkdirTemp(s.seedScratch.dir(), "from-")` removed by `defer`; `rctx, cancel := context.WithTimeout(r.Context(), 30*time.Minute)`; `restoreWatch(rctx, cancel, dir, plan.RestoredBytes)` stats `dir/carlos-db-*/replica.db` every second and cancels when it grows past the reservation; `dbdoor.RestoreToTarget(rctx, client, dir, from.TXID)`. Errors: the watcher fired → 413 SD-N32; deadline → 504 SD-N31; `ErrCheckFailed` → 422 SD-N30; the client went away → no answer; `ErrHistoryGone` → 409 SD-N24; other → 502 SD-N39. Each aborts. |
| 1000 | 13. Hash the restored file (sha256, size); `release.PutObjectFrom(ctx, s.cfg.SeedData, seeds.SnapshotKey(host, id), f, size, "")`; failure → abort, 502 SD-N39. |
| 1001 | 14. Publish: `provision.SetRecordSeed(…, provision.SeedPublishReplica(id, strconv.FormatUint(restored.TXID, 10), sha, size))`; failure → abort, 502 SD-N39. |
| 1002 | 15. Audit `instance-seed` (SD-N41, the `Message` the caller gave), `bumpSerial`, 201 with `recordView(published)` and `RegionFrom`/`RegionName` as the ordinary create sets them. |
| 1003 | |
| 1004 | - [ ] **Step 1: Write the failing tests** |
| 1005 | |
| 1006 | `internal/console/apiinstanceseed_test.go` (the fixture builds an app without a pool, the source record, a file replica for it, and a ready home region): |
| 1007 | |
| 1008 | ```go |
| 1009 | package console |
| 1010 | |
| 1011 | import ( |
| 1012 | "context" |
| 1013 | "encoding/json" |
| 1014 | "net/http" |
| 1015 | "os" |
| 1016 | "path/filepath" |
| 1017 | "strings" |
| 1018 | "testing" |
| 1019 | "time" |
| 1020 | |
| 1021 | lsfile "github.com/benbjohnson/litestream/file" |
| 1022 | |
| 1023 | "github.com/carlosframework/platform/internal/dbdoor" |
| 1024 | "github.com/carlosframework/platform/internal/dbdoor/dbdoortest" |
| 1025 | "github.com/carlosframework/platform/internal/provision" |
| 1026 | "github.com/carlosframework/platform/internal/release" |
| 1027 | "github.com/carlosframework/platform/internal/seeds" |
| 1028 | ) |
| 1029 | |
| 1030 | type fromFixture struct { |
| 1031 | fx *consoleFixture |
| 1032 | token string |
| 1033 | data release.Store |
| 1034 | url string |
| 1035 | } |
| 1036 | |
| 1037 | func newFromFixture(t *testing.T, txns int) fromFixture { |
| 1038 | t.Helper() |
| 1039 | fx, token := instanceFixture(t, "code-from") |
| 1040 | m := titoManifest() |
| 1041 | m.Pool = "" |
| 1042 | m.Regions = nil |
| 1043 | enableInstances(t, fx, "titogo", m) |
| 1044 | fx.srv.cfg.HibernateBucket = "data-test" |
| 1045 | fx.srv.cfg.HibernatePrefix = "instances" |
| 1046 | fx.srv.cfg.Region = "" |
| 1047 | data := release.NewDirStore(t.TempDir()) |
| 1048 | fx.srv.cfg.SeedData = data |
| 1049 | scratch := filepath.Join(t.TempDir(), "scratch", "seeds") |
| 1050 | if err := clearSeedScratch(scratch); err != nil { |
| 1051 | t.Fatal(err) |
| 1052 | } |
| 1053 | fx.srv.seedScratch = newSeedScratch(scratch, func(string) (int64, error) { return 64 << 30, nil }) |
| 1054 | root := t.TempDir() |
| 1055 | dbdoortest.Replicate(t, filepath.Join(root, "instances", "acme.go.tito.io"), txns) |
| 1056 | fx.srv.dbReplica = func(_ context.Context, prefix string) (replicaReader, error) { |
| 1057 | return lsfile.NewReplicaClient(filepath.Join(root, prefix)), nil |
| 1058 | } |
| 1059 | if err := provision.CreateRecord(context.Background(), fx.rel, provision.Record{Schema: 1, Host: "acme.go.tito.io", |
| 1060 | Slug: "acme", App: "titogo", Account: "bab", Channel: "stable", CreatedAt: time.Now().UTC()}); err != nil { |
| 1061 | t.Fatal(err) |
| 1062 | } |
| 1063 | publishSeedsFloorFx(t, fx, "v0.27.0") |
| 1064 | seedReadyBox(t, fx, "kass-1", "", 1, "v0.27.0", "v0.27.0", "nested", false) |
| 1065 | return fromFixture{fx: fx, token: token, data: data, url: instancesURL(fx, "bab", "titogo")} |
| 1066 | } |
| 1067 | |
| 1068 | func (f fromFixture) create(t *testing.T, body string) (int, map[string]any) { |
| 1069 | t.Helper() |
| 1070 | code, _, raw := rawReq(t, http.MethodPost, f.url, f.token, body) |
| 1071 | var out map[string]any |
| 1072 | _ = json.Unmarshal([]byte(raw), &out) |
| 1073 | return code, out |
| 1074 | } |
| 1075 | |
| 1076 | func TestFromCreatesAReadySeededInstance(t *testing.T) { |
| 1077 | f := newFromFixture(t, 30) |
| 1078 | code, out := f.create(t, `{"host":"canary.go.tito.io","from":{"host":"acme.go.tito.io"}}`) |
| 1079 | if code != http.StatusCreated { |
| 1080 | t.Fatalf("create = %d %v", code, out) |
| 1081 | } |
| 1082 | rec, err := provision.GetRecord(context.Background(), f.fx.rel, "bab", "titogo", "canary.go.tito.io") |
| 1083 | if err != nil || rec.Seed == nil || rec.Seed.State != seeds.StateReady || rec.Seed.Kind != seeds.KindReplica || |
| 1084 | rec.Seed.From != "acme.go.tito.io" || rec.Seed.TXID == "" || rec.Seed.Bytes == 0 || len(rec.Seed.SHA256) != 64 { |
| 1085 | t.Fatalf("record seed = %+v, %v", rec.Seed, err) |
| 1086 | } |
| 1087 | if _, _, err := f.data.Get(context.Background(), seeds.SnapshotKey(rec.Host, rec.Seed.ID)); err != nil { |
| 1088 | t.Fatalf("snapshot: %v", err) |
| 1089 | } |
| 1090 | if ents, _ := os.ReadDir(f.fx.srv.seedScratch.dir()); len(ents) != 0 { |
| 1091 | t.Fatalf("scratch left behind: %v", ents) |
| 1092 | } |
| 1093 | if rel, ref := f.fx.srv.seedScratch.reserve("bab/titogo", 1); ref != nil { |
| 1094 | t.Fatalf("the reservation was not released: %+v", ref) |
| 1095 | } else { |
| 1096 | rel() |
| 1097 | } |
| 1098 | } |
| 1099 | |
| 1100 | func TestFromRefusals(t *testing.T) { |
| 1101 | f := newFromFixture(t, 30) |
| 1102 | for _, c := range []struct { |
| 1103 | body string |
| 1104 | code int |
| 1105 | want string |
| 1106 | }{ |
| 1107 | {`{"host":"c1.go.tito.io","from":{"host":"nope.go.tito.io"}}`, http.StatusConflict, "no instance nope.go.tito.io in titogo"}, |
| 1108 | {`{"host":"c2.go.tito.io","kind":"static","from":{"host":"acme.go.tito.io"}}`, http.StatusBadRequest, "ordinary instance"}, |
| 1109 | {`{"host":"c3.go.tito.io","from":{"host":"acme.go.tito.io","txid":1}}`, http.StatusCreated, ""}, |
| 1110 | {`{"host":"c4.go.tito.io","from":{"host":"acme.go.tito.io","txid":999999}}`, http.StatusCreated, ""}, // past the end: the latest copy (RestorePlan.Target 0) |
| 1111 | } { |
| 1112 | code, out := f.create(t, c.body) |
| 1113 | if code != c.code || (c.want != "" && !strings.Contains(str(t, out, "error"), c.want)) { |
| 1114 | t.Errorf("%s = %d %v, want %d %q", c.body, code, out, c.code, c.want) |
| 1115 | } |
| 1116 | } |
| 1117 | } |
| 1118 | |
| 1119 | func TestFromRefusedBeforeTheFloor(t *testing.T) { |
| 1120 | f := newFromFixture(t, 10) |
| 1121 | seedReadyBox(t, f.fx, "kass-1", "", 0, "v0.27.0", "v0.27.0", "nested", false) |
| 1122 | code, out := f.create(t, `{"host":"canary.go.tito.io","from":{"host":"acme.go.tito.io"}}`) |
| 1123 | if code != http.StatusConflict || !strings.Contains(str(t, out, "error"), "below the seeds floor") { |
| 1124 | t.Fatalf("= %d %v", code, out) |
| 1125 | } |
| 1126 | if _, err := provision.GetRecord(context.Background(), f.fx.rel, "bab", "titogo", "canary.go.tito.io"); err == nil { |
| 1127 | t.Fatal("a record was written before the floor check") |
| 1128 | } |
| 1129 | } |
| 1130 | |
| 1131 | func TestFromTooLarge(t *testing.T) { |
| 1132 | f := newFromFixture(t, 10) |
| 1133 | old := seedMaxBytes |
| 1134 | seedMaxBytes = 1024 // the test's database is larger than 1 KiB |
| 1135 | defer func() { seedMaxBytes = old }() |
| 1136 | code, out := f.create(t, `{"host":"canary.go.tito.io","from":{"host":"acme.go.tito.io"}}`) |
| 1137 | if code != http.StatusRequestEntityTooLarge || !strings.Contains(str(t, out, "error"), "over the 4 GiB limit") { |
| 1138 | t.Fatalf("= %d %v", code, out) |
| 1139 | } |
| 1140 | } |
| 1141 | |
| 1142 | func TestFromWatcherCancelsAnOvergrownRestore(t *testing.T) { |
| 1143 | f := newFromFixture(t, 30) |
| 1144 | oldPlan := planRestore |
| 1145 | planRestore = func(ctx context.Context, c *dbdoor.Client, target uint64) (dbdoor.RestorePlan, error) { |
| 1146 | p, err := oldPlan(ctx, c, target) |
| 1147 | p.RestoredBytes = 1 // reserve far less than the restore will write |
| 1148 | return p, err |
| 1149 | } |
| 1150 | defer func() { planRestore = oldPlan }() |
| 1151 | code, out := f.create(t, `{"host":"canary.go.tito.io","from":{"host":"acme.go.tito.io"}}`) |
| 1152 | if code != http.StatusRequestEntityTooLarge || !strings.Contains(str(t, out, "error"), "grew past its planned size") { |
| 1153 | t.Fatalf("= %d %v", code, out) |
| 1154 | } |
| 1155 | rec, err := provision.GetRecord(context.Background(), f.fx.rel, "bab", "titogo", "canary.go.tito.io") |
| 1156 | if err != nil || rec.Seed == nil || rec.Seed.State != seeds.StateAborted { |
| 1157 | t.Fatalf("record after an overgrown restore = %+v, %v; want aborted", rec, err) |
| 1158 | } |
| 1159 | } |
| 1160 | |
| 1161 | func TestFromDisconnectAbortsTheRecordAndCleansScratch(t *testing.T) { |
| 1162 | f := newFromFixture(t, 30) |
| 1163 | ctx, cancel := context.WithCancel(context.Background()) |
| 1164 | oldRestore := restoreToTarget |
| 1165 | restoreToTarget = func(rctx context.Context, c *dbdoor.Client, parent string, target uint64) (*dbdoor.Restored, error) { |
| 1166 | cancel() // the client goes away mid-restore |
| 1167 | <-rctx.Done() |
| 1168 | return nil, rctx.Err() |
| 1169 | } |
| 1170 | defer func() { restoreToTarget = oldRestore }() |
| 1171 | req, _ := http.NewRequestWithContext(ctx, http.MethodPost, f.url, strings.NewReader(`{"host":"canary.go.tito.io","from":{"host":"acme.go.tito.io"}}`)) |
| 1172 | req.Header.Set("Authorization", "Bearer "+f.token) |
| 1173 | _, _ = http.DefaultClient.Do(req) |
| 1174 | deadline := time.Now().Add(5 * time.Second) |
| 1175 | for time.Now().Before(deadline) { |
| 1176 | rec, err := provision.GetRecord(context.Background(), f.fx.rel, "bab", "titogo", "canary.go.tito.io") |
| 1177 | if err == nil && rec.Seed != nil && rec.Seed.State == seeds.StateAborted { |
| 1178 | if ents, _ := os.ReadDir(f.fx.srv.seedScratch.dir()); len(ents) == 0 { |
| 1179 | return |
| 1180 | } |
| 1181 | } |
| 1182 | time.Sleep(20 * time.Millisecond) |
| 1183 | } |
| 1184 | t.Fatal("a disconnected --from did not abort its record and clear its scratch") |
| 1185 | } |
| 1186 | |
| 1187 | func TestFromPrivateStorageRules(t *testing.T) { |
| 1188 | f := newFromFixture(t, 10) |
| 1189 | m := titoManifest() |
| 1190 | m.Pool = "" |
| 1191 | m.Regions = nil |
| 1192 | m.PrivateStorage = true |
| 1193 | enableInstances(t, f.fx, "titogo", m) |
| 1194 | code, out := f.create(t, `{"host":"canary.go.tito.io","environment":"acme","from":{"host":"acme.go.tito.io"}}`) |
| 1195 | if code != http.StatusConflict || !strings.Contains(str(t, out, "error"), "own environment and an empty store") { |
| 1196 | t.Fatalf("explicit environment = %d %v", code, out) |
| 1197 | } |
| 1198 | // A store already provisioned for the derived environment, by no seed. |
| 1199 | putStoreRecord(t, f.fx, "titogo", "canary", `{"schema":1,"declared":{"env_prefix":"S3","at":"2026-10-08T09:00:00Z","by":"x"},"grant":{"backend":"aws","bucket":"carlos-bab-titogo-canary","source":"account-approval","at":"2026-10-08T09:00:00Z","by":"x"}}`) |
| 1200 | code, out = f.create(t, `{"host":"canary.go.tito.io","from":{"host":"acme.go.tito.io"}}`) |
| 1201 | if code != http.StatusConflict || !strings.Contains(str(t, out, "error"), "already has a store (carlos-bab-titogo-canary)") { |
| 1202 | t.Fatalf("an existing store = %d %v", code, out) |
| 1203 | } |
| 1204 | } |
| 1205 | ``` |
| 1206 | |
| 1207 | `putStoreRecord(t, fx, app, env, body)` writes `apps/bab/<app>/stores/<env>.json` with `fx.rel.Put` (add it to the test file if no such helper exists; the key is `objstore.KeyIn("bab", app, env)`). The seams the tests swap are package vars in `apiinstanceseed.go`: `seedMaxBytes = seeds.MaxBytes`, `planRestore = dbdoor.PlanRestore`, `restoreToTarget = dbdoor.RestoreToTarget`. |
| 1208 | |
| 1209 | - [ ] **Step 2: Run the tests to verify they fail** |
| 1210 | |
| 1211 | Run: `go test ./internal/console/ -run 'TestFrom' -count=1` |
| 1212 | Expected: FAIL to compile (`fx.srv.cfg.SeedData undefined`, `seedMaxBytes undefined`). |
| 1213 | |
| 1214 | - [ ] **Step 3: Write `apiinstanceseed.go`** |
| 1215 | |
| 1216 | ```go |
| 1217 | package console |
| 1218 | |
| 1219 | import ( |
| 1220 | "context" |
| 1221 | "crypto/sha256" |
| 1222 | "encoding/hex" |
| 1223 | "errors" |
| 1224 | "fmt" |
| 1225 | "io" |
| 1226 | "log" |
| 1227 | "net/http" |
| 1228 | "os" |
| 1229 | "path/filepath" |
| 1230 | "strconv" |
| 1231 | "strings" |
| 1232 | "time" |
| 1233 | |
| 1234 | "github.com/carlosframework/platform/internal/dbdoor" |
| 1235 | "github.com/carlosframework/platform/internal/objstore" |
| 1236 | "github.com/carlosframework/platform/internal/provision" |
| 1237 | "github.com/carlosframework/platform/internal/registry" |
| 1238 | "github.com/carlosframework/platform/internal/release" |
| 1239 | "github.com/carlosframework/platform/internal/seeds" |
| 1240 | ) |
| 1241 | |
| 1242 | // Seams for the tests. |
| 1243 | var ( |
| 1244 | seedMaxBytes = seeds.MaxBytes |
| 1245 | planRestore = dbdoor.PlanRestore |
| 1246 | restoreToTarget = dbdoor.RestoreToTarget |
| 1247 | ) |
| 1248 | |
| 1249 | const seedRestoreDeadline = 30 * time.Minute |
| 1250 | |
| 1251 | // seedFromReq is the create body's "from" (seeded-instances spec §4). |
| 1252 | type seedFromReq struct { |
| 1253 | Host string `json:"host"` |
| 1254 | TXID uint64 `json:"txid,omitempty"` |
| 1255 | } |
| 1256 | |
| 1257 | // seedCreate is what handleAPIInstanceCreate has decided by the time it |
| 1258 | // hands a seed door over: every ordinary check has passed. |
| 1259 | type seedCreate struct { |
| 1260 | acct *Account |
| 1261 | app, principal, message string |
| 1262 | m *provision.Manifest |
| 1263 | rec provision.Record |
| 1264 | placed, regionFrom string |
| 1265 | } |
| 1266 | |
| 1267 | // seedCommonRefusal is both seed doors' shape rule: an ordinary instance. |
| 1268 | func seedCommonRefusal(kind string, addrs []string, role string, replicas []string) *doorRefusal { |
| 1269 | if (kind != "" && kind != "instance") || len(addrs) > 0 || role != "" || len(replicas) > 0 { |
| 1270 | return &doorRefusal{http.StatusBadRequest, "a seeded instance is an ordinary instance. Drop --kind, --addr and --role."} |
| 1271 | } |
| 1272 | return nil |
| 1273 | } |
| 1274 | |
| 1275 | // seedPlacementRefusal is both seed doors' placement rule. |
| 1276 | func seedPlacementRefusal(c seedCreate) *doorRefusal { |
| 1277 | if c.placed != "" || c.m.Pool != "" { |
| 1278 | return &doorRefusal{http.StatusConflict, fmt.Sprintf("%s places its instances in a pool or on a fleet, so it cannot take a seeded instance.", c.app)} |
| 1279 | } |
| 1280 | return nil |
| 1281 | } |
| 1282 | |
| 1283 | // seedStoreRefusal is spec §4 Storage's existing-store rule for both doors. |
| 1284 | func (s *Server) seedStoreRefusal(ctx context.Context, c seedCreate, seedID string) *doorRefusal { |
| 1285 | if !c.m.PrivateStorage { |
| 1286 | return nil |
| 1287 | } |
| 1288 | rec, _, err := objstore.GetIn(ctx, s.cfg.Releases, c.acct.Sqid, c.app, c.rec.Environment) |
| 1289 | if err != nil { |
| 1290 | log.Printf("console: seed store check %s/%s/%s: %v", c.acct.Sqid, c.app, c.rec.Environment, err) |
| 1291 | return &doorRefusal{http.StatusBadGateway, fmt.Sprintf("could not copy %s. Nothing was created. Try again.", c.rec.Host)} |
| 1292 | } |
| 1293 | if !seedOwnsStore(rec, seedID) { |
| 1294 | return &doorRefusal{http.StatusConflict, fmt.Sprintf("environment %s already has a store (%s). A seeded instance needs a new, empty one.", c.rec.Environment, storeBucketName(rec))} |
| 1295 | } |
| 1296 | return nil |
| 1297 | } |
| 1298 | |
| 1299 | func gib(n int64) string { return fmt.Sprintf("%.1f GiB", float64(n)/(1<<30)) } |
| 1300 | |
| 1301 | // createFrom is --from (spec §4). |
| 1302 | func (s *Server) createFrom(w http.ResponseWriter, r *http.Request, c seedCreate, from seedFromReq) { |
| 1303 | ctx := r.Context() |
| 1304 | acct, app, host := c.acct, c.app, c.rec.Host |
| 1305 | source := strings.TrimSpace(from.Host) |
| 1306 | if ref := seedPlacementRefusal(c); ref != nil { |
| 1307 | jsonError(w, ref.Status, ref.Msg) |
| 1308 | return |
| 1309 | } |
| 1310 | if s.cfg.SeedData == nil || s.cfg.HibernateBucket == "" { |
| 1311 | jsonError(w, http.StatusServiceUnavailable, "this console has no data bucket set up, so it cannot create a seeded instance.") |
| 1312 | return |
| 1313 | } |
| 1314 | if ref := s.seedRegionReady(ctx, c.rec.Region, false); ref != nil { |
| 1315 | jsonError(w, ref.Status, ref.Msg) |
| 1316 | return |
| 1317 | } |
| 1318 | src, err := s.resolveDBSource(ctx, acct, c.principal, app, source) |
| 1319 | if ref := s.fromSourceRefusal(src, source, app, err); ref != nil { |
| 1320 | jsonError(w, ref.Status, ref.Msg) |
| 1321 | return |
| 1322 | } |
| 1323 | if src.rec.Region != c.rec.Region { |
| 1324 | jsonError(w, http.StatusConflict, "a copy stays in its source's region. To run it elsewhere, create it here and then run carlos move.") |
| 1325 | return |
| 1326 | } |
| 1327 | rd, err := s.dbReplica(ctx, src.prefix) |
| 1328 | if err != nil { |
| 1329 | log.Printf("console: from %s: replica client: %v", source, err) |
| 1330 | jsonError(w, http.StatusBadGateway, fmt.Sprintf("could not copy %s. Nothing was created. Try again.", source)) |
| 1331 | return |
| 1332 | } |
| 1333 | client := dbdoor.NewClient(replicaDoer(rd)) |
| 1334 | plan, err := planRestore(ctx, client, from.TXID) |
| 1335 | switch { |
| 1336 | case errors.Is(err, dbdoor.ErrHistoryGone): |
| 1337 | jsonError(w, http.StatusConflict, fmt.Sprintf("history before %d is no longer kept", from.TXID)) |
| 1338 | return |
| 1339 | case errors.Is(err, dbdoor.ErrNothingReplicated): |
| 1340 | jsonError(w, http.StatusConflict, fmt.Sprintf("%s has not replicated anything yet. Wake it once, then try again.", source)) |
| 1341 | return |
| 1342 | case err != nil: |
| 1343 | log.Printf("console: from %s: plan: %v", source, err) |
| 1344 | jsonError(w, http.StatusBadGateway, fmt.Sprintf("could not copy %s. Nothing was created. Try again.", source)) |
| 1345 | return |
| 1346 | } |
| 1347 | if plan.RestoredBytes > seedMaxBytes { |
| 1348 | jsonError(w, http.StatusRequestEntityTooLarge, fmt.Sprintf("a copy of %s would be %s, over the 4 GiB limit.", source, gib(plan.RestoredBytes))) |
| 1349 | return |
| 1350 | } |
| 1351 | release_, ref := s.seedScratch.reserve(acct.Sqid+"/"+app, plan.RestoredBytes+plan.LTXBytes) |
| 1352 | if ref != nil { |
| 1353 | jsonError(w, ref.Status, ref.Msg) |
| 1354 | return |
| 1355 | } |
| 1356 | defer release_() |
| 1357 | id, err := seeds.NewID() |
| 1358 | if err != nil { |
| 1359 | jsonError(w, http.StatusBadGateway, fmt.Sprintf("could not copy %s. Nothing was created. Try again.", source)) |
| 1360 | return |
| 1361 | } |
| 1362 | if ref := s.seedStoreRefusal(ctx, c, id); ref != nil { |
| 1363 | jsonError(w, ref.Status, ref.Msg) |
| 1364 | return |
| 1365 | } |
| 1366 | rec := c.rec |
| 1367 | rec.Seed = &provision.SeedSource{ID: id, Kind: seeds.KindReplica, From: source, State: seeds.StateReserving, |
| 1368 | By: c.principal, At: time.Now().UTC()} |
| 1369 | if err := provision.CreateRecord(ctx, s.cfg.Releases, rec); err != nil { |
| 1370 | if errors.Is(err, release.ErrCASConflict) { |
| 1371 | jsonError(w, http.StatusConflict, "instance "+host+" already exists") |
| 1372 | return |
| 1373 | } |
| 1374 | log.Printf("console: from %s: reserve %s: %v", source, host, err) |
| 1375 | jsonError(w, http.StatusBadGateway, fmt.Sprintf("could not copy %s. Nothing was created. Try again.", source)) |
| 1376 | return |
| 1377 | } |
| 1378 | // From here every failure leaves an aborted record for the box and the |
| 1379 | // console's finish to clean up (spec §4 step 7), written under a |
| 1380 | // context the client's departure cannot cancel. |
| 1381 | abort := func(why string) { |
| 1382 | bg, cancel := context.WithTimeout(context.WithoutCancel(ctx), instanceStoreTimeout) |
| 1383 | defer cancel() |
| 1384 | if _, err := provision.SetRecordSeed(bg, s.cfg.Releases, acct.Sqid, app, host, provision.SeedAbort(id)); err != nil { |
| 1385 | log.Printf("console: from %s: abort %s after %s: %v", source, host, why, err) |
| 1386 | } |
| 1387 | s.bumpSerial(bg) |
| 1388 | } |
| 1389 | if c.m.PrivateStorage { |
| 1390 | if _, ok := s.instanceStoreProvisionFor(ctx, w, acct, app, rec.Environment, c.principal, id); !ok { |
| 1391 | abort("store provisioning") |
| 1392 | return |
| 1393 | } |
| 1394 | } |
| 1395 | dir, err := os.MkdirTemp(s.seedScratch.dir(), "from-") |
| 1396 | if err != nil { |
| 1397 | abort("scratch") |
| 1398 | jsonError(w, http.StatusInsufficientStorage, "the console has no room for this right now. Try again later.") |
| 1399 | return |
| 1400 | } |
| 1401 | defer os.RemoveAll(dir) |
| 1402 | rctx, cancel := context.WithTimeout(ctx, seedRestoreDeadline) |
| 1403 | defer cancel() |
| 1404 | grew := restoreWatch(rctx, cancel, dir, plan.RestoredBytes) |
| 1405 | restored, err := restoreToTarget(rctx, client, dir, plan.Target) |
| 1406 | if err != nil { |
| 1407 | abort("restore") |
| 1408 | switch { |
| 1409 | case grew(): |
| 1410 | jsonError(w, http.StatusRequestEntityTooLarge, fmt.Sprintf("the copy of %s grew past its planned size, so it was stopped and no instance was created.", source)) |
| 1411 | case ctx.Err() != nil: |
| 1412 | // the client went away: nobody to answer |
| 1413 | case errors.Is(rctx.Err(), context.DeadlineExceeded): |
| 1414 | jsonError(w, http.StatusGatewayTimeout, fmt.Sprintf("copying %s took longer than 30 minutes, so no instance was created.", source)) |
| 1415 | case errors.Is(err, dbdoor.ErrCheckFailed): |
| 1416 | jsonError(w, http.StatusUnprocessableEntity, fmt.Sprintf("the copy of %s failed its integrity check, so no instance was created.", source)) |
| 1417 | case errors.Is(err, dbdoor.ErrHistoryGone): |
| 1418 | jsonError(w, http.StatusConflict, fmt.Sprintf("history before %d is no longer kept", from.TXID)) |
| 1419 | default: |
| 1420 | log.Printf("console: from %s: restore: %v", source, err) |
| 1421 | jsonError(w, http.StatusBadGateway, fmt.Sprintf("could not copy %s. Nothing was created. Try again.", source)) |
| 1422 | } |
| 1423 | return |
| 1424 | } |
| 1425 | sha, size, err := uploadSnapshot(ctx, s.cfg.SeedData, restored.Path, seeds.SnapshotKey(host, id)) |
| 1426 | if err != nil { |
| 1427 | abort("upload") |
| 1428 | log.Printf("console: from %s: upload %s: %v", source, host, err) |
| 1429 | jsonError(w, http.StatusBadGateway, fmt.Sprintf("could not copy %s. Nothing was created. Try again.", source)) |
| 1430 | return |
| 1431 | } |
| 1432 | published, err := provision.SetRecordSeed(ctx, s.cfg.Releases, acct.Sqid, app, host, |
| 1433 | provision.SeedPublishReplica(id, strconv.FormatUint(restored.TXID, 10), sha, size)) |
| 1434 | if err != nil { |
| 1435 | abort("publish") |
| 1436 | log.Printf("console: from %s: publish %s: %v", source, host, err) |
| 1437 | jsonError(w, http.StatusBadGateway, fmt.Sprintf("could not copy %s. Nothing was created. Try again.", source)) |
| 1438 | return |
| 1439 | } |
| 1440 | s.auditActivity(acct, app, c.principal, Activity{ |
| 1441 | Action: "instance-seed", TargetType: "instance", Target: host, Message: c.message, |
| 1442 | Detail: fmt.Sprintf("seeded %s from %s at txid %d", host, source, restored.TXID), |
| 1443 | }) |
| 1444 | s.bumpSerial(ctx) |
| 1445 | out := recordView(*published) |
| 1446 | out.RegionFrom = c.regionFrom |
| 1447 | writeJSON(w, http.StatusCreated, out) |
| 1448 | } |
| 1449 | |
| 1450 | // fromSourceRefusal maps resolveDBSource's answer to --from's sentences. |
| 1451 | func (s *Server) fromSourceRefusal(src dbSource, source, app string, err error) *doorRefusal { |
| 1452 | switch { |
| 1453 | case err == nil: |
| 1454 | return nil |
| 1455 | case errors.Is(err, os.ErrNotExist), errors.Is(err, ErrBadHost): |
| 1456 | return &doorRefusal{http.StatusConflict, fmt.Sprintf("no instance %s in %s. A copy can only come from an instance of the same app in the same account.", source, app)} |
| 1457 | case errors.Is(err, errSourceClass): |
| 1458 | return &doorRefusal{http.StatusConflict, fmt.Sprintf("%s is not an instance the platform keeps a replica of, so it cannot be copied.", source)} |
| 1459 | case errors.Is(err, ErrPooled): |
| 1460 | return &doorRefusal{http.StatusConflict, fmt.Sprintf("%s runs in the pool %s, so the platform keeps no replica of it to copy.", source, src.rec.Pool)} |
| 1461 | case errors.Is(err, ErrResidency): |
| 1462 | if fleet, ok := registry.FleetPlacement(src.rec.Region); ok { |
| 1463 | return &doorRefusal{http.StatusConflict, fmt.Sprintf("%s runs on your fleet %s, so the platform keeps no replica of it to copy.", source, fleet)} |
| 1464 | } |
| 1465 | return &doorRefusal{http.StatusConflict, s.dbResidencyRefusal(src.rec)} |
| 1466 | } |
| 1467 | log.Printf("console: from %s: resolve: %v", source, err) |
| 1468 | return &doorRefusal{http.StatusBadGateway, fmt.Sprintf("could not copy %s. Nothing was created. Try again.", source)} |
| 1469 | } |
| 1470 | |
| 1471 | // uploadSnapshot hashes path and writes it to key with one create-only |
| 1472 | // PutFrom (spec §4 step 7.2; seekable and under the 5 GB single-PUT limit). |
| 1473 | func uploadSnapshot(ctx context.Context, data release.Store, path, key string) (sha string, size int64, err error) { |
| 1474 | f, err := os.Open(path) |
| 1475 | if err != nil { |
| 1476 | return "", 0, err |
| 1477 | } |
| 1478 | defer f.Close() |
| 1479 | h := sha256.New() |
| 1480 | if size, err = io.Copy(h, f); err != nil { |
| 1481 | return "", 0, err |
| 1482 | } |
| 1483 | if _, err := f.Seek(0, io.SeekStart); err != nil { |
| 1484 | return "", 0, err |
| 1485 | } |
| 1486 | if _, err := release.PutObjectFrom(ctx, data, key, f, size, ""); err != nil { |
| 1487 | return "", 0, err |
| 1488 | } |
| 1489 | return hex.EncodeToString(h.Sum(nil)), size, nil |
| 1490 | } |
| 1491 | |
| 1492 | // restoreWatch stats the restore's output every second and cancels the |
| 1493 | // restore when it grows past limit (spec §4 step 5). grew reports whether |
| 1494 | // it fired. |
| 1495 | func restoreWatch(ctx context.Context, cancel context.CancelFunc, dir string, limit int64) (grew func() bool) { |
| 1496 | var fired atomicBool |
| 1497 | go func() { |
| 1498 | t := time.NewTicker(time.Second) |
| 1499 | defer t.Stop() |
| 1500 | for { |
| 1501 | select { |
| 1502 | case <-ctx.Done(): |
| 1503 | return |
| 1504 | case <-t.C: |
| 1505 | } |
| 1506 | matches, _ := filepath.Glob(filepath.Join(dir, "carlos-db-*", "replica.db")) |
| 1507 | for _, m := range matches { |
| 1508 | if st, err := os.Stat(m); err == nil && st.Size() > limit { |
| 1509 | fired.set() |
| 1510 | cancel() |
| 1511 | return |
| 1512 | } |
| 1513 | } |
| 1514 | } |
| 1515 | }() |
| 1516 | return fired.get |
| 1517 | } |
| 1518 | ``` |
| 1519 | |
| 1520 | `atomicBool` is `struct{ v atomic.Bool }` with `set()` and `get() bool`; write it in the same file (import `sync/atomic`). `restoreWatch` ticks once a second, so `TestFromWatcherCancelsAnOvergrownRestore` needs a restore that runs for more than a second: if the fixture's 30-transaction restore finishes faster, wrap `restoreToTarget` in that test to sleep 1.5 s after it returns its result and check `grew` before using the result. Make `createFrom` check `grew()` after a successful restore too: a file over the reservation is refused (SD-N32) and aborted even if the watcher's tick came late. |
| 1521 | |
| 1522 | `os.Open` here is a census site: add `internal/console/apiinstanceseed.go|uploadSnapshot|os.Open(path)` to `testdata/claim-authority-sites.txt` (the exact line the census test prints). |
| 1523 | |
| 1524 | - [ ] **Step 4: Hand off from the create door** |
| 1525 | |
| 1526 | In `handleAPIInstanceCreate`'s request struct, add: |
| 1527 | |
| 1528 | ```go |
| 1529 | // The seed doors (seeded-instances spec §4, §5.1): a create that |
| 1530 | // starts from a sibling's replica or an uploaded snapshot. Both run |
| 1531 | // every check above, then leave this handler (apiinstanceseed.go, |
| 1532 | // apiinstanceimport.go). |
| 1533 | From *seedFromReq `json:"from,omitempty"` |
| 1534 | ``` |
| 1535 | |
| 1536 | Right after `readJSON`/`checkedActivityMessage`: |
| 1537 | |
| 1538 | ```go |
| 1539 | seeded := req.From != nil |
| 1540 | if seeded { |
| 1541 | if ref := seedCommonRefusal(strings.TrimSpace(req.Kind), req.Addrs, strings.TrimSpace(req.Role), req.Replicas); ref != nil { |
| 1542 | jsonError(w, ref.Status, ref.Msg) |
| 1543 | return |
| 1544 | } |
| 1545 | } |
| 1546 | ``` |
| 1547 | |
| 1548 | Before `if m.PrivateStorage {` (`:1096`): |
| 1549 | |
| 1550 | ```go |
| 1551 | if seeded && m.PrivateStorage && strings.TrimSpace(req.Environment) != "" { |
| 1552 | s.refuseSeededEnvironment(w) |
| 1553 | return |
| 1554 | } |
| 1555 | ``` |
| 1556 | |
| 1557 | with `refuseSeededEnvironment` in `apiinstanceseed.go` (the sentence lives there, not in the legacy-exempt file): |
| 1558 | |
| 1559 | ```go |
| 1560 | func (s *Server) refuseSeededEnvironment(w http.ResponseWriter) { |
| 1561 | jsonError(w, http.StatusConflict, "a seeded instance gets its own environment and an empty store") |
| 1562 | } |
| 1563 | ``` |
| 1564 | |
| 1565 | After the `leftoverReplicaRefusal` block (`:1169`), before `cancel()`: |
| 1566 | |
| 1567 | ```go |
| 1568 | if req.From != nil { |
| 1569 | s.createFrom(w, r, seedCreate{acct: acct, app: app, principal: principal, message: message, m: m, |
| 1570 | rec: rec, placed: placed, regionFrom: regionFrom}, *req.From) |
| 1571 | return |
| 1572 | } |
| 1573 | ``` |
| 1574 | |
| 1575 | `apiinstances.go` gains no sentence: the moves gate's legacy exemption stays honest. |
| 1576 | |
| 1577 | - [ ] **Step 5: Configure it** |
| 1578 | |
| 1579 | `server.go` `Config`: `SeedData release.Store // the home region's data bucket, where seeds/ lives; nil: the seed doors answer 503`. `cmd/console/main.go`, in the `console.Config` literal: `SeedScratchDir: filepath.Join(filepath.Dir(envOr("CONSOLE_DB", "/data/console/console.db")), "scratch", "seeds"),` and, before the literal: |
| 1580 | |
| 1581 | ```go |
| 1582 | var seedData release.Store |
| 1583 | if dbBucket != "" { |
| 1584 | if sd, err := release.NewS3StoreIn(context.Background(), dbBucket, dbRegion); err != nil { |
| 1585 | log.Printf("console: seed data store: %v (the seed doors will answer 503)", err) |
| 1586 | } else { |
| 1587 | seedData = sd |
| 1588 | } |
| 1589 | } |
| 1590 | ``` |
| 1591 | |
| 1592 | and `SeedData: seedData,` in the literal. Add `"NewS3StoreIn": true` to `claimAuthoritySites`' `selected` set and the line `../../cmd/console/main.go|main|release.NewS3StoreIn(context.Background(), dbBucket, dbRegion)` to `testdata/claim-authority-sites.txt` (use the exact line the failing census test prints; the function name is the enclosing function's). |
| 1593 | |
| 1594 | - [ ] **Step 6: Run the tests to verify they pass** |
| 1595 | |
| 1596 | Run: `go test ./internal/console/ -run 'TestFrom|TestCreate|TestAPIInstanceCreate|TestClaimAuthoritySourceCensus' -count=1 && go test . -run 'TestMoves' -count=1` |
| 1597 | Expected: PASS. |
| 1598 | |
| 1599 | - [ ] **Step 7: Wire SD-N20 to SD-N26, SD-N30 to SD-N39, SD-N41 and commit** |
| 1600 | |
| 1601 | Append those ids; run `go test . -run TestSeededWiredCopyIsApplied -count=1`. |
| 1602 | |
| 1603 | ```bash |
| 1604 | git add internal/console/ cmd/console/main.go approved_seeded_copy_test.go |
| 1605 | git commit -m "console: carlos instances create --from, reserve, restore, upload, publish (Plan 5 Task 5) |
| 1606 | |
| 1607 | Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>" |
| 1608 | ``` |
| 1609 | |
| 1610 | ### Task 6: `--from` and `--at` in the CLI, the SEED column, the docs |
| 1611 | |
| 1612 | **Files:** |
| 1613 | - Modify: `instances_cli.go:288` (`runInstancesCreate`), `:420` (`runInstancesList`) |
| 1614 | - Modify: `apimode.go:3118` (`apiInstanceRecord`), `:3257` (add `instanceCreateFrom`) |
| 1615 | - Create: `instances_seed.go` (`seedCell`, `apiSeed`), `instances_seed_test.go` |
| 1616 | - Modify: `docs/site/cli.md` (under `### carlos instances`, `:612`) |
| 1617 | |
| 1618 | **Interfaces:** |
| 1619 | - Produces: `type apiSeed struct{ ID, Kind, From, TXID, SHA256, State, Reason string; Bytes int64 }` (json tags as the record's); `apiInstanceRecord.Seed *apiSeed` (`json:"seed,omitempty"`); `(s *apiSession) instanceCreateFrom(ctx, a apiAcct, app, host, region, channel, environment, from string, at uint64, messages ...string) (apiInstanceRecord, error)`; `seedCell(r apiInstanceRecord) string`. |
| 1620 | |
| 1621 | `seedCell` renders spec §7's provenance: `-` for no seed; `reserving` SD-N46; replica `ready`/`placed` SD-N43; import `awaiting-upload` SD-N84, `uploaded` SD-N85, `ready` SD-N85 (the box has not swapped yet), `swapped`/`placed` SD-N86, `committed` SD-N83, `refused` SD-N87; `aborted`/`rolled-back` SD-N88. The import strings are wired here for the list's sake and their rows are wired by this task. |
| 1622 | |
| 1623 | - [ ] **Step 1: Write the failing tests** |
| 1624 | |
| 1625 | `instances_seed_test.go`: |
| 1626 | |
| 1627 | ```go |
| 1628 | package main |
| 1629 | |
| 1630 | import ( |
| 1631 | "context" |
| 1632 | "encoding/json" |
| 1633 | "io" |
| 1634 | "net/http" |
| 1635 | "net/http/httptest" |
| 1636 | "testing" |
| 1637 | ) |
| 1638 | |
| 1639 | func TestSeedCell(t *testing.T) { |
| 1640 | for _, c := range []struct { |
| 1641 | seed *apiSeed |
| 1642 | want string |
| 1643 | }{ |
| 1644 | {nil, "-"}, |
| 1645 | {&apiSeed{Kind: "replica", State: "reserving", From: "acme.go.tito.io"}, "copying from acme.go.tito.io"}, |
| 1646 | {&apiSeed{Kind: "replica", State: "placed", From: "acme.go.tito.io", TXID: "4211"}, "seeded from acme.go.tito.io at txid 4211"}, |
| 1647 | {&apiSeed{Kind: "import", State: "awaiting-upload"}, "import awaiting upload"}, |
| 1648 | {&apiSeed{Kind: "import", State: "uploaded"}, "awaiting stop"}, |
| 1649 | {&apiSeed{Kind: "import", State: "swapped"}, "swapped, awaiting verify"}, |
| 1650 | {&apiSeed{Kind: "import", State: "committed", SHA256: "0123456789abcdef0123"}, "imported 0123456789ab"}, |
| 1651 | {&apiSeed{Kind: "import", State: "refused", Reason: "the route belongs to app kass"}, "refused: the route belongs to app kass"}, |
| 1652 | {&apiSeed{Kind: "import", State: "aborted"}, "aborting"}, |
| 1653 | } { |
| 1654 | if got := seedCell(apiInstanceRecord{Seed: c.seed}); got != c.want { |
| 1655 | t.Errorf("seedCell(%+v) = %q, want %q", c.seed, got, c.want) |
| 1656 | } |
| 1657 | } |
| 1658 | } |
| 1659 | |
| 1660 | func TestInstanceCreateFromSendsTheSource(t *testing.T) { |
| 1661 | var body map[string]any |
| 1662 | ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { |
| 1663 | raw, _ := io.ReadAll(r.Body) |
| 1664 | _ = json.Unmarshal(raw, &body) |
| 1665 | w.WriteHeader(http.StatusCreated) |
| 1666 | w.Write([]byte(`{"host":"canary.go.tito.io","channel":"stable","seed":{"id":"sd_x","kind":"replica","from":"acme.go.tito.io","txid":"4211","state":"ready"}}`)) |
| 1667 | })) |
| 1668 | defer ts.Close() |
| 1669 | rec, err := newTestSession(ts.URL).instanceCreateFrom(context.Background(), apiAcct{Seg: "bab"}, "titogo", |
| 1670 | "canary.go.tito.io", "", "", "", "acme.go.tito.io", 4200) |
| 1671 | if err != nil { |
| 1672 | t.Fatal(err) |
| 1673 | } |
| 1674 | from, _ := body["from"].(map[string]any) |
| 1675 | if from["host"] != "acme.go.tito.io" || from["txid"] != float64(4200) || body["host"] != "canary.go.tito.io" { |
| 1676 | t.Fatalf("body = %v", body) |
| 1677 | } |
| 1678 | if rec.Seed == nil || rec.Seed.TXID != "4211" { |
| 1679 | t.Fatalf("record = %+v", rec) |
| 1680 | } |
| 1681 | } |
| 1682 | ``` |
| 1683 | |
| 1684 | - [ ] **Step 2: Run the tests to verify they fail** |
| 1685 | |
| 1686 | Run: `go test . -run 'TestSeedCell|TestInstanceCreateFrom' -count=1` |
| 1687 | Expected: FAIL to compile. |
| 1688 | |
| 1689 | - [ ] **Step 3: Write `instances_seed.go` and the client call** |
| 1690 | |
| 1691 | ```go |
| 1692 | package main |
| 1693 | |
| 1694 | import "fmt" |
| 1695 | |
| 1696 | // apiSeed is the record's seed as the CLI renders it |
| 1697 | // (provision.SeedSource's wire shape). |
| 1698 | type apiSeed struct { |
| 1699 | ID string `json:"id"` |
| 1700 | Kind string `json:"kind"` |
| 1701 | From string `json:"from,omitempty"` |
| 1702 | TXID string `json:"txid,omitempty"` |
| 1703 | SHA256 string `json:"sha256"` |
| 1704 | Bytes int64 `json:"bytes"` |
| 1705 | State string `json:"state"` |
| 1706 | Reason string `json:"reason,omitempty"` |
| 1707 | } |
| 1708 | |
| 1709 | // seedCell is the list's SEED column (seeded-instances spec §7). |
| 1710 | func seedCell(r apiInstanceRecord) string { |
| 1711 | s := r.Seed |
| 1712 | if s == nil { |
| 1713 | return "-" |
| 1714 | } |
| 1715 | switch s.State { |
| 1716 | case "reserving": |
| 1717 | return fmt.Sprintf("copying from %s", s.From) |
| 1718 | case "aborted", "rolled-back": |
| 1719 | return "aborting" |
| 1720 | case "refused": |
| 1721 | return fmt.Sprintf("refused: %s", s.Reason) |
| 1722 | } |
| 1723 | if s.Kind != "import" { |
| 1724 | return fmt.Sprintf("seeded from %s at txid %s", s.From, s.TXID) |
| 1725 | } |
| 1726 | switch s.State { |
| 1727 | case "awaiting-upload": |
| 1728 | return "import awaiting upload" |
| 1729 | case "uploaded", "ready": |
| 1730 | return "awaiting stop" |
| 1731 | case "swapped", "placed": |
| 1732 | return "swapped, awaiting verify" |
| 1733 | } |
| 1734 | sha := s.SHA256 |
| 1735 | if len(sha) > 12 { |
| 1736 | sha = sha[:12] |
| 1737 | } |
| 1738 | return fmt.Sprintf("imported %s", sha) |
| 1739 | } |
| 1740 | ``` |
| 1741 | |
| 1742 | `ready` for an import means the app has said it is stopped and the box has not swapped yet; SD-N85 ("awaiting stop") is the closest approved text, and the review can split it. |
| 1743 | |
| 1744 | In `apimode.go`, `apiInstanceRecord` gains `Seed *apiSeed `json:"seed,omitempty"`` (comment: "the record's seed (seeded-instances spec §3.2); nil on an unseeded record and on an older console"), and: |
| 1745 | |
| 1746 | ```go |
| 1747 | // instanceCreateFrom is `carlos instances create --from` (seeded-instances |
| 1748 | // spec §4): the ordinary create body plus "from". The console answers when |
| 1749 | // the copy is uploaded and published, which can take minutes, so the |
| 1750 | // request uses its own client with no absolute timeout beyond the door's |
| 1751 | // own 30-minute deadline. |
| 1752 | func (s *apiSession) instanceCreateFrom(ctx context.Context, a apiAcct, app, host, region, channel, environment, from string, at uint64, messages ...string) (apiInstanceRecord, error) { |
| 1753 | in := struct { |
| 1754 | Host string `json:"host"` |
| 1755 | Region string `json:"region,omitempty"` |
| 1756 | Channel string `json:"channel,omitempty"` |
| 1757 | Environment string `json:"environment,omitempty"` |
| 1758 | Message string `json:"message,omitempty"` |
| 1759 | From struct { |
| 1760 | Host string `json:"host"` |
| 1761 | TXID uint64 `json:"txid,omitempty"` |
| 1762 | } `json:"from"` |
| 1763 | }{Host: host, Region: region, Channel: channel, Environment: environment, Message: firstMutationMessage(messages)} |
| 1764 | in.From.Host, in.From.TXID = from, at |
| 1765 | body, _ := json.Marshal(in) |
| 1766 | ctx, cancel := context.WithTimeout(ctx, 35*time.Minute) |
| 1767 | defer cancel() |
| 1768 | res, err := s.c.doStream(ctx, http.MethodPost, apiPath(a, app, "/instances"), bytes.NewReader(body), int64(len(body)), http.Header{"Content-Type": {"application/json"}}) |
| 1769 | if err != nil { |
| 1770 | return apiInstanceRecord{}, err |
| 1771 | } |
| 1772 | defer res.Body.Close() |
| 1773 | raw, _ := io.ReadAll(io.LimitReader(res.Body, 1<<20)) |
| 1774 | if res.StatusCode != http.StatusCreated { |
| 1775 | return apiInstanceRecord{}, apiErrorFromBody(res.StatusCode, raw) |
| 1776 | } |
| 1777 | var out apiInstanceRecord |
| 1778 | if err := json.Unmarshal(raw, &out); err != nil { |
| 1779 | return apiInstanceRecord{}, fmt.Errorf("instances create: unreadable response: %w", err) |
| 1780 | } |
| 1781 | return out, nil |
| 1782 | } |
| 1783 | ``` |
| 1784 | |
| 1785 | `doStream` builds its own `http.Client{Timeout: shipHTTPClientTimeout}` (`auth.go:564`), whose zero value means no timeout; the 35-minute context bounds it. |
| 1786 | |
| 1787 | - [ ] **Step 4: The flags, the output and the column** |
| 1788 | |
| 1789 | In `runInstancesCreate`, after `replicas`: |
| 1790 | |
| 1791 | ```go |
| 1792 | from := fs.String("from", "", "copy this instance's database into the new one. The source must be an instance of the same app, in this console's region") |
| 1793 | at := fs.Int64("at", 0, "with --from, the transaction id to copy at (default: the latest)") |
| 1794 | ``` |
| 1795 | |
| 1796 | After `message, merr := …`: |
| 1797 | |
| 1798 | ```go |
| 1799 | if *from != "" || set["at"] { |
| 1800 | if *at < 0 || (set["at"] && *at == 0) || (*from == "" && set["at"]) { |
| 1801 | log.Fatalf("instances create: %s", "at must be a transaction id, a whole number above 0") |
| 1802 | } |
| 1803 | fmt.Fprintln(stdout, fmt.Sprintf("copying %s into %s. A large database can take several minutes.", *from, strings.TrimSpace(*host))) |
| 1804 | rec, err := sess.instanceCreateFrom(context.Background(), acct, app2, strings.TrimSpace(*host), strings.TrimSpace(*region), |
| 1805 | strings.TrimSpace(*channel), strings.TrimSpace(*environment), strings.TrimSpace(*from), uint64(*at), message) |
| 1806 | if err != nil { |
| 1807 | log.Fatalf("instances create: %v", err) |
| 1808 | } |
| 1809 | fmt.Fprintf(stdout, "%s %s\n", colorFor(stdout).ok("created instance"), instanceCreatedLine(rec, environmentSuffix(rec.Environment))) |
| 1810 | if rec.Seed != nil { |
| 1811 | mutedf(stdout, " %s\n", seedCell(rec)) |
| 1812 | } |
| 1813 | mutedf(stdout, " the host comes live on the next reconcile pass on its box") |
| 1814 | return |
| 1815 | } |
| 1816 | ``` |
| 1817 | |
| 1818 | `--kind`, `--addr`, `--role` and `--replicas` with `--from` are refused by the console (SD-N36): `instanceCreateFrom` sends none of them, so a caller who combines them is told by the CLI before anything is sent. Add before the block: `if *from != "" && (*kind != "" || set["addr"] || *role != "" || *replicas != "") { log.Fatalf("instances create: %s", "a seeded instance is an ordinary instance. Drop --kind, --addr and --role.") }`. |
| 1819 | |
| 1820 | In `runInstancesList`, the header becomes `fmt.Fprintln(w, "HOST\tREGION\tCHANNEL\tPOOL\tVERSION\tHEALTH\tUPSTREAMS\tDB\tCREATED\tBY\t"+"SEED")` (SD-N114 as its own literal, so the gate can find it), and each row's `Fprintf` gains one more `\t%s` with `seedCell(r)`. |
| 1821 | |
| 1822 | In `docs/site/cli.md`, under `### carlos instances`, after the create examples (`:634`), add a paragraph with SD-N47's approved text and the example: |
| 1823 | |
| 1824 | ````markdown |
| 1825 | ``` |
| 1826 | carlos instances create --app titogo --host canary.go.tito.io --from acme.go.tito.io |
| 1827 | carlos instances create --app titogo --host canary.go.tito.io --from acme.go.tito.io --at 4211 |
| 1828 | ``` |
| 1829 | ```` |
| 1830 | |
| 1831 | - [ ] **Step 5: Run the tests to verify they pass** |
| 1832 | |
| 1833 | Run: `go test . -run 'TestSeedCell|TestInstanceCreateFrom|TestInstances|TestSeeded' -count=1` |
| 1834 | Expected: PASS. |
| 1835 | |
| 1836 | - [ ] **Step 6: Wire SD-N40, SD-N42 to SD-N47, SD-N83 to SD-N88, SD-N114 and commit** |
| 1837 | |
| 1838 | Append those ids to `seededWiredIDs`. SD-N40 is the CLI's alone (the wire's `uint64` cannot carry a negative, and 0 means latest), and its deck row names `instances_cli.go`. Run `go test . -run TestSeededWiredCopyIsApplied -count=1`. |
| 1839 | |
| 1840 | ```bash |
| 1841 | git add instances_cli.go instances_seed.go instances_seed_test.go apimode.go docs/site/cli.md approved_seeded_copy_test.go |
| 1842 | git commit -m "carlos instances create --from and --at, and the SEED column (Plan 5 Task 6) |
| 1843 | |
| 1844 | Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>" |
| 1845 | ``` |
| 1846 | |
| 1847 | ### Task 7: Plan gate |
| 1848 | |
| 1849 | - [ ] **Step 1: Cross-compile and run the touched packages under -race** |
| 1850 | |
| 1851 | Run: |
| 1852 | ```bash |
| 1853 | GOOS=darwin go build ./... && GOOS=windows go build ./... && GOOS=linux GOARCH=arm64 go build ./... && \ |
| 1854 | nice go test -race ./internal/dbdoor/ ./internal/objstore/ ./internal/console/ . -count=1 |
| 1855 | ``` |
| 1856 | Expected: builds clean; tests PASS, including `TestClaimAuthoritySourceCensus` and the moves copy gate. |
| 1857 | |
| 1858 | --- |
| 1859 | |
| 1860 | ## Self-review |
| 1861 | |
| 1862 | - Spec coverage: §4's gate (Task 5 through `cliRoute`'s instance role; Task 1 leaves `dbGate` person-only), `resolveDBSource` (Task 1), steps 1 to 7 (Task 5), step 4's TXID target and the history refusal (Task 2), step 5's exact size, 413, the pool, 507, the limits, 429, the deadline and the watcher (Tasks 2, 3, 5), step 6's private 0700 scratch and `CheckDBContext` through `RestoreToTarget` (Task 5), step 7's reserve, upload with `PutFrom`, publish, and abort on failure (Task 5), "Cleanup" on every path and at start (Tasks 3, 5), "Environment" and "Storage" (Tasks 4, 5), §7's list cell and audit row (Tasks 5, 6). |
| 1863 | - §3.1 residency: the target region must equal the source's, and `resolveDBSource` refuses a source outside this console's region with the existing sentence. |
| 1864 | - Deviation: the seed `ID` is recorded on the store record's `Declared` block, not its `GrantBlock` (Task 4's note). The guarantee is the same: only that seed may reuse or tear down the store. |
| 1865 | - Deviation: `--from` keeps the HTTP request open until the copy is published (up to the 30-minute deadline); the spec does not say whether the door is synchronous. |
| 1866 | - Deviation: an `--at` at or past the latest transaction copies the latest (Task 2's `RestorePlan.Target`); the spec names only the too-old case. |
| 1867 | |