| 1 | # Seeded Instances, Plan 7: The Swap and the Rollback on the Box Implementation Plan |
| 2 | |
| 3 | > **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. |
| 4 | |
| 5 | **Goal:** On the box that holds a legacy addr route, an import whose record is `ready` replaces that exact route with the platform's own nested exec row in one registry transaction, journaled with the complete original row, behind an import gate that only the verify token passes; commit clears the gate; abort (or a record that vanished or names another seed) rolls back, fenced and idempotent across any crash, to the original route exactly. |
| 6 | |
| 7 | **Architecture:** The registry gains an `import_gate` column and an `import_journal` table, with transactions that change the route and the journal together (`SwapForImport`, `CloseImportGate`, `RestoreImportRoute`, `CommitImport`, `DeleteImportJournal`). The provision pass reads the journal before the bucket records on every pass: a journal row drives its host (resume the swap's record CAS, clear the gate after commit, or roll back), and the removal phase never touches a journaled host. A record whose import seed is `ready` and has no journal row goes through the swap's predicates (claims off, nested, shape, protected columns, addr, app, ownership with the operator assignment) and is either swapped or refused with the condition named. The rollback follows `internal/layout/pass.go`'s fence handling through a host-supplied hook. The edge and the edge door answer 503 to every request on a gated row except a matching `Carlos-Import-Verify`; the schedule runner skips gated rows. This plan's last task raises `seeds.ProtocolVersion` to 1. |
| 8 | |
| 9 | **Tech Stack:** Go 1.25, modernc sqlite (registry transactions), `internal/activator`'s maintenance fence API, `internal/instancefs`, `release.Store`. |
| 10 | |
| 11 | **Spec:** `docs/superpowers/specs/2026-10-08-seeded-instances-design.md` rev 7: §3.2's dispatch (codex 13) for an import's `ready` and the journal-first rule, §5.4 (codex 10, 11, 12, 14, 15, R1-10, R2-3), §5.6's rollback (codex R2-4, R2-7, R3-1, R4-1), §5.7, §6.3's box half, §6.4's use of the assignment (codex 9). Index: `2026-10-08-seeded-instances-0-index.md`. Copy deck rows SD-N91 to SD-N99. |
| 12 | |
| 13 | ## Global Constraints |
| 14 | |
| 15 | - Every user-facing string (CLI output, errors, console messages, docs prose) is copy-reviewed by Paul before it is written into code; implementation tasks use only strings from the copy deck. Copy voice: no em dashes; full stops and new sentences instead. |
| 16 | - CI cross-compiles darwin and windows: linux-only code needs `//go:build linux` plus a `!linux` stub. |
| 17 | - Local test commands: targeted tests (`go test ./internal/<pkg>/ -run TestX`), not the full suite in a loop. Inside the sandbox the default Go build cache may be read-only; if so use `GOCACHE=$TMPDIR/gocache` re-exported per command. Socket tests need a short TMPDIR (`TMPDIR=/tmp/g`). The local gate may need `GOFLAGS=-buildvcs=false`. |
| 18 | - Heavy loops run under `nice`; no -race full suite until the end of a plan. |
| 19 | - Production use needs nested boxes; the claim-armed path is refused here and left to #250 (spec §3.4). |
| 20 | - Commits end with `Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>`. |
| 21 | - No new file in this series imports `internal/moves`. New sentences go in new files (`internal/provision/seedswap.go`, `internal/edge/importgate.go`), never in a file listed in `movesScanFiles` (`registry.go`, `reconcile.go`, `edge.go`, `runner.go` are listed; they gain only calls and columns). |
| 22 | - When a task wires a deck row, append its id to `seededWiredIDs` in `approved_seeded_copy_test.go` in the same commit. |
| 23 | - Write each deck sentence as exactly one string literal or one format string holding the row's text, with `%s` or `%d` at each `<placeholder>` and nothing else: no prefix, no trailing `\n`, no leading spaces. The gate's matcher (`movesRowSpoken`, `approved_moves_copy_test.go:811`) anchors both ends. |
| 24 | - Every registry change that touches both a route and the journal is one SQLite transaction. Every rollback step is idempotent, and the journal's state says where to resume (spec §5.6). |
| 25 | - The import door refuses a region with more than one box (Plan 3's `oneBox`), so the box that holds the legacy row is the region's only box. This plan does not handle multi-box regions. |
| 26 | |
| 27 | ## Review Focus |
| 28 | |
| 29 | 1. A transient read failure of the bucket (or a record body that cannot be parsed) must never read as "the record is gone" and start a rollback: the journal-first pass rolls back for a missing record only when the pass read every record cleanly and the host is not protected. Test in Task 5 (`TestJournalDoesNotRollBackOnAReadFailure`), against Task 4's pass. |
| 30 | 2. A row that changed between the predicates and the transaction (an operator re-added it with another addr) must make the swap transaction fail, not replace the changed row. `SwapForImport` compares the stored row to the one the predicates read, inside the transaction. Test in Task 1 (`TestSwapForImportRefusesAChangedRow`). |
| 31 | 3. Rolling back after an agent restart (the activator's in-memory fence is gone, the host is cold and has no stop history) must proceed only when the box swept its instances at start and the host is quiesced, and must otherwise lift its fence and defer. Test in Task 5 (`TestRollbackFenceAfterRestart`). |
| 32 | 4. A resumed `cleanup-pending` must delete only this seed's snapshot and must never fence or stop the restored legacy process. Test in Task 5 (`TestCleanupPendingNeverFences`). |
| 33 | 5. A verify request whose token matches must reach the instance without the `Carlos-Import-Verify` header (the token never reaches the app's logs), and every other request on a gated row gets 503 with `Retry-After: 5`, including through the edge door. Test in Task 6 (`TestImportGateAtTheEdgeAndTheDoor`). |
| 34 | |
| 35 | --- |
| 36 | |
| 37 | ## File structure |
| 38 | |
| 39 | | path | responsibility | |
| 40 | |---|---| |
| 41 | | `internal/registry/registry.go` | `Route.ImportGate`, column `import_gate`, `Fingerprint`, `SnapshotFingerprint`, `routeArgs` shared by `Upsert` and the transactions, `OpsAccountSqid` | |
| 42 | | `internal/registry/importjournal.go` | the `import_journal` table, `ImportJournal`, the five transactions | |
| 43 | | `internal/seeds/legacyowners.go` | `control/legacy-route-owners.json`: type, read, lookup | |
| 44 | | `internal/provision/seedreconcile.go` | `SeedHooks` gains the swap and rollback hooks; the journal-first pass | |
| 45 | | `internal/provision/seedswap.go` | the predicates and the swap | |
| 46 | | `internal/provision/seedrollback.go` | the rollback, steps 1 to 8 | |
| 47 | | `internal/provision/reconcile.go` | calls: the journal pass, the import branch, the removal exclusion | |
| 48 | | `internal/edge/importgate.go` | the gate | |
| 49 | | `internal/edge/edge.go` | one call in `ServeHTTP` | |
| 50 | | `internal/schedule/runner.go` | gated rows skipped | |
| 51 | | `internal/host/seedwiring.go`, `internal/host/agent.go` | the fence hook, the swept flag, local removal, replica delete, owners, nested probe | |
| 52 | | `internal/seeds/floor.go` | `ProtocolVersion = 1` | |
| 53 | |
| 54 | --- |
| 55 | |
| 56 | ### Task 1: The import gate column and the journal |
| 57 | |
| 58 | **Files:** |
| 59 | - Modify: `internal/registry/registry.go` (`Route`; `migrate`; `routeCols`; `scanRoute`; `Upsert`; `Fingerprint` `:2427`; `SnapshotFingerprint` `:2657`) |
| 60 | - Create: `internal/registry/importjournal.go`, `internal/registry/importjournal_test.go` |
| 61 | |
| 62 | **Interfaces:** |
| 63 | - Produces: `Route.ImportGate string` (column `import_gate`; `Upsert`'s insert writes it, its conflict path never changes it); `OpsAccountSqid = "bab"` (the pinned ops account, `migrateAccounts`'s `INSERT OR IGNORE … (1, 'bab', 'ops')`); `type ImportJournal struct{ Host, SeedID, RecordKey, State string; Original Route; FenceGen uint64; At int64 }`; `JournalSwapped`, `JournalRollingBack`, `JournalCleanupPending` (the spec's strings); `(*Registry).ImportJournal(host string) (*ImportJournal, error)`; `(*Registry).ImportJournals() ([]ImportJournal, error)`; `(*Registry).SwapForImport(j ImportJournal, next Route) error`; `(*Registry).CloseImportGate(host, seedID string, fenceGen uint64) error`; `(*Registry).RestoreImportRoute(host, seedID string) error`; `(*Registry).CommitImport(host, seedID string) error`; `(*Registry).DeleteImportJournal(host, seedID string) error`; `ErrImportJournal` (a transaction's precondition failed). |
| 64 | |
| 65 | `SwapForImport` checks, inside one transaction: no journal row for the host; the stored route equals `j.Original` field for field (`reflect.DeepEqual` after a scan); then deletes the route, inserts `next` and inserts the journal row (`state = swapped`). `CloseImportGate` sets the journal row (which must name `seedID` and be `swapped` or `rolling-back`) to `rolling-back` with `fenceGen` and the route's `import_gate` to `seeds.ClosedGate`'s value (`"closed"`; the registry does not import `seeds`, so the literal lives in a registry constant `ImportGateClosed = "closed"` that `seeds.ClosedGate` must equal, pinned by a test in Task 7). `RestoreImportRoute` deletes the current route, inserts the journal's `Original`, and sets the journal to `cleanup-pending`. `CommitImport` clears `import_gate` and deletes the journal row. Each names the seed `ID` and fails with `ErrImportJournal` when the journal does not. |
| 66 | |
| 67 | - [ ] **Step 1: Write the failing tests** |
| 68 | |
| 69 | ```go |
| 70 | package registry |
| 71 | |
| 72 | import ( |
| 73 | "errors" |
| 74 | "reflect" |
| 75 | "testing" |
| 76 | ) |
| 77 | |
| 78 | func legacyRow() Route { |
| 79 | return Route{Host: "old.go.tito.io", Kind: "service", Addr: "127.0.0.1:8401", Status: "ready", |
| 80 | App: "titogo", Account: "bab", Channel: "stable", Unit: "titogo@old.service", Caps: "", CreatedAt: 1700000000} |
| 81 | } |
| 82 | |
| 83 | func execRow() Route { |
| 84 | return Route{Host: "old.go.tito.io", Kind: "instance", Socket: "/run/carlos/instances/old/app.sock", |
| 85 | DBPath: "/data/instances/old/old.go.tito.io.db", Status: "ready", App: "titogo", Account: "bae", |
| 86 | Channel: "stable", Hibernate: true, Provisioned: true, ImportGate: "abc123", CreatedAt: 1700000001} |
| 87 | } |
| 88 | |
| 89 | func TestSwapCommitAndRestore(t *testing.T) { |
| 90 | reg := openTestRegistry(t) |
| 91 | old := legacyRow() |
| 92 | if err := reg.Upsert(old); err != nil { |
| 93 | t.Fatal(err) |
| 94 | } |
| 95 | stored, _ := reg.Get(old.Host) |
| 96 | j := ImportJournal{Host: old.Host, SeedID: "sd_aaaaaaaaaaaaaaaaaaaaaaaaaa", RecordKey: "apps/bae/titogo/instances/old.go.tito.io.json", Original: *stored, At: 1} |
| 97 | if err := reg.SwapForImport(j, execRow()); err != nil { |
| 98 | t.Fatal(err) |
| 99 | } |
| 100 | got, _ := reg.Get(old.Host) |
| 101 | if got.Kind != "instance" || got.ImportGate != "abc123" || got.Account != "bae" { |
| 102 | t.Fatalf("after swap: %+v", got) |
| 103 | } |
| 104 | jr, _ := reg.ImportJournal(old.Host) |
| 105 | if jr == nil || jr.State != JournalSwapped || !reflect.DeepEqual(jr.Original, *stored) { |
| 106 | t.Fatalf("journal = %+v", jr) |
| 107 | } |
| 108 | // A second swap for the host is refused: the journal is there. |
| 109 | if err := reg.SwapForImport(j, execRow()); !errors.Is(err, ErrImportJournal) { |
| 110 | t.Fatalf("second swap: %v", err) |
| 111 | } |
| 112 | // Close the gate, restore the route exactly, and the journal says where it is. |
| 113 | if err := reg.CloseImportGate(old.Host, j.SeedID, 77); err != nil { |
| 114 | t.Fatal(err) |
| 115 | } |
| 116 | if got, _ := reg.Get(old.Host); got.ImportGate != ImportGateClosed { |
| 117 | t.Fatalf("gate = %q, want closed", got.ImportGate) |
| 118 | } |
| 119 | if err := reg.RestoreImportRoute(old.Host, j.SeedID); err != nil { |
| 120 | t.Fatal(err) |
| 121 | } |
| 122 | if got, _ := reg.Get(old.Host); !reflect.DeepEqual(*got, *stored) { |
| 123 | t.Fatalf("restored %+v, want %+v", got, stored) |
| 124 | } |
| 125 | if jr, _ := reg.ImportJournal(old.Host); jr == nil || jr.State != JournalCleanupPending || jr.FenceGen != 77 { |
| 126 | t.Fatalf("journal after restore = %+v", jr) |
| 127 | } |
| 128 | if err := reg.DeleteImportJournal(old.Host, "sd_bbbbbbbbbbbbbbbbbbbbbbbbbb"); !errors.Is(err, ErrImportJournal) { |
| 129 | t.Fatalf("delete with another id: %v", err) |
| 130 | } |
| 131 | if err := reg.DeleteImportJournal(old.Host, j.SeedID); err != nil { |
| 132 | t.Fatal(err) |
| 133 | } |
| 134 | if jr, _ := reg.ImportJournal(old.Host); jr != nil { |
| 135 | t.Fatalf("journal after delete = %+v", jr) |
| 136 | } |
| 137 | } |
| 138 | |
| 139 | func TestCommitImportClearsTheGateAndTheJournal(t *testing.T) { |
| 140 | reg := openTestRegistry(t) |
| 141 | old := legacyRow() |
| 142 | reg.Upsert(old) |
| 143 | stored, _ := reg.Get(old.Host) |
| 144 | j := ImportJournal{Host: old.Host, SeedID: "sd_aaaaaaaaaaaaaaaaaaaaaaaaaa", Original: *stored} |
| 145 | if err := reg.SwapForImport(j, execRow()); err != nil { |
| 146 | t.Fatal(err) |
| 147 | } |
| 148 | before, _ := reg.Fingerprint() |
| 149 | if err := reg.CommitImport(old.Host, j.SeedID); err != nil { |
| 150 | t.Fatal(err) |
| 151 | } |
| 152 | got, _ := reg.Get(old.Host) |
| 153 | jr, _ := reg.ImportJournal(old.Host) |
| 154 | after, _ := reg.Fingerprint() |
| 155 | if got.ImportGate != "" || jr != nil || before == after { |
| 156 | t.Fatalf("after commit: gate %q journal %+v fingerprint moved %v", got.ImportGate, jr, before != after) |
| 157 | } |
| 158 | } |
| 159 | |
| 160 | func TestSwapForImportRefusesAChangedRow(t *testing.T) { |
| 161 | reg := openTestRegistry(t) |
| 162 | old := legacyRow() |
| 163 | reg.Upsert(old) |
| 164 | stored, _ := reg.Get(old.Host) |
| 165 | changed := *stored |
| 166 | changed.Addr = "127.0.0.1:9999" |
| 167 | if err := reg.Upsert(changed); err != nil { |
| 168 | t.Fatal(err) |
| 169 | } |
| 170 | j := ImportJournal{Host: old.Host, SeedID: "sd_aaaaaaaaaaaaaaaaaaaaaaaaaa", Original: *stored} |
| 171 | if err := reg.SwapForImport(j, execRow()); !errors.Is(err, ErrImportJournal) { |
| 172 | t.Fatalf("swap over a changed row: %v", err) |
| 173 | } |
| 174 | if got, _ := reg.Get(old.Host); got.Addr != "127.0.0.1:9999" { |
| 175 | t.Fatal("the changed row was replaced") |
| 176 | } |
| 177 | } |
| 178 | |
| 179 | func TestUpsertNeverChangesTheImportGate(t *testing.T) { |
| 180 | reg := openTestRegistry(t) |
| 181 | row := execRow() |
| 182 | reg.Upsert(row) |
| 183 | row.ImportGate = "" |
| 184 | if err := reg.Upsert(row); err != nil { |
| 185 | t.Fatal(err) |
| 186 | } |
| 187 | if got, _ := reg.Get(row.Host); got.ImportGate != "abc123" { |
| 188 | t.Fatalf("an Upsert cleared the gate: %q", got.ImportGate) |
| 189 | } |
| 190 | } |
| 191 | ``` |
| 192 | |
| 193 | - [ ] **Step 2: Run the tests to verify they fail** |
| 194 | |
| 195 | Run: `go test ./internal/registry/ -run 'TestSwap|TestCommitImport|TestUpsertNeverChangesTheImportGate' -count=1` |
| 196 | Expected: FAIL to compile. |
| 197 | |
| 198 | - [ ] **Step 3: Add the column** |
| 199 | |
| 200 | In `Route`, after `SeedSource`: |
| 201 | |
| 202 | ```go |
| 203 | // ImportGate is an uncommitted import's gate (seeded-instances spec |
| 204 | // §5.4): the sha256 of its verify token, or ImportGateClosed during a |
| 205 | // rollback. Set: the edge answers 503 to every request without the |
| 206 | // matching token, and ticks skip the row. Only the import transactions |
| 207 | // (importjournal.go) write it; Upsert's conflict path never changes it. |
| 208 | // In Fingerprint (the edge must reload on a change), folded only when |
| 209 | // non-empty. |
| 210 | ImportGate string |
| 211 | ``` |
| 212 | |
| 213 | `migrate`: `` `ALTER TABLE routes ADD COLUMN import_gate TEXT NOT NULL DEFAULT ''`, ``. `routeCols`: append `, import_gate` (49). `scanRoute`: `&t.ImportGate`. `Upsert`: one more `?`, `t.ImportGate` appended to the arguments; no `SET` entry, so a conflict keeps the stored value. Factor the argument list into `func routeArgs(t Route) []any` (after `Upsert`'s two normalisations) so the journal's transactions insert a row with exactly `Upsert`'s columns: `Upsert` becomes `r.sql.Exec(upsertSQL, routeArgs(t)...)`. |
| 214 | |
| 215 | `Fingerprint`: add `import_gate` to the `SELECT` and fold it like `hold_op`: `if importGate != "" { fmt.Fprintf(h, "\x1fimport_gate=%q", importGate) }`. `SnapshotFingerprint`: give `import_gate` `hold_op`'s rule. |
| 216 | |
| 217 | Add beside `opsAccountID`: |
| 218 | |
| 219 | ```go |
| 220 | // OpsAccountSqid is the ops account's pinned sqid (migrateAccounts inserts |
| 221 | // it): the account a direct-mode `carlos add` files a route under. |
| 222 | const OpsAccountSqid = "bab" |
| 223 | |
| 224 | // ImportGateClosed is the import gate a rollback installs: it matches no |
| 225 | // token's hash (seeds.ClosedGate holds the same value). |
| 226 | const ImportGateClosed = "closed" |
| 227 | ``` |
| 228 | |
| 229 | - [ ] **Step 4: Write `importjournal.go`** |
| 230 | |
| 231 | ```go |
| 232 | package registry |
| 233 | |
| 234 | import ( |
| 235 | "database/sql" |
| 236 | "encoding/json" |
| 237 | "errors" |
| 238 | "fmt" |
| 239 | "reflect" |
| 240 | ) |
| 241 | |
| 242 | // The import journal (seeded-instances spec §5.7): one row per host this |
| 243 | // box has swapped for an import, holding the complete original route, so |
| 244 | // the box always knows what it replaced, whatever happened to the record. |
| 245 | const ( |
| 246 | JournalSwapped = "swapped" |
| 247 | JournalRollingBack = "rolling-back" |
| 248 | JournalCleanupPending = "cleanup-pending" |
| 249 | ) |
| 250 | |
| 251 | // ErrImportJournal is a journal transaction whose precondition did not |
| 252 | // hold: no row or another seed's, a row already there, a changed route. |
| 253 | var ErrImportJournal = errors.New("registry: the import journal does not allow this") |
| 254 | |
| 255 | type ImportJournal struct { |
| 256 | Host string |
| 257 | SeedID string |
| 258 | Original Route |
| 259 | RecordKey string |
| 260 | State string |
| 261 | FenceGen uint64 |
| 262 | At int64 |
| 263 | } |
| 264 | |
| 265 | func (r *Registry) migrateImportJournal() error { |
| 266 | _, err := r.sql.Exec(`CREATE TABLE IF NOT EXISTS import_journal ( |
| 267 | host TEXT PRIMARY KEY, |
| 268 | seed_id TEXT NOT NULL, |
| 269 | original TEXT NOT NULL, |
| 270 | record_key TEXT NOT NULL DEFAULT '', |
| 271 | state TEXT NOT NULL, |
| 272 | fence_gen INTEGER NOT NULL DEFAULT 0, |
| 273 | at INTEGER NOT NULL DEFAULT 0 |
| 274 | );`) |
| 275 | return err |
| 276 | } |
| 277 | |
| 278 | type queryer interface { |
| 279 | QueryRow(query string, args ...any) *sql.Row |
| 280 | } |
| 281 | |
| 282 | func scanJournal(q queryer, host string) (*ImportJournal, error) { |
| 283 | var j ImportJournal |
| 284 | var orig string |
| 285 | err := q.QueryRow(`SELECT host, seed_id, original, record_key, state, fence_gen, at FROM import_journal WHERE host = ?`, host). |
| 286 | Scan(&j.Host, &j.SeedID, &orig, &j.RecordKey, &j.State, &j.FenceGen, &j.At) |
| 287 | if errors.Is(err, sql.ErrNoRows) { |
| 288 | return nil, nil |
| 289 | } |
| 290 | if err != nil { |
| 291 | return nil, err |
| 292 | } |
| 293 | if err := json.Unmarshal([]byte(orig), &j.Original); err != nil { |
| 294 | return nil, fmt.Errorf("registry: import journal %s: original row: %w", host, err) |
| 295 | } |
| 296 | return &j, nil |
| 297 | } |
| 298 | |
| 299 | func (r *Registry) ImportJournal(host string) (*ImportJournal, error) { return scanJournal(r.sql, host) } |
| 300 | |
| 301 | func (r *Registry) ImportJournals() ([]ImportJournal, error) { |
| 302 | rows, err := r.sql.Query(`SELECT host FROM import_journal ORDER BY host`) |
| 303 | if err != nil { |
| 304 | return nil, err |
| 305 | } |
| 306 | var hosts []string |
| 307 | for rows.Next() { |
| 308 | var h string |
| 309 | if err := rows.Scan(&h); err != nil { |
| 310 | rows.Close() |
| 311 | return nil, err |
| 312 | } |
| 313 | hosts = append(hosts, h) |
| 314 | } |
| 315 | rows.Close() |
| 316 | out := make([]ImportJournal, 0, len(hosts)) |
| 317 | for _, h := range hosts { |
| 318 | j, err := r.ImportJournal(h) |
| 319 | if err != nil { |
| 320 | return nil, err |
| 321 | } |
| 322 | if j != nil { |
| 323 | out = append(out, *j) |
| 324 | } |
| 325 | } |
| 326 | return out, nil |
| 327 | } |
| 328 | |
| 329 | // inTx runs fn in one transaction. The registry's pool holds one |
| 330 | // connection (Open), so in-process writers serialise on it; another |
| 331 | // process's write waits on busy_timeout. |
| 332 | func (r *Registry) inTx(fn func(tx *sql.Tx) error) error { |
| 333 | tx, err := r.sql.Begin() |
| 334 | if err != nil { |
| 335 | return err |
| 336 | } |
| 337 | defer tx.Rollback() |
| 338 | if err := fn(tx); err != nil { |
| 339 | return err |
| 340 | } |
| 341 | return tx.Commit() |
| 342 | } |
| 343 | |
| 344 | func getRouteTx(tx *sql.Tx, host string) (*Route, error) { |
| 345 | t, err := scanRoute(tx.QueryRow(`SELECT `+routeCols+` FROM routes WHERE host = ?`, host).Scan) |
| 346 | if errors.Is(err, sql.ErrNoRows) { |
| 347 | return nil, nil |
| 348 | } |
| 349 | if err != nil { |
| 350 | return nil, err |
| 351 | } |
| 352 | return &t, nil |
| 353 | } |
| 354 | |
| 355 | func insertRouteTx(tx *sql.Tx, t Route) error { |
| 356 | _, err := tx.Exec(`INSERT INTO routes (`+routeCols+`) VALUES (`+placeholders(routeColCount)+`)`, routeArgs(t)...) |
| 357 | return err |
| 358 | } |
| 359 | |
| 360 | // SwapForImport is the swap's one transaction (spec §5.4, codex 14). |
| 361 | func (r *Registry) SwapForImport(j ImportJournal, next Route) error { |
| 362 | orig, err := json.Marshal(j.Original) |
| 363 | if err != nil { |
| 364 | return err |
| 365 | } |
| 366 | return r.inTx(func(tx *sql.Tx) error { |
| 367 | if cur, err := scanJournal(tx, j.Host); err != nil { |
| 368 | return err |
| 369 | } else if cur != nil { |
| 370 | return fmt.Errorf("%w: %s already has a journal row for seed %s", ErrImportJournal, j.Host, cur.SeedID) |
| 371 | } |
| 372 | stored, err := getRouteTx(tx, j.Host) |
| 373 | if err != nil { |
| 374 | return err |
| 375 | } |
| 376 | if stored == nil || !reflect.DeepEqual(*stored, j.Original) { |
| 377 | return fmt.Errorf("%w: the route at %s changed since it was checked", ErrImportJournal, j.Host) |
| 378 | } |
| 379 | if _, err := tx.Exec(`DELETE FROM routes WHERE host = ?`, j.Host); err != nil { |
| 380 | return err |
| 381 | } |
| 382 | if err := insertRouteTx(tx, next); err != nil { |
| 383 | return err |
| 384 | } |
| 385 | _, err = tx.Exec(`INSERT INTO import_journal (host, seed_id, original, record_key, state, fence_gen, at) VALUES (?,?,?,?,?,0,?)`, |
| 386 | j.Host, j.SeedID, string(orig), j.RecordKey, JournalSwapped, j.At) |
| 387 | return err |
| 388 | }) |
| 389 | } |
| 390 | |
| 391 | func journalFor(tx *sql.Tx, host, seedID string, states ...string) (*ImportJournal, error) { |
| 392 | j, err := scanJournal(tx, host) |
| 393 | if err != nil { |
| 394 | return nil, err |
| 395 | } |
| 396 | if j == nil || j.SeedID != seedID { |
| 397 | return nil, fmt.Errorf("%w: no journal row for %s seed %s", ErrImportJournal, host, seedID) |
| 398 | } |
| 399 | for _, s := range states { |
| 400 | if j.State == s { |
| 401 | return j, nil |
| 402 | } |
| 403 | } |
| 404 | return nil, fmt.Errorf("%w: %s's journal is %s", ErrImportJournal, host, j.State) |
| 405 | } |
| 406 | |
| 407 | // CloseImportGate is the rollback's step 1 (spec §5.6): the journal says |
| 408 | // rolling-back with the fence generation, and the gate refuses even verify |
| 409 | // requests, in one transaction. |
| 410 | func (r *Registry) CloseImportGate(host, seedID string, fenceGen uint64) error { |
| 411 | return r.inTx(func(tx *sql.Tx) error { |
| 412 | if _, err := journalFor(tx, host, seedID, JournalSwapped, JournalRollingBack); err != nil { |
| 413 | return err |
| 414 | } |
| 415 | if _, err := tx.Exec(`UPDATE import_journal SET state = ?, fence_gen = ? WHERE host = ?`, JournalRollingBack, fenceGen, host); err != nil { |
| 416 | return err |
| 417 | } |
| 418 | _, err := tx.Exec(`UPDATE routes SET import_gate = ? WHERE host = ?`, ImportGateClosed, host) |
| 419 | return err |
| 420 | }) |
| 421 | } |
| 422 | |
| 423 | // RestoreImportRoute is the rollback's step 4 (spec §5.6, codex R3-1): the |
| 424 | // original route back exactly, and the journal at cleanup-pending. |
| 425 | func (r *Registry) RestoreImportRoute(host, seedID string) error { |
| 426 | return r.inTx(func(tx *sql.Tx) error { |
| 427 | j, err := journalFor(tx, host, seedID, JournalRollingBack) |
| 428 | if err != nil { |
| 429 | return err |
| 430 | } |
| 431 | if _, err := tx.Exec(`DELETE FROM routes WHERE host = ?`, host); err != nil { |
| 432 | return err |
| 433 | } |
| 434 | if err := insertRouteTx(tx, j.Original); err != nil { |
| 435 | return err |
| 436 | } |
| 437 | _, err = tx.Exec(`UPDATE import_journal SET state = ? WHERE host = ?`, JournalCleanupPending, host) |
| 438 | return err |
| 439 | }) |
| 440 | } |
| 441 | |
| 442 | // CommitImport clears the gate and retires the journal in one transaction |
| 443 | // (spec §5.4 "Commit"). |
| 444 | func (r *Registry) CommitImport(host, seedID string) error { |
| 445 | return r.inTx(func(tx *sql.Tx) error { |
| 446 | if _, err := journalFor(tx, host, seedID, JournalSwapped); err != nil { |
| 447 | return err |
| 448 | } |
| 449 | if _, err := tx.Exec(`UPDATE routes SET import_gate = '' WHERE host = ?`, host); err != nil { |
| 450 | return err |
| 451 | } |
| 452 | _, err := tx.Exec(`DELETE FROM import_journal WHERE host = ?`, host) |
| 453 | return err |
| 454 | }) |
| 455 | } |
| 456 | |
| 457 | // DeleteImportJournal is the rollback's step 7. |
| 458 | func (r *Registry) DeleteImportJournal(host, seedID string) error { |
| 459 | return r.inTx(func(tx *sql.Tx) error { |
| 460 | if _, err := journalFor(tx, host, seedID, JournalCleanupPending); err != nil { |
| 461 | return err |
| 462 | } |
| 463 | _, err := tx.Exec(`DELETE FROM import_journal WHERE host = ?`, host) |
| 464 | return err |
| 465 | }) |
| 466 | } |
| 467 | ``` |
| 468 | |
| 469 | Add `routeColCount = 49` and `placeholders(n int) string` (n comma-joined `?`) beside `routeCols`, and use `placeholders(routeColCount)` in `Upsert`'s `VALUES` too, so a column added later changes one number. Call `r.migrateImportJournal()` from `migrate` before `migrateAccounts`. `Registry.sql` uses `MaxOpenConns(1)`, so the transaction serialises with every other write on this handle; another process's write waits on `busy_timeout`. |
| 470 | |
| 471 | - [ ] **Step 5: Run the registry package** |
| 472 | |
| 473 | Run: `go test ./internal/registry/ -count=1` |
| 474 | Expected: PASS, including `TestUpsertRoundTripsAllColumns` with `ImportGate` added to its `want` and the snapshot digest's stability test. |
| 475 | |
| 476 | - [ ] **Step 6: Commit** |
| 477 | |
| 478 | ```bash |
| 479 | git add internal/registry/ |
| 480 | git commit -m "registry: the import gate column and the import journal's transactions (Plan 7 Task 1) |
| 481 | |
| 482 | Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>" |
| 483 | ``` |
| 484 | |
| 485 | ### Task 2: The operator assignment reader |
| 486 | |
| 487 | **Files:** |
| 488 | - Create: `internal/seeds/legacyowners.go`, `internal/seeds/legacyowners_test.go` |
| 489 | |
| 490 | **Interfaces:** |
| 491 | - Produces: `LegacyOwnersKey = "control/legacy-route-owners.json"`; `type LegacyOwner struct{ Account, App string }`; `type LegacyOwners struct{ V int; Names map[string]LegacyOwner }` (json `v`, `names`, `account`, `app`); `ReadLegacyOwners(ctx, s release.Store) (*LegacyOwners, string, error)` (absent: an empty set, `""`, nil); `(*LegacyOwners).Lookup(name string) (LegacyOwner, bool)`. Plan 8 writes the file. |
| 492 | |
| 493 | - [ ] **Step 1: Write the failing test** |
| 494 | |
| 495 | ```go |
| 496 | package seeds |
| 497 | |
| 498 | import ( |
| 499 | "context" |
| 500 | "testing" |
| 501 | |
| 502 | "github.com/carlosframework/platform/internal/release" |
| 503 | ) |
| 504 | |
| 505 | func TestLegacyOwners(t *testing.T) { |
| 506 | ctx := context.Background() |
| 507 | s := release.NewDirStore(t.TempDir()) |
| 508 | o, etag, err := ReadLegacyOwners(ctx, s) |
| 509 | if err != nil || etag != "" || o == nil || len(o.Names) != 0 { |
| 510 | t.Fatalf("absent = %+v %q %v", o, etag, err) |
| 511 | } |
| 512 | if _, err := s.Put(ctx, LegacyOwnersKey, []byte(`{"v":1,"names":{"titogo":{"account":"bae","app":"titogo"}}}`), ""); err != nil { |
| 513 | t.Fatal(err) |
| 514 | } |
| 515 | o, etag, err = ReadLegacyOwners(ctx, s) |
| 516 | if err != nil || etag == "" { |
| 517 | t.Fatal(err) |
| 518 | } |
| 519 | if got, ok := o.Lookup("titogo"); !ok || got != (LegacyOwner{Account: "bae", App: "titogo"}) { |
| 520 | t.Fatalf("lookup = %+v %v", got, ok) |
| 521 | } |
| 522 | if _, ok := o.Lookup("kass"); ok { |
| 523 | t.Fatal("an unassigned name resolved") |
| 524 | } |
| 525 | if _, err := s.Put(ctx, LegacyOwnersKey, []byte(`{garbage`), etag); err != nil { |
| 526 | t.Fatal(err) |
| 527 | } |
| 528 | if _, _, err := ReadLegacyOwners(ctx, s); err == nil { |
| 529 | t.Fatal("garbage read as no assignments") |
| 530 | } |
| 531 | } |
| 532 | ``` |
| 533 | |
| 534 | - [ ] **Step 2: Run it to verify it fails** |
| 535 | |
| 536 | Run: `go test ./internal/seeds/ -run TestLegacyOwners -count=1` |
| 537 | Expected: FAIL to compile. |
| 538 | |
| 539 | - [ ] **Step 3: Write `legacyowners.go`** |
| 540 | |
| 541 | ```go |
| 542 | package seeds |
| 543 | |
| 544 | import ( |
| 545 | "context" |
| 546 | "encoding/json" |
| 547 | "errors" |
| 548 | "fmt" |
| 549 | "os" |
| 550 | |
| 551 | "github.com/carlosframework/platform/internal/release" |
| 552 | ) |
| 553 | |
| 554 | // LegacyOwnersKey is the operator assignment of legacy route names |
| 555 | // (seeded-instances spec §6.4, codex 9): which account and app a route |
| 556 | // filed under the operator account belongs to. |
| 557 | const LegacyOwnersKey = "control/legacy-route-owners.json" |
| 558 | |
| 559 | type LegacyOwner struct { |
| 560 | Account string `json:"account"` |
| 561 | App string `json:"app"` |
| 562 | } |
| 563 | |
| 564 | type LegacyOwners struct { |
| 565 | V int `json:"v"` |
| 566 | Names map[string]LegacyOwner `json:"names"` |
| 567 | } |
| 568 | |
| 569 | // ReadLegacyOwners reads the assignments. Absent is an empty set; present |
| 570 | // and unreadable is an error, never "no assignment". |
| 571 | func ReadLegacyOwners(ctx context.Context, s release.Store) (*LegacyOwners, string, error) { |
| 572 | body, etag, err := s.Get(ctx, LegacyOwnersKey) |
| 573 | if errors.Is(err, os.ErrNotExist) { |
| 574 | return &LegacyOwners{V: 1, Names: map[string]LegacyOwner{}}, "", nil |
| 575 | } |
| 576 | if err != nil { |
| 577 | return nil, "", fmt.Errorf("seeds: read legacy route owners: %w", err) |
| 578 | } |
| 579 | var o LegacyOwners |
| 580 | if err := json.Unmarshal(body, &o); err != nil { |
| 581 | return nil, "", fmt.Errorf("seeds: parse legacy route owners: %w", err) |
| 582 | } |
| 583 | if o.Names == nil { |
| 584 | o.Names = map[string]LegacyOwner{} |
| 585 | } |
| 586 | return &o, etag, nil |
| 587 | } |
| 588 | |
| 589 | func (o *LegacyOwners) Lookup(name string) (LegacyOwner, bool) { |
| 590 | if o == nil { |
| 591 | return LegacyOwner{}, false |
| 592 | } |
| 593 | v, ok := o.Names[name] |
| 594 | return v, ok |
| 595 | } |
| 596 | ``` |
| 597 | |
| 598 | - [ ] **Step 4: Run it to verify it passes, and commit** |
| 599 | |
| 600 | Run: `go test ./internal/seeds/ -count=1` (PASS). |
| 601 | |
| 602 | ```bash |
| 603 | git add internal/seeds/legacyowners.go internal/seeds/legacyowners_test.go |
| 604 | git commit -m "seeds: the legacy route owners reader (Plan 7 Task 2) |
| 605 | |
| 606 | Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>" |
| 607 | ``` |
| 608 | |
| 609 | ### Task 3: The swap |
| 610 | |
| 611 | **Files:** |
| 612 | - Create: `internal/provision/seedswap.go`, `internal/provision/seedswap_test.go` |
| 613 | - Modify: `internal/provision/seedreconcile.go` (`SeedHooks` fields; `seedAction`'s import `ready` case) |
| 614 | - Modify: `internal/provision/reconcile.go` (the record loop calls `swapImport` for `seedSwap`) |
| 615 | |
| 616 | **Interfaces:** |
| 617 | - Consumes: Task 1's registry calls; Task 2's `ReadLegacyOwners`; `routeFor`, `SeedColumn`, `RecordKey`, `SetRecordSeed`, `SeedSwapped`, `SeedRefuse`; `mintLayoutReader`. |
| 618 | - Produces: `SeedHooks` gains `LegacyOwners func(ctx context.Context) (*seeds.LegacyOwners, error)`, `NestedOK func() error`, `Now func() time.Time`; `seedVerdict` gains `seedSwap`; `swapImport(ctx, s release.Store, reg *registry.Registry, cfg ReconcileConfig, rec Record, existing *registry.Route, nested func() (bool, error))`; `importRefusal(rec Record, existing *registry.Route, cfg ReconcileConfig, owners *seeds.LegacyOwners, nested bool) string` (the reason, `""` to swap). |
| 619 | |
| 620 | The predicates, in order, each refused with its reason (spec §5.4, §6.3, §6.4): |
| 621 | |
| 622 | 1. Claims armed: SD-N98. |
| 623 | 2. Not nested (the layout intent through `mintLayoutReader`, and `SeedHooks.NestedOK` when set): SD-N97. A layout read error defers (no refusal). |
| 624 | 3. No row for the host: SD-N91. |
| 625 | 4. Shape: `kind` other than `instance` or `service` (`<what>`: `kind <kind>`), a `socket` (`socket`), no `addr` and no `addrs` (`no address`), `provisioned` (`provisioned`), and any of `backing`, `tunnel`, `caps`, `platform`, `canary` (the column's name): SD-N96. |
| 626 | 5. The address: `addr` equals `Replaces`, or `addr` is empty and `addrs` is exactly `Replaces`; otherwise SD-N92 naming the row's address. |
| 627 | 6. The app: SD-N93 unless the row's app is the record's. |
| 628 | 7. The account: the record's account passes; `registry.OpsAccountSqid` passes only when the assignment maps the row's app name to the record's account and app (SD-N94 otherwise; an unreadable assignment file defers); any other account: SD-N95. |
| 629 | |
| 630 | Then: the exec row is `routeFor(rec, cfg.DataDir, "", true)` (nested explicitly, never `NestedPaths(existing)`, codex 12) with `ImportGate = rec.Seed.VerifyHash`; `cfg.ConfigGate` must admit it (a refusal defers); the journal row carries the complete stored row, the record key, the seed `ID` and the time; `SwapForImport`; then `SetRecordSeed(SeedSwapped(id))`. A failed CAS is redone by Task 4's journal pass. |
| 631 | |
| 632 | - [ ] **Step 1: Write the failing tests** |
| 633 | |
| 634 | ```go |
| 635 | package provision |
| 636 | |
| 637 | import ( |
| 638 | "context" |
| 639 | "strings" |
| 640 | "testing" |
| 641 | "time" |
| 642 | |
| 643 | "github.com/carlosframework/platform/internal/registry" |
| 644 | "github.com/carlosframework/platform/internal/release" |
| 645 | "github.com/carlosframework/platform/internal/seeds" |
| 646 | ) |
| 647 | |
| 648 | func importRecord(t *testing.T, s release.Store, sqid string) Record { |
| 649 | t.Helper() |
| 650 | rec := acmeRecord(sqid) |
| 651 | rec.Pool = "" |
| 652 | id, _ := seeds.NewID() |
| 653 | rec.Seed = &SeedSource{ID: id, Kind: seeds.KindImport, State: seeds.StateReady, SHA256: strings.Repeat("a", 64), Bytes: 10, |
| 654 | Replaces: "127.0.0.1:8401", VerifyHash: strings.Repeat("b", 64), By: "t", At: time.Now()} |
| 655 | if err := CreateRecord(ctx, s, rec); err != nil { |
| 656 | t.Fatal(err) |
| 657 | } |
| 658 | return rec |
| 659 | } |
| 660 | |
| 661 | func addrRow(rec Record, account string) registry.Route { |
| 662 | return registry.Route{Host: rec.Host, Kind: "service", Addr: "127.0.0.1:8401", App: rec.App, Account: account, |
| 663 | Channel: "stable", Status: "ready", Unit: "titogo@acme.service"} |
| 664 | } |
| 665 | |
| 666 | func importCfg(t *testing.T, owners map[string]seeds.LegacyOwner) ReconcileConfig { |
| 667 | cfg := nestedSeedCfg(dirStore(t)) |
| 668 | cfg.Pools = nil |
| 669 | cfg.Seeds.LegacyOwners = func(context.Context) (*seeds.LegacyOwners, error) { |
| 670 | return &seeds.LegacyOwners{V: 1, Names: owners}, nil |
| 671 | } |
| 672 | return cfg |
| 673 | } |
| 674 | |
| 675 | func TestSwapReplacesTheAddrRowAndJournalsIt(t *testing.T) { |
| 676 | s, sqid := setup(t) |
| 677 | rec := importRecord(t, s, sqid) |
| 678 | reg := regWith(t, addrRow(rec, sqid)) |
| 679 | original, _ := reg.Get(rec.Host) |
| 680 | if err := Reconcile(ctx, s, reg, importCfg(t, nil)); err != nil { |
| 681 | t.Fatal(err) |
| 682 | } |
| 683 | got := mustGet(t, reg, rec.Host) |
| 684 | if got.Kind != "instance" || got.Socket == "" || got.Addr != "" || got.ImportGate != rec.Seed.VerifyHash || |
| 685 | got.DBPath != NestedDBPath("/data", rec.Host) || got.Account != sqid || !got.Provisioned || got.Seed != "" { |
| 686 | t.Fatalf("swapped row = %+v", got) |
| 687 | } |
| 688 | j, _ := reg.ImportJournal(rec.Host) |
| 689 | if j == nil || j.SeedID != rec.Seed.ID || j.State != registry.JournalSwapped || j.Original != *original || |
| 690 | j.RecordKey != RecordKey(rec.Account, rec.App, rec.Host) { |
| 691 | t.Fatalf("journal = %+v", j) |
| 692 | } |
| 693 | r, _ := GetRecord(ctx, s, rec.Account, rec.App, rec.Host) |
| 694 | if r.Seed.State != seeds.StateSwapped { |
| 695 | t.Fatalf("record = %s", r.Seed.State) |
| 696 | } |
| 697 | } |
| 698 | |
| 699 | func TestSwapRefusesEachCondition(t *testing.T) { |
| 700 | for _, c := range []struct { |
| 701 | name string |
| 702 | row func(Record, string) *registry.Route |
| 703 | owners map[string]seeds.LegacyOwner |
| 704 | cfg func(*ReconcileConfig) |
| 705 | want string |
| 706 | }{ |
| 707 | {"no row", func(Record, string) *registry.Route { return nil }, nil, nil, "there is no route at"}, |
| 708 | {"addr changed", func(r Record, a string) *registry.Route { x := addrRow(r, a); x.Addr = "127.0.0.1:9"; return &x }, nil, nil, "the route's address is 127.0.0.1:9, not 127.0.0.1:8401"}, |
| 709 | {"another app", func(r Record, a string) *registry.Route { x := addrRow(r, a); x.App = "kass"; return &x }, nil, nil, "belongs to app kass"}, |
| 710 | {"a third account", func(r Record, _ string) *registry.Route { x := addrRow(r, "zzz"); return &x }, nil, nil, "filed under another account"}, |
| 711 | {"operator row, no assignment", func(r Record, _ string) *registry.Route { x := addrRow(r, registry.OpsAccountSqid); return &x }, nil, nil, "no assignment maps tito"}, |
| 712 | {"operator row, assigned elsewhere", func(r Record, _ string) *registry.Route { x := addrRow(r, registry.OpsAccountSqid); return &x }, |
| 713 | map[string]seeds.LegacyOwner{"tito": {Account: "other", App: "tito"}}, nil, "no assignment maps tito"}, |
| 714 | {"socket backed", func(r Record, a string) *registry.Route { x := addrRow(r, a); x.Socket = "/run/x.sock"; return &x }, nil, nil, "(socket)"}, |
| 715 | {"provisioned", func(r Record, a string) *registry.Route { x := addrRow(r, a); x.Provisioned = true; return &x }, nil, nil, "(provisioned)"}, |
| 716 | {"static", func(r Record, a string) *registry.Route { x := addrRow(r, a); x.Kind = "static"; return &x }, nil, nil, "(kind static)"}, |
| 717 | {"backing", func(r Record, a string) *registry.Route { x := addrRow(r, a); x.Backing = registry.BackingUnit; return &x }, nil, nil, "(backing)"}, |
| 718 | {"tunnel", func(r Record, a string) *registry.Route { x := addrRow(r, a); x.Tunnel = "t"; return &x }, nil, nil, "(tunnel)"}, |
| 719 | {"caps", func(r Record, a string) *registry.Route { x := addrRow(r, a); x.Caps = "host-role"; return &x }, nil, nil, "(caps)"}, |
| 720 | {"platform", func(r Record, a string) *registry.Route { x := addrRow(r, a); x.Platform = true; return &x }, nil, nil, "(platform)"}, |
| 721 | {"canary", func(r Record, a string) *registry.Route { x := addrRow(r, a); x.Canary = true; return &x }, nil, nil, "(canary)"}, |
| 722 | {"flat", func(r Record, a string) *registry.Route { x := addrRow(r, a); return &x }, nil, func(c *ReconcileConfig) { c.Layout = nil }, "does not run instances nested"}, |
| 723 | {"claimed", func(r Record, a string) *registry.Route { x := addrRow(r, a); return &x }, nil, func(c *ReconcileConfig) { c.ClaimEnabled = true }, "claim control is on"}, |
| 724 | } { |
| 725 | t.Run(c.name, func(t *testing.T) { |
| 726 | s, sqid := setup(t) |
| 727 | rec := importRecord(t, s, sqid) |
| 728 | var reg *registry.Registry |
| 729 | if row := c.row(rec, sqid); row != nil { |
| 730 | reg = regWith(t, *row) |
| 731 | } else { |
| 732 | reg = regWith(t) |
| 733 | } |
| 734 | cfg := importCfg(t, c.owners) |
| 735 | if c.cfg != nil { |
| 736 | c.cfg(&cfg) |
| 737 | } |
| 738 | if err := Reconcile(ctx, s, reg, cfg); err != nil { |
| 739 | t.Fatal(err) |
| 740 | } |
| 741 | r, _ := GetRecord(ctx, s, rec.Account, rec.App, rec.Host) |
| 742 | if r.Seed.State != seeds.StateRefused || !strings.Contains(r.Seed.Reason, c.want) { |
| 743 | t.Fatalf("seed = %s %q, want refused with %q", r.Seed.State, r.Seed.Reason, c.want) |
| 744 | } |
| 745 | if j, _ := reg.ImportJournal(rec.Host); j != nil { |
| 746 | t.Fatalf("a refused import left a journal row: %+v", j) |
| 747 | } |
| 748 | }) |
| 749 | } |
| 750 | } |
| 751 | |
| 752 | func TestSwapAcceptsAnAssignedOperatorRow(t *testing.T) { |
| 753 | s, sqid := setup(t) |
| 754 | rec := importRecord(t, s, sqid) |
| 755 | reg := regWith(t, addrRow(rec, registry.OpsAccountSqid)) |
| 756 | if err := Reconcile(ctx, s, reg, importCfg(t, map[string]seeds.LegacyOwner{"tito": {Account: sqid, App: "tito"}})); err != nil { |
| 757 | t.Fatal(err) |
| 758 | } |
| 759 | if got := mustGet(t, reg, rec.Host); got.Account != sqid || got.Kind != "instance" { |
| 760 | t.Fatalf("row = %+v; want it filed under the record's account", got) |
| 761 | } |
| 762 | } |
| 763 | ``` |
| 764 | |
| 765 | The setup manifest's app is `tito` (`setup`, `reconcile_test.go:127`), so the row's app and the assignment's name are `tito`. |
| 766 | |
| 767 | - [ ] **Step 2: Run the tests to verify they fail** |
| 768 | |
| 769 | Run: `go test ./internal/provision/ -run 'TestSwap' -count=1` |
| 770 | Expected: FAIL: the import is skipped (Plan 4's `seedAction`), so no row changes and no refusal is written. |
| 771 | |
| 772 | - [ ] **Step 3: Write `seedswap.go`** |
| 773 | |
| 774 | ```go |
| 775 | package provision |
| 776 | |
| 777 | import ( |
| 778 | "context" |
| 779 | "errors" |
| 780 | "fmt" |
| 781 | "log" |
| 782 | "time" |
| 783 | |
| 784 | "github.com/carlosframework/platform/internal/registry" |
| 785 | "github.com/carlosframework/platform/internal/release" |
| 786 | "github.com/carlosframework/platform/internal/seeds" |
| 787 | ) |
| 788 | |
| 789 | // importShape is the legacy row's shape check (spec §5.4, codex 10, R1-10): |
| 790 | // "" when the row is an addr-backed instance or service this import may |
| 791 | // take over, else what makes it not one. |
| 792 | func importShape(rt registry.Route) string { |
| 793 | switch { |
| 794 | case rt.Kind != "instance" && rt.Kind != "service": |
| 795 | return "kind " + rt.Kind |
| 796 | case rt.Socket != "": |
| 797 | return "socket" |
| 798 | case rt.Addr == "" && rt.Addrs == "": |
| 799 | return "no address" |
| 800 | case rt.Provisioned: |
| 801 | return "provisioned" |
| 802 | case rt.Backing != "": |
| 803 | return "backing" |
| 804 | case rt.Tunnel != "": |
| 805 | return "tunnel" |
| 806 | case rt.Caps != "": |
| 807 | return "caps" |
| 808 | case rt.Platform: |
| 809 | return "platform" |
| 810 | case rt.Canary: |
| 811 | return "canary" |
| 812 | } |
| 813 | return "" |
| 814 | } |
| 815 | |
| 816 | func importAddr(rt registry.Route) string { |
| 817 | if rt.Addr != "" { |
| 818 | return rt.Addr |
| 819 | } |
| 820 | return rt.Addrs |
| 821 | } |
| 822 | |
| 823 | // importRefusal is §5.4's predicates in order: "" to swap, else the |
| 824 | // refusal reason the record keeps. owners is consulted only for an |
| 825 | // operator-filed row. |
| 826 | func importRefusal(rec Record, existing *registry.Route, cfg ReconcileConfig, owners *seeds.LegacyOwners, nested bool) string { |
| 827 | switch { |
| 828 | case cfg.ClaimEnabled: |
| 829 | return "claim control is on for this box" |
| 830 | case !nested: |
| 831 | return "this box does not run instances nested" |
| 832 | case existing == nil: |
| 833 | return fmt.Sprintf("there is no route at %s to replace", rec.Host) |
| 834 | } |
| 835 | if what := importShape(*existing); what != "" { |
| 836 | return fmt.Sprintf("the route is not an address route this import can take over (%s)", what) |
| 837 | } |
| 838 | if importAddr(*existing) != rec.Seed.Replaces { |
| 839 | return fmt.Sprintf("the route's address is %s, not %s", importAddr(*existing), rec.Seed.Replaces) |
| 840 | } |
| 841 | if existing.App != rec.App { |
| 842 | return fmt.Sprintf("the route belongs to app %s", existing.App) |
| 843 | } |
| 844 | switch existing.Account { |
| 845 | case rec.Account: |
| 846 | return "" |
| 847 | case registry.OpsAccountSqid: |
| 848 | if o, ok := owners.Lookup(existing.App); ok && o.Account == rec.Account && o.App == rec.App { |
| 849 | return "" |
| 850 | } |
| 851 | return fmt.Sprintf("the route is filed under the operator account, and no assignment maps %s to this app", existing.App) |
| 852 | } |
| 853 | return "the route is filed under another account" |
| 854 | } |
| 855 | |
| 856 | // swapImport is the swap (spec §5.4): the predicates, then one registry |
| 857 | // transaction, then the record's CAS to swapped. Transient failures defer |
| 858 | // to the next pass; a failed predicate is a durable refusal. |
| 859 | func swapImport(ctx context.Context, s release.Store, reg *registry.Registry, cfg ReconcileConfig, rec Record, existing *registry.Route, nested func() (bool, error)) { |
| 860 | if cfg.DryRun { |
| 861 | log.Printf("provision: dry-run: would consider swapping %s for import %s", rec.Host, rec.Seed.ID) |
| 862 | return |
| 863 | } |
| 864 | isNested := false |
| 865 | if !cfg.ClaimEnabled { |
| 866 | ok, err := nested() |
| 867 | if err != nil { |
| 868 | log.Printf("provision: %s: import deferred: the box's layout intent could not be read: %v", rec.Host, err) |
| 869 | return |
| 870 | } |
| 871 | isNested = ok |
| 872 | if ok && cfg.Seeds.NestedOK != nil { |
| 873 | isNested = cfg.Seeds.NestedOK() == nil |
| 874 | } |
| 875 | } |
| 876 | var owners *seeds.LegacyOwners |
| 877 | if existing != nil && existing.Account == registry.OpsAccountSqid && cfg.Seeds.LegacyOwners != nil { |
| 878 | o, err := cfg.Seeds.LegacyOwners(ctx) |
| 879 | if err != nil { |
| 880 | log.Printf("provision: %s: import deferred: the legacy route assignments could not be read: %v", rec.Host, err) |
| 881 | return |
| 882 | } |
| 883 | owners = o |
| 884 | } |
| 885 | if reason := importRefusal(rec, existing, cfg, owners, isNested); reason != "" { |
| 886 | if _, err := SetRecordSeed(ctx, s, rec.Account, rec.App, rec.Host, SeedRefuse(rec.Seed.ID, reason)); err != nil { |
| 887 | log.Printf("provision: %s: record the import's refusal (%s): %v", rec.Host, reason, err) |
| 888 | return |
| 889 | } |
| 890 | log.Printf("provision: %s: import %s refused: %s", rec.Host, rec.Seed.ID, reason) |
| 891 | return |
| 892 | } |
| 893 | next := routeFor(rec, cfg.DataDir, "", true) |
| 894 | next.ImportGate = rec.Seed.VerifyHash |
| 895 | if cfg.ConfigGate != nil { |
| 896 | if err := cfg.ConfigGate(ctx, next); err != nil { |
| 897 | return // the gate logs its own refusal; retried every pass |
| 898 | } |
| 899 | } |
| 900 | now := time.Now |
| 901 | if cfg.Seeds.Now != nil { |
| 902 | now = cfg.Seeds.Now |
| 903 | } |
| 904 | j := registry.ImportJournal{Host: rec.Host, SeedID: rec.Seed.ID, Original: *existing, |
| 905 | RecordKey: RecordKey(rec.Account, rec.App, rec.Host), At: now().Unix()} |
| 906 | if err := reg.SwapForImport(j, next); err != nil { |
| 907 | if errors.Is(err, registry.ErrImportJournal) { |
| 908 | log.Printf("provision: %s: import swap deferred: %v", rec.Host, err) |
| 909 | return |
| 910 | } |
| 911 | log.Printf("provision: %s: import swap: %v", rec.Host, err) |
| 912 | return |
| 913 | } |
| 914 | log.Printf("provision: %s: swapped for import %s (the original route is journaled)", rec.Host, rec.Seed.ID) |
| 915 | if _, err := SetRecordSeed(ctx, s, rec.Account, rec.App, rec.Host, SeedSwapped(rec.Seed.ID)); err != nil { |
| 916 | log.Printf("provision: %s: import %s swapped, record CAS deferred to the next pass: %v", rec.Host, rec.Seed.ID, err) |
| 917 | } |
| 918 | } |
| 919 | ``` |
| 920 | |
| 921 | In `seedreconcile.go`: add the three `SeedHooks` fields, the verdict `seedSwap`, and in `seedAction`'s `StateReady` case replace `if src.Import() { return seedSkip }` with `if src.Import() { return seedSwap }`. In `reconcile.go`'s dispatch: |
| 922 | |
| 923 | ```go |
| 924 | case seedSwap: |
| 925 | var row *registry.Route |
| 926 | if ok { |
| 927 | row = &existing |
| 928 | } |
| 929 | swapImport(ctx, s, reg, cfg, rec, row, mintLayout) |
| 930 | continue |
| 931 | ``` |
| 932 | |
| 933 | Plan 4's `TestReconcileSkipsSeedsThatAreNotReady` lists an import `ready` as skipped; move that case to this task's tests (it is now a swap or a refusal) by deleting it from Plan 4's table. |
| 934 | |
| 935 | - [ ] **Step 4: Run the tests to verify they pass** |
| 936 | |
| 937 | Run: `go test ./internal/provision/ -run 'TestSwap|TestReconcile|TestSeed' -count=1` |
| 938 | Expected: PASS. |
| 939 | |
| 940 | - [ ] **Step 5: Wire SD-N91 to SD-N98 and commit** |
| 941 | |
| 942 | ```bash |
| 943 | git add internal/provision/ approved_seeded_copy_test.go |
| 944 | git commit -m "provision: the import swap, its predicates and its refusals (Plan 7 Task 3) |
| 945 | |
| 946 | Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>" |
| 947 | ``` |
| 948 | |
| 949 | ### Task 4: The journal comes first |
| 950 | |
| 951 | **Files:** |
| 952 | - Modify: `internal/provision/seedreconcile.go` (add `journalPass`), `internal/provision/reconcile.go` (call it after `byHost` is built; skip its hosts in the record loop and the removal phase; `seedAction` refuses import work when the journal could not be read) |
| 953 | - Test: `internal/provision/seedjournal_test.go` |
| 954 | |
| 955 | **Interfaces:** |
| 956 | - Consumes: `reg.ImportJournals`, `reg.CommitImport`, `SetRecordSeed`, `SeedSwapped`; Task 5's `rollbackImport` (this task calls a stub that Task 5 fills). |
| 957 | - Produces: `journalPass(ctx, s release.Store, reg *registry.Registry, cfg ReconcileConfig, recs map[string]Record, readFailed bool, protect map[string]bool) (handled map[string]bool, journaled map[string]bool, ok bool)`. |
| 958 | |
| 959 | For each journal row (spec §5.4 "Crashes", §5.6, §5.7): |
| 960 | |
| 961 | | journal | record | action | |
| 962 | |---|---|---| |
| 963 | | any | the pass could not read every record, or the host is protected | nothing this pass (handled) | |
| 964 | | `cleanup-pending` | any | resume the rollback at step 6 (handled) | |
| 965 | | `rolling-back` | any | resume the rollback at step 1's check, then 2 (handled) | |
| 966 | | `swapped` | gone, no seed, another seed `ID`, or this seed `aborted` | roll back (handled) | |
| 967 | | `swapped` | this seed `ready` | redo the CAS to `swapped` (handled) | |
| 968 | | `swapped` | this seed `committed` | `CommitImport` (not handled: the loop converges the row) | |
| 969 | | `swapped` | this seed `swapped` or `placed` | nothing (not handled: ordinary handling) | |
| 970 | |
| 971 | `journaled` is every journal host; the removal phase skips them (the journal owns the row until it is gone). If `ImportJournals` fails, `ok` is false: every import seed is skipped this pass (`seedAction` returns `seedSkip` for an import when the journal is unknown), so the no-journal finish (Plan 4) can never run while a journal row might exist. |
| 972 | |
| 973 | - [ ] **Step 1: Write the failing tests** |
| 974 | |
| 975 | ```go |
| 976 | package provision |
| 977 | |
| 978 | import ( |
| 979 | "testing" |
| 980 | |
| 981 | "github.com/carlosframework/platform/internal/registry" |
| 982 | "github.com/carlosframework/platform/internal/seeds" |
| 983 | ) |
| 984 | |
| 985 | func swappedImport(t *testing.T) (*registry.Registry, Record, ReconcileConfig, func() error) { |
| 986 | t.Helper() |
| 987 | s, sqid := setup(t) |
| 988 | rec := importRecord(t, s, sqid) |
| 989 | reg := regWith(t, addrRow(rec, sqid)) |
| 990 | cfg := importCfg(t, nil) |
| 991 | run := func() error { return Reconcile(ctx, s, reg, cfg) } |
| 992 | if err := run(); err != nil { |
| 993 | t.Fatal(err) |
| 994 | } |
| 995 | return reg, rec, cfg, run |
| 996 | } |
| 997 | |
| 998 | func TestJournalResumesTheRecordCAS(t *testing.T) { |
| 999 | s, sqid := setup(t) |
| 1000 | rec := importRecord(t, s, sqid) |
| 1001 | reg := regWith(t, addrRow(rec, sqid)) |
| 1002 | stored, _ := reg.Get(rec.Host) |
| 1003 | // The box crashed after the transaction and before the record CAS. |
| 1004 | next := routeFor(rec, "/data", "", true) |
| 1005 | next.ImportGate = rec.Seed.VerifyHash |
| 1006 | if err := reg.SwapForImport(registry.ImportJournal{Host: rec.Host, SeedID: rec.Seed.ID, Original: *stored}, next); err != nil { |
| 1007 | t.Fatal(err) |
| 1008 | } |
| 1009 | if err := Reconcile(ctx, s, reg, importCfg(t, nil)); err != nil { |
| 1010 | t.Fatal(err) |
| 1011 | } |
| 1012 | r, _ := GetRecord(ctx, s, rec.Account, rec.App, rec.Host) |
| 1013 | if r.Seed.State != seeds.StateSwapped { |
| 1014 | t.Fatalf("state = %s, want the CAS redone", r.Seed.State) |
| 1015 | } |
| 1016 | if got := mustGet(t, reg, rec.Host); got.Kind != "instance" { |
| 1017 | t.Fatalf("the swapped row was refused or changed: %+v", got) |
| 1018 | } |
| 1019 | } |
| 1020 | |
| 1021 | func TestCommitClearsTheGateAndTheJournal(t *testing.T) { |
| 1022 | s, sqid := setup(t) |
| 1023 | rec := importRecord(t, s, sqid) |
| 1024 | reg := regWith(t, addrRow(rec, sqid)) |
| 1025 | cfg := importCfg(t, nil) |
| 1026 | Reconcile(ctx, s, reg, cfg) |
| 1027 | for _, step := range []func(string) func(*Record) error{SeedPlace, SeedCommit} { |
| 1028 | if _, err := SetRecordSeed(ctx, s, rec.Account, rec.App, rec.Host, step(rec.Seed.ID)); err != nil { |
| 1029 | t.Fatal(err) |
| 1030 | } |
| 1031 | } |
| 1032 | if err := Reconcile(ctx, s, reg, cfg); err != nil { |
| 1033 | t.Fatal(err) |
| 1034 | } |
| 1035 | if got := mustGet(t, reg, rec.Host); got.ImportGate != "" { |
| 1036 | t.Fatalf("gate after commit = %q", got.ImportGate) |
| 1037 | } |
| 1038 | if j, _ := reg.ImportJournal(rec.Host); j != nil { |
| 1039 | t.Fatalf("journal after commit = %+v", j) |
| 1040 | } |
| 1041 | } |
| 1042 | |
| 1043 | ``` |
| 1044 | |
| 1045 | Task 5 adds `TestJournalDoesNotRollBackOnAReadFailure`, once the rollback exists to be wrongly started. |
| 1046 | |
| 1047 | - [ ] **Step 2: Run the tests to verify they fail** |
| 1048 | |
| 1049 | Run: `go test ./internal/provision/ -run 'TestJournalResumes|TestCommitClears' -count=1` |
| 1050 | Expected: FAIL: the CAS is never redone (the row now fails the predicates and the seed is refused), and the gate is never cleared. |
| 1051 | |
| 1052 | - [ ] **Step 3: Write `journalPass` and wire it** |
| 1053 | |
| 1054 | In `seedreconcile.go`: |
| 1055 | |
| 1056 | ```go |
| 1057 | // journalPass is spec §5.7: the box's import journal is read before the |
| 1058 | // bucket records, every pass, and each row drives its host. handled hosts |
| 1059 | // are skipped by the record loop; journaled hosts by the removal phase. |
| 1060 | func journalPass(ctx context.Context, s release.Store, reg *registry.Registry, cfg ReconcileConfig, |
| 1061 | recs map[string]Record, readFailed bool, protect map[string]bool) (handled, journaled map[string]bool, ok bool) { |
| 1062 | handled, journaled = map[string]bool{}, map[string]bool{} |
| 1063 | if cfg.Seeds == nil { |
| 1064 | return handled, journaled, true |
| 1065 | } |
| 1066 | rows, err := reg.ImportJournals() |
| 1067 | if err != nil { |
| 1068 | log.Printf("provision: import journal unreadable: %v (no import is acted on this pass)", err) |
| 1069 | return handled, journaled, false |
| 1070 | } |
| 1071 | for _, j := range rows { |
| 1072 | journaled[j.Host] = true |
| 1073 | rec, found := recs[j.Host] |
| 1074 | if !found && (readFailed || protect[j.Host]) { |
| 1075 | handled[j.Host] = true |
| 1076 | continue |
| 1077 | } |
| 1078 | switch j.State { |
| 1079 | case registry.JournalCleanupPending, registry.JournalRollingBack: |
| 1080 | rollbackImport(ctx, s, reg, cfg, j, recordOrNil(found, rec)) |
| 1081 | handled[j.Host] = true |
| 1082 | continue |
| 1083 | } |
| 1084 | var seed *SeedSource |
| 1085 | if found { |
| 1086 | seed = rec.Seed |
| 1087 | } |
| 1088 | switch { |
| 1089 | case seed == nil || seed.ID != j.SeedID || seed.State == seeds.StateAborted: |
| 1090 | rollbackImport(ctx, s, reg, cfg, j, recordOrNil(found, rec)) |
| 1091 | handled[j.Host] = true |
| 1092 | case seed.State == seeds.StateReady: |
| 1093 | if _, err := SetRecordSeed(ctx, s, rec.Account, rec.App, rec.Host, SeedSwapped(seed.ID)); err != nil { |
| 1094 | log.Printf("provision: %s: redo the swap's record CAS: %v", j.Host, err) |
| 1095 | } |
| 1096 | handled[j.Host] = true |
| 1097 | case seed.State == seeds.StateCommitted: |
| 1098 | if cfg.DryRun { |
| 1099 | log.Printf("provision: dry-run: would clear %s's import gate", j.Host) |
| 1100 | } else if err := reg.CommitImport(j.Host, j.SeedID); err != nil { |
| 1101 | log.Printf("provision: %s: clear the import gate: %v", j.Host, err) |
| 1102 | } |
| 1103 | } |
| 1104 | } |
| 1105 | return handled, journaled, true |
| 1106 | } |
| 1107 | |
| 1108 | func recordOrNil(found bool, rec Record) *Record { |
| 1109 | if !found { |
| 1110 | return nil |
| 1111 | } |
| 1112 | return &rec |
| 1113 | } |
| 1114 | |
| 1115 | // rollbackImport is Task 5's. Until then, a stub that logs. |
| 1116 | func rollbackImport(ctx context.Context, s release.Store, reg *registry.Registry, cfg ReconcileConfig, j registry.ImportJournal, rec *Record) { |
| 1117 | log.Printf("provision: %s: import %s needs a rollback (not built yet)", j.Host, j.SeedID) |
| 1118 | } |
| 1119 | ``` |
| 1120 | |
| 1121 | In `Reconcile`, after `byHost` is built and `mintLayout` is created: |
| 1122 | |
| 1123 | ```go |
| 1124 | journalHandled, journaled, journalOK := journalPass(ctx, s, reg, cfg, recs, readFailed, protect) |
| 1125 | ``` |
| 1126 | |
| 1127 | In the record loop, before the seed dispatch: `if journalHandled[rec.Host] { continue }`, and pass `journalOK` into `seedAction` (signature gains `journalOK bool`; an import seed with `!journalOK` returns `seedSkip`; `aborted` with a journal row never reaches `seedAction` because it is handled). In the removal phase, before the `protect` check: `if journaled[rt.Host] { continue }`. |
| 1128 | |
| 1129 | A journal row whose route was minted afterwards in the same pass (the ordinary owned branch): it is the swapped exec row, `Provisioned` and filed under the record's account, so the owned branch converges its columns and leaves `import_gate` alone. |
| 1130 | |
| 1131 | - [ ] **Step 4: Run the tests to verify they pass** |
| 1132 | |
| 1133 | Run: `go test ./internal/provision/ -run 'TestJournal|TestCommitClears|TestSwap|TestReconcile' -count=1` |
| 1134 | Expected: PASS. |
| 1135 | |
| 1136 | - [ ] **Step 5: Commit** |
| 1137 | |
| 1138 | ```bash |
| 1139 | git add internal/provision/ |
| 1140 | git commit -m "provision: the import journal drives its hosts before the records (Plan 7 Task 4) |
| 1141 | |
| 1142 | Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>" |
| 1143 | ``` |
| 1144 | |
| 1145 | ### Task 5: The rollback |
| 1146 | |
| 1147 | **Files:** |
| 1148 | - Create: `internal/provision/seedrollback.go` (replaces Task 4's stub), `internal/provision/seedrollback_test.go` |
| 1149 | - Modify: `internal/provision/seedreconcile.go` (`SeedHooks` gains `FenceForRollback`, `LiftFence`, `RemoveLocal`, `DeleteReplica`, `NextFenceGen`) |
| 1150 | |
| 1151 | **Interfaces:** |
| 1152 | - Consumes: `reg.CloseImportGate`, `reg.RestoreImportRoute`, `reg.DeleteImportJournal`, `cfg.Seeds.Data`, `SetRecordSeed`, `SeedRolledBack`. |
| 1153 | - Produces: `SeedHooks.FenceForRollback func(ctx context.Context, host string, gen uint64) (proceed bool, err error)` (the layout pass's rule, implemented by the host in Task 7); `SeedHooks.LiftFence func(host string, gen uint64)`; `SeedHooks.RemoveLocal func(rt registry.Route) error`; `SeedHooks.DeleteReplica func(ctx context.Context, host string) error`; `SeedHooks.NextFenceGen func() uint64`; `rollbackImport(...)`. |
| 1154 | |
| 1155 | The steps (spec §5.6), each idempotent, resumed by the journal's state: |
| 1156 | |
| 1157 | 1. `swapped`: `CloseImportGate(host, id, NextFenceGen())`. The journal is now `rolling-back` with that generation. |
| 1158 | 2. `rolling-back`: `FenceForRollback(host, j.FenceGen)`; not proceeding: defer (the hook lifted any fence it installed). Proceeding: |
| 1159 | 3. `RemoveLocal(current row)`: the database, `-wal`, `-shm`, `<db>.seed`, litestream's metadata. |
| 1160 | 4. `RestoreImportRoute(host, id)`: journal `cleanup-pending`. |
| 1161 | 5. `LiftFence(host, j.FenceGen)`. |
| 1162 | 6. `cleanup-pending`: delete `seeds/<host>/<id>/snapshot.db` (this `ID` only) and `DeleteReplica(host)` (the host's `instances/` replica, which is this box's to delete while the journal row exists). Never fences or stops anything: the route is the legacy row and its process is the app's own. |
| 1163 | 7. `DeleteImportJournal(host, id)`. |
| 1164 | 8. When the record still names this `ID`: CAS to `rolled-back` (a conflict or a missing record is done). |
| 1165 | |
| 1166 | - [ ] **Step 1: Write the failing tests** |
| 1167 | |
| 1168 | ```go |
| 1169 | package provision |
| 1170 | |
| 1171 | import ( |
| 1172 | "context" |
| 1173 | "errors" |
| 1174 | "os" |
| 1175 | "reflect" |
| 1176 | "testing" |
| 1177 | |
| 1178 | "github.com/carlosframework/platform/internal/registry" |
| 1179 | "github.com/carlosframework/platform/internal/seeds" |
| 1180 | ) |
| 1181 | |
| 1182 | type rollbackRecorder struct { |
| 1183 | steps []string |
| 1184 | proceed bool |
| 1185 | fenceErr error |
| 1186 | gen uint64 |
| 1187 | } |
| 1188 | |
| 1189 | func (r *rollbackRecorder) hooks(h *SeedHooks) { |
| 1190 | h.NextFenceGen = func() uint64 { r.gen++; return r.gen } |
| 1191 | h.FenceForRollback = func(_ context.Context, host string, gen uint64) (bool, error) { |
| 1192 | r.steps = append(r.steps, "fence") |
| 1193 | return r.proceed, r.fenceErr |
| 1194 | } |
| 1195 | h.LiftFence = func(host string, gen uint64) { r.steps = append(r.steps, "lift") } |
| 1196 | h.RemoveLocal = func(rt registry.Route) error { r.steps = append(r.steps, "remove-local:"+rt.Kind); return nil } |
| 1197 | h.DeleteReplica = func(_ context.Context, host string) error { r.steps = append(r.steps, "delete-replica"); return nil } |
| 1198 | } |
| 1199 | |
| 1200 | func TestRollbackAfterAbortRestoresTheRowExactly(t *testing.T) { |
| 1201 | s, sqid := setup(t) |
| 1202 | rec := importRecord(t, s, sqid) |
| 1203 | reg := regWith(t, addrRow(rec, sqid)) |
| 1204 | original, _ := reg.Get(rec.Host) |
| 1205 | cfg := importCfg(t, nil) |
| 1206 | rr := &rollbackRecorder{proceed: true} |
| 1207 | rr.hooks(cfg.Seeds) |
| 1208 | cfg.Seeds.Data.Put(ctx, seeds.SnapshotKey(rec.Host, rec.Seed.ID), []byte("db"), "") |
| 1209 | other, _ := seeds.NewID() |
| 1210 | cfg.Seeds.Data.Put(ctx, seeds.SnapshotKey(rec.Host, other), []byte("db"), "") |
| 1211 | Reconcile(ctx, s, reg, cfg) // swap |
| 1212 | if _, err := SetRecordSeed(ctx, s, rec.Account, rec.App, rec.Host, SeedAbort(rec.Seed.ID)); err != nil { |
| 1213 | t.Fatal(err) |
| 1214 | } |
| 1215 | if err := Reconcile(ctx, s, reg, cfg); err != nil { |
| 1216 | t.Fatal(err) |
| 1217 | } |
| 1218 | if got := mustGet(t, reg, rec.Host); !reflect.DeepEqual(*got, *original) { |
| 1219 | t.Fatalf("restored %+v, want %+v", got, original) |
| 1220 | } |
| 1221 | if j, _ := reg.ImportJournal(rec.Host); j != nil { |
| 1222 | t.Fatalf("journal = %+v", j) |
| 1223 | } |
| 1224 | want := []string{"fence", "remove-local:instance", "lift", "delete-replica"} |
| 1225 | if !reflect.DeepEqual(rr.steps, want) { |
| 1226 | t.Fatalf("steps = %q, want %q", rr.steps, want) |
| 1227 | } |
| 1228 | if _, _, err := cfg.Seeds.Data.Get(ctx, seeds.SnapshotKey(rec.Host, rec.Seed.ID)); !errors.Is(err, os.ErrNotExist) { |
| 1229 | t.Fatal("this seed's snapshot survived") |
| 1230 | } |
| 1231 | if _, _, err := cfg.Seeds.Data.Get(ctx, seeds.SnapshotKey(rec.Host, other)); err != nil { |
| 1232 | t.Fatal("another seed's snapshot was deleted") |
| 1233 | } |
| 1234 | r, _ := GetRecord(ctx, s, rec.Account, rec.App, rec.Host) |
| 1235 | if r.Seed.State != seeds.StateRolledBack { |
| 1236 | t.Fatalf("record = %s", r.Seed.State) |
| 1237 | } |
| 1238 | } |
| 1239 | |
| 1240 | func TestRollbackForAGoneOrReplacedRecord(t *testing.T) { |
| 1241 | for _, mutate := range []string{"deleted", "another seed"} { |
| 1242 | t.Run(mutate, func(t *testing.T) { |
| 1243 | s, sqid := setup(t) |
| 1244 | rec := importRecord(t, s, sqid) |
| 1245 | reg := regWith(t, addrRow(rec, sqid)) |
| 1246 | original, _ := reg.Get(rec.Host) |
| 1247 | cfg := importCfg(t, nil) |
| 1248 | (&rollbackRecorder{proceed: true}).hooks(cfg.Seeds) |
| 1249 | Reconcile(ctx, s, reg, cfg) |
| 1250 | if err := DeleteRecord(ctx, s, rec.Account, rec.App, rec.Host); err != nil { |
| 1251 | t.Fatal(err) |
| 1252 | } |
| 1253 | if mutate == "another seed" { |
| 1254 | importRecord(t, s, sqid) // the same host, a new seed id, state ready |
| 1255 | } |
| 1256 | if err := Reconcile(ctx, s, reg, cfg); err != nil { |
| 1257 | t.Fatal(err) |
| 1258 | } |
| 1259 | if got := mustGet(t, reg, rec.Host); !reflect.DeepEqual(*got, *original) { |
| 1260 | t.Fatalf("restored %+v", got) |
| 1261 | } |
| 1262 | }) |
| 1263 | } |
| 1264 | } |
| 1265 | |
| 1266 | func TestRollbackDefersWhenTheFenceIsNotOurs(t *testing.T) { |
| 1267 | s, sqid := setup(t) |
| 1268 | rec := importRecord(t, s, sqid) |
| 1269 | reg := regWith(t, addrRow(rec, sqid)) |
| 1270 | cfg := importCfg(t, nil) |
| 1271 | rr := &rollbackRecorder{proceed: false} |
| 1272 | rr.hooks(cfg.Seeds) |
| 1273 | Reconcile(ctx, s, reg, cfg) |
| 1274 | SetRecordSeed(ctx, s, rec.Account, rec.App, rec.Host, SeedAbort(rec.Seed.ID)) |
| 1275 | Reconcile(ctx, s, reg, cfg) |
| 1276 | j, _ := reg.ImportJournal(rec.Host) |
| 1277 | got := mustGet(t, reg, rec.Host) |
| 1278 | if j == nil || j.State != registry.JournalRollingBack || got.ImportGate != registry.ImportGateClosed || got.Kind != "instance" { |
| 1279 | t.Fatalf("after a deferred fence: journal %+v row %+v; want rolling-back, gate closed, row kept", j, got) |
| 1280 | } |
| 1281 | // Next pass the fence proceeds, with the SAME generation. |
| 1282 | gen := j.FenceGen |
| 1283 | rr.proceed = true |
| 1284 | Reconcile(ctx, s, reg, cfg) |
| 1285 | if rr.gen != gen { |
| 1286 | t.Fatalf("a resumed rollback minted a new generation: %d, journal had %d", rr.gen, gen) |
| 1287 | } |
| 1288 | if j, _ := reg.ImportJournal(rec.Host); j != nil { |
| 1289 | t.Fatalf("journal = %+v", j) |
| 1290 | } |
| 1291 | } |
| 1292 | |
| 1293 | func TestCleanupPendingNeverFences(t *testing.T) { |
| 1294 | s, sqid := setup(t) |
| 1295 | rec := importRecord(t, s, sqid) |
| 1296 | reg := regWith(t, addrRow(rec, sqid)) |
| 1297 | cfg := importCfg(t, nil) |
| 1298 | rr := &rollbackRecorder{proceed: true} |
| 1299 | rr.hooks(cfg.Seeds) |
| 1300 | Reconcile(ctx, s, reg, cfg) |
| 1301 | SetRecordSeed(ctx, s, rec.Account, rec.App, rec.Host, SeedAbort(rec.Seed.ID)) |
| 1302 | // Crash after the route restore, before the bucket cleanup. |
| 1303 | if err := reg.CloseImportGate(rec.Host, rec.Seed.ID, 9); err != nil { |
| 1304 | t.Fatal(err) |
| 1305 | } |
| 1306 | if err := reg.RestoreImportRoute(rec.Host, rec.Seed.ID); err != nil { |
| 1307 | t.Fatal(err) |
| 1308 | } |
| 1309 | rr.steps = nil |
| 1310 | if err := Reconcile(ctx, s, reg, cfg); err != nil { |
| 1311 | t.Fatal(err) |
| 1312 | } |
| 1313 | if !reflect.DeepEqual(rr.steps, []string{"delete-replica"}) { |
| 1314 | t.Fatalf("steps = %q; a cleanup-pending resume fenced, stopped or removed something", rr.steps) |
| 1315 | } |
| 1316 | } |
| 1317 | |
| 1318 | func TestJournalDoesNotRollBackOnAReadFailure(t *testing.T) { |
| 1319 | s, sqid := setup(t) |
| 1320 | rec := importRecord(t, s, sqid) |
| 1321 | reg := regWith(t, addrRow(rec, sqid)) |
| 1322 | cfg := importCfg(t, nil) |
| 1323 | rr := &rollbackRecorder{proceed: true} |
| 1324 | rr.hooks(cfg.Seeds) |
| 1325 | Reconcile(ctx, s, reg, cfg) |
| 1326 | // The record body becomes unreadable: protected, never "gone". |
| 1327 | putReplace(t, s, RecordKey(rec.Account, rec.App, rec.Host), []byte("{not json")) |
| 1328 | if err := Reconcile(ctx, s, reg, cfg); err != nil { |
| 1329 | t.Fatal(err) |
| 1330 | } |
| 1331 | if len(rr.steps) != 0 || rr.gen != 0 { |
| 1332 | t.Fatalf("an unreadable record started a rollback: steps %q", rr.steps) |
| 1333 | } |
| 1334 | if j, _ := reg.ImportJournal(rec.Host); j == nil || j.State != registry.JournalSwapped { |
| 1335 | t.Fatalf("journal = %+v", j) |
| 1336 | } |
| 1337 | if got := mustGet(t, reg, rec.Host); got.Kind != "instance" { |
| 1338 | t.Fatal("the removal phase touched a journaled host") |
| 1339 | } |
| 1340 | } |
| 1341 | ``` |
| 1342 | |
| 1343 | - [ ] **Step 2: Run the tests to verify they fail** |
| 1344 | |
| 1345 | Run: `go test ./internal/provision/ -run 'TestRollback|TestCleanupPending|TestJournalDoesNotRollBack' -count=1` |
| 1346 | Expected: FAIL: the stub logs and nothing is restored (the read-failure test passes already; it guards Task 4's pass from here on). |
| 1347 | |
| 1348 | - [ ] **Step 3: Write `seedrollback.go`** (delete the stub from `seedreconcile.go`) |
| 1349 | |
| 1350 | ```go |
| 1351 | package provision |
| 1352 | |
| 1353 | import ( |
| 1354 | "context" |
| 1355 | "errors" |
| 1356 | "log" |
| 1357 | "os" |
| 1358 | |
| 1359 | "github.com/carlosframework/platform/internal/registry" |
| 1360 | "github.com/carlosframework/platform/internal/release" |
| 1361 | "github.com/carlosframework/platform/internal/seeds" |
| 1362 | ) |
| 1363 | |
| 1364 | // rollbackImport is spec §5.6's rollback, driven by the journal and not the |
| 1365 | // record (codex R2-7): each step idempotent, the journal's state the |
| 1366 | // resume point. rec is nil when the record is gone. |
| 1367 | func rollbackImport(ctx context.Context, s release.Store, reg *registry.Registry, cfg ReconcileConfig, j registry.ImportJournal, rec *Record) { |
| 1368 | h := cfg.Seeds |
| 1369 | if cfg.DryRun { |
| 1370 | log.Printf("provision: dry-run: would roll back import %s of %s (%s)", j.SeedID, j.Host, j.State) |
| 1371 | return |
| 1372 | } |
| 1373 | if h.FenceForRollback == nil || h.LiftFence == nil || h.RemoveLocal == nil || h.DeleteReplica == nil || h.NextFenceGen == nil { |
| 1374 | log.Printf("provision: %s: import %s cannot roll back on this box: the activator is not wired", j.Host, j.SeedID) |
| 1375 | return |
| 1376 | } |
| 1377 | if j.State == registry.JournalSwapped { |
| 1378 | gen := h.NextFenceGen() |
| 1379 | if err := reg.CloseImportGate(j.Host, j.SeedID, gen); err != nil { |
| 1380 | log.Printf("provision: %s: close the import gate: %v", j.Host, err) |
| 1381 | return |
| 1382 | } |
| 1383 | j.State, j.FenceGen = registry.JournalRollingBack, gen |
| 1384 | } |
| 1385 | if j.State == registry.JournalRollingBack { |
| 1386 | proceed, err := h.FenceForRollback(ctx, j.Host, j.FenceGen) |
| 1387 | if !proceed { |
| 1388 | log.Printf("provision: %s: import %s rollback deferred: the host is not fenced and stopped yet (%v)", j.Host, j.SeedID, err) |
| 1389 | return |
| 1390 | } |
| 1391 | row, err := reg.Get(j.Host) |
| 1392 | if err != nil { |
| 1393 | log.Printf("provision: %s: read the swapped row: %v", j.Host, err) |
| 1394 | return |
| 1395 | } |
| 1396 | if row != nil && row.Kind == "instance" && row.DBPath != "" && row.Socket != "" { |
| 1397 | if err := h.RemoveLocal(*row); err != nil { |
| 1398 | log.Printf("provision: %s: remove the import's local data: %v", j.Host, err) |
| 1399 | return |
| 1400 | } |
| 1401 | } |
| 1402 | if err := reg.RestoreImportRoute(j.Host, j.SeedID); err != nil { |
| 1403 | log.Printf("provision: %s: restore the original route: %v", j.Host, err) |
| 1404 | return |
| 1405 | } |
| 1406 | h.LiftFence(j.Host, j.FenceGen) |
| 1407 | j.State = registry.JournalCleanupPending |
| 1408 | log.Printf("provision: %s: import %s rolled back to its original route", j.Host, j.SeedID) |
| 1409 | } |
| 1410 | if j.State != registry.JournalCleanupPending { |
| 1411 | return |
| 1412 | } |
| 1413 | if err := h.Data.Delete(ctx, seeds.SnapshotKey(j.Host, j.SeedID)); err != nil && !errors.Is(err, os.ErrNotExist) { |
| 1414 | log.Printf("provision: %s: delete import %s's snapshot: %v", j.Host, j.SeedID, err) |
| 1415 | return |
| 1416 | } |
| 1417 | if err := h.DeleteReplica(ctx, j.Host); err != nil { |
| 1418 | log.Printf("provision: %s: delete the import's replica: %v", j.Host, err) |
| 1419 | return |
| 1420 | } |
| 1421 | if err := reg.DeleteImportJournal(j.Host, j.SeedID); err != nil { |
| 1422 | log.Printf("provision: %s: retire the import journal: %v", j.Host, err) |
| 1423 | return |
| 1424 | } |
| 1425 | if rec != nil && rec.Seed != nil && rec.Seed.ID == j.SeedID { |
| 1426 | if _, err := SetRecordSeed(ctx, s, rec.Account, rec.App, rec.Host, SeedRolledBack(j.SeedID)); err != nil && |
| 1427 | !errors.Is(err, ErrSeedConflict) && !errors.Is(err, os.ErrNotExist) { |
| 1428 | log.Printf("provision: %s: mark import %s rolled back: %v (the no-journal path finishes it)", j.Host, j.SeedID, err) |
| 1429 | } |
| 1430 | } |
| 1431 | } |
| 1432 | ``` |
| 1433 | |
| 1434 | A crash after step 7 leaves an `aborted` record and no journal row; Plan 4's `finishAbortedSeed` then deletes this seed's snapshot (already gone) and CASes `rolled-back`. A record whose seed is still `swapped` or `placed` but whose journal row was rolled back because the record named another `ID` keeps its own state; `SeedRolledBack` is only applied to a record naming this `ID`. |
| 1435 | |
| 1436 | - [ ] **Step 4: Run the tests to verify they pass** |
| 1437 | |
| 1438 | Run: `go test ./internal/provision/ -run 'TestRollback|TestCleanupPending|TestJournal|TestSwap|TestReconcile' -count=1` |
| 1439 | Expected: PASS. |
| 1440 | |
| 1441 | - [ ] **Step 5: Commit** |
| 1442 | |
| 1443 | ```bash |
| 1444 | git add internal/provision/ |
| 1445 | git commit -m "provision: the fenced, journal-driven import rollback (Plan 7 Task 5) |
| 1446 | |
| 1447 | Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>" |
| 1448 | ``` |
| 1449 | |
| 1450 | ### Task 6: The gate at the edge and in the schedule runner |
| 1451 | |
| 1452 | **Files:** |
| 1453 | - Create: `internal/edge/importgate.go`, `internal/edge/importgate_test.go` |
| 1454 | - Modify: `internal/edge/edge.go:1559-1562` (one call after the door-scope check), `internal/schedule/runner.go:196` (skip gated rows) |
| 1455 | - Test: `internal/schedule/runner_test.go` |
| 1456 | |
| 1457 | **Interfaces:** |
| 1458 | - Consumes: `seeds.TokenMatches`, `registry.Route.ImportGate`, `MarkAnswered` (`internal/edge/answered.go`). |
| 1459 | - Produces: `ImportVerifyHeader = "Carlos-Import-Verify"`; `(e *Edge) importGateAdmit(w http.ResponseWriter, r *http.Request, rt registry.Route) bool`. |
| 1460 | |
| 1461 | A row with `ImportGate` set: a request whose `Carlos-Import-Verify` hashes to it passes, with the header removed; every other request gets 503, `Retry-After: 5`, `Cache-Control: no-store` and SD-N99. Placed after the door's scope check and before the meter, the password gate, the cache and the activator, so nothing ordinary is served, cached or woken. The edge door re-enters `ServeHTTP` (`door.go:91`), so the same check covers it. The schedule runner skips a gated row in its pre-filter whether or not the hold wiring is armed. |
| 1462 | |
| 1463 | - [ ] **Step 1: Write the failing tests** |
| 1464 | |
| 1465 | `internal/edge/importgate_test.go`: |
| 1466 | |
| 1467 | ```go |
| 1468 | package edge |
| 1469 | |
| 1470 | import ( |
| 1471 | "net/http" |
| 1472 | "net/http/httptest" |
| 1473 | "testing" |
| 1474 | |
| 1475 | "github.com/carlosframework/platform/internal/registry" |
| 1476 | "github.com/carlosframework/platform/internal/seeds" |
| 1477 | ) |
| 1478 | |
| 1479 | func TestImportGateAtTheEdgeAndTheDoor(t *testing.T) { |
| 1480 | var sawHeader string |
| 1481 | backend := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { |
| 1482 | sawHeader = r.Header.Get(ImportVerifyHeader) |
| 1483 | w.Write([]byte("served")) |
| 1484 | })) |
| 1485 | defer backend.Close() |
| 1486 | token, hash, _ := seeds.NewVerifyToken() |
| 1487 | reg := openReg(t) |
| 1488 | reg.Upsert(registry.Route{Host: "old.example.com", Kind: "service", Addr: backend.Listener.Addr().String(), Status: "ready"}) |
| 1489 | // The gate is set by the swap transaction; for the edge test, set the |
| 1490 | // column directly through the registry's swap with an original row. |
| 1491 | stored, _ := reg.Get("old.example.com") |
| 1492 | next := *stored |
| 1493 | next.ImportGate = hash |
| 1494 | if err := reg.SwapForImport(registry.ImportJournal{Host: "old.example.com", SeedID: "sd_aaaaaaaaaaaaaaaaaaaaaaaaaa", Original: *stored}, next); err != nil { |
| 1495 | t.Fatal(err) |
| 1496 | } |
| 1497 | e, err := New(reg, nil) |
| 1498 | if err != nil { |
| 1499 | t.Fatal(err) |
| 1500 | } |
| 1501 | srv := httptest.NewServer(e) |
| 1502 | defer srv.Close() |
| 1503 | get := func(tok string) *http.Response { |
| 1504 | req, _ := http.NewRequest(http.MethodGet, srv.URL+"/up", nil) |
| 1505 | req.Host = "old.example.com" |
| 1506 | if tok != "" { |
| 1507 | req.Header.Set(ImportVerifyHeader, tok) |
| 1508 | } |
| 1509 | res, err := http.DefaultClient.Do(req) |
| 1510 | if err != nil { |
| 1511 | t.Fatal(err) |
| 1512 | } |
| 1513 | return res |
| 1514 | } |
| 1515 | for _, tok := range []string{"", "wrong"} { |
| 1516 | res := get(tok) |
| 1517 | if res.StatusCode != http.StatusServiceUnavailable || res.Header.Get("Retry-After") != "5" { |
| 1518 | t.Fatalf("token %q = %d retry %q, want 503 and 5", tok, res.StatusCode, res.Header.Get("Retry-After")) |
| 1519 | } |
| 1520 | } |
| 1521 | res := get(token) |
| 1522 | if res.StatusCode != http.StatusOK || sawHeader != "" { |
| 1523 | t.Fatalf("verify = %d, header at the app %q; want served without the header", res.StatusCode, sawHeader) |
| 1524 | } |
| 1525 | // Closed during a rollback: even the token is refused. |
| 1526 | if err := reg.CloseImportGate("old.example.com", "sd_aaaaaaaaaaaaaaaaaaaaaaaaaa", 1); err != nil { |
| 1527 | t.Fatal(err) |
| 1528 | } |
| 1529 | if err := e.Reload(); err != nil { |
| 1530 | t.Fatal(err) |
| 1531 | } |
| 1532 | if res := get(token); res.StatusCode != http.StatusServiceUnavailable { |
| 1533 | t.Fatalf("a closed gate let the token through: %d", res.StatusCode) |
| 1534 | } |
| 1535 | } |
| 1536 | ``` |
| 1537 | |
| 1538 | The edge door path goes through the same `ServeHTTP` (`door.go:91`); add a case to `door_test.go` with `newDoorScene(t)` that gates one of its routes the same way and asserts `wantUnknownHost` is not what comes back but the 503 (the door caller passes the scope check, then meets the gate). |
| 1539 | |
| 1540 | In `internal/schedule/runner_test.go`, beside the hold tests (`runnerFixture` `:69`, `holdHost` `:984`, `deliverFunc` `:1100`): |
| 1541 | |
| 1542 | ```go |
| 1543 | func TestTickSkipsGatedRows(t *testing.T) { |
| 1544 | reg, r := runnerFixture(t, nil) |
| 1545 | now := at("2026-08-23T13:07:00Z") |
| 1546 | r.Now = func() time.Time { return now } |
| 1547 | delivered := 0 |
| 1548 | r.Deliver = deliverFunc(func(registry.Route, *http.Request) (int, error) { delivered++; return 204, nil }) |
| 1549 | stored, _ := reg.Get(holdHost) |
| 1550 | next := *stored |
| 1551 | next.ImportGate = strings.Repeat("a", 64) |
| 1552 | if err := reg.SwapForImport(registry.ImportJournal{Host: holdHost, SeedID: "sd_aaaaaaaaaaaaaaaaaaaaaaaaaa", Original: *stored}, next); err != nil { |
| 1553 | t.Fatal(err) |
| 1554 | } |
| 1555 | reg.UpsertTimer(registry.Timer{Host: holdHost, Name: "remind", Kind: KindOnce, Spec: "at 1", Path: "/r", NextAt: 1}) |
| 1556 | r.tick() // r.Gate is nil: the hold wiring is off, and a gated row is still skipped |
| 1557 | if delivered != 0 { |
| 1558 | t.Fatalf("delivered %d ticks to a gated row", delivered) |
| 1559 | } |
| 1560 | if err := reg.CommitImport(holdHost, "sd_aaaaaaaaaaaaaaaaaaaaaaaaaa"); err != nil { |
| 1561 | t.Fatal(err) |
| 1562 | } |
| 1563 | r.tick() |
| 1564 | if delivered != 1 { |
| 1565 | t.Fatalf("after commit delivered %d, want 1", delivered) |
| 1566 | } |
| 1567 | } |
| 1568 | ``` |
| 1569 | |
| 1570 | (add `strings` to the file's imports if it lacks it). |
| 1571 | |
| 1572 | - [ ] **Step 2: Run the tests to verify they fail** |
| 1573 | |
| 1574 | Run: `go test ./internal/edge/ -run 'TestImportGate' -count=1; go test ./internal/schedule/ -run 'TestTickSkipsGatedRows' -count=1` |
| 1575 | Expected: FAIL. |
| 1576 | |
| 1577 | - [ ] **Step 3: Write the gate and the two calls** |
| 1578 | |
| 1579 | `internal/edge/importgate.go`: |
| 1580 | |
| 1581 | ```go |
| 1582 | package edge |
| 1583 | |
| 1584 | import ( |
| 1585 | "net/http" |
| 1586 | |
| 1587 | "github.com/carlosframework/platform/internal/registry" |
| 1588 | "github.com/carlosframework/platform/internal/seeds" |
| 1589 | ) |
| 1590 | |
| 1591 | // ImportVerifyHeader carries an uncommitted import's verify token |
| 1592 | // (seeded-instances spec §5.4). |
| 1593 | const ImportVerifyHeader = "Carlos-Import-Verify" |
| 1594 | |
| 1595 | // importGateAdmit is the import gate (spec §5.4, codex 15). A row without a |
| 1596 | // gate passes. A gated row passes only a request whose verify token hashes |
| 1597 | // to the gate, and that request reaches the app without the header. Every |
| 1598 | // other request is answered 503 with Retry-After: 5, before the meter, the |
| 1599 | // password gate, the cache and the activator: nothing ordinary is served, |
| 1600 | // cached or woken until commit. |
| 1601 | func (e *Edge) importGateAdmit(w http.ResponseWriter, r *http.Request, rt registry.Route) bool { |
| 1602 | if rt.ImportGate == "" { |
| 1603 | return true |
| 1604 | } |
| 1605 | if seeds.TokenMatches(rt.ImportGate, r.Header.Get(ImportVerifyHeader)) { |
| 1606 | r.Header.Del(ImportVerifyHeader) |
| 1607 | return true |
| 1608 | } |
| 1609 | w.Header().Set("Retry-After", "5") |
| 1610 | w.Header().Set("Cache-Control", "no-store") |
| 1611 | http.Error(w, "this instance is being imported. Try again in a few seconds.", http.StatusServiceUnavailable) |
| 1612 | return false |
| 1613 | } |
| 1614 | ``` |
| 1615 | |
| 1616 | In `edge.go`, right after the door-scope check (`if isDoor && !doorOwnsRoute(door, up.route) { … }`): |
| 1617 | |
| 1618 | ```go |
| 1619 | if !e.importGateAdmit(w, r, up.route) { |
| 1620 | return |
| 1621 | } |
| 1622 | ``` |
| 1623 | |
| 1624 | In `runner.go` `Tick`, before the `if r.holdsArmed() && registry.IsHold(rt.Status)` pre-filter: |
| 1625 | |
| 1626 | ```go |
| 1627 | // An uncommitted import's row gets no ticks (seeded-instances spec |
| 1628 | // §5.4): only verify requests reach it until commit. |
| 1629 | if rt.ImportGate != "" { |
| 1630 | continue |
| 1631 | } |
| 1632 | ``` |
| 1633 | |
| 1634 | - [ ] **Step 4: Run the tests to verify they pass** |
| 1635 | |
| 1636 | Run: `go test ./internal/edge/ -run 'TestImportGate|TestDoor|TestGate' -count=1 && go test ./internal/schedule/ -count=1` |
| 1637 | Expected: PASS. |
| 1638 | |
| 1639 | - [ ] **Step 5: Wire SD-N99 and commit** |
| 1640 | |
| 1641 | ```bash |
| 1642 | git add internal/edge/importgate.go internal/edge/importgate_test.go internal/edge/edge.go internal/edge/door_test.go internal/schedule/ approved_seeded_copy_test.go |
| 1643 | git commit -m "edge and schedule: the import gate (Plan 7 Task 6) |
| 1644 | |
| 1645 | Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>" |
| 1646 | ``` |
| 1647 | |
| 1648 | ### Task 7: The box's wiring, and the protocol raised |
| 1649 | |
| 1650 | **Files:** |
| 1651 | - Modify: `internal/host/seedwiring.go` (`fenceForRollback`, `removeSeedLocal`, `deleteReplica`, `seedSweptAtStart`), `internal/host/agent.go` (the arming of Plan 4 Task 6 gains the new hooks; the `recoverLayout` closure records `swept`) |
| 1652 | - Modify: `internal/seeds/floor.go` (`ProtocolVersion = 1`), `internal/seeds/floor_test.go` |
| 1653 | - Test: `internal/host/seedwiring_test.go`, `internal/seeds/seeds_test.go` (the gate constant) |
| 1654 | |
| 1655 | **Interfaces:** |
| 1656 | - Consumes: `activator.Service`'s `FenceAndStopIfUnfenced`, `LiftFence`, `FenceGeneration`, `Quiesced`, `StopUncertain`; `activator.ErrFenceHeld`, `activator.ErrStopUnproven`; `activator.PurgeLocalDBArtifacts`, `activator.SeedMarkerSuffix`; `instancefs.RemoveBeneath`; `eb.NestedOK`; `eb.Cfg.hibernatePrefix` (unexported: use `activator.HibernatePrefix(cfg.Prefix, host)`, `activator.go:141`). |
| 1657 | - Produces: `type rollbackFence interface{ FenceAndStopIfUnfenced(ctx context.Context, host string, generation uint64) (*activator.StopEvidence, error); LiftFence(host string, generation uint64) bool; FenceGeneration(host string) uint64; Quiesced(host string) bool; StopUncertain(host string) bool }`; `fenceForRollback(f rollbackFence, swept func() bool) func(ctx context.Context, host string, gen uint64) (bool, error)`; `var seedSweptAtStart atomic.Bool`. |
| 1658 | |
| 1659 | `fenceForRollback` is `internal/layout/pass.go:654-680`'s rule (spec §5.6 step 2): our fence already (by generation): proceed; otherwise `FenceAndStopIfUnfenced` under a 30-second deadline; nil: proceed; `ErrFenceHeld`: defer, touching nothing; `ErrStopUnproven` with the box swept at start, the host quiesced and its stop not uncertain: proceed; anything else: lift this attempt's fence and defer. |
| 1660 | |
| 1661 | - [ ] **Step 1: Write the failing tests** |
| 1662 | |
| 1663 | Append to `internal/host/seedwiring_test.go`: |
| 1664 | |
| 1665 | ```go |
| 1666 | type fakeFence struct { |
| 1667 | gens map[string]uint64 |
| 1668 | err error |
| 1669 | quiesced bool |
| 1670 | uncertain bool |
| 1671 | lifted []uint64 |
| 1672 | } |
| 1673 | |
| 1674 | func (f *fakeFence) FenceAndStopIfUnfenced(_ context.Context, host string, gen uint64) (*activator.StopEvidence, error) { |
| 1675 | if f.err == nil || errors.Is(f.err, activator.ErrStopUnproven) { |
| 1676 | f.gens[host] = gen |
| 1677 | } |
| 1678 | return nil, f.err |
| 1679 | } |
| 1680 | func (f *fakeFence) LiftFence(host string, gen uint64) bool { |
| 1681 | f.lifted = append(f.lifted, gen) |
| 1682 | delete(f.gens, host) |
| 1683 | return true |
| 1684 | } |
| 1685 | func (f *fakeFence) FenceGeneration(host string) uint64 { return f.gens[host] } |
| 1686 | func (f *fakeFence) Quiesced(string) bool { return f.quiesced } |
| 1687 | func (f *fakeFence) StopUncertain(string) bool { return f.uncertain } |
| 1688 | |
| 1689 | func TestRollbackFenceAfterRestart(t *testing.T) { |
| 1690 | ctx := context.Background() |
| 1691 | swept := true |
| 1692 | f := &fakeFence{gens: map[string]uint64{}, err: activator.ErrStopUnproven, quiesced: true} |
| 1693 | fence := fenceForRollback(f, func() bool { return swept }) |
| 1694 | if ok, _ := fence(ctx, "h", 7); !ok { |
| 1695 | t.Fatal("cold, swept, quiesced: want proceed") |
| 1696 | } |
| 1697 | // Ours already (an earlier pass installed it): proceed without asking. |
| 1698 | f.err = errors.New("must not be called") |
| 1699 | if ok, _ := fence(ctx, "h", 7); !ok { |
| 1700 | t.Fatal("our own fence: want proceed") |
| 1701 | } |
| 1702 | // Not swept at start: lift and defer. |
| 1703 | f = &fakeFence{gens: map[string]uint64{}, err: activator.ErrStopUnproven, quiesced: true} |
| 1704 | swept = false |
| 1705 | fence = fenceForRollback(f, func() bool { return swept }) |
| 1706 | if ok, _ := fence(ctx, "h", 8); ok || len(f.lifted) != 1 || f.lifted[0] != 8 { |
| 1707 | t.Fatalf("not swept: proceed %v lifted %v", ok, f.lifted) |
| 1708 | } |
| 1709 | // Another operation's fence: defer and touch nothing. |
| 1710 | f = &fakeFence{gens: map[string]uint64{}, err: activator.ErrFenceHeld} |
| 1711 | fence = fenceForRollback(f, func() bool { return true }) |
| 1712 | if ok, _ := fence(ctx, "h", 9); ok || len(f.lifted) != 0 { |
| 1713 | t.Fatalf("held: proceed %v lifted %v", ok, f.lifted) |
| 1714 | } |
| 1715 | } |
| 1716 | ``` |
| 1717 | |
| 1718 | `internal/seeds/seeds_test.go`: `if ClosedGate != registry.ImportGateClosed { t.Fatal(…) }` (seeds may import registry in a test only if registry does not import seeds; it does not). |
| 1719 | |
| 1720 | `internal/seeds/floor_test.go`: replace `TestProtocolStartsBelowWhatTheConsoleRequires` with `TestProtocolMeetsWhatTheConsoleRequires` asserting `ProtocolVersion >= RequiredProtocol`. |
| 1721 | |
| 1722 | - [ ] **Step 2: Run the tests to verify they fail** |
| 1723 | |
| 1724 | Run: `go test ./internal/host/ -run TestRollbackFenceAfterRestart -count=1; go test ./internal/seeds/ -count=1` |
| 1725 | Expected: FAIL. |
| 1726 | |
| 1727 | - [ ] **Step 3: Write the hooks and arm them** |
| 1728 | |
| 1729 | Append to `seedwiring.go`: |
| 1730 | |
| 1731 | ```go |
| 1732 | // seedSweptAtStart is whether the startup unit sweep succeeded, for the |
| 1733 | // import rollback's stop proof (the layout pass reads the same fact). |
| 1734 | var seedSweptAtStart atomic.Bool |
| 1735 | |
| 1736 | type rollbackFence interface { |
| 1737 | FenceAndStopIfUnfenced(ctx context.Context, host string, generation uint64) (*activator.StopEvidence, error) |
| 1738 | LiftFence(host string, generation uint64) bool |
| 1739 | FenceGeneration(host string) uint64 |
| 1740 | Quiesced(host string) bool |
| 1741 | StopUncertain(host string) bool |
| 1742 | } |
| 1743 | |
| 1744 | // fenceForRollback is the import rollback's step 2 (seeded-instances spec |
| 1745 | // §5.6), internal/layout/pass.go's rule for the same API. |
| 1746 | func fenceForRollback(f rollbackFence, swept func() bool) func(ctx context.Context, host string, gen uint64) (bool, error) { |
| 1747 | return func(ctx context.Context, host string, gen uint64) (bool, error) { |
| 1748 | if f.FenceGeneration(host) == gen { |
| 1749 | return true, nil |
| 1750 | } |
| 1751 | fctx, cancel := context.WithTimeout(ctx, 30*time.Second) |
| 1752 | defer cancel() |
| 1753 | _, err := f.FenceAndStopIfUnfenced(fctx, host, gen) |
| 1754 | switch { |
| 1755 | case err == nil: |
| 1756 | return true, nil |
| 1757 | case errors.Is(err, activator.ErrFenceHeld): |
| 1758 | return false, err |
| 1759 | case errors.Is(err, activator.ErrStopUnproven) && swept() && f.Quiesced(host) && !f.StopUncertain(host): |
| 1760 | return true, nil |
| 1761 | } |
| 1762 | f.LiftFence(host, gen) |
| 1763 | return false, err |
| 1764 | } |
| 1765 | } |
| 1766 | |
| 1767 | // removeSeedLocal is the rollback's step 3: the database, its WAL and SHM, |
| 1768 | // the seed marker and litestream's metadata. The files go through |
| 1769 | // instancefs; litestream's metadata directories through |
| 1770 | // PurgeLocalDBArtifacts (instancefs removes files only). |
| 1771 | func removeSeedLocal(rt registry.Route) error { |
| 1772 | dir, base := filepath.Dir(rt.DBPath), filepath.Base(rt.DBPath) |
| 1773 | for _, name := range []string{base, base + "-wal", base + "-shm", base + activator.SeedMarkerSuffix} { |
| 1774 | if err := instancefs.RemoveBeneath(dir, name); err != nil && !errors.Is(err, os.ErrNotExist) { |
| 1775 | return err |
| 1776 | } |
| 1777 | } |
| 1778 | activator.PurgeLocalDBArtifacts(rt.DBPath, false) |
| 1779 | return nil |
| 1780 | } |
| 1781 | |
| 1782 | // deleteReplica is the rollback's step 6 for the host's instances/ replica. |
| 1783 | func deleteReplica(data release.Store, prefix string) func(ctx context.Context, host string) error { |
| 1784 | return func(ctx context.Context, host string) error { |
| 1785 | keys, err := data.List(ctx, activator.HibernatePrefix(prefix, host)+"/") |
| 1786 | if err != nil { |
| 1787 | return err |
| 1788 | } |
| 1789 | for _, k := range keys { |
| 1790 | if err := data.Delete(ctx, k); err != nil && !errors.Is(err, os.ErrNotExist) { |
| 1791 | return err |
| 1792 | } |
| 1793 | } |
| 1794 | return nil |
| 1795 | } |
| 1796 | } |
| 1797 | ``` |
| 1798 | |
| 1799 | (imports: `path/filepath`, `time`, `internal/activator`, `internal/instancefs`, `internal/registry`.) `instancefs` is Linux-only at run time; `removeSeedLocal` compiles everywhere and returns `errors.ErrUnsupported` off Linux, where no box runs. |
| 1800 | |
| 1801 | In `agent.go`, in Plan 4's seed arming block, replace `armSeedHooks(&provision.SeedHooks{Data: data})` with: |
| 1802 | |
| 1803 | ```go |
| 1804 | armSeedHooks(&provision.SeedHooks{ |
| 1805 | Data: data, |
| 1806 | NestedOK: eb.NestedOK, |
| 1807 | LegacyOwners: func(ctx context.Context) (*seeds.LegacyOwners, error) { o, _, err := seeds.ReadLegacyOwners(ctx, deploy); return o, err }, |
| 1808 | FenceForRollback: fenceForRollback(svc, seedSweptAtStart.Load), |
| 1809 | LiftFence: func(host string, gen uint64) { svc.LiftFence(host, gen) }, |
| 1810 | RemoveLocal: removeSeedLocal, |
| 1811 | DeleteReplica: deleteReplica(data, cfg.Prefix), |
| 1812 | NextFenceGen: func() uint64 { return uint64(time.Now().UnixNano()) }, |
| 1813 | }) |
| 1814 | ``` |
| 1815 | |
| 1816 | In the `recoverLayout` closure passed to `startActivator` (`agent.go:427`), first line: `seedSweptAtStart.Store(swept)`. With claims armed the agent starts the activator without a sweep (`agent.go:418`); `seedSweptAtStart` stays false there, and the seed doors refuse claim-armed boxes anyway. |
| 1817 | |
| 1818 | `internal/seeds/floor.go`: `const ProtocolVersion = 1`, its comment: "1: this build places, swaps and rolls back a seed (Plans 4 and 7)." |
| 1819 | |
| 1820 | - [ ] **Step 4: Run the tests to verify they pass** |
| 1821 | |
| 1822 | Run: `go test ./internal/host/ -run 'TestRollbackFence|TestSeed|TestWithSeedsFields' -count=1 && go test ./internal/seeds/ -count=1 && go test ./internal/console/ -run 'TestSeedRegionReady|TestFrom|TestImport' -count=1` |
| 1823 | Expected: PASS. The console tests write their own edges files, so the raised protocol changes nothing there. |
| 1824 | |
| 1825 | - [ ] **Step 5: Commit** |
| 1826 | |
| 1827 | ```bash |
| 1828 | git add internal/host/ internal/seeds/ |
| 1829 | git commit -m "host: wire the import swap and rollback; seeds protocol 1 (Plan 7 Task 7) |
| 1830 | |
| 1831 | Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>" |
| 1832 | ``` |
| 1833 | |
| 1834 | ### Task 8: Plan gate |
| 1835 | |
| 1836 | - [ ] **Step 1: Cross-compile and run the touched packages under -race** |
| 1837 | |
| 1838 | Run: |
| 1839 | ```bash |
| 1840 | GOOS=darwin go build ./... && GOOS=windows go build ./... && GOOS=linux GOARCH=arm64 go build ./... && \ |
| 1841 | nice go test -race ./internal/registry/ ./internal/seeds/ ./internal/provision/ ./internal/edge/ ./internal/schedule/ ./internal/host/ . -count=1 |
| 1842 | ``` |
| 1843 | Expected: builds clean; tests PASS; the moves copy gate passes. |
| 1844 | |
| 1845 | --- |
| 1846 | |
| 1847 | ## Self-review |
| 1848 | |
| 1849 | - Spec coverage: §5.4's journal first (Task 4), shape with the `shapeMismatch` protected columns (Task 3), ownership with the assignment (Tasks 2, 3), layout with nested derived explicitly (Task 3), the swap transaction with the complete original row (Tasks 1, 3), crash resume (Task 4), the gate at the edge, the door and the runner (Task 6; `carlos run` is the run series'), commit's clear in one transaction (Tasks 1, 4); §5.6's rollback triggers (aborted, another ID, gone), steps 1 to 8 with the layout pass's fence handling, `cleanup-pending` resume that never fences (Task 5), record CAS last and the no-journal finish after a crash (Plan 4); §5.7 (Tasks 1, 4); §6.3 box half (Task 3); §9's reconcile and gate tests (Tasks 3 to 6). |
| 1850 | - Not built here: the claim CAS at the swap (claims are refused, §5.4), and multi-box regions (refused by the import door). |
| 1851 | - The registry's `ImportGateClosed` and `seeds.ClosedGate` are one value, pinned by Task 7's test. |
| 1852 | |