carlos / platform Public

Clone
git clone https://amadan.net/carlos/platform

Plain git — no account needed to clone.

Download

Download this file

Seeded Instances, Plan 4: Placing a Seed on First Wake Implementation Plan

> For agentic workers: REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (- [ ]) syntax for tracking.

Goal: On a nested box with claims off, an instance whose record carries a placeable seed starts from that seed's snapshot on its first wake, never from an empty database, and the seed becomes placed only after the host's own replica exists.

Architecture: Reconcile carries a record's seed onto its registry row as a new column, seed_source (a seeds.RowSeed), dispatches on the seed's state before any other per-record handling, mints a --from row only nested and only with claims off, never stamps the manifest's seed command on a seeded row, and finishes an aborted seed that no box swapped (delete this seed's snapshot, CAS to rolled-back). The activator's ExecBackend.Restore gains a fallback beside the staging fallback: for a row whose seed is ready, swapped or placed, it restores normally when the host has a replica of its own, keeps a local database only when its <db>.seed marker names this seed, and otherwise purges, fetches the snapshot into the instance directory, checks length, sha256 and CheckDBContext, writes the marker and renames the file into place. After replication starts, doWake waits for the first upload (Replicator.SyncAndWait) and the host package CASes the record to placed and deletes seeds/<host>/<id>/.

Tech Stack: Go 1.25, modernc sqlite registry, internal/instancefs (openat2, Linux), litestream v0.5.14 replica client (lease), release.S3Store.

Spec: docs/superpowers/specs/2026-10-08-seeded-instances-design.md rev 7: §3.2's dispatch table (codex 13) for every state except an import's ready (Plan 7), §3.4 the plain path (codex 8, R2-1), §6.3 the box half for placement. Index: 2026-10-08-seeded-instances-0-index.md. Copy deck rows SD-N14 to SD-N19.

Global Constraints

  • Every user-facing string (CLI output, errors, console messages, docs prose) is copy-reviewed by Paul before it is written into code; implementation tasks use only strings from the copy deck. Copy voice: no em dashes; full stops and new sentences instead.
  • CI cross-compiles darwin and windows: linux-only code needs //go:build linux plus a !linux stub.
  • Local test commands: targeted tests (go test ./internal/<pkg>/ -run TestX), not the full suite in a loop. Inside the sandbox the default Go build cache may be read-only; if so use GOCACHE=$TMPDIR/gocache re-exported per command. Socket tests need a short TMPDIR (TMPDIR=/tmp/g). The local gate may need GOFLAGS=-buildvcs=false.
  • Heavy loops run under nice; no -race full suite until the end of a plan.
  • Production use needs nested boxes; the claim-armed path is refused here and left to #250 (spec §3.4).
  • Commits end with Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>.
  • No new file in this series imports internal/moves. New sentences go in new files (internal/activator/seedplace.go), never in a file listed in movesScanFiles (backend_exec.go, activator.go's neighbours registry.go, reconcile.go, provision.go, lease.go, agent.go are listed).
  • When a task wires a deck row, append its id to seededWiredIDs in approved_seeded_copy_test.go in the same commit.
  • Write each deck sentence as exactly one string literal or one format string holding the row's text, with %s or %d at each <placeholder> and nothing else: no prefix, no trailing \n, no leading spaces. The gate's matcher (movesRowSpoken, approved_moves_copy_test.go:811) anchors both ends. Print with fmt.Fprintln(w, sentence) or a " %s\n" format around a separately built sentence.
  • instancefs is Linux-only (instancefs_other.go returns errors.ErrUnsupported); the placement code compiles everywhere and its tests carry //go:build linux.

Review Focus

  1. A wake that cannot tell whether the host has a replica of its own (a bucket error) must fail, never fall through to the snapshot or to an empty database: placing over a replica that exists would fork the host's lineage. Test in Task 3 (TestSeededRestoreFailsWhenTheReplicaCannotBeListed).
  2. A local database left by an earlier, aborted seed of the same host (marker names another ID) or with no marker must be purged with its -wal, -shm and litestream metadata before the snapshot is placed. Test in Task 3 (TestSeededRestorePurgesAnotherSeedsDatabase).
  3. The manifest's seed command must never run for a seeded row, whatever Restore reports: the row's seed column stays empty, and Restore never returns fresh for a placeable seed. Tests in Task 5 (TestReconcileMintsAFromSeedNestedWithoutTheSeedCommand) and Task 3.
  4. A ready --from record on a flat or claim-armed box must not be minted at all, so nothing can wake it there. Test in Task 5 (TestReconcileDoesNotMintASeedFlatOrClaimed).
  5. When the placed CAS fails after the first upload, the next wake (which now finds the host's own replica) must finish it: CAS and snapshot delete, idempotently. Test in Task 4 (TestSeedPlacedIsFinishedOnTheNextWake).

---

File structure

| path | responsibility | |---|---| | internal/registry/registry.go | Route.SeedSource, column seed_source, SetSeedSource | | internal/release/store.go | NewS3StoreIn(ctx, bucket, region) | | internal/lease/check.go | HasReplica | | internal/activator/seedplace.go | SeedObjects, SeedPlacementError, seededRestore, SeedPlaced | | internal/activator/seedplace_linux_test.go | the placement tests | | internal/activator/backend_exec.go | ExecBackend.Seeds, ClaimArmed, OnSeedPlaced; Restore dispatches | | internal/activator/backend_unit.go | SplitBackend.SeedPlaced | | internal/activator/activator.go | doWake calls SeedPlaced after replication starts | | internal/activator/logsink.go | memberSafeWakeReason passes a SeedPlacementError sentence through | | internal/provision/seedreconcile.go | SeedHooks, seedAction, finishAbortedSeed | | internal/provision/reconcile.go | the dispatch call, the seed column on mint and converge | | internal/host/seedwiring.go | storeSeedObjects, seedPlacedHook, armSeedHooks | | internal/host/agent.go | arming beside the staging-fallback mirror |

---

Task 1: The registry's seed_source column

Files:

  • Modify: internal/registry/registry.go (Route struct after HoldOp; migrate :524; routeCols :776; scanRoute :783; Upsert :977)
  • Test: internal/registry/registry_test.go (TestUpsertRoundTripsAllColumns :64), internal/registry/seedsource_test.go

Interfaces:

  • Produces: Route.SeedSource string (column seed_source TEXT NOT NULL DEFAULT ''; seeds.RowSeed's encoding, "" for none); (*Registry).SetSeedSource(host, v string) error (errors when no row matched). Upsert's conflict path keeps a non-empty existing value when the incoming one is empty, like seed. Not in Fingerprint: the edge does not route on it. In SnapshotFingerprint only when set.
  • [ ] Step 1: Write the failing tests

In TestUpsertRoundTripsAllColumns, add SeedSource: {"id":"sd_aaaaaaaaaaaaaaaaaaaaaaaaaa","kind":"replica","state":"ready"}, to want.

internal/registry/seedsource_test.go:

package registry

import "testing"

func TestSeedSourceColumn(t *testing.T) {
	reg := openTestRegistry(t)
	const col = `{"id":"sd_aaaaaaaaaaaaaaaaaaaaaaaaaa","kind":"replica","state":"ready"}`
	if err := reg.Upsert(Route{Host: "c.example.com", Kind: "instance", SeedSource: col}); err != nil {
		t.Fatal(err)
	}
	// A re-add that knows nothing of seeds keeps the column.
	if err := reg.Upsert(Route{Host: "c.example.com", Kind: "instance"}); err != nil {
		t.Fatal(err)
	}
	if got, _ := reg.Get("c.example.com"); got.SeedSource != col {
		t.Fatalf("after a seedless Upsert: %q", got.SeedSource)
	}
	placed := `{"id":"sd_aaaaaaaaaaaaaaaaaaaaaaaaaa","kind":"replica","state":"placed"}`
	if err := reg.SetSeedSource("c.example.com", placed); err != nil {
		t.Fatal(err)
	}
	if got, _ := reg.Get("c.example.com"); got.SeedSource != placed {
		t.Fatalf("after SetSeedSource: %q", got.SeedSource)
	}
	if err := reg.SetSeedSource("nope.example.com", placed); err == nil {
		t.Fatal("SetSeedSource on no row: want an error")
	}
	before, _ := reg.Fingerprint()
	_ = reg.SetSeedSource("c.example.com", col)
	if after, _ := reg.Fingerprint(); after != before {
		t.Fatal("the seed column moved the edge's fingerprint")
	}
}
  • [ ] Step 2: Run the tests to verify they fail

Run: go test ./internal/registry/ -run 'TestUpsertRoundTripsAllColumns|TestSeedSourceColumn' -count=1 Expected: FAIL to compile: unknown field SeedSource.

  • [ ] Step 3: Add the column in its four places
  • Route, after HoldOp:
	// SeedSource is the record's seed as this row needs it
	// (seeds.RowSeed's JSON, "" for none), written by the provision pass
	// (seeded-instances spec §3.4): the activator's restore fallback reads
	// it, never the record. Upsert keeps a non-empty value over an empty
	// one; SetSeedSource converges it. Not in Fingerprint.
	SeedSource string
  • migrate's ALTER list, after hold_op: ` ALTER TABLE routes ADD COLUMN seed_source TEXT NOT NULL DEFAULT '', `
  • routeCols: append , seed_source after hold_op, and update its comment's count from 47 to 48.
  • scanRoute: append &t.SeedSource after &t.HoldOp.
  • Upsert: add one ? to VALUES (48), append t.SeedSource to the argument list after t.HoldOp, and add to the ON CONFLICT … DO UPDATE SET list: seed_source=CASE WHEN excluded.seed_source = '' THEN routes.seed_source ELSE excluded.seed_source END, (before args=).
  • SnapshotFingerprint (:2657) hashes every routeCols column; give seed_source hold_op's rule there (hashed only when non-empty, tagged seed_source=), so a registry with no seeded rows digests exactly as before. Add a case to the snapshot digest's existing stability test (find it with grep -n SnapshotFingerprint internal/registry/*_test.go) asserting an unseeded registry's digest is unchanged by the migration.

Then:

// SetSeedSource converges a row's seed column from its record.
func (r *Registry) SetSeedSource(host, v string) error {
	res, err := r.sql.Exec(`UPDATE routes SET seed_source = ? WHERE host = ?`, v, host)
	if err != nil {
		return err
	}
	n, err := res.RowsAffected()
	if err != nil {
		return err
	}
	if n == 0 {
		return fmt.Errorf("no route for host %q", host)
	}
	return nil
}
  • [ ] Step 4: Run the registry package

Run: go test ./internal/registry/ -count=1 Expected: PASS.

  • [ ] Step 5: Commit
git add internal/registry/registry.go internal/registry/registry_test.go internal/registry/seedsource_test.go
git commit -m "registry: the seed_source column (Plan 4 Task 1)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>"

Task 2: lease.HasReplica and release.NewS3StoreIn

Files:

  • Modify: internal/lease/check.go (add HasReplica), internal/release/store.go:687 (add NewS3StoreIn)
  • Test: internal/lease/hasreplica_test.go, internal/release/store_test.go

Interfaces:

  • Produces: lease.HasReplica(ctx context.Context, bucket, prefix, region string) (bool, error): true when any LTX file exists at any level 0 to 9 under prefix; an error when the listing fails. release.NewS3StoreIn(ctx context.Context, bucket, region string) (*S3Store, error): NewS3Store with the AWS region pinned when region != "".
  • [ ] Step 1: Write the failing tests

internal/lease/hasreplica_test.go. The listing itself is a small function over anything with litestream's LTXFiles, so the test needs no S3:

package lease

import (
	"context"
	"errors"
	"testing"

	"github.com/superfly/ltx"
)

type fakeLister struct {
	levels map[int][]*ltx.FileInfo
	err    error
	asked  []int
}

func (f *fakeLister) LTXFiles(_ context.Context, level int, _ ltx.TXID, _ bool) (ltx.FileIterator, error) {
	f.asked = append(f.asked, level)
	if f.err != nil {
		return nil, f.err
	}
	return ltx.NewFileInfoSliceIterator(f.levels[level]), nil
}

func TestHasReplicaIn(t *testing.T) {
	ctx := context.Background()
	empty := &fakeLister{}
	if has, err := hasReplicaIn(ctx, empty); has || err != nil {
		t.Fatalf("empty = %v, %v", has, err)
	}
	if len(empty.asked) != 10 {
		t.Fatalf("levels asked = %v, want 0 to 9", empty.asked)
	}
	snap := &fakeLister{levels: map[int][]*ltx.FileInfo{9: {{Level: 9, MinTXID: 1, MaxTXID: 7, Size: 4096}}}}
	if has, err := hasReplicaIn(ctx, snap); !has || err != nil {
		t.Fatalf("a snapshot at level 9 = %v, %v", has, err)
	}
	broken := &fakeLister{err: errors.New("listing failed")}
	if has, err := hasReplicaIn(ctx, broken); has || err == nil {
		t.Fatalf("a failed listing = %v, %v; want an error, never false", has, err)
	}
}

In internal/release/store_test.go:

func TestNewS3StoreInPinsTheRegion(t *testing.T) {
	t.Setenv("AWS_REGION", "us-east-1")
	t.Setenv("AWS_ACCESS_KEY_ID", "k")
	t.Setenv("AWS_SECRET_ACCESS_KEY", "s")
	s, err := NewS3StoreIn(context.Background(), "b", "eu-west-1")
	if err != nil {
		t.Fatal(err)
	}
	if got := s.Client.Options().Region; got != "eu-west-1" {
		t.Fatalf("region = %q, want eu-west-1", got)
	}
}
  • [ ] Step 2: Run the tests to verify they fail

Run: go test ./internal/lease/ -run TestHasReplicaIn -count=1; go test ./internal/release/ -run TestNewS3StoreIn -count=1 Expected: FAIL to compile.

  • [ ] Step 3: Write both

In internal/lease/check.go:

// HasReplica reports whether s3://bucket/prefix holds any LTX file at any
// level: the seed fallback's "has this host a replica of its own" (seeded
// instances spec §3.4 step 1). A listing error is returned, never read as
// "no replica".
func HasReplica(ctx context.Context, bucket, prefix, region string) (bool, error) {
	region, err := resolveRegion(ctx, region)
	if err != nil {
		return false, err
	}
	return hasReplicaIn(ctx, newReplicaClient(bucket, prefix, region))
}

// ltxLister is the one replica-client method HasReplica needs.
type ltxLister interface {
	LTXFiles(ctx context.Context, level int, seek ltx.TXID, useMetadata bool) (ltx.FileIterator, error)
}

func hasReplicaIn(ctx context.Context, client ltxLister) (bool, error) {
	for level := 0; level <= litestream.SnapshotLevel; level++ {
		itr, err := client.LTXFiles(ctx, level, 0, false)
		if err != nil {
			return false, err
		}
		found := itr.Next()
		ierr := itr.Err()
		cerr := itr.Close()
		if ierr != nil {
			return false, ierr
		}
		if cerr != nil {
			return false, cerr
		}
		if found {
			return true, nil
		}
	}
	return false, nil
}

(litestream.SnapshotLevel is 9 in v0.5.14; add the litestream and github.com/superfly/ltx imports if check.go lacks them.)

In internal/release/store.go, refactor NewS3Store's body into newS3Store(ctx, bucket string, opts ...func(*awsconfig.LoadOptions) error) passing opts to awsconfig.LoadDefaultConfig(ctx, opts...), keep NewS3Store(ctx, bucket) as return newS3Store(ctx, bucket), and add:

// NewS3StoreIn is NewS3Store with the bucket's region pinned: a box's or a
// console's data bucket is named with its region beside it
// (CARLOS_HIBERNATE_REGION), and the default chain's region may be
// another. An empty region is NewS3Store.
func NewS3StoreIn(ctx context.Context, bucket, region string) (*S3Store, error) {
	if region == "" {
		return newS3Store(ctx, bucket)
	}
	return newS3Store(ctx, bucket, awsconfig.WithRegion(region))
}
  • [ ] Step 4: Run the tests to verify they pass

Run: go test ./internal/lease/ -run TestHasReplicaIn -count=1 && go test ./internal/release/ -run 'TestNewS3Store' -count=1 Expected: PASS.

  • [ ] Step 5: Commit
git add internal/lease/check.go internal/lease/hasreplica_test.go internal/release/store.go internal/release/store_test.go
git commit -m "lease.HasReplica and release.NewS3StoreIn for the seed fallback (Plan 4 Task 2)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>"

Task 3: The restore fallback

Files:

  • Create: internal/activator/seedplace.go, internal/activator/seedplace_linux_test.go
  • Modify: internal/activator/backend_exec.go:611 (Restore dispatches; the existing body becomes restorePlain), the ExecBackend struct (fields Seeds, ClaimArmed)
  • Modify: internal/activator/logsink.go:49 (memberSafeWakeReason)

Interfaces:

  • Consumes: seeds.ParseRowSeed, seeds.RowSeed.Placeable, lease.HasReplica, lease.CheckDBContext, instancefs.OpenBeneath, instancefs.ReadBeneathLimit, instancefs.WriteAtomic, instancefs.RenameBetween, instancefs.RemoveBeneath, PurgeLocalDBArtifacts, b.pathsFor(rt), LayoutNested, b.Cfg.Bucket, b.Cfg.hibernatePrefix(host), b.Cfg.Region.
  • Produces: type SeedObjects interface{ Open(ctx context.Context, host, id string) (io.ReadCloser, error) }; ExecBackend.Seeds SeedObjects; ExecBackend.ClaimArmed bool; type SeedPlacementError struct{ Sentence string; Err error }; SeedMarkerSuffix = ".seed"; package vars hasOwnReplica = lease.HasReplica, checkSeedDB = lease.CheckDBContext.

The fallback's order (spec §3.4):

  1. Claim armed: refuse (SD-N18). Not nested (pathsFor(rt).layout != LayoutNested): refuse (SD-N17).
  2. The host has a replica of its own (hasOwnReplica): restore plainly; if the seed is not yet placed, finish placement best-effort (Task 4's finishPlacement); never report fresh. A listing error fails the wake.
  3. A local database whose <db>.seed names this seed's ID: keep it; replication starts in doWake.
  4. Otherwise purge the local database, its -wal and -shm, its marker and litestream's metadata; fetch the snapshot to .seed-<id>.tmp in the instance directory, hashing as it writes; refuse a length or sha256 that differs (SD-N14); run CheckDBContext (SD-N15); write the marker, then rename the file onto the database path (SD-N19 on failure). A read failure is SD-N16.
  • [ ] Step 1: Write the failing tests

internal/activator/seedplace_linux_test.go:

//go:build linux

package activator

import (
	"bytes"
	"context"
	"crypto/sha256"
	"encoding/hex"
	"errors"
	"io"
	"os"
	"path/filepath"
	"strings"
	"testing"

	"github.com/carlosframework/platform/internal/registry"
	"github.com/carlosframework/platform/internal/seeds"
)

type fakeSeedObjects struct {
	body  map[string][]byte // id -> snapshot
	err   error
	opens int
}

func (f *fakeSeedObjects) Open(_ context.Context, host, id string) (io.ReadCloser, error) {
	f.opens++
	if f.err != nil {
		return nil, f.err
	}
	b, ok := f.body[id]
	if !ok {
		return nil, os.ErrNotExist
	}
	return io.NopCloser(bytes.NewReader(b)), nil
}

// seededRoute is nestedBackend's row with a seed column for a snapshot
// made by writeMinimalSQLite.
func seededRoute(t *testing.T, b *ExecBackend, rt registry.Route, state string) (registry.Route, *fakeSeedObjects, string) {
	t.Helper()
	id, _ := seeds.NewID()
	snap := filepath.Join(t.TempDir(), "snap.db")
	if err := writeMinimalSQLite(snap); err != nil {
		t.Fatal(err)
	}
	body, _ := os.ReadFile(snap)
	sum := sha256.Sum256(body)
	rt.SeedSource = seeds.RowSeed{ID: id, Kind: seeds.KindReplica, SHA256: hex.EncodeToString(sum[:]), Bytes: int64(len(body)), State: state}.Encode()
	objs := &fakeSeedObjects{body: map[string][]byte{id: body}}
	b.Seeds = objs
	return rt, objs, id
}

func noOwnReplica(t *testing.T) {
	t.Helper()
	old := hasOwnReplica
	hasOwnReplica = func(context.Context, string, string, string) (bool, error) { return false, nil }
	t.Cleanup(func() { hasOwnReplica = old })
}

func TestSeededRestorePlacesTheSnapshot(t *testing.T) {
	noOwnReplica(t)
	b, rt := nestedBackend(t, &fakeControl{}, &fakeUnit{})
	rt, objs, id := seededRoute(t, b, rt, seeds.StateReady)
	fresh, err := b.Restore(context.Background(), rt)
	if err != nil || fresh {
		t.Fatalf("Restore = fresh %v, %v; want placed, never fresh", fresh, err)
	}
	if objs.opens != 1 {
		t.Fatalf("snapshot opened %d times", objs.opens)
	}
	marker, err := os.ReadFile(rt.DBPath + SeedMarkerSuffix)
	if err != nil || strings.TrimSpace(string(marker)) != id {
		t.Fatalf("marker = %q, %v; want the seed id", marker, err)
	}
	if err := checkSeedDB(context.Background(), rt.DBPath); err != nil {
		t.Fatalf("the placed database does not check: %v", err)
	}
	// A second wake (a crash after the rename, before the first upload)
	// keeps the placed file and does not fetch again.
	if fresh, err := b.Restore(context.Background(), rt); err != nil || fresh || objs.opens != 1 {
		t.Fatalf("resume: fresh %v err %v opens %d", fresh, err, objs.opens)
	}
}

func TestSeededRestorePurgesAnotherSeedsDatabase(t *testing.T) {
	noOwnReplica(t)
	b, rt := nestedBackend(t, &fakeControl{}, &fakeUnit{})
	rt, _, id := seededRoute(t, b, rt, seeds.StateSwapped)
	os.MkdirAll(filepath.Dir(rt.DBPath), 0o700)
	os.WriteFile(rt.DBPath, []byte("an aborted import's data"), 0o600)
	os.WriteFile(rt.DBPath+"-wal", []byte("w"), 0o600)
	os.WriteFile(rt.DBPath+"-shm", []byte("s"), 0o600)
	os.WriteFile(rt.DBPath+SeedMarkerSuffix, []byte("sd_bbbbbbbbbbbbbbbbbbbbbbbbbb\n"), 0o600)
	meta := filepath.Join(filepath.Dir(rt.DBPath), ".activator-"+filepath.Base(rt.DBPath)+"-litestream")
	os.MkdirAll(meta, 0o700)
	if _, err := b.Restore(context.Background(), rt); err != nil {
		t.Fatal(err)
	}
	for _, p := range []string{rt.DBPath + "-wal", rt.DBPath + "-shm", meta} {
		if _, err := os.Stat(p); !os.IsNotExist(err) {
			t.Errorf("%s survived the purge", p)
		}
	}
	if m, _ := os.ReadFile(rt.DBPath + SeedMarkerSuffix); strings.TrimSpace(string(m)) != id {
		t.Fatalf("marker = %q, want %s", m, id)
	}
	if b, _ := os.ReadFile(rt.DBPath); bytes.Equal(b, []byte("an aborted import's data")) {
		t.Fatal("the other seed's database stood in for this one")
	}
}

func TestSeededRestoreRefusesABadSnapshot(t *testing.T) {
	noOwnReplica(t)
	for _, c := range []struct {
		name   string
		mangle func(*fakeSeedObjects, string)
		want   string
	}{
		{"short", func(f *fakeSeedObjects, id string) { f.body[id] = f.body[id][:100] }, "length or sha256"},
		{"wrong bytes", func(f *fakeSeedObjects, id string) { f.body[id] = bytes.Repeat([]byte{'x'}, len(f.body[id])) }, "length or sha256"},
		{"unreadable", func(f *fakeSeedObjects, id string) { f.err = errors.New("bucket down") }, "could not be read"},
	} {
		t.Run(c.name, func(t *testing.T) {
			b, rt := nestedBackend(t, &fakeControl{}, &fakeUnit{})
			rt, objs, id := seededRoute(t, b, rt, seeds.StateReady)
			c.mangle(objs, id)
			fresh, err := b.Restore(context.Background(), rt)
			var spe *SeedPlacementError
			if fresh || !errors.As(err, &spe) || !strings.Contains(spe.Sentence, c.want) {
				t.Fatalf("Restore = fresh %v, %v; want a %q refusal", fresh, err, c.want)
			}
			if _, err := os.Stat(rt.DBPath); !os.IsNotExist(err) {
				t.Fatal("a refused snapshot left a database at the row's path")
			}
		})
	}
}

func TestSeededRestoreRefusesANonDatabase(t *testing.T) {
	noOwnReplica(t)
	b, rt := nestedBackend(t, &fakeControl{}, &fakeUnit{})
	id, _ := seeds.NewID()
	body := []byte("not sqlite at all, but the right length and hash")
	sum := sha256.Sum256(body)
	rt.SeedSource = seeds.RowSeed{ID: id, Kind: seeds.KindImport, SHA256: hex.EncodeToString(sum[:]), Bytes: int64(len(body)), State: seeds.StateSwapped}.Encode()
	b.Seeds = &fakeSeedObjects{body: map[string][]byte{id: body}}
	_, err := b.Restore(context.Background(), rt)
	var spe *SeedPlacementError
	if !errors.As(err, &spe) || !strings.Contains(spe.Sentence, "integrity check") {
		t.Fatalf("Restore = %v; want the integrity refusal", err)
	}
}

func TestSeededRestoreRefusesClaimAndFlat(t *testing.T) {
	noOwnReplica(t)
	b, rt := nestedBackend(t, &fakeControl{}, &fakeUnit{})
	rt, _, _ = seededRoute(t, b, rt, seeds.StateReady)
	b.ClaimArmed = true
	var spe *SeedPlacementError
	if _, err := b.Restore(context.Background(), rt); !errors.As(err, &spe) || !strings.Contains(spe.Sentence, "claim control") {
		t.Fatalf("claim-armed: %v", err)
	}
	b.ClaimArmed = false
	flat := testExecBackend(t)
	flat.Seeds = b.Seeds
	frt := registry.Route{Host: rt.Host, Kind: "instance", Hibernate: true, SeedSource: rt.SeedSource,
		Socket: filepath.Join(t.TempDir(), "x.sock")} // DBPath empty: the flat default
	if _, err := flat.Restore(context.Background(), frt); !errors.As(err, &spe) || !strings.Contains(spe.Sentence, "nested") {
		t.Fatalf("flat: %v", err)
	}
}

func TestSeededRestoreFailsWhenTheReplicaCannotBeListed(t *testing.T) {
	old := hasOwnReplica
	hasOwnReplica = func(context.Context, string, string, string) (bool, error) { return false, errors.New("listing failed") }
	defer func() { hasOwnReplica = old }()
	b, rt := nestedBackend(t, &fakeControl{}, &fakeUnit{})
	rt, objs, _ := seededRoute(t, b, rt, seeds.StateReady)
	if fresh, err := b.Restore(context.Background(), rt); err == nil || fresh || objs.opens != 0 {
		t.Fatalf("Restore = fresh %v, %v, opens %d; want a failed wake and no fetch", fresh, err, objs.opens)
	}
}

func TestSeededRestoreUsesItsOwnReplicaOnceItHasOne(t *testing.T) {
	old := hasOwnReplica
	hasOwnReplica = func(context.Context, string, string, string) (bool, error) { return true, nil }
	defer func() { hasOwnReplica = old }()
	oldRestore := restoreReplica
	restored := 0
	restoreReplica = func(_ context.Context, dbPath, _, _, _ string) (bool, error) {
		restored++
		return true, writeMinimalSQLite(dbPath)
	}
	defer func() { restoreReplica = oldRestore }()
	b, rt := nestedBackend(t, &fakeControl{}, &fakeUnit{})
	rt, objs, _ := seededRoute(t, b, rt, seeds.StatePlaced)
	if fresh, err := b.Restore(context.Background(), rt); err != nil || fresh || restored != 1 || objs.opens != 0 {
		t.Fatalf("fresh %v err %v restored %d opens %d; want the own replica, not the seed", fresh, err, restored, objs.opens)
	}
}

func TestMemberSafeWakeReasonPassesTheSeedSentence(t *testing.T) {
	err := fmtWrap(seedRefusedFlat("canary.go.tito.io"))
	if got := memberSafeWakeReason(err); got != "wake failed: canary.go.tito.io could not start from its seed: this box does not run instances nested." {
		t.Fatalf("reason = %q", got)
	}
}

func fmtWrap(err error) error { return errors.Join(errors.New("restore canary.go.tito.io"), err) }
  • [ ] Step 2: Run the tests to verify they fail

Run: go test ./internal/activator/ -run 'TestSeeded|TestMemberSafeWakeReasonPassesTheSeedSentence' -count=1 Expected: FAIL to compile: undefined: SeedMarkerSuffix, b.Seeds undefined.

  • [ ] Step 3: Add the fields and the dispatch

In the ExecBackend struct (backend_exec.go), beside NestedOK:

	// Seeds reads a seed's snapshot from this region's data bucket
	// (seeded-instances spec §3.4). Nil: no seed can be placed here, and a
	// seeded row's wake fails rather than starting empty.
	Seeds SeedObjects
	// ClaimArmed is CARLOS_HOST_CLAIM on this box. The seed fallback is the
	// plain path only; with claims armed it refuses (spec §3.4, #250).
	ClaimArmed bool
	// OnSeedPlaced records a seed placed (Task 4): the record's CAS to
	// placed, then the snapshot's delete. Nil: not recorded here.
	OnSeedPlaced func(ctx context.Context, account, app, host, id string) error

Rename the current Restore (:611-701) to restorePlain, unchanged, and add above it:

func (b *ExecBackend) Restore(ctx context.Context, rt registry.Route) (bool, error) {
	src, err := seeds.ParseRowSeed(rt.SeedSource)
	if err != nil {
		return false, fmt.Errorf("restore %s: %w", rt.Host, err)
	}
	if src != nil && src.Placeable() {
		return b.seededRestore(ctx, rt, src)
	}
	return b.restorePlain(ctx, rt)
}
  • [ ] Step 4: Write internal/activator/seedplace.go
package activator

import (
	"context"
	"crypto/sha256"
	"encoding/hex"
	"errors"
	"fmt"
	"io"
	"os"
	"path/filepath"
	"strings"

	"github.com/carlosframework/platform/internal/instancefs"
	"github.com/carlosframework/platform/internal/lease"
	"github.com/carlosframework/platform/internal/registry"
	"github.com/carlosframework/platform/internal/seeds"
)

// SeedObjects reads a seed's snapshot, seeds/<host>/<id>/snapshot.db, from
// this box's own region data bucket.
type SeedObjects interface {
	Open(ctx context.Context, host, id string) (io.ReadCloser, error)
}

// SeedMarkerSuffix names the installation evidence beside a placed
// database: <db>.seed holds the seed's id (spec §3.4 step 3, codex R2-1).
const SeedMarkerSuffix = ".seed"

// hasOwnReplica and checkSeedDB are seams for the tests.
var (
	hasOwnReplica = lease.HasReplica
	checkSeedDB   = lease.CheckDBContext
)

// SeedPlacementError is a wake the seed fallback failed or refused. Its
// Sentence is the member's (SD-N14 to SD-N19): memberSafeWakeReason passes
// it to carlos logs whole.
type SeedPlacementError struct {
	Sentence string
	Err      error
}

func (e *SeedPlacementError) Error() string { return e.Sentence }
func (e *SeedPlacementError) Unwrap() error { return e.Err }

func seedMismatch(host string) error {
	return &SeedPlacementError{Sentence: fmt.Sprintf("%s could not start from its seed: the snapshot's length or sha256 does not match its record.", host)}
}

func seedNotDB(host string, err error) error {
	return &SeedPlacementError{Sentence: fmt.Sprintf("%s could not start from its seed: the snapshot is not a database that passes an integrity check.", host), Err: err}
}

func seedUnread(host string, err error) error {
	return &SeedPlacementError{Sentence: fmt.Sprintf("%s could not start from its seed: the snapshot could not be read. The next wake tries again.", host), Err: err}
}

func seedRefusedFlat(host string) error {
	return &SeedPlacementError{Sentence: fmt.Sprintf("%s could not start from its seed: this box does not run instances nested.", host)}
}

func seedRefusedClaim(host string) error {
	return &SeedPlacementError{Sentence: fmt.Sprintf("%s could not start from its seed: claim control is on for this box.", host)}
}

func seedPlaceFailed(host string, err error) error {
	return &SeedPlacementError{Sentence: fmt.Sprintf("%s could not start from its seed: it could not be put in place. The next wake tries again.", host), Err: err}
}

// seededRestore is the plain-path seed fallback (spec §3.4). It never
// reports fresh: a seeded row never runs the manifest's seed command, and
// every failure fails the wake instead of falling through to an empty
// database.
func (b *ExecBackend) seededRestore(ctx context.Context, rt registry.Route, src *seeds.RowSeed) (bool, error) {
	host := rt.Host
	if b.ClaimArmed {
		return false, seedRefusedClaim(host)
	}
	paths := b.pathsFor(rt)
	if paths.layout != LayoutNested {
		return false, seedRefusedFlat(host)
	}
	// Step 1: a replica of its own means this host was placed already.
	has, err := hasOwnReplica(ctx, b.Cfg.Bucket, b.Cfg.hibernatePrefix(host), b.Cfg.Region)
	if err != nil {
		return false, fmt.Errorf("seed fallback for %s: cannot tell whether it has a replica of its own: %w", host, err)
	}
	if has {
		if _, err := b.restorePlain(ctx, rt); err != nil {
			return false, err
		}
		if src.State != seeds.StatePlaced {
			b.finishPlacement(ctx, rt, src)
		}
		return false, nil
	}
	dbPath := paths.dbPath
	dir, base := filepath.Dir(dbPath), filepath.Base(dbPath)
	if _, err := os.Stat(dbPath); err == nil {
		if m, err := instancefs.ReadBeneathLimit(dir, base+SeedMarkerSuffix, 128); err == nil && strings.TrimSpace(string(m)) == src.ID {
			return false, nil // this seed, placed by an interrupted first wake
		}
	} else if !errors.Is(err, os.ErrNotExist) {
		return false, err
	}
	if b.Seeds == nil {
		return false, seedUnread(host, errors.New("no seed reader on this box"))
	}
	// Anything else is purged: another seed's database, one with no
	// marker, their WAL and SHM, the marker, litestream's metadata.
	PurgeLocalDBArtifacts(dbPath, false)
	for _, name := range []string{base, base + SeedMarkerSuffix} {
		if err := instancefs.RemoveBeneath(dir, name); err != nil && !errors.Is(err, os.ErrNotExist) {
			return false, seedPlaceFailed(host, err)
		}
	}
	if err := os.MkdirAll(dir, 0o700); err != nil {
		return false, seedPlaceFailed(host, err)
	}
	tmp := ".seed-" + src.ID + ".tmp"
	_ = instancefs.RemoveBeneath(dir, tmp)
	if err := b.fetchSeed(ctx, host, src, dir, tmp); err != nil {
		_ = instancefs.RemoveBeneath(dir, tmp)
		return false, err
	}
	if err := checkSeedDB(ctx, filepath.Join(dir, tmp)); err != nil {
		_ = instancefs.RemoveBeneath(dir, tmp)
		return false, seedNotDB(host, err)
	}
	// The marker first: a database at the row's path always has one.
	if err := instancefs.WriteAtomic(dir, base+SeedMarkerSuffix, []byte(src.ID+"\n"), 0o600); err != nil {
		_ = instancefs.RemoveBeneath(dir, tmp)
		return false, seedPlaceFailed(host, err)
	}
	if err := instancefs.RenameBetween(dir, tmp, dir, base); err != nil {
		_ = instancefs.RemoveBeneath(dir, tmp)
		return false, seedPlaceFailed(host, err)
	}
	return false, nil
}

// fetchSeed copies the snapshot into dir/tmp, checking its length and
// sha256 against the row as it writes.
func (b *ExecBackend) fetchSeed(ctx context.Context, host string, src *seeds.RowSeed, dir, tmp string) error {
	rc, err := b.Seeds.Open(ctx, host, src.ID)
	if err != nil {
		return seedUnread(host, err)
	}
	defer rc.Close()
	f, err := instancefs.OpenBeneath(dir, tmp, os.O_CREATE|os.O_EXCL|os.O_WRONLY, 0o600)
	if err != nil {
		return seedPlaceFailed(host, err)
	}
	h := sha256.New()
	n, cerr := io.Copy(io.MultiWriter(f, h), io.LimitReader(rc, src.Bytes+1))
	serr := f.Sync()
	ferr := f.Close()
	if cerr != nil {
		return seedUnread(host, cerr)
	}
	if serr != nil || ferr != nil {
		return seedPlaceFailed(host, errors.Join(serr, ferr))
	}
	if n != src.Bytes || hex.EncodeToString(h.Sum(nil)) != src.SHA256 {
		return seedMismatch(host)
	}
	return nil
}

finishPlacement is Task 4's; for this task add a stub in seedplace.go that does nothing (func (b *ExecBackend) finishPlacement(context.Context, registry.Route, *seeds.RowSeed) {}), replaced in Task 4.

In logsink.go's memberSafeWakeReason, first thing:

	var seed *SeedPlacementError
	if errors.As(err, &seed) {
		return "wake failed: " + seed.Sentence
	}
  • [ ] Step 5: Run the tests to verify they pass

Run: go test ./internal/activator/ -run 'TestSeeded|TestMemberSafeWakeReason|TestExecRestore|TestStaging|TestRestore' -count=1 Expected: PASS.

  • [ ] Step 6: Wire SD-N14 to SD-N19 and commit

Append "SD-N14", "SD-N15", "SD-N16", "SD-N17", "SD-N18", "SD-N19" to seededWiredIDs; run go test . -run TestSeededWiredCopyIsApplied -count=1 (PASS).

git add internal/activator/seedplace.go internal/activator/seedplace_linux_test.go internal/activator/backend_exec.go internal/activator/logsink.go approved_seeded_copy_test.go
git commit -m "activator: the seed fallback in Restore, marker, purge and checks (Plan 4 Task 3)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>"

Task 4: Placed after the first upload

Files:

  • Modify: internal/activator/seedplace.go (SeedPlaced, finishPlacement)
  • Modify: internal/activator/backend_unit.go:311 (SplitBackend.SeedPlaced)
  • Modify: internal/activator/activator.go:1321-1330 (doWake, after lease.Replicate)
  • Test: internal/activator/seedplace_linux_test.go, internal/activator/activator_test.go

Interfaces:

  • Consumes: ExecBackend.OnSeedPlaced (Task 3's field), Rep.
  • Produces: (b *ExecBackend) SeedPlaced(ctx context.Context, rt registry.Route, rep Rep) error; (b *SplitBackend) SeedPlaced(...); unexported seedPlacer interface in activator.go.

SeedPlaced does nothing for a row whose seed is absent, not placeable, or already placed. Otherwise it waits for the first upload (SyncAndWait on the replicator, when rep offers it) and calls OnSeedPlaced. doWake logs a failure and goes on serving: the next wake finds the host's own replica and finishPlacement records it then.

  • [ ] Step 1: Write the failing tests

Append to seedplace_linux_test.go:

type syncRep struct {
	synced int
	err    error
}

func (r *syncRep) Err() error                         { return nil }
func (r *syncRep) Close(context.Context) error        { return nil }
func (r *syncRep) SyncAndWait(context.Context) error  { r.synced++; return r.err }

func TestSeedPlacedWaitsForTheFirstUpload(t *testing.T) {
	b, rt := nestedBackend(t, &fakeControl{}, &fakeUnit{})
	rt, _, id := seededRoute(t, b, rt, seeds.StateReady)
	rt.Account, rt.App = "bae", "titogo"
	var calls []string
	b.OnSeedPlaced = func(_ context.Context, acct, app, host, got string) error {
		calls = append(calls, acct+"/"+app+"/"+host+"/"+got)
		return nil
	}
	rep := &syncRep{}
	if err := b.SeedPlaced(context.Background(), rt, rep); err != nil {
		t.Fatal(err)
	}
	if rep.synced != 1 || len(calls) != 1 || calls[0] != "bae/titogo/"+rt.Host+"/"+id {
		t.Fatalf("synced %d, calls %q", rep.synced, calls)
	}
	// A failed first upload records nothing.
	rep = &syncRep{err: errors.New("upload failed")}
	calls = nil
	if err := b.SeedPlaced(context.Background(), rt, rep); err == nil || len(calls) != 0 {
		t.Fatalf("after a failed sync: err %v calls %q", err, calls)
	}
	// Placed already, or no seed: nothing at all.
	rt.SeedSource = strings.Replace(rt.SeedSource, `"state":"ready"`, `"state":"placed"`, 1)
	rep = &syncRep{}
	if err := b.SeedPlaced(context.Background(), rt, rep); err != nil || rep.synced != 0 || len(calls) != 0 {
		t.Fatalf("placed row: err %v synced %d calls %q", err, rep.synced, calls)
	}
}

func TestSeedPlacedIsFinishedOnTheNextWake(t *testing.T) {
	old := hasOwnReplica
	hasOwnReplica = func(context.Context, string, string, string) (bool, error) { return true, nil }
	defer func() { hasOwnReplica = old }()
	oldRestore := restoreReplica
	restoreReplica = func(_ context.Context, dbPath, _, _, _ string) (bool, error) { return true, writeMinimalSQLite(dbPath) }
	defer func() { restoreReplica = oldRestore }()
	b, rt := nestedBackend(t, &fakeControl{}, &fakeUnit{})
	rt, _, id := seededRoute(t, b, rt, seeds.StateReady) // the CAS was lost last wake
	rt.Account, rt.App = "bae", "titogo"
	var got []string
	b.OnSeedPlaced = func(_ context.Context, _, _, _, seed string) error { got = append(got, seed); return nil }
	if _, err := b.Restore(context.Background(), rt); err != nil {
		t.Fatal(err)
	}
	if len(got) != 1 || got[0] != id {
		t.Fatalf("finishPlacement calls = %q, want one for %s", got, id)
	}
}

In activator_test.go, add a doWake-level test on the package's fakeBackend (activator_test.go:215), wired the way testActivator (:620) wires it:

// seedCountingBackend is fakeBackend plus the optional seed recorder.
type seedCountingBackend struct {
	fakeBackend
	placed []string
}

func (b *seedCountingBackend) SeedPlaced(_ context.Context, rt registry.Route, _ Rep) error {
	b.placed = append(b.placed, rt.Host)
	return nil
}

func TestDoWakeRecordsSeedPlacementAfterReplication(t *testing.T) {
	b := &seedCountingBackend{}
	reg, err := registry.Open(filepath.Join(t.TempDir(), "registry.db"))
	if err != nil {
		t.Fatal(err)
	}
	a := New(Config{Bucket: "test-bucket", Idle: time.Hour}, b, reg)
	rt := testRoute("canary.example.com")
	rt.SeedSource = seeds.RowSeed{ID: "sd_aaaaaaaaaaaaaaaaaaaaaaaaaa", Kind: seeds.KindReplica, State: seeds.StateReady}.Encode()
	if err := reg.Upsert(rt); err != nil {
		t.Fatal(err)
	}
	arrangeAwake(t, a, rt)
	if len(b.placed) != 1 || b.placed[0] != rt.Host {
		t.Fatalf("SeedPlaced calls = %q, want one, after replication started", b.placed)
	}
	if len(b.reps) != 1 {
		t.Fatalf("replications = %d: SeedPlaced must follow Replicate", len(b.reps))
	}
}

(add the seeds import to activator_test.go).

  • [ ] Step 2: Run the tests to verify they fail

Run: go test ./internal/activator/ -run 'TestSeedPlaced|TestDoWakeRecordsSeedPlacement' -count=1 Expected: FAIL to compile: b.SeedPlaced undefined.

  • [ ] Step 3: Write SeedPlaced and finishPlacement

Replace Task 3's stub in seedplace.go:

// SeedPlaced records a seed placed once the host's own replica holds it
// (spec §3.4 steps 4 and 5): it waits for the first upload, then the host
// package CASes the record to placed and deletes seeds/<host>/<id>/. The
// snapshot goes only after the replica exists, so a crash at any point
// retries from the placed local file or the snapshot.
func (b *ExecBackend) SeedPlaced(ctx context.Context, rt registry.Route, rep Rep) error {
	src, err := seeds.ParseRowSeed(rt.SeedSource)
	if err != nil || src == nil || !src.Placeable() || src.State == seeds.StatePlaced {
		return err
	}
	if s, ok := rep.(interface{ SyncAndWait(context.Context) error }); ok {
		if err := s.SyncAndWait(ctx); err != nil {
			return fmt.Errorf("first upload of %s: %w", rt.Host, err)
		}
	}
	if b.OnSeedPlaced == nil {
		return errors.New("no seed recorder on this box")
	}
	return b.OnSeedPlaced(ctx, rt.Account, rt.App, rt.Host, src.ID)
}

// finishPlacement is SeedPlaced for a host whose own replica already
// exists (a placed CAS lost on an earlier wake). Best effort: logged.
func (b *ExecBackend) finishPlacement(ctx context.Context, rt registry.Route, src *seeds.RowSeed) {
	if b.OnSeedPlaced == nil {
		return
	}
	if err := b.OnSeedPlaced(ctx, rt.Account, rt.App, rt.Host, src.ID); err != nil {
		log.Printf("activator: %s: seed %s not recorded as placed yet: %v", rt.Host, src.ID, err)
	}
}

(add "log" to the imports).

In backend_unit.go, beside AdmitConfig:

// SeedPlaced dispatches on Backing, like every other call. A unit tenant
// never carries a seed.
func (b *SplitBackend) SeedPlaced(ctx context.Context, rt registry.Route, rep Rep) error {
	if p, ok := b.pick(rt).(seedPlacer); ok {
		return p.SeedPlaced(ctx, rt, rep)
	}
	return nil
}

In activator.go, beside the Backend interface:

// seedPlacer is the optional half of a Backend that records a seed placed
// after the first upload (seeded-instances spec §3.4).
type seedPlacer interface {
	SeedPlaced(ctx context.Context, rt registry.Route, rep Rep) error
}

In doWake, immediately after the successful lease.Replicate (w.rep = rep and the error check), before step 6:

	// 5b. A seeded row's first upload, then its record (spec §3.4 step 5).
	// A failure here does not fail the wake: the instance is serving from
	// the placed file, and the next wake records it (finishPlacement).
	if p, ok := a.backend.(seedPlacer); ok {
		if err := p.SeedPlaced(ctx, rt, rep); err != nil {
			log.Printf("activator: %s: seed placement not recorded yet: %v", host, err)
		}
	}
  • [ ] Step 4: Run the tests to verify they pass

Run: go test ./internal/activator/ -run 'TestSeed|TestDoWake|TestMemberSafe' -count=1 Expected: PASS.

  • [ ] Step 5: Commit
git add internal/activator/
git commit -m "activator: record a seed placed after the first upload (Plan 4 Task 4)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>"

Task 5: Reconcile's seed dispatch

Files:

  • Create: internal/provision/seedreconcile.go
  • Modify: internal/provision/reconcile.go (ReconcileConfig :35; routeFor :425; the record loop :1072; the owned branch's converge :1232)
  • Test: internal/provision/seedreconcile_test.go

Interfaces:

  • Consumes: seeds.*, SetRecordSeed, SeedRolledBack, SeedColumn, mintLayoutReader.
  • Produces: type SeedHooks struct{ Data release.Store } (Plan 7 adds fields); ReconcileConfig.Seeds *SeedHooks; unexported seedAction(rec Record, hasRow bool, cfg ReconcileConfig) seedVerdict with seedOrdinary, seedSkip, seedFinishAbort; finishAbortedSeed(ctx, s release.Store, cfg ReconcileConfig, rec Record).

The dispatch, before any other per-record handling (spec §3.2):

| seed | verdict | |---|---| | none | ordinary (as today) | | any, cfg.Seeds == nil | skip (a box not wired for seeds never mints one) | | aborted | finish: delete seeds/<host>/<id>/snapshot.db, CAS rolled-back (Plan 7 puts the journal check first) | | rolled-back, reserving, awaiting-upload, uploaded, refused | skip | | ready, replica | ordinary, but only nested and only with claims off; otherwise skip and log | | ready, import | skip (Plan 7 swaps) | | swapped, placed, committed | ordinary |

For every seeded record that is handled ordinarily, the manifest seed is "" and known (the seed command never runs for a seeded row), and the row carries SeedColumn(rec).

  • [ ] Step 1: Write the failing tests

internal/provision/seedreconcile_test.go:

package provision

import (
	"context"
	"errors"
	"os"
	"testing"
	"time"

	"github.com/carlosframework/platform/internal/release"
	"github.com/carlosframework/platform/internal/seeds"
)

func seededRecord(t *testing.T, s release.Store, sqid, kind, state string) Record {
	t.Helper()
	rec := acmeRecord(sqid)
	id, _ := seeds.NewID()
	rec.Seed = &SeedSource{ID: id, Kind: kind, State: state, SHA256: "a", Bytes: 1, By: "t", At: time.Now()}
	if err := CreateRecord(ctx, s, rec); err != nil {
		t.Fatal(err)
	}
	return rec
}

func nestedSeedCfg(data release.Store) ReconcileConfig {
	return ReconcileConfig{Region: "ie", Pools: []string{"tito"}, DataDir: "/data",
		Layout: func(context.Context, release.Store) (string, error) { return release.LayoutModeNested, nil },
		Seeds:  &SeedHooks{Data: data}}
}

func TestReconcileMintsAFromSeedNestedWithoutTheSeedCommand(t *testing.T) {
	s, sqid := setup(t) // the manifest declares Seed ["seed"]
	rec := seededRecord(t, s, sqid, seeds.KindReplica, seeds.StateReady)
	reg := regWith(t)
	if err := Reconcile(ctx, s, reg, nestedSeedCfg(dirStore(t))); err != nil {
		t.Fatal(err)
	}
	got := mustGet(t, reg, rec.Host)
	if got.Seed != "" || got.DBPath != NestedDBPath("/data", rec.Host) || got.SeedSource != SeedColumn(rec) {
		t.Fatalf("row = seed %q db %q col %q", got.Seed, got.DBPath, got.SeedSource)
	}
	// The record moves on (placed), and the next pass converges the column
	// and still never stamps the seed command.
	if _, err := SetRecordSeed(ctx, s, rec.Account, rec.App, rec.Host, SeedPlace(rec.Seed.ID)); err != nil {
		t.Fatal(err)
	}
	if err := Reconcile(ctx, s, reg, nestedSeedCfg(dirStore(t))); err != nil {
		t.Fatal(err)
	}
	got = mustGet(t, reg, rec.Host)
	row, _ := seeds.ParseRowSeed(got.SeedSource)
	if got.Seed != "" || row == nil || row.State != seeds.StatePlaced {
		t.Fatalf("after placed: seed %q col %+v", got.Seed, row)
	}
}

func TestReconcileDoesNotMintASeedFlatOrClaimed(t *testing.T) {
	s, sqid := setup(t)
	rec := seededRecord(t, s, sqid, seeds.KindReplica, seeds.StateReady)
	for name, cfg := range map[string]ReconcileConfig{
		"flat":    {Region: "ie", Pools: []string{"tito"}, DataDir: "/data", Seeds: &SeedHooks{Data: dirStore(t)}},
		"claimed": func() ReconcileConfig { c := nestedSeedCfg(dirStore(t)); c.ClaimEnabled = true; return c }(),
		"unwired": {Region: "ie", Pools: []string{"tito"}, DataDir: "/data",
			Layout: func(context.Context, release.Store) (string, error) { return release.LayoutModeNested, nil }},
	} {
		reg := regWith(t)
		if err := Reconcile(ctx, s, reg, cfg); err != nil {
			t.Fatal(err)
		}
		if rt, _ := reg.Get(rec.Host); rt != nil {
			t.Errorf("%s: minted %+v", name, rt)
		}
	}
}

func TestReconcileSkipsSeedsThatAreNotReady(t *testing.T) {
	for _, c := range []struct{ kind, state string }{
		{seeds.KindReplica, seeds.StateReserving},
		{seeds.KindImport, seeds.StateAwaitingUpload},
		{seeds.KindImport, seeds.StateUploaded},
		{seeds.KindImport, seeds.StateRefused},
		{seeds.KindImport, seeds.StateReady}, // Plan 7 swaps; here, nothing
		{seeds.KindReplica, seeds.StateRolledBack},
	} {
		s, sqid := setup(t)
		rec := seededRecord(t, s, sqid, c.kind, c.state)
		reg := regWith(t)
		if err := Reconcile(ctx, s, reg, nestedSeedCfg(dirStore(t))); err != nil {
			t.Fatal(err)
		}
		if rt, _ := reg.Get(rec.Host); rt != nil {
			t.Errorf("%s %s: minted %+v", c.kind, c.state, rt)
		}
	}
}

func TestReconcileFinishesAnAbortedSeedWithNoJournal(t *testing.T) {
	s, sqid := setup(t)
	data := dirStore(t)
	rec := seededRecord(t, s, sqid, seeds.KindReplica, seeds.StateAborted)
	other, _ := seeds.NewID()
	mine := seeds.SnapshotKey(rec.Host, rec.Seed.ID)
	theirs := seeds.SnapshotKey(rec.Host, other)
	for _, k := range []string{mine, theirs} {
		if _, err := data.Put(ctx, k, []byte("db"), ""); err != nil {
			t.Fatal(err)
		}
	}
	reg := regWith(t)
	if err := Reconcile(ctx, s, reg, nestedSeedCfg(data)); err != nil {
		t.Fatal(err)
	}
	if _, _, err := data.Get(ctx, mine); !errors.Is(err, os.ErrNotExist) {
		t.Fatalf("this seed's snapshot: %v, want it deleted", err)
	}
	if _, _, err := data.Get(ctx, theirs); err != nil {
		t.Fatalf("another seed's snapshot was touched: %v", err)
	}
	got, err := GetRecord(ctx, s, rec.Account, rec.App, rec.Host)
	if err != nil || got.Seed.State != seeds.StateRolledBack {
		t.Fatalf("record = %+v, %v; want rolled-back", got.Seed, err)
	}
	// Idempotent: a second pass over a rolled-back record does nothing.
	if err := Reconcile(ctx, s, reg, nestedSeedCfg(data)); err != nil {
		t.Fatal(err)
	}
}
  • [ ] Step 2: Run the tests to verify they fail

Run: go test ./internal/provision/ -run 'TestReconcile.*Seed|TestReconcileSkipsSeeds|TestReconcileFinishesAnAborted' -count=1 Expected: FAIL to compile: unknown field Seeds in struct literal of type ReconcileConfig.

  • [ ] Step 3: Write internal/provision/seedreconcile.go
package provision

import (
	"context"
	"errors"
	"log"
	"os"

	"github.com/carlosframework/platform/internal/release"
	"github.com/carlosframework/platform/internal/seeds"
)

// SeedHooks is what the provision pass needs to act on a seeded record
// (seeded-instances spec §3.2). Nil on ReconcileConfig: no seeded record is
// minted or acted on, which is the safe answer for a box that cannot place
// one.
type SeedHooks struct {
	// Data is this region's data bucket (seeds/ lives there).
	Data release.Store
}

type seedVerdict int

const (
	seedOrdinary seedVerdict = iota
	seedSkip
	seedFinishAbort
)

// seedAction is §3.2's dispatch, run before any other per-record handling.
// hasRow is whether this box already holds a row for the host. nested is
// asked only for a mint, through the pass's lazy layout reader.
func seedAction(rec Record, hasRow bool, cfg ReconcileConfig, nested func() (bool, error)) seedVerdict {
	src := rec.Seed
	if src == nil {
		return seedOrdinary
	}
	if cfg.Seeds == nil {
		return seedSkip
	}
	switch src.State {
	case seeds.StateAborted:
		return seedFinishAbort
	case seeds.StateRolledBack, seeds.StateReserving, seeds.StateAwaitingUpload, seeds.StateUploaded, seeds.StateRefused:
		return seedSkip
	case seeds.StateReady:
		if src.Import() {
			return seedSkip // the swap is Plan 7's
		}
		if hasRow {
			return seedOrdinary
		}
		if cfg.ClaimEnabled {
			log.Printf("provision: %s: not minting a seeded instance: claim control is on for this box", rec.Host)
			return seedSkip
		}
		if ok, err := nested(); err != nil || !ok {
			log.Printf("provision: %s: not minting a seeded instance: this box does not mint nested (%v)", rec.Host, err)
			return seedSkip
		}
		return seedOrdinary
	case seeds.StateSwapped, seeds.StatePlaced, seeds.StateCommitted:
		return seedOrdinary
	}
	log.Printf("provision: %s: seed %s is in an unknown state %q; leaving it alone", rec.Host, src.ID, src.State)
	return seedSkip
}

// finishAbortedSeed is §3.2's no-journal path for an aborted seed: delete
// this seed's snapshot (only this seed's id, codex R4-1), then CAS the
// record to rolled-back. Idempotent: an absent snapshot and a lost CAS are
// both done. The console finishes from rolled-back (Plan 8).
func finishAbortedSeed(ctx context.Context, s release.Store, cfg ReconcileConfig, rec Record) {
	if cfg.DryRun {
		log.Printf("provision: dry-run: would finish aborted seed %s of %s", rec.Seed.ID, rec.Host)
		return
	}
	key := seeds.SnapshotKey(rec.Host, rec.Seed.ID)
	if err := cfg.Seeds.Data.Delete(ctx, key); err != nil && !errors.Is(err, os.ErrNotExist) {
		log.Printf("provision: %s: delete aborted seed's snapshot: %v", rec.Host, err)
		return
	}
	if _, err := SetRecordSeed(ctx, s, rec.Account, rec.App, rec.Host, SeedRolledBack(rec.Seed.ID)); err != nil && !errors.Is(err, ErrSeedConflict) {
		log.Printf("provision: %s: mark seed %s rolled back: %v", rec.Host, rec.Seed.ID, err)
		return
	}
	log.Printf("provision: %s: aborted seed %s rolled back (no box had swapped it)", rec.Host, rec.Seed.ID)
}
  • [ ] Step 4: Wire it into Reconcile
  • ReconcileConfig, after HoldWritten:
	// Seeds is the seeded-instances wiring (seedreconcile.go). Nil: every
	// seeded record is left alone.
	Seeds *SeedHooks
  • routeFor (:425): add SeedSource: SeedColumn(rec), to the registry.Route literal.
  • In the record loop, right after existing, ok := byHost[rec.Host] and before the cfg.Hold block:
		// A seed is dispatched before anything else (seeded-instances spec
		// §3.2, codex 13): an aborted one is finished, an unready one is
		// skipped, and a seeded row never carries the manifest's seed
		// command.
		if rec.Seed != nil {
			switch seedAction(rec, ok, cfg, mintLayout) {
			case seedSkip:
				continue
			case seedFinishAbort:
				finishAbortedSeed(ctx, s, cfg, rec)
				continue
			}
			seed, seedIsKnown = "", true
		}
  • In the owned branch, after the SetSeed converge (:1232-1240):
				if col := SeedColumn(rec); existing.SeedSource != col {
					if cfg.DryRun {
						log.Printf("provision: dry-run: would converge %s seed source", rec.Host)
					} else if err := reg.SetSeedSource(rec.Host, col); err != nil {
						log.Printf("provision: converge seed source on %s: %v", rec.Host, err)
					}
				}
  • [ ] Step 5: Run the tests to verify they pass

Run: go test ./internal/provision/ -run 'TestReconcile|TestSeed' -count=1 Expected: PASS, including every existing reconcile test (an unseeded record mints exactly as before: SeedColumn is "").

  • [ ] Step 6: Commit
git add internal/provision/seedreconcile.go internal/provision/seedreconcile_test.go internal/provision/reconcile.go
git commit -m "provision: reconcile dispatches on the seed state first (Plan 4 Task 5)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>"

Task 6: The box's wiring

Files:

  • Create: internal/host/seedwiring.go, internal/host/seedwiring_test.go
  • Modify: internal/host/agent.go (beside the staging-fallback mirror, :404-410; reconcileSerialized :1131)

Interfaces:

  • Consumes: release.NewS3StoreIn, openDeployStore(ctx), claims.Writes.Enabled, provision.SetRecordSeed, provision.SeedPlace, provision.GetRecord, provision.ErrSeedConflict.
  • Produces: type storeSeedObjects struct{ s release.Store } implementing activator.SeedObjects; seedPlacedHook(deploy, data release.Store) func(ctx context.Context, acct, app, host, id string) error; var armedSeedHooks atomic.Pointer[provision.SeedHooks]; armSeedHooks(h *provision.SeedHooks).
  • [ ] Step 1: Write the failing tests

internal/host/seedwiring_test.go:

package host

import (
	"context"
	"errors"
	"io"
	"os"
	"testing"
	"time"

	"github.com/carlosframework/platform/internal/provision"
	"github.com/carlosframework/platform/internal/release"
	"github.com/carlosframework/platform/internal/seeds"
)

func TestSeedPlacedHookCASesThenDeletesOnlyThisSeed(t *testing.T) {
	ctx := context.Background()
	deploy, data := release.NewDirStore(t.TempDir()), release.NewDirStore(t.TempDir())
	id, _ := seeds.NewID()
	other, _ := seeds.NewID()
	rec := provision.Record{Schema: 1, Host: "canary.go.tito.io", Slug: "canary", App: "titogo", Account: "bae",
		Channel: "stable", CreatedAt: time.Now().UTC(),
		Seed: &provision.SeedSource{ID: id, Kind: seeds.KindReplica, State: seeds.StateReady, By: "t", At: time.Now().UTC()}}
	if err := provision.CreateRecord(ctx, deploy, rec); err != nil {
		t.Fatal(err)
	}
	for _, k := range []string{seeds.SnapshotKey(rec.Host, id), seeds.SnapshotKey(rec.Host, other)} {
		data.Put(ctx, k, []byte("db"), "")
	}
	hook := seedPlacedHook(deploy, data)
	if err := hook(ctx, "bae", "titogo", rec.Host, id); err != nil {
		t.Fatal(err)
	}
	got, _ := provision.GetRecord(ctx, deploy, "bae", "titogo", rec.Host)
	if got.Seed.State != seeds.StatePlaced {
		t.Fatalf("state = %s", got.Seed.State)
	}
	if _, _, err := data.Get(ctx, seeds.SnapshotKey(rec.Host, id)); !errors.Is(err, os.ErrNotExist) {
		t.Fatalf("this seed's snapshot: %v", err)
	}
	if _, _, err := data.Get(ctx, seeds.SnapshotKey(rec.Host, other)); err != nil {
		t.Fatalf("another seed's snapshot: %v", err)
	}
	// Again (a retried wake): done, no error.
	if err := hook(ctx, "bae", "titogo", rec.Host, id); err != nil {
		t.Fatalf("idempotent: %v", err)
	}
	// Another id than the record's: refused, nothing deleted.
	data.Put(ctx, seeds.SnapshotKey(rec.Host, other), []byte("db"), "")
	if err := hook(ctx, "bae", "titogo", rec.Host, other); err == nil {
		t.Fatal("placing another seed's id succeeded")
	}
	if _, _, err := data.Get(ctx, seeds.SnapshotKey(rec.Host, other)); err != nil {
		t.Fatal("a refused placement deleted a snapshot")
	}
}

func TestStoreSeedObjectsOpensTheSnapshotKey(t *testing.T) {
	ctx := context.Background()
	data := release.NewDirStore(t.TempDir())
	id, _ := seeds.NewID()
	data.Put(ctx, seeds.SnapshotKey("h.example.com", id), []byte("db-bytes"), "")
	rc, err := storeSeedObjects{data}.Open(ctx, "h.example.com", id)
	if err != nil {
		t.Fatal(err)
	}
	b, _ := io.ReadAll(rc)
	rc.Close()
	if string(b) != "db-bytes" {
		t.Fatalf("read %q", b)
	}
}
  • [ ] Step 2: Run the tests to verify they fail

Run: go test ./internal/host/ -run 'TestSeedPlacedHook|TestStoreSeedObjects' -count=1 Expected: FAIL to compile.

  • [ ] Step 3: Write internal/host/seedwiring.go
package host

import (
	"context"
	"errors"
	"fmt"
	"io"
	"os"
	"sync/atomic"

	"github.com/carlosframework/platform/internal/provision"
	"github.com/carlosframework/platform/internal/release"
	"github.com/carlosframework/platform/internal/seeds"
)

// storeSeedObjects reads seed snapshots from this box's data bucket.
type storeSeedObjects struct{ s release.Store }

func (o storeSeedObjects) Open(ctx context.Context, host, id string) (io.ReadCloser, error) {
	op, ok := o.s.(release.Opener)
	if !ok {
		return nil, errors.New("the data bucket store cannot stream")
	}
	rc, _, err := op.Open(ctx, seeds.SnapshotKey(host, id))
	return rc, err
}

// seedPlacedHook is ExecBackend.OnSeedPlaced (seeded-instances spec §3.4
// step 5): the record's CAS to placed, then the delete of this seed's
// snapshot, never another's. A record already placed (or committed) with
// this id is a retried wake: the delete still runs.
func seedPlacedHook(deploy, data release.Store) func(ctx context.Context, acct, app, host, id string) error {
	return func(ctx context.Context, acct, app, host, id string) error {
		if deploy == nil || data == nil {
			return errors.New("seed recording is not wired on this box")
		}
		_, err := provision.SetRecordSeed(ctx, deploy, acct, app, host, provision.SeedPlace(id))
		if errors.Is(err, provision.ErrSeedConflict) {
			rec, gerr := provision.GetRecord(ctx, deploy, acct, app, host)
			if gerr != nil {
				return gerr
			}
			if rec.Seed == nil || rec.Seed.ID != id || (rec.Seed.State != seeds.StatePlaced && rec.Seed.State != seeds.StateCommitted) {
				return fmt.Errorf("seed %s of %s: %w", id, host, err)
			}
		} else if err != nil {
			return err
		}
		if err := data.Delete(ctx, seeds.SnapshotKey(host, id)); err != nil && !errors.Is(err, os.ErrNotExist) {
			return err
		}
		return nil
	}
}

// armedSeedHooks is what reconcileSerialized hands the provision pass
// (ReconcileConfig.Seeds). Nil until the activator is armed with a data
// bucket.
var armedSeedHooks atomic.Pointer[provision.SeedHooks]

func armSeedHooks(h *provision.SeedHooks) { armedSeedHooks.Store(h) }
  • [ ] Step 4: Arm it in agent.go

Beside the staging-fallback mirror (agent.go:404), after eb.Mirror = m:

		// Seeded instances (seeded-instances spec §3.4): the data bucket
		// for snapshots, the record store for the placed CAS, and the
		// claim flag that makes the fallback refuse. Best-effort like the
		// mirror: a box that cannot build them places no seed, and a
		// seeded row's wake fails instead of starting empty.
		eb.ClaimArmed = claims.Writes.Enabled
		if data, err := release.NewS3StoreIn(context.Background(), cfg.Bucket, cfg.Region); err != nil {
			log.Printf("agent: seed reader: %v (no seed can be placed on this box)", err)
		} else if deploy, err := openDeployStore(context.Background()); err != nil {
			log.Printf("agent: seed recorder: %v (no seed can be placed on this box)", err)
		} else {
			eb.Seeds = storeSeedObjects{data}
			eb.OnSeedPlaced = seedPlacedHook(deploy, data)
			armSeedHooks(&provision.SeedHooks{Data: data})
		}

In reconcileSerialized (:1131), before cfg = withMovesHolds(cfg):

	if cfg.Seeds == nil {
		cfg.Seeds = armedSeedHooks.Load()
	}
  • [ ] Step 5: Run the tests to verify they pass

Run: go test ./internal/host/ -run 'TestSeedPlacedHook|TestStoreSeedObjects|TestReconcileSerialized|TestAgent' -count=1 Expected: PASS.

  • [ ] Step 6: Commit
git add internal/host/seedwiring.go internal/host/seedwiring_test.go internal/host/agent.go
git commit -m "host: wire seed placement into the activator and the provision pass (Plan 4 Task 6)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>"

Task 7: Plan gate

  • [ ] Step 1: Cross-compile and run the touched packages under -race

Run:

GOOS=darwin go build ./... && GOOS=windows go build ./... && GOOS=linux GOARCH=arm64 go build ./... && \
nice go test -race ./internal/registry/ ./internal/lease/ ./internal/release/ ./internal/activator/ ./internal/provision/ ./internal/host/ . -count=1

Expected: builds clean; tests PASS; the moves copy gate passes (go test . -run TestMoves).

---

Self-review

  • Spec coverage: §3.4 steps 1 to 5 and "Failures" (Tasks 3, 4); "the manifest's seed command is never run for a seeded row" (Task 5 blanks the column; Task 3 never reports fresh); "the snapshot is deleted only after the host's own replica exists" (Task 4: SyncAndWait before the CAS and delete); refusal with claim control armed and on a flat box (Tasks 3 and 5); §3.2's dispatch for every state, with an import's ready and the journal-first branch left to Plan 7; §3.1's seed-ID-scoped delete (Tasks 5 and 6).
  • §9 box placement tests: each failure fails the wake (Task 3), purge of another seed's database (Task 3), crash after rename resumes from the local file (Task 3's second Restore), crash before rename retries from the snapshot (the .tmp is removed and refetched), delete only after the first upload (Task 4), claim refusal (Task 3).
  • Multi-box regions: any box holding a --from row may place it, under the wake lease; finishing an aborted --from seed is idempotent on every box.