amadan

carlos / platform Public

plan: region HA 1a, the foundations that need no claim, and its Astra brief

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Paul Campbell pushed by claude 7a8cf520c252247807b5ba3ff201ab44aa096940 parent 2cb9d12
2 files changed, +4927 −0
  • docs/superpowers/plans/2026-10-05-region-ha-1a-foundations.md +4877 −0
  • docs/superpowers/plans/CODEX-PLAN-REGION-HA-1A-BRIEF-1.md +50 −0

CI failed — run details

diff --git a/docs/superpowers/plans/2026-10-05-region-ha-1a-foundations.md b/docs/superpowers/plans/2026-10-05-region-ha-1a-foundations.md
new file mode 100644
index 000000000..80a6a874a
--- /dev/null
+++ b/docs/superpowers/plans/2026-10-05-region-ha-1a-foundations.md
@@ -0,0 +1,4877 @@
+# Region HA 1a: Foundations Implementation Plan
+
+> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
+
+**Goal:** Build every part of region HA spec 1 that does not need the host lifecycle claim: box identity records, box evidence and choice, stamp-ordered home map, the ordered host-files publisher, wake-state export, termination-fence verification, the installation-complete marker, and the infrastructure they need.
+
+**Architecture:** Eleven small tasks, each inert on a one-box region. Nothing here changes which box runs an instance: that is plan 1b, which wires these into create, reconcile, the activator and the `relocating` operation once the claim (`internal/claim`, not yet on main) lands. Every piece is a pure function or a store-backed type with its own tests.
+
+**Tech Stack:** Go 1.26, module `github.com/carlosframework/platform`; `release.Store` (create-only and ETag compare-and-swap puts); SQLite registry; `golang.org/x/sys/unix` inotify; OpenTofu modules under `infra/modules/`.
+
+**Spec:** `docs/superpowers/specs/2026-10-04-region-ha-box-placement-design.md` (rev 4.2 plus rulings H7 to H9, at 2cb9d12e). Executors read it first: §2 words, §3 invariants, §5.1, §5.4 to §5.7, §5.9, §6.
+
+## Why this is plan 1a and not the whole of spec 1
+
+Spec 1 sits on the host lifecycle claim, which exists only as a spec (rev 6.12 on `design-replica-ownership`; spec 1 was reviewed against rev 6.3) and on move-any-app's install contract (rev 7.12, also unbuilt). There is no `internal/claim` or `internal/hostguard` on main. So spec 1 splits:
+
+- **1a (this plan):** everything with no claim dependency. Mergeable to main now, dormant.
+- **1b (written when the claim's code lands):** create names a box (§5.2), reconcile's three row cases (§5.3), the activator's warm-copy check wired in (§5.5), relocation and the release sequence (§5.8), the multi-box gate (§8), `carlos ops box bind`, the manifest's `files_durability` projection onto routes, and starting the publisher from the agent. Before 1b is written, spec 1 is re-read against the claim's then-current revision.
+
+## Global Constraints
+
+- Nothing in this plan may change behaviour on a one-box region. Every new field is additive and `omitempty`; every reader tolerates its absence (AGENTS.md: "Wire-shape changes are additive").
+- When the edges-file shape changes, every parser changes in the same task: `internal/host`, `internal/edge`, `internal/console`, and check `internal/edgesfile` and `internal/steering` (AGENTS.md: "the consumer list is everything that PARSES the shape").
+- Linux-only syscalls live in `*_linux.go` with `//go:build linux` and a `*_other.go` stub with `//go:build !linux`. CI cross-compiles darwin and windows.
+- Do not add a method to `release.Store` (eleven implementations). Use the existing `Get`/`Put`/`List`/`Delete`; `Put(ctx, key, body, "")` is create-only, a non-empty `ifMatch` is replace-only, and a lost race is `release.ErrCASConflict`.
+- Tests use `release.NewDirStore(t.TempDir())`; inject failures with a struct embedding `release.Store` that overrides one method.
+- No new Go module dependencies. `service/ec2` and `x/sys` are already direct. Do **not** add `service/sqs`, `service/autoscaling` or a PKCS7 library in this plan.
+- No user-facing copy is added here. Log lines are operator-facing and follow the surrounding `agent: …` / `edges: …` style. If a task finds it must add console or CLI copy, stop: that needs the copy-review gate.
+- Commit messages: lowercase `area:` prefix and a sentence stating the behaviour, for example `boxid: a box record verifies only under AWS's signature`. End each with `Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>`.
+- Go test environment on this machine: use the default Go build cache. Do not set `GOCACHE`, `GOMODCACHE` or `GOTMPDIR` under `/tmp`. Scratch files go under `$TMPDIR`. Unix-socket tests need a short `TMPDIR`.
+- Gate before each commit: `go build ./... && go vet ./... && gofmt -l . && go test ./internal/<pkg>/... -count=1`. Full gate before the branch lands: `make test`.
+- Exact values from the spec: continuous deadline **1 second** from the first unpublished modification; repair scan **10 minutes**; polling fallback **1 second**; snapshot interval default **15 minutes**; `files_durability` default **`snapshot`** (H8); capability string **`region-ha-1`**; termination fence accepts **only** state `terminated`.
+
+## Three deliberate departures from the spec's wording
+
+Each keeps the spec's property and is recorded in Task 11 as a spec note.
+
+1. **Exact versions by content hash, not S3 version id.** Spec §5.6 says restore fetches "by the exact versions it lists". `release.Store` has no version-id read and the box role has no `s3:GetObjectVersion`. Objects here are immutable, written create-only under a key that contains their SHA-256, and the manifest records each SHA-256. A reader verifies the hash. That pins content exactly without version ids.
+2. **Identity signature endpoint.** Spec §5.1 says "PKCS7 signature". IMDS also serves `/latest/dynamic/instance-identity/signature`, a base64 RSA-SHA256 signature over the identity document, verifiable with `crypto/x509` and no PKCS7 library. This plan uses that endpoint. Task 1's last step checks it against a real box before the branch lands.
+3. **`snapshot` is a periodic full publish, not a sealed archive.** Spec §5.6 gives the manifest a sealed-archive `base`. Here `hostfiles.SyncDir` stages every difference between the directory and the manifest; run on the snapshot interval it is the `snapshot` setting, with no second format and no whole-directory archive held in memory. The manifest has no `base`.
+
+## Review Focus
+
+1. **A restore path that escapes its directory.** A manifest entry named `../x`, `/etc/x` or `a/../../x` must be refused, never written (Task 5).
+2. **Non-regular files in the files directory.** Symlinks, sockets, FIFOs and devices must be skipped and counted, never followed or opened (Tasks 5, 6, 7).
+3. **A file larger than memory allows.** `release.Store.Put` takes a whole `[]byte`. A file over the 64 MiB cap is skipped and reported, never read whole, and never silently dropped from the manifest (Tasks 5, 7).
+4. **A lookalike termination.** An event or describe result for the right instance id in the wrong account or region, or with state `shutting-down`, `stopped` or missing, must not verify (Task 9).
+5. **A box record whose id is someone else's.** A validly signed identity document for instance A, stored under box id B, must not verify (Task 1).
+
+## File Structure
+
+| Path | Responsibility |
+|---|---|
+| `internal/boxid/record.go` | The box record type, `Publish` (create-only), `Load` |
+| `internal/boxid/verify.go` | `Verifier`: signature, account, region, id binding, legacy bind |
+| `internal/boxid/certs.go`, `certs/*.pem` | AWS regional signing certificates, embedded |
+| `internal/host/edgesfile.go` | New edges-file fields; `Stamp` on `edgeHost` |
+| `internal/host/headroom_linux.go`, `headroom_other.go` | Reading memory, disk and awake counts |
+| `internal/edge/homemap.go` | `Stamp` in the edge's view; stamp-ordered fold |
+| `internal/console/instancesplacement.go` | New fields in the console's view |
+| `internal/console/boxchoice.go` | `eligibleBoxes`, `chooseBox` |
+| `internal/hostfiles/manifest.go` | Manifest and entry types, keys, path validation |
+| `internal/hostfiles/publisher.go` | Stage, commit (CAS), tombstones |
+| `internal/hostfiles/restore.go` | Restore a directory from a manifest |
+| `internal/hostfiles/sync.go` | `SyncDir`: stage a whole directory's differences; `readStable` |
+| `internal/hostfiles/run.go`, `deadline.go`, `watch_linux.go`, `watch_other.go` | The publish loop, deadline queue, inotify watcher, polling fallback |
+| `internal/registry/timers.go` | `ReplaceTimers` |
+| `internal/hostfiles/wake.go` | Wake state: timers, `NextDue`, `Replication` |
+| `internal/termfence/fence.go` | Fence type, the three evidence sources, `Verify` |
+| `internal/activator/installed.go` | Installation-complete marker and `warmCopyTrusted` |
+| `infra/modules/region-host/*.tf`, `infra/modules/deployment-buckets/*.tf` | Volume flag, termination queue, IAM, bucket policies |
+
+---
+
+### Task 1: Box records and their verification (`internal/boxid`)
+
+**Files:**
+- Create: `internal/boxid/record.go`, `internal/boxid/verify.go`, `internal/boxid/certs.go`, `internal/boxid/certs/README.md`
+- Test: `internal/boxid/boxid_test.go`
+
+**Interfaces:**
+- Consumes: `release.Store`.
+- Produces:
+ ```go
+ package boxid
+ const Prefix = "control/boxes/"
+ func Key(box string) string // Prefix + box + ".json"
+ func BindKey(box string) string // Prefix + box + ".bind.json"
+ type Record struct {
+ V int `json:"v"`
+ Box string `json:"box"`
+ IdentityDocument string `json:"identity_document"` // base64 of the raw IMDS document bytes
+ IdentitySignature string `json:"identity_signature"` // base64 RSA-SHA256, as IMDS serves it
+ WrittenAt string `json:"written_at"`
+ }
+ type Identity struct{ AccountID, Region, InstanceID string }
+ type Bind struct {
+ RecordSHA256 string `json:"record_sha256"`
+ InstanceID string `json:"instance_id"`
+ TofuInstanceID string `json:"tofu_instance_id"`
+ Operator string `json:"operator"`
+ At string `json:"at"`
+ }
+ type Verifier struct {
+ Account string // the deployment's AWS account id
+ Certs map[string]*x509.Certificate // by AWS region
+ }
+ func (v Verifier) Verify(rec Record, raw []byte, bind *Bind) (Identity, error)
+ func Publish(ctx context.Context, s release.Store, rec Record) error // create-only; an identical existing record is success
+ func Load(ctx context.Context, s release.Store, box string) (Record, []byte, *Bind, error)
+ func EmbeddedCerts() (map[string]*x509.Certificate, error)
+ var ErrUnverified = errors.New("boxid: record does not verify")
+ ```
+ `Bind.RecordSHA256` replaces the spec's `record_version_id` (see "Three deliberate departures").
+
+- [ ] **Step 1: Write the failing tests**
+
+`internal/boxid/boxid_test.go`:
+
+```go
+package boxid
+
+import (
+ "context"
+ "crypto"
+ "crypto/rand"
+ "crypto/rsa"
+ "crypto/sha256"
+ "crypto/x509"
+ "crypto/x509/pkix"
+ "encoding/base64"
+ "encoding/hex"
+ "encoding/json"
+ "errors"
+ "math/big"
+ "testing"
+ "time"
+
+ "github.com/carlosframework/platform/internal/release"
+)
+
+// signer stands in for AWS: one key and certificate per test.
+type signer struct {
+ key *rsa.PrivateKey
+ cert *x509.Certificate
+}
+
+func newSigner(t *testing.T) signer {
+ t.Helper()
+ key, err := rsa.GenerateKey(rand.Reader, 2048)
+ if err != nil {
+ t.Fatal(err)
+ }
+ tmpl := &x509.Certificate{SerialNumber: big.NewInt(1), Subject: pkix.Name{CommonName: "test"},
+ NotBefore: time.Now().Add(-time.Hour), NotAfter: time.Now().Add(time.Hour)}
+ der, err := x509.CreateCertificate(rand.Reader, tmpl, tmpl, &key.PublicKey, key)
+ if err != nil {
+ t.Fatal(err)
+ }
+ cert, err := x509.ParseCertificate(der)
+ if err != nil {
+ t.Fatal(err)
+ }
+ return signer{key, cert}
+}
+
+func (s signer) record(t *testing.T, box, account, region, instance string) (Record, []byte) {
+ t.Helper()
+ doc, _ := json.Marshal(map[string]string{"accountId": account, "region": region, "instanceId": instance})
+ sum := sha256.Sum256(doc)
+ sig, err := rsa.SignPKCS1v15(rand.Reader, s.key, crypto.SHA256, sum[:])
+ if err != nil {
+ t.Fatal(err)
+ }
+ rec := Record{V: 1, Box: box, IdentityDocument: base64.StdEncoding.EncodeToString(doc),
+ IdentitySignature: base64.StdEncoding.EncodeToString(sig), WrittenAt: "2026-10-05T00:00:00Z"}
+ raw, _ := json.Marshal(rec)
+ return rec, raw
+}
+
+func (s signer) verifier() Verifier {
+ return Verifier{Account: "111122223333", Certs: map[string]*x509.Certificate{"eu-west-1": s.cert}}
+}
+
+func TestAutoScalingBoxVerifiesWhenIDIsItsInstance(t *testing.T) {
+ s := newSigner(t)
+ rec, raw := s.record(t, "i-0abc", "111122223333", "eu-west-1", "i-0abc")
+ id, err := s.verifier().Verify(rec, raw, nil)
+ if err != nil {
+ t.Fatal(err)
+ }
+ if id.InstanceID != "i-0abc" || id.Region != "eu-west-1" || id.AccountID != "111122223333" {
+ t.Fatalf("identity = %+v", id)
+ }
+}
+
+// Review Focus 5: a valid document for instance A under box id B.
+func TestRecordUnderAnotherInstancesIDIsRefused(t *testing.T) {
+ s := newSigner(t)
+ rec, raw := s.record(t, "i-0other", "111122223333", "eu-west-1", "i-0abc")
+ if _, err := s.verifier().Verify(rec, raw, nil); !errors.Is(err, ErrUnverified) {
+ t.Fatalf("err = %v, want ErrUnverified", err)
+ }
+}
+
+func TestRefusals(t *testing.T) {
+ s, other := newSigner(t), newSigner(t)
+ cases := map[string]func() (Record, []byte, Verifier){
+ "wrong account": func() (Record, []byte, Verifier) {
+ r, raw := s.record(t, "i-0abc", "999999999999", "eu-west-1", "i-0abc")
+ return r, raw, s.verifier()
+ },
+ "region with no certificate": func() (Record, []byte, Verifier) {
+ r, raw := s.record(t, "i-0abc", "111122223333", "ap-south-1", "i-0abc")
+ return r, raw, s.verifier()
+ },
+ "signed by someone else": func() (Record, []byte, Verifier) {
+ r, raw := other.record(t, "i-0abc", "111122223333", "eu-west-1", "i-0abc")
+ return r, raw, s.verifier()
+ },
+ "document altered after signing": func() (Record, []byte, Verifier) {
+ r, _ := s.record(t, "i-0abc", "111122223333", "eu-west-1", "i-0abc")
+ doc, _ := json.Marshal(map[string]string{"accountId": "111122223333", "region": "eu-west-1", "instanceId": "i-0zzz"})
+ r.IdentityDocument, r.Box = base64.StdEncoding.EncodeToString(doc), "i-0zzz"
+ raw, _ := json.Marshal(r)
+ return r, raw, s.verifier()
+ },
+ "empty box": func() (Record, []byte, Verifier) {
+ r, raw := s.record(t, "", "111122223333", "eu-west-1", "i-0abc")
+ return r, raw, s.verifier()
+ },
+ }
+ for name, mk := range cases {
+ t.Run(name, func(t *testing.T) {
+ r, raw, v := mk()
+ if _, err := v.Verify(r, raw, nil); !errors.Is(err, ErrUnverified) {
+ t.Fatalf("err = %v, want ErrUnverified", err)
+ }
+ })
+ }
+}
+
+func TestLegacyLabelNeedsABindNamingThisExactRecord(t *testing.T) {
+ s := newSigner(t)
+ rec, raw := s.record(t, "annie-welding", "111122223333", "eu-west-1", "i-0abc")
+ if _, err := s.verifier().Verify(rec, raw, nil); !errors.Is(err, ErrUnverified) {
+ t.Fatalf("no bind: err = %v, want ErrUnverified", err)
+ }
+ sum := sha256.Sum256(raw)
+ good := &Bind{RecordSHA256: hex.EncodeToString(sum[:]), InstanceID: "i-0abc", TofuInstanceID: "i-0abc", Operator: "paul", At: "2026-10-05T00:00:00Z"}
+ if _, err := s.verifier().Verify(rec, raw, good); err != nil {
+ t.Fatalf("good bind: %v", err)
+ }
+ for name, b := range map[string]Bind{
+ "another record's hash": {RecordSHA256: "00", InstanceID: "i-0abc", TofuInstanceID: "i-0abc", Operator: "paul"},
+ "tofu names another box": {RecordSHA256: good.RecordSHA256, InstanceID: "i-0abc", TofuInstanceID: "i-0zzz", Operator: "paul"},
+ "bind names another box": {RecordSHA256: good.RecordSHA256, InstanceID: "i-0zzz", TofuInstanceID: "i-0zzz", Operator: "paul"},
+ "no operator": {RecordSHA256: good.RecordSHA256, InstanceID: "i-0abc", TofuInstanceID: "i-0abc"},
+ } {
+ b := b
+ t.Run(name, func(t *testing.T) {
+ if _, err := s.verifier().Verify(rec, raw, &b); !errors.Is(err, ErrUnverified) {
+ t.Fatalf("err = %v, want ErrUnverified", err)
+ }
+ })
+ }
+}
+
+func TestPublishIsCreateOnlyAndIdempotent(t *testing.T) {
+ ctx, st, s := context.Background(), release.NewDirStore(t.TempDir()), newSigner(t)
+ rec, _ := s.record(t, "i-0abc", "111122223333", "eu-west-1", "i-0abc")
+ if err := Publish(ctx, st, rec); err != nil {
+ t.Fatal(err)
+ }
+ if err := Publish(ctx, st, rec); err != nil {
+ t.Fatalf("republishing the same record: %v", err)
+ }
+ other, _ := s.record(t, "i-0abc", "111122223333", "eu-west-1", "i-0abc")
+ other.WrittenAt = "2027-01-01T00:00:00Z"
+ if err := Publish(ctx, st, other); !errors.Is(err, release.ErrCASConflict) {
+ t.Fatalf("a different record for the same box: err = %v, want ErrCASConflict", err)
+ }
+ got, raw, bind, err := Load(ctx, st, "i-0abc")
+ if err != nil || got.WrittenAt != rec.WrittenAt || len(raw) == 0 || bind != nil {
+ t.Fatalf("Load = %+v, %d bytes, bind %v, err %v", got, len(raw), bind, err)
+ }
+}
+
+func TestLoadReadsABindWhenThereIsOne(t *testing.T) {
+ ctx, st, s := context.Background(), release.NewDirStore(t.TempDir()), newSigner(t)
+ rec, _ := s.record(t, "annie-welding", "111122223333", "eu-west-1", "i-0abc")
+ if err := Publish(ctx, st, rec); err != nil {
+ t.Fatal(err)
+ }
+ body, _ := json.Marshal(Bind{RecordSHA256: "aa", InstanceID: "i-0abc", TofuInstanceID: "i-0abc", Operator: "paul"})
+ if _, err := st.Put(ctx, BindKey("annie-welding"), body, ""); err != nil {
+ t.Fatal(err)
+ }
+ _, _, bind, err := Load(ctx, st, "annie-welding")
+ if err != nil || bind == nil || bind.Operator != "paul" {
+ t.Fatalf("bind = %+v, err %v", bind, err)
+ }
+}
+```
+
+- [ ] **Step 2: Run the tests to see them fail**
+
+Run: `go test ./internal/boxid/ -count=1`
+Expected: FAIL to build, `undefined: Record`.
+
+- [ ] **Step 3: Write `record.go`**
+
+```go
+// Package boxid binds the name a claim uses for a box to one AWS instance
+// (region HA spec 1 §5.1). The binding is the box's own AWS-signed identity
+// document, written once. Nothing else is evidence of a box's identity: not
+// an edges file, not a status snapshot.
+package boxid
+
+import (
+ "bytes"
+ "context"
+ "encoding/json"
+ "errors"
+ "fmt"
+ "os"
+
+ "github.com/carlosframework/platform/internal/release"
+)
+
+const Prefix = "control/boxes/"
+
+func Key(box string) string { return Prefix + box + ".json" }
+func BindKey(box string) string { return Prefix + box + ".bind.json" }
+
+type Record struct {
+ V int `json:"v"`
+ Box string `json:"box"`
+ IdentityDocument string `json:"identity_document"`
+ IdentitySignature string `json:"identity_signature"`
+ WrittenAt string `json:"written_at"`
+}
+
+// Bind is an operator's statement that a legacy label belongs to one
+// instance. It names the exact record bytes it vouches for.
+type Bind struct {
+ RecordSHA256 string `json:"record_sha256"`
+ InstanceID string `json:"instance_id"`
+ TofuInstanceID string `json:"tofu_instance_id"`
+ Operator string `json:"operator"`
+ At string `json:"at"`
+}
+
+// Publish writes a box's record create-only. Writing the same bytes again
+// succeeds; a different record for the same box is ErrCASConflict, because
+// a record is never replaced.
+func Publish(ctx context.Context, s release.Store, rec Record) error {
+ body, err := json.Marshal(rec)
+ if err != nil {
+ return fmt.Errorf("boxid: marshal: %w", err)
+ }
+ _, err = s.Put(ctx, Key(rec.Box), body, "")
+ if !errors.Is(err, release.ErrCASConflict) {
+ return err
+ }
+ cur, _, gerr := s.Get(ctx, Key(rec.Box))
+ if gerr != nil {
+ return fmt.Errorf("boxid: %s exists and cannot be read: %w", rec.Box, gerr)
+ }
+ if bytes.Equal(cur, body) {
+ return nil
+ }
+ return fmt.Errorf("boxid: %s already has a different record: %w", rec.Box, release.ErrCASConflict)
+}
+
+// Load returns a box's record, its exact bytes (which a Bind's hash is over)
+// and its bind when one exists. A missing record is os.ErrNotExist-wrapped.
+func Load(ctx context.Context, s release.Store, box string) (Record, []byte, *Bind, error) {
+ raw, _, err := s.Get(ctx, Key(box))
+ if err != nil {
+ return Record{}, nil, nil, err
+ }
+ var rec Record
+ if err := json.Unmarshal(raw, &rec); err != nil {
+ return Record{}, nil, nil, fmt.Errorf("boxid: %s: %w", box, err)
+ }
+ braw, _, err := s.Get(ctx, BindKey(box))
+ if errors.Is(err, os.ErrNotExist) {
+ return rec, raw, nil, nil
+ }
+ if err != nil {
+ return Record{}, nil, nil, err
+ }
+ var b Bind
+ if err := json.Unmarshal(braw, &b); err != nil {
+ return Record{}, nil, nil, fmt.Errorf("boxid: %s bind: %w", box, err)
+ }
+ return rec, raw, &b, nil
+}
+```
+
+- [ ] **Step 4: Write `verify.go`**
+
+```go
+package boxid
+
+import (
+ "crypto/sha256"
+ "crypto/x509"
+ "encoding/base64"
+ "encoding/hex"
+ "encoding/json"
+ "errors"
+ "fmt"
+)
+
+// ErrUnverified wraps every reason a record is not evidence of identity.
+var ErrUnverified = errors.New("boxid: record does not verify")
+
+type Identity struct{ AccountID, Region, InstanceID string }
+
+type Verifier struct {
+ Account string
+ Certs map[string]*x509.Certificate
+}
+
+func refuse(format string, a ...any) error {
+ return fmt.Errorf("%w: %s", ErrUnverified, fmt.Sprintf(format, a...))
+}
+
+// Verify checks, in order: the signature under the certificate for the
+// document's own region, the deployment account, and the binding of the
+// record's box id to the signed instance id. A box whose id is its instance
+// id needs nothing more. Any other id is a legacy label and needs a bind
+// that names these exact record bytes and agrees on the instance.
+func (v Verifier) Verify(rec Record, raw []byte, bind *Bind) (Identity, error) {
+ if rec.Box == "" {
+ return Identity{}, refuse("no box id")
+ }
+ doc, err := base64.StdEncoding.DecodeString(rec.IdentityDocument)
+ if err != nil {
+ return Identity{}, refuse("document is not base64")
+ }
+ sig, err := base64.StdEncoding.DecodeString(rec.IdentitySignature)
+ if err != nil {
+ return Identity{}, refuse("signature is not base64")
+ }
+ var d struct {
+ AccountID string `json:"accountId"`
+ Region string `json:"region"`
+ InstanceID string `json:"instanceId"`
+ }
+ if err := json.Unmarshal(doc, &d); err != nil {
+ return Identity{}, refuse("document is not JSON")
+ }
+ cert := v.Certs[d.Region]
+ if cert == nil {
+ return Identity{}, refuse("no signing certificate for region %q", d.Region)
+ }
+ if err := cert.CheckSignature(x509.SHA256WithRSA, doc, sig); err != nil {
+ return Identity{}, refuse("signature: %v", err)
+ }
+ if d.InstanceID == "" {
+ return Identity{}, refuse("document names no instance")
+ }
+ if v.Account == "" || d.AccountID != v.Account {
+ return Identity{}, refuse("account %q is not the deployment's", d.AccountID)
+ }
+ id := Identity{AccountID: d.AccountID, Region: d.Region, InstanceID: d.InstanceID}
+ if rec.Box == d.InstanceID {
+ return id, nil
+ }
+ if bind == nil {
+ return Identity{}, refuse("%q is a legacy label with no operator bind", rec.Box)
+ }
+ sum := sha256.Sum256(raw)
+ switch {
+ case bind.RecordSHA256 != hex.EncodeToString(sum[:]):
+ return Identity{}, refuse("bind is for a different record")
+ case bind.InstanceID != d.InstanceID:
+ return Identity{}, refuse("bind names instance %q, the record %q", bind.InstanceID, d.InstanceID)
+ case bind.TofuInstanceID != d.InstanceID:
+ return Identity{}, refuse("tofu records instance %q, the record %q", bind.TofuInstanceID, d.InstanceID)
+ case bind.Operator == "":
+ return Identity{}, refuse("bind names no operator")
+ }
+ return id, nil
+}
+```
+
+- [ ] **Step 5: Write `certs.go` and add the certificates**
+
+```go
+package boxid
+
+import (
+ "crypto/x509"
+ "embed"
+ "encoding/pem"
+ "fmt"
+ "strings"
+)
+
+//go:embed certs/*.pem
+var certFS embed.FS
+
+// EmbeddedCerts returns AWS's instance-identity signing certificates by
+// region, from certs/<aws-region>.pem. A region with no file has no
+// certificate, so its boxes never verify: that is the safe default.
+func EmbeddedCerts() (map[string]*x509.Certificate, error) {
+ entries, err := certFS.ReadDir("certs")
+ if err != nil {
+ return nil, err
+ }
+ out := map[string]*x509.Certificate{}
+ for _, e := range entries {
+ region, ok := strings.CutSuffix(e.Name(), ".pem")
+ if !ok {
+ continue
+ }
+ raw, err := certFS.ReadFile("certs/" + e.Name())
+ if err != nil {
+ return nil, err
+ }
+ block, _ := pem.Decode(raw)
+ if block == nil {
+ return nil, fmt.Errorf("boxid: certs/%s is not PEM", e.Name())
+ }
+ cert, err := x509.ParseCertificate(block.Bytes)
+ if err != nil {
+ return nil, fmt.Errorf("boxid: certs/%s: %w", e.Name(), err)
+ }
+ out[region] = cert
+ }
+ return out, nil
+}
+```
+
+Download the four regional certificates the flagship uses. They are on the AWS page "AWS public certificates for instance identity signatures", https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/regions-certs.html . For each of `eu-west-1`, `eu-west-2`, `us-east-1` and `ap-southeast-2`, open the region, take the certificate under the **"Base64-encoded signature"** tab (not the PKCS7 or RSA-2048 tabs), and save it verbatim, including the `BEGIN`/`END CERTIFICATE` lines, as `internal/boxid/certs/<region>.pem`.
+
+Write `internal/boxid/certs/README.md`:
+
+```markdown
+AWS's public certificates for the base64-encoded instance identity
+signature (IMDS `/latest/dynamic/instance-identity/signature`), one per
+region, from
+https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/regions-certs.html
+(the "Base64-encoded signature" tab). A region with no file here has no
+certificate, and a box there never verifies. Add a region's file before
+running a second box in it.
+```
+
+Append to `boxid_test.go`:
+
+```go
+func TestEmbeddedCertsCoverTheFlagshipRegions(t *testing.T) {
+ certs, err := EmbeddedCerts()
+ if err != nil {
+ t.Fatal(err)
+ }
+ for _, region := range []string{"eu-west-1", "eu-west-2", "us-east-1", "ap-southeast-2"} {
+ c := certs[region]
+ if c == nil {
+ t.Errorf("no certificate for %s", region)
+ continue
+ }
+ if c.PublicKeyAlgorithm != x509.RSA {
+ t.Errorf("%s: %v key; the base64-signature certificate is RSA (the PKCS7 one is DSA: wrong tab)", region, c.PublicKeyAlgorithm)
+ }
+ }
+}
+```
+
+- [ ] **Step 6: Run the tests**
+
+Run: `go test ./internal/boxid/ -count=1`
+Expected: PASS.
+
+- [ ] **Step 7: Check the signature endpoint against a real box**
+
+This is the one step that needs AWS. On the dev box (ask Paul for the session; do not run it against a production box unprompted), run:
+
+```bash
+TOKEN=$(curl -sX PUT http://169.254.169.254/latest/api/token -H 'X-aws-ec2-metadata-token-ttl-seconds: 60')
+curl -sH "X-aws-ec2-metadata-token: $TOKEN" http://169.254.169.254/latest/dynamic/instance-identity/document > doc.json
+curl -sH "X-aws-ec2-metadata-token: $TOKEN" http://169.254.169.254/latest/dynamic/instance-identity/signature | base64 -d > sig.bin
+openssl x509 -in <region>.pem -pubkey -noout > pub.pem
+openssl dgst -sha256 -verify pub.pem -signature sig.bin doc.json
+```
+
+Expected: `Verified OK`. If it prints `Verification Failure`, stop and report: the certificate tab or the digest is wrong, and `Verify` must not ship until this passes. Record the result in the task's commit message.
+
+- [ ] **Step 8: Commit**
+
+```bash
+git add internal/boxid
+git commit -m "boxid: a box record verifies only under AWS's signature, for its own instance
+
+Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>"
+```
+
+---
+
+### Task 2: Box evidence in the edges file
+
+**Files:**
+- Modify: `internal/host/edgesfile.go` (struct at `:156-232`, ident at `:638-650`), `internal/console/instancesplacement.go:38`
+- Create: `internal/host/headroom_linux.go`, `internal/host/headroom_other.go`
+- Test: `internal/host/edgesfile_test.go`, `internal/host/headroom_test.go`, `internal/console/regioncandidates_test.go`
+
+**Interfaces:**
+- Consumes: `status.InstanceID() string`; `edgesFilePass(ctx, s, reg, ident, eip, &last)`.
+- Produces, in `internal/host`:
+ ```go
+ type edgeHeadroom struct {
+ MemFreeMiB int64 `json:"mem_free_mib"`
+ DiskFreeMiB int64 `json:"disk_free_mib"`
+ Awake int `json:"awake"`
+ }
+ // new edgesFile fields:
+ BoxID string `json:"box_id,omitempty"`
+ Capabilities []string `json:"capabilities,omitempty"`
+ Draining bool `json:"draining,omitempty"`
+ Headroom *edgeHeadroom `json:"headroom,omitempty"`
+ const capRegionHA1 = "region-ha-1"
+ func edgesFileBoxID(instanceID, label string) string
+ func edgesFileCapabilities(getenv func(string) string) []string
+ func readHeadroom(dataDir string, awake int) *edgeHeadroom
+ ```
+ and in `internal/console`, on `edgesFilePlacementView`:
+ ```go
+ BoxID string `json:"box_id"`
+ Capabilities []string `json:"capabilities"`
+ Draining bool `json:"draining"`
+ Headroom *struct {
+ MemFreeMiB int64 `json:"mem_free_mib"`
+ DiskFreeMiB int64 `json:"disk_free_mib"`
+ Awake int `json:"awake"`
+ } `json:"headroom"`
+ ```
+
+Headroom changes constantly, so it must not defeat the pass's "nothing changed, skip the write" gate. The gate compares a marshalled file with `Generation` blanked; this task blanks `Headroom` in the gate copy too, so headroom is refreshed only when the file is written for another reason or on the loop's five-minute tick.
+
+- [ ] **Step 1: Write the failing tests**
+
+Append to `internal/host/edgesfile_test.go`:
+
+```go
+func TestEdgesFileBoxIDIsTheInstanceIDWhenThereIsOne(t *testing.T) {
+ if got := edgesFileBoxID("i-0abc", "annie-welding"); got != "i-0abc" {
+ t.Fatalf("on EC2: %q", got)
+ }
+ if got := edgesFileBoxID("", "annie-welding"); got != "annie-welding" {
+ t.Fatalf("off EC2: %q", got)
+ }
+}
+
+func TestEdgesFileCapabilityFollowsTheFlag(t *testing.T) {
+ env := func(v string) func(string) string {
+ return func(k string) string {
+ if k == "CARLOS_REGION_HA" {
+ return v
+ }
+ return ""
+ }
+ }
+ if got := edgesFileCapabilities(env("")); len(got) != 0 {
+ t.Fatalf("unset: %v", got)
+ }
+ if got := edgesFileCapabilities(env("1")); len(got) != 1 || got[0] != "region-ha-1" {
+ t.Fatalf("set: %v", got)
+ }
+ if got := edgesFileCapabilities(env("0")); len(got) != 0 {
+ t.Fatalf("\"0\" is off: %v", got)
+ }
+}
+
+func TestEdgesFileRawWireBodyCarriesBoxEvidence(t *testing.T) {
+ s, reg := release.NewDirStore(t.TempDir()), edgesTestRegistry(t)
+ seedRoute(t, reg, "app.example.com")
+ var last []byte
+ ident := edgesFile{Box: "annie", BoxID: "i-0abc", Capabilities: []string{"region-ha-1"},
+ Headroom: &edgeHeadroom{MemFreeMiB: 2210, DiskFreeMiB: 31200, Awake: 14}}
+ if err := edgesFilePass(context.Background(), s, reg, ident, "1.2.3.4", &last); err != nil {
+ t.Fatal(err)
+ }
+ body, _, err := s.Get(context.Background(), "control/edges/annie.json")
+ if err != nil {
+ t.Fatal(err)
+ }
+ var raw map[string]json.RawMessage
+ if err := json.Unmarshal(body, &raw); err != nil {
+ t.Fatal(err)
+ }
+ for key, want := range map[string]string{
+ "box_id": `"i-0abc"`,
+ "capabilities": `["region-ha-1"]`,
+ "headroom": `{"mem_free_mib":2210,"disk_free_mib":31200,"awake":14}`,
+ } {
+ if string(raw[key]) != want {
+ t.Errorf("%s = %s, want %s", key, raw[key], want)
+ }
+ }
+ if _, ok := raw["draining"]; ok {
+ t.Error("draining:false must be omitted")
+ }
+}
+
+func TestEdgesFileOmitsBoxEvidenceWhenAbsent(t *testing.T) {
+ s, reg := release.NewDirStore(t.TempDir()), edgesTestRegistry(t)
+ var last []byte
+ if err := edgesFilePass(context.Background(), s, reg, edgesFile{Box: "annie"}, "1.2.3.4", &last); err != nil {
+ t.Fatal(err)
+ }
+ body, _, _ := s.Get(context.Background(), "control/edges/annie.json")
+ for _, key := range []string{"box_id", "capabilities", "draining", "headroom"} {
+ if bytes.Contains(body, []byte(`"`+key+`"`)) {
+ t.Errorf("%s present in %s", key, body)
+ }
+ }
+}
+
+func TestEdgesFileHeadroomAloneDoesNotRewriteTheFile(t *testing.T) {
+ st := &putCountingStore{Store: release.NewDirStore(t.TempDir())}
+ reg := edgesTestRegistry(t)
+ seedRoute(t, reg, "app.example.com")
+ var last []byte
+ ident := edgesFile{Box: "annie", Headroom: &edgeHeadroom{MemFreeMiB: 100}}
+ if err := edgesFilePass(context.Background(), st, reg, ident, "1.2.3.4", &last); err != nil {
+ t.Fatal(err)
+ }
+ ident.Headroom = &edgeHeadroom{MemFreeMiB: 99}
+ if err := edgesFilePass(context.Background(), st, reg, ident, "1.2.3.4", &last); err != nil {
+ t.Fatal(err)
+ }
+ if st.puts != 1 {
+ t.Fatalf("puts = %d, want 1: a headroom change alone is not a reason to write", st.puts)
+ }
+}
+```
+
+Create `internal/host/headroom_test.go`:
+
+```go
+package host
+
+import "testing"
+
+func TestParseMemAvailable(t *testing.T) {
+ const meminfo = "MemTotal: 16000000 kB\nMemFree: 500000 kB\nMemAvailable: 2263040 kB\n"
+ if got := parseMemAvailableMiB(meminfo); got != 2210 {
+ t.Fatalf("got %d, want 2210", got)
+ }
+ if got := parseMemAvailableMiB("MemTotal: 1 kB\n"); got != 0 {
+ t.Fatalf("no MemAvailable line: got %d, want 0", got)
+ }
+ if got := parseMemAvailableMiB("MemAvailable: lots kB\n"); got != 0 {
+ t.Fatalf("unparseable: got %d, want 0", got)
+ }
+}
+```
+
+Append to `internal/console/regioncandidates_test.go`:
+
+```go
+func TestEdgesViewParsesBoxEvidenceAndToleratesItsAbsence(t *testing.T) {
+ ctx, s := context.Background(), release.NewDirStore(t.TempDir())
+ with := `{"v":1,"box":"i-0abc","eip":"1.2.3.4","region":"eu-west-2","placement":"uk","hosts":{},"box_id":"i-0abc","capabilities":["region-ha-1"],"draining":true,"headroom":{"mem_free_mib":2210,"disk_free_mib":31200,"awake":14}}`
+ without := `{"v":1,"box":"old-box","eip":"5.6.7.8","region":"eu-west-1","hosts":{}}`
+ for k, b := range map[string]string{"i-0abc": with, "old-box": without} {
+ if _, err := s.Put(ctx, "control/edges/"+k+".json", []byte(b), ""); err != nil {
+ t.Fatal(err)
+ }
+ }
+ snap, err := readEdgesSnapshot(ctx, s)
+ if err != nil || !snap.Complete {
+ t.Fatalf("snap complete=%v err=%v", snap.Complete, err)
+ }
+ by := map[string]edgesFilePlacementView{}
+ for _, f := range snap.Files {
+ by[f.Box] = f
+ }
+ n := by["i-0abc"]
+ if n.BoxID != "i-0abc" || len(n.Capabilities) != 1 || !n.Draining || n.Headroom == nil || n.Headroom.Awake != 14 {
+ t.Fatalf("new box parsed as %+v", n)
+ }
+ o := by["old-box"]
+ if o.BoxID != "" || o.Capabilities != nil || o.Draining || o.Headroom != nil {
+ t.Fatalf("old box parsed as %+v", o)
+ }
+}
+```
+
+- [ ] **Step 2: Run them to see them fail**
+
+Run: `go test ./internal/host/ -run 'EdgesFile(BoxID|Capability|RawWireBodyCarries|OmitsBoxEvidence|Headroom)|ParseMemAvailable' -count=1`
+Expected: FAIL to build, `undefined: edgesFileBoxID`.
+
+- [ ] **Step 3: Add the fields and helpers to `internal/host/edgesfile.go`**
+
+After `SecretsKeyFP` (`:228`) in `edgesFile`:
+
+```go
+ // BoxID is the name claims use for this box (region HA spec 1 §2): the
+ // EC2 instance id, or the label off EC2. It is NOT evidence of identity:
+ // that is the box record (internal/boxid). A reader compares the two and
+ // treats a mismatch as an ineligible box.
+ BoxID string `json:"box_id,omitempty"`
+ // Capabilities gates multi-box regions (spec §8): "region-ha-1" once
+ // CARLOS_REGION_HA is on.
+ Capabilities []string `json:"capabilities,omitempty"`
+ // Draining is set while the placer empties this box (spec 4).
+ Draining bool `json:"draining,omitempty"`
+ // Headroom is what placement chooses by (spec 2). It is refreshed when
+ // the file is written for another reason and on the loop's own tick; a
+ // headroom change alone never forces a write (see edgesFilePass).
+ Headroom *edgeHeadroom `json:"headroom,omitempty"`
+```
+
+Above `edgesFile`:
+
+```go
+// edgeHeadroom is this box's spare capacity at the time the file was
+// written. Zero values mean "could not read", never "none left": a chooser
+// ranks an unreadable box last, it does not exclude it.
+type edgeHeadroom struct {
+ MemFreeMiB int64 `json:"mem_free_mib"`
+ DiskFreeMiB int64 `json:"disk_free_mib"`
+ Awake int `json:"awake"`
+}
+
+const capRegionHA1 = "region-ha-1"
+
+// edgesFileBoxID is the instance id on EC2 and the label everywhere else.
+func edgesFileBoxID(instanceID, label string) string {
+ if instanceID != "" {
+ return instanceID
+ }
+ return label
+}
+
+// edgesFileCapabilities publishes region-ha-1 only when CARLOS_REGION_HA is
+// set to something other than "" or "0".
+func edgesFileCapabilities(getenv func(string) string) []string {
+ switch strings.TrimSpace(getenv("CARLOS_REGION_HA")) {
+ case "", "0":
+ return nil
+ }
+ return []string{capRegionHA1}
+}
+```
+
+In `edgesFilePass`, change the gate (`:485`) so headroom is not part of it:
+
+```go
+ f := ident
+ f.V, f.EIP, f.Hosts, f.Generation = 1, eip, hosts, ""
+ gateCopy := f
+ gateCopy.Headroom = nil // a changing reading is not a reason to write
+ gate, err := json.Marshal(gateCopy)
+ if err != nil {
+ return fmt.Errorf("edges: marshal: %w", err)
+ }
+ if bytes.Equal(gate, *last) {
+ return nil
+ }
+```
+
+Leave the rest of the function as it is: `body` is still marshalled from `f`, which keeps its headroom.
+
+`seedEdgesFileGate` (`:529`) rebuilds the gate from the stored file by blanking `Generation`. Make it blank `Headroom` too, the same way, so a restart does not rewrite an unchanged file. Read the function and add `prior.Headroom = nil` beside the line that clears `Generation`.
+
+In `edgesFileLoop`, where `ident` is built (`:638`), add the three static fields:
+
+```go
+ BoxID: edgesFileBoxID(status.InstanceID(), box),
+ Capabilities: edgesFileCapabilities(os.Getenv),
+```
+
+and, immediately before each `edgesFilePass` call in the loop, refresh the reading:
+
+```go
+ ident.Headroom = readHeadroom(provisionTickConfig(os.Getenv).DataDir, awakeCount(reg))
+```
+
+Add beside the loop:
+
+```go
+// awakeCount is how many routes this box has running now. An error reads
+// as zero: headroom is advisory.
+func awakeCount(reg *registry.Registry) int {
+ routes, err := reg.List()
+ if err != nil {
+ return 0
+ }
+ n := 0
+ for _, r := range routes {
+ if r.Status == "ready" && !r.Hibernate {
+ n++
+ }
+ }
+ return n
+}
+```
+
+`Hibernate` true means the route sleeps between requests, so a ready route with `Hibernate` false is always running. This under-counts hibernating routes that happen to be awake. That is acceptable here: the field is advisory and spec 2 decides what the chooser really needs.
+
+- [ ] **Step 4: Write the headroom readers**
+
+`internal/host/headroom_linux.go`:
+
+```go
+//go:build linux
+
+package host
+
+import (
+ "os"
+ "strconv"
+ "strings"
+
+ "golang.org/x/sys/unix"
+)
+
+// readHeadroom reads spare memory from /proc/meminfo and spare disk from
+// the data directory's filesystem. A reading that fails is left at zero.
+func readHeadroom(dataDir string, awake int) *edgeHeadroom {
+ h := &edgeHeadroom{Awake: awake}
+ if b, err := os.ReadFile("/proc/meminfo"); err == nil {
+ h.MemFreeMiB = parseMemAvailableMiB(string(b))
+ }
+ var st unix.Statfs_t
+ if err := unix.Statfs(dataDir, &st); err == nil {
+ h.DiskFreeMiB = int64(st.Bavail) * int64(st.Bsize) / (1 << 20)
+ }
+ return h
+}
+
+// parseMemAvailableMiB returns the MemAvailable line in MiB, or 0.
+func parseMemAvailableMiB(meminfo string) int64 {
+ for _, line := range strings.Split(meminfo, "\n") {
+ rest, ok := strings.CutPrefix(line, "MemAvailable:")
+ if !ok {
+ continue
+ }
+ fields := strings.Fields(rest)
+ if len(fields) == 0 {
+ return 0
+ }
+ kb, err := strconv.ParseInt(fields[0], 10, 64)
+ if err != nil {
+ return 0
+ }
+ return kb / 1024
+ }
+ return 0
+}
+```
+
+`internal/host/headroom_other.go`:
+
+```go
+//go:build !linux
+
+package host
+
+import (
+ "strconv"
+ "strings"
+)
+
+// readHeadroom has no readings off Linux: only the awake count.
+func readHeadroom(dataDir string, awake int) *edgeHeadroom { return &edgeHeadroom{Awake: awake} }
+
+func parseMemAvailableMiB(meminfo string) int64 {
+ for _, line := range strings.Split(meminfo, "\n") {
+ rest, ok := strings.CutPrefix(line, "MemAvailable:")
+ if !ok {
+ continue
+ }
+ fields := strings.Fields(rest)
+ if len(fields) == 0 {
+ return 0
+ }
+ kb, err := strconv.ParseInt(fields[0], 10, 64)
+ if err != nil {
+ return 0
+ }
+ return kb / 1024
+ }
+ return 0
+}
+```
+
+The parser is duplicated so `headroom_test.go` builds on every platform without a third file. Keep the two bodies identical.
+
+- [ ] **Step 5: Add the fields to the console's view**
+
+In `internal/console/instancesplacement.go`, after `SecretsKeyFP` in `edgesFilePlacementView`:
+
+```go
+ // Box evidence (region HA spec 1 §5.1). All absent on a writer that
+ // predates it: BoxID "", Capabilities nil, Draining false, Headroom nil.
+ BoxID string `json:"box_id"`
+ Capabilities []string `json:"capabilities"`
+ Draining bool `json:"draining"`
+ Headroom *struct {
+ MemFreeMiB int64 `json:"mem_free_mib"`
+ DiskFreeMiB int64 `json:"disk_free_mib"`
+ Awake int `json:"awake"`
+ } `json:"headroom"`
+```
+
+- [ ] **Step 6: Check the other parsers**
+
+Run: `grep -rn "control/edges\|edgesFileView\|edgesFilePrefix" --include=*.go internal/edgesfile internal/steering internal/edge | grep -v _test`
+
+For each struct that unmarshals an edges file, confirm it ignores unknown fields (Go's default) and has no `DisallowUnknownFields`. Run `grep -rn "DisallowUnknownFields" internal/edgesfile internal/steering internal/edge internal/console`. Expected: no hit on an edges-file decoder. If there is one, add the four fields to that struct in this task.
+
+- [ ] **Step 7: Run the tests**
+
+Run: `go test ./internal/host/ ./internal/console/ ./internal/edge/ ./internal/edgesfile/ -count=1 && GOOS=darwin go build ./... && GOOS=windows go build ./...`
+Expected: PASS, and both cross-builds succeed.
+
+- [ ] **Step 8: Commit**
+
+```bash
+git add internal/host internal/console
+git commit -m "edges: a box publishes its id, the HA capability, draining and headroom
+
+Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>"
+```
+
+---
+
+### Task 3: Which boxes are eligible, and which one to choose (`internal/console/boxchoice.go`)
+
+**Files:**
+- Create: `internal/console/boxchoice.go`
+- Test: `internal/console/boxchoice_test.go`
+
+**Interfaces:**
+- Consumes: `edgesSnapshot`, `edgesFilePlacementView` (with Task 2's fields), `isTrue`, `provision.ValidRegion`, `boxid.Identity`.
+- Produces:
+ ```go
+ type boxCandidate struct {
+ BoxID string
+ Region string // AWS region, from the verified box record
+ MemFreeMiB, DiskFreeMiB int64
+ Awake int
+ }
+ // verified maps box id to its verified identity; a box absent from it is never eligible.
+ // lost is the set of box ids a claim has quarantined.
+ func eligibleBoxes(snap edgesSnapshot, verified map[string]boxid.Identity, placement, homeName, expectedFP string, lost map[string]bool) []boxCandidate
+ // running is, per box id, how many hosts claims place there and which apps ("acct/app") they belong to.
+ type boxLoad struct {
+ Hosts int
+ Apps map[string]int
+ }
+ func chooseBox(cands []boxCandidate, running map[string]boxLoad, app string) (boxCandidate, bool)
+ ```
+ `placement` is the instance's region code (`""` for home). Pools are not an input yet: nearest-region's predicate covers the shared pool only (`serves_shared_pool`), and pool-aware candidates are its stated non-goal. Plan 1b decides pooled records.
+
+The eligibility predicate is nearest-region's, applied to one box, plus spec 1's additions: a verified record whose instance matches `box_id`, the capability, not draining, not lost.
+
+- [ ] **Step 1: Write the failing tests**
+
+`internal/console/boxchoice_test.go`:
+
+```go
+package console
+
+import (
+ "encoding/json"
+ "testing"
+
+ "github.com/carlosframework/platform/internal/boxid"
+)
+
+// haBox is an eligible box's edges file in placement "uk".
+func haBox(id string, mut func(m map[string]any)) string {
+ m := map[string]any{
+ "v": 1, "box": id, "box_id": id, "eip": "1.2.3.4", "region": "eu-west-2", "placement": "uk",
+ "hosts": map[string]any{}, "hibernation_configured": true, "serves_shared_pool": true,
+ "accepts_inbound": true, "secrets_key_fp": testKeyFP, "capabilities": []string{"region-ha-1"},
+ "headroom": map[string]any{"mem_free_mib": 1000, "disk_free_mib": 1000, "awake": 1},
+ }
+ if mut != nil {
+ mut(m)
+ }
+ b, _ := json.Marshal(m)
+ return string(b)
+}
+
+func ids(cs []boxCandidate) []string {
+ out := make([]string, len(cs))
+ for i, c := range cs {
+ out[i] = c.BoxID
+ }
+ return out
+}
+
+func verifiedFor(boxes ...string) map[string]boxid.Identity {
+ out := map[string]boxid.Identity{}
+ for _, b := range boxes {
+ out[b] = boxid.Identity{AccountID: "1", Region: "eu-west-2", InstanceID: b}
+ }
+ return out
+}
+
+func TestEligibleBoxesAppliesThePredicateOneBoxAtATime(t *testing.T) {
+ del := func(k string) func(map[string]any) { return func(m map[string]any) { delete(m, k) } }
+ set := func(k string, v any) func(map[string]any) { return func(m map[string]any) { m[k] = v } }
+ cases := []struct {
+ name string
+ mut func(map[string]any)
+ want bool
+ }{
+ {"all present", nil, true},
+ {"no dial address", set("eip", ""), false},
+ {"hibernation unconfigured", set("hibernation_configured", false), false},
+ {"hibernation unknown", del("hibernation_configured"), false},
+ {"refuses the shared pool", set("serves_shared_pool", false), false},
+ {"refuses inbound", set("accepts_inbound", false), false},
+ {"wrong secrets key", set("secrets_key_fp", "sha256:other"), false},
+ {"no capability", del("capabilities"), false},
+ {"draining", set("draining", true), false},
+ {"another placement", set("placement", "us"), false},
+ {"box_id disagrees with its record", set("box_id", "i-0zzz"), false},
+ }
+ for _, tc := range cases {
+ t.Run(tc.name, func(t *testing.T) {
+ snap := views(t, haBox("i-0aaa", tc.mut))
+ got := eligibleBoxes(snap, verifiedFor("i-0aaa"), "uk", "ie", testKeyFP, nil)
+ if (len(got) == 1) != tc.want {
+ t.Fatalf("eligible = %v, want %v", ids(got), tc.want)
+ }
+ })
+ }
+}
+
+func TestEligibleBoxesNeedsAVerifiedRecordFreshnessAndNotLost(t *testing.T) {
+ snap := views(t, haBox("i-0aaa", nil), haBox("i-0bbb", nil), haBox("i-0ccc", nil))
+ snap = markStale(snap, "i-0bbb")
+ got := eligibleBoxes(snap, verifiedFor("i-0aaa", "i-0bbb"), "uk", "ie", testKeyFP, nil)
+ if len(got) != 1 || got[0].BoxID != "i-0aaa" {
+ t.Fatalf("got %v, want only i-0aaa (i-0bbb is stale, i-0ccc has no verified record)", ids(got))
+ }
+ got = eligibleBoxes(snap, verifiedFor("i-0aaa"), "uk", "ie", testKeyFP, map[string]bool{"i-0aaa": true})
+ if len(got) != 0 {
+ t.Fatalf("a lost box is eligible: %v", ids(got))
+ }
+}
+
+// TestEligibilityNeverCombinesBoxes is nearest-region's rule for boxes: one
+// box holding the key and another accepting inbound do not add up to one.
+func TestEligibleBoxesNeverCombinesTwoHalfBoxes(t *testing.T) {
+ snap := views(t,
+ haBox("i-0aaa", func(m map[string]any) { m["secrets_key_fp"] = "" }),
+ haBox("i-0bbb", func(m map[string]any) { m["accepts_inbound"] = false }))
+ if got := eligibleBoxes(snap, verifiedFor("i-0aaa", "i-0bbb"), "uk", "ie", testKeyFP, nil); len(got) != 0 {
+ t.Fatalf("got %v, want none", ids(got))
+ }
+}
+
+func TestEligibleBoxesAnIncompleteSnapshotOffersNothing(t *testing.T) {
+ snap := views(t, haBox("i-0aaa", nil))
+ snap.Complete = false
+ if got := eligibleBoxes(snap, verifiedFor("i-0aaa"), "uk", "ie", testKeyFP, nil); len(got) != 0 {
+ t.Fatalf("got %v from an incomplete snapshot", ids(got))
+ }
+}
+
+func TestEligibleBoxesHomePlacementMatchesTheEmptyCode(t *testing.T) {
+ home := haBox("i-0aaa", func(m map[string]any) { m["placement"] = "" })
+ named := haBox("i-0bbb", func(m map[string]any) { m["placement"] = "ie" })
+ snap := views(t, home, named)
+ got := eligibleBoxes(snap, verifiedFor("i-0aaa", "i-0bbb"), "", "ie", testKeyFP, nil)
+ if len(got) != 2 {
+ t.Fatalf("got %v, want both home boxes", ids(got))
+ }
+}
+
+func TestChooseBox(t *testing.T) {
+ a, b, c := boxCandidate{BoxID: "i-0aaa"}, boxCandidate{BoxID: "i-0bbb"}, boxCandidate{BoxID: "i-0ccc"}
+ load := func(hosts int, apps ...string) boxLoad {
+ l := boxLoad{Hosts: hosts, Apps: map[string]int{}}
+ for _, x := range apps {
+ l.Apps[x]++
+ }
+ return l
+ }
+ if _, ok := chooseBox(nil, nil, "bab/jam"); ok {
+ t.Fatal("chose from nothing")
+ }
+ got, _ := chooseBox([]boxCandidate{a, b}, map[string]boxLoad{"i-0aaa": load(5), "i-0bbb": load(2)}, "bab/jam")
+ if got.BoxID != "i-0bbb" {
+ t.Fatalf("fewest hosts: got %s", got.BoxID)
+ }
+ got, _ = chooseBox([]boxCandidate{a, b}, map[string]boxLoad{"i-0aaa": load(5), "i-0bbb": load(2, "bab/jam")}, "bab/jam")
+ if got.BoxID != "i-0aaa" {
+ t.Fatalf("spread beats load: got %s", got.BoxID)
+ }
+ got, ok := chooseBox([]boxCandidate{a, b}, map[string]boxLoad{"i-0aaa": load(5, "bab/jam"), "i-0bbb": load(2, "bab/jam", "bab/jam")}, "bab/jam")
+ if !ok || got.BoxID != "i-0aaa" {
+ t.Fatalf("every box runs the app: fewest of that app, never a refusal; got %s ok=%v", got.BoxID, ok)
+ }
+ got, _ = chooseBox([]boxCandidate{c, b, a}, nil, "bab/jam")
+ if got.BoxID != "i-0aaa" {
+ t.Fatalf("a tie breaks by box id: got %s", got.BoxID)
+ }
+}
+```
+
+- [ ] **Step 2: Run them to see them fail**
+
+Run: `go test ./internal/console/ -run 'EligibleBoxes|ChooseBox' -count=1`
+Expected: FAIL to build, `undefined: eligibleBoxes`.
+
+- [ ] **Step 3: Write `boxchoice.go`**
+
+```go
+package console
+
+import (
+ "slices"
+ "sort"
+
+ "github.com/carlosframework/platform/internal/boxid"
+)
+
+// boxCandidate is one box that may be named as a host's executor (region HA
+// spec 1 §2, "eligible box").
+type boxCandidate struct {
+ BoxID string
+ Region string
+ MemFreeMiB, DiskFreeMiB int64
+ Awake int
+}
+
+// eligibleBoxes is regionCandidates' predicate applied to one box at a time,
+// plus what a multi-box region adds: a verified box record that agrees with
+// the file's box_id, the HA capability, not draining, not lost. Two boxes
+// that each satisfy half of it never add up to one.
+//
+// An incomplete snapshot offers nothing: an unread file could be the one
+// box that would change the answer.
+func eligibleBoxes(snap edgesSnapshot, verified map[string]boxid.Identity, placement, homeName, expectedFP string, lost map[string]bool) []boxCandidate {
+ if !snap.complete() || expectedFP == "" {
+ return nil
+ }
+ var out []boxCandidate
+ for _, f := range snap.Files {
+ code := f.Placement
+ if code == "" || (homeName != "" && code == homeName) {
+ code = ""
+ }
+ want := placement
+ if homeName != "" && want == homeName {
+ want = ""
+ }
+ id, ok := verified[f.BoxID]
+ switch {
+ case code != want:
+ case !f.Fresh, f.EIP == "", f.BoxID == "":
+ case !ok || id.InstanceID == "":
+ case !isTrue(f.HibernationConfigured), !isTrue(f.ServesSharedPool), !isTrue(f.AcceptsInbound):
+ case f.SecretsKeyFP != expectedFP:
+ case !slices.Contains(f.Capabilities, "region-ha-1"):
+ case f.Draining, lost[f.BoxID]:
+ default:
+ c := boxCandidate{BoxID: f.BoxID, Region: id.Region}
+ if f.Headroom != nil {
+ c.MemFreeMiB, c.DiskFreeMiB, c.Awake = f.Headroom.MemFreeMiB, f.Headroom.DiskFreeMiB, f.Headroom.Awake
+ }
+ out = append(out, c)
+ }
+ }
+ sort.Slice(out, func(i, j int) bool { return out[i].BoxID < out[j].BoxID })
+ return out
+}
+
+// boxLoad is what claims already place on a box.
+type boxLoad struct {
+ Hosts int
+ Apps map[string]int // "acct/app" -> hosts of that app
+}
+
+// chooseBox is spec 1 §5.2's interim rule until spec 2: prefer a box running
+// the fewest hosts of this app (soft spread: it never refuses), then the
+// fewest hosts overall, then the lowest box id.
+func chooseBox(cands []boxCandidate, running map[string]boxLoad, app string) (boxCandidate, bool) {
+ if len(cands) == 0 {
+ return boxCandidate{}, false
+ }
+ best := cands[0]
+ for _, c := range cands[1:] {
+ cl, bl := running[c.BoxID], running[best.BoxID]
+ switch {
+ case cl.Apps[app] != bl.Apps[app]:
+ if cl.Apps[app] < bl.Apps[app] {
+ best = c
+ }
+ case cl.Hosts != bl.Hosts:
+ if cl.Hosts < bl.Hosts {
+ best = c
+ }
+ case c.BoxID < best.BoxID:
+ best = c
+ }
+ }
+ return best, true
+}
+```
+
+- [ ] **Step 4: Run the tests**
+
+Run: `go test ./internal/console/ -run 'EligibleBoxes|ChooseBox' -count=1`
+Expected: PASS.
+
+If "box_id disagrees with its record" fails: the test changes the file's `box_id` to `i-0zzz` while `verified` holds only `i-0aaa`, so the lookup misses. That is the intended path; check `verified[f.BoxID]` is keyed by the file's `box_id`, not its `box`.
+
+- [ ] **Step 5: Commit**
+
+```bash
+git add internal/console/boxchoice.go internal/console/boxchoice_test.go
+git commit -m "console: eligible boxes, one at a time, and a choice that spreads an app
+
+Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>"
+```
+
+---
+
+### Task 4: The home map follows the newest installation
+
+**Files:**
+- Modify: `internal/host/edgesfile.go` (`edgeHost`, `:41-83`), `internal/edge/homemap.go` (`edgesHostView` `:139`, `homeEntry` `:161`, `foldHomes` `:538-602`)
+- Test: `internal/edge/homemap_test.go`, `internal/host/edgesfile_test.go`
+
+**Interfaces:**
+- Produces: `edgeHost.Stamp int64 \`json:"stamp,omitempty"\`` (writer), `edgesHostView.Stamp int64 \`json:"stamp,omitempty"\`` (reader), `homeEntry.stamp int64`.
+- Nothing sets `Stamp` yet: plan 1b sets it from the claim's `holder.stamp`. Until then every entry is unstamped and today's rule applies unchanged. That is what makes this task safe to land alone.
+
+The rule (spec HA4, §5.4), applied when two files claim one host as a dial target:
+1. A stamped entry beats an unstamped one.
+2. Between two stamped entries, the higher stamp wins.
+3. Between two unstamped entries, today's rule: a route beats an alias, any dial target beats none, otherwise the first box in sorted order.
+4. Equal non-zero stamps cannot happen for two different boxes (a stamp is one claim revision naming one executor). Treat it as rule 3 and log once per refresh.
+
+- [ ] **Step 1: Write the failing tests**
+
+Append to `internal/edge/homemap_test.go`:
+
+```go
+func TestHomeMapHigherStampWinsWhateverTheSortOrder(t *testing.T) {
+ s := meshTestStore(t)
+ // "annie" sorts first and is the dead box; "betty" holds the newer installation.
+ putJSON(t, s, "control/edges/annie.json", `{"v":1,"box":"annie","eip":"1.1.1.1","hosts":{"shop.test":{"role":"home","stamp":41}}}`)
+ putJSON(t, s, "control/edges/betty.json", `{"v":1,"box":"betty","eip":"2.2.2.2","hosts":{"shop.test":{"role":"home","stamp":57}}}`)
+ m := newHomeMap(s, "carol")
+ refreshHomeMap(t, m)
+ if eip, box, ok := m.Home("shop.test"); !ok || box != "betty" || eip != "2.2.2.2" {
+ t.Fatalf("Home = %q %q %v, want betty", eip, box, ok)
+ }
+ // And the other way round, so it is the stamp and not the name.
+ putJSON(t, s, "control/edges/annie.json", `{"v":1,"box":"annie","eip":"1.1.1.1","hosts":{"shop.test":{"role":"home","stamp":90}}}`)
+ refreshHomeMap(t, m)
+ if _, box, _ := m.Home("shop.test"); box != "annie" {
+ t.Fatalf("Home box = %q, want annie", box)
+ }
+}
+
+func TestHomeMapStampedBeatsUnstamped(t *testing.T) {
+ s := meshTestStore(t)
+ putJSON(t, s, "control/edges/annie.json", `{"v":1,"box":"annie","eip":"1.1.1.1","hosts":{"shop.test":{"role":"home"}}}`)
+ putJSON(t, s, "control/edges/betty.json", `{"v":1,"box":"betty","eip":"2.2.2.2","hosts":{"shop.test":{"role":"home","stamp":3}}}`)
+ m := newHomeMap(s, "carol")
+ refreshHomeMap(t, m)
+ if _, box, _ := m.Home("shop.test"); box != "betty" {
+ t.Fatalf("Home box = %q, want betty", box)
+ }
+}
+
+func TestHomeMapUnstampedKeepsTodaysRule(t *testing.T) {
+ s := meshTestStore(t)
+ putJSON(t, s, "control/edges/annie.json", `{"v":1,"box":"annie","eip":"1.1.1.1","hosts":{"shop.test":{"role":"home","alias":true}}}`)
+ putJSON(t, s, "control/edges/betty.json", `{"v":1,"box":"betty","eip":"2.2.2.2","hosts":{"shop.test":{"role":"home"}}}`)
+ m := newHomeMap(s, "carol")
+ refreshHomeMap(t, m)
+ if _, box, _ := m.Home("shop.test"); box != "betty" {
+ t.Fatalf("a route still beats an alias: got %q", box)
+ }
+ putJSON(t, s, "control/edges/annie.json", `{"v":1,"box":"annie","eip":"1.1.1.1","hosts":{"shop.test":{"role":"home"}}}`)
+ refreshHomeMap(t, m)
+ if _, box, _ := m.Home("shop.test"); box != "annie" {
+ t.Fatalf("two plain routes: first box in sorted order; got %q", box)
+ }
+}
+
+func TestHomeMapAStampedAliasDoesNotBeatAnUnstampedRoute(t *testing.T) {
+ s := meshTestStore(t)
+ putJSON(t, s, "control/edges/annie.json", `{"v":1,"box":"annie","eip":"1.1.1.1","hosts":{"shop.test":{"role":"home"}}}`)
+ putJSON(t, s, "control/edges/betty.json", `{"v":1,"box":"betty","eip":"2.2.2.2","hosts":{"shop.test":{"role":"home","alias":true,"stamp":9}}}`)
+ m := newHomeMap(s, "carol")
+ refreshHomeMap(t, m)
+ if _, box, _ := m.Home("shop.test"); box != "annie" {
+ t.Fatalf("got %q: #287's rule (a route beats an alias) comes before stamps", box)
+ }
+}
+
+func TestHomeMapAStampedEntryWithNoDialAddressDoesNotWin(t *testing.T) {
+ s := meshTestStore(t)
+ putJSON(t, s, "control/edges/annie.json", `{"v":1,"box":"annie","eip":"1.1.1.1","hosts":{"shop.test":{"role":"home","stamp":2}}}`)
+ putJSON(t, s, "control/edges/betty.json", `{"v":1,"box":"betty","eip":"","hosts":{"shop.test":{"role":"home","stamp":99}}}`)
+ m := newHomeMap(s, "carol")
+ refreshHomeMap(t, m)
+ if eip, box, ok := m.Home("shop.test"); !ok || box != "annie" || eip != "1.1.1.1" {
+ t.Fatalf("Home = %q %q %v, want annie: a box nothing can dial is skipped today and stays skipped", eip, box, ok)
+ }
+}
+```
+
+Append to `internal/host/edgesfile_test.go`:
+
+```go
+func TestEdgesFileStampIsOmittedWhenZero(t *testing.T) {
+ b, _ := json.Marshal(edgeHost{Role: "home"})
+ if bytes.Contains(b, []byte("stamp")) {
+ t.Fatalf("zero stamp on the wire: %s", b)
+ }
+ b, _ = json.Marshal(edgeHost{Role: "home", Stamp: 41})
+ if !bytes.Contains(b, []byte(`"stamp":41`)) {
+ t.Fatalf("stamp missing: %s", b)
+ }
+}
+```
+
+- [ ] **Step 2: Run them to see them fail**
+
+Run: `go test ./internal/edge/ -run 'HomeMap(HigherStamp|StampedBeats|UnstampedKeeps|AStamped)' -count=1`
+Expected: `TestHomeMapHigherStampWinsWhateverTheSortOrder` and `TestHomeMapStampedBeatsUnstamped` FAIL (annie wins by sort order); the other three PASS already. `go test ./internal/host/ -run StampIsOmitted` fails to build, `unknown field Stamp`.
+
+- [ ] **Step 3: Add the field on both sides**
+
+`internal/host/edgesfile.go`, in `edgeHost` after `App`:
+
+```go
+ // Stamp is the claim's holder.stamp for the installation this box is
+ // serving (region HA spec 1 HA4). 0 = unstamped: no claim, or a writer
+ // that predates it. Two files claiming one host are settled by the
+ // higher stamp on the edge (homemap.go foldHomes).
+ Stamp int64 `json:"stamp,omitempty"`
+```
+
+`internal/edge/homemap.go`, in `edgesHostView` after `App`:
+
+```go
+ // Stamp orders two boxes' claims on one host: see foldHomes.
+ Stamp int64 `json:"stamp,omitempty"`
+```
+
+and in `homeEntry` after `account, app`:
+
+```go
+ // stamp is the dial winner's installation stamp, 0 when unstamped.
+ stamp int64
+```
+
+- [ ] **Step 4: Change the fold**
+
+In `foldHomes`, set the stamp on the incoming entry. Change:
+
+```go
+ in := homeEntry{eip: f.EIP, box: f.Box, platform: h.Platform && !h.Alias && !h.Wildcard, alias: h.Alias, wildcard: h.Wildcard, dial: h.Role == edgeRoleHome,
+ account: h.Account, app: h.App}
+```
+
+to:
+
+```go
+ in := homeEntry{eip: f.EIP, box: f.Box, platform: h.Platform && !h.Alias && !h.Wildcard, alias: h.Alias, wildcard: h.Wildcard, dial: h.Role == edgeRoleHome,
+ account: h.Account, app: h.App, stamp: h.Stamp}
+```
+
+Then replace the merge condition:
+
+```go
+ if in.dial && (!prev.dial || (prev.alias && !in.alias)) {
+ merged.eip, merged.box, merged.alias, merged.dial = in.eip, in.box, in.alias, true
+ merged.account, merged.app = in.account, in.app
+ }
+```
+
+with:
+
+```go
+ if dialBeats(in, prev) {
+ merged.eip, merged.box, merged.alias, merged.dial = in.eip, in.box, in.alias, true
+ merged.account, merged.app, merged.stamp = in.account, in.app, in.stamp
+ }
+```
+
+and add below `foldHomes`:
+
+```go
+// dialBeats reports whether a later file's entry takes the dial from the
+// one already folded. In order:
+//
+// - only a dial target can take it, and any dial target beats none;
+// - a ROUTE beats an ALIAS whatever else is true (#287);
+// - between two of the same kind, the newer installation wins: a stamped
+// entry beats an unstamped one, and a higher stamp beats a lower one
+// (region HA spec 1 HA4). A dead box's file lingers; its successor's
+// stamp is always higher, so the stale one loses without any liveness
+// rule;
+// - otherwise the earlier file keeps it, which is the first box in
+// sorted order, as before.
+func dialBeats(in, prev homeEntry) bool {
+ if !in.dial {
+ return false
+ }
+ if !prev.dial {
+ return true
+ }
+ if prev.alias != in.alias {
+ return prev.alias && !in.alias
+ }
+ return in.stamp > prev.stamp
+}
+```
+
+An entry with no dial address never reaches `dialBeats` as a dial target: the existing `case edgeRoleHome: if f.EIP == "" { … continue }` skips it first. That is what `TestHomeMapAStampedEntryWithNoDialAddressDoesNotWin` pins.
+
+- [ ] **Step 5: Run the tests**
+
+Run: `go test ./internal/edge/ ./internal/host/ -count=1`
+Expected: PASS, including every pre-existing home-map test.
+
+- [ ] **Step 6: Commit**
+
+```bash
+git add internal/edge/homemap.go internal/edge/homemap_test.go internal/host/edgesfile.go internal/host/edgesfile_test.go
+git commit -m "edge: two boxes claiming one host are settled by the newer installation's stamp
+
+Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>"
+```
+
+---
+### Task 5: The ordered host-files manifest (`internal/hostfiles`)
+
+**Files:**
+- Create: `internal/hostfiles/manifest.go`, `internal/hostfiles/publisher.go`, `internal/hostfiles/restore.go`
+- Test: `internal/hostfiles/hostfiles_test.go`
+
+**Interfaces:**
+- Consumes: `release.Store`, `release.ErrCASConflict`.
+- Produces:
+ ```go
+ package hostfiles
+ const MaxFileBytes = 64 << 20
+ type Entry struct {
+ Object string `json:"object,omitempty"` // key relative to the prefix
+ SHA256 string `json:"sha256,omitempty"`
+ Size int64 `json:"size,omitempty"`
+ Mode uint32 `json:"mode,omitempty"` // permission bits only
+ Seq int64 `json:"seq"`
+ Deleted bool `json:"deleted,omitempty"`
+ Torn bool `json:"torn,omitempty"` // the file changed while it was read
+ Skipped string `json:"skipped,omitempty"` // "too-large": present on disk, not published
+ }
+ type Manifest struct {
+ V int `json:"v"`
+ Seq int64 `json:"seq"`
+ Entries map[string]Entry `json:"entries"` // files directory, by clean relative path
+ Platform map[string]Entry `json:"platform"` // platform-managed items, by name; never written into the files directory
+ }
+ func ManifestKey(prefix string) string
+ func CleanPath(p string) (string, error)
+ var ErrSuperseded = errors.New("hostfiles: another writer published this manifest")
+ type Publisher struct{ /* unexported */ }
+ func Open(ctx context.Context, s release.Store, prefix string) (*Publisher, error)
+ func (p *Publisher) Stage(ctx context.Context, path string, body []byte, mode uint32, torn bool) error
+ func (p *Publisher) StageSkipped(path, why string) error
+ func (p *Publisher) StageDelete(path string) error
+ func (p *Publisher) StagePlatform(ctx context.Context, name string, body []byte) error
+ func (p *Publisher) Commit(ctx context.Context) (seq int64, err error)
+ func (p *Publisher) Dirty() bool
+ func (p *Publisher) Seq() int64 // the last committed seq
+ func (p *Publisher) Paths() []string // live (not deleted) entry paths, sorted
+ type RestoreStats struct{ Written, Removed int; Skipped []string }
+ func Load(ctx context.Context, s release.Store, prefix string) (Manifest, bool, error)
+ func Restore(ctx context.Context, s release.Store, prefix, dir string) (Manifest, RestoreStats, error)
+ func ReadPlatform(ctx context.Context, s release.Store, prefix string, m Manifest, name string) ([]byte, error)
+ ```
+ `prefix` ends in `/`. Plan 1b supplies `<holder.location>/files/<generation>/`.
+
+**How ordering works (spec §5.6).** Every staged change takes the next sequence number. Its bytes go to a new key, `objects/<seq, 16 digits>-<sha256>`, written create-only, so nothing is ever overwritten. Only the manifest says which object is a path's current content, and the manifest is replaced by compare-and-swap. An upload that stalls and lands late creates an object no manifest names. A second writer's commit fails with `ErrSuperseded` instead of overwriting.
+
+- [ ] **Step 1: Write the failing tests**
+
+`internal/hostfiles/hostfiles_test.go`:
+
+```go
+package hostfiles
+
+import (
+ "context"
+ "errors"
+ "os"
+ "path/filepath"
+ "strings"
+ "testing"
+
+ "github.com/carlosframework/platform/internal/release"
+)
+
+const pfx = "instances/shop.test/abc/files/g1/"
+
+func open(t *testing.T, s release.Store) *Publisher {
+ t.Helper()
+ p, err := Open(context.Background(), s, pfx)
+ if err != nil {
+ t.Fatal(err)
+ }
+ return p
+}
+
+func stage(t *testing.T, p *Publisher, path, body string) {
+ t.Helper()
+ if err := p.Stage(context.Background(), path, []byte(body), 0o600, false); err != nil {
+ t.Fatal(err)
+ }
+}
+
+func commit(t *testing.T, p *Publisher) int64 {
+ t.Helper()
+ seq, err := p.Commit(context.Background())
+ if err != nil {
+ t.Fatal(err)
+ }
+ return seq
+}
+
+func read(t *testing.T, dir, rel string) string {
+ t.Helper()
+ b, err := os.ReadFile(filepath.Join(dir, rel))
+ if err != nil {
+ t.Fatal(err)
+ }
+ return string(b)
+}
+
+func TestPublishThenRestoreRoundTrips(t *testing.T) {
+ ctx, s := context.Background(), release.NewDirStore(t.TempDir())
+ p := open(t, s)
+ stage(t, p, "instance.key", "k1")
+ stage(t, p, "uploads/a/b.txt", "hello")
+ if seq := commit(t, p); seq != 2 {
+ t.Fatalf("seq = %d, want 2", seq)
+ }
+ dir := t.TempDir()
+ m, st, err := Restore(ctx, s, pfx, dir)
+ if err != nil {
+ t.Fatal(err)
+ }
+ if m.Seq != 2 || st.Written != 2 {
+ t.Fatalf("manifest seq %d, written %d", m.Seq, st.Written)
+ }
+ if read(t, dir, "instance.key") != "k1" || read(t, dir, "uploads/a/b.txt") != "hello" {
+ t.Fatal("content differs")
+ }
+ fi, _ := os.Stat(filepath.Join(dir, "instance.key"))
+ if fi.Mode().Perm() != 0o600 {
+ t.Fatalf("mode = %v, want 0600", fi.Mode().Perm())
+ }
+}
+
+func TestALaterWriteWinsAndAnUnchangedOneTakesNoSequence(t *testing.T) {
+ ctx, s := context.Background(), release.NewDirStore(t.TempDir())
+ p := open(t, s)
+ stage(t, p, "instance.key", "old")
+ commit(t, p)
+ stage(t, p, "instance.key", "new")
+ stage(t, p, "instance.key", "new") // unchanged: no new sequence number
+ if seq := commit(t, p); seq != 2 {
+ t.Fatalf("seq = %d, want 2", seq)
+ }
+ dir := t.TempDir()
+ if _, _, err := Restore(ctx, s, pfx, dir); err != nil {
+ t.Fatal(err)
+ }
+ if got := read(t, dir, "instance.key"); got != "new" {
+ t.Fatalf("restored %q, want new", got)
+ }
+}
+
+// A1-4: a stalled upload of the OLD content that lands after the new
+// content was published must not come back.
+func TestALateUploadCannotResurrectOldContent(t *testing.T) {
+ ctx, s := context.Background(), release.NewDirStore(t.TempDir())
+ p := open(t, s)
+ stage(t, p, "instance.key", "new")
+ commit(t, p)
+ // The late arrival: any object at all under objects/, with a higher number.
+ if _, err := s.Put(ctx, pfx+"objects/0000000000000099-deadbeef", []byte("old"), ""); err != nil {
+ t.Fatal(err)
+ }
+ dir := t.TempDir()
+ if _, _, err := Restore(ctx, s, pfx, dir); err != nil {
+ t.Fatal(err)
+ }
+ if got := read(t, dir, "instance.key"); got != "new" {
+ t.Fatalf("restored %q, want new", got)
+ }
+}
+
+func TestADeleteIsATombstoneAndRestoreRemovesTheFile(t *testing.T) {
+ ctx, s := context.Background(), release.NewDirStore(t.TempDir())
+ p := open(t, s)
+ stage(t, p, "a.txt", "a")
+ stage(t, p, "b.txt", "b")
+ commit(t, p)
+ dir := t.TempDir()
+ if _, _, err := Restore(ctx, s, pfx, dir); err != nil {
+ t.Fatal(err)
+ }
+ if err := p.StageDelete("a.txt"); err != nil {
+ t.Fatal(err)
+ }
+ commit(t, p)
+ _, st, err := Restore(ctx, s, pfx, dir)
+ if err != nil {
+ t.Fatal(err)
+ }
+ if _, err := os.Stat(filepath.Join(dir, "a.txt")); !os.IsNotExist(err) {
+ t.Fatalf("a.txt still there: %v", err)
+ }
+ if st.Removed != 1 || read(t, dir, "b.txt") != "b" {
+ t.Fatalf("removed %d", st.Removed)
+ }
+ if got := p.Paths(); len(got) != 1 || got[0] != "b.txt" {
+ t.Fatalf("Paths = %v", got)
+ }
+}
+
+func TestASecondWriterIsSupersededNotMerged(t *testing.T) {
+ s := release.NewDirStore(t.TempDir())
+ a, b := open(t, s), open(t, s)
+ stage(t, a, "x", "from a")
+ commit(t, a)
+ stage(t, b, "x", "from b")
+ if _, err := b.Commit(context.Background()); !errors.Is(err, ErrSuperseded) {
+ t.Fatalf("err = %v, want ErrSuperseded", err)
+ }
+ dir := t.TempDir()
+ if _, _, err := Restore(context.Background(), s, pfx, dir); err != nil {
+ t.Fatal(err)
+ }
+ if got := read(t, dir, "x"); got != "from a" {
+ t.Fatalf("restored %q, want from a", got)
+ }
+}
+
+func TestAReopenedPublisherContinuesTheSequence(t *testing.T) {
+ s := release.NewDirStore(t.TempDir())
+ p := open(t, s)
+ stage(t, p, "x", "1")
+ commit(t, p)
+ q := open(t, s)
+ stage(t, q, "y", "2")
+ if seq := commit(t, q); seq != 2 {
+ t.Fatalf("seq = %d, want 2", seq)
+ }
+}
+
+// Review Focus 1.
+func TestCleanPathRefusesEscapes(t *testing.T) {
+ for _, bad := range []string{"", "/etc/passwd", "../x", "a/../../x", "a/../b", "./a", "a//b", "a/", ".", "..", "a\x00b"} {
+ if _, err := CleanPath(bad); err == nil {
+ t.Errorf("CleanPath(%q) accepted", bad)
+ }
+ }
+ for _, ok := range []string{"a", "a/b.txt", ".hidden", "a b/c", `back\slash`} {
+ if got, err := CleanPath(ok); err != nil || got != ok {
+ t.Errorf("CleanPath(%q) = %q, %v", ok, got, err)
+ }
+ }
+}
+
+func TestStageAndRestoreRefuseAnEscapingPath(t *testing.T) {
+ ctx, s := context.Background(), release.NewDirStore(t.TempDir())
+ p := open(t, s)
+ if err := p.Stage(ctx, "../evil", []byte("x"), 0o600, false); err == nil {
+ t.Fatal("Stage accepted ../evil")
+ }
+ // A manifest written by something else with a hostile path.
+ body := `{"v":1,"seq":1,"entries":{"../evil":{"object":"objects/0000000000000001-aa","sha256":"aa","seq":1}},"platform":{}}`
+ hostile := "instances/h/i/files/g/"
+ if _, err := s.Put(ctx, ManifestKey(hostile), []byte(body), ""); err != nil {
+ t.Fatal(err)
+ }
+ parent := t.TempDir()
+ dir := filepath.Join(parent, "files")
+ if err := os.Mkdir(dir, 0o755); err != nil {
+ t.Fatal(err)
+ }
+ if _, _, err := Restore(ctx, s, hostile, dir); err == nil {
+ t.Fatal("Restore accepted a manifest with ../evil")
+ }
+ if _, err := os.Stat(filepath.Join(parent, "evil")); !os.IsNotExist(err) {
+ t.Fatal("../evil was written")
+ }
+}
+
+func TestRestoreRefusesContentThatDoesNotMatchItsHash(t *testing.T) {
+ ctx, s := context.Background(), release.NewDirStore(t.TempDir())
+ p := open(t, s)
+ stage(t, p, "x", "good")
+ commit(t, p)
+ m, _, err := Load(ctx, s, pfx)
+ if err != nil {
+ t.Fatal(err)
+ }
+ key := pfx + m.Entries["x"].Object
+ _, etag, _ := s.Get(ctx, key)
+ if _, err := s.Put(ctx, key, []byte("tampered"), etag); err != nil {
+ t.Fatal(err)
+ }
+ if _, _, err := Restore(ctx, s, pfx, t.TempDir()); err == nil || !strings.Contains(err.Error(), "sha256") {
+ t.Fatalf("err = %v, want a sha256 mismatch", err)
+ }
+}
+
+func TestAnAbsentManifestRestoresNothing(t *testing.T) {
+ m, st, err := Restore(context.Background(), release.NewDirStore(t.TempDir()), pfx, t.TempDir())
+ if err != nil || m.Seq != 0 || st.Written != 0 {
+ t.Fatalf("m=%+v st=%+v err=%v", m, st, err)
+ }
+}
+
+func TestPlatformItemsAreNotWrittenIntoTheFilesDirectory(t *testing.T) {
+ ctx, s := context.Background(), release.NewDirStore(t.TempDir())
+ p := open(t, s)
+ if err := p.StagePlatform(ctx, "wake.json", []byte(`{"v":1}`)); err != nil {
+ t.Fatal(err)
+ }
+ commit(t, p)
+ dir := t.TempDir()
+ m, _, err := Restore(ctx, s, pfx, dir)
+ if err != nil {
+ t.Fatal(err)
+ }
+ if entries, _ := os.ReadDir(dir); len(entries) != 0 {
+ t.Fatalf("files directory has %d entries, want 0", len(entries))
+ }
+ got, err := ReadPlatform(ctx, s, pfx, m, "wake.json")
+ if err != nil || string(got) != `{"v":1}` {
+ t.Fatalf("ReadPlatform = %q, %v", got, err)
+ }
+ if _, err := ReadPlatform(ctx, s, pfx, m, "absent"); !errors.Is(err, os.ErrNotExist) {
+ t.Fatalf("absent item: err = %v, want os.ErrNotExist", err)
+ }
+}
+
+// Review Focus 3: a skipped file stays in the manifest, visibly.
+func TestASkippedFileIsRecordedAndReportedOnRestore(t *testing.T) {
+ ctx, s := context.Background(), release.NewDirStore(t.TempDir())
+ p := open(t, s)
+ if err := p.StageSkipped("huge.bin", "too-large"); err != nil {
+ t.Fatal(err)
+ }
+ commit(t, p)
+ _, st, err := Restore(ctx, s, pfx, t.TempDir())
+ if err != nil {
+ t.Fatal(err)
+ }
+ if len(st.Skipped) != 1 || st.Skipped[0] != "huge.bin" {
+ t.Fatalf("Skipped = %v", st.Skipped)
+ }
+}
+```
+
+- [ ] **Step 2: Run them to see them fail**
+
+Run: `go test ./internal/hostfiles/ -count=1`
+Expected: FAIL to build, `undefined: Open`.
+
+- [ ] **Step 3: Write `manifest.go`**
+
+```go
+// Package hostfiles publishes an instance's files directory, and the small
+// platform-managed items that travel with it, to object storage in an order
+// that survives late and duplicate uploads (region HA spec 1 §5.6).
+//
+// Nothing is overwritten: each change is a new immutable object, and one
+// manifest, replaced by compare-and-swap, says which object is current.
+package hostfiles
+
+import (
+ "context"
+ "encoding/json"
+ "errors"
+ "fmt"
+ "os"
+ "path"
+ "strings"
+
+ "github.com/carlosframework/platform/internal/release"
+)
+
+// MaxFileBytes is the largest file published as one object. release.Store
+// takes a whole body in memory, so a larger file is recorded as skipped
+// rather than read. Plan 1b lifts this with a streaming put.
+const MaxFileBytes = 64 << 20
+
+type Entry struct {
+ Object string `json:"object,omitempty"`
+ SHA256 string `json:"sha256,omitempty"`
+ Size int64 `json:"size,omitempty"`
+ Mode uint32 `json:"mode,omitempty"`
+ Seq int64 `json:"seq"`
+ Deleted bool `json:"deleted,omitempty"`
+ Torn bool `json:"torn,omitempty"`
+ Skipped string `json:"skipped,omitempty"`
+}
+
+type Manifest struct {
+ V int `json:"v"`
+ Seq int64 `json:"seq"`
+ Entries map[string]Entry `json:"entries"`
+ Platform map[string]Entry `json:"platform"`
+}
+
+func ManifestKey(prefix string) string { return prefix + "manifest.json" }
+
+func objectKey(seq int64, sha string) string { return fmt.Sprintf("objects/%016d-%s", seq, sha) }
+
+// CleanPath accepts only a relative, already-clean, slash-separated path
+// that stays inside its root. It never repairs a path: a manifest is
+// written by this package, so anything else is damage or an attack.
+func CleanPath(p string) (string, error) {
+ if p == "" || strings.ContainsRune(p, 0) || strings.HasPrefix(p, "/") {
+ return "", fmt.Errorf("hostfiles: path %q is not a relative path", p)
+ }
+ c := path.Clean(p)
+ if c != p || c == "." || c == ".." || strings.HasPrefix(c, "../") {
+ return "", fmt.Errorf("hostfiles: path %q is not clean or leaves its root", p)
+ }
+ return c, nil
+}
+
+// Load reads the manifest. The bool is false when there is none yet.
+func Load(ctx context.Context, s release.Store, prefix string) (Manifest, bool, error) {
+ m, _, found, err := load(ctx, s, prefix)
+ return m, found, err
+}
+
+func load(ctx context.Context, s release.Store, prefix string) (Manifest, string, bool, error) {
+ empty := Manifest{V: 1, Entries: map[string]Entry{}, Platform: map[string]Entry{}}
+ raw, etag, err := s.Get(ctx, ManifestKey(prefix))
+ if errors.Is(err, os.ErrNotExist) {
+ return empty, "", false, nil
+ }
+ if err != nil {
+ return Manifest{}, "", false, err
+ }
+ var m Manifest
+ if err := json.Unmarshal(raw, &m); err != nil {
+ return Manifest{}, "", false, fmt.Errorf("hostfiles: manifest %s: %w", prefix, err)
+ }
+ if m.V != 1 {
+ return Manifest{}, "", false, fmt.Errorf("hostfiles: manifest %s is version %d; this binary reads 1", prefix, m.V)
+ }
+ if m.Entries == nil {
+ m.Entries = map[string]Entry{}
+ }
+ if m.Platform == nil {
+ m.Platform = map[string]Entry{}
+ }
+ return m, etag, true, nil
+}
+```
+
+- [ ] **Step 4: Write `publisher.go`**
+
+```go
+package hostfiles
+
+import (
+ "context"
+ "crypto/sha256"
+ "encoding/hex"
+ "encoding/json"
+ "errors"
+ "fmt"
+ "sort"
+ "strings"
+ "sync"
+
+ "github.com/carlosframework/platform/internal/release"
+)
+
+// ErrSuperseded means another writer replaced the manifest. The caller
+// stops publishing: it is no longer this host's executor, or two
+// publishers were started for one host.
+var ErrSuperseded = errors.New("hostfiles: another writer published this manifest")
+
+// Publisher stages changes and commits them as one manifest revision. It
+// is safe for concurrent use.
+type Publisher struct {
+ s release.Store
+ prefix string
+
+ mu sync.Mutex
+ man Manifest
+ etag string
+ next int64 // the last sequence number handed out; man.Seq is the last committed
+ dirty bool
+}
+
+func Open(ctx context.Context, s release.Store, prefix string) (*Publisher, error) {
+ if !strings.HasSuffix(prefix, "/") {
+ return nil, fmt.Errorf("hostfiles: prefix %q must end in /", prefix)
+ }
+ m, etag, _, err := load(ctx, s, prefix)
+ if err != nil {
+ return nil, err
+ }
+ return &Publisher{s: s, prefix: prefix, man: m, etag: etag, next: m.Seq}, nil
+}
+
+func sum(body []byte) string {
+ h := sha256.Sum256(body)
+ return hex.EncodeToString(h[:])
+}
+
+// put writes body as the next immutable object and returns its entry.
+// Callers hold p.mu.
+func (p *Publisher) put(ctx context.Context, body []byte, mode uint32, torn bool) (Entry, error) {
+ sha := sum(body)
+ seq := p.next + 1
+ key := objectKey(seq, sha)
+ if _, err := p.s.Put(ctx, p.prefix+key, body, ""); err != nil && !errors.Is(err, release.ErrCASConflict) {
+ // A conflict means this exact key exists: the same sequence number
+ // and the same hash, left by an attempt that crashed before its
+ // commit. The bytes are the ones we were about to write.
+ return Entry{}, err
+ }
+ p.next = seq
+ return Entry{Object: key, SHA256: sha, Size: int64(len(body)), Mode: mode & 0o777, Seq: seq, Torn: torn}, nil
+}
+
+// Stage records path's content. Content identical to what is already
+// staged or committed takes no sequence number and writes nothing.
+func (p *Publisher) Stage(ctx context.Context, path string, body []byte, mode uint32, torn bool) error {
+ clean, err := CleanPath(path)
+ if err != nil {
+ return err
+ }
+ p.mu.Lock()
+ defer p.mu.Unlock()
+ if cur, ok := p.man.Entries[clean]; ok && !cur.Deleted && cur.Skipped == "" &&
+ cur.SHA256 == sum(body) && cur.Mode == mode&0o777 && cur.Torn == torn {
+ return nil
+ }
+ e, err := p.put(ctx, body, mode, torn)
+ if err != nil {
+ return err
+ }
+ p.man.Entries[clean], p.dirty = e, true
+ return nil
+}
+
+// StageSkipped records that path exists but was not published, and why.
+func (p *Publisher) StageSkipped(path, why string) error {
+ clean, err := CleanPath(path)
+ if err != nil {
+ return err
+ }
+ p.mu.Lock()
+ defer p.mu.Unlock()
+ if cur, ok := p.man.Entries[clean]; ok && cur.Skipped == why {
+ return nil
+ }
+ p.next++
+ p.man.Entries[clean], p.dirty = Entry{Seq: p.next, Skipped: why}, true
+ return nil
+}
+
+// StageDelete records that path is gone. Deleting a path the manifest never
+// had, or already has as deleted, is a no-op.
+func (p *Publisher) StageDelete(path string) error {
+ clean, err := CleanPath(path)
+ if err != nil {
+ return err
+ }
+ p.mu.Lock()
+ defer p.mu.Unlock()
+ if cur, ok := p.man.Entries[clean]; !ok || cur.Deleted {
+ return nil
+ }
+ p.next++
+ p.man.Entries[clean], p.dirty = Entry{Seq: p.next, Deleted: true}, true
+ return nil
+}
+
+// StagePlatform records a platform-managed item (wake state, an env file).
+// It shares the manifest's ordering but is never written into the files
+// directory by Restore.
+func (p *Publisher) StagePlatform(ctx context.Context, name string, body []byte) error {
+ if name == "" || strings.ContainsAny(name, "/\x00") {
+ return fmt.Errorf("hostfiles: platform item %q is not a plain name", name)
+ }
+ p.mu.Lock()
+ defer p.mu.Unlock()
+ if cur, ok := p.man.Platform[name]; ok && cur.SHA256 == sum(body) {
+ return nil
+ }
+ e, err := p.put(ctx, body, 0o600, false)
+ if err != nil {
+ return err
+ }
+ p.man.Platform[name], p.dirty = e, true
+ return nil
+}
+
+// Commit publishes everything staged as one manifest revision and returns
+// its sequence number. With nothing staged it returns the current one. A
+// failed commit leaves the staged changes in place, so the caller retries;
+// ErrSuperseded is final.
+func (p *Publisher) Commit(ctx context.Context) (int64, error) {
+ p.mu.Lock()
+ defer p.mu.Unlock()
+ if !p.dirty {
+ return p.man.Seq, nil
+ }
+ out := p.man
+ out.Seq = p.next
+ body, err := json.Marshal(out)
+ if err != nil {
+ return 0, fmt.Errorf("hostfiles: marshal manifest: %w", err)
+ }
+ etag, err := p.s.Put(ctx, ManifestKey(p.prefix), body, p.etag)
+ if errors.Is(err, release.ErrCASConflict) {
+ return 0, fmt.Errorf("%w: %s", ErrSuperseded, p.prefix)
+ }
+ if err != nil {
+ return 0, err
+ }
+ p.man.Seq, p.etag, p.dirty = out.Seq, etag, false
+ return out.Seq, nil
+}
+
+func (p *Publisher) Dirty() bool { p.mu.Lock(); defer p.mu.Unlock(); return p.dirty }
+
+func (p *Publisher) Seq() int64 { p.mu.Lock(); defer p.mu.Unlock(); return p.man.Seq }
+
+// Paths lists every path the manifest holds as present: published or
+// skipped, not deleted.
+func (p *Publisher) Paths() []string {
+ p.mu.Lock()
+ defer p.mu.Unlock()
+ var out []string
+ for path, e := range p.man.Entries {
+ if !e.Deleted {
+ out = append(out, path)
+ }
+ }
+ sort.Strings(out)
+ return out
+}
+```
+
+`gofmt` will expand the three one-line methods; let it.
+
+- [ ] **Step 5: Write `restore.go`**
+
+```go
+package hostfiles
+
+import (
+ "context"
+ "fmt"
+ "os"
+ "path/filepath"
+ "sort"
+
+ "github.com/carlosframework/platform/internal/release"
+)
+
+type RestoreStats struct {
+ Written, Removed int
+ Skipped []string // paths the manifest holds as present but unpublished
+}
+
+// Restore makes dir match the manifest under prefix: it writes every
+// published entry, removes every tombstoned one, and reports the skipped
+// ones. It reads nothing but the manifest and the objects the manifest
+// names, and verifies each object against the hash the manifest records.
+//
+// Every path is validated before anything is written, so a manifest with
+// one bad path changes nothing.
+func Restore(ctx context.Context, s release.Store, prefix, dir string) (Manifest, RestoreStats, error) {
+ var st RestoreStats
+ m, _, _, err := load(ctx, s, prefix)
+ if err != nil {
+ return Manifest{}, st, err
+ }
+ paths := make([]string, 0, len(m.Entries))
+ for p := range m.Entries {
+ if _, err := CleanPath(p); err != nil {
+ return Manifest{}, st, err
+ }
+ paths = append(paths, p)
+ }
+ sort.Strings(paths)
+ for _, p := range paths {
+ e := m.Entries[p]
+ dst := filepath.Join(dir, filepath.FromSlash(p))
+ switch {
+ case e.Deleted:
+ err := os.Remove(dst)
+ if err == nil {
+ st.Removed++
+ } else if !os.IsNotExist(err) {
+ return Manifest{}, st, fmt.Errorf("hostfiles: remove %s: %w", p, err)
+ }
+ case e.Skipped != "":
+ st.Skipped = append(st.Skipped, p)
+ default:
+ body, err := fetch(ctx, s, prefix, e)
+ if err != nil {
+ return Manifest{}, st, fmt.Errorf("hostfiles: %s: %w", p, err)
+ }
+ if err := writeAtomic(dst, body, os.FileMode(e.Mode)); err != nil {
+ return Manifest{}, st, fmt.Errorf("hostfiles: write %s: %w", p, err)
+ }
+ st.Written++
+ }
+ }
+ return m, st, nil
+}
+
+// ReadPlatform returns a platform-managed item's bytes, verified. A name
+// the manifest does not hold is os.ErrNotExist.
+func ReadPlatform(ctx context.Context, s release.Store, prefix string, m Manifest, name string) ([]byte, error) {
+ e, ok := m.Platform[name]
+ if !ok {
+ return nil, fmt.Errorf("hostfiles: platform item %q: %w", name, os.ErrNotExist)
+ }
+ return fetch(ctx, s, prefix, e)
+}
+
+func fetch(ctx context.Context, s release.Store, prefix string, e Entry) ([]byte, error) {
+ body, _, err := s.Get(ctx, prefix+e.Object)
+ if err != nil {
+ return nil, err
+ }
+ if got := sum(body); got != e.SHA256 {
+ return nil, fmt.Errorf("object %s has sha256 %s, the manifest says %s", e.Object, got, e.SHA256)
+ }
+ return body, nil
+}
+
+// writeAtomic writes beside the destination and renames into place, so a
+// crash leaves the old file or the new one, never part of one.
+func writeAtomic(dst string, body []byte, mode os.FileMode) error {
+ if mode == 0 {
+ mode = 0o600
+ }
+ if err := os.MkdirAll(filepath.Dir(dst), 0o755); err != nil {
+ return err
+ }
+ tmp, err := os.CreateTemp(filepath.Dir(dst), ".carlos-restore-*")
+ if err != nil {
+ return err
+ }
+ defer os.Remove(tmp.Name())
+ if _, err := tmp.Write(body); err != nil {
+ tmp.Close()
+ return err
+ }
+ if err := tmp.Chmod(mode); err != nil {
+ tmp.Close()
+ return err
+ }
+ if err := tmp.Sync(); err != nil {
+ tmp.Close()
+ return err
+ }
+ if err := tmp.Close(); err != nil {
+ return err
+ }
+ return os.Rename(tmp.Name(), dst)
+}
+```
+
+- [ ] **Step 6: Run the tests**
+
+Run: `gofmt -w internal/hostfiles && go test ./internal/hostfiles/ -count=1`
+Expected: PASS.
+
+`TestRestoreRefusesContentThatDoesNotMatchItsHash` overwrites an object through the test store with its etag. That is a test-only tamper: production objects are create-only and Task 11's bucket policy enforces it.
+
+- [ ] **Step 7: Commit**
+
+```bash
+git add internal/hostfiles
+git commit -m "hostfiles: immutable objects and one manifest decide which file content is current
+
+Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>"
+```
+
+---
+
+### Task 6: Publishing a whole directory (`hostfiles.SyncDir`)
+
+**Files:**
+- Create: `internal/hostfiles/sync.go`
+- Test: `internal/hostfiles/sync_test.go`
+
+**Interfaces:**
+- Consumes: Task 5's `Publisher`.
+- Produces:
+ ```go
+ type SyncStats struct {
+ Published, Deleted int
+ TooLarge, NotRegular []string // relative paths
+ }
+ // SyncDir stages every difference between dir and the manifest: changed
+ // and new regular files, and a tombstone for each manifest path no longer
+ // on disk. It does not commit. skip, when non-nil, excludes a relative
+ // path (and, for a directory, everything under it).
+ func SyncDir(ctx context.Context, p *Publisher, dir string, skip func(rel string, isDir bool) bool) (SyncStats, error)
+ // readStable reads one file for publishing.
+ func readStable(path string) (body []byte, mode uint32, state fileState, err error)
+ type fileState int
+ const (
+ fileOK fileState = iota
+ fileTorn // changed while it was read; body is the image read
+ fileGone
+ fileNotRegular
+ fileTooLarge
+ )
+ ```
+
+This one function serves three callers in plan 1b: the `snapshot` setting (run every `files_snapshot_interval`), the continuous publisher's ten-minute repair scan, and the release sequence's "publish every pending mutation" step. That is why this plan has no separate sealed-archive snapshot: a periodic full `SyncDir` gives the `snapshot` setting the same result through the same manifest (recorded as a spec note in Task 11).
+
+- [ ] **Step 1: Write the failing tests**
+
+`internal/hostfiles/sync_test.go`:
+
+```go
+package hostfiles
+
+import (
+ "context"
+ "os"
+ "path/filepath"
+ "reflect"
+ "testing"
+
+ "github.com/carlosframework/platform/internal/release"
+)
+
+func write(t *testing.T, dir, rel, body string) {
+ t.Helper()
+ full := filepath.Join(dir, rel)
+ if err := os.MkdirAll(filepath.Dir(full), 0o755); err != nil {
+ t.Fatal(err)
+ }
+ if err := os.WriteFile(full, []byte(body), 0o640); err != nil {
+ t.Fatal(err)
+ }
+}
+
+func TestSyncDirPublishesNewChangedAndDeleted(t *testing.T) {
+ ctx, s, src := context.Background(), release.NewDirStore(t.TempDir()), t.TempDir()
+ p := open(t, s)
+ write(t, src, "instance.key", "k")
+ write(t, src, "uploads/a.txt", "a")
+ st, err := SyncDir(ctx, p, src, nil)
+ if err != nil || st.Published != 2 {
+ t.Fatalf("first sync: %+v, %v", st, err)
+ }
+ commit(t, p)
+
+ write(t, src, "uploads/a.txt", "a2")
+ if err := os.Remove(filepath.Join(src, "instance.key")); err != nil {
+ t.Fatal(err)
+ }
+ write(t, src, "new.txt", "n")
+ st, err = SyncDir(ctx, p, src, nil)
+ if err != nil || st.Published != 2 || st.Deleted != 1 {
+ t.Fatalf("second sync: %+v, %v", st, err)
+ }
+ commit(t, p)
+
+ dst := t.TempDir()
+ if _, _, err := Restore(ctx, s, pfx, dst); err != nil {
+ t.Fatal(err)
+ }
+ if read(t, dst, "uploads/a.txt") != "a2" || read(t, dst, "new.txt") != "n" {
+ t.Fatal("content differs")
+ }
+ if _, err := os.Stat(filepath.Join(dst, "instance.key")); !os.IsNotExist(err) {
+ t.Fatal("deleted file restored")
+ }
+ fi, _ := os.Stat(filepath.Join(dst, "new.txt"))
+ if fi.Mode().Perm() != 0o640 {
+ t.Fatalf("mode = %v, want 0640", fi.Mode().Perm())
+ }
+}
+
+func TestSyncDirOfAnUnchangedDirectoryStagesNothing(t *testing.T) {
+ ctx, s, src := context.Background(), release.NewDirStore(t.TempDir()), t.TempDir()
+ p := open(t, s)
+ write(t, src, "a", "a")
+ if _, err := SyncDir(ctx, p, src, nil); err != nil {
+ t.Fatal(err)
+ }
+ commit(t, p)
+ st, err := SyncDir(ctx, p, src, nil)
+ if err != nil || st.Published != 0 || st.Deleted != 0 || p.Dirty() {
+ t.Fatalf("second sync: %+v dirty=%v err=%v", st, p.Dirty(), err)
+ }
+}
+
+// Review Focus 2: never follow or open what is not a regular file.
+func TestSyncDirSkipsWhatIsNotARegularFile(t *testing.T) {
+ ctx, s, src := context.Background(), release.NewDirStore(t.TempDir()), t.TempDir()
+ outside := filepath.Join(t.TempDir(), "secret")
+ if err := os.WriteFile(outside, []byte("do not publish"), 0o600); err != nil {
+ t.Fatal(err)
+ }
+ write(t, src, "real.txt", "r")
+ if err := os.Symlink(outside, filepath.Join(src, "link")); err != nil {
+ t.Fatal(err)
+ }
+ if err := os.Symlink(filepath.Dir(outside), filepath.Join(src, "linkdir")); err != nil {
+ t.Fatal(err)
+ }
+ p := open(t, s)
+ st, err := SyncDir(ctx, p, src, nil)
+ if err != nil {
+ t.Fatal(err)
+ }
+ if st.Published != 1 || !reflect.DeepEqual(st.NotRegular, []string{"link", "linkdir"}) {
+ t.Fatalf("stats = %+v", st)
+ }
+ if got := p.Paths(); !reflect.DeepEqual(got, []string{"real.txt"}) {
+ t.Fatalf("Paths = %v", got)
+ }
+}
+
+// Review Focus 3: too large is recorded, never read whole, never dropped.
+func TestSyncDirRecordsAFileOverTheCapWithoutReadingIt(t *testing.T) {
+ ctx, s, src := context.Background(), release.NewDirStore(t.TempDir()), t.TempDir()
+ big := filepath.Join(src, "huge.bin")
+ f, err := os.Create(big)
+ if err != nil {
+ t.Fatal(err)
+ }
+ if err := f.Truncate(MaxFileBytes + 1); err != nil { // sparse: costs no disk
+ t.Fatal(err)
+ }
+ f.Close()
+ p := open(t, s)
+ st, err := SyncDir(ctx, p, src, nil)
+ if err != nil {
+ t.Fatal(err)
+ }
+ if !reflect.DeepEqual(st.TooLarge, []string{"huge.bin"}) || st.Published != 0 {
+ t.Fatalf("stats = %+v", st)
+ }
+ commit(t, p)
+ _, rs, err := Restore(ctx, s, pfx, t.TempDir())
+ if err != nil || !reflect.DeepEqual(rs.Skipped, []string{"huge.bin"}) {
+ t.Fatalf("restore skipped %v, err %v", rs.Skipped, err)
+ }
+}
+
+func TestSyncDirHonoursSkip(t *testing.T) {
+ ctx, s, src := context.Background(), release.NewDirStore(t.TempDir()), t.TempDir()
+ write(t, src, "keep.txt", "k")
+ write(t, src, ".files-gen", "marker")
+ write(t, src, "cache/x", "x")
+ p := open(t, s)
+ skip := func(rel string, isDir bool) bool { return rel == ".files-gen" || (isDir && rel == "cache") }
+ if _, err := SyncDir(ctx, p, src, skip); err != nil {
+ t.Fatal(err)
+ }
+ if got := p.Paths(); !reflect.DeepEqual(got, []string{"keep.txt"}) {
+ t.Fatalf("Paths = %v", got)
+ }
+}
+
+func TestSyncDirOfAMissingDirectoryTombstonesNothing(t *testing.T) {
+ ctx, s, src := context.Background(), release.NewDirStore(t.TempDir()), t.TempDir()
+ p := open(t, s)
+ write(t, src, "a", "a")
+ if _, err := SyncDir(ctx, p, src, nil); err != nil {
+ t.Fatal(err)
+ }
+ commit(t, p)
+ if _, err := SyncDir(ctx, p, filepath.Join(src, "does-not-exist"), nil); err == nil {
+ t.Fatal("a missing root must be an error, not an empty directory")
+ }
+ if p.Dirty() {
+ t.Fatal("a missing root staged tombstones")
+ }
+}
+```
+
+- [ ] **Step 2: Run them to see them fail**
+
+Run: `go test ./internal/hostfiles/ -run SyncDir -count=1`
+Expected: FAIL to build, `undefined: SyncDir`.
+
+- [ ] **Step 3: Write `sync.go`**
+
+```go
+package hostfiles
+
+import (
+ "context"
+ "fmt"
+ "io"
+ "io/fs"
+ "os"
+ "path/filepath"
+ "sort"
+)
+
+type fileState int
+
+const (
+ fileOK fileState = iota
+ fileTorn
+ fileGone
+ fileNotRegular
+ fileTooLarge
+)
+
+// readStable reads one file for publishing. It never follows a symlink and
+// never opens anything but a regular file. If the file's size, mtime or
+// identity changed between the first look and the last byte, the bytes
+// read are returned as fileTorn: a possibly inconsistent image the caller
+// publishes flagged and reads again later.
+func readStable(path string) ([]byte, uint32, fileState, error) {
+ before, err := os.Lstat(path)
+ if os.IsNotExist(err) {
+ return nil, 0, fileGone, nil
+ }
+ if err != nil {
+ return nil, 0, fileOK, err
+ }
+ if !before.Mode().IsRegular() {
+ return nil, 0, fileNotRegular, nil
+ }
+ if before.Size() > MaxFileBytes {
+ return nil, 0, fileTooLarge, nil
+ }
+ f, err := os.Open(path)
+ if os.IsNotExist(err) {
+ return nil, 0, fileGone, nil
+ }
+ if err != nil {
+ return nil, 0, fileOK, err
+ }
+ defer f.Close()
+ opened, err := f.Stat()
+ if err != nil {
+ return nil, 0, fileOK, err
+ }
+ if !os.SameFile(before, opened) || !opened.Mode().IsRegular() {
+ // Replaced between the look and the open, possibly by a symlink
+ // that Open followed. Do not read it.
+ return nil, 0, fileNotRegular, nil
+ }
+ body, err := io.ReadAll(io.LimitReader(f, MaxFileBytes+1))
+ if err != nil {
+ return nil, 0, fileOK, err
+ }
+ if int64(len(body)) > MaxFileBytes {
+ return nil, 0, fileTooLarge, nil
+ }
+ after, err := os.Lstat(path)
+ state := fileOK
+ if err != nil || !os.SameFile(before, after) || after.Size() != before.Size() || !after.ModTime().Equal(before.ModTime()) {
+ state = fileTorn
+ }
+ return body, uint32(before.Mode().Perm()), state, nil
+}
+
+type SyncStats struct {
+ Published, Deleted int
+ TooLarge, NotRegular []string
+}
+
+// SyncDir stages every difference between dir and the manifest. It does
+// not commit. A root that cannot be read is an error and stages nothing:
+// an unreadable directory is not an empty one.
+func SyncDir(ctx context.Context, p *Publisher, dir string, skip func(rel string, isDir bool) bool) (SyncStats, error) {
+ var st SyncStats
+ if fi, err := os.Lstat(dir); err != nil {
+ return st, fmt.Errorf("hostfiles: sync %s: %w", dir, err)
+ } else if !fi.IsDir() {
+ return st, fmt.Errorf("hostfiles: sync %s: not a directory", dir)
+ }
+ seen := map[string]bool{}
+ err := filepath.WalkDir(dir, func(full string, d fs.DirEntry, err error) error {
+ if err != nil {
+ return err
+ }
+ if err := ctx.Err(); err != nil {
+ return err
+ }
+ if full == dir {
+ return nil
+ }
+ rel, err := filepath.Rel(dir, full)
+ if err != nil {
+ return err
+ }
+ rel = filepath.ToSlash(rel)
+ if skip != nil && skip(rel, d.IsDir()) {
+ if d.IsDir() {
+ return filepath.SkipDir
+ }
+ return nil
+ }
+ if d.IsDir() {
+ return nil
+ }
+ // WalkDir does not follow symlinks: a symlink to a directory
+ // arrives here as a non-directory entry.
+ body, mode, state, err := readStable(full)
+ if err != nil {
+ return err
+ }
+ switch state {
+ case fileGone:
+ return nil
+ case fileNotRegular:
+ st.NotRegular = append(st.NotRegular, rel)
+ return nil
+ case fileTooLarge:
+ seen[rel] = true
+ st.TooLarge = append(st.TooLarge, rel)
+ return p.StageSkipped(rel, "too-large")
+ }
+ seen[rel] = true
+ before := p.pending()
+ if err := p.Stage(ctx, rel, body, mode, state == fileTorn); err != nil {
+ return err
+ }
+ if p.pending() != before {
+ st.Published++
+ }
+ return nil
+ })
+ if err != nil {
+ return st, fmt.Errorf("hostfiles: sync %s: %w", dir, err)
+ }
+ for _, rel := range p.Paths() {
+ if seen[rel] || (skip != nil && skip(rel, false)) {
+ continue
+ }
+ if err := p.StageDelete(rel); err != nil {
+ return st, err
+ }
+ st.Deleted++
+ }
+ sort.Strings(st.TooLarge)
+ sort.Strings(st.NotRegular)
+ return st, nil
+}
+```
+
+Add to `publisher.go`:
+
+```go
+// pending is the last sequence number handed out, committed or not.
+func (p *Publisher) pending() int64 {
+ p.mu.Lock()
+ defer p.mu.Unlock()
+ return p.next
+}
+```
+
+`SyncDir` returns an error from the walk before it reaches the tombstone loop, so a directory that vanishes or fails partway never tombstones the files it did not reach.
+
+- [ ] **Step 4: Run the tests**
+
+Run: `go test ./internal/hostfiles/ -count=1`
+Expected: PASS.
+
+- [ ] **Step 5: Commit**
+
+```bash
+git add internal/hostfiles
+git commit -m "hostfiles: SyncDir stages a directory's differences, and never follows a link
+
+Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>"
+```
+
+---
+
+### Task 7: Continuous publishing with a deadline (`hostfiles.Run`)
+
+**Files:**
+- Create: `internal/hostfiles/deadline.go`, `internal/hostfiles/run.go`, `internal/hostfiles/watch_linux.go`, `internal/hostfiles/watch_other.go`
+- Test: `internal/hostfiles/deadline_test.go`, `internal/hostfiles/run_test.go`, `internal/hostfiles/watch_linux_test.go`
+
+**Interfaces:**
+- Consumes: `Publisher`, `SyncDir`, `readStable`.
+- Produces:
+ ```go
+ type Options struct {
+ Deadline time.Duration // first unpublished modification to publish; spec: 1s
+ Scan time.Duration // repair scan; spec: 10m
+ Poll time.Duration // full-scan period while a watch is missing; spec: 1s
+ Skip func(rel string, isDir bool) bool
+ OnStats func(SyncStats) // called after each full scan; may be nil
+ }
+ func DefaultOptions() Options
+ // Run publishes dir's changes until ctx ends or the manifest is superseded.
+ func Run(ctx context.Context, p *Publisher, dir string, o Options) error
+ // internal:
+ type event struct{ rel string; rescan bool }
+ type watcher interface {
+ Events() <-chan event
+ Degraded() bool // some directory has no working watch
+ Repair() error // try to restore full coverage
+ Close() error
+ }
+ func newWatcher(root string, skip func(string, bool) bool) (watcher, error) // per-OS
+ ```
+
+Spec §5.6, continuous mode: a file is published within one second of its **first** unpublished modification, whether or not it has been closed; a file still being written is published again at the next deadline; an unstable read is published anyway, flagged; overflow triggers a full scan; a missing watch means polling every second until coverage returns.
+
+- [ ] **Step 1: Write the deadline queue's failing test**
+
+`internal/hostfiles/deadline_test.go`:
+
+```go
+package hostfiles
+
+import (
+ "reflect"
+ "testing"
+ "time"
+)
+
+func TestQueueArmsOnTheFirstTouchAndLaterTouchesDoNotPushItBack(t *testing.T) {
+ q := newQueue(time.Second)
+ t0 := time.Unix(1000, 0)
+ q.Touch("a", t0)
+ q.Touch("a", t0.Add(900*time.Millisecond)) // still being written
+ if got := q.Due(t0.Add(999 * time.Millisecond)); len(got) != 0 {
+ t.Fatalf("due early: %v", got)
+ }
+ if got := q.Due(t0.Add(time.Second)); !reflect.DeepEqual(got, []string{"a"}) {
+ t.Fatalf("due = %v, want [a]", got)
+ }
+ if got := q.Due(t0.Add(2 * time.Second)); len(got) != 0 {
+ t.Fatalf("due twice: %v", got)
+ }
+ q.Touch("a", t0.Add(3*time.Second)) // a new modification arms a new deadline
+ if got := q.Due(t0.Add(4 * time.Second)); !reflect.DeepEqual(got, []string{"a"}) {
+ t.Fatalf("second deadline: %v", got)
+ }
+}
+
+func TestQueueReturnsDuePathsSorted(t *testing.T) {
+ q := newQueue(time.Second)
+ t0 := time.Unix(1000, 0)
+ for _, p := range []string{"c", "a", "b"} {
+ q.Touch(p, t0)
+ }
+ if got := q.Due(t0.Add(time.Second)); !reflect.DeepEqual(got, []string{"a", "b", "c"}) {
+ t.Fatalf("due = %v", got)
+ }
+}
+```
+
+- [ ] **Step 2: Run it to see it fail**
+
+Run: `go test ./internal/hostfiles/ -run Queue -count=1`
+Expected: FAIL to build, `undefined: newQueue`.
+
+- [ ] **Step 3: Write `deadline.go`**
+
+```go
+package hostfiles
+
+import (
+ "sort"
+ "time"
+)
+
+// queue holds, per path, the time by which its first unpublished
+// modification must be published. A later modification of a path already
+// waiting does not move its deadline, so a file written without pause is
+// still published every period.
+type queue struct {
+ d time.Duration
+ due map[string]time.Time
+}
+
+func newQueue(d time.Duration) *queue { return &queue{d: d, due: map[string]time.Time{}} }
+
+func (q *queue) Touch(rel string, now time.Time) {
+ if _, waiting := q.due[rel]; !waiting {
+ q.due[rel] = now.Add(q.d)
+ }
+}
+
+// Due removes and returns every path whose deadline has arrived.
+func (q *queue) Due(now time.Time) []string {
+ var out []string
+ for rel, at := range q.due {
+ if !at.After(now) {
+ out = append(out, rel)
+ delete(q.due, rel)
+ }
+ }
+ sort.Strings(out)
+ return out
+}
+```
+
+Run: `go test ./internal/hostfiles/ -run Queue -count=1`. Expected: PASS.
+
+- [ ] **Step 4: Write the run loop's failing tests, with a fake watcher**
+
+`internal/hostfiles/run_test.go`:
+
+```go
+package hostfiles
+
+import (
+ "context"
+ "errors"
+ "os"
+ "path/filepath"
+ "sync"
+ "testing"
+ "time"
+
+ "github.com/carlosframework/platform/internal/release"
+)
+
+type fakeWatcher struct {
+ ch chan event
+ mu sync.Mutex
+ degraded bool
+ repairs int
+}
+
+func newFake() *fakeWatcher { return &fakeWatcher{ch: make(chan event, 64)} }
+
+func (f *fakeWatcher) Events() <-chan event { return f.ch }
+func (f *fakeWatcher) Degraded() bool { f.mu.Lock(); defer f.mu.Unlock(); return f.degraded }
+func (f *fakeWatcher) Close() error { return nil }
+func (f *fakeWatcher) Repair() error {
+ f.mu.Lock()
+ defer f.mu.Unlock()
+ f.repairs++
+ return nil
+}
+func (f *fakeWatcher) setDegraded(v bool) { f.mu.Lock(); f.degraded = v; f.mu.Unlock() }
+
+var fast = Options{Deadline: 40 * time.Millisecond, Scan: time.Hour, Poll: 40 * time.Millisecond}
+
+// startRun runs the loop against a fake watcher and returns a stop func.
+func startRun(t *testing.T, p *Publisher, dir string, w watcher, o Options) (stop func() error) {
+ t.Helper()
+ ctx, cancel := context.WithCancel(context.Background())
+ done := make(chan error, 1)
+ go func() { done <- run(ctx, p, dir, o, w) }()
+ return func() error { cancel(); return <-done }
+}
+
+// eventually polls until ok returns true or two seconds pass.
+func eventually(t *testing.T, what string, ok func() bool) {
+ t.Helper()
+ deadline := time.Now().Add(2 * time.Second)
+ for time.Now().Before(deadline) {
+ if ok() {
+ return
+ }
+ time.Sleep(10 * time.Millisecond)
+ }
+ t.Fatalf("timed out waiting for: %s", what)
+}
+
+func manifestHas(s release.Store, path, sha string) bool {
+ m, found, err := Load(context.Background(), s, pfx)
+ if err != nil || !found {
+ return false
+ }
+ e, ok := m.Entries[path]
+ return ok && !e.Deleted && e.SHA256 == sha
+}
+
+func TestRunPublishesATouchedFileAfterTheDeadline(t *testing.T) {
+ s, dir, w := release.NewDirStore(t.TempDir()), t.TempDir(), newFake()
+ p := open(t, s)
+ stop := startRun(t, p, dir, w, fast)
+ defer stop()
+ write(t, dir, "instance.key", "k1")
+ w.ch <- event{rel: "instance.key"}
+ eventually(t, "instance.key published", func() bool { return manifestHas(s, "instance.key", sum([]byte("k1"))) })
+}
+
+func TestRunPublishesADeleteAsATombstone(t *testing.T) {
+ s, dir, w := release.NewDirStore(t.TempDir()), t.TempDir(), newFake()
+ p := open(t, s)
+ write(t, dir, "a", "a")
+ stop := startRun(t, p, dir, w, fast)
+ defer stop()
+ w.ch <- event{rel: "a"}
+ eventually(t, "a published", func() bool { return manifestHas(s, "a", sum([]byte("a"))) })
+ if err := os.Remove(filepath.Join(dir, "a")); err != nil {
+ t.Fatal(err)
+ }
+ w.ch <- event{rel: "a"}
+ eventually(t, "a tombstoned", func() bool {
+ m, _, _ := Load(context.Background(), s, pfx)
+ return m.Entries["a"].Deleted
+ })
+}
+
+func TestRunRescansOnOverflow(t *testing.T) {
+ s, dir, w := release.NewDirStore(t.TempDir()), t.TempDir(), newFake()
+ p := open(t, s)
+ stop := startRun(t, p, dir, w, fast)
+ defer stop()
+ write(t, dir, "missed.txt", "m") // no event for it: the queue overflowed
+ w.ch <- event{rescan: true}
+ eventually(t, "missed.txt found by the rescan", func() bool { return manifestHas(s, "missed.txt", sum([]byte("m"))) })
+}
+
+// A2-5 residual: with a watch missing, changes are found by polling, not
+// by waiting for the ten-minute scan.
+func TestRunPollsWhileTheWatcherIsDegradedAndTriesToRepair(t *testing.T) {
+ s, dir, w := release.NewDirStore(t.TempDir()), t.TempDir(), newFake()
+ w.setDegraded(true)
+ p := open(t, s)
+ stop := startRun(t, p, dir, w, fast)
+ defer stop()
+ write(t, dir, "quiet.txt", "q") // no event at all
+ eventually(t, "quiet.txt found by polling", func() bool { return manifestHas(s, "quiet.txt", sum([]byte("q"))) })
+ eventually(t, "a repair attempt", func() bool { w.mu.Lock(); defer w.mu.Unlock(); return w.repairs > 0 })
+}
+
+func TestRunWithNoWatcherPolls(t *testing.T) {
+ s, dir := release.NewDirStore(t.TempDir()), t.TempDir()
+ p := open(t, s)
+ stop := startRun(t, p, dir, nil, fast)
+ defer stop()
+ write(t, dir, "x", "x")
+ eventually(t, "x found by polling", func() bool { return manifestHas(s, "x", sum([]byte("x"))) })
+}
+
+func TestRunStopsWhenSuperseded(t *testing.T) {
+ s, dir, w := release.NewDirStore(t.TempDir()), t.TempDir(), newFake()
+ p, other := open(t, s), open(t, s)
+ ctx, cancel := context.WithCancel(context.Background())
+ defer cancel()
+ done := make(chan error, 1)
+ go func() { done <- run(ctx, p, dir, fast, w) }()
+ stage(t, other, "theirs", "t")
+ commit(t, other)
+ write(t, dir, "mine", "m")
+ w.ch <- event{rel: "mine"}
+ select {
+ case err := <-done:
+ if !errors.Is(err, ErrSuperseded) {
+ t.Fatalf("err = %v, want ErrSuperseded", err)
+ }
+ case <-time.After(2 * time.Second):
+ t.Fatal("run did not stop after another writer published")
+ }
+}
+
+func TestRunIgnoresAnEventForAPathThatEscapes(t *testing.T) {
+ s, dir, w := release.NewDirStore(t.TempDir()), t.TempDir(), newFake()
+ p := open(t, s)
+ stop := startRun(t, p, dir, w, fast)
+ w.ch <- event{rel: "../outside"}
+ write(t, dir, "ok", "ok")
+ w.ch <- event{rel: "ok"}
+ eventually(t, "ok published", func() bool { return manifestHas(s, "ok", sum([]byte("ok"))) })
+ if err := stop(); !errors.Is(err, context.Canceled) {
+ t.Fatalf("run ended with %v; a bad path must be skipped, not fatal", err)
+ }
+}
+```
+
+- [ ] **Step 5: Run them to see them fail**
+
+Run: `go test ./internal/hostfiles/ -run Run -count=1`
+Expected: FAIL to build, `undefined: run`.
+
+- [ ] **Step 6: Write `run.go`**
+
+```go
+package hostfiles
+
+import (
+ "context"
+ "errors"
+ "log"
+ "path/filepath"
+ "time"
+)
+
+type Options struct {
+ Deadline time.Duration
+ Scan time.Duration
+ Poll time.Duration
+ Skip func(rel string, isDir bool) bool
+ OnStats func(SyncStats)
+}
+
+// DefaultOptions are region HA spec 1 §5.6's values.
+func DefaultOptions() Options {
+ return Options{Deadline: time.Second, Scan: 10 * time.Minute, Poll: time.Second}
+}
+
+type event struct {
+ rel string
+ rescan bool
+}
+
+type watcher interface {
+ Events() <-chan event
+ Degraded() bool
+ Repair() error
+ Close() error
+}
+
+// Run publishes dir's changes until ctx ends (it returns ctx.Err()) or
+// another writer supersedes the manifest (ErrSuperseded). Where the OS has
+// no watcher, it polls.
+func Run(ctx context.Context, p *Publisher, dir string, o Options) error {
+ w, err := newWatcher(dir, o.Skip)
+ if err != nil {
+ log.Printf("hostfiles: no watcher for %s (%v): polling every %s", dir, err, o.Poll)
+ w = nil
+ }
+ if w != nil {
+ defer w.Close()
+ }
+ return run(ctx, p, dir, o, w)
+}
+
+func run(ctx context.Context, p *Publisher, dir string, o Options, w watcher) error {
+ var events <-chan event
+ if w != nil {
+ events = w.Events()
+ }
+ q := newQueue(o.Deadline)
+ tick := time.NewTicker(o.Deadline / 4)
+ defer tick.Stop()
+
+ needScan := true // the first pass is a full scan: changes made while nothing watched
+ var lastScan time.Time
+ for {
+ select {
+ case <-ctx.Done():
+ return ctx.Err()
+ case ev := <-events:
+ if ev.rescan {
+ needScan = true
+ } else {
+ q.Touch(ev.rel, time.Now())
+ }
+ continue
+ case <-tick.C:
+ }
+ now := time.Now()
+ polling := w == nil || w.Degraded()
+ switch {
+ case polling && now.Sub(lastScan) >= o.Poll:
+ needScan = true
+ case !polling && now.Sub(lastScan) >= o.Scan:
+ needScan = true
+ }
+ if needScan {
+ if w != nil && w.Degraded() {
+ if err := w.Repair(); err != nil {
+ log.Printf("hostfiles: %s: watch coverage still missing: %v", dir, err)
+ }
+ }
+ st, err := SyncDir(ctx, p, dir, o.Skip)
+ if err != nil {
+ if ctx.Err() != nil {
+ return ctx.Err()
+ }
+ log.Printf("hostfiles: %v", err)
+ } else {
+ needScan, lastScan = false, now
+ if o.OnStats != nil {
+ o.OnStats(st)
+ }
+ }
+ }
+ for _, rel := range q.Due(now) {
+ again, err := publishOne(ctx, p, dir, rel, o.Skip)
+ if err != nil {
+ log.Printf("hostfiles: %s: %v", rel, err)
+ q.Touch(rel, now) // try again at the next deadline
+ continue
+ }
+ if again {
+ q.Touch(rel, now)
+ }
+ }
+ if p.Dirty() {
+ if _, err := p.Commit(ctx); errors.Is(err, ErrSuperseded) {
+ return err
+ } else if err != nil {
+ log.Printf("hostfiles: commit: %v", err) // staged changes stay; the next tick retries
+ }
+ }
+ }
+}
+
+// publishOne stages one path's current state. again is true when the file
+// changed while it was read: its image is published flagged, and the
+// caller reads it again at the next deadline.
+func publishOne(ctx context.Context, p *Publisher, dir, rel string, skip func(string, bool) bool) (again bool, err error) {
+ clean, err := CleanPath(rel)
+ if err != nil {
+ return false, nil // a path the watcher should never have produced: skip it
+ }
+ if skip != nil && skip(clean, false) {
+ return false, nil
+ }
+ body, mode, state, err := readStable(filepath.Join(dir, filepath.FromSlash(clean)))
+ if err != nil {
+ return false, err
+ }
+ switch state {
+ case fileGone, fileNotRegular:
+ return false, p.StageDelete(clean)
+ case fileTooLarge:
+ return false, p.StageSkipped(clean, "too-large")
+ }
+ return state == fileTorn, p.Stage(ctx, clean, body, mode, state == fileTorn)
+}
+```
+
+A path that turns out to be a directory arrives as `fileNotRegular` and `StageDelete` is a no-op for a path the manifest never had, so directory events need no special case.
+
+- [ ] **Step 7: Write the non-Linux stub**
+
+`internal/hostfiles/watch_other.go`:
+
+```go
+//go:build !linux
+
+package hostfiles
+
+import "errors"
+
+// newWatcher has no implementation off Linux; Run polls instead.
+func newWatcher(root string, skip func(string, bool) bool) (watcher, error) {
+ return nil, errors.New("no file watcher on this OS")
+}
+```
+
+- [ ] **Step 8: Run the loop tests**
+
+Run: `go test ./internal/hostfiles/ -run 'Run|Queue' -count=1` (on Linux this needs Step 9's file to build; if so, do Step 9 first and run both).
+Expected: PASS.
+
+- [ ] **Step 9: Write the Linux watcher and its test**
+
+`internal/hostfiles/watch_linux_test.go`:
+
+```go
+//go:build linux
+
+package hostfiles
+
+import (
+ "context"
+ "os"
+ "path/filepath"
+ "testing"
+ "time"
+
+ "github.com/carlosframework/platform/internal/release"
+)
+
+var realFast = Options{Deadline: 50 * time.Millisecond, Scan: time.Hour, Poll: time.Hour}
+
+func startReal(t *testing.T, s release.Store, dir string) {
+ t.Helper()
+ p := open(t, s)
+ ctx, cancel := context.WithCancel(context.Background())
+ done := make(chan error, 1)
+ go func() { done <- Run(ctx, p, dir, realFast) }()
+ t.Cleanup(func() { cancel(); <-done })
+ // Let the first full scan finish, so what follows is found by events.
+ time.Sleep(100 * time.Millisecond)
+}
+
+func TestWatcherPublishesAFileStillHeldOpen(t *testing.T) {
+ s, dir := release.NewDirStore(t.TempDir()), t.TempDir()
+ startReal(t, s, dir)
+ f, err := os.Create(filepath.Join(dir, "open.log"))
+ if err != nil {
+ t.Fatal(err)
+ }
+ defer f.Close() // deliberately not closed before the assertion
+ if _, err := f.WriteString("line one\n"); err != nil {
+ t.Fatal(err)
+ }
+ eventually(t, "open.log published while still open", func() bool {
+ return manifestHas(s, "open.log", sum([]byte("line one\n")))
+ })
+}
+
+func TestWatcherSeesAFileInADirectoryCreatedAfterItStarted(t *testing.T) {
+ s, dir := release.NewDirStore(t.TempDir()), t.TempDir()
+ startReal(t, s, dir)
+ write(t, dir, "new/deep/a.txt", "a")
+ eventually(t, "new/deep/a.txt published", func() bool {
+ return manifestHas(s, "new/deep/a.txt", sum([]byte("a")))
+ })
+ // And a later write inside the new directory is an event, not a scan.
+ time.Sleep(150 * time.Millisecond)
+ write(t, dir, "new/deep/b.txt", "b")
+ eventually(t, "new/deep/b.txt published", func() bool {
+ return manifestHas(s, "new/deep/b.txt", sum([]byte("b")))
+ })
+}
+
+func TestWatcherPublishesARenameAsDeleteAndCreate(t *testing.T) {
+ s, dir := release.NewDirStore(t.TempDir()), t.TempDir()
+ write(t, dir, "tmp", "v")
+ startReal(t, s, dir)
+ if err := os.Rename(filepath.Join(dir, "tmp"), filepath.Join(dir, "final")); err != nil {
+ t.Fatal(err)
+ }
+ eventually(t, "final present and tmp tombstoned", func() bool {
+ m, _, _ := Load(context.Background(), s, pfx)
+ return m.Entries["tmp"].Deleted && m.Entries["final"].SHA256 == sum([]byte("v"))
+ })
+}
+
+func TestWatcherDoesNotWatchASkippedDirectory(t *testing.T) {
+ root := t.TempDir()
+ if err := os.MkdirAll(filepath.Join(root, "cache"), 0o755); err != nil {
+ t.Fatal(err)
+ }
+ w, err := newWatcher(root, func(rel string, isDir bool) bool { return isDir && rel == "cache" })
+ if err != nil {
+ t.Fatal(err)
+ }
+ defer w.Close()
+ write(t, root, "cache/x", "x")
+ select {
+ case ev := <-w.Events():
+ t.Fatalf("event from a skipped directory: %+v", ev)
+ case <-time.After(150 * time.Millisecond):
+ }
+}
+```
+
+`internal/hostfiles/watch_linux.go`:
+
+```go
+//go:build linux
+
+package hostfiles
+
+import (
+ "errors"
+ "fmt"
+ "io/fs"
+ "os"
+ "path/filepath"
+ "sync"
+ "sync/atomic"
+ "unsafe"
+
+ "golang.org/x/sys/unix"
+)
+
+const watchMask = unix.IN_MODIFY | unix.IN_CLOSE_WRITE | unix.IN_MOVED_FROM | unix.IN_MOVED_TO |
+ unix.IN_CREATE | unix.IN_DELETE | unix.IN_ATTRIB
+
+type inotifyWatcher struct {
+ root string
+ skip func(string, bool) bool
+ f *os.File // the inotify descriptor, non-blocking, so Close unblocks Read
+ ch chan event
+
+ mu sync.Mutex
+ wds map[int32]string // watch descriptor -> directory, relative to root ("" is root)
+
+ degraded atomic.Bool
+}
+
+func newWatcher(root string, skip func(string, bool) bool) (watcher, error) {
+ fd, err := unix.InotifyInit1(unix.IN_CLOEXEC | unix.IN_NONBLOCK)
+ if err != nil {
+ return nil, fmt.Errorf("inotify init: %w", err)
+ }
+ w := &inotifyWatcher{root: root, skip: skip, f: os.NewFile(uintptr(fd), "inotify"),
+ ch: make(chan event, 4096), wds: map[int32]string{}}
+ if err := w.addTree(""); err != nil {
+ w.degraded.Store(true) // partial coverage: Run polls and calls Repair
+ }
+ go w.loop()
+ return w, nil
+}
+
+func (w *inotifyWatcher) Events() <-chan event { return w.ch }
+func (w *inotifyWatcher) Degraded() bool { return w.degraded.Load() }
+func (w *inotifyWatcher) Close() error { return w.f.Close() }
+
+// Repair re-adds a watch on every directory. InotifyAddWatch on a path
+// already watched returns the same descriptor, so this is safe to repeat.
+func (w *inotifyWatcher) Repair() error {
+ if err := w.addTree(""); err != nil {
+ return err
+ }
+ w.degraded.Store(false)
+ return nil
+}
+
+// addTree watches rel and every directory under it. It keeps going after a
+// failure, so one unreadable directory does not cost the rest their
+// watches, and returns the first error.
+func (w *inotifyWatcher) addTree(rel string) error {
+ var first error
+ start := filepath.Join(w.root, filepath.FromSlash(rel))
+ err := filepath.WalkDir(start, func(full string, d fs.DirEntry, err error) error {
+ if err != nil {
+ if first == nil {
+ first = err
+ }
+ return nil
+ }
+ if !d.IsDir() {
+ return nil // WalkDir does not follow symlinks, so a linked directory is never watched
+ }
+ r, rerr := filepath.Rel(w.root, full)
+ if rerr != nil {
+ return nil
+ }
+ r = filepath.ToSlash(r)
+ if r == "." {
+ r = ""
+ }
+ if r != "" && w.skip != nil && w.skip(r, true) {
+ return filepath.SkipDir
+ }
+ wd, aerr := unix.InotifyAddWatch(int(w.f.Fd()), full, watchMask|unix.IN_DONT_FOLLOW|unix.IN_ONLYDIR)
+ if aerr != nil {
+ if first == nil {
+ first = fmt.Errorf("watch %s: %w", full, aerr)
+ }
+ return nil
+ }
+ w.mu.Lock()
+ w.wds[int32(wd)] = r
+ w.mu.Unlock()
+ return nil
+ })
+ if err != nil && first == nil {
+ first = err
+ }
+ return first
+}
+
+func (w *inotifyWatcher) send(ev event) {
+ select {
+ case w.ch <- ev:
+ default:
+ // Our own buffer is full: the same remedy as a kernel overflow.
+ // A rescan request must not be lost, so make room for it.
+ select {
+ case <-w.ch:
+ default:
+ }
+ select {
+ case w.ch <- event{rescan: true}:
+ default:
+ }
+ }
+}
+
+func (w *inotifyWatcher) loop() {
+ defer close(w.ch)
+ buf := make([]byte, 64<<10)
+ for {
+ n, err := w.f.Read(buf)
+ if err != nil {
+ if !errors.Is(err, os.ErrClosed) {
+ w.degraded.Store(true)
+ }
+ return
+ }
+ for off := 0; off+unix.SizeofInotifyEvent <= n; {
+ raw := (*unix.InotifyEvent)(unsafe.Pointer(&buf[off]))
+ nameLen := int(raw.Len)
+ name := ""
+ if nameLen > 0 {
+ b := buf[off+unix.SizeofInotifyEvent : off+unix.SizeofInotifyEvent+nameLen]
+ for i, c := range b {
+ if c == 0 {
+ b = b[:i]
+ break
+ }
+ }
+ name = string(b)
+ }
+ off += unix.SizeofInotifyEvent + nameLen
+ w.handle(raw.Wd, raw.Mask, name)
+ }
+ }
+}
+
+func (w *inotifyWatcher) handle(wd int32, mask uint32, name string) {
+ if mask&unix.IN_Q_OVERFLOW != 0 {
+ w.send(event{rescan: true})
+ return
+ }
+ w.mu.Lock()
+ dir, known := w.wds[wd]
+ if mask&unix.IN_IGNORED != 0 {
+ delete(w.wds, wd)
+ }
+ w.mu.Unlock()
+ if !known || name == "" {
+ return
+ }
+ rel := name
+ if dir != "" {
+ rel = dir + "/" + name
+ }
+ if mask&unix.IN_ISDIR != 0 {
+ if mask&(unix.IN_CREATE|unix.IN_MOVED_TO) != 0 {
+ if w.skip != nil && w.skip(rel, true) {
+ return
+ }
+ if err := w.addTree(rel); err != nil {
+ w.degraded.Store(true)
+ }
+ }
+ // Files may have appeared in the new directory before its watch
+ // did, and a directory moved away takes its files with it: only a
+ // scan finds either.
+ w.send(event{rescan: true})
+ return
+ }
+ if w.skip != nil && w.skip(rel, false) {
+ return
+ }
+ w.send(event{rel: rel})
+}
+```
+
+- [ ] **Step 10: Run everything, and the cross-builds**
+
+Run: `gofmt -l internal/hostfiles; go vet ./internal/hostfiles/ && go test ./internal/hostfiles/ -count=1 -race && GOOS=darwin go build ./internal/hostfiles/ && GOOS=windows go build ./internal/hostfiles/`
+Expected: no gofmt output, PASS, both cross-builds succeed.
+
+If `TestWatcherPublishesAFileStillHeldOpen` is flaky under `-race` on a loaded machine, do not raise `eventually`'s two-second bound past five seconds: look for a lost event first (a `send` that dropped a path rather than requesting a rescan).
+
+- [ ] **Step 11: Commit**
+
+```bash
+git add internal/hostfiles
+git commit -m "hostfiles: a changed file is published a second after its first change, open or not
+
+Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>"
+```
+
+---
+
+### Task 8: Wake state travels with the files (`registry.ReplaceTimers`, `hostfiles/wake.go`)
+
+**Files:**
+- Modify: `internal/registry/timers.go`
+- Create: `internal/hostfiles/wake.go`
+- Test: `internal/registry/timers_test.go`, `internal/hostfiles/wake_test.go`
+
+**Interfaces:**
+- Consumes: `registry.Timer` (its `Registration` field is `json:"-"`), `(*Registry).TimersFor(host) ([]Timer, error)`, `(*Registry).Get(host) (*Route, error)`, `SetNextDue(host string, due int64) error`, `SetReplication(host, class string) error`, `registry.ValidReplication`, `Publisher.StagePlatform`, `ReadPlatform`.
+- Produces:
+ ```go
+ // registry
+ func (r *Registry) ReplaceTimers(host string, rows []Timer) error
+ // hostfiles
+ const WakeItem = "wake.json"
+ type WakeTimer struct {
+ Name, Kind, Spec, Path string
+ NextAt, LastAt int64
+ LastStatus string
+ LastMs int64
+ Attempts int
+ CreatedAt, DueAt int64
+ Registration string
+ }
+ type WakeState struct {
+ V int `json:"v"`
+ Host string `json:"host"`
+ Timers []WakeTimer `json:"timers"`
+ NextDue int64 `json:"next_due"`
+ Replication string `json:"replication"`
+ }
+ func ExportWake(reg *registry.Registry, host string) (WakeState, error)
+ func ImportWake(reg *registry.Registry, host string, ws WakeState) error
+ func StageWake(ctx context.Context, p *Publisher, ws WakeState) error
+ func LoadWake(ctx context.Context, s release.Store, prefix string, m Manifest) (WakeState, bool, error)
+ ```
+
+Spec §5.7: the wake state is the host's timer rows with their bookkeeping, the route's `NextDue`, and its `Replication` class. Completed once-timers (`kind == "once"` with `next_at == 0`) are history and are left out, as the moves spec's digest leaves them out, so the runner's daily prune never changes what is exported. An occurrence that fell due while no box ran the host fires once after import, because `DueTimers` selects by the imported `next_at`.
+
+- [ ] **Step 1: Write the failing registry test**
+
+Append to `internal/registry/timers_test.go`:
+
+```go
+func TestReplaceTimersSwapsAHostsRowsExactly(t *testing.T) {
+ r := openTimersReg(t)
+ for _, tm := range []Timer{
+ {Host: "a.test", Name: "old", Kind: "schedule", Spec: "@every 1m", Path: "/t", NextAt: 100, CreatedAt: 1},
+ {Host: "b.test", Name: "other", Kind: "schedule", Spec: "@every 1m", Path: "/t", NextAt: 100, CreatedAt: 1},
+ } {
+ if err := r.UpsertTimer(tm); err != nil {
+ t.Fatal(err)
+ }
+ }
+ rows := []Timer{
+ {Host: "a.test", Name: "nightly", Kind: "schedule", Spec: "0 3 * * *", Path: "/n", NextAt: 5000, LastAt: 4000,
+ LastStatus: "ok", LastMs: 12, Attempts: 2, CreatedAt: 10, DueAt: 4900, Registration: "reg-nightly"},
+ {Host: "a.test", Name: "once-1", Kind: "once", Spec: "2026-10-06T00:00:00Z", Path: "/o", NextAt: 6000, CreatedAt: 11, DueAt: 6000, Registration: "reg-once"},
+ }
+ if err := r.ReplaceTimers("a.test", rows); err != nil {
+ t.Fatal(err)
+ }
+ got, err := r.TimersFor("a.test")
+ if err != nil {
+ t.Fatal(err)
+ }
+ if len(got) != 2 {
+ t.Fatalf("a.test has %d timers, want 2: %+v", len(got), got)
+ }
+ n := got[0] // ordered by name: nightly, once-1
+ if n.Name != "nightly" || n.NextAt != 5000 || n.LastAt != 4000 || n.LastStatus != "ok" || n.LastMs != 12 ||
+ n.Attempts != 2 || n.CreatedAt != 10 || n.DueAt != 4900 || n.Registration != "reg-nightly" {
+ t.Fatalf("bookkeeping not preserved: %+v", n)
+ }
+ if other, _ := r.TimersFor("b.test"); len(other) != 1 {
+ t.Fatalf("b.test has %d timers, want 1: another host's rows were touched", len(other))
+ }
+}
+
+func TestReplaceTimersIsAllOrNothing(t *testing.T) {
+ r := openTimersReg(t)
+ if err := r.UpsertTimer(Timer{Host: "a.test", Name: "keep", Kind: "schedule", Spec: "@every 1m", Path: "/t", NextAt: 100, CreatedAt: 1}); err != nil {
+ t.Fatal(err)
+ }
+ bad := []Timer{
+ {Host: "a.test", Name: "x", Kind: "schedule", Spec: "@every 1m", Path: "/t", NextAt: 1, CreatedAt: 1},
+ {Host: "elsewhere.test", Name: "y", Kind: "schedule", Spec: "@every 1m", Path: "/t", NextAt: 1, CreatedAt: 1},
+ }
+ if err := r.ReplaceTimers("a.test", bad); err == nil {
+ t.Fatal("a row for another host was accepted")
+ }
+ got, _ := r.TimersFor("a.test")
+ if len(got) != 1 || got[0].Name != "keep" {
+ t.Fatalf("a failed replace changed the rows: %+v", got)
+ }
+}
+
+func TestReplaceTimersWithNoRowsClearsTheHost(t *testing.T) {
+ r := openTimersReg(t)
+ if err := r.UpsertTimer(Timer{Host: "a.test", Name: "gone", Kind: "schedule", Spec: "@every 1m", Path: "/t", NextAt: 100, CreatedAt: 1}); err != nil {
+ t.Fatal(err)
+ }
+ if err := r.ReplaceTimers("a.test", nil); err != nil {
+ t.Fatal(err)
+ }
+ if got, _ := r.TimersFor("a.test"); len(got) != 0 {
+ t.Fatalf("rows left: %+v", got)
+ }
+}
+```
+
+- [ ] **Step 2: Run it to see it fail**
+
+Run: `go test ./internal/registry/ -run ReplaceTimers -count=1`
+Expected: FAIL to build, `r.ReplaceTimers undefined`.
+
+- [ ] **Step 3: Add `ReplaceTimers` to `internal/registry/timers.go`**
+
+Add after `UpsertTimer`:
+
+```go
+// ReplaceTimers makes host's timer rows exactly rows, in one transaction,
+// keeping every column as given: next_at, due_at, attempts and
+// registration included. It is how an installation imports the wake state
+// it was handed (region HA spec 1 §5.7, moves spec §9.4).
+//
+// UpsertTimer is the wrong tool for that: it mints a fresh registration
+// and resets the runner's bookkeeping on a rearm.
+func (r *Registry) ReplaceTimers(host string, rows []Timer) error {
+ for _, t := range rows {
+ if t.Host != host {
+ return fmt.Errorf("registry: timer %q is for host %q, not %q", t.Name, t.Host, host)
+ }
+ }
+ tx, err := r.sql.Begin()
+ if err != nil {
+ return err
+ }
+ defer tx.Rollback()
+ if _, err := tx.Exec(`DELETE FROM timers WHERE host = ?`, host); err != nil {
+ return err
+ }
+ for _, t := range rows {
+ if _, err := tx.Exec(`INSERT INTO timers (`+timerCols+`) VALUES (?,?,?,?,?,?,?,?,?,?,?,?,?)`,
+ t.Host, t.Name, t.Kind, t.Spec, t.Path, t.NextAt, t.LastAt, t.LastStatus, t.LastMs, t.Attempts, t.CreatedAt, t.DueAt, t.Registration); err != nil {
+ return fmt.Errorf("registry: insert timer %q: %w", t.Name, err)
+ }
+ }
+ return tx.Commit()
+}
+```
+
+If `timers.go` does not already import `fmt`, add it.
+
+Run: `go test ./internal/registry/ -count=1`. Expected: PASS.
+
+- [ ] **Step 4: Write the failing wake-state tests**
+
+`internal/hostfiles/wake_test.go`:
+
+```go
+package hostfiles
+
+import (
+ "context"
+ "path/filepath"
+ "testing"
+
+ "github.com/carlosframework/platform/internal/registry"
+ "github.com/carlosframework/platform/internal/release"
+)
+
+func wakeReg(t *testing.T) *registry.Registry {
+ t.Helper()
+ r, err := registry.Open(filepath.Join(t.TempDir(), "registry.db"))
+ if err != nil {
+ t.Fatal(err)
+ }
+ t.Cleanup(func() { r.Close() })
+ return r
+}
+
+func seedHost(t *testing.T, r *registry.Registry, host string) {
+ t.Helper()
+ if err := r.Upsert(registry.Route{Host: host, Kind: "service", Addr: "10.0.0.1:3000", Status: "ready", Hibernate: true}); err != nil {
+ t.Fatal(err)
+ }
+}
+
+func TestWakeStateRoundTripsThroughTheManifest(t *testing.T) {
+ ctx, s := context.Background(), release.NewDirStore(t.TempDir())
+ src, dst := wakeReg(t), wakeReg(t)
+ seedHost(t, src, "shop.test")
+ seedHost(t, dst, "shop.test")
+ if err := src.ReplaceTimers("shop.test", []registry.Timer{
+ {Host: "shop.test", Name: "nightly", Kind: "schedule", Spec: "0 3 * * *", Path: "/n", NextAt: 5000, LastAt: 4000,
+ LastStatus: "ok", Attempts: 1, CreatedAt: 10, DueAt: 4900, Registration: "reg-n"},
+ {Host: "shop.test", Name: "remind", Kind: "once", Spec: "2026-10-06T00:00:00Z", Path: "/r", NextAt: 6000, CreatedAt: 11, DueAt: 6000, Registration: "reg-r"},
+ {Host: "shop.test", Name: "done", Kind: "once", Spec: "2026-10-01T00:00:00Z", Path: "/d", NextAt: 0, LastAt: 3000, CreatedAt: 9, Registration: "reg-d"},
+ }); err != nil {
+ t.Fatal(err)
+ }
+ if err := src.SetNextDue("shop.test", 7777); err != nil {
+ t.Fatal(err)
+ }
+ if err := src.SetReplication("shop.test", registry.ReplicationBatched); err != nil {
+ t.Fatal(err)
+ }
+
+ ws, err := ExportWake(src, "shop.test")
+ if err != nil {
+ t.Fatal(err)
+ }
+ if len(ws.Timers) != 2 {
+ t.Fatalf("exported %d timers, want 2: a finished once-timer is history, not wake state", len(ws.Timers))
+ }
+ p := open(t, s)
+ if err := StageWake(ctx, p, ws); err != nil {
+ t.Fatal(err)
+ }
+ commit(t, p)
+
+ m, _, err := Load(ctx, s, pfx)
+ if err != nil {
+ t.Fatal(err)
+ }
+ got, found, err := LoadWake(ctx, s, pfx, m)
+ if err != nil || !found {
+ t.Fatalf("LoadWake found=%v err=%v", found, err)
+ }
+ if err := ImportWake(dst, "shop.test", got); err != nil {
+ t.Fatal(err)
+ }
+ rows, _ := dst.TimersFor("shop.test")
+ if len(rows) != 2 || rows[0].Name != "nightly" || rows[0].Registration != "reg-n" || rows[0].NextAt != 5000 || rows[0].Attempts != 1 {
+ t.Fatalf("imported timers: %+v", rows)
+ }
+ rt, err := dst.Get("shop.test")
+ if err != nil || rt == nil {
+ t.Fatalf("route: %v %v", rt, err)
+ }
+ if rt.NextDue != 7777 || rt.Replication != registry.ReplicationBatched {
+ t.Fatalf("NextDue=%d Replication=%q", rt.NextDue, rt.Replication)
+ }
+}
+
+func TestStageWakeOfUnchangedStateStagesNothing(t *testing.T) {
+ ctx, s := context.Background(), release.NewDirStore(t.TempDir())
+ p := open(t, s)
+ ws := WakeState{V: 1, Host: "shop.test", NextDue: 5}
+ if err := StageWake(ctx, p, ws); err != nil {
+ t.Fatal(err)
+ }
+ commit(t, p)
+ if err := StageWake(ctx, p, ws); err != nil {
+ t.Fatal(err)
+ }
+ if p.Dirty() {
+ t.Fatal("unchanged wake state was staged again")
+ }
+}
+
+func TestLoadWakeWithNoneReportsNotFound(t *testing.T) {
+ ctx, s := context.Background(), release.NewDirStore(t.TempDir())
+ m, _, _ := Load(ctx, s, pfx)
+ if _, found, err := LoadWake(ctx, s, pfx, m); found || err != nil {
+ t.Fatalf("found=%v err=%v", found, err)
+ }
+}
+
+func TestImportWakeRefusesAnotherHostsState(t *testing.T) {
+ r := wakeReg(t)
+ seedHost(t, r, "shop.test")
+ if err := ImportWake(r, "shop.test", WakeState{V: 1, Host: "other.test"}); err == nil {
+ t.Fatal("imported another host's wake state")
+ }
+}
+
+func TestImportWakeRefusesAnUnknownReplicationClassAndChangesNothing(t *testing.T) {
+ r := wakeReg(t)
+ seedHost(t, r, "shop.test")
+ if err := r.ReplaceTimers("shop.test", []registry.Timer{{Host: "shop.test", Name: "keep", Kind: "schedule", Spec: "@every 1m", Path: "/k", NextAt: 1, CreatedAt: 1}}); err != nil {
+ t.Fatal(err)
+ }
+ err := ImportWake(r, "shop.test", WakeState{V: 1, Host: "shop.test", Replication: "hourly"})
+ if err == nil {
+ t.Fatal("accepted replication class \"hourly\"")
+ }
+ if rows, _ := r.TimersFor("shop.test"); len(rows) != 1 {
+ t.Fatalf("a refused import changed the timers: %+v", rows)
+ }
+}
+
+func TestExportWakeOfAnUnknownHostIsAnError(t *testing.T) {
+ if _, err := ExportWake(wakeReg(t), "nobody.test"); err == nil {
+ t.Fatal("exported wake state for a host with no route")
+ }
+}
+```
+
+- [ ] **Step 5: Run them to see them fail**
+
+Run: `go test ./internal/hostfiles/ -run Wake -count=1`
+Expected: FAIL to build, `undefined: ExportWake`.
+
+- [ ] **Step 6: Write `wake.go`**
+
+```go
+package hostfiles
+
+import (
+ "context"
+ "encoding/json"
+ "errors"
+ "fmt"
+ "os"
+
+ "github.com/carlosframework/platform/internal/registry"
+ "github.com/carlosframework/platform/internal/release"
+)
+
+// WakeItem is the wake state's name among the manifest's platform items.
+const WakeItem = "wake.json"
+
+// WakeTimer is registry.Timer on the wire. It exists because
+// registry.Timer hides Registration from JSON, and an import must keep it:
+// the runner's delivery bookkeeping is keyed by it.
+type WakeTimer struct {
+ Name string `json:"name"`
+ Kind string `json:"kind"`
+ Spec string `json:"spec"`
+ Path string `json:"path"`
+ NextAt int64 `json:"next_at"`
+ LastAt int64 `json:"last_at"`
+ LastStatus string `json:"last_status"`
+ LastMs int64 `json:"last_ms"`
+ Attempts int `json:"attempts"`
+ CreatedAt int64 `json:"created_at"`
+ DueAt int64 `json:"due_at"`
+ Registration string `json:"registration"`
+}
+
+// WakeState is everything that decides when a host next runs without a
+// request: its timers, the app's own next wake, and its replication class
+// (region HA spec 1 §5.7).
+type WakeState struct {
+ V int `json:"v"`
+ Host string `json:"host"`
+ Timers []WakeTimer `json:"timers"`
+ NextDue int64 `json:"next_due"`
+ Replication string `json:"replication"`
+}
+
+// ExportWake reads host's wake state. A finished once-timer (next_at 0) is
+// history: it is left out, so the runner's daily prune of such rows never
+// changes what is exported.
+func ExportWake(reg *registry.Registry, host string) (WakeState, error) {
+ rt, err := reg.Get(host)
+ if err != nil {
+ return WakeState{}, err
+ }
+ if rt == nil {
+ return WakeState{}, fmt.Errorf("hostfiles: wake state for %s: no route", host)
+ }
+ rows, err := reg.TimersFor(host)
+ if err != nil {
+ return WakeState{}, err
+ }
+ ws := WakeState{V: 1, Host: host, Timers: []WakeTimer{}, NextDue: rt.NextDue, Replication: rt.Replication}
+ for _, t := range rows {
+ if t.Kind == "once" && t.NextAt == 0 {
+ continue
+ }
+ ws.Timers = append(ws.Timers, WakeTimer{Name: t.Name, Kind: t.Kind, Spec: t.Spec, Path: t.Path,
+ NextAt: t.NextAt, LastAt: t.LastAt, LastStatus: t.LastStatus, LastMs: t.LastMs, Attempts: t.Attempts,
+ CreatedAt: t.CreatedAt, DueAt: t.DueAt, Registration: t.Registration})
+ }
+ return ws, nil
+}
+
+// ImportWake makes host's wake state exactly ws. It validates everything
+// before it writes anything.
+func ImportWake(reg *registry.Registry, host string, ws WakeState) error {
+ if ws.V != 1 {
+ return fmt.Errorf("hostfiles: wake state version %d; this binary reads 1", ws.V)
+ }
+ if ws.Host != host {
+ return fmt.Errorf("hostfiles: wake state is for %q, not %q", ws.Host, host)
+ }
+ if !registry.ValidReplication(ws.Replication) {
+ return fmt.Errorf("hostfiles: wake state for %s names replication class %q", host, ws.Replication)
+ }
+ rows := make([]registry.Timer, 0, len(ws.Timers))
+ for _, t := range ws.Timers {
+ rows = append(rows, registry.Timer{Host: host, Name: t.Name, Kind: t.Kind, Spec: t.Spec, Path: t.Path,
+ NextAt: t.NextAt, LastAt: t.LastAt, LastStatus: t.LastStatus, LastMs: t.LastMs, Attempts: t.Attempts,
+ CreatedAt: t.CreatedAt, DueAt: t.DueAt, Registration: t.Registration})
+ }
+ if err := reg.ReplaceTimers(host, rows); err != nil {
+ return err
+ }
+ if err := reg.SetNextDue(host, ws.NextDue); err != nil {
+ return err
+ }
+ if ws.Replication != "" {
+ return reg.SetReplication(host, ws.Replication)
+ }
+ return nil
+}
+
+// StageWake stages the wake state as a platform item. Unchanged state
+// stages nothing.
+func StageWake(ctx context.Context, p *Publisher, ws WakeState) error {
+ body, err := json.Marshal(ws)
+ if err != nil {
+ return fmt.Errorf("hostfiles: marshal wake state: %w", err)
+ }
+ return p.StagePlatform(ctx, WakeItem, body)
+}
+
+// LoadWake reads the wake state a manifest carries. The bool is false when
+// it carries none.
+func LoadWake(ctx context.Context, s release.Store, prefix string, m Manifest) (WakeState, bool, error) {
+ body, err := ReadPlatform(ctx, s, prefix, m, WakeItem)
+ if errors.Is(err, os.ErrNotExist) {
+ return WakeState{}, false, nil
+ }
+ if err != nil {
+ return WakeState{}, false, err
+ }
+ var ws WakeState
+ if err := json.Unmarshal(body, &ws); err != nil {
+ return WakeState{}, false, fmt.Errorf("hostfiles: wake state: %w", err)
+ }
+ return ws, true, nil
+}
+```
+
+- [ ] **Step 7: Check the registry facts the code relies on**
+
+Run: `grep -n "func ValidReplication" -A8 internal/registry/registry.go; grep -n "func (r \*Registry) SetReplication" -A12 internal/registry/registry.go`
+
+Confirm two things and adjust if either is false:
+- `ValidReplication("")` returns true (`""` is the pre-ladder default). If it returns false, change the check in `ImportWake` to `ws.Replication != "" && !registry.ValidReplication(ws.Replication)`.
+- `SetReplication` on a route with `Hibernate` false: the report says Upsert clears `Replication` when `!t.Hibernate`. If `SetReplication` errors for such a route, `TestWakeStateRoundTripsThroughTheManifest` already seeds `Hibernate: true`; keep it, and make `ImportWake` skip `SetReplication` when the route's `Hibernate` is false (read it with `reg.Get`), with a comment saying why.
+
+- [ ] **Step 8: Run the tests**
+
+Run: `go test ./internal/hostfiles/ ./internal/registry/ -count=1`
+Expected: PASS.
+
+- [ ] **Step 9: Commit**
+
+```bash
+git add internal/registry/timers.go internal/registry/timers_test.go internal/hostfiles/wake.go internal/hostfiles/wake_test.go
+git commit -m "hostfiles: a host's timers, next wake and replication class travel with its files
+
+Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>"
+```
+
+---
+
+### Task 9: Termination fences (`internal/termfence`)
+
+**Files:**
+- Create: `internal/termfence/fence.go`
+- Test: `internal/termfence/fence_test.go`
+
+**Interfaces:**
+- Consumes: `boxid.Identity`; `github.com/aws/aws-sdk-go-v2/service/ec2` and its `types` (already a dependency).
+- Produces:
+ ```go
+ package termfence
+ const (
+ ViaDescribe = "describe"
+ ViaEvent = "event"
+ )
+ type Observation struct {
+ Via string
+ Found bool // the source said something about this instance
+ InstanceID string
+ Account string
+ Region string
+ State string // EC2's state name, verbatim
+ EventID string
+ }
+ type Fence struct {
+ Box string `json:"box"`
+ InstanceID string `json:"instance_id"`
+ Account string `json:"account"`
+ AWSRegion string `json:"aws_region"`
+ State string `json:"state"`
+ ObservedAt string `json:"observed_at"`
+ Via string `json:"via"`
+ EventID string `json:"event_id,omitempty"`
+ }
+ var ErrNotEvidence = errors.New("termfence: not evidence of termination")
+ func FromDescribe(out *ec2.DescribeInstancesOutput, instanceID, region string) Observation
+ func FromEvent(body []byte) (Observation, error)
+ func Verify(box string, id boxid.Identity, o Observation, now time.Time) (Fence, error)
+ ```
+
+Spec §5.9 and HA7. Only `terminated` counts. A missing instance, an error, another account or another region is never evidence. This task builds the two observation sources that exist today; the third (Auto Scaling activity history) is off until spec 4 verifies it on dev, so it is not built here. Reading the queue and calling EC2 belong to the placer (spec 3): this package is pure.
+
+- [ ] **Step 1: Write the failing tests**
+
+`internal/termfence/fence_test.go`:
+
+```go
+package termfence
+
+import (
+ "errors"
+ "testing"
+ "time"
+
+ "github.com/aws/aws-sdk-go-v2/aws"
+ "github.com/aws/aws-sdk-go-v2/service/ec2"
+ "github.com/aws/aws-sdk-go-v2/service/ec2/types"
+
+ "github.com/carlosframework/platform/internal/boxid"
+)
+
+var (
+ id = boxid.Identity{AccountID: "111122223333", Region: "eu-west-1", InstanceID: "i-0abc"}
+ now = time.Date(2026, 10, 5, 12, 0, 0, 0, time.UTC)
+)
+
+func describe(owner, instance string, state types.InstanceStateName) *ec2.DescribeInstancesOutput {
+ return &ec2.DescribeInstancesOutput{Reservations: []types.Reservation{{
+ OwnerId: aws.String(owner),
+ Instances: []types.Instance{{InstanceId: aws.String(instance), State: &types.InstanceState{Name: state}}},
+ }}}
+}
+
+func TestATerminatedInstanceOfThisBoxIsAFence(t *testing.T) {
+ o := FromDescribe(describe("111122223333", "i-0abc", types.InstanceStateNameTerminated), "i-0abc", "eu-west-1")
+ f, err := Verify("i-0abc", id, o, now)
+ if err != nil {
+ t.Fatal(err)
+ }
+ want := Fence{Box: "i-0abc", InstanceID: "i-0abc", Account: "111122223333", AWSRegion: "eu-west-1",
+ State: "terminated", ObservedAt: "2026-10-05T12:00:00Z", Via: "describe"}
+ if f != want {
+ t.Fatalf("fence = %+v\nwant %+v", f, want)
+ }
+}
+
+// Review Focus 4: every lookalike is refused.
+func TestLookalikesAreNotEvidence(t *testing.T) {
+ cases := map[string]Observation{
+ "stopped": FromDescribe(describe("111122223333", "i-0abc", types.InstanceStateNameStopped), "i-0abc", "eu-west-1"),
+ "stopping": FromDescribe(describe("111122223333", "i-0abc", types.InstanceStateNameStopping), "i-0abc", "eu-west-1"),
+ "shutting-down": FromDescribe(describe("111122223333", "i-0abc", types.InstanceStateNameShuttingDown), "i-0abc", "eu-west-1"),
+ "running": FromDescribe(describe("111122223333", "i-0abc", types.InstanceStateNameRunning), "i-0abc", "eu-west-1"),
+ "another account": FromDescribe(describe("999999999999", "i-0abc", types.InstanceStateNameTerminated), "i-0abc", "eu-west-1"),
+ "another region": FromDescribe(describe("111122223333", "i-0abc", types.InstanceStateNameTerminated), "i-0abc", "us-east-1"),
+ "another instance": FromDescribe(describe("111122223333", "i-0zzz", types.InstanceStateNameTerminated), "i-0zzz", "eu-west-1"),
+ "not in the reply": FromDescribe(describe("111122223333", "i-0zzz", types.InstanceStateNameTerminated), "i-0abc", "eu-west-1"),
+ "empty reply": FromDescribe(&ec2.DescribeInstancesOutput{}, "i-0abc", "eu-west-1"),
+ "nil reply": FromDescribe(nil, "i-0abc", "eu-west-1"),
+ "no state": FromDescribe(&ec2.DescribeInstancesOutput{Reservations: []types.Reservation{{OwnerId: aws.String("111122223333"), Instances: []types.Instance{{InstanceId: aws.String("i-0abc")}}}}}, "i-0abc", "eu-west-1"),
+ "unknown source": {Via: "cloudtrail", Found: true, InstanceID: "i-0abc", Account: "111122223333", Region: "eu-west-1", State: "terminated"},
+ "zero observation": {},
+ }
+ for name, o := range cases {
+ t.Run(name, func(t *testing.T) {
+ if f, err := Verify("i-0abc", id, o, now); !errors.Is(err, ErrNotEvidence) {
+ t.Fatalf("fence = %+v, err = %v; want ErrNotEvidence", f, err)
+ }
+ })
+ }
+}
+
+func TestAnUnverifiedIdentityIsNeverFenced(t *testing.T) {
+ o := FromDescribe(describe("", "", types.InstanceStateNameTerminated), "", "")
+ if _, err := Verify("i-0abc", boxid.Identity{}, o, now); !errors.Is(err, ErrNotEvidence) {
+ t.Fatalf("err = %v: an empty identity matched an empty observation", err)
+ }
+}
+
+const terminatedEvent = `{"version":"0","id":"ev-123","detail-type":"EC2 Instance State-change Notification","source":"aws.ec2","account":"111122223333","time":"2026-10-05T11:59:00Z","region":"eu-west-1","resources":["arn:aws:ec2:eu-west-1:111122223333:instance/i-0abc"],"detail":{"instance-id":"i-0abc","state":"terminated"}}`
+
+func TestATerminatedEventIsAFence(t *testing.T) {
+ o, err := FromEvent([]byte(terminatedEvent))
+ if err != nil {
+ t.Fatal(err)
+ }
+ f, err := Verify("i-0abc", id, o, now)
+ if err != nil {
+ t.Fatal(err)
+ }
+ if f.Via != "event" || f.EventID != "ev-123" || f.State != "terminated" {
+ t.Fatalf("fence = %+v", f)
+ }
+}
+
+func TestEventsThatAreNotEC2StateChangesAreRefused(t *testing.T) {
+ for name, body := range map[string]string{
+ "not JSON": `terminated`,
+ "another source": `{"id":"e","detail-type":"EC2 Instance State-change Notification","source":"my.app","account":"111122223333","region":"eu-west-1","detail":{"instance-id":"i-0abc","state":"terminated"}}`,
+ "another detail-type": `{"id":"e","detail-type":"EC2 Spot Instance Interruption Warning","source":"aws.ec2","account":"111122223333","region":"eu-west-1","detail":{"instance-id":"i-0abc","state":"terminated"}}`,
+ "no event id": `{"detail-type":"EC2 Instance State-change Notification","source":"aws.ec2","account":"111122223333","region":"eu-west-1","detail":{"instance-id":"i-0abc","state":"terminated"}}`,
+ } {
+ t.Run(name, func(t *testing.T) {
+ if _, err := FromEvent([]byte(body)); err == nil {
+ t.Fatal("accepted")
+ }
+ })
+ }
+}
+
+func TestAShuttingDownEventIsNotEvidence(t *testing.T) {
+ body := `{"id":"e","detail-type":"EC2 Instance State-change Notification","source":"aws.ec2","account":"111122223333","region":"eu-west-1","detail":{"instance-id":"i-0abc","state":"shutting-down"}}`
+ o, err := FromEvent([]byte(body))
+ if err != nil {
+ t.Fatal(err)
+ }
+ if _, err := Verify("i-0abc", id, o, now); !errors.Is(err, ErrNotEvidence) {
+ t.Fatalf("err = %v, want ErrNotEvidence", err)
+ }
+}
+
+func TestALegacyLabelFencesByItsBoundInstance(t *testing.T) {
+ o := FromDescribe(describe("111122223333", "i-0abc", types.InstanceStateNameTerminated), "i-0abc", "eu-west-1")
+ f, err := Verify("annie-welding", id, o, now)
+ if err != nil || f.Box != "annie-welding" || f.InstanceID != "i-0abc" {
+ t.Fatalf("fence = %+v, err = %v", f, err)
+ }
+}
+```
+
+- [ ] **Step 2: Run them to see them fail**
+
+Run: `go test ./internal/termfence/ -count=1`
+Expected: FAIL to build, `undefined: FromDescribe`.
+
+- [ ] **Step 3: Write `fence.go`**
+
+```go
+// Package termfence decides whether an observation proves that a box's
+// machine has been terminated (region HA spec 1 §5.9, invariant HA7).
+//
+// It is pure: something else calls EC2 or reads the event queue and hands
+// the result here. The rule is narrow on purpose. Only the state
+// "terminated" counts, only for the instance, account and region that the
+// box's AWS-signed record names, and silence is never evidence.
+package termfence
+
+import (
+ "encoding/json"
+ "errors"
+ "fmt"
+ "time"
+
+ "github.com/aws/aws-sdk-go-v2/aws"
+ "github.com/aws/aws-sdk-go-v2/service/ec2"
+
+ "github.com/carlosframework/platform/internal/boxid"
+)
+
+const (
+ ViaDescribe = "describe"
+ ViaEvent = "event"
+
+ stateTerminated = "terminated"
+ eventSource = "aws.ec2"
+ eventDetailType = "EC2 Instance State-change Notification"
+)
+
+// ErrNotEvidence wraps every reason an observation does not fence a box.
+var ErrNotEvidence = errors.New("termfence: not evidence of termination")
+
+// Observation is what one source said about one instance. Found is false
+// when the source said nothing about it: an empty reply, another instance.
+type Observation struct {
+ Via string
+ Found bool
+ InstanceID string
+ Account string
+ Region string
+ State string
+ EventID string
+}
+
+// Fence is the record a relocation journals in place of stop evidence.
+type Fence struct {
+ Box string `json:"box"`
+ InstanceID string `json:"instance_id"`
+ Account string `json:"account"`
+ AWSRegion string `json:"aws_region"`
+ State string `json:"state"`
+ ObservedAt string `json:"observed_at"`
+ Via string `json:"via"`
+ EventID string `json:"event_id,omitempty"`
+}
+
+// FromDescribe reads one instance out of a DescribeInstances reply. region
+// is the region the call was made in: the reply does not carry it. A reply
+// that does not mention instanceID yields Found false, which Verify
+// refuses: AWS forgets a terminated instance within about an hour, so
+// absence proves nothing.
+func FromDescribe(out *ec2.DescribeInstancesOutput, instanceID, region string) Observation {
+ o := Observation{Via: ViaDescribe, Region: region}
+ if out == nil || instanceID == "" {
+ return o
+ }
+ for _, r := range out.Reservations {
+ for _, inst := range r.Instances {
+ if aws.ToString(inst.InstanceId) != instanceID {
+ continue
+ }
+ o.Found, o.InstanceID, o.Account = true, instanceID, aws.ToString(r.OwnerId)
+ if inst.State != nil {
+ o.State = string(inst.State.Name)
+ }
+ return o
+ }
+ }
+ return o
+}
+
+// FromEvent reads an EventBridge "EC2 Instance State-change Notification".
+// It refuses anything else: the queue's policy admits only the rule that
+// forwards these, and this is the second check.
+func FromEvent(body []byte) (Observation, error) {
+ var ev struct {
+ ID string `json:"id"`
+ DetailType string `json:"detail-type"`
+ Source string `json:"source"`
+ Account string `json:"account"`
+ Region string `json:"region"`
+ Detail struct {
+ InstanceID string `json:"instance-id"`
+ State string `json:"state"`
+ } `json:"detail"`
+ }
+ if err := json.Unmarshal(body, &ev); err != nil {
+ return Observation{}, fmt.Errorf("termfence: event is not JSON: %w", err)
+ }
+ switch {
+ case ev.Source != eventSource:
+ return Observation{}, fmt.Errorf("termfence: event source %q is not %s", ev.Source, eventSource)
+ case ev.DetailType != eventDetailType:
+ return Observation{}, fmt.Errorf("termfence: event detail-type %q is not a state change", ev.DetailType)
+ case ev.ID == "":
+ return Observation{}, errors.New("termfence: event has no id")
+ }
+ return Observation{Via: ViaEvent, Found: ev.Detail.InstanceID != "", InstanceID: ev.Detail.InstanceID,
+ Account: ev.Account, Region: ev.Region, State: ev.Detail.State, EventID: ev.ID}, nil
+}
+
+func refuse(format string, a ...any) error {
+ return fmt.Errorf("%w: %s", ErrNotEvidence, fmt.Sprintf(format, a...))
+}
+
+// Verify returns a fence for box when o proves that the instance in id,
+// the box's verified identity, is terminated.
+func Verify(box string, id boxid.Identity, o Observation, now time.Time) (Fence, error) {
+ switch {
+ case box == "" || id.InstanceID == "" || id.AccountID == "" || id.Region == "":
+ return Fence{}, refuse("the box has no verified identity")
+ case o.Via != ViaDescribe && o.Via != ViaEvent:
+ return Fence{}, refuse("source %q is not one this binary accepts", o.Via)
+ case !o.Found:
+ return Fence{}, refuse("the source said nothing about %s; absence is not termination", id.InstanceID)
+ case o.InstanceID != id.InstanceID:
+ return Fence{}, refuse("observed %s, the box is %s", o.InstanceID, id.InstanceID)
+ case o.Account != id.AccountID:
+ return Fence{}, refuse("observed in account %q, the box is in %q", o.Account, id.AccountID)
+ case o.Region != id.Region:
+ return Fence{}, refuse("observed in region %q, the box is in %q", o.Region, id.Region)
+ case o.State != stateTerminated:
+ return Fence{}, refuse("state is %q; only %q counts", o.State, stateTerminated)
+ }
+ return Fence{Box: box, InstanceID: o.InstanceID, Account: o.Account, AWSRegion: o.Region, State: o.State,
+ ObservedAt: now.UTC().Format(time.RFC3339), Via: o.Via, EventID: o.EventID}, nil
+}
+```
+
+- [ ] **Step 4: Run the tests**
+
+Run: `gofmt -w internal/termfence && go test ./internal/termfence/ -count=1`
+Expected: PASS.
+
+- [ ] **Step 5: Commit**
+
+```bash
+git add internal/termfence
+git commit -m "termfence: only a terminated state, for the box's own verified instance, is a fence
+
+Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>"
+```
+
+---
+
+### Task 10: The installation-complete marker (`internal/activator/installed.go`)
+
+**Files:**
+- Create: `internal/activator/installed.go`
+- Test: `internal/activator/installed_test.go`
+
+**Interfaces:**
+- Produces:
+ ```go
+ type InstallMarker struct {
+ Incarnation string `json:"incarnation"`
+ Generation string `json:"generation"`
+ Stamp int64 `json:"stamp"`
+ DBMarker string `json:"db_marker"`
+ FilesSeq int64 `json:"files_seq"`
+ }
+ // Installation is what the claim says is current for a host.
+ type Installation struct {
+ Incarnation string
+ Generation string
+ Stamp int64
+ }
+ func InstallMarkerPath(dataDir, host string) string // <dataDir>/.installed/<InstanceID(host)>.json
+ func WriteInstallMarker(dataDir, host string, m InstallMarker) error
+ func ReadInstallMarker(dataDir, host string) (InstallMarker, bool, error)
+ func RemoveInstallMarker(dataDir, host string) error
+ func WarmCopyTrusted(want Installation, activeStamp int64, m InstallMarker, found bool, dbMarker string) bool
+ ```
+
+Spec §5.5 and HA3: authority (the box's `active_stamp`, from the claim) and evidence (this marker) are separate, and a box skips restore only when both match. This task builds the marker and the decision. It does **not** change `ExecBackend.Restore`: wiring it in needs the claim's fresh read and host guard, which is plan 1b. Until then `WarmCopyTrusted` has no caller outside its tests.
+
+The marker's file name uses `provision.InstanceID(host)`, as the nested layout does, so a long host name cannot exceed the file-name limit.
+
+- [ ] **Step 1: Write the failing tests**
+
+`internal/activator/installed_test.go`:
+
+```go
+package activator
+
+import (
+ "os"
+ "path/filepath"
+ "strings"
+ "testing"
+)
+
+func TestInstallMarkerRoundTrips(t *testing.T) {
+ dir := t.TempDir()
+ if _, found, err := ReadInstallMarker(dir, "shop.test"); found || err != nil {
+ t.Fatalf("before any write: found=%v err=%v", found, err)
+ }
+ m := InstallMarker{Incarnation: "inc1", Generation: "g-op7", Stamp: 41, DBMarker: "dbm", FilesSeq: 1882}
+ if err := WriteInstallMarker(dir, "shop.test", m); err != nil {
+ t.Fatal(err)
+ }
+ got, found, err := ReadInstallMarker(dir, "shop.test")
+ if err != nil || !found || got != m {
+ t.Fatalf("got %+v found=%v err=%v", got, found, err)
+ }
+ m.Stamp = 57
+ if err := WriteInstallMarker(dir, "shop.test", m); err != nil {
+ t.Fatal(err)
+ }
+ if got, _, _ := ReadInstallMarker(dir, "shop.test"); got.Stamp != 57 {
+ t.Fatalf("a second write did not replace the first: %+v", got)
+ }
+ if err := RemoveInstallMarker(dir, "shop.test"); err != nil {
+ t.Fatal(err)
+ }
+ if err := RemoveInstallMarker(dir, "shop.test"); err != nil {
+ t.Fatalf("removing an absent marker: %v", err)
+ }
+ if _, found, _ := ReadInstallMarker(dir, "shop.test"); found {
+ t.Fatal("marker still there")
+ }
+}
+
+func TestInstallMarkerLeavesNoTemporaryFile(t *testing.T) {
+ dir := t.TempDir()
+ if err := WriteInstallMarker(dir, "shop.test", InstallMarker{Stamp: 1}); err != nil {
+ t.Fatal(err)
+ }
+ entries, err := os.ReadDir(filepath.Join(dir, ".installed"))
+ if err != nil {
+ t.Fatal(err)
+ }
+ if len(entries) != 1 {
+ t.Fatalf("%d files in .installed, want 1", len(entries))
+ }
+}
+
+func TestInstallMarkerForALongHostHasAShortFileName(t *testing.T) {
+ host := strings.Repeat("a", 200) + ".example.com"
+ p := InstallMarkerPath(t.TempDir(), host)
+ if n := len(filepath.Base(p)); n > 64 {
+ t.Fatalf("file name is %d bytes", n)
+ }
+}
+
+func TestADamagedMarkerIsAnErrorNotAbsence(t *testing.T) {
+ dir := t.TempDir()
+ if err := WriteInstallMarker(dir, "shop.test", InstallMarker{Stamp: 1}); err != nil {
+ t.Fatal(err)
+ }
+ if err := os.WriteFile(InstallMarkerPath(dir, "shop.test"), []byte("{half"), 0o600); err != nil {
+ t.Fatal(err)
+ }
+ if _, found, err := ReadInstallMarker(dir, "shop.test"); err == nil || found {
+ t.Fatalf("found=%v err=%v: a damaged marker must be an error", found, err)
+ }
+}
+
+func TestWarmCopyTrusted(t *testing.T) {
+ want := Installation{Incarnation: "inc1", Generation: "g-op7", Stamp: 41}
+ good := InstallMarker{Incarnation: "inc1", Generation: "g-op7", Stamp: 41, DBMarker: "dbm", FilesSeq: 9}
+ if !WarmCopyTrusted(want, 41, good, true, "dbm") {
+ t.Fatal("a complete, current installation was not trusted")
+ }
+ for name, tc := range map[string]struct {
+ active int64
+ m InstallMarker
+ found bool
+ db string
+ }{
+ "no marker: crashed before the install finished": {41, InstallMarker{}, false, "dbm"},
+ "authority without evidence": {41, InstallMarker{Incarnation: "inc1", Generation: "g-op7", Stamp: 40, DBMarker: "dbm"}, true, "dbm"},
+ "evidence without authority": {40, good, true, "dbm"},
+ "no authority at all": {0, good, true, "dbm"},
+ "another generation": {41, InstallMarker{Incarnation: "inc1", Generation: "g-op6", Stamp: 41, DBMarker: "dbm"}, true, "dbm"},
+ "another incarnation": {41, InstallMarker{Incarnation: "inc0", Generation: "g-op7", Stamp: 41, DBMarker: "dbm"}, true, "dbm"},
+ "the database is not the one installed": {41, good, true, "other"},
+ "the database has no marker": {41, good, true, ""},
+ } {
+ t.Run(name, func(t *testing.T) {
+ if WarmCopyTrusted(want, tc.active, tc.m, tc.found, tc.db) {
+ t.Fatal("trusted")
+ }
+ })
+ }
+ if WarmCopyTrusted(Installation{}, 0, InstallMarker{}, true, "") {
+ t.Fatal("an empty claim matched an empty marker")
+ }
+}
+```
+
+- [ ] **Step 2: Run them to see them fail**
+
+Run: `go test ./internal/activator/ -run 'InstallMarker|DamagedMarker|WarmCopyTrusted' -count=1`
+Expected: FAIL to build, `undefined: ReadInstallMarker`.
+
+- [ ] **Step 3: Write `installed.go`**
+
+```go
+package activator
+
+import (
+ "encoding/json"
+ "errors"
+ "fmt"
+ "os"
+ "path/filepath"
+
+ "github.com/carlosframework/platform/internal/provision"
+)
+
+// InstallMarker is a box's own record that it finished installing one
+// generation of a host: the database restored and synced, the files
+// applied, the wake state imported (region HA spec 1 §5.5). It is written
+// last, so its presence vouches for all of them.
+//
+// It is evidence, not authority. Authority is the claim naming this box,
+// which the box holds as its active stamp. A box that crashed mid-install
+// has authority and no marker, and must restore.
+type InstallMarker struct {
+ Incarnation string `json:"incarnation"`
+ Generation string `json:"generation"`
+ Stamp int64 `json:"stamp"`
+ DBMarker string `json:"db_marker"`
+ FilesSeq int64 `json:"files_seq"`
+}
+
+// Installation is what the claim says is current for a host.
+type Installation struct {
+ Incarnation string
+ Generation string
+ Stamp int64
+}
+
+func InstallMarkerPath(dataDir, host string) string {
+ return filepath.Join(dataDir, ".installed", provision.InstanceID(host)+".json")
+}
+
+// WriteInstallMarker replaces the marker atomically and syncs both the
+// file and its directory, so a marker that is read back was fully written.
+func WriteInstallMarker(dataDir, host string, m InstallMarker) error {
+ path := InstallMarkerPath(dataDir, host)
+ dir := filepath.Dir(path)
+ if err := os.MkdirAll(dir, 0o700); err != nil {
+ return err
+ }
+ body, err := json.Marshal(m)
+ if err != nil {
+ return err
+ }
+ tmp, err := os.CreateTemp(dir, ".marker-*")
+ if err != nil {
+ return err
+ }
+ defer os.Remove(tmp.Name())
+ if _, err := tmp.Write(body); err != nil {
+ tmp.Close()
+ return err
+ }
+ if err := tmp.Sync(); err != nil {
+ tmp.Close()
+ return err
+ }
+ if err := tmp.Close(); err != nil {
+ return err
+ }
+ if err := os.Rename(tmp.Name(), path); err != nil {
+ return err
+ }
+ d, err := os.Open(dir)
+ if err != nil {
+ return err
+ }
+ defer d.Close()
+ return d.Sync()
+}
+
+// ReadInstallMarker returns the marker and whether there is one. A marker
+// that exists but cannot be parsed is an error, never "absent": the caller
+// restores either way, but it must not mistake damage for a clean slate.
+func ReadInstallMarker(dataDir, host string) (InstallMarker, bool, error) {
+ body, err := os.ReadFile(InstallMarkerPath(dataDir, host))
+ if errors.Is(err, os.ErrNotExist) {
+ return InstallMarker{}, false, nil
+ }
+ if err != nil {
+ return InstallMarker{}, false, err
+ }
+ var m InstallMarker
+ if err := json.Unmarshal(body, &m); err != nil {
+ return InstallMarker{}, false, fmt.Errorf("activator: install marker for %s: %w", host, err)
+ }
+ return m, true, nil
+}
+
+func RemoveInstallMarker(dataDir, host string) error {
+ err := os.Remove(InstallMarkerPath(dataDir, host))
+ if errors.Is(err, os.ErrNotExist) {
+ return nil
+ }
+ return err
+}
+
+// WarmCopyTrusted is invariant HA3: a box may skip restore and serve its
+// local copy only when it has authority for the current installation
+// (activeStamp equals the claim's stamp) AND evidence that it finished
+// installing exactly that one (the marker matches the claim, and the
+// database on disk is the one the marker was written for).
+//
+// dbMarker is the generation marker read from beside the local database.
+// Every empty value fails: nothing matches by being absent on both sides.
+func WarmCopyTrusted(want Installation, activeStamp int64, m InstallMarker, found bool, dbMarker string) bool {
+ if !found || want.Stamp == 0 || want.Incarnation == "" || want.Generation == "" || dbMarker == "" {
+ return false
+ }
+ return activeStamp == want.Stamp &&
+ m.Stamp == want.Stamp &&
+ m.Incarnation == want.Incarnation &&
+ m.Generation == want.Generation &&
+ m.DBMarker == dbMarker
+}
+```
+
+- [ ] **Step 4: Run the tests**
+
+Run: `go test ./internal/activator/ -run 'InstallMarker|DamagedMarker|WarmCopyTrusted' -count=1 && GOOS=windows go build ./internal/activator/`
+Expected: PASS and the cross-build succeeds.
+
+If the Windows build fails on the directory `Sync`, it is a compile error only if the code is wrong: `(*os.File).Sync` exists on every OS. A runtime failure syncing a directory on Windows does not matter here, because the agent runs on Linux.
+
+- [ ] **Step 5: Commit**
+
+```bash
+git add internal/activator/installed.go internal/activator/installed_test.go
+git commit -m "activator: an installation-complete marker, and the rule for trusting a local copy
+
+Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>"
+```
+
+---
+
+### Task 11: Infrastructure, the spec note, and the branch gate
+
+**Files:**
+- Modify: `infra/modules/region-host/main.tf` (instance at `:461`, role policy `host_buckets` at `:139`), `infra/modules/region-host/outputs.tf`, `infra/modules/region-host/README.md`
+- Modify: `infra/modules/deployment-buckets/main.tf`, `infra/modules/deployment-buckets/variables.tf`
+- Modify: `docs/superpowers/specs/2026-10-04-region-ha-box-placement-design.md`
+
+**Interfaces:**
+- Produces, from `region-host`: outputs `instance_id` (if not already there) and `termination_queue_url`. From `deployment-buckets`: variable `operator_role_arns` (list of strings, default `[]`).
+
+Nothing here is applied. The flagship's root lives in another repository and consumes these modules at a release tag; applying is a separate, gated operation. Two cautions from past incidents, both binding on this task:
+
+- A whole-root apply of the flagship restarts every box (user-data drift). This task changes no `user_data`.
+- A Deny on these buckets once covered `s3:DeleteObject` and broke every lease release for the full 15-minute TTL. Every Deny added here names the `control/boxes/` prefix and nothing wider.
+
+- [ ] **Step 1: Make the root volume die with the instance**
+
+In `infra/modules/region-host/main.tf`, in `aws_instance.host`'s `root_block_device`:
+
+```hcl
+ root_block_device {
+ volume_size = var.root_volume_size
+ volume_type = "gp3"
+ encrypted = true
+ # A terminated box must leave no disk behind: termination is the proof
+ # that its instances may run elsewhere (region HA spec 1 §5.9).
+ delete_on_termination = true
+ }
+```
+
+`true` is AWS's default for a root volume, so this changes nothing on a box created by this module; it pins the property so it cannot be lost.
+
+- [ ] **Step 2: Add the termination queue**
+
+Append to `infra/modules/region-host/main.tf`:
+
+```hcl
+# --- Termination evidence (region HA spec 1 §5.9) ---------------------------
+# EC2 tells EventBridge when an instance in this account and region reaches
+# "terminated"; the rule forwards that to a queue the region's boxes read.
+# Delivery is best-effort on AWS's side: the queue keeps what arrived, and a
+# box that finds nothing here still asks EC2 directly.
+
+data "aws_caller_identity" "current" {}
+
+resource "aws_sqs_queue" "terminations" {
+ name = "${var.iam_name}-terminations"
+ message_retention_seconds = 1209600 # 14 days, the maximum
+ sqs_managed_sse_enabled = true
+}
+
+resource "aws_cloudwatch_event_rule" "terminations" {
+ name = "${var.iam_name}-terminations"
+ description = "EC2 instances reaching terminated, for region HA failover."
+ event_pattern = jsonencode({
+ source = ["aws.ec2"]
+ "detail-type" = ["EC2 Instance State-change Notification"]
+ detail = { state = ["terminated"] }
+ })
+}
+
+resource "aws_cloudwatch_event_target" "terminations" {
+ rule = aws_cloudwatch_event_rule.terminations.name
+ arn = aws_sqs_queue.terminations.arn
+}
+
+# Only this rule may put a message on the queue. A message is evidence, so
+# who can write one matters as much as who can read one.
+resource "aws_sqs_queue_policy" "terminations" {
+ queue_url = aws_sqs_queue.terminations.id
+ policy = jsonencode({
+ Version = "2012-10-17"
+ Statement = [{
+ Sid = "OnlyTheTerminationRule"
+ Effect = "Allow"
+ Principal = { Service = "events.amazonaws.com" }
+ Action = "sqs:SendMessage"
+ Resource = aws_sqs_queue.terminations.arn
+ Condition = { ArnEquals = { "aws:SourceArn" = aws_cloudwatch_event_rule.terminations.arn } }
+ }]
+ })
+}
+
+resource "aws_iam_role_policy" "host_terminations" {
+ name = "${var.iam_name}-terminations"
+ role = aws_iam_role.host.id
+ policy = jsonencode({
+ Version = "2012-10-17"
+ Statement = [{
+ Effect = "Allow"
+ Action = ["sqs:ReceiveMessage", "sqs:DeleteMessage", "sqs:GetQueueAttributes"]
+ Resource = aws_sqs_queue.terminations.arn
+ }]
+ })
+}
+```
+
+First run `grep -n 'data "aws_caller_identity"' infra/modules/region-host/*.tf`. If the module already declares one, delete the declaration above (it is not otherwise used here) rather than adding a second.
+
+The box role already has `ec2:DescribeInstances` on `*` for the infra inventory (`main.tf`, the read-only statement in `host_buckets`). Spec §6 asks for an `ec2:Region` condition on it. Do **not** narrow the existing statement in this task: the inventory reads other regions. Record it as an open item in the README text below; plan 1b decides it with the placer.
+
+- [ ] **Step 3: Add the outputs**
+
+Run `grep -n 'output "instance_id"' infra/modules/region-host/outputs.tf`. Add whichever of these is missing to `outputs.tf`:
+
+```hcl
+output "instance_id" {
+ description = "The box's EC2 instance id. `carlos ops box bind` checks a legacy box's record against this."
+ value = aws_instance.host.id
+}
+
+output "termination_queue_url" {
+ description = "The queue that receives this region's EC2 terminated events."
+ value = aws_sqs_queue.terminations.id
+}
+```
+
+- [ ] **Step 4: Protect box records in the deployment bucket**
+
+Append to `infra/modules/deployment-buckets/variables.tf`:
+
+```hcl
+variable "operator_role_arns" {
+ description = "IAM role ARNs of human operators. Only these may write a box bind (control/boxes/*.bind.json). Empty means nobody can: the safe default until operators are named."
+ type = list(string)
+ default = []
+}
+```
+
+Append to `infra/modules/deployment-buckets/main.tf`:
+
+```hcl
+# --- Box records (region HA spec 1 §5.1) -----------------------------------
+# control/boxes/<id>.json binds a box's name to one AWS instance, and
+# control/boxes/<id>.bind.json is an operator's word for a legacy label.
+# Both are written once and never changed. Storage enforces that, so no
+# bug or stolen box credential can rebind a name to another machine.
+#
+# EVERY statement here names control/boxes/ and nothing wider. A broad Deny
+# on this bucket has broken lease releases before.
+
+resource "aws_s3_bucket_policy" "deployment_box_records" {
+ bucket = aws_s3_bucket.deployment.id
+ policy = jsonencode({
+ Version = "2012-10-17"
+ Statement = [
+ {
+ # A record or bind may be created, never replaced. The qualifier
+ # leaves multipart preparation alone and still covers its completion.
+ Sid = "BoxRecordsAreCreateOnly"
+ Effect = "Deny"
+ Principal = "*"
+ Action = "s3:PutObject"
+ Resource = "${aws_s3_bucket.deployment.arn}/control/boxes/*"
+ Condition = {
+ Null = { "s3:if-none-match" = "true" }
+ Bool = { "s3:ObjectCreationOperation" = "true" }
+ }
+ },
+ {
+ Sid = "BoxRecordsAreNeverDeleted"
+ Effect = "Deny"
+ Principal = "*"
+ Action = ["s3:DeleteObject", "s3:DeleteObjectVersion"]
+ Resource = "${aws_s3_bucket.deployment.arn}/control/boxes/*"
+ },
+ {
+ # With no operators named, the list is one ARN nobody has, so the
+ # Deny covers everyone.
+ Sid = "OnlyOperatorsWriteBinds"
+ Effect = "Deny"
+ Principal = "*"
+ Action = "s3:PutObject"
+ Resource = "${aws_s3_bucket.deployment.arn}/control/boxes/*.bind.json"
+ Condition = {
+ ArnNotLike = {
+ "aws:PrincipalArn" = length(var.operator_role_arns) > 0 ? var.operator_role_arns : ["arn:aws:iam::000000000000:role/nobody"]
+ }
+ }
+ },
+ ]
+ })
+}
+```
+
+Before adding it, run `grep -rn "aws_s3_bucket_policy" infra/modules/`. The report found no bucket policy in this module. If one exists for the deployment bucket elsewhere, a bucket takes only one policy: merge these three statements into it instead of creating a second resource.
+
+- [ ] **Step 5: Validate both modules**
+
+Run:
+
+```bash
+(cd infra/modules/region-host && tofu fmt -check && tofu init -backend=false -input=false >/dev/null && tofu validate)
+(cd infra/modules/deployment-buckets && tofu fmt -check && tofu init -backend=false -input=false >/dev/null && tofu validate)
+```
+
+Expected: `Success! The configuration is valid.` twice. `tofu init` downloads the AWS provider: it needs network access to `registry.opentofu.org` and `github.com`, and writes a `.terraform` directory in each module. Delete both `.terraform` directories and any `.terraform.lock.hcl` this created before committing (`git status` must show only the `.tf` and `.md` changes). If `tofu fmt -check` lists a file, run `tofu fmt` on it.
+
+- [ ] **Step 6: Document the module changes**
+
+Append to `infra/modules/region-host/README.md`:
+
+```markdown
+## Termination evidence
+
+The module creates a queue, `<iam_name>-terminations`, that receives an
+event whenever an EC2 instance in this account and region reaches
+`terminated`. Region HA reads it to prove a dead box is gone before its
+instances run elsewhere. Only the module's own EventBridge rule can write
+to the queue, and the box role can only read and delete from it.
+
+The root volume is deleted when the instance is terminated. Region HA
+depends on that: a terminated box must leave no disk behind.
+
+Open item for region HA plan 1b: the box role's `ec2:DescribeInstances`
+is unconditioned, because the infrastructure inventory reads every
+region. The HA spec asks for an `ec2:Region` condition on the call the
+placer makes. Decide there whether that is a second, narrower statement.
+```
+
+- [ ] **Step 7: Record this plan's three departures in the spec**
+
+Append to `docs/superpowers/specs/2026-10-04-region-ha-box-placement-design.md`:
+
+```markdown
+
+## 18. Notes from plan 1a (2026-10-05)
+
+Plan `docs/superpowers/plans/2026-10-05-region-ha-1a-foundations.md` builds
+the parts of this spec that need no claim. It keeps every property above
+and changes three mechanisms:
+
+1. **Exact content by hash, not by S3 version id (§5.6, §5.1).** The store
+ abstraction has no version-id read and the box role has no
+ `s3:GetObjectVersion`. Objects are immutable, their key contains their
+ SHA-256, the manifest records it, and a reader verifies it. A bind
+ names its record by `record_sha256` in place of `record_version_id`.
+2. **The identity signature (§5.1).** The record carries IMDS's
+ base64 RSA-SHA256 signature (`/latest/dynamic/instance-identity/signature`),
+ verified with the regional certificate, in place of the PKCS7 form.
+ Same document, same signer, no new dependency.
+3. **`snapshot` is a periodic full publish, not a sealed archive (§5.6).**
+ `hostfiles.SyncDir` stages every difference between the directory and
+ the manifest. Run on the snapshot interval it gives the `snapshot`
+ setting; run every ten minutes it is the continuous setting's repair
+ scan; run before stop evidence it is the release flush. The manifest
+ therefore has no `base` or `base_seq`, and restore reads one manifest
+ and the objects it names.
+
+Two limits to lift in plan 1b: a file over 64 MiB is recorded in the
+manifest as skipped and is not published (the store takes a whole body in
+memory), and the third termination-evidence source (Auto Scaling activity
+history) is not built until spec 4 verifies it.
+```
+
+- [ ] **Step 8: Run the whole gate**
+
+Run: `go build ./... && make test`
+Expected: `gofmt -l .: clean` and every package `ok`. `make test` takes up to thirty minutes. Then the cross-builds CI runs: `GOOS=darwin go build ./... && GOOS=windows go build ./... && GOOS=linux GOARCH=arm64 go build ./...`. Expected: all three succeed.
+
+If a package unrelated to this plan fails, do not fix it here and do not skip it: run that package alone, confirm it also fails on `origin/main`, and report both results.
+
+- [ ] **Step 9: Commit**
+
+```bash
+git add infra/modules/region-host infra/modules/deployment-buckets docs/superpowers/specs/2026-10-04-region-ha-box-placement-design.md
+git commit -m "infra: a termination queue per region, and box records that storage will not let anyone rewrite
+
+Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>"
+```
+
+- [ ] **Step 10: Push and read CI**
+
+```bash
+git push origin design-region-ha
+amadan ci status carlos/platform -branch design-region-ha
+```
+
+Expected: exit 0 (pass). Exit 2 is pending: wait and read it again. Exit 1 is a failure: read the log, fix it, and do not ask for the branch to be merged. Merging is Paul's call, through `amadan branch merge`, never a push to `main`.
+
+---
+
+## What plan 1b picks up
+
+Written when `internal/claim` and `internal/hostguard` are on main, after re-reading spec 1 against the claim's then-current revision:
+
+1. Create names a box: `chooseBox(eligibleBoxes(…))` in the claim's create step (§5.2), with `verified` built from `boxid.Load` and `Verifier.Verify`.
+2. Reconcile's three row cases and the ten-minute sweep (§5.3).
+3. The edges-file pass sets `edgeHost.Stamp` from the claim, and publishes `home` only for the executor (§5.4).
+4. `ExecBackend.Restore` consults `WarmCopyTrusted` under the host guard (§5.5).
+5. `files_durability` and `files_snapshot_interval` on the app manifest, projected onto the route as Seed is, and the agent starting `hostfiles.Run` or a periodic `SyncDir` per host (§5.6), with the files directory's location and skip list for both layouts.
+6. The agent publishing its box record at start, and `carlos ops box bind` (§5.1), with `docs/site/cli.md`.
+7. The `relocating` operation, the release sequence and the pinned recovery set (§5.8).
+8. The multi-box gate and the HA floor (§8).
+9. A streaming put, lifting the 64 MiB limit; the data bucket's conditional-write policy for the files prefix (§6).
diff --git a/docs/superpowers/plans/CODEX-PLAN-REGION-HA-1A-BRIEF-1.md b/docs/superpowers/plans/CODEX-PLAN-REGION-HA-1A-BRIEF-1.md
new file mode 100644
index 000000000..c0b3d649d
--- /dev/null
+++ b/docs/superpowers/plans/CODEX-PLAN-REGION-HA-1A-BRIEF-1.md
@@ -0,0 +1,50 @@
+# Review brief: region HA plan 1a (foundations), round 1
+
+You are an independent second reader. Review the implementation plan
+`docs/superpowers/plans/2026-10-05-region-ha-1a-foundations.md` at the
+commit named in the prompt, against the tree at the working directory
+(Go, module `github.com/carlosframework/platform`) and the spec
+`docs/superpowers/specs/2026-10-04-region-ha-box-placement-design.md`.
+
+## Fixed: do not re-argue
+
+- The spec's rulings H1 to H9.
+- The split into 1a (no claim dependency, this plan) and 1b (after
+ `internal/claim` lands). Do not ask 1a to wire anything into create,
+ reconcile, the activator's restore path or relocation.
+- The three departures the plan states at its top (content hash in place
+ of S3 version ids; the base64 RSA identity signature; `snapshot` as a
+ periodic full publish). Report only if one fails to keep the spec's
+ property.
+
+## What to check
+
+1. **Does the code compile and do the tests pass as written?** Read each
+ task's code against the real tree: every identifier, signature,
+ import and file path it uses from existing code (`release.Store`,
+ `registry.Timer`, `registry.Route`, `edgesFile`, `foldHomes`,
+ `edgesFilePlacementView`, `views`, `markStale`, `testKeyFP`,
+ `provision.InstanceID`, the ec2 SDK types, `unix.Inotify*`). Name the
+ exact line that would not compile or the test that would fail, and why.
+2. **Does each test actually pin the behaviour its name claims?** A test
+ that passes for the wrong reason, or passes before the change.
+3. **Correctness of the mechanisms**: the manifest's ordering under
+ concurrent or late writes; `readStable` and symlink races; the inotify
+ watcher (event parsing, lost events, new directories, close);
+ `dialBeats` against every pre-existing home-map rule; `Verify` in
+ `boxid` and `termfence`; the bucket policy statements (would any
+ statement deny something that works today, especially lease releases
+ or existing writers under `control/`?); the EventBridge and SQS tofu.
+4. **Additivity**: anything that changes behaviour on a one-box region
+ or breaks a reader of an existing wire shape.
+5. **The Review Focus list**: is each of the five pinned by a test in
+ the task that owns it?
+6. **Task boundaries**: a task that cannot be built and tested alone, in
+ order, with only the earlier tasks present.
+
+## Output
+
+Findings ranked P1 (would not compile, a test fails, or unsafe), P2
+(wrong or missing), P3 (clarity). Each: task and step, the problem, the
+exact evidence (file:line in the tree, or the plan's own line), and a
+concrete fix. End with a verdict: ready to build, or not ready.