carlos / platform Public

Clone
git clone https://amadan.net/carlos/platform

Plain git — no account needed to clone.

Download

Download this file

Region HA 1b: Evidence and Claim Readers Implementation Plan

> For agentic workers: REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (- [ ]) syntax for tracking.

Goal: Build every part of region HA spec 1 rev 5.4 that needs none of the claim and moves changes L1 to L20: enrollment evidence from internal/boxid, §19's corrections to plan 1a, the agent publishing its identity record, the termination fence bound to the roster's cloud, volume records and the discharge check, the edges file following the claim's executor, and root's refusal to update a box that takes part in a claim operation. All of it dormant behind CARLOS_REGION_HA.

Architecture: Eight tasks. Four are pure libraries that read and decide but write nothing a decision rests on (boxid.Enroll, termfence.Verify, boxvolumes, chooseBox/eligibleBoxes). Three change what a box does, each only with CARLOS_REGION_HA on: the agent publishes its identity record once, the edges file publishes a relocatable host as home only on the box its claim's op.exec names, and root refuses an executable update while <DataDir>/.hostexec/ holds anything. One task is storage policy. Nothing adds a claim op kind, changes hostclaim's admission rules, or changes moves; claims are only read, through hostclaim.Read.

Tech Stack: Go 1.26, module github.com/carlosframework/platform; release.Store (create-only and ETag compare-and-swap puts); aws-sdk-go-v2/service/ec2 and smithy-go (already direct dependencies); SQLite registry; OpenTofu modules under infra/modules/.

Spec: docs/superpowers/specs/2026-10-04-region-ha-box-placement-design.md, rev 5.4 (approved by Paul 2026-10-06). Executors read §0 (rulings H1 to H19), §2, §5.1, §5.4, §5.9, §5.10, §6, §8, §9 (the L-table), §18 and §19 first. Plan 1a (docs/superpowers/plans/2026-10-05-region-ha-1a-foundations.md, merged as aadb860) built the pieces this plan corrects; read its "After the build" section.

Controller rulings, 2026-10-06

Made overnight by the controller on Paul's behalf, on the six points the first draft of this plan left open. Each names what it costs if it is wrong.

  1. Discharge also needs a recent last record. The newest running-time volume record must be no more than 2 hours older than the termination observation (boxvolumes.LastRecordWindow); otherwise the dead box's park-local chores stay open for the operator path. Pinned by TestDischargeNeedsARecentLastRecord (Task 5). Cost if wrong: a recorder outage longer than two hours before a death sends that box's cleanup to an operator; nothing about recovery waits on it (H19).
  2. §5.4 is followed literally. The edges file withholds a relocatable host whose claim's op.exec names another box or none (Task 6). Cost if wrong: with the flag on in a one-box region, a host whose claim briefly names no executor (a move's cut, once moves exist) is missing from remote edges for that window and answers there as unknown; moves are not built, so nothing is affected today.
  3. Root's refusal stays behind CARLOS_REGION_HA in 1b (Task 7). It becomes unconditional with claim D6 (see "Waits for the claim"). Cost if wrong: until D6 lands nothing writes .hostexec/, so no update is wrongly allowed today; the risk is forgetting to lift the flag when D6 lands, which the waits table records.
  4. The stricter enrollment refusal stands: one instance is never enrolled under two references (Task 1). Cost if wrong: re-enrolling a machine under a new label needs the old entry handled by an operator first.
  5. The bind statement and operator_role_arns stay in deployment-buckets (Task 8). Retiring both is a follow-up once no consuming root sets the variable. Cost if wrong: one harmless Deny on a prefix nothing writes.
  6. No README change in this plan. The region-host README paragraph needs Paul's copy review, which waits for the morning; its proposed text is under "Copy waiting for Paul" at the end, and Task 8 does not touch the README. Cost if wrong: the module README keeps saying region HA depends on root volumes being deleted, which H19 made untrue, until the copy lands.

Why this is plan 1b, and what it leaves out

The spec asks the host lifecycle claim and move-any-app sessions for twenty changes (§9, L1 to L20). Neither session has accepted them. So this plan builds only what needs none of them, and stops at every point where the next step would write a claim, add an op kind, depend on the roster (control/host-actors.json, claim D7, not on main), or change moves. The section "Waits for the claim", after the tasks, maps everything left to its L-number.

Three of the candidates the brief named were checked against the spec and main and left out:

  • Reading the termination queue. The spec gives the placer loop that reads the queue to spec 3 (§12), and an event read is only useful to a writer of fence records (L2 to L4). A reader that cannot write a fence could only leave messages on the queue or delete evidence. It also needs service/sqs, which is not a dependency.
  • The Auto Scaling membership confirmation (H16, H17). It reads the target region's group ARN from the roster's regions entry (claim P5-D2's extension) and lists the roster's enrollments in that region. There is no roster and no Auto Scaling group yet (spec 4), and the console may accept the record only once L19 is accepted. It needs service/autoscaling too.
  • IAM for those two. Nothing in this plan calls SQS or Auto Scaling, and this repository grants nothing for later. ec2:DescribeInstances, the one call this plan's code makes, is already allowed in every region by the inventory statement (infra/modules/region-host/main.tf:213): a second, narrower Allow could not take that away, and the code pins every describe to the box's enrolled region instead.

Global Constraints

  • Dormant and inert. Nothing in this plan may change behaviour on a box where CARLOS_REGION_HA is unset or 0, or on a one-box region (spec §4, §8 step 2: "Ship §5 dormant behind CARLOS_REGION_HA … With one box per region, every rule is a no-op"). The flag is read by one helper, regionHAEnabled (Task 3).
  • Scope (the brief). Do not add a claim op kind, change hostclaim's admission rules, or change moves. Read claims only through hostclaim.Read. Use hostguard, stopevidence and genselect as they are.
  • H19, verbatim: "The failover fence rests on the terminated read alone; disk evidence only decides discharge of local chores." Nothing in this plan lets a volume record gate execution, admission, relocation, recovery, row minting or a box's start (spec §8, B4-1).
  • Absence is never evidence (spec §5.9): "stopped and stopping do not count … A missing instance, InvalidInstanceID.NotFound, an API error, a partial reply, or a lookup in another account or region is never evidence." Only state terminated fences.
  • Box references are labels (#212 option B, spec §2): one DNS label (controlname.ValidLabel), so never containing . or /. The instance id is private and lives in the roster entry's cloud.
  • Wire-shape changes are additive (AGENTS.md). This plan adds no edges-file field (stamp exists since plan 1a). Records it defines are new objects under new prefixes.
  • Do not add a method to release.Store (eleven implementations). Put(ctx, key, body, "") is create-only; a lost race is release.ErrCASConflict.
  • No new Go module dependencies. service/ec2, aws/middleware and smithy-go are already direct. Do not add service/sqs or service/autoscaling.
  • AWS is behind interfaces. Every EC2 call goes through termfence.EC2; tests use fakes and never call AWS. IMDS is reached through an *http.Client whose transport tests replace, so no test opens a socket.
  • Linux-only syscalls live in *_linux.go with a !linux stub. This plan adds none: systemparticipation.go uses os only. CI cross-compiles darwin, windows and linux/arm64.
  • Tests use release.NewDirStore(t.TempDir()); inject failures with a struct embedding release.Store that overrides one method.
  • Copy. No console, CLI or docs copy is added (controller ruling 6). Log lines are operator-facing and follow the surrounding agent: … / system apply: … style. If any task finds it needs user-facing copy, stop: it waits for Paul's copy review.
  • Commit messages: lowercase area: prefix and a sentence stating the behaviour, then a blank line and Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>.
  • Go environment. Run Go with GOCACHE=$TMPDIR/gocache GOFLAGS="-mod=mod -buildvcs=false" (every command below assumes both are exported). Tests that open Unix sockets fail inside the Claude sandbox with socket: operation not permitted; on this branch's base those are, in internal/host, TestSleepingInstanceRearmsItsNextDeliveryBeforeReturning, TestScheduledTickWakesASleepingInstanceAndItHibernatesAgain, TestDeliveryReusesTheConnectionForASocket, TestArmScheduleFiresOnceTimersWithNoDeployStore, TestProxyCacheIsStableAndBounded, TestDeliveryDoesNotHoldTheInstanceAwake, TestDeliveryStillCarriesScheduleAtHeader and TestSocketServingProbe; in internal/activator, the TestExecBackendStart…, TestStartUnit…, TestUnitProc…, TestSeedUnit…, TestStop… and TestPrepareRemovesAStaleSocketFile tests; and internal/edge as a whole. That is the sandbox, not the change: run those packages with the sandbox off, or rely on CI. Scratch files go under $TMPDIR. Never use git stash, reset --hard, checkout -- or clean.
  • OpenTofu. tofu test starts the AWS provider over a local socket, so it needs the sandbox off, and tofu init needs network access to registry.opentofu.org and GitHub.
  • Gate before each commit: go build ./... && go vet ./<pkgs>/ && gofmt -l <dirs> && go test ./<pkgs>/ -run '<tests>' -count=1, then GOOS=darwin go build ./... && GOOS=windows go build ./.... Full gate before the branch lands: go build ./... && make test, and the three cross-builds.
  • Exact values. Claim read cache on the edges file 10 minutes (H9's sweep); at most 256 claim reads per edges pass, with an extra pass every 30 seconds while reads are due; a volume record at least once an hour when nothing changes; the first volume record must be taken within 1 hour of enrollment; the newest within 2 hours before the termination observation; capability region-ha-1; fence state terminated only; root's marker /opt/carlos/system/guard/update-begun.json; controller records <DataDir>/.hostexec/; volume records control/box-volumes/<enrollment>/<observed_at>.json.
  • Work on a branch cut from origin/main (suggested name region-ha-1b). Merging is Paul's call, through amadan branch merge.
  • This plan's code was dry-run on 2026-10-06: every task was applied in order to a clean copy of origin/main (bf813540), built, vetted, cross-built for darwin, windows and linux/arm64, and its tests run; the five Review Focus tests and five others were each watched to fail under a mutation of the code they pin; Task 8's tofu test was run against the AWS provider's mock and watched to fail against main's policy. After Astra's plan round 1 the plan was dry-run again from its own text (every task applied in order to a clean tree by a script that reads the plan), and each round 1 fix's test was watched to fail with its mechanism removed, including five mutations of the bucket policy. Then the code was removed. What the dry run could not do is listed at the end.

Review Focus

  1. A copied identity record. Box A's own record, validly signed, written under box B's reference with its box field rewritten. The box field is not signed, so the signature still verifies; enrollment must refuse because the instance is already enrolled as A (Task 1, TestEnrollRefusesAnInstanceAlreadyEnrolledUnderAnotherReference).
  2. A claim that cannot be read. A store error on a claim read must neither take a served host off this box's edges file nor publish a host this box was told not to; a host never read yet publishes as it did before claims (Task 6, TestEdgesFileKeepsTheLastDecisionWhenAClaimCannotBeRead).
  3. A terminated describe offered as a volume observation. Its BlockDeviceMappings is empty, so recorded it would list every volume as removed. It must not be recordable; and a removal that no record listed must still keep the chores open (Task 5, TestObserveRefusesWhatIsNotTheInstancesMapping, TestDischargeSeesARemovalNoRecordListed).
  4. A controller directory root cannot read. A temporary file, a record that does not parse, a symlink or a file in place of .hostexec, or a directory root cannot list: each must refuse the update, never pass it (Task 7, TestApplyRefusesWhenItCannotTellWhetherThisBoxTakesPart, TestApplyRefusesWhenItCannotListTheControllerDirectory).
  5. A fence read from another region. A worker enrolled in another AWS region reading a perfect terminated observation must not fence the box (spec §6: the fence refuses any observation not made in the box's own region) (Task 4, TestAWorkerInAnotherRegionNeverFences).

File Structure

| Path | Responsibility | |---|---| | internal/boxid/record.go | The identity record, ValidBox, Key, create-only Publish, Load (exact bytes) | | internal/boxid/verify.go | Cloud, Verifier.Enroll: the record as enrollment evidence | | internal/console/boxchoice.go | eligibleBoxes over the roster's proven cloud; chooseBox with P5-D2's tie-break | | internal/activator/installed.go, activator.go | WarmCopyTrusted removed; comments point at claim R6 | | internal/host/boxrecord.go | regionHAEnabled; reading IMDS; publishing this box's record at start | | internal/termfence/fence.go | Observations, DescribeInstance, Verify bound to the roster entry, §5.9's record shape | | internal/boxvolumes/volumes.go | Volume records: observe, decide, write create-only, list, decide discharge | | internal/host/edgesclaims.go | The edges file's claim cache and the home/stamp decision | | internal/host/edgesfile.go | Wiring the claim cache into the publisher | | internal/host/systemparticipation.go | Root's update-begun marker and .hostexec check | | internal/host/systemapply.go | Wiring them into the apply | | infra/modules/deployment-buckets/main.tf, tests/evidence.tftest.hcl | Create-only, never-deleted evidence prefixes |

---

Task 1: Enrollment evidence (internal/boxid)

Spec §5.1 and §19: control/boxes/<ref>.json is no longer an authority. It is the evidence enrollment reads, and enrollment records the proven account, region and instance id in the roster entry's cloud. The operator bind is superseded (its tofu check moves into enrollment), and box references are labels.

Files:

  • Modify (replace whole file): internal/boxid/record.go, internal/boxid/verify.go
  • Test (replace whole file): internal/boxid/boxid_test.go

Interfaces:

  • Consumes: release.Store, controlname.ValidLabel.
  • Produces: ``go package boxid const Prefix = "control/boxes/" func ValidBox(ref string) bool func Key(box string) string type Record struct{ V int; Box, IdentityDocument, IdentitySignature, WrittenAt string } // json as before func Publish(ctx context.Context, s release.Store, rec Record) error // create-only; same bytes again is success func Load(ctx context.Context, s release.Store, box string) ([]byte, error) // exact bytes; os.ErrNotExist-wrapped when absent var ErrUnverified error type Identity struct{ AccountID, Region, InstanceID string } // kept only until Task 4 type Cloud struct{ Provider, Account, Region, InstanceID string } // json: provider, account, region, instance_id func (c Cloud) Complete() bool type Verifier struct{ Account string; Certs map[string]*x509.Certificate } type Enrolled struct{ Box string; Cloud Cloud } type Expect struct{ Box, AWSRegion, TofuInstanceID string; Roster []Enrolled } type Evidence struct{ Box string; Cloud Cloud; RecordSHA256 string } func (v Verifier) Enroll(raw []byte, e Expect) (Evidence, error) func EmbeddedCerts() (map[string]*x509.Certificate, error) // unchanged ` Removed: BindKey, Bind, Verifier.Verify, and Load's record and bind returns. Nothing outside internal/boxid` used them.

Why Enroll takes raw bytes. The hash enrollment records must be of the bytes it verified. Decoding inside Enroll makes that one value, where a Record and a separately supplied []byte could disagree.

Why the roster check runs both ways. The spec asks enrollment to "refuse a reference already enrolled to a different instance". The record's box field is not covered by AWS's signature, so a box can copy another box's record to its own key, rewrite box, and still verify. Enroll therefore also refuses an instance the roster already holds under another reference. Entries in every state count: a retired reference never names a working box again, and a terminated instance never returns.

  • [ ] Step 1: Write the failing tests

Replace internal/boxid/boxid_test.go with:

package boxid

import (
	"context"
	"crypto"
	"crypto/rand"
	"crypto/rsa"
	"crypto/sha256"
	"crypto/x509"
	"crypto/x509/pkix"
	"encoding/base64"
	"encoding/hex"
	"encoding/json"
	"errors"
	"math/big"
	"os"
	"testing"
	"time"

	"github.com/carlosframework/platform/internal/release"
)

// signer stands in for AWS: one key and certificate per test.
type signer struct {
	key  *rsa.PrivateKey
	cert *x509.Certificate
}

func newSigner(t *testing.T) signer {
	t.Helper()
	key, err := rsa.GenerateKey(rand.Reader, 2048)
	if err != nil {
		t.Fatal(err)
	}
	tmpl := &x509.Certificate{SerialNumber: big.NewInt(1), Subject: pkix.Name{CommonName: "test"},
		NotBefore: time.Now().Add(-time.Hour), NotAfter: time.Now().Add(time.Hour)}
	der, err := x509.CreateCertificate(rand.Reader, tmpl, tmpl, &key.PublicKey, key)
	if err != nil {
		t.Fatal(err)
	}
	cert, err := x509.ParseCertificate(der)
	if err != nil {
		t.Fatal(err)
	}
	return signer{key, cert}
}

func (s signer) record(t *testing.T, box, account, region, instance string) (Record, []byte) {
	t.Helper()
	doc, _ := json.Marshal(map[string]string{"accountId": account, "region": region, "instanceId": instance})
	sum := sha256.Sum256(doc)
	sig, err := rsa.SignPKCS1v15(rand.Reader, s.key, crypto.SHA256, sum[:])
	if err != nil {
		t.Fatal(err)
	}
	rec := Record{V: 1, Box: box, IdentityDocument: base64.StdEncoding.EncodeToString(doc),
		IdentitySignature: base64.StdEncoding.EncodeToString(sig), WrittenAt: "2026-10-06T00:00:00Z"}
	raw, _ := json.Marshal(rec)
	return rec, raw
}

// verifier certifies eu-west-1 and eu-west-2 with the same key, so a test
// can show that a supported but wrong region is still refused.
func (s signer) verifier() Verifier {
	return Verifier{Account: "111122223333", Certs: map[string]*x509.Certificate{"eu-west-1": s.cert, "eu-west-2": s.cert}}
}

func expect(box string) Expect { return Expect{Box: box, AWSRegion: "eu-west-1"} }

func TestEnrollProvesTheCloudOfALabel(t *testing.T) {
	s := newSigner(t)
	_, raw := s.record(t, "annie-welding", "111122223333", "eu-west-1", "i-0abc")
	ev, err := s.verifier().Enroll(raw, expect("annie-welding"))
	if err != nil {
		t.Fatal(err)
	}
	sum := sha256.Sum256(raw)
	want := Evidence{Box: "annie-welding", RecordSHA256: hex.EncodeToString(sum[:]),
		Cloud: Cloud{Provider: "aws", Account: "111122223333", Region: "eu-west-1", InstanceID: "i-0abc"}}
	if ev != want {
		t.Fatalf("evidence = %+v\nwant       %+v", ev, want)
	}
	if !ev.Cloud.Complete() {
		t.Fatal("a proven cloud is not complete")
	}
}

func TestEnrollRefusals(t *testing.T) {
	s, other := newSigner(t), newSigner(t)
	good := func() []byte {
		_, raw := s.record(t, "annie-welding", "111122223333", "eu-west-1", "i-0abc")
		return raw
	}
	cases := map[string]struct {
		raw []byte
		e   Expect
	}{
		"wrong account": {func() []byte {
			_, raw := s.record(t, "annie-welding", "999999999999", "eu-west-1", "i-0abc")
			return raw
		}(), expect("annie-welding")},
		"region with no certificate": {func() []byte {
			_, raw := s.record(t, "annie-welding", "111122223333", "ap-south-1", "i-0abc")
			return raw
		}(), Expect{Box: "annie-welding", AWSRegion: "ap-south-1"}},
		"a supported region that is not the placement's": {func() []byte {
			_, raw := s.record(t, "annie-welding", "111122223333", "eu-west-2", "i-0abc")
			return raw
		}(), expect("annie-welding")},
		"signed by someone else": {func() []byte {
			_, raw := other.record(t, "annie-welding", "111122223333", "eu-west-1", "i-0abc")
			return raw
		}(), expect("annie-welding")},
		"document altered after signing": {func() []byte {
			r, _ := s.record(t, "annie-welding", "111122223333", "eu-west-1", "i-0abc")
			doc, _ := json.Marshal(map[string]string{"accountId": "111122223333", "region": "eu-west-1", "instanceId": "i-0zzz"})
			r.IdentityDocument = base64.StdEncoding.EncodeToString(doc)
			raw, _ := json.Marshal(r)
			return raw
		}(), expect("annie-welding")},
		"record names another box":    {good(), expect("bert-sawing")},
		"reference is not a label":    {good(), expect("annie.welding")},
		"no expected region":          {good(), Expect{Box: "annie-welding"}},
		"tofu names another instance": {good(), Expect{Box: "annie-welding", AWSRegion: "eu-west-1", TofuInstanceID: "i-0zzz"}},
		"unknown field in the record": {[]byte(`{"v":1,"box":"annie-welding","extra":1}`), expect("annie-welding")},
		"trailing data":               {append(good(), []byte(`{}`)...), expect("annie-welding")},
		"version 2": {func() []byte {
			r, _ := s.record(t, "annie-welding", "111122223333", "eu-west-1", "i-0abc")
			r.V = 2
			raw, _ := json.Marshal(r)
			return raw
		}(), expect("annie-welding")},
		"reference enrolled elsewhere": {good(), Expect{Box: "annie-welding", AWSRegion: "eu-west-1", Roster: []Enrolled{{Box: "annie-welding", Cloud: Cloud{InstanceID: "i-0old"}}}}},
	}
	for name, tc := range cases {
		t.Run(name, func(t *testing.T) {
			if ev, err := s.verifier().Enroll(tc.raw, tc.e); !errors.Is(err, ErrUnverified) {
				t.Fatalf("evidence = %+v, err = %v; want ErrUnverified", ev, err)
			}
		})
	}
}

// Review Focus 1: box A's record, intact and validly signed, copied under
// box B's key with its box field rewritten. The signature still verifies,
// because the box field is not signed. Only the roster check refuses it.
func TestEnrollRefusesAnInstanceAlreadyEnrolledUnderAnotherReference(t *testing.T) {
	s := newSigner(t)
	_, raw := s.record(t, "bert-sawing", "111122223333", "eu-west-1", "i-0abc")
	roster := []Enrolled{{Box: "annie-welding", Cloud: Cloud{Provider: "aws", Account: "111122223333", Region: "eu-west-1", InstanceID: "i-0abc"}}}
	if _, err := s.verifier().Enroll(raw, Expect{Box: "bert-sawing", AWSRegion: "eu-west-1", Roster: roster}); !errors.Is(err, ErrUnverified) {
		t.Fatalf("err = %v, want ErrUnverified", err)
	}
	// The same box re-enrolled as the same instance is not a conflict.
	roster[0].Box = "bert-sawing"
	if _, err := s.verifier().Enroll(raw, Expect{Box: "bert-sawing", AWSRegion: "eu-west-1", Roster: roster}); err != nil {
		t.Fatalf("re-enrolling the same instance: %v", err)
	}
}

func TestEnrollAcceptsTheTofuInstanceForALegacyReference(t *testing.T) {
	s := newSigner(t)
	_, raw := s.record(t, "annie-welding", "111122223333", "eu-west-1", "i-0abc")
	if _, err := s.verifier().Enroll(raw, Expect{Box: "annie-welding", AWSRegion: "eu-west-1", TofuInstanceID: "i-0abc"}); err != nil {
		t.Fatal(err)
	}
}

func TestPublishIsCreateOnlyAndIdempotent(t *testing.T) {
	ctx, st, s := context.Background(), release.NewDirStore(t.TempDir()), newSigner(t)
	rec, _ := s.record(t, "annie-welding", "111122223333", "eu-west-1", "i-0abc")
	if err := Publish(ctx, st, rec); err != nil {
		t.Fatal(err)
	}
	if err := Publish(ctx, st, rec); err != nil {
		t.Fatalf("republishing the same record: %v", err)
	}
	other := rec
	other.WrittenAt = "2027-01-01T00:00:00Z"
	if err := Publish(ctx, st, other); !errors.Is(err, release.ErrCASConflict) {
		t.Fatalf("a different record for the same box: err = %v, want ErrCASConflict", err)
	}
	raw, err := Load(ctx, st, "annie-welding")
	if err != nil {
		t.Fatal(err)
	}
	if _, err := s.verifier().Enroll(raw, expect("annie-welding")); err != nil {
		t.Fatalf("the published record does not enroll: %v", err)
	}
}

func TestPublishAndLoadRefuseAReferenceThatIsNotALabel(t *testing.T) {
	ctx, st, s := context.Background(), release.NewDirStore(t.TempDir()), newSigner(t)
	rec, _ := s.record(t, "x.bind", "111122223333", "eu-west-1", "i-0abc")
	if err := Publish(ctx, st, rec); err == nil {
		t.Fatal("published under a reference with a dot")
	}
	if _, err := Load(ctx, st, "../edges/x"); !errors.Is(err, ErrUnverified) {
		t.Fatalf("err = %v, want ErrUnverified", err)
	}
	if _, err := Load(ctx, st, "nobody-here"); !errors.Is(err, os.ErrNotExist) {
		t.Fatalf("err = %v, want os.ErrNotExist", err)
	}
}

func TestEmbeddedCertsCoverTheFlagshipRegions(t *testing.T) {
	certs, err := EmbeddedCerts()
	if err != nil {
		t.Fatal(err)
	}
	for _, region := range []string{"eu-west-1", "eu-west-2", "us-east-1", "ap-southeast-2"} {
		c := certs[region]
		if c == nil {
			t.Errorf("no certificate for %s", region)
			continue
		}
		if c.PublicKeyAlgorithm != x509.RSA {
			t.Errorf("%s: %v key; the base64-signature certificate is RSA (the PKCS7 one is DSA: wrong tab)", region, c.PublicKeyAlgorithm)
		}
	}
}
  • [ ] Step 2: Run them to see them fail

Run: go test ./internal/boxid/ -count=1 Expected: FAIL to build, undefined: Expect and s.verifier().Enroll undefined.

  • [ ] Step 3: Write the record

Replace internal/boxid/record.go with:

// Package boxid is the evidence a platform box presents to be enrolled in
// the roster (region HA spec 1 §5.1, ruling H11): the box's own AWS-signed
// instance identity document, published once under its box reference. The
// record grants nothing by itself. Enrollment verifies it (Verifier.Enroll)
// and copies the proven account, region and instance id into the roster
// entry's cloud field; from then on the roster, never this record, is what
// binds a reference to an instance.
package boxid

import (
	"bytes"
	"context"
	"encoding/json"
	"errors"
	"fmt"

	"github.com/carlosframework/platform/internal/controlname"
	"github.com/carlosframework/platform/internal/release"
)

const Prefix = "control/boxes/"

// ValidBox reports whether ref can be a box reference: one DNS label, so
// it can never contain "." or "/" and no two references share a key.
func ValidBox(ref string) bool { return controlname.ValidLabel(ref) }

// Key is where a box's record lives. Callers check ValidBox first.
func Key(box string) string { return Prefix + box + ".json" }

type Record struct {
	V                 int    `json:"v"`
	Box               string `json:"box"`
	IdentityDocument  string `json:"identity_document"`
	IdentitySignature string `json:"identity_signature"`
	WrittenAt         string `json:"written_at"`
}

// Publish writes a box's record create-only. Writing the same bytes again
// succeeds; a different record for the same box is ErrCASConflict, because
// a record is never replaced.
func Publish(ctx context.Context, s release.Store, rec Record) error {
	if !ValidBox(rec.Box) {
		return fmt.Errorf("boxid: %q is not a box reference", rec.Box)
	}
	body, err := json.Marshal(rec)
	if err != nil {
		return fmt.Errorf("boxid: marshal: %w", err)
	}
	_, err = s.Put(ctx, Key(rec.Box), body, "")
	if !errors.Is(err, release.ErrCASConflict) {
		return err
	}
	cur, _, gerr := s.Get(ctx, Key(rec.Box))
	if gerr != nil {
		return fmt.Errorf("boxid: %s exists and cannot be read: %w", rec.Box, gerr)
	}
	if bytes.Equal(cur, body) {
		return nil
	}
	return fmt.Errorf("boxid: %s already has a different record: %w", rec.Box, release.ErrCASConflict)
}

// Load returns the exact bytes of a box's record, which is what enrollment
// verifies and hashes. A missing record is os.ErrNotExist-wrapped.
func Load(ctx context.Context, s release.Store, box string) ([]byte, error) {
	if !ValidBox(box) {
		return nil, fmt.Errorf("%w: %q is not a box reference", ErrUnverified, box)
	}
	raw, _, err := s.Get(ctx, Key(box))
	return raw, err
}
  • [ ] Step 4: Write enrollment

Replace internal/boxid/verify.go with:

package boxid

import (
	"bytes"
	"crypto/sha256"
	"crypto/x509"
	"encoding/base64"
	"encoding/hex"
	"encoding/json"
	"errors"
	"fmt"
	"io"
)

// ErrUnverified wraps every reason a record is not evidence for enrollment.
var ErrUnverified = errors.New("boxid: record does not verify")

// Identity is what an identity document proves. termfence binds to it
// until it moves to Cloud; nothing else uses it.
type Identity struct{ AccountID, Region, InstanceID string }

// Cloud is the roster entry's cloud field (spec §2, §5.1): the instance a
// box reference is enrolled as. Enrollment fills it from a verified record;
// every later check (the termination fence, volume records) compares
// against it and never against anything a box says about itself.
type Cloud struct {
	Provider   string `json:"provider"`
	Account    string `json:"account"`
	Region     string `json:"region"`
	InstanceID string `json:"instance_id"`
}

// Complete reports whether every field is set and the provider is AWS.
func (c Cloud) Complete() bool {
	return c.Provider == "aws" && c.Account != "" && c.Region != "" && c.InstanceID != ""
}

type Verifier struct {
	Account string                       // the deployment's AWS account id
	Certs   map[string]*x509.Certificate // by AWS region
}

// Enrolled is one roster entry as enrollment needs to see it: its
// reference and the instance it was enrolled as. Entries in every state
// count, retired ones included: a retired reference never names a working
// box again (spec §5.9), and a terminated instance never comes back.
type Enrolled struct {
	Box   string
	Cloud Cloud
}

// Expect is what the enrolling side knows before it reads the record.
type Expect struct {
	Box       string // the reference being enrolled
	AWSRegion string // the AWS region of the placement it is enrolled into
	// TofuInstanceID is the region's tofu region-host instance_id output.
	// The caller sets it when Box already appears in claims or edges files,
	// as a box predating region HA does, and leaves it "" otherwise.
	TofuInstanceID string
	Roster         []Enrolled
}

// Evidence is what enrollment records: the roster entry's cloud, and the
// SHA-256 of the exact record bytes it was proven from.
type Evidence struct {
	Box          string
	Cloud        Cloud
	RecordSHA256 string
}

func refuse(format string, a ...any) error {
	return fmt.Errorf("%w: %s", ErrUnverified, fmt.Sprintf(format, a...))
}

func decodeRecord(raw []byte) (Record, error) {
	dec := json.NewDecoder(bytes.NewReader(raw))
	dec.DisallowUnknownFields()
	var rec Record
	if err := dec.Decode(&rec); err != nil {
		return Record{}, refuse("record is not JSON: %v", err)
	}
	if _, err := dec.Token(); err != io.EOF {
		return Record{}, refuse("trailing data after the record")
	}
	return rec, nil
}

// Enroll checks raw, the exact bytes stored at Key(e.Box), as enrollment
// evidence (spec §5.1, H11). In order: the reference, the record's own box
// field, AWS's signature under the certificate for the document's region,
// the deployment account, the AWS region expected for the placement, the
// tofu instance id for a reference predating region HA, and the roster:
// the reference enrolled as another instance, or the instance enrolled
// under another reference, is refused. A record copied from one box to
// another's key carries the first box's signed instance id, so the last
// check is what keeps one machine from holding two references.
func (v Verifier) Enroll(raw []byte, e Expect) (Evidence, error) {
	if !ValidBox(e.Box) {
		return Evidence{}, refuse("%q is not a box reference", e.Box)
	}
	if e.AWSRegion == "" {
		return Evidence{}, refuse("no expected AWS region for %s", e.Box)
	}
	rec, err := decodeRecord(raw)
	if err != nil {
		return Evidence{}, err
	}
	if rec.V != 1 {
		return Evidence{}, refuse("record version %d", rec.V)
	}
	if rec.Box != e.Box {
		return Evidence{}, refuse("the record stored for %q names %q", e.Box, rec.Box)
	}
	doc, err := base64.StdEncoding.DecodeString(rec.IdentityDocument)
	if err != nil {
		return Evidence{}, refuse("document is not base64")
	}
	sig, err := base64.StdEncoding.DecodeString(rec.IdentitySignature)
	if err != nil {
		return Evidence{}, refuse("signature is not base64")
	}
	var d struct {
		AccountID  string `json:"accountId"`
		Region     string `json:"region"`
		InstanceID string `json:"instanceId"`
	}
	if err := json.Unmarshal(doc, &d); err != nil {
		return Evidence{}, refuse("document is not JSON")
	}
	cert := v.Certs[d.Region]
	if cert == nil {
		return Evidence{}, refuse("no signing certificate for region %q", d.Region)
	}
	if err := cert.CheckSignature(x509.SHA256WithRSA, doc, sig); err != nil {
		return Evidence{}, refuse("signature: %v", err)
	}
	switch {
	case d.InstanceID == "":
		return Evidence{}, refuse("document names no instance")
	case v.Account == "" || d.AccountID != v.Account:
		return Evidence{}, refuse("account %q is not the deployment's", d.AccountID)
	case d.Region != e.AWSRegion:
		return Evidence{}, refuse("the box is in %q, the placement is in %q", d.Region, e.AWSRegion)
	case e.TofuInstanceID != "" && e.TofuInstanceID != d.InstanceID:
		return Evidence{}, refuse("tofu records instance %q for this region, the record %q", e.TofuInstanceID, d.InstanceID)
	}
	for _, en := range e.Roster {
		if en.Box == e.Box && en.Cloud.InstanceID != d.InstanceID {
			return Evidence{}, refuse("%s is enrolled as instance %q", e.Box, en.Cloud.InstanceID)
		}
		if en.Box != e.Box && en.Cloud.InstanceID == d.InstanceID {
			return Evidence{}, refuse("instance %s is enrolled as %s", d.InstanceID, en.Box)
		}
	}
	sum := sha256.Sum256(raw)
	return Evidence{
		Box:          e.Box,
		Cloud:        Cloud{Provider: "aws", Account: d.AccountID, Region: d.Region, InstanceID: d.InstanceID},
		RecordSHA256: hex.EncodeToString(sum[:]),
	}, nil
}
  • [ ] Step 5: Run the tests

Run: go test ./internal/boxid/ -count=1 -v Expected: PASS, including TestEnrollRefusesAnInstanceAlreadyEnrolledUnderAnotherReference and every subtest of TestEnrollRefusals.

  • [ ] Step 6: Gate and commit
go build ./... && go vet ./internal/boxid/ && gofmt -l internal/boxid
go test ./internal/boxid/ ./internal/termfence/ ./internal/console/ -run 'Enroll|Publish|Load|Embedded|Eligible|ChooseBox|Fence|Lookalike|Event|Legacy|Unverified' -count=1
GOOS=darwin go build ./... && GOOS=windows go build ./...
git add internal/boxid
git commit -m "boxid: a box record is enrollment evidence, and one instance never holds two references

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>"

Expected: gofmt -l prints nothing; every package ok.

---

Task 2: §19's corrections to choice and the warm copy

Spec §19: eligibleBoxes takes its identities from the roster, not from loading records; chooseBox's last tie-break becomes claim P5-D2's greatest SHA-256 of host + "\x00" + reference (H12); WarmCopyTrusted and the activator's warm path are withdrawn (H13), and the activator's one-box comment points at claim R6.

Files:

  • Modify (replace whole file): internal/console/boxchoice.go
  • Test (replace whole file): internal/console/boxchoice_test.go
  • Modify: internal/activator/installed.go, internal/activator/installed_test.go, internal/activator/activator.go:1164-1168

Interfaces:

  • Consumes: boxid.Cloud (Task 1).
  • Produces: ``go // package console type rosterBox struct{ Placement string; Cloud boxid.Cloud } func eligibleBoxes(snap edgesSnapshot, roster map[string]rosterBox, placement, awsRegion, homeName, expectedFP string, lost map[string]bool) []boxCandidate func chooseBox(host string, cands []boxCandidate, running map[string]boxLoad, app string) (boxCandidate, bool) ` Neither has a caller outside its tests yet: wiring them into create is L8. Removed from activator: Installation, WarmCopyTrusted`.
  • [ ] Step 1: Write the failing tests

Replace internal/console/boxchoice_test.go with:

package console

import (
	"crypto/sha256"
	"encoding/hex"
	"encoding/json"
	"testing"

	"github.com/carlosframework/platform/internal/boxid"
)

// haBox is an eligible box's edges file in placement "uk".
func haBox(id string, mut func(m map[string]any)) string {
	m := map[string]any{
		"v": 1, "box": id, "box_id": id, "eip": "1.2.3.4", "region": "eu-west-2", "placement": "uk",
		"hosts": map[string]any{}, "hibernation_configured": true, "serves_shared_pool": true,
		"accepts_inbound": true, "secrets_key_fp": testKeyFP, "capabilities": []string{"region-ha-1"},
		"headroom": map[string]any{"mem_free_mib": 1000, "disk_free_mib": 1000, "awake": 1},
	}
	if mut != nil {
		mut(m)
	}
	b, _ := json.Marshal(m)
	return string(b)
}

func ids(cs []boxCandidate) []string {
	out := make([]string, len(cs))
	for i, c := range cs {
		out[i] = c.BoxID
	}
	return out
}

// rosterFor enrolls each box in placement "uk" as an instance in eu-west-2.
func rosterFor(boxes ...string) map[string]rosterBox {
	out := map[string]rosterBox{}
	for _, b := range boxes {
		out[b] = rosterBox{Placement: "uk", Cloud: boxid.Cloud{Provider: "aws", Account: "1", Region: "eu-west-2", InstanceID: "i-" + b}}
	}
	return out
}

func TestEligibleBoxesAppliesThePredicateOneBoxAtATime(t *testing.T) {
	del := func(k string) func(map[string]any) { return func(m map[string]any) { delete(m, k) } }
	set := func(k string, v any) func(map[string]any) { return func(m map[string]any) { m[k] = v } }
	cases := []struct {
		name string
		mut  func(map[string]any)
		want bool
	}{
		{"all present", nil, true},
		{"no dial address", set("eip", ""), false},
		{"hibernation unconfigured", set("hibernation_configured", false), false},
		{"hibernation unknown", del("hibernation_configured"), false},
		{"refuses the shared pool", set("serves_shared_pool", false), false},
		{"refuses inbound", set("accepts_inbound", false), false},
		{"wrong secrets key", set("secrets_key_fp", "sha256:other"), false},
		{"no capability", del("capabilities"), false},
		{"draining", set("draining", true), false},
		{"another placement", set("placement", "us"), false},
		{"box_id names no roster entry", set("box_id", "i-0zzz"), false},
	}
	for _, tc := range cases {
		t.Run(tc.name, func(t *testing.T) {
			snap := views(t, haBox("i-0aaa", tc.mut))
			got := eligibleBoxes(snap, rosterFor("i-0aaa"), "uk", "eu-west-2", "ie", testKeyFP, nil)
			if (len(got) == 1) != tc.want {
				t.Fatalf("eligible = %v, want %v", ids(got), tc.want)
			}
		})
	}
}

func TestEligibleBoxesNeedsARosterEntryFreshnessAndNotLost(t *testing.T) {
	snap := views(t, haBox("i-0aaa", nil), haBox("i-0bbb", nil), haBox("i-0ccc", nil))
	snap = markStale(snap, "i-0bbb")
	got := eligibleBoxes(snap, rosterFor("i-0aaa", "i-0bbb"), "uk", "eu-west-2", "ie", testKeyFP, nil)
	if len(got) != 1 || got[0].BoxID != "i-0aaa" {
		t.Fatalf("got %v, want only i-0aaa (i-0bbb is stale, i-0ccc has no roster entry)", ids(got))
	}
	got = eligibleBoxes(snap, rosterFor("i-0aaa"), "uk", "eu-west-2", "ie", testKeyFP, map[string]bool{"i-0aaa": true})
	if len(got) != 0 {
		t.Fatalf("a lost box is eligible: %v", ids(got))
	}
}

// TestEligibilityNeverCombinesBoxes is nearest-region's rule for boxes: one
// box holding the key and another accepting inbound do not add up to one.
func TestEligibleBoxesNeverCombinesTwoHalfBoxes(t *testing.T) {
	snap := views(t,
		haBox("i-0aaa", func(m map[string]any) { m["secrets_key_fp"] = "" }),
		haBox("i-0bbb", func(m map[string]any) { m["accepts_inbound"] = false }))
	if got := eligibleBoxes(snap, rosterFor("i-0aaa", "i-0bbb"), "uk", "eu-west-2", "ie", testKeyFP, nil); len(got) != 0 {
		t.Fatalf("got %v, want none", ids(got))
	}
}

func TestEligibleBoxesAnIncompleteSnapshotOffersNothing(t *testing.T) {
	snap := views(t, haBox("i-0aaa", nil))
	snap.Complete = false
	if got := eligibleBoxes(snap, rosterFor("i-0aaa"), "uk", "eu-west-2", "ie", testKeyFP, nil); len(got) != 0 {
		t.Fatalf("got %v from an incomplete snapshot", ids(got))
	}
}

func TestEligibleBoxesHomePlacementMatchesTheEmptyCode(t *testing.T) {
	home := haBox("i-0aaa", func(m map[string]any) { m["placement"] = "" })
	named := haBox("i-0bbb", func(m map[string]any) { m["placement"] = "ie" })
	snap := views(t, home, named)
	roster := rosterFor("i-0aaa", "i-0bbb")
	for _, b := range []string{"i-0aaa", "i-0bbb"} {
		rb := roster[b]
		rb.Placement = "ie"
		roster[b] = rb
	}
	got := eligibleBoxes(snap, roster, "", "eu-west-2", "ie", testKeyFP, nil)
	if len(got) != 2 {
		t.Fatalf("got %v, want both home boxes", ids(got))
	}
}

func TestChooseBox(t *testing.T) {
	a, b := boxCandidate{BoxID: "i-0aaa"}, boxCandidate{BoxID: "i-0bbb"}
	load := func(hosts int, apps ...string) boxLoad {
		l := boxLoad{Hosts: hosts, Apps: map[string]int{}}
		for _, x := range apps {
			l.Apps[x]++
		}
		return l
	}
	const host = "shop.bab.oncarlos.com"
	if _, ok := chooseBox(host, nil, nil, "bab/jam"); ok {
		t.Fatal("chose from nothing")
	}
	got, _ := chooseBox(host, []boxCandidate{a, b}, map[string]boxLoad{"i-0aaa": load(5), "i-0bbb": load(2)}, "bab/jam")
	if got.BoxID != "i-0bbb" {
		t.Fatalf("fewest hosts: got %s", got.BoxID)
	}
	got, _ = chooseBox(host, []boxCandidate{a, b}, map[string]boxLoad{"i-0aaa": load(5), "i-0bbb": load(2, "bab/jam")}, "bab/jam")
	if got.BoxID != "i-0aaa" {
		t.Fatalf("spread beats load: got %s", got.BoxID)
	}
	got, ok := chooseBox(host, []boxCandidate{a, b}, map[string]boxLoad{"i-0aaa": load(5, "bab/jam"), "i-0bbb": load(2, "bab/jam", "bab/jam")}, "bab/jam")
	if !ok || got.BoxID != "i-0bbb" {
		t.Fatalf("every box runs the app: fewest hosts overall, never a refusal; got %s ok=%v", got.BoxID, ok)
	}
}

// H12, claim P5-D2: an equal load breaks by the greatest SHA-256 of
// host + "\x00" + reference, so the choice differs host by host instead
// of always landing on the lowest name, and does not depend on the order
// the candidates arrive in.
func TestChooseBoxBreaksATieByTheHostsHash(t *testing.T) {
	cands := []boxCandidate{{BoxID: "annie-welding"}, {BoxID: "bert-sawing"}, {BoxID: "cleo-mixing"}}
	want := func(host string) string {
		best, bestSum := "", ""
		for _, c := range cands {
			sum := sha256.Sum256([]byte(host + "\x00" + c.BoxID))
			if h := hex.EncodeToString(sum[:]); h > bestSum {
				best, bestSum = c.BoxID, h
			}
		}
		return best
	}
	seen := map[string]bool{}
	for _, host := range []string{"a.bab.oncarlos.com", "b.bab.oncarlos.com", "c.bab.oncarlos.com", "d.bab.oncarlos.com", "e.bab.oncarlos.com", "f.bab.oncarlos.com"} {
		forward, _ := chooseBox(host, cands, nil, "bab/jam")
		backward, _ := chooseBox(host, []boxCandidate{cands[2], cands[1], cands[0]}, nil, "bab/jam")
		if forward.BoxID != want(host) || backward.BoxID != want(host) {
			t.Fatalf("%s: chose %s and %s, want %s", host, forward.BoxID, backward.BoxID, want(host))
		}
		seen[forward.BoxID] = true
	}
	if len(seen) < 2 {
		t.Fatalf("six hosts all chose %v: the tie-break is not per host", seen)
	}
}

// A box enrolled in another AWS region than the placement's is not in that
// placement, even when its own file says it is.
func TestEligibleBoxesRefusesABoxEnrolledInAnotherRegion(t *testing.T) {
	snap := views(t, haBox("i-0aaa", nil), haBox("i-0bbb", nil))
	roster := rosterFor("i-0aaa", "i-0bbb")
	roster["i-0bbb"] = rosterBox{Placement: "uk", Cloud: boxid.Cloud{Provider: "aws", Account: "1", Region: "us-east-1", InstanceID: "i-0bbb"}}
	got := eligibleBoxes(snap, roster, "uk", "eu-west-2", "ie", testKeyFP, nil)
	if len(got) != 1 || got[0].BoxID != "i-0aaa" {
		t.Fatalf("got %v, want only i-0aaa", ids(got))
	}
	// And a file that advertises another AWS region than the one expected.
	snap = views(t, haBox("i-0aaa", func(m map[string]any) { m["region"] = "us-east-1" }))
	if got := eligibleBoxes(snap, rosterFor("i-0aaa"), "uk", "eu-west-2", "ie", testKeyFP, nil); len(got) != 0 {
		t.Fatalf("got %v, want none", ids(got))
	}
	if got := eligibleBoxes(views(t, haBox("i-0aaa", nil)), rosterFor("i-0aaa"), "uk", "", "ie", testKeyFP, nil); len(got) != 0 {
		t.Fatalf("got %v with no expected region", ids(got))
	}
}

// The roster's enrolled placement is the enrollment's word and must agree
// with the file's; an incomplete cloud is no identity at all.
func TestEligibleBoxesNeedsTheRosterToAgree(t *testing.T) {
	snap := views(t, haBox("i-0aaa", nil), haBox("i-0bbb", nil))
	roster := rosterFor("i-0aaa", "i-0bbb")
	roster["i-0aaa"] = rosterBox{Placement: "us", Cloud: roster["i-0aaa"].Cloud}
	roster["i-0bbb"] = rosterBox{Placement: "uk", Cloud: boxid.Cloud{Provider: "aws", Region: "eu-west-2", InstanceID: "i-0bbb"}}
	if got := eligibleBoxes(snap, roster, "uk", "eu-west-2", "ie", testKeyFP, nil); len(got) != 0 {
		t.Fatalf("got %v, want none", ids(got))
	}
}
  • [ ] Step 2: Run them to see them fail

Run: go test ./internal/console/ -run 'EligibleBoxes|ChooseBox' -count=1 Expected: FAIL to build, undefined: rosterBox and too many arguments in call to chooseBox.

  • [ ] Step 3: Write the choice

Replace internal/console/boxchoice.go with:

package console

import (
	"crypto/sha256"
	"encoding/hex"
	"slices"
	"sort"

	"github.com/carlosframework/platform/internal/boxid"
)

// boxCandidate is one box that may be named as a host's executor (region HA
// spec 1 §2, "eligible box").
type boxCandidate struct {
	BoxID                   string
	Region                  string
	MemFreeMiB, DiskFreeMiB int64
	Awake                   int
}

// rosterBox is what the roster (control/host-actors.json, claim D7) says
// about one active box: the placement it is enrolled in and the cloud
// identity enrollment proved from its boxid record (spec §5.1, H11). The
// caller passes active entries only.
type rosterBox struct {
	Placement string
	Cloud     boxid.Cloud
}

// eligibleBoxes is regionCandidates' predicate applied to one box at a time,
// plus what a multi-box region adds: a roster entry for the file's box_id,
// enrolled in this placement, whose proven cloud is in awsRegion (the AWS
// region the roster records for this placement), the HA capability, not
// draining, not lost. Two boxes that each satisfy half of it never add up
// to one. The edges file is evidence of liveness and configuration only:
// identity and region come from the roster (spec §5.1).
//
// An incomplete snapshot offers nothing: an unread file could be the one
// box that would change the answer.
func eligibleBoxes(snap edgesSnapshot, roster map[string]rosterBox, placement, awsRegion, homeName, expectedFP string, lost map[string]bool) []boxCandidate {
	if !snap.complete() || expectedFP == "" || awsRegion == "" {
		return nil
	}
	home := func(code string) string {
		if homeName != "" && code == homeName {
			return ""
		}
		return code
	}
	var out []boxCandidate
	for _, f := range snap.Files {
		want := home(placement)
		rb, ok := roster[f.BoxID]
		switch {
		case home(f.Placement) != want:
		case !f.Fresh, f.EIP == "", f.BoxID == "":
		case !ok || !rb.Cloud.Complete():
		case home(rb.Placement) != want:
			// The file's placement is the box's own word; the roster's is
			// the enrollment's. Both must agree.
		case rb.Cloud.Region != awsRegion, f.Region != awsRegion:
			// The AWS region was proven at enrollment: a box enrolled in
			// another region is not in this placement, whatever its file
			// says.
		case !isTrue(f.HibernationConfigured), !isTrue(f.ServesSharedPool), !isTrue(f.AcceptsInbound):
		case f.SecretsKeyFP != expectedFP:
		case !slices.Contains(f.Capabilities, "region-ha-1"):
		case f.Draining, lost[f.BoxID]:
		default:
			c := boxCandidate{BoxID: f.BoxID, Region: rb.Cloud.Region}
			if f.Headroom != nil {
				c.MemFreeMiB, c.DiskFreeMiB, c.Awake = f.Headroom.MemFreeMiB, f.Headroom.DiskFreeMiB, f.Headroom.Awake
			}
			out = append(out, c)
		}
	}
	sort.Slice(out, func(i, j int) bool { return out[i].BoxID < out[j].BoxID })
	return out
}

// boxLoad is what claims already place on a box.
type boxLoad struct {
	Hosts int
	Apps  map[string]int // "acct/app" -> hosts of that app
}

// chooseBox is spec 1 §5.2's choose() until spec 2 (H12): prefer a box
// that runs no instance of this app (soft spread: it never refuses), then
// the fewest hosts overall, then the greatest hexadecimal SHA-256 of
// host + "\x00" + box reference (claim P5-D2), then the smallest reference
// if two hashes are equal.
func chooseBox(host string, cands []boxCandidate, running map[string]boxLoad, app string) (boxCandidate, bool) {
	if len(cands) == 0 {
		return boxCandidate{}, false
	}
	tie := func(ref string) string {
		sum := sha256.Sum256([]byte(host + "\x00" + ref))
		return hex.EncodeToString(sum[:])
	}
	best := cands[0]
	for _, c := range cands[1:] {
		cl, bl := running[c.BoxID], running[best.BoxID]
		cRuns, bRuns := cl.Apps[app] > 0, bl.Apps[app] > 0
		switch {
		case cRuns != bRuns:
			if !cRuns {
				best = c
			}
		case cl.Hosts != bl.Hosts:
			if cl.Hosts < bl.Hosts {
				best = c
			}
		case tie(c.BoxID) != tie(best.BoxID):
			if tie(c.BoxID) > tie(best.BoxID) {
				best = c
			}
		case c.BoxID < best.BoxID:
			best = c
		}
	}
	return best, true
}
  • [ ] Step 4: Remove the warm-copy check

In internal/activator/installed.go, replace the InstallMarker doc comment:

// InstallMarker is a box's own record that it finished installing one
// generation of a host: the database restored and synced, the files
// applied, the wake state imported (region HA spec 1 §5.5). It is written
// last, so its presence vouches for all of them.
//
// It is evidence, not authority. Authority is the claim naming this box,
// which the box holds as its active stamp. A box that crashed mid-install
// has authority and no marker, and must restore.

with:

// InstallMarker is a box's own note that it finished a local install of
// one generation of a host. Nothing reads it to trust a local database:
// claim ruling R6 has every admitted writer purge and restore from the
// certified remote (region HA spec 1 H13, §5.5 withdrawn). It is kept for
// relocation's local install journal (§5.8), which plan 1b does not build.

Delete the Installation type (the block starting // Installation is what the claim says is current for a host.) and the whole of WarmCopyTrusted with its doc comment, from // WarmCopyTrusted is invariant HA3 to the end of the file. In internal/activator/installed_test.go, delete TestWarmCopyTrusted, from func TestWarmCopyTrusted(t *testing.T) { to the end of the file.

In internal/activator/activator.go, in doWake, replace:

	// S3 state written by the true holder). When it's still on disk it is the
	// warm cache: on one box, with the lease held, the local copy can only be
	// what we ourselves checkpointed at the last sleep.

with:

	// S3 state written by the true holder). When it's still on disk it is the
	// warm cache: on one box, with the lease held, the local copy can only be
	// what we ourselves checkpointed at the last sleep. That holds only while
	// a region has one box. Claim ruling R6 ends it (region HA spec 1 H13):
	// once wakes are admitted by hostclaim's AdmitWriter, every admitted
	// writer purges its local copy and restores from the certified remote,
	// and no marker is ever read to skip that.

The behaviour of doWake does not change: the warm cache stays correct while every region has one box, and replacing it is the claim's AdmitWriter wiring, not this plan's.

  • [ ] Step 5: Run the tests

Run:

go test ./internal/console/ -run 'EligibleBoxes|ChooseBox|CandidateLocation' -count=1 -v
go test ./internal/activator/ -run 'InstallMarker|DamagedMarker' -count=1

Expected: PASS, including TestChooseBoxBreaksATieByTheHostsHash and TestEligibleBoxesNeedsTheRosterToAgree.

  • [ ] Step 6: Gate and commit
go build ./... && go vet ./internal/console/ ./internal/activator/ && gofmt -l internal/console internal/activator
GOOS=darwin go build ./... && GOOS=windows go build ./...
git add internal/console/boxchoice.go internal/console/boxchoice_test.go internal/activator/installed.go internal/activator/installed_test.go internal/activator/activator.go
git commit -m "console: eligible boxes come from the roster's proven cloud, and a tie breaks by the host's hash; activator: no local copy is trusted

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>"

---

Task 3: The agent publishes its identity record (internal/host/boxrecord.go)

Spec §19: "Item 6 becomes publishing the enrollment evidence at start." Plan 1a's "After the build" item 13: the record is written under the box's label. With CARLOS_REGION_HA on, the agent reads IMDS's identity document and signature once and publishes them create-only. The record grants nothing; enrollment reads it later.

Files:

  • Create: internal/host/boxrecord.go
  • Test: internal/host/boxrecord_test.go
  • Modify: internal/host/edgesfile.go (edgesFileCapabilities), internal/host/agent.go:524

Interfaces:

  • Consumes: boxid.Record, boxid.ValidBox, boxid.Load, boxid.Publish (Task 1); openDeployStore, metricsBoxLabel, readMachineID, shouldSkipPublisherElection (existing).
  • Produces: ``go func regionHAEnabled(getenv func(string) string) bool // used by Tasks 6 and 7 func readIdentityRecord(client *http.Client, base, box string, now time.Time) (boxid.Record, error) func publishBoxRecord(ctx context.Context, s release.Store, box string, read func() (boxid.Record, error)) (done bool, err error) func boxRecordLoop() ``

Two things IMDS does that matter. The identity document is signed byte for byte, so it is stored exactly as served: imdsLeaf trims, and is not used. The signature may arrive wrapped across lines, which Go's base64 decoder refuses, so whitespace is removed from it.

  • [ ] Step 1: Write the failing tests

Create internal/host/boxrecord_test.go:

package host

import (
	"context"
	"encoding/base64"
	"errors"
	"io"
	"net/http"
	"strings"
	"testing"
	"time"

	"github.com/carlosframework/platform/internal/boxid"
	"github.com/carlosframework/platform/internal/release"
)

// fakeIMDS answers IMDSv2 in-process: no socket, so it runs in any sandbox.
type fakeIMDS struct {
	doc, sig string
	status   int // for the document and signature; 0 = 200
	calls    []string
}

func (f *fakeIMDS) RoundTrip(r *http.Request) (*http.Response, error) {
	f.calls = append(f.calls, r.Method+" "+r.URL.Path)
	answer := func(code int, body string) (*http.Response, error) {
		return &http.Response{StatusCode: code, Body: io.NopCloser(strings.NewReader(body)), Header: http.Header{}, Request: r}, nil
	}
	if r.Method == http.MethodPut && r.URL.Path == "/latest/api/token" {
		return answer(200, "tok\n")
	}
	if r.Header.Get("X-aws-ec2-metadata-token") != "tok" {
		return answer(401, "")
	}
	code := f.status
	if code == 0 {
		code = 200
	}
	switch r.URL.Path {
	case imdsIdentityDocument:
		return answer(code, f.doc)
	case imdsIdentitySignature:
		return answer(code, f.sig)
	}
	return answer(404, "")
}

func imdsClient(f *fakeIMDS) *http.Client { return &http.Client{Transport: f} }

var recordNow = time.Date(2026, 10, 6, 9, 0, 0, 0, time.UTC)

// The document is signed byte for byte, so its trailing newline must
// survive; the signature arrives wrapped and must not.
func TestReadIdentityRecordKeepsTheDocumentsExactBytes(t *testing.T) {
	doc := "{\n  \"accountId\" : \"111122223333\",\n  \"instanceId\" : \"i-0abc\"\n}\n"
	sig := base64.StdEncoding.EncodeToString([]byte(strings.Repeat("s", 256)))
	wrapped := sig[:64] + "\n" + sig[64:] + "\n"
	f := &fakeIMDS{doc: doc, sig: wrapped}
	rec, err := readIdentityRecord(imdsClient(f), "http://imds.test", "annie-welding", recordNow)
	if err != nil {
		t.Fatal(err)
	}
	got, _ := base64.StdEncoding.DecodeString(rec.IdentityDocument)
	if string(got) != doc {
		t.Fatalf("document = %q, want the bytes IMDS served, %q", got, doc)
	}
	if rec.IdentitySignature != sig || rec.V != 1 || rec.Box != "annie-welding" || rec.WrittenAt != "2026-10-06T09:00:00Z" {
		t.Fatalf("record = %+v", rec)
	}
}

func TestReadIdentityRecordRefusesWhatIsNotAnAnswer(t *testing.T) {
	for name, f := range map[string]*fakeIMDS{
		"document missing":     {doc: "{}", sig: "c2lnCg==", status: 404},
		"signature not base64": {doc: "{}", sig: "not base64!"},
		"signature empty":      {doc: "{}", sig: "\n"},
	} {
		t.Run(name, func(t *testing.T) {
			if rec, err := readIdentityRecord(imdsClient(f), "http://imds.test", "annie-welding", recordNow); err == nil {
				t.Fatalf("record = %+v, want an error", rec)
			}
		})
	}
}

func TestPublishBoxRecordWritesOnceAndNeverRewrites(t *testing.T) {
	ctx, st := context.Background(), release.NewDirStore(t.TempDir())
	reads := 0
	read := func() (boxid.Record, error) {
		reads++
		return boxid.Record{V: 1, Box: "annie-welding", IdentityDocument: "ZG9j", IdentitySignature: "c2ln", WrittenAt: "2026-10-06T09:00:00Z"}, nil
	}
	if done, err := publishBoxRecord(ctx, st, "annie-welding", read); !done || err != nil {
		t.Fatalf("first publish: done=%v err=%v", done, err)
	}
	if done, err := publishBoxRecord(ctx, st, "annie-welding", read); !done || err != nil {
		t.Fatalf("second publish: done=%v err=%v", done, err)
	}
	if reads != 1 {
		t.Fatalf("IMDS read %d times; an existing record must not be re-read or rewritten", reads)
	}
}

func TestPublishBoxRecordRetriesAReadFailureAndStopsOnABadLabel(t *testing.T) {
	ctx, st := context.Background(), release.NewDirStore(t.TempDir())
	failing := func() (boxid.Record, error) { return boxid.Record{}, errors.New("IMDS down") }
	if done, err := publishBoxRecord(ctx, st, "annie-welding", failing); done || err == nil {
		t.Fatalf("done=%v err=%v: a read failure is retried", done, err)
	}
	if done, err := publishBoxRecord(ctx, st, "Annie.Welding", failing); !done || err == nil {
		t.Fatalf("done=%v err=%v: a reference that is not a label is not retried", done, err)
	}
}

// A store that refuses the read with something other than not-found must
// not be taken as "no record yet": that would publish over an outage.
type unreadableBoxStore struct{ release.Store }

func (unreadableBoxStore) Get(context.Context, string) ([]byte, string, error) {
	return nil, "", errors.New("AccessDenied")
}

func TestPublishBoxRecordDoesNotPublishWhenItCannotRead(t *testing.T) {
	ctx := context.Background()
	st := unreadableBoxStore{release.NewDirStore(t.TempDir())}
	read := func() (boxid.Record, error) {
		t.Fatal("IMDS read although the store could not say whether a record exists")
		return boxid.Record{}, nil
	}
	if done, err := publishBoxRecord(ctx, st, "annie-welding", read); done || err == nil {
		t.Fatalf("done=%v err=%v", done, err)
	}
}

func TestRegionHAEnabled(t *testing.T) {
	for v, want := range map[string]bool{"": false, "0": false, " 0 ": false, "1": true, "yes": true} {
		if got := regionHAEnabled(func(string) string { return v }); got != want {
			t.Errorf("CARLOS_REGION_HA=%q: %v, want %v", v, got, want)
		}
	}
}
  • [ ] Step 2: Run them to see them fail

Run: go test ./internal/host/ -run 'IdentityRecord|BoxRecord|RegionHAEnabled' -count=1 Expected: FAIL to build, undefined: readIdentityRecord.

  • [ ] Step 3: Write the publisher

Create internal/host/boxrecord.go:

package host

import (
	"context"
	"encoding/base64"
	"errors"
	"fmt"
	"io"
	"log"
	"net/http"
	"os"
	"strings"
	"time"

	"github.com/carlosframework/platform/internal/boxid"
	"github.com/carlosframework/platform/internal/release"
)

// regionHAEnabled is CARLOS_REGION_HA: on for any value but "" and "0".
// Every region HA behaviour on a box is behind it (spec 1 §8 step 2).
func regionHAEnabled(getenv func(string) string) bool {
	switch strings.TrimSpace(getenv("CARLOS_REGION_HA")) {
	case "", "0":
		return false
	}
	return true
}

const (
	imdsBase              = "http://169.254.169.254"
	imdsIdentityDocument  = "/latest/dynamic/instance-identity/document"
	imdsIdentitySignature = "/latest/dynamic/instance-identity/signature"
	boxRecordRetry        = time.Minute
)

// imdsGetRaw reads one IMDSv2 path and returns its body untouched. The
// identity document is signed byte for byte, so unlike imdsLeaf nothing is
// trimmed.
func imdsGetRaw(client *http.Client, base, token, path string) ([]byte, error) {
	req, err := http.NewRequest(http.MethodGet, base+path, nil)
	if err != nil {
		return nil, err
	}
	req.Header.Set("X-aws-ec2-metadata-token", token)
	resp, err := client.Do(req)
	if err != nil {
		return nil, err
	}
	defer resp.Body.Close()
	body, err := io.ReadAll(io.LimitReader(resp.Body, 64<<10))
	if err != nil {
		return nil, err
	}
	if resp.StatusCode != http.StatusOK {
		return nil, fmt.Errorf("IMDS %s answered %d", path, resp.StatusCode)
	}
	return body, nil
}

// readIdentityRecord builds this box's record (spec §5.1) from IMDS: the
// identity document exactly as served, and its base64 RSA-SHA256
// signature with any line breaks removed (base64 ignores them; Go's
// decoder does not).
func readIdentityRecord(client *http.Client, base, box string, now time.Time) (boxid.Record, error) {
	req, err := http.NewRequest(http.MethodPut, base+"/latest/api/token", nil)
	if err != nil {
		return boxid.Record{}, err
	}
	req.Header.Set("X-aws-ec2-metadata-token-ttl-seconds", "60")
	resp, err := client.Do(req)
	if err != nil {
		return boxid.Record{}, fmt.Errorf("IMDS token: %w", err)
	}
	tok, _ := io.ReadAll(io.LimitReader(resp.Body, 4<<10))
	resp.Body.Close()
	if resp.StatusCode != http.StatusOK || len(strings.TrimSpace(string(tok))) == 0 {
		return boxid.Record{}, fmt.Errorf("IMDS token answered %d", resp.StatusCode)
	}
	token := strings.TrimSpace(string(tok))
	doc, err := imdsGetRaw(client, base, token, imdsIdentityDocument)
	if err != nil {
		return boxid.Record{}, err
	}
	sig, err := imdsGetRaw(client, base, token, imdsIdentitySignature)
	if err != nil {
		return boxid.Record{}, err
	}
	cleaned := strings.Join(strings.Fields(string(sig)), "")
	if _, err := base64.StdEncoding.DecodeString(cleaned); err != nil || cleaned == "" {
		return boxid.Record{}, errors.New("IMDS signature is not base64")
	}
	return boxid.Record{V: 1, Box: box, IdentityDocument: base64.StdEncoding.EncodeToString(doc),
		IdentitySignature: cleaned, WrittenAt: now.UTC().Format(time.RFC3339)}, nil
}

// publishBoxRecord makes one attempt to publish this box's enrollment
// evidence. done is true once there is nothing left to retry: the record
// is in place, or a different one already holds this reference, which
// only an operator can sort out and which is logged loudly.
func publishBoxRecord(ctx context.Context, s release.Store, box string, read func() (boxid.Record, error)) (done bool, err error) {
	if !boxid.ValidBox(box) {
		return true, fmt.Errorf("box reference %q is not a label; no enrollment evidence is published", box)
	}
	if _, err := boxid.Load(ctx, s, box); err == nil {
		return true, nil // create-only: an existing record is never rewritten
	} else if !errors.Is(err, os.ErrNotExist) {
		return false, err
	}
	rec, err := read()
	if err != nil {
		return false, err
	}
	err = boxid.Publish(ctx, s, rec)
	if errors.Is(err, release.ErrCASConflict) {
		return true, fmt.Errorf("another record already holds %s: %w", box, err)
	}
	return err == nil, err
}

// boxRecordLoop publishes this box's enrollment evidence once, at start,
// when CARLOS_REGION_HA is on, retrying each minute until it is in place.
// The record grants nothing: enrollment (an operator, or for Auto Scaling
// boxes the console with a membership confirmation) reads it later.
// Platform-box job: skipped on fleet boxes.
func boxRecordLoop() {
	if !regionHAEnabled(os.Getenv) || shouldSkipPublisherElection(os.Getenv) {
		return
	}
	ctx := context.Background()
	store, err := openDeployStore(ctx)
	if err != nil {
		log.Printf("agent: box record not published: %v", err)
		return
	}
	box := metricsBoxLabel(os.Getenv, readMachineID())
	client := &http.Client{Timeout: 2 * time.Second}
	read := func() (boxid.Record, error) { return readIdentityRecord(client, imdsBase, box, time.Now()) }
	for attempt := 1; ; attempt++ {
		done, err := publishBoxRecord(ctx, store, box, read)
		if done {
			if err != nil {
				log.Printf("agent: box record for %s: %v (region HA cannot enroll this box until an operator resolves it)", box, err)
			}
			return
		}
		if attempt == 1 || attempt%60 == 0 {
			log.Printf("agent: box record for %s not published yet (#%d): %v", box, attempt, err)
		}
		time.Sleep(boxRecordRetry)
	}
}
  • [ ] Step 4: Read the flag in one place

In internal/host/edgesfile.go, replace the body of edgesFileCapabilities:

func edgesFileCapabilities(getenv func(string) string) []string {
	switch strings.TrimSpace(getenv("CARLOS_REGION_HA")) {
	case "", "0":
		return nil
	}
	return []string{capRegionHA1}
}

with:

func edgesFileCapabilities(getenv func(string) string) []string {
	if !regionHAEnabled(getenv) {
		return nil
	}
	return []string{capRegionHA1}
}
  • [ ] Step 5: Start the loop

In internal/host/agent.go, in RunAgent, replace:

	go edgesFileLoop(reg)

with:

	go edgesFileLoop(reg)
	// Region HA enrollment evidence (spec 1 §5.1): once, at start, and only
	// with CARLOS_REGION_HA on. See boxrecord.go.
	go boxRecordLoop()
  • [ ] Step 6: Run the tests

Run: go test ./internal/host/ -run 'IdentityRecord|BoxRecord|RegionHAEnabled|EdgesFileCapability|Headroom' -count=1 -v Expected: PASS.

  • [ ] Step 7: Gate and commit
go build ./... && go vet ./internal/host/ && gofmt -l internal/host
GOOS=darwin go build ./... && GOOS=windows go build ./...
git add internal/host/boxrecord.go internal/host/boxrecord_test.go internal/host/edgesfile.go internal/host/agent.go
git commit -m "agent: with region HA on, a box publishes its AWS-signed identity once, under its label

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>"

---

Task 4: The fence binds to the roster entry (internal/termfence)

Spec §19: "Verify binds to a boxid.Identity; it must bind to the roster entry's cloud and produce §5.9's record. It takes no volume input (H19) … FromDescribe and FromEvent stay as the two terminated reads; a shutting-down observation is never accepted." §6 adds that the fence refuses any observation not made in the box's own region. This task also gives the describe a seam: one call, pinned to the box's region, with InvalidInstanceID.NotFound read as absence.

Writing the record to control/box-fences/<enrollment>.json, and citing it in a claim CAS, are L2 to L4 and are not built.

Files:

  • Modify (replace whole file): internal/termfence/fence.go
  • Test (replace whole file): internal/termfence/fence_test.go
  • Modify: internal/boxid/verify.go (remove Identity)

Interfaces:

  • Consumes: boxid.Cloud (Task 1).
  • Produces: ``go package termfence const ViaDescribe, ViaEvent = "describe", "event" var ErrNotEvidence error type Observation struct{ Via string; Found bool; InstanceID, Account, Region, State, EventID, RequestID string } type Box struct{ Ref, Enrollment string; Cloud boxid.Cloud } type WrittenBy struct{ Box, Enrollment string } // json: box, enrollment type Record struct{ V int; Box, Enrollment string; Cloud boxid.Cloud; State, ObservedAt, Via, RequestID, EventID string; WrittenBy WrittenBy } // §5.9 type EC2 interface{ DescribeInstances(ctx, *ec2.DescribeInstancesInput, ...func(*ec2.Options)) (*ec2.DescribeInstancesOutput, error) } func FromDescribe(out *ec2.DescribeInstancesOutput, instanceID, region string) Observation func FromEvent(body []byte) (Observation, error) func DescribeInstance(ctx context.Context, api EC2, c boxid.Cloud) (*ec2.DescribeInstancesOutput, error) // used by Task 5's callers too func Verify(box, worker Box, o Observation, now time.Time) (Record, error) ` Removed: Fence, and Verify's (string, boxid.Identity, …)` form.
  • [ ] Step 1: Write the failing tests

Replace internal/termfence/fence_test.go with:

package termfence

import (
	"context"
	"errors"
	"testing"
	"time"

	"github.com/aws/aws-sdk-go-v2/aws"
	awsmiddleware "github.com/aws/aws-sdk-go-v2/aws/middleware"
	"github.com/aws/aws-sdk-go-v2/service/ec2"
	"github.com/aws/aws-sdk-go-v2/service/ec2/types"
	"github.com/aws/smithy-go"
	smithymiddleware "github.com/aws/smithy-go/middleware"

	"github.com/carlosframework/platform/internal/boxid"
)

var (
	box = Box{Ref: "annie-welding", Enrollment: "enr-annie-1",
		Cloud: boxid.Cloud{Provider: "aws", Account: "111122223333", Region: "eu-west-1", InstanceID: "i-0abc"}}
	worker = Box{Ref: "bert-sawing", Enrollment: "enr-bert-1",
		Cloud: boxid.Cloud{Provider: "aws", Account: "111122223333", Region: "eu-west-1", InstanceID: "i-0bbb"}}
	now = time.Date(2026, 10, 6, 12, 0, 0, 0, time.UTC)
)

func describe(owner, instance string, state types.InstanceStateName) *ec2.DescribeInstancesOutput {
	return &ec2.DescribeInstancesOutput{Reservations: []types.Reservation{{
		OwnerId:   aws.String(owner),
		Instances: []types.Instance{{InstanceId: aws.String(instance), State: &types.InstanceState{Name: state}}},
	}}}
}

func withRequestID(out *ec2.DescribeInstancesOutput, id string) *ec2.DescribeInstancesOutput {
	var md smithymiddleware.Metadata
	awsmiddleware.SetRequestIDMetadata(&md, id)
	out.ResultMetadata = md
	return out
}

func TestATerminatedInstanceOfThisBoxIsAFence(t *testing.T) {
	out := withRequestID(describe("111122223333", "i-0abc", types.InstanceStateNameTerminated), "req-1")
	rec, err := Verify(box, worker, FromDescribe(out, "i-0abc", "eu-west-1"), now)
	if err != nil {
		t.Fatal(err)
	}
	want := Record{V: 1, Box: "annie-welding", Enrollment: "enr-annie-1", Cloud: box.Cloud, State: "terminated",
		ObservedAt: "2026-10-06T12:00:00Z", Via: "describe", RequestID: "req-1",
		WrittenBy: WrittenBy{Box: "bert-sawing", Enrollment: "enr-bert-1"}}
	if rec != want {
		t.Fatalf("record = %+v\nwant     %+v", rec, want)
	}
}

// Every lookalike is refused.
func TestLookalikesAreNotEvidence(t *testing.T) {
	cases := map[string]Observation{
		"stopped":          FromDescribe(describe("111122223333", "i-0abc", types.InstanceStateNameStopped), "i-0abc", "eu-west-1"),
		"stopping":         FromDescribe(describe("111122223333", "i-0abc", types.InstanceStateNameStopping), "i-0abc", "eu-west-1"),
		"shutting-down":    FromDescribe(describe("111122223333", "i-0abc", types.InstanceStateNameShuttingDown), "i-0abc", "eu-west-1"),
		"running":          FromDescribe(describe("111122223333", "i-0abc", types.InstanceStateNameRunning), "i-0abc", "eu-west-1"),
		"another account":  FromDescribe(describe("999999999999", "i-0abc", types.InstanceStateNameTerminated), "i-0abc", "eu-west-1"),
		"another region":   FromDescribe(describe("111122223333", "i-0abc", types.InstanceStateNameTerminated), "i-0abc", "us-east-1"),
		"another instance": FromDescribe(describe("111122223333", "i-0zzz", types.InstanceStateNameTerminated), "i-0zzz", "eu-west-1"),
		"not in the reply": FromDescribe(describe("111122223333", "i-0zzz", types.InstanceStateNameTerminated), "i-0abc", "eu-west-1"),
		"empty reply":      FromDescribe(&ec2.DescribeInstancesOutput{}, "i-0abc", "eu-west-1"),
		"nil reply":        FromDescribe(nil, "i-0abc", "eu-west-1"),
		"no state":         FromDescribe(&ec2.DescribeInstancesOutput{Reservations: []types.Reservation{{OwnerId: aws.String("111122223333"), Instances: []types.Instance{{InstanceId: aws.String("i-0abc")}}}}}, "i-0abc", "eu-west-1"),
		"unknown source":   {Via: "cloudtrail", Found: true, InstanceID: "i-0abc", Account: "111122223333", Region: "eu-west-1", State: "terminated"},
		"zero observation": {},
	}
	for name, o := range cases {
		t.Run(name, func(t *testing.T) {
			if rec, err := Verify(box, worker, o, now); !errors.Is(err, ErrNotEvidence) {
				t.Fatalf("record = %+v, err = %v; want ErrNotEvidence", rec, err)
			}
		})
	}
}

// Review Focus 5: the fence is a regional worker's own read in the box's
// own region. A worker enrolled elsewhere is refused even when what it
// read is a perfect terminated observation.
func TestAWorkerInAnotherRegionNeverFences(t *testing.T) {
	far := worker
	far.Cloud.Region = "us-east-1"
	o := FromDescribe(describe("111122223333", "i-0abc", types.InstanceStateNameTerminated), "i-0abc", "eu-west-1")
	if _, err := Verify(box, far, o, now); !errors.Is(err, ErrNotEvidence) {
		t.Fatalf("err = %v, want ErrNotEvidence", err)
	}
}

func TestAnIncompleteRosterEntryIsNeverFenced(t *testing.T) {
	// A found, terminated observation whose every coordinate is empty, so
	// it MATCHES an empty cloud field for field. Only the guard on the
	// entry itself can refuse it.
	o := Observation{Via: ViaDescribe, Found: true, State: "terminated"}
	for name, b := range map[string]Box{
		"empty cloud":   {Ref: "annie-welding", Enrollment: "enr-annie-1"},
		"no enrollment": {Ref: "annie-welding", Cloud: box.Cloud},
		"no reference":  {Enrollment: "enr-annie-1", Cloud: box.Cloud},
		"not aws":       {Ref: "annie-welding", Enrollment: "enr-annie-1", Cloud: boxid.Cloud{Provider: "gcp", Account: "111122223333", Region: "eu-west-1", InstanceID: "i-0abc"}},
	} {
		t.Run(name, func(t *testing.T) {
			full := Observation{Via: ViaDescribe, Found: true, InstanceID: b.Cloud.InstanceID, Account: b.Cloud.Account, Region: b.Cloud.Region, State: "terminated"}
			for _, obs := range []Observation{o, full} {
				if _, err := Verify(b, worker, obs, now); !errors.Is(err, ErrNotEvidence) {
					t.Fatalf("err = %v, want ErrNotEvidence", err)
				}
			}
		})
	}
	if _, err := Verify(box, Box{Ref: "bert-sawing"}, FromDescribe(describe("111122223333", "i-0abc", types.InstanceStateNameTerminated), "i-0abc", "eu-west-1"), now); !errors.Is(err, ErrNotEvidence) {
		t.Fatalf("a worker with no proven identity: err = %v", err)
	}
}

const terminatedEvent = `{"version":"0","id":"ev-123","detail-type":"EC2 Instance State-change Notification","source":"aws.ec2","account":"111122223333","time":"2026-10-06T11:59:00Z","region":"eu-west-1","resources":["arn:aws:ec2:eu-west-1:111122223333:instance/i-0abc"],"detail":{"instance-id":"i-0abc","state":"terminated"}}`

func TestATerminatedEventIsAFence(t *testing.T) {
	o, err := FromEvent([]byte(terminatedEvent))
	if err != nil {
		t.Fatal(err)
	}
	rec, err := Verify(box, worker, o, now)
	if err != nil {
		t.Fatal(err)
	}
	if rec.Via != "event" || rec.EventID != "ev-123" || rec.State != "terminated" || rec.RequestID != "" {
		t.Fatalf("record = %+v", rec)
	}
}

func TestEventsThatAreNotEC2StateChangesAreRefused(t *testing.T) {
	for name, body := range map[string]string{
		"not JSON":            `terminated`,
		"another source":      `{"id":"e","detail-type":"EC2 Instance State-change Notification","source":"my.app","account":"111122223333","region":"eu-west-1","detail":{"instance-id":"i-0abc","state":"terminated"}}`,
		"another detail-type": `{"id":"e","detail-type":"EC2 Spot Instance Interruption Warning","source":"aws.ec2","account":"111122223333","region":"eu-west-1","detail":{"instance-id":"i-0abc","state":"terminated"}}`,
		"no event id":         `{"detail-type":"EC2 Instance State-change Notification","source":"aws.ec2","account":"111122223333","region":"eu-west-1","detail":{"instance-id":"i-0abc","state":"terminated"}}`,
	} {
		t.Run(name, func(t *testing.T) {
			if _, err := FromEvent([]byte(body)); err == nil {
				t.Fatal("accepted")
			}
		})
	}
}

func TestAShuttingDownEventIsNotEvidence(t *testing.T) {
	body := `{"id":"e","detail-type":"EC2 Instance State-change Notification","source":"aws.ec2","account":"111122223333","region":"eu-west-1","detail":{"instance-id":"i-0abc","state":"shutting-down"}}`
	o, err := FromEvent([]byte(body))
	if err != nil {
		t.Fatal(err)
	}
	if _, err := Verify(box, worker, o, now); !errors.Is(err, ErrNotEvidence) {
		t.Fatalf("err = %v, want ErrNotEvidence", err)
	}
}

// fakeEC2 answers DescribeInstances and records the region each call was
// forced into.
type fakeEC2 struct {
	out     *ec2.DescribeInstancesOutput
	err     error
	regions []string
	ids     [][]string
}

func (f *fakeEC2) DescribeInstances(_ context.Context, in *ec2.DescribeInstancesInput, optFns ...func(*ec2.Options)) (*ec2.DescribeInstancesOutput, error) {
	o := ec2.Options{Region: "client-default"}
	for _, fn := range optFns {
		fn(&o)
	}
	f.regions = append(f.regions, o.Region)
	f.ids = append(f.ids, in.InstanceIds)
	return f.out, f.err
}

func TestDescribeInstanceAsksInTheBoxsOwnRegion(t *testing.T) {
	f := &fakeEC2{out: describe("111122223333", "i-0abc", types.InstanceStateNameTerminated)}
	out, err := DescribeInstance(context.Background(), f, box.Cloud)
	if err != nil {
		t.Fatal(err)
	}
	if len(f.regions) != 1 || f.regions[0] != "eu-west-1" || len(f.ids[0]) != 1 || f.ids[0][0] != "i-0abc" {
		t.Fatalf("called in %v for %v", f.regions, f.ids)
	}
	if _, err := Verify(box, worker, FromDescribe(out, "i-0abc", "eu-west-1"), now); err != nil {
		t.Fatal(err)
	}
}

func TestDescribeInstanceNotFoundIsAbsenceAndOtherErrorsAreErrors(t *testing.T) {
	notFound := &fakeEC2{err: &smithy.GenericAPIError{Code: "InvalidInstanceID.NotFound", Message: "gone"}}
	out, err := DescribeInstance(context.Background(), notFound, box.Cloud)
	if err != nil {
		t.Fatalf("NotFound: err = %v", err)
	}
	if _, err := Verify(box, worker, FromDescribe(out, "i-0abc", "eu-west-1"), now); !errors.Is(err, ErrNotEvidence) {
		t.Fatalf("NotFound fenced the box: err = %v", err)
	}
	denied := &fakeEC2{err: &smithy.GenericAPIError{Code: "UnauthorizedOperation", Message: "no"}}
	if _, err := DescribeInstance(context.Background(), denied, box.Cloud); err == nil {
		t.Fatal("an access error read as an answer")
	}
	if _, err := DescribeInstance(context.Background(), &fakeEC2{}, boxid.Cloud{InstanceID: "i-0abc"}); err == nil {
		t.Fatal("described an incomplete cloud")
	}
}
  • [ ] Step 2: Run them to see them fail

Run: go test ./internal/termfence/ -count=1 Expected: FAIL to build, undefined: Box and undefined: DescribeInstance.

  • [ ] Step 3: Write the fence

Replace internal/termfence/fence.go with:

// Package termfence decides whether an observation proves that a box's
// machine has been terminated (region HA spec 1 §5.9, invariants HA7 and
// H19).
//
// Only the state "terminated" counts, only for the instance, account and
// region the box's roster entry records under cloud, and only when the
// worker that read it is in that same region. Silence is never evidence,
// and no disk condition applies: volume records decide only whether the
// dead box's local chores are discharged (internal/boxvolumes).
//
// This package reads and decides. Writing the fence record and citing it
// in a claim CAS wait for the claim's acceptance of L2 to L5.
package termfence

import (
	"context"
	"encoding/json"
	"errors"
	"fmt"
	"time"

	"github.com/aws/aws-sdk-go-v2/aws"
	awsmiddleware "github.com/aws/aws-sdk-go-v2/aws/middleware"
	"github.com/aws/aws-sdk-go-v2/service/ec2"
	"github.com/aws/smithy-go"

	"github.com/carlosframework/platform/internal/boxid"
)

const (
	ViaDescribe = "describe"
	ViaEvent    = "event"

	stateTerminated = "terminated"
	eventSource     = "aws.ec2"
	eventDetailType = "EC2 Instance State-change Notification"
)

// ErrNotEvidence wraps every reason an observation does not fence a box.
var ErrNotEvidence = errors.New("termfence: not evidence of termination")

// Observation is what one source said about one instance. Found is false
// when the source said nothing about it: an empty reply, another instance.
type Observation struct {
	Via        string
	Found      bool
	InstanceID string
	Account    string
	Region     string
	State      string
	EventID    string
	RequestID  string
}

// Box is a roster entry as the fence needs it: the reference, the
// enrollment, and the cloud identity enrollment proved (boxid.Enroll).
type Box struct {
	Ref        string
	Enrollment string
	Cloud      boxid.Cloud
}

func (b Box) complete() bool { return b.Ref != "" && b.Enrollment != "" && b.Cloud.Complete() }

// WrittenBy names the regional worker that made the read.
type WrittenBy struct {
	Box        string `json:"box"`
	Enrollment string `json:"enrollment"`
}

// Record is §5.9's fence record. Its key (control/box-fences/<enrollment>.json)
// and its writer belong to the claim's L4 and are not built here.
type Record struct {
	V          int         `json:"v"`
	Box        string      `json:"box"`
	Enrollment string      `json:"enrollment"`
	Cloud      boxid.Cloud `json:"cloud"`
	State      string      `json:"state"`
	ObservedAt string      `json:"observed_at"`
	Via        string      `json:"via"`
	RequestID  string      `json:"request_id,omitempty"`
	EventID    string      `json:"event_id,omitempty"`
	WrittenBy  WrittenBy   `json:"written_by"`
}

// FromDescribe reads one instance out of a DescribeInstances reply. region
// is the region the call was made in: the reply does not carry it. A reply
// that does not mention instanceID yields Found false, which Verify
// refuses: AWS forgets a terminated instance within about an hour, so
// absence proves nothing.
func FromDescribe(out *ec2.DescribeInstancesOutput, instanceID, region string) Observation {
	o := Observation{Via: ViaDescribe, Region: region}
	if out == nil || instanceID == "" {
		return o
	}
	if id, ok := awsmiddleware.GetRequestIDMetadata(out.ResultMetadata); ok {
		o.RequestID = id
	}
	for _, r := range out.Reservations {
		for _, inst := range r.Instances {
			if aws.ToString(inst.InstanceId) != instanceID {
				continue
			}
			o.Found, o.InstanceID, o.Account = true, instanceID, aws.ToString(r.OwnerId)
			if inst.State != nil {
				o.State = string(inst.State.Name)
			}
			return o
		}
	}
	return o
}

// FromEvent reads an EventBridge "EC2 Instance State-change Notification".
// It refuses anything else: the queue's policy admits only the rule that
// forwards these, and this is the second check.
func FromEvent(body []byte) (Observation, error) {
	var ev struct {
		ID         string `json:"id"`
		DetailType string `json:"detail-type"`
		Source     string `json:"source"`
		Account    string `json:"account"`
		Region     string `json:"region"`
		Detail     struct {
			InstanceID string `json:"instance-id"`
			State      string `json:"state"`
		} `json:"detail"`
	}
	if err := json.Unmarshal(body, &ev); err != nil {
		return Observation{}, fmt.Errorf("termfence: event is not JSON: %w", err)
	}
	switch {
	case ev.Source != eventSource:
		return Observation{}, fmt.Errorf("termfence: event source %q is not %s", ev.Source, eventSource)
	case ev.DetailType != eventDetailType:
		return Observation{}, fmt.Errorf("termfence: event detail-type %q is not a state change", ev.DetailType)
	case ev.ID == "":
		return Observation{}, errors.New("termfence: event has no id")
	}
	return Observation{Via: ViaEvent, Found: ev.Detail.InstanceID != "", InstanceID: ev.Detail.InstanceID,
		Account: ev.Account, Region: ev.Region, State: ev.Detail.State, EventID: ev.ID}, nil
}

// EC2 is the one call this package makes. *ec2.Client satisfies it; tests
// use a fake. It is a read: the box role holds no EC2 write right.
type EC2 interface {
	DescribeInstances(ctx context.Context, in *ec2.DescribeInstancesInput, optFns ...func(*ec2.Options)) (*ec2.DescribeInstancesOutput, error)
}

// DescribeInstance asks EC2 about c.InstanceID alone, in c.Region whatever
// region the client was built for. InvalidInstanceID.NotFound comes back
// as an empty reply, which FromDescribe turns into Found false: absence,
// never termination. Every other error is returned.
func DescribeInstance(ctx context.Context, api EC2, c boxid.Cloud) (*ec2.DescribeInstancesOutput, error) {
	if !c.Complete() {
		return nil, errors.New("termfence: describe of an incomplete cloud identity")
	}
	region := c.Region
	out, err := api.DescribeInstances(ctx, &ec2.DescribeInstancesInput{InstanceIds: []string{c.InstanceID}},
		func(o *ec2.Options) { o.Region = region })
	var apiErr smithy.APIError
	if errors.As(err, &apiErr) && apiErr.ErrorCode() == "InvalidInstanceID.NotFound" {
		return &ec2.DescribeInstancesOutput{}, nil
	}
	return out, err
}

func refuse(format string, a ...any) error {
	return fmt.Errorf("%w: %s", ErrNotEvidence, fmt.Sprintf(format, a...))
}

// Verify returns box's fence record when o proves that the instance its
// roster entry records is terminated, read by worker, a regional worker of
// the same AWS region (spec §6: the fence refuses any observation not made
// in the box's own region). Whether worker is active in the roster is the
// caller's check, made before it reads.
func Verify(box, worker Box, o Observation, now time.Time) (Record, error) {
	c := box.Cloud
	switch {
	case !box.complete():
		return Record{}, refuse("the box has no proven cloud identity")
	case !worker.complete():
		return Record{}, refuse("the worker has no proven cloud identity")
	case worker.Cloud.Region != c.Region:
		return Record{}, refuse("worker %s is in %q, the box in %q", worker.Ref, worker.Cloud.Region, c.Region)
	case o.Via != ViaDescribe && o.Via != ViaEvent:
		return Record{}, refuse("source %q is not one this binary accepts", o.Via)
	case !o.Found:
		return Record{}, refuse("the source said nothing about %s; absence is not termination", c.InstanceID)
	case o.InstanceID != c.InstanceID:
		return Record{}, refuse("observed %s, the box is %s", o.InstanceID, c.InstanceID)
	case o.Account != c.Account:
		return Record{}, refuse("observed in account %q, the box is in %q", o.Account, c.Account)
	case o.Region != c.Region:
		return Record{}, refuse("observed in region %q, the box is in %q", o.Region, c.Region)
	case o.State != stateTerminated:
		return Record{}, refuse("state is %q; only %q counts", o.State, stateTerminated)
	}
	return Record{V: 1, Box: box.Ref, Enrollment: box.Enrollment, Cloud: c, State: o.State,
		ObservedAt: now.UTC().Format(time.RFC3339), Via: o.Via, RequestID: o.RequestID, EventID: o.EventID,
		WrittenBy: WrittenBy{Box: worker.Ref, Enrollment: worker.Enrollment}}, nil
}
  • [ ] Step 4: Remove boxid.Identity

Nothing uses it now. In internal/boxid/verify.go, delete:

// Identity is what an identity document proves. termfence binds to it
// until it moves to Cloud; nothing else uses it.
type Identity struct{ AccountID, Region, InstanceID string }

Then check: grep -rn "boxid.Identity" --include=*.go . prints nothing.

  • [ ] Step 5: Run the tests

Run: go test ./internal/termfence/ ./internal/boxid/ -count=1 -v Expected: PASS, including TestAWorkerInAnotherRegionNeverFences and TestDescribeInstanceNotFoundIsAbsenceAndOtherErrorsAreErrors.

  • [ ] Step 6: Gate and commit
go build ./... && go vet ./internal/termfence/ ./internal/boxid/ && gofmt -l internal/termfence internal/boxid
GOOS=darwin go build ./... && GOOS=windows go build ./...
git add internal/termfence internal/boxid/verify.go
git commit -m "termfence: a fence binds to the roster entry's cloud and is read by a worker in the box's own region

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>"

---

Task 5: Volume records and the discharge check (internal/boxvolumes)

Spec §5.9 under H19: regional workers record each enrolled box's block device mappings while it runs, create-only, on change and at least hourly; a removal or any false flag rules out automatic discharge for good. A separate function decides discharge from the records. This task builds the record, the decision to write one, the writer, the reader and the discharge check. The loop that describes every enrolled box in the region needs the roster's enrollments (L7), and the discharge CAS is L3: both wait.

Files:

  • Create: internal/boxvolumes/volumes.go
  • Test: internal/boxvolumes/volumes_test.go

Interfaces:

  • Consumes: boxid.Cloud (Task 1); release.Store. Callers get the describe reply from termfence.DescribeInstance (Task 4).
  • Produces: ``go package boxvolumes const Prefix = "control/box-volumes/" const RefreshEvery, EnrollmentWindow = time.Hour, time.Hour type Device struct{ Device, VolumeID string; DeleteOnTermination bool; Status string } type WrittenBy struct{ Box, Enrollment string } type Record struct{ V int; Box, Enrollment string; Cloud boxid.Cloud; InstanceState string; BlockDevices []Device; RemovedVolumes []string; ObservedAt, RequestID string; WrittenBy WrittenBy } // §5.9 type Entry struct{ Box, Enrollment string; Cloud boxid.Cloud } type Observation struct{ State string; Devices []Device; RequestID string } var ErrNotRecordable error func ValidEnrollment(e string) bool func Key(enrollment string, t time.Time) string func Observe(out *ec2.DescribeInstancesOutput, c boxid.Cloud) (Observation, error) func Next(prev *Record, e Entry, writer WrittenBy, o Observation, now time.Time) (Record, bool) func Write(ctx context.Context, s release.Store, rec Record) (string, error) func List(ctx context.Context, s release.Store, enrollment string) ([]Record, error) type Verdict struct{ Discharge bool; Reason string } const LastRecordWindow = 2 * time.Hour func Discharge(records []Record, e Entry, enrolledAt, terminatedAt time.Time) Verdict ``

Five choices the spec leaves to the plan.

  • A terminated read is never recorded. Its mappings are empty (the dev test of 2026-10-06), and a record of it would list every volume as removed.
  • "The first record taken at enrollment" is checked as: the earliest record for this enrollment must be observed within EnrollmentWindow (one hour, the refresh period) of the roster's enrollment time. A later first record cannot speak for a volume attached and removed before it.
  • The newest record must be no more than LastRecordWindow (2 hours) older than terminatedAt, the fence's observed_at (controller ruling 1). A recorder that stopped long before the box died cannot speak for its last hours.
  • Discharge checks every stored record of this enrollment before using it (Astra plan R1-6, R2-2): a version other than 1, a state that carries no mapping (terminated, or none), an empty mapping, or a volume with no id or no device keeps the chores open. The test has one stored case per check, each failing that check alone. List returns whatever is stored, so a record Write would never produce is still met here.
  • Discharge recomputes removals from consecutive records as well as reading removed_volumes, so a record one worker failed to write cannot hide a removal.

Keys use a fixed-width UTC time with - in place of :, so they sort in observation order.

  • [ ] Step 1: Write the failing tests

Create internal/boxvolumes/volumes_test.go:

package boxvolumes

import (
	"context"
	"errors"
	"testing"
	"time"

	"github.com/aws/aws-sdk-go-v2/aws"
	awsmiddleware "github.com/aws/aws-sdk-go-v2/aws/middleware"
	"github.com/aws/aws-sdk-go-v2/service/ec2"
	"github.com/aws/aws-sdk-go-v2/service/ec2/types"
	smithymiddleware "github.com/aws/smithy-go/middleware"

	"github.com/carlosframework/platform/internal/boxid"
	"github.com/carlosframework/platform/internal/release"
)

var (
	cloud  = boxid.Cloud{Provider: "aws", Account: "111122223333", Region: "eu-west-1", InstanceID: "i-0abc"}
	entry  = Entry{Box: "annie-welding", Enrollment: "enr-annie-1", Cloud: cloud}
	writer = WrittenBy{Box: "bert-sawing", Enrollment: "enr-bert-1"}
	t0     = time.Date(2026, 10, 6, 9, 0, 0, 0, time.UTC)
)

type vol struct {
	dev, id string
	dot     bool
}

func reply(owner string, state types.InstanceStateName, vols ...vol) *ec2.DescribeInstancesOutput {
	var maps []types.InstanceBlockDeviceMapping
	for _, v := range vols {
		maps = append(maps, types.InstanceBlockDeviceMapping{DeviceName: aws.String(v.dev),
			Ebs: &types.EbsInstanceBlockDevice{VolumeId: aws.String(v.id), DeleteOnTermination: aws.Bool(v.dot), Status: types.AttachmentStatusAttached}})
	}
	var md smithymiddleware.Metadata
	awsmiddleware.SetRequestIDMetadata(&md, "req-1")
	return &ec2.DescribeInstancesOutput{ResultMetadata: md, Reservations: []types.Reservation{{OwnerId: aws.String(owner),
		Instances: []types.Instance{{InstanceId: aws.String("i-0abc"), State: &types.InstanceState{Name: state}, BlockDeviceMappings: maps}}}}}
}

var root, data, scratch = vol{"/dev/xvda", "vol-root", true}, vol{"/dev/xvdb", "vol-data", true}, vol{"/dev/xvdc", "vol-keep", false}

func observe(t *testing.T, out *ec2.DescribeInstancesOutput) Observation {
	t.Helper()
	o, err := Observe(out, cloud)
	if err != nil {
		t.Fatal(err)
	}
	return o
}

func TestObserveReadsEBSMappingsInVolumeOrder(t *testing.T) {
	out := reply("111122223333", types.InstanceStateNameRunning, root, data)
	out.Reservations[0].Instances[0].BlockDeviceMappings = append(out.Reservations[0].Instances[0].BlockDeviceMappings,
		types.InstanceBlockDeviceMapping{DeviceName: aws.String("/dev/sdz")}) // instance store: no Ebs
	o := observe(t, out)
	want := []Device{{"/dev/xvdb", "vol-data", true, "attached"}, {"/dev/xvda", "vol-root", true, "attached"}}
	if o.State != "running" || o.RequestID != "req-1" || len(o.Devices) != 2 || o.Devices[0] != want[0] || o.Devices[1] != want[1] {
		t.Fatalf("observation = %+v", o)
	}
}

// Review Focus 3: a terminated instance's reply has no mappings. Recorded,
// it would claim every volume was removed; it must not be recordable.
func TestObserveRefusesWhatIsNotTheInstancesMapping(t *testing.T) {
	for name, out := range map[string]*ec2.DescribeInstancesOutput{
		"terminated":       reply("111122223333", types.InstanceStateNameTerminated),
		"another owner":    reply("999999999999", types.InstanceStateNameRunning, root),
		"another instance": {Reservations: []types.Reservation{{OwnerId: aws.String("111122223333"), Instances: []types.Instance{{InstanceId: aws.String("i-0zzz")}}}}},
		"empty reply":      {},
		"nil reply":        nil,
	} {
		t.Run(name, func(t *testing.T) {
			if o, err := Observe(out, cloud); !errors.Is(err, ErrNotRecordable) {
				t.Fatalf("observation = %+v, err = %v; want ErrNotRecordable", o, err)
			}
		})
	}
	// A shutting-down read still lists the mappings and may be recorded.
	if o := observe(t, reply("111122223333", types.InstanceStateNameShuttingDown, root)); len(o.Devices) != 1 {
		t.Fatalf("shutting-down: %+v", o)
	}
}

func TestNextWritesOnChangeAndAtLeastHourly(t *testing.T) {
	first, ok := Next(nil, entry, writer, observe(t, reply("111122223333", types.InstanceStateNameRunning, root)), t0)
	if !ok || first.ObservedAt != "2026-10-06T09:00:00Z" || len(first.RemovedVolumes) != 0 || first.RemovedVolumes == nil {
		t.Fatalf("first = %+v ok=%v", first, ok)
	}
	same := observe(t, reply("111122223333", types.InstanceStateNameRunning, root))
	if _, ok := Next(&first, entry, writer, same, t0.Add(59*time.Minute)); ok {
		t.Fatal("wrote an unchanged mapping inside the hour")
	}
	if _, ok := Next(&first, entry, writer, same, t0.Add(time.Hour)); !ok {
		t.Fatal("no record after an hour")
	}
	grown, ok := Next(&first, entry, writer, observe(t, reply("111122223333", types.InstanceStateNameRunning, root, scratch)), t0.Add(time.Minute))
	if !ok || len(grown.BlockDevices) != 2 {
		t.Fatalf("an attached volume was not recorded: %+v", grown)
	}
	shrunk, ok := Next(&grown, entry, writer, observe(t, reply("111122223333", types.InstanceStateNameRunning, root)), t0.Add(2*time.Minute))
	if !ok || len(shrunk.RemovedVolumes) != 1 || shrunk.RemovedVolumes[0] != "vol-keep" {
		t.Fatalf("a detached volume is not in removed_volumes: %+v", shrunk)
	}
	if _, ok := Next(&first, entry, writer, observe(t, reply("111122223333", types.InstanceStateNameShuttingDown, root)), t0.Add(time.Minute)); !ok {
		t.Fatal("a state change was not recorded")
	}
}

func TestWriteIsCreateOnlyAndListReturnsObservationOrder(t *testing.T) {
	ctx, st := context.Background(), release.NewDirStore(t.TempDir())
	a, _ := Next(nil, entry, writer, observe(t, reply("111122223333", types.InstanceStateNameRunning, root)), t0)
	b, _ := Next(&a, entry, writer, observe(t, reply("111122223333", types.InstanceStateNameRunning, root, data)), t0.Add(10*time.Minute))
	for _, r := range []Record{b, a} {
		if _, err := Write(ctx, st, r); err != nil {
			t.Fatal(err)
		}
	}
	if _, err := Write(ctx, st, a); err != nil {
		t.Fatalf("the same record twice: %v", err)
	}
	other := a
	other.RequestID = "req-other"
	if _, err := Write(ctx, st, other); !errors.Is(err, release.ErrCASConflict) {
		t.Fatalf("a different record under the same key: err = %v", err)
	}
	got, err := List(ctx, st, entry.Enrollment)
	if err != nil {
		t.Fatal(err)
	}
	if len(got) != 2 || got[0].ObservedAt != a.ObservedAt || got[1].ObservedAt != b.ObservedAt {
		t.Fatalf("list = %+v", got)
	}
	if _, err := Write(ctx, st, Record{Enrollment: "../edges", ObservedAt: a.ObservedAt}); err == nil {
		t.Fatal("wrote outside the prefix")
	}
}

func TestListFailsOnARecordItCannotRead(t *testing.T) {
	ctx, st := context.Background(), release.NewDirStore(t.TempDir())
	if _, err := st.Put(ctx, Key(entry.Enrollment, t0), []byte(`{"v":1,"surprise":true}`), ""); err != nil {
		t.Fatal(err)
	}
	if _, err := List(ctx, st, entry.Enrollment); err == nil {
		t.Fatal("an unreadable record read as a gap")
	}
}

func records(t *testing.T, steps ...[]vol) []Record {
	t.Helper()
	var out []Record
	var prev *Record
	for i, vs := range steps {
		rec, _ := Next(prev, entry, writer, observe(t, reply("111122223333", types.InstanceStateNameRunning, vs...)), t0.Add(time.Duration(i)*10*time.Minute))
		out = append(out, rec)
		prev = &out[len(out)-1]
	}
	return out
}

func TestDischarge(t *testing.T) {
	enrolled := t0.Add(-5 * time.Minute)
	terminated := t0.Add(30 * time.Minute)
	if v := Discharge(records(t, []vol{root}, []vol{root, data}, []vol{root, data}), entry, enrolled, terminated); !v.Discharge {
		t.Fatalf("all true, nothing removed: %+v", v)
	}
	cases := map[string]struct {
		recs []Record
		at   time.Time
	}{
		"no record at all":          {nil, enrolled},
		"a false flag":              {records(t, []vol{root}, []vol{root, scratch}), enrolled},
		"a removed volume":          {records(t, []vol{root, data}, []vol{root}), enrolled},
		"a replaced root volume":    {records(t, []vol{root}, []vol{{"/dev/xvda", "vol-root2", true}}), enrolled},
		"first record long after":   {records(t, []vol{root}), t0.Add(-2 * time.Hour)},
		"only another enrollment's": {func() []Record { r := records(t, []vol{root}); r[0].Enrollment = "enr-other"; return r }(), enrolled},
		"only another instance's":   {func() []Record { r := records(t, []vol{root}); r[0].Cloud.InstanceID = "i-0zzz"; return r }(), enrolled},
	}
	for name, tc := range cases {
		t.Run(name, func(t *testing.T) {
			if v := Discharge(tc.recs, entry, tc.at, terminated); v.Discharge || v.Reason == "" {
				t.Fatalf("verdict = %+v, want the chores left open with a reason", v)
			}
		})
	}
}

// A removal is caught from the sequence even when the record that should
// have listed it was never written (two workers, one missed write).
func TestDischargeSeesARemovalNoRecordListed(t *testing.T) {
	recs := records(t, []vol{root, data}, []vol{root})
	recs[1].RemovedVolumes = []string{}
	if v := Discharge(recs, entry, t0, t0.Add(time.Hour)); v.Discharge {
		t.Fatal("discharged across a silent removal")
	}
}

// Controller ruling, 2026-10-06: the newest record must be no more than
// LastRecordWindow older than the termination observation. A recorder that
// stopped long before the box died cannot speak for its last hours.
func TestDischargeNeedsARecentLastRecord(t *testing.T) {
	recs := records(t, []vol{root}, []vol{root}) // newest at t0+10m
	newest := t0.Add(10 * time.Minute)
	if v := Discharge(recs, entry, t0, newest.Add(LastRecordWindow)); !v.Discharge {
		t.Fatalf("exactly at the window: %+v", v)
	}
	if v := Discharge(recs, entry, t0, newest.Add(LastRecordWindow+time.Second)); v.Discharge || v.Reason == "" {
		t.Fatalf("a newest record older than the window discharged: %+v", v)
	}
}

// Astra plan R1-6: what List returns is whatever is stored. A stored record
// that is not evidence of a running instance's mappings keeps the chores
// open, however it got there. Each bad record sits beside a good one so
// that no removal is seen between them: only the record's own check can
// refuse it.
func TestDischargeRefusesStoredRecordsThatAreNotEvidence(t *testing.T) {
	ctx := context.Background()
	good := records(t, []vol{root})
	const before = `"enrollment":"enr-annie-1","cloud":{"provider":"aws","account":"111122223333","region":"eu-west-1","instance_id":"i-0abc"},"observed_at":"2026-10-06T08:55:00Z"`
	const after = `"enrollment":"enr-annie-1","cloud":{"provider":"aws","account":"111122223333","region":"eu-west-1","instance_id":"i-0abc"},"observed_at":"2026-10-06T09:05:00Z"`
	type bad struct {
		body string
		at   time.Duration // from t0; the good record is at t0
	}
	cases := map[string]bad{
		// Before the good record: an empty mapping followed by one volume is
		// an attach, not a removal.
		"only the binding fields": {`{"v":1,` + before + `}`, -5 * time.Minute},
		"a terminated read":       {`{"v":1,` + before + `,"instance_state":"terminated","block_devices":[],"removed_volumes":[]}`, -5 * time.Minute},
		// After it, keeping vol-root, so nothing is removed.
		"a volume with no id": {`{"v":1,` + after + `,"instance_state":"running","block_devices":[{"device":"/dev/xvda","volume_id":"vol-root","delete_on_termination":true,"status":"attached"},{"device":"/dev/xvdb","volume_id":"","delete_on_termination":true,"status":"attached"}],"removed_volumes":[]}`, 5 * time.Minute},
		"a version 2 record":  {`{"v":2,` + after + `,"instance_state":"running","block_devices":[{"device":"/dev/xvda","volume_id":"vol-root","delete_on_termination":true,"status":"attached"}],"removed_volumes":[]}`, 5 * time.Minute},
		// Astra plan R2-2: one case per check, each failing only that check.
		"a running read with no volume":           {`{"v":1,` + before + `,"instance_state":"running","block_devices":[],"removed_volumes":[]}`, -5 * time.Minute},
		"a terminated read with its root mapping": {`{"v":1,` + after + `,"instance_state":"terminated","block_devices":[{"device":"/dev/xvda","volume_id":"vol-root","delete_on_termination":true,"status":"attached"}],"removed_volumes":[]}`, 5 * time.Minute},
		"a volume with no device":                 {`{"v":1,` + after + `,"instance_state":"running","block_devices":[{"device":"/dev/xvda","volume_id":"vol-root","delete_on_termination":true,"status":"attached"},{"device":"","volume_id":"vol-data","delete_on_termination":true,"status":"attached"}],"removed_volumes":[]}`, 5 * time.Minute},
	}
	for name, tc := range cases {
		t.Run(name, func(t *testing.T) {
			st := release.NewDirStore(t.TempDir())
			if _, err := Write(ctx, st, good[0]); err != nil {
				t.Fatal(err)
			}
			if _, err := st.Put(ctx, Key(entry.Enrollment, t0.Add(tc.at)), []byte(tc.body), ""); err != nil {
				t.Fatal(err)
			}
			recs, err := List(ctx, st, entry.Enrollment)
			if err != nil {
				t.Fatal(err)
			}
			if len(recs) != 2 {
				t.Fatalf("listed %d records, want 2", len(recs))
			}
			if v := Discharge(recs, entry, t0, t0.Add(time.Hour)); v.Discharge || v.Reason == "" {
				t.Fatalf("verdict = %+v, want the chores left open", v)
			}
		})
	}
	// The good record alone, stored and listed, still discharges.
	st := release.NewDirStore(t.TempDir())
	if _, err := Write(ctx, st, good[0]); err != nil {
		t.Fatal(err)
	}
	recs, _ := List(ctx, st, entry.Enrollment)
	if v := Discharge(recs, entry, t0, t0.Add(time.Hour)); !v.Discharge {
		t.Fatalf("a stored good record: %+v", v)
	}
}
  • [ ] Step 2: Run them to see them fail

Run: go test ./internal/boxvolumes/ -count=1 Expected: FAIL, no non-test Go files or undefined: Observe.

  • [ ] Step 3: Write the package

Create internal/boxvolumes/volumes.go:

// Package boxvolumes records a platform box's block device mappings while
// it runs (region HA spec 1 §5.9, ruling H19), and decides from those
// records whether a terminated box's park-local chores may be discharged
// without retirement (claim D5's destroyed-local exception, L3).
//
// Nothing here is a termination fence, and nothing about execution,
// admission, relocation or recovery reads it (spec §8, B4-1). A record that
// is wrong or missing can leave a disk unretired for an operator; it can
// never admit a writer.
//
// This package builds, writes and reads records. The loop that describes
// every enrolled box in the region needs the roster's enrollments, and the
// discharge CAS needs L3: both wait for the claim.
package boxvolumes

import (
	"bytes"
	"context"
	"encoding/json"
	"errors"
	"fmt"
	"io"
	"slices"
	"sort"
	"strings"
	"time"

	"github.com/aws/aws-sdk-go-v2/aws"
	awsmiddleware "github.com/aws/aws-sdk-go-v2/aws/middleware"
	"github.com/aws/aws-sdk-go-v2/service/ec2"

	"github.com/carlosframework/platform/internal/boxid"
	"github.com/carlosframework/platform/internal/release"
)

const (
	Prefix = "control/box-volumes/"

	// RefreshEvery is the longest a running box goes without a new record
	// when nothing changes (spec §5.9: "at least once an hour").
	RefreshEvery = time.Hour
	// EnrollmentWindow is how soon after enrollment the first record must
	// be taken for it to stand for the launch-time mappings.
	EnrollmentWindow = time.Hour
	// LastRecordWindow is the most the newest record may predate the
	// termination observation (controller ruling, 2026-10-06). An older one
	// cannot speak for a volume attached after it.
	LastRecordWindow = 2 * time.Hour

	keyTime = "2006-01-02T15-04-05.000000000Z"
)

// Device is one EBS volume in the instance's block device mappings.
type Device struct {
	Device              string `json:"device"`
	VolumeID            string `json:"volume_id"`
	DeleteOnTermination bool   `json:"delete_on_termination"`
	Status              string `json:"status"`
}

// WrittenBy names the regional worker that made the read.
type WrittenBy struct {
	Box        string `json:"box"`
	Enrollment string `json:"enrollment"`
}

// Record is §5.9's volume record,
// control/box-volumes/<enrollment>/<observed_at>.json.
type Record struct {
	V              int         `json:"v"`
	Box            string      `json:"box"`
	Enrollment     string      `json:"enrollment"`
	Cloud          boxid.Cloud `json:"cloud"`
	InstanceState  string      `json:"instance_state"`
	BlockDevices   []Device    `json:"block_devices"`
	RemovedVolumes []string    `json:"removed_volumes"`
	ObservedAt     string      `json:"observed_at"`
	RequestID      string      `json:"request_id"`
	WrittenBy      WrittenBy   `json:"written_by"`
}

// Entry is the roster entry a record describes.
type Entry struct {
	Box        string
	Enrollment string
	Cloud      boxid.Cloud
}

// Observation is one DescribeInstances read of one enrolled box.
type Observation struct {
	State     string
	Devices   []Device
	RequestID string
}

var ErrNotRecordable = errors.New("boxvolumes: not a recordable observation")

// ValidEnrollment reports whether e can be a key segment: 1 to 128 of
// [A-Za-z0-9._-], never "." or "..".
func ValidEnrollment(e string) bool {
	if e == "" || len(e) > 128 || e == "." || e == ".." {
		return false
	}
	for _, r := range e {
		ok := r >= 'a' && r <= 'z' || r >= 'A' && r <= 'Z' || r >= '0' && r <= '9' || r == '.' || r == '_' || r == '-'
		if !ok {
			return false
		}
	}
	return true
}

// Key is where a record observed at t lives. The time is fixed-width UTC,
// so keys sort in observation order.
func Key(enrollment string, t time.Time) string {
	return Prefix + enrollment + "/" + t.UTC().Format(keyTime) + ".json"
}

// recordable is every state in which EC2 still reports the instance's
// mappings. "terminated" is not one: its BlockDeviceMappings is empty (the
// dev test of 2026-10-06), which would read as every volume removed.
var recordable = map[string]bool{"pending": true, "running": true, "shutting-down": true, "stopping": true, "stopped": true}

// Observe reads the entry's instance out of a DescribeInstances reply (see
// termfence.DescribeInstance for the call). It refuses a reply that does
// not name the instance, names it under another owner, or reports a state
// whose mappings are not the instance's.
func Observe(out *ec2.DescribeInstancesOutput, c boxid.Cloud) (Observation, error) {
	if out == nil || !c.Complete() {
		return Observation{}, fmt.Errorf("%w: no reply, or no proven cloud", ErrNotRecordable)
	}
	var o Observation
	if id, ok := awsmiddleware.GetRequestIDMetadata(out.ResultMetadata); ok {
		o.RequestID = id
	}
	for _, r := range out.Reservations {
		for _, inst := range r.Instances {
			if aws.ToString(inst.InstanceId) != c.InstanceID {
				continue
			}
			if aws.ToString(r.OwnerId) != c.Account {
				return Observation{}, fmt.Errorf("%w: %s is owned by %q, the roster says %q", ErrNotRecordable, c.InstanceID, aws.ToString(r.OwnerId), c.Account)
			}
			if inst.State != nil {
				o.State = string(inst.State.Name)
			}
			if !recordable[o.State] {
				return Observation{}, fmt.Errorf("%w: state %q", ErrNotRecordable, o.State)
			}
			for _, m := range inst.BlockDeviceMappings {
				if m.Ebs == nil {
					continue // instance store dies with the instance
				}
				o.Devices = append(o.Devices, Device{Device: aws.ToString(m.DeviceName), VolumeID: aws.ToString(m.Ebs.VolumeId),
					DeleteOnTermination: aws.ToBool(m.Ebs.DeleteOnTermination), Status: string(m.Ebs.Status)})
			}
			sort.Slice(o.Devices, func(i, j int) bool { return o.Devices[i].VolumeID < o.Devices[j].VolumeID })
			return o, nil
		}
	}
	return Observation{}, fmt.Errorf("%w: the reply does not name %s", ErrNotRecordable, c.InstanceID)
}

func volumeIDs(ds []Device) []string {
	out := make([]string, 0, len(ds))
	for _, d := range ds {
		out = append(out, d.VolumeID)
	}
	return out
}

func removed(prev, cur []Device) []string {
	have := volumeIDs(cur)
	var out []string
	for _, id := range volumeIDs(prev) {
		if !slices.Contains(have, id) {
			out = append(out, id)
		}
	}
	sort.Strings(out)
	return out
}

// Next decides whether o, made by writer at now, is a new record after
// prev (nil when the enrollment has none). It is one when there is no
// record yet, when the mapping or the instance state differs from prev,
// and otherwise once RefreshEvery has passed.
func Next(prev *Record, e Entry, writer WrittenBy, o Observation, now time.Time) (Record, bool) {
	rec := Record{V: 1, Box: e.Box, Enrollment: e.Enrollment, Cloud: e.Cloud, InstanceState: o.State,
		BlockDevices: o.Devices, RemovedVolumes: []string{}, ObservedAt: now.UTC().Format(time.RFC3339Nano),
		RequestID: o.RequestID, WrittenBy: writer}
	if rec.BlockDevices == nil {
		rec.BlockDevices = []Device{}
	}
	if prev == nil {
		return rec, true
	}
	if r := removed(prev.BlockDevices, o.Devices); len(r) > 0 {
		rec.RemovedVolumes = r
	}
	if !slices.Equal(prev.BlockDevices, rec.BlockDevices) || prev.InstanceState != o.State {
		return rec, true
	}
	at, err := time.Parse(time.RFC3339Nano, prev.ObservedAt)
	return rec, err != nil || now.Sub(at) >= RefreshEvery
}

// Write stores rec create-only at its key. Records are never replaced or
// deleted (spec §6); the same bytes written twice are one record.
func Write(ctx context.Context, s release.Store, rec Record) (string, error) {
	if !ValidEnrollment(rec.Enrollment) {
		return "", fmt.Errorf("boxvolumes: %q is not an enrollment", rec.Enrollment)
	}
	at, err := time.Parse(time.RFC3339Nano, rec.ObservedAt)
	if err != nil {
		return "", fmt.Errorf("boxvolumes: observed_at: %w", err)
	}
	body, err := json.Marshal(rec)
	if err != nil {
		return "", err
	}
	key := Key(rec.Enrollment, at)
	_, err = s.Put(ctx, key, body, "")
	if errors.Is(err, release.ErrCASConflict) {
		cur, _, gerr := s.Get(ctx, key)
		if gerr == nil && bytes.Equal(cur, body) {
			return key, nil
		}
		return "", fmt.Errorf("boxvolumes: %s already holds another record: %w", key, err)
	}
	return key, err
}

func decode(body []byte) (Record, error) {
	dec := json.NewDecoder(bytes.NewReader(body))
	dec.DisallowUnknownFields()
	var rec Record
	if err := dec.Decode(&rec); err != nil {
		return Record{}, err
	}
	if _, err := dec.Token(); err != io.EOF {
		return Record{}, errors.New("trailing data")
	}
	return rec, nil
}

// List returns every record stored for an enrollment, in key order. Any
// record that cannot be read or parsed is an error, never a gap: a gap
// could hide the record that shows a removal.
func List(ctx context.Context, s release.Store, enrollment string) ([]Record, error) {
	if !ValidEnrollment(enrollment) {
		return nil, fmt.Errorf("boxvolumes: %q is not an enrollment", enrollment)
	}
	keys, err := s.List(ctx, Prefix+enrollment+"/")
	if err != nil {
		return nil, err
	}
	sort.Strings(keys)
	out := make([]Record, 0, len(keys))
	for _, k := range keys {
		if !strings.HasSuffix(k, ".json") {
			continue
		}
		body, _, err := s.Get(ctx, k)
		if err != nil {
			return nil, fmt.Errorf("boxvolumes: %s: %w", k, err)
		}
		rec, err := decode(body)
		if err != nil {
			return nil, fmt.Errorf("boxvolumes: %s: %w", k, err)
		}
		out = append(out, rec)
	}
	return out, nil
}

// incomplete says why a stored record is not evidence of a running
// instance's mappings, or "" when it is. Write never produces one of
// these; a record that is one anyway (an older writer, a hand edit, a
// bug) keeps the chores open rather than being read as "nothing attached".
func incomplete(r Record) string {
	if !recordable[r.InstanceState] {
		return fmt.Sprintf("has state %q, which carries no mapping", r.InstanceState)
	}
	if len(r.BlockDevices) == 0 {
		return "lists no EBS volume" // a platform box always has its root volume
	}
	for _, d := range r.BlockDevices {
		if d.VolumeID == "" || d.Device == "" {
			return "lists a volume with no id or device"
		}
	}
	return ""
}

// Verdict is whether a terminated box's park-local chores may be
// discharged without retirement, and if not, why.
type Verdict struct {
	Discharge bool
	Reason    string
}

// Discharge decides from records (as List returns them) whether every EBS
// volume e's instance had attached, from the first record taken within
// EnrollmentWindow of enrolledAt to the last, taken within LastRecordWindow
// of terminatedAt (the fence's observed_at), was DeleteOnTermination, with
// none ever removed (spec §5.9, H19). Records for another enrollment or
// another instance are ignored; a record of this enrollment's that is
// incomplete, carries no mapping, or has a version this binary does not
// read keeps the chores open. Anything short of that keeps the chores
// open for an operator, which costs cleanup and never recovery.
func Discharge(records []Record, e Entry, enrolledAt, terminatedAt time.Time) Verdict {
	no := func(format string, a ...any) Verdict { return Verdict{Reason: fmt.Sprintf(format, a...)} }
	if !e.Cloud.Complete() || e.Enrollment == "" {
		return no("the roster entry has no proven cloud")
	}
	type stamped struct {
		at  time.Time
		rec Record
	}
	var mine []stamped
	for _, r := range records {
		if r.Enrollment != e.Enrollment {
			continue
		}
		// A version this binary does not read could hold the adverse
		// record; it is never skipped.
		if r.V != 1 {
			return no("a record for %s has version %d", e.Enrollment, r.V)
		}
		if r.Cloud != e.Cloud {
			continue // another instance's (spec §5.9: ignored)
		}
		if reason := incomplete(r); reason != "" {
			return no("the record observed at %q %s", r.ObservedAt, reason)
		}
		at, err := time.Parse(time.RFC3339Nano, r.ObservedAt)
		if err != nil {
			return no("record with unreadable observed_at %q", r.ObservedAt)
		}
		mine = append(mine, stamped{at, r})
	}
	if len(mine) == 0 {
		return no("no volume record for %s", e.Enrollment)
	}
	sort.SliceStable(mine, func(i, j int) bool { return mine[i].at.Before(mine[j].at) })
	if first := mine[0].at; first.After(enrolledAt.Add(EnrollmentWindow)) {
		return no("the first record (%s) is more than %s after enrollment", first.Format(time.RFC3339), EnrollmentWindow)
	}
	if last := mine[len(mine)-1].at; terminatedAt.Sub(last) > LastRecordWindow {
		return no("the newest record (%s) is more than %s before the termination", last.Format(time.RFC3339), LastRecordWindow)
	}
	for i, s := range mine {
		for _, d := range s.rec.BlockDevices {
			if !d.DeleteOnTermination {
				return no("volume %s was not DeleteOnTermination at %s", d.VolumeID, s.rec.ObservedAt)
			}
		}
		if len(s.rec.RemovedVolumes) > 0 {
			return no("volumes %v were removed by %s", s.rec.RemovedVolumes, s.rec.ObservedAt)
		}
		if i > 0 {
			if r := removed(mine[i-1].rec.BlockDevices, s.rec.BlockDevices); len(r) > 0 {
				return no("volumes %v left the mapping by %s", r, s.rec.ObservedAt)
			}
		}
	}
	return Verdict{Discharge: true}
}
  • [ ] Step 4: Run the tests

Run: go test ./internal/boxvolumes/ -count=1 -v Expected: PASS, including TestObserveRefusesWhatIsNotTheInstancesMapping/terminated, TestDischargeSeesARemovalNoRecordListed, TestDischargeNeedsARecentLastRecord and TestDischargeRefusesStoredRecordsThatAreNotEvidence.

  • [ ] Step 5: Gate and commit
go build ./... && go vet ./internal/boxvolumes/ && gofmt -l internal/boxvolumes
GOOS=darwin go build ./... && GOOS=windows go build ./...
git add internal/boxvolumes
git commit -m "boxvolumes: a running box's volumes are recorded create-only, and only an unbroken all-true record discharges its chores

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>"

---

Task 6: The edges file follows the claim's executor (internal/host/edgesclaims.go)

Spec §5.4: "The edges-file pass publishes a host as home only when this box is its executor and its row is ready, and adds the installation's stamp." Plan 1a built the edge's side (foldHomes believes the higher stamp). This task builds the publisher's side, with CARLOS_REGION_HA on.

Decisions.

  • Which rows. Only relocatable hosts (spec §2): a provisioned instance row that the activator runs as its own exec child (Backing == "", no Unit, no Addr/Addrs, a Socket). Unit-, addr- and socket-only routes, static sites and services keep their box and are published as before. Sidecars are never published (unchanged).
  • The decision. A claim whose op.exec is this box's reference: home, stamped with holder.stamp. A claim whose op.exec names another box, or none, or which has no holder: withheld. No claim: home, unstamped, exactly as before; §8's gate admits a second box only once every host has a claim.
  • Errors. A read that fails keeps the last decision (claim C6: an error is never absence); a host never read publishes as before.
  • Cost. Each read stands 10 minutes (H9), at most 256 reads a pass, never-attempted hosts first, then oldest attempt first. A failed read counts as an attempt, so a claim that keeps failing goes to the back of the queue and cannot starve the rest (Astra plan R1-4). About 288 thousand GETs a day per 2,000 hosts, under $0.12.
  • Refresh. The registry timer alone runs a pass every five to six minutes, which at 256 reads a pass would take 40 minutes to cover 2,000 hosts (Astra plan R1-5). So, with the flag on, the loop also runs a pass every 30 seconds while any claim read is due (edgesLoopState). Every claim is then re-read within about 14 minutes; the edges file is still written only when its body changes. A claim pass is a publication like any other (Astra plan R2-1): it waits out the schedule's retry backoff after a failed write, and its own outcome goes through edgesFileSchedule.done, so a bucket that refuses every write is asked on the existing 2-second-to-5-minute ladder, not every 30 seconds. With the flag off, edgesLoopState is exactly edgesFileSchedule.
  • Aliases and catch-alls carry their primary's stamp (Astra plan R1-2). An attached domain and a served *.<parent> catch-all are dialled through the primary's installation. Unstamped, a dead predecessor's alias would tie with the successor's at 0 and the edge (dialBeats) would keep the box that sorts first.
  • §18's open question. A positive stamp is the mark: only relocatable hosts with a claim carry one, so the edge's report of two boxes at one positive stamp already concerns them alone. No separate marker field.

The withholding and stamping are applied before owners and aliases are computed, so an alias of a withheld host is withheld with it, and an alias of a stamped host reads its stamp from the map.

Files:

  • Create: internal/host/edgesclaims.go
  • Test: internal/host/edgesclaims_test.go, internal/edge/homemap_alias_stamp_test.go
  • Modify: internal/host/edgesfile.go

Interfaces:

  • Consumes: hostclaim.Read, hostclaim.Observed, hostclaim.Claim (main); regionHAEnabled (Task 3); edgeHost.Stamp (plan 1a).
  • Produces: ``go const edgesClaimTTL = 10 * time.Minute const edgesClaimReadsPerPass = 256 const edgesClaimPassEvery = 30 * time.Second type edgesClaims struct{ /* self, read, now, entries, hosts */ } func (c *edgesClaims) backlog(now time.Time) bool type edgesLoopState struct{ sched *edgesFileSchedule; claims *edgesClaims; claimsNext time.Time; schedDue, claimDue bool } func (s *edgesLoopState) due(now time.Time, fp string, fpOK bool) bool func (s *edgesLoopState) timed() bool func (s *edgesLoopState) done(now time.Time, ok bool) func newEdgesClaims(self string, read func(context.Context, string) (hostclaim.Observed, error)) *edgesClaims func relocatableRow(r registry.Route) bool func applyEdgesClaims(ctx context.Context, claims *edgesClaims, routes []registry.Route, hosts map[string]edgeHost) func edgesFileWriteClaims(ctx context.Context, s release.Store, reg *registry.Registry, ident edgesFile, eip string, last *[]byte, refresh bool, claims *edgesClaims) error ` edgesFileWrite keeps its signature and calls edgesFileWriteClaims with nil`.
  • [ ] Step 1: Write the failing tests

Create internal/host/edgesclaims_test.go:

package host

import (
	"context"
	"encoding/json"
	"errors"
	"fmt"
	"testing"
	"time"

	"github.com/carlosframework/platform/internal/hostclaim"
	"github.com/carlosframework/platform/internal/registry"
	"github.com/carlosframework/platform/internal/release"
)

// seedInstance upserts a ready, provisioned exec instance: a relocatable
// host (spec 1 §2).
func seedInstance(t *testing.T, reg *registry.Registry, host string) {
	t.Helper()
	if err := reg.Upsert(registry.Route{Host: host, Kind: "instance", Socket: "/run/carlos/" + host + ".sock",
		Provisioned: true, App: "jam", Account: "bab", Status: "ready"}); err != nil {
		t.Fatal(err)
	}
}

// claimsBook answers claim reads from a map; a missing host is an absent
// claim, a host in errs fails.
type claimsBook struct {
	claims map[string]*hostclaim.Claim
	errs   map[string]bool
	reads  []string
}

func (b *claimsBook) read(_ context.Context, host string) (hostclaim.Observed, error) {
	b.reads = append(b.reads, host)
	if b.errs[host] {
		return hostclaim.Observed{}, errors.New("SlowDown")
	}
	c, ok := b.claims[host]
	if !ok {
		return hostclaim.Observed{Host: host, Absent: true}, nil
	}
	return hostclaim.Observed{Host: host, Claim: c, ETag: "e"}, nil
}

func serving(exec string, stamp uint64) *hostclaim.Claim {
	return &hostclaim.Claim{V: 2, Gen: 7, Holder: &hostclaim.Holder{Stamp: stamp},
		Op: hostclaim.Op{Kind: hostclaim.KindServing, ID: "op-1", Exec: exec}}
}

func publishWithClaims(t *testing.T, reg *registry.Registry, claims *edgesClaims) map[string]edgeHost {
	t.Helper()
	st := release.NewDirStore(t.TempDir())
	var last []byte
	if err := edgesFileWriteClaims(context.Background(), st, reg, edgesFile{Box: "annie", BoxID: "annie"}, "1.2.3.4", &last, false, claims); err != nil {
		t.Fatal(err)
	}
	body, _, err := st.Get(context.Background(), edgesFileKey("annie"))
	if err != nil {
		t.Fatal(err)
	}
	var f edgesFile
	if err := json.Unmarshal(body, &f); err != nil {
		t.Fatal(err)
	}
	return f.Hosts
}

func TestEdgesFileFollowsTheClaimsExecutor(t *testing.T) {
	reg := edgesTestRegistry(t)
	for _, h := range []string{"mine.bab.test", "theirs.bab.test", "between.bab.test", "unclaimed.bab.test"} {
		seedInstance(t, reg, h)
	}
	seedRoute(t, reg, "static.example.com") // not relocatable: never read, always home
	book := &claimsBook{claims: map[string]*hostclaim.Claim{
		"mine.bab.test":    serving("annie", 41),
		"theirs.bab.test":  serving("bert", 42),
		"between.bab.test": serving("", 43), // op.exec none: nobody serves it
	}}
	hosts := publishWithClaims(t, reg, newEdgesClaims("annie", book.read))
	if h, ok := hosts["mine.bab.test"]; !ok || h.Stamp != 41 || h.Role != "home" {
		t.Fatalf("executor's host = %+v, %v; want home stamped 41", h, ok)
	}
	for _, h := range []string{"theirs.bab.test", "between.bab.test"} {
		if _, ok := hosts[h]; ok {
			t.Errorf("%s published by a box its claim does not name", h)
		}
	}
	if h, ok := hosts["unclaimed.bab.test"]; !ok || h.Stamp != 0 {
		t.Errorf("a host with no claim = %+v, %v; want home, unstamped, as before", h, ok)
	}
	if h, ok := hosts["static.example.com"]; !ok || h.Stamp != 0 {
		t.Errorf("a service route = %+v, %v; want home, unstamped", h, ok)
	}
	for _, r := range book.reads {
		if r == "static.example.com" {
			t.Error("read a claim for a route that is not relocatable")
		}
	}
}

func TestEdgesFileWithholdsTheAliasOfAWithheldHost(t *testing.T) {
	reg := edgesTestRegistry(t)
	seedInstance(t, reg, "theirs.bab.test")
	if err := reg.UpsertAlias(registry.RouteAlias{Alias: "www.customer.com", Host: "theirs.bab.test", Account: "bab", App: "jam", DNSStatus: "pointing"}); err != nil {
		t.Fatal(err)
	}
	book := &claimsBook{claims: map[string]*hostclaim.Claim{"theirs.bab.test": serving("bert", 42)}}
	hosts := publishWithClaims(t, reg, newEdgesClaims("annie", book.read))
	if len(hosts) != 0 {
		t.Fatalf("hosts = %v, want none", hosts)
	}
}

// Review Focus 2: a claim that cannot be read never moves a host. It keeps
// the last decision, or, never read, publishes as it did before claims.
func TestEdgesFileKeepsTheLastDecisionWhenAClaimCannotBeRead(t *testing.T) {
	reg := edgesTestRegistry(t)
	seedInstance(t, reg, "mine.bab.test")
	seedInstance(t, reg, "theirs.bab.test")
	book := &claimsBook{claims: map[string]*hostclaim.Claim{"mine.bab.test": serving("annie", 41), "theirs.bab.test": serving("bert", 42)}}
	clock := time.Date(2026, 10, 6, 9, 0, 0, 0, time.UTC)
	claims := newEdgesClaims("annie", book.read)
	claims.now = func() time.Time { return clock }
	publishWithClaims(t, reg, claims)
	clock = clock.Add(edgesClaimTTL)
	book.errs = map[string]bool{"mine.bab.test": true, "theirs.bab.test": true}
	hosts := publishWithClaims(t, reg, claims)
	if h, ok := hosts["mine.bab.test"]; !ok || h.Stamp != 41 {
		t.Fatalf("mine = %+v, %v: an unreadable claim dropped or unstamped a served host", h, ok)
	}
	if _, ok := hosts["theirs.bab.test"]; ok {
		t.Fatal("an unreadable claim published a host this box was told not to")
	}

	fresh := newEdgesClaims("annie", book.read)
	if h, ok := publishWithClaims(t, reg, fresh)["mine.bab.test"]; !ok || h.Stamp != 0 {
		t.Fatalf("never read = %+v, %v; want home, unstamped", h, ok)
	}
}

func TestEdgesClaimsRereadOnlyAfterTheTTL(t *testing.T) {
	book := &claimsBook{claims: map[string]*hostclaim.Claim{"mine.bab.test": serving("annie", 41)}}
	clock := time.Date(2026, 10, 6, 9, 0, 0, 0, time.UTC)
	c := newEdgesClaims("annie", book.read)
	c.now = func() time.Time { return clock }
	ctx := context.Background()
	c.decide(ctx, []string{"mine.bab.test"})
	clock = clock.Add(edgesClaimTTL - time.Second)
	c.decide(ctx, []string{"mine.bab.test"})
	if len(book.reads) != 1 {
		t.Fatalf("%d reads inside the TTL, want 1", len(book.reads))
	}
	book.claims["mine.bab.test"] = serving("bert", 50)
	clock = clock.Add(time.Second)
	if d := c.decide(ctx, []string{"mine.bab.test"})["mine.bab.test"]; !d.withhold {
		t.Fatalf("decision after the TTL = %+v; the move to another box was not seen", d)
	}
	c.decide(ctx, nil)
	if len(c.entries) != 0 {
		t.Fatalf("a host no longer served kept its entry: %v", c.entries)
	}
}

func TestEdgesClaimsBoundReadsPerPassNeverReadFirst(t *testing.T) {
	book := &claimsBook{claims: map[string]*hostclaim.Claim{}}
	clock := time.Date(2026, 10, 6, 9, 0, 0, 0, time.UTC)
	c := newEdgesClaims("annie", book.read)
	c.now = func() time.Time { return clock }
	var hosts []string
	for i := 0; i < edgesClaimReadsPerPass+10; i++ {
		hosts = append(hosts, fmt.Sprintf("h%04d.bab.test", i))
	}
	ctx := context.Background()
	c.decide(ctx, hosts)
	if len(book.reads) != edgesClaimReadsPerPass {
		t.Fatalf("first pass read %d claims, want %d", len(book.reads), edgesClaimReadsPerPass)
	}
	book.reads = nil
	c.decide(ctx, hosts)
	if len(book.reads) != 10 {
		t.Fatalf("second pass read %d claims, want the 10 never read", len(book.reads))
	}
}

// Astra plan R1-2: an alias and a served catch-all are dialled through
// their primary's installation, so they carry its stamp. Unstamped, a dead
// predecessor's alias would tie with the successor's and the edge would
// keep the first box by name.
func TestEdgesFileStampsAliasesAndCatchAllsWithTheirPrimary(t *testing.T) {
	reg := edgesTestRegistry(t)
	seedInstance(t, reg, "mine.bab.test")
	for _, a := range []registry.RouteAlias{
		{Alias: "www.customer.com", Host: "mine.bab.test", Account: "bab", App: "jam", DNSStatus: "pointing"},
		{Alias: "*.customer.com", Host: "mine.bab.test", Account: "bab", App: "jam", DNSStatus: "pointing"},
	} {
		if err := reg.UpsertAlias(a); err != nil {
			t.Fatal(err)
		}
	}
	if err := reg.UpsertWildcardDomain(registry.WildcardDomain{Host: "customer.com", Account: "bab", App: "jam"}); err != nil {
		t.Fatal(err)
	}
	book := &claimsBook{claims: map[string]*hostclaim.Claim{"mine.bab.test": serving("annie", 57)}}
	hosts := publishWithClaims(t, reg, newEdgesClaims("annie", book.read))
	for _, key := range []string{"mine.bab.test", "www.customer.com", "*.customer.com"} {
		if h, ok := hosts[key]; !ok || h.Stamp != 57 || h.Role != "home" {
			t.Errorf("%s = %+v, %v; want home at stamp 57", key, h, ok)
		}
	}
	// With no claim reader the file is what it always was: no stamps.
	for key, h := range publishWithClaims(t, reg, nil) {
		if h.Stamp != 0 {
			t.Errorf("%s stamped %d with region HA off", key, h.Stamp)
		}
	}
}

// Astra plan R1-4: claims that keep failing go to the back of the queue,
// so they cannot hold every read of every pass.
func TestEdgesClaimsAFailingClaimDoesNotStarveTheRest(t *testing.T) {
	book := &claimsBook{claims: map[string]*hostclaim.Claim{"zz-healthy.bab.test": serving("annie", 7)}, errs: map[string]bool{}}
	var hosts []string
	for i := 0; i < edgesClaimReadsPerPass; i++ {
		h := fmt.Sprintf("a%04d.bab.test", i)
		book.errs[h] = true
		hosts = append(hosts, h)
	}
	hosts = append(hosts, "zz-healthy.bab.test")
	clock := time.Date(2026, 10, 6, 9, 0, 0, 0, time.UTC)
	c := newEdgesClaims("annie", book.read)
	c.now = func() time.Time { return clock }
	ctx := context.Background()
	c.decide(ctx, hosts) // the 256 failing claims sort first and all fail
	clock = clock.Add(edgesClaimPassEvery)
	if d, ok := c.decide(ctx, hosts)["zz-healthy.bab.test"]; !ok || d.stamp != 7 {
		t.Fatalf("healthy claim = %+v, %v: starved by claims that cannot be read", d, ok)
	}
}

// Astra plan R1-5: the loop's own schedule, with a stable registry, keeps
// every claim of a 2,000-host box fresh. Timer passes alone come every
// five to six minutes and read 256 claims each.
func TestEdgesLoopRefreshesEveryClaimWithinTheTTL(t *testing.T) {
	const n = 2000
	book := &claimsBook{claims: map[string]*hostclaim.Claim{}}
	var hosts []string
	for i := 0; i < n; i++ {
		h := fmt.Sprintf("h%04d.bab.test", i)
		book.claims[h] = serving("annie", uint64(i+1))
		hosts = append(hosts, h)
	}
	start := time.Date(2026, 10, 6, 9, 0, 0, 0, time.UTC)
	clock := start
	claims := newEdgesClaims("annie", book.read)
	claims.now = func() time.Time { return clock }
	state := &edgesLoopState{sched: newEdgesFileSchedule(start, func() time.Duration { return 5*time.Minute + 30*time.Second }), claims: claims}
	lastRead := map[string]time.Time{}
	worst := time.Duration(0)
	for clock = start; clock.Before(start.Add(3 * time.Hour)); clock = clock.Add(edgesFileWatchInterval) {
		if !state.due(clock, "fp", true) {
			continue
		}
		book.reads = nil
		claims.decide(context.Background(), hosts)
		for _, h := range book.reads {
			lastRead[h] = clock
		}
		state.done(clock, true)
		if clock.Sub(start) < time.Hour {
			continue // warm-up: the first reads of every host
		}
		for _, h := range hosts {
			if age := clock.Sub(lastRead[h]); age > worst {
				worst = age
			}
		}
	}
	if limit := edgesClaimTTL + 5*time.Minute; worst > limit {
		t.Fatalf("a claim went %s without a read, want at most %s", worst, limit)
	}
}

// Astra plan R2-1: a claim pass is a publication. While the bucket refuses
// every write, claim passes wait out the schedule's backoff and their own
// failures move it; once writes succeed, the 30-second claim passes resume.
func TestEdgesLoopClaimPassesObeyTheRetryBackoff(t *testing.T) {
	book := &claimsBook{claims: map[string]*hostclaim.Claim{}}
	var hosts []string
	for i := 0; i < 4000; i++ { // enough that reads stay due throughout
		h := fmt.Sprintf("h%04d.bab.test", i)
		book.claims[h] = serving("annie", 1)
		hosts = append(hosts, h)
	}
	start := time.Date(2026, 10, 6, 9, 0, 0, 0, time.UTC)
	clock := start
	claims := newEdgesClaims("annie", book.read)
	claims.now = func() time.Time { return clock }
	state := &edgesLoopState{sched: newEdgesFileSchedule(start, func() time.Duration { return 5*time.Minute + 30*time.Second }), claims: claims}
	failUntil := start.Add(20 * time.Minute)
	var during, after []time.Time
	for clock = start; clock.Before(failUntil.Add(8 * time.Minute)); clock = clock.Add(edgesFileWatchInterval) {
		if !state.due(clock, "fp", true) {
			continue
		}
		claims.decide(context.Background(), hosts)
		ok := !clock.Before(failUntil)
		if ok {
			after = append(after, clock)
		} else {
			during = append(during, clock)
		}
		state.done(clock, ok)
	}
	// Unthrottled, a claim pass every 30 seconds would be 40 attempts in
	// 20 minutes. The backoff (2 s doubling to 5 min) plus the timer allows
	// about a dozen.
	t.Logf("%d attempts while failing, %d after", len(during), len(after))
	if len(during) > 16 {
		t.Fatalf("%d publication attempts while every write failed, want at most 16: %v", len(during), during)
	}
	// Recovery waits out the backoff (at most five minutes), then the
	// 30-second claim passes resume.
	if len(after) < 3 || after[len(after)-1].Sub(after[len(after)-2]) != edgesClaimPassEvery {
		t.Fatalf("passes after writes recovered = %v, want the 30-second claim passes back within eight minutes", after)
	}
}

And create internal/edge/homemap_alias_stamp_test.go, which pins the edge's half: with stamped aliases, the successor's alias wins over a dead box's that sorts first. It passes on main already (dialBeats compares stamps after the alias rule); it is here so the reason for stamping aliases is tested where the decision is made.

package edge

import "testing"

// Region HA plan 1b (Astra plan R1-2): after a failover the dead box's file
// is stale but still listed. Its alias carries the old installation's
// stamp and the successor's the new one, so the successor wins however the
// boxes sort.
func TestHomeMapAnAliasFollowsTheNewerInstallation(t *testing.T) {
	s := meshTestStore(t)
	putJSON(t, s, "control/edges/annie.json", `{"v":1,"box":"annie","eip":"1.1.1.1","hosts":{"shop.test":{"role":"home","stamp":41},"www.customer.com":{"role":"home","alias":true,"stamp":41}}}`)
	putJSON(t, s, "control/edges/betty.json", `{"v":1,"box":"betty","eip":"2.2.2.2","hosts":{"shop.test":{"role":"home","stamp":57},"www.customer.com":{"role":"home","alias":true,"stamp":57}}}`)
	m := newHomeMap(s, "carol")
	refreshHomeMap(t, m)
	if eip, box, ok := m.Home("www.customer.com"); !ok || box != "betty" || eip != "2.2.2.2" {
		t.Fatalf("Home(www.customer.com) = %q %q %v, want betty", eip, box, ok)
	}
}
  • [ ] Step 2: Run them to see them fail

Run: go test ./internal/host/ -run 'EdgesFileFollows|EdgesFileWithholds|EdgesFileKeeps|EdgesFileStamps|EdgesClaims|EdgesLoop' -count=1 Expected: FAIL to build, undefined: edgesClaims.

  • [ ] Step 3: Write the claim cache

Create internal/host/edgesclaims.go:

package host

import (
	"context"
	"math"
	"sort"
	"time"

	"github.com/carlosframework/platform/internal/hostclaim"
	"github.com/carlosframework/platform/internal/registry"
)

// Region HA spec 1 §5.4: with CARLOS_REGION_HA on, a relocatable host is
// published as home only by the box its claim's op.exec names, carrying
// the claim's holder.stamp so the edge's home map believes the newest
// installation (HA4). The edge never reads claims; this publisher does, on
// a slow cache, because the safety gate is the activator's fresh read at
// every spawn (claim C4), not this file.

const (
	// edgesClaimTTL is how long one claim read stands: reconcile's sweep
	// (H9). A held row's entry can be stale this long, which can only send
	// a request to a box that then refuses to spawn.
	edgesClaimTTL = 10 * time.Minute
	// edgesClaimReadsPerPass bounds the GETs one pass makes, so a box with
	// thousands of hosts spreads its reads over several passes instead of
	// stalling one. A host not read yet publishes as it did before this
	// file existed.
	edgesClaimReadsPerPass = 256
	// edgesClaimPassEvery is how often the loop runs an extra pass while
	// claim reads are due. The registry timer alone (5 to 6 minutes) would
	// take 40 minutes to refresh 2,000 hosts at 256 a pass; with these
	// passes every host is re-read within about edgesClaimTTL plus four
	// minutes.
	edgesClaimPassEvery = 30 * time.Second
)

// edgesDecision is what one claim says about this box and one host.
type edgesDecision struct {
	withhold bool  // the claim names another executor, or none
	stamp    int64 // holder.stamp when this box is the executor
}

// edgesClaimEntry is one host's last decision and last read attempt. A
// failed read moves attempted and keeps the decision, so a claim that
// cannot be read goes to the back of the queue instead of holding its
// place at the front (claim C6: an error is never absence).
type edgesClaimEntry struct {
	decision  edgesDecision
	decided   bool // some read of this claim has succeeded
	attempted time.Time
}

// edgesClaims caches one decision per host. It is used only by the edges
// file loop's goroutine.
type edgesClaims struct {
	self    string // this box's reference: the edges file's box_id
	read    func(ctx context.Context, host string) (hostclaim.Observed, error)
	now     func() time.Time
	entries map[string]edgesClaimEntry
	hosts   []string // the relocatable hosts of the last pass
}

func newEdgesClaims(self string, read func(context.Context, string) (hostclaim.Observed, error)) *edgesClaims {
	return &edgesClaims{self: self, read: read, now: time.Now, entries: map[string]edgesClaimEntry{}}
}

// relocatableRow is spec 1 §2's relocatable host as a registry row shows
// it: a provisioned instance whose process is the activator's own exec
// child. Unit-backed, addr-backed, static and service routes keep their
// box and are published as before; sidecars are never published at all.
func relocatableRow(r registry.Route) bool {
	return r.Provisioned && r.Kind == "instance" && r.Backing == "" && r.Unit == "" &&
		r.Addr == "" && r.Addrs == "" && r.Socket != ""
}

// decideClaim turns one observation into a decision. An absent claim
// leaves the host as it was before claims existed: a region can only be
// given a second box once every host has one (spec §8).
func (c *edgesClaims) decideClaim(obs hostclaim.Observed) edgesDecision {
	if obs.Absent || obs.Claim == nil {
		return edgesDecision{}
	}
	cl := obs.Claim
	if cl.Holder == nil || cl.Op.Exec != c.self {
		return edgesDecision{withhold: true}
	}
	stamp := cl.Holder.Stamp
	if stamp > math.MaxInt64 {
		stamp = 0
	}
	return edgesDecision{stamp: int64(stamp)}
}

func (c *edgesClaims) dueAt(h string, now time.Time) bool {
	e, ok := c.entries[h]
	return !ok || now.Sub(e.attempted) >= edgesClaimTTL
}

// backlog reports whether any host of the last pass has a read due.
func (c *edgesClaims) backlog(now time.Time) bool {
	for _, h := range c.hosts {
		if c.dueAt(h, now) {
			return true
		}
	}
	return false
}

// decide returns a decision for each host that has one. Hosts whose last
// attempt is older than edgesClaimTTL are read, never-attempted first and
// then oldest attempt first, up to edgesClaimReadsPerPass. A read that
// fails keeps the previous decision; a host with no successful read has
// none, and publishes as before.
func (c *edgesClaims) decide(ctx context.Context, hosts []string) map[string]edgesDecision {
	now := c.now()
	c.hosts = append(c.hosts[:0], hosts...)
	want := make(map[string]bool, len(hosts))
	var due []string
	for _, h := range hosts {
		want[h] = true
		if c.dueAt(h, now) {
			due = append(due, h)
		}
	}
	for h := range c.entries {
		if !want[h] {
			delete(c.entries, h)
		}
	}
	sort.Slice(due, func(i, j int) bool {
		ei, iok := c.entries[due[i]]
		ej, jok := c.entries[due[j]]
		if iok != jok {
			return !iok // never attempted first
		}
		if !ei.attempted.Equal(ej.attempted) {
			return ei.attempted.Before(ej.attempted)
		}
		return due[i] < due[j]
	})
	if len(due) > edgesClaimReadsPerPass {
		due = due[:edgesClaimReadsPerPass]
	}
	for _, h := range due {
		e := c.entries[h]
		e.attempted = now
		if obs, err := c.read(ctx, h); err == nil {
			e.decision, e.decided = c.decideClaim(obs), true
		}
		c.entries[h] = e
	}
	out := make(map[string]edgesDecision, len(c.entries))
	for h, e := range c.entries {
		if e.decided {
			out[h] = e.decision
		}
	}
	return out
}

// applyEdgesClaims withholds or stamps every relocatable row in hosts
// (edgesFileWrite's served map, before aliases join it, so an alias of a
// withheld host is withheld with it and an alias of a stamped host takes
// its stamp). claims nil leaves hosts untouched.
func applyEdgesClaims(ctx context.Context, claims *edgesClaims, routes []registry.Route, hosts map[string]edgeHost) {
	if claims == nil {
		return
	}
	var reloc []string
	for _, r := range routes {
		if _, served := hosts[r.Host]; served && relocatableRow(r) {
			reloc = append(reloc, r.Host)
		}
	}
	for host, d := range claims.decide(ctx, reloc) {
		h, ok := hosts[host]
		if !ok {
			continue
		}
		if d.withhold {
			delete(hosts, host)
			continue
		}
		h.Stamp = d.stamp
		hosts[host] = h
	}
}

// edgesLoopState is edgesFileLoop's per-tick decision: the registry and
// timer schedule, plus, with region HA on, an extra pass every
// edgesClaimPassEvery while claim reads are due. A claim pass is a
// publication like any other: it waits out the schedule's retry backoff
// after a failed write, and its own outcome moves that backoff, so a bucket
// that refuses every write is not asked every 30 seconds (Astra plan R2-1).
type edgesLoopState struct {
	sched      *edgesFileSchedule
	claims     *edgesClaims
	claimsNext time.Time
	schedDue   bool
	claimDue   bool
}

// due reports whether to run a pass on this tick.
func (s *edgesLoopState) due(now time.Time, fp string, fpOK bool) bool {
	s.schedDue = s.sched.due(now, fp, fpOK)
	s.claimDue = s.claims != nil && !now.Before(s.claimsNext) && !now.Before(s.sched.retryAt) && s.claims.backlog(now)
	return s.schedDue || s.claimDue
}

// timed reports whether the pass due just allowed is the timer's.
func (s *edgesLoopState) timed() bool { return s.schedDue && s.sched.timed }

// done records the pass's outcome. Every pass, claim passes included, goes
// through the schedule: a failure advances its backoff, and a success
// clears it and records the fingerprint the pass published. A claim pass
// is never the timer's (due cleared timed), so the timer does not move.
func (s *edgesLoopState) done(now time.Time, ok bool) {
	if s.schedDue || s.claimDue {
		s.sched.done(now, ok)
	}
	if s.claims != nil {
		s.claimsNext = now.Add(edgesClaimPassEvery)
	}
}
  • [ ] Step 4: Wire it into the publisher

In internal/host/edgesfile.go, add the import (keep the group sorted):

	"github.com/carlosframework/platform/internal/appconfig"
	"github.com/carlosframework/platform/internal/hostclaim"
	"github.com/carlosframework/platform/internal/registry"

Replace the head of edgesFileWrite:

func edgesFileWrite(ctx context.Context, s release.Store, reg *registry.Registry, ident edgesFile, eip string, last *[]byte, refresh bool) error {
	box := ident.Box

with:

func edgesFileWrite(ctx context.Context, s release.Store, reg *registry.Registry, ident edgesFile, eip string, last *[]byte, refresh bool) error {
	return edgesFileWriteClaims(ctx, s, reg, ident, eip, last, refresh, nil)
}

// edgesFileWriteClaims is edgesFileWrite with the claim reader region HA
// adds (edgesclaims.go). claims is nil unless CARLOS_REGION_HA is on, and
// nil publishes exactly what edgesFileWrite always has.
func edgesFileWriteClaims(ctx context.Context, s release.Store, reg *registry.Registry, ident edgesFile, eip string, last *[]byte, refresh bool, claims *edgesClaims) error {
	box := ident.Box

After the routes loop, replace:

		hosts[r.Host] = edgeHost{Role: "home", Platform: r.Platform || r.Provisioned, Replica: r.Replica}
	}

with:

		hosts[r.Host] = edgeHost{Role: "home", Platform: r.Platform || r.Provisioned, Replica: r.Replica}
	}
	// Region HA (spec 1 §5.4): a relocatable host is home only on the box
	// its claim names, stamped with the installation. Before owners and
	// aliases, so an alias of a withheld host is withheld with it.
	applyEdgesClaims(ctx, claims, routes, hosts)

In edgesFileLoop, replace:

	dataDir := provisionTickConfig(os.Getenv).DataDir
	log.Printf("agent: edges file loop armed (5min + jitter, and %s after a registry change, %s)", edgesFileWatchInterval, edgesFileKey(box))

with:

	dataDir := provisionTickConfig(os.Getenv).DataDir
	// Region HA reads each relocatable host's claim (spec 1 §5.4); off, the
	// file is exactly what it always was.
	var claims *edgesClaims
	if regionHAEnabled(os.Getenv) {
		claims = newEdgesClaims(box, func(ctx context.Context, host string) (hostclaim.Observed, error) {
			return hostclaim.Read(ctx, store, host)
		})
	}
	log.Printf("agent: edges file loop armed (5min + jitter, and %s after a registry change, %s)", edgesFileWatchInterval, edgesFileKey(box))

Then, where the loop builds its schedule, replace:

	sched := newEdgesFileSchedule(time.Now(), func() time.Duration {
		return 5*time.Minute + rand.N(time.Minute)
	})
	tick := time.NewTicker(edgesFileWatchInterval)

with:

	sched := newEdgesFileSchedule(time.Now(), func() time.Duration {
		return 5*time.Minute + rand.N(time.Minute)
	})
	state := &edgesLoopState{sched: sched, claims: claims}
	tick := time.NewTicker(edgesFileWatchInterval)

Then, in the loop body, replace:

		if !sched.due(now, fp, err == nil) {
			continue
		}
		if sched.timed {
			eip = readPublicIP()
		}
		ident.Headroom = headroomFor(ident.Capabilities, dataDir, reg)
		err = edgesFileWrite(ctx, store, reg, ident, eip, &last, sched.timed)

with:

		if !state.due(now, fp, err == nil) {
			continue
		}
		if state.timed() {
			eip = readPublicIP()
		}
		ident.Headroom = headroomFor(ident.Capabilities, dataDir, reg)
		err = edgesFileWriteClaims(ctx, store, reg, ident, eip, &last, state.timed(), claims)

Then, at the end of the loop body, replace:

		sched.done(now, err == nil)

with:

		state.done(now, err == nil)

Then, in the alias loop, replace:

		acct, app := owners[a.Host].owner()
		hosts[a.Alias] = edgeHost{Role: "home", Alias: true, Account: acct, App: app}

with:

		acct, app := owners[a.Host].owner()
		// The alias is served by its primary's installation, so it carries
		// the primary's stamp (region HA HA4): an unstamped alias would tie
		// with a dead predecessor's and the edge would keep the first box
		// by name.
		hosts[a.Alias] = edgeHost{Role: "home", Alias: true, Account: acct, App: app, Stamp: hosts[a.Host].Stamp}

Then, in the delegated wildcard loop, replace:

		role, alias := "wildcard", false
		var acct, app string
		if primary, served := catchallServed[wd.Host]; served {
			role, alias = "home", true

with:

		role, alias := "wildcard", false
		var acct, app string
		var stamp int64
		if primary, served := catchallServed[wd.Host]; served {
			role, alias = "home", true
			stamp = hosts[primary].Stamp // the dial half is the primary's installation

Then, at the end of the same loop, replace:

		hosts[key] = edgeHost{Role: role, Wildcard: true, Alias: alias, Account: acct, App: app}

with:

		hosts[key] = edgeHost{Role: role, Wildcard: true, Alias: alias, Account: acct, App: app, Stamp: stamp}

With region HA off every primary's stamp is 0, so every alias and catch-all is published exactly as before.

box here is metricsBoxLabel(…), the same value the file publishes as box_id, which is the reference claims name (spec §2).

  • [ ] Step 5: Run the tests

Run:

go test ./internal/host/ -run 'EdgesFile|EdgesClaims|EdgesLoop' -count=1 -v
go test ./internal/edge/ -run 'TestHomeMap' -count=1

Expected: PASS: the nine new host tests (including TestEdgesFileStampsAliasesAndCatchAllsWithTheirPrimary, TestEdgesClaimsAFailingClaimDoesNotStarveTheRest, TestEdgesLoopRefreshesEveryClaimWithinTheTTL and TestEdgesLoopClaimPassesObeyTheRetryBackoff), every existing edges-file test, and the home map tests. The internal/edge run of TestHomeMap alone needs no socket.

  • [ ] Step 6: Gate and commit
go build ./... && go vet ./internal/host/ ./internal/edge/ && gofmt -l internal/host internal/edge
GOOS=darwin go build ./... && GOOS=windows go build ./...
git add internal/host/edgesclaims.go internal/host/edgesclaims_test.go internal/host/edgesfile.go internal/edge/homemap_alias_stamp_test.go
git commit -m "edges: with region HA on, a relocatable host is home only on the box its claim names, stamped

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>"

---

Task 7: Root refuses an update while the box takes part in a claim operation (internal/host/systemparticipation.go)

Spec §5.10, root's side (H14): under root's update lock, before an executable change, write /opt/carlos/system/guard/update-begun.json durably, then scan <DataDir>/.hostexec/ completely. Any record there, in any state, or anything root cannot read, list or parse, removes the marker and refuses the update, to be retried later. Root records that refusal as deferred, not refused (Astra plan R1-3): refused is terminal for that promote (systemResult.terminal, systemleg.go), and the agent would never ask again; deferred is re-requested on the agent's backoff ladder, at most every five minutes, until the operation ends. Otherwise the update proceeds and the marker stays until it finishes or is recovered.

On main every executable change goes through one path, systemApplier.install (an operator's carlos system rollback re-points the channel, and the box applies it the same way); the guarded activation and the console's activation reservation are claim D1 and D7 and not on main. Recovery of an update already begun needs no check (spec §5.10: no participant acknowledged meanwhile). The participant's half (write pending, then refuse to acknowledge while the marker exists) belongs to claim D6's controller and waits for L6.

Nothing writes .hostexec/ on main, so with the flag on this changes nothing but a marker file during an update.

Files:

  • Create: internal/host/systemparticipation.go
  • Test: internal/host/systemparticipation_test.go
  • Modify: internal/host/systemapply.go (systemApplier, RunSystemApply, Run, install)

Interfaces:

  • Consumes: regionHAEnabled (Task 3); writeJSONDurable, syncDir, readSystemJournal, errSystemDeferred, systemApplier.record (existing).
  • Produces: ``go func systemGuardDir(optDir string) string func updateBegunPath(optDir string) string func hostexecParticipation(dataDir string) error func (a *systemApplier) beginExecutableChange(target string, promotedAt time.Time) error func (a *systemApplier) clearUpdateBegun() error func (a *systemApplier) settleUpdateBegun() // systemApplier gains: regionHA bool ``
  • [ ] Step 1: Write the failing tests

Create internal/host/systemparticipation_test.go:

package host

import (
	"context"
	"os"
	"path/filepath"
	"runtime"
	"strings"
	"testing"
)

func hostexec(t *testing.T, f *applyFixture) string {
	t.Helper()
	dir := filepath.Join(f.a.dataDir, hostexecDir)
	if err := os.MkdirAll(dir, 0o755); err != nil {
		t.Fatal(err)
	}
	return dir
}

func assertNotUpdated(t *testing.T, f *applyFixture, why string) {
	t.Helper()
	r := f.result()
	if r == nil || r.Outcome != systemOutcomeDeferred || !strings.Contains(r.Reason, why) {
		t.Fatalf("result = %+v, want deferred naming %q", r, why)
	}
	if r.terminal() {
		t.Fatal("the agent would never retry this promote")
	}
	if bin, _ := os.ReadFile(systemBinaryPath(f.a.optDir)); !strings.Contains(string(bin), "carlos v1") {
		t.Fatal("the binary changed under a refusal")
	}
	if j, _ := readSystemJournal(f.a.optDir); j != nil {
		t.Fatalf("a refusal opened a journal: %+v", j)
	}
	if _, err := os.Stat(updateBegunPath(f.a.optDir)); !os.IsNotExist(err) {
		t.Fatalf("the update-begun marker outlived the refusal: %v", err)
	}
	if f.called("systemctl stop") || f.called("systemctl restart") {
		t.Fatal("a refused update touched a unit")
	}
}

// Spec §5.10: any controller record, in any state, refuses the update.
func TestApplyRefusesWhileThisBoxTakesPartInAClaimOperation(t *testing.T) {
	for _, state := range []string{"pending", "active", "lost"} {
		t.Run(state, func(t *testing.T) {
			f, _ := applyHappy(t)
			f.a.regionHA = true
			os.WriteFile(filepath.Join(hostexec(t, f), "shop.bab.test.json"), []byte(`{"state":"`+state+`"}`), 0o600)
			if err := f.a.Run(context.Background(), false); err != nil {
				t.Fatal(err)
			}
			assertNotUpdated(t, f, "takes part in a running claim operation")
		})
	}
}

// Review Focus 4: what root cannot read is a refusal, never a pass.
func TestApplyRefusesWhenItCannotTellWhetherThisBoxTakesPart(t *testing.T) {
	cases := map[string]func(t *testing.T, f *applyFixture){
		"a temporary file": func(t *testing.T, f *applyFixture) {
			os.WriteFile(filepath.Join(hostexec(t, f), ".shop.bab.test.json.tmp"), nil, 0o600)
		},
		"garbage": func(t *testing.T, f *applyFixture) {
			os.WriteFile(filepath.Join(hostexec(t, f), "shop.bab.test.json"), []byte("{half"), 0o600)
		},
		"a symlink in place of the directory": func(t *testing.T, f *applyFixture) {
			elsewhere := t.TempDir()
			if err := os.Symlink(elsewhere, filepath.Join(f.a.dataDir, hostexecDir)); err != nil {
				t.Skip("no symlinks here:", err)
			}
		},
		"a file in place of the directory": func(t *testing.T, f *applyFixture) {
			os.WriteFile(filepath.Join(f.a.dataDir, hostexecDir), nil, 0o600)
		},
	}
	for name, setup := range cases {
		t.Run(name, func(t *testing.T) {
			f, _ := applyHappy(t)
			f.a.regionHA = true
			setup(t, f)
			if err := f.a.Run(context.Background(), false); err != nil {
				t.Fatal(err)
			}
			assertNotUpdated(t, f, "hostexec")
		})
	}
}

func TestApplyRefusesWhenItCannotListTheControllerDirectory(t *testing.T) {
	if runtime.GOOS == "windows" || os.Geteuid() == 0 {
		t.Skip("permission bits do not stop this user")
	}
	f, _ := applyHappy(t)
	f.a.regionHA = true
	dir := hostexec(t, f)
	os.Chmod(dir, 0)
	t.Cleanup(func() { os.Chmod(dir, 0o755) })
	if err := f.a.Run(context.Background(), false); err != nil {
		t.Fatal(err)
	}
	assertNotUpdated(t, f, "cannot be listed")
}

// The marker is written before the box changes and is gone once the
// update is done; an empty controller directory is no participation.
func TestApplyHoldsTheUpdateBegunMarkerAcrossTheInstall(t *testing.T) {
	f, req := applyHappy(t)
	f.a.regionHA = true
	hostexec(t, f) // present and empty
	seen := false
	orig := f.a.ops.systemctl
	f.a.ops.systemctl = func(args ...string) (string, error) {
		if args[0] == "restart" && args[len(args)-1] == systemEdgeUnit {
			_, err := os.Stat(updateBegunPath(f.a.optDir))
			seen = err == nil
		}
		return orig(args...)
	}
	if err := f.a.Run(context.Background(), false); err != nil {
		t.Fatal(err)
	}
	if r := f.result(); r == nil || r.Outcome != systemOutcomeApplied || !r.matches("stable", "v2", req.PromotedAt) {
		t.Fatalf("result = %+v", r)
	}
	if !seen {
		t.Fatal("no update-begun marker while the edge restarted onto the new binary")
	}
	if _, err := os.Stat(updateBegunPath(f.a.optDir)); !os.IsNotExist(err) {
		t.Fatalf("marker left after the update finished: %v", err)
	}
}

func TestApplyWithoutRegionHAWritesNoMarkerAndIgnoresControllerRecords(t *testing.T) {
	f, _ := applyHappy(t)
	os.WriteFile(filepath.Join(hostexec(t, f), "shop.bab.test.json"), []byte(`{}`), 0o600)
	if err := f.a.Run(context.Background(), false); err != nil {
		t.Fatal(err)
	}
	if r := f.result(); r == nil || r.Outcome != systemOutcomeApplied {
		t.Fatalf("result = %+v: the flag is off, so nothing changes", r)
	}
	if _, err := os.Stat(systemGuardDir(f.a.optDir)); !os.IsNotExist(err) {
		t.Fatalf("the guard directory was created with the flag off: %v", err)
	}
}

// A crash between the marker and the journal leaves a marker with no
// update behind it; the next run, --recover included, clears it.
func TestAStaleUpdateBegunMarkerIsClearedByTheNextRun(t *testing.T) {
	f := newApplyFixture(t, "stable", "v1")
	os.MkdirAll(systemGuardDir(f.a.optDir), 0o755)
	os.WriteFile(updateBegunPath(f.a.optDir), []byte(`{"target":"v2"}`), 0o644)
	if err := f.a.Run(context.Background(), true); err != nil {
		t.Fatal(err)
	}
	if _, err := os.Stat(updateBegunPath(f.a.optDir)); !os.IsNotExist(err) {
		t.Fatalf("stale marker kept: %v", err)
	}
}

// A rollback that could not finish keeps its journal for the next run, and
// the marker with it: the update is not over.
func TestTheMarkerStaysWhileAJournalIsOpen(t *testing.T) {
	f := newApplyFixture(t, "stable", "v1")
	f.a.regionHA = true
	os.MkdirAll(systemGuardDir(f.a.optDir), 0o755)
	os.WriteFile(updateBegunPath(f.a.optDir), []byte(`{"target":"v2"}`), 0o644)
	// A journal recovery cannot act on: no carlos.prev to restore.
	if err := writeJSONDurable(f.a.file("journal.json"), systemJournal{Phase: systemPhaseInstalled, Target: "v2"}, 0o644); err != nil {
		t.Fatal(err)
	}
	_ = f.a.Run(context.Background(), true)
	if j, _ := readSystemJournal(f.a.optDir); j == nil {
		t.Fatal("recovery closed the journal; this test needs one it cannot finish")
	}
	if _, err := os.Stat(updateBegunPath(f.a.optDir)); err != nil {
		t.Fatalf("marker cleared while the journal is still open: %v", err)
	}
}

// Astra plan R1-3: participation defers, it does not refuse. Once the
// operation ends and its record is cleared, the same promote applies. The
// agent's side, re-requesting a deferred apply, is
// TestSystemLegReusesAVerifiedCacheOnARetry.
func TestApplyDeferredForParticipationAppliesOnceItEnds(t *testing.T) {
	f, req := applyHappy(t)
	f.a.regionHA = true
	rec := filepath.Join(hostexec(t, f), "shop.bab.test.json")
	os.WriteFile(rec, []byte(`{"state":"active"}`), 0o600)
	if err := f.a.Run(context.Background(), false); err != nil {
		t.Fatal(err)
	}
	if r := f.result(); r == nil || r.Outcome != systemOutcomeDeferred || r.terminal() {
		t.Fatalf("result = %+v, want deferred, which the agent retries", r)
	}
	os.Remove(rec)
	if err := f.a.Run(context.Background(), false); err != nil {
		t.Fatal(err)
	}
	if r := f.result(); r == nil || r.Outcome != systemOutcomeApplied || !r.matches("stable", "v2", req.PromotedAt) {
		t.Fatalf("result after the operation ended = %+v, want applied", r)
	}
}
  • [ ] Step 2: Run them to see them fail

Run: go test ./internal/host/ -run 'TakesPart|CannotTell|CannotList|UpdateBegun|RegionHAWrites|MarkerStays' -count=1 Expected: FAIL to build, f.a.regionHA undefined and undefined: hostexecDir.

  • [ ] Step 3: Write root's check

Create internal/host/systemparticipation.go:

package host

import (
	"errors"
	"fmt"
	"os"
	"path/filepath"
	"sort"
	"strings"
	"time"
)

// Region HA spec 1 §5.10 (H14), root's half. A box that takes part in a
// running claim operation never takes an executable update. Each side
// records its own intent durably, then checks the other's, so of two that
// race at least one sees the other:
//
//   - root writes its update-begun marker, then scans <DataDir>/.hostexec/
//     and refuses the update while any controller record is there, in any
//     state, or while it cannot tell;
//   - a participant writes its .hostexec record (claim D6), then reads the
//     marker and refuses to acknowledge while one exists. That half waits
//     for the claim's D6 controller and is not built here.
//
// The marker stays until the update finishes or is recovered. Dormant
// unless CARLOS_REGION_HA is on.

const (
	updateBegunFile = "update-begun.json"
	hostexecDir     = ".hostexec"
)

// systemGuardDir is /opt/carlos/system/guard, beside root's update state.
func systemGuardDir(optDir string) string { return filepath.Join(optDir, "system", "guard") }

func updateBegunPath(optDir string) string {
	return filepath.Join(systemGuardDir(optDir), updateBegunFile)
}

type updateBegun struct {
	Target     string    `json:"target"`
	PromotedAt time.Time `json:"promoted_at"`
	At         time.Time `json:"at"`
}

// hostexecParticipation reports why this box may still be a participant
// in a claim operation, or nil when it certainly is not. The directory
// absent is the only "not". Anything in it, whatever its name or state,
// and anything root cannot list or that is not a plain directory, is a
// refusal (spec §5.10, B1-5). Only names are read: root never opens a
// record here.
func hostexecParticipation(dataDir string) error {
	dir := filepath.Join(dataDir, hostexecDir)
	fi, err := os.Lstat(dir)
	if errors.Is(err, os.ErrNotExist) {
		return nil
	}
	if err != nil {
		return fmt.Errorf("%s cannot be read: %v", dir, err)
	}
	if !fi.IsDir() {
		return fmt.Errorf("%s is not a directory", dir)
	}
	entries, err := os.ReadDir(dir)
	if err != nil {
		return fmt.Errorf("%s cannot be listed: %v", dir, err)
	}
	if len(entries) == 0 {
		return nil
	}
	names := make([]string, 0, len(entries))
	for _, e := range entries {
		names = append(names, e.Name())
	}
	sort.Strings(names)
	if len(names) > 3 {
		names = append(names[:3], fmt.Sprintf("and %d more", len(entries)-3))
	}
	return fmt.Errorf("this box takes part in a running claim operation (%s holds %s)", dir, strings.Join(names, ", "))
}

// beginExecutableChange is §5.10's root side, run under the apply lock
// just before the first box-changing step: write the marker durably, then
// look for participation. On a refusal the marker is removed again and the
// error, wrapping errSystemDeferred, says why. The caller records it as
// "deferred", never "refused": a refusal is terminal for that promote, and
// this one must be retried once the operation ends (spec §5.10: the
// update is refused and retried later). The agent's leg re-requests a
// deferred apply on its backoff ladder, at most every five minutes.
func (a *systemApplier) beginExecutableChange(target string, promotedAt time.Time) error {
	if !a.regionHA {
		return nil
	}
	if err := os.MkdirAll(systemGuardDir(a.optDir), 0o755); err != nil {
		return err
	}
	if err := writeJSONDurable(updateBegunPath(a.optDir), updateBegun{Target: target, PromotedAt: promotedAt, At: a.ops.now().UTC()}, 0o644); err != nil {
		return err
	}
	if reason := hostexecParticipation(a.dataDir); reason != nil {
		if err := a.clearUpdateBegun(); err != nil {
			a.ops.logf("system apply: %v", err)
		}
		return fmt.Errorf("%w: %v", errSystemDeferred, reason)
	}
	return nil
}

// clearUpdateBegun removes the marker durably. A marker that is not there
// is already cleared.
func (a *systemApplier) clearUpdateBegun() error {
	err := os.Remove(updateBegunPath(a.optDir))
	if errors.Is(err, os.ErrNotExist) {
		return nil
	}
	if err != nil {
		return fmt.Errorf("remove %s: %w", updateBegunPath(a.optDir), err)
	}
	return syncDir(systemGuardDir(a.optDir))
}

// settleUpdateBegun clears the marker once no update is in progress: no
// journal means the update finished, rolled back, was recovered, or never
// got past its first step. A kept journal (a rollback that could not
// finish) keeps the marker with it. Run with or without CARLOS_REGION_HA,
// so a marker left behind after the flag is turned off still goes.
func (a *systemApplier) settleUpdateBegun() {
	if j, err := readSystemJournal(a.optDir); err != nil || j != nil {
		return
	}
	if err := a.clearUpdateBegun(); err != nil {
		a.ops.logf("system apply: %v", err)
	}
}
  • [ ] Step 4: Wire it into the apply

In internal/host/systemapply.go, add the field at the end of systemApplier:

	platform       string // "<goos>-<goarch>"; the caller sets it from runtime
	gateWait, soak time.Duration
	// regionHA is CARLOS_REGION_HA: root writes its update-begun marker and
	// refuses an update while this box takes part in a claim operation
	// (region HA spec 1 §5.10; systemparticipation.go).
	regionHA bool
}

In RunSystemApply, set it:

		platform: runtime.GOOS + "-" + runtime.GOARCH, gateWait: systemGateWait, soak: systemGateSoak,
		regionHA: regionHAEnabled(os.Getenv),
	}

In Run, right after defer os.RemoveAll(a.stagingDir()), add:

	// Whatever this run does, a finished or recovered update leaves no
	// update-begun marker behind (spec §5.10).
	defer a.settleUpdateBegun()

In install, immediately before // PromotedAt rides along for the whole transaction, add:

	// Region HA (spec 1 §5.10): the marker, then the participation check,
	// before the first step that changes the box. Either failing defers the
	// update, which the agent retries; the box is left exactly as it was.
	if err := a.beginExecutableChange(target, req.PromotedAt); err != nil {
		if !errors.Is(err, errSystemDeferred) {
			err = fmt.Errorf("%w: update-begun marker: %v", errSystemDeferred, err)
		}
		return a.record(req, err)
	}

The deferral is recorded in result.json; the agent's leg re-requests a deferred apply (pinned on the agent's side by TestSystemLegReusesAVerifiedCacheOnARetry).

  • [ ] Step 5: Run the tests

Run: go test ./internal/host/ -run 'Apply|Recover|Rollback|Prev|Layout|UpdateBegun|MarkerStays|TakesPart|CannotTell|CannotList|Deferred' -count=1 -v Expected: PASS: the eight new tests (including TestApplyDeferredForParticipationAppliesOnceItEnds) and every existing apply and recovery test.

  • [ ] Step 6: Gate and commit
go build ./... && go vet ./internal/host/ && gofmt -l internal/host
GOOS=darwin go build ./... && GOOS=windows go build ./... && GOOS=linux GOARCH=arm64 go build ./...
git add internal/host/systemparticipation.go internal/host/systemparticipation_test.go internal/host/systemapply.go
git commit -m "system: with region HA on, root marks an update begun and refuses it while the box takes part in a claim operation

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>"

---

Task 8: Storage policy, the spec note and the branch gate

Spec §6: "The control bucket denies PutObject without s3:if-none-match: *, and DeleteObject and DeleteObjectVersion, on control/boxes/* (plan 1a's policy) and on control/box-volumes/*, control/box-enrollment/* and control/box-fences/*. Each Deny is scoped to its prefix only." control/boxes/ is already covered. Task 5's writer uses control/box-volumes/; control/box-enrollment/ and control/box-fences/ have no writer yet, but a Deny grants nothing, and putting all four in place at once means one apply. The *.bind.json statement and the operator_role_arns variable stay (§19: "harmless and can go at the next apply"): removing the variable would break every root that sets it.

Nothing here is applied. The flagship's root lives in carlosframework/carloku-infrastructure and consumes these modules at a release tag. Two cautions bind this task: a whole-root apply of the flagship restarts every box (user-data drift), and this task changes no user_data; a broad Deny on these buckets once broke every lease release, and every Deny here names one evidence prefix and nothing wider.

Files:

  • Modify: infra/modules/deployment-buckets/main.tf
  • Create: infra/modules/deployment-buckets/tests/evidence.tftest.hcl
  • Modify: docs/superpowers/specs/2026-10-04-region-ha-box-placement-design.md (append §24)
  • [ ] Step 1: Write the failing module test

Create infra/modules/deployment-buckets/tests/evidence.tftest.hcl:

# Region HA spec 1 §6: every box-evidence prefix is create-only and never
# deleted, and the Denies name those prefixes and nothing wider (a broad
# Deny on this bucket once broke every lease release). Each statement is
# checked whole: effect, principal, actions, condition and resources, so
# turning a Deny into an Allow, dropping an action or loosening the
# conditional-write condition fails here.

mock_provider "aws" {}

override_resource {
  target = aws_s3_bucket.deployment
  values = { arn = "arn:aws:s3:::dep" }
}

variables {
  deployment_bucket_name = "dep"
  data_bucket_name       = "data"
}

run "evidence_prefixes_are_create_only_and_never_deleted" {
  command = plan

  assert {
    condition = jsonencode([
      for s in jsondecode(aws_s3_bucket_policy.deployment_box_records.policy).Statement : s if s.Sid == "BoxRecordsAreCreateOnly"
      ]) == jsonencode([{
        Action = "s3:PutObject"
        Condition = {
          Bool = { "s3:ObjectCreationOperation" = "true" }
          Null = { "s3:if-none-match" = "true" }
        }
        Effect    = "Deny"
        Principal = "*"
        Resource = [
          "arn:aws:s3:::dep/control/boxes/*",
          "arn:aws:s3:::dep/control/box-volumes/*",
          "arn:aws:s3:::dep/control/box-enrollment/*",
          "arn:aws:s3:::dep/control/box-fences/*",
        ]
        Sid = "BoxRecordsAreCreateOnly"
    }])
    error_message = "BoxRecordsAreCreateOnly must Deny, to everyone, every PutObject without If-None-Match on exactly the four box-evidence prefixes"
  }

  assert {
    condition = jsonencode([
      for s in jsondecode(aws_s3_bucket_policy.deployment_box_records.policy).Statement : s if s.Sid == "BoxRecordsAreNeverDeleted"
      ]) == jsonencode([{
        Action    = ["s3:DeleteObject", "s3:DeleteObjectVersion"]
        Effect    = "Deny"
        Principal = "*"
        Resource = [
          "arn:aws:s3:::dep/control/boxes/*",
          "arn:aws:s3:::dep/control/box-volumes/*",
          "arn:aws:s3:::dep/control/box-enrollment/*",
          "arn:aws:s3:::dep/control/box-fences/*",
        ]
        Sid = "BoxRecordsAreNeverDeleted"
    }])
    error_message = "BoxRecordsAreNeverDeleted must Deny, to everyone, both delete actions on exactly the four box-evidence prefixes"
  }
}
  • [ ] Step 2: Run it to see it fail

Run (sandbox off; tofu init needs the network once):

cd infra/modules/deployment-buckets && tofu init -backend=false -input=false && tofu test; cd -

Expected: Failure! 0 passed, 1 failed., both assertions naming their statement. Leave .terraform/ and .terraform.lock.hcl in place until Step 4 has run: Step 4 needs the initialized provider.

  • [ ] Step 3: Extend the policy

In infra/modules/deployment-buckets/main.tf, replace the section comment:

# --- Box records (region HA spec 1 §5.1) -----------------------------------
# control/boxes/<id>.json binds a box's name to one AWS instance, and
# control/boxes/<id>.bind.json is an operator's word for a legacy label.
# Both are written once and never changed. Storage enforces that, so no
# bug or stolen box credential can rebind a name to another machine.
#
# EVERY statement here names control/boxes/ and nothing wider. A broad Deny
# on this bucket has broken lease releases before.

with:

# --- Box evidence (region HA spec 1 §5.1, §5.9, §6) ------------------------
# control/boxes/<ref>.json is a box's AWS-signed identity, the evidence its
# enrollment reads. control/box-volumes/, control/box-enrollment/ and
# control/box-fences/ are the volume records, membership confirmations
# and termination fence records regional workers write. Every one is
# written once and never changed or deleted. Storage enforces that, so no
# bug or stolen box credential can rewrite the evidence behind a decision.
# (control/boxes/*.bind.json is no longer written: enrollment replaced the
# operator bind. Its statement stays harmless until operator_role_arns is
# retired with it.)
#
# EVERY statement here names one of these prefixes and nothing wider. A
# broad Deny on this bucket has broken lease releases before.

locals {
  box_evidence_prefixes = ["control/boxes", "control/box-volumes", "control/box-enrollment", "control/box-fences"]
}

and replace the first two statements of aws_s3_bucket_policy.deployment_box_records:

      {
        # A record or bind may be created, never replaced. The qualifier
        # leaves multipart preparation alone and still covers its completion.
        Sid       = "BoxRecordsAreCreateOnly"
        Effect    = "Deny"
        Principal = "*"
        Action    = "s3:PutObject"
        Resource  = "${aws_s3_bucket.deployment.arn}/control/boxes/*"
        Condition = {
          Null = { "s3:if-none-match" = "true" }
          Bool = { "s3:ObjectCreationOperation" = "true" }
        }
      },
      {
        Sid       = "BoxRecordsAreNeverDeleted"
        Effect    = "Deny"
        Principal = "*"
        Action    = ["s3:DeleteObject", "s3:DeleteObjectVersion"]
        Resource  = "${aws_s3_bucket.deployment.arn}/control/boxes/*"
      },

with:

      {
        # Evidence may be created, never replaced. The qualifier leaves
        # multipart preparation alone and still covers its completion.
        Sid       = "BoxRecordsAreCreateOnly"
        Effect    = "Deny"
        Principal = "*"
        Action    = "s3:PutObject"
        Resource  = [for p in local.box_evidence_prefixes : "${aws_s3_bucket.deployment.arn}/${p}/*"]
        Condition = {
          Null = { "s3:if-none-match" = "true" }
          Bool = { "s3:ObjectCreationOperation" = "true" }
        }
      },
      {
        Sid       = "BoxRecordsAreNeverDeleted"
        Effect    = "Deny"
        Principal = "*"
        Action    = ["s3:DeleteObject", "s3:DeleteObjectVersion"]
        Resource  = [for p in local.box_evidence_prefixes : "${aws_s3_bucket.deployment.arn}/${p}/*"]
      },

The Sids are kept, so a reader of the bucket policy's history sees one statement widened, not two replaced.

  • [ ] Step 4: Run the module test

Run (sandbox off; this re-runs tofu init, which is a no-op if Step 2's initialization is still there):

cd infra/modules/deployment-buckets && tofu init -backend=false -input=false && tofu fmt -check && tofu validate && tofu test; cd -

Expected: Success! The configuration is valid. and Success! 1 passed, 0 failed. Only now remove .terraform/ and .terraform.lock.hcl from the module directory, as no module here commits them.

  • [ ] Step 5: Record this plan's decisions in the spec

Append to docs/superpowers/specs/2026-10-04-region-ha-box-placement-design.md:


## 24. Notes from plan 1b (2026-10-06)

Plan `docs/superpowers/plans/2026-10-06-region-ha-1b-claim-readers.md`
builds the parts of rev 5.4 that need none of L1 to L20: enrollment
evidence, §19's corrections, the agent publishing its identity record,
the fence bound to the roster's `cloud`, volume records and the discharge
check, the edges file following `op.exec`, and root's half of §5.10. All
of it is dormant behind `CARLOS_REGION_HA`. It keeps every property above
and settles five details:

1. **Enrollment refuses one instance under two references (§5.1).** The
   record's `box` field is not covered by AWS's signature, so box A's
   record copied to B's key with that field rewritten still verifies.
   `boxid.Verifier.Enroll` refuses an instance the roster holds under another
   reference, as well as a reference the roster holds for another
   instance, counting entries in every state.
2. **A positive stamp is the relocatable mark (§18's open question).**
   Only relocatable hosts with a claim, and their aliases and served
   catch-alls, carry a stamp, so the edge's report
   of two boxes at one positive stamp already concerns them alone. No
   separate marker.
3. **The edges file's claim reads (§5.4).** A relocatable host with no
   claim is published as before, unstamped: §8's gate admits a second box
   only once every host has a claim. A read that fails keeps the last
   decision. Each read stands 10 minutes (H9), at most 256 reads a pass.
4. **"The first record taken at enrollment" (§5.9)** is the earliest
   volume record for the enrollment, observed within one hour of the
   roster's enrollment time. A later first record cannot speak for a
   volume attached and removed before it, so the chores stay open. And
   (controller ruling, 2026-10-06) the newest record must be no more
   than two hours older than the termination observation; otherwise the
   chores stay open for the operator path.
5. **No volume record from a `terminated` read.** Its mappings are empty
   (§8 step 5's dev test), and recording it would list every volume as
   removed.

It also removes plan 1a's `Bind`, `BindKey`, `Verifier.Verify` and
`activator.WarmCopyTrusted`, and makes every box-evidence prefix
create-only and never deleted in the deployment bucket's policy.

Controller rulings of 2026-10-06, on Paul's behalf: the edges file
withholds a relocatable host when `op.exec` names another box or none
(§5.4 as written); root's §5.10 refusal stays behind `CARLOS_REGION_HA`
until claim D6 lands, then becomes unconditional; the `*.bind.json`
statement and `operator_role_arns` stay until no root sets the variable.
  • [ ] Step 6: Run the whole gate

Run: go build ./... && make test, then GOOS=darwin go build ./... && GOOS=windows go build ./... && GOOS=linux GOARCH=arm64 go build ./.... Expected: gofmt -l . clean and every package ok, with the sandbox off (see Global Constraints for the socket tests that fail inside it). make test takes up to thirty minutes. If a package unrelated to this plan fails, do not fix it here and do not skip it: run that package alone, confirm it also fails on origin/main, and report both results.

  • [ ] Step 7: Commit
git add infra/modules/deployment-buckets docs/superpowers/specs/2026-10-04-region-ha-box-placement-design.md
git commit -m "infra: box evidence prefixes are create-only and never deleted; spec: plan 1b's decisions

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>"
  • [ ] Step 8: Push and read CI
git push origin region-ha-1b
amadan ci status carlos/platform -branch region-ha-1b

Expected: exit 0 (pass). Exit 2 is pending: wait and read it again. Exit 1 is a failure: read the log, fix it, and do not ask for the branch to be merged. Merging is Paul's call, through amadan branch merge, never a push to main.

---

Waits for the claim

Everything spec 1 rev 5.4 asks for that this plan does not build, with the request it waits on (§9). Item numbers are §19's revised list for plan 1b.

| What | Waits for | Why it cannot be built now | |---|---|---| | Relocation: the relocating op kind, its phases, start CAS, replacement table, source-loss conversion, two-replacement bound (§5.8; item 7) | L1 | a new claim op kind | | The release sequence's writers-stopped check (§5.8 step 1) | L10 | changes stopevidence's contract | | Coverage released on the first probe admission (§5.8) | L9 | changes hostclaim.AdmitWriter's admission rules | | A relocation start despite a regional park-generation chore | L11 | changes hostclaim.StartGuard (CC11) | | Building the prepared generation from a pinned set; the local install receipt | L16, L17 (and L1) | shared with moves' landing; changes moves | | Reconcile's row cases, held participant rows, former executor rows, the 10-minute sweep as an exception to C4 (§5.3; item 2) | L12 | changes claim C4 and moves §9.6 | | Create names a box: chooseBox(eligibleBoxes(…)) in the create intent's selected_executor (§5.2; item 1) | L8 (claim P5-D6, the claim's plan 5) | changes create's step 2 and step 4 CASes | | A move into a multi-box region naming its target box (§5.2) | L15 | changes moves | | A target terminated mid-move taken off the move by the fence | L14 | changes moves | | Writing fence records to control/box-fences/, the lost list, the fence CAS (§5.9) | L2, L4 | a lost-box fence with no operator is a claim rule change | | Discharging park-local chores in the fence CAS from boxvolumes.Discharge | L3 | a claim D5 change | | The census excluding a fenced box and a console on it; the compute binding (§5.9) | L5 | a claim D7 change | | A stale console report accepted for box-started relocations (H18) | L20 | a claim D7 change | | Root's §5.10 refusal made unconditional, no longer behind CARLOS_REGION_HA (controller ruling 3) | claim D6 (with L6) | until D6's controller writes .hostexec/, the flag is the only thing keeping a one-box region unchanged | | The participant's half of §5.10: write pending, then refuse to acknowledge while root's marker exists | L6 (and claim D6's controller, not on main) | .hostexec/ is D6's; nothing writes it yet | | Enrollment itself: the console verifying records with boxid.Enroll into control/host-actors.json, the tofu check, refusals | L7 (claim D7's roster, not on main) | there is no roster to write | | The volume recorder loop: describing every enrolled box in the region on the 10-minute sweep and on shutting-down | L7 | records are keyed by enrollment, which only the roster assigns | | eligibleBoxes' caller, reading the roster and the region's AWS region from it (§18's second correction) | L7, L8 | no roster | | Auto Scaling membership confirmations, and the cross-region confirmer for an emptied region (H16, H17), with the cross-region IAM statement | L19 (and spec 4's group and the roster's regions entry) | the console may accept them only under L19; nothing to confirm yet | | The files publisher as a generation-bound writer, the copied manifest at each cold admission, retirement by park-generation (§5.6); files_durability on routes and the agent starting hostfiles.Run (item 5) | L13 | a second ordinary-replication exception to claim C3 | | The multi-box gate's roster half and the HA floor (§8; item 8) | L7 and claim D1's floor | no roster, no floor on main | | The joint model with two boxes per region | L18 | the model belongs to the claim and moves sessions | | Reading the termination queue and the placer loop | spec 3 (§12), then L2 to L4 | nothing can be done with an event before a fence can be written |

Follow-up, not waiting for the claim. Retire the OnlyOperatorsWriteBinds statement and the operator_role_arns variable in deployment-buckets once no consuming root sets the variable (controller ruling 5).

Not waiting for the claim, and not in this plan. Plan 1a's "After the build" items 9 (a streaming put, lifting the 64 MiB limit, and the data bucket's conditional-write policy for files/), 11 (one inotify instance per agent) and 12 (a size cap in StagePlatform, a rate limit on Commit) need no claim change. They matter only once the files publisher is switched on, which waits for L13, so they belong with that plan.

Turning the flag on. CARLOS_REGION_HA must be set in /etc/carlos/host.env, where both the agent and root's carlos-system-update.service read it. A drop-in for carlos-edge.service alone would turn on the edges file's claim reads and the box record but not root's §5.10 refusal. It is on only for a value strconv.ParseBool reads as true (1, t, true). Two things must be in place before any box sets it: the per-pass deadline on the edges file's claim reads (built: see "After the build", item 4), and a roster-wide check that no instance holds more than one reference (enrollment's per-record check cannot see two entries the roster already holds; it belongs with L7's roster).

What the dry run could not check

  • readIdentityRecord against a real box's IMDS. Plan 1a verified the regional certificates and the signature on the dev box; this plan's reader was tested only against an in-process fake. Before the branch lands, on the dev box with CARLOS_REGION_HA=1 set for the agent, confirm control/boxes/<label>.json appears once and that boxid.Enroll accepts its bytes with the dev account and region.
  • termfence.DescribeInstance and boxvolumes.Observe against EC2. Both were tested with fakes and the SDK's own types. The first live use is spec 3's placer.
  • Root's marker under a real carlos-system-update.service. The apply's tests run the whole Run with faked systemctl; whether /etc/carlos/host.env carries CARLOS_REGION_HA on a given box is a deployment fact.
  • The tests listed under Global Constraints that open Unix sockets, inside the sandbox.

Self-review

  • Spec coverage. §19's corrections: Tasks 1, 2 and 4 (box records as evidence, the bind's tofu check moved into enrollment, eligibleBoxes from the roster, chooseBox's tie-break, WarmCopyTrusted, termfence on cloud with no volume input). §5.1 enrollment evidence: Tasks 1 and 3. §5.4: Task 6. §5.9 under H19: Tasks 4 and 5. §5.10 root's half: Task 7. §6 storage: Task 8. §10's unit tests that need no claim: enrollment evidence (Task 1), eligibleBoxes and choose (Task 2), termination-fence acceptance (Task 4: NotFound, wrong account, wrong region, stopped, an event alone, a describe alone, shutting-down offered as evidence, accepted with no volume record), the volume recorder and chore discharge, including the 2-hour last-record window (Task 5), root's refusal on any controller record (Task 7). Everything else is in "Waits for the claim".
  • Placeholders. None: every code step is a whole file or an exact replacement.
  • Type consistency. boxid.Cloud is defined in Task 1 and used by Tasks 2, 4 and 5; boxid.Identity is kept through Task 3 and removed in Task 4, its last user; regionHAEnabled is defined in Task 3 and used by Tasks 6 and 7; termfence.DescribeInstance (Task 4) returns the *ec2.DescribeInstancesOutput that boxvolumes.Observe (Task 5) reads. The dry run applied the tasks in order to a clean tree and built each.
  • Review Focus. Each of the five has a test in its owning task, named above, and each was watched to fail with its guard removed.

Copy waiting for Paul

Controller ruling 6 took this out of Task 8 until Paul has reviewed it (the copy-review skill). Proposed for infra/modules/region-host/README.md, replacing the two paragraphs that begin "The root volume is deleted when the instance is terminated." and "Open item for region HA plan 1b":

The root volume is deleted when the instance is terminated. The fence
does not depend on that: a `terminated` read is enough. It decides only
whether a dead box's local cleanup can be skipped automatically, so keep
it, and keep Recycle Bin, Data Lifecycle Manager and AWS Backup rules off
platform box volumes.

The box role's `ec2:DescribeInstances` has no region condition, because
the infrastructure inventory reads every region, and a second, narrower
Allow could not take that away. Region HA keeps to the box's own region
in code instead: every describe it makes for a fence or a volume record
is pinned to the region the box was enrolled in.

Astra plan round 1: dispositions

Findings are in CODEX-PLAN-REGION-HA-1B-R1.md (reviewed at 856767d8). All seven were checked against the tree and accepted. The fixes were dry-run again from the plan's own text: a script applied Tasks 1 to 8 in order to a clean copy of bf813540 and each built, vetted and cross-built; the tests of every task passed (with the sandbox's eight known socket failures in internal/host, unchanged); and each new test failed with its mechanism removed. No controller ruling was needed.

| # | Finding | Disposition | |---|---|---| | 1 | P1 Task 8: Step 2's cleanup leaves Step 4 uninitialized | fixed: Step 2 keeps .terraform/; Step 4 runs tofu init again (a no-op when still initialized) before validate and test, and only then removes it | | 2 | P2 Task 6: aliases and served catch-alls lose the stamp, so a dead box's alias ties and can win | fixed: both take their primary's stamp from the served map; TestEdgesFileStampsAliasesAndCatchAllsWithTheirPrimary (killed by removing either stamp), and internal/edge's TestHomeMapAnAliasFollowsTheNewerInstallation pins the edge's half. With the flag off every stamp is 0, so the file is unchanged | | 3 | P2 Task 7: refused is terminal for the promote, so participation suppresses the release for good | fixed: participation and a marker failure are recorded as deferred, which the agent's leg re-requests on its backoff ladder; TestApplyDeferredForParticipationAppliesOnceItEnds (deferred, not terminal, then applied once the record is gone; killed when recorded as refused) | | 4 | P2 Task 6: failing reads keep their place at the front and starve healthy claims | fixed: an entry records its last attempt separately from its last decision; a failed read moves the attempt and keeps the decision, so it goes to the back of the queue; TestEdgesClaimsAFailingClaimDoesNotStarveTheRest (256 failing claims then a healthy one; killed by the old bookkeeping) | | 5 | P2 Task 6: 256 reads a pass at the timer's 5 to 6 minutes cannot refresh 2,000 hosts in 10 minutes | fixed: edgesLoopState adds a pass every 30 seconds while claim reads are due (flag on only); TestEdgesLoopRefreshesEveryClaimWithinTheTTL simulates the loop's schedule for three hours with 2,000 hosts and requires every claim re-read within 15 minutes (killed with the extra passes off) | | 6 | P2 Task 5: incomplete, non-recordable or unknown-version records can authorize discharge | fixed: Discharge validates every stored record of the enrollment (version 1, a state that carries mappings, a non-empty mapping, every volume with an id and device); anything else keeps the chores open. TestDischargeRefusesStoredRecordsThatAreNotEvidence writes each bad record through the store and lists it back, placed so that no removal masks it; each case killed by removing its check | | 7 | P2 Task 8: the policy test checks resources only | fixed: each statement is compared whole (effect, principal, actions, condition, resources); five mutations (Deny to Allow, a dropped delete action, the wrong condition key, a narrower principal, a wider prefix) and main's policy each fail it |

Astra plan round 2: dispositions

Findings are in CODEX-PLAN-REGION-HA-1B-R2.md (reviewed at 161728ab). Round 2 marked R1-1 to R1-4 and R1-7 resolved and R1-5 and R1-6 partly resolved, and raised two findings, both checked and accepted. Tasks 5 and 6 were dry-run again from the plan's text, with every task applied in order to a clean copy of bf813540, built, vetted, cross-built and tested; each check below was removed in turn and its test watched to fail.

| # | Finding | Disposition | |---|---|---| | R2-1 | P2 Task 6: claim passes bypass the publication retry backoff | fixed: a claim pass is gated on the schedule's retryAt, and its outcome goes through edgesFileSchedule.done (a failure advances the backoff, a success clears it; the timer does not move). TestEdgesLoopClaimPassesObeyTheRetryBackoff runs the loop's schedule against a bucket that fails for 20 minutes then recovers: 12 attempts while failing (40 unthrottled), and the 30-second passes back after recovery. It fails with the retryAt gate removed, with claim outcomes not recorded, and with claim passes off | | R2-2 | P2 Task 5: three validation checks survive removal | fixed: three more stored cases, each failing exactly one check: running with no volume (empty mapping), terminated with its root mapping (state), a volume with an id but no device (device). Removing each of the five checks (state, empty mapping, volume id, device, version) fails its own case |

After the build

The final whole-branch review (2026-10-06) raised seven findings, all ruled by the controller and fixed on the branch:

  1. boxvolumes: a shutting-down record whose DeleteOnTermination volumes show detaching (EC2 detaching them on the way to deletion) is evidence, not a refusal. In any other state, and for a volume whose flag is false, only attached and attaching count.
  2. boxid.Verifier.Enroll: a roster entry for the same reference with no proven cloud yet does not refuse it; only an entry naming another instance does.
  3. Enroll's doc says the record's box is not signed, so for a reference the roster does not yet hold the caller must bind instance to reference independently (operator approval, the tofu id, or H16's group confirmation). The agent's publishBoxRecord checks a record already under its label against its own identity and logs loudly, once per run, when it is signed for another instance or cannot be read.
  4. The edges file's claim reads share a 20-second deadline per pass; a read it cuts short keeps the last decision, and reads not reached wait for a later pass. Failed reads are logged on the first failing pass, every 60th after, and on recovery. A host hostclaim.Key rejects is never read and publishes as before.
  5. regionHAEnabled accepts only what strconv.ParseBool reads as true; "", 0, false, off, no and anything unparseable are off.
  6. boxid's "unknown field" test uses a fully signed record plus one extra field, so only DisallowUnknownFields can refuse it.
  7. Spec §24 names Verifier.Enroll and includes aliases and served catch-alls among the stamped entries; this plan says where the flag is set and what must exist before it is turned on.