carlos / platform Public

#162: the shadow-record bypass — key the record map by identity, not by host

Adversarial verification broke the four tenancy rails 1a5f819f added and
reproduced the bypass. The record map both passes read was keyed by HOST,
so it had one slot per host. A writer holding nothing but their own prefix
wrote a record there carrying the VICTIM's host — their account, their app,
so it agreed with its own path — and a bucket listing is lexicographic, so
a later-sorting prefix took the victim's slot. Three things followed, each
a P1: the attacker's replica passed all four rails against its own shadow
and minted, which aliases the victim's public host to the attacker's
process on that box; the victim's own siblings.json named the attacker with
a dial address and a live pair key, both sides agreeing because both derive
the label from the attacker's account; and the victim's LEGITIMATE replica
was refused as "another account's", a denial of service on the real pair
from a record the victim cannot see and cannot delete.

The map is now keyed by the record's identity, RecordKey(account, app,
host), and a replica's home is looked up inside the replica's OWN account
and app. That makes rails one and two true by construction rather than by
comparison, and removes the collision itself: no record can occupy
another's slot. ReconcileConfig.OnRecords says so, because the key is the
safety property its readers inherit.

Both passes now ask one function, provision.PairableHome — they defend the
same property from two sides, so a rail only one of them holds is a rail
the other bypasses. Rails three (a home is not itself a replica) and four
(the placement is named in the home's `replicas`) stay explicit checks.

A fifth rail, found while testing the fix: the home must be the record that
OWNS its host — the first-sorting claimant, the same arbitration the pass
already makes when two records claim one host. Identity keying alone does
not close the mint: the attacker's shadow is a perfectly good home for the
attacker's own replica, so the rails pass and the victim's host is aliased
anyway. The loser of the host arbitration serves nothing anywhere, so a
replica vouched for by one is vouched for by a record with no route, and
the alias it mints diverts the winner's public host. buildSiblings asks the
same question of the file's own host, in both loops: it writes files keyed
by host and has no route arbitration of its own, so without it a record
claiming another account's replica host hands that instance a file naming a
stranger as its home.

Failure handling splits, per the same verification. Rails that can never
fail on a door-written record clear the link (SetReplicaOf ""), and the
domains pass sweeps the alias next run — protecting those leaves a row
nobody vouched for serving an account's public host indefinitely for no
fail-safe benefit. Not reg.Delete: removal carries a grant purge and none
of this is a retirement the bucket asked for. An unreadable home and the
opt-in rail keep `protect`: a bucket blip must not sweep a live placement,
and an OLD CONSOLE legitimately strips `replicas` by round-tripping a home
record, so a console deploy must not be able to take a placement down.

republishRetired now takes the pair's label off the entry buildSiblings
wrote rather than re-deriving it from all[host].Account — that was the last
host-keyed read in the file, and deriving one label twice is how two
derivations drift.

Spec §6 gains a paragraph on what these rails are and are NOT: they defend
an operator's typo and a prefix-scoped writer, and they are not a tenancy
boundary against a writer holding bucket-wide s3:PutObject (a region box's
IAM role does), who can author a record in the victim's OWN prefix where
every rail passes trivially — or a plain instance record claiming the
victim's host, with no replica involved at all.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
claude agent, operated by paul@keymail.dev adf29ebdb1a94c6995673c4a9087c936adb6a75b parent 1a5f819
5 files changed, +489 −115
  • docs/superpowers/specs/2026-09-21-replica-placements-design.md +24 −0
  • internal/host/siblings.go +66 −31
  • internal/host/siblings_test.go +113 −39
  • internal/provision/reconcile.go +156 −42
  • internal/provision/reconcile_test.go +130 −3

CI passed — run details

diff --git a/docs/superpowers/specs/2026-09-21-replica-placements-design.md b/docs/superpowers/specs/2026-09-21-replica-placements-design.md
index 7dda807c1..96426072b 100644
--- a/docs/superpowers/specs/2026-09-21-replica-placements-design.md
+++ b/docs/superpowers/specs/2026-09-21-replica-placements-design.md
@@ -405,6 +405,30 @@ differs from the stored one. So:
build carrying the fields. That is a sequencing constraint on the
deployment, and it belongs in the branch handoff.
+**What the box-side rails are, and what they are not.** Both box passes —
+the reconcile mint/converge and the siblings writer — ask the same
+questions of the fleet-wide record map before they pair anything
+(`provision.PairableHome`), because a door is not a guarantee about what is
+in the bucket. The map is keyed by the record's identity
+(`RecordKey(account, app, host)`, not by host), so the home is looked up
+*inside the replica's own account and app*: the tenancy halves are the key
+rather than a comparison. On top of that the home must be the record that
+**owns** its host — the first-sorting claimant, the same arbitration the
+pass already makes when two records claim one host, since the loser's
+record serves nothing anywhere — must not itself be a replica, and must
+name the placement in its `replicas`. What that defends is an operator's
+typo and a **writer scoped to its own prefix**: neither can reach into
+another account's pairing, take a live pair away from its owner, or land in
+another account's `siblings.json` with a dial address and a pair key. What
+it is **not** is a tenancy boundary against a writer holding bucket-wide
+`s3:PutObject`, which every region box's IAM role does hold. Such a writer
+can author a record in the **victim's own prefix**, where every rail passes
+trivially — and could as easily write a plain instance record claiming the
+victim's host, with no replica involved at all. Host ownership is decided
+at the console's create door (the manifest domain, the custom-domain claim)
+and by who can write which bucket prefix. The pass can refuse only what it
+can see, and what it sees is records, not the authority behind them.
+
## 7. Sibling addressing (Ask 3): no new channel
An instance reaches a sibling by dialling **that box's EIP with the real
diff --git a/internal/host/siblings.go b/internal/host/siblings.go
index f3639826a..f2a947fbc 100644
--- a/internal/host/siblings.go
+++ b/internal/host/siblings.go
@@ -17,7 +17,6 @@ import (
"log"
"os"
"path/filepath"
- "slices"
"sort"
"sync/atomic"
@@ -55,6 +54,12 @@ type siblingEntry struct {
Placement string `json:"placement,omitempty"`
Addr string `json:"addr"`
Keys []string `json:"keys"`
+ // label is the pair's identity (pairLabel), carried on the entry rather
+ // than re-derived later: republishRetired needs it to remember a pair's
+ // derivative, and deriving it twice off two different lookups is how the
+ // two derivations drift. Never rendered — it is the HMAC message, and
+ // the file is read by the tenant's own process.
+ label string
}
// pairLabel is the HMAC message that makes a key belong to ONE pair:
@@ -87,7 +92,13 @@ func deriveSiblingKey(meshKey []byte, label string) string {
//
// `all` is the fleet-wide record map ReconcileConfig.OnRecords hands the
// pass — both sides of a pair are in it even though only one of them is in
-// this box's scope, which is the whole reason the callback exists.
+// this box's scope, which is the whole reason the callback exists. It is
+// keyed by the record's IDENTITY (provision.RecordKey), never by host, and
+// nothing here may look a record up by host alone: one slot per host is
+// exactly what let a record written in one account's prefix be read as
+// another account's home. The returned map IS keyed by host, because a file
+// on a box is addressed by host — so every record that reaches it has been
+// asked whether it owns that host.
//
// eipOf is the edge's home map: the sibling box's EIP, or false for a box
// that has published no edges file yet. keys is the mesh key list, current
@@ -115,30 +126,48 @@ func buildSiblings(all map[string]provision.Record, eipOf func(host string) (str
// A home that has opted in gets a file even before any replica record
// exists, and even when no replica is reachable: "no replicas" and "no
// file" must stay different answers (spec §7).
- for host, rec := range all {
- if rec.Role == "replica" || len(rec.Replicas) == 0 {
+ //
+ // Written under the record's HOST, which is what a file on a box is
+ // addressed by — so the record must be the one that OWNS that host
+ // (provision.OwnsItsHost). Without that, a record naming another
+ // account's host, written in any prefix, decides what goes in the file
+ // the victim's own box writes for the victim's own instance.
+ for _, rec := range all {
+ if rec.Role == "replica" || len(rec.Replicas) == 0 || !provision.OwnsItsHost(all, rec) {
continue
}
- out[host] = siblingsFile{Role: "home", Host: host, Placement: rec.Region}
+ out[rec.Host] = siblingsFile{Role: "home", Host: rec.Host, Placement: rec.Region}
}
- for host, rec := range all {
+ for _, rec := range all {
if rec.Role != "replica" || rec.Home == "" {
continue
}
- // The four CROSS-RECORD rails of the console's create door (spec §6),
- // asked again here because a bucket record is not a record the door
- // wrote: every region box's IAM role can put one there and an
- // operator can hand-write one. Pairing on the home record's mere
- // EXISTENCE would write the HOME's file — on the home's own box, in
- // the home's own account — naming a stranger's instance with a dial
- // address and a live pair key, telling the home's app to grant
- // leases to it and accept drains from it. Nothing to pair with is
- // the answer to all four: a file naming an address nobody vouched
- // for is worse than no file.
- homeRec, ok := all[rec.Home]
- if !ok || homeRec.Account != rec.Account || homeRec.App != rec.App ||
- homeRec.Role != "" || !slices.Contains(homeRec.Replicas, rec.Region) {
+ // The replica's own file is written under the replica's host, so the
+ // same ownership question is asked of it: a record in any prefix
+ // naming ANOTHER account's replica host, with a perfectly valid home
+ // of its own behind it, would otherwise hand that instance a file
+ // naming a stranger as its home. The reconcile pass settles this for
+ // routes with its duplicate-host arbitration; this pass has no
+ // routes, so it asks the map the same question.
+ if !provision.OwnsItsHost(all, rec) {
+ continue
+ }
+ host := rec.Host
+ // The CROSS-RECORD rails of the console's create door (spec §6),
+ // asked through the same function the reconcile pass asks — the two
+ // passes defend the same property from two sides, so a rail only one
+ // of them holds is a rail the other bypasses. A bucket record is not
+ // a record the door wrote: every region box's IAM role can put one
+ // there and an operator can hand-write one. Pairing on the home
+ // record's mere EXISTENCE would write the HOME's file — on the
+ // home's own box, in the home's own account — naming a stranger's
+ // instance with a dial address and a live pair key, telling the
+ // home's app to grant leases to it and accept drains from it.
+ // Nothing to pair with is the answer to every one of them: a file
+ // naming an address nobody vouched for is worse than no file.
+ homeRec, why, _ := provision.PairableHome(all, rec)
+ if why != "" {
continue
}
label := pairLabel(homeRec.Account, rec.Home, host)
@@ -156,7 +185,7 @@ func buildSiblings(all map[string]provision.Record, eipOf func(host string) (str
hf := out[rec.Home]
if a, ok := addr(host); ok {
hf.Replicas = append(hf.Replicas, siblingEntry{
- Host: host, Placement: rec.Region, Addr: a, Keys: ks,
+ Host: host, Placement: rec.Region, Addr: a, Keys: ks, label: label,
})
}
out[rec.Home] = hf
@@ -164,7 +193,7 @@ func buildSiblings(all map[string]provision.Record, eipOf func(host string) (str
rf := siblingsFile{Role: "replica", Host: host, Placement: rec.Region}
if a, ok := addr(rec.Home); ok {
rf.Home = &siblingEntry{
- Host: rec.Home, Placement: homeRec.Region, Addr: a, Keys: ks,
+ Host: rec.Home, Placement: homeRec.Region, Addr: a, Keys: ks, label: label,
}
}
out[host] = rf
@@ -226,7 +255,7 @@ func (w *siblingsWriter) pass(all map[string]provision.Record, eipOf func(host s
return nil
}
built := buildSiblings(all, eipOf, keys)
- w.republishRetired(built, all)
+ w.republishRetired(built)
routes, err := w.reg.List()
if err != nil {
@@ -294,7 +323,12 @@ func (w *siblingsWriter) dirFor(rt registry.Route) string {
// republishRetired folds the previous pass's derivative into each pair's key
// list and remembers this pass's current one. It mutates built in place: the
// list the file is rendered from is the list the pair's memory covers.
-func (w *siblingsWriter) republishRetired(built map[string]siblingsFile, all map[string]provision.Record) {
+//
+// The pair's identity comes off the entry buildSiblings wrote (its label),
+// never from a second lookup in the record map. It used to re-derive the
+// label from all[host].Account — a host-keyed read, which is exactly the
+// read the shadow bypass turned against the pass.
+func (w *siblingsWriter) republishRetired(built map[string]siblingsFile) {
next := make(map[string]string, len(w.last))
remember := func(label string, ks []string) []string {
if len(ks) == 0 {
@@ -312,13 +346,11 @@ func (w *siblingsWriter) republishRetired(built map[string]siblingsFile, all map
switch f.Role {
case "home":
for i := range f.Replicas {
- label := pairLabel(all[host].Account, host, f.Replicas[i].Host)
- f.Replicas[i].Keys = remember(label, f.Replicas[i].Keys)
+ f.Replicas[i].Keys = remember(f.Replicas[i].label, f.Replicas[i].Keys)
}
case "replica":
if f.Home != nil {
- label := pairLabel(all[f.Home.Host].Account, f.Home.Host, host)
- f.Home.Keys = remember(label, f.Home.Keys)
+ f.Home.Keys = remember(f.Home.label, f.Home.Keys)
}
}
built[host] = f
@@ -452,15 +484,18 @@ func siblingInputs() (eipOf func(host string) (string, bool), keys [][]byte, ok
// both reconcile legs carry it) and read by instancesLoop after the pass
// returns. An atomic.Value rather than a field on the loop because the two
// legs share one callback and run under reconcileMu.
-var lastRecords atomic.Value // map[string]provision.Record
+var lastRecords atomic.Value // map[string]provision.Record, keyed by RecordKey
// rememberRecords is the OnRecords callback. It COPIES: the map belongs to
// the reconcile pass, the callback runs while reconcileMu is held, and the
-// reader is another goroutine's tick.
+// reader is another goroutine's tick. The keys are carried through exactly
+// as the pass wrote them (provision.RecordKey — account, app and host), the
+// key being the property that keeps one account's record out of another's
+// slot.
func rememberRecords(all map[string]provision.Record) {
cp := make(map[string]provision.Record, len(all))
- for host, rec := range all {
- cp[host] = rec
+ for key, rec := range all {
+ cp[key] = rec
}
lastRecords.Store(cp)
}
diff --git a/internal/host/siblings_test.go b/internal/host/siblings_test.go
index 61b62baef..330d0a053 100644
--- a/internal/host/siblings_test.go
+++ b/internal/host/siblings_test.go
@@ -11,18 +11,31 @@ import (
"github.com/carlosframework/platform/internal/registry"
)
+// recordMap keys records the way ReconcileConfig.OnRecords hands them over:
+// by the record's IDENTITY (account, app and host), never by host alone. The
+// key is the safety property — a host-keyed map has one slot per host, so a
+// record written in one account's prefix can stand in for another's, which
+// is what TestBuildSiblingsRefusesAShadowOfTheVictimsHost is about.
+func recordMap(recs ...provision.Record) map[string]provision.Record {
+ all := make(map[string]provision.Record, len(recs))
+ for _, r := range recs {
+ all[provision.RecordKey(r.Account, r.App, r.Host)] = r
+ }
+ return all
+}
+
// The pair a home and its replica make: each side's file names the other,
// with the other's dial address and the key of THIS pair (spec §7). The two
// files are written by two different boxes, from the same fleet-wide record
// map, so the derivation must land on the same bytes on both sides — that
// last assertion is the whole protocol.
func TestBuildSiblingsPairsHomeAndReplica(t *testing.T) {
- all := map[string]provision.Record{
- "acme.go.tito.io": {Host: "acme.go.tito.io", Account: "bab", App: "tito", Region: "ie",
+ all := recordMap(
+ provision.Record{Host: "acme.go.tito.io", Account: "bab", App: "tito", Region: "ie",
Steering: "latency", Replicas: []string{"uk"}},
- "r7x2.go.tito.io": {Host: "r7x2.go.tito.io", Account: "bab", App: "tito", Region: "uk",
+ provision.Record{Host: "r7x2.go.tito.io", Account: "bab", App: "tito", Region: "uk",
Role: "replica", Home: "acme.go.tito.io"},
- }
+ )
eip := func(host string) (string, bool) {
return map[string]string{"acme.go.tito.io": "198.51.100.4", "r7x2.go.tito.io": "203.0.113.9"}[host], true
}
@@ -49,11 +62,11 @@ func TestBuildSiblingsPairsHomeAndReplica(t *testing.T) {
// flight publishes both derivatives, current first, because the two sides'
// files are rewritten by two passes minutes apart.
func TestBuildSiblingsKeysArePerPairAndRotate(t *testing.T) {
- all := map[string]provision.Record{
- "acme.go.tito.io": {Host: "acme.go.tito.io", Account: "bab", App: "tito", Region: "ie", Steering: "latency", Replicas: []string{"uk", "de"}},
- "r7x2.go.tito.io": {Host: "r7x2.go.tito.io", Account: "bab", App: "tito", Region: "uk", Role: "replica", Home: "acme.go.tito.io"},
- "r8y3.go.tito.io": {Host: "r8y3.go.tito.io", Account: "bab", App: "tito", Region: "de", Role: "replica", Home: "acme.go.tito.io"},
- }
+ all := recordMap(
+ provision.Record{Host: "acme.go.tito.io", Account: "bab", App: "tito", Region: "ie", Steering: "latency", Replicas: []string{"uk", "de"}},
+ provision.Record{Host: "r7x2.go.tito.io", Account: "bab", App: "tito", Region: "uk", Role: "replica", Home: "acme.go.tito.io"},
+ provision.Record{Host: "r8y3.go.tito.io", Account: "bab", App: "tito", Region: "de", Role: "replica", Home: "acme.go.tito.io"},
+ )
eip := func(host string) (string, bool) { return "203.0.113.9", true }
got := buildSiblings(all, eip, [][]byte{[]byte("cur"), []byte("prev")})
home := got["acme.go.tito.io"]
@@ -73,10 +86,10 @@ func TestBuildSiblingsKeysArePerPairAndRotate(t *testing.T) {
// own file is still written, so the app can tell "no replicas" from "no
// file" (spec §7).
func TestBuildSiblingsOmitsASiblingWithNoEIPAndStillWritesAnEmptyHome(t *testing.T) {
- all := map[string]provision.Record{
- "acme.go.tito.io": {Host: "acme.go.tito.io", Account: "bab", App: "tito", Region: "ie", Steering: "latency", Replicas: []string{"uk"}},
- "r7x2.go.tito.io": {Host: "r7x2.go.tito.io", Account: "bab", App: "tito", Region: "uk", Role: "replica", Home: "acme.go.tito.io"},
- }
+ all := recordMap(
+ provision.Record{Host: "acme.go.tito.io", Account: "bab", App: "tito", Region: "ie", Steering: "latency", Replicas: []string{"uk"}},
+ provision.Record{Host: "r7x2.go.tito.io", Account: "bab", App: "tito", Region: "uk", Role: "replica", Home: "acme.go.tito.io"},
+ )
got := buildSiblings(all, func(string) (string, bool) { return "", false }, [][]byte{[]byte("cur")})
home, ok := got["acme.go.tito.io"]
if !ok {
@@ -91,21 +104,24 @@ func TestBuildSiblingsOmitsASiblingWithNoEIPAndStillWritesAnEmptyHome(t *testing
// produce no file at all, not an empty one (global constraint: additive
// only).
func TestBuildSiblingsIgnoresAnAccountWithNoReplicas(t *testing.T) {
- all := map[string]provision.Record{
- "plain.go.tito.io": {Host: "plain.go.tito.io", Account: "zed", App: "tito", Region: "ie"},
- }
+ all := recordMap(provision.Record{Host: "plain.go.tito.io", Account: "zed", App: "tito", Region: "ie"})
if got := buildSiblings(all, func(string) (string, bool) { return "203.0.113.9", true }, [][]byte{[]byte("cur")}); len(got) != 0 {
t.Errorf("buildSiblings = %+v, want nothing for an account with no replicas", got)
}
}
-// The four cross-record rails of the console's create door (spec §6), held
-// here too because a bucket record is not a record the door wrote: any
-// region box's IAM role can put one, and an operator can hand-write one.
-// Pairing an unvouched-for record would write the VICTIM's siblings file
-// naming a stranger's instance, with a dial address and a live pair key —
-// telling the victim's own app to grant leases to it and accept drains from
-// it. Each case below is a record that pairs today and must not.
+// The cross-record rails of the console's create door (spec §6), held here
+// too because a bucket record is not a record the door wrote: any region
+// box's IAM role can put one, and an operator can hand-write one. Pairing an
+// unvouched-for record would write the VICTIM's siblings file naming a
+// stranger's instance, with a dial address and a live pair key — telling the
+// victim's own app to grant leases to it and accept drains from it. Each
+// case below is a record that pairs today and must not.
+//
+// The account and app halves reach this as "no home record at all": the map
+// is keyed by identity and the home is looked up inside the replica's own
+// account and app, so a home in another tenancy is not a home this replica
+// can name. That is the rail, stated as a lookup rather than a comparison.
func TestBuildSiblingsRefusesAnUnvouchedForReplica(t *testing.T) {
const (
victim = "acme.go.tito.io"
@@ -143,10 +159,7 @@ func TestBuildSiblingsRefusesAnUnvouchedForReplica(t *testing.T) {
},
} {
t.Run(tc.name, func(t *testing.T) {
- all := map[string]provision.Record{victim: homeRec, tc.replica.Host: tc.replica}
- for _, r := range tc.extra {
- all[r.Host] = r
- }
+ all := recordMap(append([]provision.Record{homeRec, tc.replica}, tc.extra...)...)
got := buildSiblings(all, eip, [][]byte{[]byte("cur")})
if _, ok := got[tc.replica.Host]; ok {
t.Errorf("the unvouched-for replica got a file of its own: %+v", got[tc.replica.Host])
@@ -175,6 +188,67 @@ func TestBuildSiblingsRefusesAnUnvouchedForReplica(t *testing.T) {
}
}
+// The SHADOW record, this pass's half (the reconcile half is
+// TestReconcileShadowHomeRecordCannotDisplaceTheVictims). A writer who holds
+// nothing but their own prefix writes a record there carrying the VICTIM's
+// host — their account, their app, so it agrees with its own path — and a
+// replica of it. Every rail then passes against that shadow, and the
+// attacker lands in the VICTIM's own siblings file with a dial address and a
+// live pair key: the victim's app is told to grant leases to a stranger.
+//
+// A host belongs to ONE record — the one the bucket listing reaches first,
+// which is the arbitration the reconcile pass already makes when two records
+// claim a host. A replica may only be vouched for by the record that WON the
+// host it names, because that is the record whose route the public host is
+// actually served from.
+func TestBuildSiblingsRefusesAShadowOfTheVictimsHost(t *testing.T) {
+ const victim = "acme.go.tito.io"
+ all := recordMap(
+ // The victim, in account "bab", and the replica it really opted into.
+ provision.Record{Host: victim, Account: "bab", App: "tito", Region: "ie",
+ Steering: "latency", Replicas: []string{"uk"}},
+ provision.Record{Host: "r7x2.go.tito.io", Account: "bab", App: "tito", Region: "uk",
+ Role: "replica", Home: victim},
+ // The attacker, in "mal" — a prefix that sorts AFTER the victim's, so
+ // a host-keyed map would hold this record and not the victim's.
+ provision.Record{Host: victim, Account: "mal", App: "tito", Region: "ie",
+ Steering: "latency", Replicas: []string{"de"}},
+ provision.Record{Host: "r9z9.go.tito.io", Account: "mal", App: "tito", Region: "de",
+ Role: "replica", Home: victim},
+ // And the other direction: a record of the attacker's own, entirely
+ // valid on its own terms — a home it does own, a placement it opted
+ // into — except that it claims the VICTIM's REPLICA host. The file
+ // this pass writes is addressed by host, so without the same
+ // ownership question it would hand the victim's replica a file
+ // naming a stranger as its home.
+ provision.Record{Host: "mal.go.tito.io", Account: "mal", App: "other", Region: "ie",
+ Steering: "latency", Replicas: []string{"uk"}},
+ provision.Record{Host: "r7x2.go.tito.io", Account: "mal", App: "other", Region: "uk",
+ Role: "replica", Home: "mal.go.tito.io"},
+ )
+ got := buildSiblings(all, func(string) (string, bool) { return "203.0.113.9", true }, [][]byte{[]byte("cur")})
+
+ home, ok := got[victim]
+ if !ok {
+ t.Fatal("the victim's own home file went missing: a shadow in another prefix took it")
+ }
+ if len(home.Replicas) != 1 || home.Replicas[0].Host != "r7x2.go.tito.io" {
+ t.Errorf("the victim's file lists %+v, want only the replica it opted into — anything else "+
+ "is an address and a pair key the victim never vouched for", home.Replicas)
+ }
+ if _, ok := got["r9z9.go.tito.io"]; ok {
+ t.Errorf("the shadow's replica got a file of its own: %+v", got["r9z9.go.tito.io"])
+ }
+ rf, ok := got["r7x2.go.tito.io"]
+ if !ok || rf.Home == nil {
+ t.Fatalf("the victim's replica lost its own file: %+v", rf)
+ }
+ if rf.Home.Host != victim {
+ t.Errorf("the victim's replica is told its home is %q — a record claiming that host in "+
+ "another account's prefix rewrote the file", rf.Home.Host)
+ }
+}
+
// --- the writer ---
const (
@@ -185,12 +259,12 @@ const (
// siblingsPair is the two records every writer test below works from: a home
// opted into one replica placement, and the replica record itself.
func siblingsPair() map[string]provision.Record {
- return map[string]provision.Record{
- siblingsHomeHost: {Host: siblingsHomeHost, Account: "bab", App: "tito", Region: "ie",
+ return recordMap(
+ provision.Record{Host: siblingsHomeHost, Account: "bab", App: "tito", Region: "ie",
Steering: "latency", Replicas: []string{"uk"}},
- siblingsReplicaHost: {Host: siblingsReplicaHost, Account: "bab", App: "tito", Region: "uk",
+ provision.Record{Host: siblingsReplicaHost, Account: "bab", App: "tito", Region: "uk",
Role: "replica", Home: siblingsHomeHost},
- }
+ )
}
func siblingsEIP(host string) (string, bool) {
@@ -262,8 +336,9 @@ func TestSiblingsPassWritesEachLayoutsDirectoryAt0600(t *testing.T) {
all := siblingsPair()
// A third record, homed on some other box: in the map, not in the
// registry, so no file of any kind lands here for it.
- all["other.go.tito.io"] = provision.Record{Host: "other.go.tito.io", Account: "zed", App: "tito",
+ other := provision.Record{Host: "other.go.tito.io", Account: "zed", App: "tito",
Region: "de", Steering: "latency", Replicas: []string{"fr"}}
+ all[provision.RecordKey(other.Account, other.App, other.Host)] = other
w := &siblingsWriter{reg: reg, dataDir: dd}
if err := w.pass(all, siblingsEIP, [][]byte{[]byte("mesh-A")}); err != nil {
@@ -345,9 +420,7 @@ func TestSiblingsPassRemovesAStaleFileWithoutFollowingASymlink(t *testing.T) {
// The account's replicas are withdrawn: the record no longer names a
// placement and the replica record is gone.
- plain := map[string]provision.Record{
- siblingsHomeHost: {Host: siblingsHomeHost, Account: "bab", App: "tito", Region: "ie"},
- }
+ plain := recordMap(provision.Record{Host: siblingsHomeHost, Account: "bab", App: "tito", Region: "ie"})
if err := w.pass(plain, siblingsEIP, [][]byte{[]byte("mesh-A")}); err != nil {
t.Fatalf("second pass: %v", err)
}
@@ -395,8 +468,9 @@ func TestSiblingsPassRefusesTwoFlatHostsSharingOneDirectory(t *testing.T) {
const other = "zed.go.tito.io"
addFlatRoute(t, reg, dd, other)
all := siblingsPair()
- all[other] = provision.Record{Host: other, Account: "zed", App: "tito", Region: "ie",
+ rec := provision.Record{Host: other, Account: "zed", App: "tito", Region: "ie",
Steering: "latency", Replicas: []string{"uk"}}
+ all[provision.RecordKey(rec.Account, rec.App, rec.Host)] = rec
if err := w.pass(all, siblingsEIP, [][]byte{[]byte("mesh-A")}); err != nil {
t.Fatalf("second pass: %v", err)
}
@@ -464,9 +538,9 @@ func TestSiblingsPassForgetsAPairThatIsGone(t *testing.T) {
if len(w.last) != 1 {
t.Fatalf("remembered %d derivatives, want one per pair", len(w.last))
}
- if err := w.pass(map[string]provision.Record{
- siblingsHomeHost: {Host: siblingsHomeHost, Account: "bab", App: "tito", Region: "ie"},
- }, siblingsEIP, [][]byte{[]byte("mesh-A")}); err != nil {
+ if err := w.pass(recordMap(
+ provision.Record{Host: siblingsHomeHost, Account: "bab", App: "tito", Region: "ie"},
+ ), siblingsEIP, [][]byte{[]byte("mesh-A")}); err != nil {
t.Fatalf("pass 2: %v", err)
}
if len(w.last) != 0 {
diff --git a/internal/provision/reconcile.go b/internal/provision/reconcile.go
index 508842a3d..ad500e195 100644
--- a/internal/provision/reconcile.go
+++ b/internal/provision/reconcile.go
@@ -80,6 +80,14 @@ type ReconcileConfig struct {
// needs both sides of a home/replica pair and only one is ever in
// scope on a given box; this hands it the map the pass already built
// rather than making it list apps/ a second time.
+ //
+ // The map is keyed by the record's IDENTITY — RecordKey(account, app,
+ // host), the bucket key it was read from — and that is not a detail of
+ // convenience, it is the safety property the reader inherits. A map
+ // keyed by host has one slot per host, so a record written in any
+ // account's prefix can occupy another account's slot and be read as
+ // that account's record. Look a host up the way this pass does:
+ // all[RecordKey(acct, app, host)], inside the tenancy asking.
OnRecords func(all map[string]Record)
}
@@ -700,6 +708,77 @@ func convergeOwned(reg *registry.Registry, existing registry.Route, rec Record,
}
}
+// OwnsItsHost reports whether rec is the record that OWNS its host in the
+// fleet-wide record map — the one a bucket listing reaches FIRST when two
+// records claim the same host. That is the arbitration Reconcile already
+// makes when it mints (the duplicate-host skip below, "whichever record this
+// listing reaches first wins the host"); the map is keyed by RecordKey and a
+// listing is lexicographic, so "first" is the smallest key, and every box
+// reaches the same answer off the same bucket with nobody coordinating.
+//
+// One host, two records, is not a state one account's create door produces —
+// CreateRecord is create-only on the key — but the key carries the account
+// and the app, so nothing stops a SECOND account's record from naming the
+// same host: a hand edit, a writer with its own bucket access, or two
+// accounts reaching for one custom domain. Whoever loses that arbitration
+// has no route anywhere: the host is served from the winner's row. So a
+// replica vouched for by a losing record is vouched for by a record that
+// serves nothing, and the alias it mints would divert the WINNER's public
+// host to a box nobody pointed it at.
+func OwnsItsHost(all map[string]Record, rec Record) bool {
+ mine := RecordKey(rec.Account, rec.App, rec.Host)
+ for key, other := range all {
+ if other.Host == rec.Host && key < mine {
+ return false
+ }
+ }
+ return true
+}
+
+// PairableHome resolves the home record a replica record names and reports
+// whether the pair may be made at all — the cross-record rails of the
+// console's create door (spec §6), asked of the bucket rather than of the
+// caller, because a door is not a guarantee about what is in the bucket:
+// every region box's IAM role can PutObject there and an operator can
+// hand-write a record.
+//
+// BOTH passes that make a pair ask this one function — the reconcile pass
+// that mints the route, and the box's siblings writer that hands the two
+// sides each other's address and key. They defend the same property from two
+// sides, so a rail only one of them holds is a rail the other bypasses.
+//
+// `all` is keyed by the record's IDENTITY (RecordKey), and that keying is
+// itself two of the rails: the home is looked up INSIDE the replica's own
+// account and app, so "same account" and "same app" are true by construction
+// rather than by comparison. They used to be comparisons against a map keyed
+// by host, which has one slot per host — and a record written in any prefix
+// could therefore stand in for the victim's, pass the comparisons against
+// itself, and take the victim's real record out of the map on the way past.
+//
+// why is "" when the pair may be made. Otherwise it names the condition that
+// failed, for the log, because "not minting" with no reason is how an
+// operator's typo becomes an afternoon — and protect reports whether the
+// caller should leave an existing row exactly as it is (see the two answers
+// at Reconcile's replica branch).
+func PairableHome(all map[string]Record, rec Record) (home Record, why string, protect bool) {
+ home, ok := all[RecordKey(rec.Account, rec.App, rec.Home)]
+ switch {
+ case !ok:
+ // Absent, unreadable this pass, or in another account or app — one
+ // answer, because the lookup cannot tell them apart and the pass must
+ // not guess. A mirror of nothing: protected, since a bucket blip must
+ // never sweep a live placement's alias.
+ return Record{}, "no record for its home " + rec.Home + " in this account and app", true
+ case !OwnsItsHost(all, home):
+ return Record{}, "its home " + rec.Home + " is not the record that owns that host", false
+ case home.Role != "":
+ return Record{}, "its home " + rec.Home + " is itself a replica", false
+ case !slices.Contains(home.Replicas, rec.Region):
+ return Record{}, "its home " + rec.Home + " has not opted into placement " + rec.Region, true
+ }
+ return home, "", false
+}
+
// Reconcile materializes bucket instance records into registry routes for
// THIS agent's region and pools: mint on appearance, converge seed, remove
// on record deletion — removal only for routes this pass (or an earlier one
@@ -725,7 +804,7 @@ func Reconcile(ctx context.Context, s release.Store, reg *registry.Registry, cfg
return fmt.Errorf("provision: list apps: %w", err)
}
recs := map[string]Record{} // host -> record, in-scope only
- all := map[string]Record{} // host -> record, EVERY decoded record
+ all := map[string]Record{} // RecordKey -> record, EVERY decoded record
// protect holds hosts whose record KEY parsed (ParseRecordKey succeeded)
// but whose body could not be trusted — unparseable JSON, or a body that
// disagrees with its own path. Such a host must NEVER be treated as "the
@@ -810,7 +889,19 @@ func Reconcile(ctx context.Context, s release.Store, reg *registry.Registry, cfg
// scope check: a record ValidateShape rejected, or one carrying no
// channel, is absent from this map too, so a replica pointed at it
// is protected rather than minted off a record no box will serve.
- all[rec.Host] = rec
+ //
+ // Keyed by the record's IDENTITY — account, app and host — never by
+ // host alone. The key IS a safety property, and a host-keyed map had
+ // one slot per host: a record written in ANY prefix, carrying its
+ // own account and app so it agreed with its own path, overwrote the
+ // slot belonging to a record in another account (a bucket listing is
+ // lexicographic, so the later-sorting prefix won). The cross-record
+ // rails below then read that record as the victim's home — vouching
+ // for the writer's own replica, and refusing the victim's real one.
+ // The duplicate-host check further down guards `recs` and only
+ // `recs`; this map is where the collision had to stop being possible
+ // at all.
+ all[RecordKey(rec.Account, rec.App, rec.Host)] = rec
// Scope. For a home-boxed record this is the region/pool test it
// always was; for a static or public-addr one the region half widens
// to every platform box in the deployment (everywhereClass, above).
@@ -922,48 +1013,71 @@ func Reconcile(ctx context.Context, s release.Store, reg *registry.Registry, cfg
}
for _, rec := range recs {
- // A replica's route takes its channel from its HOME record (#162),
+ // home is the replica's HOME record, and the zero Record for every
+ // record that is not a replica — which is what routeFor and
+ // convergeOwned below already read a non-replica's home as. Resolved
+ // once, here, so the rest of the loop cannot reach into the record
+ // map for it by host and undo the rails.
+ var home Record
+ // A replica's route takes its channel from its home record (#162),
// so a pass that cannot see that record can neither mint the row nor
// converge it: both would write an empty channel off a zero Record.
- // Protect rather than guess — the row keeps serving what it serves,
- // and the removal phase leaves it alone (spec §2). Asked before
- // trashedFor and seedFor, so a mirror of nothing costs no reads.
+ // Asked before trashedFor and seedFor, so a mirror of nothing costs
+ // no reads.
//
- // The same four CROSS-RECORD rails the console's create door
- // enforces (spec §6) are asked here, because the door is not a
- // guarantee about what is in the bucket: every region box's IAM role
- // can PutObject into it and an operator can hand-write a record, and
- // this is the one place in the pass where the other record is
- // already in hand. Without them a record in ANY account naming
- // another account's host as its home mints here, has the domains
- // pass alias the victim's public host to this box, and is written
- // into the VICTIM's own siblings file as a vouched-for replica with
- // a live pair key.
+ // PairableHome holds the CROSS-RECORD rails the console's create
+ // door enforces (spec §6), because the door is not a guarantee about
+ // what is in the bucket: every region box's IAM role can PutObject
+ // into it and an operator can hand-write a record, and this is the
+ // one place in the pass where the other record is already in hand.
+ // Without them a record naming another account's host as its home
+ // mints here, has the domains pass alias the victim's public host to
+ // this box, and is written into the VICTIM's own siblings file as a
+ // vouched-for replica with a live pair key.
//
- // Protect rather than retire, in every one of these cases: a home
- // whose `replicas` an old console stripped keeps its existing row
- // serving, which is the fail-safe direction the spec chose for a
- // home that cannot be verified. The condition that failed is named
- // in the log, because "not minting" with no reason is how an
- // operator's typo becomes an afternoon.
+ // A failure has TWO answers, and which one is right depends on
+ // whether a legitimate writer could have caused it:
+ //
+ // protect — the home is not readable this pass, or its `replicas`
+ // no longer names this placement. An OLD CONSOLE strips that
+ // field by round-tripping the record (spec §6), and a bucket
+ // read can simply fail, so the existing row goes on serving what
+ // it serves. That is the fail-safe direction: a console deploy
+ // must not be able to take a live placement down.
+ //
+ // clear the link — every other rail. None of them can fail on a
+ // record the doors wrote, so what failed them is a writer
+ // outside the doors, and protecting THAT leaves a row nobody
+ // vouched for serving an account's public host indefinitely, for
+ // no fail-safe benefit at all. SetReplicaOf("") drops the link;
+ // the domains pass sweeps the alias on its next run. Not
+ // reg.Delete: removal carries a grant purge (see the removal
+ // phase below) and nothing here is a retirement the bucket
+ // asked for.
+ //
+ // The condition that failed is named in the log either way, because
+ // "not minting" with no reason is how an operator's typo becomes an
+ // afternoon.
if rec.Role == "replica" {
- home, ok := all[rec.Home]
var why string
- switch {
- case !ok:
- why = "its home record " + rec.Home + " is absent or unreadable this pass"
- case home.Account != rec.Account:
- why = "its home " + rec.Home + " belongs to another account"
- case home.App != rec.App:
- why = "its home " + rec.Home + " belongs to another app"
- case home.Role != "":
- why = "its home " + rec.Home + " is itself a replica"
- case !slices.Contains(home.Replicas, rec.Region):
- why = "its home " + rec.Home + " has not opted into placement " + rec.Region
- }
+ var keep bool
+ home, why, keep = PairableHome(all, rec)
if why != "" {
log.Printf("provision: not minting replica %s: %s", rec.Host, why)
- protect[rec.Host] = true
+ switch existing, ok := byHost[rec.Host]; {
+ case keep:
+ protect[rec.Host] = true
+ case !ok || existing.ReplicaOf == "":
+ // Nothing minted here, or nothing linked: nothing to clear.
+ case cfg.DryRun:
+ log.Printf("provision: dry-run: would clear %s replica_of %q", rec.Host, existing.ReplicaOf)
+ default:
+ if err := reg.SetReplicaOf(rec.Host, ""); err != nil {
+ log.Printf("provision: clear replica_of on %s: %v", rec.Host, err)
+ } else {
+ log.Printf("provision: cleared %s replica_of %q: %s", rec.Host, existing.ReplicaOf, why)
+ }
+ }
continue
}
}
@@ -999,7 +1113,7 @@ func Reconcile(ctx context.Context, s release.Store, reg *registry.Registry, cfg
// exact route this branch hardcoded before they existed — that is
// the additivity rule, pinned by TestRecordWireShapeIsAdditive and
// TestReconcileMints.
- nr := routeFor(rec, all[rec.Home], cfg.DataDir, seed)
+ nr := routeFor(rec, home, cfg.DataDir, seed)
if cfg.DryRun {
log.Printf("provision: dry-run: would mint %s route %s (app %q acct %q channel %q)",
nr.Kind, rec.Host, rec.App, rec.Account, rec.Channel)
@@ -1016,7 +1130,7 @@ func Reconcile(ctx context.Context, s release.Store, reg *registry.Registry, cfg
// deploy's SetVersions or the activator's SetNextDue would
// clobber those, and Upsert cannot express "clear Environment"
// at all (it preserves a non-empty value by design).
- convergeOwned(reg, existing, rec, all[rec.Home], cfg.DryRun)
+ convergeOwned(reg, existing, rec, home, cfg.DryRun)
// Absent means "this record does not speak to hibernation" —
// never a derivation stamped over an operator's own setting.
//
@@ -1066,7 +1180,7 @@ func Reconcile(ctx context.Context, s release.Store, reg *registry.Registry, cfg
// widened to prevent. What remains are the columns nothing
// converges in place: the route CLASS and the two filesystem paths
// derived from it.
- if want := routeFor(rec, all[rec.Home], cfg.DataDir, seed); want.Kind != existing.Kind ||
+ if want := routeFor(rec, home, cfg.DataDir, seed); want.Kind != existing.Kind ||
(rec.Unit != "" && want.Unit != existing.Unit) ||
want.Socket != existing.Socket || want.DBPath != existing.DBPath {
log.Printf("provision: %s: record describes a different route SHAPE than the row "+
@@ -1084,7 +1198,7 @@ func Reconcile(ctx context.Context, s release.Store, reg *registry.Registry, cfg
// as before: adoption grants this pass's removal phase authority
// over the row, so the match test is the only thing standing
// between a backfill and a retired production route.
- want := routeFor(rec, all[rec.Home], cfg.DataDir, seed)
+ want := routeFor(rec, home, cfg.DataDir, seed)
if reason := shapeMismatch(existing, want); reason != "" {
log.Printf("provision: %s exists but is not the route this record describes (%s) — leaving it alone",
rec.Host, reason)
@@ -1101,7 +1215,7 @@ func Reconcile(ctx context.Context, s release.Store, reg *registry.Registry, cfg
// already on the record's channel. Every write inside
// convergeOwned is dry-run guarded there, so this call
// writes nothing.
- convergeOwned(reg, existing, rec, all[rec.Home], true)
+ convergeOwned(reg, existing, rec, home, true)
continue
}
if err := reg.SetProvisioned(rec.Host, true); err != nil {
@@ -1116,7 +1230,7 @@ func Reconcile(ctx context.Context, s release.Store, reg *registry.Registry, cfg
// live process is allowed to sleep. It converges on the next
// pass through the owned branch above, by which point the
// adoption is already visible in the log.
- convergeOwned(reg, existing, rec, all[rec.Home], cfg.DryRun)
+ convergeOwned(reg, existing, rec, home, cfg.DryRun)
}
}
diff --git a/internal/provision/reconcile_test.go b/internal/provision/reconcile_test.go
index 3a84d8b2b..eba11f8ee 100644
--- a/internal/provision/reconcile_test.go
+++ b/internal/provision/reconcile_test.go
@@ -2837,20 +2837,27 @@ func TestReconcileProtectsReplicaWhoseHomeIsGone(t *testing.T) {
}
}
+// Every record, in scope or not — and keyed by the record's IDENTITY, which
+// is the contract the siblings pass reads it under: a host-keyed map has one
+// slot per host, and one account's record in another account's slot is the
+// whole of the shadow bypass below.
func TestReconcileReportsEveryRecord(t *testing.T) {
- s, _, home, replica := replicaSetup(t)
+ s, sqid, home, replica := replicaSetup(t)
var seen map[string]Record
cfg := ReconcileConfig{Region: "uk", Pools: []string{"tito"}, DataDir: "/data",
OnRecords: func(all map[string]Record) { seen = all }}
if err := Reconcile(ctx, s, regWith(t), cfg); err != nil {
t.Fatal(err)
}
- if _, ok := seen[home.Host]; !ok {
+ if _, ok := seen[RecordKey(sqid, "tito", home.Host)]; !ok {
t.Error("OnRecords did not carry the out-of-scope home record")
}
- if _, ok := seen[replica.Host]; !ok {
+ if _, ok := seen[RecordKey(sqid, "tito", replica.Host)]; !ok {
t.Error("OnRecords did not carry the in-scope replica record")
}
+ if _, ok := seen[home.Host]; ok {
+ t.Error("OnRecords is keyed by host: any account's record can occupy another's slot")
+ }
}
// The pass holds the same four cross-record rails the console's create door
@@ -2954,6 +2961,126 @@ func TestReconcileRefusesAReplicaInAPlacementTheHomeNeverOptedInto(t *testing.T)
}
}
+// The SHADOW record. The rails above ask their four questions of "the
+// record for this host", and a record map keyed by HOST has exactly one slot
+// per host — so a writer with nothing but their own prefix can fill that slot
+// with a record of their own: same host, their account, their app. S3 lists
+// lexicographically, so a prefix that sorts after the victim's is written
+// into the map LAST and wins the slot.
+//
+// Three things followed, and every one of them is a P1: the attacker's
+// replica passed all four rails against its own shadow and minted, which
+// aliases the victim's public host to the attacker's process on that box;
+// the victim's own siblings file named the attacker (the other half of this
+// test, in internal/host); and the victim's LEGITIMATE replica was refused
+// as "another account's", because the record the rails read for its home was
+// no longer the victim's. A denial of service on the real pair, from a
+// record the victim cannot see and cannot delete.
+//
+// The map is now keyed by the record's IDENTITY — account, app and host —
+// so no record can occupy another's slot, and the home a replica names is
+// looked up inside the replica's own account and app.
+func TestReconcileShadowHomeRecordCannotDisplaceTheVictims(t *testing.T) {
+ s, sqid, home, replica := replicaSetup(t)
+ mallory := writeAccountManifest(t, s, 3, "Mallory")
+ if mallory <= sqid {
+ t.Fatalf("fixture: the attacker's prefix %q must sort AFTER the victim's %q, or the "+
+ "shadow never reaches the map slot this test is about", mallory, sqid)
+ }
+ if err := PutManifest(ctx, s, mallory, "tito", Manifest{
+ Schema: 1, App: "tito", Domain: "go.tito.io", Pool: "tito", Seed: []string{"seed"},
+ }); err != nil {
+ t.Fatal(err)
+ }
+ // The shadow: the VICTIM's host, in the ATTACKER's prefix, carrying the
+ // attacker's own account and app so it agrees with its own path — and
+ // opted into uk, so the attacker's replica below can point at it.
+ shadow := Record{Schema: 1, Host: home.Host, Slug: "acme", App: "tito",
+ Account: mallory, Pool: "tito", Region: "ie", Channel: "stable", CreatedAt: time.Now(),
+ Steering: "latency", Replicas: []string{"uk"}}
+ if err := CreateRecord(ctx, s, shadow); err != nil {
+ t.Fatal(err)
+ }
+ evil := Record{Schema: 1, Host: "r4444444444.go.tito.io", Slug: "r4444444444", App: "tito",
+ Account: mallory, Pool: "tito", Region: "uk", Channel: "stable", CreatedAt: time.Now(),
+ Role: "replica", Home: home.Host}
+ if err := CreateRecord(ctx, s, evil); err != nil {
+ t.Fatal(err)
+ }
+ uk := regWith(t)
+ if err := Reconcile(ctx, s, uk, ReconcileConfig{Region: "uk", Pools: []string{"tito"}, DataDir: "/data"}); err != nil {
+ t.Fatal(err)
+ }
+ // Get reports a miss as (nil, nil), never an error.
+ if rt, err := uk.Get(evil.Host); err != nil {
+ t.Fatal(err)
+ } else if rt != nil {
+ t.Errorf("a replica vouched for by a SHADOW of the victim's host was minted: %+v — the "+
+ "victim's public host is now aliased to this row", rt)
+ }
+ // And the denial-of-service half: the victim's own pair is untouched by
+ // a record in a prefix the victim has no sight of.
+ if got := mustGet(t, uk, replica.Host); got.ReplicaOf != home.Host {
+ t.Errorf("the victim's own replica has ReplicaOf %q, want %q — a shadow record in another "+
+ "account's prefix took its home away", got.ReplicaOf, home.Host)
+ }
+}
+
+// Rails that can never fail on a door-written record fail here only because
+// something outside the doors wrote the record — so leaving the row serving
+// buys no fail-safety, it just leaves a link nobody vouched for in place.
+// The link is CLEARED (SetReplicaOf "", which the domains pass sweeps on its
+// next alias reconcile) rather than the row deleted: registry.Delete carries
+// a grant purge, and none of this is a retirement the bucket asked for.
+func TestReconcileClearsTheLinkWhenTheHomeIsItselfAReplica(t *testing.T) {
+ s, sqid, home, replica := replicaSetup(t)
+ uk := regWith(t)
+ cfg := ReconcileConfig{Region: "uk", Pools: []string{"tito"}, DataDir: "/data"}
+ if err := Reconcile(ctx, s, uk, cfg); err != nil {
+ t.Fatal(err)
+ }
+ if got := mustGet(t, uk, replica.Host).ReplicaOf; got != home.Host {
+ t.Fatalf("fixture: ReplicaOf = %q, want %q", got, home.Host)
+ }
+ // The home turns into a replica underneath its own replica: a chain, and
+ // no door makes one. (A record carrying both `role` and `replicas` is
+ // refused by ValidateShape before any of this, so the rewrite is the
+ // shape a hand edit would actually leave behind.)
+ home.Role, home.Home, home.Replicas, home.Steering = "replica", "somewhere.go.tito.io", nil, ""
+ putReplace(t, s, RecordKey(sqid, "tito", home.Host), mustJSON(t, home))
+ if err := Reconcile(ctx, s, uk, cfg); err != nil {
+ t.Fatal(err)
+ }
+ rt := mustGet(t, uk, replica.Host) // still serving; only the link goes
+ if rt.ReplicaOf != "" {
+ t.Errorf("ReplicaOf = %q, want empty: the alias goes on sending the home's public host "+
+ "to this row for as long as the link stands", rt.ReplicaOf)
+ }
+}
+
+// The opt-in is the one rail an OLD CONSOLE can legitimately fail: it
+// marshals a record it has never heard the `replicas` field on, and writing
+// a home back strips the list (spec §6). Clearing the link there would take
+// a live placement down on a console deploy, so this rail alone keeps the
+// fail-safe answer — the existing row goes on serving what it serves.
+func TestReconcileProtectsTheRowWhenTheHomeWithdrawsThePlacement(t *testing.T) {
+ s, sqid, home, replica := replicaSetup(t)
+ uk := regWith(t)
+ cfg := ReconcileConfig{Region: "uk", Pools: []string{"tito"}, DataDir: "/data"}
+ if err := Reconcile(ctx, s, uk, cfg); err != nil {
+ t.Fatal(err)
+ }
+ home.Replicas = nil // an old console's UpdateRecord, round-tripping
+ putReplace(t, s, RecordKey(sqid, "tito", home.Host), mustJSON(t, home))
+ if err := Reconcile(ctx, s, uk, cfg); err != nil {
+ t.Fatal(err)
+ }
+ if got := mustGet(t, uk, replica.Host).ReplicaOf; got != home.Host {
+ t.Errorf("ReplicaOf = %q, want %q: a console that cannot write the field must not be able "+
+ "to take a placement down by deploying", got, home.Host)
+ }
+}
+
// A gated home must not have an ungated mirror of itself: the alias sends the
// account's public host to the replica's row, so that row carries the home's
// gate or the password gate simply does not exist in that placement. The