carlos / platform Public

#162: sort order is not authority — a contested host vouches for nothing

adf29ebd's fifth rail said the home must be the FIRST-SORTING claimant of
its host. A second adversarial verification broke it twice, in the same
afternoon, and both breaks are the same mistake: a bucket listing is
lexicographic, so "first" is a property an attacker chooses, not a property
the platform grants.

  1. An attacker whose account prefix sorts BEFORE the victim's IS the
     owner under that rail. Their shadow of the victim's host vouched for
     their own replica, it passed all five rails, and the victim's public
     host was aliased to the attacker's process on that box — the original
     P1, intact. The test that shipped with the rail covered only the
     later-sorting half, and guarded its own fixture with
     `if mallory <= sqid { t.Fatalf }` so the other half never ran.

  2. An earlier-sorting PLAIN record carrying the victim's host, in a region
     no box serves, made the victim's own box CLEAR the victim's real
     replica link: the rail answered protect=false, the domains pass swept
     the alias, and another box went on minting the victim's home. A live
     pair severed by a record the victim cannot see and cannot delete.

There is no attacker-proof ownership source available to this pass — it
reads records, not the authority behind them — so the rail stops trying to
pick a winner. A host claimed by more than one RecordKey is CONTESTED
(provision.ContestedHosts), and a contested host is never used to vouch for
anything: a replica whose home is contested is protected, a replica whose
OWN host is contested is protected, and buildSiblings writes no file for
either side of such a pairing. Claims are counted off the bucket LISTING
rather than off the decoded records, so a record whose body will not read
this pass still holds its claim — otherwise a read failure hands the shadow
sole ownership and mints the attacker's replica during the blip, which it
did. It is logged once per pass naming both claimants: two records claiming
one instance host is a thing an operator has to see.

Nothing in the pass clears replica_of any more. The SetReplicaOf(host, "")
path adf29ebd added for rails one to three is gone; it was the weapon in the
second break, and protection is the correct fail-safe for every one of these
rails. The argument for clearing was that no door produces those states, so
whatever produced them was outside the doors — but "outside the doors"
includes a writer attacking the account, and clearing was the only
irreversible thing the branch could do. The siblings writer holds the same
line: a directory holding a contested host is left exactly as it is, no
write and no removal, since "nothing built" is otherwise how that pass
learns a placement was withdrawn.

The identity keying of `all` stays — that part was right and is load-bearing.

Both of the verifier's adversarial files are adopted as
replica_adversarial_test.go, every case kept, plus the cases they did not
write: the earlier-sorting shadow refused, the earlier-sorting plain record
not severing a live pair, and the legitimate pair still working when no
shadow exists. Every case runs in BOTH sort orders — the fixture guard is
gone — and each new assertion was verified red against adf29ebd by
overlaying the files onto a `git archive` of it.

Spec §6 now says plainly what these rails are: fail-safe only, they refuse
to vouch and never revoke; a writer holding bucket-wide s3:PutObject (a
region box's IAM role does) can still deny service by contesting a host and
CANNOT be stopped by this pass; and the real fix is narrowing that grant,
which is infrastructure work outside this branch.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
claude agent, operated by paul@keymail.dev f0ea03866b06f2141309f03eaf4e55f4088d8f27 parent adf29eb
7 files changed, +854 −232
  • docs/superpowers/specs/2026-09-21-replica-placements-design.md +47 −19
  • internal/host/replica_adversarial_test.go +190 −0
  • internal/host/siblings.go +49 −20
  • internal/host/siblings_test.go +12 −52
  • internal/provision/reconcile.go +118 −68
  • internal/provision/reconcile_test.go +51 −73
  • internal/provision/replica_adversarial_test.go +387 −0

CI passed — run details

diff --git a/docs/superpowers/specs/2026-09-21-replica-placements-design.md b/docs/superpowers/specs/2026-09-21-replica-placements-design.md
index 96426072b..44dc8192c 100644
--- a/docs/superpowers/specs/2026-09-21-replica-placements-design.md
+++ b/docs/superpowers/specs/2026-09-21-replica-placements-design.md
@@ -384,9 +384,11 @@ struct it was handed (`provision.go:928-933`) and `DecodeRecord` polices
only `maintenance` (`maintenance.go:38-78`), so a console build that
predates these fields reads a record without them and writes it back
without them — `replicas` vanishes from a home (bypassing the refusal
-above), `role`/`home` vanish from a replica, which clears `replica_of` and
-sweeps the alias on the next pass, silently. Tito's console is pinned on an
-older build today.
+above), and `role`/`home` vanish from a replica, so the record stops
+describing the pair while the row goes on serving it: nothing in the pass
+clears `replica_of` (see the rails below), so the alias keeps standing and
+the two drift silently instead. Tito's console is pinned on an older build
+today.
Revision 2 answered this by adding all three fields to
`protectedFieldsEqual`. The plan review showed that is the wrong seam
@@ -412,22 +414,48 @@ questions of the fleet-wide record map before they pair anything
in the bucket. The map is keyed by the record's identity
(`RecordKey(account, app, host)`, not by host), so the home is looked up
*inside the replica's own account and app*: the tenancy halves are the key
-rather than a comparison. On top of that the home must be the record that
-**owns** its host — the first-sorting claimant, the same arbitration the
-pass already makes when two records claim one host, since the loser's
-record serves nothing anywhere — must not itself be a replica, and must
-name the placement in its `replicas`. What that defends is an operator's
-typo and a **writer scoped to its own prefix**: neither can reach into
-another account's pairing, take a live pair away from its owner, or land in
-another account's `siblings.json` with a dial address and a pair key. What
-it is **not** is a tenancy boundary against a writer holding bucket-wide
-`s3:PutObject`, which every region box's IAM role does hold. Such a writer
-can author a record in the **victim's own prefix**, where every rail passes
-trivially — and could as easily write a plain instance record claiming the
-victim's host, with no replica involved at all. Host ownership is decided
-at the console's create door (the manifest domain, the custom-domain claim)
-and by who can write which bucket prefix. The pass can refuse only what it
-can see, and what it sees is records, not the authority behind them.
+rather than a comparison. The home must also not itself be a replica, and
+must name the placement in its `replicas`.
+
+**The rails are fail-safe only: they refuse to vouch, they never revoke.**
+That is the whole of their posture, and it is a correction. Two earlier
+attempts tried to pick a winner when two records claimed one host — the
+first-sorting claimant "owns" it, the arbitration the pass already makes
+when it mints — and an attacker simply chose an account prefix that sorts
+first, which made the *attacker* the owner and had the rail vouch for their
+replica. Worse, a losing record was treated as grounds to clear a live
+`replica_of`, so one planted record in a region no box serves severed a
+running pair the victim could neither see nor delete. Sort order is not
+authority, and no attacker-proof ownership source is available to a pass
+that reads records rather than the authority behind them. So a host claimed
+by more than one record identity is **contested**
+(`provision.ContestedHosts`), and a contested host vouches for nothing and
+is vouched for by nothing: the replica is not minted, not converged, its
+link is left exactly as it stands, and neither side of the pairing gets a
+`siblings.json` written or removed. It is logged once per pass, naming both
+claimants, because two records claiming one instance host is a thing an
+operator has to see. **No path in either pass clears `replica_of`.**
+
+**What that defends** is an operator's typo and a **writer scoped to its
+own prefix**: neither can reach into another account's pairing, take a live
+pair away from its owner, or land in another account's `siblings.json` with
+a dial address and a pair key. What it is **not** is a tenancy boundary
+against a writer holding bucket-wide `s3:PutObject`, which every region
+box's IAM role does hold. Such a writer can author a record in the
+**victim's own prefix**, where every rail passes trivially, and can as
+easily write a plain instance record claiming the victim's host with no
+replica involved at all. They can also **deny service** by contesting a
+host: one record naming the victim's host freezes that host's pairings —
+nothing new mints, nothing converges — until it is removed. That is the
+deliberate trade, because the alternative is picking a claimant, and every
+rule for picking one hands the attacker the victim's public host instead of
+a stalled pairing. **This pass cannot stop them.** The real fix is
+narrowing that grant so a box's role can write only its own deployment's
+records — infrastructure work, outside this branch, and the thing to do
+next if any of this matters. Until then, host ownership is decided at the
+console's create door (the manifest domain, the custom-domain claim) and by
+who can write which bucket prefix; the pass can refuse only what it can
+see, and what it sees is records, not the authority behind them.
## 7. Sibling addressing (Ask 3): no new channel
diff --git a/internal/host/replica_adversarial_test.go b/internal/host/replica_adversarial_test.go
new file mode 100644
index 000000000..f4b020705
--- /dev/null
+++ b/internal/host/replica_adversarial_test.go
@@ -0,0 +1,190 @@
+package host
+
+import (
+ "os"
+ "path/filepath"
+ "testing"
+
+ "github.com/carlosframework/platform/internal/provision"
+)
+
+// The adversarial half of the siblings pass (#162), the companion of
+// internal/provision/replica_adversarial_test.go. The files this pass writes
+// are what tells one instance to dial another and hands it a live pair key,
+// so a record that reaches into someone else's file is a record that gets
+// leases granted to it and drains accepted from it.
+//
+// The rail here is the same one the reconcile pass holds, asked of the same
+// map: a host claimed by more than one record IDENTITY is CONTESTED, and a
+// contested host vouches for nothing. Neither side of such a pairing gets a
+// file, and any file already on disk is left exactly as it is — because the
+// alternative, deciding between the claimants, is what broke twice.
+//
+// Sort order is not authority. Both halves of the alphabet are covered.
+
+// The shadow of the victim's host, in both sort orders. The account labels
+// are the sqids the provision fixtures use ("bac" is Sqid(2), the victim);
+// "baaa" is a four-letter sqid, which sorts BEFORE it, and "mal" sorts after.
+//
+// adf29ebd wrote the later-sorting half of this test only, and passed. The
+// earlier-sorting half is the P1 it left open: the attacker was the
+// first-sorting claimant, so the pass vouched for the attacker's replica and
+// wrote it into the VICTIM's own file, with a dial address and a live key.
+func TestBuildSiblingsRefusesAContestedHostInEitherSortOrder(t *testing.T) {
+ const (
+ victim = "acme.go.tito.io"
+ victimReplica = "r7x2.go.tito.io"
+ evilReplica = "r9z9.go.tito.io"
+ )
+ for _, tc := range []struct {
+ name, attacker string
+ }{
+ {"attacker prefix sorts EARLIER than the victim's", "baaa"},
+ {"attacker prefix sorts LATER than the victim's", "mal"},
+ } {
+ t.Run(tc.name, func(t *testing.T) {
+ all := recordMap(
+ // The victim, and the replica it really opted into.
+ provision.Record{Host: victim, Account: "bac", App: "tito", Region: "ie",
+ Steering: "latency", Replicas: []string{"uk"}},
+ provision.Record{Host: victimReplica, Account: "bac", App: "tito", Region: "uk",
+ Role: "replica", Home: victim},
+ // The attacker's shadow of the victim's host, out of every
+ // box's scope ("de"), opted into uk — and a replica of its
+ // own to be vouched for by it.
+ provision.Record{Host: victim, Account: tc.attacker, App: "tito", Region: "de",
+ Steering: "latency", Replicas: []string{"uk"}},
+ provision.Record{Host: evilReplica, Account: tc.attacker, App: "tito", Region: "uk",
+ Role: "replica", Home: victim},
+ )
+ got := buildSiblings(all, func(string) (string, bool) { return "203.0.113.9", true }, [][]byte{[]byte("cur")})
+
+ // The file the victim's own box would write for the victim's own
+ // instance must never name a stranger.
+ for _, e := range got[victim].Replicas {
+ if e.Host == evilReplica {
+ t.Errorf("the victim's home file names the attacker's replica with addr %s and a "+
+ "live pair key: the victim's app is told to grant leases to it", e.Addr)
+ }
+ }
+ // A contested host vouches for nothing at all, in either
+ // direction: no file for the shadow, none for the replica it
+ // vouches for, and none for the victim's side either. Refusing
+ // to vouch is the whole remedy — this pass has no way to tell
+ // the two claimants apart, and picking one is what broke.
+ for _, host := range []string{victim, victimReplica, evilReplica} {
+ if f, ok := got[host]; ok {
+ t.Errorf("a contested pairing produced a file for %s: %+v", host, f)
+ }
+ }
+ })
+ }
+}
+
+// The other direction: a record of the attacker's own, entirely valid on its
+// own terms — a home it does own, a placement it opted into — except that it
+// claims the VICTIM's REPLICA host. The files this pass writes are addressed
+// by host, so without the contested rail it would hand the victim's replica a
+// file naming a stranger as its home, and a key derived from the stranger's
+// account.
+func TestBuildSiblingsRefusesAContestedReplicaHost(t *testing.T) {
+ const (
+ victim = "acme.go.tito.io"
+ victimReplica = "r7x2.go.tito.io"
+ )
+ all := recordMap(
+ provision.Record{Host: victim, Account: "bac", App: "tito", Region: "ie",
+ Steering: "latency", Replicas: []string{"uk"}},
+ provision.Record{Host: victimReplica, Account: "bac", App: "tito", Region: "uk",
+ Role: "replica", Home: victim},
+ provision.Record{Host: "mal.go.tito.io", Account: "baaa", App: "other", Region: "ie",
+ Steering: "latency", Replicas: []string{"uk"}},
+ provision.Record{Host: victimReplica, Account: "baaa", App: "other", Region: "uk",
+ Role: "replica", Home: "mal.go.tito.io"},
+ )
+ got := buildSiblings(all, func(string) (string, bool) { return "203.0.113.9", true }, [][]byte{[]byte("cur")})
+
+ if f, ok := got[victimReplica]; ok {
+ t.Errorf("the contested replica host got a file: %+v — whichever claimant it was written for, "+
+ "the other one reads it", f)
+ }
+ // The victim's home host is NOT contested here, so its own file is still
+ // written: the rail is per host, not per account. It simply names no
+ // replica, because the replica host it would have named is contested.
+ hf, ok := got[victim]
+ if !ok {
+ t.Fatalf("the victim's home file went missing: its own host is claimed by one record")
+ }
+ for _, e := range hf.Replicas {
+ if e.Host == victimReplica {
+ t.Errorf("the victim's home file names a contested replica host (%s): the file on that "+
+ "box could belong to either claimant", e.Host)
+ }
+ }
+}
+
+// Refusing to vouch is not refusing to work. Take the shadow away and the
+// same fixture pairs exactly as it always did, on both sides, with one key.
+func TestBuildSiblingsStillPairsWhenNoShadowExists(t *testing.T) {
+ const (
+ victim = "acme.go.tito.io"
+ victimReplica = "r7x2.go.tito.io"
+ )
+ all := recordMap(
+ provision.Record{Host: victim, Account: "bac", App: "tito", Region: "ie",
+ Steering: "latency", Replicas: []string{"uk"}},
+ provision.Record{Host: victimReplica, Account: "bac", App: "tito", Region: "uk",
+ Role: "replica", Home: victim},
+ )
+ got := buildSiblings(all, func(string) (string, bool) { return "203.0.113.9", true }, [][]byte{[]byte("cur")})
+
+ hf, ok := got[victim]
+ if !ok || len(hf.Replicas) != 1 || hf.Replicas[0].Host != victimReplica {
+ t.Fatalf("the home's file = %+v, want exactly the replica it opted into", hf)
+ }
+ rf, ok := got[victimReplica]
+ if !ok || rf.Home == nil || rf.Home.Host != victim {
+ t.Fatalf("the replica's file = %+v, want its home named", rf)
+ }
+ if len(rf.Home.Keys) != 1 || rf.Home.Keys[0] != hf.Replicas[0].Keys[0] {
+ t.Errorf("the two sides derived different keys: %v vs %v", rf.Home.Keys, hf.Replicas[0].Keys)
+ }
+}
+
+// A contested host must not cost the instance the file it already has. The
+// writer removes a file when nothing is built for the directory — which is
+// how a withdrawn placement is cleaned up — and that removal is a revocation
+// an attacker must not be able to trigger by planting one record.
+func TestSiblingsPassLeavesTheFileAloneWhenTheHostIsContested(t *testing.T) {
+ dd := t.TempDir()
+ reg := openSiblingsRegistry(t, dd)
+ addNestedRoute(t, reg, dd, siblingsHomeHost)
+ w := &siblingsWriter{reg: reg, dataDir: dd}
+ if err := w.pass(siblingsPair(), siblingsEIP, [][]byte{[]byte("mesh-A")}); err != nil {
+ t.Fatalf("first pass: %v", err)
+ }
+ path := filepath.Join(provision.InstanceDir(dd, siblingsHomeHost), siblingsFileName)
+ before, err := os.ReadFile(path)
+ if err != nil {
+ t.Fatalf("the first pass wrote no file to start from: %v", err)
+ }
+
+ // One record, planted in another account's prefix, carrying this box's
+ // host. Nothing else about the deployment changes.
+ all := siblingsPair()
+ shadow := provision.Record{Host: siblingsHomeHost, Account: "baaa", App: "tito", Region: "de",
+ Steering: "latency", Replicas: []string{"fr"}}
+ all[provision.RecordKey(shadow.Account, shadow.App, shadow.Host)] = shadow
+ if err := w.pass(all, siblingsEIP, [][]byte{[]byte("mesh-A")}); err != nil {
+ t.Fatalf("second pass: %v", err)
+ }
+
+ after, err := os.ReadFile(path)
+ if err != nil {
+ t.Fatalf("REVOKED: a record in another account's prefix removed this instance's %s: %v",
+ siblingsFileName, err)
+ }
+ if string(after) != string(before) {
+ t.Errorf("the file was rewritten under a contested host:\n before %s\n after %s", before, after)
+ }
+}
diff --git a/internal/host/siblings.go b/internal/host/siblings.go
index f2a947fbc..f25ff945c 100644
--- a/internal/host/siblings.go
+++ b/internal/host/siblings.go
@@ -18,6 +18,7 @@ import (
"os"
"path/filepath"
"sort"
+ "strings"
"sync/atomic"
"github.com/carlosframework/platform/internal/activator"
@@ -96,15 +97,26 @@ func deriveSiblingKey(meshKey []byte, label string) string {
// keyed by the record's IDENTITY (provision.RecordKey), never by host, and
// nothing here may look a record up by host alone: one slot per host is
// exactly what let a record written in one account's prefix be read as
-// another account's home. The returned map IS keyed by host, because a file
-// on a box is addressed by host — so every record that reaches it has been
-// asked whether it owns that host.
+// another account's home.
+//
+// The returned map IS keyed by host, because a file on a box is addressed by
+// host — so no CONTESTED host reaches it. A host two record identities claim
+// is a host this pass cannot write a file for: whichever claimant the file
+// was built from, the other one's instance is the process that reads it, and
+// the file hands out a dial address and a live pair key. Refusing both sides
+// is the only answer available, because there is no authority here to pick
+// between them (provision.ContestedHosts says why sort order is not one).
//
// eipOf is the edge's home map: the sibling box's EIP, or false for a box
// that has published no edges file yet. keys is the mesh key list, current
// first.
func buildSiblings(all map[string]provision.Record, eipOf func(host string) (string, bool), keys [][]byte) map[string]siblingsFile {
out := map[string]siblingsFile{}
+ // A pure function of `all`, so siblingsWriter.pass asking the same
+ // question of the same map cannot land on a different answer — the two
+ // derivations this file warns about elsewhere are two DIFFERENT lookups,
+ // which this is not.
+ contested := provision.ContestedHosts(all)
addr := func(host string) (string, bool) {
eip, ok := eipOf(host)
if !ok || eip == "" {
@@ -128,12 +140,13 @@ func buildSiblings(all map[string]provision.Record, eipOf func(host string) (str
// file" must stay different answers (spec §7).
//
// Written under the record's HOST, which is what a file on a box is
- // addressed by — so the record must be the one that OWNS that host
- // (provision.OwnsItsHost). Without that, a record naming another
- // account's host, written in any prefix, decides what goes in the file
- // the victim's own box writes for the victim's own instance.
+ // addressed by — so a CONTESTED host gets nothing. Without that, a
+ // record naming another account's host, written in any prefix, decides
+ // what goes in the file the victim's own box writes for the victim's own
+ // instance: it took the victim's placement, its replica list, and the
+ // account the pair key is derived from.
for _, rec := range all {
- if rec.Role == "replica" || len(rec.Replicas) == 0 || !provision.OwnsItsHost(all, rec) {
+ if rec.Role == "replica" || len(rec.Replicas) == 0 || len(contested[rec.Host]) > 0 {
continue
}
out[rec.Host] = siblingsFile{Role: "home", Host: rec.Host, Placement: rec.Region}
@@ -143,16 +156,6 @@ func buildSiblings(all map[string]provision.Record, eipOf func(host string) (str
if rec.Role != "replica" || rec.Home == "" {
continue
}
- // The replica's own file is written under the replica's host, so the
- // same ownership question is asked of it: a record in any prefix
- // naming ANOTHER account's replica host, with a perfectly valid home
- // of its own behind it, would otherwise hand that instance a file
- // naming a stranger as its home. The reconcile pass settles this for
- // routes with its duplicate-host arbitration; this pass has no
- // routes, so it asks the map the same question.
- if !provision.OwnsItsHost(all, rec) {
- continue
- }
host := rec.Host
// The CROSS-RECORD rails of the console's create door (spec §6),
// asked through the same function the reconcile pass asks — the two
@@ -165,8 +168,12 @@ func buildSiblings(all map[string]provision.Record, eipOf func(host string) (str
// instance with a dial address and a live pair key, telling the
// home's app to grant leases to it and accept drains from it.
// Nothing to pair with is the answer to every one of them: a file
- // naming an address nobody vouched for is worse than no file.
- homeRec, why, _ := provision.PairableHome(all, rec)
+ // naming an address nobody vouched for is worse than no file. That
+ // includes a contested host on either side of the pairing — the
+ // replica's own host as much as its home's, since this file is
+ // written under the replica's host and read by whichever instance
+ // actually holds it.
+ homeRec, why := provision.PairableHome(all, contested, rec)
if why != "" {
continue
}
@@ -268,8 +275,18 @@ func (w *siblingsWriter) pass(all map[string]provision.Record, eipOf func(host s
// write keyed by host would leave two flat rows overwriting each other's
// keys turn by turn, with whichever app read last holding a key minted
// for someone else's pair.
+ //
+ // held is the same decision for a CONTESTED host: buildSiblings writes
+ // nothing for one, and "nothing built" is otherwise how this pass learns
+ // a placement was withdrawn and removes the file. Removing it here would
+ // hand a writer who plants one record the power to take a live pair's
+ // file off the box — the revocation the rails exist to refuse. A
+ // directory holding a contested host is left exactly as it is instead:
+ // no write, no removal.
+ contested := provision.ContestedHosts(all)
var dirs []string
owners := map[string][]string{}
+ held := map[string][]string{}
for _, rt := range routes {
dir := w.dirFor(rt)
if _, seen := owners[dir]; !seen {
@@ -279,6 +296,9 @@ func (w *siblingsWriter) pass(all map[string]provision.Record, eipOf func(host s
if _, ok := built[rt.Host]; ok {
owners[dir] = append(owners[dir], rt.Host)
}
+ if len(contested[rt.Host]) > 0 {
+ held[dir] = append(held[dir], rt.Host)
+ }
}
sort.Strings(dirs)
@@ -289,6 +309,15 @@ func (w *siblingsWriter) pass(all map[string]provision.Record, eipOf func(host s
}
}
for _, dir := range dirs {
+ if h := held[dir]; len(h) > 0 {
+ sort.Strings(h)
+ for _, host := range h {
+ log.Printf("agent: siblings: %s is claimed by more than one record (%s), so %s in %s "+
+ "is left exactly as it is — this pass cannot tell which claimant a file there "+
+ "would belong to", host, strings.Join(contested[host], ", "), siblingsFileName, dir)
+ }
+ continue
+ }
hosts := owners[dir]
sort.Strings(hosts)
if len(hosts) > 1 {
diff --git a/internal/host/siblings_test.go b/internal/host/siblings_test.go
index 330d0a053..75d1edf2c 100644
--- a/internal/host/siblings_test.go
+++ b/internal/host/siblings_test.go
@@ -196,58 +196,18 @@ func TestBuildSiblingsRefusesAnUnvouchedForReplica(t *testing.T) {
// attacker lands in the VICTIM's own siblings file with a dial address and a
// live pair key: the victim's app is told to grant leases to a stranger.
//
-// A host belongs to ONE record — the one the bucket listing reaches first,
-// which is the arbitration the reconcile pass already makes when two records
-// claim a host. A replica may only be vouched for by the record that WON the
-// host it names, because that is the record whose route the public host is
-// actually served from.
-func TestBuildSiblingsRefusesAShadowOfTheVictimsHost(t *testing.T) {
- const victim = "acme.go.tito.io"
- all := recordMap(
- // The victim, in account "bab", and the replica it really opted into.
- provision.Record{Host: victim, Account: "bab", App: "tito", Region: "ie",
- Steering: "latency", Replicas: []string{"uk"}},
- provision.Record{Host: "r7x2.go.tito.io", Account: "bab", App: "tito", Region: "uk",
- Role: "replica", Home: victim},
- // The attacker, in "mal" — a prefix that sorts AFTER the victim's, so
- // a host-keyed map would hold this record and not the victim's.
- provision.Record{Host: victim, Account: "mal", App: "tito", Region: "ie",
- Steering: "latency", Replicas: []string{"de"}},
- provision.Record{Host: "r9z9.go.tito.io", Account: "mal", App: "tito", Region: "de",
- Role: "replica", Home: victim},
- // And the other direction: a record of the attacker's own, entirely
- // valid on its own terms — a home it does own, a placement it opted
- // into — except that it claims the VICTIM's REPLICA host. The file
- // this pass writes is addressed by host, so without the same
- // ownership question it would hand the victim's replica a file
- // naming a stranger as its home.
- provision.Record{Host: "mal.go.tito.io", Account: "mal", App: "other", Region: "ie",
- Steering: "latency", Replicas: []string{"uk"}},
- provision.Record{Host: "r7x2.go.tito.io", Account: "mal", App: "other", Region: "uk",
- Role: "replica", Home: "mal.go.tito.io"},
- )
- got := buildSiblings(all, func(string) (string, bool) { return "203.0.113.9", true }, [][]byte{[]byte("cur")})
-
- home, ok := got[victim]
- if !ok {
- t.Fatal("the victim's own home file went missing: a shadow in another prefix took it")
- }
- if len(home.Replicas) != 1 || home.Replicas[0].Host != "r7x2.go.tito.io" {
- t.Errorf("the victim's file lists %+v, want only the replica it opted into — anything else "+
- "is an address and a pair key the victim never vouched for", home.Replicas)
- }
- if _, ok := got["r9z9.go.tito.io"]; ok {
- t.Errorf("the shadow's replica got a file of its own: %+v", got["r9z9.go.tito.io"])
- }
- rf, ok := got["r7x2.go.tito.io"]
- if !ok || rf.Home == nil {
- t.Fatalf("the victim's replica lost its own file: %+v", rf)
- }
- if rf.Home.Host != victim {
- t.Errorf("the victim's replica is told its home is %q — a record claiming that host in "+
- "another account's prefix rewrote the file", rf.Home.Host)
- }
-}
+// The answer is NOT that a host belongs to whichever record the listing
+// reaches first. That was tried, and an attacker who picks an account prefix
+// sorting before the victim's is simply "first" — so the rail vouched for
+// them. A host two identities claim is CONTESTED and vouches for nothing at
+// all, on either side.
+//
+// The cases live in replica_adversarial_test.go, in both sort orders:
+// TestBuildSiblingsRefusesAContestedHostInEitherSortOrder,
+// TestBuildSiblingsRefusesAContestedReplicaHost, and
+// TestSiblingsPassLeavesTheFileAloneWhenTheHostIsContested — which pins the
+// other half of fail-safe: a contested host does not cost an instance the
+// file it already has.
// --- the writer ---
diff --git a/internal/provision/reconcile.go b/internal/provision/reconcile.go
index ad500e195..d48a73a31 100644
--- a/internal/provision/reconcile.go
+++ b/internal/provision/reconcile.go
@@ -17,6 +17,7 @@ import (
"errors"
"fmt"
"log"
+ "maps"
"net"
"os"
"slices"
@@ -88,6 +89,15 @@ type ReconcileConfig struct {
// account's prefix can occupy another account's slot and be read as
// that account's record. Look a host up the way this pass does:
// all[RecordKey(acct, app, host)], inside the tenancy asking.
+ //
+ // A reader that pairs records must also ask ContestedHosts of this map
+ // before it believes any host in it: two identities can still claim one
+ // host, and neither of them may be used to vouch for the other side of a
+ // pairing. The one thing this map cannot show is a claim whose body did
+ // not decode — the pass counts those off the bucket listing, which the
+ // callback never sees — so a reader is fail-safe against a shadow, not
+ // against a shadow plus a simultaneous read failure of the record it
+ // shadows (spec §6).
OnRecords func(all map[string]Record)
}
@@ -708,31 +718,56 @@ func convergeOwned(reg *registry.Registry, existing registry.Route, rec Record,
}
}
-// OwnsItsHost reports whether rec is the record that OWNS its host in the
-// fleet-wide record map — the one a bucket listing reaches FIRST when two
-// records claim the same host. That is the arbitration Reconcile already
-// makes when it mints (the duplicate-host skip below, "whichever record this
-// listing reaches first wins the host"); the map is keyed by RecordKey and a
-// listing is lexicographic, so "first" is the smallest key, and every box
-// reaches the same answer off the same bucket with nobody coordinating.
+// ContestedHosts reports every host that more than one record IDENTITY
+// claims, mapped to those identities' record keys, sorted. One claim is the
+// ordinary case; two or more is CONTESTED, and a contested host is one this
+// pass refuses to reason about at all.
//
-// One host, two records, is not a state one account's create door produces —
+// One host, two records, is not a state a create door produces —
// CreateRecord is create-only on the key — but the key carries the account
// and the app, so nothing stops a SECOND account's record from naming the
-// same host: a hand edit, a writer with its own bucket access, or two
-// accounts reaching for one custom domain. Whoever loses that arbitration
-// has no route anywhere: the host is served from the winner's row. So a
-// replica vouched for by a losing record is vouched for by a record that
-// serves nothing, and the alias it mints would divert the WINNER's public
-// host to a box nobody pointed it at.
-func OwnsItsHost(all map[string]Record, rec Record) bool {
- mine := RecordKey(rec.Account, rec.App, rec.Host)
- for key, other := range all {
- if other.Host == rec.Host && key < mine {
- return false
+// same host: a hand edit, two accounts reaching for one custom domain, or a
+// writer holding bucket-wide s3:PutObject, which every region box's IAM role
+// does.
+//
+// What this pass must NOT do is pick a winner. It tried twice. Sort order
+// was the only tiebreak available — a bucket listing is lexicographic, so
+// "first" is the smallest key, and every box reaches the same answer with
+// nobody coordinating — and sort order is not authority: an attacker choosing
+// an account prefix that sorts before the victim's simply became the owner,
+// and the rail vouched for THEM. The other half was worse: when the victim's
+// record lost, the victim's own live replica failed the rail, and the pass
+// answered that by clearing the link, so one planted record severed a
+// running pair.
+//
+// There is no attacker-proof ownership source available here, so the rail is
+// fail-safe only. It refuses to vouch, in both directions, and it never
+// revokes. A writer who can plant a record can therefore deny service by
+// contesting a host, and no arrangement of this pass can stop them — the fix
+// for that is narrowing the grant, which is infrastructure work (spec §6).
+func ContestedHosts(all map[string]Record) map[string][]string {
+ claims := map[string][]string{}
+ for key, rec := range all {
+ claims[rec.Host] = append(claims[rec.Host], key)
+ }
+ return contestedOf(claims)
+}
+
+// contestedOf keeps the hosts with more than one claimant and sorts each
+// list, so every box logs and refuses off the same string.
+func contestedOf(claims map[string][]string) map[string][]string {
+ var contested map[string][]string
+ for host, keys := range claims {
+ if len(keys) < 2 {
+ continue
}
+ if contested == nil {
+ contested = map[string][]string{}
+ }
+ slices.Sort(keys)
+ contested[host] = keys
}
- return true
+ return contested
}
// PairableHome resolves the home record a replica record names and reports
@@ -755,28 +790,39 @@ func OwnsItsHost(all map[string]Record, rec Record) bool {
// could therefore stand in for the victim's, pass the comparisons against
// itself, and take the victim's real record out of the map on the way past.
//
+// contested is ContestedHosts' answer for the same map (the reconcile pass
+// counts claims off the bucket LISTING, so a record whose body would not
+// read still holds its claim). A contested host vouches for nothing and is
+// vouched for by nothing: neither the replica's own host nor its home may be
+// one, because the file and the route this pairing produces are addressed by
+// host, and two records claiming one host means the pass cannot say whose
+// they would be.
+//
// why is "" when the pair may be made. Otherwise it names the condition that
// failed, for the log, because "not minting" with no reason is how an
-// operator's typo becomes an afternoon — and protect reports whether the
-// caller should leave an existing row exactly as it is (see the two answers
-// at Reconcile's replica branch).
-func PairableHome(all map[string]Record, rec Record) (home Record, why string, protect bool) {
+// operator's typo becomes an afternoon. There is no second answer: EVERY
+// failure here leaves the caller's existing state exactly as it is.
+func PairableHome(all map[string]Record, contested map[string][]string, rec Record) (home Record, why string) {
+ if by, dup := contested[rec.Host]; dup {
+ return Record{}, "its own host is claimed by more than one record (" + strings.Join(by, ", ") + ")"
+ }
+ if by, dup := contested[rec.Home]; dup {
+ return Record{}, "its home " + rec.Home + " is claimed by more than one record (" + strings.Join(by, ", ") + ")"
+ }
home, ok := all[RecordKey(rec.Account, rec.App, rec.Home)]
switch {
case !ok:
// Absent, unreadable this pass, or in another account or app — one
// answer, because the lookup cannot tell them apart and the pass must
- // not guess. A mirror of nothing: protected, since a bucket blip must
- // never sweep a live placement's alias.
- return Record{}, "no record for its home " + rec.Home + " in this account and app", true
- case !OwnsItsHost(all, home):
- return Record{}, "its home " + rec.Home + " is not the record that owns that host", false
+ // not guess. A mirror of nothing, and a bucket blip must never sweep
+ // a live placement's alias.
+ return Record{}, "no record for its home " + rec.Home + " in this account and app"
case home.Role != "":
- return Record{}, "its home " + rec.Home + " is itself a replica", false
+ return Record{}, "its home " + rec.Home + " is itself a replica"
case !slices.Contains(home.Replicas, rec.Region):
- return Record{}, "its home " + rec.Home + " has not opted into placement " + rec.Region, true
+ return Record{}, "its home " + rec.Home + " has not opted into placement " + rec.Region
}
- return home, "", false
+ return home, ""
}
// Reconcile materializes bucket instance records into registry routes for
@@ -816,11 +862,20 @@ func Reconcile(ctx context.Context, s release.Store, reg *registry.Registry, cfg
// back — every other record still converges/removes normally the same
// pass.
protect := map[string]bool{}
+ // claims is host -> the record KEYS naming it, counted off the LISTING
+ // rather than off the decoded records: a key is a claim on its host
+ // whether or not its body reads, and a claim that disappears when a body
+ // fails to decode is a claim an attacker can make disappear. A record
+ // the victim's box could not read this pass once left the attacker's
+ // shadow of that host as the only claimant, which made it the owner and
+ // minted the attacker's replica during the blip.
+ claims := map[string][]string{}
for _, key := range keys {
acct, app, host, ok := ParseRecordKey(key, known)
if !ok {
continue
}
+ claims[host] = append(claims[host], RecordKey(acct, app, host))
raw, _, err := s.Get(ctx, key)
if err != nil {
log.Printf("provision: skipping record %s: read failed: %v", key, err)
@@ -919,6 +974,18 @@ func Reconcile(ctx context.Context, s release.Store, reg *registry.Registry, cfg
}
recs[rec.Host] = rec
}
+ // One line per contested host per pass, naming both claimants. Two
+ // records claiming one host in a deployment is not a condition the pass
+ // can resolve and it is not one it should swallow: it is either an
+ // operator's mistake or a writer attacking an account, and both want a
+ // human. The list is sorted inside contestedOf so the line is the same
+ // on every box and does not churn between passes.
+ contested := contestedOf(claims)
+ for _, host := range slices.Sorted(maps.Keys(contested)) {
+ log.Printf("provision: host %s is contested — claimed by %s. No replica pairing may use it "+
+ "until one of them goes; existing rows are left exactly as they are",
+ host, strings.Join(contested[host], ", "))
+ }
if cfg.OnRecords != nil {
cfg.OnRecords(all)
}
@@ -1035,49 +1102,32 @@ func Reconcile(ctx context.Context, s release.Store, reg *registry.Registry, cfg
// this box, and is written into the VICTIM's own siblings file as a
// vouched-for replica with a live pair key.
//
- // A failure has TWO answers, and which one is right depends on
- // whether a legitimate writer could have caused it:
+ // A failure has ONE answer: PROTECT. The existing row goes on
+ // serving exactly what it was serving, nothing is minted where
+ // nothing is, and no column is written. Nothing in this pass clears
+ // a replica link, and that is a deliberate reversal.
//
- // protect — the home is not readable this pass, or its `replicas`
- // no longer names this placement. An OLD CONSOLE strips that
- // field by round-tripping the record (spec §6), and a bucket
- // read can simply fail, so the existing row goes on serving what
- // it serves. That is the fail-safe direction: a console deploy
- // must not be able to take a live placement down.
+ // adf29ebd split the answer: rails that "cannot fail on a record the
+ // doors wrote" cleared the link (SetReplicaOf ""), on the argument
+ // that whatever failed them was a writer outside the doors, so
+ // protecting the row bought no fail-safety. The argument is wrong in
+ // its first move — "outside the doors" includes a writer attacking
+ // this account, and clearing is the only irreversible thing this
+ // branch could do. A single record planted in another account's
+ // prefix, naming the victim's host, in a region no box even serves,
+ // was enough to make the victim's own box clear the victim's own
+ // live link and have the domains pass sweep the alias behind it.
+ // Every rail is a refusal to VOUCH; none of them is a revocation.
//
- // clear the link — every other rail. None of them can fail on a
- // record the doors wrote, so what failed them is a writer
- // outside the doors, and protecting THAT leaves a row nobody
- // vouched for serving an account's public host indefinitely, for
- // no fail-safe benefit at all. SetReplicaOf("") drops the link;
- // the domains pass sweeps the alias on its next run. Not
- // reg.Delete: removal carries a grant purge (see the removal
- // phase below) and nothing here is a retirement the bucket
- // asked for.
- //
- // The condition that failed is named in the log either way, because
- // "not minting" with no reason is how an operator's typo becomes an
+ // The condition that failed is named in the log, because "not
+ // minting" with no reason is how an operator's typo becomes an
// afternoon.
if rec.Role == "replica" {
var why string
- var keep bool
- home, why, keep = PairableHome(all, rec)
+ home, why = PairableHome(all, contested, rec)
if why != "" {
log.Printf("provision: not minting replica %s: %s", rec.Host, why)
- switch existing, ok := byHost[rec.Host]; {
- case keep:
- protect[rec.Host] = true
- case !ok || existing.ReplicaOf == "":
- // Nothing minted here, or nothing linked: nothing to clear.
- case cfg.DryRun:
- log.Printf("provision: dry-run: would clear %s replica_of %q", rec.Host, existing.ReplicaOf)
- default:
- if err := reg.SetReplicaOf(rec.Host, ""); err != nil {
- log.Printf("provision: clear replica_of on %s: %v", rec.Host, err)
- } else {
- log.Printf("provision: cleared %s replica_of %q: %s", rec.Host, existing.ReplicaOf, why)
- }
- }
+ protect[rec.Host] = true
continue
}
}
diff --git a/internal/provision/reconcile_test.go b/internal/provision/reconcile_test.go
index eba11f8ee..f1344c42e 100644
--- a/internal/provision/reconcile_test.go
+++ b/internal/provision/reconcile_test.go
@@ -2980,83 +2980,61 @@ func TestReconcileRefusesAReplicaInAPlacementTheHomeNeverOptedInto(t *testing.T)
// The map is now keyed by the record's IDENTITY — account, app and host —
// so no record can occupy another's slot, and the home a replica names is
// looked up inside the replica's own account and app.
+//
+// Identity keying alone does not close the mint, though: the attacker's
+// shadow is a perfectly good home for the attacker's OWN replica, so every
+// per-record rail passes and the victim's host is aliased anyway. What
+// closes it is that the host is CONTESTED — two identities claim it — and a
+// contested host vouches for nothing. This test pins the cost of that
+// answer, which is a denial of service the pass cannot prevent: while the
+// shadow stands, the victim's own replica does not mint either. Refusing to
+// mint is reversible the moment the shadow goes; aliasing the victim's
+// public host to a stranger is not.
+//
+// Both sort orders, because the rail this replaced held for only one of
+// them — and its test guarded the fixture so the other half never ran. The
+// rest of the family is in replica_adversarial_test.go.
func TestReconcileShadowHomeRecordCannotDisplaceTheVictims(t *testing.T) {
- s, sqid, home, replica := replicaSetup(t)
- mallory := writeAccountManifest(t, s, 3, "Mallory")
- if mallory <= sqid {
- t.Fatalf("fixture: the attacker's prefix %q must sort AFTER the victim's %q, or the "+
- "shadow never reaches the map slot this test is about", mallory, sqid)
- }
- if err := PutManifest(ctx, s, mallory, "tito", Manifest{
- Schema: 1, App: "tito", Domain: "go.tito.io", Pool: "tito", Seed: []string{"seed"},
- }); err != nil {
- t.Fatal(err)
- }
- // The shadow: the VICTIM's host, in the ATTACKER's prefix, carrying the
- // attacker's own account and app so it agrees with its own path — and
- // opted into uk, so the attacker's replica below can point at it.
- shadow := Record{Schema: 1, Host: home.Host, Slug: "acme", App: "tito",
- Account: mallory, Pool: "tito", Region: "ie", Channel: "stable", CreatedAt: time.Now(),
- Steering: "latency", Replicas: []string{"uk"}}
- if err := CreateRecord(ctx, s, shadow); err != nil {
- t.Fatal(err)
- }
- evil := Record{Schema: 1, Host: "r4444444444.go.tito.io", Slug: "r4444444444", App: "tito",
- Account: mallory, Pool: "tito", Region: "uk", Channel: "stable", CreatedAt: time.Now(),
- Role: "replica", Home: home.Host}
- if err := CreateRecord(ctx, s, evil); err != nil {
- t.Fatal(err)
- }
- uk := regWith(t)
- if err := Reconcile(ctx, s, uk, ReconcileConfig{Region: "uk", Pools: []string{"tito"}, DataDir: "/data"}); err != nil {
- t.Fatal(err)
- }
- // Get reports a miss as (nil, nil), never an error.
- if rt, err := uk.Get(evil.Host); err != nil {
- t.Fatal(err)
- } else if rt != nil {
- t.Errorf("a replica vouched for by a SHADOW of the victim's host was minted: %+v — the "+
- "victim's public host is now aliased to this row", rt)
- }
- // And the denial-of-service half: the victim's own pair is untouched by
- // a record in a prefix the victim has no sight of.
- if got := mustGet(t, uk, replica.Host); got.ReplicaOf != home.Host {
- t.Errorf("the victim's own replica has ReplicaOf %q, want %q — a shadow record in another "+
- "account's prefix took its home away", got.ReplicaOf, home.Host)
+ for _, order := range bothSortOrders(t) {
+ t.Run(order.name, func(t *testing.T) {
+ s, _, home, replica := replicaSetup(t)
+ mallory := plantShadow(t, s, order.id, home.Host, []string{"uk"})
+ evil := Record{Schema: 1, Host: "r4444444444.go.tito.io", Slug: "r4444444444", App: "tito",
+ Account: mallory, Pool: "tito", Region: "uk", Channel: "stable", CreatedAt: time.Now(),
+ Role: "replica", Home: home.Host}
+ if err := CreateRecord(ctx, s, evil); err != nil {
+ t.Fatal(err)
+ }
+ uk := regWith(t)
+ if err := Reconcile(ctx, s, uk, ReconcileConfig{Region: "uk", Pools: []string{"tito"}, DataDir: "/data"}); err != nil {
+ t.Fatal(err)
+ }
+ // Get reports a miss as (nil, nil), never an error.
+ if rt, err := uk.Get(evil.Host); err != nil {
+ t.Fatal(err)
+ } else if rt != nil {
+ t.Errorf("a replica vouched for by a SHADOW of the victim's host was minted: %+v — the "+
+ "victim's public host is now aliased to this row", rt)
+ }
+ // The accepted cost, stated out loud so nobody mistakes it for
+ // an oversight: the victim's own replica is refused too, because
+ // the pass has no way to tell the two claimants apart and
+ // guessing is what it stopped doing.
+ if rt, err := uk.Get(replica.Host); err != nil {
+ t.Fatal(err)
+ } else if rt != nil {
+ t.Errorf("the victim's replica minted off a contested host: %+v — one of the two "+
+ "claimants was believed, and the pass has nothing to believe one with", rt)
+ }
+ })
}
}
-// Rails that can never fail on a door-written record fail here only because
-// something outside the doors wrote the record — so leaving the row serving
-// buys no fail-safety, it just leaves a link nobody vouched for in place.
-// The link is CLEARED (SetReplicaOf "", which the domains pass sweeps on its
-// next alias reconcile) rather than the row deleted: registry.Delete carries
-// a grant purge, and none of this is a retirement the bucket asked for.
-func TestReconcileClearsTheLinkWhenTheHomeIsItselfAReplica(t *testing.T) {
- s, sqid, home, replica := replicaSetup(t)
- uk := regWith(t)
- cfg := ReconcileConfig{Region: "uk", Pools: []string{"tito"}, DataDir: "/data"}
- if err := Reconcile(ctx, s, uk, cfg); err != nil {
- t.Fatal(err)
- }
- if got := mustGet(t, uk, replica.Host).ReplicaOf; got != home.Host {
- t.Fatalf("fixture: ReplicaOf = %q, want %q", got, home.Host)
- }
- // The home turns into a replica underneath its own replica: a chain, and
- // no door makes one. (A record carrying both `role` and `replicas` is
- // refused by ValidateShape before any of this, so the rewrite is the
- // shape a hand edit would actually leave behind.)
- home.Role, home.Home, home.Replicas, home.Steering = "replica", "somewhere.go.tito.io", nil, ""
- putReplace(t, s, RecordKey(sqid, "tito", home.Host), mustJSON(t, home))
- if err := Reconcile(ctx, s, uk, cfg); err != nil {
- t.Fatal(err)
- }
- rt := mustGet(t, uk, replica.Host) // still serving; only the link goes
- if rt.ReplicaOf != "" {
- t.Errorf("ReplicaOf = %q, want empty: the alias goes on sending the home's public host "+
- "to this row for as long as the link stands", rt.ReplicaOf)
- }
-}
+// Nothing in this pass clears a replica link any more — see
+// TestReconcileNeverClearsALiveReplicaLink in replica_adversarial_test.go,
+// which takes every rail in turn against a LIVE row. adf29ebd cleared the
+// link for the rails it judged undoor-able; a planted record then used that
+// as a revocation and severed a running pair.
// The opt-in is the one rail an OLD CONSOLE can legitimately fail: it
// marshals a record it has never heard the `replicas` field on, and writing
diff --git a/internal/provision/replica_adversarial_test.go b/internal/provision/replica_adversarial_test.go
new file mode 100644
index 000000000..9afa12a6c
--- /dev/null
+++ b/internal/provision/replica_adversarial_test.go
@@ -0,0 +1,387 @@
+package provision
+
+import (
+ "testing"
+ "time"
+
+ "github.com/carlosframework/platform/internal/domains"
+ "github.com/carlosframework/platform/internal/release"
+)
+
+// The adversarial half of the replica rails (#162). Every record here is one
+// a WRITER planted — not one any door wrote — and every case was reproduced
+// against a version of this pass that believed it was already defended.
+//
+// The history is the point. 1a5f819f asked the console door's four
+// cross-record questions of a record map keyed by HOST, so a record in any
+// prefix could occupy the victim's one slot. adf29ebd keyed the map by the
+// record's IDENTITY and added a fifth rail — the home must be the
+// FIRST-SORTING claimant of its host — and that rail broke twice, both times
+// because sort order was being read as authority:
+//
+// - an attacker whose account prefix sorts BEFORE the victim's IS the
+// first-sorting claimant, so the rail vouched for THEIR replica and
+// aliased the victim's public host to the attacker's process;
+// - an earlier-sorting PLAIN record carrying the victim's host, in a region
+// no box serves, made the victim's own box CLEAR the victim's real
+// replica link — a live pair severed by a record the victim cannot see
+// and cannot delete.
+//
+// No attacker-proof ownership source is available to this pass, so it stopped
+// trying to pick a winner. A host claimed by more than one record IDENTITY is
+// CONTESTED; a contested host vouches for nothing; and every refusal is
+// fail-safe — nothing minted, nothing converged, and NOTHING cleared.
+//
+// Each case runs in BOTH sort orders. A rail that holds for half the alphabet
+// is not a rail, and the test that shipped with the fifth rail guarded its
+// own fixture with `if mallory <= sqid { t.Fatalf }` — which is how half a
+// rail passes for an afternoon.
+
+// earlierSortingID and laterSortingID hand back an account id whose sqid
+// sorts before (after) the victim's. A bucket listing is lexicographic and a
+// record key begins with the account's sqid, so this is the only thing "first"
+// ever meant. Searched rather than hardcoded, for the same reason
+// writeAccountManifest computes its sqid instead of writing a literal.
+func earlierSortingID(t *testing.T, victim string) int64 {
+ t.Helper()
+ for id := int64(3); id < 100000; id++ {
+ if domains.Sqid(id) < victim {
+ return id
+ }
+ }
+ t.Fatalf("no account id encodes to a sqid sorting before %q", victim)
+ return 0
+}
+
+func laterSortingID(t *testing.T, victim string) int64 {
+ t.Helper()
+ for id := int64(3); id < 100000; id++ {
+ if domains.Sqid(id) > victim {
+ return id
+ }
+ }
+ t.Fatalf("no account id encodes to a sqid sorting after %q", victim)
+ return 0
+}
+
+// sortOrder is one attacker fixture: a name for the subtest and the account
+// id whose sqid lands on the chosen side of the victim's.
+type sortOrder struct {
+ name string
+ id int64
+}
+
+// bothSortOrders is the pair every case below runs under.
+func bothSortOrders(t *testing.T) []sortOrder {
+ t.Helper()
+ victim := domains.Sqid(acctID)
+ return []sortOrder{
+ {"attacker prefix sorts EARLIER than the victim's", earlierSortingID(t, victim)},
+ {"attacker prefix sorts LATER than the victim's", laterSortingID(t, victim)},
+ }
+}
+
+// plantShadow writes an attacker account and its app manifest, then a SHADOW
+// of the victim's host inside the attacker's OWN prefix: the attacker's
+// account, the attacker's app, so the record agrees with its own path and
+// every per-record rail passes it. It hands back the attacker's sqid.
+func plantShadow(t *testing.T, s release.Store, id int64, victimHost string, replicas []string) string {
+ t.Helper()
+ eve := writeAccountManifest(t, s, id, "Eve")
+ if err := PutManifest(ctx, s, eve, "tito", Manifest{
+ Schema: 1, App: "tito", Domain: "go.tito.io", Pool: "tito", Seed: []string{"seed"},
+ }); err != nil {
+ t.Fatal(err)
+ }
+ shadow := Record{Schema: 1, Host: victimHost, Slug: "acme", App: "tito",
+ Account: eve, Pool: "tito", Region: "ie", Channel: "stable", CreatedAt: time.Now(),
+ Replicas: replicas}
+ if len(replicas) > 0 {
+ shadow.Steering = "latency"
+ }
+ if err := CreateRecord(ctx, s, shadow); err != nil {
+ t.Fatal(err)
+ }
+ return eve
+}
+
+// A replica vouched for by a SHADOW of the victim's host must never mint.
+// Minting it hands the attacker's process the victim's public host on that
+// box: the domains pass derives the alias from the row's replica link, so the
+// victim's customers arrive at the attacker.
+//
+// adf29ebd refused this only when the attacker's prefix sorted LATER. In the
+// other half of the alphabet the attacker WAS the first-sorting claimant, so
+// the fifth rail vouched for them — the original P1, intact.
+func TestReconcileRefusesAReplicaVouchedForByAShadowInEitherSortOrder(t *testing.T) {
+ for _, order := range bothSortOrders(t) {
+ t.Run(order.name, func(t *testing.T) {
+ s, _, home, replica := replicaSetup(t)
+ uk := regWith(t)
+ cfg := ReconcileConfig{Region: "uk", Pools: []string{"tito"}, DataDir: "/data"}
+ // The victim's pair is LIVE before the attacker arrives. That is
+ // the state the fail-safe exists for, and a fresh registry would
+ // hide a link being cleared.
+ if err := Reconcile(ctx, s, uk, cfg); err != nil {
+ t.Fatal(err)
+ }
+ if got := mustGet(t, uk, replica.Host).ReplicaOf; got != home.Host {
+ t.Fatalf("fixture: the victim's replica has ReplicaOf %q, want %q", got, home.Host)
+ }
+
+ eve := plantShadow(t, s, order.id, home.Host, []string{"uk"})
+ evil := Record{Schema: 1, Host: "r4444444444.go.tito.io", Slug: "r4444444444", App: "tito",
+ Account: eve, Pool: "tito", Region: "uk", Channel: "stable", CreatedAt: time.Now(),
+ Role: "replica", Home: home.Host}
+ if err := CreateRecord(ctx, s, evil); err != nil {
+ t.Fatal(err)
+ }
+ if err := Reconcile(ctx, s, uk, cfg); err != nil {
+ t.Fatal(err)
+ }
+
+ // Get reports a miss as (nil, nil), never an error.
+ if rt, err := uk.Get(evil.Host); err != nil {
+ t.Fatal(err)
+ } else if rt != nil {
+ t.Errorf("a replica vouched for by a SHADOW of the victim's host was minted: %+v — the "+
+ "victim's public host is now aliased to the attacker's process on this box", rt)
+ }
+ // The fail-safe half: the victim's live link is left exactly as
+ // it was. A refusal refuses to VOUCH; it never revokes something
+ // already serving.
+ if got := mustGet(t, uk, replica.Host).ReplicaOf; got != home.Host {
+ t.Errorf("the victim's live replica link is %q, want %q — a record the victim cannot "+
+ "see severed a running pair", got, home.Host)
+ }
+ })
+ }
+}
+
+// The second break, and the one that made "clear the link" indefensible. An
+// earlier-sorting PLAIN record — no attacker replica anywhere, nothing to
+// mint — carrying the victim's host, in a region no box in the deployment
+// serves. Under adf29ebd the victim's own record was then not the
+// first-sorting claimant of its own host, so the victim's replica failed the
+// fifth rail, that rail answered `protect=false`, and the victim's box
+// CLEARED the victim's real replica link. The domains pass swept the alias
+// next run while another box went on minting the victim's home.
+//
+// Nothing in this pass clears a link any more. The refusal is the whole
+// remedy, and a refusal leaves what is already serving alone.
+func TestReconcilePlainShadowNeverSeversTheVictimsLivePair(t *testing.T) {
+ for _, order := range bothSortOrders(t) {
+ t.Run(order.name, func(t *testing.T) {
+ s, sqid, home, replica := replicaSetup(t)
+ uk := regWith(t)
+ cfg := ReconcileConfig{Region: "uk", Pools: []string{"tito"}, DataDir: "/data"}
+ if err := Reconcile(ctx, s, uk, cfg); err != nil {
+ t.Fatal(err)
+ }
+ if got := mustGet(t, uk, replica.Host).ReplicaOf; got != home.Host {
+ t.Fatalf("fixture: the victim's replica has ReplicaOf %q, want %q", got, home.Host)
+ }
+
+ // A plain record: no `replicas`, no role, region "de" — no box in
+ // this test serves it, so it materializes nowhere and there is
+ // nothing for an operator to notice but the log line.
+ eve := writeAccountManifest(t, s, order.id, "Eve")
+ if err := PutManifest(ctx, s, eve, "tito", Manifest{
+ Schema: 1, App: "tito", Domain: "go.tito.io", Pool: "tito",
+ }); err != nil {
+ t.Fatal(err)
+ }
+ shadow := Record{Schema: 1, Host: home.Host, Slug: "acme", App: "tito",
+ Account: eve, Pool: "tito", Region: "de", Channel: "stable", CreatedAt: time.Now()}
+ if err := CreateRecord(ctx, s, shadow); err != nil {
+ t.Fatal(err)
+ }
+ if err := Reconcile(ctx, s, uk, cfg); err != nil {
+ t.Fatal(err)
+ }
+
+ rt := mustGet(t, uk, replica.Host)
+ if rt.ReplicaOf != home.Host {
+ t.Errorf("SEVERED: the victim's replica link is %q, want %q — an out-of-scope plain "+
+ "record in another account's prefix took a live pair down", rt.ReplicaOf, home.Host)
+ }
+ // Protection means the row goes on serving exactly what it was
+ // serving, not just that it keeps its link.
+ if rt.Channel != "stable" {
+ t.Errorf("the victim's replica channel is %q, want stable", rt.Channel)
+ }
+
+ // The other half of the severed pair: the home's own box still
+ // mints the victim's home for the VICTIM, so before this fix the
+ // deployment held a home nothing pointed at and a replica row
+ // that had forgotten it.
+ ie := regWith(t)
+ if err := Reconcile(ctx, s, ie, ReconcileConfig{Region: "ie", Pools: []string{"tito"}, DataDir: "/data"}); err != nil {
+ t.Fatal(err)
+ }
+ if got := mustGet(t, ie, home.Host).Account; got != sqid {
+ t.Errorf("the home box minted %s for account %q, want the victim's %q", home.Host, got, sqid)
+ }
+ })
+ }
+}
+
+// A transient read failure of the VICTIM's record is not a licence to vouch
+// for the attacker. The record is still there and the listing still names its
+// key, so the host is contested by the KEY alone — the pass counts claims off
+// the bucket listing, before any body is decoded, exactly so a body it cannot
+// read cannot be made to disappear.
+//
+// Under adf29ebd an unreadable victim record simply vanished from the map,
+// which left the shadow the only claimant — first-sorting by default — and
+// the attacker's replica minted during the blip, in BOTH sort orders.
+func TestReconcileRefusesAShadowVouchedReplicaWhileTheVictimsRecordBlips(t *testing.T) {
+ for _, order := range bothSortOrders(t) {
+ t.Run(order.name, func(t *testing.T) {
+ s, sqid, home, _ := replicaSetup(t)
+ eve := plantShadow(t, s, order.id, home.Host, []string{"uk"})
+ evil := Record{Schema: 1, Host: "r4444444444.go.tito.io", Slug: "r4444444444", App: "tito",
+ Account: eve, Pool: "tito", Region: "uk", Channel: "stable", CreatedAt: time.Now(),
+ Role: "replica", Home: home.Host}
+ if err := CreateRecord(ctx, s, evil); err != nil {
+ t.Fatal(err)
+ }
+
+ fs := &failStore{Store: s}
+ fs.failReadsFor(RecordKey(sqid, "tito", home.Host))
+ uk := regWith(t)
+ cfg := ReconcileConfig{Region: "uk", Pools: []string{"tito"}, DataDir: "/data"}
+ if err := Reconcile(ctx, fs, uk, cfg); err != nil {
+ t.Fatal(err)
+ }
+ if rt, err := uk.Get(evil.Host); err != nil {
+ t.Fatal(err)
+ } else if rt != nil {
+ t.Errorf("BLIP MINT: a replica vouched for by a shadow was minted while the victim's "+
+ "own record was unreadable: %+v", rt)
+ }
+ // And the healthy pass after it, where both records decode and
+ // the host is plainly contested.
+ if err := Reconcile(ctx, s, uk, cfg); err != nil {
+ t.Fatal(err)
+ }
+ if rt, err := uk.Get(evil.Host); err != nil {
+ t.Fatal(err)
+ } else if rt != nil {
+ t.Errorf("the attacker's replica is still minted after a healthy pass: %+v", rt)
+ }
+ })
+ }
+}
+
+// Nothing in this pass clears a replica link. Every rail now leads to
+// protection, so SetReplicaOf(host, "") — the one write that could remove a
+// link — has no caller here at all. This is the end-to-end statement of that:
+// break each rail in turn, on a LIVE row, and the link survives every one.
+//
+// adf29ebd cleared the link for three of them, on the argument that no door
+// can produce those states, so whatever produced them was outside the doors.
+// It is the argument that is wrong — "outside the doors" includes a writer
+// attacking this account, and handing that writer a revocation hands them the
+// outage.
+func TestReconcileNeverClearsALiveReplicaLink(t *testing.T) {
+ for _, tc := range []struct {
+ name string
+ plant func(t *testing.T, s *release.DirStore, sqid string, home Record)
+ }{
+ {
+ // A chain: the home turns into a replica underneath its own
+ // replica. No door makes one (a record carrying both `role` and
+ // `replicas` is refused by ValidateShape), so this is the shape a
+ // hand edit leaves behind — and adf29ebd cleared the link for it.
+ name: "its home became a replica itself",
+ plant: func(t *testing.T, s *release.DirStore, sqid string, home Record) {
+ home.Role, home.Home = "replica", "somewhere.go.tito.io"
+ home.Replicas, home.Steering = nil, ""
+ putReplace(t, s, RecordKey(sqid, "tito", home.Host), mustJSON(t, home))
+ },
+ },
+ {
+ // The home's host is contested. adf29ebd cleared the link for
+ // this one too, whenever the other claimant sorted first.
+ name: "its home host is contested by a shadow",
+ plant: func(t *testing.T, s *release.DirStore, sqid string, home Record) {
+ plantShadow(t, s, earlierSortingID(t, sqid), home.Host, []string{"uk"})
+ },
+ },
+ {
+ // The home record is gone entirely: a mirror of nothing, which
+ // has always been protected.
+ name: "its home record is gone",
+ plant: func(t *testing.T, s *release.DirStore, sqid string, home Record) {
+ if err := DeleteRecord(ctx, s, sqid, "tito", home.Host); err != nil {
+ t.Fatal(err)
+ }
+ },
+ },
+ {
+ // An old console round-trips a home record and strips `replicas`
+ // it has never heard of (spec §6). A console deploy must not be
+ // able to take a live placement down.
+ name: "its home withdrew the placement",
+ plant: func(t *testing.T, s *release.DirStore, sqid string, home Record) {
+ home.Replicas = nil
+ putReplace(t, s, RecordKey(sqid, "tito", home.Host), mustJSON(t, home))
+ },
+ },
+ } {
+ t.Run(tc.name, func(t *testing.T) {
+ s, sqid, home, replica := replicaSetup(t)
+ uk := regWith(t)
+ cfg := ReconcileConfig{Region: "uk", Pools: []string{"tito"}, DataDir: "/data"}
+ if err := Reconcile(ctx, s, uk, cfg); err != nil {
+ t.Fatal(err)
+ }
+ if got := mustGet(t, uk, replica.Host).ReplicaOf; got != home.Host {
+ t.Fatalf("fixture: ReplicaOf = %q, want %q", got, home.Host)
+ }
+ tc.plant(t, s, sqid, home)
+ if err := Reconcile(ctx, s, uk, cfg); err != nil {
+ t.Fatal(err)
+ }
+ rt := mustGet(t, uk, replica.Host)
+ if rt.ReplicaOf != home.Host {
+ t.Errorf("ReplicaOf = %q, want %q: the row goes on serving the account's public host "+
+ "either way, and clearing the link only takes the pair down", rt.ReplicaOf, home.Host)
+ }
+ if rt.Channel != "stable" {
+ t.Errorf("channel = %q, want stable: a protected row converges nothing", rt.Channel)
+ }
+ })
+ }
+}
+
+// The rail refuses to VOUCH; it does not refuse to work. With no shadow
+// anywhere the legitimate pair mints, links and converges exactly as it did
+// before any of this — the assertion that stops a fail-safe from quietly
+// becoming "fail".
+func TestReconcilePairsALegitimateReplicaWhenNoShadowExists(t *testing.T) {
+ s, sqid, home, replica := replicaSetup(t)
+ uk := regWith(t)
+ cfg := ReconcileConfig{Region: "uk", Pools: []string{"tito"}, DataDir: "/data"}
+ if err := Reconcile(ctx, s, uk, cfg); err != nil {
+ t.Fatal(err)
+ }
+ rt := mustGet(t, uk, replica.Host)
+ if rt.ReplicaOf != home.Host {
+ t.Errorf("ReplicaOf = %q, want %q", rt.ReplicaOf, home.Host)
+ }
+ if rt.Channel != "stable" {
+ t.Errorf("channel = %q, want the home's stable", rt.Channel)
+ }
+ // And it still FOLLOWS its home: a pairing that mints once and then stops
+ // converging is a pairing that drifts.
+ home.Channel = "canary"
+ putReplace(t, s, RecordKey(sqid, "tito", home.Host), mustJSON(t, home))
+ if err := Reconcile(ctx, s, uk, cfg); err != nil {
+ t.Fatal(err)
+ }
+ if got := mustGet(t, uk, replica.Host).Channel; got != "canary" {
+ t.Errorf("channel = %q, want canary: the replica must follow its home", got)
+ }
+}