amadan
The replicator's fleet stops sending when ownership changes, holds every day in doubt, and takes one strict inventory of files it has checked
Built on the request gate and the registry's ownership generation (second review's findings, DISPOSITIONS-metered-replicator.md, third build): - every replica client is gated on the ownership witness, so the first request after a route changes hands is refused before it is sent, and the replicator's own writes to the registry no longer make it stale; - a stale fleet tells every recorder the days in doubt, a watch notices without a request, and the fleet that replaces it inherits the hold; - one strict walk of each tenant tree: what it will not replicate must be named, and where it cannot look refuses the store; - a database is owned only as a checked file: one name, ordinary siblings, no link or mount on the way, the tree owner's, under a directory the platform owns; the fleet looks again on a timer; - shadow state is made private before litestream can make it. Tested through the clients litestream writes through and with the store opened over real databases. What a look at one instant cannot establish is stated where it is checked. Nothing calls this and nothing attests.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
11 files changed,
+2132
−543
docs/superpowers/specs/DISPOSITIONS-metered-replicator.md+30 −0internal/lease/meteredstore.go+20 −14internal/lease/meteredstore_test.go+13 −1internal/lease/requestgate_test.go+1 −0internal/ops/filefacts_other.go+16 −0internal/ops/filefacts_unix.go+27 −0internal/ops/litestream.go+45 −7internal/ops/litestream_metered.go+436 −134internal/ops/litestream_metered_test.go+886 −188internal/ops/litestream_owners.go+325 −91internal/ops/litestream_owners_test.go+333 −108
CI passed — run details
diff --git a/docs/superpowers/specs/DISPOSITIONS-metered-replicator.md b/docs/superpowers/specs/DISPOSITIONS-metered-replicator.md| index 8531ec7fa..514ada5a3 100644 |
| --- a/docs/superpowers/specs/DISPOSITIONS-metered-replicator.md |
| +++ b/docs/superpowers/specs/DISPOSITIONS-metered-replicator.md |
| @@ -44,3 +44,33 @@ Both rejected it again, and the main reason is a fault in the fix made after rev |
| | Opus P3: one mutex and one SQLite read on every request; `Close`'s comment sat on `Open` | The comment is fixed. The rest is open. | |
| What this means for `OUTSTANDING.md` item 1: path B is partial, unwired, and rejected twice. The pieces that hold are the ownership-by-file rule's first half, the unclaimed-file and unreadable-directory refusals, the platform identities and the private store. The ownership fence does not hold as built and needs two changes outside this package (a request gate in `internal/lease`, a routes-only version in the registry) before a third review is worth running. |
| + |
| +## Third build (after review two; not yet read by a reviewer) |
| + |
| +Three commits: a request gate in `internal/lease`, an ownership generation in the registry, and the fleet rebuilt on both with one strict inventory. Nothing calls this code and nothing attests. What follows is what was built against each of review two's findings, and what was not. |
| + |
| +| Review two | Now | Where | Reproduction | |
| +| --- | --- | --- | --- | |
| +| Both, P1: a stale-owner refusal does not stop the request | Fixed. Permission to send is its own middleware (`lease.RequestGate`), outside the request meter in the same step whichever option is applied first. A refusal returns `ErrRequestRefused` before dispatch: nothing reaches the network or the meter, the SDK does not retry it, and litestream sees a failed upload. Every metered database must name a gate. The fleet's gate asks the ownership witness on every attempt. | `lease/requestgate.go`, `lease/meteredstore.go`, `ops/litestream_metered.go` `admit` | `TestRequestGateStopsAnAttemptBeforeItIsSentOrCounted` (both option orders), `TestRequestGateIsAskedAgainForARetry`, `TestMeteredStoreClientIsStoppedByItsGate`, `TestMeteredFleetStopsSendingWhenOwnershipChanges` | |
| +| Opus P1, Astra P2: the witness is moved by the replicator's own writes and by every unrelated commit | Fixed. The registry keeps an ownership generation, a counter moved by triggers when a route is added, removed, or changes host, account, app, database path, backing or hibernation to a different value. `RouteObservation.OwnershipUnchanged` reads it on the pinned connection. A change made and undone moves it twice. | `registry/registry.go` `migrateRouteOwnership`, `registry/observe.go` | `TestOwnershipWitnessMovesOnlyWhenOwnershipChanges` (ten writes that move it, eight that do not, litestream's bookkeeping among them), `TestMeteredFleetOpenedReplicatesCountsAndStops` (litestream really writing its tables into the registry it replicates) | |
| +| Astra P2: replacing the registry file leaves the witness unchanged | Fixed for a replacement that stands. The observation records which file the path named and answers changed when it names another. Discovery and attribution now work from the witnessed rows, not a second read. A path swapped away and back between two looks is not told apart. | `registry/observe.go` | `TestOwnershipWitnessSeesTheRegistryReplaced` | |
| +| Astra P1: requests crossing a change stay with the old owner and nobody else is told; invalidation waits for another request | Changed, not closed. A request sent before the change is the old owner's by rule. The ones in doubt were admitted before the change and sent after it: when the fleet goes stale it tells every recorder it was given that each day from a minute before the last proof to now is incomplete, a watch asks the witness every second whether or not anyone makes a request, and the fleet built to replace it tells its own recorders the same days, which reaches a new owner. Still open: a process that dies between the change and the rebuild takes the hold with it. The started rows it leaves are held by the ledger; the new owner's recorder is told nothing. | `ops/litestream_metered.go` `goStale`, `holdDays`, `watch`, `Replaces` | `TestMeteredFleetWatchHoldsEveryRecorderWithoutARequest`, `TestStaleFleetHoldsEveryDayInDoubt`, `TestMeteredFleetReplacementInheritsTheHold` | |
| +| Both, P1: `Close` left the store running | Fixed in the previous commit; now shown on an opened store. Closing a current fleet lets its last sync through the gate and holds nobody's day. Closing a stale one tells litestream to try its last sync once, not for its whole retry window. A replacement does not open while the fleet it replaces is open. | `ops/litestream_metered.go` `Close`, `Open` | `TestMeteredFleetCloseStopsTheStore`, `TestMeteredFleetOpenedReplicatesCountsAndStops` | |
| +| Opus P1, Astra P1: the `-wal`, `-shm` and `-journal` siblings, hard links, and the claim root were unchecked | Fixed as a check at one instant, not as a guard. A database is owned only if it is a regular file with one name; its siblings are absent or regular files with one name; nothing from the claim root down is a link or on another filesystem; beneath a tenant's tree it belongs to the user the tree belongs to; the directory above a tenant's tree (as spelled and as resolved) and the directory of a platform database are root's or this process's and writable by nobody else. The fleet looks again every thirty seconds and goes stale, with the hold, if a database is no longer the file it was or no longer passes. Still open, completely: a tenant who can write the directory can swap a file, a sibling or a directory after a look and before litestream's next open by path, or between two looks. Closing it needs litestream to open beneath a pinned root without following links, a litestream patch that is not made. | `ops/litestream_owners.go` `checkOwnedFile`, `platformOwned`; `ops/litestream_metered.go` `lookAgain` | `TestFleetDBOwnersCheckTheFileAndItsSiblings` (seven facts, each made false), `TestFleetDBOwnersRefuseANamedFileThatIsALink`, `TestMeteredFleetLooksAtTheFilesAgain` | |
| +| Astra P1: shadow state under the data directory is not the platform's unless something makes it so | Fixed. Every shadow directory is made before litestream can make it, owned by this process and closed to everyone else, under a root that is checked; a data directory, root or component that is a link, somebody else's or open to others refuses the store. What litestream creates beneath a key's directory may carry the tenant's owner and cannot be reached through a directory the tenant cannot enter. | `ops/litestream_metered.go` `prepareShadowState` | `TestMeteredFleetKeepsShadowStateOutOfTenantsReach`, and the permissions checked again after litestream has initialised in `TestMeteredFleetOpenedReplicatesCountsAndStops` | |
| +| Astra P2: every single-file unit `DBPath` is refused | Fixed. A unit route that names a file is checked as a file; only a directory is walked. | `ops/litestream.go` `fleetDBsFromRoutes` | `TestFleetDBOwnersNamesAnOwnerOrSaysThereIsNone` (the `single` route) | |
| +| Both, P2: databases drop out unseen | Changed. One strict walk of each tenant tree returns what it found, what it will not replicate, and where it could not look, together. A database in a dot-directory, deeper than discovery goes, under another SQLite name, or a file with no extension that begins with SQLite's header is listed and must be named by the caller; a link that may lead to a database, a directory that cannot be read, and a tree deeper than thirty-two levels refuse the store. The fleet's second look reports a database that appears or goes as `ErrInventoryChanged`: it goes on streaming what it has and must be rebuilt. Still open: an empty `.db` is passed over until it is not empty; nothing compares against the external replicator's own list. | `ops/litestream_owners.go` `strictTreeDBs`, `fleetInventory`; `ops/litestream_metered.go` `lookAgain` | `TestStrictTreeWalkReportsWhatDiscoveryPassesOver`, `TestMeteredFleetStoreRefusesAnUnclaimedDatabaseNobodyNamed`, `TestMeteredFleetLooksAtTheFilesAgain` | |
| +| Astra P2, Opus P3-1: the registry's real path is discarded before it is classified | Fixed for the platform's replicator: discovery is told where the registry is open and lists it there. The config generator's behaviour is unchanged on purpose. | `ops/litestream.go` `fleetDBsFromRoutes` | `TestFleetDBOwnersFindTheRegistryWhereItIsOpen` | |
| +| Opus P2: any tenant can refuse the box's replicator; the check refused at random | Changed. The walk uses `Lstat`, ignores an entry that vanishes while it is looking, and ignores a link that is neither named like a database nor leads to a directory. A refusal names the tenant's host. Still open, and a decision: a tenant can still stop a rebuild for the whole box by making a link that may lead to a database, or a database discovery does not take, until an operator acts. A running fleet is not stopped by either. | `ops/litestream_owners.go` `strictTreeDBs` | `TestFleetDBOwnersDoesNotFollowATenantsLink` | |
| +| Both: the ownership test bypassed the client it claimed to prove | Fixed. The tests write through the clients the fleet's store gave its databases, against a server that counts, and one opens the store and lets litestream replicate real databases. The console's identity and a relocated registry have fixtures. | `ops/litestream_metered_test.go` | the tests above | |
| +| Opus P3-2: one mutex and one SQLite read inside the meter's 25 ms budget | Changed. The read is on the gate, before the meter and outside its budget. It is still one read per attempt behind one lock. | `ops/litestream_metered.go` `admit` | none | |
| +| Opus P2-1: a `Stat` failure in `NewMeteredStore` skipped the identity check | Fixed: a database file that cannot be looked at is refused. | `lease/meteredstore.go` | `TestMeteredStoreRefusesWhatItCannotAttribute` | |
| +| Opus P2-2 (review one): nothing enforces that the replica prefix is not the external binary's lineage | Open. | | | |
| +| Opus P2-4 (review one): no shrink guard against the external replicator's list | Open. | | | |
| + |
| +Found on the way: `registry.ObserveRoutes` panicked on any failure after its connection opened (its cleanup closed a result the failing return had cleared). Fixed here, and on the launch branch's own small branch `fix/carloku-observe-routes-cleanup-20261001`, which this branch will conflict with trivially. |
| + |
| +Mutation checks in a scratch copy, twenty-five. Twenty-two failed their tests: the gate letting a refused attempt through; the gate inside the meter; the generation moving on any write of an owner column; hibernation dropped from the columns; the witness ignoring a replaced file; the fleet's clients given a gate that always admits; the fleet asking the whole-database witness; a stale fleet telling one recorder; the watch not asking; siblings unchecked; a second name unchecked; the directory above a tenant unchecked; shadow directories unverified; shadow directories left to litestream; the second look not comparing the file; a replacement not inheriting the hold; discovery not told where the registry is; a replacement opening beside the fleet it replaces; the hold not reaching back before the last proof; the walk ignoring every link; a stale close retrying; the walk passing over a hidden database. Three survived, and are stated as untested: a link at the shadow root (it is refused by the mode check instead, so the explicit check is redundant on Linux); the owner check beneath a tenant's tree and the filesystem check (a test needs root, or a second filesystem). |
| + |
| +Not built, and still needed before this could replace the external binary: the replica credentials out of everything else's reach; a restore path; the prefix-lineage guard; the shrink guard; opening databases beneath a pinned root; a caller that rebuilds the fleet; the registry migration reviewed as a schema change on every box (it adds one table and three triggers, and an older build writing to a migrated registry still moves the counter). |
| + |
diff --git a/internal/lease/meteredstore.go b/internal/lease/meteredstore.go| index 6533ecb74..9934ecc97 100644 |
| --- a/internal/lease/meteredstore.go |
| +++ b/internal/lease/meteredstore.go |
| @@ -35,8 +35,9 @@ import ( |
| // meter sends when it cannot record (and holds the day), so stopping a |
| // request is the gate's job and not the meter's. |
| // |
| -// What it is not, yet. Nothing calls it. It does not replace the external |
| -// binary, change a unit, or attest anything. Replacing the binary also |
| +// What it is not, yet. Nothing calls it but the fleet in internal/ops, |
| +// which nothing calls. It does not replace the external binary, change a |
| +// unit, or attest anything. Replacing the binary also |
| // needs the replica credentials to stop being readable by anything else on |
| // the box, a restore path that does not shell out to the external binary, |
| // and a replica prefix the old binary's files do not share. None of that is |
| @@ -71,8 +72,9 @@ type MeteredDB struct { |
| // NewMeteredStore builds one litestream store over dbs, each replica client |
| // metered by its own recorder. The store is returned unopened. It refuses |
| -// an empty list, a database with no path, prefix, meter or gate, and two |
| -// databases that share a file or a replica prefix (two owners must never |
| +// an empty list, a database with no path, prefix, meter or gate, a |
| +// database file that is not there to be looked at, and two databases that |
| +// share a file or a replica prefix (two owners must never |
| // be billed for one stream, nor one stream written twice). |
| func NewMeteredStore(bucket, region string, dbs []MeteredDB) (*litestream.Store, error) { |
| if bucket == "" { |
| @@ -85,16 +87,6 @@ func NewMeteredStore(bucket, region string, dbs []MeteredDB) (*litestream.Store, |
| var files []os.FileInfo |
| built := make([]*litestream.DB, 0, len(dbs)) |
| for _, d := range dbs { |
| - // Two entries that are one file, however their paths are spelled |
| - // (a link, a second mount point): compared as files, not as text. |
| - if fi, serr := os.Stat(d.Path); serr == nil { |
| - for _, other := range files { |
| - if os.SameFile(fi, other) { |
| - return nil, fmt.Errorf("lease: metered store: %s is a file already listed under another path", d.Path) |
| - } |
| - } |
| - files = append(files, fi) |
| - } |
| switch { |
| case d.Path == "" || !filepath.IsAbs(d.Path): |
| return nil, fmt.Errorf("lease: metered store: database path %q is not absolute", d.Path) |
| @@ -113,6 +105,20 @@ func NewMeteredStore(bucket, region string, dbs []MeteredDB) (*litestream.Store, |
| case prefixes[d.Prefix]: |
| return nil, fmt.Errorf("lease: metered store: replica prefix %q is used by two databases", d.Prefix) |
| } |
| + // Two entries that are one file, however their paths are spelled |
| + // (a link, a second mount point): compared as files, not as text. |
| + // A file that cannot be looked at cannot be compared, and one |
| + // that appears later would never be: it is refused. |
| + fi, serr := os.Stat(d.Path) |
| + if serr != nil { |
| + return nil, fmt.Errorf("lease: metered store: %s cannot be looked at: %w", d.Path, serr) |
| + } |
| + for _, other := range files { |
| + if os.SameFile(fi, other) { |
| + return nil, fmt.Errorf("lease: metered store: %s is a file already listed under another path", d.Path) |
| + } |
| + } |
| + files = append(files, fi) |
| paths[filepath.Clean(d.Path)], prefixes[d.Prefix], metas[filepath.Clean(d.MetaDir)] = true, true, true |
| option, err := requestMeterOption(d.Meter) |
| if err != nil { |
diff --git a/internal/lease/meteredstore_test.go b/internal/lease/meteredstore_test.go| index a78abe152..ed7456d17 100644 |
| --- a/internal/lease/meteredstore_test.go |
| +++ b/internal/lease/meteredstore_test.go |
| @@ -18,6 +18,7 @@ func TestMeteredStoreCountsEveryRequestOnItsOwnersMeter(t *testing.T) { |
| fakeCreds(t) |
| t.Setenv("CARLOS_S3_ENDPOINT", endpoint) |
| dir := t.TempDir() |
| + touchDBs(t, dir, "one.db", "two.db") |
| one, two := &memoryRequestRecorder{}, &memoryRequestRecorder{} |
| store, err := NewMeteredStore("test-bucket", "eu-west-1", []MeteredDB{ |
| {Path: filepath.Join(dir, "one.db"), Prefix: "replicas/bab/one", Meter: one, Gate: admitEveryRequest, MetaDir: filepath.Join(dir, ".replicator", "one")}, |
| @@ -71,6 +72,7 @@ func TestMeteredStoreRefusesWhatItCannotAttribute(t *testing.T) { |
| if err := os.WriteFile(ok.Path, []byte("x"), 0o644); err != nil { |
| t.Fatal(err) |
| } |
| + touchDBs(t, dir, "b.db") |
| link := filepath.Join(dir, "link.db") |
| if err := os.Symlink(ok.Path, link); err != nil { |
| t.Fatal(err) |
| @@ -84,7 +86,8 @@ func TestMeteredStoreRefusesWhatItCannotAttribute(t *testing.T) { |
| "no metadata directory": {{Path: ok.Path, Prefix: ok.Prefix, Meter: m, Gate: admitEveryRequest}}, |
| "a relative metadata directory": {{Path: ok.Path, Prefix: ok.Prefix, Meter: m, Gate: admitEveryRequest, MetaDir: ".replicator/a"}}, |
| "one metadata directory for two": {ok, {Path: filepath.Join(dir, "b.db"), Prefix: "replicas/b", Meter: m, Gate: admitEveryRequest, MetaDir: metaA}}, |
| - "one file listed twice": {ok, {Path: filepath.Join(dir, ".", "a.db"), Prefix: "replicas/b", Meter: m, Gate: admitEveryRequest, MetaDir: metaB}}, |
| + "one file listed twice": {ok, {Path: dir + "/./a.db", Prefix: "replicas/b", Meter: m, Gate: admitEveryRequest, MetaDir: metaB}}, |
| + "a file that is not there": {{Path: filepath.Join(dir, "absent.db"), Prefix: "replicas/c", Meter: m, Gate: admitEveryRequest, MetaDir: metaB}}, |
| "one file under a link": {ok, {Path: link, Prefix: "replicas/b", Meter: m, Gate: admitEveryRequest, MetaDir: metaB}}, |
| "one prefix for two": {ok, {Path: filepath.Join(dir, "b.db"), Prefix: ok.Prefix, Meter: m, Gate: admitEveryRequest, MetaDir: metaB}}, |
| "a meter missing in many": {ok, {Path: filepath.Join(dir, "b.db"), Prefix: "replicas/b", Gate: admitEveryRequest, MetaDir: metaB}}, |
| @@ -106,3 +109,12 @@ func TestMeteredStoreRefusesWhatItCannotAttribute(t *testing.T) { |
| t.Errorf("a well-formed database: %v", err) |
| } |
| } |
| + |
| +func touchDBs(t *testing.T, dir string, names ...string) { |
| + t.Helper() |
| + for _, n := range names { |
| + if err := os.WriteFile(filepath.Join(dir, n), []byte("x"), 0o644); err != nil { |
| + t.Fatal(err) |
| + } |
| + } |
| +} |
diff --git a/internal/lease/requestgate_test.go b/internal/lease/requestgate_test.go| index c358ee62b..b5b758d94 100644 |
| --- a/internal/lease/requestgate_test.go |
| +++ b/internal/lease/requestgate_test.go |
| @@ -185,6 +185,7 @@ func TestMeteredStoreClientIsStoppedByItsGate(t *testing.T) { |
| fakeCreds(t) |
| t.Setenv("CARLOS_S3_ENDPOINT", endpoint) |
| dir := t.TempDir() |
| + touchDBs(t, dir, "one.db", "two.db") |
| one, two := &memoryRequestRecorder{}, &memoryRequestRecorder{} |
| var closed atomic.Bool |
| store, err := NewMeteredStore("test-bucket", "eu-west-1", []MeteredDB{ |
diff --git a/internal/ops/filefacts_other.go b/internal/ops/filefacts_other.go| new file mode 100644 |
| index 000000000..fa268986d |
| --- /dev/null |
| +++ b/internal/ops/filefacts_other.go |
| @@ -0,0 +1,16 @@ |
| +//go:build !unix |
| + |
| +package ops |
| + |
| +import "io/fs" |
| + |
| +// fileFacts: see filefacts_unix.go. Where a file's identity, link count and |
| +// owner cannot be read, nothing can be shown to be owned, and the |
| +// replicator refuses every database. |
| +type fileFacts struct { |
| + Dev, Ino uint64 |
| + Nlink uint64 |
| + UID uint32 |
| +} |
| + |
| +func factsOf(fs.FileInfo) (fileFacts, bool) { return fileFacts{}, false } |
diff --git a/internal/ops/filefacts_unix.go b/internal/ops/filefacts_unix.go| new file mode 100644 |
| index 000000000..e09b27bf9 |
| --- /dev/null |
| +++ b/internal/ops/filefacts_unix.go |
| @@ -0,0 +1,27 @@ |
| +//go:build unix |
| + |
| +package ops |
| + |
| +import ( |
| + "io/fs" |
| + "syscall" |
| +) |
| + |
| +// fileFacts is what the replicator's ownership checks need to know about a |
| +// file beyond its mode: which file it is (device and inode), how many names |
| +// it has, and whose it is. Split by build tag because internal/ops is linked |
| +// into the CLI, which is built for platforms where syscall.Stat_t does not |
| +// exist; the replicator itself only ever runs on a box. |
| +type fileFacts struct { |
| + Dev, Ino uint64 |
| + Nlink uint64 |
| + UID uint32 |
| +} |
| + |
| +func factsOf(fi fs.FileInfo) (fileFacts, bool) { |
| + st, ok := fi.Sys().(*syscall.Stat_t) |
| + if !ok { |
| + return fileFacts{}, false |
| + } |
| + return fileFacts{Dev: uint64(st.Dev), Ino: uint64(st.Ino), Nlink: uint64(st.Nlink), UID: st.Uid}, true |
| +} |
diff --git a/internal/ops/litestream.go b/internal/ops/litestream.go| index 21396ef01..5fa1dc8a2 100644 |
| --- a/internal/ops/litestream.go |
| +++ b/internal/ops/litestream.go |
| @@ -333,8 +333,26 @@ func FleetDBsWithSkipped(reg *registry.Registry, dataDir string) (dbs []string, |
| if err != nil { |
| return nil, nil, err |
| } |
| + dbs, skipped, _, err = fleetDBsFromRoutes(routes, "", dataDir, nil) |
| + return dbs, skipped, err |
| +} |
| + |
| +// fleetDBsFromRoutes is FleetDBsWithSkipped over routes already read, so a |
| +// caller that read them under a witness discovers from the same rows. |
| +// |
| +// registryPath, when set, is the file the registry is open at: it is the |
| +// box's own database wherever it lives, claimed and listed like the |
| +// conventional <dataDir>/registry.db. The config generator passes none and |
| +// keeps its behaviour. |
| +// |
| +// tree walks a unit tenant's directory. nil is treeDBs, which passes over |
| +// what it cannot read or recognise, as a config generator should. The |
| +// platform's own replicator passes a walk that reports those instead |
| +// (strictTreeDBs): what it adds to skipped and to problems comes back to |
| +// the caller, and a tree walk's problems carry the claiming host. |
| +func fleetDBsFromRoutes(routes []registry.Route, registryPath, dataDir string, tree func(root string) (dbs, skipped, problems []string)) (dbs []string, skipped []string, problems []string, err error) { |
| hibernating := map[string]bool{} |
| - unitPaths := map[string]bool{} |
| + unitPaths := map[string]string{} // claimed path -> the first host that claims it |
| for _, r := range routes { |
| if r.Backing == registry.BackingUnit { |
| // Unit tenants' data lives outside dataDir (their systemd |
| @@ -342,7 +360,9 @@ func FleetDBsWithSkipped(reg *registry.Registry, dataDir string) (dbs []string, |
| // own — DBPath, when the operator set it, is the only way this |
| // replicator learns where they are. |
| if r.DBPath != "" { |
| - unitPaths[filepath.Clean(r.DBPath)] = true |
| + if _, seen := unitPaths[filepath.Clean(r.DBPath)]; !seen { |
| + unitPaths[filepath.Clean(r.DBPath)] = r.Host |
| + } |
| } |
| continue |
| } |
| @@ -353,7 +373,7 @@ func FleetDBsWithSkipped(reg *registry.Registry, dataDir string) (dbs []string, |
| matches, err := filepath.Glob(filepath.Join(dataDir, "*.db")) |
| if err != nil { |
| - return nil, nil, err |
| + return nil, nil, nil, err |
| } |
| // A nested row's DB lives one level down, under |
| // <dataDir>/instances/<id>/<host>.db (isolated-exec design §7): the flat |
| @@ -363,7 +383,7 @@ func FleetDBsWithSkipped(reg *registry.Registry, dataDir string) (dbs []string, |
| // claimed by the very same loop that adds a flat row's. |
| nested, err := filepath.Glob(filepath.Join(dataDir, "instances", "*", "*.db")) |
| if err != nil { |
| - return nil, nil, err |
| + return nil, nil, nil, err |
| } |
| matches = append(matches, nested...) |
| // THE GLOB PROPOSES, THE REGISTRY DISPOSES. A *.db under dataDir is |
| @@ -383,6 +403,9 @@ func FleetDBsWithSkipped(reg *registry.Registry, dataDir string) (dbs []string, |
| filepath.Clean(filepath.Join(dataDir, "registry.db")): true, |
| filepath.Clean(consoleDBPath(dataDir)): true, |
| } |
| + if registryPath != "" { |
| + claimed[filepath.Clean(registryPath)] = true |
| + } |
| for _, r := range routes { |
| if r.DBPath != "" { |
| claimed[filepath.Clean(r.DBPath)] = true |
| @@ -402,13 +425,27 @@ func FleetDBsWithSkipped(reg *registry.Registry, dataDir string) (dbs []string, |
| // entirely outside dataDir, at their StateDirectory path. A DBPath naming |
| // a DIRECTORY is a tree — see treeDBs. These are claimed by construction: |
| // a route named them. |
| - for p := range unitPaths { |
| + for p, host := range unitPaths { |
| if fi, err := os.Stat(p); err == nil && fi.IsDir() { |
| - matches = append(matches, treeDBs(p)...) |
| + if tree == nil { |
| + matches = append(matches, treeDBs(p)...) |
| + continue |
| + } |
| + found, passed, bad := tree(p) |
| + matches = append(matches, found...) |
| + skipped = append(skipped, passed...) |
| + for _, b := range bad { |
| + problems = append(problems, b+" (in the state directory of "+host+")") |
| + } |
| continue |
| } |
| matches = append(matches, p) |
| } |
| + // The registry at the path it is open at, when that is not the |
| + // conventional one: it is the box's own wherever it lives. |
| + if registryPath != "" { |
| + matches = append(matches, registryPath) |
| + } |
| // And the console's own database, for the same reason in miniature: it |
| // sits at <data>/console/console.db — ONE DIRECTORY BELOW the flat glob, |
| // which is exactly why it went unreplicated. It holds no platform state |
| @@ -435,7 +472,8 @@ func FleetDBsWithSkipped(reg *registry.Registry, dataDir string) (dbs []string, |
| dbs = append(dbs, p) |
| } |
| sort.Strings(skipped) |
| - return dbs, skipped, nil |
| + sort.Strings(problems) |
| + return dbs, skipped, problems, nil |
| } |
| // restoreAndCheckFn is the per-database restore behind a package var, so the |
diff --git a/internal/ops/litestream_metered.go b/internal/ops/litestream_metered.go| index aabd216b1..c6afca93a 100644 |
| --- a/internal/ops/litestream_metered.go |
| +++ b/internal/ops/litestream_metered.go |
| @@ -4,9 +4,12 @@ import ( |
| "context" |
| "errors" |
| "fmt" |
| + "os" |
| "path/filepath" |
| + "sort" |
| "strings" |
| "sync" |
| + "sync/atomic" |
| "time" |
| "github.com/benbjohnson/litestream" |
| @@ -22,64 +25,226 @@ import ( |
| // through its ledger. |
| var ErrUnmeterable = errors.New("ops: a replicated database has no owner to meter it against") |
| -// ErrOwnershipStale: the registry has changed since this replicator read |
| -// who owns each database. A route may have moved to another account, been |
| -// deleted, been made again, or started hibernating; the meters this store |
| -// was built with may now be the wrong ones. Every replica request is |
| -// refused from then on, and the store has to be built again. |
| -var ErrOwnershipStale = errors.New("ops: the registry changed since ownership was read; the replicator must be rebuilt") |
| +// ErrOwnershipStale: who owns the databases this replicator streams can no |
| +// longer be vouched for. The registry's ownership generation has moved (a |
| +// route was added or removed, or changed account, app, database path, |
| +// backing or hibernation), or the registry can no longer be read, or a |
| +// database is no longer the file it was when its ownership was checked. |
| +// Every replica request is refused from then on, every recorder's day is |
| +// held, and the store has to be closed and built again. A closed fleet |
| +// answers the same. |
| +var ErrOwnershipStale = errors.New("ops: ownership of the replicated databases can no longer be vouched for; the replicator must be rebuilt") |
| + |
| +// ErrInventoryChanged: the databases on the box are no longer the ones |
| +// this replicator was built over (a tenant made a new one, or one has |
| +// gone). Nothing it streams is in doubt, so it goes on streaming and |
| +// counting; it has to be rebuilt to cover what is new. |
| +var ErrInventoryChanged = errors.New("ops: the box's databases have changed since the replicator was built; rebuild it to cover them") |
| + |
| +// How often an opened fleet asks the registry's ownership witness without |
| +// waiting for a request to ask, and how often it looks at the files again. |
| +// Variables so a test can shorten them. |
| +var ( |
| + ownershipWatchInterval = time.Second |
| + inventoryWatchInterval = 30 * time.Second |
| +) |
| + |
| +// ownershipMargin is how far before the last proven instant a stale |
| +// fleet's hold reaches. A request is admitted by the gate and then |
| +// recorded and sent; the recording has a budget of milliseconds. The |
| +// margin is far wider than that, so the hold covers every request that |
| +// was admitted before a change and sent after it. |
| +const ownershipMargin = time.Minute |
| + |
| +// MeteredFleetOptions says what the platform's replicator is built over. |
| +type MeteredFleetOptions struct { |
| + Registry *registry.Registry |
| + DataDir string |
| + // Bucket, Prefix and Region are where the replicas go. Prefix is the |
| + // replica lineage this replicator writes. It must not be one the |
| + // external binary has written to: the two do not share a format or a |
| + // metadata directory, and a restore from a prefix both have touched is |
| + // not something anyone has tested. Nothing here enforces that. |
| + Bucket, Prefix, Region string |
| + // Accept names the files the inventory skips (FleetInventory.Skipped) |
| + // that the caller has seen and means to leave unreplicated. A skipped |
| + // file that is not named refuses the store: a database the external |
| + // replicator still streams, whose route has gone, must not drop out of |
| + // replication because nobody looked. |
| + Accept []string |
| + // Meter returns the recorder for one database, bound by the caller to |
| + // the box's durability ledger. It is asked once for each database. A |
| + // nil recorder refuses the whole store: there is no unmetered |
| + // fallback. |
| + // |
| + // The recorders should all write to one ledger whose incompleteness is |
| + // the box's (the durability ledger's is). The fleet does not rely on |
| + // it: when ownership goes stale it tells every recorder it was given. |
| + Meter func(OwnedDB) lease.RequestRecorder |
| + // Replaces is the fleet this one is built to take over from, if any. |
| + // If that fleet went stale, every recorder of this one is told the |
| + // same days are incomplete: the new owner of a database that changed |
| + // hands is one of them, and the old fleet never had its recorder. The |
| + // old fleet must be closed before this one is opened. |
| + Replaces *MeteredFleet |
| + // SyncInterval is litestream's replica sync tick. Zero keeps its |
| + // default. |
| + SyncInterval time.Duration |
| +} |
| // MeteredFleet is the platform's replicator for one box, with the witness |
| -// its ownership was read under. Close releases the witness. |
| +// its ownership was read under. The caller must Close it. |
| type MeteredFleet struct { |
| // Owned is every database the store replicates, with its owner. |
| Owned []OwnedDB |
| - // Skipped lists the files under the data directory that look like |
| - // databases and that no route claims. They are not replicated. The |
| - // caller named each of them (MeteredFleetStore's accept), so none was |
| - // left out unseen. |
| + // Skipped lists the files that look like databases and are not |
| + // replicated. The caller named each of them (MeteredFleetOptions.Accept), |
| + // so none was left out unseen. |
| Skipped []string |
| + routes []registry.Route |
| + registryPath string |
| + dataDir string |
| + facts map[string]fileFacts // by owned path: the file each was when it was checked |
| + inventory string // the inventory this fleet was built over, rendered |
| + recorders []lease.RequestRecorder |
| + replaces *MeteredFleet |
| + |
| // store is kept private. A litestream store takes any database through |
| // its own RegisterDB, with an ordinary unmetered client, and its DBs |
| // hand out live clients that can be replaced: the constructor's |
| // refusals would mean nothing to a caller holding it. |
| + // |
| + // mu guards store and obs, and serialises reads of the witness (one |
| + // pinned connection). |
| + mu sync.Mutex |
| store *litestream.Store |
| + obs *registry.RouteObservation |
| - mu sync.Mutex |
| - obs *registry.RouteObservation |
| - stale bool |
| - recorders map[string]lease.RequestRecorder // by database path; what each client was given |
| + stale atomic.Bool |
| + drift atomic.Bool |
| + closed atomic.Bool |
| + proven atomic.Int64 // the last instant ownership was shown unchanged, in Unix nanoseconds |
| + |
| + staleMu sync.Mutex |
| + staleFrom, staleAt time.Time |
| + |
| + watchOnce sync.Once |
| + stopOnce sync.Once |
| + stop chan struct{} |
| + watching sync.WaitGroup |
| } |
| -// Current reports whether ownership is still as it was read: nil, or |
| -// ErrOwnershipStale once any commit has landed on the registry since. It |
| -// asks the registry's own change witness (registry.RouteObservation), which |
| -// sees a change that was made and undone as well as one that stands. Once |
| -// stale, always stale. A witness that cannot be read is not proof of no |
| -// change, and counts as one. |
| +// Current reports whether the fleet may go on as it is: nil, |
| +// ErrInventoryChanged (still streaming; rebuild to cover what is new), or |
| +// ErrOwnershipStale (stopped; close and rebuild). Stale is for good. |
| func (f *MeteredFleet) Current(ctx context.Context) error { |
| - f.mu.Lock() |
| - defer f.mu.Unlock() |
| - if f.stale { |
| + if err := f.admit(ctx); err != nil { |
| + return err |
| + } |
| + if f.drift.Load() { |
| + return ErrInventoryChanged |
| + } |
| + return nil |
| +} |
| + |
| +// admit is the request gate every replica client is given |
| +// (lease.RequestGate): nil when a request may be sent, an error when it |
| +// must not. It asks the registry's ownership witness each time, so the |
| +// first request after a change of ownership is refused and not only the |
| +// first one after somebody noticed. |
| +// |
| +// A request whose own context has ended is refused without a verdict on |
| +// ownership. A witness that cannot be read is not proof of no change, and |
| +// counts as one. |
| +func (f *MeteredFleet) admit(ctx context.Context) error { |
| + if f.closed.Load() || f.stale.Load() { |
| return ErrOwnershipStale |
| } |
| - if f.obs == nil { |
| - f.stale = true |
| + if err := ctx.Err(); err != nil { |
| + return err |
| + } |
| + f.mu.Lock() |
| + defer f.mu.Unlock() |
| + if f.obs == nil || f.closed.Load() { |
| return ErrOwnershipStale |
| } |
| - ctx, cancel := context.WithTimeout(ctx, 5*time.Second) |
| + asked := time.Now() |
| + // Not the request's context: a request cancelled while the witness is |
| + // being read must not be mistaken for a registry that cannot be read. |
| + wctx, cancel := context.WithTimeout(context.Background(), 5*time.Second) |
| defer cancel() |
| - if ok, err := f.obs.Unchanged(ctx); err != nil || !ok { |
| - f.stale = true |
| + if ok, err := f.obs.OwnershipUnchanged(wctx); err != nil || !ok { |
| + f.goStale(time.Now()) |
| return ErrOwnershipStale |
| } |
| + // The instant before the read, not after it: what was shown is that |
| + // nothing had changed by some moment during the read. |
| + f.proven.Store(asked.UnixNano()) |
| return nil |
| } |
| -// Open starts replication. It refuses once ownership is stale. |
| +// goStale stops the fleet for good and holds the days in doubt on every |
| +// recorder it was given. It does not wait for a request: the caller may be |
| +// the watch, with no request in sight. |
| +// |
| +// The days in doubt run from a margin before the last instant ownership |
| +// was shown unchanged, to now. A request sent before the registry changed |
| +// was sent for the owner of that moment and is counted to them. The ones |
| +// in doubt were admitted before the change and sent after it; they were |
| +// admitted no earlier than a moment before the change, which is later than |
| +// the last proof less the margin. |
| +func (f *MeteredFleet) goStale(at time.Time) { |
| + if !f.stale.CompareAndSwap(false, true) { |
| + return |
| + } |
| + from := time.Unix(0, f.proven.Load()).Add(-ownershipMargin) |
| + f.staleMu.Lock() |
| + f.staleFrom, f.staleAt = from, at |
| + f.staleMu.Unlock() |
| + holdDays(f.recorders, from, at) |
| +} |
| + |
| +// staleWindow is the span a stale fleet held, for the fleet that replaces |
| +// it. |
| +func (f *MeteredFleet) staleWindow() (from, at time.Time, stale bool) { |
| + if !f.stale.Load() { |
| + return time.Time{}, time.Time{}, false |
| + } |
| + f.staleMu.Lock() |
| + defer f.staleMu.Unlock() |
| + return f.staleFrom, f.staleAt, true |
| +} |
| + |
| +// holdDays tells every recorder that each UTC day touched by [from, to] is |
| +// incomplete. |
| +func holdDays(recorders []lease.RequestRecorder, from, to time.Time) { |
| + from, to = from.UTC(), to.UTC() |
| + if to.Before(from) { |
| + to = from |
| + } |
| + for at := from; ; { |
| + for _, r := range recorders { |
| + r.Incomplete(at) |
| + } |
| + next := time.Date(at.Year(), at.Month(), at.Day(), 0, 0, 0, 0, time.UTC).AddDate(0, 0, 1) |
| + if next.After(to) { |
| + return |
| + } |
| + at = next |
| + } |
| +} |
| + |
| +// Open starts replication, and the watch that asks the ownership witness |
| +// and looks at the files again whether or not any request is being made. |
| +// It refuses once ownership is stale, and while the fleet this one |
| +// replaces is still open: two replicators over one database share its |
| +// shadow state and its replica prefix. |
| func (f *MeteredFleet) Open(ctx context.Context) error { |
| - if err := f.Current(ctx); err != nil { |
| + if f.replaces != nil && !f.replaces.closed.Load() { |
| + return errors.New("ops: the replicator this one replaces has not been closed") |
| + } |
| + if err := f.admit(ctx); err != nil { |
| return err |
| } |
| f.mu.Lock() |
| @@ -88,7 +253,86 @@ func (f *MeteredFleet) Open(ctx context.Context) error { |
| if store == nil { |
| return ErrOwnershipStale |
| } |
| - return store.Open(ctx) |
| + if err := store.Open(ctx); err != nil { |
| + return err |
| + } |
| + f.watchOnce.Do(func() { |
| + f.watching.Add(1) |
| + go f.watch() |
| + }) |
| + return nil |
| +} |
| + |
| +func (f *MeteredFleet) watch() { |
| + defer f.watching.Done() |
| + owner := time.NewTicker(ownershipWatchInterval) |
| + defer owner.Stop() |
| + files := time.NewTicker(inventoryWatchInterval) |
| + defer files.Stop() |
| + for { |
| + select { |
| + case <-f.stop: |
| + return |
| + case <-owner.C: |
| + _ = f.admit(context.Background()) |
| + case <-files.C: |
| + f.lookAgain() |
| + } |
| + } |
| +} |
| + |
| +// lookAgain takes the box's inventory again, from the routes this fleet was |
| +// built over, and compares it with what it was built over. |
| +// |
| +// A database this fleet streams that is no longer the file it was when its |
| +// ownership was checked (another inode at its path, or a check that now |
| +// fails: a link where a file was, a second name, a sibling replaced) makes |
| +// the fleet stale, with the hold that goes with it: what has been shipped |
| +// from that path since can no longer be said to be its owner's. Any other |
| +// difference (a new database, a new file to be named, one that has gone) |
| +// is ErrInventoryChanged. |
| +// |
| +// This is a second look, not a guard. Between two looks a tenant can swap |
| +// a file and swap it back. See checkOwnedFile. |
| +func (f *MeteredFleet) lookAgain() { |
| + if f.closed.Load() || f.stale.Load() { |
| + return |
| + } |
| + inv, err := fleetInventory(f.routes, f.registryPath, f.dataDir) |
| + if err != nil { |
| + f.drift.Store(true) |
| + return |
| + } |
| + for path, was := range f.facts { |
| + now, still := inv.facts[path] |
| + if still && now.Dev == was.Dev && now.Ino == was.Ino { |
| + continue |
| + } |
| + if _, err := os.Lstat(path); errors.Is(err, os.ErrNotExist) { |
| + // Gone altogether: nothing is being shipped from it. |
| + continue |
| + } |
| + f.goStale(time.Now()) |
| + return |
| + } |
| + if inv.render() != f.inventory { |
| + f.drift.Store(true) |
| + } |
| +} |
| + |
| +// render is an inventory as one comparable string: what is owned and by |
| +// whom, what is unowned, skipped, or could not be looked at. |
| +func (inv FleetInventory) render() string { |
| + var b strings.Builder |
| + for _, o := range inv.Owned { |
| + fmt.Fprintf(&b, "owned %s %s/%s %s\n", o.Path, o.Account, o.App, o.Host) |
| + } |
| + for _, list := range [][]string{inv.Unowned, inv.Skipped, inv.Problems} { |
| + sorted := append([]string(nil), list...) |
| + sort.Strings(sorted) |
| + b.WriteString("--\n" + strings.Join(sorted, "\n") + "\n") |
| + } |
| + return b.String() |
| } |
| // Databases is how many databases the store replicates (none once closed). |
| @@ -101,26 +345,38 @@ func (f *MeteredFleet) Databases() int { |
| return len(f.store.DBs()) |
| } |
| -// Close stops replication and releases the registry witness, in that |
| -// order. The fleet is stale afterwards and cannot be opened again. |
| - |
| +// Close stops replication, then the watch, then releases the registry |
| +// witness, in that order. The fleet answers ErrOwnershipStale afterwards |
| +// and cannot be opened again. Closing a fleet that was current holds |
| +// nobody's day: its last requests are admitted and counted like any |
| +// others, and only when the store has stopped does the gate shut. |
| func (f *MeteredFleet) Close() error { |
| - // The store first, and outside the fleet's lock: its workers ask |
| - // Current on their way out, and Current takes the lock. A fleet whose |
| + // The store first, and outside the fleet's lock: its workers ask the |
| + // gate on their way out, and the gate takes the lock. A fleet whose |
| // witness was released while its store kept running would go on |
| // replicating beside whatever store replaced it, over the same shadow |
| // state. |
| f.mu.Lock() |
| store := f.store |
| f.store = nil |
| - f.stale = true |
| f.mu.Unlock() |
| var first error |
| if store != nil { |
| + // Litestream's last act is to sync each replica, trying again for |
| + // a while if that fails. A stale fleet's requests are refused and |
| + // will go on being refused, so it is told to try once: closing a |
| + // stale fleet is the first step of replacing it, and should not |
| + // take the whole retry window for every database. |
| + if f.stale.Load() { |
| + store.SetShutdownSyncTimeout(0) |
| + } |
| ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second) |
| first = store.Close(ctx) |
| cancel() |
| } |
| + f.closed.Store(true) |
| + f.stopOnce.Do(func() { close(f.stop) }) |
| + f.watching.Wait() |
| f.mu.Lock() |
| defer f.mu.Unlock() |
| if f.obs == nil { |
| @@ -134,147 +390,193 @@ func (f *MeteredFleet) Close() error { |
| return err |
| } |
| -// ownedRecorder is the recorder a replica client is actually given: the |
| -// owner's own, behind a check that the owner is still the owner. A request |
| -// that would be made after the registry changed is not started and is not |
| -// counted to anybody. The owner's ledger is told its day is incomplete, |
| -// because a replica that stops being written is a gap, not a zero. |
| -type ownedRecorder struct { |
| - lease.RequestRecorder |
| - fleet *MeteredFleet |
| -} |
| - |
| -func (r ownedRecorder) Start(operation string, at time.Time) (uint64, error) { |
| - if err := r.fleet.Current(context.Background()); err != nil { |
| - r.RequestRecorder.Incomplete(at) |
| - return 0, err |
| - } |
| - return r.RequestRecorder.Start(operation, at) |
| -} |
| - |
| // MeteredFleetStore builds the platform's own replicator for this box: one |
| -// litestream store over every database FleetDBs lists, each replica client |
| -// metered against the database's owner (path B in |
| +// litestream store over every database the box's inventory lists, each |
| +// replica client metered against the database's owner and gated on that |
| +// ownership still standing (path B in |
| // docs/superpowers/plans/2026-10-01-durability-attestation-path.md). |
| // |
| -// meter returns the recorder for one database, bound by the caller to the |
| -// box's durability ledger. It is asked once for each database. A nil |
| -// recorder refuses the whole store: there is no unmetered fallback. |
| -// |
| -// NOT YET TRUE AS BUILT (second review, DISPOSITIONS-metered-replicator.md): |
| -// the refusal below does not stop a request, because the request meter |
| -// sends on a failed Start, and the witness is moved by this replicator's |
| -// own bookkeeping in the registry it replicates. Both need changes outside |
| -// this package. What follows describes the intent. |
| -// |
| // Who owns each database is read once, here, and a route can change hands |
| -// afterwards. So the read is taken under the registry's change witness, |
| -// and every recorder is wrapped (ownedRecorder): the first request after |
| -// ANY commit to the registry is refused, and every one after it. The rule |
| -// is deliberately blunt. This code cannot tell a change that moves an |
| -// owner from one that does not, and counting a write to an account that |
| -// no longer owns the database is the one thing it must never do. The |
| -// caller rebuilds the store when Current says so. One request already |
| -// past its check when the registry changes can still be counted to the |
| -// owner as read; closing that needs the registry's writer and this |
| -// replicator to share a lock, which nothing here does. |
| +// afterwards. So the routes are read under the registry's ownership |
| +// witness (registry.RouteObservation.OwnershipUnchanged), discovery and |
| +// attribution both work from those same rows, and every replica client |
| +// asks the witness before each request it sends (admit). From the first |
| +// request after a route is added or removed, or changes account, app, |
| +// database path, backing or hibernation, nothing is sent: litestream sees |
| +// a failed upload and does not advance, its local files accumulate, and |
| +// the caller closes the fleet and builds another. The rule is blunt on |
| +// purpose. This code cannot tell a change that moves an owner from one |
| +// that does not, and counting a write to an account that no longer owns |
| +// the database is the one thing it must never do. |
| // |
| -// prefix is the replica lineage this replicator writes. It must not be one |
| -// the external binary has written to: the two do not share a format or a |
| -// metadata directory, and a restore from a prefix both have touched is not |
| -// something anyone has tested. |
| +// What that leaves. A request admitted a moment before the change and |
| +// sent a moment after it is counted to the owner it was admitted for; the |
| +// days it could fall in are held on every recorder (goStale), and on the |
| +// recorders of the fleet that replaces this one. A process that dies |
| +// between the change and the rebuild takes that hold with it: the |
| +// requests it had started are left as started rows, which the ledger |
| +// holds, and the new owner's recorder is told nothing. Closing that needs |
| +// the registry's writer and the replicator to share a lock, or the hold to |
| +// be written down before the request is sent; neither is built. |
| // |
| -// The store is returned unopened, inside a MeteredFleet the caller must |
| -// Close, and nothing here attests anything. A box |
| +// The store is returned unopened, and nothing here attests anything. A box |
| // may say that every replica write was counted only when this is the only |
| // thing on it that holds the replica bucket's credentials, which is a |
| // change to the box and not to this code. |
| -// |
| -// accept names the files discovery skips (database-like files under the |
| -// data directory that no route claims) that the caller has seen and means |
| -// to leave unreplicated. A skipped file that is not in accept refuses the |
| -// store: a database the external replicator still streams, whose route has |
| -// gone, must not drop out of replication because nobody looked. |
| -func MeteredFleetStore(reg *registry.Registry, dataDir, bucket, prefix, region string, accept []string, meter func(OwnedDB) lease.RequestRecorder) (fleet *MeteredFleet, err error) { |
| - if meter == nil { |
| +func MeteredFleetStore(o MeteredFleetOptions) (fleet *MeteredFleet, err error) { |
| + if o.Meter == nil { |
| return nil, fmt.Errorf("%w: no meter", ErrUnmeterable) |
| } |
| - // The witness first, then the ownership read, then the witness again: |
| - // what was read is what the registry held for the whole of the read. |
| - obs, err := reg.ObserveRoutes(context.Background()) |
| + if o.Registry == nil || o.DataDir == "" || !filepath.IsAbs(o.DataDir) { |
| + return nil, errors.New("ops: the replicator needs a registry and an absolute data directory") |
| + } |
| + // The witness first, then the files, then the witness again: what was |
| + // read is what the registry held for the whole of the reading. |
| + obs, err := o.Registry.ObserveRoutes(context.Background()) |
| if err != nil { |
| return nil, err |
| } |
| - built := &MeteredFleet{obs: obs, recorders: map[string]lease.RequestRecorder{}} |
| - fleet = built |
| + built := &MeteredFleet{ |
| + obs: obs, routes: obs.Routes, registryPath: o.Registry.Path(), dataDir: filepath.Clean(o.DataDir), |
| + replaces: o.Replaces, stop: make(chan struct{}), |
| + } |
| + built.proven.Store(time.Now().UnixNano()) |
| defer func() { |
| if err != nil { |
| built.Close() |
| fleet = nil |
| } |
| }() |
| - owned, unowned, skipped, err := FleetDBOwners(reg, dataDir) |
| + inv, err := fleetInventory(built.routes, built.registryPath, built.dataDir) |
| if err != nil { |
| return nil, err |
| } |
| - // Discovery passes over a directory it cannot read. Here that is a |
| - // directory that may hold a database nobody counted. |
| - for _, r := range obs.Routes { |
| - if r.Backing != registry.BackingUnit || r.DBPath == "" { |
| - continue |
| - } |
| - if bad, unreadable := unreadableUnder(filepath.Clean(r.DBPath)); unreadable { |
| - return nil, fmt.Errorf("%w: cannot read %s under %s's state directory", ErrUnmeterable, bad, r.Host) |
| - } |
| - } |
| - if err := fleet.Current(context.Background()); err != nil { |
| - return nil, fmt.Errorf("%w (while ownership was being read)", err) |
| + if len(inv.Problems) > 0 { |
| + return nil, fmt.Errorf("%w: places a database could be that cannot be vouched for: %s", ErrUnmeterable, strings.Join(inv.Problems, "; ")) |
| } |
| accepted := map[string]bool{} |
| - for _, p := range accept { |
| + for _, p := range o.Accept { |
| accepted[filepath.Clean(p)] = true |
| } |
| var unseen []string |
| - for _, p := range skipped { |
| + for _, p := range inv.Skipped { |
| if !accepted[filepath.Clean(p)] { |
| unseen = append(unseen, p) |
| } |
| } |
| if len(unseen) > 0 { |
| - return nil, fmt.Errorf("%w: database files no route claims, not named as accepted: %s", ErrUnmeterable, strings.Join(unseen, ", ")) |
| + return nil, fmt.Errorf("%w: files that look like databases, not replicated and not named as accepted: %s", ErrUnmeterable, strings.Join(unseen, ", ")) |
| } |
| - built.Skipped = skipped |
| - if len(unowned) > 0 { |
| - return nil, fmt.Errorf("%w: %s", ErrUnmeterable, strings.Join(unowned, ", ")) |
| + if len(inv.Unowned) > 0 { |
| + return nil, fmt.Errorf("%w: %s", ErrUnmeterable, strings.Join(inv.Unowned, "; ")) |
| } |
| - if len(owned) == 0 { |
| + if len(inv.Owned) == 0 { |
| return nil, errors.New("ops: no databases to replicate") |
| } |
| - prefix = strings.Trim(prefix, "/") |
| - dbs := make([]lease.MeteredDB, 0, len(owned)) |
| - for _, o := range owned { |
| - inner := meter(o) |
| - if inner == nil { |
| - return nil, fmt.Errorf("%w: no recorder for %s (%s/%s)", ErrUnmeterable, o.Path, o.Account, o.App) |
| + if err := built.admit(context.Background()); err != nil { |
| + return nil, fmt.Errorf("%w (while ownership was being read)", err) |
| + } |
| + keys := make([]string, 0, len(inv.Owned)) |
| + for _, owned := range inv.Owned { |
| + keys = append(keys, replicaKey(owned.Path)) |
| + } |
| + metas, err := prepareShadowState(built.dataDir, keys) |
| + if err != nil { |
| + return nil, err |
| + } |
| + prefix := strings.Trim(o.Prefix, "/") |
| + dbs := make([]lease.MeteredDB, 0, len(inv.Owned)) |
| + for _, owned := range inv.Owned { |
| + rec := o.Meter(owned) |
| + if rec == nil { |
| + return nil, fmt.Errorf("%w: no recorder for %s (%s/%s)", ErrUnmeterable, owned.Path, owned.Account, owned.App) |
| } |
| - rec := ownedRecorder{RequestRecorder: inner, fleet: fleet} |
| - fleet.recorders[o.Path] = rec |
| - key := replicaKey(o.Path) |
| + built.recorders = append(built.recorders, rec) |
| + key := replicaKey(owned.Path) |
| + replica := key |
| if prefix != "" { |
| - key = prefix + "/" + key |
| + replica = prefix + "/" + key |
| } |
| - // Shadow state lives under the platform's data directory, keyed by |
| - // the replica key, never beside a database in a tenant's tree. |
| - meta := filepath.Join(dataDir, ".replicator", replicaKey(o.Path)) |
| - dbs = append(dbs, lease.MeteredDB{Path: o.Path, Prefix: key, Meter: rec, MetaDir: meta}) |
| + dbs = append(dbs, lease.MeteredDB{Path: owned.Path, Prefix: replica, Meter: rec, Gate: built.admit, MetaDir: metas[key], SyncInterval: o.SyncInterval}) |
| } |
| - store, err := lease.NewMeteredStore(bucket, region, dbs) |
| + store, err := lease.NewMeteredStore(o.Bucket, o.Region, dbs) |
| if err != nil { |
| return nil, err |
| } |
| // The fleet's recovery window, not litestream's one-day default: the |
| // external replicator this replaces keeps a week of snapshots. |
| store.SnapshotInterval, store.SnapshotRetention = SnapshotInterval, SnapshotRetention |
| - fleet.store, fleet.Owned = store, owned |
| - return fleet, nil |
| + built.store, built.Owned, built.Skipped = store, inv.Owned, inv.Skipped |
| + built.facts, built.inventory = inv.facts, inv.render() |
| + // The fleet this one replaces went stale over some span of days. A |
| + // database that changed hands has a new owner, whose recorder the old |
| + // fleet never had. |
| + if o.Replaces != nil { |
| + if from, at, stale := o.Replaces.staleWindow(); stale { |
| + holdDays(built.recorders, from, at) |
| + } |
| + } |
| + return built, nil |
| +} |
| + |
| +// prepareShadowState makes the directories litestream keeps each |
| +// database's shadow state in, under <data dir>/.replicator, and answers |
| +// with each key's directory. |
| +// |
| +// Litestream makes a missing metadata directory, and every missing |
| +// directory above it, with the mode and owner of the database's own |
| +// directory. For a unit tenant's database that is the tenant: left to |
| +// litestream, the shadow state of a tenant's database would be the |
| +// tenant's to rewrite, in a tree the replicator then writes and deletes in |
| +// as root. So every directory is made here first, owned by this process |
| +// and closed to everyone else, and litestream finds them made. The root is |
| +// what keeps tenants out: what litestream creates beneath a key's |
| +// directory may carry the tenant's owner, and cannot be reached through a |
| +// directory the tenant cannot enter. |
| +// |
| +// It refuses a data directory that is not the platform's, and any |
| +// component that is already there and is a link, somebody else's, or open |
| +// to anybody else. |
| +func prepareShadowState(dataDir string, keys []string) (map[string]string, error) { |
| + if why := platformOwned(dataDir); why != "" { |
| + return nil, fmt.Errorf("%w: the data directory is not the platform's: %s", ErrUnmeterable, why) |
| + } |
| + private := func(dir string) error { |
| + if err := os.Mkdir(dir, 0o700); err != nil && !errors.Is(err, os.ErrExist) { |
| + return err |
| + } |
| + fi, err := os.Lstat(dir) |
| + if err != nil { |
| + return err |
| + } |
| + facts, ok := factsOf(fi) |
| + switch { |
| + case !fi.IsDir(): |
| + return fmt.Errorf("%w: %s is not a directory", ErrUnmeterable, dir) |
| + case !ok || int(facts.UID) != os.Geteuid(): |
| + return fmt.Errorf("%w: %s does not belong to the platform", ErrUnmeterable, dir) |
| + case fi.Mode().Perm()&0o077 != 0: |
| + return fmt.Errorf("%w: %s is open to users other than the platform's", ErrUnmeterable, dir) |
| + } |
| + return nil |
| + } |
| + root := filepath.Join(dataDir, ".replicator") |
| + if err := private(root); err != nil { |
| + return nil, err |
| + } |
| + out := map[string]string{} |
| + for _, key := range keys { |
| + at := root |
| + for _, part := range strings.Split(filepath.Clean(key), string(filepath.Separator)) { |
| + if part == "" || part == "." || part == ".." { |
| + return nil, fmt.Errorf("%w: replica key %q is not a plain path", ErrUnmeterable, key) |
| + } |
| + at = filepath.Join(at, part) |
| + if err := private(at); err != nil { |
| + return nil, err |
| + } |
| + } |
| + out[key] = at |
| + } |
| + return out, nil |
| } |
diff --git a/internal/ops/litestream_metered_test.go b/internal/ops/litestream_metered_test.go| index c3b850d31..80f76e3bc 100644 |
| --- a/internal/ops/litestream_metered_test.go |
| +++ b/internal/ops/litestream_metered_test.go |
| @@ -1,17 +1,25 @@ |
| package ops |
| import ( |
| + "bytes" |
| "context" |
| + "database/sql" |
| "errors" |
| + "io" |
| + "net/http" |
| + "net/http/httptest" |
| "os" |
| "path/filepath" |
| "strings" |
| + "sync" |
| + "sync/atomic" |
| "testing" |
| "time" |
| + "github.com/superfly/ltx" |
| + |
| "github.com/carlosframework/platform/internal/lease" |
| "github.com/carlosframework/platform/internal/registry" |
| - durability "github.com/carlosframework/platform/internal/usage" |
| ) |
| type nopRecorder struct{} |
| @@ -22,291 +30,665 @@ func (nopRecorder) Abandon(uint64, time.Time) {} |
| func (nopRecorder) Incomplete(time.Time) {} |
| func (nopRecorder) Hold(time.Time) func() { return func() {} } |
| -// The platform's replicator starts only when every database it would |
| -// stream has an owner and a recorder. One that has neither stops the whole |
| -// store: it is not left out quietly and it is not streamed uncounted. |
| -func TestMeteredFleetStoreRefusesADatabaseItCannotMeter(t *testing.T) { |
| - dir := t.TempDir() |
| - reg, err := registry.Open(filepath.Join(dir, "registry.db")) |
| +// countingRecorder counts what it is asked to start and finish and |
| +// remembers each day it is told is incomplete. |
| +type countingRecorder struct { |
| + mu sync.Mutex |
| + started int |
| + finished int |
| + incomplete []time.Time |
| +} |
| + |
| +func (c *countingRecorder) Start(string, time.Time) (uint64, error) { |
| + c.mu.Lock() |
| + defer c.mu.Unlock() |
| + c.started++ |
| + return uint64(c.started), nil |
| +} |
| +func (c *countingRecorder) Finish(uint64, lease.RequestOutcome) error { |
| + c.mu.Lock() |
| + defer c.mu.Unlock() |
| + c.finished++ |
| + return nil |
| +} |
| +func (c *countingRecorder) Abandon(uint64, time.Time) {} |
| +func (c *countingRecorder) Incomplete(at time.Time) { |
| + c.mu.Lock() |
| + defer c.mu.Unlock() |
| + c.incomplete = append(c.incomplete, at) |
| +} |
| +func (c *countingRecorder) Hold(time.Time) func() { return func() {} } |
| + |
| +func (c *countingRecorder) counts() (started, finished, incomplete int) { |
| + c.mu.Lock() |
| + defer c.mu.Unlock() |
| + return c.started, c.finished, len(c.incomplete) |
| +} |
| + |
| +func (c *countingRecorder) days() map[string]bool { |
| + c.mu.Lock() |
| + defer c.mu.Unlock() |
| + out := map[string]bool{} |
| + for _, at := range c.incomplete { |
| + out[at.UTC().Format("2006-01-02")] = true |
| + } |
| + return out |
| +} |
| + |
| +// recorders hands each database its own counting recorder, by path. |
| +type recorders struct { |
| + mu sync.Mutex |
| + by map[string]*countingRecorder |
| +} |
| + |
| +func (r *recorders) meter(o OwnedDB) lease.RequestRecorder { |
| + r.mu.Lock() |
| + defer r.mu.Unlock() |
| + if r.by == nil { |
| + r.by = map[string]*countingRecorder{} |
| + } |
| + c := &countingRecorder{} |
| + r.by[o.Path] = c |
| + return c |
| +} |
| + |
| +func (r *recorders) of(path string) *countingRecorder { |
| + r.mu.Lock() |
| + defer r.mu.Unlock() |
| + return r.by[path] |
| +} |
| + |
| +func (r *recorders) all() map[string]*countingRecorder { |
| + r.mu.Lock() |
| + defer r.mu.Unlock() |
| + out := map[string]*countingRecorder{} |
| + for k, v := range r.by { |
| + out[k] = v |
| + } |
| + return out |
| +} |
| + |
| +func fleetOptions(reg *registry.Registry, dir string) MeteredFleetOptions { |
| + return MeteredFleetOptions{ |
| + Registry: reg, DataDir: dir, Bucket: "replica-bucket", Prefix: "metered/v1", Region: "eu-west-1", |
| + Meter: func(OwnedDB) lease.RequestRecorder { return nopRecorder{} }, |
| + } |
| +} |
| + |
| +// fakeBucket is an S3 endpoint that takes whatever litestream sends and |
| +// remembers each request. |
| +type fakeBucket struct { |
| + mu sync.Mutex |
| + reqs []string |
| + inFlight atomic.Int32 |
| +} |
| + |
| +func newFakeBucket(t *testing.T) *fakeBucket { |
| + t.Helper() |
| + f := &fakeBucket{} |
| + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { |
| + f.inFlight.Add(1) |
| + defer f.inFlight.Add(-1) |
| + _, _ = io.Copy(io.Discard, r.Body) |
| + f.mu.Lock() |
| + f.reqs = append(f.reqs, r.Method+" "+r.URL.Path) |
| + f.mu.Unlock() |
| + w.Header().Set("Content-Type", "application/xml") |
| + switch { |
| + case r.Method == http.MethodPut: |
| + w.Header().Set("ETag", `"etag"`) |
| + w.WriteHeader(http.StatusOK) |
| + case r.Method == http.MethodGet && r.URL.Query().Has("list-type"): |
| + _, _ = io.WriteString(w, `<ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><KeyCount>0</KeyCount><IsTruncated>false</IsTruncated></ListBucketResult>`) |
| + case r.Method == http.MethodGet || r.Method == http.MethodHead: |
| + w.WriteHeader(http.StatusNotFound) |
| + _, _ = io.WriteString(w, `<?xml version="1.0" encoding="UTF-8"?><Error><Code>NoSuchKey</Code><Message>no such key</Message></Error>`) |
| + case r.Method == http.MethodDelete: |
| + w.WriteHeader(http.StatusNoContent) |
| + case r.Method == http.MethodPost: |
| + _, _ = io.WriteString(w, `<DeleteResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"></DeleteResult>`) |
| + default: |
| + w.WriteHeader(http.StatusMethodNotAllowed) |
| + } |
| + })) |
| + t.Cleanup(srv.Close) |
| + t.Setenv("CARLOS_S3_ENDPOINT", srv.URL) |
| + t.Setenv("AWS_ACCESS_KEY_ID", "test-key") |
| + t.Setenv("AWS_SECRET_ACCESS_KEY", "test-secret") |
| + t.Setenv("AWS_SESSION_TOKEN", "") |
| + t.Setenv("AWS_REGION", "") |
| + t.Setenv("AWS_DEFAULT_REGION", "") |
| + t.Setenv("AWS_EC2_METADATA_DISABLED", "true") |
| + return f |
| +} |
| + |
| +func (f *fakeBucket) count() int { |
| + f.mu.Lock() |
| + defer f.mu.Unlock() |
| + return len(f.reqs) |
| +} |
| + |
| +// puts is how many PUTs landed under a key that contains part. |
| +func (f *fakeBucket) puts(part string) int { |
| + f.mu.Lock() |
| + defer f.mu.Unlock() |
| + n := 0 |
| + for _, r := range f.reqs { |
| + if strings.HasPrefix(r, "PUT ") && strings.Contains(r, part) { |
| + n++ |
| + } |
| + } |
| + return n |
| +} |
| + |
| +func ltxBytes(t *testing.T) []byte { |
| + t.Helper() |
| + buf := new(bytes.Buffer) |
| + enc, err := ltx.NewEncoder(buf) |
| if err != nil { |
| t.Fatal(err) |
| } |
| - owned := filepath.Join(dir, "a.example.com.db") |
| - if err := os.WriteFile(owned, []byte("x"), 0o644); err != nil { |
| + if err := enc.EncodeHeader(ltx.Header{Version: ltx.Version, PageSize: 4096, MinTXID: 2, MaxTXID: 2, Timestamp: time.Now().UnixMilli(), PreApplyChecksum: ltx.ChecksumFlag | 1}); err != nil { |
| + t.Fatal(err) |
| + } |
| + enc.SetPostApplyChecksum(ltx.ChecksumFlag) |
| + if err := enc.Close(); err != nil { |
| + t.Fatal(err) |
| + } |
| + return buf.Bytes() |
| +} |
| + |
| +// send makes one replica write through the client the fleet's store gave |
| +// the database at path: the client litestream itself writes through. |
| +func send(t *testing.T, f *MeteredFleet, path string, txid int) error { |
| + t.Helper() |
| + for _, db := range f.store.DBs() { |
| + if db.Path() == path { |
| + _, err := db.Replica.Client.WriteLTXFile(context.Background(), 0, ltx.TXID(txid), ltx.TXID(txid), bytes.NewReader(ltxBytes(t))) |
| + return err |
| + } |
| + } |
| + t.Fatalf("the store has no database at %s", path) |
| + return nil |
| +} |
| + |
| +// rawExec runs a statement against a SQLite file over its own connection: |
| +// some other process writing to it. |
| +func rawExec(t *testing.T, path, stmt string) { |
| + t.Helper() |
| + db, err := sql.Open("sqlite", path) |
| + if err != nil { |
| t.Fatal(err) |
| } |
| - if err := reg.Upsert(registry.Route{Host: "a.example.com", Account: "bac", App: "one", DBPath: owned}); err != nil { |
| + defer db.Close() |
| + if _, err := db.Exec("PRAGMA busy_timeout=5000"); err != nil { |
| t.Fatal(err) |
| } |
| + if _, err := db.Exec(stmt); err != nil { |
| + t.Fatalf("%s: %v", stmt, err) |
| + } |
| +} |
| + |
| +func eventually(t *testing.T, what string, ok func() bool) { |
| + t.Helper() |
| + deadline := time.Now().Add(20 * time.Second) |
| + for !ok() { |
| + if time.Now().After(deadline) { |
| + t.Fatalf("timed out waiting for %s", what) |
| + } |
| + time.Sleep(10 * time.Millisecond) |
| + } |
| +} |
| + |
| +// The platform's replicator starts only when every database it would |
| +// stream has an owner and a recorder. One that has neither stops the whole |
| +// store: it is not left out quietly and it is not streamed uncounted. |
| +func TestMeteredFleetStoreRefusesADatabaseItCannotMeter(t *testing.T) { |
| + dir := platformDir(t) |
| + reg := openRegistry(t, filepath.Join(dir, "registry.db")) |
| + owned := writeFile(t, filepath.Join(dir, "a.example.com.db")) |
| + upsert(t, reg, registry.Route{Host: "a.example.com", Account: "bac", App: "one", DBPath: owned}) |
| var asked []OwnedDB |
| - meter := func(o OwnedDB) lease.RequestRecorder { |
| - asked = append(asked, o) |
| + o := fleetOptions(reg, dir) |
| + o.Prefix = "/metered/v1/" |
| + o.Meter = func(db OwnedDB) lease.RequestRecorder { |
| + asked = append(asked, db) |
| return nopRecorder{} |
| } |
| - fleet, err := MeteredFleetStore(reg, dir, "replica-bucket", "/metered/v1/", "eu-west-1", nil, meter) |
| + fleet, err := MeteredFleetStore(o) |
| if err != nil { |
| t.Fatal(err) |
| } |
| defer fleet.Close() |
| - store, dbs := fleet.store, fleet.Owned |
| - if len(dbs) != 2 || len(asked) != 2 || len(store.DBs()) != 2 { |
| - t.Fatalf("store over %d databases, %d owners, meter asked %d times: want the route's database and the registry", len(store.DBs()), len(dbs), len(asked)) |
| + if len(fleet.Owned) != 2 || len(asked) != 2 || len(fleet.store.DBs()) != 2 { |
| + t.Fatalf("store over %d databases, %d owners, meter asked %d times: want the route's database and the registry", len(fleet.store.DBs()), len(fleet.Owned), len(asked)) |
| } |
| - for _, o := range asked { |
| - if o.Path == owned && (o.Account != "bac" || o.App != "one" || o.Platform) { |
| - t.Errorf("the route's database was metered against %+v", o) |
| + for _, db := range asked { |
| + if db.Path == owned && (db.Account != "bac" || db.App != "one" || db.Platform) { |
| + t.Errorf("the route's database was metered against %+v", db) |
| } |
| - if o.Path != owned && !o.Platform { |
| - t.Errorf("the box's own database was not metered against the platform: %+v", o) |
| + if db.Path != owned && !db.Platform { |
| + t.Errorf("the box's own database was not metered against the platform: %+v", db) |
| } |
| } |
| // A recorder that cannot be bound: nothing is replicated unmetered. |
| - _, err = MeteredFleetStore(reg, dir, "replica-bucket", "metered/v1", "eu-west-1", nil, func(o OwnedDB) lease.RequestRecorder { |
| - if o.Path == owned { |
| + o = fleetOptions(reg, dir) |
| + o.Meter = func(db OwnedDB) lease.RequestRecorder { |
| + if db.Path == owned { |
| return nil |
| } |
| return nopRecorder{} |
| - }) |
| - if !errors.Is(err, ErrUnmeterable) || !strings.Contains(err.Error(), owned) { |
| + } |
| + if _, err = MeteredFleetStore(o); !errors.Is(err, ErrUnmeterable) || !strings.Contains(err.Error(), owned) { |
| t.Fatalf("a database with no recorder: %v, want ErrUnmeterable naming it", err) |
| } |
| - if _, err := MeteredFleetStore(reg, dir, "replica-bucket", "metered/v1", "eu-west-1", nil, nil); !errors.Is(err, ErrUnmeterable) { |
| + o.Meter = nil |
| + if _, err := MeteredFleetStore(o); !errors.Is(err, ErrUnmeterable) { |
| t.Fatalf("no meter at all: %v", err) |
| } |
| // A database two accounts claim. |
| - shared := filepath.Join(dir, "shared.example.com.db") |
| - if err := os.WriteFile(shared, []byte("x"), 0o644); err != nil { |
| - t.Fatal(err) |
| - } |
| - for _, r := range []registry.Route{ |
| - {Host: "shared.example.com", Account: "bac", App: "one", DBPath: shared}, |
| - {Host: "shared2.example.com", Account: "bae", App: "two", DBPath: shared}, |
| - } { |
| - if err := reg.Upsert(r); err != nil { |
| - t.Fatal(err) |
| - } |
| - } |
| - _, err = MeteredFleetStore(reg, dir, "replica-bucket", "metered/v1", "eu-west-1", nil, meter) |
| - if !errors.Is(err, ErrUnmeterable) || !strings.Contains(err.Error(), shared) { |
| + shared := writeFile(t, filepath.Join(dir, "shared.example.com.db")) |
| + upsert(t, reg, |
| + registry.Route{Host: "shared.example.com", Account: "bac", App: "one", DBPath: shared}, |
| + registry.Route{Host: "shared2.example.com", Account: "bae", App: "two", DBPath: shared}, |
| + ) |
| + if _, err = MeteredFleetStore(fleetOptions(reg, dir)); !errors.Is(err, ErrUnmeterable) || !strings.Contains(err.Error(), shared) { |
| t.Fatalf("a database with no single owner: %v, want ErrUnmeterable naming it", err) |
| } |
| } |
| -// countingRecorder counts what it is asked to start and remembers being |
| -// told its day is incomplete. |
| -type countingRecorder struct { |
| - started int |
| - incomplete int |
| -} |
| - |
| -func (c *countingRecorder) Start(string, time.Time) (uint64, error) { |
| - c.started++ |
| - return uint64(c.started), nil |
| -} |
| -func (c *countingRecorder) Finish(uint64, lease.RequestOutcome) error { return nil } |
| -func (c *countingRecorder) Abandon(uint64, time.Time) {} |
| -func (c *countingRecorder) Incomplete(time.Time) { c.incomplete++ } |
| -func (c *countingRecorder) Hold(time.Time) func() { return func() {} } |
| - |
| -// Who owns a database is read when the replicator is built. A route can |
| -// change hands afterwards: moved to another account, deleted and made |
| -// again, put to sleep. The recorder a replica client holds must not go on |
| -// counting to the owner it was built for. After any commit to the |
| -// registry, a request is not started and not counted to anybody, the old |
| -// owner's ledger is told its day is incomplete, and the fleet says it must |
| -// be rebuilt. |
| -func TestMeteredFleetStopsCountingWhenTheRegistryChanges(t *testing.T) { |
| - for name, change := range map[string]func(t *testing.T, reg *registry.Registry, db string){ |
| - "the route moves to another account": func(t *testing.T, reg *registry.Registry, db string) { |
| - if err := reg.Upsert(registry.Route{Host: "a.example.com", Account: "bae", App: "one", DBPath: db}); err != nil { |
| +// Who owns a database is read when the replicator is built, and a route |
| +// can change hands afterwards. This drives the clients the fleet's store |
| +// gave its databases, the ones litestream writes through, against a |
| +// server that counts what reaches it. |
| +// |
| +// Before a change: a write goes out, and is counted on the recorder of the |
| +// database it was made for and on no other. After a change of ownership: |
| +// the write fails, nothing reaches the server, nothing is counted to |
| +// anybody, and every recorder is told its day is incomplete, the ones that |
| +// made no request too. A commit to the registry that has nothing to do |
| +// with ownership changes none of that, the replicator's own bookkeeping in |
| +// the registry it replicates included. |
| +func TestMeteredFleetStopsSendingWhenOwnershipChanges(t *testing.T) { |
| + type change func(t *testing.T, reg *registry.Registry, db string) |
| + run := func(name string, do change, stops bool) { |
| + t.Run(name, func(t *testing.T) { |
| + bucket := newFakeBucket(t) |
| + dir := platformDir(t) |
| + reg := openRegistry(t, filepath.Join(dir, "registry.db")) |
| + db := writeFile(t, filepath.Join(dir, "a.example.com.db")) |
| + upsert(t, reg, registry.Route{Host: "a.example.com", Account: "bac", App: "one", DBPath: db, Version: "v1"}) |
| + recs := &recorders{} |
| + o := fleetOptions(reg, dir) |
| + o.Meter = recs.meter |
| + fleet, err := MeteredFleetStore(o) |
| + if err != nil { |
| t.Fatal(err) |
| } |
| - }, |
| - "the route starts hibernating": func(t *testing.T, reg *registry.Registry, db string) { |
| - if err := reg.Upsert(registry.Route{Host: "a.example.com", Account: "bac", App: "one", DBPath: db, Hibernate: true}); err != nil { |
| - t.Fatal(err) |
| + defer fleet.Close() |
| + regPath := filepath.Join(dir, "registry.db") |
| + if err := send(t, fleet, db, 2); err != nil { |
| + t.Fatalf("a write before any change: %v", err) |
| } |
| - }, |
| - "an unrelated route is added": func(t *testing.T, reg *registry.Registry, db string) { |
| - if err := reg.Upsert(registry.Route{Host: "other.example.com", Account: "bae", App: "two"}); err != nil { |
| + if s, f, _ := recs.of(db).counts(); s != 1 || f != 1 { |
| + t.Fatalf("the write was not counted whole on its own database's recorder: %d started, %d finished", s, f) |
| + } |
| + if s, _, _ := recs.of(regPath).counts(); s != 0 { |
| + t.Fatalf("the tenant's write was counted on the platform's recorder (%d)", s) |
| + } |
| + if bucket.count() != 1 || bucket.puts("a.example.com.db") != 1 { |
| + t.Fatalf("requests at the bucket before any change: %v", bucket.reqs) |
| + } |
| + do(t, reg, db) |
| + err = send(t, fleet, db, 3) |
| + if !stops { |
| + if err != nil { |
| + t.Fatalf("a write after a commit that changes nobody's ownership: %v", err) |
| + } |
| + if s, _, inc := recs.of(db).counts(); s != 2 || inc != 0 { |
| + t.Fatalf("after it: %d started, %d incomplete; want 2 and 0", s, inc) |
| + } |
| + if err := fleet.Current(context.Background()); err != nil { |
| + t.Fatalf("the fleet says %v after a commit that changes nobody's ownership", err) |
| + } |
| + return |
| + } |
| + if !errors.Is(err, ErrOwnershipStale) || !errors.Is(err, lease.ErrRequestRefused) { |
| + t.Fatalf("a write after the change: %v, want a refusal for stale ownership", err) |
| + } |
| + // The platform's own database too: the fleet is stale, not one client. |
| + if err := send(t, fleet, regPath, 2); !errors.Is(err, ErrOwnershipStale) { |
| + t.Fatalf("the platform's write after the change: %v", err) |
| + } |
| + if bucket.count() != 1 { |
| + t.Fatalf("requests reached the bucket after the change: %v", bucket.reqs) |
| + } |
| + for path, rec := range recs.all() { |
| + s, _, inc := rec.counts() |
| + if path == db && s != 1 { |
| + t.Errorf("a refused write was counted to the owner as first read (%d started)", s) |
| + } |
| + if path != db && s != 0 { |
| + t.Errorf("%s: %d started", path, s) |
| + } |
| + if inc == 0 { |
| + t.Errorf("%s: its recorder was not told its day is incomplete", path) |
| + } |
| + } |
| + if err := fleet.Current(context.Background()); !errors.Is(err, ErrOwnershipStale) { |
| + t.Fatalf("the fleet does not say it must be rebuilt: %v", err) |
| + } |
| + }) |
| + } |
| + run("the route moves to another account", func(t *testing.T, reg *registry.Registry, db string) { |
| + upsert(t, reg, registry.Route{Host: "a.example.com", Account: "bae", App: "one", DBPath: db, Version: "v1"}) |
| + }, true) |
| + run("the route moves to another account and back", func(t *testing.T, reg *registry.Registry, db string) { |
| + for _, acct := range []string{"bae", "bac"} { |
| + if err := reg.SetAccount("a.example.com", acct); err != nil { |
| t.Fatal(err) |
| } |
| - }, |
| - } { |
| - dir := t.TempDir() |
| - reg, err := registry.Open(filepath.Join(dir, "registry.db")) |
| - if err != nil { |
| - t.Fatal(err) |
| } |
| - db := filepath.Join(dir, "a.example.com.db") |
| - if err := os.WriteFile(db, []byte("x"), 0o644); err != nil { |
| + }, true) |
| + run("the route starts hibernating", func(t *testing.T, reg *registry.Registry, db string) { |
| + upsert(t, reg, registry.Route{Host: "a.example.com", Account: "bac", App: "one", DBPath: db, Version: "v1", Hibernate: true}) |
| + }, true) |
| + run("the route is deleted", func(t *testing.T, reg *registry.Registry, db string) { |
| + if err := reg.Delete("a.example.com"); err != nil { |
| t.Fatal(err) |
| } |
| - if err := reg.Upsert(registry.Route{Host: "a.example.com", Account: "bac", App: "one", DBPath: db}); err != nil { |
| + }, true) |
| + run("another route is added", func(t *testing.T, reg *registry.Registry, db string) { |
| + upsert(t, reg, registry.Route{Host: "other.example.com", Account: "bae", App: "two"}) |
| + }, true) |
| + run("a deploy writes the row with the same owner", func(t *testing.T, reg *registry.Registry, db string) { |
| + upsert(t, reg, registry.Route{Host: "a.example.com", Account: "bac", App: "one", DBPath: db, Version: "v2", Status: "ready"}) |
| + }, false) |
| + run("a worker's next due time", func(t *testing.T, reg *registry.Registry, db string) { |
| + if err := reg.SetNextDue("a.example.com", 1234); err != nil { |
| t.Fatal(err) |
| } |
| - inner := map[string]*countingRecorder{} |
| - fleet, err := MeteredFleetStore(reg, dir, "replica-bucket", "metered/v1", "eu-west-1", nil, func(o OwnedDB) lease.RequestRecorder { |
| - c := &countingRecorder{} |
| - inner[o.Path] = c |
| - return c |
| - }) |
| - if err != nil { |
| - t.Fatalf("%s: %v", name, err) |
| - } |
| - rec := fleet.recorders[db] |
| - if rec == nil || inner[db] == nil { |
| - t.Fatalf("%s: fixture: no recorder for the route's database", name) |
| - } |
| - at := time.Date(2026, 10, 1, 10, 0, 0, 0, time.UTC) |
| - if _, err := rec.Start("PutObject", at); err != nil || inner[db].started != 1 { |
| - t.Fatalf("%s: a request before any change: %v, started %d", name, err, inner[db].started) |
| - } |
| - if err := fleet.Current(context.Background()); err != nil { |
| - t.Fatalf("%s: current before any change: %v", name, err) |
| - } |
| - change(t, reg, db) |
| - if _, err := rec.Start("PutObject", at.Add(time.Second)); !errors.Is(err, ErrOwnershipStale) { |
| - t.Fatalf("%s: a request after the change: %v, want ErrOwnershipStale", name, err) |
| - } |
| - if inner[db].started != 1 { |
| - t.Fatalf("%s: a request after the change was counted to the owner as first read (%d started)", name, inner[db].started) |
| + }, false) |
| + run("a certificate's expiry and a DNS check", func(t *testing.T, reg *registry.Registry, db string) { |
| + if err := reg.SetCertExpiresAt("a.example.com", 99); err != nil { |
| + t.Fatal(err) |
| } |
| - if inner[db].incomplete != 1 { |
| - t.Fatalf("%s: the old owner's ledger was not told its day is incomplete", name) |
| + if err := reg.SetDNSStatus("a.example.com", "ok", 99); err != nil { |
| + t.Fatal(err) |
| } |
| - if err := fleet.Current(context.Background()); !errors.Is(err, ErrOwnershipStale) { |
| - t.Fatalf("%s: the fleet does not say it must be rebuilt: %v", name, err) |
| + }, false) |
| + run("the replicator's own bookkeeping in the registry", func(t *testing.T, reg *registry.Registry, db string) { |
| + rawExec(t, reg.Path(), `CREATE TABLE IF NOT EXISTS _litestream_seq (id INTEGER PRIMARY KEY, seq INTEGER)`) |
| + rawExec(t, reg.Path(), `INSERT INTO _litestream_seq (id, seq) VALUES (1, 1) ON CONFLICT (id) DO UPDATE SET seq = seq + 1`) |
| + }, false) |
| +} |
| + |
| +// Nobody has to make a request for a change of ownership to be noticed. |
| +// With the fleet's watch running and no request made at all, every |
| +// recorder is told its day is incomplete soon after the change. |
| +func TestMeteredFleetWatchHoldsEveryRecorderWithoutARequest(t *testing.T) { |
| + bucket := newFakeBucket(t) |
| + prev := ownershipWatchInterval |
| + ownershipWatchInterval = 5 * time.Millisecond |
| + t.Cleanup(func() { ownershipWatchInterval = prev }) |
| + dir := platformDir(t) |
| + reg := openRegistry(t, filepath.Join(dir, "registry.db")) |
| + db := writeFile(t, filepath.Join(dir, "a.example.com.db")) |
| + upsert(t, reg, registry.Route{Host: "a.example.com", Account: "bac", App: "one", DBPath: db}) |
| + recs := &recorders{} |
| + o := fleetOptions(reg, dir) |
| + o.Meter = recs.meter |
| + fleet, err := MeteredFleetStore(o) |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + defer fleet.Close() |
| + // The watch alone, without the store: no database is open, so nothing |
| + // can make a request. |
| + fleet.watchOnce.Do(func() { |
| + fleet.watching.Add(1) |
| + go fleet.watch() |
| + }) |
| + time.Sleep(50 * time.Millisecond) |
| + for path, rec := range recs.all() { |
| + if _, _, inc := rec.counts(); inc != 0 { |
| + t.Fatalf("%s was told its day is incomplete before anything changed", path) |
| } |
| - // Stale for good: every recorder, the platform's included. |
| - for path, r := range fleet.recorders { |
| - if _, err := r.Start("PutObject", at.Add(2*time.Second)); !errors.Is(err, ErrOwnershipStale) { |
| - t.Fatalf("%s: %s still counts after the change: %v", name, path, err) |
| + } |
| + if err := reg.SetAccount("a.example.com", "bae"); err != nil { |
| + t.Fatal(err) |
| + } |
| + eventually(t, "every recorder to be told", func() bool { |
| + for _, rec := range recs.all() { |
| + if _, _, inc := rec.counts(); inc == 0 { |
| + return false |
| } |
| } |
| - fleet.Close() |
| + return true |
| + }) |
| + if bucket.count() != 0 { |
| + t.Fatalf("requests were made: %v", bucket.reqs) |
| + } |
| + if err := fleet.Close(); err != nil { |
| + t.Fatal(err) |
| } |
| } |
| -// The identities the box's own databases are metered under are ones the |
| -// real durability ledger accepts for the platform's purpose. A no-op |
| -// recorder would never notice if they were not. |
| -func TestPlatformDatabasesHaveIdentitiesTheLedgerAccepts(t *testing.T) { |
| - dir := t.TempDir() |
| - reg, err := registry.Open(filepath.Join(dir, "registry.db")) |
| +// The days a stale fleet holds run from a margin before the last instant |
| +// ownership was shown unchanged to the instant it was found changed, every |
| +// UTC day in between, on every recorder. |
| +func TestStaleFleetHoldsEveryDayInDoubt(t *testing.T) { |
| + day := func(d int, h, m, s int) time.Time { return time.Date(2026, 10, d, h, m, s, 0, time.UTC) } |
| + for name, tc := range map[string]struct { |
| + proven, found time.Time |
| + want []string |
| + }{ |
| + "within one day": {day(2, 10, 0, 0), day(2, 10, 0, 1), []string{"2026-10-02"}}, |
| + "found after midnight": {day(2, 23, 59, 50), day(3, 0, 0, 5), []string{"2026-10-02", "2026-10-03"}}, |
| + "proven just after midnight": {day(3, 0, 0, 20), day(3, 0, 0, 40), []string{"2026-10-02", "2026-10-03"}}, |
| + "a watch that did not run for some days": {day(2, 12, 0, 0), day(4, 12, 0, 0), []string{"2026-10-02", "2026-10-03", "2026-10-04"}}, |
| + } { |
| + t.Run(name, func(t *testing.T) { |
| + a, b := &countingRecorder{}, &countingRecorder{} |
| + f := &MeteredFleet{recorders: []lease.RequestRecorder{a, b}} |
| + f.proven.Store(tc.proven.UnixNano()) |
| + f.goStale(tc.found) |
| + for _, rec := range []*countingRecorder{a, b} { |
| + got := rec.days() |
| + if len(got) != len(tc.want) { |
| + t.Fatalf("held %v, want %v", got, tc.want) |
| + } |
| + for _, d := range tc.want { |
| + if !got[d] { |
| + t.Fatalf("held %v, want %v", got, tc.want) |
| + } |
| + } |
| + } |
| + // Once: a second finding holds nothing more. |
| + _, _, before := a.counts() |
| + f.goStale(tc.found.Add(time.Hour)) |
| + if _, _, after := a.counts(); after != before { |
| + t.Fatal("a fleet went stale twice") |
| + } |
| + }) |
| + } |
| +} |
| + |
| +// The fleet built to take over from one that went stale tells its own |
| +// recorders the same days are incomplete: a database that changed hands |
| +// has a new owner, whose recorder the old fleet never had. And it does not |
| +// open while the fleet it replaces is still open. |
| +func TestMeteredFleetReplacementInheritsTheHold(t *testing.T) { |
| + newFakeBucket(t) |
| + dir := platformDir(t) |
| + reg := openRegistry(t, filepath.Join(dir, "registry.db")) |
| + db := writeFile(t, filepath.Join(dir, "a.example.com.db")) |
| + upsert(t, reg, registry.Route{Host: "a.example.com", Account: "bac", App: "one", DBPath: db}) |
| + o := fleetOptions(reg, dir) |
| + old, err := MeteredFleetStore(o) |
| if err != nil { |
| t.Fatal(err) |
| } |
| - owned, unowned, _, err := FleetDBOwners(reg, dir) |
| - if err != nil || len(unowned) != 0 || len(owned) == 0 { |
| - t.Fatalf("owned %+v, unowned %v, err %v", owned, unowned, err) |
| + defer old.Close() |
| + if err := reg.SetAccount("a.example.com", "bae"); err != nil { |
| + t.Fatal(err) |
| + } |
| + if err := old.Current(context.Background()); !errors.Is(err, ErrOwnershipStale) { |
| + t.Fatalf("the old fleet after the change: %v", err) |
| } |
| - for _, o := range owned { |
| - if !o.Platform { |
| - t.Fatalf("fixture: %+v is not the platform's", o) |
| + recs := &recorders{} |
| + o.Meter, o.Replaces = recs.meter, old |
| + next, err := MeteredFleetStore(o) |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + defer next.Close() |
| + var owner OwnedDB |
| + for _, od := range next.Owned { |
| + if od.Path == db { |
| + owner = od |
| } |
| - attr := durability.ReplicationAttribution{Account: o.Account, App: o.App, Host: o.Host, Purpose: durability.PurposePlatform} |
| - if err := attr.Validate(); err != nil { |
| - t.Errorf("%s: the ledger refuses the platform's identity %+v: %v", o.Path, attr, err) |
| + } |
| + if owner.Account != "bae" { |
| + t.Fatalf("the new fleet's owner of the database: %+v", owner) |
| + } |
| + for path, rec := range recs.all() { |
| + if _, _, inc := rec.counts(); inc == 0 { |
| + t.Errorf("%s: the new fleet's recorder was not told the days the old fleet held", path) |
| } |
| } |
| -} |
| - |
| -// A database-like file under the data directory that no route claims is |
| -// not replicated. It may be one the external replicator still streams, for |
| -// a route that has gone. The store is refused until the caller names it as |
| -// seen; then it is listed, not dropped. |
| -func TestMeteredFleetStoreRefusesAnUnclaimedDatabaseNobodyNamed(t *testing.T) { |
| - dir := t.TempDir() |
| - reg, err := registry.Open(filepath.Join(dir, "registry.db")) |
| + if err := next.Open(context.Background()); err == nil || !strings.Contains(err.Error(), "not been closed") { |
| + t.Fatalf("opening beside the fleet it replaces: %v", err) |
| + } |
| + // A fleet that replaces one that was current inherits nothing. |
| + if err := old.Close(); err != nil { |
| + t.Fatal(err) |
| + } |
| + if err := next.Close(); err != nil { |
| + t.Fatal(err) |
| + } |
| + o.Replaces = nil |
| + clean, err := MeteredFleetStore(o) |
| if err != nil { |
| t.Fatal(err) |
| } |
| - leftover := filepath.Join(dir, "gone.example.com.db") |
| - if err := os.WriteFile(leftover, []byte("x"), 0o644); err != nil { |
| + if err := clean.Close(); err != nil { |
| + t.Fatal(err) |
| + } |
| + recs2 := &recorders{} |
| + o.Meter, o.Replaces = recs2.meter, clean |
| + after, err := MeteredFleetStore(o) |
| + if err != nil { |
| t.Fatal(err) |
| } |
| - meter := func(OwnedDB) lease.RequestRecorder { return nopRecorder{} } |
| - fleet, err := MeteredFleetStore(reg, dir, "replica-bucket", "metered/v1", "eu-west-1", nil, meter) |
| - if !errors.Is(err, ErrUnmeterable) || !strings.Contains(err.Error(), leftover) || fleet != nil { |
| - t.Fatalf("an unclaimed database nobody named: fleet %v, err %v; want ErrUnmeterable naming it and no fleet", fleet, err) |
| + defer after.Close() |
| + for path, rec := range recs2.all() { |
| + if _, _, inc := rec.counts(); inc != 0 { |
| + t.Errorf("%s: told its day is incomplete after a clean handover", path) |
| + } |
| } |
| - fleet, err = MeteredFleetStore(reg, dir, "replica-bucket", "metered/v1", "eu-west-1", []string{leftover}, meter) |
| +} |
| + |
| +// A database-like file under the data directory that no route claims is |
| +// not replicated. It may be one the external replicator still streams, for |
| +// a route that has gone. The store is refused until the caller names it as |
| +// seen; then it is listed, not dropped. The same goes for a database in a |
| +// tenant's tree that discovery does not take. |
| +func TestMeteredFleetStoreRefusesAnUnclaimedDatabaseNobodyNamed(t *testing.T) { |
| + dir := platformDir(t) |
| + reg := openRegistry(t, filepath.Join(dir, "registry.db")) |
| + leftover := writeFile(t, filepath.Join(dir, "gone.example.com.db")) |
| + state := filepath.Join(platformDir(t), "unit-state") |
| + writeFile(t, filepath.Join(state, "main.db")) |
| + hidden := writeFile(t, filepath.Join(state, ".cache", "hidden.db")) |
| + upsert(t, reg, registry.Route{Host: "unit.example.com", Account: "bad", App: "unit", DBPath: state, Backing: registry.BackingUnit}) |
| + o := fleetOptions(reg, dir) |
| + fleet, err := MeteredFleetStore(o) |
| + if !errors.Is(err, ErrUnmeterable) || !strings.Contains(err.Error(), leftover) || !strings.Contains(err.Error(), hidden) || fleet != nil { |
| + t.Fatalf("unclaimed databases nobody named: fleet %v, err %v; want ErrUnmeterable naming both and no fleet", fleet, err) |
| + } |
| + o.Accept = []string{leftover} |
| + if _, err := MeteredFleetStore(o); !errors.Is(err, ErrUnmeterable) || !strings.Contains(err.Error(), hidden) || strings.Contains(err.Error(), leftover) { |
| + t.Fatalf("with one of the two named: %v; want a refusal naming only the other", err) |
| + } |
| + o.Accept = []string{leftover, hidden} |
| + fleet, err = MeteredFleetStore(o) |
| if err != nil { |
| - t.Fatalf("with the file named as seen: %v", err) |
| + t.Fatalf("with both named as seen: %v", err) |
| } |
| defer fleet.Close() |
| - if len(fleet.Skipped) != 1 || fleet.Skipped[0] != leftover { |
| - t.Fatalf("Skipped = %v, want the named file", fleet.Skipped) |
| + if len(fleet.Skipped) != 2 { |
| + t.Fatalf("Skipped = %v, want the two named files", fleet.Skipped) |
| } |
| // The fleet's recovery window, not litestream's default day. |
| if fleet.store.SnapshotRetention != SnapshotRetention || fleet.store.SnapshotInterval != SnapshotInterval { |
| t.Fatalf("snapshot policy %s every %s, want %s every %s", fleet.store.SnapshotRetention, fleet.store.SnapshotInterval, SnapshotRetention, SnapshotInterval) |
| } |
| - if fleet.Databases() != 1 { |
| - t.Fatalf("%d databases, want the registry alone", fleet.Databases()) |
| + if fleet.Databases() != 2 { |
| + t.Fatalf("%d databases, want the registry and the tenant's", fleet.Databases()) |
| } |
| } |
| -// Discovery passes over a directory it cannot read. For the replicator |
| -// that is a directory that may hold a database nobody counted, and it |
| -// refuses to start. |
| +// A directory in a tenant's tree that cannot be read may hold a database |
| +// nobody counted. The replicator refuses to start, and says whose tree. |
| func TestMeteredFleetStoreRefusesAStateDirectoryItCannotRead(t *testing.T) { |
| if os.Geteuid() == 0 { |
| t.Skip("root reads every directory") |
| } |
| - dir := t.TempDir() |
| - reg, err := registry.Open(filepath.Join(dir, "registry.db")) |
| - if err != nil { |
| - t.Fatal(err) |
| - } |
| - state := filepath.Join(t.TempDir(), "unit-state") |
| + dir := platformDir(t) |
| + reg := openRegistry(t, filepath.Join(dir, "registry.db")) |
| + state := filepath.Join(platformDir(t), "unit-state") |
| hidden := filepath.Join(state, "tenants") |
| - if err := os.MkdirAll(hidden, 0o755); err != nil { |
| - t.Fatal(err) |
| - } |
| - if err := os.WriteFile(filepath.Join(state, "main.db"), []byte("x"), 0o644); err != nil { |
| - t.Fatal(err) |
| - } |
| - if err := os.WriteFile(filepath.Join(hidden, "t1.db"), []byte("x"), 0o644); err != nil { |
| - t.Fatal(err) |
| - } |
| - if err := reg.Upsert(registry.Route{Host: "unit.example.com", Account: "bad", App: "unit", DBPath: state, Backing: registry.BackingUnit}); err != nil { |
| - t.Fatal(err) |
| - } |
| + writeFile(t, filepath.Join(state, "main.db")) |
| + writeFile(t, filepath.Join(hidden, "t1.db")) |
| + upsert(t, reg, registry.Route{Host: "unit.example.com", Account: "bad", App: "unit", DBPath: state, Backing: registry.BackingUnit}) |
| if err := os.Chmod(hidden, 0o000); err != nil { |
| t.Fatal(err) |
| } |
| t.Cleanup(func() { os.Chmod(hidden, 0o755) }) |
| - fleet, err := MeteredFleetStore(reg, dir, "replica-bucket", "metered/v1", "eu-west-1", nil, func(OwnedDB) lease.RequestRecorder { return nopRecorder{} }) |
| - if !errors.Is(err, ErrUnmeterable) || !strings.Contains(err.Error(), hidden) || fleet != nil { |
| - t.Fatalf("a state directory with an unreadable part: fleet %v, err %v; want ErrUnmeterable naming it and no fleet", fleet, err) |
| + fleet, err := MeteredFleetStore(fleetOptions(reg, dir)) |
| + if !errors.Is(err, ErrUnmeterable) || !strings.Contains(err.Error(), hidden) || !strings.Contains(err.Error(), "unit.example.com") || fleet != nil { |
| + t.Fatalf("a state directory with an unreadable part: fleet %v, err %v; want ErrUnmeterable naming it and its tenant, and no fleet", fleet, err) |
| } |
| } |
| -// Close stops the store before it lets go of the witness, and a closed |
| -// fleet does not open again: the caller builds a new one. A fleet that |
| -// released its witness and left its store running would replicate beside |
| -// its replacement, over the same shadow state. |
| +// Close stops the store before it shuts the gate and lets go of the |
| +// witness, and a closed fleet does not open again: the caller builds a new |
| +// one. Closing a fleet that was current holds nobody's day. |
| func TestMeteredFleetCloseStopsTheStore(t *testing.T) { |
| - dir := t.TempDir() |
| - reg, err := registry.Open(filepath.Join(dir, "registry.db")) |
| - if err != nil { |
| - t.Fatal(err) |
| - } |
| - fleet, err := MeteredFleetStore(reg, dir, "replica-bucket", "metered/v1", "eu-west-1", nil, func(OwnedDB) lease.RequestRecorder { return nopRecorder{} }) |
| + bucket := newFakeBucket(t) |
| + dir := platformDir(t) |
| + reg := openRegistry(t, filepath.Join(dir, "registry.db")) |
| + recs := &recorders{} |
| + o := fleetOptions(reg, dir) |
| + o.Meter = recs.meter |
| + fleet, err := MeteredFleetStore(o) |
| if err != nil { |
| t.Fatal(err) |
| } |
| if fleet.Databases() != 1 { |
| t.Fatalf("%d databases before Close", fleet.Databases()) |
| } |
| + regPath := filepath.Join(dir, "registry.db") |
| + var client interface { |
| + WriteLTXFile(context.Context, int, ltx.TXID, ltx.TXID, io.Reader) (*ltx.FileInfo, error) |
| + } = fleet.store.DBs()[0].Replica.Client |
| if err := fleet.Close(); err != nil { |
| t.Fatalf("Close: %v", err) |
| } |
| if fleet.store != nil || fleet.Databases() != 0 { |
| t.Fatal("Close left the store in place") |
| } |
| + if _, _, inc := recs.of(regPath).counts(); inc != 0 { |
| + t.Fatal("closing a current fleet told a recorder its day is incomplete") |
| + } |
| + // A client somebody kept from before the close sends nothing. |
| + if _, err := client.WriteLTXFile(context.Background(), 0, 2, 2, bytes.NewReader(ltxBytes(t))); !errors.Is(err, lease.ErrRequestRefused) { |
| + t.Fatalf("a write through a closed fleet's client: %v", err) |
| + } |
| + if bucket.count() != 0 { |
| + t.Fatalf("requests after Close: %v", bucket.reqs) |
| + } |
| if err := fleet.Open(context.Background()); !errors.Is(err, ErrOwnershipStale) { |
| t.Fatalf("Open after Close: %v, want ErrOwnershipStale", err) |
| } |
| @@ -314,3 +696,319 @@ func TestMeteredFleetCloseStopsTheStore(t *testing.T) { |
| t.Fatalf("a second Close: %v", err) |
| } |
| } |
| + |
| +// Litestream makes a missing metadata directory with the owner and mode of |
| +// the database's own directory, which for a tenant's database is the |
| +// tenant's. The fleet makes them first: its own, closed to everyone else, |
| +// under a root it has checked. It refuses a root somebody else could have |
| +// prepared. |
| +func TestMeteredFleetKeepsShadowStateOutOfTenantsReach(t *testing.T) { |
| + private := func(t *testing.T, dir string) { |
| + t.Helper() |
| + fi, err := os.Lstat(dir) |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + facts, ok := factsOf(fi) |
| + if !fi.IsDir() || fi.Mode().Perm() != 0o700 || !ok || int(facts.UID) != os.Geteuid() { |
| + t.Fatalf("%s: mode %v, owner %d; want a directory of this process's, closed to everyone else", dir, fi.Mode(), facts.UID) |
| + } |
| + } |
| + t.Run("made private", func(t *testing.T) { |
| + dir := platformDir(t) |
| + reg := openRegistry(t, filepath.Join(dir, "registry.db")) |
| + state := filepath.Join(platformDir(t), "unit-state") |
| + db := writeFile(t, filepath.Join(state, "tenants", "t1.db")) |
| + // The tenant's directories are open to its group, as a state |
| + // directory may be: what litestream would copy. |
| + if err := os.Chmod(filepath.Join(state, "tenants"), 0o775); err != nil { |
| + t.Fatal(err) |
| + } |
| + upsert(t, reg, registry.Route{Host: "unit.example.com", Account: "bad", App: "unit", DBPath: state, Backing: registry.BackingUnit}) |
| + fleet, err := MeteredFleetStore(fleetOptions(reg, dir)) |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + defer fleet.Close() |
| + root := filepath.Join(dir, ".replicator") |
| + private(t, root) |
| + for _, sdb := range fleet.store.DBs() { |
| + meta := sdb.MetaPath() |
| + if !strings.HasPrefix(meta, root+string(filepath.Separator)) { |
| + t.Fatalf("%s keeps its shadow state at %s, outside %s", sdb.Path(), meta, root) |
| + } |
| + for at := meta; at != root; at = filepath.Dir(at) { |
| + private(t, at) |
| + } |
| + } |
| + if got := filepath.Join(root, replicaKey(db)); func() bool { _, err := os.Stat(got); return err != nil }() { |
| + t.Fatalf("no shadow directory for the tenant's database at %s", got) |
| + } |
| + }) |
| + for name, spoil := range map[string]func(t *testing.T, dir string){ |
| + "the root is a link": func(t *testing.T, dir string) { |
| + if err := os.Symlink(platformDir(t), filepath.Join(dir, ".replicator")); err != nil { |
| + t.Fatal(err) |
| + } |
| + }, |
| + "the root is open to others": func(t *testing.T, dir string) { |
| + if err := os.Mkdir(filepath.Join(dir, ".replicator"), 0o755); err != nil { |
| + t.Fatal(err) |
| + } |
| + }, |
| + "a directory beneath the root is open to others": func(t *testing.T, dir string) { |
| + if err := os.MkdirAll(filepath.Join(dir, ".replicator"), 0o700); err != nil { |
| + t.Fatal(err) |
| + } |
| + first := strings.Split(replicaKey(filepath.Join(dir, "registry.db")), string(filepath.Separator))[0] |
| + if err := os.Mkdir(filepath.Join(dir, ".replicator", first), 0o777); err != nil { |
| + t.Fatal(err) |
| + } |
| + if err := os.Chmod(filepath.Join(dir, ".replicator", first), 0o777); err != nil { |
| + t.Fatal(err) |
| + } |
| + }, |
| + "the data directory can be written by anyone": func(t *testing.T, dir string) { |
| + if err := os.Chmod(dir, 0o777); err != nil { |
| + t.Fatal(err) |
| + } |
| + }, |
| + } { |
| + t.Run(name, func(t *testing.T) { |
| + dir := platformDir(t) |
| + reg := openRegistry(t, filepath.Join(dir, "registry.db")) |
| + spoil(t, dir) |
| + fleet, err := MeteredFleetStore(fleetOptions(reg, dir)) |
| + if !errors.Is(err, ErrUnmeterable) || fleet != nil { |
| + t.Fatalf("fleet %v, err %v; want a refusal", fleet, err) |
| + } |
| + }) |
| + } |
| +} |
| + |
| +// The fleet looks at the files again on a timer. A database that is no |
| +// longer the file it was when its ownership was checked stops the fleet |
| +// and holds every recorder's day. A database that is new, or gone, leaves |
| +// it streaming and says it has to be rebuilt. |
| +func TestMeteredFleetLooksAtTheFilesAgain(t *testing.T) { |
| + type fixture struct { |
| + reg *registry.Registry |
| + state, db string |
| + } |
| + for name, tc := range map[string]struct { |
| + change func(t *testing.T, f fixture) |
| + want error |
| + }{ |
| + "nothing changes": {func(*testing.T, fixture) {}, nil}, |
| + "the tenant makes another database": {func(t *testing.T, f fixture) { |
| + writeFile(t, filepath.Join(f.state, "second.db")) |
| + }, ErrInventoryChanged}, |
| + "the tenant makes one discovery does not take": {func(t *testing.T, f fixture) { |
| + writeFile(t, filepath.Join(f.state, ".hidden", "x.db")) |
| + }, ErrInventoryChanged}, |
| + "the database is removed": {func(t *testing.T, f fixture) { |
| + if err := os.Remove(f.db); err != nil { |
| + t.Fatal(err) |
| + } |
| + }, ErrInventoryChanged}, |
| + "another file is put at the database's path": {func(t *testing.T, f fixture) { |
| + other := writeFile(t, filepath.Join(f.state, "other")) |
| + if err := os.Rename(other, f.db); err != nil { |
| + t.Fatal(err) |
| + } |
| + }, ErrOwnershipStale}, |
| + "the database becomes a link": {func(t *testing.T, f fixture) { |
| + elsewhere := writeFile(t, filepath.Join(platformDir(t), "elsewhere.db")) |
| + if err := os.Remove(f.db); err != nil { |
| + t.Fatal(err) |
| + } |
| + if err := os.Symlink(elsewhere, f.db); err != nil { |
| + t.Fatal(err) |
| + } |
| + }, ErrOwnershipStale}, |
| + "its -wal becomes a link": {func(t *testing.T, f fixture) { |
| + elsewhere := writeFile(t, filepath.Join(platformDir(t), "elsewhere.db-wal")) |
| + if err := os.Symlink(elsewhere, f.db+"-wal"); err != nil { |
| + t.Fatal(err) |
| + } |
| + }, ErrOwnershipStale}, |
| + "it gains a second name": {func(t *testing.T, f fixture) { |
| + if err := os.Link(f.db, filepath.Join(filepath.Dir(f.state), "second-name")); err != nil { |
| + t.Fatal(err) |
| + } |
| + }, ErrOwnershipStale}, |
| + } { |
| + t.Run(name, func(t *testing.T) { |
| + bucket := newFakeBucket(t) |
| + dir := platformDir(t) |
| + f := fixture{reg: openRegistry(t, filepath.Join(dir, "registry.db")), state: filepath.Join(platformDir(t), "unit-state")} |
| + f.db = writeFile(t, filepath.Join(f.state, "main.db")) |
| + upsert(t, f.reg, registry.Route{Host: "unit.example.com", Account: "bad", App: "unit", DBPath: f.state, Backing: registry.BackingUnit}) |
| + recs := &recorders{} |
| + o := fleetOptions(f.reg, dir) |
| + o.Meter = recs.meter |
| + fleet, err := MeteredFleetStore(o) |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + defer fleet.Close() |
| + fleet.lookAgain() |
| + if err := fleet.Current(context.Background()); err != nil { |
| + t.Fatalf("a second look at files nobody touched: %v", err) |
| + } |
| + tc.change(t, f) |
| + fleet.lookAgain() |
| + if err := fleet.Current(context.Background()); !errors.Is(err, tc.want) || (tc.want == nil && err != nil) { |
| + t.Fatalf("Current = %v, want %v", err, tc.want) |
| + } |
| + regPath := filepath.Join(dir, "registry.db") |
| + sendErr := send(t, fleet, regPath, 2) |
| + _, _, inc := recs.of(regPath).counts() |
| + if errors.Is(tc.want, ErrOwnershipStale) { |
| + if !errors.Is(sendErr, ErrOwnershipStale) || inc == 0 || bucket.count() != 0 { |
| + t.Fatalf("a write after a file changed under the fleet: %v, %d days held, %d requests; want a refusal and a hold", sendErr, inc, bucket.count()) |
| + } |
| + return |
| + } |
| + if sendErr != nil || inc != 0 || bucket.count() != 1 { |
| + t.Fatalf("a write while only the inventory has changed: %v, %d days held, %d requests; want it sent and nothing held", sendErr, inc, bucket.count()) |
| + } |
| + }) |
| + } |
| +} |
| + |
| +// The whole thing, opened: litestream itself replicating real databases |
| +// through the fleet's clients to a server that counts. |
| +// |
| +// It shows what the pieces above cannot: that the requests litestream makes |
| +// of its own accord are counted on the right recorder; that the replicator |
| +// writing its own bookkeeping into the registry it replicates does not |
| +// make the fleet stale; that the shadow state stays the platform's once |
| +// litestream has initialised it; that after a change of ownership nothing |
| +// more reaches the server while the databases go on being written; and |
| +// that Close returns with the store stopped. |
| +func TestMeteredFleetOpenedReplicatesCountsAndStops(t *testing.T) { |
| + bucket := newFakeBucket(t) |
| + prevWatch := ownershipWatchInterval |
| + ownershipWatchInterval = 20 * time.Millisecond |
| + t.Cleanup(func() { ownershipWatchInterval = prevWatch }) |
| + dir := platformDir(t) |
| + regPath := filepath.Join(dir, "registry.db") |
| + reg := openRegistry(t, regPath) |
| + db := filepath.Join(dir, "a.example.com.db") |
| + rawExec(t, db, `PRAGMA journal_mode=WAL`) |
| + rawExec(t, db, `CREATE TABLE t (id INTEGER PRIMARY KEY, v TEXT)`) |
| + upsert(t, reg, registry.Route{Host: "a.example.com", Account: "bac", App: "one", DBPath: db}) |
| + recs := &recorders{} |
| + o := fleetOptions(reg, dir) |
| + o.Meter, o.SyncInterval = recs.meter, 20*time.Millisecond |
| + fleet, err := MeteredFleetStore(o) |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + defer fleet.Close() |
| + if err := fleet.Open(context.Background()); err != nil { |
| + t.Fatal(err) |
| + } |
| + write := func() { rawExec(t, db, `INSERT INTO t (v) VALUES ('x')`) } |
| + eventually(t, "litestream's own writes for both databases to reach the bucket and be counted", func() bool { |
| + write() |
| + // A commit to the registry that is nobody's ownership, so the |
| + // registry has something to replicate too. |
| + if err := reg.SetNextDue("a.example.com", time.Now().UnixNano()); err != nil { |
| + t.Fatal(err) |
| + } |
| + tenant, _, _ := recs.of(db).counts() |
| + platform, _, _ := recs.of(regPath).counts() |
| + return bucket.puts("a.example.com.db") > 0 && bucket.puts("registry.db") > 0 && tenant > 0 && platform > 0 |
| + }) |
| + // Litestream has by now written its own tables into the registry. |
| + var seq int |
| + check, err := sql.Open("sqlite", regPath) |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + if err := check.QueryRow(`SELECT count(*) FROM sqlite_master WHERE name = '_litestream_seq'`).Scan(&seq); err != nil || seq != 1 { |
| + t.Fatalf("fixture: litestream's bookkeeping table in the registry: %d, %v", seq, err) |
| + } |
| + check.Close() |
| + if err := fleet.Current(context.Background()); err != nil { |
| + t.Fatalf("the fleet went stale on the replicator's own writes: %v", err) |
| + } |
| + for path, rec := range recs.all() { |
| + if _, _, inc := rec.counts(); inc != 0 { |
| + t.Fatalf("%s was told its day is incomplete while nothing had changed", path) |
| + } |
| + } |
| + // Every request at the bucket was counted by somebody. |
| + eventually(t, "the counts to settle", func() bool { return bucket.inFlight.Load() == 0 }) |
| + // The shadow state is still the platform's. |
| + root := filepath.Join(dir, ".replicator") |
| + for at := filepath.Join(root, replicaKey(db)); ; at = filepath.Dir(at) { |
| + fi, err := os.Lstat(at) |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + if facts, ok := factsOf(fi); !ok || int(facts.UID) != os.Geteuid() || fi.Mode().Perm() != 0o700 { |
| + t.Fatalf("%s after litestream initialised it: mode %v", at, fi.Mode()) |
| + } |
| + if at == root { |
| + break |
| + } |
| + } |
| + // The route changes hands. Nobody tells the fleet; the databases go on |
| + // being written. |
| + if err := reg.SetAccount("a.example.com", "bae"); err != nil { |
| + t.Fatal(err) |
| + } |
| + eventually(t, "the fleet to go stale", func() bool { |
| + return errors.Is(fleet.Current(context.Background()), ErrOwnershipStale) |
| + }) |
| + for path, rec := range recs.all() { |
| + if _, _, inc := rec.counts(); inc == 0 { |
| + t.Errorf("%s was not told its day is incomplete", path) |
| + } |
| + } |
| + // Requests admitted before the change may still be on their way. |
| + settled, since := bucket.count(), time.Now() |
| + eventually(t, "requests already admitted to land", func() bool { |
| + if n := bucket.count(); n != settled || bucket.inFlight.Load() != 0 { |
| + settled, since = n, time.Now() |
| + } |
| + return time.Since(since) > 500*time.Millisecond |
| + }) |
| + startedBefore := map[string]int{} |
| + for path, rec := range recs.all() { |
| + startedBefore[path], _, _ = rec.counts() |
| + } |
| + for i := 0; i < 20; i++ { |
| + write() |
| + time.Sleep(20 * time.Millisecond) |
| + } |
| + if n := bucket.count(); n != settled { |
| + t.Fatalf("%d requests reached the bucket after the fleet went stale", n-settled) |
| + } |
| + for path, rec := range recs.all() { |
| + if s, _, _ := rec.counts(); s != startedBefore[path] { |
| + t.Errorf("%s: %d requests were counted after the fleet went stale", path, s-startedBefore[path]) |
| + } |
| + } |
| + // Closing a stale fleet is prompt: its last sync is refused, and it is |
| + // not tried again for the whole of litestream's shutdown window. |
| + closing := time.Now() |
| + if err := fleet.Close(); err != nil { |
| + t.Logf("Close: %v", err) |
| + } |
| + if took := time.Since(closing); took > 15*time.Second { |
| + t.Fatalf("closing a stale fleet took %s", took) |
| + } |
| + if fleet.Databases() != 0 { |
| + t.Fatal("Close left the store in place") |
| + } |
| + after := bucket.count() |
| + write() |
| + time.Sleep(200 * time.Millisecond) |
| + if bucket.count() != after { |
| + t.Fatal("requests reached the bucket after Close") |
| + } |
| +} |
diff --git a/internal/ops/litestream_owners.go b/internal/ops/litestream_owners.go| index 11c633695..2b0381b8b 100644 |
| --- a/internal/ops/litestream_owners.go |
| +++ b/internal/ops/litestream_owners.go |
| @@ -1,6 +1,10 @@ |
| package ops |
| import ( |
| + "errors" |
| + "fmt" |
| + "io" |
| + "io/fs" |
| "os" |
| "path/filepath" |
| "sort" |
| @@ -33,40 +37,67 @@ const ( |
| PlatformConsoleHost = "console.platform.internal" |
| ) |
| -// FleetDBOwners is FleetDBs with an owner for each database, for the |
| +// FleetInventory is one strict reading of what the box replicates and who |
| +// pays for each database's replication writes. |
| +type FleetInventory struct { |
| + // Owned is every database with a single owner, sorted by path. |
| + Owned []OwnedDB |
| + // Unowned names the databases no single account can be shown to own, |
| + // each with why. They are named, never dropped and never guessed. |
| + Unowned []string |
| + // Skipped lists files that look like databases and are not replicated: |
| + // a file under the data directory that no route claims, and, beneath a |
| + // unit tenant's directory, a database in a dot-directory, deeper than |
| + // discovery goes, or under a name discovery does not recognise. A |
| + // replicator's caller has to name each one before it may start. |
| + Skipped []string |
| + // Problems are places a database could be and that could not be |
| + // looked at, or must not be followed: a directory that cannot be |
| + // read, a link in a tenant's tree. Each names the tenant's host. |
| + Problems []string |
| + |
| + facts map[string]fileFacts // by owned path: the file each was when it was checked |
| +} |
| + |
| +// FleetDBOwners reads the box's inventory from the registry as it is now. |
| +func FleetDBOwners(reg *registry.Registry, dataDir string) (FleetInventory, error) { |
| + routes, err := reg.List() |
| + if err != nil { |
| + return FleetInventory{}, err |
| + } |
| + return fleetInventory(routes, reg.Path(), dataDir) |
| +} |
| + |
| +// fleetInventory is FleetDBs with an owner for each database, for the |
| // platform's own replicator (path B in |
| // docs/superpowers/plans/2026-10-01-durability-attestation-path.md): every |
| // replica write has to be metered against an account, so every database has |
| -// to have one. |
| +// to have one. It works from routes the caller has already read, so that a |
| +// caller holding a witness discovers and attributes from the rows the |
| +// witness covers. |
| // |
| // A database is owned by the route whose DBPath is that file, or whose |
| // DBPath is a directory the file lies under (a unit tenant's tree). The |
| -// box's registry and console databases are the platform's. |
| +// box's registry (at the path it is open at) and console databases are the |
| +// platform's. |
| // |
| // Ownership is decided on the file, not on the spelling of its path. A unit |
| -// tenant owns its state directory and can put a symbolic link in it. A |
| -// link named like a database and pointing at another tenant's database, or |
| -// at the console's, would otherwise be listed under the tenant's directory, |
| +// tenant owns its state directory and can put a link in it. A link named |
| +// like a database and pointing at another tenant's database, or at the |
| +// console's, would otherwise be listed under the tenant's directory, |
| // claimed by the tenant's route as text, streamed into the tenant's replica |
| -// prefix and metered to the tenant. So a listed path is owned only if no |
| -// part of it beneath the claiming route's directory is a link, and no two |
| -// listed paths are the same file. |
| +// prefix and metered to the tenant. So every listed path is checked as a |
| +// file (checkOwnedFile), and no two listed paths may be one file. |
| // |
| -// unowned lists the databases FleetDBs would replicate that no single |
| -// account can be shown to own: a file two routes of different accounts |
| -// both claim, a route with no account, a path that reaches its file |
| -// through a link, or a file listed twice. They are named, never dropped |
| -// and never guessed. A replicator must not stream them unmetered, and a box |
| -// that replicates one cannot say that every write was counted. |
| -func FleetDBOwners(reg *registry.Registry, dataDir string) (owned []OwnedDB, unowned []string, skipped []string, err error) { |
| - dbs, skipped, err := FleetDBsWithSkipped(reg, dataDir) |
| +// Everything here is a look at one instant. See checkOwnedFile for what |
| +// that does not establish. |
| +func fleetInventory(routes []registry.Route, registryPath, dataDir string) (FleetInventory, error) { |
| + inv := FleetInventory{facts: map[string]fileFacts{}} |
| + dbs, skipped, problems, err := fleetDBsFromRoutes(routes, registryPath, dataDir, strictTreeDBs) |
| if err != nil { |
| - return nil, nil, nil, err |
| - } |
| - routes, err := reg.List() |
| - if err != nil { |
| - return nil, nil, nil, err |
| + return inv, err |
| } |
| + inv.Skipped, inv.Problems = skipped, problems |
| // The box's own databases, by what they are: the registry this |
| // process has open (wherever it lives), the conventional registry |
| // path, and the console's database. |
| @@ -74,41 +105,55 @@ func FleetDBOwners(reg *registry.Registry, dataDir string) (owned []OwnedDB, uno |
| filepath.Clean(filepath.Join(dataDir, "registry.db")): {Account: release.OpsAccount, App: PlatformRegistryApp, Host: PlatformRegistryHost, Platform: true}, |
| filepath.Clean(consoleDBPath(dataDir)): {Account: release.OpsAccount, App: PlatformConsoleApp, Host: PlatformConsoleHost, Platform: true}, |
| } |
| - if p := reg.Path(); p != "" { |
| - platform[filepath.Clean(p)] = OwnedDB{Account: release.OpsAccount, App: PlatformRegistryApp, Host: PlatformRegistryHost, Platform: true} |
| + if registryPath != "" { |
| + platform[filepath.Clean(registryPath)] = OwnedDB{Account: release.OpsAccount, App: PlatformRegistryApp, Host: PlatformRegistryHost, Platform: true} |
| } |
| - // The file each listed path is, by its resolved name. Two listed paths |
| - // that are one file are both unowned: one stream must never be written |
| - // twice, nor two owners billed for it. |
| - real := map[string]string{} |
| - times := map[string]int{} |
| + unowned := func(p, why string) { inv.Unowned = append(inv.Unowned, p+": "+why) } |
| + // The file each listed path is. Two listed paths that are one file are |
| + // both unowned: one stream must never be written twice, nor two |
| + // owners billed for it. |
| + type listed struct { |
| + path string |
| + fi os.FileInfo |
| + } |
| + var files []listed |
| for _, p := range dbs { |
| p = filepath.Clean(p) |
| - if r, rerr := filepath.EvalSymlinks(p); rerr == nil { |
| - real[p] = filepath.Clean(r) |
| - times[real[p]]++ |
| + fi, serr := os.Stat(p) |
| + if serr != nil { |
| + unowned(p, "it cannot be looked at") |
| + continue |
| } |
| + files = append(files, listed{p, fi}) |
| } |
| - for _, p := range dbs { |
| - p = filepath.Clean(p) |
| - if real[p] == "" || times[real[p]] > 1 { |
| - // It cannot be resolved, or another listed path is this file. |
| - unowned = append(unowned, p) |
| + for i, f := range files { |
| + p := f.path |
| + twice := false |
| + for j, other := range files { |
| + if i != j && os.SameFile(f.fi, other.fi) { |
| + twice = true |
| + break |
| + } |
| + } |
| + if twice { |
| + unowned(p, "another listed path is this same file") |
| continue |
| } |
| if own, ok := platform[p]; ok { |
| - if !noLinkBeneath(filepath.Dir(p), p) { |
| - unowned = append(unowned, p) |
| + facts, why := checkOwnedFile(filepath.Dir(p), p, platformDirectory) |
| + if why != "" { |
| + unowned(p, why) |
| continue |
| } |
| own.Path = p |
| - owned = append(owned, own) |
| + inv.Owned = append(inv.Owned, own) |
| + inv.facts[p] = facts |
| continue |
| } |
| // Every route that claims the file, exactly or by a directory above |
| - // it. The most specific claim wins only when it is the only one of |
| - // its kind; two accounts on one file is not something to choose |
| - // between. |
| + // it. Any number of routes may claim it as long as they all name |
| + // one account and one app; two accounts on one file is not |
| + // something to choose between. |
| var claims []registry.Route |
| for _, r := range routes { |
| if r.DBPath == "" { |
| @@ -121,26 +166,26 @@ func FleetDBOwners(reg *registry.Registry, dataDir string) (owned []OwnedDB, uno |
| } |
| owner, ok := singleOwner(claims) |
| if !ok { |
| - unowned = append(unowned, p) |
| + unowned(p, "no single account claims it") |
| continue |
| } |
| // The claim is on a directory the tenant can write in, or on the |
| - // file itself. Either way the path must lead to its file without a |
| - // link: from the claimed directory down for a unit tenant, and the |
| - // file itself for an exact claim. |
| - root := filepath.Clean(owner.DBPath) |
| + // file itself. |
| + root, kind := filepath.Clean(owner.DBPath), tenantTree |
| if root == p { |
| - root = filepath.Dir(p) |
| + root, kind = filepath.Dir(p), namedFile |
| } |
| - if !noLinkBeneath(root, p) { |
| - unowned = append(unowned, p) |
| + facts, why := checkOwnedFile(root, p, kind) |
| + if why != "" { |
| + unowned(p, why) |
| continue |
| } |
| - owned = append(owned, OwnedDB{Path: p, Account: owner.Account, App: owner.App, Host: owner.Host, Platform: owner.Account == release.OpsAccount}) |
| + inv.Owned = append(inv.Owned, OwnedDB{Path: p, Account: owner.Account, App: owner.App, Host: owner.Host, Platform: owner.Account == release.OpsAccount}) |
| + inv.facts[p] = facts |
| } |
| - sort.Slice(owned, func(i, j int) bool { return owned[i].Path < owned[j].Path }) |
| - sort.Strings(unowned) |
| - return owned, unowned, skipped, nil |
| + sort.Slice(inv.Owned, func(i, j int) bool { return inv.Owned[i].Path < inv.Owned[j].Path }) |
| + sort.Strings(inv.Unowned) |
| + return inv, nil |
| } |
| // singleOwner picks the route that owns a database from the routes that |
| @@ -164,63 +209,252 @@ func singleOwner(claims []registry.Route) (registry.Route, bool) { |
| return first, true |
| } |
| -// noLinkBeneath reports whether path leads to its file from root without |
| -// passing through a symbolic link: every component after root, the file |
| -// included, is what it is named and not a link to something else. root |
| -// itself may be reached however the operator arranged it (a data directory |
| -// behind a link is the operator's, not a tenant's). |
| -func noLinkBeneath(root, path string) bool { |
| +// How a database is claimed decides what has to be true of the directory |
| +// it is reached through. |
| +type claimKind int |
| + |
| +const ( |
| + // platformDirectory: the box's own database. The directory it is in |
| + // must be the platform's: owned by root or by this process's user, |
| + // and not writable by anyone else. |
| + platformDirectory claimKind = iota |
| + // tenantTree: a file beneath a directory a unit tenant owns. The |
| + // directory ABOVE the tenant's must be the platform's (or the tenant |
| + // could swap its whole tree for a link), and the file must belong to |
| + // the user the tree belongs to. |
| + tenantTree |
| + // namedFile: a route names the file itself. Nothing is asked of its |
| + // directory here: an exec instance's directory may be the tenant's. |
| + namedFile |
| +) |
| + |
| +// checkOwnedFile decides whether path, claimed from root, is a file its |
| +// claimant can be billed for. It answers with why not, or with the file's |
| +// identity for a later look to compare against. |
| +// |
| +// What it establishes, at the instant it looks: |
| +// |
| +// - every component from root down to the file is what it is named and |
| +// not a link, and none of them is on another filesystem than root (a |
| +// mount beneath a claim is somebody else's tree); |
| +// - the file is a regular file with exactly one name. A second name is a |
| +// hard link, and a hard link to a database nobody listed (one the |
| +// activator replicates, or any file outside discovery) would be |
| +// streamed and billed as the linking tenant's; |
| +// - beneath a tenant's tree, the file belongs to the user the tree |
| +// belongs to; |
| +// - its -wal, -shm and -journal siblings are absent or are regular files |
| +// with one name. SQLite and litestream open those by name, as root: a |
| +// -wal that is a link into another tenant's tree would have that |
| +// tenant's pages shipped into this one's replica and its WAL truncated |
| +// by this one's checkpoint; |
| +// - the directory the claim hangs from is the platform's where that can |
| +// be asked (claimKind). |
| +// |
| +// What it does not establish: any of that a moment later. A tenant who can |
| +// write the directory can replace the file, a sibling or a directory with |
| +// a link after this look, and litestream opens by path afterwards, for the |
| +// database, for its WAL on every sync, and for its shared memory. The |
| +// fleet looks again on a timer and stops if a file is no longer the file |
| +// it was, which narrows the window and does not close it. Closing it needs |
| +// litestream to open beneath a pinned root without following links |
| +// (openat2 with RESOLVE_BENEATH and RESOLVE_NO_SYMLINKS), which is a |
| +// change to litestream and is not made. |
| +func checkOwnedFile(root, path string, kind claimKind) (fileFacts, string) { |
| + rootInfo, err := os.Stat(root) |
| + if err != nil || !rootInfo.IsDir() { |
| + return fileFacts{}, "the directory it is claimed from cannot be looked at" |
| + } |
| + rootFacts, ok := factsOf(rootInfo) |
| + if !ok { |
| + return fileFacts{}, "file ownership cannot be read on this platform" |
| + } |
| + switch kind { |
| + case platformDirectory: |
| + if why := platformOwned(root); why != "" { |
| + return fileFacts{}, why |
| + } |
| + case tenantTree: |
| + // As spelled, and as resolved: a state directory may be reached |
| + // through a link the operator (or systemd) made, and both the |
| + // directory holding that link and the one holding the real tree |
| + // have to be out of the tenant's reach. |
| + if why := platformOwned(filepath.Dir(root)); why != "" { |
| + return fileFacts{}, why |
| + } |
| + if real, rerr := filepath.EvalSymlinks(root); rerr != nil { |
| + return fileFacts{}, "the directory it is claimed from cannot be resolved" |
| + } else if filepath.Clean(real) != filepath.Clean(root) { |
| + if why := platformOwned(filepath.Dir(real)); why != "" { |
| + return fileFacts{}, why |
| + } |
| + } |
| + } |
| rel, err := filepath.Rel(root, path) |
| if err != nil || rel == "." || rel == ".." || strings.HasPrefix(rel, ".."+string(filepath.Separator)) { |
| - return false |
| + return fileFacts{}, "it is not beneath the directory it is claimed from" |
| } |
| at := root |
| + var last os.FileInfo |
| for _, part := range strings.Split(rel, string(filepath.Separator)) { |
| at = filepath.Join(at, part) |
| - fi, err := os.Lstat(at) |
| - if err != nil || fi.Mode()&os.ModeSymlink != 0 { |
| - return false |
| + fi, lerr := os.Lstat(at) |
| + if lerr != nil { |
| + return fileFacts{}, "part of its path cannot be looked at" |
| + } |
| + if fi.Mode()&os.ModeSymlink != 0 { |
| + return fileFacts{}, "its path passes through a link (" + at + ")" |
| } |
| + f, ok := factsOf(fi) |
| + if !ok { |
| + return fileFacts{}, "file ownership cannot be read on this platform" |
| + } |
| + if f.Dev != rootFacts.Dev { |
| + return fileFacts{}, "its path crosses onto another filesystem (" + at + ")" |
| + } |
| + last = fi |
| + } |
| + if !last.Mode().IsRegular() { |
| + return fileFacts{}, "it is not a regular file" |
| + } |
| + facts, _ := factsOf(last) |
| + if facts.Nlink != 1 { |
| + return fileFacts{}, "it has more than one name (a hard link)" |
| } |
| - return true |
| + if kind == tenantTree && facts.UID != rootFacts.UID { |
| + return fileFacts{}, "it does not belong to the user its tenant's directory belongs to" |
| + } |
| + for _, suffix := range []string{"-wal", "-shm", "-journal"} { |
| + fi, lerr := os.Lstat(path + suffix) |
| + if errors.Is(lerr, fs.ErrNotExist) { |
| + continue |
| + } |
| + if lerr != nil { |
| + return fileFacts{}, "its " + suffix + " file cannot be looked at" |
| + } |
| + if !fi.Mode().IsRegular() { |
| + return fileFacts{}, "its " + suffix + " file is a link or not a regular file" |
| + } |
| + if f, ok := factsOf(fi); !ok || f.Nlink != 1 { |
| + return fileFacts{}, "its " + suffix + " file has more than one name (a hard link)" |
| + } |
| + } |
| + return facts, "" |
| } |
| -// unreadableUnder walks a unit tenant's state directory the way discovery |
| -// does (treeDBs: the same depth, dot names skipped) and names the first |
| -// directory that cannot be read. Discovery passes such a directory over in |
| -// silence, which is right for a config generator and wrong for a |
| -// replicator that has to say every database was counted: a directory it |
| -// cannot read may hold one. |
| -func unreadableUnder(root string) (string, bool) { |
| - var bad string |
| - var walk func(dir string, depth int) bool |
| - walk = func(dir string, depth int) bool { |
| - if depth > treeDBWalkDepth { |
| - return true |
| - } |
| +// platformOwned says why dir is not the platform's, or nothing: it must be |
| +// a real directory (not a link), owned by root or by this process's user, |
| +// and writable by nobody else. |
| +func platformOwned(dir string) string { |
| + fi, err := os.Lstat(dir) |
| + if err != nil || !fi.IsDir() { |
| + return fmt.Sprintf("%s is not a directory that can be looked at", dir) |
| + } |
| + f, ok := factsOf(fi) |
| + if !ok { |
| + return "file ownership cannot be read on this platform" |
| + } |
| + if f.UID != 0 && int(f.UID) != os.Geteuid() { |
| + return fmt.Sprintf("%s belongs to a user other than root or the platform's", dir) |
| + } |
| + if fi.Mode().Perm()&0o022 != 0 { |
| + return fmt.Sprintf("%s can be written by its group or by anyone", dir) |
| + } |
| + return "" |
| +} |
| + |
| +// strictTreeMaxDepth bounds strictTreeDBs. Past treeDBWalkDepth a database |
| +// is reported and not replicated; past this the walk says it cannot vouch |
| +// for what lies below. |
| +const strictTreeMaxDepth = 32 |
| + |
| +// strictTreeDBs walks a unit tenant's state directory for the platform's |
| +// own replicator. It finds what treeDBs finds (non-empty *.db files, no |
| +// deeper than treeDBWalkDepth, outside dot-directories) and, unlike it, |
| +// passes nothing over in silence, because a replicator that has to say |
| +// every database was counted cannot have places it did not look: |
| +// |
| +// - a database it would not replicate is reported in skipped: one in a |
| +// dot-directory, one deeper than treeDBWalkDepth, one under another |
| +// name SQLite files go by (.sqlite, .sqlite3, .db3), and a file with no |
| +// extension that begins with SQLite's header; |
| +// - a directory that cannot be read, and one deeper than |
| +// strictTreeMaxDepth, is a problem; |
| +// - a link is never followed. One that is named like a database, or that |
| +// leads to a directory, is a problem: the tenant's own software would |
| +// follow it, so there may be a database behind it. Any other link is |
| +// ignored. |
| +// |
| +// An entry that vanishes between being listed and being looked at is |
| +// ordinary churn (SQLite's journals come and go) and is ignored. An empty |
| +// *.db is a placeholder, as in treeDBs; the fleet's periodic look notices |
| +// when it stops being empty. |
| +func strictTreeDBs(root string) (dbs, skipped, problems []string) { |
| + var walk func(dir string, depth int, hidden bool) |
| + walk = func(dir string, depth int, hidden bool) { |
| entries, err := os.ReadDir(dir) |
| if err != nil { |
| - bad = dir |
| - return false |
| + if depth > 0 && errors.Is(err, fs.ErrNotExist) { |
| + return |
| + } |
| + problems = append(problems, dir+": cannot be read") |
| + return |
| } |
| for _, e := range entries { |
| - if strings.HasPrefix(e.Name(), ".") { |
| + name := e.Name() |
| + p := filepath.Join(dir, name) |
| + fi, err := os.Lstat(p) |
| + if errors.Is(err, fs.ErrNotExist) { |
| continue |
| } |
| - p := filepath.Join(dir, e.Name()) |
| - fi, err := os.Stat(p) |
| if err != nil { |
| - bad = p |
| - return false |
| + problems = append(problems, p+": cannot be looked at") |
| + continue |
| } |
| - if fi.IsDir() && !walk(p, depth+1) { |
| - return false |
| + switch { |
| + case fi.Mode()&os.ModeSymlink != 0: |
| + target, terr := os.Stat(p) |
| + if sqliteName(name) || (terr == nil && target.IsDir()) { |
| + problems = append(problems, p+": a link that may lead to a database") |
| + } |
| + case fi.IsDir(): |
| + if depth+1 > strictTreeMaxDepth { |
| + problems = append(problems, p+": too deep to walk") |
| + continue |
| + } |
| + walk(p, depth+1, hidden || strings.HasPrefix(name, ".") || depth+1 > treeDBWalkDepth) |
| + case !fi.Mode().IsRegular() || fi.Size() == 0: |
| + case strings.HasSuffix(name, ".db") && !hidden && !strings.HasPrefix(name, "."): |
| + dbs = append(dbs, p) |
| + case sqliteName(name) || (filepath.Ext(name) == "" && sqliteHeader(p)): |
| + skipped = append(skipped, p) |
| } |
| } |
| + } |
| + walk(root, 0, false) |
| + return dbs, skipped, problems |
| +} |
| + |
| +// sqliteName reports whether a file name is one SQLite databases go by. |
| +func sqliteName(name string) bool { |
| + switch filepath.Ext(name) { |
| + case ".db", ".sqlite", ".sqlite3", ".db3": |
| return true |
| } |
| - if walk(root, 0) { |
| - return "", false |
| + return false |
| +} |
| + |
| +// sqliteHeader reports whether the file begins with SQLite's header. It is |
| +// asked only of files with no extension. |
| +func sqliteHeader(path string) bool { |
| + f, err := os.Open(path) |
| + if err != nil { |
| + return false |
| + } |
| + defer f.Close() |
| + var head [16]byte |
| + if _, err := io.ReadFull(f, head[:]); err != nil { |
| + return false |
| } |
| - return bad, true |
| + return string(head[:]) == "SQLite format 3\x00" |
| } |
diff --git a/internal/ops/litestream_owners_test.go b/internal/ops/litestream_owners_test.go| index e9d461daa..4748c3cd8 100644 |
| --- a/internal/ops/litestream_owners_test.go |
| +++ b/internal/ops/litestream_owners_test.go |
| @@ -1,68 +1,117 @@ |
| package ops |
| import ( |
| - "errors" |
| - "github.com/carlosframework/platform/internal/lease" |
| "os" |
| "path/filepath" |
| + "strings" |
| "testing" |
| "github.com/carlosframework/platform/internal/registry" |
| "github.com/carlosframework/platform/internal/release" |
| + durability "github.com/carlosframework/platform/internal/usage" |
| ) |
| -// Every database the box replicates gets the account that pays for its |
| -// replication writes, or is named as having none. Nothing is guessed. |
| -func TestFleetDBOwnersNamesAnOwnerOrSaysThereIsNone(t *testing.T) { |
| +// platformDir is a temporary directory as the platform's directories are on |
| +// a box: its owner's, and writable by nobody else. t.TempDir alone follows |
| +// the umask, which on a developer's machine often leaves the group able to |
| +// write, and the ownership checks refuse a directory like that. |
| +func platformDir(t *testing.T) string { |
| + t.Helper() |
| dir := t.TempDir() |
| - reg, err := registry.Open(filepath.Join(dir, "registry.db")) |
| + if err := os.Chmod(dir, 0o755); err != nil { |
| + t.Fatal(err) |
| + } |
| + return dir |
| +} |
| + |
| +func writeFile(t *testing.T, p string) string { |
| + t.Helper() |
| + if err := os.MkdirAll(filepath.Dir(p), 0o755); err != nil { |
| + t.Fatal(err) |
| + } |
| + if err := os.WriteFile(p, []byte("x"), 0o644); err != nil { |
| + t.Fatal(err) |
| + } |
| + return p |
| +} |
| + |
| +func openRegistry(t *testing.T, path string) *registry.Registry { |
| + t.Helper() |
| + reg, err := registry.Open(path) |
| if err != nil { |
| t.Fatal(err) |
| } |
| - write := func(p string) string { |
| - t.Helper() |
| - if err := os.MkdirAll(filepath.Dir(p), 0o755); err != nil { |
| + t.Cleanup(func() { reg.Close() }) |
| + return reg |
| +} |
| + |
| +func upsert(t *testing.T, reg *registry.Registry, routes ...registry.Route) { |
| + t.Helper() |
| + for _, r := range routes { |
| + if err := reg.Upsert(r); err != nil { |
| t.Fatal(err) |
| } |
| - if err := os.WriteFile(p, []byte("x"), 0o644); err != nil { |
| - t.Fatal(err) |
| + } |
| +} |
| + |
| +// named reports whether list has an entry that begins with path and, when |
| +// why is given, mentions it. |
| +func named(list []string, path, why string) bool { |
| + for _, e := range list { |
| + if (e == path || strings.HasPrefix(e, path+":") || strings.HasPrefix(e, path+" ")) && strings.Contains(e, why) { |
| + return true |
| } |
| - return p |
| - } |
| - exec := write(filepath.Join(dir, "a.example.com.db")) |
| - state := filepath.Join(t.TempDir(), "unit-state") |
| - unitFile := write(filepath.Join(state, "main.db")) |
| - unitNested := write(filepath.Join(state, "tenants", "t1.db")) |
| - shared := write(filepath.Join(dir, "shared.example.com.db")) |
| - orphan := write(filepath.Join(dir, "no-account.example.com.db")) |
| - asleep := write(filepath.Join(dir, "asleep.example.com.db")) |
| - leftover := write(filepath.Join(dir, "leftover.db")) |
| - for _, r := range []registry.Route{ |
| - {Host: "a.example.com", Account: "bac", App: "one", DBPath: exec}, |
| - {Host: "unit.example.com", Account: "bad", App: "unit", DBPath: state, Backing: registry.BackingUnit}, |
| + } |
| + return false |
| +} |
| + |
| +func ownedByPath(inv FleetInventory) map[string]OwnedDB { |
| + got := map[string]OwnedDB{} |
| + for _, o := range inv.Owned { |
| + got[o.Path] = o |
| + } |
| + return got |
| +} |
| + |
| +// Every database the box replicates gets the account that pays for its |
| +// replication writes, or is named as having none. Nothing is guessed. |
| +func TestFleetDBOwnersNamesAnOwnerOrSaysThereIsNone(t *testing.T) { |
| + dir := platformDir(t) |
| + reg := openRegistry(t, filepath.Join(dir, "registry.db")) |
| + exec := writeFile(t, filepath.Join(dir, "a.example.com.db")) |
| + state := filepath.Join(platformDir(t), "unit-state") |
| + unitFile := writeFile(t, filepath.Join(state, "main.db")) |
| + unitNested := writeFile(t, filepath.Join(state, "tenants", "t1.db")) |
| + // A unit route that names one file, not a directory. |
| + single := writeFile(t, filepath.Join(platformDir(t), "single", "app.db")) |
| + shared := writeFile(t, filepath.Join(dir, "shared.example.com.db")) |
| + orphan := writeFile(t, filepath.Join(dir, "no-account.example.com.db")) |
| + asleep := writeFile(t, filepath.Join(dir, "asleep.example.com.db")) |
| + leftover := writeFile(t, filepath.Join(dir, "leftover.db")) |
| + console := writeFile(t, filepath.Join(dir, "console", "console.db")) |
| + upsert(t, reg, |
| + registry.Route{Host: "a.example.com", Account: "bac", App: "one", DBPath: exec}, |
| + registry.Route{Host: "unit.example.com", Account: "bad", App: "unit", DBPath: state, Backing: registry.BackingUnit}, |
| + registry.Route{Host: "single.example.com", Account: "baf", App: "single", DBPath: single, Backing: registry.BackingUnit}, |
| // Two accounts claim one file. |
| - {Host: "shared.example.com", Account: "bac", App: "one", DBPath: shared}, |
| - {Host: "shared2.example.com", Account: "bae", App: "two", DBPath: shared}, |
| + registry.Route{Host: "shared.example.com", Account: "bac", App: "one", DBPath: shared}, |
| + registry.Route{Host: "shared2.example.com", Account: "bae", App: "two", DBPath: shared}, |
| // A route with no account. |
| - {Host: "no-account.example.com", DBPath: orphan}, |
| + registry.Route{Host: "no-account.example.com", DBPath: orphan}, |
| // Hibernating: the activator replicates it, metered already. |
| - {Host: "asleep.example.com", Account: "bac", App: "one", DBPath: asleep, Hibernate: true}, |
| - } { |
| - if err := reg.Upsert(r); err != nil { |
| - t.Fatal(err) |
| - } |
| - } |
| - owned, unowned, skipped, err := FleetDBOwners(reg, dir) |
| + registry.Route{Host: "asleep.example.com", Account: "bac", App: "one", DBPath: asleep, Hibernate: true}, |
| + ) |
| + inv, err := FleetDBOwners(reg, dir) |
| if err != nil { |
| t.Fatal(err) |
| } |
| - got := map[string]OwnedDB{} |
| - for _, o := range owned { |
| - got[o.Path] = o |
| - } |
| - if o := got[filepath.Join(dir, "registry.db")]; o.Account != release.OpsAccount || !o.Platform { |
| + got := ownedByPath(inv) |
| + if o := got[filepath.Join(dir, "registry.db")]; o.Account != release.OpsAccount || !o.Platform || o.App != PlatformRegistryApp { |
| t.Errorf("the registry is not the platform's: %+v", o) |
| } |
| + if o := got[console]; o.Account != release.OpsAccount || !o.Platform || o.App != PlatformConsoleApp || o.Host != PlatformConsoleHost { |
| + t.Errorf("the console's database is not the platform's: %+v", o) |
| + } |
| if o := got[exec]; o.Account != "bac" || o.App != "one" || o.Host != "a.example.com" || o.Platform { |
| t.Errorf("an exec route's database: %+v", o) |
| } |
| @@ -71,108 +120,284 @@ func TestFleetDBOwnersNamesAnOwnerOrSaysThereIsNone(t *testing.T) { |
| t.Errorf("a file under a unit tenant's directory (%s): %+v", p, o) |
| } |
| } |
| - want := map[string]bool{shared: true, orphan: true} |
| - if len(unowned) != len(want) { |
| - t.Fatalf("unowned = %v, want the file two accounts claim and the file of a route with no account", unowned) |
| + if o := got[single]; o.Account != "baf" || o.App != "single" { |
| + t.Errorf("a unit route that names one file: %+v (unowned %v, problems %v)", o, inv.Unowned, inv.Problems) |
| } |
| - for _, p := range unowned { |
| - if !want[p] { |
| - t.Errorf("unowned names %s", p) |
| - } |
| - if _, ok := got[p]; ok { |
| - t.Errorf("%s is both owned and unowned", p) |
| - } |
| + if len(inv.Unowned) != 2 || !named(inv.Unowned, shared, "no single account") || !named(inv.Unowned, orphan, "no single account") { |
| + t.Fatalf("unowned = %v, want the file two accounts claim and the file of a route with no account", inv.Unowned) |
| } |
| if _, ok := got[asleep]; ok { |
| t.Error("a hibernating route's database is listed: the activator replicates it") |
| } |
| - if len(skipped) != 1 || skipped[0] != leftover { |
| - t.Errorf("skipped = %v, want the leftover file no route claims", skipped) |
| + if len(inv.Skipped) != 1 || inv.Skipped[0] != leftover { |
| + t.Errorf("skipped = %v, want the leftover file no route claims", inv.Skipped) |
| + } |
| + if len(inv.Problems) != 0 { |
| + t.Errorf("problems = %v, want none", inv.Problems) |
| } |
| // Everything FleetDBs replicates is either owned or named as unowned. |
| dbs, err := FleetDBs(reg, dir) |
| if err != nil { |
| t.Fatal(err) |
| } |
| - if len(dbs) != len(owned)+len(unowned) { |
| - t.Fatalf("FleetDBs lists %d databases; %d owned and %d unowned", len(dbs), len(owned), len(unowned)) |
| + if len(dbs) != len(inv.Owned)+len(inv.Unowned) { |
| + t.Fatalf("FleetDBs lists %d databases; %d owned and %d unowned", len(dbs), len(inv.Owned), len(inv.Unowned)) |
| + } |
| + // The identities the box's own databases are metered under are ones |
| + // the real durability ledger accepts for the platform's purpose, the |
| + // console's as well as the registry's. |
| + platform := 0 |
| + for _, o := range inv.Owned { |
| + if !o.Platform { |
| + continue |
| + } |
| + platform++ |
| + attr := durability.ReplicationAttribution{Account: o.Account, App: o.App, Host: o.Host, Purpose: durability.PurposePlatform} |
| + if err := attr.Validate(); err != nil { |
| + t.Errorf("%s: the ledger refuses the platform's identity %+v: %v", o.Path, attr, err) |
| + } |
| + } |
| + if platform != 2 { |
| + t.Fatalf("%d platform databases, want the registry and the console", platform) |
| + } |
| +} |
| + |
| +// The registry is the platform's at the path it is open at, under the data |
| +// directory or outside it, and is listed there: it is not passed over as |
| +// somebody's leftover, and it does not need naming as accepted. |
| +func TestFleetDBOwnersFindTheRegistryWhereItIsOpen(t *testing.T) { |
| + for name, at := range map[string]func(dataDir string) string{ |
| + "another name in the data directory": func(dataDir string) string { return filepath.Join(dataDir, "fleet.db") }, |
| + "outside the data directory": func(string) string { return filepath.Join(platformDir(t), "elsewhere", "reg.db") }, |
| + } { |
| + t.Run(name, func(t *testing.T) { |
| + dir := platformDir(t) |
| + path := at(dir) |
| + if err := os.MkdirAll(filepath.Dir(path), 0o755); err != nil { |
| + t.Fatal(err) |
| + } |
| + reg := openRegistry(t, path) |
| + inv, err := FleetDBOwners(reg, dir) |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + o, ok := ownedByPath(inv)[path] |
| + if !ok || !o.Platform || o.App != PlatformRegistryApp { |
| + t.Fatalf("the registry at %s: owned %+v, skipped %v, unowned %v", path, inv.Owned, inv.Skipped, inv.Unowned) |
| + } |
| + if len(inv.Skipped) != 0 { |
| + t.Fatalf("skipped = %v: the registry was passed over", inv.Skipped) |
| + } |
| + }) |
| } |
| } |
| // A unit tenant owns its state directory and can put a link in it. A link |
| -// named like a database and pointing at another tenant's database, or at |
| -// the platform's own, must not be streamed and metered as the tenant's: it |
| -// is another owner's data and another owner's stream. Such a path is |
| -// unowned, and so is the real file when it is listed too (one file, two |
| -// paths). The replicator then refuses to start. |
| +// is never followed: one named like a database, or one that leads to a |
| +// directory, is reported as a place a database could be that cannot be |
| +// vouched for, and the replicator refuses to start. Another tenant's |
| +// database reached through such a link is not listed under this tenant at |
| +// all, so it stays the other tenant's and is listed once. |
| func TestFleetDBOwnersDoesNotFollowATenantsLink(t *testing.T) { |
| - dir := t.TempDir() |
| - reg, err := registry.Open(filepath.Join(dir, "registry.db")) |
| - if err != nil { |
| - t.Fatal(err) |
| - } |
| - write := func(p string) string { |
| - t.Helper() |
| - if err := os.MkdirAll(filepath.Dir(p), 0o755); err != nil { |
| - t.Fatal(err) |
| - } |
| - if err := os.WriteFile(p, []byte("x"), 0o644); err != nil { |
| - t.Fatal(err) |
| - } |
| - return p |
| - } |
| - stateA := filepath.Join(t.TempDir(), "unit-a") |
| - stateB := filepath.Join(t.TempDir(), "unit-b") |
| - ownA := write(filepath.Join(stateA, "main.db")) |
| - theirs := write(filepath.Join(stateB, "main.db")) |
| - // Not a database by name, so it is not listed by itself; reached only |
| - // through the link. |
| - outside := write(filepath.Join(t.TempDir(), "elsewhere", "secrets.sqlite")) |
| + dir := platformDir(t) |
| + reg := openRegistry(t, filepath.Join(dir, "registry.db")) |
| + stateA := filepath.Join(platformDir(t), "unit-a") |
| + stateB := filepath.Join(platformDir(t), "unit-b") |
| + ownA := writeFile(t, filepath.Join(stateA, "main.db")) |
| + theirs := writeFile(t, filepath.Join(stateB, "main.db")) |
| + outside := writeFile(t, filepath.Join(platformDir(t), "elsewhere", "secrets.sqlite")) |
| + notes := writeFile(t, filepath.Join(stateA, "notes.txt")) |
| toB := filepath.Join(stateA, "link-to-b.db") |
| toOutside := filepath.Join(stateA, "link-outside.db") |
| linkedDir := filepath.Join(stateA, "linked") |
| - for target, link := range map[string]string{theirs: toB, outside: toOutside, stateB: linkedDir} { |
| + harmless := filepath.Join(stateA, "latest-notes") |
| + dangling := filepath.Join(stateA, "gone") |
| + for link, target := range map[string]string{toB: theirs, toOutside: outside, linkedDir: stateB, harmless: notes, dangling: filepath.Join(stateA, "nothing-here")} { |
| if err := os.Symlink(target, link); err != nil { |
| t.Fatal(err) |
| } |
| } |
| - for _, r := range []registry.Route{ |
| - {Host: "a.example.com", Account: "bac", App: "one", DBPath: stateA, Backing: registry.BackingUnit}, |
| - {Host: "b.example.com", Account: "bae", App: "two", DBPath: stateB, Backing: registry.BackingUnit}, |
| - } { |
| - if err := reg.Upsert(r); err != nil { |
| - t.Fatal(err) |
| + upsert(t, reg, |
| + registry.Route{Host: "a.example.com", Account: "bac", App: "one", DBPath: stateA, Backing: registry.BackingUnit}, |
| + registry.Route{Host: "b.example.com", Account: "bae", App: "two", DBPath: stateB, Backing: registry.BackingUnit}, |
| + ) |
| + inv, err := FleetDBOwners(reg, dir) |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + got := ownedByPath(inv) |
| + if o := got[ownA]; o.Account != "bac" { |
| + t.Errorf("the tenant's own database: %+v", o) |
| + } |
| + if o := got[theirs]; o.Account != "bae" { |
| + t.Errorf("the other tenant's database, listed once under its own route: %+v (unowned %v)", o, inv.Unowned) |
| + } |
| + for _, p := range []string{toB, toOutside, linkedDir} { |
| + if _, ok := got[p]; ok { |
| + t.Errorf("%s is a link and is owned", p) |
| } |
| + if !named(inv.Problems, p, "a.example.com") { |
| + t.Errorf("%s is a link that may lead to a database and is not reported with its tenant (problems: %v)", p, inv.Problems) |
| + } |
| + } |
| + if len(inv.Problems) != 3 { |
| + t.Errorf("problems = %v, want the three links and nothing else: a link to an ordinary file and a dangling one are not places a database could be", inv.Problems) |
| + } |
| + if _, err := MeteredFleetStore(fleetOptions(reg, dir)); err == nil || !strings.Contains(err.Error(), toB) { |
| + t.Fatalf("the replicator over a tenant's link: %v, want a refusal naming it", err) |
| } |
| - owned, unowned, _, err := FleetDBOwners(reg, dir) |
| +} |
| + |
| +// A route that names a file which is itself a link is not the owner of |
| +// what the link leads to. |
| +func TestFleetDBOwnersRefuseANamedFileThatIsALink(t *testing.T) { |
| + dir := platformDir(t) |
| + reg := openRegistry(t, filepath.Join(dir, "registry.db")) |
| + real := writeFile(t, filepath.Join(platformDir(t), "elsewhere", "console-copy.db")) |
| + link := filepath.Join(dir, "a.example.com.db") |
| + if err := os.Symlink(real, link); err != nil { |
| + t.Fatal(err) |
| + } |
| + upsert(t, reg, registry.Route{Host: "a.example.com", Account: "bac", App: "one", DBPath: link}) |
| + inv, err := FleetDBOwners(reg, dir) |
| if err != nil { |
| t.Fatal(err) |
| } |
| - isUnowned := map[string]bool{} |
| - for _, p := range unowned { |
| - isUnowned[p] = true |
| + if _, ok := ownedByPath(inv)[link]; ok || !named(inv.Unowned, link, "link") { |
| + t.Fatalf("a named file that is a link: owned %+v, unowned %v", inv.Owned, inv.Unowned) |
| + } |
| +} |
| + |
| +// What has to be true of a file before a tenant is billed for it, beyond |
| +// its path not passing through a link. Each case is one fact made false. |
| +func TestFleetDBOwnersCheckTheFileAndItsSiblings(t *testing.T) { |
| + type fixture struct { |
| + state, db, other string |
| } |
| - for _, o := range owned { |
| - switch { |
| - case o.Path == ownA: |
| - if o.Account != "bac" { |
| - t.Errorf("the tenant's own database: %+v", o) |
| + for name, tc := range map[string]struct { |
| + spoil func(t *testing.T, f fixture) |
| + why string |
| + }{ |
| + "a second name for the database (a hard link)": {func(t *testing.T, f fixture) { |
| + if err := os.Link(f.db, filepath.Join(filepath.Dir(f.state), "second-name")); err != nil { |
| + t.Fatal(err) |
| } |
| - case o.Platform: |
| - default: |
| - t.Errorf("owned: %+v; only the tenant's own database and the platform's may be owned here", o) |
| - } |
| + }, "more than one name"}, |
| + "the database is a hard link to a file nobody listed": {func(t *testing.T, f fixture) { |
| + if err := os.Remove(f.db); err != nil { |
| + t.Fatal(err) |
| + } |
| + if err := os.Link(f.other, f.db); err != nil { |
| + t.Fatal(err) |
| + } |
| + }, "more than one name"}, |
| + "its -wal is a link into another tree": {func(t *testing.T, f fixture) { |
| + if err := os.Symlink(f.other+"-wal", f.db+"-wal"); err != nil { |
| + t.Fatal(err) |
| + } |
| + }, "-wal"}, |
| + "its -shm is a link": {func(t *testing.T, f fixture) { |
| + if err := os.Symlink(f.other, f.db+"-shm"); err != nil { |
| + t.Fatal(err) |
| + } |
| + }, "-shm"}, |
| + "its -journal is a directory": {func(t *testing.T, f fixture) { |
| + if err := os.Mkdir(f.db+"-journal", 0o755); err != nil { |
| + t.Fatal(err) |
| + } |
| + }, "-journal"}, |
| + "its -wal has a second name": {func(t *testing.T, f fixture) { |
| + writeFile(t, f.other+"-wal") |
| + if err := os.Link(f.other+"-wal", f.db+"-wal"); err != nil { |
| + t.Fatal(err) |
| + } |
| + }, "-wal"}, |
| + "the directory above the tenant's can be written by anyone": {func(t *testing.T, f fixture) { |
| + if err := os.Chmod(filepath.Dir(f.state), 0o777); err != nil { |
| + t.Fatal(err) |
| + } |
| + }, "can be written"}, |
| + } { |
| + t.Run(name, func(t *testing.T) { |
| + dir := platformDir(t) |
| + reg := openRegistry(t, filepath.Join(dir, "registry.db")) |
| + f := fixture{state: filepath.Join(platformDir(t), "unit-a")} |
| + f.db = writeFile(t, filepath.Join(f.state, "main.db")) |
| + // Somebody else's file, in a place discovery never looks. |
| + f.other = writeFile(t, filepath.Join(filepath.Dir(f.state), "not-listed", "theirs.db")) |
| + upsert(t, reg, registry.Route{Host: "a.example.com", Account: "bac", App: "one", DBPath: f.state, Backing: registry.BackingUnit}) |
| + // Sound before it is spoiled: with ordinary siblings present. |
| + writeFile(t, f.db+"-wal") |
| + inv, err := FleetDBOwners(reg, dir) |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + if _, ok := ownedByPath(inv)[f.db]; !ok { |
| + t.Fatalf("fixture: the database is not owned before the case spoils it (unowned %v, problems %v)", inv.Unowned, inv.Problems) |
| + } |
| + if err := os.Remove(f.db + "-wal"); err != nil { |
| + t.Fatal(err) |
| + } |
| + tc.spoil(t, f) |
| + inv, err = FleetDBOwners(reg, dir) |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + if _, ok := ownedByPath(inv)[f.db]; ok || !named(inv.Unowned, f.db, tc.why) { |
| + t.Fatalf("owned %+v, unowned %v; want the database unowned, saying %q", inv.Owned, inv.Unowned, tc.why) |
| + } |
| + }) |
| } |
| - for _, p := range []string{toB, toOutside} { |
| - if !isUnowned[p] { |
| - t.Errorf("%s is a link out of the tenant's directory and is not named as unowned (unowned: %v)", p, unowned) |
| +} |
| + |
| +// The strict walk reports what the config generator's walk passes over: a |
| +// database in a dot-directory, deeper than discovery goes, or under another |
| +// name SQLite files go by. Each has to be named by the caller before the |
| +// replicator starts; none is replicated, and none is dropped unseen. |
| +func TestStrictTreeWalkReportsWhatDiscoveryPassesOver(t *testing.T) { |
| + root := filepath.Join(platformDir(t), "state") |
| + plain := writeFile(t, filepath.Join(root, "main.db")) |
| + hidden := writeFile(t, filepath.Join(root, ".cache", "hidden.db")) |
| + deep := writeFile(t, filepath.Join(root, "a", "b", "c", "d", "e", "f", "g", "h", "i", "deep.db")) |
| + other := writeFile(t, filepath.Join(root, "data.sqlite3")) |
| + bare := filepath.Join(root, "store") |
| + if err := os.WriteFile(bare, append([]byte("SQLite format 3\x00"), make([]byte, 100)...), 0o644); err != nil { |
| + t.Fatal(err) |
| + } |
| + writeFile(t, filepath.Join(root, "README")) |
| + writeFile(t, filepath.Join(root, "main.db-wal")) |
| + if err := os.WriteFile(filepath.Join(root, "empty.db"), nil, 0o644); err != nil { |
| + t.Fatal(err) |
| + } |
| + dbs, skipped, problems := strictTreeDBs(root) |
| + if len(dbs) != 1 || dbs[0] != plain { |
| + t.Errorf("dbs = %v, want only %s", dbs, plain) |
| + } |
| + want := map[string]bool{hidden: true, deep: true, other: true, bare: true} |
| + if len(skipped) != len(want) { |
| + t.Errorf("skipped = %v, want %v", skipped, want) |
| + } |
| + for _, p := range skipped { |
| + if !want[p] { |
| + t.Errorf("skipped names %s", p) |
| } |
| } |
| - // B's database is one file under two listed paths. |
| - if !isUnowned[theirs] { |
| - t.Errorf("a file listed twice (once through a link) is not named as unowned: %v", unowned) |
| + if len(problems) != 0 { |
| + t.Errorf("problems = %v", problems) |
| + } |
| + // The lenient walk sees only the plain one, which is the point. |
| + if lenient := treeDBs(root); len(lenient) != 1 || lenient[0] != plain { |
| + t.Errorf("treeDBs = %v", lenient) |
| } |
| - if _, err := MeteredFleetStore(reg, dir, "test-bucket", "replicas", "eu-west-1", nil, func(OwnedDB) lease.RequestRecorder { return nopRecorder{} }); !errors.Is(err, ErrUnmeterable) { |
| - t.Fatalf("the replicator over a tenant's link: %v, want ErrUnmeterable", err) |
| + if os.Geteuid() != 0 { |
| + locked := filepath.Join(root, "locked") |
| + writeFile(t, filepath.Join(locked, "x.db")) |
| + if err := os.Chmod(locked, 0o000); err != nil { |
| + t.Fatal(err) |
| + } |
| + t.Cleanup(func() { os.Chmod(locked, 0o755) }) |
| + if _, _, problems := strictTreeDBs(root); len(problems) != 1 || !strings.HasPrefix(problems[0], locked) { |
| + t.Errorf("a directory that cannot be read: problems = %v", problems) |
| + } |
| } |
| } |