amadan
Required coverage (plan Task 3): RequiredDatabases (the registry this process has open, the console database when CONSOLE_DB names it; no role configuration exists on this branch to name a console box, recorded), each looked at through its root in fleetInventory with why it cannot be reached, an unreachable one a problem and, at a fleet build, the options' Hold told the instant before the build is refused; carlos ops inventory, the read-only command rendering the external replicator's configured list beside the platform replicator's inventory and the required databases' reachability
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
6 files changed,
+240
−2
docs/superpowers/plans/2026-10-03-replicator-authority-plan.md+2 −2internal/ops/authoritybridge.go+3 −0internal/ops/litestream_metered.go+10 −0internal/ops/litestream_owners.go+57 −0internal/ops/litestream_owners_unix_test.go+71 −0internal/ops/ops.go+97 −0
CI passed — run details
diff --git a/docs/superpowers/plans/2026-10-03-replicator-authority-plan.md b/docs/superpowers/plans/2026-10-03-replicator-authority-plan.md| index 7d81d6544..6424da614 100644 |
| --- a/docs/superpowers/plans/2026-10-03-replicator-authority-plan.md |
| +++ b/docs/superpowers/plans/2026-10-03-replicator-authority-plan.md |
| @@ -22,9 +22,9 @@ Date: 2026-10-03. For `docs/superpowers/specs/2026-10-03-replicator-pinned-roots |
| **Tests.** Every existing owners test keeps passing with its strings; added, from the design's Part A list at the inventory level: each link class rendered as written and none silent (a directory target, a database-named link, a regular file in the tree, a regular file under another pin, a regular file in a sibling tenant under the same pin, an absolute target inside the tenant's own tree (skipped, not out of the tree), a link to a link, a FIFO target, an absent target, an ascent past the tenant root, an absolute target outside every pin, a non-platform component in an absolute target); a tenant root reached through systemd's relative link shape and through an absolute one; Astra 2's layout walked, with the root above the tenant-owned component and the chain beneath retained; a swap of the tenant directory between the listing and the classification changing nothing; the retained chain's `Recheck` at the next look reporting a moved or replaced directory; the descriptor count per fleet equal to the retained stacks' total. |
| -## Task 3: required coverage (Part D) |
| +## Task 3 (built): required coverage (Part D) |
| -The platform's required databases as a list (the registry this process has open, the console database when the role configuration says console box, `CONSOLE_DB` when set), each resolved through its root like any claim; one unreachable through its root is a problem that makes the inventory incomplete and is written as a durable box-wide hold for the day with today's `DurabilityClient.Hold` (box-wide is the default; owner scope is decision 10), before and independent of whether a fleet is built. A read-only operator command (`carlos ops replication inventory`, name subject to the operator CLI's conventions) lists the inventory beside the external replicator's configuration for the cutover comparison. Tests: a required database missing, replaced by a link, and reachable; the hold written once per day; the command's output against a fixture. |
| +`RequiredDatabases(registryPath, dataDir)`: the registry this process has open and the console database when `CONSOLE_DB` names it. The design also names the console database when the box's role configuration says console box; no such role configuration exists on this branch, so `CONSOLE_DB` is the only way a console database is required here, and that gap is recorded for the owner with the deployment support conditions. `fleetInventory` looks at each required database through its root (`lookAt`: the retained chain, then `Lookat` of the name; a link in its place is "it is a link") and records `Required` with why each is unreachable; an unreachable one is a problem (the inventory is incomplete) and `MeteredFleetStore` tells the options' `Hold` the instant (the caller binds it to the durability ledger's `Incomplete`, today's box-wide durable marker; owner scope is decision 10) before it refuses the build. `carlos ops inventory` is the read-only operator command: the external replicator's configured list beside the platform replicator's inventory (owned, skipped, unowned, problems, each side's differences) and the required databases with their reachability; it builds no fleet and writes nothing. Tests: every required database reachable (no problem, the report says reachable); a link in the console database's place (missing, a problem, the report says MISSING); missing altogether (a fleet build holds the day once and is refused); restored; without `CONSOLE_DB` only the registry is required. |
| ## Task 4: Part C's ledger changes (after the model) |
diff --git a/internal/ops/authoritybridge.go b/internal/ops/authoritybridge.go| index 69b181708..fedacf9ba 100644 |
| --- a/internal/ops/authoritybridge.go |
| +++ b/internal/ops/authoritybridge.go |
| @@ -467,5 +467,8 @@ func (p *fleetPins) lookAt(path string) (authority.Facts, string) { |
| if err != nil || k == authority.Absent { |
| return authority.Facts{}, "it cannot be looked at" |
| } |
| + if k == authority.Link { |
| + return authority.Facts{}, "it is a link" |
| + } |
| return f, "" |
| } |
diff --git a/internal/ops/litestream_metered.go b/internal/ops/litestream_metered.go| index f84f0a9e7..c54b98072 100644 |
| --- a/internal/ops/litestream_metered.go |
| +++ b/internal/ops/litestream_metered.go |
| @@ -98,6 +98,13 @@ type MeteredFleetOptions struct { |
| // the box's (the durability ledger's is). The fleet does not rely on |
| // it: when ownership goes stale it tells every recorder it was given. |
| Meter func(OwnedDB) lease.RequestRecorder |
| + // Hold, when set, is told the instant at which a required database |
| + // (RequiredDatabases) could not be reached through its root while the |
| + // fleet was being built: the caller binds it to the box's durability |
| + // ledger (DurabilityClient.Incomplete), so the day is held, durably |
| + // and box-wide, whether or not a fleet is then built. The build is |
| + // refused as well, since the inventory is incomplete. |
| + Hold func(at time.Time) |
| // Replaces is the fleet this one is built to take over from, if any. |
| // The old fleet must be closed before this one is opened, and closing |
| // it looks at the files and asks the witness one last time each (a |
| @@ -890,6 +897,9 @@ func MeteredFleetStore(o MeteredFleetOptions) (fleet *MeteredFleet, err error) { |
| inv.Close() |
| return nil, err |
| } |
| + if missing := inv.RequiredMissing(); len(missing) > 0 && o.Hold != nil { |
| + o.Hold(time.Now()) |
| + } |
| if len(inv.Problems) > 0 { |
| inv.Close() |
| return nil, fmt.Errorf("%w: places a database could be that cannot be vouched for: %s", ErrUnmeterable, quotedList(inv.Problems)) |
diff --git a/internal/ops/litestream_owners.go b/internal/ops/litestream_owners.go| index f258ad649..93173e7cb 100644 |
| --- a/internal/ops/litestream_owners.go |
| +++ b/internal/ops/litestream_owners.go |
| @@ -42,6 +42,12 @@ const ( |
| // FleetInventory is one strict reading of what the box replicates and who |
| // pays for each database's replication writes. |
| type FleetInventory struct { |
| + // Required is every database the platform requires to be covered |
| + // (RequiredDatabases), each with why it cannot be reached through its |
| + // root, or "" when it can. One that cannot be reached is also a |
| + // problem: the inventory is incomplete, and the box's day is held |
| + // (MeteredFleetOptions.Hold) whether or not a fleet is built. |
| + Required []RequiredDB |
| // pins are the pinned slash and the platform roots this inventory was |
| // walked from; the chains of the owned files (facts[p].keep) are |
| // retained beneath them until Close. A caller that keeps the inventory |
| @@ -78,6 +84,42 @@ func FleetDBOwners(reg *registry.Registry, dataDir string) (FleetInventory, erro |
| return inv, err |
| } |
| +// RequiredDB is one database the platform requires to be covered, and why |
| +// it is not reachable through its root ("" when it is). |
| +type RequiredDB struct { |
| + Path, Why string |
| +} |
| + |
| +// RequiredDatabases is the platform's own list: the registry this process |
| +// has open (registryPath, when known) and the console database when |
| +// CONSOLE_DB names it. The design also names the console database when |
| +// the box's role configuration says console box; no such role |
| +// configuration exists on this branch, so CONSOLE_DB is the only way a |
| +// console database is required here, and the conventional path is only |
| +// listed when it exists (as fleetInventory lists it today). |
| +func RequiredDatabases(registryPath, dataDir string) []string { |
| + var out []string |
| + if registryPath != "" { |
| + out = append(out, filepath.Clean(registryPath)) |
| + } |
| + if p := strings.TrimSpace(os.Getenv("CONSOLE_DB")); p != "" { |
| + out = append(out, filepath.Clean(p)) |
| + } |
| + return out |
| +} |
| + |
| +// RequiredMissing is the paths of the required databases that cannot be |
| +// reached through their roots. |
| +func (inv *FleetInventory) RequiredMissing() []string { |
| + var out []string |
| + for _, r := range inv.Required { |
| + if r.Why != "" { |
| + out = append(out, r.Path) |
| + } |
| + } |
| + return out |
| +} |
| + |
| // Close releases every retained chain and the pins. The facts stay, their |
| // chains gone; a second Close is harmless. |
| func (inv *FleetInventory) Close() { |
| @@ -176,6 +218,21 @@ func fleetInventory(routes []registry.Route, registryPath, dataDir string) (Flee |
| return inv, err |
| } |
| inv.Skipped, inv.Problems = skipped, problems |
| + // Required coverage: each required database must be reachable through |
| + // its root as a regular file, before and independent of whether a |
| + // fleet is built over the rest. |
| + for _, p := range RequiredDatabases(registryPath, dataDir) { |
| + why := "" |
| + if id, lerr := pins.lookAt(p); lerr != "" { |
| + why = lerr |
| + } else if id.Mode&syscall.S_IFMT != syscall.S_IFREG { |
| + why = "it is not a regular file" |
| + } |
| + inv.Required = append(inv.Required, RequiredDB{Path: p, Why: why}) |
| + if why != "" { |
| + inv.Problems = append(inv.Problems, p+": a required database cannot be reached through its root ("+why+")") |
| + } |
| + } |
| // The box's own databases, by what they are: the registry this |
| // process has open (wherever it lives), the conventional registry |
| // path, and the console's database. |
diff --git a/internal/ops/litestream_owners_unix_test.go b/internal/ops/litestream_owners_unix_test.go| index 8ad9cf4d8..260d31e01 100644 |
| --- a/internal/ops/litestream_owners_unix_test.go |
| +++ b/internal/ops/litestream_owners_unix_test.go |
| @@ -1101,3 +1101,74 @@ func TestFleetInventoryRetainsAndRechecksChains(t *testing.T) { |
| t.Fatalf("a report holds %d descriptors (%d open, %d before)", rep.Descriptors(), n, before) |
| } |
| } |
| + |
| +// Required coverage: the registry this process has open and the console |
| +// database CONSOLE_DB names are required; one that cannot be reached |
| +// through its root (missing, or a link in its place) is a problem that |
| +// makes the inventory incomplete and, when a fleet is built, holds the |
| +// box's day through the options' Hold, before the build is refused; a |
| +// reachable one is not a problem. The read-only report says which. |
| +func TestRequiredCoverage(t *testing.T) { |
| + dir := platformDir(t) |
| + reg := openRegistry(t, filepath.Join(dir, "registry.db")) |
| + console := writeFile(t, filepath.Join(dir, "console", "console.db")) |
| + t.Setenv("CONSOLE_DB", console) |
| + inv, err := FleetDBOwners(reg, dir) |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + if len(inv.Required) != 2 || inv.Required[0].Path != reg.Path() || inv.Required[1].Path != console || len(inv.RequiredMissing()) != 0 { |
| + t.Fatalf("required %+v missing %v", inv.Required, inv.RequiredMissing()) |
| + } |
| + if len(inv.Problems) != 0 { |
| + t.Fatalf("problems with every required database reachable: %v", inv.Problems) |
| + } |
| + report, err := InventoryReport(reg, dir) |
| + if err != nil || !strings.Contains(report, "required "+console+" reachable") || !strings.Contains(report, "required: 2 databases") { |
| + t.Fatalf("report: %v\n%s", err, report) |
| + } |
| + // the console database replaced by a link to itself: not reachable |
| + if err := os.Rename(console, console+".real"); err != nil { |
| + t.Fatal(err) |
| + } |
| + if err := os.Symlink(console+".real", console); err != nil { |
| + t.Fatal(err) |
| + } |
| + inv, err = FleetDBOwners(reg, dir) |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + if m := inv.RequiredMissing(); len(m) != 1 || m[0] != console || !named(inv.Problems, console, "a required database cannot be reached through its root") || !named(inv.Problems, console, "it is a link") { |
| + t.Fatalf("a link in the required database's place: missing %v problems %v", m, inv.Problems) |
| + } |
| + report, _ = InventoryReport(reg, dir) |
| + if !strings.Contains(report, "required "+console+" MISSING: it is a link") { |
| + t.Fatalf("report:\n%s", report) |
| + } |
| + // missing altogether: a problem, and a fleet build holds the day once |
| + // and is refused |
| + os.Remove(console) |
| + os.Remove(console + ".real") |
| + var held []time.Time |
| + o := fleetOptions(reg, dir) |
| + o.Hold = func(at time.Time) { held = append(held, at) } |
| + if _, err := MeteredFleetStore(o); err == nil || !strings.Contains(err.Error(), "a required database cannot be reached") { |
| + t.Fatalf("a fleet built with a required database missing: %v", err) |
| + } |
| + if len(held) != 1 { |
| + t.Fatalf("the day was held %d times, want once", len(held)) |
| + } |
| + // reachable again: no hold, no problem |
| + writeFile(t, console) |
| + held = nil |
| + inv, err = FleetDBOwners(reg, dir) |
| + if err != nil || len(inv.RequiredMissing()) != 0 || len(inv.Problems) != 0 { |
| + t.Fatalf("restored: %v missing %v problems %v", err, inv.RequiredMissing(), inv.Problems) |
| + } |
| + // without CONSOLE_DB only the registry is required |
| + t.Setenv("CONSOLE_DB", "") |
| + inv, _ = FleetDBOwners(reg, dir) |
| + if len(inv.Required) != 1 || inv.Required[0].Path != reg.Path() { |
| + t.Fatalf("required without CONSOLE_DB: %+v", inv.Required) |
| + } |
| +} |
diff --git a/internal/ops/ops.go b/internal/ops/ops.go| index 6ff589464..bac280379 100644 |
| --- a/internal/ops/ops.go |
| +++ b/internal/ops/ops.go |
| @@ -41,6 +41,8 @@ func Run(args []string) { |
| runRestoreVerify(args) |
| case "hibernate-verify": |
| runHibernateVerify(args) |
| + case "inventory": |
| + runInventory(args) |
| case "", "-h", "--help", "help": |
| usage() |
| if verb == "" { |
| @@ -61,6 +63,11 @@ func usage() { |
| so the caller knows whether to restart replication |
| carlos ops restore-verify restore every configured replica from S3 and |
| integrity-check each (run by the daily timer) |
| + carlos ops inventory read-only: the platform replicator's inventory of |
| + this box (owned, skipped, unowned, problems, the |
| + required databases) beside the external |
| + replicator's configured list, for the cutover |
| + comparison; writes nothing |
| carlos ops hibernate-verify |
| the same drill for the replicas the config does |
| NOT name: hibernating instances, whose |
| @@ -343,3 +350,93 @@ func openDeployStore(ctx context.Context) (release.Store, error) { |
| } |
| return nil, errors.New("neither CARLOS_DEPLOYMENT_BUCKET nor CARLOS_DEPLOYMENT_DIR is set") |
| } |
| + |
| +// runInventory prints the platform replicator's inventory beside the |
| +// external replicator's configured list. Read-only: nothing is written, |
| +// no fleet is built, no hold is placed. |
| +func runInventory(args []string) { |
| + fs := flag.NewFlagSet("ops inventory", flag.ExitOnError) |
| + regPath := fs.String("registry", "", "registry database (default <data>/registry.db)") |
| + data := fs.String("data", "", "data directory holding the fleet's *.db files (default $CARLOS_DATA_DIR or /data)") |
| + fs.Usage = cliflag.Usage(fs, "") |
| + _ = fs.Parse(args) |
| + dir := dataDir(*data) |
| + rp := *regPath |
| + if rp == "" { |
| + rp = dir + "/registry.db" |
| + } |
| + reg, err := registry.Open(rp) |
| + if err != nil { |
| + log.Fatalf("inventory: open registry %s: %v", rp, err) |
| + } |
| + defer reg.Close() |
| + report, err := InventoryReport(reg, dir) |
| + if err != nil { |
| + log.Fatalf("inventory: %v", err) |
| + } |
| + fmt.Print(report) |
| +} |
| + |
| +// InventoryReport renders, for one box, the external replicator's |
| +// configured databases (what carlos ops litestream would write), the |
| +// platform replicator's inventory (FleetDBOwners), and the required |
| +// databases with whether each is reachable through its root. It reads |
| +// and writes nothing but the report. |
| +func InventoryReport(reg *registry.Registry, dir string) (string, error) { |
| + dbs, skipped, err := FleetDBsWithSkipped(reg, dir) |
| + if err != nil { |
| + return "", err |
| + } |
| + inv, err := FleetDBOwners(reg, dir) |
| + if err != nil { |
| + return "", err |
| + } |
| + var b strings.Builder |
| + fmt.Fprintf(&b, "external replicator: %d databases configured, %d skipped\n", len(dbs), len(skipped)) |
| + for _, p := range dbs { |
| + fmt.Fprintf(&b, " configured %s\n", p) |
| + } |
| + for _, p := range skipped { |
| + fmt.Fprintf(&b, " skipped %s\n", p) |
| + } |
| + fmt.Fprintf(&b, "platform replicator: %d owned, %d skipped, %d unowned, %d problems\n", len(inv.Owned), len(inv.Skipped), len(inv.Unowned), len(inv.Problems)) |
| + for _, o := range inv.Owned { |
| + fmt.Fprintf(&b, " owned %s %s/%s %s\n", o.Path, o.Account, o.App, o.Host) |
| + } |
| + for _, p := range inv.Skipped { |
| + fmt.Fprintf(&b, " skipped %s\n", p) |
| + } |
| + for _, p := range inv.Unowned { |
| + fmt.Fprintf(&b, " unowned %s\n", p) |
| + } |
| + for _, p := range inv.Problems { |
| + fmt.Fprintf(&b, " problem %s\n", p) |
| + } |
| + owned := map[string]bool{} |
| + for _, o := range inv.Owned { |
| + owned[o.Path] = true |
| + } |
| + for _, p := range dbs { |
| + if !owned[p] { |
| + fmt.Fprintf(&b, " configured but not owned by the platform replicator: %s\n", p) |
| + } |
| + } |
| + configured := map[string]bool{} |
| + for _, p := range dbs { |
| + configured[p] = true |
| + } |
| + for _, o := range inv.Owned { |
| + if !configured[o.Path] { |
| + fmt.Fprintf(&b, " owned but not configured for the external replicator: %s\n", o.Path) |
| + } |
| + } |
| + fmt.Fprintf(&b, "required: %d databases\n", len(inv.Required)) |
| + for _, r := range inv.Required { |
| + if r.Why == "" { |
| + fmt.Fprintf(&b, " required %s reachable\n", r.Path) |
| + } else { |
| + fmt.Fprintf(&b, " required %s MISSING: %s\n", r.Path, r.Why) |
| + } |
| + } |
| + return b.String(), nil |
| +} |