amadan
Merge origin/main (56ebee16) into moves-phase1-3
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
23 files changed,
+1636
−73
docs/site/cli.md+38 −3internal/activator/activator_test.go+3 −0internal/activator/backend_exec.go+20 −5internal/activator/backend_exec_test.go+34 −1internal/activator/unitproc_test.go+5 −0internal/host/systemapply.go+71 −2internal/host/systemapply_test.go+117 −3internal/host/units/carlos-instance-seed@.service+9 −0internal/host/units/carlos-instance-sidecar@.service+9 −0internal/host/units/carlos-instance@.service+9 −0internal/host/units_isolation_test.go+13 −2internal/instancectl/client.go+43 −16internal/instancectl/client_test.go+39 −0internal/instancectl/names.go+8 −0internal/instancectl/server.go+60 −9internal/instancectl/server_test.go+59 −0internal/layout/extras.go+357 −0internal/layout/extras_pass_linux_test.go+327 −0internal/layout/extras_test.go+194 −0internal/layout/journal.go+25 −0internal/layout/pass.go+113 −31internal/layout/plan.go+21 −1internal/layout/preflight.go+62 −0
CI passed — run details
diff --git a/docs/site/cli.md b/docs/site/cli.md| index 8e1963123..44dc7d488 100644 |
| --- a/docs/site/cli.md |
| +++ b/docs/site/cli.md |
| @@ -674,6 +674,31 @@ holds the account's authoritative data, and until a batch of orders drains |
| home it holds order data in its own region — so the placement set is an |
| explicit decision on the home's record, never a default. |
| +#### What an instance is started with |
| + |
| +A hibernating instance is started by the box when a request arrives for it, |
| +as `<your binary> --socket <path> --db <path>`: the unix socket to serve on, |
| +and the SQLite database that is yours. On a box with the nested layout its |
| +working directory is its own data directory, the one the database is in, so |
| +a relative path such as `./data/blobs` lands beside the database and inside |
| +the instance's namespace. (Under the flat layout it was `/`, so the same path |
| +landed in the shared `/data`; `layout migrate` leaves such shared directories |
| +where they are and names them in the agent log for an operator to place.) |
| +Its environment is your app's config bundle, with the instance's own |
| +`<host>.env` layered on top, plus a few variables the platform sets and that |
| +no bundle can override: |
| + |
| +| Variable | Value | |
| +|---|---| |
| +| `CARLOS_HOST` | the instance's hostname, exactly as in `carlos instances list`. Read this, never the socket's file name, which says nothing about the host on an isolated box | |
| +| `CARLOS_REGION` | the box's region code, such as `ie` | |
| +| `CARLOS_CONTROL_SOCKET` | the socket the platform listens on for your app's purge and scheduling calls | |
| +| `CARLOS_PURGE` | `header` when the edge honours a `Carlos-Purge` response header | |
| +| `CARLOS_ADMIN_TOKEN` | a secret shared with this one instance: the platform presents it when it calls your app, and your app presents it on `CARLOS_CONTROL_SOCKET` | |
| + |
| +A seed (the `--seed` command of `carlos instances enable`) runs with the same |
| +environment, `CARLOS_HOST` included, and no control socket. |
| + |
| #### Storage of its own, for every instance |
| `--private-storage` on `enable` is one declaration that changes what every |
| @@ -1845,9 +1870,9 @@ owns. |
| `update` promotes a public release (or, with `--file`, a binary you built) to |
| `canary` or `stable`; every box on that channel fetches it through its normal |
| -store, verifies it against the signed pointer, installs it, restarts its edge, |
| -and rolls back to the previous binary if the new one does not come up and stay |
| -up. `status` shows each box's platform, channel, running build, the channel's |
| +store, verifies it against the signed pointer, installs it, restarts its |
| +instance-control service and then its edge, and rolls back to the previous |
| +binary if the new one does not come up and stay up. `status` shows each box's platform, channel, running build, the channel's |
| target, and the last apply outcome with its reason. `rollback` points a channel |
| back at the release before this one, and every box on it rolls back the same |
| way it rolled forward. All three talk to your console. You need to be an owner |
| @@ -1872,6 +1897,8 @@ Every instance the activator runs used to live in one shared directory as one us |
| `layout migrate` is refused while any box in scope runs a platform release from before the layout, so a fleet is rolled forward first and migrated second. |
| +What moves with an instance is everything in the box's data directory that carries its hostname: its database and the platform's files beside it, and anything the app made for itself under that name, such as a `<host>/` directory holding its keys or a `<host>-blobs` store. They land in the instance's directory under the same names, which is where an app that derives paths from its database path now looks. An entry that cannot be moved safely, such as a file hard-linked from outside the instance's own files, or a symlink, stops the move with its name in the reason, and the instance is held until it is cleared. A directory that carries no hostname, such as a `blobs` directory that several apps wrote into with the working directory `/`, is not moved: the box cannot tell whose it is. The agent log names those once a move has happened; an operator moves what an app owns into that app's instance directory by hand, and it comes along on any later revert or migration. A file named like another instance's platform files (`other.example.com.env`, `other.example.com.db`), including one left by an instance that has since been deleted or moved away, is never taken; the log names it. `revert` brings the same names back. What an app wrote relative to its working directory while nested (a `data/blobs` under its instance directory) is not among them: it stays in the instance's directory, where the flat app does not look, but nothing is lost. An instance that was moved before the platform carried entries along still has them in the shared directory; `carlos status --layout` and the agent log say so on every pass until an operator moves them. |
| + |
| You rarely need `migrate` at all. With no intent recorded, a box uses the nested layout by default as soon as its isolation is ready (the probe passes), and moves its instances on its own, with the same hour of grace for an awake one. `carlos status --layout` marks such a box `(nested by default)`, and says why a box the default has not reached is still flat. To keep box-by-box control, record `migrate --box` or `revert` first; a recorded intent always wins. A deployment that is not ready for this at all puts `CARLOS_LAYOUT_DEFAULT=flat` in each box's host.env. |
| `layout revert` moves them back. The order for rolling the platform back to a release from before the layout is: `layout revert`, wait until `carlos status --layout` shows every instance `flat`, then `system rollback`. The console will not move a channel pointer onto such a release while any box reports otherwise, and a revert is refused while a maintenance worker is attached to a box in scope. `layout status` prints the recorded intent; what each box has done is `carlos status --layout`. |
| @@ -2030,6 +2057,14 @@ its slice before a start, and runs the isolation probe at start and every |
| five minutes. It accepts a connection only from the edge unit's own main |
| process. Not for operators: there is nothing to type. |
| +It stays up once the first wake has started it, so it has to be restarted |
| +whenever the platform binary changes; `carlos system update` does that, and |
| +a hand roll must too. A box checks both ends: the service refuses to be |
| +ready while the binary it runs is no longer `/opt/carlos/carlos`, and the |
| +agent refuses to wake isolated instances while the service reports another |
| +build than its own, or none. Either way the reason in `carlos logs` ends |
| +with `restart carlos-instance-control.service`. |
| + |
| ### carlos instance-exec |
| The first thing an isolated instance unit runs. It reads the launch file |
diff --git a/internal/activator/activator_test.go b/internal/activator/activator_test.go| index f6edb8d93..8edb0a379 100644 |
| --- a/internal/activator/activator_test.go |
| +++ b/internal/activator/activator_test.go |
| @@ -5649,6 +5649,9 @@ func TestExecBackendSeedRunsDeclaredArgvWithDBPath(t *testing.T) { |
| if got := child["CARLOS_ADMIN_TOKEN"]; got != "real" { |
| t.Fatalf("seed CARLOS_ADMIN_TOKEN = %q, want %q — the seed gets the instance's own env file", got, "real") |
| } |
| + if got := child["CARLOS_HOST"]; got != host { |
| + t.Fatalf("seed CARLOS_HOST = %q, want %q (#256: the seed is told its host like the instance is)", got, host) |
| + } |
| if _, ok := child["CARLOS_HIBERNATE_BUCKET"]; ok { |
| t.Fatal("the agent's own CARLOS_* config leaked into the seed process") |
| } |
diff --git a/internal/activator/backend_exec.go b/internal/activator/backend_exec.go| index 4ae1ea1e7..638c1d616 100644 |
| --- a/internal/activator/backend_exec.go |
| +++ b/internal/activator/backend_exec.go |
| @@ -810,7 +810,7 @@ func (b *ExecBackend) seedChild(ctx context.Context, rt registry.Route) error { |
| // serving cache to purge, nor any business setting timers, in any case. |
| // This is what makes the contract above true rather than merely |
| // intended. |
| - cmd.Env = childEnv(withoutVars(fileEnv, "CARLOS_CONTROL_SOCKET", "CARLOS_PURGE")) |
| + cmd.Env = childEnv(seedVars(fileEnv, host)) |
| // Like the instance child, the seed logs through the agent's own streams |
| // (journald under systemd) — and its stderr is ALSO teed into a bounded |
| // buffer, because the wake error this returns is the whole diagnosis an |
| @@ -949,7 +949,7 @@ func (b *ExecBackend) seedUnit(ctx context.Context, rt registry.Route) error { |
| return nil // defensive: doWake only calls us for a non-empty seed |
| } |
| argv := append([]string{b.liveLink(host)}, seedArgs...) |
| - env := childEnv(withoutVars(fileEnv, "CARLOS_CONTROL_SOCKET", "CARLOS_PURGE")) |
| + env := childEnv(seedVars(fileEnv, host)) |
| if err := b.writeLaunch(p, instancectl.RoleSeed, host, argv, env); err != nil { |
| return err |
| } |
| @@ -1342,7 +1342,22 @@ func (b *ExecBackend) agentOwnedVars(rt registry.Route, fileEnv map[string]strin |
| } else { |
| childVars = withoutVars(childVars, "CARLOS_REGION") |
| } |
| - return childVars |
| + // The hostname, always, from the row (discussion #256 item 2). An app |
| + // used to learn it from its socket's file name, /run/carlos/<host>.sock, |
| + // and the nested layout's app.sock names nothing; three instances |
| + // crashed at start over that on the first roll. Agent-owned like the |
| + // three above: a tenant's env file does not get to tell an app it is |
| + // some other host. |
| + return withVar(childVars, "CARLOS_HOST", rt.Host) |
| +} |
| + |
| +// seedVars is the seed's environment over fileEnv: the control-channel |
| +// variables stripped (a seed has no listener for either; see seedChild's |
| +// comment), and CARLOS_HOST set, since a seed initialising a host's |
| +// database is as entitled to know which host as the instance is. One |
| +// composer for both seed paths, as agentOwnedVars is for both run paths. |
| +func seedVars(fileEnv map[string]string, host string) map[string]string { |
| + return withVar(withoutVars(fileEnv, "CARLOS_CONTROL_SOCKET", "CARLOS_PURGE"), "CARLOS_HOST", host) |
| } |
| // normalizeSocketMode is the PLATFORM's 0660 on a spawned instance's socket |
| @@ -1522,8 +1537,8 @@ func (b *ExecBackend) instanceEnvParts(rt registry.Route) (cfgEnv, ownEnv map[st |
| // exact composition Start uses (agentOwnedVars(rt, fileEnv, haveControl)), |
| // decoded back to a map. agentOwnedVars REMOVES tenant-provided |
| // CARLOS_CONTROL_SOCKET, CARLOS_PURGE and CARLOS_REGION when the agent |
| -// does not provide them, so a report over mergeEnv alone would misstate |
| -// the environment. haveControl is true: startChild calls |
| +// does not provide them, and always sets CARLOS_HOST, so a report over |
| +// mergeEnv alone would misstate the environment. haveControl is true: startChild calls |
| // edge.ListenControl for EVERY exec child, and b.Control is the nested |
| // root-control client, which says nothing about the flat channel, so |
| // before the spawn the honest composition is "the agent will bind it"; a |
diff --git a/internal/activator/backend_exec_test.go b/internal/activator/backend_exec_test.go| index 8bd1e88b6..8d86084f4 100644 |
| --- a/internal/activator/backend_exec_test.go |
| +++ b/internal/activator/backend_exec_test.go |
| @@ -114,6 +114,35 @@ func TestExecBackendStartChildRegion(t *testing.T) { |
| } |
| } |
| +// Every exec child is told which host it is, in its environment, from the |
| +// row (discussion #256): an app that used to read its hostname off its |
| +// socket's file name has nothing to read nested (app.sock). The variable |
| +// is agent-owned: a tenant's own env file cannot make an app another host. |
| +func TestExecBackendStartChildHandsTheHost(t *testing.T) { |
| + b := testExecBackend(t) |
| + host := "host-env.example.com" |
| + _, envPath := stageRecordingLiveFullEnv(t, b, host, "v1") |
| + if err := os.WriteFile(filepath.Join(b.DataDir, host+".env"), []byte("CARLOS_HOST=forged.example.com\n"), 0o600); err != nil { |
| + t.Fatal(err) |
| + } |
| + rt := registry.Route{Host: host, Kind: "instance", Socket: filepath.Join(t.TempDir(), "instance.sock")} |
| + ctx, cancel := context.WithTimeout(context.Background(), 2*time.Second) |
| + defer cancel() |
| + proc, _, _ := b.Start(ctx, rt) // fake binary never binds the socket; health-wait error expected |
| + if proc != nil { |
| + defer proc.Stop(context.Background()) |
| + } |
| + child := map[string]string{} |
| + for _, line := range strings.Split(waitForFile(t, envPath, time.Second), "\n") { |
| + if k, v, ok := strings.Cut(line, "="); ok { |
| + child[k] = v |
| + } |
| + } |
| + if got := child["CARLOS_HOST"]; got != host { |
| + t.Fatalf("child CARLOS_HOST = %q, want %q", got, host) |
| + } |
| +} |
| + |
| // The staging fallback (issue #331): a host whose ordinary prefix is empty |
| // restores from a move's staged copy — promoted server-side, dropped after |
| // a clean restore — and the ordinary prefix ALWAYS wins when it exists. |
| @@ -348,7 +377,11 @@ func TestLaunchEnvironmentComposesTheSpawnEnvironment(t *testing.T) { |
| env["MINE"] == "1", env["UNRELATED"] == "yes", env["CARLOS_SECRETS_KEY"] != "", env["CARLOS_ADMIN_TOKEN"] == "t") |
| } |
| // The agent-owned layer, exactly as agentOwnedVars composes it: the |
| - // tenant's region and purge values are overridden or removed. |
| + // tenant's region and purge values are overridden or removed, and the |
| + // host is the row's (#256). |
| + if env["CARLOS_HOST"] != rt.Host { |
| + t.Fatalf("CARLOS_HOST = %q, want the row's host %q", env["CARLOS_HOST"], rt.Host) |
| + } |
| if env["CARLOS_REGION"] != "ie" || env["CARLOS_CONTROL_SOCKET"] != rt.Socket+".control" { |
| t.Fatalf("agent-owned vars: CARLOS_REGION is the backend's: %v; CARLOS_CONTROL_SOCKET is the route's control path: %v", |
| env["CARLOS_REGION"] == "ie", env["CARLOS_CONTROL_SOCKET"] == rt.Socket+".control") |
diff --git a/internal/activator/unitproc_test.go b/internal/activator/unitproc_test.go| index 9c9d89297..7b867e7c5 100644 |
| --- a/internal/activator/unitproc_test.go |
| +++ b/internal/activator/unitproc_test.go |
| @@ -286,10 +286,15 @@ func TestSeedUnitCleansUpOnlyItsOwnDBOnFailure(t *testing.T) { |
| if len(l.Argv) != 5 || l.Argv[1] != "seed" || l.Argv[2] != "--fast" || l.Argv[3] != "--db" || l.Argv[4] != rt.DBPath { |
| t.Errorf("seed argv = %v", l.Argv) |
| } |
| + gotHost := false |
| for _, kv := range l.Env { |
| if strings.HasPrefix(kv, "CARLOS_CONTROL_SOCKET=") || strings.HasPrefix(kv, "CARLOS_PURGE=") { |
| t.Errorf("seed env carries %s", kv) |
| } |
| + gotHost = gotHost || kv == "CARLOS_HOST="+rt.Host |
| + } |
| + if !gotHost { |
| + t.Errorf("seed env lacks CARLOS_HOST=%s (#256): %v", rt.Host, l.Env) |
| } |
| if _, err := os.Stat(rt.DBPath); !os.IsNotExist(err) { |
| t.Errorf("failed seed left its database: %v", err) |
diff --git a/internal/host/systemapply.go b/internal/host/systemapply.go| index 0fea54ddd..09330955d 100644 |
| --- a/internal/host/systemapply.go |
| +++ b/internal/host/systemapply.go |
| @@ -44,8 +44,13 @@ import ( |
| // finds is refuse the install. |
| const ( |
| - systemEdgeUnit = "carlos-edge" |
| - systemRelayUnit = "carlos-relay" |
| + systemEdgeUnit = "carlos-edge" |
| + systemRelayUnit = "carlos-relay" |
| + // systemControlUnit is the root instance-control service (isolated-exec |
| + // design §5). It execs /opt/carlos/carlos and stays up for weeks once |
| + // the first wake activated it, so every install that changes the |
| + // binary restarts it (restartInstanceControl; discussion #256). |
| + systemControlUnit = "carlos-instance-control.service" |
| systemLockFile = "lock" |
| systemMaxRequest = 4 << 10 |
| systemGateWait = 20 * time.Second |
| @@ -528,6 +533,11 @@ func (a *systemApplier) install(ctx context.Context, req systemRequest, p *relea |
| if err := a.journal(j); err != nil { |
| return fail("journal", err) |
| } |
| + // The control service before the edge; see restartInstanceControl for |
| + // why that order, and why a restart that may drop a wake is safe here. |
| + if err := a.restartInstanceControl(); err != nil { |
| + return fail("instance-control restart", err) |
| + } |
| if err := a.restartEdge(); err != nil { |
| return fail("restart", err) |
| } |
| @@ -1280,6 +1290,48 @@ func (a *systemApplier) restartEdge() error { |
| return err |
| } |
| +// restartInstanceControl restarts carlos-instance-control.service, the |
| +// root service that starts every nested instance (design §5). It is |
| +// socket-activated and then stays up: on the flagship it had run the |
| +// 2026-09-16 binary for three weeks when v0.25.1 was installed around it |
| +// (discussion #256). A control service of the previous release serves the |
| +// new release's unit templates, which bind sources only the new code |
| +// writes (/run/carlos-control/exe/<iid>/), so every nested start failed |
| +// at namespace setup and the probe unit, run by the same stale service, |
| +// failed the same way: the box reported not ready with the right binary |
| +// on disk and nothing to say why. |
| +// |
| +// Order: after the new units are installed and daemon-reload has run, and |
| +// BEFORE the edge restarts. The new service then runs its start-up probe |
| +// against the old edge, which is up, so the verdict it records is real |
| +// (a probe with no edge cgroup fails on cgroup-ro); and by the time the |
| +// new agent starts, the service it calls is already the new build, so |
| +// there is no window in which a new agent asks an old service to start |
| +// units it cannot. A wake in flight at this moment is lost, and that is |
| +// safe: the edge restart one step later stops every instance anyway |
| +// (BindsTo=carlos-edge.service), so the wake fails either way; the |
| +// systemctl start the dropped request was running is a job systemd |
| +// finishes on its own; the agent reads the dropped connection as a failed |
| +// wake and sends the stop, which the socket unit, up throughout, queues |
| +// for the restarted service. Flat hosts never pass through this service. |
| +// |
| +// A build that ships no control service (a downgrade below v0.20, whose |
| +// unit installUnits has just disabled and removed) has nothing to |
| +// restart; that is the one case this returns nil without calling |
| +// systemctl. reset-failed first, as restartEdge does, so a unit that hit |
| +// its start limit can be restarted at all. |
| +func (a *systemApplier) restartInstanceControl() error { |
| + if _, err := os.Stat(filepath.Join(a.etcDir, "systemd", "system", systemControlUnit)); err != nil { |
| + if errors.Is(err, os.ErrNotExist) { |
| + return nil |
| + } |
| + return err |
| + } |
| + a.ops.systemctl("reset-failed", systemControlUnit) |
| + _, err := a.ops.systemctl("restart", systemControlUnit) |
| + return err |
| +} |
| + |
| // restartRelay mirrors restartEdge for the relay (relay spec §3), with one |
| // difference that decides its shape: the relay is optional. A box with no |
| // /etc/carlos/relay-masters must not be handed a relay it never asked for. |
| @@ -1464,6 +1516,14 @@ func (a *systemApplier) rollback(ctx context.Context, req systemRequest, reason |
| // carlos.prev, now on disk: what this rollback's journal finish makes |
| // current, checked again under activationGuard at that finish. |
| restored, _ := readLine(prevVersionPath(a.optDir)) |
| + // The previous binary is on disk, so the control service goes back |
| + // too, before the edge, as on the way forward. Logged, not fatal: a |
| + // rollback's job is a good edge, and a control service that would |
| + // not restart is caught by the agent's readiness check (it refuses |
| + // nested wakes until the service runs the installed binary). |
| + if err := a.restartInstanceControl(); err != nil { |
| + a.ops.logf("system apply: rollback: instance-control restart: %v (nested wakes refuse until it is restarted)", err) |
| + } |
| if err := a.restartEdge(); err != nil { |
| return failed("restart", err) // journal kept: the edge is still to come back |
| } |
| @@ -1712,6 +1772,15 @@ func (a *systemApplier) recoverIfNeeded(ctx context.Context) (bool, error) { |
| // carlos.prev, now on disk: what the finish below makes current. |
| restored, _ := readLine(prevVersionPath(a.optDir)) |
| if a.ops.edgeStartedThisBoot() { |
| + // The control service first, as in install and rollback. Not at |
| + // boot: it is socket-activated and not yet running, and started |
| + // before the edge it would run its probe against an edge whose |
| + // cgroup does not exist yet and record a failing verdict for the |
| + // next five minutes. Blocking is safe: nothing orders this unit |
| + // against carlos-system-recover.service. |
| + if err := a.restartInstanceControl(); err != nil { |
| + a.ops.logf("system apply: recovery: instance-control restart: %v (nested wakes refuse until it is restarted)", err) |
| + } |
| a.ops.systemctl("reset-failed", systemEdgeUnit) |
| // --no-block: this can be running as the edge's own OnFailure= |
| // handler, and a blocking restart of the unit that triggered it |
diff --git a/internal/host/systemapply_test.go b/internal/host/systemapply_test.go| index a37a18ad6..ddca8e505 100644 |
| --- a/internal/host/systemapply_test.go |
| +++ b/internal/host/systemapply_test.go |
| @@ -131,7 +131,7 @@ func newApplyFixture(t *testing.T, channel, build string) *applyFixture { |
| os.WriteFile(systemBinaryPath(f.a.optDir), []byte("#!/bin/sh\necho carlos "+build+"\n"), 0o755) |
| f.answer["version:"+systemBinaryPath(f.a.optDir)] = "carlos " + build + "\n" |
| f.answer["plan:"+systemBinaryPath(f.a.optDir)] = `{"enabled":["carlos-edge.service","carlos-adopt.timer","carlos-adopt.path","carlos-stage.path","carlos-restart.path","carlos-system-update.path","carlos-system-recover.service"],"inert":["carlos-restore-verify.timer"]}` |
| - f.answer["units:"+systemBinaryPath(f.a.optDir)] = "carlos-edge.service carlos-adopt.timer carlos-adopt.path carlos-stage.path carlos-restart.path carlos-system-update.path carlos-system-update.service carlos-system-recover.service carlos-old-only.timer" |
| + f.answer["units:"+systemBinaryPath(f.a.optDir)] = "carlos-edge.service carlos-adopt.timer carlos-adopt.path carlos-stage.path carlos-restart.path carlos-system-update.path carlos-system-update.service carlos-system-recover.service carlos-instance-control.service carlos-old-only.timer" |
| return f |
| } |
| @@ -416,7 +416,7 @@ func applyHappyVersion(t *testing.T, version string) (*applyFixture, systemReque |
| staged := filepath.Join(f.a.stagingDir(), "carlos") |
| f.answer["version:"+staged] = "carlos " + version + "\n" |
| f.answer["plan:"+staged] = `{"enabled":["carlos-edge.service","carlos-adopt.timer","carlos-adopt.path","carlos-stage.path","carlos-restart.path","carlos-system-update.path","carlos-system-recover.service","carlos-new-only.path"],"inert":["carlos-restore-verify.timer"]}` |
| - f.answer["units:"+staged] = "carlos-edge.service carlos-adopt.timer carlos-adopt.path carlos-stage.path carlos-restart.path carlos-system-update.path carlos-system-update.service carlos-system-recover.service carlos-new-only.path" |
| + f.answer["units:"+staged] = "carlos-edge.service carlos-adopt.timer carlos-adopt.path carlos-stage.path carlos-restart.path carlos-system-update.path carlos-system-update.service carlos-system-recover.service carlos-instance-control.service carlos-new-only.path" |
| // The unit v1 owns and v2 drops is on the box, so removing it is a |
| // thing that can be seen to happen — or not to. |
| os.WriteFile(filepath.Join(f.a.etcDir, "systemd", "system", "carlos-old-only.timer"), []byte("[Unit]\n# v1\n"), 0o644) |
| @@ -740,6 +740,113 @@ func TestApplyRestartsTheRelayAfterTheEdge(t *testing.T) { |
| } |
| } |
| +// Discussion #256: the root instance-control service execs the platform |
| +// binary and stays up for weeks, so an install that leaves it running the |
| +// previous build leaves every nested wake failing at namespace setup. It |
| +// is restarted after the new units are in place (daemon-reload) and before |
| +// the edge, so the new agent never meets an old control service. |
| +func TestApplyRestartsTheInstanceControlServiceBeforeTheEdge(t *testing.T) { |
| + f, _ := applyHappy(t) |
| + if err := f.a.Run(context.Background(), false); err != nil { |
| + t.Fatal(err) |
| + } |
| + joined := strings.Join(f.calls, "\n") |
| + reload := strings.Index(joined, "systemctl daemon-reload") |
| + reset := strings.Index(joined, "systemctl reset-failed carlos-instance-control.service") |
| + restart := strings.Index(joined, "systemctl restart carlos-instance-control.service") |
| + edge := strings.Index(joined, "systemctl restart carlos-edge") |
| + switch { |
| + case reset < 0 || restart < 0: |
| + t.Fatalf("instance-control never restarted:\n%s", joined) |
| + case !(reload < reset && reset < restart && restart < edge): |
| + t.Fatalf("want daemon-reload, then reset-failed and restart carlos-instance-control.service, then restart carlos-edge:\n%s", joined) |
| + } |
| + if strings.Contains(joined, "systemctl try-restart carlos-instance-control.service") { |
| + t.Errorf("try-restart cannot start an inactive (socket-activated, never connected) service:\n%s", joined) |
| + } |
| +} |
| + |
| +// A control service that will not restart fails the install, and the |
| +// rollback restarts it again onto the previous binary (so the restart is |
| +// attempted twice over the run), rather than leaving a new edge beside a |
| +// control service of unknown vintage. |
| +func TestApplyInstanceControlRestartFailureRollsBack(t *testing.T) { |
| + f, req := applyHappy(t) |
| + orig := f.a.ops.systemctl |
| + failed := false |
| + f.a.ops.systemctl = func(args ...string) (string, error) { |
| + out, err := orig(args...) |
| + if args[0] == "restart" && args[1] == systemControlUnit && !failed { |
| + failed = true |
| + return "", errors.New("Job for carlos-instance-control.service failed") |
| + } |
| + return out, err |
| + } |
| + if err := f.a.Run(context.Background(), false); err == nil { |
| + t.Fatal("a control service that would not restart passed the install") |
| + } |
| + r := f.result() |
| + if r == nil || r.Outcome != systemOutcomeRolledBack || !r.matches("stable", "v2", req.PromotedAt) || !strings.Contains(r.Reason, "instance-control restart") { |
| + t.Fatalf("result = %+v", r) |
| + } |
| + if n := strings.Count(strings.Join(f.calls, "\n"), "systemctl restart carlos-instance-control.service"); n != 2 { |
| + t.Errorf("control service restarted %d times, want 2 (install then rollback):\n%s", n, strings.Join(f.calls, "\n")) |
| + } |
| + if bin, _ := os.ReadFile(systemBinaryPath(f.a.optDir)); !strings.Contains(string(bin), "carlos v1") { |
| + t.Fatal("binary not rolled back") |
| + } |
| +} |
| + |
| +// The rollback of a failed gate restarts the control service onto the |
| +// previous binary too, before the edge, for the same reason the relay is. |
| +func TestApplyRollbackRestartsTheInstanceControlService(t *testing.T) { |
| + f, _ := applyHappy(t) |
| + f.edgeUp = false |
| + orig := f.a.ops.systemctl |
| + f.a.ops.systemctl = func(args ...string) (string, error) { |
| + if args[0] == "restart" && args[1] == systemEdgeUnit { |
| + if b, _ := os.ReadFile(systemBinaryPath(f.a.optDir)); strings.Contains(string(b), "carlos v1") { |
| + f.edgeUp = true |
| + } |
| + } |
| + return orig(args...) |
| + } |
| + if err := f.a.Run(context.Background(), false); err == nil { |
| + t.Fatal("a failed gate must exit non-zero") |
| + } |
| + joined := strings.Join(f.calls, "\n") |
| + edge := strings.LastIndex(joined, "systemctl restart carlos-edge") |
| + ctl := strings.LastIndex(joined, "systemctl restart carlos-instance-control.service") |
| + if ctl < 0 || ctl > edge { |
| + t.Fatalf("the rollback left the control service on the withdrawn build, or restarted it after the edge:\n%s", joined) |
| + } |
| + if n := strings.Count(joined, "systemctl restart carlos-instance-control.service"); n != 2 { |
| + t.Errorf("control service restarted %d times, want 2 (apply then rollback):\n%s", n, joined) |
| + } |
| +} |
| + |
| +// A candidate that ships no control service (a downgrade below the |
| +// isolated-exec releases) has nothing to restart: installUnits has just |
| +// disabled and removed the unit, and a restart would fail the install. |
| +func TestApplyDoesNotRestartAControlServiceTheCandidateLacks(t *testing.T) { |
| + f, req := applyHappy(t) |
| + staged := filepath.Join(f.a.stagingDir(), "carlos") |
| + f.answer["units:"+staged] = strings.ReplaceAll(f.answer["units:"+staged], " carlos-instance-control.service", "") |
| + if err := f.a.Run(context.Background(), false); err != nil { |
| + t.Fatal(err) |
| + } |
| + if r := f.result(); r == nil || r.Outcome != systemOutcomeApplied || !r.matches("stable", "v2", req.PromotedAt) { |
| + t.Fatalf("result = %+v", r) |
| + } |
| + joined := strings.Join(f.calls, "\n") |
| + if !strings.Contains(joined, "systemctl disable --now carlos-instance-control.service") { |
| + t.Fatalf("the unit the candidate lacks was not disabled:\n%s", joined) |
| + } |
| + if strings.Contains(joined, "systemctl restart carlos-instance-control.service") { |
| + t.Fatalf("restarted a unit the candidate removed:\n%s", joined) |
| + } |
| +} |
| + |
| // A relay that cannot be restarted is not a failed update. The commonest |
| // case is the ordinary one — a box with no masters file, whose unit systemd |
| // skips on its condition and may well answer about — and the edge, the |
| @@ -860,7 +967,7 @@ func TestRecoveryRollsBackFilesOnlyAtBootAndRestartsAtRuntime(t *testing.T) { |
| t.Error("removed unit not restored") |
| } |
| joined := strings.Join(f.calls, "\n") |
| - if strings.Contains(joined, "systemctl restart carlos-edge") || strings.Contains(joined, "systemctl start ") { |
| + if strings.Contains(joined, "systemctl restart carlos-edge") || strings.Contains(joined, "systemctl start ") || strings.Contains(joined, "systemctl restart carlos-instance-control") { |
| t.Fatalf("boot recovery restarted or started something:\n%s", joined) |
| } |
| if !strings.Contains(joined, "systemctl enable carlos-edge.service") { |
| @@ -890,6 +997,13 @@ func TestRecoveryRollsBackFilesOnlyAtBootAndRestartsAtRuntime(t *testing.T) { |
| t.Errorf("runtime recovery lacks %q:\n%s", want, joined) |
| } |
| } |
| + // The control service goes back to the previous binary with the edge |
| + // (#256), and before it; at boot (above) neither is touched. |
| + ctl := strings.Index(joined, "systemctl restart carlos-instance-control.service") |
| + edge := strings.Index(joined, "systemctl restart --no-block carlos-edge") |
| + if ctl < 0 || ctl > edge { |
| + t.Errorf("runtime recovery does not restart the control service before the edge:\n%s", joined) |
| + } |
| } |
| func TestRecoveryFinishesForwardFromGated(t *testing.T) { |
diff --git a/internal/host/units/carlos-instance-seed@.service b/internal/host/units/carlos-instance-seed@.service| index 42facdaae..24e0eac7e 100644 |
| --- a/internal/host/units/carlos-instance-seed@.service |
| +++ b/internal/host/units/carlos-instance-seed@.service |
| @@ -53,6 +53,15 @@ StandardError=inherit |
| # The instance's own two directories, the only writable data paths. |
| BindPaths=/data/instances/%I /run/carlos/instances/%I |
| ReadWritePaths=/data/instances/%I /run/carlos/instances/%I |
| +# The working directory is the instance's own data directory, the one its |
| +# database is in (discussion #256 item D). A flat child ran with cwd /, |
| +# and an app writing relative paths (./data/blobs) wrote under /data; |
| +# without this line a unit's cwd is / on a read-only tmpfs and that app |
| +# exits at start with "read-only file system". systemd applies it after |
| +# the namespace is set up, so the bind above is what it chdirs into. Same |
| +# value in all three templates: the seed and the sidecar see the files |
| +# the instance made from the same place. |
| +WorkingDirectory=/data/instances/%I |
| # Its own build, and no other app's (#214 item 2). The namespace block |
| # empties /opt/carlos; these put back /opt/carlos/live/<host>, a link to |
| # ../app, and /opt/carlos/app, which PID 1 binds by chasing the source |
diff --git a/internal/host/units/carlos-instance-sidecar@.service b/internal/host/units/carlos-instance-sidecar@.service| index 18c672e28..4a43cf115 100644 |
| --- a/internal/host/units/carlos-instance-sidecar@.service |
| +++ b/internal/host/units/carlos-instance-sidecar@.service |
| @@ -52,6 +52,15 @@ StandardError=inherit |
| # The instance's own two directories, the only writable data paths. |
| BindPaths=/data/instances/%I /run/carlos/instances/%I |
| ReadWritePaths=/data/instances/%I /run/carlos/instances/%I |
| +# The working directory is the instance's own data directory, the one its |
| +# database is in (discussion #256 item D). A flat child ran with cwd /, |
| +# and an app writing relative paths (./data/blobs) wrote under /data; |
| +# without this line a unit's cwd is / on a read-only tmpfs and that app |
| +# exits at start with "read-only file system". systemd applies it after |
| +# the namespace is set up, so the bind above is what it chdirs into. Same |
| +# value in all three templates: the seed and the sidecar see the files |
| +# the instance made from the same place. |
| +WorkingDirectory=/data/instances/%I |
| # Its own build, and no other app's (#214 item 2). The namespace block |
| # empties /opt/carlos; these put back /opt/carlos/live/<host>, a link to |
| # ../app, and /opt/carlos/app, which PID 1 binds by chasing the source |
diff --git a/internal/host/units/carlos-instance@.service b/internal/host/units/carlos-instance@.service| index f979ff7d6..c8cec39bf 100644 |
| --- a/internal/host/units/carlos-instance@.service |
| +++ b/internal/host/units/carlos-instance@.service |
| @@ -52,6 +52,15 @@ StandardError=inherit |
| # The instance's own two directories, the only writable data paths. |
| BindPaths=/data/instances/%I /run/carlos/instances/%I |
| ReadWritePaths=/data/instances/%I /run/carlos/instances/%I |
| +# The working directory is the instance's own data directory, the one its |
| +# database is in (discussion #256 item D). A flat child ran with cwd /, |
| +# and an app writing relative paths (./data/blobs) wrote under /data; |
| +# without this line a unit's cwd is / on a read-only tmpfs and that app |
| +# exits at start with "read-only file system". systemd applies it after |
| +# the namespace is set up, so the bind above is what it chdirs into. Same |
| +# value in all three templates: the seed and the sidecar see the files |
| +# the instance made from the same place. |
| +WorkingDirectory=/data/instances/%I |
| # Its own build, and no other app's (#214 item 2). The namespace block |
| # empties /opt/carlos; these put back /opt/carlos/live/<host>, a link to |
| # ../app, and /opt/carlos/app, which PID 1 binds by chasing the source |
diff --git a/internal/host/units_isolation_test.go b/internal/host/units_isolation_test.go| index 9bc3ba317..f7a939f57 100644 |
| --- a/internal/host/units_isolation_test.go |
| +++ b/internal/host/units_isolation_test.go |
| @@ -266,16 +266,21 @@ func TestInstanceTemplatesShareOneNamespace(t *testing.T) { |
| for name, wants := range map[string][]string{ |
| "carlos-instance@.service": {"ExecStart=/opt/carlos/carlos instance-exec run %I", "ExecStopPost=/opt/carlos/carlos instance-exit run %I", |
| "BindPaths=/data/instances/%I /run/carlos/instances/%I", "StandardOutput=file:/run/carlos/logs/%I.run.sock", |
| + // The instance's own data directory, where a flat child's |
| + // relative paths now resolve (#256 D); the IE box's emergency |
| + // drop-in 50-workdir-incident-256.conf carries this exact line, |
| + // so removing it changes nothing. |
| + "WorkingDirectory=/data/instances/%I", |
| "Slice=carlos-instances-%i.slice", "Restart=no", "KillMode=mixed", "TimeoutStopSec=20", "BindsTo=carlos-edge.service", |
| "Requires=data-instances.mount run-carlos-instances.mount run-carlos-logs.mount", "OOMScoreAdjust=500", "OOMPolicy=kill", "TasksMax=256", "Type=simple", |
| // The trampoline's init exits 128+n for a death by signal n (#126). |
| "SuccessExitStatus=129 130 141 143"}, |
| "carlos-instance-seed@.service": {"Type=oneshot", "RemainAfterExit=yes", "TimeoutStartSec=120", "ExecStart=/opt/carlos/carlos instance-exec seed %I", |
| "ExecStopPost=/opt/carlos/carlos instance-exit seed %I", "StandardOutput=file:/run/carlos/logs/%I.seed.sock", |
| - "KillMode=mixed", "TimeoutStopSec=20"}, |
| + "KillMode=mixed", "TimeoutStopSec=20", "WorkingDirectory=/data/instances/%I"}, |
| "carlos-instance-sidecar@.service": {"Type=simple", "ExecStart=/opt/carlos/carlos instance-exec sidecar %I", |
| "ExecStopPost=/opt/carlos/carlos instance-exit sidecar %I", "StandardOutput=file:/run/carlos/logs/%I.sidecar.sock", |
| - "SuccessExitStatus=129 130 141 143"}, |
| + "SuccessExitStatus=129 130 141 143", "WorkingDirectory=/data/instances/%I"}, |
| "carlos-instance-probe.service": {"Type=oneshot", "ExecStart=/opt/carlos/carlos instance-probe", |
| "BindPaths=/data/instances/_probe /run/carlos/instances/_probe", "StandardOutput=truncate:/run/carlos-control/probe.out"}, |
| } { |
| @@ -285,6 +290,12 @@ func TestInstanceTemplatesShareOneNamespace(t *testing.T) { |
| t.Errorf("%s lacks %q", name, w) |
| } |
| } |
| + // Exactly one WorkingDirectory= per tenant template, and none in |
| + // the probe, whose directory is _probe and whose cwd does not |
| + // matter: a second line would silently win (#256 D). |
| + if n := strings.Count(u, "\nWorkingDirectory="); (name != "carlos-instance-probe.service" && n != 1) || (name == "carlos-instance-probe.service" && n != 0) { |
| + t.Errorf("%s has %d WorkingDirectory= lines", name, n) |
| + } |
| if strings.Contains(u, "ExecStopPost=+") { |
| t.Errorf("%s uses ExecStopPost=+: the plus prefix keeps the unit's tmpfs (codex R3-2)", name) |
| } |
diff --git a/internal/instancectl/client.go b/internal/instancectl/client.go| index 33dfa32cc..cae0ca4cb 100644 |
| --- a/internal/instancectl/client.go |
| +++ b/internal/instancectl/client.go |
| @@ -12,6 +12,8 @@ import ( |
| "strconv" |
| "strings" |
| "time" |
| + |
| + "github.com/carlosframework/platform/internal/version" |
| ) |
| // Deadlines per verb (design §5.3). A start or stop that exceeds its |
| @@ -34,9 +36,25 @@ type Client struct { |
| // Dial is the dialer; nil means net.Dialer. Tests do not need to override |
| // it, but a caller composing several clients may. |
| Dial func(ctx context.Context, path string) (net.Conn, error) |
| + // Build is the build this agent runs. Ready refuses a service whose |
| + // BuildHeader does not name it (#256); empty skips the comparison, |
| + // for tests that build a Client literal. NewClient sets version.Build. |
| + Build string |
| } |
| -func NewClient(path string) *Client { return &Client{Path: path} } |
| +func NewClient(path string) *Client { return &Client{Path: path, Build: version.Build} } |
| + |
| +// StaleBuildError is Ready's answer to a control service running a build |
| +// other than the agent's own: the service outlived a binary change and |
| +// serves this release's units with the previous release's code (#256). |
| +type StaleBuildError struct{ Running, Want string } |
| + |
| +func (e *StaleBuildError) Error() string { |
| + if e.Running == "" { |
| + return fmt.Sprintf("instance control: carlos-instance-control.service is running an older build than this agent (%s); restart carlos-instance-control.service", e.Want) |
| + } |
| + return fmt.Sprintf("instance control: carlos-instance-control.service is running build %s, this agent is %s; restart carlos-instance-control.service", e.Running, e.Want) |
| +} |
| // NotReadyError carries the probe verdict a 503 /ready returned. |
| type NotReadyError struct{ Report ProbeReport } |
| @@ -76,7 +94,7 @@ func (c *Client) httpClient() *http.Client { |
| // there is one. A non-2xx answer becomes an error carrying the server's |
| // message; a dial failure names the socket so a forgotten enable reads as |
| // what it is from carlos logs. |
| -func (c *Client) do(ctx context.Context, deadline time.Duration, method, path string, q url.Values, out any) (int, error) { |
| +func (c *Client) do(ctx context.Context, deadline time.Duration, method, path string, q url.Values, out any) (int, http.Header, error) { |
| ctx, cancel := context.WithTimeout(ctx, deadline) |
| defer cancel() |
| u := "http://control" + path |
| @@ -85,18 +103,18 @@ func (c *Client) do(ctx context.Context, deadline time.Duration, method, path st |
| } |
| req, err := http.NewRequestWithContext(ctx, method, u, nil) |
| if err != nil { |
| - return 0, err |
| + return 0, nil, err |
| } |
| resp, err := c.httpClient().Do(req) |
| if err != nil { |
| - return 0, fmt.Errorf("instance control socket %s: %w", c.Path, err) |
| + return 0, nil, fmt.Errorf("instance control socket %s: %w", c.Path, err) |
| } |
| defer resp.Body.Close() |
| body, _ := io.ReadAll(io.LimitReader(resp.Body, 64<<10)) |
| if resp.StatusCode == http.StatusServiceUnavailable && path == "/ready" { |
| var r ProbeReport |
| _ = json.Unmarshal(body, &r) |
| - return resp.StatusCode, &NotReadyError{Report: r} |
| + return resp.StatusCode, resp.Header, &NotReadyError{Report: r} |
| } |
| // A 502 from /sweep still carries the units it did and didn't manage to |
| // stop, so it is decoded into out here even though it is about to be |
| @@ -104,13 +122,13 @@ func (c *Client) do(ctx context.Context, deadline time.Duration, method, path st |
| // the failure, not instead of it. |
| if out != nil && len(body) > 0 && (resp.StatusCode/100 == 2 || (resp.StatusCode == http.StatusBadGateway && path == "/sweep")) { |
| if err := json.Unmarshal(body, out); err != nil { |
| - return resp.StatusCode, fmt.Errorf("instance control %s: %v", path, err) |
| + return resp.StatusCode, resp.Header, fmt.Errorf("instance control %s: %v", path, err) |
| } |
| } |
| if resp.StatusCode/100 != 2 { |
| - return resp.StatusCode, fmt.Errorf("instance control %s: %d: %s", path, resp.StatusCode, strings.TrimSpace(string(body))) |
| + return resp.StatusCode, resp.Header, fmt.Errorf("instance control %s: %d: %s", path, resp.StatusCode, strings.TrimSpace(string(body))) |
| } |
| - return resp.StatusCode, nil |
| + return resp.StatusCode, resp.Header, nil |
| } |
| func hostRole(host string, role Role) url.Values { |
| @@ -127,7 +145,7 @@ func (c *Client) Start(ctx context.Context, host string, role Role, memoryMax in |
| q := hostRole(host, role) |
| q.Set("memory_max", strconv.FormatInt(memoryMax, 10)) |
| var out StartResponse |
| - if _, err := c.do(ctx, DeadlineControl, http.MethodPost, "/start", q, &out); err != nil { |
| + if _, _, err := c.do(ctx, DeadlineControl, http.MethodPost, "/start", q, &out); err != nil { |
| return "", err |
| } |
| return out.InvocationID, nil |
| @@ -138,17 +156,17 @@ func (c *Client) Seed(ctx context.Context, host string, memoryMax int64) (SeedRe |
| q := hostRole(host, "") |
| q.Set("memory_max", strconv.FormatInt(memoryMax, 10)) |
| var out SeedResponse |
| - _, err := c.do(ctx, DeadlineSeed, http.MethodPost, "/seed", q, &out) |
| + _, _, err := c.do(ctx, DeadlineSeed, http.MethodPost, "/seed", q, &out) |
| return out, err |
| } |
| func (c *Client) Stop(ctx context.Context, host string, role Role) error { |
| - _, err := c.do(ctx, DeadlineControl, http.MethodPost, "/stop", hostRole(host, role), nil) |
| + _, _, err := c.do(ctx, DeadlineControl, http.MethodPost, "/stop", hostRole(host, role), nil) |
| return err |
| } |
| func (c *Client) Reset(ctx context.Context, host string, role Role) error { |
| - _, err := c.do(ctx, DeadlineControl, http.MethodPost, "/reset", hostRole(host, role), nil) |
| + _, _, err := c.do(ctx, DeadlineControl, http.MethodPost, "/reset", hostRole(host, role), nil) |
| return err |
| } |
| @@ -157,15 +175,24 @@ func (c *Client) Reset(ctx context.Context, host string, role Role) error { |
| // returned with the error. |
| func (c *Client) Sweep(ctx context.Context) (SweepResponse, error) { |
| var out SweepResponse |
| - status, err := c.do(ctx, DeadlineSweep, http.MethodPost, "/sweep", nil, &out) |
| + status, _, err := c.do(ctx, DeadlineSweep, http.MethodPost, "/sweep", nil, &out) |
| if err != nil && status == http.StatusBadGateway { |
| return out, errors.Join(err, errors.New("some instance units would not stop")) |
| } |
| return out, err |
| } |
| -// Ready is nil on 204 and a *NotReadyError on 503. |
| +// Ready is nil on 204 from a service of this agent's own build, a |
| +// *NotReadyError on 503, and a *StaleBuildError on a 204 whose BuildHeader |
| +// is missing or names another build (#256). The verdict is asked first: |
| +// a service that is both stale and not ready says why it is not ready. |
| func (c *Client) Ready(ctx context.Context) error { |
| - _, err := c.do(ctx, DeadlineReady, http.MethodGet, "/ready", nil, nil) |
| - return err |
| + _, hdr, err := c.do(ctx, DeadlineReady, http.MethodGet, "/ready", nil, nil) |
| + if err != nil { |
| + return err |
| + } |
| + if got := hdr.Get(BuildHeader); c.Build != "" && got != c.Build { |
| + return &StaleBuildError{Running: got, Want: c.Build} |
| + } |
| + return nil |
| } |
diff --git a/internal/instancectl/client_test.go b/internal/instancectl/client_test.go| index dc84c5837..a02f4e140 100644 |
| --- a/internal/instancectl/client_test.go |
| +++ b/internal/instancectl/client_test.go |
| @@ -119,3 +119,42 @@ func TestClientRefusesWhenTheSocketIsAbsent(t *testing.T) { |
| t.Fatalf("absent socket: %v", err) |
| } |
| } |
| + |
| +// The agent side of #256: a 204 from a service that reports another build, |
| +// or none at all (a release before the header), is not ready. Both are |
| +// the service carlos system apply or the roll script left running across |
| +// a binary change, and the error names the restart. |
| +func TestClientReadyRefusesAControlServiceOfAnotherBuild(t *testing.T) { |
| + ctl := &fakeCtl{} |
| + s, c := newClient(t, ctl, func(s *Server) { s.Build = "" }) |
| + b, _ := json.Marshal(ProbeReport{Ready: true, Lines: []VerdictLine{{Check: "pid1", Verdict: "ok"}}}) |
| + if err := os.WriteFile(s.ProbeOut, b, 0o644); err != nil { |
| + t.Fatal(err) |
| + } |
| + s.RunProbe(context.Background()) |
| + c.Build = "new1234" |
| + err := c.Ready(context.Background()) |
| + var stale *StaleBuildError |
| + if !errors.As(err, &stale) || stale.Running != "" || !strings.Contains(err.Error(), "older build") || !strings.Contains(err.Error(), "restart carlos-instance-control.service") { |
| + t.Fatalf("no header: Ready = %v", err) |
| + } |
| + s.Build = "old5678" |
| + err = c.Ready(context.Background()) |
| + if !errors.As(err, &stale) || stale.Running != "old5678" || !strings.Contains(err.Error(), "running build old5678, this agent is new1234") { |
| + t.Fatalf("other build: Ready = %v", err) |
| + } |
| + s.Build = "new1234" |
| + if err := c.Ready(context.Background()); err != nil { |
| + t.Fatalf("same build: Ready = %v", err) |
| + } |
| + // A client with no build of its own (a test's literal) does not judge. |
| + c.Build = "" |
| + s.Build = "" |
| + if err := c.Ready(context.Background()); err != nil { |
| + t.Fatalf("unjudged: Ready = %v", err) |
| + } |
| + // NewClient judges with this process's own build. |
| + if NewClient(c.Path).Build == "" { |
| + t.Fatal("NewClient left Build empty; production would never check") |
| + } |
| +} |
diff --git a/internal/instancectl/names.go b/internal/instancectl/names.go| index 7de0f994c..f07a94624 100644 |
| --- a/internal/instancectl/names.go |
| +++ b/internal/instancectl/names.go |
| @@ -168,6 +168,14 @@ type ProbeReport struct { |
| Lines []VerdictLine `json:"lines"` |
| } |
| +// BuildHeader carries the control service's version.Build on every /ready |
| +// answer (#256). The agent compares it with its own: both are meant to be |
| +// /opt/carlos/carlos, and a service that reports another build, or none |
| +// (a release before this header), is one carlos system apply or the roll |
| +// script left running across a binary change. Nested wakes refuse until |
| +// it is restarted, with that as the reason. |
| +const BuildHeader = "X-Carlos-Build" |
| + |
| // Responses of the control verbs (design §5.3). |
| type StartResponse struct { |
| InvocationID string `json:"invocation_id"` |
diff --git a/internal/instancectl/server.go b/internal/instancectl/server.go| index 58ef6c0d8..64b96d55c 100644 |
| --- a/internal/instancectl/server.go |
| +++ b/internal/instancectl/server.go |
| @@ -23,6 +23,7 @@ import ( |
| "github.com/carlosframework/platform/internal/cliflag" |
| "github.com/carlosframework/platform/internal/controlsock" |
| "github.com/carlosframework/platform/internal/provision" |
| + "github.com/carlosframework/platform/internal/version" |
| ) |
| // Server is the root instance-control service (design §5). It does four |
| @@ -78,6 +79,19 @@ type Server struct { |
| // constant: nothing from a request reaches it. |
| Run func(ctx context.Context, args ...string) (string, error) |
| + // Build is the build this process runs (version.Build), answered on |
| + // every /ready as the BuildHeader so the agent can see it talks to |
| + // its own release (#256). Empty sends no header, which a client of |
| + // this release reads as an older service. |
| + Build string |
| + // ExeCurrent reports whether this process still runs the installed |
| + // platform binary; nil skips the check. Production compares |
| + // /proc/self/exe with /opt/carlos/carlos by identity |
| + // (installedBinaryCurrent). Asked on every /ready, not once at probe |
| + // time, because the binary can change at any moment after the probe |
| + // ran: that is exactly the stale-service case of #256. |
| + ExeCurrent func() error |
| + |
| // cgroupWaitBudget bounds a wait for an empty cgroup; tests shorten it. |
| cgroupWaitBudget time.Duration |
| @@ -130,6 +144,8 @@ func runWithClaim(args []string, claim http.Handler) { |
| DataRoot: "/data/instances", |
| Statfs: statfsBytes, |
| DiskEvery: 30 * time.Second, |
| + Build: version.Build, |
| + ExeCurrent: installedBinaryCurrent, |
| } |
| ln, activated, err := activatedListener() |
| if err != nil { |
| @@ -578,6 +594,9 @@ func (s *Server) waitInactiveOrFailed(ctx context.Context, unit string, every ti |
| } |
| func (s *Server) handleReady(w http.ResponseWriter, r *http.Request) { |
| + if s.Build != "" { |
| + w.Header().Set(BuildHeader, s.Build) |
| + } |
| rep, ready := s.Ready() |
| if ready { |
| w.WriteHeader(http.StatusNoContent) |
| @@ -586,18 +605,50 @@ func (s *Server) handleReady(w http.ResponseWriter, r *http.Request) { |
| writeJSON(w, http.StatusServiceUnavailable, rep) |
| } |
| -// Ready returns the last probe verdict. Before the first probe has ever |
| -// run, there is no verdict to report, so a synthetic line names that |
| -// reason explicitly rather than answering with an empty Lines (final |
| -// review, plan 2, item 6): a caller reading /ready during startup should |
| -// see why, not a report that looks like every check silently passed. |
| +// Ready returns the last probe verdict, plus the one check that cannot |
| +// wait for the next probe: that this process still runs the installed |
| +// binary (ExeCurrent). Before the first probe has ever run, there is no |
| +// verdict to report, so a synthetic line names that reason explicitly |
| +// rather than answering with an empty Lines (final review, plan 2, item |
| +// 6): a caller reading /ready during startup should see why, not a report |
| +// that looks like every check silently passed. |
| func (s *Server) Ready() (ProbeReport, bool) { |
| s.mu.Lock() |
| - defer s.mu.Unlock() |
| - if !s.probed { |
| - return ProbeReport{Lines: []VerdictLine{{Check: "probe", Verdict: "fail", Detail: "not yet run"}}}, false |
| + rep, probed := s.report, s.probed |
| + s.mu.Unlock() |
| + if !probed { |
| + rep = ProbeReport{Lines: []VerdictLine{{Check: "probe", Verdict: "fail", Detail: "not yet run"}}} |
| + } |
| + if s.ExeCurrent != nil { |
| + if err := s.ExeCurrent(); err != nil { |
| + // A copy: the stored report must not grow a line per call. |
| + rep = ProbeReport{Ready: false, Lines: append(append([]VerdictLine(nil), rep.Lines...), |
| + VerdictLine{Check: "binary", Verdict: "fail", Detail: err.Error()})} |
| + } |
| + } |
| + return rep, rep.Ready |
| +} |
| + |
| +// installedBinaryCurrent is ExeCurrent in production: /proc/self/exe |
| +// resolves to the file this process was started from, even after that |
| +// file was renamed over (the kernel reports it as deleted), and |
| +// os.SameFile compares device and inode, so a service started from the |
| +// previous release's binary answers a mismatch the moment carlos system |
| +// apply renames the new one into place, and keeps answering it until it |
| +// is restarted (#256). |
| +func installedBinaryCurrent() error { |
| + running, err := os.Stat("/proc/self/exe") |
| + if err != nil { |
| + return fmt.Errorf("stat /proc/self/exe: %v", err) |
| + } |
| + installed, err := os.Stat(SystemBinary) |
| + if err != nil { |
| + return fmt.Errorf("stat %s: %v", SystemBinary, err) |
| + } |
| + if !os.SameFile(running, installed) { |
| + return fmt.Errorf("instance control is running a binary that is no longer %s; restart carlos-instance-control.service", SystemBinary) |
| } |
| - return s.report, s.report.Ready |
| + return nil |
| } |
| func (s *Server) waitBudget() time.Duration { |
diff --git a/internal/instancectl/server_test.go b/internal/instancectl/server_test.go| index aaf822f57..5ef76bc3c 100644 |
| --- a/internal/instancectl/server_test.go |
| +++ b/internal/instancectl/server_test.go |
| @@ -13,6 +13,8 @@ import ( |
| "sync" |
| "testing" |
| "time" |
| + |
| + "github.com/carlosframework/platform/internal/version" |
| ) |
| type fakeCtl struct { |
| @@ -96,6 +98,9 @@ func serveOnSocket(t *testing.T, ctl *fakeCtl, configure ...func(*Server)) (*Ser |
| FixtureOK: func() error { return nil }, |
| ProbeEvery: time.Hour, |
| Logf: t.Logf, |
| + // The same binary as the Client this test process is, as in |
| + // production, where both are /opt/carlos/carlos. |
| + Build: version.Build, |
| } |
| for _, configure := range configure { |
| configure(s) |
| @@ -630,3 +635,57 @@ func TestMountpointInspectionUnverifiedDoesNotBlock(t *testing.T) { |
| t.Error("a failing probe read as ready") |
| } |
| } |
| + |
| +// Discussion #256: carlos-instance-control.service ran the previous |
| +// release's binary for three weeks and served the new release's units. |
| +// /ready now asks, on every call, whether this process still runs the |
| +// installed binary, and a mismatch is a failing line whatever the probe |
| +// said; it names the restart that clears it. |
| +func TestReadyRefusesWhenTheRunningBinaryIsNotTheInstalledOne(t *testing.T) { |
| + ctl := &fakeCtl{} |
| + stale := errors.New("instance control is running a binary that is no longer /opt/carlos/carlos; restart carlos-instance-control.service") |
| + var exeErr error |
| + s, c := newServer(t, ctl, func(s *Server) { |
| + s.ExeCurrent = func() error { return exeErr } |
| + s.Build = "abc1234" |
| + }) |
| + b, _ := json.Marshal(ProbeReport{Ready: true, Lines: []VerdictLine{{Check: "pid1", Verdict: "ok"}}}) |
| + if err := os.WriteFile(s.ProbeOut, b, 0o644); err != nil { |
| + t.Fatal(err) |
| + } |
| + s.RunProbe(context.Background()) |
| + resp, err := c.Get("http://unix/ready") |
| + if err != nil || resp.StatusCode != 204 { |
| + t.Fatalf("current binary: %v %v", resp, err) |
| + } |
| + if got := resp.Header.Get(BuildHeader); got != "abc1234" { |
| + t.Fatalf("%s = %q, want the service's build", BuildHeader, got) |
| + } |
| + exeErr = stale |
| + resp, err = c.Get("http://unix/ready") |
| + if err != nil || resp.StatusCode != 503 { |
| + t.Fatalf("stale binary: %v %v", resp, err) |
| + } |
| + if got := resp.Header.Get(BuildHeader); got != "abc1234" { |
| + t.Fatalf("%s on a 503 = %q, want the service's build", BuildHeader, got) |
| + } |
| + var got ProbeReport |
| + if err := json.NewDecoder(resp.Body).Decode(&got); err != nil { |
| + t.Fatal(err) |
| + } |
| + if got.Ready || len(got.Lines) != 2 || got.Lines[1].Check != "binary" || got.Lines[1].Verdict != "fail" || !strings.Contains(got.Lines[1].Detail, "restart carlos-instance-control.service") { |
| + t.Fatalf("body = %+v", got) |
| + } |
| + // The stored report is not what grew: the next call carries one |
| + // binary line, not two. |
| + resp, _ = c.Get("http://unix/ready") |
| + got = ProbeReport{} |
| + _ = json.NewDecoder(resp.Body).Decode(&got) |
| + if len(got.Lines) != 2 { |
| + t.Fatalf("second call = %+v, want the probe line and one binary line", got) |
| + } |
| + exeErr = nil |
| + if resp, err := c.Get("http://unix/ready"); err != nil || resp.StatusCode != 204 { |
| + t.Fatalf("after the restart: %v %v", resp, err) |
| + } |
| +} |
diff --git a/internal/layout/extras.go b/internal/layout/extras.go| new file mode 100644 |
| index 000000000..5fef52ff6 |
| --- /dev/null |
| +++ b/internal/layout/extras.go |
| @@ -0,0 +1,357 @@ |
| +package layout |
| + |
| +import ( |
| + "errors" |
| + "fmt" |
| + "os" |
| + "path/filepath" |
| + "sort" |
| + "strings" |
| + |
| + "github.com/carlosframework/platform/internal/controlname" |
| +) |
| + |
| +// The entries an app makes for itself (discussion #256). |
| +// |
| +// The plan's nine artifacts are the files the PLATFORM names for a host. |
| +// A flat instance was started with cwd / and `--db /data/<host>.db`, and |
| +// an app that derives a directory from its database path makes more: |
| +// bdn/ips kept /data/<host>/instance.key and /data/<host>/report.token |
| +// and a /data/<host>-blobs directory beside its database. The first |
| +// migration moved the nine names and left those behind, the app started |
| +// nested, looked for them under its new database's directory, found |
| +// nothing, and minted a fresh key. Three apps did, on the flagship, on |
| +// the first day. |
| +// |
| +// So a move to nested takes every entry in the data directory that the |
| +// flat layout's naming rules make the host's, into the instance |
| +// directory at the same relative name, where `filepath.Dir(dbPath)` now |
| +// points (nested the database is <instance dir>/<host>.db, and the |
| +// instance sees that directory as /data/instances/<iid>); a revert brings |
| +// the same names back. Nothing the host could own is left behind |
| +// silently: an entry that cannot be moved safely refuses the move with |
| +// its name in the reason. These entries are scanned, not derived, so they |
| +// are recorded in the journal by name and re-derived from the host and |
| +// the name on recovery; a name a journal records must still pass the |
| +// same rules (extraNameOK), so a journal cannot name a move the pass |
| +// would not have chosen itself. |
| +// |
| +// The naming rules are the flat layout's own. A file is <host>'s when it |
| +// is <host> itself, <host> plus a dot or a dash and more (<host>.db, |
| +// <host>-sidecar.env, <host>-blobs), or the activator's own prefix form |
| +// of those (.activator-<host>.db-litestream). A name can match more than |
| +// one host on the box (`a.b.oncarlos.com-x` is `a.b.oncarlos.com` plus |
| +// `-x`, and also the host `a.b.oncarlos.com-x` itself), so an entry |
| +// belongs to the LONGEST host on the box whose rule it matches, and a |
| +// pass moves only what belongs to its own host. That is exact for every |
| +// host the box has a row for. A name that matches no row is not moved by |
| +// anyone and is reported (UnattributedEntries), because a shared |
| +// directory such as /data/blobs, written by every flat app whose cwd was |
| +// /, cannot be given to one host. |
| + |
| +// belongsTo reports whether name is one the flat layout's rules make |
| +// host's: the host itself, or the host with a platform or app suffix. |
| +func belongsTo(name, host string) bool { |
| + if name == host { |
| + return true |
| + } |
| + for _, prefix := range []string{host + ".", host + "-", ".activator-" + host + ".", ".activator-" + host + "-"} { |
| + if strings.HasPrefix(name, prefix) { |
| + return true |
| + } |
| + } |
| + return false |
| +} |
| + |
| +// ownerOf is the longest host among hosts whose rules name matches, or |
| +// "" when none does. |
| +func ownerOf(name string, hosts []string) string { |
| + owner := "" |
| + for _, h := range hosts { |
| + if len(h) > len(owner) && belongsTo(name, h) { |
| + owner = h |
| + } |
| + } |
| + return owner |
| +} |
| + |
| +// fixedNames is the base names of the plan's platform-named artifacts |
| +// (the same on both sides of a move; the socket is never a file the pass |
| +// moves and has a different name nested, so it is left out). |
| +func fixedNames(p *Plan) map[string]bool { |
| + names := map[string]bool{} |
| + for _, a := range p.Artifacts { |
| + if a.Extra || a.Name == "socket" { |
| + continue |
| + } |
| + names[filepath.Base(a.Src)] = true |
| + names[filepath.Base(a.Dst)] = true |
| + } |
| + return names |
| +} |
| + |
| +// fixedSuffixes are the platform's per-host file shapes under the flat |
| +// layout, X plus suffix (flatFiles), longest first so ".db.corrupt" is |
| +// read before ".db" and ".config.env" before ".env". |
| +var fixedSuffixes = []string{".db.corrupt", ".config.env", "-sidecar.env", ".db-wal", ".db-shm", ".db", ".env"} |
| + |
| +// fixedShapeHost reads the host X out of a name that has one of the |
| +// platform's fixed per-host shapes (X.db, X.env, X.config.env, |
| +// X-sidecar.env, X.db-wal, X.db-shm, X.db.corrupt, .activator-X.*), and |
| +// reports whether it has one. X must be a valid host shape with a dot in |
| +// it (every host the platform serves has one; registry.db and |
| +// litestream.env are the box's, not a host's); a name that is not |
| +// host-shaped under any suffix is nobody's platform file. |
| +func fixedShapeHost(name string) (string, bool) { |
| + hostLike := func(x string) bool { return strings.Contains(x, ".") && controlname.ValidHostShape(x) } |
| + if rest, ok := strings.CutPrefix(name, ".activator-"); ok { |
| + if x, ok := fixedShapeHost(rest); ok { |
| + return x, true |
| + } |
| + if i := strings.Index(rest, ".db-"); i > 0 && hostLike(rest[:i]) { |
| + return rest[:i], true |
| + } |
| + return "", false |
| + } |
| + for _, suffix := range fixedSuffixes { |
| + if x, ok := strings.CutSuffix(name, suffix); ok && x != "" { |
| + if hostLike(x) { |
| + return x, true |
| + } |
| + return "", false |
| + } |
| + } |
| + return "", false |
| +} |
| + |
| +// knownHosts is every host an entry can belong to: the box's rows, and |
| +// every host the directory's own platform-named files are evidence of. A |
| +// host deleted from the box, or moved away, leaves X.env (its admin |
| +// token), X.config.env, X-blobs and the rest behind under its name, and |
| +// when X extends this host's name (app.example.com.au beside |
| +// app.example.com; <host>-staging beside <host>) a scan that knew only |
| +// the rows would give them to the shorter host. Its files say X existed, |
| +// so X owns what its rules match, exactly as a row would. |
| +func knownHosts(entries []os.DirEntry, rows []string) []string { |
| + seen := map[string]bool{} |
| + var hosts []string |
| + for _, h := range rows { |
| + if !seen[h] { |
| + seen[h] = true |
| + hosts = append(hosts, h) |
| + } |
| + } |
| + for _, e := range entries { |
| + if x, ok := fixedShapeHost(e.Name()); ok && !seen[x] { |
| + seen[x] = true |
| + hosts = append(hosts, x) |
| + } |
| + } |
| + return hosts |
| +} |
| + |
| +// hostEntries sorts a directory's entries that match host's rules into |
| +// the ones host owns (fixed platform names of host itself left out when |
| +// skipFixed) and the ones a longer known host owns, by owner. |
| +func hostEntries(entries []os.DirEntry, host string, known []string, skipFixed map[string]bool) (ours []string, others map[string]string) { |
| + others = map[string]string{} |
| + for _, e := range entries { |
| + name := e.Name() |
| + if skipFixed[name] || !belongsTo(name, host) { |
| + continue |
| + } |
| + if owner := ownerOf(name, known); owner != host { |
| + others[name] = owner |
| + continue |
| + } |
| + ours = append(ours, name) |
| + } |
| + sort.Strings(ours) |
| + return ours, others |
| +} |
| + |
| +// reportOthers says, once per scan, what matched the host's rules but |
| +// belongs to a longer host, and whether that host still has a row: the |
| +// operator reads which files a departed host left in the shared |
| +// directory. Sorted by name so a log reads the same way twice. |
| +func reportOthers(p *Plan, dir string, others map[string]string, rows []string) { |
| + if len(others) == 0 { |
| + return |
| + } |
| + hasRow := map[string]bool{} |
| + for _, h := range rows { |
| + hasRow[h] = true |
| + } |
| + names := make([]string, 0, len(others)) |
| + for name := range others { |
| + names = append(names, name) |
| + } |
| + sort.Strings(names) |
| + var parts []string |
| + for _, name := range names { |
| + owner := others[name] |
| + if hasRow[owner] { |
| + parts = append(parts, fmt.Sprintf("%s (%s's)", name, owner)) |
| + } else { |
| + parts = append(parts, fmt.Sprintf("%s (%s's, which has no row on this box)", name, owner)) |
| + } |
| + } |
| + p.logf("layout: %s: left in %s, named for another host: %s", p.Host, dir, strings.Join(parts, ", ")) |
| +} |
| + |
| +// extraNameOK is the rule a recorded or scanned extra name must pass: one |
| +// path element, not one of the plan's own names, and host's by belongsTo. |
| +// It is what keeps a journal from being an instruction (design §3.3): a |
| +// name that fails it is refused however it got into the journal. |
| +func extraNameOK(p *Plan, name string) error { |
| + if name == "" || name == "." || name == ".." || strings.ContainsAny(name, "/\x00") { |
| + return fmt.Errorf("layout: %s: %q is not a file name", p.Host, name) |
| + } |
| + if fixedNames(p)[name] { |
| + return fmt.Errorf("layout: %s: %q is one of the move's own names, not an extra", p.Host, name) |
| + } |
| + if !belongsTo(name, p.Host) { |
| + return fmt.Errorf("layout: %s: %q is not named for the host", p.Host, name) |
| + } |
| + return nil |
| +} |
| + |
| +// extraDirs is where a host's extra entries are before and after this |
| +// plan's move: the data directory and the raw instance directory, in the |
| +// plan's direction. |
| +func (p *Plan) extraDirs() (src, dst string) { |
| + flat, nested := p.DataDir, rawInstanceDir(p.DataDir, p.Host) |
| + if p.Mode == ModeFlat { |
| + return nested, flat |
| + } |
| + return flat, nested |
| +} |
| + |
| +// AddExtras appends the named entries to the plan as extra artifacts, |
| +// present, at their source and destination for the plan's direction. |
| +// Each name must pass extraNameOK, and no name twice. |
| +func (p *Plan) AddExtras(names []string) error { |
| + src, dst := p.extraDirs() |
| + seen := map[string]bool{} |
| + for _, a := range p.Artifacts { |
| + if a.Extra { |
| + seen[a.Name] = true |
| + } |
| + } |
| + for _, name := range names { |
| + if err := extraNameOK(p, name); err != nil { |
| + return err |
| + } |
| + if seen[name] { |
| + return fmt.Errorf("layout: %s: extra %q listed twice", p.Host, name) |
| + } |
| + seen[name] = true |
| + p.Artifacts = append(p.Artifacts, Artifact{ |
| + Name: name, Src: filepath.Join(src, name), Dst: filepath.Join(dst, name), |
| + Extra: true, Present: true, |
| + }) |
| + } |
| + return nil |
| +} |
| + |
| +// ExtraNames is the names of the plan's extra artifacts, in plan order. |
| +func (p *Plan) ExtraNames() []string { |
| + var names []string |
| + for _, a := range p.Artifacts { |
| + if a.Extra { |
| + names = append(names, a.Name) |
| + } |
| + } |
| + return names |
| +} |
| + |
| +// ScanExtras lists the entries in the plan's source directory that belong |
| +// to the host and that the plan does not already name, sorted. hosts is |
| +// every host with a row on the box; with the hosts the directory's own |
| +// platform files are evidence of (knownHosts), an entry a longer host's |
| +// rule also matches is that host's, is left alone, and is reported. An |
| +// entry whose destination is already taken is a refusal naming both, |
| +// never a choice: on a migration the destination check has just cleared |
| +// the instance directory of everything the plan did not name, so a taken |
| +// destination was planted since (a hold, like any planted destination); |
| +// on a revert it is a flat file the pass will not overwrite. |
| +// |
| +// The scan runs under the fence, with the host stopped, so the listing is |
| +// the whole of what the app made; and inside the serialized section with |
| +// the journal write, so what it records is what the journal records. |
| +func ScanExtras(p *Plan, hosts []string) ([]string, error) { |
| + src, dst := p.extraDirs() |
| + entries, err := os.ReadDir(src) |
| + if errors.Is(err, os.ErrNotExist) { |
| + return nil, nil |
| + } |
| + if err != nil { |
| + return nil, fmt.Errorf("layout: %s: list %s: %w", p.Host, src, err) |
| + } |
| + rows := append([]string{p.Host}, hosts...) |
| + names, others := hostEntries(entries, p.Host, knownHosts(entries, rows), fixedNames(p)) |
| + reportOthers(p, src, others, rows) |
| + for _, name := range names { |
| + taken, err := lstatPresent(filepath.Join(dst, name)) |
| + if err != nil { |
| + return nil, fmt.Errorf("layout: %s: %w", p.Host, err) |
| + } |
| + if taken { |
| + return nil, fmt.Errorf("layout: %s: %s exists at both %s and %s; refusing to choose", p.Host, name, filepath.Join(src, name), filepath.Join(dst, name)) |
| + } |
| + } |
| + return names, nil |
| +} |
| + |
| +// FlatLeftovers lists what a host that is already nested still has in |
| +// the data directory under its own name: the entries a move made before |
| +// #256 left behind, platform-named ones included (a <host>.db there |
| +// beside the nested one is the thing an operator most wants to know |
| +// about). Ownership is judged as the scan judges it. Nothing is moved; |
| +// the pass reports these on every pass for a nested host (convergeHost). |
| +func FlatLeftovers(dataDir, host string, hosts []string) []string { |
| + entries, err := os.ReadDir(dataDir) |
| + if err != nil { |
| + return nil |
| + } |
| + ours, _ := hostEntries(entries, host, knownHosts(entries, append([]string{host}, hosts...)), nil) |
| + for i, name := range ours { |
| + ours[i] = filepath.Join(dataDir, name) |
| + } |
| + return ours |
| +} |
| + |
| +// platformNames are the entries the platform itself keeps directly under |
| +// the data directory. UnattributedEntries leaves them out of its report. |
| +var platformNames = map[string]bool{ |
| + "registry.db": true, "registry.db-wal": true, "registry.db-shm": true, |
| + "instances": true, ".instances": true, "layout": true, "system": true, |
| + "moves": true, ".moves": true, ".hostexec": true, "litestream.env": true, |
| + "lost+found": true, |
| +} |
| + |
| +// UnattributedEntries lists the entries directly under dataDir that no |
| +// host on the box owns and that are not the platform's own: what a flat |
| +// app wrote relative to its working directory, which was /, or under a |
| +// name of its own choosing. The pass cannot move these, because it |
| +// cannot tell whose they are (a /data/blobs that several apps wrote into |
| +// is the live example), so it reports them for an operator to place by |
| +// hand. Sorted; nil when there are none. |
| +func UnattributedEntries(dataDir string, hosts []string) []string { |
| + entries, err := os.ReadDir(dataDir) |
| + if err != nil { |
| + return nil |
| + } |
| + // Rows only, not knownHosts: a departed host's files are exactly what |
| + // this report is for, and "no host on this box" is true of them. |
| + var out []string |
| + for _, e := range entries { |
| + name := e.Name() |
| + if platformNames[name] || ownerOf(name, hosts) != "" { |
| + continue |
| + } |
| + out = append(out, filepath.Join(dataDir, name)) |
| + } |
| + sort.Strings(out) |
| + return out |
| +} |
diff --git a/internal/layout/extras_pass_linux_test.go b/internal/layout/extras_pass_linux_test.go| new file mode 100644 |
| index 000000000..56ce10c82 |
| --- /dev/null |
| +++ b/internal/layout/extras_pass_linux_test.go |
| @@ -0,0 +1,327 @@ |
| +//go:build linux |
| + |
| +package layout |
| + |
| +import ( |
| + "context" |
| + "errors" |
| + "fmt" |
| + "os" |
| + "path/filepath" |
| + "strings" |
| + "testing" |
| + |
| + "github.com/carlosframework/platform/internal/registry" |
| +) |
| + |
| +// seedAppEntries writes what an app makes for itself beside its flat |
| +// database (discussion #256): a directory named for the host holding its |
| +// key, a `-blobs` directory, and a dot-suffixed file. |
| +func seedAppEntries(t *testing.T, dd, h string) { |
| + t.Helper() |
| + for path, body := range map[string]string{ |
| + filepath.Join(dd, h, "instance.key"): "key " + h, |
| + filepath.Join(dd, h, "report.token"): "token " + h, |
| + filepath.Join(dd, h+"-blobs", "ab", "cd"): "blob " + h, |
| + filepath.Join(dd, h+".cache"): "cache " + h, |
| + } { |
| + if err := os.MkdirAll(filepath.Dir(path), 0o700); err != nil { |
| + t.Fatal(err) |
| + } |
| + if err := os.WriteFile(path, []byte(body), 0o600); err != nil { |
| + t.Fatal(err) |
| + } |
| + } |
| +} |
| + |
| +// Discussion #256: the first migration moved the nine platform names and |
| +// left behind /data/<host>/instance.key and /data/<host>-blobs; the app |
| +// started nested without them and minted a new key. Every entry the flat |
| +// naming rules make the host's now moves into the instance directory at |
| +// the same relative name, and comes back on revert. An entry whose name a |
| +// longer host owns (`<host>-x` is the host `<host>-x` itself) stays, and so |
| +// does a shared directory no host owns, which is reported. |
| +func TestConvergeMovesAppCreatedEntriesWithTheHost(t *testing.T) { |
| + p, f := box(t) |
| + var logged []string |
| + p.Logf = func(format string, a ...any) { logged = append(logged, fmt.Sprintf(format, a...)) } |
| + seedAppEntries(t, p.DataDir, host) |
| + // A route whose name is host plus "-x": an externally supervised one |
| + // (not activator-managed), so its own entries must be left alone. |
| + decoy := host + "-x" |
| + if err := p.Registry.Upsert(registry.Route{Host: decoy, Kind: "instance", Socket: filepath.Join(p.SocketDir, decoy+".sock")}); err != nil { |
| + t.Fatal(err) |
| + } |
| + seedAppEntries(t, p.DataDir, decoy) |
| + // What a flat app with cwd / wrote as ./blobs. |
| + if err := os.MkdirAll(filepath.Join(p.DataDir, "blobs", "aa"), 0o700); err != nil { |
| + t.Fatal(err) |
| + } |
| + |
| + res := byHost(t, p.Converge(context.Background(), ModeNested)) |
| + if res[host].Outcome != "migrated" { |
| + t.Fatalf("%s: %+v", host, res[host]) |
| + } |
| + raw := rawInstanceDir(p.DataDir, host) |
| + for rel, want := range map[string]string{ |
| + filepath.Join(host, "instance.key"): "key " + host, |
| + filepath.Join(host, "report.token"): "token " + host, |
| + filepath.Join(host+"-blobs", "ab", "cd"): "blob " + host, |
| + host + ".cache": "cache " + host, |
| + } { |
| + if b, err := os.ReadFile(filepath.Join(raw, rel)); err != nil || string(b) != want { |
| + t.Errorf("%s not in the instance directory: %q %v", rel, b, err) |
| + } |
| + if _, err := os.Lstat(filepath.Join(p.DataDir, rel)); !errors.Is(err, os.ErrNotExist) { |
| + t.Errorf("%s left behind in the data directory: %v", rel, err) |
| + } |
| + } |
| + for _, rel := range []string{filepath.Join(decoy, "instance.key"), filepath.Join(decoy+"-blobs", "ab", "cd"), decoy + ".cache", filepath.Join("blobs", "aa")} { |
| + if _, err := os.Lstat(filepath.Join(p.DataDir, rel)); err != nil { |
| + t.Errorf("%s, not this host's, was moved: %v", rel, err) |
| + } |
| + if _, err := os.Lstat(filepath.Join(raw, rel)); !errors.Is(err, os.ErrNotExist) { |
| + t.Errorf("%s, not this host's, is in the instance directory: %v", rel, err) |
| + } |
| + } |
| + if JournalPresent(p.DataDir, host) || f.fences[host] != 0 { |
| + t.Fatal("journal or fence left after the move") |
| + } |
| + if n := p.Note(host); !strings.Contains(n, "moved with it: "+host+", "+host+"-blobs, "+host+".cache") { |
| + t.Errorf("note = %q", n) |
| + } |
| + // The aside directory holds nothing: these were moved, not quarantined. |
| + if q, err := os.ReadDir(asideDir(p.DataDir, host)); err == nil && len(q) > 0 { |
| + t.Errorf("quarantined instead of moved: %v", q) |
| + } |
| + joined := strings.Join(logged, "\n") |
| + if !strings.Contains(joined, "owned by no host on this box") || !strings.Contains(joined, filepath.Join(p.DataDir, "blobs")) { |
| + t.Errorf("the shared directory was not reported:\n%s", joined) |
| + } |
| + if strings.Contains(joined, filepath.Join(p.DataDir, decoy)) { |
| + t.Errorf("the decoy host's own directory was reported as unowned:\n%s", joined) |
| + } |
| + if !strings.Contains(joined, "named for another host: "+decoy+" ("+decoy+"'s), "+decoy+"-blobs ("+decoy+"'s)") { |
| + t.Errorf("the decoy host's entries were not reported as left for it:\n%s", joined) |
| + } |
| + |
| + // Revert brings the same names back, and only those. |
| + res = byHost(t, p.Converge(context.Background(), ModeFlat)) |
| + if res[host].Outcome != "reverted" { |
| + t.Fatalf("revert %s: %+v", host, res[host]) |
| + } |
| + assertFlatUntouched(t, p, host) |
| + for rel, want := range map[string]string{ |
| + filepath.Join(host, "instance.key"): "key " + host, |
| + filepath.Join(host+"-blobs", "ab", "cd"): "blob " + host, |
| + host + ".cache": "cache " + host, |
| + } { |
| + if b, err := os.ReadFile(filepath.Join(p.DataDir, rel)); err != nil || string(b) != want { |
| + t.Errorf("%s not back in the data directory: %q %v", rel, b, err) |
| + } |
| + } |
| + if entries, _ := os.ReadDir(raw); len(entries) != 0 { |
| + t.Errorf("instance directory not emptied by the revert: %v", entries) |
| + } |
| +} |
| + |
| +// The same protections the platform names get, on the way in: a file |
| +// inside an app's directory that has another name OUTSIDE it would be |
| +// bound read-write into the namespace while something else still reaches |
| +// it, so the move is refused with the inode and the find, and the host is |
| +// held (a tenant can plant exactly this). A hard link whose every name is |
| +// inside the directory is the app's own deduplication and moves. |
| +func TestConvergeRefusesAnAppEntryWithANameOutsideIt(t *testing.T) { |
| + p, f := box(t) |
| + seedAppEntries(t, p.DataDir, host) |
| + secret := filepath.Join(p.DataDir, "litestream.env") |
| + if err := os.WriteFile(secret, []byte("AWS_SECRET_ACCESS_KEY=x\n"), 0o600); err != nil { |
| + t.Fatal(err) |
| + } |
| + if err := os.Link(secret, filepath.Join(p.DataDir, host, "creds")); err != nil { |
| + t.Skipf("hard links unavailable here: %v", err) |
| + } |
| + // And a legitimate pair, both names inside the blob store. |
| + if err := os.Link(filepath.Join(p.DataDir, host+"-blobs", "ab", "cd"), filepath.Join(p.DataDir, host+"-blobs", "dup")); err != nil { |
| + t.Fatal(err) |
| + } |
| + res := byHost(t, p.Converge(context.Background(), ModeNested)) |
| + r := res[host] |
| + if r.Outcome != "held" || !strings.Contains(r.Reason, "creds") || !strings.Contains(r.Reason, "2 names") || !strings.Contains(r.Reason, "only 1 of them are inside") || !strings.Contains(r.Reason, "-samefile") { |
| + t.Fatalf("linked outside: %+v", r) |
| + } |
| + assertFlatUntouched(t, p, host) |
| + if f.fences[host] == 0 { |
| + t.Fatal("the fence was lifted; the host would wake flat with the link in place") |
| + } |
| + if _, err := os.Stat(filepath.Join(p.DataDir, host, "instance.key")); err != nil { |
| + t.Fatalf("a refused move touched the app's directory: %v", err) |
| + } |
| + // The operator removes the outside name; the inside pair is fine. |
| + os.Remove(filepath.Join(p.DataDir, host, "creds")) |
| + res = byHost(t, p.Converge(context.Background(), ModeNested)) |
| + if res[host].Outcome != "migrated" || f.fences[host] != 0 { |
| + t.Fatalf("after the link is gone: %+v fence=%d", res[host], f.fences[host]) |
| + } |
| + raw := rawInstanceDir(p.DataDir, host) |
| + a, err1 := os.Stat(filepath.Join(raw, host+"-blobs", "ab", "cd")) |
| + b, err2 := os.Stat(filepath.Join(raw, host+"-blobs", "dup")) |
| + if err1 != nil || err2 != nil || !os.SameFile(a, b) { |
| + t.Fatalf("the app's own hard-link pair did not move together: %v %v", err1, err2) |
| + } |
| +} |
| + |
| +// A symlink inside an app's directory is refused like one anywhere else |
| +// the pass moves, and the host is held. |
| +func TestConvergeRefusesASymlinkInsideAnAppEntry(t *testing.T) { |
| + p, f := box(t) |
| + seedAppEntries(t, p.DataDir, host) |
| + if err := os.Symlink("/etc/passwd", filepath.Join(p.DataDir, host, "link")); err != nil { |
| + t.Fatal(err) |
| + } |
| + res := byHost(t, p.Converge(context.Background(), ModeNested)) |
| + if r := res[host]; r.Outcome != "held" || !strings.Contains(r.Reason, "link") || !strings.Contains(r.Reason, "not a regular file") { |
| + t.Fatalf("symlink: %+v", r) |
| + } |
| + if f.fences[host] == 0 { |
| + t.Fatal("the fence was lifted") |
| + } |
| + assertFlatUntouched(t, p, host) |
| +} |
| + |
| +// An extra whose destination is already taken is refused before the |
| +// journal, naming both, never a choice. On the way to nested the |
| +// destination check has just cleared the instance directory, so a taken |
| +// destination was planted since (here: inside the serialized section, |
| +// after the check and before the scan), and the host is held. |
| +func TestConvergeRefusesAnAppEntryWhoseDestinationIsTaken(t *testing.T) { |
| + p, f := box(t) |
| + seedAppEntries(t, p.DataDir, host) |
| + p.Serialize = func(fn func()) { |
| + if !JournalPresent(p.DataDir, host) { |
| + os.MkdirAll(filepath.Join(rawInstanceDir(p.DataDir, host), host), 0o700) |
| + } |
| + fn() |
| + } |
| + res := byHost(t, p.Converge(context.Background(), ModeNested)) |
| + r := res[host] |
| + if r.Outcome != "held" || !strings.Contains(r.Reason, "exists at both") || !strings.Contains(r.Reason, "refusing to choose") { |
| + t.Fatalf("taken destination: %+v", r) |
| + } |
| + if f.fences[host] == 0 { |
| + t.Fatal("the fence was lifted") |
| + } |
| + assertFlatUntouched(t, p, host) |
| + if _, err := os.Stat(filepath.Join(p.DataDir, host, "instance.key")); err != nil { |
| + t.Fatalf("a refused move touched the app's directory: %v", err) |
| + } |
| +} |
| + |
| +// Recovery completes a move with extras from its journal: the names are |
| +// recorded, re-derived on recovery, and moved with the rest. |
| +func TestRecoverCompletesAMoveWithAppEntries(t *testing.T) { |
| + for _, step := range []string{"after-journal", "mid-files", "after-files"} { |
| + t.Run(step, func(t *testing.T) { |
| + p, _ := box(t) |
| + seedAppEntries(t, p.DataDir, host) |
| + p.abortAt = step |
| + p.Converge(context.Background(), ModeNested) |
| + p.abortAt = "" |
| + j, found, err := ReadJournal(p.DataDir, host) |
| + if err != nil || !found || len(j.Extra) != 3 { |
| + t.Fatalf("journal = %+v %v %v, want three extras recorded", j, found, err) |
| + } |
| + res := byHost(t, p.Recover(context.Background(), ModeNested)) |
| + if res[host].Outcome != "recovered" { |
| + t.Fatalf("recover = %+v", res[host]) |
| + } |
| + raw := rawInstanceDir(p.DataDir, host) |
| + for _, rel := range []string{filepath.Join(host, "instance.key"), filepath.Join(host+"-blobs", "ab", "cd"), host + ".cache"} { |
| + if _, err := os.Stat(filepath.Join(raw, rel)); err != nil { |
| + t.Errorf("%s not at destination after recovery from %s: %v", rel, step, err) |
| + } |
| + if _, err := os.Lstat(filepath.Join(p.DataDir, rel)); !errors.Is(err, os.ErrNotExist) { |
| + t.Errorf("%s left behind after recovery from %s", rel, step) |
| + } |
| + } |
| + if JournalPresent(p.DataDir, host) { |
| + t.Error("journal remains") |
| + } |
| + }) |
| + } |
| +} |
| + |
| +// A forged journal for host H that names H-x, the directory of the host |
| +// H-x itself, is refused on recovery: the name passes the shape rule, but |
| +// ownership is judged against the box's rows as they stand, and it is not |
| +// H's. The host stays fenced, nothing moves. |
| +func TestRecoverRefusesAJournalNamingAnotherHostsEntry(t *testing.T) { |
| + p, _ := box(t) |
| + decoy := host + "-x" |
| + if err := p.Registry.Upsert(registry.Route{Host: decoy, Kind: "instance", Socket: filepath.Join(p.SocketDir, decoy+".sock")}); err != nil { |
| + t.Fatal(err) |
| + } |
| + seedAppEntries(t, p.DataDir, decoy) |
| + plan, err := PlanFor(p.DataDir, p.SocketDir, host, ModeNested) |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + present := make([]bool, len(plan.Artifacts)) |
| + present[0] = true |
| + j := plan.Journal(present) |
| + j.Extra = []JournalArtifact{{Name: decoy, Present: true}} |
| + if err := WriteJournal(p.DataDir, j); err != nil { |
| + t.Fatal(err) |
| + } |
| + res := byHost(t, p.Recover(context.Background(), ModeNested)) |
| + if r := res[host]; r.Outcome != "refused" || !strings.Contains(r.Reason, "belongs to "+decoy) || !strings.HasSuffix(r.Reason, "not acting") { |
| + t.Fatalf("forged extra: %+v", r) |
| + } |
| + if _, err := os.Stat(filepath.Join(p.DataDir, decoy, "instance.key")); err != nil { |
| + t.Fatalf("the other host's directory was moved: %v", err) |
| + } |
| + if !JournalPresent(p.DataDir, host) { |
| + t.Fatal("the refused journal was removed") |
| + } |
| +} |
| + |
| +// Review (#256 B): the hosts migrated before the pass carried the app's |
| +// entries along (the 104 on IE) still have them in the data directory, |
| +// where the namespace cannot see them. The pass never moves a nested |
| +// host's files, but it says so on every pass, in the log and in the |
| +// host's status note, until an operator has moved them. |
| +func TestConvergeReportsFlatLeftoversOfANestedHost(t *testing.T) { |
| + p, _ := box(t) |
| + if res := byHost(t, p.Converge(context.Background(), ModeNested)); res[host].Outcome != "migrated" { |
| + t.Fatalf("setup: %+v", res[host]) |
| + } |
| + // What an earlier release left behind: the app's directory, and a |
| + // platform-named file minted at the flat path since. |
| + seedAppEntries(t, p.DataDir, host) |
| + if err := os.WriteFile(filepath.Join(p.DataDir, host+".env"), []byte("CARLOS_ADMIN_TOKEN=old\n"), 0o600); err != nil { |
| + t.Fatal(err) |
| + } |
| + var logged []string |
| + p.Logf = func(f string, a ...any) { logged = append(logged, fmt.Sprintf(f, a...)) } |
| + res := byHost(t, p.Converge(context.Background(), ModeNested)) |
| + if res[host].Outcome != "unchanged" { |
| + t.Fatalf("%+v", res[host]) |
| + } |
| + want := "still in " + p.DataDir + " from before its move, where the instance cannot see them: " + |
| + filepath.Join(p.DataDir, host) + ", " + filepath.Join(p.DataDir, host+"-blobs") + ", " + filepath.Join(p.DataDir, host+".cache") + ", " + filepath.Join(p.DataDir, host+".env") + |
| + ". move what the app needs into " + rawInstanceDir(p.DataDir, host) + " by hand" |
| + if j := strings.Join(logged, "\n"); !strings.Contains(j, "layout: "+host+": "+want) { |
| + t.Errorf("log:\n%s\nwant a line with:\n%s", j, want) |
| + } |
| + if n := p.Note(host); n != want { |
| + t.Errorf("note = %q", n) |
| + } |
| + for _, rel := range []string{filepath.Join(host, "instance.key"), host + ".env"} { |
| + if _, err := os.Stat(filepath.Join(p.DataDir, rel)); err != nil { |
| + t.Errorf("a report moved %s: %v", rel, err) |
| + } |
| + } |
| + if n := p.Note(replica); strings.Contains(n, "still in") { |
| + t.Errorf("the replica, with nothing left behind, got the note: %q", n) |
| + } |
| +} |
diff --git a/internal/layout/extras_test.go b/internal/layout/extras_test.go| new file mode 100644 |
| index 000000000..42b105af4 |
| --- /dev/null |
| +++ b/internal/layout/extras_test.go |
| @@ -0,0 +1,194 @@ |
| +//go:build linux |
| + |
| +package layout |
| + |
| +import ( |
| + "fmt" |
| + "os" |
| + "path/filepath" |
| + "strings" |
| + "testing" |
| + |
| + "github.com/carlosframework/platform/internal/registry" |
| +) |
| + |
| +// The naming rules of extras.go: an entry is a host's when it is the host |
| +// or the host plus a dot or dash suffix (or the activator's prefix form), |
| +// and when two hosts on the box both match, the longer one owns it. |
| +func TestExtraOwnershipIsTheLongestMatchingHost(t *testing.T) { |
| + hosts := []string{"a.b.oncarlos.com", "a.b.oncarlos.com-x", "other.example.com"} |
| + for name, want := range map[string]string{ |
| + "a.b.oncarlos.com": "a.b.oncarlos.com", |
| + "a.b.oncarlos.com-blobs": "a.b.oncarlos.com", |
| + "a.b.oncarlos.com.cache": "a.b.oncarlos.com", |
| + ".activator-a.b.oncarlos.com.db-litestream": "a.b.oncarlos.com", |
| + "a.b.oncarlos.com-x": "a.b.oncarlos.com-x", |
| + "a.b.oncarlos.com-x-blobs": "a.b.oncarlos.com-x", |
| + "a.b.oncarlos.com-x.db": "a.b.oncarlos.com-x", |
| + "a.b.oncarlos.comx": "", |
| + "blobs": "", |
| + "registry.db": "", |
| + } { |
| + if got := ownerOf(name, hosts); got != want { |
| + t.Errorf("ownerOf(%q) = %q, want %q", name, got, want) |
| + } |
| + } |
| +} |
| + |
| +// A journal's extra names pass the scan's own rule or the journal is |
| +// refused: one path element, not one of the plan's own names, named for |
| +// the host, and never absent. Journals from before #256 have no extras. |
| +func TestValidateJudgesTheJournalsExtraNames(t *testing.T) { |
| + dd, rd := t.TempDir(), t.TempDir() |
| + good := func() *Journal { |
| + p, _ := PlanFor(dd, rd, host, ModeNested) |
| + present := make([]bool, len(artifactNames)) |
| + present[0] = true |
| + return p.Journal(present) |
| + } |
| + row := registry.Route{Host: host, DBPath: good().From.DB, Socket: good().From.Socket} |
| + j := good() |
| + j.Extra = []JournalArtifact{{Name: host, Present: true}, {Name: host + "-blobs", Present: true}} |
| + if _, err := Validate(j, dd, rd, row); err != nil { |
| + t.Fatalf("two good extras: %v", err) |
| + } |
| + for name, extra := range map[string]JournalArtifact{ |
| + "a path": {Name: "../" + host, Present: true}, |
| + "a fixed name": {Name: host + ".db", Present: true}, |
| + "another host's": {Name: replica + "-blobs", Present: true}, |
| + "the data dir": {Name: ".", Present: true}, |
| + "recorded absent": {Name: host + "-blobs", Present: false}, |
| + } { |
| + j := good() |
| + j.Extra = []JournalArtifact{extra} |
| + if _, err := Validate(j, dd, rd, row); err == nil || !strings.HasSuffix(err.Error(), "not acting") { |
| + t.Errorf("%s (%q): Validate = %v, want a refusal ending in not acting", name, extra.Name, err) |
| + } |
| + } |
| + j = good() |
| + j.Extra = []JournalArtifact{{Name: host, Present: true}, {Name: host, Present: true}} |
| + if _, err := Validate(j, dd, rd, row); err == nil || !strings.Contains(err.Error(), "twice") { |
| + t.Errorf("a name twice: Validate = %v", err) |
| + } |
| + // The wire shape is additive: a journal without the field reads as |
| + // one with no extras, and one with the field round-trips. |
| + j = good() |
| + j.Extra = []JournalArtifact{{Name: host + "-blobs", Present: true}} |
| + if err := WriteJournal(dd, j); err != nil { |
| + t.Fatal(err) |
| + } |
| + back, found, err := ReadJournal(dd, host) |
| + if err != nil || !found || len(back.Extra) != 1 || back.Extra[0].Name != host+"-blobs" { |
| + t.Fatalf("round trip = %+v %v %v", back, found, err) |
| + } |
| +} |
| + |
| +// UnattributedEntries names what no host on the box owns and the |
| +// platform did not make: the shared directories a flat app with cwd / |
| +// wrote, which no move can take. |
| +func TestUnattributedEntriesSkipsHostsAndPlatformNames(t *testing.T) { |
| + dd := t.TempDir() |
| + for _, name := range []string{"blobs", "registry.db", "layout", ".instances", "instances", host, host + "-blobs", "cache.tmp"} { |
| + if err := os.MkdirAll(filepath.Join(dd, name), 0o700); err != nil { |
| + t.Fatal(err) |
| + } |
| + } |
| + got := UnattributedEntries(dd, []string{host, replica}) |
| + want := []string{filepath.Join(dd, "blobs"), filepath.Join(dd, "cache.tmp")} |
| + if strings.Join(got, " ") != strings.Join(want, " ") { |
| + t.Fatalf("UnattributedEntries = %v, want %v", got, want) |
| + } |
| +} |
| + |
| +// Review (#256 B): ownership knows more than the rows. A host deleted or |
| +// moved away leaves its platform-named files behind, and when its name |
| +// extends this host's, the shorter host's scan must not sweep them in, |
| +// nor the app entries beside them that the files are evidence of. Each |
| +// platform shape, for a dotted and a dashed extension, with no row for |
| +// either: none is an extra, and each is reported with "no row". |
| +func TestScanExtrasLeavesAnotherHostsPlatformFilesAlone(t *testing.T) { |
| + dd, rd := t.TempDir(), t.TempDir() |
| + shapes := func(x string) []string { |
| + return []string{x + ".env", x + ".db", x + ".db-wal", x + ".db-shm", x + ".config.env", x + "-sidecar.env", ".activator-" + x + ".db-litestream", x + ".db.corrupt"} |
| + } |
| + var planted []string |
| + for _, x := range []string{host + ".au", host + "-staging"} { |
| + planted = append(planted, shapes(x)...) |
| + // The app entries of that host, which only its files evidence. |
| + planted = append(planted, x, x+"-blobs") |
| + } |
| + planted = append(planted, host, host+"-blobs") |
| + for _, name := range planted { |
| + if err := os.MkdirAll(filepath.Join(dd, name), 0o700); err != nil { |
| + t.Fatal(err) |
| + } |
| + } |
| + for _, mode := range []Mode{ModeNested, ModeFlat} { |
| + p, err := PlanFor(dd, rd, host, mode) |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + var logged []string |
| + p.Logf = func(f string, a ...any) { logged = append(logged, fmt.Sprintf(f, a...)) } |
| + if mode == ModeFlat { |
| + // The revert scans the instance directory; stage the same |
| + // names there to show the rule holds in that direction too. |
| + raw := rawInstanceDir(dd, host) |
| + for _, name := range planted { |
| + if err := os.MkdirAll(filepath.Join(raw, name), 0o700); err != nil { |
| + t.Fatal(err) |
| + } |
| + } |
| + os.RemoveAll(filepath.Join(dd, host)) |
| + os.RemoveAll(filepath.Join(dd, host+"-blobs")) |
| + } |
| + got, err := ScanExtras(p, []string{replica}) |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + if strings.Join(got, " ") != host+" "+host+"-blobs" { |
| + t.Errorf("%s: ScanExtras = %v, want only the host's own two entries", mode, got) |
| + } |
| + joined := strings.Join(logged, "\n") |
| + for _, x := range []string{host + ".au", host + "-staging"} { |
| + for _, name := range append(shapes(x), x, x+"-blobs") { |
| + if !strings.Contains(joined, name+" ("+x+"'s, which has no row on this box)") { |
| + t.Errorf("%s: %s not reported as %s's with no row:\n%s", mode, name, x, joined) |
| + } |
| + } |
| + } |
| + } |
| + // And a host that is a row: reported as its own, without the "no row". |
| + p, _ := PlanFor(dd, rd, host, ModeNested) |
| + var logged []string |
| + p.Logf = func(f string, a ...any) { logged = append(logged, fmt.Sprintf(f, a...)) } |
| + if _, err := ScanExtras(p, []string{host + ".au"}); err != nil { |
| + t.Fatal(err) |
| + } |
| + if j := strings.Join(logged, "\n"); !strings.Contains(j, host+".au.env ("+host+".au's)") || strings.Contains(j, host+".au.env ("+host+".au's, which") { |
| + t.Errorf("a host with a row is reported wrongly:\n%s", j) |
| + } |
| +} |
| + |
| +func TestFixedShapeHostReadsEveryPlatformShape(t *testing.T) { |
| + for name, want := range map[string]string{ |
| + "a.example.com.env": "a.example.com", |
| + "a.example.com.db": "a.example.com", |
| + "a.example.com.db-wal": "a.example.com", |
| + "a.example.com.db-shm": "a.example.com", |
| + "a.example.com.config.env": "a.example.com", |
| + "a.example.com-sidecar.env": "a.example.com", |
| + "a.example.com.db.corrupt": "a.example.com", |
| + ".activator-a.example.com.db-litestream": "a.example.com", |
| + "a.example.com": "", |
| + "a.example.com-blobs": "", |
| + "litestream.env": "", |
| + "registry.db": "", |
| + ".env": "", |
| + } { |
| + got, ok := fixedShapeHost(name) |
| + if got != want || ok != (want != "") { |
| + t.Errorf("fixedShapeHost(%q) = %q %v, want %q", name, got, ok, want) |
| + } |
| + } |
| +} |
diff --git a/internal/layout/journal.go b/internal/layout/journal.go| index 2068259e8..a8a66255f 100644 |
| --- a/internal/layout/journal.go |
| +++ b/internal/layout/journal.go |
| @@ -37,10 +37,25 @@ type Journal struct { |
| From Endpoint `json:"from"` |
| To Endpoint `json:"to"` |
| Artifacts []JournalArtifact `json:"artifacts"` |
| + // Extra is the app-made entries moved with the host (extras.go), by |
| + // base name. Absent in journals written before #256, which moved |
| + // none. Each is re-derived from the host and the name on recovery |
| + // and must pass extraNameOK, so the journal cannot direct a move |
| + // PlanFor and ScanExtras would not have chosen. |
| + Extra []JournalArtifact `json:"extra,omitempty"` |
| Phase Phase `json:"phase"` |
| WrittenAt string `json:"written_at"` |
| } |
| +// extraNames is the recorded extra entries' names, in journal order. |
| +func (j *Journal) extraNames() []string { |
| + var names []string |
| + for _, e := range j.Extra { |
| + names = append(names, e.Name) |
| + } |
| + return names |
| +} |
| + |
| // JournalArtifact records one artifact's presence at the time the journal |
| // was written. The path itself is not stored: it is re-derived by |
| // PlanFor from the host and mode on every read, so a journal can never be |
| @@ -329,6 +344,16 @@ func Validate(j *Journal, dataDir, socketDir string, row registry.Route) (atFrom |
| if !j.Artifacts[0].Present { |
| return false, fmt.Errorf("layout: %s: journal's db artifact is not marked present; not acting", j.Host) |
| } |
| + // The extra names pass the same rule a scan applies, and an extra is |
| + // never absent: the plan carries one because it was seen. |
| + if err := p.AddExtras(j.extraNames()); err != nil { |
| + return false, fmt.Errorf("%w; not acting", err) |
| + } |
| + for _, e := range j.Extra { |
| + if !e.Present { |
| + return false, fmt.Errorf("layout: %s: journal records extra %q absent; not acting", j.Host, e.Name) |
| + } |
| + } |
| switch (Endpoint{DB: row.DBPath, Socket: row.Socket}) { |
| case j.From: |
| return true, nil |
diff --git a/internal/layout/pass.go b/internal/layout/pass.go| index be711ffca..26a3870cd 100644 |
| --- a/internal/layout/pass.go |
| +++ b/internal/layout/pass.go |
| @@ -259,6 +259,7 @@ func (p *Pass) Converge(ctx context.Context, mode Mode) []Result { |
| return append(out, Result{Outcome: "refused", Reason: "list routes: " + err.Error()}) |
| } |
| sort.Slice(routes, func(i, j int) bool { return routes[i].Host < routes[j].Host }) |
| + migrated := false |
| for _, rt := range routes { |
| if !activator.ActivatorManaged(rt) || seen[rt.Host] { |
| continue |
| @@ -266,7 +267,18 @@ func (p *Pass) Converge(ctx context.Context, mode Mode) []Result { |
| if ctx.Err() != nil { |
| break |
| } |
| - out = append(out, p.convergeHost(ctx, rt, routes, mode)) |
| + r := p.convergeHost(ctx, rt, routes, mode) |
| + migrated = migrated || r.Outcome == "migrated" |
| + out = append(out, r) |
| + } |
| + if migrated { |
| + // What no host's move could take (extras.go): a flat app with cwd |
| + // / wrote relative paths straight into the data directory, and a |
| + // name no row owns cannot be given to one host. Said once per pass |
| + // that moved something, for the operator who has to place them. |
| + if left := UnattributedEntries(p.DataDir, hostsOf(routes)); len(left) > 0 { |
| + p.logf("layout: left in %s, owned by no host on this box: %s. a nested instance cannot see them; move what an app wrote into that app's instance directory by hand", p.DataDir, strings.Join(left, ", ")) |
| + } |
| } |
| return out |
| } |
| @@ -284,6 +296,17 @@ func (p *Pass) convergeHost(ctx context.Context, rt registry.Route, routes []reg |
| case activator.LayoutNested: |
| if mode == ModeNested { |
| p.forgetAwake(rt.Host) |
| + // A host moved before the pass took the app's own entries |
| + // with it (#256) has them still in the data directory, |
| + // where its namespace cannot see them. Nothing moves here: |
| + // the host is running, and what the app has minted since |
| + // may be the copy it wants. Said on every pass, so an |
| + // operator reading status or the log sees it while it lasts. |
| + if left := FlatLeftovers(p.DataDir, rt.Host, hostsOf(routes)); len(left) > 0 { |
| + text := fmt.Sprintf("still in %s from before its move, where the instance cannot see them: %s. move what the app needs into %s by hand", p.DataDir, strings.Join(left, ", "), rawInstanceDir(p.DataDir, rt.Host)) |
| + p.logf("layout: %s: %s", rt.Host, text) |
| + p.note(rt.Host, text) |
| + } |
| return Result{rt.Host, "unchanged", ""} |
| } |
| case activator.LayoutFlat: |
| @@ -318,6 +341,7 @@ func (p *Pass) migrateHost(ctx context.Context, host string, routes []registry.R |
| if err != nil { |
| return Result{host, "refused", err.Error()} |
| } |
| + plan.Logf = p.logf |
| if err := Preflight(plan, routes); err != nil { |
| return Result{host, "refused", err.Error()} |
| } |
| @@ -537,18 +561,30 @@ func (p *Pass) recheck(host string, routes []registry.Route) (reason string, pla |
| if err != nil { |
| return err.Error(), false |
| } |
| + plan.Logf = p.logf |
| + |
| if err := Preflight(plan, routes); err != nil { |
| return err.Error(), false |
| } |
| if err := EnsureDirs(p.DataDir, p.socketDir(), host, p.SocketsGID); err != nil { |
| return err.Error(), true |
| } |
| - if _, err := presence(plan); err != nil { |
| + if _, err := presence(plan, hostsOf(routes)); err != nil { |
| return err.Error(), true |
| } |
| return "", false |
| } |
| +// hostsOf is every host with a row on the box, whatever its kind: the set |
| +// an entry's owner is chosen from (extras.go). |
| +func hostsOf(routes []registry.Route) []string { |
| + hosts := make([]string, 0, len(routes)) |
| + for _, rt := range routes { |
| + hosts = append(hosts, rt.Host) |
| + } |
| + return hosts |
| +} |
| + |
| // Held reports whether the pass keeps host fenced and down after a |
| // refused nested move, and why. The fence must still be the pass's own: |
| // a record the activator no longer matches is dropped. |
| @@ -734,6 +770,8 @@ func (p *Pass) shift(host string, mode Mode, prior *Journal) (res Result, h hand |
| if err != nil { |
| return Result{host, "refused", err.Error()}, handoverNone, false |
| } |
| + plan.Logf = p.logf |
| + |
| if err := Preflight(plan, routes); err != nil { |
| return Result{host, "refused", err.Error()}, handoverNone, false |
| } |
| @@ -751,7 +789,7 @@ func (p *Pass) shift(host string, mode Mode, prior *Journal) (res Result, h hand |
| // of, with the journal write: a bundle rendered between a scan outside |
| // it and the journal would be recorded absent and left at the old path. |
| p.serialize(func() { |
| - present, err := presence(plan) |
| + present, err := presence(plan, hostsOf(routes)) |
| if err != nil { |
| res, h, hold = Result{host, "refused", err.Error()}, handoverNone, true |
| return |
| @@ -765,8 +803,11 @@ func (p *Pass) shift(host string, mode Mode, prior *Journal) (res Result, h hand |
| } |
| // presence records which artifacts are at their sources, in artifactNames |
| -// order, and refuses a plan whose database is not. |
| -func presence(plan *Plan) ([]bool, error) { |
| +// order, and refuses a plan whose database is not. It then adds to the |
| +// plan every entry the app made for the host (ScanExtras; hosts is the |
| +// box's rows, which decide ownership), so the journal written from the |
| +// plan names them and the move carries them. |
| +func presence(plan *Plan, hosts []string) ([]bool, error) { |
| present := make([]bool, len(plan.Artifacts)) |
| for i, a := range plan.Artifacts { |
| // Nothing is ever renamed for the socket: the listener is gone |
| @@ -784,6 +825,13 @@ func presence(plan *Plan) ([]bool, error) { |
| present[i] = ok |
| plan.Artifacts[i].Present = ok |
| } |
| + extras, err := ScanExtras(plan, hosts) |
| + if err != nil { |
| + return nil, err |
| + } |
| + if err := plan.AddExtras(extras); err != nil { |
| + return nil, err |
| + } |
| // With presence known, every file about to move must have one name. |
| // This runs under the same lock as the journal write, so nothing the |
| // agent itself does can add a name between the check and the journal. |
| @@ -793,6 +841,28 @@ func presence(plan *Plan) ([]bool, error) { |
| return present, nil |
| } |
| +// applyJournal gives a plan rebuilt from the host name what the journal |
| +// recorded about the move: which fixed artifacts were present, and the |
| +// extra entries by name. Validate has already checked the names' shape; |
| +// this checks their ownership against the box's rows as they stand now, |
| +// because a journal is not an instruction (design §3.3): a forged journal |
| +// for H naming H-x, where H-x is a host on the box, would otherwise move |
| +// that host's own directory into H's tree on recovery. |
| +func applyJournal(plan *Plan, j *Journal, hosts []string) error { |
| + for i := range plan.Artifacts { |
| + if i < len(j.Artifacts) { |
| + plan.Artifacts[i].Present = j.Artifacts[i].Present |
| + } |
| + } |
| + all := append([]string{plan.Host}, hosts...) |
| + for _, name := range j.extraNames() { |
| + if owner := ownerOf(name, all); owner != plan.Host { |
| + return fmt.Errorf("layout: %s: journal names extra %q, which belongs to %s", plan.Host, name, owner) |
| + } |
| + } |
| + return plan.AddExtras(j.extraNames()) |
| +} |
| + |
| // commit is steps 6 to 9 up to the journal's removal. The caller reloads |
| // routing afterwards, outside Serialize's lock and still under the fence. |
| // Every return after the journal is written leaves it for Recover, and the |
| @@ -835,7 +905,7 @@ func (p *Pass) commitWritten(plan *Plan, j *Journal, aside []string) Result { |
| if p.abortAt == "after-journal" { |
| return Result{host, "refused", "aborted"} |
| } |
| - if err := p.moveFiles(plan, j); err != nil { |
| + if err := p.moveFiles(plan); err != nil { |
| return Result{host, "refused", err.Error()} |
| } |
| moved, err := VerifyMoved(plan) |
| @@ -868,9 +938,16 @@ func (p *Pass) commitWritten(plan *Plan, j *Journal, aside []string) Result { |
| if plan.Mode == ModeFlat { |
| outcome = "reverted" |
| } |
| + var events []string |
| + if extras := plan.ExtraNames(); len(extras) > 0 { |
| + events = append(events, "moved with it: "+strings.Join(extras, ", ")) |
| + } |
| if len(aside) > 0 { |
| - p.logf("layout: %s: %s -> %s (kept aside: %s)", host, plan.From.DB, plan.To.DB, strings.Join(aside, ", ")) |
| - p.note(host, fmt.Sprintf("%s at %s: kept aside %s", outcome, p.now().UTC().Format(time.RFC3339), strings.Join(aside, ", "))) |
| + events = append(events, "kept aside: "+strings.Join(aside, ", ")) |
| + } |
| + if len(events) > 0 { |
| + p.logf("layout: %s: %s -> %s (%s)", host, plan.From.DB, plan.To.DB, strings.Join(events, "; ")) |
| + p.note(host, fmt.Sprintf("%s at %s: %s", outcome, p.now().UTC().Format(time.RFC3339), strings.Join(events, "; "))) |
| } else { |
| p.logf("layout: %s: %s -> %s", host, plan.From.DB, plan.To.DB) |
| } |
| @@ -879,16 +956,17 @@ func (p *Pass) commitWritten(plan *Plan, j *Journal, aside []string) Result { |
| var errAborted = errors.New("layout: aborted by test seam") |
| -// moveFiles renames every artifact the journal records present. One |
| -// already at its destination is done; one still at its source is renamed; |
| -// one at neither is lost, and one at both was not left that way by a |
| -// crash (the destination check cleared every destination before the |
| -// journal was written, and a rename leaves a file in exactly one place). |
| -// Either of the last two is an error that keeps the journal and so the |
| -// fence. |
| -func (p *Pass) moveFiles(plan *Plan, j *Journal) error { |
| - for i, a := range plan.Artifacts { |
| - if a.Name == "socket" || !j.Artifacts[i].Present { |
| +// moveFiles renames every artifact the plan records present (from the |
| +// scan on a first attempt, from the journal on recovery), the app's own |
| +// entries included. One already at its destination is done; one still at |
| +// its source is renamed; one at neither is lost, and one at both was not |
| +// left that way by a crash (the destination check cleared every |
| +// destination before the journal was written, and a rename leaves a file |
| +// in exactly one place). Either of the last two is an error that keeps |
| +// the journal and so the fence. |
| +func (p *Pass) moveFiles(plan *Plan) error { |
| + for _, a := range plan.Artifacts { |
| + if a.Name == "socket" || !a.Present { |
| continue |
| } |
| atDst, err := lstatPresent(a.Dst) |
| @@ -1032,6 +1110,15 @@ func (p *Pass) recoverHost(ctx context.Context, host string, byHost map[string]r |
| if err != nil { |
| return Result{host, "refused", notActing(err)} |
| } |
| + plan.Logf = p.logf |
| + |
| + hosts := make([]string, 0, len(byHost)) |
| + for h := range byHost { |
| + hosts = append(hosts, h) |
| + } |
| + if err := applyJournal(plan, j, hosts); err != nil { |
| + return Result{host, "refused", notActing(err)} |
| + } |
| keepFence := false |
| r, fence, gen := p.quiesce(ctx, host, p.generation()) |
| switch fence { |
| @@ -1141,8 +1228,8 @@ func (p *Pass) recoverHost(ctx context.Context, host string, byHost map[string]r |
| // it records present being at its destination before the row is swapped. |
| func (p *Pass) complete(plan *Plan, j *Journal, atFrom, remove bool) Result { |
| host, phase := plan.Host, j.Phase |
| - for i, a := range plan.Artifacts { |
| - if a.Name == "socket" || j.Artifacts[i].Present { |
| + for _, a := range plan.Artifacts { |
| + if a.Name == "socket" || a.Present { |
| continue |
| } |
| for _, path := range []string{a.Src, a.Dst} { |
| @@ -1157,17 +1244,12 @@ func (p *Pass) complete(plan *Plan, j *Journal, atFrom, remove bool) Result { |
| } |
| switch j.Phase { |
| case PhaseFiles: |
| - if err := p.moveFiles(plan, j); err != nil { |
| + if err := p.moveFiles(plan); err != nil { |
| return Result{host, "refused", notActing(err)} |
| } |
| - // The same check a first attempt makes before its row flips: the |
| - // plan was rebuilt from the host name, so it carries the journal's |
| - // presence for the check to know which files it moved. |
| - for i := range plan.Artifacts { |
| - if i < len(j.Artifacts) { |
| - plan.Artifacts[i].Present = j.Artifacts[i].Present |
| - } |
| - } |
| + // The same check a first attempt makes before its row flips; the |
| + // plan carries the journal's presence (applyJournal) so the check |
| + // knows which files it moved. |
| moved, err := VerifyMoved(plan) |
| if len(moved) > 0 { |
| p.logf("layout: %s: quarantined on recovery -> %s", host, strings.Join(moved, ", ")) |
| @@ -1184,8 +1266,8 @@ func (p *Pass) complete(plan *Plan, j *Journal, atFrom, remove bool) Result { |
| // The row is about to name the destinations, so every file the |
| // journal records present must be there: a row pointed at a |
| // missing database has the next wake restore a second one. |
| - for i, a := range plan.Artifacts { |
| - if a.Name == "socket" || !j.Artifacts[i].Present { |
| + for _, a := range plan.Artifacts { |
| + if a.Name == "socket" || !a.Present { |
| continue |
| } |
| present, err := lstatPresent(a.Dst) |
diff --git a/internal/layout/plan.go b/internal/layout/plan.go| index a396b6ce8..ed1b17194 100644 |
| --- a/internal/layout/plan.go |
| +++ b/internal/layout/plan.go |
| @@ -54,6 +54,11 @@ type Artifact struct { |
| // name alone, §3.3), so Present is always false here; the pass fills |
| // it in during preflight. |
| Present bool |
| + // Extra marks an entry the app made for itself under the host's name |
| + // (extras.go), found by ScanExtras or recorded in the journal rather |
| + // than derived by PlanFor. Always Present: it is in the plan because |
| + // it was seen. Name is the entry's own base name. |
| + Extra bool |
| } |
| // Plan is everything the pass will do for one host, computed before it |
| @@ -68,6 +73,16 @@ type Plan struct { |
| // the plan can find the pass's own directory under it (the aside |
| // directory, outside any instance tree) without being handed it again. |
| DataDir string |
| + // Logf is the pass's logger, set by the pass after PlanFor so the |
| + // checks that run on the plan (the extras scan, the link walk) can |
| + // say what they are doing and what they left. Nil logs nothing. |
| + Logf func(string, ...any) |
| +} |
| + |
| +func (p *Plan) logf(format string, a ...any) { |
| + if p.Logf != nil { |
| + p.Logf(format, a...) |
| + } |
| } |
| // artifactNames is the fixed order of the nine artifacts the pass tracks. |
| @@ -184,10 +199,15 @@ func PlanFor(dataDir, socketDir, host string, mode Mode) (*Plan, error) { |
| } |
| // Journal builds the journal for this plan with the given presence |
| -// vector (one bool per artifact, artifactNames order). |
| +// vector (one bool per fixed artifact, artifactNames order), plus the |
| +// plan's extra entries by name. |
| func (p *Plan) Journal(present []bool) *Journal { |
| j := &Journal{Host: p.Host, Mode: p.Mode, From: p.From, To: p.To, Phase: PhaseFiles} |
| for i, a := range p.Artifacts { |
| + if a.Extra { |
| + j.Extra = append(j.Extra, JournalArtifact{Name: a.Name, Present: true}) |
| + continue |
| + } |
| j.Artifacts = append(j.Artifacts, JournalArtifact{Name: a.Name, Present: i < len(present) && present[i]}) |
| } |
| return j |
diff --git a/internal/layout/preflight.go b/internal/layout/preflight.go| index b29513bdf..da96d6d25 100644 |
| --- a/internal/layout/preflight.go |
| +++ b/internal/layout/preflight.go |
| @@ -290,6 +290,16 @@ func LinkCheck(p *Plan, at func(Artifact) string) error { |
| } |
| continue |
| } |
| + if a.Extra && fi.IsDir() { |
| + // Said before the walk, which stats every file: a blob store |
| + // can be large, the host is stopped meanwhile, and the walk |
| + // runs twice (sources, then destinations). |
| + p.logf("layout: %s: checking every file under %s for hard links and symlinks before the move goes on", p.Host, path) |
| + if err := walkExtraLinks(p.Host, p.DataDir, path); err != nil { |
| + return err |
| + } |
| + continue |
| + } |
| if err := oneName(p.Host, p.DataDir, path, fi); err != nil { |
| return err |
| } |
| @@ -297,6 +307,58 @@ func LinkCheck(p *Plan, at func(Artifact) string) error { |
| return nil |
| } |
| +// walkExtraLinks is walkLinks for a directory an app made for itself |
| +// (extras.go), with one allowance the shadow directory does not need: a |
| +// file with several names may be moved when EVERY one of those names is |
| +// inside the directory, since the move carries them all and nothing |
| +// outside can reach the file afterwards. Apps deduplicate content by |
| +// hard-linking inside their own blob store, and refusing that would hold |
| +// a host down for its own ordinary files. A name outside the directory |
| +// is still a second reader and writer the move does not know about, and |
| +// is refused with the inode and the find that lists it. Symlinks and |
| +// special files are refused as in walkLinks. |
| +func walkExtraLinks(host, dataDir, root string) error { |
| + type linked struct { |
| + path string |
| + nlink uint64 |
| + seen uint64 |
| + } |
| + byInode := map[uint64]*linked{} |
| + err := filepath.WalkDir(root, func(path string, d os.DirEntry, err error) error { |
| + if err != nil { |
| + return fmt.Errorf("layout: %s: %w", host, err) |
| + } |
| + if d.IsDir() { |
| + return nil |
| + } |
| + fi, err := d.Info() |
| + if err != nil { |
| + return fmt.Errorf("layout: %s: %w", host, err) |
| + } |
| + if !fi.Mode().IsRegular() { |
| + return fmt.Errorf("layout: %s: %s is not a regular file (%v); refusing to move it", host, path, fi.Mode().Type()) |
| + } |
| + if n, ino := linkCount(fi); n > 1 { |
| + l := byInode[ino] |
| + if l == nil { |
| + l = &linked{path: path, nlink: n} |
| + byInode[ino] = l |
| + } |
| + l.seen++ |
| + } |
| + return nil |
| + }) |
| + if err != nil { |
| + return err |
| + } |
| + for ino, l := range byInode { |
| + if l.seen < l.nlink { |
| + return fmt.Errorf("layout: %s: %s has %d names (hard links, inode %d) and only %d of them are inside %s; refusing to move a file something else can still reach. find the others with: find %s -xdev -samefile %s", host, l.path, l.nlink, ino, l.seen, root, dataDir, l.path) |
| + } |
| + } |
| + return nil |
| +} |
| + |
| func walkLinks(host, dataDir, root string) error { |
| return filepath.WalkDir(root, func(path string, d os.DirEntry, err error) error { |
| if err != nil { |