amadan
layout: move the app's own entries with the host, or refuse (#256 B)
The layout pass moved its nine platform-named files and left behind what the app had made for itself under the host's name: on the flagship, /data/<host>/instance.key and report.token and /data/<host>-blobs for three instances. Nested, filepath.Dir(dbPath) pointed at an empty instance directory, so the apps minted fresh keys. A move now scans the data directory, under the fence and inside the serialized section with the journal write, for every entry the flat layout's naming rules make the host's (<host>, <host>.*, <host>-*, and the activator's .activator-<host>.* form), and moves each into the instance directory at the same name; a revert brings them back. A name two hosts on the box both match (`a.b.oncarlos.com-x` is also the host `a.b.oncarlos.com-x`) belongs to the longest, so no pass moves another host's entry. The entries are recorded in the journal by name (an additive field, absent in earlier journals) and re-derived from host and name on recovery; a recorded name must pass the same shape rule, and its ownership is judged again against the box's rows, so a forged journal cannot move a neighbour's directory into a host's tree. The same protections apply: a symlink, a special file, or a file whose other hard links lie outside the entry refuse the move with the inode and the find that lists them, and the host is held as for any planted refusal. Hard links whose every name is inside the entry are the app's own deduplication and move together. A taken destination is a refusal naming both paths, never a choice. What no host's rule names (a shared /data/blobs written by every flat app whose cwd was /) is not moved, since it cannot be attributed; the pass logs those once per pass that moved something, for an operator to place by hand, and docs/site/cli.md says so.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
8 files changed,
+805
−32
docs/site/cli.md+2 −0internal/layout/extras.go+231 −0internal/layout/extras_pass_linux_test.go+283 −0internal/layout/extras_test.go+100 −0internal/layout/journal.go+25 −0internal/layout/pass.go+95 −31internal/layout/plan.go+11 −1internal/layout/preflight.go+58 −0
diff --git a/docs/site/cli.md b/docs/site/cli.md| index b849f63c0..ca0d32ced 100644 |
| --- a/docs/site/cli.md |
| +++ b/docs/site/cli.md |
| @@ -1872,6 +1872,8 @@ Every instance the activator runs used to live in one shared directory as one us |
| `layout migrate` is refused while any box in scope runs a platform release from before the layout, so a fleet is rolled forward first and migrated second. |
| +What moves with an instance is everything in the box's data directory that carries its hostname: its database and the platform's files beside it, and anything the app made for itself under that name, such as a `<host>/` directory holding its keys or a `<host>-blobs` store. They land in the instance's directory under the same names, which is where an app that derives paths from its database path now looks. An entry that cannot be moved safely, such as a file hard-linked from outside the instance's own files, or a symlink, stops the move with its name in the reason, and the instance is held until it is cleared. A directory that carries no hostname, such as a `blobs` directory that several apps wrote into with the working directory `/`, is not moved: the box cannot tell whose it is. The agent log names those once a move has happened; an operator moves what an app owns into that app's instance directory by hand, and it comes along on any later revert or migration. `revert` brings the same names back. |
| + |
| You rarely need `migrate` at all. With no intent recorded, a box uses the nested layout by default as soon as its isolation is ready (the probe passes), and moves its instances on its own, with the same hour of grace for an awake one. `carlos status --layout` marks such a box `(nested by default)`, and says why a box the default has not reached is still flat. To keep box-by-box control, record `migrate --box` or `revert` first; a recorded intent always wins. A deployment that is not ready for this at all puts `CARLOS_LAYOUT_DEFAULT=flat` in each box's host.env. |
| `layout revert` moves them back. The order for rolling the platform back to a release from before the layout is: `layout revert`, wait until `carlos status --layout` shows every instance `flat`, then `system rollback`. The console will not move a channel pointer onto such a release while any box reports otherwise, and a revert is refused while a maintenance worker is attached to a box in scope. `layout status` prints the recorded intent; what each box has done is `carlos status --layout`. |
diff --git a/internal/layout/extras.go b/internal/layout/extras.go| new file mode 100644 |
| index 000000000..38f924488 |
| --- /dev/null |
| +++ b/internal/layout/extras.go |
| @@ -0,0 +1,231 @@ |
| +package layout |
| + |
| +import ( |
| + "errors" |
| + "fmt" |
| + "os" |
| + "path/filepath" |
| + "sort" |
| + "strings" |
| +) |
| + |
| +// The entries an app makes for itself (discussion #256). |
| +// |
| +// The plan's nine artifacts are the files the PLATFORM names for a host. |
| +// A flat instance was started with cwd / and `--db /data/<host>.db`, and |
| +// an app that derives a directory from its database path makes more: |
| +// bdn/ips kept /data/<host>/instance.key and /data/<host>/report.token |
| +// and a /data/<host>-blobs directory beside its database. The first |
| +// migration moved the nine names and left those behind, the app started |
| +// nested, looked for them under its new database's directory, found |
| +// nothing, and minted a fresh key. Three apps did, on the flagship, on |
| +// the first day. |
| +// |
| +// So a move to nested takes every entry in the data directory that the |
| +// flat layout's naming rules make the host's, into the instance |
| +// directory at the same relative name, where `filepath.Dir(dbPath)` now |
| +// points (nested the database is <instance dir>/<host>.db, and the |
| +// instance sees that directory as /data/instances/<iid>); a revert brings |
| +// the same names back. Nothing the host could own is left behind |
| +// silently: an entry that cannot be moved safely refuses the move with |
| +// its name in the reason. These entries are scanned, not derived, so they |
| +// are recorded in the journal by name and re-derived from the host and |
| +// the name on recovery; a name a journal records must still pass the |
| +// same rules (extraNameOK), so a journal cannot name a move the pass |
| +// would not have chosen itself. |
| +// |
| +// The naming rules are the flat layout's own. A file is <host>'s when it |
| +// is <host> itself, <host> plus a dot or a dash and more (<host>.db, |
| +// <host>-sidecar.env, <host>-blobs), or the activator's own prefix form |
| +// of those (.activator-<host>.db-litestream). A name can match more than |
| +// one host on the box (`a.b.oncarlos.com-x` is `a.b.oncarlos.com` plus |
| +// `-x`, and also the host `a.b.oncarlos.com-x` itself), so an entry |
| +// belongs to the LONGEST host on the box whose rule it matches, and a |
| +// pass moves only what belongs to its own host. That is exact for every |
| +// host the box has a row for. A name that matches no row is not moved by |
| +// anyone and is reported (UnattributedEntries), because a shared |
| +// directory such as /data/blobs, written by every flat app whose cwd was |
| +// /, cannot be given to one host. |
| + |
| +// belongsTo reports whether name is one the flat layout's rules make |
| +// host's: the host itself, or the host with a platform or app suffix. |
| +func belongsTo(name, host string) bool { |
| + if name == host { |
| + return true |
| + } |
| + for _, prefix := range []string{host + ".", host + "-", ".activator-" + host + ".", ".activator-" + host + "-"} { |
| + if strings.HasPrefix(name, prefix) { |
| + return true |
| + } |
| + } |
| + return false |
| +} |
| + |
| +// ownerOf is the longest host among hosts whose rules name matches, or |
| +// "" when none does. |
| +func ownerOf(name string, hosts []string) string { |
| + owner := "" |
| + for _, h := range hosts { |
| + if len(h) > len(owner) && belongsTo(name, h) { |
| + owner = h |
| + } |
| + } |
| + return owner |
| +} |
| + |
| +// fixedNames is the base names of the plan's platform-named artifacts |
| +// (the same on both sides of a move; the socket is never a file the pass |
| +// moves and has a different name nested, so it is left out). |
| +func fixedNames(p *Plan) map[string]bool { |
| + names := map[string]bool{} |
| + for _, a := range p.Artifacts { |
| + if a.Extra || a.Name == "socket" { |
| + continue |
| + } |
| + names[filepath.Base(a.Src)] = true |
| + names[filepath.Base(a.Dst)] = true |
| + } |
| + return names |
| +} |
| + |
| +// extraNameOK is the rule a recorded or scanned extra name must pass: one |
| +// path element, not one of the plan's own names, and host's by belongsTo. |
| +// It is what keeps a journal from being an instruction (design §3.3): a |
| +// name that fails it is refused however it got into the journal. |
| +func extraNameOK(p *Plan, name string) error { |
| + if name == "" || name == "." || name == ".." || strings.ContainsAny(name, "/\x00") { |
| + return fmt.Errorf("layout: %s: %q is not a file name", p.Host, name) |
| + } |
| + if fixedNames(p)[name] { |
| + return fmt.Errorf("layout: %s: %q is one of the move's own names, not an extra", p.Host, name) |
| + } |
| + if !belongsTo(name, p.Host) { |
| + return fmt.Errorf("layout: %s: %q is not named for the host", p.Host, name) |
| + } |
| + return nil |
| +} |
| + |
| +// extraDirs is where a host's extra entries are before and after this |
| +// plan's move: the data directory and the raw instance directory, in the |
| +// plan's direction. |
| +func (p *Plan) extraDirs() (src, dst string) { |
| + flat, nested := p.DataDir, rawInstanceDir(p.DataDir, p.Host) |
| + if p.Mode == ModeFlat { |
| + return nested, flat |
| + } |
| + return flat, nested |
| +} |
| + |
| +// AddExtras appends the named entries to the plan as extra artifacts, |
| +// present, at their source and destination for the plan's direction. |
| +// Each name must pass extraNameOK, and no name twice. |
| +func (p *Plan) AddExtras(names []string) error { |
| + src, dst := p.extraDirs() |
| + seen := map[string]bool{} |
| + for _, a := range p.Artifacts { |
| + if a.Extra { |
| + seen[a.Name] = true |
| + } |
| + } |
| + for _, name := range names { |
| + if err := extraNameOK(p, name); err != nil { |
| + return err |
| + } |
| + if seen[name] { |
| + return fmt.Errorf("layout: %s: extra %q listed twice", p.Host, name) |
| + } |
| + seen[name] = true |
| + p.Artifacts = append(p.Artifacts, Artifact{ |
| + Name: name, Src: filepath.Join(src, name), Dst: filepath.Join(dst, name), |
| + Extra: true, Present: true, |
| + }) |
| + } |
| + return nil |
| +} |
| + |
| +// ExtraNames is the names of the plan's extra artifacts, in plan order. |
| +func (p *Plan) ExtraNames() []string { |
| + var names []string |
| + for _, a := range p.Artifacts { |
| + if a.Extra { |
| + names = append(names, a.Name) |
| + } |
| + } |
| + return names |
| +} |
| + |
| +// ScanExtras lists the entries in the plan's source directory that belong |
| +// to the host and that the plan does not already name, sorted. hosts is |
| +// every host with a row on the box; an entry a longer host's rule also |
| +// matches is that host's and is left alone. An entry whose destination is |
| +// already taken is a refusal naming both, never a choice: on a migration |
| +// the destination check has just cleared the instance directory of |
| +// everything the plan did not name, so a taken destination was planted |
| +// since (a hold, like any planted destination); on a revert it is a flat |
| +// file the pass will not overwrite. |
| +// |
| +// The scan runs under the fence, with the host stopped, so the listing is |
| +// the whole of what the app made; and inside the serialized section with |
| +// the journal write, so what it records is what the journal records. |
| +func ScanExtras(p *Plan, hosts []string) ([]string, error) { |
| + src, dst := p.extraDirs() |
| + entries, err := os.ReadDir(src) |
| + if errors.Is(err, os.ErrNotExist) { |
| + return nil, nil |
| + } |
| + if err != nil { |
| + return nil, fmt.Errorf("layout: %s: list %s: %w", p.Host, src, err) |
| + } |
| + fixed := fixedNames(p) |
| + all := append([]string{p.Host}, hosts...) |
| + var names []string |
| + for _, e := range entries { |
| + name := e.Name() |
| + if fixed[name] || !belongsTo(name, p.Host) || ownerOf(name, all) != p.Host { |
| + continue |
| + } |
| + taken, err := lstatPresent(filepath.Join(dst, name)) |
| + if err != nil { |
| + return nil, fmt.Errorf("layout: %s: %w", p.Host, err) |
| + } |
| + if taken { |
| + return nil, fmt.Errorf("layout: %s: %s exists at both %s and %s; refusing to choose", p.Host, name, filepath.Join(src, name), filepath.Join(dst, name)) |
| + } |
| + names = append(names, name) |
| + } |
| + sort.Strings(names) |
| + return names, nil |
| +} |
| + |
| +// platformNames are the entries the platform itself keeps directly under |
| +// the data directory. UnattributedEntries leaves them out of its report. |
| +var platformNames = map[string]bool{ |
| + "registry.db": true, "registry.db-wal": true, "registry.db-shm": true, |
| + "instances": true, ".instances": true, "layout": true, "system": true, |
| + "moves": true, ".moves": true, ".hostexec": true, "litestream.env": true, |
| + "lost+found": true, |
| +} |
| + |
| +// UnattributedEntries lists the entries directly under dataDir that no |
| +// host on the box owns and that are not the platform's own: what a flat |
| +// app wrote relative to its working directory, which was /, or under a |
| +// name of its own choosing. The pass cannot move these, because it |
| +// cannot tell whose they are (a /data/blobs that several apps wrote into |
| +// is the live example), so it reports them for an operator to place by |
| +// hand. Sorted; nil when there are none. |
| +func UnattributedEntries(dataDir string, hosts []string) []string { |
| + entries, err := os.ReadDir(dataDir) |
| + if err != nil { |
| + return nil |
| + } |
| + var out []string |
| + for _, e := range entries { |
| + name := e.Name() |
| + if platformNames[name] || ownerOf(name, hosts) != "" { |
| + continue |
| + } |
| + out = append(out, filepath.Join(dataDir, name)) |
| + } |
| + sort.Strings(out) |
| + return out |
| +} |
diff --git a/internal/layout/extras_pass_linux_test.go b/internal/layout/extras_pass_linux_test.go| new file mode 100644 |
| index 000000000..efe3a80b3 |
| --- /dev/null |
| +++ b/internal/layout/extras_pass_linux_test.go |
| @@ -0,0 +1,283 @@ |
| +//go:build linux |
| + |
| +package layout |
| + |
| +import ( |
| + "context" |
| + "errors" |
| + "fmt" |
| + "os" |
| + "path/filepath" |
| + "strings" |
| + "testing" |
| + |
| + "github.com/carlosframework/platform/internal/registry" |
| +) |
| + |
| +// seedAppEntries writes what an app makes for itself beside its flat |
| +// database (discussion #256): a directory named for the host holding its |
| +// key, a `-blobs` directory, and a dot-suffixed file. |
| +func seedAppEntries(t *testing.T, dd, h string) { |
| + t.Helper() |
| + for path, body := range map[string]string{ |
| + filepath.Join(dd, h, "instance.key"): "key " + h, |
| + filepath.Join(dd, h, "report.token"): "token " + h, |
| + filepath.Join(dd, h+"-blobs", "ab", "cd"): "blob " + h, |
| + filepath.Join(dd, h+".cache"): "cache " + h, |
| + } { |
| + if err := os.MkdirAll(filepath.Dir(path), 0o700); err != nil { |
| + t.Fatal(err) |
| + } |
| + if err := os.WriteFile(path, []byte(body), 0o600); err != nil { |
| + t.Fatal(err) |
| + } |
| + } |
| +} |
| + |
| +// Discussion #256: the first migration moved the nine platform names and |
| +// left behind /data/<host>/instance.key and /data/<host>-blobs; the app |
| +// started nested without them and minted a new key. Every entry the flat |
| +// naming rules make the host's now moves into the instance directory at |
| +// the same relative name, and comes back on revert. An entry whose name a |
| +// longer host owns (`<host>-x` is the host `<host>-x` itself) stays, and so |
| +// does a shared directory no host owns, which is reported. |
| +func TestConvergeMovesAppCreatedEntriesWithTheHost(t *testing.T) { |
| + p, f := box(t) |
| + var logged []string |
| + p.Logf = func(format string, a ...any) { logged = append(logged, fmt.Sprintf(format, a...)) } |
| + seedAppEntries(t, p.DataDir, host) |
| + // A route whose name is host plus "-x": an externally supervised one |
| + // (not activator-managed), so its own entries must be left alone. |
| + decoy := host + "-x" |
| + if err := p.Registry.Upsert(registry.Route{Host: decoy, Kind: "instance", Socket: filepath.Join(p.SocketDir, decoy+".sock")}); err != nil { |
| + t.Fatal(err) |
| + } |
| + seedAppEntries(t, p.DataDir, decoy) |
| + // What a flat app with cwd / wrote as ./blobs. |
| + if err := os.MkdirAll(filepath.Join(p.DataDir, "blobs", "aa"), 0o700); err != nil { |
| + t.Fatal(err) |
| + } |
| + |
| + res := byHost(t, p.Converge(context.Background(), ModeNested)) |
| + if res[host].Outcome != "migrated" { |
| + t.Fatalf("%s: %+v", host, res[host]) |
| + } |
| + raw := rawInstanceDir(p.DataDir, host) |
| + for rel, want := range map[string]string{ |
| + filepath.Join(host, "instance.key"): "key " + host, |
| + filepath.Join(host, "report.token"): "token " + host, |
| + filepath.Join(host+"-blobs", "ab", "cd"): "blob " + host, |
| + host + ".cache": "cache " + host, |
| + } { |
| + if b, err := os.ReadFile(filepath.Join(raw, rel)); err != nil || string(b) != want { |
| + t.Errorf("%s not in the instance directory: %q %v", rel, b, err) |
| + } |
| + if _, err := os.Lstat(filepath.Join(p.DataDir, rel)); !errors.Is(err, os.ErrNotExist) { |
| + t.Errorf("%s left behind in the data directory: %v", rel, err) |
| + } |
| + } |
| + for _, rel := range []string{filepath.Join(decoy, "instance.key"), filepath.Join(decoy+"-blobs", "ab", "cd"), decoy + ".cache", filepath.Join("blobs", "aa")} { |
| + if _, err := os.Lstat(filepath.Join(p.DataDir, rel)); err != nil { |
| + t.Errorf("%s, not this host's, was moved: %v", rel, err) |
| + } |
| + if _, err := os.Lstat(filepath.Join(raw, rel)); !errors.Is(err, os.ErrNotExist) { |
| + t.Errorf("%s, not this host's, is in the instance directory: %v", rel, err) |
| + } |
| + } |
| + if JournalPresent(p.DataDir, host) || f.fences[host] != 0 { |
| + t.Fatal("journal or fence left after the move") |
| + } |
| + if n := p.Note(host); !strings.Contains(n, "moved with it: "+host+", "+host+"-blobs, "+host+".cache") { |
| + t.Errorf("note = %q", n) |
| + } |
| + // The aside directory holds nothing: these were moved, not quarantined. |
| + if q, err := os.ReadDir(asideDir(p.DataDir, host)); err == nil && len(q) > 0 { |
| + t.Errorf("quarantined instead of moved: %v", q) |
| + } |
| + joined := strings.Join(logged, "\n") |
| + if !strings.Contains(joined, "owned by no host on this box") || !strings.Contains(joined, filepath.Join(p.DataDir, "blobs")) { |
| + t.Errorf("the shared directory was not reported:\n%s", joined) |
| + } |
| + if strings.Contains(joined, filepath.Join(p.DataDir, decoy)) { |
| + t.Errorf("the decoy host's own directory was reported as unowned:\n%s", joined) |
| + } |
| + |
| + // Revert brings the same names back, and only those. |
| + res = byHost(t, p.Converge(context.Background(), ModeFlat)) |
| + if res[host].Outcome != "reverted" { |
| + t.Fatalf("revert %s: %+v", host, res[host]) |
| + } |
| + assertFlatUntouched(t, p, host) |
| + for rel, want := range map[string]string{ |
| + filepath.Join(host, "instance.key"): "key " + host, |
| + filepath.Join(host+"-blobs", "ab", "cd"): "blob " + host, |
| + host + ".cache": "cache " + host, |
| + } { |
| + if b, err := os.ReadFile(filepath.Join(p.DataDir, rel)); err != nil || string(b) != want { |
| + t.Errorf("%s not back in the data directory: %q %v", rel, b, err) |
| + } |
| + } |
| + if entries, _ := os.ReadDir(raw); len(entries) != 0 { |
| + t.Errorf("instance directory not emptied by the revert: %v", entries) |
| + } |
| +} |
| + |
| +// The same protections the platform names get, on the way in: a file |
| +// inside an app's directory that has another name OUTSIDE it would be |
| +// bound read-write into the namespace while something else still reaches |
| +// it, so the move is refused with the inode and the find, and the host is |
| +// held (a tenant can plant exactly this). A hard link whose every name is |
| +// inside the directory is the app's own deduplication and moves. |
| +func TestConvergeRefusesAnAppEntryWithANameOutsideIt(t *testing.T) { |
| + p, f := box(t) |
| + seedAppEntries(t, p.DataDir, host) |
| + secret := filepath.Join(p.DataDir, "litestream.env") |
| + if err := os.WriteFile(secret, []byte("AWS_SECRET_ACCESS_KEY=x\n"), 0o600); err != nil { |
| + t.Fatal(err) |
| + } |
| + if err := os.Link(secret, filepath.Join(p.DataDir, host, "creds")); err != nil { |
| + t.Skipf("hard links unavailable here: %v", err) |
| + } |
| + // And a legitimate pair, both names inside the blob store. |
| + if err := os.Link(filepath.Join(p.DataDir, host+"-blobs", "ab", "cd"), filepath.Join(p.DataDir, host+"-blobs", "dup")); err != nil { |
| + t.Fatal(err) |
| + } |
| + res := byHost(t, p.Converge(context.Background(), ModeNested)) |
| + r := res[host] |
| + if r.Outcome != "held" || !strings.Contains(r.Reason, "creds") || !strings.Contains(r.Reason, "2 names") || !strings.Contains(r.Reason, "only 1 of them are inside") || !strings.Contains(r.Reason, "-samefile") { |
| + t.Fatalf("linked outside: %+v", r) |
| + } |
| + assertFlatUntouched(t, p, host) |
| + if f.fences[host] == 0 { |
| + t.Fatal("the fence was lifted; the host would wake flat with the link in place") |
| + } |
| + if _, err := os.Stat(filepath.Join(p.DataDir, host, "instance.key")); err != nil { |
| + t.Fatalf("a refused move touched the app's directory: %v", err) |
| + } |
| + // The operator removes the outside name; the inside pair is fine. |
| + os.Remove(filepath.Join(p.DataDir, host, "creds")) |
| + res = byHost(t, p.Converge(context.Background(), ModeNested)) |
| + if res[host].Outcome != "migrated" || f.fences[host] != 0 { |
| + t.Fatalf("after the link is gone: %+v fence=%d", res[host], f.fences[host]) |
| + } |
| + raw := rawInstanceDir(p.DataDir, host) |
| + a, err1 := os.Stat(filepath.Join(raw, host+"-blobs", "ab", "cd")) |
| + b, err2 := os.Stat(filepath.Join(raw, host+"-blobs", "dup")) |
| + if err1 != nil || err2 != nil || !os.SameFile(a, b) { |
| + t.Fatalf("the app's own hard-link pair did not move together: %v %v", err1, err2) |
| + } |
| +} |
| + |
| +// A symlink inside an app's directory is refused like one anywhere else |
| +// the pass moves, and the host is held. |
| +func TestConvergeRefusesASymlinkInsideAnAppEntry(t *testing.T) { |
| + p, f := box(t) |
| + seedAppEntries(t, p.DataDir, host) |
| + if err := os.Symlink("/etc/passwd", filepath.Join(p.DataDir, host, "link")); err != nil { |
| + t.Fatal(err) |
| + } |
| + res := byHost(t, p.Converge(context.Background(), ModeNested)) |
| + if r := res[host]; r.Outcome != "held" || !strings.Contains(r.Reason, "link") || !strings.Contains(r.Reason, "not a regular file") { |
| + t.Fatalf("symlink: %+v", r) |
| + } |
| + if f.fences[host] == 0 { |
| + t.Fatal("the fence was lifted") |
| + } |
| + assertFlatUntouched(t, p, host) |
| +} |
| + |
| +// An extra whose destination is already taken is refused before the |
| +// journal, naming both, never a choice. On the way to nested the |
| +// destination check has just cleared the instance directory, so a taken |
| +// destination was planted since (here: inside the serialized section, |
| +// after the check and before the scan), and the host is held. |
| +func TestConvergeRefusesAnAppEntryWhoseDestinationIsTaken(t *testing.T) { |
| + p, f := box(t) |
| + seedAppEntries(t, p.DataDir, host) |
| + p.Serialize = func(fn func()) { |
| + if !JournalPresent(p.DataDir, host) { |
| + os.MkdirAll(filepath.Join(rawInstanceDir(p.DataDir, host), host), 0o700) |
| + } |
| + fn() |
| + } |
| + res := byHost(t, p.Converge(context.Background(), ModeNested)) |
| + r := res[host] |
| + if r.Outcome != "held" || !strings.Contains(r.Reason, "exists at both") || !strings.Contains(r.Reason, "refusing to choose") { |
| + t.Fatalf("taken destination: %+v", r) |
| + } |
| + if f.fences[host] == 0 { |
| + t.Fatal("the fence was lifted") |
| + } |
| + assertFlatUntouched(t, p, host) |
| + if _, err := os.Stat(filepath.Join(p.DataDir, host, "instance.key")); err != nil { |
| + t.Fatalf("a refused move touched the app's directory: %v", err) |
| + } |
| +} |
| + |
| +// Recovery completes a move with extras from its journal: the names are |
| +// recorded, re-derived on recovery, and moved with the rest. |
| +func TestRecoverCompletesAMoveWithAppEntries(t *testing.T) { |
| + for _, step := range []string{"after-journal", "mid-files", "after-files"} { |
| + t.Run(step, func(t *testing.T) { |
| + p, _ := box(t) |
| + seedAppEntries(t, p.DataDir, host) |
| + p.abortAt = step |
| + p.Converge(context.Background(), ModeNested) |
| + p.abortAt = "" |
| + j, found, err := ReadJournal(p.DataDir, host) |
| + if err != nil || !found || len(j.Extra) != 3 { |
| + t.Fatalf("journal = %+v %v %v, want three extras recorded", j, found, err) |
| + } |
| + res := byHost(t, p.Recover(context.Background(), ModeNested)) |
| + if res[host].Outcome != "recovered" { |
| + t.Fatalf("recover = %+v", res[host]) |
| + } |
| + raw := rawInstanceDir(p.DataDir, host) |
| + for _, rel := range []string{filepath.Join(host, "instance.key"), filepath.Join(host+"-blobs", "ab", "cd"), host + ".cache"} { |
| + if _, err := os.Stat(filepath.Join(raw, rel)); err != nil { |
| + t.Errorf("%s not at destination after recovery from %s: %v", rel, step, err) |
| + } |
| + if _, err := os.Lstat(filepath.Join(p.DataDir, rel)); !errors.Is(err, os.ErrNotExist) { |
| + t.Errorf("%s left behind after recovery from %s", rel, step) |
| + } |
| + } |
| + if JournalPresent(p.DataDir, host) { |
| + t.Error("journal remains") |
| + } |
| + }) |
| + } |
| +} |
| + |
| +// A forged journal for host H that names H-x, the directory of the host |
| +// H-x itself, is refused on recovery: the name passes the shape rule, but |
| +// ownership is judged against the box's rows as they stand, and it is not |
| +// H's. The host stays fenced, nothing moves. |
| +func TestRecoverRefusesAJournalNamingAnotherHostsEntry(t *testing.T) { |
| + p, _ := box(t) |
| + decoy := host + "-x" |
| + if err := p.Registry.Upsert(registry.Route{Host: decoy, Kind: "instance", Socket: filepath.Join(p.SocketDir, decoy+".sock")}); err != nil { |
| + t.Fatal(err) |
| + } |
| + seedAppEntries(t, p.DataDir, decoy) |
| + plan, err := PlanFor(p.DataDir, p.SocketDir, host, ModeNested) |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + present := make([]bool, len(plan.Artifacts)) |
| + present[0] = true |
| + j := plan.Journal(present) |
| + j.Extra = []JournalArtifact{{Name: decoy, Present: true}} |
| + if err := WriteJournal(p.DataDir, j); err != nil { |
| + t.Fatal(err) |
| + } |
| + res := byHost(t, p.Recover(context.Background(), ModeNested)) |
| + if r := res[host]; r.Outcome != "refused" || !strings.Contains(r.Reason, "belongs to "+decoy) || !strings.HasSuffix(r.Reason, "not acting") { |
| + t.Fatalf("forged extra: %+v", r) |
| + } |
| + if _, err := os.Stat(filepath.Join(p.DataDir, decoy, "instance.key")); err != nil { |
| + t.Fatalf("the other host's directory was moved: %v", err) |
| + } |
| + if !JournalPresent(p.DataDir, host) { |
| + t.Fatal("the refused journal was removed") |
| + } |
| +} |
diff --git a/internal/layout/extras_test.go b/internal/layout/extras_test.go| new file mode 100644 |
| index 000000000..7bb43fd2b |
| --- /dev/null |
| +++ b/internal/layout/extras_test.go |
| @@ -0,0 +1,100 @@ |
| +//go:build linux |
| + |
| +package layout |
| + |
| +import ( |
| + "os" |
| + "path/filepath" |
| + "strings" |
| + "testing" |
| + |
| + "github.com/carlosframework/platform/internal/registry" |
| +) |
| + |
| +// The naming rules of extras.go: an entry is a host's when it is the host |
| +// or the host plus a dot or dash suffix (or the activator's prefix form), |
| +// and when two hosts on the box both match, the longer one owns it. |
| +func TestExtraOwnershipIsTheLongestMatchingHost(t *testing.T) { |
| + hosts := []string{"a.b.oncarlos.com", "a.b.oncarlos.com-x", "other.example.com"} |
| + for name, want := range map[string]string{ |
| + "a.b.oncarlos.com": "a.b.oncarlos.com", |
| + "a.b.oncarlos.com-blobs": "a.b.oncarlos.com", |
| + "a.b.oncarlos.com.cache": "a.b.oncarlos.com", |
| + ".activator-a.b.oncarlos.com.db-litestream": "a.b.oncarlos.com", |
| + "a.b.oncarlos.com-x": "a.b.oncarlos.com-x", |
| + "a.b.oncarlos.com-x-blobs": "a.b.oncarlos.com-x", |
| + "a.b.oncarlos.com-x.db": "a.b.oncarlos.com-x", |
| + "a.b.oncarlos.comx": "", |
| + "blobs": "", |
| + "registry.db": "", |
| + } { |
| + if got := ownerOf(name, hosts); got != want { |
| + t.Errorf("ownerOf(%q) = %q, want %q", name, got, want) |
| + } |
| + } |
| +} |
| + |
| +// A journal's extra names pass the scan's own rule or the journal is |
| +// refused: one path element, not one of the plan's own names, named for |
| +// the host, and never absent. Journals from before #256 have no extras. |
| +func TestValidateJudgesTheJournalsExtraNames(t *testing.T) { |
| + dd, rd := t.TempDir(), t.TempDir() |
| + good := func() *Journal { |
| + p, _ := PlanFor(dd, rd, host, ModeNested) |
| + present := make([]bool, len(artifactNames)) |
| + present[0] = true |
| + return p.Journal(present) |
| + } |
| + row := registry.Route{Host: host, DBPath: good().From.DB, Socket: good().From.Socket} |
| + j := good() |
| + j.Extra = []JournalArtifact{{Name: host, Present: true}, {Name: host + "-blobs", Present: true}} |
| + if _, err := Validate(j, dd, rd, row); err != nil { |
| + t.Fatalf("two good extras: %v", err) |
| + } |
| + for name, extra := range map[string]JournalArtifact{ |
| + "a path": {Name: "../" + host, Present: true}, |
| + "a fixed name": {Name: host + ".db", Present: true}, |
| + "another host's": {Name: replica + "-blobs", Present: true}, |
| + "the data dir": {Name: ".", Present: true}, |
| + "recorded absent": {Name: host + "-blobs", Present: false}, |
| + } { |
| + j := good() |
| + j.Extra = []JournalArtifact{extra} |
| + if _, err := Validate(j, dd, rd, row); err == nil || !strings.HasSuffix(err.Error(), "not acting") { |
| + t.Errorf("%s (%q): Validate = %v, want a refusal ending in not acting", name, extra.Name, err) |
| + } |
| + } |
| + j = good() |
| + j.Extra = []JournalArtifact{{Name: host, Present: true}, {Name: host, Present: true}} |
| + if _, err := Validate(j, dd, rd, row); err == nil || !strings.Contains(err.Error(), "twice") { |
| + t.Errorf("a name twice: Validate = %v", err) |
| + } |
| + // The wire shape is additive: a journal without the field reads as |
| + // one with no extras, and one with the field round-trips. |
| + j = good() |
| + j.Extra = []JournalArtifact{{Name: host + "-blobs", Present: true}} |
| + if err := WriteJournal(dd, j); err != nil { |
| + t.Fatal(err) |
| + } |
| + back, found, err := ReadJournal(dd, host) |
| + if err != nil || !found || len(back.Extra) != 1 || back.Extra[0].Name != host+"-blobs" { |
| + t.Fatalf("round trip = %+v %v %v", back, found, err) |
| + } |
| +} |
| + |
| +// UnattributedEntries names what no host on the box owns and the |
| +// platform did not make: the shared directories a flat app with cwd / |
| +// wrote, which no move can take. |
| +func TestUnattributedEntriesSkipsHostsAndPlatformNames(t *testing.T) { |
| + dd := t.TempDir() |
| + for _, name := range []string{"blobs", "registry.db", "layout", ".instances", "instances", host, host + "-blobs", "cache.tmp"} { |
| + if err := os.MkdirAll(filepath.Join(dd, name), 0o700); err != nil { |
| + t.Fatal(err) |
| + } |
| + } |
| + got := UnattributedEntries(dd, []string{host, replica}) |
| + want := []string{filepath.Join(dd, "blobs"), filepath.Join(dd, "cache.tmp")} |
| + if strings.Join(got, " ") != strings.Join(want, " ") { |
| + t.Fatalf("UnattributedEntries = %v, want %v", got, want) |
| + } |
| +} |
diff --git a/internal/layout/journal.go b/internal/layout/journal.go| index 2068259e8..a8a66255f 100644 |
| --- a/internal/layout/journal.go |
| +++ b/internal/layout/journal.go |
| @@ -37,10 +37,25 @@ type Journal struct { |
| From Endpoint `json:"from"` |
| To Endpoint `json:"to"` |
| Artifacts []JournalArtifact `json:"artifacts"` |
| + // Extra is the app-made entries moved with the host (extras.go), by |
| + // base name. Absent in journals written before #256, which moved |
| + // none. Each is re-derived from the host and the name on recovery |
| + // and must pass extraNameOK, so the journal cannot direct a move |
| + // PlanFor and ScanExtras would not have chosen. |
| + Extra []JournalArtifact `json:"extra,omitempty"` |
| Phase Phase `json:"phase"` |
| WrittenAt string `json:"written_at"` |
| } |
| +// extraNames is the recorded extra entries' names, in journal order. |
| +func (j *Journal) extraNames() []string { |
| + var names []string |
| + for _, e := range j.Extra { |
| + names = append(names, e.Name) |
| + } |
| + return names |
| +} |
| + |
| // JournalArtifact records one artifact's presence at the time the journal |
| // was written. The path itself is not stored: it is re-derived by |
| // PlanFor from the host and mode on every read, so a journal can never be |
| @@ -329,6 +344,16 @@ func Validate(j *Journal, dataDir, socketDir string, row registry.Route) (atFrom |
| if !j.Artifacts[0].Present { |
| return false, fmt.Errorf("layout: %s: journal's db artifact is not marked present; not acting", j.Host) |
| } |
| + // The extra names pass the same rule a scan applies, and an extra is |
| + // never absent: the plan carries one because it was seen. |
| + if err := p.AddExtras(j.extraNames()); err != nil { |
| + return false, fmt.Errorf("%w; not acting", err) |
| + } |
| + for _, e := range j.Extra { |
| + if !e.Present { |
| + return false, fmt.Errorf("layout: %s: journal records extra %q absent; not acting", j.Host, e.Name) |
| + } |
| + } |
| switch (Endpoint{DB: row.DBPath, Socket: row.Socket}) { |
| case j.From: |
| return true, nil |
diff --git a/internal/layout/pass.go b/internal/layout/pass.go| index be711ffca..6c9514e83 100644 |
| --- a/internal/layout/pass.go |
| +++ b/internal/layout/pass.go |
| @@ -259,6 +259,7 @@ func (p *Pass) Converge(ctx context.Context, mode Mode) []Result { |
| return append(out, Result{Outcome: "refused", Reason: "list routes: " + err.Error()}) |
| } |
| sort.Slice(routes, func(i, j int) bool { return routes[i].Host < routes[j].Host }) |
| + migrated := false |
| for _, rt := range routes { |
| if !activator.ActivatorManaged(rt) || seen[rt.Host] { |
| continue |
| @@ -266,7 +267,18 @@ func (p *Pass) Converge(ctx context.Context, mode Mode) []Result { |
| if ctx.Err() != nil { |
| break |
| } |
| - out = append(out, p.convergeHost(ctx, rt, routes, mode)) |
| + r := p.convergeHost(ctx, rt, routes, mode) |
| + migrated = migrated || r.Outcome == "migrated" |
| + out = append(out, r) |
| + } |
| + if migrated { |
| + // What no host's move could take (extras.go): a flat app with cwd |
| + // / wrote relative paths straight into the data directory, and a |
| + // name no row owns cannot be given to one host. Said once per pass |
| + // that moved something, for the operator who has to place them. |
| + if left := UnattributedEntries(p.DataDir, hostsOf(routes)); len(left) > 0 { |
| + p.logf("layout: left in %s, owned by no host on this box: %s. a nested instance cannot see them; move what an app wrote into that app's instance directory by hand", p.DataDir, strings.Join(left, ", ")) |
| + } |
| } |
| return out |
| } |
| @@ -543,12 +555,22 @@ func (p *Pass) recheck(host string, routes []registry.Route) (reason string, pla |
| if err := EnsureDirs(p.DataDir, p.socketDir(), host, p.SocketsGID); err != nil { |
| return err.Error(), true |
| } |
| - if _, err := presence(plan); err != nil { |
| + if _, err := presence(plan, hostsOf(routes)); err != nil { |
| return err.Error(), true |
| } |
| return "", false |
| } |
| +// hostsOf is every host with a row on the box, whatever its kind: the set |
| +// an entry's owner is chosen from (extras.go). |
| +func hostsOf(routes []registry.Route) []string { |
| + hosts := make([]string, 0, len(routes)) |
| + for _, rt := range routes { |
| + hosts = append(hosts, rt.Host) |
| + } |
| + return hosts |
| +} |
| + |
| // Held reports whether the pass keeps host fenced and down after a |
| // refused nested move, and why. The fence must still be the pass's own: |
| // a record the activator no longer matches is dropped. |
| @@ -751,7 +773,7 @@ func (p *Pass) shift(host string, mode Mode, prior *Journal) (res Result, h hand |
| // of, with the journal write: a bundle rendered between a scan outside |
| // it and the journal would be recorded absent and left at the old path. |
| p.serialize(func() { |
| - present, err := presence(plan) |
| + present, err := presence(plan, hostsOf(routes)) |
| if err != nil { |
| res, h, hold = Result{host, "refused", err.Error()}, handoverNone, true |
| return |
| @@ -765,8 +787,11 @@ func (p *Pass) shift(host string, mode Mode, prior *Journal) (res Result, h hand |
| } |
| // presence records which artifacts are at their sources, in artifactNames |
| -// order, and refuses a plan whose database is not. |
| -func presence(plan *Plan) ([]bool, error) { |
| +// order, and refuses a plan whose database is not. It then adds to the |
| +// plan every entry the app made for the host (ScanExtras; hosts is the |
| +// box's rows, which decide ownership), so the journal written from the |
| +// plan names them and the move carries them. |
| +func presence(plan *Plan, hosts []string) ([]bool, error) { |
| present := make([]bool, len(plan.Artifacts)) |
| for i, a := range plan.Artifacts { |
| // Nothing is ever renamed for the socket: the listener is gone |
| @@ -784,6 +809,13 @@ func presence(plan *Plan) ([]bool, error) { |
| present[i] = ok |
| plan.Artifacts[i].Present = ok |
| } |
| + extras, err := ScanExtras(plan, hosts) |
| + if err != nil { |
| + return nil, err |
| + } |
| + if err := plan.AddExtras(extras); err != nil { |
| + return nil, err |
| + } |
| // With presence known, every file about to move must have one name. |
| // This runs under the same lock as the journal write, so nothing the |
| // agent itself does can add a name between the check and the journal. |
| @@ -793,6 +825,28 @@ func presence(plan *Plan) ([]bool, error) { |
| return present, nil |
| } |
| +// applyJournal gives a plan rebuilt from the host name what the journal |
| +// recorded about the move: which fixed artifacts were present, and the |
| +// extra entries by name. Validate has already checked the names' shape; |
| +// this checks their ownership against the box's rows as they stand now, |
| +// because a journal is not an instruction (design §3.3): a forged journal |
| +// for H naming H-x, where H-x is a host on the box, would otherwise move |
| +// that host's own directory into H's tree on recovery. |
| +func applyJournal(plan *Plan, j *Journal, hosts []string) error { |
| + for i := range plan.Artifacts { |
| + if i < len(j.Artifacts) { |
| + plan.Artifacts[i].Present = j.Artifacts[i].Present |
| + } |
| + } |
| + all := append([]string{plan.Host}, hosts...) |
| + for _, name := range j.extraNames() { |
| + if owner := ownerOf(name, all); owner != plan.Host { |
| + return fmt.Errorf("layout: %s: journal names extra %q, which belongs to %s", plan.Host, name, owner) |
| + } |
| + } |
| + return plan.AddExtras(j.extraNames()) |
| +} |
| + |
| // commit is steps 6 to 9 up to the journal's removal. The caller reloads |
| // routing afterwards, outside Serialize's lock and still under the fence. |
| // Every return after the journal is written leaves it for Recover, and the |
| @@ -835,7 +889,7 @@ func (p *Pass) commitWritten(plan *Plan, j *Journal, aside []string) Result { |
| if p.abortAt == "after-journal" { |
| return Result{host, "refused", "aborted"} |
| } |
| - if err := p.moveFiles(plan, j); err != nil { |
| + if err := p.moveFiles(plan); err != nil { |
| return Result{host, "refused", err.Error()} |
| } |
| moved, err := VerifyMoved(plan) |
| @@ -868,9 +922,16 @@ func (p *Pass) commitWritten(plan *Plan, j *Journal, aside []string) Result { |
| if plan.Mode == ModeFlat { |
| outcome = "reverted" |
| } |
| + var events []string |
| + if extras := plan.ExtraNames(); len(extras) > 0 { |
| + events = append(events, "moved with it: "+strings.Join(extras, ", ")) |
| + } |
| if len(aside) > 0 { |
| - p.logf("layout: %s: %s -> %s (kept aside: %s)", host, plan.From.DB, plan.To.DB, strings.Join(aside, ", ")) |
| - p.note(host, fmt.Sprintf("%s at %s: kept aside %s", outcome, p.now().UTC().Format(time.RFC3339), strings.Join(aside, ", "))) |
| + events = append(events, "kept aside: "+strings.Join(aside, ", ")) |
| + } |
| + if len(events) > 0 { |
| + p.logf("layout: %s: %s -> %s (%s)", host, plan.From.DB, plan.To.DB, strings.Join(events, "; ")) |
| + p.note(host, fmt.Sprintf("%s at %s: %s", outcome, p.now().UTC().Format(time.RFC3339), strings.Join(events, "; "))) |
| } else { |
| p.logf("layout: %s: %s -> %s", host, plan.From.DB, plan.To.DB) |
| } |
| @@ -879,16 +940,17 @@ func (p *Pass) commitWritten(plan *Plan, j *Journal, aside []string) Result { |
| var errAborted = errors.New("layout: aborted by test seam") |
| -// moveFiles renames every artifact the journal records present. One |
| -// already at its destination is done; one still at its source is renamed; |
| -// one at neither is lost, and one at both was not left that way by a |
| -// crash (the destination check cleared every destination before the |
| -// journal was written, and a rename leaves a file in exactly one place). |
| -// Either of the last two is an error that keeps the journal and so the |
| -// fence. |
| -func (p *Pass) moveFiles(plan *Plan, j *Journal) error { |
| - for i, a := range plan.Artifacts { |
| - if a.Name == "socket" || !j.Artifacts[i].Present { |
| +// moveFiles renames every artifact the plan records present (from the |
| +// scan on a first attempt, from the journal on recovery), the app's own |
| +// entries included. One already at its destination is done; one still at |
| +// its source is renamed; one at neither is lost, and one at both was not |
| +// left that way by a crash (the destination check cleared every |
| +// destination before the journal was written, and a rename leaves a file |
| +// in exactly one place). Either of the last two is an error that keeps |
| +// the journal and so the fence. |
| +func (p *Pass) moveFiles(plan *Plan) error { |
| + for _, a := range plan.Artifacts { |
| + if a.Name == "socket" || !a.Present { |
| continue |
| } |
| atDst, err := lstatPresent(a.Dst) |
| @@ -1032,6 +1094,13 @@ func (p *Pass) recoverHost(ctx context.Context, host string, byHost map[string]r |
| if err != nil { |
| return Result{host, "refused", notActing(err)} |
| } |
| + hosts := make([]string, 0, len(byHost)) |
| + for h := range byHost { |
| + hosts = append(hosts, h) |
| + } |
| + if err := applyJournal(plan, j, hosts); err != nil { |
| + return Result{host, "refused", notActing(err)} |
| + } |
| keepFence := false |
| r, fence, gen := p.quiesce(ctx, host, p.generation()) |
| switch fence { |
| @@ -1141,8 +1210,8 @@ func (p *Pass) recoverHost(ctx context.Context, host string, byHost map[string]r |
| // it records present being at its destination before the row is swapped. |
| func (p *Pass) complete(plan *Plan, j *Journal, atFrom, remove bool) Result { |
| host, phase := plan.Host, j.Phase |
| - for i, a := range plan.Artifacts { |
| - if a.Name == "socket" || j.Artifacts[i].Present { |
| + for _, a := range plan.Artifacts { |
| + if a.Name == "socket" || a.Present { |
| continue |
| } |
| for _, path := range []string{a.Src, a.Dst} { |
| @@ -1157,17 +1226,12 @@ func (p *Pass) complete(plan *Plan, j *Journal, atFrom, remove bool) Result { |
| } |
| switch j.Phase { |
| case PhaseFiles: |
| - if err := p.moveFiles(plan, j); err != nil { |
| + if err := p.moveFiles(plan); err != nil { |
| return Result{host, "refused", notActing(err)} |
| } |
| - // The same check a first attempt makes before its row flips: the |
| - // plan was rebuilt from the host name, so it carries the journal's |
| - // presence for the check to know which files it moved. |
| - for i := range plan.Artifacts { |
| - if i < len(j.Artifacts) { |
| - plan.Artifacts[i].Present = j.Artifacts[i].Present |
| - } |
| - } |
| + // The same check a first attempt makes before its row flips; the |
| + // plan carries the journal's presence (applyJournal) so the check |
| + // knows which files it moved. |
| moved, err := VerifyMoved(plan) |
| if len(moved) > 0 { |
| p.logf("layout: %s: quarantined on recovery -> %s", host, strings.Join(moved, ", ")) |
| @@ -1184,8 +1248,8 @@ func (p *Pass) complete(plan *Plan, j *Journal, atFrom, remove bool) Result { |
| // The row is about to name the destinations, so every file the |
| // journal records present must be there: a row pointed at a |
| // missing database has the next wake restore a second one. |
| - for i, a := range plan.Artifacts { |
| - if a.Name == "socket" || !j.Artifacts[i].Present { |
| + for _, a := range plan.Artifacts { |
| + if a.Name == "socket" || !a.Present { |
| continue |
| } |
| present, err := lstatPresent(a.Dst) |
diff --git a/internal/layout/plan.go b/internal/layout/plan.go| index a396b6ce8..35988def1 100644 |
| --- a/internal/layout/plan.go |
| +++ b/internal/layout/plan.go |
| @@ -54,6 +54,11 @@ type Artifact struct { |
| // name alone, §3.3), so Present is always false here; the pass fills |
| // it in during preflight. |
| Present bool |
| + // Extra marks an entry the app made for itself under the host's name |
| + // (extras.go), found by ScanExtras or recorded in the journal rather |
| + // than derived by PlanFor. Always Present: it is in the plan because |
| + // it was seen. Name is the entry's own base name. |
| + Extra bool |
| } |
| // Plan is everything the pass will do for one host, computed before it |
| @@ -184,10 +189,15 @@ func PlanFor(dataDir, socketDir, host string, mode Mode) (*Plan, error) { |
| } |
| // Journal builds the journal for this plan with the given presence |
| -// vector (one bool per artifact, artifactNames order). |
| +// vector (one bool per fixed artifact, artifactNames order), plus the |
| +// plan's extra entries by name. |
| func (p *Plan) Journal(present []bool) *Journal { |
| j := &Journal{Host: p.Host, Mode: p.Mode, From: p.From, To: p.To, Phase: PhaseFiles} |
| for i, a := range p.Artifacts { |
| + if a.Extra { |
| + j.Extra = append(j.Extra, JournalArtifact{Name: a.Name, Present: true}) |
| + continue |
| + } |
| j.Artifacts = append(j.Artifacts, JournalArtifact{Name: a.Name, Present: i < len(present) && present[i]}) |
| } |
| return j |
diff --git a/internal/layout/preflight.go b/internal/layout/preflight.go| index b29513bdf..3d118c5d6 100644 |
| --- a/internal/layout/preflight.go |
| +++ b/internal/layout/preflight.go |
| @@ -290,6 +290,12 @@ func LinkCheck(p *Plan, at func(Artifact) string) error { |
| } |
| continue |
| } |
| + if a.Extra && fi.IsDir() { |
| + if err := walkExtraLinks(p.Host, p.DataDir, path); err != nil { |
| + return err |
| + } |
| + continue |
| + } |
| if err := oneName(p.Host, p.DataDir, path, fi); err != nil { |
| return err |
| } |
| @@ -297,6 +303,58 @@ func LinkCheck(p *Plan, at func(Artifact) string) error { |
| return nil |
| } |
| +// walkExtraLinks is walkLinks for a directory an app made for itself |
| +// (extras.go), with one allowance the shadow directory does not need: a |
| +// file with several names may be moved when EVERY one of those names is |
| +// inside the directory, since the move carries them all and nothing |
| +// outside can reach the file afterwards. Apps deduplicate content by |
| +// hard-linking inside their own blob store, and refusing that would hold |
| +// a host down for its own ordinary files. A name outside the directory |
| +// is still a second reader and writer the move does not know about, and |
| +// is refused with the inode and the find that lists it. Symlinks and |
| +// special files are refused as in walkLinks. |
| +func walkExtraLinks(host, dataDir, root string) error { |
| + type linked struct { |
| + path string |
| + nlink uint64 |
| + seen uint64 |
| + } |
| + byInode := map[uint64]*linked{} |
| + err := filepath.WalkDir(root, func(path string, d os.DirEntry, err error) error { |
| + if err != nil { |
| + return fmt.Errorf("layout: %s: %w", host, err) |
| + } |
| + if d.IsDir() { |
| + return nil |
| + } |
| + fi, err := d.Info() |
| + if err != nil { |
| + return fmt.Errorf("layout: %s: %w", host, err) |
| + } |
| + if !fi.Mode().IsRegular() { |
| + return fmt.Errorf("layout: %s: %s is not a regular file (%v); refusing to move it", host, path, fi.Mode().Type()) |
| + } |
| + if n, ino := linkCount(fi); n > 1 { |
| + l := byInode[ino] |
| + if l == nil { |
| + l = &linked{path: path, nlink: n} |
| + byInode[ino] = l |
| + } |
| + l.seen++ |
| + } |
| + return nil |
| + }) |
| + if err != nil { |
| + return err |
| + } |
| + for ino, l := range byInode { |
| + if l.seen < l.nlink { |
| + return fmt.Errorf("layout: %s: %s has %d names (hard links, inode %d) and only %d of them are inside %s; refusing to move a file something else can still reach. find the others with: find %s -xdev -samefile %s", host, l.path, l.nlink, ino, l.seen, root, dataDir, l.path) |
| + } |
| + } |
| + return nil |
| +} |
| + |
| func walkLinks(host, dataDir, root string) error { |
| return filepath.WalkDir(root, func(path string, d os.DirEntry, err error) error { |
| if err != nil { |