amadan
server: bootstrap a specific box AS the short-link apex (#1226)
* server: bootstrap enables the links redirector unit `lchat bootstrap` has always WRITTEN lchat-links.service, but nothing ever enabled it — so the short-link redirector shipped in the repo and ran on no box. That is why 11ch.at serves nothing today: no unit, and so no route and no certificate either. Enabling it everywhere is safe and deliberate. It holds no secrets, no database and no env file (just an in-memory verification cache), and it listens on a unix socket the router only reaches if a route points at it — so a box with no short-link route runs one idle process and nothing more. That matches the intent already written above the embed: "safe to enable immediately, no operator env-file step first". Registering the route stays a separate, deliberate step (`lchat add-service <apex> --socket /run/lchat/links.sock`), because the apex is fleet-specific: a self-hosted box has no business claiming 11ch.at. enable + restart, not `enable --now`, for the same reason the router does it: on the installer's update path `--now` is a no-op for an already-active unit, leaving the OLD process running against the NEW binary. The unit execs /opt/lchat/lchat directly rather than a versioned live symlink, so a bootstrap re-run is the only thing that picks a new build up. A failure warns rather than dies: the redirector is inert without a route, so it must never be why a box fails to come up. Co-Authored-By: Claude <noreply@anthropic.com> * server: the auto-updater restarts lchat-links too Units that exec /opt/lchat/lchat directly — rather than a per-instance versioned live symlink — keep running their OLD inode until something restarts them. The updater already bounced lchat-control and lchat-cloud for exactly that reason; links has the same shape and was missing, so once enabled it would have been the one service that never picked up a new build. It is also the cheapest of the three to bounce: no database, no connections, just an in-memory verification cache that re-populates lazily. try-restart is a no-op where the unit isn't running, so boxes that route no short links are unaffected. The unit list moves into one `controlBinaryUnits` var. Two existing tests asserted a literal "exactly 2 restarts" — their real intent is "one restart each, no rollback round", so they now count against that list and adding a unit to the update path no longer reads as a spurious extra restart. Co-Authored-By: Claude <noreply@anthropic.com> * server: record why the links redirector is ours-only in practice Enabling the unit everywhere is safe, but "safe" is not the same as "useful to a self-hoster", and the comment claiming the former should say so. Two values in linkscmd.go are Eleven-specific and not configurable: the apex bounces to linksMarketingURL (elevenmessenger.com), and the AASA advertises appleAppID (our iOS app). A self-hoster who routed their own apex at the socket would hand their visitors to our marketing site and publish our app id under their domain. Nothing leads them there — no route is ever registered automatically, and 11ch.at is not theirs to claim — so this is latent, not live, and the unit sits idle on their box. Noted rather than fixed: making those two configurable is what self-hostable short links would actually take. Co-Authored-By: Claude <noreply@anthropic.com> * server: bootstrap a specific box AS the short-link apex Supersedes the earlier "enable the links unit on every box" approach in this branch. Exactly one box in a fleet is the short-link apex, so only that box should run the redirector — every other box, and every self-hoster's, has no use for the process at all. `lchat bootstrap --links-apex <host>` starts the redirector and registers <host> as a service route pointing at /run/lchat/links.sock. Without the flag, bootstrap converges to whatever the registry already says: routed -> enable + restart, not routed -> disable --now. Keying the state on the ROUTE rather than the flag is the point. A flag-keyed design has a footgun --no-update-timer is stuck with (no timer state persists, so you must repeat the flag or it flips back): an operator re-running plain bootstrap on the apex box would silently switch short links off. The route persists in the registry, so it can be the source of truth — bootstrap just makes the box match it, and re-running is always safe. The apex is matched on the SOCKET, not a hostname, so the same code recognises 11ch.at in production and dev.11ch.at on the dev mesh with neither named here. An unreadable registry reads as "not the apex" — an idle box beats starting a service we couldn't justify. The route is registered only once the socket answers (the check add-service makes), because a route to a dead socket is a 502-shaped hole rather than a pending setup step; and repointing a host that already routes elsewhere is refused rather than silently taken over. The known limitation moves to where an operator meets it: linksMarketingURL and appleAppID are Eleven-specific and not configurable, so pointing your own apex here serves our marketing site and our app id under your domain. Co-Authored-By: Claude <noreply@anthropic.com> * server: a links read must never create the registry linksRouted runs on every plain bootstrap — including every self-hoster's install.sh — and openRegistry CREATES the database (sql.Open plus migrate's CREATE TABLE IF NOT EXISTS). bootstrap is root, while the router runs as the lchat user and creates the registry itself moments earlier. If this read won that race it would leave a ROOT-OWNED registry the router cannot write, breaking all routing on a fresh box. Stat first, and answer "not the apex" when the file is absent. A read must never be the thing that creates it. The existing "unreadable registry" case passed for the wrong reason — its path had a missing PARENT dir, so openRegistry failed rather than creating anything. The new case is the real fresh-box shape (dir present, file absent) and asserts the directory is still empty afterwards. Verified it catches the bug: without the guard it fails on registry.db, -shm and -wal. Co-Authored-By: Claude <noreply@anthropic.com> ---------
Co-authored-by: Claude <noreply@anthropic.com>
6 files changed,
+368
−18
bootstrap.go+147 −2bootstrap_test.go+181 −0control_selfupdate_test.go+7 −5docs/planet-scale-guide/processes.html+3 −3docs/planet-scale-guide/reference.html+6 −1updater.go+24 −7
diff --git a/bootstrap.go b/bootstrap.go| index 24b8eba..cd93916 100644 |
| --- a/bootstrap.go |
| +++ b/bootstrap.go |
| @@ -9,6 +9,7 @@ import ( |
| "os/user" |
| "path/filepath" |
| "strings" |
| + "time" |
| ) |
| // The systemd units ship inside the binary (source of truth: deploy/systemd), so |
| @@ -50,8 +51,11 @@ var unitConnect string |
| // The links redirector (11ch.at short links, issue #905) is also a sibling |
| // service, but unlike push/home it's a subcommand of this same binary, not a |
| -// separate one, and holds no secrets or database — so it installs everywhere |
| -// and is safe to enable immediately, no operator env-file step first. |
| +// separate one, and holds no secrets or database. The unit installs |
| +// everywhere; only the box declared the short-link apex actually RUNS it |
| +// (`bootstrap --links-apex <host>`, converged from the registry thereafter — |
| +// see syncLinks). Exactly one box in a fleet is the apex, so every other box |
| +// keeps the unit written and disabled. |
| // |
| //go:embed deploy/systemd/lchat-links.service |
| var unitLinks string |
| @@ -270,6 +274,136 @@ func enableCollectorTimers() { |
| } |
| } |
| +// linksSocket is where the redirector listens — the contract between |
| +// lchat-links.service's ExecStart and the registry route that fronts it. |
| +const linksSocket = "/run/lchat/links.sock" |
| + |
| +// linksRouted reports whether THIS box is the short-link apex, by asking the |
| +// only thing that actually knows: the router's registry. Any route pointing at |
| +// linksSocket means an operator has declared this box the apex. |
| +// |
| +// Matching on the SOCKET rather than a hostname is what keeps this general — |
| +// it recognises 11ch.at in production and dev.11ch.at on the dev mesh without |
| +// either being named in the code. |
| +// |
| +// A registry we can't read is reported as "not the apex": that is the safe |
| +// direction (an idle box rather than a service we couldn't justify starting), |
| +// and on a fresh box the registry legitimately has nothing in it yet. |
| +// |
| +// The Stat guard matters more than it looks. openRegistry CREATES the database |
| +// (sql.Open + migrate's CREATE TABLE IF NOT EXISTS), and this runs on every |
| +// plain bootstrap — including every self-hoster's install.sh. bootstrap is |
| +// root, while the router runs as the lchat user and creates the registry |
| +// itself moments earlier; if this read won that race it would leave a |
| +// ROOT-OWNED registry the router cannot write, breaking all routing on a |
| +// fresh box. A read must never be the thing that creates it. |
| +func linksRouted(registryPath string) bool { |
| + if _, err := os.Stat(registryPath); err != nil { |
| + return false |
| + } |
| + reg, err := openRegistry(registryPath) |
| + if err != nil { |
| + return false |
| + } |
| + routes, err := reg.list() |
| + if err != nil { |
| + return false |
| + } |
| + for _, rt := range routes { |
| + if rt.Socket == linksSocket { |
| + return true |
| + } |
| + } |
| + return false |
| +} |
| + |
| +// syncLinks converges the short-link redirector (`lchat links run`, issue #905) |
| +// to whether this box is the apex. |
| +// |
| +// The unit has always been written by bootstrap, but nothing ever enabled it — |
| +// so the service existed in the repo and ran nowhere, which is why 11ch.at |
| +// served nothing long after the code merged. |
| +// |
| +// The state is keyed on the ROUTE, not on a flag, and that distinction is the |
| +// whole design. Only one box in a fleet is the short-link apex; every other |
| +// box (and every self-hoster's) has no business running the process at all. |
| +// But if a *flag* decided it, an operator who later ran a plain `bootstrap` on |
| +// the apex box would silently turn short links off — the same footgun |
| +// --no-update-timer has, and has to have, because no timer state persists |
| +// anywhere. Here the route persists in the registry, so it can be the source |
| +// of truth: bootstrap simply makes the box match what the registry says, and |
| +// re-running it is always safe. |
| +// |
| +// restart, not `enable --now`, for exactly the reason the router does it: on |
| +// the installer's update path `--now` is a no-op for an already-active unit, |
| +// which would leave the OLD process running against the NEW binary. The unit |
| +// execs /opt/lchat/lchat directly (not a versioned live symlink), so a |
| +// bootstrap re-run is the only thing that picks up a new build. |
| +// |
| +// Failures warn rather than die(): short links are one optional service, and |
| +// must never be the reason a box fails to come up. |
| +func syncLinks(apex bool) { |
| + if !apex { |
| + // Not the apex — make sure it isn't running. Idempotent: bootstrap has |
| + // just written the unit file, and disabling an already-disabled unit is |
| + // a no-op, so this is quiet on every ordinary box. |
| + if err := systemctl("disable", "--now", "lchat-links"); err != nil { |
| + fmt.Printf("bootstrap: warning: disable lchat-links: %v\n", err) |
| + } |
| + return |
| + } |
| + if err := systemctl("enable", "lchat-links"); err != nil { |
| + fmt.Printf("bootstrap: warning: enable lchat-links: %v (short links will not serve on this box)\n", err) |
| + return |
| + } |
| + if err := systemctl("restart", "lchat-links"); err != nil { |
| + fmt.Printf("bootstrap: warning: start lchat-links: %v (short links will not serve on this box)\n", err) |
| + } |
| +} |
| + |
| +// registerLinksApex makes this box the short-link apex for host: start the |
| +// redirector, then register the route that both points the router at it and |
| +// records the decision durably (so later bootstraps converge to it). |
| +// |
| +// Order matters — the route is registered only once the socket answers, the |
| +// same check `add-service` makes, because a route to a dead socket is a |
| +// 502-shaped hole in the router rather than a pending setup step. |
| +// |
| +// KNOWN LIMITATION worth stating where an operator will meet it: the |
| +// redirector is really only OURS to serve. Two values in linkscmd.go are |
| +// Eleven-specific and not configurable — the apex bounces to |
| +// linksMarketingURL (elevenmessenger.com), and the apple-app-site-association |
| +// advertises appleAppID (our iOS app). Pointing your own apex here therefore |
| +// sends your visitors to our marketing site and publishes our app id under |
| +// your domain. Making those configurable is what self-hostable short links |
| +// would take; until then this flag is for our fleet. |
| +func registerLinksApex(host, registryPath string) { |
| + host = strings.ToLower(strings.TrimSpace(host)) |
| + if !validHost(host) { |
| + die("bootstrap: --links-apex %q is not a valid hostname", host) |
| + } |
| + syncLinks(true) |
| + if err := waitReachable(linksSocket, 10*time.Second); err != nil { |
| + die("bootstrap: links redirector not answering on %s: %v (check: journalctl -u lchat-links)", linksSocket, err) |
| + } |
| + reg, err := openRegistry(registryPath) |
| + if err != nil { |
| + die("bootstrap: open registry: %v", err) |
| + } |
| + if existing, err := reg.get(host); err != nil { |
| + die("bootstrap: %v", err) |
| + } else if existing != nil && existing.Socket != linksSocket { |
| + die("bootstrap: %s is already routed to %s (kind %s) — refusing to repoint it at the links redirector", host, existing.Socket, existing.Kind) |
| + } |
| + if err := reg.upsert(Route{Host: host, Kind: "service", Socket: linksSocket, Status: "ready"}); err != nil { |
| + die("bootstrap: register %s: %v", host, err) |
| + } |
| + if err := systemctl("reload", "lchat-router"); err != nil { |
| + die("bootstrap: reload router: %v (route added; run: systemctl reload lchat-router)", err) |
| + } |
| + fmt.Printf("bootstrap: short-link apex %s -> %s\n", host, linksSocket) |
| +} |
| + |
| // runBootstrap installs the systemd units + runtime dir and starts the router, so |
| // provisioning a fresh box is one command after the binary is copied to |
| // /opt/lchat/lchat. Idempotent — safe to re-run. See docs/instance-router.md. |
| @@ -279,6 +413,7 @@ func runBootstrap(args []string) { |
| unitDir := fs.String("unit-dir", "/etc/systemd/system", "systemd unit directory") |
| noStart := fs.Bool("no-start", false, "install units but don't enable/start the router") |
| collectorsOnly := fs.Bool("collectors-only", false, "refresh ONLY the status collectors (scripts, units, viewer pages) + enable their timers — no service user, no control token, no router restart") |
| + linksApex := fs.String("links-apex", "", "make this box the short-link apex for this host (starts the redirector and routes the host at it); without it, a box that isn't already routed as the apex doesn't run the redirector at all") |
| _ = fs.Parse(args) |
| // --collectors-only is the no-root-shell refresh path: with the fleet |
| @@ -365,6 +500,16 @@ func runBootstrap(args []string) { |
| if err := systemctl("enable", "--now", "lchat-update.timer"); err != nil { |
| die("bootstrap: enable update timer: %v", err) |
| } |
| + // Short links: --links-apex declares this box the apex (and records that in |
| + // the registry); otherwise converge to whatever the registry already says, |
| + // so a re-run neither turns a live apex off nor starts the redirector on a |
| + // box that has no use for it. |
| + registryPath := filepath.Join(*dataDir, "registry.db") |
| + if *linksApex != "" { |
| + registerLinksApex(*linksApex, registryPath) |
| + } else { |
| + syncLinks(linksRouted(registryPath)) |
| + } |
| // The collector units + pages were written above and picked up by the reload; |
| // enable their 5-minute timers (safe everywhere — a pass with no /data/status.env |
| // creds is a near no-op). |
diff --git a/bootstrap_test.go b/bootstrap_test.go| index cc41ace..b67adbf 100644 |
| --- a/bootstrap_test.go |
| +++ b/bootstrap_test.go |
| @@ -156,6 +156,187 @@ func TestEnableCollectorTimers(t *testing.T) { |
| } |
| } |
| +// TestSyncLinks: the redirector runs on the apex box and nowhere else. |
| +// enable+restart (never `enable --now`) so the installer's update path |
| +// re-execs the new binary rather than leaving the old process up; disable |
| +// --now on every other box so a fleet doesn't carry idle redirectors. |
| +func TestSyncLinks(t *testing.T) { |
| + for _, tc := range []struct { |
| + name string |
| + apex bool |
| + want []string |
| + }{ |
| + {"apex box runs it", true, []string{"enable lchat-links", "restart lchat-links"}}, |
| + {"every other box does not", false, []string{"disable --now lchat-links"}}, |
| + } { |
| + t.Run(tc.name, func(t *testing.T) { |
| + oldSysctl := systemctl |
| + t.Cleanup(func() { systemctl = oldSysctl }) |
| + var calls []string |
| + systemctl = func(args ...string) error { |
| + calls = append(calls, strings.Join(args, " ")) |
| + return nil |
| + } |
| + |
| + syncLinks(tc.apex) |
| + |
| + if len(calls) != len(tc.want) { |
| + t.Fatalf("systemctl calls = %v, want %v", calls, tc.want) |
| + } |
| + for i, w := range tc.want { |
| + if calls[i] != w { |
| + t.Errorf("call %d = %q, want %q", i, calls[i], w) |
| + } |
| + } |
| + }) |
| + } |
| +} |
| + |
| +// TestSyncLinksSurvivesFailure: short links are one optional service, so a box |
| +// that can't drive the unit must still finish bootstrapping rather than die(). |
| +func TestSyncLinksSurvivesFailure(t *testing.T) { |
| + oldSysctl := systemctl |
| + t.Cleanup(func() { systemctl = oldSysctl }) |
| + systemctl = func(args ...string) error { return os.ErrPermission } |
| + |
| + syncLinks(true) // must return normally rather than exiting |
| + syncLinks(false) // ditto for the disable path |
| +} |
| + |
| +// TestLinksRouted: the registry is what decides whether this box is the apex, |
| +// and it is matched on the SOCKET, not a hostname — that is what lets the same |
| +// code recognise 11ch.at in production and dev.11ch.at on the dev mesh without |
| +// either being named here. |
| +// |
| +// It is also what makes re-running bootstrap safe: the apex box keeps its |
| +// route, so a later plain `bootstrap` sees routed and leaves short links up, |
| +// instead of silently switching them off the way a flag-keyed design would. |
| +func TestLinksRouted(t *testing.T) { |
| + newRegistry := func(t *testing.T, routes ...Route) string { |
| + t.Helper() |
| + path := filepath.Join(t.TempDir(), "registry.db") |
| + reg, err := openRegistry(path) |
| + if err != nil { |
| + t.Fatalf("open registry: %v", err) |
| + } |
| + for _, rt := range routes { |
| + if err := reg.upsert(rt); err != nil { |
| + t.Fatalf("seed %s: %v", rt.Host, err) |
| + } |
| + } |
| + return path |
| + } |
| + |
| + t.Run("no routes at all", func(t *testing.T) { |
| + if linksRouted(newRegistry(t)) { |
| + t.Error("an empty registry must not read as the apex") |
| + } |
| + }) |
| + |
| + t.Run("instances only", func(t *testing.T) { |
| + path := newRegistry(t, |
| + Route{Host: "a.example.com", Kind: "instance", Socket: "/run/lchat/a.example.com.sock", Status: "ready"}, |
| + Route{Host: "home.example.com", Kind: "service", Socket: "/run/lchat/home.sock", Status: "ready"}, |
| + ) |
| + if linksRouted(path) { |
| + t.Error("a box with instances and other services must not read as the apex") |
| + } |
| + }) |
| + |
| + t.Run("production apex", func(t *testing.T) { |
| + path := newRegistry(t, |
| + Route{Host: "a.example.com", Kind: "instance", Socket: "/run/lchat/a.example.com.sock", Status: "ready"}, |
| + Route{Host: "11ch.at", Kind: "service", Socket: linksSocket, Status: "ready"}, |
| + ) |
| + if !linksRouted(path) { |
| + t.Error("a route pointing at the links socket means this box IS the apex") |
| + } |
| + }) |
| + |
| + t.Run("dev apex — recognised without being named", func(t *testing.T) { |
| + path := newRegistry(t, Route{Host: "dev.11ch.at", Kind: "service", Socket: linksSocket, Status: "ready"}) |
| + if !linksRouted(path) { |
| + t.Error("the apex is identified by socket, so any hostname must work") |
| + } |
| + }) |
| + |
| + t.Run("unreadable registry is not the apex", func(t *testing.T) { |
| + // Fail safe: an idle box beats starting a service we couldn't justify. |
| + if linksRouted(filepath.Join(t.TempDir(), "nonexistent", "registry.db")) { |
| + t.Error("an unreadable registry must not read as the apex") |
| + } |
| + }) |
| + |
| + // The fresh-box case, and the one that actually guards a footgun: /data |
| + // exists but the registry does not yet. openRegistry CREATES the database, |
| + // and this read runs as root on every plain bootstrap (including every |
| + // self-hoster's install.sh) — while the router, running as the lchat user, |
| + // creates the registry itself moments earlier. If this read won that race |
| + // it would leave a root-owned registry the router cannot write, breaking |
| + // all routing on a fresh box. So it must answer "not the apex" WITHOUT |
| + // creating anything. |
| + t.Run("absent registry is not the apex and is not created", func(t *testing.T) { |
| + dir := t.TempDir() |
| + path := filepath.Join(dir, "registry.db") |
| + |
| + if linksRouted(path) { |
| + t.Error("an absent registry must not read as the apex") |
| + } |
| + |
| + entries, err := os.ReadDir(dir) |
| + if err != nil { |
| + t.Fatalf("read dir: %v", err) |
| + } |
| + for _, e := range entries { |
| + t.Errorf("linksRouted created %s — a read must never create the registry", e.Name()) |
| + } |
| + }) |
| +} |
| + |
| +// TestRestartControlUnitsIncludesLinks: the units that exec /opt/lchat/lchat |
| +// directly (rather than a versioned live symlink) keep their OLD inode until |
| +// something restarts them, so every one of them must ride the updater's |
| +// restart pass — otherwise the redirector is the one service that never picks |
| +// up a new build. |
| +func TestRestartControlUnitsIncludesLinks(t *testing.T) { |
| + oldSysctl := systemctl |
| + t.Cleanup(func() { systemctl = oldSysctl }) |
| + var calls []string |
| + systemctl = func(args ...string) error { |
| + calls = append(calls, strings.Join(args, " ")) |
| + return nil |
| + } |
| + |
| + restartControlUnits() |
| + |
| + want := []string{ |
| + "try-restart lchat-control.service", |
| + "try-restart lchat-cloud.service", |
| + "try-restart lchat-links.service", |
| + } |
| + if len(calls) != len(want) { |
| + t.Fatalf("systemctl calls = %v, want %v", calls, want) |
| + } |
| + for i, w := range want { |
| + if calls[i] != w { |
| + t.Errorf("call %d = %q, want %q", i, calls[i], w) |
| + } |
| + } |
| +} |
| + |
| +// TestLinksUnitShape guards the embedded unit against drift: it must exec the |
| +// links subcommand on the socket bootstrap's route registration expects, and |
| +// must NOT carry an EnvironmentFile (it holds no secrets — the reason enabling |
| +// it everywhere is safe). |
| +func TestLinksUnitShape(t *testing.T) { |
| + if !strings.Contains(unitLinks, "ExecStart=/opt/lchat/lchat links run --socket /run/lchat/links.sock") { |
| + t.Error("links unit missing its ExecStart line (socket path is the route contract)") |
| + } |
| + if strings.Contains(unitLinks, "EnvironmentFile") { |
| + t.Error("links unit should carry no EnvironmentFile — it holds no secrets") |
| + } |
| +} |
| + |
| // assertFileMode asserts a file exists with exactly the given permission bits. |
| func assertFileMode(t *testing.T, path string, want os.FileMode) { |
| t.Helper() |
diff --git a/control_selfupdate_test.go b/control_selfupdate_test.go| index 6159ecc..94bc96b 100644 |
| --- a/control_selfupdate_test.go |
| +++ b/control_selfupdate_test.go |
| @@ -156,9 +156,11 @@ func TestControlSelfUpdateHealthyCommits(t *testing.T) { |
| if got, _ := os.ReadFile(self); string(got) != string(newBin) { |
| t.Fatalf("healthy build should stay swapped in: %q", got) |
| } |
| - // One restart of each backend, never a second (rollback) round. |
| - if n := strings.Count(strings.Join(*calls, ";"), "try-restart"); n != 2 { |
| - t.Fatalf("expected exactly 2 restarts (no rollback), got %d: %v", n, *calls) |
| + // One restart of each backend, never a second (rollback) round. Counted |
| + // against controlBinaryUnits rather than a literal, so adding a unit to the |
| + // update path doesn't read as a spurious extra restart here. |
| + if n, want := strings.Count(strings.Join(*calls, ";"), "try-restart"), len(controlBinaryUnits); n != want { |
| + t.Fatalf("expected exactly %d restarts (no rollback), got %d: %v", want, n, *calls) |
| } |
| } |
| @@ -234,8 +236,8 @@ func TestControlSelfUpdateInactiveControldCommits(t *testing.T) { |
| t.Fatalf("good binary must stay committed, not rolled back: %q", got) |
| } |
| joined := strings.Join(*calls, ";") |
| - if n := strings.Count(joined, "try-restart"); n != 2 { |
| - t.Fatalf("expected exactly 2 restarts (commit, no rollback), got %d: %v", n, *calls) |
| + if n, want := strings.Count(joined, "try-restart"), len(controlBinaryUnits); n != want { |
| + t.Fatalf("expected exactly %d restarts (commit, no rollback), got %d: %v", want, n, *calls) |
| } |
| if strings.Contains(joined, "lchat-router") { |
| t.Fatalf("router must never restart: %v", *calls) |
diff --git a/docs/planet-scale-guide/processes.html b/docs/planet-scale-guide/processes.html| index 354daff..5c19fe1 100644 |
| --- a/docs/planet-scale-guide/processes.html |
| +++ b/docs/planet-scale-guide/processes.html |
| @@ -41,7 +41,7 @@ |
| <tr><td><code>lchat-hooks@<host>.service</code></td><td>The optional webhook-ingest sidecar for one instance (template unit): serves <code>in.<host></code> behind the router, turning inbound webhook HTTP into encrypted events. Stood up by <code>lchat add-hooks <host></code>. <span class="status partial">built, not yet deployed</span></td><td><code>/opt/lchat/live/<host> hooks run</code></td><td><code>lchat</code></td></tr> |
| <tr><td><code>lchat-home.service</code></td><td>The home server: the anonymous, passkey-gated backup store (a separate binary from the same repository).</td><td><code>/opt/lchat/home --addr /run/lchat/home.sock --db /data/home.db</code></td><td><code>lchat</code></td></tr> |
| <tr><td><code>lchat-push.service</code></td><td>The push relay: the central Apple push-notification signer (separate binary).</td><td><code>/opt/lchat/push --addr /run/lchat/push.sock --db /data/push.db</code></td><td><code>lchat</code></td></tr> |
| - <tr><td><code>lchat-links.service</code></td><td>The <code>11ch.at</code> short-link redirector for message permalinks (issue #905) — unlike home/push, a subcommand of the main <code>lchat</code> binary, not a separate one; holds no database (an in-memory, TTL-bounded verification cache only). <span class="status partial">built, not yet deployed</span></td><td><code>/opt/lchat/lchat links run --socket /run/lchat/links.sock</code></td><td><code>lchat</code></td></tr> |
| + <tr><td><code>lchat-links.service</code></td><td>The <code>11ch.at</code> short-link redirector for message permalinks (issue #905) — unlike home/push, a subcommand of the main <code>lchat</code> binary, not a separate one; holds no database (an in-memory, TTL-bounded verification cache only). Runs on the <strong>one</strong> box declared the short-link apex (<code>bootstrap --links-apex</code>); written and disabled everywhere else. <span class="status partial">built, not yet deployed</span></td><td><code>/opt/lchat/lchat links run --socket /run/lchat/links.sock</code></td><td><code>lchat</code></td></tr> |
| <tr><td><code>lchat-control.service</code></td><td>The control daemon: the root-only local API that creates and deletes instances. See <a href="spaces.html#controld">part 05</a>.</td><td><code>/opt/lchat/lchat controld</code></td><td>root (deliberately)</td></tr> |
| <tr><td><code>lchat-cloud.service</code></td><td>The cloud provisioner: the public, unprivileged self-serve space-creation service. See <a href="spaces.html#cloud">part 05</a>.</td><td><code>/opt/lchat/lchat cloud --socket /run/lchat/cloud.sock --db /data/cloud.db</code></td><td><code>lchat</code></td></tr> |
| <tr><td><code>lchat-update.timer</code> + <code>.service</code></td><td>The hourly auto-updater (a timer that fires a one-shot pass). See <a href="deploys.html#auto-updates">part 04</a>.</td><td><code>/opt/lchat/lchat update run</code></td><td>root (deliberately)</td></tr> |
| @@ -152,9 +152,9 @@ curl -s --unix-socket /run/lchat/chat-a.example.com.sock http://localhost/api/ve |
| <ul> |
| <li><strong>The home server</strong> (<code>lchat-home.service</code>) is the account-recovery backup store: members’ encrypted key backups, gated by passkeys. Its one operational constraint is that <strong>its hostname must never change</strong> — the hostname is part of the passkey security model (the WebAuthn “relying party” origin), and changing it orphans every passkey. Its environment file <code>/data/home.env</code> is <em>required</em> (no leading <code>-</code> in the unit) because a misconfigured home server is worse than a stopped one.</li> |
| <li><strong>The push relay</strong> (<code>lchat-push.service</code>) holds the one Apple push-notification signing key for the whole fleet, so that individual instances never do. Instances authenticate to it with per-instance revocable tokens. Its environment file <code>/data/push.env</code> is likewise required.</li> |
| - <li><strong>The links redirector</strong> (<code>lchat-links.service</code>, <span class="status partial">built, not yet deployed</span>) fronts <code>11ch.at</code>, the shared short-domain for message permalinks (issue #905 — <code>linkscmd.go</code>): a bare request to the apex bounces to <code>elevenmessenger.com</code>, and <code>/l/<host></code> live-checks the target (an SSRF-guarded fetch of its own <code>/api/version</code>, cached in memory) before redirecting. Unlike home and push it is a <em>subcommand</em> of the main <code>lchat</code> binary rather than a separate one, and it holds no database at all — a restart just re-populates its verification cache from scratch, which is cheap because entries are looked up lazily.</li> |
| + <li><strong>The links redirector</strong> (<code>lchat-links.service</code>, <span class="status partial">built, not yet deployed</span>) fronts <code>11ch.at</code>, the shared short-domain for message permalinks (issue #905 — <code>linkscmd.go</code>): a bare request to the apex bounces to <code>elevenmessenger.com</code>, and <code>/l/<host></code> live-checks the target (an SSRF-guarded fetch of its own <code>/api/version</code>, cached in memory) before redirecting. Unlike home and push it is a <em>subcommand</em> of the main <code>lchat</code> binary rather than a separate one, and it holds no database at all — a restart just re-populates its verification cache from scratch, which is cheap because entries are looked up lazily. Exactly <em>one</em> box in a fleet is the short-link apex, and only that box runs it: <code>lchat bootstrap --links-apex <host></code> starts the redirector and registers the host as a route pointing at its socket. Everywhere else the unit is written and left disabled, so no box carries a process it has no use for. The route is then the source of truth — later bootstraps read the registry and converge to it, so the apex stays up across re-runs and every other box stays quiet.</li> |
| </ul> |
| -<p>All three are registered with the router by <code>lchat add-service <host> --socket <path></code>, which simply adds a <code>service</code> row to the registry after checking the backend answers. Deploying home or push is manual and rare: copy the new binary over the old one and <code>systemctl restart</code> the unit (<a href="deploys.html#sibling-deploys">part 04</a>); links, being a subcommand of the main binary, updates whenever <code>/opt/lchat/lchat</code> itself does.</p> |
| +<p>All three are registered with the router by <code>lchat add-service <host> --socket <path></code>, which simply adds a <code>service</code> row to the registry after checking the backend answers. Deploying home or push is manual and rare: copy the new binary over the old one and <code>systemctl restart</code> the unit (<a href="deploys.html#sibling-deploys">part 04</a>); links, being a subcommand of the main binary, picks up a new build whenever <code>/opt/lchat/lchat</code> is replaced <em>and its unit is restarted</em> — which both <code>lchat bootstrap</code> and the hourly auto-updater do for you (bootstrap restarts rather than <code>enable --now</code>s precisely so an update re-execs the new binary instead of leaving the old process running; the updater bounces it alongside <code>lchat-control</code> and <code>lchat-cloud</code>, the other units that exec the control binary directly).</p> |
| <h2 id="bot-sidecars">Bot sidecars: a member that runs on a server</h2> |
| <p>An instance can have a <strong>bot sidecar</strong>: a process hosting the built-in helper personas (Claude and Eleven). Architecturally, a bot is an ordinary end-to-end encrypted <em>member</em> of the chat whose “device” happens to be a server process: it holds its own private keys (in the session files under <code>/data/<host>-bot/</code>) and connects to its instance as a WebSocket <em>client</em>, over the instance’s public hostname, exactly like a browser would. It serves no inbound traffic at all — no socket, no port, no route.</p> |
diff --git a/docs/planet-scale-guide/reference.html b/docs/planet-scale-guide/reference.html| index 3845e8e..ee3a679 100644 |
| --- a/docs/planet-scale-guide/reference.html |
| +++ b/docs/planet-scale-guide/reference.html |
| @@ -28,7 +28,7 @@ |
| <h3 id="cmd-lifecycle">Box and instance lifecycle</h3> |
| <dl> |
| <dt><code>lchat bootstrap</code></dt> |
| - <dd>One-time box setup: writes every systemd unit file, creates the <code>lchat</code> system user and the <code>/data</code>, <code>/run/lchat</code>, and certificate-cache directories, mints the root-only control token, enables and starts the router and the hourly update timer. Idempotent — safe to re-run to refresh unit files. Explained in <a href="deploys.html#bootstrap">part 04</a>.</dd> |
| + <dd>One-time box setup: writes every systemd unit file, creates the <code>lchat</code> system user and the <code>/data</code>, <code>/run/lchat</code>, and certificate-cache directories, mints the root-only control token, and enables and starts the router and the hourly update timer. Idempotent — safe to re-run to refresh unit files, and re-running is the <em>only</em> thing that installs a newly-added unit on an existing box (<code>deploy</code> restarts only units that already exist). <code>--links-apex <host></code> additionally makes this box the <a href="#cmd-links">short-link apex</a>. Explained in <a href="deploys.html#bootstrap">part 04</a>.</dd> |
| <dt><code>lchat add-instance <host> [--admin-token <tok>] [--hibernate] [--home-region <cc>] [--app-store-url <url>] [--home-origin <url>] [--push-relay-url <url>] [--push-relay-token <tok>]</code></dt> |
| <dd>Creates a space: mints (or, with <code>--admin-token</code>, imports) the admin token, writes <code>/data/<host>.env</code>, registers the route, points the version link, starts and health-checks the unit (skipped with <code>--hibernate</code>), reloads the router, warms the certificate, and prints the one-time setup link and the admin link. Explained in <a href="spaces.html#adding-manual">part 05</a>.</dd> |
| <dt><code>lchat add-service <host> --socket <path></code></dt> |
| @@ -77,6 +77,11 @@ |
| <dd>The root-only control daemon — run by <code>lchat-control.service</code>. Flags/env for socket, token file, and capacity policy are under <a href="#env-control">environment variables</a>. <a href="spaces.html#controld">Part 05</a>.</dd> |
| <dt><code>lchat cloud --socket <path> --db <path></code></dt> |
| <dd>The self-serve provisioner — run by <code>lchat-cloud.service</code>. <a href="spaces.html#cloud">Part 05</a>.</dd> |
| + <dt id="cmd-links"><code>lchat links run --socket <path></code></dt> |
| + <dd>The short-link redirector behind <code>11ch.at</code> (message permalinks, issue #905) — run by <code>lchat-links.service</code>. Exactly <strong>one box in a fleet is the short-link apex</strong>, and only that box runs the redirector; everywhere else the unit is written and left disabled. You declare the apex with <code>lchat bootstrap --links-apex <host></code>, which starts the redirector and registers <code><host></code> as a <code>service</code> route pointing at <code>/run/lchat/links.sock</code>. |
| + <p>That route — not the flag — is then the source of truth: later plain <code>bootstrap</code> runs read the registry and converge to it, so re-running bootstrap on the apex box keeps short links up, and running it on any other box keeps the redirector off. (A flag-keyed design would silently switch the apex off the first time someone re-bootstrapped it without remembering the flag.)</p> |
| + <p>Note the apex needs its own DNS A record and certificate: a <code>*.example.com</code> wildcard does <em>not</em> cover bare <code>example.com</code>. The router issues the certificate automatically once the route exists — an unrouted host gets no certificate at all, which is how a missing route surfaces: a TLS handshake failure rather than a 404.</p> |
| + <p><strong>Ours to serve, in practice.</strong> Two values in <code>linkscmd.go</code> are Eleven-specific and not configurable: the apex bounces to <code>https://elevenmessenger.com</code>, and the <code>apple-app-site-association</code> advertises our iOS app id. Pointing your own apex here would send your visitors to our marketing site and publish our app id under your domain. Making those configurable is what self-hostable short links would take.</p></dd> |
| <dt><code>lchat bot run [--base <url>]</code> / <code>lchat bot provision</code></dt> |
| <dd>Run a bot sidecar (with no arguments, configuration comes from the unit’s environment file) / claim an agent invite into a session file. <a href="processes.html#bot-sidecars">Part 03</a>.</dd> |
| <dt><code>lchat release keygen|sign|verify|payload</code></dt> |
diff --git a/updater.go b/updater.go| index 455f107..352e361 100644 |
| --- a/updater.go |
| +++ b/updater.go |
| @@ -768,20 +768,37 @@ func rollbackControl(self, prev, newVer string, healthErr error) error { |
| return fmt.Errorf("control binary %s failed its health check and was rolled back to %s: %v", newVer, buildVersion, healthErr) |
| } |
| -// controlUnit / cloudUnit are the systemd units that exec the control binary. |
| +// controlUnit / cloudUnit / linksUnit are the systemd units that exec the |
| +// control binary (/opt/lchat/lchat) directly, rather than a per-instance |
| +// versioned live symlink — so they keep running the OLD inode until something |
| +// restarts them. |
| const ( |
| controlUnit = "lchat-control.service" |
| cloudUnit = "lchat-cloud.service" |
| + linksUnit = "lchat-links.service" |
| ) |
| +// controlBinaryUnits is the one list of them, so the updater and the tests that |
| +// assert "one restart each, no rollback round" can't disagree about how many |
| +// that is. Adding a unit here is all it takes to put it on the update path. |
| +var controlBinaryUnits = []string{controlUnit, cloudUnit, linksUnit} |
| + |
| // restartControlUnits bounces the backend units that exec the control binary — |
| -// lchat-control and lchat-cloud — onto whatever /opt/lchat/lchat now is. The |
| -// ROUTER is deliberately excluded: it's the fleet's only ingress, has no |
| -// /healthz to probe, and must never blip on the updater's schedule (it keeps |
| -// its old inode until an operator restarts it). Restart failures only log — the |
| -// health probe that follows is the real gate on whether the new build is up. |
| +// lchat-control, lchat-cloud, and lchat-links — onto whatever /opt/lchat/lchat |
| +// now is. The ROUTER is deliberately excluded: it's the fleet's only ingress, |
| +// has no /healthz to probe, and must never blip on the updater's schedule (it |
| +// keeps its old inode until an operator restarts it). Restart failures only |
| +// log — the health probe that follows is the real gate on whether the new |
| +// build is up. |
| +// |
| +// links belongs here for the same reason cloud does, and is the cheapest of |
| +// the three to bounce: it holds no database and no connections, just an |
| +// in-memory verification cache that re-populates lazily. Without it the |
| +// redirector would be the one service that never picks up a new build at all — |
| +// try-restart is a no-op on a box where it isn't running, so boxes that route |
| +// no short links are unaffected. |
| func restartControlUnits() { |
| - for _, unit := range []string{controlUnit, cloudUnit} { |
| + for _, unit := range controlBinaryUnits { |
| if err := systemctl("try-restart", unit); err != nil { |
| fmt.Printf("update: %s restart: %v (new build applies on its next start)\n", unit, err) |
| } |