eleven / ocho Public

deploy: harden the systemd units (non-root, sandbox, resource limits) (#186)

Every unit now runs as a locked system user `lchat` instead of root:
bootstrap creates the user and owns /data + /run/lchat to it. Only the
router keeps CAP_NET_BIND_SERVICE (to bind :80/:443 unprivileged); the
rest bind unix sockets and hold no capabilities. All units get the usual
systemd sandbox (ProtectSystem=strict + ReadWritePaths, NoNewPrivileges,
ProtectHome/Kernel*/ControlGroups, RestrictAddressFamilies,
SystemCallFilter=@system-service) and a MemoryMax cap.

The Node sidecar is the one exception to MemoryDenyWriteExecute — V8's
JIT needs W+X memory, so only the Go units set it. Documented, with the
OS-level steps (firewall, key-only SSH, unattended-upgrades), in a new
Hardening section of docs/instance-router.md.

systemd-analyze security drops each unit from ~9.6 (UNSAFE) to ~1.5 (OK).

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Paul Campbell 4 months ago 9497037ebbb323e6ec4005757133037853e00563 parent 1b0a161

7 files changed, +231 −3

bootstrap.go +38 −3
deploy/systemd/lchat-bot@.service +34 −0
deploy/systemd/lchat-home.service +29 −0
deploy/systemd/lchat-push.service +29 −0
deploy/systemd/lchat-router.service +34 −0
deploy/systemd/lchat@.service +30 −0
docs/instance-router.md +37 −0

diff --git a/bootstrap.go b/bootstrap.go

index 628ffc3..e155f5b 100644

--- a/bootstrap.go

+++ b/bootstrap.go

@@ -5,7 +5,10 @@ import (

"flag"

"fmt"

"os"

+ "os/exec"

+ "os/user"

"path/filepath"

+ "strings"

)

// The systemd units ship inside the binary (source of truth: deploy/systemd), so

@@ -30,9 +33,37 @@ var unitPush string

//go:embed deploy/systemd/lchat-home.service

var unitHome string

+// serviceUser is the unprivileged account every unit runs as (see the hardened

+// units in deploy/systemd). bootstrap creates it and hands it the data + runtime

+// dirs; the router keeps CAP_NET_BIND_SERVICE to still bind :80/:443.

+const serviceUser = "lchat"

// tmpfilesConf keeps /run/lchat across reboots (/run is tmpfs); bootstrap also

-// creates it now.

-const tmpfilesConf = "d /run/lchat 0710 root root -\n"

+// creates it now. Owned by the service user so the non-root units can bind their

+// sockets there.

+const tmpfilesConf = "d /run/lchat 0750 " + serviceUser + " " + serviceUser + " -\n"

+// ensureServiceUser creates the locked, home-less system user the units run as,

+// idempotently.

+func ensureServiceUser() {

+ if _, err := user.Lookup(serviceUser); err == nil {

+ return

+ }

+ out, err := exec.Command("useradd", "--system", "--user-group",

+ "--no-create-home", "--shell", "/usr/sbin/nologin", serviceUser).CombinedOutput()

+ if err != nil {

+ die("bootstrap: create user %s: %v: %s", serviceUser, err, strings.TrimSpace(string(out)))

+ }

+// chownToService recursively gives a path to the service user (best-effort exec of

+// chown -R, so it also fixes ownership of any data already present).

+func chownToService(path string) {

+ out, err := exec.Command("chown", "-R", serviceUser+":"+serviceUser, path).CombinedOutput()

+ if err != nil {

+ die("bootstrap: chown %s: %v: %s", path, err, strings.TrimSpace(string(out)))

+ }

// runBootstrap installs the systemd units + runtime dir and starts the router, so

// provisioning a fresh box is one command after the binary is copied to

@@ -44,14 +75,18 @@ func runBootstrap(args []string) {

noStart := fs.Bool("no-start", false, "install units but don't enable/start the router")

_ = fs.Parse(args)

+ ensureServiceUser()

for _, d := range []string{*dataDir, filepath.Join(*dataDir, "acme"), "/run/lchat"} {

if err := os.MkdirAll(d, 0o750); err != nil {

die("bootstrap: mkdir %s: %v", d, err)

}

- if err := os.Chmod("/run/lchat", 0o710); err != nil {

+ if err := os.Chmod("/run/lchat", 0o750); err != nil {

die("bootstrap: chmod /run/lchat: %v", err)

}

+ // The units run as serviceUser, so it must own where they read/write.

+ chownToService(*dataDir)

+ chownToService("/run/lchat")

write := func(path, content string) {

if err := os.WriteFile(path, []byte(content), 0o644); err != nil {

diff --git a/deploy/systemd/lchat-bot@.service b/deploy/systemd/lchat-bot@.service

index 4d826c7..fa35a3c 100644

--- a/deploy/systemd/lchat-bot@.service

+++ b/deploy/systemd/lchat-bot@.service

@@ -15,5 +15,39 @@ ExecStart=/usr/bin/node /opt/lchat/agent/agent.mjs

Restart=always

RestartSec=2

+# --- hardening (see docs/instance-router.md § Hardening) ---

+# Pure outbound WebSocket client: no listening socket, no capabilities.

+User=lchat

+Group=lchat

+CapabilityBoundingSet=

+NoNewPrivileges=yes

+ProtectSystem=strict

+ProtectHome=yes

+# Writes only its persisted agent sessions under /data/%i-bot/. HOME lands in the

+# private /tmp so nothing reaches for a real home dir.

+ReadWritePaths=/data

+Environment=HOME=/tmp

+PrivateTmp=yes

+PrivateDevices=yes

+ProtectClock=yes

+ProtectHostname=yes

+ProtectKernelTunables=yes

+ProtectKernelModules=yes

+ProtectKernelLogs=yes

+ProtectControlGroups=yes

+ProtectProc=invisible

+RestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX

+RestrictNamespaces=yes

+RestrictRealtime=yes

+RestrictSUIDSGID=yes

+LockPersonality=yes

+# NOTE: no MemoryDenyWriteExecute — V8's JIT maps writable+executable memory, so

+# enabling it (as the Go units do) would crash Node on boot.

+SystemCallFilter=@system-service

+SystemCallErrorNumber=EPERM

+SystemCallArchitectures=native

+MemoryMax=512M

+TasksMax=512

[Install]

WantedBy=multi-user.target

diff --git a/deploy/systemd/lchat-home.service b/deploy/systemd/lchat-home.service

index 1d1845e..c437e99 100644

--- a/deploy/systemd/lchat-home.service

+++ b/deploy/systemd/lchat-home.service

@@ -11,5 +11,34 @@ ExecStart=/opt/lchat/home --addr /run/lchat/home.sock --db /data/home.db

Restart=always

RestartSec=1

+# --- hardening (see docs/instance-router.md § Hardening) ---

+User=lchat

+Group=lchat

+CapabilityBoundingSet=

+NoNewPrivileges=yes

+ProtectSystem=strict

+ProtectHome=yes

+ReadWritePaths=/data /run/lchat

+PrivateTmp=yes

+PrivateDevices=yes

+ProtectClock=yes

+ProtectHostname=yes

+ProtectKernelTunables=yes

+ProtectKernelModules=yes

+ProtectKernelLogs=yes

+ProtectControlGroups=yes

+ProtectProc=invisible

+RestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX

+RestrictNamespaces=yes

+RestrictRealtime=yes

+RestrictSUIDSGID=yes

+LockPersonality=yes

+MemoryDenyWriteExecute=yes

+SystemCallFilter=@system-service

+SystemCallErrorNumber=EPERM

+SystemCallArchitectures=native

+MemoryMax=256M

+TasksMax=256

[Install]

WantedBy=multi-user.target

diff --git a/deploy/systemd/lchat-push.service b/deploy/systemd/lchat-push.service

index 04aec21..a2c998c 100644

--- a/deploy/systemd/lchat-push.service

+++ b/deploy/systemd/lchat-push.service

@@ -12,5 +12,34 @@ ExecStart=/opt/lchat/push --addr /run/lchat/push.sock --db /data/push.db

Restart=always

RestartSec=1

+# --- hardening (see docs/instance-router.md § Hardening) ---

+User=lchat

+Group=lchat

+CapabilityBoundingSet=

+NoNewPrivileges=yes

+ProtectSystem=strict

+ProtectHome=yes

+ReadWritePaths=/data /run/lchat

+PrivateTmp=yes

+PrivateDevices=yes

+ProtectClock=yes

+ProtectHostname=yes

+ProtectKernelTunables=yes

+ProtectKernelModules=yes

+ProtectKernelLogs=yes

+ProtectControlGroups=yes

+ProtectProc=invisible

+RestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX

+RestrictNamespaces=yes

+RestrictRealtime=yes

+RestrictSUIDSGID=yes

+LockPersonality=yes

+MemoryDenyWriteExecute=yes

+SystemCallFilter=@system-service

+SystemCallErrorNumber=EPERM

+SystemCallArchitectures=native

+MemoryMax=256M

+TasksMax=256

[Install]

WantedBy=multi-user.target

diff --git a/deploy/systemd/lchat-router.service b/deploy/systemd/lchat-router.service

index c8c5c28..6943e07 100644

--- a/deploy/systemd/lchat-router.service

+++ b/deploy/systemd/lchat-router.service

@@ -10,5 +10,39 @@ ExecReload=/bin/kill -HUP $MAINPID

Restart=always

RestartSec=1

+# --- hardening (see docs/instance-router.md § Hardening) ---

+User=lchat

+Group=lchat

+# The router is the ONLY unit that binds privileged ports (:80 + :443); this cap

+# lets it do so as a non-root user. Every other unit binds a unix socket only.

+AmbientCapabilities=CAP_NET_BIND_SERVICE

+CapabilityBoundingSet=CAP_NET_BIND_SERVICE

+NoNewPrivileges=yes

+ProtectSystem=strict

+ProtectHome=yes

+ReadWritePaths=/data /run/lchat

+PrivateTmp=yes

+PrivateDevices=yes

+ProtectClock=yes

+ProtectHostname=yes

+ProtectKernelTunables=yes

+ProtectKernelModules=yes

+ProtectKernelLogs=yes

+ProtectControlGroups=yes

+ProtectProc=invisible

+RestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX

+RestrictNamespaces=yes

+RestrictRealtime=yes

+RestrictSUIDSGID=yes

+LockPersonality=yes

+# The Go binary never maps writable+executable memory, so this is safe (unlike the

+# Node sidecar, whose JIT needs it — see lchat-bot@.service).

+MemoryDenyWriteExecute=yes

+SystemCallFilter=@system-service

+SystemCallErrorNumber=EPERM

+SystemCallArchitectures=native

+MemoryMax=256M

+TasksMax=256

[Install]

WantedBy=multi-user.target

diff --git a/deploy/systemd/lchat@.service b/deploy/systemd/lchat@.service

index 068d5a0..87135d9 100644

--- a/deploy/systemd/lchat@.service

+++ b/deploy/systemd/lchat@.service

@@ -11,5 +11,35 @@ ExecStart=/opt/lchat/lchat --socket /run/lchat/%i.sock --db /data/%i.db

Restart=always

RestartSec=1

+# --- hardening (see docs/instance-router.md § Hardening) ---

+User=lchat

+Group=lchat

+# Binds a unix socket only — no capabilities needed.

+CapabilityBoundingSet=

+NoNewPrivileges=yes

+ProtectSystem=strict

+ProtectHome=yes

+ReadWritePaths=/data /run/lchat

+PrivateTmp=yes

+PrivateDevices=yes

+ProtectClock=yes

+ProtectHostname=yes

+ProtectKernelTunables=yes

+ProtectKernelModules=yes

+ProtectKernelLogs=yes

+ProtectControlGroups=yes

+ProtectProc=invisible

+RestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX

+RestrictNamespaces=yes

+RestrictRealtime=yes

+RestrictSUIDSGID=yes

+LockPersonality=yes

+MemoryDenyWriteExecute=yes

+SystemCallFilter=@system-service

+SystemCallErrorNumber=EPERM

+SystemCallArchitectures=native

+MemoryMax=512M

+TasksMax=256

[Install]

WantedBy=multi-user.target

diff --git a/docs/instance-router.md b/docs/instance-router.md

index 475536a..6959b7a 100644

--- a/docs/instance-router.md

+++ b/docs/instance-router.md

@@ -337,6 +337,43 @@ zero-downtime via `SO_REUSEPORT` / systemd socket activation.

- `lchat-bot@.service` — the optional per-instance sidecar (Node; see *Bots*).

`Restart=always`; a deploy is just a restart (it backfills on reconnect).

+## Hardening

+Everything on one box means the box is the blast radius, so treat it as the

+security boundary. The shipped units and `bootstrap` already apply the service-level

+items below; the OS-level ones are a few one-time commands.

+**Run unprivileged.** `bootstrap` creates a locked system user (`lchat`) and owns

+`/data` + `/run/lchat` to it; every unit runs as that user. Only the router touches

+privileged ports, so it alone keeps `CAP_NET_BIND_SERVICE` (`AmbientCapabilities` +

+a matching `CapabilityBoundingSet`) to bind `:80`/`:443` as non-root; everything

+else binds a unix socket and needs no capabilities (`CapabilityBoundingSet=`).

+**Sandbox each unit.** The units set the usual systemd confinement:

+`NoNewPrivileges`, `ProtectSystem=strict` with `ReadWritePaths=/data /run/lchat`

+(the whole FS is read-only otherwise), `ProtectHome`, `PrivateTmp`,

+`PrivateDevices`, `ProtectKernel*`, `ProtectControlGroups`, `ProtectProc=invisible`,

+`RestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX`, `RestrictNamespaces`,

+`SystemCallFilter=@system-service`, and `MemoryMax`. `systemd-analyze security

+<unit>` should score each in the low single digits.

+> **Go vs Node caveat.** The Go units add `MemoryDenyWriteExecute=yes` (the static

+> binary never maps W+X memory). The Node **sidecar must not** set it — V8's JIT

+> maps writable+executable pages, so it would crash on boot. That's the one

+> directive that differs between the two.

+**OS level** (one-time):

+- **Firewall** — allow only `22`, `80`, `443` inbound, default-deny the rest

+ (`ufw allow 22/tcp && ufw allow 80/tcp && ufw allow 443/tcp && ufw enable`; add

+ the `allow`s *before* enabling so you don't lock out SSH). Instances and services

+ talk over unix sockets, so nothing else needs a public port.

+- **SSH** — key-only: `PasswordAuthentication no`, `PermitRootLogin

+ prohibit-password` (or a sudo user + `PermitRootLogin no`).

+- **Automatic security updates** — enable `unattended-upgrades`.

+- If the box hosts its own services, pin their hostnames to loopback in

+ `/etc/hosts` (see *Bots*).

## Backups

Collapsing onto one box means every instance's data is on one disk — a single