amadan

carlos / platform Public

isolated exec: CPU/IO shares, a disk floor, own-build /opt, an egress fence and private IPC for nested instances (#214 items 1-3, #178)

The nested layout isolates files, pids and cgroupfs. Four things were
still shared between tenants on a box: CPU, IO and the data disk; every
app's build under /opt/carlos/versions; the box's loopback and private
networks; and POSIX mqueues and abstract unix sockets.

- Item 1: the control service writes CPUWeight=, CPUQuota= and
  IOWeight= next to the memory caps on each host slice. The values come
  from the same tier (cgroup.SharesOf: 100/100%/100, children
  100/200%/100), so nothing new goes over the wire. A per-tenant disk
  quota is not possible here: /data is a directory on the mounted root
  ext4, and project quotas need an offline tune2fs. Usage also cannot
  be attributed reliably: unlinked-but-open files appear in no
  directory, and a stop from the control service bypasses the
  activator's fence. So the guard refuses rather than stops: below
  max(1 GiB, 10%) free, starts and seeds answer 507, and a warning is
  logged every 30s.
- Item 2: the namespace replaces /opt/carlos with an empty tmpfs and
  binds back only /opt/carlos/carlos. Each instance gets its own build
  at /opt/carlos/live/<host> through links the control service writes
  under /run/carlos-control/exe/<iid> before every start. carlos-app@
  binds /opt/carlos/live/%i over an empty /opt/carlos.
- Item 3: IPAddressDeny= now covers loopback, RFC 1918, CGNAT,
  link-local, ULA, NAT64, site-local, multicast and reserved ranges.
  An allow is per address, both ways and on every port, so no loopback
  resolver is ever opened, resolved's 127.0.0.53/54 stubs included.
  Instead the namespace binds resolved's upstream list over
  /etc/resolv.conf. Before every probe run, the control service
  re-opens the nameservers in that same file on carlos-instances.slice.
  Link-local resolvers are refused except 169.254.169.253, since an
  allow carries no zone.
- #178: PrivateIPC=yes and a private /dev/mqueue. Before exec, the
  trampoline scopes abstract unix sockets with Landlock (ABI 6), and it
  refuses to exec if it cannot apply the scope.

The probe gains a line for each property: opt-own-build, dns,
resolver-ports, loopback-v4/v6, private-v4/v6, ipc-private,
abstract-unix, cpu-limits and io-weight, so ready reflects them. The
unit tests pin every new directive. The leftover gaps are written up in
spec §11.1.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Paul Campbell pushed by claude 7b7571bd28c4e0f3bcab8a18530f06e3997ab144 parent 01f890c
23 files changed, +1716 −38
  • docs/superpowers/specs/2026-09-11-isolated-exec-design.md +61 −0
  • internal/cgroup/cgroup.go +29 −0
  • internal/host/units/carlos-app@.service +6 −0
  • internal/host/units/carlos-instance-probe.service +54 −1
  • internal/host/units/carlos-instance-seed@.service +48 −1
  • internal/host/units/carlos-instance-sidecar@.service +48 −1
  • internal/host/units/carlos-instance@.service +48 −1
  • internal/host/units_hardening_test.go +148 −0
  • internal/host/units_isolation_test.go +1 −1
  • internal/instancectl/hardening.go +342 −0
  • internal/instancectl/hardening_test.go +185 −0
  • internal/instancectl/peer_linux.go +11 −0
  • internal/instancectl/peer_other.go +1 −0
  • internal/instancectl/server.go +56 −7
  • internal/instancectl/server_test.go +2 −2
  • internal/instanceexec/exec.go +2 −2
  • internal/instanceexec/probe.go +60 −2
  • internal/instanceexec/probe_hardening.go +193 −0
  • internal/instanceexec/probe_hardening_test.go +221 −0
  • internal/instanceexec/probe_linux.go +1 −1
  • internal/instanceexec/probe_test.go +40 −19
  • internal/instanceexec/scope_linux.go +146 −0
  • internal/instanceexec/scope_other.go +13 −0

CI passed — run details

diff --git a/docs/superpowers/specs/2026-09-11-isolated-exec-design.md b/docs/superpowers/specs/2026-09-11-isolated-exec-design.md
index 95586eeda..cebaa53b7 100644
--- a/docs/superpowers/specs/2026-09-11-isolated-exec-design.md
+++ b/docs/superpowers/specs/2026-09-11-isolated-exec-design.md
@@ -2012,6 +2012,67 @@ Box level, in the plan's verification task and in the rollout:
- **Discussion #1 (memory pressure)**: no new numbers here (§4.3); the
slice shape makes a later change a unit-file edit, never a drop-in.
+### 11.1 Hardening after the layout (#214 items 1-3, #178), and what it leaves
+
+Added to the templates, the control service and the probe (each property
+has a probe line, so `ready` reflects it):
+
+- **CPU and IO.** The control service writes the tier's `CPUWeight=`,
+ `CPUQuota=` and `IOWeight=` beside its memory caps on the host slice
+ (`cgroup.SharesOf`: 100/100%/100, children tier 100/200%/100). Nothing
+ new on the wire: the shares follow from the memory tier.
+ *Residual:* `IOWeight=` is enforced only by io.cost or BFQ; the probe
+ says `unverified` on a box with neither, which is the default on an
+ EBS `none` scheduler. Enabling io.cost is a box change, not made here.
+- **Disk.** No per-tenant quota: `/data` is a directory on the root ext4
+ filesystem, and project quotas need `tune2fs -O project,quota` on an
+ unmounted filesystem plus a `prjquota` mount; every writer is uid
+ carlos, so reserved blocks cannot tell a tenant from the agent. The
+ guard refuses rather than stops: below max(1 GiB, 10%) free, starts
+ and seeds answer 507, and the control service logs a warning every
+ 30s while it lasts. Stopping the "biggest" tenant was tried and
+ dropped in review: space held by unlinked-but-open files appears in
+ no directory, so attribution can be gamed into stopping honest
+ tenants, and a stop from the control service bypasses the activator's
+ fence and holds. *Residual:* tenants already awake can keep writing
+ past the floor until the disk is full. A dedicated `/data` volume
+ with project quotas (or systemd's quota directives on top of them)
+ is the real fix.
+- **The version store.** Each instance sees an empty `/opt/carlos` with
+ `/opt/carlos/carlos` and its own build put back: the control service
+ writes `/run/carlos-control/exe/<iid>/{app -> /opt/carlos/live/<host>,
+ live/<host> -> ../app}` before every start, and the unit binds them at
+ `/opt/carlos/app` and `/opt/carlos/live`. `carlos-app@` gets the same
+ view by binding `/opt/carlos/live/%i` over an empty `/opt/carlos`.
+- **Egress.** `IPAddressDeny=` covers loopback, RFC 1918, CGNAT,
+ link-local, ULA, NAT64, site-local, multicast and reserved ranges.
+ An allow is per address, both ways and on every port, so no loopback
+ resolver is ever opened. That includes resolved's 127.0.0.53/54
+ stubs, because a tenant could bind the stub address and reach every
+ wildcard listener through it. The namespace binds
+ `/run/systemd/resolve/resolv.conf` (resolved's upstreams, when
+ present) over `/etc/resolv.conf`. Before every probe, the control
+ service re-opens the nameservers in that same file with
+ `IPAddressAllow=` on `carlos-instances.slice`. Link-local resolvers
+ are refused except 169.254.169.253: an allow carries no zone, so
+ `fe80::1` would open that address on every link. The probe's `dns`
+ and `resolver-ports` lines check that each allowed resolver is
+ reachable, is not an address of the box, and does not answer on
+ port 22. *Residuals:* an allowed upstream is open on every port (it
+ is a remote resolver, not the box), and a resolver change reaches
+ running instances only at their next start. Loopback inside one instance is
+ closed too, so a family that talks to itself over TCP loopback must
+ use a unix socket in its own directory. Services the box exposes on a
+ public interface address remain reachable, as from the internet.
+- **IPC (#178).** `PrivateIPC=yes` and a private `/dev/mqueue`. Abstract
+ unix sockets are scoped with Landlock (`LANDLOCK_SCOPE_ABSTRACT_UNIX_SOCKET`,
+ ABI 6, Linux 6.12) by the trampoline before it execs; a kernel without
+ it fails the probe and the trampoline refuses to exec. *Residual:* a
+ sibling can still bind an abstract name first, which is a refusal of
+ service to the owner, never a connection to it. A per-unit network
+ namespace would close that too, at the cost of a veth or proxy for
+ egress; not done.
+
---
## 12. Approaches considered
diff --git a/internal/cgroup/cgroup.go b/internal/cgroup/cgroup.go
index 912208afc..e040cbb4a 100644
--- a/internal/cgroup/cgroup.go
+++ b/internal/cgroup/cgroup.go
@@ -61,6 +61,35 @@ var (
TierChildren = Limit{MemoryMax: 1 << 30}
)
+// Shares is the CPU and IO half of a tier (#214 item 1). Memory alone let
+// one tenant take every core, or the disk's whole queue, from every other
+// tenant on the box. Like the memory caps these are PLATFORM-chosen: the
+// tier a route gets is decided by its manifest's one bit (children), and
+// its shares follow from the tier, so nothing new travels on the wire and
+// no number comes from a manifest.
+//
+// The weights are the kernel's default, written explicitly: a weight is
+// what makes the per-host slices take turns under contention, and setting
+// one is also what turns the controller on for the subtree. The quota is
+// the hard ceiling, in percent of one CPU. IOWeight is honoured only where
+// the disk has a weight-aware controller (io.cost or BFQ); the probe
+// reports whether this box has one.
+type Shares struct {
+ CPUWeight int
+ CPUQuotaPercent int
+ IOWeight int
+}
+
+// SharesOf is the tier's shares. A forking family gets two CPUs' worth of
+// quota, as it gets twice the memory; anything else, including a limit
+// that is not a tier, gets one: the smaller, and so the safe direction.
+func SharesOf(l Limit) Shares {
+ if l == TierChildren {
+ return Shares{CPUWeight: 100, CPUQuotaPercent: 200, IOWeight: 100}
+ }
+ return Shares{CPUWeight: 100, CPUQuotaPercent: 100, IOWeight: 100}
+}
+
// HighOf is the memory.high for a memory.max: 90 percent, computed as the
// cgroup manager always has, so a unit tier and an exec-child tier are the
// same number.
diff --git a/internal/host/units/carlos-app@.service b/internal/host/units/carlos-app@.service
index d8509d698..3b79b44bf 100644
--- a/internal/host/units/carlos-app@.service
+++ b/internal/host/units/carlos-app@.service
@@ -40,6 +40,12 @@ After=network-online.target
DynamicUser=yes
StateDirectory=carlos-app/%i
ExecStart=/opt/carlos/live/%i serve
+# The version store holds every app's build on the box (#214 item 2). This
+# unit sees an empty /opt/carlos with its own build put back at the path
+# ExecStart= names: PID 1 chases the live link when it binds it, so a
+# repoint takes effect at the next start, as it always has.
+TemporaryFileSystem=/opt/carlos:ro
+BindReadOnlyPaths=/opt/carlos/live/%i
EnvironmentFile=-/etc/carlos/apps/%i.env
Restart=always
NoNewPrivileges=yes
diff --git a/internal/host/units/carlos-instance-probe.service b/internal/host/units/carlos-instance-probe.service
index c45b75f53..8a31572d5 100644
--- a/internal/host/units/carlos-instance-probe.service
+++ b/internal/host/units/carlos-instance-probe.service
@@ -23,6 +23,12 @@ TimeoutStartSec=30
# The instance's own two directories, the only writable data paths.
BindPaths=/data/instances/_probe /run/carlos/instances/_probe
ReadWritePaths=/data/instances/_probe /run/carlos/instances/_probe
+# The same two binds the instance templates make (#214 item 2), for the
+# name _probe: /opt/carlos/live/_probe is a link to ../app, and
+# /opt/carlos/app is whatever the source link points at, here the
+# platform binary standing in for a build. The control service writes
+# both sources before every probe run.
+BindReadOnlyPaths=/run/carlos-control/exe/_probe/app:/opt/carlos/app /run/carlos-control/exe/_probe/live:/opt/carlos/live
# --- the namespace (titogo@.service, 2026-07-31 design, on systemd 257) ---
ProtectSystem=strict
@@ -33,6 +39,13 @@ TemporaryFileSystem=/run/carlos:ro
# /var/lib as a class. An instance needs nothing from either.
TemporaryFileSystem=/var/cache:ro
TemporaryFileSystem=/var/lib:ro
+# The version store holds every app's build on the box, world-readable so
+# a unit can exec its own (#214 item 2). An instance sees none of it: an
+# empty /opt/carlos with the platform binary that ExecStart= and
+# ExecStopPost= run put back, and, outside this block because it is per
+# instance, its own build at /opt/carlos/live/<host>.
+TemporaryFileSystem=/opt/carlos:ro
+BindReadOnlyPaths=/opt/carlos/carlos
# Root-written credentials a carlos-GROUP process may read live here:
# /etc/carlos/relay-masters is root:carlos 0640 for carlos-relay.service,
# and an instance's Group=carlos would read it too (codex R10-1). An
@@ -50,6 +63,15 @@ PrivateDevices=yes
# and every instance is uid carlos, so a sibling's shared memory would be
# one open away. Each unit gets its own, charged to its own cgroup.
TemporaryFileSystem=/dev/shm:mode=1777,nosuid,nodev
+# POSIX message queues and SysV IPC belong to an IPC namespace, and every
+# instance is uid carlos, so a sibling's queue would be one mq_open away
+# (#178). PrivateIPC= gives each unit its own namespace, and mq_open()
+# goes through that namespace's own internal mount. PrivateDevices=
+# carries the host's /dev/mqueue into the private /dev the way it carries
+# /dev/shm, and that mount would still list and open the host namespace's
+# queues, so it is replaced the same way.
+PrivateIPC=yes
+TemporaryFileSystem=/dev/mqueue:ro
# /sys/fs/cgroup read-only: the edge's delegated subtree is carlos-owned,
# and its cgroup.kill / cgroup.freeze would otherwise be one write away
# for a same-uid tenant (codex R9-1).
@@ -61,7 +83,32 @@ RestrictNamespaces=yes
NoNewPrivileges=yes
CapabilityBoundingSet=
RestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX
-IPAddressDeny=169.254.169.254/32 fd00:ec2::254/128
+# Abstract unix sockets live in the network namespace, which instances
+# share with the box because they need outbound TCP, and the line above
+# cannot tell them from path sockets (#178). The trampoline scopes them
+# with Landlock before it execs the app (instanceexec.ScopeAbstractUnix):
+# an instance can connect only to abstract sockets its own processes
+# bound. A sibling can still bind a name first; that is a refusal of
+# service, never a connection.
+#
+# Egress (#214 item 3): the internet, and nothing on the box or private.
+# Loopback is where the console and other tenants' addr upstreams listen;
+# RFC 1918, ULA, link-local and CGNAT are the VPC, other boxes and the
+# cloud's internal services. IMDS stays named so the property is
+# greppable. A cgroup filter: it drops in both directions for every
+# socket the unit opens. The one thing an instance needs from these
+# ranges is the box's DNS resolver, so the control service re-opens
+# exactly the nameservers the instance sees with IPAddressAllow= on
+# carlos-instances.slice (an allow wins, and a slice's lists reach every
+# unit below it). The probe checks both halves.
+IPAddressDeny=localhost multicast 0.0.0.0/8 10.0.0.0/8 100.64.0.0/10 169.254.0.0/16 169.254.169.254/32 172.16.0.0/12 192.0.0.0/24 192.168.0.0/16 198.18.0.0/15 240.0.0.0/4 64:ff9b::/96 fc00::/7 fd00:ec2::254/128 fe80::/10 fec0::/10
+# An allow opens an address both ways and on every port, so a loopback
+# resolver can never be opened: a tenant could bind systemd-resolved's
+# 127.0.0.53 itself and reach every listener bound to all addresses
+# through it. Where resolved runs, instances resolve through its
+# upstreams instead, and the control service opens those. The "-" makes
+# this a no-op on a box without resolved.
+BindReadOnlyPaths=-/run/systemd/resolve/resolv.conf:/etc/resolv.conf
ProtectKernelTunables=yes
ProtectKernelModules=yes
ProtectKernelLogs=yes
@@ -84,3 +131,9 @@ SystemCallArchitectures=native
# --- per-service resources; the tier lives on the host slice (§4.3) ---
TasksMax=64
MemoryMax=64M
+# Read back by the probe from its own cgroup (#214 item 1): proof that CPU
+# and IO settings take under carlos-instances.slice, where the control
+# service sets each host slice's from its tier.
+CPUQuota=50%
+CPUWeight=50
+IOWeight=50
diff --git a/internal/host/units/carlos-instance-seed@.service b/internal/host/units/carlos-instance-seed@.service
index 7dd47f549..706bf5723 100644
--- a/internal/host/units/carlos-instance-seed@.service
+++ b/internal/host/units/carlos-instance-seed@.service
@@ -50,6 +50,12 @@ StandardError=inherit
# The instance's own two directories, the only writable data paths.
BindPaths=/data/instances/%I /run/carlos/instances/%I
ReadWritePaths=/data/instances/%I /run/carlos/instances/%I
+# Its own build, and no other app's (#214 item 2). The namespace block
+# empties /opt/carlos; these put back /opt/carlos/live/<host>, a link to
+# ../app, and /opt/carlos/app, which PID 1 binds by chasing the source
+# link to this host's live build at every start. The control service
+# writes both sources before it starts the unit.
+BindReadOnlyPaths=/run/carlos-control/exe/%I/app:/opt/carlos/app /run/carlos-control/exe/%I/live:/opt/carlos/live
# --- the namespace (titogo@.service, 2026-07-31 design, on systemd 257) ---
ProtectSystem=strict
@@ -60,6 +66,13 @@ TemporaryFileSystem=/run/carlos:ro
# /var/lib as a class. An instance needs nothing from either.
TemporaryFileSystem=/var/cache:ro
TemporaryFileSystem=/var/lib:ro
+# The version store holds every app's build on the box, world-readable so
+# a unit can exec its own (#214 item 2). An instance sees none of it: an
+# empty /opt/carlos with the platform binary that ExecStart= and
+# ExecStopPost= run put back, and, outside this block because it is per
+# instance, its own build at /opt/carlos/live/<host>.
+TemporaryFileSystem=/opt/carlos:ro
+BindReadOnlyPaths=/opt/carlos/carlos
# Root-written credentials a carlos-GROUP process may read live here:
# /etc/carlos/relay-masters is root:carlos 0640 for carlos-relay.service,
# and an instance's Group=carlos would read it too (codex R10-1). An
@@ -77,6 +90,15 @@ PrivateDevices=yes
# and every instance is uid carlos, so a sibling's shared memory would be
# one open away. Each unit gets its own, charged to its own cgroup.
TemporaryFileSystem=/dev/shm:mode=1777,nosuid,nodev
+# POSIX message queues and SysV IPC belong to an IPC namespace, and every
+# instance is uid carlos, so a sibling's queue would be one mq_open away
+# (#178). PrivateIPC= gives each unit its own namespace, and mq_open()
+# goes through that namespace's own internal mount. PrivateDevices=
+# carries the host's /dev/mqueue into the private /dev the way it carries
+# /dev/shm, and that mount would still list and open the host namespace's
+# queues, so it is replaced the same way.
+PrivateIPC=yes
+TemporaryFileSystem=/dev/mqueue:ro
# /sys/fs/cgroup read-only: the edge's delegated subtree is carlos-owned,
# and its cgroup.kill / cgroup.freeze would otherwise be one write away
# for a same-uid tenant (codex R9-1).
@@ -88,7 +110,32 @@ RestrictNamespaces=yes
NoNewPrivileges=yes
CapabilityBoundingSet=
RestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX
-IPAddressDeny=169.254.169.254/32 fd00:ec2::254/128
+# Abstract unix sockets live in the network namespace, which instances
+# share with the box because they need outbound TCP, and the line above
+# cannot tell them from path sockets (#178). The trampoline scopes them
+# with Landlock before it execs the app (instanceexec.ScopeAbstractUnix):
+# an instance can connect only to abstract sockets its own processes
+# bound. A sibling can still bind a name first; that is a refusal of
+# service, never a connection.
+#
+# Egress (#214 item 3): the internet, and nothing on the box or private.
+# Loopback is where the console and other tenants' addr upstreams listen;
+# RFC 1918, ULA, link-local and CGNAT are the VPC, other boxes and the
+# cloud's internal services. IMDS stays named so the property is
+# greppable. A cgroup filter: it drops in both directions for every
+# socket the unit opens. The one thing an instance needs from these
+# ranges is the box's DNS resolver, so the control service re-opens
+# exactly the nameservers the instance sees with IPAddressAllow= on
+# carlos-instances.slice (an allow wins, and a slice's lists reach every
+# unit below it). The probe checks both halves.
+IPAddressDeny=localhost multicast 0.0.0.0/8 10.0.0.0/8 100.64.0.0/10 169.254.0.0/16 169.254.169.254/32 172.16.0.0/12 192.0.0.0/24 192.168.0.0/16 198.18.0.0/15 240.0.0.0/4 64:ff9b::/96 fc00::/7 fd00:ec2::254/128 fe80::/10 fec0::/10
+# An allow opens an address both ways and on every port, so a loopback
+# resolver can never be opened: a tenant could bind systemd-resolved's
+# 127.0.0.53 itself and reach every listener bound to all addresses
+# through it. Where resolved runs, instances resolve through its
+# upstreams instead, and the control service opens those. The "-" makes
+# this a no-op on a box without resolved.
+BindReadOnlyPaths=-/run/systemd/resolve/resolv.conf:/etc/resolv.conf
ProtectKernelTunables=yes
ProtectKernelModules=yes
ProtectKernelLogs=yes
diff --git a/internal/host/units/carlos-instance-sidecar@.service b/internal/host/units/carlos-instance-sidecar@.service
index 88f3be41f..7d34f2f89 100644
--- a/internal/host/units/carlos-instance-sidecar@.service
+++ b/internal/host/units/carlos-instance-sidecar@.service
@@ -46,6 +46,12 @@ StandardError=inherit
# The instance's own two directories, the only writable data paths.
BindPaths=/data/instances/%I /run/carlos/instances/%I
ReadWritePaths=/data/instances/%I /run/carlos/instances/%I
+# Its own build, and no other app's (#214 item 2). The namespace block
+# empties /opt/carlos; these put back /opt/carlos/live/<host>, a link to
+# ../app, and /opt/carlos/app, which PID 1 binds by chasing the source
+# link to this host's live build at every start. The control service
+# writes both sources before it starts the unit.
+BindReadOnlyPaths=/run/carlos-control/exe/%I/app:/opt/carlos/app /run/carlos-control/exe/%I/live:/opt/carlos/live
# --- the namespace (titogo@.service, 2026-07-31 design, on systemd 257) ---
ProtectSystem=strict
@@ -56,6 +62,13 @@ TemporaryFileSystem=/run/carlos:ro
# /var/lib as a class. An instance needs nothing from either.
TemporaryFileSystem=/var/cache:ro
TemporaryFileSystem=/var/lib:ro
+# The version store holds every app's build on the box, world-readable so
+# a unit can exec its own (#214 item 2). An instance sees none of it: an
+# empty /opt/carlos with the platform binary that ExecStart= and
+# ExecStopPost= run put back, and, outside this block because it is per
+# instance, its own build at /opt/carlos/live/<host>.
+TemporaryFileSystem=/opt/carlos:ro
+BindReadOnlyPaths=/opt/carlos/carlos
# Root-written credentials a carlos-GROUP process may read live here:
# /etc/carlos/relay-masters is root:carlos 0640 for carlos-relay.service,
# and an instance's Group=carlos would read it too (codex R10-1). An
@@ -73,6 +86,15 @@ PrivateDevices=yes
# and every instance is uid carlos, so a sibling's shared memory would be
# one open away. Each unit gets its own, charged to its own cgroup.
TemporaryFileSystem=/dev/shm:mode=1777,nosuid,nodev
+# POSIX message queues and SysV IPC belong to an IPC namespace, and every
+# instance is uid carlos, so a sibling's queue would be one mq_open away
+# (#178). PrivateIPC= gives each unit its own namespace, and mq_open()
+# goes through that namespace's own internal mount. PrivateDevices=
+# carries the host's /dev/mqueue into the private /dev the way it carries
+# /dev/shm, and that mount would still list and open the host namespace's
+# queues, so it is replaced the same way.
+PrivateIPC=yes
+TemporaryFileSystem=/dev/mqueue:ro
# /sys/fs/cgroup read-only: the edge's delegated subtree is carlos-owned,
# and its cgroup.kill / cgroup.freeze would otherwise be one write away
# for a same-uid tenant (codex R9-1).
@@ -84,7 +106,32 @@ RestrictNamespaces=yes
NoNewPrivileges=yes
CapabilityBoundingSet=
RestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX
-IPAddressDeny=169.254.169.254/32 fd00:ec2::254/128
+# Abstract unix sockets live in the network namespace, which instances
+# share with the box because they need outbound TCP, and the line above
+# cannot tell them from path sockets (#178). The trampoline scopes them
+# with Landlock before it execs the app (instanceexec.ScopeAbstractUnix):
+# an instance can connect only to abstract sockets its own processes
+# bound. A sibling can still bind a name first; that is a refusal of
+# service, never a connection.
+#
+# Egress (#214 item 3): the internet, and nothing on the box or private.
+# Loopback is where the console and other tenants' addr upstreams listen;
+# RFC 1918, ULA, link-local and CGNAT are the VPC, other boxes and the
+# cloud's internal services. IMDS stays named so the property is
+# greppable. A cgroup filter: it drops in both directions for every
+# socket the unit opens. The one thing an instance needs from these
+# ranges is the box's DNS resolver, so the control service re-opens
+# exactly the nameservers the instance sees with IPAddressAllow= on
+# carlos-instances.slice (an allow wins, and a slice's lists reach every
+# unit below it). The probe checks both halves.
+IPAddressDeny=localhost multicast 0.0.0.0/8 10.0.0.0/8 100.64.0.0/10 169.254.0.0/16 169.254.169.254/32 172.16.0.0/12 192.0.0.0/24 192.168.0.0/16 198.18.0.0/15 240.0.0.0/4 64:ff9b::/96 fc00::/7 fd00:ec2::254/128 fe80::/10 fec0::/10
+# An allow opens an address both ways and on every port, so a loopback
+# resolver can never be opened: a tenant could bind systemd-resolved's
+# 127.0.0.53 itself and reach every listener bound to all addresses
+# through it. Where resolved runs, instances resolve through its
+# upstreams instead, and the control service opens those. The "-" makes
+# this a no-op on a box without resolved.
+BindReadOnlyPaths=-/run/systemd/resolve/resolv.conf:/etc/resolv.conf
ProtectKernelTunables=yes
ProtectKernelModules=yes
ProtectKernelLogs=yes
diff --git a/internal/host/units/carlos-instance@.service b/internal/host/units/carlos-instance@.service
index 2baae24e8..0f6ca71b8 100644
--- a/internal/host/units/carlos-instance@.service
+++ b/internal/host/units/carlos-instance@.service
@@ -46,6 +46,12 @@ StandardError=inherit
# The instance's own two directories, the only writable data paths.
BindPaths=/data/instances/%I /run/carlos/instances/%I
ReadWritePaths=/data/instances/%I /run/carlos/instances/%I
+# Its own build, and no other app's (#214 item 2). The namespace block
+# empties /opt/carlos; these put back /opt/carlos/live/<host>, a link to
+# ../app, and /opt/carlos/app, which PID 1 binds by chasing the source
+# link to this host's live build at every start. The control service
+# writes both sources before it starts the unit.
+BindReadOnlyPaths=/run/carlos-control/exe/%I/app:/opt/carlos/app /run/carlos-control/exe/%I/live:/opt/carlos/live
# --- the namespace (titogo@.service, 2026-07-31 design, on systemd 257) ---
ProtectSystem=strict
@@ -56,6 +62,13 @@ TemporaryFileSystem=/run/carlos:ro
# /var/lib as a class. An instance needs nothing from either.
TemporaryFileSystem=/var/cache:ro
TemporaryFileSystem=/var/lib:ro
+# The version store holds every app's build on the box, world-readable so
+# a unit can exec its own (#214 item 2). An instance sees none of it: an
+# empty /opt/carlos with the platform binary that ExecStart= and
+# ExecStopPost= run put back, and, outside this block because it is per
+# instance, its own build at /opt/carlos/live/<host>.
+TemporaryFileSystem=/opt/carlos:ro
+BindReadOnlyPaths=/opt/carlos/carlos
# Root-written credentials a carlos-GROUP process may read live here:
# /etc/carlos/relay-masters is root:carlos 0640 for carlos-relay.service,
# and an instance's Group=carlos would read it too (codex R10-1). An
@@ -73,6 +86,15 @@ PrivateDevices=yes
# and every instance is uid carlos, so a sibling's shared memory would be
# one open away. Each unit gets its own, charged to its own cgroup.
TemporaryFileSystem=/dev/shm:mode=1777,nosuid,nodev
+# POSIX message queues and SysV IPC belong to an IPC namespace, and every
+# instance is uid carlos, so a sibling's queue would be one mq_open away
+# (#178). PrivateIPC= gives each unit its own namespace, and mq_open()
+# goes through that namespace's own internal mount. PrivateDevices=
+# carries the host's /dev/mqueue into the private /dev the way it carries
+# /dev/shm, and that mount would still list and open the host namespace's
+# queues, so it is replaced the same way.
+PrivateIPC=yes
+TemporaryFileSystem=/dev/mqueue:ro
# /sys/fs/cgroup read-only: the edge's delegated subtree is carlos-owned,
# and its cgroup.kill / cgroup.freeze would otherwise be one write away
# for a same-uid tenant (codex R9-1).
@@ -84,7 +106,32 @@ RestrictNamespaces=yes
NoNewPrivileges=yes
CapabilityBoundingSet=
RestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX
-IPAddressDeny=169.254.169.254/32 fd00:ec2::254/128
+# Abstract unix sockets live in the network namespace, which instances
+# share with the box because they need outbound TCP, and the line above
+# cannot tell them from path sockets (#178). The trampoline scopes them
+# with Landlock before it execs the app (instanceexec.ScopeAbstractUnix):
+# an instance can connect only to abstract sockets its own processes
+# bound. A sibling can still bind a name first; that is a refusal of
+# service, never a connection.
+#
+# Egress (#214 item 3): the internet, and nothing on the box or private.
+# Loopback is where the console and other tenants' addr upstreams listen;
+# RFC 1918, ULA, link-local and CGNAT are the VPC, other boxes and the
+# cloud's internal services. IMDS stays named so the property is
+# greppable. A cgroup filter: it drops in both directions for every
+# socket the unit opens. The one thing an instance needs from these
+# ranges is the box's DNS resolver, so the control service re-opens
+# exactly the nameservers the instance sees with IPAddressAllow= on
+# carlos-instances.slice (an allow wins, and a slice's lists reach every
+# unit below it). The probe checks both halves.
+IPAddressDeny=localhost multicast 0.0.0.0/8 10.0.0.0/8 100.64.0.0/10 169.254.0.0/16 169.254.169.254/32 172.16.0.0/12 192.0.0.0/24 192.168.0.0/16 198.18.0.0/15 240.0.0.0/4 64:ff9b::/96 fc00::/7 fd00:ec2::254/128 fe80::/10 fec0::/10
+# An allow opens an address both ways and on every port, so a loopback
+# resolver can never be opened: a tenant could bind systemd-resolved's
+# 127.0.0.53 itself and reach every listener bound to all addresses
+# through it. Where resolved runs, instances resolve through its
+# upstreams instead, and the control service opens those. The "-" makes
+# this a no-op on a box without resolved.
+BindReadOnlyPaths=-/run/systemd/resolve/resolv.conf:/etc/resolv.conf
ProtectKernelTunables=yes
ProtectKernelModules=yes
ProtectKernelLogs=yes
diff --git a/internal/host/units_hardening_test.go b/internal/host/units_hardening_test.go
new file mode 100644
index 000000000..0cd6fc864
--- /dev/null
+++ b/internal/host/units_hardening_test.go
@@ -0,0 +1,148 @@
+package host
+
+import (
+ "strings"
+ "testing"
+)
+
+// directives returns every "Key=value" line of a unit, comments and
+// section headers dropped, in file order.
+func directives(u string) []string {
+ var out []string
+ for _, line := range strings.Split(u, "\n") {
+ line = strings.TrimSpace(line)
+ if line == "" || strings.HasPrefix(line, "#") || strings.HasPrefix(line, "[") {
+ continue
+ }
+ out = append(out, line)
+ }
+ return out
+}
+
+func hasDirective(u, want string) bool {
+ for _, d := range directives(u) {
+ if d == want {
+ return true
+ }
+ }
+ return false
+}
+
+// instanceDeny is the namespace block's egress fence (#214 item 3): every
+// address a tenant has no business dialling, loopback first. A tenant
+// reaches the internet; it does not reach the box, the VPC, or the cloud's
+// internal ranges. The box's resolver is re-opened per box by the control
+// service on carlos-instances.slice (IPAddressAllow=, which wins).
+const instanceDeny = "IPAddressDeny=localhost multicast " +
+ "0.0.0.0/8 10.0.0.0/8 100.64.0.0/10 169.254.0.0/16 169.254.169.254/32 172.16.0.0/12 192.0.0.0/24 192.168.0.0/16 198.18.0.0/15 240.0.0.0/4 " +
+ "64:ff9b::/96 fc00::/7 fd00:ec2::254/128 fe80::/10 fec0::/10"
+
+// Every directive the hardening adds to the shared namespace block (#214
+// items 2 and 3, #178), pinned exactly: the block is byte-identical across
+// the three templates and the probe (TestInstanceTemplatesShareOneNamespace),
+// so pinning it on carlos-instance@ pins it on all four.
+func TestNamespaceBlockCarriesTheHardening(t *testing.T) {
+ for _, name := range []string{"carlos-instance@.service", "carlos-instance-seed@.service", "carlos-instance-sidecar@.service", "carlos-instance-probe.service"} {
+ u := readUnit(t, name)
+ for _, want := range []string{
+ // #214 item 2: no other app's build is visible.
+ "TemporaryFileSystem=/opt/carlos:ro",
+ "BindReadOnlyPaths=/opt/carlos/carlos",
+ // #214 item 3: loopback, private and cloud-internal ranges.
+ instanceDeny,
+ // #178: SysV and POSIX mqueue namespaces of its own, and the
+ // host's mqueue mount out of sight.
+ // #214 item 3: resolved's upstreams, never its loopback stub.
+ "BindReadOnlyPaths=-/run/systemd/resolve/resolv.conf:/etc/resolv.conf",
+ "PrivateIPC=yes",
+ "TemporaryFileSystem=/dev/mqueue:ro",
+ } {
+ if !hasDirective(u, want) {
+ t.Errorf("%s lacks %q", name, want)
+ }
+ }
+ // The old IMDS-only line is gone, not left beside the new one:
+ // two IPAddressDeny= lines would still union, but a reader would
+ // take the first for the whole fence.
+ if hasDirective(u, "IPAddressDeny=169.254.169.254/32 fd00:ec2::254/128") {
+ t.Errorf("%s still carries the IMDS-only IPAddressDeny line", name)
+ }
+ // Nothing in the namespace re-opens what the fence closes: the
+ // resolver allow lives on the slice, set per box at run time.
+ for _, d := range directives(u) {
+ if strings.HasPrefix(d, "IPAddressAllow=") {
+ t.Errorf("%s carries %q; the resolver allow belongs on carlos-instances.slice", name, d)
+ }
+ }
+ }
+}
+
+// The doc-engine fence (caps/render.go) is the platform's other private-
+// range list. The instance fence must cover every range it does, plus
+// loopback, which the doc engine deliberately leaves open.
+func TestInstanceDenyCoversTheDocEngineFence(t *testing.T) {
+ have := map[string]bool{}
+ for _, f := range strings.Fields(strings.TrimPrefix(instanceDeny, "IPAddressDeny=")) {
+ have[f] = true
+ }
+ // localhost and multicast are systemd's names for 127.0.0.0/8 ::1/128
+ // and 224.0.0.0/4 ff00::/8. Link-local is spelled out: systemd's
+ // "link-local" is fe80::/64, narrower than the fe80::/10 here.
+ named := map[string]string{"224.0.0.0/4": "multicast", "ff00::/8": "multicast"}
+ for _, r := range []string{"0.0.0.0/8", "10.0.0.0/8", "100.64.0.0/10", "169.254.0.0/16", "169.254.169.254/32", "172.16.0.0/12",
+ "192.0.0.0/24", "192.168.0.0/16", "198.18.0.0/15", "224.0.0.0/4", "240.0.0.0/4",
+ "64:ff9b::/96", "fc00::/7", "fd00:ec2::254/128", "fe80::/10", "fec0::/10", "ff00::/8"} {
+ if !have[r] && !have[named[r]] {
+ t.Errorf("instance fence lacks %s, which the doc-engine fence denies", r)
+ }
+ }
+ if !have["localhost"] {
+ t.Error("instance fence leaves loopback open")
+ }
+}
+
+// Each template binds back exactly its own build (#214 item 2): the control
+// service writes the two sources before every start (instancectl ensureExe).
+func TestInstanceTemplatesBindOnlyTheirOwnBuild(t *testing.T) {
+ for name, iid := range map[string]string{
+ "carlos-instance@.service": "%I",
+ "carlos-instance-seed@.service": "%I",
+ "carlos-instance-sidecar@.service": "%I",
+ "carlos-instance-probe.service": "_probe",
+ } {
+ u := readUnit(t, name)
+ want := "BindReadOnlyPaths=/run/carlos-control/exe/" + iid + "/app:/opt/carlos/app /run/carlos-control/exe/" + iid + "/live:/opt/carlos/live"
+ if !hasDirective(u, want) {
+ t.Errorf("%s lacks %q", name, want)
+ }
+ for _, d := range directives(u) {
+ if strings.Contains(d, "/opt/carlos/versions") {
+ t.Errorf("%s names the version store: %q", name, d)
+ }
+ }
+ }
+}
+
+// The probe's own resource section carries a CPU and IO setting it can read
+// back from its own cgroup (#214 item 1): proof that systemd applies them
+// under carlos-instances.slice on this box, which is where the per-host
+// slices get the tier's.
+func TestProbeCarriesReadableCPUAndIOSettings(t *testing.T) {
+ u := readUnit(t, "carlos-instance-probe.service")
+ for _, want := range []string{"CPUQuota=50%", "CPUWeight=50", "IOWeight=50"} {
+ if !hasDirective(u, want) {
+ t.Errorf("probe lacks %q", want)
+ }
+ }
+}
+
+// carlos-app@ gets the same /opt view (#214 item 2): its ExecStart is
+// /opt/carlos/live/%i, which PID 1 chases to the build when it binds it.
+func TestAppTemplateSeesOnlyItsOwnBuild(t *testing.T) {
+ u := readUnit(t, "carlos-app@.service")
+ for _, want := range []string{"TemporaryFileSystem=/opt/carlos:ro", "BindReadOnlyPaths=/opt/carlos/live/%i", "ExecStart=/opt/carlos/live/%i serve"} {
+ if !hasDirective(u, want) {
+ t.Errorf("carlos-app@.service lacks %q", want)
+ }
+ }
+}
diff --git a/internal/host/units_isolation_test.go b/internal/host/units_isolation_test.go
index 5d6357aaf..527c48f05 100644
--- a/internal/host/units_isolation_test.go
+++ b/internal/host/units_isolation_test.go
@@ -250,7 +250,7 @@ func TestInstanceTemplatesShareOneNamespace(t *testing.T) {
"TemporaryFileSystem=/dev/shm:mode=1777,nosuid,nodev",
"ProtectControlGroups=yes", "ProtectProc=invisible", "ProcSubset=pid", "PrivatePIDs=yes",
"RestrictNamespaces=yes", "NoNewPrivileges=yes", "CapabilityBoundingSet=",
- "IPAddressDeny=169.254.169.254/32 fd00:ec2::254/128", "MemoryDenyWriteExecute=yes",
+ "IPAddressDeny=localhost ", " 169.254.169.254/32 ", " fd00:ec2::254/128 ", "PrivateIPC=yes", "MemoryDenyWriteExecute=yes",
} {
if !strings.Contains(ref, want) {
t.Errorf("namespace block lacks %q", want)
diff --git a/internal/instancectl/hardening.go b/internal/instancectl/hardening.go
new file mode 100644
index 000000000..17f452979
--- /dev/null
+++ b/internal/instancectl/hardening.go
@@ -0,0 +1,342 @@
+package instancectl
+
+import (
+ "bufio"
+ "bytes"
+ "context"
+ "errors"
+ "fmt"
+ "io/fs"
+ "net/http"
+ "net/netip"
+ "os"
+ "path/filepath"
+ "sort"
+ "strings"
+ "time"
+
+ "github.com/carlosframework/platform/internal/provision"
+)
+
+// What the control service does beyond start and stop, so the isolation
+// the templates promise holds (#214, #178): it lays out each instance's
+// view of its own build, re-opens the box's DNS resolver through the
+// egress fence, and keeps tenants from filling the shared data disk.
+
+// Paths the instance units and the probe bind from (#214 item 2). The
+// templates name them literally; the units test pins both sides.
+const (
+ // LiveDir is where the adopt pass keeps each host's live link, and
+ // where an instance finds its own build inside its namespace.
+ LiveDir = "/opt/carlos/live"
+ // SystemBinary is the platform binary every instance unit runs as
+ // ExecStart=; the probe's stand-in build.
+ SystemBinary = "/opt/carlos/carlos"
+ // InstancesSlice is the parent of every per-host slice and of the
+ // probe: an IPAddressAllow= on it reaches every instance unit.
+ InstancesSlice = "carlos-instances.slice"
+)
+
+// ensureExe writes the two sources an instance unit binds over its empty
+// /opt/carlos (#214 item 2), under root/<iid>/:
+//
+// app -> target (the host's live link; the probe's: the platform binary)
+// live/<name> -> ../app
+//
+// PID 1 chases app to the build when it binds it at /opt/carlos/app, and
+// binds live/ at /opt/carlos/live, so inside the namespace
+// /opt/carlos/live/<host> resolves to the build and nothing else of the
+// version store exists. Both are links, never copies: a deploy repoints
+// the host's live link and the next start binds the new build, exactly as
+// before. Only root writes under root, so nothing here can be planted;
+// anything else found in live/ is removed all the same, because live/ is
+// bound into a tenant.
+func ensureExe(root, iid, target, name string) error {
+ if root == "" {
+ return errors.New("no exe root")
+ }
+ if iid == "" || strings.ContainsAny(iid, "/\x00") || iid == "." || iid == ".." ||
+ name == "" || strings.ContainsAny(name, "/\x00") || name == "." || name == ".." {
+ return fmt.Errorf("exe: %q/%q is not a name", iid, name)
+ }
+ dir := filepath.Join(root, iid)
+ live := filepath.Join(dir, "live")
+ for _, d := range []string{root, dir, live} {
+ if err := os.Mkdir(d, 0o755); err != nil && !errors.Is(err, fs.ErrExist) {
+ return err
+ }
+ fi, err := os.Lstat(d)
+ if err != nil {
+ return err
+ }
+ if !fi.IsDir() {
+ return fmt.Errorf("exe: %s is not a directory", d)
+ }
+ }
+ if err := replaceLink(filepath.Join(dir, "app"), target); err != nil {
+ return err
+ }
+ if err := replaceLink(filepath.Join(live, name), "../app"); err != nil {
+ return err
+ }
+ entries, err := os.ReadDir(live)
+ if err != nil {
+ return err
+ }
+ for _, e := range entries {
+ if e.Name() != name {
+ if err := os.RemoveAll(filepath.Join(live, e.Name())); err != nil {
+ return err
+ }
+ }
+ }
+ return nil
+}
+
+// replaceLink points path at target, atomically: a temp link renamed over
+// whatever was there, so a unit starting at that moment binds the old
+// target or the new one, never nothing.
+func replaceLink(path, target string) error {
+ if cur, err := os.Readlink(path); err == nil && cur == target {
+ return nil
+ }
+ tmp := path + ".tmp"
+ _ = os.Remove(tmp)
+ if err := os.Symlink(target, tmp); err != nil {
+ return err
+ }
+ if err := os.Rename(tmp, path); err != nil {
+ os.Remove(tmp)
+ return err
+ }
+ return nil
+}
+
+// ensureHostExe is ensureExe for a host about to start.
+func (s *Server) ensureHostExe(host string) error {
+ return ensureExe(s.ExeRoot, provision.InstanceID(host), filepath.Join(LiveDir, host), host)
+}
+
+// ParseResolvers returns the nameserver addresses of a resolv.conf, in
+// order, zones dropped. Lines it cannot read are skipped: the resolver
+// library skips them too.
+func ParseResolvers(b []byte) []netip.Addr {
+ var out []netip.Addr
+ sc := bufio.NewScanner(bytes.NewReader(b))
+ for sc.Scan() {
+ f := strings.Fields(sc.Text())
+ if len(f) < 2 || f[0] != "nameserver" {
+ continue
+ }
+ a, err := netip.ParseAddr(f[1])
+ if err != nil {
+ continue
+ }
+ out = append(out, a.WithZone("").Unmap())
+ }
+ return out
+}
+
+// ResolverAllowed says whether a nameserver may be re-opened through the
+// instance egress fence (#214 item 3), and if not, why. IPAddressAllow=
+// opens an address, not a port, and in both directions, so an allowed
+// address must be one where nothing but a resolver can answer:
+//
+// - no loopback address, systemd-resolved's stubs (127.0.0.53/54)
+// included: a tenant can bind the address itself and then reach every
+// listener on the box bound to all addresses through it. Instances
+// resolve through resolved's UPSTREAMS instead (the namespace binds
+// /run/systemd/resolve/resolv.conf over /etc/resolv.conf);
+// - no unspecified address, which means "this box";
+// - no metadata service;
+// - no link-local address but Amazon's resolver, 169.254.169.253. An
+// allow cannot carry a zone, so fe80::1 would open that address on
+// every link, and link-local v4 is where IMDS and the time service
+// live. Boxes resolve through the VPC resolver; a link-local one is
+// refused and the probe's dns line says so.
+//
+// Everything else is opened as a single address: the VPC resolver (VPC
+// base + 2, inside RFC 1918), fd00:ec2::253, or a public one. That the
+// address is not one of the box's own is checked by the probe, which can
+// see the interfaces; this service cannot (AF_UNIX only).
+func ResolverAllowed(a netip.Addr) (bool, string) {
+ switch {
+ case !a.IsValid():
+ return false, "not an address"
+ case a == netip.MustParseAddr("127.0.0.53"), a == netip.MustParseAddr("127.0.0.54"):
+ return false, "systemd-resolved's stub is on loopback; instances use its upstreams"
+ case a.IsLoopback():
+ return false, "a loopback resolver shares its address with services on this box"
+ case a.IsUnspecified():
+ return false, "the unspecified address is this box"
+ case a == netip.MustParseAddr("169.254.169.254"), a == netip.MustParseAddr("fd00:ec2::254"):
+ return false, "the metadata service is not a resolver"
+ case a == netip.MustParseAddr("169.254.169.253"):
+ return true, ""
+ case a.IsLinkLocalUnicast():
+ return false, "a link-local resolver cannot be opened for one link only"
+ }
+ return true, ""
+}
+
+// resolverAllow is the IPAddressAllow= value for a resolv.conf: each
+// allowed nameserver as a single-address prefix, sorted and deduplicated
+// so the drop-in systemd writes does not churn.
+func resolverAllow(b []byte) []string {
+ seen := map[string]bool{}
+ var out []string
+ for _, a := range ParseResolvers(b) {
+ if ok, _ := ResolverAllowed(a); !ok {
+ continue
+ }
+ p := netip.PrefixFrom(a, a.BitLen()).String()
+ if !seen[p] {
+ seen[p] = true
+ out = append(out, p)
+ }
+ }
+ sort.Strings(out)
+ return out
+}
+
+// ResolvedConf is systemd-resolved's list of its upstreams. When it
+// exists, the namespace binds it over /etc/resolv.conf (the "-" bind in
+// the templates), so it is what instances resolve through and what the
+// allow must open.
+const ResolvedConf = "/run/systemd/resolve/resolv.conf"
+
+// resolverFile is the resolv.conf an instance sees: resolved's upstream
+// list when there is one, the box's own otherwise. Mirrors the template.
+func (s *Server) resolverFile() string {
+ if s.ResolvedConf != "" {
+ if _, err := os.Stat(s.ResolvedConf); err == nil {
+ return s.ResolvedConf
+ }
+ }
+ return s.ResolvConf
+}
+
+// allowResolvers re-opens the resolvers instances see on
+// carlos-instances.slice. The empty assignment first resets the list, so
+// a resolver that left the file is closed again. It runs before every
+// probe, so the allow is in place before the first start and follows a
+// resolver change within one probe period; the probe then checks the
+// result from inside.
+func (s *Server) allowResolvers(ctx context.Context) error {
+ b, err := os.ReadFile(s.resolverFile())
+ if err != nil {
+ return err
+ }
+ args := []string{"set-property", "--runtime", InstancesSlice, "IPAddressAllow="}
+ if allow := resolverAllow(b); len(allow) > 0 {
+ args = append(args, "IPAddressAllow="+strings.Join(allow, " "))
+ }
+ out, err := s.Systemctl(ctx, args...)
+ if err != nil {
+ return fmt.Errorf("%s: %s", strings.TrimSpace(out), err)
+ }
+ return nil
+}
+
+// The data disk (#214 item 1). Every instance's directory, the registry,
+// the activator's state and litestream's shadow WAL share one filesystem,
+// /data, on the root volume. A per-tenant quota is not achievable there
+// today: ext4 project quotas need the "project" and "quota" features,
+// which tune2fs sets only on an unmounted filesystem, plus a prjquota
+// mount, and the root filesystem is never unmounted on a running box.
+// Every process that writes /data is uid carlos, so reserved blocks
+// cannot tell a tenant from the agent either.
+//
+// Nor can this service attribute usage well enough to stop a tenant for
+// it: space held by unlinked-but-open files appears in no directory, so
+// the guilty tenant can look small while honest ones look large, and a
+// stop from here bypasses the activator's fence and holds (a host mid-
+// migration). So the guard refuses, it never stops: below the floor no
+// instance starts or seeds, and a warning names the shortfall every
+// DiskEvery while it lasts. Instances already awake keep running.
+
+const minStartFloor = 1 << 30 // 1 GiB
+
+// DiskFloor is the free space a start needs on a filesystem of total
+// bytes: the larger of 1 GiB and 10%.
+func DiskFloor(total uint64) uint64 { return max(minStartFloor, total/10) }
+
+// ErrDiskFull is a start refused for disk space.
+var ErrDiskFull = errors.New("data disk too full")
+
+// diskRoom is nil when there is room to start an instance.
+func (s *Server) diskRoom() error {
+ if s.Statfs == nil {
+ return nil
+ }
+ avail, total, err := s.Statfs(s.DataRoot)
+ if err != nil {
+ return fmt.Errorf("statfs %s: %w", s.DataRoot, err)
+ }
+ if floor := DiskFloor(total); avail < floor {
+ return fmt.Errorf("%w: %s free, below the %s floor", ErrDiskFull, humanBytes(avail), humanBytes(floor))
+ }
+ return nil
+}
+
+// refuseStart answers a start the disk or the exe layout cannot take. It
+// reports whether it answered.
+func (s *Server) refuseStart(w http.ResponseWriter, host, unit string) bool {
+ if err := s.diskRoom(); err != nil {
+ s.logf("instance-control: %s: not starting %s: %v", host, unit, err)
+ status := http.StatusBadGateway
+ if errors.Is(err, ErrDiskFull) {
+ status = http.StatusInsufficientStorage
+ }
+ http.Error(w, "not starting "+unit+": "+err.Error(), status)
+ return true
+ }
+ if s.ExeRoot != "" {
+ if err := s.ensureHostExe(host); err != nil {
+ http.Error(w, "exe: "+err.Error(), http.StatusBadGateway)
+ return true
+ }
+ }
+ return false
+}
+
+// warnIfFull logs the alert while the disk is below the floor. It reports
+// whether it warned.
+func (s *Server) warnIfFull() bool {
+ err := s.diskRoom()
+ if errors.Is(err, ErrDiskFull) {
+ s.logf("instance-control: disk: %v; refusing every instance start until there is room (awake instances keep running)", err)
+ return true
+ }
+ if err != nil {
+ s.logf("instance-control: disk: %v", err)
+ }
+ return false
+}
+
+// diskLoop runs warnIfFull every DiskEvery.
+func (s *Server) diskLoop(ctx context.Context) {
+ if s.DiskEvery <= 0 {
+ return
+ }
+ t := time.NewTicker(s.DiskEvery)
+ defer t.Stop()
+ for {
+ select {
+ case <-ctx.Done():
+ return
+ case <-t.C:
+ s.warnIfFull()
+ }
+ }
+}
+
+func humanBytes(n uint64) string {
+ switch {
+ case n >= 1<<30:
+ return fmt.Sprintf("%.1f GiB", float64(n)/(1<<30))
+ case n >= 1<<20:
+ return fmt.Sprintf("%.0f MiB", float64(n)/(1<<20))
+ }
+ return fmt.Sprintf("%d bytes", n)
+}
diff --git a/internal/instancectl/hardening_test.go b/internal/instancectl/hardening_test.go
new file mode 100644
index 000000000..0a0db345c
--- /dev/null
+++ b/internal/instancectl/hardening_test.go
@@ -0,0 +1,185 @@
+package instancectl
+
+import (
+ "context"
+ "fmt"
+ "net/http"
+ "net/netip"
+ "os"
+ "path/filepath"
+ "strings"
+ "testing"
+
+ "github.com/carlosframework/platform/internal/provision"
+)
+
+// ensureExe lays out exactly the two links the templates bind (#214 item
+// 2): app at the host's live link, live/<host> at ../app, nothing else in
+// live/, and a second call repoints rather than failing.
+func TestEnsureExeLaysOutOneBuild(t *testing.T) {
+ root := filepath.Join(t.TempDir(), "exe")
+ if err := ensureExe(root, "h.example.com", "/opt/carlos/live/h.example.com", "h.example.com"); err != nil {
+ t.Fatal(err)
+ }
+ if err := os.WriteFile(filepath.Join(root, "h.example.com", "live", "stray"), nil, 0o644); err != nil {
+ t.Fatal(err)
+ }
+ if err := ensureExe(root, "h.example.com", "/opt/carlos/live/h.example.com", "h.example.com"); err != nil {
+ t.Fatal(err)
+ }
+ if got, _ := os.Readlink(filepath.Join(root, "h.example.com", "app")); got != "/opt/carlos/live/h.example.com" {
+ t.Errorf("app -> %q", got)
+ }
+ if got, _ := os.Readlink(filepath.Join(root, "h.example.com", "live", "h.example.com")); got != "../app" {
+ t.Errorf("live/h.example.com -> %q", got)
+ }
+ entries, _ := os.ReadDir(filepath.Join(root, "h.example.com", "live"))
+ if len(entries) != 1 {
+ t.Errorf("live/ holds %d entries, want 1", len(entries))
+ }
+ for _, bad := range [][2]string{{"..", "h"}, {"a/b", "h"}, {"x", "../h"}, {"", "h"}} {
+ if err := ensureExe(root, bad[0], "/t", bad[1]); err == nil {
+ t.Errorf("ensureExe(%q, %q) accepted", bad[0], bad[1])
+ }
+ }
+}
+
+// A long host's hashed identifier keys the directory; the link inside is
+// still named for the host, the path the trampoline execs.
+func TestStartWritesTheHostsExeBeforeStarting(t *testing.T) {
+ exe := filepath.Join(t.TempDir(), "exe")
+ ctl := &fakeCtl{}
+ _, c := newServer(t, ctl, func(s *Server) { s.ExeRoot = exe })
+ host := "a-very-long-hostname-that-gets-hashed.bdm.oncarlos.com"
+ resp := post(t, c, "/start?host="+host+"&role=run&memory_max=536870912")
+ if resp.StatusCode != 200 {
+ t.Fatalf("status %d", resp.StatusCode)
+ }
+ iid := provision.InstanceID(host)
+ if iid == host {
+ t.Fatal("fixture host is not long enough to be hashed")
+ }
+ if got, _ := os.Readlink(filepath.Join(exe, iid, "live", host)); got != "../app" {
+ t.Errorf("live/%s -> %q (dir %s)", host, got, iid)
+ }
+}
+
+func TestResolverAllowPolicy(t *testing.T) {
+ conf := []byte("nameserver 172.31.0.2\nnameserver 127.0.0.53\nnameserver 127.0.0.54\nnameserver 127.0.0.1\nnameserver ::1\n" +
+ "nameserver 169.254.169.254\nnameserver 169.254.169.253\nnameserver 169.254.1.1\nnameserver fd00:ec2::253\n" +
+ "nameserver fe80::1%eth0\nnameserver 0.0.0.0\nnameserver 172.31.0.2\noptions edns0\n")
+ got := strings.Join(resolverAllow(conf), " ")
+ if want := "169.254.169.253/32 172.31.0.2/32 fd00:ec2::253/128"; got != want {
+ t.Errorf("allow = %q, want %q", got, want)
+ }
+ // review F1: resolved's stubs are loopback, and an allow on them is
+ // an allow on every wildcard listener of the box.
+ for _, stub := range []string{"127.0.0.53", "127.0.0.54"} {
+ if ok, why := ResolverAllowed(netip.MustParseAddr(stub)); ok || !strings.Contains(why, "upstreams") {
+ t.Errorf("%s allowed=%v %q", stub, ok, why)
+ }
+ }
+}
+
+// Where resolved runs, its upstream list is what instances see (the
+// template binds it over /etc/resolv.conf), so it is what gets opened;
+// without it, the box's own file.
+func TestAllowFollowsTheFileInstancesSee(t *testing.T) {
+ dir := t.TempDir()
+ etc := filepath.Join(dir, "resolv.conf")
+ up := filepath.Join(dir, "upstream.conf")
+ if err := os.WriteFile(etc, []byte("nameserver 127.0.0.53\n"), 0o644); err != nil {
+ t.Fatal(err)
+ }
+ ctl := &fakeCtl{}
+ s := &Server{Systemctl: ctl.run, Logf: t.Logf, ResolvConf: etc, ResolvedConf: up}
+ if err := s.allowResolvers(context.Background()); err != nil {
+ t.Fatal(err)
+ }
+ if got := ctl.joined(); got != "set-property --runtime carlos-instances.slice IPAddressAllow=" {
+ t.Errorf("stub only: %q", got)
+ }
+ if err := os.WriteFile(up, []byte("nameserver 10.0.0.2\n"), 0o644); err != nil {
+ t.Fatal(err)
+ }
+ ctl2 := &fakeCtl{}
+ s.Systemctl = ctl2.run
+ if err := s.allowResolvers(context.Background()); err != nil {
+ t.Fatal(err)
+ }
+ if got := ctl2.joined(); got != "set-property --runtime carlos-instances.slice IPAddressAllow= IPAddressAllow=10.0.0.2/32" {
+ t.Errorf("upstreams: %q", got)
+ }
+}
+
+// The allow is reset and re-written on carlos-instances.slice before the
+// probe unit runs (#214 item 3).
+func TestProbeRunReopensTheResolversFirst(t *testing.T) {
+ dir := t.TempDir()
+ resolv := filepath.Join(dir, "resolv.conf")
+ if err := os.WriteFile(resolv, []byte("nameserver 10.0.0.2\n"), 0o644); err != nil {
+ t.Fatal(err)
+ }
+ ctl := &fakeCtl{}
+ s, _ := newServer(t, ctl, func(s *Server) { s.ResolvConf = resolv; s.ExeRoot = filepath.Join(dir, "exe") })
+ s.RunProbe(context.Background())
+ calls := ctl.joined()
+ want := "set-property --runtime carlos-instances.slice IPAddressAllow= IPAddressAllow=10.0.0.2/32\nstart carlos-instance-probe.service"
+ if !strings.Contains(calls, want) {
+ t.Errorf("calls:\n%s\nwant in order:\n%s", calls, want)
+ }
+ if got, _ := os.Readlink(filepath.Join(dir, "exe", "_probe", "app")); got != SystemBinary {
+ t.Errorf("probe app -> %q", got)
+ }
+}
+
+func TestDiskFloor(t *testing.T) {
+ if f := DiskFloor(8 << 30); f != 1<<30 {
+ t.Errorf("8 GiB: %d", f)
+ }
+ if f := DiskFloor(100 << 30); f != 10<<30 {
+ t.Errorf("100 GiB: %d", f)
+ }
+}
+
+// Below the floor a start and a seed are refused with 507 and no unit is
+// touched (#214 item 1).
+func TestStartAndSeedRefuseBelowTheDiskFloor(t *testing.T) {
+ ctl := &fakeCtl{}
+ _, c := newServer(t, ctl, func(s *Server) {
+ s.Statfs = func(string) (uint64, uint64, error) { return 900 << 20, 8 << 30, nil }
+ })
+ for _, path := range []string{"/start?host=h.example.com&role=run&memory_max=536870912", "/seed?host=h.example.com&memory_max=536870912"} {
+ resp := post(t, c, path)
+ if resp.StatusCode != http.StatusInsufficientStorage {
+ t.Errorf("%s: status %d", path, resp.StatusCode)
+ }
+ }
+ if calls := ctl.joined(); calls != "" {
+ t.Errorf("systemctl called: %s", calls)
+ }
+}
+
+// The guard warns and never stops (review F2): usage cannot be attributed
+// reliably from here, and a stop from here would bypass the activator's
+// fence, so a full disk refuses new starts and leaves awake instances be.
+func TestFullDiskWarnsAndStopsNothing(t *testing.T) {
+ ctl := &fakeCtl{}
+ var logged []string
+ s := &Server{Systemctl: ctl.run, DataRoot: t.TempDir(),
+ Logf: func(f string, a ...any) { logged = append(logged, fmt.Sprintf(f, a...)) },
+ Statfs: func(string) (uint64, uint64, error) { return 100 << 20, 8 << 30, nil }}
+ if !s.warnIfFull() {
+ t.Fatal("no warning below the floor")
+ }
+ if len(logged) != 1 || !strings.Contains(logged[0], "refusing every instance start") {
+ t.Errorf("logged %q", logged)
+ }
+ if calls := ctl.joined(); calls != "" {
+ t.Errorf("systemctl called: %s", calls)
+ }
+ s.Statfs = func(string) (uint64, uint64, error) { return 4 << 30, 8 << 30, nil }
+ if s.warnIfFull() {
+ t.Error("warned with room")
+ }
+}
diff --git a/internal/instancectl/peer_linux.go b/internal/instancectl/peer_linux.go
index 0e4085b5d..8d3cc1e5f 100644
--- a/internal/instancectl/peer_linux.go
+++ b/internal/instancectl/peer_linux.go
@@ -188,3 +188,14 @@ func probeFixturesOK() error {
}
return nil
}
+
+// statfsBytes is the free space an unprivileged writer can use (f_bavail,
+// so root's reserved blocks do not count) and the filesystem's size.
+func statfsBytes(path string) (avail, total uint64, err error) {
+ var st unix.Statfs_t
+ if err := unix.Statfs(path, &st); err != nil {
+ return 0, 0, err
+ }
+ bs := uint64(st.Bsize)
+ return st.Bavail * bs, st.Blocks * bs, nil
+}
diff --git a/internal/instancectl/peer_other.go b/internal/instancectl/peer_other.go
index 17c373f1d..6c82e3409 100644
--- a/internal/instancectl/peer_other.go
+++ b/internal/instancectl/peer_other.go
@@ -13,3 +13,4 @@ func productionPeerOK() func(net.Conn) error {
func activatedListener() (net.Listener, bool, error) { return nil, false, errors.ErrUnsupported }
func bindControlSocket(string) (net.Listener, error) { return nil, errors.ErrUnsupported }
func probeFixturesOK() error { return errors.ErrUnsupported }
+func statfsBytes(string) (uint64, uint64, error) { return 0, 0, errors.ErrUnsupported }
diff --git a/internal/instancectl/server.go b/internal/instancectl/server.go
index 580cbfe86..379f6d02d 100644
--- a/internal/instancectl/server.go
+++ b/internal/instancectl/server.go
@@ -51,6 +51,21 @@ type Server struct {
ProbeEvery time.Duration
Logf func(string, ...any)
+ // ExeRoot is where ensureExe lays out each instance's view of its own
+ // build (#214 item 2); empty skips it, for tests of other verbs.
+ ExeRoot string
+ // ResolvConf is the box's resolv.conf and ResolvedConf resolved's
+ // upstream list; whichever an instance sees (resolverFile) has its
+ // nameservers re-opened through the egress fence before every probe
+ // (#214 item 3).
+ ResolvConf string
+ ResolvedConf string
+ // DataRoot is the instance data tree, and Statfs reports its free and
+ // total bytes; a nil Statfs skips the disk floors (#214 item 1).
+ DataRoot string
+ Statfs func(path string) (avail, total uint64, err error)
+ DiskEvery time.Duration
+
// cgroupWaitBudget bounds a wait for an empty cgroup; tests shorten it.
cgroupWaitBudget time.Duration
@@ -78,12 +93,18 @@ func Run(args []string) {
out, err := exec.CommandContext(ctx, "systemctl", args...).CombinedOutput()
return string(out), err
},
- PeerOK: productionPeerOK(),
- CgroupRoot: "/sys/fs/cgroup",
- ProbeOut: filepath.Join(controlsock.Dir, "probe.out"),
- FixtureOK: probeFixturesOK,
- ProbeEvery: 5 * time.Minute,
- Logf: log.Printf,
+ PeerOK: productionPeerOK(),
+ CgroupRoot: "/sys/fs/cgroup",
+ ProbeOut: filepath.Join(controlsock.Dir, "probe.out"),
+ FixtureOK: probeFixturesOK,
+ ProbeEvery: 5 * time.Minute,
+ Logf: log.Printf,
+ ExeRoot: filepath.Join(controlsock.Dir, "exe"),
+ ResolvConf: "/etc/resolv.conf",
+ ResolvedConf: ResolvedConf,
+ DataRoot: "/data/instances",
+ Statfs: statfsBytes,
+ DiskEvery: 30 * time.Second,
}
ln, activated, err := activatedListener()
if err != nil {
@@ -100,6 +121,7 @@ func Run(args []string) {
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
go s.probeLoop(ctx)
+ go s.diskLoop(ctx)
if err := s.Serve(ctx, ln); err != nil {
log.Fatalf("instance-control: %v", err)
}
@@ -237,11 +259,17 @@ func parseRequest(r *http.Request, roles []Role, wantTier bool) (request, error)
return req, nil
}
+// setTier writes the whole tier onto the host slice in one call: the
+// memory caps, and the CPU and IO shares that follow from the same tier
+// (#214 item 1). Seed, instance and sidecar share all of it.
func (s *Server) setTier(ctx context.Context, host string, tier cgroup.Limit) error {
max := strconv.FormatInt(tier.MemoryMax, 10)
high := strconv.FormatInt(cgroup.HighOf(tier.MemoryMax), 10)
+ sh := cgroup.SharesOf(tier)
_, err := s.Systemctl(ctx, "set-property", "--runtime", SliceName(host),
- "MemoryMax="+max, "MemoryHigh="+high, "MemorySwapMax="+max)
+ "MemoryMax="+max, "MemoryHigh="+high, "MemorySwapMax="+max,
+ "CPUWeight="+strconv.Itoa(sh.CPUWeight), "CPUQuota="+strconv.Itoa(sh.CPUQuotaPercent)+"%",
+ "IOWeight="+strconv.Itoa(sh.IOWeight))
return err
}
@@ -259,6 +287,9 @@ func (s *Server) handleStart(w http.ResponseWriter, r *http.Request) {
}
ctx := r.Context()
unit := UnitName(req.role, req.host)
+ if s.refuseStart(w, req.host, unit) {
+ return
+ }
if err := s.setTier(ctx, req.host, req.tier); err != nil {
http.Error(w, "set-property: "+err.Error(), http.StatusBadGateway)
return
@@ -317,6 +348,9 @@ func (s *Server) handleSeed(w http.ResponseWriter, r *http.Request) {
}
ctx := r.Context()
unit := UnitName(RoleSeed, req.host)
+ if s.refuseStart(w, req.host, unit) {
+ return
+ }
if err := s.setTier(ctx, req.host, req.tier); err != nil {
http.Error(w, "set-property: "+err.Error(), http.StatusBadGateway)
return
@@ -595,6 +629,21 @@ func (s *Server) RunProbe(ctx context.Context) {
set(ProbeReport{Lines: []VerdictLine{{Check: "fixture", Verdict: "fail", Detail: err.Error()}}})
return
}
+ // The probe binds the same /opt view an instance does, so its sources
+ // must exist (#214 item 2), and the resolver allow it checks must be
+ // in place (#214 item 3).
+ if s.ExeRoot != "" {
+ if err := ensureExe(s.ExeRoot, ProbeDirName, SystemBinary, ProbeDirName); err != nil {
+ set(ProbeReport{Lines: []VerdictLine{{Check: "probe-exe", Verdict: "fail", Detail: err.Error()}}})
+ return
+ }
+ }
+ if s.ResolvConf != "" {
+ if err := s.allowResolvers(ctx); err != nil {
+ set(ProbeReport{Lines: []VerdictLine{{Check: "resolver-allow", Verdict: "fail", Detail: err.Error()}}})
+ return
+ }
+ }
if out, err := s.Systemctl(ctx, "start", probeUnit); err != nil {
set(ProbeReport{Lines: []VerdictLine{{Check: "probe-unit", Verdict: "fail", Detail: strings.TrimSpace(out + " " + err.Error())}}})
return
diff --git a/internal/instancectl/server_test.go b/internal/instancectl/server_test.go
index 0ed37a3f6..13bc53db1 100644
--- a/internal/instancectl/server_test.go
+++ b/internal/instancectl/server_test.go
@@ -137,7 +137,7 @@ func TestStartSetsTheTierThenStartsThenReadsTheInvocation(t *testing.T) {
if err := json.NewDecoder(resp.Body).Decode(&out); err != nil || out.InvocationID != "abc123" {
t.Errorf("response = %+v, %v", out, err)
}
- want := "set-property --runtime carlos-instances-bill\\x2d2.example.com.slice MemoryMax=536870912 MemoryHigh=483183819 MemorySwapMax=536870912\n" +
+ want := "set-property --runtime carlos-instances-bill\\x2d2.example.com.slice MemoryMax=536870912 MemoryHigh=483183819 MemorySwapMax=536870912 CPUWeight=100 CPUQuota=100% IOWeight=100\n" +
"start carlos-instance@bill\\x2d2.example.com.service\n" +
"show -p InvocationID --value carlos-instance@bill\\x2d2.example.com.service"
if got := ctl.joined(); got != want {
@@ -172,7 +172,7 @@ func TestStartReportsSystemctlFailureAs502(t *testing.T) {
if resp.StatusCode != 502 {
t.Errorf("status %d, want 502", resp.StatusCode)
}
- if !strings.Contains(ctl.joined(), "MemoryMax=1073741824 MemoryHigh=966367638 MemorySwapMax=1073741824") {
+ if !strings.Contains(ctl.joined(), "MemoryMax=1073741824 MemoryHigh=966367638 MemorySwapMax=1073741824 CPUWeight=100 CPUQuota=200% IOWeight=100") {
t.Errorf("children tier not applied: %s", ctl.joined())
}
}
diff --git a/internal/instanceexec/exec.go b/internal/instanceexec/exec.go
index 1778ec16e..ad72521d7 100644
--- a/internal/instanceexec/exec.go
+++ b/internal/instanceexec/exec.go
@@ -15,7 +15,6 @@ import (
"os"
"path/filepath"
"strings"
- "syscall"
"github.com/carlosframework/platform/internal/cliflag"
"github.com/carlosframework/platform/internal/instancectl"
@@ -52,7 +51,8 @@ func RunExec(args []string) {
if !ok {
log.Fatalf("instance-exec: role %q is not run, seed or sidecar", fs.Arg(0))
}
- if err := execLaunch(dataDir(), role, fs.Arg(1), syscall.Exec); err != nil {
+ // The exec is scoped (#178): see ScopeAbstractUnix.
+ if err := execLaunch(dataDir(), role, fs.Arg(1), scopedExec); err != nil {
log.Fatalf("instance-exec: %v", err)
}
}
diff --git a/internal/instanceexec/probe.go b/internal/instanceexec/probe.go
index 05bd909d4..6317577b2 100644
--- a/internal/instanceexec/probe.go
+++ b/internal/instanceexec/probe.go
@@ -8,6 +8,7 @@ import (
"flag"
"fmt"
"net"
+ "net/netip"
"os"
"path/filepath"
"sort"
@@ -15,6 +16,7 @@ import (
"syscall"
"time"
+ "github.com/carlosframework/platform/internal/cgroup"
"github.com/carlosframework/platform/internal/cliflag"
"github.com/carlosframework/platform/internal/controlsock"
"github.com/carlosframework/platform/internal/instancectl"
@@ -51,6 +53,26 @@ type probeEnv struct {
// nosymfollow mount, so the real check would fail there for a reason
// unrelated to the property under test in the other cases.
symlinkCheck func(dir string) error
+
+ // The hardening of #214 and #178.
+ optRoot string // /opt/carlos, which must hold only this unit's own build
+ resolvConf string // /etc/resolv.conf, whose resolvers must be reachable
+ devMqueue string // /dev/mqueue, which must not be the host's
+ // ipcInode is this process's IPC namespace inode; the box's own is
+ // initIPCInode.
+ ipcInode func() (uint64, error)
+ // abstractScoped proves the trampoline's Landlock scope on this
+ // kernel: nil when a scoped connect to an outside abstract name was
+ // refused with EPERM.
+ abstractScoped func() error
+ // selfCgroup is this unit's own cgroup directory, where the probe
+ // unit's CPUQuota=, CPUWeight= and IOWeight= must have landed.
+ selfCgroup func() (string, error)
+ // ioEnforced says whether io.weight has a controller behind it.
+ ioEnforced func(selfCgroup string) bool
+ // localAddrs are this box's interface addresses (the network
+ // namespace is the box's).
+ localAddrs func() ([]netip.Addr, error)
}
// cgroupMountpoint is the fixed system path the cgroup-ro-mount check
@@ -96,9 +118,45 @@ func productionProbeEnv() probeEnv {
mountNoSym: instancefs.MountNoSymfollow,
mountOption: instancefs.MountOptionOf,
symlinkCheck: realSymlinkCheck,
+
+ optRoot: "/opt/carlos", resolvConf: "/etc/resolv.conf", devMqueue: "/dev/mqueue",
+ ipcInode: ipcNamespaceInode,
+ abstractScoped: realAbstractCheck,
+ selfCgroup: ownCgroupDir,
+ ioEnforced: ioWeightEnforced,
+ localAddrs: interfaceAddrs,
}
}
+// interfaceAddrs lists the box's addresses, zones and prefixes dropped.
+func interfaceAddrs() ([]netip.Addr, error) {
+ addrs, err := net.InterfaceAddrs()
+ if err != nil {
+ return nil, err
+ }
+ out := make([]netip.Addr, 0, len(addrs))
+ for _, a := range addrs {
+ if p, err := netip.ParsePrefix(a.String()); err == nil {
+ out = append(out, p.Addr().WithZone("").Unmap())
+ }
+ }
+ return out, nil
+}
+
+// ownCgroupDir is /sys/fs/cgroup plus this process's unified-hierarchy
+// path from /proc/self/cgroup.
+func ownCgroupDir() (string, error) {
+ b, err := os.ReadFile("/proc/self/cgroup")
+ if err != nil {
+ return "", err
+ }
+ rel, ok := cgroup.ParseSelfCgroup(string(b))
+ if !ok {
+ return "", errors.New("no cgroup v2 path in /proc/self/cgroup")
+ }
+ return filepath.Join(cgroupMountpoint, rel), nil
+}
+
func listNames(dir string) ([]string, error) {
entries, err := os.ReadDir(dir)
if err != nil {
@@ -228,11 +286,11 @@ func runChecks(e probeEnv) instancectl.ProbeReport {
default:
add("imds-v6", "fail", fmt.Sprintf("sendto = %v", v6))
}
- lo := e.sendUDP("udp4", "127.0.0.1:9")
- pass("loopback", fmt.Sprintf("sendto = %v", lo), lo == nil)
tcp := e.connectTCP("169.254.169.254:80", 2*time.Second)
pass("imds-tcp", fmt.Sprintf("connect = %v", tcp), tcp != nil)
+ hardeningChecks(e, add, pass)
+
r.Ready = true
for _, l := range r.Lines {
if l.Verdict == "fail" {
diff --git a/internal/instanceexec/probe_hardening.go b/internal/instanceexec/probe_hardening.go
new file mode 100644
index 000000000..438b8f110
--- /dev/null
+++ b/internal/instanceexec/probe_hardening.go
@@ -0,0 +1,193 @@
+package instanceexec
+
+import (
+ "errors"
+ "fmt"
+ "net"
+ "net/netip"
+ "os"
+ "path/filepath"
+ "strings"
+ "syscall"
+ "time"
+
+ "github.com/carlosframework/platform/internal/instancectl"
+)
+
+// What the probe unit's own resource section sets (the units test pins
+// the unit side), read back from its cgroup by the cpu-limits and
+// io-weight lines.
+const (
+ probeCPUMax = "50000 100000" // CPUQuota=50%
+ probeCPUWeight = "50"
+ probeIOWeight = "default 50"
+)
+
+// resolverProbePort is the non-DNS port the resolver-ports line dials on
+// each allowed resolver: ssh, the listener a box is most likely to have
+// bound to all addresses.
+const resolverProbePort = 22
+
+// Addresses the egress fence must refuse (#214 item 3), one per range,
+// none of them anything a box would bind. A send to each is answered by
+// the cgroup filter with EPERM before a packet leaves.
+var (
+ privateV4 = []string{"10.255.255.254", "172.31.255.254", "192.168.255.254", "100.127.255.254", "169.254.0.1", "198.18.0.1", "192.0.0.1"}
+ // v6 has no route on most boxes; ENETUNREACH there is unverified.
+ privateV6 = []string{"fd00::1", "fec0::1", "64:ff9b::a00:1"}
+)
+
+// hardeningChecks are the verdict lines for #214 items 1-3 and #178.
+func hardeningChecks(e probeEnv, add func(check, verdict, detail string), pass func(check, detail string, ok bool)) {
+ // Item 2: /opt/carlos holds the platform binary and this unit's own
+ // build, and no version store.
+ o1, ok1 := exactly(e.optRoot, "app", "carlos", "live")
+ o2, ok2 := exactly(filepath.Join(e.optRoot, "live"), instancectl.ProbeDirName)
+ pass("opt-own-build", o1+o2, ok1 && ok2)
+
+ // Item 3: the box's resolvers are reachable through the fence. This
+ // is also the live-stack control the EPERM lines below rest on.
+ resolvers := map[string]bool{}
+ var allowed []netip.Addr
+ var dnsFail []string
+ if b, err := os.ReadFile(e.resolvConf); err != nil {
+ dnsFail = append(dnsFail, err.Error())
+ } else if addrs := instancectl.ParseResolvers(b); len(addrs) == 0 {
+ dnsFail = append(dnsFail, "no nameserver in "+e.resolvConf)
+ } else {
+ for _, a := range addrs {
+ resolvers[a.String()] = true
+ if ok, why := instancectl.ResolverAllowed(a); !ok {
+ dnsFail = append(dnsFail, a.String()+": "+why)
+ continue
+ }
+ allowed = append(allowed, a)
+ network := "udp4"
+ if a.Is6() {
+ network = "udp6"
+ }
+ if err := e.sendUDP(network, netip.AddrPortFrom(a, 53).String()); err != nil {
+ dnsFail = append(dnsFail, fmt.Sprintf("%s: sendto = %v", a, err))
+ }
+ }
+ }
+ pass("dns", strings.Join(dnsFail, "; "), len(dnsFail) == 0)
+
+ // Item 3: an allowed resolver opens only a resolver. The allow is per
+ // address, both ways and every port, so it must not be an address of
+ // this box, where a tenant could reach a listener bound to all
+ // addresses, and a non-DNS port on it must not answer.
+ var portFail []string
+ local, lerr := e.localAddrs()
+ if lerr != nil {
+ portFail = append(portFail, "interface addresses: "+lerr.Error())
+ }
+ for _, a := range allowed {
+ for _, l := range local {
+ if l == a {
+ portFail = append(portFail, a.String()+" is an address of this box")
+ }
+ }
+ if err := e.connectTCP(netip.AddrPortFrom(a, resolverProbePort).String(), 2*time.Second); err == nil {
+ portFail = append(portFail, fmt.Sprintf("%s: port %d answered", a, resolverProbePort))
+ }
+ }
+ pass("resolver-ports", strings.Join(portFail, "; "), len(portFail) == 0)
+
+ // Item 3: loopback is closed.
+ lo := e.sendUDP("udp4", "127.0.0.1:9")
+ pass("loopback-v4", fmt.Sprintf("sendto = %v", lo), errors.Is(lo, syscall.EPERM))
+ lo6 := e.sendUDP("udp6", "[::1]:9")
+ switch {
+ case errors.Is(lo6, syscall.EPERM):
+ add("loopback-v6", "ok", "")
+ case noV6(lo6):
+ add("loopback-v6", "unverified", fmt.Sprintf("no IPv6 on this box (%v); deny not exercised", lo6))
+ default:
+ add("loopback-v6", "fail", fmt.Sprintf("sendto = %v", lo6))
+ }
+
+ // Item 3: private and cloud-internal ranges are closed, except an
+ // address the box resolves through, which is open on purpose.
+ var v4Fail []string
+ for _, a := range privateV4 {
+ if resolvers[a] {
+ continue
+ }
+ if err := e.sendUDP("udp4", net.JoinHostPort(a, "9")); !errors.Is(err, syscall.EPERM) {
+ v4Fail = append(v4Fail, fmt.Sprintf("%s: sendto = %v", a, err))
+ }
+ }
+ pass("private-v4", strings.Join(v4Fail, "; "), len(v4Fail) == 0)
+ var v6Fail, v6Unrouted []string
+ for _, a := range privateV6 {
+ if resolvers[a] {
+ continue
+ }
+ err := e.sendUDP("udp6", net.JoinHostPort(a, "9"))
+ switch {
+ case errors.Is(err, syscall.EPERM):
+ case noV6(err):
+ v6Unrouted = append(v6Unrouted, a)
+ default:
+ v6Fail = append(v6Fail, fmt.Sprintf("%s: sendto = %v", a, err))
+ }
+ }
+ switch {
+ case len(v6Fail) > 0:
+ add("private-v6", "fail", strings.Join(v6Fail, "; "))
+ case len(v6Unrouted) > 0:
+ add("private-v6", "unverified", "no route to "+strings.Join(v6Unrouted, ", ")+"; deny not exercised")
+ default:
+ add("private-v6", "ok", "")
+ }
+
+ // #178: an IPC namespace of its own, and no host mqueue mount.
+ ino, ierr := e.ipcInode()
+ m, mok := exactly(e.devMqueue)
+ pass("ipc-private", fmt.Sprintf("ipc ns inode=%d err=%v %s", ino, ierr, m), ierr == nil && ino != initIPCInode && mok)
+
+ // #178: the trampoline's abstract socket scope holds on this kernel.
+ aerr := e.abstractScoped()
+ pass("abstract-unix", fmt.Sprint(aerr), aerr == nil)
+
+ // Item 1: CPU settings land on a unit under carlos-instances.slice.
+ cg, cerr := e.selfCgroup()
+ if cerr != nil {
+ add("cpu-limits", "fail", cerr.Error())
+ add("io-weight", "fail", cerr.Error())
+ return
+ }
+ cpuMax := readTrim(filepath.Join(cg, "cpu.max"))
+ cpuWeight := readTrim(filepath.Join(cg, "cpu.weight"))
+ pass("cpu-limits", fmt.Sprintf("cpu.max=%q cpu.weight=%q", cpuMax, cpuWeight), cpuMax == probeCPUMax && cpuWeight == probeCPUWeight)
+
+ // Item 1: the IO weight lands too, and says whether anything on this
+ // box enforces it.
+ io, ioErr := os.ReadFile(filepath.Join(cg, "io.weight"))
+ ioFirst, _, _ := strings.Cut(strings.TrimSpace(string(io)), "\n")
+ switch {
+ case ioErr != nil:
+ add("io-weight", "unverified", "no io.weight here ("+ioErr.Error()+"); IO is not weighted")
+ case ioFirst != probeIOWeight:
+ add("io-weight", "fail", fmt.Sprintf("io.weight=%q, want %q", ioFirst, probeIOWeight))
+ case !e.ioEnforced(cg):
+ add("io-weight", "unverified", "weights are set, but no disk here has io.cost or BFQ to enforce them")
+ default:
+ add("io-weight", "ok", "")
+ }
+}
+
+// noV6 is an answer that means the box has no IPv6 path for that
+// address, so the fence was never asked.
+func noV6(err error) bool {
+ return errors.Is(err, syscall.ENETUNREACH) || errors.Is(err, syscall.EADDRNOTAVAIL) || errors.Is(err, syscall.EAFNOSUPPORT)
+}
+
+func readTrim(path string) string {
+ b, err := os.ReadFile(path)
+ if err != nil {
+ return "<" + err.Error() + ">"
+ }
+ return strings.TrimSpace(string(b))
+}
diff --git a/internal/instanceexec/probe_hardening_test.go b/internal/instanceexec/probe_hardening_test.go
new file mode 100644
index 000000000..0f83241a2
--- /dev/null
+++ b/internal/instanceexec/probe_hardening_test.go
@@ -0,0 +1,221 @@
+//go:build linux
+
+package instanceexec
+
+import (
+ "errors"
+ "net"
+ "net/netip"
+ "os"
+ "path/filepath"
+ "runtime"
+ "syscall"
+ "testing"
+ "time"
+)
+
+// The probe refuses readiness for each property the hardening adds (#214
+// items 1-3, #178), naming the line.
+func TestProbeFailsOnEachMissingHardening(t *testing.T) {
+ cases := map[string]func(t *testing.T, e *probeEnv){
+ "opt-own-build": func(t *testing.T, e *probeEnv) {
+ if err := os.MkdirAll(filepath.Join(e.optRoot, "versions"), 0o755); err != nil {
+ t.Fatal(err)
+ }
+ },
+ "dns": func(t *testing.T, e *probeEnv) {
+ e.sendUDP = func(string, string) error { return syscall.EPERM }
+ },
+ "loopback-v4": func(t *testing.T, e *probeEnv) {
+ e.sendUDP = func(_, addr string) error {
+ if addr == "127.0.0.1:9" || addr == "10.0.0.2:53" {
+ return nil
+ }
+ return syscall.EPERM
+ }
+ },
+ "private-v4": func(t *testing.T, e *probeEnv) {
+ e.sendUDP = func(_, addr string) error {
+ if addr == "172.31.255.254:9" || addr == "10.0.0.2:53" {
+ return nil
+ }
+ return syscall.EPERM
+ }
+ },
+ "private-v6": func(t *testing.T, e *probeEnv) {
+ e.sendUDP = func(_, addr string) error {
+ if addr == "[fd00::1]:9" || addr == "10.0.0.2:53" {
+ return nil
+ }
+ return syscall.EPERM
+ }
+ },
+ "ipc-private": func(t *testing.T, e *probeEnv) {
+ e.ipcInode = func() (uint64, error) { return initIPCInode, nil }
+ },
+ "abstract-unix": func(t *testing.T, e *probeEnv) {
+ e.abstractScoped = func() error { return errAbstractReached }
+ },
+ "cpu-limits": func(t *testing.T, e *probeEnv) {
+ cg, _ := e.selfCgroup()
+ if err := os.WriteFile(filepath.Join(cg, "cpu.max"), []byte("max 100000\n"), 0o644); err != nil {
+ t.Fatal(err)
+ }
+ },
+ "io-weight": func(t *testing.T, e *probeEnv) {
+ cg, _ := e.selfCgroup()
+ if err := os.WriteFile(filepath.Join(cg, "io.weight"), []byte("default 100\n"), 0o644); err != nil {
+ t.Fatal(err)
+ }
+ },
+ }
+ for check, breakIt := range cases {
+ t.Run(check, func(t *testing.T) {
+ e := goodEnv(t)
+ breakIt(t, &e)
+ r := runChecks(e)
+ if r.Ready || verdict(r, check) != "fail" {
+ t.Errorf("%s broken yet ready=%v verdict=%s", check, r.Ready, verdict(r, check))
+ }
+ })
+ }
+}
+
+// A host mqueue visible in the namespace fails ipc-private even with a
+// private namespace inode.
+func TestProbeFailsOnAVisibleHostMqueue(t *testing.T) {
+ e := goodEnv(t)
+ if err := os.WriteFile(filepath.Join(e.devMqueue, "q"), nil, 0o600); err != nil {
+ t.Fatal(err)
+ }
+ if r := runChecks(e); r.Ready || verdict(r, "ipc-private") != "fail" {
+ t.Errorf("host mqueue visible yet ready=%v", r.Ready)
+ }
+}
+
+// A loopback resolver other than resolved's stub is refused by policy,
+// and the probe names it rather than passing on a reachable address.
+func TestProbeRefusesALoopbackResolver(t *testing.T) {
+ e := goodEnv(t)
+ if err := os.WriteFile(e.resolvConf, []byte("nameserver 127.0.0.1\n"), 0o644); err != nil {
+ t.Fatal(err)
+ }
+ e.sendUDP = func(string, string) error { return syscall.EPERM }
+ if r := runChecks(e); r.Ready || verdict(r, "dns") != "fail" {
+ t.Errorf("loopback resolver yet ready=%v", r.Ready)
+ }
+}
+
+// Unverified, not failed: no IPv6 on the box, and IO weights with no
+// controller to enforce them.
+func TestProbeUnverifiedLinesDoNotBlock(t *testing.T) {
+ e := goodEnv(t)
+ e.sendUDP = func(network, addr string) error {
+ if addr == "10.0.0.2:53" {
+ return nil
+ }
+ if network == "udp6" {
+ return syscall.ENETUNREACH
+ }
+ return syscall.EPERM
+ }
+ e.ioEnforced = func(string) bool { return false }
+ r := runChecks(e)
+ if !r.Ready {
+ t.Fatalf("not ready: %+v", r.Lines)
+ }
+ for _, c := range []string{"loopback-v6", "private-v6", "io-weight"} {
+ if v := verdict(r, c); v != "unverified" {
+ t.Errorf("%s = %s, want unverified", c, v)
+ }
+ }
+}
+
+// The real Landlock scope on this kernel (#178): a scoped thread cannot
+// connect to an abstract name bound outside its domain, while the
+// unscoped process still can. Skipped where the kernel lacks ABI 6.
+func TestScopeAbstractUnixRefusesAnOutsideName(t *testing.T) {
+ name := "@carlos-scope-test-" + randomName()
+ ln, err := net.Listen("unix", name)
+ if err != nil {
+ t.Skipf("abstract listen: %v", err)
+ }
+ defer ln.Close()
+ go func() {
+ for {
+ c, err := ln.Accept()
+ if err != nil {
+ return
+ }
+ c.Close()
+ }
+ }()
+ res := make(chan error, 1)
+ go func() {
+ runtime.LockOSThread() // never unlocked: the thread dies scoped
+ if err := ScopeAbstractUnix(); err != nil {
+ res <- err
+ return
+ }
+ c, err := net.Dial("unix", name)
+ if err == nil {
+ c.Close()
+ res <- errAbstractReached
+ return
+ }
+ res <- unwrapErrno(err)
+ }()
+ err = <-res
+ if err != nil && !errors.Is(err, syscall.EPERM) && !errors.Is(err, errAbstractReached) {
+ t.Skipf("no Landlock scope here: %v", err)
+ }
+ if !errors.Is(err, syscall.EPERM) {
+ t.Fatalf("scoped connect = %v, want EPERM", err)
+ }
+ c, err := net.Dial("unix", name)
+ if err != nil {
+ t.Fatalf("unscoped connect after the scoped thread: %v", err)
+ }
+ c.Close()
+ if err := realAbstractCheck(); err != nil {
+ t.Errorf("realAbstractCheck = %v", err)
+ }
+}
+
+// An allowed resolver must open only a resolver (#214 item 3, review F1):
+// one that is an address of this box, or that answers on a non-DNS port,
+// fails resolver-ports.
+func TestProbeResolverPortsLine(t *testing.T) {
+ e := goodEnv(t)
+ if r := runChecks(e); verdict(r, "resolver-ports") != "ok" {
+ t.Fatalf("good fixture: resolver-ports = %s", verdict(r, "resolver-ports"))
+ }
+ e.localAddrs = func() ([]netip.Addr, error) { return []netip.Addr{netip.MustParseAddr("10.0.0.2")}, nil }
+ if r := runChecks(e); r.Ready || verdict(r, "resolver-ports") != "fail" {
+ t.Errorf("resolver on a box address yet ready=%v", r.Ready)
+ }
+ e = goodEnv(t)
+ e.connectTCP = func(addr string, _ time.Duration) error {
+ if addr == "10.0.0.2:22" {
+ return nil
+ }
+ return syscall.ETIMEDOUT
+ }
+ if r := runChecks(e); r.Ready || verdict(r, "resolver-ports") != "fail" {
+ t.Errorf("resolver answering on 22 yet ready=%v", r.Ready)
+ }
+}
+
+// resolved's stub is never a resolver an instance may use: the namespace
+// should have replaced it with the upstream list, and if it did not, dns
+// fails with the reason.
+func TestProbeRefusesTheResolvedStub(t *testing.T) {
+ e := goodEnv(t)
+ if err := os.WriteFile(e.resolvConf, []byte("nameserver 127.0.0.53\noptions edns0 trust-ad\n"), 0o644); err != nil {
+ t.Fatal(err)
+ }
+ e.sendUDP = func(string, string) error { return nil }
+ if r := runChecks(e); r.Ready || verdict(r, "dns") != "fail" {
+ t.Errorf("stub resolver yet ready=%v", r.Ready)
+ }
+}
diff --git a/internal/instanceexec/probe_linux.go b/internal/instanceexec/probe_linux.go
index 7083f6cf9..9f7716e7d 100644
--- a/internal/instanceexec/probe_linux.go
+++ b/internal/instanceexec/probe_linux.go
@@ -32,7 +32,7 @@ func sendOneDatagram(network, addr string) error {
}
func connectWithTimeout(addr string, timeout time.Duration) error {
- c, err := net.DialTimeout("tcp4", addr, timeout)
+ c, err := net.DialTimeout("tcp", addr, timeout)
if err != nil {
return unwrapErrno(err)
}
diff --git a/internal/instanceexec/probe_test.go b/internal/instanceexec/probe_test.go
index d8b28a9e7..cef018748 100644
--- a/internal/instanceexec/probe_test.go
+++ b/internal/instanceexec/probe_test.go
@@ -5,6 +5,7 @@ package instanceexec
import (
"errors"
"net"
+ "net/netip"
"os"
"path/filepath"
"strings"
@@ -39,6 +40,27 @@ func goodEnv(t *testing.T) probeEnv {
proc := mk("proc")
mk("proc/1")
mk("proc/self")
+ opt := mk("opt/carlos")
+ mk("opt/carlos/live")
+ for _, f := range []string{"opt/carlos/app", "opt/carlos/carlos"} {
+ if err := os.WriteFile(filepath.Join(root, f), nil, 0o755); err != nil {
+ t.Fatal(err)
+ }
+ }
+ if err := os.Symlink("../app", filepath.Join(opt, "live", "_probe")); err != nil {
+ t.Fatal(err)
+ }
+ resolv := filepath.Join(root, "resolv.conf")
+ if err := os.WriteFile(resolv, []byte("# fixture\nnameserver 10.0.0.2\nsearch eu-west-1.compute.internal\n"), 0o644); err != nil {
+ t.Fatal(err)
+ }
+ mqueue := mk("dev/mqueue")
+ cg := mk("cgroup/self")
+ for name, v := range map[string]string{"cpu.max": "50000 100000\n", "cpu.weight": "50\n", "io.weight": "default 50\n"} {
+ if err := os.WriteFile(filepath.Join(cg, name), []byte(v), 0o644); err != nil {
+ t.Fatal(err)
+ }
+ }
return probeEnv{
dataRoot: data, runRoot: run, varCache: varCache, varLib: varLib, etcCarlos: etc, devShm: shm, procRoot: proc,
controlDir: filepath.Join(root, "run", "carlos-control"), // deliberately absent: ReadDir and dial fail
@@ -46,17 +68,26 @@ func goodEnv(t *testing.T) probeEnv {
cgroupProcs: filepath.Join(root, "sys", "cgroup.procs"),
pid: func() int { return 1 },
writable: func(string) error { return syscall.EROFS },
+ // The fence: everything is refused but the box's resolver.
sendUDP: func(network, addr string) error {
- switch addr {
- case "169.254.169.254:80", "[fd00:ec2::254]:80":
- return syscall.EPERM
+ if addr == "10.0.0.2:53" {
+ return nil
}
- return nil
+ return syscall.EPERM
},
connectTCP: func(string, time.Duration) error { return syscall.ETIMEDOUT },
mountNoSym: func(string) (bool, error) { return true, nil },
mountOption: func(string, string) (bool, error) { return true, nil },
symlinkCheck: func(string) error { return nil },
+
+ optRoot: opt, resolvConf: resolv, devMqueue: mqueue,
+ ipcInode: func() (uint64, error) { return 4026532412, nil },
+ abstractScoped: func() error { return nil },
+ selfCgroup: func() (string, error) { return cg, nil },
+ ioEnforced: func(string) bool { return true },
+ localAddrs: func() ([]netip.Addr, error) {
+ return []netip.Addr{netip.MustParseAddr("127.0.0.1"), netip.MustParseAddr("10.0.3.17")}, nil
+ },
}
}
@@ -74,7 +105,8 @@ func TestProbePassesOnAnIsolatedNamespace(t *testing.T) {
if !r.Ready {
t.Fatalf("not ready: %+v", r.Lines)
}
- for _, c := range []string{"data-tree", "run-tree", "var-empty", "pid1", "cgroup-ro", "cgroup-ro-mount", "control-unreachable", "etc-carlos", "shm-private", "symlink-eloop", "nosymfollow", "imds-v4", "imds-v6", "loopback", "imds-tcp"} {
+ for _, c := range []string{"data-tree", "run-tree", "var-empty", "pid1", "cgroup-ro", "cgroup-ro-mount", "control-unreachable", "etc-carlos", "shm-private", "symlink-eloop", "nosymfollow", "imds-v4", "imds-v6", "imds-tcp",
+ "opt-own-build", "dns", "resolver-ports", "loopback-v4", "loopback-v6", "private-v4", "private-v6", "ipc-private", "abstract-unix", "cpu-limits", "io-weight"} {
if v := verdict(r, c); v != "ok" {
t.Errorf("%s = %s", c, v)
}
@@ -275,10 +307,10 @@ func TestProbeIMDSVerdicts(t *testing.T) {
if addr == "[fd00:ec2::254]:80" {
return syscall.ENETUNREACH
}
- if addr == "169.254.169.254:80" {
- return syscall.EPERM
+ if addr == "10.0.0.2:53" {
+ return nil
}
- return nil
+ return syscall.EPERM
}
if r := runChecks(e); !r.Ready || verdict(r, "imds-v6") != "unverified" {
t.Errorf("v6 no route: ready=%v v6=%s", r.Ready, verdict(r, "imds-v6"))
@@ -289,17 +321,6 @@ func TestProbeIMDSVerdicts(t *testing.T) {
if r := runChecks(e); r.Ready || verdict(r, "imds-v4") != "fail" {
t.Errorf("v4 success: ready=%v", r.Ready)
}
- // A dead loopback means the stack, not the filter, said no.
- e = goodEnv(t)
- e.sendUDP = func(_, addr string) error {
- if addr == "127.0.0.1:9" {
- return syscall.ENETUNREACH
- }
- return syscall.EPERM
- }
- if r := runChecks(e); r.Ready || verdict(r, "loopback") != "fail" {
- t.Errorf("dead loopback: ready=%v", r.Ready)
- }
// A TCP connect that succeeds is a failure.
e = goodEnv(t)
e.connectTCP = func(string, time.Duration) error { return nil }
diff --git a/internal/instanceexec/scope_linux.go b/internal/instanceexec/scope_linux.go
new file mode 100644
index 000000000..557a8f303
--- /dev/null
+++ b/internal/instanceexec/scope_linux.go
@@ -0,0 +1,146 @@
+//go:build linux
+
+package instanceexec
+
+import (
+ "errors"
+ "fmt"
+ "net"
+ "os"
+ "runtime"
+ "strings"
+ "syscall"
+ "unsafe"
+
+ "golang.org/x/sys/unix"
+)
+
+// landlockScopeABI is the first Landlock ABI with scopes (Linux 6.12).
+const landlockScopeABI = 6
+
+// ScopeAbstractUnix confines the CALLING THREAD, and everything it execs,
+// to abstract unix sockets bound inside its own Landlock domain (#178).
+// Abstract names live in the network namespace, which every instance
+// shares with the box, and RestrictAddressFamilies= cannot tell them from
+// path sockets; LANDLOCK_SCOPE_ABSTRACT_UNIX_SOCKET can. After this, a
+// connect to a name a sibling bound answers EPERM. The app's own
+// processes, all descended from the restricted thread, still reach each
+// other's.
+//
+// Landlock restricts a thread, not a process, so the caller must hold
+// runtime.LockOSThread and either exec or let the thread die locked.
+// Nothing here needs privilege: no_new_privs (which the unit already sets
+// with NoNewPrivileges=) is the only precondition.
+func ScopeAbstractUnix() error {
+ abi, _, e := unix.Syscall(unix.SYS_LANDLOCK_CREATE_RULESET, 0, 0, unix.LANDLOCK_CREATE_RULESET_VERSION)
+ if e != 0 {
+ return fmt.Errorf("landlock unavailable: %w", e)
+ }
+ if abi < landlockScopeABI {
+ return fmt.Errorf("landlock ABI %d cannot scope abstract unix sockets (needs %d, Linux 6.12)", abi, landlockScopeABI)
+ }
+ attr := unix.LandlockRulesetAttr{Scoped: unix.LANDLOCK_SCOPE_ABSTRACT_UNIX_SOCKET}
+ fd, _, e := unix.Syscall(unix.SYS_LANDLOCK_CREATE_RULESET, uintptr(unsafe.Pointer(&attr)), unsafe.Sizeof(attr), 0)
+ if e != 0 {
+ return fmt.Errorf("landlock ruleset: %w", e)
+ }
+ defer unix.Close(int(fd))
+ if err := unix.Prctl(unix.PR_SET_NO_NEW_PRIVS, 1, 0, 0, 0); err != nil {
+ return fmt.Errorf("no_new_privs: %w", err)
+ }
+ if _, _, e := unix.Syscall(unix.SYS_LANDLOCK_RESTRICT_SELF, fd, 0, 0); e != 0 {
+ return fmt.Errorf("landlock restrict: %w", e)
+ }
+ return nil
+}
+
+// scopedExec is the trampoline's exec: scope, then exec on the same
+// thread. A scope that cannot be applied refuses the exec; the probe has
+// already failed readiness on such a box, so no nested wake gets here.
+func scopedExec(argv0 string, argv, env []string) error {
+ runtime.LockOSThread() // never unlocked: the thread execs or the process exits
+ if err := ScopeAbstractUnix(); err != nil {
+ return fmt.Errorf("not starting without the abstract socket scope: %w", err)
+ }
+ return syscall.Exec(argv0, argv, env)
+}
+
+var errAbstractReached = errors.New("connected to an abstract socket bound outside the scope")
+
+// realAbstractCheck proves the scope on this kernel, end to end: it binds
+// an abstract name, connects to it once unscoped (the stack works), then
+// connects again from a scoped thread, which must answer EPERM. The scoped
+// thread is left locked so it dies with its restriction.
+func realAbstractCheck() error {
+ name := "@carlos-probe-" + randomName()
+ ln, err := net.Listen("unix", name)
+ if err != nil {
+ return fmt.Errorf("listen %s: %w", name, err)
+ }
+ defer ln.Close()
+ go func() {
+ for {
+ c, err := ln.Accept()
+ if err != nil {
+ return
+ }
+ c.Close()
+ }
+ }()
+ c, err := net.Dial("unix", name)
+ if err != nil {
+ return fmt.Errorf("unscoped connect: %w", err)
+ }
+ c.Close()
+ res := make(chan error, 1)
+ go func() {
+ runtime.LockOSThread()
+ if err := ScopeAbstractUnix(); err != nil {
+ res <- err
+ return
+ }
+ c, err := net.Dial("unix", name)
+ if err == nil {
+ c.Close()
+ res <- errAbstractReached
+ return
+ }
+ res <- unwrapErrno(err)
+ }()
+ if err := <-res; !errors.Is(err, syscall.EPERM) {
+ return err
+ }
+ return nil
+}
+
+// initIPCInode is PROC_IPC_INIT_INO, the inode of the box's own IPC
+// namespace; any other value is a private one.
+const initIPCInode = 0xEFFFFFFF
+
+func ipcNamespaceInode() (uint64, error) {
+ fi, err := os.Stat("/proc/self/ns/ipc")
+ if err != nil {
+ return 0, err
+ }
+ st, ok := fi.Sys().(*syscall.Stat_t)
+ if !ok {
+ return 0, errors.New("no inode")
+ }
+ return st.Ino, nil
+}
+
+// ioWeightEnforced reports whether io.weight has a controller behind it on
+// this box: io.cost enabled for some device (a line with enable=1 in the
+// root's io.cost.qos), or BFQ, which exposes io.bfq.weight in the unit's
+// own cgroup.
+func ioWeightEnforced(selfCgroup string) bool {
+ if b, err := os.ReadFile("/sys/fs/cgroup/io.cost.qos"); err == nil {
+ for _, line := range strings.Split(string(b), "\n") {
+ if strings.Contains(" "+line+" ", " enable=1 ") {
+ return true
+ }
+ }
+ }
+ _, err := os.Stat(selfCgroup + "/io.bfq.weight")
+ return err == nil
+}
diff --git a/internal/instanceexec/scope_other.go b/internal/instanceexec/scope_other.go
new file mode 100644
index 000000000..9176634f2
--- /dev/null
+++ b/internal/instanceexec/scope_other.go
@@ -0,0 +1,13 @@
+//go:build !linux
+
+package instanceexec
+
+import "errors"
+
+func ScopeAbstractUnix() error { return errors.ErrUnsupported }
+func scopedExec(string, []string, []string) error { return errors.ErrUnsupported }
+func realAbstractCheck() error { return errors.ErrUnsupported }
+func ipcNamespaceInode() (uint64, error) { return 0, errors.ErrUnsupported }
+func ioWeightEnforced(string) bool { return false }
+
+const initIPCInode = 0xEFFFFFFF