amadan
units: every instance template runs in the instance's own data directory (#256 D)
A flat child ran with cwd /, so an app writing ./data/blobs wrote under /data. A nested unit had no WorkingDirectory=, so its cwd was / on the read-only tmpfs the namespace puts there, and bdn/sharingangels exited at start with "mkdir ./data/blobs: read-only file system" for seventeen hours after the migration, until a drop-in on the IE box set WorkingDirectory=/data/instances/%I for that one unit. The run, seed and sidecar templates now carry that exact line, beside the BindPaths= it chdirs into (systemd applies the working directory after the mount namespace), so the three roles see the instance's files from one place. The unit test pins the line in all three and its absence from the probe, and that each template has exactly one. The IE drop-in, 50-workdir-incident-256.conf, carries the same value, so removing it after this rolls changes nothing. What a flat app wrote relative to / is not moved by the layout pass: a shared /data/blobs cannot be given to one host. Fix B reports such directories once per pass that moved something; docs/site/cli.md says where relative paths land under each layout and what an operator does with a shared one.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
5 files changed,
+49
−5
docs/site/cli.md+9 −3internal/host/units/carlos-instance-seed@.service+9 −0internal/host/units/carlos-instance-sidecar@.service+9 −0internal/host/units/carlos-instance@.service+9 −0internal/host/units_isolation_test.go+13 −2
CI failed — run details
diff --git a/docs/site/cli.md b/docs/site/cli.md| index b106c9ffc..1b2da95f3 100644 |
| --- a/docs/site/cli.md |
| +++ b/docs/site/cli.md |
| @@ -678,9 +678,15 @@ explicit decision on the home's record, never a default. |
| A hibernating instance is started by the box when a request arrives for it, |
| as `<your binary> --socket <path> --db <path>`: the unix socket to serve on, |
| -and the SQLite database that is yours. Its environment is your app's config |
| -bundle, with the instance's own `<host>.env` layered on top, plus a few |
| -variables the platform sets and that no bundle can override: |
| +and the SQLite database that is yours. On a box with the nested layout its |
| +working directory is its own data directory, the one the database is in, so |
| +a relative path such as `./data/blobs` lands beside the database and inside |
| +the instance's namespace. (Under the flat layout it was `/`, so the same path |
| +landed in the shared `/data`; `layout migrate` leaves such shared directories |
| +where they are and names them in the agent log for an operator to place.) |
| +Its environment is your app's config bundle, with the instance's own |
| +`<host>.env` layered on top, plus a few variables the platform sets and that |
| +no bundle can override: |
| | Variable | Value | |
| |---|---| |
diff --git a/internal/host/units/carlos-instance-seed@.service b/internal/host/units/carlos-instance-seed@.service| index 42facdaae..24e0eac7e 100644 |
| --- a/internal/host/units/carlos-instance-seed@.service |
| +++ b/internal/host/units/carlos-instance-seed@.service |
| @@ -53,6 +53,15 @@ StandardError=inherit |
| # The instance's own two directories, the only writable data paths. |
| BindPaths=/data/instances/%I /run/carlos/instances/%I |
| ReadWritePaths=/data/instances/%I /run/carlos/instances/%I |
| +# The working directory is the instance's own data directory, the one its |
| +# database is in (discussion #256 item D). A flat child ran with cwd /, |
| +# and an app writing relative paths (./data/blobs) wrote under /data; |
| +# without this line a unit's cwd is / on a read-only tmpfs and that app |
| +# exits at start with "read-only file system". systemd applies it after |
| +# the namespace is set up, so the bind above is what it chdirs into. Same |
| +# value in all three templates: the seed and the sidecar see the files |
| +# the instance made from the same place. |
| +WorkingDirectory=/data/instances/%I |
| # Its own build, and no other app's (#214 item 2). The namespace block |
| # empties /opt/carlos; these put back /opt/carlos/live/<host>, a link to |
| # ../app, and /opt/carlos/app, which PID 1 binds by chasing the source |
diff --git a/internal/host/units/carlos-instance-sidecar@.service b/internal/host/units/carlos-instance-sidecar@.service| index 18c672e28..4a43cf115 100644 |
| --- a/internal/host/units/carlos-instance-sidecar@.service |
| +++ b/internal/host/units/carlos-instance-sidecar@.service |
| @@ -52,6 +52,15 @@ StandardError=inherit |
| # The instance's own two directories, the only writable data paths. |
| BindPaths=/data/instances/%I /run/carlos/instances/%I |
| ReadWritePaths=/data/instances/%I /run/carlos/instances/%I |
| +# The working directory is the instance's own data directory, the one its |
| +# database is in (discussion #256 item D). A flat child ran with cwd /, |
| +# and an app writing relative paths (./data/blobs) wrote under /data; |
| +# without this line a unit's cwd is / on a read-only tmpfs and that app |
| +# exits at start with "read-only file system". systemd applies it after |
| +# the namespace is set up, so the bind above is what it chdirs into. Same |
| +# value in all three templates: the seed and the sidecar see the files |
| +# the instance made from the same place. |
| +WorkingDirectory=/data/instances/%I |
| # Its own build, and no other app's (#214 item 2). The namespace block |
| # empties /opt/carlos; these put back /opt/carlos/live/<host>, a link to |
| # ../app, and /opt/carlos/app, which PID 1 binds by chasing the source |
diff --git a/internal/host/units/carlos-instance@.service b/internal/host/units/carlos-instance@.service| index f979ff7d6..c8cec39bf 100644 |
| --- a/internal/host/units/carlos-instance@.service |
| +++ b/internal/host/units/carlos-instance@.service |
| @@ -52,6 +52,15 @@ StandardError=inherit |
| # The instance's own two directories, the only writable data paths. |
| BindPaths=/data/instances/%I /run/carlos/instances/%I |
| ReadWritePaths=/data/instances/%I /run/carlos/instances/%I |
| +# The working directory is the instance's own data directory, the one its |
| +# database is in (discussion #256 item D). A flat child ran with cwd /, |
| +# and an app writing relative paths (./data/blobs) wrote under /data; |
| +# without this line a unit's cwd is / on a read-only tmpfs and that app |
| +# exits at start with "read-only file system". systemd applies it after |
| +# the namespace is set up, so the bind above is what it chdirs into. Same |
| +# value in all three templates: the seed and the sidecar see the files |
| +# the instance made from the same place. |
| +WorkingDirectory=/data/instances/%I |
| # Its own build, and no other app's (#214 item 2). The namespace block |
| # empties /opt/carlos; these put back /opt/carlos/live/<host>, a link to |
| # ../app, and /opt/carlos/app, which PID 1 binds by chasing the source |
diff --git a/internal/host/units_isolation_test.go b/internal/host/units_isolation_test.go| index 9bc3ba317..f7a939f57 100644 |
| --- a/internal/host/units_isolation_test.go |
| +++ b/internal/host/units_isolation_test.go |
| @@ -266,16 +266,21 @@ func TestInstanceTemplatesShareOneNamespace(t *testing.T) { |
| for name, wants := range map[string][]string{ |
| "carlos-instance@.service": {"ExecStart=/opt/carlos/carlos instance-exec run %I", "ExecStopPost=/opt/carlos/carlos instance-exit run %I", |
| "BindPaths=/data/instances/%I /run/carlos/instances/%I", "StandardOutput=file:/run/carlos/logs/%I.run.sock", |
| + // The instance's own data directory, where a flat child's |
| + // relative paths now resolve (#256 D); the IE box's emergency |
| + // drop-in 50-workdir-incident-256.conf carries this exact line, |
| + // so removing it changes nothing. |
| + "WorkingDirectory=/data/instances/%I", |
| "Slice=carlos-instances-%i.slice", "Restart=no", "KillMode=mixed", "TimeoutStopSec=20", "BindsTo=carlos-edge.service", |
| "Requires=data-instances.mount run-carlos-instances.mount run-carlos-logs.mount", "OOMScoreAdjust=500", "OOMPolicy=kill", "TasksMax=256", "Type=simple", |
| // The trampoline's init exits 128+n for a death by signal n (#126). |
| "SuccessExitStatus=129 130 141 143"}, |
| "carlos-instance-seed@.service": {"Type=oneshot", "RemainAfterExit=yes", "TimeoutStartSec=120", "ExecStart=/opt/carlos/carlos instance-exec seed %I", |
| "ExecStopPost=/opt/carlos/carlos instance-exit seed %I", "StandardOutput=file:/run/carlos/logs/%I.seed.sock", |
| - "KillMode=mixed", "TimeoutStopSec=20"}, |
| + "KillMode=mixed", "TimeoutStopSec=20", "WorkingDirectory=/data/instances/%I"}, |
| "carlos-instance-sidecar@.service": {"Type=simple", "ExecStart=/opt/carlos/carlos instance-exec sidecar %I", |
| "ExecStopPost=/opt/carlos/carlos instance-exit sidecar %I", "StandardOutput=file:/run/carlos/logs/%I.sidecar.sock", |
| - "SuccessExitStatus=129 130 141 143"}, |
| + "SuccessExitStatus=129 130 141 143", "WorkingDirectory=/data/instances/%I"}, |
| "carlos-instance-probe.service": {"Type=oneshot", "ExecStart=/opt/carlos/carlos instance-probe", |
| "BindPaths=/data/instances/_probe /run/carlos/instances/_probe", "StandardOutput=truncate:/run/carlos-control/probe.out"}, |
| } { |
| @@ -285,6 +290,12 @@ func TestInstanceTemplatesShareOneNamespace(t *testing.T) { |
| t.Errorf("%s lacks %q", name, w) |
| } |
| } |
| + // Exactly one WorkingDirectory= per tenant template, and none in |
| + // the probe, whose directory is _probe and whose cwd does not |
| + // matter: a second line would silently win (#256 D). |
| + if n := strings.Count(u, "\nWorkingDirectory="); (name != "carlos-instance-probe.service" && n != 1) || (name == "carlos-instance-probe.service" && n != 0) { |
| + t.Errorf("%s has %d WorkingDirectory= lines", name, n) |
| + } |
| if strings.Contains(u, "ExecStopPost=+") { |
| t.Errorf("%s uses ExecStopPost=+: the plus prefix keeps the unit's tmpfs (codex R3-2)", name) |
| } |