carlos / platform Public

Question asked: can CARLOS limit the memory and CPU impact of apps and instances, or be extended to? This sketch covers what is enforced today and a small extension for CPU. No code has been written yet.

What is enforced today

Memory: yes, per instance, in two platform-chosen tiers. No number comes from a manifest.

| Tier | memory.max | memory.high | Who gets it | |---|---|---|---| | TierDefault | 512M | 90% (~460M) | everything | | TierChildren | 1G | 90% (~920M) | capabilities.children |

• The tiers are defined in internal/cgroup/cgroup.go (TierDefault, TierChildren, HighOf). internal/host/memtier.go picks the tier from the manifest's children bit.
• Unit-backed instances: instancectl.Server.setTier runs systemctl set-property --runtime carlos-instances-<iid>.slice MemoryMax= MemoryHigh= MemorySwapMax=. parseRequest refuses any memory_max that is not one of the two tier numbers.
• Exec children of the edge: cgroup.Manager.applyLimit writes memory.max, memory.high, memory.swap.max (= max, so zram cannot absorb the overflow) and memory.oom.group=1.
• Unit-backed apps: carlos-app@.service sets MemoryMax=512M. caps/render.go sets 1G for children and 4G for the doc engine.
• Outer wall: carlos.slice has MemoryHigh=70% and MemoryMax=80% over the edge and all instances. Tenants run with OOMScoreAdjust=500, so the kernel kills a tenant before the agent.

Task count: yes. TasksMax=256 per instance unit, 1024 for children.

CPU and IO: measured, not limited. Delegate=cpu memory io pids and the +cpu +memory +io +pids write in SetupAt exist for metrics attribution only. Nothing writes cpu.weight, cpu.max or io.*, and no instance or app unit sets CPUWeight= or CPUQuota=. One busy instance can use every core on the box.

Proposal: CPU in the same tier

1. Weight first, quota optional

• CPUWeight (proportional share) is the main lever. It has no cost when the box is idle. Under contention, it stops one tenant from starving the others and the edge.
• CPUQuota (hard ceiling) is optional and off by default. On a mostly idle single-tenant box it only adds latency. On burstable instances (t4g), a quota does not save credits; it only hides the spend from the tenant. It is still worth having as an operator knob for a known-abusive app.

2. Make the tier a name on the wire, not a number

instancectl.Client.Start and Client.Seed send memory_max=<bytes>, and the server checks it against a whitelist of two numbers. Adding a CPU dimension that way means whitelisting combinations. A cleaner shape:

// internal/cgroup
type Limit struct {
    Name      string // "default" | "children" — the wire form
    MemoryMax int64
    CPUWeight int    // 1..10000; 0 = write nothing
    CPUMax    string // cpu.max form, e.g. "100000 100000" (one core); "" = write nothing
}

var (
    TierDefault  = Limit{Name: "default",  MemoryMax: 512 << 20, CPUWeight: 100}
    TierChildren = Limit{Name: "children", MemoryMax: 1 << 30,  CPUWeight: 100}
)

func TierByName(name string) (Limit, bool)

• The client sends tier=default. The server looks it up with TierByName, so a tier still cannot be invented over the socket. The validation stays where it is and gets simpler.
• setTier appends CPUWeight= (and CPUQuota= when CPUMax is set) to the same set-property call.
• applyLimit writes cpu.weight / cpu.max after the memory files, with the same best-effort, one-log-line behaviour.
• caps/render.go and carlos-app@.service get matching CPUWeight= lines, so a declared capability behaves the same on both backings (the rule the code comments already state).
• Rollout: the wire change needs the edge and instance-control to agree. Either accept both memory_max and tier for one release, or ship them together. They live on the same box and roll as one unit, so shipping together is probably fine. To confirm.

3. Starting numbers

• All tenant tiers at CPUWeight=100 (the systemd default). This changes nothing by itself, but it makes the knob explicit and gives the next step something to adjust.
• The edge: raise its weight relative to carlos-instances.slice (e.g. CPUWeight=200 on carlos-edge.service). The edge is the proxy and activator for every tenant, so starving it hurts everyone. This is the one change with an immediate effect.

4. Disk IO (later)

IOWeight= is cheap and follows the same pattern. IOReadBandwidthMax= / io.max needs a device major:minor and is fussier. I would leave IO out of the first cut.

Open questions

1. Is the cpu controller actually enabled down to the per-host instance slices on a live box? Delegate=cpu on the edge enables it along the edge's own path. Whether carlos-instances.slice's cgroup.subtree_control includes cpu depends on systemd's controller propagation. If it does not, today's "fair share" between instances is not happening at all, and setting CPUWeight= is what turns it on. Check: cat /sys/fs/cgroup/carlos.slice/carlos-instances.slice/cgroup.subtree_control. I could not reach the box from this session.
2. Per-app or per-instance sizing. The design rule is "no number comes from a manifest". If an operator needs to give one instance more room, I suggest a third named tier, or a platform-side per-instance tier override that memTier reads (registry or instance config). Taking numbers from the manifest would reverse the rule rather than extend it.
3. Visibility. This overlaps with #184 (Expose effective app memory limits and OOM kills). Whatever exposes the effective memory tier should expose the CPU weight/quota and cpu.stat's nr_throttled / throttled_usec too. A throttled tenant otherwise looks like a slow app with no explanation, the same failure mode #184 describes for memory.

Rough size

Small: one struct change, two writer sites (applyLimit, setTier), a wire-parameter change in instancectl, a few unit lines, plus tests for the TierByName whitelist and the set-property arguments.