amadan

carlos / platform Public

← Discussions

Cursory load testing for tenant apps (carlos load) Feature Request open

claude driven by paul@keymail.dev session 785b0a91 · claude-code/2.1.292 · DOCS-T1

Paul asked whether CARLOS should offer cursory load testing. Answer: yes, but narrow, and only in a form a generic tool can't match. Parked until the paid launch and claim activation are done. It overlaps Plan 8 (abuse guards) of carloku-plans-caps-v1, so design the two together.

Why build it into the platform

  • Server-side view. The platform can report the server side alongside client latency: wake events, cgroup CPU throttling, memory peak against the tier, edge queueing and Litestream lag. k6 or hey from a laptop sees latency only.
  • Hibernation. Every app hibernates, so the useful answers are the cold-wake time and the warm p50/p95/p99, reported separately. Generic tools mix the two together.
  • Plans and caps. The report can name the cap that binds first ("memory ceiling at ~40 concurrent"). That ties straight into tiers and packs.

Constraints (non-negotiable)

  • Not DoS-as-a-service. Targets are only the caller's own apps and verified domains, resolved through the edge. Never an arbitrary URL.
  • Noisy neighbours. Apps share boxes. Rate, concurrency and duration get hard ceilings set by the platform, not the user. Free accounts get a tiny smoke test or none at all. Every run counts against the bandwidth pack.
  • Real path. Generate the load from another region's edge, so it goes through public TLS and the edge rather than skipping them.
  • Provider-neutral. Nothing in the generator should depend on AWS.

Shape, v1

Two fixed profiles, with no knobs at first:

  • smoke: about 10 requests a second for 30 seconds.
  • ramp: climb to the tier's ceiling, then stop.

Report: p50/p95/p99, error rate, cold-wake time, resource peak (CPU throttle, memory, edge queue), and the first cap that would bind.

Open questions

  1. Is this a CLI subcommand (carlos load <app>, in the carlos repo), a console door, or both?
  2. Where does the generator run: edge boxes in another region, or a dedicated short-lived runner?
  3. Which census entries and claim routes does the new door need?
  4. Is it paid-only, or does Free get the smoke profile?

claude driven by paul@keymail.dev session 785b0a91 · claude-code/2.1.292 · DOCS-T1

Paul's answers to the four open questions (2026-10-24):

  1. Surface: both, a CLI command and a console door.
  2. Generator: an edge box in another region makes sense. Paul also raised AWS Lambda, pointing at the new web functions (https://aws.amazon.com/blogs/compute/exploring-aws-lambda-web-functions-for-building-web-applications/).
  3. Census and claim routes: open. Work it out at spec time.
  4. Plans: paid-only on Carloku. Self-hosters get it and pay for their own load.

Agent notes on the Lambda option, from reading the post:

  • Web functions are built to serve inbound HTTP: an HTTP server per environment, up to 128 concurrent requests each, 2 vCPU / 2 GB, Node.js 24 on arm64 at launch, billed on active CPU. The post says nothing about outbound networking, egress IPs or the maximum duration, and those are exactly what a load generator depends on. Verify them before relying on Lambda.
  • Active-CPU billing would suit a generator well, since it spends most of its time waiting on responses. But ordinary event Lambda invocations fanned out would also work, and wouldn't need web functions at all.
  • Under provider-neutral-core the generator has to be an interface. Proposal: the edge-in-another-region generator is the default, and it's the one self-hosters use as they are. Lambda is an optional AWS adapter added later if the edge path turns out short on capacity or isolation. Running the generator on an edge box also loads that box, which is a cost to weigh against Lambda's isolation.