carlos / platform Public

What and why

Tito's console (carlos.tito.io, on carlos-tito-ie, eu-west-1) runs cf02a4d5-tito: platform 16593de5 (2026-09-11) plus the short-alias commits (on main as e36cdb6a and 86d10e28) and one CI-only commit. Main is 314 commits ahead. It is pinned because main's console refuses to boot with CARLOS_STORE_BACKEND set and no CARLOS_STORE_USER_BOUNDARY, and Tito's account has no boundary policy. This thread creates the boundary, fixes the console env, and rolls the console to main. The env change also carries CONSOLE_RESERVE_PLATFORM_DOMAINS=true (#156; Tito hosts other people's apps), the full edge list (#179), and CARLOS_SYSTEM_OPERATORS for Bill (#176, building on branch system-operators-176).

The production change (needs Paul's scoped approval)

Account and region: titogo (665106592142), eu-west-1. IAM only. The only principal that changes is the role carlos-tito-host-ie.

Repo: teamtito/carlos-infrastructure, as a PR on GitHub, its primary. The region-host module pin 7f7f941797fa77830893c4615a5bbaaeb01c0e8a stays where it is.

Why not the module's store_bucket_prefix input

That input would create module.host_ie.aws_iam_policy.store_user_boundary[0] and module.host_ie.aws_iam_role_policy.host_store[0]. Tito's pin predates it; v0.19.0 is the first tag with it. It is not safe today, for three reasons:

1. The lease outage would come back. Every tag from v0.19.0 to v0.22.2, and main (infra/modules/region-host/main.tf, host_store), ships an explicit Deny on s3:DeleteBucket, s3:DeleteObject and s3:CreateBucket against the deployment bucket and the data bucket. Releasing an instance lease is a DeleteObject on instances/<host>/lock.json. On the flagship this made every hibernating instance answer 503 for the full 15-minute lease TTL. The fix (907c3c9e on worktree-lease-release-hardening) is not on main and no tag carries it.
2. The pin bump brings other changes. It also adds host_ssm_parameter_scope[0] (scope_ssm_parameter_reads defaults to true; Tito's deny_foreign_parameters already covers it) and VPC data sources. Nobody has checked those plan as a no-op on the security group and the instance.
3. It would add nothing. Tito's own aws_iam_role_policy.host_ie_object_store already allows iam:CreateUser and iam:PutUserPolicy on user/carlos-store/* with no condition. IAM combines Allow statements across policies, so the module's boundary-conditioned Allow changes nothing while the unconditioned one stands.

So the boundary and the condition go into Tito's own object-store.tf. Once a fixed tag exists, a later change can move them into the module.

Phase 1 (additions only; safe for the console running today)

• Create aws_iam_policy.store_user_boundary, named carlos-tito-host-ie-store-boundary (the module's name). It mirrors the module document: - Allow s3:ListBucket and s3:GetBucketLocation on arn:aws:s3:::carlos-store-tito-*. - Allow s3:GetObject, s3:PutObject and s3:DeleteObject on arn:aws:s3:::carlos-store-tito-*/*. - Two possible additions, Paul to decide: - s3:GetObjectTagging. The store user's inline policy grants it and the module's boundary drops it. - titogo-uploads-665106592142 and /*, if that adopted bucket is served through a store user.
• Update aws_iam_role_policy.host_ie_tenant_mail: add ses:DeleteTenantResourceAssociation to MailTenants. Main's console calls it before deleting a sending identity (internal/mail/aws.go). Carloku's tofu lacks it too.
• Add output "store_user_boundary_arn". No resource.

Phase 2 (after the new console has run cleanly for a day)

• Update aws_iam_role_policy.host_ie_object_store by splitting StoreUsers in two: - iam:CreateUser and iam:PutUserPolicy on user/carlos-store/*, with StringEquals {"iam:PermissionsBoundary" = aws_iam_policy.store_user_boundary.arn}. - The rest (GetUser, DeleteUser, DeleteUserPolicy, CreateAccessKey, DeleteAccessKey, ListAccessKeys), unconditioned, as today. - StoreBuckets is unchanged. In particular, no Deny on the deployment or data buckets.

Purpose. Phase 1 lets main's console boot. Phase 2 closes the escalation the boundary exists for: without it, a compromised console role can create a /carlos-store/ user with any inline policy it likes.

Permissions effect. In phase 1 the role gains one SES action. In phase 2 it loses the ability to create or re-policy a store user without the boundary. Store users that existed before this change carry no boundary, so phase 2 refuses PutUserPolicy on them and a re-grant on such a user fails. An inventory is therefore a precondition of phase 2. Fixing existing users means an admin attaching the boundary with iam:PutUserPermissionsBoundary, and that is part of this approval only if Paul says so.

Traffic effect. None on app traffic.

Cost. $0.

Duration. Each apply takes minutes and is permanent. The phases are about a day apart.

Apply. Targeted:

• Phase 1: -target=aws_iam_policy.store_user_boundary -target=aws_iam_role_policy.host_ie_tenant_mail
• Phase 2: -target=aws_iam_role_policy.host_ie_object_store

Read the plan first and stop on anything else, especially under module.host_ie.aws_instance.host. Don't combine with PR #17 (region-ca).

Verification. aws iam get-policy on the boundary ARN, then aws iam simulate-principal-policy against carlos-tito-host-ie:

• ses:DeleteTenantResourceAssociation is allowed.
• After phase 2, iam:CreateUser on user/carlos-store/x is allowed with iam:PermissionsBoundary set to the ARN and implicitDeny without it.
• As a regression check against the flagship outage, s3:DeleteObject on <IE data bucket>/instances/x/lock.json is still allowed.

Cleanup and rollback. Revert phase 2 and re-apply; that restores the unconditioned grant. Phase 1 needs no rollback, because the old console ignores it. A boundary policy can't be deleted while any user carries it, so leave it in place.

Console env changes

In the file console.service reads on carlos-tito-ie. Back it up first, then re-run scripts/escrow-host-env.sh titogo eu-west-1 i-025a0b70f3d93304d ie so the SSM escrow matches.

CARLOS_STORE_USER_BOUNDARY=<store_user_boundary_arn from tofu output; expected arn:aws:iam::665106592142:policy/carlos-tito-host-ie-store-boundary>
CONSOLE_RESERVE_PLATFORM_DOMAINS=true
PLATFORM_EDGE_IPS=54.228.234.85,35.177.77.51,16.16.113.244,13.233.36.247,3.92.98.101,63.181.123.82
CARLOS_SYSTEM_OPERATORS=bill@keymail.dev

• CARLOS_STORE_USER_BOUNDARY: main's console exits at boot without it, because CARLOS_STORE_BACKEND=aws is set.
• CONSOLE_RESERVE_PLATFORM_DOMAINS: on main, reserving the apps domain and the console's hostnames is opt-in. Without it, any Tito account can attach names under Tito's apps domain, and the bap alias delegation loses its limits. A value that isn't a boolean stops the boot.
• PLATFORM_EDGE_IPS: 3 of the 6 edges today, which makes the custom-domain check flap (#179). The console reads it at start. carlos-edge reads it too, and picks it up when the box roll below restarts the edge. If PR #17 adds Canada, its IP joins the list.
• CARLOS_SYSTEM_OPERATORS: set it only on a build that carries #176.

Keep these as they are: CARLOS_STORE_BACKEND, CARLOS_STORE_REGION, CARLOS_STORE_BUCKET_PREFIX=carlos-store-tito-, CARLOS_STORE_OPERATORS, CONSOLE_APPS_DOMAIN_ALIAS_ACCOUNTS (bap) and CARLOS_REGION (main's console now reads it for the carlos db door).

Leave these unset:

• CARLOS_MAIL_LEGACY_SWEEP. It deletes the legacy email.json records and drops mail_counts.
• CARLOS_STORE_MAX_PER_APP. Unset means the default; if set, it must be a positive whole number or the boot fails.

The old console doesn't read the new keys, so writing them before the roll is harmless.

Roll order

1. Preconditions. - Read the live console env and the EnvironmentFile of console.service. - Confirm cf02a4d5-tito is still a console release in carlos-tito-platform-665106592142, or keep the binary to ship again. - Pick the console commit: the main commit carrying #176, or v0.22.2 (7edd4b33) with CARLOS_SYSTEM_OPERATORS added in a later console roll.
2. Apply phase 1 and verify it.
3. Edit the console env after backing it up, and re-run the escrow script.
4. Roll the console: platform-infrastructure/scripts/roll-console.sh <clean checkout at the chosen commit> tito. Then verify: - curl -sI https://carlos.tito.io/ | grep -i x-carlos-version; - carlos system status --console https://carlos.tito.io; - journalctl -u console has no console: fatal line.
5. Check domain reservation. From an account not on the alias list, try to attach an unused name under the apps domain. It should be refused as platform-managed. If it is accepted, remove the claim and stop.
6. Roll the boxes close behind the console (v0.22.0's operator note): carlos system update --console https://carlos.tito.io --version v0.22.2, then carlos system status. The apply restarts carlos-edge, which is how IE's agent picks up the new PLATFORM_EDGE_IPS. IE's edges: alias ... not published lines should stop.
7. After a day of clean running, take the phase 2 inventory, then apply phase 2 and verify it.

Rollback

1. Console: in direct-bucket mode (AWS_PROFILE=titogo, CARLOS_DEPLOYMENT_BUCKET=carlos-tito-platform-665106592142, CARLOS_RELEASE_KEY from SSM /carlos/release-key for the one command), run carlos promote --app console --hotfix cf02a4d5-tito canary/console.
2. Recover the unit: over SSM, run systemctl reset-failed console && systemctl restart console. On 2026-09-11 the promote alone didn't recover a crash-looping console. roll-console.sh doesn't run reset-failed.
3. What a console rollback loses: email pause and rotate state made under main's console (sender records moved to senders/app.json), and mail counts recorded in between. New registry tables and columns stay behind, unused.
4. Never roll the console back after phase 2. The old console creates store users without a boundary, and phase 2 refuses that. Revert phase 2 first.
5. Boxes: carlos system rollback, or carlos system update --version v0.21.0.

Open questions (couldn't answer read-only)

• What the console env file on carlos-tito-ie holds today: - whether it is /etc/carlos/host.env; - the values of CARLOS_STORE_OPERATORS and CONSOLE_APPS_DOMAIN_ALIAS_ACCOUNTS; - whether CARLOS_REGION is present; - whether PLATFORM_EDGE_IPS is still the three-address list.
• Whether any /carlos-store/ IAM users exist in titogo, and whether any carries a boundary. Phase 2 waits on this.
• Whether titogo-uploads-665106592142 is served as a store, and so whether the boundary must name it.
• Whether the other five boxes carry a short PLATFORM_EDGE_IPS too.
• Whether any Tito app relies on a store user reading object tags.
• Whether cf02a4d5-tito is still in the bucket.
• Whether the boxes are still on v0.21.0.