| 1 | # A deploy credential for CI: promote through the console, limited to rungs |
| 2 | |
| 3 | **Date:** 2026-10-05 |
| 4 | **Revision:** 5.2. |
| 5 | |
| 6 | - Rev 1 (d30a74c2) went to Astra (via codex). The findings are in |
| 7 | `CODEX-FINDINGS-DEPLOY-CREDENTIAL-R1.md`, unedited. All six were checked |
| 8 | against the code and accepted. Changes are marked "(codex N)" and §15 |
| 9 | records each disposition. |
| 10 | - **Rev 2 changes one thing Paul was told about rev 1.** Rev 1 moved CI's |
| 11 | rung limit from a grant every box verifies to a check in the console, and |
| 12 | flagged that as the trade-off. Rev 2 keeps the box-verified limit: the |
| 13 | console signs a `deploy` credential's pointer with a request-local scoped |
| 14 | key (§7.4, codex 6). |
| 15 | - Rev 2 (38dfb240) went back. The findings are in |
| 16 | `CODEX-FINDINGS-DEPLOY-CREDENTIAL-R2.md`. All three were checked and |
| 17 | held. Changes are marked "(codex R2-N)". |
| 18 | - **Rev 3 withdraws a claim rev 2 made.** Rev 2 said a `deploy` promote |
| 19 | racing a pipeline install always ends in a state some serial order |
| 20 | produces. Astra gave a three-operation counterexample (codex R2-1). Rev 3 |
| 21 | reorders the reads so the pointer's own compare-and-swap defeats that |
| 22 | counterexample, states the guarantee as an invariant (§7.2, I1), and |
| 23 | names the one window it does not close. **That window is a decision for |
| 24 | Paul: §16.** |
| 25 | - Rev 3 (2d9d528e) went back. The findings are in |
| 26 | `CODEX-FINDINGS-DEPLOY-CREDENTIAL-R3.md`. All four were checked and held. |
| 27 | Changes are marked "(codex R3-N)". |
| 28 | - **Rev 4 stops patching interleavings.** Three rounds each found a new |
| 29 | ordering of the same fault: R2's pipeline rule reads one object and |
| 30 | writes another, and the store cannot make that atomic. Rev 3's I1(b) |
| 31 | said "nothing wrote C", which the store cannot promise either, because |
| 32 | an ETag is a hash of the content (codex R3-1). Rev 4 states only what |
| 33 | the store does guarantee (§7.2, G1 and G2), says in one sentence what |
| 34 | that leaves open, and requires a randomized interleaving test of exactly |
| 35 | those two properties (§12). §16's decision is unchanged in substance and |
| 36 | is now stated in those terms. |
| 37 | - Rev 4 (2af4eba4) went back. The findings are in |
| 38 | `CODEX-FINDINGS-DEPLOY-CREDENTIAL-R4.md`. Astra found no outcome outside |
| 39 | the class §7.2's open sentence describes, and found the three hook call |
| 40 | points sufficient. Two findings remained and both held: G2 assumed more |
| 41 | of the directory store than it provides, and §12's randomized test |
| 42 | asserted something G1 and G2 do not imply. Rev 5 qualifies G2 by backend |
| 43 | and corrects the test's oracle. Changes are marked "(codex R4-N)". |
| 44 | - Rev 5 (e764b20c) went back. Astra's round 5 |
| 45 | (`CODEX-FINDINGS-DEPLOY-CREDENTIAL-R5.md`) found nothing at P1 or P2 and |
| 46 | accepted. Rev 5 is unchanged by it. |
| 47 | - **Rev 5.1 records Paul's ruling on §16 as R5** (2026-10-05): the |
| 48 | in-flight gap is accepted. Nothing else changes. |
| 49 | - **Rev 5.2 records Paul's ruling R6** (2026-10-05): CI must be able to run |
| 50 | `carlos deploy`. §6 gains the two read doors its routine path needs. |
| 51 | **Discussion:** carlos/platform#229. |
| 52 | **Code read at:** main `8d8e4bae`. Every line reference below is at that commit. |
| 53 | |
| 54 | ## 1. The problem, as found |
| 55 | |
| 56 | titogo's CI promotes every merge to `edge` with `carlos promote` in |
| 57 | direct-bucket mode. It does that because the README tells it to |
| 58 | (`README.md:170`): "CI and operators keep the existing path unchanged ... |
| 59 | logging in is for a human at a laptop, never a box or a pipeline." |
| 60 | |
| 61 | A direct-bucket promote signs a pointer and writes it to S3. The console |
| 62 | never sees it, so the promote skips everything the console adds: |
| 63 | |
| 64 | - the audit line; |
| 65 | - pipeline rules and pending approvals; |
| 66 | - the serial bump (`s.bumpSerial`, `internal/console/apicli.go:906`) that |
| 67 | makes a box's `fastConvergeLoop` adopt within seconds. |
| 68 | |
| 69 | Without the bump, a box picks the promote up on `carlos-adopt.timer`, hourly |
| 70 | by default. Tito's box carries a hand-made 2-minute drop-in to make that |
| 71 | bearable. Measured on 2026-10-05, it adds 30 seconds to 3.5 minutes to every |
| 72 | titogo deploy. |
| 73 | |
| 74 | CI cannot move to the console today, because the narrowest credential it |
| 75 | could hold there is too wide. A `publish` service credential bound to one |
| 76 | app reaches (`internal/console/authcli.go:476-739`): |
| 77 | |
| 78 | - promote and rollback on `beta`, and a hotfix that skips the ladder order, |
| 79 | for an app with no pipeline (`apicli.go:802-811`: only the stable rung is |
| 80 | refused when `pipe == nil`). Promote refuses the stable rung when the app |
| 81 | has no pipeline or its pipeline does not name `stable`. A pipeline that |
| 82 | does name `stable` is governed by that channel's own rules and nothing |
| 83 | else (`apicli.go:813`), and a pipeline may name it without a passkey rule |
| 84 | (`internal/release/pipeline.go:255`). Rollback refuses stable |
| 85 | unconditionally (codex 4); |
| 86 | - trash and restore of the app, retention, version-target, domains. |
| 87 | |
| 88 | Today's CI key is a scoped release key for `canary,edge`. It cannot sign |
| 89 | beta or stable, and it cannot do anything else. Moving CI to `publish` would |
| 90 | widen its authority, so the Tito session asked for the feature instead of |
| 91 | working around the gap. That is the right call, and every Carloku customer |
| 92 | with a CI pipeline will need the same thing: on Carloku a customer never |
| 93 | holds bucket credentials, so the console is their only door. |
| 94 | |
| 95 | ## 2. Goals and non-goals |
| 96 | |
| 97 | **Goals** |
| 98 | |
| 99 | 1. A console service credential that CI can hold with authority no wider |
| 100 | than a scoped release key: ship, and promote or roll back only the rungs |
| 101 | it was granted. |
| 102 | 2. CI can supply that credential to the CLI from its environment, with no |
| 103 | credentials file. |
| 104 | 3. The README and CLI docs tell CI to use the console. Direct-bucket mode is |
| 105 | documented as the operator's and break-glass path. |
| 106 | 4. A console rolled back to a build from before this change refuses the new |
| 107 | credential at every door. |
| 108 | |
| 109 | **What "adopts in seconds" means here.** This spec claims adoption: the box |
| 110 | stages the new version and repoints to it within seconds of the promote. It |
| 111 | does not claim the new build is serving within seconds. |
| 112 | |
| 113 | - For a unit-backed route, which is what titogo's fleet runs on, serving |
| 114 | follows adoption as soon as the unit restarts, so #229's goal stands. |
| 115 | - For an awake exec-backed instance, adoption does not cycle the running |
| 116 | process today. On a real box the root adopt pass stages and repoints |
| 117 | through `rootRestartRoute` (`internal/host/agent.go:3033`), which cycles |
| 118 | no exec child, and the agent then sees the version already current and |
| 119 | never calls its own restart. The activator's sweep catches it later: a |
| 120 | 30-second tick plus `ConvergeGrace` of 90 seconds |
| 121 | (`internal/activator/activator.go:532`). The platform session measured |
| 122 | 110 to 200 seconds from ship to serving on the flagship over 3 to 5 |
| 123 | October. |
| 124 | |
| 125 | That gap is being fixed on its own branch by another session, and the |
| 126 | handover that makes a shipped release reach running instances immediately |
| 127 | is a separate spec. Neither is part of this one. |
| 128 | |
| 129 | **Non-goals** |
| 130 | |
| 131 | - Making an adopted release serve sooner on exec-backed instances (above). |
| 132 | |
| 133 | - Scopes that name pipeline channels (ruling R2). |
| 134 | - Narrowing or changing existing `publish` credentials. |
| 135 | - Expiry on service credentials. Rotate and revoke stay as they are. |
| 136 | - Any change to direct-bucket mode. In particular, a direct promote still |
| 137 | does not bump the serial (ruling R4). |
| 138 | - Changing who restarts titogo's units. That is titogo's own work |
| 139 | (teamtito/titogo#3872). |
| 140 | |
| 141 | ## 3. Rulings (Paul, 2026-10-05) |
| 142 | |
| 143 | - **R1.** A new role, `deploy`, plus a rung scope on the credential. Existing |
| 144 | roles are unchanged. |
| 145 | - **R2.** The scope names rungs only, exactly as a scoped release key does. |
| 146 | For an app with a pipeline a `deploy` credential reaches `canary/*` and |
| 147 | nothing else. |
| 148 | - **R3.** The console is the normal path for CI. Direct-bucket mode stays as |
| 149 | the operator's tool: bootstrap before a console exists, deployments with |
| 150 | no console, recovery when the console is down, and the platform's own |
| 151 | releases. |
| 152 | - **R4.** #229's option 2 is declined in both forms. A direct promote that |
| 153 | bumps the serial makes the bypass more attractive. Polling every followed |
| 154 | channel pointer multiplies the one serial read per 2-second tick that the |
| 155 | serial exists to protect. |
| 156 | - **R5.** A `deploy` request in flight when a pipeline is installed may |
| 157 | complete, within G1 and G2 (§7.2, §16). The gap is accepted and is not |
| 158 | closed by this spec. |
| 159 | - **R6.** CI must be able to run `carlos deploy`, not only `ship` and |
| 160 | `promote`. A `deploy` credential therefore reaches the read doors |
| 161 | `carlos deploy` uses on an app that is already set up (§6). |
| 162 | |
| 163 | ## 4. The shape, in one paragraph |
| 164 | |
| 165 | `carlos services create --role deploy --app titogo --rungs canary,edge ci` |
| 166 | mints a service credential. CI exports it as `CARLOS_SERVICE_TOKEN` with |
| 167 | `CARLOS_CONSOLE`, and runs the same `carlos ship` and `carlos promote` it |
| 168 | runs today. The console checks the role at the door and the target channel |
| 169 | against the credential's rungs. It then signs the pointer with a scoped key |
| 170 | minted for that one request and limited to the same rungs, so every box |
| 171 | still verifies the limit. The promote is audited and bumps the serial, so |
| 172 | the box adopts in seconds. The runner holds no AWS role and no signing key. |
| 173 | |
| 174 | ## 5. The credential |
| 175 | |
| 176 | ### 5.1 Role |
| 177 | |
| 178 | `internal/svcrole/svcrole.go` gains `Deploy Role = "deploy"` in the |
| 179 | catalogue. `Allow()` keeps adding `Admin` to every set, unchanged. |
| 180 | |
| 181 | ### 5.2 Rungs |
| 182 | |
| 183 | - Allowed values: `canary`, `edge`, `beta` (`release.RungCanary`, |
| 184 | `RungEdge`, `RungBeta`, `internal/release/ladder.go:30-32`). `canary` |
| 185 | covers every `canary/<branch>` channel, as it does in a key grant. |
| 186 | - `stable` is refused at mint and again at the door, mirroring |
| 187 | `internal/release/grant.go:103` and `:219`. No delegated key has ever |
| 188 | been able to reach stable, and no `deploy` credential can (codex 4). |
| 189 | - Stored canonically: deduplicated, in ladder order, comma-joined. |
| 190 | |
| 191 | ### 5.3 Storage |
| 192 | |
| 193 | One additive column on `cli_tokens`, beside the service columns at |
| 194 | `internal/console/authstore.go:210-216`: |
| 195 | |
| 196 | ALTER TABLE cli_tokens ADD COLUMN rungs TEXT NOT NULL DEFAULT '' |
| 197 | |
| 198 | `Principal` (`internal/console/svcprincipal.go:48`) gains `Rungs []string`. |
| 199 | |
| 200 | ### 5.4 Mint rules |
| 201 | |
| 202 | Both mint doors (`carlos services create` through `handleAPIServiceCreate`, |
| 203 | and the browser form at `POST /a/{sqid}/services/new`) write through |
| 204 | `Store.NewServicePrincipal` (`svcprincipal.go:171`). The rules live there, |
| 205 | so neither door can skip them: |
| 206 | |
| 207 | - role `deploy` requires a non-empty app, the same way `service` does at |
| 208 | `svcprincipal.go:189`; |
| 209 | - role `deploy` requires at least one rung, each one of the three in §5.2; |
| 210 | - any other role with a non-empty rungs value is refused. A rung scope on |
| 211 | `publish` would read as a limit and enforce nothing. |
| 212 | |
| 213 | There is no default rung set. The person minting says what the credential |
| 214 | may move. |
| 215 | |
| 216 | ### 5.5 Reading it back |
| 217 | |
| 218 | A `deploy` principal whose stored rungs are empty or do not parse is refused |
| 219 | when the token is resolved. It is never treated as unscoped. This is the |
| 220 | case a hand-edited row or a future bug would produce, and it must fail |
| 221 | closed. |
| 222 | |
| 223 | Rotate keeps the rungs. There is no edit: to change a credential's rungs, |
| 224 | mint a new one and revoke the old. |
| 225 | |
| 226 | ### 5.6 CLI and browser |
| 227 | |
| 228 | - `carlos services create` gains `--rungs <list>`. The usage line and the |
| 229 | role list in `services.go:33` and `:157` gain `deploy`. |
| 230 | - `printServiceCreated` (`services.go:236`) states the rungs in words next to |
| 231 | the app scope, and its delivery hint covers the CI case (§8). |
| 232 | - `carlos services ls` and the services page show the rungs. |
| 233 | - The browser mint form offers the role and the three rungs. |
| 234 | |
| 235 | ## 6. Doors |
| 236 | |
| 237 | `deploy` is added to the allow set of exactly these routes in |
| 238 | `internal/console/authcli.go`: |
| 239 | |
| 240 | | Door | Why | |
| 241 | | --- | --- | |
| 242 | | `GET /api/cli/apps` | the CLI resolves the app | |
| 243 | | `GET .../{app}/channels` | read what is promoted where | |
| 244 | | `POST .../{app}/releases` | ship | |
| 245 | | `GET .../{app}/releases` | list what was shipped | |
| 246 | | `POST .../{app}/promote` | scoped, §7 | |
| 247 | | `POST .../{app}/rollback` | scoped, §7 | |
| 248 | | `GET .../{app}/tags`, `GET .../{app}/tags/{version}` | read | |
| 249 | | `GET .../{app}/version-target` | read | |
| 250 | | `GET .../{app}/pipeline` | read | |
| 251 | | `GET .../{app}/pending` | read | |
| 252 | | `GET .../{app}/instances` | read: `carlos deploy` lists the instances it will watch (R6) | |
| 253 | | `GET .../{app}/regions` | read: `carlos deploy --region` checks the app's region (R6) | |
| 254 | |
| 255 | It is added to nothing else. In particular it does not reach: app create, |
| 256 | trash, restore, retention (read or write), version-target write, domains, |
| 257 | features, logs, or any other `instance`, `operate` or `admin` door. |
| 258 | |
| 259 | **What `carlos deploy` does with a `deploy` credential (R6).** Its routine |
| 260 | path on an app that is already set up reads the instance list, reads the |
| 261 | channels, ships, promotes, and watches the app's own hosts, and with |
| 262 | `--region` reads the app's region. All of that is in the table. Three |
| 263 | things it does only in particular cases stay out of reach: |
| 264 | |
| 265 | - **First-deploy setup.** Creating the app, enabling instances and minting |
| 266 | the first platform host are writes a person does once. A `deploy` |
| 267 | credential is refused at those doors with the role gate's message, and |
| 268 | the deploy stops there. |
| 269 | - **`--host` on a static deploy** attaches a domain, which is a domains |
| 270 | write. Refused the same way. |
| 271 | - **The watch's diagnosis.** When a watched host times out, `carlos deploy` |
| 272 | reads one line of the app's platform log to say why. That read is |
| 273 | best-effort: refused, the deploy reports the timeout without the reason. |
| 274 | Logs can carry customer data, so a CI credential does not read them. |
| 275 | |
| 276 | The existing app binding (`apiAccountAppBy`, `apicli.go:70`) applies as it |
| 277 | does to every app-bound service credential: another app's path is a 404. |
| 278 | `refuseServiceControlPlane` (`controlplane.go:108`) applies unchanged. |
| 279 | |
| 280 | Ship is not rung-scoped. Shipping uploads a release and never moves a |
| 281 | pointer. |
| 282 | |
| 283 | ## 7. Enforcement at promote and rollback |
| 284 | |
| 285 | ### 7.1 One function |
| 286 | |
| 287 | A pure function decides, so it can be table-tested without a server: |
| 288 | |
| 289 | deployScopeAllows(p Principal, pipelined bool, channel string, hotfix bool) error |
| 290 | |
| 291 | For any principal whose role is not `deploy` it returns nil. For a `deploy` |
| 292 | principal it refuses, in this order: |
| 293 | |
| 294 | 1. `hotfix` is set. This is a deliberate restriction beyond what a scoped |
| 295 | release key carries: a scoped signer may skip ladder order with `Hotfix` |
| 296 | (`internal/release/release.go:945`). What a scoped key cannot do is |
| 297 | `FastTrack`, which waives a bake window and is a separate field |
| 298 | (`release.go:383`). A `deploy` credential gets neither. The console's |
| 299 | promote door has no fast-track input, and the plan pins that with a test |
| 300 | (codex 5). |
| 301 | 2. `release.Rung(channel)` (`ladder.go:55`) says the channel is not a rung. |
| 302 | 3. The rung is `stable`. |
| 303 | 4. The rung is `canary` and the channel is not `canary/<name>`. |
| 304 | `release.Rung("canary")` answers `canary`, so the bare name would |
| 305 | otherwise pass. It is refused for every app, pipelined or not (codex 2). |
| 306 | 5. The rung is not in the credential's rungs. |
| 307 | 6. The app has a pipeline and the rung is not `canary`. |
| 308 | |
| 309 | ### 7.2 Where it runs, and what is guaranteed (codex 1, R2-1, R2-2, R3-1, R3-2, R3-3) |
| 310 | |
| 311 | **The store gives one tool.** Every write in the release store is a |
| 312 | conditional write on a single object: create-only, or replace-only against |
| 313 | an ETag (`internal/release/store.go:33-44`). An ETag is a hash of the |
| 314 | object's bytes (`store.go:279`; S3's is too). So a conditional write proves |
| 315 | one thing: at write time the object's bytes were the bytes the writer |
| 316 | read. It does not prove nobody wrote the object in between. Two rollbacks |
| 317 | inside one second can restore a pointer byte for byte, because signing |
| 318 | truncates `promoted_at` to the second (`release.go:371`) (codex R3-1). |
| 319 | |
| 320 | There is no lock in the console and no write that spans two objects. A |
| 321 | pipeline and a channel pointer are two objects. No ordering of reads makes |
| 322 | "there is no pipeline" and "write this pointer" one atomic step. |
| 323 | |
| 324 | **The two backends differ, and G2 below is only as good as the backend** |
| 325 | (codex R4-1). |
| 326 | |
| 327 | - `S3Store` sends the condition with the write (`store.go:613-628`), and S3 |
| 328 | evaluates it atomically. G2 holds against every writer. |
| 329 | - `DirStore` compares and then renames under a mutex that belongs to one |
| 330 | handle (`store.go:258-262`, `:316-348`). Its own comment calls this |
| 331 | "single-process fidelity, which is what the tests and a one-box |
| 332 | deployment need" (`store.go:254-257`). Two handles on one directory, |
| 333 | such as a console and a direct-mode CLI (`cmd/console/main.go:482`, |
| 334 | `cli.go:1948`), can both pass the comparison and the later rename wins. |
| 335 | On a directory store G2 holds only among writers that share one handle. |
| 336 | |
| 337 | That is an existing limit of every conditional write on a directory |
| 338 | store, for every caller. This spec does not change it and does not depend |
| 339 | on a directory store being safe across processes (§14). |
| 340 | |
| 341 | **What a `deploy` request therefore is.** It is authorized against the |
| 342 | pipeline it read, and it may complete after a pipeline is installed. That |
| 343 | is what an in-flight request is under any authorization system: a policy |
| 344 | change governs the requests that start after it. The mechanism below makes |
| 345 | the in-flight case as narrow as one object's conditional write allows, and |
| 346 | the spec claims nothing beyond that. |
| 347 | |
| 348 | **The mechanism.** The console passes an authorization hook, |
| 349 | `deployScopeAllows` bound to the principal, the channel and the hotfix |
| 350 | flag. The primitives call it with a pipeline they have just read, at |
| 351 | three points: |
| 352 | |
| 353 | 1. **On the ladder's pipeline read** in `PromoteOnLadder` |
| 354 | (`release.go:930`), before the order check. This is not the authority. |
| 355 | It exists so that a pipeline installed after the handler's read is |
| 356 | refused with §7.3's 403, and not with the ladder's 409 about channel |
| 357 | order (codex R3-3). |
| 358 | 2. **After the pointer read and before the write**, in `Signer.Promote` |
| 359 | (`release.go:732`) and `Signer.Rollback` (`release.go:1043`). The |
| 360 | primitive reads the pointer and its ETag as it does today, reads the |
| 361 | pipeline fresh, calls the hook, then signs and writes against that |
| 362 | ETag. This is the authority for a write. |
| 363 | 3. **On the repeat path.** `PromoteOnLadder` returns a no-op when the |
| 364 | channel already points at the requested version (`release.go:887`). |
| 365 | With a hook, it reads the pipeline after that pointer read and calls |
| 366 | the hook before returning (codex R2-2). |
| 367 | |
| 368 | A refusal is a typed error the handler maps to §7.3's 403. A failed |
| 369 | conditional write is returned as a conflict and is not retried inside the |
| 370 | primitive, because a retry must start again from the pointer read. A |
| 371 | caller that passes no hook gets today's behaviour at all three points, |
| 372 | with no extra read. |
| 373 | |
| 374 | The handlers also call `deployScopeAllows` on their own pipeline read, |
| 375 | before the rules switch at `apicli.go:803`. This is what keeps a `deploy` |
| 376 | credential off the proposal path: the handler creates a proposal only |
| 377 | when its own read shows a pipeline that names the channel, and that same |
| 378 | read refuses a `deploy` credential for any channel but `canary/<name>`. |
| 379 | |
| 380 | **What is guaranteed.** For a write by a `deploy` credential to channel C: |
| 381 | |
| 382 | - **G1.** It was authorized against a pipeline read made after the read of |
| 383 | C's pointer. |
| 384 | - **G2.** It lands only if C's pointer, at write time, is byte for byte |
| 385 | what the request read. For a first promote, it lands only if C has no |
| 386 | pointer at write time. This holds to the extent the backend's |
| 387 | conditional write does, as set out above. |
| 388 | |
| 389 | From G1: a pipeline that existed before the request read C, and still |
| 390 | exists when the authorization read is made, is seen. A `deploy` request |
| 391 | for any channel but `canary/<name>` is then refused. A pipeline that was |
| 392 | installed and removed again before the authorization read is not seen, |
| 393 | and nothing requires it to be: the request is judged on the pipeline the |
| 394 | authorization read returned (codex R4-2). |
| 395 | |
| 396 | From G2: a write to C that leaves it different from what the request read |
| 397 | defeats the `deploy` write. That covers Astra's round 2 counterexample, |
| 398 | where an owner promotes B under a new pipeline. |
| 399 | |
| 400 | **What is not guaranteed, in one sentence.** A pipeline installed after |
| 401 | the request's authorization read is not seen, and the write lands if G2 |
| 402 | holds. |
| 403 | |
| 404 | That sentence covers every case found so far, and it is the whole of what |
| 405 | R5 accepts (§16): |
| 406 | |
| 407 | - the pipeline is installed and nothing touches C before the write; |
| 408 | - the pipeline is installed and C is written and restored to identical |
| 409 | bytes, as two rollbacks in one second do (codex R3-1); |
| 410 | - C has no pointer, a pipeline is installed, and the create-only write |
| 411 | lands on a channel that is still empty. |
| 412 | |
| 413 | In each, the pointer the `deploy` write replaces is exactly the state the |
| 414 | request read, and the version was authorized when the request read it. |
| 415 | This is not serializability and the spec does not claim it. Another |
| 416 | operation that reads C before the `deploy` write lands sees the earlier |
| 417 | state, as it would between two people promoting today. |
| 418 | |
| 419 | **A no-op is an observation, not a write** (codex R3-2). A repeat promote |
| 420 | answers "at the moment this request read C, C pointed at this version, |
| 421 | and the credential was in scope on a pipeline read after that". It makes |
| 422 | no conditional write, so it detects no later change to C. If an owner |
| 423 | installs a pipeline and promotes B after the request's reads, the request |
| 424 | still reports the no-op for A. It has changed nothing, and its answer was |
| 425 | true when it was read. No conflict is promised for this path. |
| 426 | |
| 427 | The rung limit depends on none of this. The grant enforces it (§7.4) on |
| 428 | one object, with no race. |
| 429 | |
| 430 | Everything after the check is unchanged. The ladder order, the audit line |
| 431 | and the serial bump all still run for a `deploy` credential. |
| 432 | |
| 433 | ### 7.3 The refusal |
| 434 | |
| 435 | A 403 with a JSON `error` that names the credential and the rungs it holds, |
| 436 | so a CI log says what to fix. The wording goes through copy review (§11). |
| 437 | |
| 438 | ### 7.4 The pointer is signed by a request-local scoped key (codex 6) |
| 439 | |
| 440 | Today the limit on titogo's CI is cryptographic: every box verifies the key |
| 441 | grant, and a pointer signed outside the grant is rejected wherever it |
| 442 | lands. Rev 1 gave that up, because the console signs with its own key |
| 443 | (`s.cfg.Key`). It does not have to. |
| 444 | |
| 445 | For a `deploy` principal, promote and rollback sign like this: |
| 446 | |
| 447 | 1. The console mints a scoped keypair and a root-signed grant with |
| 448 | `release.MintKeyGrant` (`grant.go:88`), for exactly the credential's |
| 449 | rungs, with a lifetime of five minutes. The grant's name identifies the |
| 450 | service credential and satisfies `grantNameRe` (`grant.go:81`). |
| 451 | 2. It builds a `release.ScopedSigner` (`signer.go:42`) from them and passes |
| 452 | that signer to `PromoteOnLadder` or `Rollback`. |
| 453 | 3. The key is never stored and never leaves the request. |
| 454 | |
| 455 | What this buys: |
| 456 | |
| 457 | - A box verifies the pointer against the grant, as it does for titogo's CI |
| 458 | today. If §7.1 were wrong about a rung, the signer refuses |
| 459 | (`Signer.allows`, `signer.go:71`; `Grant.allows`, `grant.go:254`), and a |
| 460 | pointer that somehow got written would still fail verification on every |
| 461 | box. |
| 462 | - Stable gets a refusal that does not depend on §7.1: no grant can name it |
| 463 | (`grant.go:219`) and no grant can sign for it (`grant.go:263`) (codex 4). |
| 464 | - The pointer itself records which credential moved it. |
| 465 | |
| 466 | What it does not buy, stated plainly: |
| 467 | |
| 468 | - It does not defend against a compromised console. The console holds the |
| 469 | root key and could sign anything. |
| 470 | - It does not enforce R2's pipeline rule. A grant knows rungs, not |
| 471 | pipelines, so a pipeline that names `edge` is `edge` to a grant. That |
| 472 | rule rests on §7.1 and §7.2. |
| 473 | - It does not refuse bare `canary`, which a grant accepts. That rests on |
| 474 | §7.1. |
| 475 | |
| 476 | Failure is closed. If the grant cannot be minted or the signer cannot be |
| 477 | built (for instance, the console's key is not a root key), the request is |
| 478 | refused. A `deploy` request never falls back to the root signer. |
| 479 | |
| 480 | A grant's lifetime is checked against the pointer's own `promoted_at` |
| 481 | (`release.go:480`), so a five-minute grant does not expire a pointer |
| 482 | it has already signed. A pointer dated before its grant was minted is |
| 483 | refused, so the grant is minted before the pointer is stamped. |
| 484 | |
| 485 | Ship is unchanged. It uploads immutable artifacts and an unsigned |
| 486 | manifest (`release.Ship`, `release.go:584`, which takes no signer) and |
| 487 | moves no pointer (codex R2-3). |
| 488 | |
| 489 | ## 8. The CLI in CI |
| 490 | |
| 491 | Today the CLI finds a console credential only in a credentials file |
| 492 | (`auth.go:140`). `CARLOS_SERVICE_TOKEN` is printed by `services create` as |
| 493 | a delivery hint and read by nothing in the CLI. |
| 494 | |
| 495 | - `resolveMode` (`apimode.go:93`) reads `CARLOS_SERVICE_TOKEN`. When it is |
| 496 | set, the session uses that token and the console named by `--console` or |
| 497 | `CARLOS_CONSOLE`. The credentials file is not consulted. |
| 498 | - `CARLOS_SERVICE_TOKEN` with no console named is an error that names |
| 499 | `CARLOS_CONSOLE`. There is no default console for a token. |
| 500 | - `CARLOS_SERVICE_TOKEN` together with `CARLOS_DEPLOYMENT_BUCKET` or |
| 501 | `CARLOS_DEPLOYMENT_DIR` is refused, with an error naming both variables. |
| 502 | The two modes carry different authority, and `resolveMode` already |
| 503 | refuses `--console` beside either store variable for the same reason |
| 504 | (`apimode.go:86-99`). |
| 505 | - A bare store variable with no token still selects direct mode silently, |
| 506 | unchanged. |
| 507 | - The token is never accepted as a flag, so it never lands in a process |
| 508 | list or a CI log's command echo. |
| 509 | - `carlos auth whoami` is unchanged and stays person-only. It reads the |
| 510 | credentials file and never calls `resolveMode` (`auth.go:785`), and the |
| 511 | console's whoami door refuses every service token |
| 512 | (`authcli.go:1531`, `authstore.go:515`). Rev 1 promised a service |
| 513 | identity here with no door behind it. CI does not need one, so the |
| 514 | promise is withdrawn rather than built (codex 3). |
| 515 | |
| 516 | **To check in the plan.** `CARLOS_SERVICE_TOKEN` is also the name an app |
| 517 | receives its own service credential under at run time |
| 518 | (`internal/console/svcpage.go:143`). A `carlos` invoked inside an instance |
| 519 | would now pick it up. Confirm that is harmless for every box-side and |
| 520 | in-instance caller, or choose a second name for the CLI. |
| 521 | |
| 522 | ## 9. Mixed versions and rollback |
| 523 | |
| 524 | - **Old console, `deploy` token.** Two checks refuse it, and the first is |
| 525 | the one that fires (codex R3-4). `CLIPrincipal` rejects a stored service |
| 526 | role it cannot parse (`svcprincipal.go:126`), so `cliRoute` answers as it |
| 527 | does for no credential at all, before its role gate. If a role did |
| 528 | resolve, `cliRoute` checks `allow.Permits(p.Role)` before any handler |
| 529 | runs (`authcli.go:430`), and an old binary's sets do not contain |
| 530 | `deploy`. The plan pins both, separately: a stored role the binary does |
| 531 | not know is refused as not signed in at every registered `cliRoute`; and |
| 532 | a known role outside a door's set is refused by the role gate. It must |
| 533 | also confirm both hold in v0.21.0 by reading it. |
| 534 | - **Old console, new column.** An old binary ignores `rungs`. That is safe |
| 535 | only because of the line above: it never lets the credential through. |
| 536 | - **New console, old CLI.** `services create --role deploy` fails in the old |
| 537 | CLI's own role check (`services.go:174`). Promote and ship with a token |
| 538 | need the new CLI for §8 only. |
| 539 | - **Roll order.** Console first, then mint, then CI. |
| 540 | |
| 541 | ## 10. Audit |
| 542 | |
| 543 | No new audit shape. A `deploy` promote writes the same `promote` activity |
| 544 | line as any CLI promote, attributed to the service credential. Minting, |
| 545 | rotating and revoking a `deploy` credential are audited as service |
| 546 | credentials already are, and the mint line records the rungs. |
| 547 | |
| 548 | ## 11. Docs and copy |
| 549 | |
| 550 | - `README.md:170-180` is rewritten: CI uses the console with a `deploy` |
| 551 | credential. Direct-bucket mode is described as the operator's path, with |
| 552 | the four cases in R3. |
| 553 | - `docs/site/cli.md` (the `ci-shipper` example at `:1419`) uses |
| 554 | `--role deploy --rungs canary,edge`. |
| 555 | - Every new user-facing string goes through the copy-review gate before it |
| 556 | is written into a file: CLI usage and help, the mint output, the three |
| 557 | CLI errors in §8, the 403 in §7.3, the browser form, and the two doc |
| 558 | passages above. |
| 559 | |
| 560 | ## 12. Testing |
| 561 | |
| 562 | - **Scope function.** A table over role, pipelined, channel and hotfix, |
| 563 | covering every refusal in §7.1 and each allowed case, including bare |
| 564 | `canary` for both a pipelined and an unpipelined app. |
| 565 | - **Handlers.** For promote and rollback: an in-scope rung succeeds, writes |
| 566 | the audit line and bumps the serial; an out-of-scope rung, `stable`, bare |
| 567 | `canary`, a hotfix, and a non-canary channel on a pipelined app each give |
| 568 | a 403 and write nothing. A pipeline that names `stable` with no passkey |
| 569 | rule still refuses a `deploy` credential. |
| 570 | - **G1 and G2, for promote and for rollback (§7.2).** Each with the store |
| 571 | paused at the named point: |
| 572 | - a pipeline installed before the primitive's pointer read is refused |
| 573 | with §7.3's 403; |
| 574 | - a pipeline that excludes the requested rung, installed between the |
| 575 | handler's read and the ladder's read, is refused with §7.3's 403 and |
| 576 | not the ladder's 409 (codex R3-3); |
| 577 | - a pipeline installed and then an owner's promote of B to C, before |
| 578 | the `deploy` write: conflict, and C still points at B; |
| 579 | - a pipeline installed and then two rollbacks of C that restore its |
| 580 | bytes, before the `deploy` write: the write lands. The test names |
| 581 | this as the accepted in-flight case (codex R3-1); |
| 582 | - a pipeline installed with no write to C, and a first promote onto an |
| 583 | empty C: each lands, named the same way. |
| 584 | These three pin R5. If the gap is ever closed, they flip. |
| 585 | - **No-op (§7.2).** Its own assertions, never a conflict (codex R3-2): |
| 586 | refused when a pipeline existed before the pointer read; a successful |
| 587 | no-op when the pipeline, or an owner's later promote, arrives after the |
| 588 | request's reads. |
| 589 | - **Randomized interleavings.** One example per round was how three |
| 590 | rounds went. The plan adds a test that runs random schedules of `deploy` |
| 591 | promote, `deploy` rollback, owner promote, owner rollback, pipeline |
| 592 | install and pipeline removal against a store that can pause between any |
| 593 | two of a request's steps, over many seeds. |
| 594 | - It records three events for every `deploy` request, independently: the |
| 595 | pointer read (with the bytes read), the authorization read (with the |
| 596 | pipeline returned), and the write (with the bytes replaced). |
| 597 | - The oracle is G1 and G2 and nothing else (codex R4-2). G1: the |
| 598 | authorization read came after the pointer read, and the pipeline it |
| 599 | returned permits the request under §7.1. G2: the bytes replaced equal |
| 600 | the bytes read. It does not assert anything about a pipeline that |
| 601 | existed at some other moment, since a pipeline can be removed again |
| 602 | before the authorization read. |
| 603 | - Two mutants must each fail it, on a schedule the test prints so the |
| 604 | failure can be replayed: the authoritative hook moved before the |
| 605 | pointer read (breaks G1), and the write made unconditional (breaks G2 |
| 606 | when another writer changes C between the read and the write). |
| 607 | - It runs on one shared store handle, which is where G2 holds for a |
| 608 | directory store. It therefore cannot show the two-handle directory |
| 609 | race (codex R4-1), and does not claim to. The S3 path's conditional |
| 610 | write is covered by the existing store tests |
| 611 | (`internal/release/store_test.go`, `store_rgw_test.go`). |
| 612 | - **No hook, no change.** A caller that passes no hook makes the same |
| 613 | store calls in the same order as before, for promote, repeat promote |
| 614 | and rollback. |
| 615 | - **Signer.** A pointer written by a `deploy` promote carries a grant for |
| 616 | exactly the credential's rungs and verifies with `verifyPointer`. With |
| 617 | the scope function stubbed to allow everything, an out-of-rung promote |
| 618 | and a stable promote are still refused by the signer. A grant that |
| 619 | cannot be minted refuses the request, and the root signer is not used. |
| 620 | - **Hotfix and fast-track.** Each is refused, tested separately. |
| 621 | - **Doors.** A test walks every registered `cliRoute` and asserts `deploy` |
| 622 | is permitted on exactly the §6 list. A route added later is refused for |
| 623 | `deploy` unless someone adds it to the list on purpose. |
| 624 | - **Mint.** `deploy` with no app, with no rungs, with `stable`, with an |
| 625 | unknown rung; another role with rungs. Each refused through both doors. |
| 626 | - **Read-back.** A `deploy` row with empty rungs is refused at resolve. |
| 627 | - **Unknown role.** §9's two fail-closed tests. |
| 628 | - **CLI.** Token plus console selects API mode; token with no console |
| 629 | errors; token plus a store variable errors; a bare store variable is |
| 630 | still direct mode. |
| 631 | |
| 632 | ## 13. Rollout |
| 633 | |
| 634 | 1. Merge and release the platform. |
| 635 | 2. Roll Tito's console and boxes. Tito runs its own console. |
| 636 | 3. An owner mints `deploy` for app `titogo`, rungs `canary,edge`. |
| 637 | 4. titogo's CI switches: drops its scoped key and grant and |
| 638 | `CARLOS_DEPLOYMENT_BUCKET` (and its AWS role, if nothing else in the |
| 639 | workflow uses it); sets `CARLOS_SERVICE_TOKEN` and |
| 640 | `CARLOS_CONSOLE`. This is a titogo change, tracked on #229. |
| 641 | 5. Confirm on the box that an edge promote is adopted (staged and |
| 642 | repointed) within seconds, then |
| 643 | remove `carlos-adopt.timer.d/canary-cadence.conf`. |
| 644 | 6. Remove the scoped release key and its grant from the runner's secrets. |
| 645 | The grant expires on its own within its 365-day life. |
| 646 | |
| 647 | **To check before step 4.** |
| 648 | |
| 649 | - The console's ship door caps one request at 256 MiB |
| 650 | (`apiShipMaxBytes`, `apicli.go:1404`). Confirm titogo's artifact fits. |
| 651 | - The GitHub runner can reach Tito's console. |
| 652 | - Which doors `carlos canary` uses in API mode. If it needs an `instance` |
| 653 | door, session canaries keep a separate credential, and that is out of |
| 654 | scope here. |
| 655 | |
| 656 | ## 14. Out of scope, recorded so nobody rediscovers it |
| 657 | |
| 658 | - A `publish` credential can promote `beta` and hotfix past the ladder on an |
| 659 | app with no pipeline (§1). That is existing behaviour for a role that was |
| 660 | designed to be wide. This spec gives CI a narrower role and does not |
| 661 | change `publish`. |
| 662 | - Pipeline channel scopes (R2). If a customer with a pipeline needs CI to |
| 663 | propose onto a pipeline channel, that is a follow-on spec. |
| 664 | - For every caller, the promote handler judges pipeline rules (approvals, |
| 665 | passkey) on its own pipeline read, and `PromoteOnLadder` acts on a second |
| 666 | one. A pipeline changed between the two is judged on the old rules. §7.2 |
| 667 | narrows this for `deploy` to G1 and G2, and leaves it as found for people |
| 668 | and for `publish`. Closing it for everyone means binding pipeline writes and |
| 669 | pointer writes to one generation, which is its own piece of work (§16). |
| 670 | - A pipeline may name `stable` with no passkey rule, and promote then |
| 671 | applies only that channel's rules (§1). Existing behaviour, unchanged. |
| 672 | - A directory store's conditional write is not safe across handles or |
| 673 | processes (§7.2, codex R4-1). It affects every conditional write on a |
| 674 | directory-backed deployment, whoever makes it. Fixing it, with a file |
| 675 | lock or by refusing a second writer, is separate work. |
| 676 | |
| 677 | ## 15. Review dispositions |
| 678 | |
| 679 | Astra round 1, `CODEX-FINDINGS-DEPLOY-CREDENTIAL-R1.md`, against d30a74c2. |
| 680 | Every citation was checked at main `8d8e4bae`. |
| 681 | |
| 682 | | Finding | Severity | Disposition | |
| 683 | | --- | --- | --- | |
| 684 | | R1-1 scope judged on a different pipeline read than the one executed | P1 | Accepted. §7.2: the check runs inside the primitive on its own read. Full serialization of pipeline writes against pointer writes is declined, with the reason in §7.2 and §14. | |
| 685 | | R1-2 bare `canary` passes | P1 | Accepted. §7.1 rule 4. | |
| 686 | | R1-3 `auth whoami` has no service path | P2 | Accepted by withdrawing the claim. §8. | |
| 687 | | R1-4 promote's stable refusal is not unconditional | P2 | Accepted. §1 and §5.2 corrected; the independent refusal is now the grant, §7.4. | |
| 688 | | R1-5 hotfix and fast-track conflated | P2 | Accepted. §7.1 rule 1. | |
| 689 | | R1-6 keep box-verified limits with a request-local scoped signer | P3 | Accepted and adopted. §7.4. | |
| 690 | | R2-1 a write can land over an owner's promote made under a new pipeline | P1 | Accepted in part. §7.2: the hook moves between the pointer read and the write, so that counterexample fails on the pointer's compare-and-swap. The serializability claim is withdrawn and replaced by I1. Full serialization is not built; the remaining window is stated and put to Paul in §16. | |
| 691 | | R2-2 the no-op return skips the hook | P2 | Accepted. §7.2. | |
| 692 | | R2-3 ship does not use a signer; wrong citation for the lifetime check | P2 | Accepted. §7.4. | |
| 693 | | R3-1 a restored ETag makes I1(b) false | P1 | Accepted. §7.2: I1 is replaced by G1 and G2, which claim only what a content-hash conditional write proves. The restored-bytes case is named as part of the in-flight gap. | |
| 694 | | R3-2 the no-op path detects no conflict | P2 | Accepted. §7.2 and §12: a no-op is an observation, with its own assertions. | |
| 695 | | R3-3 the relocated hook loses the scope refusal to a ladder error | P2 | Accepted. §7.2: the hook is also called on the ladder's pipeline read. | |
| 696 | | R3-4 wrong location and missing first refusal in §9 | P2 | Accepted. §9. | |
| 697 | | R4-1 G2 fails across two `DirStore` handles on one directory | P1 | Accepted as a qualification. §7.2 states G2 per backend. The directory store's cross-handle limit is existing, documented in the store, and recorded in §14 as separate work. | |
| 698 | | R4-2 the randomized test's oracle rejects a write that satisfies G1 and G2 | P2 | Accepted. §7.2's inference and §12's oracle are restated on the pipeline the authorization read returned. Mutants must fail on a printed, replayable schedule. | |
| 699 | |
| 700 | ## 16. Requests in flight when a pipeline is installed (R5) |
| 701 | |
| 702 | §7.2 guarantees G1 and G2 and nothing more: a pipeline installed after a |
| 703 | `deploy` request's authorization read is not seen, and the request's write |
| 704 | lands if the pointer is still byte for byte what it read. |
| 705 | |
| 706 | **Ruled (R5, Paul, 2026-10-05): accepted.** This is how an in-flight |
| 707 | request behaves under any authorization system: the policy change governs |
| 708 | requests that start after it. R2 is a rule about which channels CI is |
| 709 | offered. The trust boundary is the rung limit, which the grant enforces |
| 710 | with no race. Reaching the gap takes an owner installing a pipeline while |
| 711 | a CI promote is between two store calls, and the result is what a promote |
| 712 | a moment earlier would have produced. The same gap exists today for every |
| 713 | person's promote against pipeline rules (§14). |
| 714 | |
| 715 | **Not chosen.** Closing it for everyone: stamp a pipeline generation into |
| 716 | every pointer, or add a per-app lock in the store, and take it in pipeline |
| 717 | writes, promotes, rollbacks and proposal execution. That changes the |
| 718 | release store's concurrency model. If it is ever wanted it is its own |
| 719 | spec. |
| 720 | |
amadan