plan: host lifecycle claim Plan 2, versioned storage primitives
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
1 file changed,
+1317
−0
docs/superpowers/plans/2026-10-05-host-lifecycle-claim-2-versioned-storage.md+1317 −0
CI passed — run details
diff --git a/docs/superpowers/plans/2026-10-05-host-lifecycle-claim-2-versioned-storage.md b/docs/superpowers/plans/2026-10-05-host-lifecycle-claim-2-versioned-storage.md| new file mode 100644 |
| index 000000000..db5b48f47 |
| --- /dev/null |
| +++ b/docs/superpowers/plans/2026-10-05-host-lifecycle-claim-2-versioned-storage.md |
| @@ -0,0 +1,1317 @@ |
| +# Host Lifecycle Claim, Plan 2: versioned storage primitives |
| + |
| +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. |
| + |
| +**Goal:** Give the data bucket a small, exact-version storage interface. |
| +It lists every version of every object, delete markers included and |
| +paginated, and copies, reads and deletes **by version id**. It also has a |
| +create-only put. That is everything Plans 3 to 8 need to park data, write |
| +certificates and read pinned versions, without ever issuing a bare-key |
| +destructive call (spec C3). |
| + |
| +**Architecture:** |
| + |
| +- **A new interface, `lease.VersionedStore`.** It sits beside the |
| + existing `MirrorStore`. `MirrorStore` and its two test fakes are left |
| + untouched. |
| +- **The production implementation is the existing `s3Mirror`.** It |
| + reuses the same regional client routing. |
| +- **One exported fake, `leasetest.VersionedBucket`,** models versions, |
| + delete markers and ETag rules, with failure injection. It is the |
| + reference every later plan tests against. |
| +- **An `http.Handler` that speaks the S3 wire protocol over that fake.** |
| + The real `s3Mirror` code is tested against it, using an actual AWS SDK |
| + client, over the wire. A single contract test suite runs against both |
| + the fake directly and the `s3Mirror`. |
| +- **IAM:** the box role gains the three S3 version actions on its own |
| + data bucket. |
| + |
| +**Tech Stack:** Go, aws-sdk-go-v2 `service/s3` v1.105.0 (already in |
| +go.mod), `net/http/httptest`, OpenTofu HCL. |
| + |
| +**Spec:** `docs/superpowers/specs/2026-10-02-host-lifecycle-claim-design.md`, |
| +rev 6.12. Read C3, §6.2 and §13 first. §13 lists the S3 behaviour already |
| +verified on real AWS, and this plan encodes exactly that behaviour in the |
| +fake. |
| + |
| +**Depends on:** nothing. It can be built in parallel with Plan 1's merge. |
| + |
| +## Global Constraints |
| + |
| +- **Every destructive call names a version id.** No method in this |
| + interface deletes or overwrites by bare key. |
| +- **Delete by version id is idempotent.** A version or key that is |
| + already gone is success, not an error (§13 row 3). |
| +- **Listings never return partial results.** A page error fails the |
| + whole listing, with no partial result. A listing returns every version |
| + **and** every delete marker, in key order, with versions newest first |
| + within a key. |
| +- **Copy semantics.** |
| + - Copy reads the source **by version id** (`CopySource` carries |
| + `?versionId=`). |
| + - A copy may also be made create-only at the destination: `If-None-Match: |
| + *` on the copy. A 412 then returns `ErrDestinationExists`. |
| + - A copy never needs the source ETag, but it records the destination's |
| + new version id and ETag. |
| + - A multipart source ETag becomes single-part on copy (§13 row 9). The |
| + fake models this. |
| +- **`PutIfAbsent`** uses `If-None-Match: *`. A 412 returns `ErrExists`. |
| +- **The fake follows §13 exactly:** |
| + - a bare delete adds a delete marker; |
| + - deleting a version removes only that version; |
| + - deleting the current version makes the next newest version current |
| + (a delete marker counts); |
| + - versions are retained. |
| +- **IAM:** `s3:ListBucketVersions` on the data bucket ARN, and |
| + `s3:GetObjectVersion` and `s3:DeleteObjectVersion` on `<data bucket>/*`, |
| + in **both** `infra/modules/region-host/main.tf` and |
| + `infra/modules/region-remote-host/main.tf`. No other grants change. |
| +- **Gate (AGENTS.md):** `go build ./...`, `go vet ./...`, |
| + `go test ./... -count=1` and `gofmt -l .` must all be clean. Use the |
| + default Go cache. In a read-only sandbox, use `GOCACHE=$TMPDIR/gocache`, |
| + never `/tmp`. CI is the authority for any test that only fails in the |
| + sandbox. |
| + |
| +## Review Focus |
| + |
| +- **A listing over 1,000 objects**, where S3 paginates by key and version |
| + marker. Every version must come back exactly once (Task 3). |
| +- **Keys with characters that need escaping** (`+`, spaces, `%`, `?` in |
| + a key) in `CopySource` and in paths. The copy must hit the exact |
| + version of the exact key (Task 4). |
| +- **Deleting the current version when an older one exists.** The older |
| + one becomes current, in both the fake and S3. Callers must expect it |
| + (Tasks 2 and 5). |
| +- **A page error mid-listing** must surface as an error, never as a short |
| + list (Tasks 2 and 3). |
| +- **Copy create-only at a destination that only holds a delete marker.** |
| + S3 lets `If-None-Match: *` succeed there (§13 row 5's sibling), and the |
| + fake must match (Task 2). |
| + |
| +--- |
| + |
| +## File Structure |
| + |
| +| File | Responsibility | |
| +|---|---| |
| +| `internal/lease/versioned.go` | `VersionedStore`, `ObjectVersion`, the errors, and the `s3Mirror` methods | |
| +| `internal/lease/leasetest/bucket.go` | `VersionedBucket`, the in-memory versioned fake (exported, for every later plan) | |
| +| `internal/lease/leasetest/s3server.go` | `NewS3Server(*VersionedBucket) http.Handler`, the S3 wire protocol over the fake | |
| +| `internal/lease/leasetest/contract.go` | `RunVersionedStoreContract(t, newStore)`, the shared behaviour suite | |
| +| `internal/lease/leasetest/bucket_test.go` | Runs the contract against the fake | |
| +| `internal/lease/versioned_test.go` | Runs the contract against `s3Mirror` over `NewS3Server`, plus wire-specific tests | |
| +| `infra/modules/region-host/main.tf`, `infra/modules/region-remote-host/main.tf` | IAM grants | |
| + |
| +--- |
| + |
| +### Task 1: The interface, types and errors |
| + |
| +**Files:** |
| +- Create: `internal/lease/versioned.go` |
| + |
| +**Interfaces:** |
| +- Produces: |
| + - `type ObjectVersion struct { Key, VersionID, ETag string; Size int64; IsLatest, DeleteMarker bool; LastModified time.Time }`; |
| + - `type VersionedStore interface` (below); |
| + - `ErrExists` and `ErrDestinationExists`. |
| + |
| +- [ ] **Step 1: Write the file** |
| + |
| +```go |
| +package lease |
| + |
| +import ( |
| + "context" |
| + "errors" |
| + "time" |
| +) |
| + |
| +// ObjectVersion is one entry of a versioned listing: an object version, or |
| +// a delete marker (DeleteMarker true, no ETag or Size). |
| +type ObjectVersion struct { |
| + Key string |
| + VersionID string |
| + ETag string // unquoted |
| + Size int64 |
| + IsLatest bool |
| + DeleteMarker bool |
| + LastModified time.Time |
| +} |
| + |
| +var ( |
| + // ErrExists is PutIfAbsent finding the key already holds a current |
| + // object version. |
| + ErrExists = errors.New("lease: object already exists") |
| + // ErrDestinationExists is a create-only CopyVersion whose destination |
| + // key already holds a current object version. |
| + ErrDestinationExists = errors.New("lease: copy destination already exists") |
| +) |
| + |
| +// VersionedStore is the data bucket addressed by exact object version (host |
| +// lifecycle claim spec C3, §6.2). No method deletes or overwrites by bare |
| +// key. |
| +type VersionedStore interface { |
| + // ListVersions returns every version and delete marker under prefix, |
| + // in key order, newest first within a key. It reads every page and |
| + // fails on any page error: never a partial listing. |
| + ListVersions(ctx context.Context, bucket, prefix string) ([]ObjectVersion, error) |
| + // GetVersion reads one exact version. A missing version is |
| + // ErrMirrorNotFound. |
| + GetVersion(ctx context.Context, bucket, key, versionID string) ([]byte, error) |
| + // CopyVersion copies one exact source version to dstKey. With |
| + // createOnly the copy carries If-None-Match: *, and an existing |
| + // current destination object gives ErrDestinationExists. It returns |
| + // the destination's new version id and its ETag. |
| + CopyVersion(ctx context.Context, srcBucket, srcKey, srcVersionID, dstBucket, dstKey string, createOnly bool) (dstVersionID, dstETag string, err error) |
| + // DeleteVersion removes one exact version. A version or key already |
| + // gone is success. |
| + DeleteVersion(ctx context.Context, bucket, key, versionID string) error |
| + // PutIfAbsent writes body only if key holds no current object version |
| + // (If-None-Match: *); otherwise ErrExists. It returns the new version |
| + // id. |
| + PutIfAbsent(ctx context.Context, bucket, key string, body []byte) (versionID string, err error) |
| +} |
| +``` |
| + |
| +- [ ] **Step 2: Build** |
| + |
| +Run: `go build ./internal/lease/` |
| +Expected: success. |
| + |
| +- [ ] **Step 3: Commit** |
| + |
| +```bash |
| +git add internal/lease/versioned.go |
| +git commit -m "lease: the VersionedStore interface (exact-version data bucket access)" |
| +``` |
| + |
| +--- |
| + |
| +### Task 2: The fake versioned bucket and the contract suite |
| + |
| +**Files:** |
| +- Create: `internal/lease/leasetest/bucket.go`, `internal/lease/leasetest/contract.go`, `internal/lease/leasetest/bucket_test.go` |
| + |
| +**Interfaces:** |
| +- Consumes: Task 1. |
| +- Produces: |
| + - `type VersionedBucket struct` with `NewVersionedBucket() *VersionedBucket`; |
| + - the methods of `lease.VersionedStore`, with the bucket name argument |
| + checked against a set of known bucket names (`AddBucket(name string)`); |
| + - helpers for later plans: |
| + - `PutMultipart(bucket, key string, body []byte)`, which stores an |
| + ETag with a `-2` suffix; |
| + - `BareDelete(bucket, key string)`, which adds a delete marker; |
| + - `Current(bucket, key string) ([]byte, bool)`; |
| + - failure hooks: `FailListAfterPages int` (fail the listing after this |
| + many pages, 0 = never), `PageSize int` (default 1000), and |
| + `FailOn func(op, key string) error`; |
| + - `RunVersionedStoreContract(t *testing.T, mk func(t *testing.T) (lease.VersionedStore, *VersionedBucket, string))`. |
| + `mk` returns the store under test, the backing fake (for setup and |
| + inspection) and the bucket name. |
| + |
| +**Model:** |
| +- Each key holds a list of versions, newest first. Each version is |
| + `{id, body, etag, size, deleteMarker, modified}`. |
| +- Version ids are `v000001`, `v000002` and so on, from a counter, so |
| + ordering is deterministic. |
| +- The ETag of a put is the hex md5 of the body. For `PutMultipart` it is |
| + md5 + `"-2"`. A copy always gets the single-part md5 of the body. |
| +- "Current" means the newest entry, if it isn't a delete marker. |
| + |
| +- [ ] **Step 1: Write the contract suite (it fails: nothing implements it yet)** |
| + |
| +```go |
| +package leasetest |
| + |
| +import ( |
| + "bytes" |
| + "context" |
| + "errors" |
| + "fmt" |
| + "testing" |
| + |
| + "github.com/carlosframework/platform/internal/lease" |
| +) |
| + |
| +// RunVersionedStoreContract is the behaviour every lease.VersionedStore must |
| +// have (spec §13). mk returns the store under test, the fake that backs it |
| +// (used to set up and inspect state) and the bucket name. |
| +func RunVersionedStoreContract(t *testing.T, mk func(t *testing.T) (lease.VersionedStore, *VersionedBucket, string)) { |
| + ctx := context.Background() |
| + |
| + t.Run("versions and delete markers, newest first", func(t *testing.T) { |
| + st, fb, b := mk(t) |
| + fb.Put(b, "p/a", []byte("one")) |
| + fb.Put(b, "p/a", []byte("two")) |
| + fb.BareDelete(b, "p/a") |
| + fb.Put(b, "p/b", []byte("x")) |
| + fb.Put(b, "q/c", []byte("other prefix")) |
| + vs, err := st.ListVersions(ctx, b, "p/") |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + var got []string |
| + for _, v := range vs { |
| + got = append(got, fmt.Sprintf("%s:%v:%v", v.Key, v.DeleteMarker, v.IsLatest)) |
| + } |
| + want := []string{"p/a:true:true", "p/a:false:false", "p/a:false:false", "p/b:false:true"} |
| + if fmt.Sprint(got) != fmt.Sprint(want) { |
| + t.Fatalf("ListVersions = %v, want %v", got, want) |
| + } |
| + }) |
| + |
| + t.Run("listing spans pages and fails whole on a page error", func(t *testing.T) { |
| + st, fb, b := mk(t) |
| + fb.PageSize = 3 |
| + for i := 0; i < 10; i++ { |
| + fb.Put(b, fmt.Sprintf("p/k%02d", i), []byte{byte(i)}) |
| + } |
| + vs, err := st.ListVersions(ctx, b, "p/") |
| + if err != nil || len(vs) != 10 { |
| + t.Fatalf("paged listing = %d, %v; want 10", len(vs), err) |
| + } |
| + fb.FailListAfterPages = 2 |
| + if vs, err := st.ListVersions(ctx, b, "p/"); err == nil { |
| + t.Fatalf("listing with a failing page returned %d entries and no error", len(vs)) |
| + } |
| + }) |
| + |
| + t.Run("get and copy by exact version", func(t *testing.T) { |
| + st, fb, b := mk(t) |
| + fb.Put(b, "src/a b+c%", []byte("old")) |
| + v1 := fb.LatestID(b, "src/a b+c%") |
| + fb.Put(b, "src/a b+c%", []byte("new")) |
| + body, err := st.GetVersion(ctx, b, "src/a b+c%", v1) |
| + if err != nil || string(body) != "old" { |
| + t.Fatalf("GetVersion(v1) = %q, %v", body, err) |
| + } |
| + dv, etag, err := st.CopyVersion(ctx, b, "src/a b+c%", v1, b, "dst/x", true) |
| + if err != nil || dv == "" || etag == "" { |
| + t.Fatalf("CopyVersion = %q %q %v", dv, etag, err) |
| + } |
| + if cur, ok := fb.Current(b, "dst/x"); !ok || string(cur) != "old" { |
| + t.Fatalf("copied body = %q", cur) |
| + } |
| + if _, _, err := st.CopyVersion(ctx, b, "src/a b+c%", v1, b, "dst/x", true); !errors.Is(err, lease.ErrDestinationExists) { |
| + t.Fatalf("create-only copy onto an existing object = %v", err) |
| + } |
| + if _, err := st.GetVersion(ctx, b, "src/a b+c%", "v999999"); !errors.Is(err, lease.ErrMirrorNotFound) { |
| + t.Fatalf("missing version = %v", err) |
| + } |
| + }) |
| + |
| + t.Run("create-only copy succeeds over a delete marker", func(t *testing.T) { |
| + st, fb, b := mk(t) |
| + fb.Put(b, "src/k", []byte("s")) |
| + sv := fb.LatestID(b, "src/k") |
| + fb.Put(b, "dst/k", []byte("gone")) |
| + fb.BareDelete(b, "dst/k") |
| + if _, _, err := st.CopyVersion(ctx, b, "src/k", sv, b, "dst/k", true); err != nil { |
| + t.Fatalf("create-only copy over a delete marker = %v", err) |
| + } |
| + }) |
| + |
| + t.Run("multipart source etag becomes single-part on copy", func(t *testing.T) { |
| + st, fb, b := mk(t) |
| + fb.PutMultipart(b, "src/m", []byte("payload")) |
| + sv := fb.LatestID(b, "src/m") |
| + _, etag, err := st.CopyVersion(ctx, b, "src/m", sv, b, "dst/m", false) |
| + if err != nil || bytes.Contains([]byte(etag), []byte("-")) { |
| + t.Fatalf("copied etag = %q, %v; want single-part", etag, err) |
| + } |
| + }) |
| + |
| + t.Run("delete by version is exact and idempotent", func(t *testing.T) { |
| + st, fb, b := mk(t) |
| + fb.Put(b, "k", []byte("one")) |
| + v1 := fb.LatestID(b, "k") |
| + fb.Put(b, "k", []byte("two")) |
| + v2 := fb.LatestID(b, "k") |
| + if err := st.DeleteVersion(ctx, b, "k", v2); err != nil { |
| + t.Fatal(err) |
| + } |
| + if cur, ok := fb.Current(b, "k"); !ok || string(cur) != "one" { |
| + t.Fatalf("after deleting the current version, current = %q %v; want the older version", cur, ok) |
| + } |
| + if err := st.DeleteVersion(ctx, b, "k", v2); err != nil { |
| + t.Fatalf("deleting an already-deleted version = %v, want success", err) |
| + } |
| + if err := st.DeleteVersion(ctx, b, "k", v1); err != nil { |
| + t.Fatal(err) |
| + } |
| + if _, ok := fb.Current(b, "k"); ok { |
| + t.Fatal("key still has a current version after deleting every version") |
| + } |
| + if err := st.DeleteVersion(ctx, b, "never", "v000001"); err != nil { |
| + t.Fatalf("deleting a version of a missing key = %v, want success", err) |
| + } |
| + }) |
| + |
| + t.Run("put if absent", func(t *testing.T) { |
| + st, fb, b := mk(t) |
| + v, err := st.PutIfAbsent(ctx, b, "m/closed.json", []byte("{}")) |
| + if err != nil || v == "" { |
| + t.Fatalf("first PutIfAbsent = %q %v", v, err) |
| + } |
| + if _, err := st.PutIfAbsent(ctx, b, "m/closed.json", []byte("{}")); !errors.Is(err, lease.ErrExists) { |
| + t.Fatalf("second PutIfAbsent = %v, want ErrExists", err) |
| + } |
| + fb.BareDelete(b, "m/closed.json") |
| + if _, err := st.PutIfAbsent(ctx, b, "m/closed.json", []byte("{}")); err != nil { |
| + t.Fatalf("PutIfAbsent over a delete marker = %v, want success", err) |
| + } |
| + }) |
| +} |
| +``` |
| + |
| +- [ ] **Step 2: Write `bucket_test.go`, which runs the contract against the fake** |
| + |
| +```go |
| +package leasetest |
| + |
| +import ( |
| + "testing" |
| + |
| + "github.com/carlosframework/platform/internal/lease" |
| +) |
| + |
| +func TestVersionedBucketContract(t *testing.T) { |
| + RunVersionedStoreContract(t, func(t *testing.T) (lease.VersionedStore, *VersionedBucket, string) { |
| + fb := NewVersionedBucket() |
| + fb.AddBucket("data") |
| + return fb, fb, "data" |
| + }) |
| +} |
| +``` |
| + |
| +Run: `go test ./internal/lease/leasetest/ -count=1` |
| +Expected: FAIL (`undefined: NewVersionedBucket`). |
| + |
| +- [ ] **Step 3: Implement `bucket.go`** |
| + |
| +```go |
| +// Package leasetest holds test doubles for internal/lease. Its |
| +// VersionedBucket is the reference model of a versioned S3 bucket that |
| +// every plan of the host lifecycle claim tests against (spec §13). |
| +package leasetest |
| + |
| +import ( |
| + "context" |
| + "crypto/md5" |
| + "encoding/hex" |
| + "fmt" |
| + "sort" |
| + "strings" |
| + "sync" |
| + "time" |
| + |
| + "github.com/carlosframework/platform/internal/lease" |
| +) |
| + |
| +type version struct { |
| + id string |
| + body []byte |
| + etag string |
| + deleteMarker bool |
| + modified time.Time |
| +} |
| + |
| +// VersionedBucket is an in-memory, versioned, multi-bucket S3 model. |
| +type VersionedBucket struct { |
| + mu sync.Mutex |
| + buckets map[string]map[string][]version // bucket -> key -> versions, newest first |
| + next int |
| + clock time.Time |
| + |
| + // PageSize is the listing page size the S3 server and ListVersions |
| + // use (default 1000). |
| + PageSize int |
| + // FailListAfterPages fails a listing once this many pages have been |
| + // served (0: never). |
| + FailListAfterPages int |
| + // FailOn, when set, is consulted before every operation, with op one |
| + // of "list", "get", "copy", "delete" or "put" and the key; a non-nil |
| + // result fails the call. |
| + FailOn func(op, key string) error |
| +} |
| + |
| +func NewVersionedBucket() *VersionedBucket { |
| + return &VersionedBucket{buckets: map[string]map[string][]version{}, PageSize: 1000, |
| + clock: time.Date(2026, 10, 5, 0, 0, 0, 0, time.UTC)} |
| +} |
| + |
| +func (f *VersionedBucket) AddBucket(name string) { |
| + f.mu.Lock() |
| + defer f.mu.Unlock() |
| + if f.buckets[name] == nil { |
| + f.buckets[name] = map[string][]version{} |
| + } |
| +} |
| + |
| +func (f *VersionedBucket) newID() string { f.next++; return fmt.Sprintf("v%06d", f.next) } |
| +func (f *VersionedBucket) tick() time.Time { f.clock = f.clock.Add(time.Second); return f.clock } |
| +func md5hex(b []byte) string { s := md5.Sum(b); return hex.EncodeToString(s[:]) } |
| + |
| +func (f *VersionedBucket) objs(bucket string) (map[string][]version, error) { |
| + o, ok := f.buckets[bucket] |
| + if !ok { |
| + return nil, fmt.Errorf("leasetest: no bucket %q", bucket) |
| + } |
| + return o, nil |
| +} |
| + |
| +func (f *VersionedBucket) fail(op, key string) error { |
| + if f.FailOn != nil { |
| + return f.FailOn(op, key) |
| + } |
| + return nil |
| +} |
| + |
| +func (f *VersionedBucket) push(bucket, key string, v version) string { |
| + o := f.buckets[bucket] |
| + v.id, v.modified = f.newID(), f.tick() |
| + o[key] = append([]version{v}, o[key]...) |
| + return v.id |
| +} |
| + |
| +// Put stores a new current version (single-part ETag). |
| +func (f *VersionedBucket) Put(bucket, key string, body []byte) string { |
| + f.mu.Lock() |
| + defer f.mu.Unlock() |
| + return f.push(bucket, key, version{body: append([]byte(nil), body...), etag: md5hex(body)}) |
| +} |
| + |
| +// PutMultipart stores a new current version with a multipart-shaped ETag. |
| +func (f *VersionedBucket) PutMultipart(bucket, key string, body []byte) string { |
| + f.mu.Lock() |
| + defer f.mu.Unlock() |
| + return f.push(bucket, key, version{body: append([]byte(nil), body...), etag: md5hex(body) + "-2"}) |
| +} |
| + |
| +// BareDelete adds a delete marker, as a delete without a version id does. |
| +func (f *VersionedBucket) BareDelete(bucket, key string) string { |
| + f.mu.Lock() |
| + defer f.mu.Unlock() |
| + return f.push(bucket, key, version{deleteMarker: true}) |
| +} |
| + |
| +// LatestID is the id of the newest entry (version or marker) of key. |
| +func (f *VersionedBucket) LatestID(bucket, key string) string { |
| + f.mu.Lock() |
| + defer f.mu.Unlock() |
| + if vs := f.buckets[bucket][key]; len(vs) > 0 { |
| + return vs[0].id |
| + } |
| + return "" |
| +} |
| + |
| +// Current returns the current object body: false when the key is absent or |
| +// its newest entry is a delete marker. |
| +func (f *VersionedBucket) Current(bucket, key string) ([]byte, bool) { |
| + f.mu.Lock() |
| + defer f.mu.Unlock() |
| + vs := f.buckets[bucket][key] |
| + if len(vs) == 0 || vs[0].deleteMarker { |
| + return nil, false |
| + } |
| + return vs[0].body, true |
| +} |
| + |
| +func (f *VersionedBucket) hasCurrent(bucket, key string) bool { |
| + vs := f.buckets[bucket][key] |
| + return len(vs) > 0 && !vs[0].deleteMarker |
| +} |
| + |
| +// listPage returns one page of the listing and whether it is truncated; |
| +// page counts served pages for FailListAfterPages. |
| +func (f *VersionedBucket) listPage(bucket, prefix string, start, page int) ([]lease.ObjectVersion, int, bool, error) { |
| + if f.FailListAfterPages > 0 && page >= f.FailListAfterPages { |
| + return nil, 0, false, fmt.Errorf("leasetest: injected listing failure on page %d", page+1) |
| + } |
| + if err := f.fail("list", prefix); err != nil { |
| + return nil, 0, false, err |
| + } |
| + o, err := f.objs(bucket) |
| + if err != nil { |
| + return nil, 0, false, err |
| + } |
| + keys := make([]string, 0, len(o)) |
| + for k := range o { |
| + if strings.HasPrefix(k, prefix) { |
| + keys = append(keys, k) |
| + } |
| + } |
| + sort.Strings(keys) |
| + var all []lease.ObjectVersion |
| + for _, k := range keys { |
| + for i, v := range o[k] { |
| + ov := lease.ObjectVersion{Key: k, VersionID: v.id, IsLatest: i == 0, DeleteMarker: v.deleteMarker, LastModified: v.modified} |
| + if !v.deleteMarker { |
| + ov.ETag, ov.Size = v.etag, int64(len(v.body)) |
| + } |
| + all = append(all, ov) |
| + } |
| + } |
| + size := f.PageSize |
| + if size <= 0 { |
| + size = 1000 |
| + } |
| + end := start + size |
| + if end >= len(all) { |
| + return all[start:], len(all), false, nil |
| + } |
| + return all[start:end], end, true, nil |
| +} |
| + |
| +func (f *VersionedBucket) ListVersions(ctx context.Context, bucket, prefix string) ([]lease.ObjectVersion, error) { |
| + f.mu.Lock() |
| + defer f.mu.Unlock() |
| + var out []lease.ObjectVersion |
| + for start, page := 0, 0; ; page++ { |
| + vs, next, more, err := f.listPage(bucket, prefix, start, page) |
| + if err != nil { |
| + return nil, err |
| + } |
| + out = append(out, vs...) |
| + if !more { |
| + return out, nil |
| + } |
| + start = next |
| + } |
| +} |
| + |
| +func (f *VersionedBucket) find(bucket, key, id string) (version, bool) { |
| + for _, v := range f.buckets[bucket][key] { |
| + if v.id == id { |
| + return v, true |
| + } |
| + } |
| + return version{}, false |
| +} |
| + |
| +func (f *VersionedBucket) GetVersion(ctx context.Context, bucket, key, versionID string) ([]byte, error) { |
| + f.mu.Lock() |
| + defer f.mu.Unlock() |
| + if err := f.fail("get", key); err != nil { |
| + return nil, err |
| + } |
| + if _, err := f.objs(bucket); err != nil { |
| + return nil, err |
| + } |
| + v, ok := f.find(bucket, key, versionID) |
| + if !ok || v.deleteMarker { |
| + return nil, lease.ErrMirrorNotFound |
| + } |
| + return append([]byte(nil), v.body...), nil |
| +} |
| + |
| +func (f *VersionedBucket) CopyVersion(ctx context.Context, srcBucket, srcKey, srcVersionID, dstBucket, dstKey string, createOnly bool) (string, string, error) { |
| + f.mu.Lock() |
| + defer f.mu.Unlock() |
| + if err := f.fail("copy", srcKey); err != nil { |
| + return "", "", err |
| + } |
| + if _, err := f.objs(srcBucket); err != nil { |
| + return "", "", err |
| + } |
| + if _, err := f.objs(dstBucket); err != nil { |
| + return "", "", err |
| + } |
| + v, ok := f.find(srcBucket, srcKey, srcVersionID) |
| + if !ok || v.deleteMarker { |
| + return "", "", lease.ErrMirrorNotFound |
| + } |
| + if createOnly && f.hasCurrent(dstBucket, dstKey) { |
| + return "", "", lease.ErrDestinationExists |
| + } |
| + etag := md5hex(v.body) // a copy is always single-part (§13 row 9) |
| + id := f.push(dstBucket, dstKey, version{body: append([]byte(nil), v.body...), etag: etag}) |
| + return id, etag, nil |
| +} |
| + |
| +func (f *VersionedBucket) DeleteVersion(ctx context.Context, bucket, key, versionID string) error { |
| + f.mu.Lock() |
| + defer f.mu.Unlock() |
| + if err := f.fail("delete", key); err != nil { |
| + return err |
| + } |
| + o, err := f.objs(bucket) |
| + if err != nil { |
| + return err |
| + } |
| + vs := o[key] |
| + for i, v := range vs { |
| + if v.id == versionID { |
| + o[key] = append(vs[:i:i], vs[i+1:]...) |
| + if len(o[key]) == 0 { |
| + delete(o, key) |
| + } |
| + return nil |
| + } |
| + } |
| + return nil // already gone: success |
| +} |
| + |
| +func (f *VersionedBucket) PutIfAbsent(ctx context.Context, bucket, key string, body []byte) (string, error) { |
| + f.mu.Lock() |
| + defer f.mu.Unlock() |
| + if err := f.fail("put", key); err != nil { |
| + return "", err |
| + } |
| + if _, err := f.objs(bucket); err != nil { |
| + return "", err |
| + } |
| + if f.hasCurrent(bucket, key) { |
| + return "", lease.ErrExists |
| + } |
| + return f.push(bucket, key, version{body: append([]byte(nil), body...), etag: md5hex(body)}), nil |
| +} |
| + |
| +var _ lease.VersionedStore = (*VersionedBucket)(nil) |
| +``` |
| + |
| +- [ ] **Step 4: Run the contract against the fake** |
| + |
| +Run: `go test ./internal/lease/leasetest/ -count=1 -race` |
| +Expected: PASS (every subtest). |
| + |
| +- [ ] **Step 5: Commit** |
| + |
| +```bash |
| +git add internal/lease/leasetest/ |
| +git commit -m "leasetest: the versioned bucket model and the VersionedStore contract suite (spec §13)" |
| +``` |
| + |
| +--- |
| + |
| +### Task 3: The S3 server over the fake, and `s3Mirror.ListVersions` |
| + |
| +**Files:** |
| +- Create: `internal/lease/leasetest/s3server.go`, `internal/lease/versioned_test.go` |
| +- Modify: `internal/lease/versioned.go` (add `s3Mirror.ListVersions`) |
| + |
| +**Interfaces:** |
| +- Consumes: Tasks 1 and 2, and the existing `s3Mirror` in |
| + `internal/lease/mirror.go` (`clientFor`, the `byBucket` memo, |
| + `normalizeETag`). |
| +- Produces: |
| + - `leasetest.NewS3Server(f *VersionedBucket) http.Handler`; |
| + - `func (m *s3Mirror) ListVersions(ctx, bucket, prefix string) ([]ObjectVersion, error)`; |
| + - the test helper `newWireMirror(t) (*s3Mirror, *leasetest.VersionedBucket)` |
| + in `versioned_test.go`. |
| + |
| +**The server handles path-style requests, `/{bucket}/{key...}`:** |
| +- `GET /{bucket}?versions&prefix=&key-marker=&version-id-marker=` |
| + returns a `ListVersionsResult` XML page. The marker pair resumes after |
| + the given (key, version id). |
| +- `GET /{bucket}/{key}?versionId=` returns the body. A missing version |
| + gets 404 with `<Code>NoSuchVersion</Code>`. |
| +- `PUT /{bucket}/{key}` with `x-amz-copy-source: /{srcBucket}/{escaped key}?versionId=…` |
| + is a copy: |
| + - `If-None-Match: *` makes it create-only (412 `PreconditionFailed` |
| + when a current version exists); |
| + - it returns `CopyObjectResult` XML with ETag, and the header |
| + `x-amz-version-id`. |
| +- `PUT /{bucket}/{key}` without the copy header is a put, with |
| + `If-None-Match: *` honoured. It returns header `x-amz-version-id`. |
| +- `DELETE /{bucket}/{key}?versionId=` deletes that version and returns |
| + 204. A missing version also returns 204. A DELETE **without** |
| + `versionId` adds a delete marker; the interface never sends one, but |
| + the server models it. |
| +- Anything else gets 405. |
| + |
| +Paging is built on the fake's `listPage`. The server uses the fake's |
| +`PageSize` and `FailListAfterPages`, and an injected failure returns 500 |
| +`<Code>InternalError</Code>`. |
| + |
| +- [ ] **Step 1: Write the failing wire test** |
| + |
| +```go |
| +package lease |
| + |
| +import ( |
| + "context" |
| + "net/http/httptest" |
| + "testing" |
| + |
| + "github.com/aws/aws-sdk-go-v2/aws" |
| + "github.com/aws/aws-sdk-go-v2/credentials" |
| + awss3 "github.com/aws/aws-sdk-go-v2/service/s3" |
| + |
| + "github.com/carlosframework/platform/internal/lease/leasetest" |
| +) |
| + |
| +// newWireMirror is an s3Mirror whose clients talk to NewS3Server over the |
| +// wire, so the production request building and response parsing are what |
| +// is tested. |
| +func newWireMirror(t *testing.T) (*s3Mirror, *leasetest.VersionedBucket) { |
| + t.Helper() |
| + fb := leasetest.NewVersionedBucket() |
| + fb.AddBucket("data") |
| + fb.AddBucket("other") |
| + srv := httptest.NewServer(leasetest.NewS3Server(fb)) |
| + t.Cleanup(srv.Close) |
| + c := awss3.New(awss3.Options{ |
| + Region: "eu-west-1", BaseEndpoint: aws.String(srv.URL), UsePathStyle: true, |
| + Credentials: credentials.NewStaticCredentialsProvider("k", "s", ""), |
| + RetryMaxAttempts: 1, |
| + }) |
| + return &s3Mirror{src: c, dst: c, byBucket: map[string]*awss3.Client{}}, fb |
| +} |
| + |
| +func TestS3MirrorVersionedContract(t *testing.T) { |
| + leasetest.RunVersionedStoreContract(t, func(t *testing.T) (VersionedStore, *leasetest.VersionedBucket, string) { |
| + m, fb := newWireMirror(t) |
| + return m, fb, "data" |
| + }) |
| +} |
| + |
| +func TestS3MirrorListVersionsPagesAcrossMarkers(t *testing.T) { |
| + m, fb := newWireMirror(t) |
| + fb.PageSize = 2 |
| + for i := 0; i < 3; i++ { |
| + fb.Put("data", "p/a", []byte{byte(i)}) // three versions of one key span pages |
| + } |
| + fb.Put("data", "p/b", []byte("b")) |
| + vs, err := m.ListVersions(context.Background(), "data", "p/") |
| + if err != nil || len(vs) != 4 { |
| + t.Fatalf("ListVersions over pages = %d, %v; want 4", len(vs), err) |
| + } |
| +} |
| +``` |
| + |
| +Run: `go test ./internal/lease/ -run 'TestS3MirrorVersioned|TestS3MirrorListVersions' -count=1` |
| +Expected: FAIL (`undefined: leasetest.NewS3Server`). |
| + |
| +- [ ] **Step 2: Implement `s3server.go`** |
| + |
| +```go |
| +package leasetest |
| + |
| +import ( |
| + "encoding/xml" |
| + "io" |
| + "net/http" |
| + "net/url" |
| + "strings" |
| + "time" |
| +) |
| + |
| +type xmlVersion struct { |
| + XMLName xml.Name `xml:"Version"` |
| + Key string |
| + VersionId string |
| + IsLatest bool |
| + LastModified string |
| + ETag string |
| + Size int64 |
| +} |
| + |
| +type xmlMarker struct { |
| + XMLName xml.Name `xml:"DeleteMarker"` |
| + Key string |
| + VersionId string |
| + IsLatest bool |
| + LastModified string |
| +} |
| + |
| +type listVersionsResult struct { |
| + XMLName xml.Name `xml:"ListVersionsResult"` |
| + Name string |
| + Prefix string |
| + KeyMarker string |
| + VersionIdMarker string |
| + NextKeyMarker string `xml:",omitempty"` |
| + NextVersionIdMarker string `xml:",omitempty"` |
| + MaxKeys int |
| + IsTruncated bool |
| + Entries []any |
| +} |
| + |
| +func s3err(w http.ResponseWriter, status int, code string) { |
| + w.Header().Set("Content-Type", "application/xml") |
| + w.WriteHeader(status) |
| + io.WriteString(w, "<Error><Code>"+code+"</Code><Message>"+code+"</Message></Error>") |
| +} |
| + |
| +// NewS3Server speaks the subset of the S3 wire protocol VersionedStore uses, |
| +// path-style, over f. |
| +func NewS3Server(f *VersionedBucket) http.Handler { |
| + return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { |
| + path := strings.TrimPrefix(r.URL.EscapedPath(), "/") |
| + bucket, rawKey, _ := strings.Cut(path, "/") |
| + key, err := url.PathUnescape(rawKey) |
| + if err != nil { |
| + s3err(w, http.StatusBadRequest, "InvalidURI") |
| + return |
| + } |
| + q := r.URL.Query() |
| + f.mu.Lock() |
| + _, known := f.buckets[bucket] |
| + f.mu.Unlock() |
| + if !known { |
| + s3err(w, http.StatusNotFound, "NoSuchBucket") |
| + return |
| + } |
| + switch { |
| + case r.Method == http.MethodGet && key == "" && q.Has("versions"): |
| + f.serveList(w, bucket, q) |
| + case r.Method == http.MethodGet && key != "": |
| + body, err := f.GetVersion(r.Context(), bucket, key, q.Get("versionId")) |
| + if err != nil { |
| + s3err(w, http.StatusNotFound, "NoSuchVersion") |
| + return |
| + } |
| + w.Write(body) |
| + case r.Method == http.MethodPut && r.Header.Get("x-amz-copy-source") != "": |
| + f.serveCopy(w, r, bucket, key) |
| + case r.Method == http.MethodPut: |
| + body, _ := io.ReadAll(r.Body) |
| + f.mu.Lock() |
| + if r.Header.Get("If-None-Match") == "*" && f.hasCurrent(bucket, key) { |
| + f.mu.Unlock() |
| + s3err(w, http.StatusPreconditionFailed, "PreconditionFailed") |
| + return |
| + } |
| + id := f.push(bucket, key, version{body: body, etag: md5hex(body)}) |
| + f.mu.Unlock() |
| + w.Header().Set("x-amz-version-id", id) |
| + w.Header().Set("ETag", `"`+md5hex(body)+`"`) |
| + case r.Method == http.MethodDelete && key != "": |
| + if v := q.Get("versionId"); v != "" { |
| + _ = f.DeleteVersion(r.Context(), bucket, key, v) |
| + } else { |
| + f.BareDelete(bucket, key) |
| + } |
| + w.WriteHeader(http.StatusNoContent) |
| + default: |
| + s3err(w, http.StatusMethodNotAllowed, "MethodNotAllowed") |
| + } |
| + }) |
| +} |
| + |
| +func (f *VersionedBucket) serveList(w http.ResponseWriter, bucket string, q url.Values) { |
| + prefix, km, vm := q.Get("prefix"), q.Get("key-marker"), q.Get("version-id-marker") |
| + f.mu.Lock() |
| + // Resume after (km, vm): find its position in the full ordered listing. |
| + all, _, _, err := f.listPageAll(bucket, prefix) |
| + start := 0 |
| + if err == nil && km != "" { |
| + for i, v := range all { |
| + if v.Key == km && v.VersionID == vm { |
| + start = i + 1 |
| + break |
| + } |
| + } |
| + } |
| + page := 0 |
| + if start > 0 { |
| + page = (start + f.pageSize() - 1) / f.pageSize() |
| + } |
| + var vs []objectRow |
| + var more bool |
| + if err == nil { |
| + vs, more, err = f.slice(all, start, page) |
| + } |
| + f.mu.Unlock() |
| + if err != nil { |
| + s3err(w, http.StatusInternalServerError, "InternalError") |
| + return |
| + } |
| + res := listVersionsResult{Name: bucket, Prefix: prefix, KeyMarker: km, VersionIdMarker: vm, MaxKeys: f.pageSize(), IsTruncated: more} |
| + for _, v := range vs { |
| + lm := v.LastModified.UTC().Format(time.RFC3339) |
| + if v.DeleteMarker { |
| + res.Entries = append(res.Entries, xmlMarker{Key: v.Key, VersionId: v.VersionID, IsLatest: v.IsLatest, LastModified: lm}) |
| + } else { |
| + res.Entries = append(res.Entries, xmlVersion{Key: v.Key, VersionId: v.VersionID, IsLatest: v.IsLatest, LastModified: lm, ETag: `"` + v.ETag + `"`, Size: v.Size}) |
| + } |
| + } |
| + if more && len(vs) > 0 { |
| + last := vs[len(vs)-1] |
| + res.NextKeyMarker, res.NextVersionIdMarker = last.Key, last.VersionID |
| + } |
| + w.Header().Set("Content-Type", "application/xml") |
| + xml.NewEncoder(w).Encode(res) |
| +} |
| + |
| +func (f *VersionedBucket) serveCopy(w http.ResponseWriter, r *http.Request, bucket, key string) { |
| + src := strings.TrimPrefix(r.Header.Get("x-amz-copy-source"), "/") |
| + srcPath, rawQ, _ := strings.Cut(src, "?") |
| + sq, _ := url.ParseQuery(rawQ) |
| + sb, sk, _ := strings.Cut(srcPath, "/") |
| + srcKey, err := url.PathUnescape(sk) |
| + if err != nil { |
| + s3err(w, http.StatusBadRequest, "InvalidArgument") |
| + return |
| + } |
| + id, etag, err := f.CopyVersion(r.Context(), sb, srcKey, sq.Get("versionId"), bucket, key, r.Header.Get("If-None-Match") == "*") |
| + switch { |
| + case err == nil: |
| + case strings.Contains(err.Error(), "already exists"): |
| + s3err(w, http.StatusPreconditionFailed, "PreconditionFailed") |
| + return |
| + default: |
| + s3err(w, http.StatusNotFound, "NoSuchVersion") |
| + return |
| + } |
| + w.Header().Set("x-amz-version-id", id) |
| + w.Header().Set("Content-Type", "application/xml") |
| + io.WriteString(w, `<CopyObjectResult><ETag>"`+etag+`"</ETag><LastModified>`+time.Now().UTC().Format(time.RFC3339)+`</LastModified></CopyObjectResult>`) |
| +} |
| +``` |
| + |
| +**Two small additions to `bucket.go`** support the server: `pageSize()`, |
| +and a `listPageAll` and `slice` pair that splits the ordered listing. |
| +Implement them so that `ListVersions` and the server share **one** |
| +ordering and **one** page-failure rule (`FailListAfterPages`), with |
| +`objectRow = lease.ObjectVersion`. Keep `listPage` (Task 2) as a thin |
| +wrapper over them. |
| + |
| +- [ ] **Step 3: Implement `s3Mirror.ListVersions` in `versioned.go`** |
| + |
| +```go |
| +func (m *s3Mirror) ListVersions(ctx context.Context, bucket, prefix string) ([]ObjectVersion, error) { |
| + c := m.clientFor(bucket, m.src) |
| + p := awss3.NewListObjectVersionsPaginator(c, &awss3.ListObjectVersionsInput{ |
| + Bucket: aws.String(bucket), Prefix: aws.String(prefix), |
| + }) |
| + var out []ObjectVersion |
| + for p.HasMorePages() { |
| + page, err := p.NextPage(ctx) |
| + if err != nil { |
| + return nil, fmt.Errorf("lease: list versions %s/%s: %w", bucket, prefix, err) // never partial |
| + } |
| + for _, v := range page.Versions { |
| + out = append(out, ObjectVersion{Key: aws.ToString(v.Key), VersionID: aws.ToString(v.VersionId), |
| + ETag: normalizeETag(aws.ToString(v.ETag)), Size: aws.ToInt64(v.Size), |
| + IsLatest: aws.ToBool(v.IsLatest), LastModified: aws.ToTime(v.LastModified)}) |
| + } |
| + for _, d := range page.DeleteMarkers { |
| + out = append(out, ObjectVersion{Key: aws.ToString(d.Key), VersionID: aws.ToString(d.VersionId), |
| + IsLatest: aws.ToBool(d.IsLatest), DeleteMarker: true, LastModified: aws.ToTime(d.LastModified)}) |
| + } |
| + } |
| + sortVersions(out) // key order; newest first within a key (S3 returns versions and markers in separate lists) |
| + return out, nil |
| +} |
| + |
| +// sortVersions orders a listing by key, then newest first: LastModified |
| +// descending, then the IsLatest entry first. |
| +func sortVersions(vs []ObjectVersion) { |
| + sort.SliceStable(vs, func(i, j int) bool { |
| + if vs[i].Key != vs[j].Key { |
| + return vs[i].Key < vs[j].Key |
| + } |
| + if vs[i].IsLatest != vs[j].IsLatest { |
| + return vs[i].IsLatest |
| + } |
| + return vs[i].LastModified.After(vs[j].LastModified) |
| + }) |
| +} |
| +``` |
| + |
| +Add `"fmt"`, `"sort"`, `"github.com/aws/aws-sdk-go-v2/aws"` and |
| +`awss3 "github.com/aws/aws-sdk-go-v2/service/s3"` to `versioned.go`'s |
| +imports. `clientFor` and `normalizeETag` already exist in `mirror.go`. If |
| +`clientFor`'s signature differs from `clientFor(bucket string, fallback |
| +*awss3.Client) *awss3.Client`, use the existing one and note the |
| +difference in the report. |
| + |
| +The other four methods are still missing, so the contract test doesn't |
| +compile yet. Add temporary stubs that return |
| +`errors.New("not implemented")`, so that this task's two tests can run |
| +(`TestS3MirrorListVersionsPagesAcrossMarkers`, and the contract's two |
| +listing subtests via `-run`). |
| + |
| +- [ ] **Step 4: Run** |
| + |
| +Run: `go test ./internal/lease/ -run 'TestS3MirrorListVersions|TestS3MirrorVersionedContract/versions|TestS3MirrorVersionedContract/listing' -count=1` |
| +Expected: PASS. |
| + |
| +- [ ] **Step 5: Commit** |
| + |
| +```bash |
| +git add internal/lease/versioned.go internal/lease/versioned_test.go internal/lease/leasetest/ |
| +git commit -m "lease: ListVersions over S3 (paginated, never partial), tested over the wire" |
| +``` |
| + |
| +--- |
| + |
| +### Task 4: `GetVersion` and `CopyVersion` over S3 |
| + |
| +**Files:** |
| +- Modify: `internal/lease/versioned.go`, `internal/lease/versioned_test.go` |
| + |
| +**Interfaces:** |
| +- Produces: `s3Mirror.GetVersion` and `s3Mirror.CopyVersion`, replacing |
| + the stubs. |
| + |
| +- [ ] **Step 1: Write a failing wire test for escaping** |
| + |
| +```go |
| +func TestS3MirrorCopyEscapesKeyAndPinsVersion(t *testing.T) { |
| + m, fb := newWireMirror(t) |
| + key := "src/a b+c%?d" |
| + v1 := fb.Put("data", key, []byte("old")) |
| + fb.Put("data", key, []byte("new")) |
| + if _, _, err := m.CopyVersion(context.Background(), "data", key, v1, "other", "dst/k", true); err != nil { |
| + t.Fatal(err) |
| + } |
| + if cur, ok := fb.Current("other", "dst/k"); !ok || string(cur) != "old" { |
| + t.Fatalf("copied %q, want the pinned old version", cur) |
| + } |
| +} |
| +``` |
| + |
| +Run: `go test ./internal/lease/ -run TestS3MirrorCopyEscapes -count=1` |
| +Expected: FAIL (`not implemented`). |
| + |
| +- [ ] **Step 2: Implement** |
| + |
| +```go |
| +func (m *s3Mirror) GetVersion(ctx context.Context, bucket, key, versionID string) ([]byte, error) { |
| + out, err := m.clientFor(bucket, m.src).GetObject(ctx, &awss3.GetObjectInput{ |
| + Bucket: aws.String(bucket), Key: aws.String(key), VersionId: aws.String(versionID)}) |
| + if err != nil { |
| + if isNotFound(err) { |
| + return nil, ErrMirrorNotFound |
| + } |
| + return nil, fmt.Errorf("lease: get %s/%s@%s: %w", bucket, key, versionID, err) |
| + } |
| + defer out.Body.Close() |
| + return io.ReadAll(out.Body) |
| +} |
| + |
| +func (m *s3Mirror) CopyVersion(ctx context.Context, srcBucket, srcKey, srcVersionID, dstBucket, dstKey string, createOnly bool) (string, string, error) { |
| + in := &awss3.CopyObjectInput{ |
| + Bucket: aws.String(dstBucket), |
| + Key: aws.String(dstKey), |
| + CopySource: aws.String(srcBucket + "/" + url.PathEscape(srcKey) + "?versionId=" + url.QueryEscape(srcVersionID)), |
| + } |
| + if createOnly { |
| + in.IfNoneMatch = aws.String("*") |
| + } |
| + out, err := m.dst.CopyObject(ctx, in) |
| + if err != nil { |
| + if isPreconditionFailed(err) { |
| + return "", "", ErrDestinationExists |
| + } |
| + if isNotFound(err) { |
| + return "", "", ErrMirrorNotFound |
| + } |
| + return "", "", fmt.Errorf("lease: copy %s/%s@%s -> %s/%s: %w", srcBucket, srcKey, srcVersionID, dstBucket, dstKey, err) |
| + } |
| + var etag string |
| + if out.CopyObjectResult != nil { |
| + etag = normalizeETag(aws.ToString(out.CopyObjectResult.ETag)) |
| + } |
| + return aws.ToString(out.VersionId), etag, nil |
| +} |
| +``` |
| + |
| +**Escaping a key that contains `/`.** `url.PathEscape` escapes `/` as |
| +`%2F`, but S3 copy sources keep `/` literal inside the key. Escape each |
| +path segment with `url.PathEscape`, then join them with `/`. |
| + |
| +**Helpers:** |
| +- `isNotFound` should recognise smithy codes `NoSuchKey`, |
| + `NoSuchVersion` and `NotFound`, and HTTP 404. |
| +- `isPreconditionFailed` should recognise code `PreconditionFailed` and |
| + HTTP 412. |
| + |
| +If equivalents exist in `mirror.go` or `internal/release` (for example |
| +`s3NotFound` or `s3CASConflict`), reuse their logic rather than copying |
| +it. Shared helpers must not change behaviour for existing callers. |
| + |
| +**`IfNoneMatch` on `CopyObjectInput`:** if this SDK version lacks the |
| +field, set the `If-None-Match` header with a smithy middleware option on |
| +that one call, and say so in the report. |
| + |
| +- [ ] **Step 3: Run** |
| + |
| +Run: `go test ./internal/lease/ -run 'TestS3MirrorCopyEscapes|TestS3MirrorVersionedContract/get|TestS3MirrorVersionedContract/create-only|TestS3MirrorVersionedContract/multipart' -count=1` |
| +Expected: PASS. |
| + |
| +- [ ] **Step 4: Commit** |
| + |
| +```bash |
| +git add internal/lease/versioned.go internal/lease/versioned_test.go |
| +git commit -m "lease: GetVersion and CopyVersion by exact source version, create-only copies" |
| +``` |
| + |
| +--- |
| + |
| +### Task 5: `DeleteVersion` and `PutIfAbsent` over S3 |
| + |
| +**Files:** |
| +- Modify: `internal/lease/versioned.go`, `internal/lease/versioned_test.go` |
| + |
| +**Interfaces:** |
| +- Produces: `s3Mirror.DeleteVersion` and `s3Mirror.PutIfAbsent`. |
| + |
| +- [ ] **Step 1: Write a failing wire test (a bare delete must never be sent)** |
| + |
| +```go |
| +func TestS3MirrorDeleteVersionNeverBare(t *testing.T) { |
| + m, fb := newWireMirror(t) |
| + fb.Put("data", "k", []byte("one")) |
| + if err := m.DeleteVersion(context.Background(), "data", "k", ""); err == nil { |
| + t.Fatal("DeleteVersion with an empty version id must refuse rather than send a bare delete") |
| + } |
| + if fb.LatestID("data", "k") == "" { |
| + t.Fatal("a delete marker or deletion happened") |
| + } |
| +} |
| +``` |
| + |
| +Run: `go test ./internal/lease/ -run TestS3MirrorDeleteVersionNeverBare -count=1` |
| +Expected: FAIL. |
| + |
| +- [ ] **Step 2: Implement** |
| + |
| +```go |
| +var errNoVersion = errors.New("lease: a version id is required") |
| + |
| +func (m *s3Mirror) DeleteVersion(ctx context.Context, bucket, key, versionID string) error { |
| + if versionID == "" { |
| + return errNoVersion // never a bare-key delete (spec C3) |
| + } |
| + _, err := m.clientFor(bucket, m.dst).DeleteObject(ctx, &awss3.DeleteObjectInput{ |
| + Bucket: aws.String(bucket), Key: aws.String(key), VersionId: aws.String(versionID)}) |
| + if err != nil && !isNotFound(err) { |
| + return fmt.Errorf("lease: delete %s/%s@%s: %w", bucket, key, versionID, err) |
| + } |
| + return nil |
| +} |
| + |
| +func (m *s3Mirror) PutIfAbsent(ctx context.Context, bucket, key string, body []byte) (string, error) { |
| + out, err := m.clientFor(bucket, m.src).PutObject(ctx, &awss3.PutObjectInput{ |
| + Bucket: aws.String(bucket), Key: aws.String(key), Body: bytes.NewReader(body), |
| + IfNoneMatch: aws.String("*")}) |
| + if err != nil { |
| + if isPreconditionFailed(err) { |
| + return "", ErrExists |
| + } |
| + return "", fmt.Errorf("lease: put-if-absent %s/%s: %w", bucket, key, err) |
| + } |
| + return aws.ToString(out.VersionId), nil |
| +} |
| + |
| +var _ VersionedStore = (*s3Mirror)(nil) |
| +``` |
| + |
| +Also make the fake's `DeleteVersion` refuse an empty version id the same |
| +way, returning an error, and add that case to the contract suite (Task |
| +2's file), so both implementations agree. |
| + |
| +- [ ] **Step 3: Run the full contract over the wire, and the fake** |
| + |
| +Run: `go test ./internal/lease/... -count=1 -race` |
| +Expected: PASS (the fake contract, the wire contract and the wire tests). |
| + |
| +- [ ] **Step 4: Commit** |
| + |
| +```bash |
| +git add internal/lease/ internal/lease/leasetest/ |
| +git commit -m "lease: DeleteVersion (never bare) and PutIfAbsent over S3" |
| +``` |
| + |
| +--- |
| + |
| +### Task 6: IAM grants |
| + |
| +**Files:** |
| +- Modify: `infra/modules/region-host/main.tf` (the `host_buckets` policy, |
| + around lines 153-232) and `infra/modules/region-remote-host/main.tf` |
| + (around lines 85-134). |
| + |
| +- [ ] **Step 1: Add the statements** |
| + |
| +In both policies' `Statement` lists, beside the existing data-bucket |
| +object statement, add these two statements, using each module's own |
| +data bucket ARN variable: |
| + |
| +```hcl |
| + { |
| + Effect = "Allow" |
| + Action = ["s3:GetObjectVersion", "s3:DeleteObjectVersion"] |
| + Resource = ["${var.data_bucket_arn}/*"] |
| + }, |
| + { |
| + Effect = "Allow" |
| + Action = ["s3:ListBucketVersions"] |
| + Resource = [var.data_bucket_arn] |
| + }, |
| +``` |
| + |
| +Leave the deployment bucket, the sibling `moves/*` staging statements |
| +and the store policies unchanged. Above the statements, add a comment |
| +naming spec C3 and Plan 2. |
| + |
| +- [ ] **Step 2: Format and validate** |
| + |
| +Run: `tofu fmt -check infra/modules/region-host infra/modules/region-remote-host`. Then run |
| +`tofu -chdir=infra/modules/region-host validate` and the same for |
| +`region-remote-host`, if `tofu` is available. If it isn't, run |
| +`terraform fmt -check` instead, and record which one you used. |
| +Expected: no formatting diff, and validation passes. Without either |
| +tool, inspect the diff for HCL syntax and say so in the report. |
| + |
| +- [ ] **Step 3: Commit** |
| + |
| +```bash |
| +git add infra/modules/region-host/main.tf infra/modules/region-remote-host/main.tf |
| +git commit -m "infra: box role may list, read and delete data-bucket object versions (claim Plan 2)" |
| +``` |
| + |
| +--- |
| + |
| +### Task 7: The repository gate |
| + |
| +- [ ] **Step 1: Run the full gate (AGENTS.md)** |
| + |
| +```bash |
| +go build ./... && go vet ./... && gofmt -l . && go test ./... -count=1 |
| +``` |
| +Expected: build and vet are clean, `gofmt -l .` prints nothing, and every |
| +package passes. Confirm sandbox-only failures through `amadan ci status`. |
| + |
| +- [ ] **Step 2: Commit any formatting fixes** |
| + |
| +```bash |
| +git add -A internal/lease infra/modules |
| +git commit -m "lease: Plan 2 gate" || true |
| +``` |
| + |
| +--- |
| + |
| +## Rollout note |
| + |
| +The IAM change has to reach each deployment before Plans 4 and 6 run |
| +anything that calls these methods. Dev tracks `main`. Flagship applies |
| +through `carloku-infrastructure` at a release tag, and that apply must |
| +**target** the policy resources only: a whole-root apply touches |
| +`user_data` and restarts every box (memory: carloku infra split). Plan 2 |
| +itself calls nothing in production, so merging it is inert. |
| + |
| +## Self-review |
| + |
| +- **Spec coverage:** |
| + - C3, exact-version destructive calls: Tasks 1, 4 and 5; |
| + - §6.2, list every version and archive by version: Tasks 2 to 4; |
| + - §6.4a certificates, create-only: Task 5 `PutIfAbsent`; |
| + - §6.5, pinned reads: Task 4 `GetVersion`; |
| + - §13 rows 1 to 9 in the fake: Task 2; |
| + - IAM for §6.2 and §6.5: Task 6. |
| +- **Placeholders:** none. Two conditional instructions are explicit: the |
| + SDK `IfNoneMatch` fallback, and reusing the existing not-found and |
| + precondition helpers. |
| +- **Types:** |
| + - `ObjectVersion` and `VersionedStore` are used consistently. |
| + - `VersionedBucket`'s exported helpers (`Put`, `PutMultipart`, |
| + `BareDelete`, `LatestID`, `Current`, `AddBucket`, `PageSize`, |
| + `FailListAfterPages`, `FailOn`) are used by the contract and later |
| + plans. |
| + - `newWireMirror` is test-only. |