From 17a142cf565c84583c35ff8b3bdab533f0bfd6ab Mon Sep 17 00:00:00 2001 From: vickydotbat Date: Sat, 4 Jul 2026 23:28:42 +0200 Subject: [PATCH] docs: depot core increment 1 plan + spec updates Co-Authored-By: Claude Fable 5 --- .../plans/2026-07-04-crucible-depot-core.md | 330 ++++++++++++++++++ .../2026-07-04-crucible-depot-core-design.md | 132 +++++-- 2 files changed, 441 insertions(+), 21 deletions(-) create mode 100644 docs/superpowers/plans/2026-07-04-crucible-depot-core.md diff --git a/docs/superpowers/plans/2026-07-04-crucible-depot-core.md b/docs/superpowers/plans/2026-07-04-crucible-depot-core.md new file mode 100644 index 0000000..e1bc1f8 --- /dev/null +++ b/docs/superpowers/plans/2026-07-04-crucible-depot-core.md @@ -0,0 +1,330 @@ +# Crucible Depot Core (Increment 1) Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Implement `crucible depot {status,push,verify,get,pull}` with local/cdn/bunny backends in sow-tools, then cut sow-assets-manifest over to it and delete the bash it replaces. + +**Architecture:** New stdlib-only `internal/depot` package with its own `Run(args) int` entrypoint (exit contract 0/1/2/70 is richer than `internal/app.Run`'s 0/1, so dispatch special-cases depot instead of routing through `app.Run`). Backends implement a small interface; a sweep engine does parallel range-GET probes with serial re-confirm. Cutover repo work happens in sow-assets-manifest on its own branch. + +**Tech Stack:** Go 1.26, stdlib `net/http`/`crypto/sha256`/`sync`, `gopkg.in/yaml.v3` (already a dep — add nothing to go.mod, vendorHash must not change). + +**Spec:** `docs/superpowers/specs/2026-07-04-crucible-depot-core-design.md` — read it before any task; it is authoritative. + +## Global Constraints + +- **No new Go dependencies.** stdlib + existing `gopkg.in/yaml.v3` only. +- **HEAD is banned** for existence checks. All probes: IPv4, `GET` with `Range: bytes=0-0`, expect `206` (accept any 2xx). A test must fail if HEAD is introduced. +- **Never prompt, never read stdin** for credentials — read path included. Missing key → clear error on stderr, exit `70`. +- **No cache of "what I uploaded"** — presence is always probed against the real target backend. +- `unconfirmed` is a distinct state from `missing`; exit `2` when unconfirmed-only. +- Exit codes: `0` clean, `1` drift (referenced-but-absent), `2` unconfirmed-only, `64` usage, `70` internal/backend error. +- Blob key: `sha256///`; sha must match `^[0-9a-f]{64}$`. +- Env names (unchanged from bash): `DEPOT_CDN_BASE`/`BUNNY_CDN_BASE` (cdn read base, default `https://cdn-a7f3k9.westgate.pw`), `BUNNY_STORAGE_HOST` (**required for bunny, no default** — spec overrides bash), `BUNNY_STORAGE_ZONE` (default `sow-assets-depot`), `BUNNY_STORAGE_READ_PASSWORD` (falls back to `BUNNY_STORAGE_PASSWORD`) for reads/probes, `BUNNY_STORAGE_PASSWORD` for PUT, `DEPOT_PROBE_JOBS` (default 16), `DEPOT_JOBS` (default 16), `DEPOT_CONNECT_TIMEOUT` (default 10s), `DEPOT_PROBE_MAX_TIME` (default 30s), `DEPOT_CONFIRM_RETRIES` (default 3), `DEPOT_CONFIRM_MAX` (default 200, 0 = unlimited). +- Upload: `PUT` with `AccessKey: ` and `Checksum: ` headers; each sha uploaded at most once per run; probe-before-put. +- Every downloaded blob is re-hashed; deleted on mismatch. +- `cdn` backend writes fail closed (error, never fake success). +- Verify with `make vet test` in sow-tools (run inside `nix develop` if go missing from PATH). +- Commit after each task. **Never commit to main.** sow-tools branch: `depot-core-increment-1`. sow-assets-manifest branch: `crucible-depot-cutover`. sow-tools has an unrelated modified spec file on main — leave it out of commits unless the task says otherwise. + +--- + +### Task 1: Package skeleton — config + manifest parsing + +**Files:** +- Create: `internal/depot/config.go`, `internal/depot/config_test.go` +- Create: `internal/depot/manifest.go`, `internal/depot/manifest_test.go` + +**Interfaces (Produces):** + +```go +package depot + +// config.go +type Config struct { + CDNBase string // DEPOT_CDN_BASE || BUNNY_CDN_BASE || default + StorageHost string // BUNNY_STORAGE_HOST, no default + StorageZone string // BUNNY_STORAGE_ZONE, default "sow-assets-depot" + ReadKey string // BUNNY_STORAGE_READ_PASSWORD || BUNNY_STORAGE_PASSWORD + WriteKey string // BUNNY_STORAGE_PASSWORD + ProbeJobs int // DEPOT_PROBE_JOBS, default 16 + Jobs int // DEPOT_JOBS, default 16 + ConnectTimeout time.Duration // DEPOT_CONNECT_TIMEOUT seconds, default 10s + ProbeMaxTime time.Duration // DEPOT_PROBE_MAX_TIME seconds, default 30s + ConfirmRetries int // DEPOT_CONFIRM_RETRIES, default 3 + ConfirmMax int // DEPOT_CONFIRM_MAX, default 200 (0 = unlimited) +} +func LoadConfig(getenv func(string) string) Config + +// manifest.go +// ReferencedSHAs parses every *.yml in dir (schema: assets[].{path,sha256,size,restype,hak}) +// and returns the deduplicated sha set plus per-sha size (for pull). +func ReferencedSHAs(dir string) (map[string]int64, error) +func ValidSHA(s string) bool // ^[0-9a-f]{64}$ +func BlobKey(sha string) string // "sha256/ab/cd/" +``` + +`LoadConfig` takes `getenv` so tests inject env without `t.Setenv` races. Invalid ints/durations in env → fall back to default (do not error). `ReferencedSHAs` errors on: unreadable dir, no `*.yml` files, YAML parse failure, or an entry whose `sha256` fails `ValidSHA` (report file + path). Manifest struct: + +```go +type manifestFile struct { + Assets []struct { + Path string `yaml:"path"` + SHA256 string `yaml:"sha256"` + Size int64 `yaml:"size"` + } `yaml:"assets"` +} +``` + +- [ ] **Step 1:** `git -C sow-tools switch -c depot-core-increment-1` (from main). +- [ ] **Step 2:** Write failing tests: `TestLoadConfigDefaults`, `TestLoadConfigReadKeyFallback` (READ unset → WriteKey value; READ set → its own), `TestLoadConfigBadIntFallsBack`, `TestBlobKey` (`BlobKey("0b1d…")=="sha256/0b/1d/0b1d…"`), `TestValidSHA` (rejects uppercase, short, non-hex), `TestReferencedSHAs` (temp dir with two yml files sharing one sha → dedup; bad sha → error naming the file; empty dir → error). +- [ ] **Step 3:** `go test ./internal/depot/` — verify FAIL (compile error is fine). +- [ ] **Step 4:** Implement `config.go` + `manifest.go` minimally. +- [ ] **Step 5:** `go test ./internal/depot/` PASS; `gofmt -l -w internal/depot`. +- [ ] **Step 6:** Commit: `feat(depot): config resolution and manifest sha parsing`. + +### Task 2: Local backend + +**Files:** +- Create: `internal/depot/backend.go`, `internal/depot/local.go`, `internal/depot/local_test.go` + +**Interfaces (Produces):** + +```go +// backend.go +type ProbeState int +const ( + Present ProbeState = iota + Absent + Unconfirmed +) +type Backend interface { + Name() string + // Probe returns the existence state of one sha. transient=true means a + // retry might change the answer (feeds the serial confirm loop). + Probe(ctx context.Context, sha string) (state ProbeState, transient bool, err error) + // Get fetches sha into dest (temp file + rename), re-hashes, deletes on mismatch. + Get(ctx context.Context, sha, dest string) error + // Put uploads bytes from src for sha. Read-only backends return an error. + Put(ctx context.Context, sha, src string) error +} + +// local.go +type LocalBackend struct{ Root string } // implements Backend; Name()=="local" +``` + +Local semantics: `Probe` = `os.Stat(Root/BlobKey(sha))`, never transient. `Put` = copy to temp file in dest dir then `os.Rename` (MkdirAll parents). `Get` = copy out + re-hash (hash the source bytes while copying via `io.TeeReader` into `sha256.New()`), delete dest and error on mismatch. `err` from Probe only for real I/O errors (permission), not absence. + +- [ ] **Step 1:** Failing tests: `TestLocalRoundTrip` (Put→Probe Present→Get→bytes equal), `TestLocalProbeAbsent`, `TestLocalGetHashMismatch` (hand-write a corrupt blob file at the keyed path, Get returns error and dest does not exist), `TestLocalPutIdempotent`. +- [ ] **Step 2:** Verify FAIL. **Step 3:** Implement. **Step 4:** PASS + gofmt. +- [ ] **Step 5:** Commit: `feat(depot): backend interface and local backend`. + +### Task 3: Remote backends (cdn, bunny) with fake-Bunny tests + +**Files:** +- Create: `internal/depot/remote.go`, `internal/depot/remote_test.go` + +**Interfaces (Consumes):** `Backend`, `ProbeState`, `Config`, `BlobKey`. +**Produces:** + +```go +// NewBackend returns the backend for name ("local"|"cdn"|"bunny"). +// localRoot is used only for "local". Fails closed: "bunny" with empty +// StorageHost or empty ReadKey returns an error (never prompts); +// unknown name returns an error. +func NewBackend(name, localRoot string, cfg Config) (Backend, error) + +type httpBackend struct { ... } // shared by cdn and bunny +``` + +Implementation requirements: + +```go +// IPv4-only transport — REQUIRED, spec bans IPv6/HEAD probe hazards. +transport := &http.Transport{ + DialContext: func(ctx context.Context, network, addr string) (net.Conn, error) { + d := net.Dialer{Timeout: cfg.ConnectTimeout} + return d.DialContext(ctx, "tcp4", addr) + }, + MaxIdleConnsPerHost: cfg.ProbeJobs, +} +client := &http.Client{Transport: transport, Timeout: cfg.ProbeMaxTime} +``` + +- **Probe** (both remotes): `GET /` with header `Range: bytes=0-0`; drain/close body. 2xx → `Present`; 404/410 → `Absent`; anything else (incl. transport error) → `Unconfirmed, transient=true`. **Never http.Head / MethodHead.** +- cdn: base = `cfg.CDNBase`, no auth headers, `Put` returns `errors.New("cdn backend is read-only")`. +- bunny: probe/read against storage `https:////` with `AccessKey: `. `Get`: try CDN base first, on non-2xx fall back to storage URL with ReadKey (mirrors bash `_depot_bunny_get`); re-hash, delete on mismatch. `Put`: probe first, skip if Present; else `PUT` storage URL, headers `AccessKey: `, `Checksum: `, body streamed from src file; non-2xx → error. `Put` with empty WriteKey → error before any I/O. +- Both `Get`s write via temp file + rename and re-hash exactly like local. + +httptest note: `httptest.NewServer` listens on `127.0.0.1`, which the tcp4 dialer reaches — no test shim needed. + +- [ ] **Step 1:** Failing tests against an `httptest` fake Bunny (a handler recording method+path+headers, serving a blob map): + - `TestProbeUsesRangeGetNotHead` — probe a present sha; assert recorded request has `Method=="GET"` and `Range=="bytes=0-0"`; **also** assert the depot source contains no HEAD probe: the handler returns 500 for any HEAD request, and the test asserts zero HEAD requests were received across the whole test run (this is the anti-HEAD regression tripwire from field note 3). + - `TestProbeStates` — 200/206→Present, 404→Absent, 428→Unconfirmed+transient, 503→Unconfirmed+transient. + - `TestBunnyPutSequence` — Put of an absent sha: recorded sequence is [GET range probe, PUT]; PUT has `Checksum` = uppercase sha and `AccessKey` = write key; Put of a present sha: probe only, no PUT. + - `TestReadWriteKeySplit` — probe uses ReadKey, PUT uses WriteKey (distinct values asserted). + - `TestBunnyNoReadKeyFailsClosed` — `NewBackend("bunny",…)` with empty read key returns error **without reading stdin**: replace `os.Stdin` with the read end of an `os.Pipe()` for the test; after the call, write+close and confirm the pipe is undrained (or simply assert error occurs with no stdin interaction because the code path has no stdin reference — grep-based assertion acceptable: test fails if `internal/depot` sources mention `os.Stdin` or `/dev/tty`). + - `TestBunnyNoStorageHostFailsClosed`. + - `TestGetHashMismatchDeletes` — server returns wrong bytes; Get errors, dest removed. + - `TestCDNPutReadOnly`. +- [ ] **Step 2:** Verify FAIL. **Step 3:** Implement `remote.go`. **Step 4:** PASS + gofmt + `go vet ./...`. +- [ ] **Step 5:** Commit: `feat(depot): cdn and bunny HTTP backends with fake-Bunny tests`. + +### Task 4: Sweep engine — parallel probe + serial confirm + +**Files:** +- Create: `internal/depot/sweep.go`, `internal/depot/sweep_test.go` + +**Interfaces (Consumes):** `Backend`, `Config`, `ProbeState`. +**Produces:** + +```go +type SweepResult struct { + Present []string + Missing []string // confirmed absent (404/410 after retries) + Unconfirmed []string // probe budget exhausted, state unknown +} +// Sweep probes every sha against b: one parallel pass (cfg.ProbeJobs workers, +// sync.WaitGroup + buffered-channel work queue), then any non-Present result +// is re-probed serially with linear backoff (attempt t sleeps t seconds, +// cfg.ConfirmRetries attempts) — unless the non-Present count exceeds +// cfg.ConfirmMax (>0), in which case the remainder is reported Unconfirmed +// without serial re-probe (never collapsed into Missing). +// A transient result that stays non-2xx after retries is Unconfirmed; +// a 404/410 is Missing. Backend errors (non-transient) abort with error. +func Sweep(ctx context.Context, b Backend, shas []string, cfg Config, progress io.Writer) (SweepResult, error) +``` + +Sort `shas` before sweeping (deterministic output). Progress: print `probed X/N` roughly every 1000 blobs to `progress` (may be `io.Discard`). For tests, make the backoff sleeper injectable: package-level `var confirmSleep = time.Sleep`, overridden in tests. + +- [ ] **Step 1:** Failing tests using a stub Backend (function fields): + - `TestSweepAllPresent`, `TestSweepMissing` (404 → Missing, no retries beyond first serial confirm). + - `TestSweepThrottledUnconfirmed` — stub always returns Unconfirmed/transient → after retries lands in `Unconfirmed`, never `Missing`; assert `confirmSleep` called with 1s then 2s. + - `TestSweepTransientRecovers` — first probe 428, serial re-probe 206 → Present. + - `TestSweepConfirmMaxCap` — 5 non-present, ConfirmMax=2 → all 5 Unconfirmed, zero serial re-probes. + - `TestSweepParallelBounded` — 100 shas, ProbeJobs=4; stub counts concurrent in-flight probes (atomic), assert max ≤ 4. +- [ ] **Step 2:** FAIL. **Step 3:** Implement. **Step 4:** PASS + gofmt. +- [ ] **Step 5:** Commit: `feat(depot): parallel probe sweep with serial confirm and unconfirmed state`. + +### Task 5: Commands + Run entrypoint + integration test + +**Files:** +- Create: `internal/depot/run.go`, `internal/depot/run_test.go`, `internal/depot/integration_test.go` + +**Interfaces (Consumes):** everything above. +**Produces:** + +```go +// Run executes a depot subcommand. args[0] is the subcommand +// (status|push|verify|get|pull); returns the process exit code. +// Exit: 0 clean, 1 drift, 2 unconfirmed-only, 64 usage, 70 internal. +func Run(args []string, stdout, stderr io.Writer, getenv func(string) string) int +``` + +Flag parsing: one `flag.FlagSet` per subcommand (stdlib; `internal/depot` is a fresh package, the hand-rolled loop in `internal/app` is not a constraint here). Flags per spec: + +``` +status [--manifests DIR] [--source LOCAL_DEPOT] --target bunny|cdn|local +push [--manifests DIR] --source LOCAL_DEPOT --target bunny +verify [--manifests DIR] --target bunny|cdn [--sample N] +get --target cdn|bunny|local +pull [--manifests DIR] --dest DIR --target cdn|bunny +``` + +`--manifests` default `assets`. `--target local` uses `--source` as root for status, and `--target`-side root comes from `DEPOT_DIR` env when target is local (matches bash `DEPOT_DIR`); document in usage string. Missing required flag / unknown subcommand / no args → usage on stderr, exit 64. + +Command semantics (all build shas via `ReferencedSHAs`, backend via `NewBackend`, sweep via `Sweep`): + +- **status**: sweep; print `referenced= present= missing= unconfirmed=` to stdout, then each missing sha on its own line prefixed `missing ` and each unconfirmed prefixed `unconfirmed `. Exit 1 if missing>0; else 2 if unconfirmed>0; else 0. +- **push**: target must be `bunny` (enforce). Sweep; treat missing∪unconfirmed as to-upload (unconfirmed→re-upload is safe/idempotent per spec). For each, read bytes from `--source` local root; a sha absent from source → report and count as failed. Upload with `cfg.Jobs` parallel workers, each sha at most once. Exit 70 on upload error, 1 if any sha was missing from source, else 0. Print `uploaded= failed=`. +- **verify**: sweep (exit 1/2 as status) **plus**, when clean, download `--sample` N (default 3, 0=skip) random blobs (`math/rand` seeded — pick via deterministic-enough `rand.New(rand.NewSource(time.Now().UnixNano()))`) to temp files via `Get` (which re-hashes); any failure → exit 1. +- **get**: `Get(sha, dest)`; validate sha; exit 0/70. +- **pull**: for every referenced sha, dest path `--dest/BlobKey(sha)`. Incremental: if dest file exists, re-hash it; matching → skip, mismatch → re-download. Download absent/mismatched with `cfg.Jobs` workers via `Get`. Print `pulled= present=`; any failed download → exit 70 after finishing the batch (report each failure). Non-zero missing at the source (404) → exit 1 listing them. + +Wire the binary: **Modify** `internal/dispatch/dispatch.go` — in `runBuilder`, before the `Wired` check/`delegateLegacy` routing, special-case: + +```go +if b.Name == "depot" && b.Wired { + return depot.Run(args[1:], os.Stdout, errw, os.Getenv) +} +``` + +(Exact insertion point: the implementer must read `runBuilder` and place this after subcommand resolution is bypassed — depot does its own subcommand parsing, so pass the raw args through, i.e. `args[1:]` where `args[0]=="depot"` at the dispatcher level, or the full arg slice in `RunBuilder("depot", …)` context. Verify both `crucible depot status …` and `crucible-depot status …` paths hit `depot.Run`.) + +- [ ] **Step 1:** Failing unit tests (`run_test.go`): `TestRunUsage` (no args→64, unknown cmd→64, push --target cdn→64), `TestStatusExitCodes` (drift→1, unconfirmed-only→2 using a bunny backend pointed at an httptest server that 428s), `TestGetInvalidSHA` (→64 or 70, pick 64), `TestNoKeyStatusBunnyFailsClosedNoStdin` (getenv returns empty keys → exit 70, stderr mentions the env var name). +- [ ] **Step 2:** Failing integration test (`integration_test.go`), pure local→local in temp dirs: build manifests yml referencing 3 blobs; source depot has all 3; target depot has 1. Then: `status --target local` → exit 1, missing=2 → `push` (local→local: for the test, allow `push --target local` **only** via an unexported hook? No — keep the spec surface: instead run the integration through bunny backend backed by httptest fake-Bunny with an in-memory blob map, which exercises the real push path) → `status` → 0 → `push` again → 0 uploads (idempotent, assert fake recorded no second PUT). Also `pull` into a pre-populated dest: present-and-hash-ok skipped (fake records no GET for it), corrupt pre-existing file re-downloaded. +- [ ] **Step 3:** FAIL. **Step 4:** Implement `run.go`. **Step 5:** `make vet test` PASS + gofmt. +- [ ] **Step 6:** Commit: `feat(depot): status/push/verify/get/pull commands with exit-code contract`. + +### Task 6: Registry wiring + docs + +**Files:** +- Modify: `internal/dispatch/dispatch.go` (depot Registry entry) +- Modify: `internal/dispatch/dispatch_test.go` (if it asserts wired/unwired sets) + +Flip the depot entry: + +```go +{ + Name: "depot", + Bin: "crucible-depot", + Summary: "content-addressed asset depot (local/cdn/bunny)", + Commands: []Command{ + {Name: "status", Summary: "report referenced-vs-present drift against a backend", Usage: "crucible depot status [--manifests DIR] [--source DIR] --target bunny|cdn|local"}, + {Name: "push", Summary: "upload referenced-but-absent blobs from a local depot", Usage: "crucible depot push [--manifests DIR] --source DIR --target bunny"}, + {Name: "verify", Summary: "existence sweep plus sampled download re-hash", Usage: "crucible depot verify [--manifests DIR] --target bunny|cdn [--sample N]"}, + {Name: "get", Summary: "fetch one blob with sha re-verify", Usage: "crucible depot get --target cdn|bunny|local"}, + {Name: "pull", Summary: "incremental verified pull of every referenced blob", Usage: "crucible depot pull [--manifests DIR] --dest DIR --target cdn|bunny"}, + }, + Wired: true, +}, +``` + +The stale "(SeaweedFS)" wording must not survive anywhere: `grep -ri seaweedfs` across the repo and fix hits. + +- [ ] **Step 1:** Update registry; run `go test ./internal/dispatch/` — fix any test that pinned depot as unwired; add/extend a test asserting `crucible depot` routes to `depot.Run` (e.g. `RunBuilder("depot", []string{"status"})` with no manifests dir returns 70/64, not the unwired 70 message — assert on stderr text). +- [ ] **Step 2:** `make check` (vet+test+shellcheck+yamllint) PASS. Also `make build && ./bin/crucible depot` prints depot usage; `./bin/crucible` interactive menu shows depot entries (manual eyeball via `printf 'q\n' | ./bin/crucible` or equivalent). +- [ ] **Step 3:** Commit: `feat(depot): wire depot into dispatch registry and menu`. + +### Task 7: sow-assets-manifest cutover + +**Repo:** `/home/vicky/Projects/westgate/repositories/migration/sow-assets-manifest`, new branch `crucible-depot-cutover` from main. + +**Files:** +- Delete: `scripts/mirror.sh` +- Modify: `scripts/build-haks.sh` (the `--full` depot-verify block, ~L138-192), `Makefile` (`mirror` target, `haks` target), `.gitea/workflows/release.yml`, `.gitea/workflows/publish-haks.yml`, `.gitea/workflows/build-haks.yml`, `scripts/lib.sh` (header comments only), `AGENTS.md`/`README.md` (author-facing docs) + +**Consumes:** the `crucible depot` CLI from Tasks 1–6. NOTE: CI resolves crucible via the flake input tracking sow-tools' default branch — the sow-tools branch must be merged before this repo's CI can pass. Local verification uses a locally built binary (`sow-tools/bin/crucible` on PATH). Do not run `nix flake update` here; note in the PR description that a `nix flake update sow-tools` (or lock bump) must land with/after the sow-tools merge. + +Changes: + +1. **`release.yml`** "Warm depot cache (verified full pull)" step body (keep name, job env unchanged; drop the ticker/compgen bash): +```yaml + - name: Warm depot cache (verified full pull) + env: + BUNNY_STORAGE_READ_PASSWORD: ${{ secrets.BUNNY_STORAGE_READ_PASSWORD }} + BUNNY_STORAGE_HOST: storage.bunnycdn.com + BUNNY_STORAGE_ZONE: sow-assets-depot + run: | + nix develop --command bash -c ' + set -euo pipefail + crucible depot status --manifests assets --target bunny + crucible depot pull --manifests assets --dest "$DEPOT_CACHE_DIR" --target cdn + ' +``` + (If the `BUNNY_STORAGE_READ_PASSWORD` secret does not exist in the repo, use `secrets.BUNNY_STORAGE_PASSWORD` and say so in the PR body. Check what other steps use.) +2. **`publish-haks.yml`** — same swap of the identical step. +3. **`build-haks.yml`** — the PR-check step keeps `build-haks.sh --dry-run --no-verify --base "$base"` (structural check) and **adds** `crucible depot verify --manifests assets --target cdn --sample 3` as the cdn-backend verification. +4. **`scripts/build-haks.sh`** — delete the depot existence-sweep + sample-download block inside `--dry-run` (the `depot_has_many`/`depot_get` verify at ~L138-185); replace with a comment pointing at `crucible depot verify`. Remove now-unused locals. `shellcheck scripts/build-haks.sh` must pass. +5. **`Makefile`** — `mirror:` target now runs `crucible depot pull --manifests assets --dest "$(DEPOT_DIR)" --target cdn`. `haks:` gains a trailing non-fatal drift report: `-crucible depot status --manifests assets --target cdn` (leading `-` so drift never fails a local build) and an echo pointing at `crucible depot push --source workspace/depot --target bunny` as the standard post-edit step. +6. **`scripts/lib.sh`** — no function deletions (caller analysis: every depot function used by mirror.sh/build-haks.sh retains Increment 2–3 callers). Add a header comment block at the top of the depot section: `# NOTE: this bash depot layer is being retired. Increment 1 moved mirror/verify to 'crucible depot' (sow-tools). import/sync go in Increment 2; export/pack/publish/artifact/cdn_purge in Increment 3. Do not add new callers.` **Do not touch the backend-keyed stat-cache in import.sh** (spec forbids it). +7. **Docs** — in README.md (or AGENTS.md where depot workflow is documented): document `crucible depot status --manifests assets --target cdn` as the canonical credential-free "is my content live?" author check, and `crucible depot push --source workspace/depot --target bunny` as the standard post-edit publish step. + +- [ ] **Step 1:** Branch; make all edits above. +- [ ] **Step 2:** Verify: `shellcheck scripts/*.sh`; `yamllint .gitea`; `grep -rn "mirror.sh" . --exclude-dir=.git` returns nothing; with sow-tools' `bin` on PATH run `crucible depot status --manifests assets --target cdn` for real (credential-free) and confirm it produces a sane report (exit 0 or 2 acceptable; exit 1 means real drift — report it, don't hide it). +- [ ] **Step 3:** Commit: `feat: cut depot mirror/verify over to crucible depot, retire mirror.sh`. + +## Self-Review Notes + +- Spec coverage: command surface (T5), drift model + unconfirmed/exit 2 (T4/T5), backends + env + no-prompt (T1/T3), IPv4 range probe + HEAD ban test (T3), Checksum PUT (T3), pull incremental (T5), cutover of all three workflows + Makefile + lib.sh headers + author docs (T7), registry/SeaweedFS fix + menu (T6), testing section requirements all mapped (T1–T5), interim contract `make haks` drift report (T7.5). +- Deliberate deviations from spec, justified: `status --target local` root via `DEPOT_DIR`; push treats source-missing blobs as exit 1; pull download failure exits 70 (backend error) but source-404 exits 1 (drift). +- Out of scope, per spec non-goals: import/sync, prune/gc/export, mdl checks, rich TUI, cdn_purge. diff --git a/docs/superpowers/specs/2026-07-04-crucible-depot-core-design.md b/docs/superpowers/specs/2026-07-04-crucible-depot-core-design.md index ec13993..bb9b4bd 100644 --- a/docs/superpowers/specs/2026-07-04-crucible-depot-core-design.md +++ b/docs/superpowers/specs/2026-07-04-crucible-depot-core-design.md @@ -124,9 +124,10 @@ Increment 1. This spec covers **Increment 1** only. Roadmap for context: -- **Increment 1 (this spec):** `crucible depot {status, push, verify, get}` + - blob backends (`local` / `cdn` / `bunny`). Cut over `mirror`/verify/release - gate. Delete the corresponding `lib.sh` depot layer. +- **Increment 1 (this spec):** `crucible depot {status, push, verify, get, pull}` + + blob backends (`local` / `cdn` / `bunny`). Cut over `mirror`/verify/release + gate. Delete the `lib.sh` depot functions that lose their last caller (see + Cutover for what actually remains until Increments 2–3). - **Increment 2:** `crucible depot import` / `sync` (edit-tree → manifest+depot), replacing `import.sh` / `sync-assets.sh`. Retire the stat-cache concept. - **Increment 3:** mdl integrity checks, `mirror`/`pull` ergonomics, `prune`, @@ -141,8 +142,17 @@ crucible depot status [--manifests DIR] [--source LOCAL_DEPOT] --target bunny|c crucible depot push [--manifests DIR] --source LOCAL_DEPOT --target bunny crucible depot verify [--manifests DIR] --target bunny|cdn [--sample N] crucible depot get --target cdn|bunny|local +crucible depot pull [--manifests DIR] --dest DIR --target cdn|bunny ``` +- `pull` is the bulk fetch that replaces `mirror.sh`: every referenced sha into + `--dest/sha256/ab/cd/`, **incremental** (skip blobs already present with a + matching hash — the release warm step re-runs against a persistent + `/var/cache` dir holding ~69k blobs; a non-incremental pull would re-download + the corpus every release), sha re-verified on download, parallel with the same + probe/transfer bounds as `status`. Without `pull`, Increment 1 cannot delete + `mirror.sh` (single-blob `get` is not a replacement). + - `--manifests` defaults to `assets/` (the repo's manifest dir). Confirmed decision: `crucible depot` parses `assets/*.yml` directly (same coupling model as `crucible-hak` reading hak manifests); the caller does not pre-extract sha @@ -159,8 +169,26 @@ no cache of "what I uploaded." - **`status`** — collect every `sha256` referenced by `--manifests/*.yml`, run one parallel existence sweep against `--target`, report - `{referenced, present, missing}`. Non-zero exit if any referenced blob is - missing. This is the release pre-flight gate. + `{referenced, present, missing, unconfirmed}`. Non-zero exit if any referenced + blob is missing. This is the release pre-flight gate. + + **`unconfirmed` is a distinct state, not `missing`.** Any non-2xx from the + parallel sweep is re-probed serially with backoff (the bash + `depot_confirm_absent` behavior). If the serial re-probe budget is exhausted + (bash caps at 200; keep a cap, `DEPOT_CONFIRM_MAX`), the remainder is reported + as `unconfirmed` — **not** collapsed into `missing`. For `push` the + distinction is harmless (unconfirmed → treat as absent → re-upload; wasteful + but idempotent and safe). For `status` as the release gate it is critical: a + throttled CDN edge must produce "N unconfirmed, retry" (distinct exit code, + proposed `2`), never a false "thousands missing" hard-fail — exactly the + over-reporting the corrected v0.1.5 drift measurement documented above. + + **Target authority split:** `bunny` probes storage — the origin, authoritative, + needs the read key; this is what the release gate uses. `cdn` probes the edge — + credential-free, what players actually fetch, the author-facing "is my content + live?" check; the edge cache can briefly lag storage (bash carries `cdn_purge` + for this reason). A blob `present` on bunny but `missing` on cdn is + propagation lag, not drift. - **`push`** — run `status`, then for each missing sha read the bytes from `--source` and upload to `--target`. Stateless and idempotent; re-running does nothing once clean. **Cannot silently skip an upload** — "should I upload this?" @@ -189,13 +217,25 @@ untouched; flags are local aliases): - `BUNNY_STORAGE_HOST`, `BUNNY_STORAGE_ZONE` — Bunny storage endpoint. - **Read/write key split (preserved):** `BUNNY_STORAGE_READ_PASSWORD` (falls back to `BUNNY_STORAGE_PASSWORD`) for probes/reads; `BUNNY_STORAGE_PASSWORD` for - `PUT`. + `PUT`. Note for the `release.yml` cutover: `status --target bunny` as the gate + step needs at least the read key exported in that step's env. +- **No prompting, read path included.** The no-prompt rule from field note 1 + applies to every credentialed operation, not just `push`: `status`/`verify`/ + `get` against `bunny` with no read key fail closed with a clear message — + never a hang, never a tty read. +- `BUNNY_STORAGE_HOST` stays **env-required, no baked default** — the bash never + defaulted it (`depot_require_write` defaults only the zone + `sow-assets-depot` and the CDN base); resolving the earlier open question. - Concurrency: `DEPOT_PROBE_JOBS` (read probes), `DEPOT_JOBS` (uploads); `DEPOT_CONNECT_TIMEOUT`, `DEPOT_PROBE_MAX_TIME` per-transfer bounds. Invariants carried over exactly: -- **Existence probe** is a 1-byte range request (no body download). +- **Existence probe** is an IPv4 1-byte range request (no body download) — for + **all** backends. This is a deliberate deviation from `lib.sh`, whose `cdn` + backend probes with HEAD (`_depot_cdn_probe_code`); the "faithful port" + clause above does not extend to the probe method. HEAD is banned (see field + note 3 and the test requirement there). - **Upload** is `PUT` with `Checksum: ` so Bunny rejects corrupt writes server-side; each sha uploaded at most once per run. - **Fail-safe:** a blob with no confirmed-present reply is treated as absent and @@ -223,26 +263,68 @@ assets/*.yml ──parse──▶ referenced sha set ─┐ Increment 1 deletes the bash it replaces (no wrappers left behind): -- `scripts/mirror.sh` → `crucible depot get`/pull path; script removed. +- `scripts/mirror.sh` → `crucible depot pull`; script removed. - `build-haks.sh`'s `--full` depot-verify block → `crucible depot verify`. -- `release.yml` step "Warm depot cache" → `crucible depot status` (gate) then the - fetch path; the inline bash ticker/compgen logic goes away. -- `lib.sh` depot layer (`depot_require`, `depot_require_write`, `_depot_*_has`, - `_depot_*_get`, `_depot_*_put*`, `_depot_*_probe*`, `blob_key`, backend - dispatch — ~250+ lines) is removed once no remaining bash references it. - Anything still needed by Increments 2–3 is deleted when those land, not kept as - a compatibility shim. +- **All three depot-touching workflows switch in the same change** (not just + `release.yml`): + - `release.yml` "Warm depot cache (verified full pull)" → + `crucible depot status --target bunny` (gate) then + `crucible depot pull --dest $DEPOT_CACHE_DIR`; the inline bash + ticker/compgen logic goes away. + - `publish-haks.yml` mirrors the same warm+verify+pack flow — same swap. + - `build-haks.yml` (PR check, cdn-backend verification) → + `crucible depot verify --target cdn`. +- Registry wiring: the `depot` entry in `internal/dispatch` flips to + `Wired: true` with its `Commands` list populated (status/push/verify/get/pull) + and the stale "(SeaweedFS)" summary corrected. This automatically puts depot + in the interactive `crucible` menu alongside the other wired tools — no + menu-specific code needed. + +**What `lib.sh` deletion actually looks like.** The depot layer cannot be fully +removed in Increment 1 — it is still referenced by scripts outside this +increment's scope: + +- `import.sh` / `sync-assets.sh` (`make haks`, `make import`, `make sync`) — + Increment 2. +- `export.sh` (`make pull`), `pack-haks.sh` (release-time `DEPOT_BACKEND=bunny` + packing), `publish-release.sh` + `release_put`, and the path-addressed + `artifact_has/get/put` + `cdn_purge` helpers — Increment 3 (these were + previously unlisted; they are now explicitly Increment 3 scope). + +Increment 1 deletes the functions whose last caller it removes (the +mirror/verify probe-and-get paths) and leaves the rest with a header comment +marking them scheduled for deletion in Increments 2–3. No compatibility shims, +no new wrappers — but no pretending `lib.sh` dies before its last caller does. Consumers call `crucible depot …` directly. Nothing new is wrapped. +### Interim contract for local builds (until Increment 2) + +`make haks` keeps running bash `sync-assets.sh` with `DEPOT_BACKEND=local` until +Increment 2 replaces import/sync. The gap this leaves — local imports creating +blobs Bunny never sees — is held closed by two things: + +- The backend-keyed stat-cache + ([`sow-assets-manifest#29`](https://git.westgate.pw/ShadowsOverWestgate/sow-assets-manifest/pulls/29)) + stays exactly as it is; do not "simplify" it (see the 2026-07-04 update above). +- `make haks` gains a trailing `crucible depot status --manifests assets + --target cdn` (credential-free, report-only, non-fatal) so drift is visible + the moment it is created instead of at release time; and + `crucible depot push --source workspace/depot --target bunny` is documented + as the standard post-edit step (field note 4's ask, landed here). Upload when + you want the content live; build local-only when you don't — but either way + the drift is *reported*, never masked. + ## Exit-code contract Matches Crucible's fail-closed convention: - `0` — clean / success. -- distinct non-zero (proposed `1`) — **drift found** (`status`/`push` saw - referenced-but-absent blobs), so `release.yml` can gate on `status` before - packing. +- `1` — **drift found** (`status`/`push` saw referenced-but-absent blobs), so + `release.yml` can gate on `status` before packing. +- `2` — **unconfirmed only**: no confirmed-missing blobs, but the probe budget + ran out before every blob got a confirmed reply (throttled edge). The gate + should retry, not report drift. - `70` — internal/backend error; never a faked result. ## Testing @@ -255,6 +337,10 @@ Matches Crucible's fail-closed convention: `Checksum: ` header, and the read/write key split — no real Bunny. - **Integration (local→local)**: `status` shows drift → `push` clears it → `status` clean → `push` again is a no-op (idempotency). +- **New-surface tests**: fake-Bunny throttling (`428`) → blobs land in + `unconfirmed`, exit `2`, never `missing`; `pull` into a pre-populated dest + skips present-and-hash-ok blobs (incremental); credentialed op with no key + exits non-zero without reading stdin (read path and write path both). ## Non-goals (Increment 1) @@ -262,7 +348,11 @@ Matches Crucible's fail-closed convention: - mdl integrity checks, `prune`, `gc`, `export`, `mirror` ergonomics (Increment 3). - Any change to the depot wire protocol, blob layout, or Bunny account config. -- A TUI. +- A depot-specific rich TUI. **Direction note:** Crucible is headed toward being + a TUI overall — the existing interactive dispatcher menu is the first step, + and depot joins it for free via the registry wiring above. What Increment 1 + does not build is anything richer than that menu (progress UIs, pickers, + dashboards); those come with the Crucible-wide TUI work, not per-tool. ## Immediate operational payoff @@ -276,8 +366,8 @@ Increment 1 first both fixes the tooling and clears the current outage. ## Risks / open questions -- `BUNNY_STORAGE_HOST` default: confirm the canonical storage host to bake as the - fallback (bash referenced it without a visible default in the reviewed range). +- ~~`BUNNY_STORAGE_HOST` default~~ — resolved: env-required, no baked default + (see Backends & config). - Manifest schema coupling: `crucible depot` now depends on the `assets/*.yml` shape (`assets[].{path,sha256,size,restype,hak}`). Acceptable and intentional, but a schema change now touches Crucible.