diff --git a/docs/superpowers/specs/2026-07-04-crucible-depot-core-design.md b/docs/superpowers/specs/2026-07-04-crucible-depot-core-design.md new file mode 100644 index 0000000..ec13993 --- /dev/null +++ b/docs/superpowers/specs/2026-07-04-crucible-depot-core-design.md @@ -0,0 +1,285 @@ +# Crucible Depot Core Design (Increment 1) + +## Goal + +Give Crucible real ownership of the content-addressed asset depot. Today +`crucible-depot` is a registered-but-unwired stub (fails closed, exit `70`), and +the actual depot logic lives as ~2000 lines of bash embedded in +`sow-assets-manifest/scripts` — a direct violation of the Crucible principle that +*artifact repos invoke Crucible through wrappers and never embed a toolkit*. + +Increment 1 wires the depot **core** — blob backends plus `status` / `push` / +`verify` / `get` — into `crucible depot`, and cuts the manifest repo over to call +it directly. The bash depot layer is **retired, not wrapped**: one tool owns +depot byte-motion for both local workflows and CI/CD, with complete parity. + +## Why now (root cause that triggered this) + +Release run 222 (`v0.1.5`) failed in 45s at the "Warm depot cache" step. PR #27's +mdl-name normalization recompiled the corpus, producing **5903 net-new sha256s** +in `assets/*.yml`. Sampling proved **15/15 net-new blobs return HTTP 404 on the +CDN** while **10/10 old blobs return 200** — the CDN works; the new content was +never uploaded. The blobs exist in the local depot (`workspace/depot`, 69,473 +blobs) but never reached Bunny. `mirror.sh` then 404s and dies — the release is +correctly failing closed on a depot missing data. + +Root cause of the *missing upload*: `import.sh`'s stat-cache +(`.cache/import/.tsv`) records `(path,size,mtime,sha)` after **any** import +and excludes "unchanged" files from the only upload call (`depot_put_many`) — +**without tracking which backend those bytes actually landed in**. A prior +`local` import poisons the cache so a later `bunny` import silently skips the +upload. Combined with `depot_require_write` only normalizing `cdn`/unset→`bunny` +(never `local`→anything), it is very easy to leave the manifests referencing +blobs that exist only in the local depot. + +The design principle that kills this bug class: **presence is always checked +against the real target backend, never a cache.** + +### Update 2026-07-04 — interim bash fix landed + corrected drift measurement + +The bug above was fixed in bash ahead of this rewrite (this spec's Increment 2 +retires the stat-cache concept entirely, making the fix structural): +[`sow-assets-manifest#29`](https://git.westgate.pw/ShadowsOverWestgate/sow-assets-manifest/pulls/29) +keys the stat-cache dir by write backend + target +(`.cache/import//`, `scripts/import.sh`), so a `local` import +(`make haks`) can no longer mark a blob "done" in a cache the next `bunny` sync +trusts. A cross-backend regression test guards it. When Increment 1/2 delete the +bash depot layer, this becomes moot — but until then, **do not "simplify" the +stat-cache back to a single shared dir**; that reintroduces the exact orphaning +that broke run 222. + +**Corrected drift numbers.** The "15/15 sampled net-new blobs 404" reading in the +section above is a *sampling artifact*, not the real drift. The CDN edge +(`cdn-a7f3k9.westgate.pw`) is IPv6-reachable but resets HTTP/2 from some hosts, +and it throttles concurrent HEADs with `428`/`000` — which produce **both** +false-404s and false-200s. A reliable sweep (force `curl -4`, 1-byte range GET +`-r 0-0` → expect `206`, low concurrency, then serial re-confirm any non-2xx) +shows the true missing set was **exactly 2 net-new blobs**, not ~5903: + +- `item/_shared_aenea_weaponvfx/wblpl_fxshblk.mdl` (`0b1d3f…`) +- `item/_shared_aenea_weaponvfx/wxdbsc_fxshpur.mdl` (`4e8d4f…`) + +A 600-blob random sample of the pre-existing corpus was 100% present. `mirror.sh` +still correctly failed the release — it dies on the first missing blob — but the +gap was two files a local `make haks` had primed in the shared cache, not a +wholesale upload failure. Both blobs were re-uploaded to Bunny from +`workspace/depot` (present in the local depot, sha-verified), unblocking the +`v0.1.5` re-run. + +Takeaway for `crucible depot status`/`verify`: probe over IPv4, treat a single +concurrent-probe non-2xx as *unconfirmed* (re-check serially) before reporting a +blob missing, so the tool doesn't over-report drift the way an ad-hoc HEAD sweep +did here. + +### Field notes 2026-07-04 (content-side session) — issues the migration must fix + +Surfaced while verifying a content edit (`over/ravenloft_potm_blood`, 18 blobs) had +actually reached the CDN. All corroborate the design above; #1 is a new gap to close in +Increment 1. + +1. **`push` must be non-interactive — no tty prompt (new gap).** Today the only write + path, `depot_require_write` (`lib.sh`), *prompts* for `BUNNY_STORAGE_PASSWORD` on a tty + when it is unset. Headless agents and CI-without-a-pre-exported-secret cannot answer + that prompt, so there is no way to push except with a human at a terminal — which is + exactly why the current operational workaround is "an agent/human manually pushes each + fixed asset." `crucible depot push` must take the write key **from env only**, fail + closed with a clear message when it is absent, and **never prompt**. The Backends + section lists the env names but does not state the no-prompt requirement — make it + explicit and add a test that a missing write key exits non-zero without reading stdin. + +2. **`status --target cdn` is the everyday author check, not only the release gate.** The + spec frames `status` as the release pre-flight (correct), but the same command, + credential-free against `cdn`, is the "did my edit actually reach the CDN?" check + content authors need routinely. This session answered that question by hand-probing 18 + sha256s with `curl` — precisely the toil `crucible depot status --manifests assets + --target cdn` should replace. Document it as the canonical author-facing "is my content + live?" command so people stop ad-hoc probing. + +3. **Probe reliability confirmed again — bake it into the tests.** Reproduced the HEAD + hazard directly: a `curl -sI` HEAD sweep of the 18 blobs returned all-`200`; the + prescribed reliable method (`curl -4`, 1-byte range GET `-r 0-0` → `206`, serial) also + returned all-`206` with the fake-sha control at `404`. HEAD did not false-negative this + time, but it stays the wrong tool. Make the IPv4 range-GET probe a hard requirement and + add a test that fails if a HEAD-based existence check is introduced. + +4. **Drift is recurring, not the single v0.1.5 incident.** Operationally, local→CDN sync is + treated as "basically broken": every `make haks` / local import can re-orphan blobs from + Bunny, so authors push by hand after edits. That raises Increment 1's priority and argues + for making `crucible depot push` a standard post-edit step (or an on-edit CI hook), not + only a release-time gate. + +## Ownership principle (from the maintainer) + +- Crucible **owns** the depot tools. No wrapper scripts preserved for their own + sake. +- Every consumer repo already has `crucible` in its flake; that is the only + dependency. Consumers call `crucible depot …` directly (Makefile targets and + CI workflow steps alike). +- **Complete parity**: the same `crucible depot` commands serve local workflows + and CI/CD. No divergent code paths, no "CI-only" or "local-only" logic. +- Total clean-out over legacy preservation: bash depot code is **deleted** as its + Go replacement lands, not kept as a fallback. + +## Scope + +This spec covers **Increment 1** only. Roadmap for context: + +- **Increment 1 (this spec):** `crucible depot {status, push, verify, get}` + + blob backends (`local` / `cdn` / `bunny`). Cut over `mirror`/verify/release + gate. Delete the corresponding `lib.sh` depot layer. +- **Increment 2:** `crucible depot import` / `sync` (edit-tree → manifest+depot), + replacing `import.sh` / `sync-assets.sh`. Retire the stat-cache concept. +- **Increment 3:** mdl integrity checks, `mirror`/`pull` ergonomics, `prune`, + `gc`, `export`. Delete remaining bash. + +Each increment ships independently and deletes the bash it replaces. + +## Command surface (Increment 1) + +``` +crucible depot status [--manifests DIR] [--source LOCAL_DEPOT] --target bunny|cdn|local +crucible depot push [--manifests DIR] --source LOCAL_DEPOT --target bunny +crucible depot verify [--manifests DIR] --target bunny|cdn [--sample N] +crucible depot get --target cdn|bunny|local +``` + +- `--manifests` defaults to `assets/` (the repo's manifest dir). Confirmed + decision: `crucible depot` parses `assets/*.yml` directly (same coupling model + as `crucible-hak` reading hak manifests); the caller does not pre-extract sha + lists. +- `--source` is the local content-addressed store used as the byte source for + `push` (e.g. `workspace/depot`). +- Config/auth is env-first with flags as local aliases (see Backends). + +## Drift model (the fix) + +The manifest is the source of truth for **what** must exist; the target backend +is the source of truth for **what does** exist; `push` reconciles them. There is +no cache of "what I uploaded." + +- **`status`** — collect every `sha256` referenced by `--manifests/*.yml`, run + one parallel existence sweep against `--target`, report + `{referenced, present, missing}`. Non-zero exit if any referenced blob is + missing. This is the release pre-flight gate. +- **`push`** — run `status`, then for each missing sha read the bytes from + `--source` and upload to `--target`. Stateless and idempotent; re-running does + nothing once clean. **Cannot silently skip an upload** — "should I upload this?" + is answered by probing the target, not by trusting a local record. +- **`verify`** — existence sweep plus a bounded random sample downloaded and + re-hashed (the current `--full` tag-build check), decoupled from packing. +- **`get`** — single-blob fetch with sha re-verify (backfill / debugging). + +## Backends & config + +Faithful port of `lib.sh` semantics — this is a re-home, not a redesign of the +wire protocol. + +| Backend | Read | Write | Role | +|---|---|---|---| +| `local` | filesystem `sha256/ab/cd/` | filesystem | dev/CI; byte source for `push` | +| `cdn` | CDN GET + sha re-verify | ✗ read-only, fails closed | credential-free public reads | +| `bunny` | CDN GET, storage fallback | Storage `PUT` | the only writable remote | + +Layout: `blob_key(sha) = sha256///`. + +Config (env names unchanged so existing CI secrets and `release.yml` keep working +untouched; flags are local aliases): + +- `DEPOT_CDN_BASE` / `BUNNY_CDN_BASE` — CDN read base. +- `BUNNY_STORAGE_HOST`, `BUNNY_STORAGE_ZONE` — Bunny storage endpoint. +- **Read/write key split (preserved):** `BUNNY_STORAGE_READ_PASSWORD` (falls back + to `BUNNY_STORAGE_PASSWORD`) for probes/reads; `BUNNY_STORAGE_PASSWORD` for + `PUT`. +- Concurrency: `DEPOT_PROBE_JOBS` (read probes), `DEPOT_JOBS` (uploads); + `DEPOT_CONNECT_TIMEOUT`, `DEPOT_PROBE_MAX_TIME` per-transfer bounds. + +Invariants carried over exactly: + +- **Existence probe** is a 1-byte range request (no body download). +- **Upload** is `PUT` with `Checksum: ` so Bunny rejects corrupt + writes server-side; each sha uploaded at most once per run. +- **Fail-safe:** a blob with no confirmed-present reply is treated as absent and + (re)uploaded — never silently skipped. This is the invariant the stat-cache + violated; here it is structural, not cached. +- **`get`** re-hashes every downloaded blob and deletes on mismatch. +- `cdn` write operations fail closed (never fake a write). + +Stale-wording note to reconcile in code/docs: the dispatch registry summarizes +depot as *"(SeaweedFS)"*, but the real depot is Bunny-CDN + local +(`sow-assets-manifest/AGENTS.md`: "No S3, no SeaweedFS"). The Go implementation +and the registry summary adopt the Bunny/local/cdn reality. + +## Data flow + +``` +assets/*.yml ──parse──▶ referenced sha set ─┐ + ├─▶ target.HasMany() ─▶ missing set + --target (bunny) ───┘ │ + ▼ + --source (workspace/depot) ──read bytes──▶ target.PutMany() +``` + +## Cutover / clean-out + +Increment 1 deletes the bash it replaces (no wrappers left behind): + +- `scripts/mirror.sh` → `crucible depot get`/pull path; script removed. +- `build-haks.sh`'s `--full` depot-verify block → `crucible depot verify`. +- `release.yml` step "Warm depot cache" → `crucible depot status` (gate) then the + fetch path; the inline bash ticker/compgen logic goes away. +- `lib.sh` depot layer (`depot_require`, `depot_require_write`, `_depot_*_has`, + `_depot_*_get`, `_depot_*_put*`, `_depot_*_probe*`, `blob_key`, backend + dispatch — ~250+ lines) is removed once no remaining bash references it. + Anything still needed by Increments 2–3 is deleted when those land, not kept as + a compatibility shim. + +Consumers call `crucible depot …` directly. Nothing new is wrapped. + +## Exit-code contract + +Matches Crucible's fail-closed convention: + +- `0` — clean / success. +- distinct non-zero (proposed `1`) — **drift found** (`status`/`push` saw + referenced-but-absent blobs), so `release.yml` can gate on `status` before + packing. +- `70` — internal/backend error; never a faked result. + +## Testing + +- **Go unit tests** (`internal/depot`): `local` backend round-trip against a temp + dir; manifest parsing; drift computation (referenced − present = missing) as + table tests; backend selection + config resolution (incl. read/write key + fallback). +- **httptest fake-Bunny**: assert the range-probe-then-PUT sequence, the + `Checksum: ` header, and the read/write key split — no real Bunny. +- **Integration (local→local)**: `status` shows drift → `push` clears it → + `status` clean → `push` again is a no-op (idempotency). + +## Non-goals (Increment 1) + +- `import` / `sync` / edit-tree ingestion (Increment 2). +- mdl integrity checks, `prune`, `gc`, `export`, `mirror` ergonomics + (Increment 3). +- Any change to the depot wire protocol, blob layout, or Bunny account config. +- A TUI. + +## Immediate operational payoff + +`crucible depot push --source workspace/depot --target bunny` is exactly the +command that unblocks the stuck `v0.1.5` release: it probes the target and +uploads whatever is actually 404 while it re-hydrates from the local depot. +(The `5903` figure below is superseded — see the 2026-07-04 update above: the +real drift was **2** blobs, already re-uploaded; `push` would have been the +clean way to do it and is idempotent once the depot is whole.) Building +Increment 1 first both fixes the tooling and clears the current outage. + +## Risks / open questions + +- `BUNNY_STORAGE_HOST` default: confirm the canonical storage host to bake as the + fallback (bash referenced it without a visible default in the reviewed range). +- Manifest schema coupling: `crucible depot` now depends on the `assets/*.yml` + shape (`assets[].{path,sha256,size,restype,hak}`). Acceptable and intentional, + but a schema change now touches Crucible. +- Cutover ordering: `release.yml` and Makefile must switch to `crucible depot` in + the same change that removes the bash, to avoid a window where both exist.