sow-tools/docs/superpowers/specs/2026-07-04-crucible-depot-core-design.md

# Crucible Depot Core Design (Increment 1)

## Goal

Give Crucible real ownership of the content-addressed asset depot. Today
`crucible-depot` is a registered-but-unwired stub (fails closed, exit `70`), and
the actual depot logic lives as ~2000 lines of bash embedded in
`sow-assets-manifest/scripts` — a direct violation of the Crucible principle that
*artifact repos invoke Crucible through wrappers and never embed a toolkit*.

Increment 1 wires the depot **core** — blob backends plus `status` / `push` /
`verify` / `get` — into `crucible depot`, and cuts the manifest repo over to call
it directly. The bash depot layer is **retired, not wrapped**: one tool owns
depot byte-motion for both local workflows and CI/CD, with complete parity.

## Why now (root cause that triggered this)

Release run 222 (`v0.1.5`) failed in 45s at the "Warm depot cache" step. PR #27's
mdl-name normalization recompiled the corpus, producing **5903 net-new sha256s**
in `assets/*.yml`. Sampling proved **15/15 net-new blobs return HTTP 404 on the
CDN** while **10/10 old blobs return 200** — the CDN works; the new content was
never uploaded. The blobs exist in the local depot (`workspace/depot`, 69,473
blobs) but never reached Bunny. `mirror.sh` then 404s and dies — the release is
correctly failing closed on a depot missing data.

Root cause of the *missing upload*: `import.sh`'s stat-cache
(`.cache/import/<cat>.tsv`) records `(path,size,mtime,sha)` after **any** import
and excludes "unchanged" files from the only upload call (`depot_put_many`) —
**without tracking which backend those bytes actually landed in**. A prior
`local` import poisons the cache so a later `bunny` import silently skips the
upload. Combined with `depot_require_write` only normalizing `cdn`/unset→`bunny`
(never `local`→anything), it is very easy to leave the manifests referencing
blobs that exist only in the local depot.

The design principle that kills this bug class: **presence is always checked
against the real target backend, never a cache.**

### Update 2026-07-04 — interim bash fix landed + corrected drift measurement

The bug above was fixed in bash ahead of this rewrite (this spec's Increment 2
retires the stat-cache concept entirely, making the fix structural):
[`sow-assets-manifest#29`](https://git.westgate.pw/ShadowsOverWestgate/sow-assets-manifest/pulls/29)
keys the stat-cache dir by write backend + target
(`.cache/import/<target>/`, `scripts/import.sh`), so a `local` import
(`make haks`) can no longer mark a blob "done" in a cache the next `bunny` sync
trusts. A cross-backend regression test guards it. When Increment 1/2 delete the
bash depot layer, this becomes moot — but until then, **do not "simplify" the
stat-cache back to a single shared dir**; that reintroduces the exact orphaning
that broke run 222.

**Corrected drift numbers.** The "15/15 sampled net-new blobs 404" reading in the
section above is a *sampling artifact*, not the real drift. The CDN edge
(`cdn-a7f3k9.westgate.pw`) is IPv6-reachable but resets HTTP/2 from some hosts,
and it throttles concurrent HEADs with `428`/`000` — which produce **both**
false-404s and false-200s. A reliable sweep (force `curl -4`, 1-byte range GET
`-r 0-0` → expect `206`, low concurrency, then serial re-confirm any non-2xx)
shows the true missing set was **exactly 2 net-new blobs**, not ~5903:

- `item/_shared_aenea_weaponvfx/wblpl_fxshblk.mdl` (`0b1d3f…`)
- `item/_shared_aenea_weaponvfx/wxdbsc_fxshpur.mdl` (`4e8d4f…`)

A 600-blob random sample of the pre-existing corpus was 100% present. `mirror.sh`
still correctly failed the release — it dies on the first missing blob — but the
gap was two files a local `make haks` had primed in the shared cache, not a
wholesale upload failure. Both blobs were re-uploaded to Bunny from
`workspace/depot` (present in the local depot, sha-verified), unblocking the
`v0.1.5` re-run.

Takeaway for `crucible depot status`/`verify`: probe over IPv4, treat a single
concurrent-probe non-2xx as *unconfirmed* (re-check serially) before reporting a
blob missing, so the tool doesn't over-report drift the way an ad-hoc HEAD sweep
did here.

### Field notes 2026-07-04 (content-side session) — issues the migration must fix

Surfaced while verifying a content edit (`over/ravenloft_potm_blood`, 18 blobs) had
actually reached the CDN. All corroborate the design above; #1 is a new gap to close in
Increment 1.

1. **`push` must be non-interactive — no tty prompt (new gap).** Today the only write
   path, `depot_require_write` (`lib.sh`), *prompts* for `BUNNY_STORAGE_PASSWORD` on a tty
   when it is unset. Headless agents and CI-without-a-pre-exported-secret cannot answer
   that prompt, so there is no way to push except with a human at a terminal — which is
   exactly why the current operational workaround is "an agent/human manually pushes each
   fixed asset." `crucible depot push` must take the write key **from env only**, fail
   closed with a clear message when it is absent, and **never prompt**. The Backends
   section lists the env names but does not state the no-prompt requirement — make it
   explicit and add a test that a missing write key exits non-zero without reading stdin.

2. **`status --target cdn` is the everyday author check, not only the release gate.** The
   spec frames `status` as the release pre-flight (correct), but the same command,
   credential-free against `cdn`, is the "did my edit actually reach the CDN?" check
   content authors need routinely. This session answered that question by hand-probing 18
   sha256s with `curl` — precisely the toil `crucible depot status --manifests assets
   --target cdn` should replace. Document it as the canonical author-facing "is my content
   live?" command so people stop ad-hoc probing.

3. **Probe reliability confirmed again — bake it into the tests.** Reproduced the HEAD
   hazard directly: a `curl -sI` HEAD sweep of the 18 blobs returned all-`200`; the
   prescribed reliable method (`curl -4`, 1-byte range GET `-r 0-0` → `206`, serial) also
   returned all-`206` with the fake-sha control at `404`. HEAD did not false-negative this
   time, but it stays the wrong tool. Make the IPv4 range-GET probe a hard requirement and
   add a test that fails if a HEAD-based existence check is introduced.

4. **Drift is recurring, not the single v0.1.5 incident.** Operationally, local→CDN sync is
   treated as "basically broken": every `make haks` / local import can re-orphan blobs from
   Bunny, so authors push by hand after edits. That raises Increment 1's priority and argues
   for making `crucible depot push` a standard post-edit step (or an on-edit CI hook), not
   only a release-time gate.

## Ownership principle (from the maintainer)

- Crucible **owns** the depot tools. No wrapper scripts preserved for their own
  sake.
- Every consumer repo already has `crucible` in its flake; that is the only
  dependency. Consumers call `crucible depot …` directly (Makefile targets and
  CI workflow steps alike).
- **Complete parity**: the same `crucible depot` commands serve local workflows
  and CI/CD. No divergent code paths, no "CI-only" or "local-only" logic.
- Total clean-out over legacy preservation: bash depot code is **deleted** as its
  Go replacement lands, not kept as a fallback.

## Scope

This spec covers **Increment 1** only. Roadmap for context:

- **Increment 1 (this spec):** `crucible depot {status, push, verify, get, pull}`
  + blob backends (`local` / `cdn` / `bunny`). Cut over `mirror`/verify/release
  gate. Delete the `lib.sh` depot functions that lose their last caller (see
  Cutover for what actually remains until Increments 2–3).
- **Increment 2:** `crucible depot import` / `sync` (edit-tree → manifest+depot),
  replacing `import.sh` / `sync-assets.sh`. Retire the stat-cache concept.
- **Increment 3:** mdl integrity checks, `mirror`/`pull` ergonomics, `prune`,
  `gc`, `export`. Delete remaining bash.

Each increment ships independently and deletes the bash it replaces.

## Command surface (Increment 1)

```
crucible depot status  [--manifests DIR] [--source LOCAL_DEPOT] --target bunny|cdn|local
crucible depot push    [--manifests DIR] --source LOCAL_DEPOT   --target bunny
crucible depot verify  [--manifests DIR] --target bunny|cdn     [--sample N]
crucible depot get     <sha> <dest>       --target cdn|bunny|local
crucible depot pull    [--manifests DIR] --dest DIR             --target cdn|bunny
```

- `pull` is the bulk fetch that replaces `mirror.sh`: every referenced sha into
  `--dest/sha256/ab/cd/<sha>`, **incremental** (skip blobs already present with a
  matching hash — the release warm step re-runs against a persistent
  `/var/cache` dir holding ~69k blobs; a non-incremental pull would re-download
  the corpus every release), sha re-verified on download, parallel with the same
  probe/transfer bounds as `status`. Without `pull`, Increment 1 cannot delete
  `mirror.sh` (single-blob `get` is not a replacement).

- `--manifests` defaults to `assets/` (the repo's manifest dir). Confirmed
  decision: `crucible depot` parses `assets/*.yml` directly (same coupling model
  as `crucible-hak` reading hak manifests); the caller does not pre-extract sha
  lists.
- `--source` is the local content-addressed store used as the byte source for
  `push` (e.g. `workspace/depot`).
- Config/auth is env-first with flags as local aliases (see Backends).

## Drift model (the fix)

The manifest is the source of truth for **what** must exist; the target backend
is the source of truth for **what does** exist; `push` reconciles them. There is
no cache of "what I uploaded."

- **`status`** — collect every `sha256` referenced by `--manifests/*.yml`, run
  one parallel existence sweep against `--target`, report
  `{referenced, present, missing, unconfirmed}`. Non-zero exit if any referenced
  blob is missing. This is the release pre-flight gate.

  **`unconfirmed` is a distinct state, not `missing`.** Any non-2xx from the
  parallel sweep is re-probed serially with backoff (the bash
  `depot_confirm_absent` behavior). If the serial re-probe budget is exhausted
  (bash caps at 200; keep a cap, `DEPOT_CONFIRM_MAX`), the remainder is reported
  as `unconfirmed` — **not** collapsed into `missing`. For `push` the
  distinction is harmless (unconfirmed → treat as absent → re-upload; wasteful
  but idempotent and safe). For `status` as the release gate it is critical: a
  throttled CDN edge must produce "N unconfirmed, retry" (distinct exit code,
  proposed `2`), never a false "thousands missing" hard-fail — exactly the
  over-reporting the corrected v0.1.5 drift measurement documented above.

  **Target authority split:** `bunny` probes storage — the origin, authoritative,
  needs the read key; this is what the release gate uses. `cdn` probes the edge —
  credential-free, what players actually fetch, the author-facing "is my content
  live?" check; the edge cache can briefly lag storage (bash carries `cdn_purge`
  for this reason). A blob `present` on bunny but `missing` on cdn is
  propagation lag, not drift.
- **`push`** — run `status`, then for each missing sha read the bytes from
  `--source` and upload to `--target`. Stateless and idempotent; re-running does
  nothing once clean. **Cannot silently skip an upload** — "should I upload this?"
  is answered by probing the target, not by trusting a local record.
- **`verify`** — existence sweep plus a bounded random sample downloaded and
  re-hashed (the current `--full` tag-build check), decoupled from packing.
- **`get`** — single-blob fetch with sha re-verify (backfill / debugging).

## Backends & config

Faithful port of `lib.sh` semantics — this is a re-home, not a redesign of the
wire protocol.

| Backend | Read | Write | Role |
|---|---|---|---|
| `local` | filesystem `sha256/ab/cd/<sha>` | filesystem | dev/CI; byte source for `push` |
| `cdn`   | CDN GET + sha re-verify | ✗ read-only, fails closed | credential-free public reads |
| `bunny` | CDN GET, storage fallback | Storage `PUT` | the only writable remote |

Layout: `blob_key(sha) = sha256/<sha[0:2]>/<sha[2:4]>/<sha>`.

Config (env names unchanged so existing CI secrets and `release.yml` keep working
untouched; flags are local aliases):

- `DEPOT_CDN_BASE` / `BUNNY_CDN_BASE` — CDN read base.
- `BUNNY_STORAGE_HOST`, `BUNNY_STORAGE_ZONE` — Bunny storage endpoint.
- **Read/write key split (preserved):** `BUNNY_STORAGE_READ_PASSWORD` (falls back
  to `BUNNY_STORAGE_PASSWORD`) for probes/reads; `BUNNY_STORAGE_PASSWORD` for
  `PUT`. Note for the `release.yml` cutover: `status --target bunny` as the gate
  step needs at least the read key exported in that step's env.
- **No prompting, read path included.** The no-prompt rule from field note 1
  applies to every credentialed operation, not just `push`: `status`/`verify`/
  `get` against `bunny` with no read key fail closed with a clear message —
  never a hang, never a tty read.
- `BUNNY_STORAGE_HOST` stays **env-required, no baked default** — the bash never
  defaulted it (`depot_require_write` defaults only the zone
  `sow-assets-depot` and the CDN base); resolving the earlier open question.
- Concurrency: `DEPOT_PROBE_JOBS` (read probes), `DEPOT_JOBS` (uploads);
  `DEPOT_CONNECT_TIMEOUT`, `DEPOT_PROBE_MAX_TIME` per-transfer bounds.

Invariants carried over exactly:

- **Existence probe** is an IPv4 1-byte range request (no body download) — for
  **all** backends. This is a deliberate deviation from `lib.sh`, whose `cdn`
  backend probes with HEAD (`_depot_cdn_probe_code`); the "faithful port"
  clause above does not extend to the probe method. HEAD is banned (see field
  note 3 and the test requirement there).
- **Upload** is `PUT` with `Checksum: <UPPER-sha>` so Bunny rejects corrupt
  writes server-side; each sha uploaded at most once per run.
- **Fail-safe:** a blob with no confirmed-present reply is treated as absent and
  (re)uploaded — never silently skipped. This is the invariant the stat-cache
  violated; here it is structural, not cached.
- **`get`** re-hashes every downloaded blob and deletes on mismatch.
- `cdn` write operations fail closed (never fake a write).

Stale-wording note to reconcile in code/docs: the dispatch registry summarizes
depot as *"(SeaweedFS)"*, but the real depot is Bunny-CDN + local
(`sow-assets-manifest/AGENTS.md`: "No S3, no SeaweedFS"). The Go implementation
and the registry summary adopt the Bunny/local/cdn reality.

## Data flow

```
assets/*.yml ──parse──▶ referenced sha set ─┐
                                            ├─▶ target.HasMany() ─▶ missing set
                        --target (bunny) ───┘                          │
                                                                       ▼
                     --source (workspace/depot) ──read bytes──▶ target.PutMany()
```

## Cutover / clean-out

Increment 1 deletes the bash it replaces (no wrappers left behind):

- `scripts/mirror.sh` → `crucible depot pull`; script removed.
- `build-haks.sh`'s `--full` depot-verify block → `crucible depot verify`.
- **All three depot-touching workflows switch in the same change** (not just
  `release.yml`):
  - `release.yml` "Warm depot cache (verified full pull)" →
    `crucible depot status --target bunny` (gate) then
    `crucible depot pull --dest $DEPOT_CACHE_DIR`; the inline bash
    ticker/compgen logic goes away.
  - `publish-haks.yml` mirrors the same warm+verify+pack flow — same swap.
  - `build-haks.yml` (PR check, cdn-backend verification) →
    `crucible depot verify --target cdn`.
- Registry wiring: the `depot` entry in `internal/dispatch` flips to
  `Wired: true` with its `Commands` list populated (status/push/verify/get/pull)
  and the stale "(SeaweedFS)" summary corrected. This automatically puts depot
  in the interactive `crucible` menu alongside the other wired tools — no
  menu-specific code needed.

**What `lib.sh` deletion actually looks like.** The depot layer cannot be fully
removed in Increment 1 — it is still referenced by scripts outside this
increment's scope:

- `import.sh` / `sync-assets.sh` (`make haks`, `make import`, `make sync`) —
  Increment 2.
- `export.sh` (`make pull`), `pack-haks.sh` (release-time `DEPOT_BACKEND=bunny`
  packing), `publish-release.sh` + `release_put`, and the path-addressed
  `artifact_has/get/put` + `cdn_purge` helpers — Increment 3 (these were
  previously unlisted; they are now explicitly Increment 3 scope).

Increment 1 deletes the functions whose last caller it removes (the
mirror/verify probe-and-get paths) and leaves the rest with a header comment
marking them scheduled for deletion in Increments 2–3. No compatibility shims,
no new wrappers — but no pretending `lib.sh` dies before its last caller does.

Consumers call `crucible depot …` directly. Nothing new is wrapped.

### Interim contract for local builds (until Increment 2)

`make haks` keeps running bash `sync-assets.sh` with `DEPOT_BACKEND=local` until
Increment 2 replaces import/sync. The gap this leaves — local imports creating
blobs Bunny never sees — is held closed by two things:

- The backend-keyed stat-cache
  ([`sow-assets-manifest#29`](https://git.westgate.pw/ShadowsOverWestgate/sow-assets-manifest/pulls/29))
  stays exactly as it is; do not "simplify" it (see the 2026-07-04 update above).
- `make haks` gains a trailing `crucible depot status --manifests assets
  --target cdn` (credential-free, report-only, non-fatal) so drift is visible
  the moment it is created instead of at release time; and
  `crucible depot push --source workspace/depot --target bunny` is documented
  as the standard post-edit step (field note 4's ask, landed here). Upload when
  you want the content live; build local-only when you don't — but either way
  the drift is *reported*, never masked.

## Exit-code contract

Matches Crucible's fail-closed convention:

- `0` — clean / success.
- `1` — **drift found** (`status`/`push` saw referenced-but-absent blobs), so
  `release.yml` can gate on `status` before packing.
- `2` — **unconfirmed only**: no confirmed-missing blobs, but the probe budget
  ran out before every blob got a confirmed reply (throttled edge). The gate
  should retry, not report drift.
- `70` — internal/backend error; never a faked result.

## Testing

- **Go unit tests** (`internal/depot`): `local` backend round-trip against a temp
  dir; manifest parsing; drift computation (referenced − present = missing) as
  table tests; backend selection + config resolution (incl. read/write key
  fallback).
- **httptest fake-Bunny**: assert the range-probe-then-PUT sequence, the
  `Checksum: <UPPER-sha>` header, and the read/write key split — no real Bunny.
- **Integration (local→local)**: `status` shows drift → `push` clears it →
  `status` clean → `push` again is a no-op (idempotency).
- **New-surface tests**: fake-Bunny throttling (`428`) → blobs land in
  `unconfirmed`, exit `2`, never `missing`; `pull` into a pre-populated dest
  skips present-and-hash-ok blobs (incremental); credentialed op with no key
  exits non-zero without reading stdin (read path and write path both).

## Non-goals (Increment 1)

- `import` / `sync` / edit-tree ingestion (Increment 2).
- mdl integrity checks, `prune`, `gc`, `export`, `mirror` ergonomics
  (Increment 3).
- Any change to the depot wire protocol, blob layout, or Bunny account config.
- A depot-specific rich TUI. **Direction note:** Crucible is headed toward being
  a TUI overall — the existing interactive dispatcher menu is the first step,
  and depot joins it for free via the registry wiring above. What Increment 1
  does not build is anything richer than that menu (progress UIs, pickers,
  dashboards); those come with the Crucible-wide TUI work, not per-tool.

## Immediate operational payoff

`crucible depot push --source workspace/depot --target bunny` is exactly the
command that unblocks the stuck `v0.1.5` release: it probes the target and
uploads whatever is actually 404 while it re-hydrates from the local depot.
(The `5903` figure below is superseded — see the 2026-07-04 update above: the
real drift was **2** blobs, already re-uploaded; `push` would have been the
clean way to do it and is idempotent once the depot is whole.) Building
Increment 1 first both fixes the tooling and clears the current outage.

## Risks / open questions

- ~~`BUNNY_STORAGE_HOST` default~~ — resolved: env-required, no baked default
  (see Backends & config).
- Manifest schema coupling: `crucible depot` now depends on the `assets/*.yml`
  shape (`assets[].{path,sha256,size,restype,hak}`). Acceptable and intentional,
  but a schema change now touches Crucible.
- Cutover ordering: `release.yml` and Makefile must switch to `crucible depot` in
  the same change that removes the bash, to avoid a window where both exist.