sow-tools/docs/superpowers/specs/2026-07-04-crucible-depot-core-design.md

Crucible Depot Core Design (Increment 1)

Goal

Give Crucible real ownership of the content-addressed asset depot. Today crucible-depot is a registered-but-unwired stub (fails closed, exit 70), and the actual depot logic lives as ~2000 lines of bash embedded in sow-assets-manifest/scripts — a direct violation of the Crucible principle that artifact repos invoke Crucible through wrappers and never embed a toolkit.

Increment 1 wires the depot core — blob backends plus status / push / verify / get — into crucible depot, and cuts the manifest repo over to call it directly. The bash depot layer is retired, not wrapped: one tool owns depot byte-motion for both local workflows and CI/CD, with complete parity.

Why now (root cause that triggered this)

Release run 222 (v0.1.5) failed in 45s at the "Warm depot cache" step. PR #27's mdl-name normalization recompiled the corpus, producing 5903 net-new sha256s in assets/*.yml. Sampling proved 15/15 net-new blobs return HTTP 404 on the CDN while 10/10 old blobs return 200 — the CDN works; the new content was never uploaded. The blobs exist in the local depot (workspace/depot, 69,473 blobs) but never reached Bunny. mirror.sh then 404s and dies — the release is correctly failing closed on a depot missing data.

Root cause of the missing upload: import.sh's stat-cache (.cache/import/<cat>.tsv) records (path,size,mtime,sha) after any import and excludes "unchanged" files from the only upload call (depot_put_many) — without tracking which backend those bytes actually landed in. A prior local import poisons the cache so a later bunny import silently skips the upload. Combined with depot_require_write only normalizing cdn/unset→bunny (never local→anything), it is very easy to leave the manifests referencing blobs that exist only in the local depot.

The design principle that kills this bug class: presence is always checked against the real target backend, never a cache.

Update 2026-07-04 — interim bash fix landed + corrected drift measurement

The bug above was fixed in bash ahead of this rewrite (this spec's Increment 2 retires the stat-cache concept entirely, making the fix structural): sow-assets-manifest#29 keys the stat-cache dir by write backend + target (.cache/import/<target>/, scripts/import.sh), so a local import (make haks) can no longer mark a blob "done" in a cache the next bunny sync trusts. A cross-backend regression test guards it. When Increment 1/2 delete the bash depot layer, this becomes moot — but until then, do not "simplify" the stat-cache back to a single shared dir; that reintroduces the exact orphaning that broke run 222.

Corrected drift numbers. The "15/15 sampled net-new blobs 404" reading in the section above is a sampling artifact, not the real drift. The CDN edge (cdn-a7f3k9.westgate.pw) is IPv6-reachable but resets HTTP/2 from some hosts, and it throttles concurrent HEADs with 428/000 — which produce both false-404s and false-200s. A reliable sweep (force curl -4, 1-byte range GET -r 0-0 → expect 206, low concurrency, then serial re-confirm any non-2xx) shows the true missing set was exactly 2 net-new blobs, not ~5903:

• item/_shared_aenea_weaponvfx/wblpl_fxshblk.mdl (0b1d3f…)
• item/_shared_aenea_weaponvfx/wxdbsc_fxshpur.mdl (4e8d4f…)

A 600-blob random sample of the pre-existing corpus was 100% present. mirror.sh still correctly failed the release — it dies on the first missing blob — but the gap was two files a local make haks had primed in the shared cache, not a wholesale upload failure. Both blobs were re-uploaded to Bunny from workspace/depot (present in the local depot, sha-verified), unblocking the v0.1.5 re-run.

Takeaway for crucible depot status/verify: probe over IPv4, treat a single concurrent-probe non-2xx as unconfirmed (re-check serially) before reporting a blob missing, so the tool doesn't over-report drift the way an ad-hoc HEAD sweep did here.

Field notes 2026-07-04 (content-side session) — issues the migration must fix

Surfaced while verifying a content edit (over/ravenloft_potm_blood, 18 blobs) had actually reached the CDN. All corroborate the design above; #1 is a new gap to close in Increment 1.

1. push must be non-interactive — no tty prompt (new gap). Today the only write path, depot_require_write (lib.sh), prompts for BUNNY_STORAGE_PASSWORD on a tty when it is unset. Headless agents and CI-without-a-pre-exported-secret cannot answer that prompt, so there is no way to push except with a human at a terminal — which is exactly why the current operational workaround is "an agent/human manually pushes each fixed asset." crucible depot push must take the write key from env only, fail closed with a clear message when it is absent, and never prompt. The Backends section lists the env names but does not state the no-prompt requirement — make it explicit and add a test that a missing write key exits non-zero without reading stdin.

2. status --target cdn is the everyday author check, not only the release gate. The spec frames status as the release pre-flight (correct), but the same command, credential-free against cdn, is the "did my edit actually reach the CDN?" check content authors need routinely. This session answered that question by hand-probing 18 sha256s with curl — precisely the toil crucible depot status --manifests assets --target cdn should replace. Document it as the canonical author-facing "is my content live?" command so people stop ad-hoc probing.

3. Probe reliability confirmed again — bake it into the tests. Reproduced the HEAD hazard directly: a curl -sI HEAD sweep of the 18 blobs returned all-200; the prescribed reliable method (curl -4, 1-byte range GET -r 0-0 → 206, serial) also returned all-206 with the fake-sha control at 404. HEAD did not false-negative this time, but it stays the wrong tool. Make the IPv4 range-GET probe a hard requirement and add a test that fails if a HEAD-based existence check is introduced.

4. Drift is recurring, not the single v0.1.5 incident. Operationally, local→CDN sync is treated as "basically broken": every make haks / local import can re-orphan blobs from Bunny, so authors push by hand after edits. That raises Increment 1's priority and argues for making crucible depot push a standard post-edit step (or an on-edit CI hook), not only a release-time gate.

Ownership principle (from the maintainer)

• Crucible owns the depot tools. No wrapper scripts preserved for their own sake.
• Every consumer repo already has crucible in its flake; that is the only dependency. Consumers call crucible depot … directly (Makefile targets and CI workflow steps alike).
• Complete parity: the same crucible depot commands serve local workflows and CI/CD. No divergent code paths, no "CI-only" or "local-only" logic.
• Total clean-out over legacy preservation: bash depot code is deleted as its Go replacement lands, not kept as a fallback.

Scope

This spec covers Increment 1 only. Roadmap for context:

• Increment 1 (this spec): crucible depot {status, push, verify, get, pull}
  • blob backends (local / cdn / bunny). Cut over mirror/verify/release gate. Delete the lib.sh depot functions that lose their last caller (see Cutover for what actually remains until Increments 2–3).
• Increment 2: crucible depot import / sync (edit-tree → manifest+depot), replacing import.sh / sync-assets.sh. Retire the stat-cache concept.
• Increment 3: mdl integrity checks, mirror/pull ergonomics, prune, gc, export. Delete remaining bash.

Each increment ships independently and deletes the bash it replaces.

Command surface (Increment 1)

crucible depot status  [--manifests DIR] [--source LOCAL_DEPOT] --target bunny|cdn|local
crucible depot push    [--manifests DIR] --source LOCAL_DEPOT   --target bunny
crucible depot verify  [--manifests DIR] --target bunny|cdn     [--sample N]
crucible depot get     <sha> <dest>       --target cdn|bunny|local
crucible depot pull    [--manifests DIR] --dest DIR             --target cdn|bunny

• pull is the bulk fetch that replaces mirror.sh: every referenced sha into --dest/sha256/ab/cd/<sha>, incremental (skip blobs already present with a matching hash — the release warm step re-runs against a persistent /var/cache dir holding ~69k blobs; a non-incremental pull would re-download the corpus every release), sha re-verified on download, parallel with the same probe/transfer bounds as status. Without pull, Increment 1 cannot delete mirror.sh (single-blob get is not a replacement).

• --manifests defaults to assets/ (the repo's manifest dir). Confirmed decision: crucible depot parses assets/*.yml directly (same coupling model as crucible-hak reading hak manifests); the caller does not pre-extract sha lists.

• --source is the local content-addressed store used as the byte source for push (e.g. workspace/depot).

• Config/auth is env-first with flags as local aliases (see Backends).

Drift model (the fix)

The manifest is the source of truth for what must exist; the target backend is the source of truth for what does exist; push reconciles them. There is no cache of "what I uploaded."

• status — collect every sha256 referenced by --manifests/*.yml, run one parallel existence sweep against --target, report {referenced, present, missing, unconfirmed}. Non-zero exit if any referenced blob is missing. This is the release pre-flight gate.

  unconfirmed is a distinct state, not missing. Any non-2xx from the parallel sweep is re-probed serially with backoff (the bash depot_confirm_absent behavior). If the serial re-probe budget is exhausted (bash caps at 200; keep a cap, DEPOT_CONFIRM_MAX), the remainder is reported as unconfirmed — not collapsed into missing. For push the distinction is harmless (unconfirmed → treat as absent → re-upload; wasteful but idempotent and safe). For status as the release gate it is critical: a throttled CDN edge must produce "N unconfirmed, retry" (distinct exit code, proposed 2), never a false "thousands missing" hard-fail — exactly the over-reporting the corrected v0.1.5 drift measurement documented above.

  Target authority split: bunny probes storage — the origin, authoritative, needs the read key; this is what the release gate uses. cdn probes the edge — credential-free, what players actually fetch, the author-facing "is my content live?" check; the edge cache can briefly lag storage (bash carries cdn_purge for this reason). A blob present on bunny but missing on cdn is propagation lag, not drift.

• push — run status, then for each missing sha read the bytes from --source and upload to --target. Stateless and idempotent; re-running does nothing once clean. Cannot silently skip an upload — "should I upload this?" is answered by probing the target, not by trusting a local record.

• verify — existence sweep plus a bounded random sample downloaded and re-hashed (the current --full tag-build check), decoupled from packing.

• get — single-blob fetch with sha re-verify (backfill / debugging).

Backends & config

Faithful port of lib.sh semantics — this is a re-home, not a redesign of the wire protocol.

Backend	Read	Write	Role
local	filesystem sha256/ab/cd/<sha>	filesystem	dev/CI; byte source for push
cdn	CDN GET + sha re-verify	✗ read-only, fails closed	credential-free public reads
bunny	CDN GET, storage fallback	Storage PUT	the only writable remote

Layout: blob_key(sha) = sha256/<sha[0:2]>/<sha[2:4]>/<sha>.

Config (env names unchanged so existing CI secrets and release.yml keep working untouched; flags are local aliases):

• DEPOT_CDN_BASE / BUNNY_CDN_BASE — CDN read base.
• BUNNY_STORAGE_HOST, BUNNY_STORAGE_ZONE — Bunny storage endpoint.
• Read/write key split (preserved): BUNNY_STORAGE_READ_PASSWORD (falls back to BUNNY_STORAGE_PASSWORD) for probes/reads; BUNNY_STORAGE_PASSWORD for PUT. Note for the release.yml cutover: status --target bunny as the gate step needs at least the read key exported in that step's env.
• No prompting, read path included. The no-prompt rule from field note 1 applies to every credentialed operation, not just push: status/verify/ get against bunny with no read key fail closed with a clear message — never a hang, never a tty read.
• BUNNY_STORAGE_HOST stays env-required, no baked default — the bash never defaulted it (depot_require_write defaults only the zone sow-assets-depot and the CDN base); resolving the earlier open question.
• Concurrency: DEPOT_PROBE_JOBS (read probes), DEPOT_JOBS (uploads); DEPOT_CONNECT_TIMEOUT, DEPOT_PROBE_MAX_TIME per-transfer bounds.

Invariants carried over exactly:

• Existence probe is an IPv4 1-byte range request (no body download) — for all backends. This is a deliberate deviation from lib.sh, whose cdn backend probes with HEAD (_depot_cdn_probe_code); the "faithful port" clause above does not extend to the probe method. HEAD is banned (see field note 3 and the test requirement there).
• Upload is PUT with Checksum: <UPPER-sha> so Bunny rejects corrupt writes server-side; each sha uploaded at most once per run.
• Fail-safe: a blob with no confirmed-present reply is treated as absent and (re)uploaded — never silently skipped. This is the invariant the stat-cache violated; here it is structural, not cached.
• get re-hashes every downloaded blob and deletes on mismatch.
• cdn write operations fail closed (never fake a write).

Stale-wording note to reconcile in code/docs: the dispatch registry summarizes depot as "(SeaweedFS)", but the real depot is Bunny-CDN + local (sow-assets-manifest/AGENTS.md: "No S3, no SeaweedFS"). The Go implementation and the registry summary adopt the Bunny/local/cdn reality.

Data flow

assets/*.yml ──parse──▶ referenced sha set ─┐
                                            ├─▶ target.HasMany() ─▶ missing set
                        --target (bunny) ───┘                          │
                                                                       ▼
                     --source (workspace/depot) ──read bytes──▶ target.PutMany()

Cutover / clean-out

Increment 1 deletes the bash it replaces (no wrappers left behind):

• scripts/mirror.sh → crucible depot pull; script removed.
• build-haks.sh's --full depot-verify block → crucible depot verify.
• All three depot-touching workflows switch in the same change (not just release.yml):
  • release.yml "Warm depot cache (verified full pull)" → crucible depot status --target bunny (gate) then crucible depot pull --dest $DEPOT_CACHE_DIR; the inline bash ticker/compgen logic goes away.
  • publish-haks.yml mirrors the same warm+verify+pack flow — same swap.
  • build-haks.yml (PR check, cdn-backend verification) → crucible depot verify --target cdn.
• Registry wiring: the depot entry in internal/dispatch flips to Wired: true with its Commands list populated (status/push/verify/get/pull) and the stale "(SeaweedFS)" summary corrected. This automatically puts depot in the interactive crucible menu alongside the other wired tools — no menu-specific code needed.

What lib.sh deletion actually looks like. The depot layer cannot be fully removed in Increment 1 — it is still referenced by scripts outside this increment's scope:

• import.sh / sync-assets.sh (make haks, make import, make sync) — Increment 2.
• export.sh (make pull), pack-haks.sh (release-time DEPOT_BACKEND=bunny packing), publish-release.sh + release_put, and the path-addressed artifact_has/get/put + cdn_purge helpers — Increment 3 (these were previously unlisted; they are now explicitly Increment 3 scope).

Increment 1 deletes the functions whose last caller it removes (the mirror/verify probe-and-get paths) and leaves the rest with a header comment marking them scheduled for deletion in Increments 2–3. No compatibility shims, no new wrappers — but no pretending lib.sh dies before its last caller does.

Consumers call crucible depot … directly. Nothing new is wrapped.

Interim contract for local builds (until Increment 2)

make haks keeps running bash sync-assets.sh with DEPOT_BACKEND=local until Increment 2 replaces import/sync. The gap this leaves — local imports creating blobs Bunny never sees — is held closed by two things:

• The backend-keyed stat-cache (sow-assets-manifest#29) stays exactly as it is; do not "simplify" it (see the 2026-07-04 update above).
• make haks gains a trailing crucible depot status --manifests assets --target cdn (credential-free, report-only, non-fatal) so drift is visible the moment it is created instead of at release time; and crucible depot push --source workspace/depot --target bunny is documented as the standard post-edit step (field note 4's ask, landed here). Upload when you want the content live; build local-only when you don't — but either way the drift is reported, never masked.

Exit-code contract

Matches Crucible's fail-closed convention:

• 0 — clean / success.
• 1 — drift found (status/push saw referenced-but-absent blobs), so release.yml can gate on status before packing.
• 2 — unconfirmed only: no confirmed-missing blobs, but the probe budget ran out before every blob got a confirmed reply (throttled edge). The gate should retry, not report drift.
• 70 — internal/backend error; never a faked result.

Testing

• Go unit tests (internal/depot): local backend round-trip against a temp dir; manifest parsing; drift computation (referenced − present = missing) as table tests; backend selection + config resolution (incl. read/write key fallback).
• httptest fake-Bunny: assert the range-probe-then-PUT sequence, the Checksum: <UPPER-sha> header, and the read/write key split — no real Bunny.
• Integration (local→local): status shows drift → push clears it → status clean → push again is a no-op (idempotency).
• New-surface tests: fake-Bunny throttling (428) → blobs land in unconfirmed, exit 2, never missing; pull into a pre-populated dest skips present-and-hash-ok blobs (incremental); credentialed op with no key exits non-zero without reading stdin (read path and write path both).

Non-goals (Increment 1)

• import / sync / edit-tree ingestion (Increment 2).
• mdl integrity checks, prune, gc, export, mirror ergonomics (Increment 3).
• Any change to the depot wire protocol, blob layout, or Bunny account config.
• A depot-specific rich TUI. Direction note: Crucible is headed toward being a TUI overall — the existing interactive dispatcher menu is the first step, and depot joins it for free via the registry wiring above. What Increment 1 does not build is anything richer than that menu (progress UIs, pickers, dashboards); those come with the Crucible-wide TUI work, not per-tool.

Immediate operational payoff

crucible depot push --source workspace/depot --target bunny is exactly the command that unblocks the stuck v0.1.5 release: it probes the target and uploads whatever is actually 404 while it re-hydrates from the local depot. (The 5903 figure below is superseded — see the 2026-07-04 update above: the real drift was 2 blobs, already re-uploaded; push would have been the clean way to do it and is idempotent once the depot is whole.) Building Increment 1 first both fixes the tooling and clears the current outage.

Risks / open questions

• BUNNY_STORAGE_HOST default — resolved: env-required, no baked default (see Backends & config).
• Manifest schema coupling: crucible depot now depends on the assets/*.yml shape (assets[].{path,sha256,size,restype,hak}). Acceptable and intentional, but a schema change now touches Crucible.
• Cutover ordering: release.yml and Makefile must switch to crucible depot in the same change that removes the bash, to avoid a window where both exist.