sow-tools/docs/superpowers/specs/2026-07-04-crucible-depot-core-design.md

Crucible Depot Core Design (Increment 1)

Goal

Give Crucible real ownership of the content-addressed asset depot. Today crucible-depot is a registered-but-unwired stub (fails closed, exit 70), and the actual depot logic lives as ~2000 lines of bash embedded in sow-assets-manifest/scripts — a direct violation of the Crucible principle that artifact repos invoke Crucible through wrappers and never embed a toolkit.

Increment 1 wires the depot core — blob backends plus status / push / verify / get — into crucible depot, and cuts the manifest repo over to call it directly. The bash depot layer is retired, not wrapped: one tool owns depot byte-motion for both local workflows and CI/CD, with complete parity.

Why now (root cause that triggered this)

Release run 222 (v0.1.5) failed in 45s at the "Warm depot cache" step. PR #27's mdl-name normalization recompiled the corpus, producing 5903 net-new sha256s in assets/*.yml. Sampling proved 15/15 net-new blobs return HTTP 404 on the CDN while 10/10 old blobs return 200 — the CDN works; the new content was never uploaded. The blobs exist in the local depot (workspace/depot, 69,473 blobs) but never reached Bunny. mirror.sh then 404s and dies — the release is correctly failing closed on a depot missing data.

Root cause of the missing upload: import.sh's stat-cache (.cache/import/<cat>.tsv) records (path,size,mtime,sha) after any import and excludes "unchanged" files from the only upload call (depot_put_many) — without tracking which backend those bytes actually landed in. A prior local import poisons the cache so a later bunny import silently skips the upload. Combined with depot_require_write only normalizing cdn/unset→bunny (never local→anything), it is very easy to leave the manifests referencing blobs that exist only in the local depot.

The design principle that kills this bug class: presence is always checked against the real target backend, never a cache.

Update 2026-07-04 — interim bash fix landed + corrected drift measurement

The bug above was fixed in bash ahead of this rewrite (this spec's Increment 2 retires the stat-cache concept entirely, making the fix structural): sow-assets-manifest#29 keys the stat-cache dir by write backend + target (.cache/import/<target>/, scripts/import.sh), so a local import (make haks) can no longer mark a blob "done" in a cache the next bunny sync trusts. A cross-backend regression test guards it. When Increment 1/2 delete the bash depot layer, this becomes moot — but until then, do not "simplify" the stat-cache back to a single shared dir; that reintroduces the exact orphaning that broke run 222.

Corrected drift numbers. The "15/15 sampled net-new blobs 404" reading in the section above is a sampling artifact, not the real drift. The CDN edge (cdn-a7f3k9.westgate.pw) is IPv6-reachable but resets HTTP/2 from some hosts, and it throttles concurrent HEADs with 428/000 — which produce both false-404s and false-200s. A reliable sweep (force curl -4, 1-byte range GET -r 0-0 → expect 206, low concurrency, then serial re-confirm any non-2xx) shows the true missing set was exactly 2 net-new blobs, not ~5903:

• item/_shared_aenea_weaponvfx/wblpl_fxshblk.mdl (0b1d3f…)
• item/_shared_aenea_weaponvfx/wxdbsc_fxshpur.mdl (4e8d4f…)

A 600-blob random sample of the pre-existing corpus was 100% present. mirror.sh still correctly failed the release — it dies on the first missing blob — but the gap was two files a local make haks had primed in the shared cache, not a wholesale upload failure. Both blobs were re-uploaded to Bunny from workspace/depot (present in the local depot, sha-verified), unblocking the v0.1.5 re-run.

Takeaway for crucible depot status/verify: probe over IPv4, treat a single concurrent-probe non-2xx as unconfirmed (re-check serially) before reporting a blob missing, so the tool doesn't over-report drift the way an ad-hoc HEAD sweep did here.

Field notes 2026-07-04 (content-side session) — issues the migration must fix

Surfaced while verifying a content edit (over/ravenloft_potm_blood, 18 blobs) had actually reached the CDN. All corroborate the design above; #1 is a new gap to close in Increment 1.

1. push must be non-interactive — no tty prompt (new gap). Today the only write path, depot_require_write (lib.sh), prompts for BUNNY_STORAGE_PASSWORD on a tty when it is unset. Headless agents and CI-without-a-pre-exported-secret cannot answer that prompt, so there is no way to push except with a human at a terminal — which is exactly why the current operational workaround is "an agent/human manually pushes each fixed asset." crucible depot push must take the write key from env only, fail closed with a clear message when it is absent, and never prompt. The Backends section lists the env names but does not state the no-prompt requirement — make it explicit and add a test that a missing write key exits non-zero without reading stdin.

2. status --target cdn is the everyday author check, not only the release gate. The spec frames status as the release pre-flight (correct), but the same command, credential-free against cdn, is the "did my edit actually reach the CDN?" check content authors need routinely. This session answered that question by hand-probing 18 sha256s with curl — precisely the toil crucible depot status --manifests assets --target cdn should replace. Document it as the canonical author-facing "is my content live?" command so people stop ad-hoc probing.

3. Probe reliability confirmed again — bake it into the tests. Reproduced the HEAD hazard directly: a curl -sI HEAD sweep of the 18 blobs returned all-200; the prescribed reliable method (curl -4, 1-byte range GET -r 0-0 → 206, serial) also returned all-206 with the fake-sha control at 404. HEAD did not false-negative this time, but it stays the wrong tool. Make the IPv4 range-GET probe a hard requirement and add a test that fails if a HEAD-based existence check is introduced.

4. Drift is recurring, not the single v0.1.5 incident. Operationally, local→CDN sync is treated as "basically broken": every make haks / local import can re-orphan blobs from Bunny, so authors push by hand after edits. That raises Increment 1's priority and argues for making crucible depot push a standard post-edit step (or an on-edit CI hook), not only a release-time gate.

Ownership principle (from the maintainer)

• Crucible owns the depot tools. No wrapper scripts preserved for their own sake.
• Every consumer repo already has crucible in its flake; that is the only dependency. Consumers call crucible depot … directly (Makefile targets and CI workflow steps alike).
• Complete parity: the same crucible depot commands serve local workflows and CI/CD. No divergent code paths, no "CI-only" or "local-only" logic.
• Total clean-out over legacy preservation: bash depot code is deleted as its Go replacement lands, not kept as a fallback.

Scope

This spec covers Increment 1 only. Roadmap for context:

• Increment 1 (this spec): crucible depot {status, push, verify, get} + blob backends (local / cdn / bunny). Cut over mirror/verify/release gate. Delete the corresponding lib.sh depot layer.
• Increment 2: crucible depot import / sync (edit-tree → manifest+depot), replacing import.sh / sync-assets.sh. Retire the stat-cache concept.
• Increment 3: mdl integrity checks, mirror/pull ergonomics, prune, gc, export. Delete remaining bash.

Each increment ships independently and deletes the bash it replaces.

Command surface (Increment 1)

crucible depot status  [--manifests DIR] [--source LOCAL_DEPOT] --target bunny|cdn|local
crucible depot push    [--manifests DIR] --source LOCAL_DEPOT   --target bunny
crucible depot verify  [--manifests DIR] --target bunny|cdn     [--sample N]
crucible depot get     <sha> <dest>       --target cdn|bunny|local

• --manifests defaults to assets/ (the repo's manifest dir). Confirmed decision: crucible depot parses assets/*.yml directly (same coupling model as crucible-hak reading hak manifests); the caller does not pre-extract sha lists.
• --source is the local content-addressed store used as the byte source for push (e.g. workspace/depot).
• Config/auth is env-first with flags as local aliases (see Backends).

Drift model (the fix)

The manifest is the source of truth for what must exist; the target backend is the source of truth for what does exist; push reconciles them. There is no cache of "what I uploaded."

• status — collect every sha256 referenced by --manifests/*.yml, run one parallel existence sweep against --target, report {referenced, present, missing}. Non-zero exit if any referenced blob is missing. This is the release pre-flight gate.
• push — run status, then for each missing sha read the bytes from --source and upload to --target. Stateless and idempotent; re-running does nothing once clean. Cannot silently skip an upload — "should I upload this?" is answered by probing the target, not by trusting a local record.
• verify — existence sweep plus a bounded random sample downloaded and re-hashed (the current --full tag-build check), decoupled from packing.
• get — single-blob fetch with sha re-verify (backfill / debugging).

Backends & config

Faithful port of lib.sh semantics — this is a re-home, not a redesign of the wire protocol.

Backend	Read	Write	Role
local	filesystem sha256/ab/cd/<sha>	filesystem	dev/CI; byte source for push
cdn	CDN GET + sha re-verify	✗ read-only, fails closed	credential-free public reads
bunny	CDN GET, storage fallback	Storage PUT	the only writable remote

Layout: blob_key(sha) = sha256/<sha[0:2]>/<sha[2:4]>/<sha>.

Config (env names unchanged so existing CI secrets and release.yml keep working untouched; flags are local aliases):

• DEPOT_CDN_BASE / BUNNY_CDN_BASE — CDN read base.
• BUNNY_STORAGE_HOST, BUNNY_STORAGE_ZONE — Bunny storage endpoint.
• Read/write key split (preserved): BUNNY_STORAGE_READ_PASSWORD (falls back to BUNNY_STORAGE_PASSWORD) for probes/reads; BUNNY_STORAGE_PASSWORD for PUT.
• Concurrency: DEPOT_PROBE_JOBS (read probes), DEPOT_JOBS (uploads); DEPOT_CONNECT_TIMEOUT, DEPOT_PROBE_MAX_TIME per-transfer bounds.

Invariants carried over exactly:

• Existence probe is a 1-byte range request (no body download).
• Upload is PUT with Checksum: <UPPER-sha> so Bunny rejects corrupt writes server-side; each sha uploaded at most once per run.
• Fail-safe: a blob with no confirmed-present reply is treated as absent and (re)uploaded — never silently skipped. This is the invariant the stat-cache violated; here it is structural, not cached.
• get re-hashes every downloaded blob and deletes on mismatch.
• cdn write operations fail closed (never fake a write).

Stale-wording note to reconcile in code/docs: the dispatch registry summarizes depot as "(SeaweedFS)", but the real depot is Bunny-CDN + local (sow-assets-manifest/AGENTS.md: "No S3, no SeaweedFS"). The Go implementation and the registry summary adopt the Bunny/local/cdn reality.

Data flow

assets/*.yml ──parse──▶ referenced sha set ─┐
                                            ├─▶ target.HasMany() ─▶ missing set
                        --target (bunny) ───┘                          │
                                                                       ▼
                     --source (workspace/depot) ──read bytes──▶ target.PutMany()

Cutover / clean-out

Increment 1 deletes the bash it replaces (no wrappers left behind):

• scripts/mirror.sh → crucible depot get/pull path; script removed.
• build-haks.sh's --full depot-verify block → crucible depot verify.
• release.yml step "Warm depot cache" → crucible depot status (gate) then the fetch path; the inline bash ticker/compgen logic goes away.
• lib.sh depot layer (depot_require, depot_require_write, _depot_*_has, _depot_*_get, _depot_*_put*, _depot_*_probe*, blob_key, backend dispatch — ~250+ lines) is removed once no remaining bash references it. Anything still needed by Increments 2–3 is deleted when those land, not kept as a compatibility shim.

Consumers call crucible depot … directly. Nothing new is wrapped.

Exit-code contract

Matches Crucible's fail-closed convention:

• 0 — clean / success.
• distinct non-zero (proposed 1) — drift found (status/push saw referenced-but-absent blobs), so release.yml can gate on status before packing.
• 70 — internal/backend error; never a faked result.

Testing

• Go unit tests (internal/depot): local backend round-trip against a temp dir; manifest parsing; drift computation (referenced − present = missing) as table tests; backend selection + config resolution (incl. read/write key fallback).
• httptest fake-Bunny: assert the range-probe-then-PUT sequence, the Checksum: <UPPER-sha> header, and the read/write key split — no real Bunny.
• Integration (local→local): status shows drift → push clears it → status clean → push again is a no-op (idempotency).

Non-goals (Increment 1)

• import / sync / edit-tree ingestion (Increment 2).
• mdl integrity checks, prune, gc, export, mirror ergonomics (Increment 3).
• Any change to the depot wire protocol, blob layout, or Bunny account config.
• A TUI.

Immediate operational payoff

crucible depot push --source workspace/depot --target bunny is exactly the command that unblocks the stuck v0.1.5 release: it probes the target and uploads whatever is actually 404 while it re-hydrates from the local depot. (The 5903 figure below is superseded — see the 2026-07-04 update above: the real drift was 2 blobs, already re-uploaded; push would have been the clean way to do it and is idempotent once the depot is whole.) Building Increment 1 first both fixes the tooling and clears the current outage.

Risks / open questions

• BUNNY_STORAGE_HOST default: confirm the canonical storage host to bake as the fallback (bash referenced it without a visible default in the reviewed range).
• Manifest schema coupling: crucible depot now depends on the assets/*.yml shape (assets[].{path,sha256,size,restype,hak}). Acceptable and intentional, but a schema change now touches Crucible.
• Cutover ordering: release.yml and Makefile must switch to crucible depot in the same change that removes the bash, to avoid a window where both exist.