(docs): crucible depot redesign

## Summary
- skip overwriting existing `.are.json` files when only the root GFF `Version` differs
- keep real extraction changes writing normally
- add regression coverage for version-only area extraction

## Test Plan
- `nix develop --command make check`
- sample extraction in a clean temp `sow-module` clone using the patched `crucible` binary

Reviewed-on: #28
Reviewed-by: xtul <mpiasecki720@protonmail.com>
Co-authored-by: vickydotbat <vickydotbat@tutamail.com>
Co-committed-by: vickydotbat <vickydotbat@tutamail.com>

.gitea/workflows/build-binaries.yml

@@ -9,6 +9,11 @@ on:
   push:
     branches: [main]
     tags: ['v*']
+    paths-ignore:
+      - "docs/**"
+      - "README.md"
+      - "AGENTS.md"
+      - "LICENSE"
 
 jobs:
   build-binaries:

.gitea/workflows/test.yml

@@ -4,6 +4,11 @@ name: test
 on:
   push:
     branches: [main]
+    paths-ignore:
+      - "docs/**"
+      - "README.md"
+      - "AGENTS.md"
+      - "LICENSE"
   pull_request:
 
 jobs:

docs/superpowers/specs/2026-07-04-crucible-depot-core-design.md

@@ -0,0 +1,285 @@
+# Crucible Depot Core Design (Increment 1)
+
+## Goal
+
+Give Crucible real ownership of the content-addressed asset depot. Today
+`crucible-depot` is a registered-but-unwired stub (fails closed, exit `70`), and
+the actual depot logic lives as ~2000 lines of bash embedded in
+`sow-assets-manifest/scripts` — a direct violation of the Crucible principle that
+*artifact repos invoke Crucible through wrappers and never embed a toolkit*.
+
+Increment 1 wires the depot **core** — blob backends plus `status` / `push` /
+`verify` / `get` — into `crucible depot`, and cuts the manifest repo over to call
+it directly. The bash depot layer is **retired, not wrapped**: one tool owns
+depot byte-motion for both local workflows and CI/CD, with complete parity.
+
+## Why now (root cause that triggered this)
+
+Release run 222 (`v0.1.5`) failed in 45s at the "Warm depot cache" step. PR #27's
+mdl-name normalization recompiled the corpus, producing **5903 net-new sha256s**
+in `assets/*.yml`. Sampling proved **15/15 net-new blobs return HTTP 404 on the
+CDN** while **10/10 old blobs return 200** — the CDN works; the new content was
+never uploaded. The blobs exist in the local depot (`workspace/depot`, 69,473
+blobs) but never reached Bunny. `mirror.sh` then 404s and dies — the release is
+correctly failing closed on a depot missing data.
+
+Root cause of the *missing upload*: `import.sh`'s stat-cache
+(`.cache/import/<cat>.tsv`) records `(path,size,mtime,sha)` after **any** import
+and excludes "unchanged" files from the only upload call (`depot_put_many`) —
+**without tracking which backend those bytes actually landed in**. A prior
+`local` import poisons the cache so a later `bunny` import silently skips the
+upload. Combined with `depot_require_write` only normalizing `cdn`/unset→`bunny`
+(never `local`→anything), it is very easy to leave the manifests referencing
+blobs that exist only in the local depot.
+
+The design principle that kills this bug class: **presence is always checked
+against the real target backend, never a cache.**
+
+### Update 2026-07-04 — interim bash fix landed + corrected drift measurement
+
+The bug above was fixed in bash ahead of this rewrite (this spec's Increment 2
+retires the stat-cache concept entirely, making the fix structural):
+[`sow-assets-manifest#29`](https://git.westgate.pw/ShadowsOverWestgate/sow-assets-manifest/pulls/29)
+keys the stat-cache dir by write backend + target
+(`.cache/import/<target>/`, `scripts/import.sh`), so a `local` import
+(`make haks`) can no longer mark a blob "done" in a cache the next `bunny` sync
+trusts. A cross-backend regression test guards it. When Increment 1/2 delete the
+bash depot layer, this becomes moot — but until then, **do not "simplify" the
+stat-cache back to a single shared dir**; that reintroduces the exact orphaning
+that broke run 222.
+
+**Corrected drift numbers.** The "15/15 sampled net-new blobs 404" reading in the
+section above is a *sampling artifact*, not the real drift. The CDN edge
+(`cdn-a7f3k9.westgate.pw`) is IPv6-reachable but resets HTTP/2 from some hosts,
+and it throttles concurrent HEADs with `428`/`000` — which produce **both**
+false-404s and false-200s. A reliable sweep (force `curl -4`, 1-byte range GET
+`-r 0-0` → expect `206`, low concurrency, then serial re-confirm any non-2xx)
+shows the true missing set was **exactly 2 net-new blobs**, not ~5903:
+
+- `item/_shared_aenea_weaponvfx/wblpl_fxshblk.mdl` (`0b1d3f…`)
+- `item/_shared_aenea_weaponvfx/wxdbsc_fxshpur.mdl` (`4e8d4f…`)
+
+A 600-blob random sample of the pre-existing corpus was 100% present. `mirror.sh`
+still correctly failed the release — it dies on the first missing blob — but the
+gap was two files a local `make haks` had primed in the shared cache, not a
+wholesale upload failure. Both blobs were re-uploaded to Bunny from
+`workspace/depot` (present in the local depot, sha-verified), unblocking the
+`v0.1.5` re-run.
+
+Takeaway for `crucible depot status`/`verify`: probe over IPv4, treat a single
+concurrent-probe non-2xx as *unconfirmed* (re-check serially) before reporting a
+blob missing, so the tool doesn't over-report drift the way an ad-hoc HEAD sweep
+did here.
+
+### Field notes 2026-07-04 (content-side session) — issues the migration must fix
+
+Surfaced while verifying a content edit (`over/ravenloft_potm_blood`, 18 blobs) had
+actually reached the CDN. All corroborate the design above; #1 is a new gap to close in
+Increment 1.
+
+1. **`push` must be non-interactive — no tty prompt (new gap).** Today the only write
+   path, `depot_require_write` (`lib.sh`), *prompts* for `BUNNY_STORAGE_PASSWORD` on a tty
+   when it is unset. Headless agents and CI-without-a-pre-exported-secret cannot answer
+   that prompt, so there is no way to push except with a human at a terminal — which is
+   exactly why the current operational workaround is "an agent/human manually pushes each
+   fixed asset." `crucible depot push` must take the write key **from env only**, fail
+   closed with a clear message when it is absent, and **never prompt**. The Backends
+   section lists the env names but does not state the no-prompt requirement — make it
+   explicit and add a test that a missing write key exits non-zero without reading stdin.
+
+2. **`status --target cdn` is the everyday author check, not only the release gate.** The
+   spec frames `status` as the release pre-flight (correct), but the same command,
+   credential-free against `cdn`, is the "did my edit actually reach the CDN?" check
+   content authors need routinely. This session answered that question by hand-probing 18
+   sha256s with `curl` — precisely the toil `crucible depot status --manifests assets
+   --target cdn` should replace. Document it as the canonical author-facing "is my content
+   live?" command so people stop ad-hoc probing.
+
+3. **Probe reliability confirmed again — bake it into the tests.** Reproduced the HEAD
+   hazard directly: a `curl -sI` HEAD sweep of the 18 blobs returned all-`200`; the
+   prescribed reliable method (`curl -4`, 1-byte range GET `-r 0-0` → `206`, serial) also
+   returned all-`206` with the fake-sha control at `404`. HEAD did not false-negative this
+   time, but it stays the wrong tool. Make the IPv4 range-GET probe a hard requirement and
+   add a test that fails if a HEAD-based existence check is introduced.
+
+4. **Drift is recurring, not the single v0.1.5 incident.** Operationally, local→CDN sync is
+   treated as "basically broken": every `make haks` / local import can re-orphan blobs from
+   Bunny, so authors push by hand after edits. That raises Increment 1's priority and argues
+   for making `crucible depot push` a standard post-edit step (or an on-edit CI hook), not
+   only a release-time gate.
+
+## Ownership principle (from the maintainer)
+
+- Crucible **owns** the depot tools. No wrapper scripts preserved for their own
+  sake.
+- Every consumer repo already has `crucible` in its flake; that is the only
+  dependency. Consumers call `crucible depot …` directly (Makefile targets and
+  CI workflow steps alike).
+- **Complete parity**: the same `crucible depot` commands serve local workflows
+  and CI/CD. No divergent code paths, no "CI-only" or "local-only" logic.
+- Total clean-out over legacy preservation: bash depot code is **deleted** as its
+  Go replacement lands, not kept as a fallback.
+
+## Scope
+
+This spec covers **Increment 1** only. Roadmap for context:
+
+- **Increment 1 (this spec):** `crucible depot {status, push, verify, get}` +
+  blob backends (`local` / `cdn` / `bunny`). Cut over `mirror`/verify/release
+  gate. Delete the corresponding `lib.sh` depot layer.
+- **Increment 2:** `crucible depot import` / `sync` (edit-tree → manifest+depot),
+  replacing `import.sh` / `sync-assets.sh`. Retire the stat-cache concept.
+- **Increment 3:** mdl integrity checks, `mirror`/`pull` ergonomics, `prune`,
+  `gc`, `export`. Delete remaining bash.
+
+Each increment ships independently and deletes the bash it replaces.
+
+## Command surface (Increment 1)
+
+```
+crucible depot status  [--manifests DIR] [--source LOCAL_DEPOT] --target bunny|cdn|local
+crucible depot push    [--manifests DIR] --source LOCAL_DEPOT   --target bunny
+crucible depot verify  [--manifests DIR] --target bunny|cdn     [--sample N]
+crucible depot get     <sha> <dest>       --target cdn|bunny|local
+```
+
+- `--manifests` defaults to `assets/` (the repo's manifest dir). Confirmed
+  decision: `crucible depot` parses `assets/*.yml` directly (same coupling model
+  as `crucible-hak` reading hak manifests); the caller does not pre-extract sha
+  lists.
+- `--source` is the local content-addressed store used as the byte source for
+  `push` (e.g. `workspace/depot`).
+- Config/auth is env-first with flags as local aliases (see Backends).
+
+## Drift model (the fix)
+
+The manifest is the source of truth for **what** must exist; the target backend
+is the source of truth for **what does** exist; `push` reconciles them. There is
+no cache of "what I uploaded."
+
+- **`status`** — collect every `sha256` referenced by `--manifests/*.yml`, run
+  one parallel existence sweep against `--target`, report
+  `{referenced, present, missing}`. Non-zero exit if any referenced blob is
+  missing. This is the release pre-flight gate.
+- **`push`** — run `status`, then for each missing sha read the bytes from
+  `--source` and upload to `--target`. Stateless and idempotent; re-running does
+  nothing once clean. **Cannot silently skip an upload** — "should I upload this?"
+  is answered by probing the target, not by trusting a local record.
+- **`verify`** — existence sweep plus a bounded random sample downloaded and
+  re-hashed (the current `--full` tag-build check), decoupled from packing.
+- **`get`** — single-blob fetch with sha re-verify (backfill / debugging).
+
+## Backends & config
+
+Faithful port of `lib.sh` semantics — this is a re-home, not a redesign of the
+wire protocol.
+
+| Backend | Read | Write | Role |
+|---|---|---|---|
+| `local` | filesystem `sha256/ab/cd/<sha>` | filesystem | dev/CI; byte source for `push` |
+| `cdn`   | CDN GET + sha re-verify | ✗ read-only, fails closed | credential-free public reads |
+| `bunny` | CDN GET, storage fallback | Storage `PUT` | the only writable remote |
+
+Layout: `blob_key(sha) = sha256/<sha[0:2]>/<sha[2:4]>/<sha>`.
+
+Config (env names unchanged so existing CI secrets and `release.yml` keep working
+untouched; flags are local aliases):
+
+- `DEPOT_CDN_BASE` / `BUNNY_CDN_BASE` — CDN read base.
+- `BUNNY_STORAGE_HOST`, `BUNNY_STORAGE_ZONE` — Bunny storage endpoint.
+- **Read/write key split (preserved):** `BUNNY_STORAGE_READ_PASSWORD` (falls back
+  to `BUNNY_STORAGE_PASSWORD`) for probes/reads; `BUNNY_STORAGE_PASSWORD` for
+  `PUT`.
+- Concurrency: `DEPOT_PROBE_JOBS` (read probes), `DEPOT_JOBS` (uploads);
+  `DEPOT_CONNECT_TIMEOUT`, `DEPOT_PROBE_MAX_TIME` per-transfer bounds.
+
+Invariants carried over exactly:
+
+- **Existence probe** is a 1-byte range request (no body download).
+- **Upload** is `PUT` with `Checksum: <UPPER-sha>` so Bunny rejects corrupt
+  writes server-side; each sha uploaded at most once per run.
+- **Fail-safe:** a blob with no confirmed-present reply is treated as absent and
+  (re)uploaded — never silently skipped. This is the invariant the stat-cache
+  violated; here it is structural, not cached.
+- **`get`** re-hashes every downloaded blob and deletes on mismatch.
+- `cdn` write operations fail closed (never fake a write).
+
+Stale-wording note to reconcile in code/docs: the dispatch registry summarizes
+depot as *"(SeaweedFS)"*, but the real depot is Bunny-CDN + local
+(`sow-assets-manifest/AGENTS.md`: "No S3, no SeaweedFS"). The Go implementation
+and the registry summary adopt the Bunny/local/cdn reality.
+
+## Data flow
+
+```
+assets/*.yml ──parse──▶ referenced sha set ─┐
+                                            ├─▶ target.HasMany() ─▶ missing set
+                        --target (bunny) ───┘                          │
+                                                                       ▼
+                     --source (workspace/depot) ──read bytes──▶ target.PutMany()
+```
+
+## Cutover / clean-out
+
+Increment 1 deletes the bash it replaces (no wrappers left behind):
+
+- `scripts/mirror.sh` → `crucible depot get`/pull path; script removed.
+- `build-haks.sh`'s `--full` depot-verify block → `crucible depot verify`.
+- `release.yml` step "Warm depot cache" → `crucible depot status` (gate) then the
+  fetch path; the inline bash ticker/compgen logic goes away.
+- `lib.sh` depot layer (`depot_require`, `depot_require_write`, `_depot_*_has`,
+  `_depot_*_get`, `_depot_*_put*`, `_depot_*_probe*`, `blob_key`, backend
+  dispatch — ~250+ lines) is removed once no remaining bash references it.
+  Anything still needed by Increments 2–3 is deleted when those land, not kept as
+  a compatibility shim.
+
+Consumers call `crucible depot …` directly. Nothing new is wrapped.
+
+## Exit-code contract
+
+Matches Crucible's fail-closed convention:
+
+- `0` — clean / success.
+- distinct non-zero (proposed `1`) — **drift found** (`status`/`push` saw
+  referenced-but-absent blobs), so `release.yml` can gate on `status` before
+  packing.
+- `70` — internal/backend error; never a faked result.
+
+## Testing
+
+- **Go unit tests** (`internal/depot`): `local` backend round-trip against a temp
+  dir; manifest parsing; drift computation (referenced − present = missing) as
+  table tests; backend selection + config resolution (incl. read/write key
+  fallback).
+- **httptest fake-Bunny**: assert the range-probe-then-PUT sequence, the
+  `Checksum: <UPPER-sha>` header, and the read/write key split — no real Bunny.
+- **Integration (local→local)**: `status` shows drift → `push` clears it →
+  `status` clean → `push` again is a no-op (idempotency).
+
+## Non-goals (Increment 1)
+
+- `import` / `sync` / edit-tree ingestion (Increment 2).
+- mdl integrity checks, `prune`, `gc`, `export`, `mirror` ergonomics
+  (Increment 3).
+- Any change to the depot wire protocol, blob layout, or Bunny account config.
+- A TUI.
+
+## Immediate operational payoff
+
+`crucible depot push --source workspace/depot --target bunny` is exactly the
+command that unblocks the stuck `v0.1.5` release: it probes the target and
+uploads whatever is actually 404 while it re-hydrates from the local depot.
+(The `5903` figure below is superseded — see the 2026-07-04 update above: the
+real drift was **2** blobs, already re-uploaded; `push` would have been the
+clean way to do it and is idempotent once the depot is whole.) Building
+Increment 1 first both fixes the tooling and clears the current outage.
+
+## Risks / open questions
+
+- `BUNNY_STORAGE_HOST` default: confirm the canonical storage host to bake as the
+  fallback (bash referenced it without a visible default in the reviewed range).
+- Manifest schema coupling: `crucible depot` now depends on the `assets/*.yml`
+  shape (`assets[].{path,sha256,size,restype,hak}`). Acceptable and intentional,
+  but a schema change now touches Crucible.
+- Cutover ordering: `release.yml` and Makefile must switch to `crucible depot` in
+  the same change that removes the bash, to avoid a window where both exist.

internal/pipeline/extract.go

@@ -501,6 +501,9 @@ func writeManagedFile(path string, data []byte) (writeState, error) {
 		if bytes.Equal(existing, data) {
 			return writeSkipped, nil
 		}
+		if areaGFFJSONEqualIgnoringRootVersion(path, existing, data) {
+			return writeSkipped, nil
+		}
 		if err := os.WriteFile(path, data, 0o644); err != nil {
 			return writeSkipped, fmt.Errorf("overwrite %s: %w", path, err)
 		}
@@ -516,6 +519,38 @@ func writeManagedFile(path string, data []byte) (writeState, error) {
 	return writeNew, nil
 }
 
+func areaGFFJSONEqualIgnoringRootVersion(path string, existing, extracted []byte) bool {
+	if !strings.HasSuffix(strings.ToLower(filepath.Base(path)), ".are.json") {
+		return false
+	}
+
+	var left, right gff.Document
+	if err := json.Unmarshal(existing, &left); err != nil {
+		return false
+	}
+	if err := json.Unmarshal(extracted, &right); err != nil {
+		return false
+	}
+	removeRootField(&left.Root, "Version")
+	removeRootField(&right.Root, "Version")
+
+	leftRaw, err := json.Marshal(left)
+	if err != nil {
+		return false
+	}
+	rightRaw, err := json.Marshal(right)
+	if err != nil {
+		return false
+	}
+	return bytes.Equal(leftRaw, rightRaw)
+}
+
+func removeRootField(s *gff.Struct, label string) {
+	s.Fields = slices.DeleteFunc(s.Fields, func(field gff.Field) bool {
+		return field.Label == label
+	})
+}
+
 func cleanupStaleFiles(p *project.Project, desired map[string]struct{}, scope extractionScope) (int, []error) {
 	candidates := make([]string, 0, len(p.Inventory.SourceFiles)+len(p.Inventory.ScriptFiles)+len(p.Inventory.AssetFiles))
 	if scope.Source && safeCleanupRoot(p.SourceDir(), p.Root) {

internal/pipeline/pipeline_test.go

@@ -134,6 +134,113 @@ func TestBuildThenExtract(t *testing.T) {
 	}
 }
 
+func TestExtractSkipsAreaWhenOnlyRootVersionChanges(t *testing.T) {
+	root := t.TempDir()
+	mustMkdir(t, filepath.Join(root, "src", "module"))
+	mustMkdir(t, filepath.Join(root, "src", "areas"))
+	mustMkdir(t, filepath.Join(root, "assets"))
+	mustMkdir(t, filepath.Join(root, "build"))
+
+	mustWriteFile(t, filepath.Join(root, "nwn-tool.json"), `{
+  "module": {
+    "name": "Test Module",
+    "resref": "testmod"
+  },
+  "paths": {
+    "source": "src",
+    "assets": "assets",
+    "build": "build"
+  }
+}
+`)
+	mustWriteFile(t, filepath.Join(root, "src", "module", "module.ifo.json"), `{
+  "file_type": "IFO ",
+  "file_version": "V3.2",
+  "root": {
+    "struct_type": 0,
+    "fields": [
+      {
+        "label": "Mod_Name",
+        "type": "CExoString",
+        "value": "Test Module"
+      }
+    ]
+  }
+}
+`)
+	mustWriteFile(t, filepath.Join(root, "src", "areas", "area001.are.json"), `{
+  "file_type": "ARE ",
+  "file_version": "V3.2",
+  "root": {
+    "struct_type": 0,
+    "fields": [
+      {
+        "label": "Version",
+        "type": "DWord",
+        "value": 2
+      },
+      {
+        "label": "Tag",
+        "type": "CExoString",
+        "value": "area001"
+      }
+    ]
+  }
+}
+`)
+
+	p, err := project.Load(root)
+	if err != nil {
+		t.Fatalf("load project: %v", err)
+	}
+	if err := p.ValidateLayout(); err != nil {
+		t.Fatalf("validate layout: %v", err)
+	}
+	if err := p.Scan(); err != nil {
+		t.Fatalf("scan: %v", err)
+	}
+	if _, err := BuildModule(p); err != nil {
+		t.Fatalf("build module: %v", err)
+	}
+
+	mustWriteFile(t, filepath.Join(root, "src", "areas", "area001.are.json"), `{
+  "file_type": "ARE ",
+  "file_version": "V3.2",
+  "root": {
+    "struct_type": 0,
+    "fields": [
+      {
+        "label": "Version",
+        "type": "DWord",
+        "value": 1
+      },
+      {
+        "label": "Tag",
+        "type": "CExoString",
+        "value": "area001"
+      }
+    ]
+  }
+}
+`)
+	if err := p.Scan(); err != nil {
+		t.Fatalf("rescan: %v", err)
+	}
+
+	result, err := Extract(p)
+	if err != nil {
+		t.Fatalf("extract: %v", err)
+	}
+	if result.Overwritten != 0 {
+		t.Fatalf("Version-only area extraction must not overwrite, got %d overwritten", result.Overwritten)
+	}
+
+	document := readGFFJSON(t, filepath.Join(root, "src", "areas", "area001.are.json"))
+	if got, want := fieldValue(t, document.Root, "Version"), gff.DWordValue(1); got != want {
+		t.Fatalf("expected existing Version %#v to remain, got %#v", want, got)
+	}
+}
+
 func TestExtractReadsHAKAssets(t *testing.T) {
 	root := t.TempDir()
 	mustMkdir(t, filepath.Join(root, "src"))