diff --git a/docs/superpowers/specs/2026-07-12-crucible-assets-builder-design.md b/docs/superpowers/specs/2026-07-12-crucible-assets-builder-design.md index 806be48..831f754 100644 --- a/docs/superpowers/specs/2026-07-12-crucible-assets-builder-design.md +++ b/docs/superpowers/specs/2026-07-12-crucible-assets-builder-design.md @@ -30,8 +30,8 @@ Seven commands: model-name mismatches (read-only). - `crucible assets fix-mdl` — lowercase `.mdl` filenames and rewrite ASCII model/root names to match each file's stem. -- `crucible assets check-dupes` — report runtime-name (basename) collisions in - directories, or hak-scoped collisions across `assets/*.yml` manifests. +- `crucible assets check-dupes` — report runtime-name (basename) collisions + across directories. - `crucible assets clean-dupes` — delete files from a "clean" tree whose basename collides with anything in a "primary" tree. @@ -235,21 +235,21 @@ candidate only if it needs a content fix (name mismatch) or a filename lowercase, so the expensive per-file read/rewrite runs only on the few that matter, not the whole tree. -## `crucible assets check-dupes [--manifests] …` +## `crucible assets check-dupes …` -Port of `check-duplicate-names.sh`. Read-only. NWN packs hak entries by -basename, not path, so files at different paths with the same basename -(case-insensitively) silently clobber on pack. This flags them. - -- Directory mode (default): across all `` args, report any two files whose - lowercased basename matches. -- `--manifests`: additionally read every `assets/*.yml` in the current repo and - report basename collisions **within the same hak** (`.assets[].path` grouped - by `.assets[].hak`), reusing Crucible's existing `gopkg.in/yaml.v3` parsing. - Requires no external `yq`. +Port of `check-duplicate-names.sh`'s **directory mode only**. Read-only. NWN +packs hak entries by basename, not path, so files at different paths with the +same basename (case-insensitively) silently clobber on pack. Across all `` +args, report any two files whose lowercased basename matches. Exit non-zero if any collision is found. +The script's `--manifests` mode (hak-scoped collisions read from `assets/*.yml`) +is **not** ported here — it inspects manifest structure, not files on disk, so +it belongs with the manifest/depot tooling. It moves in the follow-up spec (see +Out of scope); until then `sow-assets-manifest` keeps +`check-duplicate-names.sh` solely for that mode. + ## `crucible assets clean-dupes [--dry-run] ` Port of `clean-duplicate-names.sh`. Deletes files from the `` tree whose @@ -269,22 +269,24 @@ Exit zero on success with a count of removed (or would-remove) files. (`crucible.sh`) and uses `crucible depot` from its Makefile, so it resolves a released `crucible` with zero extra setup. -- **Delete** the four folded driver scripts: `scripts/check-ascii-mdl.sh`, - `scripts/fix-mdl-model-names.sh`, `scripts/check-duplicate-names.sh`, - `scripts/clean-duplicate-names.sh`. +- **Delete** the three fully-folded driver scripts: `scripts/check-ascii-mdl.sh`, + `scripts/fix-mdl-model-names.sh`, `scripts/clean-duplicate-names.sh`. - **Repoint** the Makefile targets to the wrapper: - `check-mdl` → `./crucible.sh assets check-mdl …` - `fix-mdl` → `./crucible.sh assets fix-mdl [--dry-run] …` - - `check-dupes` → `./crucible.sh assets check-dupes …` and - `./crucible.sh assets check-dupes --manifests` + - `check-dupes` — its directory scan → `./crucible.sh assets check-dupes …`; + the `check-duplicate-names.sh --manifests` line stays as-is. - `clean-dupes` → `./crucible.sh assets clean-dupes [--dry-run] ` - - the `haks` target's `check-duplicate-names.sh --manifests` guard → - `./crucible.sh assets check-dupes --manifests` -- **Keep for now:** `scripts/mdl-scan.awk` and `scripts/mdl-name-lib.sh`. They - are still sourced by `import.sh` for its batched pre-import gate (~28× faster - than per-file, on a hot path). They retire in the follow-up below. Because - both the awk classifier and the Go `mdl` module must agree, the port keeps - detection byte-for-byte identical (a shared fixture set checks this). +- **Keep for now:** + - `scripts/check-duplicate-names.sh` — kept **only** for its `--manifests` + mode, still called by the `check` and `haks` targets. Its directory mode is + superseded by `crucible assets check-dupes`. Ported and deleted in the + follow-up. + - `scripts/mdl-scan.awk` and `scripts/mdl-name-lib.sh` — still sourced by + `import.sh` for its batched pre-import gate (~28× faster than per-file, on a + hot path). They retire in the follow-up too. Because both the awk classifier + and the Go `mdl` module must agree, the port keeps detection byte-for-byte + identical (a shared fixture set checks this). Nothing in `sow-assets-manifest` is committed by this project's plan except the script deletions and Makefile edits; that repo's change lands as its own commit @@ -292,12 +294,15 @@ once the new `crucible` release with `assets` is available. ## Out of scope — follow-up spec -Moving the **depot-interacting pipeline** (`import.sh`, `sync-assets.sh`, -`export.sh`, and the inline mdl gate) into `crucible depot` is a separate -project with its own spec. When `import.sh` becomes `crucible depot import`, its -gate uses the in-process `internal/assets/mdl` module and -`scripts/mdl-scan.awk` + `scripts/mdl-name-lib.sh` are deleted. This spec -deliberately stops at the standalone integrity tools to keep one focused plan. +Moving the **depot-interacting pipeline** into `crucible depot` is a separate +project with its own spec +(`2026-07-12-crucible-depot-pipeline-migration-design.md`). It covers +`import.sh`, `sync-assets.sh`, `export.sh`, the inline mdl gate, and the +`check-duplicate-names.sh --manifests` manifest-collision check. When that lands: +its gate uses the in-process `internal/assets/mdl` module, and +`scripts/mdl-scan.awk`, `scripts/mdl-name-lib.sh`, and +`scripts/check-duplicate-names.sh` are all deleted. This spec deliberately stops +at the standalone, file-on-disk integrity tools to keep one focused plan. ## Error handling @@ -327,8 +332,7 @@ Go tests in `internal/assets`: a model that then passes `CheckNames`. A parity fixture set shared in intent with `mdl-scan.awk` so the Go port and the still-present awk agree. - `check-dupes`/`clean-dupes`: temp trees asserting basename-collision - detection (case-insensitive), the `--manifests` hak-scoped path against a - small `assets/*.yml` fixture, the overlap guard, and `--dry-run` mutating + detection (case-insensitive), the overlap guard, and `--dry-run` mutating nothing. External engine/upscaler calls are exercised through the `run` indirection with diff --git a/docs/superpowers/specs/2026-07-12-crucible-depot-pipeline-migration-design.md b/docs/superpowers/specs/2026-07-12-crucible-depot-pipeline-migration-design.md new file mode 100644 index 0000000..fcbdeab --- /dev/null +++ b/docs/superpowers/specs/2026-07-12-crucible-depot-pipeline-migration-design.md @@ -0,0 +1,90 @@ +# Crucible `depot` pipeline migration — design (charter / draft) + +Date: 2026-07-12 +Status: **draft — needs its own brainstorming pass before implementation.** +Depends on: `2026-07-12-crucible-assets-builder-design.md` (the `assets` builder +and `internal/assets/mdl` module) landing first. + +This is a scoping charter, not an implementation-ready design. The scripts it +covers are nontrivial and lean heavily on `sow-assets-manifest/scripts/lib.sh` +(~35 KB). A full design requires studying `lib.sh` and the manifest format in +depth; that work happens in this spec's own brainstorming before any code. + +## Why this exists + +`sow-assets-manifest` is the last consumer still carrying real builder logic in +shell. The `assets` spec folded its standalone integrity tools into Crucible but +deliberately left the **depot-interacting pipeline** behind, because those +scripts read/write the content-addressed depot and the asset manifests — depot +domain, and a bigger lift. This charter completes the migration so +`sow-assets-manifest` becomes what the other consumers already are: a thin +caller of released `crucible` binaries with no vendored toolkit logic. + +## What moves into `crucible depot` + +The depot builder already owns `status/push/verify/get/pull`. This adds the +authoring side: + +| Script | Proposed command | What it does today | +| --- | --- | --- | +| `import.sh` (318 lines) | `crucible depot import [prefix…]` | Import an edit tree into the depot + manifests: sticky per-path member assignment, new paths to the tail member, per-category `max_bytes` rebalance (spill overflow forward, no full repack), a local stat-cache to skip unchanged files, batched hashing + parallel `depot_put_many`. | +| `sync-assets.sh` (61 lines) | `crucible depot sync [prefix…]` | Full reconcile = import + prune manifest entries whose path no longer exists (scoped by prefix when given). Delegates to `import.sh`. | +| `export.sh` (96 lines) | `crucible depot export [glob] [--hak m] [--restype ext] --to ` | Materialize a manifest-selected subset out of the depot into a clean edit tree (category paths preserved, no metadata). | +| `check-duplicate-names.sh --manifests` | `crucible depot check-dupes` (name TBD) | Hak-scoped basename collisions read from `assets/*.yml` (`.assets[].path` grouped by `.assets[].hak`). Manifest-structure check, not files on disk. | + +The inline **pre-import mdl gate** in `import.sh` (batched `mdl-scan.awk` pass +that aborts on an H/B model-name mismatch before any blob is hashed/uploaded) +is reimplemented with the in-process `internal/assets/mdl` module from the +`assets` spec — same H/B contract, no subprocess. + +## What retires when this lands + +Deleted from `sow-assets-manifest/scripts/` once the commands above are wired +and the Makefile is repointed: + +- `import.sh`, `sync-assets.sh`, `export.sh` +- `mdl-scan.awk`, `mdl-name-lib.sh` (only remaining consumer was `import.sh`) +- `check-duplicate-names.sh` (its directory mode already superseded by + `crucible assets check-dupes`; the `--manifests` mode moves here) +- whatever else in `lib.sh` becomes dead once the above are gone (assess during + design — `lib.sh` also serves `build-haks.sh`, `pack-haks.sh`, `promote.sh`, + etc., which are **not** in scope here, so `lib.sh` likely shrinks, not dies) + +Makefile targets `import`, `sync`, `haks` (its sync/import/dupe steps), `export` +repoint to `./crucible.sh depot …`. + +## Known hard parts (resolve in brainstorming) + +- **Manifest read/write parity.** Byte-stable output, member `max_bytes` + rebalance semantics, sticky path→member assignment. Must match the current + awk/`lib.sh` behavior or diff-review of manifests becomes noise. Crucible + already parses these manifests (`internal/pipeline`, hak build) — reuse, don't + reinvent. +- **Depot backends.** `import`/`export` drive bunny/cdn/local through + `lib.sh`'s `depot_put_many`/`depot_require_write` (incl. the + `BUNNY_STORAGE_PASSWORD` prompt and cdn→bunny write normalization). Crucible's + `internal/depot` already speaks these backends for pull/push — the authoring + side should share that client, not a second implementation. +- **Stat-cache.** `import.sh`'s `.cache/import//.tsv` skip-unchanged + optimization is the dominant re-import speedup. Decide whether Crucible + reproduces it, replaces it (content-addressing already dedupes uploads), or + drops it with a measured justification. +- **Throughput.** The shell versions are already batched to avoid per-file + forks; a Go port should be at least as fast (in-process, no forks) — verify, + don't assume. +- **Scoping semantics.** Prefix-scoped import/prune (segment-boundary match) and + export glob matching (`**`/`*`/literal dots, no shell expansion) must port + exactly. + +## Out of scope + +The HAK build/pack/promote/publish scripts (`build-haks.sh`, `pack-haks.sh`, +`promote.sh`, `publish-release*.sh`, `hak-artifact-record.sh`) and their +`lib.sh` support. `crucible hak build` already exists; whether these thin +release wrappers also migrate is a separate question, not this charter. + +## Next step + +Brainstorm this charter into a full design: read `lib.sh` and the manifest +format, settle the hard parts above, then write the implementation-ready spec +and plan. Do not implement from this document.