assets builder tools (#39)

Reviewed-on: #39
Reviewed-by: xtul <mpiasecki720@protonmail.com>

docs/superpowers/specs/2026-07-12-crucible-assets-builder-design.md

@@ -0,0 +1,342 @@
+# Crucible `assets` builder — design
+
+Date: 2026-07-12
+Status: approved, pre-implementation
+
+## Goal
+
+Fold the NWN:EE asset tools out of two standalone toolkits into Crucible as a
+new self-contained builder, `crucible assets`:
+
+- from the Python toolkit (`nwnee-asset-processing`) — model compile, texture
+  convert, texture upscale. Those scripts are references only; broken as-is and
+  carrying a heavy config/staging/report layer that we drop.
+- from `sow-assets-manifest/scripts/` — the model-name and duplicate-name
+  integrity tools (`check-ascii-mdl.sh`, `fix-mdl-model-names.sh`,
+  `check-duplicate-names.sh`, `clean-duplicate-names.sh`). These are ported into
+  Go and the shell drivers removed; `sow-assets-manifest` calls
+  `./crucible.sh assets …` through its bootstrap wrapper instead.
+
+Seven commands:
+
+- `crucible assets compile` — compile every ASCII `.mdl` in a directory to
+  binary, in place.
+- `crucible assets convert` — convert every texture in a directory to NWN:EE
+  DDS (or back to PNG/TGA), flipping vertically **every time** so a DDS is
+  always upside-down relative to its source.
+- `crucible assets upscale` — upscale every texture in a directory through
+  whatever upscaling backend is installed.
+- `crucible assets check-mdl` — report uncompiled ASCII `.mdl` files and
+  model-name mismatches (read-only).
+- `crucible assets fix-mdl` — lowercase `.mdl` filenames and rewrite ASCII
+  model/root names to match each file's stem.
+- `crucible assets check-dupes` — report runtime-name (basename) collisions
+  across directories.
+- `crucible assets clean-dupes` — delete files from a "clean" tree whose
+  basename collides with anything in a "primary" tree.
+
+## Principles
+
+- **No configuration files. Arguments only.** No `processing/` staging dirs, no
+  discards, no YAML, no reports. Operates in place on the target directory.
+- **Discover, don't configure.** External tools (the NWN engine, ImageMagick, an
+  upscaler) are found on `PATH` and at standard install locations. Each has a
+  single override flag when discovery is not enough.
+- **Fail closed, never fake** (Crucible rule #1). A missing backend is a clear
+  error naming what to install and a non-zero exit — never a placeholder
+  artifact.
+- **Simplify.** Drop the reference tools' padding / power-of-two / `nwn-safe` /
+  explicit-format knobs (convert auto-picks DXT1 vs DXT5) and the small-texture
+  staging (upscale). These can return later if wanted.
+
+## Architecture
+
+A new self-contained builder that follows the existing `depot` pattern exactly
+(delegates straight to its own internal package, bypassing the legacy
+`internal/app` surface):
+
+- `internal/assets/run.go` — `func Run(args []string, stdout, stderr io.Writer,
+  getenv func(string) string) int`. Parses the subcommand
+  (`compile|convert|upscale|check-mdl|fix-mdl|check-dupes|clean-dupes`) and
+  dispatches. Returns a sysexits-style code.
+- `internal/assets/` — one file per command plus small shared helpers
+  (recursion/file-selection, external-command runner, backend discovery).
+- `cmd/crucible-assets/main.go` — one-line shim:
+  `func main() { os.Exit(dispatch.RunBuilder("assets", os.Args[1:])) }`.
+- `internal/dispatch`:
+  - one `Registry` entry for `assets` (`Wired: true`) listing the seven
+    commands with `Usage`/`Options`;
+  - extend the direct-delegate branch that today reads
+    `if b.Name == "depot" && b.Wired` so `assets` also routes to
+    `assets.Run(...)` instead of the legacy `app.Run`.
+- `docs/command-surface.md` — seven visible-command rows.
+- `wrappers/consumers.txt` / wrappers are unchanged (no new consumer; the
+  consumers listed already vendor the wrapper).
+
+### Shared helpers (`internal/assets`)
+
+- `walk(roots, exts)` — collect files under each root (recursive by default)
+  whose extension is in `exts`, case-insensitively. Reject nothing fancy; these
+  are arguments the caller chose.
+- `run(cmd, args...)` — thin `exec.Command` wrapper capturing combined output,
+  returning `(output, error)`. A single indirection point so tests can observe
+  invocations. cwd and env overrides passed as needed.
+- `look(names...)` — return the first name found via `exec.LookPath`, or the
+  first path that exists from a supplied list of candidates. Used by every
+  backend discovery.
+
+### Shared MDL name module (`internal/assets/mdl`)
+
+A pure-Go port of `mdl-name-lib.sh` + `mdl-scan.awk`, no external tools. One
+place three commands share (`compile`'s pre-check, `check-mdl`, `fix-mdl`):
+
+- `IsASCII(path)` — NUL in the first 256 bytes → binary; else the leading
+  keyword must be `#`/`newmodel`/`node`/`setsupermodel`. Mirrors `mdl_is_ascii`.
+- `ExpectedName(path)` — the file stem (sans `.mdl`).
+- `CheckNames(path) []Mismatch` — for an ASCII model, report header-token
+  mismatches (`newmodel`, `setsupermodel`, `beginmodelgeom`, `endmodelgeom`,
+  `donemodel`, `newanim`, `doneanim` — case-insensitive vs the stem) and the
+  geometry **base node** mismatch (the node inside `beginmodelgeom` whose parent
+  is `null`; empty if none). Mirrors `mdl_check_model_name` + `mdl_base_name`.
+- `FixNames(in) (out []byte, changed bool)` — rewrite only the model identity:
+  the header tokens above, plus the base node's declaration / any `parent` /
+  `animroot` pointing at the *old* base name → the expected name. Animation bone
+  names and supermodel animroots are left untouched so inheritance keeps working.
+  Mirrors `mdl_fix_model_name` byte-for-byte in intent.
+
+Detection must stay identical to the awk classifier so behavior does not drift
+while `import.sh` still runs the awk version (see Consumer migration).
+
+## `crucible assets compile <dir>…`
+
+Usage: `crucible assets compile [--nwn <install>] [--non-recursive] <dir>…`
+
+Recursively find ASCII `.mdl` files under each `<dir>` and compile each to a
+binary `.mdl` in place (the compiled result replaces the source).
+
+The NWN:EE engine is the only real ASCII→binary MDL compiler, so this drives it
+exactly as the reference does:
+
+- **Discovery.** User data dir is fixed on Linux at
+  `~/.local/share/Neverwinter Nights` (holds the flat `development/` and
+  `modelcompiler/` folders the engine uses). The `nwmain` binary is found by
+  probing standard install roots, first hit wins:
+  - `~/.local/share/Steam/steamapps/common/Neverwinter Nights/bin/linux-x86/nwmain-linux`
+  - GOG: `~/GOG Games/Neverwinter Nights Enhanced Edition/.../nwmain-linux`
+  - Beamdog install dirs.
+  - `--nwn <install>` overrides (path to the install root or directly to the
+    binary).
+- **Headless.** The engine is a GUI binary. If there is no `DISPLAY` and
+  `xvfb-run` is present, wrap the call:
+  `xvfb-run -a --server-args=-screen 0 1024x768x24 nwmain-linux compilemodel <stem>`.
+- **Per model (one at a time — the engine's `development/` and `modelcompiler/`
+  folders are flat and single-slot):**
+  1. Skip if the file is not ASCII (binary/compiled `.mdl`) — reported, not an
+     error.
+  2. Copy source into `development/<name>`.
+  3. Run `nwmain-linux compilemodel <stem>` with cwd = the binary dir.
+  4. Collect the compiled artifact from `modelcompiler/` matched
+     case-insensitively by stem.
+  5. Move it back over the source path, lowercased.
+  6. Clean the temp files created in `development/` and `modelcompiler/`.
+- **Collisions.** A pre-existing `development/<name>` or `modelcompiler/<stem>`
+  aborts that model before any mutation.
+
+Exit: non-zero if any model fails; each failure prints a short excerpt of the
+engine log (`~/.local/share/Neverwinter Nights/logs/nwengineLog.txt`) as
+advisory diagnostics.
+
+Dropped vs reference: internal-name-mismatch pre-checks and the
+`Model Names Differ` fail rule are kept (cheap, prevents silent wrong output),
+using the shared `internal/assets/mdl` module — `compile` aborts a model whose
+names differ and tells the user to run `crucible assets fix-mdl`. It does **not**
+auto-fix (kept separate, by decision). The discard-manifest, dry-run/report
+layer, and `processing/in|out` staging are dropped.
+
+## `crucible assets convert <dir>…`
+
+Usage: `crucible assets convert [--to dds|png|tga] [--backend <path>]
+[--non-recursive] <dir>…`
+
+Recursively convert textures in place. `--to` defaults to `dds`. The source set
+is every texture under the roots whose format is not already the target, among
+`{.dds, .png, .tga}`.
+
+**Every conversion flips vertically exactly once.** Converting to DDS produces
+an upside-down DDS (NWN's convention); converting a DDS back to PNG/TGA flips it
+upright again. The flip is unconditional — it is the defining behavior, not a
+policy.
+
+Backend: ImageMagick (`magick`), one tool for decode + flip + encode across all
+three formats:
+
+- to DDS: `magick <src> -flip -define dds:mipmaps=<n> -define
+  dds:compression=<dxt1|dxt5> <dst>.dds`, choosing DXT1 when the source is
+  opaque and DXT5 when it has alpha (detected with `magick identify`).
+- from DDS: `magick <src>.dds -flip <dst>.<png|tga>`.
+- `--backend <path>` points at a different ImageMagick-compatible binary or a
+  dedicated encoder for higher-quality DXT (e.g. compressonator); default is
+  `magick`.
+
+The original file is replaced in place — canonical DDS is the point. Exit
+non-zero if any file fails.
+
+Dropped vs reference: `--pad-color`, POT/legacy-safe/`nwn-safe`/`--strict`
+sizing, explicit `--format`, alpha-detect toggles, discard staging. Auto DXT
+selection replaces the format knobs.
+
+## `crucible assets upscale <dir>…`
+
+Usage: `crucible assets upscale [--scale N] [--backend <path>]
+[--non-recursive] <dir>…`
+
+Recursively upscale textures in place.
+
+- **Backend discovery**, first found wins: `upscayl-bin`, `upscayl`,
+  `realesrgan-ncnn-vulkan`, `waifu2x-ncnn-vulkan`. These share a compatible
+  ncnn-vulkan CLI shape (`-i <in> -o <out> -s <scale>`). `--backend <path>`
+  overrides.
+- `--scale N` default 4.
+- Backends operate on PNG; real NWN textures are DDS, so for a `.dds` input the
+  command reuses the convert helpers: dds→png (flip), run the backend, png→dds
+  (flip back) — yielding a correctly-flipped upscaled DDS. PNG/TGA inputs are
+  upscaled directly in place.
+- No backend found → fail closed with a message naming the supported backends.
+
+Dropped vs reference: min-dimension small-texture staging, the separate
+`--dds-backend` plumbing (it reuses convert), configured backend-path table.
+
+## `crucible assets check-mdl <path>…`
+
+Port of `check-ascii-mdl.sh`. Read-only. Each `<path>` is a file or a directory
+(recursed for `*.mdl`). For every model, using `internal/assets/mdl`:
+
+- report an **uncompiled ASCII** `.mdl` (a binary/compiled one is fine, skipped);
+- report every model-name mismatch (header tokens and geometry base node) with
+  file, line, the offending token, and the expected stem.
+
+Exit non-zero if any ASCII model or mismatch is found; zero and an `OK` line
+otherwise. No mutation.
+
+## `crucible assets fix-mdl [--dry-run] <path>…`
+
+Port of `fix-mdl-model-names.sh`. For each `*.mdl` under the paths:
+
+1. **Lowercase the filename** if it is not already lowercase (`mv`; abort that
+   file on a case-collision with an existing target).
+2. For ASCII models with a name mismatch, rewrite the model identity in place
+   via `mdl.FixNames` (binary models are skipped — the engine owns those).
+
+`--dry-run` reports every "would lowercase" / "would fix" without touching disk.
+Prints a per-file log and a final `no broken mdl model names found` when clean.
+
+Selection optimization from the reference is preserved: a file is a fix
+candidate only if it needs a content fix (name mismatch) or a filename
+lowercase, so the expensive per-file read/rewrite runs only on the few that
+matter, not the whole tree.
+
+## `crucible assets check-dupes <dir>…`
+
+Port of `check-duplicate-names.sh`'s **directory mode only**. Read-only. NWN
+packs hak entries by basename, not path, so files at different paths with the
+same basename (case-insensitively) silently clobber on pack. Across all `<dir>`
+args, report any two files whose lowercased basename matches.
+
+Exit non-zero if any collision is found.
+
+The script's `--manifests` mode (hak-scoped collisions read from `assets/*.yml`)
+is **not** ported here — it inspects manifest structure, not files on disk, so
+it belongs with the manifest/depot tooling. It moves in the follow-up spec (see
+Out of scope); until then `sow-assets-manifest` keeps
+`check-duplicate-names.sh` solely for that mode.
+
+## `crucible assets clean-dupes [--dry-run] <primary> <clean>`
+
+Port of `clean-duplicate-names.sh`. Deletes files from the `<clean>` tree whose
+basename collides (case-insensitively) with any file in the `<primary>` tree.
+Mutates only `<clean>`. Same rule as `check-dupes` directory mode, but resolves
+the collision by removal instead of reporting.
+
+- Refuses to run if `<primary>` and `<clean>` overlap (either contains the
+  other), matching the script's guard.
+- `--dry-run` reports every "would delete" without removing.
+
+Exit zero on success with a count of removed (or would-remove) files.
+
+## Consumer migration (`sow-assets-manifest`)
+
+`sow-assets-manifest` already ships the `crucible` bootstrap wrapper
+(`crucible.sh`) and uses `crucible depot` from its Makefile, so it resolves a
+released `crucible` with zero extra setup.
+
+- **Delete** the three fully-folded driver scripts: `scripts/check-ascii-mdl.sh`,
+  `scripts/fix-mdl-model-names.sh`, `scripts/clean-duplicate-names.sh`.
+- **Repoint** the Makefile targets to the wrapper:
+  - `check-mdl` → `./crucible.sh assets check-mdl …`
+  - `fix-mdl` → `./crucible.sh assets fix-mdl [--dry-run] …`
+  - `check-dupes` — its directory scan → `./crucible.sh assets check-dupes …`;
+    the `check-duplicate-names.sh --manifests` line stays as-is.
+  - `clean-dupes` → `./crucible.sh assets clean-dupes [--dry-run] <PRIMARY> <CLEAN>`
+- **Keep for now:**
+  - `scripts/check-duplicate-names.sh` — kept **only** for its `--manifests`
+    mode, still called by the `check` and `haks` targets. Its directory mode is
+    superseded by `crucible assets check-dupes`. Ported and deleted in the
+    follow-up.
+  - `scripts/mdl-scan.awk` and `scripts/mdl-name-lib.sh` — still sourced by
+    `import.sh` for its batched pre-import gate (~28× faster than per-file, on a
+    hot path). They retire in the follow-up too. Because both the awk classifier
+    and the Go `mdl` module must agree, the port keeps detection byte-for-byte
+    identical (a shared fixture set checks this).
+
+Nothing in `sow-assets-manifest` is committed by this project's plan except the
+script deletions and Makefile edits; that repo's change lands as its own commit
+once the new `crucible` release with `assets` is available.
+
+## Out of scope — follow-up spec
+
+Moving the **depot-interacting pipeline** into `crucible depot` is a separate
+project with its own spec
+(`2026-07-12-crucible-depot-pipeline-migration-design.md`). It covers
+`import.sh`, `sync-assets.sh`, `export.sh`, the inline mdl gate, and the
+`check-duplicate-names.sh --manifests` manifest-collision check. When that lands:
+its gate uses the in-process `internal/assets/mdl` module, and
+`scripts/mdl-scan.awk`, `scripts/mdl-name-lib.sh`, and
+`scripts/check-duplicate-names.sh` are all deleted. This spec deliberately stops
+at the standalone, file-on-disk integrity tools to keep one focused plan.
+
+## Error handling
+
+- Missing external tool (engine, `magick`, upscaler) → non-zero exit, message
+  names the tool and how to get it. Never a faked artifact. The pure-Go
+  integrity commands (`check-mdl`, `fix-mdl`, `check-dupes`, `clean-dupes`) need
+  no external tool.
+- Per-file failures are collected and printed as a summary line at the end; the
+  command exits non-zero if any file failed but still processes the rest.
+- Report commands (`check-mdl`, `check-dupes`) exit non-zero when they *find*
+  problems — that is their contract as CI/pre-import gates, not a tool error.
+- Unknown subcommand / bad flags → usage error, exit 64.
+
+## Testing
+
+Go tests in `internal/assets`:
+
+- flag/subcommand parsing and usage errors;
+- backend discovery against a fabricated `PATH` (temp dir with stub
+  executables) — asserts first-match ordering and the `--backend` override;
+- `walk` recursion and extension filtering, including `--non-recursive`;
+- one real convert round-trip using `magick` (present in the dev shell):
+  encode a small PNG to DDS and back, asserting the pixels are vertically
+  flipped after a single conversion and restored after the round trip.
+- `internal/assets/mdl`: table-driven fixtures — ASCII vs binary detection,
+  each header-token mismatch, the base-node mismatch, and `FixNames` producing
+  a model that then passes `CheckNames`. A parity fixture set shared in intent
+  with `mdl-scan.awk` so the Go port and the still-present awk agree.
+- `check-dupes`/`clean-dupes`: temp trees asserting basename-collision
+  detection (case-insensitive), the overlap guard, and `--dry-run` mutating
+  nothing.
+
+External engine/upscaler calls are exercised through the `run` indirection with
+a stub in tests; they are not invoked for real in CI.
+
+`make check` must stay green; add an `assets` expectation to `make smoke` for
+the wired exit.

docs/superpowers/specs/2026-07-12-crucible-depot-pipeline-migration-design.md

@@ -0,0 +1,90 @@
+# Crucible `depot` pipeline migration — design (charter / draft)
+
+Date: 2026-07-12
+Status: **draft — needs its own brainstorming pass before implementation.**
+Depends on: `2026-07-12-crucible-assets-builder-design.md` (the `assets` builder
+and `internal/assets/mdl` module) landing first.
+
+This is a scoping charter, not an implementation-ready design. The scripts it
+covers are nontrivial and lean heavily on `sow-assets-manifest/scripts/lib.sh`
+(~35 KB). A full design requires studying `lib.sh` and the manifest format in
+depth; that work happens in this spec's own brainstorming before any code.
+
+## Why this exists
+
+`sow-assets-manifest` is the last consumer still carrying real builder logic in
+shell. The `assets` spec folded its standalone integrity tools into Crucible but
+deliberately left the **depot-interacting pipeline** behind, because those
+scripts read/write the content-addressed depot and the asset manifests — depot
+domain, and a bigger lift. This charter completes the migration so
+`sow-assets-manifest` becomes what the other consumers already are: a thin
+caller of released `crucible` binaries with no vendored toolkit logic.
+
+## What moves into `crucible depot`
+
+The depot builder already owns `status/push/verify/get/pull`. This adds the
+authoring side:
+
+| Script | Proposed command | What it does today |
+| --- | --- | --- |
+| `import.sh` (318 lines) | `crucible depot import <edit-dir> [prefix…]` | Import an edit tree into the depot + manifests: sticky per-path member assignment, new paths to the tail member, per-category `max_bytes` rebalance (spill overflow forward, no full repack), a local stat-cache to skip unchanged files, batched hashing + parallel `depot_put_many`. |
+| `sync-assets.sh` (61 lines) | `crucible depot sync <tree> [prefix…]` | Full reconcile = import + prune manifest entries whose path no longer exists (scoped by prefix when given). Delegates to `import.sh`. |
+| `export.sh` (96 lines) | `crucible depot export [glob] [--hak m] [--restype ext] --to <dir>` | Materialize a manifest-selected subset out of the depot into a clean edit tree (category paths preserved, no metadata). |
+| `check-duplicate-names.sh --manifests` | `crucible depot check-dupes` (name TBD) | Hak-scoped basename collisions read from `assets/*.yml` (`.assets[].path` grouped by `.assets[].hak`). Manifest-structure check, not files on disk. |
+
+The inline **pre-import mdl gate** in `import.sh` (batched `mdl-scan.awk` pass
+that aborts on an H/B model-name mismatch before any blob is hashed/uploaded)
+is reimplemented with the in-process `internal/assets/mdl` module from the
+`assets` spec — same H/B contract, no subprocess.
+
+## What retires when this lands
+
+Deleted from `sow-assets-manifest/scripts/` once the commands above are wired
+and the Makefile is repointed:
+
+- `import.sh`, `sync-assets.sh`, `export.sh`
+- `mdl-scan.awk`, `mdl-name-lib.sh` (only remaining consumer was `import.sh`)
+- `check-duplicate-names.sh` (its directory mode already superseded by
+  `crucible assets check-dupes`; the `--manifests` mode moves here)
+- whatever else in `lib.sh` becomes dead once the above are gone (assess during
+  design — `lib.sh` also serves `build-haks.sh`, `pack-haks.sh`, `promote.sh`,
+  etc., which are **not** in scope here, so `lib.sh` likely shrinks, not dies)
+
+Makefile targets `import`, `sync`, `haks` (its sync/import/dupe steps), `export`
+repoint to `./crucible.sh depot …`.
+
+## Known hard parts (resolve in brainstorming)
+
+- **Manifest read/write parity.** Byte-stable output, member `max_bytes`
+  rebalance semantics, sticky path→member assignment. Must match the current
+  awk/`lib.sh` behavior or diff-review of manifests becomes noise. Crucible
+  already parses these manifests (`internal/pipeline`, hak build) — reuse, don't
+  reinvent.
+- **Depot backends.** `import`/`export` drive bunny/cdn/local through
+  `lib.sh`'s `depot_put_many`/`depot_require_write` (incl. the
+  `BUNNY_STORAGE_PASSWORD` prompt and cdn→bunny write normalization). Crucible's
+  `internal/depot` already speaks these backends for pull/push — the authoring
+  side should share that client, not a second implementation.
+- **Stat-cache.** `import.sh`'s `.cache/import/<target>/<cat>.tsv` skip-unchanged
+  optimization is the dominant re-import speedup. Decide whether Crucible
+  reproduces it, replaces it (content-addressing already dedupes uploads), or
+  drops it with a measured justification.
+- **Throughput.** The shell versions are already batched to avoid per-file
+  forks; a Go port should be at least as fast (in-process, no forks) — verify,
+  don't assume.
+- **Scoping semantics.** Prefix-scoped import/prune (segment-boundary match) and
+  export glob matching (`**`/`*`/literal dots, no shell expansion) must port
+  exactly.
+
+## Out of scope
+
+The HAK build/pack/promote/publish scripts (`build-haks.sh`, `pack-haks.sh`,
+`promote.sh`, `publish-release*.sh`, `hak-artifact-record.sh`) and their
+`lib.sh` support. `crucible hak build` already exists; whether these thin
+release wrappers also migrate is a separate question, not this charter.
+
+## Next step
+
+Brainstorm this charter into a full design: read `lib.sh` and the manifest
+format, settle the hard parts above, then write the implementation-ready spec
+and plan. Do not implement from this document.