sow-tools/docs/superpowers/specs/2026-07-12-crucible-assets-builder-design.md

Crucible assets builder — design

Date: 2026-07-12 Status: approved, pre-implementation

Goal

Fold the NWN:EE asset tools out of two standalone toolkits into Crucible as a new self-contained builder, crucible assets:

• from the Python toolkit (nwnee-asset-processing) — model compile, texture convert, texture upscale. Those scripts are references only; broken as-is and carrying a heavy config/staging/report layer that we drop.
• from sow-assets-manifest/scripts/ — the model-name and duplicate-name integrity tools (check-ascii-mdl.sh, fix-mdl-model-names.sh, check-duplicate-names.sh, clean-duplicate-names.sh). These are ported into Go and the shell drivers removed; sow-assets-manifest calls ./crucible.sh assets … through its bootstrap wrapper instead.

Seven commands:

• crucible assets compile — compile every ASCII .mdl in a directory to binary, in place.
• crucible assets convert — convert every texture in a directory to NWN:EE DDS (or back to PNG/TGA), flipping vertically every time so a DDS is always upside-down relative to its source.
• crucible assets upscale — upscale every texture in a directory through whatever upscaling backend is installed.
• crucible assets check-mdl — report uncompiled ASCII .mdl files and model-name mismatches (read-only).
• crucible assets fix-mdl — lowercase .mdl filenames and rewrite ASCII model/root names to match each file's stem.
• crucible assets check-dupes — report runtime-name (basename) collisions in directories, or hak-scoped collisions across assets/*.yml manifests.
• crucible assets clean-dupes — delete files from a "clean" tree whose basename collides with anything in a "primary" tree.

Principles

• No configuration files. Arguments only. No processing/ staging dirs, no discards, no YAML, no reports. Operates in place on the target directory.
• Discover, don't configure. External tools (the NWN engine, ImageMagick, an upscaler) are found on PATH and at standard install locations. Each has a single override flag when discovery is not enough.
• Fail closed, never fake (Crucible rule #1). A missing backend is a clear error naming what to install and a non-zero exit — never a placeholder artifact.
• Simplify. Drop the reference tools' padding / power-of-two / nwn-safe / explicit-format knobs (convert auto-picks DXT1 vs DXT5) and the small-texture staging (upscale). These can return later if wanted.

Architecture

A new self-contained builder that follows the existing depot pattern exactly (delegates straight to its own internal package, bypassing the legacy internal/app surface):

• internal/assets/run.go — func Run(args []string, stdout, stderr io.Writer, getenv func(string) string) int. Parses the subcommand (compile|convert|upscale|check-mdl|fix-mdl|check-dupes|clean-dupes) and dispatches. Returns a sysexits-style code.
• internal/assets/ — one file per command plus small shared helpers (recursion/file-selection, external-command runner, backend discovery).
• cmd/crucible-assets/main.go — one-line shim: func main() { os.Exit(dispatch.RunBuilder("assets", os.Args[1:])) }.
• internal/dispatch:
  • one Registry entry for assets (Wired: true) listing the seven commands with Usage/Options;
  • extend the direct-delegate branch that today reads if b.Name == "depot" && b.Wired so assets also routes to assets.Run(...) instead of the legacy app.Run.
• docs/command-surface.md — seven visible-command rows.
• wrappers/consumers.txt / wrappers are unchanged (no new consumer; the consumers listed already vendor the wrapper).

Shared helpers (internal/assets)

• walk(roots, exts) — collect files under each root (recursive by default) whose extension is in exts, case-insensitively. Reject nothing fancy; these are arguments the caller chose.
• run(cmd, args...) — thin exec.Command wrapper capturing combined output, returning (output, error). A single indirection point so tests can observe invocations. cwd and env overrides passed as needed.
• look(names...) — return the first name found via exec.LookPath, or the first path that exists from a supplied list of candidates. Used by every backend discovery.

Shared MDL name module (internal/assets/mdl)

A pure-Go port of mdl-name-lib.sh + mdl-scan.awk, no external tools. One place three commands share (compile's pre-check, check-mdl, fix-mdl):

• IsASCII(path) — NUL in the first 256 bytes → binary; else the leading keyword must be #/newmodel/node/setsupermodel. Mirrors mdl_is_ascii.
• ExpectedName(path) — the file stem (sans .mdl).
• CheckNames(path) []Mismatch — for an ASCII model, report header-token mismatches (newmodel, setsupermodel, beginmodelgeom, endmodelgeom, donemodel, newanim, doneanim — case-insensitive vs the stem) and the geometry base node mismatch (the node inside beginmodelgeom whose parent is null; empty if none). Mirrors mdl_check_model_name + mdl_base_name.
• FixNames(in) (out []byte, changed bool) — rewrite only the model identity: the header tokens above, plus the base node's declaration / any parent / animroot pointing at the old base name → the expected name. Animation bone names and supermodel animroots are left untouched so inheritance keeps working. Mirrors mdl_fix_model_name byte-for-byte in intent.

Detection must stay identical to the awk classifier so behavior does not drift while import.sh still runs the awk version (see Consumer migration).

crucible assets compile <dir>…

Usage: crucible assets compile [--nwn <install>] [--non-recursive] <dir>…

Recursively find ASCII .mdl files under each <dir> and compile each to a binary .mdl in place (the compiled result replaces the source).

The NWN:EE engine is the only real ASCII→binary MDL compiler, so this drives it exactly as the reference does:

• Discovery. User data dir is fixed on Linux at ~/.local/share/Neverwinter Nights (holds the flat development/ and modelcompiler/ folders the engine uses). The nwmain binary is found by probing standard install roots, first hit wins:
  • ~/.local/share/Steam/steamapps/common/Neverwinter Nights/bin/linux-x86/nwmain-linux
  • GOG: ~/GOG Games/Neverwinter Nights Enhanced Edition/.../nwmain-linux
  • Beamdog install dirs.
  • --nwn <install> overrides (path to the install root or directly to the binary).
• Headless. The engine is a GUI binary. If there is no DISPLAY and xvfb-run is present, wrap the call: xvfb-run -a --server-args=-screen 0 1024x768x24 nwmain-linux compilemodel <stem>.
• Per model (one at a time — the engine's development/ and modelcompiler/ folders are flat and single-slot):
  1. Skip if the file is not ASCII (binary/compiled .mdl) — reported, not an error.
  2. Copy source into development/<name>.
  3. Run nwmain-linux compilemodel <stem> with cwd = the binary dir.
  4. Collect the compiled artifact from modelcompiler/ matched case-insensitively by stem.
  5. Move it back over the source path, lowercased.
  6. Clean the temp files created in development/ and modelcompiler/.
• Collisions. A pre-existing development/<name> or modelcompiler/<stem> aborts that model before any mutation.

Exit: non-zero if any model fails; each failure prints a short excerpt of the engine log (~/.local/share/Neverwinter Nights/logs/nwengineLog.txt) as advisory diagnostics.

Dropped vs reference: internal-name-mismatch pre-checks and the Model Names Differ fail rule are kept (cheap, prevents silent wrong output), using the shared internal/assets/mdl module — compile aborts a model whose names differ and tells the user to run crucible assets fix-mdl. It does not auto-fix (kept separate, by decision). The discard-manifest, dry-run/report layer, and processing/in|out staging are dropped.

crucible assets convert <dir>…

Usage: crucible assets convert [--to dds|png|tga] [--backend <path>] [--non-recursive] <dir>…

Recursively convert textures in place. --to defaults to dds. The source set is every texture under the roots whose format is not already the target, among {.dds, .png, .tga}.

Every conversion flips vertically exactly once. Converting to DDS produces an upside-down DDS (NWN's convention); converting a DDS back to PNG/TGA flips it upright again. The flip is unconditional — it is the defining behavior, not a policy.

Backend: ImageMagick (magick), one tool for decode + flip + encode across all three formats:

• to DDS: magick <src> -flip -define dds:mipmaps=<n> -define dds:compression=<dxt1|dxt5> <dst>.dds, choosing DXT1 when the source is opaque and DXT5 when it has alpha (detected with magick identify).
• from DDS: magick <src>.dds -flip <dst>.<png|tga>.
• --backend <path> points at a different ImageMagick-compatible binary or a dedicated encoder for higher-quality DXT (e.g. compressonator); default is magick.

The original file is replaced in place — canonical DDS is the point. Exit non-zero if any file fails.

Dropped vs reference: --pad-color, POT/legacy-safe/nwn-safe/--strict sizing, explicit --format, alpha-detect toggles, discard staging. Auto DXT selection replaces the format knobs.

crucible assets upscale <dir>…

Usage: crucible assets upscale [--scale N] [--backend <path>] [--non-recursive] <dir>…

Recursively upscale textures in place.

• Backend discovery, first found wins: upscayl-bin, upscayl, realesrgan-ncnn-vulkan, waifu2x-ncnn-vulkan. These share a compatible ncnn-vulkan CLI shape (-i <in> -o <out> -s <scale>). --backend <path> overrides.
• --scale N default 4.
• Backends operate on PNG; real NWN textures are DDS, so for a .dds input the command reuses the convert helpers: dds→png (flip), run the backend, png→dds (flip back) — yielding a correctly-flipped upscaled DDS. PNG/TGA inputs are upscaled directly in place.
• No backend found → fail closed with a message naming the supported backends.

Dropped vs reference: min-dimension small-texture staging, the separate --dds-backend plumbing (it reuses convert), configured backend-path table.

crucible assets check-mdl <path>…

Port of check-ascii-mdl.sh. Read-only. Each <path> is a file or a directory (recursed for *.mdl). For every model, using internal/assets/mdl:

• report an uncompiled ASCII .mdl (a binary/compiled one is fine, skipped);
• report every model-name mismatch (header tokens and geometry base node) with file, line, the offending token, and the expected stem.

Exit non-zero if any ASCII model or mismatch is found; zero and an OK line otherwise. No mutation.

crucible assets fix-mdl [--dry-run] <path>…

Port of fix-mdl-model-names.sh. For each *.mdl under the paths:

1. Lowercase the filename if it is not already lowercase (mv; abort that file on a case-collision with an existing target).
2. For ASCII models with a name mismatch, rewrite the model identity in place via mdl.FixNames (binary models are skipped — the engine owns those).

--dry-run reports every "would lowercase" / "would fix" without touching disk. Prints a per-file log and a final no broken mdl model names found when clean.

Selection optimization from the reference is preserved: a file is a fix candidate only if it needs a content fix (name mismatch) or a filename lowercase, so the expensive per-file read/rewrite runs only on the few that matter, not the whole tree.

crucible assets check-dupes [--manifests] <dir>…

Port of check-duplicate-names.sh. Read-only. NWN packs hak entries by basename, not path, so files at different paths with the same basename (case-insensitively) silently clobber on pack. This flags them.

• Directory mode (default): across all <dir> args, report any two files whose lowercased basename matches.
• --manifests: additionally read every assets/*.yml in the current repo and report basename collisions within the same hak (.assets[].path grouped by .assets[].hak), reusing Crucible's existing gopkg.in/yaml.v3 parsing. Requires no external yq.

Exit non-zero if any collision is found.

crucible assets clean-dupes [--dry-run] <primary> <clean>

Port of clean-duplicate-names.sh. Deletes files from the <clean> tree whose basename collides (case-insensitively) with any file in the <primary> tree. Mutates only <clean>. Same rule as check-dupes directory mode, but resolves the collision by removal instead of reporting.

• Refuses to run if <primary> and <clean> overlap (either contains the other), matching the script's guard.
• --dry-run reports every "would delete" without removing.

Exit zero on success with a count of removed (or would-remove) files.

Consumer migration (sow-assets-manifest)

sow-assets-manifest already ships the crucible bootstrap wrapper (crucible.sh) and uses crucible depot from its Makefile, so it resolves a released crucible with zero extra setup.

• Delete the four folded driver scripts: scripts/check-ascii-mdl.sh, scripts/fix-mdl-model-names.sh, scripts/check-duplicate-names.sh, scripts/clean-duplicate-names.sh.
• Repoint the Makefile targets to the wrapper:
  • check-mdl → ./crucible.sh assets check-mdl …
  • fix-mdl → ./crucible.sh assets fix-mdl [--dry-run] …
  • check-dupes → ./crucible.sh assets check-dupes … and ./crucible.sh assets check-dupes --manifests
  • clean-dupes → ./crucible.sh assets clean-dupes [--dry-run] <PRIMARY> <CLEAN>
  • the haks target's check-duplicate-names.sh --manifests guard → ./crucible.sh assets check-dupes --manifests
• Keep for now: scripts/mdl-scan.awk and scripts/mdl-name-lib.sh. They are still sourced by import.sh for its batched pre-import gate (~28× faster than per-file, on a hot path). They retire in the follow-up below. Because both the awk classifier and the Go mdl module must agree, the port keeps detection byte-for-byte identical (a shared fixture set checks this).

Nothing in sow-assets-manifest is committed by this project's plan except the script deletions and Makefile edits; that repo's change lands as its own commit once the new crucible release with assets is available.

Out of scope — follow-up spec

Moving the depot-interacting pipeline (import.sh, sync-assets.sh, export.sh, and the inline mdl gate) into crucible depot is a separate project with its own spec. When import.sh becomes crucible depot import, its gate uses the in-process internal/assets/mdl module and scripts/mdl-scan.awk + scripts/mdl-name-lib.sh are deleted. This spec deliberately stops at the standalone integrity tools to keep one focused plan.

Error handling

• Missing external tool (engine, magick, upscaler) → non-zero exit, message names the tool and how to get it. Never a faked artifact. The pure-Go integrity commands (check-mdl, fix-mdl, check-dupes, clean-dupes) need no external tool.
• Per-file failures are collected and printed as a summary line at the end; the command exits non-zero if any file failed but still processes the rest.
• Report commands (check-mdl, check-dupes) exit non-zero when they find problems — that is their contract as CI/pre-import gates, not a tool error.
• Unknown subcommand / bad flags → usage error, exit 64.

Testing

Go tests in internal/assets:

• flag/subcommand parsing and usage errors;
• backend discovery against a fabricated PATH (temp dir with stub executables) — asserts first-match ordering and the --backend override;
• walk recursion and extension filtering, including --non-recursive;
• one real convert round-trip using magick (present in the dev shell): encode a small PNG to DDS and back, asserting the pixels are vertically flipped after a single conversion and restored after the round trip.
• internal/assets/mdl: table-driven fixtures — ASCII vs binary detection, each header-token mismatch, the base-node mismatch, and FixNames producing a model that then passes CheckNames. A parity fixture set shared in intent with mdl-scan.awk so the Go port and the still-present awk agree.
• check-dupes/clean-dupes: temp trees asserting basename-collision detection (case-insensitive), the --manifests hak-scoped path against a small assets/*.yml fixture, the overlap guard, and --dry-run mutating nothing.

External engine/upscaler calls are exercised through the run indirection with a stub in tests; they are not invoked for real in CI.

make check must stay green; add an assets expectation to make smoke for the wired exit.