sow-tools/docs/superpowers/specs/2026-07-12-crucible-assets-builder-design.md

# Crucible `assets` builder — design

Date: 2026-07-12
Status: approved, pre-implementation

## Goal

Fold the NWN:EE asset tools out of two standalone toolkits into Crucible as a
new self-contained builder, `crucible assets`:

- from the Python toolkit (`nwnee-asset-processing`) — model compile, texture
  convert, texture upscale. Those scripts are references only; broken as-is and
  carrying a heavy config/staging/report layer that we drop.
- from `sow-assets-manifest/scripts/` — the model-name and duplicate-name
  integrity tools (`check-ascii-mdl.sh`, `fix-mdl-model-names.sh`,
  `check-duplicate-names.sh`, `clean-duplicate-names.sh`). These are ported into
  Go and the shell drivers removed; `sow-assets-manifest` calls
  `./crucible.sh assets …` through its bootstrap wrapper instead.

Seven commands:

- `crucible assets compile` — compile every ASCII `.mdl` in a directory to
  binary, in place.
- `crucible assets convert` — convert every texture in a directory to NWN:EE
  DDS (or back to PNG/TGA), flipping vertically **every time** so a DDS is
  always upside-down relative to its source.
- `crucible assets upscale` — upscale every texture in a directory through
  whatever upscaling backend is installed.
- `crucible assets check-mdl` — report uncompiled ASCII `.mdl` files and
  model-name mismatches (read-only).
- `crucible assets fix-mdl` — lowercase `.mdl` filenames and rewrite ASCII
  model/root names to match each file's stem.
- `crucible assets check-dupes` — report runtime-name (basename) collisions in
  directories, or hak-scoped collisions across `assets/*.yml` manifests.
- `crucible assets clean-dupes` — delete files from a "clean" tree whose
  basename collides with anything in a "primary" tree.

## Principles

- **No configuration files. Arguments only.** No `processing/` staging dirs, no
  discards, no YAML, no reports. Operates in place on the target directory.
- **Discover, don't configure.** External tools (the NWN engine, ImageMagick, an
  upscaler) are found on `PATH` and at standard install locations. Each has a
  single override flag when discovery is not enough.
- **Fail closed, never fake** (Crucible rule #1). A missing backend is a clear
  error naming what to install and a non-zero exit — never a placeholder
  artifact.
- **Simplify.** Drop the reference tools' padding / power-of-two / `nwn-safe` /
  explicit-format knobs (convert auto-picks DXT1 vs DXT5) and the small-texture
  staging (upscale). These can return later if wanted.

## Architecture

A new self-contained builder that follows the existing `depot` pattern exactly
(delegates straight to its own internal package, bypassing the legacy
`internal/app` surface):

- `internal/assets/run.go` — `func Run(args []string, stdout, stderr io.Writer,
  getenv func(string) string) int`. Parses the subcommand
  (`compile|convert|upscale|check-mdl|fix-mdl|check-dupes|clean-dupes`) and
  dispatches. Returns a sysexits-style code.
- `internal/assets/` — one file per command plus small shared helpers
  (recursion/file-selection, external-command runner, backend discovery).
- `cmd/crucible-assets/main.go` — one-line shim:
  `func main() { os.Exit(dispatch.RunBuilder("assets", os.Args[1:])) }`.
- `internal/dispatch`:
  - one `Registry` entry for `assets` (`Wired: true`) listing the seven
    commands with `Usage`/`Options`;
  - extend the direct-delegate branch that today reads
    `if b.Name == "depot" && b.Wired` so `assets` also routes to
    `assets.Run(...)` instead of the legacy `app.Run`.
- `docs/command-surface.md` — seven visible-command rows.
- `wrappers/consumers.txt` / wrappers are unchanged (no new consumer; the
  consumers listed already vendor the wrapper).

### Shared helpers (`internal/assets`)

- `walk(roots, exts)` — collect files under each root (recursive by default)
  whose extension is in `exts`, case-insensitively. Reject nothing fancy; these
  are arguments the caller chose.
- `run(cmd, args...)` — thin `exec.Command` wrapper capturing combined output,
  returning `(output, error)`. A single indirection point so tests can observe
  invocations. cwd and env overrides passed as needed.
- `look(names...)` — return the first name found via `exec.LookPath`, or the
  first path that exists from a supplied list of candidates. Used by every
  backend discovery.

### Shared MDL name module (`internal/assets/mdl`)

A pure-Go port of `mdl-name-lib.sh` + `mdl-scan.awk`, no external tools. One
place three commands share (`compile`'s pre-check, `check-mdl`, `fix-mdl`):

- `IsASCII(path)` — NUL in the first 256 bytes → binary; else the leading
  keyword must be `#`/`newmodel`/`node`/`setsupermodel`. Mirrors `mdl_is_ascii`.
- `ExpectedName(path)` — the file stem (sans `.mdl`).
- `CheckNames(path) []Mismatch` — for an ASCII model, report header-token
  mismatches (`newmodel`, `setsupermodel`, `beginmodelgeom`, `endmodelgeom`,
  `donemodel`, `newanim`, `doneanim` — case-insensitive vs the stem) and the
  geometry **base node** mismatch (the node inside `beginmodelgeom` whose parent
  is `null`; empty if none). Mirrors `mdl_check_model_name` + `mdl_base_name`.
- `FixNames(in) (out []byte, changed bool)` — rewrite only the model identity:
  the header tokens above, plus the base node's declaration / any `parent` /
  `animroot` pointing at the *old* base name → the expected name. Animation bone
  names and supermodel animroots are left untouched so inheritance keeps working.
  Mirrors `mdl_fix_model_name` byte-for-byte in intent.

Detection must stay identical to the awk classifier so behavior does not drift
while `import.sh` still runs the awk version (see Consumer migration).

## `crucible assets compile <dir>…`

Usage: `crucible assets compile [--nwn <install>] [--non-recursive] <dir>…`

Recursively find ASCII `.mdl` files under each `<dir>` and compile each to a
binary `.mdl` in place (the compiled result replaces the source).

The NWN:EE engine is the only real ASCII→binary MDL compiler, so this drives it
exactly as the reference does:

- **Discovery.** User data dir is fixed on Linux at
  `~/.local/share/Neverwinter Nights` (holds the flat `development/` and
  `modelcompiler/` folders the engine uses). The `nwmain` binary is found by
  probing standard install roots, first hit wins:
  - `~/.local/share/Steam/steamapps/common/Neverwinter Nights/bin/linux-x86/nwmain-linux`
  - GOG: `~/GOG Games/Neverwinter Nights Enhanced Edition/.../nwmain-linux`
  - Beamdog install dirs.
  - `--nwn <install>` overrides (path to the install root or directly to the
    binary).
- **Headless.** The engine is a GUI binary. If there is no `DISPLAY` and
  `xvfb-run` is present, wrap the call:
  `xvfb-run -a --server-args=-screen 0 1024x768x24 nwmain-linux compilemodel <stem>`.
- **Per model (one at a time — the engine's `development/` and `modelcompiler/`
  folders are flat and single-slot):**
  1. Skip if the file is not ASCII (binary/compiled `.mdl`) — reported, not an
     error.
  2. Copy source into `development/<name>`.
  3. Run `nwmain-linux compilemodel <stem>` with cwd = the binary dir.
  4. Collect the compiled artifact from `modelcompiler/` matched
     case-insensitively by stem.
  5. Move it back over the source path, lowercased.
  6. Clean the temp files created in `development/` and `modelcompiler/`.
- **Collisions.** A pre-existing `development/<name>` or `modelcompiler/<stem>`
  aborts that model before any mutation.

Exit: non-zero if any model fails; each failure prints a short excerpt of the
engine log (`~/.local/share/Neverwinter Nights/logs/nwengineLog.txt`) as
advisory diagnostics.

Dropped vs reference: internal-name-mismatch pre-checks and the
`Model Names Differ` fail rule are kept (cheap, prevents silent wrong output),
using the shared `internal/assets/mdl` module — `compile` aborts a model whose
names differ and tells the user to run `crucible assets fix-mdl`. It does **not**
auto-fix (kept separate, by decision). The discard-manifest, dry-run/report
layer, and `processing/in|out` staging are dropped.

## `crucible assets convert <dir>…`

Usage: `crucible assets convert [--to dds|png|tga] [--backend <path>]
[--non-recursive] <dir>…`

Recursively convert textures in place. `--to` defaults to `dds`. The source set
is every texture under the roots whose format is not already the target, among
`{.dds, .png, .tga}`.

**Every conversion flips vertically exactly once.** Converting to DDS produces
an upside-down DDS (NWN's convention); converting a DDS back to PNG/TGA flips it
upright again. The flip is unconditional — it is the defining behavior, not a
policy.

Backend: ImageMagick (`magick`), one tool for decode + flip + encode across all
three formats:

- to DDS: `magick <src> -flip -define dds:mipmaps=<n> -define
  dds:compression=<dxt1|dxt5> <dst>.dds`, choosing DXT1 when the source is
  opaque and DXT5 when it has alpha (detected with `magick identify`).
- from DDS: `magick <src>.dds -flip <dst>.<png|tga>`.
- `--backend <path>` points at a different ImageMagick-compatible binary or a
  dedicated encoder for higher-quality DXT (e.g. compressonator); default is
  `magick`.

The original file is replaced in place — canonical DDS is the point. Exit
non-zero if any file fails.

Dropped vs reference: `--pad-color`, POT/legacy-safe/`nwn-safe`/`--strict`
sizing, explicit `--format`, alpha-detect toggles, discard staging. Auto DXT
selection replaces the format knobs.

## `crucible assets upscale <dir>…`

Usage: `crucible assets upscale [--scale N] [--backend <path>]
[--non-recursive] <dir>…`

Recursively upscale textures in place.

- **Backend discovery**, first found wins: `upscayl-bin`, `upscayl`,
  `realesrgan-ncnn-vulkan`, `waifu2x-ncnn-vulkan`. These share a compatible
  ncnn-vulkan CLI shape (`-i <in> -o <out> -s <scale>`). `--backend <path>`
  overrides.
- `--scale N` default 4.
- Backends operate on PNG; real NWN textures are DDS, so for a `.dds` input the
  command reuses the convert helpers: dds→png (flip), run the backend, png→dds
  (flip back) — yielding a correctly-flipped upscaled DDS. PNG/TGA inputs are
  upscaled directly in place.
- No backend found → fail closed with a message naming the supported backends.

Dropped vs reference: min-dimension small-texture staging, the separate
`--dds-backend` plumbing (it reuses convert), configured backend-path table.

## `crucible assets check-mdl <path>…`

Port of `check-ascii-mdl.sh`. Read-only. Each `<path>` is a file or a directory
(recursed for `*.mdl`). For every model, using `internal/assets/mdl`:

- report an **uncompiled ASCII** `.mdl` (a binary/compiled one is fine, skipped);
- report every model-name mismatch (header tokens and geometry base node) with
  file, line, the offending token, and the expected stem.

Exit non-zero if any ASCII model or mismatch is found; zero and an `OK` line
otherwise. No mutation.

## `crucible assets fix-mdl [--dry-run] <path>…`

Port of `fix-mdl-model-names.sh`. For each `*.mdl` under the paths:

1. **Lowercase the filename** if it is not already lowercase (`mv`; abort that
   file on a case-collision with an existing target).
2. For ASCII models with a name mismatch, rewrite the model identity in place
   via `mdl.FixNames` (binary models are skipped — the engine owns those).

`--dry-run` reports every "would lowercase" / "would fix" without touching disk.
Prints a per-file log and a final `no broken mdl model names found` when clean.

Selection optimization from the reference is preserved: a file is a fix
candidate only if it needs a content fix (name mismatch) or a filename
lowercase, so the expensive per-file read/rewrite runs only on the few that
matter, not the whole tree.

## `crucible assets check-dupes [--manifests] <dir>…`

Port of `check-duplicate-names.sh`. Read-only. NWN packs hak entries by
basename, not path, so files at different paths with the same basename
(case-insensitively) silently clobber on pack. This flags them.

- Directory mode (default): across all `<dir>` args, report any two files whose
  lowercased basename matches.
- `--manifests`: additionally read every `assets/*.yml` in the current repo and
  report basename collisions **within the same hak** (`.assets[].path` grouped
  by `.assets[].hak`), reusing Crucible's existing `gopkg.in/yaml.v3` parsing.
  Requires no external `yq`.

Exit non-zero if any collision is found.

## `crucible assets clean-dupes [--dry-run] <primary> <clean>`

Port of `clean-duplicate-names.sh`. Deletes files from the `<clean>` tree whose
basename collides (case-insensitively) with any file in the `<primary>` tree.
Mutates only `<clean>`. Same rule as `check-dupes` directory mode, but resolves
the collision by removal instead of reporting.

- Refuses to run if `<primary>` and `<clean>` overlap (either contains the
  other), matching the script's guard.
- `--dry-run` reports every "would delete" without removing.

Exit zero on success with a count of removed (or would-remove) files.

## Consumer migration (`sow-assets-manifest`)

`sow-assets-manifest` already ships the `crucible` bootstrap wrapper
(`crucible.sh`) and uses `crucible depot` from its Makefile, so it resolves a
released `crucible` with zero extra setup.

- **Delete** the four folded driver scripts: `scripts/check-ascii-mdl.sh`,
  `scripts/fix-mdl-model-names.sh`, `scripts/check-duplicate-names.sh`,
  `scripts/clean-duplicate-names.sh`.
- **Repoint** the Makefile targets to the wrapper:
  - `check-mdl` → `./crucible.sh assets check-mdl …`
  - `fix-mdl` → `./crucible.sh assets fix-mdl [--dry-run] …`
  - `check-dupes` → `./crucible.sh assets check-dupes …` and
    `./crucible.sh assets check-dupes --manifests`
  - `clean-dupes` → `./crucible.sh assets clean-dupes [--dry-run] <PRIMARY> <CLEAN>`
  - the `haks` target's `check-duplicate-names.sh --manifests` guard →
    `./crucible.sh assets check-dupes --manifests`
- **Keep for now:** `scripts/mdl-scan.awk` and `scripts/mdl-name-lib.sh`. They
  are still sourced by `import.sh` for its batched pre-import gate (~28× faster
  than per-file, on a hot path). They retire in the follow-up below. Because
  both the awk classifier and the Go `mdl` module must agree, the port keeps
  detection byte-for-byte identical (a shared fixture set checks this).

Nothing in `sow-assets-manifest` is committed by this project's plan except the
script deletions and Makefile edits; that repo's change lands as its own commit
once the new `crucible` release with `assets` is available.

## Out of scope — follow-up spec

Moving the **depot-interacting pipeline** (`import.sh`, `sync-assets.sh`,
`export.sh`, and the inline mdl gate) into `crucible depot` is a separate
project with its own spec. When `import.sh` becomes `crucible depot import`, its
gate uses the in-process `internal/assets/mdl` module and
`scripts/mdl-scan.awk` + `scripts/mdl-name-lib.sh` are deleted. This spec
deliberately stops at the standalone integrity tools to keep one focused plan.

## Error handling

- Missing external tool (engine, `magick`, upscaler) → non-zero exit, message
  names the tool and how to get it. Never a faked artifact. The pure-Go
  integrity commands (`check-mdl`, `fix-mdl`, `check-dupes`, `clean-dupes`) need
  no external tool.
- Per-file failures are collected and printed as a summary line at the end; the
  command exits non-zero if any file failed but still processes the rest.
- Report commands (`check-mdl`, `check-dupes`) exit non-zero when they *find*
  problems — that is their contract as CI/pre-import gates, not a tool error.
- Unknown subcommand / bad flags → usage error, exit 64.

## Testing

Go tests in `internal/assets`:

- flag/subcommand parsing and usage errors;
- backend discovery against a fabricated `PATH` (temp dir with stub
  executables) — asserts first-match ordering and the `--backend` override;
- `walk` recursion and extension filtering, including `--non-recursive`;
- one real convert round-trip using `magick` (present in the dev shell):
  encode a small PNG to DDS and back, asserting the pixels are vertically
  flipped after a single conversion and restored after the round trip.
- `internal/assets/mdl`: table-driven fixtures — ASCII vs binary detection,
  each header-token mismatch, the base-node mismatch, and `FixNames` producing
  a model that then passes `CheckNames`. A parity fixture set shared in intent
  with `mdl-scan.awk` so the Go port and the still-present awk agree.
- `check-dupes`/`clean-dupes`: temp trees asserting basename-collision
  detection (case-insensitive), the `--manifests` hak-scoped path against a
  small `assets/*.yml` fixture, the overlap guard, and `--dry-run` mutating
  nothing.

External engine/upscaler calls are exercised through the `run` indirection with
a stub in tests; they are not invoked for real in CI.

`make check` must stay green; add an `assets` expectation to `make smoke` for
the wired exit.