sow-tools/MODEL_COMPILATION_ON_BUILDTIME_CONTRACT.md

# Model Compilation Build-Time Contract

## Status Snapshot

Current state as of 2026-05-13:

- This contract is still outstanding.
- `build-haks` does not currently compile authored ASCII `.mdl` files into
  binary build artifacts.
- Toolkit validation and extraction understand `.mdl`, but there is no
  config-backed build-time model compiler pipeline in `internal/pipeline/`.

Open work remains exactly in the areas this contract describes:

- schema and validation for `models.compile.*`
- compiler/tool resolution and staging
- generated-asset overlay integration in HAK builds
- cache/reuse semantics and regression coverage

## Scope

Expand the toolkit so authored ASCII Neverwinter Nights `.mdl` assets can be compiled into binary `.mdl` build artifacts during `sow-toolkit build-haks` without changing the authored source tree.

This contract covers:

- `toolkit/` as the implementation owner
- `sow-assets/` as the primary consumer repository
- any future consumer repo that packages NWN model assets through the same HAK build pipeline

This contract does not cover:

- replacing authored ASCII `.mdl` files in source control
- introducing a separate standalone shell-only build system as the primary path
- downloading or redistributing copyrighted NWN game data
- building an in-game runtime compiler workflow

## Goal

Make model compilation a first-class, deterministic, configurable build stage in the Go toolkit so generated HAKs contain compiled binary models while authored repositories continue to keep ASCII `.mdl` files as source of truth.

## Why This Exists

Repo-local context already shows that ASCII `.mdl` files are an intentional part of authored asset workflows:

- validation reports ASCII/decompiled models as informational, not erroneous
- topdata autogen manifest generation scans authored `.mdl` paths directly
- part discovery derives IDs from authored `.mdl` filenames
- HAK building already has other generated-asset stages such as NWScript and music conversion

Upstream NWN documentation also supports compiling before release:

- `nwn.wiki` documents that ASCII models are slower because the game compiles them at load time and recommends shipping compiled models
- `nwnmdlcomp` usage docs confirm ASCII-to-binary compilation as a standard flow
- community documentation shows `nwnmdlcomp.exe` expects old NWN registry/data layout and may need a 1.69-era data root

Therefore the correct implementation is not “replace asset scanning with binary models everywhere”. The correct implementation is:

1. keep authored ASCII `.mdl` in `paths.assets`
2. preserve current validation and manifest derivation behavior against source
3. compile only the build payload that becomes generated HAK content

## Ownership Split

- `toolkit/`
  - owns config schema, validation, discovery, cache policy, build staging, and HAK integration
  - owns Wine execution and NWN data preparation logic
  - owns tests and user-facing CLI behavior
- `sow-assets/`
  - owns opting into the feature through `nwn-tool.yaml`
  - owns providing local tool binaries and any fake/minimal NWN data tree it wants to use
  - owns wrapper scripts and CI invocation details
- `sow-module/`
  - is an indirect consumer only when it consumes generated HAKs
  - must not need authored-source changes for this feature to work

## Current Project-Needs Analysis

## Existing architecture

Current HAK packaging is Go-native and centered on:

- `cmd/nwn-tool/main.go`
- `internal/app/app.go`
- `internal/pipeline/build.go`
- `internal/project/*`
- `internal/validator/*`

`build-haks` already does all asset discovery, planning, manifest writing, HAK reuse, and generated-asset preparation inside the toolkit. Adding model compilation as a disconnected shell script would duplicate the existing build graph and create drift.

## Existing relevant behavior

- `project.AssetExtensions` already includes `.mdl`
- `validator.ValidateProject` detects text `.mdl` and reports them via `Report.DecompiledModels`
- `emitValidatorReport` already surfaces that information in CLI output
- `collectAssetResources` packages raw files from `paths.assets` directly into HAK resources
- `BuildHAKs` already supports generated assets and temporary staging through the music pipeline
- script compilation already establishes a precedent for:
  - config-backed tool resolution
  - environment discovery
  - cache directories under `.cache`
  - build-time generated resources that do not modify authored sources

## Constraints the plan must preserve

- authored `.mdl` files remain canonical inputs
- autogen and topdata logic must continue to see source `.mdl` paths
- validation must still be able to detect ASCII source models
- HAK duplicate detection and manifest generation must remain deterministic
- build output reuse must not falsely reuse HAKs built from stale compiled model content
- consumer repos must be able to opt out cleanly

## Architecture Decision

Implement model compilation as a generated-asset overlay inside the existing HAK build pipeline, not as a separate top-level shell build.

That means:

- source scanning still indexes authored `.mdl`
- a model preparation stage produces compiled build artifacts in cache/staging
- `collectAssetResources` substitutes compiled `.mdl` bytes for eligible source models during HAK packaging
- non-model assets continue to flow unchanged
- the written HAK manifest still records the authored relative asset path, not a cache path

This mirrors the existing music pipeline more than the script pipeline:

- source asset exists in `paths.assets`
- generated build artifact lives in cache/staging
- HAK includes the generated bytes under the original logical resource name

## Non-Goals

- no automatic decompilation
- no mutation of files under `paths.assets`
- no requirement to compile every `.mdl` in every repo by default
- no requirement to support Windows-only shell wrappers as the primary interface
- no requirement to support arbitrary model compilers on day one beyond `nwnmdlcomp.exe` via Wine
- no attempt to “fix” malformed source models automatically

## Config Contract

Add a new top-level config section:

```yaml
models:
  compile:
    enabled: true
    source_extensions: [.mdl]
    include: []
    exclude: []
    cache: "{paths.cache}/models"
    compiler:
      path: ""
      env:
        path: SOW_NWN_MODEL_COMPILER
        wine: SOW_NWN_WINE
        winepath: SOW_NWN_WINEPATH
        prefix: SOW_NWN_WINEPREFIX
        nwn_data: [SOW_NWN_MODEL_DATA, SOW_NWN_ROOT, NWN_ROOT]
      search:
        - "{paths.tools}/nwnmdlcomp/nwnmdlcomp.exe"
        - "{paths.tools}/nwnmdlcomp.exe"
    runtime:
      mode: wine
      jobs: 1
      force: false
      setup_registry: true
      compiler_cwd: "{paths.tools}/nwnmdlcomp"
    data:
      root: ""
      require_keys: [chitin.key, nwn_base.key, data/nwn_base.key]
      writable_aliases: [chitin.key, nwn_base.key]
    policy:
      compile_ascii_only: true
      preserve_binary_sources: true
      fail_on_missing_output: true
      fail_on_zero_byte_output: true
      fail_on_warning: false
```

## Config design rules

- `models.compile.enabled` must default to `false`
- the feature must be opt-in
- config must be repository-driven, not hardcoded per consumer
- search/discovery semantics should match `scripts.compiler` where practical
- path templating should reuse the existing effective config path expansion model
- include/exclude filters apply to asset-relative paths under `paths.assets`
- default jobs should be serial unless parallel execution is explicitly enabled

## Environment precedence

Use this precedence for the compiler executable:

1. `models.compile.compiler.path`
2. env var named by `models.compile.compiler.env.path`
3. configured search paths

Use this precedence for NWN data root:

1. `models.compile.data.root`
2. env vars listed in `models.compile.compiler.env.nwn_data`
3. no implicit auto-discovery unless explicitly implemented and documented

Wine tools should resolve as:

1. env override for `wine`
2. `PATH:wine`

and:

1. env override for `winepath`
2. `PATH:winepath`

Do not silently guess a fake NWN data tree.

## Build Contract

## Eligibility rules

A model is eligible for build-time compilation when all of the following are true:

- asset extension is `.mdl`
- `models.compile.enabled` is true
- path passes include/exclude filters
- file is detected as ASCII when `compile_ascii_only` is true

If a `.mdl` file is already binary and `preserve_binary_sources` is true:

- do not recompile it
- package it as-is
- optionally log a verbose skip reason

## Integration point

Insert model preparation into `planOrBuildHAKs` before `collectAssetResources` finalizes HAK resource content, alongside other generated-asset preparation.

Expected flow:

1. validate project
2. prepare music assets if enabled
3. prepare compiled model assets if enabled
4. collect asset resources using source inventory plus prepared overlays
5. plan/write HAKs as today

## Prepared asset representation

Introduce a prepared-model result similar in spirit to `preparedMusicAssets`, for example:

- generated assets keyed by original asset-relative path
- skipped source-relative paths
- summary counts and loggable actions
- cleanup hook for temporary staging if needed

The HAK layer should continue to see:

- resource name from the original source path stem
- resource type `mdl`
- relative manifest path equal to the original source path

Only the file bytes should differ.

## Cache and staging contract

Compiled models are generated artifacts and must live outside the authored asset tree.

Default cache root:

```text
{paths.cache}/models
```

Suggested cache layout:

```text
.cache/models/
  compiled/<asset-relative-path>.mdl
  logs/<asset-relative-path>.log
  metadata.json
  wine-prefix/           # only if the repo chooses a toolkit-managed prefix
  nwn-data-links/        # only if alias files/symlinks are staged
```

The exact layout may differ, but it must support:

- deterministic mapping from source asset path to compiled output path
- per-model logging for failures
- incremental stale detection
- cleanup without touching authored content

## Incremental contract

The plan must support incremental recompilation for persistent cache dirs.

At minimum, recompile when any of these change:

- source file content hash changed
- source file modtime is newer than compiled output
- compiler executable path or content fingerprint changed
- NWN data root fingerprint changed if the implementation tracks it
- relevant config changed
- `--force` or config `force: true` is active
- compiled output is missing or zero bytes

Recommended metadata key:

- source asset relative path
- source SHA-256
- output SHA-256
- compiler path
- compiler binary mtime or hash
- NWN data root path
- build config fingerprint
- compiled timestamp

If metadata is unavailable or invalid, fail open to recompilation, not reuse.

## NWN Data Contract

`nwnmdlcomp.exe` requires a classic NWN data/registry environment.

The implementation must:

- validate the configured NWN data root exists
- require at least one of:
  - `chitin.key`
  - `nwn_base.key`
  - `data/nwn_base.key`
- never download game data
- never synthesize copyrighted content

If only `data/nwn_base.key` exists, the implementation may create a staged alias view for the compiler by:

- symlinking inside a cache-owned staging dir, or
- copying just the key file names into a cache-owned staging dir when symlinks are unavailable

Do not mutate the user’s real install directory unless they explicitly pointed the config at a writable fake data tree and the implementation clearly documents that behavior.

## Wine Registry Contract

When `runtime.setup_registry` is enabled, prepare the Wine registry keys needed by `nwnmdlcomp.exe`:

```text
HKLM\Software\WOW6432Node\BioWare\NWN\Neverwinter
  Location = <wine path>
  Path     = <wine path>
  Version  = 1.69
  GUID     = 00000000-0000-0000-0000-000000000000
  Language = 0
```

Implementation rules:

- convert Linux paths with `winepath -w`
- target the Wine prefix selected by config/env
- make registry setup idempotent
- separate “prepare registry” failure messages from “compile model” failures

The implementation may either:

- manage a dedicated toolkit prefix, or
- use a user-provided existing prefix

The contract should prefer an isolated prefix to avoid polluting unrelated Wine state.

## Compiler Invocation Contract

Primary supported compiler:

- `nwnmdlcomp.exe`

Expected invocation shape:

```bash
WINEDEBUG=-all wine "$COMPILER" -c "$INPUT_WIN" "$OUTPUT_WIN" -e
```

Implementation requirements:

- invoke through `os/exec`, not shell string concatenation
- convert input/output paths with `winepath -w`
- allow compiler working directory configuration if supermodel resolution or relative lookup requires it
- capture combined stdout/stderr per compile attempt

## Supermodel and sibling dependency considerations

The plan must explicitly account for model compilation not being a pure single-file transform.

Potential dependencies include:

- `setsupermodel` references
- sibling helper models in the same asset family
- textures or material references that may not block compilation but do matter for diagnostics

Minimum contract:

- compile against a staging layout that preserves original relative structure
- ensure each model’s directory context exists in staging
- document that some supermodel cases may require staging additional authored `.mdl` siblings into the compiler-visible workspace

Do not assume “compile one file in isolation” is always sufficient.

## Staging strategy

Preferred strategy:

1. keep source inventory rooted in `paths.assets`
2. materialize a compiler workspace under cache that mirrors required relative directories for eligible `.mdl`
3. compile outputs into a sibling compiled tree
4. package compiled output bytes while preserving source-relative logical names

This avoids:

- polluting source trees
- output collisions between models with the same basename in different folders
- broken relative assumptions inside asset packs

## Logging Contract

Progress output should fit existing `ProgressFunc` style.

Minimum messages:

- `Preparing model compiler...`
- `Preparing model compilation workspace...`
- `Compiling model 12/324: part/belt/pfa0_belt018.mdl`
- `Reusing compiled model: vfxs/head_accessories/hfx_bandana.mdl`
- `Skipping binary model source: creatures/foo.mdl`
- `FAILED model compile: placeables/broken_model.mdl`

At summary level, surface:

- models considered
- compiled
- reused
- skipped binary
- skipped filtered
- failed

Verbose/debug output may include:

- chosen compiler path
- chosen Wine/Wineprefix/Winepath tools
- chosen NWN data root
- per-model log file location

## Error Handling Contract

Model compilation is a build-critical stage once enabled.

The build must fail when an eligible model compile:

- exits non-zero
- does not produce the expected output file
- produces a zero-byte output file
- cannot be staged or fingerprinted

Warnings from the compiler do not fail the build unless future config enables that policy.

Failure output must include:

- original source-relative asset path
- compiler command context sufficient for debugging
- log file path if written

At the end of a failed run, include a concise failed-path summary.

## CLI Contract

The canonical user-facing entrypoint remains:

```bash
sow-toolkit build-haks
```

Add targeted flags only if they materially improve operability and match existing CLI patterns. Recommended flags:

- `build-haks --force-models`
- `build-haks --skip-models`
- `build-haks --model <glob-or-prefix>` optional, only if there is a real use case for scoped rebuilds

Flag rules:

- flags override config for the current invocation only
- `--skip-models` must bypass model preparation even if config is enabled
- `--force-models` must invalidate incremental reuse for model outputs only

Do not introduce a parallel standalone command as the only way to use the feature. A helper/diagnostic subcommand is acceptable later, but build-haks must own the real integration.

## Validation Contract

Current validation already reports authored ASCII `.mdl` as informational. That behavior should remain, but the plan should extend validation to cover model compilation readiness when the feature is enabled.

Recommended validation additions:

- compiler path resolves or search candidates exist
- `wine` and `winepath` are resolvable on non-Windows platforms when mode is `wine`
- NWN data root exists and has required keys
- cache path is safe and repo-relative unless explicitly documented otherwise
- include/exclude glob config is well-formed
- jobs is >= 1

Validation severity:

- config/discovery failures that make enabled compilation impossible should be errors
- authored ASCII `.mdl` detection remains info
- binary `.mdl` sources alongside enabled compilation are not errors

## Manifest And Reuse Contract

Generated `haks.json` must continue to describe authored asset-relative paths, not cache file paths.

HAK reuse logic must account for compiled model output bytes. Reuse must not mistakenly treat an unchanged authored path as unchanged content if the compiled artifact changed.

That means one of:

- the existing asset `ContentID` for `.mdl` must reflect compiled output hash when compilation is enabled, or
- chunk hashing must incorporate the prepared compiled output fingerprint

Without this, stale HAK reuse would be incorrect.

## Parallelism Contract

Support configurable parallel compilation with conservative defaults.

Requirements:

- default jobs is `1`
- jobs > 1 must not corrupt outputs or logs
- each compile must write to a unique output path and unique log path
- shared registry/prefix setup must happen before parallel work or under safe synchronization

If `nwnmdlcomp.exe` or Wine proves unstable under parallel execution:

- keep serial default
- document the risk
- allow opting into higher parallelism with clear caveats

## Testing Contract

Implementation is not complete without automated tests. Use fake executables and temporary directories rather than requiring a real Wine/NWN environment in unit tests.

## Unit tests

Add tests for:

- config loading and effective config defaults for `models.compile`
- compiler path/env/search resolution
- Wine tool resolution
- NWN data root validation and alias handling
- ASCII-vs-binary `.mdl` eligibility logic
- include/exclude filtering
- incremental reuse decisions
- metadata invalidation on source/config/compiler changes
- content hashing based on compiled output, not source text alone

## Pipeline tests

Add or extend `internal/pipeline/pipeline_test.go` coverage for:

- HAK build packages compiled `.mdl` bytes instead of source ASCII bytes
- non-`.mdl` assets remain unchanged
- binary source `.mdl` passes through unchanged
- failed model compile aborts build
- missing output aborts build
- zero-byte output aborts build
- `--skip-models` bypasses compilation
- `--force-models` rebuilds compiled outputs
- unchanged compiled output allows HAK reuse
- changed compiled output forces HAK rewrite
- duplicate asset collision rules still behave as before

## Validator tests

Keep and extend tests proving:

- ASCII source models are still reported as informational
- enabled compilation readiness errors surface when config is invalid

## Integration/manual acceptance tests

Document real-world manual checks for a workstation with Wine and `nwnmdlcomp.exe` available.

### Test 1: readiness validation

```bash
sow-toolkit validate
```

Expected:

- no model-compiler readiness errors when config/env are correct
- ASCII `.mdl` info line still appears for source models

### Test 2: single-model HAK packaging

Given an authored ASCII model under `paths.assets`, run:

```bash
sow-toolkit build-haks --force-models
```

Expected:

- build succeeds
- resulting HAK contains a binary `.mdl`
- authored source file remains unchanged

### Test 3: incremental reuse

Run `build-haks` twice without changes.

Expected:

- second run reports model reuse
- unchanged HAKs are reused when manifest/chunk fingerprints match

### Test 4: source change invalidation

Edit one ASCII `.mdl` source and rebuild.

Expected:

- only affected compiled output is rebuilt
- affected HAK chunk is rewritten
- unrelated HAK chunks remain reusable

### Test 5: missing NWN data

Misconfigure `models.compile.data.root` and run build.

Expected:

- validation/build fails clearly before or during preparation
- error names the missing key/data requirement

## Documentation Contract

Update `README.md` to include:

- why compiled models are shipped even though ASCII remains authored source
- the new `models.compile` config section
- required local dependencies:
  - `nwnmdlcomp.exe`
  - `wine`
  - `winepath`
  - NWN data root
- environment override behavior
- cache/staging behavior
- `build-haks` flags related to model compilation
- limitations and troubleshooting notes

Optional but recommended:

- `tools/nwnmdlcomp/README.md` for consumer repo operators

## Suggested Implementation Breakdown

1. Schema and config foundation
   - add `models.compile` config structs to `internal/project/project.go`
   - add effective config defaults and path expansion in `internal/project/effective.go`
   - add project helper methods for compiler/cache/env resolution

2. Validation foundation
   - extend project/config validation for enabled compilation readiness
   - extend CLI validation reporting where needed

3. Preparation layer
   - add model compiler resolution, Wine resolution, NWN data staging, registry setup, and incremental metadata logic
   - add a `preparedModelAssets` representation

4. Pipeline integration
   - plug model preparation into `planOrBuildHAKs`
   - teach `collectAssetResources` to substitute compiled model payloads
   - ensure `ContentID` and HAK reuse reflect compiled output

5. CLI polish
   - add build-haks flags if approved
   - add progress messages and summary output

6. Tests
   - config tests
   - validator tests
   - pipeline tests with fake compiler/Wine shims

7. Docs
   - repo README updates
   - optional operator README under `tools/`

## Open Design Decisions To Resolve Before Coding

These should be decided explicitly during implementation, not left implicit:

1. Should the toolkit own a dedicated Wine prefix under cache by default, or require a user-provided prefix?
2. Should NWN data alias files be created in a separate staging tree, or should the configured data root be used directly when already compatible?
3. Is parallel model compilation stable enough to expose beyond best-effort?
4. Do we want dedicated `build-haks` flags now, or only config for the first implementation?
5. Should per-model logs always be persisted, or only on failure / debug mode?

## Acceptance Criteria

The feature is complete when all of the following are true:

- `sow-toolkit build-haks` can package compiled binary `.mdl` content from authored ASCII `.mdl` source when `models.compile.enabled: true`
- authored `.mdl` files remain unchanged on disk
- validation and topdata/autogen behaviors that depend on authored `.mdl` source still work
- generated HAK manifests still reference authored asset-relative paths
- HAK reuse logic correctly tracks compiled output changes
- the feature is opt-in, deterministic, and tested
- failures are actionable and name the offending model paths
- README/config docs are updated

## Relevant References

- Repo code:
  - `internal/pipeline/build.go`
  - `internal/validator/validator.go`
  - `internal/topdata/autogen.go`
  - `internal/topdata/parts_discovery.go`
  - `README.md`
- Upstream docs consulted for this contract:
  - `nwn.wiki` MDL page: compiled models are recommended for release because ASCII loads more slowly
  - Neverwinter Vault `nwnmdlcomp` page: standard compile/decompile CLI usage
  - Neverwinter Vault forum discussions: `nwnmdlcomp` expects old NWN registry keys/data layout and some supermodel cases need additional care