sow-tools/MODEL_COMPILATION_ON_BUILDTIME_CONTRACT.md

Model Compilation Build-Time Contract

Scope

Expand the toolkit so authored ASCII Neverwinter Nights .mdl assets can be compiled into binary .mdl build artifacts during sow-toolkit build-haks without changing the authored source tree.

This contract covers:

• toolkit/ as the implementation owner
• sow-assets/ as the primary consumer repository
• any future consumer repo that packages NWN model assets through the same HAK build pipeline

This contract does not cover:

• replacing authored ASCII .mdl files in source control
• introducing a separate standalone shell-only build system as the primary path
• downloading or redistributing copyrighted NWN game data
• building an in-game runtime compiler workflow

Goal

Make model compilation a first-class, deterministic, configurable build stage in the Go toolkit so generated HAKs contain compiled binary models while authored repositories continue to keep ASCII .mdl files as source of truth.

Why This Exists

Repo-local context already shows that ASCII .mdl files are an intentional part of authored asset workflows:

• validation reports ASCII/decompiled models as informational, not erroneous
• topdata autogen manifest generation scans authored .mdl paths directly
• part discovery derives IDs from authored .mdl filenames
• HAK building already has other generated-asset stages such as NWScript and music conversion

Upstream NWN documentation also supports compiling before release:

• nwn.wiki documents that ASCII models are slower because the game compiles them at load time and recommends shipping compiled models
• nwnmdlcomp usage docs confirm ASCII-to-binary compilation as a standard flow
• community documentation shows nwnmdlcomp.exe expects old NWN registry/data layout and may need a 1.69-era data root

Therefore the correct implementation is not “replace asset scanning with binary models everywhere”. The correct implementation is:

1. keep authored ASCII .mdl in paths.assets
2. preserve current validation and manifest derivation behavior against source
3. compile only the build payload that becomes generated HAK content

Ownership Split

• toolkit/
  • owns config schema, validation, discovery, cache policy, build staging, and HAK integration
  • owns Wine execution and NWN data preparation logic
  • owns tests and user-facing CLI behavior
• sow-assets/
  • owns opting into the feature through nwn-tool.yaml
  • owns providing local tool binaries and any fake/minimal NWN data tree it wants to use
  • owns wrapper scripts and CI invocation details
• sow-module/
  • is an indirect consumer only when it consumes generated HAKs
  • must not need authored-source changes for this feature to work

Current Project-Needs Analysis

Existing architecture

Current HAK packaging is Go-native and centered on:

• cmd/nwn-tool/main.go
• internal/app/app.go
• internal/pipeline/build.go
• internal/project/*
• internal/validator/*

build-haks already does all asset discovery, planning, manifest writing, HAK reuse, and generated-asset preparation inside the toolkit. Adding model compilation as a disconnected shell script would duplicate the existing build graph and create drift.

Existing relevant behavior

• project.AssetExtensions already includes .mdl
• validator.ValidateProject detects text .mdl and reports them via Report.DecompiledModels
• emitValidatorReport already surfaces that information in CLI output
• collectAssetResources packages raw files from paths.assets directly into HAK resources
• BuildHAKs already supports generated assets and temporary staging through the music pipeline
• script compilation already establishes a precedent for:
  • config-backed tool resolution
  • environment discovery
  • cache directories under .cache
  • build-time generated resources that do not modify authored sources

Constraints the plan must preserve

• authored .mdl files remain canonical inputs
• autogen and topdata logic must continue to see source .mdl paths
• validation must still be able to detect ASCII source models
• HAK duplicate detection and manifest generation must remain deterministic
• build output reuse must not falsely reuse HAKs built from stale compiled model content
• consumer repos must be able to opt out cleanly

Architecture Decision

Implement model compilation as a generated-asset overlay inside the existing HAK build pipeline, not as a separate top-level shell build.

That means:

• source scanning still indexes authored .mdl
• a model preparation stage produces compiled build artifacts in cache/staging
• collectAssetResources substitutes compiled .mdl bytes for eligible source models during HAK packaging
• non-model assets continue to flow unchanged
• the written HAK manifest still records the authored relative asset path, not a cache path

This mirrors the existing music pipeline more than the script pipeline:

• source asset exists in paths.assets
• generated build artifact lives in cache/staging
• HAK includes the generated bytes under the original logical resource name

Non-Goals

• no automatic decompilation
• no mutation of files under paths.assets
• no requirement to compile every .mdl in every repo by default
• no requirement to support Windows-only shell wrappers as the primary interface
• no requirement to support arbitrary model compilers on day one beyond nwnmdlcomp.exe via Wine
• no attempt to “fix” malformed source models automatically

Config Contract

Add a new top-level config section:

models:
  compile:
    enabled: true
    source_extensions: [.mdl]
    include: []
    exclude: []
    cache: "{paths.cache}/models"
    compiler:
      path: ""
      env:
        path: SOW_NWN_MODEL_COMPILER
        wine: SOW_NWN_WINE
        winepath: SOW_NWN_WINEPATH
        prefix: SOW_NWN_WINEPREFIX
        nwn_data: [SOW_NWN_MODEL_DATA, SOW_NWN_ROOT, NWN_ROOT]
      search:
        - "{paths.tools}/nwnmdlcomp/nwnmdlcomp.exe"
        - "{paths.tools}/nwnmdlcomp.exe"
    runtime:
      mode: wine
      jobs: 1
      force: false
      setup_registry: true
      compiler_cwd: "{paths.tools}/nwnmdlcomp"
    data:
      root: ""
      require_keys: [chitin.key, nwn_base.key, data/nwn_base.key]
      writable_aliases: [chitin.key, nwn_base.key]
    policy:
      compile_ascii_only: true
      preserve_binary_sources: true
      fail_on_missing_output: true
      fail_on_zero_byte_output: true
      fail_on_warning: false

Config design rules

• models.compile.enabled must default to false
• the feature must be opt-in
• config must be repository-driven, not hardcoded per consumer
• search/discovery semantics should match scripts.compiler where practical
• path templating should reuse the existing effective config path expansion model
• include/exclude filters apply to asset-relative paths under paths.assets
• default jobs should be serial unless parallel execution is explicitly enabled

Environment precedence

Use this precedence for the compiler executable:

1. models.compile.compiler.path
2. env var named by models.compile.compiler.env.path
3. configured search paths

Use this precedence for NWN data root:

1. models.compile.data.root
2. env vars listed in models.compile.compiler.env.nwn_data
3. no implicit auto-discovery unless explicitly implemented and documented

Wine tools should resolve as:

1. env override for wine
2. PATH:wine

and:

1. env override for winepath
2. PATH:winepath

Do not silently guess a fake NWN data tree.

Build Contract

Eligibility rules

A model is eligible for build-time compilation when all of the following are true:

• asset extension is .mdl
• models.compile.enabled is true
• path passes include/exclude filters
• file is detected as ASCII when compile_ascii_only is true

If a .mdl file is already binary and preserve_binary_sources is true:

• do not recompile it
• package it as-is
• optionally log a verbose skip reason

Integration point

Insert model preparation into planOrBuildHAKs before collectAssetResources finalizes HAK resource content, alongside other generated-asset preparation.

Expected flow:

1. validate project
2. prepare music assets if enabled
3. prepare compiled model assets if enabled
4. collect asset resources using source inventory plus prepared overlays
5. plan/write HAKs as today

Prepared asset representation

Introduce a prepared-model result similar in spirit to preparedMusicAssets, for example:

• generated assets keyed by original asset-relative path
• skipped source-relative paths
• summary counts and loggable actions
• cleanup hook for temporary staging if needed

The HAK layer should continue to see:

• resource name from the original source path stem
• resource type mdl
• relative manifest path equal to the original source path

Only the file bytes should differ.

Cache and staging contract

Compiled models are generated artifacts and must live outside the authored asset tree.

Default cache root:

{paths.cache}/models

Suggested cache layout:

.cache/models/
  compiled/<asset-relative-path>.mdl
  logs/<asset-relative-path>.log
  metadata.json
  wine-prefix/           # only if the repo chooses a toolkit-managed prefix
  nwn-data-links/        # only if alias files/symlinks are staged

The exact layout may differ, but it must support:

• deterministic mapping from source asset path to compiled output path
• per-model logging for failures
• incremental stale detection
• cleanup without touching authored content

Incremental contract

The plan must support incremental recompilation for persistent cache dirs.

At minimum, recompile when any of these change:

• source file content hash changed
• source file modtime is newer than compiled output
• compiler executable path or content fingerprint changed
• NWN data root fingerprint changed if the implementation tracks it
• relevant config changed
• --force or config force: true is active
• compiled output is missing or zero bytes

Recommended metadata key:

• source asset relative path
• source SHA-256
• output SHA-256
• compiler path
• compiler binary mtime or hash
• NWN data root path
• build config fingerprint
• compiled timestamp

If metadata is unavailable or invalid, fail open to recompilation, not reuse.

NWN Data Contract

nwnmdlcomp.exe requires a classic NWN data/registry environment.

The implementation must:

• validate the configured NWN data root exists
• require at least one of:
  • chitin.key
  • nwn_base.key
  • data/nwn_base.key
• never download game data
• never synthesize copyrighted content

If only data/nwn_base.key exists, the implementation may create a staged alias view for the compiler by:

• symlinking inside a cache-owned staging dir, or
• copying just the key file names into a cache-owned staging dir when symlinks are unavailable

Do not mutate the user’s real install directory unless they explicitly pointed the config at a writable fake data tree and the implementation clearly documents that behavior.

Wine Registry Contract

When runtime.setup_registry is enabled, prepare the Wine registry keys needed by nwnmdlcomp.exe:

HKLM\Software\WOW6432Node\BioWare\NWN\Neverwinter
  Location = <wine path>
  Path     = <wine path>
  Version  = 1.69
  GUID     = 00000000-0000-0000-0000-000000000000
  Language = 0

Implementation rules:

• convert Linux paths with winepath -w
• target the Wine prefix selected by config/env
• make registry setup idempotent
• separate “prepare registry” failure messages from “compile model” failures

The implementation may either:

• manage a dedicated toolkit prefix, or
• use a user-provided existing prefix

The contract should prefer an isolated prefix to avoid polluting unrelated Wine state.

Compiler Invocation Contract

Primary supported compiler:

• nwnmdlcomp.exe

Expected invocation shape:

WINEDEBUG=-all wine "$COMPILER" -c "$INPUT_WIN" "$OUTPUT_WIN" -e

Implementation requirements:

• invoke through os/exec, not shell string concatenation
• convert input/output paths with winepath -w
• allow compiler working directory configuration if supermodel resolution or relative lookup requires it
• capture combined stdout/stderr per compile attempt

Supermodel and sibling dependency considerations

The plan must explicitly account for model compilation not being a pure single-file transform.

Potential dependencies include:

• setsupermodel references
• sibling helper models in the same asset family
• textures or material references that may not block compilation but do matter for diagnostics

Minimum contract:

• compile against a staging layout that preserves original relative structure
• ensure each model’s directory context exists in staging
• document that some supermodel cases may require staging additional authored .mdl siblings into the compiler-visible workspace

Do not assume “compile one file in isolation” is always sufficient.

Staging strategy

Preferred strategy:

1. keep source inventory rooted in paths.assets
2. materialize a compiler workspace under cache that mirrors required relative directories for eligible .mdl
3. compile outputs into a sibling compiled tree
4. package compiled output bytes while preserving source-relative logical names

This avoids:

• polluting source trees
• output collisions between models with the same basename in different folders
• broken relative assumptions inside asset packs

Logging Contract

Progress output should fit existing ProgressFunc style.

Minimum messages:

• Preparing model compiler...
• Preparing model compilation workspace...
• Compiling model 12/324: part/belt/pfa0_belt018.mdl
• Reusing compiled model: vfxs/head_accessories/hfx_bandana.mdl
• Skipping binary model source: creatures/foo.mdl
• FAILED model compile: placeables/broken_model.mdl

At summary level, surface:

• models considered
• compiled
• reused
• skipped binary
• skipped filtered
• failed

Verbose/debug output may include:

• chosen compiler path
• chosen Wine/Wineprefix/Winepath tools
• chosen NWN data root
• per-model log file location

Error Handling Contract

Model compilation is a build-critical stage once enabled.

The build must fail when an eligible model compile:

• exits non-zero
• does not produce the expected output file
• produces a zero-byte output file
• cannot be staged or fingerprinted

Warnings from the compiler do not fail the build unless future config enables that policy.

Failure output must include:

• original source-relative asset path
• compiler command context sufficient for debugging
• log file path if written

At the end of a failed run, include a concise failed-path summary.

CLI Contract

The canonical user-facing entrypoint remains:

sow-toolkit build-haks

Add targeted flags only if they materially improve operability and match existing CLI patterns. Recommended flags:

• build-haks --force-models
• build-haks --skip-models
• build-haks --model <glob-or-prefix> optional, only if there is a real use case for scoped rebuilds

Flag rules:

• flags override config for the current invocation only
• --skip-models must bypass model preparation even if config is enabled
• --force-models must invalidate incremental reuse for model outputs only

Do not introduce a parallel standalone command as the only way to use the feature. A helper/diagnostic subcommand is acceptable later, but build-haks must own the real integration.

Validation Contract

Current validation already reports authored ASCII .mdl as informational. That behavior should remain, but the plan should extend validation to cover model compilation readiness when the feature is enabled.

Recommended validation additions:

• compiler path resolves or search candidates exist
• wine and winepath are resolvable on non-Windows platforms when mode is wine
• NWN data root exists and has required keys
• cache path is safe and repo-relative unless explicitly documented otherwise
• include/exclude glob config is well-formed
• jobs is >= 1

Validation severity:

• config/discovery failures that make enabled compilation impossible should be errors
• authored ASCII .mdl detection remains info
• binary .mdl sources alongside enabled compilation are not errors

Manifest And Reuse Contract

Generated haks.json must continue to describe authored asset-relative paths, not cache file paths.

HAK reuse logic must account for compiled model output bytes. Reuse must not mistakenly treat an unchanged authored path as unchanged content if the compiled artifact changed.

That means one of:

• the existing asset ContentID for .mdl must reflect compiled output hash when compilation is enabled, or
• chunk hashing must incorporate the prepared compiled output fingerprint

Without this, stale HAK reuse would be incorrect.

Parallelism Contract

Support configurable parallel compilation with conservative defaults.

Requirements:

• default jobs is 1
• jobs > 1 must not corrupt outputs or logs
• each compile must write to a unique output path and unique log path
• shared registry/prefix setup must happen before parallel work or under safe synchronization

If nwnmdlcomp.exe or Wine proves unstable under parallel execution:

• keep serial default
• document the risk
• allow opting into higher parallelism with clear caveats

Testing Contract

Implementation is not complete without automated tests. Use fake executables and temporary directories rather than requiring a real Wine/NWN environment in unit tests.

Unit tests

Add tests for:

• config loading and effective config defaults for models.compile
• compiler path/env/search resolution
• Wine tool resolution
• NWN data root validation and alias handling
• ASCII-vs-binary .mdl eligibility logic
• include/exclude filtering
• incremental reuse decisions
• metadata invalidation on source/config/compiler changes
• content hashing based on compiled output, not source text alone

Pipeline tests

Add or extend internal/pipeline/pipeline_test.go coverage for:

• HAK build packages compiled .mdl bytes instead of source ASCII bytes
• non-.mdl assets remain unchanged
• binary source .mdl passes through unchanged
• failed model compile aborts build
• missing output aborts build
• zero-byte output aborts build
• --skip-models bypasses compilation
• --force-models rebuilds compiled outputs
• unchanged compiled output allows HAK reuse
• changed compiled output forces HAK rewrite
• duplicate asset collision rules still behave as before

Validator tests

Keep and extend tests proving:

• ASCII source models are still reported as informational
• enabled compilation readiness errors surface when config is invalid

Integration/manual acceptance tests

Document real-world manual checks for a workstation with Wine and nwnmdlcomp.exe available.

Test 1: readiness validation

sow-toolkit validate

Expected:

• no model-compiler readiness errors when config/env are correct
• ASCII .mdl info line still appears for source models

Test 2: single-model HAK packaging

Given an authored ASCII model under paths.assets, run:

sow-toolkit build-haks --force-models

Expected:

• build succeeds
• resulting HAK contains a binary .mdl
• authored source file remains unchanged

Test 3: incremental reuse

Run build-haks twice without changes.

Expected:

• second run reports model reuse
• unchanged HAKs are reused when manifest/chunk fingerprints match

Test 4: source change invalidation

Edit one ASCII .mdl source and rebuild.

Expected:

• only affected compiled output is rebuilt
• affected HAK chunk is rewritten
• unrelated HAK chunks remain reusable

Test 5: missing NWN data

Misconfigure models.compile.data.root and run build.

Expected:

• validation/build fails clearly before or during preparation
• error names the missing key/data requirement

Documentation Contract

Update README.md to include:

• why compiled models are shipped even though ASCII remains authored source
• the new models.compile config section
• required local dependencies:
  • nwnmdlcomp.exe
  • wine
  • winepath
  • NWN data root
• environment override behavior
• cache/staging behavior
• build-haks flags related to model compilation
• limitations and troubleshooting notes

Optional but recommended:

• tools/nwnmdlcomp/README.md for consumer repo operators

Suggested Implementation Breakdown

1. Schema and config foundation

   • add models.compile config structs to internal/project/project.go
   • add effective config defaults and path expansion in internal/project/effective.go
   • add project helper methods for compiler/cache/env resolution

2. Validation foundation

   • extend project/config validation for enabled compilation readiness
   • extend CLI validation reporting where needed

3. Preparation layer

   • add model compiler resolution, Wine resolution, NWN data staging, registry setup, and incremental metadata logic
   • add a preparedModelAssets representation

4. Pipeline integration

   • plug model preparation into planOrBuildHAKs
   • teach collectAssetResources to substitute compiled model payloads
   • ensure ContentID and HAK reuse reflect compiled output

5. CLI polish

   • add build-haks flags if approved
   • add progress messages and summary output

6. Tests

   • config tests
   • validator tests
   • pipeline tests with fake compiler/Wine shims

7. Docs

   • repo README updates
   • optional operator README under tools/

Open Design Decisions To Resolve Before Coding

These should be decided explicitly during implementation, not left implicit:

1. Should the toolkit own a dedicated Wine prefix under cache by default, or require a user-provided prefix?
2. Should NWN data alias files be created in a separate staging tree, or should the configured data root be used directly when already compatible?
3. Is parallel model compilation stable enough to expose beyond best-effort?
4. Do we want dedicated build-haks flags now, or only config for the first implementation?
5. Should per-model logs always be persisted, or only on failure / debug mode?

Acceptance Criteria

The feature is complete when all of the following are true:

• sow-toolkit build-haks can package compiled binary .mdl content from authored ASCII .mdl source when models.compile.enabled: true
• authored .mdl files remain unchanged on disk
• validation and topdata/autogen behaviors that depend on authored .mdl source still work
• generated HAK manifests still reference authored asset-relative paths
• HAK reuse logic correctly tracks compiled output changes
• the feature is opt-in, deterministic, and tested
• failures are actionable and name the offending model paths
• README/config docs are updated

Relevant References

• Repo code:
  • internal/pipeline/build.go
  • internal/validator/validator.go
  • internal/topdata/autogen.go
  • internal/topdata/parts_discovery.go
  • README.md
• Upstream docs consulted for this contract:
  • nwn.wiki MDL page: compiled models are recommended for release because ASCII loads more slowly
  • Neverwinter Vault nwnmdlcomp page: standard compile/decompile CLI usage
  • Neverwinter Vault forum discussions: nwnmdlcomp expects old NWN registry keys/data layout and some supermodel cases need additional care