ShadowsOverWestgate/sow-tools
7
0
Fork 0
Code Issues 2 Pull Requests Actions Packages Projects Releases 38 Activity

Model compilation contract

Browse Source
This commit is contained in:
archvillainette
2026-05-08 22:29:46 +02:00
parent c7fd63a80c
commit dee53e8eb4
1 changed files with 770 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
MODEL_COMPILATION_ON_BUILDTIME_CONTRACT.md
+770
View File
@@ -0,0 +1,770 @@
# Model Compilation Build-Time Contract
## Scope
Expand the toolkit so authored ASCII Neverwinter Nights `.mdl` assets can be
compiled into binary `.mdl` build artifacts during `sow-toolkit build-haks`
without changing the authored source tree.
This contract covers:
- `toolkit/` as the implementation owner
- `sow-assets/` as the primary consumer repository
- any future consumer repo that packages NWN model assets through the same HAK
build pipeline
This contract does not cover:
- replacing authored ASCII `.mdl` files in source control
- introducing a separate standalone shell-only build system as the primary path
- downloading or redistributing copyrighted NWN game data
- building an in-game runtime compiler workflow
## Goal
Make model compilation a first-class, deterministic, configurable build stage in
the Go toolkit so generated HAKs contain compiled binary models while authored
repositories continue to keep ASCII `.mdl` files as source of truth.
## Why This Exists
Repo-local context already shows that ASCII `.mdl` files are an intentional part
of authored asset workflows:
- validation reports ASCII/decompiled models as informational, not erroneous
- topdata autogen manifest generation scans authored `.mdl` paths directly
- part discovery derives IDs from authored `.mdl` filenames
- HAK building already has other generated-asset stages such as NWScript and
music conversion
Upstream NWN documentation also supports compiling before release:
- `nwn.wiki` documents that ASCII models are slower because the game compiles
them at load time and recommends shipping compiled models
- `nwnmdlcomp` usage docs confirm ASCII-to-binary compilation as a standard flow
- community documentation shows `nwnmdlcomp.exe` expects old NWN registry/data
layout and may need a 1.69-era data root
Therefore the correct implementation is not “replace asset scanning with binary
models everywhere”. The correct implementation is:
1. keep authored ASCII `.mdl` in `paths.assets`
2. preserve current validation and manifest derivation behavior against source
3. compile only the build payload that becomes generated HAK content
## Ownership Split
- `toolkit/`
- owns config schema, validation, discovery, cache policy, build staging, and
HAK integration
- owns Wine execution and NWN data preparation logic
- owns tests and user-facing CLI behavior
- `sow-assets/`
- owns opting into the feature through `nwn-tool.yaml`
- owns providing local tool binaries and any fake/minimal NWN data tree it
wants to use
- owns wrapper scripts and CI invocation details
- `sow-module/`
- is an indirect consumer only when it consumes generated HAKs
- must not need authored-source changes for this feature to work
## Current Project-Needs Analysis
## Existing architecture
Current HAK packaging is Go-native and centered on:
- `cmd/nwn-tool/main.go`
- `internal/app/app.go`
- `internal/pipeline/build.go`
- `internal/project/*`
- `internal/validator/*`
`build-haks` already does all asset discovery, planning, manifest writing, HAK
reuse, and generated-asset preparation inside the toolkit. Adding model
compilation as a disconnected shell script would duplicate the existing build
graph and create drift.
## Existing relevant behavior
- `project.AssetExtensions` already includes `.mdl`
- `validator.ValidateProject` detects text `.mdl` and reports them via
`Report.DecompiledModels`
- `emitValidatorReport` already surfaces that information in CLI output
- `collectAssetResources` packages raw files from `paths.assets` directly into
HAK resources
- `BuildHAKs` already supports generated assets and temporary staging through the
music pipeline
- script compilation already establishes a precedent for:
- config-backed tool resolution
- environment discovery
- cache directories under `.cache`
- build-time generated resources that do not modify authored sources
## Constraints the plan must preserve
- authored `.mdl` files remain canonical inputs
- autogen and topdata logic must continue to see source `.mdl` paths
- validation must still be able to detect ASCII source models
- HAK duplicate detection and manifest generation must remain deterministic
- build output reuse must not falsely reuse HAKs built from stale compiled model
content
- consumer repos must be able to opt out cleanly
## Architecture Decision
Implement model compilation as a generated-asset overlay inside the existing HAK
build pipeline, not as a separate top-level shell build.
That means:
- source scanning still indexes authored `.mdl`
- a model preparation stage produces compiled build artifacts in cache/staging
- `collectAssetResources` substitutes compiled `.mdl` bytes for eligible source
models during HAK packaging
- non-model assets continue to flow unchanged
- the written HAK manifest still records the authored relative asset path, not a
cache path
This mirrors the existing music pipeline more than the script pipeline:
- source asset exists in `paths.assets`
- generated build artifact lives in cache/staging
- HAK includes the generated bytes under the original logical resource name
## Non-Goals
- no automatic decompilation
- no mutation of files under `paths.assets`
- no requirement to compile every `.mdl` in every repo by default
- no requirement to support Windows-only shell wrappers as the primary interface
- no requirement to support arbitrary model compilers on day one beyond
`nwnmdlcomp.exe` via Wine
- no attempt to “fix” malformed source models automatically
## Config Contract
Add a new top-level config section:
```yaml
models:
compile:
enabled: true
source_extensions: [.mdl]
include: []
exclude: []
cache: "{paths.cache}/models"
compiler:
path: ""
env:
path: SOW_NWN_MODEL_COMPILER
wine: SOW_NWN_WINE
winepath: SOW_NWN_WINEPATH
prefix: SOW_NWN_WINEPREFIX
nwn_data: [SOW_NWN_MODEL_DATA, SOW_NWN_ROOT, NWN_ROOT]
search:
- "{paths.tools}/nwnmdlcomp/nwnmdlcomp.exe"
- "{paths.tools}/nwnmdlcomp.exe"
runtime:
mode: wine
jobs: 1
force: false
setup_registry: true
compiler_cwd: "{paths.tools}/nwnmdlcomp"
data:
root: ""
require_keys: [chitin.key, nwn_base.key, data/nwn_base.key]
writable_aliases: [chitin.key, nwn_base.key]
policy:
compile_ascii_only: true
preserve_binary_sources: true
fail_on_missing_output: true
fail_on_zero_byte_output: true
fail_on_warning: false
```
## Config design rules
- `models.compile.enabled` must default to `false`
- the feature must be opt-in
- config must be repository-driven, not hardcoded per consumer
- search/discovery semantics should match `scripts.compiler` where practical
- path templating should reuse the existing effective config path expansion model
- include/exclude filters apply to asset-relative paths under `paths.assets`
- default jobs should be serial unless parallel execution is explicitly enabled
## Environment precedence
Use this precedence for the compiler executable:
1. `models.compile.compiler.path`
2. env var named by `models.compile.compiler.env.path`
3. configured search paths
Use this precedence for NWN data root:
1. `models.compile.data.root`
2. env vars listed in `models.compile.compiler.env.nwn_data`
3. no implicit auto-discovery unless explicitly implemented and documented
Wine tools should resolve as:
1. env override for `wine`
2. `PATH:wine`
and:
1. env override for `winepath`
2. `PATH:winepath`
Do not silently guess a fake NWN data tree.
## Build Contract
## Eligibility rules
A model is eligible for build-time compilation when all of the following are
true:
- asset extension is `.mdl`
- `models.compile.enabled` is true
- path passes include/exclude filters
- file is detected as ASCII when `compile_ascii_only` is true
If a `.mdl` file is already binary and `preserve_binary_sources` is true:
- do not recompile it
- package it as-is
- optionally log a verbose skip reason
## Integration point
Insert model preparation into `planOrBuildHAKs` before `collectAssetResources`
finalizes HAK resource content, alongside other generated-asset preparation.
Expected flow:
1. validate project
2. prepare music assets if enabled
3. prepare compiled model assets if enabled
4. collect asset resources using source inventory plus prepared overlays
5. plan/write HAKs as today
## Prepared asset representation
Introduce a prepared-model result similar in spirit to `preparedMusicAssets`,
for example:
- generated assets keyed by original asset-relative path
- skipped source-relative paths
- summary counts and loggable actions
- cleanup hook for temporary staging if needed
The HAK layer should continue to see:
- resource name from the original source path stem
- resource type `mdl`
- relative manifest path equal to the original source path
Only the file bytes should differ.
## Cache and staging contract
Compiled models are generated artifacts and must live outside the authored asset
tree.
Default cache root:
```text
{paths.cache}/models
```
Suggested cache layout:
```text
.cache/models/
compiled/<asset-relative-path>.mdl
logs/<asset-relative-path>.log
metadata.json
wine-prefix/ # only if the repo chooses a toolkit-managed prefix
nwn-data-links/ # only if alias files/symlinks are staged
```
The exact layout may differ, but it must support:
- deterministic mapping from source asset path to compiled output path
- per-model logging for failures
- incremental stale detection
- cleanup without touching authored content
## Incremental contract
The plan must support incremental recompilation for persistent cache dirs.
At minimum, recompile when any of these change:
- source file content hash changed
- source file modtime is newer than compiled output
- compiler executable path or content fingerprint changed
- NWN data root fingerprint changed if the implementation tracks it
- relevant config changed
- `--force` or config `force: true` is active
- compiled output is missing or zero bytes
Recommended metadata key:
- source asset relative path
- source SHA-256
- output SHA-256
- compiler path
- compiler binary mtime or hash
- NWN data root path
- build config fingerprint
- compiled timestamp
If metadata is unavailable or invalid, fail open to recompilation, not reuse.
## NWN Data Contract
`nwnmdlcomp.exe` requires a classic NWN data/registry environment.
The implementation must:
- validate the configured NWN data root exists
- require at least one of:
- `chitin.key`
- `nwn_base.key`
- `data/nwn_base.key`
- never download game data
- never synthesize copyrighted content
If only `data/nwn_base.key` exists, the implementation may create a staged alias
view for the compiler by:
- symlinking inside a cache-owned staging dir, or
- copying just the key file names into a cache-owned staging dir when symlinks
are unavailable
Do not mutate the user’s real install directory unless they explicitly pointed
the config at a writable fake data tree and the implementation clearly documents
that behavior.
## Wine Registry Contract
When `runtime.setup_registry` is enabled, prepare the Wine registry keys needed
by `nwnmdlcomp.exe`:
```text
HKLM\Software\WOW6432Node\BioWare\NWN\Neverwinter
Location = <wine path>
Path = <wine path>
Version = 1.69
GUID = 00000000-0000-0000-0000-000000000000
Language = 0
```
Implementation rules:
- convert Linux paths with `winepath -w`
- target the Wine prefix selected by config/env
- make registry setup idempotent
- separate “prepare registry” failure messages from “compile model” failures
The implementation may either:
- manage a dedicated toolkit prefix, or
- use a user-provided existing prefix
The contract should prefer an isolated prefix to avoid polluting unrelated Wine
state.
## Compiler Invocation Contract
Primary supported compiler:
- `nwnmdlcomp.exe`
Expected invocation shape:
```bash
WINEDEBUG=-all wine "$COMPILER" -c "$INPUT_WIN" "$OUTPUT_WIN" -e
```
Implementation requirements:
- invoke through `os/exec`, not shell string concatenation
- convert input/output paths with `winepath -w`
- allow compiler working directory configuration if supermodel resolution or
relative lookup requires it
- capture combined stdout/stderr per compile attempt
## Supermodel and sibling dependency considerations
The plan must explicitly account for model compilation not being a pure
single-file transform.
Potential dependencies include:
- `setsupermodel` references
- sibling helper models in the same asset family
- textures or material references that may not block compilation but do matter
for diagnostics
Minimum contract:
- compile against a staging layout that preserves original relative structure
- ensure each model’s directory context exists in staging
- document that some supermodel cases may require staging additional authored
`.mdl` siblings into the compiler-visible workspace
Do not assume “compile one file in isolation” is always sufficient.
## Staging strategy
Preferred strategy:
1. keep source inventory rooted in `paths.assets`
2. materialize a compiler workspace under cache that mirrors required relative
directories for eligible `.mdl`
3. compile outputs into a sibling compiled tree
4. package compiled output bytes while preserving source-relative logical names
This avoids:
- polluting source trees
- output collisions between models with the same basename in different folders
- broken relative assumptions inside asset packs
## Logging Contract
Progress output should fit existing `ProgressFunc` style.
Minimum messages:
- `Preparing model compiler...`
- `Preparing model compilation workspace...`
- `Compiling model 12/324: part/belt/pfa0_belt018.mdl`
- `Reusing compiled model: vfxs/head_accessories/hfx_bandana.mdl`
- `Skipping binary model source: creatures/foo.mdl`
- `FAILED model compile: placeables/broken_model.mdl`
At summary level, surface:
- models considered
- compiled
- reused
- skipped binary
- skipped filtered
- failed
Verbose/debug output may include:
- chosen compiler path
- chosen Wine/Wineprefix/Winepath tools
- chosen NWN data root
- per-model log file location
## Error Handling Contract
Model compilation is a build-critical stage once enabled.
The build must fail when an eligible model compile:
- exits non-zero
- does not produce the expected output file
- produces a zero-byte output file
- cannot be staged or fingerprinted
Warnings from the compiler do not fail the build unless future config enables
that policy.
Failure output must include:
- original source-relative asset path
- compiler command context sufficient for debugging
- log file path if written
At the end of a failed run, include a concise failed-path summary.
## CLI Contract
The canonical user-facing entrypoint remains:
```bash
sow-toolkit build-haks
```
Add targeted flags only if they materially improve operability and match
existing CLI patterns. Recommended flags:
- `build-haks --force-models`
- `build-haks --skip-models`
- `build-haks --model <glob-or-prefix>` optional, only if there is a real use
case for scoped rebuilds
Flag rules:
- flags override config for the current invocation only
- `--skip-models` must bypass model preparation even if config is enabled
- `--force-models` must invalidate incremental reuse for model outputs only
Do not introduce a parallel standalone command as the only way to use the
feature. A helper/diagnostic subcommand is acceptable later, but build-haks must
own the real integration.
## Validation Contract
Current validation already reports authored ASCII `.mdl` as informational.
That behavior should remain, but the plan should extend validation to cover model
compilation readiness when the feature is enabled.
Recommended validation additions:
- compiler path resolves or search candidates exist
- `wine` and `winepath` are resolvable on non-Windows platforms when mode is
`wine`
- NWN data root exists and has required keys
- cache path is safe and repo-relative unless explicitly documented otherwise
- include/exclude glob config is well-formed
- jobs is >= 1
Validation severity:
- config/discovery failures that make enabled compilation impossible should be
errors
- authored ASCII `.mdl` detection remains info
- binary `.mdl` sources alongside enabled compilation are not errors
## Manifest And Reuse Contract
Generated `haks.json` must continue to describe authored asset-relative paths,
not cache file paths.
HAK reuse logic must account for compiled model output bytes. Reuse must not
mistakenly treat an unchanged authored path as unchanged content if the compiled
artifact changed.
That means one of:
- the existing asset `ContentID` for `.mdl` must reflect compiled output hash
when compilation is enabled, or
- chunk hashing must incorporate the prepared compiled output fingerprint
Without this, stale HAK reuse would be incorrect.
## Parallelism Contract
Support configurable parallel compilation with conservative defaults.
Requirements:
- default jobs is `1`
- jobs > 1 must not corrupt outputs or logs
- each compile must write to a unique output path and unique log path
- shared registry/prefix setup must happen before parallel work or under safe
synchronization
If `nwnmdlcomp.exe` or Wine proves unstable under parallel execution:
- keep serial default
- document the risk
- allow opting into higher parallelism with clear caveats
## Testing Contract
Implementation is not complete without automated tests. Use fake executables and
temporary directories rather than requiring a real Wine/NWN environment in unit
tests.
## Unit tests
Add tests for:
- config loading and effective config defaults for `models.compile`
- compiler path/env/search resolution
- Wine tool resolution
- NWN data root validation and alias handling
- ASCII-vs-binary `.mdl` eligibility logic
- include/exclude filtering
- incremental reuse decisions
- metadata invalidation on source/config/compiler changes
- content hashing based on compiled output, not source text alone
## Pipeline tests
Add or extend `internal/pipeline/pipeline_test.go` coverage for:
- HAK build packages compiled `.mdl` bytes instead of source ASCII bytes
- non-`.mdl` assets remain unchanged
- binary source `.mdl` passes through unchanged
- failed model compile aborts build
- missing output aborts build
- zero-byte output aborts build
- `--skip-models` bypasses compilation
- `--force-models` rebuilds compiled outputs
- unchanged compiled output allows HAK reuse
- changed compiled output forces HAK rewrite
- duplicate asset collision rules still behave as before
## Validator tests
Keep and extend tests proving:
- ASCII source models are still reported as informational
- enabled compilation readiness errors surface when config is invalid
## Integration/manual acceptance tests
Document real-world manual checks for a workstation with Wine and
`nwnmdlcomp.exe` available.
### Test 1: readiness validation
```bash
sow-toolkit validate
```
Expected:
- no model-compiler readiness errors when config/env are correct
- ASCII `.mdl` info line still appears for source models
### Test 2: single-model HAK packaging
Given an authored ASCII model under `paths.assets`, run:
```bash
sow-toolkit build-haks --force-models
```
Expected:
- build succeeds
- resulting HAK contains a binary `.mdl`
- authored source file remains unchanged
### Test 3: incremental reuse
Run `build-haks` twice without changes.
Expected:
- second run reports model reuse
- unchanged HAKs are reused when manifest/chunk fingerprints match
### Test 4: source change invalidation
Edit one ASCII `.mdl` source and rebuild.
Expected:
- only affected compiled output is rebuilt
- affected HAK chunk is rewritten
- unrelated HAK chunks remain reusable
### Test 5: missing NWN data
Misconfigure `models.compile.data.root` and run build.
Expected:
- validation/build fails clearly before or during preparation
- error names the missing key/data requirement
## Documentation Contract
Update `README.md` to include:
- why compiled models are shipped even though ASCII remains authored source
- the new `models.compile` config section
- required local dependencies:
- `nwnmdlcomp.exe`
- `wine`
- `winepath`
- NWN data root
- environment override behavior
- cache/staging behavior
- `build-haks` flags related to model compilation
- limitations and troubleshooting notes
Optional but recommended:
- `tools/nwnmdlcomp/README.md` for consumer repo operators
## Suggested Implementation Breakdown
1. Schema and config foundation
- add `models.compile` config structs to `internal/project/project.go`
- add effective config defaults and path expansion in
`internal/project/effective.go`
- add project helper methods for compiler/cache/env resolution
2. Validation foundation
- extend project/config validation for enabled compilation readiness
- extend CLI validation reporting where needed
3. Preparation layer
- add model compiler resolution, Wine resolution, NWN data staging, registry
setup, and incremental metadata logic
- add a `preparedModelAssets` representation
4. Pipeline integration
- plug model preparation into `planOrBuildHAKs`
- teach `collectAssetResources` to substitute compiled model payloads
- ensure `ContentID` and HAK reuse reflect compiled output
5. CLI polish
- add build-haks flags if approved
- add progress messages and summary output
6. Tests
- config tests
- validator tests
- pipeline tests with fake compiler/Wine shims
7. Docs
- repo README updates
- optional operator README under `tools/`
## Open Design Decisions To Resolve Before Coding
These should be decided explicitly during implementation, not left implicit:
1. Should the toolkit own a dedicated Wine prefix under cache by default, or
require a user-provided prefix?
2. Should NWN data alias files be created in a separate staging tree, or should
the configured data root be used directly when already compatible?
3. Is parallel model compilation stable enough to expose beyond best-effort?
4. Do we want dedicated `build-haks` flags now, or only config for the first
implementation?
5. Should per-model logs always be persisted, or only on failure / debug mode?
## Acceptance Criteria
The feature is complete when all of the following are true:
- `sow-toolkit build-haks` can package compiled binary `.mdl` content from
authored ASCII `.mdl` source when `models.compile.enabled: true`
- authored `.mdl` files remain unchanged on disk
- validation and topdata/autogen behaviors that depend on authored `.mdl` source
still work
- generated HAK manifests still reference authored asset-relative paths
- HAK reuse logic correctly tracks compiled output changes
- the feature is opt-in, deterministic, and tested
- failures are actionable and name the offending model paths
- README/config docs are updated
## Relevant References
- Repo code:
- `internal/pipeline/build.go`
- `internal/validator/validator.go`
- `internal/topdata/autogen.go`
- `internal/topdata/parts_discovery.go`
- `README.md`
- Upstream docs consulted for this contract:
- `nwn.wiki` MDL page: compiled models are recommended for release because
ASCII loads more slowly
- Neverwinter Vault `nwnmdlcomp` page: standard compile/decompile CLI usage
- Neverwinter Vault forum discussions: `nwnmdlcomp` expects old NWN registry
keys/data layout and some supermodel cases need additional care