diff --git a/MODEL_COMPILATION_ON_BUILDTIME_CONTRACT.md b/MODEL_COMPILATION_ON_BUILDTIME_CONTRACT.md new file mode 100644 index 0000000..724e2c4 --- /dev/null +++ b/MODEL_COMPILATION_ON_BUILDTIME_CONTRACT.md @@ -0,0 +1,770 @@ +# Model Compilation Build-Time Contract + +## Scope + +Expand the toolkit so authored ASCII Neverwinter Nights `.mdl` assets can be +compiled into binary `.mdl` build artifacts during `sow-toolkit build-haks` +without changing the authored source tree. + +This contract covers: + +- `toolkit/` as the implementation owner +- `sow-assets/` as the primary consumer repository +- any future consumer repo that packages NWN model assets through the same HAK + build pipeline + +This contract does not cover: + +- replacing authored ASCII `.mdl` files in source control +- introducing a separate standalone shell-only build system as the primary path +- downloading or redistributing copyrighted NWN game data +- building an in-game runtime compiler workflow + +## Goal + +Make model compilation a first-class, deterministic, configurable build stage in +the Go toolkit so generated HAKs contain compiled binary models while authored +repositories continue to keep ASCII `.mdl` files as source of truth. + +## Why This Exists + +Repo-local context already shows that ASCII `.mdl` files are an intentional part +of authored asset workflows: + +- validation reports ASCII/decompiled models as informational, not erroneous +- topdata autogen manifest generation scans authored `.mdl` paths directly +- part discovery derives IDs from authored `.mdl` filenames +- HAK building already has other generated-asset stages such as NWScript and + music conversion + +Upstream NWN documentation also supports compiling before release: + +- `nwn.wiki` documents that ASCII models are slower because the game compiles + them at load time and recommends shipping compiled models +- `nwnmdlcomp` usage docs confirm ASCII-to-binary compilation as a standard flow +- community documentation shows `nwnmdlcomp.exe` expects old NWN registry/data + layout and may need a 1.69-era data root + +Therefore the correct implementation is not “replace asset scanning with binary +models everywhere”. The correct implementation is: + +1. keep authored ASCII `.mdl` in `paths.assets` +2. preserve current validation and manifest derivation behavior against source +3. compile only the build payload that becomes generated HAK content + +## Ownership Split + +- `toolkit/` + - owns config schema, validation, discovery, cache policy, build staging, and + HAK integration + - owns Wine execution and NWN data preparation logic + - owns tests and user-facing CLI behavior +- `sow-assets/` + - owns opting into the feature through `nwn-tool.yaml` + - owns providing local tool binaries and any fake/minimal NWN data tree it + wants to use + - owns wrapper scripts and CI invocation details +- `sow-module/` + - is an indirect consumer only when it consumes generated HAKs + - must not need authored-source changes for this feature to work + +## Current Project-Needs Analysis + +## Existing architecture + +Current HAK packaging is Go-native and centered on: + +- `cmd/nwn-tool/main.go` +- `internal/app/app.go` +- `internal/pipeline/build.go` +- `internal/project/*` +- `internal/validator/*` + +`build-haks` already does all asset discovery, planning, manifest writing, HAK +reuse, and generated-asset preparation inside the toolkit. Adding model +compilation as a disconnected shell script would duplicate the existing build +graph and create drift. + +## Existing relevant behavior + +- `project.AssetExtensions` already includes `.mdl` +- `validator.ValidateProject` detects text `.mdl` and reports them via + `Report.DecompiledModels` +- `emitValidatorReport` already surfaces that information in CLI output +- `collectAssetResources` packages raw files from `paths.assets` directly into + HAK resources +- `BuildHAKs` already supports generated assets and temporary staging through the + music pipeline +- script compilation already establishes a precedent for: + - config-backed tool resolution + - environment discovery + - cache directories under `.cache` + - build-time generated resources that do not modify authored sources + +## Constraints the plan must preserve + +- authored `.mdl` files remain canonical inputs +- autogen and topdata logic must continue to see source `.mdl` paths +- validation must still be able to detect ASCII source models +- HAK duplicate detection and manifest generation must remain deterministic +- build output reuse must not falsely reuse HAKs built from stale compiled model + content +- consumer repos must be able to opt out cleanly + +## Architecture Decision + +Implement model compilation as a generated-asset overlay inside the existing HAK +build pipeline, not as a separate top-level shell build. + +That means: + +- source scanning still indexes authored `.mdl` +- a model preparation stage produces compiled build artifacts in cache/staging +- `collectAssetResources` substitutes compiled `.mdl` bytes for eligible source + models during HAK packaging +- non-model assets continue to flow unchanged +- the written HAK manifest still records the authored relative asset path, not a + cache path + +This mirrors the existing music pipeline more than the script pipeline: + +- source asset exists in `paths.assets` +- generated build artifact lives in cache/staging +- HAK includes the generated bytes under the original logical resource name + +## Non-Goals + +- no automatic decompilation +- no mutation of files under `paths.assets` +- no requirement to compile every `.mdl` in every repo by default +- no requirement to support Windows-only shell wrappers as the primary interface +- no requirement to support arbitrary model compilers on day one beyond + `nwnmdlcomp.exe` via Wine +- no attempt to “fix” malformed source models automatically + +## Config Contract + +Add a new top-level config section: + +```yaml +models: + compile: + enabled: true + source_extensions: [.mdl] + include: [] + exclude: [] + cache: "{paths.cache}/models" + compiler: + path: "" + env: + path: SOW_NWN_MODEL_COMPILER + wine: SOW_NWN_WINE + winepath: SOW_NWN_WINEPATH + prefix: SOW_NWN_WINEPREFIX + nwn_data: [SOW_NWN_MODEL_DATA, SOW_NWN_ROOT, NWN_ROOT] + search: + - "{paths.tools}/nwnmdlcomp/nwnmdlcomp.exe" + - "{paths.tools}/nwnmdlcomp.exe" + runtime: + mode: wine + jobs: 1 + force: false + setup_registry: true + compiler_cwd: "{paths.tools}/nwnmdlcomp" + data: + root: "" + require_keys: [chitin.key, nwn_base.key, data/nwn_base.key] + writable_aliases: [chitin.key, nwn_base.key] + policy: + compile_ascii_only: true + preserve_binary_sources: true + fail_on_missing_output: true + fail_on_zero_byte_output: true + fail_on_warning: false +``` + +## Config design rules + +- `models.compile.enabled` must default to `false` +- the feature must be opt-in +- config must be repository-driven, not hardcoded per consumer +- search/discovery semantics should match `scripts.compiler` where practical +- path templating should reuse the existing effective config path expansion model +- include/exclude filters apply to asset-relative paths under `paths.assets` +- default jobs should be serial unless parallel execution is explicitly enabled + +## Environment precedence + +Use this precedence for the compiler executable: + +1. `models.compile.compiler.path` +2. env var named by `models.compile.compiler.env.path` +3. configured search paths + +Use this precedence for NWN data root: + +1. `models.compile.data.root` +2. env vars listed in `models.compile.compiler.env.nwn_data` +3. no implicit auto-discovery unless explicitly implemented and documented + +Wine tools should resolve as: + +1. env override for `wine` +2. `PATH:wine` + +and: + +1. env override for `winepath` +2. `PATH:winepath` + +Do not silently guess a fake NWN data tree. + +## Build Contract + +## Eligibility rules + +A model is eligible for build-time compilation when all of the following are +true: + +- asset extension is `.mdl` +- `models.compile.enabled` is true +- path passes include/exclude filters +- file is detected as ASCII when `compile_ascii_only` is true + +If a `.mdl` file is already binary and `preserve_binary_sources` is true: + +- do not recompile it +- package it as-is +- optionally log a verbose skip reason + +## Integration point + +Insert model preparation into `planOrBuildHAKs` before `collectAssetResources` +finalizes HAK resource content, alongside other generated-asset preparation. + +Expected flow: + +1. validate project +2. prepare music assets if enabled +3. prepare compiled model assets if enabled +4. collect asset resources using source inventory plus prepared overlays +5. plan/write HAKs as today + +## Prepared asset representation + +Introduce a prepared-model result similar in spirit to `preparedMusicAssets`, +for example: + +- generated assets keyed by original asset-relative path +- skipped source-relative paths +- summary counts and loggable actions +- cleanup hook for temporary staging if needed + +The HAK layer should continue to see: + +- resource name from the original source path stem +- resource type `mdl` +- relative manifest path equal to the original source path + +Only the file bytes should differ. + +## Cache and staging contract + +Compiled models are generated artifacts and must live outside the authored asset +tree. + +Default cache root: + +```text +{paths.cache}/models +``` + +Suggested cache layout: + +```text +.cache/models/ + compiled/.mdl + logs/.log + metadata.json + wine-prefix/ # only if the repo chooses a toolkit-managed prefix + nwn-data-links/ # only if alias files/symlinks are staged +``` + +The exact layout may differ, but it must support: + +- deterministic mapping from source asset path to compiled output path +- per-model logging for failures +- incremental stale detection +- cleanup without touching authored content + +## Incremental contract + +The plan must support incremental recompilation for persistent cache dirs. + +At minimum, recompile when any of these change: + +- source file content hash changed +- source file modtime is newer than compiled output +- compiler executable path or content fingerprint changed +- NWN data root fingerprint changed if the implementation tracks it +- relevant config changed +- `--force` or config `force: true` is active +- compiled output is missing or zero bytes + +Recommended metadata key: + +- source asset relative path +- source SHA-256 +- output SHA-256 +- compiler path +- compiler binary mtime or hash +- NWN data root path +- build config fingerprint +- compiled timestamp + +If metadata is unavailable or invalid, fail open to recompilation, not reuse. + +## NWN Data Contract + +`nwnmdlcomp.exe` requires a classic NWN data/registry environment. + +The implementation must: + +- validate the configured NWN data root exists +- require at least one of: + - `chitin.key` + - `nwn_base.key` + - `data/nwn_base.key` +- never download game data +- never synthesize copyrighted content + +If only `data/nwn_base.key` exists, the implementation may create a staged alias +view for the compiler by: + +- symlinking inside a cache-owned staging dir, or +- copying just the key file names into a cache-owned staging dir when symlinks + are unavailable + +Do not mutate the user’s real install directory unless they explicitly pointed +the config at a writable fake data tree and the implementation clearly documents +that behavior. + +## Wine Registry Contract + +When `runtime.setup_registry` is enabled, prepare the Wine registry keys needed +by `nwnmdlcomp.exe`: + +```text +HKLM\Software\WOW6432Node\BioWare\NWN\Neverwinter + Location = + Path = + Version = 1.69 + GUID = 00000000-0000-0000-0000-000000000000 + Language = 0 +``` + +Implementation rules: + +- convert Linux paths with `winepath -w` +- target the Wine prefix selected by config/env +- make registry setup idempotent +- separate “prepare registry” failure messages from “compile model” failures + +The implementation may either: + +- manage a dedicated toolkit prefix, or +- use a user-provided existing prefix + +The contract should prefer an isolated prefix to avoid polluting unrelated Wine +state. + +## Compiler Invocation Contract + +Primary supported compiler: + +- `nwnmdlcomp.exe` + +Expected invocation shape: + +```bash +WINEDEBUG=-all wine "$COMPILER" -c "$INPUT_WIN" "$OUTPUT_WIN" -e +``` + +Implementation requirements: + +- invoke through `os/exec`, not shell string concatenation +- convert input/output paths with `winepath -w` +- allow compiler working directory configuration if supermodel resolution or + relative lookup requires it +- capture combined stdout/stderr per compile attempt + +## Supermodel and sibling dependency considerations + +The plan must explicitly account for model compilation not being a pure +single-file transform. + +Potential dependencies include: + +- `setsupermodel` references +- sibling helper models in the same asset family +- textures or material references that may not block compilation but do matter + for diagnostics + +Minimum contract: + +- compile against a staging layout that preserves original relative structure +- ensure each model’s directory context exists in staging +- document that some supermodel cases may require staging additional authored + `.mdl` siblings into the compiler-visible workspace + +Do not assume “compile one file in isolation” is always sufficient. + +## Staging strategy + +Preferred strategy: + +1. keep source inventory rooted in `paths.assets` +2. materialize a compiler workspace under cache that mirrors required relative + directories for eligible `.mdl` +3. compile outputs into a sibling compiled tree +4. package compiled output bytes while preserving source-relative logical names + +This avoids: + +- polluting source trees +- output collisions between models with the same basename in different folders +- broken relative assumptions inside asset packs + +## Logging Contract + +Progress output should fit existing `ProgressFunc` style. + +Minimum messages: + +- `Preparing model compiler...` +- `Preparing model compilation workspace...` +- `Compiling model 12/324: part/belt/pfa0_belt018.mdl` +- `Reusing compiled model: vfxs/head_accessories/hfx_bandana.mdl` +- `Skipping binary model source: creatures/foo.mdl` +- `FAILED model compile: placeables/broken_model.mdl` + +At summary level, surface: + +- models considered +- compiled +- reused +- skipped binary +- skipped filtered +- failed + +Verbose/debug output may include: + +- chosen compiler path +- chosen Wine/Wineprefix/Winepath tools +- chosen NWN data root +- per-model log file location + +## Error Handling Contract + +Model compilation is a build-critical stage once enabled. + +The build must fail when an eligible model compile: + +- exits non-zero +- does not produce the expected output file +- produces a zero-byte output file +- cannot be staged or fingerprinted + +Warnings from the compiler do not fail the build unless future config enables +that policy. + +Failure output must include: + +- original source-relative asset path +- compiler command context sufficient for debugging +- log file path if written + +At the end of a failed run, include a concise failed-path summary. + +## CLI Contract + +The canonical user-facing entrypoint remains: + +```bash +sow-toolkit build-haks +``` + +Add targeted flags only if they materially improve operability and match +existing CLI patterns. Recommended flags: + +- `build-haks --force-models` +- `build-haks --skip-models` +- `build-haks --model ` optional, only if there is a real use + case for scoped rebuilds + +Flag rules: + +- flags override config for the current invocation only +- `--skip-models` must bypass model preparation even if config is enabled +- `--force-models` must invalidate incremental reuse for model outputs only + +Do not introduce a parallel standalone command as the only way to use the +feature. A helper/diagnostic subcommand is acceptable later, but build-haks must +own the real integration. + +## Validation Contract + +Current validation already reports authored ASCII `.mdl` as informational. +That behavior should remain, but the plan should extend validation to cover model +compilation readiness when the feature is enabled. + +Recommended validation additions: + +- compiler path resolves or search candidates exist +- `wine` and `winepath` are resolvable on non-Windows platforms when mode is + `wine` +- NWN data root exists and has required keys +- cache path is safe and repo-relative unless explicitly documented otherwise +- include/exclude glob config is well-formed +- jobs is >= 1 + +Validation severity: + +- config/discovery failures that make enabled compilation impossible should be + errors +- authored ASCII `.mdl` detection remains info +- binary `.mdl` sources alongside enabled compilation are not errors + +## Manifest And Reuse Contract + +Generated `haks.json` must continue to describe authored asset-relative paths, +not cache file paths. + +HAK reuse logic must account for compiled model output bytes. Reuse must not +mistakenly treat an unchanged authored path as unchanged content if the compiled +artifact changed. + +That means one of: + +- the existing asset `ContentID` for `.mdl` must reflect compiled output hash + when compilation is enabled, or +- chunk hashing must incorporate the prepared compiled output fingerprint + +Without this, stale HAK reuse would be incorrect. + +## Parallelism Contract + +Support configurable parallel compilation with conservative defaults. + +Requirements: + +- default jobs is `1` +- jobs > 1 must not corrupt outputs or logs +- each compile must write to a unique output path and unique log path +- shared registry/prefix setup must happen before parallel work or under safe + synchronization + +If `nwnmdlcomp.exe` or Wine proves unstable under parallel execution: + +- keep serial default +- document the risk +- allow opting into higher parallelism with clear caveats + +## Testing Contract + +Implementation is not complete without automated tests. Use fake executables and +temporary directories rather than requiring a real Wine/NWN environment in unit +tests. + +## Unit tests + +Add tests for: + +- config loading and effective config defaults for `models.compile` +- compiler path/env/search resolution +- Wine tool resolution +- NWN data root validation and alias handling +- ASCII-vs-binary `.mdl` eligibility logic +- include/exclude filtering +- incremental reuse decisions +- metadata invalidation on source/config/compiler changes +- content hashing based on compiled output, not source text alone + +## Pipeline tests + +Add or extend `internal/pipeline/pipeline_test.go` coverage for: + +- HAK build packages compiled `.mdl` bytes instead of source ASCII bytes +- non-`.mdl` assets remain unchanged +- binary source `.mdl` passes through unchanged +- failed model compile aborts build +- missing output aborts build +- zero-byte output aborts build +- `--skip-models` bypasses compilation +- `--force-models` rebuilds compiled outputs +- unchanged compiled output allows HAK reuse +- changed compiled output forces HAK rewrite +- duplicate asset collision rules still behave as before + +## Validator tests + +Keep and extend tests proving: + +- ASCII source models are still reported as informational +- enabled compilation readiness errors surface when config is invalid + +## Integration/manual acceptance tests + +Document real-world manual checks for a workstation with Wine and +`nwnmdlcomp.exe` available. + +### Test 1: readiness validation + +```bash +sow-toolkit validate +``` + +Expected: + +- no model-compiler readiness errors when config/env are correct +- ASCII `.mdl` info line still appears for source models + +### Test 2: single-model HAK packaging + +Given an authored ASCII model under `paths.assets`, run: + +```bash +sow-toolkit build-haks --force-models +``` + +Expected: + +- build succeeds +- resulting HAK contains a binary `.mdl` +- authored source file remains unchanged + +### Test 3: incremental reuse + +Run `build-haks` twice without changes. + +Expected: + +- second run reports model reuse +- unchanged HAKs are reused when manifest/chunk fingerprints match + +### Test 4: source change invalidation + +Edit one ASCII `.mdl` source and rebuild. + +Expected: + +- only affected compiled output is rebuilt +- affected HAK chunk is rewritten +- unrelated HAK chunks remain reusable + +### Test 5: missing NWN data + +Misconfigure `models.compile.data.root` and run build. + +Expected: + +- validation/build fails clearly before or during preparation +- error names the missing key/data requirement + +## Documentation Contract + +Update `README.md` to include: + +- why compiled models are shipped even though ASCII remains authored source +- the new `models.compile` config section +- required local dependencies: + - `nwnmdlcomp.exe` + - `wine` + - `winepath` + - NWN data root +- environment override behavior +- cache/staging behavior +- `build-haks` flags related to model compilation +- limitations and troubleshooting notes + +Optional but recommended: + +- `tools/nwnmdlcomp/README.md` for consumer repo operators + +## Suggested Implementation Breakdown + +1. Schema and config foundation + - add `models.compile` config structs to `internal/project/project.go` + - add effective config defaults and path expansion in + `internal/project/effective.go` + - add project helper methods for compiler/cache/env resolution + +2. Validation foundation + - extend project/config validation for enabled compilation readiness + - extend CLI validation reporting where needed + +3. Preparation layer + - add model compiler resolution, Wine resolution, NWN data staging, registry + setup, and incremental metadata logic + - add a `preparedModelAssets` representation + +4. Pipeline integration + - plug model preparation into `planOrBuildHAKs` + - teach `collectAssetResources` to substitute compiled model payloads + - ensure `ContentID` and HAK reuse reflect compiled output + +5. CLI polish + - add build-haks flags if approved + - add progress messages and summary output + +6. Tests + - config tests + - validator tests + - pipeline tests with fake compiler/Wine shims + +7. Docs + - repo README updates + - optional operator README under `tools/` + +## Open Design Decisions To Resolve Before Coding + +These should be decided explicitly during implementation, not left implicit: + +1. Should the toolkit own a dedicated Wine prefix under cache by default, or + require a user-provided prefix? +2. Should NWN data alias files be created in a separate staging tree, or should + the configured data root be used directly when already compatible? +3. Is parallel model compilation stable enough to expose beyond best-effort? +4. Do we want dedicated `build-haks` flags now, or only config for the first + implementation? +5. Should per-model logs always be persisted, or only on failure / debug mode? + +## Acceptance Criteria + +The feature is complete when all of the following are true: + +- `sow-toolkit build-haks` can package compiled binary `.mdl` content from + authored ASCII `.mdl` source when `models.compile.enabled: true` +- authored `.mdl` files remain unchanged on disk +- validation and topdata/autogen behaviors that depend on authored `.mdl` source + still work +- generated HAK manifests still reference authored asset-relative paths +- HAK reuse logic correctly tracks compiled output changes +- the feature is opt-in, deterministic, and tested +- failures are actionable and name the offending model paths +- README/config docs are updated + +## Relevant References + +- Repo code: + - `internal/pipeline/build.go` + - `internal/validator/validator.go` + - `internal/topdata/autogen.go` + - `internal/topdata/parts_discovery.go` + - `README.md` +- Upstream docs consulted for this contract: + - `nwn.wiki` MDL page: compiled models are recommended for release because + ASCII loads more slowly + - Neverwinter Vault `nwnmdlcomp` page: standard compile/decompile CLI usage + - Neverwinter Vault forum discussions: `nwnmdlcomp` expects old NWN registry + keys/data layout and some supermodel cases need additional care