From 7919ded8b4616989050e60667c92020476f286e0 Mon Sep 17 00:00:00 2001 From: vickydotbat Date: Sun, 10 May 2026 09:11:06 +0200 Subject: [PATCH] Model compilation contract --- MODEL_COMPILATION_ON_BUILDTIME_CONTRACT.md | 174 +++++++-------------- 1 file changed, 55 insertions(+), 119 deletions(-) diff --git a/MODEL_COMPILATION_ON_BUILDTIME_CONTRACT.md b/MODEL_COMPILATION_ON_BUILDTIME_CONTRACT.md index 724e2c4..d475d22 100644 --- a/MODEL_COMPILATION_ON_BUILDTIME_CONTRACT.md +++ b/MODEL_COMPILATION_ON_BUILDTIME_CONTRACT.md @@ -2,16 +2,13 @@ ## Scope -Expand the toolkit so authored ASCII Neverwinter Nights `.mdl` assets can be -compiled into binary `.mdl` build artifacts during `sow-toolkit build-haks` -without changing the authored source tree. +Expand the toolkit so authored ASCII Neverwinter Nights `.mdl` assets can be compiled into binary `.mdl` build artifacts during `sow-toolkit build-haks` without changing the authored source tree. This contract covers: - `toolkit/` as the implementation owner - `sow-assets/` as the primary consumer repository -- any future consumer repo that packages NWN model assets through the same HAK - build pipeline +- any future consumer repo that packages NWN model assets through the same HAK build pipeline This contract does not cover: @@ -22,31 +19,24 @@ This contract does not cover: ## Goal -Make model compilation a first-class, deterministic, configurable build stage in -the Go toolkit so generated HAKs contain compiled binary models while authored -repositories continue to keep ASCII `.mdl` files as source of truth. +Make model compilation a first-class, deterministic, configurable build stage in the Go toolkit so generated HAKs contain compiled binary models while authored repositories continue to keep ASCII `.mdl` files as source of truth. ## Why This Exists -Repo-local context already shows that ASCII `.mdl` files are an intentional part -of authored asset workflows: +Repo-local context already shows that ASCII `.mdl` files are an intentional part of authored asset workflows: - validation reports ASCII/decompiled models as informational, not erroneous - topdata autogen manifest generation scans authored `.mdl` paths directly - part discovery derives IDs from authored `.mdl` filenames -- HAK building already has other generated-asset stages such as NWScript and - music conversion +- HAK building already has other generated-asset stages such as NWScript and music conversion Upstream NWN documentation also supports compiling before release: -- `nwn.wiki` documents that ASCII models are slower because the game compiles - them at load time and recommends shipping compiled models +- `nwn.wiki` documents that ASCII models are slower because the game compiles them at load time and recommends shipping compiled models - `nwnmdlcomp` usage docs confirm ASCII-to-binary compilation as a standard flow -- community documentation shows `nwnmdlcomp.exe` expects old NWN registry/data - layout and may need a 1.69-era data root +- community documentation shows `nwnmdlcomp.exe` expects old NWN registry/data layout and may need a 1.69-era data root -Therefore the correct implementation is not “replace asset scanning with binary -models everywhere”. The correct implementation is: +Therefore the correct implementation is not “replace asset scanning with binary models everywhere”. The correct implementation is: 1. keep authored ASCII `.mdl` in `paths.assets` 2. preserve current validation and manifest derivation behavior against source @@ -55,14 +45,12 @@ models everywhere”. The correct implementation is: ## Ownership Split - `toolkit/` - - owns config schema, validation, discovery, cache policy, build staging, and - HAK integration + - owns config schema, validation, discovery, cache policy, build staging, and HAK integration - owns Wine execution and NWN data preparation logic - owns tests and user-facing CLI behavior - `sow-assets/` - owns opting into the feature through `nwn-tool.yaml` - - owns providing local tool binaries and any fake/minimal NWN data tree it - wants to use + - owns providing local tool binaries and any fake/minimal NWN data tree it wants to use - owns wrapper scripts and CI invocation details - `sow-module/` - is an indirect consumer only when it consumes generated HAKs @@ -80,21 +68,15 @@ Current HAK packaging is Go-native and centered on: - `internal/project/*` - `internal/validator/*` -`build-haks` already does all asset discovery, planning, manifest writing, HAK -reuse, and generated-asset preparation inside the toolkit. Adding model -compilation as a disconnected shell script would duplicate the existing build -graph and create drift. +`build-haks` already does all asset discovery, planning, manifest writing, HAK reuse, and generated-asset preparation inside the toolkit. Adding model compilation as a disconnected shell script would duplicate the existing build graph and create drift. ## Existing relevant behavior - `project.AssetExtensions` already includes `.mdl` -- `validator.ValidateProject` detects text `.mdl` and reports them via - `Report.DecompiledModels` +- `validator.ValidateProject` detects text `.mdl` and reports them via `Report.DecompiledModels` - `emitValidatorReport` already surfaces that information in CLI output -- `collectAssetResources` packages raw files from `paths.assets` directly into - HAK resources -- `BuildHAKs` already supports generated assets and temporary staging through the - music pipeline +- `collectAssetResources` packages raw files from `paths.assets` directly into HAK resources +- `BuildHAKs` already supports generated assets and temporary staging through the music pipeline - script compilation already establishes a precedent for: - config-backed tool resolution - environment discovery @@ -107,24 +89,20 @@ graph and create drift. - autogen and topdata logic must continue to see source `.mdl` paths - validation must still be able to detect ASCII source models - HAK duplicate detection and manifest generation must remain deterministic -- build output reuse must not falsely reuse HAKs built from stale compiled model - content +- build output reuse must not falsely reuse HAKs built from stale compiled model content - consumer repos must be able to opt out cleanly ## Architecture Decision -Implement model compilation as a generated-asset overlay inside the existing HAK -build pipeline, not as a separate top-level shell build. +Implement model compilation as a generated-asset overlay inside the existing HAK build pipeline, not as a separate top-level shell build. That means: - source scanning still indexes authored `.mdl` - a model preparation stage produces compiled build artifacts in cache/staging -- `collectAssetResources` substitutes compiled `.mdl` bytes for eligible source - models during HAK packaging +- `collectAssetResources` substitutes compiled `.mdl` bytes for eligible source models during HAK packaging - non-model assets continue to flow unchanged -- the written HAK manifest still records the authored relative asset path, not a - cache path +- the written HAK manifest still records the authored relative asset path, not a cache path This mirrors the existing music pipeline more than the script pipeline: @@ -138,8 +116,7 @@ This mirrors the existing music pipeline more than the script pipeline: - no mutation of files under `paths.assets` - no requirement to compile every `.mdl` in every repo by default - no requirement to support Windows-only shell wrappers as the primary interface -- no requirement to support arbitrary model compilers on day one beyond - `nwnmdlcomp.exe` via Wine +- no requirement to support arbitrary model compilers on day one beyond `nwnmdlcomp.exe` via Wine - no attempt to “fix” malformed source models automatically ## Config Contract @@ -223,8 +200,7 @@ Do not silently guess a fake NWN data tree. ## Eligibility rules -A model is eligible for build-time compilation when all of the following are -true: +A model is eligible for build-time compilation when all of the following are true: - asset extension is `.mdl` - `models.compile.enabled` is true @@ -239,8 +215,7 @@ If a `.mdl` file is already binary and `preserve_binary_sources` is true: ## Integration point -Insert model preparation into `planOrBuildHAKs` before `collectAssetResources` -finalizes HAK resource content, alongside other generated-asset preparation. +Insert model preparation into `planOrBuildHAKs` before `collectAssetResources` finalizes HAK resource content, alongside other generated-asset preparation. Expected flow: @@ -252,8 +227,7 @@ Expected flow: ## Prepared asset representation -Introduce a prepared-model result similar in spirit to `preparedMusicAssets`, -for example: +Introduce a prepared-model result similar in spirit to `preparedMusicAssets`, for example: - generated assets keyed by original asset-relative path - skipped source-relative paths @@ -270,8 +244,7 @@ Only the file bytes should differ. ## Cache and staging contract -Compiled models are generated artifacts and must live outside the authored asset -tree. +Compiled models are generated artifacts and must live outside the authored asset tree. Default cache root: @@ -338,21 +311,16 @@ The implementation must: - never download game data - never synthesize copyrighted content -If only `data/nwn_base.key` exists, the implementation may create a staged alias -view for the compiler by: +If only `data/nwn_base.key` exists, the implementation may create a staged alias view for the compiler by: - symlinking inside a cache-owned staging dir, or -- copying just the key file names into a cache-owned staging dir when symlinks - are unavailable +- copying just the key file names into a cache-owned staging dir when symlinks are unavailable -Do not mutate the user’s real install directory unless they explicitly pointed -the config at a writable fake data tree and the implementation clearly documents -that behavior. +Do not mutate the user’s real install directory unless they explicitly pointed the config at a writable fake data tree and the implementation clearly documents that behavior. ## Wine Registry Contract -When `runtime.setup_registry` is enabled, prepare the Wine registry keys needed -by `nwnmdlcomp.exe`: +When `runtime.setup_registry` is enabled, prepare the Wine registry keys needed by `nwnmdlcomp.exe`: ```text HKLM\Software\WOW6432Node\BioWare\NWN\Neverwinter @@ -375,8 +343,7 @@ The implementation may either: - manage a dedicated toolkit prefix, or - use a user-provided existing prefix -The contract should prefer an isolated prefix to avoid polluting unrelated Wine -state. +The contract should prefer an isolated prefix to avoid polluting unrelated Wine state. ## Compiler Invocation Contract @@ -394,28 +361,24 @@ Implementation requirements: - invoke through `os/exec`, not shell string concatenation - convert input/output paths with `winepath -w` -- allow compiler working directory configuration if supermodel resolution or - relative lookup requires it +- allow compiler working directory configuration if supermodel resolution or relative lookup requires it - capture combined stdout/stderr per compile attempt ## Supermodel and sibling dependency considerations -The plan must explicitly account for model compilation not being a pure -single-file transform. +The plan must explicitly account for model compilation not being a pure single-file transform. Potential dependencies include: - `setsupermodel` references - sibling helper models in the same asset family -- textures or material references that may not block compilation but do matter - for diagnostics +- textures or material references that may not block compilation but do matter for diagnostics Minimum contract: - compile against a staging layout that preserves original relative structure - ensure each model’s directory context exists in staging -- document that some supermodel cases may require staging additional authored - `.mdl` siblings into the compiler-visible workspace +- document that some supermodel cases may require staging additional authored `.mdl` siblings into the compiler-visible workspace Do not assume “compile one file in isolation” is always sufficient. @@ -424,8 +387,7 @@ Do not assume “compile one file in isolation” is always sufficient. Preferred strategy: 1. keep source inventory rooted in `paths.assets` -2. materialize a compiler workspace under cache that mirrors required relative - directories for eligible `.mdl` +2. materialize a compiler workspace under cache that mirrors required relative directories for eligible `.mdl` 3. compile outputs into a sibling compiled tree 4. package compiled output bytes while preserving source-relative logical names @@ -475,8 +437,7 @@ The build must fail when an eligible model compile: - produces a zero-byte output file - cannot be staged or fingerprinted -Warnings from the compiler do not fail the build unless future config enables -that policy. +Warnings from the compiler do not fail the build unless future config enables that policy. Failure output must include: @@ -494,13 +455,11 @@ The canonical user-facing entrypoint remains: sow-toolkit build-haks ``` -Add targeted flags only if they materially improve operability and match -existing CLI patterns. Recommended flags: +Add targeted flags only if they materially improve operability and match existing CLI patterns. Recommended flags: - `build-haks --force-models` - `build-haks --skip-models` -- `build-haks --model ` optional, only if there is a real use - case for scoped rebuilds +- `build-haks --model ` optional, only if there is a real use case for scoped rebuilds Flag rules: @@ -508,21 +467,16 @@ Flag rules: - `--skip-models` must bypass model preparation even if config is enabled - `--force-models` must invalidate incremental reuse for model outputs only -Do not introduce a parallel standalone command as the only way to use the -feature. A helper/diagnostic subcommand is acceptable later, but build-haks must -own the real integration. +Do not introduce a parallel standalone command as the only way to use the feature. A helper/diagnostic subcommand is acceptable later, but build-haks must own the real integration. ## Validation Contract -Current validation already reports authored ASCII `.mdl` as informational. -That behavior should remain, but the plan should extend validation to cover model -compilation readiness when the feature is enabled. +Current validation already reports authored ASCII `.mdl` as informational. That behavior should remain, but the plan should extend validation to cover model compilation readiness when the feature is enabled. Recommended validation additions: - compiler path resolves or search candidates exist -- `wine` and `winepath` are resolvable on non-Windows platforms when mode is - `wine` +- `wine` and `winepath` are resolvable on non-Windows platforms when mode is `wine` - NWN data root exists and has required keys - cache path is safe and repo-relative unless explicitly documented otherwise - include/exclude glob config is well-formed @@ -530,24 +484,19 @@ Recommended validation additions: Validation severity: -- config/discovery failures that make enabled compilation impossible should be - errors +- config/discovery failures that make enabled compilation impossible should be errors - authored ASCII `.mdl` detection remains info - binary `.mdl` sources alongside enabled compilation are not errors ## Manifest And Reuse Contract -Generated `haks.json` must continue to describe authored asset-relative paths, -not cache file paths. +Generated `haks.json` must continue to describe authored asset-relative paths, not cache file paths. -HAK reuse logic must account for compiled model output bytes. Reuse must not -mistakenly treat an unchanged authored path as unchanged content if the compiled -artifact changed. +HAK reuse logic must account for compiled model output bytes. Reuse must not mistakenly treat an unchanged authored path as unchanged content if the compiled artifact changed. That means one of: -- the existing asset `ContentID` for `.mdl` must reflect compiled output hash - when compilation is enabled, or +- the existing asset `ContentID` for `.mdl` must reflect compiled output hash when compilation is enabled, or - chunk hashing must incorporate the prepared compiled output fingerprint Without this, stale HAK reuse would be incorrect. @@ -561,8 +510,7 @@ Requirements: - default jobs is `1` - jobs > 1 must not corrupt outputs or logs - each compile must write to a unique output path and unique log path -- shared registry/prefix setup must happen before parallel work or under safe - synchronization +- shared registry/prefix setup must happen before parallel work or under safe synchronization If `nwnmdlcomp.exe` or Wine proves unstable under parallel execution: @@ -572,9 +520,7 @@ If `nwnmdlcomp.exe` or Wine proves unstable under parallel execution: ## Testing Contract -Implementation is not complete without automated tests. Use fake executables and -temporary directories rather than requiring a real Wine/NWN environment in unit -tests. +Implementation is not complete without automated tests. Use fake executables and temporary directories rather than requiring a real Wine/NWN environment in unit tests. ## Unit tests @@ -615,8 +561,7 @@ Keep and extend tests proving: ## Integration/manual acceptance tests -Document real-world manual checks for a workstation with Wine and -`nwnmdlcomp.exe` available. +Document real-world manual checks for a workstation with Wine and `nwnmdlcomp.exe` available. ### Test 1: readiness validation @@ -695,8 +640,7 @@ Optional but recommended: 1. Schema and config foundation - add `models.compile` config structs to `internal/project/project.go` - - add effective config defaults and path expansion in - `internal/project/effective.go` + - add effective config defaults and path expansion in `internal/project/effective.go` - add project helper methods for compiler/cache/env resolution 2. Validation foundation @@ -704,8 +648,7 @@ Optional but recommended: - extend CLI validation reporting where needed 3. Preparation layer - - add model compiler resolution, Wine resolution, NWN data staging, registry - setup, and incremental metadata logic + - add model compiler resolution, Wine resolution, NWN data staging, registry setup, and incremental metadata logic - add a `preparedModelAssets` representation 4. Pipeline integration @@ -730,24 +673,19 @@ Optional but recommended: These should be decided explicitly during implementation, not left implicit: -1. Should the toolkit own a dedicated Wine prefix under cache by default, or - require a user-provided prefix? -2. Should NWN data alias files be created in a separate staging tree, or should - the configured data root be used directly when already compatible? +1. Should the toolkit own a dedicated Wine prefix under cache by default, or require a user-provided prefix? +2. Should NWN data alias files be created in a separate staging tree, or should the configured data root be used directly when already compatible? 3. Is parallel model compilation stable enough to expose beyond best-effort? -4. Do we want dedicated `build-haks` flags now, or only config for the first - implementation? +4. Do we want dedicated `build-haks` flags now, or only config for the first implementation? 5. Should per-model logs always be persisted, or only on failure / debug mode? ## Acceptance Criteria The feature is complete when all of the following are true: -- `sow-toolkit build-haks` can package compiled binary `.mdl` content from - authored ASCII `.mdl` source when `models.compile.enabled: true` +- `sow-toolkit build-haks` can package compiled binary `.mdl` content from authored ASCII `.mdl` source when `models.compile.enabled: true` - authored `.mdl` files remain unchanged on disk -- validation and topdata/autogen behaviors that depend on authored `.mdl` source - still work +- validation and topdata/autogen behaviors that depend on authored `.mdl` source still work - generated HAK manifests still reference authored asset-relative paths - HAK reuse logic correctly tracks compiled output changes - the feature is opt-in, deterministic, and tested @@ -763,8 +701,6 @@ The feature is complete when all of the following are true: - `internal/topdata/parts_discovery.go` - `README.md` - Upstream docs consulted for this contract: - - `nwn.wiki` MDL page: compiled models are recommended for release because - ASCII loads more slowly + - `nwn.wiki` MDL page: compiled models are recommended for release because ASCII loads more slowly - Neverwinter Vault `nwnmdlcomp` page: standard compile/decompile CLI usage - - Neverwinter Vault forum discussions: `nwnmdlcomp` expects old NWN registry - keys/data layout and some supermodel cases need additional care + - Neverwinter Vault forum discussions: `nwnmdlcomp` expects old NWN registry keys/data layout and some supermodel cases need additional care