Model compilation contract

2026-05-08 22:29:46 +02:00
parent c7fd63a80c
commit dee53e8eb4
1 changed files with 770 additions and 0 deletions
@@ -0,0 +1,770 @@
 # Model Compilation Build-Time Contract
 ## Scope
 Expand the toolkit so authored ASCII Neverwinter Nights `.mdl` assets can be
 compiled into binary `.mdl` build artifacts during `sow-toolkit build-haks`
 without changing the authored source tree.
 This contract covers:
 - `toolkit/` as the implementation owner
 - `sow-assets/` as the primary consumer repository
 - any future consumer repo that packages NWN model assets through the same HAK
  build pipeline
 This contract does not cover:
 - replacing authored ASCII `.mdl` files in source control
 - introducing a separate standalone shell-only build system as the primary path
 - downloading or redistributing copyrighted NWN game data
 - building an in-game runtime compiler workflow
 ## Goal
 Make model compilation a first-class, deterministic, configurable build stage in
 the Go toolkit so generated HAKs contain compiled binary models while authored
 repositories continue to keep ASCII `.mdl` files as source of truth.
 ## Why This Exists
 Repo-local context already shows that ASCII `.mdl` files are an intentional part
 of authored asset workflows:
 - validation reports ASCII/decompiled models as informational, not erroneous
 - topdata autogen manifest generation scans authored `.mdl` paths directly
 - part discovery derives IDs from authored `.mdl` filenames
 - HAK building already has other generated-asset stages such as NWScript and
  music conversion
 Upstream NWN documentation also supports compiling before release:
 - `nwn.wiki` documents that ASCII models are slower because the game compiles
  them at load time and recommends shipping compiled models
 - `nwnmdlcomp` usage docs confirm ASCII-to-binary compilation as a standard flow
 - community documentation shows `nwnmdlcomp.exe` expects old NWN registry/data
  layout and may need a 1.69-era data root
 Therefore the correct implementation is not “replace asset scanning with binary
 models everywhere”. The correct implementation is:
 1. keep authored ASCII `.mdl` in `paths.assets`
 2. preserve current validation and manifest derivation behavior against source
 3. compile only the build payload that becomes generated HAK content
 ## Ownership Split
 - `toolkit/`
  - owns config schema, validation, discovery, cache policy, build staging, and
    HAK integration
  - owns Wine execution and NWN data preparation logic
  - owns tests and user-facing CLI behavior
 - `sow-assets/`
  - owns opting into the feature through `nwn-tool.yaml`
  - owns providing local tool binaries and any fake/minimal NWN data tree it
    wants to use
  - owns wrapper scripts and CI invocation details
 - `sow-module/`
  - is an indirect consumer only when it consumes generated HAKs
  - must not need authored-source changes for this feature to work
 ## Current Project-Needs Analysis
 ## Existing architecture
 Current HAK packaging is Go-native and centered on:
 - `cmd/nwn-tool/main.go`
 - `internal/app/app.go`
 - `internal/pipeline/build.go`
 - `internal/project/*`
 - `internal/validator/*`
 `build-haks` already does all asset discovery, planning, manifest writing, HAK
 reuse, and generated-asset preparation inside the toolkit. Adding model
 compilation as a disconnected shell script would duplicate the existing build
 graph and create drift.
 ## Existing relevant behavior
 - `project.AssetExtensions` already includes `.mdl`
 - `validator.ValidateProject` detects text `.mdl` and reports them via
  `Report.DecompiledModels`
 - `emitValidatorReport` already surfaces that information in CLI output
 - `collectAssetResources` packages raw files from `paths.assets` directly into
  HAK resources
 - `BuildHAKs` already supports generated assets and temporary staging through the
  music pipeline
 - script compilation already establishes a precedent for:
  - config-backed tool resolution
  - environment discovery
  - cache directories under `.cache`
  - build-time generated resources that do not modify authored sources
 ## Constraints the plan must preserve
 - authored `.mdl` files remain canonical inputs
 - autogen and topdata logic must continue to see source `.mdl` paths
 - validation must still be able to detect ASCII source models
 - HAK duplicate detection and manifest generation must remain deterministic
 - build output reuse must not falsely reuse HAKs built from stale compiled model
  content
 - consumer repos must be able to opt out cleanly
 ## Architecture Decision
 Implement model compilation as a generated-asset overlay inside the existing HAK
 build pipeline, not as a separate top-level shell build.
 That means:
 - source scanning still indexes authored `.mdl`
 - a model preparation stage produces compiled build artifacts in cache/staging
 - `collectAssetResources` substitutes compiled `.mdl` bytes for eligible source
  models during HAK packaging
 - non-model assets continue to flow unchanged
 - the written HAK manifest still records the authored relative asset path, not a
  cache path
 This mirrors the existing music pipeline more than the script pipeline:
 - source asset exists in `paths.assets`
 - generated build artifact lives in cache/staging
 - HAK includes the generated bytes under the original logical resource name
 ## Non-Goals
 - no automatic decompilation
 - no mutation of files under `paths.assets`
 - no requirement to compile every `.mdl` in every repo by default
 - no requirement to support Windows-only shell wrappers as the primary interface
 - no requirement to support arbitrary model compilers on day one beyond
  `nwnmdlcomp.exe` via Wine
 - no attempt to “fix” malformed source models automatically
 ## Config Contract
 Add a new top-level config section:
 ```yaml
 models:
  compile:
    enabled: true
    source_extensions: [.mdl]
    include: []
    exclude: []
    cache: "{paths.cache}/models"
    compiler:
      path: ""
      env:
        path: SOW_NWN_MODEL_COMPILER
        wine: SOW_NWN_WINE
        winepath: SOW_NWN_WINEPATH
        prefix: SOW_NWN_WINEPREFIX
        nwn_data: [SOW_NWN_MODEL_DATA, SOW_NWN_ROOT, NWN_ROOT]
      search:
        - "{paths.tools}/nwnmdlcomp/nwnmdlcomp.exe"
        - "{paths.tools}/nwnmdlcomp.exe"
    runtime:
      mode: wine
      jobs: 1
      force: false
      setup_registry: true
      compiler_cwd: "{paths.tools}/nwnmdlcomp"
    data:
      root: ""
      require_keys: [chitin.key, nwn_base.key, data/nwn_base.key]
      writable_aliases: [chitin.key, nwn_base.key]
    policy:
      compile_ascii_only: true
      preserve_binary_sources: true
      fail_on_missing_output: true
      fail_on_zero_byte_output: true
      fail_on_warning: false
 ```
 ## Config design rules
 - `models.compile.enabled` must default to `false`
 - the feature must be opt-in
 - config must be repository-driven, not hardcoded per consumer
 - search/discovery semantics should match `scripts.compiler` where practical
 - path templating should reuse the existing effective config path expansion model
 - include/exclude filters apply to asset-relative paths under `paths.assets`
 - default jobs should be serial unless parallel execution is explicitly enabled
 ## Environment precedence
 Use this precedence for the compiler executable:
 1. `models.compile.compiler.path`
 2. env var named by `models.compile.compiler.env.path`
 3. configured search paths
 Use this precedence for NWN data root:
 1. `models.compile.data.root`
 2. env vars listed in `models.compile.compiler.env.nwn_data`
 3. no implicit auto-discovery unless explicitly implemented and documented
 Wine tools should resolve as:
 1. env override for `wine`
 2. `PATH:wine`
 and:
 1. env override for `winepath`
 2. `PATH:winepath`
 Do not silently guess a fake NWN data tree.
 ## Build Contract
 ## Eligibility rules
 A model is eligible for build-time compilation when all of the following are
 true:
 - asset extension is `.mdl`
 - `models.compile.enabled` is true
 - path passes include/exclude filters
 - file is detected as ASCII when `compile_ascii_only` is true
 If a `.mdl` file is already binary and `preserve_binary_sources` is true:
 - do not recompile it
 - package it as-is
 - optionally log a verbose skip reason
 ## Integration point
 Insert model preparation into `planOrBuildHAKs` before `collectAssetResources`
 finalizes HAK resource content, alongside other generated-asset preparation.
 Expected flow:
 1. validate project
 2. prepare music assets if enabled
 3. prepare compiled model assets if enabled
 4. collect asset resources using source inventory plus prepared overlays
 5. plan/write HAKs as today
 ## Prepared asset representation
 Introduce a prepared-model result similar in spirit to `preparedMusicAssets`,
 for example:
 - generated assets keyed by original asset-relative path
 - skipped source-relative paths
 - summary counts and loggable actions
 - cleanup hook for temporary staging if needed
 The HAK layer should continue to see:
 - resource name from the original source path stem
 - resource type `mdl`
 - relative manifest path equal to the original source path
 Only the file bytes should differ.
 ## Cache and staging contract
 Compiled models are generated artifacts and must live outside the authored asset
 tree.
 Default cache root:
 ```text
 {paths.cache}/models
 ```
 Suggested cache layout:
 ```text
 .cache/models/
  compiled/<asset-relative-path>.mdl
  logs/<asset-relative-path>.log
  metadata.json
  wine-prefix/           # only if the repo chooses a toolkit-managed prefix
  nwn-data-links/        # only if alias files/symlinks are staged
 ```
 The exact layout may differ, but it must support:
 - deterministic mapping from source asset path to compiled output path
 - per-model logging for failures
 - incremental stale detection
 - cleanup without touching authored content
 ## Incremental contract
 The plan must support incremental recompilation for persistent cache dirs.
 At minimum, recompile when any of these change:
 - source file content hash changed
 - source file modtime is newer than compiled output
 - compiler executable path or content fingerprint changed
 - NWN data root fingerprint changed if the implementation tracks it
 - relevant config changed
 - `--force` or config `force: true` is active
 - compiled output is missing or zero bytes
 Recommended metadata key:
 - source asset relative path
 - source SHA-256
 - output SHA-256
 - compiler path
 - compiler binary mtime or hash
 - NWN data root path
 - build config fingerprint
 - compiled timestamp
 If metadata is unavailable or invalid, fail open to recompilation, not reuse.
 ## NWN Data Contract
 `nwnmdlcomp.exe` requires a classic NWN data/registry environment.
 The implementation must:
 - validate the configured NWN data root exists
 - require at least one of:
  - `chitin.key`
  - `nwn_base.key`
  - `data/nwn_base.key`
 - never download game data
 - never synthesize copyrighted content
 If only `data/nwn_base.key` exists, the implementation may create a staged alias
 view for the compiler by:
 - symlinking inside a cache-owned staging dir, or
 - copying just the key file names into a cache-owned staging dir when symlinks
  are unavailable
 Do not mutate the user’s real install directory unless they explicitly pointed
 the config at a writable fake data tree and the implementation clearly documents
 that behavior.
 ## Wine Registry Contract
 When `runtime.setup_registry` is enabled, prepare the Wine registry keys needed
 by `nwnmdlcomp.exe`:
 ```text
 HKLM\Software\WOW6432Node\BioWare\NWN\Neverwinter
  Location = <wine path>
  Path     = <wine path>
  Version  = 1.69
  GUID     = 00000000-0000-0000-0000-000000000000
  Language = 0
 ```
 Implementation rules:
 - convert Linux paths with `winepath -w`
 - target the Wine prefix selected by config/env
 - make registry setup idempotent
 - separate “prepare registry” failure messages from “compile model” failures
 The implementation may either:
 - manage a dedicated toolkit prefix, or
 - use a user-provided existing prefix
 The contract should prefer an isolated prefix to avoid polluting unrelated Wine
 state.
 ## Compiler Invocation Contract
 Primary supported compiler:
 - `nwnmdlcomp.exe`
 Expected invocation shape:
 ```bash
 WINEDEBUG=-all wine "$COMPILER" -c "$INPUT_WIN" "$OUTPUT_WIN" -e
 ```
 Implementation requirements:
 - invoke through `os/exec`, not shell string concatenation
 - convert input/output paths with `winepath -w`
 - allow compiler working directory configuration if supermodel resolution or
  relative lookup requires it
 - capture combined stdout/stderr per compile attempt
 ## Supermodel and sibling dependency considerations
 The plan must explicitly account for model compilation not being a pure
 single-file transform.
 Potential dependencies include:
 - `setsupermodel` references
 - sibling helper models in the same asset family
 - textures or material references that may not block compilation but do matter
  for diagnostics
 Minimum contract:
 - compile against a staging layout that preserves original relative structure
 - ensure each model’s directory context exists in staging
 - document that some supermodel cases may require staging additional authored
  `.mdl` siblings into the compiler-visible workspace
 Do not assume “compile one file in isolation” is always sufficient.
 ## Staging strategy
 Preferred strategy:
 1. keep source inventory rooted in `paths.assets`
 2. materialize a compiler workspace under cache that mirrors required relative
   directories for eligible `.mdl`
 3. compile outputs into a sibling compiled tree
 4. package compiled output bytes while preserving source-relative logical names
 This avoids:
 - polluting source trees
 - output collisions between models with the same basename in different folders
 - broken relative assumptions inside asset packs
 ## Logging Contract
 Progress output should fit existing `ProgressFunc` style.
 Minimum messages:
 - `Preparing model compiler...`
 - `Preparing model compilation workspace...`
 - `Compiling model 12/324: part/belt/pfa0_belt018.mdl`
 - `Reusing compiled model: vfxs/head_accessories/hfx_bandana.mdl`
 - `Skipping binary model source: creatures/foo.mdl`
 - `FAILED model compile: placeables/broken_model.mdl`
 At summary level, surface:
 - models considered
 - compiled
 - reused
 - skipped binary
 - skipped filtered
 - failed
 Verbose/debug output may include:
 - chosen compiler path
 - chosen Wine/Wineprefix/Winepath tools
 - chosen NWN data root
 - per-model log file location
 ## Error Handling Contract
 Model compilation is a build-critical stage once enabled.
 The build must fail when an eligible model compile:
 - exits non-zero
 - does not produce the expected output file
 - produces a zero-byte output file
 - cannot be staged or fingerprinted
 Warnings from the compiler do not fail the build unless future config enables
 that policy.
 Failure output must include:
 - original source-relative asset path
 - compiler command context sufficient for debugging
 - log file path if written
 At the end of a failed run, include a concise failed-path summary.
 ## CLI Contract
 The canonical user-facing entrypoint remains:
 ```bash
 sow-toolkit build-haks
 ```
 Add targeted flags only if they materially improve operability and match
 existing CLI patterns. Recommended flags:
 - `build-haks --force-models`
 - `build-haks --skip-models`
 - `build-haks --model <glob-or-prefix>` optional, only if there is a real use
  case for scoped rebuilds
 Flag rules:
 - flags override config for the current invocation only
 - `--skip-models` must bypass model preparation even if config is enabled
 - `--force-models` must invalidate incremental reuse for model outputs only
 Do not introduce a parallel standalone command as the only way to use the
 feature. A helper/diagnostic subcommand is acceptable later, but build-haks must
 own the real integration.
 ## Validation Contract
 Current validation already reports authored ASCII `.mdl` as informational.
 That behavior should remain, but the plan should extend validation to cover model
 compilation readiness when the feature is enabled.
 Recommended validation additions:
 - compiler path resolves or search candidates exist
 - `wine` and `winepath` are resolvable on non-Windows platforms when mode is
  `wine`
 - NWN data root exists and has required keys
 - cache path is safe and repo-relative unless explicitly documented otherwise
 - include/exclude glob config is well-formed
 - jobs is >= 1
 Validation severity:
 - config/discovery failures that make enabled compilation impossible should be
  errors
 - authored ASCII `.mdl` detection remains info
 - binary `.mdl` sources alongside enabled compilation are not errors
 ## Manifest And Reuse Contract
 Generated `haks.json` must continue to describe authored asset-relative paths,
 not cache file paths.
 HAK reuse logic must account for compiled model output bytes. Reuse must not
 mistakenly treat an unchanged authored path as unchanged content if the compiled
 artifact changed.
 That means one of:
 - the existing asset `ContentID` for `.mdl` must reflect compiled output hash
  when compilation is enabled, or
 - chunk hashing must incorporate the prepared compiled output fingerprint
 Without this, stale HAK reuse would be incorrect.
 ## Parallelism Contract
 Support configurable parallel compilation with conservative defaults.
 Requirements:
 - default jobs is `1`
 - jobs > 1 must not corrupt outputs or logs
 - each compile must write to a unique output path and unique log path
 - shared registry/prefix setup must happen before parallel work or under safe
  synchronization
 If `nwnmdlcomp.exe` or Wine proves unstable under parallel execution:
 - keep serial default
 - document the risk
 - allow opting into higher parallelism with clear caveats
 ## Testing Contract
 Implementation is not complete without automated tests. Use fake executables and
 temporary directories rather than requiring a real Wine/NWN environment in unit
 tests.
 ## Unit tests
 Add tests for:
 - config loading and effective config defaults for `models.compile`
 - compiler path/env/search resolution
 - Wine tool resolution
 - NWN data root validation and alias handling
 - ASCII-vs-binary `.mdl` eligibility logic
 - include/exclude filtering
 - incremental reuse decisions
 - metadata invalidation on source/config/compiler changes
 - content hashing based on compiled output, not source text alone
 ## Pipeline tests
 Add or extend `internal/pipeline/pipeline_test.go` coverage for:
 - HAK build packages compiled `.mdl` bytes instead of source ASCII bytes
 - non-`.mdl` assets remain unchanged
 - binary source `.mdl` passes through unchanged
 - failed model compile aborts build
 - missing output aborts build
 - zero-byte output aborts build
 - `--skip-models` bypasses compilation
 - `--force-models` rebuilds compiled outputs
 - unchanged compiled output allows HAK reuse
 - changed compiled output forces HAK rewrite
 - duplicate asset collision rules still behave as before
 ## Validator tests
 Keep and extend tests proving:
 - ASCII source models are still reported as informational
 - enabled compilation readiness errors surface when config is invalid
 ## Integration/manual acceptance tests
 Document real-world manual checks for a workstation with Wine and
 `nwnmdlcomp.exe` available.
 ### Test 1: readiness validation
 ```bash
 sow-toolkit validate
 ```
 Expected:
 - no model-compiler readiness errors when config/env are correct
 - ASCII `.mdl` info line still appears for source models
 ### Test 2: single-model HAK packaging
 Given an authored ASCII model under `paths.assets`, run:
 ```bash
 sow-toolkit build-haks --force-models
 ```
 Expected:
 - build succeeds
 - resulting HAK contains a binary `.mdl`
 - authored source file remains unchanged
 ### Test 3: incremental reuse
 Run `build-haks` twice without changes.
 Expected:
 - second run reports model reuse
 - unchanged HAKs are reused when manifest/chunk fingerprints match
 ### Test 4: source change invalidation
 Edit one ASCII `.mdl` source and rebuild.
 Expected:
 - only affected compiled output is rebuilt
 - affected HAK chunk is rewritten
 - unrelated HAK chunks remain reusable
 ### Test 5: missing NWN data
 Misconfigure `models.compile.data.root` and run build.
 Expected:
 - validation/build fails clearly before or during preparation
 - error names the missing key/data requirement
 ## Documentation Contract
 Update `README.md` to include:
 - why compiled models are shipped even though ASCII remains authored source
 - the new `models.compile` config section
 - required local dependencies:
  - `nwnmdlcomp.exe`
  - `wine`
  - `winepath`
  - NWN data root
 - environment override behavior
 - cache/staging behavior
 - `build-haks` flags related to model compilation
 - limitations and troubleshooting notes
 Optional but recommended:
 - `tools/nwnmdlcomp/README.md` for consumer repo operators
 ## Suggested Implementation Breakdown
 1. Schema and config foundation
   - add `models.compile` config structs to `internal/project/project.go`
   - add effective config defaults and path expansion in
     `internal/project/effective.go`
   - add project helper methods for compiler/cache/env resolution
 2. Validation foundation
   - extend project/config validation for enabled compilation readiness
   - extend CLI validation reporting where needed
 3. Preparation layer
   - add model compiler resolution, Wine resolution, NWN data staging, registry
     setup, and incremental metadata logic
   - add a `preparedModelAssets` representation
 4. Pipeline integration
   - plug model preparation into `planOrBuildHAKs`
   - teach `collectAssetResources` to substitute compiled model payloads
   - ensure `ContentID` and HAK reuse reflect compiled output
 5. CLI polish
   - add build-haks flags if approved
   - add progress messages and summary output
 6. Tests
   - config tests
   - validator tests
   - pipeline tests with fake compiler/Wine shims
 7. Docs
   - repo README updates
   - optional operator README under `tools/`
 ## Open Design Decisions To Resolve Before Coding
 These should be decided explicitly during implementation, not left implicit:
 1. Should the toolkit own a dedicated Wine prefix under cache by default, or
   require a user-provided prefix?
 2. Should NWN data alias files be created in a separate staging tree, or should
   the configured data root be used directly when already compatible?
 3. Is parallel model compilation stable enough to expose beyond best-effort?
 4. Do we want dedicated `build-haks` flags now, or only config for the first
   implementation?
 5. Should per-model logs always be persisted, or only on failure / debug mode?
 ## Acceptance Criteria
 The feature is complete when all of the following are true:
 - `sow-toolkit build-haks` can package compiled binary `.mdl` content from
  authored ASCII `.mdl` source when `models.compile.enabled: true`
 - authored `.mdl` files remain unchanged on disk
 - validation and topdata/autogen behaviors that depend on authored `.mdl` source
  still work
 - generated HAK manifests still reference authored asset-relative paths
 - HAK reuse logic correctly tracks compiled output changes
 - the feature is opt-in, deterministic, and tested
 - failures are actionable and name the offending model paths
 - README/config docs are updated
 ## Relevant References
 - Repo code:
  - `internal/pipeline/build.go`
  - `internal/validator/validator.go`
  - `internal/topdata/autogen.go`
  - `internal/topdata/parts_discovery.go`
  - `README.md`
 - Upstream docs consulted for this contract:
  - `nwn.wiki` MDL page: compiled models are recommended for release because
    ASCII loads more slowly
  - Neverwinter Vault `nwnmdlcomp` page: standard compile/decompile CLI usage
  - Neverwinter Vault forum discussions: `nwnmdlcomp` expects old NWN registry
    keys/data layout and some supermodel cases need additional care