sow-tools/CONFIGURATION_HARDENING_CONTRACT.md

Create a long-term multi-repository hardening plan focused on making YAML configuration the authoritative source of truth.

Goal

Reduce hardcoded behavior across all repositories and toolkit code.

All behavior that touches repository-specific configuration must be driven by YAML. If a behavior currently exists only as an explicit rule in application code, move it into configuration or create a new configuration option for it.

Core principle

Configuration is fact.

Code should interpret and enforce configuration, not silently override it.

Requirements

• YAML is the only human-authored repository configuration format.
• JSON remains acceptable only for generated artifacts, datasets, manifests, caches, and machine-readable outputs.
• Any code path that touches configurable behavior must lock to YAML-defined values.
• If YAML defines a value, code must not substitute another value unless explicitly configured as a fallback.
• If behavior varies by repository, dataset, build target, asset type, or command, it belongs in YAML.
• If no configuration option exists for a behavior, create one.
• Remove implicit repository assumptions from code.
• Remove hidden defaults unless they are documented toolkit-level defaults.
• Make all defaults visible, documented, and overrideable.

Audit scope

Audit all repositories and toolkit code for hardcoded or implicit behavior related to:

• paths
• module metadata
• resrefs
• HAK names
• HAK order
• HAK grouping
• HAK priority
• HAK splitting
• HAK size limits
• optional HAKs
• include/exclude globs
• generated manifest names
• cache names
• release tags
• dataset names
• autogen producers
• autogen consumers
• derive rules
• music prefixes
• credits behavior
• extract behavior
• topdata behavior
• package names
• TLK names
• build directories
• source directories
• validation rules
• cleanup rules
• normalization rules
• naming schemes
• file extension rules
• ignore rules
• command-specific behavior
• logging verbosity
• deterministic ordering rules

Hardening rules

1. YAML wins. If YAML declares a value, the application must use it exactly.

2. No silent fallbacks. Fallbacks must be explicit, documented, and visible in logs.

3. No hidden repository assumptions. Repository-specific behavior must not live in toolkit code.

4. No magic names. Names like sow_top, sow_core, content, build, .cache, or specific manifest names must come from YAML unless they are generated from a configured rule.

5. No implicit behavior drift. Existing behavior should remain stable unless the YAML explicitly changes it.

6. Deterministic by default. Ordering, output paths, generated names, cache keys, manifests, and logs must be stable.

7. Validation before execution. Invalid or incomplete configuration should fail early with clear errors.

8. Configuration gaps become schema work. If code needs a behavior that YAML cannot express, extend the schema.

Implementation direction

• Build a central configuration model.
• Route all repository behavior through that model.
• Add typed config objects or schema validation.
• Normalize config once at startup.
• Pass normalized config into commands instead of reading globals or hardcoded constants.
• Make command behavior depend on normalized config only.
• Add explicit toolkit defaults in one place.
• Generate an effective configuration view for debugging.
• Add a command such as:

toolkit config inspect
toolkit config validate
toolkit config explain <key>
toolkit config effective

Effective config

Add support for printing the final resolved configuration after defaults, overrides, and dataset-specific settings have been applied.

This should help verify that:

• YAML was loaded.
• defaults were applied intentionally.
• no hidden behavior was used.
• paths resolved as expected.
• command behavior is explainable.

Repository hardening process

For each repository:

1. Inventory existing YAML config.
2. Inventory generated JSON artifacts separately.
3. Search code for hardcoded repository behavior.
4. Map each hardcoded behavior to an existing YAML entry.
5. If no YAML entry exists, add one.
6. Update schema and validation.
7. Update code to consume normalized config.
8. Add tests proving YAML controls the behavior.
9. Add regression tests for existing outputs.
10. Document the config option.

Testing requirements

Add tests for:

• YAML config loading
• config validation
• effective config generation
• deterministic ordering
• missing required config
• duplicate config entries
• invalid paths
• invalid HAK definitions
• invalid autogen definitions
• invalid dataset definitions
• command behavior matching YAML
• no hardcoded repository names
• JSON artifacts remaining generated data only

Add regression tests where possible:

• same YAML input produces same build plan
• same YAML input produces same HAK order
• same YAML input produces same manifest names
• same YAML input produces same cache paths
• same YAML input produces same autogen behavior

Code audit targets

Look specifically for:

• string literals that look like paths, names, prefixes, manifests, or cache files
• constants that duplicate config values
• default values inside command implementations
• repository-specific branch logic
• command-specific assumptions
• filename conventions not declared in config
• implicit glob patterns
• implicit output names
• implicit ordering rules
• duplicated config parsing
• separate code paths that bypass central config

Deliverables

1. Long-term hardening plan
2. Config authority rules
3. Audit checklist
4. Proposed YAML schema expansions
5. Effective config design
6. Migration strategy per repository
7. Testing strategy
8. Documentation strategy
9. List of hardcoded assumptions found
10. Refactor tickets grouped by risk and priority

Acceptance criteria

• YAML is the only human-authored repository config.
• Code does not override YAML silently.
• Repository-specific behavior is not hardcoded.
• Any behavior touching config is lock-set to the normalized YAML value.
• Missing config options are added instead of encoded in application logic.
• Generated JSON artifacts remain unchanged unless intentionally modified.
• Builds remain deterministic.
• Existing repositories continue to build.
• Developers can inspect the effective config and understand why each value was used.

Current project-needs analysis

Current state

• Root repository configuration is already discovered as:
  1. nwn-tool.yaml
  2. nwn-tool.yml
  3. legacy nwn-tool.json
• YAML is decoded with KnownFields(true).
• Legacy JSON is decoded with DisallowUnknownFields().
• internal/app.loadProject prints the loaded config filename and warns when legacy JSON is used.
• The root config model currently lives in internal/project.Config.
• internal/project.defaultConfig currently only defaults paths.build to build.
• Generated and machine-readable JSON remains widespread and should stay JSON:
  • HAK manifest: build/haks.json
  • canonical GFF JSON documents
  • topdata authored/generated datasets
  • topdata wiki state
  • topdata deploy manifest
  • autogen manifests and release caches
  • credits inventory
• Some behavior is already YAML-driven:
  • module name/resref/description
  • module HAK order
  • source/assets/build paths
  • HAK names, priorities, split behavior, max bytes, optional flag, include globs
  • music prefix mapping
  • topdata source/build/reference/assets/package names
  • extract ignore extensions and post-extract module deletion
  • autogen producer/consumer ids, roots, includes, modes, derive rules, release tags, asset names, cache names
• Some behavior is still implicit in code and should be made explicit or documented as toolkit defaults.

Important naming clarification

The current repository root config file is nwn-tool.yaml, not config.yaml.

Hardening assumption:

• Keep nwn-tool.yaml as the canonical root config filename.
• Keep nwn-tool.yml as an accepted alternate YAML filename.
• Treat nwn-tool.json as temporary legacy compatibility only.
• Do not rename generated or nested JSON files.
• Do not treat topdata dataset JSON as root repository configuration.

If a future decision changes the canonical name to config.yaml, update project.ConfigFile, docs, consumer wrappers, tests, and migration scripts together.

Current hardcoded assumptions found

Project config and path defaults

• paths.build defaults to build in internal/project.defaultConfig.
• HAKManifestPath always returns <build>/haks.json.
• ModuleArchivePath always uses <build>/<module.resref>.mod.
• HAKArchivePath always uses <build>/<hak.name>.hak.
• SourceDir, AssetsDir, and BuildDir directly join configured paths with the repo root and do not yet expose a richer path policy.
• TopDataBuildDir falls back to <paths.build>/topdata when topdata.build is empty.
• TopDataUsesRepoCache special-cases topdata.build == .cache.
• TopDataCompiled2DADir always appends 2da.
• TopDataCompiledTLKPath uses sow_tlk.tlk either under build or topdata build output.
• TopDataWikiRootDir uses .cache/wiki for repo-cache mode and <topdata.build>/wiki otherwise.
• TopDataWikiStatePath always uses state.json.
• TopDataPackageHAKName defaults to sow_top.hak.
• TopDataPackageTLKName defaults to sow_tlk.tlk.

Hardening direction:

• Split these into documented toolkit defaults and repository-overridable fields.
• Keep existing values as defaults only for compatibility.
• Show all derived values in effective config output.

Inventory and extension rules

• Source extensions are hardcoded in project.SourceExtensions.
• Asset extensions are hardcoded in project.AssetExtensions.
• Project.Scan treats .json and .nss as source inputs.
• Source JSON extension detection assumes resource filenames like <resref>.<ext>.json.
• Validator required fields are hardcoded, currently including ifo -> Mod_Name.
• Built-in script prefixes are hardcoded in internal/validator.
• Validator reference checks hardcode specific field labels and expected resource extensions:
  • Conversation -> .dlg
  • Mod_Entry_Area -> .are
  • Model -> .mdl
  • Sound -> .wav

Hardening direction:

• Add a resources or inventory config section for extension policy and source/asset classification.
• Keep NWN defaults built in, but make them inspectable and overrideable.
• Consider allowing validators to be named profiles, for example nwn_module, instead of requiring every repository to spell out base NWN rules.

Build and HAK behavior

• build-haks reads/writes the hardcoded manifest name from Project.HAKManifestPath.
• HAK files are always emitted under paths.build.
• HAK archive filenames are always derived from haks[].name.
• extract scans all *.hak in the build directory rather than only configured/generated HAKs.
• build-haks has command flags that mutate behavior independently of config:
  • --hak
  • --archive
  • --source-manifest
  • --plan-only
  • --quiet
  • --verbose
  • --debug
• SOW_BUILD_HAKS_KEEP_EXISTING controls HAK cleanup behavior outside YAML.

Hardening direction:

• Add configurable manifest names and cleanup policy.
• Define command-line flags as explicit per-run overrides in the effective config model.
• Log all active environment overrides.
• Consider config-controlled archive discovery for extract.

Script compiler behavior

• Compiled scripts are cached under .cache/ncs.
• Compiler discovery is hardcoded to:
  • SOW_NWN_SCRIPT_COMPILER
  • tools/script-compiler/nwn_script_comp
  • tools/script-compiler/nwn_script_comp.exe
  • tools/nwn_script_comp
  • tools/nwn_script_comp.exe
  • PATH
• Script source directory is always <paths.source>/scripts.
• Compiler environment probing uses SOW_NWN_USER_DIRECTORY, NWN_HOME, NWN_USER_DIRECTORY, SOW_NWN_ROOT, and NWN_ROOT.

Hardening direction:

• Add tools.script_compiler and script.cache config.
• Keep environment variables as explicit override sources.
• Include resolved compiler path and cache path in effective config and debug logs.

Music behavior

• Music conversion/scanning is hardcoded around envi/music.
• Music staging is hardcoded to .cache/music.
• Credits output is hardcoded to .cache/credits.
• Credits overlay filename is hardcoded to CREDITS.md.
• FFmpeg/ffprobe discovery is environment/PATH based.
• BMU output naming rules and 16-character NWN stem limits are hardcoded.

Hardening direction:

• Apply MUSIC_REFACTOR_CONTRACT.md.
• Convert music behavior to dataset configuration.
• Keep old music.prefixes only as compatibility shorthand until repositories migrate.

Topdata behavior

• Topdata source layout is partly convention-based:
  • data
  • tlk
  • base_dialog.json
  • lock.json
  • .tlk_state.json
  • templates
  • assets
• Topdata compiled output subdirectories are hardcoded:
  • 2da
  • wiki
  • wiki/pages
  • wiki/state.json
• Wiki generation has hardcoded managed namespaces and status labels.
• Wiki deploy defaults are hardcoded:
  • .wiki_deploy_manifest.json
  • Auto-generated from native builder
  • managed namespaces including classes, feat, itemtypes, races, skills, spells, meta
• Topdata package fallback names are sow_top.hak and sow_tlk.tlk.
• Autogen released manifest cache path is .cache/<cache_name>.
• Autogen released manifest max age is one hour.
• SOW_AUTOGEN_MANIFEST_REFRESH forces refresh outside YAML.
• Released autogen manifest discovery derives a Shadows Over Westgate assets repo spec from git remote and environment overrides.
• Supported autogen modes are hardcoded:
  • parts_rows
  • head_visualeffects
• Supported derive kinds are hardcoded:
  • trailing_numeric_suffix
  • model_stem
• Supported derive group source is hardcoded:
  • first_path_segment

Hardening direction:

• Separate topdata toolkit defaults from repository-specific defaults.
• Add explicit config for output subpaths, wiki generation/deploy defaults, autogen cache policy, and release source.
• Keep dataset JSON shape unchanged.
• Treat topdata dataset internals as authored data, not root YAML config, unless the behavior genuinely varies by repository.

Extract behavior

• Extract output layout is hardcoded:
  • .nss -> <paths.source>/scripts/<resref>.nss
  • GFF resources -> <paths.source>/<sourceSubdir(ext)>/<resref>.<ext>.json
  • other resources -> <paths.assets>/<extension>/<resref>.<extension>
• Stale cleanup uses the current project inventory.
• Ignore behavior is configurable only by extension.

Hardening direction:

• Add extract output layout settings or named extraction profiles.
• Keep current layout as nwn_canonical_json.
• Make cleanup scope explicit and visible.

App and logging behavior

• Command descriptions mention hardcoded paths like src/, assets/, build/, .cache/2da, .cache/wiki, sow_top.hak, and sow_tlk.tlk.
• Verbosity flags are command-specific, especially for build-haks.
• Spinner and phase labels are hardcoded in internal/app/app.go.
• Config load diagnostics are not yet a full effective-config explanation.

Hardening direction:

• Update descriptions to avoid implying fixed repository layouts.
• Introduce shared global verbosity flags or a documented command-level policy.
• Expose effective config and active overrides through config commands.

Proposed schema expansions

These are target concepts, not a requirement to implement every field in one pass.

toolkit:
  defaults_profile: nwn
  generated_artifacts:
    format: json
  logging:
    default_verbosity: normal
    paths: relative

paths:
  source: src
  assets: assets
  build: build
  cache: .cache
  tools: tools

outputs:
  module_archive: "{module.resref}.mod"
  hak_manifest: haks.json
  hak_archive: "{hak.name}.hak"

inventory:
  source_extensions:
    - .are
    - .dlg
    - .fac
    - .git
    - .ifo
    - .jrl
    - .utc
    - .utd
    - .ute
    - .uti
    - .utm
    - .utp
    - .uts
    - .utt
    - .utw
  asset_extensions:
    - .2da
    - .bmu
    - .dds
    - .mdl
    - .wav
  source_json_pattern: "{resref}.{extension}.json"

validation:
  profile: nwn_module
  builtin_script_prefixes:
    - nw_
    - x0_
    - x1_
    - x2_
    - x3_
    - ga_
    - gc_
    - gen_
    - gui_
    - nwg_
    - ta_

scripts:
  source_dir: scripts
  cache: "{paths.cache}/ncs"
  compiler:
    path: ""
    env:
      path: SOW_NWN_SCRIPT_COMPILER
      nwn_root: SOW_NWN_ROOT
      nwn_user_directory: SOW_NWN_USER_DIRECTORY
    search:
      - "{paths.tools}/script-compiler/nwn_script_comp"
      - "{paths.tools}/script-compiler/nwn_script_comp.exe"
      - "{paths.tools}/nwn_script_comp"
      - "{paths.tools}/nwn_script_comp.exe"
      - PATH:nwn_script_comp

extract:
  layout: nwn_canonical_json
  ignore_extensions: []
  cleanup_stale: true
  delete_module_archive_after_success: false

topdata:
  source: topdata
  build: "{paths.build}/topdata"
  compiled_2da_dir: 2da
  compiled_tlk: sow_tlk.tlk
  package_hak: sow_top.hak
  package_tlk: sow_tlk.tlk
  wiki:
    enabled_by_default: false
    output_root: wiki
    pages_dir: pages
    state_file: state.json
    managed_namespaces:
      - classes
      - feat
      - itemtypes
      - races
      - skills
      - spells
    deploy_manifest: .wiki_deploy_manifest.json
    deploy_edit_summary: Auto-generated from native builder

autogen:
  cache:
    root: "{paths.cache}"
    max_age: 1h
    refresh_env: SOW_AUTOGEN_MANIFEST_REFRESH
  release_source:
    provider: gitea
    server_url_env: SOW_ASSETS_SERVER_URL
    repo_env: SOW_ASSETS_REPO

Implementation does not need to preserve exactly this naming, but each hardcoded behavior above should either become:

• a field in schema,
• a documented toolkit default visible in effective config, or
• a deliberate non-configurable engine invariant with a clear rationale.

Effective config design

Commands

Add config inspection commands:

sow-toolkit config validate
sow-toolkit config effective [--json|--yaml]
sow-toolkit config inspect [<key>] [--json]
sow-toolkit config explain <key>
sow-toolkit config sources

config validate should load, normalize, and validate without scanning/building.

config effective should print the complete normalized config after defaults and compatibility migrations.

config explain <key> should show:

• final value
• source of the value:
  • toolkit default
  • root YAML
  • compatibility migration
  • environment override
  • command-line override
• whether the value is deprecated
• validation rules that apply

config sources should print:

• loaded config path and format
• legacy status
• accepted root config candidates
• active environment overrides
• command-line overrides used for the current invocation

Internal model

Introduce a normalized/effective config layer separate from raw YAML structs.

Suggested distinction:

• RawConfig: direct YAML representation with pointer fields where inheritance or "unset" matters.
• EffectiveConfig: fully resolved values consumed by commands.
• ConfigProvenance: source metadata for inspect/explain commands.
• ConfigDiagnostics: warnings/errors/deprecations produced during normalization and validation.

Commands should consume EffectiveConfig or methods derived from it, not raw YAML fields.

Override rules

Order of authority:

1. toolkit built-in defaults
2. YAML root config
3. compatibility migrations from deprecated YAML fields
4. environment overrides
5. command-line overrides

For each override:

• document whether it is allowed
• record provenance
• show it in debug/effective output
• avoid silent behavior changes

Environment variables should not be removed immediately, but they should be treated as visible overrides, not hidden behavior.

Hardening phases

Phase 1: Baseline audit and effective config shell

• Add a config audit document generated from code search.
• Implement config validate.
• Implement config effective --json using current config and current defaults.
• Add provenance for:
  • config filename
  • legacy JSON use
  • paths.build default
  • topdata default package names
  • HAK manifest default name
• Do not change build behavior in this phase.

Phase 2: Centralize derived paths and names

• Move path/name derivation out of scattered methods and into effective config.
• Cover:
  • module archive path
  • HAK archive path template
  • HAK manifest path
  • cache root
  • script cache
  • topdata compiled dirs
  • topdata package names
  • wiki output paths
  • autogen cache root
• Keep old methods as wrappers around effective config during migration.
• Add tests proving old and new derivations match.

Phase 3: Schema expansion for current hardcoded behavior

• Add schema fields for outputs, cache roots, inventory extensions, script compiler settings, extract layout, topdata output subpaths, wiki deploy defaults, and autogen cache policy.
• Preserve current defaults.
• Add strict validation for new fields.
• Update README examples.

Phase 4: Command integration

• Update build, extract, validate, compare, topdata, autogen, music, and wiki commands to consume effective config.
• Convert environment variables into explicit override records.
• Update logs to report configured names/paths instead of hardcoded labels.
• Add --debug output for effective values used by each command.

Phase 5: Repository migrations

• Update consumer repository YAML to explicitly declare any behavior that is repository-specific.
• Remove compatibility shims only after all active repositories have migrated.
• Keep generated JSON artifacts stable unless a separate contract says otherwise.
• Add regression checks comparing pre/post build plans and generated manifests.

Phase 6: Enforcement

• Add tests or static checks that reject new repository-specific literals in core command paths.
• Remove legacy JSON loading after the agreed migration window.
• Turn deprecated YAML shorthand warnings into validation errors where appropriate.

Repository-by-repository checklist

For each consumer repository:

1. Identify root config file and confirm it is nwn-tool.yaml or nwn-tool.yml.
2. Confirm no root nwn-tool.json remains except as deliberate migration fixture.
3. List all wrapper scripts and environment variables they set.
4. Record local tool paths and whether they need YAML fields.
5. List generated artifact paths and confirm they are not treated as config.
6. Run current build and save:
   • HAK manifest
   • module HAK order
   • topdata package paths
   • autogen manifest paths
   • credits inventory paths
7. Convert implicit behavior to explicit YAML.
8. Run config effective and verify values.
9. Run build/validate workflows and compare generated outputs.
10. Document intentional diffs.

Test strategy expansion

Add tests for:

• config validate succeeds without scanning missing optional trees.
• config effective --json is deterministic.
• config explain paths.build identifies toolkit default when omitted.
• config explain topdata.package_hak identifies YAML source when configured.
• legacy nwn-tool.json emits warning and provenance.
• YAML wins over legacy JSON when both exist.
• unknown YAML fields fail before any scan/build side effects.
• environment overrides appear in effective config.
• command-line overrides appear in effective config for the command invocation.
• HAK manifest name can be configured without changing generated JSON schema.
• topdata package names can be configured and all command output follows them.
• script compiler cache path can be configured.
• extract layout profile preserves current output paths.
• autogen cache root and max age are configurable.
• wiki deploy manifest name is configurable.
• source/asset extension defaults match current scanning behavior.
• invalid path traversal in output/cache config fails early.
• absolute paths are rejected unless the field explicitly allows them.
• duplicate or case-colliding generated output paths fail validation.
• config-generated effective values are sorted deterministically.

Regression tests:

• current minimal YAML still builds to the same default paths.
• current Shadows Over Westgate-style topdata package output remains sow_top.hak / sow_tlk.tlk unless YAML changes it.
• current HAK manifest remains haks.json unless YAML changes it.
• current .cache outputs remain stable unless YAML changes them.
• old music.prefixes compatibility still maps to current behavior until removed.

Priority refactor tickets

P0: Authority and safety

• Add effective config command output.
• Record config provenance and legacy warnings.
• Centralize output/cache path derivation.
• Validate path traversal and unsafe output roots.
• Make environment overrides visible.

P1: Repository-specific names and paths

• Configure HAK manifest filename.
• Configure cache root.
• Configure script compiler path/search/cache.
• Configure topdata compiled output subpaths.
• Configure wiki deploy manifest/edit summary/namespaces.
• Configure autogen cache root/max age/release source.

P2: Inventory and validation profiles

• Configure source and asset extension lists.
• Configure or profile validator reference rules.
• Configure built-in script prefixes.
• Configure extract output layout.

P3: Migration cleanup

• Deprecate and remove legacy JSON root config loading.
• Deprecate music prefix shorthand after dataset migration.
• Replace command descriptions that mention hardcoded defaults.
• Add static/lint checks for new repository-specific literals.

Documentation strategy

• Document nwn-tool.yaml as the canonical root config file.
• Maintain a generated effective-config reference in docs.
• Document every toolkit default separately from repository examples.
• For every config field, document:
  • type
  • default
  • whether relative paths are repo-relative or output-root-relative
  • whether absolute paths are allowed
  • validation rules
  • generated artifacts affected
  • migration notes
• Keep a clear line between:
  • human-authored root YAML config
  • human-authored topdata dataset JSON
  • generated JSON artifacts
  • compatibility/legacy files

Expanded acceptance criteria

• sow-toolkit config effective --json shows all values that commands use.
• Every effective value can be traced to YAML, toolkit default, environment override, compatibility migration, or command override.
• Existing hardcoded defaults are either configurable or documented as toolkit defaults.
• Command implementations do not invent repository-specific paths after config normalization.
• Topdata, HAK, music, extract, autogen, wiki, and script compiler paths are inspectable.
• Legacy JSON root config use is visible and eventually removable.
• Generated JSON schemas remain stable unless intentionally changed.
• Existing consumer repositories build before and after migration.