Configuration Hardening

Key changes: - Added internal/project/effective.go with normalized effective config, defaults, provenance, and JSON output. - Added config subcommands: - config validate - config effective [--json|--yaml] - config inspect [<key>] - config explain <key> - config sources - Added YAML schema fields for configurable outputs, cache roots, script cache, inventory extensions, topdata output/wiki paths, and autogen cache root. - Routed existing hardcoded paths through effective config wrappers for module archives, HAK manifests/archives, script cache, music cache/credits, topdata outputs, and autogen caches. - Added validation for unsafe generated output/cache paths. - Updated command descriptions to avoid fixed build/, .cache, sow_top, etc. - Added regression tests for defaults, provenance, deterministic effective config, YAML-controlled derivation, config commands, and invalid paths.
2026-05-07 14:09:20 +02:00
parent 000d31ad24
commit 60d2de9f2d
12 changed files with 2481 additions and 62 deletions
@@ -0,0 +1,782 @@
+Create a long-term multi-repository hardening plan focused on making YAML configuration the authoritative source of truth.
+
+# Goal
+
+Reduce hardcoded behavior across all repositories and toolkit code.
+
+All behavior that touches repository-specific configuration must be driven by YAML. If a behavior currently exists only as an explicit rule in application code, move it into configuration or create a new configuration option for it.
+
+# Core principle
+
+Configuration is fact.
+
+Code should interpret and enforce configuration, not silently override it.
+
+# Requirements
+
+- YAML is the only human-authored repository configuration format.
+- JSON remains acceptable only for generated artifacts, datasets, manifests, caches, and machine-readable outputs.
+- Any code path that touches configurable behavior must lock to YAML-defined values.
+- If YAML defines a value, code must not substitute another value unless explicitly configured as a fallback.
+- If behavior varies by repository, dataset, build target, asset type, or command, it belongs in YAML.
+- If no configuration option exists for a behavior, create one.
+- Remove implicit repository assumptions from code.
+- Remove hidden defaults unless they are documented toolkit-level defaults.
+- Make all defaults visible, documented, and overrideable.
+
+# Audit scope
+
+Audit all repositories and toolkit code for hardcoded or implicit behavior related to:
+
+- paths
+- module metadata
+- resrefs
+- HAK names
+- HAK order
+- HAK grouping
+- HAK priority
+- HAK splitting
+- HAK size limits
+- optional HAKs
+- include/exclude globs
+- generated manifest names
+- cache names
+- release tags
+- dataset names
+- autogen producers
+- autogen consumers
+- derive rules
+- music prefixes
+- credits behavior
+- extract behavior
+- topdata behavior
+- package names
+- TLK names
+- build directories
+- source directories
+- validation rules
+- cleanup rules
+- normalization rules
+- naming schemes
+- file extension rules
+- ignore rules
+- command-specific behavior
+- logging verbosity
+- deterministic ordering rules
+
+# Hardening rules
+
+1. YAML wins. If YAML declares a value, the application must use it exactly.
+
+2. No silent fallbacks. Fallbacks must be explicit, documented, and visible in logs.
+
+3. No hidden repository assumptions. Repository-specific behavior must not live in toolkit code.
+
+4. No magic names. Names like `sow_top`, `sow_core`, `content`, `build`, `.cache`, or specific manifest names must come from YAML unless they are generated from a configured rule.
+
+5. No implicit behavior drift. Existing behavior should remain stable unless the YAML explicitly changes it.
+
+6. Deterministic by default. Ordering, output paths, generated names, cache keys, manifests, and logs must be stable.
+
+7. Validation before execution. Invalid or incomplete configuration should fail early with clear errors.
+
+8. Configuration gaps become schema work. If code needs a behavior that YAML cannot express, extend the schema.
+
+# Implementation direction
+
+- Build a central configuration model.
+- Route all repository behavior through that model.
+- Add typed config objects or schema validation.
+- Normalize config once at startup.
+- Pass normalized config into commands instead of reading globals or hardcoded constants.
+- Make command behavior depend on normalized config only.
+- Add explicit toolkit defaults in one place.
+- Generate an effective configuration view for debugging.
+- Add a command such as:
+
+```text
+toolkit config inspect
+toolkit config validate
+toolkit config explain <key>
+toolkit config effective
+```
+
+# Effective config
+
+Add support for printing the final resolved configuration after defaults, overrides, and dataset-specific settings have been applied.
+
+This should help verify that:
+
+- YAML was loaded.
+- defaults were applied intentionally.
+- no hidden behavior was used.
+- paths resolved as expected.
+- command behavior is explainable.
+
+# Repository hardening process
+
+For each repository:
+
+1. Inventory existing YAML config.
+2. Inventory generated JSON artifacts separately.
+3. Search code for hardcoded repository behavior.
+4. Map each hardcoded behavior to an existing YAML entry.
+5. If no YAML entry exists, add one.
+6. Update schema and validation.
+7. Update code to consume normalized config.
+8. Add tests proving YAML controls the behavior.
+9. Add regression tests for existing outputs.
+10. Document the config option.
+
+# Testing requirements
+
+Add tests for:
+
+- YAML config loading
+- config validation
+- effective config generation
+- deterministic ordering
+- missing required config
+- duplicate config entries
+- invalid paths
+- invalid HAK definitions
+- invalid autogen definitions
+- invalid dataset definitions
+- command behavior matching YAML
+- no hardcoded repository names
+- JSON artifacts remaining generated data only
+
+Add regression tests where possible:
+
+- same YAML input produces same build plan
+- same YAML input produces same HAK order
+- same YAML input produces same manifest names
+- same YAML input produces same cache paths
+- same YAML input produces same autogen behavior
+
+# Code audit targets
+
+Look specifically for:
+
+- string literals that look like paths, names, prefixes, manifests, or cache files
+- constants that duplicate config values
+- default values inside command implementations
+- repository-specific branch logic
+- command-specific assumptions
+- filename conventions not declared in config
+- implicit glob patterns
+- implicit output names
+- implicit ordering rules
+- duplicated config parsing
+- separate code paths that bypass central config
+
+# Deliverables
+
+1. Long-term hardening plan
+2. Config authority rules
+3. Audit checklist
+4. Proposed YAML schema expansions
+5. Effective config design
+6. Migration strategy per repository
+7. Testing strategy
+8. Documentation strategy
+9. List of hardcoded assumptions found
+10. Refactor tickets grouped by risk and priority
+
+# Acceptance criteria
+
+- YAML is the only human-authored repository config.
+- Code does not override YAML silently.
+- Repository-specific behavior is not hardcoded.
+- Any behavior touching config is lock-set to the normalized YAML value.
+- Missing config options are added instead of encoded in application logic.
+- Generated JSON artifacts remain unchanged unless intentionally modified.
+- Builds remain deterministic.
+- Existing repositories continue to build.
+- Developers can inspect the effective config and understand why each value was used.
+
+# Current project-needs analysis
+
+## Current state
+
+- Root repository configuration is already discovered as:
+  1. `nwn-tool.yaml`
+  2. `nwn-tool.yml`
+  3. legacy `nwn-tool.json`
+- YAML is decoded with `KnownFields(true)`.
+- Legacy JSON is decoded with `DisallowUnknownFields()`.
+- `internal/app.loadProject` prints the loaded config filename and warns when legacy JSON is used.
+- The root config model currently lives in `internal/project.Config`.
+- `internal/project.defaultConfig` currently only defaults `paths.build` to `build`.
+- Generated and machine-readable JSON remains widespread and should stay JSON:
+  - HAK manifest: `build/haks.json`
+  - canonical GFF JSON documents
+  - topdata authored/generated datasets
+  - topdata wiki state
+  - topdata deploy manifest
+  - autogen manifests and release caches
+  - credits inventory
+- Some behavior is already YAML-driven:
+  - module name/resref/description
+  - module HAK order
+  - source/assets/build paths
+  - HAK names, priorities, split behavior, max bytes, optional flag, include globs
+  - music prefix mapping
+  - topdata source/build/reference/assets/package names
+  - extract ignore extensions and post-extract module deletion
+  - autogen producer/consumer ids, roots, includes, modes, derive rules, release tags, asset names, cache names
+- Some behavior is still implicit in code and should be made explicit or documented as toolkit defaults.
+
+## Important naming clarification
+
+The current repository root config file is `nwn-tool.yaml`, not `config.yaml`.
+
+Hardening assumption:
+
+- Keep `nwn-tool.yaml` as the canonical root config filename.
+- Keep `nwn-tool.yml` as an accepted alternate YAML filename.
+- Treat `nwn-tool.json` as temporary legacy compatibility only.
+- Do not rename generated or nested JSON files.
+- Do not treat topdata dataset JSON as root repository configuration.
+
+If a future decision changes the canonical name to `config.yaml`, update `project.ConfigFile`, docs, consumer wrappers, tests, and migration scripts together.
+
+# Current hardcoded assumptions found
+
+## Project config and path defaults
+
+- `paths.build` defaults to `build` in `internal/project.defaultConfig`.
+- `HAKManifestPath` always returns `<build>/haks.json`.
+- `ModuleArchivePath` always uses `<build>/<module.resref>.mod`.
+- `HAKArchivePath` always uses `<build>/<hak.name>.hak`.
+- `SourceDir`, `AssetsDir`, and `BuildDir` directly join configured paths with the repo root and do not yet expose a richer path policy.
+- `TopDataBuildDir` falls back to `<paths.build>/topdata` when `topdata.build` is empty.
+- `TopDataUsesRepoCache` special-cases `topdata.build == .cache`.
+- `TopDataCompiled2DADir` always appends `2da`.
+- `TopDataCompiledTLKPath` uses `sow_tlk.tlk` either under `build` or topdata build output.
+- `TopDataWikiRootDir` uses `.cache/wiki` for repo-cache mode and `<topdata.build>/wiki` otherwise.
+- `TopDataWikiStatePath` always uses `state.json`.
+- `TopDataPackageHAKName` defaults to `sow_top.hak`.
+- `TopDataPackageTLKName` defaults to `sow_tlk.tlk`.
+
+Hardening direction:
+
+- Split these into documented toolkit defaults and repository-overridable fields.
+- Keep existing values as defaults only for compatibility.
+- Show all derived values in effective config output.
+
+## Inventory and extension rules
+
+- Source extensions are hardcoded in `project.SourceExtensions`.
+- Asset extensions are hardcoded in `project.AssetExtensions`.
+- `Project.Scan` treats `.json` and `.nss` as source inputs.
+- Source JSON extension detection assumes resource filenames like `<resref>.<ext>.json`.
+- Validator required fields are hardcoded, currently including `ifo -> Mod_Name`.
+- Built-in script prefixes are hardcoded in `internal/validator`.
+- Validator reference checks hardcode specific field labels and expected resource extensions:
+  - `Conversation -> .dlg`
+  - `Mod_Entry_Area -> .are`
+  - `Model -> .mdl`
+  - `Sound -> .wav`
+
+Hardening direction:
+
+- Add a `resources` or `inventory` config section for extension policy and source/asset classification.
+- Keep NWN defaults built in, but make them inspectable and overrideable.
+- Consider allowing validators to be named profiles, for example `nwn_module`, instead of requiring every repository to spell out base NWN rules.
+
+## Build and HAK behavior
+
+- `build-haks` reads/writes the hardcoded manifest name from `Project.HAKManifestPath`.
+- HAK files are always emitted under `paths.build`.
+- HAK archive filenames are always derived from `haks[].name`.
+- `extract` scans all `*.hak` in the build directory rather than only configured/generated HAKs.
+- `build-haks` has command flags that mutate behavior independently of config:
+  - `--hak`
+  - `--archive`
+  - `--source-manifest`
+  - `--plan-only`
+  - `--quiet`
+  - `--verbose`
+  - `--debug`
+- `SOW_BUILD_HAKS_KEEP_EXISTING` controls HAK cleanup behavior outside YAML.
+
+Hardening direction:
+
+- Add configurable manifest names and cleanup policy.
+- Define command-line flags as explicit per-run overrides in the effective config model.
+- Log all active environment overrides.
+- Consider config-controlled archive discovery for `extract`.
+
+## Script compiler behavior
+
+- Compiled scripts are cached under `.cache/ncs`.
+- Compiler discovery is hardcoded to:
+  - `SOW_NWN_SCRIPT_COMPILER`
+  - `tools/script-compiler/nwn_script_comp`
+  - `tools/script-compiler/nwn_script_comp.exe`
+  - `tools/nwn_script_comp`
+  - `tools/nwn_script_comp.exe`
+  - `PATH`
+- Script source directory is always `<paths.source>/scripts`.
+- Compiler environment probing uses `SOW_NWN_USER_DIRECTORY`, `NWN_HOME`, `NWN_USER_DIRECTORY`, `SOW_NWN_ROOT`, and `NWN_ROOT`.
+
+Hardening direction:
+
+- Add `tools.script_compiler` and `script.cache` config.
+- Keep environment variables as explicit override sources.
+- Include resolved compiler path and cache path in effective config and debug logs.
+
+## Music behavior
+
+- Music conversion/scanning is hardcoded around `envi/music`.
+- Music staging is hardcoded to `.cache/music`.
+- Credits output is hardcoded to `.cache/credits`.
+- Credits overlay filename is hardcoded to `CREDITS.md`.
+- FFmpeg/ffprobe discovery is environment/PATH based.
+- BMU output naming rules and 16-character NWN stem limits are hardcoded.
+
+Hardening direction:
+
+- Apply `MUSIC_REFACTOR_CONTRACT.md`.
+- Convert music behavior to dataset configuration.
+- Keep old `music.prefixes` only as compatibility shorthand until repositories migrate.
+
+## Topdata behavior
+
+- Topdata source layout is partly convention-based:
+  - `data`
+  - `tlk`
+  - `base_dialog.json`
+  - `lock.json`
+  - `.tlk_state.json`
+  - `templates`
+  - `assets`
+- Topdata compiled output subdirectories are hardcoded:
+  - `2da`
+  - `wiki`
+  - `wiki/pages`
+  - `wiki/state.json`
+- Wiki generation has hardcoded managed namespaces and status labels.
+- Wiki deploy defaults are hardcoded:
+  - `.wiki_deploy_manifest.json`
+  - `Auto-generated from native builder`
+  - managed namespaces including `classes`, `feat`, `itemtypes`, `races`, `skills`, `spells`, `meta`
+- Topdata package fallback names are `sow_top.hak` and `sow_tlk.tlk`.
+- Autogen released manifest cache path is `.cache/<cache_name>`.
+- Autogen released manifest max age is one hour.
+- `SOW_AUTOGEN_MANIFEST_REFRESH` forces refresh outside YAML.
+- Released autogen manifest discovery derives a Shadows Over Westgate assets repo spec from git remote and environment overrides.
+- Supported autogen modes are hardcoded:
+  - `parts_rows`
+  - `head_visualeffects`
+- Supported derive kinds are hardcoded:
+  - `trailing_numeric_suffix`
+  - `model_stem`
+- Supported derive group source is hardcoded:
+  - `first_path_segment`
+
+Hardening direction:
+
+- Separate topdata toolkit defaults from repository-specific defaults.
+- Add explicit config for output subpaths, wiki generation/deploy defaults, autogen cache policy, and release source.
+- Keep dataset JSON shape unchanged.
+- Treat topdata dataset internals as authored data, not root YAML config, unless the behavior genuinely varies by repository.
+
+## Extract behavior
+
+- Extract output layout is hardcoded:
+  - `.nss` -> `<paths.source>/scripts/<resref>.nss`
+  - GFF resources -> `<paths.source>/<sourceSubdir(ext)>/<resref>.<ext>.json`
+  - other resources -> `<paths.assets>/<extension>/<resref>.<extension>`
+- Stale cleanup uses the current project inventory.
+- Ignore behavior is configurable only by extension.
+
+Hardening direction:
+
+- Add extract output layout settings or named extraction profiles.
+- Keep current layout as `nwn_canonical_json`.
+- Make cleanup scope explicit and visible.
+
+## App and logging behavior
+
+- Command descriptions mention hardcoded paths like `src/`, `assets/`, `build/`, `.cache/2da`, `.cache/wiki`, `sow_top.hak`, and `sow_tlk.tlk`.
+- Verbosity flags are command-specific, especially for `build-haks`.
+- Spinner and phase labels are hardcoded in `internal/app/app.go`.
+- Config load diagnostics are not yet a full effective-config explanation.
+
+Hardening direction:
+
+- Update descriptions to avoid implying fixed repository layouts.
+- Introduce shared global verbosity flags or a documented command-level policy.
+- Expose effective config and active overrides through `config` commands.
+
+# Proposed schema expansions
+
+These are target concepts, not a requirement to implement every field in one pass.
+
+```yaml
+toolkit:
+  defaults_profile: nwn
+  generated_artifacts:
+    format: json
+  logging:
+    default_verbosity: normal
+    paths: relative
+
+paths:
+  source: src
+  assets: assets
+  build: build
+  cache: .cache
+  tools: tools
+
+outputs:
+  module_archive: "{module.resref}.mod"
+  hak_manifest: haks.json
+  hak_archive: "{hak.name}.hak"
+
+inventory:
+  source_extensions:
+    - .are
+    - .dlg
+    - .fac
+    - .git
+    - .ifo
+    - .jrl
+    - .utc
+    - .utd
+    - .ute
+    - .uti
+    - .utm
+    - .utp
+    - .uts
+    - .utt
+    - .utw
+  asset_extensions:
+    - .2da
+    - .bmu
+    - .dds
+    - .mdl
+    - .wav
+  source_json_pattern: "{resref}.{extension}.json"
+
+validation:
+  profile: nwn_module
+  builtin_script_prefixes:
+    - nw_
+    - x0_
+    - x1_
+    - x2_
+    - x3_
+    - ga_
+    - gc_
+    - gen_
+    - gui_
+    - nwg_
+    - ta_
+
+scripts:
+  source_dir: scripts
+  cache: "{paths.cache}/ncs"
+  compiler:
+    path: ""
+    env:
+      path: SOW_NWN_SCRIPT_COMPILER
+      nwn_root: SOW_NWN_ROOT
+      nwn_user_directory: SOW_NWN_USER_DIRECTORY
+    search:
+      - "{paths.tools}/script-compiler/nwn_script_comp"
+      - "{paths.tools}/script-compiler/nwn_script_comp.exe"
+      - "{paths.tools}/nwn_script_comp"
+      - "{paths.tools}/nwn_script_comp.exe"
+      - PATH:nwn_script_comp
+
+extract:
+  layout: nwn_canonical_json
+  ignore_extensions: []
+  cleanup_stale: true
+  delete_module_archive_after_success: false
+
+topdata:
+  source: topdata
+  build: "{paths.build}/topdata"
+  compiled_2da_dir: 2da
+  compiled_tlk: sow_tlk.tlk
+  package_hak: sow_top.hak
+  package_tlk: sow_tlk.tlk
+  wiki:
+    enabled_by_default: false
+    output_root: wiki
+    pages_dir: pages
+    state_file: state.json
+    managed_namespaces:
+      - classes
+      - feat
+      - itemtypes
+      - races
+      - skills
+      - spells
+    deploy_manifest: .wiki_deploy_manifest.json
+    deploy_edit_summary: Auto-generated from native builder
+
+autogen:
+  cache:
+    root: "{paths.cache}"
+    max_age: 1h
+    refresh_env: SOW_AUTOGEN_MANIFEST_REFRESH
+  release_source:
+    provider: gitea
+    server_url_env: SOW_ASSETS_SERVER_URL
+    repo_env: SOW_ASSETS_REPO
+```
+
+Implementation does not need to preserve exactly this naming, but each hardcoded behavior above should either become:
+
+- a field in schema,
+- a documented toolkit default visible in effective config, or
+- a deliberate non-configurable engine invariant with a clear rationale.
+
+# Effective config design
+
+## Commands
+
+Add config inspection commands:
+
+```text
+sow-toolkit config validate
+sow-toolkit config effective [--json|--yaml]
+sow-toolkit config inspect [<key>] [--json]
+sow-toolkit config explain <key>
+sow-toolkit config sources
+```
+
+`config validate` should load, normalize, and validate without scanning/building.
+
+`config effective` should print the complete normalized config after defaults and compatibility migrations.
+
+`config explain <key>` should show:
+
+- final value
+- source of the value:
+  - toolkit default
+  - root YAML
+  - compatibility migration
+  - environment override
+  - command-line override
+- whether the value is deprecated
+- validation rules that apply
+
+`config sources` should print:
+
+- loaded config path and format
+- legacy status
+- accepted root config candidates
+- active environment overrides
+- command-line overrides used for the current invocation
+
+## Internal model
+
+Introduce a normalized/effective config layer separate from raw YAML structs.
+
+Suggested distinction:
+
+- `RawConfig`: direct YAML representation with pointer fields where inheritance or "unset" matters.
+- `EffectiveConfig`: fully resolved values consumed by commands.
+- `ConfigProvenance`: source metadata for inspect/explain commands.
+- `ConfigDiagnostics`: warnings/errors/deprecations produced during normalization and validation.
+
+Commands should consume `EffectiveConfig` or methods derived from it, not raw YAML fields.
+
+## Override rules
+
+Order of authority:
+
+1. toolkit built-in defaults
+2. YAML root config
+3. compatibility migrations from deprecated YAML fields
+4. environment overrides
+5. command-line overrides
+
+For each override:
+
+- document whether it is allowed
+- record provenance
+- show it in debug/effective output
+- avoid silent behavior changes
+
+Environment variables should not be removed immediately, but they should be treated as visible overrides, not hidden behavior.
+
+# Hardening phases
+
+## Phase 1: Baseline audit and effective config shell
+
+- Add a config audit document generated from code search.
+- Implement `config validate`.
+- Implement `config effective --json` using current config and current defaults.
+- Add provenance for:
+  - config filename
+  - legacy JSON use
+  - `paths.build` default
+  - topdata default package names
+  - HAK manifest default name
+- Do not change build behavior in this phase.
+
+## Phase 2: Centralize derived paths and names
+
+- Move path/name derivation out of scattered methods and into effective config.
+- Cover:
+  - module archive path
+  - HAK archive path template
+  - HAK manifest path
+  - cache root
+  - script cache
+  - topdata compiled dirs
+  - topdata package names
+  - wiki output paths
+  - autogen cache root
+- Keep old methods as wrappers around effective config during migration.
+- Add tests proving old and new derivations match.
+
+## Phase 3: Schema expansion for current hardcoded behavior
+
+- Add schema fields for outputs, cache roots, inventory extensions, script compiler settings, extract layout, topdata output subpaths, wiki deploy defaults, and autogen cache policy.
+- Preserve current defaults.
+- Add strict validation for new fields.
+- Update README examples.
+
+## Phase 4: Command integration
+
+- Update build, extract, validate, compare, topdata, autogen, music, and wiki commands to consume effective config.
+- Convert environment variables into explicit override records.
+- Update logs to report configured names/paths instead of hardcoded labels.
+- Add `--debug` output for effective values used by each command.
+
+## Phase 5: Repository migrations
+
+- Update consumer repository YAML to explicitly declare any behavior that is repository-specific.
+- Remove compatibility shims only after all active repositories have migrated.
+- Keep generated JSON artifacts stable unless a separate contract says otherwise.
+- Add regression checks comparing pre/post build plans and generated manifests.
+
+## Phase 6: Enforcement
+
+- Add tests or static checks that reject new repository-specific literals in core command paths.
+- Remove legacy JSON loading after the agreed migration window.
+- Turn deprecated YAML shorthand warnings into validation errors where appropriate.
+
+# Repository-by-repository checklist
+
+For each consumer repository:
+
+1. Identify root config file and confirm it is `nwn-tool.yaml` or `nwn-tool.yml`.
+2. Confirm no root `nwn-tool.json` remains except as deliberate migration fixture.
+3. List all wrapper scripts and environment variables they set.
+4. Record local tool paths and whether they need YAML fields.
+5. List generated artifact paths and confirm they are not treated as config.
+6. Run current build and save:
+   - HAK manifest
+   - module HAK order
+   - topdata package paths
+   - autogen manifest paths
+   - credits inventory paths
+7. Convert implicit behavior to explicit YAML.
+8. Run `config effective` and verify values.
+9. Run build/validate workflows and compare generated outputs.
+10. Document intentional diffs.
+
+# Test strategy expansion
+
+Add tests for:
+
+- `config validate` succeeds without scanning missing optional trees.
+- `config effective --json` is deterministic.
+- `config explain paths.build` identifies toolkit default when omitted.
+- `config explain topdata.package_hak` identifies YAML source when configured.
+- legacy `nwn-tool.json` emits warning and provenance.
+- YAML wins over legacy JSON when both exist.
+- unknown YAML fields fail before any scan/build side effects.
+- environment overrides appear in effective config.
+- command-line overrides appear in effective config for the command invocation.
+- HAK manifest name can be configured without changing generated JSON schema.
+- topdata package names can be configured and all command output follows them.
+- script compiler cache path can be configured.
+- extract layout profile preserves current output paths.
+- autogen cache root and max age are configurable.
+- wiki deploy manifest name is configurable.
+- source/asset extension defaults match current scanning behavior.
+- invalid path traversal in output/cache config fails early.
+- absolute paths are rejected unless the field explicitly allows them.
+- duplicate or case-colliding generated output paths fail validation.
+- config-generated effective values are sorted deterministically.
+
+Regression tests:
+
+- current minimal YAML still builds to the same default paths.
+- current Shadows Over Westgate-style topdata package output remains `sow_top.hak` / `sow_tlk.tlk` unless YAML changes it.
+- current HAK manifest remains `haks.json` unless YAML changes it.
+- current `.cache` outputs remain stable unless YAML changes them.
+- old `music.prefixes` compatibility still maps to current behavior until removed.
+
+# Priority refactor tickets
+
+## P0: Authority and safety
+
+- Add effective config command output.
+- Record config provenance and legacy warnings.
+- Centralize output/cache path derivation.
+- Validate path traversal and unsafe output roots.
+- Make environment overrides visible.
+
+## P1: Repository-specific names and paths
+
+- Configure HAK manifest filename.
+- Configure cache root.
+- Configure script compiler path/search/cache.
+- Configure topdata compiled output subpaths.
+- Configure wiki deploy manifest/edit summary/namespaces.
+- Configure autogen cache root/max age/release source.
+
+## P2: Inventory and validation profiles
+
+- Configure source and asset extension lists.
+- Configure or profile validator reference rules.
+- Configure built-in script prefixes.
+- Configure extract output layout.
+
+## P3: Migration cleanup
+
+- Deprecate and remove legacy JSON root config loading.
+- Deprecate music prefix shorthand after dataset migration.
+- Replace command descriptions that mention hardcoded defaults.
+- Add static/lint checks for new repository-specific literals.
+
+# Documentation strategy
+
+- Document `nwn-tool.yaml` as the canonical root config file.
+- Maintain a generated effective-config reference in docs.
+- Document every toolkit default separately from repository examples.
+- For every config field, document:
+  - type
+  - default
+  - whether relative paths are repo-relative or output-root-relative
+  - whether absolute paths are allowed
+  - validation rules
+  - generated artifacts affected
+  - migration notes
+- Keep a clear line between:
+  - human-authored root YAML config
+  - human-authored topdata dataset JSON
+  - generated JSON artifacts
+  - compatibility/legacy files
+
+# Expanded acceptance criteria
+
+- `sow-toolkit config effective --json` shows all values that commands use.
+- Every effective value can be traced to YAML, toolkit default, environment override, compatibility migration, or command override.
+- Existing hardcoded defaults are either configurable or documented as toolkit defaults.
+- Command implementations do not invent repository-specific paths after config normalization.
+- Topdata, HAK, music, extract, autogen, wiki, and script compiler paths are inspectable.
+- Legacy JSON root config use is visible and eventually removable.
+- Generated JSON schemas remain stable unless intentionally changed.
+- Existing consumer repositories build before and after migration.