sow-tools/CONFIGURATION_REFACTOR_CONTRACT.md

Refactor toolkit configuration from JSON to YAML only.

Status Snapshot

Current state as of 2026-05-13:

• This refactor is largely implemented.
• Root config discovery now prefers nwn-tool.yaml, then nwn-tool.yml, then legacy nwn-tool.json.
• YAML is canonical in the consumer repos, and the toolkit exposes config validate|effective|inspect|explain|sources.
• Legacy JSON compatibility still exists by design and is still used heavily in test fixtures.

Remaining work:

• continue migrating representative tests from nwn-tool.json fixtures to YAML while keeping only targeted legacy coverage
• decide when to deprecate or remove legacy shorthand such as music.prefixes after consumer configs are fully dataset-based
• keep documentation/examples synchronized with the normalized YAML schema as it expands

Active Scope

This contract remains active and is one of the current priorities.

The implementation gap is no longer "load YAML at all". The real gap is configuration authority drift:

• some repository behavior is still encoded as toolkit defaults when it should become explicit schema
• some tests still model old JSON-era expectations
• some topdata behavior is still driven by explicit namespace/path checks rather than generic dataset rules

Current Findings

The most important remaining configuration work is topdata-related:

• topdata dataset discovery already derives default output names when output is absent, but validation still hard-fails for topdata/data/masterfeats/base.json when output is missing
• internal/topdata/topdata.go dispatches validation through explicit path checks for masterfeats, feat, parts overrides, and racialtypes registry
• topdata defaults in internal/project still mix generic toolkit defaults with behavior that may need clearer schema ownership

This means the config-refactor track now overlaps directly with topdata hardening. The core question is no longer file format migration; it is whether a behavior belongs in:

• root YAML
• dataset-authored JSON
• or a documented intrinsic toolkit invariant

Deferred Scope

Defer unrelated schema expansion for now:

• model compilation config
• broader asset-maintenance policy schema
• release-announcement config

Immediate Plan

1. Continue shrinking legacy nwn-tool.json coverage down to deliberate compatibility tests.
2. Audit hardcoded topdata behaviors and classify each as:
   • root YAML concern
   • dataset JSON concern
   • toolkit invariant
3. Remove outdated namespace-specific validation assumptions where generic dataset logic already exists.
4. Keep generated artifacts in JSON, but stop using explicit path-based logic when the same rule can be inferred from generic dataset shape.
5. Document every remaining intentional toolkit default in config effective terms.

Context

Repositories currently use config.json for human-authored toolkit configuration.

However, JSON is also used heavily for generated datasets, manifests, caches, and machine-readable artifacts. This makes repository configuration harder to distinguish from generated data.

Goal

Make YAML the source of truth for all human-authored repository configuration.

JSON should remain valid for generated data artifacts, but not as the preferred format for repository configuration.

Core requirements

• Replace config.json with config.yaml.
• Load repository configuration from YAML.
• Treat YAML as the canonical source-of-truth.
• Keep generated datasets, manifests, caches, and machine-readable outputs as JSON.
• Do not move dataset artifacts to YAML.
• Do not hardcode configuration behavior in code if it can reasonably be expressed in config.
• The toolkit should be deterministic.

Configuration philosophy

Configuration should control repository behavior.

Avoid hardcoded assumptions for:

• module names
• resrefs
• paths
• HAK names
• HAK priority
• HAK split behavior
• HAK size limits
• include/exclude globs
• optional HAKs
• music prefixes
• extract rules
• topdata paths/packages
• autogen producers
• autogen consumers
• manifest names
• cache names
• release tags
• dataset names
• derivation rules

If behavior varies by repository, it belongs in YAML.

Example target config

Convert this kind of repository config:

{
  "module": {
    "name": "Shadows Over Westgate",
    "resref": "sow_module",
    "description": "Shadows Over Westgate asset pipeline.",
    "hak_order": ["sow_top", "group:sow_over", "group:sow_core"]
  },
  "paths": {
    "assets": "content",
    "build": "build"
  },
  "music": {
    "prefixes": {
      "envi/music/westgate": "mus_wg_"
    }
  }
}

Into:

module:
  name: Shadows Over Westgate
  resref: sow_module
  description: Shadows Over Westgate asset pipeline.
  hak_order:
    - sow_top
    - group:sow_over
    - group:sow_core

paths:
  assets: content
  build: build

music:
  prefixes:
    envi/music/westgate: mus_wg_

Implementation requirements

• Add YAML config loading.
• Prefer config.yaml.
• Optionally accept config.yml.
• Remove reliance on config.json for repository configuration.
• If config.json support is kept temporarily, mark it as legacy/deprecated.
• If both YAML and JSON configs exist, YAML must win.
• Emit a clear warning when legacy JSON config is used.
• Preserve existing schema semantics.
• Preserve existing defaults only where they are intentional toolkit defaults.
• Move repository-specific defaults into YAML.
• Keep validation strict.
• Fail clearly on malformed YAML or unknown required fields.
• Preserve deterministic ordering where configs are serialized, logged, or normalized.

Validation requirements

• Validate YAML against the same schema expectations currently used for JSON.
• Ensure required sections are present when needed.
• Ensure paths are strings.
• Ensure HAK entries have required fields.
• Ensure autogen producers/consumers have valid ids, roots, includes, derive rules, and manifest config.
• Ensure music prefixes are deterministic and unambiguous.
• Ensure duplicate HAK names, producer ids, consumer ids, or dataset ids are rejected.
• Ensure priority ordering is deterministic even when priorities match.
• Ensure include glob ordering is stable.

Migration requirements

• Convert existing config.json files to config.yaml.

• Preserve all existing values exactly unless a change is intentional.

• Update code paths, scripts, tests, and documentation to reference YAML.

• Add or update tests for:

  • loading config.yaml
  • loading config.yml
  • legacy config.json fallback, if retained
  • YAML precedence over JSON
  • validation failures
  • deterministic ordering
  • representative configs matching the current repository examples

• Do not change generated artifact formats unless explicitly required.

Generated artifacts

Keep these as JSON unless there is a separate reason to change them:

• datasets
• manifests
• caches
• credits output
• build summaries
• machine-readable reports

Rationale:

• YAML is for human-authored repository configuration.
• JSON is for generated or machine-readable data.

Deliverables

1. YAML config loader design
2. Updated config schema
3. JSON-to-YAML migration plan
4. Converted example configs
5. Legacy JSON handling policy
6. Validation updates
7. Test plan
8. Documentation updates
9. List of code-level assumptions that should be moved into config
10. Before/after examples

Initial project-needs analysis

Current state

• Main repository configuration is currently modeled by internal/project.Config.
• The toolkit currently discovers and loads nwn-tool.json, not config.json.
• Config loading is centralized in internal/project.FindRoot and internal/project.Load.
• Config structs currently have JSON tags only.
• There is no YAML dependency in go.mod.
• Generated and machine-readable JSON is widespread and should remain JSON:
  • build HAK manifest: build/haks.json
  • topdata authored/generated datasets
  • topdata template conversion metadata: topdata/templates/config.json
  • topdata wiki state and deploy manifests
  • autogen manifest caches under .cache
  • credits inventory output
  • GFF canonical JSON documents

Naming decision needed

The contract says to replace config.json with config.yaml, but the current toolkit root config file is nwn-tool.json.

Plan assumption:

• Treat nwn-tool.yaml as the direct YAML replacement for the existing root config.
• Optionally accept nwn-tool.yml.
• Keep temporary legacy fallback to nwn-tool.json with a warning.
• Do not rename generated or nested metadata files unless they are confirmed to be human-authored repository root configuration.

If the intended final filename is instead literally config.yaml, update the loader plan before implementation.

Code areas affected

• internal/project/project.go
  • config file discovery
  • YAML/legacy JSON loading
  • schema tags and strict decode behavior
  • validation for duplicate names, path fields, include ordering, and required config sections
  • currently hardcoded defaults such as build, .cache, sow_top.hak, sow_tlk.tlk, and haks.json
• internal/app/app.go
  • project-load diagnostics and logging so commands clearly state which config file was loaded
  • user-facing descriptions that mention hardcoded paths or outputs
• internal/pipeline/*
  • HAK ordering, manifest naming, music cache/credits paths, and music prefix behavior
  • generated JSON outputs must remain JSON
• internal/topdata/*
  • package names, cache names, released manifest metadata, wiki state paths, and autogen consumer cache paths
  • topdata datasets and manifests must remain JSON
• internal/validator/*
  • validation tests and duplicate/ordering rules
  • potential config control for built-in script prefixes if repository-specific
• Tests under internal/**/*_test.go
  • many tests currently write nwn-tool.json; representative tests should move to YAML with targeted legacy JSON coverage retained
• README.md and workflow docs
  • update root configuration references to YAML
  • explicitly document JSON artifacts that remain JSON

Hardcoded assumptions to audit for config ownership

Move to YAML if repository-specific:

• root config filename policy and migration behavior
• build output directory and HAK manifest filename
• topdata build/cache paths
• topdata package HAK/TLK filenames and derived TLK behavior
• autogen release tags, asset names, cache names, producers, consumers, roots, includes, modes, and derive rules
• released parts manifest repository/server metadata when not a universal toolkit default
• HAK names, priorities, split behavior, optional flags, include globs, and module HAK order
• music source roots, prefixes, credit overlay names, generated credits paths, and music stem constraints where repository-specific
• source/asset include extension sets where repository-specific
• validator script-prefix exclusions if they vary by repository

Keep in code if toolkit-intrinsic:

• generated JSON encoding for datasets, manifests, caches, GFF documents, and reports
• NWN resource format constraints such as resref length limits
• deterministic sort rules after config values are loaded
• supported autogen derive kinds and consumer modes unless extensibility is implemented

Implementation plan

Phase 1: Loader and schema foundation

• Add YAML parsing dependency, likely gopkg.in/yaml.v3.
• Add YAML tags to all config structs while preserving JSON tags for legacy decode.
• Introduce config discovery order:
  1. nwn-tool.yaml
  2. nwn-tool.yml
  3. legacy nwn-tool.json
• Return loaded config metadata from project.Load, including path, format, and legacy flag.
• Make YAML win when both YAML and JSON are present.
• Emit a clear legacy warning when nwn-tool.json is used.
• Emit/log the selected config path for normal command execution.
• Fail clearly for malformed YAML and unknown fields.

Phase 2: Validation and determinism

• Preserve existing schema semantics while tightening validation.
• Validate required module fields and path field types.
• Reject duplicate HAK names, autogen producer IDs, autogen consumer IDs, and dataset IDs where represented in root config.
• Validate HAK entries for name, priority, max bytes, include globs, and output-safe names.
• Validate autogen manifest filenames and deterministic producer/consumer references.
• Validate music prefixes for ambiguous duplicate normalized roots.
• Ensure include globs and map-derived outputs are sorted before use where order matters.
• Add deterministic tie-breakers for HAK priority sorting.

Phase 3: Move repository-specific defaults into YAML

• Keep only intentional toolkit defaults in defaultConfig.
• Move Shadows Over Westgate-specific defaults into repo YAML:
  • top package names
  • topdata cache/build paths
  • autogen manifest metadata
  • music prefixes
  • HAK ordering and include globs
• Add config fields where needed before removing hardcoded assumptions.
• Leave generated artifact paths as JSON outputs unless the path itself must be configurable.

Phase 4: Migration

• Convert consumer root configs from nwn-tool.json to nwn-tool.yaml.
• Preserve existing values exactly unless a separate behavior change is documented.
• Keep topdata/templates/config.json unchanged for now because it is topdata conversion metadata, not root repository configuration.
• Keep generated datasets, manifests, caches, credits inventory, and GFF JSON unchanged.
• Add before/after examples for root config migration.

Phase 5: Tests

• Add focused loader tests for:
  • nwn-tool.yaml
  • nwn-tool.yml
  • legacy nwn-tool.json
  • YAML precedence over JSON
  • malformed YAML
  • unknown YAML fields
• Update representative build, validator, pipeline, topdata, and app tests to use YAML root config.
• Keep targeted JSON tests only for legacy fallback and generated artifact behavior.
• Add validation tests for duplicates, path type failures, deterministic HAK ordering, and ambiguous music prefixes.
• Run go test ./....

Phase 6: Documentation

• Update README.md repository layout and consumer instructions to reference nwn-tool.yaml.
• Document legacy JSON fallback and planned removal policy.
• Document which JSON files remain intentional generated or machine-readable artifacts.
• Document config precedence and command log behavior.

Acceptance criteria

• A repository can build from the chosen YAML root config file (nwn-tool.yaml under the current plan, or config.yaml if the naming decision changes).
• Existing generated JSON datasets and manifests still work.
• No repository-specific behavior remains hardcoded if it can be expressed in YAML.
• Builds remain deterministic.
• Existing behavior is preserved after migration.
• Logs clearly state which config file was loaded.