sow-tools/CONFIGURATION_REFACTOR_CONTRACT.md

Refactor toolkit configuration from JSON to YAML only.

Context

Repositories currently use config.json for human-authored toolkit configuration.

However, JSON is also used heavily for generated datasets, manifests, caches, and machine-readable artifacts. This makes repository configuration harder to distinguish from generated data.

Goal

Make YAML the source of truth for all human-authored repository configuration.

JSON should remain valid for generated data artifacts, but not as the preferred format for repository configuration.

Core requirements

• Replace config.json with config.yaml.
• Load repository configuration from YAML.
• Treat YAML as the canonical source-of-truth.
• Keep generated datasets, manifests, caches, and machine-readable outputs as JSON.
• Do not move dataset artifacts to YAML.
• Do not hardcode configuration behavior in code if it can reasonably be expressed in config.
• The toolkit should be deterministic.

Configuration philosophy

Configuration should control repository behavior.

Avoid hardcoded assumptions for:

• module names
• resrefs
• paths
• HAK names
• HAK priority
• HAK split behavior
• HAK size limits
• include/exclude globs
• optional HAKs
• music prefixes
• extract rules
• topdata paths/packages
• autogen producers
• autogen consumers
• manifest names
• cache names
• release tags
• dataset names
• derivation rules

If behavior varies by repository, it belongs in YAML.

Example target config

Convert this kind of repository config:

{
  "module": {
    "name": "Shadows Over Westgate",
    "resref": "sow_module",
    "description": "Shadows Over Westgate asset pipeline.",
    "hak_order": ["sow_top", "group:sow_over", "group:sow_core"]
  },
  "paths": {
    "assets": "content",
    "build": "build"
  },
  "music": {
    "prefixes": {
      "envi/music/westgate": "mus_wg_"
    }
  }
}

Into:

module:
  name: Shadows Over Westgate
  resref: sow_module
  description: Shadows Over Westgate asset pipeline.
  hak_order:
    - sow_top
    - group:sow_over
    - group:sow_core

paths:
  assets: content
  build: build

music:
  prefixes:
    envi/music/westgate: mus_wg_

Implementation requirements

• Add YAML config loading.
• Prefer config.yaml.
• Optionally accept config.yml.
• Remove reliance on config.json for repository configuration.
• If config.json support is kept temporarily, mark it as legacy/deprecated.
• If both YAML and JSON configs exist, YAML must win.
• Emit a clear warning when legacy JSON config is used.
• Preserve existing schema semantics.
• Preserve existing defaults only where they are intentional toolkit defaults.
• Move repository-specific defaults into YAML.
• Keep validation strict.
• Fail clearly on malformed YAML or unknown required fields.
• Preserve deterministic ordering where configs are serialized, logged, or normalized.

Validation requirements

• Validate YAML against the same schema expectations currently used for JSON.
• Ensure required sections are present when needed.
• Ensure paths are strings.
• Ensure HAK entries have required fields.
• Ensure autogen producers/consumers have valid ids, roots, includes, derive rules, and manifest config.
• Ensure music prefixes are deterministic and unambiguous.
• Ensure duplicate HAK names, producer ids, consumer ids, or dataset ids are rejected.
• Ensure priority ordering is deterministic even when priorities match.
• Ensure include glob ordering is stable.

Migration requirements

• Convert existing config.json files to config.yaml.

• Preserve all existing values exactly unless a change is intentional.

• Update code paths, scripts, tests, and documentation to reference YAML.

• Add or update tests for:

  • loading config.yaml
  • loading config.yml
  • legacy config.json fallback, if retained
  • YAML precedence over JSON
  • validation failures
  • deterministic ordering
  • representative configs matching the current repository examples

• Do not change generated artifact formats unless explicitly required.

Generated artifacts

Keep these as JSON unless there is a separate reason to change them:

• datasets
• manifests
• caches
• credits output
• build summaries
• machine-readable reports

Rationale:

• YAML is for human-authored repository configuration.
• JSON is for generated or machine-readable data.

Deliverables

1. YAML config loader design
2. Updated config schema
3. JSON-to-YAML migration plan
4. Converted example configs
5. Legacy JSON handling policy
6. Validation updates
7. Test plan
8. Documentation updates
9. List of code-level assumptions that should be moved into config
10. Before/after examples

Initial project-needs analysis

Current state

• Main repository configuration is currently modeled by internal/project.Config.
• The toolkit currently discovers and loads nwn-tool.json, not config.json.
• Config loading is centralized in internal/project.FindRoot and internal/project.Load.
• Config structs currently have JSON tags only.
• There is no YAML dependency in go.mod.
• Generated and machine-readable JSON is widespread and should remain JSON:
  • build HAK manifest: build/haks.json
  • topdata authored/generated datasets
  • topdata template conversion metadata: topdata/templates/config.json
  • topdata wiki state and deploy manifests
  • autogen manifest caches under .cache
  • credits inventory output
  • GFF canonical JSON documents

Naming decision needed

The contract says to replace config.json with config.yaml, but the current toolkit root config file is nwn-tool.json.

Plan assumption:

• Treat nwn-tool.yaml as the direct YAML replacement for the existing root config.
• Optionally accept nwn-tool.yml.
• Keep temporary legacy fallback to nwn-tool.json with a warning.
• Do not rename generated or nested metadata files unless they are confirmed to be human-authored repository root configuration.

If the intended final filename is instead literally config.yaml, update the loader plan before implementation.

Code areas affected

• internal/project/project.go
  • config file discovery
  • YAML/legacy JSON loading
  • schema tags and strict decode behavior
  • validation for duplicate names, path fields, include ordering, and required config sections
  • currently hardcoded defaults such as build, .cache, sow_top.hak, sow_tlk.tlk, and haks.json
• internal/app/app.go
  • project-load diagnostics and logging so commands clearly state which config file was loaded
  • user-facing descriptions that mention hardcoded paths or outputs
• internal/pipeline/*
  • HAK ordering, manifest naming, music cache/credits paths, and music prefix behavior
  • generated JSON outputs must remain JSON
• internal/topdata/*
  • package names, cache names, released manifest metadata, wiki state paths, and autogen consumer cache paths
  • topdata datasets and manifests must remain JSON
• internal/validator/*
  • validation tests and duplicate/ordering rules
  • potential config control for built-in script prefixes if repository-specific
• Tests under internal/**/*_test.go
  • many tests currently write nwn-tool.json; representative tests should move to YAML with targeted legacy JSON coverage retained
• README.md and workflow docs
  • update root configuration references to YAML
  • explicitly document JSON artifacts that remain JSON

Hardcoded assumptions to audit for config ownership

Move to YAML if repository-specific:

• root config filename policy and migration behavior
• build output directory and HAK manifest filename
• topdata build/cache paths
• topdata package HAK/TLK filenames and derived TLK behavior
• autogen release tags, asset names, cache names, producers, consumers, roots, includes, modes, and derive rules
• released parts manifest repository/server metadata when not a universal toolkit default
• HAK names, priorities, split behavior, optional flags, include globs, and module HAK order
• music source roots, prefixes, credit overlay names, generated credits paths, and music stem constraints where repository-specific
• source/asset include extension sets where repository-specific
• validator script-prefix exclusions if they vary by repository

Keep in code if toolkit-intrinsic:

• generated JSON encoding for datasets, manifests, caches, GFF documents, and reports
• NWN resource format constraints such as resref length limits
• deterministic sort rules after config values are loaded
• supported autogen derive kinds and consumer modes unless extensibility is implemented

Implementation plan

Phase 1: Loader and schema foundation

• Add YAML parsing dependency, likely gopkg.in/yaml.v3.
• Add YAML tags to all config structs while preserving JSON tags for legacy decode.
• Introduce config discovery order:
  1. nwn-tool.yaml
  2. nwn-tool.yml
  3. legacy nwn-tool.json
• Return loaded config metadata from project.Load, including path, format, and legacy flag.
• Make YAML win when both YAML and JSON are present.
• Emit a clear legacy warning when nwn-tool.json is used.
• Emit/log the selected config path for normal command execution.
• Fail clearly for malformed YAML and unknown fields.

Phase 2: Validation and determinism

• Preserve existing schema semantics while tightening validation.
• Validate required module fields and path field types.
• Reject duplicate HAK names, autogen producer IDs, autogen consumer IDs, and dataset IDs where represented in root config.
• Validate HAK entries for name, priority, max bytes, include globs, and output-safe names.
• Validate autogen manifest filenames and deterministic producer/consumer references.
• Validate music prefixes for ambiguous duplicate normalized roots.
• Ensure include globs and map-derived outputs are sorted before use where order matters.
• Add deterministic tie-breakers for HAK priority sorting.

Phase 3: Move repository-specific defaults into YAML

• Keep only intentional toolkit defaults in defaultConfig.
• Move Shadows Over Westgate-specific defaults into repo YAML:
  • top package names
  • topdata cache/build paths
  • autogen manifest metadata
  • music prefixes
  • HAK ordering and include globs
• Add config fields where needed before removing hardcoded assumptions.
• Leave generated artifact paths as JSON outputs unless the path itself must be configurable.

Phase 4: Migration

• Convert consumer root configs from nwn-tool.json to nwn-tool.yaml.
• Preserve existing values exactly unless a separate behavior change is documented.
• Keep topdata/templates/config.json unchanged for now because it is topdata conversion metadata, not root repository configuration.
• Keep generated datasets, manifests, caches, credits inventory, and GFF JSON unchanged.
• Add before/after examples for root config migration.

Phase 5: Tests

• Add focused loader tests for:
  • nwn-tool.yaml
  • nwn-tool.yml
  • legacy nwn-tool.json
  • YAML precedence over JSON
  • malformed YAML
  • unknown YAML fields
• Update representative build, validator, pipeline, topdata, and app tests to use YAML root config.
• Keep targeted JSON tests only for legacy fallback and generated artifact behavior.
• Add validation tests for duplicates, path type failures, deterministic HAK ordering, and ambiguous music prefixes.
• Run go test ./....

Phase 6: Documentation

• Update README.md repository layout and consumer instructions to reference nwn-tool.yaml.
• Document legacy JSON fallback and planned removal policy.
• Document which JSON files remain intentional generated or machine-readable artifacts.
• Document config precedence and command log behavior.

Acceptance criteria

• A repository can build from the chosen YAML root config file (nwn-tool.yaml under the current plan, or config.yaml if the naming decision changes).
• Existing generated JSON datasets and manifests still work.
• No repository-specific behavior remains hardcoded if it can be expressed in YAML.
• Builds remain deterministic.
• Existing behavior is preserved after migration.
• Logs clearly state which config file was loaded.