sow-tools/CONFIGURATION_REFACTOR_CONTRACT.md

Refactor toolkit configuration from JSON to YAML only.

## Status Snapshot

Current state as of 2026-05-13:

- This refactor is largely implemented.
- Root config discovery now prefers `nwn-tool.yaml`, then `nwn-tool.yml`, then
  legacy `nwn-tool.json`.
- YAML is canonical in the consumer repos, and the toolkit exposes
  `config validate|effective|inspect|explain|sources`.
- Legacy JSON compatibility still exists by design and is still used heavily in
  test fixtures.

Remaining work:

- continue migrating representative tests from `nwn-tool.json` fixtures to YAML
  while keeping only targeted legacy coverage
- decide when to deprecate or remove legacy shorthand such as `music.prefixes`
  after consumer configs are fully dataset-based
- keep documentation/examples synchronized with the normalized YAML schema as it
  expands

## Active Scope

This contract remains active and is one of the current priorities.

The implementation gap is no longer "load YAML at all". The real gap is
configuration authority drift:

- some repository behavior is still encoded as toolkit defaults when it should
  become explicit schema
- some tests still model old JSON-era expectations
- some topdata behavior is still driven by explicit namespace/path checks rather
  than generic dataset rules

## Current Findings

The most important remaining configuration work is topdata-related:

- topdata dataset discovery already derives default output names when `output` is
  absent, but validation still hard-fails for
  `topdata/data/masterfeats/base.json` when `output` is missing
- `internal/topdata/topdata.go` dispatches validation through explicit path
  checks for `masterfeats`, `feat`, parts overrides, and racialtypes registry
- topdata defaults in `internal/project` still mix generic toolkit defaults with
  behavior that may need clearer schema ownership

This means the config-refactor track now overlaps directly with topdata
hardening. The core question is no longer file format migration; it is whether a
behavior belongs in:

- root YAML
- dataset-authored JSON
- or a documented intrinsic toolkit invariant

## Deferred Scope

Defer unrelated schema expansion for now:

- model compilation config
- broader asset-maintenance policy schema
- release-announcement config

## Immediate Plan

1. Continue shrinking legacy `nwn-tool.json` coverage down to deliberate
   compatibility tests.
2. Audit hardcoded topdata behaviors and classify each as:
   - root YAML concern
   - dataset JSON concern
   - toolkit invariant
3. Remove outdated namespace-specific validation assumptions where generic
   dataset logic already exists.
4. Keep generated artifacts in JSON, but stop using explicit path-based logic
   when the same rule can be inferred from generic dataset shape.
5. Document every remaining intentional toolkit default in `config effective`
   terms.

# Context

Repositories currently use `config.json` for human-authored toolkit configuration.

However, JSON is also used heavily for generated datasets, manifests, caches, and machine-readable artifacts. This makes repository configuration harder to distinguish from generated data.

# Goal

Make YAML the source of truth for all human-authored repository configuration.

JSON should remain valid for generated data artifacts, but not as the preferred format for repository configuration.

# Core requirements

- Replace `config.json` with `config.yaml`.
- Load repository configuration from YAML.
- Treat YAML as the canonical source-of-truth.
- Keep generated datasets, manifests, caches, and machine-readable outputs as JSON.
- Do not move dataset artifacts to YAML.
- Do not hardcode configuration behavior in code if it can reasonably be expressed in config.
- The toolkit should be deterministic.

# Configuration philosophy

Configuration should control repository behavior.

Avoid hardcoded assumptions for:

- module names
- resrefs
- paths
- HAK names
- HAK priority
- HAK split behavior
- HAK size limits
- include/exclude globs
- optional HAKs
- music prefixes
- extract rules
- topdata paths/packages
- autogen producers
- autogen consumers
- manifest names
- cache names
- release tags
- dataset names
- derivation rules

If behavior varies by repository, it belongs in YAML.

# Example target config

Convert this kind of repository config:

```json
{
  "module": {
    "name": "Shadows Over Westgate",
    "resref": "sow_module",
    "description": "Shadows Over Westgate asset pipeline.",
    "hak_order": ["sow_top", "group:sow_over", "group:sow_core"]
  },
  "paths": {
    "assets": "content",
    "build": "build"
  },
  "music": {
    "prefixes": {
      "envi/music/westgate": "mus_wg_"
    }
  }
}
```

Into:

```yaml
module:
  name: Shadows Over Westgate
  resref: sow_module
  description: Shadows Over Westgate asset pipeline.
  hak_order:
    - sow_top
    - group:sow_over
    - group:sow_core

paths:
  assets: content
  build: build

music:
  prefixes:
    envi/music/westgate: mus_wg_
```

# Implementation requirements

- Add YAML config loading.
- Prefer `config.yaml`.
- Optionally accept `config.yml`.
- Remove reliance on `config.json` for repository configuration.
- If `config.json` support is kept temporarily, mark it as legacy/deprecated.
- If both YAML and JSON configs exist, YAML must win.
- Emit a clear warning when legacy JSON config is used.
- Preserve existing schema semantics.
- Preserve existing defaults only where they are intentional toolkit defaults.
- Move repository-specific defaults into YAML.
- Keep validation strict.
- Fail clearly on malformed YAML or unknown required fields.
- Preserve deterministic ordering where configs are serialized, logged, or normalized.

# Validation requirements

- Validate YAML against the same schema expectations currently used for JSON.
- Ensure required sections are present when needed.
- Ensure paths are strings.
- Ensure HAK entries have required fields.
- Ensure autogen producers/consumers have valid ids, roots, includes, derive rules, and manifest config.
- Ensure music prefixes are deterministic and unambiguous.
- Ensure duplicate HAK names, producer ids, consumer ids, or dataset ids are rejected.
- Ensure priority ordering is deterministic even when priorities match.
- Ensure include glob ordering is stable.

# Migration requirements

- Convert existing `config.json` files to `config.yaml`.
- Preserve all existing values exactly unless a change is intentional.
- Update code paths, scripts, tests, and documentation to reference YAML.
- Add or update tests for:
  - loading `config.yaml`
  - loading `config.yml`
  - legacy `config.json` fallback, if retained
  - YAML precedence over JSON
  - validation failures
  - deterministic ordering
  - representative configs matching the current repository examples

- Do not change generated artifact formats unless explicitly required.

# Generated artifacts

Keep these as JSON unless there is a separate reason to change them:

- datasets
- manifests
- caches
- credits output
- build summaries
- machine-readable reports

Rationale:

- YAML is for human-authored repository configuration.
- JSON is for generated or machine-readable data.

# Deliverables

1. YAML config loader design
2. Updated config schema
3. JSON-to-YAML migration plan
4. Converted example configs
5. Legacy JSON handling policy
6. Validation updates
7. Test plan
8. Documentation updates
9. List of code-level assumptions that should be moved into config
10. Before/after examples

# Initial project-needs analysis

## Current state

- Main repository configuration is currently modeled by `internal/project.Config`.
- The toolkit currently discovers and loads `nwn-tool.json`, not `config.json`.
- Config loading is centralized in `internal/project.FindRoot` and `internal/project.Load`.
- Config structs currently have JSON tags only.
- There is no YAML dependency in `go.mod`.
- Generated and machine-readable JSON is widespread and should remain JSON:
  - build HAK manifest: `build/haks.json`
  - topdata authored/generated datasets
  - topdata template conversion metadata: `topdata/templates/config.yaml`
  - topdata wiki state and deploy manifests
  - autogen manifest caches under `.cache`
  - credits inventory output
  - GFF canonical JSON documents

## Naming decision needed

The contract says to replace `config.json` with `config.yaml`, but the current toolkit root config file is `nwn-tool.json`.

Plan assumption:

- Treat `nwn-tool.yaml` as the direct YAML replacement for the existing root config.
- Optionally accept `nwn-tool.yml`.
- Keep temporary legacy fallback to `nwn-tool.json` with a warning.
- Do not rename generated or nested metadata files unless they are confirmed to be human-authored repository root configuration.

If the intended final filename is instead literally `config.yaml`, update the loader plan before implementation.

## Code areas affected

- `internal/project/project.go`
  - config file discovery
  - YAML/legacy JSON loading
  - schema tags and strict decode behavior
  - validation for duplicate names, path fields, include ordering, and required config sections
  - currently hardcoded defaults such as `build`, `.cache`, `sow_top.hak`, `sow_tlk.tlk`, and `haks.json`
- `internal/app/app.go`
  - project-load diagnostics and logging so commands clearly state which config file was loaded
  - user-facing descriptions that mention hardcoded paths or outputs
- `internal/pipeline/*`
  - HAK ordering, manifest naming, music cache/credits paths, and music prefix behavior
  - generated JSON outputs must remain JSON
- `internal/topdata/*`
  - package names, cache names, released manifest metadata, wiki state paths, and autogen consumer cache paths
  - topdata datasets and manifests must remain JSON
- `internal/validator/*`
  - validation tests and duplicate/ordering rules
  - potential config control for built-in script prefixes if repository-specific
- Tests under `internal/**/*_test.go`
  - many tests currently write `nwn-tool.json`; representative tests should move to YAML with targeted legacy JSON coverage retained
- `README.md` and workflow docs
  - update root configuration references to YAML
  - explicitly document JSON artifacts that remain JSON

## Hardcoded assumptions to audit for config ownership

Move to YAML if repository-specific:

- root config filename policy and migration behavior
- build output directory and HAK manifest filename
- topdata build/cache paths
- topdata package HAK/TLK filenames and derived TLK behavior
- autogen release tags, asset names, cache names, producers, consumers, roots, includes, modes, and derive rules
- released parts manifest repository/server metadata when not a universal toolkit default
- HAK names, priorities, split behavior, optional flags, include globs, and module HAK order
- music source roots, prefixes, credit overlay names, generated credits paths, and music stem constraints where repository-specific
- source/asset include extension sets where repository-specific
- validator script-prefix exclusions if they vary by repository

Keep in code if toolkit-intrinsic:

- generated JSON encoding for datasets, manifests, caches, GFF documents, and reports
- NWN resource format constraints such as resref length limits
- deterministic sort rules after config values are loaded
- supported autogen derive kinds and consumer modes unless extensibility is implemented

# Implementation plan

## Phase 1: Loader and schema foundation

- Add YAML parsing dependency, likely `gopkg.in/yaml.v3`.
- Add YAML tags to all config structs while preserving JSON tags for legacy decode.
- Introduce config discovery order:
  1. `nwn-tool.yaml`
  2. `nwn-tool.yml`
  3. legacy `nwn-tool.json`
- Return loaded config metadata from `project.Load`, including path, format, and legacy flag.
- Make YAML win when both YAML and JSON are present.
- Emit a clear legacy warning when `nwn-tool.json` is used.
- Emit/log the selected config path for normal command execution.
- Fail clearly for malformed YAML and unknown fields.

## Phase 2: Validation and determinism

- Preserve existing schema semantics while tightening validation.
- Validate required module fields and path field types.
- Reject duplicate HAK names, autogen producer IDs, autogen consumer IDs, and dataset IDs where represented in root config.
- Validate HAK entries for name, priority, max bytes, include globs, and output-safe names.
- Validate autogen manifest filenames and deterministic producer/consumer references.
- Validate music prefixes for ambiguous duplicate normalized roots.
- Ensure include globs and map-derived outputs are sorted before use where order matters.
- Add deterministic tie-breakers for HAK priority sorting.

## Phase 3: Move repository-specific defaults into YAML

- Keep only intentional toolkit defaults in `defaultConfig`.
- Move Shadows Over Westgate-specific defaults into repo YAML:
  - top package names
  - topdata cache/build paths
  - autogen manifest metadata
  - music prefixes
  - HAK ordering and include globs
- Add config fields where needed before removing hardcoded assumptions.
- Leave generated artifact paths as JSON outputs unless the path itself must be configurable.

## Phase 4: Migration

- Convert consumer root configs from `nwn-tool.json` to `nwn-tool.yaml`.
- Preserve existing values exactly unless a separate behavior change is documented.
- Treat `topdata/templates/config.yaml` as authored converter configuration, distinct from JSON data artifacts and root repository config.
- Keep generated datasets, manifests, caches, credits inventory, and GFF JSON unchanged.
- Add before/after examples for root config migration.

## Phase 5: Tests

- Add focused loader tests for:
  - `nwn-tool.yaml`
  - `nwn-tool.yml`
  - legacy `nwn-tool.json`
  - YAML precedence over JSON
  - malformed YAML
  - unknown YAML fields
- Update representative build, validator, pipeline, topdata, and app tests to use YAML root config.
- Keep targeted JSON tests only for legacy fallback and generated artifact behavior.
- Add validation tests for duplicates, path type failures, deterministic HAK ordering, and ambiguous music prefixes.
- Run `go test ./...`.

## Phase 6: Documentation

- Update `README.md` repository layout and consumer instructions to reference `nwn-tool.yaml`.
- Document legacy JSON fallback and planned removal policy.
- Document which JSON files remain intentional generated or machine-readable artifacts.
- Document config precedence and command log behavior.

# Acceptance criteria

- A repository can build from the chosen YAML root config file (`nwn-tool.yaml` under the current plan, or `config.yaml` if the naming decision changes).
- Existing generated JSON datasets and manifests still work.
- No repository-specific behavior remains hardcoded if it can be expressed in YAML.
- Builds remain deterministic.
- Existing behavior is preserved after migration.
- Logs clearly state which config file was loaded.