Refactor toolkit configuration from JSON to YAML only. # Context Repositories currently use `config.json` for human-authored toolkit configuration. However, JSON is also used heavily for generated datasets, manifests, caches, and machine-readable artifacts. This makes repository configuration harder to distinguish from generated data. # Goal Make YAML the source of truth for all human-authored repository configuration. JSON should remain valid for generated data artifacts, but not as the preferred format for repository configuration. # Core requirements - Replace `config.json` with `config.yaml`. - Load repository configuration from YAML. - Treat YAML as the canonical source-of-truth. - Keep generated datasets, manifests, caches, and machine-readable outputs as JSON. - Do not move dataset artifacts to YAML. - Do not hardcode configuration behavior in code if it can reasonably be expressed in config. - The toolkit should be deterministic. # Configuration philosophy Configuration should control repository behavior. Avoid hardcoded assumptions for: - module names - resrefs - paths - HAK names - HAK priority - HAK split behavior - HAK size limits - include/exclude globs - optional HAKs - music prefixes - extract rules - topdata paths/packages - autogen producers - autogen consumers - manifest names - cache names - release tags - dataset names - derivation rules If behavior varies by repository, it belongs in YAML. # Example target config Convert this kind of repository config: ```json { "module": { "name": "Shadows Over Westgate", "resref": "sow_module", "description": "Shadows Over Westgate asset pipeline.", "hak_order": ["sow_top", "group:sow_over", "group:sow_core"] }, "paths": { "assets": "content", "build": "build" }, "music": { "prefixes": { "envi/music/westgate": "mus_wg_" } } } ``` Into: ```yaml module: name: Shadows Over Westgate resref: sow_module description: Shadows Over Westgate asset pipeline. hak_order: - sow_top - group:sow_over - group:sow_core paths: assets: content build: build music: prefixes: envi/music/westgate: mus_wg_ ``` # Implementation requirements - Add YAML config loading. - Prefer `config.yaml`. - Optionally accept `config.yml`. - Remove reliance on `config.json` for repository configuration. - If `config.json` support is kept temporarily, mark it as legacy/deprecated. - If both YAML and JSON configs exist, YAML must win. - Emit a clear warning when legacy JSON config is used. - Preserve existing schema semantics. - Preserve existing defaults only where they are intentional toolkit defaults. - Move repository-specific defaults into YAML. - Keep validation strict. - Fail clearly on malformed YAML or unknown required fields. - Preserve deterministic ordering where configs are serialized, logged, or normalized. # Validation requirements - Validate YAML against the same schema expectations currently used for JSON. - Ensure required sections are present when needed. - Ensure paths are strings. - Ensure HAK entries have required fields. - Ensure autogen producers/consumers have valid ids, roots, includes, derive rules, and manifest config. - Ensure music prefixes are deterministic and unambiguous. - Ensure duplicate HAK names, producer ids, consumer ids, or dataset ids are rejected. - Ensure priority ordering is deterministic even when priorities match. - Ensure include glob ordering is stable. # Migration requirements - Convert existing `config.json` files to `config.yaml`. - Preserve all existing values exactly unless a change is intentional. - Update code paths, scripts, tests, and documentation to reference YAML. - Add or update tests for: - loading `config.yaml` - loading `config.yml` - legacy `config.json` fallback, if retained - YAML precedence over JSON - validation failures - deterministic ordering - representative configs matching the current repository examples - Do not change generated artifact formats unless explicitly required. # Generated artifacts Keep these as JSON unless there is a separate reason to change them: - datasets - manifests - caches - credits output - build summaries - machine-readable reports Rationale: - YAML is for human-authored repository configuration. - JSON is for generated or machine-readable data. # Deliverables 1. YAML config loader design 2. Updated config schema 3. JSON-to-YAML migration plan 4. Converted example configs 5. Legacy JSON handling policy 6. Validation updates 7. Test plan 8. Documentation updates 9. List of code-level assumptions that should be moved into config 10. Before/after examples # Initial project-needs analysis ## Current state - Main repository configuration is currently modeled by `internal/project.Config`. - The toolkit currently discovers and loads `nwn-tool.json`, not `config.json`. - Config loading is centralized in `internal/project.FindRoot` and `internal/project.Load`. - Config structs currently have JSON tags only. - There is no YAML dependency in `go.mod`. - Generated and machine-readable JSON is widespread and should remain JSON: - build HAK manifest: `build/haks.json` - topdata authored/generated datasets - topdata template conversion metadata: `topdata/templates/config.json` - topdata wiki state and deploy manifests - autogen manifest caches under `.cache` - credits inventory output - GFF canonical JSON documents ## Naming decision needed The contract says to replace `config.json` with `config.yaml`, but the current toolkit root config file is `nwn-tool.json`. Plan assumption: - Treat `nwn-tool.yaml` as the direct YAML replacement for the existing root config. - Optionally accept `nwn-tool.yml`. - Keep temporary legacy fallback to `nwn-tool.json` with a warning. - Do not rename generated or nested metadata files unless they are confirmed to be human-authored repository root configuration. If the intended final filename is instead literally `config.yaml`, update the loader plan before implementation. ## Code areas affected - `internal/project/project.go` - config file discovery - YAML/legacy JSON loading - schema tags and strict decode behavior - validation for duplicate names, path fields, include ordering, and required config sections - currently hardcoded defaults such as `build`, `.cache`, `sow_top.hak`, `sow_tlk.tlk`, and `haks.json` - `internal/app/app.go` - project-load diagnostics and logging so commands clearly state which config file was loaded - user-facing descriptions that mention hardcoded paths or outputs - `internal/pipeline/*` - HAK ordering, manifest naming, music cache/credits paths, and music prefix behavior - generated JSON outputs must remain JSON - `internal/topdata/*` - package names, cache names, released manifest metadata, wiki state paths, and autogen consumer cache paths - topdata datasets and manifests must remain JSON - `internal/validator/*` - validation tests and duplicate/ordering rules - potential config control for built-in script prefixes if repository-specific - Tests under `internal/**/*_test.go` - many tests currently write `nwn-tool.json`; representative tests should move to YAML with targeted legacy JSON coverage retained - `README.md` and workflow docs - update root configuration references to YAML - explicitly document JSON artifacts that remain JSON ## Hardcoded assumptions to audit for config ownership Move to YAML if repository-specific: - root config filename policy and migration behavior - build output directory and HAK manifest filename - topdata build/cache paths - topdata package HAK/TLK filenames and derived TLK behavior - autogen release tags, asset names, cache names, producers, consumers, roots, includes, modes, and derive rules - released parts manifest repository/server metadata when not a universal toolkit default - HAK names, priorities, split behavior, optional flags, include globs, and module HAK order - music source roots, prefixes, credit overlay names, generated credits paths, and music stem constraints where repository-specific - source/asset include extension sets where repository-specific - validator script-prefix exclusions if they vary by repository Keep in code if toolkit-intrinsic: - generated JSON encoding for datasets, manifests, caches, GFF documents, and reports - NWN resource format constraints such as resref length limits - deterministic sort rules after config values are loaded - supported autogen derive kinds and consumer modes unless extensibility is implemented # Implementation plan ## Phase 1: Loader and schema foundation - Add YAML parsing dependency, likely `gopkg.in/yaml.v3`. - Add YAML tags to all config structs while preserving JSON tags for legacy decode. - Introduce config discovery order: 1. `nwn-tool.yaml` 2. `nwn-tool.yml` 3. legacy `nwn-tool.json` - Return loaded config metadata from `project.Load`, including path, format, and legacy flag. - Make YAML win when both YAML and JSON are present. - Emit a clear legacy warning when `nwn-tool.json` is used. - Emit/log the selected config path for normal command execution. - Fail clearly for malformed YAML and unknown fields. ## Phase 2: Validation and determinism - Preserve existing schema semantics while tightening validation. - Validate required module fields and path field types. - Reject duplicate HAK names, autogen producer IDs, autogen consumer IDs, and dataset IDs where represented in root config. - Validate HAK entries for name, priority, max bytes, include globs, and output-safe names. - Validate autogen manifest filenames and deterministic producer/consumer references. - Validate music prefixes for ambiguous duplicate normalized roots. - Ensure include globs and map-derived outputs are sorted before use where order matters. - Add deterministic tie-breakers for HAK priority sorting. ## Phase 3: Move repository-specific defaults into YAML - Keep only intentional toolkit defaults in `defaultConfig`. - Move Shadows Over Westgate-specific defaults into repo YAML: - top package names - topdata cache/build paths - autogen manifest metadata - music prefixes - HAK ordering and include globs - Add config fields where needed before removing hardcoded assumptions. - Leave generated artifact paths as JSON outputs unless the path itself must be configurable. ## Phase 4: Migration - Convert consumer root configs from `nwn-tool.json` to `nwn-tool.yaml`. - Preserve existing values exactly unless a separate behavior change is documented. - Keep `topdata/templates/config.json` unchanged for now because it is topdata conversion metadata, not root repository configuration. - Keep generated datasets, manifests, caches, credits inventory, and GFF JSON unchanged. - Add before/after examples for root config migration. ## Phase 5: Tests - Add focused loader tests for: - `nwn-tool.yaml` - `nwn-tool.yml` - legacy `nwn-tool.json` - YAML precedence over JSON - malformed YAML - unknown YAML fields - Update representative build, validator, pipeline, topdata, and app tests to use YAML root config. - Keep targeted JSON tests only for legacy fallback and generated artifact behavior. - Add validation tests for duplicates, path type failures, deterministic HAK ordering, and ambiguous music prefixes. - Run `go test ./...`. ## Phase 6: Documentation - Update `README.md` repository layout and consumer instructions to reference `nwn-tool.yaml`. - Document legacy JSON fallback and planned removal policy. - Document which JSON files remain intentional generated or machine-readable artifacts. - Document config precedence and command log behavior. # Acceptance criteria - A repository can build from the chosen YAML root config file (`nwn-tool.yaml` under the current plan, or `config.yaml` if the naming decision changes). - Existing generated JSON datasets and manifests still work. - No repository-specific behavior remains hardcoded if it can be expressed in YAML. - Builds remain deterministic. - Existing behavior is preserved after migration. - Logs clearly state which config file was loaded.