YAML Configuration Refactor

2026-05-07 13:44:46 +02:00
parent 7abcbf1a38
commit 000d31ad24
8 changed files with 860 additions and 87 deletions
@@ -0,0 +1,325 @@
+Refactor toolkit configuration from JSON to YAML only.
+
+# Context
+
+Repositories currently use `config.json` for human-authored toolkit configuration.
+
+However, JSON is also used heavily for generated datasets, manifests, caches, and machine-readable artifacts. This makes repository configuration harder to distinguish from generated data.
+
+# Goal
+
+Make YAML the source of truth for all human-authored repository configuration.
+
+JSON should remain valid for generated data artifacts, but not as the preferred format for repository configuration.
+
+# Core requirements
+
+- Replace `config.json` with `config.yaml`.
+- Load repository configuration from YAML.
+- Treat YAML as the canonical source-of-truth.
+- Keep generated datasets, manifests, caches, and machine-readable outputs as JSON.
+- Do not move dataset artifacts to YAML.
+- Do not hardcode configuration behavior in code if it can reasonably be expressed in config.
+- The toolkit should be deterministic.
+
+# Configuration philosophy
+
+Configuration should control repository behavior.
+
+Avoid hardcoded assumptions for:
+
+- module names
+- resrefs
+- paths
+- HAK names
+- HAK priority
+- HAK split behavior
+- HAK size limits
+- include/exclude globs
+- optional HAKs
+- music prefixes
+- extract rules
+- topdata paths/packages
+- autogen producers
+- autogen consumers
+- manifest names
+- cache names
+- release tags
+- dataset names
+- derivation rules
+
+If behavior varies by repository, it belongs in YAML.
+
+# Example target config
+
+Convert this kind of repository config:
+
+```json
+{
+  "module": {
+    "name": "Shadows Over Westgate",
+    "resref": "sow_module",
+    "description": "Shadows Over Westgate asset pipeline.",
+    "hak_order": ["sow_top", "group:sow_over", "group:sow_core"]
+  },
+  "paths": {
+    "assets": "content",
+    "build": "build"
+  },
+  "music": {
+    "prefixes": {
+      "envi/music/westgate": "mus_wg_"
+    }
+  }
+}
+```
+
+Into:
+
+```yaml
+module:
+  name: Shadows Over Westgate
+  resref: sow_module
+  description: Shadows Over Westgate asset pipeline.
+  hak_order:
+    - sow_top
+    - group:sow_over
+    - group:sow_core
+
+paths:
+  assets: content
+  build: build
+
+music:
+  prefixes:
+    envi/music/westgate: mus_wg_
+```
+
+# Implementation requirements
+
+- Add YAML config loading.
+- Prefer `config.yaml`.
+- Optionally accept `config.yml`.
+- Remove reliance on `config.json` for repository configuration.
+- If `config.json` support is kept temporarily, mark it as legacy/deprecated.
+- If both YAML and JSON configs exist, YAML must win.
+- Emit a clear warning when legacy JSON config is used.
+- Preserve existing schema semantics.
+- Preserve existing defaults only where they are intentional toolkit defaults.
+- Move repository-specific defaults into YAML.
+- Keep validation strict.
+- Fail clearly on malformed YAML or unknown required fields.
+- Preserve deterministic ordering where configs are serialized, logged, or normalized.
+
+# Validation requirements
+
+- Validate YAML against the same schema expectations currently used for JSON.
+- Ensure required sections are present when needed.
+- Ensure paths are strings.
+- Ensure HAK entries have required fields.
+- Ensure autogen producers/consumers have valid ids, roots, includes, derive rules, and manifest config.
+- Ensure music prefixes are deterministic and unambiguous.
+- Ensure duplicate HAK names, producer ids, consumer ids, or dataset ids are rejected.
+- Ensure priority ordering is deterministic even when priorities match.
+- Ensure include glob ordering is stable.
+
+# Migration requirements
+
+- Convert existing `config.json` files to `config.yaml`.
+- Preserve all existing values exactly unless a change is intentional.
+- Update code paths, scripts, tests, and documentation to reference YAML.
+- Add or update tests for:
+  - loading `config.yaml`
+  - loading `config.yml`
+  - legacy `config.json` fallback, if retained
+  - YAML precedence over JSON
+  - validation failures
+  - deterministic ordering
+  - representative configs matching the current repository examples
+
+- Do not change generated artifact formats unless explicitly required.
+
+# Generated artifacts
+
+Keep these as JSON unless there is a separate reason to change them:
+
+- datasets
+- manifests
+- caches
+- credits output
+- build summaries
+- machine-readable reports
+
+Rationale:
+
+- YAML is for human-authored repository configuration.
+- JSON is for generated or machine-readable data.
+
+# Deliverables
+
+1. YAML config loader design
+2. Updated config schema
+3. JSON-to-YAML migration plan
+4. Converted example configs
+5. Legacy JSON handling policy
+6. Validation updates
+7. Test plan
+8. Documentation updates
+9. List of code-level assumptions that should be moved into config
+10. Before/after examples
+
+# Initial project-needs analysis
+
+## Current state
+
+- Main repository configuration is currently modeled by `internal/project.Config`.
+- The toolkit currently discovers and loads `nwn-tool.json`, not `config.json`.
+- Config loading is centralized in `internal/project.FindRoot` and `internal/project.Load`.
+- Config structs currently have JSON tags only.
+- There is no YAML dependency in `go.mod`.
+- Generated and machine-readable JSON is widespread and should remain JSON:
+  - build HAK manifest: `build/haks.json`
+  - topdata authored/generated datasets
+  - topdata template conversion metadata: `topdata/templates/config.json`
+  - topdata wiki state and deploy manifests
+  - autogen manifest caches under `.cache`
+  - credits inventory output
+  - GFF canonical JSON documents
+
+## Naming decision needed
+
+The contract says to replace `config.json` with `config.yaml`, but the current toolkit root config file is `nwn-tool.json`.
+
+Plan assumption:
+
+- Treat `nwn-tool.yaml` as the direct YAML replacement for the existing root config.
+- Optionally accept `nwn-tool.yml`.
+- Keep temporary legacy fallback to `nwn-tool.json` with a warning.
+- Do not rename generated or nested metadata files unless they are confirmed to be human-authored repository root configuration.
+
+If the intended final filename is instead literally `config.yaml`, update the loader plan before implementation.
+
+## Code areas affected
+
+- `internal/project/project.go`
+  - config file discovery
+  - YAML/legacy JSON loading
+  - schema tags and strict decode behavior
+  - validation for duplicate names, path fields, include ordering, and required config sections
+  - currently hardcoded defaults such as `build`, `.cache`, `sow_top.hak`, `sow_tlk.tlk`, and `haks.json`
+- `internal/app/app.go`
+  - project-load diagnostics and logging so commands clearly state which config file was loaded
+  - user-facing descriptions that mention hardcoded paths or outputs
+- `internal/pipeline/*`
+  - HAK ordering, manifest naming, music cache/credits paths, and music prefix behavior
+  - generated JSON outputs must remain JSON
+- `internal/topdata/*`
+  - package names, cache names, released manifest metadata, wiki state paths, and autogen consumer cache paths
+  - topdata datasets and manifests must remain JSON
+- `internal/validator/*`
+  - validation tests and duplicate/ordering rules
+  - potential config control for built-in script prefixes if repository-specific
+- Tests under `internal/**/*_test.go`
+  - many tests currently write `nwn-tool.json`; representative tests should move to YAML with targeted legacy JSON coverage retained
+- `README.md` and workflow docs
+  - update root configuration references to YAML
+  - explicitly document JSON artifacts that remain JSON
+
+## Hardcoded assumptions to audit for config ownership
+
+Move to YAML if repository-specific:
+
+- root config filename policy and migration behavior
+- build output directory and HAK manifest filename
+- topdata build/cache paths
+- topdata package HAK/TLK filenames and derived TLK behavior
+- autogen release tags, asset names, cache names, producers, consumers, roots, includes, modes, and derive rules
+- released parts manifest repository/server metadata when not a universal toolkit default
+- HAK names, priorities, split behavior, optional flags, include globs, and module HAK order
+- music source roots, prefixes, credit overlay names, generated credits paths, and music stem constraints where repository-specific
+- source/asset include extension sets where repository-specific
+- validator script-prefix exclusions if they vary by repository
+
+Keep in code if toolkit-intrinsic:
+
+- generated JSON encoding for datasets, manifests, caches, GFF documents, and reports
+- NWN resource format constraints such as resref length limits
+- deterministic sort rules after config values are loaded
+- supported autogen derive kinds and consumer modes unless extensibility is implemented
+
+# Implementation plan
+
+## Phase 1: Loader and schema foundation
+
+- Add YAML parsing dependency, likely `gopkg.in/yaml.v3`.
+- Add YAML tags to all config structs while preserving JSON tags for legacy decode.
+- Introduce config discovery order:
+  1. `nwn-tool.yaml`
+  2. `nwn-tool.yml`
+  3. legacy `nwn-tool.json`
+- Return loaded config metadata from `project.Load`, including path, format, and legacy flag.
+- Make YAML win when both YAML and JSON are present.
+- Emit a clear legacy warning when `nwn-tool.json` is used.
+- Emit/log the selected config path for normal command execution.
+- Fail clearly for malformed YAML and unknown fields.
+
+## Phase 2: Validation and determinism
+
+- Preserve existing schema semantics while tightening validation.
+- Validate required module fields and path field types.
+- Reject duplicate HAK names, autogen producer IDs, autogen consumer IDs, and dataset IDs where represented in root config.
+- Validate HAK entries for name, priority, max bytes, include globs, and output-safe names.
+- Validate autogen manifest filenames and deterministic producer/consumer references.
+- Validate music prefixes for ambiguous duplicate normalized roots.
+- Ensure include globs and map-derived outputs are sorted before use where order matters.
+- Add deterministic tie-breakers for HAK priority sorting.
+
+## Phase 3: Move repository-specific defaults into YAML
+
+- Keep only intentional toolkit defaults in `defaultConfig`.
+- Move Shadows Over Westgate-specific defaults into repo YAML:
+  - top package names
+  - topdata cache/build paths
+  - autogen manifest metadata
+  - music prefixes
+  - HAK ordering and include globs
+- Add config fields where needed before removing hardcoded assumptions.
+- Leave generated artifact paths as JSON outputs unless the path itself must be configurable.
+
+## Phase 4: Migration
+
+- Convert consumer root configs from `nwn-tool.json` to `nwn-tool.yaml`.
+- Preserve existing values exactly unless a separate behavior change is documented.
+- Keep `topdata/templates/config.json` unchanged for now because it is topdata conversion metadata, not root repository configuration.
+- Keep generated datasets, manifests, caches, credits inventory, and GFF JSON unchanged.
+- Add before/after examples for root config migration.
+
+## Phase 5: Tests
+
+- Add focused loader tests for:
+  - `nwn-tool.yaml`
+  - `nwn-tool.yml`
+  - legacy `nwn-tool.json`
+  - YAML precedence over JSON
+  - malformed YAML
+  - unknown YAML fields
+- Update representative build, validator, pipeline, topdata, and app tests to use YAML root config.
+- Keep targeted JSON tests only for legacy fallback and generated artifact behavior.
+- Add validation tests for duplicates, path type failures, deterministic HAK ordering, and ambiguous music prefixes.
+- Run `go test ./...`.
+
+## Phase 6: Documentation
+
+- Update `README.md` repository layout and consumer instructions to reference `nwn-tool.yaml`.
+- Document legacy JSON fallback and planned removal policy.
+- Document which JSON files remain intentional generated or machine-readable artifacts.
+- Document config precedence and command log behavior.
+
+# Acceptance criteria
+
+- A repository can build from the chosen YAML root config file (`nwn-tool.yaml` under the current plan, or `config.yaml` if the naming decision changes).
+- Existing generated JSON datasets and manifests still work.
+- No repository-specific behavior remains hardcoded if it can be expressed in YAML.
+- Builds remain deterministic.
+- Existing behavior is preserved after migration.
+- Logs clearly state which config file was loaded.