Add topdata bridge commands and handoff

2026-04-02 21:40:59 +02:00
parent 7c8d58586a
commit d67b6e3f12
6 changed files with 1083 additions and 5 deletions
@@ -0,0 +1,110 @@
+# TopData Rebuild Handoff - 2026-04-02
+
+## Current State
+
+This handoff captures the first implementation milestone of the topdata rebuild.
+
+The migration direction is:
+
+- Python `2dabuilder` in `sow-dev` remains authoritative
+- `sow-toolkit` now owns the new topdata command surface
+- `sow-module` is the new long-term source home for migrated topdata content
+- wiki generation, wiki deploy, and tophak packaging are still deferred
+
+## Implemented In This Milestone
+
+### New `sow-toolkit` commands
+
+- `validate-topdata`
+- `build-topdata`
+- `compare-topdata`
+
+Current behavior:
+
+- `validate-topdata` validates the new `topdata/` source root in `sow-module`
+- `build-topdata` currently uses the Python builder as a reference backend and writes outputs into `build/topdata/`
+- `compare-topdata` rebuilds fresh reference output from Python and compares it against `build/topdata/`
+
+This is an intentional bridge: it preserves current production behavior while the deeper merge/build rules are audited and ported.
+
+### New `sow-module` layout
+
+Recommended migration root now exists:
+
+```text
+topdata/
+  data/
+  tlk/
+  base_dialog.json
+```
+
+Generated outputs are expected under:
+
+```text
+build/topdata/
+  2da/
+  tlk/
+```
+
+### Canonical compatibility validation
+
+The new validator already recognizes the current family of authoring shapes:
+
+- dataset `base.json` with `columns` + `rows`
+- lockfiles as object maps of key -> numeric id
+- module files using `entries`
+- module files using `overrides`
+- plain row files using `rows`
+- TLK files with `entries`
+
+It is intentionally permissive for now:
+
+- legacy parseable shapes can still load
+- warnings are used where a file is parseable but not canonical yet
+- this keeps migration practical while still nudging toward a formal schema
+
+## Important Rules Captured So Far
+
+These behaviors are considered core and must be preserved in the eventual native compiler:
+
+- base fallback from `base.json`
+- TLK fallback from `base_dialog.json` / TLK base data
+- snippet files and grouped files both being valid
+- grouped dataset families and plain tables both being valid
+- lockfile-backed IDs
+- ID allocation when a key has no existing lock entry
+- “only override what we intentionally touch”
+- deterministic output and deterministic compare behavior
+
+## What Is Still Deferred
+
+- native Go merge/build parity for current Python rules
+- shorthand expansion porting
+- dataset-specific rule porting
+- local wiki generation
+- wiki deploy
+- tophak packaging
+- canonical normalize/export command
+
+## Suggested Next Steps
+
+1. Audit the current Python rules dataset-by-dataset, especially:
+   - `classes`
+   - `feat`
+   - `itemprops`
+   - TLK merge behavior
+2. Capture representative parity fixtures from the Python builder.
+3. Replace the reference-backed `build-topdata` bridge with native Go build logic one dataset family at a time.
+4. Add a normalize/migrate command once the canonical JSON contract is nailed down.
+5. Only migrate wiki generation after compiler parity is stable.
+
+## Related UX State
+
+Separate from topdata, the content repos already have:
+
+- auto-installing `sow-toolkit` wrappers
+- Linux/macOS shell wrappers
+- Windows PowerShell and `.cmd` wrappers
+- simplified `START-HERE.md` docs in `sow-module` and `sow-assets`
+
+Windows still needs a real smoke test from an actual Windows machine.