Expand topdata handoff notes

2026-04-02 21:42:57 +02:00
parent d67b6e3f12
commit 6777e2baef
1 changed files with 91 additions and 4 deletions
@@ -11,6 +11,11 @@ The migration direction is:
 - `sow-module` is the new long-term source home for migrated topdata content
 - wiki generation, wiki deploy, and tophak packaging are still deferred

+Relevant commits from this milestone:
+
+- `sow-tools`: `d67b6e3` `Add topdata bridge commands and handoff`
+- `sow-module`: `dfb7fd2` `Add topdata migration roots and wrappers`
+
 ## Implemented In This Milestone

 ### New `sow-toolkit` commands
@@ -27,6 +32,13 @@ Current behavior:

 This is an intentional bridge: it preserves current production behavior while the deeper merge/build rules are audited and ported.

+Important implementation detail:
+
+- the Python builder is run from a copied temporary workspace, not in place from `sow-dev`
+- this prevents the reference bridge from mutating repo-tracked state in `sow-dev`
+- the copied bridge rewrites its default asset/staging roots to temp-local paths before running
+- this was necessary because the Python builder also compiles `sow_top.hak` as a side effect of its normal build flow
+
 ### New `sow-module` layout

 Recommended migration root now exists:
@@ -46,6 +58,12 @@ build/topdata/
  tlk/
 ```

+Current `sow-module` config now points at:
+
+- `topdata.source = "topdata"`
+- `topdata.build = "build/topdata"`
+- `topdata.reference_builder = "../sow-dev/tools/2dabuilder"`
+
 ### Canonical compatibility validation

 The new validator already recognizes the current family of authoring shapes:
@@ -63,6 +81,27 @@ It is intentionally permissive for now:
 - warnings are used where a file is parseable but not canonical yet
 - this keeps migration practical while still nudging toward a formal schema

+It does **not** yet rewrite or normalize files on disk. This milestone is validation and bridge-first, not migration automation.
+
+## What Was Verified
+
+The new bridge was verified on the real project, not only on unit tests.
+
+Verified successfully:
+
+- `go test ./...` in `sow-tools`
+- `sow-toolkit validate-topdata`
+- `sow-toolkit build-topdata`
+- `sow-toolkit compare-topdata`
+
+Observed real output:
+
+- `build/topdata/2da` produced `113` generated `.2da` files
+- `build/topdata/tlk` produced `1` TLK output
+- `compare-topdata` passed against a fresh Python reference build
+
+This means the first milestone is not just scaffolding; it already gives `sow-module` a working topdata command surface backed by the authoritative Python builder.
+
 ## Important Rules Captured So Far

 These behaviors are considered core and must be preserved in the eventual native compiler:
@@ -75,12 +114,30 @@ These behaviors are considered core and must be preserved in the eventual native
 - ID allocation when a key has no existing lock entry
 - “only override what we intentionally touch”
 - deterministic output and deterministic compare behavior
+- grouped dataset families such as `classes`
+- plain-table handling alongside base+module datasets
+- compatibility with both snippet files and grouped files
+
+## Current Rough Edges
+
+These are known limitations of the bridge and should be expected when resuming:
+
+- `build-topdata` and `compare-topdata` are slower than the future native implementation
+  - they currently create a temporary copied Python builder workspace
+  - they install the Python requirements into a temporary venv each run
+- the bridge still mirrors the Python builder’s semantics, including the tophak side-effect path, but now safely sandboxed in temp space
+- canonical schema validation is only partial so far
+  - enough to validate root shape families and obvious errors
+  - not enough yet to encode all dataset-specific rules
+- migrated data has not actually been moved into `sow-module/topdata` yet beyond the new root placeholders

 ## What Is Still Deferred

 - native Go merge/build parity for current Python rules
 - shorthand expansion porting
 - dataset-specific rule porting
+- parity fixtures checked into `sow-tools`
+- schema documentation for the canonical topdata JSON contract
 - local wiki generation
 - wiki deploy
 - tophak packaging
@@ -93,10 +150,40 @@ These behaviors are considered core and must be preserved in the eventual native
   - `feat`
   - `itemprops`
   - TLK merge behavior
-2. Capture representative parity fixtures from the Python builder.
-3. Replace the reference-backed `build-topdata` bridge with native Go build logic one dataset family at a time.
-4. Add a normalize/migrate command once the canonical JSON contract is nailed down.
-5. Only migrate wiki generation after compiler parity is stable.
+2. Write the canonical topdata JSON contract down explicitly.
+   - root object families
+   - row/entry/override forms
+   - keyed refs
+   - TLK refs
+   - lockfile expectations
+3. Capture representative parity fixtures from the Python builder.
+4. Replace the reference-backed `build-topdata` bridge with native Go build logic one dataset family at a time.
+5. Add a normalize/migrate command once the canonical JSON contract is nailed down.
+6. Only migrate wiki generation after compiler parity is stable.
+
+## Resume Commands
+
+From `sow-module`:
+
+```bash
+../sow-tools/tools/sow-toolkit validate-topdata
+../sow-tools/tools/sow-toolkit build-topdata
+../sow-tools/tools/sow-toolkit compare-topdata
+```
+
+Or via the repo wrappers:
+
+```bash
+./validate-topdata.sh
+./build-topdata.sh
+./compare-topdata.sh
+```
+
+Expected success output today:
+
+- `validate-topdata` prints `topdata validation: ok`
+- `build-topdata` prints `mode: reference` and output directories under `build/topdata`
+- `compare-topdata` prints `topdata compare: ok`

 ## Related UX State