From 6777e2baef5c16ee8873660fb074acfa0c003a5a Mon Sep 17 00:00:00 2001 From: vickydotbat Date: Thu, 2 Apr 2026 21:42:57 +0200 Subject: [PATCH] Expand topdata handoff notes --- TOPDATA_HANDOFF_2026-04-02.md | 95 +++++++++++++++++++++++++++++++++-- 1 file changed, 91 insertions(+), 4 deletions(-) diff --git a/TOPDATA_HANDOFF_2026-04-02.md b/TOPDATA_HANDOFF_2026-04-02.md index af0add0..490c1d0 100644 --- a/TOPDATA_HANDOFF_2026-04-02.md +++ b/TOPDATA_HANDOFF_2026-04-02.md @@ -11,6 +11,11 @@ The migration direction is: - `sow-module` is the new long-term source home for migrated topdata content - wiki generation, wiki deploy, and tophak packaging are still deferred +Relevant commits from this milestone: + +- `sow-tools`: `d67b6e3` `Add topdata bridge commands and handoff` +- `sow-module`: `dfb7fd2` `Add topdata migration roots and wrappers` + ## Implemented In This Milestone ### New `sow-toolkit` commands @@ -27,6 +32,13 @@ Current behavior: This is an intentional bridge: it preserves current production behavior while the deeper merge/build rules are audited and ported. +Important implementation detail: + +- the Python builder is run from a copied temporary workspace, not in place from `sow-dev` +- this prevents the reference bridge from mutating repo-tracked state in `sow-dev` +- the copied bridge rewrites its default asset/staging roots to temp-local paths before running +- this was necessary because the Python builder also compiles `sow_top.hak` as a side effect of its normal build flow + ### New `sow-module` layout Recommended migration root now exists: @@ -46,6 +58,12 @@ build/topdata/ tlk/ ``` +Current `sow-module` config now points at: + +- `topdata.source = "topdata"` +- `topdata.build = "build/topdata"` +- `topdata.reference_builder = "../sow-dev/tools/2dabuilder"` + ### Canonical compatibility validation The new validator already recognizes the current family of authoring shapes: @@ -63,6 +81,27 @@ It is intentionally permissive for now: - warnings are used where a file is parseable but not canonical yet - this keeps migration practical while still nudging toward a formal schema +It does **not** yet rewrite or normalize files on disk. This milestone is validation and bridge-first, not migration automation. + +## What Was Verified + +The new bridge was verified on the real project, not only on unit tests. + +Verified successfully: + +- `go test ./...` in `sow-tools` +- `sow-toolkit validate-topdata` +- `sow-toolkit build-topdata` +- `sow-toolkit compare-topdata` + +Observed real output: + +- `build/topdata/2da` produced `113` generated `.2da` files +- `build/topdata/tlk` produced `1` TLK output +- `compare-topdata` passed against a fresh Python reference build + +This means the first milestone is not just scaffolding; it already gives `sow-module` a working topdata command surface backed by the authoritative Python builder. + ## Important Rules Captured So Far These behaviors are considered core and must be preserved in the eventual native compiler: @@ -75,12 +114,30 @@ These behaviors are considered core and must be preserved in the eventual native - ID allocation when a key has no existing lock entry - “only override what we intentionally touch” - deterministic output and deterministic compare behavior +- grouped dataset families such as `classes` +- plain-table handling alongside base+module datasets +- compatibility with both snippet files and grouped files + +## Current Rough Edges + +These are known limitations of the bridge and should be expected when resuming: + +- `build-topdata` and `compare-topdata` are slower than the future native implementation + - they currently create a temporary copied Python builder workspace + - they install the Python requirements into a temporary venv each run +- the bridge still mirrors the Python builder’s semantics, including the tophak side-effect path, but now safely sandboxed in temp space +- canonical schema validation is only partial so far + - enough to validate root shape families and obvious errors + - not enough yet to encode all dataset-specific rules +- migrated data has not actually been moved into `sow-module/topdata` yet beyond the new root placeholders ## What Is Still Deferred - native Go merge/build parity for current Python rules - shorthand expansion porting - dataset-specific rule porting +- parity fixtures checked into `sow-tools` +- schema documentation for the canonical topdata JSON contract - local wiki generation - wiki deploy - tophak packaging @@ -93,10 +150,40 @@ These behaviors are considered core and must be preserved in the eventual native - `feat` - `itemprops` - TLK merge behavior -2. Capture representative parity fixtures from the Python builder. -3. Replace the reference-backed `build-topdata` bridge with native Go build logic one dataset family at a time. -4. Add a normalize/migrate command once the canonical JSON contract is nailed down. -5. Only migrate wiki generation after compiler parity is stable. +2. Write the canonical topdata JSON contract down explicitly. + - root object families + - row/entry/override forms + - keyed refs + - TLK refs + - lockfile expectations +3. Capture representative parity fixtures from the Python builder. +4. Replace the reference-backed `build-topdata` bridge with native Go build logic one dataset family at a time. +5. Add a normalize/migrate command once the canonical JSON contract is nailed down. +6. Only migrate wiki generation after compiler parity is stable. + +## Resume Commands + +From `sow-module`: + +```bash +../sow-tools/tools/sow-toolkit validate-topdata +../sow-tools/tools/sow-toolkit build-topdata +../sow-tools/tools/sow-toolkit compare-topdata +``` + +Or via the repo wrappers: + +```bash +./validate-topdata.sh +./build-topdata.sh +./compare-topdata.sh +``` + +Expected success output today: + +- `validate-topdata` prints `topdata validation: ok` +- `build-topdata` prints `mode: reference` and output directories under `build/topdata` +- `compare-topdata` prints `topdata compare: ok` ## Related UX State