sow-tools/TOPDATA_HANDOFF_2026-04-02.md

TopData Rebuild Handoff - 2026-04-02

Current State

This handoff captures the first implementation milestone of the topdata rebuild.

The migration direction is:

• Python 2dabuilder in sow-dev remains authoritative
• sow-toolkit now owns the new topdata command surface
• sow-module is the new long-term source home for migrated topdata content
• wiki generation, wiki deploy, and tophak packaging are still deferred

Relevant commits from this milestone:

• sow-tools: d67b6e3 Add topdata bridge commands and handoff
• sow-module: dfb7fd2 Add topdata migration roots and wrappers

Implemented In This Milestone

New sow-toolkit commands

• validate-topdata
• build-topdata
• compare-topdata

Current behavior:

• validate-topdata validates the new topdata/ source root in sow-module
• build-topdata currently uses the Python builder as a reference backend and writes outputs into build/topdata/
• compare-topdata rebuilds fresh reference output from Python and compares it against build/topdata/

This is an intentional bridge: it preserves current production behavior while the deeper merge/build rules are audited and ported.

Important implementation detail:

• the Python builder is run from a copied temporary workspace, not in place from sow-dev
• this prevents the reference bridge from mutating repo-tracked state in sow-dev
• the copied bridge rewrites its default asset/staging roots to temp-local paths before running
• this was necessary because the Python builder also compiles sow_top.hak as a side effect of its normal build flow

New sow-module layout

Recommended migration root now exists:

topdata/
  data/
  tlk/
  base_dialog.json

Generated outputs are expected under:

build/topdata/
  2da/
  tlk/

Current sow-module config now points at:

• topdata.source = "topdata"
• topdata.build = "build/topdata"
• topdata.reference_builder = "../sow-dev/tools/2dabuilder"

Canonical compatibility validation

The new validator already recognizes the current family of authoring shapes:

• dataset base.json with columns + rows
• lockfiles as object maps of key -> numeric id
• module files using entries
• module files using overrides
• plain row files using rows
• TLK files with entries

It is intentionally permissive for now:

• legacy parseable shapes can still load
• warnings are used where a file is parseable but not canonical yet
• this keeps migration practical while still nudging toward a formal schema

It does not yet rewrite or normalize files on disk. This milestone is validation and bridge-first, not migration automation.

What Was Verified

The new bridge was verified on the real project, not only on unit tests.

Verified successfully:

• go test ./... in sow-tools
• sow-toolkit validate-topdata
• sow-toolkit build-topdata
• sow-toolkit compare-topdata

Observed real output:

• build/topdata/2da produced 113 generated .2da files
• build/topdata/tlk produced 1 TLK output
• compare-topdata passed against a fresh Python reference build

This means the first milestone is not just scaffolding; it already gives sow-module a working topdata command surface backed by the authoritative Python builder.

Important Rules Captured So Far

These behaviors are considered core and must be preserved in the eventual native compiler:

• base fallback from base.json
• TLK fallback from base_dialog.json / TLK base data
• snippet files and grouped files both being valid
• grouped dataset families and plain tables both being valid
• lockfile-backed IDs
• ID allocation when a key has no existing lock entry
• “only override what we intentionally touch”
• deterministic output and deterministic compare behavior
• grouped dataset families such as classes
• plain-table handling alongside base+module datasets
• compatibility with both snippet files and grouped files

Current Rough Edges

These are known limitations of the bridge and should be expected when resuming:

• build-topdata and compare-topdata are slower than the future native implementation
  • they currently create a temporary copied Python builder workspace
  • they install the Python requirements into a temporary venv each run
• the bridge still mirrors the Python builder’s semantics, including the tophak side-effect path, but now safely sandboxed in temp space
• canonical schema validation is only partial so far
  • enough to validate root shape families and obvious errors
  • not enough yet to encode all dataset-specific rules
• migrated data has not actually been moved into sow-module/topdata yet beyond the new root placeholders

What Is Still Deferred

• native Go merge/build parity for current Python rules
• shorthand expansion porting
• dataset-specific rule porting
• parity fixtures checked into sow-tools
• schema documentation for the canonical topdata JSON contract
• local wiki generation
• wiki deploy
• tophak packaging
• canonical normalize/export command

Suggested Next Steps

1. Audit the current Python rules dataset-by-dataset, especially:
   • classes
   • feat
   • itemprops
   • TLK merge behavior
2. Write the canonical topdata JSON contract down explicitly.
   • root object families
   • row/entry/override forms
   • keyed refs
   • TLK refs
   • lockfile expectations
3. Capture representative parity fixtures from the Python builder.
4. Replace the reference-backed build-topdata bridge with native Go build logic one dataset family at a time.
5. Add a normalize/migrate command once the canonical JSON contract is nailed down.
6. Only migrate wiki generation after compiler parity is stable.

Resume Commands

From sow-module:

../sow-tools/tools/sow-toolkit validate-topdata
../sow-tools/tools/sow-toolkit build-topdata
../sow-tools/tools/sow-toolkit compare-topdata

Or via the repo wrappers:

./validate-topdata.sh
./build-topdata.sh
./compare-topdata.sh

Expected success output today:

• validate-topdata prints topdata validation: ok
• build-topdata prints mode: reference and output directories under build/topdata
• compare-topdata prints topdata compare: ok

Related UX State

Separate from topdata, the content repos already have:

• auto-installing sow-toolkit wrappers
• Linux/macOS shell wrappers
• Windows PowerShell and .cmd wrappers
• simplified START-HERE.md docs in sow-module and sow-assets

Windows still needs a real smoke test from an actual Windows machine.