sow-tools/TOPDATA_HANDOFF_2026-04-02.md

TopData Rebuild Handoff - 2026-04-02

Current State

This handoff captures the first implementation milestone of the topdata rebuild.

The migration direction is:

• Python 2dabuilder in sow-dev remains authoritative
• sow-toolkit now owns the new topdata command surface
• sow-module is the new long-term source home for migrated topdata content
• wiki generation, wiki deploy, and tophak packaging are still deferred

Implemented In This Milestone

New sow-toolkit commands

• validate-topdata
• build-topdata
• compare-topdata

Current behavior:

• validate-topdata validates the new topdata/ source root in sow-module
• build-topdata currently uses the Python builder as a reference backend and writes outputs into build/topdata/
• compare-topdata rebuilds fresh reference output from Python and compares it against build/topdata/

This is an intentional bridge: it preserves current production behavior while the deeper merge/build rules are audited and ported.

New sow-module layout

Recommended migration root now exists:

topdata/
  data/
  tlk/
  base_dialog.json

Generated outputs are expected under:

build/topdata/
  2da/
  tlk/

Canonical compatibility validation

The new validator already recognizes the current family of authoring shapes:

• dataset base.json with columns + rows
• lockfiles as object maps of key -> numeric id
• module files using entries
• module files using overrides
• plain row files using rows
• TLK files with entries

It is intentionally permissive for now:

• legacy parseable shapes can still load
• warnings are used where a file is parseable but not canonical yet
• this keeps migration practical while still nudging toward a formal schema

Important Rules Captured So Far

These behaviors are considered core and must be preserved in the eventual native compiler:

• base fallback from base.json
• TLK fallback from base_dialog.json / TLK base data
• snippet files and grouped files both being valid
• grouped dataset families and plain tables both being valid
• lockfile-backed IDs
• ID allocation when a key has no existing lock entry
• “only override what we intentionally touch”
• deterministic output and deterministic compare behavior

What Is Still Deferred

• native Go merge/build parity for current Python rules
• shorthand expansion porting
• dataset-specific rule porting
• local wiki generation
• wiki deploy
• tophak packaging
• canonical normalize/export command

Suggested Next Steps

1. Audit the current Python rules dataset-by-dataset, especially:
   • classes
   • feat
   • itemprops
   • TLK merge behavior
2. Capture representative parity fixtures from the Python builder.
3. Replace the reference-backed build-topdata bridge with native Go build logic one dataset family at a time.
4. Add a normalize/migrate command once the canonical JSON contract is nailed down.
5. Only migrate wiki generation after compiler parity is stable.

Related UX State

Separate from topdata, the content repos already have:

• auto-installing sow-toolkit wrappers
• Linux/macOS shell wrappers
• Windows PowerShell and .cmd wrappers
• simplified START-HERE.md docs in sow-module and sow-assets

Windows still needs a real smoke test from an actual Windows machine.