sow-tools/TOPDATA_HANDOFF_2026-04-02.md

# TopData Rebuild Handoff - 2026-04-02

## Current State

This handoff captures the first implementation milestone of the topdata rebuild.

The migration direction is:

- Python `2dabuilder` in `sow-dev` remains authoritative
- `sow-toolkit` now owns the new topdata command surface
- `sow-module` is the new long-term source home for migrated topdata content
- wiki generation, wiki deploy, and tophak packaging are still deferred

Relevant commits from this milestone:

- `sow-tools`: `d67b6e3` `Add topdata bridge commands and handoff`
- `sow-module`: `dfb7fd2` `Add topdata migration roots and wrappers`

## Implemented In This Milestone

### New `sow-toolkit` commands

- `validate-topdata`
- `build-topdata`
- `compare-topdata`

Current behavior:

- `validate-topdata` validates the new `topdata/` source root in `sow-module`
- `build-topdata` currently uses the Python builder as a reference backend and writes outputs into `build/topdata/`
- `compare-topdata` rebuilds fresh reference output from Python and compares it against `build/topdata/`

This is an intentional bridge: it preserves current production behavior while the deeper merge/build rules are audited and ported.

Important implementation detail:

- the Python builder is run from a copied temporary workspace, not in place from `sow-dev`
- this prevents the reference bridge from mutating repo-tracked state in `sow-dev`
- the copied bridge rewrites its default asset/staging roots to temp-local paths before running
- this was necessary because the Python builder also compiles `sow_top.hak` as a side effect of its normal build flow

### New `sow-module` layout

Recommended migration root now exists:

```text
topdata/
  data/
  tlk/
  base_dialog.json
```

Generated outputs are expected under:

```text
build/topdata/
  2da/
  tlk/
```

Current `sow-module` config now points at:

- `topdata.source = "topdata"`
- `topdata.build = "build/topdata"`
- `topdata.reference_builder = "../sow-dev/tools/2dabuilder"`

### Canonical compatibility validation

The new validator already recognizes the current family of authoring shapes:

- dataset `base.json` with `columns` + `rows`
- lockfiles as object maps of key -> numeric id
- module files using `entries`
- module files using `overrides`
- plain row files using `rows`
- TLK files with `entries`

It is intentionally permissive for now:

- legacy parseable shapes can still load
- warnings are used where a file is parseable but not canonical yet
- this keeps migration practical while still nudging toward a formal schema

It does **not** yet rewrite or normalize files on disk. This milestone is validation and bridge-first, not migration automation.

## What Was Verified

The new bridge was verified on the real project, not only on unit tests.

Verified successfully:

- `go test ./...` in `sow-tools`
- `sow-toolkit validate-topdata`
- `sow-toolkit build-topdata`
- `sow-toolkit compare-topdata`

Observed real output:

- `build/topdata/2da` produced `113` generated `.2da` files
- `build/topdata/tlk` produced `1` TLK output
- `compare-topdata` passed against a fresh Python reference build

This means the first milestone is not just scaffolding; it already gives `sow-module` a working topdata command surface backed by the authoritative Python builder.

## Important Rules Captured So Far

These behaviors are considered core and must be preserved in the eventual native compiler:

- base fallback from `base.json`
- TLK fallback from `base_dialog.json` / TLK base data
- snippet files and grouped files both being valid
- grouped dataset families and plain tables both being valid
- lockfile-backed IDs
- ID allocation when a key has no existing lock entry
- “only override what we intentionally touch”
- deterministic output and deterministic compare behavior
- grouped dataset families such as `classes`
- plain-table handling alongside base+module datasets
- compatibility with both snippet files and grouped files

## Current Rough Edges

These are known limitations of the bridge and should be expected when resuming:

- `build-topdata` and `compare-topdata` are slower than the future native implementation
  - they currently create a temporary copied Python builder workspace
  - they install the Python requirements into a temporary venv each run
- the bridge still mirrors the Python builder’s semantics, including the tophak side-effect path, but now safely sandboxed in temp space
- canonical schema validation is only partial so far
  - enough to validate root shape families and obvious errors
  - not enough yet to encode all dataset-specific rules
- migrated data has not actually been moved into `sow-module/topdata` yet beyond the new root placeholders

## What Is Still Deferred

- native Go merge/build parity for current Python rules
- shorthand expansion porting
- dataset-specific rule porting
- parity fixtures checked into `sow-tools`
- schema documentation for the canonical topdata JSON contract
- local wiki generation
- wiki deploy
- tophak packaging
- canonical normalize/export command

## Suggested Next Steps

1. Audit the current Python rules dataset-by-dataset, especially:
   - `classes`
   - `feat`
   - `itemprops`
   - TLK merge behavior
2. Write the canonical topdata JSON contract down explicitly.
   - root object families
   - row/entry/override forms
   - keyed refs
   - TLK refs
   - lockfile expectations
3. Capture representative parity fixtures from the Python builder.
4. Replace the reference-backed `build-topdata` bridge with native Go build logic one dataset family at a time.
5. Add a normalize/migrate command once the canonical JSON contract is nailed down.
6. Only migrate wiki generation after compiler parity is stable.

## Resume Commands

From `sow-module`:

```bash
../sow-tools/tools/sow-toolkit validate-topdata
../sow-tools/tools/sow-toolkit build-topdata
../sow-tools/tools/sow-toolkit compare-topdata
```

Or via the repo wrappers:

```bash
./validate-topdata.sh
./build-topdata.sh
./compare-topdata.sh
```

Expected success output today:

- `validate-topdata` prints `topdata validation: ok`
- `build-topdata` prints `mode: reference` and output directories under `build/topdata`
- `compare-topdata` prints `topdata compare: ok`

## Related UX State

Separate from topdata, the content repos already have:

- auto-installing `sow-toolkit` wrappers
- Linux/macOS shell wrappers
- Windows PowerShell and `.cmd` wrappers
- simplified `START-HERE.md` docs in `sow-module` and `sow-assets`

Windows still needs a real smoke test from an actual Windows machine.