# TopData Rebuild Handoff - 2026-04-02

## Current State

This handoff captures the first implementation milestone of the topdata rebuild.

The migration direction is:

- Python `2dabuilder` in `sow-dev` remains authoritative
- `sow-toolkit` now owns the new topdata command surface
- `sow-module` is the new long-term source home for migrated topdata content
- wiki generation, wiki deploy, and tophak packaging are still deferred

## Implemented In This Milestone

### New `sow-toolkit` commands

- `validate-topdata`
- `build-topdata`
- `compare-topdata`

Current behavior:

- `validate-topdata` validates the new `topdata/` source root in `sow-module`
- `build-topdata` currently uses the Python builder as a reference backend and writes outputs into `build/topdata/`
- `compare-topdata` rebuilds fresh reference output from Python and compares it against `build/topdata/`

This is an intentional bridge: it preserves current production behavior while the deeper merge/build rules are audited and ported.

### New `sow-module` layout

Recommended migration root now exists:

```text
topdata/
  data/
  tlk/
  base_dialog.json
```

Generated outputs are expected under:

```text
build/topdata/
  2da/
  tlk/
```

### Canonical compatibility validation

The new validator already recognizes the current family of authoring shapes:

- dataset `base.json` with `columns` + `rows`
- lockfiles as object maps of key -> numeric id
- module files using `entries`
- module files using `overrides`
- plain row files using `rows`
- TLK files with `entries`

It is intentionally permissive for now:

- legacy parseable shapes can still load
- warnings are used where a file is parseable but not canonical yet
- this keeps migration practical while still nudging toward a formal schema

## Important Rules Captured So Far

These behaviors are considered core and must be preserved in the eventual native compiler:

- base fallback from `base.json`
- TLK fallback from `base_dialog.json` / TLK base data
- snippet files and grouped files both being valid
- grouped dataset families and plain tables both being valid
- lockfile-backed IDs
- ID allocation when a key has no existing lock entry
- “only override what we intentionally touch”
- deterministic output and deterministic compare behavior

## What Is Still Deferred

- native Go merge/build parity for current Python rules
- shorthand expansion porting
- dataset-specific rule porting
- local wiki generation
- wiki deploy
- tophak packaging
- canonical normalize/export command

## Suggested Next Steps

1. Audit the current Python rules dataset-by-dataset, especially:
   - `classes`
   - `feat`
   - `itemprops`
   - TLK merge behavior
2. Capture representative parity fixtures from the Python builder.
3. Replace the reference-backed `build-topdata` bridge with native Go build logic one dataset family at a time.
4. Add a normalize/migrate command once the canonical JSON contract is nailed down.
5. Only migrate wiki generation after compiler parity is stable.

## Related UX State

Separate from topdata, the content repos already have:

- auto-installing `sow-toolkit` wrappers
- Linux/macOS shell wrappers
- Windows PowerShell and `.cmd` wrappers
- simplified `START-HERE.md` docs in `sow-module` and `sow-assets`

Windows still needs a real smoke test from an actual Windows machine.