111 lines
3.3 KiB
Markdown
111 lines
3.3 KiB
Markdown
# TopData Rebuild Handoff - 2026-04-02
|
|
|
|
## Current State
|
|
|
|
This handoff captures the first implementation milestone of the topdata rebuild.
|
|
|
|
The migration direction is:
|
|
|
|
- Python `2dabuilder` in `sow-dev` remains authoritative
|
|
- `sow-toolkit` now owns the new topdata command surface
|
|
- `sow-module` is the new long-term source home for migrated topdata content
|
|
- wiki generation, wiki deploy, and tophak packaging are still deferred
|
|
|
|
## Implemented In This Milestone
|
|
|
|
### New `sow-toolkit` commands
|
|
|
|
- `validate-topdata`
|
|
- `build-topdata`
|
|
- `compare-topdata`
|
|
|
|
Current behavior:
|
|
|
|
- `validate-topdata` validates the new `topdata/` source root in `sow-module`
|
|
- `build-topdata` currently uses the Python builder as a reference backend and writes outputs into `build/topdata/`
|
|
- `compare-topdata` rebuilds fresh reference output from Python and compares it against `build/topdata/`
|
|
|
|
This is an intentional bridge: it preserves current production behavior while the deeper merge/build rules are audited and ported.
|
|
|
|
### New `sow-module` layout
|
|
|
|
Recommended migration root now exists:
|
|
|
|
```text
|
|
topdata/
|
|
data/
|
|
tlk/
|
|
base_dialog.json
|
|
```
|
|
|
|
Generated outputs are expected under:
|
|
|
|
```text
|
|
build/topdata/
|
|
2da/
|
|
tlk/
|
|
```
|
|
|
|
### Canonical compatibility validation
|
|
|
|
The new validator already recognizes the current family of authoring shapes:
|
|
|
|
- dataset `base.json` with `columns` + `rows`
|
|
- lockfiles as object maps of key -> numeric id
|
|
- module files using `entries`
|
|
- module files using `overrides`
|
|
- plain row files using `rows`
|
|
- TLK files with `entries`
|
|
|
|
It is intentionally permissive for now:
|
|
|
|
- legacy parseable shapes can still load
|
|
- warnings are used where a file is parseable but not canonical yet
|
|
- this keeps migration practical while still nudging toward a formal schema
|
|
|
|
## Important Rules Captured So Far
|
|
|
|
These behaviors are considered core and must be preserved in the eventual native compiler:
|
|
|
|
- base fallback from `base.json`
|
|
- TLK fallback from `base_dialog.json` / TLK base data
|
|
- snippet files and grouped files both being valid
|
|
- grouped dataset families and plain tables both being valid
|
|
- lockfile-backed IDs
|
|
- ID allocation when a key has no existing lock entry
|
|
- “only override what we intentionally touch”
|
|
- deterministic output and deterministic compare behavior
|
|
|
|
## What Is Still Deferred
|
|
|
|
- native Go merge/build parity for current Python rules
|
|
- shorthand expansion porting
|
|
- dataset-specific rule porting
|
|
- local wiki generation
|
|
- wiki deploy
|
|
- tophak packaging
|
|
- canonical normalize/export command
|
|
|
|
## Suggested Next Steps
|
|
|
|
1. Audit the current Python rules dataset-by-dataset, especially:
|
|
- `classes`
|
|
- `feat`
|
|
- `itemprops`
|
|
- TLK merge behavior
|
|
2. Capture representative parity fixtures from the Python builder.
|
|
3. Replace the reference-backed `build-topdata` bridge with native Go build logic one dataset family at a time.
|
|
4. Add a normalize/migrate command once the canonical JSON contract is nailed down.
|
|
5. Only migrate wiki generation after compiler parity is stable.
|
|
|
|
## Related UX State
|
|
|
|
Separate from topdata, the content repos already have:
|
|
|
|
- auto-installing `sow-toolkit` wrappers
|
|
- Linux/macOS shell wrappers
|
|
- Windows PowerShell and `.cmd` wrappers
|
|
- simplified `START-HERE.md` docs in `sow-module` and `sow-assets`
|
|
|
|
Windows still needs a real smoke test from an actual Windows machine.
|