sow-tools/TOPDATA_HARDENING_CONTRACT.md

Native Topdata Hardening Contract

Scope

This contract governs the current topdata hardening and bugfixing phase across:

• toolkit/ as implementation owner
• module/ as the primary authored topdata consumer

This is an active-priority contract.

Current Priority

Focus now on:

1. removing outdated explicit namespace/path hardcoding
2. fixing validation/build mismatches
3. making native topdata behavior generic where the data model is already generic
4. improving diagnostics and operator-facing output for topdata commands

Defer for now:

• model compilation
• release patch-note automation
• unrelated asset-processing expansion
• non-topdata feature work that does not unblock wiki deployment

Wiki deployment itself is a follow-on phase after this hardening pass, because build-wiki and deploy-wiki depend on topdata outputs and metadata being stable.

Problem Statement

Native topdata has reached the point where most major functionality exists, but too much behavior is still implemented as explicit namespace knowledge instead of generic dataset rules.

One concrete example already observed:

• dataset discovery in internal/topdata/native.go already derives a default output name when base.json omits it
• validation in internal/topdata/topdata.go still hard-fails specifically for topdata/data/masterfeats/base.json if output is missing

That mismatch proves the current methodology is stale:

• the runtime/build side treats masterfeats like a generic dataset
• the validator still treats it like a special namespace with explicit required metadata

This same category of problem appears in several other places.

Findings From Current Audit

1. Validation dispatch is path-special-cased

validateDataObject() currently routes behavior through explicit path checks for:

• masterfeats/base.json
• masterfeats/lock.json
• feat/base.json
• feat/lock.json
• feat/generated/*
• parts/overrides/*
• racialtypes/registry/races/*

Some of these are legitimate dataset-shape differences. Others are stale special-cases that should be generalized.

2. masterfeats still carries outdated explicit validation rules

validateMasterfeatsBaseFile() currently requires:

• output == masterfeats.2da
• specific required columns
• masterfeats: row keys

The key-prefix and column requirements may still be valid dataset invariants. The output requirement is not clearly justified anymore when generic dataset discovery already computes a default output name from location.

3. Migration still seeds explicit dataset exceptions

feat_migrate.go explicitly writes:

• output = "feat.2da"
• compare_reference = false

This may still be operationally correct, but it means migration continues to materialize special-case control fields instead of relying on a clearer generic contract for dataset reference comparison.

4. Project-level topdata defaults are still mixed

internal/project still owns several topdata defaults directly:

• build directory derivation
• compiled 2DA subdirectory
• compiled TLK name
• wiki root/pages/state defaults
• wiki deploy manifest and edit summary defaults
• package HAK/TLK fallback names

Some of these are valid toolkit defaults. Some should be reviewed as explicit schema-backed config.

5. Wiki generation and deploy depend on stable dataset semantics

build-wiki and deploy-wiki are downstream consumers of native topdata state. If dataset rules remain partly explicit and partly generic, wiki generation will keep inheriting brittle assumptions.

Hardening Rules

1. If a rule can be inferred generically from dataset shape, do not hardcode it to a namespace path.
2. If a rule is truly dataset-specific, document why it is intrinsic rather than configuration-driven.
3. Validation and build behavior must agree. Discovery defaults and validation requirements cannot contradict each other.
4. Dataset-authored JSON should remain the source of truth for dataset-level semantics; root YAML should not absorb dataset internals just to avoid code cleanup.
5. Error messages must point to the actual invariant being violated, not an outdated historical implementation detail.

Immediate Work Items

Work Item 1: Remove outdated masterfeats.output hard requirement

Investigate and likely change validateMasterfeatsBaseFile() so:

• missing output is accepted when generic dataset discovery can derive the same output name deterministically
• explicit output remains allowed
• wrong explicit output still fails clearly if the dataset truly requires a fixed emitted file name

This should be test-covered.

Work Item 2: Audit path-based validator routing

Classify every explicit route in validateDataObject() as one of:

• generic canonical shape handling
• dataset-family behavior that should become generic
• deliberate intrinsic special-case with justification

Reduce the explicit route set where possible.

Work Item 3: Define topdata invariant boundaries

For each current explicit topdata behavior, decide whether it belongs in:

• root YAML config
• dataset JSON
• toolkit invariant

Do not move dataset-authored semantics into YAML unless the behavior truly varies at repository level rather than dataset level.

Work Item 4: Revisit compare_reference ownership

feat.2da is still excluded via compare_reference = false. Confirm whether:

• this remains a temporary rollout flag
• or compare coverage needs a more explicit generic contract for generated-family datasets

Work Item 5: Bring topdata logging up to current CLI standards

Coordinate with LOG_REFACTOR_CONTRACT.md so topdata commands emit:

• deterministic phase summaries
• concise validation/build/compare totals
• clear file/namespace diagnostics

Testing Requirements

Add or update tests for:

• masterfeats/base.json without explicit output when the derived output would still be masterfeats.2da
• explicit wrong output for masterfeats
• validator/build parity for dataset output naming
• any reduction in path-special-case validation routing
• topdata command output summaries if presentation changes land in the same phase

Acceptance Criteria

• topdata validation no longer rejects data solely because of stale explicit namespace assumptions
• build and validation agree on dataset output-name behavior
• path-special-casing is reduced or explicitly justified
• remaining dataset-specific invariants are documented as such
• topdata hardening leaves wiki generation/deployment with a cleaner and more stable contract