# Native Topdata Hardening Contract

## Scope

This contract governs the current topdata hardening and bugfixing phase across:

- `toolkit/` as implementation owner
- `module/` as the primary authored topdata consumer

This is an active-priority contract.

## Current Priority

Focus now on:

1. removing outdated explicit namespace/path hardcoding
2. fixing validation/build mismatches
3. making native topdata behavior generic where the data model is already generic
4. improving diagnostics and operator-facing output for topdata commands

Defer for now:

- model compilation
- release patch-note automation
- unrelated asset-processing expansion
- non-topdata feature work that does not unblock wiki deployment

Wiki deployment itself is a follow-on phase after this hardening pass, because
`build-wiki` and `deploy-wiki` depend on topdata outputs and metadata being
stable.

## Problem Statement

Native topdata has reached the point where most major functionality exists, but
too much behavior is still implemented as explicit namespace knowledge instead of
generic dataset rules.

One concrete example already observed:

- dataset discovery in `internal/topdata/native.go` already derives a default
  `output` name when `base.json` omits it
- validation in `internal/topdata/topdata.go` still hard-fails specifically for
  `topdata/data/masterfeats/base.json` if `output` is missing

That mismatch proved the current methodology was stale:

- the runtime/build side treats `masterfeats` like a generic dataset
- the validator still treats it like a special namespace with explicit required
  metadata

Current status: the output-name mismatch has been fixed for `masterfeats` and
the equivalent `feat` base dataset rule by sharing the derived output-name rule
between discovery and validation.

This same category of problem appears in several other places.

## Findings From Current Audit

### 1. Validation dispatch is path-special-cased

`validateDataObject()` currently routes behavior through explicit path checks for:

- `masterfeats/base.json`
- `masterfeats/lock.json`
- `feat/base.json`
- `feat/lock.json`
- `feat/generated/*`
- `parts/overrides/*`
- `racialtypes/registry/races/*`

Some of these are legitimate dataset-shape differences. Others are stale
special-cases that should be generalized.

### 2. `masterfeats` carries dataset-specific invariants

`validateMasterfeatsBaseFile()` currently enforces:

- explicit `output == masterfeats.2da` when `output` is authored
- specific required columns
- `masterfeats:` row keys

The key-prefix and column requirements may still be valid dataset invariants.
The stale requirement that `output` must be authored has been removed; validation
now shares the same derived default output-name rule as native discovery.

### 3. Migration still seeds explicit dataset exceptions

`feat_migrate.go` explicitly writes:

- `output = "feat.2da"`
- `compare_reference = false`

This may still be operationally correct, but it means migration continues to
materialize special-case control fields instead of relying on a clearer generic
contract for dataset reference comparison.

### 4. Project-level topdata defaults are still mixed

`internal/project` still owns several topdata defaults directly:

- build directory derivation
- compiled 2DA subdirectory
- compiled TLK name
- wiki root/pages/state defaults
- wiki deploy manifest and edit summary defaults
- package HAK/TLK fallback names

Some of these are valid toolkit defaults. Some should be reviewed as explicit
schema-backed config.

### 5. Wiki generation and deploy depend on stable dataset semantics

`build-wiki` and `deploy-wiki` are downstream consumers of native topdata state.
If dataset rules remain partly explicit and partly generic, wiki generation will
keep inheriting brittle assumptions.

## Hardening Rules

1. If a rule can be inferred generically from dataset shape, do not hardcode it
   to a namespace path.
2. If a rule is truly dataset-specific, document why it is intrinsic rather than
   configuration-driven.
3. Validation and build behavior must agree. Discovery defaults and validation
   requirements cannot contradict each other.
4. Dataset-authored JSON should remain the source of truth for dataset-level
   semantics; root YAML should not absorb dataset internals just to avoid code
   cleanup.
5. Error messages must point to the actual invariant being violated, not an
   outdated historical implementation detail.

## Immediate Work Items

### Work Item 1: Remove outdated `masterfeats.output` hard requirement

Status: implemented.

`validateMasterfeatsBaseFile()` now:

- missing `output` is accepted when generic dataset discovery can derive the same
  output name deterministically
- explicit `output` remains allowed
- wrong explicit `output` still fails clearly if the dataset truly requires a
  fixed emitted file name

This should be test-covered.

### Work Item 2: Audit path-based validator routing

Status: implemented for root dataset invariants; family-specific routes remain
where the authored shape is not a generic base/module/lock file.

Classify every explicit route in `validateDataObject()` as one of:

- generic canonical shape handling
- dataset-family behavior that should become generic
- deliberate intrinsic special-case with justification

Reduce the explicit route set where possible.

Current classification:

- `masterfeats/base.json`, `masterfeats/lock.json`, `feat/base.json`, and
  `feat/lock.json` are now handled by a generic dataset invariant table. These
  remain toolkit invariants because the row key namespaces and required columns
  are intrinsic to those native datasets, not repository-level YAML choices.
- `feat/generated/*` remains an intrinsic dataset-family route. Generated feat
  files support compact family expansion and generated entries/overrides, which
  are not generic module shapes.
- `parts/overrides/*` remains a dataset-family route for non-module override
  files outside the standard `modules/` tree.
- `racialtypes/registry/races/*` remains a registry-family route because those
  files describe race registry records, optional feat-table generation, and core
  row payloads rather than generic row/module documents.

### Work Item 3: Define topdata invariant boundaries

For each current explicit topdata behavior, decide whether it belongs in:

- root YAML config
- dataset JSON
- toolkit invariant

Do not move dataset-authored semantics into YAML unless the behavior truly varies
at repository level rather than dataset level.

### Work Item 4: Revisit `compare_reference` ownership

`feat.2da` is still excluded via `compare_reference = false`. Confirm whether:

- this remains a temporary rollout flag
- or compare coverage needs a more explicit generic contract for generated-family
  datasets

### Work Item 5: Bring topdata logging up to current CLI standards

Coordinate with `LOG_REFACTOR_CONTRACT.md` so topdata commands emit:

- deterministic phase summaries
- concise validation/build/compare totals
- clear file/namespace diagnostics

## Testing Requirements

Add or update tests for:

- `masterfeats/base.json` without explicit `output` when the derived output would
  still be `masterfeats.2da`
- explicit wrong `output` for `masterfeats`
- validator/build parity for dataset output naming
- any reduction in path-special-case validation routing
- topdata command output summaries if presentation changes land in the same phase

## Acceptance Criteria

- topdata validation no longer rejects data solely because of stale explicit
  namespace assumptions
- build and validation agree on dataset output-name behavior
- path-special-casing is reduced or explicitly justified
- remaining dataset-specific invariants are documented as such
- topdata hardening leaves wiki generation/deployment with a cleaner and more
  stable contract