stricter topdata validation (#24)
makes topdata json validation much stricter, currently still too permissive Reviewed-on: #24 Reviewed-by: xtul <mpiasecki720@protonmail.com> Co-authored-by: vickydotbat <vickydotbat@tutamail.com> Co-committed-by: vickydotbat <vickydotbat@tutamail.com>
This commit was merged in pull request #24.
This commit is contained in:
@@ -0,0 +1,269 @@
|
||||
# Topdata JSON Validation Tightening Design
|
||||
|
||||
**Date:** 2026-06-25
|
||||
**Status:** Implemented and reviewed
|
||||
**Scope:** `sow-tools`
|
||||
|
||||
## Problem
|
||||
|
||||
Topdata JSON validation checks many required properties and value types, but it
|
||||
often ignores unknown structural properties. This allows plausible typos to
|
||||
pass validation and then be ignored by the builder.
|
||||
|
||||
The immediate failure occurred in
|
||||
`topdata/data/classes/feats/global.json`: an injection used
|
||||
`when_present` even though the implemented condition is `require_present`.
|
||||
Validation succeeded, the builder ignored `when_present`, and the injection was
|
||||
therefore applied without the intended condition.
|
||||
|
||||
This is a fail-open authoring contract. A misspelled build-control property must
|
||||
not silently change generated game data.
|
||||
|
||||
## Goals
|
||||
|
||||
- Reject unsupported structural properties in canonical topdata JSON files.
|
||||
- Add an `any_present` global injection condition.
|
||||
- Keep standalone validation and direct native builds consistent.
|
||||
- Preserve dataset-specific row flexibility.
|
||||
- Tighten only established canonical formats; do not turn this work into a
|
||||
repository-wide schema rewrite.
|
||||
- Produce diagnostics that identify the file, object, and unsupported property.
|
||||
|
||||
## Non-goals
|
||||
|
||||
- No schema migration or formatting change to authored topdata.
|
||||
- No generic JSON Schema framework or new validation dependency.
|
||||
- No blanket closure of every specialized JSON dialect.
|
||||
- No change to generated feat family, registry, spellbook, TLK, or other
|
||||
specialized formats unless they use the canonical containers covered here.
|
||||
- No restriction of row values beyond existing column, metadata, inheritance,
|
||||
reference, TLK, and authoring-sugar contracts.
|
||||
- No compatibility alias for `when_present`; it is invalid syntax.
|
||||
|
||||
## Design
|
||||
|
||||
### Validation boundary
|
||||
|
||||
Structural objects use closed property sets. Dataset payload objects remain
|
||||
dataset-aware.
|
||||
|
||||
Closed structural objects include:
|
||||
|
||||
| Object | Supported properties |
|
||||
| --- | --- |
|
||||
| Base or plain rows-file root | `output`, `key`, `columns`, `rows` |
|
||||
| Canonical module root | `output`, `key`, `columns`, `entries`, `overrides`, `rows` |
|
||||
| `global.json` root | `columns`, `entries`, `overrides`, `defaults`, `position`, `injections` |
|
||||
| Global injection | `row`, `require_present`, `any_present`, `unless_present` |
|
||||
| Global condition | `field`, `id` |
|
||||
| Global default rule | `match`, `values` |
|
||||
| Global default match object | `source` |
|
||||
| Global default format object | `format` |
|
||||
|
||||
The allowed root set is selected from the file's existing canonical context;
|
||||
properties from unrelated formats are not accepted merely because another
|
||||
topdata file type supports them.
|
||||
|
||||
When a canonical root contains more than one supported container, each present
|
||||
container must be validated. For example, a module containing both `entries`
|
||||
and `overrides` must not stop validation after recognizing `entries`.
|
||||
|
||||
The following remain payload objects rather than global allowlists:
|
||||
|
||||
- objects inside `rows`;
|
||||
- values inside `entries`;
|
||||
- row-like objects inside `overrides`;
|
||||
- an injection's `row`;
|
||||
- a default rule's `values`.
|
||||
|
||||
Those objects continue to accept declared dataset columns plus documented row
|
||||
control and metadata fields. The builder already rejects unknown row and entry
|
||||
columns when it canonicalizes a dataset. This work must reuse or align with
|
||||
that knowledge where practical, but must not invent a single fixed row schema
|
||||
across all datasets.
|
||||
|
||||
### Moderate strictness
|
||||
|
||||
This change closes only canonical containers whose grammar is already
|
||||
established by the parser and current authored data.
|
||||
|
||||
Before adding a closed root check, implementation must inventory active
|
||||
`sow-topdata` files and existing `sow-tools` fixtures for legitimate property
|
||||
combinations. A property used by an established canonical format must be added
|
||||
to that format's explicit set rather than removed from authored data merely to
|
||||
satisfy the validator.
|
||||
|
||||
Specialized parsers remain responsible for their own object shapes. Extending
|
||||
closed-property checks to those formats is separate work.
|
||||
|
||||
### Global injection conditions
|
||||
|
||||
Global injections support three condition groups:
|
||||
|
||||
- `require_present`: every listed condition must match a current row.
|
||||
- `any_present`: at least one listed condition must match a current row.
|
||||
- `unless_present`: no listed condition may match a current row.
|
||||
|
||||
When more than one group is present, every group must pass. Conditions continue
|
||||
to use the existing shape:
|
||||
|
||||
```json
|
||||
{
|
||||
"field": "FeatIndex",
|
||||
"id": "feat:example"
|
||||
}
|
||||
```
|
||||
|
||||
Example:
|
||||
|
||||
```json
|
||||
{
|
||||
"row": {
|
||||
"FeatIndex": {
|
||||
"id": "feat:example"
|
||||
}
|
||||
},
|
||||
"require_present": [
|
||||
{
|
||||
"field": "FeatIndex",
|
||||
"id": "feat:base_requirement"
|
||||
}
|
||||
],
|
||||
"any_present": [
|
||||
{
|
||||
"field": "FeatIndex",
|
||||
"id": "masterfeats:metamagic"
|
||||
},
|
||||
{
|
||||
"field": "FeatIndex",
|
||||
"id": "masterfeats:combat"
|
||||
}
|
||||
],
|
||||
"unless_present": [
|
||||
{
|
||||
"field": "FeatIndex",
|
||||
"id": "feat:example"
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
This injection applies only when the required feat exists, either listed
|
||||
masterfeat exists, and the injected feat does not already exist.
|
||||
|
||||
`any_present` must contain at least one condition. An empty list is an
|
||||
authoring error because it cannot express a useful successful condition.
|
||||
Existing empty-list behavior for `require_present` and `unless_present` is not
|
||||
changed by this work.
|
||||
|
||||
Conditions are evaluated against the same current-row view and in the same
|
||||
injection order used today. This preserves the existing behavior where an
|
||||
earlier injection can affect a later injection's conditions.
|
||||
|
||||
### Validation and build consistency
|
||||
|
||||
`ValidateProject` must report unsupported structural properties before a build.
|
||||
The native builder must also fail closed when invoked directly with the same
|
||||
invalid control syntax.
|
||||
|
||||
The implementation should centralize:
|
||||
|
||||
- deterministic unsupported-property checks;
|
||||
- global condition parsing and validation;
|
||||
- condition group evaluation.
|
||||
|
||||
Validation may collect multiple diagnostics while building returns the first
|
||||
blocking error. Both paths must accept and reject the same property names and
|
||||
condition shapes.
|
||||
|
||||
### Diagnostics
|
||||
|
||||
Diagnostics must name the containing object and unsupported property. Property
|
||||
lists must be deterministic.
|
||||
|
||||
For the original error, an acceptable diagnostic is:
|
||||
|
||||
```text
|
||||
global injection 10 contains unsupported key "when_present";
|
||||
supported keys are row, require_present, any_present, and unless_present
|
||||
```
|
||||
|
||||
Equivalent concise wording is acceptable if tests assert the stable semantic
|
||||
parts rather than the entire sentence.
|
||||
|
||||
## Compatibility
|
||||
|
||||
Current canonical `sow-topdata` roots fit the property sets above. The
|
||||
implementation must rerun the inventory immediately before tightening checks
|
||||
and run validation against the active `sow-topdata` checkout.
|
||||
|
||||
Unknown properties are not retained as compatibility behavior. They currently
|
||||
have no defined effect, and silently accepting them is the bug being fixed.
|
||||
Existing documented properties and dataset-specific row fields remain valid.
|
||||
|
||||
`require_present` and `unless_present` retain their current semantics.
|
||||
`any_present` is additive.
|
||||
|
||||
## Testing
|
||||
|
||||
Tests must follow red-green development and cover validation and direct-build
|
||||
paths.
|
||||
|
||||
### Unsupported properties
|
||||
|
||||
- Reject an unknown `global.json` root property.
|
||||
- Reject `when_present` and other unknown injection properties.
|
||||
- Reject an unknown condition property.
|
||||
- Reject unknown properties in default rules, match objects, and format
|
||||
objects.
|
||||
- Reject unknown canonical base/plain rows-file root properties.
|
||||
- Reject unknown canonical module root properties.
|
||||
- Continue accepting legitimate combinations such as:
|
||||
- a columns-only module;
|
||||
- entries-only and overrides-only modules;
|
||||
- a module containing both entries and overrides;
|
||||
- current base and plain rows-file roots.
|
||||
- Continue accepting dataset columns and documented row control fields inside
|
||||
rows, entries, overrides, and injection rows.
|
||||
|
||||
### `any_present`
|
||||
|
||||
- Validation accepts a non-empty `any_present` condition list.
|
||||
- Validation rejects a non-array or empty `any_present`.
|
||||
- A build injects when the first alternative is present.
|
||||
- A build injects when a later alternative is present.
|
||||
- A build skips the injection when no alternative is present.
|
||||
- `any_present` combines correctly with `require_present`.
|
||||
- `any_present` combines correctly with `unless_present`.
|
||||
- Existing `require_present` and `unless_present` behavior remains unchanged.
|
||||
- Direct building rejects unsupported condition syntax even when project
|
||||
validation was not called first.
|
||||
|
||||
### Integration
|
||||
|
||||
- Run focused `internal/topdata` tests.
|
||||
- Run `nix develop --command make check`.
|
||||
- Validate the active `sow-topdata` checkout with the changed Crucible.
|
||||
- Build topdata far enough to exercise global injections and confirm no
|
||||
generated files are tracked.
|
||||
|
||||
## Documentation
|
||||
|
||||
Update the class feat global injection contract to document `require_present`,
|
||||
`any_present`, and `unless_present`, including their all/any/none semantics and
|
||||
how multiple groups combine.
|
||||
|
||||
No broad validation guide is added unless implementation reveals an existing
|
||||
operator document that would otherwise become stale.
|
||||
|
||||
## Acceptance criteria
|
||||
|
||||
- `when_present` in a global injection fails validation and direct building.
|
||||
- `any_present` is valid only as a non-empty list of valid conditions.
|
||||
- `any_present` applies an injection when one or more alternatives exist and
|
||||
skips it when none exist.
|
||||
- Unknown canonical structural properties fail with actionable diagnostics.
|
||||
- Dataset-specific row fields remain valid when declared by the dataset.
|
||||
- Existing active `sow-topdata` validates and builds after replacing invalid
|
||||
syntax with the documented grammar.
|
||||
- No specialized topdata format is tightened outside the stated scope.
|
||||
Reference in New Issue
Block a user