stricter topdata validation (#24)
makes topdata json validation much stricter, currently still too permissive Reviewed-on: #24 Reviewed-by: xtul <mpiasecki720@protonmail.com> Co-authored-by: vickydotbat <vickydotbat@tutamail.com> Co-committed-by: vickydotbat <vickydotbat@tutamail.com>
This commit is contained in:
@@ -0,0 +1,858 @@
|
||||
# Topdata JSON Validation Tightening Implementation Plan
|
||||
|
||||
> **For agentic workers:** REQUIRED SUB-SKILL: Use
|
||||
> `superpowers:subagent-driven-development` (recommended) or
|
||||
> `superpowers:executing-plans` to implement this plan task-by-task. Steps use
|
||||
> checkbox (`- [ ]`) syntax for tracking.
|
||||
|
||||
**Goal:** Reject unsupported properties in established canonical topdata JSON
|
||||
containers, add the non-empty `any_present` global-injection condition, and keep
|
||||
standalone validation and direct native builds fail-closed and consistent.
|
||||
|
||||
**Architecture:** Add one focused `internal/topdata/json_contract.go` unit that
|
||||
owns deterministic closed-key checks plus parsing and evaluation of global
|
||||
injection condition groups. Existing validation in `topdata.go` uses that unit
|
||||
to collect actionable diagnostics, while `native.go` uses the same parsed
|
||||
conditions during ordered injection evaluation. Dataset payload rows continue
|
||||
through the existing column-aware canonicalization paths and are not assigned a
|
||||
global schema.
|
||||
|
||||
**Tech Stack:** Go 1.26, standard-library `encoding/json`, Go tests, Nix flakes,
|
||||
Make, authored JSON in the sibling `sow-topdata` checkout.
|
||||
|
||||
## Global Constraints
|
||||
|
||||
- Work on a fresh non-`main` branch and open all changes as a pull request.
|
||||
- Never commit or push secrets, `.sops` data, generated binaries, `.cache/`,
|
||||
`generated/`, HAK/TLK artifacts, or result symlinks.
|
||||
- Add no JSON Schema framework and no new validation dependency.
|
||||
- Close only the canonical base/plain rows root, canonical module root,
|
||||
`global.json` root, global injection, global condition, global default rule,
|
||||
global default match object, and global default format object.
|
||||
- Preserve specialized parser-owned JSON dialects outside those canonical
|
||||
containers.
|
||||
- Preserve dataset-specific columns and documented row controls inside `rows`,
|
||||
`entries`, `overrides`, injection `row`, and default `values`.
|
||||
- Preserve current-row, ordered-injection behavior: an earlier injection may
|
||||
satisfy or block a later injection.
|
||||
- Preserve empty-list behavior for `require_present` and `unless_present`;
|
||||
reject an empty `any_present`.
|
||||
- Do not add a compatibility alias for `when_present`.
|
||||
- Use deterministic property ordering in diagnostics.
|
||||
- Inventory evidence requires `compare_reference` to remain valid on
|
||||
base/plain rows roots: `internal/topdata/native.go` reads it,
|
||||
`internal/topdata/feat_migrate.go` emits it, and existing tests exercise it.
|
||||
This is an established canonical property covered by the design's moderate
|
||||
strictness rule even though the initial property table omitted it.
|
||||
- The active `sow-topdata` branch
|
||||
`fix-always-present-metamagic` already replaces the known `when_present` with
|
||||
`require_present` in commit `ed139f5`; preserve that change and use the
|
||||
checkout for integration validation rather than duplicating or rewriting it.
|
||||
|
||||
---
|
||||
|
||||
### Task 1: Introduce the shared closed-object and condition contract
|
||||
|
||||
**Files:**
|
||||
|
||||
- Create: `internal/topdata/json_contract.go`
|
||||
- Create: `internal/topdata/json_contract_test.go`
|
||||
|
||||
**Interfaces:**
|
||||
|
||||
- Produces:
|
||||
|
||||
```go
|
||||
type globalCondition struct {
|
||||
Field string
|
||||
ID string
|
||||
}
|
||||
|
||||
type globalConditionGroups struct {
|
||||
RequirePresent []globalCondition
|
||||
AnyPresent []globalCondition
|
||||
UnlessPresent []globalCondition
|
||||
}
|
||||
|
||||
func unsupportedObjectKeys(obj map[string]any, supported ...string) []string
|
||||
func unsupportedObjectKeysError(label string, obj map[string]any, supported ...string) error
|
||||
func parseGlobalConditionGroups(injection map[string]any) (globalConditionGroups, error)
|
||||
func globalConditionGroupsMatch(groups globalConditionGroups, rows []map[string]any) bool
|
||||
```
|
||||
|
||||
- `unsupportedObjectKeys` returns unknown keys in lexical order.
|
||||
- `parseGlobalConditionGroups` accepts absent groups, permits empty
|
||||
`require_present` and `unless_present`, rejects empty `any_present`, validates
|
||||
condition keys as exactly `field` and `id`, and returns the first deterministic
|
||||
error.
|
||||
- `globalConditionGroupsMatch` implements all/any/none semantics without
|
||||
reparsing JSON.
|
||||
|
||||
- [ ] **Step 1: Create the feature branch after checking it was not previously
|
||||
merged and deleted**
|
||||
|
||||
```bash
|
||||
git fetch origin
|
||||
git branch --show-current
|
||||
git ls-remote --heads origin feat/topdata-json-validation
|
||||
git log --all --oneline --decorate --grep='topdata json validation'
|
||||
```
|
||||
|
||||
Expected: current branch is not `main`. If
|
||||
`feat/topdata-json-validation` has never been used, create it from the approved
|
||||
design commit:
|
||||
|
||||
```bash
|
||||
git switch -c feat/topdata-json-validation
|
||||
```
|
||||
|
||||
If that exact branch name has prior remote history, choose a new descriptive
|
||||
name and record it in the PR; do not reuse the old branch.
|
||||
|
||||
- [ ] **Step 2: Re-run the canonical-format inventory before writing the
|
||||
allowlists**
|
||||
|
||||
```bash
|
||||
for f in $(find ../sow-topdata/data -type f -name 'base.json' | sort); do
|
||||
jq -r --arg f "${f#../sow-topdata/}" \
|
||||
'[$f, (keys | sort | join(","))] | @tsv' "$f"
|
||||
done
|
||||
|
||||
for f in $(find ../sow-topdata/data -type f -name 'global.json' | sort); do
|
||||
jq -r --arg f "${f#../sow-topdata/}" \
|
||||
'[$f, (keys | sort | join(","))] | @tsv' "$f"
|
||||
done
|
||||
|
||||
find ../sow-topdata/data -type f -path '*/modules/*.json' -print0 |
|
||||
while IFS= read -r -d '' f; do
|
||||
jq -r --arg f "${f#../sow-topdata/}" \
|
||||
'[$f, (keys | sort | join(","))] | @tsv' "$f"
|
||||
done | sort
|
||||
|
||||
rg -n 'compare_reference' internal/topdata ../sow-topdata/data
|
||||
```
|
||||
|
||||
Expected: active base roots use `columns`, `key`, `output`, and `rows`;
|
||||
active modules use `columns`, `entries`, `overrides`, or the legitimate
|
||||
`entries`+`overrides` combination; active globals use the approved global
|
||||
keys. `sow-tools` additionally confirms established `compare_reference`
|
||||
support.
|
||||
|
||||
- [ ] **Step 3: Write failing unit tests for deterministic unsupported-key
|
||||
diagnostics**
|
||||
|
||||
Add table-driven tests equivalent to:
|
||||
|
||||
```go
|
||||
func TestUnsupportedObjectKeysAreSorted(t *testing.T) {
|
||||
obj := map[string]any{
|
||||
"row": map[string]any{},
|
||||
"when_present": []any{},
|
||||
"aaa": true,
|
||||
}
|
||||
|
||||
got := unsupportedObjectKeys(
|
||||
obj,
|
||||
"row",
|
||||
"require_present",
|
||||
"any_present",
|
||||
"unless_present",
|
||||
)
|
||||
want := []string{"aaa", "when_present"}
|
||||
if !slices.Equal(got, want) {
|
||||
t.Fatalf("unsupportedObjectKeys() = %v, want %v", got, want)
|
||||
}
|
||||
}
|
||||
|
||||
func TestUnsupportedObjectKeysErrorNamesObjectAndContract(t *testing.T) {
|
||||
err := unsupportedObjectKeysError(
|
||||
"global injection 10",
|
||||
map[string]any{"row": map[string]any{}, "when_present": []any{}},
|
||||
"row",
|
||||
"require_present",
|
||||
"any_present",
|
||||
"unless_present",
|
||||
)
|
||||
if err == nil {
|
||||
t.Fatal("expected unsupported-property error")
|
||||
}
|
||||
for _, want := range []string{
|
||||
"global injection 10",
|
||||
`"when_present"`,
|
||||
"row",
|
||||
"require_present",
|
||||
"any_present",
|
||||
"unless_present",
|
||||
} {
|
||||
if !strings.Contains(err.Error(), want) {
|
||||
t.Fatalf("error %q does not contain %q", err, want)
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 4: Write failing parser tests for all condition-group contracts**
|
||||
|
||||
Cover these exact cases in `json_contract_test.go`:
|
||||
|
||||
```go
|
||||
tests := []struct {
|
||||
name string
|
||||
injection map[string]any
|
||||
want globalConditionGroups
|
||||
wantError string
|
||||
}{
|
||||
{
|
||||
name: "all groups",
|
||||
injection: map[string]any{
|
||||
"require_present": []any{
|
||||
map[string]any{"field": "FeatIndex", "id": "feat:required"},
|
||||
},
|
||||
"any_present": []any{
|
||||
map[string]any{"field": "FeatIndex", "id": "feat:first"},
|
||||
map[string]any{"field": "FeatIndex", "id": "feat:second"},
|
||||
},
|
||||
"unless_present": []any{
|
||||
map[string]any{"field": "FeatIndex", "id": "feat:blocked"},
|
||||
},
|
||||
},
|
||||
want: globalConditionGroups{
|
||||
RequirePresent: []globalCondition{
|
||||
{Field: "FeatIndex", ID: "feat:required"},
|
||||
},
|
||||
AnyPresent: []globalCondition{
|
||||
{Field: "FeatIndex", ID: "feat:first"},
|
||||
{Field: "FeatIndex", ID: "feat:second"},
|
||||
},
|
||||
UnlessPresent: []globalCondition{
|
||||
{Field: "FeatIndex", ID: "feat:blocked"},
|
||||
},
|
||||
},
|
||||
},
|
||||
{
|
||||
name: "empty existing groups remain valid",
|
||||
injection: map[string]any{
|
||||
"require_present": []any{},
|
||||
"unless_present": []any{},
|
||||
},
|
||||
},
|
||||
{
|
||||
name: "any present must be array",
|
||||
injection: map[string]any{
|
||||
"any_present": "feat:first",
|
||||
},
|
||||
wantError: "any_present must be an array",
|
||||
},
|
||||
{
|
||||
name: "any present must not be empty",
|
||||
injection: map[string]any{
|
||||
"any_present": []any{},
|
||||
},
|
||||
wantError: "any_present must contain at least one condition",
|
||||
},
|
||||
{
|
||||
name: "condition is closed",
|
||||
injection: map[string]any{
|
||||
"require_present": []any{
|
||||
map[string]any{
|
||||
"field": "FeatIndex",
|
||||
"id": "feat:required",
|
||||
"typo": true,
|
||||
},
|
||||
},
|
||||
},
|
||||
wantError: `require_present[0] contains unsupported key "typo"`,
|
||||
},
|
||||
{
|
||||
name: "field is required",
|
||||
injection: map[string]any{
|
||||
"require_present": []any{
|
||||
map[string]any{"id": "feat:required"},
|
||||
},
|
||||
},
|
||||
wantError: "require_present[0].field is required",
|
||||
},
|
||||
{
|
||||
name: "id is required",
|
||||
injection: map[string]any{
|
||||
"require_present": []any{
|
||||
map[string]any{"field": "FeatIndex"},
|
||||
},
|
||||
},
|
||||
wantError: "require_present[0].id is required",
|
||||
},
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 5: Write failing evaluator tests for all/any/none and group
|
||||
conjunction**
|
||||
|
||||
Build rows with `FeatIndex` references for `feat:required`,
|
||||
`feat:second`, and `feat:blocked`, then assert:
|
||||
|
||||
```go
|
||||
if !globalConditionGroupsMatch(globalConditionGroups{
|
||||
RequirePresent: []globalCondition{
|
||||
{Field: "FeatIndex", ID: "feat:required"},
|
||||
},
|
||||
AnyPresent: []globalCondition{
|
||||
{Field: "FeatIndex", ID: "feat:first"},
|
||||
{Field: "FeatIndex", ID: "feat:second"},
|
||||
},
|
||||
}, rowsWithoutBlocked) {
|
||||
t.Fatal("expected required + later any_present alternative to match")
|
||||
}
|
||||
|
||||
if globalConditionGroupsMatch(globalConditionGroups{
|
||||
AnyPresent: []globalCondition{
|
||||
{Field: "FeatIndex", ID: "feat:missing"},
|
||||
},
|
||||
}, rowsWithoutBlocked) {
|
||||
t.Fatal("expected missing any_present alternatives to reject")
|
||||
}
|
||||
|
||||
if globalConditionGroupsMatch(globalConditionGroups{
|
||||
UnlessPresent: []globalCondition{
|
||||
{Field: "FeatIndex", ID: "feat:blocked"},
|
||||
},
|
||||
}, rowsWithBlocked) {
|
||||
t.Fatal("expected unless_present match to reject")
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 6: Run the focused tests and confirm they fail**
|
||||
|
||||
```bash
|
||||
nix develop --command go test ./internal/topdata \
|
||||
-run 'TestUnsupportedObjectKeys|TestParseGlobalConditionGroups|TestGlobalConditionGroupsMatch' \
|
||||
-v
|
||||
```
|
||||
|
||||
Expected: build failure because the shared contract types and functions do
|
||||
not exist.
|
||||
|
||||
- [ ] **Step 7: Implement the minimal shared contract**
|
||||
|
||||
In `json_contract.go`, define fixed key sets and implement:
|
||||
|
||||
```go
|
||||
var (
|
||||
baseOrPlainRootKeys = []string{
|
||||
"output", "key", "columns", "rows", "compare_reference",
|
||||
}
|
||||
canonicalModuleRootKeys = []string{
|
||||
"output", "key", "columns", "entries", "overrides", "rows",
|
||||
}
|
||||
globalRootKeys = []string{
|
||||
"columns", "entries", "overrides", "defaults", "position", "injections",
|
||||
}
|
||||
globalInjectionKeys = []string{
|
||||
"row", "require_present", "any_present", "unless_present",
|
||||
}
|
||||
globalConditionKeys = []string{"field", "id"}
|
||||
globalDefaultRuleKeys = []string{"match", "values"}
|
||||
globalDefaultMatchKeys = []string{"source"}
|
||||
globalDefaultFormatKeys = []string{"format"}
|
||||
)
|
||||
```
|
||||
|
||||
`unsupportedObjectKeysError` must report one deterministic error per call,
|
||||
using the first lexically sorted unsupported key and a supported-key list in
|
||||
declaration order. Implement condition parsing through one private helper:
|
||||
|
||||
```go
|
||||
func parseGlobalConditionList(
|
||||
name string,
|
||||
raw any,
|
||||
requireNonEmpty bool,
|
||||
) ([]globalCondition, error)
|
||||
```
|
||||
|
||||
It must reject non-arrays, reject non-object elements, run the closed-key
|
||||
check before required-field checks, trim `field` and `id`, and enforce the
|
||||
`any_present` non-empty rule.
|
||||
|
||||
`globalConditionGroupsMatch` must call the existing
|
||||
`globalReferencePresent` helper and implement:
|
||||
|
||||
```text
|
||||
require_present: every condition is found
|
||||
any_present: at least one condition is found
|
||||
unless_present: no condition is found
|
||||
all groups: every present group passes
|
||||
```
|
||||
|
||||
- [ ] **Step 8: Run the focused and package tests**
|
||||
|
||||
```bash
|
||||
nix develop --command go test ./internal/topdata \
|
||||
-run 'TestUnsupportedObjectKeys|TestParseGlobalConditionGroups|TestGlobalConditionGroupsMatch' \
|
||||
-v
|
||||
nix develop --command go test ./internal/topdata
|
||||
```
|
||||
|
||||
Expected: pass.
|
||||
|
||||
- [ ] **Step 9: Commit the shared contract**
|
||||
|
||||
```bash
|
||||
git add internal/topdata/json_contract.go internal/topdata/json_contract_test.go
|
||||
git commit -m "feat(topdata): define canonical json contracts"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Task 2: Close canonical structural objects during project validation
|
||||
|
||||
**Files:**
|
||||
|
||||
- Modify: `internal/topdata/topdata.go`
|
||||
- Modify: `internal/topdata/json_contract_test.go`
|
||||
|
||||
**Interfaces:**
|
||||
|
||||
- Consumes the key sets and condition parser from Task 1.
|
||||
- `validateDataObject` selects exactly one root contract from file context:
|
||||
base/plain rows root, canonical module root, or `global.json` root.
|
||||
- A module validates every present supported container; it does not return
|
||||
after the first of `entries`, `overrides`, `rows`, or `columns`.
|
||||
- Specialized paths continue to return to their existing validators before
|
||||
canonical root closure.
|
||||
|
||||
- [ ] **Step 1: Add failing validation tests for unsupported canonical root
|
||||
properties**
|
||||
|
||||
Add table-driven temporary-project cases covering:
|
||||
- base root with `"typo": true`;
|
||||
- loose plain rows root with `"typo": true`;
|
||||
- module root with `"typo": true`;
|
||||
- `global.json` root with `"typo": true`.
|
||||
|
||||
For every case, call `ValidateProject`, require errors, and assert only stable
|
||||
semantic fragments: the file/object label, `typo`, and `unsupported`.
|
||||
|
||||
- [ ] **Step 2: Add failing validation tests for closed nested global objects**
|
||||
|
||||
Add cases for:
|
||||
|
||||
```json
|
||||
{"injections":[{"row":{},"when_present":[]}]}
|
||||
{"injections":[{"row":{},"require_present":[{"field":"FeatIndex","id":"feat:x","typo":true}]}]}
|
||||
{"defaults":[{"match":"all","values":{},"typo":true}]}
|
||||
{"defaults":[{"match":{"source":"entries","typo":true},"values":{}}]}
|
||||
{"defaults":[{"match":"all","values":{"ImpactScript":{"format":"ss_{id}","typo":true}}}]}
|
||||
```
|
||||
|
||||
Expected diagnostics name `global injection 0`, the condition label,
|
||||
`global default 0`, `global default 0 match`, or
|
||||
`global default 0 values.ImpactScript` respectively.
|
||||
|
||||
- [ ] **Step 3: Add failing validation tests for `any_present` shape**
|
||||
|
||||
Test one accepted non-empty list and rejected string/empty-list cases. The
|
||||
accepted case must use a declared dataset column in the injection row so this
|
||||
also protects payload flexibility.
|
||||
|
||||
- [ ] **Step 4: Add regression tests for canonical combinations and payload
|
||||
flexibility**
|
||||
|
||||
Build one table-driven test that validates without errors:
|
||||
- a columns-only module;
|
||||
- an entries-only module;
|
||||
- an overrides-only module;
|
||||
- a module containing both entries and overrides;
|
||||
- a rows module;
|
||||
- a current base root;
|
||||
- a loose plain rows root;
|
||||
- a base/plain root with `compare_reference: false`;
|
||||
- dataset columns plus `id`, `key`, `inherit`, `_tlk`, and `meta` in row-like
|
||||
payloads where their existing contracts allow them;
|
||||
- an injection row using a declared dataset column.
|
||||
|
||||
The combined entries+overrides case must contain an independently invalid
|
||||
override value first, prove validation catches it, then fix only that value
|
||||
and prove the module passes. This specifically prevents the current
|
||||
first-container early return.
|
||||
|
||||
- [ ] **Step 5: Run the focused tests and confirm failures**
|
||||
|
||||
```bash
|
||||
nix develop --command go test ./internal/topdata \
|
||||
-run 'TestValidateProject(RejectsUnsupportedCanonicalJSON|RejectsUnsupportedGlobalJSON|ValidatesEveryCanonicalModuleContainer|AcceptsCanonicalJSONContracts|AcceptsAnyPresent|RejectsInvalidAnyPresent)' \
|
||||
-v
|
||||
```
|
||||
|
||||
Expected: unknown keys are currently ignored, `any_present` is not validated,
|
||||
and the combined module does not validate all containers.
|
||||
|
||||
- [ ] **Step 6: Add a validation adapter for shared contract errors**
|
||||
|
||||
In `topdata.go`, add:
|
||||
|
||||
```go
|
||||
func reportTopdataContractError(path string, err error, report *ValidationReport) {
|
||||
if err == nil {
|
||||
return
|
||||
}
|
||||
report.Diagnostics = append(report.Diagnostics, Diagnostic{
|
||||
Severity: SeverityError,
|
||||
Path: path,
|
||||
Message: err.Error(),
|
||||
})
|
||||
}
|
||||
```
|
||||
|
||||
Use `unsupportedObjectKeysError` for canonical roots and nested structural
|
||||
objects. Keep row payload validation in existing `validateRowCollection`,
|
||||
`validateEntriesFile`, and `validateOverridesFile` paths.
|
||||
|
||||
- [ ] **Step 7: Refactor canonical module dispatch to validate all containers**
|
||||
|
||||
After specialized-path checks, classify module files by directory context.
|
||||
For recognized canonical modules:
|
||||
1. check `canonicalModuleRootKeys`;
|
||||
2. call `validateColumnsFile` when `columns` is present;
|
||||
3. call `validateEntriesFile` when `entries` is present;
|
||||
4. call `validateOverridesFile` when `overrides` is present;
|
||||
5. call `validateRowsFile(path, obj, report, false)` when `rows` is present;
|
||||
6. emit the existing warning only when none of those containers is present.
|
||||
|
||||
Avoid duplicate `validateColumnsFile` calls by removing the nested column
|
||||
calls from this module dispatch path or by adding a caller-controlled flag;
|
||||
retain current behavior for callers outside canonical module dispatch.
|
||||
|
||||
- [ ] **Step 8: Close global roots, injections, defaults, matches, and format
|
||||
objects**
|
||||
- Run the global-root key check before validating present containers.
|
||||
- Run the injection key check before validating `row` and condition groups.
|
||||
- Replace calls to `validateGlobalConditionList` with
|
||||
`parseGlobalConditionGroups`; report its error through
|
||||
`reportTopdataContractError`.
|
||||
- Run default-rule and match-object key checks before existing type/value
|
||||
checks.
|
||||
- Treat an object containing `format` as a format object, reject every key
|
||||
other than `format`, and preserve non-format object literals as payload
|
||||
values.
|
||||
|
||||
- [ ] **Step 9: Run focused and full package tests**
|
||||
|
||||
```bash
|
||||
nix develop --command go test ./internal/topdata \
|
||||
-run 'TestValidateProject(RejectsUnsupportedCanonicalJSON|RejectsUnsupportedGlobalJSON|ValidatesEveryCanonicalModuleContainer|AcceptsCanonicalJSONContracts|AcceptsAnyPresent|RejectsInvalidAnyPresent)' \
|
||||
-v
|
||||
nix develop --command go test ./internal/topdata
|
||||
```
|
||||
|
||||
Expected: pass.
|
||||
|
||||
- [ ] **Step 10: Commit validation tightening**
|
||||
|
||||
```bash
|
||||
git add internal/topdata/topdata.go internal/topdata/json_contract_test.go
|
||||
git commit -m "feat(topdata): reject unsupported canonical json properties"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Task 3: Use shared condition parsing and `any_present` in native builds
|
||||
|
||||
**Files:**
|
||||
|
||||
- Modify: `internal/topdata/native.go`
|
||||
- Modify: `internal/topdata/json_contract_test.go`
|
||||
|
||||
**Interfaces:**
|
||||
|
||||
- Consumes `parseGlobalConditionGroups` and
|
||||
`globalConditionGroupsMatch` from Task 1.
|
||||
- Removes the old boolean-mode
|
||||
`globalConditionListMatches(name, raw, rows, required)` path.
|
||||
- Direct `BuildNativeWithOptions` calls fail before output mutation when
|
||||
canonical control syntax is invalid because the existing build entry point
|
||||
runs `ValidateProject` first and native evaluation uses the same parser.
|
||||
|
||||
- [ ] **Step 1: Add failing native-build tests for `any_present` alternatives**
|
||||
|
||||
Use one compact plain-dataset fixture with:
|
||||
- an authored row satisfying the first alternative;
|
||||
- a second subtest satisfying only the later alternative;
|
||||
- a third subtest satisfying neither alternative.
|
||||
|
||||
Assert the injected row exists in the first two outputs and is absent in the
|
||||
third. Inspect the built 2DA content rather than internal slices.
|
||||
|
||||
- [ ] **Step 2: Add failing native-build tests for condition-group
|
||||
conjunction**
|
||||
|
||||
Add subtests proving:
|
||||
- `require_present` + `any_present` injects only when both pass;
|
||||
- `any_present` + `unless_present` injects only when both pass;
|
||||
- existing `require_present` all semantics remain unchanged;
|
||||
- existing `unless_present` none semantics remain unchanged.
|
||||
|
||||
- [ ] **Step 3: Add a failing ordered-injection test**
|
||||
|
||||
Author injection 0 so it adds the row referenced by injection 1's
|
||||
`any_present`. Assert injection 1 applies. This protects evaluation against
|
||||
the same current-row view and ordered mutation used today.
|
||||
|
||||
- [ ] **Step 4: Add direct-build fail-closed tests**
|
||||
|
||||
Call `BuildNativeWithOptions` without a prior explicit `ValidateProject` call
|
||||
for fixtures containing:
|
||||
- `when_present` on an injection;
|
||||
- an unknown condition key;
|
||||
- an empty `any_present`;
|
||||
- an unknown canonical root key.
|
||||
|
||||
Require an error containing the stable object/property fragments and assert
|
||||
the expected output 2DA does not exist.
|
||||
|
||||
- [ ] **Step 5: Run focused tests and confirm failures**
|
||||
|
||||
```bash
|
||||
nix develop --command go test ./internal/topdata \
|
||||
-run 'TestBuildNative(AnyPresent|CombinesGlobalConditionGroups|UsesCurrentRowsForAnyPresent|RejectsUnsupportedGlobalControlSyntax|RejectsUnsupportedCanonicalRoot)' \
|
||||
-v
|
||||
```
|
||||
|
||||
Expected: `any_present` is ignored by current native evaluation and the new
|
||||
parser/evaluator is not wired.
|
||||
|
||||
- [ ] **Step 6: Replace native condition evaluation with the shared contract**
|
||||
|
||||
Replace `globalInjectionConditionsMatch` and
|
||||
`globalConditionListMatches` with:
|
||||
|
||||
```go
|
||||
func globalInjectionConditionsMatch(
|
||||
injection map[string]any,
|
||||
rows []map[string]any,
|
||||
) (bool, error) {
|
||||
groups, err := parseGlobalConditionGroups(injection)
|
||||
if err != nil {
|
||||
return false, err
|
||||
}
|
||||
return globalConditionGroupsMatch(groups, rows), nil
|
||||
}
|
||||
```
|
||||
|
||||
Keep `applyPlainDatasetGlobalInjections` calling this function immediately
|
||||
before deduplication/canonicalization, using `currentRows()` for each
|
||||
injection.
|
||||
|
||||
- [ ] **Step 7: Run focused, package, and repository tests**
|
||||
|
||||
```bash
|
||||
nix develop --command go test ./internal/topdata \
|
||||
-run 'TestBuildNative(AnyPresent|CombinesGlobalConditionGroups|UsesCurrentRowsForAnyPresent|RejectsUnsupportedGlobalControlSyntax|RejectsUnsupportedCanonicalRoot)' \
|
||||
-v
|
||||
nix develop --command go test ./internal/topdata
|
||||
nix develop --command go test ./...
|
||||
```
|
||||
|
||||
Expected: pass.
|
||||
|
||||
- [ ] **Step 8: Commit native `any_present` support**
|
||||
|
||||
```bash
|
||||
git add internal/topdata/native.go internal/topdata/json_contract_test.go
|
||||
git commit -m "feat(topdata): add any-present injection conditions"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Task 4: Document the global injection grammar
|
||||
|
||||
**Files:**
|
||||
|
||||
- Modify: `internal/topdata/CLASS_FEAT_GLOBAL_INJECTS_CONTRACT.md`
|
||||
|
||||
**Interfaces:**
|
||||
|
||||
- Documents the exact injection keys and all/any/none condition semantics.
|
||||
- States that all present groups must pass and that conditions observe current
|
||||
rows in injection order.
|
||||
- States that `any_present` must be a non-empty array and `when_present` is
|
||||
invalid.
|
||||
|
||||
- [ ] **Step 1: Add a documentation contract check**
|
||||
|
||||
Run this before editing:
|
||||
|
||||
```bash
|
||||
rg -n 'any_present|when_present|all.*any.*none|require_present.*unless_present' \
|
||||
internal/topdata/CLASS_FEAT_GLOBAL_INJECTS_CONTRACT.md
|
||||
```
|
||||
|
||||
Expected: no complete description of `any_present` and group conjunction.
|
||||
|
||||
- [ ] **Step 2: Update the manifest shape and semantics**
|
||||
|
||||
Add an example containing all three groups:
|
||||
|
||||
```json
|
||||
{
|
||||
"row": {
|
||||
"FeatIndex": { "id": "feat:example" }
|
||||
},
|
||||
"require_present": [{ "field": "FeatIndex", "id": "feat:required" }],
|
||||
"any_present": [
|
||||
{ "field": "FeatIndex", "id": "masterfeats:metamagic" },
|
||||
{ "field": "FeatIndex", "id": "masterfeats:combat" }
|
||||
],
|
||||
"unless_present": [{ "field": "FeatIndex", "id": "feat:example" }]
|
||||
}
|
||||
```
|
||||
|
||||
Document:
|
||||
- `require_present`: all listed references must exist;
|
||||
- `any_present`: at least one listed reference must exist and the list cannot
|
||||
be empty;
|
||||
- `unless_present`: none of the listed references may exist;
|
||||
- every present group must pass;
|
||||
- earlier injections affect later conditions;
|
||||
- supported injection keys are `row`, `require_present`, `any_present`, and
|
||||
`unless_present`;
|
||||
- `when_present` is rejected rather than aliased.
|
||||
|
||||
- [ ] **Step 3: Verify the documentation contains the complete contract**
|
||||
|
||||
```bash
|
||||
rg -n 'require_present|any_present|unless_present|when_present|earlier injection|every present group' \
|
||||
internal/topdata/CLASS_FEAT_GLOBAL_INJECTS_CONTRACT.md
|
||||
```
|
||||
|
||||
Expected: each semantic point appears in the contract.
|
||||
|
||||
- [ ] **Step 4: Commit documentation**
|
||||
|
||||
```bash
|
||||
git add internal/topdata/CLASS_FEAT_GLOBAL_INJECTS_CONTRACT.md
|
||||
git commit -m "docs(topdata): define global injection conditions"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Task 5: Validate the active topdata checkout and complete repository checks
|
||||
|
||||
**Files:**
|
||||
|
||||
- No authored files should change in `sow-tools`.
|
||||
- Preserve existing sibling change:
|
||||
`../sow-topdata/data/classes/feats/global.json`.
|
||||
- Do not retain: `bin/`, `.cache/`, `generated/`, `result`, HAK/TLK files, or
|
||||
temporary Crucible binaries.
|
||||
|
||||
**Interfaces:**
|
||||
|
||||
- The built Crucible validates the active `sow-topdata` checkout.
|
||||
- The active checkout's single-condition correction uses `require_present`;
|
||||
`any_present` remains available for future multi-alternative authoring.
|
||||
|
||||
- [ ] **Step 1: Format and run the full `sow-tools` checks**
|
||||
|
||||
```bash
|
||||
nix develop --command gofmt -w \
|
||||
internal/topdata/json_contract.go \
|
||||
internal/topdata/json_contract_test.go \
|
||||
internal/topdata/topdata.go \
|
||||
internal/topdata/native.go
|
||||
nix develop --command make check
|
||||
```
|
||||
|
||||
Expected: vet, Go tests, shellcheck, and yamllint pass.
|
||||
|
||||
- [ ] **Step 2: Build a temporary Crucible binary without tracking it**
|
||||
|
||||
```bash
|
||||
nix develop --command go build \
|
||||
-o /tmp/crucible-topdata-json-validation \
|
||||
./cmd/crucible
|
||||
```
|
||||
|
||||
Expected: `/tmp/crucible-topdata-json-validation` exists outside the
|
||||
repository.
|
||||
|
||||
- [ ] **Step 3: Confirm the active consumer syntax and repository state**
|
||||
|
||||
```bash
|
||||
git -C ../sow-topdata status --short --branch
|
||||
rg -n 'when_present|require_present|any_present|unless_present' \
|
||||
../sow-topdata/data/classes/feats/global.json
|
||||
```
|
||||
|
||||
Expected: the existing consumer branch contains `require_present` and no
|
||||
`when_present`. Do not alter unrelated consumer work.
|
||||
|
||||
- [ ] **Step 4: Validate and build active `sow-topdata` with the changed
|
||||
Crucible**
|
||||
|
||||
```bash
|
||||
(
|
||||
cd ../sow-topdata
|
||||
nix develop --command /tmp/crucible-topdata-json-validation topdata validate
|
||||
nix develop --command /tmp/crucible-topdata-json-validation topdata build
|
||||
)
|
||||
```
|
||||
|
||||
Expected: validation and build pass, global injections execute, and generated
|
||||
outputs remain ignored.
|
||||
|
||||
- [ ] **Step 5: Verify no generated files became tracked**
|
||||
|
||||
```bash
|
||||
git status --short
|
||||
git -C ../sow-topdata status --short
|
||||
git ls-files --error-unmatch bin generated .cache result 2>/dev/null
|
||||
git -C ../sow-topdata ls-files --error-unmatch generated .cache result 2>/dev/null
|
||||
```
|
||||
|
||||
Expected: the first two commands show only intentional source/doc changes;
|
||||
the `git ls-files --error-unmatch` commands find no newly tracked generated
|
||||
output. Remove only untracked artifacts created by this plan.
|
||||
|
||||
- [ ] **Step 6: Run final semantic searches and diff checks**
|
||||
|
||||
```bash
|
||||
rg -n 'when_present' internal/topdata --glob '!docs/superpowers/**'
|
||||
rg -n 'any_present' \
|
||||
internal/topdata/json_contract.go \
|
||||
internal/topdata/json_contract_test.go \
|
||||
internal/topdata/CLASS_FEAT_GLOBAL_INJECTS_CONTRACT.md
|
||||
git diff --check
|
||||
git status --short --branch
|
||||
git log --oneline origin/main..HEAD
|
||||
```
|
||||
|
||||
Expected: `when_present` appears only in rejection tests/documentation;
|
||||
implementation, tests, and contract document `any_present`; diff check is
|
||||
clean.
|
||||
|
||||
- [ ] **Step 7: Remove the temporary binary**
|
||||
|
||||
```bash
|
||||
rm -f /tmp/crucible-topdata-json-validation
|
||||
```
|
||||
|
||||
- [ ] **Step 8: Push the feature branch and open the required pull request**
|
||||
|
||||
```bash
|
||||
git push -u origin HEAD
|
||||
tea pr create \
|
||||
--title "feat: tighten topdata JSON validation" \
|
||||
--description $'## Summary\n- reject unsupported canonical topdata JSON properties\n- add non-empty any_present global injection conditions\n- keep validation and direct builds on one shared contract\n\n## Verification\n- nix develop --command make check\n- active sow-topdata validate and build with the changed Crucible'
|
||||
```
|
||||
|
||||
Expected: a PR targeting `main`; no production workflow or release tag is
|
||||
triggered.
|
||||
|
||||
- [ ] **Step 9: Record the handoff**
|
||||
|
||||
Report:
|
||||
- the `sow-tools` branch and PR URL;
|
||||
- the existing `sow-topdata` consumer branch/commit used for integration;
|
||||
- exact focused and full verification commands with outcomes;
|
||||
- confirmation that no generated files were tracked;
|
||||
- any diagnostics wording intentionally asserted as public semantic
|
||||
contract.
|
||||
@@ -0,0 +1,269 @@
|
||||
# Topdata JSON Validation Tightening Design
|
||||
|
||||
**Date:** 2026-06-25
|
||||
**Status:** Implemented and reviewed
|
||||
**Scope:** `sow-tools`
|
||||
|
||||
## Problem
|
||||
|
||||
Topdata JSON validation checks many required properties and value types, but it
|
||||
often ignores unknown structural properties. This allows plausible typos to
|
||||
pass validation and then be ignored by the builder.
|
||||
|
||||
The immediate failure occurred in
|
||||
`topdata/data/classes/feats/global.json`: an injection used
|
||||
`when_present` even though the implemented condition is `require_present`.
|
||||
Validation succeeded, the builder ignored `when_present`, and the injection was
|
||||
therefore applied without the intended condition.
|
||||
|
||||
This is a fail-open authoring contract. A misspelled build-control property must
|
||||
not silently change generated game data.
|
||||
|
||||
## Goals
|
||||
|
||||
- Reject unsupported structural properties in canonical topdata JSON files.
|
||||
- Add an `any_present` global injection condition.
|
||||
- Keep standalone validation and direct native builds consistent.
|
||||
- Preserve dataset-specific row flexibility.
|
||||
- Tighten only established canonical formats; do not turn this work into a
|
||||
repository-wide schema rewrite.
|
||||
- Produce diagnostics that identify the file, object, and unsupported property.
|
||||
|
||||
## Non-goals
|
||||
|
||||
- No schema migration or formatting change to authored topdata.
|
||||
- No generic JSON Schema framework or new validation dependency.
|
||||
- No blanket closure of every specialized JSON dialect.
|
||||
- No change to generated feat family, registry, spellbook, TLK, or other
|
||||
specialized formats unless they use the canonical containers covered here.
|
||||
- No restriction of row values beyond existing column, metadata, inheritance,
|
||||
reference, TLK, and authoring-sugar contracts.
|
||||
- No compatibility alias for `when_present`; it is invalid syntax.
|
||||
|
||||
## Design
|
||||
|
||||
### Validation boundary
|
||||
|
||||
Structural objects use closed property sets. Dataset payload objects remain
|
||||
dataset-aware.
|
||||
|
||||
Closed structural objects include:
|
||||
|
||||
| Object | Supported properties |
|
||||
| --- | --- |
|
||||
| Base or plain rows-file root | `output`, `key`, `columns`, `rows` |
|
||||
| Canonical module root | `output`, `key`, `columns`, `entries`, `overrides`, `rows` |
|
||||
| `global.json` root | `columns`, `entries`, `overrides`, `defaults`, `position`, `injections` |
|
||||
| Global injection | `row`, `require_present`, `any_present`, `unless_present` |
|
||||
| Global condition | `field`, `id` |
|
||||
| Global default rule | `match`, `values` |
|
||||
| Global default match object | `source` |
|
||||
| Global default format object | `format` |
|
||||
|
||||
The allowed root set is selected from the file's existing canonical context;
|
||||
properties from unrelated formats are not accepted merely because another
|
||||
topdata file type supports them.
|
||||
|
||||
When a canonical root contains more than one supported container, each present
|
||||
container must be validated. For example, a module containing both `entries`
|
||||
and `overrides` must not stop validation after recognizing `entries`.
|
||||
|
||||
The following remain payload objects rather than global allowlists:
|
||||
|
||||
- objects inside `rows`;
|
||||
- values inside `entries`;
|
||||
- row-like objects inside `overrides`;
|
||||
- an injection's `row`;
|
||||
- a default rule's `values`.
|
||||
|
||||
Those objects continue to accept declared dataset columns plus documented row
|
||||
control and metadata fields. The builder already rejects unknown row and entry
|
||||
columns when it canonicalizes a dataset. This work must reuse or align with
|
||||
that knowledge where practical, but must not invent a single fixed row schema
|
||||
across all datasets.
|
||||
|
||||
### Moderate strictness
|
||||
|
||||
This change closes only canonical containers whose grammar is already
|
||||
established by the parser and current authored data.
|
||||
|
||||
Before adding a closed root check, implementation must inventory active
|
||||
`sow-topdata` files and existing `sow-tools` fixtures for legitimate property
|
||||
combinations. A property used by an established canonical format must be added
|
||||
to that format's explicit set rather than removed from authored data merely to
|
||||
satisfy the validator.
|
||||
|
||||
Specialized parsers remain responsible for their own object shapes. Extending
|
||||
closed-property checks to those formats is separate work.
|
||||
|
||||
### Global injection conditions
|
||||
|
||||
Global injections support three condition groups:
|
||||
|
||||
- `require_present`: every listed condition must match a current row.
|
||||
- `any_present`: at least one listed condition must match a current row.
|
||||
- `unless_present`: no listed condition may match a current row.
|
||||
|
||||
When more than one group is present, every group must pass. Conditions continue
|
||||
to use the existing shape:
|
||||
|
||||
```json
|
||||
{
|
||||
"field": "FeatIndex",
|
||||
"id": "feat:example"
|
||||
}
|
||||
```
|
||||
|
||||
Example:
|
||||
|
||||
```json
|
||||
{
|
||||
"row": {
|
||||
"FeatIndex": {
|
||||
"id": "feat:example"
|
||||
}
|
||||
},
|
||||
"require_present": [
|
||||
{
|
||||
"field": "FeatIndex",
|
||||
"id": "feat:base_requirement"
|
||||
}
|
||||
],
|
||||
"any_present": [
|
||||
{
|
||||
"field": "FeatIndex",
|
||||
"id": "masterfeats:metamagic"
|
||||
},
|
||||
{
|
||||
"field": "FeatIndex",
|
||||
"id": "masterfeats:combat"
|
||||
}
|
||||
],
|
||||
"unless_present": [
|
||||
{
|
||||
"field": "FeatIndex",
|
||||
"id": "feat:example"
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
This injection applies only when the required feat exists, either listed
|
||||
masterfeat exists, and the injected feat does not already exist.
|
||||
|
||||
`any_present` must contain at least one condition. An empty list is an
|
||||
authoring error because it cannot express a useful successful condition.
|
||||
Existing empty-list behavior for `require_present` and `unless_present` is not
|
||||
changed by this work.
|
||||
|
||||
Conditions are evaluated against the same current-row view and in the same
|
||||
injection order used today. This preserves the existing behavior where an
|
||||
earlier injection can affect a later injection's conditions.
|
||||
|
||||
### Validation and build consistency
|
||||
|
||||
`ValidateProject` must report unsupported structural properties before a build.
|
||||
The native builder must also fail closed when invoked directly with the same
|
||||
invalid control syntax.
|
||||
|
||||
The implementation should centralize:
|
||||
|
||||
- deterministic unsupported-property checks;
|
||||
- global condition parsing and validation;
|
||||
- condition group evaluation.
|
||||
|
||||
Validation may collect multiple diagnostics while building returns the first
|
||||
blocking error. Both paths must accept and reject the same property names and
|
||||
condition shapes.
|
||||
|
||||
### Diagnostics
|
||||
|
||||
Diagnostics must name the containing object and unsupported property. Property
|
||||
lists must be deterministic.
|
||||
|
||||
For the original error, an acceptable diagnostic is:
|
||||
|
||||
```text
|
||||
global injection 10 contains unsupported key "when_present";
|
||||
supported keys are row, require_present, any_present, and unless_present
|
||||
```
|
||||
|
||||
Equivalent concise wording is acceptable if tests assert the stable semantic
|
||||
parts rather than the entire sentence.
|
||||
|
||||
## Compatibility
|
||||
|
||||
Current canonical `sow-topdata` roots fit the property sets above. The
|
||||
implementation must rerun the inventory immediately before tightening checks
|
||||
and run validation against the active `sow-topdata` checkout.
|
||||
|
||||
Unknown properties are not retained as compatibility behavior. They currently
|
||||
have no defined effect, and silently accepting them is the bug being fixed.
|
||||
Existing documented properties and dataset-specific row fields remain valid.
|
||||
|
||||
`require_present` and `unless_present` retain their current semantics.
|
||||
`any_present` is additive.
|
||||
|
||||
## Testing
|
||||
|
||||
Tests must follow red-green development and cover validation and direct-build
|
||||
paths.
|
||||
|
||||
### Unsupported properties
|
||||
|
||||
- Reject an unknown `global.json` root property.
|
||||
- Reject `when_present` and other unknown injection properties.
|
||||
- Reject an unknown condition property.
|
||||
- Reject unknown properties in default rules, match objects, and format
|
||||
objects.
|
||||
- Reject unknown canonical base/plain rows-file root properties.
|
||||
- Reject unknown canonical module root properties.
|
||||
- Continue accepting legitimate combinations such as:
|
||||
- a columns-only module;
|
||||
- entries-only and overrides-only modules;
|
||||
- a module containing both entries and overrides;
|
||||
- current base and plain rows-file roots.
|
||||
- Continue accepting dataset columns and documented row control fields inside
|
||||
rows, entries, overrides, and injection rows.
|
||||
|
||||
### `any_present`
|
||||
|
||||
- Validation accepts a non-empty `any_present` condition list.
|
||||
- Validation rejects a non-array or empty `any_present`.
|
||||
- A build injects when the first alternative is present.
|
||||
- A build injects when a later alternative is present.
|
||||
- A build skips the injection when no alternative is present.
|
||||
- `any_present` combines correctly with `require_present`.
|
||||
- `any_present` combines correctly with `unless_present`.
|
||||
- Existing `require_present` and `unless_present` behavior remains unchanged.
|
||||
- Direct building rejects unsupported condition syntax even when project
|
||||
validation was not called first.
|
||||
|
||||
### Integration
|
||||
|
||||
- Run focused `internal/topdata` tests.
|
||||
- Run `nix develop --command make check`.
|
||||
- Validate the active `sow-topdata` checkout with the changed Crucible.
|
||||
- Build topdata far enough to exercise global injections and confirm no
|
||||
generated files are tracked.
|
||||
|
||||
## Documentation
|
||||
|
||||
Update the class feat global injection contract to document `require_present`,
|
||||
`any_present`, and `unless_present`, including their all/any/none semantics and
|
||||
how multiple groups combine.
|
||||
|
||||
No broad validation guide is added unless implementation reveals an existing
|
||||
operator document that would otherwise become stale.
|
||||
|
||||
## Acceptance criteria
|
||||
|
||||
- `when_present` in a global injection fails validation and direct building.
|
||||
- `any_present` is valid only as a non-empty list of valid conditions.
|
||||
- `any_present` applies an injection when one or more alternatives exist and
|
||||
skips it when none exist.
|
||||
- Unknown canonical structural properties fail with actionable diagnostics.
|
||||
- Dataset-specific row fields remain valid when declared by the dataset.
|
||||
- Existing active `sow-topdata` validates and builds after replacing invalid
|
||||
syntax with the documented grammar.
|
||||
- No specialized topdata format is tightened outside the stated scope.
|
||||
@@ -65,6 +65,33 @@ effective output while keeping project policy in topdata authoring files.
|
||||
`position` defaults to `append`. The module uses `prepend` for class feat
|
||||
injections to keep generated policy rows before per-class authored rows.
|
||||
|
||||
## Global Injection Grammar
|
||||
|
||||
Each injection entry may use only `row`, `require_present`, `any_present`, and
|
||||
`unless_present`. `when_present` is invalid and rejected; it is not an alias.
|
||||
|
||||
```json
|
||||
{
|
||||
"row": {
|
||||
"FeatIndex": { "id": "feat:example" }
|
||||
},
|
||||
"require_present": [{ "field": "FeatIndex", "id": "feat:required" }],
|
||||
"any_present": [
|
||||
{ "field": "FeatIndex", "id": "masterfeats:metamagic" },
|
||||
{ "field": "FeatIndex", "id": "masterfeats:combat" }
|
||||
],
|
||||
"unless_present": [{ "field": "FeatIndex", "id": "feat:example" }]
|
||||
}
|
||||
```
|
||||
|
||||
- `require_present` means every listed reference must exist.
|
||||
- `any_present` means at least one listed reference must exist, and the array
|
||||
must not be empty.
|
||||
- `unless_present` means none of the listed references may exist.
|
||||
- Every present group on an entry must pass.
|
||||
- Conditions observe the current rows in injection order, so earlier injections
|
||||
can affect later conditions.
|
||||
|
||||
## Acceptance Criteria
|
||||
|
||||
- Class feat output contains the same effective shared feats as the prior YAML
|
||||
|
||||
@@ -0,0 +1,195 @@
|
||||
package topdata
|
||||
|
||||
import (
|
||||
"errors"
|
||||
"fmt"
|
||||
"slices"
|
||||
"strings"
|
||||
)
|
||||
|
||||
type globalCondition struct {
|
||||
Field string
|
||||
ID string
|
||||
}
|
||||
|
||||
type globalConditionGroups struct {
|
||||
RequirePresent []globalCondition
|
||||
AnyPresent []globalCondition
|
||||
UnlessPresent []globalCondition
|
||||
}
|
||||
|
||||
var (
|
||||
baseOrPlainRootKeys = []string{
|
||||
"output", "key", "columns", "rows", "compare_reference",
|
||||
}
|
||||
canonicalModuleRootKeys = []string{
|
||||
"output", "key", "columns", "entries", "overrides", "rows",
|
||||
}
|
||||
globalRootKeys = []string{
|
||||
"columns", "entries", "overrides", "defaults", "position", "injections",
|
||||
}
|
||||
globalInjectionKeys = []string{
|
||||
"row", "require_present", "any_present", "unless_present",
|
||||
}
|
||||
globalConditionKeys = []string{"field", "id"}
|
||||
globalDefaultRuleKeys = []string{"match", "values"}
|
||||
globalDefaultMatchKeys = []string{"source"}
|
||||
globalDefaultFormatKeys = []string{"format"}
|
||||
)
|
||||
|
||||
func unsupportedObjectKeys(obj map[string]any, supported ...string) []string {
|
||||
if len(obj) == 0 {
|
||||
return nil
|
||||
}
|
||||
supportedSet := make(map[string]struct{}, len(supported))
|
||||
for _, key := range supported {
|
||||
supportedSet[key] = struct{}{}
|
||||
}
|
||||
unsupported := make([]string, 0, len(obj))
|
||||
for key := range obj {
|
||||
if _, ok := supportedSet[key]; ok {
|
||||
continue
|
||||
}
|
||||
unsupported = append(unsupported, key)
|
||||
}
|
||||
slices.Sort(unsupported)
|
||||
return unsupported
|
||||
}
|
||||
|
||||
func unsupportedObjectKeysError(label string, obj map[string]any, supported ...string) error {
|
||||
unsupported := unsupportedObjectKeys(obj, supported...)
|
||||
if len(unsupported) == 0 {
|
||||
return nil
|
||||
}
|
||||
return fmt.Errorf("%s contains unsupported key %q (supported keys: %s)", label, unsupported[0], strings.Join(supported, ", "))
|
||||
}
|
||||
|
||||
type globalConditionParseOptions struct {
|
||||
Label string
|
||||
Report func(error)
|
||||
}
|
||||
|
||||
func parseGlobalConditionGroups(injection map[string]any, options ...globalConditionParseOptions) (globalConditionGroups, error) {
|
||||
if injection == nil {
|
||||
return globalConditionGroups{}, nil
|
||||
}
|
||||
|
||||
label := "global injection"
|
||||
var report func(error)
|
||||
if len(options) > 0 {
|
||||
if strings.TrimSpace(options[0].Label) != "" {
|
||||
label = options[0].Label
|
||||
}
|
||||
report = options[0].Report
|
||||
}
|
||||
|
||||
var errs []error
|
||||
addError := func(err error) {
|
||||
if err == nil {
|
||||
return
|
||||
}
|
||||
errs = append(errs, err)
|
||||
if report != nil {
|
||||
report(err)
|
||||
}
|
||||
}
|
||||
if err := unsupportedObjectKeysError(label, injection, globalInjectionKeys...); err != nil {
|
||||
addError(err)
|
||||
}
|
||||
|
||||
var groups globalConditionGroups
|
||||
if raw, ok := injection["require_present"]; ok {
|
||||
parsed, err := parseGlobalConditionList(label+" require_present", raw, false)
|
||||
if err != nil {
|
||||
addError(err)
|
||||
} else {
|
||||
groups.RequirePresent = parsed
|
||||
}
|
||||
}
|
||||
if raw, ok := injection["any_present"]; ok {
|
||||
parsed, err := parseGlobalConditionList(label+" any_present", raw, true)
|
||||
if err != nil {
|
||||
addError(err)
|
||||
} else {
|
||||
groups.AnyPresent = parsed
|
||||
}
|
||||
}
|
||||
if raw, ok := injection["unless_present"]; ok {
|
||||
parsed, err := parseGlobalConditionList(label+" unless_present", raw, false)
|
||||
if err != nil {
|
||||
addError(err)
|
||||
} else {
|
||||
groups.UnlessPresent = parsed
|
||||
}
|
||||
}
|
||||
return groups, errors.Join(errs...)
|
||||
}
|
||||
|
||||
func parseGlobalConditionList(name string, raw any, requireNonEmpty bool) ([]globalCondition, error) {
|
||||
if raw == nil {
|
||||
return nil, fmt.Errorf("%s must be an array", name)
|
||||
}
|
||||
|
||||
conditions, ok := raw.([]any)
|
||||
if !ok {
|
||||
return nil, fmt.Errorf("%s must be an array", name)
|
||||
}
|
||||
if requireNonEmpty && len(conditions) == 0 {
|
||||
return nil, fmt.Errorf("%s must contain at least one condition", name)
|
||||
}
|
||||
|
||||
parsed := make([]globalCondition, 0, len(conditions))
|
||||
for index, rawCondition := range conditions {
|
||||
condition, ok := rawCondition.(map[string]any)
|
||||
if !ok {
|
||||
return nil, fmt.Errorf("%s[%d] must be an object", name, index)
|
||||
}
|
||||
if err := unsupportedObjectKeysError(fmt.Sprintf("%s[%d]", name, index), condition, globalConditionKeys...); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
field, _ := condition["field"].(string)
|
||||
field = strings.TrimSpace(field)
|
||||
if field == "" {
|
||||
return nil, fmt.Errorf("%s[%d].field is required", name, index)
|
||||
}
|
||||
|
||||
id, _ := condition["id"].(string)
|
||||
id = strings.TrimSpace(id)
|
||||
if id == "" {
|
||||
return nil, fmt.Errorf("%s[%d].id is required", name, index)
|
||||
}
|
||||
|
||||
parsed = append(parsed, globalCondition{Field: field, ID: id})
|
||||
}
|
||||
return parsed, nil
|
||||
}
|
||||
|
||||
func globalConditionGroupsMatch(groups globalConditionGroups, rows []map[string]any) bool {
|
||||
for _, condition := range groups.RequirePresent {
|
||||
if !globalReferencePresent(rows, condition.Field, condition.ID) {
|
||||
return false
|
||||
}
|
||||
}
|
||||
if groups.AnyPresent != nil {
|
||||
if len(groups.AnyPresent) == 0 {
|
||||
return false
|
||||
}
|
||||
matched := false
|
||||
for _, condition := range groups.AnyPresent {
|
||||
if globalReferencePresent(rows, condition.Field, condition.ID) {
|
||||
matched = true
|
||||
break
|
||||
}
|
||||
}
|
||||
if !matched {
|
||||
return false
|
||||
}
|
||||
}
|
||||
for _, condition := range groups.UnlessPresent {
|
||||
if globalReferencePresent(rows, condition.Field, condition.ID) {
|
||||
return false
|
||||
}
|
||||
}
|
||||
return true
|
||||
}
|
||||
File diff suppressed because it is too large
Load Diff
@@ -3886,48 +3886,11 @@ func globalManifestPosition(module nativeGeneratedModule) (string, error) {
|
||||
}
|
||||
|
||||
func globalInjectionConditionsMatch(injection map[string]any, rows []map[string]any) (bool, error) {
|
||||
if rawRequired, ok := injection["require_present"]; ok {
|
||||
matches, err := globalConditionListMatches("require_present", rawRequired, rows, true)
|
||||
if err != nil || !matches {
|
||||
return matches, err
|
||||
groups, err := parseGlobalConditionGroups(injection)
|
||||
if err != nil {
|
||||
return false, err
|
||||
}
|
||||
}
|
||||
if rawBlocked, ok := injection["unless_present"]; ok {
|
||||
matches, err := globalConditionListMatches("unless_present", rawBlocked, rows, false)
|
||||
if err != nil || !matches {
|
||||
return matches, err
|
||||
}
|
||||
}
|
||||
return true, nil
|
||||
}
|
||||
|
||||
func globalConditionListMatches(name string, raw any, rows []map[string]any, required bool) (bool, error) {
|
||||
conditions, ok := raw.([]any)
|
||||
if !ok {
|
||||
return false, fmt.Errorf("%s must be an array", name)
|
||||
}
|
||||
for index, rawCondition := range conditions {
|
||||
condition, ok := rawCondition.(map[string]any)
|
||||
if !ok {
|
||||
return false, fmt.Errorf("%s[%d] must be an object", name, index)
|
||||
}
|
||||
field, _ := condition["field"].(string)
|
||||
id, _ := condition["id"].(string)
|
||||
if strings.TrimSpace(field) == "" {
|
||||
return false, fmt.Errorf("%s[%d].field is required", name, index)
|
||||
}
|
||||
if strings.TrimSpace(id) == "" {
|
||||
return false, fmt.Errorf("%s[%d].id is required", name, index)
|
||||
}
|
||||
found := globalReferencePresent(rows, field, id)
|
||||
if required && !found {
|
||||
return false, nil
|
||||
}
|
||||
if !required && found {
|
||||
return false, nil
|
||||
}
|
||||
}
|
||||
return true, nil
|
||||
return globalConditionGroupsMatch(groups, rows), nil
|
||||
}
|
||||
|
||||
func globalRowIdentityPresent(row map[string]any, rows []map[string]any) bool {
|
||||
|
||||
+59
-73
@@ -333,6 +333,17 @@ func validateDuplicateJSONKeys(path string, raw []byte, report *ValidationReport
|
||||
}
|
||||
}
|
||||
|
||||
func reportTopdataContractError(path string, err error, report *ValidationReport) {
|
||||
if err == nil {
|
||||
return
|
||||
}
|
||||
report.Diagnostics = append(report.Diagnostics, Diagnostic{
|
||||
Severity: SeverityError,
|
||||
Path: path,
|
||||
Message: err.Error(),
|
||||
})
|
||||
}
|
||||
|
||||
func scanJSONValueForDuplicateKeys(decoder *json.Decoder, path string, report *ValidationReport) error {
|
||||
token, err := decoder.Token()
|
||||
if err != nil {
|
||||
@@ -425,35 +436,40 @@ func validateDataObject(path string, obj map[string]any, report *ValidationRepor
|
||||
return
|
||||
}
|
||||
if base == "base.json" {
|
||||
reportTopdataContractError(path, unsupportedObjectKeysError("base.json root", obj, baseOrPlainRootKeys...), report)
|
||||
validateRowsFile(path, obj, report, true)
|
||||
return
|
||||
}
|
||||
if inModules {
|
||||
if _, hasEntries := obj["entries"]; hasEntries {
|
||||
validateEntriesFile(path, obj, report)
|
||||
return
|
||||
}
|
||||
if _, hasOverrides := obj["overrides"]; hasOverrides {
|
||||
validateOverridesFile(path, obj, report)
|
||||
return
|
||||
}
|
||||
if _, hasRows := obj["rows"]; hasRows {
|
||||
validateRowsFile(path, obj, report, false)
|
||||
return
|
||||
}
|
||||
if _, hasColumns := obj["columns"]; hasColumns {
|
||||
reportTopdataContractError(path, unsupportedObjectKeysError("module root", obj, canonicalModuleRootKeys...), report)
|
||||
_, hasColumns := obj["columns"]
|
||||
_, hasEntries := obj["entries"]
|
||||
_, hasOverrides := obj["overrides"]
|
||||
_, hasRows := obj["rows"]
|
||||
if hasColumns {
|
||||
validateColumnsFile(path, obj, report)
|
||||
return
|
||||
}
|
||||
if hasEntries {
|
||||
validateEntriesFileWithColumns(path, obj, report, false)
|
||||
}
|
||||
if hasOverrides {
|
||||
validateOverridesFileWithColumns(path, obj, report, false)
|
||||
}
|
||||
if hasRows {
|
||||
validateRowsFileWithColumns(path, obj, report, false, false)
|
||||
}
|
||||
if !hasColumns && !hasEntries && !hasOverrides && !hasRows {
|
||||
report.Diagnostics = append(report.Diagnostics, Diagnostic{
|
||||
Severity: SeverityWarning,
|
||||
Path: path,
|
||||
Message: "module file does not use a recognized canonical shape yet; expected one of columns, entries, overrides, or rows",
|
||||
})
|
||||
}
|
||||
return
|
||||
}
|
||||
|
||||
if _, hasRows := obj["rows"]; hasRows {
|
||||
reportTopdataContractError(path, unsupportedObjectKeysError("plain rows root", obj, baseOrPlainRootKeys...), report)
|
||||
validateRowsFile(path, obj, report, false)
|
||||
return
|
||||
}
|
||||
@@ -516,6 +532,7 @@ func datasetInvariantForPath(dataPath topdataDataPath) (datasetValidationInvaria
|
||||
}
|
||||
|
||||
func validateDatasetInvariantBaseFile(path string, obj map[string]any, invariant datasetValidationInvariant, report *ValidationReport) {
|
||||
reportTopdataContractError(path, unsupportedObjectKeysError("base.json root", obj, baseOrPlainRootKeys...), report)
|
||||
validateRowsFile(path, obj, report, true)
|
||||
validateDerivedBaseOutput(path, obj, invariant.Name, report)
|
||||
validateRequiredColumns(path, obj, invariant, report)
|
||||
@@ -826,6 +843,10 @@ func validateLockObject(path string, obj map[string]any, report *ValidationRepor
|
||||
}
|
||||
|
||||
func validateRowsFile(path string, obj map[string]any, report *ValidationReport, requireColumns bool) {
|
||||
validateRowsFileWithColumns(path, obj, report, requireColumns, true)
|
||||
}
|
||||
|
||||
func validateRowsFileWithColumns(path string, obj map[string]any, report *ValidationReport, requireColumns bool, validateColumns bool) {
|
||||
rows, ok := obj["rows"]
|
||||
if !ok {
|
||||
report.Diagnostics = append(report.Diagnostics, Diagnostic{
|
||||
@@ -845,7 +866,7 @@ func validateRowsFile(path string, obj map[string]any, report *ValidationReport,
|
||||
if rowsList, ok := rows.([]any); ok {
|
||||
validateRowCollection(path, obj, rowsList, report)
|
||||
}
|
||||
if _, ok := obj["columns"]; ok {
|
||||
if _, ok := obj["columns"]; ok && validateColumns {
|
||||
validateColumnsFile(path, obj, report)
|
||||
} else if requireColumns {
|
||||
report.Diagnostics = append(report.Diagnostics, Diagnostic{
|
||||
@@ -896,6 +917,10 @@ func validateColumnsFile(path string, obj map[string]any, report *ValidationRepo
|
||||
}
|
||||
|
||||
func validateEntriesFile(path string, obj map[string]any, report *ValidationReport) {
|
||||
validateEntriesFileWithColumns(path, obj, report, true)
|
||||
}
|
||||
|
||||
func validateEntriesFileWithColumns(path string, obj map[string]any, report *ValidationReport, validateColumns bool) {
|
||||
entries, ok := obj["entries"]
|
||||
if !ok {
|
||||
report.Diagnostics = append(report.Diagnostics, Diagnostic{
|
||||
@@ -931,12 +956,16 @@ func validateEntriesFile(path string, obj map[string]any, report *ValidationRepo
|
||||
validateInlineTextUsage(path, key, entry, report)
|
||||
}
|
||||
}
|
||||
if _, ok := obj["columns"]; ok {
|
||||
if _, ok := obj["columns"]; ok && validateColumns {
|
||||
validateColumnsFile(path, obj, report)
|
||||
}
|
||||
}
|
||||
|
||||
func validateOverridesFile(path string, obj map[string]any, report *ValidationReport) {
|
||||
validateOverridesFileWithColumns(path, obj, report, true)
|
||||
}
|
||||
|
||||
func validateOverridesFileWithColumns(path string, obj map[string]any, report *ValidationReport, validateColumns bool) {
|
||||
overrides, ok := obj["overrides"]
|
||||
if !ok {
|
||||
report.Diagnostics = append(report.Diagnostics, Diagnostic{
|
||||
@@ -957,20 +986,21 @@ func validateOverridesFile(path string, obj map[string]any, report *ValidationRe
|
||||
container := map[string]any{"rows": overrideList}
|
||||
validateRowCollection(path, container, overrideList, report)
|
||||
}
|
||||
if _, ok := obj["columns"]; ok {
|
||||
if _, ok := obj["columns"]; ok && validateColumns {
|
||||
validateColumnsFile(path, obj, report)
|
||||
}
|
||||
}
|
||||
|
||||
func validateGlobalJSONFile(path string, obj map[string]any, report *ValidationReport) {
|
||||
reportTopdataContractError(path, unsupportedObjectKeysError("global.json root", obj, globalRootKeys...), report)
|
||||
if _, ok := obj["columns"]; ok {
|
||||
validateColumnsFile(path, obj, report)
|
||||
}
|
||||
if _, ok := obj["entries"]; ok {
|
||||
validateEntriesFile(path, obj, report)
|
||||
validateEntriesFileWithColumns(path, obj, report, false)
|
||||
}
|
||||
if _, ok := obj["overrides"]; ok {
|
||||
validateOverridesFile(path, obj, report)
|
||||
validateOverridesFileWithColumns(path, obj, report, false)
|
||||
}
|
||||
if _, ok := obj["defaults"]; ok {
|
||||
validateGlobalDefaults(path, obj, report)
|
||||
@@ -1029,8 +1059,12 @@ func validateGlobalJSONFile(path string, obj map[string]any, report *ValidationR
|
||||
} else {
|
||||
rows = append(rows, row)
|
||||
}
|
||||
validateGlobalConditionList(path, fmt.Sprintf("global injection %d require_present", index), injection["require_present"], report)
|
||||
validateGlobalConditionList(path, fmt.Sprintf("global injection %d unless_present", index), injection["unless_present"], report)
|
||||
_, _ = parseGlobalConditionGroups(injection, globalConditionParseOptions{
|
||||
Label: fmt.Sprintf("global injection %d", index),
|
||||
Report: func(err error) {
|
||||
reportTopdataContractError(path, err, report)
|
||||
},
|
||||
})
|
||||
}
|
||||
validateRowCollection(path, obj, rows, report)
|
||||
}
|
||||
@@ -1099,6 +1133,7 @@ func validateGlobalDefaults(path string, obj map[string]any, report *ValidationR
|
||||
})
|
||||
continue
|
||||
}
|
||||
reportTopdataContractError(path, unsupportedObjectKeysError(fmt.Sprintf("global default %d", index), defaultRule, globalDefaultRuleKeys...), report)
|
||||
validateGlobalDefaultMatch(path, index, defaultRule["match"], report)
|
||||
rawValues, ok := defaultRule["values"]
|
||||
if !ok {
|
||||
@@ -1155,15 +1190,13 @@ func validateGlobalDefaultMatch(path string, index int, rawMatch any, report *Va
|
||||
})
|
||||
return
|
||||
}
|
||||
for key := range match {
|
||||
if key != "source" {
|
||||
for _, key := range unsupportedObjectKeys(match, globalDefaultMatchKeys...) {
|
||||
report.Diagnostics = append(report.Diagnostics, Diagnostic{
|
||||
Severity: SeverityError,
|
||||
Path: path,
|
||||
Message: fmt.Sprintf("global default %d match.%s is not supported", index, key),
|
||||
Message: fmt.Sprintf("global default %d match.%s is not supported (unsupported key; supported keys: %s)", index, key, strings.Join(globalDefaultMatchKeys, ", ")),
|
||||
})
|
||||
}
|
||||
}
|
||||
source, ok := match["source"].(string)
|
||||
if !ok || strings.TrimSpace(source) == "" {
|
||||
report.Diagnostics = append(report.Diagnostics, Diagnostic{
|
||||
@@ -1193,14 +1226,7 @@ func validateGlobalDefaultValue(path string, index int, field string, value any,
|
||||
if !hasFormat {
|
||||
return
|
||||
}
|
||||
if len(obj) != 1 {
|
||||
report.Diagnostics = append(report.Diagnostics, Diagnostic{
|
||||
Severity: SeverityError,
|
||||
Path: path,
|
||||
Message: fmt.Sprintf("global default %d values.%s format object must contain only format", index, field),
|
||||
})
|
||||
return
|
||||
}
|
||||
reportTopdataContractError(path, unsupportedObjectKeysError(fmt.Sprintf("global default %d values.%s", index, field), obj, globalDefaultFormatKeys...), report)
|
||||
format, ok := rawFormat.(string)
|
||||
if !ok {
|
||||
report.Diagnostics = append(report.Diagnostics, Diagnostic{
|
||||
@@ -1220,46 +1246,6 @@ func validateGlobalDefaultValue(path string, index int, field string, value any,
|
||||
}
|
||||
}
|
||||
|
||||
func validateGlobalConditionList(path, label string, raw any, report *ValidationReport) {
|
||||
if raw == nil {
|
||||
return
|
||||
}
|
||||
conditions, ok := raw.([]any)
|
||||
if !ok {
|
||||
report.Diagnostics = append(report.Diagnostics, Diagnostic{
|
||||
Severity: SeverityError,
|
||||
Path: path,
|
||||
Message: fmt.Sprintf("%s must be a JSON array", label),
|
||||
})
|
||||
return
|
||||
}
|
||||
for index, rawCondition := range conditions {
|
||||
condition, ok := rawCondition.(map[string]any)
|
||||
if !ok {
|
||||
report.Diagnostics = append(report.Diagnostics, Diagnostic{
|
||||
Severity: SeverityError,
|
||||
Path: path,
|
||||
Message: fmt.Sprintf("%s[%d] must be an object", label, index),
|
||||
})
|
||||
continue
|
||||
}
|
||||
if field, ok := condition["field"].(string); !ok || strings.TrimSpace(field) == "" {
|
||||
report.Diagnostics = append(report.Diagnostics, Diagnostic{
|
||||
Severity: SeverityError,
|
||||
Path: path,
|
||||
Message: fmt.Sprintf("%s[%d].field is required", label, index),
|
||||
})
|
||||
}
|
||||
if id, ok := condition["id"].(string); !ok || strings.TrimSpace(id) == "" {
|
||||
report.Diagnostics = append(report.Diagnostics, Diagnostic{
|
||||
Severity: SeverityError,
|
||||
Path: path,
|
||||
Message: fmt.Sprintf("%s[%d].id is required", label, index),
|
||||
})
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func validateStateObject(path string, obj map[string]any, report *ValidationReport) {
|
||||
entries, ok := obj["entries"].(map[string]any)
|
||||
if !ok {
|
||||
|
||||
Reference in New Issue
Block a user