Compare commits

..
Author SHA1 Message Date
archvillainette 2d184bf686 docs: plan topdata JSON validation tightening
build-binaries / build-binaries (pull_request) Successful in 2m12s
test-image / build-image (pull_request) Successful in 37s
test / test (pull_request) Successful in 1m23s
2026-06-25 21:54:08 +02:00
archvillainette ba76278983 docs: specify topdata JSON validation tightening 2026-06-25 21:34:42 +02:00
19 changed files with 145 additions and 2447 deletions
-5
View File
@@ -9,11 +9,6 @@ on:
push: push:
branches: [main] branches: [main]
tags: ['v*'] tags: ['v*']
paths-ignore:
- "docs/**"
- "README.md"
- "AGENTS.md"
- "LICENSE"
jobs: jobs:
build-binaries: build-binaries:
-5
View File
@@ -4,11 +4,6 @@ name: test
on: on:
push: push:
branches: [main] branches: [main]
paths-ignore:
- "docs/**"
- "README.md"
- "AGENTS.md"
- "LICENSE"
pull_request: pull_request:
jobs: jobs:
@@ -1,7 +1,7 @@
# Topdata JSON Validation Tightening Design # Topdata JSON Validation Tightening Design
**Date:** 2026-06-25 **Date:** 2026-06-25
**Status:** Implemented and reviewed **Status:** Approved design; implementation pending
**Scope:** `sow-tools` **Scope:** `sow-tools`
## Problem ## Problem
@@ -1,285 +0,0 @@
# Crucible Depot Core Design (Increment 1)
## Goal
Give Crucible real ownership of the content-addressed asset depot. Today
`crucible-depot` is a registered-but-unwired stub (fails closed, exit `70`), and
the actual depot logic lives as ~2000 lines of bash embedded in
`sow-assets-manifest/scripts` — a direct violation of the Crucible principle that
*artifact repos invoke Crucible through wrappers and never embed a toolkit*.
Increment 1 wires the depot **core** — blob backends plus `status` / `push` /
`verify` / `get` — into `crucible depot`, and cuts the manifest repo over to call
it directly. The bash depot layer is **retired, not wrapped**: one tool owns
depot byte-motion for both local workflows and CI/CD, with complete parity.
## Why now (root cause that triggered this)
Release run 222 (`v0.1.5`) failed in 45s at the "Warm depot cache" step. PR #27's
mdl-name normalization recompiled the corpus, producing **5903 net-new sha256s**
in `assets/*.yml`. Sampling proved **15/15 net-new blobs return HTTP 404 on the
CDN** while **10/10 old blobs return 200** — the CDN works; the new content was
never uploaded. The blobs exist in the local depot (`workspace/depot`, 69,473
blobs) but never reached Bunny. `mirror.sh` then 404s and dies — the release is
correctly failing closed on a depot missing data.
Root cause of the *missing upload*: `import.sh`'s stat-cache
(`.cache/import/<cat>.tsv`) records `(path,size,mtime,sha)` after **any** import
and excludes "unchanged" files from the only upload call (`depot_put_many`) —
**without tracking which backend those bytes actually landed in**. A prior
`local` import poisons the cache so a later `bunny` import silently skips the
upload. Combined with `depot_require_write` only normalizing `cdn`/unset→`bunny`
(never `local`→anything), it is very easy to leave the manifests referencing
blobs that exist only in the local depot.
The design principle that kills this bug class: **presence is always checked
against the real target backend, never a cache.**
### Update 2026-07-04 — interim bash fix landed + corrected drift measurement
The bug above was fixed in bash ahead of this rewrite (this spec's Increment 2
retires the stat-cache concept entirely, making the fix structural):
[`sow-assets-manifest#29`](https://git.westgate.pw/ShadowsOverWestgate/sow-assets-manifest/pulls/29)
keys the stat-cache dir by write backend + target
(`.cache/import/<target>/`, `scripts/import.sh`), so a `local` import
(`make haks`) can no longer mark a blob "done" in a cache the next `bunny` sync
trusts. A cross-backend regression test guards it. When Increment 1/2 delete the
bash depot layer, this becomes moot — but until then, **do not "simplify" the
stat-cache back to a single shared dir**; that reintroduces the exact orphaning
that broke run 222.
**Corrected drift numbers.** The "15/15 sampled net-new blobs 404" reading in the
section above is a *sampling artifact*, not the real drift. The CDN edge
(`cdn-a7f3k9.westgate.pw`) is IPv6-reachable but resets HTTP/2 from some hosts,
and it throttles concurrent HEADs with `428`/`000` — which produce **both**
false-404s and false-200s. A reliable sweep (force `curl -4`, 1-byte range GET
`-r 0-0` → expect `206`, low concurrency, then serial re-confirm any non-2xx)
shows the true missing set was **exactly 2 net-new blobs**, not ~5903:
- `item/_shared_aenea_weaponvfx/wblpl_fxshblk.mdl` (`0b1d3f…`)
- `item/_shared_aenea_weaponvfx/wxdbsc_fxshpur.mdl` (`4e8d4f…`)
A 600-blob random sample of the pre-existing corpus was 100% present. `mirror.sh`
still correctly failed the release — it dies on the first missing blob — but the
gap was two files a local `make haks` had primed in the shared cache, not a
wholesale upload failure. Both blobs were re-uploaded to Bunny from
`workspace/depot` (present in the local depot, sha-verified), unblocking the
`v0.1.5` re-run.
Takeaway for `crucible depot status`/`verify`: probe over IPv4, treat a single
concurrent-probe non-2xx as *unconfirmed* (re-check serially) before reporting a
blob missing, so the tool doesn't over-report drift the way an ad-hoc HEAD sweep
did here.
### Field notes 2026-07-04 (content-side session) — issues the migration must fix
Surfaced while verifying a content edit (`over/ravenloft_potm_blood`, 18 blobs) had
actually reached the CDN. All corroborate the design above; #1 is a new gap to close in
Increment 1.
1. **`push` must be non-interactive — no tty prompt (new gap).** Today the only write
path, `depot_require_write` (`lib.sh`), *prompts* for `BUNNY_STORAGE_PASSWORD` on a tty
when it is unset. Headless agents and CI-without-a-pre-exported-secret cannot answer
that prompt, so there is no way to push except with a human at a terminal — which is
exactly why the current operational workaround is "an agent/human manually pushes each
fixed asset." `crucible depot push` must take the write key **from env only**, fail
closed with a clear message when it is absent, and **never prompt**. The Backends
section lists the env names but does not state the no-prompt requirement — make it
explicit and add a test that a missing write key exits non-zero without reading stdin.
2. **`status --target cdn` is the everyday author check, not only the release gate.** The
spec frames `status` as the release pre-flight (correct), but the same command,
credential-free against `cdn`, is the "did my edit actually reach the CDN?" check
content authors need routinely. This session answered that question by hand-probing 18
sha256s with `curl` — precisely the toil `crucible depot status --manifests assets
--target cdn` should replace. Document it as the canonical author-facing "is my content
live?" command so people stop ad-hoc probing.
3. **Probe reliability confirmed again — bake it into the tests.** Reproduced the HEAD
hazard directly: a `curl -sI` HEAD sweep of the 18 blobs returned all-`200`; the
prescribed reliable method (`curl -4`, 1-byte range GET `-r 0-0``206`, serial) also
returned all-`206` with the fake-sha control at `404`. HEAD did not false-negative this
time, but it stays the wrong tool. Make the IPv4 range-GET probe a hard requirement and
add a test that fails if a HEAD-based existence check is introduced.
4. **Drift is recurring, not the single v0.1.5 incident.** Operationally, local→CDN sync is
treated as "basically broken": every `make haks` / local import can re-orphan blobs from
Bunny, so authors push by hand after edits. That raises Increment 1's priority and argues
for making `crucible depot push` a standard post-edit step (or an on-edit CI hook), not
only a release-time gate.
## Ownership principle (from the maintainer)
- Crucible **owns** the depot tools. No wrapper scripts preserved for their own
sake.
- Every consumer repo already has `crucible` in its flake; that is the only
dependency. Consumers call `crucible depot …` directly (Makefile targets and
CI workflow steps alike).
- **Complete parity**: the same `crucible depot` commands serve local workflows
and CI/CD. No divergent code paths, no "CI-only" or "local-only" logic.
- Total clean-out over legacy preservation: bash depot code is **deleted** as its
Go replacement lands, not kept as a fallback.
## Scope
This spec covers **Increment 1** only. Roadmap for context:
- **Increment 1 (this spec):** `crucible depot {status, push, verify, get}` +
blob backends (`local` / `cdn` / `bunny`). Cut over `mirror`/verify/release
gate. Delete the corresponding `lib.sh` depot layer.
- **Increment 2:** `crucible depot import` / `sync` (edit-tree → manifest+depot),
replacing `import.sh` / `sync-assets.sh`. Retire the stat-cache concept.
- **Increment 3:** mdl integrity checks, `mirror`/`pull` ergonomics, `prune`,
`gc`, `export`. Delete remaining bash.
Each increment ships independently and deletes the bash it replaces.
## Command surface (Increment 1)
```
crucible depot status [--manifests DIR] [--source LOCAL_DEPOT] --target bunny|cdn|local
crucible depot push [--manifests DIR] --source LOCAL_DEPOT --target bunny
crucible depot verify [--manifests DIR] --target bunny|cdn [--sample N]
crucible depot get <sha> <dest> --target cdn|bunny|local
```
- `--manifests` defaults to `assets/` (the repo's manifest dir). Confirmed
decision: `crucible depot` parses `assets/*.yml` directly (same coupling model
as `crucible-hak` reading hak manifests); the caller does not pre-extract sha
lists.
- `--source` is the local content-addressed store used as the byte source for
`push` (e.g. `workspace/depot`).
- Config/auth is env-first with flags as local aliases (see Backends).
## Drift model (the fix)
The manifest is the source of truth for **what** must exist; the target backend
is the source of truth for **what does** exist; `push` reconciles them. There is
no cache of "what I uploaded."
- **`status`** — collect every `sha256` referenced by `--manifests/*.yml`, run
one parallel existence sweep against `--target`, report
`{referenced, present, missing}`. Non-zero exit if any referenced blob is
missing. This is the release pre-flight gate.
- **`push`** — run `status`, then for each missing sha read the bytes from
`--source` and upload to `--target`. Stateless and idempotent; re-running does
nothing once clean. **Cannot silently skip an upload** — "should I upload this?"
is answered by probing the target, not by trusting a local record.
- **`verify`** — existence sweep plus a bounded random sample downloaded and
re-hashed (the current `--full` tag-build check), decoupled from packing.
- **`get`** — single-blob fetch with sha re-verify (backfill / debugging).
## Backends & config
Faithful port of `lib.sh` semantics — this is a re-home, not a redesign of the
wire protocol.
| Backend | Read | Write | Role |
|---|---|---|---|
| `local` | filesystem `sha256/ab/cd/<sha>` | filesystem | dev/CI; byte source for `push` |
| `cdn` | CDN GET + sha re-verify | ✗ read-only, fails closed | credential-free public reads |
| `bunny` | CDN GET, storage fallback | Storage `PUT` | the only writable remote |
Layout: `blob_key(sha) = sha256/<sha[0:2]>/<sha[2:4]>/<sha>`.
Config (env names unchanged so existing CI secrets and `release.yml` keep working
untouched; flags are local aliases):
- `DEPOT_CDN_BASE` / `BUNNY_CDN_BASE` — CDN read base.
- `BUNNY_STORAGE_HOST`, `BUNNY_STORAGE_ZONE` — Bunny storage endpoint.
- **Read/write key split (preserved):** `BUNNY_STORAGE_READ_PASSWORD` (falls back
to `BUNNY_STORAGE_PASSWORD`) for probes/reads; `BUNNY_STORAGE_PASSWORD` for
`PUT`.
- Concurrency: `DEPOT_PROBE_JOBS` (read probes), `DEPOT_JOBS` (uploads);
`DEPOT_CONNECT_TIMEOUT`, `DEPOT_PROBE_MAX_TIME` per-transfer bounds.
Invariants carried over exactly:
- **Existence probe** is a 1-byte range request (no body download).
- **Upload** is `PUT` with `Checksum: <UPPER-sha>` so Bunny rejects corrupt
writes server-side; each sha uploaded at most once per run.
- **Fail-safe:** a blob with no confirmed-present reply is treated as absent and
(re)uploaded — never silently skipped. This is the invariant the stat-cache
violated; here it is structural, not cached.
- **`get`** re-hashes every downloaded blob and deletes on mismatch.
- `cdn` write operations fail closed (never fake a write).
Stale-wording note to reconcile in code/docs: the dispatch registry summarizes
depot as *"(SeaweedFS)"*, but the real depot is Bunny-CDN + local
(`sow-assets-manifest/AGENTS.md`: "No S3, no SeaweedFS"). The Go implementation
and the registry summary adopt the Bunny/local/cdn reality.
## Data flow
```
assets/*.yml ──parse──▶ referenced sha set ─┐
├─▶ target.HasMany() ─▶ missing set
--target (bunny) ───┘ │
--source (workspace/depot) ──read bytes──▶ target.PutMany()
```
## Cutover / clean-out
Increment 1 deletes the bash it replaces (no wrappers left behind):
- `scripts/mirror.sh``crucible depot get`/pull path; script removed.
- `build-haks.sh`'s `--full` depot-verify block → `crucible depot verify`.
- `release.yml` step "Warm depot cache" → `crucible depot status` (gate) then the
fetch path; the inline bash ticker/compgen logic goes away.
- `lib.sh` depot layer (`depot_require`, `depot_require_write`, `_depot_*_has`,
`_depot_*_get`, `_depot_*_put*`, `_depot_*_probe*`, `blob_key`, backend
dispatch — ~250+ lines) is removed once no remaining bash references it.
Anything still needed by Increments 23 is deleted when those land, not kept as
a compatibility shim.
Consumers call `crucible depot …` directly. Nothing new is wrapped.
## Exit-code contract
Matches Crucible's fail-closed convention:
- `0` — clean / success.
- distinct non-zero (proposed `1`) — **drift found** (`status`/`push` saw
referenced-but-absent blobs), so `release.yml` can gate on `status` before
packing.
- `70` — internal/backend error; never a faked result.
## Testing
- **Go unit tests** (`internal/depot`): `local` backend round-trip against a temp
dir; manifest parsing; drift computation (referenced present = missing) as
table tests; backend selection + config resolution (incl. read/write key
fallback).
- **httptest fake-Bunny**: assert the range-probe-then-PUT sequence, the
`Checksum: <UPPER-sha>` header, and the read/write key split — no real Bunny.
- **Integration (local→local)**: `status` shows drift → `push` clears it →
`status` clean → `push` again is a no-op (idempotency).
## Non-goals (Increment 1)
- `import` / `sync` / edit-tree ingestion (Increment 2).
- mdl integrity checks, `prune`, `gc`, `export`, `mirror` ergonomics
(Increment 3).
- Any change to the depot wire protocol, blob layout, or Bunny account config.
- A TUI.
## Immediate operational payoff
`crucible depot push --source workspace/depot --target bunny` is exactly the
command that unblocks the stuck `v0.1.5` release: it probes the target and
uploads whatever is actually 404 while it re-hydrates from the local depot.
(The `5903` figure below is superseded — see the 2026-07-04 update above: the
real drift was **2** blobs, already re-uploaded; `push` would have been the
clean way to do it and is idempotent once the depot is whole.) Building
Increment 1 first both fixes the tooling and clears the current outage.
## Risks / open questions
- `BUNNY_STORAGE_HOST` default: confirm the canonical storage host to bake as the
fallback (bash referenced it without a visible default in the reviewed range).
- Manifest schema coupling: `crucible depot` now depends on the `assets/*.yml`
shape (`assets[].{path,sha256,size,restype,hak}`). Acceptable and intentional,
but a schema change now touches Crucible.
- Cutover ordering: `release.yml` and Makefile must switch to `crucible depot` in
the same change that removes the bash, to avoid a window where both exist.
-35
View File
@@ -501,9 +501,6 @@ func writeManagedFile(path string, data []byte) (writeState, error) {
if bytes.Equal(existing, data) { if bytes.Equal(existing, data) {
return writeSkipped, nil return writeSkipped, nil
} }
if areaGFFJSONEqualIgnoringRootVersion(path, existing, data) {
return writeSkipped, nil
}
if err := os.WriteFile(path, data, 0o644); err != nil { if err := os.WriteFile(path, data, 0o644); err != nil {
return writeSkipped, fmt.Errorf("overwrite %s: %w", path, err) return writeSkipped, fmt.Errorf("overwrite %s: %w", path, err)
} }
@@ -519,38 +516,6 @@ func writeManagedFile(path string, data []byte) (writeState, error) {
return writeNew, nil return writeNew, nil
} }
func areaGFFJSONEqualIgnoringRootVersion(path string, existing, extracted []byte) bool {
if !strings.HasSuffix(strings.ToLower(filepath.Base(path)), ".are.json") {
return false
}
var left, right gff.Document
if err := json.Unmarshal(existing, &left); err != nil {
return false
}
if err := json.Unmarshal(extracted, &right); err != nil {
return false
}
removeRootField(&left.Root, "Version")
removeRootField(&right.Root, "Version")
leftRaw, err := json.Marshal(left)
if err != nil {
return false
}
rightRaw, err := json.Marshal(right)
if err != nil {
return false
}
return bytes.Equal(leftRaw, rightRaw)
}
func removeRootField(s *gff.Struct, label string) {
s.Fields = slices.DeleteFunc(s.Fields, func(field gff.Field) bool {
return field.Label == label
})
}
func cleanupStaleFiles(p *project.Project, desired map[string]struct{}, scope extractionScope) (int, []error) { func cleanupStaleFiles(p *project.Project, desired map[string]struct{}, scope extractionScope) (int, []error) {
candidates := make([]string, 0, len(p.Inventory.SourceFiles)+len(p.Inventory.ScriptFiles)+len(p.Inventory.AssetFiles)) candidates := make([]string, 0, len(p.Inventory.SourceFiles)+len(p.Inventory.ScriptFiles)+len(p.Inventory.AssetFiles))
if scope.Source && safeCleanupRoot(p.SourceDir(), p.Root) { if scope.Source && safeCleanupRoot(p.SourceDir(), p.Root) {
-107
View File
@@ -134,113 +134,6 @@ func TestBuildThenExtract(t *testing.T) {
} }
} }
func TestExtractSkipsAreaWhenOnlyRootVersionChanges(t *testing.T) {
root := t.TempDir()
mustMkdir(t, filepath.Join(root, "src", "module"))
mustMkdir(t, filepath.Join(root, "src", "areas"))
mustMkdir(t, filepath.Join(root, "assets"))
mustMkdir(t, filepath.Join(root, "build"))
mustWriteFile(t, filepath.Join(root, "nwn-tool.json"), `{
"module": {
"name": "Test Module",
"resref": "testmod"
},
"paths": {
"source": "src",
"assets": "assets",
"build": "build"
}
}
`)
mustWriteFile(t, filepath.Join(root, "src", "module", "module.ifo.json"), `{
"file_type": "IFO ",
"file_version": "V3.2",
"root": {
"struct_type": 0,
"fields": [
{
"label": "Mod_Name",
"type": "CExoString",
"value": "Test Module"
}
]
}
}
`)
mustWriteFile(t, filepath.Join(root, "src", "areas", "area001.are.json"), `{
"file_type": "ARE ",
"file_version": "V3.2",
"root": {
"struct_type": 0,
"fields": [
{
"label": "Version",
"type": "DWord",
"value": 2
},
{
"label": "Tag",
"type": "CExoString",
"value": "area001"
}
]
}
}
`)
p, err := project.Load(root)
if err != nil {
t.Fatalf("load project: %v", err)
}
if err := p.ValidateLayout(); err != nil {
t.Fatalf("validate layout: %v", err)
}
if err := p.Scan(); err != nil {
t.Fatalf("scan: %v", err)
}
if _, err := BuildModule(p); err != nil {
t.Fatalf("build module: %v", err)
}
mustWriteFile(t, filepath.Join(root, "src", "areas", "area001.are.json"), `{
"file_type": "ARE ",
"file_version": "V3.2",
"root": {
"struct_type": 0,
"fields": [
{
"label": "Version",
"type": "DWord",
"value": 1
},
{
"label": "Tag",
"type": "CExoString",
"value": "area001"
}
]
}
}
`)
if err := p.Scan(); err != nil {
t.Fatalf("rescan: %v", err)
}
result, err := Extract(p)
if err != nil {
t.Fatalf("extract: %v", err)
}
if result.Overwritten != 0 {
t.Fatalf("Version-only area extraction must not overwrite, got %d overwritten", result.Overwritten)
}
document := readGFFJSON(t, filepath.Join(root, "src", "areas", "area001.are.json"))
if got, want := fieldValue(t, document.Root, "Version"), gff.DWordValue(1); got != want {
t.Fatalf("expected existing Version %#v to remain, got %#v", want, got)
}
}
func TestExtractReadsHAKAssets(t *testing.T) { func TestExtractReadsHAKAssets(t *testing.T) {
root := t.TempDir() root := t.TempDir()
mustMkdir(t, filepath.Join(root, "src")) mustMkdir(t, filepath.Join(root, "src"))
-9
View File
@@ -1581,15 +1581,6 @@ func validateAutogenConfig(cfg AutogenConfig) []error {
func validateAutogenConsumerSources(consumers []AutogenConsumerConfig) []error { func validateAutogenConsumerSources(consumers []AutogenConsumerConfig) []error {
var failures []error var failures []error
partsRows := 0
for _, c := range consumers {
if strings.TrimSpace(c.Mode) == "parts_rows" {
partsRows++
}
}
if partsRows > 1 {
failures = append(failures, fmt.Errorf("at most one autogen consumer may use mode parts_rows; found %d (override precedence would be ambiguous)", partsRows))
}
for _, c := range consumers { for _, c := range consumers {
if strings.TrimSpace(c.Source.Kind) != "cdn_channel" { if strings.TrimSpace(c.Source.Kind) != "cdn_channel" {
continue continue
-14
View File
@@ -1864,20 +1864,6 @@ func TestValidateAutogenConsumerSourcesCDNChannel(t *testing.T) {
} }
} }
func TestValidateRejectsMultiplePartsRowsConsumers(t *testing.T) {
consumers := []AutogenConsumerConfig{
{ID: "parts-a", Mode: "parts_rows"},
{ID: "parts-b", Mode: "parts_rows"},
}
if failures := validateAutogenConsumerSources(consumers); len(failures) == 0 {
t.Fatal("expected error for two parts_rows consumers")
}
single := []AutogenConsumerConfig{{ID: "parts", Mode: "parts_rows"}}
if failures := validateAutogenConsumerSources(single); len(failures) != 0 {
t.Fatalf("single parts_rows consumer must validate, got %v", failures)
}
}
func loadAndValidate(root string) error { func loadAndValidate(root string) error {
proj, err := Load(root) proj, err := Load(root)
if err != nil { if err != nil {
@@ -65,33 +65,6 @@ effective output while keeping project policy in topdata authoring files.
`position` defaults to `append`. The module uses `prepend` for class feat `position` defaults to `append`. The module uses `prepend` for class feat
injections to keep generated policy rows before per-class authored rows. injections to keep generated policy rows before per-class authored rows.
## Global Injection Grammar
Each injection entry may use only `row`, `require_present`, `any_present`, and
`unless_present`. `when_present` is invalid and rejected; it is not an alias.
```json
{
"row": {
"FeatIndex": { "id": "feat:example" }
},
"require_present": [{ "field": "FeatIndex", "id": "feat:required" }],
"any_present": [
{ "field": "FeatIndex", "id": "masterfeats:metamagic" },
{ "field": "FeatIndex", "id": "masterfeats:combat" }
],
"unless_present": [{ "field": "FeatIndex", "id": "feat:example" }]
}
```
- `require_present` means every listed reference must exist.
- `any_present` means at least one listed reference must exist, and the array
must not be empty.
- `unless_present` means none of the listed references may exist.
- Every present group on an entry must pass.
- Conditions observe the current rows in injection order, so earlier injections
can affect later conditions.
## Acceptance Criteria ## Acceptance Criteria
- Class feat output contains the same effective shared feats as the prior YAML - Class feat output contains the same effective shared feats as the prior YAML
+17 -104
View File
@@ -299,34 +299,23 @@ func resolveCDNChannelManifest(p *project.Project, consumer project.AutogenConsu
progress = func(string) {} progress = func(string) {}
} }
// Manifest basename + provenance label are derived from config, not hardcoded, // 1. Offline / air-gapped override: a vfxs.yml path or a manifest-repo
// so the same resolver serves vfxs.yml (accessory VFX) and part.yml (parts). // checkout root containing assets/vfxs.yml. Skips the network entirely.
manifestBase := filepath.Base(filepath.FromSlash(strings.TrimSpace(src.ManifestPath)))
if manifestBase == "." || manifestBase == "" || manifestBase == string(filepath.Separator) {
manifestBase = "manifest.yml"
}
label := strings.TrimSpace(consumer.ID)
if label == "" {
label = "autogen"
}
// 1. Offline / air-gapped override: a manifest file path or a manifest-repo
// checkout root containing assets/<manifestBase>. Skips the network entirely.
if envName := strings.TrimSpace(src.OfflineOverrideEnv); envName != "" { if envName := strings.TrimSpace(src.OfflineOverrideEnv); envName != "" {
if override := strings.TrimSpace(os.Getenv(envName)); override != "" { if override := strings.TrimSpace(os.Getenv(envName)); override != "" {
manifestPath := override vfxsPath := override
if info, err := os.Stat(override); err == nil && info.IsDir() { if info, err := os.Stat(override); err == nil && info.IsDir() {
manifestPath = filepath.Join(override, "assets", manifestBase) vfxsPath = filepath.Join(override, "assets", "vfxs.yml")
} }
raw, err := os.ReadFile(manifestPath) raw, err := os.ReadFile(vfxsPath)
if err != nil { if err != nil {
return nil, fmt.Errorf("offline %s override %s: %w", manifestBase, manifestPath, err) return nil, fmt.Errorf("offline vfxs override %s: %w", vfxsPath, err)
} }
entries, err := filterCDNChannelEntries(raw, consumer) entries, err := filterCDNChannelEntries(raw, consumer)
if err != nil { if err != nil {
return nil, err return nil, err
} }
progress(fmt.Sprintf("Using offline %s override for %s from %s...", manifestBase, consumer.ID, manifestPath)) progress(fmt.Sprintf("Using offline vfxs override for %s from %s...", consumer.ID, vfxsPath))
return &autogenManifest{ID: consumer.Producer, Ref: "local", Entries: entries}, nil return &autogenManifest{ID: consumer.Producer, Ref: "local", Entries: entries}, nil
} }
} }
@@ -370,13 +359,13 @@ func resolveCDNChannelManifest(p *project.Project, consumer project.AutogenConsu
if tag == "" { if tag == "" {
return nil, unavailableAutogenManifest(fmt.Errorf("channel %q absent in channels.json (%s)", channel, channelsURL)) return nil, unavailableAutogenManifest(fmt.Errorf("channel %q absent in channels.json (%s)", channel, channelsURL))
} }
progress(fmt.Sprintf("%s: channel %s -> tag %s", label, channel, tag)) progress(fmt.Sprintf("Accessory VFX: channel %s -> tag %s", channel, tag))
// 5. per-tag manifest for the tag. // 5. vfxs.yml for the tag.
manifestURL := join(src.ManifestPath, tag) manifestURL := join(src.ManifestPath, tag)
vstatus, vbody, err := httpGetStatus(manifestURL) vstatus, vbody, err := httpGetStatus(manifestURL)
if err != nil { if err != nil {
return nil, unavailableAutogenManifest(fmt.Errorf("%s unreachable %s: %w", manifestBase, manifestURL, err)) return nil, unavailableAutogenManifest(fmt.Errorf("vfxs.yml unreachable %s: %w", manifestURL, err))
} }
switch vstatus { switch vstatus {
case http.StatusOK: case http.StatusOK:
@@ -388,103 +377,27 @@ func resolveCDNChannelManifest(p *project.Project, consumer project.AutogenConsu
if marker := strings.TrimSpace(src.ReleaseMarkerPath); marker != "" { if marker := strings.TrimSpace(src.ReleaseMarkerPath); marker != "" {
markerURL := join(marker, tag) markerURL := join(marker, tag)
if mstatus, _, merr := httpGetStatus(markerURL); merr == nil && mstatus == http.StatusOK { if mstatus, _, merr := httpGetStatus(markerURL); merr == nil && mstatus == http.StatusOK {
return nil, fmt.Errorf("release %s has %s but no %s (%s) — rows would silently vanish; backfill/republish %s for %s", tag, marker, manifestBase, manifestURL, manifestBase, tag) return nil, fmt.Errorf("release %s has %s but no vfxs.yml (%s) — accessory rows would silently vanish; backfill/republish vfxs.yml for %s", tag, marker, manifestURL, tag)
} }
} }
return nil, unavailableAutogenManifest(fmt.Errorf("%s 404 %s (no published release for %s)", manifestBase, manifestURL, tag)) return nil, unavailableAutogenManifest(fmt.Errorf("vfxs.yml 404 %s (no published release for %s)", manifestURL, tag))
default: default:
return nil, unavailableAutogenManifest(fmt.Errorf("%s HTTP %d %s", manifestBase, vstatus, manifestURL)) return nil, unavailableAutogenManifest(fmt.Errorf("vfxs.yml HTTP %d %s", vstatus, manifestURL))
} }
entries, err := filterCDNChannelEntries(vbody, consumer) entries, err := filterCDNChannelEntries(vbody, consumer)
if err != nil { if err != nil {
return nil, err // malformed manifest = HARD fail return nil, err // malformed vfxs.yml = HARD fail
} }
progress(fmt.Sprintf("%s: resolved %d model entries from %s", label, len(entries), manifestURL)) progress(fmt.Sprintf("Accessory VFX: resolved %d model entries from %s", len(entries), manifestURL))
return &autogenManifest{ID: consumer.Producer, Ref: tag, Entries: entries}, nil return &autogenManifest{ID: consumer.Producer, Ref: tag, Entries: entries}, nil
} }
// filterCDNChannelEntries dispatches per-tag manifest parsing on the consumer // filterCDNChannelEntries parses vfxs.yml and keeps restype==mdl assets under
// mode. parts_rows consumes part.yml (the parts inventory contract); every other
// mode consumes vfxs.yml (the accessory-VFX contract).
func filterCDNChannelEntries(raw []byte, consumer project.AutogenConsumerConfig) ([]autogenManifestEntry, error) {
switch strings.TrimSpace(consumer.Mode) {
case "parts_rows":
return filterPartsCDNChannelEntries(raw)
default:
return filterVFXCDNChannelEntries(raw, consumer)
}
}
// filterPartsCDNChannelEntries parses part.yml and keeps restype==mdl assets
// under part/<supported-category>/.../<stem><digits>.mdl. The asset category is
// the path segment after part/; the row ID is the trailing decimal of the
// filename stem. Body/race/gender/left-right variants dedup by (category,rowID).
// Zero accepted rows is a HARD fail (silent-drop guard).
func filterPartsCDNChannelEntries(raw []byte) ([]autogenManifestEntry, error) {
var manifest struct {
Assets *[]struct {
Path string `yaml:"path"`
Restype string `yaml:"restype"`
} `yaml:"assets"`
}
if err := yaml.Unmarshal(raw, &manifest); err != nil {
return nil, fmt.Errorf("malformed part.yml (cannot parse): %w", err)
}
if manifest.Assets == nil {
return nil, fmt.Errorf("malformed part.yml (no assets array)")
}
seen := map[string]struct{}{}
var entries []autogenManifestEntry
for _, a := range *manifest.Assets {
if a.Restype != "mdl" {
continue
}
path := filepath.ToSlash(strings.TrimSpace(a.Path))
if !strings.HasPrefix(path, "part/") {
continue
}
rel := strings.TrimPrefix(path, "part/")
segs := strings.Split(rel, "/")
if len(segs) < 2 {
continue
}
category := segs[0]
if !isSupportedPartCategory(category) {
continue // ignores _masters, cloak, head, helm, tail, wings
}
stem := strings.TrimSuffix(segs[len(segs)-1], ".mdl")
match := trailingNumberRegex.FindString(stem)
if match == "" {
return nil, fmt.Errorf("part.yml: supported-category model %q has no trailing row number", path)
}
rowID, err := strconv.Atoi(match)
if err != nil || rowID == 0 {
return nil, fmt.Errorf("part.yml: supported-category model %q resolves to invalid row id %q", path, match)
}
key := category + "/" + strconv.Itoa(rowID)
if _, dup := seen[key]; dup {
continue
}
seen[key] = struct{}{}
entries = append(entries, autogenManifestEntry{
Source: rel, Group: category, ModelStem: stem, RowID: rowID,
})
}
if len(entries) == 0 {
return nil, fmt.Errorf("part.yml: zero supported part rows after filtering")
}
slices.SortFunc(entries, func(a, b autogenManifestEntry) int {
return strings.Compare(a.Source, b.Source)
})
return entries, nil
}
// filterVFXCDNChannelEntries parses vfxs.yml and keeps restype==mdl assets under
// vfxs/<group>/ for the consumer's 4 accessory groups, stripping the leading // vfxs/<group>/ for the consumer's 4 accessory groups, stripping the leading
// vfxs/ from each source path. A present-but-malformed manifest (unparseable, or // vfxs/ from each source path. A present-but-malformed manifest (unparseable, or
// no assets array) is an error; an empty assets array yields zero entries. // no assets array) is an error; an empty assets array yields zero entries.
func filterVFXCDNChannelEntries(raw []byte, consumer project.AutogenConsumerConfig) ([]autogenManifestEntry, error) { func filterCDNChannelEntries(raw []byte, consumer project.AutogenConsumerConfig) ([]autogenManifestEntry, error) {
var manifest struct { var manifest struct {
Assets *[]struct { Assets *[]struct {
Path string `yaml:"path"` Path string `yaml:"path"`
@@ -5,7 +5,6 @@ import (
"fmt" "fmt"
"net/http" "net/http"
"net/http/httptest" "net/http/httptest"
"os"
"path/filepath" "path/filepath"
"testing" "testing"
@@ -195,35 +194,6 @@ func TestResolveCDNChannelOfflineOverride(t *testing.T) {
} }
} }
// TestResolveCDNChannelOfflineUsesManifestBasename proves the resolver derives
// the offline checkout filename from manifest_path's basename (part.yml here),
// not a hardcoded assets/vfxs.yml, and routes parts_rows through the parts filter.
func TestResolveCDNChannelOfflineUsesManifestBasename(t *testing.T) {
root := testProjectRoot(t)
dir := filepath.Join(root, "manifest-checkout")
if err := os.MkdirAll(filepath.Join(dir, "assets"), 0o755); err != nil {
t.Fatal(err)
}
writeFile(t, filepath.Join(dir, "assets", "part.yml"),
"assets:\n - {path: part/belt/m/pfa0_belt017.mdl, restype: mdl}\n")
t.Setenv("SOW_PART_MANIFEST", dir)
c := projectPartsConsumer()
c.Source = project.AutogenSourceConfig{
Kind: "cdn_channel",
ManifestPath: "releases/haks/{tag}/part.yml",
OfflineOverrideEnv: "SOW_PART_MANIFEST",
}
m, err := resolveCDNChannelManifest(testProject(root), c, c.Source, nil)
if err != nil {
t.Fatalf("resolve: %v", err)
}
if len(m.Entries) != 1 || m.Entries[0].RowID != 17 || m.Entries[0].Group != "belt" {
t.Fatalf("unexpected entries: %+v", m.Entries)
}
}
func errorIsUnavailable(err error) bool { func errorIsUnavailable(err error) bool {
return errors.Is(err, errAutogenManifestUnavailable) return errors.Is(err, errAutogenManifestUnavailable)
} }
-195
View File
@@ -1,195 +0,0 @@
package topdata
import (
"errors"
"fmt"
"slices"
"strings"
)
type globalCondition struct {
Field string
ID string
}
type globalConditionGroups struct {
RequirePresent []globalCondition
AnyPresent []globalCondition
UnlessPresent []globalCondition
}
var (
baseOrPlainRootKeys = []string{
"output", "key", "columns", "rows", "compare_reference",
}
canonicalModuleRootKeys = []string{
"output", "key", "columns", "entries", "overrides", "rows",
}
globalRootKeys = []string{
"columns", "entries", "overrides", "defaults", "position", "injections",
}
globalInjectionKeys = []string{
"row", "require_present", "any_present", "unless_present",
}
globalConditionKeys = []string{"field", "id"}
globalDefaultRuleKeys = []string{"match", "values"}
globalDefaultMatchKeys = []string{"source"}
globalDefaultFormatKeys = []string{"format"}
)
func unsupportedObjectKeys(obj map[string]any, supported ...string) []string {
if len(obj) == 0 {
return nil
}
supportedSet := make(map[string]struct{}, len(supported))
for _, key := range supported {
supportedSet[key] = struct{}{}
}
unsupported := make([]string, 0, len(obj))
for key := range obj {
if _, ok := supportedSet[key]; ok {
continue
}
unsupported = append(unsupported, key)
}
slices.Sort(unsupported)
return unsupported
}
func unsupportedObjectKeysError(label string, obj map[string]any, supported ...string) error {
unsupported := unsupportedObjectKeys(obj, supported...)
if len(unsupported) == 0 {
return nil
}
return fmt.Errorf("%s contains unsupported key %q (supported keys: %s)", label, unsupported[0], strings.Join(supported, ", "))
}
type globalConditionParseOptions struct {
Label string
Report func(error)
}
func parseGlobalConditionGroups(injection map[string]any, options ...globalConditionParseOptions) (globalConditionGroups, error) {
if injection == nil {
return globalConditionGroups{}, nil
}
label := "global injection"
var report func(error)
if len(options) > 0 {
if strings.TrimSpace(options[0].Label) != "" {
label = options[0].Label
}
report = options[0].Report
}
var errs []error
addError := func(err error) {
if err == nil {
return
}
errs = append(errs, err)
if report != nil {
report(err)
}
}
if err := unsupportedObjectKeysError(label, injection, globalInjectionKeys...); err != nil {
addError(err)
}
var groups globalConditionGroups
if raw, ok := injection["require_present"]; ok {
parsed, err := parseGlobalConditionList(label+" require_present", raw, false)
if err != nil {
addError(err)
} else {
groups.RequirePresent = parsed
}
}
if raw, ok := injection["any_present"]; ok {
parsed, err := parseGlobalConditionList(label+" any_present", raw, true)
if err != nil {
addError(err)
} else {
groups.AnyPresent = parsed
}
}
if raw, ok := injection["unless_present"]; ok {
parsed, err := parseGlobalConditionList(label+" unless_present", raw, false)
if err != nil {
addError(err)
} else {
groups.UnlessPresent = parsed
}
}
return groups, errors.Join(errs...)
}
func parseGlobalConditionList(name string, raw any, requireNonEmpty bool) ([]globalCondition, error) {
if raw == nil {
return nil, fmt.Errorf("%s must be an array", name)
}
conditions, ok := raw.([]any)
if !ok {
return nil, fmt.Errorf("%s must be an array", name)
}
if requireNonEmpty && len(conditions) == 0 {
return nil, fmt.Errorf("%s must contain at least one condition", name)
}
parsed := make([]globalCondition, 0, len(conditions))
for index, rawCondition := range conditions {
condition, ok := rawCondition.(map[string]any)
if !ok {
return nil, fmt.Errorf("%s[%d] must be an object", name, index)
}
if err := unsupportedObjectKeysError(fmt.Sprintf("%s[%d]", name, index), condition, globalConditionKeys...); err != nil {
return nil, err
}
field, _ := condition["field"].(string)
field = strings.TrimSpace(field)
if field == "" {
return nil, fmt.Errorf("%s[%d].field is required", name, index)
}
id, _ := condition["id"].(string)
id = strings.TrimSpace(id)
if id == "" {
return nil, fmt.Errorf("%s[%d].id is required", name, index)
}
parsed = append(parsed, globalCondition{Field: field, ID: id})
}
return parsed, nil
}
func globalConditionGroupsMatch(groups globalConditionGroups, rows []map[string]any) bool {
for _, condition := range groups.RequirePresent {
if !globalReferencePresent(rows, condition.Field, condition.ID) {
return false
}
}
if groups.AnyPresent != nil {
if len(groups.AnyPresent) == 0 {
return false
}
matched := false
for _, condition := range groups.AnyPresent {
if globalReferencePresent(rows, condition.Field, condition.ID) {
matched = true
break
}
}
if !matched {
return false
}
}
for _, condition := range groups.UnlessPresent {
if globalReferencePresent(rows, condition.Field, condition.ID) {
return false
}
}
return true
}
File diff suppressed because it is too large Load Diff
+42 -13
View File
@@ -460,15 +460,7 @@ func buildNativeUnchecked(p *project.Project, opts NativeBuildOptions, progress
if err != nil { if err != nil {
return BuildResult{}, err return BuildResult{}, err
} }
// Single explicit parts sequence: augment (inside applyAutogenConsumers) -> collected, err = applyPartOverrides(sourceDir, collected)
// normalize -> apply overrides exactly once, all under the one configured
// parts_rows consumer's policy.
partsCfg := partsRowsConfigForProject(p)
collected, err = normalizePartsRowsACBonus(collected, partsCfg)
if err != nil {
return BuildResult{}, err
}
collected, err = applyPartOverridesWithConfig(sourceDir, collected, partsCfg)
if err != nil { if err != nil {
return BuildResult{}, err return BuildResult{}, err
} }
@@ -3894,11 +3886,48 @@ func globalManifestPosition(module nativeGeneratedModule) (string, error) {
} }
func globalInjectionConditionsMatch(injection map[string]any, rows []map[string]any) (bool, error) { func globalInjectionConditionsMatch(injection map[string]any, rows []map[string]any) (bool, error) {
groups, err := parseGlobalConditionGroups(injection) if rawRequired, ok := injection["require_present"]; ok {
if err != nil { matches, err := globalConditionListMatches("require_present", rawRequired, rows, true)
return false, err if err != nil || !matches {
return matches, err
} }
return globalConditionGroupsMatch(groups, rows), nil }
if rawBlocked, ok := injection["unless_present"]; ok {
matches, err := globalConditionListMatches("unless_present", rawBlocked, rows, false)
if err != nil || !matches {
return matches, err
}
}
return true, nil
}
func globalConditionListMatches(name string, raw any, rows []map[string]any, required bool) (bool, error) {
conditions, ok := raw.([]any)
if !ok {
return false, fmt.Errorf("%s must be an array", name)
}
for index, rawCondition := range conditions {
condition, ok := rawCondition.(map[string]any)
if !ok {
return false, fmt.Errorf("%s[%d] must be an object", name, index)
}
field, _ := condition["field"].(string)
id, _ := condition["id"].(string)
if strings.TrimSpace(field) == "" {
return false, fmt.Errorf("%s[%d].field is required", name, index)
}
if strings.TrimSpace(id) == "" {
return false, fmt.Errorf("%s[%d].id is required", name, index)
}
found := globalReferencePresent(rows, field, id)
if required && !found {
return false, nil
}
if !required && found {
return false, nil
}
}
return true, nil
} }
func globalRowIdentityPresent(row map[string]any, rows []map[string]any) bool { func globalRowIdentityPresent(row map[string]any, rows []map[string]any) bool {
-44
View File
@@ -1,8 +1,6 @@
package topdata package topdata
import ( import (
"os"
"path/filepath"
"testing" "testing"
"git.westgate.pw/ShadowsOverWestgate/sow-tools/internal/project" "git.westgate.pw/ShadowsOverWestgate/sow-tools/internal/project"
@@ -56,45 +54,3 @@ func TestParityGuardCDNChannelProducesAccessoryRows(t *testing.T) {
t.Fatalf("PARITY GUARD: expected lock id for %s, got %#v", wantKey, got[0].LockData) t.Fatalf("PARITY GUARD: expected lock id for %s, got %#v", wantKey, got[0].LockData)
} }
} }
// PARITY GUARD (parts): a bare project whose only parts source is the offline
// CDN-equivalent (SOW_PART_MANIFEST -> a part.yml), with no NWN_ROOT, no token,
// no asset checkout, must produce parts rows from the filtered inventory. If
// parts resolution ever drifts back into a wrapper, this fails.
func TestParityGuardCDNChannelProducesPartsRows(t *testing.T) {
root := testProjectRoot(t)
checkout := filepath.Join(root, "manifest-checkout")
if err := os.MkdirAll(filepath.Join(checkout, "assets"), 0o755); err != nil {
t.Fatal(err)
}
writeFile(t, filepath.Join(checkout, "assets", "part.yml"),
"assets:\n - {path: part/belt/m/pfa0_belt017.mdl, restype: mdl}\n - {path: part/belt/m/pfa0_belt018.mdl, restype: mdl}\n")
t.Setenv("SOW_PART_MANIFEST", checkout)
t.Setenv("NWN_ROOT", "")
c := projectPartsConsumer()
c.Source = project.AutogenSourceConfig{
Kind: "cdn_channel",
ChannelsPath: "releases/haks/channels.json",
ManifestPath: "releases/haks/{tag}/part.yml",
ReleaseMarkerPath: "releases/haks/{tag}/haks.json",
OfflineOverrideEnv: "SOW_PART_MANIFEST",
}
p := testProject(root)
p.Config.Autogen.Consumers = []project.AutogenConsumerConfig{c}
collected := []nativeCollectedDataset{partsDatasetWithIDs("parts/belt", nil)}
got, err := applyAutogenConsumers(p, collected, nil)
if err != nil {
t.Fatalf("applyAutogenConsumers failed: %v", err)
}
ids := map[int]bool{}
for _, row := range got[0].Rows {
ids[row["id"].(int)] = true
}
if !ids[17] || !ids[18] {
t.Fatalf("PARITY GUARD: expected belt rows 17 and 18 from inventory, got %#v", got[0].Rows)
}
}
@@ -1,87 +0,0 @@
package topdata
import (
"testing"
"git.westgate.pw/ShadowsOverWestgate/sow-tools/internal/project"
)
func projectPartsConsumer() project.AutogenConsumerConfig {
return project.AutogenConsumerConfig{ID: "parts", Producer: "parts", Mode: "parts_rows"}
}
const partYML = `assets:
- {path: part/belt/m/pfa0_belt017.mdl, restype: mdl}
- {path: part/hand/m/pfh0_handl105.mdl, restype: mdl}
- {path: part/hand/m/pfh0_handr105.mdl, restype: mdl}
- {path: part/leg/m/pfa0_legl001.mdl, restype: mdl}
- {path: part/leg/m/pfa0_legr001.mdl, restype: mdl}
- {path: part/cloak/m/cloak001.mdl, restype: mdl}
- {path: part/belt/m/readme.txt, restype: txt}
`
func TestFilterPartsKeepsSupportedDedupsVariants(t *testing.T) {
consumer := projectPartsConsumer()
entries, err := filterCDNChannelEntries([]byte(partYML), consumer)
if err != nil {
t.Fatalf("filter: %v", err)
}
inv := autogenPartsInventory(entries)
if _, ok := inv["belt"][17]; !ok {
t.Errorf("missing belt 17")
}
if len(inv["hand"]) != 1 {
t.Errorf("hand should dedup l/r to 1 row, got %d", len(inv["hand"]))
}
if _, ok := inv["hand"][105]; !ok {
t.Errorf("missing hand 105")
}
if len(inv["leg"]) != 1 {
t.Errorf("leg should dedup l/r to 1 row, got %d", len(inv["leg"]))
}
if _, ok := inv["cloak"]; ok {
t.Errorf("cloak must be ignored")
}
}
func TestFilterPartsRejectsZeroAccepted(t *testing.T) {
_, err := filterCDNChannelEntries([]byte("assets:\n - {path: part/cloak/m/cloak001.mdl, restype: mdl}\n"), projectPartsConsumer())
if err == nil {
t.Fatal("expected hard error on zero accepted part rows")
}
}
func TestFilterPartsRejectsRowZeroAndNonNumeric(t *testing.T) {
for _, doc := range []string{
"assets:\n - {path: part/belt/m/pfa0_belt000.mdl, restype: mdl}\n",
"assets:\n - {path: part/belt/m/pfa0_belt.mdl, restype: mdl}\n",
} {
if _, err := filterCDNChannelEntries([]byte(doc), projectPartsConsumer()); err == nil {
t.Fatalf("expected error for %q", doc)
}
}
}
func TestFilterPartsRejectsMalformedAndMissingAssets(t *testing.T) {
if _, err := filterCDNChannelEntries([]byte("assets:\n - path: [unterminated"), projectPartsConsumer()); err == nil {
t.Fatal("expected malformed error")
}
if _, err := filterCDNChannelEntries([]byte("other: 1\n"), projectPartsConsumer()); err == nil {
t.Fatal("expected no-assets error")
}
}
func TestSupportedPartCategoriesCoverTwelveDatasets(t *testing.T) {
want := []string{"belt", "bicep", "chest", "foot", "forearm", "hand", "leg", "neck", "pelvis", "robe", "shin", "shoulder"}
for _, c := range want {
if !isSupportedPartCategory(c) {
t.Errorf("category %q not supported", c)
}
}
if partDatasetToAssetCategory["hand"] != "hand" {
t.Errorf("parts/hand must map to asset category hand")
}
if partDatasetToAssetCategory["legs"] != "leg" {
t.Errorf("parts/legs must map to asset category leg")
}
}
+3 -15
View File
@@ -21,7 +21,6 @@ var supportedPartCategories = []string{
"chest", "chest",
"foot", "foot",
"forearm", "forearm",
"hand",
"leg", "leg",
"neck", "neck",
"pelvis", "pelvis",
@@ -36,7 +35,6 @@ var partDatasetToAssetCategory = map[string]string{
"chest": "chest", "chest": "chest",
"foot": "foot", "foot": "foot",
"forearm": "forearm", "forearm": "forearm",
"hand": "hand",
"legs": "leg", "legs": "leg",
"neck": "neck", "neck": "neck",
"pelvis": "pelvis", "pelvis": "pelvis",
@@ -392,18 +390,6 @@ func applyPartOverrides(sourceDir string, collected []nativeCollectedDataset) ([
return applyPartOverridesWithConfig(sourceDir, collected, project.PartsRowsConfig{}) return applyPartOverridesWithConfig(sourceDir, collected, project.PartsRowsConfig{})
} }
// partsRowsConfigForProject returns the PartsRows policy of the single configured
// parts_rows autogen consumer, or the zero value if none. Project validation
// guarantees at most one parts_rows consumer, so the first match is canonical.
func partsRowsConfigForProject(p *project.Project) project.PartsRowsConfig {
for _, c := range p.Config.Autogen.Consumers {
if strings.TrimSpace(c.Mode) == "parts_rows" {
return c.PartsRows
}
}
return project.PartsRowsConfig{}
}
func applyPartOverridesWithConfig(sourceDir string, collected []nativeCollectedDataset, cfg project.PartsRowsConfig) ([]nativeCollectedDataset, error) { func applyPartOverridesWithConfig(sourceDir string, collected []nativeCollectedDataset, cfg project.PartsRowsConfig) ([]nativeCollectedDataset, error) {
result := make([]nativeCollectedDataset, len(collected)) result := make([]nativeCollectedDataset, len(collected))
copy(result, collected) copy(result, collected)
@@ -444,7 +430,9 @@ func applyPartOverridesWithConfig(sourceDir string, collected []nativeCollectedD
} }
row, ok := rowByID[rowID] row, ok := rowByID[rowID]
if !ok { if !ok {
return nil, fmt.Errorf("parts/%s override %d targets row id %d which is neither a baseline row nor a discovered model row; author the row in data/parts/%s.json instead", category, index, rowID, category) row = createDefaultPartRowForDataset(rowID, dataset.Dataset.Name, dataset.Columns, cfg)
rows = append(rows, row)
rowByID[rowID] = row
} }
for field, value := range override { for field, value := range override {
if field == "id" { if field == "id" {
-51
View File
@@ -1,51 +0,0 @@
package topdata
import (
"os"
"path/filepath"
"testing"
"git.westgate.pw/ShadowsOverWestgate/sow-tools/internal/project"
)
func partsDatasetWithIDs(name string, ids []int) nativeCollectedDataset {
rows := make([]map[string]any, 0, len(ids))
for _, id := range ids {
rows = append(rows, map[string]any{"id": id, "COSTMODIFIER": "0", "ACBONUS": "0.00"})
}
return nativeCollectedDataset{
Dataset: nativeDataset{Name: name, Kind: nativeDatasetBase},
Columns: []string{"COSTMODIFIER", "ACBONUS"},
Rows: rows,
}
}
func TestApplyPartOverridesRejectsOrphan(t *testing.T) {
dir := t.TempDir()
if err := os.MkdirAll(filepath.Join(dir, "data", "parts", "overrides"), 0o755); err != nil {
t.Fatal(err)
}
writeFile(t, filepath.Join(dir, "data", "parts", "overrides", "belt.json"),
`{"overrides": [{"id": 999, "COSTMODIFIER": "5"}]}`)
collected := []nativeCollectedDataset{partsDatasetWithIDs("parts/belt", []int{17})}
if _, err := applyPartOverridesWithConfig(dir, collected, project.PartsRowsConfig{}); err == nil {
t.Fatal("expected orphan override to fail, not synthesize a row")
}
}
func TestApplyPartOverridesUpdatesDiscoveredRow(t *testing.T) {
dir := t.TempDir()
if err := os.MkdirAll(filepath.Join(dir, "data", "parts", "overrides"), 0o755); err != nil {
t.Fatal(err)
}
writeFile(t, filepath.Join(dir, "data", "parts", "overrides", "belt.json"),
`{"overrides": [{"id": 17, "COSTMODIFIER": "5"}]}`)
collected := []nativeCollectedDataset{partsDatasetWithIDs("parts/belt", []int{17})}
out, err := applyPartOverridesWithConfig(dir, collected, project.PartsRowsConfig{})
if err != nil {
t.Fatalf("override: %v", err)
}
if got := out[0].Rows[0]["COSTMODIFIER"]; got != "5" {
t.Fatalf("row 17 COSTMODIFIER want 5, got %v", got)
}
}
+73 -59
View File
@@ -333,17 +333,6 @@ func validateDuplicateJSONKeys(path string, raw []byte, report *ValidationReport
} }
} }
func reportTopdataContractError(path string, err error, report *ValidationReport) {
if err == nil {
return
}
report.Diagnostics = append(report.Diagnostics, Diagnostic{
Severity: SeverityError,
Path: path,
Message: err.Error(),
})
}
func scanJSONValueForDuplicateKeys(decoder *json.Decoder, path string, report *ValidationReport) error { func scanJSONValueForDuplicateKeys(decoder *json.Decoder, path string, report *ValidationReport) error {
token, err := decoder.Token() token, err := decoder.Token()
if err != nil { if err != nil {
@@ -436,40 +425,35 @@ func validateDataObject(path string, obj map[string]any, report *ValidationRepor
return return
} }
if base == "base.json" { if base == "base.json" {
reportTopdataContractError(path, unsupportedObjectKeysError("base.json root", obj, baseOrPlainRootKeys...), report)
validateRowsFile(path, obj, report, true) validateRowsFile(path, obj, report, true)
return return
} }
if inModules { if inModules {
reportTopdataContractError(path, unsupportedObjectKeysError("module root", obj, canonicalModuleRootKeys...), report) if _, hasEntries := obj["entries"]; hasEntries {
_, hasColumns := obj["columns"] validateEntriesFile(path, obj, report)
_, hasEntries := obj["entries"] return
_, hasOverrides := obj["overrides"] }
_, hasRows := obj["rows"] if _, hasOverrides := obj["overrides"]; hasOverrides {
if hasColumns { validateOverridesFile(path, obj, report)
return
}
if _, hasRows := obj["rows"]; hasRows {
validateRowsFile(path, obj, report, false)
return
}
if _, hasColumns := obj["columns"]; hasColumns {
validateColumnsFile(path, obj, report) validateColumnsFile(path, obj, report)
return
} }
if hasEntries {
validateEntriesFileWithColumns(path, obj, report, false)
}
if hasOverrides {
validateOverridesFileWithColumns(path, obj, report, false)
}
if hasRows {
validateRowsFileWithColumns(path, obj, report, false, false)
}
if !hasColumns && !hasEntries && !hasOverrides && !hasRows {
report.Diagnostics = append(report.Diagnostics, Diagnostic{ report.Diagnostics = append(report.Diagnostics, Diagnostic{
Severity: SeverityWarning, Severity: SeverityWarning,
Path: path, Path: path,
Message: "module file does not use a recognized canonical shape yet; expected one of columns, entries, overrides, or rows", Message: "module file does not use a recognized canonical shape yet; expected one of columns, entries, overrides, or rows",
}) })
}
return return
} }
if _, hasRows := obj["rows"]; hasRows { if _, hasRows := obj["rows"]; hasRows {
reportTopdataContractError(path, unsupportedObjectKeysError("plain rows root", obj, baseOrPlainRootKeys...), report)
validateRowsFile(path, obj, report, false) validateRowsFile(path, obj, report, false)
return return
} }
@@ -532,7 +516,6 @@ func datasetInvariantForPath(dataPath topdataDataPath) (datasetValidationInvaria
} }
func validateDatasetInvariantBaseFile(path string, obj map[string]any, invariant datasetValidationInvariant, report *ValidationReport) { func validateDatasetInvariantBaseFile(path string, obj map[string]any, invariant datasetValidationInvariant, report *ValidationReport) {
reportTopdataContractError(path, unsupportedObjectKeysError("base.json root", obj, baseOrPlainRootKeys...), report)
validateRowsFile(path, obj, report, true) validateRowsFile(path, obj, report, true)
validateDerivedBaseOutput(path, obj, invariant.Name, report) validateDerivedBaseOutput(path, obj, invariant.Name, report)
validateRequiredColumns(path, obj, invariant, report) validateRequiredColumns(path, obj, invariant, report)
@@ -843,10 +826,6 @@ func validateLockObject(path string, obj map[string]any, report *ValidationRepor
} }
func validateRowsFile(path string, obj map[string]any, report *ValidationReport, requireColumns bool) { func validateRowsFile(path string, obj map[string]any, report *ValidationReport, requireColumns bool) {
validateRowsFileWithColumns(path, obj, report, requireColumns, true)
}
func validateRowsFileWithColumns(path string, obj map[string]any, report *ValidationReport, requireColumns bool, validateColumns bool) {
rows, ok := obj["rows"] rows, ok := obj["rows"]
if !ok { if !ok {
report.Diagnostics = append(report.Diagnostics, Diagnostic{ report.Diagnostics = append(report.Diagnostics, Diagnostic{
@@ -866,7 +845,7 @@ func validateRowsFileWithColumns(path string, obj map[string]any, report *Valida
if rowsList, ok := rows.([]any); ok { if rowsList, ok := rows.([]any); ok {
validateRowCollection(path, obj, rowsList, report) validateRowCollection(path, obj, rowsList, report)
} }
if _, ok := obj["columns"]; ok && validateColumns { if _, ok := obj["columns"]; ok {
validateColumnsFile(path, obj, report) validateColumnsFile(path, obj, report)
} else if requireColumns { } else if requireColumns {
report.Diagnostics = append(report.Diagnostics, Diagnostic{ report.Diagnostics = append(report.Diagnostics, Diagnostic{
@@ -917,10 +896,6 @@ func validateColumnsFile(path string, obj map[string]any, report *ValidationRepo
} }
func validateEntriesFile(path string, obj map[string]any, report *ValidationReport) { func validateEntriesFile(path string, obj map[string]any, report *ValidationReport) {
validateEntriesFileWithColumns(path, obj, report, true)
}
func validateEntriesFileWithColumns(path string, obj map[string]any, report *ValidationReport, validateColumns bool) {
entries, ok := obj["entries"] entries, ok := obj["entries"]
if !ok { if !ok {
report.Diagnostics = append(report.Diagnostics, Diagnostic{ report.Diagnostics = append(report.Diagnostics, Diagnostic{
@@ -956,16 +931,12 @@ func validateEntriesFileWithColumns(path string, obj map[string]any, report *Val
validateInlineTextUsage(path, key, entry, report) validateInlineTextUsage(path, key, entry, report)
} }
} }
if _, ok := obj["columns"]; ok && validateColumns { if _, ok := obj["columns"]; ok {
validateColumnsFile(path, obj, report) validateColumnsFile(path, obj, report)
} }
} }
func validateOverridesFile(path string, obj map[string]any, report *ValidationReport) { func validateOverridesFile(path string, obj map[string]any, report *ValidationReport) {
validateOverridesFileWithColumns(path, obj, report, true)
}
func validateOverridesFileWithColumns(path string, obj map[string]any, report *ValidationReport, validateColumns bool) {
overrides, ok := obj["overrides"] overrides, ok := obj["overrides"]
if !ok { if !ok {
report.Diagnostics = append(report.Diagnostics, Diagnostic{ report.Diagnostics = append(report.Diagnostics, Diagnostic{
@@ -986,21 +957,20 @@ func validateOverridesFileWithColumns(path string, obj map[string]any, report *V
container := map[string]any{"rows": overrideList} container := map[string]any{"rows": overrideList}
validateRowCollection(path, container, overrideList, report) validateRowCollection(path, container, overrideList, report)
} }
if _, ok := obj["columns"]; ok && validateColumns { if _, ok := obj["columns"]; ok {
validateColumnsFile(path, obj, report) validateColumnsFile(path, obj, report)
} }
} }
func validateGlobalJSONFile(path string, obj map[string]any, report *ValidationReport) { func validateGlobalJSONFile(path string, obj map[string]any, report *ValidationReport) {
reportTopdataContractError(path, unsupportedObjectKeysError("global.json root", obj, globalRootKeys...), report)
if _, ok := obj["columns"]; ok { if _, ok := obj["columns"]; ok {
validateColumnsFile(path, obj, report) validateColumnsFile(path, obj, report)
} }
if _, ok := obj["entries"]; ok { if _, ok := obj["entries"]; ok {
validateEntriesFileWithColumns(path, obj, report, false) validateEntriesFile(path, obj, report)
} }
if _, ok := obj["overrides"]; ok { if _, ok := obj["overrides"]; ok {
validateOverridesFileWithColumns(path, obj, report, false) validateOverridesFile(path, obj, report)
} }
if _, ok := obj["defaults"]; ok { if _, ok := obj["defaults"]; ok {
validateGlobalDefaults(path, obj, report) validateGlobalDefaults(path, obj, report)
@@ -1059,12 +1029,8 @@ func validateGlobalJSONFile(path string, obj map[string]any, report *ValidationR
} else { } else {
rows = append(rows, row) rows = append(rows, row)
} }
_, _ = parseGlobalConditionGroups(injection, globalConditionParseOptions{ validateGlobalConditionList(path, fmt.Sprintf("global injection %d require_present", index), injection["require_present"], report)
Label: fmt.Sprintf("global injection %d", index), validateGlobalConditionList(path, fmt.Sprintf("global injection %d unless_present", index), injection["unless_present"], report)
Report: func(err error) {
reportTopdataContractError(path, err, report)
},
})
} }
validateRowCollection(path, obj, rows, report) validateRowCollection(path, obj, rows, report)
} }
@@ -1133,7 +1099,6 @@ func validateGlobalDefaults(path string, obj map[string]any, report *ValidationR
}) })
continue continue
} }
reportTopdataContractError(path, unsupportedObjectKeysError(fmt.Sprintf("global default %d", index), defaultRule, globalDefaultRuleKeys...), report)
validateGlobalDefaultMatch(path, index, defaultRule["match"], report) validateGlobalDefaultMatch(path, index, defaultRule["match"], report)
rawValues, ok := defaultRule["values"] rawValues, ok := defaultRule["values"]
if !ok { if !ok {
@@ -1190,13 +1155,15 @@ func validateGlobalDefaultMatch(path string, index int, rawMatch any, report *Va
}) })
return return
} }
for _, key := range unsupportedObjectKeys(match, globalDefaultMatchKeys...) { for key := range match {
if key != "source" {
report.Diagnostics = append(report.Diagnostics, Diagnostic{ report.Diagnostics = append(report.Diagnostics, Diagnostic{
Severity: SeverityError, Severity: SeverityError,
Path: path, Path: path,
Message: fmt.Sprintf("global default %d match.%s is not supported (unsupported key; supported keys: %s)", index, key, strings.Join(globalDefaultMatchKeys, ", ")), Message: fmt.Sprintf("global default %d match.%s is not supported", index, key),
}) })
} }
}
source, ok := match["source"].(string) source, ok := match["source"].(string)
if !ok || strings.TrimSpace(source) == "" { if !ok || strings.TrimSpace(source) == "" {
report.Diagnostics = append(report.Diagnostics, Diagnostic{ report.Diagnostics = append(report.Diagnostics, Diagnostic{
@@ -1226,7 +1193,14 @@ func validateGlobalDefaultValue(path string, index int, field string, value any,
if !hasFormat { if !hasFormat {
return return
} }
reportTopdataContractError(path, unsupportedObjectKeysError(fmt.Sprintf("global default %d values.%s", index, field), obj, globalDefaultFormatKeys...), report) if len(obj) != 1 {
report.Diagnostics = append(report.Diagnostics, Diagnostic{
Severity: SeverityError,
Path: path,
Message: fmt.Sprintf("global default %d values.%s format object must contain only format", index, field),
})
return
}
format, ok := rawFormat.(string) format, ok := rawFormat.(string)
if !ok { if !ok {
report.Diagnostics = append(report.Diagnostics, Diagnostic{ report.Diagnostics = append(report.Diagnostics, Diagnostic{
@@ -1246,6 +1220,46 @@ func validateGlobalDefaultValue(path string, index int, field string, value any,
} }
} }
func validateGlobalConditionList(path, label string, raw any, report *ValidationReport) {
if raw == nil {
return
}
conditions, ok := raw.([]any)
if !ok {
report.Diagnostics = append(report.Diagnostics, Diagnostic{
Severity: SeverityError,
Path: path,
Message: fmt.Sprintf("%s must be a JSON array", label),
})
return
}
for index, rawCondition := range conditions {
condition, ok := rawCondition.(map[string]any)
if !ok {
report.Diagnostics = append(report.Diagnostics, Diagnostic{
Severity: SeverityError,
Path: path,
Message: fmt.Sprintf("%s[%d] must be an object", label, index),
})
continue
}
if field, ok := condition["field"].(string); !ok || strings.TrimSpace(field) == "" {
report.Diagnostics = append(report.Diagnostics, Diagnostic{
Severity: SeverityError,
Path: path,
Message: fmt.Sprintf("%s[%d].field is required", label, index),
})
}
if id, ok := condition["id"].(string); !ok || strings.TrimSpace(id) == "" {
report.Diagnostics = append(report.Diagnostics, Diagnostic{
Severity: SeverityError,
Path: path,
Message: fmt.Sprintf("%s[%d].id is required", label, index),
})
}
}
}
func validateStateObject(path string, obj map[string]any, report *ValidationReport) { func validateStateObject(path string, obj map[string]any, report *ValidationReport) {
entries, ok := obj["entries"].(map[string]any) entries, ok := obj["entries"].(map[string]any)
if !ok { if !ok {