Compare commits

..
Author SHA1 Message Date
archvillainette 2d184bf686 docs: plan topdata JSON validation tightening
build-binaries / build-binaries (pull_request) Successful in 2m12s
test-image / build-image (pull_request) Successful in 37s
test / test (pull_request) Successful in 1m23s
2026-06-25 21:54:08 +02:00
archvillainette ba76278983 docs: specify topdata JSON validation tightening 2026-06-25 21:34:42 +02:00
archvillainette da6bf17f69 fix: stdenv system error
test / test (push) Successful in 1m22s
build-binaries / build-binaries (push) Successful in 2m11s
2026-06-25 12:14:47 +02:00
archvillainette 396236f242 stop drift checks on prs (#22)
build-binaries / build-binaries (push) Successful in 2m6s
test / test (push) Successful in 1m21s
Reviewed-on: #22
Co-authored-by: vickydotbat <vickydotbat@tutamail.com>
Co-committed-by: vickydotbat <vickydotbat@tutamail.com>
2026-06-25 09:58:59 +00:00
archvillainette f257672427 command and help ux pass (#21)
sync-wrappers / sync (push) Successful in 16s
test / test (push) Successful in 1m22s
build-binaries / build-binaries (push) Successful in 2m7s
build-image / publish (push) Successful in 38s
Reviewed-on: #21
Co-authored-by: vickydotbat <vickydotbat@tutamail.com>
Co-committed-by: vickydotbat <vickydotbat@tutamail.com>
2026-06-25 09:29:39 +00:00
archvillainette 6afac1a4d9 chore: register sow-assets-manifest as a wrapper consumer (#20)
test / test (push) Successful in 1m25s
build-binaries / build-binaries (push) Successful in 2m9s
build-image / publish (push) Successful in 55s
It vendors crucible.sh/.ps1 for off-Nix local dev and now carries the
drift-check workflow, so sync-wrappers should open update PRs to it too.

Requires: grant the gitea-bot org token write access to
ShadowsOverWestgate/sow-assets-manifest (per the header note in this file).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Reviewed-on: #20
Reviewed-by: xtul <mpiasecki720@protonmail.com>
Co-authored-by: vickydotbat <vickydotbat@tutamail.com>
Co-committed-by: vickydotbat <vickydotbat@tutamail.com>
2026-06-25 08:34:35 +00:00
archvillainetteandarchvillainette 79595e55be chore: add direnv .envrc for per-repo nix shell (#19)
test / test (push) Successful in 1m24s
build-binaries / build-binaries (push) Successful in 2m8s
Auto-loads the repo's own flake devshell on `cd` via direnv. No-op for non-Nix users (direnv simply isn't installed). `.direnv/` eval cache gitignored.

Part of dropping the redundant root migration/ flake aggregator (its non-git path: input copied the whole tree and broke on the local postgres data dir). Each repo's own git-scoped flake is the source of truth now.

Co-authored-by: archvillainette <zoelynne.victoria@gmail.com>
Reviewed-on: #19
2026-06-25 07:00:50 +00:00
archvillainette eefe00ebc5 chore: add tea to flake (#18)
build-binaries / build-binaries (push) Has been cancelled
test / test (push) Has been cancelled
Reviewed-on: #18
Co-authored-by: vickydotbat <vickydotbat@tutamail.com>
Co-committed-by: vickydotbat <vickydotbat@tutamail.com>
2026-06-25 06:55:32 +00:00
archvillainette eb5d15061c fix(pipeline): make the full module build work for module-only projects (#17)
test / test (push) Successful in 1m23s
build-binaries / build-binaries (push) Successful in 2m9s
build-image / publish (push) Successful in 52s
`crucible module build` runs the full pipeline (module + HAK producer +
topdata). A module-only project consumes prebuilt HAKs and has no
`paths.assets`, so the HAK stage crashed on the unset assets dir: the
music-credits walk hit `filepath.WalkDir("")` (`lstat : no such file or
directory`) and asset collection hit `filepath.Rel(root, "")`.

Guard both on an unset `AssetsDir()`: an absent assets tree simply means
no authored credits and no asset files to collect, so the HAK stage
no-ops cleanly while topdata generated-2DA assets still build. Honors
spec Decision 3 (canonical `crucible module build`, no `build-module`).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Reviewed-on: #17
Co-authored-by: vickydotbat <vickydotbat@tutamail.com>
Co-committed-by: vickydotbat <vickydotbat@tutamail.com>
2026-06-24 15:32:01 +00:00
archvillainette 5c46824ebd fix depot cdn url (#16)
build-binaries / build-binaries (push) Successful in 2m10s
test / test (push) Successful in 1m23s
build-image / publish (push) Successful in 55s
Reviewed-on: #16
Co-authored-by: vickydotbat <vickydotbat@tutamail.com>
Co-committed-by: vickydotbat <vickydotbat@tutamail.com>
2026-06-22 12:34:12 +00:00
archvillainette 2ea7959693 paths.build fix (#15)
test / test (push) Successful in 1m27s
build-binaries / build-binaries (push) Successful in 2m8s
build-image / publish (push) Successful in 52s
Reviewed-on: #15
Co-authored-by: vickydotbat <vickydotbat@tutamail.com>
Co-committed-by: vickydotbat <vickydotbat@tutamail.com>
2026-06-21 18:33:45 +00:00
archvillainette 3315f8b7eb crucible build parity (#14)
build-binaries / build-binaries (push) Successful in 2m16s
test / test (push) Successful in 1m31s
build-image / publish (push) Successful in 57s
forces build parity with crucible = local builds and CI/CD builds use different tools.

Reviewed-on: #14
Co-authored-by: vickydotbat <vickydotbat@tutamail.com>
Co-committed-by: vickydotbat <vickydotbat@tutamail.com>
2026-06-21 13:17:58 +00:00
archvillainette faee2cde95 fix failing checks + cosmetic renames (#13)
test / test (push) Successful in 1m23s
build-binaries / build-binaries (push) Successful in 2m7s
build-image / publish (push) Successful in 52s
Reviewed-on: #13
Co-authored-by: vickydotbat <vickydotbat@tutamail.com>
Co-committed-by: vickydotbat <vickydotbat@tutamail.com>
2026-06-21 07:40:46 +00:00
archvillainette d2f7c420ba autogen activation (#12)
test / test (push) Successful in 1m25s
build-binaries / build-binaries (push) Successful in 2m14s
build-image / publish (push) Successful in 1m12s
re-activates VFX accessory autogen

Reviewed-on: #12
Reviewed-by: xtul <mpiasecki720@protonmail.com>
Co-authored-by: vickydotbat <vickydotbat@tutamail.com>
Co-committed-by: vickydotbat <vickydotbat@tutamail.com>
2026-06-21 07:16:58 +00:00
archvillainette 97f4c00393 bot token (#11)
build-binaries / build-binaries (push) Successful in 2m8s
test / test (push) Successful in 1m23s
Reviewed-on: #11
Co-authored-by: vickydotbat <vickydotbat@tutamail.com>
Co-committed-by: vickydotbat <vickydotbat@tutamail.com>
2026-06-20 19:33:10 +00:00
archvillainette 4bab09412d workflow cleanups (#10)
build-binaries / build-binaries (push) Successful in 2m13s
test / test (push) Successful in 1m24s
Reviewed-on: #10
Co-authored-by: vickydotbat <vickydotbat@tutamail.com>
Co-committed-by: vickydotbat <vickydotbat@tutamail.com>
2026-06-20 15:41:33 +00:00
archvillainette 13c8ced5e8 clean brittle tests (#9)
build-binaries / build-binaries (push) Successful in 2m7s
test-image / build-image (push) Successful in 43s
test / test (push) Successful in 1m26s
Reviewed-on: #9
Reviewed-by: xtul <mpiasecki720@protonmail.com>
Co-authored-by: vickydotbat <vickydotbat@tutamail.com>
Co-committed-by: vickydotbat <vickydotbat@tutamail.com>
2026-06-20 10:32:20 +00:00
archvillainette f1fd03ee83 pipeline: reject resref collisions when chunking; document erf post-write hashing (#8)
test-image / build-image (push) Successful in 47s
test / test (push) Successful in 1m27s
build-binaries / build-binaries (push) Successful in 2m14s
build-image / publish (push) Successful in 13s
## What

Two correctness fixes surfaced in review of the direct-depot HAK artifact work.

### Reject resref+type collisions in `chunksFromManifest`

A manifest can name two distinct source paths that collapse to the same
`resref+type` (e.g. `creature/foo.tga` and `placeable/foo.tga` — resref is the
lowercase basename minus extension). The ERF writer keys resources on
`Name:Type`, so the second silently shadowed the first: an asset would vanish
from the packed HAK with no error.

`chunksFromManifest` now runs `ensureUniqueChunkResources` per chunk and fails
the build on a duplicate. This guards **both** build paths — the legacy
`--source-manifest` flow and the direct content-addressed flow
(`chunksFromSourceManifest` → `chunksFromManifest`).

### Document erf post-write hash coupling

`writeResourceData` streams the source into the output while hashing, so
size/SHA mismatches are only detected *after* the bytes are written. A non-nil
return therefore means the writer holds partial, unverified output and the
caller must discard it. Added a comment making that contract explicit;
`writeHAKArchive` already honours it (writes to a temp file, removes on any
Write error, never renames a bad archive into place).

## Tests

- `TestChunksFromManifestRejectsResrefCollision`: collision → error, distinct
  resrefs → clean. Asserts only error presence/absence — silent asset loss is
  the contract, not any specific wording.
- `go vet ./internal/erf/ ./internal/pipeline/` clean.
- `go test ./internal/erf/ ./internal/pipeline/` green.

## Follow-up

A new crucible release must be cut after this merges so the guard ships in the
binary `sow-assets-manifest` pins.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Reviewed-on: #8
Co-authored-by: vickydotbat <vickydotbat@tutamail.com>
Co-committed-by: vickydotbat <vickydotbat@tutamail.com>
2026-06-20 07:53:04 +00:00
archvillainette cdbbba3181 add content-addressed hak source mode (#7)
test-image / build-image (push) Successful in 46s
test / test (push) Successful in 1m26s
build-binaries / build-binaries (push) Successful in 2m20s
build-image / publish (push) Successful in 31s
Reviewed-on: #7
Co-authored-by: vickydotbat <vickydotbat@tutamail.com>
Co-committed-by: vickydotbat <vickydotbat@tutamail.com>
2026-06-19 20:56:36 +00:00
archvillainette 223b9831c5 Always Gitkeep Cache (#6)
build-binaries / build-binaries (push) Successful in 2m2s
test-image / build-image (push) Successful in 43s
test / test (push) Successful in 1m20s
Reviewed-on: #6
Co-authored-by: vickydotbat <vickydotbat@tutamail.com>
Co-committed-by: vickydotbat <vickydotbat@tutamail.com>
2026-06-18 07:29:09 +00:00
archvillainette 16d5586587 Crucible: Fix Formatting Churn (#5)
test-image / build-image (push) Successful in 43s
test / test (push) Successful in 1m21s
build-binaries / build-binaries (push) Successful in 2m2s
build-image / publish (push) Successful in 13s
RC#1 — build wrote 2-space instead of .editorconfig's 4-space
- editorconfig_format.go: the glob translator escaped {/} as literals, so [*.{json,jsonc}]→indent_size=4 never matched any file and formatting fell back to [*]→2. Implemented real editorconfig brace expansion — {a,b,c} alternation and {n..m} numeric ranges. Nothing is hardcoded; saveLockfile already read .editorconfig, it just got the wrong section.

RC#2 — validate wrote a lockfile (it must be read-only)
- The three registry collectors persisted lockfiles as a side effect of collection; ValidateProject calls collection outside its snapshot guard, so the write leaked. Threaded a persistLocks bool through collectGeneratedRegistryDatasets and the damagetypes/itemprops/racialtypes collectors. Only buildNativeUnchecked (the real build allocator) passes true; validate, discovery, packaging queries, and wiki discovery pass false.

Tests added: editorconfig_glob_test.go (brace match + 4-space via brace glob), registry_readonly_test.go (read-only writes nothing; persist writes allocations).

Reviewed-on: #5
Co-authored-by: vickydotbat <vickydotbat@tutamail.com>
Co-committed-by: vickydotbat <vickydotbat@tutamail.com>
2026-06-17 07:57:54 +00:00
archvillainette b7c0f43064 Fix autogen to fail-open on missing asset manifests (#4)
test-image / build-image (push) Successful in 45s
test / test (push) Successful in 1m22s
build-binaries / build-binaries (push) Successful in 2m2s
build-image / publish (push) Successful in 13s
Reviewed-on: #4
Co-authored-by: vickydotbat <vickydotbat@tutamail.com>
Co-committed-by: vickydotbat <vickydotbat@tutamail.com>
2026-06-16 21:50:45 +00:00
66 changed files with 7851 additions and 3514 deletions
View File
+1
View File
@@ -0,0 +1 @@
use flake
+18 -5
View File
@@ -1,7 +1,10 @@
# Auto-PR canonical wrapper updates to consumer repos when wrappers/ changes on # Auto-PR canonical wrapper updates to consumer repos when wrappers/ changes on
# main. Maintenance automation (not artifact publishing), so it is allowed on a # main. Maintenance automation (not artifact publishing), so it is allowed on a
# main push under the D7 trigger standard. Requires WRAPPER_SYNC_TOKEN: a bot # main push under the D7 trigger standard. Requires BOT_TOKEN: the gitea-bot
# user's token with content+PR write to the consumer repos (never committed). # org token (content+PR write to the consumer repos; never committed).
#
# Consumer drift checks run only after their sync PRs merge to main. They must
# not run on the sync PR itself, which can create recursive cross-repo checks.
name: sync-wrappers name: sync-wrappers
on: on:
@@ -19,12 +22,13 @@ jobs:
- name: Open sync PRs to consumers - name: Open sync PRs to consumers
env: env:
TOKEN: ${{ secrets.WRAPPER_SYNC_TOKEN }} TOKEN: ${{ secrets.BOT_TOKEN }}
SERVER: ${{ github.server_url }} SERVER: ${{ github.server_url }}
SRC_SHA: ${{ github.sha }} SRC_SHA: ${{ github.sha }}
run: | run: |
nix develop --command bash -c ' nix develop --command bash -c '
set -euo pipefail set -euo pipefail
[ -n "${TOKEN}" ] || { echo "::error::BOT_TOKEN secret is empty — set the gitea-bot org token"; exit 1; }
host="$(echo "$SERVER" | sed -E "s#https?://##")" host="$(echo "$SERVER" | sed -E "s#https?://##")"
branch="chore/sync-wrappers-$(echo "$SRC_SHA" | cut -c1-12)" branch="chore/sync-wrappers-$(echo "$SRC_SHA" | cut -c1-12)"
grep -vE "^\s*#|^\s*$" wrappers/consumers.txt | while read -r target; do grep -vE "^\s*#|^\s*$" wrappers/consumers.txt | while read -r target; do
@@ -40,10 +44,19 @@ jobs:
git add crucible.sh crucible.ps1 git add crucible.sh crucible.ps1
git commit -m "chore: sync crucible wrappers from sow-tools@${SRC_SHA}" git commit -m "chore: sync crucible wrappers from sow-tools@${SRC_SHA}"
git push -f origin "$branch" git push -f origin "$branch"
curl -fsS -X POST \ # Capture HTTP status: 201=created, 422=PR already open for this
# branch (fine, the force-push above refreshed it). Anything else
# (401/404/...) is a real failure — fail loud, do not swallow it.
resp="$(mktemp)"
code="$(curl -sS -o "$resp" -w "%{http_code}" -X POST \
-H "Authorization: token ${TOKEN}" -H "Content-Type: application/json" \ -H "Authorization: token ${TOKEN}" -H "Content-Type: application/json" \
"${SERVER}/api/v1/repos/${target}/pulls" \ "${SERVER}/api/v1/repos/${target}/pulls" \
-d "{\"head\":\"${branch}\",\"base\":\"main\",\"title\":\"chore: sync crucible wrappers from sow-tools\"}" || true -d "{\"head\":\"${branch}\",\"base\":\"main\",\"title\":\"chore: sync crucible wrappers from sow-tools\"}")"
case "$code" in
201) echo "opened sync PR for $target" ;;
422) echo "sync PR already open for $target; refreshed its branch" ;;
*) echo "::error::PR create failed for $target (HTTP $code)"; cat "$resp"; exit 1 ;;
esac
) )
rm -rf "$work" rm -rf "$work"
done done
+5 -5
View File
@@ -1,11 +1,11 @@
# Test-build the Crucible image on PRs and main. Proves `nix build .#image` # Test-build the Crucible image on PRs. Proves `nix build .#image` still works
# still works (daemonless, Nix-built OCI tarball) but does NOT publish — # (daemonless, Nix-built OCI tarball) but does NOT publish — release publishing
# release publishing happens in build-image.yml on v* tags. # happens in build-image.yml on v* tags. PR-only: with up-to-date-before-merge
# protection, main == the tested PR head, so a throwaway post-merge rebuild that
# publishes nothing is pure waste.
name: test-image name: test-image
on: on:
push:
branches: [main]
pull_request: pull_request:
jobs: jobs:
+3 -2
View File
@@ -6,8 +6,8 @@ nwn-tool
sow-toolkit sow-toolkit
# Go / build cache # Go / build cache
.cache/ .cache/*
.cache/** !.cache/.gitkeep
# nix build symlink # nix build symlink
result result
@@ -18,3 +18,4 @@ result-*
*.swp *.swp
.idea/ .idea/
.vscode/ .vscode/
.direnv/
+11 -23
View File
@@ -5,34 +5,25 @@ alwaysApply: true
# sow-tools / Crucible Agent Guide # sow-tools / Crucible Agent Guide
This repo owns the **builder logic** for the migration: one Go module This repo owns the **builder logic:** one Go module
(`git.westgate.pw/ShadowsOverWestgate/sow-tools`) producing the `crucible` (`git.westgate.pw/ShadowsOverWestgate/sow-tools`) producing the `crucible`
dispatcher and the `crucible-<name>` binaries (D11). Read dispatcher and the `crucible-<name>` binaries.
`../AGENTS.md` (migration hub) and `../../KICKOFF_PROMPT.md` first.
## What this repo owns / does not own ## What this repo owns / does not own
Owns: build/extract/validate/compare pipeline, ERF/HAK packing, topdata 2da/tlk Owns: build/extract/validate/compare pipeline, ERF/HAK packing, topdata 2da/tlk
compilation, wiki rendering/deploy, depot blob verify, music conversion, compilation, wiki rendering/deploy, depot blob verify, changelog. Does **not**
changelog. Does **not** own authored game content (that is `sow-module` / own authored game content (that is `sow-module` /
`sow-topdata` / `sow-assets-manifest`) or any production deploy authority (that `sow-topdata` / `sow-assets-manifest`) or any production deploy authority (that
is `sow-platform`). is `sow-platform`).
## Rules ## Rules
1. **Migrated logic, not a fresh rewrite.** The `internal/` packages (`app`, 1. **Fail closed, never fake.** A builder with no migrated logic yet (`depot`)
`pipeline`, `project`, `erf`, `gff`, `topdata`, `music`, `changelog`,
`validator`) were folded in from `gitea/sow-tools` at cutover; wired builders
delegate to `internal/app`'s command surface. Keep them in step with upstream
fixes rather than diverging silently.
2. **Fail closed, never fake.** A builder with no migrated logic yet (`depot`)
exits `70`. Do not stub a builder to emit a placeholder artifact. exits `70`. Do not stub a builder to emit a placeholder artifact.
3. **Binaries are not committed.** They are CI artifacts / image layers (D19). 2. **Binaries are not committed.** They are CI artifacts / image layers.
`/bin/`, `*.exe`, `nwn-tool`, `sow-toolkit` are gitignored. `/bin/`, `*.exe`, `nwn-tool`, `sow-toolkit` are gitignored.
4. **No home-dir / `NWN_ROOT` guessing.** Builders take roots explicitly via 3. **The registry is the command surface.** `internal/dispatch.Registry` is the
flag or env (project resolution is CWD-based, never `$HOME`). See
[`docs/consumer-contract.md`](docs/consumer-contract.md).
5. **The registry is the command surface.** `internal/dispatch.Registry` is the
single source of truth; keep it in sync with `cmd/` and single source of truth; keep it in sync with `cmd/` and
[`docs/command-surface.md`](docs/command-surface.md). Adding a builder = a [`docs/command-surface.md`](docs/command-surface.md). Adding a builder = a
`cmd/crucible-<name>/main.go` shim + a `Registry` entry + a doc row. `cmd/crucible-<name>/main.go` shim + a `Registry` entry + a doc row.
@@ -40,11 +31,8 @@ is `sow-platform`).
## Wiring a builder ## Wiring a builder
1. Ensure the relevant `internal/` package(s) cover the work. 1. Ensure the relevant `internal/` package(s) cover the work.
2. Add the legacy command(s) to the builder's `Legacy`/`Extra` set and set 2. Add tests; keep outputs deterministic (same input → same bytes).
`Wired: true` in `internal/dispatch`; the dispatcher delegates to 3. `make check` must stay green; update `make smoke` to expect the wired exit.
`app.Run`. `depot` is the remaining unwired builder.
3. Add tests; keep outputs deterministic (same input → same bytes).
4. `make check` must stay green; update `make smoke` to expect the wired exit.
## Commands ## Commands
@@ -55,6 +43,6 @@ make smoke # assert fail-closed contract
make image # crucible:<sha> make image # crucible:<sha>
``` ```
## Git ## Tests
Never commit, branch, or push. Suggest a commit message; let the operator do it. Tests must survive harmless changes to constants, defaults, wording, ordering, fixture data, and internal implementation details. A test that fails merely because a basic value changed is usually a bad test. Only assert exact values when the value is part of a documented public contract, external protocol, compatibility requirement, security rule, migration, or business rule.
+674
View File
@@ -0,0 +1,674 @@
GNU GENERAL PUBLIC LICENSE
Version 3, 29 June 2007
Copyright (C) 2007 Free Software Foundation, Inc. <https://fsf.org/>
Everyone is permitted to copy and distribute verbatim copies
of this license document, but changing it is not allowed.
Preamble
The GNU General Public License is a free, copyleft license for
software and other kinds of works.
The licenses for most software and other practical works are designed
to take away your freedom to share and change the works. By contrast,
the GNU General Public License is intended to guarantee your freedom to
share and change all versions of a program--to make sure it remains free
software for all its users. We, the Free Software Foundation, use the
GNU General Public License for most of our software; it applies also to
any other work released this way by its authors. You can apply it to
your programs, too.
When we speak of free software, we are referring to freedom, not
price. Our General Public Licenses are designed to make sure that you
have the freedom to distribute copies of free software (and charge for
them if you wish), that you receive source code or can get it if you
want it, that you can change the software or use pieces of it in new
free programs, and that you know you can do these things.
To protect your rights, we need to prevent others from denying you
these rights or asking you to surrender the rights. Therefore, you have
certain responsibilities if you distribute copies of the software, or if
you modify it: responsibilities to respect the freedom of others.
For example, if you distribute copies of such a program, whether
gratis or for a fee, you must pass on to the recipients the same
freedoms that you received. You must make sure that they, too, receive
or can get the source code. And you must show them these terms so they
know their rights.
Developers that use the GNU GPL protect your rights with two steps:
(1) assert copyright on the software, and (2) offer you this License
giving you legal permission to copy, distribute and/or modify it.
For the developers' and authors' protection, the GPL clearly explains
that there is no warranty for this free software. For both users' and
authors' sake, the GPL requires that modified versions be marked as
changed, so that their problems will not be attributed erroneously to
authors of previous versions.
Some devices are designed to deny users access to install or run
modified versions of the software inside them, although the manufacturer
can do so. This is fundamentally incompatible with the aim of
protecting users' freedom to change the software. The systematic
pattern of such abuse occurs in the area of products for individuals to
use, which is precisely where it is most unacceptable. Therefore, we
have designed this version of the GPL to prohibit the practice for those
products. If such problems arise substantially in other domains, we
stand ready to extend this provision to those domains in future versions
of the GPL, as needed to protect the freedom of users.
Finally, every program is threatened constantly by software patents.
States should not allow patents to restrict development and use of
software on general-purpose computers, but in those that do, we wish to
avoid the special danger that patents applied to a free program could
make it effectively proprietary. To prevent this, the GPL assures that
patents cannot be used to render the program non-free.
The precise terms and conditions for copying, distribution and
modification follow.
TERMS AND CONDITIONS
0. Definitions.
"This License" refers to version 3 of the GNU General Public License.
"Copyright" also means copyright-like laws that apply to other kinds of
works, such as semiconductor masks.
"The Program" refers to any copyrightable work licensed under this
License. Each licensee is addressed as "you". "Licensees" and
"recipients" may be individuals or organizations.
To "modify" a work means to copy from or adapt all or part of the work
in a fashion requiring copyright permission, other than the making of an
exact copy. The resulting work is called a "modified version" of the
earlier work or a work "based on" the earlier work.
A "covered work" means either the unmodified Program or a work based
on the Program.
To "propagate" a work means to do anything with it that, without
permission, would make you directly or secondarily liable for
infringement under applicable copyright law, except executing it on a
computer or modifying a private copy. Propagation includes copying,
distribution (with or without modification), making available to the
public, and in some countries other activities as well.
To "convey" a work means any kind of propagation that enables other
parties to make or receive copies. Mere interaction with a user through
a computer network, with no transfer of a copy, is not conveying.
An interactive user interface displays "Appropriate Legal Notices"
to the extent that it includes a convenient and prominently visible
feature that (1) displays an appropriate copyright notice, and (2)
tells the user that there is no warranty for the work (except to the
extent that warranties are provided), that licensees may convey the
work under this License, and how to view a copy of this License. If
the interface presents a list of user commands or options, such as a
menu, a prominent item in the list meets this criterion.
1. Source Code.
The "source code" for a work means the preferred form of the work
for making modifications to it. "Object code" means any non-source
form of a work.
A "Standard Interface" means an interface that either is an official
standard defined by a recognized standards body, or, in the case of
interfaces specified for a particular programming language, one that
is widely used among developers working in that language.
The "System Libraries" of an executable work include anything, other
than the work as a whole, that (a) is included in the normal form of
packaging a Major Component, but which is not part of that Major
Component, and (b) serves only to enable use of the work with that
Major Component, or to implement a Standard Interface for which an
implementation is available to the public in source code form. A
"Major Component", in this context, means a major essential component
(kernel, window system, and so on) of the specific operating system
(if any) on which the executable work runs, or a compiler used to
produce the work, or an object code interpreter used to run it.
The "Corresponding Source" for a work in object code form means all
the source code needed to generate, install, and (for an executable
work) run the object code and to modify the work, including scripts to
control those activities. However, it does not include the work's
System Libraries, or general-purpose tools or generally available free
programs which are used unmodified in performing those activities but
which are not part of the work. For example, Corresponding Source
includes interface definition files associated with source files for
the work, and the source code for shared libraries and dynamically
linked subprograms that the work is specifically designed to require,
such as by intimate data communication or control flow between those
subprograms and other parts of the work.
The Corresponding Source need not include anything that users
can regenerate automatically from other parts of the Corresponding
Source.
The Corresponding Source for a work in source code form is that
same work.
2. Basic Permissions.
All rights granted under this License are granted for the term of
copyright on the Program, and are irrevocable provided the stated
conditions are met. This License explicitly affirms your unlimited
permission to run the unmodified Program. The output from running a
covered work is covered by this License only if the output, given its
content, constitutes a covered work. This License acknowledges your
rights of fair use or other equivalent, as provided by copyright law.
You may make, run and propagate covered works that you do not
convey, without conditions so long as your license otherwise remains
in force. You may convey covered works to others for the sole purpose
of having them make modifications exclusively for you, or provide you
with facilities for running those works, provided that you comply with
the terms of this License in conveying all material for which you do
not control copyright. Those thus making or running the covered works
for you must do so exclusively on your behalf, under your direction
and control, on terms that prohibit them from making any copies of
your copyrighted material outside their relationship with you.
Conveying under any other circumstances is permitted solely under
the conditions stated below. Sublicensing is not allowed; section 10
makes it unnecessary.
3. Protecting Users' Legal Rights From Anti-Circumvention Law.
No covered work shall be deemed part of an effective technological
measure under any applicable law fulfilling obligations under article
11 of the WIPO copyright treaty adopted on 20 December 1996, or
similar laws prohibiting or restricting circumvention of such
measures.
When you convey a covered work, you waive any legal power to forbid
circumvention of technological measures to the extent such circumvention
is effected by exercising rights under this License with respect to
the covered work, and you disclaim any intention to limit operation or
modification of the work as a means of enforcing, against the work's
users, your or third parties' legal rights to forbid circumvention of
technological measures.
4. Conveying Verbatim Copies.
You may convey verbatim copies of the Program's source code as you
receive it, in any medium, provided that you conspicuously and
appropriately publish on each copy an appropriate copyright notice;
keep intact all notices stating that this License and any
non-permissive terms added in accord with section 7 apply to the code;
keep intact all notices of the absence of any warranty; and give all
recipients a copy of this License along with the Program.
You may charge any price or no price for each copy that you convey,
and you may offer support or warranty protection for a fee.
5. Conveying Modified Source Versions.
You may convey a work based on the Program, or the modifications to
produce it from the Program, in the form of source code under the
terms of section 4, provided that you also meet all of these conditions:
a) The work must carry prominent notices stating that you modified
it, and giving a relevant date.
b) The work must carry prominent notices stating that it is
released under this License and any conditions added under section
7. This requirement modifies the requirement in section 4 to
"keep intact all notices".
c) You must license the entire work, as a whole, under this
License to anyone who comes into possession of a copy. This
License will therefore apply, along with any applicable section 7
additional terms, to the whole of the work, and all its parts,
regardless of how they are packaged. This License gives no
permission to license the work in any other way, but it does not
invalidate such permission if you have separately received it.
d) If the work has interactive user interfaces, each must display
Appropriate Legal Notices; however, if the Program has interactive
interfaces that do not display Appropriate Legal Notices, your
work need not make them do so.
A compilation of a covered work with other separate and independent
works, which are not by their nature extensions of the covered work,
and which are not combined with it such as to form a larger program,
in or on a volume of a storage or distribution medium, is called an
"aggregate" if the compilation and its resulting copyright are not
used to limit the access or legal rights of the compilation's users
beyond what the individual works permit. Inclusion of a covered work
in an aggregate does not cause this License to apply to the other
parts of the aggregate.
6. Conveying Non-Source Forms.
You may convey a covered work in object code form under the terms
of sections 4 and 5, provided that you also convey the
machine-readable Corresponding Source under the terms of this License,
in one of these ways:
a) Convey the object code in, or embodied in, a physical product
(including a physical distribution medium), accompanied by the
Corresponding Source fixed on a durable physical medium
customarily used for software interchange.
b) Convey the object code in, or embodied in, a physical product
(including a physical distribution medium), accompanied by a
written offer, valid for at least three years and valid for as
long as you offer spare parts or customer support for that product
model, to give anyone who possesses the object code either (1) a
copy of the Corresponding Source for all the software in the
product that is covered by this License, on a durable physical
medium customarily used for software interchange, for a price no
more than your reasonable cost of physically performing this
conveying of source, or (2) access to copy the
Corresponding Source from a network server at no charge.
c) Convey individual copies of the object code with a copy of the
written offer to provide the Corresponding Source. This
alternative is allowed only occasionally and noncommercially, and
only if you received the object code with such an offer, in accord
with subsection 6b.
d) Convey the object code by offering access from a designated
place (gratis or for a charge), and offer equivalent access to the
Corresponding Source in the same way through the same place at no
further charge. You need not require recipients to copy the
Corresponding Source along with the object code. If the place to
copy the object code is a network server, the Corresponding Source
may be on a different server (operated by you or a third party)
that supports equivalent copying facilities, provided you maintain
clear directions next to the object code saying where to find the
Corresponding Source. Regardless of what server hosts the
Corresponding Source, you remain obligated to ensure that it is
available for as long as needed to satisfy these requirements.
e) Convey the object code using peer-to-peer transmission, provided
you inform other peers where the object code and Corresponding
Source of the work are being offered to the general public at no
charge under subsection 6d.
A separable portion of the object code, whose source code is excluded
from the Corresponding Source as a System Library, need not be
included in conveying the object code work.
A "User Product" is either (1) a "consumer product", which means any
tangible personal property which is normally used for personal, family,
or household purposes, or (2) anything designed or sold for incorporation
into a dwelling. In determining whether a product is a consumer product,
doubtful cases shall be resolved in favor of coverage. For a particular
product received by a particular user, "normally used" refers to a
typical or common use of that class of product, regardless of the status
of the particular user or of the way in which the particular user
actually uses, or expects or is expected to use, the product. A product
is a consumer product regardless of whether the product has substantial
commercial, industrial or non-consumer uses, unless such uses represent
the only significant mode of use of the product.
"Installation Information" for a User Product means any methods,
procedures, authorization keys, or other information required to install
and execute modified versions of a covered work in that User Product from
a modified version of its Corresponding Source. The information must
suffice to ensure that the continued functioning of the modified object
code is in no case prevented or interfered with solely because
modification has been made.
If you convey an object code work under this section in, or with, or
specifically for use in, a User Product, and the conveying occurs as
part of a transaction in which the right of possession and use of the
User Product is transferred to the recipient in perpetuity or for a
fixed term (regardless of how the transaction is characterized), the
Corresponding Source conveyed under this section must be accompanied
by the Installation Information. But this requirement does not apply
if neither you nor any third party retains the ability to install
modified object code on the User Product (for example, the work has
been installed in ROM).
The requirement to provide Installation Information does not include a
requirement to continue to provide support service, warranty, or updates
for a work that has been modified or installed by the recipient, or for
the User Product in which it has been modified or installed. Access to a
network may be denied when the modification itself materially and
adversely affects the operation of the network or violates the rules and
protocols for communication across the network.
Corresponding Source conveyed, and Installation Information provided,
in accord with this section must be in a format that is publicly
documented (and with an implementation available to the public in
source code form), and must require no special password or key for
unpacking, reading or copying.
7. Additional Terms.
"Additional permissions" are terms that supplement the terms of this
License by making exceptions from one or more of its conditions.
Additional permissions that are applicable to the entire Program shall
be treated as though they were included in this License, to the extent
that they are valid under applicable law. If additional permissions
apply only to part of the Program, that part may be used separately
under those permissions, but the entire Program remains governed by
this License without regard to the additional permissions.
When you convey a copy of a covered work, you may at your option
remove any additional permissions from that copy, or from any part of
it. (Additional permissions may be written to require their own
removal in certain cases when you modify the work.) You may place
additional permissions on material, added by you to a covered work,
for which you have or can give appropriate copyright permission.
Notwithstanding any other provision of this License, for material you
add to a covered work, you may (if authorized by the copyright holders of
that material) supplement the terms of this License with terms:
a) Disclaiming warranty or limiting liability differently from the
terms of sections 15 and 16 of this License; or
b) Requiring preservation of specified reasonable legal notices or
author attributions in that material or in the Appropriate Legal
Notices displayed by works containing it; or
c) Prohibiting misrepresentation of the origin of that material, or
requiring that modified versions of such material be marked in
reasonable ways as different from the original version; or
d) Limiting the use for publicity purposes of names of licensors or
authors of the material; or
e) Declining to grant rights under trademark law for use of some
trade names, trademarks, or service marks; or
f) Requiring indemnification of licensors and authors of that
material by anyone who conveys the material (or modified versions of
it) with contractual assumptions of liability to the recipient, for
any liability that these contractual assumptions directly impose on
those licensors and authors.
All other non-permissive additional terms are considered "further
restrictions" within the meaning of section 10. If the Program as you
received it, or any part of it, contains a notice stating that it is
governed by this License along with a term that is a further
restriction, you may remove that term. If a license document contains
a further restriction but permits relicensing or conveying under this
License, you may add to a covered work material governed by the terms
of that license document, provided that the further restriction does
not survive such relicensing or conveying.
If you add terms to a covered work in accord with this section, you
must place, in the relevant source files, a statement of the
additional terms that apply to those files, or a notice indicating
where to find the applicable terms.
Additional terms, permissive or non-permissive, may be stated in the
form of a separately written license, or stated as exceptions;
the above requirements apply either way.
8. Termination.
You may not propagate or modify a covered work except as expressly
provided under this License. Any attempt otherwise to propagate or
modify it is void, and will automatically terminate your rights under
this License (including any patent licenses granted under the third
paragraph of section 11).
However, if you cease all violation of this License, then your
license from a particular copyright holder is reinstated (a)
provisionally, unless and until the copyright holder explicitly and
finally terminates your license, and (b) permanently, if the copyright
holder fails to notify you of the violation by some reasonable means
prior to 60 days after the cessation.
Moreover, your license from a particular copyright holder is
reinstated permanently if the copyright holder notifies you of the
violation by some reasonable means, this is the first time you have
received notice of violation of this License (for any work) from that
copyright holder, and you cure the violation prior to 30 days after
your receipt of the notice.
Termination of your rights under this section does not terminate the
licenses of parties who have received copies or rights from you under
this License. If your rights have been terminated and not permanently
reinstated, you do not qualify to receive new licenses for the same
material under section 10.
9. Acceptance Not Required for Having Copies.
You are not required to accept this License in order to receive or
run a copy of the Program. Ancillary propagation of a covered work
occurring solely as a consequence of using peer-to-peer transmission
to receive a copy likewise does not require acceptance. However,
nothing other than this License grants you permission to propagate or
modify any covered work. These actions infringe copyright if you do
not accept this License. Therefore, by modifying or propagating a
covered work, you indicate your acceptance of this License to do so.
10. Automatic Licensing of Downstream Recipients.
Each time you convey a covered work, the recipient automatically
receives a license from the original licensors, to run, modify and
propagate that work, subject to this License. You are not responsible
for enforcing compliance by third parties with this License.
An "entity transaction" is a transaction transferring control of an
organization, or substantially all assets of one, or subdividing an
organization, or merging organizations. If propagation of a covered
work results from an entity transaction, each party to that
transaction who receives a copy of the work also receives whatever
licenses to the work the party's predecessor in interest had or could
give under the previous paragraph, plus a right to possession of the
Corresponding Source of the work from the predecessor in interest, if
the predecessor has it or can get it with reasonable efforts.
You may not impose any further restrictions on the exercise of the
rights granted or affirmed under this License. For example, you may
not impose a license fee, royalty, or other charge for exercise of
rights granted under this License, and you may not initiate litigation
(including a cross-claim or counterclaim in a lawsuit) alleging that
any patent claim is infringed by making, using, selling, offering for
sale, or importing the Program or any portion of it.
11. Patents.
A "contributor" is a copyright holder who authorizes use under this
License of the Program or a work on which the Program is based. The
work thus licensed is called the contributor's "contributor version".
A contributor's "essential patent claims" are all patent claims
owned or controlled by the contributor, whether already acquired or
hereafter acquired, that would be infringed by some manner, permitted
by this License, of making, using, or selling its contributor version,
but do not include claims that would be infringed only as a
consequence of further modification of the contributor version. For
purposes of this definition, "control" includes the right to grant
patent sublicenses in a manner consistent with the requirements of
this License.
Each contributor grants you a non-exclusive, worldwide, royalty-free
patent license under the contributor's essential patent claims, to
make, use, sell, offer for sale, import and otherwise run, modify and
propagate the contents of its contributor version.
In the following three paragraphs, a "patent license" is any express
agreement or commitment, however denominated, not to enforce a patent
(such as an express permission to practice a patent or covenant not to
sue for patent infringement). To "grant" such a patent license to a
party means to make such an agreement or commitment not to enforce a
patent against the party.
If you convey a covered work, knowingly relying on a patent license,
and the Corresponding Source of the work is not available for anyone
to copy, free of charge and under the terms of this License, through a
publicly available network server or other readily accessible means,
then you must either (1) cause the Corresponding Source to be so
available, or (2) arrange to deprive yourself of the benefit of the
patent license for this particular work, or (3) arrange, in a manner
consistent with the requirements of this License, to extend the patent
license to downstream recipients. "Knowingly relying" means you have
actual knowledge that, but for the patent license, your conveying the
covered work in a country, or your recipient's use of the covered work
in a country, would infringe one or more identifiable patents in that
country that you have reason to believe are valid.
If, pursuant to or in connection with a single transaction or
arrangement, you convey, or propagate by procuring conveyance of, a
covered work, and grant a patent license to some of the parties
receiving the covered work authorizing them to use, propagate, modify
or convey a specific copy of the covered work, then the patent license
you grant is automatically extended to all recipients of the covered
work and works based on it.
A patent license is "discriminatory" if it does not include within
the scope of its coverage, prohibits the exercise of, or is
conditioned on the non-exercise of one or more of the rights that are
specifically granted under this License. You may not convey a covered
work if you are a party to an arrangement with a third party that is
in the business of distributing software, under which you make payment
to the third party based on the extent of your activity of conveying
the work, and under which the third party grants, to any of the
parties who would receive the covered work from you, a discriminatory
patent license (a) in connection with copies of the covered work
conveyed by you (or copies made from those copies), or (b) primarily
for and in connection with specific products or compilations that
contain the covered work, unless you entered into that arrangement,
or that patent license was granted, prior to 28 March 2007.
Nothing in this License shall be construed as excluding or limiting
any implied license or other defenses to infringement that may
otherwise be available to you under applicable patent law.
12. No Surrender of Others' Freedom.
If conditions are imposed on you (whether by court order, agreement or
otherwise) that contradict the conditions of this License, they do not
excuse you from the conditions of this License. If you cannot convey a
covered work so as to satisfy simultaneously your obligations under this
License and any other pertinent obligations, then as a consequence you may
not convey it at all. For example, if you agree to terms that obligate you
to collect a royalty for further conveying from those to whom you convey
the Program, the only way you could satisfy both those terms and this
License would be to refrain entirely from conveying the Program.
13. Use with the GNU Affero General Public License.
Notwithstanding any other provision of this License, you have
permission to link or combine any covered work with a work licensed
under version 3 of the GNU Affero General Public License into a single
combined work, and to convey the resulting work. The terms of this
License will continue to apply to the part which is the covered work,
but the special requirements of the GNU Affero General Public License,
section 13, concerning interaction through a network will apply to the
combination as such.
14. Revised Versions of this License.
The Free Software Foundation may publish revised and/or new versions of
the GNU General Public License from time to time. Such new versions will
be similar in spirit to the present version, but may differ in detail to
address new problems or concerns.
Each version is given a distinguishing version number. If the
Program specifies that a certain numbered version of the GNU General
Public License "or any later version" applies to it, you have the
option of following the terms and conditions either of that numbered
version or of any later version published by the Free Software
Foundation. If the Program does not specify a version number of the
GNU General Public License, you may choose any version ever published
by the Free Software Foundation.
If the Program specifies that a proxy can decide which future
versions of the GNU General Public License can be used, that proxy's
public statement of acceptance of a version permanently authorizes you
to choose that version for the Program.
Later license versions may give you additional or different
permissions. However, no additional obligations are imposed on any
author or copyright holder as a result of your choosing to follow a
later version.
15. Disclaimer of Warranty.
THERE IS NO WARRANTY FOR THE PROGRAM, TO THE EXTENT PERMITTED BY
APPLICABLE LAW. EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT
HOLDERS AND/OR OTHER PARTIES PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY
OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO,
THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
PURPOSE. THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE PROGRAM
IS WITH YOU. SHOULD THE PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF
ALL NECESSARY SERVICING, REPAIR OR CORRECTION.
16. Limitation of Liability.
IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING
WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MODIFIES AND/OR CONVEYS
THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, INCLUDING ANY
GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING OUT OF THE
USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED TO LOSS OF
DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY YOU OR THIRD
PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER PROGRAMS),
EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF
SUCH DAMAGES.
17. Interpretation of Sections 15 and 16.
If the disclaimer of warranty and limitation of liability provided
above cannot be given local legal effect according to their terms,
reviewing courts shall apply local law that most closely approximates
an absolute waiver of all civil liability in connection with the
Program, unless a warranty or assumption of liability accompanies a
copy of the Program in return for a fee.
END OF TERMS AND CONDITIONS
How to Apply These Terms to Your New Programs
If you develop a new program, and you want it to be of the greatest
possible use to the public, the best way to achieve this is to make it
free software which everyone can redistribute and change under these terms.
To do so, attach the following notices to the program. It is safest
to attach them to the start of each source file to most effectively
state the exclusion of warranty; and each file should have at least
the "copyright" line and a pointer to where the full notice is found.
<one line to give the program's name and a brief idea of what it does.>
Copyright (C) <year> <name of author>
This program is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.
You should have received a copy of the GNU General Public License
along with this program. If not, see <https://www.gnu.org/licenses/>.
Also add information on how to contact you by electronic and paper mail.
If the program does terminal interaction, make it output a short
notice like this when it starts in an interactive mode:
<program> Copyright (C) <year> <name of author>
This program comes with ABSOLUTELY NO WARRANTY; for details type `show w'.
This is free software, and you are welcome to redistribute it
under certain conditions; type `show c' for details.
The hypothetical commands `show w' and `show c' should show the appropriate
parts of the General Public License. Of course, your program's commands
might be different; for a GUI interface, you would use an "about box".
You should also get your employer (if you work as a programmer) or school,
if any, to sign a "copyright disclaimer" for the program, if necessary.
For more information on this, and how to apply and follow the GNU GPL, see
<https://www.gnu.org/licenses/>.
The GNU General Public License does not permit incorporating your program
into proprietary programs. If your program is a subroutine library, you
may consider it more useful to permit linking proprietary applications with
the library. If this is what you want to do, use the GNU Lesser General
Public License instead of this License. But first, please read
<https://www.gnu.org/licenses/why-not-lgpl.html>.
+6 -7
View File
@@ -29,7 +29,7 @@ command lands is mapped in [`docs/command-surface.md`](docs/command-surface.md).
## Status (cutover performed) ## Status (cutover performed)
The internal `app`/`pipeline`/`project`/`erf`/`gff`/`topdata`/`music`/`changelog`/ The internal `app`/`pipeline`/`project`/`erf`/`gff`/`topdata`/`changelog`/
`validator` packages from `gitea/sow-tools` have been migrated into this tree, and `validator` packages from `gitea/sow-tools` have been migrated into this tree, and
the `module`, `topdata`, `hak`, and `wiki` builders now **delegate to the migrated the `module`, `topdata`, `hak`, and `wiki` builders now **delegate to the migrated
`nwn-tool` command surface** (mapped in `nwn-tool` command surface** (mapped in
@@ -49,7 +49,7 @@ which downloads the latest released `crucible` for your OS and runs it:
```bash ```bash
./crucible # interactive menu (pick a command) ./crucible # interactive menu (pick a command)
./crucible module build ./crucible module build
./crucible topdata validate-topdata ./crucible topdata validate
``` ```
Windows (PowerShell): Windows (PowerShell):
@@ -62,15 +62,12 @@ The binary is cached under `~/.cache/crucible/<version>/` (`%LOCALAPPDATA%\cruci
on Windows); `--repo-local` caches inside the repo instead. Private releases: on Windows); `--repo-local` caches inside the repo instead. Private releases:
set `CRUCIBLE_TOKEN` or write the token to `~/.config/crucible/token`. set `CRUCIBLE_TOKEN` or write the token to `~/.config/crucible/token`.
Only the **music** builder needs `ffmpeg`; everything else has zero dependencies.
If you run a music command without it, Crucible prints a per-OS install hint.
## Develop ## Develop
Self-contained (D8) — a host with only Nix can run everything: Self-contained (D8) — a host with only Nix can run everything:
```bash ```bash
nix develop # Go + ffmpeg + shellcheck + yamllint + make nix develop # Go + shellcheck + yamllint + make
make check # go vet + go test + shellcheck + yamllint make check # go vet + go test + shellcheck + yamllint
make build # build every cmd/* into ./bin (gitignored) make build # build every cmd/* into ./bin (gitignored)
make smoke # build + assert the fail-closed contract make smoke # build + assert the fail-closed contract
@@ -94,8 +91,10 @@ publish event is a `v*` tag (see `sow-docs/runbooks/ci-trigger-standard.md`).
- `publish-image.yml` — manual `workflow_dispatch` break-glass republish. - `publish-image.yml` — manual `workflow_dispatch` break-glass republish.
- `sync-wrappers.yml` — on a `main` push that touches `wrappers/`, auto-PR the - `sync-wrappers.yml` — on a `main` push that touches `wrappers/`, auto-PR the
canonical wrappers to the consumer repos in `wrappers/consumers.txt`. canonical wrappers to the consumer repos in `wrappers/consumers.txt`.
Consumer drift checks run after those PRs merge to `main`, not on the PRs
themselves, to avoid recursive cross-repo checks.
## Consumers ## Consumers
How the artifact repos resolve a Crucible binary (and the `NWN_ROOT` rule) is How the artifact repos resolve a Crucible binary is
documented in [`docs/consumer-contract.md`](docs/consumer-contract.md). documented in [`docs/consumer-contract.md`](docs/consumer-contract.md).
+1 -6
View File
@@ -6,10 +6,6 @@
# and ships them on a static base. The `crucible` dispatcher is the entrypoint; # and ships them on a static base. The `crucible` dispatcher is the entrypoint;
# consumer CI can also call the standalone crucible-<name> binaries by path. # consumer CI can also call the standalone crucible-<name> binaries by path.
# #
# NOTE: the internal pipeline is migrated and the music conversion path is wired,
# so the runtime stage is debian-slim with ffmpeg on PATH (BMU encode/decode).
# See docs/migration-from-nwn-tool.md.
FROM golang:1.26-alpine AS build FROM golang:1.26-alpine AS build
WORKDIR /src WORKDIR /src
RUN apk add --no-cache git RUN apk add --no-cache git
@@ -30,11 +26,10 @@ RUN set -eux; \
done done
FROM debian:12-slim FROM debian:12-slim
# ffmpeg: the migrated music pipeline shells out to it for BMU conversion.
# ca-certificates: builders fetch published manifests over HTTPS. # ca-certificates: builders fetch published manifests over HTTPS.
RUN set -eux; \ RUN set -eux; \
apt-get update; \ apt-get update; \
apt-get install -y --no-install-recommends ffmpeg ca-certificates; \ apt-get install -y --no-install-recommends ca-certificates; \
rm -rf /var/lib/apt/lists/*; \ rm -rf /var/lib/apt/lists/*; \
useradd --system --create-home --uid 65532 nonroot useradd --system --create-home --uid 65532 nonroot
COPY --from=build /out/ /usr/local/bin/ COPY --from=build /out/ /usr/local/bin/
+61 -35
View File
@@ -1,51 +1,77 @@
# Crucible command surface # Crucible command surface
Crucible re-homes the single `nwn-tool` (a.k.a. `sow-toolkit`) command surface `internal/dispatch.Registry` is the single source of truth for Crucible's
into five builders plus a dispatcher (D11). This table maps every legacy command builders, visible commands, descriptions, help, and hidden compatibility
to its Crucible home. It is the migration contract — keep it in sync with aliases.
`internal/dispatch.Registry`.
## Builders ## Visible commands
| Builder | Legacy `nwn-tool` commands subsumed | | Builder | Command | Purpose |
| ------------------ | ------------------------------------------------------------------------------------- | | --- | --- | --- |
| `crucible-module` | `build`, `build-module`, `extract`, `validate`, `compare`, `apply-hak-manifest` | | `hak` | `build` | Build configured HAK archives and their manifest. |
| `crucible-topdata` | `validate-topdata`, `build-topdata`, `build-top-package`, `compare-topdata`, `convert-topdata` | | `hak` | `manifest` | Apply a generated HAK list to module source. |
| `crucible-hak` | `build-haks`, `apply-hak-manifest` | | `module` | `build` | Build the module and configured project outputs. |
| `crucible-wiki` | `build-wiki`, `deploy-wiki` | | `module` | `extract` | Extract built archives into configured source trees. |
| `crucible-depot` | (new) depot blob verify/move — was shell `mc`/S3 in `sow-assets-manifest` | | `module` | `validate` | Validate project layout, config, and source inventory. |
| `module` | `compare` | Compare source content with built archives. |
| `module` | `manifest` | Apply a generated HAK list to module source. |
| `topdata` | `validate` | Validate topdata source and canonical JSON compatibility. |
| `topdata` | `build` | Compile topdata and refresh the configured package. |
| `topdata` | `package` | Package cached outputs into HAK and TLK files. |
| `topdata` | `compare` | Compare current output with a fresh native build. |
| `topdata` | `convert` | Convert between 2DA and native JSON/module formats. |
| `wiki` | `build` | Render wiki page drafts from compiled topdata. |
| `wiki` | `deploy` | Deploy generated wiki pages to NodeBB. |
`apply-hak-manifest` is HAK-list maintenance on a module; it is reachable from `crucible-depot` remains registered but unwired. It fails closed with exit `70`
both `crucible-module` and `crucible-hak`. Canonical home is `crucible-hak`. and never emits placeholder artifacts.
## Folded / global commands ## Hidden compatibility aliases
These legacy commands are **not** top-level builders: Existing scripts may continue using these names indefinitely. They are accepted
but omitted from routine help and the interactive menu:
- `music *` (`list-datasets`, `scan`, `build`, `credits`, `validate`, `manifest`, | Builder | Hidden alias | Implementation |
`normalize`) → folds into the **module/hak build pipeline** (music BMUs are | --- | --- | --- |
packed into HAKs at build time). Exposed as `crucible-module music ...` / | `hak` | `build-haks` | `build-haks` |
`crucible-hak music ...`. Needs `ffmpeg` at runtime (in the image). | `hak` | `apply-hak-manifest` | `apply-hak-manifest` |
- `config *` (`validate`, `effective`, `inspect`, `explain`, `sources`) → a | `module` | `build-module` | module-only `build-module` |
**global** concern shared by every builder; exposed as the global | `module` | `apply-hak-manifest` | `apply-hak-manifest` |
`crucible config ...`, not its own binary. | `topdata` | `validate-topdata` | `validate-topdata` |
- `build-changelog` → release tooling; exposed as the global `crucible changelog` | `topdata` | `build-topdata` | `build-topdata` |
rather than a content builder. | `topdata` | `build-top-package` | `build-top-package` |
| `topdata` | `compare-topdata` | `compare-topdata` |
| `topdata` | `convert-topdata` | `convert-topdata` |
| `wiki` | `build-wiki` | `build-wiki` |
| `wiki` | `deploy-wiki` | `deploy-wiki` |
## Global commands (implemented) `module build-module` deliberately retains its narrower module-only behavior;
it is not redirected to the broader `module build`.
## Global commands
```text ```text
crucible help | -h usage + builder list crucible help | -h usage + builder list
crucible version | -V build version (git sha via -ldflags) crucible version | -V build version
crucible list builders, one per line: name<TAB>bin<TAB>summary crucible list name<TAB>binary<TAB>summary
crucible config [args] -> legacy `config` command group crucible config [args] inspect and validate effective configuration
crucible changelog [args] -> legacy `build-changelog` crucible changelog [args] generate a release changelog
``` ```
`crucible list` is machine-readable so CI can enumerate builders without parsing ## HAK build options
help text.
## Global flags (planned, at wiring time) ```text
crucible hak build
[--hak <hak-name> ...]
[--archive <archive-name> ...]
[--source-manifest <path>]
[--content-addressed-root <path>]
[--plan-only]
[--quiet|--verbose|--debug]
```
`--quiet`, `--verbose`, `--debug` (the legacy verbosity model), passed after the When a source manifest contains `asset_sources`, pass
builder name, e.g. `crucible topdata build-topdata --verbose`. `--content-addressed-root <directory>`. Crucible resolves each declared SHA-256
under `sha256/<first-two>/<next-two>/<full-sha256>` and verifies streamed bytes
while writing selected HAK archives. Authored `.bmu` files are ordinary assets
and require no conversion pipeline.
+24 -14
View File
@@ -22,27 +22,37 @@ If none resolve, the wrapper exits non-zero with a Phase 5 message. The
wrappers prefer the single-token `crucible-<name>` shim so `"$builder" args` wrappers prefer the single-token `crucible-<name>` shim so `"$builder" args`
quoting stays correct. quoting stays correct.
## In CI (preferred) ## Parity contract — wrappers are not authoritative for build inputs
Run the consumer job inside the pinned image and the binaries are on `PATH`: A consumer build wrapper may only:
```yaml 1. **validate** — a pre-build gate that does not change output;
container: 2. **publish** — a post-build step on a separate artifact;
image: registry.westgate.pw/deployment/crucible:<sha> # pinned, immutable 3. **resolve the crucible binary** — the bootstrap above.
```
No host install, no `$HOME` layout, no developer machine assumptions. A wrapper MUST NOT fetch, resolve, generate, or stage any input Crucible reads to
produce the artifact. If Crucible needs an input, Crucible acquires it. This makes
the "no local-machine assumptions" + "same inputs → identical output" rules
explicit: local `crucible <build>` and CI `crucible <build>` against the same
committed config must produce identical bytes, with no CI-only pre-steps.
## `NWN_ROOT` rule Builders read every input from the project tree. No NWN install is required. A
builder that one day needs NWN game data auto-detects the install (the NWN home
and Steam path are reliably discoverable), with an optional explicit override for
non-standard layouts — never a required hand-set env.
Crucible **never guesses `NWN_ROOT` from `$HOME`** (this was a deploy-notes ## In CI
pain point). The NWN install/data root is passed explicitly:
- env `NWN_ROOT=/path/to/nwn`, or Builds run the Crucible **binary** — from the `crucible.sh` / `crucible.ps1`
- flag `--nwn-root /path/to/nwn`. bootstrap (anonymous download) or, for Nix users, from the flake devshell pinned
by `flake.lock`. CI runs the _same_ `crucible <build>` as local dev, inside the
nix devshell (binary-cache fast). No build container, no token, no CI-only
pre-steps.
A builder that needs `NWN_ROOT` and receives neither must fail closed with a The `registry.westgate.pw/deployment/crucible:<sha>` image is **deployment-only**:
clear message, not fall back to a home-directory default. `prod.yml` pins it so tools travel with the runtime host (`nwn.enable`), a
disabled placeholder until the NWN stack lands. It is **not** a build tool and no
build/CI job consumes it.
## Determinism ## Determinism
+2 -5
View File
@@ -27,7 +27,7 @@ checklist is kept as the record of what was done.
## Cutover steps ## Cutover steps
1. **Move the internal packages.** Bring `internal/{app,pipeline,project,erf, 1. **Move the internal packages.** Bring `internal/{app,pipeline,project,erf,
gff,topdata,music,changelog,validator}` from `gitea/sow-tools` into this repo. gff,topdata,changelog,validator}` from `gitea/sow-tools` into this repo.
Re-home `internal/app` command wiring behind `internal/dispatch` instead of Re-home `internal/app` command wiring behind `internal/dispatch` instead of
the old flag parser in `cmd/nwn-tool`. the old flag parser in `cmd/nwn-tool`.
2. **Restore dependencies.** Add `golang.org/x/text` and `gopkg.in/yaml.v3` back 2. **Restore dependencies.** Add `golang.org/x/text` and `gopkg.in/yaml.v3` back
@@ -43,10 +43,7 @@ checklist is kept as the record of what was done.
artifacts and image layers only. artifacts and image layers only.
6. **Carry the determinism test.** Port 6. **Carry the determinism test.** Port
`TestBuildNativeKeepsGeneratedIDsStableAcrossRepeatedBuilds` and keep it green. `TestBuildNativeKeepsGeneratedIDsStableAcrossRepeatedBuilds` and keep it green.
7. **Runtime base for music.** If music conversion is wired, switch the final 7. **Update `make smoke`.** Once a builder is wired, change the smoke
Docker stage from `distroless/static` to a debian-slim base that includes
`ffmpeg`.
8. **Update `make smoke`.** Once a builder is wired, change the smoke
expectation for it from exit `70` to a real success path. expectation for it from exit `70` to a real success path.
## Wrapper contract carried over ## Wrapper contract carried over
File diff suppressed because it is too large Load Diff
@@ -0,0 +1,346 @@
# Crucible Command Surface Cleanup Implementation Plan
> **For agentic workers:** REQUIRED SUB-SKILL: Use
> `superpowers:subagent-driven-development` or
> `superpowers:executing-plans` to implement this plan task-by-task. Steps use
> checkbox (`- [ ]`) syntax for tracking.
**Goal:** Replace repetitive Crucible command names with a descriptive,
registry-driven public surface, retain long names as hidden aliases, and remove
the unused music conversion pipeline without affecting authored `.bmu` assets.
**Architecture:** `internal/dispatch.Registry` gains explicit command records
that drive lookup, help, and the interactive menu while translating to unchanged
`internal/app` implementation names. The unused music subsystem is removed
through its app, pipeline, project-config, validator, packaging, and consumer
boundaries. The two active `sow-assets-manifest` call sites are decoupled in the
same worktree session.
**Tech Stack:** Go 1.26, Go tests, Nix flakes, Dockerfile, Bash/Bats.
## Global Constraints
- Never commit or push.
- Do not touch secrets or `.sops` data.
- Keep `internal/dispatch.Registry` synchronized with
`docs/command-surface.md`.
- Preserve deterministic artifact behavior.
- Preserve every existing long command as a hidden alias except all music
commands, which are removed.
- Preserve `.bmu` as an ordinary HAK asset.
- Do not retain generated binaries or build output.
---
### Task 1: Registry-driven canonical commands and hidden aliases
**Files:**
- Modify: `internal/dispatch/dispatch.go`
- Modify: `internal/dispatch/dispatch_test.go`
**Interfaces:**
- Produces a `Command` record owned by each `Builder`.
- Produces lookup that returns the command record and exact app target.
- Preserves `Builder.Legacy` only if required by migration tests; otherwise
replaces it with command-derived compatibility metadata.
- [ ] Add failing tests covering the visible command table:
`hak build/manifest`, `module build/extract/validate/compare/manifest`,
`topdata validate/build/package/compare/convert`, and `wiki build/deploy`.
- [ ] Add failing tests proving long aliases resolve to the same app target and
that `module build-module` resolves to `build-module`, not `build`.
- [ ] Add a failing test proving all music commands are rejected.
- [ ] Run:
```bash
nix develop --command go test ./internal/dispatch -run 'TestCanonical|TestHidden|TestMusic' -v
```
Expected: failures caused by the old registry shape and exposed music
commands.
- [ ] Introduce command records with fields equivalent to:
```go
type Command struct {
Name string
Summary string
AppCommand string
Usage string
Options []string
HiddenAlias []string
}
```
Use the minimum final field names that keep lookup, menu, and help readable.
- [ ] Replace builder-level subcommand acceptance with command lookup. Translate
canonical and hidden aliases to `AppCommand` before calling `app.Run`.
- [ ] Define the approved command table and remove dispatcher music entries.
- [ ] Re-run the focused dispatch tests and the full dispatch package:
```bash
nix develop --command go test ./internal/dispatch -v
```
Expected: pass.
### Task 2: Descriptive menu and command-specific help
**Files:**
- Modify: `internal/menu/menu.go`
- Modify: `internal/menu/menu_test.go`
- Modify: `internal/dispatch/dispatch.go`
- Modify: `internal/dispatch/dispatch_test.go`
**Interfaces:**
- `menu.Item` carries display label, summary, dispatcher args, usage, and option
lines.
- `menu.Select` returns selected dispatcher args plus entered arguments.
- Dispatcher command help uses the same registry guidance as the menu.
- [ ] Add failing menu tests proving descriptions start at one aligned column
for labels of different lengths and hidden aliases never enter `menuItems()`.
- [ ] Add a failing test selecting a command and asserting output contains its
usage, option guidance, and
`arguments (press Enter to use defaults):`.
- [ ] Add failing dispatcher tests proving
`crucible topdata build --help` returns `0`, prints canonical usage/options,
and does not delegate into project loading.
- [ ] Run:
```bash
nix develop --command go test ./internal/menu ./internal/dispatch -run 'Test.*Menu|Test.*Help' -v
```
Expected: fail against the current fixed-width menu and builder-only help.
- [ ] Compute menu label width from visible items and render descriptions with
that width.
- [ ] Print command guidance immediately after a selection and before reading
optional arguments.
- [ ] Add command-level help handling before app delegation.
- [ ] Ensure builder help lists only canonical names and their summaries.
- [ ] Run:
```bash
nix develop --command go test ./internal/menu ./internal/dispatch -v
```
Expected: pass.
### Task 3: Remove music from HAK app and pipeline
**Files:**
- Delete: `internal/pipeline/music.go`
- Modify: `internal/pipeline/build.go`
- Modify: `internal/pipeline/pipeline_test.go`
- Modify: `internal/app/app.go`
- Modify: `internal/app/app_test.go`
- Delete: `internal/music/config.go`
- Delete: `internal/music/credits.go`
- Delete: `internal/music/metadata.go`
- Delete: `internal/music/music.go`
- Delete: `internal/music/music_test.go`
- Delete: `internal/music/naming.go`
**Interfaces:**
- `pipeline.BuildHAKOptions` retains only HAK selection, source-manifest,
content-addressed-root, and progress controls.
- `pipeline.BuildResult` retains artifact fields unrelated to generated music
credits.
- `build-haks` accepts no music-specific flags.
- [ ] Replace music pipeline tests with a failing contract test that builds a
HAK containing a pre-authored `.bmu` and proves the file appears in the
manifest/archive without conversion setup.
- [ ] Add failing parser tests proving `--skip-music` and `--music-dataset`
return unknown-argument errors and help does not mention them.
- [ ] Run:
```bash
nix develop --command go test ./internal/app ./internal/pipeline -run 'Test.*Music|Test.*BMU|TestParseBuildHAK' -v
```
Expected: at least the removed-flag assertions fail.
- [ ] Remove the app `music` command, argument parser, emitters, music HAK
options, and credits console output.
- [ ] Remove music fields from `BuildHAKOptions` and `BuildResult`.
- [ ] Remove preparation/cleanup calls from HAK builds so asset collection reads
authored resources directly.
- [ ] Delete the music implementation packages and obsolete conversion tests.
- [ ] Keep `.bmu` handling in ordinary ERF/resource and asset collection paths.
- [ ] Run:
```bash
nix develop --command go test ./internal/app ./internal/pipeline -v
```
Expected: pass.
### Task 4: Remove music configuration and validation
**Files:**
- Modify: `internal/project/project.go`
- Modify: `internal/project/effective.go`
- Modify: `internal/project/project_test.go`
- Modify: `internal/validator/validator.go`
- Modify: `internal/validator/validator_test.go`
**Interfaces:**
- `project.Config` and `project.EffectiveConfig` contain no music section.
- Project scanning uses only configured inventory extensions.
- `.bmu` and `.wav` remain default asset extensions; `.mp3` and `.ogg` do not.
- [ ] Add or adjust failing project tests asserting default extensions include
`.bmu` and `.wav` but exclude `.mp3` and `.ogg`.
- [ ] Add a failing strict-decoder test proving a top-level `music:` field is
rejected as unknown.
- [ ] Run:
```bash
nix develop --command go test ./internal/project ./internal/validator -run 'Test.*Music|Test.*AssetExtension|Test.*BMU' -v
```
Expected: fail while music config is still accepted.
- [ ] Remove music config types, defaults, normalization, validation, accessors,
effective config, provenance defaults, and environment overrides.
- [ ] Simplify `Project.Scan` to accept assets solely from
`effective.Inventory.AssetExtensions`.
- [ ] Remove validator logic that exempts conversion source files.
- [ ] Remove conversion-only `.mp3` and `.ogg` from default asset extensions;
retain `.bmu` and `.wav`.
- [ ] Remove obsolete music-focused project and validator tests.
- [ ] Run:
```bash
nix develop --command go test ./internal/project ./internal/validator -v
```
Expected: pass.
### Task 5: Remove ffmpeg packaging and stale documentation
**Files:**
- Modify: `flake.nix`
- Modify: `docker/Dockerfile`
- Modify: `wrappers/crucible.sh`
- Modify: `README.md`
- Modify: `AGENTS.md`
- Modify: `docs/command-surface.md`
- Modify: `docs/migration-from-nwn-tool.md`
**Interfaces:**
- Nix development shell and image no longer contain ffmpeg.
- Docker runtime contains only certificates and the non-root runtime support
needed by current builders.
- User docs advertise canonical short commands.
- [ ] Add a shell verification that fails while active packaging still mentions
ffmpeg or removed command names:
```bash
! rg -n 'ffmpeg|ffprobe|skip-music|music-dataset|module music|hak music' \
flake.nix docker wrappers README.md AGENTS.md docs/command-surface.md
```
- [ ] Remove `pkgs.ffmpeg-headless` and `ffmpeg` from Nix image/dev-shell
contents and update comments.
- [ ] Return the Docker runtime to a minimal base without ffmpeg while retaining
CA certificates and non-root execution.
- [ ] Remove the wrapper's `ffmpeg-free` wording.
- [ ] Rewrite command-surface documentation around canonical names and a
concise hidden-alias compatibility section.
- [ ] Update README examples and ownership/status prose.
- [ ] Re-run the search. Expected: no matches in active files.
### Task 6: Decouple active consumer scripts
**Files in `../sow-assets-manifest`:**
- Modify: `scripts/pack-haks.sh`
- Modify: `scripts/build-local-haks.sh`
- Modify: `tests/pack-haks.bats`
- Modify only directly related active comments/tests if a focused test proves
they require adjustment.
**Interfaces:**
- Both scripts call the hidden-compatible `hak build-haks` or canonical
`hak build` without `--skip-music`.
- [ ] Read `../sow-assets-manifest/AGENTS.md` before editing.
- [ ] Add or adjust a focused test to reject `--skip-music` in generated
Crucible invocations.
- [ ] Run the focused test and confirm it fails while the flag is present.
- [ ] Remove `--skip-music` and its lockstep comments from both scripts.
- [ ] Update the HAK parser test invocation to omit the removed flag.
- [ ] Run:
```bash
cd ../sow-assets-manifest
nix develop --command bats tests/pack-haks.bats
```
Expected: pass.
### Task 7: Full verification and scratchpad cleanup
**Files:**
- Modify: `../SCRATCHPAD.md`
- Do not retain: `bin/`, `.cache/go-build/`, result symlinks, image tarballs, or
other generated outputs.
- [ ] Run formatting:
```bash
nix develop --command gofmt -w internal/app internal/dispatch internal/menu internal/pipeline internal/project internal/validator
```
- [ ] Run full repository checks:
```bash
nix develop --command make check
make build
make smoke
```
- [ ] Run focused consumer verification from Task 6.
- [ ] Verify deleted pipeline identifiers are absent from active code:
```bash
rg -n 'BuildMusic|CreditsSummary|MusicDataset|skip-music|music-dataset|ffmpeg|ffprobe' \
--glob '!docs/superpowers/**' .
```
Expected: no matches.
- [ ] Verify active command docs/examples do not advertise long names:
```bash
rg -n 'topdata (validate-topdata|build-topdata|build-top-package|compare-topdata|convert-topdata)|wiki (build-wiki|deploy-wiki)|hak (build-haks|apply-hak-manifest)' \
README.md docs/command-surface.md
```
Expected: matches only inside the explicitly labeled hidden-alias
compatibility section.
- [ ] Inspect `git diff --check`, `git status --short`, and both repository
diffs. Remove only generated files created by this work.
- [ ] Remove the completed “Simplify crucible commands” to-do and the full
“Crucible commands issue” section from `../SCRATCHPAD.md`; do not archive it.
- [ ] Record exact verification commands and outcomes in the final handoff.
@@ -0,0 +1,858 @@
# Topdata JSON Validation Tightening Implementation Plan
> **For agentic workers:** REQUIRED SUB-SKILL: Use
> `superpowers:subagent-driven-development` (recommended) or
> `superpowers:executing-plans` to implement this plan task-by-task. Steps use
> checkbox (`- [ ]`) syntax for tracking.
**Goal:** Reject unsupported properties in established canonical topdata JSON
containers, add the non-empty `any_present` global-injection condition, and keep
standalone validation and direct native builds fail-closed and consistent.
**Architecture:** Add one focused `internal/topdata/json_contract.go` unit that
owns deterministic closed-key checks plus parsing and evaluation of global
injection condition groups. Existing validation in `topdata.go` uses that unit
to collect actionable diagnostics, while `native.go` uses the same parsed
conditions during ordered injection evaluation. Dataset payload rows continue
through the existing column-aware canonicalization paths and are not assigned a
global schema.
**Tech Stack:** Go 1.26, standard-library `encoding/json`, Go tests, Nix flakes,
Make, authored JSON in the sibling `sow-topdata` checkout.
## Global Constraints
- Work on a fresh non-`main` branch and open all changes as a pull request.
- Never commit or push secrets, `.sops` data, generated binaries, `.cache/`,
`generated/`, HAK/TLK artifacts, or result symlinks.
- Add no JSON Schema framework and no new validation dependency.
- Close only the canonical base/plain rows root, canonical module root,
`global.json` root, global injection, global condition, global default rule,
global default match object, and global default format object.
- Preserve specialized parser-owned JSON dialects outside those canonical
containers.
- Preserve dataset-specific columns and documented row controls inside `rows`,
`entries`, `overrides`, injection `row`, and default `values`.
- Preserve current-row, ordered-injection behavior: an earlier injection may
satisfy or block a later injection.
- Preserve empty-list behavior for `require_present` and `unless_present`;
reject an empty `any_present`.
- Do not add a compatibility alias for `when_present`.
- Use deterministic property ordering in diagnostics.
- Inventory evidence requires `compare_reference` to remain valid on
base/plain rows roots: `internal/topdata/native.go` reads it,
`internal/topdata/feat_migrate.go` emits it, and existing tests exercise it.
This is an established canonical property covered by the design's moderate
strictness rule even though the initial property table omitted it.
- The active `sow-topdata` branch
`fix-always-present-metamagic` already replaces the known `when_present` with
`require_present` in commit `ed139f5`; preserve that change and use the
checkout for integration validation rather than duplicating or rewriting it.
---
### Task 1: Introduce the shared closed-object and condition contract
**Files:**
- Create: `internal/topdata/json_contract.go`
- Create: `internal/topdata/json_contract_test.go`
**Interfaces:**
- Produces:
```go
type globalCondition struct {
Field string
ID string
}
type globalConditionGroups struct {
RequirePresent []globalCondition
AnyPresent []globalCondition
UnlessPresent []globalCondition
}
func unsupportedObjectKeys(obj map[string]any, supported ...string) []string
func unsupportedObjectKeysError(label string, obj map[string]any, supported ...string) error
func parseGlobalConditionGroups(injection map[string]any) (globalConditionGroups, error)
func globalConditionGroupsMatch(groups globalConditionGroups, rows []map[string]any) bool
```
- `unsupportedObjectKeys` returns unknown keys in lexical order.
- `parseGlobalConditionGroups` accepts absent groups, permits empty
`require_present` and `unless_present`, rejects empty `any_present`, validates
condition keys as exactly `field` and `id`, and returns the first deterministic
error.
- `globalConditionGroupsMatch` implements all/any/none semantics without
reparsing JSON.
- [ ] **Step 1: Create the feature branch after checking it was not previously
merged and deleted**
```bash
git fetch origin
git branch --show-current
git ls-remote --heads origin feat/topdata-json-validation
git log --all --oneline --decorate --grep='topdata json validation'
```
Expected: current branch is not `main`. If
`feat/topdata-json-validation` has never been used, create it from the approved
design commit:
```bash
git switch -c feat/topdata-json-validation
```
If that exact branch name has prior remote history, choose a new descriptive
name and record it in the PR; do not reuse the old branch.
- [ ] **Step 2: Re-run the canonical-format inventory before writing the
allowlists**
```bash
for f in $(find ../sow-topdata/data -type f -name 'base.json' | sort); do
jq -r --arg f "${f#../sow-topdata/}" \
'[$f, (keys | sort | join(","))] | @tsv' "$f"
done
for f in $(find ../sow-topdata/data -type f -name 'global.json' | sort); do
jq -r --arg f "${f#../sow-topdata/}" \
'[$f, (keys | sort | join(","))] | @tsv' "$f"
done
find ../sow-topdata/data -type f -path '*/modules/*.json' -print0 |
while IFS= read -r -d '' f; do
jq -r --arg f "${f#../sow-topdata/}" \
'[$f, (keys | sort | join(","))] | @tsv' "$f"
done | sort
rg -n 'compare_reference' internal/topdata ../sow-topdata/data
```
Expected: active base roots use `columns`, `key`, `output`, and `rows`;
active modules use `columns`, `entries`, `overrides`, or the legitimate
`entries`+`overrides` combination; active globals use the approved global
keys. `sow-tools` additionally confirms established `compare_reference`
support.
- [ ] **Step 3: Write failing unit tests for deterministic unsupported-key
diagnostics**
Add table-driven tests equivalent to:
```go
func TestUnsupportedObjectKeysAreSorted(t *testing.T) {
obj := map[string]any{
"row": map[string]any{},
"when_present": []any{},
"aaa": true,
}
got := unsupportedObjectKeys(
obj,
"row",
"require_present",
"any_present",
"unless_present",
)
want := []string{"aaa", "when_present"}
if !slices.Equal(got, want) {
t.Fatalf("unsupportedObjectKeys() = %v, want %v", got, want)
}
}
func TestUnsupportedObjectKeysErrorNamesObjectAndContract(t *testing.T) {
err := unsupportedObjectKeysError(
"global injection 10",
map[string]any{"row": map[string]any{}, "when_present": []any{}},
"row",
"require_present",
"any_present",
"unless_present",
)
if err == nil {
t.Fatal("expected unsupported-property error")
}
for _, want := range []string{
"global injection 10",
`"when_present"`,
"row",
"require_present",
"any_present",
"unless_present",
} {
if !strings.Contains(err.Error(), want) {
t.Fatalf("error %q does not contain %q", err, want)
}
}
}
```
- [ ] **Step 4: Write failing parser tests for all condition-group contracts**
Cover these exact cases in `json_contract_test.go`:
```go
tests := []struct {
name string
injection map[string]any
want globalConditionGroups
wantError string
}{
{
name: "all groups",
injection: map[string]any{
"require_present": []any{
map[string]any{"field": "FeatIndex", "id": "feat:required"},
},
"any_present": []any{
map[string]any{"field": "FeatIndex", "id": "feat:first"},
map[string]any{"field": "FeatIndex", "id": "feat:second"},
},
"unless_present": []any{
map[string]any{"field": "FeatIndex", "id": "feat:blocked"},
},
},
want: globalConditionGroups{
RequirePresent: []globalCondition{
{Field: "FeatIndex", ID: "feat:required"},
},
AnyPresent: []globalCondition{
{Field: "FeatIndex", ID: "feat:first"},
{Field: "FeatIndex", ID: "feat:second"},
},
UnlessPresent: []globalCondition{
{Field: "FeatIndex", ID: "feat:blocked"},
},
},
},
{
name: "empty existing groups remain valid",
injection: map[string]any{
"require_present": []any{},
"unless_present": []any{},
},
},
{
name: "any present must be array",
injection: map[string]any{
"any_present": "feat:first",
},
wantError: "any_present must be an array",
},
{
name: "any present must not be empty",
injection: map[string]any{
"any_present": []any{},
},
wantError: "any_present must contain at least one condition",
},
{
name: "condition is closed",
injection: map[string]any{
"require_present": []any{
map[string]any{
"field": "FeatIndex",
"id": "feat:required",
"typo": true,
},
},
},
wantError: `require_present[0] contains unsupported key "typo"`,
},
{
name: "field is required",
injection: map[string]any{
"require_present": []any{
map[string]any{"id": "feat:required"},
},
},
wantError: "require_present[0].field is required",
},
{
name: "id is required",
injection: map[string]any{
"require_present": []any{
map[string]any{"field": "FeatIndex"},
},
},
wantError: "require_present[0].id is required",
},
}
```
- [ ] **Step 5: Write failing evaluator tests for all/any/none and group
conjunction**
Build rows with `FeatIndex` references for `feat:required`,
`feat:second`, and `feat:blocked`, then assert:
```go
if !globalConditionGroupsMatch(globalConditionGroups{
RequirePresent: []globalCondition{
{Field: "FeatIndex", ID: "feat:required"},
},
AnyPresent: []globalCondition{
{Field: "FeatIndex", ID: "feat:first"},
{Field: "FeatIndex", ID: "feat:second"},
},
}, rowsWithoutBlocked) {
t.Fatal("expected required + later any_present alternative to match")
}
if globalConditionGroupsMatch(globalConditionGroups{
AnyPresent: []globalCondition{
{Field: "FeatIndex", ID: "feat:missing"},
},
}, rowsWithoutBlocked) {
t.Fatal("expected missing any_present alternatives to reject")
}
if globalConditionGroupsMatch(globalConditionGroups{
UnlessPresent: []globalCondition{
{Field: "FeatIndex", ID: "feat:blocked"},
},
}, rowsWithBlocked) {
t.Fatal("expected unless_present match to reject")
}
```
- [ ] **Step 6: Run the focused tests and confirm they fail**
```bash
nix develop --command go test ./internal/topdata \
-run 'TestUnsupportedObjectKeys|TestParseGlobalConditionGroups|TestGlobalConditionGroupsMatch' \
-v
```
Expected: build failure because the shared contract types and functions do
not exist.
- [ ] **Step 7: Implement the minimal shared contract**
In `json_contract.go`, define fixed key sets and implement:
```go
var (
baseOrPlainRootKeys = []string{
"output", "key", "columns", "rows", "compare_reference",
}
canonicalModuleRootKeys = []string{
"output", "key", "columns", "entries", "overrides", "rows",
}
globalRootKeys = []string{
"columns", "entries", "overrides", "defaults", "position", "injections",
}
globalInjectionKeys = []string{
"row", "require_present", "any_present", "unless_present",
}
globalConditionKeys = []string{"field", "id"}
globalDefaultRuleKeys = []string{"match", "values"}
globalDefaultMatchKeys = []string{"source"}
globalDefaultFormatKeys = []string{"format"}
)
```
`unsupportedObjectKeysError` must report one deterministic error per call,
using the first lexically sorted unsupported key and a supported-key list in
declaration order. Implement condition parsing through one private helper:
```go
func parseGlobalConditionList(
name string,
raw any,
requireNonEmpty bool,
) ([]globalCondition, error)
```
It must reject non-arrays, reject non-object elements, run the closed-key
check before required-field checks, trim `field` and `id`, and enforce the
`any_present` non-empty rule.
`globalConditionGroupsMatch` must call the existing
`globalReferencePresent` helper and implement:
```text
require_present: every condition is found
any_present: at least one condition is found
unless_present: no condition is found
all groups: every present group passes
```
- [ ] **Step 8: Run the focused and package tests**
```bash
nix develop --command go test ./internal/topdata \
-run 'TestUnsupportedObjectKeys|TestParseGlobalConditionGroups|TestGlobalConditionGroupsMatch' \
-v
nix develop --command go test ./internal/topdata
```
Expected: pass.
- [ ] **Step 9: Commit the shared contract**
```bash
git add internal/topdata/json_contract.go internal/topdata/json_contract_test.go
git commit -m "feat(topdata): define canonical json contracts"
```
---
### Task 2: Close canonical structural objects during project validation
**Files:**
- Modify: `internal/topdata/topdata.go`
- Modify: `internal/topdata/json_contract_test.go`
**Interfaces:**
- Consumes the key sets and condition parser from Task 1.
- `validateDataObject` selects exactly one root contract from file context:
base/plain rows root, canonical module root, or `global.json` root.
- A module validates every present supported container; it does not return
after the first of `entries`, `overrides`, `rows`, or `columns`.
- Specialized paths continue to return to their existing validators before
canonical root closure.
- [ ] **Step 1: Add failing validation tests for unsupported canonical root
properties**
Add table-driven temporary-project cases covering:
- base root with `"typo": true`;
- loose plain rows root with `"typo": true`;
- module root with `"typo": true`;
- `global.json` root with `"typo": true`.
For every case, call `ValidateProject`, require errors, and assert only stable
semantic fragments: the file/object label, `typo`, and `unsupported`.
- [ ] **Step 2: Add failing validation tests for closed nested global objects**
Add cases for:
```json
{"injections":[{"row":{},"when_present":[]}]}
{"injections":[{"row":{},"require_present":[{"field":"FeatIndex","id":"feat:x","typo":true}]}]}
{"defaults":[{"match":"all","values":{},"typo":true}]}
{"defaults":[{"match":{"source":"entries","typo":true},"values":{}}]}
{"defaults":[{"match":"all","values":{"ImpactScript":{"format":"ss_{id}","typo":true}}}]}
```
Expected diagnostics name `global injection 0`, the condition label,
`global default 0`, `global default 0 match`, or
`global default 0 values.ImpactScript` respectively.
- [ ] **Step 3: Add failing validation tests for `any_present` shape**
Test one accepted non-empty list and rejected string/empty-list cases. The
accepted case must use a declared dataset column in the injection row so this
also protects payload flexibility.
- [ ] **Step 4: Add regression tests for canonical combinations and payload
flexibility**
Build one table-driven test that validates without errors:
- a columns-only module;
- an entries-only module;
- an overrides-only module;
- a module containing both entries and overrides;
- a rows module;
- a current base root;
- a loose plain rows root;
- a base/plain root with `compare_reference: false`;
- dataset columns plus `id`, `key`, `inherit`, `_tlk`, and `meta` in row-like
payloads where their existing contracts allow them;
- an injection row using a declared dataset column.
The combined entries+overrides case must contain an independently invalid
override value first, prove validation catches it, then fix only that value
and prove the module passes. This specifically prevents the current
first-container early return.
- [ ] **Step 5: Run the focused tests and confirm failures**
```bash
nix develop --command go test ./internal/topdata \
-run 'TestValidateProject(RejectsUnsupportedCanonicalJSON|RejectsUnsupportedGlobalJSON|ValidatesEveryCanonicalModuleContainer|AcceptsCanonicalJSONContracts|AcceptsAnyPresent|RejectsInvalidAnyPresent)' \
-v
```
Expected: unknown keys are currently ignored, `any_present` is not validated,
and the combined module does not validate all containers.
- [ ] **Step 6: Add a validation adapter for shared contract errors**
In `topdata.go`, add:
```go
func reportTopdataContractError(path string, err error, report *ValidationReport) {
if err == nil {
return
}
report.Diagnostics = append(report.Diagnostics, Diagnostic{
Severity: SeverityError,
Path: path,
Message: err.Error(),
})
}
```
Use `unsupportedObjectKeysError` for canonical roots and nested structural
objects. Keep row payload validation in existing `validateRowCollection`,
`validateEntriesFile`, and `validateOverridesFile` paths.
- [ ] **Step 7: Refactor canonical module dispatch to validate all containers**
After specialized-path checks, classify module files by directory context.
For recognized canonical modules:
1. check `canonicalModuleRootKeys`;
2. call `validateColumnsFile` when `columns` is present;
3. call `validateEntriesFile` when `entries` is present;
4. call `validateOverridesFile` when `overrides` is present;
5. call `validateRowsFile(path, obj, report, false)` when `rows` is present;
6. emit the existing warning only when none of those containers is present.
Avoid duplicate `validateColumnsFile` calls by removing the nested column
calls from this module dispatch path or by adding a caller-controlled flag;
retain current behavior for callers outside canonical module dispatch.
- [ ] **Step 8: Close global roots, injections, defaults, matches, and format
objects**
- Run the global-root key check before validating present containers.
- Run the injection key check before validating `row` and condition groups.
- Replace calls to `validateGlobalConditionList` with
`parseGlobalConditionGroups`; report its error through
`reportTopdataContractError`.
- Run default-rule and match-object key checks before existing type/value
checks.
- Treat an object containing `format` as a format object, reject every key
other than `format`, and preserve non-format object literals as payload
values.
- [ ] **Step 9: Run focused and full package tests**
```bash
nix develop --command go test ./internal/topdata \
-run 'TestValidateProject(RejectsUnsupportedCanonicalJSON|RejectsUnsupportedGlobalJSON|ValidatesEveryCanonicalModuleContainer|AcceptsCanonicalJSONContracts|AcceptsAnyPresent|RejectsInvalidAnyPresent)' \
-v
nix develop --command go test ./internal/topdata
```
Expected: pass.
- [ ] **Step 10: Commit validation tightening**
```bash
git add internal/topdata/topdata.go internal/topdata/json_contract_test.go
git commit -m "feat(topdata): reject unsupported canonical json properties"
```
---
### Task 3: Use shared condition parsing and `any_present` in native builds
**Files:**
- Modify: `internal/topdata/native.go`
- Modify: `internal/topdata/json_contract_test.go`
**Interfaces:**
- Consumes `parseGlobalConditionGroups` and
`globalConditionGroupsMatch` from Task 1.
- Removes the old boolean-mode
`globalConditionListMatches(name, raw, rows, required)` path.
- Direct `BuildNativeWithOptions` calls fail before output mutation when
canonical control syntax is invalid because the existing build entry point
runs `ValidateProject` first and native evaluation uses the same parser.
- [ ] **Step 1: Add failing native-build tests for `any_present` alternatives**
Use one compact plain-dataset fixture with:
- an authored row satisfying the first alternative;
- a second subtest satisfying only the later alternative;
- a third subtest satisfying neither alternative.
Assert the injected row exists in the first two outputs and is absent in the
third. Inspect the built 2DA content rather than internal slices.
- [ ] **Step 2: Add failing native-build tests for condition-group
conjunction**
Add subtests proving:
- `require_present` + `any_present` injects only when both pass;
- `any_present` + `unless_present` injects only when both pass;
- existing `require_present` all semantics remain unchanged;
- existing `unless_present` none semantics remain unchanged.
- [ ] **Step 3: Add a failing ordered-injection test**
Author injection 0 so it adds the row referenced by injection 1's
`any_present`. Assert injection 1 applies. This protects evaluation against
the same current-row view and ordered mutation used today.
- [ ] **Step 4: Add direct-build fail-closed tests**
Call `BuildNativeWithOptions` without a prior explicit `ValidateProject` call
for fixtures containing:
- `when_present` on an injection;
- an unknown condition key;
- an empty `any_present`;
- an unknown canonical root key.
Require an error containing the stable object/property fragments and assert
the expected output 2DA does not exist.
- [ ] **Step 5: Run focused tests and confirm failures**
```bash
nix develop --command go test ./internal/topdata \
-run 'TestBuildNative(AnyPresent|CombinesGlobalConditionGroups|UsesCurrentRowsForAnyPresent|RejectsUnsupportedGlobalControlSyntax|RejectsUnsupportedCanonicalRoot)' \
-v
```
Expected: `any_present` is ignored by current native evaluation and the new
parser/evaluator is not wired.
- [ ] **Step 6: Replace native condition evaluation with the shared contract**
Replace `globalInjectionConditionsMatch` and
`globalConditionListMatches` with:
```go
func globalInjectionConditionsMatch(
injection map[string]any,
rows []map[string]any,
) (bool, error) {
groups, err := parseGlobalConditionGroups(injection)
if err != nil {
return false, err
}
return globalConditionGroupsMatch(groups, rows), nil
}
```
Keep `applyPlainDatasetGlobalInjections` calling this function immediately
before deduplication/canonicalization, using `currentRows()` for each
injection.
- [ ] **Step 7: Run focused, package, and repository tests**
```bash
nix develop --command go test ./internal/topdata \
-run 'TestBuildNative(AnyPresent|CombinesGlobalConditionGroups|UsesCurrentRowsForAnyPresent|RejectsUnsupportedGlobalControlSyntax|RejectsUnsupportedCanonicalRoot)' \
-v
nix develop --command go test ./internal/topdata
nix develop --command go test ./...
```
Expected: pass.
- [ ] **Step 8: Commit native `any_present` support**
```bash
git add internal/topdata/native.go internal/topdata/json_contract_test.go
git commit -m "feat(topdata): add any-present injection conditions"
```
---
### Task 4: Document the global injection grammar
**Files:**
- Modify: `internal/topdata/CLASS_FEAT_GLOBAL_INJECTS_CONTRACT.md`
**Interfaces:**
- Documents the exact injection keys and all/any/none condition semantics.
- States that all present groups must pass and that conditions observe current
rows in injection order.
- States that `any_present` must be a non-empty array and `when_present` is
invalid.
- [ ] **Step 1: Add a documentation contract check**
Run this before editing:
```bash
rg -n 'any_present|when_present|all.*any.*none|require_present.*unless_present' \
internal/topdata/CLASS_FEAT_GLOBAL_INJECTS_CONTRACT.md
```
Expected: no complete description of `any_present` and group conjunction.
- [ ] **Step 2: Update the manifest shape and semantics**
Add an example containing all three groups:
```json
{
"row": {
"FeatIndex": { "id": "feat:example" }
},
"require_present": [{ "field": "FeatIndex", "id": "feat:required" }],
"any_present": [
{ "field": "FeatIndex", "id": "masterfeats:metamagic" },
{ "field": "FeatIndex", "id": "masterfeats:combat" }
],
"unless_present": [{ "field": "FeatIndex", "id": "feat:example" }]
}
```
Document:
- `require_present`: all listed references must exist;
- `any_present`: at least one listed reference must exist and the list cannot
be empty;
- `unless_present`: none of the listed references may exist;
- every present group must pass;
- earlier injections affect later conditions;
- supported injection keys are `row`, `require_present`, `any_present`, and
`unless_present`;
- `when_present` is rejected rather than aliased.
- [ ] **Step 3: Verify the documentation contains the complete contract**
```bash
rg -n 'require_present|any_present|unless_present|when_present|earlier injection|every present group' \
internal/topdata/CLASS_FEAT_GLOBAL_INJECTS_CONTRACT.md
```
Expected: each semantic point appears in the contract.
- [ ] **Step 4: Commit documentation**
```bash
git add internal/topdata/CLASS_FEAT_GLOBAL_INJECTS_CONTRACT.md
git commit -m "docs(topdata): define global injection conditions"
```
---
### Task 5: Validate the active topdata checkout and complete repository checks
**Files:**
- No authored files should change in `sow-tools`.
- Preserve existing sibling change:
`../sow-topdata/data/classes/feats/global.json`.
- Do not retain: `bin/`, `.cache/`, `generated/`, `result`, HAK/TLK files, or
temporary Crucible binaries.
**Interfaces:**
- The built Crucible validates the active `sow-topdata` checkout.
- The active checkout's single-condition correction uses `require_present`;
`any_present` remains available for future multi-alternative authoring.
- [ ] **Step 1: Format and run the full `sow-tools` checks**
```bash
nix develop --command gofmt -w \
internal/topdata/json_contract.go \
internal/topdata/json_contract_test.go \
internal/topdata/topdata.go \
internal/topdata/native.go
nix develop --command make check
```
Expected: vet, Go tests, shellcheck, and yamllint pass.
- [ ] **Step 2: Build a temporary Crucible binary without tracking it**
```bash
nix develop --command go build \
-o /tmp/crucible-topdata-json-validation \
./cmd/crucible
```
Expected: `/tmp/crucible-topdata-json-validation` exists outside the
repository.
- [ ] **Step 3: Confirm the active consumer syntax and repository state**
```bash
git -C ../sow-topdata status --short --branch
rg -n 'when_present|require_present|any_present|unless_present' \
../sow-topdata/data/classes/feats/global.json
```
Expected: the existing consumer branch contains `require_present` and no
`when_present`. Do not alter unrelated consumer work.
- [ ] **Step 4: Validate and build active `sow-topdata` with the changed
Crucible**
```bash
(
cd ../sow-topdata
nix develop --command /tmp/crucible-topdata-json-validation topdata validate
nix develop --command /tmp/crucible-topdata-json-validation topdata build
)
```
Expected: validation and build pass, global injections execute, and generated
outputs remain ignored.
- [ ] **Step 5: Verify no generated files became tracked**
```bash
git status --short
git -C ../sow-topdata status --short
git ls-files --error-unmatch bin generated .cache result 2>/dev/null
git -C ../sow-topdata ls-files --error-unmatch generated .cache result 2>/dev/null
```
Expected: the first two commands show only intentional source/doc changes;
the `git ls-files --error-unmatch` commands find no newly tracked generated
output. Remove only untracked artifacts created by this plan.
- [ ] **Step 6: Run final semantic searches and diff checks**
```bash
rg -n 'when_present' internal/topdata --glob '!docs/superpowers/**'
rg -n 'any_present' \
internal/topdata/json_contract.go \
internal/topdata/json_contract_test.go \
internal/topdata/CLASS_FEAT_GLOBAL_INJECTS_CONTRACT.md
git diff --check
git status --short --branch
git log --oneline origin/main..HEAD
```
Expected: `when_present` appears only in rejection tests/documentation;
implementation, tests, and contract document `any_present`; diff check is
clean.
- [ ] **Step 7: Remove the temporary binary**
```bash
rm -f /tmp/crucible-topdata-json-validation
```
- [ ] **Step 8: Push the feature branch and open the required pull request**
```bash
git push -u origin HEAD
tea pr create \
--title "feat: tighten topdata JSON validation" \
--description $'## Summary\n- reject unsupported canonical topdata JSON properties\n- add non-empty any_present global injection conditions\n- keep validation and direct builds on one shared contract\n\n## Verification\n- nix develop --command make check\n- active sow-topdata validate and build with the changed Crucible'
```
Expected: a PR targeting `main`; no production workflow or release tag is
triggered.
- [ ] **Step 9: Record the handoff**
Report:
- the `sow-tools` branch and PR URL;
- the existing `sow-topdata` consumer branch/commit used for integration;
- exact focused and full verification commands with outcomes;
- confirmation that no generated files were tracked;
- any diagnostics wording intentionally asserted as public semantic
contract.
@@ -0,0 +1,357 @@
# Crucible build parity — local == CI, zero-config reproducible
Date: 2026-06-21
Status: approved (design); implementation pending
Repos touched: `sow-tools` (Crucible + contract doc + image notes),
`sow-topdata` (config, flake, wrapper, CI), `sow-assets-manifest` (flake pin),
and the consumer wrapper/flake pattern shared by every Crucible repo.
Supersedes the resolver-script approach in
`2026-06-21-topdata-vfx-autogen-channel-design.md` (the channel→tag→`vfxs.yml`
resolution moves out of a bash wrapper and into Crucible).
## Problem
`crucible topdata build-topdata` run by hand does **not** reproduce the CI
build. CI runs `scripts/build-topdata.sh`, which resolves the accessory-VFX
model list with `scripts/resolve-accessory-vfx.sh` and writes
`.cache/sow-accessory-vfx-manifest.json` **before** Crucible runs; Crucible only
reads that pre-staged file (`nwn-tool.yaml: manifest_file:`). Run Crucible
directly and the file never exists, the `optional` consumer skips, and
`visualeffects.2da` ships **zero accessory rows** — exactly the local build the
operator observed (637 rows in CI, 0 locally).
Root cause: a **build-output-affecting input is resolved outside Crucible**, in
a wrapper script. The wrapper, not Crucible, is authoritative for that input.
## Goal
Crucible is the single authority for everything that lands in a build artifact.
A clone of any consumer repo, on any system, builds the same bytes CI builds —
with no secret beyond what cloning already required and no hand-written
configuration.
### Zero-config reproducibility requirements
- **R1 — No extra secrets.** All build-input reads are anonymous: the asset
channel pointer (`releases/haks/channels.json`) and `vfxs.yml` are public CDN
text. The crucible binary is fetchable anonymously — **verified**: the
sow-tools release assets return HTTP 200 without auth, and the bootstrap tries
anon before any token, so `CRUCIBLE_TOKEN` in consumer CI is **vestigial and is
removed**. Git-LFS asset bytes use the **same** credentials the clone already
used — never a separate key.
- **R2 — No hand-written config.** Every default needed to build is baked into
committed `nwn-tool.yaml` and Crucible defaults: CDN base, channel selection,
source paths. The user types `crucible topdata build-topdata` and nothing else.
- **R3 — No host data assumptions, no `NWN_ROOT`.** Builders read every input
from the project tree (`nwn-tool.yaml` + committed `data/**/base.json`); no NWN
install is required. `NWN_ROOT` is **dead** today — nothing in Crucible reads it
(see *NWN_ROOT cleanup* below). The accessory build reads
`data/visualeffects/base.json`. (Enforced by the CI parity guard, which runs in
a bare checkout with no `NWN_ROOT`.)
- **R4 — Nix is convenience, not a requirement.** The flake devshell provides
`crucible`, `git-lfs`, `jq`, `yq`, etc. on `PATH` for Nix users. Non-Nix users
(incl. Windows) get the same Crucible via the `crucible.sh`/`crucible.ps1`
bootstrap and a host `git-lfs`; the build path through Crucible is identical.
- **R5 — Same Crucible, both worlds.** Local dev and CI invoke the same Crucible
subcommand against the same committed config. No CI-only pre-steps that change
output.
## Parity contract (prevents recurrence)
Added to `sow-tools/docs/consumer-contract.md`. A consumer build wrapper may only:
1. **validate** — a pre-build gate that does not change output;
2. **publish** — post-build, on a separate artifact;
3. **resolve the crucible binary** — the bootstrap in `consumer-contract.md`.
A wrapper MUST NOT fetch, resolve, generate, or stage any input Crucible reads
to produce the artifact. If Crucible needs an input, Crucible acquires it. This
is the existing "no local-machine assumptions" + "same inputs → identical
output" rule, made explicit: **the wrapper is not allowed to be authoritative
for build inputs.**
## Design
### A. Accessory-VFX resolution moves into Crucible
**Config (`sow-topdata/nwn-tool.yaml`).** Drop `manifest_file:`; declare the
source on the consumer. The 4-group + `mdl` filter is **not** re-declared — it
is derived from the keys already present under `accessory_visualeffects.groups`:
```yaml
autogen:
consumers:
- id: accessory_visualeffects
producer: accessory_visualeffects
dataset: visualeffects
mode: accessory_visualeffects
optional: true
source:
kind: cdn_channel
cdn_base_env: BUNNY_CDN_BASE # default https://depot.westgate.pw
channels_path: releases/haks/channels.json
manifest_path: releases/haks/{tag}/vfxs.yml
release_marker_path: releases/haks/{tag}/haks.json # the broken-release guard probe
channel_env: SOW_TOPDATA_ASSET_CHANNEL # else derive from git tag
offline_override_env: SOW_VFXS_MANIFEST # dev/air-gapped vfxs.yml path
accessory_visualeffects:
groups: { chest_accessories: {...}, head_accessories: {...},
head_decorations: {...}, head_features: {...} }
# ... rest of the policy block unchanged
```
New struct field on `AutogenConsumerConfig` (sow-tools
`internal/project/project.go`): `Source AutogenSourceConfig` with the keys
above. Validation: when `source.kind == cdn_channel`, `channels_path` and
`manifest_path` are required and `manifest_path` must contain `{tag}`.
**Resolution (`internal/topdata/autogen.go`,
`resolveAutogenConsumerManifest`).** A new branch for `source.kind ==
cdn_channel`, reusing the existing `fetchJSON` HTTP client and the project YAML
parser:
1. **Offline override.** If `offline_override_env` names a readable file (or a
manifest-repo checkout root containing `assets/vfxs.yml`), parse it and skip
the network — the dev/air-gapped path.
2. **Channel.** `channel_env` if set; else from the git tag / `GITHUB_REF_NAME`
(`v*-*``testing`, `v*``current`, otherwise `current`).
3. **CDN base.** `cdn_base_env` value, else the baked default.
4. `GET {cdn}/{channels_path}` → JSON → `tag = channels[channel]`.
5. `GET {cdn}/{manifest_path with {tag}}` → parse YAML `assets[]`.
6. **Filter** to `restype == mdl` whose `path` is under `vfxs/<g>/` for `<g>` in
the `accessory_visualeffects.groups` keys. Derive each entry's `source`
(leading `vfxs/` stripped), `group`, `subgroup`, `model_stem`. Sort by
`source`. Hand the entries to the existing accessory augmentor unchanged.
**Fail policy (ported verbatim from the resolver — this is the v0.1.4-bug
guard).** Distinguish *unreachable* (fail open) from *broken* (hard fail):
- channels.json unreachable / not JSON / channel absent → **fail open**: the
`optional` consumer contributes no rows and `data/visualeffects/lock.json` IDs
are preserved.
- vfxs.yml network error, or 404 **and** the `release_marker_path` (haks.json)
is **absent** for that tag (no published release) → **fail open**.
- vfxs.yml 404 **and** haks.json **present****HARD fail** ("release `<tag>`
has haks.json but no vfxs.yml — accessory rows would silently vanish").
- vfxs.yml present but malformed / no `assets` array → **HARD fail**.
Fail-open writes nothing and lets the optional consumer skip; hard-fail aborts
the build with the message above.
**Deleted:** `sow-topdata/scripts/resolve-accessory-vfx.sh`,
`tests/resolve-accessory-vfx-contract.sh`, the `manifest_file:` key, and the
resolver pre-step at `build-topdata.sh:19`. The four bash contract cases
(offline override, unreachable fail-open, broken-release hard-fail, no-release
fail-open) are re-expressed as Go tests in `internal/topdata`.
### B. Git-LFS materialization moves into Crucible
So a bare clone + `crucible` packs real bytes, not pointer stubs (today only
`build-topdata.sh` pulls LFS, so direct Crucible would ship stubs in CI's
non-smudging checkout).
Before packing assets into a hak, Crucible: detects git-LFS pointer stubs under
the asset tree; if any, runs `git lfs install --local` then `git lfs pull` in
the repo; re-checks and **hard-fails** if stubs remain ("refusing to build from
pointer stubs"). Uses the clone's own credentials (R1). Requires `git` +
`git-lfs` on `PATH` — provided by the Nix devshell, a standard host install
otherwise (a clone without git-lfs already yields stubs, so this is the existing
host expectation, now enforced loudly by Crucible instead of a wrapper).
A skip control replaces `TOPDATA_SKIP_LFS` for the maintain-tree flow, which
regenerates the `data/` tree and **discards** the package: a Crucible flag/env
(e.g. `--skip-lfs` / `CRUCIBLE_SKIP_LFS`) so that one job can opt out. Default
is fail-closed (pull).
### C. Wrappers thin out
`scripts/build-topdata.sh` (and the shared pattern in every Crucible repo)
reduces to: validate (gate) → resolve crucible → `crucible <build>`. No input
resolution, no LFS pull. `sow-module/scripts/package-module.sh` already matches
this shape (validate → resolve → build); it stays the reference.
### D. CI parity guard (catches the next regression)
A `sow-tools` integration test builds the topdata package via Crucible **alone**
in a bare checkout (no wrapper, clean `.cache`, no `NWN_ROOT`, anonymous CDN)
and asserts `visualeffects.2da` contains accessory rows. This is the test that
would have caught today's bug: it fails if any build-output-affecting step ever
drifts back out of Crucible into a wrapper, and it enforces R2/R3/R5 mechanically.
The contract rule in `consumer-contract.md` is the written half; this test is
the enforced half.
### E. NWN_ROOT cleanup (stale, contradicts the goal)
`NWN_ROOT` is unused — no `os.Getenv`, no `--nwn-root` flag, no consumer. The
only traces are documentation/help-text describing a rule for a thing that does
not exist: the `## NWN_ROOT rule` section in `consumer-contract.md`, the help
text in `internal/dispatch/dispatch.go` ("if a builder ever needs an NWN
root… crucible never guesses from $HOME"), and the `README.md` pointer. That
old rule — *never guess, require an explicit env* — directly contradicts the
zero-config goal (R2): it would make a future builder demand a hand-set path.
- **Remove** the `NWN_ROOT` references: the `consumer-contract.md` section, the
dispatch help-text lines, the README pointer. Leave the
`migration-from-nwn-tool.md` bullet as historical record.
- **Replace the forward rule.** Should a future builder ever need NWN game data,
it **auto-detects** it (the NWN home — containing `modules/`, `hak/`, `tlk/`
and the Steam install path are reliably discoverable), with an optional
explicit override for non-standard layouts. Auto-detect first, never a required
hand-set env. This keeps "clone → run → Just Work" intact even for a builder
that one day reads game data. None do today.
### F. Toolchain convergence — one Crucible, three modes
Parity needs everyone to run the *same* Crucible. Today three mechanisms
diverge: non-Nix uses the `crucible.sh` bootstrap download; sow-assets-manifest's
flake pins `crucible v0.2.5` via `fetchurl`+hash; sow-topdata's flake ships no
crucible at all (falls back to the bootstrap, with a needless `CRUCIBLE_TOKEN`).
The pin is scattered (a hardcoded flake version, none in another flake, an image
sha in `prod.yml`, a HEAD-floating CI binary build).
Converge on **one binary, one pin, three modes — all anonymous, all identical**:
1. **Non-Nix (local + CI fallback).** `crucible.sh` / `crucible.ps1` bootstrap
downloads the release binary anonymously. `CRUCIBLE_TOKEN` is dropped from
consumer CI (R1).
2. **Nix (local + CI optimized).** Every consumer flake provides `crucible` in
its devshell from a **sow-tools flake input**, pinned by that repo's
`flake.lock`. Idiomatic, reproducible, no manual hashes; bump with
`nix flake update`. This replaces sow-assets-manifest's manual `fetchurl`
pin and **adds crucible to sow-topdata's devshell** (today it has none).
3. **CI** runs the *same* `crucible <build>` as local, inside the nix devshell
(binary-cache fast). No build container, no token, no CI-only pre-steps.
**Pin source of truth = `flake.lock` per repo**, bumped deliberately in a PR. We
**reject floating "latest"**: it would let local drift from CI between releases
and break reproducibility. The non-Nix bootstrap, which resolves "latest" by
default, is the convenience path; reproducible/parity-critical builds pin (nix
devshell, or `CRUCIBLE_*` pin env for the bootstrap).
### G. The crucible container is deployment-only
`sow-tools` builds and publishes `registry.westgate.pw/deployment/crucible:<sha>`
on `v*` tags (`nix build .#image``skopeo`). It exists for **deployment**
`prod.yml` pins it so tools travel with the runtime host (`nwn.enable`), and it
is a **disabled placeholder** until the NWN stack lands. No build/CI job consumes
it, and the host-mode runners have no container runtime to run it. It is **not a
build tool.**
- Keep the image build (cheap, `v*`-gated) for its deployment purpose.
- **Fix `consumer-contract.md`:** remove the crossed-out "run CI inside the
pinned image" guidance and its OPERATOR NOTE; state plainly that builds use the
binary (bootstrap or nix devshell) and the image is deployment-only. This kills
the "why do we build a container we don't use?" confusion.
### H. Autogen lock identity — model-anchored, immutable
**General lock contract (hard rule, all datasets).** A dataset lock maps a stable
key to a row id. The build MUST: reuse the existing id for a key that still
exists; allocate the first free id (lexicographic 0→1→2…) for a new key; and free
an id only when its key is gone everywhere. An id, once assigned to a live key,
never changes. (Already implemented for the accessory consumer via
`historicalLockData` reuse + `nextAvailableAutogenID`; this section makes it the
written rule and fixes the autogen-specific anchor below.)
**The autogen problem.** Autogen rows describe models that do **not** exist in
`sow-topdata/data/` — they live in the asset `vfxs.yml` manifest. Today the lock
key is the *config-derived presentation key*:
`KeyFormat = {dataset}:{group}{delimiter}{category_segment}{stem}`. Group token,
category, delimiter, case, and prefix-stripping all come from `nwn-tool.yaml`, so
a **config-only** change (`key_format`, `delimiter`, `case`, a group rename,
`category_from`, `strip_model_prefixes`) changes the key → the lock no longer
matches → a new id is allocated → **every row shifts**. That is a config edit
silently reshuffling ids for models that never changed.
**Fix — anchor the lock identity on the dataset + model, not the presentation.**
The lock key for an autogen row becomes `{dataset}:{model_source_path}` — the
stable `dataset` namespace prefix (kept as the 2da/namespace linker developers use
instead of raw ids) plus the **model source path** from `vfxs.yml`, e.g.
`visualeffects:head_accessories/hat/hfx_bandana.mdl` (leading `vfxs/` stripped —
the path is the value already on `entry.Source`). Dropped from the key are only
the config-derived parts (group token, `category_segment`, `delimiter`, case,
prefix-stripping). The row's `key`/`label` *columns* still derive from config via
`KeyFormat`/`LabelFormat` — only the **lock identity** is decoupled and
model-anchored. Consequences:
- A config-only change keeps every id (the path is unchanged).
- A model re-export (same path, new `sha256`) keeps its id — a content change is
not an identity change. (Audit trail comes free from `git diff` on the committed
`vfxs.yml`, which carries `path` + `sha256`; the lock stays a flat `string→int`
map — no schema change, no fork of the shared lock format. A future `id`+`sha`
object form is a clean later add if observability ever needs it, without
changing id semantics.)
- A renamed/moved/removed model is a genuinely different (or absent) identity.
**One-time cutover remap.** On the first build under `{dataset}:{path}` keys, for
each current model compute its *old* config-key (with the current policy) and its
new `{dataset}:{path}` key; if the new key is absent but the old key carries an id
in the lock, adopt that id under the new key. So ids carry over with zero shift on
cutover; the now-unreferenced old keys fall stale and are pruned (below).
**Stale pruning.** A lock key absent from the resolved entry set is dropped and
its id freed for first-free reuse — but **only on a successful, non-empty
resolution**. The fail-open paths (section A) contribute no entries and skip the
augmentor entirely, so an unreachable/missing manifest can never mass-prune the
lock. This realizes the general "free a fully-stale id" rule without risking the
lock on a transient source outage.
**Testing.** Go tests in `internal/topdata`: config-only change (rename a group
token, change `key_format`/`delimiter`/`case`) → ids unchanged; model re-export
(same path, different sha) → id unchanged; model rename/remove → old id freed and
reusable; cutover remap → existing ids preserved across the key-scheme switch;
fail-open → no pruning, lock untouched.
## Determinism vs. channel mutability (decision)
The committed topdata commit pins everything **except** the asset channel
pointer. `current` resolves through `channels.json`, which is intentionally
mutable, so "reproducible" here means **same commit + same channel state →
identical bytes**, not "this commit is frozen forever." This is deliberate: the
accessory rows must track the asset HAKs shipped beside `sow_top.hak`, which is
exactly what the channel pointer expresses. A given asset release **tag** is
immutable; the pointer moving is a real input change, and `lock.json` preserves
accessory IDs across it (no reshuffle). Builds against an explicit tag
(`SOW_TOPDATA_ASSET_CHANNEL` set, or a `v*` ref) are fully pinned.
This satisfies R1R5: no secret, no config, deterministic for a fixed input
set, on any system.
## Testing
- **Go unit tests (`sow-tools/internal/topdata`)** for the `cdn_channel`
resolution: offline-override path; unreachable channels.json → fail open;
channel absent → fail open; vfxs.yml 404 + no haks.json → fail open; vfxs.yml
404 + haks.json present → hard fail (asserts message); malformed vfxs.yml →
hard fail; happy path → correct filtered/sorted entries (mdl-only, 4-groups-
only, `vfxs/` stripped, group/subgroup/stem). Mirrors the deleted bash
contract cases.
- **Go config validation test** for the new `source` block.
- **CI parity guard** (section D) in the topdata build workflow.
- Existing accessory augmentor / lock-preserve tests unchanged.
## Rollout
1. sow-tools: add `AutogenSourceConfig`, the `cdn_channel` resolution branch,
LFS materialization, the model-anchored lock identity + cutover remap + stale
pruning (section H), the parity-guard test; update `consumer-contract.md`
(add the parity contract, remove the `NWN_ROOT` rule per section E, replace the
image guidance with deployment-only per section G).
2. Release Crucible (assets already anon-downloadable — R1).
3. Toolchain convergence (section F): expose `crucible` from a sow-tools flake
input in each consumer flake — add it to sow-topdata's devshell, switch
sow-assets-manifest off its manual `fetchurl` pin; drop `CRUCIBLE_TOKEN` from
consumer CI; CI builds inside the nix devshell.
4. sow-topdata: swap `manifest_file:``source:` in `nwn-tool.yaml`; delete the
resolver script + bash contract test; thin `build-topdata.sh` (validate →
resolve crucible → build); pin Crucible via `flake.lock`.
5. Verify: bare clone → `crucible topdata build-topdata` → 637 accessory rows,
no env, no key; and the same command inside `nix develop`.
## Follow-ups (not this work)
- Parts `2da` / `cachedmodels` reinstatement (separate spec) can reuse the same
`cdn_channel` source type.
- Roll the thinned-wrapper pattern review across `sow-module` / `sow-codebase`.
@@ -0,0 +1,147 @@
# Crucible Command Surface Cleanup Design
**Date:** 2026-06-25
**Status:** Approved
**Scope:** `sow-tools`, with the two active `sow-assets-manifest` call sites
updated in lockstep
## Problem
Crucible exposes migrated `nwn-tool` implementation names directly. Several
public commands repeat their builder name (`topdata build-topdata`,
`wiki build-wiki`), the interactive menu repeats builder-level descriptions
instead of explaining each action, and selected commands only prompt for
unexplained "extra args".
The registry also exposes a migrated music-conversion subsystem that is not part
of the current project workflow. Live HAK builds explicitly bypass it with
`--skip-music`; authored `.bmu` files are already packed as ordinary assets.
## Decisions
### Canonical command names
The visible command surface is:
| Builder | Visible commands |
| --- | --- |
| `hak` | `build`, `manifest` |
| `module` | `build`, `extract`, `validate`, `compare`, `manifest` |
| `topdata` | `validate`, `build`, `package`, `compare`, `convert` |
| `wiki` | `build`, `deploy` |
The dispatcher translates these names to the existing `internal/app`
implementation commands. The app implementation names do not need to be
renamed.
### Hidden compatibility aliases
Existing long names remain callable indefinitely but are omitted from the
interactive menu and routine builder help:
- `hak build-haks`
- `hak apply-hak-manifest`
- `module build-module`
- `module apply-hak-manifest`
- `topdata validate-topdata`
- `topdata build-topdata`
- `topdata build-top-package`
- `topdata compare-topdata`
- `topdata convert-topdata`
- `wiki build-wiki`
- `wiki deploy-wiki`
Aliases preserve their existing implementation target. In particular,
`module build-module` continues to run the module-only implementation; it must
not silently become the broader `module build`.
### Registry model
`internal/dispatch.Registry` remains the single source of truth. Each wired
builder owns command records containing:
- visible command name;
- action-oriented summary;
- underlying `internal/app` command;
- command usage and option guidance;
- hidden compatibility aliases, where applicable.
Execution, menu items, and builder help are generated from these records.
Unknown commands continue to fail with exit `64`.
### Interactive behavior
The main menu:
- lists only visible commands;
- aligns the description column from the longest visible label;
- shows a distinct description for every command.
After selection, it prints the selected command's usage and available options,
then prompts:
```text
arguments (press Enter to use defaults):
```
Arguments continue to use shell-like whitespace splitting, matching current
behavior. Quoted argument parsing is out of scope.
`crucible <builder> <command> --help` prints the same command-specific guidance,
returns success, and performs no project loading or build work.
### Remove the unused music pipeline
The music conversion subsystem is deleted rather than hidden:
- remove the `music` app and dispatcher commands;
- remove `internal/music` and `internal/pipeline/music.go`;
- remove music config, effective config, validation, environment overrides, and
project accessors;
- remove HAK music preparation, conversion, credits, generated-manifest result
fields, `--skip-music`, and `--music-dataset`;
- remove ffmpeg/ffprobe runtime and development dependencies;
- remove music-pipeline documentation and wrapper wording.
`.bmu` remains a supported ordinary asset extension and continues through the
same content-addressed HAK packing path as other authored files. Actual game
content, category names, or prose that happens to use the word "music" is not
part of this deletion.
The default asset-extension list drops conversion-only `.mp3` and `.ogg`.
`.wav` remains because it is an NWN resource type independent of the deleted
conversion pipeline.
Because `sow-assets-manifest` currently passes `--skip-music`, its
`scripts/pack-haks.sh` and `scripts/build-local-haks.sh` call sites are updated
in the same change before verification.
## Error handling
- Canonical and hidden alias lookup is deterministic within one builder.
- Registry tests reject duplicate visible names or aliases.
- Command help never delegates to `internal/app`.
- Removed music commands and flags fail as unknown usage.
- Existing project files containing a top-level `music:` configuration become
invalid under strict YAML decoding. No active consumer configuration contains
that field.
## Testing
Tests assert public contracts rather than exact incidental formatting:
- visible command names and hidden alias translation;
- preservation of the distinct `module build-module` target;
- hidden aliases absent from menu and normal help;
- command-specific help exits successfully without project setup;
- menu columns align and selected-command guidance is displayed;
- `.bmu` remains accepted as an asset;
- `.mp3`/`.ogg` are no longer default packable assets;
- HAK builds no longer expose or execute music preparation;
- ffmpeg is absent from package/image definitions;
- both consumer scripts no longer pass `--skip-music`.
Final verification is `nix develop --command make check`, `make build`,
`make smoke`, focused consumer tests, and repository-wide searches for deleted
pipeline identifiers.
@@ -0,0 +1,269 @@
# Topdata JSON Validation Tightening Design
**Date:** 2026-06-25
**Status:** Approved design; implementation pending
**Scope:** `sow-tools`
## Problem
Topdata JSON validation checks many required properties and value types, but it
often ignores unknown structural properties. This allows plausible typos to
pass validation and then be ignored by the builder.
The immediate failure occurred in
`topdata/data/classes/feats/global.json`: an injection used
`when_present` even though the implemented condition is `require_present`.
Validation succeeded, the builder ignored `when_present`, and the injection was
therefore applied without the intended condition.
This is a fail-open authoring contract. A misspelled build-control property must
not silently change generated game data.
## Goals
- Reject unsupported structural properties in canonical topdata JSON files.
- Add an `any_present` global injection condition.
- Keep standalone validation and direct native builds consistent.
- Preserve dataset-specific row flexibility.
- Tighten only established canonical formats; do not turn this work into a
repository-wide schema rewrite.
- Produce diagnostics that identify the file, object, and unsupported property.
## Non-goals
- No schema migration or formatting change to authored topdata.
- No generic JSON Schema framework or new validation dependency.
- No blanket closure of every specialized JSON dialect.
- No change to generated feat family, registry, spellbook, TLK, or other
specialized formats unless they use the canonical containers covered here.
- No restriction of row values beyond existing column, metadata, inheritance,
reference, TLK, and authoring-sugar contracts.
- No compatibility alias for `when_present`; it is invalid syntax.
## Design
### Validation boundary
Structural objects use closed property sets. Dataset payload objects remain
dataset-aware.
Closed structural objects include:
| Object | Supported properties |
| --- | --- |
| Base or plain rows-file root | `output`, `key`, `columns`, `rows` |
| Canonical module root | `output`, `key`, `columns`, `entries`, `overrides`, `rows` |
| `global.json` root | `columns`, `entries`, `overrides`, `defaults`, `position`, `injections` |
| Global injection | `row`, `require_present`, `any_present`, `unless_present` |
| Global condition | `field`, `id` |
| Global default rule | `match`, `values` |
| Global default match object | `source` |
| Global default format object | `format` |
The allowed root set is selected from the file's existing canonical context;
properties from unrelated formats are not accepted merely because another
topdata file type supports them.
When a canonical root contains more than one supported container, each present
container must be validated. For example, a module containing both `entries`
and `overrides` must not stop validation after recognizing `entries`.
The following remain payload objects rather than global allowlists:
- objects inside `rows`;
- values inside `entries`;
- row-like objects inside `overrides`;
- an injection's `row`;
- a default rule's `values`.
Those objects continue to accept declared dataset columns plus documented row
control and metadata fields. The builder already rejects unknown row and entry
columns when it canonicalizes a dataset. This work must reuse or align with
that knowledge where practical, but must not invent a single fixed row schema
across all datasets.
### Moderate strictness
This change closes only canonical containers whose grammar is already
established by the parser and current authored data.
Before adding a closed root check, implementation must inventory active
`sow-topdata` files and existing `sow-tools` fixtures for legitimate property
combinations. A property used by an established canonical format must be added
to that format's explicit set rather than removed from authored data merely to
satisfy the validator.
Specialized parsers remain responsible for their own object shapes. Extending
closed-property checks to those formats is separate work.
### Global injection conditions
Global injections support three condition groups:
- `require_present`: every listed condition must match a current row.
- `any_present`: at least one listed condition must match a current row.
- `unless_present`: no listed condition may match a current row.
When more than one group is present, every group must pass. Conditions continue
to use the existing shape:
```json
{
"field": "FeatIndex",
"id": "feat:example"
}
```
Example:
```json
{
"row": {
"FeatIndex": {
"id": "feat:example"
}
},
"require_present": [
{
"field": "FeatIndex",
"id": "feat:base_requirement"
}
],
"any_present": [
{
"field": "FeatIndex",
"id": "masterfeats:metamagic"
},
{
"field": "FeatIndex",
"id": "masterfeats:combat"
}
],
"unless_present": [
{
"field": "FeatIndex",
"id": "feat:example"
}
]
}
```
This injection applies only when the required feat exists, either listed
masterfeat exists, and the injected feat does not already exist.
`any_present` must contain at least one condition. An empty list is an
authoring error because it cannot express a useful successful condition.
Existing empty-list behavior for `require_present` and `unless_present` is not
changed by this work.
Conditions are evaluated against the same current-row view and in the same
injection order used today. This preserves the existing behavior where an
earlier injection can affect a later injection's conditions.
### Validation and build consistency
`ValidateProject` must report unsupported structural properties before a build.
The native builder must also fail closed when invoked directly with the same
invalid control syntax.
The implementation should centralize:
- deterministic unsupported-property checks;
- global condition parsing and validation;
- condition group evaluation.
Validation may collect multiple diagnostics while building returns the first
blocking error. Both paths must accept and reject the same property names and
condition shapes.
### Diagnostics
Diagnostics must name the containing object and unsupported property. Property
lists must be deterministic.
For the original error, an acceptable diagnostic is:
```text
global injection 10 contains unsupported key "when_present";
supported keys are row, require_present, any_present, and unless_present
```
Equivalent concise wording is acceptable if tests assert the stable semantic
parts rather than the entire sentence.
## Compatibility
Current canonical `sow-topdata` roots fit the property sets above. The
implementation must rerun the inventory immediately before tightening checks
and run validation against the active `sow-topdata` checkout.
Unknown properties are not retained as compatibility behavior. They currently
have no defined effect, and silently accepting them is the bug being fixed.
Existing documented properties and dataset-specific row fields remain valid.
`require_present` and `unless_present` retain their current semantics.
`any_present` is additive.
## Testing
Tests must follow red-green development and cover validation and direct-build
paths.
### Unsupported properties
- Reject an unknown `global.json` root property.
- Reject `when_present` and other unknown injection properties.
- Reject an unknown condition property.
- Reject unknown properties in default rules, match objects, and format
objects.
- Reject unknown canonical base/plain rows-file root properties.
- Reject unknown canonical module root properties.
- Continue accepting legitimate combinations such as:
- a columns-only module;
- entries-only and overrides-only modules;
- a module containing both entries and overrides;
- current base and plain rows-file roots.
- Continue accepting dataset columns and documented row control fields inside
rows, entries, overrides, and injection rows.
### `any_present`
- Validation accepts a non-empty `any_present` condition list.
- Validation rejects a non-array or empty `any_present`.
- A build injects when the first alternative is present.
- A build injects when a later alternative is present.
- A build skips the injection when no alternative is present.
- `any_present` combines correctly with `require_present`.
- `any_present` combines correctly with `unless_present`.
- Existing `require_present` and `unless_present` behavior remains unchanged.
- Direct building rejects unsupported condition syntax even when project
validation was not called first.
### Integration
- Run focused `internal/topdata` tests.
- Run `nix develop --command make check`.
- Validate the active `sow-topdata` checkout with the changed Crucible.
- Build topdata far enough to exercise global injections and confirm no
generated files are tracked.
## Documentation
Update the class feat global injection contract to document `require_present`,
`any_present`, and `unless_present`, including their all/any/none semantics and
how multiple groups combine.
No broad validation guide is added unless implementation reveals an existing
operator document that would otherwise become stale.
## Acceptance criteria
- `when_present` in a global injection fails validation and direct building.
- `any_present` is valid only as a non-empty list of valid conditions.
- `any_present` applies an injection when one or more alternatives exist and
skips it when none exist.
- Unknown canonical structural properties fail with actionable diagnostics.
- Dataset-specific row fields remain valid when declared by the dataset.
- Existing active `sow-topdata` validates and builds after replacing invalid
syntax with the documented grammar.
- No specialized topdata format is tightened outside the stated scope.
+4 -5
View File
@@ -10,12 +10,11 @@
in in
{ {
# Self-contained (D8): a host with ONLY nix can enter this shell and run # Self-contained (D8): a host with ONLY nix can enter this shell and run
# `make check`. ffmpeg is present because the migrated music pipeline needs # `make check`.
# it; the scaffold itself is pure stdlib Go.
# Daemonless image build (D8): `nix build .#image` produces an OCI tarball # Daemonless image build (D8): `nix build .#image` produces an OCI tarball
# with no docker/podman daemon. CI loads/pushes it with skopeo. This is the # with no docker/podman daemon. CI loads/pushes it with skopeo. This is the
# Nix-native replacement for `docker build` on the host-mode runner, which # Nix-native replacement for `docker build` on the host-mode runner, which
# has no container runtime. ffmpeg-headless ships the BMU codec path. # has no container runtime.
packages = forAllSystems (pkgs: packages = forAllSystems (pkgs:
let let
version = self.shortRev or self.dirtyShortRev or "unknown"; version = self.shortRev or self.dirtyShortRev or "unknown";
@@ -49,7 +48,7 @@
image = pkgs.dockerTools.buildLayeredImage { image = pkgs.dockerTools.buildLayeredImage {
name = "registry.westgate.pw/deployment/crucible"; name = "registry.westgate.pw/deployment/crucible";
tag = version; tag = version;
contents = [ crucible pkgs.cacert pkgs.ffmpeg-headless pkgs.fakeNss ]; contents = [ crucible pkgs.cacert pkgs.fakeNss ];
config = { config = {
Entrypoint = [ "/bin/crucible" ]; Entrypoint = [ "/bin/crucible" ];
Cmd = [ "help" ]; Cmd = [ "help" ];
@@ -69,8 +68,8 @@
go go
gopls gopls
gotools gotools
ffmpeg
git git
tea
gnumake gnumake
bashInteractive bashInteractive
shellcheck shellcheck
+39 -213
View File
@@ -9,7 +9,6 @@ import (
"os/exec" "os/exec"
"path/filepath" "path/filepath"
"runtime" "runtime"
"sort"
"strconv" "strconv"
"strings" "strings"
"sync" "sync"
@@ -184,11 +183,6 @@ var commands = []command{
description: "Inspect, explain, and validate the resolved project configuration.", description: "Inspect, explain, and validate the resolved project configuration.",
run: runConfig, run: runConfig,
}, },
{
name: "music",
description: "List, scan, build, and validate configured music datasets.",
run: runMusic,
},
{ {
name: "compare", name: "compare",
description: "Compare current source content against the built archives.", description: "Compare current source content against the built archives.",
@@ -469,8 +463,7 @@ type buildHAKOptions struct {
filteredHAKs []string filteredHAKs []string
filteredArchives []string filteredArchives []string
sourceManifest string sourceManifest string
musicDatasets []string contentAddressedRoot string
skipMusic bool
planOnly bool planOnly bool
logLevel logLevel logLevel logLevel
} }
@@ -720,46 +713,11 @@ func (c *buildHAKConsole) emitResult(result pipeline.BuildResult) {
spin.linebreak() spin.linebreak()
fmt.Fprintln(c.stdout, "Build HAKs ----------") fmt.Fprintln(c.stdout, "Build HAKs ----------")
if c.level != logLevelQuiet { if c.level != logLevelQuiet {
c.emitCreditsSummary(result)
c.emitCoreSteps(result) c.emitCoreSteps(result)
} }
c.emitFinalSummary(result) c.emitFinalSummary(result)
} }
func (c *buildHAKConsole) emitCreditsSummary(result pipeline.BuildResult) {
summary := result.CreditsSummary
if len(summary.ArtifactPaths) == 0 {
return
}
fmt.Fprintln(c.stdout, "Refreshing credits cache")
if len(summary.ScannedDirs) > 0 {
fmt.Fprintf(c.stdout, " scanned: %s\n", strings.Join(summary.ScannedDirs, ", "))
}
fmt.Fprintf(c.stdout, " tracks: %d\n", summary.TrackCount)
if len(summary.GeneratedPaths) > 0 {
for _, path := range summary.GeneratedPaths {
fmt.Fprintf(c.stdout, " output: %s\n", c.relPath(path))
}
}
switch {
case summary.ChangedFiles == 0:
fmt.Fprintln(c.stdout, " status: unchanged")
default:
fmt.Fprintf(c.stdout, " updated: %d artifact(s)\n", summary.ChangedFiles)
}
if summary.TrackCount > 0 {
if c.level >= logLevelVerbose {
fmt.Fprintln(c.stdout, " mappings:")
for _, mapping := range summary.Mappings {
fmt.Fprintf(c.stdout, " %s -> %s\n", mapping.Source, mapping.Output)
}
} else {
fmt.Fprintf(c.stdout, " mapped: %d music file(s); use --verbose to list mappings\n", summary.TrackCount)
}
}
fmt.Fprintln(c.stdout)
}
func (c *buildHAKConsole) emitCoreSteps(result pipeline.BuildResult) { func (c *buildHAKConsole) emitCoreSteps(result pipeline.BuildResult) {
fmt.Fprintln(c.stdout, "Validating project") fmt.Fprintln(c.stdout, "Validating project")
fmt.Fprintln(c.stdout, "Collecting asset resources") fmt.Fprintln(c.stdout, "Collecting asset resources")
@@ -807,9 +765,6 @@ func (c *buildHAKConsole) emitFinalSummary(result pipeline.BuildResult) {
if result.Manifest != "" { if result.Manifest != "" {
fmt.Fprintf(c.stdout, "manifest: %s\n", c.relPath(result.Manifest)) fmt.Fprintf(c.stdout, "manifest: %s\n", c.relPath(result.Manifest))
} }
if len(result.CreditsArtifactPaths) > 0 {
fmt.Fprintf(c.stdout, "credits artifacts: %d\n", len(result.CreditsArtifactPaths))
}
if len(result.AutogenManifestPaths) > 0 { if len(result.AutogenManifestPaths) > 0 {
fmt.Fprintf(c.stdout, "autogen manifests: %d\n", len(result.AutogenManifestPaths)) fmt.Fprintf(c.stdout, "autogen manifests: %d\n", len(result.AutogenManifestPaths))
} }
@@ -1048,8 +1003,7 @@ func runBuildHAKs(ctx context) error {
Progress: console.progress, Progress: console.progress,
ArchiveNames: opts.filteredArchives, ArchiveNames: opts.filteredArchives,
SourceManifestPath: opts.sourceManifest, SourceManifestPath: opts.sourceManifest,
SkipMusic: opts.skipMusic, ContentAddressedRoot: opts.contentAddressedRoot,
MusicDatasetIDs: opts.musicDatasets,
} }
if opts.planOnly { if opts.planOnly {
result, err = pipeline.PlanHAKsWithOptions(p, pipelineOpts) result, err = pipeline.PlanHAKsWithOptions(p, pipelineOpts)
@@ -1074,7 +1028,7 @@ func parseBuildHAKArgs(args []string) (buildHAKOptions, error) {
arg := args[index] arg := args[index]
switch arg { switch arg {
case "-h", "--help": case "-h", "--help":
return opts, errors.New("usage: build-haks [--hak <hak-name> ...] [--archive <archive-name> ...] [--source-manifest <path>] [--plan-only] [--skip-music] [--music-dataset <id> ...] [--quiet|--verbose|--debug]") return opts, errors.New("usage: build-haks [--hak <hak-name> ...] [--archive <archive-name> ...] [--source-manifest <path>] [--content-addressed-root <path>] [--plan-only] [--quiet|--verbose|--debug]")
case "--hak": case "--hak":
index++ index++
if index >= len(args) { if index >= len(args) {
@@ -1089,14 +1043,6 @@ func parseBuildHAKArgs(args []string) (buildHAKOptions, error) {
opts.filteredArchives = append(opts.filteredArchives, args[index]) opts.filteredArchives = append(opts.filteredArchives, args[index])
case "--plan-only": case "--plan-only":
opts.planOnly = true opts.planOnly = true
case "--skip-music":
opts.skipMusic = true
case "--music-dataset":
index++
if index >= len(args) {
return opts, errors.New("--music-dataset requires a value")
}
opts.musicDatasets = append(opts.musicDatasets, args[index])
case "--quiet": case "--quiet":
opts.logLevel = logLevelQuiet opts.logLevel = logLevelQuiet
case "--verbose": case "--verbose":
@@ -1109,6 +1055,12 @@ func parseBuildHAKArgs(args []string) (buildHAKOptions, error) {
return opts, errors.New("--source-manifest requires a value") return opts, errors.New("--source-manifest requires a value")
} }
opts.sourceManifest = args[index] opts.sourceManifest = args[index]
case "--content-addressed-root":
index++
if index >= len(args) {
return opts, errors.New("--content-addressed-root requires a value")
}
opts.contentAddressedRoot = args[index]
default: default:
if value, ok, err := requireInlineFlagValue(arg, "--hak"); ok || err != nil { if value, ok, err := requireInlineFlagValue(arg, "--hak"); ok || err != nil {
if err != nil { if err != nil {
@@ -1131,11 +1083,11 @@ func parseBuildHAKArgs(args []string) (buildHAKOptions, error) {
opts.sourceManifest = value opts.sourceManifest = value
continue continue
} }
if value, ok, err := requireInlineFlagValue(arg, "--music-dataset"); ok || err != nil { if value, ok, err := requireInlineFlagValue(arg, "--content-addressed-root"); ok || err != nil {
if err != nil { if err != nil {
return opts, err return opts, err
} }
opts.musicDatasets = append(opts.musicDatasets, value) opts.contentAddressedRoot = value
continue continue
} }
if arg == "--quiet=true" { if arg == "--quiet=true" {
@@ -1167,149 +1119,6 @@ func requireInlineFlagValue(arg, name string) (string, bool, error) {
return value, true, nil return value, true, nil
} }
type musicCommandOptions struct {
datasets []string
json bool
dryRun bool
check bool
force bool
}
func runMusic(ctx context) error {
if len(ctx.args) < 2 {
return errors.New("usage: music <list-datasets|scan|build|credits|validate|manifest|normalize> [--dataset <id>] [--json] [--dry-run] [--check] [--force]")
}
p, err := loadProject(ctx)
if err != nil {
return err
}
if err := p.Scan(); err != nil {
return err
}
subcommand := ctx.args[1]
opts, err := parseMusicCommandArgs(ctx.args[2:])
if err != nil {
return err
}
switch subcommand {
case "list-datasets":
return emitMusicDatasets(ctx, p, opts)
case "scan":
result, err := pipeline.BuildMusic(p, pipeline.MusicBuildOptions{DatasetIDs: opts.datasets, Write: false})
if err != nil {
return err
}
return emitMusicResult(ctx, "scan", result, opts)
case "build", "credits":
write := true
if subcommand == "build" && opts.dryRun {
write = false
}
if subcommand == "credits" && opts.check {
write = false
}
result, err := pipeline.BuildMusic(p, pipeline.MusicBuildOptions{DatasetIDs: opts.datasets, Write: write})
if err != nil {
return err
}
return emitMusicResult(ctx, subcommand, result, opts)
case "manifest", "normalize":
result, err := pipeline.BuildMusic(p, pipeline.MusicBuildOptions{DatasetIDs: opts.datasets, Write: false})
if err != nil {
return err
}
return emitMusicResult(ctx, subcommand, result, opts)
case "validate":
if err := p.ValidateLayout(); err != nil {
return err
}
result, err := pipeline.BuildMusic(p, pipeline.MusicBuildOptions{DatasetIDs: opts.datasets, Write: false})
if err != nil {
return err
}
return emitMusicResult(ctx, "validate", result, opts)
default:
return fmt.Errorf("unknown music subcommand %q", subcommand)
}
}
func parseMusicCommandArgs(args []string) (musicCommandOptions, error) {
opts := musicCommandOptions{}
for index := 0; index < len(args); index++ {
arg := args[index]
switch arg {
case "--dataset":
index++
if index >= len(args) || strings.TrimSpace(args[index]) == "" {
return opts, errors.New("--dataset requires a value")
}
opts.datasets = append(opts.datasets, args[index])
case "--json":
opts.json = true
case "--dry-run":
opts.dryRun = true
case "--check":
opts.check = true
case "--force":
opts.force = true
default:
if value, ok, err := requireInlineFlagValue(arg, "--dataset"); ok || err != nil {
if err != nil {
return opts, err
}
opts.datasets = append(opts.datasets, value)
continue
}
return opts, fmt.Errorf("unknown music argument %q", arg)
}
}
return opts, nil
}
func emitMusicDatasets(ctx context, p *project.Project, opts musicCommandOptions) error {
effective := p.EffectiveConfig()
if opts.json {
return emitConfigValue(ctx.stdout, effective.Music.Datasets, "json")
}
if len(effective.Music.Datasets) == 0 {
fmt.Fprintln(ctx.stdout, "music datasets: none")
return nil
}
fmt.Fprintln(ctx.stdout, "music datasets:")
ids := make([]string, 0, len(effective.Music.Datasets))
for id := range effective.Music.Datasets {
ids = append(ids, id)
}
sort.Strings(ids)
for _, id := range ids {
ds := effective.Music.Datasets[id]
fmt.Fprintf(ctx.stdout, " %s: source=%s output=%s prefix=%s\n", id, ds.Source, ds.Output, ds.Prefix)
}
return nil
}
func emitMusicResult(ctx context, action string, result pipeline.BuildResult, opts musicCommandOptions) error {
if opts.json {
return emitConfigValue(ctx.stdout, result.CreditsSummary, "json")
}
fmt.Fprintf(ctx.stdout, "music %s\n", action)
if len(result.CreditsSummary.ScannedDirs) > 0 {
fmt.Fprintf(ctx.stdout, "scanned: %s\n", strings.Join(result.CreditsSummary.ScannedDirs, ", "))
}
fmt.Fprintf(ctx.stdout, "tracks: %d\n", result.CreditsSummary.TrackCount)
if action != "scan" && len(result.CreditsArtifactPaths) > 0 {
fmt.Fprintf(ctx.stdout, "credits artifacts: %d\n", len(result.CreditsArtifactPaths))
}
if len(result.CreditsSummary.Mappings) > 0 && ctx.logLevel >= logLevelVerbose {
fmt.Fprintln(ctx.stdout, "mappings:")
for _, mapping := range result.CreditsSummary.Mappings {
fmt.Fprintf(ctx.stdout, " %s -> %s\n", mapping.Source, mapping.Output)
}
}
fmt.Fprintln(ctx.stdout, "status: ok")
return nil
}
func runExtract(ctx context) error { func runExtract(ctx context) error {
p, err := loadProject(ctx) p, err := loadProject(ctx)
if err != nil { if err != nil {
@@ -1605,16 +1414,20 @@ func runBuildTopData(ctx context) error {
return err return err
} }
opts, err := parseBuildTopDataArgs("build-topdata", ctx.args[1:]) parsed, err := parseBuildTopDataArgs("build-topdata", ctx.args[1:])
if err != nil { if err != nil {
return err return err
} }
if parsed.skipLFS {
os.Setenv("CRUCIBLE_SKIP_LFS", "1") //nolint:errcheck
}
console := newTopdataConsole(ctx, p, "build-topdata") console := newTopdataConsole(ctx, p, "build-topdata")
spin.configure(ctx.stderr, console.spinnerEnabled) spin.configure(ctx.stderr, console.spinnerEnabled)
spin.start("Build Topdata: starting") spin.start("Build Topdata: starting")
defer spin.stop() defer spin.stop()
result, err := topdata.BuildAndPackageWithOptions(p, opts, console.progress) result, err := topdata.BuildAndPackageWithOptions(p, parsed.opts, console.progress)
if err != nil { if err != nil {
return err return err
} }
@@ -1629,15 +1442,19 @@ func runBuildTopPackage(ctx context) error {
return err return err
} }
opts, err := parseBuildTopDataArgs("build-top-package", ctx.args[1:]) parsed, err := parseBuildTopDataArgs("build-top-package", ctx.args[1:])
if err != nil { if err != nil {
return err return err
} }
if opts.BuildWiki { if parsed.opts.BuildWiki {
return fmt.Errorf("--wiki is not supported with build-top-package; use build-topdata when wiki generation is required") return fmt.Errorf("--wiki is not supported with build-top-package; use build-topdata when wiki generation is required")
} }
if parsed.skipLFS {
os.Setenv("CRUCIBLE_SKIP_LFS", "1") //nolint:errcheck
}
console := newTopdataConsole(ctx, p, "build-top-package") console := newTopdataConsole(ctx, p, "build-top-package")
spin.configure(ctx.stderr, console.spinnerEnabled) spin.configure(ctx.stderr, console.spinnerEnabled)
spin.start("Build Top Package: starting") spin.start("Build Top Package: starting")
@@ -1651,21 +1468,30 @@ func runBuildTopPackage(ctx context) error {
return nil return nil
} }
func parseBuildTopDataArgs(commandName string, args []string) (topdata.BuildAndPackageOptions, error) { type buildTopDataArgs struct {
opts := topdata.BuildAndPackageOptions{} opts topdata.BuildAndPackageOptions
skipLFS bool
}
func parseBuildTopDataArgs(commandName string, args []string) (buildTopDataArgs, error) {
var parsed buildTopDataArgs
for _, arg := range args { for _, arg := range args {
switch arg { switch arg {
case "--force": case "--force":
opts.Force = true parsed.opts.Force = true
case "--wiki": case "--wiki":
opts.BuildWiki = true parsed.opts.BuildWiki = true
case "--skip-lfs":
// ponytail: CRUCIBLE_SKIP_LFS env is the single skip mechanism; set it at
// the CLI entry point rather than carrying a dead field through BuildAndPackageOptions.
parsed.skipLFS = true
case "-h", "--help": case "-h", "--help":
return opts, fmt.Errorf("usage: %s [--force] [--wiki]", commandName) return parsed, fmt.Errorf("usage: %s [--force] [--wiki] [--skip-lfs]", commandName)
default: default:
return opts, fmt.Errorf("unknown %s argument %q", commandName, arg) return parsed, fmt.Errorf("unknown %s argument %q", commandName, arg)
} }
} }
return opts, nil return parsed, nil
} }
func runCompareTopData(ctx context) error { func runCompareTopData(ctx context) error {
+92 -207
View File
@@ -2,6 +2,7 @@ package app
import ( import (
"bytes" "bytes"
"encoding/json"
"os" "os"
"path/filepath" "path/filepath"
"strings" "strings"
@@ -11,6 +12,70 @@ import (
"git.westgate.pw/ShadowsOverWestgate/sow-tools/internal/pipeline" "git.westgate.pw/ShadowsOverWestgate/sow-tools/internal/pipeline"
) )
func TestParseBuildHAKArgsContentAddressedRoot(t *testing.T) {
tests := []struct {
name string
args []string
want string
}{
{
name: "separated",
args: []string{"--content-addressed-root", "/var/cache/blobs"},
want: "/var/cache/blobs",
},
{
name: "inline",
args: []string{"--content-addressed-root=/var/cache/blobs"},
want: "/var/cache/blobs",
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
opts, err := parseBuildHAKArgs(tt.args)
if err != nil {
t.Fatalf("parse build-haks args: %v", err)
}
if opts.contentAddressedRoot != tt.want {
t.Fatalf("content-addressed root = %q, want %q", opts.contentAddressedRoot, tt.want)
}
})
}
}
func TestParseBuildHAKArgsHelpListsContentAddressedRoot(t *testing.T) {
_, err := parseBuildHAKArgs([]string{"--help"})
if err == nil {
t.Fatal("expected help usage error")
}
for _, flag := range []string{
"--hak",
"--archive",
"--source-manifest",
"--content-addressed-root",
"--plan-only",
"--quiet",
"--verbose",
"--debug",
} {
if !strings.Contains(err.Error(), flag) {
t.Errorf("help usage missing documented flag %q: %v", flag, err)
}
}
}
func TestParseBuildHAKArgsRejectsRemovedMusicFlags(t *testing.T) {
for _, args := range [][]string{
{"--skip-music"},
{"--music-dataset", "westgate"},
{"--music-dataset=westgate"},
} {
if _, err := parseBuildHAKArgs(args); err == nil {
t.Errorf("expected removed music arguments %v to fail", args)
}
}
}
func TestRunBuildTopPackageUsesCachedCompiledOutputs(t *testing.T) { func TestRunBuildTopPackageUsesCachedCompiledOutputs(t *testing.T) {
root := t.TempDir() root := t.TempDir()
mkdirAll(t, filepath.Join(root, "build")) mkdirAll(t, filepath.Join(root, "build"))
@@ -43,9 +108,8 @@ topdata:
setFileTime(t, filepath.Join(root, ".cache", "2da", "repadjust.2da"), outputTime) setFileTime(t, filepath.Join(root, ".cache", "2da", "repadjust.2da"), outputTime)
setFileTime(t, filepath.Join(root, "build", "sow_tlk.tlk"), outputTime) setFileTime(t, filepath.Join(root, "build", "sow_tlk.tlk"), outputTime)
var stdout bytes.Buffer
ctx := context{ ctx := context{
stdout: &stdout, stdout: &bytes.Buffer{},
stderr: &bytes.Buffer{}, stderr: &bytes.Buffer{},
cwd: root, cwd: root,
args: []string{"build-top-package"}, args: []string{"build-top-package"},
@@ -54,19 +118,12 @@ topdata:
if err := runBuildTopPackage(ctx); err != nil { if err := runBuildTopPackage(ctx); err != nil {
t.Fatalf("runBuildTopPackage failed: %v", err) t.Fatalf("runBuildTopPackage failed: %v", err)
} }
output := stdout.String()
if !strings.Contains(output, "top package hak: build/sow_top.hak") {
t.Fatalf("expected build-top-package output, got %q", output)
}
if strings.Contains(output, "[build-top-package]") {
t.Fatalf("did not expect raw progress lines in normal output, got %q", output)
}
if _, err := os.Stat(filepath.Join(root, "build", "sow_top.hak")); err != nil { if _, err := os.Stat(filepath.Join(root, "build", "sow_top.hak")); err != nil {
t.Fatalf("expected packaged hak output: %v", err) t.Fatalf("expected packaged hak output: %v", err)
} }
} }
func TestRunBuildHAKsEmitsCompactSummary(t *testing.T) { func TestRunBuildHAKsPacksAuthoredBMU(t *testing.T) {
root := t.TempDir() root := t.TempDir()
mkdirAll(t, filepath.Join(root, "assets", "envi", "music", "westgate")) mkdirAll(t, filepath.Join(root, "assets", "envi", "music", "westgate"))
mkdirAll(t, filepath.Join(root, "build")) mkdirAll(t, filepath.Join(root, "build"))
@@ -88,23 +145,7 @@ haks:
include: include:
- envi/** - envi/**
`) `)
writeFile(t, filepath.Join(root, "assets", "envi", "music", "westgate", "AleandAnecdotes.mp3"), "source-mp3") writeFile(t, filepath.Join(root, "assets", "envi", "music", "westgate", "westgate_theme.bmu"), "authored-bmu")
writeFile(t, filepath.Join(root, "assets", "envi", "music", "westgate", "CREDITS.md"), "# NWN Music Pack Credits\n\n| Artist | Title | Output File | Original File | Album | Date | License / Copyright | Notes |\n|---|---|---|---|---|---|---|---|\n| Darren Curtis | Ale and Anecdotes | `mus_wg_andnc.bmu` | `AleandAnecdotes.mp3` | Darren's Commercially Free Music | | Darren Curtis Music | |\n")
ffprobePath := filepath.Join(root, "ffprobe")
writeFile(t, ffprobePath, "#!/bin/sh\nprintf '%s\\n' '{\"format\":{\"tags\":{\"artist\":\"Unknown\",\"title\":\"Broken Metadata\"}}}'\n")
if err := os.Chmod(ffprobePath, 0o755); err != nil {
t.Fatalf("chmod ffprobe: %v", err)
}
ffmpegPath := filepath.Join(root, "ffmpeg")
writeFile(t, ffmpegPath, "#!/bin/sh\nfor last; do :; done\nprintf 'bmu-data' > \"$last\"\n")
if err := os.Chmod(ffmpegPath, 0o755); err != nil {
t.Fatalf("chmod ffmpeg: %v", err)
}
t.Setenv("SOW_FFMPEG", ffmpegPath)
t.Setenv("SOW_FFPROBE", ffprobePath)
var stdout bytes.Buffer var stdout bytes.Buffer
ctx := context{ ctx := context{
@@ -118,18 +159,12 @@ haks:
t.Fatalf("runBuildHAKs failed: %v", err) t.Fatalf("runBuildHAKs failed: %v", err)
} }
output := stdout.String() manifest, err := os.ReadFile(filepath.Join(root, "build", "haks.json"))
if !strings.Contains(output, "Build HAKs ----------") { if err != nil {
t.Fatalf("expected build header, got %q", output) t.Fatalf("read HAK manifest: %v", err)
} }
if !strings.Contains(output, "mapped: 1 music file(s); use --verbose to list mappings") { if !strings.Contains(string(manifest), "envi/music/westgate/westgate_theme.bmu") {
t.Fatalf("expected compact mapping summary, got %q", output) t.Fatalf("authored BMU missing from HAK manifest:\n%s", manifest)
}
if strings.Contains(output, "AleandAnecdotes.mp3 -> mus_wg_andnc.bmu") {
t.Fatalf("did not expect verbose mapping in normal mode, got %q", output)
}
if !strings.Contains(output, "manifest: build/haks.json") {
t.Fatalf("expected relative manifest path, got %q", output)
} }
} }
@@ -146,70 +181,6 @@ func setTreeTime(t *testing.T, root string, modTime time.Time) {
} }
} }
func TestRunBuildHAKsVerboseListsMappings(t *testing.T) {
root := t.TempDir()
mkdirAll(t, filepath.Join(root, "assets", "envi", "music", "westgate"))
mkdirAll(t, filepath.Join(root, "build"))
mkdirAll(t, filepath.Join(root, "src"))
writeFile(t, filepath.Join(root, "nwn-tool.yaml"), `
module:
name: Test Module
resref: testmod
paths:
source: src
assets: assets
build: build
haks:
- name: envi
priority: 1
max_bytes: 1048576
split: false
include:
- envi/**
`)
writeFile(t, filepath.Join(root, "assets", "envi", "music", "westgate", "AleandAnecdotes.mp3"), "source-mp3")
writeFile(t, filepath.Join(root, "assets", "envi", "music", "westgate", "CREDITS.md"), "# NWN Music Pack Credits\n\n| Artist | Title | Output File | Original File | Album | Date | License / Copyright | Notes |\n|---|---|---|---|---|---|---|---|\n| Darren Curtis | Ale and Anecdotes | `mus_wg_andnc.bmu` | `AleandAnecdotes.mp3` | Darren's Commercially Free Music | | Darren Curtis Music | |\n")
ffprobePath := filepath.Join(root, "ffprobe")
writeFile(t, ffprobePath, "#!/bin/sh\nprintf '%s\\n' '{\"format\":{\"tags\":{\"artist\":\"Unknown\",\"title\":\"Broken Metadata\"}}}'\n")
if err := os.Chmod(ffprobePath, 0o755); err != nil {
t.Fatalf("chmod ffprobe: %v", err)
}
ffmpegPath := filepath.Join(root, "ffmpeg")
writeFile(t, ffmpegPath, "#!/bin/sh\nfor last; do :; done\nprintf 'bmu-data' > \"$last\"\n")
if err := os.Chmod(ffmpegPath, 0o755); err != nil {
t.Fatalf("chmod ffmpeg: %v", err)
}
t.Setenv("SOW_FFMPEG", ffmpegPath)
t.Setenv("SOW_FFPROBE", ffprobePath)
var stdout bytes.Buffer
ctx := context{
stdout: &stdout,
stderr: &bytes.Buffer{},
cwd: root,
args: []string{"build-haks", "--verbose"},
}
if err := runBuildHAKs(ctx); err != nil {
t.Fatalf("runBuildHAKs failed: %v", err)
}
output := stdout.String()
if !strings.Contains(output, "mappings:") {
t.Fatalf("expected verbose mappings header, got %q", output)
}
if !strings.Contains(output, "AleandAnecdotes.mp3 -> mus_wg_andnc.bmu") {
t.Fatalf("expected verbose mapping output, got %q", output)
}
if !strings.Contains(output, "wrote: envi (1 assets)") {
t.Fatalf("expected verbose archive action, got %q", output)
}
}
func TestTopdataConsoleSuppressesProgressInNormalMode(t *testing.T) { func TestTopdataConsoleSuppressesProgressInNormalMode(t *testing.T) {
var stdout bytes.Buffer var stdout bytes.Buffer
console := &topdataConsole{ console := &topdataConsole{
@@ -250,16 +221,10 @@ func TestTopdataConsoleDebugProgressAndRelativePaths(t *testing.T) {
console.emitWikiDeployResult(10, 1, 2, 3, 4, 5, 6, 0, "/workspace/project/build/wiki/deploy-manifest.json") console.emitWikiDeployResult(10, 1, 2, 3, 4, 5, 6, 0, "/workspace/project/build/wiki/deploy-manifest.json")
output := stdout.String() output := stdout.String()
if !strings.Contains(output, "[debug] NodeBB wiki plan: create 1, update 2, skip 3, stale 4, archive 5, purge 6, drift 0") { if !strings.Contains(output, "NodeBB wiki plan") {
t.Fatalf("expected debug progress line, got %q", output) t.Fatalf("expected debug progress line, got %q", output)
} }
if !strings.Contains(output, "archived: 5") { if !strings.Contains(output, "build/wiki/deploy-manifest.json") || strings.Contains(output, "/workspace/project/") {
t.Fatalf("expected archived deploy count, got %q", output)
}
if !strings.Contains(output, "purged: 6") {
t.Fatalf("expected purged deploy count, got %q", output)
}
if !strings.Contains(output, "manifest: build/wiki/deploy-manifest.json") {
t.Fatalf("expected relative deploy manifest path, got %q", output) t.Fatalf("expected relative deploy manifest path, got %q", output)
} }
} }
@@ -317,85 +282,11 @@ func TestProjectConsoleEmitsRelativePaths(t *testing.T) {
}) })
output := stdout.String() output := stdout.String()
if !strings.Contains(output, "module: build/test.mod") { if !strings.Contains(output, "build/test.mod") || strings.Contains(output, "/workspace/project/") {
t.Fatalf("expected relative module path, got %q", output) t.Fatalf("expected relative module path, got %q", output)
} }
} }
func TestRunMusicScanUsesConfiguredDatasetsWithoutWriting(t *testing.T) {
root := t.TempDir()
mkdirAll(t, filepath.Join(root, "assets", "audio", "westgate"))
mkdirAll(t, filepath.Join(root, "build"))
writeFile(t, filepath.Join(root, "nwn-tool.yaml"), `
module:
name: Test Module
resref: testmod
paths:
assets: assets
build: build
music:
datasets:
westgate_audio:
source: audio/westgate
output: generated/music
prefix: wg_
`)
writeFile(t, filepath.Join(root, "assets", "audio", "westgate", "Theme Song.mp3"), "source-mp3")
var stdout bytes.Buffer
ctx := context{
stdout: &stdout,
stderr: &bytes.Buffer{},
cwd: root,
args: []string{"music", "scan", "--dataset", "westgate_audio"},
}
if err := runMusic(ctx); err != nil {
t.Fatalf("runMusic failed: %v", err)
}
output := stdout.String()
if !strings.Contains(output, "music scan") || !strings.Contains(output, "tracks: 1") {
t.Fatalf("unexpected music scan output: %q", output)
}
if _, err := os.Stat(filepath.Join(root, ".cache", "credits")); !os.IsNotExist(err) {
t.Fatalf("music scan should not write credits artifacts, stat err=%v", err)
}
}
func TestRunMusicListDatasets(t *testing.T) {
root := t.TempDir()
mkdirAll(t, filepath.Join(root, "assets"))
mkdirAll(t, filepath.Join(root, "build"))
writeFile(t, filepath.Join(root, "nwn-tool.yaml"), `
module:
name: Test Module
resref: testmod
paths:
assets: assets
music:
datasets:
westgate_audio:
source: audio/westgate
output: generated/music
prefix: wg_
`)
var stdout bytes.Buffer
ctx := context{
stdout: &stdout,
stderr: &bytes.Buffer{},
cwd: root,
args: []string{"music", "list-datasets"},
}
if err := runMusic(ctx); err != nil {
t.Fatalf("runMusic failed: %v", err)
}
if !strings.Contains(stdout.String(), "westgate_audio: source=audio/westgate output=generated/music prefix=wg_") {
t.Fatalf("unexpected dataset list: %q", stdout.String())
}
}
func TestRunConfigEffectiveJSONShowsDefaultsAndProvenance(t *testing.T) { func TestRunConfigEffectiveJSONShowsDefaultsAndProvenance(t *testing.T) {
root := t.TempDir() root := t.TempDir()
writeFile(t, filepath.Join(root, "nwn-tool.yaml"), ` writeFile(t, filepath.Join(root, "nwn-tool.yaml"), `
@@ -417,12 +308,16 @@ paths:
if err := runConfig(ctx); err != nil { if err := runConfig(ctx); err != nil {
t.Fatalf("runConfig failed: %v", err) t.Fatalf("runConfig failed: %v", err)
} }
output := stdout.String() var effective map[string]any
if !strings.Contains(output, `"hak_manifest": "haks.json"`) { if err := json.Unmarshal(stdout.Bytes(), &effective); err != nil {
t.Fatalf("expected HAK manifest default in effective config, got %q", output) t.Fatalf("effective config is not JSON: %v\n%s", err, stdout.String())
} }
if !strings.Contains(output, `"paths.build":`) || !strings.Contains(output, `"toolkit default"`) { provenance, ok := effective["provenance"].(map[string]any)
t.Fatalf("expected default provenance in effective config, got %q", output) if !ok || len(provenance) == 0 {
t.Fatalf("effective config missing provenance: %#v", effective["provenance"])
}
if _, ok := provenance["paths.build"]; !ok {
t.Fatalf("effective config missing provenance for omitted paths.build: %#v", provenance)
} }
} }
@@ -448,10 +343,10 @@ paths:
t.Fatalf("runConfig failed: %v", err) t.Fatalf("runConfig failed: %v", err)
} }
output := stdout.String() output := stdout.String()
if !strings.Contains(output, "value: \"output\"") { if !strings.Contains(output, "output") {
t.Fatalf("expected configured build value, got %q", output) t.Fatalf("expected configured build value, got %q", output)
} }
if !strings.Contains(output, "source: yaml") { if !strings.Contains(strings.ToLower(output), "yaml") {
t.Fatalf("expected YAML source, got %q", output) t.Fatalf("expected YAML source, got %q", output)
} }
} }
@@ -468,9 +363,8 @@ paths:
build: build build: build
`) `)
var stdout bytes.Buffer
ctx := context{ ctx := context{
stdout: &stdout, stdout: &bytes.Buffer{},
stderr: &bytes.Buffer{}, stderr: &bytes.Buffer{},
cwd: root, cwd: root,
args: []string{"config", "validate"}, args: []string{"config", "validate"},
@@ -479,9 +373,6 @@ paths:
if err := runConfig(ctx); err != nil { if err := runConfig(ctx); err != nil {
t.Fatalf("runConfig failed: %v", err) t.Fatalf("runConfig failed: %v", err)
} }
if !strings.Contains(stdout.String(), "config: ok") {
t.Fatalf("expected config validation output, got %q", stdout.String())
}
} }
func TestRunConfigSourcesListsActiveOverrides(t *testing.T) { func TestRunConfigSourcesListsActiveOverrides(t *testing.T) {
@@ -507,25 +398,19 @@ paths:
t.Fatalf("runConfig failed: %v", err) t.Fatalf("runConfig failed: %v", err)
} }
output := stdout.String() output := stdout.String()
if !strings.Contains(output, "active overrides:") { if !strings.Contains(output, "build.keep_existing_haks") {
t.Fatalf("expected active overrides section, got %q", output)
}
if !strings.Contains(output, "build.keep_existing_haks=1") {
t.Fatalf("expected keep existing override, got %q", output) t.Fatalf("expected keep existing override, got %q", output)
} }
} }
func TestInlineFlagParsersRejectEmptyValues(t *testing.T) { func TestInlineFlagParsersRejectEmptyValues(t *testing.T) {
if _, err := parseBuildHAKArgs([]string{"--hak="}); err == nil || !strings.Contains(err.Error(), "--hak requires a value") { if _, err := parseBuildHAKArgs([]string{"--hak="}); err == nil {
t.Fatalf("expected empty --hak inline value error, got %v", err) t.Fatalf("expected empty --hak inline value error, got %v", err)
} }
if _, err := parseMusicCommandArgs([]string{"--dataset="}); err == nil || !strings.Contains(err.Error(), "--dataset requires a value") { if _, err := parseDeployWikiArgs("deploy-wiki", []string{"--endpoint="}); err == nil {
t.Fatalf("expected empty --dataset inline value error, got %v", err)
}
if _, err := parseDeployWikiArgs("deploy-wiki", []string{"--endpoint="}); err == nil || !strings.Contains(err.Error(), "--endpoint requires a value") {
t.Fatalf("expected empty --endpoint inline value error, got %v", err) t.Fatalf("expected empty --endpoint inline value error, got %v", err)
} }
if _, err := parseBuildChangelogArgs("build-changelog", []string{"--output="}); err == nil || !strings.Contains(err.Error(), "--output requires a value") { if _, err := parseBuildChangelogArgs("build-changelog", []string{"--output="}); err == nil {
t.Fatalf("expected empty --output inline value error, got %v", err) t.Fatalf("expected empty --output inline value error, got %v", err)
} }
} }
+11 -5
View File
@@ -178,11 +178,17 @@ func TestGenerateIncludesPullRequestsAndDirectPushes(t *testing.T) {
} }
rendered := stdout.String() rendered := stdout.String()
if !strings.Contains(rendered, "- Add release summary ([#12]("+repoURL+"/pulls/12)) - From Patch Author") { for _, want := range []string{
t.Fatalf("rendered changelog missing pull entry:\n%s", rendered) "Add release summary",
repoURL + "/pulls/12",
"Patch Author",
"Fix direct push handling",
repoURL + "/commit/" + directHash,
"Test User",
} {
if !strings.Contains(rendered, want) {
t.Errorf("rendered changelog missing %q:\n%s", want, rendered)
} }
if !strings.Contains(rendered, "- Fix direct push handling (["+shortCommitHash(directHash)+"]("+repoURL+"/commit/"+directHash+")) - From Test User") {
t.Fatalf("rendered changelog missing direct-push entry:\n%s", rendered)
} }
if pullRequests != 1 { if pullRequests != 1 {
t.Fatalf("expected 1 pull lookup, got %d", pullRequests) t.Fatalf("expected 1 pull lookup, got %d", pullRequests)
@@ -192,7 +198,7 @@ func TestGenerateIncludesPullRequestsAndDirectPushes(t *testing.T) {
func runGit(t *testing.T, repoRoot string, args ...string) string { func runGit(t *testing.T, repoRoot string, args ...string) string {
t.Helper() t.Helper()
cmd := exec.Command("git", args...) cmd := exec.Command("git", append([]string{"-c", "commit.gpgsign=false", "-c", "tag.gpgsign=false"}, args...)...)
cmd.Dir = repoRoot cmd.Dir = repoRoot
output, err := cmd.CombinedOutput() output, err := cmd.CombinedOutput()
if err != nil { if err != nil {
+229 -34
View File
@@ -2,7 +2,7 @@
// builders shared by the `crucible` dispatcher and the standalone // builders shared by the `crucible` dispatcher and the standalone
// `crucible-<name>` shims. // `crucible-<name>` shims.
// //
// Cutover state (Phase 5 → 6): the internal pipeline/topdata/erf/wiki/music // Cutover state (Phase 5 → 6): the internal pipeline/topdata/erf/wiki
// packages migrated from gitea/sow-tools now live in this tree, so wired // packages migrated from gitea/sow-tools now live in this tree, so wired
// builders delegate to internal/app's legacy command surface (mapped per // builders delegate to internal/app's legacy command surface (mapped per
// docs/command-surface.md). A builder with no migrated logic yet (depot) keeps // docs/command-surface.md). A builder with no migrated logic yet (depot) keeps
@@ -13,7 +13,6 @@ import (
"fmt" "fmt"
"io" "io"
"os" "os"
"text/tabwriter"
"git.westgate.pw/ShadowsOverWestgate/sow-tools/internal/app" "git.westgate.pw/ShadowsOverWestgate/sow-tools/internal/app"
"git.westgate.pw/ShadowsOverWestgate/sow-tools/internal/buildinfo" "git.westgate.pw/ShadowsOverWestgate/sow-tools/internal/buildinfo"
@@ -33,22 +32,56 @@ type Builder struct {
Name string // canonical dispatcher subcommand, e.g. "module" Name string // canonical dispatcher subcommand, e.g. "module"
Bin string // standalone binary name, e.g. "crucible-module" Bin string // standalone binary name, e.g. "crucible-module"
Summary string // one-line description Summary string // one-line description
Legacy []string // nwn-tool commands this subsumes (migration aid; see docs/command-surface.md) Commands []Command // visible commands plus hidden compatibility aliases
Extra []string // additional callable subcommands beyond Legacy (e.g. cross-listed "music")
Wired bool // true once the builder delegates to migrated internal/app logic Wired bool // true once the builder delegates to migrated internal/app logic
} }
// subcommands returns every legacy command a wired builder accepts: the // Command is one public builder action. AppCommand names the unchanged
// canonical Legacy set plus any cross-listed Extra commands. // internal/app implementation command. Aliases remain callable for compatibility
func (b Builder) subcommands() []string { return append(append([]string{}, b.Legacy...), b.Extra...) } // but are omitted from routine help and the interactive menu.
type Command struct {
Name string
Summary string
AppCommand string
Usage string
Options []string
Aliases []CommandAlias
}
type CommandAlias struct {
Name string
AppCommand string
}
func (b Builder) subcommands() []string {
commands := make([]string, 0, len(b.Commands))
for _, command := range b.Commands {
commands = append(commands, command.Name)
}
return commands
}
func (b Builder) command(name string) (Command, string, bool) {
for _, command := range b.Commands {
if command.Name == name {
return command, command.AppCommand, true
}
for _, alias := range command.Aliases {
if alias.Name == name {
target := alias.AppCommand
if target == "" {
target = command.AppCommand
}
return command, target, true
}
}
}
return Command{}, "", false
}
func (b Builder) accepts(sub string) bool { func (b Builder) accepts(sub string) bool {
for _, s := range b.subcommands() { _, _, ok := b.command(sub)
if s == sub { return ok
return true
}
}
return false
} }
// Registry is the single source of truth for the Crucible command surface. // Registry is the single source of truth for the Crucible command surface.
@@ -58,39 +91,171 @@ var Registry = []Builder{
Name: "depot", Name: "depot",
Bin: "crucible-depot", Bin: "crucible-depot",
Summary: "verify/move content-addressed depot blobs (SeaweedFS)", Summary: "verify/move content-addressed depot blobs (SeaweedFS)",
Legacy: nil,
Wired: false, // no migrated logic yet; fails closed (exit 70) Wired: false, // no migrated logic yet; fails closed (exit 70)
}, },
{ {
Name: "hak", Name: "hak",
Bin: "crucible-hak", Bin: "crucible-hak",
Summary: "pack/unpack ERF/HAK archives + hak manifests", Summary: "pack/unpack ERF/HAK archives + hak manifests",
Legacy: []string{"build-haks", "apply-hak-manifest"}, Commands: []Command{
Extra: []string{"music"}, // music BMUs are packed into HAKs (command-surface.md) {
Name: "build",
Summary: "build configured HAK archives and their manifest",
AppCommand: "build-haks",
Usage: "crucible hak build [options]",
Options: []string{
"--hak <name> build selected configured HAKs",
"--archive <name> build selected archive members",
"--source-manifest <path> read a generated source manifest",
"--content-addressed-root <path> resolve source-manifest blobs here",
"--plan-only plan outputs without writing archives",
"--quiet | --verbose | --debug select output detail",
},
Aliases: []CommandAlias{{Name: "build-haks"}},
},
{
Name: "manifest",
Summary: "apply a generated HAK list to the module source",
AppCommand: "apply-hak-manifest",
Usage: "crucible hak manifest [manifest-path]",
Aliases: []CommandAlias{{Name: "apply-hak-manifest"}},
},
},
Wired: true, Wired: true,
}, },
{ {
Name: "module", Name: "module",
Bin: "crucible-module", Bin: "crucible-module",
Summary: "build/extract/validate/compare the .mod", Summary: "build/extract/validate/compare the .mod",
Legacy: []string{"build", "build-module", "extract", "validate", "compare"}, Commands: []Command{
// apply-hak-manifest is canonically a hak command but reachable here too; {
// music is folded into the module build pipeline (command-surface.md). Name: "build",
Extra: []string{"apply-hak-manifest", "music"}, Summary: "build the module and configured project outputs",
AppCommand: "build",
Usage: "crucible module build [--quiet|--verbose|--debug]",
Aliases: []CommandAlias{{Name: "build-module", AppCommand: "build-module"}},
},
{
Name: "extract",
Summary: "extract built archives into configured source trees",
AppCommand: "extract",
Usage: "crucible module extract [resource ...]",
},
{
Name: "validate",
Summary: "validate project layout, config, and source inventory",
AppCommand: "validate",
Usage: "crucible module validate",
},
{
Name: "compare",
Summary: "compare source content with built module and HAK archives",
AppCommand: "compare",
Usage: "crucible module compare",
},
{
Name: "manifest",
Summary: "apply a generated HAK list to the module source",
AppCommand: "apply-hak-manifest",
Usage: "crucible module manifest [manifest-path]",
Aliases: []CommandAlias{{Name: "apply-hak-manifest"}},
},
},
Wired: true, Wired: true,
}, },
{ {
Name: "topdata", Name: "topdata",
Bin: "crucible-topdata", Bin: "crucible-topdata",
Summary: "compile 2da/tlk topdata + packages", Summary: "compile 2da/tlk topdata + packages",
Legacy: []string{"validate-topdata", "build-topdata", "build-top-package", "compare-topdata", "convert-topdata"}, Commands: []Command{
{
Name: "validate",
Summary: "validate topdata layout and canonical JSON compatibility",
AppCommand: "validate-topdata",
Usage: "crucible topdata validate",
Aliases: []CommandAlias{{Name: "validate-topdata"}},
},
{
Name: "build",
Summary: "compile topdata and refresh the configured package",
AppCommand: "build-topdata",
Usage: "crucible topdata build [--force] [--wiki] [--skip-lfs]",
Options: []string{
"--force rebuild outputs even when current",
"--wiki build wiki drafts after compilation",
"--skip-lfs skip Git LFS materialization",
},
Aliases: []CommandAlias{{Name: "build-topdata"}},
},
{
Name: "package",
Summary: "package cached topdata outputs into HAK and TLK files",
AppCommand: "build-top-package",
Usage: "crucible topdata package [--force] [--skip-lfs]",
Options: []string{
"--force rebuild outputs even when current",
"--skip-lfs skip Git LFS materialization",
},
Aliases: []CommandAlias{{Name: "build-top-package"}},
},
{
Name: "compare",
Summary: "compare current output with a fresh native build",
AppCommand: "compare-topdata",
Usage: "crucible topdata compare",
Aliases: []CommandAlias{{Name: "compare-topdata"}},
},
{
Name: "convert",
Summary: "convert between 2DA and native JSON/module formats",
AppCommand: "convert-topdata",
Usage: "crucible topdata convert <2da-to-json|2da-to-module|json-to-2da> ...",
Options: []string{
"2da-to-json <input.2da> <output.json>",
"2da-to-module [flags] <input.2da> [output.json]",
"json-to-2da <input.json> <output.2da>",
},
Aliases: []CommandAlias{{Name: "convert-topdata"}},
},
},
Wired: true, Wired: true,
}, },
{ {
Name: "wiki", Name: "wiki",
Bin: "crucible-wiki", Bin: "crucible-wiki",
Summary: "render + deploy mechanical wiki pages", Summary: "render + deploy mechanical wiki pages",
Legacy: []string{"build-wiki", "deploy-wiki"}, Commands: []Command{
{
Name: "build",
Summary: "render wiki page drafts from compiled topdata",
AppCommand: "build-wiki",
Usage: "crucible wiki build [--force]",
Options: []string{"--force rebuild drafts even when current"},
Aliases: []CommandAlias{{Name: "build-wiki"}},
},
{
Name: "deploy",
Summary: "deploy generated wiki pages to NodeBB",
AppCommand: "deploy-wiki",
Usage: "crucible wiki deploy [options]",
Options: []string{
"--source-dir <path> read generated pages here",
"--endpoint <url> override the NodeBB endpoint",
"--token <token> authenticate with an API token",
"--username/--password <value> authenticate with credentials",
"--version <version> label the deployed revision",
"--namespace <name> deploy selected namespaces",
"--category <namespace=id> override a namespace category",
"--manifest <path> write deployment state here",
"--stale-policy <policy> report, archive, or purge",
"--dry-run report changes without writing",
"--create allow missing pages to be created",
"--force update unchanged pages",
"--reset-managed-namespaces reset managed namespace state",
},
Aliases: []CommandAlias{{Name: "deploy-wiki"}},
},
},
Wired: true, Wired: true,
}, },
} }
@@ -129,10 +294,14 @@ func menuItems() []menu.Item {
continue continue
} }
for _, sub := range b.subcommands() { for _, sub := range b.subcommands() {
command, _, _ := b.command(sub)
items = append(items, menu.Item{ items = append(items, menu.Item{
Label: b.Name + " " + sub, Label: b.Name + " " + sub,
Desc: b.Summary, Desc: command.Summary,
Args: []string{b.Name, sub}, Args: []string{b.Name, sub},
Usage: command.Usage,
Options: append([]string(nil),
command.Options...),
}) })
} }
} }
@@ -207,14 +376,21 @@ func runBuilder(name string, args []string, out, errw io.Writer) int {
return exitUsage return exitUsage
} }
sub := args[0] sub := args[0]
if !b.accepts(sub) { command, appCommand, ok := b.command(sub)
if !ok {
fmt.Fprintf(errw, "crucible %s: unknown subcommand %q\n\n", b.Name, sub) fmt.Fprintf(errw, "crucible %s: unknown subcommand %q\n\n", b.Name, sub)
builderHelp(errw, b) builderHelp(errw, b)
return exitUsage return exitUsage
} }
// Delegate to the migrated legacy command surface. args[0] is already the if len(args) > 1 {
// legacy command name, so it is forwarded unchanged. switch args[1] {
return delegateLegacy(args, errw) case "-h", "--help", "help":
commandHelp(out, command)
return exitOK
}
}
legacyArgs := append([]string{appCommand}, args[1:]...)
return delegateLegacy(legacyArgs, errw)
} }
// delegateLegacy runs a migrated nwn-tool command via internal/app and maps its // delegateLegacy runs a migrated nwn-tool command via internal/app and maps its
@@ -249,16 +425,16 @@ func usage(w io.Writer) {
fmt.Fprintf(w, " list list builders (one per line: name<TAB>bin<TAB>summary)\n") fmt.Fprintf(w, " list list builders (one per line: name<TAB>bin<TAB>summary)\n")
fmt.Fprintf(w, " config [args] inspect/validate effective configuration\n") fmt.Fprintf(w, " config [args] inspect/validate effective configuration\n")
fmt.Fprintf(w, " changelog [args] generate the release changelog\n") fmt.Fprintf(w, " changelog [args] generate the release changelog\n")
fmt.Fprintf(w, "\nNWN_ROOT must be passed explicitly (env or flag); crucible never guesses\n") fmt.Fprintf(w, "\nBuilders read all inputs from the project tree (nwn-tool.yaml + data/); no\n")
fmt.Fprintf(w, "from $HOME. See docs/consumer-contract.md.\n") fmt.Fprintf(w, "NWN install is required. A builder that one day needs NWN game data\n")
fmt.Fprintf(w, "auto-detects the install; it never requires a hand-set path. See\n")
fmt.Fprintf(w, "docs/consumer-contract.md.\n")
} }
func list(w io.Writer) { func list(w io.Writer) {
tw := tabwriter.NewWriter(w, 0, 2, 2, ' ', 0)
for _, b := range Registry { for _, b := range Registry {
fmt.Fprintf(tw, " %s\t%s\t%s\n", b.Name, b.Bin, b.Summary) fmt.Fprintf(w, "%s\t%s\t%s\n", b.Name, b.Bin, b.Summary)
} }
tw.Flush()
} }
func builderHelp(w io.Writer, b Builder) { func builderHelp(w io.Writer, b Builder) {
@@ -270,8 +446,27 @@ func builderHelp(w io.Writer, b Builder) {
} }
fmt.Fprintf(w, "usage: crucible %s <subcommand> [args] (or: %s <subcommand> [args])\n\n", b.Name, b.Bin) fmt.Fprintf(w, "usage: crucible %s <subcommand> [args] (or: %s <subcommand> [args])\n\n", b.Name, b.Bin)
fmt.Fprintf(w, "subcommands:\n") fmt.Fprintf(w, "subcommands:\n")
for _, c := range b.subcommands() { width := 0
fmt.Fprintf(w, " %s\n", c) for _, command := range b.Commands {
if len(command.Name) > width {
width = len(command.Name)
}
}
for _, command := range b.Commands {
fmt.Fprintf(w, " %-*s %s\n", width, command.Name, command.Summary)
} }
fmt.Fprintf(w, "\nstatus: wired to migrated nwn-tool logic. See docs/command-surface.md.\n") fmt.Fprintf(w, "\nstatus: wired to migrated nwn-tool logic. See docs/command-surface.md.\n")
} }
func commandHelp(w io.Writer, command Command) {
fmt.Fprintf(w, "%s\n", command.Summary)
if command.Usage != "" {
fmt.Fprintf(w, "\nusage: %s\n", command.Usage)
}
if len(command.Options) > 0 {
fmt.Fprintln(w, "\noptions:")
for _, option := range command.Options {
fmt.Fprintf(w, " %s\n", option)
}
}
}
+177 -30
View File
@@ -25,22 +25,54 @@ func TestHelpAndNoArgs(t *testing.T) {
if code := run([]string{"help"}, &out, &errw); code != exitOK { if code := run([]string{"help"}, &out, &errw); code != exitOK {
t.Fatalf("help exit=%d want %d", code, exitOK) t.Fatalf("help exit=%d want %d", code, exitOK)
} }
if !strings.Contains(out.String(), "builders:") { for _, builder := range Registry {
t.Fatalf("help output missing builders section:\n%s", out.String()) if !strings.Contains(out.String(), builder.Name) {
t.Errorf("help output missing builder %q:\n%s", builder.Name, out.String())
}
} }
// No args is a usage error (exit 64) but still prints help. // No args is a usage error (exit 64) but still prints help.
out.Reset() out.Reset()
if code := run(nil, &out, &errw); code != exitUsage { if code := run(nil, &out, &errw); code != exitUsage {
t.Fatalf("no-args exit=%d want %d", code, exitUsage) t.Fatalf("no-args exit=%d want %d", code, exitUsage)
} }
if out.Len() == 0 {
t.Fatal("no-args usage error should include help")
}
} }
func TestListCoversRegistry(t *testing.T) { func TestListCoversRegistry(t *testing.T) {
var out bytes.Buffer var out bytes.Buffer
list(&out) list(&out)
for _, b := range Registry { lines := strings.Split(strings.TrimSpace(out.String()), "\n")
if !strings.Contains(out.String(), b.Name) || !strings.Contains(out.String(), b.Bin) { if len(lines) != len(Registry) {
t.Fatalf("list missing %s/%s:\n%s", b.Name, b.Bin, out.String()) t.Fatalf("list returned %d rows for %d builders:\n%s", len(lines), len(Registry), out.String())
}
builders := make(map[string]Builder, len(Registry))
for _, builder := range Registry {
builders[builder.Name] = builder
}
seen := make(map[string]bool, len(lines))
for _, line := range lines {
fields := strings.Split(strings.TrimSpace(line), "\t")
if len(fields) != 3 {
t.Fatalf("list row must be name<TAB>bin<TAB>summary, got %q", line)
}
builder, ok := builders[fields[0]]
if !ok {
t.Errorf("list returned unregistered builder %q", fields[0])
continue
}
if fields[1] != builder.Bin {
t.Errorf("builder %q listed binary %q, want %q", builder.Name, fields[1], builder.Bin)
}
if strings.TrimSpace(fields[2]) == "" {
t.Errorf("builder %q listed an empty summary", builder.Name)
}
seen[fields[0]] = true
}
for _, builder := range Registry {
if !seen[builder.Name] {
t.Errorf("list missing builder %q", builder.Name)
} }
} }
} }
@@ -50,8 +82,8 @@ func TestUnknownBuilderFailsUsage(t *testing.T) {
if code := run([]string{"frobnicate"}, &out, &errw); code != exitUsage { if code := run([]string{"frobnicate"}, &out, &errw); code != exitUsage {
t.Fatalf("unknown builder exit=%d want %d", code, exitUsage) t.Fatalf("unknown builder exit=%d want %d", code, exitUsage)
} }
if !strings.Contains(errw.String(), "unknown builder") { if errw.Len() == 0 {
t.Fatalf("unknown builder stderr=%q", errw.String()) t.Fatal("unknown builder should explain the usage error")
} }
} }
@@ -65,8 +97,8 @@ func TestUnwiredBuilderFailsClosed(t *testing.T) {
if code := run([]string{b.Name}, &out, &errw); code != exitUnwired { if code := run([]string{b.Name}, &out, &errw); code != exitUnwired {
t.Errorf("crucible %s: exit=%d want %d (must fail closed)", b.Name, code, exitUnwired) t.Errorf("crucible %s: exit=%d want %d (must fail closed)", b.Name, code, exitUnwired)
} }
if !strings.Contains(errw.String(), "not wired") { if errw.Len() == 0 {
t.Errorf("crucible %s: stderr missing fail-closed message: %q", b.Name, errw.String()) t.Errorf("crucible %s: missing fail-closed explanation", b.Name)
} }
// Via standalone shim path. // Via standalone shim path.
out.Reset() out.Reset()
@@ -93,8 +125,8 @@ func TestWiredBuilderRejectsBadInvocation(t *testing.T) {
if code := runBuilder(b.Name, []string{"frobnicate"}, &out, &errw); code != exitUsage { if code := runBuilder(b.Name, []string{"frobnicate"}, &out, &errw); code != exitUsage {
t.Errorf("crucible %s frobnicate: exit=%d want %d", b.Name, code, exitUsage) t.Errorf("crucible %s frobnicate: exit=%d want %d", b.Name, code, exitUsage)
} }
if !strings.Contains(errw.String(), "unknown subcommand") { if errw.Len() == 0 {
t.Errorf("crucible %s frobnicate: stderr=%q", b.Name, errw.String()) t.Errorf("crucible %s frobnicate: missing usage explanation", b.Name)
} }
} }
} }
@@ -105,34 +137,149 @@ func TestBuilderHelpIsOK(t *testing.T) {
if code := runBuilder(b.Name, []string{"--help"}, &out, &errw); code != exitOK { if code := runBuilder(b.Name, []string{"--help"}, &out, &errw); code != exitOK {
t.Errorf("%s --help: exit=%d want %d", b.Name, code, exitOK) t.Errorf("%s --help: exit=%d want %d", b.Name, code, exitOK)
} }
if !strings.Contains(out.String(), b.Summary) { if !strings.Contains(out.String(), b.Name) || !strings.Contains(out.String(), b.Bin) {
t.Errorf("%s --help: missing summary", b.Name) t.Errorf("%s --help: missing builder identity", b.Name)
}
for _, subcommand := range b.subcommands() {
if !strings.Contains(out.String(), subcommand) {
t.Errorf("%s --help: missing subcommand %q", b.Name, subcommand)
}
} }
} }
} }
// Every legacy command named in command-surface.md must have exactly one func TestCanonicalCommandSurface(t *testing.T) {
// canonical home (Builder.Legacy), so the migrated surface has no gaps or want := map[string][]string{
// ambiguous homes. "depot": nil,
func TestLegacyCommandsHaveExactlyOneHome(t *testing.T) { "hak": {"build", "manifest"},
legacy := []string{ "module": {"build", "extract", "validate", "compare", "manifest"},
"build", "build-module", "extract", "validate", "compare", "topdata": {"validate", "build", "package", "compare", "convert"},
"build-haks", "apply-hak-manifest", "wiki": {"build", "deploy"},
"validate-topdata", "build-topdata", "build-top-package", "compare-topdata", "convert-topdata",
"build-wiki", "deploy-wiki",
} }
for _, cmd := range legacy { for _, builder := range Registry {
homes := 0 got := builder.subcommands()
for _, b := range Registry { expected, ok := want[builder.Name]
for _, c := range b.Legacy { if !ok {
if c == cmd { t.Fatalf("unexpected builder %q", builder.Name)
homes++ }
if strings.Join(got, ",") != strings.Join(expected, ",") {
t.Errorf("%s commands = %v, want %v", builder.Name, got, expected)
} }
} }
} }
if homes != 1 {
t.Errorf("legacy command %q has %d canonical homes, want 1", cmd, homes) func TestRegistryCommandNamesAndAliasesAreUnambiguous(t *testing.T) {
for _, builder := range Registry {
seen := map[string]bool{}
for _, command := range builder.Commands {
if command.Name == "" || command.Summary == "" || command.AppCommand == "" || command.Usage == "" {
t.Errorf("%s has incomplete command metadata: %#v", builder.Name, command)
} }
if seen[command.Name] {
t.Errorf("%s repeats command name %q", builder.Name, command.Name)
}
seen[command.Name] = true
for _, alias := range command.Aliases {
if alias.Name == "" {
t.Errorf("%s %s has an empty alias", builder.Name, command.Name)
continue
}
if seen[alias.Name] {
t.Errorf("%s repeats command or alias %q", builder.Name, alias.Name)
}
seen[alias.Name] = true
}
}
}
}
func TestMusicCommandsAreNotAccepted(t *testing.T) {
for _, builderName := range []string{"hak", "module"} {
builder, ok := find(builderName)
if !ok {
t.Fatalf("missing builder %q", builderName)
}
if builder.accepts("music") {
t.Errorf("%s must not accept the removed music command", builderName)
}
}
}
func TestHiddenAliasesPreserveImplementationTargets(t *testing.T) {
tests := []struct {
builder string
alias string
target string
}{
{"hak", "build-haks", "build-haks"},
{"hak", "apply-hak-manifest", "apply-hak-manifest"},
{"module", "build-module", "build-module"},
{"module", "apply-hak-manifest", "apply-hak-manifest"},
{"topdata", "validate-topdata", "validate-topdata"},
{"topdata", "build-topdata", "build-topdata"},
{"topdata", "build-top-package", "build-top-package"},
{"topdata", "compare-topdata", "compare-topdata"},
{"topdata", "convert-topdata", "convert-topdata"},
{"wiki", "build-wiki", "build-wiki"},
{"wiki", "deploy-wiki", "deploy-wiki"},
}
for _, tt := range tests {
t.Run(tt.builder+"/"+tt.alias, func(t *testing.T) {
builder, ok := find(tt.builder)
if !ok {
t.Fatalf("missing builder %q", tt.builder)
}
command, target, ok := builder.command(tt.alias)
if !ok {
t.Fatalf("hidden alias %q is not accepted", tt.alias)
}
if target != tt.target {
t.Fatalf("alias %q target = %q, want %q", tt.alias, target, tt.target)
}
if command.Name == tt.alias {
t.Fatalf("alias %q must not be a visible command", tt.alias)
}
})
}
}
func TestBuilderHelpHidesCompatibilityAliases(t *testing.T) {
aliases := map[string][]string{
"hak": {"build-haks", "apply-hak-manifest"},
"module": {"build-module", "apply-hak-manifest"},
"topdata": {"validate-topdata", "build-topdata", "build-top-package", "compare-topdata", "convert-topdata"},
"wiki": {"build-wiki", "deploy-wiki"},
}
for builderName, hidden := range aliases {
var out, errw bytes.Buffer
if code := runBuilder(builderName, []string{"--help"}, &out, &errw); code != exitOK {
t.Fatalf("%s help exit = %d", builderName, code)
}
for _, alias := range hidden {
if strings.Contains(out.String(), alias) {
t.Errorf("%s help exposed hidden alias %q:\n%s", builderName, alias, out.String())
}
}
}
}
func TestCommandHelpUsesCanonicalGuidanceWithoutLoadingProject(t *testing.T) {
var out, errw bytes.Buffer
if code := runBuilder("topdata", []string{"build", "--help"}, &out, &errw); code != exitOK {
t.Fatalf("command help exit = %d, stderr:\n%s", code, errw.String())
}
for _, want := range []string{
"compile topdata",
"usage: crucible topdata build",
"--force",
"--skip-lfs",
} {
if !strings.Contains(out.String(), want) {
t.Errorf("command help missing %q:\n%s", want, out.String())
}
}
if errw.Len() != 0 {
t.Fatalf("command help must not load a project or emit errors:\n%s", errw.String())
} }
} }
+25 -1
View File
@@ -2,14 +2,19 @@ package erf
import ( import (
"bytes" "bytes"
"crypto/sha256"
"encoding/binary" "encoding/binary"
"encoding/hex"
"fmt" "fmt"
"io" "io"
"os" "os"
"regexp"
"sort" "sort"
"strings" "strings"
) )
var expectedSHA256Pattern = regexp.MustCompile(`^[0-9a-f]{64}$`)
const ( const (
headerSize = 160 headerSize = 160
versionV10 = "V1.0" versionV10 = "V1.0"
@@ -30,6 +35,9 @@ type Resource struct {
Data []byte Data []byte
SourcePath string SourcePath string
Size int64 Size int64
// ExpectedSHA256, when set, is the lowercase hex SHA-256 the streamed
// SourcePath payload must hash to. A mismatch fails the write.
ExpectedSHA256 string
} }
type header struct { type header struct {
@@ -407,18 +415,34 @@ func writeResourceData(w io.Writer, resource Resource) error {
return err return err
} }
if resource.ExpectedSHA256 != "" && !expectedSHA256Pattern.MatchString(resource.ExpectedSHA256) {
return fmt.Errorf("resource %q has invalid expected sha256", resource.SourcePath)
}
file, err := os.Open(resource.SourcePath) file, err := os.Open(resource.SourcePath)
if err != nil { if err != nil {
return fmt.Errorf("open resource %q: %w", resource.SourcePath, err) return fmt.Errorf("open resource %q: %w", resource.SourcePath, err)
} }
defer file.Close() defer file.Close()
written, err := io.Copy(w, file) // The hash is computed while streaming into w, so size/SHA mismatches are
// only detected AFTER the (bad) bytes have already been written. A non-nil
// return therefore means w holds partially-written, unverified output; the
// caller must discard it. writeHAKArchive does this by writing to a temp file
// and removing it on any Write error rather than renaming it into place.
hash := sha256.New()
written, err := io.Copy(io.MultiWriter(w, hash), file)
if err != nil { if err != nil {
return fmt.Errorf("copy resource %q: %w", resource.SourcePath, err) return fmt.Errorf("copy resource %q: %w", resource.SourcePath, err)
} }
if resource.Size > 0 && written != resource.Size { if resource.Size > 0 && written != resource.Size {
return fmt.Errorf("copy resource %q: expected %d bytes, wrote %d", resource.SourcePath, resource.Size, written) return fmt.Errorf("copy resource %q: expected %d bytes, wrote %d", resource.SourcePath, resource.Size, written)
} }
if resource.ExpectedSHA256 != "" {
got := hex.EncodeToString(hash.Sum(nil))
if got != resource.ExpectedSHA256 {
return fmt.Errorf("resource %q sha256 mismatch: expected %s, got %s", resource.SourcePath, resource.ExpectedSHA256, got)
}
}
return nil return nil
} }
+63
View File
@@ -2,9 +2,72 @@ package erf
import ( import (
"bytes" "bytes"
"crypto/sha256"
"encoding/hex"
"os"
"path/filepath"
"strings"
"testing" "testing"
) )
func writeTempPayload(t *testing.T, payload []byte) string {
t.Helper()
path := filepath.Join(t.TempDir(), "payload.bin")
if err := os.WriteFile(path, payload, 0o644); err != nil {
t.Fatalf("write payload: %v", err)
}
return path
}
func TestWriteRejectsSourcePathSHA256Mismatch(t *testing.T) {
payload := []byte("hello world")
path := writeTempPayload(t, payload)
archive := New("HAK ", []Resource{{
Name: "asset",
Type: 0x0003,
SourcePath: path,
Size: int64(len(payload)),
ExpectedSHA256: strings.Repeat("a", 64),
}})
var buf bytes.Buffer
err := Write(&buf, archive)
if err == nil {
t.Fatal("expected sha256 mismatch error")
}
if !strings.Contains(err.Error(), "sha256 mismatch") {
t.Fatalf("error = %v, want sha256 mismatch", err)
}
}
func TestWriteAcceptsMatchingSourcePathSHA256(t *testing.T) {
payload := []byte("hello world")
path := writeTempPayload(t, payload)
sum := sha256.Sum256(payload)
archive := New("HAK ", []Resource{{
Name: "asset",
Type: 0x0003,
SourcePath: path,
Size: int64(len(payload)),
ExpectedSHA256: hex.EncodeToString(sum[:]),
}})
var buf bytes.Buffer
if err := Write(&buf, archive); err != nil {
t.Fatalf("write: %v", err)
}
decoded, err := Read(bytes.NewReader(buf.Bytes()))
if err != nil {
t.Fatalf("read: %v", err)
}
if len(decoded.Resources) != 1 || string(decoded.Resources[0].Data) != string(payload) {
t.Fatalf("unexpected payload: %#v", decoded.Resources)
}
}
func TestArchiveRoundTrip(t *testing.T) { func TestArchiveRoundTrip(t *testing.T) {
archive := New("MOD ", []Resource{ archive := New("MOD ", []Resource{
{Name: "module", Type: 0x07DE, Data: []byte("ifo")}, {Name: "module", Type: 0x07DE, Data: []byte("ifo")},
+21 -4
View File
@@ -15,8 +15,10 @@ import (
// Item is one selectable command. // Item is one selectable command.
type Item struct { type Item struct {
Label string // e.g. "module build" Label string // e.g. "module build"
Desc string // e.g. "build/extract/validate/compare the .mod" Desc string // e.g. "build the configured module"
Args []string // dispatcher args, e.g. {"module", "build"} Args []string // dispatcher args, e.g. {"module", "build"}
Usage string
Options []string
} }
// ANSI styling; terminals that don't support it simply ignore the codes. // ANSI styling; terminals that don't support it simply ignore the codes.
@@ -35,9 +37,15 @@ func Select(out io.Writer, in io.Reader, items []Item) ([]string, error) {
return nil, fmt.Errorf("no commands available") return nil, fmt.Errorf("no commands available")
} }
fmt.Fprintf(out, "%s%sCrucible%s — pick a command:\n\n", cBold, cCyan, cReset) fmt.Fprintf(out, "%s%sCrucible%s — pick a command:\n\n", cBold, cCyan, cReset)
labelWidth := 0
for _, item := range items {
if len(item.Label) > labelWidth {
labelWidth = len(item.Label)
}
}
for i, it := range items { for i, it := range items {
fmt.Fprintf(out, " %s%2d%s %s%-24s%s %s%s%s\n", fmt.Fprintf(out, " %s%2d%s %s%-*s%s %s%s%s\n",
cCyan, i+1, cReset, cBold, it.Label, cReset, cDim, it.Desc, cReset) cCyan, i+1, cReset, cBold, labelWidth, it.Label, cReset, cDim, it.Desc, cReset)
} }
fmt.Fprintf(out, " %sq%s quit\n\nselection: ", cCyan, cReset) fmt.Fprintf(out, " %sq%s quit\n\nselection: ", cCyan, cReset)
@@ -52,7 +60,16 @@ func Select(out io.Writer, in io.Reader, items []Item) ([]string, error) {
return nil, fmt.Errorf("invalid selection %q", choice) return nil, fmt.Errorf("invalid selection %q", choice)
} }
it := items[n-1] it := items[n-1]
fmt.Fprintf(out, "extra args for %q (optional): ", it.Label) if it.Usage != "" {
fmt.Fprintf(out, "\nusage: %s\n", it.Usage)
}
if len(it.Options) > 0 {
fmt.Fprintln(out, "options:")
for _, option := range it.Options {
fmt.Fprintf(out, " %s\n", option)
}
}
fmt.Fprint(out, "\narguments (press Enter to use defaults): ")
argLine, _ := r.ReadString('\n') argLine, _ := r.ReadString('\n')
extra := strings.Fields(strings.TrimSpace(argLine)) extra := strings.Fields(strings.TrimSpace(argLine))
return append(append([]string{}, it.Args...), extra...), nil return append(append([]string{}, it.Args...), extra...), nil
+57
View File
@@ -1,8 +1,10 @@
package menu package menu
import ( import (
"bytes"
"io" "io"
"reflect" "reflect"
"regexp"
"strings" "strings"
"testing" "testing"
) )
@@ -45,3 +47,58 @@ func TestSelectInvalidErrors(t *testing.T) {
t.Fatal("out-of-range selection should error") t.Fatal("out-of-range selection should error")
} }
} }
func TestSelectAlignsDescriptionsAfterLongestLabel(t *testing.T) {
items := []Item{
{Label: "hak build", Desc: "build haks", Args: []string{"hak", "build"}},
{Label: "topdata exceptionally-long-command", Desc: "long command", Args: []string{"topdata", "long"}},
}
var out bytes.Buffer
if _, err := Select(&out, strings.NewReader("q\n"), items); err != nil {
t.Fatalf("select: %v", err)
}
text := stripANSI(out.String())
var positions []int
for _, line := range strings.Split(text, "\n") {
for _, desc := range []string{"build haks", "long command"} {
if index := strings.Index(line, desc); index >= 0 {
positions = append(positions, index)
}
}
}
if len(positions) != 2 {
t.Fatalf("expected two menu descriptions, got %d:\n%s", len(positions), text)
}
if positions[0] != positions[1] {
t.Fatalf("descriptions are not aligned: columns %v\n%s", positions, text)
}
}
func TestSelectShowsCommandGuidanceBeforeArgumentsPrompt(t *testing.T) {
items := []Item{{
Label: "topdata build",
Desc: "compile topdata",
Args: []string{"topdata", "build"},
Usage: "crucible topdata build [--force]",
Options: []string{"--force rebuild outputs"},
}}
var out bytes.Buffer
if _, err := Select(&out, strings.NewReader("1\n\n"), items); err != nil {
t.Fatalf("select: %v", err)
}
text := stripANSI(out.String())
for _, want := range []string{
"usage: crucible topdata build [--force]",
"options:",
"--force rebuild outputs",
"arguments (press Enter to use defaults):",
} {
if !strings.Contains(text, want) {
t.Errorf("selected command guidance missing %q:\n%s", want, text)
}
}
}
func stripANSI(value string) string {
return regexp.MustCompile(`\x1b\[[0-9;]*m`).ReplaceAllString(value, "")
}
-47
View File
@@ -1,47 +0,0 @@
package music
import (
"fmt"
"path/filepath"
"strings"
)
type DatasetConfig struct {
Source string `json:"source" yaml:"source"`
Prefix string `json:"prefix,omitempty" yaml:"prefix,omitempty"`
StageRoot string `json:"stage_root,omitempty" yaml:"stage_root,omitempty"`
CreditsRoot string `json:"credits_root,omitempty" yaml:"credits_root,omitempty"`
}
type DefaultsConfig struct {
StageRoot string `json:"stage_root,omitempty" yaml:"stage_root,omitempty"`
CreditsRoot string `json:"credits_root,omitempty" yaml:"credits_root,omitempty"`
CreditsOverlay string `json:"credits_overlay,omitempty" yaml:"credits_overlay,omitempty"`
MaxStemLength *int `json:"max_stem_length,omitempty" yaml:"max_stem_length,omitempty"`
}
func ValidateDatasetSource(field, source string) error {
trimmed := strings.TrimSpace(source)
if trimmed == "" {
return fmt.Errorf("%s.source is required", field)
}
if strings.Contains(trimmed, "\x00") {
return fmt.Errorf("%s.source must not contain NUL bytes", field)
}
clean := filepath.Clean(filepath.FromSlash(trimmed))
if clean == ".." || strings.HasPrefix(clean, ".."+string(filepath.Separator)) {
return fmt.Errorf("%s.source must not escape project root", field)
}
return nil
}
func ValidateDatasetID(id string) error {
trimmed := strings.TrimSpace(id)
if trimmed == "" {
return fmt.Errorf("dataset id must not be empty")
}
if strings.Contains(trimmed, "/") || strings.Contains(trimmed, "\\") {
return fmt.Errorf("dataset id %q must not contain path separators", trimmed)
}
return nil
}
-292
View File
@@ -1,292 +0,0 @@
package music
import (
"fmt"
"os"
"path/filepath"
"strings"
)
func ParseCreditsOverlay(path string) (CreditsOverlay, error) {
entries, err := ParseCreditsMarkdown(path)
if err != nil {
return CreditsOverlay{}, err
}
overlay := CreditsOverlay{
ByOriginal: make(map[string]CreditsEntry),
ByOutput: make(map[string]CreditsEntry),
Entries: entries,
}
for _, entry := range entries {
if entry.OriginalFile != "" {
key := strings.ToLower(strings.TrimSpace(entry.OriginalFile))
if _, exists := overlay.ByOriginal[key]; exists {
return CreditsOverlay{}, fmt.Errorf("duplicate manual credits row for original file %s", entry.OriginalFile)
}
overlay.ByOriginal[key] = entry
}
if entry.OutputFile != "" {
key := strings.ToLower(strings.TrimSpace(entry.OutputFile))
if _, exists := overlay.ByOutput[key]; exists {
return CreditsOverlay{}, fmt.Errorf("duplicate manual credits row for output file %s", entry.OutputFile)
}
overlay.ByOutput[key] = entry
}
}
return overlay, nil
}
func (o CreditsOverlay) Match(originalFile, outputFile string) *CreditsEntry {
if entry, ok := o.ByOutput[strings.ToLower(strings.TrimSpace(outputFile))]; ok {
entryCopy := entry
return &entryCopy
}
if entry, ok := o.ByOriginal[strings.ToLower(strings.TrimSpace(originalFile))]; ok {
entryCopy := entry
return &entryCopy
}
return nil
}
func ValidateOverlayEntries(dir string, overlay CreditsOverlay, generated []CreditsEntry) error {
if len(overlay.Entries) == 0 {
return nil
}
validOriginal := make(map[string]struct{}, len(generated))
validOutput := make(map[string]struct{}, len(generated))
for _, entry := range generated {
validOriginal[strings.ToLower(entry.OriginalFile)] = struct{}{}
validOutput[strings.ToLower(entry.OutputFile)] = struct{}{}
}
for _, entry := range overlay.Entries {
if entry.OriginalFile != "" {
if _, ok := validOriginal[strings.ToLower(entry.OriginalFile)]; !ok {
return fmt.Errorf("manual credits entry in %s references unknown original file %s", dir, entry.OriginalFile)
}
}
if entry.OutputFile != "" {
if _, ok := validOutput[strings.ToLower(entry.OutputFile)]; !ok {
return fmt.Errorf("manual credits entry in %s references unknown output file %s", dir, entry.OutputFile)
}
}
}
return nil
}
func ParseCreditsMarkdown(path string) ([]CreditsEntry, error) {
raw, err := os.ReadFile(path)
if err != nil {
if os.IsNotExist(err) {
return nil, nil
}
return nil, err
}
lines := strings.Split(strings.ReplaceAll(string(raw), "\r\n", "\n"), "\n")
header := []string(nil)
entries := make([]CreditsEntry, 0)
for _, line := range lines {
line = strings.TrimSpace(line)
if !strings.HasPrefix(line, "|") {
continue
}
cells := ParseMarkdownTableRow(line)
if len(cells) == 0 {
continue
}
if header == nil {
header = make([]string, len(cells))
for i, cell := range cells {
header[i] = NormalizeCreditsHeader(cell)
}
continue
}
if IsMarkdownSeparatorRow(cells) {
continue
}
entry := CreditsEntry{}
for i, key := range header {
if i >= len(cells) {
continue
}
value := CleanCreditsCell(cells[i])
switch key {
case "artist":
entry.Artist = value
case "title":
entry.Title = value
case "output_file":
entry.OutputFile = value
case "original_file":
entry.OriginalFile = value
case "album":
entry.Album = value
case "date":
entry.Date = value
case "rights":
entry.Rights = value
case "notes":
entry.Notes = value
}
}
if entry.Artist == "" && entry.Title == "" && entry.OutputFile == "" && entry.OriginalFile == "" {
continue
}
entries = append(entries, entry)
}
return entries, nil
}
func ParseMarkdownTableRow(line string) []string {
trimmed := strings.TrimSpace(line)
trimmed = strings.TrimPrefix(trimmed, "|")
trimmed = strings.TrimSuffix(trimmed, "|")
parts := strings.Split(trimmed, "|")
cells := make([]string, 0, len(parts))
for _, part := range parts {
cells = append(cells, strings.TrimSpace(strings.ReplaceAll(part, `\|`, "|")))
}
return cells
}
func IsMarkdownSeparatorRow(cells []string) bool {
if len(cells) == 0 {
return false
}
for _, cell := range cells {
cell = strings.TrimSpace(cell)
cell = strings.ReplaceAll(cell, "-", "")
cell = strings.ReplaceAll(cell, ":", "")
if cell != "" {
return false
}
}
return true
}
func NormalizeCreditsHeader(value string) string {
switch strings.ToLower(CleanCreditsCell(value)) {
case "artist":
return "artist"
case "title":
return "title"
case "output file":
return "output_file"
case "original file":
return "original_file"
case "album":
return "album"
case "date":
return "date"
case "license / copyright":
return "rights"
case "notes":
return "notes"
default:
return ""
}
}
func CleanCreditsCell(value string) string {
value = strings.TrimSpace(value)
value = strings.Trim(value, "`")
value = strings.ReplaceAll(value, "<br>", "\n")
return strings.TrimSpace(value)
}
func RenderCreditsMarkdown(entries []CreditsEntry) string {
var builder strings.Builder
builder.WriteString(CreditsHeader)
for _, entry := range entries {
builder.WriteString("| ")
builder.WriteString(EscapeMarkdownCell(entry.Artist))
builder.WriteString(" | ")
builder.WriteString(EscapeMarkdownCell(entry.Title))
builder.WriteString(" | ")
builder.WriteString(EscapeMarkdownCell(WrapCodeCell(entry.OutputFile)))
builder.WriteString(" | ")
builder.WriteString(EscapeMarkdownCell(WrapCodeCell(entry.OriginalFile)))
builder.WriteString(" | ")
builder.WriteString(EscapeMarkdownCell(entry.Album))
builder.WriteString(" | ")
builder.WriteString(EscapeMarkdownCell(entry.Date))
builder.WriteString(" | ")
builder.WriteString(EscapeMarkdownCell(entry.Rights))
builder.WriteString(" | ")
builder.WriteString(EscapeMarkdownCell(entry.Notes))
builder.WriteString(" |\n")
}
return builder.String()
}
func WrapCodeCell(value string) string {
if strings.TrimSpace(value) == "" {
return ""
}
return "`" + value + "`"
}
func EscapeMarkdownCell(value string) string {
value = strings.ReplaceAll(value, `\`, `\\`)
value = strings.ReplaceAll(value, "|", `\|`)
value = strings.ReplaceAll(value, "\n", "<br>")
return value
}
func ApplyCreditsOverlay(entry *CreditsEntry, overlay *CreditsEntry) {
if overlay == nil {
return
}
entry.Artist = FirstNonEmpty(overlay.Artist, entry.Artist)
entry.Title = FirstNonEmpty(overlay.Title, entry.Title)
entry.Album = FirstNonEmpty(overlay.Album, entry.Album)
entry.Date = FirstNonEmpty(overlay.Date, entry.Date)
entry.Rights = FirstNonEmpty(overlay.Rights, entry.Rights)
entry.Notes = FirstNonEmpty(overlay.Notes, entry.Notes)
}
func SyncGeneratedCreditsArtifacts(root string, desiredFiles map[string][]byte) (int, error) {
changedFiles := 0
existingFiles := make(map[string]struct{})
if err := filepath.WalkDir(root, func(path string, d os.DirEntry, walkErr error) error {
if walkErr != nil {
if os.IsNotExist(walkErr) {
return nil
}
return walkErr
}
if d.IsDir() {
return nil
}
existingFiles[path] = struct{}{}
return nil
}); err != nil {
return 0, fmt.Errorf("scan credits dir: %w", err)
}
for path, content := range desiredFiles {
current, err := os.ReadFile(path)
if err != nil && !os.IsNotExist(err) {
return 0, fmt.Errorf("read credits artifact %s: %w", path, err)
}
if err == nil && string(current) == string(content) {
delete(existingFiles, path)
continue
}
if err := os.MkdirAll(filepath.Dir(path), 0o755); err != nil {
return 0, fmt.Errorf("create generated credits dir: %w", err)
}
if err := os.WriteFile(path, content, 0o644); err != nil {
return 0, fmt.Errorf("write credits artifact %s: %w", path, err)
}
changedFiles++
delete(existingFiles, path)
}
for path := range existingFiles {
if err := os.Remove(path); err != nil && !os.IsNotExist(err) {
return 0, fmt.Errorf("remove stale credits artifact %s: %w", path, err)
}
changedFiles++
}
return changedFiles, nil
}
-62
View File
@@ -1,62 +0,0 @@
package music
import (
"encoding/json"
"fmt"
"os/exec"
"strings"
)
func ProbeMetadata(ffprobePath, sourcePath string) (Metadata, error) {
cmd := exec.Command(ffprobePath, "-v", "quiet", "-print_format", "json", "-show_format", sourcePath)
output, err := cmd.Output()
if err != nil {
return Metadata{}, err
}
var payload struct {
Format struct {
Tags map[string]string `json:"tags"`
} `json:"format"`
}
if err := json.Unmarshal(output, &payload); err != nil {
return Metadata{}, err
}
tags := make(map[string]string, len(payload.Format.Tags))
for key, value := range payload.Format.Tags {
tags[strings.ToLower(strings.TrimSpace(key))] = strings.TrimSpace(value)
}
return Metadata{
Title: FirstNonEmpty(tags["title"]),
Artist: FirstNonEmpty(tags["artist"], tags["album_artist"], tags["composer"]),
Album: FirstNonEmpty(tags["album"]),
Date: FirstNonEmpty(tags["date"], tags["year"]),
Comment: FirstNonEmpty(tags["comment"], tags["description"]),
Copyright: FirstNonEmpty(tags["copyright"]),
License: FirstNonEmpty(tags["license"], tags["license_url"]),
}, nil
}
func ConvertSource(ffmpegPath, sourcePath, outputPath string) error {
cmd := exec.Command(
ffmpegPath, "-y",
"-i", sourcePath,
"-vn",
"-map_metadata", "-1",
"-ar", "44100",
"-ac", "2",
"-b:a", "192k",
"-codec:a", "libmp3lame",
"-write_xing", "0",
"-f", "mp3",
outputPath,
)
output, err := cmd.CombinedOutput()
if err != nil {
message := strings.TrimSpace(string(output))
if message == "" {
message = err.Error()
}
return fmt.Errorf("%s", message)
}
return nil
}
-239
View File
@@ -1,239 +0,0 @@
package music
import (
"fmt"
"os"
"os/exec"
"path/filepath"
"regexp"
"sort"
"strings"
)
const (
MaxStemLen = 16
CreditsHeader = "# NWN Music Pack Credits\n\nOriginal rights remain with the respective composers and licensors.\n\n## Processed Files\n\n| Artist | Title | Output File | Original File | Album | Date | License / Copyright | Notes |\n|---|---|---|---|---|---|---|---|\n"
)
var (
defaultConvertExtensions = []string{
".flac",
".m4a",
".mp3",
".ogg",
".wav",
}
ConvertExtensions = map[string]struct{}{
".flac": {},
".m4a": {},
".mp3": {},
".ogg": {},
".wav": {},
}
PassthroughExtensions = map[string]struct{}{
".bmu": {},
".wav": {},
}
DropWords = map[string]struct{}{
"mp3": {}, "wav": {}, "flac": {}, "ogg": {}, "m4a": {},
"music": {}, "track": {}, "audio": {}, "song": {}, "soundtrack": {},
"final": {}, "version": {}, "ver": {}, "v": {}, "mix": {}, "master": {}, "remaster": {},
"free": {}, "royalty": {}, "copyright": {}, "background": {}, "bgm": {},
"loop": {}, "loops": {}, "loopable": {}, "seamless": {},
"official": {}, "download": {}, "preview": {},
"the": {}, "a": {}, "an": {}, "and": {}, "of": {}, "to": {}, "in": {}, "for": {}, "with": {}, "by": {},
}
BracketRE = regexp.MustCompile(`\[[^\]]*\]|\([^)]*\)|\{[^}]*\}`)
SplitRE = regexp.MustCompile(`[^A-Za-z0-9]+`)
YearRE = regexp.MustCompile(`^\d{4,}$`)
VowelRE = regexp.MustCompile(`[aeiou]`)
)
func DefaultConvertExtensions() []string {
return append([]string(nil), defaultConvertExtensions...)
}
type Metadata struct {
Title string
Artist string
Album string
Date string
Rights string
Notes string
Comment string
Copyright string
License string
}
type CreditsEntry struct {
Artist string `json:"artist"`
Title string `json:"title"`
OutputFile string `json:"output_file"`
OriginalFile string `json:"original_file"`
Album string `json:"album"`
Date string `json:"date"`
Rights string `json:"rights"`
Notes string `json:"notes"`
}
type CreditsSource struct {
Path string `json:"path"`
Kind string `json:"kind"`
Entries []CreditsEntry `json:"entries"`
}
type CreditsInventory struct {
Sources []CreditsSource `json:"sources"`
}
type CreditsOverlay struct {
ByOriginal map[string]CreditsEntry
ByOutput map[string]CreditsEntry
Entries []CreditsEntry
}
func ResolveFFmpeg() (string, error) {
return ResolveFFmpegWithConfig("")
}
func ResolveFFmpegWithConfig(configuredPath string) (string, error) {
if configured := strings.TrimSpace(configuredPath); configured != "" {
return configured, nil
}
if configured := strings.TrimSpace(os.Getenv("SOW_FFMPEG")); configured != "" {
return configured, nil
}
path, err := exec.LookPath("ffmpeg")
if err != nil {
return "", ffmpegMissingErr("ffmpeg")
}
return path, nil
}
func ResolveFFprobe() (string, error) {
return ResolveFFprobeWithConfig("")
}
func ResolveFFprobeWithConfig(configuredPath string) (string, error) {
if configured := strings.TrimSpace(configuredPath); configured != "" {
return configured, nil
}
if configured := strings.TrimSpace(os.Getenv("SOW_FFPROBE")); configured != "" {
return configured, nil
}
path, err := exec.LookPath("ffprobe")
if err != nil {
return "", ffmpegMissingErr("ffprobe")
}
return path, nil
}
// ffmpegMissingErr renders an actionable, cross-platform "tool not found"
// message. Only the music builder needs ffmpeg/ffprobe; every other builder
// runs with zero external dependencies.
func ffmpegMissingErr(tool string) error {
return fmt.Errorf(`%[1]s not found on PATH.
Windows: winget install Gyan.FFmpeg
macOS: brew install ffmpeg
Linux: apt install ffmpeg (or your distro's package manager)
Only the music builder needs %[1]s; set SOW_%[2]s to a full path to override.`,
tool, strings.ToUpper(tool))
}
func IsMusicAssetPath(rel string) bool {
rel = filepath.ToSlash(strings.TrimSpace(rel))
return rel == "envi/music" || strings.HasPrefix(rel, "envi/music/")
}
func PathIsUnder(root, rel string) bool {
root = TrimSlashes(filepath.ToSlash(root))
rel = TrimSlashes(filepath.ToSlash(rel))
if root == "" || rel == "" {
return false
}
return rel == root || strings.HasPrefix(rel, root+"/")
}
func PrefixForDir(prefixes map[string]string, dir string) string {
dir = TrimSlashes(filepath.ToSlash(dir))
bestPrefix := ""
bestLen := -1
keys := make([]string, 0, len(prefixes))
for key := range prefixes {
keys = append(keys, key)
}
sort.Strings(keys)
for _, key := range keys {
value := prefixes[key]
normalizedKey := TrimSlashes(filepath.ToSlash(key))
if normalizedKey == "" {
continue
}
if dir != normalizedKey && !strings.HasPrefix(dir, normalizedKey+"/") {
continue
}
if len(normalizedKey) > bestLen {
bestLen = len(normalizedKey)
bestPrefix = SanitizePrefix(value)
}
}
return bestPrefix
}
func SanitizePrefix(prefix string) string {
prefix = strings.ToLower(strings.TrimSpace(prefix))
var builder strings.Builder
for _, r := range prefix {
switch {
case r >= 'a' && r <= 'z':
builder.WriteRune(r)
case r >= '0' && r <= '9':
builder.WriteRune(r)
case r == '_':
builder.WriteRune(r)
}
}
result := builder.String()
if strings.TrimSpace(prefix) != "" && strings.HasSuffix(strings.TrimSpace(prefix), "_") && !strings.HasSuffix(result, "_") {
result += "_"
}
return result
}
func TrimSlashes(value string) string {
return strings.Trim(strings.TrimSpace(value), "/")
}
func Min(a, b int) int {
if a < b {
return a
}
return b
}
func Max(a, b int) int {
if a > b {
return a
}
return b
}
func FirstNonEmpty(values ...string) string {
for _, value := range values {
if strings.TrimSpace(value) != "" {
return strings.TrimSpace(value)
}
}
return ""
}
func JoinNonEmpty(sep string, values ...string) string {
parts := make([]string, 0, len(values))
for _, value := range values {
value = strings.TrimSpace(value)
if value != "" {
parts = append(parts, value)
}
}
return strings.Join(parts, sep)
}
-417
View File
@@ -1,417 +0,0 @@
package music
import (
"os"
"path/filepath"
"strings"
"testing"
)
func TestUniqueNameHandlesShortStemCollisions(t *testing.T) {
used := map[string]struct{}{
"mus_wg_mystc": {},
}
got := UniqueName("mus_wg_mystc", used)
if got != "mus_wg_mystc_1" {
t.Fatalf("unexpected collision result: got %q want %q", got, "mus_wg_mystc_1")
}
}
func TestUniqueNameNoCollision(t *testing.T) {
used := map[string]struct{}{}
got := UniqueName("new_stem", used)
if got != "new_stem" {
t.Fatalf("expected 'new_stem', got %q", got)
}
}
func TestUniqueNameMultipleCollisions(t *testing.T) {
used := map[string]struct{}{
"stem": {},
"stem_1": {},
"stem_2": {},
}
got := UniqueName("stem", used)
if got != "stem_3" {
t.Fatalf("expected 'stem_3', got %q", got)
}
}
func TestGenerateStem(t *testing.T) {
used := map[string]struct{}{
"existing": {},
}
stem, err := GenerateStem("My Cool Track.mp3", "mus_", used)
if err != nil {
t.Fatalf("unexpected error: %v", err)
}
if stem == "" {
t.Fatal("expected non-empty stem")
}
if len(stem) > MaxStemLen {
t.Fatalf("stem %q exceeds max length %d", stem, MaxStemLen)
}
if _, exists := used[stem]; !exists {
t.Fatal("expected stem to be registered in used set")
}
}
func TestGenerateStemPrefixTooLong(t *testing.T) {
_, err := GenerateStem("test.mp3", "this_prefix_is_way_too_long_for_stem_", nil)
if err == nil {
t.Fatal("expected error for too-long prefix")
}
}
func TestSlugWords(t *testing.T) {
words := SlugWords("My Cool Music Track (Official) [HD]")
if len(words) == 0 {
t.Fatal("expected non-empty words")
}
}
func TestSlugWordsStripsBrackets(t *testing.T) {
words := SlugWords("Song [Explicit] (Remix)")
for _, w := range words {
if w == "explicit" || w == "remix" {
t.Fatalf("bracket content should be removed, got word %q", w)
}
}
}
func TestSanitizePrefix(t *testing.T) {
if got := SanitizePrefix("Mus_WG_"); got != "mus_wg_" {
t.Fatalf("unexpected prefix: %q", got)
}
if got := SanitizePrefix(" Test "); got != "test" {
t.Fatalf("unexpected prefix: %q", got)
}
if got := SanitizePrefix(""); got != "" {
t.Fatalf("expected empty prefix, got %q", got)
}
}
func TestReserveName(t *testing.T) {
used := map[string]struct{}{}
if err := ReserveName("testname", used); err != nil {
t.Fatalf("unexpected error: %v", err)
}
if _, exists := used["testname"]; !exists {
t.Fatal("expected name to be reserved")
}
if err := ReserveName("testname", used); err == nil {
t.Fatal("expected error for duplicate name")
}
}
func TestValidateManualOutputFile(t *testing.T) {
if err := ValidateManualOutputFile("test.bmu"); err != nil {
t.Fatalf("unexpected error: %v", err)
}
if err := ValidateManualOutputFile("test.txt"); err == nil {
t.Fatal("expected error for non-bmu extension")
}
if err := ValidateManualOutputFile(".bmu"); err == nil {
t.Fatal("expected error for empty stem")
}
}
func TestParseCreditsMarkdownMissingFile(t *testing.T) {
entries, err := ParseCreditsMarkdown("nonexistent.md")
if err != nil {
t.Fatalf("unexpected error for missing file: %v", err)
}
if entries != nil {
t.Fatalf("expected nil entries for missing file, got %v", entries)
}
}
func TestParseCreditsMarkdownRealContent(t *testing.T) {
content := "# Header\n\n| Artist | Title | Output File | Original File | Album | Date | License / Copyright | Notes |\n|---|---|---|---|---|---|---|---|\n| Test Artist | Test Title | `output.bmu` | `original.mp3` | Test Album | 2024 | MIT | test |\n"
path := filepath.Join(t.TempDir(), "CREDITS.md")
if err := os.WriteFile(path, []byte(content), 0o644); err != nil {
t.Fatalf("write test credits: %v", err)
}
entries, err := ParseCreditsMarkdown(path)
if err != nil {
t.Fatalf("unexpected error: %v", err)
}
if len(entries) != 1 {
t.Fatalf("expected 1 entry, got %d", len(entries))
}
if entries[0].Artist != "Test Artist" {
t.Fatalf("expected 'Test Artist', got %q", entries[0].Artist)
}
if entries[0].Title != "Test Title" {
t.Fatalf("expected 'Test Title', got %q", entries[0].Title)
}
if entries[0].OutputFile != "output.bmu" {
t.Fatalf("expected 'output.bmu', got %q", entries[0].OutputFile)
}
}
func TestRenderCreditsMarkdown(t *testing.T) {
entries := []CreditsEntry{
{
Artist: "Test Artist",
Title: "Test Title",
OutputFile: "output.bmu",
OriginalFile: "original.mp3",
Album: "Test Album",
Date: "2024",
Rights: "MIT",
Notes: "test note",
},
}
result := RenderCreditsMarkdown(entries)
if !strings.Contains(result, "Test Artist") {
t.Fatal("expected rendered output to contain artist")
}
if !strings.Contains(result, "output.bmu") {
t.Fatal("expected rendered output to contain output file")
}
}
func TestParseCreditsOverlay(t *testing.T) {
content := "# Header\n\n| Artist | Title | Output File | Original File |\n|---|---|---|---|\n| Overlay Artist | Ovr Title | `override.bmu` | `original.mp3` |\n"
path := filepath.Join(t.TempDir(), "CREDITS.md")
if err := os.WriteFile(path, []byte(content), 0o644); err != nil {
t.Fatalf("write test overlay: %v", err)
}
overlay, err := ParseCreditsOverlay(path)
if err != nil {
t.Fatalf("unexpected error: %v", err)
}
if len(overlay.Entries) != 1 {
t.Fatalf("expected 1 overlay entry, got %d", len(overlay.Entries))
}
matched := overlay.Match("original.mp3", "")
if matched == nil {
t.Fatal("expected overlay match by original file")
}
if matched.Artist != "Overlay Artist" {
t.Fatalf("expected 'Overlay Artist', got %q", matched.Artist)
}
}
func TestApplyCreditsOverlay(t *testing.T) {
entry := &CreditsEntry{
Artist: "Original Artist",
Title: "Original Title",
}
overlay := &CreditsEntry{
Artist: "Overlay Artist",
}
ApplyCreditsOverlay(entry, overlay)
if entry.Artist != "Overlay Artist" {
t.Fatalf("expected 'Overlay Artist', got %q", entry.Artist)
}
if entry.Title != "Original Title" {
t.Fatalf("expected 'Original Title' preserved, got %q", entry.Title)
}
ApplyCreditsOverlay(entry, nil)
if entry.Artist != "Overlay Artist" {
t.Fatalf("expected nil overlay to be no-op, got %q", entry.Artist)
}
}
func TestValidateOverlayEntries(t *testing.T) {
generated := []CreditsEntry{
{OriginalFile: "track1.mp3", OutputFile: "track1.bmu"},
}
overlay := CreditsOverlay{
Entries: []CreditsEntry{
{OriginalFile: "track1.mp3"},
},
}
if err := ValidateOverlayEntries("test", overlay, generated); err != nil {
t.Fatalf("unexpected validation error: %v", err)
}
badOverlay := CreditsOverlay{
Entries: []CreditsEntry{
{OriginalFile: "nonexistent.mp3"},
},
}
if err := ValidateOverlayEntries("test", badOverlay, generated); err == nil {
t.Fatal("expected validation error for unknown original file")
}
}
func TestIsMusicAssetPath(t *testing.T) {
if !IsMusicAssetPath("envi/music") {
t.Fatal("expected 'envi/music' to be music path")
}
if !IsMusicAssetPath("envi/music/westgate") {
t.Fatal("expected 'envi/music/westgate' to be music path")
}
if IsMusicAssetPath("envi/textures") {
t.Fatal("expected 'envi/textures' not to be music path")
}
if IsMusicAssetPath("") {
t.Fatal("expected empty string not to be music path")
}
}
func TestFirstNonEmpty(t *testing.T) {
if got := FirstNonEmpty("", "hello", "world"); got != "hello" {
t.Fatalf("expected 'hello', got %q", got)
}
if got := FirstNonEmpty("", "", ""); got != "" {
t.Fatalf("expected empty, got %q", got)
}
if got := FirstNonEmpty("first"); got != "first" {
t.Fatalf("expected 'first', got %q", got)
}
}
func TestJoinNonEmpty(t *testing.T) {
if got := JoinNonEmpty(", ", "a", "", "b"); got != "a, b" {
t.Fatalf("expected 'a, b', got %q", got)
}
if got := JoinNonEmpty(", "); got != "" {
t.Fatalf("expected empty, got %q", got)
}
}
func TestTrimSlashes(t *testing.T) {
if got := TrimSlashes("/foo/bar/"); got != "foo/bar" {
t.Fatalf("expected 'foo/bar', got %q", got)
}
if got := TrimSlashes("///"); got != "" {
t.Fatalf("expected empty, got %q", got)
}
}
func TestEscapeMarkdownCell(t *testing.T) {
result := EscapeMarkdownCell("a|b\nc")
if !strings.Contains(result, `\|`) {
t.Fatal("expected pipe to be escaped")
}
if !strings.Contains(result, "<br>") {
t.Fatal("expected newline to be replaced with <br>")
}
}
func TestWrapCodeCell(t *testing.T) {
if got := WrapCodeCell("test.bmu"); got != "`test.bmu`" {
t.Fatalf("expected '`test.bmu`', got %q", got)
}
if got := WrapCodeCell(""); got != "" {
t.Fatalf("expected empty string, got %q", got)
}
}
func TestSyncGeneratedCreditsArtifactsWritesNewFiles(t *testing.T) {
root := t.TempDir()
desired := map[string][]byte{
filepath.Join(root, "test.md"): []byte("content"),
}
changed, err := SyncGeneratedCreditsArtifacts(root, desired)
if err != nil {
t.Fatalf("unexpected error: %v", err)
}
if changed != 1 {
t.Fatalf("expected 1 changed file, got %d", changed)
}
}
func TestSyncGeneratedCreditsArtifactsNoChanges(t *testing.T) {
root := t.TempDir()
path := filepath.Join(root, "test.md")
if err := os.WriteFile(path, []byte("content"), 0o644); err != nil {
t.Fatalf("write file: %v", err)
}
desired := map[string][]byte{
path: []byte("content"),
}
changed, err := SyncGeneratedCreditsArtifacts(root, desired)
if err != nil {
t.Fatalf("unexpected error: %v", err)
}
if changed != 0 {
t.Fatalf("expected 0 changed files, got %d", changed)
}
}
func TestSyncGeneratedCreditsArtifactsRemovesStale(t *testing.T) {
root := t.TempDir()
stalePath := filepath.Join(root, "stale.md")
if err := os.WriteFile(stalePath, []byte("stale"), 0o644); err != nil {
t.Fatalf("write stale file: %v", err)
}
desired := map[string][]byte{}
changed, err := SyncGeneratedCreditsArtifacts(root, desired)
if err != nil {
t.Fatalf("unexpected error: %v", err)
}
if changed != 1 {
t.Fatalf("expected 1 removed file, got %d", changed)
}
}
func TestPrefixForDir(t *testing.T) {
prefixes := map[string]string{
"envi/music/westgate": "mus_wg_",
}
if got := PrefixForDir(prefixes, "envi/music/westgate"); got != "mus_wg_" {
t.Fatalf("expected 'mus_wg_', got %q", got)
}
if got := PrefixForDir(prefixes, "envi/music/other"); got != "" {
t.Fatalf("expected empty prefix, got %q", got)
}
if got := PrefixForDir(nil, "envi/music"); got != "" {
t.Fatalf("expected empty prefix for nil map, got %q", got)
}
}
func TestMaxMin(t *testing.T) {
if got := Min(3, 5); got != 3 {
t.Fatalf("Min(3,5) = %d, want 3", got)
}
if got := Max(3, 5); got != 5 {
t.Fatalf("Max(3,5) = %d, want 5", got)
}
}
func TestResolveFFmpegEnv(t *testing.T) {
t.Setenv("SOW_FFMPEG", "/custom/ffmpeg")
path, err := ResolveFFmpeg()
if err != nil {
t.Fatalf("unexpected error: %v", err)
}
if path != "/custom/ffmpeg" {
t.Fatalf("expected '/custom/ffmpeg', got %q", path)
}
}
func TestResolveFFprobeEnv(t *testing.T) {
t.Setenv("SOW_FFPROBE", "/custom/ffprobe")
path, err := ResolveFFprobe()
if err != nil {
t.Fatalf("unexpected error: %v", err)
}
if path != "/custom/ffprobe" {
t.Fatalf("expected '/custom/ffprobe', got %q", path)
}
}
func TestFFmpegMissingErrHasInstallHints(t *testing.T) {
msg := ffmpegMissingErr("ffmpeg").Error()
for _, want := range []string{"ffmpeg", "winget", "brew", "apt", "SOW_FFMPEG"} {
if !strings.Contains(msg, want) {
t.Errorf("ffmpeg error missing %q; got:\n%s", want, msg)
}
}
probe := ffmpegMissingErr("ffprobe").Error()
if !strings.Contains(probe, "SOW_FFPROBE") {
t.Errorf("ffprobe error should mention SOW_FFPROBE; got:\n%s", probe)
}
}
-197
View File
@@ -1,197 +0,0 @@
package music
import (
"crypto/sha1"
"fmt"
"path/filepath"
"strings"
)
func GenerateStem(fileName, prefix string, used map[string]struct{}) (string, error) {
return GenerateStemWithMax(fileName, prefix, MaxStemLen, used)
}
func GenerateStemWithMax(fileName, prefix string, maxStemLen int, used map[string]struct{}) (string, error) {
if maxStemLen <= 0 {
maxStemLen = MaxStemLen
}
maxCore := maxStemLen - len(prefix)
if maxCore < 3 {
return "", fmt.Errorf("prefix too long: %q", prefix)
}
base := strings.TrimSuffix(fileName, filepath.Ext(fileName))
words := SlugWords(base)
core := MakeMusicCore(words, maxCore)
if core == "" {
sum := sha1.Sum([]byte(fileName))
core = fmt.Sprintf("%x", sum[:])[:maxCore]
}
stem := strings.TrimRight((prefix + core)[:Min(maxStemLen, len(prefix)+len(core))], "_")
if stem == "" {
return "", fmt.Errorf("could not derive music stem for %s", fileName)
}
return UniqueNameWithMax(stem, maxStemLen, used), nil
}
func SlugWords(value string) []string {
value = BracketRE.ReplaceAllString(value, " ")
value = strings.ReplaceAll(value, "&", " and ")
value = strings.ReplaceAll(value, "'", "")
value = strings.ReplaceAll(value, "\u2019", "")
value = strings.ToLower(value)
var ascii strings.Builder
for _, r := range value {
switch {
case r >= 'a' && r <= 'z':
ascii.WriteRune(r)
case r >= '0' && r <= '9':
ascii.WriteRune(r)
default:
ascii.WriteRune(' ')
}
}
rawWords := SplitRE.Split(ascii.String(), -1)
words := make([]string, 0, len(rawWords))
for _, word := range rawWords {
if word == "" {
continue
}
if _, drop := DropWords[word]; drop {
continue
}
if YearRE.MatchString(word) {
continue
}
words = append(words, word)
}
return words
}
func MakeMusicCore(words []string, maxCore int) string {
core := ""
for _, word := range words {
candidate := word
if core != "" {
candidate = core + "_" + word
}
if len(candidate) <= maxCore {
core = candidate
continue
}
break
}
if core != "" {
return core
}
abbrev := make([]string, 0, Min(4, len(words)))
for _, word := range words {
if len(abbrev) == 4 {
break
}
abbrev = append(abbrev, AbbreviateWord(word, 5))
}
for _, word := range abbrev {
candidate := word
if core != "" {
candidate = core + "_" + word
}
if len(candidate) <= maxCore {
core = candidate
continue
}
break
}
if core != "" {
return core
}
if len(words) > 0 {
return strings.Trim(words[0][:Min(maxCore, len(words[0]))], "_")
}
return ""
}
func AbbreviateWord(word string, maxLen int) string {
if len(word) <= maxLen {
return word
}
consonants := VowelRE.ReplaceAllString(word, "")
candidate := word[:1]
if len(consonants) > 1 {
candidate += consonants[1:]
}
if len(candidate) >= 3 {
return candidate[:Min(maxLen, len(candidate))]
}
return word[:Min(maxLen, len(word))]
}
func UniqueName(stem string, used map[string]struct{}) string {
return UniqueNameWithMax(stem, MaxStemLen, used)
}
func UniqueNameWithMax(stem string, maxStemLen int, used map[string]struct{}) string {
if maxStemLen <= 0 {
maxStemLen = MaxStemLen
}
if _, exists := used[stem]; !exists {
used[stem] = struct{}{}
return stem
}
for index := 1; ; index++ {
suffix := fmt.Sprintf("_%d", index)
prefixLen := Min(len(stem), Max(0, maxStemLen-len(suffix)))
candidate := strings.TrimRight(stem[:prefixLen], "_") + suffix
if _, exists := used[candidate]; exists {
continue
}
used[candidate] = struct{}{}
return candidate
}
}
func ReserveName(stem string, used map[string]struct{}) error {
stem = strings.ToLower(strings.TrimSpace(stem))
if stem == "" {
return fmt.Errorf("output filename is empty")
}
if _, exists := used[stem]; exists {
return fmt.Errorf("output filename %s collides with an existing asset", stem)
}
used[stem] = struct{}{}
return nil
}
func ValidateManualOutputFile(name string) error {
return ValidateManualOutputFileWithRules(name, ".bmu", MaxStemLen)
}
func ValidateManualOutputFileWithRules(name, extension string, maxStemLen int) error {
name = strings.ToLower(strings.TrimSpace(name))
extension = strings.ToLower(strings.TrimSpace(extension))
if extension == "" {
extension = ".bmu"
}
if maxStemLen <= 0 {
maxStemLen = MaxStemLen
}
if !strings.HasSuffix(name, extension) {
return fmt.Errorf("output file must end with %s", extension)
}
stem := strings.TrimSuffix(name, extension)
if stem == "" {
return fmt.Errorf("output file stem is empty")
}
if len(stem) > maxStemLen {
return fmt.Errorf("output file stem %q exceeds %d characters", stem, maxStemLen)
}
for _, r := range stem {
switch {
case r >= 'a' && r <= 'z':
case r >= '0' && r <= '9':
case r == '_':
default:
return fmt.Errorf("output file stem %q contains unsupported character %q", stem, string(r))
}
}
return nil
}
+141 -83
View File
@@ -26,8 +26,6 @@ type BuildResult struct {
HAKPaths []string HAKPaths []string
Manifest string Manifest string
AutogenManifestPaths []string AutogenManifestPaths []string
CreditsArtifactPaths []string
CreditsSummary CreditsRefreshSummary
HAKSummary HAKArchiveSummary HAKSummary HAKArchiveSummary
Resources int Resources int
HAKAssets int HAKAssets int
@@ -36,21 +34,6 @@ type BuildResult struct {
TopPackageFiles int TopPackageFiles int
} }
type CreditsRefreshSummary struct {
ScannedDirs []string
TrackCount int
ArtifactPaths []string
GeneratedPaths []string
InventoryPath string
ChangedFiles int
Mappings []MusicFileMapping
}
type MusicFileMapping struct {
Source string
Output string
}
type HAKArchiveSummary struct { type HAKArchiveSummary struct {
Total int Total int
Reused int Reused int
@@ -109,8 +92,7 @@ type BuildHAKOptions struct {
Progress ProgressFunc Progress ProgressFunc
ArchiveNames []string ArchiveNames []string
SourceManifestPath string SourceManifestPath string
SkipMusic bool ContentAddressedRoot string
MusicDatasetIDs []string
} }
func Build(p *project.Project) (BuildResult, error) { func Build(p *project.Project) (BuildResult, error) {
@@ -196,7 +178,7 @@ func BuildHAKsWithProgress(p *project.Project, progress ProgressFunc) (BuildResu
} }
func BuildHAKsWithOptions(p *project.Project, opts BuildHAKOptions) (BuildResult, error) { func BuildHAKsWithOptions(p *project.Project, opts BuildHAKOptions) (BuildResult, error) {
return planOrBuildHAKs(p, opts.Progress, true, opts.ArchiveNames, opts.SourceManifestPath, opts.SkipMusic, opts.MusicDatasetIDs) return planOrBuildHAKs(p, opts.Progress, true, opts)
} }
func PlanHAKs(p *project.Project) (BuildResult, error) { func PlanHAKs(p *project.Project) (BuildResult, error) {
@@ -208,18 +190,18 @@ func PlanHAKsWithProgress(p *project.Project, progress ProgressFunc) (BuildResul
} }
func PlanHAKsWithOptions(p *project.Project, opts BuildHAKOptions) (BuildResult, error) { func PlanHAKsWithOptions(p *project.Project, opts BuildHAKOptions) (BuildResult, error) {
return planOrBuildHAKs(p, opts.Progress, false, opts.ArchiveNames, opts.SourceManifestPath, opts.SkipMusic, opts.MusicDatasetIDs) return planOrBuildHAKs(p, opts.Progress, false, opts)
} }
func buildHAKs(p *project.Project, progress ProgressFunc) (BuildResult, error) { func buildHAKs(p *project.Project, progress ProgressFunc) (BuildResult, error) {
return planOrBuildHAKs(p, progress, true, nil, "", false, nil) return planOrBuildHAKs(p, progress, true, BuildHAKOptions{})
} }
func planHAKs(p *project.Project, progress ProgressFunc) (BuildResult, error) { func planHAKs(p *project.Project, progress ProgressFunc) (BuildResult, error) {
return planOrBuildHAKs(p, progress, false, nil, "", false, nil) return planOrBuildHAKs(p, progress, false, BuildHAKOptions{})
} }
func planOrBuildHAKs(p *project.Project, progress ProgressFunc, writeArchives bool, archiveNames []string, sourceManifestPath string, skipMusic bool, musicDatasetIDs []string) (result BuildResult, err error) { func planOrBuildHAKs(p *project.Project, progress ProgressFunc, writeArchives bool, opts BuildHAKOptions) (result BuildResult, err error) {
preserveExistingHAKs := p.EffectiveConfig().Build.KeepExistingHAKs || envBool("SOW_BUILD_HAKS_KEEP_EXISTING") preserveExistingHAKs := p.EffectiveConfig().Build.KeepExistingHAKs || envBool("SOW_BUILD_HAKS_KEEP_EXISTING")
progressf(progress, "Validating project...") progressf(progress, "Validating project...")
@@ -227,31 +209,24 @@ func planOrBuildHAKs(p *project.Project, progress ProgressFunc, writeArchives bo
return BuildResult{}, err return BuildResult{}, err
} }
directManifest, err := loadDirectSourceManifest(opts.SourceManifestPath)
if err != nil {
return BuildResult{}, err
}
if directManifest != nil {
return buildDirectHAKs(p, progress, writeArchives, directManifest, opts)
}
progressf(progress, "Collecting asset resources...") progressf(progress, "Collecting asset resources...")
allowedAssets, sourceManifest, err := loadSourceManifestAssetSet(sourceManifestPath) allowedAssets, sourceManifest, err := loadSourceManifestAssetSet(opts.SourceManifestPath)
if err != nil { if err != nil {
return BuildResult{}, err return BuildResult{}, err
} }
musicAssets := &preparedMusicAssets{SkipSourceRel: map[string]struct{}{}}
if !skipMusic {
musicAssets, err = prepareMusicAssetsWithOptions(p, musicPrepareOptions{
datasetIDs: musicDatasetIDs,
write: writeArchives,
})
if err != nil {
return BuildResult{}, err
}
}
defer func() {
if cleanupErr := cleanupPreparedMusicAssets(musicAssets); cleanupErr != nil && err == nil {
err = cleanupErr
}
}()
generated2DAAssets, err := topdata.BuildGenerated2DAAssets(p, progress) generated2DAAssets, err := topdata.BuildGenerated2DAAssets(p, progress)
if err != nil { if err != nil {
return BuildResult{}, err return BuildResult{}, err
} }
assetResources, err := collectAssetResources(p, false, allowedAssets, musicAssets, generated2DAAssets) assetResources, err := collectAssetResources(p, false, allowedAssets, generated2DAAssets)
if err != nil { if err != nil {
return BuildResult{}, err return BuildResult{}, err
} }
@@ -268,8 +243,6 @@ func planOrBuildHAKs(p *project.Project, progress ProgressFunc, writeArchives bo
} }
result = BuildResult{HAKAssets: len(assetResources)} result = BuildResult{HAKAssets: len(assetResources)}
result.CreditsArtifactPaths = append(result.CreditsArtifactPaths, musicAssets.Artifacts...)
result.CreditsSummary = musicAssets.Summary
if err := writeAutogenManifestOutputs(progress, autogenManifests); err != nil { if err := writeAutogenManifestOutputs(progress, autogenManifests); err != nil {
return BuildResult{}, err return BuildResult{}, err
} }
@@ -294,7 +267,7 @@ func planOrBuildHAKs(p *project.Project, progress ProgressFunc, writeArchives bo
if err != nil { if err != nil {
return BuildResult{}, err return BuildResult{}, err
} }
chunks, err = filterHAKChunksByName(chunks, archiveNames) chunks, err = filterHAKChunksByName(chunks, opts.ArchiveNames)
if err != nil { if err != nil {
return BuildResult{}, err return BuildResult{}, err
} }
@@ -369,16 +342,8 @@ func planOrBuildHAKs(p *project.Project, progress ProgressFunc, writeArchives bo
if err := os.MkdirAll(filepath.Dir(hakPath), 0o755); err != nil { if err := os.MkdirAll(filepath.Dir(hakPath), 0o755); err != nil {
return BuildResult{}, fmt.Errorf("create hak archive dir: %w", err) return BuildResult{}, fmt.Errorf("create hak archive dir: %w", err)
} }
hakOutput, err := os.Create(hakPath) if err := writeHAKArchive(hakPath, resourceSlice(chunk.Assets)); err != nil {
if err != nil { return BuildResult{}, err
return BuildResult{}, fmt.Errorf("create hak archive: %w", err)
}
if err := erf.Write(hakOutput, erf.New("HAK ", resourceSlice(chunk.Assets))); err != nil {
hakOutput.Close()
return BuildResult{}, fmt.Errorf("write hak archive: %w", err)
}
if err := hakOutput.Close(); err != nil {
return BuildResult{}, fmt.Errorf("close hak archive: %w", err)
} }
} }
@@ -408,6 +373,110 @@ func planOrBuildHAKs(p *project.Project, progress ProgressFunc, writeArchives bo
return result, nil return result, nil
} }
// buildDirectHAKs packages HAKs directly from the content-addressed source blob
// cache. It skips generated 2da discovery, git creation-time lookups, LFS
// discovery, and any materialized asset tree scan.
func buildDirectHAKs(p *project.Project, progress ProgressFunc, writeArchives bool, manifest *SourceBuildManifest, opts BuildHAKOptions) (BuildResult, error) {
progressf(progress, "Collecting asset resources...")
assetResources, err := collectDirectSourceResources(manifest, opts.ContentAddressedRoot)
if err != nil {
return BuildResult{}, err
}
result := BuildResult{HAKAssets: len(assetResources)}
progressf(progress, "Planning HAK chunks...")
chunks, err := chunksFromSourceManifest(assetResources, manifest.HAKs)
if err != nil {
return BuildResult{}, err
}
chunks, err = filterHAKChunksByName(chunks, opts.ArchiveNames)
if err != nil {
return BuildResult{}, err
}
buildManifest := BuildManifest{
ModuleHAKs: append([]string(nil), manifest.ModuleHAKs...),
HAKs: make([]BuildManifestHAK, 0, len(chunks)),
}
for _, chunk := range chunks {
buildManifest.HAKs = append(buildManifest.HAKs, buildManifestEntry(chunk))
}
manifestPath := p.HAKManifestPath()
manifestBytes, err := json.MarshalIndent(buildManifest, "", " ")
if err != nil {
return BuildResult{}, fmt.Errorf("marshal hak manifest: %w", err)
}
manifestBytes = append(manifestBytes, '\n')
if writeArchives {
progressf(progress, "Writing HAK archives...")
result.HAKSummary.Total = len(chunks)
result.HAKPaths = make([]string, 0, len(chunks))
for index, chunk := range chunks {
hakPath := p.HAKArchivePath(chunk.Name)
progressf(progress, fmt.Sprintf("Writing HAK %d/%d: %s (%d assets)", index+1, len(chunks), chunk.Name, len(chunk.Assets)))
if err := os.MkdirAll(filepath.Dir(hakPath), 0o755); err != nil {
return BuildResult{}, fmt.Errorf("create hak archive dir: %w", err)
}
if err := writeHAKArchive(hakPath, resourceSlice(chunk.Assets)); err != nil {
return BuildResult{}, err
}
result.HAKSummary.Written++
result.HAKSummary.Actions = append(result.HAKSummary.Actions, HAKArchiveAction{
Name: chunk.Name,
AssetCount: len(chunk.Assets),
OutputPath: hakPath,
})
result.HAKPaths = append(result.HAKPaths, hakPath)
}
}
progressf(progress, "Writing HAK manifest...")
if err := os.MkdirAll(filepath.Dir(manifestPath), 0o755); err != nil {
return BuildResult{}, fmt.Errorf("create hak manifest dir: %w", err)
}
if err := os.WriteFile(manifestPath, manifestBytes, 0o644); err != nil {
return BuildResult{}, fmt.Errorf("write hak manifest: %w", err)
}
result.Manifest = manifestPath
return result, nil
}
// writeHAKArchive publishes a completed HAK atomically, so a failed write never
// leaves a partial archive at the final path.
func writeHAKArchive(hakPath string, resources []erf.Resource) error {
tmpPath := hakPath + ".tmp"
if err := os.Remove(tmpPath); err != nil && !errors.Is(err, os.ErrNotExist) {
return fmt.Errorf("remove stale temporary hak %s: %w", tmpPath, err)
}
output, err := os.OpenFile(tmpPath, os.O_CREATE|os.O_EXCL|os.O_WRONLY, 0o644)
if err != nil {
return fmt.Errorf("create temporary hak archive %s: %w", tmpPath, err)
}
if err := erf.Write(output, erf.New("HAK ", resources)); err != nil {
output.Close()
os.Remove(tmpPath)
return fmt.Errorf("write hak archive: %w", err)
}
if err := output.Sync(); err != nil {
output.Close()
os.Remove(tmpPath)
return fmt.Errorf("sync hak archive: %w", err)
}
if err := output.Close(); err != nil {
os.Remove(tmpPath)
return fmt.Errorf("close hak archive: %w", err)
}
if err := os.Rename(tmpPath, hakPath); err != nil {
os.Remove(tmpPath)
return fmt.Errorf("publish hak archive %s: %w", hakPath, err)
}
return nil
}
func writeAutogenManifestOutputs(progress ProgressFunc, manifests []topdata.ProducedAutogenManifest) error { func writeAutogenManifestOutputs(progress ProgressFunc, manifests []topdata.ProducedAutogenManifest) error {
if len(manifests) == 0 { if len(manifests) == 0 {
return nil return nil
@@ -546,7 +615,7 @@ func collectModuleResources(p *project.Project, moduleHakOrder []string) ([]erf.
return moduleResources, nil return moduleResources, nil
} }
func collectAssetResources(p *project.Project, requireContent bool, allowed map[string]struct{}, musicAssets *preparedMusicAssets, generated2DAAssets []topdata.Generated2DAAsset) ([]assetResource, error) { func collectAssetResources(p *project.Project, requireContent bool, allowed map[string]struct{}, generated2DAAssets []topdata.Generated2DAAsset) ([]assetResource, error) {
var hakResources []assetResource var hakResources []assetResource
assetCreatedAt, err := collectGitAssetCreationTimes(p) assetCreatedAt, err := collectGitAssetCreationTimes(p)
if err != nil { if err != nil {
@@ -556,18 +625,19 @@ func collectAssetResources(p *project.Project, requireContent bool, allowed map[
if err != nil { if err != nil {
return nil, err return nil, err
} }
assetsRelPath, err := filepath.Rel(p.Root, p.AssetsDir()) // A module-only project (consumes prebuilt HAKs, has no paths.assets) has no
// asset files to collect — AssetFiles is empty and AssetsDir() is "". Skip the
// relative-path resolve, which would otherwise fail with `Rel: can't make ""`.
assetsRelPath := ""
if p.AssetsDir() != "" {
assetsRelPath, err = filepath.Rel(p.Root, p.AssetsDir())
if err != nil { if err != nil {
return nil, fmt.Errorf("resolve assets dir relative path: %w", err) return nil, fmt.Errorf("resolve assets dir relative path: %w", err)
} }
assetsRelPath = filepath.ToSlash(assetsRelPath) assetsRelPath = filepath.ToSlash(assetsRelPath)
}
for _, rel := range p.Inventory.AssetFiles { for _, rel := range p.Inventory.AssetFiles {
if musicAssets != nil {
if _, skip := musicAssets.SkipSourceRel[filepath.ToSlash(rel)]; skip {
continue
}
}
if len(allowed) > 0 { if len(allowed) > 0 {
if _, ok := allowed[filepath.ToSlash(rel)]; !ok { if _, ok := allowed[filepath.ToSlash(rel)]; !ok {
continue continue
@@ -595,16 +665,6 @@ func collectAssetResources(p *project.Project, requireContent bool, allowed map[
ContentID: resourceInfo.ContentID, ContentID: resourceInfo.ContentID,
}) })
} }
if musicAssets != nil {
for _, asset := range musicAssets.Generated {
if len(allowed) > 0 {
if _, ok := allowed[filepath.ToSlash(asset.Rel)]; !ok {
continue
}
}
hakResources = append(hakResources, asset)
}
}
for _, asset := range generated2DAAssets { for _, asset := range generated2DAAssets {
rel := filepath.ToSlash(asset.Rel) rel := filepath.ToSlash(asset.Rel)
if len(allowed) > 0 { if len(allowed) > 0 {
@@ -672,7 +732,7 @@ func chunksFromManifest(assets []assetResource, entries []BuildManifestHAK) ([]h
} }
chunkAssets = append(chunkAssets, asset) chunkAssets = append(chunkAssets, asset)
} }
chunks = append(chunks, hakChunk{ chunk := hakChunk{
Config: project.HAKConfig{ Config: project.HAKConfig{
Name: entry.Group, Name: entry.Group,
Priority: entry.Priority, Priority: entry.Priority,
@@ -683,7 +743,14 @@ func chunksFromManifest(assets []assetResource, entries []BuildManifestHAK) ([]h
Name: entry.Name, Name: entry.Name,
Assets: chunkAssets, Assets: chunkAssets,
Size: erf.ArchiveSize(resourceSlice(chunkAssets)), Size: erf.ArchiveSize(resourceSlice(chunkAssets)),
}) }
// A manifest can name two distinct source paths that collapse to the same
// resref+type (e.g. creature/foo.mdl and placeable/foo.mdl); without this
// guard the second would silently shadow the first in the ERF.
if err := ensureUniqueChunkResources(chunk); err != nil {
return nil, err
}
chunks = append(chunks, chunk)
} }
return chunks, nil return chunks, nil
} }
@@ -1499,20 +1566,11 @@ func plannedModuleHAKOrder(p *project.Project) (order []string, err error) {
return append([]string(nil), manifest.ModuleHAKs...), nil return append([]string(nil), manifest.ModuleHAKs...), nil
} }
musicAssets, err := prepareMusicAssets(p)
if err != nil {
return nil, err
}
defer func() {
if cleanupErr := cleanupPreparedMusicAssets(musicAssets); cleanupErr != nil && err == nil {
err = cleanupErr
}
}()
generated2DAAssets, err := topdata.BuildGenerated2DAAssets(p, nil) generated2DAAssets, err := topdata.BuildGenerated2DAAssets(p, nil)
if err != nil { if err != nil {
return nil, err return nil, err
} }
assetResources, err := collectAssetResources(p, false, nil, musicAssets, generated2DAAssets) assetResources, err := collectAssetResources(p, false, nil, generated2DAAssets)
if err != nil { if err != nil {
return nil, err return nil, err
} }
+1 -1
View File
@@ -223,7 +223,7 @@ func ensureBuildIsCurrent(p *project.Project, modulePath string, hakPaths []stri
} }
if !newestSource.IsZero() && newestSource.After(oldestArchive) { if !newestSource.IsZero() && newestSource.After(oldestArchive) {
return fmt.Errorf("built archives are older than source file %s; run build-module or build-haks before compare", newestPath) return fmt.Errorf("built archives are older than source file %s; run crucible module build or crucible hak build before compare", newestPath)
} }
return nil return nil
} }
-524
View File
@@ -1,524 +0,0 @@
package pipeline
import (
"encoding/json"
"fmt"
"os"
"path/filepath"
"sort"
"strings"
"git.westgate.pw/ShadowsOverWestgate/sow-tools/internal/erf"
"git.westgate.pw/ShadowsOverWestgate/sow-tools/internal/music"
"git.westgate.pw/ShadowsOverWestgate/sow-tools/internal/project"
)
type preparedMusicAssets struct {
Generated []assetResource
SkipSourceRel map[string]struct{}
Artifacts []string
TempRoots []string
Summary CreditsRefreshSummary
}
type MusicBuildOptions struct {
DatasetIDs []string
Write bool
}
type musicPrepareOptions struct {
datasetIDs []string
write bool
}
type musicSourceGroup struct {
DatasetID string
Dataset project.EffectiveMusicDataset
SourceDir string
OutputDir string
Sources []string
}
type musicCreditGroup struct {
CreditsRoot string
SourceDir string
Entries []music.CreditsEntry
}
func musicRootPath(p *project.Project, path string) string {
if strings.TrimSpace(path) == "" {
return p.Root
}
if filepath.IsAbs(path) {
return path
}
return filepath.Join(p.Root, filepath.FromSlash(path))
}
func effectiveMusicDatasets(p *project.Project, onlyIDs []string) (map[string]project.EffectiveMusicDataset, error) {
effective := p.EffectiveConfig()
wanted := map[string]struct{}{}
for _, id := range onlyIDs {
id = strings.TrimSpace(id)
if id != "" {
wanted[id] = struct{}{}
}
}
datasets := map[string]project.EffectiveMusicDataset{}
for id, dataset := range effective.Music.Datasets {
if len(wanted) > 0 {
if _, ok := wanted[id]; !ok {
continue
}
delete(wanted, id)
}
datasets[id] = dataset
}
if len(wanted) > 0 {
unknown := make([]string, 0, len(wanted))
for id := range wanted {
unknown = append(unknown, id)
}
sort.Strings(unknown)
return nil, fmt.Errorf("unknown music dataset(s): %s", strings.Join(unknown, ", "))
}
if len(datasets) == 0 && len(wanted) == 0 {
for _, rel := range p.Inventory.AssetFiles {
ext := strings.ToLower(filepath.Ext(rel))
if !music.PathIsUnder("envi/music", rel) {
continue
}
if _, ok := extensionSet(effective.Music.ConvertExtensions)[ext]; ok {
datasets["legacy_envi_music"] = project.EffectiveMusicDataset{
Source: "envi/music",
Output: "envi/music",
StageRoot: effective.Music.StageRoot,
CreditsRoot: effective.Music.CreditsRoot,
NamingScheme: effective.Music.NamingScheme,
OutputExtension: effective.Music.OutputExtension,
MaxStemLength: effective.Music.MaxStemLength,
PackageMode: "hak_asset",
ConvertExtensions: effective.Music.ConvertExtensions,
}
break
}
}
}
return datasets, nil
}
func extensionSet(values []string) map[string]struct{} {
out := make(map[string]struct{}, len(values))
for _, value := range values {
value = strings.ToLower(strings.TrimSpace(value))
if value != "" {
out[value] = struct{}{}
}
}
return out
}
func datasetForMusicSource(datasets map[string]project.EffectiveMusicDataset, rel, ext string) (string, project.EffectiveMusicDataset, bool) {
bestID := ""
var best project.EffectiveMusicDataset
bestLen := -1
for id, dataset := range datasets {
if _, ok := extensionSet(dataset.ConvertExtensions)[ext]; !ok {
continue
}
if !music.PathIsUnder(dataset.Source, rel) {
continue
}
if len(dataset.Source) > bestLen {
bestID = id
best = dataset
bestLen = len(dataset.Source)
}
}
return bestID, best, bestID != ""
}
func plannedMusicAsset(outputRel, sourceRel string) (assetResource, error) {
extension := strings.TrimPrefix(strings.ToLower(filepath.Ext(outputRel)), ".")
resourceType, ok := erf.HAKResourceTypeForExtension(extension)
if !ok {
return assetResource{}, fmt.Errorf("unsupported generated music resource extension %q", filepath.Ext(outputRel))
}
name := strings.ToLower(strings.TrimSuffix(filepath.Base(outputRel), filepath.Ext(outputRel)))
return assetResource{
Rel: outputRel,
Resource: erf.Resource{
Name: name,
Type: resourceType,
},
ContentID: "music-plan:" + filepath.ToSlash(sourceRel),
}, nil
}
func BuildMusic(p *project.Project, opts MusicBuildOptions) (BuildResult, error) {
prepared, err := prepareMusicAssetsWithOptions(p, musicPrepareOptions{datasetIDs: opts.DatasetIDs, write: opts.Write})
if err != nil {
return BuildResult{}, err
}
if err := cleanupPreparedMusicAssets(prepared); err != nil {
return BuildResult{}, err
}
return BuildResult{
CreditsArtifactPaths: prepared.Artifacts,
CreditsSummary: prepared.Summary,
HAKAssets: len(prepared.Generated),
}, nil
}
func prepareMusicAssets(p *project.Project) (*preparedMusicAssets, error) {
return prepareMusicAssetsWithOptions(p, musicPrepareOptions{write: true})
}
func prepareMusicAssetsWithOptions(p *project.Project, opts musicPrepareOptions) (*preparedMusicAssets, error) {
usedNames := make(map[string]struct{})
groupsByKey := make(map[string]*musicSourceGroup)
datasets, err := effectiveMusicDatasets(p, opts.datasetIDs)
if err != nil {
return nil, err
}
for _, rel := range p.Inventory.AssetFiles {
rel = filepath.ToSlash(rel)
ext := strings.ToLower(filepath.Ext(rel))
name := strings.ToLower(strings.TrimSuffix(filepath.Base(rel), filepath.Ext(rel)))
if datasetID, dataset, ok := datasetForMusicSource(datasets, rel, ext); ok {
sourceDir := filepath.ToSlash(filepath.Dir(rel))
relDir, err := filepath.Rel(filepath.FromSlash(dataset.Source), filepath.FromSlash(sourceDir))
if err != nil {
return nil, fmt.Errorf("resolve music source directory %s: %w", sourceDir, err)
}
relDir = filepath.ToSlash(relDir)
outputDir := dataset.Output
if relDir != "." && relDir != "" {
outputDir = filepath.ToSlash(filepath.Join(outputDir, filepath.FromSlash(relDir)))
}
key := datasetID + "\x00" + sourceDir
group := groupsByKey[key]
if group == nil {
group = &musicSourceGroup{
DatasetID: datasetID,
Dataset: dataset,
SourceDir: sourceDir,
OutputDir: outputDir,
}
groupsByKey[key] = group
}
group.Sources = append(group.Sources, rel)
continue
}
if name != "" {
usedNames[name] = struct{}{}
}
}
result := &preparedMusicAssets{
SkipSourceRel: make(map[string]struct{}),
}
if len(groupsByKey) == 0 {
if !opts.write {
return result, nil
}
writeResult, err := writeCreditsArtifacts(p, nil)
if err != nil {
return nil, err
}
result.Artifacts = writeResult.Artifacts
result.Summary = writeResult.Summary
return result, nil
}
groups := make([]*musicSourceGroup, 0, len(groupsByKey))
for _, group := range groupsByKey {
sort.Strings(group.Sources)
groups = append(groups, group)
}
sort.Slice(groups, func(i, j int) bool {
if groups[i].SourceDir != groups[j].SourceDir {
return groups[i].SourceDir < groups[j].SourceDir
}
return groups[i].DatasetID < groups[j].DatasetID
})
creditGroups := make([]musicCreditGroup, 0, len(groups))
for _, group := range groups {
result.Summary.ScannedDirs = append(result.Summary.ScannedDirs, group.SourceDir)
result.Summary.TrackCount += len(group.Sources)
}
ffmpegPath := ""
ffprobePath := ""
if opts.write {
var err error
ffmpegPath, err = music.ResolveFFmpegWithConfig(p.MusicFFmpegPath())
if err != nil {
return nil, err
}
ffprobePath, err = music.ResolveFFprobeWithConfig(p.MusicFFprobePath())
if err != nil {
return nil, err
}
}
for _, group := range groups {
stageRoot := musicRootPath(p, group.Dataset.StageRoot)
result.TempRoots = append(result.TempRoots, stageRoot)
prefix := music.SanitizePrefix(group.Dataset.Prefix)
overlayPath := filepath.Join(p.AssetsDir(), filepath.FromSlash(group.SourceDir), p.EffectiveConfig().Music.CreditsOverlay)
overlay, err := music.ParseCreditsOverlay(overlayPath)
if err != nil {
return nil, fmt.Errorf("parse credits overlay %s: %w", overlayPath, err)
}
entries := make([]music.CreditsEntry, 0, len(group.Sources))
for _, rel := range group.Sources {
base := filepath.Base(rel)
manual := overlay.Match(base, "")
outputFile := ""
switch {
case manual != nil && strings.TrimSpace(manual.OutputFile) != "":
outputFile = strings.ToLower(strings.TrimSpace(manual.OutputFile))
if err := music.ValidateManualOutputFileWithRules(outputFile, group.Dataset.OutputExtension, group.Dataset.MaxStemLength); err != nil {
return nil, fmt.Errorf("manual credits output for %s: %w", rel, err)
}
if err := music.ReserveName(strings.TrimSuffix(outputFile, filepath.Ext(outputFile)), usedNames); err != nil {
return nil, fmt.Errorf("manual credits output for %s: %w", rel, err)
}
default:
stem, err := music.GenerateStemWithMax(base, prefix, group.Dataset.MaxStemLength, usedNames)
if err != nil {
return nil, fmt.Errorf("generate music name for %s: %w", rel, err)
}
outputFile = stem + group.Dataset.OutputExtension
}
outputRel := filepath.ToSlash(filepath.Join(group.OutputDir, outputFile))
result.Summary.Mappings = append(result.Summary.Mappings, MusicFileMapping{
Source: rel,
Output: outputRel,
})
result.SkipSourceRel[rel] = struct{}{}
if !opts.write {
if group.Dataset.PackageMode == "hak_asset" {
asset, err := plannedMusicAsset(outputRel, rel)
if err != nil {
return nil, err
}
result.Generated = append(result.Generated, asset)
}
continue
}
metadata, err := music.ProbeMetadata(ffprobePath, filepath.Join(p.AssetsDir(), filepath.FromSlash(rel)))
if err != nil {
return nil, fmt.Errorf("probe music metadata for %s: %w", rel, err)
}
entry := music.CreditsEntry{
Artist: metadata.Artist,
Title: music.FirstNonEmpty(metadata.Title, strings.TrimSuffix(base, filepath.Ext(base))),
OutputFile: outputFile,
OriginalFile: base,
Album: metadata.Album,
Date: metadata.Date,
Rights: music.FirstNonEmpty(metadata.Rights, music.JoinNonEmpty("; ", metadata.License, metadata.Copyright)),
Notes: music.FirstNonEmpty(metadata.Notes, metadata.Comment),
}
music.ApplyCreditsOverlay(&entry, manual)
stagePath := filepath.Join(stageRoot, filepath.FromSlash(outputRel))
if err := os.MkdirAll(filepath.Dir(stagePath), 0o755); err != nil {
return nil, fmt.Errorf("create music stage dir: %w", err)
}
if err := music.ConvertSource(ffmpegPath, filepath.Join(p.AssetsDir(), filepath.FromSlash(rel)), stagePath); err != nil {
return nil, fmt.Errorf("convert music %s: %w", rel, err)
}
if group.Dataset.PackageMode == "hak_asset" {
resourceInfo, err := assetResourceFromPath(stagePath, outputRel, lfsAssetInfo{}, false)
if err != nil {
return nil, fmt.Errorf("load converted music %s: %w", outputRel, err)
}
info, err := os.Stat(stagePath)
if err != nil {
return nil, fmt.Errorf("stat converted music %s: %w", stagePath, err)
}
result.Generated = append(result.Generated, assetResource{
Rel: outputRel,
Resource: resourceInfo.Resource,
Size: erf.ArchiveSize([]erf.Resource{resourceInfo.Resource}),
CreatedAt: info.ModTime(),
ContentID: resourceInfo.ContentID,
})
}
entries = append(entries, entry)
}
if opts.write {
if err := music.ValidateOverlayEntries(group.SourceDir, overlay, entries); err != nil {
return nil, err
}
creditGroups = append(creditGroups, musicCreditGroup{
CreditsRoot: group.Dataset.CreditsRoot,
SourceDir: group.SourceDir,
Entries: entries,
})
}
}
if !opts.write {
sort.Slice(result.Generated, func(i, j int) bool {
return compareResourceKeys(result.Generated[i].Resource, result.Generated[j].Resource) < 0
})
return result, nil
}
writeResult, err := writeCreditsArtifacts(p, creditGroups)
if err != nil {
return nil, err
}
writeResult.Summary.ScannedDirs = append(writeResult.Summary.ScannedDirs, result.Summary.ScannedDirs...)
writeResult.Summary.TrackCount = result.Summary.TrackCount
result.Artifacts = writeResult.Artifacts
result.Summary = writeResult.Summary
sort.Slice(result.Generated, func(i, j int) bool {
return compareResourceKeys(result.Generated[i].Resource, result.Generated[j].Resource) < 0
})
return result, nil
}
type creditsArtifactWriteResult struct {
Artifacts []string
Summary CreditsRefreshSummary
}
func writeCreditsArtifacts(p *project.Project, generated []musicCreditGroup) (creditsArtifactWriteResult, error) {
defaultCreditsRoot := p.CreditsDir()
sources := make([]music.CreditsSource, 0)
desiredByRoot := map[string]map[string][]byte{}
summary := CreditsRefreshSummary{}
if generated != nil {
sort.Slice(generated, func(i, j int) bool {
if generated[i].SourceDir != generated[j].SourceDir {
return generated[i].SourceDir < generated[j].SourceDir
}
return generated[i].CreditsRoot < generated[j].CreditsRoot
})
for _, group := range generated {
creditsRoot := musicRootPath(p, group.CreditsRoot)
if desiredByRoot[creditsRoot] == nil {
desiredByRoot[creditsRoot] = map[string][]byte{}
}
entries := append([]music.CreditsEntry(nil), group.Entries...)
sort.Slice(entries, func(i, j int) bool {
if strings.ToLower(entries[i].Artist) != strings.ToLower(entries[j].Artist) {
return strings.ToLower(entries[i].Artist) < strings.ToLower(entries[j].Artist)
}
if strings.ToLower(entries[i].Title) != strings.ToLower(entries[j].Title) {
return strings.ToLower(entries[i].Title) < strings.ToLower(entries[j].Title)
}
return strings.ToLower(entries[i].OutputFile) < strings.ToLower(entries[j].OutputFile)
})
outputPath := filepath.Join(creditsRoot, filepath.FromSlash(group.SourceDir), p.EffectiveConfig().Music.CreditsOverlay)
desiredByRoot[creditsRoot][outputPath] = []byte(music.RenderCreditsMarkdown(entries))
summary.GeneratedPaths = append(summary.GeneratedPaths, outputPath)
for _, entry := range entries {
summary.Mappings = append(summary.Mappings, MusicFileMapping{
Source: entry.OriginalFile,
Output: entry.OutputFile,
})
}
sources = append(sources, music.CreditsSource{
Path: filepath.ToSlash(outputPath),
Kind: "generated_music",
Entries: entries,
})
}
}
err := filepath.WalkDir(p.AssetsDir(), func(path string, d os.DirEntry, walkErr error) error {
if walkErr != nil {
return walkErr
}
if d.IsDir() {
return nil
}
if strings.ToLower(d.Name()) != "credits.md" {
return nil
}
entries, err := music.ParseCreditsMarkdown(path)
if err != nil {
return fmt.Errorf("parse credits %s: %w", path, err)
}
rel, err := filepath.Rel(p.Root, path)
if err != nil {
return err
}
sources = append(sources, music.CreditsSource{
Path: filepath.ToSlash(rel),
Kind: "authored",
Entries: entries,
})
return nil
})
if err != nil {
return creditsArtifactWriteResult{}, err
}
sort.Slice(sources, func(i, j int) bool { return sources[i].Path < sources[j].Path })
payload, err := json.MarshalIndent(music.CreditsInventory{Sources: sources}, "", " ")
if err != nil {
return creditsArtifactWriteResult{}, fmt.Errorf("marshal credits inventory: %w", err)
}
payload = append(payload, '\n')
if len(desiredByRoot) == 0 {
desiredByRoot[defaultCreditsRoot] = map[string][]byte{}
}
artifacts := make([]string, 0)
changedFiles := 0
roots := make([]string, 0, len(desiredByRoot))
for root := range desiredByRoot {
roots = append(roots, root)
}
sort.Strings(roots)
for _, creditsRoot := range roots {
desiredFiles := desiredByRoot[creditsRoot]
inventoryPath := filepath.Join(creditsRoot, "credits.json")
desiredFiles[inventoryPath] = payload
if err := os.MkdirAll(creditsRoot, 0o755); err != nil {
return creditsArtifactWriteResult{}, fmt.Errorf("create credits dir: %w", err)
}
changed, err := music.SyncGeneratedCreditsArtifacts(creditsRoot, desiredFiles)
if err != nil {
return creditsArtifactWriteResult{}, err
}
changedFiles += changed
for path := range desiredFiles {
artifacts = append(artifacts, path)
}
if summary.InventoryPath == "" {
summary.InventoryPath = inventoryPath
}
}
sort.Strings(artifacts)
summary.ArtifactPaths = append(summary.ArtifactPaths, artifacts...)
summary.ChangedFiles = changedFiles
return creditsArtifactWriteResult{
Artifacts: artifacts,
Summary: summary,
}, nil
}
func cleanupPreparedMusicAssets(prepared *preparedMusicAssets) error {
if prepared == nil {
return nil
}
for _, root := range prepared.TempRoots {
if strings.TrimSpace(root) == "" {
continue
}
if err := os.RemoveAll(root); err != nil {
return fmt.Errorf("remove temporary music artifact root %s: %w", root, err)
}
}
return nil
}
+71 -392
View File
@@ -17,6 +17,35 @@ import (
"git.westgate.pw/ShadowsOverWestgate/sow-tools/internal/project" "git.westgate.pw/ShadowsOverWestgate/sow-tools/internal/project"
) )
func TestBuildHAKsCleansFailedTemporaryArchive(t *testing.T) {
root := t.TempDir()
p := loadDirectProject(t, root)
blobs := filepath.Join(root, "blobs")
sha := strings.Repeat("a", 64)
writeBlobAt(t, blobs, sha, "wxyz")
manifestPath := writeDirectManifest(t, root,
map[string]SourceAsset{"core/a.tga": {SHA256: sha, SizeBytes: 4}},
[]string{"core/a.tga"})
tmpPath := filepath.Join(root, "build", "core_01.hak.tmp")
mustWriteFile(t, tmpPath, "stale partial archive")
_, err := BuildHAKsWithOptions(p, BuildHAKOptions{
SourceManifestPath: manifestPath,
ContentAddressedRoot: blobs,
})
if err == nil {
t.Fatal("expected corrupt input to fail")
}
if _, err := os.Stat(filepath.Join(root, "build", "core_01.hak")); !errors.Is(err, os.ErrNotExist) {
t.Fatal("corrupt input must not leave a completed hak")
}
if _, err := os.Stat(tmpPath); !errors.Is(err, os.ErrNotExist) {
t.Fatal("failed build must clean temporary hak")
}
}
func TestBuildThenExtract(t *testing.T) { func TestBuildThenExtract(t *testing.T) {
root := t.TempDir() root := t.TempDir()
mustMkdir(t, filepath.Join(root, "src")) mustMkdir(t, filepath.Join(root, "src"))
@@ -1864,7 +1893,7 @@ func TestPlanHAKsWritesManifestWithoutArchives(t *testing.T) {
} }
} }
func TestBuildHAKsWritesConfiguredHeadVisualeffectsAutogenManifest(t *testing.T) { func TestBuildHAKsWritesConfiguredAccessoryVisualeffectsAutogenManifest(t *testing.T) {
root := t.TempDir() root := t.TempDir()
mustMkdir(t, filepath.Join(root, "src")) mustMkdir(t, filepath.Join(root, "src"))
mustMkdir(t, filepath.Join(root, "assets", "vfxs", "chest_accessories")) mustMkdir(t, filepath.Join(root, "assets", "vfxs", "chest_accessories"))
@@ -1951,7 +1980,7 @@ func TestBuildHAKsWritesConfiguredHeadVisualeffectsAutogenManifest(t *testing.T)
headRaw, err := os.ReadFile(filepath.Join(root, "build", "sow-accessory-vfx-manifest.json")) headRaw, err := os.ReadFile(filepath.Join(root, "build", "sow-accessory-vfx-manifest.json"))
if err != nil { if err != nil {
t.Fatalf("read head visualeffects manifest: %v", err) t.Fatalf("read accessory visualeffects manifest: %v", err)
} }
text := string(headRaw) text := string(headRaw)
for _, want := range []string{ for _, want := range []string{
@@ -1969,12 +1998,12 @@ func TestBuildHAKsWritesConfiguredHeadVisualeffectsAutogenManifest(t *testing.T)
`"model_stem": "hfx_hair_bangs"`, `"model_stem": "hfx_hair_bangs"`,
} { } {
if !strings.Contains(text, want) { if !strings.Contains(text, want) {
t.Fatalf("expected head visualeffects manifest to contain %q, got:\n%s", want, text) t.Fatalf("expected accessory visualeffects manifest to contain %q, got:\n%s", want, text)
} }
} }
} }
func TestBuildHAKsWritesFolderCategoryInHeadVisualeffectsAutogenManifest(t *testing.T) { func TestBuildHAKsWritesFolderCategoryInAccessoryVisualeffectsAutogenManifest(t *testing.T) {
root := t.TempDir() root := t.TempDir()
mustMkdir(t, filepath.Join(root, "src")) mustMkdir(t, filepath.Join(root, "src"))
mustMkdir(t, filepath.Join(root, "assets", "vfxs", "head_features", "sinfar", "ears_plt")) mustMkdir(t, filepath.Join(root, "assets", "vfxs", "head_features", "sinfar", "ears_plt"))
@@ -2053,7 +2082,7 @@ func TestBuildHAKsWritesFolderCategoryInHeadVisualeffectsAutogenManifest(t *test
} }
headRaw, err := os.ReadFile(filepath.Join(root, "build", "sow-accessory-vfx-manifest.json")) headRaw, err := os.ReadFile(filepath.Join(root, "build", "sow-accessory-vfx-manifest.json"))
if err != nil { if err != nil {
t.Fatalf("read head visualeffects manifest: %v", err) t.Fatalf("read accessory visualeffects manifest: %v", err)
} }
text := string(headRaw) text := string(headRaw)
for _, want := range []string{ for _, want := range []string{
@@ -2066,7 +2095,7 @@ func TestBuildHAKsWritesFolderCategoryInHeadVisualeffectsAutogenManifest(t *test
`"category_from": "immediate_parent"`, `"category_from": "immediate_parent"`,
} { } {
if !strings.Contains(text, want) { if !strings.Contains(text, want) {
t.Fatalf("expected head visualeffects manifest to contain %q, got:\n%s", want, text) t.Fatalf("expected accessory visualeffects manifest to contain %q, got:\n%s", want, text)
} }
} }
} }
@@ -2900,7 +2929,7 @@ func TestPlanHAKChunksPutsNewestAssetsInLastChunk(t *testing.T) {
t.Fatalf("scan: %v", err) t.Fatalf("scan: %v", err)
} }
assets, err := collectAssetResources(p, false, nil, nil, nil) assets, err := collectAssetResources(p, false, nil, nil)
if err != nil { if err != nil {
t.Fatalf("collect asset resources: %v", err) t.Fatalf("collect asset resources: %v", err)
} }
@@ -3057,7 +3086,7 @@ func TestCollectLFSAssetInfoUsesGitCommonDirForWorktree(t *testing.T) {
func runGitCommitTest(t *testing.T, dir, timestamp, message string) { func runGitCommitTest(t *testing.T, dir, timestamp, message string) {
t.Helper() t.Helper()
runGitTest(t, dir, "add", ".") runGitTest(t, dir, "add", ".")
cmd := exec.Command("git", "commit", "-m", message) cmd := exec.Command("git", "-c", "commit.gpgsign=false", "-c", "tag.gpgsign=false", "commit", "-m", message)
cmd.Dir = dir cmd.Dir = dir
cmd.Env = append(os.Environ(), cmd.Env = append(os.Environ(),
"GIT_AUTHOR_DATE="+timestamp, "GIT_AUTHOR_DATE="+timestamp,
@@ -3071,7 +3100,7 @@ func runGitCommitTest(t *testing.T, dir, timestamp, message string) {
func runGitTest(t *testing.T, dir string, args ...string) { func runGitTest(t *testing.T, dir string, args ...string) {
t.Helper() t.Helper()
cmd := exec.Command("git", args...) cmd := exec.Command("git", append([]string{"-c", "commit.gpgsign=false", "-c", "tag.gpgsign=false"}, args...)...)
if dir != "" { if dir != "" {
cmd.Dir = dir cmd.Dir = dir
} }
@@ -3647,55 +3676,39 @@ func TestBuildHAKsFailsForDuplicateResourcesInSameHAK(t *testing.T) {
} }
} }
func TestBuildHAKsConvertsMusicSourcesAndWritesCreditsArtifacts(t *testing.T) { // A module-only project (consumes prebuilt HAKs, no paths.assets) must build via
// the full `crucible module build` (pipeline.Build) without an assets tree. HAK
// collection used to crash on the unset assets dir
// (`lstat : no such file or directory` / `Rel: can't make "" relative`).
func TestBuildModuleOnlyProjectWithoutAssetsDir(t *testing.T) {
root := t.TempDir() root := t.TempDir()
mustMkdir(t, filepath.Join(root, "assets", "envi", "music", "westgate")) mustMkdir(t, filepath.Join(root, "src", "module"))
mustMkdir(t, filepath.Join(root, "build")) mustMkdir(t, filepath.Join(root, "build"))
mustWriteFile(t, filepath.Join(root, "nwn-tool.json"), `{ mustWriteFile(t, filepath.Join(root, "nwn-tool.yaml"), `
"module": { module:
"name": "Test Assets", name: Test Module
"resref": "testassets" resref: testmod
}, paths:
"paths": { source: src
"assets": "assets", build: build
"build": "build" `)
}, mustWriteFile(t, filepath.Join(root, "src", "module", "module.ifo.json"), `{
"music": { "file_type": "IFO ",
"prefixes": { "file_version": "V3.2",
"envi/music/westgate": "mus_wg_" "root": {
} "struct_type": 0,
}, "fields": [
"haks": [
{ {
"name": "envi", "label": "Mod_Name",
"priority": 1, "type": "CExoString",
"max_bytes": 1048576, "value": "Test Module"
"split": false,
"include": ["envi/**"]
} }
] ]
} }
}
`) `)
mustWriteFile(t, filepath.Join(root, "assets", "envi", "music", "westgate", "AleandAnecdotes.mp3"), "source-mp3")
mustWriteFile(t, filepath.Join(root, "assets", "envi", "music", "westgate", "CREDITS.md"), "# NWN Music Pack Credits\n\n| Artist | Title | Output File | Original File | Album | Date | License / Copyright | Notes |\n|---|---|---|---|---|---|---|---|\n| Darren Curtis | Ale and Anecdotes | `mus_wg_andnc.bmu` | `AleandAnecdotes.mp3` | Darren's Commercially Free Music | | Darren Curtis Music | |\n")
ffprobePath := filepath.Join(root, "ffprobe")
mustWriteFile(t, ffprobePath, "#!/bin/sh\nprintf '%s\\n' '{\"format\":{\"tags\":{\"artist\":\"Unknown\",\"title\":\"Broken Metadata\"}}}'\n")
if err := os.Chmod(ffprobePath, 0o755); err != nil {
t.Fatalf("chmod ffprobe: %v", err)
}
ffmpegPath := filepath.Join(root, "ffmpeg")
mustWriteFile(t, ffmpegPath, "#!/bin/sh\nfor last; do :; done\nprintf 'bmu-data' > \"$last\"\n")
if err := os.Chmod(ffmpegPath, 0o755); err != nil {
t.Fatalf("chmod ffmpeg: %v", err)
}
t.Setenv("SOW_FFMPEG", ffmpegPath)
t.Setenv("SOW_FFPROBE", ffprobePath)
p, err := project.Load(root) p, err := project.Load(root)
if err != nil { if err != nil {
t.Fatalf("load project: %v", err) t.Fatalf("load project: %v", err)
@@ -3706,352 +3719,18 @@ func TestBuildHAKsConvertsMusicSourcesAndWritesCreditsArtifacts(t *testing.T) {
if err := p.Scan(); err != nil { if err := p.Scan(); err != nil {
t.Fatalf("scan: %v", err) t.Fatalf("scan: %v", err)
} }
if p.AssetsDir() != "" {
t.Fatalf("test precondition: expected unset assets dir, got %q", p.AssetsDir())
}
result, err := BuildHAKs(p) result, err := Build(p)
if err != nil { if err != nil {
t.Fatalf("build haks: %v", err) t.Fatalf("full build of module-only project: %v", err)
} }
if len(result.CreditsArtifactPaths) == 0 { if _, err := os.Stat(result.ModulePath); err != nil {
t.Fatal("expected credits artifacts to be written") t.Fatalf("expected built .mod at %s: %v", result.ModulePath, err)
} }
if result.HAKAssets != 0 {
manifestRaw, err := os.ReadFile(filepath.Join(root, "build", "haks.json")) t.Fatalf("module-only project has no assets, expected 0 HAK assets, got %d", result.HAKAssets)
if err != nil {
t.Fatalf("read manifest: %v", err)
}
if !strings.Contains(string(manifestRaw), "envi/music/westgate/mus_wg_andnc.bmu") {
t.Fatalf("expected converted bmu in manifest, got %s", string(manifestRaw))
}
if strings.Contains(string(manifestRaw), "AleandAnecdotes.mp3") {
t.Fatalf("did not expect source mp3 in manifest, got %s", string(manifestRaw))
}
creditsRaw, err := os.ReadFile(filepath.Join(root, ".cache", "credits", "envi", "music", "westgate", "CREDITS.md"))
if err != nil {
t.Fatalf("read generated credits: %v", err)
}
creditsText := string(creditsRaw)
if !strings.Contains(creditsText, "Darren Curtis") || !strings.Contains(creditsText, "mus_wg_andnc.bmu") {
t.Fatalf("unexpected generated credits contents: %s", creditsText)
}
if strings.Contains(creditsText, "Broken Metadata") {
t.Fatalf("manual credits overlay should override bad metadata: %s", creditsText)
}
inventoryRaw, err := os.ReadFile(filepath.Join(root, ".cache", "credits", "credits.json"))
if err != nil {
t.Fatalf("read credits inventory: %v", err)
}
inventoryText := string(inventoryRaw)
if !strings.Contains(inventoryText, "assets/envi/music/westgate/CREDITS.md") {
t.Fatalf("expected authored credits source in inventory: %s", inventoryText)
}
if !strings.Contains(inventoryText, ".cache/credits/envi/music/westgate/CREDITS.md") {
t.Fatalf("expected generated credits source in inventory: %s", inventoryText)
}
if _, err := os.Stat(filepath.Join(root, ".cache", "music")); !os.IsNotExist(err) {
t.Fatalf("expected temporary music staging to be cleaned, got err=%v", err)
}
secondResult, err := BuildHAKs(p)
if err != nil {
t.Fatalf("build haks second pass: %v", err)
}
if secondResult.CreditsSummary.ChangedFiles != 0 {
t.Fatalf("expected second credits refresh to be unchanged, got %d changed files", secondResult.CreditsSummary.ChangedFiles)
}
if secondResult.CreditsSummary.TrackCount != 1 {
t.Fatalf("expected one tracked music conversion, got %d", secondResult.CreditsSummary.TrackCount)
}
}
func TestBuildHAKsUsesExplicitMusicDatasetOutsideLegacyRoot(t *testing.T) {
root := t.TempDir()
mustMkdir(t, filepath.Join(root, "assets", "audio", "westgate"))
mustMkdir(t, filepath.Join(root, "build"))
ffprobePath := filepath.Join(root, "tools", "ffprobe")
ffmpegPath := filepath.Join(root, "tools", "ffmpeg")
mustMkdir(t, filepath.Dir(ffprobePath))
mustWriteFile(t, ffprobePath, "#!/bin/sh\nprintf '%s\\n' '{\"format\":{\"tags\":{\"artist\":\"Dataset Artist\",\"title\":\"Dataset Title\"}}}'\n")
mustWriteFile(t, ffmpegPath, "#!/bin/sh\nfor last; do :; done\nprintf 'dataset-bmu' > \"$last\"\n")
if err := os.Chmod(ffprobePath, 0o755); err != nil {
t.Fatalf("chmod ffprobe: %v", err)
}
if err := os.Chmod(ffmpegPath, 0o755); err != nil {
t.Fatalf("chmod ffmpeg: %v", err)
}
mustWriteFile(t, filepath.Join(root, "nwn-tool.yaml"), `
module:
name: Test Assets
resref: testassets
paths:
assets: assets
build: build
music:
tools:
ffmpeg: tools/ffmpeg
ffprobe: tools/ffprobe
defaults:
credits_root: custom-credits
datasets:
westgate_audio:
source: audio/westgate
output: generated/music
prefix: wg_
haks:
- name: music
priority: 1
max_bytes: 1048576
split: false
include:
- generated/music/**
`)
mustWriteFile(t, filepath.Join(root, "assets", "audio", "westgate", "Theme Song.mp3"), "source-mp3")
p, err := project.Load(root)
if err != nil {
t.Fatalf("load project: %v", err)
}
if err := p.ValidateLayout(); err != nil {
t.Fatalf("validate layout: %v", err)
}
if err := p.Scan(); err != nil {
t.Fatalf("scan: %v", err)
}
result, err := BuildHAKs(p)
if err != nil {
t.Fatalf("build haks: %v", err)
}
if result.CreditsSummary.TrackCount != 1 {
t.Fatalf("expected one music track, got %#v", result.CreditsSummary)
}
manifestRaw, err := os.ReadFile(filepath.Join(root, "build", "haks.json"))
if err != nil {
t.Fatalf("read manifest: %v", err)
}
manifestText := string(manifestRaw)
if !strings.Contains(manifestText, "generated/music/wg_theme.bmu") {
t.Fatalf("expected dataset output resource in manifest, got:\n%s", manifestText)
}
if strings.Contains(manifestText, "audio/westgate/Theme Song.mp3") {
t.Fatalf("did not expect source mp3 in manifest, got:\n%s", manifestText)
}
creditsRaw, err := os.ReadFile(filepath.Join(root, "custom-credits", "audio", "westgate", "CREDITS.md"))
if err != nil {
t.Fatalf("read generated dataset credits: %v", err)
}
if !strings.Contains(string(creditsRaw), "Dataset Artist") || !strings.Contains(string(creditsRaw), "wg_theme.bmu") {
t.Fatalf("unexpected generated credits:\n%s", string(creditsRaw))
}
}
func TestBuildHAKsConvertsFLACMusicSourcesByDefault(t *testing.T) {
root := t.TempDir()
mustMkdir(t, filepath.Join(root, "assets", "audio", "westgate"))
mustMkdir(t, filepath.Join(root, "build"))
ffprobePath := filepath.Join(root, "tools", "ffprobe")
ffmpegPath := filepath.Join(root, "tools", "ffmpeg")
mustMkdir(t, filepath.Dir(ffprobePath))
mustWriteFile(t, ffprobePath, "#!/bin/sh\nprintf '%s\\n' '{\"format\":{\"tags\":{\"artist\":\"FLAC Artist\",\"title\":\"FLAC Title\"}}}'\n")
mustWriteFile(t, ffmpegPath, "#!/bin/sh\nfor last; do :; done\nprintf 'dataset-bmu' > \"$last\"\n")
if err := os.Chmod(ffprobePath, 0o755); err != nil {
t.Fatalf("chmod ffprobe: %v", err)
}
if err := os.Chmod(ffmpegPath, 0o755); err != nil {
t.Fatalf("chmod ffmpeg: %v", err)
}
mustWriteFile(t, filepath.Join(root, "nwn-tool.yaml"), `
module:
name: Test Assets
resref: testassets
paths:
assets: assets
build: build
music:
tools:
ffmpeg: tools/ffmpeg
ffprobe: tools/ffprobe
datasets:
westgate_audio:
source: audio/westgate
output: generated/music
prefix: wg_
haks:
- name: music
priority: 1
max_bytes: 1048576
split: false
include:
- generated/music/**
`)
mustWriteFile(t, filepath.Join(root, "assets", "audio", "westgate", "Theme Song.flac"), "source-flac")
p, err := project.Load(root)
if err != nil {
t.Fatalf("load project: %v", err)
}
if err := p.ValidateLayout(); err != nil {
t.Fatalf("validate layout: %v", err)
}
if err := p.Scan(); err != nil {
t.Fatalf("scan: %v", err)
}
result, err := BuildHAKs(p)
if err != nil {
t.Fatalf("build haks: %v", err)
}
if result.CreditsSummary.TrackCount != 1 {
t.Fatalf("expected one music track, got %#v", result.CreditsSummary)
}
manifestRaw, err := os.ReadFile(filepath.Join(root, "build", "haks.json"))
if err != nil {
t.Fatalf("read manifest: %v", err)
}
manifestText := string(manifestRaw)
if !strings.Contains(manifestText, "generated/music/wg_theme.bmu") {
t.Fatalf("expected generated FLAC music in manifest, got:\n%s", manifestText)
}
if strings.Contains(manifestText, "audio/westgate/Theme Song.flac") {
t.Fatalf("did not expect source flac in manifest, got:\n%s", manifestText)
}
}
func TestBuildHAKsUsesYAMLMusicConvertExtensionsForDiscovery(t *testing.T) {
root := t.TempDir()
mustMkdir(t, filepath.Join(root, "assets", "audio", "westgate"))
mustMkdir(t, filepath.Join(root, "build"))
ffprobePath := filepath.Join(root, "tools", "ffprobe")
ffmpegPath := filepath.Join(root, "tools", "ffmpeg")
mustMkdir(t, filepath.Dir(ffprobePath))
mustWriteFile(t, ffprobePath, "#!/bin/sh\nprintf '%s\\n' '{\"format\":{\"tags\":{\"artist\":\"AAC Artist\",\"title\":\"AAC Title\"}}}'\n")
mustWriteFile(t, ffmpegPath, "#!/bin/sh\nfor last; do :; done\nprintf 'dataset-bmu' > \"$last\"\n")
if err := os.Chmod(ffprobePath, 0o755); err != nil {
t.Fatalf("chmod ffprobe: %v", err)
}
if err := os.Chmod(ffmpegPath, 0o755); err != nil {
t.Fatalf("chmod ffmpeg: %v", err)
}
mustWriteFile(t, filepath.Join(root, "nwn-tool.yaml"), `
module:
name: Test Assets
resref: testassets
paths:
assets: assets
build: build
music:
defaults:
convert_extensions:
- .aac
tools:
ffmpeg: tools/ffmpeg
ffprobe: tools/ffprobe
datasets:
westgate_audio:
source: audio/westgate
output: generated/music
prefix: wg_
haks:
- name: music
priority: 1
max_bytes: 1048576
split: false
include:
- generated/music/**
`)
mustWriteFile(t, filepath.Join(root, "assets", "audio", "westgate", "Theme Song.aac"), "source-aac")
p, err := project.Load(root)
if err != nil {
t.Fatalf("load project: %v", err)
}
if err := p.ValidateLayout(); err != nil {
t.Fatalf("validate layout: %v", err)
}
if err := p.Scan(); err != nil {
t.Fatalf("scan: %v", err)
}
result, err := BuildHAKs(p)
if err != nil {
t.Fatalf("build haks: %v", err)
}
if result.CreditsSummary.TrackCount != 1 {
t.Fatalf("expected one music track discovered through YAML convert_extensions, got %#v", result.CreditsSummary)
}
}
func TestPlanHAKsDoesNotTranscodeMusicSources(t *testing.T) {
root := t.TempDir()
mustMkdir(t, filepath.Join(root, "assets", "audio", "westgate"))
mustMkdir(t, filepath.Join(root, "build"))
ffmpegPath := filepath.Join(root, "tools", "ffmpeg")
mustMkdir(t, filepath.Dir(ffmpegPath))
mustWriteFile(t, ffmpegPath, "#!/bin/sh\necho should-not-run >&2\nexit 42\n")
if err := os.Chmod(ffmpegPath, 0o755); err != nil {
t.Fatalf("chmod ffmpeg: %v", err)
}
mustWriteFile(t, filepath.Join(root, "nwn-tool.yaml"), `
module:
name: Test Assets
resref: testassets
paths:
assets: assets
build: build
music:
tools:
ffmpeg: tools/ffmpeg
ffprobe: tools/ffprobe
datasets:
westgate_audio:
source: audio/westgate
output: generated/music
prefix: wg_
haks:
- name: music
priority: 1
max_bytes: 1048576
split: false
include:
- generated/music/**
`)
mustWriteFile(t, filepath.Join(root, "assets", "audio", "westgate", "Theme Song.mp3"), "source-mp3")
p, err := project.Load(root)
if err != nil {
t.Fatalf("load project: %v", err)
}
if err := p.ValidateLayout(); err != nil {
t.Fatalf("validate layout: %v", err)
}
if err := p.Scan(); err != nil {
t.Fatalf("scan: %v", err)
}
result, err := PlanHAKs(p)
if err != nil {
t.Fatalf("plan haks: %v", err)
}
if result.CreditsSummary.TrackCount != 1 {
t.Fatalf("expected one planned music track, got %#v", result.CreditsSummary)
}
manifestRaw, err := os.ReadFile(filepath.Join(root, "build", "haks.json"))
if err != nil {
t.Fatalf("read manifest: %v", err)
}
if !strings.Contains(string(manifestRaw), "generated/music/wg_theme.bmu") {
t.Fatalf("expected planned generated music in manifest, got:\n%s", string(manifestRaw))
}
if _, err := os.Stat(filepath.Join(root, ".cache", "music")); !os.IsNotExist(err) {
t.Fatalf("plan-only should not stage music, stat err=%v", err)
} }
} }
+249
View File
@@ -0,0 +1,249 @@
package pipeline
import (
"bytes"
"crypto/sha256"
"encoding/hex"
"encoding/json"
"errors"
"os"
"path/filepath"
"strings"
"testing"
"git.westgate.pw/ShadowsOverWestgate/sow-tools/internal/buildinfo"
"git.westgate.pw/ShadowsOverWestgate/sow-tools/internal/erf"
"git.westgate.pw/ShadowsOverWestgate/sow-tools/internal/project"
)
const directProjectConfig = `{
"module": {
"name": "Test Module",
"resref": "testmod",
"hak_order": ["core_01"]
},
"paths": {
"source": "src",
"assets": "assets",
"build": "build"
}
}
`
func writeBlobAt(t *testing.T, root, sha, content string) {
t.Helper()
dir := filepath.Join(root, "sha256", sha[0:2], sha[2:4])
mustMkdir(t, dir)
mustWriteFile(t, filepath.Join(dir, sha), content)
}
func writeBlob(t *testing.T, root string, payload []byte) string {
t.Helper()
sum := sha256.Sum256(payload)
sha := hex.EncodeToString(sum[:])
writeBlobAt(t, root, sha, string(payload))
return sha
}
func writeDirectManifest(t *testing.T, root string, sources map[string]SourceAsset, assets []string) string {
t.Helper()
manifest := SourceBuildManifest{
Schema: 1,
BuilderID: buildinfo.String(),
ModuleHAKs: []string{"core_01"},
HAKs: []SourceManifestHAK{{
Name: "core_01",
Group: "core",
Priority: 1,
MaxBytes: 1 << 20,
Assets: assets,
}},
AssetSources: sources,
}
raw, err := json.Marshal(manifest)
if err != nil {
t.Fatalf("marshal manifest: %v", err)
}
path := filepath.Join(root, "build-source-manifest.json")
mustWriteFile(t, path, string(raw))
return path
}
func loadDirectProject(t *testing.T, root string) *project.Project {
t.Helper()
mustWriteFile(t, filepath.Join(root, "nwn-tool.json"), directProjectConfig)
mustMkdir(t, filepath.Join(root, "build"))
p, err := project.Load(root)
if err != nil {
t.Fatalf("load project: %v", err)
}
return p
}
func TestBuildHAKsFromContentAddressedManifestWithoutAssetsTree(t *testing.T) {
root := t.TempDir()
p := loadDirectProject(t, root)
blobs := filepath.Join(root, "blobs")
payload := []byte("tga-payload-bytes")
sha := writeBlob(t, blobs, payload)
manifestPath := writeDirectManifest(t, root,
map[string]SourceAsset{"core/a.tga": {SHA256: sha, SizeBytes: int64(len(payload))}},
[]string{"core/a.tga"})
result, err := BuildHAKsWithOptions(p, BuildHAKOptions{
SourceManifestPath: manifestPath,
ContentAddressedRoot: blobs,
})
if err != nil {
t.Fatalf("build direct haks: %v", err)
}
if result.HAKAssets != 1 {
t.Fatalf("hak assets = %d, want 1", result.HAKAssets)
}
if _, err := os.Stat(filepath.Join(root, "assets")); !errors.Is(err, os.ErrNotExist) {
t.Fatalf("direct build must not materialize an assets tree (stat err = %v)", err)
}
hakPath := filepath.Join(root, "build", "core_01.hak")
raw, err := os.ReadFile(hakPath)
if err != nil {
t.Fatalf("read hak: %v", err)
}
archive, err := erf.Read(bytes.NewReader(raw))
if err != nil {
t.Fatalf("decode hak: %v", err)
}
if len(archive.Resources) != 1 {
t.Fatalf("hak resources = %d, want 1", len(archive.Resources))
}
if got := string(archive.Resources[0].Data); got != string(payload) {
t.Fatalf("resource payload = %q, want %q", got, payload)
}
if archive.Resources[0].Name != "a" {
t.Fatalf("resref = %q, want a", archive.Resources[0].Name)
}
}
func TestBuildHAKsFromContentAddressedManifestRejectsMissingRoot(t *testing.T) {
root := t.TempDir()
p := loadDirectProject(t, root)
blobs := filepath.Join(root, "blobs")
payload := []byte("tga-payload-bytes")
sha := writeBlob(t, blobs, payload)
manifestPath := writeDirectManifest(t, root,
map[string]SourceAsset{"core/a.tga": {SHA256: sha, SizeBytes: int64(len(payload))}},
[]string{"core/a.tga"})
_, err := BuildHAKsWithOptions(p, BuildHAKOptions{SourceManifestPath: manifestPath})
if err == nil {
t.Fatal("expected failure when content-addressed root is missing")
}
}
func TestBuildHAKsFromContentAddressedManifestRejectsMissingBlob(t *testing.T) {
root := t.TempDir()
p := loadDirectProject(t, root)
blobs := filepath.Join(root, "blobs")
sha := strings.Repeat("b", 64)
manifestPath := writeDirectManifest(t, root,
map[string]SourceAsset{"core/a.tga": {SHA256: sha, SizeBytes: 4}},
[]string{"core/a.tga"})
_, err := BuildHAKsWithOptions(p, BuildHAKOptions{
SourceManifestPath: manifestPath,
ContentAddressedRoot: blobs,
})
if err == nil {
t.Fatal("expected failure for missing source blob")
}
}
func TestBuildHAKsFromContentAddressedManifestRejectsCorruptBlob(t *testing.T) {
root := t.TempDir()
p := loadDirectProject(t, root)
blobs := filepath.Join(root, "blobs")
declared := sha256.Sum256([]byte("abcd"))
sha := hex.EncodeToString(declared[:])
// Same length, different bytes: size check passes, content hash mismatches.
writeBlobAt(t, blobs, sha, "wxyz")
manifestPath := writeDirectManifest(t, root,
map[string]SourceAsset{"core/a.tga": {SHA256: sha, SizeBytes: 4}},
[]string{"core/a.tga"})
_, err := BuildHAKsWithOptions(p, BuildHAKOptions{
SourceManifestPath: manifestPath,
ContentAddressedRoot: blobs,
})
if err == nil {
t.Fatal("expected sha256 mismatch failure for corrupt blob")
}
if !strings.Contains(err.Error(), "sha256 mismatch") {
t.Fatalf("error = %v, want sha256 mismatch", err)
}
if _, statErr := os.Stat(filepath.Join(root, "build", "core_01.hak")); !errors.Is(statErr, os.ErrNotExist) {
t.Fatalf("corrupt build must not leave a completed hak (stat err = %v)", statErr)
}
}
func TestLegacySourceManifestStillBuildsFromAssetsTree(t *testing.T) {
root := t.TempDir()
mustMkdir(t, filepath.Join(root, "src", "module"))
mustMkdir(t, filepath.Join(root, "assets", "core"))
mustMkdir(t, filepath.Join(root, "build"))
mustWriteFile(t, filepath.Join(root, "nwn-tool.json"), `{
"module": {"name": "Test Module", "resref": "testmod", "hak_order": ["group:core"]},
"paths": {"source": "src", "assets": "assets", "build": "build"},
"haks": [{"name": "core", "priority": 1, "include": ["core/**"]}]
}
`)
mustWriteFile(t, filepath.Join(root, "src", "module", "module.ifo.json"), `{
"file_type": "IFO ",
"file_version": "V3.2",
"root": {"struct_type": 0, "fields": [
{"label": "Mod_Name", "type": "CExoString", "value": "Test Module"},
{"label": "Mod_HakList", "type": "List", "value": []}
]}
}
`)
mustWriteFile(t, filepath.Join(root, "assets", "core", "a.tga"), strings.Repeat("a", 16))
mustWriteFile(t, filepath.Join(root, "assets", "core", "b.tga"), strings.Repeat("b", 16))
manifestPath := filepath.Join(root, "legacy-source-manifest.json")
mustWriteFile(t, manifestPath, `{
"module_haks": ["core"],
"haks": [{"name": "core", "group": "core", "priority": 1, "assets": ["core/a.tga", "core/b.tga"]}]
}
`)
p, err := project.Load(root)
if err != nil {
t.Fatalf("load project: %v", err)
}
if err := p.Scan(); err != nil {
t.Fatalf("scan: %v", err)
}
result, err := BuildHAKsWithOptions(p, BuildHAKOptions{SourceManifestPath: manifestPath})
if err != nil {
t.Fatalf("legacy build: %v", err)
}
if len(result.HAKPaths) != 1 {
t.Fatalf("hak paths = %d, want 1", len(result.HAKPaths))
}
raw, err := os.ReadFile(filepath.Join(root, "build", "core.hak"))
if err != nil {
t.Fatalf("read legacy hak: %v", err)
}
archive, err := erf.Read(bytes.NewReader(raw))
if err != nil {
t.Fatalf("decode legacy hak: %v", err)
}
if len(archive.Resources) != 2 {
t.Fatalf("legacy hak resources = %d, want 2", len(archive.Resources))
}
}
+232
View File
@@ -0,0 +1,232 @@
package pipeline
import (
"encoding/json"
"fmt"
"os"
pathpkg "path"
"path/filepath"
"regexp"
"slices"
"strings"
"git.westgate.pw/ShadowsOverWestgate/sow-tools/internal/buildinfo"
"git.westgate.pw/ShadowsOverWestgate/sow-tools/internal/erf"
)
// SourceAsset is one content-addressed source blob referenced by a direct source
// manifest. The blob is resolved by sha256 under the content-addressed root.
type SourceAsset struct {
SHA256 string `json:"sha256"`
SizeBytes int64 `json:"size_bytes"`
}
// SourceManifestHAK describes one output HAK and the logical asset paths it packs.
type SourceManifestHAK struct {
Name string `json:"name"`
Group string `json:"group"`
Priority int `json:"priority"`
MaxBytes int64 `json:"max_bytes"`
Optional bool `json:"optional,omitempty"`
Assets []string `json:"assets"`
BuildKey string `json:"build_key"`
}
// SourceBuildManifest is the generated, content-addressed input to a HAK build.
// It carries asset_sources, so Crucible reads source bytes directly from the blob
// cache without scanning a materialized asset tree.
type SourceBuildManifest struct {
Schema int `json:"schema"`
BuilderID string `json:"builder_id"`
ModuleHAKs []string `json:"module_haks"`
HAKs []SourceManifestHAK `json:"haks"`
AssetSources map[string]SourceAsset `json:"asset_sources"`
}
var sha256Pattern = regexp.MustCompile(`^[0-9a-f]{64}$`)
// hasAssetSources reports whether raw JSON carries the direct content-addressed
// schema. A legacy BuildManifest has no asset_sources object.
func hasAssetSources(raw []byte) bool {
var header struct {
AssetSources map[string]json.RawMessage `json:"asset_sources"`
}
if err := json.Unmarshal(raw, &header); err != nil {
return false
}
return header.AssetSources != nil
}
func loadSourceBuildManifest(path string) (*SourceBuildManifest, error) {
raw, err := os.ReadFile(path)
if err != nil {
return nil, fmt.Errorf("read source manifest %s: %w", path, err)
}
var manifest SourceBuildManifest
if err := json.Unmarshal(raw, &manifest); err != nil {
return nil, fmt.Errorf("parse source manifest %s: %w", path, err)
}
return &manifest, nil
}
func validateDirectSourceManifest(manifest *SourceBuildManifest) error {
if manifest.Schema != 1 {
return fmt.Errorf("unsupported source manifest schema %d", manifest.Schema)
}
if manifest.BuilderID != buildinfo.String() {
return fmt.Errorf("source manifest builder_id %q does not match builder %q", manifest.BuilderID, buildinfo.String())
}
if len(manifest.ModuleHAKs) == 0 {
return fmt.Errorf("source manifest module_haks is empty")
}
referenced := make(map[string]struct{}, len(manifest.AssetSources))
for _, hak := range manifest.HAKs {
if strings.TrimSpace(hak.Name) == "" {
return fmt.Errorf("source manifest hak has empty name")
}
for _, rel := range hak.Assets {
if !validLogicalAssetPath(rel) {
return fmt.Errorf("source manifest hak %s has invalid asset path %q", hak.Name, rel)
}
if _, dup := referenced[rel]; dup {
return fmt.Errorf("source manifest asset %q referenced by more than one hak", rel)
}
referenced[rel] = struct{}{}
source, ok := manifest.AssetSources[rel]
if !ok {
return fmt.Errorf("source manifest hak %s asset %q has no asset_sources entry", hak.Name, rel)
}
if !sha256Pattern.MatchString(source.SHA256) {
return fmt.Errorf("source manifest asset %q has invalid sha256 %q", rel, source.SHA256)
}
if source.SizeBytes < 0 {
return fmt.Errorf("source manifest asset %q has negative size %d", rel, source.SizeBytes)
}
}
}
for rel := range manifest.AssetSources {
if !validLogicalAssetPath(rel) {
return fmt.Errorf("source manifest asset_sources has invalid path %q", rel)
}
if _, ok := referenced[rel]; !ok {
return fmt.Errorf("source manifest asset_sources entry %q is not referenced by any hak", rel)
}
}
return nil
}
func validLogicalAssetPath(path string) bool {
if path == "" || filepath.IsAbs(path) || strings.Contains(path, `\`) {
return false
}
clean := pathpkg.Clean(path)
return clean == path && clean != "." && clean != ".." &&
!strings.HasPrefix(clean, "../")
}
func contentAddressedBlobPath(root, sha string) (string, error) {
if strings.TrimSpace(root) == "" {
return "", fmt.Errorf("content-addressed root is required")
}
if !sha256Pattern.MatchString(sha) {
return "", fmt.Errorf("invalid content-addressed sha256 %q", sha)
}
return filepath.Join(root, "sha256", sha[0:2], sha[2:4], sha), nil
}
// loadDirectSourceManifest returns a parsed direct source manifest when path
// points at a content-addressed manifest. It returns (nil, nil) for an empty
// path or a legacy manifest without asset_sources, so the caller falls through
// to the legacy path-backed build.
func loadDirectSourceManifest(path string) (*SourceBuildManifest, error) {
if strings.TrimSpace(path) == "" {
return nil, nil
}
raw, err := os.ReadFile(path)
if err != nil {
return nil, fmt.Errorf("read source manifest %s: %w", path, err)
}
if !hasAssetSources(raw) {
return nil, nil
}
var manifest SourceBuildManifest
if err := json.Unmarshal(raw, &manifest); err != nil {
return nil, fmt.Errorf("parse source manifest %s: %w", path, err)
}
return &manifest, nil
}
// collectDirectSourceResources resolves each manifest-named blob under contentRoot
// and produces sorted asset resources that stream from the blob cache with their
// expected SHA-256. It never scans a materialized asset tree.
func collectDirectSourceResources(manifest *SourceBuildManifest, contentRoot string) ([]assetResource, error) {
if err := validateDirectSourceManifest(manifest); err != nil {
return nil, err
}
if strings.TrimSpace(contentRoot) == "" {
return nil, fmt.Errorf("content-addressed source manifest requires --content-addressed-root")
}
resources := make([]assetResource, 0, len(manifest.AssetSources))
for _, hak := range manifest.HAKs {
for _, logicalPath := range hak.Assets {
source := manifest.AssetSources[logicalPath]
extension := strings.TrimPrefix(strings.ToLower(filepath.Ext(logicalPath)), ".")
resourceType, ok := erf.HAKResourceTypeForExtension(extension)
if !ok {
return nil, fmt.Errorf("unsupported HAK resource extension %q for %s", filepath.Ext(logicalPath), logicalPath)
}
resref := strings.ToLower(strings.TrimSuffix(filepath.Base(logicalPath), filepath.Ext(logicalPath)))
blobPath, err := contentAddressedBlobPath(contentRoot, source.SHA256)
if err != nil {
return nil, err
}
info, err := os.Stat(blobPath)
if err != nil {
return nil, fmt.Errorf("missing source blob for %s: %w", logicalPath, err)
}
if info.Size() != source.SizeBytes {
return nil, fmt.Errorf("source blob size mismatch for %s: expected %d, got %d", logicalPath, source.SizeBytes, info.Size())
}
resource := erf.Resource{
Name: resref,
Type: resourceType,
SourcePath: blobPath,
Size: source.SizeBytes,
ExpectedSHA256: source.SHA256,
}
resources = append(resources, assetResource{
Rel: logicalPath,
Resource: resource,
Size: erf.ArchiveSize([]erf.Resource{resource}),
ContentID: "sha256:" + source.SHA256,
})
}
}
slices.SortFunc(resources, func(a, b assetResource) int {
return compareResourceKeys(a.Resource, b.Resource)
})
return resources, nil
}
// chunksFromSourceManifest converts direct manifest HAK entries into build chunks
// reusing the legacy manifest chunker.
func chunksFromSourceManifest(assets []assetResource, entries []SourceManifestHAK) ([]hakChunk, error) {
converted := make([]BuildManifestHAK, len(entries))
for index, entry := range entries {
converted[index] = BuildManifestHAK{
Name: entry.Name,
Group: entry.Group,
Priority: entry.Priority,
MaxBytes: entry.MaxBytes,
Optional: entry.Optional,
Assets: entry.Assets,
}
}
return chunksFromManifest(assets, converted)
}
+187
View File
@@ -0,0 +1,187 @@
package pipeline
import (
"encoding/json"
"path/filepath"
"testing"
"git.westgate.pw/ShadowsOverWestgate/sow-tools/internal/buildinfo"
"git.westgate.pw/ShadowsOverWestgate/sow-tools/internal/erf"
)
const validAsset = "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"
// directManifestFixture returns a minimal valid direct source manifest using the
// running builder identity, so validation does not reject it on builder mismatch.
func directManifestFixture() *SourceBuildManifest {
return &SourceBuildManifest{
Schema: 1,
BuilderID: buildinfo.String(),
ModuleHAKs: []string{"core_01"},
HAKs: []SourceManifestHAK{
{
Name: "core_01",
Group: "core",
Priority: 1,
MaxBytes: 1024,
Assets: []string{"core/a.tga"},
},
},
AssetSources: map[string]SourceAsset{
"core/a.tga": {SHA256: validAsset, SizeBytes: 4},
},
}
}
func writeManifestFixture(t *testing.T, manifest *SourceBuildManifest) string {
t.Helper()
raw, err := json.Marshal(manifest)
if err != nil {
t.Fatalf("marshal fixture: %v", err)
}
path := filepath.Join(t.TempDir(), "build-source-manifest.json")
mustWriteFile(t, path, string(raw))
return path
}
func TestLoadSourceBuildManifestDirect(t *testing.T) {
path := writeManifestFixture(t, directManifestFixture())
manifest, err := loadSourceBuildManifest(path)
if err != nil {
t.Fatalf("load: %v", err)
}
if manifest.Schema != 1 {
t.Fatalf("schema = %d, want 1", manifest.Schema)
}
if manifest.BuilderID != buildinfo.String() {
t.Fatalf("builder_id = %q, want %q", manifest.BuilderID, buildinfo.String())
}
if len(manifest.HAKs) != 1 || manifest.HAKs[0].Name != "core_01" {
t.Fatalf("unexpected haks: %#v", manifest.HAKs)
}
source, ok := manifest.AssetSources["core/a.tga"]
if !ok || source.SHA256 != validAsset || source.SizeBytes != 4 {
t.Fatalf("unexpected asset source: %#v", manifest.AssetSources)
}
if err := validateDirectSourceManifest(manifest); err != nil {
t.Fatalf("validate valid manifest: %v", err)
}
}
func TestValidateDirectSourceManifestRejectsUnsupportedSchema(t *testing.T) {
manifest := directManifestFixture()
manifest.Schema = 2
if err := validateDirectSourceManifest(manifest); err == nil {
t.Fatal("expected schema rejection")
}
}
func TestValidateDirectSourceManifestRejectsBadSHA(t *testing.T) {
manifest := directManifestFixture()
manifest.AssetSources["core/a.tga"] = SourceAsset{SHA256: "NOTHEX", SizeBytes: 4}
if err := validateDirectSourceManifest(manifest); err == nil {
t.Fatal("expected bad sha rejection")
}
}
func TestValidateDirectSourceManifestRejectsNegativeSize(t *testing.T) {
manifest := directManifestFixture()
manifest.AssetSources["core/a.tga"] = SourceAsset{SHA256: validAsset, SizeBytes: -1}
if err := validateDirectSourceManifest(manifest); err == nil {
t.Fatal("expected negative size rejection")
}
}
func TestValidateDirectSourceManifestRejectsMissingSource(t *testing.T) {
manifest := directManifestFixture()
delete(manifest.AssetSources, "core/a.tga")
if err := validateDirectSourceManifest(manifest); err == nil {
t.Fatal("expected missing source rejection")
}
}
func TestValidateDirectSourceManifestRejectsUnusedSource(t *testing.T) {
manifest := directManifestFixture()
manifest.AssetSources["core/orphan.tga"] = SourceAsset{SHA256: validAsset, SizeBytes: 4}
if err := validateDirectSourceManifest(manifest); err == nil {
t.Fatal("expected unused source rejection")
}
}
func TestValidateDirectSourceManifestRejectsDuplicateAssetAcrossHAKs(t *testing.T) {
manifest := directManifestFixture()
manifest.HAKs = append(manifest.HAKs, SourceManifestHAK{
Name: "core_02",
Group: "core",
Priority: 1,
MaxBytes: 1024,
Assets: []string{"core/a.tga"},
})
if err := validateDirectSourceManifest(manifest); err == nil {
t.Fatal("expected duplicate-asset rejection")
}
}
func TestValidateDirectSourceManifestRejectsTraversalPath(t *testing.T) {
manifest := directManifestFixture()
manifest.HAKs[0].Assets = []string{"../escape.tga"}
manifest.AssetSources = map[string]SourceAsset{
"../escape.tga": {SHA256: validAsset, SizeBytes: 4},
}
if err := validateDirectSourceManifest(manifest); err == nil {
t.Fatal("expected traversal rejection")
}
}
func TestValidateDirectSourceManifestRejectsBuilderMismatch(t *testing.T) {
manifest := directManifestFixture()
manifest.BuilderID = "crucible deadbeef"
if err := validateDirectSourceManifest(manifest); err == nil {
t.Fatal("expected builder mismatch rejection")
}
}
// Two distinct source paths can collapse to the same resref+type; the chunker
// must reject that rather than let one silently shadow the other in the ERF.
func TestChunksFromManifestRejectsResrefCollision(t *testing.T) {
tga, ok := erf.HAKResourceTypeForExtension("tga")
if !ok {
t.Fatal("tga is not a hak resource type")
}
mk := func(rel, resref string) assetResource {
r := erf.Resource{Name: resref, Type: tga}
return assetResource{Rel: rel, Resource: r, Size: erf.ArchiveSize([]erf.Resource{r})}
}
entry := BuildManifestHAK{Name: "core_01", Group: "core"}
collide := []assetResource{mk("creature/foo.tga", "foo"), mk("placeable/foo.tga", "foo")}
entry.Assets = []string{"creature/foo.tga", "placeable/foo.tga"}
if _, err := chunksFromManifest(collide, []BuildManifestHAK{entry}); err == nil {
t.Fatal("expected resref+type collision rejection")
}
distinct := []assetResource{mk("creature/foo.tga", "foo"), mk("placeable/bar.tga", "bar")}
entry.Assets = []string{"creature/foo.tga", "placeable/bar.tga"}
if _, err := chunksFromManifest(distinct, []BuildManifestHAK{entry}); err != nil {
t.Fatalf("distinct resrefs should pack cleanly: %v", err)
}
}
func TestContentAddressedBlobPath(t *testing.T) {
root := "/var/cache/blobs"
got, err := contentAddressedBlobPath(root, validAsset)
if err != nil {
t.Fatalf("blob path: %v", err)
}
want := filepath.Join(root, "sha256", "aa", "aa", validAsset)
if got != want {
t.Fatalf("blob path = %q, want %q", got, want)
}
if _, err := contentAddressedBlobPath(root, "NOTHEX"); err == nil {
t.Fatal("expected invalid sha rejection")
}
if _, err := contentAddressedBlobPath("", validAsset); err == nil {
t.Fatal("expected empty root rejection")
}
}
-130
View File
@@ -8,8 +8,6 @@ import (
"slices" "slices"
"strings" "strings"
"time" "time"
"git.westgate.pw/ShadowsOverWestgate/sow-tools/internal/music"
) )
const ( const (
@@ -21,11 +19,6 @@ const (
DefaultHAKArchiveTemplate = "{hak.name}.hak" DefaultHAKArchiveTemplate = "{hak.name}.hak"
DefaultSourceJSONPattern = "{resref}.{extension}.json" DefaultSourceJSONPattern = "{resref}.{extension}.json"
DefaultValidationProfile = "nwn_module" DefaultValidationProfile = "nwn_module"
DefaultMusicStageRoot = "{paths.cache}/music"
DefaultCreditsRoot = "{paths.cache}/credits"
DefaultCreditsOverlayFile = "CREDITS.md"
DefaultMusicNamingScheme = "nwn_bmu"
DefaultMusicOutputExtension = ".bmu"
DefaultTopDataBuild = "{paths.build}/topdata" DefaultTopDataBuild = "{paths.build}/topdata"
DefaultTopDataCompiled2DADir = "2da" DefaultTopDataCompiled2DADir = "2da"
DefaultTopDataCompiledTLK = "sow_tlk.tlk" DefaultTopDataCompiledTLK = "sow_tlk.tlk"
@@ -85,7 +78,6 @@ type EffectiveConfig struct {
Generated GeneratedConfig `json:"generated_assets" yaml:"generated_assets"` Generated GeneratedConfig `json:"generated_assets" yaml:"generated_assets"`
Inventory InventoryConfig `json:"inventory" yaml:"inventory"` Inventory InventoryConfig `json:"inventory" yaml:"inventory"`
Validation ValidationConfig `json:"validation" yaml:"validation"` Validation ValidationConfig `json:"validation" yaml:"validation"`
Music EffectiveMusicConfig `json:"music" yaml:"music"`
TopData EffectiveTopDataConfig `json:"topdata" yaml:"topdata"` TopData EffectiveTopDataConfig `json:"topdata" yaml:"topdata"`
Extract ExtractConfig `json:"extract" yaml:"extract"` Extract ExtractConfig `json:"extract" yaml:"extract"`
Autogen EffectiveAutogenConfig `json:"autogen" yaml:"autogen"` Autogen EffectiveAutogenConfig `json:"autogen" yaml:"autogen"`
@@ -108,32 +100,6 @@ type EffectiveOutputConfig struct {
HAKArchive string `json:"hak_archive" yaml:"hak_archive"` HAKArchive string `json:"hak_archive" yaml:"hak_archive"`
} }
type EffectiveMusicConfig struct {
Prefixes map[string]string `json:"prefixes" yaml:"prefixes"`
Tools MusicToolsConfig `json:"tools" yaml:"tools"`
StageRoot string `json:"stage_root" yaml:"stage_root"`
CreditsRoot string `json:"credits_root" yaml:"credits_root"`
CreditsOverlay string `json:"credits_overlay" yaml:"credits_overlay"`
MaxStemLength int `json:"max_stem_length" yaml:"max_stem_length"`
NamingScheme string `json:"naming_scheme" yaml:"naming_scheme"`
OutputExtension string `json:"output_extension" yaml:"output_extension"`
ConvertExtensions []string `json:"convert_extensions" yaml:"convert_extensions"`
Datasets map[string]EffectiveMusicDataset `json:"datasets,omitempty" yaml:"datasets,omitempty"`
}
type EffectiveMusicDataset struct {
Source string `json:"source" yaml:"source"`
Output string `json:"output" yaml:"output"`
Prefix string `json:"prefix,omitempty" yaml:"prefix,omitempty"`
StageRoot string `json:"stage_root,omitempty" yaml:"stage_root,omitempty"`
CreditsRoot string `json:"credits_root,omitempty" yaml:"credits_root,omitempty"`
NamingScheme string `json:"naming_scheme" yaml:"naming_scheme"`
OutputExtension string `json:"output_extension" yaml:"output_extension"`
MaxStemLength int `json:"max_stem_length" yaml:"max_stem_length"`
PackageMode string `json:"package_mode" yaml:"package_mode"`
ConvertExtensions []string `json:"convert_extensions" yaml:"convert_extensions"`
}
type EffectiveTopDataConfig struct { type EffectiveTopDataConfig struct {
Source string `json:"source" yaml:"source"` Source string `json:"source" yaml:"source"`
Build string `json:"build" yaml:"build"` Build string `json:"build" yaml:"build"`
@@ -275,81 +241,6 @@ func (p *Project) EffectiveConfig() EffectiveConfig {
extract.ConsumeArchives = &defaultConsume extract.ConsumeArchives = &defaultConsume
} }
musicStageRoot := expandPathTemplate(DefaultMusicStageRoot, paths)
musicCreditsRoot := expandPathTemplate(DefaultCreditsRoot, paths)
musicCreditsOverlay := DefaultCreditsOverlayFile
musicMaxStemLength := 16
musicNamingScheme := DefaultMusicNamingScheme
musicOutputExtension := DefaultMusicOutputExtension
musicConvertExtensions := music.DefaultConvertExtensions()
if d := p.Config.Music.Defaults; d != nil {
if d.StageRoot != "" {
musicStageRoot = expandPathTemplate(d.StageRoot, paths)
}
if d.CreditsRoot != "" {
musicCreditsRoot = expandPathTemplate(d.CreditsRoot, paths)
}
if d.CreditsOverlay != "" {
musicCreditsOverlay = d.CreditsOverlay
}
if d.MaxStemLength != nil {
musicMaxStemLength = *d.MaxStemLength
}
if d.NamingScheme != "" {
musicNamingScheme = d.NamingScheme
}
if d.OutputExtension != "" {
musicOutputExtension = normalizeExtension(d.OutputExtension)
}
if len(d.ConvertExtensions) > 0 {
musicConvertExtensions = normalizeExtensionSlice(d.ConvertExtensions)
}
}
musicPrefixes := cloneStringMap(p.Config.Music.Prefixes)
if len(musicPrefixes) == 0 {
musicPrefixes = make(map[string]string, len(p.Config.Music.Datasets))
for _, ds := range p.Config.Music.Datasets {
if ds.Source != "" && ds.Prefix != "" {
musicPrefixes[ds.Source] = ds.Prefix
}
}
}
effectiveDatasets := make(map[string]EffectiveMusicDataset, len(p.Config.Music.Datasets))
for id, ds := range p.Config.Music.Datasets {
dsStageRoot := ds.StageRoot
if dsStageRoot == "" {
dsStageRoot = musicStageRoot
}
dsCreditsRoot := ds.CreditsRoot
if dsCreditsRoot == "" {
dsCreditsRoot = musicCreditsRoot
}
dsOutput := strings.TrimSpace(ds.Output)
if dsOutput == "" {
dsOutput = ds.Source
}
dsNamingScheme := defaultString(ds.NamingScheme, musicNamingScheme)
dsOutputExtension := musicOutputExtension
if ds.OutputExtension != "" {
dsOutputExtension = normalizeExtension(ds.OutputExtension)
}
dsConvertExtensions := musicConvertExtensions
if len(ds.ConvertExtensions) > 0 {
dsConvertExtensions = normalizeExtensionSlice(ds.ConvertExtensions)
}
effectiveDatasets[id] = EffectiveMusicDataset{
Source: ds.Source,
Output: dsOutput,
Prefix: ds.Prefix,
StageRoot: dsStageRoot,
CreditsRoot: dsCreditsRoot,
NamingScheme: dsNamingScheme,
OutputExtension: dsOutputExtension,
MaxStemLength: musicMaxStemLength,
PackageMode: defaultString(ds.PackageMode, "hak_asset"),
ConvertExtensions: dsConvertExtensions,
}
}
effective := EffectiveConfig{ effective := EffectiveConfig{
ConfigSource: p.ConfigSource, ConfigSource: p.ConfigSource,
Module: p.Config.Module, Module: p.Config.Module,
@@ -359,18 +250,6 @@ func (p *Project) EffectiveConfig() EffectiveConfig {
Generated: generated, Generated: generated,
Inventory: inventory, Inventory: inventory,
Validation: validation, Validation: validation,
Music: EffectiveMusicConfig{
Prefixes: musicPrefixes,
Tools: p.Config.Music.Tools,
StageRoot: musicStageRoot,
CreditsRoot: musicCreditsRoot,
CreditsOverlay: musicCreditsOverlay,
MaxStemLength: musicMaxStemLength,
NamingScheme: musicNamingScheme,
OutputExtension: musicOutputExtension,
ConvertExtensions: musicConvertExtensions,
Datasets: effectiveDatasets,
},
TopData: top, TopData: top,
Extract: extract, Extract: extract,
Autogen: EffectiveAutogenConfig{ Autogen: EffectiveAutogenConfig{
@@ -517,13 +396,6 @@ func markMissingDefaults(provenance ConfigProvenance) {
"validation.profile": DefaultValidationProfile, "validation.profile": DefaultValidationProfile,
"validation.builtin_script_prefixes": "NWN built-in script prefixes", "validation.builtin_script_prefixes": "NWN built-in script prefixes",
"validation.required_fields": "NWN required GFF fields", "validation.required_fields": "NWN required GFF fields",
"music.stage_root": DefaultMusicStageRoot,
"music.credits_root": DefaultCreditsRoot,
"music.credits_overlay": DefaultCreditsOverlayFile,
"music.max_stem_length": "16",
"music.naming_scheme": DefaultMusicNamingScheme,
"music.output_extension": DefaultMusicOutputExtension,
"music.convert_extensions": strings.Join(music.DefaultConvertExtensions(), ", "),
"topdata.build": DefaultTopDataBuild, "topdata.build": DefaultTopDataBuild,
"topdata.compiled_2da_dir": DefaultTopDataCompiled2DADir, "topdata.compiled_2da_dir": DefaultTopDataCompiled2DADir,
"topdata.compiled_tlk": DefaultTopDataCompiledTLK, "topdata.compiled_tlk": DefaultTopDataCompiledTLK,
@@ -690,8 +562,6 @@ func activeOverrides(effective EffectiveConfig) []ConfigOverride {
"autogen.cache.parts_manifest_refresh": effective.Autogen.Cache.PartsManifestRefreshEnv, "autogen.cache.parts_manifest_refresh": effective.Autogen.Cache.PartsManifestRefreshEnv,
"autogen.release_source.server_url": effective.Autogen.ReleaseSource.ServerURLEnv, "autogen.release_source.server_url": effective.Autogen.ReleaseSource.ServerURLEnv,
"autogen.release_source.repo": effective.Autogen.ReleaseSource.RepoEnv, "autogen.release_source.repo": effective.Autogen.ReleaseSource.RepoEnv,
"music.ffmpeg": "SOW_FFMPEG",
"music.ffprobe": "SOW_FFPROBE",
"wiki.endpoint": "NODEBB_API_ENDPOINT", "wiki.endpoint": "NODEBB_API_ENDPOINT",
"wiki.token": "NODEBB_API_TOKEN", "wiki.token": "NODEBB_API_TOKEN",
"wiki.categories": "NODEBB_WIKI_CATEGORIES", "wiki.categories": "NODEBB_WIKI_CATEGORIES",
+65 -207
View File
@@ -31,7 +31,7 @@ var SourceExtensions = []string{
} }
var AssetExtensions = []string{ var AssetExtensions = []string{
".2da", ".bik", ".bmp", ".bmu", ".dds", ".dwk", ".gr2", ".itp", ".jpg", ".lod", ".lyt", ".mdb", ".mdl", ".mdx", ".mp3", ".mtr", ".ogg", ".plt", ".png", ".pwk", ".set", ".shd", ".tga", ".txi", ".uti", ".vis", ".wav", ".wlk", ".wok", ".xml", ".2da", ".bik", ".bmp", ".bmu", ".dds", ".dwk", ".gr2", ".itp", ".jpg", ".lod", ".lyt", ".mdb", ".mdl", ".mdx", ".mtr", ".plt", ".png", ".pwk", ".set", ".shd", ".tga", ".txi", ".uti", ".vis", ".wav", ".wlk", ".wok", ".xml",
} }
var BuiltinScriptPrefixes = []string{ var BuiltinScriptPrefixes = []string{
@@ -88,7 +88,6 @@ type Config struct {
HAKs []HAKConfig `json:"haks" yaml:"haks"` HAKs []HAKConfig `json:"haks" yaml:"haks"`
Inventory InventoryConfig `json:"inventory" yaml:"inventory"` Inventory InventoryConfig `json:"inventory" yaml:"inventory"`
Validation ValidationConfig `json:"validation" yaml:"validation"` Validation ValidationConfig `json:"validation" yaml:"validation"`
Music MusicConfig `json:"music" yaml:"music"`
TopData TopDataConfig `json:"topdata" yaml:"topdata"` TopData TopDataConfig `json:"topdata" yaml:"topdata"`
Extract ExtractConfig `json:"extract" yaml:"extract"` Extract ExtractConfig `json:"extract" yaml:"extract"`
Autogen AutogenConfig `json:"autogen" yaml:"autogen"` Autogen AutogenConfig `json:"autogen" yaml:"autogen"`
@@ -153,40 +152,6 @@ type HAKConfig struct {
Include []string `json:"include" yaml:"include"` Include []string `json:"include" yaml:"include"`
} }
type MusicConfig struct {
Prefixes map[string]string `json:"prefixes,omitempty" yaml:"prefixes,omitempty"`
Tools MusicToolsConfig `json:"tools,omitempty" yaml:"tools,omitempty"`
Defaults *MusicDefaultsConfig `json:"defaults,omitempty" yaml:"defaults,omitempty"`
Datasets map[string]MusicDatasetConfig `json:"datasets,omitempty" yaml:"datasets,omitempty"`
}
type MusicToolsConfig struct {
FFmpeg string `json:"ffmpeg,omitempty" yaml:"ffmpeg,omitempty"`
FFprobe string `json:"ffprobe,omitempty" yaml:"ffprobe,omitempty"`
}
type MusicDefaultsConfig struct {
StageRoot string `json:"stage_root,omitempty" yaml:"stage_root,omitempty"`
CreditsRoot string `json:"credits_root,omitempty" yaml:"credits_root,omitempty"`
CreditsOverlay string `json:"credits_overlay,omitempty" yaml:"credits_overlay,omitempty"`
MaxStemLength *int `json:"max_stem_length,omitempty" yaml:"max_stem_length,omitempty"`
NamingScheme string `json:"naming_scheme,omitempty" yaml:"naming_scheme,omitempty"`
OutputExtension string `json:"output_extension,omitempty" yaml:"output_extension,omitempty"`
ConvertExtensions []string `json:"convert_extensions,omitempty" yaml:"convert_extensions,omitempty"`
}
type MusicDatasetConfig struct {
Source string `json:"source" yaml:"source"`
Output string `json:"output,omitempty" yaml:"output,omitempty"`
Prefix string `json:"prefix,omitempty" yaml:"prefix,omitempty"`
StageRoot string `json:"stage_root,omitempty" yaml:"stage_root,omitempty"`
CreditsRoot string `json:"credits_root,omitempty" yaml:"credits_root,omitempty"`
NamingScheme string `json:"naming_scheme,omitempty" yaml:"naming_scheme,omitempty"`
OutputExtension string `json:"output_extension,omitempty" yaml:"output_extension,omitempty"`
PackageMode string `json:"package_mode,omitempty" yaml:"package_mode,omitempty"`
ConvertExtensions []string `json:"convert_extensions,omitempty" yaml:"convert_extensions,omitempty"`
}
type TopDataConfig struct { type TopDataConfig struct {
Source string `json:"source" yaml:"source"` Source string `json:"source" yaml:"source"`
Build string `json:"build" yaml:"build"` Build string `json:"build" yaml:"build"`
@@ -349,7 +314,7 @@ type AutogenProducerConfig struct {
Include []string `json:"include" yaml:"include"` Include []string `json:"include" yaml:"include"`
Derive AutogenDeriveConfig `json:"derive" yaml:"derive"` Derive AutogenDeriveConfig `json:"derive" yaml:"derive"`
Manifest AutogenManifestConfig `json:"manifest" yaml:"manifest"` Manifest AutogenManifestConfig `json:"manifest" yaml:"manifest"`
HeadVisualeffects HeadVisualeffectsConfig `json:"accessory_visualeffects" yaml:"accessory_visualeffects"` AccessoryVisualeffects AccessoryVisualeffectsConfig `json:"accessory_visualeffects" yaml:"accessory_visualeffects"`
} }
type AutogenConsumerConfig struct { type AutogenConsumerConfig struct {
@@ -362,13 +327,15 @@ type AutogenConsumerConfig struct {
Include []string `json:"include" yaml:"include"` Include []string `json:"include" yaml:"include"`
Derive AutogenDeriveConfig `json:"derive" yaml:"derive"` Derive AutogenDeriveConfig `json:"derive" yaml:"derive"`
PartsRows PartsRowsConfig `json:"parts_rows" yaml:"parts_rows"` PartsRows PartsRowsConfig `json:"parts_rows" yaml:"parts_rows"`
HeadVisualeffects HeadVisualeffectsConfig `json:"accessory_visualeffects" yaml:"accessory_visualeffects"` AccessoryVisualeffects AccessoryVisualeffectsConfig `json:"accessory_visualeffects" yaml:"accessory_visualeffects"`
Manifest AutogenManifestConfig `json:"manifest" yaml:"manifest"` Manifest AutogenManifestConfig `json:"manifest" yaml:"manifest"`
LocalOverrideRoot string `json:"local_override_root,omitempty" yaml:"local_override_root,omitempty"` LocalOverrideRoot string `json:"local_override_root,omitempty" yaml:"local_override_root,omitempty"`
ManifestFile string `json:"manifest_file,omitempty" yaml:"manifest_file,omitempty"`
Source AutogenSourceConfig `json:"source,omitempty" yaml:"source,omitempty"`
} }
type HeadVisualeffectsConfig struct { type AccessoryVisualeffectsConfig struct {
Groups map[string]HeadVisualeffectGroupConfig `json:"groups,omitempty" yaml:"groups"` Groups map[string]AccessoryVisualeffectGroupConfig `json:"groups,omitempty" yaml:"groups"`
GroupTokenSource string `json:"group_token_source,omitempty" yaml:"group_token_source"` GroupTokenSource string `json:"group_token_source,omitempty" yaml:"group_token_source"`
CategoryFrom string `json:"category_from,omitempty" yaml:"category_from"` CategoryFrom string `json:"category_from,omitempty" yaml:"category_from"`
Delimiter string `json:"delimiter,omitempty" yaml:"delimiter"` Delimiter string `json:"delimiter,omitempty" yaml:"delimiter"`
@@ -380,7 +347,7 @@ type HeadVisualeffectsConfig struct {
RowDefaults map[string]string `json:"row_defaults,omitempty" yaml:"row_defaults"` RowDefaults map[string]string `json:"row_defaults,omitempty" yaml:"row_defaults"`
} }
type HeadVisualeffectGroupConfig struct { type AccessoryVisualeffectGroupConfig struct {
Prefix string `json:"prefix" yaml:"prefix"` Prefix string `json:"prefix" yaml:"prefix"`
ModelColumn string `json:"model_column,omitempty" yaml:"model_column"` ModelColumn string `json:"model_column,omitempty" yaml:"model_column"`
ModelColumns []string `json:"model_columns,omitempty" yaml:"model_columns"` ModelColumns []string `json:"model_columns,omitempty" yaml:"model_columns"`
@@ -415,6 +382,17 @@ type AutogenDeriveConfig struct {
GroupFrom string `json:"group_from,omitempty" yaml:"group_from,omitempty"` GroupFrom string `json:"group_from,omitempty" yaml:"group_from,omitempty"`
} }
type AutogenSourceConfig struct {
Kind string `json:"kind,omitempty" yaml:"kind,omitempty"`
CDNBase string `json:"cdn_base,omitempty" yaml:"cdn_base,omitempty"`
CDNBaseEnv string `json:"cdn_base_env,omitempty" yaml:"cdn_base_env,omitempty"`
ChannelsPath string `json:"channels_path,omitempty" yaml:"channels_path,omitempty"`
ManifestPath string `json:"manifest_path,omitempty" yaml:"manifest_path,omitempty"`
ReleaseMarkerPath string `json:"release_marker_path,omitempty" yaml:"release_marker_path,omitempty"`
ChannelEnv string `json:"channel_env,omitempty" yaml:"channel_env,omitempty"`
OfflineOverrideEnv string `json:"offline_override_env,omitempty" yaml:"offline_override_env,omitempty"`
}
type AutogenManifestConfig struct { type AutogenManifestConfig struct {
ReleaseTag string `json:"release_tag" yaml:"release_tag"` ReleaseTag string `json:"release_tag" yaml:"release_tag"`
AssetName string `json:"asset_name" yaml:"asset_name"` AssetName string `json:"asset_name" yaml:"asset_name"`
@@ -640,6 +618,7 @@ func (p *Project) ValidateLayout() error {
failures = append(failures, validateTopDataRowGeneration(effective.TopData.RowGeneration)...) failures = append(failures, validateTopDataRowGeneration(effective.TopData.RowGeneration)...)
failures = append(failures, validateTopDataRowExtensions(effective.TopData.RowExtensions)...) failures = append(failures, validateTopDataRowExtensions(effective.TopData.RowExtensions)...)
failures = append(failures, validateTopDataClassFeatInjections(effective.TopData.ClassFeatInjections)...) failures = append(failures, validateTopDataClassFeatInjections(effective.TopData.ClassFeatInjections)...)
failures = append(failures, validateAutogenConsumerSources(effective.Autogen.Consumers)...)
failures = append(failures, validateRelativePath("topdata.compiled_2da_dir", effective.TopData.Compiled2DADir)...) failures = append(failures, validateRelativePath("topdata.compiled_2da_dir", effective.TopData.Compiled2DADir)...)
failures = append(failures, validateRelativePath("topdata.compiled_tlk", effective.TopData.CompiledTLK)...) failures = append(failures, validateRelativePath("topdata.compiled_tlk", effective.TopData.CompiledTLK)...)
failures = append(failures, validateRelativePath("topdata.wiki.output_root", effective.TopData.Wiki.OutputRoot)...) failures = append(failures, validateRelativePath("topdata.wiki.output_root", effective.TopData.Wiki.OutputRoot)...)
@@ -736,8 +715,6 @@ func (p *Project) ValidateLayout() error {
} }
} }
failures = append(failures, validateAutogenConfig(p.Config.Autogen)...) failures = append(failures, validateAutogenConfig(p.Config.Autogen)...)
failures = append(failures, validateMusicConfig(p.Config.Music)...)
hakNames := map[string]struct{}{} hakNames := map[string]struct{}{}
for index, hak := range p.Config.HAKs { for index, hak := range p.Config.HAKs {
name := strings.TrimSpace(hak.Name) name := strings.TrimSpace(hak.Name)
@@ -774,6 +751,13 @@ func (p *Project) ValidateLayout() error {
} }
info, err := os.Stat(path) info, err := os.Stat(path)
if err != nil { if err != nil {
// An output dir that does not exist yet is fine: the builder creates
// it (MkdirAll) before writing. A bare clone must validate/build from
// a clean tree with no pre-step (R2/parity) — only a path that exists
// but is NOT a directory is a real error.
if errors.Is(err, fs.ErrNotExist) {
continue
}
failures = append(failures, fmt.Errorf("%s: %w", label, err)) failures = append(failures, fmt.Errorf("%s: %w", label, err))
continue continue
} }
@@ -1038,46 +1022,9 @@ func (p *Project) Scan() error {
assetDir := p.AssetsDir() assetDir := p.AssetsDir()
var assetFiles []string var assetFiles []string
if assetDir != "" && filepath.Clean(assetDir) != filepath.Clean(p.Root) { if assetDir != "" && filepath.Clean(assetDir) != filepath.Clean(p.Root) {
musicSourceExts := make(map[string]struct{})
for _, ext := range effective.Music.ConvertExtensions {
musicSourceExts[ext] = struct{}{}
}
musicSourceRoots := make(map[string]map[string]struct{})
for _, ds := range effective.Music.Datasets {
if strings.TrimSpace(ds.Source) == "" {
continue
}
exts := musicSourceRoots[ds.Source]
if exts == nil {
exts = make(map[string]struct{})
musicSourceRoots[ds.Source] = exts
}
for _, ext := range ds.ConvertExtensions {
exts[ext] = struct{}{}
}
for ext := range musicSourceExts {
exts[ext] = struct{}{}
}
}
var err error var err error
assetFiles, _, err = scanDir(assetDir, func(path string) bool { assetFiles, _, err = scanDir(assetDir, func(path string) bool {
rel, err := filepath.Rel(assetDir, path) return slices.Contains(effective.Inventory.AssetExtensions, strings.ToLower(filepath.Ext(path)))
if err != nil {
return false
}
rel = filepath.ToSlash(rel)
ext := strings.ToLower(filepath.Ext(rel))
if slices.Contains(effective.Inventory.AssetExtensions, ext) {
return true
}
for root, exts := range musicSourceRoots {
if !pathIsUnder(root, rel) {
continue
}
_, ok := exts[ext]
return ok
}
return false
}) })
if err != nil { if err != nil {
if errors.Is(err, os.ErrNotExist) { if errors.Is(err, os.ErrNotExist) {
@@ -1322,33 +1269,6 @@ func (p *Project) TopDataPackageTLKPath() string {
return filepath.Join(p.BuildDir(), p.TopDataPackageTLKName()) return filepath.Join(p.BuildDir(), p.TopDataPackageTLKName())
} }
func (p *Project) MusicStageDir() string {
return p.rootPath(p.EffectiveConfig().Music.StageRoot)
}
func (p *Project) CreditsDir() string {
return p.rootPath(p.EffectiveConfig().Music.CreditsRoot)
}
func (p *Project) MusicToolPath(path string) string {
path = strings.TrimSpace(path)
if path == "" {
return ""
}
if filepath.IsAbs(path) {
return path
}
return filepath.Join(p.Root, filepath.FromSlash(path))
}
func (p *Project) MusicFFmpegPath() string {
return p.MusicToolPath(p.EffectiveConfig().Music.Tools.FFmpeg)
}
func (p *Project) MusicFFprobePath() string {
return p.MusicToolPath(p.EffectiveConfig().Music.Tools.FFprobe)
}
func (p *Project) AutogenCacheDir() string { func (p *Project) AutogenCacheDir() string {
return p.rootPath(p.EffectiveConfig().Autogen.Cache.Root) return p.rootPath(p.EffectiveConfig().Autogen.Cache.Root)
} }
@@ -1438,36 +1358,6 @@ func normalizeConfig(cfg *Config) {
for i := range cfg.Autogen.Consumers { for i := range cfg.Autogen.Consumers {
cfg.Autogen.Consumers[i].Include = normalizeStringSlice(cfg.Autogen.Consumers[i].Include) cfg.Autogen.Consumers[i].Include = normalizeStringSlice(cfg.Autogen.Consumers[i].Include)
} }
normalizeMusicConfig(&cfg.Music)
}
func normalizeMusicConfig(music *MusicConfig) {
if music.Defaults != nil {
music.Defaults.ConvertExtensions = normalizeExtensionSlice(music.Defaults.ConvertExtensions)
music.Defaults.OutputExtension = normalizeExtension(music.Defaults.OutputExtension)
}
if len(music.Prefixes) > 0 && len(music.Datasets) == 0 {
datasets := make(map[string]MusicDatasetConfig, len(music.Prefixes))
for dir, prefix := range music.Prefixes {
source := trimConfigSlashes(filepath.ToSlash(strings.TrimSpace(dir)))
if source == "" {
continue
}
key := strings.ReplaceAll(source, "/", "_")
datasets[key] = MusicDatasetConfig{
Source: source,
Prefix: strings.TrimSpace(prefix),
}
}
music.Datasets = datasets
}
for id, ds := range music.Datasets {
ds.Source = trimConfigSlashes(filepath.ToSlash(strings.TrimSpace(ds.Source)))
ds.Output = trimConfigSlashes(filepath.ToSlash(strings.TrimSpace(ds.Output)))
ds.OutputExtension = normalizeExtension(ds.OutputExtension)
ds.ConvertExtensions = normalizeExtensionSlice(ds.ConvertExtensions)
music.Datasets[id] = ds
}
} }
func normalizeStringSlice(input []string) []string { func normalizeStringSlice(input []string) []string {
@@ -1625,8 +1515,8 @@ func validateAutogenConfig(cfg AutogenConfig) []error {
failures = append(failures, validateGlobList(fieldPrefix+".include", producer.Include)...) failures = append(failures, validateGlobList(fieldPrefix+".include", producer.Include)...)
failures = append(failures, validateAutogenDeriveConfig(fieldPrefix+".derive", producer.Derive)...) failures = append(failures, validateAutogenDeriveConfig(fieldPrefix+".derive", producer.Derive)...)
failures = append(failures, validateAutogenManifestConfig(fieldPrefix+".manifest", producer.Manifest)...) failures = append(failures, validateAutogenManifestConfig(fieldPrefix+".manifest", producer.Manifest)...)
if headVisualeffectsConfigConfigured(producer.HeadVisualeffects) { if accessoryVisualeffectsConfigConfigured(producer.AccessoryVisualeffects) {
failures = append(failures, validateHeadVisualeffectsConfig(fieldPrefix+".accessory_visualeffects", producer.HeadVisualeffects)...) failures = append(failures, validateAccessoryVisualeffectsConfig(fieldPrefix+".accessory_visualeffects", producer.AccessoryVisualeffects)...)
} }
} }
@@ -1665,6 +1555,15 @@ func validateAutogenConfig(cfg AutogenConfig) []error {
default: default:
failures = append(failures, fmt.Errorf("%s.mode %q is not supported", fieldPrefix, consumer.Mode)) failures = append(failures, fmt.Errorf("%s.mode %q is not supported", fieldPrefix, consumer.Mode))
} }
if strings.TrimSpace(consumer.Mode) == "accessory_visualeffects" && accessoryVisualeffectsConfigConfigured(consumer.AccessoryVisualeffects) {
failures = append(failures, validateAccessoryVisualeffectsConfig(fieldPrefix+".accessory_visualeffects", consumer.AccessoryVisualeffects)...)
}
// A manifest_file consumer reads a pre-resolved local manifest, and a
// cdn_channel consumer resolves entirely from Source.* at runtime, so
// neither reads the released-source fields (root/include/derive/manifest)
// at build time — don't require them. See resolveAutogenConsumerManifest.
if strings.TrimSpace(consumer.ManifestFile) == "" &&
strings.TrimSpace(consumer.Source.Kind) != "cdn_channel" {
if strings.TrimSpace(consumer.Root) == "" { if strings.TrimSpace(consumer.Root) == "" {
failures = append(failures, fmt.Errorf("%s.root is required", fieldPrefix)) failures = append(failures, fmt.Errorf("%s.root is required", fieldPrefix))
} }
@@ -1673,15 +1572,36 @@ func validateAutogenConfig(cfg AutogenConfig) []error {
} }
failures = append(failures, validateGlobList(fieldPrefix+".include", consumer.Include)...) failures = append(failures, validateGlobList(fieldPrefix+".include", consumer.Include)...)
failures = append(failures, validateAutogenDeriveConfig(fieldPrefix+".derive", consumer.Derive)...) failures = append(failures, validateAutogenDeriveConfig(fieldPrefix+".derive", consumer.Derive)...)
if strings.TrimSpace(consumer.Mode) == "accessory_visualeffects" && headVisualeffectsConfigConfigured(consumer.HeadVisualeffects) {
failures = append(failures, validateHeadVisualeffectsConfig(fieldPrefix+".accessory_visualeffects", consumer.HeadVisualeffects)...)
}
failures = append(failures, validateAutogenManifestConfig(fieldPrefix+".manifest", consumer.Manifest)...) failures = append(failures, validateAutogenManifestConfig(fieldPrefix+".manifest", consumer.Manifest)...)
} }
}
return failures return failures
} }
func validateAutogenConsumerSources(consumers []AutogenConsumerConfig) []error {
var failures []error
for _, c := range consumers {
if strings.TrimSpace(c.Source.Kind) != "cdn_channel" {
continue
}
label := strings.TrimSpace(c.ID)
if label == "" {
label = "<unnamed>"
}
if strings.TrimSpace(c.Source.ChannelsPath) == "" {
failures = append(failures, fmt.Errorf("autogen consumer %s: source.channels_path is required for kind cdn_channel", label))
}
manifestPath := strings.TrimSpace(c.Source.ManifestPath)
if manifestPath == "" {
failures = append(failures, fmt.Errorf("autogen consumer %s: source.manifest_path is required for kind cdn_channel", label))
} else if !strings.Contains(manifestPath, "{tag}") {
failures = append(failures, fmt.Errorf("autogen consumer %s: source.manifest_path must contain {tag}", label))
}
}
return failures
}
func validateGeneratedConfig(cfg GeneratedConfig) []error { func validateGeneratedConfig(cfg GeneratedConfig) []error {
var failures []error var failures []error
seen := map[string]struct{}{} seen := map[string]struct{}{}
@@ -1736,7 +1656,7 @@ func validateGeneratedConfig(cfg GeneratedConfig) []error {
return failures return failures
} }
func headVisualeffectsConfigConfigured(cfg HeadVisualeffectsConfig) bool { func accessoryVisualeffectsConfigConfigured(cfg AccessoryVisualeffectsConfig) bool {
return len(cfg.Groups) > 0 || return len(cfg.Groups) > 0 ||
strings.TrimSpace(cfg.GroupTokenSource) != "" || strings.TrimSpace(cfg.GroupTokenSource) != "" ||
strings.TrimSpace(cfg.CategoryFrom) != "" || strings.TrimSpace(cfg.CategoryFrom) != "" ||
@@ -1749,7 +1669,7 @@ func headVisualeffectsConfigConfigured(cfg HeadVisualeffectsConfig) bool {
len(cfg.RowDefaults) > 0 len(cfg.RowDefaults) > 0
} }
func validateHeadVisualeffectsConfig(fieldPrefix string, cfg HeadVisualeffectsConfig) []error { func validateAccessoryVisualeffectsConfig(fieldPrefix string, cfg AccessoryVisualeffectsConfig) []error {
var failures []error var failures []error
groupTokenSource := strings.TrimSpace(cfg.GroupTokenSource) groupTokenSource := strings.TrimSpace(cfg.GroupTokenSource)
switch groupTokenSource { switch groupTokenSource {
@@ -1931,68 +1851,6 @@ func validatePartsRowsFormat(format string) error {
return nil return nil
} }
func validateMusicConfig(cfg MusicConfig) []error {
var failures []error
if cfg.Defaults != nil {
if cfg.Defaults.MaxStemLength != nil && *cfg.Defaults.MaxStemLength < 3 {
failures = append(failures, errors.New("music.defaults.max_stem_length must be at least 3"))
}
if cfg.Defaults.NamingScheme != "" && cfg.Defaults.NamingScheme != "nwn_bmu" && cfg.Defaults.NamingScheme != "passthrough" {
failures = append(failures, fmt.Errorf("music.defaults.naming_scheme %q is not supported", cfg.Defaults.NamingScheme))
}
if cfg.Defaults.OutputExtension != "" && !strings.HasPrefix(cfg.Defaults.OutputExtension, ".") {
failures = append(failures, errors.New("music.defaults.output_extension must start with ."))
}
}
normalizedRoots := map[string]string{}
for root, prefix := range cfg.Prefixes {
normalizedRoot := trimConfigSlashes(filepath.ToSlash(strings.TrimSpace(root)))
if normalizedRoot == "" {
failures = append(failures, errors.New("music.prefixes contains an empty root"))
continue
}
if prior, exists := normalizedRoots[normalizedRoot]; exists {
failures = append(failures, fmt.Errorf("music.prefixes roots %q and %q normalize to the same path", prior, root))
}
normalizedRoots[normalizedRoot] = root
if strings.TrimSpace(prefix) == "" {
failures = append(failures, fmt.Errorf("music.prefixes[%q] must not be empty", root))
}
}
seenSources := map[string]string{}
for id, ds := range cfg.Datasets {
if strings.TrimSpace(id) == "" {
failures = append(failures, errors.New("music.datasets contains an empty key"))
} else if strings.Contains(id, "/") || strings.Contains(id, "\\") {
failures = append(failures, fmt.Errorf("music.datasets id %q must not contain path separators", id))
}
source := trimConfigSlashes(filepath.ToSlash(strings.TrimSpace(ds.Source)))
if source == "" {
failures = append(failures, fmt.Errorf("music.datasets[%q].source is required", id))
} else if pathEscapesRoot(source) {
failures = append(failures, fmt.Errorf("music.datasets[%q].source must not escape project root", id))
} else if prior, exists := seenSources[source]; exists {
failures = append(failures, fmt.Errorf("music.datasets[%q].source %q conflicts with dataset %q", id, ds.Source, prior))
} else {
seenSources[source] = id
}
output := trimConfigSlashes(filepath.ToSlash(strings.TrimSpace(ds.Output)))
if output != "" && pathEscapesRoot(output) {
failures = append(failures, fmt.Errorf("music.datasets[%q].output must not escape project root", id))
}
if ds.NamingScheme != "" && ds.NamingScheme != "nwn_bmu" && ds.NamingScheme != "passthrough" {
failures = append(failures, fmt.Errorf("music.datasets[%q].naming_scheme %q is not supported", id, ds.NamingScheme))
}
if ds.PackageMode != "" && ds.PackageMode != "hak_asset" && ds.PackageMode != "none" {
failures = append(failures, fmt.Errorf("music.datasets[%q].package_mode %q is not supported", id, ds.PackageMode))
}
if ds.OutputExtension != "" && !strings.HasPrefix(ds.OutputExtension, ".") {
failures = append(failures, fmt.Errorf("music.datasets[%q].output_extension must start with .", id))
}
}
return failures
}
func pathEscapesRoot(path string) bool { func pathEscapesRoot(path string) bool {
clean := filepath.Clean(filepath.FromSlash(path)) clean := filepath.Clean(filepath.FromSlash(path))
return clean == ".." || strings.HasPrefix(clean, ".."+string(filepath.Separator)) return clean == ".." || strings.HasPrefix(clean, ".."+string(filepath.Separator))
+160 -110
View File
@@ -46,7 +46,7 @@ haks:
} }
} }
func TestEffectiveConfigAppliesVisibleToolkitDefaults(t *testing.T) { func TestEffectiveConfigReportsConfiguredAndDefaultProvenance(t *testing.T) {
root := t.TempDir() root := t.TempDir()
writeProjectFile(t, filepath.Join(root, ConfigFile), ` writeProjectFile(t, filepath.Join(root, ConfigFile), `
module: module:
@@ -62,35 +62,13 @@ paths:
} }
effective := proj.EffectiveConfig() effective := proj.EffectiveConfig()
if got, want := effective.Paths.Build, "build"; got != want { for _, field := range []string{"module.name", "paths.build"} {
t.Fatalf("expected default build path %q, got %q", want, got) if strings.TrimSpace(effective.Provenance[field].Source) == "" {
t.Errorf("expected %s to report provenance", field)
} }
if got, want := effective.Outputs.HAKManifest, "haks.json"; got != want {
t.Fatalf("expected default HAK manifest %q, got %q", want, got)
} }
if got, want := strings.Join(effective.Extract.Archives, ","), "testmod.mod"; got != want { if effective.Module.Name != "Test Module" {
t.Fatalf("expected default extract archives %q, got %q", want, got) t.Fatalf("configured module name was not preserved: %q", effective.Module.Name)
}
if effective.Extract.ConsumeArchives == nil || *effective.Extract.ConsumeArchives {
t.Fatalf("expected default extract consume_archives false, got %#v", effective.Extract.ConsumeArchives)
}
if got, want := effective.TopData.PackageHAK, "sow_top.hak"; got != want {
t.Fatalf("expected default topdata HAK %q, got %q", want, got)
}
if got, want := strings.Join(effective.Validation.BuiltinScriptPrefixes, ","), "ga_,gc_,gen_,gui_,nw_,nwg_,ta_,x0_,x1_,x2_,x3_"; got != want {
t.Fatalf("expected default built-in script prefixes %q, got %q", want, got)
}
if got := strings.Join(effective.Validation.RequiredFields["ifo"], ","); got != "Mod_Name" {
t.Fatalf("expected default IFO required fields, got %#v", effective.Validation.RequiredFields)
}
if got, want := strings.Join(effective.Music.ConvertExtensions, ","), ".flac,.m4a,.mp3,.ogg,.wav"; got != want {
t.Fatalf("expected default music convert extensions %q, got %q", want, got)
}
if prov := effective.Provenance["paths.build"]; prov.Source != "toolkit default" {
t.Fatalf("expected paths.build toolkit default provenance, got %#v", prov)
}
if prov := effective.Provenance["module.name"]; prov.Source != "yaml" {
t.Fatalf("expected module.name YAML provenance, got %#v", prov)
} }
} }
@@ -116,12 +94,16 @@ validation:
if err != nil { if err != nil {
t.Fatalf("Load returned error: %v", err) t.Fatalf("Load returned error: %v", err)
} }
got := strings.Join(proj.EffectiveConfig().Validation.BuiltinScriptPrefixes, ",") effective := proj.EffectiveConfig()
if want := "custom_,nw_"; got != want { for _, prefix := range []string{"custom_", "nw_"} {
t.Fatalf("expected configured validation prefixes %q, got %q", want, got) if !slices.Contains(effective.Validation.BuiltinScriptPrefixes, prefix) {
t.Errorf("configured validation prefixes missing %q: %#v", prefix, effective.Validation.BuiltinScriptPrefixes)
}
}
for _, field := range []string{"Mod_Hak", "Mod_Name"} {
if !slices.Contains(effective.Validation.RequiredFields["ifo"], field) {
t.Errorf("configured required fields missing %q: %#v", field, effective.Validation.RequiredFields)
} }
if got := strings.Join(proj.EffectiveConfig().Validation.RequiredFields["ifo"], ","); got != "Mod_Hak,Mod_Name" {
t.Fatalf("expected configured required fields, got %#v", proj.EffectiveConfig().Validation.RequiredFields)
} }
} }
@@ -402,50 +384,30 @@ func TestValidateLayoutRejectsInvalidTopDataClassFeatInjections(t *testing.T) {
} }
} }
func TestScanUsesYAMLMusicConvertExtensionsWithinConfiguredDatasetRoots(t *testing.T) { func TestDefaultAssetExtensionsKeepAuthoredBMUWithoutConversionSources(t *testing.T) {
for _, extension := range []string{".bmu", ".wav"} {
if !slices.Contains(AssetExtensions, extension) {
t.Errorf("default asset extensions must retain %s", extension)
}
}
for _, extension := range []string{".mp3", ".ogg"} {
if slices.Contains(AssetExtensions, extension) {
t.Errorf("default asset extensions must not include conversion source %s", extension)
}
}
}
func TestLoadRejectsRemovedMusicConfig(t *testing.T) {
root := t.TempDir() root := t.TempDir()
if err := os.MkdirAll(filepath.Join(root, "assets", "audio", "westgate"), 0o755); err != nil {
t.Fatalf("mkdir westgate audio: %v", err)
}
if err := os.MkdirAll(filepath.Join(root, "assets", "audio", "other"), 0o755); err != nil {
t.Fatalf("mkdir other audio: %v", err)
}
writeProjectFile(t, filepath.Join(root, ConfigFile), ` writeProjectFile(t, filepath.Join(root, ConfigFile), `
module: module:
name: Test Module name: Test Module
resref: testmod resref: testmod
paths:
source: src
assets: assets
music: music:
defaults: datasets: {}
convert_extensions:
- .flac
datasets:
westgate_audio:
source: audio/westgate
convert_extensions:
- .aac
`) `)
writeProjectFile(t, filepath.Join(root, "assets", "audio", "westgate", "theme.aac"), "aac") if _, err := Load(root); err == nil || !strings.Contains(err.Error(), "music") {
writeProjectFile(t, filepath.Join(root, "assets", "audio", "westgate", "theme.flac"), "flac") t.Fatalf("expected strict YAML decoding to reject removed music config, got %v", err)
writeProjectFile(t, filepath.Join(root, "assets", "audio", "other", "ignored.aac"), "aac")
proj, err := Load(root)
if err != nil {
t.Fatalf("Load returned error: %v", err)
}
if err := proj.Scan(); err != nil {
t.Fatalf("Scan returned error: %v", err)
}
if !slices.Contains(proj.Inventory.AssetFiles, "audio/westgate/theme.aac") {
t.Fatalf("expected configured dataset AAC file to be discovered, got %#v", proj.Inventory.AssetFiles)
}
if !slices.Contains(proj.Inventory.AssetFiles, "audio/westgate/theme.flac") {
t.Fatalf("expected configured dataset FLAC file to be discovered, got %#v", proj.Inventory.AssetFiles)
}
if slices.Contains(proj.Inventory.AssetFiles, "audio/other/ignored.aac") {
t.Fatalf("did not expect non-dataset AAC file to be discovered, got %#v", proj.Inventory.AssetFiles)
} }
} }
@@ -853,8 +815,8 @@ paths:
if !bytes.Equal(first, second) { if !bytes.Equal(first, second) {
t.Fatalf("effective config JSON is not deterministic") t.Fatalf("effective config JSON is not deterministic")
} }
if !bytes.Contains(first, []byte(`"hak_manifest": "haks.json"`)) { if !bytes.Contains(first, []byte(`"name": "Test Module"`)) {
t.Fatalf("effective config JSON missing HAK manifest default: %s", first) t.Fatalf("effective config JSON missing configured module name: %s", first)
} }
} }
@@ -1131,6 +1093,52 @@ func TestValidateLayoutAllowsMissingAssetsDir(t *testing.T) {
} }
} }
// paths.build is an OUTPUT dir the builder creates (MkdirAll) before writing, so
// a bare clone with no build dir yet must still validate/build with no pre-step
// (R2/parity). Only a build path that exists but is not a directory is an error.
func TestValidateLayoutAllowsMissingBuildDir(t *testing.T) {
root := t.TempDir()
mkdirAll(t, filepath.Join(root, "src"))
// deliberately do NOT create build/
proj := &Project{
Root: root,
Config: Config{
Module: ModuleConfig{Name: "Test", ResRef: "test"},
Paths: PathConfig{Source: "src", Build: "build"},
},
}
if err := proj.ValidateLayout(); err != nil {
t.Fatalf("ValidateLayout returned error for missing build dir: %v", err)
}
if _, err := os.Stat(filepath.Join(root, "build")); !errors.Is(err, os.ErrNotExist) {
t.Fatalf("expected missing build dir to remain absent, got err=%v", err)
}
}
// A build path that exists but is a regular file is still a hard error.
func TestValidateLayoutRejectsBuildDirThatIsAFile(t *testing.T) {
root := t.TempDir()
mkdirAll(t, filepath.Join(root, "src"))
if err := os.WriteFile(filepath.Join(root, "build"), []byte("i am a file, not a dir"), 0o644); err != nil {
t.Fatal(err)
}
proj := &Project{
Root: root,
Config: Config{
Module: ModuleConfig{Name: "Test", ResRef: "test"},
Paths: PathConfig{Source: "src", Build: "build"},
},
}
err := proj.ValidateLayout()
if err == nil || !strings.Contains(err.Error(), "paths.build must be a directory") {
t.Fatalf("expected paths.build-must-be-a-directory error, got %v", err)
}
}
func TestValidateLayoutRejectsInvalidTopDataPackageOutputNames(t *testing.T) { func TestValidateLayoutRejectsInvalidTopDataPackageOutputNames(t *testing.T) {
root := t.TempDir() root := t.TempDir()
mkdirAll(t, filepath.Join(root, "src")) mkdirAll(t, filepath.Join(root, "src"))
@@ -1331,7 +1339,7 @@ func TestValidateLayoutRejectsInvalidAutogenConsumerConfig(t *testing.T) {
} }
} }
func TestValidateLayoutRejectsInvalidHeadVisualeffectsConfig(t *testing.T) { func TestValidateLayoutRejectsInvalidAccessoryVisualeffectsConfig(t *testing.T) {
root := t.TempDir() root := t.TempDir()
mkdirAll(t, filepath.Join(root, "src")) mkdirAll(t, filepath.Join(root, "src"))
mkdirAll(t, filepath.Join(root, "build")) mkdirAll(t, filepath.Join(root, "build"))
@@ -1356,8 +1364,8 @@ func TestValidateLayoutRejectsInvalidHeadVisualeffectsConfig(t *testing.T) {
AssetName: "sow-accessory-vfx-manifest.json", AssetName: "sow-accessory-vfx-manifest.json",
CacheName: "sow-accessory-vfx-manifest.json", CacheName: "sow-accessory-vfx-manifest.json",
}, },
HeadVisualeffects: HeadVisualeffectsConfig{ AccessoryVisualeffects: AccessoryVisualeffectsConfig{
Groups: map[string]HeadVisualeffectGroupConfig{ Groups: map[string]AccessoryVisualeffectGroupConfig{
"head_jewels": { "head_jewels": {
ModelColumns: []string{"Imp Root M Node"}, ModelColumns: []string{"Imp Root M Node"},
}, },
@@ -1373,7 +1381,7 @@ func TestValidateLayoutRejectsInvalidHeadVisualeffectsConfig(t *testing.T) {
err := proj.ValidateLayout() err := proj.ValidateLayout()
if err == nil { if err == nil {
t.Fatal("expected head visualeffects validation error") t.Fatal("expected accessory visualeffects validation error")
} }
for _, needle := range []string{ for _, needle := range []string{
"autogen.consumers[0].accessory_visualeffects.groups[head_jewels].prefix", "autogen.consumers[0].accessory_visualeffects.groups[head_jewels].prefix",
@@ -1387,7 +1395,7 @@ func TestValidateLayoutRejectsInvalidHeadVisualeffectsConfig(t *testing.T) {
} }
} }
func TestValidateLayoutRejectsInvalidHeadVisualeffectsNamingConfig(t *testing.T) { func TestValidateLayoutRejectsInvalidAccessoryVisualeffectsNamingConfig(t *testing.T) {
root := t.TempDir() root := t.TempDir()
mkdirAll(t, filepath.Join(root, "src")) mkdirAll(t, filepath.Join(root, "src"))
mkdirAll(t, filepath.Join(root, "build")) mkdirAll(t, filepath.Join(root, "build"))
@@ -1412,8 +1420,8 @@ func TestValidateLayoutRejectsInvalidHeadVisualeffectsNamingConfig(t *testing.T)
AssetName: "sow-accessory-vfx-manifest.json", AssetName: "sow-accessory-vfx-manifest.json",
CacheName: "sow-accessory-vfx-manifest.json", CacheName: "sow-accessory-vfx-manifest.json",
}, },
HeadVisualeffects: HeadVisualeffectsConfig{ AccessoryVisualeffects: AccessoryVisualeffectsConfig{
Groups: map[string]HeadVisualeffectGroupConfig{ Groups: map[string]AccessoryVisualeffectGroupConfig{
"head_features": {}, "head_features": {},
}, },
GroupTokenSource: "unsupported", GroupTokenSource: "unsupported",
@@ -1429,7 +1437,7 @@ func TestValidateLayoutRejectsInvalidHeadVisualeffectsNamingConfig(t *testing.T)
err := proj.ValidateLayout() err := proj.ValidateLayout()
if err == nil { if err == nil {
t.Fatal("expected head visualeffects naming validation error") t.Fatal("expected accessory visualeffects naming validation error")
} }
for _, needle := range []string{ for _, needle := range []string{
"autogen.consumers[0].accessory_visualeffects.group_token_source", "autogen.consumers[0].accessory_visualeffects.group_token_source",
@@ -1443,7 +1451,52 @@ func TestValidateLayoutRejectsInvalidHeadVisualeffectsNamingConfig(t *testing.T)
} }
} }
func TestLoadRejectsLegacyHeadVisualeffectsNamingFields(t *testing.T) { func TestValidateLayoutAcceptsCDNChannelConsumerWithoutReleasedSourceFields(t *testing.T) {
// cdn_channel consumers resolve entirely from Source.* at runtime; they must
// not be required to supply root/include/derive/manifest fields.
root := t.TempDir()
mkdirAll(t, filepath.Join(root, "src"))
mkdirAll(t, filepath.Join(root, "build"))
proj := &Project{
Root: root,
Config: Config{
Module: ModuleConfig{Name: "Test", ResRef: "test"},
Paths: PathConfig{Source: "src", Build: "build"},
Autogen: AutogenConfig{
Consumers: []AutogenConsumerConfig{
{
ID: "accessory_visualeffects",
Producer: "accessory_visualeffects",
Dataset: "visualeffects",
Mode: "accessory_visualeffects",
Optional: true,
AccessoryVisualeffects: AccessoryVisualeffectsConfig{
Groups: map[string]AccessoryVisualeffectGroupConfig{
"head_accessories": {Prefix: "head_acc_"},
"chest_accessories": {Prefix: "chest_acc_"},
"head_decorations": {Prefix: "head_dec_"},
"head_features": {Prefix: "head_feat_"},
},
},
Source: AutogenSourceConfig{
Kind: "cdn_channel",
ChannelsPath: "releases/haks/channels.json",
ManifestPath: "releases/haks/{tag}/vfxs.yml",
},
// Intentionally omitted: Root, Include, Derive, Manifest
},
},
},
},
}
if err := proj.ValidateLayout(); err != nil {
t.Fatalf("cdn_channel consumer should pass ValidateLayout without released-source fields, got: %v", err)
}
}
func TestLoadRejectsLegacyAccessoryVisualeffectsNamingFields(t *testing.T) {
root := t.TempDir() root := t.TempDir()
writeProjectFile(t, filepath.Join(root, ConfigFile), ` writeProjectFile(t, filepath.Join(root, ConfigFile), `
module: module:
@@ -1477,7 +1530,7 @@ autogen:
_, err := Load(root) _, err := Load(root)
if err == nil { if err == nil {
t.Fatal("expected legacy head visualeffects naming fields to be rejected") t.Fatal("expected legacy accessory visualeffects naming fields to be rejected")
} }
if !strings.Contains(err.Error(), "legacy_groups") { if !strings.Contains(err.Error(), "legacy_groups") {
t.Fatalf("expected legacy_groups unknown field error, got %v", err) t.Fatalf("expected legacy_groups unknown field error, got %v", err)
@@ -1707,34 +1760,6 @@ func TestValidateLayoutRejectsDuplicateHAKNames(t *testing.T) {
} }
} }
func TestValidateLayoutRejectsAmbiguousMusicPrefixes(t *testing.T) {
root := t.TempDir()
mkdirAll(t, filepath.Join(root, "assets"))
mkdirAll(t, filepath.Join(root, "build"))
proj := &Project{
Root: root,
Config: Config{
Module: ModuleConfig{Name: "Test", ResRef: "test"},
Paths: PathConfig{Assets: "assets", Build: "build"},
Music: MusicConfig{
Prefixes: map[string]string{
"envi/music": "mus_",
"/envi/music": "mus2_",
},
},
},
}
err := proj.ValidateLayout()
if err == nil {
t.Fatal("expected music prefix validation error")
}
if !strings.Contains(err.Error(), "normalize to the same path") {
t.Fatalf("expected ambiguous prefix validation error, got %v", err)
}
}
func TestScanAllowsMissingAssetsDir(t *testing.T) { func TestScanAllowsMissingAssetsDir(t *testing.T) {
root := t.TempDir() root := t.TempDir()
mkdirAll(t, filepath.Join(root, "src")) mkdirAll(t, filepath.Join(root, "src"))
@@ -1814,6 +1839,31 @@ func TestCloneWithHAKNamesRejectsUnknownHAKs(t *testing.T) {
} }
} }
func TestValidateAutogenConsumerSourcesCDNChannel(t *testing.T) {
cases := []struct {
name string
source AutogenSourceConfig
wantFail bool
}{
{"valid", AutogenSourceConfig{Kind: "cdn_channel", ChannelsPath: "releases/haks/channels.json", ManifestPath: "releases/haks/{tag}/vfxs.yml"}, false},
{"missing channels_path", AutogenSourceConfig{Kind: "cdn_channel", ManifestPath: "releases/haks/{tag}/vfxs.yml"}, true},
{"missing manifest_path", AutogenSourceConfig{Kind: "cdn_channel", ChannelsPath: "releases/haks/channels.json"}, true},
{"manifest_path lacks {tag}", AutogenSourceConfig{Kind: "cdn_channel", ChannelsPath: "releases/haks/channels.json", ManifestPath: "releases/haks/vfxs.yml"}, true},
{"empty kind is ignored", AutogenSourceConfig{}, false},
}
for _, tc := range cases {
t.Run(tc.name, func(t *testing.T) {
failures := validateAutogenConsumerSources([]AutogenConsumerConfig{{ID: "x", Source: tc.source}})
if tc.wantFail && len(failures) == 0 {
t.Fatalf("expected validation failure, got none")
}
if !tc.wantFail && len(failures) != 0 {
t.Fatalf("expected no failure, got %v", failures)
}
})
}
}
func loadAndValidate(root string) error { func loadAndValidate(root string) error {
proj, err := Load(root) proj, err := Load(root)
if err != nil { if err != nil {
+460 -86
View File
@@ -3,7 +3,9 @@ package topdata
import ( import (
"bytes" "bytes"
"encoding/json" "encoding/json"
"errors"
"fmt" "fmt"
"net/http"
"os" "os"
"path/filepath" "path/filepath"
"slices" "slices"
@@ -12,16 +14,32 @@ import (
"time" "time"
"git.westgate.pw/ShadowsOverWestgate/sow-tools/internal/project" "git.westgate.pw/ShadowsOverWestgate/sow-tools/internal/project"
"gopkg.in/yaml.v3"
) )
const autogenManifestCacheMaxAge = time.Hour const autogenManifestCacheMaxAge = time.Hour
const defaultBunnyCDNBase = "https://cdn-a7f3k9.westgate.pw"
// errAutogenManifestUnavailable marks a released autogen manifest that could not
// be located or fetched (network failure, missing release/asset, empty payload,
// or an undeterminable sow-assets repo). It is deliberately distinct from an
// asset that was found but reports a row removed: for an optional consumer this
// sentinel triggers a fail-open path that *preserves* the consumer's existing
// pinned lock entries instead of pruning them, so StrRef/row IDs are not
// reshuffled while the asset source is merely temporarily out of reach.
var errAutogenManifestUnavailable = errors.New("autogen manifest unavailable")
func unavailableAutogenManifest(err error) error {
return fmt.Errorf("%w: %w", errAutogenManifestUnavailable, err)
}
type autogenManifest struct { type autogenManifest struct {
ID string `json:"id"` ID string `json:"id"`
Repo string `json:"repo"` Repo string `json:"repo"`
Ref string `json:"ref"` Ref string `json:"ref"`
GeneratedAt string `json:"generated_at"` GeneratedAt string `json:"generated_at"`
HeadVisualeffects *project.HeadVisualeffectsConfig `json:"accessory_visualeffects,omitempty"` AccessoryVisualeffects *project.AccessoryVisualeffectsConfig `json:"accessory_visualeffects,omitempty"`
Entries []autogenManifestEntry `json:"entries"` Entries []autogenManifestEntry `json:"entries"`
} }
@@ -61,9 +79,18 @@ func applyAutogenConsumers(p *project.Project, collected []nativeCollectedDatase
} }
manifest, err := resolveAutogenConsumerManifest(p, consumer, progress) manifest, err := resolveAutogenConsumerManifest(p, consumer, progress)
if err != nil { if err != nil {
if consumer.Optional && isIgnorableOptionalAutogenError(err) { if consumer.Optional && errors.Is(err, errAutogenManifestUnavailable) {
// Fail open: the asset manifest is merely unreachable, not
// authoritatively empty. Build without its rows, but keep the
// consumer's already-pinned lock entries so their IDs are not
// freed and reshuffled on a later build once the asset returns.
preserved, perr := preserveAutogenConsumerLockEntries(result, consumer)
if perr != nil {
return nil, perr
}
result = preserved
if progress != nil { if progress != nil {
progress(fmt.Sprintf("Skipping optional autogen consumer %s: %v", consumer.ID, err)) progress(fmt.Sprintf("Autogen consumer %s: released manifest unavailable (%v); preserving existing lock entries, generating no new rows", consumer.ID, err))
} }
continue continue
} }
@@ -77,7 +104,7 @@ func applyAutogenConsumers(p *project.Project, collected []nativeCollectedDatase
case "parts_rows": case "parts_rows":
result = augmentWithAutogeneratedParts(result, autogenPartsInventory(entries)) result = augmentWithAutogeneratedParts(result, autogenPartsInventory(entries))
case "accessory_visualeffects": case "accessory_visualeffects":
result, err = augmentWithAutogeneratedHeadVisualeffects(result, entries, consumer, manifest.HeadVisualeffects) result, err = augmentWithAutogeneratedAccessoryVisualeffects(result, entries, consumer, manifest.AccessoryVisualeffects)
if err != nil { if err != nil {
return nil, err return nil, err
} }
@@ -90,35 +117,129 @@ func applyAutogenConsumers(p *project.Project, collected []nativeCollectedDatase
func autogenConsumerTargetsCollectedDataset(collected []nativeCollectedDataset, consumer project.AutogenConsumerConfig) bool { func autogenConsumerTargetsCollectedDataset(collected []nativeCollectedDataset, consumer project.AutogenConsumerConfig) bool {
for _, dataset := range collected { for _, dataset := range collected {
switch consumer.Mode { if autogenConsumerTargetsDataset(dataset, consumer) {
case "parts_rows":
if isAutogenEligiblePartsDataset(dataset) {
return true return true
} }
case "accessory_visualeffects":
if dataset.Dataset.Name == "visualeffects" {
return true
}
case "cachedmodels_rows":
if dataset.Dataset.Name == "cachedmodels" {
return true
}
}
} }
return false return false
} }
func isIgnorableOptionalAutogenError(err error) bool { // preserveAutogenConsumerLockEntries restores the lock keys that an autogen
if err == nil { // consumer owns from the on-disk lockfile back into the in-memory dataset,
// reusing this build's already-collected (and pruned) state. It is called only
// when the consumer's released manifest is unavailable, so the rows themselves
// are not regenerated; we just keep the key->ID pins alive. Keys are matched
// against the consumer's own naming policy so unrelated (authored) rows that were
// legitimately removed are not resurrected.
func preserveAutogenConsumerLockEntries(collected []nativeCollectedDataset, consumer project.AutogenConsumerConfig) ([]nativeCollectedDataset, error) {
matches := autogenConsumerManagedLockKeyMatcher(consumer)
if matches == nil {
// No managed-key matcher for this mode: nothing safe to preserve.
return collected, nil
}
result := append([]nativeCollectedDataset(nil), collected...)
for i, dataset := range result {
if !autogenConsumerTargetsDataset(dataset, consumer) {
continue
}
if strings.TrimSpace(dataset.Dataset.LockPath) == "" {
continue
}
onDisk, err := loadLockfile(dataset.Dataset.LockPath)
if err != nil {
// No readable lockfile means there is nothing pinned to preserve.
continue
}
lockData := dataset.LockData
if lockData == nil {
lockData = map[string]int{}
}
restored := 0
for key, rowID := range onDisk {
if _, present := lockData[key]; present {
continue
}
if !matches(key) {
continue
}
lockData[key] = rowID
restored++
}
if restored == 0 {
continue
}
result[i].LockData = lockData
result[i].LockModified = !lockDataEqual(onDisk, lockData)
}
return result, nil
}
// autogenConsumerManagedLockKeyMatcher returns a predicate identifying the lock
// keys an autogen consumer is responsible for, or nil for modes whose keys we
// cannot scope precisely (in which case preservation is skipped rather than
// risk resurrecting unrelated keys).
func autogenConsumerManagedLockKeyMatcher(consumer project.AutogenConsumerConfig) func(string) bool {
switch consumer.Mode {
case "accessory_visualeffects":
dataset := strings.TrimSpace(consumer.Dataset)
if dataset == "" {
dataset = "visualeffects"
}
prefix := dataset + ":"
// Build the group set from the resolved policy so consumers that omit
// AccessoryVisualeffects.Groups still match the four default group names.
// NOTE: this matcher assumes the default folder_name+preserve group
// representation. Keys produced under case:lower or
// group_token_source:prefix would not match and would escape pruning;
// no production config uses those combinations.
policy := resolveAccessoryVisualeffectsPolicy(consumer, nil)
groups := make(map[string]struct{}, len(policy.Groups))
for group := range policy.Groups {
if group = strings.TrimSpace(group); group != "" {
groups[group] = struct{}{}
}
}
if len(groups) == 0 {
return nil
}
return func(key string) bool {
if !strings.HasPrefix(key, prefix) {
return false
}
rest := key[len(prefix):]
idx := strings.Index(rest, "/")
if idx <= 0 {
return false
}
_, ok := groups[rest[:idx]]
return ok
}
default:
return nil
}
}
func autogenConsumerTargetsDataset(dataset nativeCollectedDataset, consumer project.AutogenConsumerConfig) bool {
switch consumer.Mode {
case "parts_rows":
return isAutogenEligiblePartsDataset(dataset)
case "accessory_visualeffects":
return dataset.Dataset.Name == "visualeffects"
case "cachedmodels_rows":
return dataset.Dataset.Name == "cachedmodels"
default:
return false return false
} }
message := err.Error()
return strings.Contains(message, "HTTP 404") ||
strings.Contains(message, "not found") ||
strings.Contains(message, "entries is empty")
} }
func resolveAutogenConsumerManifest(p *project.Project, consumer project.AutogenConsumerConfig, progress func(string)) (*autogenManifest, error) { func resolveAutogenConsumerManifest(p *project.Project, consumer project.AutogenConsumerConfig, progress func(string)) (*autogenManifest, error) {
if strings.TrimSpace(consumer.Source.Kind) == "cdn_channel" {
return resolveCDNChannelManifest(p, consumer, consumer.Source, progress)
}
if manifestFile := strings.TrimSpace(consumer.ManifestFile); manifestFile != "" {
return readAutogenConsumerManifestFile(p, consumer, manifestFile, progress)
}
overrideRoot, err := resolveAutogenLocalOverrideRoot(p, consumer) overrideRoot, err := resolveAutogenLocalOverrideRoot(p, consumer)
if err != nil { if err != nil {
return nil, err return nil, err
@@ -136,6 +257,214 @@ func resolveAutogenConsumerManifest(p *project.Project, consumer project.Autogen
return resolveReleasedAutogenManifest(p, consumer, progress) return resolveReleasedAutogenManifest(p, consumer, progress)
} }
// readAutogenConsumerManifestFile reads a pre-resolved autogenManifest JSON from
// a local file (relative paths resolve against the project root). A missing file
// is an ignorable-optional condition — it returns errAutogenManifestUnavailable
// so an optional consumer fails open and preserves its pinned lock IDs. A present
// but malformed file, or an id that disagrees with the consumer's producer, is a
// hard error.
func readAutogenConsumerManifestFile(p *project.Project, consumer project.AutogenConsumerConfig, manifestFile string, progress func(string)) (*autogenManifest, error) {
path := manifestFile
if !filepath.IsAbs(path) {
path = filepath.Join(p.Root, path)
}
raw, err := os.ReadFile(path)
if err != nil {
if os.IsNotExist(err) {
return nil, unavailableAutogenManifest(fmt.Errorf("manifest file %s not found", path))
}
return nil, fmt.Errorf("read autogen manifest file %s: %w", path, err)
}
var manifest autogenManifest
if err := json.Unmarshal(raw, &manifest); err != nil {
return nil, fmt.Errorf("parse autogen manifest file %s: %w", path, err)
}
if manifest.ID != "" && manifest.ID != consumer.Producer {
return nil, fmt.Errorf("autogen manifest file %s declared id %q, expected %q", path, manifest.ID, consumer.Producer)
}
if progress != nil {
progress(fmt.Sprintf("Reading local autogen manifest for %s from %s...", consumer.ID, path))
}
return &manifest, nil
}
// resolveCDNChannelManifest resolves the accessory-VFX model list anonymously
// from the asset CDN: channel pointer -> tag -> per-tag vfxs.yml. It ports the
// fail policy of the deleted resolve-accessory-vfx.sh: unreachable inputs fail
// OPEN (errAutogenManifestUnavailable -> optional consumer preserves lock IDs);
// a published-but-broken release (vfxs.yml 404 while haks.json present) or a
// malformed vfxs.yml HARD fail.
func resolveCDNChannelManifest(p *project.Project, consumer project.AutogenConsumerConfig, src project.AutogenSourceConfig, progress func(string)) (*autogenManifest, error) {
if progress == nil {
progress = func(string) {}
}
// 1. Offline / air-gapped override: a vfxs.yml path or a manifest-repo
// checkout root containing assets/vfxs.yml. Skips the network entirely.
if envName := strings.TrimSpace(src.OfflineOverrideEnv); envName != "" {
if override := strings.TrimSpace(os.Getenv(envName)); override != "" {
vfxsPath := override
if info, err := os.Stat(override); err == nil && info.IsDir() {
vfxsPath = filepath.Join(override, "assets", "vfxs.yml")
}
raw, err := os.ReadFile(vfxsPath)
if err != nil {
return nil, fmt.Errorf("offline vfxs override %s: %w", vfxsPath, err)
}
entries, err := filterCDNChannelEntries(raw, consumer)
if err != nil {
return nil, err
}
progress(fmt.Sprintf("Using offline vfxs override for %s from %s...", consumer.ID, vfxsPath))
return &autogenManifest{ID: consumer.Producer, Ref: "local", Entries: entries}, nil
}
}
// 2. Channel: explicit env, else derived from the git tag.
channel := strings.TrimSpace(os.Getenv(strings.TrimSpace(src.ChannelEnv)))
if channel == "" {
channel = deriveAssetChannel(p.Root)
}
// 3. CDN base: baked default, then env override, then config literal (most specific wins).
cdnBase := defaultBunnyCDNBase
if envName := strings.TrimSpace(src.CDNBaseEnv); envName != "" {
if v := strings.TrimSpace(os.Getenv(envName)); v != "" {
cdnBase = v
}
}
if v := strings.TrimSpace(src.CDNBase); v != "" {
cdnBase = v
}
cdnBase = strings.TrimRight(cdnBase, "/")
join := func(path, tag string) string {
return cdnBase + "/" + strings.TrimLeft(strings.ReplaceAll(path, "{tag}", tag), "/")
}
// 4. channels.json -> tag. Unreachable / non-JSON / channel-absent = fail open.
channelsURL := join(src.ChannelsPath, "")
status, body, err := httpGetStatus(channelsURL)
if err != nil {
return nil, unavailableAutogenManifest(fmt.Errorf("channels.json unreachable %s: %w", channelsURL, err))
}
if status != http.StatusOK {
return nil, unavailableAutogenManifest(fmt.Errorf("channels.json HTTP %d %s", status, channelsURL))
}
var channels map[string]string
if err := json.Unmarshal(body, &channels); err != nil {
return nil, unavailableAutogenManifest(fmt.Errorf("channels.json not JSON %s: %w", channelsURL, err))
}
tag := strings.TrimSpace(channels[channel])
if tag == "" {
return nil, unavailableAutogenManifest(fmt.Errorf("channel %q absent in channels.json (%s)", channel, channelsURL))
}
progress(fmt.Sprintf("Accessory VFX: channel %s -> tag %s", channel, tag))
// 5. vfxs.yml for the tag.
manifestURL := join(src.ManifestPath, tag)
vstatus, vbody, err := httpGetStatus(manifestURL)
if err != nil {
return nil, unavailableAutogenManifest(fmt.Errorf("vfxs.yml unreachable %s: %w", manifestURL, err))
}
switch vstatus {
case http.StatusOK:
// fall through to parse
case http.StatusNotFound:
// 404 + release marker present = published-but-broken release: HARD fail
// (this is the v0.1.4 silent-drop bug). 404 + marker absent = no published
// release for this tag: fail open.
if marker := strings.TrimSpace(src.ReleaseMarkerPath); marker != "" {
markerURL := join(marker, tag)
if mstatus, _, merr := httpGetStatus(markerURL); merr == nil && mstatus == http.StatusOK {
return nil, fmt.Errorf("release %s has %s but no vfxs.yml (%s) — accessory rows would silently vanish; backfill/republish vfxs.yml for %s", tag, marker, manifestURL, tag)
}
}
return nil, unavailableAutogenManifest(fmt.Errorf("vfxs.yml 404 %s (no published release for %s)", manifestURL, tag))
default:
return nil, unavailableAutogenManifest(fmt.Errorf("vfxs.yml HTTP %d %s", vstatus, manifestURL))
}
entries, err := filterCDNChannelEntries(vbody, consumer)
if err != nil {
return nil, err // malformed vfxs.yml = HARD fail
}
progress(fmt.Sprintf("Accessory VFX: resolved %d model entries from %s", len(entries), manifestURL))
return &autogenManifest{ID: consumer.Producer, Ref: tag, Entries: entries}, nil
}
// filterCDNChannelEntries parses vfxs.yml and keeps restype==mdl assets under
// vfxs/<group>/ for the consumer's 4 accessory groups, stripping the leading
// vfxs/ from each source path. A present-but-malformed manifest (unparseable, or
// no assets array) is an error; an empty assets array yields zero entries.
func filterCDNChannelEntries(raw []byte, consumer project.AutogenConsumerConfig) ([]autogenManifestEntry, error) {
var manifest struct {
Assets *[]struct {
Path string `yaml:"path"`
Restype string `yaml:"restype"`
} `yaml:"assets"`
}
if err := yaml.Unmarshal(raw, &manifest); err != nil {
return nil, fmt.Errorf("malformed vfxs.yml (cannot parse): %w", err)
}
if manifest.Assets == nil {
return nil, fmt.Errorf("malformed vfxs.yml (no assets array)")
}
groups := make(map[string]struct{}, len(consumer.AccessoryVisualeffects.Groups))
for g := range consumer.AccessoryVisualeffects.Groups {
if g = strings.TrimSpace(g); g != "" {
groups[g] = struct{}{}
}
}
var entries []autogenManifestEntry
for _, a := range *manifest.Assets {
if a.Restype != "mdl" {
continue
}
path := filepath.ToSlash(strings.TrimSpace(a.Path))
if !strings.HasPrefix(path, "vfxs/") {
continue
}
rel := strings.TrimPrefix(path, "vfxs/")
parts := strings.Split(rel, "/")
if len(parts) < 2 {
continue
}
if _, ok := groups[parts[0]]; !ok {
continue
}
entry := autogenManifestEntry{
Source: rel,
Group: parts[0],
ModelStem: strings.TrimSuffix(parts[len(parts)-1], ".mdl"),
}
if len(parts) > 2 {
entry.Subgroup = strings.Join(parts[1:len(parts)-1], "/")
}
entries = append(entries, entry)
}
slices.SortFunc(entries, func(a, b autogenManifestEntry) int {
return strings.Compare(a.Source, b.Source)
})
return entries, nil
}
// deriveAssetChannel maps the current git tag (or GITHUB_REF_NAME) to a channel:
// a prerelease tag (v*-*) -> testing; a stable tag (v*) -> current; anything
// else (branch/PR/dev) -> current. Mirrors resolve-accessory-vfx.sh.
func deriveAssetChannel(root string) string {
ref := strings.TrimSpace(os.Getenv("GITHUB_REF_NAME"))
if ref == "" {
if out, err := gitOutput(root, "describe", "--tags", "--exact-match"); err == nil {
ref = strings.TrimSpace(out)
}
}
if strings.HasPrefix(ref, "v") && strings.Contains(ref, "-") {
return "testing"
}
return "current"
}
func resolveAutogenLocalOverrideRoot(p *project.Project, consumer project.AutogenConsumerConfig) (string, error) { func resolveAutogenLocalOverrideRoot(p *project.Project, consumer project.AutogenConsumerConfig) (string, error) {
root := strings.TrimSpace(consumer.LocalOverrideRoot) root := strings.TrimSpace(consumer.LocalOverrideRoot)
if root != "" { if root != "" {
@@ -366,14 +695,14 @@ func resolveReleasedAutogenManifest(p *project.Project, consumer project.Autogen
spec, err := deriveSowAssetsRepoSpec(p) spec, err := deriveSowAssetsRepoSpec(p)
if err != nil { if err != nil {
return nil, err return nil, unavailableAutogenManifest(err)
} }
manifestURL, err := resolveAutogenManifestAssetURL(spec, consumer.Manifest.ReleaseTag, consumer.Manifest.AssetName) manifestURL, err := resolveAutogenManifestAssetURL(spec, consumer.Manifest.ReleaseTag, consumer.Manifest.AssetName)
if err != nil { if err != nil {
if consumer.Mode == "parts_rows" { if consumer.Mode == "parts_rows" {
return nil, fmt.Errorf("%s", strings.Replace(err.Error(), "autogen manifest", "parts manifest", 1)) return nil, unavailableAutogenManifest(fmt.Errorf("%s", strings.Replace(err.Error(), "autogen manifest", "parts manifest", 1)))
} }
return nil, err return nil, unavailableAutogenManifest(err)
} }
if progress != nil { if progress != nil {
progress(fmt.Sprintf("Fetching released autogen manifest from %s...", manifestURL)) progress(fmt.Sprintf("Fetching released autogen manifest from %s...", manifestURL))
@@ -381,9 +710,9 @@ func resolveReleasedAutogenManifest(p *project.Project, consumer project.Autogen
manifest, err := fetchAutogenManifest(manifestURL) manifest, err := fetchAutogenManifest(manifestURL)
if err != nil { if err != nil {
if consumer.Mode == "parts_rows" { if consumer.Mode == "parts_rows" {
return nil, fmt.Errorf("%s", strings.Replace(err.Error(), "autogen manifest", "parts manifest", 1)) return nil, unavailableAutogenManifest(fmt.Errorf("%s", strings.Replace(err.Error(), "autogen manifest", "parts manifest", 1)))
} }
return nil, err return nil, unavailableAutogenManifest(err)
} }
if manifest.ID != "" && manifest.ID != consumer.Producer { if manifest.ID != "" && manifest.ID != consumer.Producer {
return nil, fmt.Errorf("autogen manifest %s declared id %q, expected %q", consumer.Manifest.AssetName, manifest.ID, consumer.Producer) return nil, fmt.Errorf("autogen manifest %s declared id %q, expected %q", consumer.Manifest.AssetName, manifest.ID, consumer.Producer)
@@ -676,12 +1005,12 @@ func augmentWithAutogeneratedCachedModels(collected []nativeCollectedDataset, en
return result, nil return result, nil
} }
func augmentWithAutogeneratedHeadVisualeffects(collected []nativeCollectedDataset, entries []autogenManifestEntry, consumer project.AutogenConsumerConfig, manifestPolicy *project.HeadVisualeffectsConfig) ([]nativeCollectedDataset, error) { func augmentWithAutogeneratedAccessoryVisualeffects(collected []nativeCollectedDataset, entries []autogenManifestEntry, consumer project.AutogenConsumerConfig, manifestPolicy *project.AccessoryVisualeffectsConfig) ([]nativeCollectedDataset, error) {
if len(entries) == 0 { if len(entries) == 0 {
return collected, nil return collected, nil
} }
policy := resolveHeadVisualeffectsPolicy(consumer, manifestPolicy) policy := resolveAccessoryVisualeffectsPolicy(consumer, manifestPolicy)
result := append([]nativeCollectedDataset(nil), collected...) result := append([]nativeCollectedDataset(nil), collected...)
for i, dataset := range result { for i, dataset := range result {
if dataset.Dataset.Name != "visualeffects" { if dataset.Dataset.Name != "visualeffects" {
@@ -730,30 +1059,61 @@ func augmentWithAutogeneratedHeadVisualeffects(collected []nativeCollectedDatase
return rowID return rowID
} }
liveLockKeys := make(map[string]struct{}, len(entries))
for _, entry := range entries { for _, entry := range entries {
key, label, modelStem, groupPolicy, ok := headVisualeffectIdentity(dataset.Dataset.Name, entry, policy) key, label, modelStem, groupPolicy, ok := accessoryVisualeffectIdentity(dataset.Dataset.Name, entry, policy)
if !ok { if !ok {
continue continue
} }
lockKey := accessoryVisualeffectLockKey(dataset.Dataset.Name, entry)
liveLockKeys[lockKey] = struct{}{}
if existing, exists := rowByKey[key]; exists { if existing, exists := rowByKey[key]; exists {
applyDiscoveredHeadVisualeffectDefaults(existing, dataset.Columns, modelStem, label, policy, groupPolicy) applyDiscoveredAccessoryVisualeffectDefaults(existing, dataset.Columns, modelStem, label, policy, groupPolicy)
if _, pinned := lockData[lockKey]; !pinned {
if rowID, ok := existing["id"].(int); ok {
lockData[lockKey] = rowID
}
}
continue continue
} }
rowID, ok := lockData[key] rowID, ok := lockData[lockKey]
if !ok { if !ok {
if preservedRowID, preserved := historicalLockData[key]; preserved { switch {
rowID = preservedRowID case hasKey(historicalLockData, lockKey):
} else { rowID = historicalLockData[lockKey]
case hasKey(historicalLockData, key):
// One-time cutover: adopt the ID from the old config-derived key
// so IDs carry over with zero shift on the key-scheme switch. The
// stale old key is pruned below.
rowID = historicalLockData[key]
default:
rowID = allocateNextID() rowID = allocateNextID()
} }
lockData[key] = rowID lockData[lockKey] = rowID
} }
newRow := createDefaultHeadVisualeffectRow(dataset.Columns, rowID, key, label, modelStem, policy, groupPolicy) newRow := createDefaultAccessoryVisualeffectRow(dataset.Columns, rowID, key, label, modelStem, policy, groupPolicy)
rows = append(rows, newRow) rows = append(rows, newRow)
rowByKey[key] = newRow rowByKey[key] = newRow
} }
// Stale pruning: drop managed lock keys (this consumer's accessory keys,
// incl. now-superseded old config-scheme keys) that no live entry resolves
// to, freeing their IDs for first-free reuse. Safe here because the
// augmentor only runs on a successful, non-empty resolution; the fail-open
// paths skip it entirely and never reach this code.
if managed := autogenConsumerManagedLockKeyMatcher(consumer); managed != nil {
for lk := range lockData {
if _, live := liveLockKeys[lk]; live {
continue
}
if managed(lk) {
delete(lockData, lk)
}
}
}
slices.SortFunc(rows, func(a, b map[string]any) int { slices.SortFunc(rows, func(a, b map[string]any) int {
return a["id"].(int) - b["id"].(int) return a["id"].(int) - b["id"].(int)
}) })
@@ -772,8 +1132,22 @@ func nextAvailableAutogenID(used map[int]struct{}) int {
} }
} }
type headVisualeffectsPolicy struct { // accessoryVisualeffectLockKey is the model-anchored autogen lock identity:
Groups map[string]headVisualeffectsGroupPolicy // the dataset namespace prefix plus the model source path from vfxs.yml (leading
// vfxs/ already stripped on entry.Source). Config-derived presentation parts
// (group token, category, delimiter, case, prefix-stripping) are deliberately
// excluded so a config-only edit never reshuffles row IDs.
func accessoryVisualeffectLockKey(dataset string, entry autogenManifestEntry) string {
return dataset + ":" + strings.TrimPrefix(filepath.ToSlash(strings.TrimSpace(entry.Source)), "vfxs/")
}
func hasKey(m map[string]int, key string) bool {
_, ok := m[key]
return ok
}
type accessoryVisualeffectsPolicy struct {
Groups map[string]accessoryVisualeffectsGroupPolicy
GroupTokenSource string GroupTokenSource string
CategoryFrom string CategoryFrom string
Delimiter string Delimiter string
@@ -785,22 +1159,22 @@ type headVisualeffectsPolicy struct {
RowDefaults map[string]string RowDefaults map[string]string
} }
type headVisualeffectsGroupPolicy struct { type accessoryVisualeffectsGroupPolicy struct {
Prefix string Prefix string
ModelColumns []string ModelColumns []string
RowDefaults map[string]string RowDefaults map[string]string
} }
func resolveHeadVisualeffectsPolicy(consumer project.AutogenConsumerConfig, manifestPolicy *project.HeadVisualeffectsConfig) headVisualeffectsPolicy { func resolveAccessoryVisualeffectsPolicy(consumer project.AutogenConsumerConfig, manifestPolicy *project.AccessoryVisualeffectsConfig) accessoryVisualeffectsPolicy {
policy := defaultHeadVisualeffectsPolicy() policy := defaultAccessoryVisualeffectsPolicy()
if manifestPolicy != nil { if manifestPolicy != nil {
applyHeadVisualeffectsConfig(&policy, *manifestPolicy) applyAccessoryVisualeffectsConfig(&policy, *manifestPolicy)
} }
applyHeadVisualeffectsConfig(&policy, consumer.HeadVisualeffects) applyAccessoryVisualeffectsConfig(&policy, consumer.AccessoryVisualeffects)
return policy return policy
} }
func applyHeadVisualeffectsConfig(policy *headVisualeffectsPolicy, cfg project.HeadVisualeffectsConfig) { func applyAccessoryVisualeffectsConfig(policy *accessoryVisualeffectsPolicy, cfg project.AccessoryVisualeffectsConfig) {
for group, groupCfg := range cfg.Groups { for group, groupCfg := range cfg.Groups {
group = strings.TrimSpace(group) group = strings.TrimSpace(group)
prefix := strings.TrimSpace(groupCfg.Prefix) prefix := strings.TrimSpace(groupCfg.Prefix)
@@ -811,13 +1185,13 @@ func applyHeadVisualeffectsConfig(policy *headVisualeffectsPolicy, cfg project.H
if prefix != "" { if prefix != "" {
groupPolicy.Prefix = prefix groupPolicy.Prefix = prefix
} }
if columns := normalizeHeadVisualeffectsModelColumns(groupCfg.ModelColumn, groupCfg.ModelColumns); len(columns) > 0 { if columns := normalizeAccessoryVisualeffectsModelColumns(groupCfg.ModelColumn, groupCfg.ModelColumns); len(columns) > 0 {
groupPolicy.ModelColumns = columns groupPolicy.ModelColumns = columns
} }
if groupPolicy.RowDefaults == nil { if groupPolicy.RowDefaults == nil {
groupPolicy.RowDefaults = map[string]string{} groupPolicy.RowDefaults = map[string]string{}
} }
mergeStringMap(groupPolicy.RowDefaults, normalizeHeadVisualeffectsRowDefaults(groupCfg.RowDefaults)) mergeStringMap(groupPolicy.RowDefaults, normalizeAccessoryVisualeffectsRowDefaults(groupCfg.RowDefaults))
policy.Groups[group] = groupPolicy policy.Groups[group] = groupPolicy
} }
if strings.TrimSpace(cfg.GroupTokenSource) != "" { if strings.TrimSpace(cfg.GroupTokenSource) != "" {
@@ -844,12 +1218,12 @@ func applyHeadVisualeffectsConfig(policy *headVisualeffectsPolicy, cfg project.H
if strings.TrimSpace(cfg.ModelColumn) != "" { if strings.TrimSpace(cfg.ModelColumn) != "" {
policy.ModelColumn = strings.TrimSpace(cfg.ModelColumn) policy.ModelColumn = strings.TrimSpace(cfg.ModelColumn)
} }
mergeStringMap(policy.RowDefaults, normalizeHeadVisualeffectsRowDefaults(cfg.RowDefaults)) mergeStringMap(policy.RowDefaults, normalizeAccessoryVisualeffectsRowDefaults(cfg.RowDefaults))
} }
func defaultHeadVisualeffectsPolicy() headVisualeffectsPolicy { func defaultAccessoryVisualeffectsPolicy() accessoryVisualeffectsPolicy {
return headVisualeffectsPolicy{ return accessoryVisualeffectsPolicy{
Groups: map[string]headVisualeffectsGroupPolicy{ Groups: map[string]accessoryVisualeffectsGroupPolicy{
"chest_accessories": {}, "chest_accessories": {},
"head_accessories": {}, "head_accessories": {},
"head_decorations": {}, "head_decorations": {},
@@ -871,7 +1245,7 @@ func defaultHeadVisualeffectsPolicy() headVisualeffectsPolicy {
} }
} }
func normalizeHeadVisualeffectsModelColumns(modelColumn string, modelColumns []string) []string { func normalizeAccessoryVisualeffectsModelColumns(modelColumn string, modelColumns []string) []string {
var result []string var result []string
if column := strings.TrimSpace(modelColumn); column != "" { if column := strings.TrimSpace(modelColumn); column != "" {
result = append(result, column) result = append(result, column)
@@ -888,7 +1262,7 @@ func normalizeHeadVisualeffectsModelColumns(modelColumn string, modelColumns []s
return result return result
} }
func normalizeHeadVisualeffectsRowDefaults(defaults map[string]string) map[string]string { func normalizeAccessoryVisualeffectsRowDefaults(defaults map[string]string) map[string]string {
result := map[string]string{} result := map[string]string{}
for column, value := range defaults { for column, value := range defaults {
column = strings.TrimSpace(column) column = strings.TrimSpace(column)
@@ -906,44 +1280,44 @@ func mergeStringMap(target map[string]string, source map[string]string) {
} }
} }
func headVisualeffectIdentity(dataset string, entry autogenManifestEntry, policy headVisualeffectsPolicy) (string, string, string, headVisualeffectsGroupPolicy, bool) { func accessoryVisualeffectIdentity(dataset string, entry autogenManifestEntry, policy accessoryVisualeffectsPolicy) (string, string, string, accessoryVisualeffectsGroupPolicy, bool) {
entry = normalizeHeadVisualeffectEntry(entry) entry = normalizeAccessoryVisualeffectEntry(entry)
group, ok := policy.Groups[entry.Group] group, ok := policy.Groups[entry.Group]
if !ok || strings.TrimSpace(entry.ModelStem) == "" { if !ok || strings.TrimSpace(entry.ModelStem) == "" {
return "", "", "", headVisualeffectsGroupPolicy{}, false return "", "", "", accessoryVisualeffectsGroupPolicy{}, false
} }
modelStem := strings.TrimSpace(entry.ModelStem) modelStem := strings.TrimSpace(entry.ModelStem)
stem := stripHeadVisualeffectModelPrefix(modelStem, policy) stem := stripAccessoryVisualeffectModelPrefix(modelStem, policy)
stem = strings.TrimSpace(stem) stem = strings.TrimSpace(stem)
if stem == "" { if stem == "" {
return "", "", "", headVisualeffectsGroupPolicy{}, false return "", "", "", accessoryVisualeffectsGroupPolicy{}, false
} }
category := headVisualeffectCategory(entry, policy) category := accessoryVisualeffectCategory(entry, policy)
groupToken := headVisualeffectGroupToken(entry.Group, group, policy) groupToken := accessoryVisualeffectGroupToken(entry.Group, group, policy)
values := map[string]string{ values := map[string]string{
"dataset": dataset, "dataset": dataset,
"group": applyHeadVisualeffectCase(groupToken, policy), "group": applyAccessoryVisualeffectCase(groupToken, policy),
"group_raw": entry.Group, "group_raw": entry.Group,
"prefix": applyHeadVisualeffectCase(group.Prefix, policy), "prefix": applyAccessoryVisualeffectCase(group.Prefix, policy),
"category": applyHeadVisualeffectCase(category, policy), "category": applyAccessoryVisualeffectCase(category, policy),
"category_upper": strings.ToUpper(category), "category_upper": strings.ToUpper(category),
"category_segment": headVisualeffectDelimitedSegment(applyHeadVisualeffectCase(category, policy), policy.Delimiter), "category_segment": accessoryVisualeffectDelimitedSegment(applyAccessoryVisualeffectCase(category, policy), policy.Delimiter),
"category_segment_upper": headVisualeffectDelimitedSegment(strings.ToUpper(category), policy.Delimiter), "category_segment_upper": accessoryVisualeffectDelimitedSegment(strings.ToUpper(category), policy.Delimiter),
"subgroup": applyHeadVisualeffectCase(entry.Subgroup, policy), "subgroup": applyAccessoryVisualeffectCase(entry.Subgroup, policy),
"stem": applyHeadVisualeffectCase(stem, policy), "stem": applyAccessoryVisualeffectCase(stem, policy),
"stem_upper": strings.ToUpper(stem), "stem_upper": strings.ToUpper(stem),
"model_stem": modelStem, "model_stem": modelStem,
"delimiter": policy.Delimiter, "delimiter": policy.Delimiter,
} }
key := expandHeadVisualeffectsFormat(policy.KeyFormat, values) key := expandAccessoryVisualeffectsFormat(policy.KeyFormat, values)
label := expandHeadVisualeffectsFormat(policy.LabelFormat, values) label := expandAccessoryVisualeffectsFormat(policy.LabelFormat, values)
if strings.TrimSpace(key) == "" || strings.TrimSpace(label) == "" { if strings.TrimSpace(key) == "" || strings.TrimSpace(label) == "" {
return "", "", "", headVisualeffectsGroupPolicy{}, false return "", "", "", accessoryVisualeffectsGroupPolicy{}, false
} }
return key, label, modelStem, group, true return key, label, modelStem, group, true
} }
func normalizeHeadVisualeffectEntry(entry autogenManifestEntry) autogenManifestEntry { func normalizeAccessoryVisualeffectEntry(entry autogenManifestEntry) autogenManifestEntry {
entry.Group = strings.TrimSpace(entry.Group) entry.Group = strings.TrimSpace(entry.Group)
if strings.TrimSpace(entry.Group) == "" && strings.TrimSpace(entry.Source) != "" { if strings.TrimSpace(entry.Group) == "" && strings.TrimSpace(entry.Source) != "" {
parts := strings.Split(filepath.ToSlash(entry.Source), "/") parts := strings.Split(filepath.ToSlash(entry.Source), "/")
@@ -966,7 +1340,7 @@ func normalizeHeadVisualeffectEntry(entry autogenManifestEntry) autogenManifestE
return entry return entry
} }
func stripHeadVisualeffectModelPrefix(modelStem string, policy headVisualeffectsPolicy) string { func stripAccessoryVisualeffectModelPrefix(modelStem string, policy accessoryVisualeffectsPolicy) string {
stem := modelStem stem := modelStem
for _, prefix := range policy.StripModelPrefixes { for _, prefix := range policy.StripModelPrefixes {
prefix = strings.TrimSpace(prefix) prefix = strings.TrimSpace(prefix)
@@ -978,7 +1352,7 @@ func stripHeadVisualeffectModelPrefix(modelStem string, policy headVisualeffects
return stem return stem
} }
func headVisualeffectGroupToken(groupName string, group headVisualeffectsGroupPolicy, policy headVisualeffectsPolicy) string { func accessoryVisualeffectGroupToken(groupName string, group accessoryVisualeffectsGroupPolicy, policy accessoryVisualeffectsPolicy) string {
if policy.GroupTokenSource == "folder_name" { if policy.GroupTokenSource == "folder_name" {
return groupName return groupName
} }
@@ -988,7 +1362,7 @@ func headVisualeffectGroupToken(groupName string, group headVisualeffectsGroupPo
return groupName return groupName
} }
func headVisualeffectCategory(entry autogenManifestEntry, policy headVisualeffectsPolicy) string { func accessoryVisualeffectCategory(entry autogenManifestEntry, policy accessoryVisualeffectsPolicy) string {
switch policy.CategoryFrom { switch policy.CategoryFrom {
case "immediate_parent": case "immediate_parent":
return strings.TrimSpace(entry.Category) return strings.TrimSpace(entry.Category)
@@ -999,14 +1373,14 @@ func headVisualeffectCategory(entry autogenManifestEntry, policy headVisualeffec
} }
} }
func headVisualeffectDelimitedSegment(value, delimiter string) string { func accessoryVisualeffectDelimitedSegment(value, delimiter string) string {
if strings.TrimSpace(value) == "" { if strings.TrimSpace(value) == "" {
return "" return ""
} }
return value + delimiter return value + delimiter
} }
func applyHeadVisualeffectCase(value string, policy headVisualeffectsPolicy) string { func applyAccessoryVisualeffectCase(value string, policy accessoryVisualeffectsPolicy) string {
switch policy.Case { switch policy.Case {
case "lower": case "lower":
return strings.ToLower(value) return strings.ToLower(value)
@@ -1017,7 +1391,7 @@ func applyHeadVisualeffectCase(value string, policy headVisualeffectsPolicy) str
} }
} }
func expandHeadVisualeffectsFormat(format string, values map[string]string) string { func expandAccessoryVisualeffectsFormat(format string, values map[string]string) string {
result := format result := format
for key, value := range values { for key, value := range values {
result = strings.ReplaceAll(result, "{"+key+"}", value) result = strings.ReplaceAll(result, "{"+key+"}", value)
@@ -1025,7 +1399,7 @@ func expandHeadVisualeffectsFormat(format string, values map[string]string) stri
return result return result
} }
func createDefaultHeadVisualeffectRow(columns []string, rowID int, key, label, modelStem string, policy headVisualeffectsPolicy, group headVisualeffectsGroupPolicy) map[string]any { func createDefaultAccessoryVisualeffectRow(columns []string, rowID int, key, label, modelStem string, policy accessoryVisualeffectsPolicy, group accessoryVisualeffectsGroupPolicy) map[string]any {
row := map[string]any{ row := map[string]any{
"id": rowID, "id": rowID,
"key": key, "key": key,
@@ -1040,13 +1414,13 @@ func createDefaultHeadVisualeffectRow(columns []string, rowID int, key, label, m
for column, value := range group.RowDefaults { for column, value := range group.RowDefaults {
row[column] = value row[column] = value
} }
for _, column := range headVisualeffectModelColumns(policy, group) { for _, column := range accessoryVisualeffectModelColumns(policy, group) {
row[column] = modelStem row[column] = modelStem
} }
return row return row
} }
func applyDiscoveredHeadVisualeffectDefaults(row map[string]any, columns []string, modelStem, label string, policy headVisualeffectsPolicy, group headVisualeffectsGroupPolicy) { func applyDiscoveredAccessoryVisualeffectDefaults(row map[string]any, columns []string, modelStem, label string, policy accessoryVisualeffectsPolicy, group accessoryVisualeffectsGroupPolicy) {
if isNullLikeValue(row["Label"]) { if isNullLikeValue(row["Label"]) {
row["Label"] = label row["Label"] = label
} }
@@ -1060,7 +1434,7 @@ func applyDiscoveredHeadVisualeffectDefaults(row map[string]any, columns []strin
row[column] = value row[column] = value
} }
} }
for _, column := range headVisualeffectModelColumns(policy, group) { for _, column := range accessoryVisualeffectModelColumns(policy, group) {
if isNullLikeValue(row[column]) { if isNullLikeValue(row[column]) {
row[column] = modelStem row[column] = modelStem
} }
@@ -1072,7 +1446,7 @@ func applyDiscoveredHeadVisualeffectDefaults(row map[string]any, columns []strin
} }
} }
func headVisualeffectModelColumns(policy headVisualeffectsPolicy, group headVisualeffectsGroupPolicy) []string { func accessoryVisualeffectModelColumns(policy accessoryVisualeffectsPolicy, group accessoryVisualeffectsGroupPolicy) []string {
if len(group.ModelColumns) > 0 { if len(group.ModelColumns) > 0 {
return group.ModelColumns return group.ModelColumns
} }
@@ -1090,14 +1464,14 @@ func formatAutogenManifest(root string, producer project.AutogenProducerConfig,
GeneratedAt: time.Now().UTC().Format(time.RFC3339), GeneratedAt: time.Now().UTC().Format(time.RFC3339),
Entries: entries, Entries: entries,
} }
if autogenHeadVisualeffectsConfigConfigured(producer.HeadVisualeffects) { if autogenAccessoryVisualeffectsConfigConfigured(producer.AccessoryVisualeffects) {
cfg := producer.HeadVisualeffects cfg := producer.AccessoryVisualeffects
manifest.HeadVisualeffects = &cfg manifest.AccessoryVisualeffects = &cfg
} }
return json.MarshalIndent(manifest, "", " ") return json.MarshalIndent(manifest, "", " ")
} }
func autogenHeadVisualeffectsConfigConfigured(cfg project.HeadVisualeffectsConfig) bool { func autogenAccessoryVisualeffectsConfigConfigured(cfg project.AccessoryVisualeffectsConfig) bool {
return len(cfg.Groups) > 0 || return len(cfg.Groups) > 0 ||
strings.TrimSpace(cfg.GroupTokenSource) != "" || strings.TrimSpace(cfg.GroupTokenSource) != "" ||
strings.TrimSpace(cfg.CategoryFrom) != "" || strings.TrimSpace(cfg.CategoryFrom) != "" ||
@@ -0,0 +1,199 @@
package topdata
import (
"errors"
"fmt"
"net/http"
"net/http/httptest"
"path/filepath"
"testing"
"git.westgate.pw/ShadowsOverWestgate/sow-tools/internal/project"
)
func cdnConsumer() project.AutogenConsumerConfig {
return project.AutogenConsumerConfig{
ID: "accessory_visualeffects", Producer: "accessory_visualeffects",
Dataset: "visualeffects", Mode: "accessory_visualeffects", Optional: true,
AccessoryVisualeffects: project.AccessoryVisualeffectsConfig{
Groups: map[string]project.AccessoryVisualeffectGroupConfig{
"head_accessories": {}, "chest_accessories": {},
"head_decorations": {}, "head_features": {},
},
},
Source: project.AutogenSourceConfig{
Kind: "cdn_channel",
ChannelsPath: "releases/haks/channels.json",
ManifestPath: "releases/haks/{tag}/vfxs.yml",
ReleaseMarkerPath: "releases/haks/{tag}/haks.json",
},
}
}
// cdnServer serves channels.json, vfxs.yml, and haks.json from in-memory maps.
// A key absent from the map returns 404.
func cdnServer(t *testing.T, files map[string]string) *httptest.Server {
t.Helper()
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
body, ok := files[r.URL.Path]
if !ok {
w.WriteHeader(http.StatusNotFound)
return
}
fmt.Fprint(w, body)
}))
t.Cleanup(srv.Close)
return srv
}
func TestResolveCDNChannelHappyPath(t *testing.T) {
root := testProjectRoot(t)
srv := cdnServer(t, map[string]string{
"/releases/haks/channels.json": `{"current":"v1.2.3"}`,
"/releases/haks/v1.2.3/vfxs.yml": `assets:
- path: vfxs/head_accessories/hat/hfx_bandana.mdl
restype: mdl
sha256: aaa
- path: vfxs/chest_accessories/cape/cfx_cloak.mdl
restype: mdl
sha256: bbb
- path: vfxs/head_accessories/hat/hfx_bandana.tga
restype: tga
sha256: ccc
- path: vfxs/weapons/sword.mdl
restype: mdl
sha256: ddd
`,
})
t.Setenv("BUNNY_CDN_BASE", srv.URL)
t.Setenv("SOW_TOPDATA_ASSET_CHANNEL", "current")
p := testProject(root)
c := cdnConsumer()
c.Source.CDNBaseEnv = "BUNNY_CDN_BASE"
c.Source.ChannelEnv = "SOW_TOPDATA_ASSET_CHANNEL"
m, err := resolveCDNChannelManifest(p, c, c.Source, nil)
if err != nil {
t.Fatalf("resolve failed: %v", err)
}
if len(m.Entries) != 2 {
t.Fatalf("want 2 mdl entries under target groups, got %d: %#v", len(m.Entries), m.Entries)
}
// Sorted by source: chest_accessories/... before head_accessories/...
if m.Entries[0].Source != "chest_accessories/cape/cfx_cloak.mdl" {
t.Fatalf("unexpected first entry: %#v", m.Entries[0])
}
if m.Entries[1].Source != "head_accessories/hat/hfx_bandana.mdl" ||
m.Entries[1].Group != "head_accessories" ||
m.Entries[1].Subgroup != "hat" ||
m.Entries[1].ModelStem != "hfx_bandana" {
t.Fatalf("unexpected derived entry: %#v", m.Entries[1])
}
if m.Ref != "v1.2.3" {
t.Fatalf("want ref v1.2.3, got %q", m.Ref)
}
}
func TestResolveCDNChannelFailOpen(t *testing.T) {
root := testProjectRoot(t)
c := cdnConsumer()
c.Source.CDNBaseEnv = "BUNNY_CDN_BASE"
c.Source.ChannelEnv = "SOW_TOPDATA_ASSET_CHANNEL"
t.Setenv("SOW_TOPDATA_ASSET_CHANNEL", "current")
t.Run("channels.json unreachable", func(t *testing.T) {
t.Setenv("BUNNY_CDN_BASE", "http://127.0.0.1:0") // unroutable
_, err := resolveCDNChannelManifest(testProject(root), c, c.Source, nil)
if err == nil || !errorIsUnavailable(err) {
t.Fatalf("want fail-open (unavailable), got %v", err)
}
})
t.Run("channel absent", func(t *testing.T) {
srv := cdnServer(t, map[string]string{"/releases/haks/channels.json": `{"testing":"v9"}`})
t.Setenv("BUNNY_CDN_BASE", srv.URL)
_, err := resolveCDNChannelManifest(testProject(root), c, c.Source, nil)
if err == nil || !errorIsUnavailable(err) {
t.Fatalf("want fail-open (unavailable), got %v", err)
}
})
t.Run("vfxs 404 no release marker", func(t *testing.T) {
srv := cdnServer(t, map[string]string{"/releases/haks/channels.json": `{"current":"v1"}`})
t.Setenv("BUNNY_CDN_BASE", srv.URL)
_, err := resolveCDNChannelManifest(testProject(root), c, c.Source, nil)
if err == nil || !errorIsUnavailable(err) {
t.Fatalf("want fail-open (unavailable), got %v", err)
}
})
}
func TestResolveCDNChannelHardFail(t *testing.T) {
root := testProjectRoot(t)
c := cdnConsumer()
c.Source.CDNBaseEnv = "BUNNY_CDN_BASE"
c.Source.ChannelEnv = "SOW_TOPDATA_ASSET_CHANNEL"
t.Setenv("SOW_TOPDATA_ASSET_CHANNEL", "current")
t.Run("broken release: vfxs 404 + haks.json present", func(t *testing.T) {
srv := cdnServer(t, map[string]string{
"/releases/haks/channels.json": `{"current":"v1"}`,
"/releases/haks/v1/haks.json": `{}`,
})
t.Setenv("BUNNY_CDN_BASE", srv.URL)
_, err := resolveCDNChannelManifest(testProject(root), c, c.Source, nil)
if err == nil || errorIsUnavailable(err) {
t.Fatalf("want HARD fail, got %v", err)
}
})
t.Run("malformed vfxs.yml: no assets array", func(t *testing.T) {
srv := cdnServer(t, map[string]string{
"/releases/haks/channels.json": `{"current":"v1"}`,
"/releases/haks/v1/vfxs.yml": "generated_at: 2026-01-01\n",
})
t.Setenv("BUNNY_CDN_BASE", srv.URL)
_, err := resolveCDNChannelManifest(testProject(root), c, c.Source, nil)
if err == nil || errorIsUnavailable(err) {
t.Fatalf("want HARD fail, got %v", err)
}
})
t.Run("malformed vfxs.yml: syntactically broken YAML", func(t *testing.T) {
srv := cdnServer(t, map[string]string{
"/releases/haks/channels.json": `{"current":"v1"}`,
"/releases/haks/v1/vfxs.yml": "assets:\n - path: [unterminated",
})
t.Setenv("BUNNY_CDN_BASE", srv.URL)
_, err := resolveCDNChannelManifest(testProject(root), c, c.Source, nil)
if err == nil || errorIsUnavailable(err) {
t.Fatalf("want HARD fail on broken YAML, got %v", err)
}
})
}
func TestResolveCDNChannelOfflineOverride(t *testing.T) {
root := testProjectRoot(t)
vfxs := filepath.Join(root, "vfxs.yml")
writeFile(t, vfxs, `assets:
- path: vfxs/head_features/scar/hfx_scar.mdl
restype: mdl
sha256: zzz
`)
t.Setenv("SOW_VFXS_MANIFEST", vfxs)
c := cdnConsumer()
c.Source.OfflineOverrideEnv = "SOW_VFXS_MANIFEST"
m, err := resolveCDNChannelManifest(testProject(root), c, c.Source, nil)
if err != nil {
t.Fatalf("offline override failed: %v", err)
}
if len(m.Entries) != 1 || m.Entries[0].Source != "head_features/scar/hfx_scar.mdl" {
t.Fatalf("unexpected offline entries: %#v", m.Entries)
}
}
func errorIsUnavailable(err error) bool {
return errors.Is(err, errAutogenManifestUnavailable)
}
@@ -0,0 +1,114 @@
package topdata
import (
"testing"
"git.westgate.pw/ShadowsOverWestgate/sow-tools/internal/project"
)
func accConsumer() project.AutogenConsumerConfig {
return project.AutogenConsumerConfig{
ID: "accessory_visualeffects", Producer: "accessory_visualeffects",
Dataset: "visualeffects", Mode: "accessory_visualeffects", Optional: true,
AccessoryVisualeffects: project.AccessoryVisualeffectsConfig{
Groups: map[string]project.AccessoryVisualeffectGroupConfig{
"head_accessories": {ModelColumns: []string{"Imp_HeadCon_Node"}},
},
GroupTokenSource: "folder_name", CategoryFrom: "immediate_parent",
Delimiter: "/", Case: "preserve",
StripModelPrefixes: []string{"hfx_"},
KeyFormat: "{dataset}:{group}{delimiter}{category_segment}{stem}",
LabelFormat: "{group}{delimiter}{category_segment}{stem}",
ModelColumn: "Imp_HeadCon_Node",
},
}
}
func vfxEntry() autogenManifestEntry {
return autogenManifestEntry{
Source: "head_accessories/hat/hfx_bandana.mdl", Group: "head_accessories",
Subgroup: "hat", ModelStem: "hfx_bandana",
}
}
func vfxDataset(lock map[string]int) nativeCollectedDataset {
return nativeCollectedDataset{
Dataset: nativeDataset{Name: "visualeffects", Kind: nativeDatasetBase},
Columns: []string{"Label", "Imp_HeadCon_Node"},
Rows: nil,
LockData: lock,
}
}
func TestAccessoryLockKeyIsModelAnchored(t *testing.T) {
got := accessoryVisualeffectLockKey("visualeffects", vfxEntry())
want := "visualeffects:head_accessories/hat/hfx_bandana.mdl"
if got != want {
t.Fatalf("lock key = %q, want %q", got, want)
}
}
// A config-only change (here: a different key_format) must NOT move the ID,
// because identity is the model path, not the presentation key.
func TestConfigOnlyChangeKeepsLockID(t *testing.T) {
lockKey := "visualeffects:head_accessories/hat/hfx_bandana.mdl"
collected := []nativeCollectedDataset{vfxDataset(map[string]int{lockKey: 555})}
c := accConsumer()
c.AccessoryVisualeffects.KeyFormat = "{dataset}:{stem}" // config-only change
got, err := augmentWithAutogeneratedAccessoryVisualeffects(collected, []autogenManifestEntry{vfxEntry()}, c, nil)
if err != nil {
t.Fatalf("augment failed: %v", err)
}
if got[0].LockData[lockKey] != 555 {
t.Fatalf("expected ID preserved at 555, got %#v", got[0].LockData)
}
}
// A model re-export (same path) keeps its ID — content change != identity change.
func TestModelReExportKeepsLockID(t *testing.T) {
lockKey := "visualeffects:head_accessories/hat/hfx_bandana.mdl"
collected := []nativeCollectedDataset{vfxDataset(map[string]int{lockKey: 777})}
got, err := augmentWithAutogeneratedAccessoryVisualeffects(collected, []autogenManifestEntry{vfxEntry()}, accConsumer(), nil)
if err != nil {
t.Fatalf("augment failed: %v", err)
}
if got[0].LockData[lockKey] != 777 {
t.Fatalf("expected ID 777 preserved, got %#v", got[0].LockData)
}
}
// A removed model frees its old ID; the surviving model keeps its lock entry.
func TestModelRemovalPrunesStaleLockKey(t *testing.T) {
keep := "visualeffects:head_accessories/hat/hfx_bandana.mdl"
gone := "visualeffects:head_accessories/hat/hfx_removed.mdl"
collected := []nativeCollectedDataset{vfxDataset(map[string]int{keep: 1, gone: 2})}
got, err := augmentWithAutogeneratedAccessoryVisualeffects(collected, []autogenManifestEntry{vfxEntry()}, accConsumer(), nil)
if err != nil {
t.Fatalf("augment failed: %v", err)
}
if _, present := got[0].LockData[gone]; present {
t.Fatalf("expected removed model's lock key pruned, got %#v", got[0].LockData)
}
if got[0].LockData[keep] != 1 {
t.Fatalf("expected surviving model ID 1 kept, got %#v", got[0].LockData)
}
}
// Cutover: a lock that only has the OLD config-derived key adopts that ID under
// the new model-path key, with zero shift, and prunes the old key.
func TestCutoverRemapPreservesID(t *testing.T) {
oldKey := "visualeffects:head_accessories/hat/bandana" // {dataset}:{group}/{category}/{stem-without-hfx_}
newKey := "visualeffects:head_accessories/hat/hfx_bandana.mdl"
collected := []nativeCollectedDataset{vfxDataset(map[string]int{oldKey: 4242})}
got, err := augmentWithAutogeneratedAccessoryVisualeffects(collected, []autogenManifestEntry{vfxEntry()}, accConsumer(), nil)
if err != nil {
t.Fatalf("augment failed: %v", err)
}
if got[0].LockData[newKey] != 4242 {
t.Fatalf("expected cutover to carry ID 4242 to new key, got %#v", got[0].LockData)
}
if _, present := got[0].LockData[oldKey]; present {
t.Fatalf("expected old config key pruned after cutover, got %#v", got[0].LockData)
}
}
@@ -0,0 +1,103 @@
package topdata
import (
"path/filepath"
"testing"
"git.westgate.pw/ShadowsOverWestgate/sow-tools/internal/project"
)
// The manifest_file branch reads a local autogenManifest JSON, augments the
// visualeffects dataset from its entries, and assigns lock IDs — no network.
func TestApplyAutogenConsumersReadsManifestFile(t *testing.T) {
root := testProjectRoot(t)
manifestPath := filepath.Join(root, ".cache", "sow-accessory-vfx-manifest.json")
mkdirAll(t, filepath.Join(root, ".cache"))
writeFile(t, manifestPath, `{
"id": "accessory_visualeffects",
"repo": "ShadowsOverWestgate/sow-assets-manifest",
"ref": "v1.2.3",
"generated_at": "2026-06-21T00:00:00Z",
"entries": [
{"source": "head_accessories/hat/hfx_bandana.mdl", "model_stem": "hfx_bandana", "group": "head_accessories", "subgroup": "hat"}
]
}`+"\n")
p := testProject(root)
p.Config.Autogen.Consumers = []project.AutogenConsumerConfig{
{
ID: "accessory_visualeffects",
Producer: "accessory_visualeffects",
Dataset: "visualeffects",
Mode: "accessory_visualeffects",
Optional: true,
ManifestFile: ".cache/sow-accessory-vfx-manifest.json",
},
}
collected := []nativeCollectedDataset{
{
Dataset: nativeDataset{Name: "visualeffects", Kind: nativeDatasetBase},
Columns: []string{"Label", "Type_FD", "OrientWithGround", "Imp_HeadCon_Node", "OrientWithObject"},
Rows: nil,
LockData: map[string]int{
"visualeffects:existing": 17,
},
},
}
got, err := applyAutogenConsumers(p, collected, nil)
if err != nil {
t.Fatalf("applyAutogenConsumers failed: %v", err)
}
if len(got) != 1 || len(got[0].Rows) != 1 {
t.Fatalf("expected one autogenerated row, got %#v", got)
}
row := got[0].Rows[0]
if row["key"] != "visualeffects:head_accessories/hat/bandana" || row["Imp_HeadCon_Node"] != "hfx_bandana" {
t.Fatalf("unexpected generated row: %#v", row)
}
if _, ok := got[0].LockData["visualeffects:head_accessories/hat/hfx_bandana.mdl"]; !ok {
t.Fatalf("expected lock id for generated accessory, got %#v", got[0].LockData)
}
// Authored key untouched.
if got[0].LockData["visualeffects:existing"] != 17 {
t.Fatalf("expected authored lock id retained at 17, got %#v", got[0].LockData)
}
}
// A missing manifest_file is an ignorable-optional condition: the optional
// consumer fails open (no rows) and preserves existing lock IDs.
func TestApplyAutogenConsumersMissingManifestFileFailsOpen(t *testing.T) {
root := testProjectRoot(t)
lockPath := filepath.Join(root, "visualeffects-lock.json")
writeFile(t, lockPath, `{"visualeffects:head_accessories/hat/bandana":10101}`+"\n")
p := testProject(root)
p.Config.Autogen.Consumers = []project.AutogenConsumerConfig{
{
ID: "accessory_visualeffects",
Producer: "accessory_visualeffects",
Dataset: "visualeffects",
Mode: "accessory_visualeffects",
Optional: true,
ManifestFile: ".cache/does-not-exist.json",
},
}
collected := []nativeCollectedDataset{
{
Dataset: nativeDataset{Name: "visualeffects", Kind: nativeDatasetBase, LockPath: lockPath},
Columns: []string{"Label"},
Rows: nil,
LockData: map[string]int{},
},
}
got, err := applyAutogenConsumers(p, collected, nil)
if err != nil {
t.Fatalf("expected fail-open, got error: %v", err)
}
if id, ok := got[0].LockData["visualeffects:head_accessories/hat/bandana"]; !ok || id != 10101 {
t.Fatalf("expected preserved lock id 10101, got %v (present=%v)", id, ok)
}
}
+109
View File
@@ -0,0 +1,109 @@
package topdata
import (
"net/http"
"net/http/httptest"
"path/filepath"
"testing"
"git.westgate.pw/ShadowsOverWestgate/sow-tools/internal/project"
)
// When an optional autogen consumer's released manifest cannot be fetched, the
// build must fail open: it keeps the consumer's already-pinned lock entries
// (so their IDs are not freed and reshuffled later) and generates no new rows.
func TestApplyAutogenConsumersPreservesLockEntriesWhenManifestUnavailable(t *testing.T) {
root := testProjectRoot(t)
// 404 for every request → released manifest is unavailable (not empty).
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
http.NotFound(w, r)
}))
t.Cleanup(srv.Close)
t.Setenv("SOW_ASSETS_SERVER_URL", srv.URL)
t.Setenv("SOW_ASSETS_REPO", "ShadowsOverWestgate/sow-assets")
p := testProject(root)
p.Config.Autogen.Consumers = []project.AutogenConsumerConfig{
{
ID: "accessory_visualeffects",
Producer: "accessory_visualeffects",
Dataset: "visualeffects",
Mode: "accessory_visualeffects",
Optional: true,
Root: "vfxs",
Include: []string{"head_accessories/**/*.mdl"},
Derive: project.AutogenDeriveConfig{Kind: "model_stem", GroupFrom: "first_path_segment"},
Manifest: project.AutogenManifestConfig{ReleaseTag: "head-vfx-manifest-current", AssetName: "sow-accessory-vfx-manifest.json", CacheName: "sow-accessory-vfx-manifest.json"},
},
}
// On-disk lock pins an autogen-owned accessory key plus an authored key.
lockPath := filepath.Join(root, "visualeffects-lock.json")
writeFile(t, lockPath, `{"visualeffects:existing":17,"visualeffects:head_accessories/hat/bandana":10101}`+"\n")
// Simulate post-prune collected state: the accessory key has already been
// dropped from in-memory LockData (as pruneLockDataToActiveRows would do).
collected := []nativeCollectedDataset{
{
Dataset: nativeDataset{Name: "visualeffects", Kind: nativeDatasetBase, LockPath: lockPath},
Columns: []string{"Label"},
Rows: []map[string]any{{"id": 17, "key": "visualeffects:existing"}},
LockData: map[string]int{"visualeffects:existing": 17},
},
}
got, err := applyAutogenConsumers(p, collected, nil)
if err != nil {
t.Fatalf("expected fail-open build, got error: %v", err)
}
if len(got) != 1 {
t.Fatalf("expected one dataset, got %d", len(got))
}
// The pinned accessory id must survive with its exact value (no jumble).
if id, ok := got[0].LockData["visualeffects:head_accessories/hat/bandana"]; !ok || id != 10101 {
t.Fatalf("expected preserved accessory lock id 10101, got %v (present=%v) lock=%#v", id, ok, got[0].LockData)
}
// The authored key is untouched.
if id, ok := got[0].LockData["visualeffects:existing"]; !ok || id != 17 {
t.Fatalf("expected authored key retained at 17, got %#v", got[0].LockData)
}
// No row is generated for the unavailable accessory entry.
for _, row := range got[0].Rows {
if key, _ := row["key"].(string); key == "visualeffects:head_accessories/hat/bandana" {
t.Fatalf("expected no generated row while manifest unavailable, got %#v", row)
}
}
}
// A key that is NOT owned by the consumer (an authored row legitimately removed)
// must not be resurrected by the preservation path.
func TestPreserveAutogenConsumerLockEntriesIgnoresUnownedKeys(t *testing.T) {
root := testProjectRoot(t)
lockPath := filepath.Join(root, "visualeffects-lock.json")
writeFile(t, lockPath, `{"visualeffects:retired_authored":3,"visualeffects:head_accessories/hat/bandana":10101}`+"\n")
consumer := project.AutogenConsumerConfig{
ID: "accessory_visualeffects",
Dataset: "visualeffects",
Mode: "accessory_visualeffects",
}
collected := []nativeCollectedDataset{
{
Dataset: nativeDataset{Name: "visualeffects", Kind: nativeDatasetBase, LockPath: lockPath},
LockData: map[string]int{},
},
}
got, err := preserveAutogenConsumerLockEntries(collected, consumer)
if err != nil {
t.Fatalf("preserveAutogenConsumerLockEntries failed: %v", err)
}
if _, ok := got[0].LockData["visualeffects:head_accessories/hat/bandana"]; !ok {
t.Fatalf("expected owned accessory key preserved, got %#v", got[0].LockData)
}
if _, ok := got[0].LockData["visualeffects:retired_authored"]; ok {
t.Fatalf("expected unowned authored key NOT resurrected, got %#v", got[0].LockData)
}
}
+10 -6
View File
@@ -21,16 +21,20 @@ type damagetypesRegistry struct {
Types []map[string]any Types []map[string]any
} }
func collectGeneratedRegistryDatasets(dataDir string) ([]nativeCollectedDataset, error) { // collectGeneratedRegistryDatasets projects the registry datasets. When
itempropsDatasets, err := collectItempropsRegistryDatasets(dataDir) // persistLocks is true (the build pipeline) newly allocated ids are written
// back to the source lockfiles; when false (validation, discovery, queries)
// collection is read-only and never touches the source tree.
func collectGeneratedRegistryDatasets(dataDir string, persistLocks bool) ([]nativeCollectedDataset, error) {
itempropsDatasets, err := collectItempropsRegistryDatasets(dataDir, persistLocks)
if err != nil { if err != nil {
return nil, err return nil, err
} }
damagetypeDatasets, err := collectDamagetypesRegistryDatasets(dataDir) damagetypeDatasets, err := collectDamagetypesRegistryDatasets(dataDir, persistLocks)
if err != nil { if err != nil {
return nil, err return nil, err
} }
racialtypesDatasets, err := collectRacialtypesRegistryDatasets(dataDir) racialtypesDatasets, err := collectRacialtypesRegistryDatasets(dataDir, persistLocks)
if err != nil { if err != nil {
return nil, err return nil, err
} }
@@ -70,7 +74,7 @@ func loadDamagetypesRegistry(dataDir string) (*damagetypesRegistry, error) {
}, nil }, nil
} }
func collectDamagetypesRegistryDatasets(dataDir string) ([]nativeCollectedDataset, error) { func collectDamagetypesRegistryDatasets(dataDir string, persistLocks bool) ([]nativeCollectedDataset, error) {
registry, err := loadDamagetypesRegistry(dataDir) registry, err := loadDamagetypesRegistry(dataDir)
if err != nil { if err != nil {
return nil, err return nil, err
@@ -83,7 +87,7 @@ func collectDamagetypesRegistryDatasets(dataDir string) ([]nativeCollectedDatase
if err != nil { if err != nil {
return nil, err return nil, err
} }
if lockModified { if lockModified && persistLocks {
if err := saveLockfile(registry.LockPath, registry.LockData); err != nil { if err := saveLockfile(registry.LockPath, registry.LockData); err != nil {
return nil, err return nil, err
} }
+136 -11
View File
@@ -203,8 +203,23 @@ func editorConfigPatternMatches(pattern, relativePath string) (bool, error) {
} }
func editorConfigGlobRegexp(pattern string) (string, error) { func editorConfigGlobRegexp(pattern string) (string, error) {
body, err := translateEditorConfigGlobSegment(pattern)
if err != nil {
return "", err
}
expression := "^" + body + "$"
if _, err := regexp.Compile(expression); err != nil {
return "", err
}
return expression, nil
}
// translateEditorConfigGlobSegment converts an editorconfig glob into a regular
// expression body. In addition to *, **, and ?, it implements brace expansion:
// comma lists ({a,b,c}) become alternations and numeric ranges ({n..m}) expand
// to the integers in the range, matching the editorconfig specification.
func translateEditorConfigGlobSegment(pattern string) (string, error) {
var builder strings.Builder var builder strings.Builder
builder.WriteByte('^')
for index := 0; index < len(pattern); { for index := 0; index < len(pattern); {
char := pattern[index] char := pattern[index]
switch char { switch char {
@@ -220,22 +235,132 @@ func editorConfigGlobRegexp(pattern string) (string, error) {
continue continue
} }
builder.WriteString("[^/]*") builder.WriteString("[^/]*")
index++
case '?': case '?':
builder.WriteString("[^/]") builder.WriteString("[^/]")
case '.', '+', '(', ')', '|', '^', '$', '{', '}', '[', ']', '\\':
builder.WriteByte('\\')
builder.WriteByte(char)
default:
builder.WriteByte(char)
}
index++ index++
case '{':
end := matchingEditorConfigBrace(pattern, index)
if end < 0 {
// An unbalanced brace is matched literally.
builder.WriteString("\\{")
index++
continue
} }
builder.WriteByte('$') expansion, err := expandEditorConfigBrace(pattern[index+1 : end])
expression := builder.String() if err != nil {
if _, err := regexp.Compile(expression); err != nil {
return "", err return "", err
} }
return expression, nil builder.WriteString(expansion)
index = end + 1
case '.', '+', '(', ')', '|', '^', '$', '}', '[', ']', '\\':
builder.WriteByte('\\')
builder.WriteByte(char)
index++
default:
builder.WriteByte(char)
index++
}
}
return builder.String(), nil
}
// matchingEditorConfigBrace returns the index of the '}' that closes the '{' at
// open, accounting for nesting, or -1 when the braces are unbalanced.
func matchingEditorConfigBrace(pattern string, open int) int {
depth := 0
for index := open; index < len(pattern); index++ {
switch pattern[index] {
case '{':
depth++
case '}':
depth--
if depth == 0 {
return index
}
}
}
return -1
}
// expandEditorConfigBrace turns the body of a brace expression into a regex
// fragment. A comma list becomes an alternation; a lone "n..m" body becomes a
// numeric range. A body with neither comma nor range is treated as literal
// braces, matching the editorconfig reference implementation.
func expandEditorConfigBrace(inner string) (string, error) {
options := splitTopLevelBraceCommas(inner)
if len(options) == 1 {
if rangeExpr, ok, err := editorConfigNumericRange(inner); err != nil {
return "", err
} else if ok {
return rangeExpr, nil
}
segment, err := translateEditorConfigGlobSegment(inner)
if err != nil {
return "", err
}
return "\\{" + segment + "\\}", nil
}
parts := make([]string, 0, len(options))
for _, option := range options {
segment, err := translateEditorConfigGlobSegment(option)
if err != nil {
return "", err
}
parts = append(parts, segment)
}
return "(?:" + strings.Join(parts, "|") + ")", nil
}
// splitTopLevelBraceCommas splits a brace body on commas that are not nested
// inside an inner brace group.
func splitTopLevelBraceCommas(inner string) []string {
parts := []string{}
depth := 0
start := 0
for index := 0; index < len(inner); index++ {
switch inner[index] {
case '{':
depth++
case '}':
depth--
case ',':
if depth == 0 {
parts = append(parts, inner[start:index])
start = index + 1
}
}
}
return append(parts, inner[start:])
}
// editorConfigNumericRange expands "n..m" into an alternation of the integers
// in [n, m]. It returns ok=false when the body is not a numeric range.
func editorConfigNumericRange(inner string) (string, bool, error) {
separator := strings.Index(inner, "..")
if separator < 0 {
return "", false, nil
}
low, err := strconv.Atoi(strings.TrimSpace(inner[:separator]))
if err != nil {
return "", false, nil
}
high, err := strconv.Atoi(strings.TrimSpace(inner[separator+2:]))
if err != nil {
return "", false, nil
}
if low > high {
low, high = high, low
}
const maxRangeSize = 65536
if high-low >= maxRangeSize {
return "", false, fmt.Errorf("editorconfig numeric range {%s} is too large", inner)
}
parts := make([]string, 0, high-low+1)
for value := low; value <= high; value++ {
parts = append(parts, strconv.Itoa(value))
}
return "(?:" + strings.Join(parts, "|") + ")", true, nil
} }
func pathBase(path string) string { func pathBase(path string) string {
@@ -0,0 +1,56 @@
package topdata
import (
"os"
"path/filepath"
"testing"
)
func TestEditorConfigPatternMatchesBraceExpansion(t *testing.T) {
cases := []struct {
pattern string
path string
want bool
}{
{"*.{json,jsonc}", "lock.json", true},
{"*.{json,jsonc}", "types.jsonc", true},
{"*.{json,jsonc}", "notes.md", false},
{"*.{js,ts,tsx}", "main.tsx", true},
{"{foo,bar}.txt", "bar.txt", true},
{"{foo,bar}.txt", "baz.txt", false},
{"page{1..3}.json", "page2.json", true},
{"page{1..3}.json", "page4.json", false},
// No comma and not a range: editorconfig treats braces literally.
{"{single}.txt", "{single}.txt", true},
{"{single}.txt", "single.txt", false},
}
for _, c := range cases {
got, err := editorConfigPatternMatches(c.pattern, c.path)
if err != nil {
t.Fatalf("pattern %q path %q: %v", c.pattern, c.path, err)
}
if got != c.want {
t.Errorf("editorConfigPatternMatches(%q, %q) = %v, want %v", c.pattern, c.path, got, c.want)
}
}
}
func TestSaveLockfileHonorsBraceGlobIndentSize(t *testing.T) {
root := t.TempDir()
writeFile(t, filepath.Join(root, ".editorconfig"), "root = true\n\n[*]\nindent_style = space\nindent_size = 2\n\n[*.{json,jsonc}]\nindent_size = 4\n")
lockPath := filepath.Join(root, "data", "damagetypes", "registry", "lock.json")
mkdirAll(t, filepath.Dir(lockPath))
writeFile(t, lockPath, "{\n \"damagetype:acid\": 4\n}\n")
if err := saveLockfile(lockPath, map[string]int{"damagetype:acid": 4}); err != nil {
t.Fatalf("saveLockfile: %v", err)
}
got, err := os.ReadFile(lockPath)
if err != nil {
t.Fatalf("read lockfile: %v", err)
}
want := "{\n \"damagetype:acid\": 4\n}\n"
if string(got) != want {
t.Fatalf("expected 4-space indent resolved from [*.{json,jsonc}]; got:\n%q", string(got))
}
}
+2 -2
View File
@@ -166,7 +166,7 @@ func loadRegistryRows(path string) ([]map[string]any, error) {
return rows, nil return rows, nil
} }
func collectItempropsRegistryDatasets(dataDir string) ([]nativeCollectedDataset, error) { func collectItempropsRegistryDatasets(dataDir string, persistLocks bool) ([]nativeCollectedDataset, error) {
registry, err := loadItempropsRegistry(dataDir) registry, err := loadItempropsRegistry(dataDir)
if err != nil { if err != nil {
return nil, err return nil, err
@@ -188,7 +188,7 @@ func collectItempropsRegistryDatasets(dataDir string) ([]nativeCollectedDataset,
return nil, err return nil, err
} }
lockModified = lockModified || costModified || paramModified lockModified = lockModified || costModified || paramModified
if lockModified { if lockModified && persistLocks {
if err := saveLockfile(registry.LockPath, registry.LockData); err != nil { if err := saveLockfile(registry.LockPath, registry.LockData); err != nil {
return nil, err return nil, err
} }
+29
View File
@@ -0,0 +1,29 @@
package topdata
import (
"os"
"path/filepath"
"testing"
)
func TestHasLFSPointerStubsDetectsStub(t *testing.T) {
root := t.TempDir()
if err := os.MkdirAll(filepath.Join(root, "a"), 0o755); err != nil {
t.Fatalf("mkdir: %v", err)
}
stub := "version https://git-lfs.github.com/spec/v1\noid sha256:abc\nsize 12345\n"
writeFile(t, filepath.Join(root, "a", "model.mdl"), stub)
writeFile(t, filepath.Join(root, "a", "real.txt"), "not a pointer, real bytes here")
if !hasLFSPointerStubs(root) {
t.Fatal("expected stub detection to be true")
}
}
func TestHasLFSPointerStubsNoStub(t *testing.T) {
root := t.TempDir()
writeFile(t, filepath.Join(root, "real.mdl"), "binary-ish content without a pointer header")
if hasLFSPointerStubs(root) {
t.Fatal("expected no stub detected")
}
}
+2 -2
View File
@@ -398,7 +398,7 @@ func buildNativeUnchecked(p *project.Project, opts NativeBuildOptions, progress
datasets = applyTopDataValueEncodings(datasets, p.Config.TopData.ValueEncodings) datasets = applyTopDataValueEncodings(datasets, p.Config.TopData.ValueEncodings)
datasets = applyTopDataValueDefaults(datasets, p.Config.TopData.ValueDefaults) datasets = applyTopDataValueDefaults(datasets, p.Config.TopData.ValueDefaults)
datasets = applyTopDataRowGeneration(datasets, p.Config.TopData.RowGeneration) datasets = applyTopDataRowGeneration(datasets, p.Config.TopData.RowGeneration)
registryDatasets, err := collectGeneratedRegistryDatasets(dataDir) registryDatasets, err := collectGeneratedRegistryDatasets(dataDir, true)
if err != nil { if err != nil {
return BuildResult{}, err return BuildResult{}, err
} }
@@ -1073,7 +1073,7 @@ func discoverNativeOutputCatalog(dataDir string) (map[string]string, error) {
if err != nil { if err != nil {
return nil, err return nil, err
} }
registryDatasets, err := collectGeneratedRegistryDatasets(dataDir) registryDatasets, err := collectGeneratedRegistryDatasets(dataDir, false)
if err != nil { if err != nil {
return nil, err return nil, err
} }
+56
View File
@@ -0,0 +1,56 @@
package topdata
import (
"testing"
"git.westgate.pw/ShadowsOverWestgate/sow-tools/internal/project"
)
// PARITY GUARD: a bare project (no manifest_file, no NWN_ROOT, no token) with a
// cdn_channel consumer must produce accessory rows. If accessory resolution ever
// drifts back out of Crucible into a wrapper, this fails.
func TestParityGuardCDNChannelProducesAccessoryRows(t *testing.T) {
root := testProjectRoot(t)
srv := cdnServer(t, map[string]string{
"/releases/haks/channels.json": `{"current":"v1.0.0"}`,
"/releases/haks/v1.0.0/vfxs.yml": `assets:
- path: vfxs/head_accessories/hat/hfx_bandana.mdl
restype: mdl
sha256: aaa
- path: vfxs/chest_accessories/cape/cfx_cloak.mdl
restype: mdl
sha256: bbb
`,
})
t.Setenv("BUNNY_CDN_BASE", srv.URL)
t.Setenv("SOW_TOPDATA_ASSET_CHANNEL", "current")
// Prove R3: no NWN_ROOT in the environment.
t.Setenv("NWN_ROOT", "")
c := cdnConsumer()
c.Source.CDNBaseEnv = "BUNNY_CDN_BASE"
c.Source.ChannelEnv = "SOW_TOPDATA_ASSET_CHANNEL"
c.AccessoryVisualeffects.ModelColumn = "Imp_HeadCon_Node"
p := testProject(root)
p.Config.Autogen.Consumers = []project.AutogenConsumerConfig{c}
collected := []nativeCollectedDataset{{
Dataset: nativeDataset{Name: "visualeffects", Kind: nativeDatasetBase},
Columns: []string{"Label", "Imp_HeadCon_Node", "Imp_Root_S_Node"},
Rows: nil,
LockData: map[string]int{},
}}
got, err := applyAutogenConsumers(p, collected, nil)
if err != nil {
t.Fatalf("applyAutogenConsumers failed: %v", err)
}
if len(got[0].Rows) != 2 {
t.Fatalf("PARITY GUARD: expected 2 accessory rows, got %d (%#v)", len(got[0].Rows), got[0].Rows)
}
wantKey := "visualeffects:head_accessories/hat/hfx_bandana.mdl"
if _, ok := got[0].LockData[wantKey]; !ok {
t.Fatalf("PARITY GUARD: expected lock id for %s, got %#v", wantKey, got[0].LockData)
}
}
+13
View File
@@ -274,6 +274,19 @@ func writePartsManifestCache(path string, manifest *partsManifest) error {
return nil return nil
} }
// httpGetStatus performs an anonymous GET and returns the status code plus the
// body (capped). No auth header: cdn_channel inputs (channels.json, vfxs.yml,
// haks.json) are public CDN text — R1.
func httpGetStatus(target string) (int, []byte, error) {
resp, err := partsManifestHTTPClient.Get(target)
if err != nil {
return 0, nil, err
}
defer resp.Body.Close()
body, _ := io.ReadAll(io.LimitReader(resp.Body, 8<<20))
return resp.StatusCode, body, nil
}
func inventoryFromPartsManifest(manifest *partsManifest) map[string]map[int]struct{} { func inventoryFromPartsManifest(manifest *partsManifest) map[string]map[int]struct{} {
result := make(map[string]map[int]struct{}, len(supportedPartCategories)) result := make(map[string]map[int]struct{}, len(supportedPartCategories))
for _, category := range supportedPartCategories { for _, category := range supportedPartCategories {
+2 -2
View File
@@ -120,7 +120,7 @@ func loadRacialtypesRegistryRaceRows(dir string) ([]map[string]any, error) {
return rows, nil return rows, nil
} }
func collectRacialtypesRegistryDatasets(dataDir string) ([]nativeCollectedDataset, error) { func collectRacialtypesRegistryDatasets(dataDir string, persistLocks bool) ([]nativeCollectedDataset, error) {
registry, err := loadRacialtypesRegistry(dataDir) registry, err := loadRacialtypesRegistry(dataDir)
if err != nil { if err != nil {
return nil, err return nil, err
@@ -138,7 +138,7 @@ func collectRacialtypesRegistryDatasets(dataDir string) ([]nativeCollectedDatase
return nil, err return nil, err
} }
lockModified = lockModified || raceLockModified lockModified = lockModified || raceLockModified
if lockModified { if lockModified && persistLocks {
if err := saveLockfile(registry.LockPath, registry.LockData); err != nil { if err := saveLockfile(registry.LockPath, registry.LockData); err != nil {
return nil, err return nil, err
} }
@@ -0,0 +1,64 @@
package topdata
import (
"os"
"path/filepath"
"testing"
)
const damagetypesRegistryTestTypes = `{
"rows": [
{"id": 0, "key": "damagetype:bludgeoning", "label": "Bludgeoning", "group_label": "Physical", "damage_type_group": 0},
{"id": 1, "key": "damagetype:piercing", "label": "Piercing", "group_label": "Physical", "damage_type_group": 0}
]
}
`
// validate-topdata must never modify source lockfiles. Collection in read-only
// mode must leave the registry lock byte-identical even though the rows carry
// explicit ids that would otherwise mark the lock "modified".
func TestCollectGeneratedRegistryDatasetsReadOnlyDoesNotWriteLock(t *testing.T) {
dataDir := t.TempDir()
regDir := filepath.Join(dataDir, "damagetypes", "registry")
mkdirAll(t, regDir)
writeFile(t, filepath.Join(regDir, "types.json"), damagetypesRegistryTestTypes)
lockPath := filepath.Join(regDir, "lock.json")
original := "{\n \"damagetype:bludgeoning\": 0,\n \"damagetype:piercing\": 1\n}\n"
writeFile(t, lockPath, original)
if _, err := collectGeneratedRegistryDatasets(dataDir, false); err != nil {
t.Fatalf("read-only collection failed: %v", err)
}
got, err := os.ReadFile(lockPath)
if err != nil {
t.Fatalf("read lockfile: %v", err)
}
if string(got) != original {
t.Fatalf("read-only collection must not modify lockfile.\nwant: %q\ngot: %q", original, string(got))
}
}
// build-topdata still persists allocated ids to source lockfiles.
func TestCollectGeneratedRegistryDatasetsPersistWritesLock(t *testing.T) {
dataDir := t.TempDir()
regDir := filepath.Join(dataDir, "damagetypes", "registry")
mkdirAll(t, regDir)
writeFile(t, filepath.Join(regDir, "types.json"), damagetypesRegistryTestTypes)
lockPath := filepath.Join(regDir, "lock.json")
writeFile(t, lockPath, "{\n \"damagetype:bludgeoning\": 0\n}\n")
if _, err := collectGeneratedRegistryDatasets(dataDir, true); err != nil {
t.Fatalf("persisting collection failed: %v", err)
}
lockData, err := loadLockfile(lockPath)
if err != nil {
t.Fatalf("load lockfile: %v", err)
}
if lockData["damagetype:piercing"] != 1 {
t.Fatalf("persisting collection should record allocated id; got %#v", lockData)
}
}
+84 -1
View File
@@ -1,6 +1,7 @@
package topdata package topdata
import ( import (
"bytes"
"errors" "errors"
"fmt" "fmt"
"io" "io"
@@ -75,6 +76,8 @@ func BuildAndPackage(p *project.Project, progress func(string)) (PackageResult,
type BuildAndPackageOptions struct { type BuildAndPackageOptions struct {
Force bool Force bool
BuildWiki bool BuildWiki bool
// ponytail: no SkipLFS field — CRUCIBLE_SKIP_LFS env is the single skip mechanism;
// the CLI flag sets it directly before calling into this package.
} }
func BuildAndPackageWithOptions(p *project.Project, opts BuildAndPackageOptions, progress func(string)) (PackageResult, error) { func BuildAndPackageWithOptions(p *project.Project, opts BuildAndPackageOptions, progress func(string)) (PackageResult, error) {
@@ -123,6 +126,9 @@ func packageBuiltTopData(p *project.Project, nativeResult BuildResult, progress
if progress == nil { if progress == nil {
progress = func(string) {} progress = func(string) {}
} }
if err := materializeAssetLFS(p, progress); err != nil {
return PackageResult{}, err
}
outputHAK := p.TopDataPackageHAKPath() outputHAK := p.TopDataPackageHAKPath()
outputTLK := p.TopDataPackageTLKPath() outputTLK := p.TopDataPackageTLKPath()
@@ -557,7 +563,7 @@ func expectedCompiled2DAOutputs(p *project.Project) (map[string]struct{}, error)
if err != nil { if err != nil {
return nil, err return nil, err
} }
registryDatasets, err := collectGeneratedRegistryDatasets(dataDir) registryDatasets, err := collectGeneratedRegistryDatasets(dataDir, false)
if err != nil { if err != nil {
return nil, err return nil, err
} }
@@ -711,3 +717,80 @@ func countCompiledFiles(dir, extension string) (int, time.Time, error) {
} }
return count, oldest, nil return count, oldest, nil
} }
// materializeAssetLFS ensures the topdata asset tree holds real bytes, not
// Git-LFS pointer stubs, before the hak is packed. A bare CI checkout does not
// smudge LFS, so a direct Crucible build would otherwise ship ~130-byte stubs.
// It uses the clone's own credentials (R1) and hard-fails if stubs survive the
// pull. Set CRUCIBLE_SKIP_LFS=1 to opt out (maintain-tree regenerates the data/
// tree and discards the package, so it must not hard-depend on LFS).
func materializeAssetLFS(p *project.Project, progress func(string)) error {
if progress == nil {
progress = func(string) {}
}
assetsDir := filepath.Join(p.TopDataSourceDir(), "assets")
info, err := os.Stat(assetsDir)
if err != nil || !info.IsDir() {
return nil // no asset tree to materialize
}
if !hasLFSPointerStubs(assetsDir) {
return nil // already real bytes (or repo does not use LFS)
}
if isTruthyEnv(os.Getenv("CRUCIBLE_SKIP_LFS")) {
progress("CRUCIBLE_SKIP_LFS set: leaving Git-LFS pointer stubs unmaterialized (package will be discarded)")
return nil
}
if _, err := gitOutput(p.Root, "lfs", "version"); err != nil {
return fmt.Errorf("asset tree has Git-LFS pointer stubs but git-lfs is unavailable; install git-lfs or enter the nix devshell: %w", err)
}
progress("Materializing Git-LFS assets (git lfs pull)...")
// A fresh clone may lack the lfs smudge/clean filters; install them locally
// first (scoped to this repo's .git/config, no host mutation). Idempotent.
if _, err := gitOutput(p.Root, "lfs", "install", "--local"); err != nil {
return fmt.Errorf("git lfs install --local failed: %w", err)
}
if _, err := gitOutput(p.Root, "lfs", "pull"); err != nil {
return fmt.Errorf("git lfs pull failed: %w", err)
}
if hasLFSPointerStubs(assetsDir) {
return fmt.Errorf("refusing to build from pointer stubs: Git-LFS pointer stubs remain under %s after git lfs pull", assetsDir)
}
return nil
}
// hasLFSPointerStubs reports whether any small text file under root is a Git-LFS
// pointer. Real assets are larger than any pointer stub, so the size gate keeps
// this cheap on a tree of binaries.
func hasLFSPointerStubs(root string) bool {
found := false
_ = filepath.WalkDir(root, func(path string, d os.DirEntry, err error) error {
if err != nil || d.IsDir() || found {
return nil
}
info, ierr := d.Info()
if ierr != nil || info.Size() == 0 || info.Size() > 1024 {
return nil
}
f, oerr := os.Open(path)
if oerr != nil {
return nil
}
defer f.Close()
buf := make([]byte, 256)
n, _ := f.Read(buf)
if bytes.Contains(buf[:n], []byte("git-lfs.github.com/spec/v1")) {
found = true
}
return nil
})
return found
}
func isTruthyEnv(v string) bool {
switch strings.TrimSpace(strings.ToLower(v)) {
case "1", "true", "yes", "on":
return true
default:
return false
}
}
+2 -2
View File
@@ -1320,7 +1320,7 @@ func validateNativeOutputCatalog(dataDir string, report *ValidationReport) {
}) })
return return
} }
registryDatasets, err := collectGeneratedRegistryDatasets(dataDir) registryDatasets, err := collectGeneratedRegistryDatasets(dataDir, false)
if err != nil { if err != nil {
report.Diagnostics = append(report.Diagnostics, Diagnostic{ report.Diagnostics = append(report.Diagnostics, Diagnostic{
Severity: SeverityError, Severity: SeverityError,
@@ -2307,7 +2307,7 @@ func validateNativeBuildability(p *project.Project, report *ValidationReport) {
}) })
return return
} }
registryDatasets, err := collectGeneratedRegistryDatasets(dataDir) registryDatasets, err := collectGeneratedRegistryDatasets(dataDir, false)
if err != nil { if err != nil {
report.Diagnostics = append(report.Diagnostics, Diagnostic{ report.Diagnostics = append(report.Diagnostics, Diagnostic{
Severity: SeverityError, Severity: SeverityError,
+12 -12
View File
@@ -7633,7 +7633,7 @@ func TestApplyAutogenConsumersAugmentsPartsFromLocalOverride(t *testing.T) {
} }
} }
func TestApplyAutogenConsumersAugmentsHeadVisualeffectsFromLocalOverride(t *testing.T) { func TestApplyAutogenConsumersAugmentsAccessoryVisualeffectsFromLocalOverride(t *testing.T) {
root := testProjectRoot(t) root := testProjectRoot(t)
overrideRoot := filepath.Join(root, "autogen-assets") overrideRoot := filepath.Join(root, "autogen-assets")
mkdirAll(t, filepath.Join(overrideRoot, "vfxs", "head_accessories", "hat")) mkdirAll(t, filepath.Join(overrideRoot, "vfxs", "head_accessories", "hat"))
@@ -7706,18 +7706,18 @@ func TestApplyAutogenConsumersAugmentsHeadVisualeffectsFromLocalOverride(t *test
t.Fatalf("unexpected head feature defaults: %#v", hair) t.Fatalf("unexpected head feature defaults: %#v", hair)
} }
if _, ok := got[0].LockData["visualeffects:head_accessories/hat/bandana"]; !ok { if _, ok := got[0].LockData["visualeffects:head_accessories/hat/hfx_bandana.mdl"]; !ok {
t.Fatalf("expected head accessory lock id, got %#v", got[0].LockData) t.Fatalf("expected head accessory lock id, got %#v", got[0].LockData)
} }
if _, ok := got[0].LockData["visualeffects:head_decorations/laurel/laurel"]; !ok { if _, ok := got[0].LockData["visualeffects:head_decorations/laurel/hfx_laurel.mdl"]; !ok {
t.Fatalf("expected head decoration lock id, got %#v", got[0].LockData) t.Fatalf("expected head decoration lock id, got %#v", got[0].LockData)
} }
if _, ok := got[0].LockData["visualeffects:head_features/hair/hair_bangs"]; !ok { if _, ok := got[0].LockData["visualeffects:head_features/hair/hfx_hair_bangs.mdl"]; !ok {
t.Fatalf("expected head feature lock id, got %#v", got[0].LockData) t.Fatalf("expected head feature lock id, got %#v", got[0].LockData)
} }
} }
func TestApplyAutogenConsumersUsesFolderDrivenSlashHeadVisualeffectsNames(t *testing.T) { func TestApplyAutogenConsumersUsesFolderDrivenSlashAccessoryVisualeffectsNames(t *testing.T) {
root := testProjectRoot(t) root := testProjectRoot(t)
overrideRoot := filepath.Join(root, "autogen-assets") overrideRoot := filepath.Join(root, "autogen-assets")
mkdirAll(t, filepath.Join(overrideRoot, "vfxs", "head_features", "sinfar", "ears_plt")) mkdirAll(t, filepath.Join(overrideRoot, "vfxs", "head_features", "sinfar", "ears_plt"))
@@ -7735,8 +7735,8 @@ func TestApplyAutogenConsumersUsesFolderDrivenSlashHeadVisualeffectsNames(t *tes
Derive: project.AutogenDeriveConfig{Kind: "model_stem", GroupFrom: "first_path_segment"}, Derive: project.AutogenDeriveConfig{Kind: "model_stem", GroupFrom: "first_path_segment"},
Manifest: project.AutogenManifestConfig{ReleaseTag: "head-vfx-manifest-current", AssetName: "sow-accessory-vfx-manifest.json", CacheName: "sow-accessory-vfx-manifest.json"}, Manifest: project.AutogenManifestConfig{ReleaseTag: "head-vfx-manifest-current", AssetName: "sow-accessory-vfx-manifest.json", CacheName: "sow-accessory-vfx-manifest.json"},
LocalOverrideRoot: "autogen-assets", LocalOverrideRoot: "autogen-assets",
HeadVisualeffects: project.HeadVisualeffectsConfig{ AccessoryVisualeffects: project.AccessoryVisualeffectsConfig{
Groups: map[string]project.HeadVisualeffectGroupConfig{ Groups: map[string]project.AccessoryVisualeffectGroupConfig{
"head_features": { "head_features": {
ModelColumns: []string{"Imp_HeadCon_Node", "Imp_Root_H_Node"}, ModelColumns: []string{"Imp_HeadCon_Node", "Imp_Root_H_Node"},
}, },
@@ -7790,7 +7790,7 @@ func TestApplyAutogenConsumersUsesFolderDrivenSlashHeadVisualeffectsNames(t *tes
} }
} }
func TestBuildNativeUsesConfiguredHeadVisualeffectsGroups(t *testing.T) { func TestBuildNativeUsesConfiguredAccessoryVisualeffectsGroups(t *testing.T) {
root := t.TempDir() root := t.TempDir()
mkdirAll(t, filepath.Join(root, "src")) mkdirAll(t, filepath.Join(root, "src"))
mkdirAll(t, filepath.Join(root, "build")) mkdirAll(t, filepath.Join(root, "build"))
@@ -7878,7 +7878,7 @@ autogen:
} }
} }
func TestBuildGenerated2DAAssetsAppendsCachedModelsFromHeadVisualeffects(t *testing.T) { func TestBuildGenerated2DAAssetsAppendsCachedModelsFromAccessoryVisualeffects(t *testing.T) {
root := testProjectRoot(t) root := testProjectRoot(t)
mkdirAll(t, filepath.Join(root, "assets", "vfxs", "chest_accessories")) mkdirAll(t, filepath.Join(root, "assets", "vfxs", "chest_accessories"))
mkdirAll(t, filepath.Join(root, "assets", "vfxs", "head_accessories")) mkdirAll(t, filepath.Join(root, "assets", "vfxs", "head_accessories"))
@@ -13139,7 +13139,7 @@ func TestBuildNativeWithReleasedPartsManifestUsesAssetsServerOverride(t *testing
} }
} }
func TestBuildNativeWithReleasedHeadVisualeffectsManifest(t *testing.T) { func TestBuildNativeWithReleasedAccessoryVisualeffectsManifest(t *testing.T) {
root := t.TempDir() root := t.TempDir()
projRoot := filepath.Join(root, "project") projRoot := filepath.Join(root, "project")
mkdirAll(t, filepath.Join(projRoot, "src")) mkdirAll(t, filepath.Join(projRoot, "src"))
@@ -13274,7 +13274,7 @@ func TestBuildNativeWithReleasedHeadVisualeffectsManifest(t *testing.T) {
} }
} }
func TestBuildNativePrefersRemoteReleasedHeadVisualeffectsManifestOverFreshCache(t *testing.T) { func TestBuildNativePrefersRemoteReleasedAccessoryVisualeffectsManifestOverFreshCache(t *testing.T) {
root := t.TempDir() root := t.TempDir()
projRoot := filepath.Join(root, "project") projRoot := filepath.Join(root, "project")
mkdirAll(t, filepath.Join(projRoot, "src")) mkdirAll(t, filepath.Join(projRoot, "src"))
@@ -15021,7 +15021,7 @@ func (t twoDATable) cell(rowID, column string) string {
func runGitTest(t *testing.T, dir string, args ...string) { func runGitTest(t *testing.T, dir string, args ...string) {
t.Helper() t.Helper()
cmd := exec.Command("git", args...) cmd := exec.Command("git", append([]string{"-c", "commit.gpgsign=false", "-c", "tag.gpgsign=false"}, args...)...)
if dir != "" { if dir != "" {
cmd.Dir = dir cmd.Dir = dir
} }
+1 -1
View File
@@ -748,7 +748,7 @@ func collectWikiRacialtypesDatasets(dataDir string, datasets []nativeDataset) ([
if len(out) > 0 { if len(out) > 0 {
return out, nil return out, nil
} }
return collectRacialtypesRegistryDatasets(dataDir) return collectRacialtypesRegistryDatasets(dataDir, false)
} }
func collectWikiRacialtypeStatuses(dataDir string) map[string]string { func collectWikiRacialtypeStatuses(dataDir string) map[string]string {
+1 -30
View File
@@ -10,7 +10,6 @@ import (
"git.westgate.pw/ShadowsOverWestgate/sow-tools/internal/erf" "git.westgate.pw/ShadowsOverWestgate/sow-tools/internal/erf"
"git.westgate.pw/ShadowsOverWestgate/sow-tools/internal/gff" "git.westgate.pw/ShadowsOverWestgate/sow-tools/internal/gff"
"git.westgate.pw/ShadowsOverWestgate/sow-tools/internal/music"
"git.westgate.pw/ShadowsOverWestgate/sow-tools/internal/project" "git.westgate.pw/ShadowsOverWestgate/sow-tools/internal/project"
) )
@@ -101,8 +100,7 @@ func ValidateProject(p *project.Project) Report {
} }
for _, rel := range p.Inventory.AssetFiles { for _, rel := range p.Inventory.AssetFiles {
musicSource := isMusicSourceAsset(p, rel) if hasUppercaseResourceName(rel) {
if hasUppercaseResourceName(rel) && !musicSource {
report.add(rel, "resource filenames should be lowercase", SeverityWarning) report.add(rel, "resource filenames should be lowercase", SeverityWarning)
} }
base := strings.ToLower(strings.TrimSuffix(filepath.Base(rel), filepath.Ext(rel))) base := strings.ToLower(strings.TrimSuffix(filepath.Base(rel), filepath.Ext(rel)))
@@ -116,9 +114,6 @@ func ValidateProject(p *project.Project) Report {
if _, exists := assetIndex[key]; !exists { if _, exists := assetIndex[key]; !exists {
assetIndex[key] = rel assetIndex[key] = rel
} }
if musicSource {
continue
}
group, err := resolver.groupFor(rel) group, err := resolver.groupFor(rel)
if err != nil { if err != nil {
@@ -507,30 +502,6 @@ func hasAsset(name string, extensions []string, assets map[string]string) bool {
return false return false
} }
func isMusicSourceAsset(p *project.Project, rel string) bool {
rel = filepath.ToSlash(strings.TrimSpace(rel))
ext := strings.ToLower(filepath.Ext(rel))
effective := p.EffectiveConfig()
for _, dataset := range effective.Music.Datasets {
if !music.PathIsUnder(dataset.Source, rel) {
continue
}
for _, candidate := range dataset.ConvertExtensions {
if ext == strings.ToLower(strings.TrimSpace(candidate)) {
return true
}
}
}
if music.PathIsUnder("envi/music", rel) {
for _, candidate := range effective.Music.ConvertExtensions {
if ext == strings.ToLower(strings.TrimSpace(candidate)) {
return true
}
}
}
return false
}
func isBuiltinScript(name string, builtinScriptPrefixes []string) bool { func isBuiltinScript(name string, builtinScriptPrefixes []string) bool {
lower := strings.ToLower(name) lower := strings.ToLower(name)
for _, prefix := range builtinScriptPrefixes { for _, prefix := range builtinScriptPrefixes {
+1
View File
@@ -3,3 +3,4 @@
# wrappers/ changes. Add a repo here AND grant the sync bot write access to it. # wrappers/ changes. Add a repo here AND grant the sync bot write access to it.
ShadowsOverWestgate/sow-module ShadowsOverWestgate/sow-module
ShadowsOverWestgate/sow-topdata ShadowsOverWestgate/sow-topdata
ShadowsOverWestgate/sow-assets-manifest
+1 -1
View File
@@ -29,7 +29,7 @@ os="$(uname -s)"; arch="$(uname -m)"
case "$os" in case "$os" in
Linux) os=linux ;; Linux) os=linux ;;
Darwin) os=darwin ;; Darwin) os=darwin ;;
*) echo "crucible: unsupported OS '$os' — install ffmpeg-free crucible manually" >&2; exit 1 ;; *) echo "crucible: unsupported OS '$os' — install crucible manually" >&2; exit 1 ;;
esac esac
case "$arch" in case "$arch" in
x86_64|amd64) arch=amd64 ;; x86_64|amd64) arch=amd64 ;;