Follow-up to the mdb removal in the previous commit. Three of the six
extensions #64 listed as deliberate local additions are NWN2 formats,
not NWN:EE ones, and belong in an NWN:EE tool no more than mdb did.
Checked against xoreos (src/aurora/types.h), which covers every Aurora
game and so distinguishes the games' tables:
gr2 4003, sits in the NWN2 block (MDB2 4000, MDA2 4001, SPT2 4002,
GR2 4003, PWC 4008). Granny is NWN2; NWN:EE does not use it.
wlk xoreos numbers it 20004 and xml 20003, both above
xml kFileTypeMAXArchive in the section headed "Entries for files not
found in archives with numerical type IDs / Found in NWN2's ZIP
files". Neither has an archive restype in any Aurora game, so
Crucible's 0x0BCC and 0x0BCD were invented outright, in a band
Aurora does use elsewhere (3022 FSM, 3023 ART).
NWN:EE covers those jobs with formats already in the table: wok, pwk
and dwk for walkmeshes, and EE's own UI XML loads from ui/ovr rather
than from a HAK by restype, which is consistent with upstream
neverwinter.nim registering no xml number.
The remaining three local additions stay. lyt 3000, vis 3001 and mdx
3008 are real Aurora archive types that NWN1 ships in its own
data/*.bif; xoreos gives the same numbers. Upstream simply does not
list them.
No asset in the corpus uses gr2, wlk or xml, so nothing stops building.
Also drops them from AssetExtensions, and records the rule in a comment
on the table plus a test, so the next NWN2 format gets caught.
Refs ShadowsOverWestgate/sow-tools#64
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Five entries in the restype table disagreed with upstream
neverwinter.nim's restype.nim, so HAKs written from those source
extensions were labelled with a type the game reads as something else:
gff 0x07F7 (2039, upstream bte) -> 0x07F5 (2037)
ltr 0x0813 (2067, upstream bak) -> 0x07F4 (2036)
jpg 0x081C (2076, upstream tml) -> 0x0821 (2081)
dfa -> dft (0x07FD/2045 was already right,
only the name was a typo)
mdb 0x0816 (2070, upstream xbc) -> removed
Upstream registers no restype for mdb, so there is no correct number to
give it; keeping it squatting on xbc silently mislabels the resource.
It is dropped from AssetExtensions too, which turns .mdb into a loud
"unsupported extension" build error instead of a corrupt HAK entry. No
asset in the current corpus uses any of these paths.
Root cause of the miss: the init() consistency guard only panicked when
a number was missing from the reverse map. Its `if canonicalExt == ext
{ continue }` branch did nothing when the two maps disagreed, so a
mismatch was never caught. The guard now panics on disagreement and
also checks the reverse direction.
Adds a conformance test pinning the shared numbers against upstream,
including the four numbers upstream reserves for other extensions, and
a test that the two maps are mutual inverses.
Refs ShadowsOverWestgate/sow-tools#64
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Adds a `set_fields` option to `extract.merge.gff_json` rules: forces existing Int fields to a fixed value on every extract, applied after preserve_fields/merge_lists so the forced value always wins. Fields absent from the document stay absent. Rule targets can now be globs (e.g. `areas/*.are.json`).
Motivation: sow-module area weather chances must stay 0 (scripted regional weather owns weather); this makes the extract pipeline enforce it instead of a post-extract script.
Covered by `TestExtractSetsConfiguredGFFJSONFields`; `make check` passes.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Reviewed-on: #40
Co-authored-by: vickydotbat <vickydotbat@tutamail.com>
Co-committed-by: vickydotbat <vickydotbat@tutamail.com>
Phase 2 of the parts-row autogen design — Crucible (sow-tools) side only.
Resolves parts 2DA rows from the immutable \`part.yml\` published with the asset HAK release, over the anonymous Bunny CDN channel, with the parts consumer **required** (no fail-open).
## Changes
- **Resolver (T3):** generalize \`resolveCDNChannelManifest\` — derive manifest basename + provenance label from config; drop hardcoded \`assets/vfxs.yml\` so \`part.yml\` resolves through the same path. VFX behavior unchanged.
- **Parts filter (T4):** \`filterCDNChannelEntries\` dispatches on consumer mode; new \`filterPartsCDNChannelEntries\` implements the inventory contract — \`restype: mdl\` under \`part/<supported-cat>/\`, row id from trailing digits, dedup l/r/race/gender variants by (category,rowID), sort by source, **zero accepted rows = hard fail**, reject row id 0 / non-numeric / malformed / no-assets.
- **Category (T5):** add \`hand\` + \`parts/hand\` mapping (12 datasets; \`leg→legs\` already present).
- **Pipeline (T6):** native build runs one explicit augment → normalize → override sequence under the configured \`parts_rows\` policy; overrides may update an existing/discovered row but **never synthesize** one (orphan override now errors).
- **Validation (T7):** reject more than one \`parts_rows\` consumer.
## Tests
9 new tests: parts filter contract, 12-dataset coverage, offline manifest-basename, orphan-override reject + discovered-row update, and a parity guard proving a bare CDN-only parts build needs no wrapper/token/NWN_ROOT/checkout.
Gate green: \`go test ./internal/topdata/... ./internal/project/...\` + full \`go build ./... && go test ./...\`.
## Scope / ordering
Phase 2 is inert until **Phase 1** (sow-assets-manifest publishes \`part.yml\`) ships and the operational **Gate** (release publish/promote) runs, then **Phase 3** enables the sow-topdata consumer. Do not enable the consumer against a channel whose current release predates \`part.yml\`.
Plan: \`sow-topdata/docs/superpowers/plans/2026-06-25-topdata-parts-autogen-channel.md\`
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Reviewed-on: #25
Co-authored-by: vickydotbat <vickydotbat@tutamail.com>
Co-committed-by: vickydotbat <vickydotbat@tutamail.com>
forces build parity with crucible = local builds and CI/CD builds use different tools.
Reviewed-on: #14
Co-authored-by: vickydotbat <vickydotbat@tutamail.com>
Co-committed-by: vickydotbat <vickydotbat@tutamail.com>
Implemented on branches `feature/topdata-row-extensions` in both `toolkit` and `module`.
**Scope Audited**
- Toolkit native topdata collection/build path.
- Project config/effective config validation.
- Module `nwn-tool.yaml` topdata config and topdata README.
**Changes Made**
- Added `topdata.row_extensions` config with `repeat_last` support.
- Added glob dataset matching and build-time row extension for plain native datasets in [native.go](/home/vicky/Projects/nwnee-shadowsoverwestgate/toolkit/internal/topdata/native.go:641).
- Extension copies emitted columns from the highest authored `Level`, increments `Level`, and assigns sequential `id`s through target level.
- Enabled rules for `classes/spellsgained/*` and `classes/spellsknown/*` to level 60 in [nwn-tool.yaml](/home/vicky/Projects/nwnee-shadowsoverwestgate/module/nwn-tool.yaml:90).
- Updated toolkit/module docs.
**Configuration / Compatibility**
- New public config type: `TopDataRowExtensionConfig` in [project.go](/home/vicky/Projects/nwnee-shadowsoverwestgate/toolkit/internal/project/project.go:233).
- Existing behavior is unchanged unless a repo opts into `topdata.row_extensions`.
- Source JSON and lockfiles are not updated for generated extension rows.
**Tests / Validation**
- Added project config tests for loading and validation.
- Added native topdata tests for extension behavior, glob application, preserving manually authored rows, unmatched datasets, and failure cases.
- Ran:
- `go test ./internal/project ./internal/topdata`
- `go test ./internal/topdata -run TestBuildNativeKeepsGeneratedIDsStableAcrossRepeatedBuilds`
- `/tmp/sow-toolkit-rowext config validate` in `module`
- `/tmp/sow-toolkit-rowext validate-topdata` in `module`
- `/tmp/sow-toolkit-rowext build-topdata --force`, then inspected generated rows: `cls_spgn_acolyte.2da` now reaches row `59` / level `60`.
**Remaining Notes**
- `validate-topdata` still reports the pre-existing portrait lock warnings for `cotbl/cotbl_` and `lantern/lantern_`.
- `build-topdata --force` briefly rewrote three spellbook JSON files as a side effect; I reverted those, leaving only the intended YAML/docs changes in `module`.
Reviewed-on: https://gitea.westgate.pw/ShadowsOverWestgate/sow-tools/pulls/12
Co-authored-by: vickydotbat <vickydotbat@tutamail.com>
Co-committed-by: vickydotbat <vickydotbat@tutamail.com>
**Scope Audited**
- Topdata configured value encoders in [native.go](/home/vicky/Projects/nwnee-shadowsoverwestgate/toolkit/internal/topdata/native.go:3894)
- Config validation in [project.go](/home/vicky/Projects/nwnee-shadowsoverwestgate/toolkit/internal/project/project.go:799)
- Wiki alignment formatting
- Module topdata config/docs and authored `PreferredAlignments` / `ProficiencyFeats` files
**Changes Made**
- Added generic `id_hex_list` support. It works for any configured dataset/column and emits packed hex row IDs from `ids`.
- Updated alignment encoding to accept explicit `alignments`; legacy `list` still works.
- Kept numeric packed lists on `list` / `all_except`.
- Configured `baseitems.ProficiencyFeats` in [nwn-tool.yaml](/home/vicky/Projects/nwnee-shadowsoverwestgate/module/nwn-tool.yaml:73).
- Migrated module proficiency authoring to `ids` and racial alignment authoring to `alignments`.
- Updated docs in [module/topdata/README.md](/home/vicky/Projects/nwnee-shadowsoverwestgate/module/topdata/README.md:157).
**Configuration / Schema Impact**
- New `topdata.value_encodings[].mode`: `id_hex_list`.
- `id_hex_list` accepts:
- `"feat:some_key"`
- `{ "id": "feat:some_key" }`
- numeric IDs, still range-checked by `min`, `max`, `hex_width`.
**Compatibility Impact**
- Existing numeric packed lists are unchanged.
- Existing alignment `{ "list": [...] }` remains accepted, but docs and module data now prefer `{ "alignments": [...] }`.
- ID-list behavior is config-driven, not hardcoded to proficiencies.
**Tests Added / Updated**
- Added build test for generic `id_hex_list`.
- Added validation test for unknown ID references.
- Updated alignment tests to use `alignments`.
- Updated wiki alignment tests for explicit `alignments` plus legacy `list`.
**Validation Performed**
- `go test ./internal/topdata` passed.
- `go test ./internal/project` passed.
- `go test ./...` passed.
- `SOW_TOOLS_DEV_BINARY=../toolkit/tools/sow-toolkit ./scripts/run-nwn-tool.sh config validate` passed in `module/`.
**Remaining Risk**
- `./validate-topdata.sh` in `module/` is currently blocked by existing staged module authored-content changes: generated feat/baseitems validation reports `dataset baseitems: override must specify id or key` from `topdata/data/feat/generated`. I did not try to repair that unrelated staged work.
## Summary
- add YAML-backed generated wiki visibility policies for topdata datasets
- centralize page and reference eligibility through a visibility index
- cover eligibility helpers, metadata precedence, derived feat sets, and link suppression with toolkit tests
## Verification
- go test ./...
- module ./validate-topdata.sh with the feature toolkit binary
- module ./build-wiki.sh --force with the feature toolkit binary
Companion module branch: codex-wiki-visibility.
Reviewed-on: https://gitea.westgate.pw/ShadowsOverWestgate/sow-tools/pulls/4
Co-authored-by: vickydotbat <vickydotbat@tutamail.com>
Co-committed-by: vickydotbat <vickydotbat@tutamail.com>
Implemented the wiki pipeline to a shippable local state across `toolkit/`, `module/`, and the NodeBB wiki plugin.
**Scope Audited**
- Toolkit config/project loading, wiki renderer, deployer, app command wiring.
- Module `nwn-tool.yaml`, wiki source/docs, release script, topdata docs.
- Plugin sanitizer/API storage contract docs and sanitizer tests.
**Changes Made**
- Added topdata-owned wiki declarations under `module/topdata/wiki/`.
- Added `topdata.wiki.*` config fields, validation, effective config output.
- Switched generated wiki pages to `.html` with HTML comment managed/manual markers.
- Added `page-index.json` generation and deterministic metadata.
- Added manual-section merge preservation and stale page `report`/`archive` handling.
- Added namespace `category_env` loading from `topdata/wiki/namespaces.yaml`, while keeping `--category`/`NODEBB_WIKI_CATEGORIES` overrides.
- Updated release deployment to keep dry-run first and make stale cleanup opt-in via `SOW_MODULE_WIKI_DEPLOY_STALE_POLICY`.
- Updated plugin sanitizer to preserve only `sow-topdata-wiki` comments.
**Configuration/Schema Impact**
- `module/nwn-tool.yaml` now declares wiki source, renderer, link strategy, templates, manual sections, managed marker policy, and stale policy.
- Deploy manifest entries now support stale/archive metadata: `stale`, `last_seen_hash`, `archived_hash`, `title`, `namespace`, `tid`, `pid`, `cid`.
**Compatibility Impact**
- Legacy Markdown managed markers are still readable for migration.
- Generated output is now HTML, not Markdown.
- Existing mapped pages update by `pid`; creates still require explicit `--create`.
**Tests Added/Updated**
- Go tests for wiki config validation, HTML rendering, page index, manual merge, stale report/archive, app summary output.
- Plugin sanitizer test for generated topdata bot HTML markers/subset.
**Validation Performed**
- `go test ./internal/project ./internal/topdata ./internal/app` passed.
- `./build-tool.sh` passed.
- `SOW_TOOLS_DEV_BINARY=../toolkit/tools/sow-toolkit ./validate-topdata.sh` passed.
- `SOW_TOOLS_DEV_BINARY=../toolkit/tools/sow-toolkit ./build-wiki.sh --force` passed, generating 1345 entity pages.
- `deploy-wiki --dry-run --create` with dummy endpoint/token and namespace CID env vars passed locally: 1377 planned creates, 0 stale/drift.
- `node tests/wiki-html-sanitizer.test.js` passed.
**Remaining Risks**
- Full plugin `npm test` is blocked by an existing unrelated drawer CSS contract failure in `tests/wiki-article-drawers.test.js`.
- Controlled live migration was not performed because it requires a non-production NodeBB namespace and credentials.
- Direct `PUT /api/v3/posts/{pid}` save-filter behavior is documented as needing live confirmation against the deployed NodeBB/plugin stack.
Reviewed-on: https://gitea.westgate.pw/ShadowsOverWestgate/sow-tools/pulls/3
Co-authored-by: vickydotbat <vickydotbat@tutamail.com>
Co-committed-by: vickydotbat <vickydotbat@tutamail.com>
Key hardening changes:
- Config validation now rejects paths.source: . and paths.assets: ., so source/asset roots cannot
resolve to the repository root.
- apply-hak-manifest now refuses to run when paths.source is unset or unsafe, instead of potentially
writing under a root-level module/.
- Inline flags across utility parsers now reject empty values consistently, e.g. --dataset=, --hak=,
--endpoint=, --output=.
- build-changelog now supports --flag=value inline syntax like the other utilities.
archvillainetteandClaude Opus 5