Implemented on branches `feature/topdata-row-extensions` in both `toolkit` and `module`.
**Scope Audited**
- Toolkit native topdata collection/build path.
- Project config/effective config validation.
- Module `nwn-tool.yaml` topdata config and topdata README.
**Changes Made**
- Added `topdata.row_extensions` config with `repeat_last` support.
- Added glob dataset matching and build-time row extension for plain native datasets in [native.go](/home/vicky/Projects/nwnee-shadowsoverwestgate/toolkit/internal/topdata/native.go:641).
- Extension copies emitted columns from the highest authored `Level`, increments `Level`, and assigns sequential `id`s through target level.
- Enabled rules for `classes/spellsgained/*` and `classes/spellsknown/*` to level 60 in [nwn-tool.yaml](/home/vicky/Projects/nwnee-shadowsoverwestgate/module/nwn-tool.yaml:90).
- Updated toolkit/module docs.
**Configuration / Compatibility**
- New public config type: `TopDataRowExtensionConfig` in [project.go](/home/vicky/Projects/nwnee-shadowsoverwestgate/toolkit/internal/project/project.go:233).
- Existing behavior is unchanged unless a repo opts into `topdata.row_extensions`.
- Source JSON and lockfiles are not updated for generated extension rows.
**Tests / Validation**
- Added project config tests for loading and validation.
- Added native topdata tests for extension behavior, glob application, preserving manually authored rows, unmatched datasets, and failure cases.
- Ran:
- `go test ./internal/project ./internal/topdata`
- `go test ./internal/topdata -run TestBuildNativeKeepsGeneratedIDsStableAcrossRepeatedBuilds`
- `/tmp/sow-toolkit-rowext config validate` in `module`
- `/tmp/sow-toolkit-rowext validate-topdata` in `module`
- `/tmp/sow-toolkit-rowext build-topdata --force`, then inspected generated rows: `cls_spgn_acolyte.2da` now reaches row `59` / level `60`.
**Remaining Notes**
- `validate-topdata` still reports the pre-existing portrait lock warnings for `cotbl/cotbl_` and `lantern/lantern_`.
- `build-topdata --force` briefly rewrote three spellbook JSON files as a side effect; I reverted those, leaving only the intended YAML/docs changes in `module`.
Reviewed-on: https://gitea.westgate.pw/ShadowsOverWestgate/sow-tools/pulls/12
Co-authored-by: vickydotbat <vickydotbat@tutamail.com>
Co-committed-by: vickydotbat <vickydotbat@tutamail.com>
**Scope Audited**
- Topdata configured value encoders in [native.go](/home/vicky/Projects/nwnee-shadowsoverwestgate/toolkit/internal/topdata/native.go:3894)
- Config validation in [project.go](/home/vicky/Projects/nwnee-shadowsoverwestgate/toolkit/internal/project/project.go:799)
- Wiki alignment formatting
- Module topdata config/docs and authored `PreferredAlignments` / `ProficiencyFeats` files
**Changes Made**
- Added generic `id_hex_list` support. It works for any configured dataset/column and emits packed hex row IDs from `ids`.
- Updated alignment encoding to accept explicit `alignments`; legacy `list` still works.
- Kept numeric packed lists on `list` / `all_except`.
- Configured `baseitems.ProficiencyFeats` in [nwn-tool.yaml](/home/vicky/Projects/nwnee-shadowsoverwestgate/module/nwn-tool.yaml:73).
- Migrated module proficiency authoring to `ids` and racial alignment authoring to `alignments`.
- Updated docs in [module/topdata/README.md](/home/vicky/Projects/nwnee-shadowsoverwestgate/module/topdata/README.md:157).
**Configuration / Schema Impact**
- New `topdata.value_encodings[].mode`: `id_hex_list`.
- `id_hex_list` accepts:
- `"feat:some_key"`
- `{ "id": "feat:some_key" }`
- numeric IDs, still range-checked by `min`, `max`, `hex_width`.
**Compatibility Impact**
- Existing numeric packed lists are unchanged.
- Existing alignment `{ "list": [...] }` remains accepted, but docs and module data now prefer `{ "alignments": [...] }`.
- ID-list behavior is config-driven, not hardcoded to proficiencies.
**Tests Added / Updated**
- Added build test for generic `id_hex_list`.
- Added validation test for unknown ID references.
- Updated alignment tests to use `alignments`.
- Updated wiki alignment tests for explicit `alignments` plus legacy `list`.
**Validation Performed**
- `go test ./internal/topdata` passed.
- `go test ./internal/project` passed.
- `go test ./...` passed.
- `SOW_TOOLS_DEV_BINARY=../toolkit/tools/sow-toolkit ./scripts/run-nwn-tool.sh config validate` passed in `module/`.
**Remaining Risk**
- `./validate-topdata.sh` in `module/` is currently blocked by existing staged module authored-content changes: generated feat/baseitems validation reports `dataset baseitems: override must specify id or key` from `topdata/data/feat/generated`. I did not try to repair that unrelated staged work.
**Scope Audited**
`module/` release/wiki wrapper flow and `toolkit/` wiki generation cache logic.
**Changes Made**
Root cause was stale wiki cache reuse. The generator cache considered digest + page count, but not generator version, so a restored VPS cache with old generated output could skip regeneration.
Changed [wiki_native.go](/home/vicky/Projects/nwnee-shadowsoverwestgate/toolkit/internal/topdata/wiki_native.go:29) to bump the wiki generator to `nodebb-tiptap-html-v3`, and changed [wiki_native.go](/home/vicky/Projects/nwnee-shadowsoverwestgate/toolkit/internal/topdata/wiki_native.go:268) so cached pages are current only when `state.version` matches.
**Configuration/Schema Impact**
No YAML/config/schema changes. Existing `.cache/wiki/state.json` files with `nodebb-tiptap-html-v2` will be invalidated and regenerated.
**Compatibility Impact**
This is compatible with existing caches; it only forces one regeneration when the new toolkit runs. After that, normal digest-based skipping resumes.
**Tests Added**
Added [TestBuildWikiRegeneratesOldGeneratorCache](/home/vicky/Projects/nwnee-shadowsoverwestgate/toolkit/internal/topdata/wiki_native_test.go:183), which simulates a `v2` cache and verifies `build-wiki` regenerates it.
**Validation Performed**
- Confirmed new test failed before the fix with `wiki status "skipped"`.
- `go test ./internal/topdata -count=1` passes.
- `./build-tool.sh` succeeds.
- `SOW_TOOLS_DEV_BINARY=../toolkit/tools/sow-toolkit ./build-wiki.sh` now reports:
- `wiki pages: 1163`
- `wiki status: generated`
- Local generated race pages: `17`.
**Remaining Risk / VPS Next Step**
The VPS release job must use this rebuilt/published toolkit. On the next run, you should see the wiki build regenerate once before deploy, and deploy should load `1163` local pages instead of `1092`, including `races:*` pages.
### Summary
- Change generated topdata wiki links to target public wiki namespace and canonical slug paths.
- Add generated public path metadata to wiki page generation and page-index output.
- Add YAML-driven generated slug overrides and public-target collision validation.
### Details
- Generated relation links now render public targets such as:
- `[[feat/hardiness-vs-enchantments|Hardiness vs. Enchantments]]`
- `[[feat/favored-enemy-elves|Favored Enemy: Elves]]`
- Topdata page IDs remain the internal identity for output layout, managed markers, deploy state, and bookkeeping.
- Generated managed markers now include the effective public slug:
- `<!-- sow-topdata-wiki:page=... wiki_slug=... -->`
- Page-index entries now include:
- `public_slug`
- `public_target`
- Added parsing and validation for `page_paths.slug_overrides` from `topdata/wiki/wiki.yaml`.
- Generated public targets are validated for duplicate namespace-local collisions.
- Table link helpers now derive public targets from target page titles when available, so display aliases do not accidentally change link destinations.
### Tests
- Added and updated coverage for:
- public target rendering
- marker `wiki_slug` output
- page-index public metadata
- duplicate generated public target rejection
- slug override parsing and validation
- override-driven target selection
- target title slugging independent of display labels
### Validation
- `go test ./...`
- `gofmt -w` on changed Go files
- `./build-tool.sh`
- `git diff --check`
Reviewed-on: https://gitea.westgate.pw/ShadowsOverWestgate/sow-tools/pulls/6
Co-authored-by: vickydotbat <vickydotbat@tutamail.com>
Co-committed-by: vickydotbat <vickydotbat@tutamail.com>
**Scope Audited**
- Module wiki visibility config and docs.
- Toolkit wiki expression engine and visibility tests.
- Feat prerequisite rendering/reference behavior.
**Changes Made**
- Added `hidden_ref(Field, "dataset")` and `all_hidden_refs(..., "dataset")` helpers in [wiki_tables.go](/home/vicky/Projects/nwnee-shadowsoverwestgate/toolkit/internal/topdata/wiki_tables.go:914).
- Added feat visibility exclusions in [visibility.yaml](/home/vicky/Projects/nwnee-shadowsoverwestgate/module/topdata/wiki/visibility.yaml:44) for:
- `PREREQFEAT1`
- `PREREQFEAT2`
- `REQSKILL`
- `REQSKILL2`
- `OrReqFeat0..OrReqFeat4`
- Documented the helpers in [README.md](/home/vicky/Projects/nwnee-shadowsoverwestgate/module/topdata/wiki/README.md:53).
- Added coverage in [wiki_native_test.go](/home/vicky/Projects/nwnee-shadowsoverwestgate/toolkit/internal/topdata/wiki_native_test.go:646).
**Configuration/Schema Impact**
- No schema file change. `visibility.yaml` can now use the two new expression helpers.
**Compatibility Impact**
- Existing visibility expressions are unchanged.
- Missing or `****` prerequisite fields are ignored.
- Present references that resolve to hidden or missing wiki rows are treated as unreachable.
**Tests/Validation**
- Confirmed the new test failed before implementation on unknown helper.
- Ran `go test ./internal/topdata -run TestWikiVisibility`.
- Ran `go test ./internal/topdata`.
- Rebuilt toolkit with `./build-tool.sh`.
- Ran module wiki build with local toolkit:
`SOW_TOOLS_DEV_BINARY=../toolkit/tools/sow-toolkit ./build-wiki.sh`
Result: `wiki pages: 771`.
## Summary
- add YAML-backed generated wiki visibility policies for topdata datasets
- centralize page and reference eligibility through a visibility index
- cover eligibility helpers, metadata precedence, derived feat sets, and link suppression with toolkit tests
## Verification
- go test ./...
- module ./validate-topdata.sh with the feature toolkit binary
- module ./build-wiki.sh --force with the feature toolkit binary
Companion module branch: codex-wiki-visibility.
Reviewed-on: https://gitea.westgate.pw/ShadowsOverWestgate/sow-tools/pulls/4
Co-authored-by: vickydotbat <vickydotbat@tutamail.com>
Co-committed-by: vickydotbat <vickydotbat@tutamail.com>
Changes Made
- page-index.json now includes generated meta/status pages, not just entity pages.
- Wiki page count now comes from the final page index, so wiki pages: matches generated HTML files.
- Page IDs now normalize /, \, and _ to : to prevent duplicate index entries for slash-bearing keys.
- Added page-index validation for duplicate page_id and duplicate output_path.
Implemented the wiki pipeline to a shippable local state across `toolkit/`, `module/`, and the NodeBB wiki plugin.
**Scope Audited**
- Toolkit config/project loading, wiki renderer, deployer, app command wiring.
- Module `nwn-tool.yaml`, wiki source/docs, release script, topdata docs.
- Plugin sanitizer/API storage contract docs and sanitizer tests.
**Changes Made**
- Added topdata-owned wiki declarations under `module/topdata/wiki/`.
- Added `topdata.wiki.*` config fields, validation, effective config output.
- Switched generated wiki pages to `.html` with HTML comment managed/manual markers.
- Added `page-index.json` generation and deterministic metadata.
- Added manual-section merge preservation and stale page `report`/`archive` handling.
- Added namespace `category_env` loading from `topdata/wiki/namespaces.yaml`, while keeping `--category`/`NODEBB_WIKI_CATEGORIES` overrides.
- Updated release deployment to keep dry-run first and make stale cleanup opt-in via `SOW_MODULE_WIKI_DEPLOY_STALE_POLICY`.
- Updated plugin sanitizer to preserve only `sow-topdata-wiki` comments.
**Configuration/Schema Impact**
- `module/nwn-tool.yaml` now declares wiki source, renderer, link strategy, templates, manual sections, managed marker policy, and stale policy.
- Deploy manifest entries now support stale/archive metadata: `stale`, `last_seen_hash`, `archived_hash`, `title`, `namespace`, `tid`, `pid`, `cid`.
**Compatibility Impact**
- Legacy Markdown managed markers are still readable for migration.
- Generated output is now HTML, not Markdown.
- Existing mapped pages update by `pid`; creates still require explicit `--create`.
**Tests Added/Updated**
- Go tests for wiki config validation, HTML rendering, page index, manual merge, stale report/archive, app summary output.
- Plugin sanitizer test for generated topdata bot HTML markers/subset.
**Validation Performed**
- `go test ./internal/project ./internal/topdata ./internal/app` passed.
- `./build-tool.sh` passed.
- `SOW_TOOLS_DEV_BINARY=../toolkit/tools/sow-toolkit ./validate-topdata.sh` passed.
- `SOW_TOOLS_DEV_BINARY=../toolkit/tools/sow-toolkit ./build-wiki.sh --force` passed, generating 1345 entity pages.
- `deploy-wiki --dry-run --create` with dummy endpoint/token and namespace CID env vars passed locally: 1377 planned creates, 0 stale/drift.
- `node tests/wiki-html-sanitizer.test.js` passed.
**Remaining Risks**
- Full plugin `npm test` is blocked by an existing unrelated drawer CSS contract failure in `tests/wiki-article-drawers.test.js`.
- Controlled live migration was not performed because it requires a non-production NodeBB namespace and credentials.
- Direct `PUT /api/v3/posts/{pid}` save-filter behavior is documented as needing live confirmation against the deployed NodeBB/plugin stack.
Reviewed-on: https://gitea.westgate.pw/ShadowsOverWestgate/sow-tools/pulls/3
Co-authored-by: vickydotbat <vickydotbat@tutamail.com>
Co-committed-by: vickydotbat <vickydotbat@tutamail.com>
archvillainette