Wiki pipeline (#3)

Implemented the wiki pipeline to a shippable local state across `toolkit/`, `module/`, and the NodeBB wiki plugin.

**Scope Audited**
- Toolkit config/project loading, wiki renderer, deployer, app command wiring.
- Module `nwn-tool.yaml`, wiki source/docs, release script, topdata docs.
- Plugin sanitizer/API storage contract docs and sanitizer tests.

**Changes Made**
- Added topdata-owned wiki declarations under `module/topdata/wiki/`.
- Added `topdata.wiki.*` config fields, validation, effective config output.
- Switched generated wiki pages to `.html` with HTML comment managed/manual markers.
- Added `page-index.json` generation and deterministic metadata.
- Added manual-section merge preservation and stale page `report`/`archive` handling.
- Added namespace `category_env` loading from `topdata/wiki/namespaces.yaml`, while keeping `--category`/`NODEBB_WIKI_CATEGORIES` overrides.
- Updated release deployment to keep dry-run first and make stale cleanup opt-in via `SOW_MODULE_WIKI_DEPLOY_STALE_POLICY`.
- Updated plugin sanitizer to preserve only `sow-topdata-wiki` comments.

**Configuration/Schema Impact**
- `module/nwn-tool.yaml` now declares wiki source, renderer, link strategy, templates, manual sections, managed marker policy, and stale policy.
- Deploy manifest entries now support stale/archive metadata: `stale`, `last_seen_hash`, `archived_hash`, `title`, `namespace`, `tid`, `pid`, `cid`.

**Compatibility Impact**
- Legacy Markdown managed markers are still readable for migration.
- Generated output is now HTML, not Markdown.
- Existing mapped pages update by `pid`; creates still require explicit `--create`.

**Tests Added/Updated**
- Go tests for wiki config validation, HTML rendering, page index, manual merge, stale report/archive, app summary output.
- Plugin sanitizer test for generated topdata bot HTML markers/subset.

**Validation Performed**
- `go test ./internal/project ./internal/topdata ./internal/app` passed.
- `./build-tool.sh` passed.
- `SOW_TOOLS_DEV_BINARY=../toolkit/tools/sow-toolkit ./validate-topdata.sh` passed.
- `SOW_TOOLS_DEV_BINARY=../toolkit/tools/sow-toolkit ./build-wiki.sh --force` passed, generating 1345 entity pages.
- `deploy-wiki --dry-run --create` with dummy endpoint/token and namespace CID env vars passed locally: 1377 planned creates, 0 stale/drift.
- `node tests/wiki-html-sanitizer.test.js` passed.

**Remaining Risks**
- Full plugin `npm test` is blocked by an existing unrelated drawer CSS contract failure in `tests/wiki-article-drawers.test.js`.
- Controlled live migration was not performed because it requires a non-production NodeBB namespace and credentials.
- Direct `PUT /api/v3/posts/{pid}` save-filter behavior is documented as needing live confirmation against the deployed NodeBB/plugin stack.

Reviewed-on: https://gitea.westgate.pw/ShadowsOverWestgate/sow-tools/pulls/3
Co-authored-by: vickydotbat <vickydotbat@tutamail.com>
Co-committed-by: vickydotbat <vickydotbat@tutamail.com>
This commit is contained in:
2026-05-15 09:09:37 +02:00
committed by archvillainette
parent d767ec0a25
commit 9b2084344d
9 changed files with 1030 additions and 166 deletions
+56 -6
View File
@@ -204,12 +204,30 @@ type TopDataConfig struct {
}
type TopDataWikiConfig struct {
OutputRoot string `json:"output_root" yaml:"output_root"`
PagesDir string `json:"pages_dir" yaml:"pages_dir"`
StateFile string `json:"state_file" yaml:"state_file"`
ManagedNamespaces []string `json:"managed_namespaces" yaml:"managed_namespaces"`
DeployManifest string `json:"deploy_manifest" yaml:"deploy_manifest"`
DeployEditSummary string `json:"deploy_edit_summary" yaml:"deploy_edit_summary"`
OutputRoot string `json:"output_root" yaml:"output_root"`
PagesDir string `json:"pages_dir" yaml:"pages_dir"`
StateFile string `json:"state_file" yaml:"state_file"`
ManagedNamespaces []string `json:"managed_namespaces" yaml:"managed_namespaces"`
DeployManifest string `json:"deploy_manifest" yaml:"deploy_manifest"`
DeployEditSummary string `json:"deploy_edit_summary" yaml:"deploy_edit_summary"`
Source string `json:"source" yaml:"source"`
Renderer string `json:"renderer" yaml:"renderer"`
LinkStrategy string `json:"link_strategy" yaml:"link_strategy"`
NamespacesFile string `json:"namespaces_file" yaml:"namespaces_file"`
TemplatesDir string `json:"templates_dir" yaml:"templates_dir"`
ManualSectionsDir string `json:"manual_sections_dir" yaml:"manual_sections_dir"`
StalePages TopDataWikiStalePagesConfig `json:"stale_pages" yaml:"stale_pages"`
ManagedRegion TopDataWikiManagedRegionConfig `json:"managed_region" yaml:"managed_region"`
}
type TopDataWikiStalePagesConfig struct {
Default string `json:"default" yaml:"default"`
LiveCleanup string `json:"live_cleanup" yaml:"live_cleanup"`
}
type TopDataWikiManagedRegionConfig struct {
MarkerFormat string `json:"marker_format" yaml:"marker_format"`
PageMarkerPrefix string `json:"page_marker_prefix" yaml:"page_marker_prefix"`
}
type ExtractConfig struct {
@@ -485,6 +503,10 @@ func (p *Project) ValidateLayout() error {
failures = append(failures, validateRelativePath("topdata.wiki.pages_dir", effective.TopData.Wiki.PagesDir)...)
failures = append(failures, validateRelativePath("topdata.wiki.state_file", effective.TopData.Wiki.StateFile)...)
failures = append(failures, validateRelativePath("topdata.wiki.deploy_manifest", effective.TopData.Wiki.DeployManifest)...)
failures = append(failures, validateTreeRootPath("topdata.wiki.source", effective.TopData.Wiki.Source)...)
failures = append(failures, validateRelativePath("topdata.wiki.namespaces_file", effective.TopData.Wiki.NamespacesFile)...)
failures = append(failures, validateTreeRootPath("topdata.wiki.templates_dir", effective.TopData.Wiki.TemplatesDir)...)
failures = append(failures, validateTreeRootPath("topdata.wiki.manual_sections_dir", effective.TopData.Wiki.ManualSectionsDir)...)
failures = append(failures, validateRelativePath("autogen.cache.root", effective.Autogen.Cache.Root)...)
if err := validateOutputFileName("outputs.module_archive", filepath.Base(effective.Outputs.ModuleArchive), ".mod"); err != nil {
failures = append(failures, err)
@@ -498,6 +520,26 @@ func (p *Project) ValidateLayout() error {
if err := validateOutputFileName("topdata.wiki.deploy_manifest", filepath.Base(effective.TopData.Wiki.DeployManifest), ".json"); err != nil {
failures = append(failures, err)
}
switch effective.TopData.Wiki.Renderer {
case "nodebb_tiptap_html":
default:
failures = append(failures, fmt.Errorf("topdata.wiki.renderer %q is not supported", effective.TopData.Wiki.Renderer))
}
switch effective.TopData.Wiki.LinkStrategy {
case "preserve_westgate_wiki_links":
default:
failures = append(failures, fmt.Errorf("topdata.wiki.link_strategy %q is not supported", effective.TopData.Wiki.LinkStrategy))
}
for key, policy := range map[string]string{
"topdata.wiki.stale_pages.default": effective.TopData.Wiki.StalePages.Default,
"topdata.wiki.stale_pages.live_cleanup": effective.TopData.Wiki.StalePages.LiveCleanup,
} {
switch policy {
case "report", "archive", "unpublish":
default:
failures = append(failures, fmt.Errorf("%s %q is not supported", key, policy))
}
}
switch effective.Extract.Layout {
case "nwn_canonical_json":
default:
@@ -834,6 +876,14 @@ func (p *Project) TopDataWikiRootDir() string {
return filepath.Join(p.TopDataBuildDir(), wikiRoot)
}
func (p *Project) TopDataWikiSourceDir() string {
source := filepath.FromSlash(p.EffectiveConfig().TopData.Wiki.Source)
if filepath.IsAbs(source) {
return source
}
return filepath.Join(p.Root, source)
}
func (p *Project) TopDataWikiPagesDir() string {
return filepath.Join(p.TopDataWikiRootDir(), filepath.FromSlash(p.EffectiveConfig().TopData.Wiki.PagesDir))
}