# sow-tools Shared NWN build tooling for Shadows Over Westgate. This repository owns the Go-based `sow-toolkit` binary used by the separate module and assets repositories. ## Build Build the development binary into `tools/sow-toolkit`: ```bash ./build-tool.sh ``` On Windows: ```powershell .\build-tool.ps1 ``` Both wrappers use `.cache/go-build` for the Go build cache and skip rebuilding when the sources are older than the existing development binary. Or run directly with Go: ```bash GOCACHE="$PWD/.cache/go-build" go run ./cmd/nwn-tool --help ``` When changing native topdata id allocation, lockfile persistence, generated families, or null-and-relocate behavior, run the repeat-build determinism test: ```bash go test ./internal/topdata -run TestBuildNativeKeepsGeneratedIDsStableAcrossRepeatedBuilds ``` That test builds the same topdata project ten times in one process and asserts that relocated/generated ids stay pinned across every pass. ## Repository Layout ```text cmd/nwn-tool/ CLI entrypoint internal/app/ command wiring and console UX internal/erf/ ERF/MOD/HAK reading and writing internal/gff/ canonical GFF JSON and binary conversion internal/pipeline/ build, extract, compare, and manifest workflows internal/project/ config loading and repo scanning internal/topdata/ native topdata, wiki, and autogen workflows internal/validator/validation and diagnostics tools/ built development binary output ``` ## Commands The shared CLI exposes these project utilities: ```bash sow-toolkit build sow-toolkit build-module sow-toolkit build-haks [--hak ] [--archive ] [--source-manifest ] [--plan-only] [--skip-music] [--music-dataset ] sow-toolkit extract [ ...] sow-toolkit validate sow-toolkit compare sow-toolkit apply-hak-manifest [] sow-toolkit config sow-toolkit music sow-toolkit validate-topdata sow-toolkit build-topdata [--force] [--wiki] sow-toolkit build-top-package [--force] sow-toolkit compare-topdata sow-toolkit convert-topdata <2da-to-json|2da-to-module|json-to-2da> ... sow-toolkit build-wiki [--force] sow-toolkit deploy-wiki [--source-dir ] [--endpoint ] [--token ] [--version ] [--namespace ] [--category ] [--manifest ] [--stale-policy ] [--dry-run] [--create] [--force] [--reset-managed-namespaces] sow-toolkit build-changelog [--config ] [--output ] [--current-tag ] [--previous-tag ] [--api-base-url ] [--token ] ``` Global logging flags `--quiet`, `--verbose`, and `--debug` may be passed after the command name. Long options that take values accept either `--flag value` or `--flag=value`; empty inline values such as `--dataset=` are rejected. ## Repository Configuration Human-authored consumer repository configuration is YAML. The toolkit discovers configuration in this order: 1. `nwn-tool.yaml` 2. `nwn-tool.yml` 3. legacy `nwn-tool.json` YAML is canonical. If both YAML and legacy JSON files are present, the YAML file wins. Legacy `nwn-tool.json` still loads for migration, but commands print a warning and should be updated to `nwn-tool.yaml`. Example: ```yaml module: name: Shadows Over Westgate resref: sow_module description: Shadows Over Westgate asset pipeline. hak_order: - sow_top - group:sow_over - group:sow_core paths: assets: content build: build music: # Legacy shorthand — automatically normalized into a dataset. prefixes: envi/music/westgate: mus_wg_ # Explicit dataset config (replaces prefixes above). tools: ffmpeg: tools/ffmpeg ffprobe: tools/ffprobe defaults: stage_root: "{paths.cache}/music" credits_root: "{paths.cache}/credits" credits_overlay: CREDITS.md max_stem_length: 16 naming_scheme: nwn_bmu output_extension: .bmu convert_extensions: [.flac, .m4a, .mp3, .ogg, .wav] datasets: westgate: source: envi/music/westgate output: envi/music/westgate prefix: mus_wg_ package_mode: hak_asset ``` Resolved configuration can be inspected without scanning or building: ```bash sow-toolkit config validate sow-toolkit config effective --json sow-toolkit config inspect paths.build sow-toolkit config explain topdata.package_hak sow-toolkit config sources ``` `config effective` prints the normalized values consumed by commands, including toolkit defaults, YAML values, legacy status, active environment overrides, and provenance. Sensitive override values such as tokens are reported as set without printing the secret. Common configurable defaults: ```yaml paths: source: src assets: assets build: build cache: .cache tools: tools outputs: module_archive: "{module.resref}.mod" hak_manifest: haks.json hak_archive: "{hak.name}.hak" build: keep_existing_haks: false generated_assets: topdata_2da: - id: parts source: topdata output: "{paths.cache}/generated-assets/parts-2da" include_datasets: [parts/**] package_root: part autogen: id: parts mode: parts_rows root: part include: ["**/*.mdl"] derive: kind: trailing_numeric_suffix group_from: first_path_segment parts_rows: row_defaults: COSTMODIFIER: "0" acbonus: default: strategy: ascending_row_id_sort_key divisor: 100 format: "%.2f" datasets: chest: strategy: fixed value: "0.00" inventory: source_extensions: [] # empty means built-in NWN source extensions asset_extensions: [] # empty means built-in NWN asset extensions source_json_pattern: "{resref}.{extension}.json" validation: profile: nwn_module builtin_script_prefixes: [nw_, x0_, x1_, x2_, x3_, ga_, gc_, gen_, gui_, nwg_, ta_] required_fields: ifo: [Mod_Name] scripts: source_dir: scripts cache: "{paths.cache}/ncs" compiler: path: "" env: path: SOW_NWN_SCRIPT_COMPILER nwn_root: [SOW_NWN_ROOT, NWN_ROOT] nwn_user_directory: [SOW_NWN_USER_DIRECTORY, NWN_HOME, NWN_USER_DIRECTORY] search: - "{paths.tools}/script-compiler/nwn_script_comp" - "{paths.tools}/script-compiler/nwn_script_comp.exe" - "{paths.tools}/nwn_script_comp" - "{paths.tools}/nwn_script_comp.exe" - PATH:nwn_script_comp extract: layout: nwn_canonical_json archives: - "{module.resref}.mod" cleanup_stale: true consume_archives: false ignore_extensions: [] delete_module_archive_after_success: false # deprecated compatibility alias topdata: build: "{paths.build}/topdata" compiled_2da_dir: 2da compiled_tlk: sow_tlk.tlk package_hak: sow_top.hak package_tlk: sow_tlk.tlk value_encodings: - dataset: racialtypes/core column: AvailableSkinColors mode: packed_hex_list min: 0 max: 175 hex_width: 2 wiki: output_root: wiki pages_dir: pages state_file: state.json managed_namespaces: [classes, feat, itemtypes, races, skills, spells, meta] deploy_manifest: .wiki_deploy_manifest.json deploy_edit_summary: Auto-generated from native builder visibility_file: visibility.yaml title_prefix_min_length: 3 status_listing_scope: all autogen: cache: root: "{paths.cache}" max_age: 1h refresh_env: SOW_AUTOGEN_MANIFEST_REFRESH release_source: provider: gitea server_url_env: SOW_ASSETS_SERVER_URL repo_env: SOW_ASSETS_REPO ``` `topdata.value_encodings` enables configured authoring sugar for compact 2DA cell values. The `packed_hex_list` mode accepts authored values such as `{"list": [0, {"range": [10, 12]}]}` and emits a single `0x...` string with fixed-width hex chunks. `paths.source` and `paths.assets` must name real repository subtrees when they are configured. They may be omitted for repositories that do not own that class of source, but they must not resolve to the repository root (`.`). Output and cache paths must stay relative to the repository root unless a specific setting documents otherwise. `generated_assets.topdata_2da` is for HAK repositories that need a small native-topdata subset packaged as HAK resources. It builds only `.2da` output, injects those generated files into `build-haks`, and rejects TLK-backed text because this path does not produce TLK packages. Static generated groups can omit `autogen.mode`; in that case the selected datasets are compiled directly, and `package_root` plus the repository's HAK include patterns determine where the resources are packaged. For `cachedmodels_rows` generated assets, the configured autogen scan appends discovered model stems to the selected `cachedmodels` dataset without `.mdl` extensions and skips models already present in authored rows. For `parts_rows` generated assets, `parts_rows.acbonus` can normalize final ACBONUS values before module overrides are applied. The `ascending_row_id_sort_key` strategy writes toolset ordering values from row IDs so higher part rows can sort before lower rows in the toolset; the legacy `descending_row_id_sort_key` strategy remains available for repositories that already depend on it. Per-dataset policies such as `chest: fixed 0.00` can override the default. Authored source ACBONUS values are not authoritative when this policy is configured, except that authored `****` values remain null unless model discovery activates the row. Dense gaps without authored rows or discovered part models are emitted as `****` null rows. Files under `parts/modules` remain the final override layer. Extraction is deliberately guarded because it writes source files from binary archives. `extract.archives` is a list of glob patterns relative to `paths.build`; the default is only the configured module `.mod`. HAK extraction is opt-in, for example: ```yaml extract: archives: - "{module.resref}.mod" - "sow_over*.hak" ``` Module resources require a configured `paths.source`; HAK assets require a configured `paths.assets`; both roots must resolve to safe subtrees. If a root is unset or resolves to the repository root, extraction fails before writing instead of creating extension directories like `mdl/` or `png/` at the repo top level. `.erf` files are rejected by extraction because they do not contain `module.ifo`, so they cannot safely update module-level area membership. `consume_archives: true` deletes the selected build archive files only after the entire extraction succeeds. The older `delete_module_archive_after_success` setting remains as a module-only compatibility alias when `consume_archives` is not set. Stale cleanup is scoped to roots actually touched by extracted resources, so the default module-only extraction does not prune assets. `topdata.wiki.status_listing_scope` controls the generated wiki-status detail pages. The default `all` writes both per-namespace listings and aggregate `meta:wikistatus:` listings. Use `namespace` when aggregate status lists would be too large for the target NodeBB post limit. Set `topdata.wiki.status_pages: false` to omit status pages entirely; existing tracked status pages then become stale deploy-manifest entries and can be removed with `deploy-wiki --stale-policy purge`. GFF JSON extraction can merge selected fields and lists from the existing source file instead of replacing the whole extracted document. Rules are matched by `target` relative to `paths.source`; files without a matching rule keep the default overwrite behavior. For example: ```yaml extract: merge: gff_json: - target: module/module.ifo.json preserve_fields: - Mod_Entry_Area - Mod_Entry_X - Mod_Entry_Y - Mod_Entry_Z - Mod_Entry_Dir_X - Mod_Entry_Dir_Y merge_lists: - field: Mod_Area_list key_field: Area_Name strategy: preserve_existing_order_append_new ``` `preserve_fields` copies matching root fields from the existing source document into the extracted document before writing. The `preserve_existing_order_append_new` list strategy keeps existing keyed list order for entries still present in the extracted archive, uses extracted entries as the value source, appends archive-only entries in archive order, and removes existing entries that are no longer present in the archive. Music defaults (used when not explicitly configured): ```yaml music: defaults: stage_root: "{paths.cache}/music" credits_root: "{paths.cache}/credits" credits_overlay: CREDITS.md max_stem_length: 16 naming_scheme: nwn_bmu output_extension: .bmu convert_extensions: [.flac, .m4a, .mp3, .ogg, .wav] ``` Music datasets define individual music collections with independent sources and naming prefixes. `source` is the authored input root under `paths.assets`; `output` is the generated HAK asset root matched by HAK include globs. If `output` is omitted, it defaults to `source`. `package_mode: hak_asset` converts configured source extensions into generated HAK resources; `package_mode: none` scans and writes credits without adding generated HAK resources. The legacy `music.prefixes` shorthand is automatically normalized into datasets. `convert_extensions` is configuration-driven; the built-in default set is `.flac`, `.m4a`, `.mp3`, `.ogg`, and `.wav`. Those extensions are discovered from YAML within configured music dataset source roots, so setting them in `music.defaults.convert_extensions` or a dataset override is sufficient; you do not need a parallel `inventory.asset_extensions` override just to make music scanning see them. Music can also be run directly: ```bash sow-toolkit music list-datasets sow-toolkit music scan --dataset westgate sow-toolkit music build --dataset westgate --dry-run sow-toolkit music build --dataset westgate sow-toolkit music credits --dataset westgate --check sow-toolkit music validate --dataset westgate --json sow-toolkit music manifest --dataset westgate sow-toolkit music normalize --dataset westgate ``` `build-haks` uses the same music pipeline. Use `--plan-only` or music `--dry-run` to compute mappings without transcoding. Use `build-haks --skip-music` to package authored assets only, or `build-haks --music-dataset westgate` to limit conversion to a configured dataset. Generated and machine-readable artifacts remain JSON, including HAK manifests, topdata datasets, caches, credits inventories, build reports, and canonical GFF documents. Nested metadata such as `topdata/templates/config.yaml` remains valid when it is local workflow configuration rather than root repository configuration. ## Wiki And Changelog Utilities `build-topdata --wiki` or `build-wiki` renders wiki pages from the current native topdata state as Tiptap-compatible NodeBB HTML. Page templates are HTML fragments; they do not need to include a visible title heading because deploy uses generated `page-index.json` metadata for page titles and public paths. `deploy-wiki` publishes those pages to NodeBB. It reads `NODEBB_API_ENDPOINT`, `NODEBB_API_TOKEN`, `GITHUB_REF_NAME`, and `NODEBB_WIKI_CATEGORIES` when equivalent flags are omitted. Category mappings use `namespace=cid`, for example: ```bash sow-toolkit deploy-wiki --dry-run --namespace spells --category spells=42 ``` Namespace declarations in `topdata/wiki/namespaces.yaml` may set `canonical_path` when a generated namespace lives below a fuller NodeBB category path, for example `canonical_path: Wiki/Feats`. The `title` remains the display label, while `canonical_path` drives generated `public_path` metadata, wiki links, page-index entries, and deploy adoption matching. When a deploy manifest is missing but matching wiki pages already exist in NodeBB, `deploy-wiki` checks the Westgate Wiki namespace directory API and adopts those topic/post mappings before planning updates. It does the same when a cached manifest mapping points to a post NodeBB no longer has. Canonical `wikiPath`/`canonicalPath` data is preferred during adoption. Nested generated paths require that exact canonical path data; leaf/title compatibility matching is only used where it cannot cross a canonical parent path. `--create` is still required for genuinely new pages that have no existing remote topic. If NodeBB rejects a create because the clean wiki URL already exists, `deploy-wiki` re-queries the namespace by page leaf, adopts the matching topic, and updates it instead. Pass `--debug` when deploying large wiki changes to emit planning and live mutation progress. Updates and stale-page archives acquire a Westgate Wiki edit lock before saving, then send the returned `wikiEditLockToken` with the NodeBB post edit request. Creates, updates, and archives write the same managed HTML to both NodeBB `content` and `sourceContent`; the deploy manifest records `source_content_synced` so older manifests receive a one-time body repair even when the generated hash is unchanged. Stale generated pages are reported by default. `--stale-policy archive` rewrites stale generated pages in place as archived managed content. `--stale-policy purge` permanently removes only stale generated topics already tracked in the wiki deploy manifest. Dry-run that policy before the live run; it never discovers manual wiki pages from NodeBB for purge. `--reset-managed-namespaces` is a stronger destructive mode for slug or path standard migrations: it permanently removes every topic currently listed by the configured managed namespace categories, ignores the previous deploy manifest, and recreates the current generated pages. It requires `--create` when local generated pages are present. For newly created NodeBB topics, `topdata.wiki.title_prefix_min_length` controls when the deployer prefixes the namespace in the topic title. The default is `3`, so one- and two-character page titles are created as `Namespace: Title`, while titles with three or more characters keep the page title alone. Existing bot-managed pages with older prefixed titles are renamed on later deploy passes when their current title no longer matches this policy. `build-changelog` renders a release changelog from tagged pushes. Pull-request style commits keep their PR links and authors from the Gitea API; direct pushes are rendered as commit links using the commit author. It defaults to `scripts/changelog.json`, reads Gitea tokens from `GITEA_API_TOKEN`, `GITEA_TOKEN`, or `SOW_TOOLS_TOKEN`, and writes to stdout unless `--output` is provided. ## Intended Consumers - `sow-module` uses `sow-toolkit` for `build`, `build-module`, `extract`, `validate`, `compare`, and `apply-hak-manifest` - `sow-assets` uses `sow-toolkit` for `build-haks`, music processing, changelog generation, and validation - validation now fails early if duplicate asset resources would collide inside the same generated HAK - duplicate resources split across different HAK groups remain warnings because later HAK order can still override earlier content at runtime - `sow-module` now also uses `sow-toolkit` for topdata: - `validate-topdata` - `build-topdata` - `build-top-package` - `compare-topdata` - `convert-topdata` - `build-wiki` - `deploy-wiki` During development, sibling repositories can point at `../sow-tools/tools/sow-toolkit` or `../sow-tools/tools/sow-toolkit.exe`. For pinned releases, each consumer repo can instead carry its own copy in its local `tools/` directory. ## Cross-Repository Behavior The current ownership split is: - `sow-tools` - owns the shared CLI and build logic - owns module build, extract, compare, and HAK manifest application logic - owns topdata validation, build, compare, conversion, and packaging logic - `sow-module` - owns canonical module source under `src/` - owns canonical topdata authored data under `topdata/` - owns the user-facing module and topdata wrapper commands - `sow-assets` - owns canonical binary asset source under `assets/` - owns generated HAK manifests and HAK-side parts 2DA assets - owns the user-facing asset wrapper commands Consumer wrapper resolution is intentionally aligned: 1. Prefer a pinned local binary in `tools/` 2. Else prefer a sibling `../sow-tools` checkout for local source development 3. Else install the latest published `sow-tools` release artifact NWScript compilation behavior is also aligned and configurable under `scripts.compiler`: 1. Prefer `SOW_NWN_SCRIPT_COMPILER` when explicitly set 2. Else prefer a local compiler in `tools/script-compiler/` or `tools/` 3. Else use a system-installed `nwn_script_comp` from `PATH` When the compiler runs, the toolkit prepares `NWN_HOME` / `NWN_USER_DIRECTORY` automatically and tries to detect `NWN_ROOT` from common Steam locations, including custom Steam library folders, before an explicit override is required. ## Release Automation This repo publishes: - `sow-toolkit-linux-amd64.tar.gz` - `sow-toolkit-windows-amd64.exe` Consumer repos can fetch those binaries into their local `tools/` directory with: - `install-tool.sh` on Linux/macOS - `install-tool.ps1` on Windows ## TopData The native topdata system is implemented in this repo and authored in `sow-module`. High-level ownership split: - `sow-tools` - owns the `sow-toolkit` implementation - owns topdata validation, build, compare, conversion, and dataset-resolution logic - `sow-module` - owns canonical topdata authored data under `topdata/data/` - owns the user-facing topdata workflow scripts For a human-readable front-to-back explanation of how the native topdata builder works and how to use it, see: - [sow-module/topdata/README.md](../sow-module/topdata/README.md) Important topdata contracts still live here under `internal/topdata/` because they are code-adjacent implementation contracts, not user workflow docs.