546 lines
20 KiB
Markdown
546 lines
20 KiB
Markdown
# sow-tools
|
|
|
|
Shared NWN build tooling for Shadows Over Westgate.
|
|
|
|
This repository owns the Go-based `sow-toolkit` binary used by the separate module and assets repositories.
|
|
|
|
## Build
|
|
|
|
Build the development binary into `tools/sow-toolkit`:
|
|
|
|
```bash
|
|
./build-tool.sh
|
|
```
|
|
|
|
On Windows:
|
|
|
|
```powershell
|
|
.\build-tool.ps1
|
|
```
|
|
|
|
Both wrappers use `.cache/go-build` for the Go build cache and skip rebuilding
|
|
when the sources are older than the existing development binary.
|
|
|
|
Or run directly with Go:
|
|
|
|
```bash
|
|
GOCACHE="$PWD/.cache/go-build" go run ./cmd/nwn-tool --help
|
|
```
|
|
|
|
When changing native topdata id allocation, lockfile persistence, generated
|
|
families, or null-and-relocate behavior, run the repeat-build determinism test:
|
|
|
|
```bash
|
|
go test ./internal/topdata -run TestBuildNativeKeepsGeneratedIDsStableAcrossRepeatedBuilds
|
|
```
|
|
|
|
That test builds the same topdata project ten times in one process and asserts
|
|
that relocated/generated ids stay pinned across every pass.
|
|
|
|
## Repository Layout
|
|
|
|
```text
|
|
cmd/nwn-tool/ CLI entrypoint
|
|
internal/app/ command wiring and console UX
|
|
internal/erf/ ERF/MOD/HAK reading and writing
|
|
internal/gff/ canonical GFF JSON and binary conversion
|
|
internal/pipeline/ build, extract, compare, and manifest workflows
|
|
internal/project/ config loading and repo scanning
|
|
internal/topdata/ native topdata, wiki, and autogen workflows
|
|
internal/validator/validation and diagnostics
|
|
tools/ built development binary output
|
|
```
|
|
|
|
## Commands
|
|
|
|
The shared CLI exposes these project utilities:
|
|
|
|
```bash
|
|
sow-toolkit build
|
|
sow-toolkit build-module
|
|
sow-toolkit build-haks [--hak <name>] [--archive <name>] [--source-manifest <path>] [--plan-only] [--skip-music] [--music-dataset <id>]
|
|
sow-toolkit extract [<build-relative-archive-or-glob> ...]
|
|
sow-toolkit validate
|
|
sow-toolkit compare
|
|
sow-toolkit apply-hak-manifest [<manifest-path>]
|
|
sow-toolkit config <validate|effective|inspect|explain|sources>
|
|
sow-toolkit music <list-datasets|scan|build|credits|validate|manifest|normalize>
|
|
sow-toolkit validate-topdata
|
|
sow-toolkit build-topdata [--force] [--wiki]
|
|
sow-toolkit build-top-package [--force]
|
|
sow-toolkit compare-topdata
|
|
sow-toolkit convert-topdata <2da-to-json|2da-to-module|json-to-2da> ...
|
|
sow-toolkit build-wiki [--force]
|
|
sow-toolkit deploy-wiki [--source-dir <path>] [--endpoint <url>] [--token <token>] [--version <ver>] [--namespace <ns>] [--category <namespace=cid>] [--manifest <path>] [--stale-policy <report|archive|purge>] [--dry-run] [--create] [--force] [--reset-managed-namespaces]
|
|
sow-toolkit build-changelog [--config <path>] [--output <path>] [--current-tag <tag>] [--previous-tag <tag>] [--api-base-url <url>] [--token <token>]
|
|
```
|
|
|
|
Global logging flags `--quiet`, `--verbose`, and `--debug` may be passed after
|
|
the command name. Long options that take values accept either `--flag value` or
|
|
`--flag=value`; empty inline values such as `--dataset=` are rejected.
|
|
|
|
## Repository Configuration
|
|
|
|
Human-authored consumer repository configuration is YAML. The toolkit discovers
|
|
configuration in this order:
|
|
|
|
1. `nwn-tool.yaml`
|
|
2. `nwn-tool.yml`
|
|
3. legacy `nwn-tool.json`
|
|
|
|
YAML is canonical. If both YAML and legacy JSON files are present, the YAML file
|
|
wins. Legacy `nwn-tool.json` still loads for migration, but commands print a
|
|
warning and should be updated to `nwn-tool.yaml`.
|
|
|
|
Example:
|
|
|
|
```yaml
|
|
module:
|
|
name: Shadows Over Westgate
|
|
resref: sow_module
|
|
description: Shadows Over Westgate asset pipeline.
|
|
hak_order:
|
|
- sow_top
|
|
- group:sow_over
|
|
- group:sow_core
|
|
|
|
paths:
|
|
assets: content
|
|
build: build
|
|
|
|
music:
|
|
# Legacy shorthand — automatically normalized into a dataset.
|
|
prefixes:
|
|
envi/music/westgate: mus_wg_
|
|
|
|
# Explicit dataset config (replaces prefixes above).
|
|
tools:
|
|
ffmpeg: tools/ffmpeg
|
|
ffprobe: tools/ffprobe
|
|
defaults:
|
|
stage_root: "{paths.cache}/music"
|
|
credits_root: "{paths.cache}/credits"
|
|
credits_overlay: CREDITS.md
|
|
max_stem_length: 16
|
|
naming_scheme: nwn_bmu
|
|
output_extension: .bmu
|
|
convert_extensions: [.flac, .m4a, .mp3, .ogg, .wav]
|
|
datasets:
|
|
westgate:
|
|
source: envi/music/westgate
|
|
output: envi/music/westgate
|
|
prefix: mus_wg_
|
|
package_mode: hak_asset
|
|
```
|
|
|
|
Resolved configuration can be inspected without scanning or building:
|
|
|
|
```bash
|
|
sow-toolkit config validate
|
|
sow-toolkit config effective --json
|
|
sow-toolkit config inspect paths.build
|
|
sow-toolkit config explain topdata.package_hak
|
|
sow-toolkit config sources
|
|
```
|
|
|
|
`config effective` prints the normalized values consumed by commands, including
|
|
toolkit defaults, YAML values, legacy status, active environment overrides, and
|
|
provenance. Sensitive override values such as tokens are reported as set without
|
|
printing the secret.
|
|
|
|
Common configurable defaults:
|
|
|
|
```yaml
|
|
paths:
|
|
source: src
|
|
assets: assets
|
|
build: build
|
|
cache: .cache
|
|
tools: tools
|
|
|
|
outputs:
|
|
module_archive: "{module.resref}.mod"
|
|
hak_manifest: haks.json
|
|
hak_archive: "{hak.name}.hak"
|
|
|
|
build:
|
|
keep_existing_haks: false
|
|
|
|
generated_assets:
|
|
topdata_2da:
|
|
- id: parts
|
|
source: topdata
|
|
output: "{paths.cache}/generated-assets/parts-2da"
|
|
include_datasets: [parts/**]
|
|
package_root: part
|
|
autogen:
|
|
id: parts
|
|
mode: parts_rows
|
|
root: part
|
|
include: ["**/*.mdl"]
|
|
derive:
|
|
kind: trailing_numeric_suffix
|
|
group_from: first_path_segment
|
|
parts_rows:
|
|
row_defaults:
|
|
COSTMODIFIER: "0"
|
|
acbonus:
|
|
default:
|
|
strategy: ascending_row_id_sort_key
|
|
divisor: 100
|
|
format: "%.2f"
|
|
datasets:
|
|
chest:
|
|
strategy: fixed
|
|
value: "0.00"
|
|
|
|
inventory:
|
|
source_extensions: [] # empty means built-in NWN source extensions
|
|
asset_extensions: [] # empty means built-in NWN asset extensions
|
|
source_json_pattern: "{resref}.{extension}.json"
|
|
|
|
validation:
|
|
profile: nwn_module
|
|
builtin_script_prefixes: [nw_, x0_, x1_, x2_, x3_, ga_, gc_, gen_, gui_, nwg_, ta_]
|
|
required_fields:
|
|
ifo: [Mod_Name]
|
|
|
|
scripts:
|
|
source_dir: scripts
|
|
cache: "{paths.cache}/ncs"
|
|
compiler:
|
|
path: ""
|
|
env:
|
|
path: SOW_NWN_SCRIPT_COMPILER
|
|
nwn_root: [SOW_NWN_ROOT, NWN_ROOT]
|
|
nwn_user_directory: [SOW_NWN_USER_DIRECTORY, NWN_HOME, NWN_USER_DIRECTORY]
|
|
search:
|
|
- "{paths.tools}/script-compiler/nwn_script_comp"
|
|
- "{paths.tools}/script-compiler/nwn_script_comp.exe"
|
|
- "{paths.tools}/nwn_script_comp"
|
|
- "{paths.tools}/nwn_script_comp.exe"
|
|
- PATH:nwn_script_comp
|
|
|
|
extract:
|
|
layout: nwn_canonical_json
|
|
archives:
|
|
- "{module.resref}.mod"
|
|
cleanup_stale: true
|
|
consume_archives: false
|
|
ignore_extensions: []
|
|
delete_module_archive_after_success: false # deprecated compatibility alias
|
|
|
|
topdata:
|
|
build: "{paths.build}/topdata"
|
|
compiled_2da_dir: 2da
|
|
compiled_tlk: sow_tlk.tlk
|
|
package_hak: sow_top.hak
|
|
package_tlk: sow_tlk.tlk
|
|
wiki:
|
|
output_root: wiki
|
|
pages_dir: pages
|
|
state_file: state.json
|
|
managed_namespaces: [classes, feat, itemtypes, races, skills, spells, meta]
|
|
deploy_manifest: .wiki_deploy_manifest.json
|
|
deploy_edit_summary: Auto-generated from native builder
|
|
visibility_file: visibility.yaml
|
|
title_prefix_min_length: 3
|
|
status_listing_scope: all
|
|
|
|
autogen:
|
|
cache:
|
|
root: "{paths.cache}"
|
|
max_age: 1h
|
|
refresh_env: SOW_AUTOGEN_MANIFEST_REFRESH
|
|
release_source:
|
|
provider: gitea
|
|
server_url_env: SOW_ASSETS_SERVER_URL
|
|
repo_env: SOW_ASSETS_REPO
|
|
```
|
|
|
|
`paths.source` and `paths.assets` must name real repository subtrees when they
|
|
are configured. They may be omitted for repositories that do not own that class
|
|
of source, but they must not resolve to the repository root (`.`). Output and
|
|
cache paths must stay relative to the repository root unless a specific setting
|
|
documents otherwise.
|
|
|
|
`generated_assets.topdata_2da` is for HAK repositories that need a small
|
|
native-topdata subset packaged as HAK resources. It builds only `.2da` output,
|
|
injects those generated files into `build-haks`, and rejects TLK-backed text
|
|
because this path does not produce TLK packages. Static generated groups can
|
|
omit `autogen.mode`; in that case the selected datasets are compiled directly,
|
|
and `package_root` plus the repository's HAK include patterns determine where
|
|
the resources are packaged.
|
|
|
|
For `cachedmodels_rows` generated assets, the configured autogen scan appends
|
|
discovered model stems to the selected `cachedmodels` dataset without `.mdl`
|
|
extensions and skips models already present in authored rows.
|
|
|
|
For `parts_rows` generated assets, `parts_rows.acbonus` can normalize final
|
|
ACBONUS values before module overrides are applied. The
|
|
`ascending_row_id_sort_key` strategy writes toolset ordering values from row
|
|
IDs so higher part rows can sort before lower rows in the toolset; the legacy
|
|
`descending_row_id_sort_key` strategy remains available for repositories that
|
|
already depend on it. Per-dataset policies such as `chest: fixed 0.00` can
|
|
override the default. Authored source ACBONUS values are not authoritative when
|
|
this policy is configured, except that authored `****` values remain null
|
|
unless model discovery activates the row. Dense gaps without authored rows or
|
|
discovered part models are emitted as `****` null rows. Files under
|
|
`parts/modules` remain the final override layer.
|
|
|
|
Extraction is deliberately guarded because it writes source files from binary
|
|
archives. `extract.archives` is a list of glob patterns relative to
|
|
`paths.build`; the default is only the configured module `.mod`. HAK extraction
|
|
is opt-in, for example:
|
|
|
|
```yaml
|
|
extract:
|
|
archives:
|
|
- "{module.resref}.mod"
|
|
- "sow_over*.hak"
|
|
```
|
|
|
|
Module resources require a configured `paths.source`; HAK assets require a
|
|
configured `paths.assets`; both roots must resolve to safe subtrees. If a root
|
|
is unset or resolves to the repository root, extraction fails before writing
|
|
instead of creating extension directories like `mdl/` or `png/` at the repo top
|
|
level. `.erf` files are rejected by extraction because they do not contain
|
|
`module.ifo`, so they cannot safely update module-level area membership.
|
|
|
|
`consume_archives: true` deletes the selected build archive files only after the
|
|
entire extraction succeeds. The older `delete_module_archive_after_success`
|
|
setting remains as a module-only compatibility alias when `consume_archives` is
|
|
not set. Stale cleanup is scoped to roots actually touched by extracted
|
|
resources, so the default module-only extraction does not prune assets.
|
|
|
|
`topdata.wiki.status_listing_scope` controls the generated wiki-status detail
|
|
pages. The default `all` writes both per-namespace listings and aggregate
|
|
`meta:wikistatus:<status>` listings. Use `namespace` when aggregate status
|
|
lists would be too large for the target NodeBB post limit.
|
|
|
|
GFF JSON extraction can merge selected fields and lists from the existing source
|
|
file instead of replacing the whole extracted document. Rules are matched by
|
|
`target` relative to `paths.source`; files without a matching rule keep the
|
|
default overwrite behavior. For example:
|
|
|
|
```yaml
|
|
extract:
|
|
merge:
|
|
gff_json:
|
|
- target: module/module.ifo.json
|
|
preserve_fields:
|
|
- Mod_Entry_Area
|
|
- Mod_Entry_X
|
|
- Mod_Entry_Y
|
|
- Mod_Entry_Z
|
|
- Mod_Entry_Dir_X
|
|
- Mod_Entry_Dir_Y
|
|
merge_lists:
|
|
- field: Mod_Area_list
|
|
key_field: Area_Name
|
|
strategy: preserve_existing_order_append_new
|
|
```
|
|
|
|
`preserve_fields` copies matching root fields from the existing source document
|
|
into the extracted document before writing. The
|
|
`preserve_existing_order_append_new` list strategy keeps existing keyed list
|
|
order for entries still present in the extracted archive, uses extracted entries
|
|
as the value source, appends archive-only entries in archive order, and removes
|
|
existing entries that are no longer present in the archive.
|
|
|
|
Music defaults (used when not explicitly configured):
|
|
|
|
```yaml
|
|
music:
|
|
defaults:
|
|
stage_root: "{paths.cache}/music"
|
|
credits_root: "{paths.cache}/credits"
|
|
credits_overlay: CREDITS.md
|
|
max_stem_length: 16
|
|
naming_scheme: nwn_bmu
|
|
output_extension: .bmu
|
|
convert_extensions: [.flac, .m4a, .mp3, .ogg, .wav]
|
|
```
|
|
|
|
Music datasets define individual music collections with independent sources and
|
|
naming prefixes. `source` is the authored input root under `paths.assets`;
|
|
`output` is the generated HAK asset root matched by HAK include globs. If
|
|
`output` is omitted, it defaults to `source`. `package_mode: hak_asset` converts
|
|
configured source extensions into generated HAK resources; `package_mode: none`
|
|
scans and writes credits without adding generated HAK resources. The legacy
|
|
`music.prefixes` shorthand is automatically normalized into datasets.
|
|
`convert_extensions` is configuration-driven; the built-in default set is
|
|
`.flac`, `.m4a`, `.mp3`, `.ogg`, and `.wav`.
|
|
Those extensions are discovered from YAML within configured music dataset source
|
|
roots, so setting them in `music.defaults.convert_extensions` or a dataset
|
|
override is sufficient; you do not need a parallel
|
|
`inventory.asset_extensions` override just to make music scanning see them.
|
|
|
|
Music can also be run directly:
|
|
|
|
```bash
|
|
sow-toolkit music list-datasets
|
|
sow-toolkit music scan --dataset westgate
|
|
sow-toolkit music build --dataset westgate --dry-run
|
|
sow-toolkit music build --dataset westgate
|
|
sow-toolkit music credits --dataset westgate --check
|
|
sow-toolkit music validate --dataset westgate --json
|
|
sow-toolkit music manifest --dataset westgate
|
|
sow-toolkit music normalize --dataset westgate
|
|
```
|
|
|
|
`build-haks` uses the same music pipeline. Use `--plan-only` or music
|
|
`--dry-run` to compute mappings without transcoding. Use `build-haks
|
|
--skip-music` to package authored assets only, or `build-haks --music-dataset
|
|
westgate` to limit conversion to a configured dataset.
|
|
|
|
Generated and machine-readable artifacts remain JSON, including HAK manifests,
|
|
topdata datasets, caches, credits inventories, build reports, and canonical GFF
|
|
documents. Nested metadata such as `topdata/templates/config.yaml` remains
|
|
valid when it is local workflow configuration rather than root repository
|
|
configuration.
|
|
|
|
## Wiki And Changelog Utilities
|
|
|
|
`build-topdata --wiki` or `build-wiki` renders wiki pages from the current
|
|
native topdata state. `deploy-wiki` publishes those pages to NodeBB. It reads
|
|
`NODEBB_API_ENDPOINT`, `NODEBB_API_TOKEN`, `GITHUB_REF_NAME`, and
|
|
`NODEBB_WIKI_CATEGORIES` when equivalent flags are omitted. Category mappings
|
|
use `namespace=cid`, for example:
|
|
|
|
```bash
|
|
sow-toolkit deploy-wiki --dry-run --namespace spells --category spells=42
|
|
```
|
|
|
|
Namespace declarations in `topdata/wiki/namespaces.yaml` may set
|
|
`canonical_path` when a generated namespace lives below a fuller NodeBB category
|
|
path, for example `canonical_path: Wiki/Feats`. The `title` remains the display
|
|
label, while `canonical_path` drives generated `public_path` metadata, wiki
|
|
links, page-index entries, and deploy adoption matching.
|
|
|
|
When a deploy manifest is missing but matching wiki pages already exist in
|
|
NodeBB, `deploy-wiki` checks the Westgate Wiki namespace directory API and
|
|
adopts those topic/post mappings before planning updates. It does the same when
|
|
a cached manifest mapping points to a post NodeBB no longer has. Canonical
|
|
`wikiPath`/`canonicalPath` data is preferred during adoption. Nested generated
|
|
paths require that exact canonical path data; leaf/title compatibility matching
|
|
is only used where it cannot cross a canonical parent path. `--create` is still
|
|
required for genuinely new pages that have no existing remote topic. If NodeBB
|
|
rejects a create because the clean wiki URL already exists, `deploy-wiki`
|
|
re-queries the namespace by page leaf, adopts the matching topic, and updates it
|
|
instead. Pass `--debug` when deploying large wiki changes to emit planning and
|
|
live mutation progress. Updates and stale-page archives acquire a Westgate Wiki
|
|
edit lock before saving, then send the returned
|
|
`wikiEditLockToken` with the NodeBB post edit request. Stale generated pages are
|
|
reported by default.
|
|
`--stale-policy archive` rewrites stale generated pages in place as archived
|
|
managed content. `--stale-policy purge` permanently removes only stale generated
|
|
topics already tracked in the wiki deploy manifest. Dry-run that policy before
|
|
the live run; it never discovers manual wiki pages from NodeBB for purge.
|
|
`--reset-managed-namespaces` is a stronger destructive mode for slug or path
|
|
standard migrations: it permanently removes every topic currently listed by the
|
|
configured managed namespace categories, ignores the previous deploy manifest,
|
|
and recreates the current generated pages. It requires `--create` when local
|
|
generated pages are present.
|
|
|
|
For newly created NodeBB topics, `topdata.wiki.title_prefix_min_length`
|
|
controls when the deployer prefixes the namespace in the topic title. The
|
|
default is `3`, so one- and two-character page titles are created as
|
|
`Namespace: Title`, while titles with three or more characters keep the page
|
|
title alone. Existing bot-managed pages with older prefixed titles are renamed
|
|
on later deploy passes when their current title no longer matches this policy.
|
|
|
|
`build-changelog` renders a release changelog from tagged pushes. Pull-request
|
|
style commits keep their PR links and authors from the Gitea API; direct pushes
|
|
are rendered as commit links using the commit author. It defaults to
|
|
`scripts/changelog.json`, reads Gitea tokens from `GITEA_API_TOKEN`,
|
|
`GITEA_TOKEN`, or `SOW_TOOLS_TOKEN`, and writes to stdout unless `--output` is
|
|
provided.
|
|
|
|
## Intended Consumers
|
|
|
|
- `sow-module` uses `sow-toolkit` for `build`, `build-module`, `extract`, `validate`, `compare`, and `apply-hak-manifest`
|
|
- `sow-assets` uses `sow-toolkit` for `build-haks`, music processing, changelog generation, and validation
|
|
- validation now fails early if duplicate asset resources would collide inside the same generated HAK
|
|
- duplicate resources split across different HAK groups remain warnings because later HAK order can still override earlier content at runtime
|
|
- `sow-module` now also uses `sow-toolkit` for topdata:
|
|
- `validate-topdata`
|
|
- `build-topdata`
|
|
- `build-top-package`
|
|
- `compare-topdata`
|
|
- `convert-topdata`
|
|
- `build-wiki`
|
|
- `deploy-wiki`
|
|
|
|
During development, sibling repositories can point at `../sow-tools/tools/sow-toolkit`
|
|
or `../sow-tools/tools/sow-toolkit.exe`. For pinned releases, each consumer repo can
|
|
instead carry its own copy in its local `tools/` directory.
|
|
|
|
## Cross-Repository Behavior
|
|
|
|
The current ownership split is:
|
|
|
|
- `sow-tools`
|
|
- owns the shared CLI and build logic
|
|
- owns module build, extract, compare, and HAK manifest application logic
|
|
- owns topdata validation, build, compare, conversion, and packaging logic
|
|
- `sow-module`
|
|
- owns canonical module source under `src/`
|
|
- owns canonical topdata authored data under `topdata/`
|
|
- owns the user-facing module and topdata wrapper commands
|
|
- `sow-assets`
|
|
- owns canonical binary asset source under `assets/`
|
|
- owns generated HAK manifests and HAK-side parts 2DA assets
|
|
- owns the user-facing asset wrapper commands
|
|
|
|
Consumer wrapper resolution is intentionally aligned:
|
|
|
|
1. Prefer a pinned local binary in `tools/`
|
|
2. Else prefer a sibling `../sow-tools` checkout for local source development
|
|
3. Else install the latest published `sow-tools` release artifact
|
|
|
|
NWScript compilation behavior is also aligned and configurable under
|
|
`scripts.compiler`:
|
|
|
|
1. Prefer `SOW_NWN_SCRIPT_COMPILER` when explicitly set
|
|
2. Else prefer a local compiler in `tools/script-compiler/` or `tools/`
|
|
3. Else use a system-installed `nwn_script_comp` from `PATH`
|
|
|
|
When the compiler runs, the toolkit prepares `NWN_HOME` / `NWN_USER_DIRECTORY`
|
|
automatically and tries to detect `NWN_ROOT` from common Steam locations, including
|
|
custom Steam library folders, before an explicit override is required.
|
|
|
|
|
|
## Release Automation
|
|
|
|
This repo publishes:
|
|
|
|
- `sow-toolkit-linux-amd64.tar.gz`
|
|
- `sow-toolkit-windows-amd64.exe`
|
|
|
|
Consumer repos can fetch those binaries into their local `tools/` directory with:
|
|
|
|
- `install-tool.sh` on Linux/macOS
|
|
- `install-tool.ps1` on Windows
|
|
|
|
## TopData
|
|
|
|
The native topdata system is implemented in this repo and authored in `sow-module`.
|
|
|
|
High-level ownership split:
|
|
|
|
- `sow-tools`
|
|
- owns the `sow-toolkit` implementation
|
|
- owns topdata validation, build, compare, conversion, and dataset-resolution logic
|
|
- `sow-module`
|
|
- owns canonical topdata authored data under `topdata/data/`
|
|
- owns the user-facing topdata workflow scripts
|
|
|
|
For a human-readable front-to-back explanation of how the native topdata builder works
|
|
and how to use it, see:
|
|
|
|
- [sow-module/topdata/README.md](../sow-module/topdata/README.md)
|
|
|
|
Important topdata contracts still live here under `internal/topdata/` because they are
|
|
code-adjacent implementation contracts, not user workflow docs.
|