sow-tools

Shared NWN build tooling for Shadows Over Westgate.

This repository owns the Go-based sow-toolkit binary used by the separate module and assets repositories.

Build

Build the development binary into tools/sow-toolkit:

./build-tool.sh

On Windows:

.\build-tool.ps1

Both wrappers use .cache/go-build for the Go build cache and skip rebuilding when the sources are older than the existing development binary.

Or run directly with Go:

GOCACHE="$PWD/.cache/go-build" go run ./cmd/nwn-tool --help

When changing native topdata id allocation, lockfile persistence, generated families, or null-and-relocate behavior, run the repeat-build determinism test:

go test ./internal/topdata -run TestBuildNativeKeepsGeneratedIDsStableAcrossRepeatedBuilds

That test builds the same topdata project ten times in one process and asserts that relocated/generated ids stay pinned across every pass.

Repository Layout

cmd/nwn-tool/      CLI entrypoint
internal/app/      command wiring and console UX
internal/erf/      ERF/MOD/HAK reading and writing
internal/gff/      canonical GFF JSON and binary conversion
internal/pipeline/ build, extract, compare, and manifest workflows
internal/project/  config loading and repo scanning
internal/topdata/  native topdata, wiki, and autogen workflows
internal/validator/validation and diagnostics
tools/             built development binary output

Commands

The shared CLI exposes these project utilities:

sow-toolkit build
sow-toolkit build-module
sow-toolkit build-haks [--hak <name>] [--archive <name>] [--source-manifest <path>] [--plan-only] [--skip-music] [--music-dataset <id>]
sow-toolkit extract [<build-relative-archive-or-glob> ...]
sow-toolkit validate
sow-toolkit compare
sow-toolkit apply-hak-manifest [<manifest-path>]
sow-toolkit config <validate|effective|inspect|explain|sources>
sow-toolkit music <list-datasets|scan|build|credits|validate|manifest|normalize>
sow-toolkit validate-topdata
sow-toolkit build-topdata [--force] [--wiki]
sow-toolkit build-top-package [--force]
sow-toolkit compare-topdata
sow-toolkit convert-topdata <2da-to-json|2da-to-module|json-to-2da> ...
sow-toolkit build-wiki [--force]
sow-toolkit deploy-wiki [--source-dir <path>] [--endpoint <url>] [--token <token>] [--version <ver>] [--namespace <ns>] [--category <namespace=cid>] [--manifest <path>] [--stale-policy <report|archive|purge>] [--dry-run] [--create] [--force]
sow-toolkit build-changelog [--config <path>] [--output <path>] [--current-tag <tag>] [--previous-tag <tag>] [--api-base-url <url>] [--token <token>]

Global logging flags --quiet, --verbose, and --debug may be passed after the command name. Long options that take values accept either --flag value or --flag=value; empty inline values such as --dataset= are rejected.

Repository Configuration

Human-authored consumer repository configuration is YAML. The toolkit discovers configuration in this order:

1. nwn-tool.yaml
2. nwn-tool.yml
3. legacy nwn-tool.json

YAML is canonical. If both YAML and legacy JSON files are present, the YAML file wins. Legacy nwn-tool.json still loads for migration, but commands print a warning and should be updated to nwn-tool.yaml.

Example:

module:
  name: Shadows Over Westgate
  resref: sow_module
  description: Shadows Over Westgate asset pipeline.
  hak_order:
    - sow_top
    - group:sow_over
    - group:sow_core

paths:
  assets: content
  build: build

music:
  # Legacy shorthand — automatically normalized into a dataset.
  prefixes:
    envi/music/westgate: mus_wg_

  # Explicit dataset config (replaces prefixes above).
  tools:
    ffmpeg: tools/ffmpeg
    ffprobe: tools/ffprobe
  defaults:
    stage_root: "{paths.cache}/music"
    credits_root: "{paths.cache}/credits"
    credits_overlay: CREDITS.md
    max_stem_length: 16
    naming_scheme: nwn_bmu
    output_extension: .bmu
    convert_extensions: [.flac, .m4a, .mp3, .ogg, .wav]
  datasets:
    westgate:
      source: envi/music/westgate
      output: envi/music/westgate
      prefix: mus_wg_
      package_mode: hak_asset

Resolved configuration can be inspected without scanning or building:

sow-toolkit config validate
sow-toolkit config effective --json
sow-toolkit config inspect paths.build
sow-toolkit config explain topdata.package_hak
sow-toolkit config sources

config effective prints the normalized values consumed by commands, including toolkit defaults, YAML values, legacy status, active environment overrides, and provenance. Sensitive override values such as tokens are reported as set without printing the secret.

Common configurable defaults:

paths:
  source: src
  assets: assets
  build: build
  cache: .cache
  tools: tools

outputs:
  module_archive: "{module.resref}.mod"
  hak_manifest: haks.json
  hak_archive: "{hak.name}.hak"

build:
  keep_existing_haks: false

generated_assets:
  topdata_2da:
    - id: parts
      source: topdata
      output: "{paths.cache}/generated-assets/parts-2da"
      include_datasets: [parts/**]
      package_root: part
      autogen:
        id: parts
        mode: parts_rows
        root: part
        include: ["**/*.mdl"]
        derive:
          kind: trailing_numeric_suffix
          group_from: first_path_segment
        parts_rows:
          row_defaults:
            COSTMODIFIER: "0"
          acbonus:
            default:
              strategy: ascending_row_id_sort_key
              divisor: 100
              format: "%.2f"
            datasets:
              chest:
                strategy: fixed
                value: "0.00"

inventory:
  source_extensions: []      # empty means built-in NWN source extensions
  asset_extensions: []       # empty means built-in NWN asset extensions
  source_json_pattern: "{resref}.{extension}.json"

validation:
  profile: nwn_module
  builtin_script_prefixes: [nw_, x0_, x1_, x2_, x3_, ga_, gc_, gen_, gui_, nwg_, ta_]
  required_fields:
    ifo: [Mod_Name]

scripts:
  source_dir: scripts
  cache: "{paths.cache}/ncs"
  compiler:
    path: ""
    env:
      path: SOW_NWN_SCRIPT_COMPILER
      nwn_root: [SOW_NWN_ROOT, NWN_ROOT]
      nwn_user_directory: [SOW_NWN_USER_DIRECTORY, NWN_HOME, NWN_USER_DIRECTORY]
    search:
      - "{paths.tools}/script-compiler/nwn_script_comp"
      - "{paths.tools}/script-compiler/nwn_script_comp.exe"
      - "{paths.tools}/nwn_script_comp"
      - "{paths.tools}/nwn_script_comp.exe"
      - PATH:nwn_script_comp

extract:
  layout: nwn_canonical_json
  archives:
    - "{module.resref}.mod"
  cleanup_stale: true
  consume_archives: false
  ignore_extensions: []
  delete_module_archive_after_success: false # deprecated compatibility alias

topdata:
  build: "{paths.build}/topdata"
  compiled_2da_dir: 2da
  compiled_tlk: sow_tlk.tlk
  package_hak: sow_top.hak
  package_tlk: sow_tlk.tlk
  wiki:
    output_root: wiki
    pages_dir: pages
    state_file: state.json
    managed_namespaces: [classes, feat, itemtypes, races, skills, spells, meta]
    deploy_manifest: .wiki_deploy_manifest.json
    deploy_edit_summary: Auto-generated from native builder
    visibility_file: visibility.yaml
    title_prefix_min_length: 3
    status_listing_scope: all

autogen:
  cache:
    root: "{paths.cache}"
    max_age: 1h
    refresh_env: SOW_AUTOGEN_MANIFEST_REFRESH
  release_source:
    provider: gitea
    server_url_env: SOW_ASSETS_SERVER_URL
    repo_env: SOW_ASSETS_REPO

paths.source and paths.assets must name real repository subtrees when they are configured. They may be omitted for repositories that do not own that class of source, but they must not resolve to the repository root (.). Output and cache paths must stay relative to the repository root unless a specific setting documents otherwise.

generated_assets.topdata_2da is for HAK repositories that need a small native-topdata subset packaged as HAK resources. It builds only .2da output, injects those generated files into build-haks, and rejects TLK-backed text because this path does not produce TLK packages.

For parts_rows generated assets, parts_rows.acbonus can normalize final ACBONUS values before module overrides are applied. The ascending_row_id_sort_key strategy writes toolset ordering values from row IDs so higher part rows can sort before lower rows in the toolset; the legacy descending_row_id_sort_key strategy remains available for repositories that already depend on it. Per-dataset policies such as chest: fixed 0.00 can override the default. Authored source ACBONUS values are not authoritative when this policy is configured, except that authored **** values remain null unless model discovery activates the row. Dense gaps without authored rows or discovered part models are emitted as **** null rows. Files under parts/modules remain the final override layer.

Extraction is deliberately guarded because it writes source files from binary archives. extract.archives is a list of glob patterns relative to paths.build; the default is only the configured module .mod. HAK extraction is opt-in, for example:

extract:
  archives:
    - "{module.resref}.mod"
    - "sow_over*.hak"

Module resources require a configured paths.source; HAK assets require a configured paths.assets; both roots must resolve to safe subtrees. If a root is unset or resolves to the repository root, extraction fails before writing instead of creating extension directories like mdl/ or png/ at the repo top level. .erf files are rejected by extraction because they do not contain module.ifo, so they cannot safely update module-level area membership.

consume_archives: true deletes the selected build archive files only after the entire extraction succeeds. The older delete_module_archive_after_success setting remains as a module-only compatibility alias when consume_archives is not set. Stale cleanup is scoped to roots actually touched by extracted resources, so the default module-only extraction does not prune assets.

topdata.wiki.status_listing_scope controls the generated wiki-status detail pages. The default all writes both per-namespace listings and aggregate meta:wikistatus:<status> listings. Use namespace when aggregate status lists would be too large for the target NodeBB post limit.

GFF JSON extraction can merge selected fields and lists from the existing source file instead of replacing the whole extracted document. Rules are matched by target relative to paths.source; files without a matching rule keep the default overwrite behavior. For example:

extract:
  merge:
    gff_json:
      - target: module/module.ifo.json
        preserve_fields:
          - Mod_Entry_Area
          - Mod_Entry_X
          - Mod_Entry_Y
          - Mod_Entry_Z
          - Mod_Entry_Dir_X
          - Mod_Entry_Dir_Y
        merge_lists:
          - field: Mod_Area_list
            key_field: Area_Name
            strategy: preserve_existing_order_append_new

preserve_fields copies matching root fields from the existing source document into the extracted document before writing. The preserve_existing_order_append_new list strategy keeps existing keyed list order for entries still present in the extracted archive, uses extracted entries as the value source, appends archive-only entries in archive order, and removes existing entries that are no longer present in the archive.

Music defaults (used when not explicitly configured):

music:
  defaults:
    stage_root: "{paths.cache}/music"
    credits_root: "{paths.cache}/credits"
    credits_overlay: CREDITS.md
    max_stem_length: 16
    naming_scheme: nwn_bmu
    output_extension: .bmu
    convert_extensions: [.flac, .m4a, .mp3, .ogg, .wav]

Music datasets define individual music collections with independent sources and naming prefixes. source is the authored input root under paths.assets; output is the generated HAK asset root matched by HAK include globs. If output is omitted, it defaults to source. package_mode: hak_asset converts configured source extensions into generated HAK resources; package_mode: none scans and writes credits without adding generated HAK resources. The legacy music.prefixes shorthand is automatically normalized into datasets. convert_extensions is configuration-driven; the built-in default set is .flac, .m4a, .mp3, .ogg, and .wav. Those extensions are discovered from YAML within configured music dataset source roots, so setting them in music.defaults.convert_extensions or a dataset override is sufficient; you do not need a parallel inventory.asset_extensions override just to make music scanning see them.

Music can also be run directly:

sow-toolkit music list-datasets
sow-toolkit music scan --dataset westgate
sow-toolkit music build --dataset westgate --dry-run
sow-toolkit music build --dataset westgate
sow-toolkit music credits --dataset westgate --check
sow-toolkit music validate --dataset westgate --json
sow-toolkit music manifest --dataset westgate
sow-toolkit music normalize --dataset westgate

build-haks uses the same music pipeline. Use --plan-only or music --dry-run to compute mappings without transcoding. Use build-haks --skip-music to package authored assets only, or build-haks --music-dataset westgate to limit conversion to a configured dataset.

Generated and machine-readable artifacts remain JSON, including HAK manifests, topdata datasets, caches, credits inventories, build reports, and canonical GFF documents. Nested metadata such as topdata/templates/config.yaml remains valid when it is local workflow configuration rather than root repository configuration.

Wiki And Changelog Utilities

build-topdata --wiki or build-wiki renders wiki pages from the current native topdata state. deploy-wiki publishes those pages to NodeBB. It reads NODEBB_API_ENDPOINT, NODEBB_API_TOKEN, GITHUB_REF_NAME, and NODEBB_WIKI_CATEGORIES when equivalent flags are omitted. Category mappings use namespace=cid, for example:

sow-toolkit deploy-wiki --dry-run --namespace spells --category spells=42

When a deploy manifest is missing but matching wiki pages already exist in NodeBB, deploy-wiki checks the Westgate Wiki namespace directory API and adopts those topic/post mappings before planning updates. It does the same when a cached manifest mapping points to a post NodeBB no longer has. Generated public slugs are preferred during adoption before title-based compatibility matching. --create is still required for genuinely new pages that have no existing remote topic. If NodeBB rejects a create because the clean wiki URL already exists, deploy-wiki re-queries the namespace by page leaf, adopts the matching topic, and updates it instead. Updates and stale-page archives acquire a Westgate Wiki edit lock before saving, then send the returned wikiEditLockToken with the NodeBB post edit request. Stale generated pages are reported by default. --stale-policy archive rewrites stale generated pages in place as archived managed content. --stale-policy purge permanently removes only stale generated topics already tracked in the wiki deploy manifest. Dry-run that policy before the live run; it never discovers manual wiki pages from NodeBB for purge.

For newly created NodeBB topics, topdata.wiki.title_prefix_min_length controls when the deployer prefixes the namespace in the topic title. The default is 3, so one- and two-character page titles are created as Namespace: Title, while titles with three or more characters keep the page title alone. Existing bot-managed pages with older prefixed titles are renamed on later deploy passes when their current title no longer matches this policy.

build-changelog renders a release changelog from tagged pushes. Pull-request style commits keep their PR links and authors from the Gitea API; direct pushes are rendered as commit links using the commit author. It defaults to scripts/changelog.json, reads Gitea tokens from GITEA_API_TOKEN, GITEA_TOKEN, or SOW_TOOLS_TOKEN, and writes to stdout unless --output is provided.

Intended Consumers

• sow-module uses sow-toolkit for build, build-module, extract, validate, compare, and apply-hak-manifest
• sow-assets uses sow-toolkit for build-haks, music processing, changelog generation, and validation
  • validation now fails early if duplicate asset resources would collide inside the same generated HAK
  • duplicate resources split across different HAK groups remain warnings because later HAK order can still override earlier content at runtime
• sow-module now also uses sow-toolkit for topdata:
  • validate-topdata
  • build-topdata
  • build-top-package
  • compare-topdata
  • convert-topdata
  • build-wiki
  • deploy-wiki

During development, sibling repositories can point at ../sow-tools/tools/sow-toolkit or ../sow-tools/tools/sow-toolkit.exe. For pinned releases, each consumer repo can instead carry its own copy in its local tools/ directory.

Cross-Repository Behavior

The current ownership split is:

• sow-tools
  • owns the shared CLI and build logic
  • owns module build, extract, compare, and HAK manifest application logic
  • owns topdata validation, build, compare, conversion, and packaging logic
• sow-module
  • owns canonical module source under src/
  • owns canonical topdata authored data under topdata/
  • owns the user-facing module and topdata wrapper commands
• sow-assets
  • owns canonical binary asset source under assets/
  • owns generated HAK manifests and HAK-side parts 2DA assets
  • owns the user-facing asset wrapper commands

Consumer wrapper resolution is intentionally aligned:

1. Prefer a pinned local binary in tools/
2. Else prefer a sibling ../sow-tools checkout for local source development
3. Else install the latest published sow-tools release artifact

NWScript compilation behavior is also aligned and configurable under scripts.compiler:

1. Prefer SOW_NWN_SCRIPT_COMPILER when explicitly set
2. Else prefer a local compiler in tools/script-compiler/ or tools/
3. Else use a system-installed nwn_script_comp from PATH

When the compiler runs, the toolkit prepares NWN_HOME / NWN_USER_DIRECTORY automatically and tries to detect NWN_ROOT from common Steam locations, including custom Steam library folders, before an explicit override is required.

Release Automation

This repo publishes:

• sow-toolkit-linux-amd64.tar.gz
• sow-toolkit-windows-amd64.exe

Consumer repos can fetch those binaries into their local tools/ directory with:

• install-tool.sh on Linux/macOS
• install-tool.ps1 on Windows

TopData

The native topdata system is implemented in this repo and authored in sow-module.

High-level ownership split:

• sow-tools
  • owns the sow-toolkit implementation
  • owns topdata validation, build, compare, conversion, and dataset-resolution logic
• sow-module
  • owns canonical topdata authored data under topdata/data/
  • owns the user-facing topdata workflow scripts

For a human-readable front-to-back explanation of how the native topdata builder works and how to use it, see:

• sow-module/topdata/README.md

Important topdata contracts still live here under internal/topdata/ because they are code-adjacent implementation contracts, not user workflow docs.