sow-tools/README.md

sow-tools

Shared NWN build tooling for Shadows Over Westgate.

This repository owns the Go-based sow-toolkit binary used by the separate module and assets repositories.

Build

Build the development binary into tools/sow-toolkit:

./build-tool.sh

On Windows:

.\build-tool.ps1

Both wrappers use .cache/go-build for the Go build cache and skip rebuilding when the sources are older than the existing development binary.

Or run directly with Go:

GOCACHE="$PWD/.cache/go-build" go run ./cmd/nwn-tool --help

When changing native topdata id allocation, lockfile persistence, generated families, or null-and-relocate behavior, run the repeat-build determinism test:

go test ./internal/topdata -run TestBuildNativeKeepsGeneratedIDsStableAcrossRepeatedBuilds

That test builds the same topdata project ten times in one process and asserts that relocated/generated ids stay pinned across every pass.

Repository Layout

cmd/nwn-tool/      CLI entrypoint
internal/app/      command wiring and console UX
internal/erf/      ERF/MOD/HAK reading and writing
internal/gff/      canonical GFF JSON and binary conversion
internal/pipeline/ build, extract, compare, and manifest workflows
internal/project/  config loading and repo scanning
internal/topdata/  native topdata, wiki, and autogen workflows
internal/validator/validation and diagnostics
tools/             built development binary output

Commands

The shared CLI exposes these project utilities:

sow-toolkit build
sow-toolkit build-module
sow-toolkit build-haks [--hak <name>] [--archive <name>] [--source-manifest <path>] [--plan-only] [--skip-music] [--music-dataset <id>]
sow-toolkit extract [<build-relative-archive-or-glob> ...]
sow-toolkit validate
sow-toolkit compare
sow-toolkit apply-hak-manifest [<manifest-path>]
sow-toolkit config <validate|effective|inspect|explain|sources>
sow-toolkit music <list-datasets|scan|build|credits|validate|manifest|normalize>
sow-toolkit validate-topdata
sow-toolkit build-topdata [--force] [--wiki]
sow-toolkit build-top-package [--force]
sow-toolkit compare-topdata
sow-toolkit convert-topdata <2da-to-json|2da-to-module|json-to-2da> ...
sow-toolkit build-wiki [--force]
sow-toolkit deploy-wiki [--source-dir <path>] [--endpoint <url>] [--token <token>] [--version <ver>] [--namespace <ns>] [--category <namespace=cid>] [--manifest <path>] [--stale-policy <report|archive|purge>] [--dry-run] [--create] [--force] [--reset-managed-namespaces]
sow-toolkit build-changelog [--config <path>] [--output <path>] [--current-tag <tag>] [--previous-tag <tag>] [--api-base-url <url>] [--token <token>]

Global logging flags --quiet, --verbose, and --debug may be passed after the command name. Long options that take values accept either --flag value or --flag=value; empty inline values such as --dataset= are rejected.

Repository Configuration

Human-authored consumer repository configuration is YAML. The toolkit discovers configuration in this order:

1. nwn-tool.yaml
2. nwn-tool.yml
3. legacy nwn-tool.json

YAML is canonical. If both YAML and legacy JSON files are present, the YAML file wins. Legacy nwn-tool.json still loads for migration, but commands print a warning and should be updated to nwn-tool.yaml.

Example:

module:
  name: Shadows Over Westgate
  resref: sow_module
  description: Shadows Over Westgate asset pipeline.
  hak_order:
    - sow_top
    - group:sow_over
    - group:sow_core

paths:
  assets: content
  build: build

music:
  # Legacy shorthand — automatically normalized into a dataset.
  prefixes:
    envi/music/westgate: mus_wg_

  # Explicit dataset config (replaces prefixes above).
  tools:
    ffmpeg: tools/ffmpeg
    ffprobe: tools/ffprobe
  defaults:
    stage_root: "{paths.cache}/music"
    credits_root: "{paths.cache}/credits"
    credits_overlay: CREDITS.md
    max_stem_length: 16
    naming_scheme: nwn_bmu
    output_extension: .bmu
    convert_extensions: [.flac, .m4a, .mp3, .ogg, .wav]
  datasets:
    westgate:
      source: envi/music/westgate
      output: envi/music/westgate
      prefix: mus_wg_
      package_mode: hak_asset

Resolved configuration can be inspected without scanning or building:

sow-toolkit config validate
sow-toolkit config effective --json
sow-toolkit config inspect paths.build
sow-toolkit config explain topdata.package_hak
sow-toolkit config sources

config effective prints the normalized values consumed by commands, including toolkit defaults, YAML values, legacy status, active environment overrides, and provenance. Sensitive override values such as tokens are reported as set without printing the secret.

Topdata value encodings under topdata.value_encodings are matched by dataset and column. Set dataset: "*" to apply an encoding to the same column name in any native topdata dataset; an exact dataset rule takes precedence over the wildcard rule for that column. Supported compact encodings include packed hex lists, external hex-list sidecar tables, alignment set masks, and ID hex lists. Explicit {"alignments": [...]} and {"ids": [...]} objects are portable in any native topdata dataset even without a column-specific rule; configured rules still override when a column needs custom bounds, widths, or alignment encoding mode. Plain {"list": [...]} objects remain config-driven because numeric lists need table-specific output semantics.

Class spell availability can be authored as JSON spellbooks under topdata/data/classes/spellbooks/. Each file names a classes:* row, and the native builder reads that row's SpellTableColumn to generate the matching spells.2da column. Spellbook files are discovered by convention, so ordinary per-class spellbook authoring does not require YAML registration.

Common configurable defaults:

paths:
  source: src
  assets: assets
  build: build
  cache: .cache
  tools: tools

outputs:
  module_archive: "{module.resref}.mod"
  hak_manifest: haks.json
  hak_archive: "{hak.name}.hak"

build:
  keep_existing_haks: false

generated_assets:
  topdata_2da:
    - id: parts
      source: topdata
      output: "{paths.cache}/generated-assets/parts-2da"
      include_datasets: [parts/**]
      package_root: part
      autogen:
        id: parts
        mode: parts_rows
        root: part
        include: ["**/*.mdl"]
        derive:
          kind: trailing_numeric_suffix
          group_from: first_path_segment
        parts_rows:
          row_defaults:
            COSTMODIFIER: "0"
          acbonus:
            default:
              strategy: ascending_row_id_sort_key
              divisor: 100
              format: "%.2f"
            datasets:
              chest:
                strategy: fixed
                value: "0.00"

inventory:
  source_extensions: []      # empty means built-in NWN source extensions
  asset_extensions: []       # empty means built-in NWN asset extensions
  source_json_pattern: "{resref}.{extension}.json"

validation:
  profile: nwn_module
  builtin_script_prefixes: [nw_, x0_, x1_, x2_, x3_, ga_, gc_, gen_, gui_, nwg_, ta_]
  required_fields:
    ifo: [Mod_Name]

scripts:
  source_dir: scripts
  cache: "{paths.cache}/ncs"
  compiler:
    path: ""
    env:
      path: SOW_NWN_SCRIPT_COMPILER
      nwn_root: [SOW_NWN_ROOT, NWN_ROOT]
      nwn_user_directory: [SOW_NWN_USER_DIRECTORY, NWN_HOME, NWN_USER_DIRECTORY]
    search:
      - "{paths.tools}/script-compiler/nwn_script_comp"
      - "{paths.tools}/script-compiler/nwn_script_comp.exe"
      - "{paths.tools}/nwn_script_comp"
      - "{paths.tools}/nwn_script_comp.exe"
      - PATH:nwn_script_comp

extract:
  layout: nwn_canonical_json
  archives:
    - "{module.resref}.mod"
  cleanup_stale: true
  consume_archives: false
  ignore_extensions: []
  delete_module_archive_after_success: false # deprecated compatibility alias

topdata:
  build: "{paths.build}/topdata"
  compiled_2da_dir: 2da
  compiled_tlk: sow_tlk.tlk
  package_hak: sow_top.hak
  package_tlk: sow_tlk.tlk
  value_encodings:
    - dataset: racialtypes/core
      column: AvailableSkinColors
      mode: packed_hex_list
      min: 0
      max: 175
      hex_width: 2
  row_generation:
    - namespace: portraits
      mode: first_null_row
      minimum_row: 15000
  class_feat_injections:
    global_feats:
      - feat: feat:literate
        list: 3
        granted_on_level: 1
        on_menu: 0
        unless_present: [feat:illiterate]
    class_skill_masterfeats:
      - masterfeat: masterfeats:skill_focus
        list: 1
        granted_on_level: -1
        on_menu: 0
  wiki:
    source: topdata/wiki
    output_root: wiki
    pages_dir: pages
    state_file: state.json
    page_index_file: page-index.json
    managed_namespaces: [classes, feat, itemtypes, races, skills, spells, meta]
    deploy_manifest: .wiki_deploy_manifest.json
    deploy_edit_summary: Auto-generated from native builder
    namespaces_file: namespaces.yaml
    data_file: data.yaml
    page_paths_file: wiki.yaml
    tables_file: tables.yaml
    visibility_file: visibility.yaml
    templates_dir: templates
    page_templates_dir: templates/pages
    manual_sections_dir: manual-sections
    manual_sections_file: manual-sections/default.yaml
    title_prefix_min_length: 3
    status_listing_scope: all

autogen:
  cache:
    root: "{paths.cache}"
    max_age: 1h
    refresh_env: SOW_AUTOGEN_MANIFEST_REFRESH
  release_source:
    provider: gitea
    server_url_env: SOW_ASSETS_SERVER_URL
    repo_env: SOW_ASSETS_REPO

topdata.value_encodings enables configured authoring sugar for compact 2DA cell values. The packed_hex_list mode accepts authored values such as {"list": [0, {"range": [10, 12]}]} and emits a single 0x... string with fixed-width hex chunks. The alignment_hex_list mode accepts two-letter alignment codes such as {"list": ["le", "CG"]} and emits the corresponding fixed-width hex chunks.

topdata.value_defaults applies configured fallback values to missing or null-like cells before 2DA emission:

topdata:
  value_defaults:
    - dataset: racialtypes/core
      column: ECL
      value: 0

topdata.row_generation controls where newly generated rows are allocated for specific datasets. The default after_base mode preserves existing behavior by allocating after the final imported base row. first_null_row fills base rows whose canonical key is empty and whose emitted 2DA columns are all null-like (null, empty string, or ****) before extending the table. minimum_row optionally prevents generated allocation below a configured row while preserving the selected allocation mode at and above that row.

topdata.row_extensions can extend plain native tables at build time without expanding the authored JSON. The repeat_last mode matches configured dataset glob patterns, finds the highest authored value in level_column, and appends generated rows through target_level by copying the last authored row's emitted columns while incrementing the level and row id. Generated extension rows are build output only and do not update source JSON or lockfiles.

topdata:
  row_extensions:
    - dataset: classes/spellsgained/*
      mode: repeat_last
      level_column: Level
      target_level: 60

topdata.class_feat_injections controls generated rows for every classes/feats/*.json table. global_feats injects concrete feat:* references unless the class already authors that feat; require_present and unless_present can make a rule conditional on other feat references already present in the same class feat table. class_skill_masterfeats expands a masterfeats:* family once for each class skill, using the configured list, granted_on_level, and on_menu values. When this block is omitted, the toolkit keeps its legacy default injections for compatibility.

topdata.wiki.alignment_links.target_pattern controls whether formatted alignment names become internal wiki links. The pattern may use {alignment} for the title-shaped alignment segment with spaces converted to underscores, and {label} for the display label. alignment_links.link_format defaults to wiki_markup, which emits [[target|label]]. Use html_anchor when the configured target should be emitted as a root URL anchor instead of being resolved as namespace-relative wiki markup:

topdata:
  wiki:
    alignment_links:
      target_pattern: /wiki/Guides/Alignment/{alignment}
      link_format: html_anchor

Generated wiki input and output paths are YAML-configurable. source points to the wiki authoring root, data_file, page_paths_file, tables_file, visibility_file, page_templates_dir, and manual_sections_file are resolved under that source root, and page_index_file, pages_dir, and state_file are resolved under the configured wiki output root. templates_dir and manual_sections_dir remain as compatibility defaults for deriving page_templates_dir and manual_sections_file when the newer explicit paths are omitted.

Generated wiki page structure should be owned by repository templates and YAML where possible. topdata/wiki/data.yaml can expose named providers for page templates, including source: facts providers whose facts.rows declare ordered label and value expressions. Empty and null-like fact values are omitted, so templates can render category fact tables with {{#each ProviderName}} while YAML controls which 2DA-backed values are shown.

paths.source and paths.assets must name real repository subtrees when they are configured. They may be omitted for repositories that do not own that class of source, but they must not resolve to the repository root (.). Output and cache paths must stay relative to the repository root unless a specific setting documents otherwise.

generated_assets.topdata_2da is for HAK repositories that need a small native-topdata subset packaged as HAK resources. It builds only .2da output, injects those generated files into build-haks, and rejects TLK-backed text because this path does not produce TLK packages. Static generated groups can omit autogen.mode; in that case the selected datasets are compiled directly, and package_root plus the repository's HAK include patterns determine where the resources are packaged.

For cachedmodels_rows generated assets, the configured autogen scan appends discovered model stems from the configured include globs to the selected cachedmodels dataset without .mdl extensions and skips models already present in authored rows.

For head_visualeffects autogen, producers can publish the group-to-row policy in autogen.producers[].head_visualeffects, and consumers can override or extend it in autogen.consumers[].head_visualeffects. The toolkit uses compatibility defaults for older configs and manifests, but YAML can own the group token source, category source, delimiter, case policy, legacy group mapping, model stem prefixes to strip, key/label formats, model column, and default row values. group_token_source: folder_name uses the manifest folder group directly instead of a prefix mapping, and category_from: immediate_parent can expose the model's category folder in keys and labels. legacy_groups maps released manifest group names to current folder groups, and legacy_key_format preserves row IDs when migrating older generated lock keys. Individual groups can override the model column with model_column or write the discovered model stem to several columns with model_columns, which is useful for root-node visualeffects. Supported format tokens are {dataset}, {group}, {group_raw}, {prefix}, {category}, {category_upper}, {category_segment}, {category_segment_upper}, {subgroup}, {stem}, {stem_upper}, {model_stem}, and {delimiter}.

For parts_rows generated assets, parts_rows.acbonus can normalize final ACBONUS values before module overrides are applied. The ascending_row_id_sort_key strategy writes toolset ordering values from row IDs so higher part rows can sort before lower rows in the toolset; the legacy descending_row_id_sort_key strategy remains available for repositories that already depend on it. Per-dataset policies such as chest: fixed 0.00 can override the default. Authored source ACBONUS values are not authoritative when this policy is configured, except that authored **** values remain null unless model discovery activates the row. Dense gaps without authored rows or discovered part models are emitted as **** null rows. Files under parts/modules remain the final override layer.

Extraction is deliberately guarded because it writes source files from binary archives. extract.archives is a list of glob patterns relative to paths.build; the default is only the configured module .mod. HAK extraction is opt-in, for example:

extract:
  archives:
    - "{module.resref}.mod"
    - "sow_over*.hak"

Module resources require a configured paths.source; HAK assets require a configured paths.assets; both roots must resolve to safe subtrees. If a root is unset or resolves to the repository root, extraction fails before writing instead of creating extension directories like mdl/ or png/ at the repo top level. .erf files are rejected by extraction because they do not contain module.ifo, so they cannot safely update module-level area membership.

consume_archives: true deletes the selected build archive files only after the entire extraction succeeds. The older delete_module_archive_after_success setting remains as a module-only compatibility alias when consume_archives is not set. Stale cleanup is scoped to roots actually touched by extracted resources, so the default module-only extraction does not prune assets.

topdata.wiki.status_listing_scope controls the generated wiki-status detail pages. The default all writes both per-namespace listings and aggregate meta:wikistatus:<status> listings. Use namespace when aggregate status lists would be too large for the target NodeBB post limit. Set topdata.wiki.status_pages: false to omit status pages entirely; existing tracked status pages then become stale deploy-manifest entries and can be removed with deploy-wiki --stale-policy purge.

GFF JSON extraction can merge selected fields and lists from the existing source file instead of replacing the whole extracted document. Rules are matched by target relative to paths.source; files without a matching rule keep the default overwrite behavior. For example:

extract:
  merge:
    gff_json:
      - target: module/module.ifo.json
        preserve_fields:
          - Mod_Entry_Area
          - Mod_Entry_X
          - Mod_Entry_Y
          - Mod_Entry_Z
          - Mod_Entry_Dir_X
          - Mod_Entry_Dir_Y
        merge_lists:
          - field: Mod_Area_list
            key_field: Area_Name
            strategy: preserve_existing_order_append_new

preserve_fields copies matching root fields from the existing source document into the extracted document before writing. The preserve_existing_order_append_new list strategy keeps existing keyed list order for entries still present in the extracted archive, uses extracted entries as the value source, appends archive-only entries in archive order, and removes existing entries that are no longer present in the archive.

Music defaults (used when not explicitly configured):

music:
  defaults:
    stage_root: "{paths.cache}/music"
    credits_root: "{paths.cache}/credits"
    credits_overlay: CREDITS.md
    max_stem_length: 16
    naming_scheme: nwn_bmu
    output_extension: .bmu
    convert_extensions: [.flac, .m4a, .mp3, .ogg, .wav]

Music datasets define individual music collections with independent sources and naming prefixes. source is the authored input root under paths.assets; output is the generated HAK asset root matched by HAK include globs. If output is omitted, it defaults to source. package_mode: hak_asset converts configured source extensions into generated HAK resources; package_mode: none scans and writes credits without adding generated HAK resources. The legacy music.prefixes shorthand is automatically normalized into datasets. convert_extensions is configuration-driven; the built-in default set is .flac, .m4a, .mp3, .ogg, and .wav. Those extensions are discovered from YAML within configured music dataset source roots, so setting them in music.defaults.convert_extensions or a dataset override is sufficient; you do not need a parallel inventory.asset_extensions override just to make music scanning see them.

Music can also be run directly:

sow-toolkit music list-datasets
sow-toolkit music scan --dataset westgate
sow-toolkit music build --dataset westgate --dry-run
sow-toolkit music build --dataset westgate
sow-toolkit music credits --dataset westgate --check
sow-toolkit music validate --dataset westgate --json
sow-toolkit music manifest --dataset westgate
sow-toolkit music normalize --dataset westgate

build-haks uses the same music pipeline. Use --plan-only or music --dry-run to compute mappings without transcoding. Use build-haks --skip-music to package authored assets only, or build-haks --music-dataset westgate to limit conversion to a configured dataset.

Generated and machine-readable artifacts remain JSON, including HAK manifests, topdata datasets, caches, credits inventories, build reports, and canonical GFF documents. Nested metadata such as topdata/templates/config.yaml remains valid when it is local workflow configuration rather than root repository configuration.

Wiki And Changelog Utilities

build-topdata --wiki or build-wiki renders wiki pages from the current native topdata state as Tiptap-compatible NodeBB HTML. Page templates are HTML fragments; they do not need to include a visible title heading because deploy uses generated page-index.json metadata for page titles and public paths. deploy-wiki publishes those pages to NodeBB. It reads NODEBB_API_ENDPOINT, NODEBB_API_TOKEN, GITHUB_REF_NAME, and NODEBB_WIKI_CATEGORIES when equivalent flags are omitted. Category mappings use namespace=cid, for example:

sow-toolkit deploy-wiki --dry-run --namespace spells --category spells=42

Namespace declarations in topdata/wiki/namespaces.yaml may set canonical_path when a generated namespace lives below a fuller NodeBB category path, for example canonical_path: Wiki/Feats. The title remains the display label, while canonical_path drives generated public_path metadata, wiki links, page-index entries, and deploy adoption matching.

When a deploy manifest is missing but matching wiki pages already exist in NodeBB, deploy-wiki checks the Westgate Wiki namespace directory API and adopts those topic/post mappings before planning updates. It does the same when a cached manifest mapping points to a post NodeBB no longer has. Canonical wikiPath/canonicalPath data is preferred during adoption. Nested generated paths require that exact canonical path data; leaf/title compatibility matching is only used where it cannot cross a canonical parent path. --create is still required for genuinely new pages that have no existing remote topic. If NodeBB rejects a create because the clean wiki URL already exists, deploy-wiki re-queries the namespace by page leaf, adopts the matching topic, and updates it instead. Pass --debug when deploying large wiki changes to emit planning and live mutation progress. Updates and stale-page archives acquire a Westgate Wiki edit lock before saving, then send the returned wikiEditLockToken with the NodeBB post edit request. Creates, updates, and archives write the same managed HTML to both NodeBB content and sourceContent; the deploy manifest records source_content_synced so older manifests receive a one-time body repair even when the generated hash is unchanged. Stale generated pages are reported by default. --stale-policy archive rewrites stale generated pages in place as archived managed content. --stale-policy purge permanently removes only stale generated topics already tracked in the wiki deploy manifest. Dry-run that policy before the live run; it never discovers manual wiki pages from NodeBB for purge. --reset-managed-namespaces is a stronger destructive mode for slug or path standard migrations: it permanently removes every topic currently listed by the configured managed namespace categories, ignores the previous deploy manifest, and recreates the current generated pages. It requires --create when local generated pages are present.

For newly created NodeBB topics, topdata.wiki.title_prefix_min_length controls when the deployer prefixes the namespace in the topic title. The default is 3, so one- and two-character page titles are created as Namespace: Title, while titles with three or more characters keep the page title alone. Existing bot-managed pages with older prefixed titles are renamed on later deploy passes when their current title no longer matches this policy.

build-changelog renders a release changelog from tagged pushes. Pull-request style commits keep their PR links and authors from the Gitea API; direct pushes are rendered as commit links using the commit author. It defaults to scripts/changelog.json, reads Gitea tokens from GITEA_API_TOKEN, GITEA_TOKEN, or SOW_TOOLS_TOKEN, and writes to stdout unless --output is provided.

Intended Consumers

• sow-module uses sow-toolkit for build, build-module, extract, validate, compare, and apply-hak-manifest
• sow-assets uses sow-toolkit for build-haks, music processing, changelog generation, and validation
  • validation now fails early if duplicate asset resources would collide inside the same generated HAK
  • duplicate resources split across different HAK groups remain warnings because later HAK order can still override earlier content at runtime
• sow-module now also uses sow-toolkit for topdata:
  • validate-topdata
  • build-topdata
  • build-top-package
  • compare-topdata
  • convert-topdata
  • build-wiki
  • deploy-wiki

During development, sibling repositories can point at ../sow-tools/tools/sow-toolkit or ../sow-tools/tools/sow-toolkit.exe. For pinned releases, each consumer repo can instead carry its own copy in its local tools/ directory.

Cross-Repository Behavior

The current ownership split is:

• sow-tools
  • owns the shared CLI and build logic
  • owns module build, extract, compare, and HAK manifest application logic
  • owns topdata validation, build, compare, conversion, and packaging logic
• sow-module
  • owns canonical module source under src/
  • owns canonical topdata authored data under topdata/
  • owns the user-facing module and topdata wrapper commands
• sow-assets
  • owns canonical binary asset source under assets/
  • owns generated HAK manifests and HAK-side parts 2DA assets
  • owns the user-facing asset wrapper commands

Consumer wrapper resolution is intentionally aligned:

1. Prefer a pinned local binary in tools/
2. Else prefer a sibling ../sow-tools checkout for local source development
3. Else install the latest published sow-tools release artifact

NWScript compilation behavior is also aligned and configurable under scripts.compiler:

1. Prefer SOW_NWN_SCRIPT_COMPILER when explicitly set
2. Else prefer a local compiler in tools/script-compiler/ or tools/
3. Else use a system-installed nwn_script_comp from PATH

When the compiler runs, the toolkit prepares NWN_HOME / NWN_USER_DIRECTORY automatically and tries to detect NWN_ROOT from common Steam locations, including custom Steam library folders, before an explicit override is required.

Release Automation

This repo publishes:

• sow-toolkit-linux-amd64.tar.gz
• sow-toolkit-windows-amd64.exe

Consumer repos can fetch those binaries into their local tools/ directory with:

• install-tool.sh on Linux/macOS
• install-tool.ps1 on Windows

TopData

The native topdata system is implemented in this repo and authored in sow-module.

High-level ownership split:

• sow-tools
  • owns the sow-toolkit implementation
  • owns topdata validation, build, compare, conversion, and dataset-resolution logic
• sow-module
  • owns canonical topdata authored data under topdata/data/
  • owns the user-facing topdata workflow scripts

For a human-readable front-to-back explanation of how the native topdata builder works and how to use it, see:

• sow-module/topdata/README.md

Important topdata contracts still live here under internal/topdata/ because they are code-adjacent implementation contracts, not user workflow docs.