sow-tools/README.md

# sow-tools

Shared NWN build tooling for Shadows Over Westgate.

This repository owns the Go-based `sow-toolkit` binary used by the separate module and assets repositories.

## Build

Build the development binary into `tools/sow-toolkit`:

```bash
./build-tool.sh
```

Or run directly with Go:

```bash
GOCACHE="$PWD/.cache/go-build" go run ./cmd/nwn-tool --help
```

## Repository Layout

```text
cmd/nwn-tool/      CLI entrypoint
internal/app/      command wiring and console UX
internal/erf/      ERF/MOD/HAK reading and writing
internal/gff/      canonical GFF JSON and binary conversion
internal/pipeline/ build, extract, compare, and manifest workflows
internal/project/  config loading and repo scanning
internal/validator/validation and diagnostics
tools/             built development binary output
```

## Repository Configuration

Human-authored consumer repository configuration is YAML. The toolkit discovers
configuration in this order:

1. `nwn-tool.yaml`
2. `nwn-tool.yml`
3. legacy `nwn-tool.json`

YAML is canonical. If both YAML and legacy JSON files are present, the YAML file
wins. Legacy `nwn-tool.json` still loads for migration, but commands print a
warning and should be updated to `nwn-tool.yaml`.

Example:

```yaml
module:
  name: Shadows Over Westgate
  resref: sow_module
  description: Shadows Over Westgate asset pipeline.
  hak_order:
    - sow_top
    - group:sow_over
    - group:sow_core

paths:
  assets: content
  build: build

music:
  # Legacy shorthand — automatically normalized into a dataset.
  prefixes:
    envi/music/westgate: mus_wg_

  # Explicit dataset config (replaces prefixes above).
  defaults:
    max_stem_length: 16
  datasets:
    westgate:
      source: envi/music/westgate
      prefix: mus_wg_
```

Resolved configuration can be inspected without scanning or building:

```bash
sow-toolkit config validate
sow-toolkit config effective --json
sow-toolkit config inspect paths.build
sow-toolkit config explain topdata.package_hak
sow-toolkit config sources
```

`config effective` prints the normalized values consumed by commands, including
toolkit defaults, YAML values, legacy status, active environment overrides, and
provenance. Sensitive override values such as tokens are reported as set without
printing the secret.

Common configurable defaults:

```yaml
paths:
  source: src
  assets: assets
  build: build
  cache: .cache
  tools: tools

outputs:
  module_archive: "{module.resref}.mod"
  hak_manifest: haks.json
  hak_archive: "{hak.name}.hak"

build:
  keep_existing_haks: false

inventory:
  source_extensions: []      # empty means built-in NWN source extensions
  asset_extensions: []       # empty means built-in NWN asset extensions
  source_json_pattern: "{resref}.{extension}.json"

scripts:
  source_dir: scripts
  cache: "{paths.cache}/ncs"
  compiler:
    path: ""
    env:
      path: SOW_NWN_SCRIPT_COMPILER
      nwn_root: [SOW_NWN_ROOT, NWN_ROOT]
      nwn_user_directory: [SOW_NWN_USER_DIRECTORY, NWN_HOME, NWN_USER_DIRECTORY]
    search:
      - "{paths.tools}/script-compiler/nwn_script_comp"
      - "{paths.tools}/script-compiler/nwn_script_comp.exe"
      - "{paths.tools}/nwn_script_comp"
      - "{paths.tools}/nwn_script_comp.exe"
      - PATH:nwn_script_comp

extract:
  layout: nwn_canonical_json
  hak_discovery: build_glob      # or configured_haks
  cleanup_stale: true
  ignore_extensions: []
  delete_module_archive_after_success: false

topdata:
  build: "{paths.build}/topdata"
  compiled_2da_dir: 2da
  compiled_tlk: sow_tlk.tlk
  package_hak: sow_top.hak
  package_tlk: sow_tlk.tlk
  wiki:
    output_root: wiki
    pages_dir: pages
    state_file: state.json
    managed_namespaces: [classes, feat, itemtypes, races, skills, spells, meta]
    deploy_manifest: .wiki_deploy_manifest.json
    deploy_edit_summary: Auto-generated from native builder

autogen:
  cache:
    root: "{paths.cache}"
    max_age: 1h
    refresh_env: SOW_AUTOGEN_MANIFEST_REFRESH
  release_source:
    provider: gitea
    server_url_env: SOW_ASSETS_SERVER_URL
    repo_env: SOW_ASSETS_REPO
```

Music defaults (used when not explicitly configured):

```yaml
music:
  defaults:
    stage_root: "{paths.cache}/music"
    credits_root: "{paths.cache}/credits"
    credits_overlay: CREDITS.md
    max_stem_length: 16
```

Music datasets define individual music collections with independent sources and
naming prefixes. The legacy `music.prefixes` shorthand is automatically normalized
into a dataset.

Generated and machine-readable artifacts remain JSON, including HAK manifests,
topdata datasets, caches, credits inventories, build reports, and canonical GFF
documents. Nested metadata such as `topdata/templates/config.json` is not root
repository configuration and is intentionally unchanged.

## Intended Consumers

- `sow-module` uses `sow-toolkit` for `build-module`, `extract`, `validate`, `compare`, and `apply-hak-manifest`
- `sow-assets` uses `sow-toolkit` for `build-haks` and validation
  - validation now fails early if duplicate asset resources would collide inside the same generated HAK
  - duplicate resources split across different HAK groups remain warnings because later HAK order can still override earlier content at runtime
- `sow-module` now also uses `sow-toolkit` for topdata:
  - `validate-topdata`
  - `build-topdata`
  - `compare-topdata`
  - `convert-topdata`

During development, sibling repositories can point at `../sow-tools/tools/sow-toolkit`
or `../sow-tools/tools/sow-toolkit.exe`. For pinned releases, each consumer repo can
instead carry its own copy in its local `tools/` directory.

## Cross-Repository Behavior

The current ownership split is:

- `sow-tools`
  - owns the shared CLI and build logic
  - owns module build, extract, compare, and HAK manifest application logic
  - owns topdata validation, build, compare, conversion, and packaging logic
- `sow-module`
  - owns canonical module source under `src/`
  - owns canonical topdata authored data under `topdata/`
  - owns the user-facing module and topdata wrapper commands
- `sow-assets`
  - owns canonical binary asset source under `assets/`
  - owns generated HAK manifests and parts manifests under `staging/`
  - owns the user-facing asset wrapper commands

Consumer wrapper resolution is intentionally aligned:

1. Prefer a pinned local binary in `tools/`
2. Else prefer a sibling `../sow-tools` checkout for local source development
3. Else install the latest published `sow-tools` release artifact

NWScript compilation behavior is also aligned and configurable under
`scripts.compiler`:

1. Prefer `SOW_NWN_SCRIPT_COMPILER` when explicitly set
2. Else prefer a local compiler in `tools/script-compiler/` or `tools/`
3. Else use a system-installed `nwn_script_comp` from `PATH`

When the compiler runs, the toolkit prepares `NWN_HOME` / `NWN_USER_DIRECTORY`
automatically and tries to detect `NWN_ROOT` from common Steam locations, including
custom Steam library folders, before an explicit override is required.


## Release Automation

This repo publishes:

- `sow-toolkit-linux-amd64.tar.gz`
- `sow-toolkit-windows-amd64.exe`

Consumer repos can fetch those binaries into their local `tools/` directory with:

- `install-tool.sh` on Linux/macOS
- `install-tool.ps1` on Windows

## TopData

The native topdata system is implemented in this repo and authored in `sow-module`.

High-level ownership split:

- `sow-tools`
  - owns the `sow-toolkit` implementation
  - owns topdata validation, build, compare, conversion, and dataset-resolution logic
- `sow-module`
  - owns canonical topdata authored data under `topdata/data/`
  - owns the user-facing topdata workflow scripts

For a human-readable front-to-back explanation of how the native topdata builder works
and how to use it, see:

- [sow-module/topdata/README.md](../sow-module/topdata/README.md)

Important topdata contracts still live here under `internal/topdata/` because they are
code-adjacent implementation contracts, not user workflow docs.