ShadowsOverWestgate/sow-tools
7
0
Fork 0
Code Issues 2 Pull Requests Actions Packages Projects Releases 38 Activity

claude: fold in old sow-tools

Browse Source
This commit is contained in:
archvillainette
2026-06-13 09:49:29 +02:00
parent d4a435883b
commit 56c67132f3
112 changed files with 70812 additions and 74 deletions
Download Patch File Download Diff File Expand all files Collapse all files
AGENTS.md
+17 -14
View File
@@ -8,8 +8,7 @@ alwaysApply: true
This repo owns the **builder logic** for the migration: one Go module
(`gitea.westgate.pw/ShadowsOverWestgate/sow-tools`) producing the `crucible`
dispatcher and the `crucible-<name>` binaries (D11). Read
[`../AGENTS.md`](../AGENTS.md) (migration hub) and
[`../../KICKOFF_PROMPT.md`](../../KICKOFF_PROMPT.md) first.
`../AGENTS.md` (migration hub) and `../../KICKOFF_PROMPT.md` first.
## What this repo owns / does not own
@@ -19,29 +18,33 @@ changelog. Does **not** own authored game content (that is `sow-module` /
`sow-topdata` / `sow-assets-manifest`) or any production deploy authority (that
is `sow-platform`).
## Scaffold rules (Phase 5)
## Rules
1. **Source is not transplanted.** The migration hard rules forbid copying the
`internal/` packages from `gitea/sow-tools` automatically. The operator
migrates them at cutover; see [`docs/migration-from-nwn-tool.md`](docs/migration-from-nwn-tool.md).
2. **Fail closed, never fake.** Unwired builders exit `70`. Do not stub a
builder to emit a placeholder artifact.
1. **Migrated logic, not a fresh rewrite.** The `internal/` packages (`app`,
`pipeline`, `project`, `erf`, `gff`, `topdata`, `music`, `changelog`,
`validator`) were folded in from `gitea/sow-tools` at cutover; wired builders
delegate to `internal/app`'s command surface. Keep them in step with upstream
fixes rather than diverging silently.
2. **Fail closed, never fake.** A builder with no migrated logic yet (`depot`)
exits `70`. Do not stub a builder to emit a placeholder artifact.
3. **Binaries are not committed.** They are CI artifacts / image layers (D19).
`/bin/`, `*.exe`, `nwn-tool`, `sow-toolkit` are gitignored.
4. **No home-dir / `NWN_ROOT` guessing.** Builders take roots explicitly via
flag or env. See [`docs/consumer-contract.md`](docs/consumer-contract.md).
flag or env (project resolution is CWD-based, never `$HOME`). See
[`docs/consumer-contract.md`](docs/consumer-contract.md).
5. **The registry is the command surface.** `internal/dispatch.Registry` is the
single source of truth; keep it in sync with `cmd/` and
[`docs/command-surface.md`](docs/command-surface.md). Adding a builder = a
`cmd/crucible-<name>/main.go` shim + a `Registry` entry + a doc row.
## Wiring a builder (operator cutover)
## Wiring a builder
1. Migrate the relevant `internal/` package(s) from `gitea/sow-tools`.
2. Register a handler and flip the builder off the unwired path in
`internal/dispatch`.
1. Ensure the relevant `internal/` package(s) cover the work.
2. Add the legacy command(s) to the builder's `Legacy`/`Extra` set and set
`Wired: true` in `internal/dispatch`; the dispatcher delegates to
`app.Run`. `depot` is the remaining unwired builder.
3. Add tests; keep outputs deterministic (same input → same bytes).
4. `make check` must stay green; `make smoke` is updated to expect a wired exit.
4. `make check` must stay green; update `make smoke` to expect the wired exit.
## Commands
README.md
+11 -8
View File
@@ -27,16 +27,19 @@ The dispatcher and the standalone shims share one registry
single-token command. The full legacy `nwn-tool` command surface and where each
command lands is mapped in [`docs/command-surface.md`](docs/command-surface.md).
## Status (Phase 5 scaffold)
## Status (cutover performed)
The suite **builds, vets, tests, and runs**, but every builder is **unwired**:
running one fails closed with exit `70` and never fakes an artifact. The internal
pipeline/topdata/erf/wiki/music packages from `gitea/sow-tools` are migrated by
the operator at cutover — the workspace hard rules forbid transplanting that
source automatically. See [`docs/migration-from-nwn-tool.md`](docs/migration-from-nwn-tool.md).
The internal `app`/`pipeline`/`project`/`erf`/`gff`/`topdata`/`music`/`changelog`/
`validator` packages from `gitea/sow-tools` have been migrated into this tree, and
the `module`, `topdata`, `hak`, and `wiki` builders now **delegate to the migrated
`nwn-tool` command surface** (mapped in
[`docs/command-surface.md`](docs/command-surface.md)). `config` and `changelog`
are global commands on the dispatcher. `depot` has no migrated logic yet, so it
keeps the fail-closed path: exit `70`, never a faked artifact.
This matches the downstream skeletons: `sow-module` / `sow-topdata` package
scripts already fail closed until they can resolve a Crucible binary.
See [`docs/migration-from-nwn-tool.md`](docs/migration-from-nwn-tool.md) for what
was done and what remains (the consumer `--manifest/--source/--out` flag contract
is the open Phase-6 item).
## Develop
docker/Dockerfile
+13 -6
View File
@@ -6,15 +6,15 @@
# and ships them on a static base. The `crucible` dispatcher is the entrypoint;
# consumer CI can also call the standalone crucible-<name> binaries by path.
#
# NOTE (Phase 5 scaffold): the binaries are pure stdlib and fail closed until
# the operator migrates the internal pipeline. When the music pipeline is wired
# it needs ffmpeg at runtime — switch the final stage to a debian-slim base with
# ffmpeg then (see docs/migration-from-nwn-tool.md).
# NOTE: the internal pipeline is migrated and the music conversion path is wired,
# so the runtime stage is debian-slim with ffmpeg on PATH (BMU encode/decode).
# See docs/migration-from-nwn-tool.md.
FROM golang:1.26-alpine AS build
WORKDIR /src
RUN apk add --no-cache git
# go.sum is optional while the scaffold has no external deps.
# go.sum is committed now that the migrated packages pull golang.org/x/text and
# gopkg.in/yaml.v3; the glob keeps the build working if it is ever absent.
COPY go.mod go.sum* ./
RUN go mod download
COPY . .
@@ -29,7 +29,14 @@ RUN set -eux; \
-o "/out/${name}" "${dir}"; \
done
FROM gcr.io/distroless/static-debian12:nonroot
FROM debian:12-slim
# ffmpeg: the migrated music pipeline shells out to it for BMU conversion.
# ca-certificates: builders fetch published manifests over HTTPS.
RUN set -eux; \
apt-get update; \
apt-get install -y --no-install-recommends ffmpeg ca-certificates; \
rm -rf /var/lib/apt/lists/*; \
useradd --system --create-home --uid 65532 nonroot
COPY --from=build /out/ /usr/local/bin/
USER nonroot
ENTRYPOINT ["/usr/local/bin/crucible"]
docs/command-surface.md
+7 -5
View File
@@ -25,19 +25,21 @@ These legacy commands are **not** top-level builders:
- `music *` (`list-datasets`, `scan`, `build`, `credits`, `validate`, `manifest`,
`normalize`) → folds into the **module/hak build pipeline** (music BMUs are
packed into HAKs at build time). Exposed as `crucible-module music ...` /
`crucible-hak music ...` when wired. Needs `ffmpeg` at runtime.
`crucible-hak music ...`. Needs `ffmpeg` at runtime (in the image).
- `config *` (`validate`, `effective`, `inspect`, `explain`, `sources`) → a
**global** concern shared by every builder; exposed as a global subcommand
group, not its own binary.
- `build-changelog` → release tooling; lands as a global `crucible changelog`
**global** concern shared by every builder; exposed as the global
`crucible config ...`, not its own binary.
- `build-changelog` → release tooling; exposed as the global `crucible changelog`
rather than a content builder.
## Global commands (implemented in the scaffold)
## Global commands (implemented)
```text
crucible help | -h usage + builder list
crucible version | -V build version (git sha via -ldflags)
crucible list builders, one per line: name<TAB>bin<TAB>summary
crucible config [args] -> legacy `config` command group
crucible changelog [args] -> legacy `build-changelog`
```
`crucible list` is machine-readable so CI can enumerate builders without parsing
docs/migration-from-nwn-tool.md
+12 -3
View File
@@ -1,9 +1,18 @@
# Migrating from `nwn-tool` to Crucible (operator cutover)
# Migrating from `nwn-tool` to Crucible (cutover — DONE)
The legacy toolchain lives at `gitea/sow-tools` as a single `nwn-tool` binary
(entry `cmd/nwn-tool`, logic under `internal/`). This repo is the Crucible
rewrite. Per the migration hard rules, the `internal/` source is **not**
transplanted automatically — the operator migrates it. This is the checklist.
rewrite. The cutover below has been **performed**: the `internal/` packages are
migrated and the `module`/`topdata`/`hak`/`wiki` builders delegate to them. The
checklist is kept as the record of what was done.
> **Status.** Steps 1–8 are complete. `depot` is the one remaining unwired
> builder (no legacy source — it was shell `mc`/S3). The open follow-up is the
> consumer flag contract: `sow-module` / `sow-topdata` / `sow-assets-manifest`
> wrappers invoke `crucible-<name>` with `--manifest/--source/--out` style flags
> (see [`consumer-contract.md`](consumer-contract.md)), whereas the migrated
> commands use the legacy CWD/project model. Reconciling the two is the Phase-6
> consumer task and is **not** part of this cutover.
## Why a clean scaffold first
go.mod
+5
View File
@@ -1,3 +1,8 @@
module git.westgate.pw/ShadowsOverWestgate/sow-tools
go 1.26.0
require (
golang.org/x/text v0.35.0
gopkg.in/yaml.v3 v3.0.1
)
go.sum
+6
View File
@@ -0,0 +1,6 @@
golang.org/x/text v0.35.0 h1:JOVx6vVDFokkpaq1AEptVzLTpDe9KGpj5tR4/X+ybL8=
golang.org/x/text v0.35.0/go.mod h1:khi/HExzZJ2pGnjenulevKNX1W67CUy0AsXcNubPGCA=
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405 h1:yhCVgyC4o1eVCa2tZl7eS0r+SDo693bJlVdllGtEeKM=
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=
gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
internal/app/app.go
+2113
View File
File diff suppressed because it is too large Load Diff
internal/app/app_test.go
+574
View File
@@ -0,0 +1,574 @@
package app
import (
"bytes"
"os"
"path/filepath"
"strings"
"testing"
"time"
"git.westgate.pw/ShadowsOverWestgate/sow-tools/internal/pipeline"
)
func TestRunBuildTopPackageUsesCachedCompiledOutputs(t *testing.T) {
root := t.TempDir()
mkdirAll(t, filepath.Join(root, "build"))
mkdirAll(t, filepath.Join(root, ".cache", "2da"))
mkdirAll(t, filepath.Join(root, "topdata", "assets", "gui"))
mkdirAll(t, filepath.Join(root, "topdata", "data", "repadjust"))
writeFile(t, filepath.Join(root, "nwn-tool.yaml"), `
module:
name: Test Module
resref: test_module
topdata:
source: topdata
build: .cache
package_hak: sow_top.hak
package_tlk: sow_tlk.tlk
`)
writeFile(t, filepath.Join(root, ".cache", "2da", "repadjust.2da"), "2DA V2.0\n\n Label\n0 TEST_LABEL\n")
writeFile(t, filepath.Join(root, "build", "sow_tlk.tlk"), "compiled tlk")
writeFile(t, filepath.Join(root, "topdata", "assets", "gui", "testicon.png"), "icon-data")
writeFile(t, filepath.Join(root, "topdata", "data", "repadjust", "base.json"), `{
"output": "repadjust.2da",
"columns": ["Label"],
"rows": [{"id": 0, "Label": "TEST_LABEL"}]
}`+"\n")
sourceTime := time.Now().Add(-2 * time.Hour)
outputTime := time.Now().Add(-1 * time.Hour)
setTreeTime(t, filepath.Join(root, "topdata"), sourceTime)
setFileTime(t, filepath.Join(root, ".cache", "2da", "repadjust.2da"), outputTime)
setFileTime(t, filepath.Join(root, "build", "sow_tlk.tlk"), outputTime)
var stdout bytes.Buffer
ctx := context{
stdout: &stdout,
stderr: &bytes.Buffer{},
cwd: root,
args: []string{"build-top-package"},
}
if err := runBuildTopPackage(ctx); err != nil {
t.Fatalf("runBuildTopPackage failed: %v", err)
}
output := stdout.String()
if !strings.Contains(output, "top package hak: build/sow_top.hak") {
t.Fatalf("expected build-top-package output, got %q", output)
}
if strings.Contains(output, "[build-top-package]") {
t.Fatalf("did not expect raw progress lines in normal output, got %q", output)
}
if _, err := os.Stat(filepath.Join(root, "build", "sow_top.hak")); err != nil {
t.Fatalf("expected packaged hak output: %v", err)
}
}
func TestRunBuildHAKsEmitsCompactSummary(t *testing.T) {
root := t.TempDir()
mkdirAll(t, filepath.Join(root, "assets", "envi", "music", "westgate"))
mkdirAll(t, filepath.Join(root, "build"))
mkdirAll(t, filepath.Join(root, "src"))
writeFile(t, filepath.Join(root, "nwn-tool.yaml"), `
module:
name: Test Module
resref: testmod
paths:
source: src
assets: assets
build: build
haks:
- name: envi
priority: 1
max_bytes: 1048576
split: false
include:
- envi/**
`)
writeFile(t, filepath.Join(root, "assets", "envi", "music", "westgate", "AleandAnecdotes.mp3"), "source-mp3")
writeFile(t, filepath.Join(root, "assets", "envi", "music", "westgate", "CREDITS.md"), "# NWN Music Pack Credits\n\n| Artist | Title | Output File | Original File | Album | Date | License / Copyright | Notes |\n|---|---|---|---|---|---|---|---|\n| Darren Curtis | Ale and Anecdotes | `mus_wg_andnc.bmu` | `AleandAnecdotes.mp3` | Darren's Commercially Free Music | | Darren Curtis Music | |\n")
ffprobePath := filepath.Join(root, "ffprobe")
writeFile(t, ffprobePath, "#!/bin/sh\nprintf '%s\\n' '{\"format\":{\"tags\":{\"artist\":\"Unknown\",\"title\":\"Broken Metadata\"}}}'\n")
if err := os.Chmod(ffprobePath, 0o755); err != nil {
t.Fatalf("chmod ffprobe: %v", err)
}
ffmpegPath := filepath.Join(root, "ffmpeg")
writeFile(t, ffmpegPath, "#!/bin/sh\nfor last; do :; done\nprintf 'bmu-data' > \"$last\"\n")
if err := os.Chmod(ffmpegPath, 0o755); err != nil {
t.Fatalf("chmod ffmpeg: %v", err)
}
t.Setenv("SOW_FFMPEG", ffmpegPath)
t.Setenv("SOW_FFPROBE", ffprobePath)
var stdout bytes.Buffer
ctx := context{
stdout: &stdout,
stderr: &bytes.Buffer{},
cwd: root,
args: []string{"build-haks"},
}
if err := runBuildHAKs(ctx); err != nil {
t.Fatalf("runBuildHAKs failed: %v", err)
}
output := stdout.String()
if !strings.Contains(output, "Build HAKs ----------") {
t.Fatalf("expected build header, got %q", output)
}
if !strings.Contains(output, "mapped: 1 music file(s); use --verbose to list mappings") {
t.Fatalf("expected compact mapping summary, got %q", output)
}
if strings.Contains(output, "AleandAnecdotes.mp3 -> mus_wg_andnc.bmu") {
t.Fatalf("did not expect verbose mapping in normal mode, got %q", output)
}
if !strings.Contains(output, "manifest: build/haks.json") {
t.Fatalf("expected relative manifest path, got %q", output)
}
}
func setTreeTime(t *testing.T, root string, modTime time.Time) {
t.Helper()
err := filepath.WalkDir(root, func(path string, d os.DirEntry, walkErr error) error {
if walkErr != nil {
return walkErr
}
return os.Chtimes(path, modTime, modTime)
})
if err != nil {
t.Fatalf("set tree time under %s: %v", root, err)
}
}
func TestRunBuildHAKsVerboseListsMappings(t *testing.T) {
root := t.TempDir()
mkdirAll(t, filepath.Join(root, "assets", "envi", "music", "westgate"))
mkdirAll(t, filepath.Join(root, "build"))
mkdirAll(t, filepath.Join(root, "src"))
writeFile(t, filepath.Join(root, "nwn-tool.yaml"), `
module:
name: Test Module
resref: testmod
paths:
source: src
assets: assets
build: build
haks:
- name: envi
priority: 1
max_bytes: 1048576
split: false
include:
- envi/**
`)
writeFile(t, filepath.Join(root, "assets", "envi", "music", "westgate", "AleandAnecdotes.mp3"), "source-mp3")
writeFile(t, filepath.Join(root, "assets", "envi", "music", "westgate", "CREDITS.md"), "# NWN Music Pack Credits\n\n| Artist | Title | Output File | Original File | Album | Date | License / Copyright | Notes |\n|---|---|---|---|---|---|---|---|\n| Darren Curtis | Ale and Anecdotes | `mus_wg_andnc.bmu` | `AleandAnecdotes.mp3` | Darren's Commercially Free Music | | Darren Curtis Music | |\n")
ffprobePath := filepath.Join(root, "ffprobe")
writeFile(t, ffprobePath, "#!/bin/sh\nprintf '%s\\n' '{\"format\":{\"tags\":{\"artist\":\"Unknown\",\"title\":\"Broken Metadata\"}}}'\n")
if err := os.Chmod(ffprobePath, 0o755); err != nil {
t.Fatalf("chmod ffprobe: %v", err)
}
ffmpegPath := filepath.Join(root, "ffmpeg")
writeFile(t, ffmpegPath, "#!/bin/sh\nfor last; do :; done\nprintf 'bmu-data' > \"$last\"\n")
if err := os.Chmod(ffmpegPath, 0o755); err != nil {
t.Fatalf("chmod ffmpeg: %v", err)
}
t.Setenv("SOW_FFMPEG", ffmpegPath)
t.Setenv("SOW_FFPROBE", ffprobePath)
var stdout bytes.Buffer
ctx := context{
stdout: &stdout,
stderr: &bytes.Buffer{},
cwd: root,
args: []string{"build-haks", "--verbose"},
}
if err := runBuildHAKs(ctx); err != nil {
t.Fatalf("runBuildHAKs failed: %v", err)
}
output := stdout.String()
if !strings.Contains(output, "mappings:") {
t.Fatalf("expected verbose mappings header, got %q", output)
}
if !strings.Contains(output, "AleandAnecdotes.mp3 -> mus_wg_andnc.bmu") {
t.Fatalf("expected verbose mapping output, got %q", output)
}
if !strings.Contains(output, "wrote: envi (1 assets)") {
t.Fatalf("expected verbose archive action, got %q", output)
}
}
func TestTopdataConsoleSuppressesProgressInNormalMode(t *testing.T) {
var stdout bytes.Buffer
console := &topdataConsole{
stdout: &stdout,
projectRoot: "/workspace/project",
projectName: "Test Module",
commandName: "build-topdata",
commandLabel: "Build Topdata",
level: logLevelNormal,
}
console.progress("Packaging compiled topdata resources into sow_top.hak...")
if stdout.String() != "" {
t.Fatalf("expected no normal-mode progress output, got %q", stdout.String())
}
}
func TestSpinnerEnabledForHonorsPlainTTYMode(t *testing.T) {
t.Setenv("SOW_TOOLS_TTY_MODE", "plain")
if spinnerEnabledFor(&bytes.Buffer{}, logLevelNormal) {
t.Fatal("expected plain tty mode to disable spinner output")
}
}
func TestTopdataConsoleDebugProgressAndRelativePaths(t *testing.T) {
var stdout bytes.Buffer
console := &topdataConsole{
stdout: &stdout,
projectRoot: "/workspace/project",
projectName: "Test Module",
commandName: "deploy-wiki",
commandLabel: "Deploy Wiki",
level: logLevelDebug,
}
console.progress("NodeBB wiki plan: create 1, update 2, skip 3, stale 4, archive 5, purge 6, drift 0")
console.emitWikiDeployResult(10, 1, 2, 3, 4, 5, 6, 0, "/workspace/project/build/wiki/deploy-manifest.json")
output := stdout.String()
if !strings.Contains(output, "[debug] NodeBB wiki plan: create 1, update 2, skip 3, stale 4, archive 5, purge 6, drift 0") {
t.Fatalf("expected debug progress line, got %q", output)
}
if !strings.Contains(output, "archived: 5") {
t.Fatalf("expected archived deploy count, got %q", output)
}
if !strings.Contains(output, "purged: 6") {
t.Fatalf("expected purged deploy count, got %q", output)
}
if !strings.Contains(output, "manifest: build/wiki/deploy-manifest.json") {
t.Fatalf("expected relative deploy manifest path, got %q", output)
}
}
func TestParseDeployWikiHelpListsPurgeStalePolicy(t *testing.T) {
_, err := parseDeployWikiArgs("deploy-wiki", []string{"--help"})
if err == nil || !strings.Contains(err.Error(), "--stale-policy <report|archive|purge>") {
t.Fatalf("expected deploy-wiki help to list purge stale policy, got %v", err)
}
}
func TestParseDeployWikiResetManagedNamespacesFlag(t *testing.T) {
opts, err := parseDeployWikiArgs("deploy-wiki", []string{"--reset-managed-namespaces"})
if err != nil {
t.Fatalf("parse deploy wiki reset flag: %v", err)
}
if !opts.ResetManagedNamespaces {
t.Fatalf("expected --reset-managed-namespaces to enable namespace reset")
}
}
func TestProjectConsoleSuppressesBuildModuleProgressInNormalMode(t *testing.T) {
var stdout bytes.Buffer
console := &projectConsole{
stdout: &stdout,
projectRoot: "/workspace/project",
projectName: "Test Module",
commandName: "build-module",
commandLabel: "Build Module",
level: logLevelNormal,
}
console.progress("Writing module archive...")
if stdout.String() != "" {
t.Fatalf("expected no normal-mode progress output, got %q", stdout.String())
}
}
func TestProjectConsoleEmitsRelativePaths(t *testing.T) {
var stdout bytes.Buffer
console := &projectConsole{
stdout: &stdout,
projectRoot: "/workspace/project",
projectName: "Test Module",
commandName: "compare",
commandLabel: "Compare",
level: logLevelNormal,
}
console.emitCompareResult(pipeline.CompareResult{
ModulePath: "/workspace/project/build/test.mod",
HAKPaths: []string{"/workspace/project/build/core.hak"},
Checked: 42,
})
output := stdout.String()
if !strings.Contains(output, "module: build/test.mod") {
t.Fatalf("expected relative module path, got %q", output)
}
}
func TestRunMusicScanUsesConfiguredDatasetsWithoutWriting(t *testing.T) {
root := t.TempDir()
mkdirAll(t, filepath.Join(root, "assets", "audio", "westgate"))
mkdirAll(t, filepath.Join(root, "build"))
writeFile(t, filepath.Join(root, "nwn-tool.yaml"), `
module:
name: Test Module
resref: testmod
paths:
assets: assets
build: build
music:
datasets:
westgate_audio:
source: audio/westgate
output: generated/music
prefix: wg_
`)
writeFile(t, filepath.Join(root, "assets", "audio", "westgate", "Theme Song.mp3"), "source-mp3")
var stdout bytes.Buffer
ctx := context{
stdout: &stdout,
stderr: &bytes.Buffer{},
cwd: root,
args: []string{"music", "scan", "--dataset", "westgate_audio"},
}
if err := runMusic(ctx); err != nil {
t.Fatalf("runMusic failed: %v", err)
}
output := stdout.String()
if !strings.Contains(output, "music scan") || !strings.Contains(output, "tracks: 1") {
t.Fatalf("unexpected music scan output: %q", output)
}
if _, err := os.Stat(filepath.Join(root, ".cache", "credits")); !os.IsNotExist(err) {
t.Fatalf("music scan should not write credits artifacts, stat err=%v", err)
}
}
func TestRunMusicListDatasets(t *testing.T) {
root := t.TempDir()
mkdirAll(t, filepath.Join(root, "assets"))
mkdirAll(t, filepath.Join(root, "build"))
writeFile(t, filepath.Join(root, "nwn-tool.yaml"), `
module:
name: Test Module
resref: testmod
paths:
assets: assets
music:
datasets:
westgate_audio:
source: audio/westgate
output: generated/music
prefix: wg_
`)
var stdout bytes.Buffer
ctx := context{
stdout: &stdout,
stderr: &bytes.Buffer{},
cwd: root,
args: []string{"music", "list-datasets"},
}
if err := runMusic(ctx); err != nil {
t.Fatalf("runMusic failed: %v", err)
}
if !strings.Contains(stdout.String(), "westgate_audio: source=audio/westgate output=generated/music prefix=wg_") {
t.Fatalf("unexpected dataset list: %q", stdout.String())
}
}
func TestRunConfigEffectiveJSONShowsDefaultsAndProvenance(t *testing.T) {
root := t.TempDir()
writeFile(t, filepath.Join(root, "nwn-tool.yaml"), `
module:
name: Test Module
resref: testmod
paths:
source: src
`)
var stdout bytes.Buffer
ctx := context{
stdout: &stdout,
stderr: &bytes.Buffer{},
cwd: root,
args: []string{"config", "effective", "--json"},
}
if err := runConfig(ctx); err != nil {
t.Fatalf("runConfig failed: %v", err)
}
output := stdout.String()
if !strings.Contains(output, `"hak_manifest": "haks.json"`) {
t.Fatalf("expected HAK manifest default in effective config, got %q", output)
}
if !strings.Contains(output, `"paths.build":`) || !strings.Contains(output, `"toolkit default"`) {
t.Fatalf("expected default provenance in effective config, got %q", output)
}
}
func TestRunConfigExplainReportsYAMLSource(t *testing.T) {
root := t.TempDir()
writeFile(t, filepath.Join(root, "nwn-tool.yaml"), `
module:
name: Test Module
resref: testmod
paths:
build: output
`)
var stdout bytes.Buffer
ctx := context{
stdout: &stdout,
stderr: &bytes.Buffer{},
cwd: root,
args: []string{"config", "explain", "paths.build"},
}
if err := runConfig(ctx); err != nil {
t.Fatalf("runConfig failed: %v", err)
}
output := stdout.String()
if !strings.Contains(output, "value: \"output\"") {
t.Fatalf("expected configured build value, got %q", output)
}
if !strings.Contains(output, "source: yaml") {
t.Fatalf("expected YAML source, got %q", output)
}
}
func TestRunConfigValidateDoesNotScanSourceInventory(t *testing.T) {
root := t.TempDir()
mkdirAll(t, filepath.Join(root, "build"))
writeFile(t, filepath.Join(root, "nwn-tool.yaml"), `
module:
name: Test Module
resref: testmod
paths:
source: missing-src
build: build
`)
var stdout bytes.Buffer
ctx := context{
stdout: &stdout,
stderr: &bytes.Buffer{},
cwd: root,
args: []string{"config", "validate"},
}
if err := runConfig(ctx); err != nil {
t.Fatalf("runConfig failed: %v", err)
}
if !strings.Contains(stdout.String(), "config: ok") {
t.Fatalf("expected config validation output, got %q", stdout.String())
}
}
func TestRunConfigSourcesListsActiveOverrides(t *testing.T) {
root := t.TempDir()
writeFile(t, filepath.Join(root, "nwn-tool.yaml"), `
module:
name: Test Module
resref: testmod
paths:
source: src
`)
t.Setenv("SOW_BUILD_HAKS_KEEP_EXISTING", "1")
var stdout bytes.Buffer
ctx := context{
stdout: &stdout,
stderr: &bytes.Buffer{},
cwd: root,
args: []string{"config", "sources"},
}
if err := runConfig(ctx); err != nil {
t.Fatalf("runConfig failed: %v", err)
}
output := stdout.String()
if !strings.Contains(output, "active overrides:") {
t.Fatalf("expected active overrides section, got %q", output)
}
if !strings.Contains(output, "build.keep_existing_haks=1") {
t.Fatalf("expected keep existing override, got %q", output)
}
}
func TestInlineFlagParsersRejectEmptyValues(t *testing.T) {
if _, err := parseBuildHAKArgs([]string{"--hak="}); err == nil || !strings.Contains(err.Error(), "--hak requires a value") {
t.Fatalf("expected empty --hak inline value error, got %v", err)
}
if _, err := parseMusicCommandArgs([]string{"--dataset="}); err == nil || !strings.Contains(err.Error(), "--dataset requires a value") {
t.Fatalf("expected empty --dataset inline value error, got %v", err)
}
if _, err := parseDeployWikiArgs("deploy-wiki", []string{"--endpoint="}); err == nil || !strings.Contains(err.Error(), "--endpoint requires a value") {
t.Fatalf("expected empty --endpoint inline value error, got %v", err)
}
if _, err := parseBuildChangelogArgs("build-changelog", []string{"--output="}); err == nil || !strings.Contains(err.Error(), "--output requires a value") {
t.Fatalf("expected empty --output inline value error, got %v", err)
}
}
func TestParseBuildChangelogArgsAcceptsInlineValues(t *testing.T) {
opts, err := parseBuildChangelogArgs("build-changelog", []string{
"--config=scripts/changelog.json",
"--output=CHANGELOG.md",
"--current-tag=v2",
"--previous-tag=v1",
"--api-base-url=https://gitea.example/api/v1",
"--token=secret",
})
if err != nil {
t.Fatalf("parseBuildChangelogArgs failed: %v", err)
}
if opts.configPath != "scripts/changelog.json" ||
opts.outputPath != "CHANGELOG.md" ||
opts.currentTag != "v2" ||
opts.previousTag != "v1" ||
opts.apiBaseURL != "https://gitea.example/api/v1" ||
opts.token != "secret" {
t.Fatalf("unexpected changelog opts: %#v", opts)
}
}
func mkdirAll(t *testing.T, path string) {
t.Helper()
if err := os.MkdirAll(path, 0o755); err != nil {
t.Fatalf("mkdir %s: %v", path, err)
}
}
func writeFile(t *testing.T, path, content string) {
t.Helper()
if err := os.WriteFile(path, []byte(content), 0o644); err != nil {
t.Fatalf("write %s: %v", path, err)
}
}
func setFileTime(t *testing.T, path string, modTime time.Time) {
t.Helper()
if err := os.Chtimes(path, modTime, modTime); err != nil {
t.Fatalf("set file time %s: %v", path, err)
}
}
internal/changelog/changelog.go
+455
View File
@@ -0,0 +1,455 @@
package changelog
import (
"bytes"
"encoding/json"
"fmt"
"io"
"net/http"
"net/url"
"os"
"os/exec"
"path/filepath"
"regexp"
"sort"
"strings"
)
type Config struct {
RepoURL string `json:"repo_url"`
Categories []Category `json:"categories"`
}
type Category struct {
Title string `json:"title"`
Sections []Section `json:"sections"`
}
type Section struct {
Title string `json:"title"`
Paths []string `json:"paths"`
}
type Options struct {
RepoRoot string
ConfigPath string
OutputPath string
CurrentTag string
PreviousTag string
APIBaseURL string
Token string
Stdout io.Writer
}
type pullInfo struct {
AuthorName string
}
type generator struct {
repoRoot string
config Config
currentTag string
previousTag string
apiBaseURL string
repoURL string
owner string
name string
token string
httpClient *http.Client
pullCache map[string]pullInfo
}
type changelogEntry struct {
Title string
ReferenceLabel string
ReferenceURL string
ReferenceSort string
AuthorName string
}
var pullNumberPattern = regexp.MustCompile(`\(#([0-9]+)\)`)
func Generate(opts Options) error {
repoRoot := strings.TrimSpace(opts.RepoRoot)
if repoRoot == "" {
detectedRepoRoot, err := gitOutput("", "rev-parse", "--show-toplevel")
if err != nil {
return fmt.Errorf("detect repo root: %w", err)
}
repoRoot = strings.TrimSpace(detectedRepoRoot)
}
configPath := opts.ConfigPath
if configPath == "" {
configPath = filepath.Join(repoRoot, "scripts", "changelog.json")
}
config, err := loadConfig(configPath)
if err != nil {
return err
}
currentTag := strings.TrimSpace(opts.CurrentTag)
if currentTag == "" {
currentTag, err = gitOutput(repoRoot, "describe", "--tags", "--exact-match")
if err != nil {
return fmt.Errorf("detect current tag: %w", err)
}
currentTag = strings.TrimSpace(currentTag)
}
if currentTag == "" {
return fmt.Errorf("could not determine current tag")
}
previousTag := strings.TrimSpace(opts.PreviousTag)
if previousTag == "" {
previousTag, err = previousTagFor(repoRoot, currentTag)
if err != nil {
return err
}
}
repoURL := strings.TrimSpace(config.RepoURL)
if repoURL == "" {
return fmt.Errorf("config %s is missing repo_url", configPath)
}
apiBaseURL := strings.TrimSpace(opts.APIBaseURL)
if apiBaseURL == "" {
if apiBaseURL = strings.TrimSpace(os.Getenv("GITEA_SERVER_URL")); apiBaseURL == "" {
apiBaseURL = deriveAPIBaseURL(repoURL)
}
}
apiBaseURL = normalizeAPIBaseURL(apiBaseURL)
if apiBaseURL == "" {
return fmt.Errorf("could not determine Gitea API base URL")
}
owner, name, err := ownerAndRepoFromURL(repoURL)
if err != nil {
return err
}
g := generator{
repoRoot: repoRoot,
config: config,
currentTag: currentTag,
previousTag: previousTag,
apiBaseURL: strings.TrimRight(apiBaseURL, "/"),
repoURL: strings.TrimRight(repoURL, "/"),
owner: owner,
name: name,
token: firstNonEmpty(opts.Token, os.Getenv("GITEA_API_TOKEN"), os.Getenv("GITEA_TOKEN"), os.Getenv("SOW_TOOLS_TOKEN")),
httpClient: http.DefaultClient,
pullCache: map[string]pullInfo{},
}
rendered, err := g.render()
if err != nil {
return err
}
if opts.OutputPath != "" {
if err := os.MkdirAll(filepath.Dir(opts.OutputPath), 0o755); err != nil {
return fmt.Errorf("create output directory: %w", err)
}
if err := os.WriteFile(opts.OutputPath, rendered, 0o644); err != nil {
return fmt.Errorf("write changelog: %w", err)
}
return nil
}
stdout := opts.Stdout
if stdout == nil {
stdout = os.Stdout
}
_, err = stdout.Write(rendered)
return err
}
func loadConfig(path string) (Config, error) {
var config Config
data, err := os.ReadFile(path)
if err != nil {
return Config{}, fmt.Errorf("read changelog config %s: %w", path, err)
}
if err := json.Unmarshal(data, &config); err != nil {
return Config{}, fmt.Errorf("parse changelog config %s: %w", path, err)
}
if len(config.Categories) == 0 {
return Config{}, fmt.Errorf("changelog config %s does not define any categories", path)
}
return config, nil
}
func previousTagFor(repoRoot string, currentTag string) (string, error) {
output, err := gitOutput(repoRoot, "for-each-ref", "--sort=-creatordate", "--format=%(refname:strip=2)", "refs/tags")
if err != nil {
return "", fmt.Errorf("list tags: %w", err)
}
tags := strings.Fields(output)
for idx, tag := range tags {
if tag != currentTag {
continue
}
if idx+1 < len(tags) {
return tags[idx+1], nil
}
return "", nil
}
return "", fmt.Errorf("current tag %s was not found in repository tag list", currentTag)
}
func deriveAPIBaseURL(repoURL string) string {
parsed, err := url.Parse(repoURL)
if err != nil || parsed.Scheme == "" || parsed.Host == "" {
return ""
}
return fmt.Sprintf("%s://%s/api/v1", parsed.Scheme, parsed.Host)
}
func normalizeAPIBaseURL(raw string) string {
raw = strings.TrimRight(strings.TrimSpace(raw), "/")
if raw == "" {
return ""
}
if strings.HasSuffix(raw, "/api/v1") {
return raw
}
return raw + "/api/v1"
}
func ownerAndRepoFromURL(repoURL string) (string, string, error) {
parsed, err := url.Parse(repoURL)
if err != nil {
return "", "", fmt.Errorf("parse repo_url %s: %w", repoURL, err)
}
parts := strings.Split(strings.Trim(parsed.Path, "/"), "/")
if len(parts) < 2 {
return "", "", fmt.Errorf("repo_url %s does not include owner/repo", repoURL)
}
owner := strings.TrimSpace(parts[0])
name := strings.TrimSuffix(strings.TrimSpace(parts[1]), ".git")
if owner == "" || name == "" {
return "", "", fmt.Errorf("repo_url %s does not include owner/repo", repoURL)
}
return owner, name, nil
}
func (g generator) render() ([]byte, error) {
publishDate, err := gitOutput(g.repoRoot, "log", "-1", "--format=%cs", g.currentTag)
if err != nil {
return nil, fmt.Errorf("resolve publish date: %w", err)
}
publishDate = strings.TrimSpace(publishDate)
var buf bytes.Buffer
fmt.Fprintf(&buf, "# %s - %s\n\n", g.currentTag, publishDate)
if g.previousTag != "" {
fmt.Fprintf(&buf, "_Compared against %s_\n\n", g.previousTag)
} else {
buf.WriteString("_Initial tagged release_\n\n")
}
for _, category := range g.config.Categories {
categoryText, err := g.renderCategory(category)
if err != nil {
return nil, err
}
if categoryText == "" {
continue
}
fmt.Fprintf(&buf, "## %s\n\n", category.Title)
buf.WriteString(categoryText)
}
return buf.Bytes(), nil
}
func (g generator) renderCategory(category Category) (string, error) {
var buf bytes.Buffer
for _, section := range category.Sections {
entries, err := g.sectionEntries(section)
if err != nil {
return "", err
}
if len(entries) == 0 {
continue
}
fmt.Fprintf(&buf, "### %s\n", section.Title)
for _, entry := range entries {
fmt.Fprintf(&buf, "- %s ([%s](%s)) - From %s\n", entry.Title, entry.ReferenceLabel, entry.ReferenceURL, entry.AuthorName)
}
buf.WriteString("\n")
}
return buf.String(), nil
}
func (g generator) sectionEntries(section Section) ([]changelogEntry, error) {
if len(section.Paths) == 0 {
return nil, nil
}
args := []string{"log", "--first-parent", "--pretty=format:%H%x1f%s%x1f%an"}
if g.previousTag != "" {
args = append(args, fmt.Sprintf("%s..%s", g.previousTag, g.currentTag))
} else {
args = append(args, g.currentTag)
}
args = append(args, "--")
args = append(args, section.Paths...)
output, err := gitOutput(g.repoRoot, args...)
if err != nil {
return nil, fmt.Errorf("list git log for section %s: %w", section.Title, err)
}
seen := map[string]struct{}{}
var entries []changelogEntry
for _, line := range strings.Split(output, "\n") {
if strings.TrimSpace(line) == "" {
continue
}
parts := strings.SplitN(line, "\x1f", 3)
commitHash := strings.TrimSpace(parts[0])
subject := ""
if len(parts) > 1 {
subject = strings.TrimSpace(parts[1])
}
fallbackAuthor := ""
if len(parts) > 2 {
fallbackAuthor = strings.TrimSpace(parts[2])
}
matches := pullNumberPattern.FindStringSubmatch(subject)
entryKey := commitHash
if len(matches) == 2 {
entryKey = "pr:" + matches[1]
}
if _, exists := seen[entryKey]; exists {
continue
}
seen[entryKey] = struct{}{}
entry := changelogEntry{
Title: sanitizeSubject(subject),
AuthorName: fallbackAuthor,
}
if len(matches) == 2 {
pullNumber := matches[1]
pullInfo, err := g.pullDetails(pullNumber)
if err != nil {
return nil, err
}
authorName := strings.TrimSpace(pullInfo.AuthorName)
if authorName == "" {
authorName = fallbackAuthor
}
entry.ReferenceLabel = "#" + pullNumber
entry.ReferenceURL = fmt.Sprintf("%s/pulls/%s", g.repoURL, pullNumber)
entry.ReferenceSort = "pr:" + pullNumber
entry.AuthorName = authorName
} else {
entry.ReferenceLabel = shortCommitHash(commitHash)
entry.ReferenceURL = fmt.Sprintf("%s/commit/%s", g.repoURL, commitHash)
entry.ReferenceSort = "commit:" + commitHash
}
entries = append(entries, entry)
}
sort.Slice(entries, func(i, j int) bool {
if entries[i].Title == entries[j].Title {
return entries[i].ReferenceSort < entries[j].ReferenceSort
}
return entries[i].Title < entries[j].Title
})
return entries, nil
}
func sanitizeSubject(subject string) string {
title := strings.TrimSpace(subject)
title = pullNumberPattern.ReplaceAllString(title, "")
title = strings.TrimSpace(title)
title = strings.TrimPrefix(title, "Merge pull request")
title = strings.TrimSpace(title)
title = strings.Trim(title, `"'`)
title = strings.TrimSpace(title)
if title == "" {
return strings.TrimSpace(subject)
}
return title
}
func shortCommitHash(commitHash string) string {
commitHash = strings.TrimSpace(commitHash)
if len(commitHash) <= 7 {
return commitHash
}
return commitHash[:7]
}
func (g generator) pullDetails(pullNumber string) (pullInfo, error) {
if cached, ok := g.pullCache[pullNumber]; ok {
return cached, nil
}
endpoint := fmt.Sprintf("%s/repos/%s/%s/pulls/%s", g.apiBaseURL, url.PathEscape(g.owner), url.PathEscape(g.name), url.PathEscape(pullNumber))
req, err := http.NewRequest(http.MethodGet, endpoint, nil)
if err != nil {
return pullInfo{}, fmt.Errorf("build Gitea request for pull #%s: %w", pullNumber, err)
}
req.Header.Set("Accept", "application/json")
if g.token != "" {
req.Header.Set("Authorization", "token "+g.token)
}
resp, err := g.httpClient.Do(req)
if err != nil {
return pullInfo{}, fmt.Errorf("request Gitea pull #%s: %w", pullNumber, err)
}
defer resp.Body.Close()
if resp.StatusCode != http.StatusOK {
body, _ := io.ReadAll(io.LimitReader(resp.Body, 512))
return pullInfo{}, fmt.Errorf("request Gitea pull #%s: unexpected HTTP %d: %s", pullNumber, resp.StatusCode, strings.TrimSpace(string(body)))
}
var payload struct {
User struct {
FullName string `json:"full_name"`
Login string `json:"login"`
} `json:"user"`
}
if err := json.NewDecoder(resp.Body).Decode(&payload); err != nil {
return pullInfo{}, fmt.Errorf("decode Gitea pull #%s: %w", pullNumber, err)
}
info := pullInfo{AuthorName: firstNonEmpty(strings.TrimSpace(payload.User.FullName), strings.TrimSpace(payload.User.Login))}
g.pullCache[pullNumber] = info
return info, nil
}
func gitOutput(repoRoot string, args ...string) (string, error) {
cmd := exec.Command("git", args...)
if repoRoot != "" {
cmd.Dir = repoRoot
}
output, err := cmd.CombinedOutput()
if err != nil {
return "", fmt.Errorf("%s: %s", strings.Join(cmd.Args, " "), strings.TrimSpace(string(output)))
}
return string(output), nil
}
func firstNonEmpty(values ...string) string {
for _, value := range values {
if strings.TrimSpace(value) != "" {
return value
}
}
return ""
}
internal/changelog/changelog_test.go
+213
View File
@@ -0,0 +1,213 @@
package changelog
import (
"bytes"
"fmt"
"net/http"
"net/http/httptest"
"os"
"os/exec"
"path/filepath"
"strings"
"testing"
)
func TestSanitizeSubject(t *testing.T) {
t.Parallel()
tests := []struct {
name string
subject string
want string
}{
{
name: "merge prefix with quotes",
subject: "Merge pull request 'Add classes overhaul (#123)'",
want: "Add classes overhaul",
},
{
name: "plain subject",
subject: "Add classes overhaul (#123)",
want: "Add classes overhaul",
},
{
name: "double quotes",
subject: `Merge pull request "Add classes overhaul (#123)"`,
want: "Add classes overhaul",
},
{
name: "direct push",
subject: "Fix direct push handling",
want: "Fix direct push handling",
},
}
for _, tt := range tests {
tt := tt
t.Run(tt.name, func(t *testing.T) {
t.Parallel()
if got := sanitizeSubject(tt.subject); got != tt.want {
t.Fatalf("sanitizeSubject(%q) = %q, want %q", tt.subject, got, tt.want)
}
})
}
}
func TestShortCommitHash(t *testing.T) {
t.Parallel()
if got := shortCommitHash("1234567890abcdef"); got != "1234567" {
t.Fatalf("shortCommitHash() = %q, want %q", got, "1234567")
}
if got := shortCommitHash("1234567"); got != "1234567" {
t.Fatalf("shortCommitHash() preserved %q unexpectedly as %q", "1234567", got)
}
}
func TestOwnerAndRepoFromURL(t *testing.T) {
t.Parallel()
owner, repo, err := ownerAndRepoFromURL("https://gitea.example.test/org/repo.git")
if err != nil {
t.Fatalf("ownerAndRepoFromURL returned error: %v", err)
}
if owner != "org" || repo != "repo" {
t.Fatalf("ownerAndRepoFromURL returned %q/%q", owner, repo)
}
}
func TestDeriveAPIBaseURL(t *testing.T) {
t.Parallel()
got := deriveAPIBaseURL("https://gitea.example.test/org/repo")
want := "https://gitea.example.test/api/v1"
if got != want {
t.Fatalf("deriveAPIBaseURL() = %q, want %q", got, want)
}
}
func TestNormalizeAPIBaseURL(t *testing.T) {
t.Parallel()
tests := []struct {
input string
want string
}{
{input: "https://gitea.example.test", want: "https://gitea.example.test/api/v1"},
{input: "https://gitea.example.test/api/v1", want: "https://gitea.example.test/api/v1"},
}
for _, tt := range tests {
if got := normalizeAPIBaseURL(tt.input); got != tt.want {
t.Fatalf("normalizeAPIBaseURL(%q) = %q, want %q", tt.input, got, tt.want)
}
}
}
func TestGenerateIncludesPullRequestsAndDirectPushes(t *testing.T) {
t.Parallel()
repoRoot := t.TempDir()
runGit(t, repoRoot, "init")
runGit(t, repoRoot, "config", "user.name", "Test User")
runGit(t, repoRoot, "config", "user.email", "test@example.com")
writeFile(t, filepath.Join(repoRoot, "content", "entry.txt"), "initial\n")
runGit(t, repoRoot, "add", ".")
runGit(t, repoRoot, "commit", "-m", "Initial import")
runGit(t, repoRoot, "tag", "v1.0.0")
writeFile(t, filepath.Join(repoRoot, "content", "entry.txt"), "direct\n")
runGit(t, repoRoot, "add", ".")
runGit(t, repoRoot, "commit", "-m", "Fix direct push handling")
directHash := strings.TrimSpace(runGit(t, repoRoot, "rev-parse", "HEAD"))
writeFile(t, filepath.Join(repoRoot, "content", "entry.txt"), "pull\n")
runGit(t, repoRoot, "add", ".")
runGit(t, repoRoot, "commit", "-m", "Add release summary (#12)")
runGit(t, repoRoot, "tag", "v1.1.0")
configPath := filepath.Join(repoRoot, "scripts", "changelog.json")
writeFile(t, configPath, `{
"repo_url": "REPO_URL",
"categories": [
{
"title": "Content",
"sections": [
{
"title": "Entries",
"paths": ["content/"]
}
]
}
]
}`)
var pullRequests int
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if r.URL.Path != "/api/v1/repos/org/repo/pulls/12" {
t.Fatalf("unexpected API path %q", r.URL.Path)
}
pullRequests++
w.Header().Set("Content-Type", "application/json")
fmt.Fprint(w, `{"user":{"full_name":"Patch Author","login":"patch-author"}}`)
}))
defer server.Close()
repoURL := server.URL + "/org/repo"
configData, err := os.ReadFile(configPath)
if err != nil {
t.Fatalf("read changelog config: %v", err)
}
configData = bytes.ReplaceAll(configData, []byte("REPO_URL"), []byte(repoURL))
if err := os.WriteFile(configPath, configData, 0o644); err != nil {
t.Fatalf("write changelog config: %v", err)
}
var stdout bytes.Buffer
err = Generate(Options{
RepoRoot: repoRoot,
ConfigPath: configPath,
CurrentTag: "v1.1.0",
PreviousTag: "v1.0.0",
APIBaseURL: server.URL + "/api/v1",
Stdout: &stdout,
})
if err != nil {
t.Fatalf("Generate() returned error: %v", err)
}
rendered := stdout.String()
if !strings.Contains(rendered, "- Add release summary ([#12]("+repoURL+"/pulls/12)) - From Patch Author") {
t.Fatalf("rendered changelog missing pull entry:\n%s", rendered)
}
if !strings.Contains(rendered, "- Fix direct push handling (["+shortCommitHash(directHash)+"]("+repoURL+"/commit/"+directHash+")) - From Test User") {
t.Fatalf("rendered changelog missing direct-push entry:\n%s", rendered)
}
if pullRequests != 1 {
t.Fatalf("expected 1 pull lookup, got %d", pullRequests)
}
}
func runGit(t *testing.T, repoRoot string, args ...string) string {
t.Helper()
cmd := exec.Command("git", args...)
cmd.Dir = repoRoot
output, err := cmd.CombinedOutput()
if err != nil {
t.Fatalf("git %s failed: %v\n%s", strings.Join(args, " "), err, output)
}
return string(output)
}
func writeFile(t *testing.T, path string, content string) {
t.Helper()
if err := os.MkdirAll(filepath.Dir(path), 0o755); err != nil {
t.Fatalf("MkdirAll(%q): %v", path, err)
}
if err := os.WriteFile(path, []byte(content), 0o644); err != nil {
t.Fatalf("WriteFile(%q): %v", path, err)
}
}
internal/dispatch/dispatch.go
+82 -25
View File
@@ -2,12 +2,11 @@
// builders shared by the `crucible` dispatcher and the standalone
// `crucible-<name>` shims.
//
// Scaffold contract (Phase 5): every builder is registered but UNWIRED. Running
// a builder fails closed (exit 70) with an operator-cutover message; it never
// fakes an artifact. The internal pipeline/topdata/erf/wiki/music packages from
// gitea/sow-tools are migrated by the operator at cutover (the workspace hard
// rules forbid transplanting source here). Once a builder's handler is wired,
// flip its Wired flag and register the handler.
// Cutover state (Phase 5 → 6): the internal pipeline/topdata/erf/wiki/music
// packages migrated from gitea/sow-tools now live in this tree, so wired
// builders delegate to internal/app's legacy command surface (mapped per
// docs/command-surface.md). A builder with no migrated logic yet (depot) keeps
// the Wired=false fail-closed path: exit 70, never a faked artifact.
package dispatch
import (
@@ -16,6 +15,7 @@ import (
"os"
"text/tabwriter"
"git.westgate.pw/ShadowsOverWestgate/sow-tools/internal/app"
"git.westgate.pw/ShadowsOverWestgate/sow-tools/internal/buildinfo"
)
@@ -33,6 +33,21 @@ type Builder struct {
Bin string // standalone binary name, e.g. "crucible-module"
Summary string // one-line description
Legacy []string // nwn-tool commands this subsumes (migration aid; see docs/command-surface.md)
Extra []string // additional callable subcommands beyond Legacy (e.g. cross-listed "music")
Wired bool // true once the builder delegates to migrated internal/app logic
}
// subcommands returns every legacy command a wired builder accepts: the
// canonical Legacy set plus any cross-listed Extra commands.
func (b Builder) subcommands() []string { return append(append([]string{}, b.Legacy...), b.Extra...) }
func (b Builder) accepts(sub string) bool {
for _, s := range b.subcommands() {
if s == sub {
return true
}
}
return false
}
// Registry is the single source of truth for the Crucible command surface.
@@ -43,30 +58,39 @@ var Registry = []Builder{
Bin: "crucible-depot",
Summary: "verify/move content-addressed depot blobs (SeaweedFS)",
Legacy: nil,
Wired: false, // no migrated logic yet; fails closed (exit 70)
},
{
Name: "hak",
Bin: "crucible-hak",
Summary: "pack/unpack ERF/HAK archives + hak manifests",
Legacy: []string{"build-haks", "apply-hak-manifest"},
Extra: []string{"music"}, // music BMUs are packed into HAKs (command-surface.md)
Wired: true,
},
{
Name: "module",
Bin: "crucible-module",
Summary: "build/extract/validate/compare the .mod",
Legacy: []string{"build", "build-module", "extract", "validate", "compare"},
// apply-hak-manifest is canonically a hak command but reachable here too;
// music is folded into the module build pipeline (command-surface.md).
Extra: []string{"apply-hak-manifest", "music"},
Wired: true,
},
{
Name: "topdata",
Bin: "crucible-topdata",
Summary: "compile 2da/tlk topdata + packages",
Legacy: []string{"validate-topdata", "build-topdata", "build-top-package", "compare-topdata", "convert-topdata"},
Wired: true,
},
{
Name: "wiki",
Bin: "crucible-wiki",
Summary: "render + deploy mechanical wiki pages",
Legacy: []string{"build-wiki", "deploy-wiki"},
Wired: true,
},
}
@@ -102,6 +126,13 @@ func run(args []string, out, errw io.Writer) int {
case "list":
list(out)
return exitOK
case "config":
// Global concern shared by every builder (command-surface.md): the legacy
// `config` command group, surfaced verbatim.
return delegateLegacy(args, errw)
case "changelog":
// Release tooling: global alias for the legacy `build-changelog` command.
return delegateLegacy(append([]string{"build-changelog"}, args[1:]...), errw)
}
return runBuilder(args[0], args[1:], out, errw)
}
@@ -120,19 +151,43 @@ func runBuilder(name string, args []string, out, errw io.Writer) int {
return exitOK
}
}
// Every builder is unwired in the Phase 5 scaffold: fail closed, never fake.
fmt.Fprintf(errw, unwiredMsg, b.Name, b.Bin)
return exitUnwired
if !b.Wired {
// No migrated logic yet (depot): fail closed, never fake an artifact.
fmt.Fprintf(errw, unwiredMsg, b.Name, b.Bin)
return exitUnwired
}
if len(args) == 0 {
fmt.Fprintf(errw, "crucible %s: a subcommand is required\n\n", b.Name)
builderHelp(errw, b)
return exitUsage
}
sub := args[0]
if !b.accepts(sub) {
fmt.Fprintf(errw, "crucible %s: unknown subcommand %q\n\n", b.Name, sub)
builderHelp(errw, b)
return exitUsage
}
// Delegate to the migrated legacy command surface. args[0] is already the
// legacy command name, so it is forwarded unchanged.
return delegateLegacy(args, errw)
}
// delegateLegacy runs a migrated nwn-tool command via internal/app and maps its
// (code, err) result onto the dispatcher's exit code. app writes its own output
// to os.Stdout/os.Stderr (the writers the entrypoints pass through), so only the
// returned error is surfaced here.
func delegateLegacy(args []string, errw io.Writer) int {
code, err := app.Run(args)
if err != nil {
fmt.Fprintln(errw, err)
}
return code
}
const unwiredMsg = `crucible %[1]s: not wired yet.
The Crucible suite is scaffolded (Phase 5). The %[2]s implementation is migrated
from gitea/sow-tools internal packages at operator cutover; the workspace hard
rules forbid transplanting that source into this tree automatically.
This command fails closed (exit 70) rather than producing a fake artifact.
See docs/migration-from-nwn-tool.md.
The %[2]s builder has no migrated logic in this tree yet, so it fails closed
(exit 70) rather than producing a fake artifact. See docs/command-surface.md.
`
func usage(w io.Writer) {
@@ -147,6 +202,8 @@ func usage(w io.Writer) {
fmt.Fprintf(w, " help | -h show this help\n")
fmt.Fprintf(w, " version | -V show the build version\n")
fmt.Fprintf(w, " list list builders (one per line: name<TAB>bin<TAB>summary)\n")
fmt.Fprintf(w, " config [args] inspect/validate effective configuration\n")
fmt.Fprintf(w, " changelog [args] generate the release changelog\n")
fmt.Fprintf(w, "\nNWN_ROOT must be passed explicitly (env or flag); crucible never guesses\n")
fmt.Fprintf(w, "from $HOME. See docs/consumer-contract.md.\n")
}
@@ -161,15 +218,15 @@ func list(w io.Writer) {
func builderHelp(w io.Writer, b Builder) {
fmt.Fprintf(w, "crucible %s (%s) — %s\n\n", b.Name, b.Bin, b.Summary)
if len(b.Legacy) > 0 {
fmt.Fprintf(w, "subsumes nwn-tool commands: ")
for i, c := range b.Legacy {
if i > 0 {
fmt.Fprintf(w, ", ")
}
fmt.Fprintf(w, "%s", c)
}
fmt.Fprintf(w, "\n\n")
if !b.Wired {
fmt.Fprintf(w, "status: not wired — no migrated logic yet; fails closed (exit 70).\n")
fmt.Fprintf(w, "See docs/command-surface.md.\n")
return
}
fmt.Fprintf(w, "status: scaffolded, not wired (Phase 5). See docs/command-surface.md.\n")
fmt.Fprintf(w, "usage: crucible %s <subcommand> [args] (or: %s <subcommand> [args])\n\n", b.Name, b.Bin)
fmt.Fprintf(w, "subcommands:\n")
for _, c := range b.subcommands() {
fmt.Fprintf(w, " %s\n", c)
}
fmt.Fprintf(w, "\nstatus: wired to migrated nwn-tool logic. See docs/command-surface.md.\n")
}
internal/dispatch/dispatch_test.go
+51 -1
View File
@@ -55,8 +55,11 @@ func TestUnknownBuilderFailsUsage(t *testing.T) {
}
}
func TestEveryBuilderFailsClosed(t *testing.T) {
func TestUnwiredBuilderFailsClosed(t *testing.T) {
for _, b := range Registry {
if b.Wired {
continue
}
var out, errw bytes.Buffer
// Via dispatcher.
if code := run([]string{b.Name}, &out, &errw); code != exitUnwired {
@@ -74,6 +77,28 @@ func TestEveryBuilderFailsClosed(t *testing.T) {
}
}
func TestWiredBuilderRejectsBadInvocation(t *testing.T) {
for _, b := range Registry {
if !b.Wired {
continue
}
// No subcommand is a usage error (it must never silently delegate).
var out, errw bytes.Buffer
if code := runBuilder(b.Name, nil, &out, &errw); code != exitUsage {
t.Errorf("crucible %s (no subcommand): exit=%d want %d", b.Name, code, exitUsage)
}
// Unknown subcommand is a usage error, not a delegate.
out.Reset()
errw.Reset()
if code := runBuilder(b.Name, []string{"frobnicate"}, &out, &errw); code != exitUsage {
t.Errorf("crucible %s frobnicate: exit=%d want %d", b.Name, code, exitUsage)
}
if !strings.Contains(errw.String(), "unknown subcommand") {
t.Errorf("crucible %s frobnicate: stderr=%q", b.Name, errw.String())
}
}
}
func TestBuilderHelpIsOK(t *testing.T) {
for _, b := range Registry {
var out, errw bytes.Buffer
@@ -85,3 +110,28 @@ func TestBuilderHelpIsOK(t *testing.T) {
}
}
}
// Every legacy command named in command-surface.md must have exactly one
// canonical home (Builder.Legacy), so the migrated surface has no gaps or
// ambiguous homes.
func TestLegacyCommandsHaveExactlyOneHome(t *testing.T) {
legacy := []string{
"build", "build-module", "extract", "validate", "compare",
"build-haks", "apply-hak-manifest",
"validate-topdata", "build-topdata", "build-top-package", "compare-topdata", "convert-topdata",
"build-wiki", "deploy-wiki",
}
for _, cmd := range legacy {
homes := 0
for _, b := range Registry {
for _, c := range b.Legacy {
if c == cmd {
homes++
}
}
}
if homes != 1 {
t.Errorf("legacy command %q has %d canonical homes, want 1", cmd, homes)
}
}
}
internal/erf/erf.go
+424
View File
@@ -0,0 +1,424 @@
package erf
import (
"bytes"
"encoding/binary"
"fmt"
"io"
"os"
"sort"
"strings"
)
const (
headerSize = 160
versionV10 = "V1.0"
strRefNone = 0xFFFFFFFF
typeMOD = "MOD "
typeHAK = "HAK "
)
type Archive struct {
FileType string
Version string
Resources []Resource
}
type Resource struct {
Name string
Type uint16
Data []byte
SourcePath string
Size int64
}
type header struct {
FileType [4]byte
Version [4]byte
LanguageCount uint32
LocalizedStringSize uint32
EntryCount uint32
LocalizedStringOffset uint32
KeyListOffset uint32
ResourceListOffset uint32
BuildYear uint32
BuildDay uint32
DescriptionStrRef uint32
Reserved [116]byte
}
type keyEntry struct {
ResRef [16]byte
ResourceID uint32
ResourceType uint16
Unused uint16
}
type resourceEntry struct {
Offset uint32
Size uint32
}
var extensionTypes = map[string]uint16{
"res": 0x0000,
"bmp": 0x0001,
"mve": 0x0002,
"tga": 0x0003,
"wav": 0x0004,
"plt": 0x0006,
"ini": 0x0007,
"bmu": 0x0008,
"txt": 0x000A,
"mdl": 0x07D2,
"nss": 0x07D9,
"ncs": 0x07DA,
"are": 0x07DC,
"set": 0x07DD,
"ifo": 0x07DE,
"bic": 0x07DF,
"wok": 0x07E0,
"2da": 0x07E1,
"tlk": 0x07E2,
"txi": 0x07E6,
"git": 0x07E7,
"uti": 0x07E9,
"utc": 0x07EB,
"dlg": 0x07ED,
"itp": 0x07EE,
"utt": 0x07F0,
"dds": 0x07F1,
"uts": 0x07F3,
"fac": 0x07F6,
"gff": 0x07F7,
"ute": 0x07F8,
"utd": 0x07FA,
"utp": 0x07FC,
"dfa": 0x07FD,
"gic": 0x07FE,
"gui": 0x07FF,
"utm": 0x0803,
"dwk": 0x0804,
"pwk": 0x0805,
"utg": 0x0807,
"jrl": 0x0808,
"utw": 0x080A,
"ssf": 0x080C,
"hak": 0x080D,
"nwm": 0x080E,
"bik": 0x080F,
"ndb": 0x0810,
"ptm": 0x0811,
"ptt": 0x0812,
"ltr": 0x0813,
"shd": 0x0815,
"mdb": 0x0816,
"mtr": 0x0818,
"jpg": 0x081C,
"lod": 0x081E,
"png": 0x0820,
"lyt": 0x0BB8,
"vis": 0x0BB9,
"mdx": 0x0BC0,
"wlk": 0x0BCC,
"xml": 0x0BCD,
"gr2": 0x0FA3,
}
var typeExtensions = map[uint16]string{
0x0000: "res",
0x0001: "bmp",
0x0002: "mve",
0x0003: "tga",
0x0004: "wav",
0x0006: "plt",
0x0007: "ini",
0x0008: "bmu",
0x000A: "txt",
0x07D2: "mdl",
0x07D9: "nss",
0x07DA: "ncs",
0x07DC: "are",
0x07DD: "set",
0x07DE: "ifo",
0x07DF: "bic",
0x07E0: "wok",
0x07E1: "2da",
0x07E2: "tlk",
0x07E6: "txi",
0x07E7: "git",
0x07E9: "uti",
0x07EB: "utc",
0x07ED: "dlg",
0x07EE: "itp",
0x07F0: "utt",
0x07F1: "dds",
0x07F3: "uts",
0x07F6: "fac",
0x07F7: "gff",
0x07F8: "ute",
0x07FA: "utd",
0x07FC: "utp",
0x07FD: "dfa",
0x07FE: "gic",
0x07FF: "gui",
0x0803: "utm",
0x0804: "dwk",
0x0805: "pwk",
0x0807: "utg",
0x0808: "jrl",
0x080A: "utw",
0x080C: "ssf",
0x080D: "hak",
0x080E: "nwm",
0x080F: "bik",
0x0810: "ndb",
0x0811: "ptm",
0x0812: "ptt",
0x0813: "ltr",
0x0815: "shd",
0x0816: "mdb",
0x0818: "mtr",
0x081C: "jpg",
0x081E: "lod",
0x0820: "png",
0x0BB8: "lyt",
0x0BB9: "vis",
0x0BC0: "mdx",
0x0BCC: "wlk",
0x0BCD: "xml",
0x0FA3: "gr2",
}
func init() {
for ext, resourceType := range extensionTypes {
canonicalExt, ok := typeExtensions[resourceType]
if !ok {
panic(fmt.Sprintf("missing canonical extension for resource type 0x%04X", resourceType))
}
if canonicalExt == ext {
continue
}
}
}
func New(fileType string, resources []Resource) Archive {
normalized := make([]Resource, len(resources))
copy(normalized, resources)
sort.Slice(normalized, func(i, j int) bool {
if normalized[i].Name == normalized[j].Name {
return normalized[i].Type < normalized[j].Type
}
return normalized[i].Name < normalized[j].Name
})
return Archive{
FileType: padFour(fileType),
Version: versionV10,
Resources: normalized,
}
}
func ArchiveSize(resources []Resource) int64 {
return int64(headerSize+len(resources)*24+len(resources)*8) + totalResourceBytes(resources)
}
func Write(w io.Writer, archive Archive) error {
if archive.Version == "" {
archive.Version = versionV10
}
if archive.FileType == "" {
archive.FileType = typeMOD
}
keys := make([]keyEntry, 0, len(archive.Resources))
entries := make([]resourceEntry, 0, len(archive.Resources))
dataOffset := uint32(headerSize + len(archive.Resources)*24 + len(archive.Resources)*8)
for index, resource := range archive.Resources {
if len(resource.Name) > 16 {
return fmt.Errorf("resource %q exceeds 16-byte resref limit", resource.Name)
}
size, err := payloadSize(resource)
if err != nil {
return err
}
if size > int64(^uint32(0)) {
return fmt.Errorf("resource %q exceeds 4 GiB resource size limit", resource.Name)
}
entry := resourceEntry{
Offset: dataOffset,
Size: uint32(size),
}
entries = append(entries, entry)
dataOffset += uint32(size)
var key keyEntry
copy(key.ResRef[:], []byte(resource.Name))
key.ResourceID = uint32(index)
key.ResourceType = resource.Type
keys = append(keys, key)
}
hdr := header{
LanguageCount: 0,
LocalizedStringSize: 0,
EntryCount: uint32(len(archive.Resources)),
LocalizedStringOffset: headerSize,
KeyListOffset: headerSize,
ResourceListOffset: uint32(headerSize + len(keys)*24),
BuildYear: 0,
BuildDay: 0,
DescriptionStrRef: strRefNone,
}
copy(hdr.FileType[:], []byte(padFour(archive.FileType)))
copy(hdr.Version[:], []byte(padFour(archive.Version)))
if err := binary.Write(w, binary.LittleEndian, hdr); err != nil {
return err
}
if err := binary.Write(w, binary.LittleEndian, keys); err != nil {
return err
}
if err := binary.Write(w, binary.LittleEndian, entries); err != nil {
return err
}
for _, resource := range archive.Resources {
if err := writeResourceData(w, resource); err != nil {
return err
}
}
return nil
}
func Read(r io.Reader) (Archive, error) {
data, err := io.ReadAll(r)
if err != nil {
return Archive{}, fmt.Errorf("read erf: %w", err)
}
if len(data) < headerSize {
return Archive{}, fmt.Errorf("erf file too small: %d bytes", len(data))
}
var hdr header
if err := binary.Read(bytes.NewReader(data[:headerSize]), binary.LittleEndian, &hdr); err != nil {
return Archive{}, fmt.Errorf("decode erf header: %w", err)
}
keyStart := int(hdr.KeyListOffset)
keyEnd := keyStart + int(hdr.EntryCount)*24
if keyEnd > len(data) {
return Archive{}, fmt.Errorf("erf key list exceeds file bounds")
}
keys := make([]keyEntry, hdr.EntryCount)
if err := binary.Read(bytes.NewReader(data[keyStart:keyEnd]), binary.LittleEndian, &keys); err != nil {
return Archive{}, fmt.Errorf("decode key list: %w", err)
}
resourceStart := int(hdr.ResourceListOffset)
resourceEnd := resourceStart + int(hdr.EntryCount)*8
if resourceEnd > len(data) {
return Archive{}, fmt.Errorf("erf resource list exceeds file bounds")
}
entries := make([]resourceEntry, hdr.EntryCount)
if err := binary.Read(bytes.NewReader(data[resourceStart:resourceEnd]), binary.LittleEndian, &entries); err != nil {
return Archive{}, fmt.Errorf("decode resource list: %w", err)
}
resources := make([]Resource, 0, hdr.EntryCount)
for index, key := range keys {
entry := entries[index]
start := int(entry.Offset)
end := start + int(entry.Size)
if end > len(data) {
return Archive{}, fmt.Errorf("resource %d exceeds file bounds", index)
}
resref := string(bytes.TrimRight(key.ResRef[:], "\x00"))
payload := make([]byte, entry.Size)
copy(payload, data[start:end])
resources = append(resources, Resource{
Name: resref,
Type: key.ResourceType,
Data: payload,
Size: int64(entry.Size),
})
}
return Archive{
FileType: string(hdr.FileType[:]),
Version: string(hdr.Version[:]),
Resources: resources,
}, nil
}
func ResourceTypeForExtension(extension string) (uint16, bool) {
resourceType, ok := extensionTypes[strings.TrimPrefix(strings.ToLower(extension), ".")]
return resourceType, ok
}
func HAKResourceTypeForExtension(extension string) (uint16, bool) {
return ResourceTypeForExtension(extension)
}
func ExtensionForResourceType(resourceType uint16) (string, bool) {
extension, ok := typeExtensions[resourceType]
return extension, ok
}
func padFour(value string) string {
value = strings.ToUpper(value)
if len(value) >= 4 {
return value[:4]
}
return value + strings.Repeat(" ", 4-len(value))
}
func totalResourceBytes(resources []Resource) int64 {
var total int64
for _, resource := range resources {
switch {
case resource.Size > 0:
total += resource.Size
default:
total += int64(len(resource.Data))
}
}
return total
}
func payloadSize(resource Resource) (int64, error) {
if resource.Size > 0 {
return resource.Size, nil
}
if resource.SourcePath != "" && len(resource.Data) == 0 {
info, err := os.Stat(resource.SourcePath)
if err != nil {
return 0, fmt.Errorf("stat resource %q: %w", resource.SourcePath, err)
}
return info.Size(), nil
}
return int64(len(resource.Data)), nil
}
func writeResourceData(w io.Writer, resource Resource) error {
if len(resource.Data) > 0 || resource.SourcePath == "" {
_, err := w.Write(resource.Data)
return err
}
file, err := os.Open(resource.SourcePath)
if err != nil {
return fmt.Errorf("open resource %q: %w", resource.SourcePath, err)
}
defer file.Close()
written, err := io.Copy(w, file)
if err != nil {
return fmt.Errorf("copy resource %q: %w", resource.SourcePath, err)
}
if resource.Size > 0 && written != resource.Size {
return fmt.Errorf("copy resource %q: expected %d bytes, wrote %d", resource.SourcePath, resource.Size, written)
}
return nil
}
internal/erf/erf_test.go
+130
View File
@@ -0,0 +1,130 @@
package erf
import (
"bytes"
"testing"
)
func TestArchiveRoundTrip(t *testing.T) {
archive := New("MOD ", []Resource{
{Name: "module", Type: 0x07DE, Data: []byte("ifo")},
{Name: "start", Type: 0x07DC, Data: []byte("are")},
})
var buf bytes.Buffer
if err := Write(&buf, archive); err != nil {
t.Fatalf("write: %v", err)
}
decoded, err := Read(bytes.NewReader(buf.Bytes()))
if err != nil {
t.Fatalf("read: %v", err)
}
if len(decoded.Resources) != 2 {
t.Fatalf("expected 2 resources, got %d", len(decoded.Resources))
}
if decoded.Resources[0].Name != "module" || string(decoded.Resources[0].Data) != "ifo" {
t.Fatalf("unexpected first resource: %#v", decoded.Resources[0])
}
}
func TestExtensionMappingsKeep2DAAndTLKDistinct(t *testing.T) {
resourceType, ok := ResourceTypeForExtension("2da")
if !ok {
t.Fatal("expected 2da resource type")
}
if resourceType != 0x07E1 {
t.Fatalf("expected 2da type 0x07E1, got 0x%04X", resourceType)
}
extension, ok := ExtensionForResourceType(resourceType)
if !ok {
t.Fatal("expected canonical extension for 2da type")
}
if extension != "2da" {
t.Fatalf("expected 2da extension, got %q", extension)
}
tlkType, ok := ResourceTypeForExtension("tlk")
if !ok {
t.Fatal("expected tlk resource type")
}
if tlkType != 0x07E2 {
t.Fatalf("expected tlk type 0x07E2, got 0x%04X", tlkType)
}
if tlkType == resourceType {
t.Fatal("expected 2da and tlk resource types to stay distinct")
}
}
func TestHAKResourceTypeForExtensionSupportsPNG(t *testing.T) {
if resourceType, ok := HAKResourceTypeForExtension("png"); !ok || resourceType != 0x0820 {
t.Fatalf("expected png restype 0x0820 (2080) for HAK generation, got ok=%v type=0x%04X", ok, resourceType)
}
if resourceType, ok := HAKResourceTypeForExtension("2da"); !ok || resourceType != 0x07E1 {
t.Fatalf("expected 2da to stay valid for HAK generation, got ok=%v type=0x%04X", ok, resourceType)
}
}
func TestExtensionMappingsSupportModernAssetTypes(t *testing.T) {
cases := map[string]uint16{
"mtr": 0x0818,
"shd": 0x0815,
"txi": 0x07E6,
"jpg": 0x081C,
"mdb": 0x0816,
"lyt": 0x0BB8,
"vis": 0x0BB9,
"mdx": 0x0BC0,
"xml": 0x0BCD,
"wlk": 0x0BCC,
"gr2": 0x0FA3,
}
for ext, wantType := range cases {
gotType, ok := ResourceTypeForExtension(ext)
if !ok {
t.Fatalf("expected %s to resolve to a resource type", ext)
}
if gotType != wantType {
t.Fatalf("expected %s type 0x%04X, got 0x%04X", ext, wantType, gotType)
}
gotExt, ok := ExtensionForResourceType(wantType)
if !ok {
t.Fatalf("expected type 0x%04X to resolve back to an extension", wantType)
}
if gotExt != ext {
t.Fatalf("expected type 0x%04X to resolve to %s, got %s", wantType, ext, gotExt)
}
}
}
func TestExtensionMappingsSupportAllUTBlueprintTypes(t *testing.T) {
cases := map[string]uint16{
"uti": 0x07E9,
"utc": 0x07EB,
"utt": 0x07F0,
"uts": 0x07F3,
"ute": 0x07F8,
"utd": 0x07FA,
"utp": 0x07FC,
"utm": 0x0803,
"utg": 0x0807,
"utw": 0x080A,
}
for ext, wantType := range cases {
gotType, ok := ResourceTypeForExtension(ext)
if !ok {
t.Fatalf("expected %s to resolve to a resource type", ext)
}
if gotType != wantType {
t.Fatalf("expected %s type 0x%04X, got 0x%04X", ext, wantType, gotType)
}
gotExt, ok := ExtensionForResourceType(wantType)
if !ok {
t.Fatalf("expected type 0x%04X to resolve back to an extension", wantType)
}
if gotExt != ext {
t.Fatalf("expected type 0x%04X to resolve to %s, got %s", wantType, ext, gotExt)
}
}
}
internal/gff/binary.go
+675
View File
@@ -0,0 +1,675 @@
package gff
import (
"bytes"
"encoding/binary"
"fmt"
"io"
"math"
)
const (
headerSize = 56
)
type header struct {
FileType [4]byte
FileVersion [4]byte
StructOffset uint32
StructCount uint32
FieldOffset uint32
FieldCount uint32
LabelOffset uint32
LabelCount uint32
FieldDataOffset uint32
FieldDataCount uint32
FieldIndicesOffset uint32
FieldIndicesCount uint32
ListIndicesOffset uint32
ListIndicesCount uint32
}
type rawStruct struct {
Type uint32
DataOrOffset uint32
FieldCount uint32
}
type rawField struct {
Type uint32
LabelIndex uint32
DataOrOffset uint32
}
func Read(r io.Reader) (Document, error) {
data, err := io.ReadAll(r)
if err != nil {
return Document{}, fmt.Errorf("read gff: %w", err)
}
if len(data) < headerSize {
return Document{}, fmt.Errorf("gff file too small: %d bytes", len(data))
}
var hdr header
if err := binary.Read(bytes.NewReader(data[:headerSize]), binary.LittleEndian, &hdr); err != nil {
return Document{}, fmt.Errorf("decode header: %w", err)
}
reader := binaryReader{data: data, hdr: hdr}
rawStructs, err := reader.readStructs()
if err != nil {
return Document{}, err
}
rawFields, err := reader.readFields()
if err != nil {
return Document{}, err
}
labels, err := reader.readLabels()
if err != nil {
return Document{}, err
}
root, err := reader.decodeStruct(0, rawStructs, rawFields, labels)
if err != nil {
return Document{}, err
}
return Document{
FileType: string(hdr.FileType[:]),
FileVersion: string(hdr.FileVersion[:]),
Root: root,
}, nil
}
func Write(w io.Writer, doc Document) error {
encoder := newBinaryEncoder(doc)
data, err := encoder.encode()
if err != nil {
return err
}
_, err = w.Write(data)
return err
}
type binaryReader struct {
data []byte
hdr header
}
func (r binaryReader) readStructs() ([]rawStruct, error) {
return readTable[rawStruct](r.data, r.hdr.StructOffset, r.hdr.StructCount)
}
func (r binaryReader) readFields() ([]rawField, error) {
return readTable[rawField](r.data, r.hdr.FieldOffset, r.hdr.FieldCount)
}
func (r binaryReader) readLabels() ([]string, error) {
start := int(r.hdr.LabelOffset)
end := start + int(r.hdr.LabelCount)*16
if end > len(r.data) {
return nil, fmt.Errorf("label table exceeds file bounds")
}
labels := make([]string, 0, r.hdr.LabelCount)
for offset := start; offset < end; offset += 16 {
chunk := r.data[offset : offset+16]
n := bytes.IndexByte(chunk, 0)
if n == -1 {
n = len(chunk)
}
labels = append(labels, string(chunk[:n]))
}
return labels, nil
}
func (r binaryReader) decodeStruct(index uint32, structs []rawStruct, fields []rawField, labels []string) (Struct, error) {
if index >= uint32(len(structs)) {
return Struct{}, fmt.Errorf("struct index %d out of range", index)
}
raw := structs[index]
fieldIndices, err := r.fieldIndices(raw)
if err != nil {
return Struct{}, err
}
out := Struct{
Type: raw.Type,
Fields: make([]Field, 0, len(fieldIndices)),
}
for _, fieldIndex := range fieldIndices {
if fieldIndex >= uint32(len(fields)) {
return Struct{}, fmt.Errorf("field index %d out of range", fieldIndex)
}
field, err := r.decodeField(fields[fieldIndex], structs, fields, labels)
if err != nil {
return Struct{}, err
}
out.Fields = append(out.Fields, field)
}
return out, nil
}
func (r binaryReader) fieldIndices(s rawStruct) ([]uint32, error) {
switch s.FieldCount {
case 0:
return nil, nil
case 1:
return []uint32{s.DataOrOffset}, nil
default:
start := int(r.hdr.FieldIndicesOffset + s.DataOrOffset)
end := start + int(s.FieldCount)*4
if end > len(r.data) {
return nil, fmt.Errorf("field indices exceed file bounds")
}
out := make([]uint32, s.FieldCount)
reader := bytes.NewReader(r.data[start:end])
if err := binary.Read(reader, binary.LittleEndian, &out); err != nil {
return nil, fmt.Errorf("decode field indices: %w", err)
}
return out, nil
}
}
func (r binaryReader) decodeField(raw rawField, structs []rawStruct, fields []rawField, labels []string) (Field, error) {
if raw.LabelIndex >= uint32(len(labels)) {
return Field{}, fmt.Errorf("label index %d out of range", raw.LabelIndex)
}
fieldType := FieldType(raw.Type)
value, err := r.decodeValue(fieldType, raw.DataOrOffset, structs, fields, labels)
if err != nil {
return Field{}, fmt.Errorf("decode field %q: %w", labels[raw.LabelIndex], err)
}
return Field{
Label: labels[raw.LabelIndex],
Type: fieldType,
Value: value,
}, nil
}
func (r binaryReader) decodeValue(fieldType FieldType, data uint32, structs []rawStruct, fields []rawField, labels []string) (Value, error) {
switch fieldType {
case TypeByte:
return ByteValue(uint8(data)), nil
case TypeChar:
return CharValue(int8(data)), nil
case TypeWord:
return WordValue(uint16(data)), nil
case TypeShort:
return ShortValue(int16(data)), nil
case TypeDWord:
return DWordValue(data), nil
case TypeInt:
return IntValue(int32(data)), nil
case TypeFloat:
return FloatValue(math.Float32frombits(data)), nil
case TypeDWord64:
raw, err := r.sliceFieldData(data, 8)
if err != nil {
return nil, err
}
return DWord64Value(binary.LittleEndian.Uint64(raw)), nil
case TypeInt64:
raw, err := r.sliceFieldData(data, 8)
if err != nil {
return nil, err
}
return Int64Value(int64(binary.LittleEndian.Uint64(raw))), nil
case TypeDouble:
raw, err := r.sliceFieldData(data, 8)
if err != nil {
return nil, err
}
return DoubleValue(math.Float64frombits(binary.LittleEndian.Uint64(raw))), nil
case TypeCExoString:
raw, err := r.prefixedFieldData(data)
if err != nil {
return nil, err
}
return StringValue(string(raw)), nil
case TypeResRef:
raw, err := r.prefixedByteFieldData(data, 1)
if err != nil {
return nil, err
}
return ResRefValue(string(raw)), nil
case TypeCExoLocString:
raw, err := r.locStringFieldData(data)
if err != nil {
return nil, err
}
return decodeLocString(raw)
case TypeVoid:
raw, err := r.prefixedFieldData(data)
if err != nil {
return nil, err
}
return VoidValue(raw), nil
case TypeStruct:
return r.decodeStruct(data, structs, fields, labels)
case TypeList:
return r.decodeList(data, structs, fields, labels)
case TypeOrientation:
raw, err := r.sliceFieldData(data, 16)
if err != nil {
return nil, err
}
return Orientation{
X: math.Float32frombits(binary.LittleEndian.Uint32(raw[0:4])),
Y: math.Float32frombits(binary.LittleEndian.Uint32(raw[4:8])),
Z: math.Float32frombits(binary.LittleEndian.Uint32(raw[8:12])),
W: math.Float32frombits(binary.LittleEndian.Uint32(raw[12:16])),
}, nil
case TypeVector:
raw, err := r.sliceFieldData(data, 12)
if err != nil {
return nil, err
}
return Vector{
X: math.Float32frombits(binary.LittleEndian.Uint32(raw[0:4])),
Y: math.Float32frombits(binary.LittleEndian.Uint32(raw[4:8])),
Z: math.Float32frombits(binary.LittleEndian.Uint32(raw[8:12])),
}, nil
default:
return nil, fmt.Errorf("unsupported field type %d", fieldType)
}
}
func (r binaryReader) decodeList(offset uint32, structs []rawStruct, fields []rawField, labels []string) (ListValue, error) {
start := int(r.hdr.ListIndicesOffset + offset)
if start+4 > len(r.data) {
return nil, fmt.Errorf("list data exceeds file bounds")
}
count := binary.LittleEndian.Uint32(r.data[start : start+4])
start += 4
end := start + int(count)*4
if end > len(r.data) {
return nil, fmt.Errorf("list indices exceed file bounds")
}
list := make(ListValue, 0, count)
for pos := start; pos < end; pos += 4 {
index := binary.LittleEndian.Uint32(r.data[pos : pos+4])
child, err := r.decodeStruct(index, structs, fields, labels)
if err != nil {
return nil, err
}
list = append(list, child)
}
return list, nil
}
func (r binaryReader) sliceFieldData(offset uint32, size int) ([]byte, error) {
start := int(r.hdr.FieldDataOffset + offset)
end := start + size
if end > len(r.data) {
return nil, fmt.Errorf("field data exceeds file bounds")
}
return r.data[start:end], nil
}
func (r binaryReader) prefixedFieldData(offset uint32) ([]byte, error) {
sizeRaw, err := r.sliceFieldData(offset, 4)
if err != nil {
return nil, err
}
size := int(binary.LittleEndian.Uint32(sizeRaw))
return r.sliceFieldData(offset+4, size)
}
func (r binaryReader) prefixedByteFieldData(offset uint32, prefixBytes uint32) ([]byte, error) {
header, err := r.sliceFieldData(offset, int(prefixBytes))
if err != nil {
return nil, err
}
size := int(header[0])
return r.sliceFieldData(offset+prefixBytes, size)
}
func (r binaryReader) locStringFieldData(offset uint32) ([]byte, error) {
sizeRaw, err := r.sliceFieldData(offset, 4)
if err != nil {
return nil, err
}
size := int(binary.LittleEndian.Uint32(sizeRaw))
return r.sliceFieldData(offset, size+4)
}
func readTable[T any](data []byte, offset uint32, count uint32) ([]T, error) {
if count == 0 {
return nil, nil
}
var row T
size := binary.Size(row)
start := int(offset)
end := start + int(count)*size
if end > len(data) {
return nil, fmt.Errorf("table exceeds file bounds")
}
out := make([]T, count)
reader := bytes.NewReader(data[start:end])
if err := binary.Read(reader, binary.LittleEndian, &out); err != nil {
return nil, fmt.Errorf("decode table: %w", err)
}
return out, nil
}
func decodeLocString(data []byte) (LocString, error) {
if len(data) < 12 {
return LocString{}, fmt.Errorf("locstring too small")
}
totalSize := binary.LittleEndian.Uint32(data[0:4])
if totalSize+4 != uint32(len(data)) {
return LocString{}, fmt.Errorf("locstring size mismatch")
}
stringRef := binary.LittleEndian.Uint32(data[4:8])
count := binary.LittleEndian.Uint32(data[8:12])
pos := 12
entries := make([]LocStringEntry, 0, count)
for i := uint32(0); i < count; i++ {
if pos+8 > len(data) {
return LocString{}, fmt.Errorf("locstring entry header truncated")
}
id := binary.LittleEndian.Uint32(data[pos : pos+4])
length := binary.LittleEndian.Uint32(data[pos+4 : pos+8])
pos += 8
if pos+int(length) > len(data) {
return LocString{}, fmt.Errorf("locstring entry %d truncated", i)
}
entries = append(entries, LocStringEntry{
ID: id,
Value: string(data[pos : pos+int(length)]),
})
pos += int(length)
}
return LocString{
StringRef: stringRef,
Entries: entries,
}, nil
}
type binaryEncoder struct {
document Document
structs []rawStruct
fields []rawField
labels []string
labelIndex map[string]uint32
fieldData bytes.Buffer
fieldIndices []uint32
listIndices []uint32
}
func newBinaryEncoder(doc Document) *binaryEncoder {
return &binaryEncoder{
document: doc,
labelIndex: map[string]uint32{},
}
}
func (e *binaryEncoder) encode() ([]byte, error) {
if err := e.addStruct(e.document.Root); err != nil {
return nil, err
}
var buf bytes.Buffer
hdr := header{}
copy(hdr.FileType[:], []byte(padFour(e.document.FileType)))
copy(hdr.FileVersion[:], []byte(padFour(defaultString(e.document.FileVersion, "V3.2"))))
hdr.StructOffset = headerSize
hdr.StructCount = uint32(len(e.structs))
hdr.FieldOffset = hdr.StructOffset + uint32(len(e.structs))*12
hdr.FieldCount = uint32(len(e.fields))
hdr.LabelOffset = hdr.FieldOffset + uint32(len(e.fields))*12
hdr.LabelCount = uint32(len(e.labels))
hdr.FieldDataOffset = hdr.LabelOffset + uint32(len(e.labels))*16
hdr.FieldDataCount = uint32(e.fieldData.Len())
hdr.FieldIndicesOffset = hdr.FieldDataOffset + uint32(e.fieldData.Len())
hdr.FieldIndicesCount = uint32(len(e.fieldIndices) * 4)
hdr.ListIndicesOffset = hdr.FieldIndicesOffset + uint32(len(e.fieldIndices))*4
hdr.ListIndicesCount = uint32(len(e.listIndices) * 4)
if err := binary.Write(&buf, binary.LittleEndian, hdr); err != nil {
return nil, err
}
if err := binary.Write(&buf, binary.LittleEndian, e.structs); err != nil {
return nil, err
}
if err := binary.Write(&buf, binary.LittleEndian, e.fields); err != nil {
return nil, err
}
for _, label := range e.labels {
var raw [16]byte
copy(raw[:], []byte(label))
if _, err := buf.Write(raw[:]); err != nil {
return nil, err
}
}
if _, err := buf.Write(e.fieldData.Bytes()); err != nil {
return nil, err
}
if err := binary.Write(&buf, binary.LittleEndian, e.fieldIndices); err != nil {
return nil, err
}
if err := binary.Write(&buf, binary.LittleEndian, e.listIndices); err != nil {
return nil, err
}
return buf.Bytes(), nil
}
func (e *binaryEncoder) addStruct(s Struct) error {
index := uint32(len(e.structs))
e.structs = append(e.structs, rawStruct{Type: s.Type})
fieldIndices := make([]uint32, 0, len(s.Fields))
for _, field := range s.Fields {
fieldIndex, err := e.addField(field)
if err != nil {
return err
}
fieldIndices = append(fieldIndices, fieldIndex)
}
entry := &e.structs[index]
entry.FieldCount = uint32(len(fieldIndices))
switch len(fieldIndices) {
case 0:
entry.DataOrOffset = 0
case 1:
entry.DataOrOffset = fieldIndices[0]
default:
entry.DataOrOffset = uint32(len(e.fieldIndices) * 4)
e.fieldIndices = append(e.fieldIndices, fieldIndices...)
}
return nil
}
func (e *binaryEncoder) addField(field Field) (uint32, error) {
labelIndex, err := e.indexLabel(field.Label)
if err != nil {
return 0, err
}
data, err := e.encodeValue(field.Value)
if err != nil {
return 0, fmt.Errorf("encode field %q: %w", field.Label, err)
}
entry := rawField{
Type: uint32(field.Type),
LabelIndex: labelIndex,
DataOrOffset: data,
}
index := uint32(len(e.fields))
e.fields = append(e.fields, entry)
return index, nil
}
func (e *binaryEncoder) encodeValue(value Value) (uint32, error) {
switch typed := value.(type) {
case ByteValue:
return uint32(uint8(typed)), nil
case CharValue:
return uint32(uint8(typed)), nil
case WordValue:
return uint32(uint16(typed)), nil
case ShortValue:
return uint32(uint16(typed)), nil
case DWordValue:
return uint32(typed), nil
case IntValue:
return uint32(int32(typed)), nil
case FloatValue:
return math.Float32bits(float32(typed)), nil
case DWord64Value:
return e.writeFieldData(func(buf *bytes.Buffer) error {
return binary.Write(buf, binary.LittleEndian, uint64(typed))
}), nil
case Int64Value:
return e.writeFieldData(func(buf *bytes.Buffer) error {
return binary.Write(buf, binary.LittleEndian, int64(typed))
}), nil
case DoubleValue:
return e.writeFieldData(func(buf *bytes.Buffer) error {
return binary.Write(buf, binary.LittleEndian, math.Float64bits(float64(typed)))
}), nil
case StringValue:
return e.writeLengthPrefixed([]byte(typed)), nil
case ResRefValue:
if len(typed) > 255 {
return 0, fmt.Errorf("resref exceeds 255 bytes")
}
return e.writeFieldData(func(buf *bytes.Buffer) error {
if err := buf.WriteByte(byte(len(typed))); err != nil {
return err
}
_, err := buf.Write([]byte(typed))
return err
}), nil
case LocString:
return e.writeFieldData(func(buf *bytes.Buffer) error {
payload := bytes.Buffer{}
if err := binary.Write(&payload, binary.LittleEndian, typed.StringRef); err != nil {
return err
}
if err := binary.Write(&payload, binary.LittleEndian, uint32(len(typed.Entries))); err != nil {
return err
}
for _, entry := range typed.Entries {
if err := binary.Write(&payload, binary.LittleEndian, entry.ID); err != nil {
return err
}
if err := binary.Write(&payload, binary.LittleEndian, uint32(len(entry.Value))); err != nil {
return err
}
if _, err := payload.Write([]byte(entry.Value)); err != nil {
return err
}
}
if err := binary.Write(buf, binary.LittleEndian, uint32(payload.Len())); err != nil {
return err
}
if _, err := buf.Write(payload.Bytes()); err != nil {
return err
}
return nil
}), nil
case VoidValue:
return e.writeLengthPrefixed([]byte(typed)), nil
case Struct:
index := uint32(len(e.structs))
if err := e.addStruct(typed); err != nil {
return 0, err
}
return index, nil
case ListValue:
return e.writeList(typed)
case Orientation:
return e.writeFieldData(func(buf *bytes.Buffer) error {
for _, component := range []float32{typed.X, typed.Y, typed.Z, typed.W} {
if err := binary.Write(buf, binary.LittleEndian, math.Float32bits(component)); err != nil {
return err
}
}
return nil
}), nil
case Vector:
return e.writeFieldData(func(buf *bytes.Buffer) error {
for _, component := range []float32{typed.X, typed.Y, typed.Z} {
if err := binary.Write(buf, binary.LittleEndian, math.Float32bits(component)); err != nil {
return err
}
}
return nil
}), nil
default:
return 0, fmt.Errorf("unsupported value type %T", value)
}
}
func (e *binaryEncoder) writeList(list ListValue) (uint32, error) {
indices := make([]uint32, 0, len(list))
for _, item := range list {
index := uint32(len(e.structs))
if err := e.addStruct(item); err != nil {
return 0, err
}
indices = append(indices, index)
}
offset := uint32(len(e.listIndices) * 4)
e.listIndices = append(e.listIndices, uint32(len(list)))
e.listIndices = append(e.listIndices, indices...)
return offset, nil
}
func (e *binaryEncoder) writeFieldData(write func(*bytes.Buffer) error) uint32 {
offset := uint32(e.fieldData.Len())
if err := write(&e.fieldData); err != nil {
panic(err)
}
return offset
}
func (e *binaryEncoder) writeLengthPrefixed(data []byte) uint32 {
return e.writeFieldData(func(buf *bytes.Buffer) error {
if err := binary.Write(buf, binary.LittleEndian, uint32(len(data))); err != nil {
return err
}
_, err := buf.Write(data)
return err
})
}
func (e *binaryEncoder) indexLabel(label string) (uint32, error) {
if len(label) > 16 {
return 0, fmt.Errorf("label %q exceeds 16 bytes", label)
}
if index, ok := e.labelIndex[label]; ok {
return index, nil
}
index := uint32(len(e.labels))
e.labels = append(e.labels, label)
e.labelIndex[label] = index
return index, nil
}
func padFour(value string) string {
if len(value) >= 4 {
return value[:4]
}
return value + string(bytes.Repeat([]byte(" "), 4-len(value)))
}
func defaultString(value, fallback string) string {
if value == "" {
return fallback
}
return value
}
internal/gff/binary_test.go
+55
View File
@@ -0,0 +1,55 @@
package gff
import (
"bytes"
"encoding/json"
"testing"
)
func TestRoundTripBinaryAndJSON(t *testing.T) {
original := Document{
FileType: "UTC ",
FileVersion: "V3.2",
Root: Struct{
Type: 0,
Fields: []Field{
NewField("Tag", StringValue("test_creature")),
NewField("TemplateResRef", ResRefValue("nw_test")),
NewField("HP", IntValue(12)),
NewField("Position", Vector{X: 1.25, Y: 2.5, Z: 3.75}),
NewField("Inventory", ListValue{
{
Type: 1,
Fields: []Field{
NewField("Slot", DWordValue(0)),
NewField("ResRef", ResRefValue("itm_sword")),
},
},
}),
},
},
}
var binaryBuf bytes.Buffer
if err := Write(&binaryBuf, original); err != nil {
t.Fatalf("write binary: %v", err)
}
decoded, err := Read(bytes.NewReader(binaryBuf.Bytes()))
if err != nil {
t.Fatalf("read binary: %v", err)
}
left, err := json.Marshal(original)
if err != nil {
t.Fatalf("marshal original json: %v", err)
}
right, err := json.Marshal(decoded)
if err != nil {
t.Fatalf("marshal decoded json: %v", err)
}
if string(left) != string(right) {
t.Fatalf("roundtrip mismatch\noriginal: %s\ndecoded: %s", left, right)
}
}
internal/gff/json.go
+317
View File
@@ -0,0 +1,317 @@
package gff
import (
"encoding/base64"
"encoding/json"
"fmt"
)
type jsonDocument struct {
FileType string `json:"file_type"`
FileVersion string `json:"file_version"`
Root jsonStruct `json:"root"`
}
type jsonStruct struct {
StructType uint32 `json:"struct_type"`
Fields []jsonField `json:"fields"`
}
type jsonField struct {
Label string `json:"label"`
Type string `json:"type"`
Value json.RawMessage `json:"value"`
}
type jsonLocString struct {
StringRef uint32 `json:"string_ref"`
Entries []LocStringEntry `json:"entries"`
}
func (s Struct) MarshalJSON() ([]byte, error) {
payload, err := marshalStruct(s)
if err != nil {
return nil, err
}
return json.Marshal(payload)
}
func (s *Struct) UnmarshalJSON(data []byte) error {
var payload jsonStruct
if err := json.Unmarshal(data, &payload); err != nil {
return err
}
decoded, err := unmarshalStruct(payload)
if err != nil {
return err
}
*s = decoded
return nil
}
func (d Document) MarshalJSON() ([]byte, error) {
root, err := marshalStruct(d.Root)
if err != nil {
return nil, err
}
return json.Marshal(jsonDocument{
FileType: d.FileType,
FileVersion: d.FileVersion,
Root: root,
})
}
func (d *Document) UnmarshalJSON(data []byte) error {
var payload jsonDocument
if err := json.Unmarshal(data, &payload); err != nil {
return err
}
root, err := unmarshalStruct(payload.Root)
if err != nil {
return err
}
d.FileType = payload.FileType
d.FileVersion = payload.FileVersion
d.Root = root
return nil
}
func marshalStruct(in Struct) (jsonStruct, error) {
fields := make([]jsonField, 0, len(in.Fields))
for _, field := range in.Fields {
payload, err := marshalValue(field.Value)
if err != nil {
return jsonStruct{}, fmt.Errorf("marshal field %q: %w", field.Label, err)
}
fields = append(fields, jsonField{
Label: field.Label,
Type: field.Type.String(),
Value: payload,
})
}
return jsonStruct{
StructType: in.Type,
Fields: fields,
}, nil
}
func unmarshalStruct(in jsonStruct) (Struct, error) {
fields := make([]Field, 0, len(in.Fields))
for _, field := range in.Fields {
ft, err := parseFieldType(field.Type)
if err != nil {
return Struct{}, fmt.Errorf("field %q: %w", field.Label, err)
}
value, err := unmarshalValue(ft, field.Value)
if err != nil {
return Struct{}, fmt.Errorf("field %q: %w", field.Label, err)
}
fields = append(fields, Field{
Label: field.Label,
Type: ft,
Value: value,
})
}
return Struct{
Type: in.StructType,
Fields: fields,
}, nil
}
func marshalValue(value Value) (json.RawMessage, error) {
switch typed := value.(type) {
case ByteValue:
return json.Marshal(uint8(typed))
case CharValue:
return json.Marshal(int8(typed))
case WordValue:
return json.Marshal(uint16(typed))
case ShortValue:
return json.Marshal(int16(typed))
case DWordValue:
return json.Marshal(uint32(typed))
case IntValue:
return json.Marshal(int32(typed))
case DWord64Value:
return json.Marshal(uint64(typed))
case Int64Value:
return json.Marshal(int64(typed))
case FloatValue:
return json.Marshal(float32(typed))
case DoubleValue:
return json.Marshal(float64(typed))
case StringValue:
return json.Marshal(string(typed))
case ResRefValue:
return json.Marshal(string(typed))
case LocString:
return json.Marshal(jsonLocString(typed))
case VoidValue:
return json.Marshal(base64.StdEncoding.EncodeToString([]byte(typed)))
case Struct:
payload, err := marshalStruct(typed)
if err != nil {
return nil, err
}
return json.Marshal(payload)
case ListValue:
payload := make([]jsonStruct, 0, len(typed))
for _, item := range typed {
entry, err := marshalStruct(item)
if err != nil {
return nil, err
}
payload = append(payload, entry)
}
return json.Marshal(payload)
case Orientation:
return json.Marshal(typed)
case Vector:
return json.Marshal(typed)
default:
return nil, fmt.Errorf("unsupported value type %T", value)
}
}
func unmarshalValue(ft FieldType, data []byte) (Value, error) {
switch ft {
case TypeByte:
var out uint8
if err := json.Unmarshal(data, &out); err != nil {
return nil, err
}
return ByteValue(out), nil
case TypeChar:
var out int8
if err := json.Unmarshal(data, &out); err != nil {
return nil, err
}
return CharValue(out), nil
case TypeWord:
var out uint16
if err := json.Unmarshal(data, &out); err != nil {
return nil, err
}
return WordValue(out), nil
case TypeShort:
var out int16
if err := json.Unmarshal(data, &out); err != nil {
return nil, err
}
return ShortValue(out), nil
case TypeDWord:
var out uint32
if err := json.Unmarshal(data, &out); err != nil {
return nil, err
}
return DWordValue(out), nil
case TypeInt:
var out int32
if err := json.Unmarshal(data, &out); err != nil {
return nil, err
}
return IntValue(out), nil
case TypeDWord64:
var out uint64
if err := json.Unmarshal(data, &out); err != nil {
return nil, err
}
return DWord64Value(out), nil
case TypeInt64:
var out int64
if err := json.Unmarshal(data, &out); err != nil {
return nil, err
}
return Int64Value(out), nil
case TypeFloat:
var out float32
if err := json.Unmarshal(data, &out); err != nil {
return nil, err
}
return FloatValue(out), nil
case TypeDouble:
var out float64
if err := json.Unmarshal(data, &out); err != nil {
return nil, err
}
return DoubleValue(out), nil
case TypeCExoString:
var out string
if err := json.Unmarshal(data, &out); err != nil {
return nil, err
}
return StringValue(out), nil
case TypeResRef:
var out string
if err := json.Unmarshal(data, &out); err != nil {
return nil, err
}
return ResRefValue(out), nil
case TypeCExoLocString:
var out jsonLocString
if err := json.Unmarshal(data, &out); err != nil {
return nil, err
}
return LocString(out), nil
case TypeVoid:
var out string
if err := json.Unmarshal(data, &out); err != nil {
return nil, err
}
decoded, err := base64.StdEncoding.DecodeString(out)
if err != nil {
return nil, fmt.Errorf("decode base64 void: %w", err)
}
return VoidValue(decoded), nil
case TypeStruct:
var payload jsonStruct
if err := json.Unmarshal(data, &payload); err != nil {
return nil, err
}
return unmarshalStruct(payload)
case TypeList:
var payload []jsonStruct
if err := json.Unmarshal(data, &payload); err != nil {
return nil, err
}
out := make(ListValue, 0, len(payload))
for _, item := range payload {
decoded, err := unmarshalStruct(item)
if err != nil {
return nil, err
}
out = append(out, decoded)
}
return out, nil
case TypeOrientation:
var out Orientation
if err := json.Unmarshal(data, &out); err != nil {
return nil, err
}
return out, nil
case TypeVector:
var out Vector
if err := json.Unmarshal(data, &out); err != nil {
return nil, err
}
return out, nil
default:
return nil, fmt.Errorf("unsupported field type %d", ft)
}
}
func parseFieldType(name string) (FieldType, error) {
for kind, candidate := range fieldTypeNames {
if candidate == name {
return kind, nil
}
}
return 0, fmt.Errorf("unknown field type %q", name)
}
internal/gff/types.go
+140
View File
@@ -0,0 +1,140 @@
package gff
import "fmt"
type FieldType uint32
const (
TypeByte FieldType = iota
TypeChar
TypeWord
TypeShort
TypeDWord
TypeInt
TypeDWord64
TypeInt64
TypeFloat
TypeDouble
TypeCExoString
TypeResRef
TypeCExoLocString
TypeVoid
TypeStruct
TypeList
TypeOrientation
TypeVector
)
var fieldTypeNames = map[FieldType]string{
TypeByte: "Byte",
TypeChar: "Char",
TypeWord: "Word",
TypeShort: "Short",
TypeDWord: "DWord",
TypeInt: "Int",
TypeDWord64: "DWord64",
TypeInt64: "Int64",
TypeFloat: "Float",
TypeDouble: "Double",
TypeCExoString: "CExoString",
TypeResRef: "ResRef",
TypeCExoLocString: "CExoLocString",
TypeVoid: "Void",
TypeStruct: "Struct",
TypeList: "List",
TypeOrientation: "Orientation",
TypeVector: "Vector",
}
type Document struct {
FileType string `json:"file_type"`
FileVersion string `json:"file_version"`
Root Struct `json:"root"`
}
type Struct struct {
Type uint32 `json:"struct_type"`
Fields []Field `json:"fields"`
}
type Field struct {
Label string `json:"label"`
Type FieldType `json:"-"`
Value Value `json:"-"`
}
type Value interface {
fieldType() FieldType
}
type LocString struct {
StringRef uint32 `json:"string_ref"`
Entries []LocStringEntry `json:"entries"`
}
type LocStringEntry struct {
ID uint32 `json:"id"`
Value string `json:"value"`
}
type Vector struct {
X float32 `json:"x"`
Y float32 `json:"y"`
Z float32 `json:"z"`
}
type Orientation struct {
X float32 `json:"x"`
Y float32 `json:"y"`
Z float32 `json:"z"`
W float32 `json:"w"`
}
type ByteValue uint8
type CharValue int8
type WordValue uint16
type ShortValue int16
type DWordValue uint32
type IntValue int32
type DWord64Value uint64
type Int64Value int64
type FloatValue float32
type DoubleValue float64
type StringValue string
type ResRefValue string
type VoidValue []byte
type ListValue []Struct
func (ByteValue) fieldType() FieldType { return TypeByte }
func (CharValue) fieldType() FieldType { return TypeChar }
func (WordValue) fieldType() FieldType { return TypeWord }
func (ShortValue) fieldType() FieldType { return TypeShort }
func (DWordValue) fieldType() FieldType { return TypeDWord }
func (IntValue) fieldType() FieldType { return TypeInt }
func (DWord64Value) fieldType() FieldType { return TypeDWord64 }
func (Int64Value) fieldType() FieldType { return TypeInt64 }
func (FloatValue) fieldType() FieldType { return TypeFloat }
func (DoubleValue) fieldType() FieldType { return TypeDouble }
func (StringValue) fieldType() FieldType { return TypeCExoString }
func (ResRefValue) fieldType() FieldType { return TypeResRef }
func (LocString) fieldType() FieldType { return TypeCExoLocString }
func (VoidValue) fieldType() FieldType { return TypeVoid }
func (Struct) fieldType() FieldType { return TypeStruct }
func (ListValue) fieldType() FieldType { return TypeList }
func (Orientation) fieldType() FieldType { return TypeOrientation }
func (Vector) fieldType() FieldType { return TypeVector }
func (t FieldType) String() string {
if name, ok := fieldTypeNames[t]; ok {
return name
}
return fmt.Sprintf("FieldType(%d)", t)
}
func NewField(label string, value Value) Field {
return Field{
Label: label,
Type: value.fieldType(),
Value: value,
}
}
internal/music/config.go
+47
View File
@@ -0,0 +1,47 @@
package music
import (
"fmt"
"path/filepath"
"strings"
)
type DatasetConfig struct {
Source string `json:"source" yaml:"source"`
Prefix string `json:"prefix,omitempty" yaml:"prefix,omitempty"`
StageRoot string `json:"stage_root,omitempty" yaml:"stage_root,omitempty"`
CreditsRoot string `json:"credits_root,omitempty" yaml:"credits_root,omitempty"`
}
type DefaultsConfig struct {
StageRoot string `json:"stage_root,omitempty" yaml:"stage_root,omitempty"`
CreditsRoot string `json:"credits_root,omitempty" yaml:"credits_root,omitempty"`
CreditsOverlay string `json:"credits_overlay,omitempty" yaml:"credits_overlay,omitempty"`
MaxStemLength *int `json:"max_stem_length,omitempty" yaml:"max_stem_length,omitempty"`
}
func ValidateDatasetSource(field, source string) error {
trimmed := strings.TrimSpace(source)
if trimmed == "" {
return fmt.Errorf("%s.source is required", field)
}
if strings.Contains(trimmed, "\x00") {
return fmt.Errorf("%s.source must not contain NUL bytes", field)
}
clean := filepath.Clean(filepath.FromSlash(trimmed))
if clean == ".." || strings.HasPrefix(clean, ".."+string(filepath.Separator)) {
return fmt.Errorf("%s.source must not escape project root", field)
}
return nil
}
func ValidateDatasetID(id string) error {
trimmed := strings.TrimSpace(id)
if trimmed == "" {
return fmt.Errorf("dataset id must not be empty")
}
if strings.Contains(trimmed, "/") || strings.Contains(trimmed, "\\") {
return fmt.Errorf("dataset id %q must not contain path separators", trimmed)
}
return nil
}
internal/music/credits.go
+292
View File
@@ -0,0 +1,292 @@
package music
import (
"fmt"
"os"
"path/filepath"
"strings"
)
func ParseCreditsOverlay(path string) (CreditsOverlay, error) {
entries, err := ParseCreditsMarkdown(path)
if err != nil {
return CreditsOverlay{}, err
}
overlay := CreditsOverlay{
ByOriginal: make(map[string]CreditsEntry),
ByOutput: make(map[string]CreditsEntry),
Entries: entries,
}
for _, entry := range entries {
if entry.OriginalFile != "" {
key := strings.ToLower(strings.TrimSpace(entry.OriginalFile))
if _, exists := overlay.ByOriginal[key]; exists {
return CreditsOverlay{}, fmt.Errorf("duplicate manual credits row for original file %s", entry.OriginalFile)
}
overlay.ByOriginal[key] = entry
}
if entry.OutputFile != "" {
key := strings.ToLower(strings.TrimSpace(entry.OutputFile))
if _, exists := overlay.ByOutput[key]; exists {
return CreditsOverlay{}, fmt.Errorf("duplicate manual credits row for output file %s", entry.OutputFile)
}
overlay.ByOutput[key] = entry
}
}
return overlay, nil
}
func (o CreditsOverlay) Match(originalFile, outputFile string) *CreditsEntry {
if entry, ok := o.ByOutput[strings.ToLower(strings.TrimSpace(outputFile))]; ok {
entryCopy := entry
return &entryCopy
}
if entry, ok := o.ByOriginal[strings.ToLower(strings.TrimSpace(originalFile))]; ok {
entryCopy := entry
return &entryCopy
}
return nil
}
func ValidateOverlayEntries(dir string, overlay CreditsOverlay, generated []CreditsEntry) error {
if len(overlay.Entries) == 0 {
return nil
}
validOriginal := make(map[string]struct{}, len(generated))
validOutput := make(map[string]struct{}, len(generated))
for _, entry := range generated {
validOriginal[strings.ToLower(entry.OriginalFile)] = struct{}{}
validOutput[strings.ToLower(entry.OutputFile)] = struct{}{}
}
for _, entry := range overlay.Entries {
if entry.OriginalFile != "" {
if _, ok := validOriginal[strings.ToLower(entry.OriginalFile)]; !ok {
return fmt.Errorf("manual credits entry in %s references unknown original file %s", dir, entry.OriginalFile)
}
}
if entry.OutputFile != "" {
if _, ok := validOutput[strings.ToLower(entry.OutputFile)]; !ok {
return fmt.Errorf("manual credits entry in %s references unknown output file %s", dir, entry.OutputFile)
}
}
}
return nil
}
func ParseCreditsMarkdown(path string) ([]CreditsEntry, error) {
raw, err := os.ReadFile(path)
if err != nil {
if os.IsNotExist(err) {
return nil, nil
}
return nil, err
}
lines := strings.Split(strings.ReplaceAll(string(raw), "\r\n", "\n"), "\n")
header := []string(nil)
entries := make([]CreditsEntry, 0)
for _, line := range lines {
line = strings.TrimSpace(line)
if !strings.HasPrefix(line, "|") {
continue
}
cells := ParseMarkdownTableRow(line)
if len(cells) == 0 {
continue
}
if header == nil {
header = make([]string, len(cells))
for i, cell := range cells {
header[i] = NormalizeCreditsHeader(cell)
}
continue
}
if IsMarkdownSeparatorRow(cells) {
continue
}
entry := CreditsEntry{}
for i, key := range header {
if i >= len(cells) {
continue
}
value := CleanCreditsCell(cells[i])
switch key {
case "artist":
entry.Artist = value
case "title":
entry.Title = value
case "output_file":
entry.OutputFile = value
case "original_file":
entry.OriginalFile = value
case "album":
entry.Album = value
case "date":
entry.Date = value
case "rights":
entry.Rights = value
case "notes":
entry.Notes = value
}
}
if entry.Artist == "" && entry.Title == "" && entry.OutputFile == "" && entry.OriginalFile == "" {
continue
}
entries = append(entries, entry)
}
return entries, nil
}
func ParseMarkdownTableRow(line string) []string {
trimmed := strings.TrimSpace(line)
trimmed = strings.TrimPrefix(trimmed, "|")
trimmed = strings.TrimSuffix(trimmed, "|")
parts := strings.Split(trimmed, "|")
cells := make([]string, 0, len(parts))
for _, part := range parts {
cells = append(cells, strings.TrimSpace(strings.ReplaceAll(part, `\|`, "|")))
}
return cells
}
func IsMarkdownSeparatorRow(cells []string) bool {
if len(cells) == 0 {
return false
}
for _, cell := range cells {
cell = strings.TrimSpace(cell)
cell = strings.ReplaceAll(cell, "-", "")
cell = strings.ReplaceAll(cell, ":", "")
if cell != "" {
return false
}
}
return true
}
func NormalizeCreditsHeader(value string) string {
switch strings.ToLower(CleanCreditsCell(value)) {
case "artist":
return "artist"
case "title":
return "title"
case "output file":
return "output_file"
case "original file":
return "original_file"
case "album":
return "album"
case "date":
return "date"
case "license / copyright":
return "rights"
case "notes":
return "notes"
default:
return ""
}
}
func CleanCreditsCell(value string) string {
value = strings.TrimSpace(value)
value = strings.Trim(value, "`")
value = strings.ReplaceAll(value, "<br>", "\n")
return strings.TrimSpace(value)
}
func RenderCreditsMarkdown(entries []CreditsEntry) string {
var builder strings.Builder
builder.WriteString(CreditsHeader)
for _, entry := range entries {
builder.WriteString("| ")
builder.WriteString(EscapeMarkdownCell(entry.Artist))
builder.WriteString(" | ")
builder.WriteString(EscapeMarkdownCell(entry.Title))
builder.WriteString(" | ")
builder.WriteString(EscapeMarkdownCell(WrapCodeCell(entry.OutputFile)))
builder.WriteString(" | ")
builder.WriteString(EscapeMarkdownCell(WrapCodeCell(entry.OriginalFile)))
builder.WriteString(" | ")
builder.WriteString(EscapeMarkdownCell(entry.Album))
builder.WriteString(" | ")
builder.WriteString(EscapeMarkdownCell(entry.Date))
builder.WriteString(" | ")
builder.WriteString(EscapeMarkdownCell(entry.Rights))
builder.WriteString(" | ")
builder.WriteString(EscapeMarkdownCell(entry.Notes))
builder.WriteString(" |\n")
}
return builder.String()
}
func WrapCodeCell(value string) string {
if strings.TrimSpace(value) == "" {
return ""
}
return "`" + value + "`"
}
func EscapeMarkdownCell(value string) string {
value = strings.ReplaceAll(value, `\`, `\\`)
value = strings.ReplaceAll(value, "|", `\|`)
value = strings.ReplaceAll(value, "\n", "<br>")
return value
}
func ApplyCreditsOverlay(entry *CreditsEntry, overlay *CreditsEntry) {
if overlay == nil {
return
}
entry.Artist = FirstNonEmpty(overlay.Artist, entry.Artist)
entry.Title = FirstNonEmpty(overlay.Title, entry.Title)
entry.Album = FirstNonEmpty(overlay.Album, entry.Album)
entry.Date = FirstNonEmpty(overlay.Date, entry.Date)
entry.Rights = FirstNonEmpty(overlay.Rights, entry.Rights)
entry.Notes = FirstNonEmpty(overlay.Notes, entry.Notes)
}
func SyncGeneratedCreditsArtifacts(root string, desiredFiles map[string][]byte) (int, error) {
changedFiles := 0
existingFiles := make(map[string]struct{})
if err := filepath.WalkDir(root, func(path string, d os.DirEntry, walkErr error) error {
if walkErr != nil {
if os.IsNotExist(walkErr) {
return nil
}
return walkErr
}
if d.IsDir() {
return nil
}
existingFiles[path] = struct{}{}
return nil
}); err != nil {
return 0, fmt.Errorf("scan credits dir: %w", err)
}
for path, content := range desiredFiles {
current, err := os.ReadFile(path)
if err != nil && !os.IsNotExist(err) {
return 0, fmt.Errorf("read credits artifact %s: %w", path, err)
}
if err == nil && string(current) == string(content) {
delete(existingFiles, path)
continue
}
if err := os.MkdirAll(filepath.Dir(path), 0o755); err != nil {
return 0, fmt.Errorf("create generated credits dir: %w", err)
}
if err := os.WriteFile(path, content, 0o644); err != nil {
return 0, fmt.Errorf("write credits artifact %s: %w", path, err)
}
changedFiles++
delete(existingFiles, path)
}
for path := range existingFiles {
if err := os.Remove(path); err != nil && !os.IsNotExist(err) {
return 0, fmt.Errorf("remove stale credits artifact %s: %w", path, err)
}
changedFiles++
}
return changedFiles, nil
}
internal/music/metadata.go
+62
View File
@@ -0,0 +1,62 @@
package music
import (
"encoding/json"
"fmt"
"os/exec"
"strings"
)
func ProbeMetadata(ffprobePath, sourcePath string) (Metadata, error) {
cmd := exec.Command(ffprobePath, "-v", "quiet", "-print_format", "json", "-show_format", sourcePath)
output, err := cmd.Output()
if err != nil {
return Metadata{}, err
}
var payload struct {
Format struct {
Tags map[string]string `json:"tags"`
} `json:"format"`
}
if err := json.Unmarshal(output, &payload); err != nil {
return Metadata{}, err
}
tags := make(map[string]string, len(payload.Format.Tags))
for key, value := range payload.Format.Tags {
tags[strings.ToLower(strings.TrimSpace(key))] = strings.TrimSpace(value)
}
return Metadata{
Title: FirstNonEmpty(tags["title"]),
Artist: FirstNonEmpty(tags["artist"], tags["album_artist"], tags["composer"]),
Album: FirstNonEmpty(tags["album"]),
Date: FirstNonEmpty(tags["date"], tags["year"]),
Comment: FirstNonEmpty(tags["comment"], tags["description"]),
Copyright: FirstNonEmpty(tags["copyright"]),
License: FirstNonEmpty(tags["license"], tags["license_url"]),
}, nil
}
func ConvertSource(ffmpegPath, sourcePath, outputPath string) error {
cmd := exec.Command(
ffmpegPath, "-y",
"-i", sourcePath,
"-vn",
"-map_metadata", "-1",
"-ar", "44100",
"-ac", "2",
"-b:a", "192k",
"-codec:a", "libmp3lame",
"-write_xing", "0",
"-f", "mp3",
outputPath,
)
output, err := cmd.CombinedOutput()
if err != nil {
message := strings.TrimSpace(string(output))
if message == "" {
message = err.Error()
}
return fmt.Errorf("%s", message)
}
return nil
}
internal/music/music.go
+226
View File
@@ -0,0 +1,226 @@
package music
import (
"os"
"os/exec"
"path/filepath"
"regexp"
"sort"
"strings"
)
const (
MaxStemLen = 16
CreditsHeader = "# NWN Music Pack Credits\n\nOriginal rights remain with the respective composers and licensors.\n\n## Processed Files\n\n| Artist | Title | Output File | Original File | Album | Date | License / Copyright | Notes |\n|---|---|---|---|---|---|---|---|\n"
)
var (
defaultConvertExtensions = []string{
".flac",
".m4a",
".mp3",
".ogg",
".wav",
}
ConvertExtensions = map[string]struct{}{
".flac": {},
".m4a": {},
".mp3": {},
".ogg": {},
".wav": {},
}
PassthroughExtensions = map[string]struct{}{
".bmu": {},
".wav": {},
}
DropWords = map[string]struct{}{
"mp3": {}, "wav": {}, "flac": {}, "ogg": {}, "m4a": {},
"music": {}, "track": {}, "audio": {}, "song": {}, "soundtrack": {},
"final": {}, "version": {}, "ver": {}, "v": {}, "mix": {}, "master": {}, "remaster": {},
"free": {}, "royalty": {}, "copyright": {}, "background": {}, "bgm": {},
"loop": {}, "loops": {}, "loopable": {}, "seamless": {},
"official": {}, "download": {}, "preview": {},
"the": {}, "a": {}, "an": {}, "and": {}, "of": {}, "to": {}, "in": {}, "for": {}, "with": {}, "by": {},
}
BracketRE = regexp.MustCompile(`\[[^\]]*\]|\([^)]*\)|\{[^}]*\}`)
SplitRE = regexp.MustCompile(`[^A-Za-z0-9]+`)
YearRE = regexp.MustCompile(`^\d{4,}$`)
VowelRE = regexp.MustCompile(`[aeiou]`)
)
func DefaultConvertExtensions() []string {
return append([]string(nil), defaultConvertExtensions...)
}
type Metadata struct {
Title string
Artist string
Album string
Date string
Rights string
Notes string
Comment string
Copyright string
License string
}
type CreditsEntry struct {
Artist string `json:"artist"`
Title string `json:"title"`
OutputFile string `json:"output_file"`
OriginalFile string `json:"original_file"`
Album string `json:"album"`
Date string `json:"date"`
Rights string `json:"rights"`
Notes string `json:"notes"`
}
type CreditsSource struct {
Path string `json:"path"`
Kind string `json:"kind"`
Entries []CreditsEntry `json:"entries"`
}
type CreditsInventory struct {
Sources []CreditsSource `json:"sources"`
}
type CreditsOverlay struct {
ByOriginal map[string]CreditsEntry
ByOutput map[string]CreditsEntry
Entries []CreditsEntry
}
func ResolveFFmpeg() (string, error) {
return ResolveFFmpegWithConfig("")
}
func ResolveFFmpegWithConfig(configuredPath string) (string, error) {
if configured := strings.TrimSpace(configuredPath); configured != "" {
return configured, nil
}
if configured := strings.TrimSpace(os.Getenv("SOW_FFMPEG")); configured != "" {
return configured, nil
}
path, err := exec.LookPath("ffmpeg")
if err != nil {
return "", err
}
return path, nil
}
func ResolveFFprobe() (string, error) {
return ResolveFFprobeWithConfig("")
}
func ResolveFFprobeWithConfig(configuredPath string) (string, error) {
if configured := strings.TrimSpace(configuredPath); configured != "" {
return configured, nil
}
if configured := strings.TrimSpace(os.Getenv("SOW_FFPROBE")); configured != "" {
return configured, nil
}
path, err := exec.LookPath("ffprobe")
if err != nil {
return "", err
}
return path, nil
}
func IsMusicAssetPath(rel string) bool {
rel = filepath.ToSlash(strings.TrimSpace(rel))
return rel == "envi/music" || strings.HasPrefix(rel, "envi/music/")
}
func PathIsUnder(root, rel string) bool {
root = TrimSlashes(filepath.ToSlash(root))
rel = TrimSlashes(filepath.ToSlash(rel))
if root == "" || rel == "" {
return false
}
return rel == root || strings.HasPrefix(rel, root+"/")
}
func PrefixForDir(prefixes map[string]string, dir string) string {
dir = TrimSlashes(filepath.ToSlash(dir))
bestPrefix := ""
bestLen := -1
keys := make([]string, 0, len(prefixes))
for key := range prefixes {
keys = append(keys, key)
}
sort.Strings(keys)
for _, key := range keys {
value := prefixes[key]
normalizedKey := TrimSlashes(filepath.ToSlash(key))
if normalizedKey == "" {
continue
}
if dir != normalizedKey && !strings.HasPrefix(dir, normalizedKey+"/") {
continue
}
if len(normalizedKey) > bestLen {
bestLen = len(normalizedKey)
bestPrefix = SanitizePrefix(value)
}
}
return bestPrefix
}
func SanitizePrefix(prefix string) string {
prefix = strings.ToLower(strings.TrimSpace(prefix))
var builder strings.Builder
for _, r := range prefix {
switch {
case r >= 'a' && r <= 'z':
builder.WriteRune(r)
case r >= '0' && r <= '9':
builder.WriteRune(r)
case r == '_':
builder.WriteRune(r)
}
}
result := builder.String()
if strings.TrimSpace(prefix) != "" && strings.HasSuffix(strings.TrimSpace(prefix), "_") && !strings.HasSuffix(result, "_") {
result += "_"
}
return result
}
func TrimSlashes(value string) string {
return strings.Trim(strings.TrimSpace(value), "/")
}
func Min(a, b int) int {
if a < b {
return a
}
return b
}
func Max(a, b int) int {
if a > b {
return a
}
return b
}
func FirstNonEmpty(values ...string) string {
for _, value := range values {
if strings.TrimSpace(value) != "" {
return strings.TrimSpace(value)
}
}
return ""
}
func JoinNonEmpty(sep string, values ...string) string {
parts := make([]string, 0, len(values))
for _, value := range values {
value = strings.TrimSpace(value)
if value != "" {
parts = append(parts, value)
}
}
return strings.Join(parts, sep)
}
internal/music/music_test.go
+404
View File
@@ -0,0 +1,404 @@
package music
import (
"os"
"path/filepath"
"strings"
"testing"
)
func TestUniqueNameHandlesShortStemCollisions(t *testing.T) {
used := map[string]struct{}{
"mus_wg_mystc": {},
}
got := UniqueName("mus_wg_mystc", used)
if got != "mus_wg_mystc_1" {
t.Fatalf("unexpected collision result: got %q want %q", got, "mus_wg_mystc_1")
}
}
func TestUniqueNameNoCollision(t *testing.T) {
used := map[string]struct{}{}
got := UniqueName("new_stem", used)
if got != "new_stem" {
t.Fatalf("expected 'new_stem', got %q", got)
}
}
func TestUniqueNameMultipleCollisions(t *testing.T) {
used := map[string]struct{}{
"stem": {},
"stem_1": {},
"stem_2": {},
}
got := UniqueName("stem", used)
if got != "stem_3" {
t.Fatalf("expected 'stem_3', got %q", got)
}
}
func TestGenerateStem(t *testing.T) {
used := map[string]struct{}{
"existing": {},
}
stem, err := GenerateStem("My Cool Track.mp3", "mus_", used)
if err != nil {
t.Fatalf("unexpected error: %v", err)
}
if stem == "" {
t.Fatal("expected non-empty stem")
}
if len(stem) > MaxStemLen {
t.Fatalf("stem %q exceeds max length %d", stem, MaxStemLen)
}
if _, exists := used[stem]; !exists {
t.Fatal("expected stem to be registered in used set")
}
}
func TestGenerateStemPrefixTooLong(t *testing.T) {
_, err := GenerateStem("test.mp3", "this_prefix_is_way_too_long_for_stem_", nil)
if err == nil {
t.Fatal("expected error for too-long prefix")
}
}
func TestSlugWords(t *testing.T) {
words := SlugWords("My Cool Music Track (Official) [HD]")
if len(words) == 0 {
t.Fatal("expected non-empty words")
}
}
func TestSlugWordsStripsBrackets(t *testing.T) {
words := SlugWords("Song [Explicit] (Remix)")
for _, w := range words {
if w == "explicit" || w == "remix" {
t.Fatalf("bracket content should be removed, got word %q", w)
}
}
}
func TestSanitizePrefix(t *testing.T) {
if got := SanitizePrefix("Mus_WG_"); got != "mus_wg_" {
t.Fatalf("unexpected prefix: %q", got)
}
if got := SanitizePrefix(" Test "); got != "test" {
t.Fatalf("unexpected prefix: %q", got)
}
if got := SanitizePrefix(""); got != "" {
t.Fatalf("expected empty prefix, got %q", got)
}
}
func TestReserveName(t *testing.T) {
used := map[string]struct{}{}
if err := ReserveName("testname", used); err != nil {
t.Fatalf("unexpected error: %v", err)
}
if _, exists := used["testname"]; !exists {
t.Fatal("expected name to be reserved")
}
if err := ReserveName("testname", used); err == nil {
t.Fatal("expected error for duplicate name")
}
}
func TestValidateManualOutputFile(t *testing.T) {
if err := ValidateManualOutputFile("test.bmu"); err != nil {
t.Fatalf("unexpected error: %v", err)
}
if err := ValidateManualOutputFile("test.txt"); err == nil {
t.Fatal("expected error for non-bmu extension")
}
if err := ValidateManualOutputFile(".bmu"); err == nil {
t.Fatal("expected error for empty stem")
}
}
func TestParseCreditsMarkdownMissingFile(t *testing.T) {
entries, err := ParseCreditsMarkdown("nonexistent.md")
if err != nil {
t.Fatalf("unexpected error for missing file: %v", err)
}
if entries != nil {
t.Fatalf("expected nil entries for missing file, got %v", entries)
}
}
func TestParseCreditsMarkdownRealContent(t *testing.T) {
content := "# Header\n\n| Artist | Title | Output File | Original File | Album | Date | License / Copyright | Notes |\n|---|---|---|---|---|---|---|---|\n| Test Artist | Test Title | `output.bmu` | `original.mp3` | Test Album | 2024 | MIT | test |\n"
path := filepath.Join(t.TempDir(), "CREDITS.md")
if err := os.WriteFile(path, []byte(content), 0o644); err != nil {
t.Fatalf("write test credits: %v", err)
}
entries, err := ParseCreditsMarkdown(path)
if err != nil {
t.Fatalf("unexpected error: %v", err)
}
if len(entries) != 1 {
t.Fatalf("expected 1 entry, got %d", len(entries))
}
if entries[0].Artist != "Test Artist" {
t.Fatalf("expected 'Test Artist', got %q", entries[0].Artist)
}
if entries[0].Title != "Test Title" {
t.Fatalf("expected 'Test Title', got %q", entries[0].Title)
}
if entries[0].OutputFile != "output.bmu" {
t.Fatalf("expected 'output.bmu', got %q", entries[0].OutputFile)
}
}
func TestRenderCreditsMarkdown(t *testing.T) {
entries := []CreditsEntry{
{
Artist: "Test Artist",
Title: "Test Title",
OutputFile: "output.bmu",
OriginalFile: "original.mp3",
Album: "Test Album",
Date: "2024",
Rights: "MIT",
Notes: "test note",
},
}
result := RenderCreditsMarkdown(entries)
if !strings.Contains(result, "Test Artist") {
t.Fatal("expected rendered output to contain artist")
}
if !strings.Contains(result, "output.bmu") {
t.Fatal("expected rendered output to contain output file")
}
}
func TestParseCreditsOverlay(t *testing.T) {
content := "# Header\n\n| Artist | Title | Output File | Original File |\n|---|---|---|---|\n| Overlay Artist | Ovr Title | `override.bmu` | `original.mp3` |\n"
path := filepath.Join(t.TempDir(), "CREDITS.md")
if err := os.WriteFile(path, []byte(content), 0o644); err != nil {
t.Fatalf("write test overlay: %v", err)
}
overlay, err := ParseCreditsOverlay(path)
if err != nil {
t.Fatalf("unexpected error: %v", err)
}
if len(overlay.Entries) != 1 {
t.Fatalf("expected 1 overlay entry, got %d", len(overlay.Entries))
}
matched := overlay.Match("original.mp3", "")
if matched == nil {
t.Fatal("expected overlay match by original file")
}
if matched.Artist != "Overlay Artist" {
t.Fatalf("expected 'Overlay Artist', got %q", matched.Artist)
}
}
func TestApplyCreditsOverlay(t *testing.T) {
entry := &CreditsEntry{
Artist: "Original Artist",
Title: "Original Title",
}
overlay := &CreditsEntry{
Artist: "Overlay Artist",
}
ApplyCreditsOverlay(entry, overlay)
if entry.Artist != "Overlay Artist" {
t.Fatalf("expected 'Overlay Artist', got %q", entry.Artist)
}
if entry.Title != "Original Title" {
t.Fatalf("expected 'Original Title' preserved, got %q", entry.Title)
}
ApplyCreditsOverlay(entry, nil)
if entry.Artist != "Overlay Artist" {
t.Fatalf("expected nil overlay to be no-op, got %q", entry.Artist)
}
}
func TestValidateOverlayEntries(t *testing.T) {
generated := []CreditsEntry{
{OriginalFile: "track1.mp3", OutputFile: "track1.bmu"},
}
overlay := CreditsOverlay{
Entries: []CreditsEntry{
{OriginalFile: "track1.mp3"},
},
}
if err := ValidateOverlayEntries("test", overlay, generated); err != nil {
t.Fatalf("unexpected validation error: %v", err)
}
badOverlay := CreditsOverlay{
Entries: []CreditsEntry{
{OriginalFile: "nonexistent.mp3"},
},
}
if err := ValidateOverlayEntries("test", badOverlay, generated); err == nil {
t.Fatal("expected validation error for unknown original file")
}
}
func TestIsMusicAssetPath(t *testing.T) {
if !IsMusicAssetPath("envi/music") {
t.Fatal("expected 'envi/music' to be music path")
}
if !IsMusicAssetPath("envi/music/westgate") {
t.Fatal("expected 'envi/music/westgate' to be music path")
}
if IsMusicAssetPath("envi/textures") {
t.Fatal("expected 'envi/textures' not to be music path")
}
if IsMusicAssetPath("") {
t.Fatal("expected empty string not to be music path")
}
}
func TestFirstNonEmpty(t *testing.T) {
if got := FirstNonEmpty("", "hello", "world"); got != "hello" {
t.Fatalf("expected 'hello', got %q", got)
}
if got := FirstNonEmpty("", "", ""); got != "" {
t.Fatalf("expected empty, got %q", got)
}
if got := FirstNonEmpty("first"); got != "first" {
t.Fatalf("expected 'first', got %q", got)
}
}
func TestJoinNonEmpty(t *testing.T) {
if got := JoinNonEmpty(", ", "a", "", "b"); got != "a, b" {
t.Fatalf("expected 'a, b', got %q", got)
}
if got := JoinNonEmpty(", "); got != "" {
t.Fatalf("expected empty, got %q", got)
}
}
func TestTrimSlashes(t *testing.T) {
if got := TrimSlashes("/foo/bar/"); got != "foo/bar" {
t.Fatalf("expected 'foo/bar', got %q", got)
}
if got := TrimSlashes("///"); got != "" {
t.Fatalf("expected empty, got %q", got)
}
}
func TestEscapeMarkdownCell(t *testing.T) {
result := EscapeMarkdownCell("a|b\nc")
if !strings.Contains(result, `\|`) {
t.Fatal("expected pipe to be escaped")
}
if !strings.Contains(result, "<br>") {
t.Fatal("expected newline to be replaced with <br>")
}
}
func TestWrapCodeCell(t *testing.T) {
if got := WrapCodeCell("test.bmu"); got != "`test.bmu`" {
t.Fatalf("expected '`test.bmu`', got %q", got)
}
if got := WrapCodeCell(""); got != "" {
t.Fatalf("expected empty string, got %q", got)
}
}
func TestSyncGeneratedCreditsArtifactsWritesNewFiles(t *testing.T) {
root := t.TempDir()
desired := map[string][]byte{
filepath.Join(root, "test.md"): []byte("content"),
}
changed, err := SyncGeneratedCreditsArtifacts(root, desired)
if err != nil {
t.Fatalf("unexpected error: %v", err)
}
if changed != 1 {
t.Fatalf("expected 1 changed file, got %d", changed)
}
}
func TestSyncGeneratedCreditsArtifactsNoChanges(t *testing.T) {
root := t.TempDir()
path := filepath.Join(root, "test.md")
if err := os.WriteFile(path, []byte("content"), 0o644); err != nil {
t.Fatalf("write file: %v", err)
}
desired := map[string][]byte{
path: []byte("content"),
}
changed, err := SyncGeneratedCreditsArtifacts(root, desired)
if err != nil {
t.Fatalf("unexpected error: %v", err)
}
if changed != 0 {
t.Fatalf("expected 0 changed files, got %d", changed)
}
}
func TestSyncGeneratedCreditsArtifactsRemovesStale(t *testing.T) {
root := t.TempDir()
stalePath := filepath.Join(root, "stale.md")
if err := os.WriteFile(stalePath, []byte("stale"), 0o644); err != nil {
t.Fatalf("write stale file: %v", err)
}
desired := map[string][]byte{}
changed, err := SyncGeneratedCreditsArtifacts(root, desired)
if err != nil {
t.Fatalf("unexpected error: %v", err)
}
if changed != 1 {
t.Fatalf("expected 1 removed file, got %d", changed)
}
}
func TestPrefixForDir(t *testing.T) {
prefixes := map[string]string{
"envi/music/westgate": "mus_wg_",
}
if got := PrefixForDir(prefixes, "envi/music/westgate"); got != "mus_wg_" {
t.Fatalf("expected 'mus_wg_', got %q", got)
}
if got := PrefixForDir(prefixes, "envi/music/other"); got != "" {
t.Fatalf("expected empty prefix, got %q", got)
}
if got := PrefixForDir(nil, "envi/music"); got != "" {
t.Fatalf("expected empty prefix for nil map, got %q", got)
}
}
func TestMaxMin(t *testing.T) {
if got := Min(3, 5); got != 3 {
t.Fatalf("Min(3,5) = %d, want 3", got)
}
if got := Max(3, 5); got != 5 {
t.Fatalf("Max(3,5) = %d, want 5", got)
}
}
func TestResolveFFmpegEnv(t *testing.T) {
t.Setenv("SOW_FFMPEG", "/custom/ffmpeg")
path, err := ResolveFFmpeg()
if err != nil {
t.Fatalf("unexpected error: %v", err)
}
if path != "/custom/ffmpeg" {
t.Fatalf("expected '/custom/ffmpeg', got %q", path)
}
}
func TestResolveFFprobeEnv(t *testing.T) {
t.Setenv("SOW_FFPROBE", "/custom/ffprobe")
path, err := ResolveFFprobe()
if err != nil {
t.Fatalf("unexpected error: %v", err)
}
if path != "/custom/ffprobe" {
t.Fatalf("expected '/custom/ffprobe', got %q", path)
}
}
internal/music/naming.go
+197
View File
@@ -0,0 +1,197 @@
package music
import (
"crypto/sha1"
"fmt"
"path/filepath"
"strings"
)
func GenerateStem(fileName, prefix string, used map[string]struct{}) (string, error) {
return GenerateStemWithMax(fileName, prefix, MaxStemLen, used)
}
func GenerateStemWithMax(fileName, prefix string, maxStemLen int, used map[string]struct{}) (string, error) {
if maxStemLen <= 0 {
maxStemLen = MaxStemLen
}
maxCore := maxStemLen - len(prefix)
if maxCore < 3 {
return "", fmt.Errorf("prefix too long: %q", prefix)
}
base := strings.TrimSuffix(fileName, filepath.Ext(fileName))
words := SlugWords(base)
core := MakeMusicCore(words, maxCore)
if core == "" {
sum := sha1.Sum([]byte(fileName))
core = fmt.Sprintf("%x", sum[:])[:maxCore]
}
stem := strings.TrimRight((prefix + core)[:Min(maxStemLen, len(prefix)+len(core))], "_")
if stem == "" {
return "", fmt.Errorf("could not derive music stem for %s", fileName)
}
return UniqueNameWithMax(stem, maxStemLen, used), nil
}
func SlugWords(value string) []string {
value = BracketRE.ReplaceAllString(value, " ")
value = strings.ReplaceAll(value, "&", " and ")
value = strings.ReplaceAll(value, "'", "")
value = strings.ReplaceAll(value, "\u2019", "")
value = strings.ToLower(value)
var ascii strings.Builder
for _, r := range value {
switch {
case r >= 'a' && r <= 'z':
ascii.WriteRune(r)
case r >= '0' && r <= '9':
ascii.WriteRune(r)
default:
ascii.WriteRune(' ')
}
}
rawWords := SplitRE.Split(ascii.String(), -1)
words := make([]string, 0, len(rawWords))
for _, word := range rawWords {
if word == "" {
continue
}
if _, drop := DropWords[word]; drop {
continue
}
if YearRE.MatchString(word) {
continue
}
words = append(words, word)
}
return words
}
func MakeMusicCore(words []string, maxCore int) string {
core := ""
for _, word := range words {
candidate := word
if core != "" {
candidate = core + "_" + word
}
if len(candidate) <= maxCore {
core = candidate
continue
}
break
}
if core != "" {
return core
}
abbrev := make([]string, 0, Min(4, len(words)))
for _, word := range words {
if len(abbrev) == 4 {
break
}
abbrev = append(abbrev, AbbreviateWord(word, 5))
}
for _, word := range abbrev {
candidate := word
if core != "" {
candidate = core + "_" + word
}
if len(candidate) <= maxCore {
core = candidate
continue
}
break
}
if core != "" {
return core
}
if len(words) > 0 {
return strings.Trim(words[0][:Min(maxCore, len(words[0]))], "_")
}
return ""
}
func AbbreviateWord(word string, maxLen int) string {
if len(word) <= maxLen {
return word
}
consonants := VowelRE.ReplaceAllString(word, "")
candidate := word[:1]
if len(consonants) > 1 {
candidate += consonants[1:]
}
if len(candidate) >= 3 {
return candidate[:Min(maxLen, len(candidate))]
}
return word[:Min(maxLen, len(word))]
}
func UniqueName(stem string, used map[string]struct{}) string {
return UniqueNameWithMax(stem, MaxStemLen, used)
}
func UniqueNameWithMax(stem string, maxStemLen int, used map[string]struct{}) string {
if maxStemLen <= 0 {
maxStemLen = MaxStemLen
}
if _, exists := used[stem]; !exists {
used[stem] = struct{}{}
return stem
}
for index := 1; ; index++ {
suffix := fmt.Sprintf("_%d", index)
prefixLen := Min(len(stem), Max(0, maxStemLen-len(suffix)))
candidate := strings.TrimRight(stem[:prefixLen], "_") + suffix
if _, exists := used[candidate]; exists {
continue
}
used[candidate] = struct{}{}
return candidate
}
}
func ReserveName(stem string, used map[string]struct{}) error {
stem = strings.ToLower(strings.TrimSpace(stem))
if stem == "" {
return fmt.Errorf("output filename is empty")
}
if _, exists := used[stem]; exists {
return fmt.Errorf("output filename %s collides with an existing asset", stem)
}
used[stem] = struct{}{}
return nil
}
func ValidateManualOutputFile(name string) error {
return ValidateManualOutputFileWithRules(name, ".bmu", MaxStemLen)
}
func ValidateManualOutputFileWithRules(name, extension string, maxStemLen int) error {
name = strings.ToLower(strings.TrimSpace(name))
extension = strings.ToLower(strings.TrimSpace(extension))
if extension == "" {
extension = ".bmu"
}
if maxStemLen <= 0 {
maxStemLen = MaxStemLen
}
if !strings.HasSuffix(name, extension) {
return fmt.Errorf("output file must end with %s", extension)
}
stem := strings.TrimSuffix(name, extension)
if stem == "" {
return fmt.Errorf("output file stem is empty")
}
if len(stem) > maxStemLen {
return fmt.Errorf("output file stem %q exceeds %d characters", stem, maxStemLen)
}
for _, r := range stem {
switch {
case r >= 'a' && r <= 'z':
case r >= '0' && r <= '9':
case r == '_':
default:
return fmt.Errorf("output file stem %q contains unsupported character %q", stem, string(r))
}
}
return nil
}
internal/pipeline/apply_manifest.go
+70
View File
@@ -0,0 +1,70 @@
package pipeline
import (
"encoding/json"
"fmt"
"os"
"path/filepath"
"strings"
"git.westgate.pw/ShadowsOverWestgate/sow-tools/internal/gff"
"git.westgate.pw/ShadowsOverWestgate/sow-tools/internal/project"
)
type ApplyManifestResult struct {
ManifestPath string
ModuleSource string
HAKCount int
}
func ApplyHAKManifest(p *project.Project, manifestPath string) (ApplyManifestResult, error) {
if manifestPath == "" {
manifestPath = p.HAKManifestPath()
}
if strings.TrimSpace(p.EffectiveConfig().Paths.Source) == "" {
return ApplyManifestResult{}, fmt.Errorf("cannot apply hak manifest: paths.source is not configured")
}
sourceRoot := filepath.Clean(p.SourceDir())
if sourceRoot == "." || sourceRoot == string(filepath.Separator) || sourceRoot == filepath.Clean(p.Root) {
return ApplyManifestResult{}, fmt.Errorf("cannot apply hak manifest: paths.source resolves to unsafe source root %s", sourceRoot)
}
raw, err := os.ReadFile(manifestPath)
if err != nil {
return ApplyManifestResult{}, fmt.Errorf("read hak manifest: %w", err)
}
var manifest BuildManifest
if err := json.Unmarshal(raw, &manifest); err != nil {
return ApplyManifestResult{}, fmt.Errorf("parse hak manifest: %w", err)
}
moduleSource := filepath.Join(sourceRoot, "module", "module.ifo.json")
sourceRaw, err := os.ReadFile(moduleSource)
if err != nil {
return ApplyManifestResult{}, fmt.Errorf("read module ifo source: %w", err)
}
var document gff.Document
if err := json.Unmarshal(sourceRaw, &document); err != nil {
return ApplyManifestResult{}, fmt.Errorf("parse module ifo source: %w", err)
}
setModuleHAKList(&document, manifest.ModuleHAKs)
formatted, err := json.MarshalIndent(document, "", " ")
if err != nil {
return ApplyManifestResult{}, fmt.Errorf("marshal updated module ifo: %w", err)
}
formatted = append(formatted, '\n')
if err := os.WriteFile(moduleSource, formatted, 0o644); err != nil {
return ApplyManifestResult{}, fmt.Errorf("write module ifo source: %w", err)
}
return ApplyManifestResult{
ManifestPath: manifestPath,
ModuleSource: moduleSource,
HAKCount: len(manifest.ModuleHAKs),
}, nil
}
internal/pipeline/build.go
+1627
View File
File diff suppressed because it is too large Load Diff
internal/pipeline/chunk_validation.go
+27
View File
@@ -0,0 +1,27 @@
package pipeline
import (
"fmt"
"git.westgate.pw/ShadowsOverWestgate/sow-tools/internal/erf"
)
func ensureUniqueChunkResources(chunk hakChunk) error {
seen := map[string]string{}
for _, asset := range chunk.Assets {
key := fmt.Sprintf("%s:%04x", asset.Resource.Name, asset.Resource.Type)
if previous, exists := seen[key]; exists {
return fmt.Errorf("resource %s from %s conflicts with %s inside generated hak %s", chunkResourceLabel(asset), asset.Rel, previous, chunk.Name)
}
seen[key] = asset.Rel
}
return nil
}
func chunkResourceLabel(asset assetResource) string {
extension, ok := erf.ExtensionForResourceType(asset.Resource.Type)
if !ok {
return asset.Resource.Name
}
return asset.Resource.Name + "." + extension
}
internal/pipeline/compare.go
+359
View File
@@ -0,0 +1,359 @@
package pipeline
import (
"bytes"
"encoding/json"
"errors"
"fmt"
"io/fs"
"os"
"path/filepath"
"slices"
"strings"
"time"
"git.westgate.pw/ShadowsOverWestgate/sow-tools/internal/erf"
"git.westgate.pw/ShadowsOverWestgate/sow-tools/internal/gff"
"git.westgate.pw/ShadowsOverWestgate/sow-tools/internal/project"
)
type CompareResult struct {
ModulePath string
HAKPaths []string
Checked int
}
type resourceExpectation struct {
Key string
Kind string
Bytes []byte
}
func Compare(p *project.Project) (CompareResult, error) {
modulePath := p.ModuleArchivePath()
var hakPaths []string
if len(p.Inventory.AssetFiles) > 0 {
var err error
hakPaths, err = manifestHAKPaths(p)
if err != nil {
return CompareResult{}, err
}
}
if err := ensureBuildIsCurrent(p, modulePath, hakPaths); err != nil {
return CompareResult{ModulePath: modulePath, HAKPaths: hakPaths}, err
}
moduleArchive, err := readArchive(modulePath)
if err != nil {
return CompareResult{}, err
}
expected, err := expectedResources(p)
if err != nil {
return CompareResult{}, err
}
actual := archiveIndex(moduleArchive)
for _, hakPath := range hakPaths {
hakArchive, err := readArchive(hakPath)
if err != nil {
return CompareResult{}, err
}
for key, value := range archiveIndex(hakArchive) {
actual[key] = value
}
}
var diagnostics []error
checked := 0
expectedKeys := make([]string, 0, len(expected))
for key := range expected {
expectedKeys = append(expectedKeys, key)
}
slices.Sort(expectedKeys)
for _, key := range expectedKeys {
want := expected[key]
got, ok := actual[key]
if !ok {
diagnostics = append(diagnostics, fmt.Errorf("missing resource in built archives: %s", key))
continue
}
if !bytes.Equal(want.Bytes, got.Data) {
diagnostics = append(diagnostics, fmt.Errorf("resource content mismatch: %s", key))
continue
}
checked++
delete(actual, key)
}
if len(actual) > 0 {
extraKeys := make([]string, 0, len(actual))
for key := range actual {
extraKeys = append(extraKeys, key)
}
slices.Sort(extraKeys)
for _, key := range extraKeys {
diagnostics = append(diagnostics, fmt.Errorf("unexpected extra resource in built archives: %s", key))
}
}
result := CompareResult{
ModulePath: modulePath,
HAKPaths: hakPaths,
Checked: checked,
}
if len(diagnostics) > 0 {
return result, errors.Join(diagnostics...)
}
return result, nil
}
func expectedResources(p *project.Project) (map[string]resourceExpectation, error) {
out := map[string]resourceExpectation{}
moduleHakOrder, err := plannedModuleHAKOrder(p)
if err != nil {
return nil, err
}
for _, rel := range p.Inventory.SourceFiles {
abs := filepath.Join(p.SourceDir(), filepath.FromSlash(rel))
name, extension, err := splitSourceName(abs)
if err != nil {
return nil, err
}
raw, err := os.ReadFile(abs)
if err != nil {
return nil, fmt.Errorf("read %s: %w", abs, err)
}
var document gff.Document
if err := json.Unmarshal(raw, &document); err != nil {
return nil, fmt.Errorf("parse gff json %s: %w", abs, err)
}
if extension == ".ifo" && strings.EqualFold(name, "module") && len(moduleHakOrder) > 0 {
setModuleHAKList(&document, moduleHakOrder)
}
canonical, err := json.Marshal(document)
if err != nil {
return nil, fmt.Errorf("marshal canonical gff json %s: %w", abs, err)
}
out[resourceKey(name, extension)] = resourceExpectation{
Key: resourceKey(name, extension),
Kind: "gff-json",
Bytes: canonical,
}
}
for _, rel := range p.Inventory.ScriptFiles {
abs := filepath.Join(p.SourceDir(), filepath.FromSlash(rel))
data, err := os.ReadFile(abs)
if err != nil {
return nil, fmt.Errorf("read %s: %w", abs, err)
}
name := strings.ToLower(strings.TrimSuffix(filepath.Base(abs), filepath.Ext(abs)))
extension := strings.TrimPrefix(strings.ToLower(filepath.Ext(abs)), ".")
out[resourceKey(name, extension)] = resourceExpectation{
Key: resourceKey(name, extension),
Kind: "raw",
Bytes: data,
}
}
for _, rel := range p.Inventory.AssetFiles {
abs := filepath.Join(p.AssetsDir(), filepath.FromSlash(rel))
data, err := os.ReadFile(abs)
if err != nil {
return nil, fmt.Errorf("read %s: %w", abs, err)
}
name := strings.ToLower(strings.TrimSuffix(filepath.Base(abs), filepath.Ext(abs)))
extension := strings.TrimPrefix(strings.ToLower(filepath.Ext(abs)), ".")
out[resourceKey(name, extension)] = resourceExpectation{
Key: resourceKey(name, extension),
Kind: "raw",
Bytes: data,
}
}
return out, nil
}
func ensureBuildIsCurrent(p *project.Project, modulePath string, hakPaths []string) error {
archivePaths := make([]string, 0, 1+len(hakPaths))
archivePaths = append(archivePaths, modulePath)
archivePaths = append(archivePaths, hakPaths...)
oldestArchive := time.Time{}
for _, path := range archivePaths {
info, err := os.Stat(path)
if err != nil {
return fmt.Errorf("stat archive %s: %w", path, err)
}
if oldestArchive.IsZero() || info.ModTime().Before(oldestArchive) {
oldestArchive = info.ModTime()
}
}
sourcePaths := make([]string, 0, len(p.Inventory.SourceFiles)+len(p.Inventory.ScriptFiles)+len(p.Inventory.AssetFiles))
for _, rel := range p.Inventory.SourceFiles {
sourcePaths = append(sourcePaths, filepath.Join(p.SourceDir(), filepath.FromSlash(rel)))
}
for _, rel := range p.Inventory.ScriptFiles {
sourcePaths = append(sourcePaths, filepath.Join(p.SourceDir(), filepath.FromSlash(rel)))
}
for _, rel := range p.Inventory.AssetFiles {
sourcePaths = append(sourcePaths, filepath.Join(p.AssetsDir(), filepath.FromSlash(rel)))
}
newestSource := time.Time{}
newestPath := ""
for _, path := range sourcePaths {
info, err := os.Stat(path)
if err != nil {
return fmt.Errorf("stat source %s: %w", path, err)
}
if newestSource.IsZero() || info.ModTime().After(newestSource) {
newestSource = info.ModTime()
newestPath = path
}
}
if !newestSource.IsZero() && newestSource.After(oldestArchive) {
return fmt.Errorf("built archives are older than source file %s; run build-module or build-haks before compare", newestPath)
}
return nil
}
func readArchive(path string) (erf.Archive, error) {
input, err := os.Open(path)
if err != nil {
return erf.Archive{}, fmt.Errorf("open archive %s: %w", path, err)
}
defer input.Close()
archive, err := erf.Read(input)
if err != nil {
return erf.Archive{}, fmt.Errorf("read archive %s: %w", path, err)
}
return archive, nil
}
func archiveIndex(archive erf.Archive) map[string]erf.Resource {
out := map[string]erf.Resource{}
for _, resource := range archive.Resources {
extension, ok := erf.ExtensionForResourceType(resource.Type)
if !ok {
continue
}
key := resourceKey(strings.ToLower(resource.Name), extension)
switch extension {
case "utc", "utd", "ute", "uti", "utm", "utp", "uts", "utt", "utw",
"are", "dlg", "fac", "gic", "git", "ifo", "itp", "jrl":
document, err := gff.Read(bytes.NewReader(resource.Data))
if err != nil {
out[key] = resource
continue
}
canonical, err := json.Marshal(document)
if err != nil {
out[key] = resource
continue
}
resource.Data = canonical
}
out[key] = resource
}
return out
}
func resourceKey(name, extension string) string {
return strings.ToLower(name) + "." + strings.TrimPrefix(strings.ToLower(extension), ".")
}
func manifestHAKPaths(p *project.Project) ([]string, error) {
manifestPath := p.HAKManifestPath()
raw, err := os.ReadFile(manifestPath)
if err == nil {
var manifest BuildManifest
if err := json.Unmarshal(raw, &manifest); err != nil {
return nil, fmt.Errorf("parse hak manifest: %w", err)
}
paths := make([]string, 0, len(manifest.HAKs))
for _, hak := range manifest.HAKs {
paths = append(paths, p.HAKArchivePath(hak.Name))
}
return paths, nil
}
if !errors.Is(err, os.ErrNotExist) {
return nil, fmt.Errorf("read hak manifest: %w", err)
}
legacy := filepath.Join(p.BuildDir(), "*.hak")
paths, err := filepath.Glob(legacy)
if err != nil {
return nil, fmt.Errorf("scan hak archives: %w", err)
}
filtered := make([]string, 0, len(paths))
for _, path := range paths {
if filepath.Base(path) == p.TopDataPackageHAKName() {
continue
}
filtered = append(filtered, path)
}
slices.Sort(filtered)
return filtered, nil
}
func topPackageSourcePaths(p *project.Project) ([]string, error) {
sourceDir := p.TopDataSourceDir()
generated2DADir := filepath.Join(sourceDir, "assets", "2da")
paths := make([]string, 0)
for _, candidate := range []string{
filepath.Join(sourceDir, ".tlk_state.json"),
filepath.Join(sourceDir, "base_dialog.json"),
} {
if _, err := os.Stat(candidate); err == nil {
paths = append(paths, candidate)
} else if err != nil && !errors.Is(err, os.ErrNotExist) {
return nil, fmt.Errorf("stat topdata source %s: %w", candidate, err)
}
}
for _, dir := range []string{
filepath.Join(sourceDir, "data"),
filepath.Join(sourceDir, "assets"),
} {
info, err := os.Stat(dir)
if err != nil {
if errors.Is(err, os.ErrNotExist) {
continue
}
return nil, fmt.Errorf("stat topdata path %s: %w", dir, err)
}
if !info.IsDir() {
continue
}
err = filepath.WalkDir(dir, func(path string, d fs.DirEntry, walkErr error) error {
if walkErr != nil {
return walkErr
}
if d.IsDir() && path == generated2DADir {
return filepath.SkipDir
}
if d.IsDir() {
return nil
}
paths = append(paths, path)
return nil
})
if err != nil {
return nil, fmt.Errorf("scan topdata path %s: %w", dir, err)
}
}
return paths, nil
}
internal/pipeline/extract.go
+621
View File
@@ -0,0 +1,621 @@
package pipeline
import (
"bytes"
"encoding/json"
"errors"
"fmt"
"os"
"path/filepath"
"slices"
"strings"
"git.westgate.pw/ShadowsOverWestgate/sow-tools/internal/erf"
"git.westgate.pw/ShadowsOverWestgate/sow-tools/internal/gff"
"git.westgate.pw/ShadowsOverWestgate/sow-tools/internal/project"
)
type ExtractResult struct {
ModulePath string
HAKPaths []string
DeletedArchivePaths []string
Written int
Overwritten int
Removed int
Skipped int
}
type extractionArchive struct {
Path string
Rel string
Extension string
}
type extractionScope struct {
Source bool
Assets bool
}
func Extract(p *project.Project, files ...string) (ExtractResult, error) {
var result ExtractResult
var failures []error
desired := map[string]struct{}{}
scope := extractionScope{}
archives, err := resolveExtractionArchives(p, files)
if err != nil {
return result, err
}
for _, archivePath := range archives {
input, err := os.Open(archivePath.Path)
if err != nil {
failures = append(failures, fmt.Errorf("open archive %s: %w", archivePath.Path, err))
continue
}
archive, err := erf.Read(input)
input.Close()
if err != nil {
failures = append(failures, fmt.Errorf("read archive %s: %w", archivePath.Path, err))
continue
}
switch archivePath.Extension {
case ".mod":
if result.ModulePath == "" {
result.ModulePath = archivePath.Path
}
case ".hak":
result.HAKPaths = append(result.HAKPaths, archivePath.Path)
}
written, overwritten, skipped, extractedScope, errs := extractArchiveResources(p, archive, desired)
result.Written += written
result.Overwritten += overwritten
result.Skipped += skipped
scope.Source = scope.Source || extractedScope.Source
scope.Assets = scope.Assets || extractedScope.Assets
failures = append(failures, errs...)
}
if p.EffectiveConfig().Extract.CleanupStale == nil || *p.EffectiveConfig().Extract.CleanupStale {
removed, errs := cleanupStaleFiles(p, desired, scope)
result.Removed = removed
failures = append(failures, errs...)
}
if len(failures) > 0 {
return result, errors.Join(failures...)
}
consumeAll, consumeModules := extractionConsumePolicy(p)
if consumeAll || consumeModules {
for _, archivePath := range archives {
if !consumeAll && archivePath.Extension != ".mod" {
continue
}
if err := os.Remove(archivePath.Path); err != nil {
return result, fmt.Errorf("delete consumed archive %s: %w", archivePath.Path, err)
}
result.DeletedArchivePaths = append(result.DeletedArchivePaths, archivePath.Path)
}
}
return result, nil
}
func extractArchiveResources(p *project.Project, archive erf.Archive, desired map[string]struct{}) (int, int, int, extractionScope, []error) {
var failures []error
writtenCount := 0
overwrittenCount := 0
skippedCount := 0
scope := extractionScope{}
ignored := make(map[string]bool)
for _, ext := range p.Config.Extract.IgnoreExtensions {
ext := strings.TrimPrefix(ext, ".")
ignored[ext] = true
}
for _, resource := range archive.Resources {
ext, ok := erf.ExtensionForResourceType(resource.Type)
if !ok {
failures = append(failures, fmt.Errorf("unsupported resource type 0x%04X for %s", resource.Type, resource.Name))
continue
}
if ignored[ext] {
skippedCount++
continue
}
target, data, err := extractedFile(p, resource, ext)
if err != nil {
failures = append(failures, err)
continue
}
desired[target] = struct{}{}
scope.Source = scope.Source || pathWithinRoot(target, p.SourceDir())
scope.Assets = scope.Assets || pathWithinRoot(target, p.AssetsDir())
state, err := writeManagedFile(target, data)
if err != nil {
failures = append(failures, err)
continue
}
switch state {
case writeNew:
writtenCount++
case writeOverwritten:
overwrittenCount++
case writeSkipped:
skippedCount++
}
}
return writtenCount, overwrittenCount, skippedCount, scope, failures
}
func resolveExtractionArchives(p *project.Project, overrides []string) ([]extractionArchive, error) {
patterns := p.EffectiveConfig().Extract.Archives
if len(overrides) > 0 {
patterns = overrides
}
if len(patterns) == 0 {
return nil, fmt.Errorf("extract.archives must contain at least one build-relative archive pattern")
}
buildRoot := filepath.Clean(p.BuildDir())
byPath := map[string]extractionArchive{}
for _, rawPattern := range patterns {
pattern, err := normalizeExtractionArchivePattern(p, rawPattern)
if err != nil {
return nil, err
}
matched, err := matchExtractionArchives(buildRoot, pattern)
if err != nil {
return nil, err
}
if len(matched) == 0 {
return nil, fmt.Errorf("extract archive pattern %q matched no files under %s", rawPattern, buildRoot)
}
for _, archive := range matched {
byPath[archive.Path] = archive
}
}
archives := make([]extractionArchive, 0, len(byPath))
for _, archive := range byPath {
archives = append(archives, archive)
}
slices.SortFunc(archives, func(a, b extractionArchive) int {
return strings.Compare(a.Rel, b.Rel)
})
return archives, nil
}
func normalizeExtractionArchivePattern(p *project.Project, raw string) (string, error) {
pattern := strings.TrimSpace(raw)
if pattern == "" {
return "", fmt.Errorf("extract archive pattern must not be empty")
}
pattern = strings.NewReplacer(
"{module.resref}", strings.TrimSpace(p.Config.Module.ResRef),
).Replace(pattern)
if strings.Contains(pattern, "\x00") {
return "", fmt.Errorf("extract archive pattern %q must not contain NUL bytes", raw)
}
if filepath.IsAbs(pattern) {
return "", fmt.Errorf("extract archive pattern %q must be relative to paths.build", raw)
}
clean := filepath.Clean(filepath.FromSlash(pattern))
if clean == "." || clean == ".." || strings.HasPrefix(clean, ".."+string(filepath.Separator)) {
return "", fmt.Errorf("extract archive pattern %q must not escape paths.build", raw)
}
return filepath.ToSlash(clean), nil
}
func matchExtractionArchives(buildRoot, pattern string) ([]extractionArchive, error) {
var archives []extractionArchive
err := filepath.WalkDir(buildRoot, func(path string, entry os.DirEntry, err error) error {
if err != nil {
return err
}
if entry.IsDir() {
return nil
}
rel, err := filepath.Rel(buildRoot, path)
if err != nil {
return err
}
rel = filepath.ToSlash(rel)
if !matchPathPattern(rel, pattern) {
return nil
}
archive, err := validateExtractionArchive(buildRoot, rel)
if err != nil {
return err
}
archives = append(archives, archive)
return nil
})
if err != nil {
if errors.Is(err, os.ErrNotExist) {
return nil, fmt.Errorf("scan build directory %s: %w", buildRoot, err)
}
return nil, err
}
return archives, nil
}
func validateExtractionArchive(buildRoot, rel string) (extractionArchive, error) {
cleanRel := filepath.Clean(filepath.FromSlash(rel))
if cleanRel == "." || cleanRel == ".." || strings.HasPrefix(cleanRel, ".."+string(filepath.Separator)) {
return extractionArchive{}, fmt.Errorf("extract archive %q must not escape paths.build", rel)
}
path := filepath.Join(buildRoot, cleanRel)
if !pathWithinRoot(path, buildRoot) {
return extractionArchive{}, fmt.Errorf("extract archive %q resolves outside paths.build", rel)
}
extension := strings.ToLower(filepath.Ext(cleanRel))
switch extension {
case ".mod", ".hak":
return extractionArchive{Path: path, Rel: filepath.ToSlash(cleanRel), Extension: extension}, nil
case ".erf":
return extractionArchive{}, fmt.Errorf("extract archive %q is an .erf; ERF extraction is unsafe because ERFs lack module.ifo context", rel)
default:
return extractionArchive{}, fmt.Errorf("extract archive %q uses unsupported extension %q", rel, extension)
}
}
func extractionConsumePolicy(p *project.Project) (consumeAll bool, consumeModules bool) {
if configured := p.Config.Extract.ConsumeArchives; configured != nil {
return *configured, false
}
return false, p.Config.Extract.DeleteModuleArchiveAfterSuccess
}
func pathWithinRoot(path, root string) bool {
root = filepath.Clean(root)
if root == "" || root == "." {
return false
}
rel, err := filepath.Rel(root, path)
if err != nil {
return false
}
return rel != ".." && !strings.HasPrefix(rel, ".."+string(filepath.Separator)) && !filepath.IsAbs(rel)
}
func extractedFile(p *project.Project, resource erf.Resource, extension string) (string, []byte, error) {
resref := strings.ToLower(resource.Name)
effective := p.EffectiveConfig()
switch extension {
case "nss":
target, err := extractionTarget(p, "paths.source", effective.Paths.Source, p.SourceDir(), "scripts", resref+".nss")
if err != nil {
return "", nil, err
}
return target, resource.Data, nil
case "utc", "utd", "ute", "uti", "utm", "utp", "uts", "utt", "utw",
"are", "dlg", "fac", "gic", "git", "ifo", "itp", "jrl":
document, err := gff.Read(bytes.NewReader(resource.Data))
if err != nil {
return "", nil, fmt.Errorf("decode gff %s.%s: %w", resource.Name, extension, err)
}
target, err := extractionTarget(p, "paths.source", effective.Paths.Source, p.SourceDir(), sourceSubdir(extension), resref+"."+extension+".json")
if err != nil {
return "", nil, err
}
if err := mergeExtractedGFFJSON(p, target, &document); err != nil {
return "", nil, fmt.Errorf("merge extracted gff json %s.%s: %w", resource.Name, extension, err)
}
formatted, err := json.MarshalIndent(document, "", " ")
if err != nil {
return "", nil, fmt.Errorf("marshal json %s.%s: %w", resource.Name, extension, err)
}
formatted = append(formatted, '\n')
return target, formatted, nil
default:
target, err := extractionTarget(p, "paths.assets", effective.Paths.Assets, p.AssetsDir(), extension, resref+"."+extension)
if err != nil {
return "", nil, err
}
return target, resource.Data, nil
}
}
func extractionTarget(p *project.Project, field, configured, root string, parts ...string) (string, error) {
if strings.TrimSpace(configured) == "" {
return "", fmt.Errorf("cannot extract resource: %s is not configured", field)
}
cleanRoot := filepath.Clean(root)
if cleanRoot == "." || cleanRoot == string(filepath.Separator) || cleanRoot == filepath.Clean(p.Root) {
return "", fmt.Errorf("cannot extract resource: %s resolves to unsafe extraction root %s", field, cleanRoot)
}
target := filepath.Join(append([]string{cleanRoot}, parts...)...)
rel, err := filepath.Rel(cleanRoot, target)
if err != nil {
return "", fmt.Errorf("resolve extraction target %s: %w", target, err)
}
if rel == ".." || strings.HasPrefix(rel, ".."+string(filepath.Separator)) || filepath.IsAbs(rel) {
return "", fmt.Errorf("refusing to extract resource outside %s: %s", cleanRoot, target)
}
return target, nil
}
func mergeExtractedGFFJSON(p *project.Project, target string, extracted *gff.Document) error {
rule, ok, err := extractGFFJSONMergeRule(p, target)
if err != nil || !ok {
return err
}
raw, err := os.ReadFile(target)
if err != nil {
if errors.Is(err, os.ErrNotExist) {
return nil
}
return fmt.Errorf("read existing source %s: %w", target, err)
}
var existing gff.Document
if err := json.Unmarshal(raw, &existing); err != nil {
return fmt.Errorf("parse existing source %s: %w", target, err)
}
for _, label := range rule.PreserveFields {
if field, ok := gffField(existing.Root, label); ok {
setGFFField(&extracted.Root, field)
}
}
for _, listRule := range rule.MergeLists {
if err := mergeGFFListByKey(&extracted.Root, existing.Root, listRule); err != nil {
return err
}
}
return nil
}
func extractGFFJSONMergeRule(p *project.Project, target string) (project.ExtractGFFJSONMergeRule, bool, error) {
rel, err := filepath.Rel(filepath.Clean(p.SourceDir()), filepath.Clean(target))
if err != nil {
return project.ExtractGFFJSONMergeRule{}, false, fmt.Errorf("resolve source-relative target %s: %w", target, err)
}
rel = filepath.ToSlash(rel)
if rel == ".." || strings.HasPrefix(rel, "../") {
return project.ExtractGFFJSONMergeRule{}, false, nil
}
for _, rule := range p.EffectiveConfig().Extract.Merge.GFFJSON {
if rule.Target == rel {
return rule, true, nil
}
}
return project.ExtractGFFJSONMergeRule{}, false, nil
}
func gffField(s gff.Struct, label string) (gff.Field, bool) {
for _, field := range s.Fields {
if field.Label == label {
return field, true
}
}
return gff.Field{}, false
}
func setGFFField(s *gff.Struct, replacement gff.Field) {
for index, field := range s.Fields {
if field.Label == replacement.Label {
s.Fields[index] = replacement
return
}
}
s.Fields = append(s.Fields, replacement)
}
func mergeGFFListByKey(extracted *gff.Struct, existing gff.Struct, rule project.ExtractListMergeRule) error {
extractedField, ok := gffField(*extracted, rule.Field)
if !ok {
return fmt.Errorf("extracted field %q not found", rule.Field)
}
extractedList, ok := extractedField.Value.(gff.ListValue)
if !ok {
return fmt.Errorf("extracted field %q is %s, not List", rule.Field, extractedField.Type)
}
existingField, ok := gffField(existing, rule.Field)
if !ok {
return nil
}
existingList, ok := existingField.Value.(gff.ListValue)
if !ok {
return fmt.Errorf("existing field %q is %s, not List", rule.Field, existingField.Type)
}
extractedByKey := map[string]gff.Struct{}
extractedOrder := make([]string, 0, len(extractedList))
for _, item := range extractedList {
key, err := gffStructKey(item, rule.KeyField)
if err != nil {
return fmt.Errorf("extracted field %q: %w", rule.Field, err)
}
if _, exists := extractedByKey[key]; exists {
return fmt.Errorf("extracted field %q has duplicate %s key %q", rule.Field, rule.KeyField, key)
}
extractedByKey[key] = item
extractedOrder = append(extractedOrder, key)
}
merged := make(gff.ListValue, 0, len(extractedList))
seen := map[string]struct{}{}
for _, item := range existingList {
key, err := gffStructKey(item, rule.KeyField)
if err != nil {
return fmt.Errorf("existing field %q: %w", rule.Field, err)
}
if _, duplicate := seen[key]; duplicate {
return fmt.Errorf("existing field %q has duplicate %s key %q", rule.Field, rule.KeyField, key)
}
if extractedItem, exists := extractedByKey[key]; exists {
merged = append(merged, extractedItem)
seen[key] = struct{}{}
}
}
for _, key := range extractedOrder {
if _, exists := seen[key]; exists {
continue
}
merged = append(merged, extractedByKey[key])
}
setGFFField(extracted, gff.NewField(rule.Field, merged))
return nil
}
func gffStructKey(s gff.Struct, keyField string) (string, error) {
field, ok := gffField(s, keyField)
if !ok {
return "", fmt.Errorf("key field %q not found", keyField)
}
switch value := field.Value.(type) {
case gff.ResRefValue:
return string(value), nil
case gff.StringValue:
return string(value), nil
default:
return "", fmt.Errorf("key field %q is %s, not ResRef or CExoString", keyField, field.Type)
}
}
type writeState int
const (
writeSkipped writeState = iota
writeNew
writeOverwritten
)
func writeManagedFile(path string, data []byte) (writeState, error) {
if err := os.MkdirAll(filepath.Dir(path), 0o755); err != nil {
return writeSkipped, fmt.Errorf("create parent directory for %s: %w", path, err)
}
existing, err := os.ReadFile(path)
if err == nil {
if bytes.Equal(existing, data) {
return writeSkipped, nil
}
if err := os.WriteFile(path, data, 0o644); err != nil {
return writeSkipped, fmt.Errorf("overwrite %s: %w", path, err)
}
return writeOverwritten, nil
}
if !errors.Is(err, os.ErrNotExist) {
return writeSkipped, fmt.Errorf("check existing file %s: %w", path, err)
}
if err := os.WriteFile(path, data, 0o644); err != nil {
return writeSkipped, fmt.Errorf("write %s: %w", path, err)
}
return writeNew, nil
}
func cleanupStaleFiles(p *project.Project, desired map[string]struct{}, scope extractionScope) (int, []error) {
candidates := make([]string, 0, len(p.Inventory.SourceFiles)+len(p.Inventory.ScriptFiles)+len(p.Inventory.AssetFiles))
if scope.Source && safeCleanupRoot(p.SourceDir(), p.Root) {
for _, rel := range p.Inventory.SourceFiles {
candidates = append(candidates, filepath.Join(p.SourceDir(), filepath.FromSlash(rel)))
}
for _, rel := range p.Inventory.ScriptFiles {
candidates = append(candidates, filepath.Join(p.SourceDir(), filepath.FromSlash(rel)))
}
}
if scope.Assets && safeCleanupRoot(p.AssetsDir(), p.Root) {
for _, rel := range p.Inventory.AssetFiles {
candidates = append(candidates, filepath.Join(p.AssetsDir(), filepath.FromSlash(rel)))
}
}
removed := 0
var failures []error
for _, path := range candidates {
if _, keep := desired[path]; keep {
continue
}
if err := os.Remove(path); err != nil {
if errors.Is(err, os.ErrNotExist) {
continue
}
failures = append(failures, fmt.Errorf("remove stale file %s: %w", path, err))
continue
}
removed++
cleanupEmptyParents(filepath.Dir(path), p.SourceDir(), p.AssetsDir())
}
return removed, failures
}
func safeCleanupRoot(root, projectRoot string) bool {
cleanRoot := filepath.Clean(root)
return cleanRoot != "." && cleanRoot != string(filepath.Separator) && cleanRoot != filepath.Clean(projectRoot)
}
func cleanupEmptyParents(dir string, roots ...string) {
for {
if dir == "." || dir == string(filepath.Separator) {
return
}
stop := false
for _, root := range roots {
if dir == root {
stop = true
break
}
}
if stop {
return
}
if err := os.Remove(dir); err != nil {
return
}
dir = filepath.Dir(dir)
}
}
func sourceSubdir(extension string) string {
switch strings.ToLower(extension) {
case "are":
return "areas"
case "dlg":
return "dialogs"
case "fac":
return "factions"
case "gic":
return "instance"
case "git":
return "instance"
case "ifo":
return "module"
case "itp":
return "palettes"
case "jrl":
return "journal"
case "utc":
return filepath.Join("blueprints", "creatures")
case "utd":
return filepath.Join("blueprints", "doors")
case "ute":
return filepath.Join("blueprints", "encounters")
case "uti":
return filepath.Join("blueprints", "items")
case "utm":
return filepath.Join("blueprints", "merchants")
case "utp":
return filepath.Join("blueprints", "placeables")
case "uts":
return filepath.Join("blueprints", "sounds")
case "utt":
return filepath.Join("blueprints", "triggers")
case "utw":
return filepath.Join("blueprints", "waypoints")
default:
return extension
}
}
internal/pipeline/music.go
+524
View File
@@ -0,0 +1,524 @@
package pipeline
import (
"encoding/json"
"fmt"
"os"
"path/filepath"
"sort"
"strings"
"git.westgate.pw/ShadowsOverWestgate/sow-tools/internal/erf"
"git.westgate.pw/ShadowsOverWestgate/sow-tools/internal/music"
"git.westgate.pw/ShadowsOverWestgate/sow-tools/internal/project"
)
type preparedMusicAssets struct {
Generated []assetResource
SkipSourceRel map[string]struct{}
Artifacts []string
TempRoots []string
Summary CreditsRefreshSummary
}
type MusicBuildOptions struct {
DatasetIDs []string
Write bool
}
type musicPrepareOptions struct {
datasetIDs []string
write bool
}
type musicSourceGroup struct {
DatasetID string
Dataset project.EffectiveMusicDataset
SourceDir string
OutputDir string
Sources []string
}
type musicCreditGroup struct {
CreditsRoot string
SourceDir string
Entries []music.CreditsEntry
}
func musicRootPath(p *project.Project, path string) string {
if strings.TrimSpace(path) == "" {
return p.Root
}
if filepath.IsAbs(path) {
return path
}
return filepath.Join(p.Root, filepath.FromSlash(path))
}
func effectiveMusicDatasets(p *project.Project, onlyIDs []string) (map[string]project.EffectiveMusicDataset, error) {
effective := p.EffectiveConfig()
wanted := map[string]struct{}{}
for _, id := range onlyIDs {
id = strings.TrimSpace(id)
if id != "" {
wanted[id] = struct{}{}
}
}
datasets := map[string]project.EffectiveMusicDataset{}
for id, dataset := range effective.Music.Datasets {
if len(wanted) > 0 {
if _, ok := wanted[id]; !ok {
continue
}
delete(wanted, id)
}
datasets[id] = dataset
}
if len(wanted) > 0 {
unknown := make([]string, 0, len(wanted))
for id := range wanted {
unknown = append(unknown, id)
}
sort.Strings(unknown)
return nil, fmt.Errorf("unknown music dataset(s): %s", strings.Join(unknown, ", "))
}
if len(datasets) == 0 && len(wanted) == 0 {
for _, rel := range p.Inventory.AssetFiles {
ext := strings.ToLower(filepath.Ext(rel))
if !music.PathIsUnder("envi/music", rel) {
continue
}
if _, ok := extensionSet(effective.Music.ConvertExtensions)[ext]; ok {
datasets["legacy_envi_music"] = project.EffectiveMusicDataset{
Source: "envi/music",
Output: "envi/music",
StageRoot: effective.Music.StageRoot,
CreditsRoot: effective.Music.CreditsRoot,
NamingScheme: effective.Music.NamingScheme,
OutputExtension: effective.Music.OutputExtension,
MaxStemLength: effective.Music.MaxStemLength,
PackageMode: "hak_asset",
ConvertExtensions: effective.Music.ConvertExtensions,
}
break
}
}
}
return datasets, nil
}
func extensionSet(values []string) map[string]struct{} {
out := make(map[string]struct{}, len(values))
for _, value := range values {
value = strings.ToLower(strings.TrimSpace(value))
if value != "" {
out[value] = struct{}{}
}
}
return out
}
func datasetForMusicSource(datasets map[string]project.EffectiveMusicDataset, rel, ext string) (string, project.EffectiveMusicDataset, bool) {
bestID := ""
var best project.EffectiveMusicDataset
bestLen := -1
for id, dataset := range datasets {
if _, ok := extensionSet(dataset.ConvertExtensions)[ext]; !ok {
continue
}
if !music.PathIsUnder(dataset.Source, rel) {
continue
}
if len(dataset.Source) > bestLen {
bestID = id
best = dataset
bestLen = len(dataset.Source)
}
}
return bestID, best, bestID != ""
}
func plannedMusicAsset(outputRel, sourceRel string) (assetResource, error) {
extension := strings.TrimPrefix(strings.ToLower(filepath.Ext(outputRel)), ".")
resourceType, ok := erf.HAKResourceTypeForExtension(extension)
if !ok {
return assetResource{}, fmt.Errorf("unsupported generated music resource extension %q", filepath.Ext(outputRel))
}
name := strings.ToLower(strings.TrimSuffix(filepath.Base(outputRel), filepath.Ext(outputRel)))
return assetResource{
Rel: outputRel,
Resource: erf.Resource{
Name: name,
Type: resourceType,
},
ContentID: "music-plan:" + filepath.ToSlash(sourceRel),
}, nil
}
func BuildMusic(p *project.Project, opts MusicBuildOptions) (BuildResult, error) {
prepared, err := prepareMusicAssetsWithOptions(p, musicPrepareOptions{datasetIDs: opts.DatasetIDs, write: opts.Write})
if err != nil {
return BuildResult{}, err
}
if err := cleanupPreparedMusicAssets(prepared); err != nil {
return BuildResult{}, err
}
return BuildResult{
CreditsArtifactPaths: prepared.Artifacts,
CreditsSummary: prepared.Summary,
HAKAssets: len(prepared.Generated),
}, nil
}
func prepareMusicAssets(p *project.Project) (*preparedMusicAssets, error) {
return prepareMusicAssetsWithOptions(p, musicPrepareOptions{write: true})
}
func prepareMusicAssetsWithOptions(p *project.Project, opts musicPrepareOptions) (*preparedMusicAssets, error) {
usedNames := make(map[string]struct{})
groupsByKey := make(map[string]*musicSourceGroup)
datasets, err := effectiveMusicDatasets(p, opts.datasetIDs)
if err != nil {
return nil, err
}
for _, rel := range p.Inventory.AssetFiles {
rel = filepath.ToSlash(rel)
ext := strings.ToLower(filepath.Ext(rel))
name := strings.ToLower(strings.TrimSuffix(filepath.Base(rel), filepath.Ext(rel)))
if datasetID, dataset, ok := datasetForMusicSource(datasets, rel, ext); ok {
sourceDir := filepath.ToSlash(filepath.Dir(rel))
relDir, err := filepath.Rel(filepath.FromSlash(dataset.Source), filepath.FromSlash(sourceDir))
if err != nil {
return nil, fmt.Errorf("resolve music source directory %s: %w", sourceDir, err)
}
relDir = filepath.ToSlash(relDir)
outputDir := dataset.Output
if relDir != "." && relDir != "" {
outputDir = filepath.ToSlash(filepath.Join(outputDir, filepath.FromSlash(relDir)))
}
key := datasetID + "\x00" + sourceDir
group := groupsByKey[key]
if group == nil {
group = &musicSourceGroup{
DatasetID: datasetID,
Dataset: dataset,
SourceDir: sourceDir,
OutputDir: outputDir,
}
groupsByKey[key] = group
}
group.Sources = append(group.Sources, rel)
continue
}
if name != "" {
usedNames[name] = struct{}{}
}
}
result := &preparedMusicAssets{
SkipSourceRel: make(map[string]struct{}),
}
if len(groupsByKey) == 0 {
if !opts.write {
return result, nil
}
writeResult, err := writeCreditsArtifacts(p, nil)
if err != nil {
return nil, err
}
result.Artifacts = writeResult.Artifacts
result.Summary = writeResult.Summary
return result, nil
}
groups := make([]*musicSourceGroup, 0, len(groupsByKey))
for _, group := range groupsByKey {
sort.Strings(group.Sources)
groups = append(groups, group)
}
sort.Slice(groups, func(i, j int) bool {
if groups[i].SourceDir != groups[j].SourceDir {
return groups[i].SourceDir < groups[j].SourceDir
}
return groups[i].DatasetID < groups[j].DatasetID
})
creditGroups := make([]musicCreditGroup, 0, len(groups))
for _, group := range groups {
result.Summary.ScannedDirs = append(result.Summary.ScannedDirs, group.SourceDir)
result.Summary.TrackCount += len(group.Sources)
}
ffmpegPath := ""
ffprobePath := ""
if opts.write {
var err error
ffmpegPath, err = music.ResolveFFmpegWithConfig(p.MusicFFmpegPath())
if err != nil {
return nil, err
}
ffprobePath, err = music.ResolveFFprobeWithConfig(p.MusicFFprobePath())
if err != nil {
return nil, err
}
}
for _, group := range groups {
stageRoot := musicRootPath(p, group.Dataset.StageRoot)
result.TempRoots = append(result.TempRoots, stageRoot)
prefix := music.SanitizePrefix(group.Dataset.Prefix)
overlayPath := filepath.Join(p.AssetsDir(), filepath.FromSlash(group.SourceDir), p.EffectiveConfig().Music.CreditsOverlay)
overlay, err := music.ParseCreditsOverlay(overlayPath)
if err != nil {
return nil, fmt.Errorf("parse credits overlay %s: %w", overlayPath, err)
}
entries := make([]music.CreditsEntry, 0, len(group.Sources))
for _, rel := range group.Sources {
base := filepath.Base(rel)
manual := overlay.Match(base, "")
outputFile := ""
switch {
case manual != nil && strings.TrimSpace(manual.OutputFile) != "":
outputFile = strings.ToLower(strings.TrimSpace(manual.OutputFile))
if err := music.ValidateManualOutputFileWithRules(outputFile, group.Dataset.OutputExtension, group.Dataset.MaxStemLength); err != nil {
return nil, fmt.Errorf("manual credits output for %s: %w", rel, err)
}
if err := music.ReserveName(strings.TrimSuffix(outputFile, filepath.Ext(outputFile)), usedNames); err != nil {
return nil, fmt.Errorf("manual credits output for %s: %w", rel, err)
}
default:
stem, err := music.GenerateStemWithMax(base, prefix, group.Dataset.MaxStemLength, usedNames)
if err != nil {
return nil, fmt.Errorf("generate music name for %s: %w", rel, err)
}
outputFile = stem + group.Dataset.OutputExtension
}
outputRel := filepath.ToSlash(filepath.Join(group.OutputDir, outputFile))
result.Summary.Mappings = append(result.Summary.Mappings, MusicFileMapping{
Source: rel,
Output: outputRel,
})
result.SkipSourceRel[rel] = struct{}{}
if !opts.write {
if group.Dataset.PackageMode == "hak_asset" {
asset, err := plannedMusicAsset(outputRel, rel)
if err != nil {
return nil, err
}
result.Generated = append(result.Generated, asset)
}
continue
}
metadata, err := music.ProbeMetadata(ffprobePath, filepath.Join(p.AssetsDir(), filepath.FromSlash(rel)))
if err != nil {
return nil, fmt.Errorf("probe music metadata for %s: %w", rel, err)
}
entry := music.CreditsEntry{
Artist: metadata.Artist,
Title: music.FirstNonEmpty(metadata.Title, strings.TrimSuffix(base, filepath.Ext(base))),
OutputFile: outputFile,
OriginalFile: base,
Album: metadata.Album,
Date: metadata.Date,
Rights: music.FirstNonEmpty(metadata.Rights, music.JoinNonEmpty("; ", metadata.License, metadata.Copyright)),
Notes: music.FirstNonEmpty(metadata.Notes, metadata.Comment),
}
music.ApplyCreditsOverlay(&entry, manual)
stagePath := filepath.Join(stageRoot, filepath.FromSlash(outputRel))
if err := os.MkdirAll(filepath.Dir(stagePath), 0o755); err != nil {
return nil, fmt.Errorf("create music stage dir: %w", err)
}
if err := music.ConvertSource(ffmpegPath, filepath.Join(p.AssetsDir(), filepath.FromSlash(rel)), stagePath); err != nil {
return nil, fmt.Errorf("convert music %s: %w", rel, err)
}
if group.Dataset.PackageMode == "hak_asset" {
resourceInfo, err := assetResourceFromPath(stagePath, outputRel, lfsAssetInfo{}, false)
if err != nil {
return nil, fmt.Errorf("load converted music %s: %w", outputRel, err)
}
info, err := os.Stat(stagePath)
if err != nil {
return nil, fmt.Errorf("stat converted music %s: %w", stagePath, err)
}
result.Generated = append(result.Generated, assetResource{
Rel: outputRel,
Resource: resourceInfo.Resource,
Size: erf.ArchiveSize([]erf.Resource{resourceInfo.Resource}),
CreatedAt: info.ModTime(),
ContentID: resourceInfo.ContentID,
})
}
entries = append(entries, entry)
}
if opts.write {
if err := music.ValidateOverlayEntries(group.SourceDir, overlay, entries); err != nil {
return nil, err
}
creditGroups = append(creditGroups, musicCreditGroup{
CreditsRoot: group.Dataset.CreditsRoot,
SourceDir: group.SourceDir,
Entries: entries,
})
}
}
if !opts.write {
sort.Slice(result.Generated, func(i, j int) bool {
return compareResourceKeys(result.Generated[i].Resource, result.Generated[j].Resource) < 0
})
return result, nil
}
writeResult, err := writeCreditsArtifacts(p, creditGroups)
if err != nil {
return nil, err
}
writeResult.Summary.ScannedDirs = append(writeResult.Summary.ScannedDirs, result.Summary.ScannedDirs...)
writeResult.Summary.TrackCount = result.Summary.TrackCount
result.Artifacts = writeResult.Artifacts
result.Summary = writeResult.Summary
sort.Slice(result.Generated, func(i, j int) bool {
return compareResourceKeys(result.Generated[i].Resource, result.Generated[j].Resource) < 0
})
return result, nil
}
type creditsArtifactWriteResult struct {
Artifacts []string
Summary CreditsRefreshSummary
}
func writeCreditsArtifacts(p *project.Project, generated []musicCreditGroup) (creditsArtifactWriteResult, error) {
defaultCreditsRoot := p.CreditsDir()
sources := make([]music.CreditsSource, 0)
desiredByRoot := map[string]map[string][]byte{}
summary := CreditsRefreshSummary{}
if generated != nil {
sort.Slice(generated, func(i, j int) bool {
if generated[i].SourceDir != generated[j].SourceDir {
return generated[i].SourceDir < generated[j].SourceDir
}
return generated[i].CreditsRoot < generated[j].CreditsRoot
})
for _, group := range generated {
creditsRoot := musicRootPath(p, group.CreditsRoot)
if desiredByRoot[creditsRoot] == nil {
desiredByRoot[creditsRoot] = map[string][]byte{}
}
entries := append([]music.CreditsEntry(nil), group.Entries...)
sort.Slice(entries, func(i, j int) bool {
if strings.ToLower(entries[i].Artist) != strings.ToLower(entries[j].Artist) {
return strings.ToLower(entries[i].Artist) < strings.ToLower(entries[j].Artist)
}
if strings.ToLower(entries[i].Title) != strings.ToLower(entries[j].Title) {
return strings.ToLower(entries[i].Title) < strings.ToLower(entries[j].Title)
}
return strings.ToLower(entries[i].OutputFile) < strings.ToLower(entries[j].OutputFile)
})
outputPath := filepath.Join(creditsRoot, filepath.FromSlash(group.SourceDir), p.EffectiveConfig().Music.CreditsOverlay)
desiredByRoot[creditsRoot][outputPath] = []byte(music.RenderCreditsMarkdown(entries))
summary.GeneratedPaths = append(summary.GeneratedPaths, outputPath)
for _, entry := range entries {
summary.Mappings = append(summary.Mappings, MusicFileMapping{
Source: entry.OriginalFile,
Output: entry.OutputFile,
})
}
sources = append(sources, music.CreditsSource{
Path: filepath.ToSlash(outputPath),
Kind: "generated_music",
Entries: entries,
})
}
}
err := filepath.WalkDir(p.AssetsDir(), func(path string, d os.DirEntry, walkErr error) error {
if walkErr != nil {
return walkErr
}
if d.IsDir() {
return nil
}
if strings.ToLower(d.Name()) != "credits.md" {
return nil
}
entries, err := music.ParseCreditsMarkdown(path)
if err != nil {
return fmt.Errorf("parse credits %s: %w", path, err)
}
rel, err := filepath.Rel(p.Root, path)
if err != nil {
return err
}
sources = append(sources, music.CreditsSource{
Path: filepath.ToSlash(rel),
Kind: "authored",
Entries: entries,
})
return nil
})
if err != nil {
return creditsArtifactWriteResult{}, err
}
sort.Slice(sources, func(i, j int) bool { return sources[i].Path < sources[j].Path })
payload, err := json.MarshalIndent(music.CreditsInventory{Sources: sources}, "", " ")
if err != nil {
return creditsArtifactWriteResult{}, fmt.Errorf("marshal credits inventory: %w", err)
}
payload = append(payload, '\n')
if len(desiredByRoot) == 0 {
desiredByRoot[defaultCreditsRoot] = map[string][]byte{}
}
artifacts := make([]string, 0)
changedFiles := 0
roots := make([]string, 0, len(desiredByRoot))
for root := range desiredByRoot {
roots = append(roots, root)
}
sort.Strings(roots)
for _, creditsRoot := range roots {
desiredFiles := desiredByRoot[creditsRoot]
inventoryPath := filepath.Join(creditsRoot, "credits.json")
desiredFiles[inventoryPath] = payload
if err := os.MkdirAll(creditsRoot, 0o755); err != nil {
return creditsArtifactWriteResult{}, fmt.Errorf("create credits dir: %w", err)
}
changed, err := music.SyncGeneratedCreditsArtifacts(creditsRoot, desiredFiles)
if err != nil {
return creditsArtifactWriteResult{}, err
}
changedFiles += changed
for path := range desiredFiles {
artifacts = append(artifacts, path)
}
if summary.InventoryPath == "" {
summary.InventoryPath = inventoryPath
}
}
sort.Strings(artifacts)
summary.ArtifactPaths = append(summary.ArtifactPaths, artifacts...)
summary.ChangedFiles = changedFiles
return creditsArtifactWriteResult{
Artifacts: artifacts,
Summary: summary,
}, nil
}
func cleanupPreparedMusicAssets(prepared *preparedMusicAssets) error {
if prepared == nil {
return nil
}
for _, root := range prepared.TempRoots {
if strings.TrimSpace(root) == "" {
continue
}
if err := os.RemoveAll(root); err != nil {
return fmt.Errorf("remove temporary music artifact root %s: %w", root, err)
}
}
return nil
}
internal/pipeline/pipeline_test.go
+4057
View File
File diff suppressed because it is too large Load Diff
internal/project/effective.go
+738
View File
@@ -0,0 +1,738 @@
package project
import (
"encoding/json"
"fmt"
"os"
"path/filepath"
"slices"
"strings"
"time"
"git.westgate.pw/ShadowsOverWestgate/sow-tools/internal/music"
)
const (
DefaultBuildPath = "build"
DefaultCachePath = ".cache"
DefaultToolsPath = "tools"
DefaultModuleArchiveTemplate = "{module.resref}.mod"
DefaultHAKManifest = "haks.json"
DefaultHAKArchiveTemplate = "{hak.name}.hak"
DefaultSourceJSONPattern = "{resref}.{extension}.json"
DefaultValidationProfile = "nwn_module"
DefaultMusicStageRoot = "{paths.cache}/music"
DefaultCreditsRoot = "{paths.cache}/credits"
DefaultCreditsOverlayFile = "CREDITS.md"
DefaultMusicNamingScheme = "nwn_bmu"
DefaultMusicOutputExtension = ".bmu"
DefaultTopDataBuild = "{paths.build}/topdata"
DefaultTopDataCompiled2DADir = "2da"
DefaultTopDataCompiledTLK = "sow_tlk.tlk"
DefaultTopDataWikiOutputRoot = "wiki"
DefaultTopDataWikiPagesDir = "pages"
DefaultTopDataWikiStateFile = "state.json"
DefaultTopDataWikiSource = "topdata/wiki"
DefaultTopDataWikiRenderer = "nodebb_tiptap_html"
DefaultTopDataWikiLinkStrategy = "preserve_westgate_wiki_links"
DefaultTopDataWikiNamespacesFile = "namespaces.yaml"
DefaultTopDataWikiTablesFile = "tables.yaml"
DefaultTopDataWikiVisibilityFile = "visibility.yaml"
DefaultTopDataWikiDataFile = "data.yaml"
DefaultTopDataWikiPagePathsFile = "wiki.yaml"
DefaultTopDataWikiTemplatesDir = "templates"
DefaultTopDataWikiPageTemplatesDir = "templates/pages"
DefaultTopDataWikiManualSectionsDir = "manual-sections"
DefaultTopDataWikiManualSectionsFile = "manual-sections/default.yaml"
DefaultTopDataWikiPageIndexFile = "page-index.json"
DefaultTopDataWikiStaleDefault = "report"
DefaultTopDataWikiStaleLiveCleanup = "archive"
DefaultTopDataWikiMarkerFormat = "html_comments"
DefaultTopDataWikiPageMarkerPrefix = "sow-topdata-wiki"
DefaultTopDataWikiTitlePrefixMinLength = 3
DefaultTopDataWikiStatusPages = true
DefaultTopDataWikiStatusListingScope = "all"
DefaultTopDataWikiAlignmentLinkFormat = "wiki_markup"
DefaultWikiDeployManifest = ".wiki_deploy_manifest.json"
DefaultWikiDeployEditSummary = "Auto-generated from native builder"
DefaultTopDataPackageHAK = "sow_top.hak"
DefaultTopDataPackageTLK = "sow_tlk.tlk"
DefaultAutogenCacheRoot = "{paths.cache}"
DefaultAutogenCacheMaxAge = time.Hour
DefaultAutogenRefreshEnv = "SOW_AUTOGEN_MANIFEST_REFRESH"
DefaultPartsManifestRefreshEnv = "SOW_PARTS_MANIFEST_REFRESH"
DefaultAutogenReleaseProvider = "gitea"
DefaultAutogenServerURLEnv = "SOW_ASSETS_SERVER_URL"
DefaultAutogenRepoEnv = "SOW_ASSETS_REPO"
DefaultExtractLayout = "nwn_canonical_json"
DefaultExtractHAKDiscovery = "build_glob"
)
type ConfigProvenance map[string]ConfigValueProvenance
type ConfigValueProvenance struct {
Source string `json:"source" yaml:"source"`
Detail string `json:"detail,omitempty" yaml:"detail,omitempty"`
Deprecated bool `json:"deprecated,omitempty" yaml:"deprecated,omitempty"`
}
type EffectiveConfig struct {
ConfigSource ConfigSource `json:"config_source" yaml:"config_source"`
Module ModuleConfig `json:"module" yaml:"module"`
Paths EffectivePathConfig `json:"paths" yaml:"paths"`
Outputs EffectiveOutputConfig `json:"outputs" yaml:"outputs"`
Build BuildConfig `json:"build" yaml:"build"`
Generated GeneratedConfig `json:"generated_assets" yaml:"generated_assets"`
Inventory InventoryConfig `json:"inventory" yaml:"inventory"`
Validation ValidationConfig `json:"validation" yaml:"validation"`
Music EffectiveMusicConfig `json:"music" yaml:"music"`
TopData EffectiveTopDataConfig `json:"topdata" yaml:"topdata"`
Extract ExtractConfig `json:"extract" yaml:"extract"`
Autogen EffectiveAutogenConfig `json:"autogen" yaml:"autogen"`
HAKs []HAKConfig `json:"haks" yaml:"haks"`
Provenance ConfigProvenance `json:"provenance" yaml:"provenance"`
Overrides []ConfigOverride `json:"overrides,omitempty" yaml:"overrides,omitempty"`
}
type EffectivePathConfig struct {
Source string `json:"source" yaml:"source"`
Assets string `json:"assets" yaml:"assets"`
Build string `json:"build" yaml:"build"`
Cache string `json:"cache" yaml:"cache"`
Tools string `json:"tools" yaml:"tools"`
}
type EffectiveOutputConfig struct {
ModuleArchive string `json:"module_archive" yaml:"module_archive"`
HAKManifest string `json:"hak_manifest" yaml:"hak_manifest"`
HAKArchive string `json:"hak_archive" yaml:"hak_archive"`
}
type EffectiveMusicConfig struct {
Prefixes map[string]string `json:"prefixes" yaml:"prefixes"`
Tools MusicToolsConfig `json:"tools" yaml:"tools"`
StageRoot string `json:"stage_root" yaml:"stage_root"`
CreditsRoot string `json:"credits_root" yaml:"credits_root"`
CreditsOverlay string `json:"credits_overlay" yaml:"credits_overlay"`
MaxStemLength int `json:"max_stem_length" yaml:"max_stem_length"`
NamingScheme string `json:"naming_scheme" yaml:"naming_scheme"`
OutputExtension string `json:"output_extension" yaml:"output_extension"`
ConvertExtensions []string `json:"convert_extensions" yaml:"convert_extensions"`
Datasets map[string]EffectiveMusicDataset `json:"datasets,omitempty" yaml:"datasets,omitempty"`
}
type EffectiveMusicDataset struct {
Source string `json:"source" yaml:"source"`
Output string `json:"output" yaml:"output"`
Prefix string `json:"prefix,omitempty" yaml:"prefix,omitempty"`
StageRoot string `json:"stage_root,omitempty" yaml:"stage_root,omitempty"`
CreditsRoot string `json:"credits_root,omitempty" yaml:"credits_root,omitempty"`
NamingScheme string `json:"naming_scheme" yaml:"naming_scheme"`
OutputExtension string `json:"output_extension" yaml:"output_extension"`
MaxStemLength int `json:"max_stem_length" yaml:"max_stem_length"`
PackageMode string `json:"package_mode" yaml:"package_mode"`
ConvertExtensions []string `json:"convert_extensions" yaml:"convert_extensions"`
}
type EffectiveTopDataConfig struct {
Source string `json:"source" yaml:"source"`
Build string `json:"build" yaml:"build"`
ReferenceBuilder string `json:"reference_builder" yaml:"reference_builder"`
Assets string `json:"assets" yaml:"assets"`
Compiled2DADir string `json:"compiled_2da_dir" yaml:"compiled_2da_dir"`
CompiledTLK string `json:"compiled_tlk" yaml:"compiled_tlk"`
PackageHAK string `json:"package_hak" yaml:"package_hak"`
PackageTLK string `json:"package_tlk" yaml:"package_tlk"`
ValueEncodings []TopDataValueEncodingConfig `json:"value_encodings" yaml:"value_encodings"`
ValueDefaults []TopDataValueDefaultConfig `json:"value_defaults" yaml:"value_defaults"`
RowGeneration []TopDataRowGenerationConfig `json:"row_generation" yaml:"row_generation"`
RowExtensions []TopDataRowExtensionConfig `json:"row_extensions" yaml:"row_extensions"`
ClassFeatInjections TopDataClassFeatInjectionConfig `json:"class_feat_injections" yaml:"class_feat_injections"`
Wiki TopDataWikiConfig `json:"wiki" yaml:"wiki"`
}
type EffectiveAutogenConfig struct {
Producers []AutogenProducerConfig `json:"producers" yaml:"producers"`
Consumers []AutogenConsumerConfig `json:"consumers" yaml:"consumers"`
Cache EffectiveAutogenCacheConfig `json:"cache" yaml:"cache"`
ReleaseSource EffectiveAutogenReleaseSourceConfig `json:"release_source" yaml:"release_source"`
}
type EffectiveAutogenCacheConfig struct {
Root string `json:"root" yaml:"root"`
MaxAge string `json:"max_age" yaml:"max_age"`
RefreshEnv string `json:"refresh_env" yaml:"refresh_env"`
PartsManifestRefreshEnv string `json:"parts_manifest_refresh_env" yaml:"parts_manifest_refresh_env"`
}
type EffectiveAutogenReleaseSourceConfig struct {
Provider string `json:"provider" yaml:"provider"`
ServerURLEnv string `json:"server_url_env" yaml:"server_url_env"`
RepoEnv string `json:"repo_env" yaml:"repo_env"`
}
type ConfigOverride struct {
Key string `json:"key" yaml:"key"`
Value string `json:"value" yaml:"value"`
Source string `json:"source" yaml:"source"`
}
func (p *Project) EffectiveConfig() EffectiveConfig {
provenance := cloneProvenance(p.Provenance)
markMissingDefaults(provenance)
paths := EffectivePathConfig{
Source: strings.TrimSpace(p.Config.Paths.Source),
Assets: strings.TrimSpace(p.Config.Paths.Assets),
Build: defaultString(p.Config.Paths.Build, DefaultBuildPath),
Cache: defaultString(p.Config.Paths.Cache, DefaultCachePath),
Tools: defaultString(p.Config.Paths.Tools, DefaultToolsPath),
}
outputs := EffectiveOutputConfig{
ModuleArchive: defaultString(p.Config.Outputs.ModuleArchive, DefaultModuleArchiveTemplate),
HAKManifest: defaultString(p.Config.Outputs.HAKManifest, DefaultHAKManifest),
HAKArchive: defaultString(p.Config.Outputs.HAKArchive, DefaultHAKArchiveTemplate),
}
inventory := InventoryConfig{
SourceExtensions: defaultStringSlice(p.Config.Inventory.SourceExtensions, SourceExtensions),
AssetExtensions: defaultStringSlice(p.Config.Inventory.AssetExtensions, AssetExtensions),
SourceJSONPattern: defaultString(p.Config.Inventory.SourceJSONPattern, DefaultSourceJSONPattern),
}
validation := ValidationConfig{
Profile: defaultString(p.Config.Validation.Profile, DefaultValidationProfile),
BuiltinScriptPrefixes: normalizeLowerStringSlice(defaultStringSlice(p.Config.Validation.BuiltinScriptPrefixes, DefaultBuiltinScriptPrefixes())),
RequiredFields: defaultRequiredFields(p.Config.Validation.RequiredFields),
}
topBuild := expandPathTemplate(defaultString(p.Config.TopData.Build, DefaultTopDataBuild), paths)
top := EffectiveTopDataConfig{
Source: strings.TrimSpace(p.Config.TopData.Source),
Build: topBuild,
ReferenceBuilder: strings.TrimSpace(p.Config.TopData.ReferenceBuilder),
Assets: strings.TrimSpace(p.Config.TopData.Assets),
Compiled2DADir: defaultString(p.Config.TopData.Compiled2DADir, DefaultTopDataCompiled2DADir),
CompiledTLK: defaultString(p.Config.TopData.CompiledTLK, DefaultTopDataCompiledTLK),
PackageHAK: defaultString(p.Config.TopData.PackageHAK, DefaultTopDataPackageHAK),
PackageTLK: defaultString(p.Config.TopData.PackageTLK, DefaultTopDataPackageTLK),
ValueEncodings: cloneTopDataValueEncodings(p.Config.TopData.ValueEncodings),
ValueDefaults: cloneTopDataValueDefaults(p.Config.TopData.ValueDefaults),
RowGeneration: cloneTopDataRowGeneration(p.Config.TopData.RowGeneration),
RowExtensions: cloneTopDataRowExtensions(p.Config.TopData.RowExtensions),
ClassFeatInjections: cloneTopDataClassFeatInjections(p.Config.TopData.ClassFeatInjections),
Wiki: TopDataWikiConfig{
OutputRoot: defaultString(p.Config.TopData.Wiki.OutputRoot, DefaultTopDataWikiOutputRoot),
PagesDir: defaultString(p.Config.TopData.Wiki.PagesDir, DefaultTopDataWikiPagesDir),
StateFile: defaultString(p.Config.TopData.Wiki.StateFile, DefaultTopDataWikiStateFile),
ManagedNamespaces: defaultStringSlice(p.Config.TopData.Wiki.ManagedNamespaces, []string{"classes", "feat", "itemtypes", "races", "skills", "spells", "meta"}),
DeployManifest: defaultString(p.Config.TopData.Wiki.DeployManifest, DefaultWikiDeployManifest),
DeployEditSummary: defaultString(p.Config.TopData.Wiki.DeployEditSummary, DefaultWikiDeployEditSummary),
Source: defaultString(p.Config.TopData.Wiki.Source, DefaultTopDataWikiSource),
Renderer: defaultString(p.Config.TopData.Wiki.Renderer, DefaultTopDataWikiRenderer),
LinkStrategy: defaultString(p.Config.TopData.Wiki.LinkStrategy, DefaultTopDataWikiLinkStrategy),
NamespacesFile: defaultString(p.Config.TopData.Wiki.NamespacesFile, DefaultTopDataWikiNamespacesFile),
TablesFile: defaultString(p.Config.TopData.Wiki.TablesFile, DefaultTopDataWikiTablesFile),
VisibilityFile: defaultString(p.Config.TopData.Wiki.VisibilityFile, DefaultTopDataWikiVisibilityFile),
DataFile: defaultString(p.Config.TopData.Wiki.DataFile, DefaultTopDataWikiDataFile),
PagePathsFile: defaultString(p.Config.TopData.Wiki.PagePathsFile, DefaultTopDataWikiPagePathsFile),
TemplatesDir: defaultString(p.Config.TopData.Wiki.TemplatesDir, DefaultTopDataWikiTemplatesDir),
PageTemplatesDir: defaultWikiPageTemplatesDir(p.Config.TopData.Wiki),
ManualSectionsDir: defaultString(p.Config.TopData.Wiki.ManualSectionsDir, DefaultTopDataWikiManualSectionsDir),
ManualSectionsFile: defaultWikiManualSectionsFile(p.Config.TopData.Wiki),
PageIndexFile: defaultString(p.Config.TopData.Wiki.PageIndexFile, DefaultTopDataWikiPageIndexFile),
AlignmentLinks: TopDataWikiAlignmentLinks{
TargetPattern: strings.TrimSpace(p.Config.TopData.Wiki.AlignmentLinks.TargetPattern),
LinkFormat: defaultString(p.Config.TopData.Wiki.AlignmentLinks.LinkFormat, DefaultTopDataWikiAlignmentLinkFormat),
},
TitlePrefixMinLength: defaultInt(p.Config.TopData.Wiki.TitlePrefixMinLength, DefaultTopDataWikiTitlePrefixMinLength),
StatusPages: defaultBoolPointer(p.Config.TopData.Wiki.StatusPages, DefaultTopDataWikiStatusPages),
StatusListingScope: defaultString(p.Config.TopData.Wiki.StatusListingScope, DefaultTopDataWikiStatusListingScope),
StalePages: TopDataWikiStalePagesConfig{
Default: defaultString(p.Config.TopData.Wiki.StalePages.Default, DefaultTopDataWikiStaleDefault),
LiveCleanup: defaultString(p.Config.TopData.Wiki.StalePages.LiveCleanup, DefaultTopDataWikiStaleLiveCleanup),
},
ManagedRegion: TopDataWikiManagedRegionConfig{
MarkerFormat: defaultString(p.Config.TopData.Wiki.ManagedRegion.MarkerFormat, DefaultTopDataWikiMarkerFormat),
PageMarkerPrefix: defaultString(p.Config.TopData.Wiki.ManagedRegion.PageMarkerPrefix, DefaultTopDataWikiPageMarkerPrefix),
},
},
}
generated := p.Config.Generated
for index := range generated.TopData2DA {
generated.TopData2DA[index].Output = expandPathTemplate(defaultString(generated.TopData2DA[index].Output, "{paths.cache}/generated-assets/"+generated.TopData2DA[index].ID), paths)
}
extract := p.Config.Extract
extract.Layout = defaultString(extract.Layout, DefaultExtractLayout)
extract.HAKDiscovery = defaultString(extract.HAKDiscovery, DefaultExtractHAKDiscovery)
if len(extract.Archives) == 0 {
extract.Archives = []string{outputs.ModuleArchiveName(p.Config.Module)}
}
if extract.CleanupStale == nil {
defaultCleanup := true
extract.CleanupStale = &defaultCleanup
}
if extract.ConsumeArchives == nil {
defaultConsume := false
extract.ConsumeArchives = &defaultConsume
}
musicStageRoot := expandPathTemplate(DefaultMusicStageRoot, paths)
musicCreditsRoot := expandPathTemplate(DefaultCreditsRoot, paths)
musicCreditsOverlay := DefaultCreditsOverlayFile
musicMaxStemLength := 16
musicNamingScheme := DefaultMusicNamingScheme
musicOutputExtension := DefaultMusicOutputExtension
musicConvertExtensions := music.DefaultConvertExtensions()
if d := p.Config.Music.Defaults; d != nil {
if d.StageRoot != "" {
musicStageRoot = expandPathTemplate(d.StageRoot, paths)
}
if d.CreditsRoot != "" {
musicCreditsRoot = expandPathTemplate(d.CreditsRoot, paths)
}
if d.CreditsOverlay != "" {
musicCreditsOverlay = d.CreditsOverlay
}
if d.MaxStemLength != nil {
musicMaxStemLength = *d.MaxStemLength
}
if d.NamingScheme != "" {
musicNamingScheme = d.NamingScheme
}
if d.OutputExtension != "" {
musicOutputExtension = normalizeExtension(d.OutputExtension)
}
if len(d.ConvertExtensions) > 0 {
musicConvertExtensions = normalizeExtensionSlice(d.ConvertExtensions)
}
}
musicPrefixes := cloneStringMap(p.Config.Music.Prefixes)
if len(musicPrefixes) == 0 {
musicPrefixes = make(map[string]string, len(p.Config.Music.Datasets))
for _, ds := range p.Config.Music.Datasets {
if ds.Source != "" && ds.Prefix != "" {
musicPrefixes[ds.Source] = ds.Prefix
}
}
}
effectiveDatasets := make(map[string]EffectiveMusicDataset, len(p.Config.Music.Datasets))
for id, ds := range p.Config.Music.Datasets {
dsStageRoot := ds.StageRoot
if dsStageRoot == "" {
dsStageRoot = musicStageRoot
}
dsCreditsRoot := ds.CreditsRoot
if dsCreditsRoot == "" {
dsCreditsRoot = musicCreditsRoot
}
dsOutput := strings.TrimSpace(ds.Output)
if dsOutput == "" {
dsOutput = ds.Source
}
dsNamingScheme := defaultString(ds.NamingScheme, musicNamingScheme)
dsOutputExtension := musicOutputExtension
if ds.OutputExtension != "" {
dsOutputExtension = normalizeExtension(ds.OutputExtension)
}
dsConvertExtensions := musicConvertExtensions
if len(ds.ConvertExtensions) > 0 {
dsConvertExtensions = normalizeExtensionSlice(ds.ConvertExtensions)
}
effectiveDatasets[id] = EffectiveMusicDataset{
Source: ds.Source,
Output: dsOutput,
Prefix: ds.Prefix,
StageRoot: dsStageRoot,
CreditsRoot: dsCreditsRoot,
NamingScheme: dsNamingScheme,
OutputExtension: dsOutputExtension,
MaxStemLength: musicMaxStemLength,
PackageMode: defaultString(ds.PackageMode, "hak_asset"),
ConvertExtensions: dsConvertExtensions,
}
}
effective := EffectiveConfig{
ConfigSource: p.ConfigSource,
Module: p.Config.Module,
Paths: paths,
Outputs: outputs,
Build: p.Config.Build,
Generated: generated,
Inventory: inventory,
Validation: validation,
Music: EffectiveMusicConfig{
Prefixes: musicPrefixes,
Tools: p.Config.Music.Tools,
StageRoot: musicStageRoot,
CreditsRoot: musicCreditsRoot,
CreditsOverlay: musicCreditsOverlay,
MaxStemLength: musicMaxStemLength,
NamingScheme: musicNamingScheme,
OutputExtension: musicOutputExtension,
ConvertExtensions: musicConvertExtensions,
Datasets: effectiveDatasets,
},
TopData: top,
Extract: extract,
Autogen: EffectiveAutogenConfig{
Producers: slices.Clone(p.Config.Autogen.Producers),
Consumers: slices.Clone(p.Config.Autogen.Consumers),
Cache: EffectiveAutogenCacheConfig{
Root: expandPathTemplate(defaultString(p.Config.Autogen.Cache.Root, DefaultAutogenCacheRoot), paths),
MaxAge: defaultString(p.Config.Autogen.Cache.MaxAge, DefaultAutogenCacheMaxAge.String()),
RefreshEnv: defaultString(p.Config.Autogen.Cache.RefreshEnv, DefaultAutogenRefreshEnv),
PartsManifestRefreshEnv: DefaultPartsManifestRefreshEnv,
},
ReleaseSource: EffectiveAutogenReleaseSourceConfig{
Provider: defaultString(p.Config.Autogen.ReleaseSource.Provider, DefaultAutogenReleaseProvider),
ServerURLEnv: defaultString(p.Config.Autogen.ReleaseSource.ServerURLEnv, DefaultAutogenServerURLEnv),
RepoEnv: defaultString(p.Config.Autogen.ReleaseSource.RepoEnv, DefaultAutogenRepoEnv),
},
},
HAKs: slices.Clone(p.Config.HAKs),
Provenance: provenance,
}
effective.Overrides = activeOverrides(effective)
return effective
}
func cloneTopDataValueEncodings(values []TopDataValueEncodingConfig) []TopDataValueEncodingConfig {
if len(values) == 0 {
return nil
}
out := make([]TopDataValueEncodingConfig, len(values))
copy(out, values)
return out
}
func cloneTopDataValueDefaults(values []TopDataValueDefaultConfig) []TopDataValueDefaultConfig {
if len(values) == 0 {
return nil
}
out := make([]TopDataValueDefaultConfig, len(values))
copy(out, values)
return out
}
func cloneTopDataRowGeneration(values []TopDataRowGenerationConfig) []TopDataRowGenerationConfig {
if len(values) == 0 {
return nil
}
out := make([]TopDataRowGenerationConfig, len(values))
copy(out, values)
for index := range out {
if strings.TrimSpace(out[index].Dataset) == "" {
out[index].Dataset = strings.TrimSpace(out[index].Namespace)
}
if strings.TrimSpace(out[index].Namespace) == "" {
out[index].Namespace = strings.TrimSpace(out[index].Dataset)
}
if strings.TrimSpace(out[index].Mode) == "" {
out[index].Mode = "after_base"
}
out[index].Dataset = filepath.ToSlash(strings.TrimSpace(out[index].Dataset))
out[index].Namespace = filepath.ToSlash(strings.TrimSpace(out[index].Namespace))
out[index].Mode = strings.TrimSpace(out[index].Mode)
}
return out
}
func cloneTopDataRowExtensions(values []TopDataRowExtensionConfig) []TopDataRowExtensionConfig {
if len(values) == 0 {
return nil
}
out := make([]TopDataRowExtensionConfig, len(values))
copy(out, values)
for index := range out {
out[index].Dataset = filepath.ToSlash(strings.TrimSpace(out[index].Dataset))
out[index].Mode = strings.TrimSpace(out[index].Mode)
out[index].LevelColumn = strings.TrimSpace(out[index].LevelColumn)
}
return out
}
func cloneTopDataClassFeatInjections(value TopDataClassFeatInjectionConfig) TopDataClassFeatInjectionConfig {
out := TopDataClassFeatInjectionConfig{
GlobalFeats: slices.Clone(value.GlobalFeats),
ClassSkillMasterfeats: slices.Clone(value.ClassSkillMasterfeats),
}
for index := range out.GlobalFeats {
out.GlobalFeats[index].RequirePresent = slices.Clone(value.GlobalFeats[index].RequirePresent)
out.GlobalFeats[index].UnlessPresent = slices.Clone(value.GlobalFeats[index].UnlessPresent)
}
return out
}
func (p *Project) EffectiveConfigJSON() ([]byte, error) {
return json.MarshalIndent(p.EffectiveConfig(), "", " ")
}
func (p *Project) ExplainConfig(key string) (any, ConfigValueProvenance, bool) {
effective := p.EffectiveConfig()
value, ok := lookupEffectiveValue(effective, key)
if !ok {
return nil, ConfigValueProvenance{}, false
}
prov := effective.Provenance[key]
if prov.Source == "" {
prov = ConfigValueProvenance{Source: "yaml", Detail: p.ConfigSource.Name}
}
return value, prov, true
}
func lookupEffectiveValue(effective EffectiveConfig, key string) (any, bool) {
raw, err := json.Marshal(effective)
if err != nil {
return nil, false
}
var data any
if err := json.Unmarshal(raw, &data); err != nil {
return nil, false
}
current := data
for _, part := range strings.Split(key, ".") {
object, ok := current.(map[string]any)
if !ok {
return nil, false
}
current, ok = object[part]
if !ok {
return nil, false
}
}
return current, true
}
func markMissingDefaults(provenance ConfigProvenance) {
defaults := map[string]string{
"paths.build": "build",
"paths.cache": ".cache",
"paths.tools": "tools",
"outputs.module_archive": DefaultModuleArchiveTemplate,
"outputs.hak_manifest": DefaultHAKManifest,
"outputs.hak_archive": DefaultHAKArchiveTemplate,
"generated_assets.topdata_2da": "no generated topdata 2DA assets",
"inventory.source_extensions": "NWN source resource extensions",
"inventory.asset_extensions": "NWN asset resource extensions",
"inventory.source_json_pattern": DefaultSourceJSONPattern,
"validation.profile": DefaultValidationProfile,
"validation.builtin_script_prefixes": "NWN built-in script prefixes",
"validation.required_fields": "NWN required GFF fields",
"music.stage_root": DefaultMusicStageRoot,
"music.credits_root": DefaultCreditsRoot,
"music.credits_overlay": DefaultCreditsOverlayFile,
"music.max_stem_length": "16",
"music.naming_scheme": DefaultMusicNamingScheme,
"music.output_extension": DefaultMusicOutputExtension,
"music.convert_extensions": strings.Join(music.DefaultConvertExtensions(), ", "),
"topdata.build": DefaultTopDataBuild,
"topdata.compiled_2da_dir": DefaultTopDataCompiled2DADir,
"topdata.compiled_tlk": DefaultTopDataCompiledTLK,
"topdata.package_hak": DefaultTopDataPackageHAK,
"topdata.package_tlk": DefaultTopDataPackageTLK,
"topdata.wiki.output_root": DefaultTopDataWikiOutputRoot,
"topdata.wiki.pages_dir": DefaultTopDataWikiPagesDir,
"topdata.wiki.state_file": DefaultTopDataWikiStateFile,
"topdata.wiki.managed_namespaces": "NWN topdata wiki namespaces",
"topdata.wiki.deploy_manifest": DefaultWikiDeployManifest,
"topdata.wiki.deploy_edit_summary": DefaultWikiDeployEditSummary,
"topdata.wiki.source": DefaultTopDataWikiSource,
"topdata.wiki.renderer": DefaultTopDataWikiRenderer,
"topdata.wiki.link_strategy": DefaultTopDataWikiLinkStrategy,
"topdata.wiki.namespaces_file": DefaultTopDataWikiNamespacesFile,
"topdata.wiki.tables_file": DefaultTopDataWikiTablesFile,
"topdata.wiki.visibility_file": DefaultTopDataWikiVisibilityFile,
"topdata.wiki.data_file": DefaultTopDataWikiDataFile,
"topdata.wiki.page_paths_file": DefaultTopDataWikiPagePathsFile,
"topdata.wiki.templates_dir": DefaultTopDataWikiTemplatesDir,
"topdata.wiki.page_templates_dir": DefaultTopDataWikiPageTemplatesDir,
"topdata.wiki.manual_sections_dir": DefaultTopDataWikiManualSectionsDir,
"topdata.wiki.manual_sections_file": DefaultTopDataWikiManualSectionsFile,
"topdata.wiki.page_index_file": DefaultTopDataWikiPageIndexFile,
"topdata.wiki.title_prefix_min_length": "3",
"topdata.wiki.status_pages": "true",
"topdata.wiki.status_listing_scope": DefaultTopDataWikiStatusListingScope,
"topdata.wiki.alignment_links.link_format": DefaultTopDataWikiAlignmentLinkFormat,
"topdata.wiki.stale_pages.default": DefaultTopDataWikiStaleDefault,
"topdata.wiki.stale_pages.live_cleanup": DefaultTopDataWikiStaleLiveCleanup,
"topdata.wiki.managed_region.marker_format": DefaultTopDataWikiMarkerFormat,
"topdata.wiki.managed_region.page_marker_prefix": DefaultTopDataWikiPageMarkerPrefix,
"extract.layout": DefaultExtractLayout,
"extract.hak_discovery": DefaultExtractHAKDiscovery,
"extract.archives": DefaultModuleArchiveTemplate,
"extract.cleanup_stale": "true",
"extract.consume_archives": "false",
"autogen.cache.root": DefaultAutogenCacheRoot,
"autogen.cache.max_age": DefaultAutogenCacheMaxAge.String(),
"autogen.cache.refresh_env": DefaultAutogenRefreshEnv,
"autogen.cache.parts_manifest_refresh_env": DefaultPartsManifestRefreshEnv,
"autogen.release_source.provider": DefaultAutogenReleaseProvider,
"autogen.release_source.server_url_env": DefaultAutogenServerURLEnv,
"autogen.release_source.repo_env": DefaultAutogenRepoEnv,
}
for key, detail := range defaults {
if _, ok := provenance[key]; !ok {
provenance[key] = ConfigValueProvenance{Source: "toolkit default", Detail: detail}
}
}
}
func cloneProvenance(input ConfigProvenance) ConfigProvenance {
out := ConfigProvenance{}
for key, value := range input {
out[key] = value
}
return out
}
func defaultString(value, fallback string) string {
trimmed := strings.TrimSpace(value)
if trimmed == "" {
return fallback
}
return trimmed
}
func defaultInt(value, fallback int) int {
if value == 0 {
return fallback
}
return value
}
func defaultBoolPointer(value *bool, fallback bool) *bool {
out := fallback
if value != nil {
out = *value
}
return &out
}
func defaultStringSlice(value, fallback []string) []string {
if len(value) == 0 {
return slices.Clone(fallback)
}
return slices.Clone(value)
}
func defaultWikiPageTemplatesDir(cfg TopDataWikiConfig) string {
if strings.TrimSpace(cfg.PageTemplatesDir) != "" {
return strings.TrimSpace(cfg.PageTemplatesDir)
}
if strings.TrimSpace(cfg.TemplatesDir) != "" {
return filepath.ToSlash(filepath.Join(filepath.FromSlash(strings.TrimSpace(cfg.TemplatesDir)), "pages"))
}
return DefaultTopDataWikiPageTemplatesDir
}
func defaultWikiManualSectionsFile(cfg TopDataWikiConfig) string {
if strings.TrimSpace(cfg.ManualSectionsFile) != "" {
return strings.TrimSpace(cfg.ManualSectionsFile)
}
if strings.TrimSpace(cfg.ManualSectionsDir) != "" {
return filepath.ToSlash(filepath.Join(filepath.FromSlash(strings.TrimSpace(cfg.ManualSectionsDir)), "default.yaml"))
}
return DefaultTopDataWikiManualSectionsFile
}
func defaultRequiredFields(configured map[string][]string) map[string][]string {
if len(configured) > 0 {
return cloneStringSliceMap(normalizeRequiredFields(configured))
}
return DefaultRequiredFields()
}
func cloneStringSliceMap(input map[string][]string) map[string][]string {
out := make(map[string][]string, len(input))
for key, values := range input {
out[key] = slices.Clone(values)
}
return out
}
func cloneStringMap(input map[string]string) map[string]string {
if len(input) == 0 {
return nil
}
out := make(map[string]string, len(input))
for key, value := range input {
out[key] = value
}
return out
}
func expandPathTemplate(template string, paths EffectivePathConfig) string {
replacer := strings.NewReplacer(
"{paths.source}", paths.Source,
"{paths.assets}", paths.Assets,
"{paths.build}", paths.Build,
"{paths.cache}", paths.Cache,
"{paths.tools}", paths.Tools,
)
return filepath.ToSlash(replacer.Replace(template))
}
func expandPathTemplates(templates []string, paths EffectivePathConfig) []string {
out := make([]string, 0, len(templates))
for _, template := range templates {
out = append(out, expandPathTemplate(template, paths))
}
return out
}
func (p *Project) ActiveOverrides() []ConfigOverride {
return activeOverrides(p.EffectiveConfig())
}
func activeOverrides(effective EffectiveConfig) []ConfigOverride {
envs := map[string]string{
"build.keep_existing_haks": effectiveBuildKeepExistingEnv,
"autogen.cache.refresh": effective.Autogen.Cache.RefreshEnv,
"autogen.cache.parts_manifest_refresh": effective.Autogen.Cache.PartsManifestRefreshEnv,
"autogen.release_source.server_url": effective.Autogen.ReleaseSource.ServerURLEnv,
"autogen.release_source.repo": effective.Autogen.ReleaseSource.RepoEnv,
"music.ffmpeg": "SOW_FFMPEG",
"music.ffprobe": "SOW_FFPROBE",
"wiki.endpoint": "NODEBB_API_ENDPOINT",
"wiki.token": "NODEBB_API_TOKEN",
"wiki.categories": "NODEBB_WIKI_CATEGORIES",
"release.version": "GITHUB_REF_NAME",
}
keys := make([]string, 0, len(envs))
for key := range envs {
keys = append(keys, key)
}
slices.Sort(keys)
overrides := make([]ConfigOverride, 0)
for _, key := range keys {
env := envs[key]
if value := strings.TrimSpace(os.Getenv(env)); value != "" {
if isSensitiveOverrideKey(key) || isSensitiveOverrideKey(env) {
value = "<set>"
}
overrides = append(overrides, ConfigOverride{Key: key, Value: value, Source: "env:" + env})
}
}
return overrides
}
const effectiveBuildKeepExistingEnv = "SOW_BUILD_HAKS_KEEP_EXISTING"
func isSensitiveOverrideKey(key string) bool {
lower := strings.ToLower(key)
return strings.Contains(lower, "token") || strings.Contains(lower, "password")
}
func (o EffectiveOutputConfig) ModuleArchiveName(module ModuleConfig) string {
return strings.NewReplacer("{module.resref}", strings.TrimSpace(module.ResRef)).Replace(o.ModuleArchive)
}
func (o EffectiveOutputConfig) HAKArchiveName(name string) string {
return strings.NewReplacer("{hak.name}", strings.TrimSpace(name)).Replace(o.HAKArchive)
}
func (p ConfigValueProvenance) String() string {
if p.Detail == "" {
return p.Source
}
return fmt.Sprintf("%s (%s)", p.Source, p.Detail)
}
internal/project/project.go
+2158
View File
File diff suppressed because it is too large Load Diff
internal/project/project_test.go
+1830
View File
File diff suppressed because it is too large Load Diff
internal/topdata/AUTO-INCLUDE_EXISTING_PART_MODELS_IN_2DA_GENERATION_CONTRACT.md
+155
View File
@@ -0,0 +1,155 @@
# AI Agent Contract: Auto-Include Existing Part Models in 2DA Generation
## Status Snapshot
Current state as of 2026-05-17:
- Superseded for current Shadows Over Westgate builds.
- The module no longer consumes a released parts manifest. Parts data is owned by
`assets/`, generated as HAK-side 2DA assets from `assets/topdata/data/parts`
plus a direct scan of `assets/content/part/**/*.mdl`, and packaged in
`sow_part`.
- This file remains as historical context for the legacy manifest-backed
consumer path.
## Objective
The native 2DA generation pipeline must automatically include existing part models already present in the **`sow-assets` repository** when building final parts tables.
This behavior is required for parity with the current asset set and must not depend on manually authored entries for models that already exist in repository assets.
---
## Source of Truth
The superseded manifest-backed native builder obtained existing model
information from a released artifact published by the **`sow-assets` repository**.
Resolution rules:
- if `topdata.assets` is configured to a local path, use that exact path
- otherwise derive the **`sow-assets` repository** from the module repo's Gitea origin
and fetch only the configured released manifest
Normal builds must not clone or scan the `sow-assets` repo tree. They may cache the
downloaded manifest locally for reuse, but the source of truth remains the released
manifest on Gitea.
Manifest-backed discovery is derived from:
```text
sow-assets/assets/part/**/*.mdl
```
The manifest generator may also accept the repo's legacy `assets/parts/` layout when
resolving the physical paths, but the native builder itself must consume the manifest,
not the repo tree.
---
## Required Scope
Process existing `.mdl` files for these parts table categories only:
- `belt`
- `bicep`
- `chest`
- `foot`
- `forearm`
- `leg`
- `neck`
- `pelvis`
- `robe`
- `shin`
- `shoulder`
Dataset mapping rules:
- `parts/legs` maps to asset category `leg`
- `parts/hand` remains unsupported and is not auto-generated
The scan must recurse through subdirectories under each applicable category path.
---
## Required Behavior
### 1. Manifest-backed discovery
For each supported part category, read the released manifest and discover all existing
numeric model IDs listed for that category.
### 2. Trailing numeric suffix extraction
The released manifest must be generated using trailing-numeric-suffix extraction from
the source `.mdl` filenames. Native topdata consumes the resulting numeric IDs only.
### 3. Row generation
If a trailing numeric suffix is present:
- use that numeric suffix as the output row ID
- emit a row for that part model in the final generated 2DA
### 4. Default emitted values
For rows generated from discovered existing models, set:
- `COSTMODIFIER = 0`
- `ACBONUS = 0.00`
Rows without discovered existing models may be intentionally emitted as dense
2DA null rows when configured. In that mode, missing rows must have
`COSTMODIFIER = ****` and `ACBONUS = ****` in the final generated 2DA.
### 5. Override precedence
If file-based overrides exist, they must take precedence over discovered defaults.
That means:
- manifest discovery provides the baseline row presence and default values
- override data may replace or augment those values
- override data wins in any conflict
---
## Non-Negotiable Requirements
- The implementation must read model presence from the **released parts manifest**,
not from assumption, hardcoded lists, or legacy output.
- Existing repository models with trailing numeric suffixes must be reflected in generated output even if they are not otherwise explicitly authored.
- Override application is mandatory and must win over auto-discovered defaults.
- This is replacement behavior for the native pipeline, not an optional enhancement.
---
## Acceptance Criteria
The superseded implementation was correct only if all of the following were true:
1. The builder fetches the configured released manifest artifact for part models.
2. Only the supported part categories are considered.
3. Manifest IDs derived from trailing numeric suffixes generate corresponding 2DA rows.
4. The numeric suffix is used as the row ID.
5. Auto-generated rows default to:
- `COSTMODIFIER = 0`
- `ACBONUS = 0.00`
6. Authored base rows remain exposed. Authored `ACBONUS = ****` rows remain
null unless model discovery activates the row.
7. Dense gaps without authored rows or discovered models are emitted as null
rows with `COSTMODIFIER = ****` and `ACBONUS = ****`.
8. File overrides are applied after discovery and take precedence.
9. The final output includes all eligible existing models present in the manifest.
10. Discovery does not require a sibling local `sow-assets` checkout or a Git clone.
---
## Implementation Directive
When generating native parts 2DAs, first scan the project-resolved assets repo for
supported part models, collect existing supported part models by category, extract
trailing numeric suffixes from filenames, emit rows using those suffixes as row IDs
with default `COSTMODIFIER = 0` and `ACBONUS = 0.00`, then apply overrides with
override values taking precedence.
internal/topdata/CLASS_FEAT_GLOBAL_INJECTS_CONTRACT.md
+76
View File
@@ -0,0 +1,76 @@
# Class Feat Global Injection Contract
## Status Snapshot
Current state as of 2026-06-05:
- Preferred class feat injection policy is authored in
`topdata/data/classes/feats/global.json`.
- `global.json` rows flow through the ordinary class feat expansion pipeline,
including successor expansion, masterfeat expansion, class-skill filtering,
reference resolution, deduplication, and final 2DA emission.
- Legacy `topdata.class_feat_injections` YAML still loads only as a
compatibility path when no applicable class feat `global.json` exists.
- Toolkit hardcoded defaults are compatibility-only and must not be the normal
source of project policy.
## Objective
Native class feat generation must produce deterministic, legacy-equivalent
effective output while keeping project policy in topdata authoring files.
## Requirements
- For each class skill, inject the configured skill-focus masterfeat rows from
`classes/feats/global.json` using `FeatIndex.filter = "classskills"`.
- Inject `feat:literate` only when the class table does not already include
`feat:illiterate`.
- Inject the shared combat/menu feat rows declared in the manifest unless a
class already authors the equivalent `FeatIndex`.
- Explicit class feat rows take precedence over injected rows.
- Injections must be deterministic for identical inputs.
## Manifest Shape
```json
{
"position": "prepend",
"injections": [
{
"row": {
"FeatIndex": { "id": "feat:literate" },
"GrantedOnLevel": 1,
"List": 3,
"OnMenu": 0
},
"unless_present": [
{ "field": "FeatIndex", "id": "feat:illiterate" }
]
},
{
"row": {
"FeatIndex": {
"id": "masterfeats:skill_focus",
"filter": "classskills"
},
"GrantedOnLevel": -1,
"List": 1,
"OnMenu": 0
}
}
]
}
```
`position` defaults to `append`. The module uses `prepend` for class feat
injections to keep generated policy rows before per-class authored rows.
## Acceptance Criteria
- Class feat output contains the same effective shared feats as the prior YAML
policy.
- Literacy injection is skipped for illiterate classes.
- Skill-focus masterfeat rows expand only for class skills.
- YAML compatibility continues to work for consumers that have not migrated.
- A project with `classes/feats/global.json` does not also receive YAML or
hardcoded default class feat injections.
internal/topdata/FAMILY_EXPANSION_CONTRACT.md
+119
View File
@@ -0,0 +1,119 @@
# Generic Family Expansion Contract
## Status Snapshot
Current state as of 2026-05-13:
- Implemented and actively used by native feat generation.
- `family_expansion.go`, `native.go`, and `topdata_test.go` cover the current
underscore-family interpretation, generated family metadata, and reapplication
behavior.
- Remaining work is mostly consumer expansion: use the same primitive in more
datasets only when the authored data actually benefits from it.
Topdata now treats underscore expansion as a structural rule:
- `parent_child` means `child` is an expansion of `parent`
- underscores are for family structure, not simulated spaces
- existing canonical keys still win when lock or TLK state already established them
This is a global interpretation rule, not a wiki-only convention.
## Identity
- `weaponspecialization_club`
- parent: `weaponspecialization`
- child: `club`
- `gnome_rock`
- parent: `gnome`
- child: `rock`
- `toughness_10`
- parent: `toughness`
- child: `10`
Standalone keys without an underscore are treated as a parent identity with no child.
## Generated Family Files
Canonical generated families declare:
```json
{
"family": "weapon_focus",
"family_key": "weaponfocus",
"template": "masterfeats:weaponfocus",
"name_prefix": "Weapon Focus",
"label_prefix": "FEAT_WEAPON_FOCUS",
"constant_prefix": "FEAT_WEAPON_FOCUS",
"legacy_family_keys": ["epic_weapon_focus"],
"default_fields": {
"DESCRIPTION": {
"tlk": {
"key": "masterfeats:weaponfocus.description",
"text": "..."
}
},
"MINATTACKBONUS": "1",
"TOOLSCATEGORIES": "1"
},
"apply_after_modules": true,
"identity_source": "child_source_value",
"auto_prereq_fields": {
"PREREQFEAT1": "weaponfocus"
},
"child_source": {
"dataset": "baseitems",
"column": "WeaponFocusFeat"
}
}
```
Rules:
- `family` is an authored label and is not a hardcoded switch key
- `family_key` is the structural parent identity used in generated child keys
- `template` is optional for the primitive in general, but required by current
masterfeat-backed `feat` families
- `name_prefix`, `label_prefix`, and `constant_prefix` define how new child rows are
named when they are not already authored
- `template_fields` lists template-backed fields copied from the referenced template row
when authored; it is optional if the family owns its shared values directly
- `default_fields` applies authored shared/default values to every generated family row
and can be used to move family-wide shared values out of the template row
- `apply_after_modules: true` re-applies the generated family after normal module files,
so authored family-shared values remain authoritative even for legacy explicitly
authored child rows
- `child_ref_field` binds the generated child row back to the source row key, e.g.
`REQSKILL`
- `identity_source: "child_source_value"` tells the builder to reuse IDs/keys from the
source column when that column already carries feat identity
- `legacy_family_keys` declares previous generated-family prefixes that should donate
existing locked row IDs to the current family when a family is renamed
- `auto_prereq_fields` binds prerequisite fields to other authored families by
`family_key`, without code changes
- `allow_existing_only` limits expansion to already-authored child rows when a family is
intentionally partial
- `child_source.dataset` identifies the canonical dataset driving expansion
- `child_source.column` is used when expansion is gated by a non-null source field
- `child_source.predicate` is used when expansion depends on a named rule such as
accessibility
## Row Metadata
Generated rows can carry:
```json
{
"meta": {
"family": {
"parent": "weaponfocus",
"child": "club",
"source": "baseitems:club",
"template": "masterfeats:weaponfocus"
}
}
}
```
This metadata is builder-owned and does not affect emitted 2DA columns. It preserves
family structure for later phases without enabling wiki generation yet.
internal/topdata/FEAT_GENERATED_FAMILIES_CONTRACT.md
+174
View File
@@ -0,0 +1,174 @@
# `feat` Generated Families Contract
## Status Snapshot
Current state as of 2026-05-13:
- Implemented for the current native `feat` pipeline.
- Native tests cover generated-family expansion, family-owned shared defaults,
masterfeat-backed inheritance, and lock/TLK identity preservation.
- The one explicitly deferred item in this contract is still current:
`feat.2da` remains native-built but excluded from `compare-topdata`
reference-catalog coverage through `compare_reference: false`.
Canonical authored paths:
- `topdata/data/feat/base.json`
- `topdata/data/feat/lock.json`
- `topdata/data/feat/generated/skill_focus.json`
- `topdata/data/feat/generated/greater_skill_focus.json`
- `topdata/data/feat/generated/weapon_focus.json`
- `topdata/data/feat/generated/weapon_specialization.json`
- `topdata/data/feat/generated/greater_weapon_focus.json`
- `topdata/data/feat/generated/greater_weapon_specialization.json`
- `topdata/data/feat/generated/improved_critical.json`
- `topdata/data/feat/generated/overwhelming_critical.json`
- `topdata/data/feat/modules/activecombat/core.json`
- `topdata/data/feat/modules/activecombat/specialattacks.json`
- `topdata/data/feat/modules/class/core.json`
- `topdata/data/feat/modules/combat/core.json`
- `topdata/data/feat/modules/defensive/core.json`
- `topdata/data/feat/modules/magical/core.json`
- `topdata/data/feat/modules/other/core.json`
- `topdata/data/feat/modules/proficiency/core.json`
- `topdata/data/feat/modules/racial/core.json`
- `topdata/data/feat/modules/removedandhidden/core.json`
- `topdata/data/feat/modules/skillfeat/affinity.json`
- `topdata/data/feat/modules/skillfeat/focus.json`
- `topdata/data/feat/modules/skillfeat/greaterfocus.json`
- `topdata/data/feat/modules/skillfeat/other.json`
`skill_affinity` is no longer authored as a generated feat file. It is synthesized from
canonical `racialtypes` race grants.
Compact family-expansion shape:
```json
{
"family": "skill_focus",
"family_key": "skillfocus",
"template": "masterfeats:skillfocus",
"name_prefix": "Skill Focus",
"label_prefix": "FEAT_SKILL_FOCUS",
"constant_prefix": "FEAT_SKILL_FOCUS",
"default_fields": {
"DESCRIPTION": {
"tlk": {
"key": "masterfeats:skillfocus.description",
"text": "..."
}
},
"CRValue": "0.5",
"ReqSkillMinRanks": "1",
"TOOLSCATEGORIES": "6"
},
"apply_after_modules": true,
"child_ref_field": "REQSKILL",
"child_source": {
"dataset": "skills",
"predicate": "accessible"
},
"title_style": {
"child_case": "lower",
"child_parenthetical": "comma"
},
"overrides": {
"skills:craft_alchemy": {
"ICON": "ife_foc_alchm"
}
}
}
```
Weapon family shape:
```json
{
"family": "weapon_focus",
"family_key": "weaponfocus",
"template": "masterfeats:weaponfocus",
"name_prefix": "Weapon Focus",
"label_prefix": "FEAT_WEAPON_FOCUS",
"constant_prefix": "FEAT_WEAPON_FOCUS",
"legacy_family_keys": ["epic_weapon_focus"],
"default_fields": {
"DESCRIPTION": {
"tlk": {
"key": "masterfeats:weaponfocus.description",
"text": "..."
}
},
"MINATTACKBONUS": "1",
"TOOLSCATEGORIES": "1"
},
"apply_after_modules": true,
"identity_source": "child_source_value",
"child_source": {
"dataset": "baseitems",
"column": "WeaponFocusFeat"
},
"title_style": {
"child_case": "lower",
"child_parenthetical": "preserve"
},
"overrides": {
"baseitems:heavymace": {
"ICON": "ife_wepfoc_Lma"
}
}
}
```
Rules:
- any generated feat file with family-expansion fields is treated as an authored
family-expansion definition
- `family` is descriptive only; the builder does not hardcode family names
- family behavior is driven by authored fields such as `template_fields`,
`default_fields`, `apply_after_modules`, `identity_source`, `child_ref_field`,
`auto_prereq_fields`, and optional `title_style`
- `default_fields` is a family-wide shared layer; it applies to existing and newly
generated child rows before per-child overrides are merged
- `apply_after_modules: true` makes the generated family the final authoritative shared
layer for legacy explicitly authored child rows, so shared baselines can live in the
generated family file instead of `masterfeats` overrides
- `legacy_family_keys` declares old generated-family prefixes whose existing row IDs
should be inherited by the current family; this is how renamed families such as
`greater_skill_focus` retaining vanilla `epic_skill_focus` rows preserve hardcoded
engine behavior without hardcoding the rename in toolkit logic
- family-expansion files must declare `template`, `family_key`, and `child_source.dataset`
- `child_source.column` is used when expansion is gated by a non-null source field
- `child_source.predicate: "accessible"` currently means `HideFromLevelUp != 1`
- `title_style` only changes generated child display text while composing titles;
generated feat keys, TLK keys, row IDs, family metadata, and source references
remain tied to generated family identity
- generated family title text is authoritative for family-managed existing rows
as well as newly allocated rows, so in-game TLK names and generated wiki titles
are derived from the same normalized `FEAT` text
- `title_style.child_case` defaults to `preserve`; `lower` lowercases the resolved
child display text after parenthetical formatting
- `title_style.child_parenthetical` defaults to `preserve`; `comma` flattens one
child parenthetical suffix such as `Knowledge (local)` into
`Knowledge, local` before the child text is wrapped by the family title
- unsupported `title_style` values fail generated-family validation instead of
silently changing title behavior
- skill focus families intentionally do not set `allow_existing_only`, so any new
accessible skill automatically receives matching focus rows
- compact `overrides` are keyed by source dataset key:
- `skills:*` for skill families
- `baseitems:*` for weapon families
- generated rows carry `meta.family` so the underscore family rule is preserved even
before wiki generation exists
- emitted feat keys must prefer existing canonical lock/TLK identities over newly
derived child spellings
- manual and irregular feat content continues to live under `feat/modules/`
- generated families and module-authored feats are merged into the same final `feat.2da`
pipeline and share inheritance, override, TLK, and family-metadata behavior
Current rollout policy:
- `feat.2da` still builds natively
- `feat.2da` is still excluded from `compare-topdata` output-catalog self-check coverage
via `compare_reference: false`
- manual and irregular feat families remain authored directly in modules rather than
compact generated-family files
internal/topdata/INHERITANCE_CONTRACT.md
+78
View File
@@ -0,0 +1,78 @@
# Topdata Inheritance Contract
## Status Snapshot
Current state as of 2026-05-13:
- Implemented.
- `native.go` supports both row-sugar inheritance and generic object
inheritance, including cycle detection and missing-target failures.
- Validation and build regression coverage live in `topdata_test.go`.
Native topdata now supports two inheritance forms:
## 1. Row Sugar Compatibility
This is the existing narrow row helper:
```json
{
"PARENT": { "id": "custom:parent" },
"inherit": {
"from": "PARENT",
"fields": ["VALUE", "ICON"]
}
}
```
It remains supported for row-local field copying.
## 2. Generic Object Inheritance
This is the new reusable primitive:
```json
{
"inherit": {
"ref": "masterfeats:skillfocus"
},
"LABEL": "FEAT_SKILL_FOCUS_ATHLETICS"
}
```
Or for a nested object:
```json
{
"DETAILS": {
"inherit": {
"ref": "custom:parent",
"field": "DETAILS"
},
"meta": {
"cost": "9"
}
}
}
```
## Rules
- `inherit.ref` must use stable `dataset:key` identity.
- `inherit.field` is optional and selects an object-valued field on the target row.
- Generic inheritance can appear on any authored object, not only top-level rows.
- Merge precedence is `local overrides inherited`.
- Scalars replace inherited values.
- Arrays replace inherited values.
- Objects merge recursively unless the local object is an atomic topdata value object such as:
- TLK payloads
- row refs
- table refs
- Cycles fail the build.
- Missing targets fail the build.
## Current Intended Consumer
`feat` is the first dataset expected to use this primitive heavily, especially for
borrowing shared properties from `masterfeats` while keeping `feat`'s own dataset
contract and row model.
internal/topdata/MASTERFEATS_CONTRACT.md
+82
View File
@@ -0,0 +1,82 @@
# `masterfeats` Canonical Contract
## Status Snapshot
Current state as of 2026-05-13:
- Implemented as a first-class native dataset.
- The repository contains `topdata/data/masterfeats/`, migration/import support,
validator checks, and native output coverage for `masterfeats.2da`.
- Remaining work is only downstream adoption and regression maintenance as other
native datasets continue to consume `masterfeats` more heavily.
`masterfeats` is a stable native canonical family consumed directly by `feat`
generation, class-feat expansion, and inheritance/ref resolution.
Current canonical scope:
- dataset root: `topdata/data/masterfeats/`
- required files:
- `base.json`
- `lock.json`
- generated output:
- `masterfeats.2da`
## `base.json`
Canonical shape:
```json
{
"output": "masterfeats.2da",
"columns": ["LABEL", "STRREF", "DESCRIPTION", "..."],
"rows": [
{
"id": 0,
"key": "masterfeats:weaponfocus",
"LABEL": "WeaponFocus",
"STRREF": "6490",
"DESCRIPTION": "436"
}
]
}
```
Rules:
- `output` may be omitted; native dataset discovery deterministically derives
`masterfeats.2da` from `topdata/data/masterfeats/base.json`
- if `output` is present, it must be exactly `masterfeats.2da`
- `columns` must include `LABEL`, `STRREF`, and `DESCRIPTION`
- every canonical row must have a non-empty `key`
- row keys must start with `masterfeats:`
- inline TLK payloads are allowed in `STRREF` and `DESCRIPTION`
- numeric/text references are both allowed where the native TLK compiler already supports them
## `lock.json`
Canonical shape:
```json
{
"masterfeats:weaponfocus": 0
}
```
Rules:
- every key must start with `masterfeats:`
- every value must be numeric
- the lockfile remains the canonical stable id source for authored keys
## Module contributions
This family continues to support the shared canonical module shapes already used by native
datasets:
- `columns`
- `entries`
- `overrides`
Those shapes are validated generically in `topdata.go`; this contract only adds the
family-specific guarantees for `masterfeats`.
internal/topdata/appearance_migrate.go
+96
View File
@@ -0,0 +1,96 @@
package topdata
import (
"encoding/json"
"os"
"path/filepath"
)
func importLegacyAppearance(referenceBuilderDir, dataDir string, legacyTLK *legacyTLKData) (int, error) {
_ = legacyTLK
legacyDir := filepath.Join(referenceBuilderDir, "data", "appearance")
if _, err := os.Stat(legacyDir); err != nil {
if os.IsNotExist(err) {
return 0, nil
}
return 0, err
}
targetDir := filepath.Join(dataDir, "appearance")
targetBasePath := filepath.Join(targetDir, "base.json")
targetLockPath := filepath.Join(targetDir, "lock.json")
targetModulesDir := filepath.Join(targetDir, "modules")
moduleNames := []string{
"cotblreaver.json",
"crawlingclaw.json",
"halfogre.json",
"zombieknight.json",
}
targetModulePaths := make([]string, 0, len(moduleNames))
for _, name := range moduleNames {
targetModulePaths = append(targetModulePaths, filepath.Join(targetModulesDir, name))
}
if fileExists(targetBasePath) && fileExists(targetLockPath) {
allModulesPresent := true
for _, path := range targetModulePaths {
if !fileExists(path) {
allModulesPresent = false
break
}
}
if allModulesPresent {
return 0, nil
}
}
baseObj, err := loadJSONObject(filepath.Join(legacyDir, "base.json"))
if err != nil {
return 0, err
}
baseObj["output"] = "appearance.2da"
lockObj, err := loadJSONObject(filepath.Join(legacyDir, "lock.json"))
if err != nil {
return 0, err
}
moduleObjs := make([]map[string]any, 0, len(moduleNames))
for _, name := range moduleNames {
obj, err := loadJSONObject(filepath.Join(legacyDir, "modules", name))
if err != nil {
return 0, err
}
moduleObjs = append(moduleObjs, obj)
}
if err := os.MkdirAll(targetModulesDir, 0o755); err != nil {
return 0, err
}
writes := []struct {
path string
obj map[string]any
}{
{path: targetBasePath, obj: baseObj},
{path: targetLockPath, obj: lockObj},
}
for i, path := range targetModulePaths {
writes = append(writes, struct {
path string
obj map[string]any
}{path: path, obj: moduleObjs[i]})
}
for _, write := range writes {
raw, err := json.MarshalIndent(write.obj, "", " ")
if err != nil {
return 0, err
}
raw = append(raw, '\n')
if err := os.WriteFile(write.path, raw, 0o644); err != nil {
return 0, err
}
}
return len(writes), nil
}
internal/topdata/armor_migrate.go
+160
View File
@@ -0,0 +1,160 @@
package topdata
import (
"encoding/json"
"os"
"path/filepath"
"sort"
)
func importLegacyArmor(referenceBuilderDir, dataDir string, legacyTLK *legacyTLKData) (int, error) {
_ = legacyTLK
legacyDir := filepath.Join(referenceBuilderDir, "data", "armor")
if _, err := os.Stat(legacyDir); err != nil {
if os.IsNotExist(err) {
return 0, nil
}
return 0, err
}
targetDir := filepath.Join(dataDir, "armor")
targetPath := filepath.Join(targetDir, "armor.json")
if _, err := os.Stat(targetPath); err == nil {
obj, err := loadJSONObject(targetPath)
if err == nil && countLegacyTLKRefsInValue(obj) == 0 {
return 0, nil
}
}
baseObj, err := loadJSONObject(filepath.Join(legacyDir, "base.json"))
if err != nil {
return 0, err
}
rows, err := mergeArmorRows(baseObj, filepath.Join(legacyDir, "modules"))
if err != nil {
return 0, err
}
if err := os.MkdirAll(targetDir, 0o755); err != nil {
return 0, err
}
if err := removePathIfExists(filepath.Join(targetDir, "base.json")); err != nil {
return 0, err
}
if err := removePathIfExists(filepath.Join(targetDir, "lock.json")); err != nil {
return 0, err
}
if err := removePathIfExists(filepath.Join(targetDir, "modules")); err != nil {
return 0, err
}
out := map[string]any{
"output": "armor.2da",
"columns": baseObj["columns"],
"rows": rows,
}
raw, err := json.MarshalIndent(out, "", " ")
if err != nil {
return 0, err
}
raw = append(raw, '\n')
if err := os.WriteFile(targetPath, raw, 0o644); err != nil {
return 0, err
}
return 1, nil
}
func mergeArmorRows(baseObj map[string]any, modulesDir string) ([]any, error) {
rawRows, ok := baseObj["rows"].([]any)
if !ok {
return nil, nil
}
rows := make([]map[string]any, 0, len(rawRows))
byID := map[int]map[string]any{}
for _, raw := range rawRows {
row, ok := deepCopyValue(raw).(map[string]any)
if !ok {
continue
}
delete(row, "key")
rawID, ok := row["id"]
if !ok {
continue
}
id, err := asInt(rawID)
if err != nil {
return nil, err
}
row["id"] = id
rows = append(rows, row)
byID[id] = row
}
modulePaths, err := collectModulePaths(modulesDir)
if err != nil {
return nil, err
}
for _, path := range modulePaths {
obj, err := loadJSONObject(path)
if err != nil {
return nil, err
}
if entries, ok := obj["entries"].(map[string]any); ok {
for _, raw := range entries {
row, ok := deepCopyValue(raw).(map[string]any)
if !ok {
continue
}
delete(row, "key")
rawID, ok := row["id"]
if !ok {
continue
}
id, err := asInt(rawID)
if err != nil {
return nil, err
}
row["id"] = id
rows = append(rows, row)
byID[id] = row
}
}
if overrides, ok := obj["overrides"].([]any); ok {
for _, raw := range overrides {
override, ok := raw.(map[string]any)
if !ok {
continue
}
rawID, ok := override["id"]
if !ok {
continue
}
id, err := asInt(rawID)
if err != nil {
return nil, err
}
row := byID[id]
if row == nil {
continue
}
for key, value := range override {
if key == "id" || key == "key" || key == "_tlk" || isMetadataField(key) {
continue
}
row[key] = deepCopyValue(value)
}
}
}
}
sort.Slice(rows, func(i, j int) bool {
return rows[i]["id"].(int) < rows[j]["id"].(int)
})
out := make([]any, 0, len(rows))
for _, row := range rows {
out = append(out, row)
}
return out, nil
}
internal/topdata/assets_repo.go
+46
View File
@@ -0,0 +1,46 @@
package topdata
import (
"fmt"
"os"
"os/exec"
"strings"
"git.westgate.pw/ShadowsOverWestgate/sow-tools/internal/project"
)
func resolvePartsAssetsOverrideDir(p *project.Project) (string, error) {
spec := strings.TrimSpace(p.Config.TopData.Assets)
if spec == "" {
return "", nil
}
if looksLikeGitRepoSpec(spec) {
return "", fmt.Errorf("topdata.assets no longer supports git repo specs for parts discovery; use a local path override or the default released manifest")
}
path := p.TopDataAssetsDir()
if path == "" {
return "", nil
}
if _, err := os.Stat(path); err != nil {
return "", fmt.Errorf("configured topdata.assets path %s is not accessible: %w", path, err)
}
return path, nil
}
func looksLikeGitRepoSpec(value string) bool {
return strings.Contains(value, "://") ||
strings.HasPrefix(value, "git@") ||
strings.HasSuffix(value, ".git")
}
func gitOutput(dir string, args ...string) (string, error) {
cmd := exec.Command("git", args...)
if dir != "" {
cmd.Dir = dir
}
output, err := cmd.CombinedOutput()
if err != nil {
return "", fmt.Errorf("%v: %s", args, strings.TrimSpace(string(output)))
}
return strings.TrimSpace(string(output)), nil
}
internal/topdata/autogen.go
+1132
View File
File diff suppressed because it is too large Load Diff
internal/topdata/base_dialog_migrate.go
+41
View File
@@ -0,0 +1,41 @@
package topdata
import (
"encoding/json"
"os"
"path/filepath"
)
func importLegacyBaseDialog(referenceBuilderDir, sourceDir string) (int, error) {
legacyPath := filepath.Join(referenceBuilderDir, "tlk", "base.json")
if _, err := os.Stat(legacyPath); err != nil {
if os.IsNotExist(err) {
return 0, nil
}
return 0, err
}
targetPath := filepath.Join(sourceDir, "base_dialog.json")
if fileExists(targetPath) {
current, err := loadJSONObject(targetPath)
if err == nil {
if entries, ok := current["entries"].(map[string]any); ok && len(entries) > 0 {
return 0, nil
}
}
}
obj, err := loadJSONObject(legacyPath)
if err != nil {
return 0, err
}
raw, err := json.MarshalIndent(obj, "", " ")
if err != nil {
return 0, err
}
raw = append(raw, '\n')
if err := os.WriteFile(targetPath, raw, 0o644); err != nil {
return 0, err
}
return 1, nil
}
internal/topdata/baseitems_migrate.go
+105
View File
@@ -0,0 +1,105 @@
package topdata
import (
"encoding/json"
"os"
"path/filepath"
)
func importLegacyBaseitems(referenceBuilderDir, dataDir string, legacyTLK *legacyTLKData) (int, error) {
_ = legacyTLK
legacyDir := filepath.Join(referenceBuilderDir, "data", "baseitems")
if _, err := os.Stat(legacyDir); err != nil {
if os.IsNotExist(err) {
return 0, nil
}
return 0, err
}
targetDir := filepath.Join(dataDir, "baseitems")
targetBasePath := filepath.Join(targetDir, "base.json")
targetLockPath := filepath.Join(targetDir, "lock.json")
targetModulesDir := filepath.Join(targetDir, "modules")
moduleNames := []string{
"blunderbuss.json",
"coin.json",
"estoc.json",
"heavymace.json",
"maulandfalchion.json",
"ovr_baseitems.json",
"ovr_cloak255.json",
"ovr_helmet255.json",
"shortspear.json",
}
targetModulePaths := make([]string, 0, len(moduleNames))
for _, name := range moduleNames {
targetModulePaths = append(targetModulePaths, filepath.Join(targetModulesDir, name))
}
if fileExists(targetBasePath) && fileExists(targetLockPath) {
allModulesPresent := true
for _, path := range targetModulePaths {
if !fileExists(path) {
allModulesPresent = false
break
}
}
if allModulesPresent {
return 0, nil
}
}
baseObj, err := loadJSONObject(filepath.Join(legacyDir, "base.json"))
if err != nil {
return 0, err
}
baseObj["output"] = "baseitems.2da"
canonicalizeWikiMetadataDocument(baseObj)
lockObj, err := loadJSONObject(filepath.Join(legacyDir, "lock.json"))
if err != nil {
return 0, err
}
moduleObjs := make([]map[string]any, 0, len(moduleNames))
for _, name := range moduleNames {
obj, err := loadJSONObject(filepath.Join(legacyDir, "modules", name))
if err != nil {
return 0, err
}
canonicalizeWikiMetadataDocument(obj)
moduleObjs = append(moduleObjs, obj)
}
if err := os.MkdirAll(targetModulesDir, 0o755); err != nil {
return 0, err
}
writes := []struct {
path string
obj map[string]any
}{
{path: targetBasePath, obj: baseObj},
{path: targetLockPath, obj: lockObj},
}
for index, path := range targetModulePaths {
writes = append(writes, struct {
path string
obj map[string]any
}{
path: path,
obj: moduleObjs[index],
})
}
for _, write := range writes {
raw, err := json.MarshalIndent(write.obj, "", " ")
if err != nil {
return 0, err
}
raw = append(raw, '\n')
if err := os.WriteFile(write.path, raw, 0o644); err != nil {
return 0, err
}
}
return len(writes), nil
}
internal/topdata/class_spellbooks.go
+373
View File
@@ -0,0 +1,373 @@
package topdata
import (
"fmt"
"path/filepath"
"slices"
"strconv"
"strings"
)
type classSpellbookSpec struct {
Path string
Key string
Class string
Levels map[int][]string
}
type classSpellbookPlan struct {
Spec classSpellbookSpec
Column string
SpellLevels map[string]int
}
func isClassSpellbookPath(path string) bool {
slashed := filepath.ToSlash(path)
return strings.Contains(slashed, "/data/classes/spellbooks/") && strings.HasSuffix(strings.ToLower(slashed), ".json")
}
func validateClassSpellbookShape(path string, obj map[string]any, report *ValidationReport) {
if _, ok := obj["class"]; !ok {
report.Diagnostics = append(report.Diagnostics, Diagnostic{Severity: SeverityError, Path: path, Message: "class spellbook requires class"})
}
if _, ok := obj["levels"]; !ok {
report.Diagnostics = append(report.Diagnostics, Diagnostic{Severity: SeverityError, Path: path, Message: "class spellbook requires levels"})
}
}
func validateClassSpellbooks(dataDir string, report *ValidationReport) {
collected, err := collectSpellbookValidationDatasets(dataDir)
if err != nil {
report.Diagnostics = append(report.Diagnostics, Diagnostic{
Severity: SeverityError,
Path: dataDir,
Message: fmt.Sprintf("validate class spellbooks: %v", err),
})
return
}
_, diagnostics := classSpellbookPlans(dataDir, collected)
report.Diagnostics = append(report.Diagnostics, diagnostics...)
if len(diagnostics) == 0 {
warnSpellModulesForManagedSpellbookColumns(dataDir, collected, report)
}
}
func collectSpellbookValidationDatasets(dataDir string) ([]nativeCollectedDataset, error) {
datasets, err := discoverNativeDatasets(dataDir)
if err != nil {
return nil, err
}
out := make([]nativeCollectedDataset, 0, 2)
for _, dataset := range datasets {
if dataset.Name != "classes/core" && dataset.Name != "spells" {
continue
}
collected, err := collectNativeDataset(dataset)
if err != nil {
return nil, err
}
out = append(out, collected)
}
return out, nil
}
func applyClassSpellbooks(sourceDir string, collected []nativeCollectedDataset) ([]nativeCollectedDataset, error) {
dataDir := filepath.Join(sourceDir, "data")
plans, diagnostics := classSpellbookPlans(dataDir, collected)
if len(diagnostics) > 0 {
return nil, fmt.Errorf("class spellbook validation failed: %s", diagnosticsTextForError(diagnostics))
}
if len(plans) == 0 {
return collected, nil
}
out := make([]nativeCollectedDataset, len(collected))
copy(out, collected)
spellsIndex := -1
for index := range out {
if out[index].Dataset.Name == "spells" {
spellsIndex = index
break
}
}
if spellsIndex == -1 {
return nil, fmt.Errorf("class spellbooks require canonical dataset %q", "spells")
}
spells := out[spellsIndex]
columns := append([]string(nil), spells.Columns...)
rows := make([]map[string]any, 0, len(spells.Rows))
for _, row := range spells.Rows {
rows = append(rows, cloneRowMap(row))
}
rowByKey := rowsByKey(rows)
for _, plan := range plans {
column := plan.Column
if _, ok := canonicalColumn(columns, column); !ok {
columns = append(columns, column)
ensureRowsExposeColumns(rows, columns)
}
for _, row := range rows {
row[column] = nullValue
}
for spellKey, level := range plan.SpellLevels {
row := rowByKey[spellKey]
if row == nil {
return nil, fmt.Errorf("%s: unknown spell %q", plan.Spec.Path, spellKey)
}
row[column] = level
}
}
spells.Columns = columns
spells.Rows = rows
out[spellsIndex] = spells
return out, nil
}
func classSpellbookPlans(dataDir string, collected []nativeCollectedDataset) ([]classSpellbookPlan, []Diagnostic) {
specs, diagnostics := loadClassSpellbookSpecs(filepath.Join(dataDir, "classes", "spellbooks"))
if len(specs) == 0 {
return nil, diagnostics
}
classes := collectedDatasetByName(collected, "classes/core")
spells := collectedDatasetByName(collected, "spells")
if classes == nil {
diagnostics = append(diagnostics, Diagnostic{Severity: SeverityError, Path: dataDir, Message: `class spellbooks require canonical dataset "classes/core"`})
return nil, diagnostics
}
if spells == nil {
diagnostics = append(diagnostics, Diagnostic{Severity: SeverityError, Path: dataDir, Message: `class spellbooks require canonical dataset "spells"`})
return nil, diagnostics
}
classRows := rowsByKey(classes.Rows)
spellRows := rowsByKey(spells.Rows)
seenColumns := map[string]string{}
plans := make([]classSpellbookPlan, 0, len(specs))
for _, spec := range specs {
classRow := classRows[spec.Class]
if classRow == nil {
diagnostics = append(diagnostics, Diagnostic{Severity: SeverityError, Path: spec.Path, Message: fmt.Sprintf("unknown class %q", spec.Class)})
}
column := ""
if classRow != nil {
column, _ = classRow["SpellTableColumn"].(string)
column = strings.TrimSpace(column)
if column == "" || column == nullValue {
diagnostics = append(diagnostics, Diagnostic{Severity: SeverityError, Path: spec.Path, Message: fmt.Sprintf("class %q has no SpellTableColumn", spec.Class)})
}
}
if column != "" && column != nullValue {
folded := strings.ToLower(column)
if previous, ok := seenColumns[folded]; ok {
diagnostics = append(diagnostics, Diagnostic{Severity: SeverityError, Path: spec.Path, Message: fmt.Sprintf("SpellTableColumn %q is already managed by %s", column, previous)})
} else {
seenColumns[folded] = spec.Path
}
}
spellLevels := map[string]int{}
for _, level := range sortedIntKeys(spec.Levels) {
for _, spellKey := range spec.Levels[level] {
if spellRows[spellKey] == nil {
diagnostics = append(diagnostics, Diagnostic{Severity: SeverityError, Path: spec.Path, Message: fmt.Sprintf("unknown spell %q", spellKey)})
continue
}
if previous, ok := spellLevels[spellKey]; ok {
diagnostics = append(diagnostics, Diagnostic{Severity: SeverityError, Path: spec.Path, Message: fmt.Sprintf("spell %q appears at both level %d and level %d", spellKey, previous, level)})
continue
}
spellLevels[spellKey] = level
}
}
if column != "" && column != nullValue {
plans = append(plans, classSpellbookPlan{Spec: spec, Column: column, SpellLevels: spellLevels})
}
}
return plans, diagnostics
}
func warnSpellModulesForManagedSpellbookColumns(dataDir string, collected []nativeCollectedDataset, report *ValidationReport) {
plans, diagnostics := classSpellbookPlans(dataDir, collected)
if len(diagnostics) > 0 || len(plans) == 0 {
return
}
managed := map[string]classSpellbookPlan{}
for _, plan := range plans {
managed[strings.ToLower(plan.Column)] = plan
}
paths, err := collectModulePaths(filepath.Join(dataDir, "spells", "modules"))
if err != nil {
report.Diagnostics = append(report.Diagnostics, Diagnostic{Severity: SeverityError, Path: dataDir, Message: fmt.Sprintf("collect spell modules for spellbook warnings: %v", err)})
return
}
for _, path := range paths {
obj, err := loadJSONObject(path)
if err != nil {
continue
}
warnSpellbookManagedColumnsInValue(dataDir, path, obj["entries"], managed, report)
warnSpellbookManagedColumnsInValue(dataDir, path, obj["overrides"], managed, report)
warnSpellbookManagedColumnsInValue(dataDir, path, obj["rows"], managed, report)
}
}
func warnSpellbookManagedColumnsInValue(dataDir, path string, value any, managed map[string]classSpellbookPlan, report *ValidationReport) {
switch typed := value.(type) {
case map[string]any:
for _, key := range sortedKeys(typed) {
row, ok := typed[key].(map[string]any)
if !ok {
continue
}
warnSpellbookManagedColumnsInRow(dataDir, path, row, managed, report)
}
case []any:
for _, raw := range typed {
row, ok := raw.(map[string]any)
if !ok {
continue
}
warnSpellbookManagedColumnsInRow(dataDir, path, row, managed, report)
}
}
}
func warnSpellbookManagedColumnsInRow(dataDir, path string, row map[string]any, managed map[string]classSpellbookPlan, report *ValidationReport) {
for field, value := range row {
plan, ok := managed[strings.ToLower(field)]
if !ok || isNullLikeValue(value) {
continue
}
spellbookPath := displayTopdataPath(dataDir, plan.Spec.Path)
report.Diagnostics = append(report.Diagnostics, Diagnostic{
Severity: SeverityWarning,
Path: path,
Message: fmt.Sprintf(
"spell module authors class spell column %q managed by class spellbook %s; use %s instead",
plan.Column,
spellbookPath,
spellbookPath,
),
})
}
}
func displayTopdataPath(dataDir, path string) string {
topdataDir := filepath.Dir(dataDir)
rel, err := filepath.Rel(filepath.Dir(topdataDir), path)
if err != nil {
return filepath.ToSlash(path)
}
return filepath.ToSlash(rel)
}
func loadClassSpellbookSpecs(dir string) ([]classSpellbookSpec, []Diagnostic) {
paths, err := collectModulePaths(dir)
if err != nil {
return nil, []Diagnostic{{Severity: SeverityError, Path: dir, Message: err.Error()}}
}
specs := make([]classSpellbookSpec, 0, len(paths))
diagnostics := []Diagnostic{}
for _, path := range paths {
obj, err := loadJSONObject(path)
if err != nil {
diagnostics = append(diagnostics, Diagnostic{Severity: SeverityError, Path: path, Message: err.Error()})
continue
}
spec, specDiagnostics := parseClassSpellbookSpec(path, obj)
diagnostics = append(diagnostics, specDiagnostics...)
if spec.Class != "" && spec.Levels != nil {
specs = append(specs, spec)
}
}
return specs, diagnostics
}
func parseClassSpellbookSpec(path string, obj map[string]any) (classSpellbookSpec, []Diagnostic) {
diagnostics := []Diagnostic{}
spec := classSpellbookSpec{Path: path, Levels: map[int][]string{}}
if key, ok := obj["key"].(string); ok {
spec.Key = strings.TrimSpace(key)
}
classKey, ok := obj["class"].(string)
if !ok || strings.TrimSpace(classKey) == "" {
diagnostics = append(diagnostics, Diagnostic{Severity: SeverityError, Path: path, Message: "class spellbook requires class"})
} else {
spec.Class = strings.TrimSpace(classKey)
if !strings.HasPrefix(spec.Class, "classes:") {
diagnostics = append(diagnostics, Diagnostic{Severity: SeverityError, Path: path, Message: "class must use a classes: reference"})
}
}
rawLevels, ok := obj["levels"].(map[string]any)
if !ok {
diagnostics = append(diagnostics, Diagnostic{Severity: SeverityError, Path: path, Message: "class spellbook levels must be an object"})
return spec, diagnostics
}
for _, rawLevel := range sortedKeys(rawLevels) {
level, err := strconv.Atoi(rawLevel)
if err != nil || level < 0 {
diagnostics = append(diagnostics, Diagnostic{Severity: SeverityError, Path: path, Message: fmt.Sprintf("level key %q must be a nonnegative integer", rawLevel)})
continue
}
rawList, ok := rawLevels[rawLevel].([]any)
if !ok {
diagnostics = append(diagnostics, Diagnostic{Severity: SeverityError, Path: path, Message: fmt.Sprintf("level %d must be an array", level)})
continue
}
for index, rawSpell := range rawList {
spellKey, ok := rawSpell.(string)
if !ok || strings.TrimSpace(spellKey) == "" {
diagnostics = append(diagnostics, Diagnostic{Severity: SeverityError, Path: path, Message: fmt.Sprintf("level %d spell %d must be a spells: reference", level, index)})
continue
}
spellKey = strings.TrimSpace(spellKey)
if !strings.HasPrefix(spellKey, "spells:") {
diagnostics = append(diagnostics, Diagnostic{Severity: SeverityError, Path: path, Message: fmt.Sprintf("level %d spell %d must use a spells: reference", level, index)})
continue
}
spec.Levels[level] = append(spec.Levels[level], spellKey)
}
}
return spec, diagnostics
}
func collectedDatasetByName(collected []nativeCollectedDataset, name string) *nativeCollectedDataset {
for index := range collected {
if collected[index].Dataset.Name == name {
return &collected[index]
}
}
return nil
}
func rowsByKey(rows []map[string]any) map[string]map[string]any {
out := make(map[string]map[string]any, len(rows))
for _, row := range rows {
key, _ := row["key"].(string)
if key != "" {
out[key] = row
}
}
return out
}
func sortedIntKeys(values map[int][]string) []int {
keys := make([]int, 0, len(values))
for key := range values {
keys = append(keys, key)
}
slices.Sort(keys)
return keys
}
func diagnosticsTextForError(diags []Diagnostic) string {
messages := make([]string, 0, len(diags))
for _, diag := range diags {
messages = append(messages, diag.Message)
}
return strings.Join(messages, "; ")
}
internal/topdata/classes_migrate.go
+226
View File
@@ -0,0 +1,226 @@
package topdata
import (
"encoding/json"
"os"
"path/filepath"
"slices"
"strings"
)
var legacyClassesPlainFamilies = []string{"feats", "skills", "savthr", "bfeat", "pres"}
func importLegacyClasses(referenceBuilderDir, dataDir string, legacyTLK *legacyTLKData) (int, error) {
legacyRoot := filepath.Join(referenceBuilderDir, "data", "classes")
if _, err := os.Stat(legacyRoot); err != nil {
if os.IsNotExist(err) {
return 0, nil
}
return 0, err
}
targetRoot := filepath.Join(dataDir, "classes")
if canonicalClassesPresent(targetRoot) {
return 0, nil
}
coreCollected, plainCollected, err := collectLegacyClassesDatasets(legacyRoot)
if err != nil {
return 0, err
}
tableKeyByOutput := map[string]string{}
for _, dataset := range plainCollected {
if dataset.TableKey == "" {
continue
}
registerLegacyClassesTableKey(tableKeyByOutput, dataset.TableKey, dataset.Dataset.OutputName)
}
coreRows := make([]map[string]any, 0, len(coreCollected.Rows))
for _, rawRow := range coreCollected.Rows {
row, ok := deepCopyValue(rawRow).(map[string]any)
if !ok {
continue
}
if legacyTLK != nil {
if _, _, err := inlineLegacyTLKValue(row, legacyTLK); err != nil {
return 0, err
}
}
rewriteLegacyClassesCoreRow(row, tableKeyByOutput)
coreRows = append(coreRows, row)
}
if err := removePathIfExists(targetRoot); err != nil {
return 0, err
}
if err := os.MkdirAll(filepath.Join(targetRoot, "core"), 0o755); err != nil {
return 0, err
}
writes := []struct {
path string
obj map[string]any
}{
{
path: filepath.Join(targetRoot, "core", "base.json"),
obj: map[string]any{
"output": coreCollected.Dataset.OutputName,
"columns": stringSliceToAny(coreCollected.Columns),
"rows": rowsToAny(coreRows),
},
},
{
path: filepath.Join(targetRoot, "core", "lock.json"),
obj: anyMapInt(coreCollected.LockData),
},
}
for _, collected := range plainCollected {
obj := map[string]any{
"output": collected.Dataset.OutputName,
"columns": stringSliceToAny(collected.Columns),
"rows": rowsToAny(collected.Rows),
}
if strings.TrimSpace(collected.TableKey) != "" {
obj["key"] = collected.TableKey
}
writes = append(writes, struct {
path string
obj map[string]any
}{
path: filepath.Join(dataDir, collected.Dataset.Name+".json"),
obj: obj,
})
}
for _, write := range writes {
if err := os.MkdirAll(filepath.Dir(write.path), 0o755); err != nil {
return 0, err
}
raw, err := json.MarshalIndent(write.obj, "", " ")
if err != nil {
return 0, err
}
raw = append(raw, '\n')
if err := os.WriteFile(write.path, raw, 0o644); err != nil {
return 0, err
}
}
return len(writes), nil
}
func canonicalClassesPresent(targetRoot string) bool {
if !fileExists(filepath.Join(targetRoot, "core", "base.json")) || !fileExists(filepath.Join(targetRoot, "core", "lock.json")) {
return false
}
for _, family := range legacyClassesPlainFamilies {
ok, err := hasJSONFiles(filepath.Join(targetRoot, family))
if err != nil || !ok {
return false
}
}
return true
}
func collectLegacyClassesDatasets(legacyRoot string) (nativeCollectedDataset, []nativeCollectedDataset, error) {
coreCollected, err := collectBaseDataset(nativeDataset{
Name: "classes/core",
BasePath: filepath.Join(legacyRoot, "core", "base.json"),
LockPath: filepath.Join(legacyRoot, "core", "lock.json"),
ModulesDir: filepath.Join(legacyRoot, "core", "modules"),
OutputName: "classes.2da",
Spec: specForDataset("classes"),
})
if err != nil {
return nativeCollectedDataset{}, nil, err
}
coreCollected.Dataset.OutputName = "classes.2da"
plainCollected := make([]nativeCollectedDataset, 0)
for _, family := range legacyClassesPlainFamilies {
familyDir := filepath.Join(legacyRoot, family)
entries, err := os.ReadDir(familyDir)
if err != nil {
if os.IsNotExist(err) {
continue
}
return nativeCollectedDataset{}, nil, err
}
for _, entry := range entries {
if entry.IsDir() || filepath.Ext(entry.Name()) != ".json" || strings.HasPrefix(entry.Name(), ".") || entry.Name() == "lock.json" {
continue
}
filePath := filepath.Join(familyDir, entry.Name())
tableData, err := loadJSONObject(filePath)
if err != nil {
return nativeCollectedDataset{}, nil, err
}
outputName, _ := tableData["output"].(string)
if strings.TrimSpace(outputName) == "" {
outputName = strings.TrimSuffix(entry.Name(), filepath.Ext(entry.Name())) + ".2da"
}
collected, err := collectPlainDataset(nativeDataset{
Name: filepath.ToSlash(filepath.Join("classes", family, strings.TrimSuffix(entry.Name(), filepath.Ext(entry.Name())))),
BasePath: filePath,
LockPath: filepath.Join(familyDir, "lock.json"),
OutputName: outputName,
Spec: specForDataset(filepath.ToSlash(filepath.Join("classes", family, strings.TrimSuffix(entry.Name(), filepath.Ext(entry.Name()))))),
})
if err != nil {
return nativeCollectedDataset{}, nil, err
}
plainCollected = append(plainCollected, collected)
}
}
slices.SortFunc(plainCollected, func(a, b nativeCollectedDataset) int {
return strings.Compare(a.Dataset.Name, b.Dataset.Name)
})
return coreCollected, plainCollected, nil
}
func registerLegacyClassesTableKey(tableKeyByOutput map[string]string, tableKey, outputName string) {
trimmedOutput := strings.TrimSpace(outputName)
if trimmedOutput != "" {
tableKeyByOutput[trimmedOutput] = tableKey
}
trimmedStem := strings.TrimSpace(outputStem(outputName))
if trimmedStem != "" {
tableKeyByOutput[trimmedStem] = tableKey
}
}
func rewriteLegacyClassesCoreRow(row map[string]any, tableKeyByOutput map[string]string) {
for _, field := range []string{"FeatsTable", "SavingThrowTable", "SkillsTable", "BonusFeatsTable", "PreReqTable"} {
tableKey := legacyClassesTableKeyForValue(row[field], tableKeyByOutput)
if tableKey == "" {
continue
}
row[field] = map[string]any{"table": tableKey}
}
}
func legacyClassesTableKeyForValue(value any, tableKeyByOutput map[string]string) string {
switch typed := value.(type) {
case map[string]any:
tableKey, _ := typed["table"].(string)
return strings.TrimSpace(tableKey)
case string:
trimmed := strings.TrimSpace(typed)
if trimmed == "" || trimmed == nullValue {
return ""
}
if trimmed != strings.ToLower(trimmed) {
return ""
}
if tableKey, ok := tableKeyByOutput[trimmed]; ok {
return tableKey
}
return ""
default:
return ""
}
}
internal/topdata/cloakmodel_migrate.go
+6
View File
@@ -0,0 +1,6 @@
package topdata
func importLegacyCloakmodel(referenceBuilderDir, dataDir string, legacyTLK *legacyTLKData) (int, error) {
_ = legacyTLK
return importLegacyDatasetMirror(referenceBuilderDir, dataDir, "cloakmodel", "cloakmodel.2da", nil)
}
internal/topdata/convert.go
+888
View File
@@ -0,0 +1,888 @@
package topdata
import (
"encoding/json"
"errors"
"fmt"
"io"
"os"
"path/filepath"
"regexp"
"sort"
"strconv"
"strings"
"git.westgate.pw/ShadowsOverWestgate/sow-tools/internal/project"
"gopkg.in/yaml.v3"
)
type convertOptions struct {
Namespace string
KeyFields []string
CollisionMode string
BaseDialog string
Type string
Name string
FilenamePrefixes map[string]string
}
func RunConvertCommand(args []string, stdout io.Writer) error {
if len(args) == 0 || args[0] == "--help" || args[0] == "-h" {
printConvertUsage(stdout)
return nil
}
switch args[0] {
case "2da-to-json":
if len(args) > 1 && (args[1] == "--help" || args[1] == "-h") {
_, _ = fmt.Fprintln(stdout, "usage: convert-topdata 2da-to-json [flags] <input.2da> <output.json>")
return nil
}
opts, input, output, err := parseConvertArgs(args[1:], false, "suffix")
if err != nil {
return err
}
data, err := parse2DAFile(input)
if err != nil {
return err
}
result, err := convert2DAToJSON(data, output, opts)
if err != nil {
return err
}
return writeJSONFile(output, result)
case "2da-to-module":
if len(args) > 1 && (args[1] == "--help" || args[1] == "-h") {
_, _ = fmt.Fprintln(stdout, "usage: convert-topdata 2da-to-module [--namespace <dataset>] [--name <module-name>] [--type entries|override] [flags] <input.2da> [output.json]")
return nil
}
opts, input, output, err := parseConvertArgs(args[1:], true, "error")
if err != nil {
return err
}
data, err := parse2DAFile(input)
if err != nil {
return err
}
if opts.Type == "override" {
result := map[string]any{"overrides": toOverridesRows(data.Rows)}
if err := writeJSONFile(output, result); err != nil {
return err
}
_, _ = fmt.Fprintf(stdout, "converted overrides: %d\n", len(result["overrides"].([]map[string]any)))
return nil
}
result, err := convert2DAToModule(data, output, opts)
if err != nil {
return err
}
if err := writeJSONFile(output, result); err != nil {
return err
}
_, _ = fmt.Fprintf(stdout, "converted entries: %d\n", len(result["entries"].(map[string]any)))
return nil
case "json-to-2da":
if len(args) > 1 && (args[1] == "--help" || args[1] == "-h") {
_, _ = fmt.Fprintln(stdout, "usage: convert-topdata json-to-2da <input.json> <output.2da>")
return nil
}
if len(args) != 3 {
return errors.New("usage: convert-topdata json-to-2da <input.json> <output.2da>")
}
data, err := readCanonicalJSON(args[1])
if err != nil {
return err
}
return write2DAFile(data, args[2])
default:
return fmt.Errorf("unknown convert-topdata subcommand %q", args[0])
}
}
func printConvertUsage(stdout io.Writer) {
_, _ = fmt.Fprintln(stdout, "usage: convert-topdata <2da-to-json|2da-to-module|json-to-2da> ...")
_, _ = fmt.Fprintln(stdout, "")
_, _ = fmt.Fprintln(stdout, "subcommands:")
_, _ = fmt.Fprintln(stdout, " 2da-to-json Convert a 2DA file into canonical JSON rows")
_, _ = fmt.Fprintln(stdout, " 2da-to-module Convert a 2DA file into module entries or override JSON")
_, _ = fmt.Fprintln(stdout, " json-to-2da Convert canonical JSON rows into a 2DA file")
_, _ = fmt.Fprintln(stdout, "")
_, _ = fmt.Fprintln(stdout, "2da-to-module notes:")
_, _ = fmt.Fprintln(stdout, " - --namespace is optional when topdata/templates/config.yaml declares it for the input table")
_, _ = fmt.Fprintln(stdout, " - --name controls inferred output naming: <prefix>_<namespace>_<name>.json")
_, _ = fmt.Fprintln(stdout, " - topdata/templates/config.yaml module_output.filename_prefixes can set entries/override prefixes")
_, _ = fmt.Fprintln(stdout, " - flags accept both --flag value and --flag=value forms")
_, _ = fmt.Fprintln(stdout, " - unkeyed rows now fail conversion instead of being skipped")
}
type parsed2DA struct {
Columns []string `json:"columns"`
Rows []map[string]any `json:"rows"`
}
func parseConvertArgs(args []string, allowFormat bool, defaultCollision string) (convertOptions, string, string, error) {
opts := convertOptions{
CollisionMode: defaultCollision,
Type: "entries",
}
positional := make([]string, 0, 2)
for index := 0; index < len(args); index++ {
arg := args[index]
switch arg {
case "--namespace":
index++
if index >= len(args) {
return opts, "", "", errors.New("--namespace requires a value")
}
opts.Namespace = args[index]
case "--key-field":
index++
if index >= len(args) {
return opts, "", "", errors.New("--key-field requires a value")
}
opts.KeyFields = append(opts.KeyFields, args[index])
case "--name":
index++
if index >= len(args) {
return opts, "", "", errors.New("--name requires a value")
}
opts.Name = args[index]
case "--collision":
index++
if index >= len(args) {
return opts, "", "", errors.New("--collision requires a value")
}
opts.CollisionMode = args[index]
case "--base-dialog":
index++
if index >= len(args) {
return opts, "", "", errors.New("--base-dialog requires a value")
}
opts.BaseDialog = args[index]
case "--format", "--type":
if !allowFormat {
return opts, "", "", fmt.Errorf("%s does not accept %s", positionalSafe(args), arg)
}
index++
if index >= len(args) {
return opts, "", "", fmt.Errorf("%s requires a value", arg)
}
opts.Type = args[index]
default:
if value, ok := parseInlineFlagValue(arg, "--namespace"); ok {
opts.Namespace = value
continue
}
if value, ok := parseInlineFlagValue(arg, "--key-field"); ok {
opts.KeyFields = append(opts.KeyFields, value)
continue
}
if value, ok := parseInlineFlagValue(arg, "--name"); ok {
opts.Name = value
continue
}
if value, ok := parseInlineFlagValue(arg, "--collision"); ok {
opts.CollisionMode = value
continue
}
if value, ok := parseInlineFlagValue(arg, "--base-dialog"); ok {
opts.BaseDialog = value
continue
}
if value, ok := parseInlineFlagValue(arg, "--format"); ok {
if !allowFormat {
return opts, "", "", fmt.Errorf("%s does not accept %s", positionalSafe(args), "--format")
}
opts.Type = value
continue
}
if value, ok := parseInlineFlagValue(arg, "--type"); ok {
if !allowFormat {
return opts, "", "", fmt.Errorf("%s does not accept %s", positionalSafe(args), "--type")
}
opts.Type = value
continue
}
if strings.HasPrefix(arg, "--") {
return opts, "", "", fmt.Errorf("unknown flag %q", arg)
}
positional = append(positional, arg)
}
}
if allowFormat {
if len(positional) < 1 || len(positional) > 2 {
return opts, "", "", errors.New("usage: convert-topdata 2da-to-module [--namespace <dataset>] [--name <module-name>] [--type entries|override] [flags] <input.2da> [output.json]")
}
} else if len(positional) != 2 {
return opts, "", "", errors.New("usage: convert-topdata 2da-to-json [flags] <input.2da> <output.json>")
}
if opts.CollisionMode != "suffix" && opts.CollisionMode != "error" {
return opts, "", "", fmt.Errorf("unsupported collision mode %q", opts.CollisionMode)
}
if allowFormat {
output := ""
if len(positional) == 2 {
output = positional[1]
}
ctx, err := resolveConvertContext(positional[0])
if err != nil {
return opts, "", "", err
}
if ctx.Project != nil {
if outputCtx, err := resolveConvertOutputContext(ctx.Project, output); err != nil {
return opts, "", "", err
} else if ctx.Template.Namespace == "" && len(ctx.Template.KeyFields) == 0 {
ctx.Template = outputCtx
}
}
if strings.TrimSpace(opts.Namespace) == "" {
opts.Namespace = ctx.Template.Namespace
}
if len(opts.KeyFields) == 0 && len(ctx.Template.KeyFields) > 0 {
opts.KeyFields = append(opts.KeyFields, ctx.Template.KeyFields...)
}
if len(opts.FilenamePrefixes) == 0 && len(ctx.Config.ModuleOutput.FilenamePrefixes) > 0 {
opts.FilenamePrefixes = cloneStringMap(ctx.Config.ModuleOutput.FilenamePrefixes)
}
if strings.TrimSpace(opts.Namespace) == "" {
return opts, "", "", errors.New("2da-to-module requires --namespace or a topdata/templates/config.yaml entry for the input table")
}
switch opts.Type {
case "entry":
opts.Type = "entries"
case "overrides":
opts.Type = "override"
}
if opts.Type != "entries" && opts.Type != "override" {
return opts, "", "", fmt.Errorf("unsupported module type %q", opts.Type)
}
if output == "" {
resolved, err := defaultModuleOutputPath(positional[0], opts, ctx.Project)
if err != nil {
return opts, "", "", err
}
output = resolved
}
return opts, positional[0], output, nil
}
ctx, err := resolveConvertContext(positional[0])
if err != nil {
return opts, "", "", err
}
if ctx.Project != nil {
if outputCtx, err := resolveConvertOutputContext(ctx.Project, positional[1]); err != nil {
return opts, "", "", err
} else if ctx.Template.Namespace == "" && len(ctx.Template.KeyFields) == 0 {
ctx.Template = outputCtx
}
}
if strings.TrimSpace(opts.Namespace) == "" {
opts.Namespace = ctx.Template.Namespace
}
if len(opts.KeyFields) == 0 && len(ctx.Template.KeyFields) > 0 {
opts.KeyFields = append(opts.KeyFields, ctx.Template.KeyFields...)
}
return opts, positional[0], positional[1], nil
}
func parseInlineFlagValue(arg, name string) (string, bool) {
prefix := name + "="
if !strings.HasPrefix(arg, prefix) {
return "", false
}
return strings.TrimSpace(arg[len(prefix):]), true
}
func positionalSafe(args []string) string {
if len(args) == 0 {
return "command"
}
return args[0]
}
func parse2DAFile(path string) (parsed2DA, error) {
raw, err := os.ReadFile(path)
if err != nil {
return parsed2DA{}, fmt.Errorf("read %s: %w", path, err)
}
lines := strings.Split(strings.ReplaceAll(string(raw), "\r\n", "\n"), "\n")
trimmed := make([]string, 0, len(lines))
for _, line := range lines {
line = strings.TrimSpace(line)
if line != "" {
trimmed = append(trimmed, line)
}
}
if len(trimmed) < 2 || !strings.HasPrefix(trimmed[0], "2DA") {
return parsed2DA{}, fmt.Errorf("%s: invalid 2DA file", path)
}
columns := split2DALine(trimmed[1])
if len(columns) == 0 {
return parsed2DA{}, fmt.Errorf("%s: missing 2DA columns", path)
}
rows := make([]map[string]any, 0, max(0, len(trimmed)-2))
seenIDs := map[int]struct{}{}
for _, line := range trimmed[2:] {
fields := split2DALine(line)
if len(fields) == 0 {
continue
}
rowID, err := strconv.Atoi(fields[0])
if err != nil {
return parsed2DA{}, fmt.Errorf("%s: invalid 2DA row id %q", path, fields[0])
}
if len(fields)-1 > len(columns) {
return parsed2DA{}, fmt.Errorf("%s: row %d has %d values but only %d columns are declared", path, rowID, len(fields)-1, len(columns))
}
if _, ok := seenIDs[rowID]; ok {
return parsed2DA{}, fmt.Errorf("%s: duplicate 2DA row id %d", path, rowID)
}
seenIDs[rowID] = struct{}{}
row := map[string]any{"id": rowID}
for index, column := range columns {
if index+1 < len(fields) {
row[column] = smartConvertScalar(fields[index+1])
continue
}
row[column] = nullValue
}
rows = append(rows, row)
}
return parsed2DA{Columns: columns, Rows: rows}, nil
}
func split2DALine(line string) []string {
fields := make([]string, 0)
var current strings.Builder
inQuotes := false
for _, ch := range line {
switch {
case ch == '"':
inQuotes = !inQuotes
case !inQuotes && (ch == '\t' || ch == ' '):
if current.Len() > 0 {
fields = append(fields, current.String())
current.Reset()
}
default:
current.WriteRune(ch)
}
}
if current.Len() > 0 {
fields = append(fields, current.String())
}
return fields
}
func smartConvertScalar(value string) any {
if value == "" {
return ""
}
if parsed, err := strconv.Atoi(value); err == nil {
return parsed
}
return value
}
func convert2DAToJSON(data parsed2DA, outputPath string, opts convertOptions) (map[string]any, error) {
rows, err := assignConvertedRowKeys(data.Rows, outputPath, opts)
if err != nil {
return nil, err
}
outRows := make([]map[string]any, 0, len(rows))
for _, row := range rows {
ordered := map[string]any{"id": row["id"]}
if key, ok := row["key"].(string); ok && key != "" {
ordered["key"] = key
}
for _, column := range data.Columns {
ordered[column] = row[column]
}
outRows = append(outRows, ordered)
}
return map[string]any{"columns": data.Columns, "rows": outRows}, nil
}
func convert2DAToModule(data parsed2DA, outputPath string, opts convertOptions) (map[string]any, error) {
rows, err := assignConvertedRowKeys(data.Rows, outputPath, opts)
if err != nil {
return nil, err
}
entries := map[string]any{}
missing := []string{}
for _, row := range rows {
key, _ := row["key"].(string)
if key == "" {
missing = append(missing, strconv.Itoa(row["id"].(int)))
continue
}
entry := map[string]any{}
for _, column := range data.Columns {
value := row[column]
if format2DAValue(value) == nullValue {
continue
}
entry[column] = value
}
entries[key] = entry
}
if len(missing) > 0 {
return nil, fmt.Errorf("unable to generate keys for row ids %s; declare --key-field or configure key_fields in topdata/templates/config.yaml", strings.Join(missing, ", "))
}
return map[string]any{"entries": entries}, nil
}
func toOverridesRows(rows []map[string]any) []map[string]any {
out := make([]map[string]any, 0, len(rows))
for _, row := range rows {
override := map[string]any{"id": row["id"]}
keys := make([]string, 0, len(row))
for key := range row {
if key == "id" || key == "key" {
continue
}
keys = append(keys, key)
}
sort.Strings(keys)
for _, key := range keys {
if format2DAValue(row[key]) == nullValue {
continue
}
override[key] = row[key]
}
out = append(out, override)
}
return out
}
func assignConvertedRowKeys(rows []map[string]any, outputPath string, opts convertOptions) ([]map[string]any, error) {
namespace := strings.TrimSpace(opts.Namespace)
out := make([]map[string]any, 0, len(rows))
used := map[string]struct{}{}
rowIDsByKey := map[string]int{}
for _, row := range rows {
cloned := cloneRowMap(row)
if namespace == "" {
out = append(out, cloned)
continue
}
candidates := convertedRowKeyCandidates(cloned, opts.KeyFields)
if len(candidates) == 0 {
out = append(out, cloned)
continue
}
key := ""
for _, candidate := range candidates {
candidateKey := namespace + ":" + candidate
if _, ok := used[candidateKey]; ok {
continue
}
key = candidateKey
break
}
if key == "" {
key = namespace + ":" + candidates[0]
if opts.CollisionMode == "error" {
return nil, fmt.Errorf("duplicate generated key %q for row %v (already used by row %d)", key, cloned["id"], rowIDsByKey[key])
}
key = fmt.Sprintf("%s_%v", key, cloned["id"])
}
used[key] = struct{}{}
rowID, _ := asInt(cloned["id"])
rowIDsByKey[key] = rowID
cloned["key"] = key
out = append(out, cloned)
}
return out, nil
}
var convertKeyWhitespace = regexp.MustCompile(`\s+`)
var convertKeyCleaner = regexp.MustCompile(`[^a-z0-9_]`)
func convertedRowKeyCandidates(row map[string]any, preferred []string) []string {
fields := preferred
if len(fields) == 0 {
fields = []string{"LABEL", "Label", "Name"}
} else {
for _, field := range fields {
value, ok := row[field]
if !ok {
return nil
}
text := strings.TrimSpace(format2DAValue(value))
if text == "" || text == nullValue {
return nil
}
}
}
values := make([]string, 0, len(fields))
seen := map[string]struct{}{}
candidates := make([]string, 0, len(fields))
for _, field := range fields {
value, ok := row[field]
if !ok {
continue
}
text := strings.TrimSpace(format2DAValue(value))
if text == "" || text == nullValue {
continue
}
normalized := normalizeConvertedKeyText(text)
if normalized == "" {
continue
}
values = append(values, normalized)
joined := strings.Join(values, "_")
if _, ok := seen[joined]; ok {
continue
}
seen[joined] = struct{}{}
candidates = append(candidates, joined)
}
return candidates
}
func normalizeConvertedKeyText(text string) string {
text = strings.TrimSpace(strings.ToLower(text))
text = convertKeyWhitespace.ReplaceAllString(text, "_")
text = convertKeyCleaner.ReplaceAllString(text, "")
text = strings.Trim(text, "_")
return text
}
type convertTemplateTable struct {
Namespace string `json:"namespace" yaml:"namespace"`
KeyFields []string `json:"key_fields" yaml:"key_fields"`
}
type convertTemplateConfig struct {
ModuleOutput convertModuleOutputConfig `json:"module_output" yaml:"module_output"`
Tables map[string]convertTemplateTable `json:"tables" yaml:"tables"`
}
type convertModuleOutputConfig struct {
FilenamePrefixes map[string]string `json:"filename_prefixes" yaml:"filename_prefixes"`
}
type convertContext struct {
Project *project.Project
Config convertTemplateConfig
Template convertTemplateTable
}
func resolveConvertContext(input string) (convertContext, error) {
cwd, err := os.Getwd()
if err != nil {
return convertContext{}, err
}
root, err := project.FindRoot(cwd)
if err != nil {
return convertContext{}, nil
}
p, err := project.Load(root)
if err != nil {
return convertContext{}, err
}
if !p.HasTopData() {
return convertContext{Project: p}, nil
}
config, err := loadConvertTemplateConfig(p)
if err != nil {
return convertContext{}, err
}
table, err := templateTableConfig(config, p, input)
if err != nil {
return convertContext{}, err
}
return convertContext{
Project: p,
Config: config,
Template: table,
}, nil
}
func loadTemplateTableConfig(p *project.Project, input string) (convertTemplateTable, error) {
config, err := loadConvertTemplateConfig(p)
if err != nil {
return convertTemplateTable{}, err
}
return templateTableConfig(config, p, input)
}
func templateTableConfig(config convertTemplateConfig, p *project.Project, input string) (convertTemplateTable, error) {
if len(config.Tables) == 0 {
return convertTemplateTable{}, nil
}
templatesDir := filepath.Join(p.TopDataSourceDir(), "templates")
inputAbs, err := filepath.Abs(input)
if err != nil {
return convertTemplateTable{}, err
}
templatesAbs, err := filepath.Abs(templatesDir)
if err != nil {
return convertTemplateTable{}, err
}
rel, err := filepath.Rel(templatesAbs, inputAbs)
if err != nil || strings.HasPrefix(rel, "..") || rel == "." {
return convertTemplateTable{}, nil
}
rel = filepath.ToSlash(rel)
base := filepath.Base(rel)
stem := strings.TrimSuffix(base, filepath.Ext(base))
for _, key := range []string{rel, base, stem} {
if table, ok := config.Tables[key]; ok {
return table, nil
}
}
return convertTemplateTable{}, nil
}
func loadConvertTemplateConfig(p *project.Project) (convertTemplateConfig, error) {
templatesDir := filepath.Join(p.TopDataSourceDir(), "templates")
for _, candidate := range []string{"config.yaml", "config.yml", "config.json"} {
configPath := filepath.Join(templatesDir, candidate)
raw, err := os.ReadFile(configPath)
if err != nil {
if errors.Is(err, os.ErrNotExist) {
continue
}
return convertTemplateConfig{}, fmt.Errorf("read %s: %w", configPath, err)
}
var config convertTemplateConfig
switch filepath.Ext(configPath) {
case ".yaml", ".yml":
if err := yaml.Unmarshal(raw, &config); err != nil {
return convertTemplateConfig{}, fmt.Errorf("parse %s: %w", configPath, err)
}
default:
if err := json.Unmarshal(raw, &config); err != nil {
return convertTemplateConfig{}, fmt.Errorf("parse %s: %w", configPath, err)
}
}
if err := validateConvertTemplateConfig(config); err != nil {
return convertTemplateConfig{}, fmt.Errorf("validate %s: %w", configPath, err)
}
return config, nil
}
return convertTemplateConfig{}, nil
}
func validateConvertTemplateConfig(config convertTemplateConfig) error {
for moduleType, prefix := range config.ModuleOutput.FilenamePrefixes {
switch moduleType {
case "entries", "override":
default:
return fmt.Errorf("unsupported module_output.filename_prefixes key %q; expected entries or override", moduleType)
}
if normalizeModuleFilenamePrefix(prefix) == "" {
return fmt.Errorf("module_output.filename_prefixes.%s must contain at least one filename-safe character", moduleType)
}
}
return nil
}
func resolveConvertOutputContext(p *project.Project, output string) (convertTemplateTable, error) {
if p == nil || !p.HasTopData() || strings.TrimSpace(output) == "" {
return convertTemplateTable{}, nil
}
config, err := loadConvertTemplateConfig(p)
if err != nil {
return convertTemplateTable{}, err
}
if len(config.Tables) == 0 {
return convertTemplateTable{}, nil
}
dataDir := filepath.Join(p.TopDataSourceDir(), "data")
dataAbs, err := filepath.Abs(dataDir)
if err != nil {
return convertTemplateTable{}, err
}
outputAbs, err := filepath.Abs(output)
if err != nil {
return convertTemplateTable{}, err
}
rel, err := filepath.Rel(dataAbs, outputAbs)
if err != nil || rel == "." || strings.HasPrefix(rel, "..") {
return convertTemplateTable{}, nil
}
rel = filepath.ToSlash(rel)
bestMatch := ""
bestTable := convertTemplateTable{}
for _, table := range config.Tables {
namespacePath := namespaceToPath(table.Namespace)
if namespacePath == "" {
continue
}
if rel != namespacePath && !strings.HasPrefix(rel, namespacePath+"/") {
continue
}
if len(namespacePath) <= len(bestMatch) {
continue
}
bestMatch = namespacePath
bestTable = table
}
return bestTable, nil
}
func namespaceToPath(namespace string) string {
namespace = strings.TrimSpace(strings.ToLower(namespace))
if namespace == "" {
return ""
}
namespace = strings.ReplaceAll(namespace, "\\", "/")
namespace = strings.ReplaceAll(namespace, ":", "/")
namespace = convertKeyWhitespace.ReplaceAllString(namespace, "_")
namespace = strings.Trim(namespace, "/")
return namespace
}
func defaultModuleOutputPath(input string, opts convertOptions, p *project.Project) (string, error) {
if p == nil {
return "", errors.New("2da-to-module requires an explicit output path when not run inside a project root")
}
if !p.HasTopData() {
return "", errors.New("2da-to-module requires topdata to be configured when inferring an output path")
}
filename, err := defaultModuleFileName(input, opts, p)
if err != nil {
return "", err
}
modulesDir := filepath.Join(p.TopDataSourceDir(), "data", filepath.FromSlash(opts.Namespace), "modules")
return filepath.Join(modulesDir, filename), nil
}
func defaultModuleFileName(input string, opts convertOptions, p *project.Project) (string, error) {
if strings.TrimSpace(opts.Name) != "" {
name := normalizeConvertedKeyText(opts.Name)
if name == "" {
return "", fmt.Errorf("invalid --name %q", opts.Name)
}
namespace := normalizeModuleFilenameNamespace(opts.Namespace)
prefix := moduleFilenamePrefix(opts)
return fmt.Sprintf("%s_%s_%s.json", prefix, namespace, name), nil
}
if isTemplateInput(p, input) {
return "", errors.New("2da-to-module requires --name when converting from topdata/templates without an explicit output path")
}
return strings.TrimSuffix(filepath.Base(input), filepath.Ext(input)) + ".json", nil
}
func moduleFilenamePrefix(opts convertOptions) string {
moduleType := strings.TrimSpace(opts.Type)
if moduleType == "entry" {
moduleType = "entries"
}
if moduleType == "overrides" {
moduleType = "override"
}
if configured := strings.TrimSpace(opts.FilenamePrefixes[moduleType]); configured != "" {
if prefix := normalizeModuleFilenamePrefix(configured); prefix != "" {
return prefix
}
}
return "add"
}
func normalizeModuleFilenamePrefix(prefix string) string {
prefix = strings.TrimSpace(strings.ToLower(prefix))
prefix = strings.ReplaceAll(prefix, "/", "_")
prefix = strings.ReplaceAll(prefix, "\\", "_")
prefix = convertKeyWhitespace.ReplaceAllString(prefix, "_")
prefix = convertKeyCleaner.ReplaceAllString(prefix, "_")
prefix = strings.Trim(prefix, "_")
return prefix
}
func normalizeModuleFilenameNamespace(namespace string) string {
namespace = strings.TrimSpace(strings.ToLower(namespace))
namespace = strings.ReplaceAll(namespace, "/", "_")
namespace = strings.ReplaceAll(namespace, "\\", "_")
namespace = convertKeyWhitespace.ReplaceAllString(namespace, "_")
namespace = convertKeyCleaner.ReplaceAllString(namespace, "_")
namespace = strings.Trim(namespace, "_")
if namespace == "" {
return "module"
}
return namespace
}
func isTemplateInput(p *project.Project, input string) bool {
if p == nil || !p.HasTopData() {
return false
}
templatesAbs, err := filepath.Abs(filepath.Join(p.TopDataSourceDir(), "templates"))
if err != nil {
return false
}
inputAbs, err := filepath.Abs(input)
if err != nil {
return false
}
rel, err := filepath.Rel(templatesAbs, inputAbs)
if err != nil {
return false
}
return rel != "." && !strings.HasPrefix(rel, "..")
}
func readCanonicalJSON(path string) (parsed2DA, error) {
raw, err := os.ReadFile(path)
if err != nil {
return parsed2DA{}, err
}
var payload struct {
Columns []string `json:"columns"`
Rows []map[string]any `json:"rows"`
}
if err := json.Unmarshal(raw, &payload); err != nil {
return parsed2DA{}, err
}
return parsed2DA{Columns: payload.Columns, Rows: payload.Rows}, nil
}
func writeJSONFile(path string, payload any) error {
if err := os.MkdirAll(filepath.Dir(path), 0o755); err != nil && filepath.Dir(path) != "." {
return err
}
raw, err := json.MarshalIndent(payload, "", " ")
if err != nil {
return err
}
raw = append(raw, '\n')
return os.WriteFile(path, raw, 0o644)
}
func write2DAFile(data parsed2DA, path string) error {
rows := make([]map[string]any, 0, len(data.Rows))
for _, row := range data.Rows {
cloned := cloneRowMap(row)
if rowID, err := asInt(cloned["id"]); err == nil {
cloned["id"] = rowID
}
rows = append(rows, cloned)
}
sort.Slice(rows, func(i, j int) bool {
left, _ := asInt(rows[i]["id"])
right, _ := asInt(rows[j]["id"])
return left < right
})
table := map[string]any{
"columns": data.Columns,
"rows": rows,
}
if err := os.MkdirAll(filepath.Dir(path), 0o755); err != nil && filepath.Dir(path) != "." {
return err
}
return write2DA(table, path, false, -1)
}
func cloneStringMap(in map[string]string) map[string]string {
out := make(map[string]string, len(in))
for key, value := range in {
out[key] = value
}
return out
}
internal/topdata/convert_test.go
+611
View File
@@ -0,0 +1,611 @@
package topdata
import (
"bytes"
"encoding/json"
"os"
"path/filepath"
"strings"
"testing"
)
func TestRunConvertCommandInfersTemplateNamespaceAndOutput(t *testing.T) {
root := testProjectRoot(t)
writeFile(t, filepath.Join(root, "nwn-tool.yaml"), `
module:
name: Test
resref: test
paths:
source: src
assets: assets
build: build
topdata:
source: topdata
build: build/topdata
`)
mkdirAll(t, filepath.Join(root, "topdata", "templates"))
mkdirAll(t, filepath.Join(root, "topdata", "data", "appearance", "modules"))
writeFile(t, filepath.Join(root, "topdata", "templates", "config.yaml"), `
tables:
template_appearance:
namespace: appearance
key_fields:
- LABEL
`)
writeFile(t, filepath.Join(root, "topdata", "templates", "template_appearance.2da"), "2DA V2.0\n\nLABEL STRING_REF NAME\n960 \"Zombie, Ash, Hot\" **** Zombie_Ash_Hot\n961 \"Zombie, Ash, Done\" **** Zombie_Ash_Done\n")
oldCwd, err := os.Getwd()
if err != nil {
t.Fatalf("getwd: %v", err)
}
t.Cleanup(func() {
_ = os.Chdir(oldCwd)
})
if err := os.Chdir(root); err != nil {
t.Fatalf("chdir: %v", err)
}
var stdout bytes.Buffer
if err := RunConvertCommand([]string{
"2da-to-module",
"--name", "ashzombies",
"topdata/templates/template_appearance.2da",
}, &stdout); err != nil {
t.Fatalf("RunConvertCommand failed: %v", err)
}
outputPath := filepath.Join(root, "topdata", "data", "appearance", "modules", "add_appearance_ashzombies.json")
raw, err := os.ReadFile(outputPath)
if err != nil {
t.Fatalf("read output: %v", err)
}
text := string(raw)
if !strings.Contains(text, `"appearance:zombie_ash_hot"`) || !strings.Contains(text, `"appearance:zombie_ash_done"`) {
t.Fatalf("unexpected output:\n%s", text)
}
if !strings.Contains(stdout.String(), "converted entries: 2") {
t.Fatalf("unexpected stdout: %s", stdout.String())
}
}
func TestRunConvertCommandInfersOverrideOutputPrefixFromTemplateConfig(t *testing.T) {
root := testProjectRoot(t)
writeFile(t, filepath.Join(root, "nwn-tool.yaml"), `
module:
name: Test
resref: test
paths:
source: src
assets: assets
build: build
topdata:
source: topdata
build: build/topdata
`)
mkdirAll(t, filepath.Join(root, "topdata", "templates"))
writeFile(t, filepath.Join(root, "topdata", "templates", "config.yaml"), `
module_output:
filename_prefixes:
entries: add
override: ovr
tables:
template_appearance:
namespace: appearance
key_fields:
- LABEL
`)
writeFile(t, filepath.Join(root, "topdata", "templates", "template_appearance.2da"), "2DA V2.0\n\nLABEL STRING_REF NAME\n960 \"Zombie, Ash, Hot\" **** Zombie_Ash_Hot\n")
oldCwd, err := os.Getwd()
if err != nil {
t.Fatalf("getwd: %v", err)
}
t.Cleanup(func() {
_ = os.Chdir(oldCwd)
})
if err := os.Chdir(root); err != nil {
t.Fatalf("chdir: %v", err)
}
var stdout bytes.Buffer
if err := RunConvertCommand([]string{
"2da-to-module",
"--type", "override",
"--name", "ashzombies",
"topdata/templates/template_appearance.2da",
}, &stdout); err != nil {
t.Fatalf("RunConvertCommand failed: %v", err)
}
outputPath := filepath.Join(root, "topdata", "data", "appearance", "modules", "ovr_appearance_ashzombies.json")
raw, err := os.ReadFile(outputPath)
if err != nil {
t.Fatalf("read output: %v", err)
}
if !strings.Contains(string(raw), `"overrides"`) {
t.Fatalf("expected override payload, got:\n%s", string(raw))
}
if _, err := os.Stat(filepath.Join(root, "topdata", "data", "appearance", "modules", "add_appearance_ashzombies.json")); !os.IsNotExist(err) {
t.Fatalf("expected add-prefixed output to be absent, stat err: %v", err)
}
if !strings.Contains(stdout.String(), "converted overrides: 1") {
t.Fatalf("unexpected stdout: %s", stdout.String())
}
}
func TestRunConvertCommand2DAToJSONInfersTemplateKeys(t *testing.T) {
root := testProjectRoot(t)
writeFile(t, filepath.Join(root, "nwn-tool.yaml"), `
module:
name: Test
resref: test
paths:
source: src
assets: assets
build: build
topdata:
source: topdata
build: build/topdata
`)
mkdirAll(t, filepath.Join(root, "topdata", "templates"))
writeFile(t, filepath.Join(root, "topdata", "templates", "config.yaml"), `
tables:
template_appearance:
namespace: appearance
key_fields:
- LABEL
`)
inputPath := filepath.Join(root, "topdata", "templates", "template_appearance.2da")
writeFile(t, inputPath, "2DA V2.0\n\nLABEL STRING_REF NAME\n960 \"Zombie, Ash, Hot\" **** Zombie_Ash_Hot\n961 \"Zombie, Ash, Done\" **** Zombie_Ash_Done\n")
oldCwd, err := os.Getwd()
if err != nil {
t.Fatalf("getwd: %v", err)
}
t.Cleanup(func() {
_ = os.Chdir(oldCwd)
})
if err := os.Chdir(root); err != nil {
t.Fatalf("chdir: %v", err)
}
outputPath := filepath.Join(root, "out", "appearance.json")
if err := RunConvertCommand([]string{
"2da-to-json",
"topdata/templates/template_appearance.2da",
outputPath,
}, &bytes.Buffer{}); err != nil {
t.Fatalf("RunConvertCommand failed: %v", err)
}
raw, err := os.ReadFile(outputPath)
if err != nil {
t.Fatalf("read output: %v", err)
}
var payload struct {
Rows []map[string]any `json:"rows"`
}
if err := json.Unmarshal(raw, &payload); err != nil {
t.Fatalf("unmarshal output: %v", err)
}
if got := payload.Rows[0]["key"]; got != "appearance:zombie_ash_hot" {
t.Fatalf("expected first key to be inferred from template config, got %#v", got)
}
if got := payload.Rows[1]["key"]; got != "appearance:zombie_ash_done" {
t.Fatalf("expected second key to be inferred from template config, got %#v", got)
}
}
func TestRunConvertCommand2DAToJSONInfersOutputDatasetKeysFromConfig(t *testing.T) {
root := testProjectRoot(t)
writeFile(t, filepath.Join(root, "nwn-tool.yaml"), `
module:
name: Test
resref: test
paths:
source: src
assets: assets
build: build
topdata:
source: topdata
build: build/topdata
`)
mkdirAll(t, filepath.Join(root, "topdata", "templates"))
mkdirAll(t, filepath.Join(root, "topdata", "data", "soundset"))
writeFile(t, filepath.Join(root, "topdata", "templates", "config.yaml"), `
tables:
template_soundset:
namespace: soundset
key_fields:
- RESREF
- LABEL
`)
inputPath := filepath.Join(root, "scratch_soundset.2da")
writeFile(t, inputPath, "2DA V2.0\n\nLABEL RESREF STRREF GENDER TYPE\n0 \"Bandit Male\" nw_bandit 100 1 4\n1 \"Bandit Female\" nw_bandit 101 2 4\n")
oldCwd, err := os.Getwd()
if err != nil {
t.Fatalf("getwd: %v", err)
}
t.Cleanup(func() {
_ = os.Chdir(oldCwd)
})
if err := os.Chdir(root); err != nil {
t.Fatalf("chdir: %v", err)
}
outputPath := filepath.Join(root, "topdata", "data", "soundset", "base.json")
if err := RunConvertCommand([]string{
"2da-to-json",
"scratch_soundset.2da",
outputPath,
}, &bytes.Buffer{}); err != nil {
t.Fatalf("RunConvertCommand failed: %v", err)
}
raw, err := os.ReadFile(outputPath)
if err != nil {
t.Fatalf("read output: %v", err)
}
var payload struct {
Rows []map[string]any `json:"rows"`
}
if err := json.Unmarshal(raw, &payload); err != nil {
t.Fatalf("unmarshal output: %v", err)
}
if got := payload.Rows[0]["key"]; got != "soundset:nw_bandit" {
t.Fatalf("expected first soundset key, got %#v", got)
}
if got := payload.Rows[1]["key"]; got != "soundset:nw_bandit_bandit_female" {
t.Fatalf("expected second soundset key to use config disambiguation, got %#v", got)
}
}
func TestRunConvertCommand2DAToJSONInfersAmbientTemplateKeysFromResource(t *testing.T) {
root := testProjectRoot(t)
writeFile(t, filepath.Join(root, "nwn-tool.yaml"), `
module:
name: Test
resref: test
paths:
source: src
assets: assets
build: build
topdata:
source: topdata
build: build/topdata
`)
mkdirAll(t, filepath.Join(root, "topdata", "templates"))
writeFile(t, filepath.Join(root, "topdata", "templates", "config.yaml"), `
tables:
template_ambientmusic:
namespace: ambientmusic
key_fields:
- Resource
`)
inputPath := filepath.Join(root, "topdata", "templates", "template_ambientmusic.2da")
writeFile(t, inputPath, "2DA V2.0\n\nDescription DisplayName Resource Stinger1 Stinger2 Stinger3\n0 61901 **** **** **** **** ****\n1 61842 **** mus_ruralday1 **** **** ****\n2 61843 **** mus_ruralday2 **** **** ****\n")
oldCwd, err := os.Getwd()
if err != nil {
t.Fatalf("getwd: %v", err)
}
t.Cleanup(func() {
_ = os.Chdir(oldCwd)
})
if err := os.Chdir(root); err != nil {
t.Fatalf("chdir: %v", err)
}
outputPath := filepath.Join(root, "out", "ambientmusic.json")
if err := RunConvertCommand([]string{
"2da-to-json",
"topdata/templates/template_ambientmusic.2da",
outputPath,
}, &bytes.Buffer{}); err != nil {
t.Fatalf("RunConvertCommand failed: %v", err)
}
raw, err := os.ReadFile(outputPath)
if err != nil {
t.Fatalf("read output: %v", err)
}
var payload struct {
Rows []map[string]any `json:"rows"`
}
if err := json.Unmarshal(raw, &payload); err != nil {
t.Fatalf("unmarshal output: %v", err)
}
if _, ok := payload.Rows[0]["key"]; ok {
t.Fatalf("expected null Resource row to remain unkeyed, got %#v", payload.Rows[0]["key"])
}
if got := payload.Rows[1]["key"]; got != "ambientmusic:mus_ruralday1" {
t.Fatalf("expected first populated resource row key, got %#v", got)
}
if got := payload.Rows[2]["key"]; got != "ambientmusic:mus_ruralday2" {
t.Fatalf("expected second populated resource row key, got %#v", got)
}
}
func TestRunConvertCommand2DAToJSONSuffixesDuplicateGeneratedKeys(t *testing.T) {
root := testProjectRoot(t)
writeFile(t, filepath.Join(root, "nwn-tool.yaml"), `
module:
name: Test
resref: test
paths:
source: src
assets: assets
build: build
topdata:
source: topdata
build: build/topdata
`)
inputPath := filepath.Join(root, "dup.2da")
writeFile(t, inputPath, "2DA V2.0\n\nResource\n1 cmp_reserved\n2 cmp_reserved\n")
oldCwd, err := os.Getwd()
if err != nil {
t.Fatalf("getwd: %v", err)
}
t.Cleanup(func() {
_ = os.Chdir(oldCwd)
})
if err := os.Chdir(root); err != nil {
t.Fatalf("chdir: %v", err)
}
outputPath := filepath.Join(root, "out", "ambientmusic.json")
if err := RunConvertCommand([]string{
"2da-to-json",
"--namespace", "ambientmusic",
"--key-field", "Resource",
inputPath,
outputPath,
}, &bytes.Buffer{}); err != nil {
t.Fatalf("RunConvertCommand failed: %v", err)
}
raw, err := os.ReadFile(outputPath)
if err != nil {
t.Fatalf("read output: %v", err)
}
var payload struct {
Rows []map[string]any `json:"rows"`
}
if err := json.Unmarshal(raw, &payload); err != nil {
t.Fatalf("unmarshal output: %v", err)
}
if got := payload.Rows[0]["key"]; got != "ambientmusic:cmp_reserved" {
t.Fatalf("expected first duplicate key to remain unsuffixed, got %#v", got)
}
if got := payload.Rows[1]["key"]; got != "ambientmusic:cmp_reserved_2" {
t.Fatalf("expected second duplicate key to gain row-id suffix, got %#v", got)
}
}
func TestParseConvertArgsSupportsEqualsSyntax(t *testing.T) {
opts, input, output, err := parseConvertArgs([]string{
"--namespace=environment",
"--name=projectq",
"--type=entries",
"--collision=suffix",
"--key-field=Label",
"input.2da",
"output.json",
}, true, "error")
if err != nil {
t.Fatalf("parseConvertArgs failed: %v", err)
}
if opts.Namespace != "environment" {
t.Fatalf("expected namespace from equals syntax, got %q", opts.Namespace)
}
if opts.Name != "projectq" {
t.Fatalf("expected name from equals syntax, got %q", opts.Name)
}
if opts.Type != "entries" {
t.Fatalf("expected type from equals syntax, got %q", opts.Type)
}
if opts.CollisionMode != "suffix" {
t.Fatalf("expected collision from equals syntax, got %q", opts.CollisionMode)
}
if len(opts.KeyFields) != 1 || opts.KeyFields[0] != "Label" {
t.Fatalf("expected key field from equals syntax, got %#v", opts.KeyFields)
}
if input != "input.2da" || output != "output.json" {
t.Fatalf("unexpected positional parse result: input=%q output=%q", input, output)
}
}
func TestConvert2DAToJSONSkipsConfiguredKeysWhenAnyKeyFieldIsNull(t *testing.T) {
result, err := convert2DAToJSON(parsed2DA{
Columns: []string{"LABEL", "RESREF", "STRREF"},
Rows: []map[string]any{
{"id": 309, "LABEL": "Unused", "RESREF": "unused", "STRREF": 1},
{"id": 310, "LABEL": nullValue, "RESREF": "unused", "STRREF": 2},
},
}, "topdata/data/soundset/base.json", convertOptions{
Namespace: "soundset",
KeyFields: []string{"RESREF", "LABEL"},
CollisionMode: "error",
})
if err != nil {
t.Fatalf("convert2DAToJSON failed: %v", err)
}
rows := result["rows"].([]map[string]any)
if got := rows[0]["key"]; got != "soundset:unused" {
t.Fatalf("expected first row key to use the primary configured key field, got %#v", got)
}
if _, ok := rows[1]["key"]; ok {
t.Fatalf("expected second row to remain unkeyed when one configured key field is null, got %#v", rows[1]["key"])
}
}
func TestRunConvertCommandFailsOnUnkeyedRows(t *testing.T) {
root := testProjectRoot(t)
writeFile(t, filepath.Join(root, "nwn-tool.json"), `{
"module": {"name": "Test", "resref": "test"},
"paths": {"source": "src", "assets": "assets", "build": "build"},
"topdata": {"source": "topdata", "build": "build/topdata"}
}`+"\n")
mkdirAll(t, filepath.Join(root, "topdata", "templates"))
writeFile(t, filepath.Join(root, "topdata", "templates", "config.yaml"), `
tables:
template_appearance:
namespace: appearance
key_fields:
- STRING_REF
`)
writeFile(t, filepath.Join(root, "topdata", "templates", "template_appearance.2da"), "2DA V2.0\n\nLABEL STRING_REF NAME\n960 \"Zombie, Ash, Hot\" **** Zombie_Ash_Hot\n")
oldCwd, err := os.Getwd()
if err != nil {
t.Fatalf("getwd: %v", err)
}
t.Cleanup(func() {
_ = os.Chdir(oldCwd)
})
if err := os.Chdir(root); err != nil {
t.Fatalf("chdir: %v", err)
}
err = RunConvertCommand([]string{
"2da-to-module",
"--name", "ashzombies",
"topdata/templates/template_appearance.2da",
}, &bytes.Buffer{})
if err == nil || !strings.Contains(err.Error(), "unable to generate keys for row ids 960") {
t.Fatalf("expected unkeyed row failure, got %v", err)
}
}
func TestRunConvertCommandFallsBackToLegacyTemplateJSONConfig(t *testing.T) {
root := testProjectRoot(t)
writeFile(t, filepath.Join(root, "nwn-tool.yaml"), `
module:
name: Test
resref: test
paths:
source: src
assets: assets
build: build
topdata:
source: topdata
build: build/topdata
`)
mkdirAll(t, filepath.Join(root, "topdata", "templates"))
writeFile(t, filepath.Join(root, "topdata", "templates", "config.json"), `{
"tables": {
"template_appearance": {
"namespace": "appearance",
"key_fields": ["LABEL"]
}
}
}`+"\n")
inputPath := filepath.Join(root, "topdata", "templates", "template_appearance.2da")
writeFile(t, inputPath, "2DA V2.0\n\nLABEL\n960 Zombie_Ash_Hot\n")
oldCwd, err := os.Getwd()
if err != nil {
t.Fatalf("getwd: %v", err)
}
t.Cleanup(func() {
_ = os.Chdir(oldCwd)
})
if err := os.Chdir(root); err != nil {
t.Fatalf("chdir: %v", err)
}
outputPath := filepath.Join(root, "out", "appearance.json")
if err := RunConvertCommand([]string{
"2da-to-json",
"topdata/templates/template_appearance.2da",
outputPath,
}, &bytes.Buffer{}); err != nil {
t.Fatalf("RunConvertCommand failed: %v", err)
}
raw, err := os.ReadFile(outputPath)
if err != nil {
t.Fatalf("read output: %v", err)
}
if !strings.Contains(string(raw), `"appearance:zombie_ash_hot"`) {
t.Fatalf("expected legacy JSON template config fallback to be honored, got %s", string(raw))
}
}
func TestRunConvertCommandRejectsUnknownTemplateFilenamePrefixType(t *testing.T) {
root := testProjectRoot(t)
writeFile(t, filepath.Join(root, "nwn-tool.yaml"), `
module:
name: Test
resref: test
paths:
source: src
assets: assets
build: build
topdata:
source: topdata
build: build/topdata
`)
mkdirAll(t, filepath.Join(root, "topdata", "templates"))
writeFile(t, filepath.Join(root, "topdata", "templates", "config.yaml"), `
module_output:
filename_prefixes:
additions: add
tables:
template_appearance:
namespace: appearance
key_fields:
- LABEL
`)
writeFile(t, filepath.Join(root, "topdata", "templates", "template_appearance.2da"), "2DA V2.0\n\nLABEL\n960 Zombie_Ash_Hot\n")
oldCwd, err := os.Getwd()
if err != nil {
t.Fatalf("getwd: %v", err)
}
t.Cleanup(func() {
_ = os.Chdir(oldCwd)
})
if err := os.Chdir(root); err != nil {
t.Fatalf("chdir: %v", err)
}
err = RunConvertCommand([]string{
"2da-to-module",
"--name", "ashzombies",
"topdata/templates/template_appearance.2da",
}, &bytes.Buffer{})
if err == nil || !strings.Contains(err.Error(), `unsupported module_output.filename_prefixes key "additions"`) {
t.Fatalf("expected invalid filename prefix type error, got %v", err)
}
}
func TestConvert2DAToModuleFailsOnDuplicateGeneratedKeysByDefault(t *testing.T) {
_, err := convert2DAToModule(parsed2DA{
Columns: []string{"LABEL"},
Rows: []map[string]any{
{"id": 1, "LABEL": "Zombie"},
{"id": 2, "LABEL": "Zombie"},
},
}, "topdata/data/appearance/modules/add_appearance_ashzombies.json", convertOptions{
Namespace: "appearance",
CollisionMode: "error",
})
if err == nil || !strings.Contains(err.Error(), `duplicate generated key "appearance:zombie"`) {
t.Fatalf("expected duplicate generated key error, got %v", err)
}
}
func TestParse2DAFileRejectsRowsWithTooManyFields(t *testing.T) {
root := t.TempDir()
inputPath := filepath.Join(root, "bad.2da")
writeFile(t, inputPath, "2DA V2.0\n\nLABEL NAME\n0 one two three\n")
_, err := parse2DAFile(inputPath)
if err == nil || !strings.Contains(err.Error(), "has 3 values but only 2 columns are declared") {
t.Fatalf("expected row width error, got %v", err)
}
}
internal/topdata/creaturespeed_migrate.go
+145
View File
@@ -0,0 +1,145 @@
package topdata
import (
"encoding/json"
"os"
"path/filepath"
)
func importLegacyCreaturespeed(referenceBuilderDir, dataDir string, legacyTLK *legacyTLKData) (int, error) {
_ = legacyTLK
legacyDir := filepath.Join(referenceBuilderDir, "data", "creaturespeed")
if _, err := os.Stat(legacyDir); err != nil {
if os.IsNotExist(err) {
return 0, nil
}
return 0, err
}
targetDir := filepath.Join(dataDir, "creaturespeed")
targetPath := filepath.Join(targetDir, "creaturespeed.json")
if _, err := os.Stat(targetPath); err == nil {
obj, err := loadJSONObject(targetPath)
if err == nil && countLegacyTLKRefsInValue(obj) == 0 {
return 0, nil
}
}
baseObj, err := loadJSONObject(filepath.Join(legacyDir, "base.json"))
if err != nil {
return 0, err
}
rows, err := mergeCreaturespeedRows(baseObj, filepath.Join(legacyDir, "modules"))
if err != nil {
return 0, err
}
if err := os.MkdirAll(targetDir, 0o755); err != nil {
return 0, err
}
if err := removePathIfExists(filepath.Join(targetDir, "base.json")); err != nil {
return 0, err
}
if err := removePathIfExists(filepath.Join(targetDir, "lock.json")); err != nil {
return 0, err
}
if err := removePathIfExists(filepath.Join(targetDir, "modules")); err != nil {
return 0, err
}
out := map[string]any{
"output": "creaturespeed.2da",
"columns": baseObj["columns"],
"rows": rows,
}
raw, err := json.MarshalIndent(out, "", " ")
if err != nil {
return 0, err
}
raw = append(raw, '\n')
if err := os.WriteFile(targetPath, raw, 0o644); err != nil {
return 0, err
}
return 1, nil
}
func mergeCreaturespeedRows(baseObj map[string]any, modulesDir string) ([]any, error) {
rawRows, ok := baseObj["rows"].([]any)
if !ok {
return nil, nil
}
rows := make([]map[string]any, 0, len(rawRows))
byID := map[int]map[string]any{}
for _, raw := range rawRows {
row, ok := deepCopyValue(raw).(map[string]any)
if !ok {
continue
}
delete(row, "key")
rows = append(rows, row)
if rawID, ok := row["id"]; ok {
id, err := asInt(rawID)
if err != nil {
return nil, err
}
byID[id] = row
}
}
modulePaths, err := collectModulePaths(modulesDir)
if err != nil {
return nil, err
}
for _, path := range modulePaths {
obj, err := loadJSONObject(path)
if err != nil {
return nil, err
}
overrideList, ok := obj["overrides"].([]any)
if !ok {
continue
}
for _, raw := range overrideList {
override, ok := raw.(map[string]any)
if !ok {
continue
}
rawID, ok := override["id"]
if !ok {
continue
}
id, err := asInt(rawID)
if err != nil {
return nil, err
}
row := byID[id]
if row == nil {
continue
}
for key, value := range override {
if key == "id" || key == "key" || key == "_tlk" {
continue
}
row[key] = deepCopyValue(value)
}
}
}
out := make([]any, 0, len(rows))
for _, row := range rows {
out = append(out, row)
}
return out, nil
}
func removePathIfExists(path string) error {
if _, err := os.Stat(path); err != nil {
if os.IsNotExist(err) {
return nil
}
return err
}
return os.RemoveAll(path)
}
internal/topdata/damagetypes_migrate.go
+288
View File
@@ -0,0 +1,288 @@
package topdata
import (
"encoding/json"
"fmt"
"os"
"path/filepath"
"sort"
"strings"
)
func importLegacyDamagetypes(referenceBuilderDir, dataDir string, legacyTLK *legacyTLKData) (int, error) {
_ = legacyTLK
legacyRoot := filepath.Join(referenceBuilderDir, "data", "damagetypes")
if _, err := os.Stat(legacyRoot); err != nil {
if os.IsNotExist(err) {
return 0, nil
}
return 0, err
}
targetDir := filepath.Join(dataDir, "damagetypes", "registry")
targetPath := filepath.Join(targetDir, "types.json")
if _, err := os.Stat(targetPath); err == nil {
obj, err := loadJSONObject(targetPath)
if err == nil && countLegacyTLKRefsInValue(obj) == 0 {
return 0, nil
}
}
coreBase, err := loadJSONObject(filepath.Join(legacyRoot, "core", "base.json"))
if err != nil {
return 0, err
}
groupBase, err := loadJSONObject(filepath.Join(legacyRoot, "groups", "base.json"))
if err != nil {
return 0, err
}
hitvisualBase, err := loadJSONObject(filepath.Join(legacyRoot, "hitvisual", "base.json"))
if err != nil {
return 0, err
}
coreModule, err := loadJSONObject(filepath.Join(legacyRoot, "core", "modules", "add_eos.json"))
if err != nil {
return 0, err
}
coreLock, err := loadLockfile(filepath.Join(legacyRoot, "core", "lock.json"))
if err != nil {
return 0, err
}
groupModule, err := loadJSONObject(filepath.Join(legacyRoot, "groups", "modules", "add_eos.json"))
if err != nil {
return 0, err
}
groupLock, err := loadLockfile(filepath.Join(legacyRoot, "groups", "lock.json"))
if err != nil {
return 0, err
}
hitvisualModule, err := loadJSONObject(filepath.Join(legacyRoot, "hitvisual", "modules", "add_eos.json"))
if err != nil {
return 0, err
}
hitvisualLock, err := loadLockfile(filepath.Join(legacyRoot, "hitvisual", "lock.json"))
if err != nil {
return 0, err
}
rows, lockData, err := mergeLegacyDamagetypes(coreBase, groupBase, hitvisualBase, coreModule, coreLock, groupModule, groupLock, hitvisualModule, hitvisualLock)
if err != nil {
return 0, err
}
if err := os.MkdirAll(targetDir, 0o755); err != nil {
return 0, err
}
if err := removePathIfExists(filepath.Join(dataDir, "damagetypes", "core")); err != nil {
return 0, err
}
if err := removePathIfExists(filepath.Join(dataDir, "damagetypes", "groups")); err != nil {
return 0, err
}
if err := removePathIfExists(filepath.Join(dataDir, "damagetypes", "hitvisual")); err != nil {
return 0, err
}
typesOut := map[string]any{"rows": rows}
raw, err := json.MarshalIndent(typesOut, "", " ")
if err != nil {
return 0, err
}
raw = append(raw, '\n')
if err := os.WriteFile(targetPath, raw, 0o644); err != nil {
return 0, err
}
if err := saveLockfile(filepath.Join(targetDir, damagetypesRegistryLock), lockData); err != nil {
return 0, err
}
return 2, nil
}
func mergeLegacyDamagetypes(coreBase, groupBase, hitvisualBase, coreModule map[string]any, coreLock map[string]int, groupModule map[string]any, groupLock map[string]int, hitvisualModule map[string]any, hitvisualLock map[string]int) ([]map[string]any, map[string]int, error) {
coreRows, err := coerceRowMapSlice(coreBase["rows"])
if err != nil {
return nil, nil, err
}
groupRows, err := coerceRowMapSlice(groupBase["rows"])
if err != nil {
return nil, nil, err
}
hitRows, err := coerceRowMapSlice(hitvisualBase["rows"])
if err != nil {
return nil, nil, err
}
groupByID := map[int]map[string]any{}
for _, row := range groupRows {
id, err := asInt(row["id"])
if err != nil {
return nil, nil, err
}
groupByID[id] = row
}
hitByID := map[int]map[string]any{}
for _, row := range hitRows {
id, err := asInt(row["id"])
if err != nil {
return nil, nil, err
}
hitByID[id] = row
}
outRows := make([]map[string]any, 0, len(coreRows))
lockData := map[string]int{}
for _, core := range coreRows {
id, err := asInt(core["id"])
if err != nil {
return nil, nil, err
}
groupID, err := asInt(core["DamageTypeGroup"])
if err != nil {
return nil, nil, err
}
group := groupByID[groupID]
if group == nil {
return nil, nil, fmt.Errorf("core id %d references missing group %d", id, groupID)
}
hit := hitByID[id]
if hit == nil {
return nil, nil, fmt.Errorf("core id %d is missing hitvisual row", id)
}
key := "damagetype:" + normalizeDamagetypeKey(core["Label"])
lockData[key] = id
outRows = append(outRows, map[string]any{
"key": key,
"id": id,
"label": deepCopyValue(core["Label"]),
"charsheet_strref": deepCopyValue(core["CharsheetStrref"]),
"damage_type_group": strconvString(groupID),
"damage_ranged_projectile": deepCopyValue(core["DamageRangedProjectile"]),
"group_label": deepCopyValue(group["Label"]),
"feedback_strref": deepCopyValue(group["FeedbackStrref"]),
"color_r": deepCopyValue(group["ColorR"]),
"color_g": deepCopyValue(group["ColorG"]),
"color_b": deepCopyValue(group["ColorB"]),
"visual_effect_id": deepCopyValue(hit["VisualEffectID"]),
"ranged_effect_id": deepCopyValue(hit["RangedEffectID"]),
})
}
coreEntries, err := coerceEntriesMap(coreModule["entries"])
if err != nil {
return nil, nil, err
}
groupEntries, err := coerceEntriesMap(groupModule["entries"])
if err != nil {
return nil, nil, err
}
hitEntries, err := coerceEntriesMap(hitvisualModule["entries"])
if err != nil {
return nil, nil, err
}
keys := make([]string, 0, len(coreEntries))
for key := range coreEntries {
keys = append(keys, key)
}
sort.Slice(keys, func(i, j int) bool {
a := strings.TrimPrefix(keys[i], "damagetypes:")
b := strings.TrimPrefix(keys[j], "damagetypes:")
return a < b
})
for _, coreKey := range keys {
core := coreEntries[coreKey]
suffix := strings.TrimPrefix(coreKey, "damagetypes:")
group := groupEntries["damagetypegroups:"+suffix]
if group == nil {
return nil, nil, fmt.Errorf("%s: missing matching group entry", coreKey)
}
hit := hitEntries["damagehitvisual:"+suffix]
if hit == nil {
return nil, nil, fmt.Errorf("%s: missing matching hitvisual entry", coreKey)
}
id, ok := coreLock[coreKey]
if !ok {
return nil, nil, fmt.Errorf("%s: missing core lock id", coreKey)
}
groupKey := "damagetypegroups:" + suffix
groupID, ok := groupLock[groupKey]
if !ok {
return nil, nil, fmt.Errorf("%s: missing group lock id", groupKey)
}
hitKey := "damagehitvisual:" + suffix
hitID, ok := hitvisualLock[hitKey]
if !ok {
return nil, nil, fmt.Errorf("%s: missing hitvisual lock id", hitKey)
}
if hitID != id {
return nil, nil, fmt.Errorf("%s: core id %d does not match hitvisual id %d", suffix, id, hitID)
}
key := "damagetype:" + suffix
lockData[key] = id
outRows = append(outRows, map[string]any{
"key": key,
"id": id,
"label": deepCopyValue(core["Label"]),
"charsheet_strref": deepCopyValue(core["CharsheetStrref"]),
"damage_type_group": strconvString(groupID),
"damage_ranged_projectile": deepCopyValue(core["DamageRangedProjectile"]),
"group_label": deepCopyValue(group["Label"]),
"feedback_strref": deepCopyValue(group["FeedbackStrref"]),
"color_r": deepCopyValue(group["ColorR"]),
"color_g": deepCopyValue(group["ColorG"]),
"color_b": deepCopyValue(group["ColorB"]),
"visual_effect_id": deepCopyValue(hit["VisualEffectID"]),
"ranged_effect_id": deepCopyValue(hit["RangedEffectID"]),
})
}
sort.Slice(outRows, func(i, j int) bool {
return outRows[i]["id"].(int) < outRows[j]["id"].(int)
})
return outRows, lockData, nil
}
func coerceRowMapSlice(value any) ([]map[string]any, error) {
rawRows, ok := value.([]any)
if !ok {
return nil, fmt.Errorf("rows must be an array")
}
rows := make([]map[string]any, 0, len(rawRows))
for _, raw := range rawRows {
row, ok := raw.(map[string]any)
if !ok {
return nil, fmt.Errorf("row must be an object")
}
rows = append(rows, row)
}
return rows, nil
}
func coerceEntriesMap(value any) (map[string]map[string]any, error) {
rawEntries, ok := value.(map[string]any)
if !ok {
return nil, fmt.Errorf("entries must be an object")
}
entries := make(map[string]map[string]any, len(rawEntries))
for key, raw := range rawEntries {
row, ok := raw.(map[string]any)
if !ok {
return nil, fmt.Errorf("entry %s must be an object", key)
}
entries[key] = row
}
return entries, nil
}
func normalizeDamagetypeKey(value any) string {
text := strings.TrimSpace(strings.ToLower(format2DAValue(value)))
text = strings.TrimSuffix(text, "damage")
text = strings.ReplaceAll(text, " ", "")
text = strings.ReplaceAll(text, "_", "")
text = strings.ReplaceAll(text, "-", "")
return text
}
func strconvString(v int) string {
return fmt.Sprintf("%d", v)
}
internal/topdata/damagetypes_registry.go
+243
View File
@@ -0,0 +1,243 @@
package topdata
import (
"fmt"
"os"
"path/filepath"
"slices"
"strconv"
"strings"
)
const (
damagetypesRegistryDirName = "damagetypes/registry"
damagetypesRegistryLock = "lock.json"
)
type damagetypesRegistry struct {
RootPath string
LockPath string
LockData map[string]int
Types []map[string]any
}
func collectGeneratedRegistryDatasets(dataDir string) ([]nativeCollectedDataset, error) {
itempropsDatasets, err := collectItempropsRegistryDatasets(dataDir)
if err != nil {
return nil, err
}
damagetypeDatasets, err := collectDamagetypesRegistryDatasets(dataDir)
if err != nil {
return nil, err
}
racialtypesDatasets, err := collectRacialtypesRegistryDatasets(dataDir)
if err != nil {
return nil, err
}
out := make([]nativeCollectedDataset, 0, len(itempropsDatasets)+len(damagetypeDatasets)+len(racialtypesDatasets))
out = append(out, itempropsDatasets...)
out = append(out, damagetypeDatasets...)
out = append(out, racialtypesDatasets...)
return out, nil
}
func loadDamagetypesRegistry(dataDir string) (*damagetypesRegistry, error) {
root := filepath.Join(dataDir, filepath.FromSlash(damagetypesRegistryDirName))
info, err := os.Stat(root)
if err != nil {
if os.IsNotExist(err) {
return nil, nil
}
return nil, err
}
if !info.IsDir() {
return nil, fmt.Errorf("damagetypes registry root must be a directory: %s", root)
}
lockData, err := loadLockfile(filepath.Join(root, damagetypesRegistryLock))
if err != nil {
return nil, err
}
types, err := loadRegistryRows(filepath.Join(root, "types.json"))
if err != nil {
return nil, err
}
return &damagetypesRegistry{
RootPath: root,
LockPath: filepath.Join(root, damagetypesRegistryLock),
LockData: lockData,
Types: types,
}, nil
}
func collectDamagetypesRegistryDatasets(dataDir string) ([]nativeCollectedDataset, error) {
registry, err := loadDamagetypesRegistry(dataDir)
if err != nil {
return nil, err
}
if registry == nil {
return nil, nil
}
typeIDs, lockModified, err := assignDamagetypeRegistryIDs(registry.Types, registry.LockData)
if err != nil {
return nil, err
}
if lockModified {
if err := saveLockfile(registry.LockPath, registry.LockData); err != nil {
return nil, err
}
}
typeRowsSorted := slices.Clone(registry.Types)
slices.SortFunc(typeRowsSorted, func(a, b map[string]any) int {
return typeIDs[stringField(a, "key")] - typeIDs[stringField(b, "key")]
})
coreRows := make([]map[string]any, 0, len(typeRowsSorted))
hitvisualRows := make([]map[string]any, 0, len(typeRowsSorted))
groupRowsByID := map[int]map[string]any{}
groupOrder := make([]int, 0)
for _, row := range typeRowsSorted {
key := stringField(row, "key")
rowID := typeIDs[key]
groupID, err := registryIntField(row, "damage_type_group")
if err != nil {
return nil, fmt.Errorf("%s: %w", key, err)
}
coreRows = append(coreRows, map[string]any{
"id": rowID,
"key": key,
"Label": registryStringField(row, "label"),
"CharsheetStrref": deepCopyValue(row["charsheet_strref"]),
"DamageTypeGroup": strconv.Itoa(groupID),
"DamageRangedProjectile": registryNullableField(row, "damage_ranged_projectile"),
})
hitvisualRows = append(hitvisualRows, map[string]any{
"id": rowID,
"key": strings.Replace(key, "damagetype:", "damagehitvisual:", 1),
"Label": registryStringField(row, "label"),
"VisualEffectID": registryNullableField(row, "visual_effect_id"),
"RangedEffectID": registryNullableField(row, "ranged_effect_id"),
})
if _, ok := groupRowsByID[groupID]; !ok {
groupRowsByID[groupID] = map[string]any{
"id": groupID,
"key": strings.Replace(key, "damagetype:", "damagetypegroup:", 1),
"Label": registryStringField(row, "group_label"),
"FeedbackStrref": deepCopyValue(row["feedback_strref"]),
"ColorR": registryNullableField(row, "color_r"),
"ColorG": registryNullableField(row, "color_g"),
"ColorB": registryNullableField(row, "color_b"),
}
groupOrder = append(groupOrder, groupID)
continue
}
existing := groupRowsByID[groupID]
for field, candidate := range map[string]any{
"Label": registryStringField(row, "group_label"),
"FeedbackStrref": deepCopyValue(row["feedback_strref"]),
"ColorR": registryNullableField(row, "color_r"),
"ColorG": registryNullableField(row, "color_g"),
"ColorB": registryNullableField(row, "color_b"),
} {
if format2DAValue(existing[field]) != format2DAValue(candidate) {
return nil, fmt.Errorf("%s: conflicting group projection for DamageTypeGroup %d field %s", key, groupID, field)
}
}
}
slices.Sort(groupOrder)
groupRows := make([]map[string]any, 0, len(groupOrder))
for _, id := range groupOrder {
groupRows = append(groupRows, groupRowsByID[id])
}
return []nativeCollectedDataset{
newGeneratedDataset("damagetypes/registry/core", "damagetypes.2da", []string{"Label", "CharsheetStrref", "DamageTypeGroup", "DamageRangedProjectile"}, coreRows),
newGeneratedDataset("damagetypes/registry/groups", "damagetypegroups.2da", []string{"Label", "FeedbackStrref", "ColorR", "ColorG", "ColorB"}, groupRows),
newGeneratedDataset("damagetypes/registry/hitvisual", "damagehitvisual.2da", []string{"Label", "VisualEffectID", "RangedEffectID"}, hitvisualRows),
}, nil
}
func assignDamagetypeRegistryIDs(rows []map[string]any, lockData map[string]int) (map[string]int, bool, error) {
ids := map[string]int{}
used := map[int]struct{}{}
for key, id := range lockData {
if strings.HasPrefix(key, "damagetype:") {
used[id] = struct{}{}
}
}
modified := false
for _, row := range rows {
key := stringField(row, "key")
if key == "" {
return nil, false, fmt.Errorf("registry row is missing key")
}
if !strings.HasPrefix(key, "damagetype:") {
return nil, false, fmt.Errorf("registry key %q must use prefix %q", key, "damagetype:")
}
if rawID, ok := row["id"]; ok {
id, err := asInt(rawID)
if err != nil {
return nil, false, fmt.Errorf("%s: invalid id: %w", key, err)
}
if existing, ok := lockData[key]; ok && existing != id {
return nil, false, fmt.Errorf("%s: explicit id %d does not match lock id %d", key, id, existing)
}
lockData[key] = id
ids[key] = id
used[id] = struct{}{}
modified = true
continue
}
if id, ok := lockData[key]; ok {
ids[key] = id
used[id] = struct{}{}
}
}
nextID := nextAvailableID(used)
for _, row := range rows {
key := stringField(row, "key")
if _, ok := ids[key]; ok {
continue
}
lockData[key] = nextID
ids[key] = nextID
used[nextID] = struct{}{}
nextID = nextAvailableID(used)
modified = true
}
return ids, modified, nil
}
func registryStringField(row map[string]any, field string) string {
text := stringField(row, field)
if text == "" {
return nullValue
}
return text
}
func registryNullableField(row map[string]any, field string) any {
value, ok := row[field]
if !ok {
return nullValue
}
switch typed := value.(type) {
case string:
if strings.TrimSpace(typed) == "" {
return nullValue
}
}
return deepCopyValue(value)
}
func registryIntField(row map[string]any, field string) (int, error) {
value, ok := row[field]
if !ok {
return 0, fmt.Errorf("missing %s", field)
}
return asInt(value)
}
internal/topdata/doortypes_migrate.go
+99
View File
@@ -0,0 +1,99 @@
package topdata
import (
"encoding/json"
"os"
"path/filepath"
)
func importLegacyDoortypes(referenceBuilderDir, dataDir string, legacyTLK *legacyTLKData) (int, error) {
_ = legacyTLK
legacyDir := filepath.Join(referenceBuilderDir, "data", "doortypes")
if _, err := os.Stat(legacyDir); err != nil {
if os.IsNotExist(err) {
return 0, nil
}
return 0, err
}
targetDir := filepath.Join(dataDir, "doortypes")
targetBasePath := filepath.Join(targetDir, "base.json")
targetLockPath := filepath.Join(targetDir, "lock.json")
targetModulesDir := filepath.Join(targetDir, "modules")
moduleNames := []string{
"add_pld01.json",
"add_sic11.json",
"add_tapr.json",
"add_tdm01.json",
"add_tdx01.json",
"add_tei01.json",
"add_tfm01.json",
}
targetModulePaths := make([]string, 0, len(moduleNames))
for _, name := range moduleNames {
targetModulePaths = append(targetModulePaths, filepath.Join(targetModulesDir, name))
}
if fileExists(targetBasePath) && fileExists(targetLockPath) {
allModulesPresent := true
for _, path := range targetModulePaths {
if !fileExists(path) {
allModulesPresent = false
break
}
}
if allModulesPresent {
return 0, nil
}
}
baseObj, err := loadJSONObject(filepath.Join(legacyDir, "base.json"))
if err != nil {
return 0, err
}
baseObj["output"] = "doortypes.2da"
lockObj, err := loadJSONObject(filepath.Join(legacyDir, "lock.json"))
if err != nil {
return 0, err
}
moduleObjs := make([]map[string]any, 0, len(moduleNames))
for _, name := range moduleNames {
obj, err := loadJSONObject(filepath.Join(legacyDir, "modules", name))
if err != nil {
return 0, err
}
moduleObjs = append(moduleObjs, obj)
}
if err := os.MkdirAll(targetModulesDir, 0o755); err != nil {
return 0, err
}
writes := []struct {
path string
obj map[string]any
}{
{path: targetBasePath, obj: baseObj},
{path: targetLockPath, obj: lockObj},
}
for i, path := range targetModulePaths {
writes = append(writes, struct {
path string
obj map[string]any
}{path: path, obj: moduleObjs[i]})
}
for _, write := range writes {
raw, err := json.MarshalIndent(write.obj, "", " ")
if err != nil {
return 0, err
}
raw = append(raw, '\n')
if err := os.WriteFile(write.path, raw, 0o644); err != nil {
return 0, err
}
}
return len(writes), nil
}
internal/topdata/editorconfig_format.go
+246
View File
@@ -0,0 +1,246 @@
package topdata
import (
"bufio"
"bytes"
"fmt"
"os"
"path/filepath"
"regexp"
"strconv"
"strings"
)
type jsonFileFormatting struct {
Indent string
LineEnding string
FinalNewline bool
}
type editorConfigDocument struct {
Dir string
Root bool
Sections []editorConfigSection
}
type editorConfigSection struct {
Pattern string
Properties map[string]string
}
func defaultJSONFileFormatting() jsonFileFormatting {
return jsonFileFormatting{
Indent: " ",
LineEnding: "\n",
FinalNewline: true,
}
}
func resolveJSONFileFormatting(path string) (jsonFileFormatting, error) {
formatting := defaultJSONFileFormatting()
documents, err := editorConfigDocumentsForPath(path)
if err != nil {
return jsonFileFormatting{}, err
}
for _, document := range documents {
relative, err := filepath.Rel(document.Dir, path)
if err != nil {
return jsonFileFormatting{}, fmt.Errorf("resolve editorconfig path for %s relative to %s: %w", path, document.Dir, err)
}
relative = filepath.ToSlash(relative)
if strings.HasPrefix(relative, "../") || relative == ".." {
continue
}
for _, section := range document.Sections {
matches, err := editorConfigPatternMatches(section.Pattern, relative)
if err != nil {
return jsonFileFormatting{}, fmt.Errorf("%s/.editorconfig section [%s]: %w", document.Dir, section.Pattern, err)
}
if !matches {
continue
}
next, err := applyEditorConfigProperties(formatting, section.Properties)
if err != nil {
return jsonFileFormatting{}, fmt.Errorf("%s/.editorconfig section [%s]: %w", document.Dir, section.Pattern, err)
}
formatting = next
}
}
return formatting, nil
}
func editorConfigDocumentsForPath(path string) ([]editorConfigDocument, error) {
dir := filepath.Dir(path)
documents := []editorConfigDocument{}
for {
configPath := filepath.Join(dir, ".editorconfig")
raw, err := os.ReadFile(configPath)
if err == nil {
document, err := parseEditorConfig(configPath, raw)
if err != nil {
return nil, err
}
document.Dir = dir
documents = append(documents, document)
if document.Root {
break
}
} else if !os.IsNotExist(err) {
return nil, fmt.Errorf("read %s: %w", configPath, err)
}
parent := filepath.Dir(dir)
if parent == dir {
break
}
dir = parent
}
for left, right := 0, len(documents)-1; left < right; left, right = left+1, right-1 {
documents[left], documents[right] = documents[right], documents[left]
}
return documents, nil
}
func parseEditorConfig(path string, raw []byte) (editorConfigDocument, error) {
document := editorConfigDocument{}
scanner := bufio.NewScanner(bytes.NewReader(raw))
var current *editorConfigSection
for lineNumber := 1; scanner.Scan(); lineNumber++ {
line := strings.TrimSpace(scanner.Text())
if line == "" || strings.HasPrefix(line, "#") || strings.HasPrefix(line, ";") {
continue
}
if strings.HasPrefix(line, "[") {
end := strings.LastIndex(line, "]")
if end < 0 {
return editorConfigDocument{}, fmt.Errorf("%s:%d: malformed section header", path, lineNumber)
}
section := editorConfigSection{
Pattern: strings.TrimSpace(line[1:end]),
Properties: map[string]string{},
}
document.Sections = append(document.Sections, section)
current = &document.Sections[len(document.Sections)-1]
continue
}
index := strings.IndexAny(line, "=:")
if index < 0 {
return editorConfigDocument{}, fmt.Errorf("%s:%d: expected key/value property", path, lineNumber)
}
key := strings.ToLower(strings.TrimSpace(line[:index]))
value := strings.TrimSpace(line[index+1:])
if current == nil {
if key == "root" {
document.Root = strings.EqualFold(value, "true")
}
continue
}
current.Properties[key] = value
}
if err := scanner.Err(); err != nil {
return editorConfigDocument{}, fmt.Errorf("scan %s: %w", path, err)
}
return document, nil
}
func applyEditorConfigProperties(formatting jsonFileFormatting, properties map[string]string) (jsonFileFormatting, error) {
style := strings.ToLower(strings.TrimSpace(properties["indent_style"]))
sizeText := strings.ToLower(strings.TrimSpace(properties["indent_size"]))
if style == "tab" {
formatting.Indent = "\t"
} else if style != "" && style != "space" && style != "unset" {
return jsonFileFormatting{}, fmt.Errorf("unsupported indent_style %q", properties["indent_style"])
}
if sizeText != "" && sizeText != "unset" && sizeText != "tab" {
size, err := strconv.Atoi(sizeText)
if err != nil || size < 1 {
return jsonFileFormatting{}, fmt.Errorf("indent_size must be a positive integer, got %q", properties["indent_size"])
}
if style != "tab" {
formatting.Indent = strings.Repeat(" ", size)
}
}
switch value := strings.ToLower(strings.TrimSpace(properties["end_of_line"])); value {
case "", "unset":
case "lf":
formatting.LineEnding = "\n"
case "crlf":
formatting.LineEnding = "\r\n"
case "cr":
formatting.LineEnding = "\r"
default:
return jsonFileFormatting{}, fmt.Errorf("unsupported end_of_line %q", properties["end_of_line"])
}
switch value := strings.ToLower(strings.TrimSpace(properties["insert_final_newline"])); value {
case "", "unset":
case "true":
formatting.FinalNewline = true
case "false":
formatting.FinalNewline = false
default:
return jsonFileFormatting{}, fmt.Errorf("insert_final_newline must be true or false, got %q", properties["insert_final_newline"])
}
return formatting, nil
}
func editorConfigPatternMatches(pattern, relativePath string) (bool, error) {
pattern = filepath.ToSlash(strings.TrimSpace(pattern))
pattern = strings.TrimPrefix(pattern, "/")
if pattern == "" {
return false, nil
}
if !strings.Contains(pattern, "/") {
relativePath = pathBase(relativePath)
}
expression, err := editorConfigGlobRegexp(pattern)
if err != nil {
return false, err
}
return regexp.MatchString(expression, relativePath)
}
func editorConfigGlobRegexp(pattern string) (string, error) {
var builder strings.Builder
builder.WriteByte('^')
for index := 0; index < len(pattern); {
char := pattern[index]
switch char {
case '*':
if index+1 < len(pattern) && pattern[index+1] == '*' {
if index+2 < len(pattern) && pattern[index+2] == '/' {
builder.WriteString("(?:.*/)?")
index += 3
} else {
builder.WriteString(".*")
index += 2
}
continue
}
builder.WriteString("[^/]*")
case '?':
builder.WriteString("[^/]")
case '.', '+', '(', ')', '|', '^', '$', '{', '}', '[', ']', '\\':
builder.WriteByte('\\')
builder.WriteByte(char)
default:
builder.WriteByte(char)
}
index++
}
builder.WriteByte('$')
expression := builder.String()
if _, err := regexp.Compile(expression); err != nil {
return "", err
}
return expression, nil
}
func pathBase(path string) string {
if index := strings.LastIndex(path, "/"); index >= 0 {
return path[index+1:]
}
return path
}
internal/topdata/expansion_native.go
+284
View File
@@ -0,0 +1,284 @@
package topdata
import (
"fmt"
"slices"
"strings"
)
func normalizeMetadataKey(key string) string {
replacer := strings.NewReplacer("-", "_", " ", "_")
return strings.ToLower(replacer.Replace(strings.TrimSpace(key)))
}
func parseRowMetadata(raw any) (map[string]any, error) {
if raw == nil {
return nil, nil
}
obj, ok := raw.(map[string]any)
if !ok {
return nil, fmt.Errorf("meta must be an object")
}
if len(obj) == 0 {
return map[string]any{}, nil
}
meta := make(map[string]any, len(obj))
for key, value := range obj {
switch normalizeMetadataKey(key) {
case "family":
family, err := parseFamilyMetadata(value)
if err != nil {
return nil, fmt.Errorf("meta.family: %w", err)
}
meta["family"] = family
case "wiki":
wiki, err := parseWikiMetadata(value)
if err != nil {
return nil, fmt.Errorf("meta.wiki: %w", err)
}
if len(wiki) > 0 {
meta["wiki"] = wiki
}
default:
return nil, fmt.Errorf("unknown metadata key %q", key)
}
}
return meta, nil
}
func mergeExpansionData(collected []nativeCollectedDataset) ([]nativeCollectedDataset, error) {
datasetIndexMap := make(map[string]int)
for i, ds := range collected {
datasetIndexMap[ds.Dataset.Name] = i
datasetIndexMap[ds.Dataset.OutputName] = i
}
rowsWithExpansion := []struct {
datasetName string
row map[string]any
}{}
for _, ds := range collected {
for _, row := range ds.Rows {
if hasExpansionData(row) {
rowsWithExpansion = append(rowsWithExpansion, struct {
datasetName string
row map[string]any
}{ds.Dataset.Name, row})
}
}
}
if len(rowsWithExpansion) == 0 {
return collected, nil
}
modifiedTargets := map[int]struct{}{}
for _, src := range rowsWithExpansion {
expansion, _ := extractExpansionData(src.row)
for targetDatasetName, targetRows := range expansion.Data {
targetIndex, ok := datasetIndexMap[targetDatasetName]
if !ok {
outputName := targetDatasetName
if !strings.HasSuffix(outputName, ".2da") {
outputName = outputName + ".2da"
}
targetIndex, ok = datasetIndexMap[outputName]
}
if !ok {
return nil, fmt.Errorf("expansion targets unknown dataset %q", targetDatasetName)
}
targetDS := collected[targetIndex]
originalLockData, _ := loadLockfile(targetDS.Dataset.LockPath)
if originalLockData == nil {
originalLockData = map[string]int{}
}
lockModified := false
usedIDs := map[int]struct{}{}
usedKeys := map[string]struct{}{}
for _, rowID := range targetDS.LockData {
usedIDs[rowID] = struct{}{}
}
for _, row := range targetDS.Rows {
if id, ok := row["id"].(int); ok {
usedIDs[id] = struct{}{}
}
if key, ok := row["key"].(string); ok {
usedKeys[key] = struct{}{}
}
}
nextID := nextAvailableIDAtLeast(usedIDs, targetDS.Dataset.RowGenerationMinRow)
for _, targetRow := range targetRows {
var rowID int
var hasID bool
if rawID, ok := targetRow["id"]; ok {
switch typed := rawID.(type) {
case int:
rowID = typed
hasID = true
case float64:
rowID = int(typed)
hasID = true
case string:
parsed, err := asInt(typed)
if err == nil {
rowID = parsed
hasID = true
}
}
}
key, _ := targetRow["key"].(string)
if strings.TrimSpace(key) == "" {
return nil, fmt.Errorf("expansion into %s: injected rows must specify key", targetDatasetName)
}
if key != "" {
if existingID, exists := targetDS.LockData[key]; exists {
rowID = existingID
hasID = true
} else if existingID, exists := originalLockData[key]; exists && targetDS.Dataset.RowGeneration != "first_null_row" {
rowID = existingID
hasID = true
targetDS.LockData[key] = existingID
lockModified = true
targetDS.LockModified = true
}
if _, seen := usedKeys[key]; seen {
continue
}
usedKeys[key] = struct{}{}
}
if !hasID {
rowID = nextID
usedIDs[rowID] = struct{}{}
nextID = nextAvailableIDAtLeast(usedIDs, targetDS.Dataset.RowGenerationMinRow)
} else {
if _, exists := usedIDs[rowID]; exists && key == "" {
return nil, fmt.Errorf("expansion into %s: row id %d already exists", targetDatasetName, rowID)
}
usedIDs[rowID] = struct{}{}
}
if key != "" {
if _, exists := targetDS.LockData[key]; !exists {
targetDS.LockData[key] = rowID
lockModified = true
targetDS.LockAdded++
targetDS.LockModified = true
}
}
newRow := map[string]any{
"id": rowID,
}
for k, v := range targetRow {
if k != "id" {
newRow[k] = v
}
}
if key != "" {
newRow["key"] = key
}
targetDS.Rows = append(targetDS.Rows, newRow)
}
if lockModified {
modifiedTargets[targetIndex] = struct{}{}
}
slices.SortFunc(targetDS.Rows, func(a, b map[string]any) int {
return a["id"].(int) - b["id"].(int)
})
collected[targetIndex] = targetDS
}
if err := applyExpansionValueToRow(src.row, expansion.Value); err != nil {
return nil, err
}
}
for targetIndex := range modifiedTargets {
targetDS := collected[targetIndex]
referencedKeys, err := collectReferencedLockKeys(nativeDatasetSourceDir(targetDS.Dataset))
if err != nil {
return nil, fmt.Errorf("dataset %s: collect referenced keys: %w", targetDS.Dataset.Name, err)
}
if pruned, updated := pruneLockDataToActiveRows(targetDS.LockData, targetDS.Rows, referencedKeys, targetDS.RetiredKeys); pruned > 0 || updated > 0 {
targetDS.LockPruned += pruned
targetDS.LockModified = true
collected[targetIndex] = targetDS
}
}
return collected, nil
}
type expansionSpec struct {
Value string
Data map[string][]map[string]any
}
func hasExpansionData(row map[string]any) bool {
for _, value := range row {
if isExpansionValue(value) {
return true
}
}
return false
}
func isExpansionValue(value any) bool {
obj, ok := value.(map[string]any)
if !ok {
return false
}
_, hasValue := obj["value"]
_, hasData := obj["data"]
return hasValue && hasData
}
func extractExpansionData(row map[string]any) (expansionSpec, bool) {
for _, value := range row {
if isExpansionValue(value) {
obj := value.(map[string]any)
valueStr, _ := obj["value"].(string)
dataMap, _ := obj["data"].(map[string]any)
result := expansionSpec{
Value: valueStr,
Data: make(map[string][]map[string]any),
}
for datasetName, rawRows := range dataMap {
switch typed := rawRows.(type) {
case []any:
rows := make([]map[string]any, 0, len(typed))
for _, r := range typed {
if rowObj, ok := r.(map[string]any); ok {
rows = append(rows, rowObj)
}
}
result.Data[datasetName] = rows
case map[string]any:
result.Data[datasetName] = []map[string]any{typed}
}
}
return result, true
}
}
return expansionSpec{}, false
}
func applyExpansionValueToRow(row map[string]any, value string) error {
for field, oldValue := range row {
if isExpansionValue(oldValue) {
row[field] = value
}
}
return nil
}
internal/topdata/family_expansion.go
+345
View File
@@ -0,0 +1,345 @@
package topdata
import (
"fmt"
"strings"
)
type familyIdentity struct {
Parent string
Child string
}
type familyExpansionSource struct {
Dataset string
Column string
Predicate string
}
type familyExpansionTitleStyle struct {
ChildCase string
ChildParenthetical string
}
type familyExpansionSpec struct {
Family string
FamilyKey string
Template string
ChildSource familyExpansionSource
NamePrefix string
LabelPrefix string
ConstantPrefix string
TemplateFields []string
DefaultFields map[string]any
ApplyAfterModules bool
ChildRefField string
IdentitySource string
AllowExistingOnly bool
AutoPrereqFields map[string]string
LegacyFamilyKeys []string
TitleStyle familyExpansionTitleStyle
}
func splitFamilyExpansionIdentity(text string) familyIdentity {
text = strings.TrimSpace(text)
if text == "" {
return familyIdentity{}
}
if idx := strings.Index(text, "_"); idx > 0 && idx < len(text)-1 {
return familyIdentity{
Parent: text[:idx],
Child: text[idx+1:],
}
}
return familyIdentity{Parent: text}
}
func parseFamilyExpansionSource(raw any) (familyExpansionSource, error) {
if raw == nil {
return familyExpansionSource{}, fmt.Errorf("child_source is required")
}
obj, ok := raw.(map[string]any)
if !ok {
return familyExpansionSource{}, fmt.Errorf("child_source must be an object")
}
dataset, ok := obj["dataset"].(string)
if !ok || strings.TrimSpace(dataset) == "" {
return familyExpansionSource{}, fmt.Errorf("child_source.dataset must be a non-empty string")
}
source := familyExpansionSource{
Dataset: strings.TrimSpace(dataset),
}
if column, ok := obj["column"].(string); ok && strings.TrimSpace(column) != "" {
source.Column = strings.TrimSpace(column)
}
if predicate, ok := obj["predicate"].(string); ok && strings.TrimSpace(predicate) != "" {
source.Predicate = strings.TrimSpace(predicate)
}
return source, nil
}
func parseFamilyExpansionSpec(path string, obj map[string]any) (familyExpansionSpec, error) {
family, ok := obj["family"].(string)
if !ok || strings.TrimSpace(family) == "" {
return familyExpansionSpec{}, fmt.Errorf("generated file %s: family must be a non-empty string", path)
}
familyKey, ok := obj["family_key"].(string)
if !ok || strings.TrimSpace(familyKey) == "" {
return familyExpansionSpec{}, fmt.Errorf("generated file %s: family_key must be a non-empty string", path)
}
template, ok := obj["template"].(string)
if !ok || strings.TrimSpace(template) == "" {
return familyExpansionSpec{}, fmt.Errorf("generated file %s: template must be a non-empty string", path)
}
source, err := parseFamilyExpansionSource(obj["child_source"])
if err != nil {
return familyExpansionSpec{}, fmt.Errorf("generated file %s: %w", path, err)
}
namePrefix, _ := optionalTrimmedString(obj, "name_prefix")
labelPrefix, _ := optionalTrimmedString(obj, "label_prefix")
constantPrefix, _ := optionalTrimmedString(obj, "constant_prefix")
templateFields, err := parseOptionalStringArray(obj["template_fields"], "template_fields")
if err != nil {
return familyExpansionSpec{}, fmt.Errorf("generated file %s: %w", path, err)
}
defaultFields, err := parseOptionalObject(obj["default_fields"], "default_fields")
if err != nil {
return familyExpansionSpec{}, fmt.Errorf("generated file %s: %w", path, err)
}
applyAfterModules, err := parseOptionalBoolField(obj["apply_after_modules"], "apply_after_modules")
if err != nil {
return familyExpansionSpec{}, fmt.Errorf("generated file %s: %w", path, err)
}
childRefField, _ := optionalTrimmedString(obj, "child_ref_field")
identitySource, _ := optionalTrimmedString(obj, "identity_source")
switch identitySource {
case "", "child_source_value":
default:
return familyExpansionSpec{}, fmt.Errorf("generated file %s: identity_source must be empty or child_source_value", path)
}
autoPrereqFields, err := parseOptionalStringMap(obj["auto_prereq_fields"], "auto_prereq_fields")
if err != nil {
return familyExpansionSpec{}, fmt.Errorf("generated file %s: %w", path, err)
}
legacyFamilyKeys, err := parseOptionalStringArray(obj["legacy_family_keys"], "legacy_family_keys")
if err != nil {
return familyExpansionSpec{}, fmt.Errorf("generated file %s: %w", path, err)
}
if err := validateLegacyFamilyKeys(familyKey, legacyFamilyKeys); err != nil {
return familyExpansionSpec{}, fmt.Errorf("generated file %s: %w", path, err)
}
titleStyle, err := parseFamilyExpansionTitleStyle(obj["title_style"])
if err != nil {
return familyExpansionSpec{}, fmt.Errorf("generated file %s: %w", path, err)
}
allowExistingOnly, err := parseOptionalBoolField(obj["allow_existing_only"], "allow_existing_only")
if err != nil {
return familyExpansionSpec{}, fmt.Errorf("generated file %s: %w", path, err)
}
return familyExpansionSpec{
Family: strings.TrimSpace(family),
FamilyKey: strings.TrimSpace(familyKey),
Template: strings.TrimSpace(template),
ChildSource: source,
NamePrefix: namePrefix,
LabelPrefix: labelPrefix,
ConstantPrefix: constantPrefix,
TemplateFields: templateFields,
DefaultFields: defaultFields,
ApplyAfterModules: applyAfterModules,
ChildRefField: childRefField,
IdentitySource: identitySource,
AllowExistingOnly: allowExistingOnly,
AutoPrereqFields: autoPrereqFields,
LegacyFamilyKeys: legacyFamilyKeys,
TitleStyle: titleStyle,
}, nil
}
func parseFamilyExpansionTitleStyle(raw any) (familyExpansionTitleStyle, error) {
style := familyExpansionTitleStyle{
ChildCase: "preserve",
ChildParenthetical: "preserve",
}
if raw == nil {
return style, nil
}
obj, ok := raw.(map[string]any)
if !ok {
return familyExpansionTitleStyle{}, fmt.Errorf("title_style must be an object")
}
if childCase, ok := obj["child_case"].(string); ok && strings.TrimSpace(childCase) != "" {
style.ChildCase = strings.TrimSpace(childCase)
}
switch style.ChildCase {
case "preserve", "lower":
default:
return familyExpansionTitleStyle{}, fmt.Errorf("title_style.child_case must be preserve or lower")
}
if childParenthetical, ok := obj["child_parenthetical"].(string); ok && strings.TrimSpace(childParenthetical) != "" {
style.ChildParenthetical = strings.TrimSpace(childParenthetical)
}
switch style.ChildParenthetical {
case "preserve", "comma":
default:
return familyExpansionTitleStyle{}, fmt.Errorf("title_style.child_parenthetical must be preserve or comma")
}
return style, nil
}
func validateLegacyFamilyKeys(familyKey string, legacyFamilyKeys []string) error {
seen := map[string]string{}
normalizedFamilyKey := normalizeKeyIdentity(familyKey)
for _, legacyKey := range legacyFamilyKeys {
normalizedLegacyKey := normalizeKeyIdentity(legacyKey)
if normalizedLegacyKey == normalizedFamilyKey {
return fmt.Errorf("legacy_family_keys must not include family_key %q", familyKey)
}
if previous, ok := seen[normalizedLegacyKey]; ok {
return fmt.Errorf("legacy_family_keys contains duplicate-equivalent keys %q and %q", previous, legacyKey)
}
seen[normalizedLegacyKey] = legacyKey
}
return nil
}
func isFamilyExpansionObject(obj map[string]any) bool {
_, hasFamilyKey := obj["family_key"]
_, hasTemplate := obj["template"]
_, hasChildSource := obj["child_source"]
_, hasNamePrefix := obj["name_prefix"]
_, hasLabelPrefix := obj["label_prefix"]
_, hasConstantPrefix := obj["constant_prefix"]
_, hasTemplateFields := obj["template_fields"]
_, hasDefaultFields := obj["default_fields"]
_, hasApplyAfterModules := obj["apply_after_modules"]
_, hasChildRefField := obj["child_ref_field"]
_, hasIdentitySource := obj["identity_source"]
_, hasAllowExistingOnly := obj["allow_existing_only"]
_, hasAutoPrereqFields := obj["auto_prereq_fields"]
_, hasLegacyFamilyKeys := obj["legacy_family_keys"]
_, hasTitleStyle := obj["title_style"]
return hasFamilyKey || hasTemplate || hasChildSource || hasNamePrefix || hasLabelPrefix ||
hasConstantPrefix || hasTemplateFields || hasDefaultFields || hasApplyAfterModules || hasChildRefField ||
hasIdentitySource || hasAllowExistingOnly || hasAutoPrereqFields || hasLegacyFamilyKeys || hasTitleStyle
}
func optionalTrimmedString(obj map[string]any, field string) (string, bool) {
text, ok := obj[field].(string)
if !ok || strings.TrimSpace(text) == "" {
return "", false
}
return strings.TrimSpace(text), true
}
func parseOptionalStringArray(raw any, field string) ([]string, error) {
if raw == nil {
return nil, nil
}
items, ok := raw.([]any)
if !ok {
return nil, fmt.Errorf("%s must be an array of strings", field)
}
out := make([]string, 0, len(items))
for _, item := range items {
text, ok := item.(string)
if !ok || strings.TrimSpace(text) == "" {
return nil, fmt.Errorf("%s must contain only non-empty strings", field)
}
out = append(out, strings.TrimSpace(text))
}
return out, nil
}
func parseOptionalObject(raw any, field string) (map[string]any, error) {
if raw == nil {
return nil, nil
}
obj, ok := raw.(map[string]any)
if !ok {
return nil, fmt.Errorf("%s must be an object", field)
}
return obj, nil
}
func parseOptionalStringMap(raw any, field string) (map[string]string, error) {
if raw == nil {
return nil, nil
}
obj, ok := raw.(map[string]any)
if !ok {
return nil, fmt.Errorf("%s must be an object of strings", field)
}
out := make(map[string]string, len(obj))
for key, value := range obj {
text, ok := value.(string)
if !ok || strings.TrimSpace(text) == "" {
return nil, fmt.Errorf("%s must contain only non-empty string values", field)
}
out[key] = strings.TrimSpace(text)
}
return out, nil
}
func parseOptionalBoolField(raw any, field string) (bool, error) {
if raw == nil {
return false, nil
}
value, ok := raw.(bool)
if !ok {
return false, fmt.Errorf("%s must be a boolean", field)
}
return value, nil
}
func familyMetadata(parent, child, source, template string) map[string]any {
meta := map[string]any{
"parent": strings.TrimSpace(parent),
}
if strings.TrimSpace(child) != "" {
meta["child"] = strings.TrimSpace(child)
}
if strings.TrimSpace(source) != "" {
meta["source"] = strings.TrimSpace(source)
}
if strings.TrimSpace(template) != "" {
meta["template"] = strings.TrimSpace(template)
}
return map[string]any{"family": meta}
}
func parseFamilyMetadata(raw any) (map[string]any, error) {
obj, ok := raw.(map[string]any)
if !ok {
return nil, fmt.Errorf("must be an object")
}
parent, ok := obj["parent"].(string)
if !ok || strings.TrimSpace(parent) == "" {
return nil, fmt.Errorf("parent must be a non-empty string")
}
meta := map[string]any{
"parent": strings.TrimSpace(parent),
}
if child, ok := obj["child"].(string); ok && strings.TrimSpace(child) != "" {
meta["child"] = strings.TrimSpace(child)
}
if source, ok := obj["source"].(string); ok && strings.TrimSpace(source) != "" {
meta["source"] = strings.TrimSpace(source)
}
if template, ok := obj["template"].(string); ok && strings.TrimSpace(template) != "" {
meta["template"] = strings.TrimSpace(template)
}
return meta, nil
}
func childTokenFromExpandedKey(key, parent string) string {
key = strings.TrimSpace(key)
key = strings.TrimPrefix(key, "feat:")
if strings.HasPrefix(key, parent+"_") {
return strings.TrimPrefix(key, parent+"_")
}
if strings.HasPrefix(key, parent) {
return strings.TrimPrefix(key, parent)
}
return ""
}
internal/topdata/feat_migrate.go
+64
View File
@@ -0,0 +1,64 @@
package topdata
import (
"encoding/json"
"os"
"path/filepath"
)
func importLegacyFeat(referenceBuilderDir, dataDir string, legacyTLK *legacyTLKData) (int, error) {
_ = legacyTLK
legacyDir := filepath.Join(referenceBuilderDir, "data", "feat")
if _, err := os.Stat(legacyDir); err != nil {
if os.IsNotExist(err) {
return 0, nil
}
return 0, err
}
legacyBasePath := filepath.Join(legacyDir, "base.json")
if _, err := os.Stat(legacyBasePath); err != nil {
if os.IsNotExist(err) {
return 0, nil
}
return 0, err
}
targetDir := filepath.Join(dataDir, "feat")
targetBasePath := filepath.Join(targetDir, "base.json")
var targetObj map[string]any
if _, err := os.Stat(targetBasePath); err == nil {
targetObj, err = loadJSONObject(targetBasePath)
if err != nil {
return 0, err
}
} else if !os.IsNotExist(err) {
return 0, err
}
if targetObj != nil {
if rows, ok := targetObj["rows"].([]any); ok && len(rows) > 0 {
return 0, nil
}
}
baseObj, err := loadJSONObject(legacyBasePath)
if err != nil {
return 0, err
}
baseObj["output"] = "feat.2da"
baseObj["compare_reference"] = false
canonicalizeWikiMetadataDocument(baseObj)
if err := os.MkdirAll(targetDir, 0o755); err != nil {
return 0, err
}
raw, err := json.MarshalIndent(baseObj, "", " ")
if err != nil {
return 0, err
}
raw = append(raw, '\n')
if err := os.WriteFile(targetBasePath, raw, 0o644); err != nil {
return 0, err
}
return 1, nil
}
internal/topdata/generated_assets.go
+205
View File
@@ -0,0 +1,205 @@
package topdata
import (
"fmt"
"os"
"path/filepath"
"strings"
"git.westgate.pw/ShadowsOverWestgate/sow-tools/internal/project"
)
type Generated2DAAsset struct {
Rel string
SourcePath string
}
func BuildGenerated2DAAssets(p *project.Project, progress func(string)) ([]Generated2DAAsset, error) {
if progress == nil {
progress = func(string) {}
}
configs := p.EffectiveConfig().Generated.TopData2DA
if len(configs) == 0 {
return nil, nil
}
results := make([]Generated2DAAsset, 0)
for _, cfg := range configs {
progress(fmt.Sprintf("Building generated 2DA assets for %s...", cfg.ID))
generated, err := buildGenerated2DAAssetGroup(p, cfg, progress)
if err != nil {
return nil, err
}
results = append(results, generated...)
}
return results, nil
}
func buildGenerated2DAAssetGroup(p *project.Project, cfg project.GeneratedTopData2DAConfig, progress func(string)) ([]Generated2DAAsset, error) {
sourceDir := filepath.Join(p.Root, filepath.FromSlash(strings.TrimSpace(cfg.Source)))
dataDir := filepath.Join(sourceDir, "data")
outputDir := filepath.Join(p.Root, filepath.FromSlash(strings.TrimSpace(cfg.Output)))
datasets, err := discoverNativeDatasets(dataDir)
if err != nil {
return nil, fmt.Errorf("discover generated topdata 2DA datasets %s: %w", cfg.ID, err)
}
selected := make([]nativeDataset, 0, len(datasets))
for _, dataset := range datasets {
if generatedDatasetIncluded(dataset.Name, cfg.IncludeDatasets) {
selected = append(selected, dataset)
}
}
if len(selected) == 0 {
return nil, fmt.Errorf("generated topdata 2DA config %s matched no datasets under %s", cfg.ID, dataDir)
}
collected := make([]nativeCollectedDataset, 0, len(selected))
for _, dataset := range selected {
collectedDataset, err := collectNativeDataset(dataset)
if err != nil {
return nil, err
}
collected = append(collected, collectedDataset)
}
consumer := cfg.Autogen
consumer.LocalOverrideRoot = p.AssetsDir()
collected, err = applyAutogenConsumersForGeneratedAssets(collected, consumer, progress)
if err != nil {
return nil, err
}
collected, err = normalizePartsRowsACBonus(collected, consumer.PartsRows)
if err != nil {
return nil, err
}
collected, err = applyPartOverridesWithConfig(sourceDir, collected, consumer.PartsRows)
if err != nil {
return nil, err
}
if err := rejectGenerated2DATLKValues(cfg.ID, collected); err != nil {
return nil, err
}
if _, _, err := saveNativeLockfiles(collected); err != nil {
return nil, fmt.Errorf("save generated topdata 2DA lockfiles %s: %w", cfg.ID, err)
}
tableRegistry, err := newResolvedTableRegistry(collected)
if err != nil {
return nil, err
}
keyToID, rowByKey := generated2DAGlobalRows(collected)
if err := os.RemoveAll(outputDir); err != nil {
return nil, fmt.Errorf("clean generated topdata 2DA output %s: %w", outputDir, err)
}
if err := os.MkdirAll(outputDir, 0o755); err != nil {
return nil, fmt.Errorf("create generated topdata 2DA output %s: %w", outputDir, err)
}
results := make([]Generated2DAAsset, 0, len(collected))
for _, dataset := range collected {
compiled, err := resolveNativeDataset(dataset, keyToID, rowByKey, tableRegistry, nil, project.TopDataClassFeatInjectionConfig{}, nil)
if err != nil {
return nil, err
}
outputPath := filepath.Join(outputDir, dataset.Dataset.OutputName)
denseRows := dataset.Dataset.Kind == nativeDatasetBase ||
(consumer.Mode == "parts_rows" && isPartsDataset(dataset.Dataset.Name))
if err := write2DA(compiled, outputPath, denseRows, -1); err != nil {
return nil, err
}
results = append(results, Generated2DAAsset{
Rel: filepath.ToSlash(filepath.Join(cfg.PackageRoot, dataset.Dataset.OutputName)),
SourcePath: outputPath,
})
}
return results, nil
}
func applyAutogenConsumersForGeneratedAssets(collected []nativeCollectedDataset, consumer project.AutogenConsumerConfig, progress func(string)) ([]nativeCollectedDataset, error) {
if strings.TrimSpace(consumer.Mode) == "" {
return collected, nil
}
if progress == nil {
progress = func(string) {}
}
if !autogenConsumerTargetsCollectedDataset(collected, consumer) {
return collected, nil
}
if progress != nil {
progress(fmt.Sprintf("Scanning local generated 2DA autogen input for %s from %s...", consumer.ID, consumer.LocalOverrideRoot))
}
entries, err := scanLocalAutogenEntries(consumer.LocalOverrideRoot, consumer)
if err != nil {
return nil, err
}
switch consumer.Mode {
case "parts_rows":
return augmentWithAutogeneratedPartsWithConfig(collected, autogenPartsInventory(entries), consumer.PartsRows), nil
case "cachedmodels_rows":
return augmentWithAutogeneratedCachedModels(collected, entries)
default:
return nil, fmt.Errorf("unsupported generated topdata 2DA autogen mode %q", consumer.Mode)
}
}
func generatedDatasetIncluded(name string, patterns []string) bool {
name = filepath.ToSlash(strings.TrimSpace(name))
for _, pattern := range patterns {
pattern = filepath.ToSlash(strings.TrimSpace(pattern))
if pattern == "" {
continue
}
if pattern == name {
return true
}
if strings.HasSuffix(pattern, "/**") {
prefix := strings.TrimSuffix(pattern, "/**")
if name == prefix || strings.HasPrefix(name, prefix+"/") {
return true
}
}
}
return false
}
func generated2DAGlobalRows(collected []nativeCollectedDataset) (map[string]int, map[string]map[string]any) {
keyToID := map[string]int{}
rowByKey := map[string]map[string]any{}
for _, dataset := range collected {
for key, rowID := range dataset.LockData {
keyToID[key] = rowID
}
for _, row := range dataset.Rows {
key, _ := row["key"].(string)
if key == "" {
continue
}
if rowID, ok := row["id"].(int); ok {
keyToID[key] = rowID
rowByKey[key] = row
}
}
}
return keyToID, rowByKey
}
func rejectGenerated2DATLKValues(configID string, collected []nativeCollectedDataset) error {
for _, dataset := range collected {
for _, row := range dataset.Rows {
for field, value := range row {
if field == "id" || field == "key" {
continue
}
allowBare := columnMatchesSpec(dataset.Dataset.Spec, field)
if _, ok, err := parseTLKPayload(value, allowBare); err != nil {
return fmt.Errorf("generated topdata 2DA config %s dataset %s field %s has invalid TLK payload: %w", configID, dataset.Dataset.Name, field, err)
} else if ok {
return fmt.Errorf("generated topdata 2DA config %s dataset %s field %s uses TLK-backed text; 2DA-only generated asset builds do not generate TLK output", configID, dataset.Dataset.Name, field)
}
}
}
}
return nil
}
internal/topdata/genericdoors_migrate.go
+6
View File
@@ -0,0 +1,6 @@
package topdata
func importLegacyGenericdoors(referenceBuilderDir, dataDir string, legacyTLK *legacyTLKData) (int, error) {
_ = legacyTLK
return importLegacyDatasetMirror(referenceBuilderDir, dataDir, "genericdoors", "genericdoors.2da", nil)
}
internal/topdata/itemprops_registry.go
+1214
View File
File diff suppressed because it is too large Load Diff
internal/topdata/legacy_dataset_migrate.go
+176
View File
@@ -0,0 +1,176 @@
package topdata
import (
"encoding/json"
"io/fs"
"os"
"path/filepath"
"sort"
"strings"
)
type legacyDatasetTransform func(relativePath string, obj map[string]any)
func importLegacyDatasetMirror(referenceBuilderDir, dataDir, datasetName, outputName string, transform legacyDatasetTransform) (int, error) {
legacyDir := filepath.Join(referenceBuilderDir, "data", datasetName)
if _, err := os.Stat(legacyDir); err != nil {
if os.IsNotExist(err) {
return 0, nil
}
return 0, err
}
targetDir := filepath.Join(dataDir, datasetName)
baseObj, err := loadJSONObject(filepath.Join(legacyDir, "base.json"))
if err != nil {
return 0, err
}
if outputName != "" {
baseObj["output"] = outputName
}
if transform != nil {
transform("base.json", baseObj)
}
lockObj, err := loadJSONObject(filepath.Join(legacyDir, "lock.json"))
if err != nil {
return 0, err
}
if transform != nil {
transform("lock.json", lockObj)
}
moduleRelPaths, err := collectJSONRelativePaths(filepath.Join(legacyDir, "modules"))
if err != nil {
return 0, err
}
moduleObjs := make(map[string]map[string]any, len(moduleRelPaths))
for _, relPath := range moduleRelPaths {
obj, err := loadJSONObject(filepath.Join(legacyDir, "modules", relPath))
if err != nil {
return 0, err
}
normalizedRel := filepath.ToSlash(relPath)
if transform != nil {
transform(filepath.ToSlash(filepath.Join("modules", normalizedRel)), obj)
}
moduleObjs[normalizedRel] = obj
}
if err := os.MkdirAll(targetDir, 0o755); err != nil {
return 0, err
}
updated := 0
for _, write := range []struct {
path string
obj map[string]any
}{
{path: filepath.Join(targetDir, "base.json"), obj: baseObj},
{path: filepath.Join(targetDir, "lock.json"), obj: lockObj},
} {
changed, err := writeJSONObjectIfChanged(write.path, write.obj)
if err != nil {
return 0, err
}
if changed {
updated++
}
}
targetModulesDir := filepath.Join(targetDir, "modules")
if len(moduleObjs) > 0 {
if err := os.MkdirAll(targetModulesDir, 0o755); err != nil {
return 0, err
}
}
for relPath, obj := range moduleObjs {
targetPath := filepath.Join(targetModulesDir, filepath.FromSlash(relPath))
if err := os.MkdirAll(filepath.Dir(targetPath), 0o755); err != nil {
return 0, err
}
changed, err := writeJSONObjectIfChanged(targetPath, obj)
if err != nil {
return 0, err
}
if changed {
updated++
}
}
staleModules, err := collectJSONRelativePaths(targetModulesDir)
if err != nil {
return 0, err
}
for _, relPath := range staleModules {
normalizedRel := filepath.ToSlash(relPath)
if _, ok := moduleObjs[normalizedRel]; ok {
continue
}
if err := os.Remove(filepath.Join(targetModulesDir, filepath.FromSlash(normalizedRel))); err != nil && !os.IsNotExist(err) {
return 0, err
}
updated++
}
return updated, nil
}
func collectJSONRelativePaths(root string) ([]string, error) {
if _, err := os.Stat(root); err != nil {
if os.IsNotExist(err) {
return nil, nil
}
return nil, err
}
var relPaths []string
err := filepath.WalkDir(root, func(path string, d fs.DirEntry, err error) error {
if err != nil {
return err
}
if d.IsDir() {
return nil
}
if !strings.EqualFold(filepath.Ext(d.Name()), ".json") {
return nil
}
relPath, err := filepath.Rel(root, path)
if err != nil {
return err
}
relPaths = append(relPaths, filepath.ToSlash(relPath))
return nil
})
if err != nil {
return nil, err
}
sort.Strings(relPaths)
return relPaths, nil
}
func writeJSONObjectIfChanged(path string, obj map[string]any) (bool, error) {
raw, err := json.MarshalIndent(obj, "", " ")
if err != nil {
return false, err
}
raw = append(raw, '\n')
current, err := os.ReadFile(path)
if err == nil && string(current) == string(raw) {
return false, nil
}
if err != nil && !os.IsNotExist(err) {
return false, err
}
if err := os.WriteFile(path, raw, 0o644); err != nil {
return false, err
}
return true, nil
}
func fileExists(path string) bool {
_, err := os.Stat(path)
return err == nil
}
internal/topdata/loadscreens_migrate.go
+6
View File
@@ -0,0 +1,6 @@
package topdata
func importLegacyLoadscreens(referenceBuilderDir, dataDir string, legacyTLK *legacyTLKData) (int, error) {
_ = legacyTLK
return importLegacyDatasetMirror(referenceBuilderDir, dataDir, "loadscreens", "loadscreens.2da", nil)
}
internal/topdata/masterfeats_migrate.go
+237
View File
@@ -0,0 +1,237 @@
package topdata
import (
"encoding/json"
"os"
"path/filepath"
"strconv"
)
func importLegacyMasterfeats(referenceBuilderDir, dataDir string, legacyTLK *legacyTLKData) (int, error) {
legacyDir := filepath.Join(referenceBuilderDir, "data", "feat", "masterfeats")
if _, err := os.Stat(legacyDir); err != nil {
if os.IsNotExist(err) {
return 0, nil
}
return 0, err
}
targetDir := filepath.Join(dataDir, "masterfeats")
targetPath := filepath.Join(targetDir, "base.json")
legacyPlainPath := filepath.Join(targetDir, "masterfeats.json")
if _, err := os.Stat(targetPath); err == nil {
obj, err := loadJSONObject(targetPath)
if err == nil && countLegacyTLKRefsInValue(obj) == 0 {
return 0, nil
}
}
collected, err := importLegacyMasterfeatsDataset(legacyDir)
if err != nil {
return 0, err
}
referenceIDs, err := collectReferenceLockIDs(filepath.Join(referenceBuilderDir, "data"))
if err != nil {
return 0, err
}
rows := make([]any, 0, len(collected.Rows))
for _, row := range collected.Rows {
copyRow, ok := deepCopyValue(row).(map[string]any)
if !ok {
continue
}
if legacyTLK != nil {
if _, _, err := inlineLegacyTLKValue(copyRow, legacyTLK); err != nil {
return 0, err
}
}
copyRow = rewriteImportedIDRefs(copyRow, referenceIDs)
rows = append(rows, copyRow)
}
if err := os.MkdirAll(targetDir, 0o755); err != nil {
return 0, err
}
writes := []struct {
path string
obj map[string]any
}{
{
path: targetPath,
obj: map[string]any{
"output": collected.Dataset.OutputName,
"columns": stringSliceToAny(collected.Columns),
"rows": rows,
},
},
{
path: filepath.Join(targetDir, "lock.json"),
obj: anyMapInt(collected.LockData),
},
}
for _, write := range writes {
raw, err := json.MarshalIndent(write.obj, "", " ")
if err != nil {
return 0, err
}
raw = append(raw, '\n')
if err := os.WriteFile(write.path, raw, 0o644); err != nil {
return 0, err
}
}
updatedFiles := len(writes)
if err := os.Remove(legacyPlainPath); err == nil {
updatedFiles++
} else if err != nil && !os.IsNotExist(err) {
return 0, err
}
return updatedFiles, nil
}
func importLegacyMasterfeatsDataset(legacyDir string) (nativeCollectedDataset, error) {
dataset := nativeDataset{
Name: "masterfeats",
BasePath: filepath.Join(legacyDir, "masterfeats.json"),
LockPath: filepath.Join(legacyDir, "lock.json"),
Spec: specForDataset("masterfeats"),
}
tableData, err := loadJSONObject(dataset.BasePath)
if err != nil {
return nativeCollectedDataset{}, err
}
columns, err := parseColumns(tableData, dataset.Name)
if err != nil {
return nativeCollectedDataset{}, err
}
rawRows, ok := tableData["rows"].([]any)
if !ok {
return nativeCollectedDataset{}, nil
}
lockData, err := loadLockfile(dataset.LockPath)
if err != nil {
return nativeCollectedDataset{}, err
}
usedIDs := map[int]struct{}{}
for _, rowID := range lockData {
usedIDs[rowID] = struct{}{}
}
rows := make([]map[string]any, 0, len(rawRows))
explicitID := make([]bool, 0, len(rawRows))
for index, raw := range rawRows {
rowObj, ok := raw.(map[string]any)
if !ok {
continue
}
rowID := index
if value, ok := rowObj["id"]; ok {
parsed, err := asInt(value)
if err != nil {
return nativeCollectedDataset{}, err
}
rowID = parsed
}
row := map[string]any{"id": rowID}
for _, column := range columns {
row[column] = nullValue
}
for key, value := range rowObj {
switch key {
case "id":
case "key":
if keyText, ok := value.(string); ok && keyText != "" {
row["key"] = keyText
}
default:
columnName, ok := canonicalColumn(columns, key)
if !ok {
continue
}
row[columnName] = value
}
}
rows = append(rows, row)
_, hasExplicitID := rowObj["id"]
explicitID = append(explicitID, hasExplicitID)
if hasExplicitID {
usedIDs[rowID] = struct{}{}
}
}
nextID := nextAvailableID(usedIDs)
lockModified := false
for index, row := range rows {
key, hasKey := row["key"].(string)
if hasKey && key != "" {
if lockedID, ok := lockData[key]; ok {
row["id"] = lockedID
continue
}
if explicitID[index] {
lockData[key] = row["id"].(int)
} else {
row["id"] = nextID
lockData[key] = nextID
usedIDs[nextID] = struct{}{}
nextID = nextAvailableID(usedIDs)
}
lockModified = true
continue
}
if !explicitID[index] {
row["id"] = index
}
}
if lockModified {
_ = lockModified
}
return nativeCollectedDataset{
Dataset: nativeDataset{
Name: dataset.Name,
OutputName: "masterfeats.2da",
Columns: columns,
Spec: dataset.Spec,
},
Columns: columns,
Rows: rows,
LockData: lockData,
}, nil
}
func rewriteImportedIDRefs(value any, ids map[string]int) map[string]any {
row, ok := rewriteImportedIDRefsValue(value, ids).(map[string]any)
if !ok {
return map[string]any{}
}
return row
}
func rewriteImportedIDRefsValue(value any, ids map[string]int) any {
switch typed := value.(type) {
case map[string]any:
if len(typed) == 1 {
if rawID, ok := typed["id"]; ok {
if key, ok := rawID.(string); ok {
if resolved, ok := ids[key]; ok {
return strconv.Itoa(resolved)
}
}
}
}
out := make(map[string]any, len(typed))
for key, child := range typed {
out[key] = rewriteImportedIDRefsValue(child, ids)
}
return out
case []any:
out := make([]any, 0, len(typed))
for _, child := range typed {
out = append(out, rewriteImportedIDRefsValue(child, ids))
}
return out
default:
return value
}
}
internal/topdata/metadata_wiki.go
+190
View File
@@ -0,0 +1,190 @@
package topdata
import (
"fmt"
"strings"
)
func isNullLike(value any) bool {
if value == nil {
return true
}
text, ok := value.(string)
if !ok {
return false
}
return strings.TrimSpace(text) == nullValue
}
func normalizeWikiMetadataKey(key string) string {
switch normalizeMetadataKey(key) {
case "reqfeats", "req_feats":
return "reqfeats"
case "generate", "wikigenerate":
return "generate"
case "canonical", "wikicanonical":
return "canonical"
case "status", "wikistatus":
return "status"
case "progressionnotes", "progression_notes", "classprogressionnotes", "class_progression_notes":
return "progression_notes"
default:
return ""
}
}
func legacyWikiMetadataField(key string) string {
switch normalizeMetadataKey(key) {
case "wikireqfeats":
return "reqfeats"
case "wikigenerate":
return "generate"
case "wikicanonical":
return "canonical"
case "wikistatus":
return "status"
default:
return ""
}
}
func parseWikiMetadata(raw any) (map[string]any, error) {
obj, ok := raw.(map[string]any)
if !ok {
return nil, fmt.Errorf("must be an object")
}
wiki := map[string]any{}
for key, value := range obj {
normalized := normalizeWikiMetadataKey(key)
if normalized == "" {
return nil, fmt.Errorf("unknown field %q", key)
}
if isNullLike(value) {
continue
}
wiki[normalized] = cloneAuthoringValue(value)
}
return wiki, nil
}
func mergeMetadataMaps(base, overlay map[string]any) map[string]any {
if len(base) == 0 && len(overlay) == 0 {
return nil
}
out := map[string]any{}
for key, value := range base {
out[key] = cloneAuthoringValue(value)
}
for key, value := range overlay {
baseValue, hasBase := out[key]
baseMap, baseIsMap := baseValue.(map[string]any)
valueMap, valueIsMap := value.(map[string]any)
if hasBase && baseIsMap && valueIsMap {
out[key] = mergeMetadataMaps(baseMap, valueMap)
continue
}
out[key] = cloneAuthoringValue(value)
}
if len(out) == 0 {
return nil
}
return out
}
func parseExistingMetadata(row map[string]any) (map[string]any, error) {
rawMeta, ok := lookupField(row, "meta")
if !ok {
return nil, nil
}
return parseRowMetadata(rawMeta)
}
func setLegacyWikiMetadata(row map[string]any, field string, value any) error {
wikiKey := legacyWikiMetadataField(field)
if wikiKey == "" {
return nil
}
meta, err := parseExistingMetadata(row)
if err != nil {
return err
}
if meta == nil {
meta = map[string]any{}
}
wiki, _ := meta["wiki"].(map[string]any)
if wiki == nil {
wiki = map[string]any{}
}
if isNullLike(value) {
delete(wiki, wikiKey)
} else {
wiki[wikiKey] = cloneAuthoringValue(value)
}
if len(wiki) == 0 {
delete(meta, "wiki")
} else {
meta["wiki"] = wiki
}
if len(meta) == 0 {
delete(row, "meta")
} else {
row["meta"] = meta
}
return nil
}
func canonicalizeWikiMetadataDocument(obj map[string]any) {
if rawColumns, ok := obj["columns"].([]any); ok {
filtered := make([]any, 0, len(rawColumns))
for _, raw := range rawColumns {
column, ok := raw.(string)
if ok && isAuthoringOnlyField(column) {
continue
}
filtered = append(filtered, raw)
}
obj["columns"] = filtered
}
if rows, ok := obj["rows"].([]any); ok {
for _, raw := range rows {
row, ok := raw.(map[string]any)
if !ok {
continue
}
canonicalizeWikiMetadataRow(row)
}
}
if entries, ok := obj["entries"].(map[string]any); ok {
for _, raw := range entries {
row, ok := raw.(map[string]any)
if !ok {
continue
}
canonicalizeWikiMetadataRow(row)
}
}
if overrides, ok := obj["overrides"].([]any); ok {
for _, raw := range overrides {
row, ok := raw.(map[string]any)
if !ok {
continue
}
canonicalizeWikiMetadataRow(row)
}
}
}
func canonicalizeWikiMetadataRow(row map[string]any) {
for _, field := range []string{"WIKIREQFEATS", "WIKIGENERATE", "WIKICANONICAL", "WIKISTATUS"} {
value, ok := row[field]
if !ok {
continue
}
_ = setLegacyWikiMetadata(row, field, value)
delete(row, field)
}
}
internal/topdata/migrate.go
+415
View File
@@ -0,0 +1,415 @@
package topdata
import (
"encoding/json"
"fmt"
"os"
"path/filepath"
"strings"
"git.westgate.pw/ShadowsOverWestgate/sow-tools/internal/project"
)
type NormalizeResult struct {
StatePath string
ScannedFiles int
UpdatedFiles int
InlineValues int
RemainingFiles int
RemainingLegacyRefs int
}
func NormalizeProject(p *project.Project) (NormalizeResult, error) {
if !p.HasTopData() {
return NormalizeResult{}, fmt.Errorf("topdata is not configured for this project")
}
sourceDir := p.TopDataSourceDir()
dataDir := filepath.Join(sourceDir, "data")
migrationRoot := resolveMigrationInputRoot(sourceDir, p.TopDataReferenceBuilderDir())
legacyTLK, err := loadLegacyTLK(filepath.Join(sourceDir, "tlk"))
if err != nil {
return NormalizeResult{}, err
}
if migrationRoot != "" {
migrationTLK, err := loadLegacyTLK(filepath.Join(migrationRoot, "tlk"))
if err != nil {
return NormalizeResult{}, err
}
legacyTLK = mergeLegacyTLK(migrationTLK, legacyTLK)
}
baseDialogImported, err := importLegacyBaseDialog(migrationRoot, sourceDir)
if err != nil {
return NormalizeResult{}, err
}
itempropsImported, err := importLegacyItempropsRegistry(migrationRoot, dataDir, legacyTLK)
if err != nil {
return NormalizeResult{}, err
}
masterfeatsImported, err := importLegacyMasterfeats(migrationRoot, dataDir, legacyTLK)
if err != nil {
return NormalizeResult{}, err
}
featImported, err := importLegacyFeat(migrationRoot, dataDir, legacyTLK)
if err != nil {
return NormalizeResult{}, err
}
creaturespeedImported, err := importLegacyCreaturespeed(migrationRoot, dataDir, legacyTLK)
if err != nil {
return NormalizeResult{}, err
}
armorImported, err := importLegacyArmor(migrationRoot, dataDir, legacyTLK)
if err != nil {
return NormalizeResult{}, err
}
baseitemsImported, err := importLegacyBaseitems(migrationRoot, dataDir, legacyTLK)
if err != nil {
return NormalizeResult{}, err
}
damagetypesImported, err := importLegacyDamagetypes(migrationRoot, dataDir, legacyTLK)
if err != nil {
return NormalizeResult{}, err
}
skyboxesImported, err := importLegacySkyboxes(migrationRoot, dataDir, legacyTLK)
if err != nil {
return NormalizeResult{}, err
}
vfxPersistentImported, err := importLegacyVFXPersistent(migrationRoot, dataDir, legacyTLK)
if err != nil {
return NormalizeResult{}, err
}
loadscreensImported, err := importLegacyLoadscreens(migrationRoot, dataDir, legacyTLK)
if err != nil {
return NormalizeResult{}, err
}
progfxImported, err := importLegacyProgfx(migrationRoot, dataDir, legacyTLK)
if err != nil {
return NormalizeResult{}, err
}
genericdoorsImported, err := importLegacyGenericdoors(migrationRoot, dataDir, legacyTLK)
if err != nil {
return NormalizeResult{}, err
}
cloakmodelImported, err := importLegacyCloakmodel(migrationRoot, dataDir, legacyTLK)
if err != nil {
return NormalizeResult{}, err
}
repadjustImported, err := importLegacyRepadjust(migrationRoot, dataDir, legacyTLK)
if err != nil {
return NormalizeResult{}, err
}
wingmodelImported, err := importLegacyWingmodel(migrationRoot, dataDir, legacyTLK)
if err != nil {
return NormalizeResult{}, err
}
tailmodelImported, err := importLegacyTailmodel(migrationRoot, dataDir, legacyTLK)
if err != nil {
return NormalizeResult{}, err
}
doortypesImported, err := importLegacyDoortypes(migrationRoot, dataDir, legacyTLK)
if err != nil {
return NormalizeResult{}, err
}
visualeffectsImported, err := importLegacyVisualeffects(migrationRoot, dataDir, legacyTLK)
if err != nil {
return NormalizeResult{}, err
}
appearanceImported, err := importLegacyAppearance(migrationRoot, dataDir, legacyTLK)
if err != nil {
return NormalizeResult{}, err
}
placeablesImported, err := importLegacyPlaceables(migrationRoot, dataDir, legacyTLK)
if err != nil {
return NormalizeResult{}, err
}
rulesetImported, err := importLegacyRuleset(migrationRoot, dataDir, legacyTLK)
if err != nil {
return NormalizeResult{}, err
}
skillsImported, err := importLegacySkills(migrationRoot, dataDir, legacyTLK)
if err != nil {
return NormalizeResult{}, err
}
spellsImported, err := importLegacySpells(migrationRoot, dataDir, legacyTLK)
if err != nil {
return NormalizeResult{}, err
}
racialtypesImported, err := importLegacyRacialtypes(migrationRoot, dataDir, legacyTLK)
if err != nil {
return NormalizeResult{}, err
}
classesImported, err := importLegacyClasses(migrationRoot, dataDir, legacyTLK)
if err != nil {
return NormalizeResult{}, err
}
portraitsImported, err := importLegacyPortraits(migrationRoot, dataDir, legacyTLK)
if err != nil {
return NormalizeResult{}, err
}
paths, err := collectDataJSONPaths(dataDir)
if err != nil {
return NormalizeResult{}, err
}
result := NormalizeResult{
StatePath: filepath.Join(sourceDir, tlkStateFile),
ScannedFiles: len(paths),
UpdatedFiles: baseDialogImported + itempropsImported + masterfeatsImported + featImported + creaturespeedImported + armorImported + baseitemsImported + damagetypesImported + skyboxesImported + vfxPersistentImported + loadscreensImported + progfxImported + genericdoorsImported + cloakmodelImported + repadjustImported + wingmodelImported + tailmodelImported + doortypesImported + visualeffectsImported + appearanceImported + placeablesImported + rulesetImported + skillsImported + spellsImported + racialtypesImported + classesImported + portraitsImported,
}
state, err := loadTLKState(result.StatePath)
if err != nil {
return result, err
}
if state.Entries == nil {
state.Entries = map[string]tlkStateMapping{}
}
if legacyTLK != nil && state.Language == "" {
state.Language = legacyTLK.Language
}
if legacyTLK != nil {
for key, id := range legacyTLK.Lock {
if _, ok := state.Entries[key]; !ok {
state.Entries[key] = tlkStateMapping{ID: id}
}
}
}
if legacyTLK != nil {
for _, path := range paths {
updated, count, err := inlineLegacyTLKFile(path, legacyTLK)
if err != nil {
return result, err
}
if updated {
result.UpdatedFiles++
result.InlineValues += count
}
}
}
if err := saveTLKState(result.StatePath, state); err != nil {
return result, err
}
if err := removeLegacyTLKAuthoredInput(filepath.Join(sourceDir, "tlk")); err != nil {
return result, err
}
remainingFiles, remainingRefs, err := countLegacyTLKRefs(paths)
if err != nil {
return result, err
}
result.RemainingFiles = remainingFiles
result.RemainingLegacyRefs = remainingRefs
return result, nil
}
func mergeLegacyTLK(base, override *legacyTLKData) *legacyTLKData {
if base == nil && override == nil {
return nil
}
merged := &legacyTLKData{
Language: "en",
Entries: map[string]tlkEntryData{},
Lock: map[string]int{},
}
if base != nil {
if base.Language != "" {
merged.Language = base.Language
}
for key, entry := range base.Entries {
merged.Entries[key] = entry
}
for key, id := range base.Lock {
merged.Lock[key] = id
}
}
if override != nil {
if override.Language != "" {
merged.Language = override.Language
}
for key, entry := range override.Entries {
merged.Entries[key] = entry
}
for key, id := range override.Lock {
merged.Lock[key] = id
}
}
return merged
}
func removeLegacyTLKAuthoredInput(tlkDir string) error {
modulesDir := filepath.Join(tlkDir, "modules")
if err := os.RemoveAll(modulesDir); err != nil {
return fmt.Errorf("remove %s: %w", modulesDir, err)
}
lockPath := filepath.Join(tlkDir, "lock.json")
if err := os.Remove(lockPath); err != nil && !os.IsNotExist(err) {
return fmt.Errorf("remove %s: %w", lockPath, err)
}
gitkeepPath := filepath.Join(tlkDir, ".gitkeep")
if err := os.Remove(gitkeepPath); err != nil && !os.IsNotExist(err) {
return fmt.Errorf("remove %s: %w", gitkeepPath, err)
}
if err := os.Remove(tlkDir); err != nil && !os.IsNotExist(err) {
if !strings.Contains(err.Error(), "directory not empty") {
return fmt.Errorf("remove %s: %w", tlkDir, err)
}
}
return nil
}
func resolveMigrationInputRoot(sourceDir, referenceBuilderDir string) string {
if strings.TrimSpace(referenceBuilderDir) != "" {
return referenceBuilderDir
}
snapshotRoot := filepath.Join(sourceDir, "migration_snapshot")
if info, err := os.Stat(snapshotRoot); err == nil && info.IsDir() {
return snapshotRoot
}
return ""
}
func inlineLegacyTLKFile(path string, legacy *legacyTLKData) (bool, int, error) {
obj, err := loadJSONObject(path)
if err != nil {
return false, 0, err
}
updated, count, err := inlineLegacyTLKValue(obj, legacy)
if err != nil {
return false, 0, err
}
if !updated {
return false, 0, nil
}
raw, err := json.MarshalIndent(obj, "", " ")
if err != nil {
return false, 0, err
}
raw = append(raw, '\n')
if err := os.WriteFile(path, raw, 0o644); err != nil {
return false, 0, err
}
return true, count, nil
}
func inlineLegacyTLKValue(value any, legacy *legacyTLKData) (bool, int, error) {
switch typed := value.(type) {
case map[string]any:
if rawTLK, ok := typed["tlk"]; ok {
key, ok := rawTLK.(string)
if ok {
entry, ok := legacy.Entries[key]
if !ok {
return false, 0, nil
}
payload := map[string]any{
"key": key,
"text": entry.Text,
}
if entry.SoundResRef != "" {
payload["sound_resref"] = entry.SoundResRef
}
if entry.SoundLength != 0 {
payload["sound_length"] = entry.SoundLength
}
typed["tlk"] = payload
return true, 1, nil
}
}
updated := false
total := 0
for key, child := range typed {
childUpdated, childCount, err := inlineLegacyTLKValue(child, legacy)
if err != nil {
return false, 0, err
}
if childUpdated {
typed[key] = child
updated = true
total += childCount
}
}
return updated, total, nil
case []any:
updated := false
total := 0
for index, child := range typed {
childUpdated, childCount, err := inlineLegacyTLKValue(child, legacy)
if err != nil {
return false, 0, err
}
if childUpdated {
typed[index] = child
updated = true
total += childCount
}
}
return updated, total, nil
default:
return false, 0, nil
}
}
func collectDataJSONPaths(dataDir string) ([]string, error) {
paths := make([]string, 0)
err := filepath.WalkDir(dataDir, func(path string, d os.DirEntry, err error) error {
if err != nil {
return err
}
if d.IsDir() {
return nil
}
if strings.HasSuffix(strings.ToLower(d.Name()), ".json") && d.Name() != "lock.json" {
paths = append(paths, path)
}
return nil
})
if err != nil {
return nil, err
}
return paths, nil
}
func countLegacyTLKRefs(paths []string) (int, int, error) {
files := 0
refs := 0
for _, path := range paths {
obj, err := loadJSONObject(path)
if err != nil {
return 0, 0, err
}
count := countLegacyTLKRefsInValue(obj)
if count == 0 {
continue
}
files++
refs += count
}
return files, refs, nil
}
func countLegacyTLKRefsInValue(value any) int {
switch typed := value.(type) {
case map[string]any:
total := 0
if rawTLK, ok := typed["tlk"]; ok {
if _, ok := rawTLK.(string); ok {
total++
}
}
for _, child := range typed {
total += countLegacyTLKRefsInValue(child)
}
return total
case []any:
total := 0
for _, child := range typed {
total += countLegacyTLKRefsInValue(child)
}
return total
default:
return 0
}
}
internal/topdata/native.go
+6936
View File
File diff suppressed because it is too large Load Diff
internal/topdata/parts_discovery.go
+692
View File
@@ -0,0 +1,692 @@
package topdata
import (
"encoding/json"
"fmt"
"os"
"path/filepath"
"regexp"
"slices"
"strconv"
"strings"
"git.westgate.pw/ShadowsOverWestgate/sow-tools/internal/project"
)
// supportedPartCategories defines the part types that should be auto-discovered
// from the sow-assets repository.
var supportedPartCategories = []string{
"belt",
"bicep",
"chest",
"foot",
"forearm",
"leg",
"neck",
"pelvis",
"robe",
"shin",
"shoulder",
}
var partDatasetToAssetCategory = map[string]string{
"belt": "belt",
"bicep": "bicep",
"chest": "chest",
"foot": "foot",
"forearm": "forearm",
"legs": "leg",
"neck": "neck",
"pelvis": "pelvis",
"robe": "robe",
"shin": "shin",
"shoulder": "shoulder",
}
// trailingNumberRegex matches trailing digits at the end of a string
var trailingNumberRegex = regexp.MustCompile(`(\d+)$`)
// isPartsDataset checks if a dataset name corresponds to a parts dataset
func isPartsDataset(datasetName string) bool {
// Check if the dataset is under parts/ directory
if strings.HasPrefix(datasetName, "parts/") {
return true
}
// Check if it's exactly a parts file (e.g., "parts/belt")
parts := strings.Split(datasetName, "/")
if len(parts) == 2 && parts[0] == "parts" {
return true
}
return false
}
func isAutogenEligiblePartsDataset(dataset nativeCollectedDataset) bool {
if !isPartsDataset(dataset.Dataset.Name) {
return false
}
for _, row := range dataset.Rows {
key, ok := row["key"].(string)
if ok && strings.TrimSpace(key) != "" {
return false
}
}
return true
}
// extractPartCategory extracts the part category from a dataset name
// e.g., "parts/belt" -> "belt", "parts/bicep" -> "bicep"
func extractPartCategory(datasetName string) string {
parts := strings.Split(datasetName, "/")
if len(parts) >= 2 {
return parts[len(parts)-1]
}
return ""
}
func partAssetCategoryForDataset(datasetName string) string {
category := extractPartCategory(datasetName)
if mapped, ok := partDatasetToAssetCategory[category]; ok {
return mapped
}
return category
}
// isSupportedPartCategory checks if the given category is in the supported list
func isSupportedPartCategory(category string) bool {
for _, c := range supportedPartCategories {
if c == category {
return true
}
}
return false
}
// extractTrailingNumericSuffix extracts the trailing numeric suffix from a filename stem
// e.g., "pfa0_belt018" -> 18, "pfa0_belt171" -> 171
// Returns 0 and false if no trailing numeric suffix is found
func extractTrailingNumericSuffix(filenameStem string) (int, bool) {
matches := trailingNumberRegex.FindStringSubmatch(filenameStem)
if len(matches) < 2 {
return 0, false
}
num, err := strconv.Atoi(matches[1])
if err != nil {
return 0, false
}
return num, true
}
// DiscoveredPartModel represents a discovered model file with its row ID
type DiscoveredPartModel struct {
RowID int
Filename string
}
// DiscoverPartModels scans the assets/part directory for .mdl files
// and returns a map of rowID -> default row data for the given category.
// The assetsDir should be the root assets directory (containing the part/ subdirectory).
func DiscoverPartModels(assetsDir string, category string) (map[int]*DiscoveredPartModel, error) {
result := make(map[int]*DiscoveredPartModel)
if !isSupportedPartCategory(category) {
return result, nil
}
partRoot, err := resolvePartsRoot(assetsDir)
if err != nil {
return nil, err
}
if partRoot == "" {
return result, nil
}
// Construct the path to the part category directory
partDir := filepath.Join(partRoot, category)
// Check if the directory exists
info, err := os.Stat(partDir)
if err != nil {
if os.IsNotExist(err) {
return result, nil
}
return nil, fmt.Errorf("failed to access part directory %s: %w", partDir, err)
}
if !info.IsDir() {
return nil, fmt.Errorf("part directory %s is not a directory", partDir)
}
// Walk the directory and find .mdl files
err = filepath.Walk(partDir, func(path string, info os.FileInfo, err error) error {
if err != nil {
return err
}
if info.IsDir() {
return nil
}
// Only process .mdl files
if !strings.HasSuffix(strings.ToLower(info.Name()), ".mdl") {
return nil
}
// Extract the filename stem (without extension)
filenameStem := strings.TrimSuffix(info.Name(), filepath.Ext(info.Name()))
// Extract trailing numeric suffix
rowID, hasSuffix := extractTrailingNumericSuffix(filenameStem)
if !hasSuffix {
return nil
}
// Store the discovered model
if existing, ok := result[rowID]; ok {
// If we already have this row ID, keep the first one found (alphabetically)
if path < existing.Filename {
result[rowID] = &DiscoveredPartModel{
RowID: rowID,
Filename: path,
}
}
} else {
result[rowID] = &DiscoveredPartModel{
RowID: rowID,
Filename: path,
}
}
return nil
})
if err != nil {
return nil, fmt.Errorf("failed to scan part directory %s: %w", partDir, err)
}
return result, nil
}
// CreateDefaultPartRow creates a default row for a discovered part model.
// Base defaults are COSTMODIFIER=0 and ACBONUS=0.00. Any robe HIDE* columns
// present in the dataset columns default to 0 as well.
func CreateDefaultPartRow(rowID int) map[string]any {
return CreateDefaultPartRowForColumns(rowID, nil)
}
func CreateDefaultPartRowForColumns(rowID int, columns []string) map[string]any {
return createDefaultPartRowForDataset(rowID, "", columns, project.PartsRowsConfig{})
}
func createDefaultPartRowForDataset(rowID int, datasetName string, columns []string, cfg project.PartsRowsConfig) map[string]any {
row := map[string]any{
"id": rowID,
"COSTMODIFIER": "0",
"ACBONUS": "0.00",
}
for field, value := range cfg.RowDefaults {
if strings.TrimSpace(field) == "" {
continue
}
row[field] = value
}
if value, ok := configuredPartsRowsACBonusValue(rowID, datasetName, cfg); ok {
row["ACBONUS"] = value
}
for _, column := range columns {
if strings.HasPrefix(column, "HIDE") {
row[column] = "0"
}
}
return row
}
func isUnsetPartValue(value any) bool {
switch typed := value.(type) {
case nil:
return true
case string:
return typed == "" || typed == "****"
}
return false
}
func applyDiscoveredPartDefaults(row map[string]any, columns []string) {
applyDiscoveredPartDefaultsWithConfig(row, "", columns, project.PartsRowsConfig{})
}
func applyDiscoveredPartDefaultsWithConfig(row map[string]any, datasetName string, columns []string, cfg project.PartsRowsConfig) {
if isUnsetPartValue(row["COSTMODIFIER"]) {
row["COSTMODIFIER"] = "0"
}
for field, value := range cfg.RowDefaults {
if strings.TrimSpace(field) == "" {
continue
}
if isUnsetPartValue(row[field]) {
row[field] = value
}
}
if isUnsetPartValue(row["ACBONUS"]) {
row["ACBONUS"] = "0.00"
}
if value, ok := configuredPartsRowsACBonusValue(rowIDFromPartRow(row), datasetName, cfg); ok {
row["ACBONUS"] = value
}
for _, column := range columns {
if !strings.HasPrefix(column, "HIDE") {
continue
}
if isUnsetPartValue(row[column]) {
row[column] = "0"
}
}
}
func rowIDFromPartRow(row map[string]any) int {
rowID, _ := row["id"].(int)
return rowID
}
func resolvePartsRoot(root string) (string, error) {
if strings.TrimSpace(root) == "" {
return "", nil
}
candidates := []string{
filepath.Join(root, "assets", "parts"),
filepath.Join(root, "assets", "part"),
filepath.Join(root, "parts"),
filepath.Join(root, "part"),
}
for _, candidate := range candidates {
info, err := os.Stat(candidate)
if err == nil && info.IsDir() {
return candidate, nil
}
if err != nil && !os.IsNotExist(err) {
return "", fmt.Errorf("failed to access parts root %s: %w", candidate, err)
}
}
return "", nil
}
// scanAutogeneratedParts scans the assets/part directory for all supported part categories
// and returns a map of category -> set of row IDs that should be auto-generated.
// The assetsDir parameter should be the path to the assets directory that contains the part/ subdirectory.
func scanAutogeneratedParts(assetsDir string) (map[string]map[int]struct{}, error) {
if assetsDir == "" {
return nil, nil
}
result := make(map[string]map[int]struct{})
for _, category := range supportedPartCategories {
discovered, err := DiscoverPartModels(assetsDir, category)
if err != nil {
return nil, err
}
// Always include the category, even if no IDs were discovered
ids := make(map[int]struct{}, len(discovered))
for rowID := range discovered {
ids[rowID] = struct{}{}
}
result[category] = ids
}
return result, nil
}
func resolveAutogeneratedPartsInventory(p *project.Project, progress func(string)) (map[string]map[int]struct{}, error) {
overrideDir, err := resolvePartsAssetsOverrideDir(p)
if err != nil {
return nil, err
}
if overrideDir != "" {
if progress != nil {
progress(fmt.Sprintf("Scanning part models from local topdata.assets override %s...", overrideDir))
}
return scanAutogeneratedParts(overrideDir)
}
return resolveReleasedPartsManifest(p, progress)
}
func hasCollectedPartsDatasets(collected []nativeCollectedDataset) bool {
for _, dataset := range collected {
if isPartsDataset(dataset.Dataset.Name) {
return true
}
}
return false
}
func loadPartOverrides(path string) ([]map[string]any, error) {
if _, err := os.Stat(path); err != nil {
if os.IsNotExist(err) {
return nil, nil
}
return nil, err
}
obj, err := loadJSONObject(path)
if err != nil {
return nil, err
}
rawOverrides, ok := obj["overrides"]
if !ok {
return nil, nil
}
overrideList, ok := rawOverrides.([]any)
if !ok {
return nil, fmt.Errorf("%s: overrides must be an array", path)
}
overrides := make([]map[string]any, 0, len(overrideList))
for index, rawOverride := range overrideList {
override, ok := rawOverride.(map[string]any)
if !ok {
return nil, fmt.Errorf("%s: override %d must be an object", path, index)
}
overrides = append(overrides, override)
}
return overrides, nil
}
func applyPartOverrides(sourceDir string, collected []nativeCollectedDataset) ([]nativeCollectedDataset, error) {
return applyPartOverridesWithConfig(sourceDir, collected, project.PartsRowsConfig{})
}
func applyPartOverridesWithConfig(sourceDir string, collected []nativeCollectedDataset, cfg project.PartsRowsConfig) ([]nativeCollectedDataset, error) {
result := make([]nativeCollectedDataset, len(collected))
copy(result, collected)
for i, dataset := range result {
if !strings.HasPrefix(dataset.Dataset.Name, "parts/") {
continue
}
category := extractPartCategory(dataset.Dataset.Name)
if category == "" {
continue
}
overrides, err := loadPartOverridesForCategory(sourceDir, category)
if err != nil {
return nil, err
}
if len(overrides) == 0 {
continue
}
rows := make([]map[string]any, len(dataset.Rows))
rowByID := make(map[int]map[string]any, len(dataset.Rows))
for index, row := range dataset.Rows {
cloned := cloneRowMap(row)
rows[index] = cloned
if rowID, ok := cloned["id"].(int); ok {
rowByID[rowID] = cloned
}
}
for index, override := range overrides {
rawID, ok := override["id"]
if !ok {
return nil, fmt.Errorf("parts/%s override %d is missing id", category, index)
}
rowID, err := asInt(rawID)
if err != nil {
return nil, fmt.Errorf("parts/%s override %d id is not numeric", category, index)
}
row, ok := rowByID[rowID]
if !ok {
row = createDefaultPartRowForDataset(rowID, dataset.Dataset.Name, dataset.Columns, cfg)
rows = append(rows, row)
rowByID[rowID] = row
}
for field, value := range override {
if field == "id" {
continue
}
row[field] = normalizePartOverrideValue(value)
}
}
slices.SortFunc(rows, func(a, b map[string]any) int {
idA := a["id"].(int)
idB := b["id"].(int)
return idA - idB
})
result[i].Rows = rows
}
return result, nil
}
func normalizePartsRowsACBonus(collected []nativeCollectedDataset, cfg project.PartsRowsConfig) ([]nativeCollectedDataset, error) {
if !partsRowsACBonusConfigured(cfg) && len(cfg.RowDefaults) == 0 {
return collected, nil
}
result := make([]nativeCollectedDataset, len(collected))
copy(result, collected)
for i, dataset := range result {
if !strings.HasPrefix(dataset.Dataset.Name, "parts/") {
continue
}
rows := make([]map[string]any, len(dataset.Rows))
for index, row := range dataset.Rows {
cloned := cloneRowMap(row)
for field, value := range cfg.RowDefaults {
if strings.TrimSpace(field) == "" {
continue
}
if isUnsetPartValue(cloned[field]) {
cloned[field] = value
}
}
rowID, ok := cloned["id"].(int)
if isUnsetPartValue(cloned["ACBONUS"]) {
cloned["COSTMODIFIER"] = nullValue
} else if ok {
if value, configured := configuredPartsRowsACBonusValue(rowID, dataset.Dataset.Name, cfg); configured {
cloned["ACBONUS"] = value
}
}
rows[index] = cloned
}
result[i].Rows = rows
}
return result, nil
}
func partsRowsACBonusConfigured(cfg project.PartsRowsConfig) bool {
if strings.TrimSpace(cfg.ACBonus.Default.Strategy) != "" {
return true
}
if len(cfg.ACBonus.Datasets) > 0 {
return true
}
for _, dataset := range cfg.Datasets {
if strings.TrimSpace(dataset.ACBonus.Strategy) != "" {
return true
}
}
return false
}
func configuredPartsRowsACBonusValue(rowID int, datasetName string, cfg project.PartsRowsConfig) (string, bool) {
policy, ok := partsRowsACBonusPolicyForDataset(datasetName, cfg)
if !ok {
return "", false
}
switch strings.TrimSpace(policy.Strategy) {
case "ascending_row_id_sort_key":
format := strings.TrimSpace(policy.Format)
if format == "" {
format = "%.2f"
}
return fmt.Sprintf(format, float64(rowID)/float64(policy.Divisor)), true
case "descending_row_id_sort_key":
format := strings.TrimSpace(policy.Format)
if format == "" {
format = "%.2f"
}
return fmt.Sprintf(format, float64(policy.MaxRowID-rowID)/float64(policy.Divisor)), true
case "fixed":
return policy.Value, true
default:
return "", false
}
}
func partsRowsACBonusPolicyForDataset(datasetName string, cfg project.PartsRowsConfig) (project.PartsRowsACBonusPolicy, bool) {
category := extractPartCategory(datasetName)
if policy, ok := cfg.ACBonus.Datasets[category]; ok && strings.TrimSpace(policy.Strategy) != "" {
return policy, true
}
if dataset, ok := cfg.Datasets[category]; ok && strings.TrimSpace(dataset.ACBonus.Strategy) != "" {
return dataset.ACBonus, true
}
if strings.TrimSpace(cfg.ACBonus.Default.Strategy) != "" {
return cfg.ACBonus.Default, true
}
return project.PartsRowsACBonusPolicy{}, false
}
func loadPartOverridesForCategory(sourceDir, category string) ([]map[string]any, error) {
overrides := []map[string]any{}
legacyPath := filepath.Join(sourceDir, "data", "parts", "overrides", category+".json")
legacyOverrides, err := loadPartOverrides(legacyPath)
if err != nil {
return nil, err
}
overrides = append(overrides, legacyOverrides...)
moduleDir := filepath.Join(sourceDir, "data", "parts", "modules")
modulePaths, err := partModuleOverridePaths(moduleDir, category)
if err != nil {
return nil, err
}
for _, path := range modulePaths {
moduleOverrides, err := loadPartOverrides(path)
if err != nil {
return nil, err
}
overrides = append(overrides, moduleOverrides...)
}
return overrides, nil
}
func partModuleOverridePaths(moduleDir, category string) ([]string, error) {
info, err := os.Stat(moduleDir)
if err != nil {
if os.IsNotExist(err) {
return nil, nil
}
return nil, err
}
if !info.IsDir() {
return nil, fmt.Errorf("parts module override path %s is not a directory", moduleDir)
}
var paths []string
err = filepath.WalkDir(moduleDir, func(path string, d os.DirEntry, err error) error {
if err != nil {
return err
}
if d.IsDir() {
return nil
}
if strings.ToLower(filepath.Ext(path)) != ".json" {
return nil
}
if partModuleOverrideMatchesCategory(path, category) {
paths = append(paths, path)
}
return nil
})
if err != nil {
return nil, err
}
slices.Sort(paths)
return paths, nil
}
func partModuleOverrideMatchesCategory(path, category string) bool {
stem := strings.TrimSuffix(strings.ToLower(filepath.Base(path)), strings.ToLower(filepath.Ext(path)))
category = strings.ToLower(strings.TrimSpace(category))
return stem == category ||
strings.HasPrefix(stem, category+"_") ||
strings.HasSuffix(stem, "_"+category) ||
strings.Contains(stem, "_"+category+"_")
}
func normalizePartOverrideValue(value any) any {
switch typed := value.(type) {
case float64:
if typed == float64(int(typed)) {
return strconv.Itoa(int(typed))
}
case int:
return strconv.Itoa(typed)
case json.Number:
return typed.String()
}
return value
}
// augmentWithAutogeneratedParts applies repository-backed discovery defaults.
// Missing discovered IDs are added with default values. Existing discovered rows
// retain authored values unless they still use placeholder values, in which case
// discovery activates them with engine-visible defaults.
func augmentWithAutogeneratedParts(collected []nativeCollectedDataset, autogenerated map[string]map[int]struct{}) []nativeCollectedDataset {
return augmentWithAutogeneratedPartsWithConfig(collected, autogenerated, project.PartsRowsConfig{})
}
func augmentWithAutogeneratedPartsWithConfig(collected []nativeCollectedDataset, autogenerated map[string]map[int]struct{}, cfg project.PartsRowsConfig) []nativeCollectedDataset {
if len(autogenerated) == 0 {
return collected
}
result := make([]nativeCollectedDataset, len(collected))
copy(result, collected)
for i, dataset := range collected {
if !isAutogenEligiblePartsDataset(dataset) {
continue
}
category := partAssetCategoryForDataset(dataset.Dataset.Name)
if !isSupportedPartCategory(category) {
continue
}
ids, ok := autogenerated[category]
if !ok || len(ids) == 0 {
continue
}
// Build a map of existing row IDs.
existingRows := make(map[int]map[string]any, len(dataset.Rows))
for _, row := range dataset.Rows {
if id, ok := row["id"].(int); ok {
existingRows[id] = row
}
}
// Add rows for discovered IDs that don't already exist.
newRows := make([]map[string]any, 0, len(dataset.Rows))
newRows = append(newRows, dataset.Rows...)
for rowID := range ids {
if row, exists := existingRows[rowID]; exists {
applyDiscoveredPartDefaultsWithConfig(row, dataset.Dataset.Name, dataset.Columns, cfg)
continue
}
if _, exists := existingRows[rowID]; !exists {
newRow := createDefaultPartRowForDataset(rowID, dataset.Dataset.Name, dataset.Columns, cfg)
newRows = append(newRows, newRow)
}
}
// Sort rows by ID
slices.SortFunc(newRows, func(a, b map[string]any) int {
idA := a["id"].(int)
idB := b["id"].(int)
return idA - idB
})
result[i].Rows = newRows
}
return result
}
internal/topdata/parts_manifest.go
+287
View File
@@ -0,0 +1,287 @@
package topdata
import (
"bytes"
"encoding/json"
"fmt"
"io"
"net/http"
"net/url"
"os"
"path/filepath"
"strings"
"time"
"git.westgate.pw/ShadowsOverWestgate/sow-tools/internal/project"
)
const (
defaultSowAssetsRepoOwner = "ShadowsOverWestgate"
defaultSowAssetsRepoName = "sow-assets"
defaultGiteaBaseURL = "https://gitea.westgate.pw"
partsManifestReleaseTag = "parts-manifest-current"
partsManifestAssetName = "sow-parts-manifest.json"
partsManifestCacheFileName = "sow-parts-manifest.json"
partsManifestCacheMaxAge = time.Hour
partsManifestRequestTimeout = 30 * time.Second
)
type partsManifest struct {
Repo string `json:"repo"`
Ref string `json:"ref"`
GeneratedAt string `json:"generated_at"`
Categories map[string][]int `json:"categories"`
}
type giteaRepoSpec struct {
BaseURL string
Owner string
Repo string
}
var partsManifestHTTPClient = &http.Client{Timeout: partsManifestRequestTimeout}
func resolveReleasedPartsManifest(p *project.Project, progress func(string)) (map[string]map[int]struct{}, error) {
if progress == nil {
progress = func(string) {}
}
cachePath := p.AutogenCachePath(partsManifestCacheFileName)
if strings.TrimSpace(os.Getenv(p.PartsManifestRefreshEnv())) == "" {
manifest, fresh, err := readFreshPartsManifestCache(cachePath, partsManifestCacheMaxAge)
if err != nil {
progress(fmt.Sprintf("Ignoring cached parts manifest: %v", err))
} else if fresh {
progress(fmt.Sprintf("Using cached released parts manifest from %s...", cachePath))
return inventoryFromPartsManifest(manifest), nil
}
}
spec, err := deriveSowAssetsRepoSpec(p)
if err != nil {
return nil, err
}
manifestURL, err := resolvePartsManifestAssetURL(spec)
if err != nil {
return nil, err
}
progress(fmt.Sprintf("Fetching released parts manifest from %s...", manifestURL))
manifest, err := fetchPartsManifest(manifestURL)
if err != nil {
return nil, err
}
if err := writePartsManifestCache(cachePath, manifest); err != nil {
return nil, err
}
return inventoryFromPartsManifest(manifest), nil
}
func deriveSowAssetsRepoSpec(p *project.Project) (giteaRepoSpec, error) {
serverOverride := strings.TrimSpace(os.Getenv(p.AssetsServerURLEnv()))
repoOverride := strings.TrimSpace(os.Getenv(p.AssetsRepoEnv()))
remote, err := gitOutput(p.Root, "remote", "get-url", "origin")
spec := giteaRepoSpec{BaseURL: defaultGiteaBaseURL, Owner: defaultSowAssetsRepoOwner, Repo: defaultSowAssetsRepoName}
if err == nil && strings.TrimSpace(remote) != "" {
if derived, ok := parseRepoSpec(strings.TrimSpace(remote)); ok {
derived.Repo = defaultSowAssetsRepoName
spec = derived
}
}
if serverOverride != "" {
spec.BaseURL = strings.TrimRight(serverOverride, "/")
}
if repoOverride != "" {
owner, repo, ok := parseOwnerRepo(repoOverride)
if !ok {
return giteaRepoSpec{}, fmt.Errorf("invalid SOW_ASSETS_REPO %q; expected owner/repo", repoOverride)
}
spec.Owner = owner
spec.Repo = repo
}
return spec, nil
}
func parseOwnerRepo(raw string) (string, string, bool) {
parts := strings.Split(strings.Trim(strings.TrimSuffix(raw, ".git"), "/"), "/")
if len(parts) < 2 {
return "", "", false
}
owner := strings.TrimSpace(parts[len(parts)-2])
repo := strings.TrimSpace(parts[len(parts)-1])
if owner == "" || repo == "" {
return "", "", false
}
return owner, repo, true
}
func parseRepoSpec(raw string) (giteaRepoSpec, bool) {
if strings.HasPrefix(raw, "ssh://") || strings.HasPrefix(raw, "http://") || strings.HasPrefix(raw, "https://") {
u, err := url.Parse(raw)
if err != nil || u.Host == "" {
return giteaRepoSpec{}, false
}
parts := strings.Split(strings.Trim(strings.TrimSuffix(u.Path, ".git"), "/"), "/")
if len(parts) < 2 {
return giteaRepoSpec{}, false
}
base := (&url.URL{Scheme: "https", Host: normalizeGiteaHTTPHost(u.Hostname())}).String()
if u.Scheme == "http" || u.Scheme == "https" {
base = (&url.URL{Scheme: u.Scheme, Host: u.Host}).String()
}
return giteaRepoSpec{BaseURL: strings.TrimRight(base, "/"), Owner: parts[len(parts)-2], Repo: parts[len(parts)-1]}, true
}
if strings.HasPrefix(raw, "git@") {
withoutUser := strings.TrimPrefix(raw, "git@")
hostAndPath := strings.SplitN(withoutUser, ":", 2)
if len(hostAndPath) != 2 {
return giteaRepoSpec{}, false
}
parts := strings.Split(strings.Trim(strings.TrimSuffix(hostAndPath[1], ".git"), "/"), "/")
if len(parts) < 2 {
return giteaRepoSpec{}, false
}
base := (&url.URL{Scheme: "https", Host: normalizeGiteaHTTPHost(hostAndPath[0])}).String()
return giteaRepoSpec{BaseURL: strings.TrimRight(base, "/"), Owner: parts[len(parts)-2], Repo: parts[len(parts)-1]}, true
}
return giteaRepoSpec{}, false
}
func normalizeGiteaHTTPHost(host string) string {
host = strings.TrimSpace(host)
if strings.HasPrefix(host, "git-ssh.") {
return "gitea." + strings.TrimPrefix(host, "git-ssh.")
}
return host
}
func resolvePartsManifestAssetURL(spec giteaRepoSpec) (string, error) {
apiURL := fmt.Sprintf("%s/api/v1/repos/%s/%s/releases/tags/%s", strings.TrimRight(spec.BaseURL, "/"), url.PathEscape(spec.Owner), url.PathEscape(spec.Repo), url.PathEscape(partsManifestReleaseTag))
var release struct {
Assets []struct {
Name string `json:"name"`
BrowserDownloadURL string `json:"browser_download_url"`
DownloadURL string `json:"download_url"`
} `json:"assets"`
}
if err := fetchJSON(apiURL, &release); err != nil {
return "", fmt.Errorf("fetch parts manifest release metadata: %w", err)
}
for _, asset := range release.Assets {
if asset.Name != partsManifestAssetName {
continue
}
if asset.DownloadURL != "" {
return asset.DownloadURL, nil
}
if asset.BrowserDownloadURL != "" {
return asset.BrowserDownloadURL, nil
}
}
return "", fmt.Errorf("parts manifest asset %s not found in release %s/%s:%s", partsManifestAssetName, spec.Owner, spec.Repo, partsManifestReleaseTag)
}
func fetchPartsManifest(manifestURL string) (*partsManifest, error) {
var manifest partsManifest
if err := fetchJSON(manifestURL, &manifest); err != nil {
return nil, fmt.Errorf("download parts manifest: %w", err)
}
if len(manifest.Categories) == 0 {
return nil, fmt.Errorf("download parts manifest: categories is empty")
}
return &manifest, nil
}
func readFreshPartsManifestCache(path string, maxAge time.Duration) (*partsManifest, bool, error) {
info, err := os.Stat(path)
if err != nil {
if os.IsNotExist(err) {
return nil, false, nil
}
return nil, false, err
}
if maxAge > 0 && time.Since(info.ModTime()) > maxAge {
return nil, false, nil
}
raw, err := os.ReadFile(path)
if err != nil {
return nil, false, err
}
var manifest partsManifest
if err := json.Unmarshal(raw, &manifest); err != nil {
return nil, false, err
}
if len(manifest.Categories) == 0 {
return nil, false, fmt.Errorf("categories is empty")
}
return &manifest, true, nil
}
func fetchJSON(target string, out any) error {
req, err := http.NewRequest(http.MethodGet, target, nil)
if err != nil {
return err
}
if token := strings.TrimSpace(os.Getenv("SOW_ASSETS_TOKEN")); token != "" {
req.Header.Set("Authorization", "token "+token)
} else if token := strings.TrimSpace(os.Getenv("GITEA_TOKEN")); token != "" {
req.Header.Set("Authorization", "token "+token)
}
resp, err := partsManifestHTTPClient.Do(req)
if err != nil {
return err
}
defer resp.Body.Close()
if resp.StatusCode < 200 || resp.StatusCode >= 300 {
body, _ := io.ReadAll(io.LimitReader(resp.Body, 4096))
if resp.StatusCode == http.StatusUnauthorized || resp.StatusCode == http.StatusForbidden {
return fmt.Errorf("HTTP %d from %s; set SOW_ASSETS_TOKEN or GITEA_TOKEN if the repo is private", resp.StatusCode, target)
}
return fmt.Errorf("HTTP %d from %s: %s", resp.StatusCode, target, strings.TrimSpace(string(body)))
}
decoder := json.NewDecoder(resp.Body)
if err := decoder.Decode(out); err != nil {
return err
}
return nil
}
func writePartsManifestCache(path string, manifest *partsManifest) error {
if err := os.MkdirAll(filepath.Dir(path), 0o755); err != nil {
return fmt.Errorf("create parts manifest cache dir: %w", err)
}
payload, err := json.MarshalIndent(manifest, "", " ")
if err != nil {
return fmt.Errorf("marshal parts manifest cache: %w", err)
}
payload = append(payload, '\n')
current, err := os.ReadFile(path)
if err == nil && bytes.Equal(current, payload) {
now := time.Now()
if err := os.Chtimes(path, now, now); err != nil {
return fmt.Errorf("refresh parts manifest cache timestamp: %w", err)
}
return nil
}
if err != nil && !os.IsNotExist(err) {
return fmt.Errorf("read parts manifest cache: %w", err)
}
tmpPath := path + ".tmp"
if err := os.WriteFile(tmpPath, payload, 0o644); err != nil {
return fmt.Errorf("write parts manifest cache: %w", err)
}
if err := os.Rename(tmpPath, path); err != nil {
return fmt.Errorf("replace parts manifest cache: %w", err)
}
return nil
}
func inventoryFromPartsManifest(manifest *partsManifest) map[string]map[int]struct{} {
result := make(map[string]map[int]struct{}, len(supportedPartCategories))
for _, category := range supportedPartCategories {
ids := make(map[int]struct{})
for _, rowID := range manifest.Categories[category] {
ids[rowID] = struct{}{}
}
result[category] = ids
}
return result
}
internal/topdata/parts_manifest_test.go
+26
View File
@@ -0,0 +1,26 @@
package topdata
import "testing"
func TestParseRepoSpecNormalizesWestgateSSHHostToPublicGiteaHost(t *testing.T) {
spec, ok := parseRepoSpec("ssh://git@git-ssh.westgate.pw:2222/ShadowsOverWestgate/sow-module.git")
if !ok {
t.Fatal("expected SSH remote to parse")
}
if spec.BaseURL != "https://gitea.westgate.pw" {
t.Fatalf("expected public Gitea base URL, got %q", spec.BaseURL)
}
if spec.Owner != "ShadowsOverWestgate" || spec.Repo != "sow-module" {
t.Fatalf("unexpected repo spec: %#v", spec)
}
}
func TestParseRepoSpecKeepsHTTPRemoteHost(t *testing.T) {
spec, ok := parseRepoSpec("https://example.invalid/owner/repo.git")
if !ok {
t.Fatal("expected HTTPS remote to parse")
}
if spec.BaseURL != "https://example.invalid" {
t.Fatalf("expected HTTPS base URL to be preserved, got %q", spec.BaseURL)
}
}
internal/topdata/placeables_migrate.go
+6
View File
@@ -0,0 +1,6 @@
package topdata
func importLegacyPlaceables(referenceBuilderDir, dataDir string, legacyTLK *legacyTLKData) (int, error) {
_ = legacyTLK
return importLegacyDatasetMirror(referenceBuilderDir, dataDir, "placeables", "placeables.2da", nil)
}
internal/topdata/portraits_migrate.go
+8
View File
@@ -0,0 +1,8 @@
package topdata
func importLegacyPortraits(referenceBuilderDir, dataDir string, legacyTLK *legacyTLKData) (int, error) {
if legacyTLK == nil {
return 0, nil
}
return importLegacyDatasetMirror(referenceBuilderDir, dataDir, "portraits", "portraits.2da", nil)
}
internal/topdata/progfx_migrate.go
+6
View File
@@ -0,0 +1,6 @@
package topdata
func importLegacyProgfx(referenceBuilderDir, dataDir string, legacyTLK *legacyTLKData) (int, error) {
_ = legacyTLK
return importLegacyDatasetMirror(referenceBuilderDir, dataDir, "progfx", "progfx.2da", nil)
}
internal/topdata/racialtypes_migrate.go
+282
View File
@@ -0,0 +1,282 @@
package topdata
import (
"encoding/json"
"fmt"
"os"
"path/filepath"
"slices"
"strings"
)
type legacyRacialtypesFeatTable struct {
Output string
Rows []map[string]any
}
func importLegacyRacialtypes(referenceBuilderDir, dataDir string, legacyTLK *legacyTLKData) (int, error) {
if strings.TrimSpace(referenceBuilderDir) == "" {
return 0, nil
}
legacyCoreDir := filepath.Join(referenceBuilderDir, "data", "racialtypes", "core")
legacyFeatsDir := filepath.Join(referenceBuilderDir, "data", "racialtypes", "feats")
if _, err := os.Stat(legacyCoreDir); err != nil {
if os.IsNotExist(err) {
return 0, nil
}
return 0, err
}
if _, err := os.Stat(legacyFeatsDir); err != nil {
if os.IsNotExist(err) {
return 0, nil
}
return 0, err
}
targetRoot := filepath.Join(dataDir, "racialtypes", "registry")
targetBasePath := filepath.Join(targetRoot, "base.json")
targetLockPath := filepath.Join(targetRoot, "lock.json")
targetRacesDir := filepath.Join(targetRoot, "races")
if fileExists(targetBasePath) && fileExists(targetLockPath) {
entries, err := os.ReadDir(targetRacesDir)
if err == nil {
raceFiles := 0
for _, entry := range entries {
if !entry.IsDir() && strings.HasSuffix(strings.ToLower(entry.Name()), ".json") {
raceFiles++
}
}
if raceFiles == 16 {
return 0, nil
}
}
}
baseColumns, baseRows, lockData, raceRows, err := collectLegacyRacialtypesRegistry(legacyCoreDir, legacyFeatsDir, legacyTLK)
if err != nil {
return 0, err
}
if err := removePathIfExists(filepath.Join(dataDir, "racialtypes", "core")); err != nil {
return 0, err
}
if err := removePathIfExists(filepath.Join(dataDir, "racialtypes", "feats")); err != nil {
return 0, err
}
if err := removePathIfExists(filepath.Join(targetRoot, "base_rows.json")); err != nil {
return 0, err
}
if err := removePathIfExists(targetRacesDir); err != nil {
return 0, err
}
if err := os.MkdirAll(targetRacesDir, 0o755); err != nil {
return 0, err
}
writes := []struct {
path string
obj any
}{
{
path: targetBasePath,
obj: map[string]any{
"columns": stringSliceToAny(baseColumns),
"rows": rowsToAny(baseRows),
},
},
{
path: targetLockPath,
obj: anyMapInt(lockData),
},
}
raceKeys := make([]string, 0, len(raceRows))
for key := range raceRows {
raceKeys = append(raceKeys, key)
}
slices.Sort(raceKeys)
for _, key := range raceKeys {
fileName := strings.TrimPrefix(key, "racialtypes:") + ".json"
writes = append(writes, struct {
path string
obj any
}{
path: filepath.Join(targetRacesDir, fileName),
obj: raceRows[key],
})
}
for _, write := range writes {
raw, err := json.MarshalIndent(write.obj, "", " ")
if err != nil {
return 0, err
}
raw = append(raw, '\n')
if err := os.WriteFile(write.path, raw, 0o644); err != nil {
return 0, err
}
}
return len(writes), nil
}
func collectLegacyRacialtypesRegistry(coreDir, featsDir string, legacyTLK *legacyTLKData) ([]string, []map[string]any, map[string]int, map[string]map[string]any, error) {
coreCollected, err := collectBaseDataset(nativeDataset{
Name: "racialtypes",
BasePath: filepath.Join(coreDir, "base.json"),
LockPath: filepath.Join(coreDir, "lock.json"),
ModulesDir: filepath.Join(coreDir, "modules"),
Spec: specForDataset("racialtypes"),
})
if err != nil {
return nil, nil, nil, nil, err
}
featTables, err := loadLegacyRacialtypesFeatTables(featsDir)
if err != nil {
return nil, nil, nil, nil, err
}
baseRows := make([]map[string]any, 0)
raceRows := map[string]map[string]any{}
lockData := map[string]int{}
for key, id := range coreCollected.LockData {
if strings.HasPrefix(key, "racialtypes:") {
lockData[key] = id
}
}
for _, collectedRow := range coreCollected.Rows {
row, ok := deepCopyValue(collectedRow).(map[string]any)
if !ok {
continue
}
if legacyTLK != nil {
if _, _, err := inlineLegacyTLKValue(row, legacyTLK); err != nil {
return nil, nil, nil, nil, err
}
}
key, _ := row["key"].(string)
if !strings.HasPrefix(key, "racialtypes:") {
assignRacialtypesBaseRowKey(row)
if key, _ := row["key"].(string); key != "" {
rowID, err := asInt(row["id"])
if err != nil {
return nil, nil, nil, nil, err
}
lockData[key] = rowID
}
baseRows = append(baseRows, row)
continue
}
delete(row, "id")
delete(row, "key")
raceFile := map[string]any{
"key": key,
"core": row,
}
if tableKey := extractRacialtypesFeatTableKey(row["FeatsTable"]); tableKey != "" {
table, ok := featTables[tableKey]
if !ok {
return nil, nil, nil, nil, fmt.Errorf("%s: unknown feat table %s", key, tableKey)
}
delete(row, "FeatsTable")
raceFile["feat_output"] = table.Output
raceFile["feats"] = rowsToAny(table.Rows)
}
raceRows[key] = raceFile
}
slices.SortFunc(baseRows, func(a, b map[string]any) int {
left, _ := asInt(a["id"])
right, _ := asInt(b["id"])
return left - right
})
return coreCollected.Columns, baseRows, lockData, raceRows, nil
}
func loadLegacyRacialtypesFeatTables(featsDir string) (map[string]legacyRacialtypesFeatTable, error) {
entries, err := os.ReadDir(featsDir)
if err != nil {
return nil, err
}
tables := map[string]legacyRacialtypesFeatTable{}
for _, entry := range entries {
if entry.IsDir() || filepath.Ext(entry.Name()) != ".json" {
continue
}
obj, err := loadJSONObject(filepath.Join(featsDir, entry.Name()))
if err != nil {
return nil, err
}
tableKey, _ := obj["key"].(string)
outputName, _ := obj["output"].(string)
rawRows, ok := obj["rows"].([]any)
if tableKey == "" || outputName == "" || !ok {
return nil, fmt.Errorf("%s: racialtypes feat table must define key, output, and rows", entry.Name())
}
rows := make([]map[string]any, 0, len(rawRows))
for index, raw := range rawRows {
row, ok := raw.(map[string]any)
if !ok {
return nil, fmt.Errorf("%s: row %d must be an object", entry.Name(), index)
}
rows = append(rows, row)
}
tables[tableKey] = legacyRacialtypesFeatTable{
Output: outputName,
Rows: rows,
}
}
return tables, nil
}
func extractRacialtypesFeatTableKey(value any) string {
obj, ok := value.(map[string]any)
if !ok {
return ""
}
tableKey, _ := obj["table"].(string)
return tableKey
}
func assignRacialtypesBaseRowKey(row map[string]any) {
name, _ := row["Name"].(string)
if strings.TrimSpace(name) == "" || name == nullValue {
delete(row, "key")
return
}
label, _ := row["Label"].(string)
keySuffix := normalizeRacialtypesKeySuffix(label)
if keySuffix == "" {
delete(row, "key")
return
}
row["key"] = "racialtypes:" + keySuffix
}
func normalizeRacialtypesKeySuffix(label string) string {
normalized := strings.ToLower(strings.TrimSpace(label))
if normalized == "" {
return ""
}
var b strings.Builder
lastUnderscore := false
for _, ch := range normalized {
isAlphaNum := (ch >= 'a' && ch <= 'z') || (ch >= '0' && ch <= '9')
if isAlphaNum {
b.WriteRune(ch)
lastUnderscore = false
continue
}
if lastUnderscore {
continue
}
b.WriteByte('_')
lastUnderscore = true
}
return strings.Trim(b.String(), "_")
}
internal/topdata/racialtypes_registry.go
+270
View File
@@ -0,0 +1,270 @@
package topdata
import (
"fmt"
"io/fs"
"os"
"path/filepath"
"slices"
"strings"
)
const (
racialtypesRegistryDirName = "racialtypes/registry"
racialtypesRegistryLock = "lock.json"
)
type racialtypesRegistry struct {
RootPath string
LockPath string
LockData map[string]int
BaseColumns []string
BaseRows []map[string]any
Races []map[string]any
}
func loadRacialtypesRegistry(dataDir string) (*racialtypesRegistry, error) {
root := filepath.Join(dataDir, filepath.FromSlash(racialtypesRegistryDirName))
info, err := os.Stat(root)
if err != nil {
if os.IsNotExist(err) {
return nil, nil
}
return nil, err
}
if !info.IsDir() {
return nil, fmt.Errorf("racialtypes registry root must be a directory: %s", root)
}
lockPath := filepath.Join(root, racialtypesRegistryLock)
lockData, err := loadLockfile(lockPath)
if err != nil {
return nil, err
}
basePath := filepath.Join(root, "base.json")
baseObj, err := loadJSONObject(basePath)
if err != nil {
return nil, err
}
baseColumns, err := parseColumns(baseObj, "racialtypes/registry/core")
if err != nil {
return nil, err
}
rawBaseRows, ok := baseObj["rows"].([]any)
if !ok {
return nil, fmt.Errorf("%s: rows must be an array", basePath)
}
baseRows := make([]map[string]any, 0, len(rawBaseRows))
for index, raw := range rawBaseRows {
row, ok := raw.(map[string]any)
if !ok {
return nil, fmt.Errorf("%s: row %d must be an object", basePath, index)
}
baseRows = append(baseRows, row)
}
raceRows, err := loadRacialtypesRegistryRaceRows(filepath.Join(root, "races"))
if err != nil {
return nil, err
}
return &racialtypesRegistry{
RootPath: root,
LockPath: lockPath,
LockData: lockData,
BaseColumns: baseColumns,
BaseRows: baseRows,
Races: raceRows,
}, nil
}
func loadRacialtypesRegistryRaceRows(dir string) ([]map[string]any, error) {
info, err := os.Stat(dir)
if err != nil {
if os.IsNotExist(err) {
return nil, nil
}
return nil, err
}
if !info.IsDir() {
return nil, fmt.Errorf("racialtypes registry races path must be a directory: %s", dir)
}
paths := make([]string, 0)
err = filepath.WalkDir(dir, func(path string, d fs.DirEntry, err error) error {
if err != nil {
return err
}
if d.IsDir() {
return nil
}
if strings.HasSuffix(strings.ToLower(d.Name()), ".json") {
paths = append(paths, path)
}
return nil
})
if err != nil {
return nil, err
}
slices.Sort(paths)
rows := make([]map[string]any, 0, len(paths))
for _, path := range paths {
obj, err := loadJSONObject(path)
if err != nil {
return nil, err
}
rows = append(rows, obj)
}
return rows, nil
}
func collectRacialtypesRegistryDatasets(dataDir string) ([]nativeCollectedDataset, error) {
registry, err := loadRacialtypesRegistry(dataDir)
if err != nil {
return nil, err
}
if registry == nil {
return nil, nil
}
lockModified, err := syncRacialtypesBaseLockData(registry.BaseRows, registry.LockData)
if err != nil {
return nil, err
}
raceIDs, raceLockModified, err := assignRegistryIDs(registry.Races, registry.LockData, "racialtypes:")
if err != nil {
return nil, err
}
lockModified = lockModified || raceLockModified
if lockModified {
if err := saveLockfile(registry.LockPath, registry.LockData); err != nil {
return nil, err
}
}
coreDataset := nativeDataset{
Kind: nativeDatasetBase,
Name: "racialtypes/registry/core",
OutputName: "racialtypes.2da",
Spec: specForDataset("racialtypes"),
}
featDataset := nativeDataset{
Kind: nativeDatasetPlain,
Name: "racialtypes/registry/feats",
Spec: specForDataset("racialtypes"),
}
coreRows := make([]map[string]any, 0, len(registry.BaseRows)+len(registry.Races))
rowIDSeen := map[int]string{}
for index, raw := range registry.BaseRows {
row, err := canonicalizeBaseRow(coreDataset, registry.BaseColumns, raw, index)
if err != nil {
return nil, fmt.Errorf("racialtypes registry base row %d: %w", index, err)
}
rowID := row["id"].(int)
if existing, ok := rowIDSeen[rowID]; ok {
return nil, fmt.Errorf("racialtypes registry base row id %d conflicts with %s", rowID, existing)
}
rowIDSeen[rowID] = "base"
coreRows = append(coreRows, row)
}
datasets := make([]nativeCollectedDataset, 0, 16)
for _, race := range registry.Races {
key := stringField(race, "key")
if key == "" {
return nil, fmt.Errorf("racialtypes registry row is missing key")
}
rowID, ok := raceIDs[key]
if !ok {
return nil, fmt.Errorf("racialtypes registry row %s is missing lock id", key)
}
if existing, ok := rowIDSeen[rowID]; ok {
return nil, fmt.Errorf("racialtypes registry row id %d for %s conflicts with %s", rowID, key, existing)
}
rowIDSeen[rowID] = key
coreRaw, ok := race["core"].(map[string]any)
if !ok {
return nil, fmt.Errorf("%s: core must be an object", key)
}
row, err := canonicalizeEntry(coreDataset, registry.BaseColumns, rowID, key, coreRaw)
if err != nil {
return nil, fmt.Errorf("%s: %w", key, err)
}
featOutput, _ := race["feat_output"].(string)
if featOutput != "" {
row["FeatsTable"] = outputStem(featOutput)
}
coreRows = append(coreRows, row)
if featOutput == "" {
continue
}
rawFeats, ok := race["feats"].([]any)
if !ok {
return nil, fmt.Errorf("%s: feats must be an array when feat_output is set", key)
}
featRows := make([]map[string]any, 0, len(rawFeats))
for index, rawFeat := range rawFeats {
featRow, ok := rawFeat.(map[string]any)
if !ok {
return nil, fmt.Errorf("%s: feat row %d must be an object", key, index)
}
row, err := canonicalizeBaseRow(featDataset, []string{"FeatLabel", "FeatIndex"}, featRow, index)
if err != nil {
return nil, fmt.Errorf("%s: feat row %d: %w", key, index, err)
}
featRows = append(featRows, row)
}
datasets = append(datasets, newGeneratedDataset(
"racialtypes/registry/"+strings.TrimPrefix(key, "racialtypes:"),
featOutput,
[]string{"FeatLabel", "FeatIndex"},
featRows,
))
}
slices.SortFunc(coreRows, func(a, b map[string]any) int {
return a["id"].(int) - b["id"].(int)
})
slices.SortFunc(datasets, func(a, b nativeCollectedDataset) int {
return strings.Compare(a.Dataset.OutputName, b.Dataset.OutputName)
})
coreCollected := newGeneratedDataset("racialtypes/registry/core", "racialtypes.2da", registry.BaseColumns, coreRows)
out := make([]nativeCollectedDataset, 0, len(datasets)+1)
out = append(out, coreCollected)
out = append(out, datasets...)
return out, nil
}
func syncRacialtypesBaseLockData(rows []map[string]any, lockData map[string]int) (bool, error) {
modified := false
for _, raw := range rows {
key := stringField(raw, "key")
if key == "" {
continue
}
rowID, ok := raw["id"]
if !ok {
return false, fmt.Errorf("racialtypes registry base row %s is missing id", key)
}
id, err := asInt(rowID)
if err != nil {
return false, fmt.Errorf("racialtypes registry base row %s has invalid id: %w", key, err)
}
if existing, ok := lockData[key]; ok {
if existing != id {
return false, fmt.Errorf("racialtypes registry base row %s id %d does not match lock id %d", key, id, existing)
}
continue
}
lockData[key] = id
modified = true
}
return modified, nil
}
internal/topdata/repadjust_migrate.go
+63
View File
@@ -0,0 +1,63 @@
package topdata
import (
"encoding/json"
"os"
"path/filepath"
)
func importLegacyRepadjust(referenceBuilderDir, dataDir string, legacyTLK *legacyTLKData) (int, error) {
_ = legacyTLK
legacyDir := filepath.Join(referenceBuilderDir, "data", "repadjust")
if _, err := os.Stat(legacyDir); err != nil {
if os.IsNotExist(err) {
return 0, nil
}
return 0, err
}
targetDir := filepath.Join(dataDir, "repadjust")
targetBasePath := filepath.Join(targetDir, "base.json")
targetLockPath := filepath.Join(targetDir, "lock.json")
if fileExists(targetBasePath) && fileExists(targetLockPath) {
obj, err := loadJSONObject(targetBasePath)
if err == nil && countLegacyTLKRefsInValue(obj) == 0 {
return 0, nil
}
}
baseObj, err := loadJSONObject(filepath.Join(legacyDir, "base.json"))
if err != nil {
return 0, err
}
baseObj["output"] = "repadjust.2da"
lockObj, err := loadJSONObject(filepath.Join(legacyDir, "lock.json"))
if err != nil {
return 0, err
}
if err := os.MkdirAll(targetDir, 0o755); err != nil {
return 0, err
}
writes := []struct {
path string
obj map[string]any
}{
{path: targetBasePath, obj: baseObj},
{path: targetLockPath, obj: lockObj},
}
for _, write := range writes {
raw, err := json.MarshalIndent(write.obj, "", " ")
if err != nil {
return 0, err
}
raw = append(raw, '\n')
if err := os.WriteFile(write.path, raw, 0o644); err != nil {
return 0, err
}
}
return len(writes), nil
}
internal/topdata/ruleset_migrate.go
+171
View File
@@ -0,0 +1,171 @@
package topdata
import (
"encoding/json"
"fmt"
"os"
"path/filepath"
"sort"
)
func importLegacyRuleset(referenceBuilderDir, dataDir string, legacyTLK *legacyTLKData) (int, error) {
_ = legacyTLK
legacyDir := filepath.Join(referenceBuilderDir, "data", "ruleset")
if _, err := os.Stat(legacyDir); err != nil {
if os.IsNotExist(err) {
return 0, nil
}
return 0, err
}
targetDir := filepath.Join(dataDir, "ruleset")
targetPath := filepath.Join(targetDir, "ruleset.json")
if _, err := os.Stat(targetPath); err == nil {
obj, err := loadJSONObject(targetPath)
if err == nil && countLegacyTLKRefsInValue(obj) == 0 {
return 0, nil
}
}
baseObj, err := loadJSONObject(filepath.Join(legacyDir, "base.json"))
if err != nil {
return 0, err
}
rows, err := mergeRulesetRows(baseObj, filepath.Join(legacyDir, "modules"))
if err != nil {
return 0, err
}
if err := os.MkdirAll(targetDir, 0o755); err != nil {
return 0, err
}
if err := removePathIfExists(filepath.Join(targetDir, "base.json")); err != nil {
return 0, err
}
if err := removePathIfExists(filepath.Join(targetDir, "lock.json")); err != nil {
return 0, err
}
if err := removePathIfExists(filepath.Join(targetDir, "modules")); err != nil {
return 0, err
}
out := map[string]any{
"output": "ruleset.2da",
"columns": baseObj["columns"],
"rows": rows,
}
raw, err := json.MarshalIndent(out, "", " ")
if err != nil {
return 0, err
}
raw = append(raw, '\n')
if err := os.WriteFile(targetPath, raw, 0o644); err != nil {
return 0, err
}
return 1, nil
}
func mergeRulesetRows(baseObj map[string]any, modulesDir string) ([]any, error) {
rawRows, ok := baseObj["rows"].([]any)
if !ok {
return nil, nil
}
rows := make([]map[string]any, 0, len(rawRows))
byLabel := map[string]map[string]any{}
for _, raw := range rawRows {
row, ok := deepCopyValue(raw).(map[string]any)
if !ok {
continue
}
delete(row, "key")
if rawID, ok := row["id"]; ok {
id, err := asInt(rawID)
if err != nil {
return nil, err
}
row["id"] = id
}
if label, ok := row["Label"].(string); ok && label != "" && label != "****" {
if _, exists := byLabel[label]; exists {
return nil, fmt.Errorf("duplicate ruleset label %q", label)
}
byLabel[label] = row
}
rows = append(rows, row)
}
modulePaths, err := collectModulePaths(modulesDir)
if err != nil {
return nil, err
}
for _, path := range modulePaths {
obj, err := loadJSONObject(path)
if err != nil {
return nil, err
}
overrideList, ok := obj["overrides"].([]any)
if !ok {
continue
}
for _, raw := range overrideList {
override, ok := raw.(map[string]any)
if !ok {
continue
}
matchObj, ok := override["match"].(map[string]any)
if !ok {
return nil, fmt.Errorf("%s: ruleset override is missing match object", path)
}
rawLabel, ok := matchObj["Label"]
if !ok {
return nil, fmt.Errorf("%s: ruleset override match is missing Label", path)
}
labels, err := normalizeRulesetMatchLabels(rawLabel)
if err != nil {
return nil, fmt.Errorf("%s: %w", path, err)
}
for _, label := range labels {
row, ok := byLabel[label]
if !ok {
return nil, fmt.Errorf("%s: ruleset override target label %q was not found", path, label)
}
for key, value := range override {
if key == "id" || key == "key" || key == "_tlk" || isMetadataField(key) || key == "match" {
continue
}
row[key] = deepCopyValue(value)
}
}
}
}
sort.Slice(rows, func(i, j int) bool {
return rows[i]["id"].(int) < rows[j]["id"].(int)
})
out := make([]any, 0, len(rows))
for _, row := range rows {
out = append(out, row)
}
return out, nil
}
func normalizeRulesetMatchLabels(raw any) ([]string, error) {
switch typed := raw.(type) {
case string:
return []string{typed}, nil
case []any:
out := make([]string, 0, len(typed))
for _, item := range typed {
label, ok := item.(string)
if !ok {
return nil, fmt.Errorf("ruleset match.Label entries must be strings")
}
out = append(out, label)
}
return out, nil
default:
return nil, fmt.Errorf("ruleset match.Label must be a string or string array")
}
}
internal/topdata/skills_migrate.go
+140
View File
@@ -0,0 +1,140 @@
package topdata
import (
"encoding/json"
"os"
"path/filepath"
)
func importLegacySkills(referenceBuilderDir, dataDir string, legacyTLK *legacyTLKData) (int, error) {
_ = legacyTLK
legacyDir := filepath.Join(referenceBuilderDir, "data", "skills")
if _, err := os.Stat(legacyDir); err != nil {
if os.IsNotExist(err) {
return 0, nil
}
return 0, err
}
targetDir := filepath.Join(dataDir, "skills")
targetBasePath := filepath.Join(targetDir, "base.json")
targetLockPath := filepath.Join(targetDir, "lock.json")
targetModulesDir := filepath.Join(targetDir, "modules")
moduleNames := []string{
"add_athletics.json",
"add_disguise.json",
"add_linguistics.json",
"add_perception.json",
"add_scriptcraft.json",
"add_sensemotive.json",
"add_stealth.json",
"add_survival.json",
"add_userope.json",
filepath.Join("craft", "add_craftalchemy.json"),
filepath.Join("craft", "add_craftcooking.json"),
filepath.Join("craft", "add_craftjewelry.json"),
filepath.Join("craft", "add_craftleatherworking.json"),
filepath.Join("craft", "add_craftstonework.json"),
filepath.Join("craft", "add_crafttextiles.json"),
filepath.Join("craft", "ovr_craftarmorsmithing.json"),
filepath.Join("craft", "ovr_crafttinkering.json"),
filepath.Join("craft", "ovr_craftweaponsmithing.json"),
filepath.Join("craft", "ovr_craftwoodworking.json"),
filepath.Join("knowledge", "add_knowledgearcana.json"),
filepath.Join("knowledge", "add_knowledgearchitecture.json"),
filepath.Join("knowledge", "add_knowledgedungeoneering.json"),
filepath.Join("knowledge", "add_knowledgegeography.json"),
filepath.Join("knowledge", "add_knowledgehistory.json"),
filepath.Join("knowledge", "add_knowledgelocal.json"),
filepath.Join("knowledge", "add_knowledgenature.json"),
filepath.Join("knowledge", "add_knowledgenobility.json"),
filepath.Join("knowledge", "add_knowledgeplanar.json"),
filepath.Join("knowledge", "add_knowledgereligion.json"),
"ovr_acrobatics.json",
"ovr_animalhandling.json",
"ovr_appraise.json",
"ovr_concentration.json",
"ovr_disabledevice.json",
"ovr_heal.json",
"ovr_hiddenskills.json",
"ovr_influence.json",
"ovr_openlock.json",
"ovr_parry.json",
"ovr_searchtowis.json",
"ovr_sleightofhand.json",
"ovr_taunttointimidate.json",
"rmv_removedskills.json",
}
targetModulePaths := make([]string, 0, len(moduleNames))
for _, name := range moduleNames {
targetModulePaths = append(targetModulePaths, filepath.Join(targetModulesDir, name))
}
if fileExists(targetBasePath) && fileExists(targetLockPath) {
allModulesPresent := true
for _, path := range targetModulePaths {
if !fileExists(path) {
allModulesPresent = false
break
}
}
if allModulesPresent {
return 0, nil
}
}
baseObj, err := loadJSONObject(filepath.Join(legacyDir, "base.json"))
if err != nil {
return 0, err
}
baseObj["output"] = "skills.2da"
canonicalizeWikiMetadataDocument(baseObj)
lockObj, err := loadJSONObject(filepath.Join(legacyDir, "lock.json"))
if err != nil {
return 0, err
}
moduleObjs := make([]map[string]any, 0, len(moduleNames))
for _, name := range moduleNames {
obj, err := loadJSONObject(filepath.Join(legacyDir, "modules", name))
if err != nil {
return 0, err
}
canonicalizeWikiMetadataDocument(obj)
moduleObjs = append(moduleObjs, obj)
}
if err := os.MkdirAll(targetModulesDir, 0o755); err != nil {
return 0, err
}
writes := []struct {
path string
obj map[string]any
}{
{path: targetBasePath, obj: baseObj},
{path: targetLockPath, obj: lockObj},
}
for i, path := range targetModulePaths {
if err := os.MkdirAll(filepath.Dir(path), 0o755); err != nil {
return 0, err
}
writes = append(writes, struct {
path string
obj map[string]any
}{path: path, obj: moduleObjs[i]})
}
for _, write := range writes {
raw, err := json.MarshalIndent(write.obj, "", " ")
if err != nil {
return 0, err
}
raw = append(raw, '\n')
if err := os.WriteFile(write.path, raw, 0o644); err != nil {
return 0, err
}
}
return len(writes), nil
}
internal/topdata/skyboxes_migrate.go
+6
View File
@@ -0,0 +1,6 @@
package topdata
func importLegacySkyboxes(referenceBuilderDir, dataDir string, legacyTLK *legacyTLKData) (int, error) {
_ = legacyTLK
return importLegacyDatasetMirror(referenceBuilderDir, dataDir, "skyboxes", "skyboxes.2da", nil)
}
internal/topdata/spells_migrate.go
+103
View File
@@ -0,0 +1,103 @@
package topdata
import (
"encoding/json"
"os"
"path/filepath"
)
func importLegacySpells(referenceBuilderDir, dataDir string, legacyTLK *legacyTLKData) (int, error) {
_ = legacyTLK
legacyDir := filepath.Join(referenceBuilderDir, "data", "spells")
if _, err := os.Stat(legacyDir); err != nil {
if os.IsNotExist(err) {
return 0, nil
}
return 0, err
}
targetDir := filepath.Join(dataDir, "spells")
targetBasePath := filepath.Join(targetDir, "base.json")
targetLockPath := filepath.Join(targetDir, "lock.json")
targetModulesDir := filepath.Join(targetDir, "modules")
moduleNames := []string{
"specialattacks.json",
"ovr_fixduplicates.json",
filepath.Join("bardicmusic", "fascinate.json"),
filepath.Join("bardicmusic", "inspirecompetence.json"),
filepath.Join("bardicmusic", "inspirecourage.json"),
filepath.Join("bardicmusic", "inspiregreatness.json"),
filepath.Join("bardicmusic", "inspireheroics.json"),
filepath.Join("bardicmusic", "songoffreedom.json"),
}
targetModulePaths := make([]string, 0, len(moduleNames))
for _, name := range moduleNames {
targetModulePaths = append(targetModulePaths, filepath.Join(targetModulesDir, name))
}
if fileExists(targetBasePath) && fileExists(targetLockPath) {
allModulesPresent := true
for _, path := range targetModulePaths {
if !fileExists(path) {
allModulesPresent = false
break
}
}
if allModulesPresent {
return 0, nil
}
}
baseObj, err := loadJSONObject(filepath.Join(legacyDir, "base.json"))
if err != nil {
return 0, err
}
baseObj["output"] = "spells.2da"
lockObj, err := loadJSONObject(filepath.Join(legacyDir, "lock.json"))
if err != nil {
return 0, err
}
moduleObjs := make([]map[string]any, 0, len(moduleNames))
for _, name := range moduleNames {
obj, err := loadJSONObject(filepath.Join(legacyDir, "modules", name))
if err != nil {
return 0, err
}
moduleObjs = append(moduleObjs, obj)
}
if err := os.MkdirAll(targetModulesDir, 0o755); err != nil {
return 0, err
}
writes := []struct {
path string
obj map[string]any
}{
{path: targetBasePath, obj: baseObj},
{path: targetLockPath, obj: lockObj},
}
for i, path := range targetModulePaths {
if err := os.MkdirAll(filepath.Dir(path), 0o755); err != nil {
return 0, err
}
writes = append(writes, struct {
path string
obj map[string]any
}{path: path, obj: moduleObjs[i]})
}
for _, write := range writes {
raw, err := json.MarshalIndent(write.obj, "", " ")
if err != nil {
return 0, err
}
raw = append(raw, '\n')
if err := os.WriteFile(write.path, raw, 0o644); err != nil {
return 0, err
}
}
return len(writes), nil
}
internal/topdata/tailmodel_migrate.go
+86
View File
@@ -0,0 +1,86 @@
package topdata
import (
"encoding/json"
"os"
"path/filepath"
)
func importLegacyTailmodel(referenceBuilderDir, dataDir string, legacyTLK *legacyTLKData) (int, error) {
_ = legacyTLK
legacyDir := filepath.Join(referenceBuilderDir, "data", "tailmodel")
if _, err := os.Stat(legacyDir); err != nil {
if os.IsNotExist(err) {
return 0, nil
}
return 0, err
}
targetDir := filepath.Join(dataDir, "tailmodel")
targetBasePath := filepath.Join(targetDir, "base.json")
targetLockPath := filepath.Join(targetDir, "lock.json")
targetModulesDir := filepath.Join(targetDir, "modules")
targetModulePaths := []string{
filepath.Join(targetModulesDir, "add.json"),
filepath.Join(targetModulesDir, "ovr_tailprefixes.json"),
}
if fileExists(targetBasePath) && fileExists(targetLockPath) {
allModulesPresent := true
for _, path := range targetModulePaths {
if !fileExists(path) {
allModulesPresent = false
break
}
}
if allModulesPresent {
return 0, nil
}
}
baseObj, err := loadJSONObject(filepath.Join(legacyDir, "base.json"))
if err != nil {
return 0, err
}
baseObj["output"] = "tailmodel.2da"
lockObj, err := loadJSONObject(filepath.Join(legacyDir, "lock.json"))
if err != nil {
return 0, err
}
moduleNames := []string{"add.json", "ovr_tailprefixes.json"}
moduleObjs := make([]map[string]any, 0, len(moduleNames))
for _, name := range moduleNames {
obj, err := loadJSONObject(filepath.Join(legacyDir, "modules", name))
if err != nil {
return 0, err
}
moduleObjs = append(moduleObjs, obj)
}
if err := os.MkdirAll(targetModulesDir, 0o755); err != nil {
return 0, err
}
writes := []struct {
path string
obj map[string]any
}{
{path: targetBasePath, obj: baseObj},
{path: targetLockPath, obj: lockObj},
{path: targetModulePaths[0], obj: moduleObjs[0]},
{path: targetModulePaths[1], obj: moduleObjs[1]},
}
for _, write := range writes {
raw, err := json.MarshalIndent(write.obj, "", " ")
if err != nil {
return 0, err
}
raw = append(raw, '\n')
if err := os.WriteFile(write.path, raw, 0o644); err != nil {
return 0, err
}
}
return len(writes), nil
}
internal/topdata/tlk_native.go
+729
View File
@@ -0,0 +1,729 @@
package topdata
import (
"bytes"
"encoding/binary"
"encoding/json"
"fmt"
"math"
"os"
"path/filepath"
"slices"
"strconv"
"strings"
"golang.org/x/text/encoding/charmap"
)
const (
defaultTLKName = "sow_tlk.tlk"
tlkStateFile = ".tlk_state.json"
customTLKBase = 0x01000000
)
type textRole string
const (
textRoleName textRole = "name"
textRoleDescription textRole = "description"
)
type datasetTextSpec struct {
NameFields []string
DescriptionFields []string
}
var datasetTextSpecs = map[string]datasetTextSpec{
"baseitems": {
NameFields: []string{"Name"},
DescriptionFields: []string{"Description"},
},
"classes": {
NameFields: []string{"Short", "Name", "Plural", "Lower", "CLASS"},
DescriptionFields: []string{"Description"},
},
"feat": {
NameFields: []string{"FEAT"},
DescriptionFields: []string{"DESCRIPTION"},
},
"masterfeats": {
NameFields: []string{"STRREF"},
DescriptionFields: []string{"DESCRIPTION"},
},
"itemprops": {
NameFields: []string{"Name", "StringRef"},
DescriptionFields: []string{"Description", "GameStrRef"},
},
"racialtypes": {
NameFields: []string{"Name", "RACIALTYPE"},
DescriptionFields: []string{"Description"},
},
"skills": {
NameFields: []string{"Name", "SKILL"},
DescriptionFields: []string{"Description"},
},
"spells": {
NameFields: []string{"Name", "SPELL"},
DescriptionFields: []string{"SpellDesc", "DESCRIPTION"},
},
}
type tlkEntryData struct {
Text string
SoundResRef string
SoundLength float32
}
type tlkStateDocument struct {
Version int `json:"version"`
Language string `json:"language"`
Entries map[string]tlkStateMapping `json:"entries"`
}
type tlkStateMapping struct {
ID int `json:"id"`
Retired bool `json:"retired,omitempty"`
}
type tlkCompiledValue struct {
Value any
TLKKey string
}
type tlkCompiler struct {
language string
statePath string
state tlkStateDocument
legacy map[string]tlkEntryData
active map[string]tlkEntryData
activeKeys map[string]struct{}
reservedByID map[int]string
nextID int
}
func newTLKCompiler(sourceDir string, legacy *legacyTLKData) (*tlkCompiler, error) {
statePath := filepath.Join(sourceDir, tlkStateFile)
state, err := loadTLKState(statePath)
if err != nil {
return nil, err
}
baseDialog, err := loadBaseDialogData(filepath.Join(sourceDir, "base_dialog.json"))
if err != nil {
return nil, err
}
legacy = mergeLegacyTLK(baseDialog, legacy)
language := state.Language
if language == "" {
language = "en"
}
if legacy != nil && legacy.Language != "" {
if state.Language == "" {
language = legacy.Language
}
for key, id := range legacy.Lock {
if _, ok := state.Entries[key]; !ok {
state.Entries[key] = tlkStateMapping{ID: id}
}
}
}
reserved := map[int]string{}
nextID := 0
for key, entry := range state.Entries {
if owner, ok := reserved[entry.ID]; ok && owner != key {
return nil, fmt.Errorf("tlk state id collision between %q and %q", owner, key)
}
reserved[entry.ID] = key
if entry.ID >= nextID {
nextID = entry.ID + 1
}
}
compiler := &tlkCompiler{
language: language,
statePath: statePath,
state: state,
legacy: map[string]tlkEntryData{},
active: map[string]tlkEntryData{},
activeKeys: map[string]struct{}{},
reservedByID: reserved,
nextID: nextID,
}
if legacy != nil {
for key, entry := range legacy.Entries {
compiler.legacy[key] = entry
}
}
return compiler, nil
}
func loadBaseDialogData(path string) (*legacyTLKData, error) {
if _, err := os.Stat(path); err != nil {
if os.IsNotExist(err) {
return nil, nil
}
return nil, fmt.Errorf("stat %s: %w", path, err)
}
obj, err := loadJSONObject(path)
if err != nil {
return nil, err
}
result := &legacyTLKData{
Language: "en",
Entries: map[string]tlkEntryData{},
Lock: map[string]int{},
}
if language, ok := obj["language"].(string); ok && strings.TrimSpace(language) != "" {
result.Language = language
}
rawEntries, ok := obj["entries"]
if !ok {
return result, nil
}
entries, err := normalizeBaseDialogEntries(rawEntries)
if err != nil {
return nil, err
}
normalized, err := normalizeLegacyTLKEntries(entries)
if err != nil {
return nil, err
}
for key, entry := range normalized {
result.Entries[key] = entry
}
return result, nil
}
func normalizeBaseDialogEntries(raw any) (map[string]any, error) {
switch typed := raw.(type) {
case map[string]any:
return typed, nil
case []any:
entries := make(map[string]any, len(typed))
for index, item := range typed {
obj, ok := item.(map[string]any)
if !ok {
return nil, fmt.Errorf("base_dialog.json entry %d must be an object", index)
}
rawID, ok := obj["id"]
if !ok {
return nil, fmt.Errorf("base_dialog.json entry %d is missing id", index)
}
key, err := legacyDialogIDKey(rawID)
if err != nil {
return nil, fmt.Errorf("base_dialog.json entry %d: %w", index, err)
}
entry := map[string]any{}
if text, ok := obj["text"]; ok {
entry["text"] = text
}
if sound, ok := obj["sound_resref"]; ok {
entry["sound_resref"] = sound
}
if length, ok := obj["sound_length"]; ok {
entry["sound_length"] = length
}
entries[key] = entry
}
return entries, nil
default:
return nil, fmt.Errorf("base_dialog.json entries must be an object or array")
}
}
func legacyDialogIDKey(raw any) (string, error) {
switch typed := raw.(type) {
case string:
if strings.TrimSpace(typed) == "" {
return "", fmt.Errorf("id must not be empty")
}
return typed, nil
case float64:
return strconv.Itoa(int(typed)), nil
case int:
return strconv.Itoa(typed), nil
default:
return "", fmt.Errorf("id must be a string or number")
}
}
func loadTLKState(path string) (tlkStateDocument, error) {
doc := tlkStateDocument{
Version: 1,
Entries: map[string]tlkStateMapping{},
}
raw, err := os.ReadFile(path)
if err != nil {
if os.IsNotExist(err) {
return doc, nil
}
return doc, fmt.Errorf("read %s: %w", path, err)
}
if err := json.Unmarshal(raw, &doc); err != nil {
return doc, fmt.Errorf("parse %s: %w", path, err)
}
if doc.Version == 0 {
doc.Version = 1
}
if doc.Entries == nil {
doc.Entries = map[string]tlkStateMapping{}
}
return doc, nil
}
func saveTLKState(path string, doc tlkStateDocument) error {
doc.Version = 1
if doc.Language == "" {
doc.Language = "en"
}
if doc.Entries == nil {
doc.Entries = map[string]tlkStateMapping{}
}
raw, err := json.MarshalIndent(doc, "", " ")
if err != nil {
return fmt.Errorf("marshal tlk state: %w", err)
}
raw = append(raw, '\n')
current, err := os.ReadFile(path)
if err == nil && bytes.Equal(current, raw) {
return nil
}
if err != nil && !os.IsNotExist(err) {
return fmt.Errorf("read %s: %w", path, err)
}
if err := os.WriteFile(path, raw, 0o644); err != nil {
return fmt.Errorf("write %s: %w", path, err)
}
return nil
}
func (c *tlkCompiler) resolveLegacyKey(key string) (tlkCompiledValue, error) {
if _, ok := c.legacy[key]; !ok {
if active, ok := c.active[key]; ok {
return tlkCompiledValue{
Value: c.customStrrefForKey(key),
TLKKey: key,
}, c.markActive(key, active)
}
if len(c.legacy) == 0 {
return tlkCompiledValue{}, fmt.Errorf("unknown TLK reference: %s (inline TLK text is required for native builds)", key)
}
return tlkCompiledValue{}, fmt.Errorf("unknown TLK reference: %s", key)
}
if err := c.markActive(key, c.legacy[key]); err != nil {
return tlkCompiledValue{}, err
}
return tlkCompiledValue{
Value: c.customStrrefForKey(key),
TLKKey: key,
}, nil
}
func (c *tlkCompiler) registerInline(key string, entry tlkEntryData) (tlkCompiledValue, error) {
if err := c.markActive(key, entry); err != nil {
return tlkCompiledValue{}, err
}
return tlkCompiledValue{
Value: c.customStrrefForKey(key),
TLKKey: key,
}, nil
}
func (c *tlkCompiler) customStrrefForKey(key string) int {
return customTLKBase + c.state.Entries[key].ID
}
func (c *tlkCompiler) markActive(key string, entry tlkEntryData) error {
mapping, ok := c.state.Entries[key]
if !ok {
mapping = tlkStateMapping{ID: c.allocateID(), Retired: false}
c.state.Entries[key] = mapping
}
existing, ok := c.active[key]
if ok && existing != entry {
return fmt.Errorf("conflicting TLK content for key %q", key)
}
c.active[key] = entry
c.activeKeys[key] = struct{}{}
mapping.Retired = false
c.state.Entries[key] = mapping
return nil
}
func (c *tlkCompiler) allocateID() int {
id := c.nextID
c.nextID++
return id
}
func (c *tlkCompiler) finish(outputDir, tlkName string) (int, error) {
if err := os.MkdirAll(outputDir, 0o755); err != nil {
return 0, fmt.Errorf("create tlk output dir: %w", err)
}
for key, mapping := range c.state.Entries {
if _, ok := c.activeKeys[key]; ok {
mapping.Retired = false
} else {
mapping.Retired = true
}
c.state.Entries[key] = mapping
}
pruneRetiredGeneratedFeatTLKEntries(filepath.Dir(c.statePath), c.state.Entries)
if err := saveTLKState(c.statePath, c.state); err != nil {
return 0, err
}
maxID := -1
for key := range c.activeKeys {
id := c.state.Entries[key].ID
if id > maxID {
maxID = id
}
}
entries := make([]tlkEntryData, maxID+1)
for key := range c.activeKeys {
id := c.state.Entries[key].ID
entries[id] = c.active[key]
}
if strings.TrimSpace(tlkName) == "" {
tlkName = defaultTLKName
}
if err := writeTLKBinary(filepath.Join(outputDir, tlkName), entries, c.language); err != nil {
return 0, err
}
if maxID < 0 {
return 1, nil
}
return 1, nil
}
func pruneRetiredGeneratedFeatTLKEntries(sourceDir string, entries map[string]tlkStateMapping) {
prefixes := collectGeneratedFeatTLKPrefixes(sourceDir)
for key, mapping := range entries {
if !mapping.Retired || !isRetiredGeneratedFeatTLKKey(key, prefixes) {
continue
}
delete(entries, key)
}
}
func isRetiredGeneratedFeatTLKKey(key string, prefixes []string) bool {
if !strings.HasPrefix(key, "feat:") {
return false
}
for _, prefix := range prefixes {
if strings.HasPrefix(key, prefix) {
return true
}
}
return false
}
func collectGeneratedFeatTLKPrefixes(sourceDir string) []string {
featGeneratedDir := filepath.Join(sourceDir, "data", "feat", "generated")
paths, err := collectModulePaths(featGeneratedDir)
if err != nil {
return nil
}
prefixes := make([]string, 0, len(paths))
seen := map[string]struct{}{}
for _, path := range paths {
obj, err := loadJSONObject(path)
if err != nil || !isFamilyExpansionObject(obj) {
continue
}
spec, err := parseFamilyExpansionSpec(path, obj)
if err != nil || spec.FamilyKey == "" {
continue
}
prefix := "feat:" + spec.FamilyKey + "_"
if _, ok := seen[prefix]; ok {
continue
}
seen[prefix] = struct{}{}
prefixes = append(prefixes, prefix)
}
slices.Sort(prefixes)
return prefixes
}
func writeTLKBinary(path string, entries []tlkEntryData, language string) error {
if err := os.MkdirAll(filepath.Dir(path), 0o755); err != nil {
return fmt.Errorf("create tlk output parent: %w", err)
}
file, err := os.Create(path)
if err != nil {
return fmt.Errorf("create tlk output: %w", err)
}
defer file.Close()
langID, err := languageID(language)
if err != nil {
return err
}
if _, err := file.Write([]byte("TLK V3.0")); err != nil {
return err
}
if err := binary.Write(file, binary.LittleEndian, uint32(langID)); err != nil {
return err
}
if err := binary.Write(file, binary.LittleEndian, uint32(len(entries))); err != nil {
return err
}
if err := binary.Write(file, binary.LittleEndian, uint32(20+len(entries)*40)); err != nil {
return err
}
stringData := make([][]byte, len(entries))
offset := 0
for index, entry := range entries {
flags := uint32(0)
if entry.Text != "" {
flags |= 0x1
}
if entry.SoundResRef != "" {
flags |= 0x2
}
if entry.SoundLength != 0 {
flags |= 0x4
}
encodedText, err := charmap.Windows1252.NewEncoder().Bytes([]byte(entry.Text))
if err != nil {
return fmt.Errorf("encode tlk text at row %d: %w", index, err)
}
stringData[index] = encodedText
soundBytes := make([]byte, 16)
if len(entry.SoundResRef) > 16 {
return fmt.Errorf("sound resref at row %d exceeds 16 characters", index)
}
copy(soundBytes, []byte(entry.SoundResRef))
if err := binary.Write(file, binary.LittleEndian, flags); err != nil {
return err
}
if _, err := file.Write(soundBytes); err != nil {
return err
}
if err := binary.Write(file, binary.LittleEndian, uint32(0)); err != nil {
return err
}
if err := binary.Write(file, binary.LittleEndian, uint32(0)); err != nil {
return err
}
if err := binary.Write(file, binary.LittleEndian, uint32(offset)); err != nil {
return err
}
if err := binary.Write(file, binary.LittleEndian, uint32(len(encodedText))); err != nil {
return err
}
if err := binary.Write(file, binary.LittleEndian, entry.SoundLength); err != nil {
return err
}
offset += len(encodedText)
}
for _, encoded := range stringData {
if _, err := file.Write(encoded); err != nil {
return err
}
}
return nil
}
func languageID(code string) (int, error) {
switch strings.ToLower(strings.TrimSpace(code)) {
case "", "en":
return 0, nil
case "fr":
return 1, nil
case "de":
return 2, nil
case "it":
return 3, nil
case "es":
return 4, nil
case "pl":
return 5, nil
default:
return 0, fmt.Errorf("unknown TLK language code %q", code)
}
}
func specForDataset(name string) datasetTextSpec {
best := datasetTextSpecs[name]
bestLength := 0
for prefix, spec := range datasetTextSpecs {
if name != prefix && !strings.HasPrefix(name, prefix+"/") {
continue
}
if len(prefix) <= bestLength {
continue
}
best = spec
bestLength = len(prefix)
}
return best
}
func roleForField(spec datasetTextSpec, field string) textRole {
for _, candidate := range spec.NameFields {
if strings.EqualFold(candidate, field) {
return textRoleName
}
}
for _, candidate := range spec.DescriptionFields {
if strings.EqualFold(candidate, field) {
return textRoleDescription
}
}
return textRole(strings.ToLower(field))
}
func columnForRole(columns []string, spec datasetTextSpec, role textRole) string {
var candidates []string
switch role {
case textRoleName:
candidates = spec.NameFields
case textRoleDescription:
candidates = spec.DescriptionFields
default:
return ""
}
for _, candidate := range candidates {
if column, ok := canonicalColumn(columns, candidate); ok {
return column
}
}
return ""
}
type parsedTLKPayload struct {
Key string
Text string
Ref string
RefField string
SoundResRef string
SoundLength float32
}
func parseTLKPayload(value any, allowBare bool) (parsedTLKPayload, bool, error) {
switch typed := value.(type) {
case map[string]any:
if rawTLK, ok := typed["tlk"]; ok {
if key, ok := rawTLK.(string); ok {
return parsedTLKPayload{Key: key}, true, nil
}
return parseTLKPayload(rawTLK, true)
}
if allowBare || hasNonRefTLKPayloadKeys(typed) {
if _, ok := typed["text"]; ok {
return buildParsedTLKPayload(typed)
}
if _, ok := typed["ref"]; ok {
return buildParsedTLKPayload(typed)
}
if _, ok := typed["key"]; ok {
return buildParsedTLKPayload(typed)
}
}
return parsedTLKPayload{}, false, nil
case string:
return parsedTLKPayload{}, false, nil
default:
return parsedTLKPayload{}, false, nil
}
}
func hasNonRefTLKPayloadKeys(obj map[string]any) bool {
for _, key := range []string{"text", "key", "sound_resref", "sound_length"} {
if _, ok := obj[key]; ok {
return true
}
}
return false
}
func buildParsedTLKPayload(obj map[string]any) (parsedTLKPayload, bool, error) {
payload := parsedTLKPayload{}
if rawKey, ok := obj["key"]; ok {
key, ok := rawKey.(string)
if !ok {
return payload, false, fmt.Errorf("tlk key must be a string")
}
payload.Key = key
}
if rawText, ok := obj["text"]; ok {
text, ok := rawText.(string)
if !ok {
return payload, false, fmt.Errorf("tlk text must be a string")
}
payload.Text = text
}
if rawRef, ok := obj["ref"]; ok {
ref, ok := rawRef.(string)
if !ok {
return payload, false, fmt.Errorf("tlk ref must be a string")
}
payload.Ref = ref
}
if rawField, ok := obj["field"]; ok {
field, ok := rawField.(string)
if !ok {
return payload, false, fmt.Errorf("tlk field must be a string")
}
payload.RefField = field
}
if rawSound, ok := obj["sound_resref"]; ok {
sound, ok := rawSound.(string)
if !ok {
return payload, false, fmt.Errorf("tlk sound_resref must be a string")
}
payload.SoundResRef = sound
}
if rawLength, ok := obj["sound_length"]; ok {
switch typed := rawLength.(type) {
case float64:
payload.SoundLength = float32(typed)
case int:
payload.SoundLength = float32(typed)
default:
return payload, false, fmt.Errorf("tlk sound_length must be numeric")
}
}
if payload.Ref != "" && payload.RefField == "" {
return payload, false, fmt.Errorf("tlk ref requires field")
}
if payload.Ref != "" && payload.Text != "" {
return payload, false, fmt.Errorf("tlk payload cannot contain both text and ref")
}
if payload.Ref == "" && payload.Text == "" && payload.Key == "" {
return payload, false, fmt.Errorf("tlk payload must contain text, ref, or key")
}
return payload, true, nil
}
func deriveAutoTLKKey(rowIdentity string, field string) string {
fieldText := strings.ToLower(strings.TrimSpace(field))
if fieldText == "" {
fieldText = "text"
}
return rowIdentity + "." + fieldText
}
func sortedKeys[M ~map[string]V, V any](input M) []string {
keys := make([]string, 0, len(input))
for key := range input {
keys = append(keys, key)
}
slices.Sort(keys)
return keys
}
func almostEqualFloat32(a, b float32) bool {
return math.Abs(float64(a-b)) < 0.0001
}
internal/topdata/top_package.go
+713
View File
@@ -0,0 +1,713 @@
package topdata
import (
"errors"
"fmt"
"io"
"io/fs"
"os"
"path/filepath"
"slices"
"strings"
"time"
"git.westgate.pw/ShadowsOverWestgate/sow-tools/internal/erf"
"git.westgate.pw/ShadowsOverWestgate/sow-tools/internal/project"
)
const (
PackageHAKResRef = "sow_top"
PackageHAKFileName = PackageHAKResRef + ".hak"
PackageTLKFileName = defaultTLKName
)
func compiled2DAOutputDir(p *project.Project) string {
return p.TopDataCompiled2DADir()
}
func topDataBuildUsesRepoCache(p *project.Project) bool {
return p.TopDataUsesRepoCache()
}
func compiledTLKOutputPath(p *project.Project) string {
return p.TopDataCompiledTLKPath()
}
func compiledTLKOutputDir(p *project.Project) string {
return p.TopDataCompiledTLKDir()
}
func wikiOutputRootDir(p *project.Project) string {
return p.TopDataWikiRootDir()
}
func wikiOutputPagesDir(p *project.Project) string {
return p.TopDataWikiPagesDir()
}
func wikiOutputStatePath(p *project.Project) string {
return p.TopDataWikiStatePath()
}
func legacyCompiled2DAOutputDir(p *project.Project) string {
return filepath.Join(p.TopDataSourceDir(), "assets", "2da")
}
func legacyWikiOutputRootDir(p *project.Project) string {
return filepath.Join(p.TopDataSourceDir(), legacyWikiRootDirName)
}
func BuildPackage(p *project.Project, progress func(string)) (PackageResult, error) {
if progress == nil {
progress = func(string) {}
}
nativeResult, err := packagedBuildResult(p)
if err != nil {
return PackageResult{}, err
}
return packageBuiltTopData(p, nativeResult, progress)
}
func BuildAndPackage(p *project.Project, progress func(string)) (PackageResult, error) {
return BuildAndPackageWithOptions(p, BuildAndPackageOptions{}, progress)
}
type BuildAndPackageOptions struct {
Force bool
BuildWiki bool
}
func BuildAndPackageWithOptions(p *project.Project, opts BuildAndPackageOptions, progress func(string)) (PackageResult, error) {
if progress == nil {
progress = func(string) {}
}
if !opts.Force {
incrementalResult, ok, err := currentTopPackageResult(p)
if err != nil {
return PackageResult{}, err
}
if ok {
progress("Topdata outputs are current; skipping native build and top package refresh.")
return incrementalResult, nil
}
nativeResult, ok, err := currentCompiledBuildResult(p)
if err != nil {
return PackageResult{}, err
}
if ok {
progress(fmt.Sprintf("Compiled topdata outputs are current; refreshing %s and %s only.", p.TopDataPackageHAKName(), p.TopDataPackageTLKName()))
return packageBuiltTopData(p, nativeResult, progress)
}
}
return buildAndPackageFull(p, opts, progress)
}
func buildAndPackageFull(p *project.Project, opts BuildAndPackageOptions, progress func(string)) (PackageResult, error) {
nativeResult, err := BuildNativeWithOptions(p, NativeBuildOptions{BuildWiki: opts.BuildWiki, Force: opts.Force}, progress)
if err != nil {
return PackageResult{}, err
}
return packageBuiltTopData(p, nativeResult, progress)
}
func BuildPackageFromBuild(p *project.Project, nativeResult BuildResult, progress func(string)) (PackageResult, error) {
if progress == nil {
progress = func(string) {}
}
return packageBuiltTopData(p, nativeResult, progress)
}
func packageBuiltTopData(p *project.Project, nativeResult BuildResult, progress func(string)) (PackageResult, error) {
if progress == nil {
progress = func(string) {}
}
outputHAK := p.TopDataPackageHAKPath()
outputTLK := p.TopDataPackageTLKPath()
progress(fmt.Sprintf("Packaging compiled topdata resources into %s...", filepath.Base(outputHAK)))
resources, assetFiles, err := collectTopPackageResources(p, nativeResult.Output2DADir)
if err != nil {
return PackageResult{}, err
}
if err := writeHAK(outputHAK, resources); err != nil {
return PackageResult{}, err
}
progress(fmt.Sprintf("Preparing compiled TLK release output %s...", filepath.Base(outputTLK)))
sourceTLK := compiledTLKOutputPath(p)
if nativeResult.OutputTLKDir != "" {
sourceTLK, err = resolveCompiledTLKPath(nativeResult.OutputTLKDir)
if err != nil {
return PackageResult{}, err
}
}
if err := copyFile(outputTLK, sourceTLK); err != nil {
return PackageResult{}, err
}
return PackageResult{
Mode: nativeResult.Mode,
Output2DADir: nativeResult.Output2DADir,
OutputTLKDir: nativeResult.OutputTLKDir,
OutputHAKPath: outputHAK,
OutputTLKPath: outputTLK,
Files2DA: nativeResult.Files2DA,
FilesTLK: nativeResult.FilesTLK,
HAKResources: len(resources),
AssetFiles: assetFiles,
WikiOutputDir: nativeResult.WikiOutputDir,
WikiPages: nativeResult.WikiPages,
WikiStatus: nativeResult.WikiStatus,
}, nil
}
func collectTopPackageResources(p *project.Project, compiled2DADir string) ([]erf.Resource, int, error) {
resourceByKey := map[string]erf.Resource{}
resourceSourceByKey := map[string]string{}
entries, err := os.ReadDir(compiled2DADir)
if err != nil {
return nil, 0, fmt.Errorf("read compiled 2da output dir: %w", err)
}
for _, entry := range entries {
if entry.IsDir() || !strings.EqualFold(filepath.Ext(entry.Name()), ".2da") {
continue
}
path := filepath.Join(compiled2DADir, entry.Name())
resource, err := topPackageResourceFromPath(path)
if err != nil {
return nil, 0, err
}
key := topPackageResourceKey(resource)
resourceByKey[key] = resource
resourceSourceByKey[key] = path
}
assetFiles := 0
assetsDir := filepath.Join(p.TopDataSourceDir(), "assets")
skipDirs := map[string]struct{}{
filepath.Clean(compiled2DADir): {},
filepath.Clean(legacyCompiled2DAOutputDir(p)): {},
}
info, err := os.Stat(assetsDir)
if err == nil && info.IsDir() {
err = filepath.WalkDir(assetsDir, func(path string, d os.DirEntry, walkErr error) error {
if walkErr != nil {
return walkErr
}
if d.IsDir() {
if _, ok := skipDirs[filepath.Clean(path)]; ok {
return filepath.SkipDir
}
}
if d.IsDir() {
return nil
}
if strings.HasPrefix(filepath.Base(path), ".") {
return nil
}
resource, err := topPackageResourceFromPath(path)
if err != nil {
return err
}
key := topPackageResourceKey(resource)
if existing, ok := resourceSourceByKey[key]; ok {
return fmt.Errorf("topdata asset %s collides with %s for top package resource %s", path, existing, key)
}
resourceByKey[key] = resource
resourceSourceByKey[key] = path
assetFiles++
return nil
})
if err != nil {
return nil, 0, err
}
} else if err != nil && !os.IsNotExist(err) {
return nil, 0, fmt.Errorf("stat topdata assets dir: %w", err)
}
keys := make([]string, 0, len(resourceByKey))
for key := range resourceByKey {
keys = append(keys, key)
}
slices.Sort(keys)
resources := make([]erf.Resource, 0, len(keys))
for _, key := range keys {
resources = append(resources, resourceByKey[key])
}
return resources, assetFiles, nil
}
func shouldSkipTopDataSourceDir(path string, skipDirs map[string]struct{}) bool {
_, ok := skipDirs[filepath.Clean(path)]
return ok
}
func topDataSourceSkipDirs(p *project.Project) map[string]struct{} {
return map[string]struct{}{
filepath.Clean(filepath.Join(p.TopDataSourceDir(), "templates")): {},
filepath.Clean(legacyWikiOutputRootDir(p)): {},
filepath.Clean(legacyCompiled2DAOutputDir(p)): {},
filepath.Clean(wikiOutputRootDir(p)): {},
}
}
func newestTopDataSource(p *project.Project) (time.Time, string, error) {
newest := time.Time{}
newestPath := ""
skipDirs := topDataSourceSkipDirs(p)
sourceDir := p.TopDataSourceDir()
err := filepath.WalkDir(sourceDir, func(path string, d fs.DirEntry, walkErr error) error {
if walkErr != nil {
return walkErr
}
if d.IsDir() && shouldSkipTopDataSourceDir(path, skipDirs) {
return filepath.SkipDir
}
if path == sourceDir {
return nil
}
info, err := d.Info()
if err != nil {
return err
}
if newest.IsZero() || info.ModTime().After(newest) {
newest = info.ModTime()
newestPath = path
}
return nil
})
if err != nil {
return time.Time{}, "", fmt.Errorf("scan topdata sources: %w", err)
}
return newest, newestPath, nil
}
func newestTopDataInput(p *project.Project, now time.Time) (time.Time, string, error) {
newest, newestPath, err := newestTopDataSource(p)
if err != nil {
return time.Time{}, "", err
}
autogenTime, autogenPath, err := newestAutogenInput(p, now)
if err != nil {
return time.Time{}, "", err
}
if autogenTime.After(newest) {
return autogenTime, autogenPath, nil
}
return newest, newestPath, nil
}
func newestAutogenInput(p *project.Project, now time.Time) (time.Time, string, error) {
newest := time.Time{}
newestPath := ""
for _, consumer := range p.Config.Autogen.Consumers {
overrideRoot, err := resolveAutogenLocalOverrideRoot(p, consumer)
if err != nil {
return time.Time{}, "", err
}
if overrideRoot != "" {
candidateRoot := filepath.Join(overrideRoot, consumer.Root)
candidateTime, candidatePath, err := newestMatchingAutogenOverrideInput(candidateRoot, consumer.Include)
if err != nil {
return time.Time{}, "", err
}
if candidateTime.After(newest) {
newest = candidateTime
newestPath = candidatePath
}
continue
}
cachePath := p.AutogenCachePath(consumer.Manifest.CacheName)
candidateTime, candidatePath, err := newestReleasedAutogenManifestInput(p, consumer, now, cachePath)
if err != nil {
return time.Time{}, "", err
}
if candidateTime.After(newest) {
newest = candidateTime
newestPath = candidatePath
}
}
return newest, newestPath, nil
}
func newestMatchingAutogenOverrideInput(scanRoot string, include []string) (time.Time, string, error) {
info, err := os.Stat(scanRoot)
if err != nil {
if os.IsNotExist(err) {
return time.Time{}, "", nil
}
return time.Time{}, "", fmt.Errorf("stat autogen override root %s: %w", scanRoot, err)
}
if !info.IsDir() {
return time.Time{}, "", fmt.Errorf("autogen override root %s is not a directory", scanRoot)
}
newest := time.Time{}
newestPath := ""
err = filepath.WalkDir(scanRoot, func(path string, d fs.DirEntry, walkErr error) error {
if walkErr != nil {
return walkErr
}
if d.IsDir() {
return nil
}
rel, err := filepath.Rel(scanRoot, path)
if err != nil {
return err
}
if !matchesAutogenInclude(filepath.ToSlash(rel), include) {
return nil
}
info, err := d.Info()
if err != nil {
return err
}
if info.ModTime().After(newest) {
newest = info.ModTime()
newestPath = path
}
return nil
})
if err != nil {
return time.Time{}, "", fmt.Errorf("scan autogen override %s: %w", scanRoot, err)
}
return newest, newestPath, nil
}
func topPackageResourceFromPath(path string) (erf.Resource, error) {
extension := strings.TrimPrefix(strings.ToLower(filepath.Ext(path)), ".")
resourceType, ok := erf.HAKResourceTypeForExtension(extension)
if !ok {
return erf.Resource{}, fmt.Errorf("unsupported top package HAK resource extension %q", filepath.Ext(path))
}
info, err := os.Stat(path)
if err != nil {
return erf.Resource{}, fmt.Errorf("stat %s: %w", path, err)
}
return erf.Resource{
Name: strings.ToLower(strings.TrimSuffix(filepath.Base(path), filepath.Ext(path))),
Type: resourceType,
SourcePath: path,
Size: info.Size(),
}, nil
}
func topPackageResourceKey(resource erf.Resource) string {
extension, ok := erf.ExtensionForResourceType(resource.Type)
if !ok {
return strings.ToLower(resource.Name)
}
return strings.ToLower(resource.Name) + "." + strings.TrimPrefix(strings.ToLower(extension), ".")
}
func writeHAK(path string, resources []erf.Resource) error {
output, err := os.Create(path)
if err != nil {
return fmt.Errorf("create top package hak: %w", err)
}
defer output.Close()
if err := erf.Write(output, erf.New("HAK ", resources)); err != nil {
return fmt.Errorf("write top package hak: %w", err)
}
return nil
}
func resolveCompiledTLKPath(outputDir string) (string, error) {
entries, err := os.ReadDir(outputDir)
if err != nil {
return "", fmt.Errorf("read compiled tlk output dir: %w", err)
}
var tlkFiles []string
for _, entry := range entries {
if entry.IsDir() || !strings.EqualFold(filepath.Ext(entry.Name()), ".tlk") {
continue
}
tlkFiles = append(tlkFiles, filepath.Join(outputDir, entry.Name()))
}
if len(tlkFiles) != 1 {
return "", fmt.Errorf("expected exactly 1 compiled tlk output, found %d", len(tlkFiles))
}
return tlkFiles[0], nil
}
func copyFile(dst, src string) error {
if samePath(dst, src) {
return nil
}
input, err := os.Open(src)
if err != nil {
return fmt.Errorf("open %s: %w", src, err)
}
defer input.Close()
output, err := os.Create(dst)
if err != nil {
return fmt.Errorf("create %s: %w", dst, err)
}
defer output.Close()
if _, err := io.Copy(output, input); err != nil {
return fmt.Errorf("copy %s to %s: %w", src, dst, err)
}
return nil
}
func samePath(left, right string) bool {
leftAbs, leftErr := filepath.Abs(left)
rightAbs, rightErr := filepath.Abs(right)
if leftErr != nil || rightErr != nil {
return left == right
}
return leftAbs == rightAbs
}
func packagedBuildResult(p *project.Project) (BuildResult, error) {
if !p.HasTopData() {
return BuildResult{}, errors.New("topdata is not configured for this project")
}
output2DA := compiled2DAOutputDir(p)
outputTLK := compiledTLKOutputDir(p)
if _, err := os.Stat(output2DA); err != nil {
return BuildResult{}, fmt.Errorf("topdata build output missing: run build-topdata first")
}
if _, err := os.Stat(compiledTLKOutputPath(p)); err != nil {
return BuildResult{}, fmt.Errorf("topdata tlk output missing: run build-topdata first")
}
files2DA, oldest2DA, err := countCompiledFiles(output2DA, ".2da")
if err != nil {
return BuildResult{}, err
}
filesTLK := 1
tlkInfo, err := os.Stat(compiledTLKOutputPath(p))
if err != nil {
return BuildResult{}, fmt.Errorf("stat compiled tlk output %s: %w", compiledTLKOutputPath(p), err)
}
oldestTLK := tlkInfo.ModTime()
if files2DA == 0 {
return BuildResult{}, fmt.Errorf("topdata build output missing: run build-topdata first")
}
if err := validateCompiled2DAOutputCatalog(p, output2DA); err != nil {
return BuildResult{}, err
}
newestSource, newestPath, err := newestTopDataInput(p, time.Now())
if err != nil {
return BuildResult{}, err
}
oldestBuild := oldest2DA
if oldestBuild.IsZero() || (!oldestTLK.IsZero() && oldestTLK.Before(oldestBuild)) {
oldestBuild = oldestTLK
}
if !newestSource.IsZero() && !oldestBuild.IsZero() && newestSource.After(oldestBuild) {
return BuildResult{}, fmt.Errorf("topdata build output is older than source file %s; run build-topdata first", newestPath)
}
return BuildResult{
Mode: "native",
Output2DADir: output2DA,
OutputTLKDir: outputTLK,
Files2DA: files2DA,
FilesTLK: filesTLK,
}, nil
}
func validateCompiled2DAOutputCatalog(p *project.Project, output2DA string) error {
expected, err := expectedCompiled2DAOutputs(p)
if err != nil {
return err
}
actual, err := compiled2DAOutputNames(output2DA)
if err != nil {
return err
}
missing := make([]string, 0)
for outputName := range expected {
if _, ok := actual[outputName]; !ok {
missing = append(missing, outputName)
}
}
stale := make([]string, 0)
for outputName := range actual {
if _, ok := expected[outputName]; !ok {
stale = append(stale, outputName)
}
}
slices.Sort(missing)
slices.Sort(stale)
if len(missing) > 0 {
return fmt.Errorf("topdata build output is missing compiled 2da %s; run build-topdata first", strings.Join(missing, ", "))
}
if len(stale) > 0 {
return fmt.Errorf("topdata build output contains stale compiled 2da %s; run build-topdata first", strings.Join(stale, ", "))
}
return nil
}
func expectedCompiled2DAOutputs(p *project.Project) (map[string]struct{}, error) {
dataDir := filepath.Join(p.TopDataSourceDir(), "data")
datasets, err := discoverNativeDatasets(dataDir)
if err != nil {
return nil, err
}
registryDatasets, err := collectGeneratedRegistryDatasets(dataDir)
if err != nil {
return nil, err
}
outputs := make(map[string]struct{}, len(datasets)+len(registryDatasets))
for _, dataset := range datasets {
outputs[dataset.OutputName] = struct{}{}
}
for _, dataset := range registryDatasets {
outputs[dataset.Dataset.OutputName] = struct{}{}
}
return outputs, nil
}
func compiled2DAOutputNames(output2DA string) (map[string]struct{}, error) {
entries, err := os.ReadDir(output2DA)
if err != nil {
return nil, fmt.Errorf("read compiled 2da output dir %s: %w", output2DA, err)
}
outputs := make(map[string]struct{})
for _, entry := range entries {
if entry.IsDir() || !strings.EqualFold(filepath.Ext(entry.Name()), ".2da") {
continue
}
outputs[entry.Name()] = struct{}{}
}
return outputs, nil
}
func currentCompiledBuildResult(p *project.Project) (BuildResult, bool, error) {
nativeResult, err := packagedBuildResult(p)
if err == nil {
return nativeResult, true, nil
}
if strings.Contains(err.Error(), "run build-topdata first") {
return BuildResult{}, false, nil
}
return BuildResult{}, false, err
}
func currentTopPackageResult(p *project.Project) (PackageResult, bool, error) {
nativeResult, ok, err := currentCompiledBuildResult(p)
if err != nil || !ok {
return PackageResult{}, ok, err
}
outputHAK := p.TopDataPackageHAKPath()
outputTLK := p.TopDataPackageTLKPath()
hakInfo, err := os.Stat(outputHAK)
if err != nil {
if os.IsNotExist(err) {
return PackageResult{}, false, nil
}
return PackageResult{}, false, fmt.Errorf("stat top package hak %s: %w", outputHAK, err)
}
tlkInfo, err := os.Stat(outputTLK)
if err != nil {
if os.IsNotExist(err) {
return PackageResult{}, false, nil
}
return PackageResult{}, false, fmt.Errorf("stat top package tlk %s: %w", outputTLK, err)
}
newestSource, _, err := newestTopDataInput(p, time.Now())
if err != nil {
return PackageResult{}, false, err
}
newestCompiled, err := newestFileModTime(nativeResult.Output2DADir, ".2da")
if err != nil {
return PackageResult{}, false, err
}
compiledTLKInfo, err := os.Stat(compiledTLKOutputPath(p))
if err != nil {
return PackageResult{}, false, fmt.Errorf("stat compiled tlk output %s: %w", compiledTLKOutputPath(p), err)
}
if compiledTLKInfo.ModTime().After(newestCompiled) {
newestCompiled = compiledTLKInfo.ModTime()
}
newestInput := newestSource
if newestCompiled.After(newestInput) {
newestInput = newestCompiled
}
oldestPackage := hakInfo.ModTime()
if tlkInfo.ModTime().Before(oldestPackage) {
oldestPackage = tlkInfo.ModTime()
}
if !newestInput.IsZero() && newestInput.After(oldestPackage) {
return PackageResult{}, false, nil
}
resources, assetFiles, err := collectTopPackageResources(p, nativeResult.Output2DADir)
if err != nil {
return PackageResult{}, false, err
}
return PackageResult{
Mode: "incremental-noop",
Output2DADir: nativeResult.Output2DADir,
OutputTLKDir: nativeResult.OutputTLKDir,
OutputHAKPath: outputHAK,
OutputTLKPath: outputTLK,
Files2DA: nativeResult.Files2DA,
FilesTLK: nativeResult.FilesTLK,
HAKResources: len(resources),
AssetFiles: assetFiles,
WikiOutputDir: nativeResult.WikiOutputDir,
WikiPages: nativeResult.WikiPages,
WikiStatus: nativeResult.WikiStatus,
}, true, nil
}
func newestFileModTime(dir, extension string) (time.Time, error) {
entries, err := os.ReadDir(dir)
if err != nil {
return time.Time{}, fmt.Errorf("read output dir %s: %w", dir, err)
}
newest := time.Time{}
for _, entry := range entries {
if entry.IsDir() || !strings.EqualFold(filepath.Ext(entry.Name()), extension) {
continue
}
info, err := entry.Info()
if err != nil {
return time.Time{}, fmt.Errorf("stat output %s: %w", filepath.Join(dir, entry.Name()), err)
}
if newest.IsZero() || info.ModTime().After(newest) {
newest = info.ModTime()
}
}
return newest, nil
}
func countCompiledFiles(dir, extension string) (int, time.Time, error) {
entries, err := os.ReadDir(dir)
if err != nil {
return 0, time.Time{}, fmt.Errorf("read compiled output dir %s: %w", dir, err)
}
count := 0
oldest := time.Time{}
for _, entry := range entries {
if entry.IsDir() || !strings.EqualFold(filepath.Ext(entry.Name()), extension) {
continue
}
info, err := entry.Info()
if err != nil {
return 0, time.Time{}, fmt.Errorf("stat compiled output %s: %w", filepath.Join(dir, entry.Name()), err)
}
count++
if oldest.IsZero() || info.ModTime().Before(oldest) {
oldest = info.ModTime()
}
}
return count, oldest, nil
}
internal/topdata/topdata.go
+3009
View File
File diff suppressed because it is too large Load Diff
internal/topdata/topdata_test.go
+15115
View File
File diff suppressed because it is too large Load Diff
internal/topdata/vfx_persistent_migrate.go
+198
View File
@@ -0,0 +1,198 @@
package topdata
import (
"os"
"path/filepath"
"strings"
)
func importLegacyVFXPersistent(referenceBuilderDir, dataDir string, legacyTLK *legacyTLKData) (int, error) {
_ = legacyTLK
if strings.TrimSpace(referenceBuilderDir) == "" {
return 0, nil
}
legacyDir := filepath.Join(referenceBuilderDir, "data", "vfx_persistent")
if _, err := os.Stat(legacyDir); err != nil {
if os.IsNotExist(err) {
return 0, nil
}
return 0, err
}
targetDir := filepath.Join(dataDir, "vfx_persistent")
targetBasePath := filepath.Join(targetDir, "base.json")
targetLockPath := filepath.Join(targetDir, "lock.json")
targetModulesDir := filepath.Join(targetDir, "modules")
targetModulePath := filepath.Join(targetModulesDir, "freeformaoes.json")
if fileExists(targetBasePath) && fileExists(targetLockPath) && fileExists(targetModulePath) {
baseObj, baseErr := loadJSONObject(targetBasePath)
lockObj, lockErr := loadJSONObject(targetLockPath)
if baseErr == nil && lockErr == nil && vfxPersistentImportComplete(baseObj, lockObj) {
legacyModulePaths, err := collectJSONRelativePaths(filepath.Join(legacyDir, "modules"))
if err == nil {
targetModulePaths, err := collectJSONRelativePaths(targetModulesDir)
if err == nil && equalStringSlices(legacyModulePaths, targetModulePaths) {
return 0, nil
}
}
}
}
baseObj, err := loadJSONObject(filepath.Join(legacyDir, "base.json"))
if err != nil {
return 0, err
}
lockObj, err := synthesizeVFXPersistentKeys(baseObj)
if err != nil {
return 0, err
}
legacyLockObj, err := loadJSONObject(filepath.Join(legacyDir, "lock.json"))
if err != nil {
return 0, err
}
for key, value := range legacyLockObj {
lockObj[key] = value
}
if err := os.MkdirAll(targetModulesDir, 0o755); err != nil {
return 0, err
}
updated := 0
for _, write := range []struct {
path string
obj map[string]any
}{
{path: targetBasePath, obj: baseObj},
{path: targetLockPath, obj: lockObj},
} {
changed, err := writeJSONObjectIfChanged(write.path, write.obj)
if err != nil {
return 0, err
}
if changed {
updated++
}
}
moduleRelPaths, err := collectJSONRelativePaths(filepath.Join(legacyDir, "modules"))
if err != nil {
return 0, err
}
for _, relPath := range moduleRelPaths {
moduleObj, err := loadJSONObject(filepath.Join(legacyDir, "modules", filepath.FromSlash(relPath)))
if err != nil {
return 0, err
}
targetPath := filepath.Join(targetModulesDir, filepath.FromSlash(relPath))
if err := os.MkdirAll(filepath.Dir(targetPath), 0o755); err != nil {
return 0, err
}
changed, err := writeJSONObjectIfChanged(targetPath, moduleObj)
if err != nil {
return 0, err
}
if changed {
updated++
}
}
targetModulePaths, err := collectJSONRelativePaths(targetModulesDir)
if err != nil {
return 0, err
}
legacyModules := make(map[string]struct{}, len(moduleRelPaths))
for _, relPath := range moduleRelPaths {
legacyModules[relPath] = struct{}{}
}
for _, relPath := range targetModulePaths {
if _, ok := legacyModules[relPath]; ok {
continue
}
if err := os.Remove(filepath.Join(targetModulesDir, filepath.FromSlash(relPath))); err != nil && !os.IsNotExist(err) {
return 0, err
}
updated++
}
return updated, nil
}
func vfxPersistentImportComplete(baseObj, lockObj map[string]any) bool {
if !vfxPersistentRowsHaveKeys(baseObj) {
return false
}
expected := map[string]int{
"vfx_persistent:blankradius5": 47,
"vfx_persistent:blankradius10": 48,
"vfx_persistent:blankradius15": 49,
"vfx_persistent:blankradius20": 50,
"vfx_persistent:blankradius25": 51,
"vfx_persistent:blankradius30": 52,
}
for key, want := range expected {
raw, ok := lockObj[key]
if !ok {
return false
}
got, err := asInt(raw)
if err != nil || got != want {
return false
}
}
return true
}
func synthesizeVFXPersistentKeys(baseObj map[string]any) (map[string]any, error) {
rawRows, ok := baseObj["rows"].([]any)
if !ok {
return map[string]any{}, nil
}
lockObj := map[string]any{}
for _, raw := range rawRows {
row, ok := raw.(map[string]any)
if !ok {
continue
}
label, _ := row["LABEL"].(string)
if label == "" {
continue
}
key := "vfx_persistent:" + strings.ToLower(label)
row["key"] = key
if rawID, ok := row["id"]; ok {
lockObj[key] = rawID
}
}
return lockObj, nil
}
func vfxPersistentRowsHaveKeys(baseObj map[string]any) bool {
rawRows, ok := baseObj["rows"].([]any)
if !ok {
return false
}
for _, raw := range rawRows {
row, ok := raw.(map[string]any)
if !ok {
return false
}
key, _ := row["key"].(string)
if key == "" {
return false
}
}
return true
}
func equalStringSlices(left, right []string) bool {
if len(left) != len(right) {
return false
}
for i := range left {
if left[i] != right[i] {
return false
}
}
return true
}
internal/topdata/visualeffects_migrate.go
+6
View File
@@ -0,0 +1,6 @@
package topdata
func importLegacyVisualeffects(referenceBuilderDir, dataDir string, legacyTLK *legacyTLKData) (int, error) {
_ = legacyTLK
return importLegacyDatasetMirror(referenceBuilderDir, dataDir, "visualeffects", "visualeffects.2da", nil)
}
internal/topdata/wiki_data_providers.go
+518
View File
@@ -0,0 +1,518 @@
package topdata
import (
"fmt"
"os"
"strconv"
"strings"
"gopkg.in/yaml.v3"
)
type wikiDataProvidersDocument struct {
Providers map[string]wikiDataProviderDefinition `json:"providers" yaml:"providers"`
}
type wikiDataProviderDefinition struct {
Source string `json:"source" yaml:"source"`
Levels string `json:"levels" yaml:"levels"`
ClassFeats wikiDataProviderClassFeatConfig `json:"class_feats" yaml:"class_feats"`
Attacks wikiDataProviderAttacksConfig `json:"attacks" yaml:"attacks"`
Facts wikiDataProviderFactsConfig `json:"facts" yaml:"facts"`
Provider string `json:"provider" yaml:"provider"`
Columns int `json:"columns" yaml:"columns"`
ItemsPerColumn int `json:"items_per_column" yaml:"items_per_column"`
ForceColumns bool `json:"force_columns" yaml:"force_columns"`
}
type wikiDataProviderClassFeatConfig struct {
Fields map[string]wikiDataProviderProjectionField `json:"fields" yaml:"fields"`
}
type wikiDataProviderProjectionField struct {
When string `json:"when" yaml:"when"`
}
type wikiDataProviderFactsConfig struct {
Rows []wikiDataProviderFactRow `json:"rows" yaml:"rows"`
}
type wikiDataProviderFactRow struct {
Key string `json:"key" yaml:"key"`
Label string `json:"label" yaml:"label"`
Value string `json:"value" yaml:"value"`
When string `json:"when" yaml:"when"`
}
type wikiDataProviderAttacksConfig struct {
Field string `json:"field" yaml:"field"`
CountField string `json:"count_field" yaml:"count_field"`
Mode string `json:"mode" yaml:"mode"`
Step int `json:"step" yaml:"step"`
Attacks int `json:"attacks" yaml:"attacks"`
Thresholds []wikiDataProviderAttackThreshold `json:"thresholds" yaml:"thresholds"`
Overrides []wikiDataProviderAttackOverrideRule `json:"overrides" yaml:"overrides"`
}
type wikiDataProviderAttackThreshold struct {
MinBAB int `json:"min_bab" yaml:"min_bab"`
Attacks int `json:"attacks" yaml:"attacks"`
}
type wikiDataProviderAttackOverrideRule struct {
Classes []string `json:"classes" yaml:"classes"`
Levels any `json:"levels" yaml:"levels"`
Attacks int `json:"attacks" yaml:"attacks"`
Add int `json:"add" yaml:"add"`
}
func loadWikiDataProviders(path string) (map[string]wikiDataProviderDefinition, error) {
raw, err := os.ReadFile(path)
if err != nil {
if os.IsNotExist(err) {
return map[string]wikiDataProviderDefinition{}, nil
}
return nil, fmt.Errorf("read wiki data providers %s: %w", path, err)
}
var doc wikiDataProvidersDocument
if err := yaml.Unmarshal(raw, &doc); err != nil {
return nil, fmt.Errorf("parse wiki data providers %s: %w", path, err)
}
for name, def := range doc.Providers {
if strings.TrimSpace(name) == "" {
return nil, fmt.Errorf("wiki data providers %s: provider name must not be empty", path)
}
if strings.TrimSpace(def.Source) == "" {
return nil, fmt.Errorf("wiki data providers %s: provider %s missing source", path, name)
}
source := strings.TrimSpace(def.Source)
switch source {
case "class_progression":
for field, projection := range def.ClassFeats.Fields {
if strings.TrimSpace(field) == "" {
return nil, fmt.Errorf("wiki data providers %s: provider %s has class feat projection without field name", path, name)
}
if strings.TrimSpace(projection.When) == "" {
return nil, fmt.Errorf("wiki data providers %s: provider %s class feat projection %s missing when", path, name, field)
}
}
if err := validateWikiDataProviderAttacks(path, name, def.Attacks); err != nil {
return nil, err
}
case "columns":
if len(def.ClassFeats.Fields) > 0 {
return nil, fmt.Errorf("wiki data providers %s: provider %s class feat projections require source class_progression", path, name)
}
if strings.TrimSpace(def.Attacks.Mode) != "" {
return nil, fmt.Errorf("wiki data providers %s: provider %s attacks config requires source class_progression", path, name)
}
if strings.TrimSpace(def.Provider) == "" {
return nil, fmt.Errorf("wiki data providers %s: provider %s columns provider missing provider", path, name)
}
if strings.TrimSpace(def.Provider) == name {
return nil, fmt.Errorf("wiki data providers %s: provider %s columns provider must not reference itself", path, name)
}
if def.Columns <= 0 {
return nil, fmt.Errorf("wiki data providers %s: provider %s columns must be a positive integer", path, name)
}
if def.ItemsPerColumn <= 0 {
return nil, fmt.Errorf("wiki data providers %s: provider %s items_per_column must be a positive integer", path, name)
}
case "facts":
if len(def.Facts.Rows) == 0 {
return nil, fmt.Errorf("wiki data providers %s: provider %s facts source requires rows", path, name)
}
for index, row := range def.Facts.Rows {
if strings.TrimSpace(row.Label) == "" {
return nil, fmt.Errorf("wiki data providers %s: provider %s facts row %d missing label", path, name, index+1)
}
if strings.TrimSpace(row.Value) == "" {
return nil, fmt.Errorf("wiki data providers %s: provider %s facts row %d missing value", path, name, index+1)
}
}
case "class_feats", "class_skills", "class_summary", "class_spellbook", "race_feats", "race_name_forms":
if len(def.ClassFeats.Fields) > 0 {
return nil, fmt.Errorf("wiki data providers %s: provider %s class feat projections require source class_progression", path, name)
}
if strings.TrimSpace(def.Attacks.Mode) != "" {
return nil, fmt.Errorf("wiki data providers %s: provider %s attacks config requires source class_progression", path, name)
}
default:
return nil, fmt.Errorf("wiki data providers %s: provider %s has unknown source %q", path, name, def.Source)
}
}
for name, def := range doc.Providers {
if strings.TrimSpace(def.Source) != "columns" {
continue
}
if _, ok := doc.Providers[strings.TrimSpace(def.Provider)]; !ok {
return nil, fmt.Errorf("wiki data providers %s: provider %s columns provider references missing provider %q", path, name, def.Provider)
}
}
if doc.Providers == nil {
return map[string]wikiDataProviderDefinition{}, nil
}
return doc.Providers, nil
}
func (ctx *wikiContext) wikiDataProviderRows(name string, page wikiTemplatePage) ([]map[string]any, error) {
def, ok := ctx.wikiDataProviders[name]
if !ok {
return nil, fmt.Errorf("missing wiki data provider %q in %s", name, ctx.wikiDataPath)
}
switch strings.TrimSpace(def.Source) {
case "columns":
sourceRows, err := ctx.wikiDataProviderRows(strings.TrimSpace(def.Provider), page)
if err != nil {
return nil, err
}
return wikiColumnProviderRows(sourceRows, def.Columns, def.ItemsPerColumn, def.ForceColumns), nil
case "class_progression":
if page.Category != "classes" {
return nil, nil
}
return ctx.classProgressionRows(def.Levels, def.ClassFeats.Fields, def.Attacks, page.Key, page.Row)
case "facts":
return ctx.wikiFactRows(name, def.Facts, page)
case "class_feats":
if page.Category != "classes" {
return nil, nil
}
return ctx.classFeatRows(page.Row), nil
case "class_skills":
if page.Category != "classes" {
return nil, nil
}
return ctx.classSkillRows(page.Row), nil
case "class_summary":
if page.Category != "classes" {
return nil, nil
}
return ctx.classSummaryRows(page.Row), nil
case "class_spellbook":
if page.Category != "classes" {
return nil, nil
}
return ctx.classSpellbookRows(page.Key, page.Row), nil
case "race_feats":
if page.Category != "racialtypes" {
return nil, nil
}
return ctx.raceFeatRows(page.Row), nil
case "race_name_forms":
if page.Category != "racialtypes" {
return nil, nil
}
return ctx.raceNameFormRows(page.Row), nil
default:
return nil, fmt.Errorf("unknown wiki data provider source %q", def.Source)
}
}
func (ctx *wikiContext) wikiFactRows(providerName string, cfg wikiDataProviderFactsConfig, page wikiTemplatePage) ([]map[string]any, error) {
out := []map[string]any{}
renderCtx := wikiTableRenderContext{Page: page, TableName: providerName, Path: ctx.wikiDataPath}
for index, spec := range cfg.Rows {
if strings.TrimSpace(spec.When) != "" {
value, err := ctx.evalWikiDataProviderTemplate(spec.When, page.Row, renderCtx)
if err != nil {
return nil, fmt.Errorf("facts provider %s row %d condition %q: %w", providerName, index+1, spec.When, err)
}
if !truthyWikiTableValue(value) {
continue
}
}
value, err := ctx.evalWikiDataProviderTemplate(spec.Value, page.Row, renderCtx)
if err != nil {
return nil, fmt.Errorf("facts provider %s row %d value %q: %w", providerName, index+1, spec.Value, err)
}
text := stringifyWikiTableValue(value)
if strings.TrimSpace(text) == "" || strings.TrimSpace(text) == nullValue {
continue
}
key := strings.TrimSpace(spec.Key)
if key == "" {
key = wikiFactKeyFromLabel(spec.Label)
}
out = append(out, map[string]any{
"Key": key,
"Label": spec.Label,
"Value": text,
})
}
return out, nil
}
func (ctx *wikiContext) evalWikiDataProviderTemplate(template string, row map[string]any, renderCtx wikiTableRenderContext) (any, error) {
template = strings.TrimSpace(template)
if strings.HasPrefix(template, "{{") && strings.HasSuffix(template, "}}") {
expr := strings.TrimSpace(strings.TrimSuffix(strings.TrimPrefix(template, "{{"), "}}"))
parser := wikiExprParser{tokens: tokenizeWikiExpr(expr), row: row, ctx: ctx, renderCtx: renderCtx, allowMissingFields: true}
value, err := parser.parseComparison()
if err != nil {
return nil, err
}
if parser.peek().kind != wikiExprEOF {
return nil, fmt.Errorf("unexpected token %q", parser.peek().text)
}
return value, nil
}
if strings.Contains(template, "{{") {
return nil, fmt.Errorf("mixed literal/expression templates are not supported")
}
return template, nil
}
func wikiFactKeyFromLabel(label string) string {
key := strings.ToLower(strings.TrimSpace(label))
key = strings.Map(func(r rune) rune {
switch {
case r >= 'a' && r <= 'z':
return r
case r >= '0' && r <= '9':
return r
case r == ' ' || r == '-' || r == '_':
return '_'
default:
return -1
}
}, key)
for strings.Contains(key, "__") {
key = strings.ReplaceAll(key, "__", "_")
}
return strings.Trim(key, "_")
}
func validateWikiDataProviderAttacks(path, provider string, cfg wikiDataProviderAttacksConfig) error {
if strings.TrimSpace(cfg.Mode) == "" {
return nil
}
switch strings.TrimSpace(cfg.Mode) {
case "fixed":
if cfg.Attacks < 1 {
return fmt.Errorf("wiki data providers %s: provider %s attacks fixed mode requires positive attacks", path, provider)
}
case "bab_thresholds":
if len(cfg.Thresholds) == 0 {
return fmt.Errorf("wiki data providers %s: provider %s attacks bab_thresholds mode requires thresholds", path, provider)
}
for i, threshold := range cfg.Thresholds {
if threshold.MinBAB < 0 {
return fmt.Errorf("wiki data providers %s: provider %s attacks threshold %d min_bab must be non-negative", path, provider, i+1)
}
if threshold.Attacks < 1 {
return fmt.Errorf("wiki data providers %s: provider %s attacks threshold %d attacks must be positive", path, provider, i+1)
}
}
default:
return fmt.Errorf("wiki data providers %s: provider %s attacks has unknown mode %q", path, provider, cfg.Mode)
}
for i, override := range cfg.Overrides {
if len(override.Classes) == 0 {
return fmt.Errorf("wiki data providers %s: provider %s attacks override %d missing classes", path, provider, i+1)
}
for _, class := range override.Classes {
if strings.TrimSpace(class) == "" {
return fmt.Errorf("wiki data providers %s: provider %s attacks override %d has empty class", path, provider, i+1)
}
}
if _, err := parseWikiAttackLevels(override.Levels); err != nil {
return fmt.Errorf("wiki data providers %s: provider %s attacks override %d bad levels: %w", path, provider, i+1, err)
}
if override.Attacks != 0 && override.Add != 0 {
return fmt.Errorf("wiki data providers %s: provider %s attacks override %d must not set both attacks and add", path, provider, i+1)
}
if override.Attacks == 0 && override.Add == 0 {
return fmt.Errorf("wiki data providers %s: provider %s attacks override %d must set attacks or add", path, provider, i+1)
}
if override.Attacks < 0 {
return fmt.Errorf("wiki data providers %s: provider %s attacks override %d attacks must be positive", path, provider, i+1)
}
}
return nil
}
func parseWikiAttackLevels(raw any) (map[int]struct{}, error) {
out := map[int]struct{}{}
addLevel := func(value any) error {
switch typed := value.(type) {
case int:
if typed < 1 {
return fmt.Errorf("level must be positive")
}
out[typed] = struct{}{}
return nil
case int64:
if typed < 1 {
return fmt.Errorf("level must be positive")
}
out[int(typed)] = struct{}{}
return nil
case float64:
level := int(typed)
if typed != float64(level) || level < 1 {
return fmt.Errorf("level must be a positive integer")
}
out[level] = struct{}{}
return nil
case string:
text := strings.TrimSpace(typed)
if text == "" {
return fmt.Errorf("level must not be empty")
}
if strings.Contains(text, "-") {
start, end, err := parseLevelRange(text)
if err != nil {
return err
}
for level := start; level <= end; level++ {
out[level] = struct{}{}
}
return nil
}
level, err := strconv.Atoi(text)
if err != nil || level < 1 {
return fmt.Errorf("level must be a positive integer")
}
out[level] = struct{}{}
return nil
default:
return fmt.Errorf("unsupported level value")
}
}
switch typed := raw.(type) {
case nil:
return nil, fmt.Errorf("levels is required")
case []any:
if len(typed) == 0 {
return nil, fmt.Errorf("levels must not be empty")
}
for _, value := range typed {
if err := addLevel(value); err != nil {
return nil, err
}
}
default:
if err := addLevel(raw); err != nil {
return nil, err
}
}
return out, nil
}
func wikiColumnProviderRows(sourceRows []map[string]any, configuredColumns, itemsPerColumn int, forceColumns bool) []map[string]any {
if configuredColumns <= 0 || itemsPerColumn <= 0 {
return nil
}
needed := 0
if len(sourceRows) > 0 {
needed = (len(sourceRows) + itemsPerColumn - 1) / itemsPerColumn
}
total := needed
if forceColumns && configuredColumns > total {
total = configuredColumns
}
if total == 0 {
return nil
}
rows := make([]map[string]any, 0, total)
for i := 0; i < total; i++ {
start := i * itemsPerColumn
end := start + itemsPerColumn
if start > len(sourceRows) {
start = len(sourceRows)
}
if end > len(sourceRows) {
end = len(sourceRows)
}
items := make([]map[string]any, 0, end-start)
for _, source := range sourceRows[start:end] {
items = append(items, cloneRowMap(source))
}
rows = append(rows, map[string]any{
"Index": i + 1,
"Count": len(items),
"Items": items,
})
}
return rows
}
func (ctx *wikiContext) classSkillRows(classRow map[string]any) []map[string]any {
table := ctx.tableForValue(fieldValue(classRow, "SkillsTable"), ctx.classSkillTables)
if table == nil {
return nil
}
rows := []map[string]any{}
for _, source := range table.Rows {
if stringValue(source, "ClassSkill") != "1" {
continue
}
key := resolveReferenceKey(fieldValue(source, "SkillIndex"), ctx.skillIDToKey)
if key == "" {
key = resolveReferenceKey(fieldValue(source, "SkillIndex"), nil)
}
if key == "" || !ctx.canReferenceWikiKey(key) {
continue
}
row := cloneRowMap(source)
row["SkillKey"] = key
row["SkillName"] = ctx.resolveSkillName(key)
if skillRow := ctx.skillRows[key]; skillRow != nil {
row["SkillAbility"] = stringValue(skillRow, "KeyAbility")
}
rows = append(rows, row)
}
return rows
}
func (ctx *wikiContext) raceFeatRows(raceRow map[string]any) []map[string]any {
values := []any{}
switch typed := fieldValue(raceRow, "FeatsTable").(type) {
case []any:
values = append(values, typed...)
case []string:
for _, value := range typed {
values = append(values, value)
}
default:
if table := ctx.tableForValue(typed, ctx.raceFeatTables); table != nil {
for _, row := range table.Rows {
values = append(values, fieldValue(row, "FeatIndex"))
}
}
}
rows := []map[string]any{}
for _, value := range values {
key := resolveReferenceKey(value, ctx.featIDToKey)
if key == "" {
key = resolveReferenceKey(value, nil)
}
if key == "" || !ctx.canReferenceWikiKey(key) {
continue
}
rows = append(rows, map[string]any{
"FeatKey": key,
"FeatName": ctx.resolveFeatName(key),
})
}
return rows
}
func (ctx *wikiContext) raceNameFormRows(raceRow map[string]any) []map[string]any {
candidates := []struct {
label string
field string
}{
{"Plural", "NamePlural"},
{"Converted Name", "ConverName"},
{"Lower Name", "ConverNameLower"},
}
rows := []map[string]any{}
for _, candidate := range candidates {
if value := ctx.resolveTextValue(fieldValue(raceRow, candidate.field), nil); value != "" {
rows = append(rows, map[string]any{"Label": candidate.label, "Value": value})
}
}
return rows
}
internal/topdata/wiki_deploy.go
+1861
View File
File diff suppressed because it is too large Load Diff
internal/topdata/wiki_deploy_test.go
+2387
View File
File diff suppressed because it is too large Load Diff
internal/topdata/wiki_formatters.go
+286
View File
@@ -0,0 +1,286 @@
package topdata
import (
"fmt"
"html"
"strings"
)
func (ctx *wikiContext) renderWikiFormatter(spec string, page wikiTemplatePage) (string, error) {
name := strings.Fields(spec)
if len(name) == 0 {
return "", fmt.Errorf("render wiki template for %s: empty formatter", page.PageID)
}
switch name[0] {
case "Description":
return renderWikiTemplatePlainHTML(ctx.resolveRowDescription(page.Category, page.Row)), nil
case "FactsTable":
return ctx.renderFactsHTML(page.Category, page.Key, page.Row), nil
case "Prerequisites":
return renderWikiFactTableHTML([]wikiHTMLFact{{Label: "Prerequisites", Value: ctx.renderFeatPrerequisites(page.Row)}}), nil
case "SkillList":
if page.Category != "classes" {
return "", nil
}
return ctx.renderClassSkillListHTML(page.Row), nil
case "ClassFeatureTable":
if page.Category != "classes" {
return "", nil
}
return ctx.renderClassFeatureTableHTML(page.Row), nil
case "FeatProgression":
if page.Category != "classes" {
return "", nil
}
return ctx.renderClassFeatProgressionHTML(page.Row), nil
case "SpellTable", "SavesProgression", "BABProgression":
return "", nil
case "RaceFeatList":
return renderWikiFactTableHTML([]wikiHTMLFact{{Label: "Racial Feats", Value: ctx.renderRaceFeatList(page.Row)}}), nil
case "StatusCallout":
return buildWikiStatusBlock(page.Status, page.Category), nil
case "RaceNameForms":
return ctx.renderRaceNameFormsHTML(page.Row), nil
case "UserBottomFallback":
return "", nil
default:
return "", fmt.Errorf("render wiki template for %s: unknown formatter %q", page.PageID, name[0])
}
}
type wikiHTMLFact struct {
Label string
Value string
}
func (ctx *wikiContext) renderFactsHTML(category, key string, row map[string]any) string {
facts := []wikiHTMLFact{}
add := func(label, value string) {
if strings.TrimSpace(value) != "" {
facts = append(facts, wikiHTMLFact{Label: label, Value: value})
}
}
switch category {
case "feat":
for _, fact := range ctx.renderFeatFactsHTML(row) {
add(fact.Label, fact.Value)
}
case "skills":
add("Key Ability", stringValue(row, "KeyAbility"))
add("Untrained", yesNoValue(stringValue(row, "Untrained")))
add("Armor Check Penalty", yesNoValue(stringValue(row, "ArmorCheckPenalty")))
case "spells":
add("Constant", stringValue(row, "Constant"))
case "baseitems":
add("Item Class", stringValue(row, "ItemClass"))
add("Weapon Type", stringValue(row, "WeaponType"))
add("Required Feats", ctx.renderReferenceList(ctx.wikiReqFeats(row)))
add("Base Item Stats", ctx.resolveTextValue(fieldValue(row, "BaseItemStatRef"), nil))
case "classes":
add("Hit Die", "d"+stringValue(row, "HitDie"))
add("Skill Points", stringValue(row, "SkillPointBase"))
add("Primary Ability", stringValue(row, "PrimaryAbil"))
case "racialtypes":
add("Ability Adjustments", formatAbilityAdjustments(row))
add("Favored Class", ctx.renderReference(fieldValue(row, "Favored"), ctx.classIDToKey, ctx.resolveClassName))
add("Favored Enemy", ctx.renderReference(fieldValue(row, "FavoredEnemyFeat"), ctx.featIDToKey, ctx.resolveFeatName))
add("Racial Feats", ctx.renderRaceFeatList(row))
}
_ = key
return renderWikiFactTableHTML(facts)
}
func (ctx *wikiContext) renderFeatFactsHTML(row map[string]any) []wikiHTMLFact {
return []wikiHTMLFact{
{Label: "Prerequisites", Value: ctx.renderFeatPrerequisites(row)},
{Label: "Minimum Attack Bonus", Value: stringValue(row, "MINATTACKBONUS")},
{Label: "Minimum Spell Level", Value: stringValue(row, "MINSPELLLVL")},
{Label: "Minimum Fortitude Save", Value: stringValue(row, "MinFortSave")},
}
}
func renderWikiFactTableHTML(facts []wikiHTMLFact) string {
rows := []string{}
for _, fact := range facts {
if strings.TrimSpace(fact.Value) == "" {
continue
}
rows = append(rows, `<tr><th scope="row">`+html.EscapeString(fact.Label)+`</th><td>`+renderMultilineInlineHTML(fact.Value)+`</td></tr>`)
}
if len(rows) == 0 {
return ""
}
return `<table class="wiki-facts"><tbody>` + strings.Join(rows, "\n") + `</tbody></table>`
}
func renderMultilineInlineHTML(text string) string {
lines := []string{}
for _, line := range strings.Split(strings.ReplaceAll(strings.TrimSpace(text), "\r\n", "\n"), "\n") {
line = strings.TrimSpace(line)
if line != "" {
lines = append(lines, renderInlineHTML(line))
}
}
return strings.Join(lines, "<br>")
}
func renderWikiListHTML(class string, items []string) string {
rendered := []string{}
for _, item := range items {
if strings.TrimSpace(item) != "" {
rendered = append(rendered, "<li>"+renderInlineHTML(item)+"</li>")
}
}
if len(rendered) == 0 {
return ""
}
classAttr := ""
if class != "" {
classAttr = ` class="` + html.EscapeString(class) + `"`
}
return "<ul" + classAttr + ">" + strings.Join(rendered, "\n") + "</ul>"
}
func (ctx *wikiContext) renderClassSkillList(row map[string]any) string {
table := ctx.tableForValue(fieldValue(row, "SkillsTable"), ctx.classSkillTables)
if table == nil {
return ""
}
skills := []string{}
for _, skillRow := range table.Rows {
if stringValue(skillRow, "ClassSkill") != "1" {
continue
}
name := ctx.renderReference(fieldValue(skillRow, "SkillIndex"), ctx.skillIDToKey, ctx.resolveSkillName)
if name != "" {
skills = append(skills, name)
}
}
if len(skills) == 0 {
return ""
}
return "==== Class Skills ====\n\n * " + strings.Join(skills, ", ")
}
func (ctx *wikiContext) renderClassSkillListHTML(row map[string]any) string {
table := ctx.tableForValue(fieldValue(row, "SkillsTable"), ctx.classSkillTables)
if table == nil {
return ""
}
skills := []string{}
for _, skillRow := range table.Rows {
if stringValue(skillRow, "ClassSkill") != "1" {
continue
}
name := ctx.renderReference(fieldValue(skillRow, "SkillIndex"), ctx.skillIDToKey, ctx.resolveSkillName)
if name != "" {
skills = append(skills, name)
}
}
if len(skills) == 0 {
return ""
}
return `<h4>Class Skills</h4>` + renderWikiListHTML("wiki-list wiki-class-skills", skills)
}
func (ctx *wikiContext) renderClassFeatProgression(row map[string]any) string {
table := ctx.tableForValue(fieldValue(row, "FeatsTable"), ctx.classFeatTables)
if table == nil {
return ""
}
lines := []string{}
byLevel := map[string][]string{}
selectable := []string{}
for _, featRow := range table.Rows {
name := ctx.renderReference(fieldValue(featRow, "FeatIndex"), ctx.featIDToKey, ctx.resolveFeatName)
if name == "" {
name = ctx.renderReference(fieldValue(featRow, "FeatIndex"), nil, func(string) string { return "" })
}
if name == "" {
continue
}
level := stringValue(featRow, "GrantedOnLevel")
if level == "" || strings.HasPrefix(level, "-") {
selectable = append(selectable, name)
continue
}
byLevel[level] = append(byLevel[level], name)
}
if len(byLevel) > 0 {
lines = append(lines, "==== Granted Feats ====", "")
for _, level := range sortedKeys(byLevel) {
lines = append(lines, " * Level "+level+": "+strings.Join(byLevel[level], ", "))
}
lines = append(lines, "")
}
if len(selectable) > 0 {
lines = append(lines, "==== Selectable Feats ====", "", " * "+strings.Join(selectable, ", "), "")
}
return strings.TrimSpace(strings.Join(lines, "\n"))
}
func (ctx *wikiContext) renderClassFeatProgressionHTML(row map[string]any) string {
table := ctx.tableForValue(fieldValue(row, "FeatsTable"), ctx.classFeatTables)
if table == nil {
return ""
}
byLevel := map[string][]string{}
selectable := []string{}
for _, featRow := range table.Rows {
name := ctx.renderReference(fieldValue(featRow, "FeatIndex"), ctx.featIDToKey, ctx.resolveFeatName)
if name == "" {
name = ctx.renderReference(fieldValue(featRow, "FeatIndex"), nil, func(string) string { return "" })
}
if name == "" {
continue
}
level := stringValue(featRow, "GrantedOnLevel")
if level == "" || strings.HasPrefix(level, "-") {
selectable = append(selectable, name)
continue
}
byLevel[level] = append(byLevel[level], name)
}
parts := []string{}
if len(byLevel) > 0 {
items := []string{}
for _, level := range sortedKeys(byLevel) {
items = append(items, "Level "+level+": "+strings.Join(byLevel[level], ", "))
}
parts = append(parts, `<h4>Granted Feats</h4>`+renderWikiListHTML("wiki-list wiki-class-granted-feats", items))
}
if len(selectable) > 0 {
parts = append(parts, `<h4>Selectable Feats</h4>`+renderWikiListHTML("wiki-list wiki-class-selectable-feats", []string{strings.Join(selectable, ", ")}))
}
return strings.Join(parts, "\n")
}
func (ctx *wikiContext) renderClassFeatureTable(row map[string]any) string {
table := ctx.tableForValue(fieldValue(row, "BonusFeatsTable"), ctx.classBonusTables)
if table == nil || len(table.Rows) == 0 {
return ""
}
return "==== Bonus Feats ====\n\n * Bonus feat table: " + table.OutputStem
}
func (ctx *wikiContext) renderClassFeatureTableHTML(row map[string]any) string {
table := ctx.tableForValue(fieldValue(row, "BonusFeatsTable"), ctx.classBonusTables)
if table == nil || len(table.Rows) == 0 {
return ""
}
return `<h4>Bonus Feats</h4>` + renderWikiListHTML("wiki-list wiki-class-bonus-feats", []string{"Bonus feat table: " + table.OutputStem})
}
func (ctx *wikiContext) renderRaceNameFormsHTML(row map[string]any) string {
facts := []wikiHTMLFact{}
if plural := ctx.resolveTextValue(fieldValue(row, "NamePlural"), nil); plural != "" {
facts = append(facts, wikiHTMLFact{Label: "Plural", Value: plural})
}
if converted := ctx.resolveTextValue(fieldValue(row, "ConverName"), nil); converted != "" {
facts = append(facts, wikiHTMLFact{Label: "Converted Name", Value: converted})
}
if lower := ctx.resolveTextValue(fieldValue(row, "ConverNameLower"), nil); lower != "" {
facts = append(facts, wikiHTMLFact{Label: "Lower Name", Value: lower})
}
return renderWikiFactTableHTML(facts)
}
internal/topdata/wiki_native.go
+2159
View File
File diff suppressed because it is too large Load Diff
internal/topdata/wiki_native_test.go
+3008
View File
File diff suppressed because it is too large Load Diff
internal/topdata/wiki_page_paths.go
+77
View File
@@ -0,0 +1,77 @@
package topdata
import (
"fmt"
"os"
"strings"
"gopkg.in/yaml.v3"
)
type wikiPagePathDocument struct {
PagePaths wikiPagePaths `json:"page_paths" yaml:"page_paths"`
}
type wikiPagePaths struct {
SlugOverrides []wikiPageSlugOverride `json:"slug_overrides" yaml:"slug_overrides"`
}
type wikiPageSlugOverride struct {
PageID string `json:"page_id" yaml:"page_id"`
Slug string `json:"slug" yaml:"slug"`
}
func loadWikiPagePathDeclarations(path string) (map[string]string, error) {
raw, err := os.ReadFile(path)
if os.IsNotExist(err) {
return map[string]string{}, nil
}
if err != nil {
return nil, err
}
return parseWikiPagePathDeclarations(raw, path)
}
func parseWikiPagePathDeclarations(raw []byte, path string) (map[string]string, error) {
var doc wikiPagePathDocument
if err := yaml.Unmarshal(raw, &doc); err != nil {
return nil, fmt.Errorf("parse wiki page paths %s: %w", path, err)
}
if len(doc.PagePaths.SlugOverrides) > 0 {
return nil, fmt.Errorf("parse wiki page paths %s: page_paths.slug_overrides is retired; generated public wiki paths are title-derived", path)
}
overrides := map[string]string{}
for _, override := range doc.PagePaths.SlugOverrides {
pageID := strings.TrimSpace(override.PageID)
slug := strings.TrimSpace(override.Slug)
if pageID == "" {
return nil, fmt.Errorf("parse wiki page paths %s: slug override has empty page_id", path)
}
if !isCanonicalWikiSlug(slug) {
return nil, fmt.Errorf("parse wiki page paths %s: page_id %q has invalid slug %q", path, pageID, override.Slug)
}
if _, ok := overrides[pageID]; ok {
return nil, fmt.Errorf("parse wiki page paths %s: duplicate page_id %q", path, pageID)
}
overrides[pageID] = slug
}
return overrides, nil
}
func isCanonicalWikiSlug(value string) bool {
if value == "" || value != strings.ToLower(value) {
return false
}
parts := strings.Split(value, "-")
for _, part := range parts {
if part == "" {
return false
}
for _, r := range part {
if !((r >= 'a' && r <= 'z') || (r >= '0' && r <= '9')) {
return false
}
}
}
return true
}
internal/topdata/wiki_preserve.go
+54
View File
@@ -0,0 +1,54 @@
package topdata
import "strings"
func normalizePreservedSectionBodies(content string) string {
var out strings.Builder
offset := 0
for {
startRel := strings.Index(content[offset:], manualStartMarkerPrefix)
if startRel < 0 {
out.WriteString(content[offset:])
break
}
start := offset + startRel
startEndRel := strings.Index(content[start:], "-->")
if startEndRel < 0 {
out.WriteString(content[offset:])
break
}
startMarker := content[start : start+startEndRel+len("-->")]
id := markerID(startMarker)
if id == "" {
out.WriteString(content[offset : start+len(startMarker)])
offset = start + len(startMarker)
continue
}
endPrefix := manualEndMarkerPrefix + " id=\"" + id + "\""
endRel := strings.Index(content[start+len(startMarker):], endPrefix)
if endRel < 0 {
out.WriteString(content[offset:])
break
}
end := start + len(startMarker) + endRel
endEndRel := strings.Index(content[end:], "-->")
if endEndRel < 0 {
out.WriteString(content[offset:])
break
}
endMarker := content[end : end+endEndRel+len("-->")]
out.WriteString(content[offset:start])
out.WriteString(startMarker)
out.WriteString("\n<!-- preserved-section-body -->\n")
out.WriteString(endMarker)
offset = end + len(endMarker)
}
return out.String()
}
func defaultWikiManualSections() []wikiManualSection {
return []wikiManualSection{
{ID: "user_top", Alias: "UserTopSection", Heading: "", InitialHTML: "<p></p>"},
{ID: "user_bottom", Alias: "UserBottomSection", Heading: "", InitialHTML: "<p></p>"},
}
}
internal/topdata/wiki_slug.go
+210
View File
@@ -0,0 +1,210 @@
package topdata
import (
"html"
"strings"
"unicode"
"golang.org/x/text/unicode/norm"
)
var reservedCanonicalWikiFirstSegments = map[string]struct{}{
"admin": {},
"api": {},
"category": {},
"compose": {},
"edit": {},
"namespace": {},
"search": {},
}
func canonicalWikiSegment(value string) string {
value = html.UnescapeString(strings.TrimSpace(value))
var b strings.Builder
lastSeparator := false
for _, r := range norm.NFKD.String(value) {
transliterated := transliterateWikiSlugRune(unicode.ToLower(r))
switch {
case (r >= 'a' && r <= 'z') || (r >= 'A' && r <= 'Z') || (r >= '0' && r <= '9'):
b.WriteRune(r)
lastSeparator = false
case r == '_':
if b.Len() > 0 && !lastSeparator {
b.WriteRune(r)
lastSeparator = true
}
case unicode.Is(unicode.Mn, r):
continue
case isWikiSlugJoinerPunctuation(r):
continue
case transliterated != "":
b.WriteString(matchWikiTransliterationCase(r, transliterated))
lastSeparator = false
default:
if b.Len() > 0 && !lastSeparator {
b.WriteByte('_')
lastSeparator = true
}
}
}
return strings.Trim(b.String(), "_")
}
func canonicalWikiSegmentFoldedKey(value string) string {
return strings.Join(strings.Fields(strings.ToLower(strings.ReplaceAll(canonicalWikiSegment(value), "_", " "))), " ")
}
func canonicalWikiSlashPath(value string) string {
segments := []string{}
for _, part := range strings.Split(strings.Trim(strings.TrimSpace(value), "/"), "/") {
canonical := canonicalWikiSegment(part)
if canonical != "" {
segments = append(segments, canonical)
}
}
return strings.Join(segments, "/")
}
func canonicalWikiSlashPathFoldedKey(value string) string {
segments := []string{}
for _, part := range strings.Split(strings.Trim(strings.TrimSpace(value), "/"), "/") {
folded := canonicalWikiSegmentFoldedKey(part)
if folded != "" {
segments = append(segments, folded)
}
}
return strings.Join(segments, "/")
}
func canonicalWikiSlashPathSegments(value string) []string {
segments := []string{}
for _, part := range strings.Split(strings.Trim(strings.TrimSpace(value), "/"), "/") {
canonical := canonicalWikiSegment(part)
if canonical != "" {
segments = append(segments, canonical)
}
}
return segments
}
func isReservedCanonicalWikiFirstSegment(path string) bool {
segments := canonicalWikiSlashPathSegments(path)
if len(segments) == 0 {
return false
}
first := strings.ToLower(segments[0])
if isCanonicalWikiNumericSegment(first) {
return true
}
_, reserved := reservedCanonicalWikiFirstSegments[first]
return reserved
}
func isCanonicalWikiNumericSegment(segment string) bool {
if segment == "" {
return false
}
for _, r := range segment {
if r < '0' || r > '9' {
return false
}
}
return true
}
func isNestedCanonicalWikiPath(path string) bool {
return len(canonicalWikiSlashPathSegments(path)) > 2
}
func matchWikiTransliterationCase(r rune, value string) string {
if unicode.IsUpper(r) {
if len(value) == 1 {
return strings.ToUpper(value)
}
return strings.ToUpper(value[:1]) + value[1:]
}
return value
}
func canonicalWikiPath(segments ...string) string {
out := []string{}
for _, segment := range segments {
canonical := canonicalWikiSegment(segment)
if canonical != "" {
out = append(out, canonical)
}
}
return strings.Join(out, "/")
}
func slugifyWikiText(value string, fallback string) string {
value = html.UnescapeString(strings.TrimSpace(value))
var b strings.Builder
lastDash := false
for _, r := range norm.NFKD.String(strings.ToLower(value)) {
transliterated := transliterateWikiSlugRune(r)
switch {
case r >= 'a' && r <= 'z':
b.WriteRune(r)
lastDash = false
case r >= '0' && r <= '9':
b.WriteRune(r)
lastDash = false
case unicode.Is(unicode.Mn, r):
continue
case isWikiSlugJoinerPunctuation(r):
continue
case transliterated != "":
b.WriteString(transliterated)
lastDash = false
default:
if b.Len() > 0 && !lastDash {
b.WriteByte('-')
lastDash = true
}
}
}
slug := strings.Trim(b.String(), "-")
if slug == "" {
return fallback
}
return slug
}
func isWikiSlugJoinerPunctuation(r rune) bool {
switch r {
case '\'', '"', '`', '´', '‘', '’', '‚', '‛', '“', '”', '„', '‟', '‹', '›', '«', '»', '′', '″', '＇', '＂':
return true
default:
return false
}
}
func transliterateWikiSlugRune(r rune) string {
switch r {
case 'æ':
return "ae"
case 'œ':
return "oe"
case 'ø':
return "o"
case 'ß':
return "ss"
case 'þ':
return "th"
case 'ð', 'đ':
return "d"
case 'ł':
return "l"
case 'ħ':
return "h"
case 'ı':
return "i"
case 'ŋ':
return "n"
case 'ŧ':
return "t"
default:
return ""
}
}
internal/topdata/wiki_tables.go
+1749
View File
File diff suppressed because it is too large Load Diff
internal/topdata/wiki_template.go
+433
View File
@@ -0,0 +1,433 @@
package topdata
import (
"fmt"
"html"
"os"
"path/filepath"
"strings"
)
type wikiTemplatePage struct {
PageID string
Category string
Key string
Title string
Status string
Row map[string]any
ManualSections []wikiManualSection
}
func (ctx *wikiContext) renderWikiPageTemplate(category string, page wikiTemplatePage) (string, error) {
name := wikiTemplateName(category)
source, path := ctx.loadWikiTemplateSource(name)
if source == "" && name != "default.html" {
source, path = ctx.loadWikiTemplateSource("default.html")
}
if source == "" {
source = builtinWikiTemplate(category)
path = "builtin:" + name
}
return ctx.renderWikiTemplateString(path, source, page)
}
func wikiTemplateName(category string) string {
switch category {
case "classes", "feat", "skills", "spells", "racialtypes", "baseitems":
return category + ".html"
default:
return "default.html"
}
}
func (ctx *wikiContext) loadWikiTemplateSource(name string) (string, string) {
if strings.TrimSpace(ctx.templateDir) == "" {
return "", ""
}
path := filepath.Join(ctx.templateDir, filepath.FromSlash(name))
raw, err := os.ReadFile(path)
if err != nil {
return "", path
}
return string(raw), path
}
func builtinWikiTemplate(category string) string {
switch category {
case "classes":
return `<h1>{{title}}</h1>
{{UserTopSection}}
{{format:StatusCallout}}
{{format:Description}}
{{format:FactsTable}}
{{format:SkillList}}
{{format:FeatProgression}}
{{format:UserBottomFallback}}`
case "racialtypes":
return `<h1>{{title}}</h1>
{{UserTopSection}}
{{format:StatusCallout}}
{{format:Description}}
{{format:FactsTable}}
{{format:RaceNameForms}}
{{format:UserBottomFallback}}`
default:
return `<h1>{{title}}</h1>
{{UserTopSection}}
{{format:StatusCallout}}
{{format:Description}}
{{format:FactsTable}}
{{format:UserBottomFallback}}`
}
}
func (ctx *wikiContext) renderWikiTemplateString(path, source string, page wikiTemplatePage) (string, error) {
state := &wikiTemplateRenderState{
path: path,
page: page,
seenSections: map[string]struct{}{},
active: true,
}
rendered, offset, closed, err := ctx.renderWikiTemplateRange(source, 0, "", state)
if err != nil {
return "", err
}
if closed {
return "", fmt.Errorf("render wiki template %s for %s: unexpected closing block", path, page.PageID)
}
if offset != len(source) {
return "", fmt.Errorf("render wiki template %s for %s: template parse stopped early", path, page.PageID)
}
return strings.TrimSpace(rendered), nil
}
type wikiTemplateRenderState struct {
path string
page wikiTemplatePage
row map[string]any
dot any
seenSections map[string]struct{}
active bool
}
func (ctx *wikiContext) renderWikiTemplateRange(source string, offset int, stopBlock string, state *wikiTemplateRenderState) (string, int, bool, error) {
var out strings.Builder
for {
start := strings.Index(source[offset:], "{{")
if start < 0 {
if stopBlock != "" {
return "", offset, false, fmt.Errorf("render wiki template %s for %s: missing {{/%s}}", state.path, state.page.PageID, stopBlock)
}
if state.active {
out.WriteString(source[offset:])
}
return out.String(), len(source), false, nil
}
start += offset
end := strings.Index(source[start+2:], "}}")
if end < 0 {
return "", offset, false, fmt.Errorf("render wiki template %s for %s: unclosed placeholder", state.path, state.page.PageID)
}
end += start + 2
if state.active {
out.WriteString(source[offset:start])
}
token := strings.TrimSpace(source[start+2 : end])
if strings.HasPrefix(token, "/") {
name := strings.TrimSpace(strings.TrimPrefix(token, "/"))
if stopBlock == "" || name != stopBlock {
return "", offset, false, fmt.Errorf("render wiki template %s for %s: unexpected {{/%s}}", state.path, state.page.PageID, name)
}
return out.String(), end + 2, true, nil
}
if strings.HasPrefix(token, "#each ") {
providerName := strings.TrimSpace(strings.TrimPrefix(token, "#each "))
body, nextOffset, err := extractWikiTemplateBlock(source, end+2, "each", state.path, state.page.PageID)
if err != nil {
return "", offset, false, err
}
if state.active {
items, err := ctx.resolveWikiTemplateEachItems(providerName, state)
if err != nil {
return "", offset, false, fmt.Errorf("render wiki template %s for %s: %w", state.path, state.page.PageID, err)
}
for _, item := range items {
child := *state
child.row = item.row
child.dot = item.dot
rendered, childOffset, closed, err := ctx.renderWikiTemplateRange(body, 0, "", &child)
if err != nil {
return "", offset, false, err
}
if closed || childOffset != len(body) {
return "", offset, false, fmt.Errorf("render wiki template %s for %s: malformed each body", state.path, state.page.PageID)
}
out.WriteString(rendered)
}
}
offset = nextOffset
continue
}
if strings.HasPrefix(token, "#if ") {
expr := strings.TrimSpace(strings.TrimPrefix(token, "#if "))
body, nextOffset, err := extractWikiTemplateBlock(source, end+2, "if", state.path, state.page.PageID)
if err != nil {
return "", offset, false, err
}
if state.active {
value, err := ctx.evalWikiTemplateExpression(expr, state)
if err != nil {
return "", offset, false, err
}
if truthyWikiTableValue(value) {
child := *state
rendered, childOffset, closed, err := ctx.renderWikiTemplateRange(body, 0, "", &child)
if err != nil {
return "", offset, false, err
}
if closed || childOffset != len(body) {
return "", offset, false, fmt.Errorf("render wiki template %s for %s: malformed if body", state.path, state.page.PageID)
}
out.WriteString(rendered)
}
}
offset = nextOffset
continue
}
if strings.HasPrefix(token, "#with ") {
expr := strings.TrimSpace(strings.TrimPrefix(token, "#with "))
body, nextOffset, err := extractWikiTemplateBlock(source, end+2, "with", state.path, state.page.PageID)
if err != nil {
return "", offset, false, err
}
if state.active {
value, err := ctx.evalWikiTemplateExpression(expr, state)
if err != nil {
return "", offset, false, err
}
if truthyWikiTableValue(value) {
child := *state
child.dot = value
rendered, childOffset, closed, err := ctx.renderWikiTemplateRange(body, 0, "", &child)
if err != nil {
return "", offset, false, err
}
if closed || childOffset != len(body) {
return "", offset, false, fmt.Errorf("render wiki template %s for %s: malformed with body", state.path, state.page.PageID)
}
out.WriteString(rendered)
}
}
offset = nextOffset
continue
}
if !state.active {
offset = end + 2
continue
}
rendered, sectionID, err := ctx.renderWikiTemplateTokenWithState(token, state)
if err != nil {
return "", offset, false, err
}
if sectionID != "" {
if _, exists := state.seenSections[sectionID]; exists {
return "", offset, false, fmt.Errorf("render wiki template %s for %s: duplicate preserved section %q", state.path, state.page.PageID, sectionID)
}
state.seenSections[sectionID] = struct{}{}
}
out.WriteString(rendered)
offset = end + 2
}
}
type wikiTemplateEachItem struct {
row map[string]any
dot any
}
func (ctx *wikiContext) resolveWikiTemplateEachItems(name string, state *wikiTemplateRenderState) ([]wikiTemplateEachItem, error) {
if value, ok := ctx.resolveWikiTemplateScopedValue(name, state); ok {
return wikiTemplateEachItemsFromValue(value), nil
}
rows, err := ctx.wikiDataProviderRows(name, state.page)
if err != nil {
return nil, err
}
items := make([]wikiTemplateEachItem, 0, len(rows))
for _, row := range rows {
items = append(items, wikiTemplateEachItem{row: row, dot: row})
}
return items, nil
}
func wikiTemplateEachItemsFromValue(value any) []wikiTemplateEachItem {
switch typed := value.(type) {
case []map[string]any:
items := make([]wikiTemplateEachItem, 0, len(typed))
for _, row := range typed {
items = append(items, wikiTemplateEachItem{row: row, dot: row})
}
return items
case []any:
items := make([]wikiTemplateEachItem, 0, len(typed))
for _, value := range typed {
if row, ok := value.(map[string]any); ok {
items = append(items, wikiTemplateEachItem{row: row, dot: row})
continue
}
items = append(items, wikiTemplateEachItem{dot: value})
}
return items
case []string:
items := make([]wikiTemplateEachItem, 0, len(typed))
for _, value := range typed {
items = append(items, wikiTemplateEachItem{dot: value})
}
return items
default:
return nil
}
}
func (ctx *wikiContext) renderWikiTemplateTokenWithState(token string, state *wikiTemplateRenderState) (string, string, error) {
if strings.HasPrefix(token, "format:") || strings.HasPrefix(token, "table:") || strings.HasPrefix(token, "field:") || strings.HasPrefix(token, "section:") {
rendered, sectionID, err := ctx.renderWikiTemplateToken(state.path, token, state.page)
return rendered, sectionID, err
}
if strings.Contains(token, "|") || strings.Contains(token, "(") || token == "." || strings.HasPrefix(token, "page.") {
value, err := ctx.evalWikiTemplateExpression(token, state)
if err != nil {
return "", "", err
}
return stringifyWikiTemplateOutput(value), "", nil
}
if state.row != nil {
if _, ok := ctx.resolveWikiTemplateScopedValue(token, state); ok {
value, err := ctx.evalWikiTemplateExpression(token, state)
if err != nil {
return "", "", err
}
return stringifyWikiTemplateOutput(value), "", nil
}
}
rendered, sectionID, err := ctx.renderWikiTemplateToken(state.path, token, state.page)
return rendered, sectionID, err
}
func (ctx *wikiContext) renderWikiTemplateToken(path, token string, page wikiTemplatePage) (string, string, error) {
switch token {
case "title":
return html.EscapeString(page.Title), "", nil
case "page_id":
return html.EscapeString(page.PageID), "", nil
case "status":
return html.EscapeString(page.Status), "", nil
}
if section, ok := wikiManualSectionForAlias(page.ManualSections, token); ok {
return renderWikiManualSection(section), section.ID, nil
}
if strings.HasPrefix(token, "section:") {
sectionName := strings.TrimSpace(strings.TrimPrefix(token, "section:"))
if section, ok := wikiManualSectionForAlias(page.ManualSections, sectionName); ok {
return renderWikiManualSection(section), section.ID, nil
}
return "", "", nil
}
if strings.HasPrefix(token, "format:") {
rendered, err := ctx.renderWikiFormatter(strings.TrimSpace(strings.TrimPrefix(token, "format:")), page)
return rendered, "", err
}
if strings.HasPrefix(token, "table:") {
rendered, err := ctx.renderConfiguredWikiTable(path, strings.TrimSpace(strings.TrimPrefix(token, "table:")), page)
return rendered, "", err
}
if strings.HasPrefix(token, "field:") {
rendered, err := ctx.renderExplicitWikiField(path, strings.TrimSpace(strings.TrimPrefix(token, "field:")), page)
return rendered, "", err
}
rendered, err := ctx.renderRequiredWikiField(path, token, page)
return rendered, "", err
}
func wikiManualSectionForAlias(sections []wikiManualSection, alias string) (wikiManualSection, bool) {
for _, section := range sections {
if section.Alias == alias {
return section, true
}
}
switch alias {
case "UserTopSection":
for _, section := range sections {
if section.ID == "user_top" {
return section, true
}
}
case "UserBottomSection":
for _, section := range sections {
if section.ID == "user_bottom" {
return section, true
}
}
}
return wikiManualSection{}, false
}
func renderWikiManualSection(section wikiManualSection) string {
return strings.Join([]string{
"<!-- sow-topdata-wiki:manual:start id=\"" + html.EscapeString(section.ID) + "\" -->",
strings.TrimSpace(section.InitialHTML),
"<!-- sow-topdata-wiki:manual:end id=\"" + html.EscapeString(section.ID) + "\" -->",
}, "\n")
}
func (ctx *wikiContext) renderExplicitWikiField(path, spec string, page wikiTemplatePage) (string, error) {
fieldSpec, defaultValue, hasDefault := strings.Cut(spec, "|default=")
fieldSpec = strings.TrimSpace(fieldSpec)
value, ok := ctx.resolveWikiTemplateField(fieldSpec, page)
if !ok {
if hasDefault {
return html.EscapeString(defaultValue), nil
}
return "", fmt.Errorf("render wiki template %s for %s: missing field %q", path, page.PageID, fieldSpec)
}
return html.EscapeString(value), nil
}
func (ctx *wikiContext) renderRequiredWikiField(path, fieldSpec string, page wikiTemplatePage) (string, error) {
value, ok := ctx.resolveWikiTemplateField(fieldSpec, page)
if !ok {
return "", fmt.Errorf("render wiki template %s for %s: missing field %q", path, page.PageID, fieldSpec)
}
return html.EscapeString(value), nil
}
func (ctx *wikiContext) resolveWikiTemplateField(fieldSpec string, page wikiTemplatePage) (string, bool) {
fieldSpec = strings.TrimSpace(fieldSpec)
if strings.HasSuffix(fieldSpec, ".text") {
base := strings.TrimSuffix(fieldSpec, ".text")
value, ok := lookupField(page.Row, base)
if !ok {
return "", false
}
text := ctx.resolveTextValue(value, nil)
return text, text != ""
}
value, ok := lookupField(page.Row, fieldSpec)
if !ok || value == nil {
return "", false
}
switch typed := value.(type) {
case string:
if typed == "" || typed == nullValue {
return "", false
}
return typed, true
case int:
return fmt.Sprintf("%d", typed), true
case float64:
return fmt.Sprintf("%d", int(typed)), true
default:
text := ctx.resolveTextValue(typed, nil)
return text, text != ""
}
}

Some files were not shown because too many files have changed in this diff Show More