Compare commits

...
15 Commits
Author SHA1 Message Date
archvillainette 4d03085996 feat(ci): prune release assets outside the newest two tags (#52) (#70)
Closes #52.

Adds `scripts/prune-release-assets.sh` and a best-effort step at the end of `build-binaries.yml`.

- Lists every release page, sorts by creation date, skips the newest two published releases, deletes the rest's attachments via `DELETE /releases/{id}/assets/{asset_id}`.
- Drafts never consume a keep slot, so a draft cannot strip the newest real release.
- Releases, tags, source archives and release notes are never touched.
- Nothing is excluded: SHA256SUMS and the wrappers go with the binaries. Checksums are only meaningful next to the files they cover, and the consumer drift-check reads the wrappers from the latest release, which stays complete.
- Failures warn and the step has `continue-on-error: true` — a stale asset is cheaper than a blocked publish.

`tests/prune-release-assets.sh` drives the script with a stub `curl`: keep window, out-of-order responses, drafts, no non-asset DELETE, short-page pagination stop, and both failure paths. Wired into `make check`.

v0.2.0 still needs its 9 assets cleared by hand in the web UI, as the issue notes.

🤖 Generated with [Claude Code](https://claude.com/claude-code)Reviewed-on: #70

Co-authored-by: vickydotbat <vickydotbat@tutamail.com>
2026-07-29 08:51:25 +00:00
archvillainette 2f4685e470 fix(erf): align restype numbers with upstream neverwinter.nim (#69)
Closes #64.

Five entries in `internal/erf/erf.go`'s restype table disagreed with upstream neverwinter.nim (`neverwinter/restype.nim`). `HAKResourceTypeForExtension` packs arbitrary source files, so a `.jpg` packed into a HAK today is written as restype 2076, which the game reads as `tml`.

| ext | was | now |
|-----|-----|-----|
| gff | 2039 (upstream `bte`) | 2037 |
| ltr | 2067 (upstream `bak`) | 2036 |
| jpg | 2076 (upstream `tml`) | 2081 |
| dfa | 2045, name typo | `dft`, 2045 (number was already right) |
| mdb | 2070 (upstream `xbc`) | removed |

Upstream registers no restype for `mdb`, so there is no correct number to give it, and leaving it squatting on `xbc` silently mislabels the resource. It is dropped from `AssetExtensions` too, which turns a `.mdb` source into a loud "unsupported extension" build error instead of a corrupt HAK entry. **This is the one judgment call the issue left open** — if `mdb` should instead stay as a deliberate local addition like `lyt`/`vis`/`gr2`, say so and I will move it to a free number in the 0x0BB8+ local range.

No asset in the current corpus uses any of these paths, so nothing in shipped content changes.

The six local additions upstream does not have (`lyt` 3000, `vis` 3001, `mdx` 3008, `wlk` 3020, `xml` 3021, `gr2` 4003) are left alone and now carry a comment saying they are deliberate.

## Root cause

The `init()` consistency guard only panicked when a number was missing from the reverse map. Its `if canonicalExt == ext { continue }` branch did nothing when the two maps disagreed, so a mismatch was never caught. The guard now panics on disagreement and also checks the reverse direction.

## Tests

- `TestRestypeTableMatchesUpstream` pins every number shared with upstream, and asserts the four numbers upstream reserves for other extensions (2039 `bte`, 2067 `bak`, 2070 `xbc`, 2076 `tml`) stay unclaimed.
- `TestRestypeTablesAreMutualInverses` is the check `init()` is meant to enforce.

`make check` passes.Reviewed-on: #69

Co-authored-by: vickydotbat <vickydotbat@tutamail.com>
2026-07-28 22:18:03 +00:00
archvillainette 00f467b932 docs(agents): note that tea reads stdin and hangs without </dev/null (#68)
`tea comment` reads stdin to EOF and appends whatever it finds to the comment body. In a non-interactive shell — which is every agent — stdin is an open pipe that never sends EOF, so the call hangs forever instead of posting.

Verified on tea 0.14.0: with `</dev/null` it posts instantly; with an open pipe it blocks until killed; with `echo "x" | tea comment N "y"` it posts `y` followed by `x`.

Documents the redirect in the tracker guide, and applies it to the wayfinder resolve step. Same trap exists on `tea issues create --description` and `tea pr create`.

🤖 Generated with [Claude Code](https://claude.com/claude-code)Reviewed-on: #68

Co-authored-by: vickydotbat <vickydotbat@tutamail.com>
2026-07-28 13:18:38 +00:00
archvillainette 22eecd1a41 depot: replace --target local with --out DIR (#67)
Closes #66.

`depot status` and `depot get` chose a depot tree on disk with `--target local`, taking the root from `DEPOT_DIR`. One decision, two flags — and `--target local` with no directory anywhere was valid but meaningless.

Now they take `--out DIR`, matching the crucible nwsync surface. `--target` names remote backends only (`bunny|cdn`). Passing both exits 64.

- `--target local` stays as a deprecated alias for `--out $DEPOT_DIR`, listed in the hidden-alias table in `docs/command-surface.md`. Nothing in `wrappers/` calls depot, but the three repos in `wrappers/consumers.txt` are out of reach from here, so the alias stays.
- `status --source` was already unused; it now warns "ignored; use --out DIR" instead of accepting silently.
- Fixes a real parse bug found on the way: the documented `depot get <sha> <dest> --target cdn` form never parsed its flags, because Go's `flag` package stops at the first positional. Positionals are split off by hand now; both orders are tested.
- Registry usage strings and `docs/command-surface.md` synced — that doc still said depot was unwired.

Not converted: `push --source` and `pull --dest`. Neither is `--target local`; one names a read source, the other a remote pull's destination. Say the word if they should become `--out` too.

`go test ./...` green.

🤖 Generated with [Claude Code](https://claude.com/claude-code)Reviewed-on: #67

Co-authored-by: vickydotbat <vickydotbat@tutamail.com>
2026-07-28 11:07:58 +00:00
archvillainette ed6945d308 Adopt workspace-wide agent standards; retire sow-docs (#51)
Part of the workspace-wide standards rollout. Establishes where work lives (issues vs ADRs vs standing law), scaffolds `docs/agents/` for the engineering skills, and adopts ADR-0001 locally. See `sow-platform` ADR-0021 for the `sow-docs` retirement.Reviewed-on: #51
Reviewed-by: xtul <mpiasecki720@protonmail.com>
Co-authored-by: vickydotbat <vickydotbat@tutamail.com>
2026-07-24 18:43:31 +00:00
archvillainette 3f500dabc8 Generate palette projections at module build (#50)
build-binaries / build-binaries (push) Successful in 2m13s
*palcus.itp files are Toolset-derived snapshots (category skeleton + blueprint descriptors). This makes Crucible the generator: module builds strip stale descriptors from the committed skeletons and inject descriptors computed from the source-tree blueprints (NAME/STRREF + RESREF, plus CR/FACTION for creatures, faction names resolved via repute.fac). Compare applies the same projection; extract never writes *palcus.itp back into source, so local Toolset palette mess stays local.

Companion PR in sow-module strips the committed palcus files down to skeletons.

🤖 Generated with [Claude Code](https://claude.com/claude-code)Reviewed-on: #50

Co-authored-by: vickydotbat <vickydotbat@tutamail.com>
2026-07-23 09:35:45 +00:00
archvillainette 8e7cead5c0 Keep model compilation headless (#49)
build-binaries / build-binaries (push) Successful in 2m14s
## Summary

- always wrap NWN model compilation with `xvfb-run`, even when the caller has `DISPLAY`
- fail closed with an actionable error when `xvfb-run` is unavailable instead of opening the client UI
- update the compiler contract and regression coverage

## Verification

- focused red/green regression tests
- `nix develop -c make check`
- real NWN compile with `DISPLAY=:0` under transient Xvfb; binary MDL output verified

Generated with Claude CodeReviewed-on: #49

Co-authored-by: vickydotbat <vickydotbat@tutamail.com>
2026-07-23 07:27:37 +00:00
archvillainette 4d38967078 fix(topdata): let pinned TLK ids evict stale state owners (#48)
build-binaries / build-binaries (push) Successful in 2m14s
## Problem

The custom palette taxonomy (#46) pins display strings to fixed TLK ids in `tlk/custom.tlk.yml`, and `registerInlineAtID` demanded each pinned id be free. But `.tlk_state.json` is **gitignored and per-machine** — each dev grows their own copy, and before #46 every ref got its id dynamically (first-come-first-served).

On any machine whose state predates #46, a ref could have already parked on a now-pinned id. That fails the build:

```
topdata validation failed with 1 error(s): error: native topdata buildability check failed:
TLK id 2689 is already reserved by "feat:yuanti/alternate_form.feat"
```

It only passes on machines whose state was regenerated after #46 (pinned window already clean). The `allocateID` reserved-skip that #46 added protects a *fresh* build, but does nothing about a *stale cache*.

## Fix

Make the pin authoritative over the per-machine cache. A stale cached owner sitting on a pinned id is evicted and reallocated a fresh id when next made active; only two pins fighting over the same id in `custom.tlk.yml` is now an error. This self-heals on the next build — no manual `.tlk_state.json` deletion needed.

Caveat: the evicted ref's strref shifts on affected machines. That's unavoidable (something must move off the pinned id), and dynamic strrefs were never stable across machines anyway.

## Tests

Added `TestBuildStandaloneTLKPinEvictsStaleStateOwner` (seeds a pre-#46 state with a feat parked on the pinned id, asserts the build succeeds, the pin owns it, and the feat is reallocated). Full `internal/topdata` suite + `go vet` pass.

🤖 Generated with [Claude Code](https://claude.com/claude-code)Reviewed-on: #48
Reviewed-by: xtul <mpiasecki720@protonmail.com>
Co-authored-by: vickydotbat <vickydotbat@tutamail.com>
2026-07-22 18:17:04 +00:00
archvillainette 7b481ca0c0 custom palette taxonomy (#46)
build-binaries / build-binaries (push) Successful in 2m12s
Implements custom palette taxonomyReviewed-on: #46

Co-authored-by: vickydotbat <vickydotbat@tutamail.com>
2026-07-22 17:03:58 +00:00
archvillainette b058846e16 chore(ci): consolidate Gitea CI (#47)
## Summary

- consolidate vet, tests, lint, smoke, and cross-build into one pull-request-only job
- keep binary releases strictly v-tag-only with release-only write permission
- retain wrapper synchronization only for canonical wrapper changes on main
- pin actions and enforce exact trigger contracts

## Verification

- nix develop --command make check
- trigger mutation contract
- 28-workflow cross-repository actionlint audit

Generated with Claude CodeReviewed-on: #47

Co-authored-by: vickydotbat <vickydotbat@tutamail.com>
2026-07-22 16:54:26 +00:00
archvillainette e509b7a90a Remove completed legacy-migration cluster from topdata (#45)
test / test (push) Successful in 1m16s
build-binaries / build-binaries (push) Successful in 2m7s
The legacy normalize migration is done (confirmed). This deletes the dead migration path: NormalizeProject + 26 *_migrate.go files, the importLegacyItempropsRegistry cluster, loadLegacyTLK, and their tests (~7.9k lines, -33 files worth). mergeLegacyTLK moved into tlk_native.go since the live TLK compiler still uses it for base_dialog.json. make check passes.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Reviewed-on: #45
Reviewed-by: gitea-bot <gitea-bot@noreply.git.westgate.pw>
Co-authored-by: vickydotbat <vickydotbat@tutamail.com>
Co-committed-by: vickydotbat <vickydotbat@tutamail.com>
2026-07-19 08:55:24 +00:00
archvillainette 0d12674838 skills: add data-driven-design (#44)
build-binaries / build-binaries (push) Successful in 2m7s
test / test (push) Successful in 1m24s
Adds .agents/skills/data-driven-design: general mechanisms in code, all specific knowledge in data. Written for the long-term cleanup that would let sow-tools be open sourced.

Grounded in two real failures: the value_encodings mode zoo (per-column mode enums + switch arms where a shape/encoding declaration in data would do) and the skills dataset rename (code hardcoding dataset names/key prefixes that the data already knows).

Tested per TDD-for-skills: a baseline subagent given the EquipSlots scenario added a fourth mode and rationalized it ("three switch cases isn't a pattern crisis"); with the skill loaded it declared shape in data, collapsed the modes into orthogonal primitives, and deleted the switch.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Reviewed-on: #44
Co-authored-by: vickydotbat <vickydotbat@tutamail.com>
Co-committed-by: vickydotbat <vickydotbat@tutamail.com>
2026-07-19 07:46:49 +00:00
archvillainette bc8fa3a6fe topdata: stop hardcoding the skills dataset location (#43)
test / test (push) Successful in 1m24s
build-binaries / build-binaries (push) Successful in 2m8s
sow-topdata moved skills into a tree (skills/core, skills/specs, skillcats, skillspecranks), which broke validate/build: the generated feat families hardcoded dataset "skills".

Resolution is now data-driven: the family spec's child_source names the dataset, child slugs come from the row key's own namespace, and the wiki loader accepts both the flat and tree layouts (topdata main still uses the flat one until the overhaul merges).

Verified against the skill-grouping-overhaul topdata branch: validate/build/wiki all clean with zero lock churn, focus feats stay on their adopted rows.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Reviewed-on: #43
Co-authored-by: vickydotbat <vickydotbat@tutamail.com>
Co-committed-by: vickydotbat <vickydotbat@tutamail.com>
2026-07-19 07:33:19 +00:00
archvillainette 24e57457b0 topdata: legacy aliases for skill-grouping renames (#42)
test / test (push) Successful in 1m27s
build-binaries / build-binaries (push) Successful in 2m11s
Extends legacyFamilyChildAliases so skill focus / greater skill focus children adopt the stock feat rows of the skills they were renamed from: medicine<-heal, investigation<-search, diplomacy<-influence/persuade, security<-open lock, intimidate<-taunt.

Existing craft_*/influence/disabledevice aliases stay: sow-topdata main still carries those skills until the overhaul merges. Needs a Crucible release before sow-topdata CI picks this up.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Reviewed-on: #42
Reviewed-by: xtul <mpiasecki720@protonmail.com>
Co-authored-by: vickydotbat <vickydotbat@tutamail.com>
Co-committed-by: vickydotbat <vickydotbat@tutamail.com>
2026-07-19 00:20:04 +00:00
archvillainette 43fe5e0373 Retire the Crucible container image (#41)
build-binaries / build-binaries (push) Successful in 2m10s
test / test (push) Successful in 1m23s
## Summary

The `registry.westgate.pw/deployment/crucible` image has no consumer: `prod.yml` no longer reserves a slot for it (contract settled in sow-platform PR #64) and no runtime or recovery path pulls it. Binaries, wrappers, and the Nix input remain the supported ways to run Crucible.

## What changed

- Deleted `.gitea/workflows/build-image.yml`, `publish-image.yml`, `test-image.yml`
- Deleted `docker/Dockerfile` and the `make image` target
- Updated README, AGENTS.md, and `docs/consumer-contract.md` to state the image is retired

Existing `deployment/crucible` images in the registry can be deleted at leisure; nothing references them.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Reviewed-on: #41
Co-authored-by: vickydotbat <vickydotbat@tutamail.com>
Co-committed-by: vickydotbat <vickydotbat@tutamail.com>
2026-07-17 07:26:58 +00:00
73 changed files with 1861 additions and 8272 deletions
@@ -0,0 +1,55 @@
---
name: data-driven-design
description: Use when adding a mode, switch case, config entry, special-cased name, or per-column/per-dataset handler to a tool that processes authored data — or when a tool "knows" a project's layout, dataset names, key prefixes, or format quirks and could not be open sourced as-is.
---
# Data-Driven Design
The tool implements **general mechanisms**; the data declares **all specific knowledge**. Every time code names a thing that lives in data — a dataset, a column, a format quirk — the tool stops being a tool and becomes a private extension of one project's layout.
**The test:** could this repo be open sourced and build a *different* project's data without code changes? Every hardcoded name, mode, and special case is a "no".
## The two smells
**1. The mode zoo.** A new data shape arrives and you add a named mode plus a switch case:
```yaml
- {column: AvailableHeadsMale, mode: external_hex_list, hex_width: 3}
- {column: PreferredAlignments, mode: alignment_set_mask, hex_width: 3}
- {column: ProficiencyFeats, mode: id_hex_list, hex_width: 4}
```
Three "modes" are one concept — *a set of ids with an engine-side encoding* — wearing three names. Each mode is code the data could have been. The fix is a smaller vocabulary of orthogonal primitives declared where the column is defined:
```json
"EquipSlots": {"shape": "id_set", "encode": "bitmask", "max": 31}
```
One parser for `id_set`, one encoder per encoding. Adding the next column is a data edit, zero code. The mode zoo *shrinks*: N special cases collapse into shape × encoding.
**2. Names in code.** `datasetRows("skills")`, `case "skills":`, `TrimPrefix(key, "skills:")` — the tool assumes the project's layout. When data moves (`skills``skills/core`), the tool breaks even though the data is self-consistent. Names must flow *from* the data: the spec that references a dataset names it; a key's namespace is whatever precedes its own colon. If code needs a name, some piece of data already knows it — read it from there.
## Deciding
Ask of every constant, case, and config knob: **whose knowledge is this?**
- *The mechanism's* (how to parse JSON, allocate rows, write a table) → code.
- *The data's* (which columns exist, how the engine encodes them, what things are called) → data, even if code is faster to write today.
Generalize only when it deletes: a primitive that replaces N modes is lazy; a framework "for future shapes" is not. One new shape may take its case *if* the case is spelled as a reusable primitive the next shape can declare.
## Rationalizations
| Excuse | Reality |
|---|---|
| "Just follow the existing pattern, it's proven" | The pattern *is* the defect. Each repetition raises the cost of the real fix. |
| "N cases isn't a crisis yet" | The crisis is per-case: every one couples the tool to one project forever. |
| "Release is tonight, generalize later" | Later never comes; tonight's mode is tomorrow's baseline. Declaring shape in data is usually the *same* hour of work. |
| "This name never changes" | Today's rename broke exactly such a name. Data moves; mechanisms shouldn't care. |
## Red flags — stop and re-shape
- A config entry that names both a dataset and a column to select behavior
- A new value in a `mode`/`kind`/`type` string enum with its own switch arm
- A literal in code that also appears in the data tree (`"skills"`, `"skills:"`)
- Per-project constants (row ranges, hex widths, id ceilings) living in Go instead of the dataset that owns them
+23 -13
View File
@@ -1,25 +1,22 @@
# Cross-platform Crucible binaries (D7 trigger standard):
# PR / push main -> cross-build ALL targets to prove they compile (no publish)
# tag v* -> build all targets + SHA256SUMS, upload to the Gitea release
# Cross-platform Crucible release binaries: a v* tag builds all targets,
# then uploads them, SHA256SUMS, and the canonical wrappers to its Gitea release.
# Crucible is pure Go (CGO_ENABLED=0), so cross-compiling is a fast loop.
name: build-binaries
on:
pull_request:
push:
branches: [main]
tags: ['v*']
paths-ignore:
- "docs/**"
- "README.md"
- "AGENTS.md"
- "LICENSE"
permissions:
code: read
releases: write
jobs:
build-binaries:
runs-on: nix-docker
timeout-minutes: 30
steps:
- uses: actions/checkout@v4
- uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
with: { fetch-depth: 0 }
- name: Cross-build all targets
@@ -45,9 +42,8 @@ jobs:
'
- name: Upload to Gitea release
if: github.event_name == 'push' && startsWith(github.ref, 'refs/tags/v')
env:
TOKEN: ${{ secrets.GITHUB_TOKEN }}
TOKEN: ${{ secrets.GITEA_TOKEN }}
TAG: ${{ github.ref_name }}
REPO: ${{ github.repository }}
SERVER: ${{ github.server_url }}
@@ -69,3 +65,17 @@ jobs:
"${api}/releases/${id}/assets?name=$(basename "$f")"
done
'
# Gitea has no asset retention (#52): without this every tag keeps its
# ~57 MB binary set forever. Best-effort — a stale asset is cheaper than
# a blocked publish, so a failure here never fails the release.
- name: Prune assets of older releases
continue-on-error: true
env:
TOKEN: ${{ secrets.GITEA_TOKEN }}
REPO: ${{ github.repository }}
SERVER: ${{ github.server_url }}
run: |
nix develop --command bash -c '
API="${SERVER}/api/v1/repos/${REPO}" scripts/prune-release-assets.sh
'
-52
View File
@@ -1,52 +0,0 @@
# Publish the immutable crucible:<sha> image on version tags.
# PR/main test builds live in test-image.yml and never publish, so registry
# packages are only generated for releases we intend to use (not on every merge).
#
# Daemonless: the host-mode runner has no container runtime, so the image is
# built by Nix (`nix build .#image`, see flake.nix) and pushed with skopeo
# straight from the OCI tarball. No `docker build`/`docker login` involved.
name: build-image
on:
push:
tags: ['v*']
env:
REGISTRY: registry.westgate.pw
IMAGE: deployment/crucible
jobs:
publish:
runs-on: nix-docker
steps:
- uses: actions/checkout@v4
with: { fetch-depth: 0 }
- name: Resolve tag
id: tag
run: echo "sha=$(git rev-parse --short=12 HEAD)" >> "$GITHUB_OUTPUT"
- name: Build OCI image (daemonless)
run: nix build .#image
# This workflow only runs on v* tags, so every run is a release publish.
- name: Publish image
env:
REGISTRY_USER: ${{ secrets.REGISTRY_USER }}
REGISTRY_PASSWORD: ${{ secrets.REGISTRY_PASSWORD }}
run: |
nix shell nixpkgs#skopeo -c skopeo copy \
--dest-creds "${REGISTRY_USER}:${REGISTRY_PASSWORD}" \
docker-archive:result \
"docker://${REGISTRY}/${IMAGE}:${{ steps.tag.outputs.sha }}"
- name: Emit release fragment
run: |
FRAG_REPO=sow-tools \
FRAG_SHA=${{ steps.tag.outputs.sha }} \
FRAG_ARTIFACT="${REGISTRY}/${IMAGE}:${{ steps.tag.outputs.sha }}" \
FRAG_URL="${REGISTRY}/${IMAGE}" \
FRAG_RUN_ID=${{ gitea.run_id }} \
bash scripts/emit-release-fragment.sh
- uses: actions/upload-artifact@v3
with: { name: release-fragment, path: release-fragment.json }
+47
View File
@@ -0,0 +1,47 @@
# Pull-request validation for Crucible. Releases and wrapper synchronization
# have their own narrow workflows because they need tag/main events.
name: ci
on:
pull_request:
permissions: read-all
jobs:
ci:
runs-on: nix-docker
timeout-minutes: 60
steps:
- uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
with:
fetch-depth: 0
- name: vet + test + lint
run: |
nix develop --command bash -c '
set -euo pipefail
go vet ./...
go test ./...
shellcheck scripts/*.sh
yamllint .gitea
'
- name: binary smoke (fail-closed contract)
run: nix develop --command make smoke
- name: Cross-build all targets
run: |
nix develop --command bash -c '
set -euo pipefail
sha="$(git rev-parse --short=12 HEAD)"
ldflags="-s -w -X git.westgate.pw/ShadowsOverWestgate/sow-tools/internal/buildinfo.Version=${sha}"
rm -rf dist && mkdir -p dist
export CGO_ENABLED=0
for target in linux/amd64 linux/arm64 darwin/amd64 darwin/arm64 windows/amd64 windows/arm64; do
os="${target%/*}"; arch="${target#*/}"
ext=""; [ "$os" = windows ] && ext=".exe"
echo "building crucible-${os}-${arch}${ext}"
GOOS="$os" GOARCH="$arch" go build -trimpath -ldflags "$ldflags" \
-o "dist/crucible-${os}-${arch}${ext}" ./cmd/crucible
done
'
-49
View File
@@ -1,49 +0,0 @@
# Manual compatibility publisher for the Crucible image.
# Normal release publishing happens in build-image.yml on v* tags; this is the
# break-glass / re-publish path, triggered by hand.
#
# Daemonless: built by Nix (`nix build .#image`) and pushed with skopeo from the
# OCI tarball. No `docker build`/`docker login` involved.
name: publish-image
on:
workflow_dispatch:
env:
REGISTRY: registry.westgate.pw
IMAGE: deployment/crucible
jobs:
publish:
runs-on: nix-docker
steps:
- uses: actions/checkout@v4
with: { fetch-depth: 0 }
- name: Resolve tag
id: tag
run: echo "sha=$(git rev-parse --short=12 HEAD)" >> "$GITHUB_OUTPUT"
- name: Build OCI image (daemonless)
run: nix build .#image
- name: Publish image
env:
REGISTRY_USER: ${{ secrets.REGISTRY_USER }}
REGISTRY_PASSWORD: ${{ secrets.REGISTRY_PASSWORD }}
run: |
nix shell nixpkgs#skopeo -c skopeo copy \
--dest-creds "${REGISTRY_USER}:${REGISTRY_PASSWORD}" \
docker-archive:result \
"docker://${REGISTRY}/${IMAGE}:${{ steps.tag.outputs.sha }}"
- name: Emit release fragment
run: |
FRAG_REPO=sow-tools \
FRAG_SHA=${{ steps.tag.outputs.sha }} \
FRAG_ARTIFACT="${REGISTRY}/${IMAGE}:${{ steps.tag.outputs.sha }}" \
FRAG_URL="${REGISTRY}/${IMAGE}" \
FRAG_RUN_ID=${{ gitea.run_id }} \
bash scripts/emit-release-fragment.sh
- uses: actions/upload-artifact@v3
with: { name: release-fragment, path: release-fragment.json }
+4 -1
View File
@@ -14,11 +14,14 @@ on:
- 'wrappers/crucible.sh'
- 'wrappers/crucible.ps1'
permissions: read-all
jobs:
sync:
runs-on: nix-docker
timeout-minutes: 30
steps:
- uses: actions/checkout@v4
- uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
- name: Open sync PRs to consumers
env:
-19
View File
@@ -1,19 +0,0 @@
# Test-build the Crucible image on PRs. Proves `nix build .#image` still works
# (daemonless, Nix-built OCI tarball) but does NOT publish — release publishing
# happens in build-image.yml on v* tags. PR-only: with up-to-date-before-merge
# protection, main == the tested PR head, so a throwaway post-merge rebuild that
# publishes nothing is pure waste.
name: test-image
on:
pull_request:
jobs:
build-image:
runs-on: nix-docker
steps:
- uses: actions/checkout@v4
with: { fetch-depth: 0 }
- name: Build OCI image (daemonless, no publish)
run: nix build .#image
-30
View File
@@ -1,30 +0,0 @@
# Lint + unit tests for the Crucible suite. PR-first: PRs + push to main (D7).
name: test
on:
push:
branches: [main]
paths-ignore:
- "docs/**"
- "README.md"
- "AGENTS.md"
- "LICENSE"
pull_request:
jobs:
test:
runs-on: nix-docker
steps:
- uses: actions/checkout@v4
with: { fetch-depth: 0 }
- name: vet + test + lint
run: |
nix develop --command bash -c '
set -euo pipefail
go vet ./...
go test ./...
shellcheck scripts/*.sh
yamllint .gitea
'
- name: binary smoke (fail-closed contract)
run: nix develop --command make smoke
+22 -4
View File
@@ -41,13 +41,14 @@ section and `.gitea/workflows/`.
- Dev loop: `nix develop`, then `make check` / `make build` / `make smoke`
(see Commands below).
- Release: push a `v*` tag. CI uploads cross-built binaries + wrappers to the
Gitea release and publishes the `crucible` container image.
Gitea release. There is no container image; Crucible ships as binaries,
wrappers, and the Nix input.
- Consumers (they download released binaries via the wrapper; they never
vendor a toolkit):
- sow-module — https://git.westgate.pw/ShadowsOverWestgate/sow-module
- sow-topdata — https://git.westgate.pw/ShadowsOverWestgate/sow-topdata
- sow-assets-manifest — https://git.westgate.pw/ShadowsOverWestgate/sow-assets-manifest
- sow-platform (deploys the released image/pins) —
- sow-platform (infra/deploy authority) —
https://git.westgate.pw/ShadowsOverWestgate/sow-platform
## What this repo owns / does not own
@@ -62,7 +63,7 @@ is `sow-platform`).
1. **Fail closed, never fake.** A builder with no migrated logic yet (`depot`)
exits `70`. Do not stub a builder to emit a placeholder artifact.
2. **Binaries are not committed.** They are CI artifacts / image layers.
2. **Binaries are not committed.** They are CI artifacts.
`/bin/`, `*.exe`, `nwn-tool`, `sow-toolkit` are gitignored.
3. **The registry is the command surface.** `internal/dispatch.Registry` is the
single source of truth; keep it in sync with `cmd/` and
@@ -81,9 +82,26 @@ is `sow-platform`).
nix develop && make check # vet + test + shellcheck + yamllint
make build # cmd/* -> ./bin
make smoke # assert fail-closed contract
make image # crucible:<sha>
```
## Tests
Tests must survive harmless changes to constants, defaults, wording, ordering, fixture data, and internal implementation details. A test that fails merely because a basic value changed is usually a bad test. Only assert exact values when the value is part of a documented public contract, external protocol, compatibility requirement, security rule, migration, or business rule.
## Agent skills
### Issue tracker
Issues live in Gitea at git.westgate.pw (`ShadowsOverWestgate/sow-tools`), managed with the `tea` CLI. Issues follow ownership — file work in the repo that owns it, not the one you happen to be standing in. See `docs/agents/issue-tracker.md`.
### Triage labels
Default label vocabulary (`needs-triage`, `needs-info`, `ready-for-agent`, `ready-for-human`, `wontfix`). See `docs/agents/triage-labels.md`.
### Domain docs
Single-context: `CONTEXT.md` at the repo root plus `docs/adr/`. See `docs/agents/domain.md`.
### Where work lives
Markdown here is **reference, law, or an ADR — nothing else** (`sow-codebase` ADR-0001). Live work -> wayfinder maps + Gitea issues (closeable, assignable, queryable). Settled decisions -> ADR files in `docs/adr/` (immutable, never closed, only superseded). Standing law -> `DOCTRINE.md` / `AGENTS.md` / `CONTEXT.md`. Current-state reference -> docs describing what the code does now. Everything else — plans, specs, concepts, handoffs, trackers — is process: it belongs in a Gitea issue, not a file. Harvest unfinished intent to an issue before deleting a process doc. `sow-docs` is deprecated and read-only.
+3 -7
View File
@@ -1,11 +1,13 @@
SHELL := bash
.ONESHELL:
.PHONY: check test vet build smoke fmt image
.PHONY: check test vet build smoke fmt
# Lint + unit tests. Mirrors the `test` CI job; runs green inside `nix develop`.
check: vet test
shellcheck scripts/*.sh
yamllint .gitea
bash tests/workflow-contract.sh
bash tests/prune-release-assets.sh
vet:
go vet ./...
@@ -22,9 +24,3 @@ smoke: build
fmt:
gofmt -l -w cmd internal
# Build the Crucible image locally. Requires docker; mirrors build-image CI.
IMAGE ?= crucible
GIT_SHA ?= $(shell git rev-parse --short=12 HEAD 2>/dev/null || echo unknown)
image:
docker build --build-arg GIT_SHA=$(GIT_SHA) -f docker/Dockerfile -t $(IMAGE):$(GIT_SHA) .
+9 -11
View File
@@ -71,25 +71,23 @@ nix develop # Go + shellcheck + yamllint + make
make check # go vet + go test + shellcheck + yamllint
make build # build every cmd/* into ./bin (gitignored)
make smoke # build + assert the fail-closed contract
make image # docker build -> crucible:<sha>
```
Binaries are **never committed** — they are CI artifacts / image layers (D19).
Binaries are **never committed** — they are CI artifacts (D19).
This retires the old habit of checking in `nwn-tool` / `sow-toolkit`.
## CI
PR-first (D7): checks run on pull requests and on push to `main`; the only
publish event is a `v*` tag (see `runbooks/ci-trigger-standard.md` in sow-docs,
PR-first (D7): checks run once on pull requests; the only publish event is a
`v*` tag (see `runbooks/ci-trigger-standard.md` in sow-docs,
https://git.westgate.pw/ShadowsOverWestgate/sow-docs).
- `test.yml` — vet, test, shellcheck, yamllint, binary smoke (PR + main).
- `test-image.yml` — build the OCI image to prove it compiles (PR + main, no push).
- `build-binaries.yml`cross-build all targets (PR + main); on a `v*` tag, upload
the binaries, `SHA256SUMS`, and the wrappers to the Gitea release.
- `build-image.yml` — on a `v*` tag, build and publish
`registry.westgate.pw/deployment/crucible:<sha>`.
- `publish-image.yml` — manual `workflow_dispatch` break-glass republish.
- `ci.yml` — vet, test, shellcheck, yamllint, binary smoke, and cross-build all
targets once per pull request.
- `build-binaries.yml`on a `v*` tag, cross-build and upload the binaries,
`SHA256SUMS`, and the wrappers to the Gitea release, then delete the assets
of every release except the newest two — Gitea keeps them forever otherwise,
and every binary is reproducible from its tag.
- `sync-wrappers.yml` — on a `main` push that touches `wrappers/`, auto-PR the
canonical wrappers to the consumer repos in `wrappers/consumers.txt`.
Consumer drift checks run after those PRs merge to `main`, not on the PRs
-38
View File
@@ -1,38 +0,0 @@
# syntax=docker/dockerfile:1
#
# Crucible toolchain image: registry.westgate.pw/deployment/crucible:<git-sha>
#
# Reproducible multi-stage build, no on-VPS build. Produces every cmd/* binary
# and ships them on a static base. The `crucible` dispatcher is the entrypoint;
# consumer CI can also call the standalone crucible-<name> binaries by path.
#
FROM golang:1.26-alpine AS build
WORKDIR /src
RUN apk add --no-cache git
# go.sum is committed now that the migrated packages pull golang.org/x/text and
# gopkg.in/yaml.v3; the glob keeps the build working if it is ever absent.
COPY go.mod go.sum* ./
RUN go mod download
COPY . .
ARG GIT_SHA=unknown
ENV CGO_ENABLED=0
RUN set -eux; \
mkdir -p /out; \
for dir in ./cmd/*/; do \
name="$(basename "${dir}")"; \
go build -trimpath \
-ldflags "-s -w -X git.westgate.pw/ShadowsOverWestgate/sow-tools/internal/buildinfo.Version=${GIT_SHA}" \
-o "/out/${name}" "${dir}"; \
done
FROM debian:12-slim
# ca-certificates: builders fetch published manifests over HTTPS.
RUN set -eux; \
apt-get update; \
apt-get install -y --no-install-recommends ca-certificates; \
rm -rf /var/lib/apt/lists/*; \
useradd --system --create-home --uid 65532 nonroot
COPY --from=build /out/ /usr/local/bin/
USER nonroot
ENTRYPOINT ["/usr/local/bin/crucible"]
CMD ["help"]
@@ -0,0 +1,55 @@
# ADR-0001: Markdown in this repo is reference, law, or an ADR — nothing else
- **Status:** Accepted
- **Date:** 2026-07-24
- **Origin:** Adopted workspace-wide from `sow-codebase` ADR-0001, which records
the full context (a `docs/` tree that had grown to 434 files, ~27MB, most of
it dead process artifacts). This repo adopts the same law so the rule is
local, not a cross-repo reference.
## Context
Markdown that *claims to describe current state or a plan* rots, because
reality diverges and nobody edits the file. Markdown that claims neither — a
dated, immutable decision record, or a standing rule — does not rot.
Two different things get conflated:
- **Rationale** — why a system is shaped the way it is. Worth keeping.
- **Process artifacts** — plans, specs, concepts, handoffs, trackers. A record
of how work happened, not what is true now.
Left unchecked, every completed effort leaves its planning behind and the repo
accumulates a "historical" pile that agents and humans must route around to
find the few live docs.
## Decision
Markdown may exist in this repo only as one of three genres:
1. **Current-state reference** — describes what the code does now, kept honest
by code review touching it.
2. **Standing law**`DOCTRINE.md`, root and folder-scoped `AGENTS.md`,
`CONTEXT.md`. States rules and vocabulary, not plans.
3. **Immutable decision record** — ADRs under `docs/adr/`. Append-only; never
edited, only superseded by a later ADR that points back.
Anything else — implementation plans, design specs, concepts, handoffs,
progress trackers, scratch — is **process** and does not live in the repo. It
lives in Gitea issues and wayfinder maps, where it can be assigned, closed, and
superseded. Small tasks need no written plan at all.
Deleting a process document is not destroying history: `git log -- <path>`
recovers it. The exception is *unfinished intent* (a design never built, an
open question still wanted) — that is live, not history, and must be harvested
to a Gitea ticket before its file is deleted. A citation from a living doc or
from code must be resolved before the cited file is deleted.
## Consequences
- The documentation map lists only current truth; there is no "historical" pile
to route around.
- No agent, under any plugin or skill, writes process-markdown into the repo.
The rule is plugin-agnostic on purpose — it binds the agent, not a named tool.
- History of deleted process docs is in git; unfinished intent is in the issue
tracker; rationale is in ADRs and law. Each thing has exactly one home.
@@ -0,0 +1,12 @@
# ADR-0002: `sow-tools` becomes the Crucible suite
- **Status:** Accepted
- **Date:** 2026-06-11
- **Migrated from:** `sow-docs` `07-decisions-log.md` entry **D11**, on the
retirement of `sow-docs`. Content is the original decision, unchanged.
- **Decision:** Rename the future toolchain surface to Crucible: one Go module,
multiple `cmd/` binaries, plus a dispatcher so wrapper commands can stay
stable.
- **Rationale:** Depot, HAK, module, topdata, and wiki tooling share internals
but have different command surfaces. One module avoids premature repo splits.
@@ -0,0 +1,29 @@
# ADR-0003: Crucible binary contract: standalone shims, fail-closed scaffold, no committed binaries
- **Status:** Accepted
- **Date:** 2026-06-11
- **Migrated from:** `sow-docs` `07-decisions-log.md` entry **D19**, on the
retirement of `sow-docs`. Content is the original decision, unchanged.
- **Decision:** Phase 5 builds `migration/sow-tools` (Crucible, D11) as a
multi-binary Go scaffold:
- One `crucible` dispatcher plus standalone `crucible-{depot,hak,module,topdata,wiki}`
binaries sharing one registry (`internal/dispatch`). The dispatcher is for
humans/CI; the **standalone shims are canonical for consumers** so wrapper
scripts resolve a single-token command and `"$builder" args` quoting stays
correct.
- Consumer resolution order: explicit env override
(`$SOW_MODULE_BUILD`/`$SOW_TOPDATA_BUILD`/`$CRUCIBLE`) → `crucible-<name>`
legacy `sow-<name>-build`. CI runs inside the pinned `crucible:<sha>` image.
- Every builder is **unwired** in the scaffold and fails closed (exit 70);
it never fakes an artifact. The `internal/` logic is migrated from
`gitea/sow-tools` by the operator at cutover (hard rule: no source
transplant by tooling).
- **Binaries are never committed** — they are CI artifacts / image layers.
Retires the committed `nwn-tool` / `tools/sow-toolkit`.
- Builders take `NWN_ROOT` only via explicit env/flag; no `$HOME` defaulting.
- **Rationale:** Locks the command surface, container, and CI shape so migrated
logic drops into a stable frame; aligns the three artifact repos onto one
Crucible contract while keeping back-compat with the pre-D11 skeletons.
- **Image name:** `registry.westgate.pw/deployment/crucible:<git-sha>` (the bare
`crucible:<sha>` is the local/dev tag).
@@ -0,0 +1,24 @@
# ADR-0004: Nix-built OCI images start server-side with Crucible; local Anvil images deferred
- **Status:** Accepted
- **Date:** 2026-06-12
- **Migrated from:** `sow-docs` `07-decisions-log.md` entry **D29**, on the
retirement of `sow-docs`. Content is the original decision, unchanged.
- **Decision:** Introduce Nix-built OCI images as an artifact-production option,
starting with `sow-tools`/Crucible in server-side CI. `sow-tools` will grow a
Nix-built `crucible` package plus `crucible-image`; PR CI builds/smokes it and
`main` publishes the normal pinned OCI image
`registry.westgate.pw/deployment/crucible:<sha>`. `docker/Dockerfile` stays during
parity and as the client-compatible fallback.
- **Local rule:** Nix users may get optional wrappers that provide tools and
call the existing Dockerfiles for local Anvil/NWServer testing. A true
`nix build .#nwserver-test-image` is explicitly deferred until real
`sow-codebase/src` exists and the Dockerfile image is proven.
- **Rationale:** Server-side image build shape is harder to change once deploy
promotion and registry gates are active, so prove Nix image builds before
cutover. Crucible is the lowest-risk pilot because it is a Go toolchain image;
NodeBB and Anvil/NWServer depend more heavily on upstream container filesystem
semantics.
- **Non-goals:** no source builds in `sow-platform`; no `nix-sidecar` or
Kubernetes-style runtime cache layer; no removal of client Dockerfiles.
+51
View File
@@ -0,0 +1,51 @@
# Domain Docs
How the engineering skills should consume this repo's domain documentation when exploring the codebase.
## Before exploring, read these
- **`CONTEXT.md`** at the repo root, or
- **`CONTEXT-MAP.md`** at the repo root if it exists — it points at one `CONTEXT.md` per context. Read each one relevant to the topic.
- **`docs/adr/`** — read ADRs that touch the area you're about to work in. In multi-context repos, also check `src/<context>/docs/adr/` for context-scoped decisions.
If any of these files don't exist, **proceed silently**. Don't flag their absence; don't suggest creating them upfront. The `/domain-modeling` skill (reached via `/grill-with-docs` and `/improve-codebase-architecture`) creates them lazily when terms or decisions actually get resolved.
## File structure
Single-context repo (most repos):
```
/
├── CONTEXT.md
├── docs/adr/
│ ├── 0001-event-sourced-orders.md
│ └── 0002-postgres-for-write-model.md
└── src/
```
Multi-context repo (presence of `CONTEXT-MAP.md` at the root):
```
/
├── CONTEXT-MAP.md
├── docs/adr/ ← system-wide decisions
└── src/
├── ordering/
│ ├── CONTEXT.md
│ └── docs/adr/ ← context-specific decisions
└── billing/
├── CONTEXT.md
└── docs/adr/
```
## Use the glossary's vocabulary
When your output names a domain concept (in an issue title, a refactor proposal, a hypothesis, a test name), use the term as defined in `CONTEXT.md`. Don't drift to synonyms the glossary explicitly avoids.
If the concept you need isn't in the glossary yet, that's a signal — either you're inventing language the project doesn't use (reconsider) or there's a real gap (note it for `/domain-modeling`).
## Flag ADR conflicts
If your output contradicts an existing ADR, surface it explicitly rather than silently overriding:
> _Contradicts ADR-0007 (event-sourced orders) — but worth reopening because…_
+88
View File
@@ -0,0 +1,88 @@
# Issue tracker: Gitea (via tea)
Issues for this repo live in Gitea at `git.westgate.pw`, repo
`ShadowsOverWestgate/sow-tools`. Use the `tea` CLI for all operations —
`gh` does not work here. For anything `tea` lacks a subcommand for, use
`tea api <endpoint>` (Gitea's API mirrors GitHub's closely).
Authenticate with your own `tea` login (`tea login add`); never commit tokens
or tea config into this repo. Note Gitea blocks self-review, so approving a PR
needs a different account than the one that opened it.
## Where work lives
Markdown in this repo is **reference, law, or an ADR — nothing else**
(`sow-codebase` ADR-0001). Four homes, no overlap:
- **Live work** → wayfinder maps + Gitea issues. Closeable, assignable,
queryable. Never a markdown file.
- **Settled decisions** → ADR files in `docs/adr/`. Immutable, findable, never
closed, never edited — only superseded by a later ADR pointing back. When an
issue ends in a durable decision, write the ADR, then close the issue
pointing at it. Decisions spanning repos go to `sow-platform/docs/adr/`.
- **Standing law** → `DOCTRINE.md`, `AGENTS.md`, `CONTEXT.md`. Rules that are
always true and vocabulary everyone shares — not the record of one decision.
- **Current-state reference** → docs describing what the code does now, kept
honest by review touching them.
Anything else — implementation plans, design specs, concepts, handoffs,
progress trackers, scratch — is **process**. It does not live in this repo. It
lives in a Gitea issue or a wayfinder map, where it can be assigned, closed,
and superseded. Small tasks need no written plan at all.
Deleting a process doc is not destroying history — `git log -- <path>` recovers
it. But *unfinished intent* (a design never built, an open question still
wanted) is live, not history: harvest it to a Gitea issue before deleting.
`sow-docs` is deprecated and read-only. Never add to it, never send work there.
## Which repo gets the issue
Issues follow ownership. File the issue in the repo that **owns the work**
see the Repo/Owns/Produces table in the workspace root `AGENTS.md`. Standing
in one repo is not a reason to file there.
If work spans repos, file it in the repo that owns the *outcome* and reference
the others from it. A wrong-repo issue is a routing bug, not a filing
preference — move it.
## Conventions
- **Create an issue**: `tea issues create --title "..." --description "..."`
- **Read an issue**: `tea issues <number>` and
`tea api repos/ShadowsOverWestgate/sow-tools/issues/<number>/comments` for comments.
- **List issues**: `tea issues list --state open` (add `--labels ...` to filter).
- **Comment**: `tea comment <number> "..." </dev/null`
Always redirect stdin. `tea` reads stdin to EOF and appends it to the body,
so any non-interactive shell (every agent) hangs forever without
`</dev/null`. Same trap on `tea issues create --description` and
`tea pr create`.
- **Apply / remove labels**: `tea api --method PATCH` on the issue, or
`tea api repos/ShadowsOverWestgate/sow-tools/issues/<number>/labels` endpoints.
- **Close**: `tea issues close <number>`
`tea` infers the repo from the git remote when run inside the clone.
Gitea shares one number space across issues and PRs.
## Pull requests as a triage surface
**PRs as a request surface: no.**
## When a skill says "publish to the issue tracker"
Create a Gitea issue with `tea issues create`.
## When a skill says "fetch the relevant ticket"
Run `tea issues <number>` plus the comments API call above.
## Wayfinding operations
Used by `/wayfinder`. The **map** is a single issue with **child** issues as tickets.
- **Map**: a single issue labelled `wayfinder:map`, holding the Notes / Decisions-so-far / Fog body. `tea issues create --title "..." --description "..." --labels wayfinder:map`.
- **Child ticket**: this Gitea instance (v1.27) has no native sub-issue hierarchy, so a child issue carries `Part of #<map>` at the top of its description, and is also added to a task list in the map body. Labels: `wayfinder:<type>` (`research`/`prototype`/`grilling`/`task`). Once claimed, the ticket is assigned to the driving dev.
- **Blocking**: Gitea's **native dependencies API** — the canonical, UI-visible representation (shows as "Depends on" / "Blocks" on the issue page). Add an edge with `tea api -X POST repos/ShadowsOverWestgate/sow-tools/issues/<child>/dependencies -f owner=ShadowsOverWestgate -f repo=sow-tools -F index=<blocker>`, where `<blocker>` is the blocker's issue **index** (its `#number` — Gitea's dependency API takes the index directly, unlike GitHub's numeric database id). Check status with `tea api repos/ShadowsOverWestgate/sow-tools/issues/<child>/dependencies` (GET) — a ticket is unblocked when every returned issue's `state` is `closed`.
- **Frontier query**: list the map's open children (`tea issues list --state open`, keep the ones whose description contains `Part of #<map>`), drop any with an open dependency (per the GET above) or an assignee; first in map order wins.
- **Claim**: `tea issues edit <n> --add-assignees <username>` — the session's first write.
- **Resolve**: `tea comment <n> "<answer>" </dev/null`, then `tea issues close <n>`, then append a context pointer (gist + link) to the map's Decisions-so-far.
+15
View File
@@ -0,0 +1,15 @@
# Triage Labels
The skills speak in terms of five canonical triage roles. This file maps those roles to the actual label strings used in this repo's issue tracker (Gitea — see `issue-tracker.md` for how to apply labels with `tea`).
| Label in mattpocock/skills | Label in our tracker | Meaning |
| -------------------------- | -------------------- | ---------------------------------------- |
| `needs-triage` | `needs-triage` | Maintainer needs to evaluate this issue |
| `needs-info` | `needs-info` | Waiting on reporter for more information |
| `ready-for-agent` | `ready-for-agent` | Fully specified, ready for an AFK agent |
| `ready-for-human` | `ready-for-human` | Requires human implementation |
| `wontfix` | `wontfix` | Will not be actioned |
When a skill mentions a role (e.g. "apply the AFK-ready triage label"), use the corresponding label string from this table.
Edit the right-hand column to match whatever vocabulary you actually use.
+9 -2
View File
@@ -29,9 +29,15 @@ aliases.
| `assets` | `fix-mdl` | Lowercase `.mdl` names and rewrite model identity to match. |
| `assets` | `check-dupes` | Report runtime-name (basename) collisions across dirs. |
| `assets` | `clean-dupes` | Delete clean-tree files whose basename collides with primary. |
| `depot` | `status` | Report referenced-vs-present drift against a backend. |
| `depot` | `push` | Upload referenced-but-absent blobs from a local depot. |
| `depot` | `verify` | Existence sweep plus sampled download re-hash. |
| `depot` | `get` | Fetch one blob with sha re-verify. |
| `depot` | `pull` | Incremental verified pull of every referenced blob. |
`crucible-depot` remains registered but unwired. It fails closed with exit `70`
and never emits placeholder artifacts.
`depot status` and `depot get` pick their backend either with `--out DIR`, a
depot tree on disk, or with `--target bunny|cdn`, a remote backend. The two
flags are mutually exclusive.
## Hidden compatibility aliases
@@ -49,6 +55,7 @@ but omitted from routine help and the interactive menu:
| `topdata` | `build-top-package` | `build-top-package` |
| `topdata` | `compare-topdata` | `compare-topdata` |
| `topdata` | `convert-topdata` | `convert-topdata` |
| `depot` | `--target local` | `--out $DEPOT_DIR` on `status`/`get` |
| `wiki` | `build-wiki` | `build-wiki` |
| `wiki` | `deploy-wiki` | `deploy-wiki` |
+4 -4
View File
@@ -49,10 +49,10 @@ by `flake.lock`. CI runs the _same_ `crucible <build>` as local dev, inside the
nix devshell (binary-cache fast). No build container, no token, no CI-only
pre-steps.
The `registry.westgate.pw/deployment/crucible:<sha>` image is **deployment-only**:
`prod.yml` pins it so tools travel with the runtime host (`nwn.enable`), a
disabled placeholder until the NWN stack lands. It is **not** a build tool and no
build/CI job consumes it.
The `registry.westgate.pw/deployment/crucible:<sha>` image is **retired**
(2026-07-17): nothing consumed it, `prod.yml` no longer reserves a slot for
it, and CI no longer builds or publishes it. Binaries, wrappers, and the Nix
input are the only supported ways to run Crucible.
## Determinism
@@ -126,8 +126,9 @@ exactly as the reference does:
- Beamdog install dirs.
- `--nwn <install>` overrides (path to the install root or directly to the
binary).
- **Headless.** The engine is a GUI binary. If there is no `DISPLAY` and
`xvfb-run` is present, wrap the call:
- **Headless.** The engine is a GUI binary. Require `xvfb-run` and wrap the call
regardless of the caller's `DISPLAY` so compilation never opens the client
UI; fail before invoking the engine when `xvfb-run` is unavailable:
`xvfb-run -a --server-args=-screen 0 1024x768x24 nwmain-linux compilemodel <stem>`.
- **Per model (one at a time — the engine's `development/` and `modelcompiler/`
folders are flat and single-slot):**
+6 -6
View File
@@ -40,13 +40,13 @@ func runCompile(args []string, stdout, stderr io.Writer, getenv func(string) str
}
binDir := filepath.Dir(nwmain)
// Headless wrap: no DISPLAY + xvfb-run present -> run under a virtual X.
var wrap []string
if getenv("DISPLAY") == "" {
if xvfb := look("xvfb-run"); xvfb != "" {
wrap = []string{xvfb, "-a", "--server-args=-screen 0 1024x768x24"}
}
// Always use a virtual X so compilation never opens the client UI.
xvfb := look("xvfb-run")
if xvfb == "" {
fmt.Fprintln(stderr, "assets compile: xvfb-run not found — install it to run the NWN model compiler headlessly")
return exitTool
}
wrap := []string{xvfb, "-a", "--server-args=-screen 0 1024x768x24"}
files, err := walk(dirs, mdlExt, !*nonRecursive)
if err != nil {
+63 -2
View File
@@ -4,6 +4,7 @@ import (
"bytes"
"os"
"path/filepath"
"strings"
"testing"
)
@@ -23,13 +24,19 @@ func TestCompileDrivesEngineAndReplacesInPlace(t *testing.T) {
if err := os.WriteFile(nwmain, []byte("#!/bin/sh\n"), 0o755); err != nil {
t.Fatal(err)
}
xvfbDir := t.TempDir()
xvfb := filepath.Join(xvfbDir, "xvfb-run")
if err := os.WriteFile(xvfb, []byte("#!/bin/sh\n"), 0o755); err != nil {
t.Fatal(err)
}
t.Setenv("PATH", xvfbDir+string(os.PathListSeparator)+os.Getenv("PATH"))
getenv := func(k string) string {
switch k {
case "HOME":
return home
case "DISPLAY":
return ":0" // pretend a display exists so no xvfb wrap is needed
return ":0" // a desktop display must not make compilation interactive
}
return ""
}
@@ -38,7 +45,11 @@ func TestCompileDrivesEngineAndReplacesInPlace(t *testing.T) {
orig := runner
defer func() { runner = orig }()
runner = func(dir string, env []string, name string, args ...string) ([]byte, error) {
// args: compilemodel <stem>
if name != xvfb || len(args) != 5 || args[0] != "-a" ||
args[1] != "--server-args=-screen 0 1024x768x24" || args[2] != nwmain ||
args[3] != "compilemodel" {
t.Fatalf("engine command = %q %q, want xvfb-run wrapping nwmain", name, args)
}
stem := args[len(args)-1]
compiled := filepath.Join(mc, stem+".mdl")
return nil, os.WriteFile(compiled, []byte("\x00\x00compiled"), 0o644)
@@ -77,6 +88,11 @@ func TestCompileAbortsOnNameMismatch(t *testing.T) {
if err := os.WriteFile(nwmain, []byte("#!/bin/sh\n"), 0o755); err != nil {
t.Fatal(err)
}
xvfbDir := t.TempDir()
if err := os.WriteFile(filepath.Join(xvfbDir, "xvfb-run"), []byte("#!/bin/sh\n"), 0o755); err != nil {
t.Fatal(err)
}
t.Setenv("PATH", xvfbDir+string(os.PathListSeparator)+os.Getenv("PATH"))
getenv := func(k string) string {
if k == "HOME" {
return home
@@ -108,3 +124,48 @@ func TestCompileAbortsOnNameMismatch(t *testing.T) {
t.Fatal("engine should not run for a name-mismatched model")
}
}
func TestCompileFailsClosedWithoutXvfb(t *testing.T) {
home := t.TempDir()
userData := filepath.Join(home, ".local", "share", "Neverwinter Nights")
for _, d := range []string{filepath.Join(userData, "development"), filepath.Join(userData, "modelcompiler")} {
if err := os.MkdirAll(d, 0o755); err != nil {
t.Fatal(err)
}
}
nwmain := filepath.Join(t.TempDir(), "nwmain-linux")
if err := os.WriteFile(nwmain, []byte("#!/bin/sh\n"), 0o755); err != nil {
t.Fatal(err)
}
t.Setenv("PATH", t.TempDir())
getenv := func(k string) string {
if k == "HOME" {
return home
}
return ""
}
orig := runner
defer func() { runner = orig }()
engineCalled := false
runner = func(string, []string, string, ...string) ([]byte, error) {
engineCalled = true
return nil, nil
}
srcDir := t.TempDir()
if err := os.WriteFile(filepath.Join(srcDir, "foo.mdl"),
[]byte("newmodel foo\nbeginmodelgeom foo\n node dummy foo\n parent null\n endnode\nendmodelgeom foo\ndonemodel foo\n"), 0o644); err != nil {
t.Fatal(err)
}
var stdout, stderr bytes.Buffer
if code := runCompile([]string{"--nwn", nwmain, srcDir}, &stdout, &stderr, getenv); code != exitTool {
t.Fatalf("compile exit = %d, want %d\n%s", code, exitTool, stderr.String())
}
if engineCalled {
t.Fatal("engine must not run without xvfb-run")
}
if !strings.Contains(stderr.String(), "xvfb-run") {
t.Fatalf("missing actionable xvfb-run error: %s", stderr.String())
}
}
+46 -28
View File
@@ -11,6 +11,7 @@ import (
"os"
"path/filepath"
"sort"
"strings"
"sync"
"time"
)
@@ -51,13 +52,15 @@ func Run(args []string, stdout, stderr io.Writer, getenv func(string) string) in
func printRunUsage(w io.Writer) {
fmt.Fprint(w, `usage:
depot status [--manifests DIR] [--source LOCAL_DEPOT] --target bunny|cdn|local
depot push [--manifests DIR] --source LOCAL_DEPOT --target bunny
depot verify [--manifests DIR] --target bunny|cdn [--sample N]
depot get <sha> <dest> --target cdn|bunny|local
depot pull [--manifests DIR] --dest DIR --target cdn|bunny
depot status [--manifests DIR] --target bunny|cdn | --out DIR
depot push [--manifests DIR] --source LOCAL_DEPOT --target bunny
depot verify [--manifests DIR] --target bunny|cdn [--sample N]
depot get <sha> <dest> --target cdn|bunny | --out DIR
depot pull [--manifests DIR] --dest DIR --target cdn|bunny
--target local uses the DEPOT_DIR environment variable as the local root.
--out DIR reads and writes a depot tree on disk at DIR instead of a remote
backend. --target names a remote backend only; the two are mutually exclusive.
--target local is a deprecated alias for --out $DEPOT_DIR.
`)
}
@@ -65,16 +68,27 @@ func printRunUsage(w io.Writer) {
// than internal/backend failures (exit 70).
var errUsage = errors.New("usage error")
// resolveBackend builds the named backend, resolving "local" against DEPOT_DIR.
func resolveBackend(target string, getenv func(string) string, cfg Config) (Backend, error) {
root := ""
if target == "local" {
root = getenv("DEPOT_DIR")
// resolveBackend picks the backend for a command that can work either against a
// depot tree on disk (--out DIR) or a remote backend (--target bunny|cdn).
// "--target local" stays as a deprecated alias for --out $DEPOT_DIR so older
// callers keep working.
func resolveBackend(out, target string, getenv func(string) string, cfg Config) (Backend, error) {
switch {
case out != "" && target != "":
return nil, fmt.Errorf("--out and --target are mutually exclusive: %w", errUsage)
case out != "":
return NewBackend("local", out, cfg)
case target == "local":
root := getenv("DEPOT_DIR")
if root == "" {
return nil, fmt.Errorf("--target local requires DEPOT_DIR to be set: %w", errUsage)
return nil, fmt.Errorf("--target local requires DEPOT_DIR to be set (prefer --out DIR): %w", errUsage)
}
return NewBackend("local", root, cfg)
case target == "":
return nil, fmt.Errorf("--out DIR or --target bunny|cdn is required: %w", errUsage)
default:
return NewBackend(target, "", cfg)
}
return NewBackend(target, root, cfg)
}
// backendErrExit maps a resolveBackend error onto the exit contract.
@@ -119,18 +133,18 @@ func runStatus(args []string, stdout, stderr io.Writer, getenv func(string) stri
fs := flag.NewFlagSet("status", flag.ContinueOnError)
fs.SetOutput(stderr)
manifests := fs.String("manifests", "assets", "directory of *.yml manifests")
_ = fs.String("source", "", "local depot root (unused for status target=local; see DEPOT_DIR)")
target := fs.String("target", "", "bunny|cdn|local")
source := fs.String("source", "", "deprecated and ignored (use --out DIR for a depot tree on disk)")
out := fs.String("out", "", "depot tree on disk to read instead of a remote backend")
target := fs.String("target", "", "bunny|cdn")
if err := fs.Parse(args); err != nil {
return exitUsage
}
if *target == "" {
fmt.Fprintln(stderr, "depot status: --target is required")
return exitUsage
if *source != "" {
fmt.Fprintln(stderr, "depot status: --source is ignored; use --out DIR")
}
cfg := LoadConfig(getenv)
backend, err := resolveBackend(*target, getenv, cfg)
backend, err := resolveBackend(*out, *target, getenv, cfg)
if err != nil {
fmt.Fprintln(stderr, "depot status:", err)
return backendErrExit(err)
@@ -309,13 +323,21 @@ func runVerify(args []string, stdout, stderr io.Writer, getenv func(string) stri
func runGet(args []string, stdout, stderr io.Writer, getenv func(string) string) int {
fs := flag.NewFlagSet("get", flag.ContinueOnError)
fs.SetOutput(stderr)
target := fs.String("target", "", "cdn|bunny|local")
if err := fs.Parse(args); err != nil {
out := fs.String("out", "", "depot tree on disk to read instead of a remote backend")
target := fs.String("target", "", "cdn|bunny")
// Positionals come first in the documented usage, and Go's flag package
// stops parsing at the first one, so split them off by hand.
split := 0
for split < len(args) && !strings.HasPrefix(args[split], "-") {
split++
}
positional := args[:split]
if err := fs.Parse(args[split:]); err != nil {
return exitUsage
}
positional := fs.Args()
positional = append(positional, fs.Args()...)
if len(positional) != 2 {
fmt.Fprintln(stderr, "depot get: usage: depot get <sha> <dest> --target cdn|bunny|local")
fmt.Fprintln(stderr, "depot get: usage: depot get <sha> <dest> --target cdn|bunny | --out DIR")
return exitUsage
}
sha, dest := positional[0], positional[1]
@@ -323,13 +345,9 @@ func runGet(args []string, stdout, stderr io.Writer, getenv func(string) string)
fmt.Fprintf(stderr, "depot get: invalid sha %q\n", sha)
return exitUsage
}
if *target == "" {
fmt.Fprintln(stderr, "depot get: --target is required")
return exitUsage
}
cfg := LoadConfig(getenv)
backend, err := resolveBackend(*target, getenv, cfg)
backend, err := resolveBackend(*out, *target, getenv, cfg)
if err != nil {
fmt.Fprintln(stderr, "depot get:", err)
return backendErrExit(err)
+96
View File
@@ -54,6 +54,91 @@ func TestRunUsage(t *testing.T) {
})
}
func TestOutFlag(t *testing.T) {
t.Run("status --out reads the given tree", func(t *testing.T) {
sha := shaOf("out-blob")
manifests := t.TempDir()
writeManifest(t, manifests, sha, 8)
depotDir := t.TempDir()
writeBlob(t, depotDir, sha, "out-blob")
var out, errb bytes.Buffer
code := Run([]string{"status", "--manifests", manifests, "--out", depotDir}, &out, &errb, testGetenv(nil))
if code != 0 {
t.Fatalf("expected 0, got %d (stdout=%s stderr=%s)", code, out.String(), errb.String())
}
if !bytesContains(out.String(), "present=1") {
t.Fatalf("expected present=1, got %s", out.String())
}
})
t.Run("get --out fetches from the given tree", func(t *testing.T) {
sha := shaOf("get-blob")
depotDir := t.TempDir()
writeBlob(t, depotDir, sha, "get-blob")
dest := filepath.Join(t.TempDir(), "fetched")
var out, errb bytes.Buffer
code := Run([]string{"get", sha, dest, "--out", depotDir}, &out, &errb, testGetenv(nil))
if code != 0 {
t.Fatalf("expected 0, got %d (stderr=%s)", code, errb.String())
}
got, err := os.ReadFile(dest)
if err != nil {
t.Fatal(err)
}
if string(got) != "get-blob" {
t.Fatalf("got %q", got)
}
})
t.Run("--out with any --target is rejected", func(t *testing.T) {
for _, target := range []string{"bunny", "cdn", "local"} {
var out, errb bytes.Buffer
code := Run([]string{"status", "--manifests", t.TempDir(), "--out", t.TempDir(), "--target", target}, &out, &errb, testGetenv(nil))
if code != 64 {
t.Fatalf("--target %s: expected 64, got %d (stderr=%s)", target, code, errb.String())
}
}
})
t.Run("get accepts flags before the positionals", func(t *testing.T) {
sha := shaOf("flags-first-blob")
depotDir := t.TempDir()
writeBlob(t, depotDir, sha, "flags-first-blob")
dest := filepath.Join(t.TempDir(), "fetched")
var out, errb bytes.Buffer
code := Run([]string{"get", "--out", depotDir, sha, dest}, &out, &errb, testGetenv(nil))
if code != 0 {
t.Fatalf("expected 0, got %d (stderr=%s)", code, errb.String())
}
})
t.Run("neither --out nor --target is rejected", func(t *testing.T) {
var out, errb bytes.Buffer
code := Run([]string{"status", "--manifests", t.TempDir()}, &out, &errb, testGetenv(nil))
if code != 64 {
t.Fatalf("expected 64, got %d (stderr=%s)", code, errb.String())
}
})
t.Run("--target local still works as a hidden alias", func(t *testing.T) {
sha := shaOf("alias-blob")
manifests := t.TempDir()
writeManifest(t, manifests, sha, 11)
depotDir := t.TempDir()
writeBlob(t, depotDir, sha, "alias-blob")
var out, errb bytes.Buffer
code := Run([]string{"status", "--manifests", manifests, "--target", "local"}, &out, &errb,
testGetenv(map[string]string{"DEPOT_DIR": depotDir}))
if code != 0 {
t.Fatalf("expected 0, got %d (stdout=%s stderr=%s)", code, out.String(), errb.String())
}
})
}
func TestGetInvalidSHA(t *testing.T) {
var out, errb bytes.Buffer
dir := t.TempDir()
@@ -172,6 +257,17 @@ func bytesContains(s, substr string) bool {
return bytes.Contains([]byte(s), []byte(substr))
}
func writeBlob(t *testing.T, root, sha, content string) {
t.Helper()
path := filepath.Join(root, BlobKey(sha))
if err := os.MkdirAll(filepath.Dir(path), 0755); err != nil {
t.Fatal(err)
}
if err := os.WriteFile(path, []byte(content), 0644); err != nil {
t.Fatal(err)
}
}
func writeManifest(t *testing.T, dir, sha string, size int64) {
t.Helper()
content := fmt.Sprintf("assets:\n - path: foo\n sha256: %s\n size: %d\n", sha, size)
+3 -3
View File
@@ -94,12 +94,12 @@ var Registry = []Builder{
{
Name: "depot",
Bin: "crucible-depot",
Summary: "content-addressed asset depot (local/cdn/bunny)",
Summary: "content-addressed asset depot (disk/cdn/bunny)",
Commands: []Command{
{Name: "status", Summary: "report referenced-vs-present drift against a backend", Usage: "crucible depot status [--manifests DIR] [--source DIR] --target bunny|cdn|local"},
{Name: "status", Summary: "report referenced-vs-present drift against a backend", Usage: "crucible depot status [--manifests DIR] --target bunny|cdn | --out DIR"},
{Name: "push", Summary: "upload referenced-but-absent blobs from a local depot", Usage: "crucible depot push [--manifests DIR] --source DIR --target bunny"},
{Name: "verify", Summary: "existence sweep plus sampled download re-hash", Usage: "crucible depot verify [--manifests DIR] --target bunny|cdn [--sample N]"},
{Name: "get", Summary: "fetch one blob with sha re-verify", Usage: "crucible depot get <sha> <dest> --target cdn|bunny|local"},
{Name: "get", Summary: "fetch one blob with sha re-verify", Usage: "crucible depot get <sha> <dest> --target cdn|bunny | --out DIR"},
{Name: "pull", Summary: "incremental verified pull of every referenced blob", Usage: "crucible depot pull [--manifests DIR] --dest DIR --target cdn|bunny"},
},
Wired: true,
+28 -18
View File
@@ -67,6 +67,19 @@ type resourceEntry struct {
Size uint32
}
// extensionTypes and typeExtensions must stay exact inverses of each other;
// init() panics if they drift apart.
//
// Numbers below 0x0BB8 follow upstream neverwinter.nim
// (neverwinter/restype.nim). The 0x0BB8 and up entries are lyt/vis/mdx: real
// Aurora archive types that NWN1 ships in its own data/*.bif but upstream
// happens not to register. xoreos corroborates those three numbers
// (src/aurora/types.h).
//
// This table is NWN:EE only. Do not add NWN2 formats (mdb, gr2, wlk, xml):
// NWN:EE either gives that number to something else (2070 is xbc here and mdb
// in NWN2) or has no number for it at all, so packing one into a HAK writes a
// resource the game misreads. See the reserved list in erf_test.go.
var extensionTypes = map[string]uint16{
"res": 0x0000,
"bmp": 0x0001,
@@ -96,12 +109,13 @@ var extensionTypes = map[string]uint16{
"utt": 0x07F0,
"dds": 0x07F1,
"uts": 0x07F3,
"ltr": 0x07F4,
"gff": 0x07F5,
"fac": 0x07F6,
"gff": 0x07F7,
"ute": 0x07F8,
"utd": 0x07FA,
"utp": 0x07FC,
"dfa": 0x07FD,
"dft": 0x07FD,
"gic": 0x07FE,
"gui": 0x07FF,
"utm": 0x0803,
@@ -117,20 +131,15 @@ var extensionTypes = map[string]uint16{
"ndb": 0x0810,
"ptm": 0x0811,
"ptt": 0x0812,
"ltr": 0x0813,
"shd": 0x0815,
"mdb": 0x0816,
"mtr": 0x0818,
"jpg": 0x081C,
"lod": 0x081E,
"gif": 0x081F,
"png": 0x0820,
"jpg": 0x0821,
"lyt": 0x0BB8,
"vis": 0x0BB9,
"mdx": 0x0BC0,
"wlk": 0x0BCC,
"xml": 0x0BCD,
"gr2": 0x0FA3,
}
var typeExtensions = map[uint16]string{
@@ -162,12 +171,13 @@ var typeExtensions = map[uint16]string{
0x07F0: "utt",
0x07F1: "dds",
0x07F3: "uts",
0x07F4: "ltr",
0x07F5: "gff",
0x07F6: "fac",
0x07F7: "gff",
0x07F8: "ute",
0x07FA: "utd",
0x07FC: "utp",
0x07FD: "dfa",
0x07FD: "dft",
0x07FE: "gic",
0x07FF: "gui",
0x0803: "utm",
@@ -183,20 +193,15 @@ var typeExtensions = map[uint16]string{
0x0810: "ndb",
0x0811: "ptm",
0x0812: "ptt",
0x0813: "ltr",
0x0815: "shd",
0x0816: "mdb",
0x0818: "mtr",
0x081C: "jpg",
0x081E: "lod",
0x081F: "gif",
0x0820: "png",
0x0821: "jpg",
0x0BB8: "lyt",
0x0BB9: "vis",
0x0BC0: "mdx",
0x0BCC: "wlk",
0x0BCD: "xml",
0x0FA3: "gr2",
}
func init() {
@@ -205,8 +210,13 @@ func init() {
if !ok {
panic(fmt.Sprintf("missing canonical extension for resource type 0x%04X", resourceType))
}
if canonicalExt == ext {
continue
if canonicalExt != ext {
panic(fmt.Sprintf("resource type 0x%04X is %q in extensionTypes but %q in typeExtensions", resourceType, ext, canonicalExt))
}
}
for resourceType, ext := range typeExtensions {
if registered, ok := extensionTypes[ext]; !ok || registered != resourceType {
panic(fmt.Sprintf("extension %q is 0x%04X in typeExtensions but 0x%04X in extensionTypes (registered=%v)", ext, resourceType, registered, ok))
}
}
}
+69 -5
View File
@@ -134,14 +134,10 @@ func TestExtensionMappingsSupportModernAssetTypes(t *testing.T) {
"mtr": 0x0818,
"shd": 0x0815,
"txi": 0x07E6,
"jpg": 0x081C,
"mdb": 0x0816,
"jpg": 0x0821,
"lyt": 0x0BB8,
"vis": 0x0BB9,
"mdx": 0x0BC0,
"xml": 0x0BCD,
"wlk": 0x0BCC,
"gr2": 0x0FA3,
}
for ext, wantType := range cases {
gotType, ok := ResourceTypeForExtension(ext)
@@ -191,3 +187,71 @@ func TestExtensionMappingsSupportAllUTBlueprintTypes(t *testing.T) {
}
}
}
// TestRestypeTableMatchesUpstream pins the restype numbers Crucible shares with
// neverwinter.nim's restype.nim. The values below are upstream's; a mismatch
// means a HAK we write is mislabelled for the game.
func TestRestypeTableMatchesUpstream(t *testing.T) {
upstream := map[string]uint16{
"res": 0, "bmp": 1, "mve": 2, "tga": 3, "wav": 4, "plt": 6, "ini": 7,
"bmu": 8, "txt": 10, "mdl": 2002, "nss": 2009, "ncs": 2010, "are": 2012,
"set": 2013, "ifo": 2014, "bic": 2015, "wok": 2016, "2da": 2017,
"tlk": 2018, "txi": 2022, "git": 2023, "uti": 2025, "utc": 2027,
"dlg": 2029, "itp": 2030, "utt": 2032, "dds": 2033, "uts": 2035,
"ltr": 2036, "gff": 2037, "fac": 2038, "ute": 2040, "utd": 2042,
"utp": 2044, "dft": 2045, "gic": 2046, "gui": 2047, "utm": 2051,
"dwk": 2052, "pwk": 2053, "utg": 2055, "jrl": 2056, "utw": 2058,
"ssf": 2060, "hak": 2061, "nwm": 2062, "bik": 2063, "ndb": 2064,
"ptm": 2065, "ptt": 2066, "shd": 2069, "mtr": 2072, "lod": 2078,
"gif": 2079, "png": 2080, "jpg": 2081,
}
for ext, want := range upstream {
got, ok := ResourceTypeForExtension(ext)
if !ok {
t.Errorf("%s: not registered, upstream has %d", ext, want)
continue
}
if got != want {
t.Errorf("%s: registered as %d, upstream has %d", ext, got, want)
}
}
// Extensions upstream registers that Crucible must not reuse for anything else.
reserved := map[uint16]string{2039: "bte", 2067: "bak", 2070: "xbc", 2076: "tml"}
for number, upstreamExt := range reserved {
if ext, ok := ExtensionForResourceType(number); ok {
t.Errorf("%d: registered as %s, upstream reserves it for %s", number, ext, upstreamExt)
}
}
}
// TestNWN2FormatsAreNotRegistered keeps NWN2 file formats out of an NWN:EE
// table. mdb and gr2 have Aurora numbers that mean something else (or nothing)
// in NWN:EE; wlk and xml have no archive number in any Aurora game, so the
// values Crucible used for them were invented. Packing any of these into a HAK
// writes a resource the game misreads.
func TestNWN2FormatsAreNotRegistered(t *testing.T) {
for _, ext := range []string{"mdb", "gr2", "wlk", "xml"} {
if number, ok := ResourceTypeForExtension(ext); ok {
t.Errorf("%s is an NWN2 format but is registered as 0x%04X", ext, number)
}
}
}
// TestRestypeTablesAreMutualInverses is the check init() is meant to enforce.
func TestRestypeTablesAreMutualInverses(t *testing.T) {
for ext, number := range extensionTypes {
canonical, ok := typeExtensions[number]
if !ok {
t.Errorf("%s: number 0x%04X missing from typeExtensions", ext, number)
continue
}
if canonical != ext {
t.Errorf("%s: maps to 0x%04X, which maps back to %s", ext, number, canonical)
}
}
for number, ext := range typeExtensions {
if got, ok := extensionTypes[ext]; !ok || got != number {
t.Errorf("0x%04X: maps to %s, which maps back to 0x%04X (ok=%v)", number, ext, got, ok)
}
}
}
+10 -2
View File
@@ -593,9 +593,14 @@ func envBool(name string) bool {
func collectModuleResources(p *project.Project, moduleHakOrder []string) ([]erf.Resource, error) {
var moduleResources []erf.Resource
palettes, err := collectPaletteDescriptors(p)
if err != nil {
return nil, err
}
for _, rel := range p.Inventory.SourceFiles {
abs := filepath.Join(p.SourceDir(), filepath.FromSlash(rel))
resource, err := resourceFromJSON(abs, moduleHakOrder)
resource, err := resourceFromJSON(abs, moduleHakOrder, palettes)
if err != nil {
return nil, err
}
@@ -1030,7 +1035,7 @@ func compareResourceKeys(a, b erf.Resource) int {
return 0
}
func resourceFromJSON(path string, moduleHakOrder []string) (erf.Resource, error) {
func resourceFromJSON(path string, moduleHakOrder []string, palettes paletteProjection) (erf.Resource, error) {
name, extension, err := splitSourceName(path)
if err != nil {
return erf.Resource{}, err
@@ -1059,6 +1064,9 @@ func resourceFromJSON(path string, moduleHakOrder []string) (erf.Resource, error
if extension == ".ifo" && name == "module" && len(moduleHakOrder) > 0 {
setModuleHAKList(&document, moduleHakOrder)
}
if extension == ".itp" && isPaletteProjectionResref(name) {
projectPaletteDocument(&document, palettes[strings.ToLower(name)])
}
var buf bytes.Buffer
if err := gff.Write(&buf, document); err != nil {
+7
View File
@@ -117,6 +117,10 @@ func expectedResources(p *project.Project) (map[string]resourceExpectation, erro
if err != nil {
return nil, err
}
palettes, err := collectPaletteDescriptors(p)
if err != nil {
return nil, err
}
for _, rel := range p.Inventory.SourceFiles {
abs := filepath.Join(p.SourceDir(), filepath.FromSlash(rel))
@@ -136,6 +140,9 @@ func expectedResources(p *project.Project) (map[string]resourceExpectation, erro
if extension == ".ifo" && strings.EqualFold(name, "module") && len(moduleHakOrder) > 0 {
setModuleHAKList(&document, moduleHakOrder)
}
if extension == ".itp" && isPaletteProjectionResref(name) {
projectPaletteDocument(&document, palettes[strings.ToLower(name)])
}
canonical, err := json.Marshal(document)
if err != nil {
+7
View File
@@ -124,6 +124,13 @@ func extractArchiveResources(p *project.Project, archive erf.Archive, desired ma
skippedCount++
continue
}
// *palcus.itp are Toolset-generated palette projections; the module
// build regenerates them from source blueprints, so extraction never
// writes them back into source.
if ext == "itp" && isPaletteProjectionResref(resource.Name) {
skippedCount++
continue
}
target, data, err := extractedFile(p, resource, ext)
if err != nil {
+298
View File
@@ -0,0 +1,298 @@
package pipeline
// Custom palette projections (*palcus.itp) are generated artifacts: the Toolset
// derives them from the category frameworks plus the blueprints present in the
// module. Crucible reproduces that projection deterministically at build time so
// the module source only carries descriptor-free category skeletons and the
// blueprint files themselves. Extract never writes palcus files back.
import (
"encoding/json"
"fmt"
"os"
"path/filepath"
"sort"
"strings"
"git.westgate.pw/ShadowsOverWestgate/sow-tools/internal/gff"
"git.westgate.pw/ShadowsOverWestgate/sow-tools/internal/project"
)
const noStrref = 0xFFFFFFFF
// hiddenPaletteID suppresses a blueprint from the Custom palette (engine
// convention; see docs in sow-module's palette research notes).
const hiddenPaletteID = 255
var paletteFamilyByExtension = map[string]string{
".utc": "creaturepalcus",
".utd": "doorpalcus",
".ute": "encounterpalcus",
".uti": "itempalcus",
".utm": "storepalcus",
".utp": "placeablepalcus",
".uts": "soundpalcus",
".utt": "triggerpalcus",
".utw": "waypointpalcus",
}
type paletteDescriptor struct {
name string
strref uint32
resref string
creature bool
cr float32
faction string
}
func (d paletteDescriptor) sortKey() string {
if d.strref != noStrref || d.name == "" {
return strings.ToLower(d.resref)
}
return strings.ToLower(d.name)
}
// palette resref (e.g. "itempalcus") -> terminal category ID -> descriptors.
type paletteProjection map[string]map[uint8][]paletteDescriptor
func isPaletteProjectionResref(name string) bool {
return strings.HasSuffix(strings.ToLower(name), "palcus")
}
func collectPaletteDescriptors(p *project.Project) (paletteProjection, error) {
projection := paletteProjection{}
factions, err := loadFactionNames(p)
if err != nil {
return nil, err
}
for _, rel := range p.Inventory.SourceFiles {
abs := filepath.Join(p.SourceDir(), filepath.FromSlash(rel))
name, extension, err := splitSourceName(abs)
if err != nil {
return nil, err
}
family, ok := paletteFamilyByExtension[extension]
if !ok {
continue
}
raw, err := os.ReadFile(abs)
if err != nil {
return nil, fmt.Errorf("read %s: %w", abs, err)
}
var document gff.Document
if err := json.Unmarshal(raw, &document); err != nil {
return nil, fmt.Errorf("parse gff json %s: %w", abs, err)
}
descriptor, paletteID, ok := blueprintDescriptor(document.Root, name, extension, factions)
if !ok {
continue
}
if projection[family] == nil {
projection[family] = map[uint8][]paletteDescriptor{}
}
projection[family][paletteID] = append(projection[family][paletteID], descriptor)
}
for _, byID := range projection {
for _, descriptors := range byID {
sort.SliceStable(descriptors, func(i, j int) bool {
a, b := descriptors[i], descriptors[j]
if a.sortKey() != b.sortKey() {
return a.sortKey() < b.sortKey()
}
return a.resref < b.resref
})
}
}
return projection, nil
}
func blueprintDescriptor(root gff.Struct, fileName, extension string, factions []string) (paletteDescriptor, uint8, bool) {
descriptor := paletteDescriptor{
strref: noStrref,
resref: strings.ToLower(fileName),
creature: extension == ".utc",
}
paletteID := -1
for _, field := range root.Fields {
switch field.Label {
case "PaletteID":
if v, ok := field.Value.(gff.ByteValue); ok {
paletteID = int(v)
}
case "TemplateResRef":
if v, ok := field.Value.(gff.ResRefValue); ok && v != "" {
descriptor.resref = strings.ToLower(string(v))
}
case "LocalizedName", "LocName", "FirstName":
if v, ok := field.Value.(gff.LocString); ok {
name, strref := locStringLabel(v)
if field.Label == "FirstName" {
descriptor.name = strings.TrimSpace(descriptor.name + " " + name)
if descriptor.strref == noStrref {
descriptor.strref = strref
}
} else {
descriptor.name = name
descriptor.strref = strref
}
}
case "LastName":
if v, ok := field.Value.(gff.LocString); ok {
name, _ := locStringLabel(v)
descriptor.name = strings.TrimSpace(descriptor.name + " " + name)
}
case "ChallengeRating":
if v, ok := field.Value.(gff.FloatValue); ok {
descriptor.cr = float32(v)
}
case "FactionID":
if v, ok := field.Value.(gff.WordValue); ok && int(v) < len(factions) {
descriptor.faction = factions[v]
}
}
}
if paletteID < 0 || paletteID == hiddenPaletteID {
return paletteDescriptor{}, 0, false
}
return descriptor, uint8(paletteID), true
}
func locStringLabel(value gff.LocString) (string, uint32) {
if value.StringRef != noStrref {
return "", value.StringRef
}
for _, entry := range value.Entries {
if entry.Value != "" {
return entry.Value, noStrref
}
}
return "", noStrref
}
func loadFactionNames(p *project.Project) ([]string, error) {
for _, rel := range p.Inventory.SourceFiles {
if !strings.HasSuffix(strings.ToLower(rel), "repute.fac.json") {
continue
}
abs := filepath.Join(p.SourceDir(), filepath.FromSlash(rel))
raw, err := os.ReadFile(abs)
if err != nil {
return nil, fmt.Errorf("read %s: %w", abs, err)
}
var document gff.Document
if err := json.Unmarshal(raw, &document); err != nil {
return nil, fmt.Errorf("parse gff json %s: %w", abs, err)
}
var names []string
for _, field := range document.Root.Fields {
if field.Label != "FactionList" {
continue
}
list, ok := field.Value.(gff.ListValue)
if !ok {
continue
}
for _, faction := range list {
name := ""
for _, f := range faction.Fields {
if f.Label == "FactionName" {
if v, ok := f.Value.(gff.StringValue); ok {
name = string(v)
}
}
}
names = append(names, name)
}
}
return names, nil
}
return nil, nil
}
// projectPaletteDocument strips every blueprint descriptor from the palette
// tree and re-inserts the descriptors derived from module source. Category
// structure (branches, terminal IDs, labels) passes through untouched.
func projectPaletteDocument(document *gff.Document, byID map[uint8][]paletteDescriptor) {
for i, field := range document.Root.Fields {
if field.Label != "MAIN" {
continue
}
if list, ok := field.Value.(gff.ListValue); ok {
document.Root.Fields[i] = gff.NewField("MAIN", projectPaletteNodes(list, byID))
}
}
}
func projectPaletteNodes(nodes gff.ListValue, byID map[uint8][]paletteDescriptor) gff.ListValue {
out := make(gff.ListValue, 0, len(nodes))
for _, node := range nodes {
if structHasField(node, "RESREF") {
continue // blueprint descriptor — regenerated below
}
terminalID := -1
fields := make([]gff.Field, 0, len(node.Fields))
for _, field := range node.Fields {
if field.Label == "LIST" {
continue // rebuilt for terminals, recursed for branches
}
if field.Label == "ID" {
if v, ok := field.Value.(gff.ByteValue); ok {
terminalID = int(v)
}
}
fields = append(fields, field)
}
switch {
case terminalID >= 0:
if descriptors := byID[uint8(terminalID)]; len(descriptors) > 0 {
fields = append(fields, gff.NewField("LIST", descriptorList(descriptors)))
}
default:
for _, field := range node.Fields {
if field.Label == "LIST" {
if list, ok := field.Value.(gff.ListValue); ok {
fields = append(fields, gff.NewField("LIST", projectPaletteNodes(list, byID)))
}
}
}
}
out = append(out, gff.Struct{Type: node.Type, Fields: fields})
}
return out
}
func descriptorList(descriptors []paletteDescriptor) gff.ListValue {
list := make(gff.ListValue, 0, len(descriptors))
for _, d := range descriptors {
fields := make([]gff.Field, 0, 4)
if d.strref != noStrref {
fields = append(fields, gff.NewField("STRREF", gff.DWordValue(d.strref)))
} else {
fields = append(fields, gff.NewField("NAME", gff.StringValue(d.name)))
}
fields = append(fields, gff.NewField("RESREF", gff.ResRefValue(d.resref)))
if d.creature {
fields = append(fields, gff.NewField("CR", gff.FloatValue(d.cr)))
fields = append(fields, gff.NewField("FACTION", gff.StringValue(d.faction)))
}
list = append(list, gff.Struct{Fields: fields})
}
return list
}
func structHasField(s gff.Struct, label string) bool {
for _, field := range s.Fields {
if field.Label == label {
return true
}
}
return false
}
+191
View File
@@ -0,0 +1,191 @@
package pipeline
import (
"bytes"
"encoding/json"
"os"
"path/filepath"
"testing"
"git.westgate.pw/ShadowsOverWestgate/sow-tools/internal/erf"
"git.westgate.pw/ShadowsOverWestgate/sow-tools/internal/gff"
"git.westgate.pw/ShadowsOverWestgate/sow-tools/internal/project"
)
const paletteTestSkeleton = `{
"file_type": "ITP ",
"file_version": "V3.2",
"root": {
"struct_type": 4294967295,
"fields": [
{
"label": "MAIN",
"type": "List",
"value": [
{
"struct_type": 0,
"fields": [
{"label": "STRREF", "type": "DWord", "value": 500},
{
"label": "LIST",
"type": "List",
"value": [
{
"struct_type": 0,
"fields": [
{"label": "STRREF", "type": "DWord", "value": 6699},
{"label": "ID", "type": "Byte", "value": 23},
{
"label": "LIST",
"type": "List",
"value": [
{
"struct_type": 0,
"fields": [
{"label": "NAME", "type": "CExoString", "value": "Stale Junk"},
{"label": "RESREF", "type": "ResRef", "value": "stalejunk"}
]
}
]
}
]
},
{
"struct_type": 0,
"fields": [
{"label": "STRREF", "type": "DWord", "value": 6753},
{"label": "ID", "type": "Byte", "value": 24}
]
}
]
}
]
}
]
}
]
}
}
`
func paletteTestItem(resref, name string, paletteID int) string {
return `{
"file_type": "UTI ",
"file_version": "V3.2",
"root": {
"struct_type": 4294967295,
"fields": [
{"label": "TemplateResRef", "type": "ResRef", "value": "` + resref + `"},
{
"label": "LocalizedName",
"type": "CExoLocString",
"value": {"string_ref": 4294967295, "entries": [{"id": 0, "value": "` + name + `"}]}
},
{"label": "PaletteID", "type": "Byte", "value": ` + itoa(paletteID) + `}
]
}
}
`
}
func itoa(v int) string {
data, _ := json.Marshal(v)
return string(data)
}
func TestBuildProjectsPaletteDescriptorsAndExtractSkipsThem(t *testing.T) {
root := t.TempDir()
mustMkdir(t, filepath.Join(root, "src", "module"))
mustMkdir(t, filepath.Join(root, "src", "palettes"))
mustMkdir(t, filepath.Join(root, "src", "blueprints", "items"))
mustMkdir(t, filepath.Join(root, "build"))
mustWriteFile(t, filepath.Join(root, "nwn-tool.json"), `{
"module": {"name": "Test Module", "resref": "testmod"},
"paths": {"source": "src", "build": "build"}
}
`)
mustWriteFile(t, filepath.Join(root, "src", "module", "module.ifo.json"), `{
"file_type": "IFO ",
"file_version": "V3.2",
"root": {"struct_type": 4294967295, "fields": [{"label": "Mod_Name", "type": "CExoString", "value": "Test Module"}]}
}
`)
mustWriteFile(t, filepath.Join(root, "src", "palettes", "itempalcus.itp.json"), paletteTestSkeleton)
mustWriteFile(t, filepath.Join(root, "src", "blueprints", "items", "i_b.uti.json"), paletteTestItem("i_b", "Bravo Item", 23))
mustWriteFile(t, filepath.Join(root, "src", "blueprints", "items", "i_a.uti.json"), paletteTestItem("i_a", "Alpha Item", 23))
mustWriteFile(t, filepath.Join(root, "src", "blueprints", "items", "i_hidden.uti.json"), paletteTestItem("i_hidden", "Hidden Item", 255))
p, err := project.Load(root)
if err != nil {
t.Fatalf("load project: %v", err)
}
if err := p.Scan(); err != nil {
t.Fatalf("scan: %v", err)
}
if _, err := BuildModule(p); err != nil {
t.Fatalf("build: %v", err)
}
archiveFile, err := os.Open(p.ModuleArchivePath())
if err != nil {
t.Fatalf("open module archive: %v", err)
}
defer archiveFile.Close()
archive, err := erf.Read(archiveFile)
if err != nil {
t.Fatalf("read module archive: %v", err)
}
var palette *gff.Document
for _, resource := range archive.Resources {
if resource.Name == "itempalcus" {
document, err := gff.Read(bytes.NewReader(resource.Data))
if err != nil {
t.Fatalf("decode itempalcus: %v", err)
}
palette = &document
}
}
if palette == nil {
t.Fatal("itempalcus missing from built module")
}
canonical, err := json.Marshal(palette)
if err != nil {
t.Fatalf("marshal palette: %v", err)
}
text := string(canonical)
if bytes.Contains(canonical, []byte("stalejunk")) {
t.Fatalf("stale descriptor survived projection: %s", text)
}
for _, resref := range []string{"i_a", "i_b"} {
if !bytes.Contains(canonical, []byte(resref)) {
t.Fatalf("descriptor %s missing from projection: %s", resref, text)
}
}
if bytes.Contains(canonical, []byte("i_hidden")) {
t.Fatalf("PaletteID 255 blueprint leaked into projection: %s", text)
}
if a, b := bytes.Index(canonical, []byte("i_a")), bytes.Index(canonical, []byte("i_b")); a > b {
t.Fatalf("descriptors not name-sorted: %s", text)
}
if _, err := Compare(p); err != nil {
t.Fatalf("compare after build: %v", err)
}
// Extraction must never write palcus files back into source.
if err := os.Remove(filepath.Join(root, "src", "palettes", "itempalcus.itp.json")); err != nil {
t.Fatalf("remove skeleton: %v", err)
}
if err := p.Scan(); err != nil {
t.Fatalf("rescan: %v", err)
}
if _, err := Extract(p); err != nil {
t.Fatalf("extract: %v", err)
}
if _, err := os.Stat(filepath.Join(root, "src", "palettes", "itempalcus.itp.json")); !os.IsNotExist(err) {
t.Fatalf("extract wrote palcus file back (stat err: %v)", err)
}
}
+1 -1
View File
@@ -31,7 +31,7 @@ var SourceExtensions = []string{
}
var AssetExtensions = []string{
".2da", ".bik", ".bmp", ".bmu", ".dds", ".dwk", ".gr2", ".itp", ".jpg", ".lod", ".lyt", ".mdb", ".mdl", ".mdx", ".mtr", ".plt", ".png", ".pwk", ".set", ".shd", ".tga", ".txi", ".uti", ".vis", ".wav", ".wlk", ".wok", ".xml",
".2da", ".bik", ".bmp", ".bmu", ".dds", ".dwk", ".itp", ".jpg", ".lod", ".lyt", ".mdl", ".mdx", ".mtr", ".plt", ".png", ".pwk", ".set", ".shd", ".tga", ".txi", ".uti", ".vis", ".wav", ".wok",
}
var BuiltinScriptPrefixes = []string{
-96
View File
@@ -1,96 +0,0 @@
package topdata
import (
"encoding/json"
"os"
"path/filepath"
)
func importLegacyAppearance(referenceBuilderDir, dataDir string, legacyTLK *legacyTLKData) (int, error) {
_ = legacyTLK
legacyDir := filepath.Join(referenceBuilderDir, "data", "appearance")
if _, err := os.Stat(legacyDir); err != nil {
if os.IsNotExist(err) {
return 0, nil
}
return 0, err
}
targetDir := filepath.Join(dataDir, "appearance")
targetBasePath := filepath.Join(targetDir, "base.json")
targetLockPath := filepath.Join(targetDir, "lock.json")
targetModulesDir := filepath.Join(targetDir, "modules")
moduleNames := []string{
"cotblreaver.json",
"crawlingclaw.json",
"halfogre.json",
"zombieknight.json",
}
targetModulePaths := make([]string, 0, len(moduleNames))
for _, name := range moduleNames {
targetModulePaths = append(targetModulePaths, filepath.Join(targetModulesDir, name))
}
if fileExists(targetBasePath) && fileExists(targetLockPath) {
allModulesPresent := true
for _, path := range targetModulePaths {
if !fileExists(path) {
allModulesPresent = false
break
}
}
if allModulesPresent {
return 0, nil
}
}
baseObj, err := loadJSONObject(filepath.Join(legacyDir, "base.json"))
if err != nil {
return 0, err
}
baseObj["output"] = "appearance.2da"
lockObj, err := loadJSONObject(filepath.Join(legacyDir, "lock.json"))
if err != nil {
return 0, err
}
moduleObjs := make([]map[string]any, 0, len(moduleNames))
for _, name := range moduleNames {
obj, err := loadJSONObject(filepath.Join(legacyDir, "modules", name))
if err != nil {
return 0, err
}
moduleObjs = append(moduleObjs, obj)
}
if err := os.MkdirAll(targetModulesDir, 0o755); err != nil {
return 0, err
}
writes := []struct {
path string
obj map[string]any
}{
{path: targetBasePath, obj: baseObj},
{path: targetLockPath, obj: lockObj},
}
for i, path := range targetModulePaths {
writes = append(writes, struct {
path string
obj map[string]any
}{path: path, obj: moduleObjs[i]})
}
for _, write := range writes {
raw, err := json.MarshalIndent(write.obj, "", " ")
if err != nil {
return 0, err
}
raw = append(raw, '\n')
if err := os.WriteFile(write.path, raw, 0o644); err != nil {
return 0, err
}
}
return len(writes), nil
}
-160
View File
@@ -1,160 +0,0 @@
package topdata
import (
"encoding/json"
"os"
"path/filepath"
"sort"
)
func importLegacyArmor(referenceBuilderDir, dataDir string, legacyTLK *legacyTLKData) (int, error) {
_ = legacyTLK
legacyDir := filepath.Join(referenceBuilderDir, "data", "armor")
if _, err := os.Stat(legacyDir); err != nil {
if os.IsNotExist(err) {
return 0, nil
}
return 0, err
}
targetDir := filepath.Join(dataDir, "armor")
targetPath := filepath.Join(targetDir, "armor.json")
if _, err := os.Stat(targetPath); err == nil {
obj, err := loadJSONObject(targetPath)
if err == nil && countLegacyTLKRefsInValue(obj) == 0 {
return 0, nil
}
}
baseObj, err := loadJSONObject(filepath.Join(legacyDir, "base.json"))
if err != nil {
return 0, err
}
rows, err := mergeArmorRows(baseObj, filepath.Join(legacyDir, "modules"))
if err != nil {
return 0, err
}
if err := os.MkdirAll(targetDir, 0o755); err != nil {
return 0, err
}
if err := removePathIfExists(filepath.Join(targetDir, "base.json")); err != nil {
return 0, err
}
if err := removePathIfExists(filepath.Join(targetDir, "lock.json")); err != nil {
return 0, err
}
if err := removePathIfExists(filepath.Join(targetDir, "modules")); err != nil {
return 0, err
}
out := map[string]any{
"output": "armor.2da",
"columns": baseObj["columns"],
"rows": rows,
}
raw, err := json.MarshalIndent(out, "", " ")
if err != nil {
return 0, err
}
raw = append(raw, '\n')
if err := os.WriteFile(targetPath, raw, 0o644); err != nil {
return 0, err
}
return 1, nil
}
func mergeArmorRows(baseObj map[string]any, modulesDir string) ([]any, error) {
rawRows, ok := baseObj["rows"].([]any)
if !ok {
return nil, nil
}
rows := make([]map[string]any, 0, len(rawRows))
byID := map[int]map[string]any{}
for _, raw := range rawRows {
row, ok := deepCopyValue(raw).(map[string]any)
if !ok {
continue
}
delete(row, "key")
rawID, ok := row["id"]
if !ok {
continue
}
id, err := asInt(rawID)
if err != nil {
return nil, err
}
row["id"] = id
rows = append(rows, row)
byID[id] = row
}
modulePaths, err := collectModulePaths(modulesDir)
if err != nil {
return nil, err
}
for _, path := range modulePaths {
obj, err := loadJSONObject(path)
if err != nil {
return nil, err
}
if entries, ok := obj["entries"].(map[string]any); ok {
for _, raw := range entries {
row, ok := deepCopyValue(raw).(map[string]any)
if !ok {
continue
}
delete(row, "key")
rawID, ok := row["id"]
if !ok {
continue
}
id, err := asInt(rawID)
if err != nil {
return nil, err
}
row["id"] = id
rows = append(rows, row)
byID[id] = row
}
}
if overrides, ok := obj["overrides"].([]any); ok {
for _, raw := range overrides {
override, ok := raw.(map[string]any)
if !ok {
continue
}
rawID, ok := override["id"]
if !ok {
continue
}
id, err := asInt(rawID)
if err != nil {
return nil, err
}
row := byID[id]
if row == nil {
continue
}
for key, value := range override {
if key == "id" || key == "key" || key == "_tlk" || isMetadataField(key) {
continue
}
row[key] = deepCopyValue(value)
}
}
}
}
sort.Slice(rows, func(i, j int) bool {
return rows[i]["id"].(int) < rows[j]["id"].(int)
})
out := make([]any, 0, len(rows))
for _, row := range rows {
out = append(out, row)
}
return out, nil
}
-41
View File
@@ -1,41 +0,0 @@
package topdata
import (
"encoding/json"
"os"
"path/filepath"
)
func importLegacyBaseDialog(referenceBuilderDir, sourceDir string) (int, error) {
legacyPath := filepath.Join(referenceBuilderDir, "tlk", "base.json")
if _, err := os.Stat(legacyPath); err != nil {
if os.IsNotExist(err) {
return 0, nil
}
return 0, err
}
targetPath := filepath.Join(sourceDir, "base_dialog.json")
if fileExists(targetPath) {
current, err := loadJSONObject(targetPath)
if err == nil {
if entries, ok := current["entries"].(map[string]any); ok && len(entries) > 0 {
return 0, nil
}
}
}
obj, err := loadJSONObject(legacyPath)
if err != nil {
return 0, err
}
raw, err := json.MarshalIndent(obj, "", " ")
if err != nil {
return 0, err
}
raw = append(raw, '\n')
if err := os.WriteFile(targetPath, raw, 0o644); err != nil {
return 0, err
}
return 1, nil
}
-105
View File
@@ -1,105 +0,0 @@
package topdata
import (
"encoding/json"
"os"
"path/filepath"
)
func importLegacyBaseitems(referenceBuilderDir, dataDir string, legacyTLK *legacyTLKData) (int, error) {
_ = legacyTLK
legacyDir := filepath.Join(referenceBuilderDir, "data", "baseitems")
if _, err := os.Stat(legacyDir); err != nil {
if os.IsNotExist(err) {
return 0, nil
}
return 0, err
}
targetDir := filepath.Join(dataDir, "baseitems")
targetBasePath := filepath.Join(targetDir, "base.json")
targetLockPath := filepath.Join(targetDir, "lock.json")
targetModulesDir := filepath.Join(targetDir, "modules")
moduleNames := []string{
"blunderbuss.json",
"coin.json",
"estoc.json",
"heavymace.json",
"maulandfalchion.json",
"ovr_baseitems.json",
"ovr_cloak255.json",
"ovr_helmet255.json",
"shortspear.json",
}
targetModulePaths := make([]string, 0, len(moduleNames))
for _, name := range moduleNames {
targetModulePaths = append(targetModulePaths, filepath.Join(targetModulesDir, name))
}
if fileExists(targetBasePath) && fileExists(targetLockPath) {
allModulesPresent := true
for _, path := range targetModulePaths {
if !fileExists(path) {
allModulesPresent = false
break
}
}
if allModulesPresent {
return 0, nil
}
}
baseObj, err := loadJSONObject(filepath.Join(legacyDir, "base.json"))
if err != nil {
return 0, err
}
baseObj["output"] = "baseitems.2da"
canonicalizeWikiMetadataDocument(baseObj)
lockObj, err := loadJSONObject(filepath.Join(legacyDir, "lock.json"))
if err != nil {
return 0, err
}
moduleObjs := make([]map[string]any, 0, len(moduleNames))
for _, name := range moduleNames {
obj, err := loadJSONObject(filepath.Join(legacyDir, "modules", name))
if err != nil {
return 0, err
}
canonicalizeWikiMetadataDocument(obj)
moduleObjs = append(moduleObjs, obj)
}
if err := os.MkdirAll(targetModulesDir, 0o755); err != nil {
return 0, err
}
writes := []struct {
path string
obj map[string]any
}{
{path: targetBasePath, obj: baseObj},
{path: targetLockPath, obj: lockObj},
}
for index, path := range targetModulePaths {
writes = append(writes, struct {
path string
obj map[string]any
}{
path: path,
obj: moduleObjs[index],
})
}
for _, write := range writes {
raw, err := json.MarshalIndent(write.obj, "", " ")
if err != nil {
return 0, err
}
raw = append(raw, '\n')
if err := os.WriteFile(write.path, raw, 0o644); err != nil {
return 0, err
}
}
return len(writes), nil
}
-226
View File
@@ -1,226 +0,0 @@
package topdata
import (
"encoding/json"
"os"
"path/filepath"
"slices"
"strings"
)
var legacyClassesPlainFamilies = []string{"feats", "skills", "savthr", "bfeat", "pres"}
func importLegacyClasses(referenceBuilderDir, dataDir string, legacyTLK *legacyTLKData) (int, error) {
legacyRoot := filepath.Join(referenceBuilderDir, "data", "classes")
if _, err := os.Stat(legacyRoot); err != nil {
if os.IsNotExist(err) {
return 0, nil
}
return 0, err
}
targetRoot := filepath.Join(dataDir, "classes")
if canonicalClassesPresent(targetRoot) {
return 0, nil
}
coreCollected, plainCollected, err := collectLegacyClassesDatasets(legacyRoot)
if err != nil {
return 0, err
}
tableKeyByOutput := map[string]string{}
for _, dataset := range plainCollected {
if dataset.TableKey == "" {
continue
}
registerLegacyClassesTableKey(tableKeyByOutput, dataset.TableKey, dataset.Dataset.OutputName)
}
coreRows := make([]map[string]any, 0, len(coreCollected.Rows))
for _, rawRow := range coreCollected.Rows {
row, ok := deepCopyValue(rawRow).(map[string]any)
if !ok {
continue
}
if legacyTLK != nil {
if _, _, err := inlineLegacyTLKValue(row, legacyTLK); err != nil {
return 0, err
}
}
rewriteLegacyClassesCoreRow(row, tableKeyByOutput)
coreRows = append(coreRows, row)
}
if err := removePathIfExists(targetRoot); err != nil {
return 0, err
}
if err := os.MkdirAll(filepath.Join(targetRoot, "core"), 0o755); err != nil {
return 0, err
}
writes := []struct {
path string
obj map[string]any
}{
{
path: filepath.Join(targetRoot, "core", "base.json"),
obj: map[string]any{
"output": coreCollected.Dataset.OutputName,
"columns": stringSliceToAny(coreCollected.Columns),
"rows": rowsToAny(coreRows),
},
},
{
path: filepath.Join(targetRoot, "core", "lock.json"),
obj: anyMapInt(coreCollected.LockData),
},
}
for _, collected := range plainCollected {
obj := map[string]any{
"output": collected.Dataset.OutputName,
"columns": stringSliceToAny(collected.Columns),
"rows": rowsToAny(collected.Rows),
}
if strings.TrimSpace(collected.TableKey) != "" {
obj["key"] = collected.TableKey
}
writes = append(writes, struct {
path string
obj map[string]any
}{
path: filepath.Join(dataDir, collected.Dataset.Name+".json"),
obj: obj,
})
}
for _, write := range writes {
if err := os.MkdirAll(filepath.Dir(write.path), 0o755); err != nil {
return 0, err
}
raw, err := json.MarshalIndent(write.obj, "", " ")
if err != nil {
return 0, err
}
raw = append(raw, '\n')
if err := os.WriteFile(write.path, raw, 0o644); err != nil {
return 0, err
}
}
return len(writes), nil
}
func canonicalClassesPresent(targetRoot string) bool {
if !fileExists(filepath.Join(targetRoot, "core", "base.json")) || !fileExists(filepath.Join(targetRoot, "core", "lock.json")) {
return false
}
for _, family := range legacyClassesPlainFamilies {
ok, err := hasJSONFiles(filepath.Join(targetRoot, family))
if err != nil || !ok {
return false
}
}
return true
}
func collectLegacyClassesDatasets(legacyRoot string) (nativeCollectedDataset, []nativeCollectedDataset, error) {
coreCollected, err := collectBaseDataset(nativeDataset{
Name: "classes/core",
BasePath: filepath.Join(legacyRoot, "core", "base.json"),
LockPath: filepath.Join(legacyRoot, "core", "lock.json"),
ModulesDir: filepath.Join(legacyRoot, "core", "modules"),
OutputName: "classes.2da",
Spec: specForDataset("classes"),
})
if err != nil {
return nativeCollectedDataset{}, nil, err
}
coreCollected.Dataset.OutputName = "classes.2da"
plainCollected := make([]nativeCollectedDataset, 0)
for _, family := range legacyClassesPlainFamilies {
familyDir := filepath.Join(legacyRoot, family)
entries, err := os.ReadDir(familyDir)
if err != nil {
if os.IsNotExist(err) {
continue
}
return nativeCollectedDataset{}, nil, err
}
for _, entry := range entries {
if entry.IsDir() || filepath.Ext(entry.Name()) != ".json" || strings.HasPrefix(entry.Name(), ".") || entry.Name() == "lock.json" {
continue
}
filePath := filepath.Join(familyDir, entry.Name())
tableData, err := loadJSONObject(filePath)
if err != nil {
return nativeCollectedDataset{}, nil, err
}
outputName, _ := tableData["output"].(string)
if strings.TrimSpace(outputName) == "" {
outputName = strings.TrimSuffix(entry.Name(), filepath.Ext(entry.Name())) + ".2da"
}
collected, err := collectPlainDataset(nativeDataset{
Name: filepath.ToSlash(filepath.Join("classes", family, strings.TrimSuffix(entry.Name(), filepath.Ext(entry.Name())))),
BasePath: filePath,
LockPath: filepath.Join(familyDir, "lock.json"),
OutputName: outputName,
Spec: specForDataset(filepath.ToSlash(filepath.Join("classes", family, strings.TrimSuffix(entry.Name(), filepath.Ext(entry.Name()))))),
})
if err != nil {
return nativeCollectedDataset{}, nil, err
}
plainCollected = append(plainCollected, collected)
}
}
slices.SortFunc(plainCollected, func(a, b nativeCollectedDataset) int {
return strings.Compare(a.Dataset.Name, b.Dataset.Name)
})
return coreCollected, plainCollected, nil
}
func registerLegacyClassesTableKey(tableKeyByOutput map[string]string, tableKey, outputName string) {
trimmedOutput := strings.TrimSpace(outputName)
if trimmedOutput != "" {
tableKeyByOutput[trimmedOutput] = tableKey
}
trimmedStem := strings.TrimSpace(outputStem(outputName))
if trimmedStem != "" {
tableKeyByOutput[trimmedStem] = tableKey
}
}
func rewriteLegacyClassesCoreRow(row map[string]any, tableKeyByOutput map[string]string) {
for _, field := range []string{"FeatsTable", "SavingThrowTable", "SkillsTable", "BonusFeatsTable", "PreReqTable"} {
tableKey := legacyClassesTableKeyForValue(row[field], tableKeyByOutput)
if tableKey == "" {
continue
}
row[field] = map[string]any{"table": tableKey}
}
}
func legacyClassesTableKeyForValue(value any, tableKeyByOutput map[string]string) string {
switch typed := value.(type) {
case map[string]any:
tableKey, _ := typed["table"].(string)
return strings.TrimSpace(tableKey)
case string:
trimmed := strings.TrimSpace(typed)
if trimmed == "" || trimmed == nullValue {
return ""
}
if trimmed != strings.ToLower(trimmed) {
return ""
}
if tableKey, ok := tableKeyByOutput[trimmed]; ok {
return tableKey
}
return ""
default:
return ""
}
}
-6
View File
@@ -1,6 +0,0 @@
package topdata
func importLegacyCloakmodel(referenceBuilderDir, dataDir string, legacyTLK *legacyTLKData) (int, error) {
_ = legacyTLK
return importLegacyDatasetMirror(referenceBuilderDir, dataDir, "cloakmodel", "cloakmodel.2da", nil)
}
-145
View File
@@ -1,145 +0,0 @@
package topdata
import (
"encoding/json"
"os"
"path/filepath"
)
func importLegacyCreaturespeed(referenceBuilderDir, dataDir string, legacyTLK *legacyTLKData) (int, error) {
_ = legacyTLK
legacyDir := filepath.Join(referenceBuilderDir, "data", "creaturespeed")
if _, err := os.Stat(legacyDir); err != nil {
if os.IsNotExist(err) {
return 0, nil
}
return 0, err
}
targetDir := filepath.Join(dataDir, "creaturespeed")
targetPath := filepath.Join(targetDir, "creaturespeed.json")
if _, err := os.Stat(targetPath); err == nil {
obj, err := loadJSONObject(targetPath)
if err == nil && countLegacyTLKRefsInValue(obj) == 0 {
return 0, nil
}
}
baseObj, err := loadJSONObject(filepath.Join(legacyDir, "base.json"))
if err != nil {
return 0, err
}
rows, err := mergeCreaturespeedRows(baseObj, filepath.Join(legacyDir, "modules"))
if err != nil {
return 0, err
}
if err := os.MkdirAll(targetDir, 0o755); err != nil {
return 0, err
}
if err := removePathIfExists(filepath.Join(targetDir, "base.json")); err != nil {
return 0, err
}
if err := removePathIfExists(filepath.Join(targetDir, "lock.json")); err != nil {
return 0, err
}
if err := removePathIfExists(filepath.Join(targetDir, "modules")); err != nil {
return 0, err
}
out := map[string]any{
"output": "creaturespeed.2da",
"columns": baseObj["columns"],
"rows": rows,
}
raw, err := json.MarshalIndent(out, "", " ")
if err != nil {
return 0, err
}
raw = append(raw, '\n')
if err := os.WriteFile(targetPath, raw, 0o644); err != nil {
return 0, err
}
return 1, nil
}
func mergeCreaturespeedRows(baseObj map[string]any, modulesDir string) ([]any, error) {
rawRows, ok := baseObj["rows"].([]any)
if !ok {
return nil, nil
}
rows := make([]map[string]any, 0, len(rawRows))
byID := map[int]map[string]any{}
for _, raw := range rawRows {
row, ok := deepCopyValue(raw).(map[string]any)
if !ok {
continue
}
delete(row, "key")
rows = append(rows, row)
if rawID, ok := row["id"]; ok {
id, err := asInt(rawID)
if err != nil {
return nil, err
}
byID[id] = row
}
}
modulePaths, err := collectModulePaths(modulesDir)
if err != nil {
return nil, err
}
for _, path := range modulePaths {
obj, err := loadJSONObject(path)
if err != nil {
return nil, err
}
overrideList, ok := obj["overrides"].([]any)
if !ok {
continue
}
for _, raw := range overrideList {
override, ok := raw.(map[string]any)
if !ok {
continue
}
rawID, ok := override["id"]
if !ok {
continue
}
id, err := asInt(rawID)
if err != nil {
return nil, err
}
row := byID[id]
if row == nil {
continue
}
for key, value := range override {
if key == "id" || key == "key" || key == "_tlk" {
continue
}
row[key] = deepCopyValue(value)
}
}
}
out := make([]any, 0, len(rows))
for _, row := range rows {
out = append(out, row)
}
return out, nil
}
func removePathIfExists(path string) error {
if _, err := os.Stat(path); err != nil {
if os.IsNotExist(err) {
return nil
}
return err
}
return os.RemoveAll(path)
}
-288
View File
@@ -1,288 +0,0 @@
package topdata
import (
"encoding/json"
"fmt"
"os"
"path/filepath"
"sort"
"strings"
)
func importLegacyDamagetypes(referenceBuilderDir, dataDir string, legacyTLK *legacyTLKData) (int, error) {
_ = legacyTLK
legacyRoot := filepath.Join(referenceBuilderDir, "data", "damagetypes")
if _, err := os.Stat(legacyRoot); err != nil {
if os.IsNotExist(err) {
return 0, nil
}
return 0, err
}
targetDir := filepath.Join(dataDir, "damagetypes", "registry")
targetPath := filepath.Join(targetDir, "types.json")
if _, err := os.Stat(targetPath); err == nil {
obj, err := loadJSONObject(targetPath)
if err == nil && countLegacyTLKRefsInValue(obj) == 0 {
return 0, nil
}
}
coreBase, err := loadJSONObject(filepath.Join(legacyRoot, "core", "base.json"))
if err != nil {
return 0, err
}
groupBase, err := loadJSONObject(filepath.Join(legacyRoot, "groups", "base.json"))
if err != nil {
return 0, err
}
hitvisualBase, err := loadJSONObject(filepath.Join(legacyRoot, "hitvisual", "base.json"))
if err != nil {
return 0, err
}
coreModule, err := loadJSONObject(filepath.Join(legacyRoot, "core", "modules", "add_eos.json"))
if err != nil {
return 0, err
}
coreLock, err := loadLockfile(filepath.Join(legacyRoot, "core", "lock.json"))
if err != nil {
return 0, err
}
groupModule, err := loadJSONObject(filepath.Join(legacyRoot, "groups", "modules", "add_eos.json"))
if err != nil {
return 0, err
}
groupLock, err := loadLockfile(filepath.Join(legacyRoot, "groups", "lock.json"))
if err != nil {
return 0, err
}
hitvisualModule, err := loadJSONObject(filepath.Join(legacyRoot, "hitvisual", "modules", "add_eos.json"))
if err != nil {
return 0, err
}
hitvisualLock, err := loadLockfile(filepath.Join(legacyRoot, "hitvisual", "lock.json"))
if err != nil {
return 0, err
}
rows, lockData, err := mergeLegacyDamagetypes(coreBase, groupBase, hitvisualBase, coreModule, coreLock, groupModule, groupLock, hitvisualModule, hitvisualLock)
if err != nil {
return 0, err
}
if err := os.MkdirAll(targetDir, 0o755); err != nil {
return 0, err
}
if err := removePathIfExists(filepath.Join(dataDir, "damagetypes", "core")); err != nil {
return 0, err
}
if err := removePathIfExists(filepath.Join(dataDir, "damagetypes", "groups")); err != nil {
return 0, err
}
if err := removePathIfExists(filepath.Join(dataDir, "damagetypes", "hitvisual")); err != nil {
return 0, err
}
typesOut := map[string]any{"rows": rows}
raw, err := json.MarshalIndent(typesOut, "", " ")
if err != nil {
return 0, err
}
raw = append(raw, '\n')
if err := os.WriteFile(targetPath, raw, 0o644); err != nil {
return 0, err
}
if err := saveLockfile(filepath.Join(targetDir, damagetypesRegistryLock), lockData); err != nil {
return 0, err
}
return 2, nil
}
func mergeLegacyDamagetypes(coreBase, groupBase, hitvisualBase, coreModule map[string]any, coreLock map[string]int, groupModule map[string]any, groupLock map[string]int, hitvisualModule map[string]any, hitvisualLock map[string]int) ([]map[string]any, map[string]int, error) {
coreRows, err := coerceRowMapSlice(coreBase["rows"])
if err != nil {
return nil, nil, err
}
groupRows, err := coerceRowMapSlice(groupBase["rows"])
if err != nil {
return nil, nil, err
}
hitRows, err := coerceRowMapSlice(hitvisualBase["rows"])
if err != nil {
return nil, nil, err
}
groupByID := map[int]map[string]any{}
for _, row := range groupRows {
id, err := asInt(row["id"])
if err != nil {
return nil, nil, err
}
groupByID[id] = row
}
hitByID := map[int]map[string]any{}
for _, row := range hitRows {
id, err := asInt(row["id"])
if err != nil {
return nil, nil, err
}
hitByID[id] = row
}
outRows := make([]map[string]any, 0, len(coreRows))
lockData := map[string]int{}
for _, core := range coreRows {
id, err := asInt(core["id"])
if err != nil {
return nil, nil, err
}
groupID, err := asInt(core["DamageTypeGroup"])
if err != nil {
return nil, nil, err
}
group := groupByID[groupID]
if group == nil {
return nil, nil, fmt.Errorf("core id %d references missing group %d", id, groupID)
}
hit := hitByID[id]
if hit == nil {
return nil, nil, fmt.Errorf("core id %d is missing hitvisual row", id)
}
key := "damagetype:" + normalizeDamagetypeKey(core["Label"])
lockData[key] = id
outRows = append(outRows, map[string]any{
"key": key,
"id": id,
"label": deepCopyValue(core["Label"]),
"charsheet_strref": deepCopyValue(core["CharsheetStrref"]),
"damage_type_group": strconvString(groupID),
"damage_ranged_projectile": deepCopyValue(core["DamageRangedProjectile"]),
"group_label": deepCopyValue(group["Label"]),
"feedback_strref": deepCopyValue(group["FeedbackStrref"]),
"color_r": deepCopyValue(group["ColorR"]),
"color_g": deepCopyValue(group["ColorG"]),
"color_b": deepCopyValue(group["ColorB"]),
"visual_effect_id": deepCopyValue(hit["VisualEffectID"]),
"ranged_effect_id": deepCopyValue(hit["RangedEffectID"]),
})
}
coreEntries, err := coerceEntriesMap(coreModule["entries"])
if err != nil {
return nil, nil, err
}
groupEntries, err := coerceEntriesMap(groupModule["entries"])
if err != nil {
return nil, nil, err
}
hitEntries, err := coerceEntriesMap(hitvisualModule["entries"])
if err != nil {
return nil, nil, err
}
keys := make([]string, 0, len(coreEntries))
for key := range coreEntries {
keys = append(keys, key)
}
sort.Slice(keys, func(i, j int) bool {
a := strings.TrimPrefix(keys[i], "damagetypes:")
b := strings.TrimPrefix(keys[j], "damagetypes:")
return a < b
})
for _, coreKey := range keys {
core := coreEntries[coreKey]
suffix := strings.TrimPrefix(coreKey, "damagetypes:")
group := groupEntries["damagetypegroups:"+suffix]
if group == nil {
return nil, nil, fmt.Errorf("%s: missing matching group entry", coreKey)
}
hit := hitEntries["damagehitvisual:"+suffix]
if hit == nil {
return nil, nil, fmt.Errorf("%s: missing matching hitvisual entry", coreKey)
}
id, ok := coreLock[coreKey]
if !ok {
return nil, nil, fmt.Errorf("%s: missing core lock id", coreKey)
}
groupKey := "damagetypegroups:" + suffix
groupID, ok := groupLock[groupKey]
if !ok {
return nil, nil, fmt.Errorf("%s: missing group lock id", groupKey)
}
hitKey := "damagehitvisual:" + suffix
hitID, ok := hitvisualLock[hitKey]
if !ok {
return nil, nil, fmt.Errorf("%s: missing hitvisual lock id", hitKey)
}
if hitID != id {
return nil, nil, fmt.Errorf("%s: core id %d does not match hitvisual id %d", suffix, id, hitID)
}
key := "damagetype:" + suffix
lockData[key] = id
outRows = append(outRows, map[string]any{
"key": key,
"id": id,
"label": deepCopyValue(core["Label"]),
"charsheet_strref": deepCopyValue(core["CharsheetStrref"]),
"damage_type_group": strconvString(groupID),
"damage_ranged_projectile": deepCopyValue(core["DamageRangedProjectile"]),
"group_label": deepCopyValue(group["Label"]),
"feedback_strref": deepCopyValue(group["FeedbackStrref"]),
"color_r": deepCopyValue(group["ColorR"]),
"color_g": deepCopyValue(group["ColorG"]),
"color_b": deepCopyValue(group["ColorB"]),
"visual_effect_id": deepCopyValue(hit["VisualEffectID"]),
"ranged_effect_id": deepCopyValue(hit["RangedEffectID"]),
})
}
sort.Slice(outRows, func(i, j int) bool {
return outRows[i]["id"].(int) < outRows[j]["id"].(int)
})
return outRows, lockData, nil
}
func coerceRowMapSlice(value any) ([]map[string]any, error) {
rawRows, ok := value.([]any)
if !ok {
return nil, fmt.Errorf("rows must be an array")
}
rows := make([]map[string]any, 0, len(rawRows))
for _, raw := range rawRows {
row, ok := raw.(map[string]any)
if !ok {
return nil, fmt.Errorf("row must be an object")
}
rows = append(rows, row)
}
return rows, nil
}
func coerceEntriesMap(value any) (map[string]map[string]any, error) {
rawEntries, ok := value.(map[string]any)
if !ok {
return nil, fmt.Errorf("entries must be an object")
}
entries := make(map[string]map[string]any, len(rawEntries))
for key, raw := range rawEntries {
row, ok := raw.(map[string]any)
if !ok {
return nil, fmt.Errorf("entry %s must be an object", key)
}
entries[key] = row
}
return entries, nil
}
func normalizeDamagetypeKey(value any) string {
text := strings.TrimSpace(strings.ToLower(format2DAValue(value)))
text = strings.TrimSuffix(text, "damage")
text = strings.ReplaceAll(text, " ", "")
text = strings.ReplaceAll(text, "_", "")
text = strings.ReplaceAll(text, "-", "")
return text
}
func strconvString(v int) string {
return fmt.Sprintf("%d", v)
}
-99
View File
@@ -1,99 +0,0 @@
package topdata
import (
"encoding/json"
"os"
"path/filepath"
)
func importLegacyDoortypes(referenceBuilderDir, dataDir string, legacyTLK *legacyTLKData) (int, error) {
_ = legacyTLK
legacyDir := filepath.Join(referenceBuilderDir, "data", "doortypes")
if _, err := os.Stat(legacyDir); err != nil {
if os.IsNotExist(err) {
return 0, nil
}
return 0, err
}
targetDir := filepath.Join(dataDir, "doortypes")
targetBasePath := filepath.Join(targetDir, "base.json")
targetLockPath := filepath.Join(targetDir, "lock.json")
targetModulesDir := filepath.Join(targetDir, "modules")
moduleNames := []string{
"add_pld01.json",
"add_sic11.json",
"add_tapr.json",
"add_tdm01.json",
"add_tdx01.json",
"add_tei01.json",
"add_tfm01.json",
}
targetModulePaths := make([]string, 0, len(moduleNames))
for _, name := range moduleNames {
targetModulePaths = append(targetModulePaths, filepath.Join(targetModulesDir, name))
}
if fileExists(targetBasePath) && fileExists(targetLockPath) {
allModulesPresent := true
for _, path := range targetModulePaths {
if !fileExists(path) {
allModulesPresent = false
break
}
}
if allModulesPresent {
return 0, nil
}
}
baseObj, err := loadJSONObject(filepath.Join(legacyDir, "base.json"))
if err != nil {
return 0, err
}
baseObj["output"] = "doortypes.2da"
lockObj, err := loadJSONObject(filepath.Join(legacyDir, "lock.json"))
if err != nil {
return 0, err
}
moduleObjs := make([]map[string]any, 0, len(moduleNames))
for _, name := range moduleNames {
obj, err := loadJSONObject(filepath.Join(legacyDir, "modules", name))
if err != nil {
return 0, err
}
moduleObjs = append(moduleObjs, obj)
}
if err := os.MkdirAll(targetModulesDir, 0o755); err != nil {
return 0, err
}
writes := []struct {
path string
obj map[string]any
}{
{path: targetBasePath, obj: baseObj},
{path: targetLockPath, obj: lockObj},
}
for i, path := range targetModulePaths {
writes = append(writes, struct {
path string
obj map[string]any
}{path: path, obj: moduleObjs[i]})
}
for _, write := range writes {
raw, err := json.MarshalIndent(write.obj, "", " ")
if err != nil {
return 0, err
}
raw = append(raw, '\n')
if err := os.WriteFile(write.path, raw, 0o644); err != nil {
return 0, err
}
}
return len(writes), nil
}
-64
View File
@@ -1,64 +0,0 @@
package topdata
import (
"encoding/json"
"os"
"path/filepath"
)
func importLegacyFeat(referenceBuilderDir, dataDir string, legacyTLK *legacyTLKData) (int, error) {
_ = legacyTLK
legacyDir := filepath.Join(referenceBuilderDir, "data", "feat")
if _, err := os.Stat(legacyDir); err != nil {
if os.IsNotExist(err) {
return 0, nil
}
return 0, err
}
legacyBasePath := filepath.Join(legacyDir, "base.json")
if _, err := os.Stat(legacyBasePath); err != nil {
if os.IsNotExist(err) {
return 0, nil
}
return 0, err
}
targetDir := filepath.Join(dataDir, "feat")
targetBasePath := filepath.Join(targetDir, "base.json")
var targetObj map[string]any
if _, err := os.Stat(targetBasePath); err == nil {
targetObj, err = loadJSONObject(targetBasePath)
if err != nil {
return 0, err
}
} else if !os.IsNotExist(err) {
return 0, err
}
if targetObj != nil {
if rows, ok := targetObj["rows"].([]any); ok && len(rows) > 0 {
return 0, nil
}
}
baseObj, err := loadJSONObject(legacyBasePath)
if err != nil {
return 0, err
}
baseObj["output"] = "feat.2da"
baseObj["compare_reference"] = false
canonicalizeWikiMetadataDocument(baseObj)
if err := os.MkdirAll(targetDir, 0o755); err != nil {
return 0, err
}
raw, err := json.MarshalIndent(baseObj, "", " ")
if err != nil {
return 0, err
}
raw = append(raw, '\n')
if err := os.WriteFile(targetBasePath, raw, 0o644); err != nil {
return 0, err
}
return 1, nil
}
-6
View File
@@ -1,6 +0,0 @@
package topdata
func importLegacyGenericdoors(referenceBuilderDir, dataDir string, legacyTLK *legacyTLKData) (int, error) {
_ = legacyTLK
return importLegacyDatasetMirror(referenceBuilderDir, dataDir, "genericdoors", "genericdoors.2da", nil)
}
-391
View File
@@ -601,397 +601,6 @@ func deepCopyValue(value any) any {
return out
}
func importLegacyItempropsRegistry(referenceBuilderDir, dataDir string, legacyTLK *legacyTLKData) (int, error) {
legacyDataDir := filepath.Join(referenceBuilderDir, "data")
registryDir := filepath.Join(dataDir, filepath.FromSlash(itempropsRegistryDirName))
if _, err := os.Stat(filepath.Join(registryDir, "properties.json")); err == nil {
if refs, err := countLegacyTLKRefsInRegistry(registryDir); err == nil && refs == 0 {
return 0, nil
}
}
if _, err := os.Stat(filepath.Join(legacyDataDir, "itemprops")); err != nil {
if os.IsNotExist(err) {
return 0, nil
}
return 0, err
}
defs, err := collectBaseDataset(nativeDataset{
Name: "itemprops/defs",
BasePath: filepath.Join(legacyDataDir, "itemprops", "defs", "base.json"),
LockPath: filepath.Join(legacyDataDir, "itemprops", "defs", "lock.json"),
ModulesDir: filepath.Join(legacyDataDir, "itemprops", "defs", "modules"),
Spec: specForDataset("itemprops"),
})
if err != nil {
return 0, err
}
sets, err := collectBaseDataset(nativeDataset{
Name: "itemprops/sets",
BasePath: filepath.Join(legacyDataDir, "itemprops", "sets", "base.json"),
LockPath: filepath.Join(legacyDataDir, "itemprops", "sets", "lock.json"),
ModulesDir: filepath.Join(legacyDataDir, "itemprops", "sets", "modules"),
Spec: specForDataset("itemprops"),
})
if err != nil {
return 0, err
}
costIndex, err := collectBaseDataset(nativeDataset{
Name: "itemprops/costtables/index",
BasePath: filepath.Join(legacyDataDir, "itemprops", "costtables", "index", "base.json"),
LockPath: filepath.Join(legacyDataDir, "itemprops", "costtables", "index", "lock.json"),
ModulesDir: filepath.Join(legacyDataDir, "itemprops", "costtables", "index", "modules"),
Spec: specForDataset("itemprops"),
})
if err != nil {
return 0, err
}
paramIndex, err := collectBaseDataset(nativeDataset{
Name: "itemprops/paramtables/index",
BasePath: filepath.Join(legacyDataDir, "itemprops", "paramtables", "index", "base.json"),
LockPath: filepath.Join(legacyDataDir, "itemprops", "paramtables", "index", "lock.json"),
ModulesDir: filepath.Join(legacyDataDir, "itemprops", "paramtables", "index", "modules"),
Spec: specForDataset("itemprops"),
})
if err != nil {
return 0, err
}
ownedCostTables, err := collectLegacyOwnedItempropTables(filepath.Join(legacyDataDir, "itemprops", "costtables"), "index")
if err != nil {
return 0, err
}
ownedParamTables, err := collectLegacyOwnedItempropTables(filepath.Join(legacyDataDir, "itemprops", "paramtables"), "index")
if err != nil {
return 0, err
}
ownedSubtypeTables, err := collectLegacyOwnedItempropTables(filepath.Join(legacyDataDir, "itemprops", "subtypes"), "")
if err != nil && !os.IsNotExist(err) {
return 0, err
}
setsByID := map[int]map[string]any{}
for _, row := range sets.Rows {
setsByID[row["id"].(int)] = row
}
costByID := map[int]map[string]any{}
for _, row := range costIndex.Rows {
costByID[row["id"].(int)] = row
}
paramByID := map[int]map[string]any{}
for _, row := range paramIndex.Rows {
paramByID[row["id"].(int)] = row
}
registryLock := map[string]int{}
legacyKeyByID := map[string]string{}
if legacyTLK != nil {
for key, id := range legacyTLK.Lock {
legacyKeyByID[strconv.Itoa(id)] = key
}
}
properties := make([]map[string]any, 0, len(defs.Rows))
subtypeRegistryByKey := map[string]map[string]any{}
for _, row := range defs.Rows {
rowID := row["id"].(int)
key := canonicalItempropPropertyKey(row)
registryLock[key] = rowID
property := map[string]any{
"key": key,
"label": row["Label"],
"name": deepCopyValue(row["Name"]),
"property_text": deepCopyValue(row["GameStrRef"]),
}
if description := nullableRegistryField(row, "Description"); description != nullValue {
property["description"] = description
}
if setRow, ok := setsByID[rowID]; ok {
property["availability"] = importLegacyAvailability(setRow)
if setName := nullableRegistryField(setRow, "StringRef"); setName != nullValue && setName != stringField(row, "Name") {
property["set_name"] = setName
}
if setLabel := stringField(setRow, "Label"); setLabel != "" && setLabel != stringField(row, "Label") {
property["set_label"] = setLabel
}
} else {
property["availability"] = []any{}
}
if subtypeResRef := stringField(row, "SubTypeResRef"); subtypeResRef != "" && subtypeResRef != nullValue {
subtypeKey := canonicalModelKey("subtype_models", subtypeResRef)
property["subtype"] = map[string]any{"ref": subtypeKey}
if _, ok := subtypeRegistryByKey[subtypeKey]; !ok {
subtypeRegistryByKey[subtypeKey] = buildLegacySubtypeModel(subtypeKey, subtypeResRef, ownedSubtypeTables)
}
}
costValue := stringField(row, "Cost")
costID := stringField(row, "CostTableResRef")
if costValue != "" && costValue != nullValue || costID != "" && costID != nullValue {
cost := map[string]any{}
if costValue != "" && costValue != nullValue {
cost["value"] = costValue
}
if costID != "" && costID != nullValue {
id, err := strconv.Atoi(costID)
if err != nil {
return 0, err
}
costRow, ok := costByID[id]
if !ok {
return 0, fmt.Errorf("%s references missing cost table index %s", key, costID)
}
cost["ref"] = canonicalModelKey("cost_models", stringField(costRow, "Name"))
}
property["cost"] = cost
}
if paramID := stringField(row, "Param1ResRef"); paramID != "" && paramID != nullValue {
id, err := strconv.Atoi(paramID)
if err != nil {
return 0, err
}
paramRow, ok := paramByID[id]
if !ok {
return 0, fmt.Errorf("%s references missing param table index %s", key, paramID)
}
property["param"] = map[string]any{"ref": canonicalModelKey("param_models", stringField(paramRow, "TableResRef"))}
}
if legacyTLK != nil {
property["name"] = importLegacyStrrefValue(property["name"], key+".name", legacyTLK, legacyKeyByID)
property["property_text"] = importLegacyStrrefValue(property["property_text"], key+".property_text", legacyTLK, legacyKeyByID)
if description, ok := property["description"]; ok {
property["description"] = importLegacyStrrefValue(description, key+".description", legacyTLK, legacyKeyByID)
}
if updated, _, err := inlineLegacyTLKValue(property, legacyTLK); err != nil {
return 0, err
} else if updated {
// property updated in place
}
}
properties = append(properties, property)
}
costModels := make([]map[string]any, 0, len(costIndex.Rows))
for _, row := range costIndex.Rows {
key := canonicalModelKey("cost_models", stringField(row, "Name"))
registryLock[key] = row["id"].(int)
model := map[string]any{
"key": key,
"index_name": row["Name"],
"label": row["Label"],
"client_load": row["ClientLoad"],
"table": buildLegacyOwnedModelTable(stringField(row, "Name"), ownedCostTables),
}
costModels = append(costModels, model)
}
paramModels := make([]map[string]any, 0, len(paramIndex.Rows))
for _, row := range paramIndex.Rows {
key := canonicalModelKey("param_models", stringField(row, "TableResRef"))
registryLock[key] = row["id"].(int)
model := map[string]any{
"key": key,
"name": deepCopyValue(row["Name"]),
"label": row["Lable"],
"table": buildLegacyOwnedModelTable(stringField(row, "TableResRef"), ownedParamTables),
}
paramModels = append(paramModels, model)
}
subtypeModels := make([]map[string]any, 0, len(subtypeRegistryByKey))
for _, key := range sortedKeysAny(subtypeRegistryByKey) {
subtypeModels = append(subtypeModels, subtypeRegistryByKey[key])
}
writes := []struct {
path string
obj map[string]any
}{
{filepath.Join(registryDir, "properties.json"), map[string]any{"rows": properties}},
{filepath.Join(registryDir, "cost_models.json"), map[string]any{"rows": costModels}},
{filepath.Join(registryDir, "param_models.json"), map[string]any{"rows": paramModels}},
{filepath.Join(registryDir, "subtype_models.json"), map[string]any{"rows": subtypeModels}},
{filepath.Join(registryDir, itempropsRegistryLock), anyMapInt(registryLock)},
}
if err := os.MkdirAll(registryDir, 0o755); err != nil {
return 0, err
}
for _, write := range writes {
raw, err := json.MarshalIndent(write.obj, "", " ")
if err != nil {
return 0, err
}
raw = append(raw, '\n')
if err := os.WriteFile(write.path, raw, 0o644); err != nil {
return 0, err
}
}
return len(writes), nil
}
func countLegacyTLKRefsInRegistry(registryDir string) (int, error) {
paths, err := collectDataJSONPaths(registryDir)
if err != nil {
return 0, err
}
refs := 0
for _, path := range paths {
obj, err := loadJSONObject(path)
if err != nil {
return 0, err
}
refs += countLegacyTLKRefsInValue(obj)
}
return refs, nil
}
func collectLegacyOwnedItempropTables(root, skipDir string) (map[string]map[string]any, error) {
result := map[string]map[string]any{}
entries, err := os.ReadDir(root)
if err != nil {
return nil, err
}
for _, entry := range entries {
if !entry.IsDir() {
continue
}
if skipDir != "" && entry.Name() == skipDir {
continue
}
dir := filepath.Join(root, entry.Name())
basePath := filepath.Join(dir, "base.json")
if _, err := os.Stat(basePath); err != nil {
continue
}
collected, err := collectBaseDataset(nativeDataset{
Name: filepath.ToSlash(filepath.Join("itemprops", filepath.Base(root), entry.Name())),
BasePath: basePath,
LockPath: filepath.Join(dir, "lock.json"),
ModulesDir: filepath.Join(dir, "modules"),
Spec: specForDataset("itemprops"),
})
if err != nil {
return nil, err
}
result[strings.ToLower(outputStem(collected.Dataset.OutputName))] = map[string]any{
"ownership": "owned",
"resref": outputStem(collected.Dataset.OutputName),
"output": collected.Dataset.OutputName,
"columns": stringSliceToAny(collected.Columns),
"rows": rowsToAny(collected.Rows),
}
}
return result, nil
}
func buildLegacySubtypeModel(key, resref string, owned map[string]map[string]any) map[string]any {
return map[string]any{
"key": key,
"table_resref": resref,
"table": buildLegacyOwnedModelTable(resref, owned),
}
}
func buildLegacyOwnedModelTable(resref string, owned map[string]map[string]any) map[string]any {
if table, ok := owned[strings.ToLower(resref)]; ok {
return table
}
return map[string]any{
"ownership": "external",
"resref": resref,
}
}
func canonicalItempropPropertyKey(row map[string]any) string {
if key := stringField(row, "key"); key != "" {
return "itemprop:" + strings.TrimPrefix(key, "itempropdef:")
}
return canonicalModelKey("itemprop", stringField(row, "Label"))
}
func canonicalModelKey(prefix, source string) string {
normalized := strings.ToLower(strings.TrimSpace(source))
normalized = strings.ReplaceAll(normalized, " ", "")
normalized = strings.ReplaceAll(normalized, "-", "")
normalized = strings.ReplaceAll(normalized, "_", "")
return prefix + ":" + normalized
}
func importLegacyStrrefValue(value any, fallbackKey string, legacy *legacyTLKData, keyByID map[string]string) any {
if legacy == nil {
return value
}
if _, ok, _ := parseTLKPayload(value, true); ok {
return value
}
_ = keyByID
key := fallbackKey
entry, ok := legacy.Entries[key]
if !ok {
return value
}
payload := map[string]any{
"key": key,
"text": entry.Text,
}
if entry.SoundResRef != "" {
payload["sound_resref"] = entry.SoundResRef
}
if entry.SoundLength != 0 {
payload["sound_length"] = entry.SoundLength
}
return map[string]any{"tlk": payload}
}
func importLegacyAvailability(row map[string]any) []any {
out := make([]any, 0)
for _, column := range itempropsAvailabilityColumns {
value, ok := row[column]
if !ok {
continue
}
text := formatRegistryScalar(value)
if text != "1" {
continue
}
out = append(out, normalizeAvailabilityName(column))
}
return out
}
func anyMapInt(values map[string]int) map[string]any {
out := make(map[string]any, len(values))
for key, value := range values {
out[key] = value
}
return out
}
func stringSliceToAny(values []string) []any {
out := make([]any, 0, len(values))
for _, value := range values {
out = append(out, value)
}
return out
}
func rowsToAny(rows []map[string]any) []any {
out := make([]any, 0, len(rows))
for _, row := range rows {
out = append(out, deepCopyValue(row))
}
return out
}
func sortedKeysAny(values map[string]map[string]any) []string {
out := make([]string, 0, len(values))
for key := range values {
out = append(out, key)
}
slices.Sort(out)
return out
}
func validateItempropsRegistryGraph(dataDir string, report *ValidationReport) {
registry, err := loadItempropsRegistry(dataDir)
if err != nil {
-176
View File
@@ -1,176 +0,0 @@
package topdata
import (
"encoding/json"
"io/fs"
"os"
"path/filepath"
"sort"
"strings"
)
type legacyDatasetTransform func(relativePath string, obj map[string]any)
func importLegacyDatasetMirror(referenceBuilderDir, dataDir, datasetName, outputName string, transform legacyDatasetTransform) (int, error) {
legacyDir := filepath.Join(referenceBuilderDir, "data", datasetName)
if _, err := os.Stat(legacyDir); err != nil {
if os.IsNotExist(err) {
return 0, nil
}
return 0, err
}
targetDir := filepath.Join(dataDir, datasetName)
baseObj, err := loadJSONObject(filepath.Join(legacyDir, "base.json"))
if err != nil {
return 0, err
}
if outputName != "" {
baseObj["output"] = outputName
}
if transform != nil {
transform("base.json", baseObj)
}
lockObj, err := loadJSONObject(filepath.Join(legacyDir, "lock.json"))
if err != nil {
return 0, err
}
if transform != nil {
transform("lock.json", lockObj)
}
moduleRelPaths, err := collectJSONRelativePaths(filepath.Join(legacyDir, "modules"))
if err != nil {
return 0, err
}
moduleObjs := make(map[string]map[string]any, len(moduleRelPaths))
for _, relPath := range moduleRelPaths {
obj, err := loadJSONObject(filepath.Join(legacyDir, "modules", relPath))
if err != nil {
return 0, err
}
normalizedRel := filepath.ToSlash(relPath)
if transform != nil {
transform(filepath.ToSlash(filepath.Join("modules", normalizedRel)), obj)
}
moduleObjs[normalizedRel] = obj
}
if err := os.MkdirAll(targetDir, 0o755); err != nil {
return 0, err
}
updated := 0
for _, write := range []struct {
path string
obj map[string]any
}{
{path: filepath.Join(targetDir, "base.json"), obj: baseObj},
{path: filepath.Join(targetDir, "lock.json"), obj: lockObj},
} {
changed, err := writeJSONObjectIfChanged(write.path, write.obj)
if err != nil {
return 0, err
}
if changed {
updated++
}
}
targetModulesDir := filepath.Join(targetDir, "modules")
if len(moduleObjs) > 0 {
if err := os.MkdirAll(targetModulesDir, 0o755); err != nil {
return 0, err
}
}
for relPath, obj := range moduleObjs {
targetPath := filepath.Join(targetModulesDir, filepath.FromSlash(relPath))
if err := os.MkdirAll(filepath.Dir(targetPath), 0o755); err != nil {
return 0, err
}
changed, err := writeJSONObjectIfChanged(targetPath, obj)
if err != nil {
return 0, err
}
if changed {
updated++
}
}
staleModules, err := collectJSONRelativePaths(targetModulesDir)
if err != nil {
return 0, err
}
for _, relPath := range staleModules {
normalizedRel := filepath.ToSlash(relPath)
if _, ok := moduleObjs[normalizedRel]; ok {
continue
}
if err := os.Remove(filepath.Join(targetModulesDir, filepath.FromSlash(normalizedRel))); err != nil && !os.IsNotExist(err) {
return 0, err
}
updated++
}
return updated, nil
}
func collectJSONRelativePaths(root string) ([]string, error) {
if _, err := os.Stat(root); err != nil {
if os.IsNotExist(err) {
return nil, nil
}
return nil, err
}
var relPaths []string
err := filepath.WalkDir(root, func(path string, d fs.DirEntry, err error) error {
if err != nil {
return err
}
if d.IsDir() {
return nil
}
if !strings.EqualFold(filepath.Ext(d.Name()), ".json") {
return nil
}
relPath, err := filepath.Rel(root, path)
if err != nil {
return err
}
relPaths = append(relPaths, filepath.ToSlash(relPath))
return nil
})
if err != nil {
return nil, err
}
sort.Strings(relPaths)
return relPaths, nil
}
func writeJSONObjectIfChanged(path string, obj map[string]any) (bool, error) {
raw, err := json.MarshalIndent(obj, "", " ")
if err != nil {
return false, err
}
raw = append(raw, '\n')
current, err := os.ReadFile(path)
if err == nil && string(current) == string(raw) {
return false, nil
}
if err != nil && !os.IsNotExist(err) {
return false, err
}
if err := os.WriteFile(path, raw, 0o644); err != nil {
return false, err
}
return true, nil
}
func fileExists(path string) bool {
_, err := os.Stat(path)
return err == nil
}
-6
View File
@@ -1,6 +0,0 @@
package topdata
func importLegacyLoadscreens(referenceBuilderDir, dataDir string, legacyTLK *legacyTLKData) (int, error) {
_ = legacyTLK
return importLegacyDatasetMirror(referenceBuilderDir, dataDir, "loadscreens", "loadscreens.2da", nil)
}
-237
View File
@@ -1,237 +0,0 @@
package topdata
import (
"encoding/json"
"os"
"path/filepath"
"strconv"
)
func importLegacyMasterfeats(referenceBuilderDir, dataDir string, legacyTLK *legacyTLKData) (int, error) {
legacyDir := filepath.Join(referenceBuilderDir, "data", "feat", "masterfeats")
if _, err := os.Stat(legacyDir); err != nil {
if os.IsNotExist(err) {
return 0, nil
}
return 0, err
}
targetDir := filepath.Join(dataDir, "masterfeats")
targetPath := filepath.Join(targetDir, "base.json")
legacyPlainPath := filepath.Join(targetDir, "masterfeats.json")
if _, err := os.Stat(targetPath); err == nil {
obj, err := loadJSONObject(targetPath)
if err == nil && countLegacyTLKRefsInValue(obj) == 0 {
return 0, nil
}
}
collected, err := importLegacyMasterfeatsDataset(legacyDir)
if err != nil {
return 0, err
}
referenceIDs, err := collectReferenceLockIDs(filepath.Join(referenceBuilderDir, "data"))
if err != nil {
return 0, err
}
rows := make([]any, 0, len(collected.Rows))
for _, row := range collected.Rows {
copyRow, ok := deepCopyValue(row).(map[string]any)
if !ok {
continue
}
if legacyTLK != nil {
if _, _, err := inlineLegacyTLKValue(copyRow, legacyTLK); err != nil {
return 0, err
}
}
copyRow = rewriteImportedIDRefs(copyRow, referenceIDs)
rows = append(rows, copyRow)
}
if err := os.MkdirAll(targetDir, 0o755); err != nil {
return 0, err
}
writes := []struct {
path string
obj map[string]any
}{
{
path: targetPath,
obj: map[string]any{
"output": collected.Dataset.OutputName,
"columns": stringSliceToAny(collected.Columns),
"rows": rows,
},
},
{
path: filepath.Join(targetDir, "lock.json"),
obj: anyMapInt(collected.LockData),
},
}
for _, write := range writes {
raw, err := json.MarshalIndent(write.obj, "", " ")
if err != nil {
return 0, err
}
raw = append(raw, '\n')
if err := os.WriteFile(write.path, raw, 0o644); err != nil {
return 0, err
}
}
updatedFiles := len(writes)
if err := os.Remove(legacyPlainPath); err == nil {
updatedFiles++
} else if err != nil && !os.IsNotExist(err) {
return 0, err
}
return updatedFiles, nil
}
func importLegacyMasterfeatsDataset(legacyDir string) (nativeCollectedDataset, error) {
dataset := nativeDataset{
Name: "masterfeats",
BasePath: filepath.Join(legacyDir, "masterfeats.json"),
LockPath: filepath.Join(legacyDir, "lock.json"),
Spec: specForDataset("masterfeats"),
}
tableData, err := loadJSONObject(dataset.BasePath)
if err != nil {
return nativeCollectedDataset{}, err
}
columns, err := parseColumns(tableData, dataset.Name)
if err != nil {
return nativeCollectedDataset{}, err
}
rawRows, ok := tableData["rows"].([]any)
if !ok {
return nativeCollectedDataset{}, nil
}
lockData, err := loadLockfile(dataset.LockPath)
if err != nil {
return nativeCollectedDataset{}, err
}
usedIDs := map[int]struct{}{}
for _, rowID := range lockData {
usedIDs[rowID] = struct{}{}
}
rows := make([]map[string]any, 0, len(rawRows))
explicitID := make([]bool, 0, len(rawRows))
for index, raw := range rawRows {
rowObj, ok := raw.(map[string]any)
if !ok {
continue
}
rowID := index
if value, ok := rowObj["id"]; ok {
parsed, err := asInt(value)
if err != nil {
return nativeCollectedDataset{}, err
}
rowID = parsed
}
row := map[string]any{"id": rowID}
for _, column := range columns {
row[column] = nullValue
}
for key, value := range rowObj {
switch key {
case "id":
case "key":
if keyText, ok := value.(string); ok && keyText != "" {
row["key"] = keyText
}
default:
columnName, ok := canonicalColumn(columns, key)
if !ok {
continue
}
row[columnName] = value
}
}
rows = append(rows, row)
_, hasExplicitID := rowObj["id"]
explicitID = append(explicitID, hasExplicitID)
if hasExplicitID {
usedIDs[rowID] = struct{}{}
}
}
nextID := nextAvailableID(usedIDs)
lockModified := false
for index, row := range rows {
key, hasKey := row["key"].(string)
if hasKey && key != "" {
if lockedID, ok := lockData[key]; ok {
row["id"] = lockedID
continue
}
if explicitID[index] {
lockData[key] = row["id"].(int)
} else {
row["id"] = nextID
lockData[key] = nextID
usedIDs[nextID] = struct{}{}
nextID = nextAvailableID(usedIDs)
}
lockModified = true
continue
}
if !explicitID[index] {
row["id"] = index
}
}
if lockModified {
_ = lockModified
}
return nativeCollectedDataset{
Dataset: nativeDataset{
Name: dataset.Name,
OutputName: "masterfeats.2da",
Columns: columns,
Spec: dataset.Spec,
},
Columns: columns,
Rows: rows,
LockData: lockData,
}, nil
}
func rewriteImportedIDRefs(value any, ids map[string]int) map[string]any {
row, ok := rewriteImportedIDRefsValue(value, ids).(map[string]any)
if !ok {
return map[string]any{}
}
return row
}
func rewriteImportedIDRefsValue(value any, ids map[string]int) any {
switch typed := value.(type) {
case map[string]any:
if len(typed) == 1 {
if rawID, ok := typed["id"]; ok {
if key, ok := rawID.(string); ok {
if resolved, ok := ids[key]; ok {
return strconv.Itoa(resolved)
}
}
}
}
out := make(map[string]any, len(typed))
for key, child := range typed {
out[key] = rewriteImportedIDRefsValue(child, ids)
}
return out
case []any:
out := make([]any, 0, len(typed))
for _, child := range typed {
out = append(out, rewriteImportedIDRefsValue(child, ids))
}
return out
default:
return value
}
}
-415
View File
@@ -1,415 +0,0 @@
package topdata
import (
"encoding/json"
"fmt"
"os"
"path/filepath"
"strings"
"git.westgate.pw/ShadowsOverWestgate/sow-tools/internal/project"
)
type NormalizeResult struct {
StatePath string
ScannedFiles int
UpdatedFiles int
InlineValues int
RemainingFiles int
RemainingLegacyRefs int
}
func NormalizeProject(p *project.Project) (NormalizeResult, error) {
if !p.HasTopData() {
return NormalizeResult{}, fmt.Errorf("topdata is not configured for this project")
}
sourceDir := p.TopDataSourceDir()
dataDir := filepath.Join(sourceDir, "data")
migrationRoot := resolveMigrationInputRoot(sourceDir, p.TopDataReferenceBuilderDir())
legacyTLK, err := loadLegacyTLK(filepath.Join(sourceDir, "tlk"))
if err != nil {
return NormalizeResult{}, err
}
if migrationRoot != "" {
migrationTLK, err := loadLegacyTLK(filepath.Join(migrationRoot, "tlk"))
if err != nil {
return NormalizeResult{}, err
}
legacyTLK = mergeLegacyTLK(migrationTLK, legacyTLK)
}
baseDialogImported, err := importLegacyBaseDialog(migrationRoot, sourceDir)
if err != nil {
return NormalizeResult{}, err
}
itempropsImported, err := importLegacyItempropsRegistry(migrationRoot, dataDir, legacyTLK)
if err != nil {
return NormalizeResult{}, err
}
masterfeatsImported, err := importLegacyMasterfeats(migrationRoot, dataDir, legacyTLK)
if err != nil {
return NormalizeResult{}, err
}
featImported, err := importLegacyFeat(migrationRoot, dataDir, legacyTLK)
if err != nil {
return NormalizeResult{}, err
}
creaturespeedImported, err := importLegacyCreaturespeed(migrationRoot, dataDir, legacyTLK)
if err != nil {
return NormalizeResult{}, err
}
armorImported, err := importLegacyArmor(migrationRoot, dataDir, legacyTLK)
if err != nil {
return NormalizeResult{}, err
}
baseitemsImported, err := importLegacyBaseitems(migrationRoot, dataDir, legacyTLK)
if err != nil {
return NormalizeResult{}, err
}
damagetypesImported, err := importLegacyDamagetypes(migrationRoot, dataDir, legacyTLK)
if err != nil {
return NormalizeResult{}, err
}
skyboxesImported, err := importLegacySkyboxes(migrationRoot, dataDir, legacyTLK)
if err != nil {
return NormalizeResult{}, err
}
vfxPersistentImported, err := importLegacyVFXPersistent(migrationRoot, dataDir, legacyTLK)
if err != nil {
return NormalizeResult{}, err
}
loadscreensImported, err := importLegacyLoadscreens(migrationRoot, dataDir, legacyTLK)
if err != nil {
return NormalizeResult{}, err
}
progfxImported, err := importLegacyProgfx(migrationRoot, dataDir, legacyTLK)
if err != nil {
return NormalizeResult{}, err
}
genericdoorsImported, err := importLegacyGenericdoors(migrationRoot, dataDir, legacyTLK)
if err != nil {
return NormalizeResult{}, err
}
cloakmodelImported, err := importLegacyCloakmodel(migrationRoot, dataDir, legacyTLK)
if err != nil {
return NormalizeResult{}, err
}
repadjustImported, err := importLegacyRepadjust(migrationRoot, dataDir, legacyTLK)
if err != nil {
return NormalizeResult{}, err
}
wingmodelImported, err := importLegacyWingmodel(migrationRoot, dataDir, legacyTLK)
if err != nil {
return NormalizeResult{}, err
}
tailmodelImported, err := importLegacyTailmodel(migrationRoot, dataDir, legacyTLK)
if err != nil {
return NormalizeResult{}, err
}
doortypesImported, err := importLegacyDoortypes(migrationRoot, dataDir, legacyTLK)
if err != nil {
return NormalizeResult{}, err
}
visualeffectsImported, err := importLegacyVisualeffects(migrationRoot, dataDir, legacyTLK)
if err != nil {
return NormalizeResult{}, err
}
appearanceImported, err := importLegacyAppearance(migrationRoot, dataDir, legacyTLK)
if err != nil {
return NormalizeResult{}, err
}
placeablesImported, err := importLegacyPlaceables(migrationRoot, dataDir, legacyTLK)
if err != nil {
return NormalizeResult{}, err
}
rulesetImported, err := importLegacyRuleset(migrationRoot, dataDir, legacyTLK)
if err != nil {
return NormalizeResult{}, err
}
skillsImported, err := importLegacySkills(migrationRoot, dataDir, legacyTLK)
if err != nil {
return NormalizeResult{}, err
}
spellsImported, err := importLegacySpells(migrationRoot, dataDir, legacyTLK)
if err != nil {
return NormalizeResult{}, err
}
racialtypesImported, err := importLegacyRacialtypes(migrationRoot, dataDir, legacyTLK)
if err != nil {
return NormalizeResult{}, err
}
classesImported, err := importLegacyClasses(migrationRoot, dataDir, legacyTLK)
if err != nil {
return NormalizeResult{}, err
}
portraitsImported, err := importLegacyPortraits(migrationRoot, dataDir, legacyTLK)
if err != nil {
return NormalizeResult{}, err
}
paths, err := collectDataJSONPaths(dataDir)
if err != nil {
return NormalizeResult{}, err
}
result := NormalizeResult{
StatePath: filepath.Join(sourceDir, tlkStateFile),
ScannedFiles: len(paths),
UpdatedFiles: baseDialogImported + itempropsImported + masterfeatsImported + featImported + creaturespeedImported + armorImported + baseitemsImported + damagetypesImported + skyboxesImported + vfxPersistentImported + loadscreensImported + progfxImported + genericdoorsImported + cloakmodelImported + repadjustImported + wingmodelImported + tailmodelImported + doortypesImported + visualeffectsImported + appearanceImported + placeablesImported + rulesetImported + skillsImported + spellsImported + racialtypesImported + classesImported + portraitsImported,
}
state, err := loadTLKState(result.StatePath)
if err != nil {
return result, err
}
if state.Entries == nil {
state.Entries = map[string]tlkStateMapping{}
}
if legacyTLK != nil && state.Language == "" {
state.Language = legacyTLK.Language
}
if legacyTLK != nil {
for key, id := range legacyTLK.Lock {
if _, ok := state.Entries[key]; !ok {
state.Entries[key] = tlkStateMapping{ID: id}
}
}
}
if legacyTLK != nil {
for _, path := range paths {
updated, count, err := inlineLegacyTLKFile(path, legacyTLK)
if err != nil {
return result, err
}
if updated {
result.UpdatedFiles++
result.InlineValues += count
}
}
}
if err := saveTLKState(result.StatePath, state); err != nil {
return result, err
}
if err := removeLegacyTLKAuthoredInput(filepath.Join(sourceDir, "tlk")); err != nil {
return result, err
}
remainingFiles, remainingRefs, err := countLegacyTLKRefs(paths)
if err != nil {
return result, err
}
result.RemainingFiles = remainingFiles
result.RemainingLegacyRefs = remainingRefs
return result, nil
}
func mergeLegacyTLK(base, override *legacyTLKData) *legacyTLKData {
if base == nil && override == nil {
return nil
}
merged := &legacyTLKData{
Language: "en",
Entries: map[string]tlkEntryData{},
Lock: map[string]int{},
}
if base != nil {
if base.Language != "" {
merged.Language = base.Language
}
for key, entry := range base.Entries {
merged.Entries[key] = entry
}
for key, id := range base.Lock {
merged.Lock[key] = id
}
}
if override != nil {
if override.Language != "" {
merged.Language = override.Language
}
for key, entry := range override.Entries {
merged.Entries[key] = entry
}
for key, id := range override.Lock {
merged.Lock[key] = id
}
}
return merged
}
func removeLegacyTLKAuthoredInput(tlkDir string) error {
modulesDir := filepath.Join(tlkDir, "modules")
if err := os.RemoveAll(modulesDir); err != nil {
return fmt.Errorf("remove %s: %w", modulesDir, err)
}
lockPath := filepath.Join(tlkDir, "lock.json")
if err := os.Remove(lockPath); err != nil && !os.IsNotExist(err) {
return fmt.Errorf("remove %s: %w", lockPath, err)
}
gitkeepPath := filepath.Join(tlkDir, ".gitkeep")
if err := os.Remove(gitkeepPath); err != nil && !os.IsNotExist(err) {
return fmt.Errorf("remove %s: %w", gitkeepPath, err)
}
if err := os.Remove(tlkDir); err != nil && !os.IsNotExist(err) {
if !strings.Contains(err.Error(), "directory not empty") {
return fmt.Errorf("remove %s: %w", tlkDir, err)
}
}
return nil
}
func resolveMigrationInputRoot(sourceDir, referenceBuilderDir string) string {
if strings.TrimSpace(referenceBuilderDir) != "" {
return referenceBuilderDir
}
snapshotRoot := filepath.Join(sourceDir, "migration_snapshot")
if info, err := os.Stat(snapshotRoot); err == nil && info.IsDir() {
return snapshotRoot
}
return ""
}
func inlineLegacyTLKFile(path string, legacy *legacyTLKData) (bool, int, error) {
obj, err := loadJSONObject(path)
if err != nil {
return false, 0, err
}
updated, count, err := inlineLegacyTLKValue(obj, legacy)
if err != nil {
return false, 0, err
}
if !updated {
return false, 0, nil
}
raw, err := json.MarshalIndent(obj, "", " ")
if err != nil {
return false, 0, err
}
raw = append(raw, '\n')
if err := os.WriteFile(path, raw, 0o644); err != nil {
return false, 0, err
}
return true, count, nil
}
func inlineLegacyTLKValue(value any, legacy *legacyTLKData) (bool, int, error) {
switch typed := value.(type) {
case map[string]any:
if rawTLK, ok := typed["tlk"]; ok {
key, ok := rawTLK.(string)
if ok {
entry, ok := legacy.Entries[key]
if !ok {
return false, 0, nil
}
payload := map[string]any{
"key": key,
"text": entry.Text,
}
if entry.SoundResRef != "" {
payload["sound_resref"] = entry.SoundResRef
}
if entry.SoundLength != 0 {
payload["sound_length"] = entry.SoundLength
}
typed["tlk"] = payload
return true, 1, nil
}
}
updated := false
total := 0
for key, child := range typed {
childUpdated, childCount, err := inlineLegacyTLKValue(child, legacy)
if err != nil {
return false, 0, err
}
if childUpdated {
typed[key] = child
updated = true
total += childCount
}
}
return updated, total, nil
case []any:
updated := false
total := 0
for index, child := range typed {
childUpdated, childCount, err := inlineLegacyTLKValue(child, legacy)
if err != nil {
return false, 0, err
}
if childUpdated {
typed[index] = child
updated = true
total += childCount
}
}
return updated, total, nil
default:
return false, 0, nil
}
}
func collectDataJSONPaths(dataDir string) ([]string, error) {
paths := make([]string, 0)
err := filepath.WalkDir(dataDir, func(path string, d os.DirEntry, err error) error {
if err != nil {
return err
}
if d.IsDir() {
return nil
}
if strings.HasSuffix(strings.ToLower(d.Name()), ".json") && d.Name() != "lock.json" {
paths = append(paths, path)
}
return nil
})
if err != nil {
return nil, err
}
return paths, nil
}
func countLegacyTLKRefs(paths []string) (int, int, error) {
files := 0
refs := 0
for _, path := range paths {
obj, err := loadJSONObject(path)
if err != nil {
return 0, 0, err
}
count := countLegacyTLKRefsInValue(obj)
if count == 0 {
continue
}
files++
refs += count
}
return files, refs, nil
}
func countLegacyTLKRefsInValue(value any) int {
switch typed := value.(type) {
case map[string]any:
total := 0
if rawTLK, ok := typed["tlk"]; ok {
if _, ok := rawTLK.(string); ok {
total++
}
}
for _, child := range typed {
total += countLegacyTLKRefsInValue(child)
}
return total
case []any:
total := 0
for _, child := range typed {
total += countLegacyTLKRefsInValue(child)
}
return total
default:
return 0
}
}
+31 -78
View File
@@ -410,6 +410,9 @@ func buildNativeUnchecked(p *project.Project, opts NativeBuildOptions, progress
if err != nil {
return BuildResult{}, err
}
if err := loadStandaloneTLKStrings(sourceDir, compiler); err != nil {
return BuildResult{}, err
}
output2DA := compiled2DAOutputDir(p)
outputTLK := compiledTLKOutputDir(p)
@@ -2317,6 +2320,18 @@ func (c *featGeneratedContext) familyHasExistingRows(familyKey string) bool {
return false
}
// skillsDatasetName returns the dataset the skill_focus family sources its
// children from. The skills dataset location is data-driven via that spec;
// "skills" is only the fallback when no spec is loaded.
func (c *featGeneratedContext) skillsDatasetName() string {
for key, spec := range c.familySpecs {
if normalizeKeyIdentity(key) == "skillfocus" && spec.ChildSource.Dataset != "" {
return spec.ChildSource.Dataset
}
}
return "skills"
}
func (c *featGeneratedContext) datasetRows(name string) (map[string]map[string]any, error) {
if rows, ok := c.rowsByDataset[name]; ok {
return rows, nil
@@ -2555,7 +2570,10 @@ func buildFamilyExpansionGeneratedModule(path string, obj map[string]any, ctx *f
if !include {
continue
}
slug := strings.TrimPrefix(sourceKey, spec.ChildSource.Dataset+":")
slug := sourceKey
if idx := strings.Index(slug, ":"); idx >= 0 {
slug = slug[idx+1:]
}
featKey, rowID, hasID, err := ctx.resolveGeneratedFeatIdentity(spec, slug, row)
if err != nil {
return nil, fmt.Errorf("generated feat file %s: %w", path, err)
@@ -2655,7 +2673,7 @@ func buildRacialtypesSkillAffinityModule(ctx *featGeneratedContext) (map[string]
if len(grants) == 0 {
return nil, nil
}
skillRows, err := ctx.datasetRows("skills")
skillRows, err := ctx.datasetRows(ctx.skillsDatasetName())
if err != nil {
return nil, err
}
@@ -2854,7 +2872,7 @@ func (c *featGeneratedContext) displayNameForGeneratedSource(dataset string, row
return text
}
switch dataset {
case "skills":
case c.skillsDatasetName():
return displayNameForSkill(row)
case "baseitems":
return displayNameForBaseitem(row)
@@ -3457,10 +3475,20 @@ func legacyFamilyChildAliases(familyKey, sourceSlug string) []string {
aliases = append(aliases, "craftweapon")
case "craftwoodworking":
aliases = append(aliases, "crafttrap")
case "diplomacy":
aliases = append(aliases, "influence", "persuade")
case "disabledevice":
aliases = append(aliases, "disabletrap")
case "influence":
aliases = append(aliases, "persuade")
case "intimidate":
aliases = append(aliases, "taunt")
case "investigation":
aliases = append(aliases, "search")
case "medicine":
aliases = append(aliases, "heal")
case "security":
aliases = append(aliases, "openlock")
case "sleightofhand":
aliases = append(aliases, "pickpocket")
}
@@ -6617,81 +6645,6 @@ func marshalOrderedLockfile(keys []string, lockData map[string]int, formatting j
return buffer.Bytes(), nil
}
func loadLegacyTLK(root string) (*legacyTLKData, error) {
info, err := os.Stat(root)
if err != nil {
if os.IsNotExist(err) {
return nil, nil
}
return nil, err
}
if !info.IsDir() {
return nil, fmt.Errorf("legacy tlk root must be a directory: %s", root)
}
result := &legacyTLKData{
Language: "en",
Entries: map[string]tlkEntryData{},
Lock: map[string]int{},
}
lockPath := filepath.Join(root, "lock.json")
lockData, err := loadLockfile(lockPath)
if err != nil && !os.IsNotExist(err) {
return nil, err
}
for key, id := range lockData {
result.Lock[key] = id
}
basePath := filepath.Join(root, "base.json")
if _, err := os.Stat(basePath); err == nil {
base, err := loadJSONObject(basePath)
if err != nil {
return nil, err
}
if language, ok := base["language"].(string); ok && language != "" {
result.Language = language
}
if entries, ok := base["entries"].(map[string]any); ok {
normalized, err := normalizeLegacyTLKEntries(entries)
if err != nil {
return nil, err
}
for key, entry := range normalized {
result.Entries[key] = entry
}
}
}
modulesDir := filepath.Join(root, "modules")
modulePaths, err := collectModulePaths(modulesDir)
if err != nil {
if os.IsNotExist(err) {
return result, nil
}
return nil, err
}
for _, path := range modulePaths {
obj, err := loadJSONObject(path)
if err != nil {
return nil, err
}
entries, ok := obj["entries"].(map[string]any)
if !ok {
continue
}
normalized, err := normalizeLegacyTLKEntries(entries)
if err != nil {
return nil, fmt.Errorf("%s: %w", path, err)
}
for key, entry := range normalized {
result.Entries[key] = entry
}
}
return result, nil
}
func normalizeLegacyTLKEntries(entries map[string]any) (map[string]tlkEntryData, error) {
normalized := map[string]tlkEntryData{}
for _, key := range sortedKeys(entries) {
-6
View File
@@ -1,6 +0,0 @@
package topdata
func importLegacyPlaceables(referenceBuilderDir, dataDir string, legacyTLK *legacyTLKData) (int, error) {
_ = legacyTLK
return importLegacyDatasetMirror(referenceBuilderDir, dataDir, "placeables", "placeables.2da", nil)
}
-8
View File
@@ -1,8 +0,0 @@
package topdata
func importLegacyPortraits(referenceBuilderDir, dataDir string, legacyTLK *legacyTLKData) (int, error) {
if legacyTLK == nil {
return 0, nil
}
return importLegacyDatasetMirror(referenceBuilderDir, dataDir, "portraits", "portraits.2da", nil)
}
-6
View File
@@ -1,6 +0,0 @@
package topdata
func importLegacyProgfx(referenceBuilderDir, dataDir string, legacyTLK *legacyTLKData) (int, error) {
_ = legacyTLK
return importLegacyDatasetMirror(referenceBuilderDir, dataDir, "progfx", "progfx.2da", nil)
}
-282
View File
@@ -1,282 +0,0 @@
package topdata
import (
"encoding/json"
"fmt"
"os"
"path/filepath"
"slices"
"strings"
)
type legacyRacialtypesFeatTable struct {
Output string
Rows []map[string]any
}
func importLegacyRacialtypes(referenceBuilderDir, dataDir string, legacyTLK *legacyTLKData) (int, error) {
if strings.TrimSpace(referenceBuilderDir) == "" {
return 0, nil
}
legacyCoreDir := filepath.Join(referenceBuilderDir, "data", "racialtypes", "core")
legacyFeatsDir := filepath.Join(referenceBuilderDir, "data", "racialtypes", "feats")
if _, err := os.Stat(legacyCoreDir); err != nil {
if os.IsNotExist(err) {
return 0, nil
}
return 0, err
}
if _, err := os.Stat(legacyFeatsDir); err != nil {
if os.IsNotExist(err) {
return 0, nil
}
return 0, err
}
targetRoot := filepath.Join(dataDir, "racialtypes", "registry")
targetBasePath := filepath.Join(targetRoot, "base.json")
targetLockPath := filepath.Join(targetRoot, "lock.json")
targetRacesDir := filepath.Join(targetRoot, "races")
if fileExists(targetBasePath) && fileExists(targetLockPath) {
entries, err := os.ReadDir(targetRacesDir)
if err == nil {
raceFiles := 0
for _, entry := range entries {
if !entry.IsDir() && strings.HasSuffix(strings.ToLower(entry.Name()), ".json") {
raceFiles++
}
}
if raceFiles == 16 {
return 0, nil
}
}
}
baseColumns, baseRows, lockData, raceRows, err := collectLegacyRacialtypesRegistry(legacyCoreDir, legacyFeatsDir, legacyTLK)
if err != nil {
return 0, err
}
if err := removePathIfExists(filepath.Join(dataDir, "racialtypes", "core")); err != nil {
return 0, err
}
if err := removePathIfExists(filepath.Join(dataDir, "racialtypes", "feats")); err != nil {
return 0, err
}
if err := removePathIfExists(filepath.Join(targetRoot, "base_rows.json")); err != nil {
return 0, err
}
if err := removePathIfExists(targetRacesDir); err != nil {
return 0, err
}
if err := os.MkdirAll(targetRacesDir, 0o755); err != nil {
return 0, err
}
writes := []struct {
path string
obj any
}{
{
path: targetBasePath,
obj: map[string]any{
"columns": stringSliceToAny(baseColumns),
"rows": rowsToAny(baseRows),
},
},
{
path: targetLockPath,
obj: anyMapInt(lockData),
},
}
raceKeys := make([]string, 0, len(raceRows))
for key := range raceRows {
raceKeys = append(raceKeys, key)
}
slices.Sort(raceKeys)
for _, key := range raceKeys {
fileName := strings.TrimPrefix(key, "racialtypes:") + ".json"
writes = append(writes, struct {
path string
obj any
}{
path: filepath.Join(targetRacesDir, fileName),
obj: raceRows[key],
})
}
for _, write := range writes {
raw, err := json.MarshalIndent(write.obj, "", " ")
if err != nil {
return 0, err
}
raw = append(raw, '\n')
if err := os.WriteFile(write.path, raw, 0o644); err != nil {
return 0, err
}
}
return len(writes), nil
}
func collectLegacyRacialtypesRegistry(coreDir, featsDir string, legacyTLK *legacyTLKData) ([]string, []map[string]any, map[string]int, map[string]map[string]any, error) {
coreCollected, err := collectBaseDataset(nativeDataset{
Name: "racialtypes",
BasePath: filepath.Join(coreDir, "base.json"),
LockPath: filepath.Join(coreDir, "lock.json"),
ModulesDir: filepath.Join(coreDir, "modules"),
Spec: specForDataset("racialtypes"),
})
if err != nil {
return nil, nil, nil, nil, err
}
featTables, err := loadLegacyRacialtypesFeatTables(featsDir)
if err != nil {
return nil, nil, nil, nil, err
}
baseRows := make([]map[string]any, 0)
raceRows := map[string]map[string]any{}
lockData := map[string]int{}
for key, id := range coreCollected.LockData {
if strings.HasPrefix(key, "racialtypes:") {
lockData[key] = id
}
}
for _, collectedRow := range coreCollected.Rows {
row, ok := deepCopyValue(collectedRow).(map[string]any)
if !ok {
continue
}
if legacyTLK != nil {
if _, _, err := inlineLegacyTLKValue(row, legacyTLK); err != nil {
return nil, nil, nil, nil, err
}
}
key, _ := row["key"].(string)
if !strings.HasPrefix(key, "racialtypes:") {
assignRacialtypesBaseRowKey(row)
if key, _ := row["key"].(string); key != "" {
rowID, err := asInt(row["id"])
if err != nil {
return nil, nil, nil, nil, err
}
lockData[key] = rowID
}
baseRows = append(baseRows, row)
continue
}
delete(row, "id")
delete(row, "key")
raceFile := map[string]any{
"key": key,
"core": row,
}
if tableKey := extractRacialtypesFeatTableKey(row["FeatsTable"]); tableKey != "" {
table, ok := featTables[tableKey]
if !ok {
return nil, nil, nil, nil, fmt.Errorf("%s: unknown feat table %s", key, tableKey)
}
delete(row, "FeatsTable")
raceFile["feat_output"] = table.Output
raceFile["feats"] = rowsToAny(table.Rows)
}
raceRows[key] = raceFile
}
slices.SortFunc(baseRows, func(a, b map[string]any) int {
left, _ := asInt(a["id"])
right, _ := asInt(b["id"])
return left - right
})
return coreCollected.Columns, baseRows, lockData, raceRows, nil
}
func loadLegacyRacialtypesFeatTables(featsDir string) (map[string]legacyRacialtypesFeatTable, error) {
entries, err := os.ReadDir(featsDir)
if err != nil {
return nil, err
}
tables := map[string]legacyRacialtypesFeatTable{}
for _, entry := range entries {
if entry.IsDir() || filepath.Ext(entry.Name()) != ".json" {
continue
}
obj, err := loadJSONObject(filepath.Join(featsDir, entry.Name()))
if err != nil {
return nil, err
}
tableKey, _ := obj["key"].(string)
outputName, _ := obj["output"].(string)
rawRows, ok := obj["rows"].([]any)
if tableKey == "" || outputName == "" || !ok {
return nil, fmt.Errorf("%s: racialtypes feat table must define key, output, and rows", entry.Name())
}
rows := make([]map[string]any, 0, len(rawRows))
for index, raw := range rawRows {
row, ok := raw.(map[string]any)
if !ok {
return nil, fmt.Errorf("%s: row %d must be an object", entry.Name(), index)
}
rows = append(rows, row)
}
tables[tableKey] = legacyRacialtypesFeatTable{
Output: outputName,
Rows: rows,
}
}
return tables, nil
}
func extractRacialtypesFeatTableKey(value any) string {
obj, ok := value.(map[string]any)
if !ok {
return ""
}
tableKey, _ := obj["table"].(string)
return tableKey
}
func assignRacialtypesBaseRowKey(row map[string]any) {
name, _ := row["Name"].(string)
if strings.TrimSpace(name) == "" || name == nullValue {
delete(row, "key")
return
}
label, _ := row["Label"].(string)
keySuffix := normalizeRacialtypesKeySuffix(label)
if keySuffix == "" {
delete(row, "key")
return
}
row["key"] = "racialtypes:" + keySuffix
}
func normalizeRacialtypesKeySuffix(label string) string {
normalized := strings.ToLower(strings.TrimSpace(label))
if normalized == "" {
return ""
}
var b strings.Builder
lastUnderscore := false
for _, ch := range normalized {
isAlphaNum := (ch >= 'a' && ch <= 'z') || (ch >= '0' && ch <= '9')
if isAlphaNum {
b.WriteRune(ch)
lastUnderscore = false
continue
}
if lastUnderscore {
continue
}
b.WriteByte('_')
lastUnderscore = true
}
return strings.Trim(b.String(), "_")
}
-63
View File
@@ -1,63 +0,0 @@
package topdata
import (
"encoding/json"
"os"
"path/filepath"
)
func importLegacyRepadjust(referenceBuilderDir, dataDir string, legacyTLK *legacyTLKData) (int, error) {
_ = legacyTLK
legacyDir := filepath.Join(referenceBuilderDir, "data", "repadjust")
if _, err := os.Stat(legacyDir); err != nil {
if os.IsNotExist(err) {
return 0, nil
}
return 0, err
}
targetDir := filepath.Join(dataDir, "repadjust")
targetBasePath := filepath.Join(targetDir, "base.json")
targetLockPath := filepath.Join(targetDir, "lock.json")
if fileExists(targetBasePath) && fileExists(targetLockPath) {
obj, err := loadJSONObject(targetBasePath)
if err == nil && countLegacyTLKRefsInValue(obj) == 0 {
return 0, nil
}
}
baseObj, err := loadJSONObject(filepath.Join(legacyDir, "base.json"))
if err != nil {
return 0, err
}
baseObj["output"] = "repadjust.2da"
lockObj, err := loadJSONObject(filepath.Join(legacyDir, "lock.json"))
if err != nil {
return 0, err
}
if err := os.MkdirAll(targetDir, 0o755); err != nil {
return 0, err
}
writes := []struct {
path string
obj map[string]any
}{
{path: targetBasePath, obj: baseObj},
{path: targetLockPath, obj: lockObj},
}
for _, write := range writes {
raw, err := json.MarshalIndent(write.obj, "", " ")
if err != nil {
return 0, err
}
raw = append(raw, '\n')
if err := os.WriteFile(write.path, raw, 0o644); err != nil {
return 0, err
}
}
return len(writes), nil
}
-171
View File
@@ -1,171 +0,0 @@
package topdata
import (
"encoding/json"
"fmt"
"os"
"path/filepath"
"sort"
)
func importLegacyRuleset(referenceBuilderDir, dataDir string, legacyTLK *legacyTLKData) (int, error) {
_ = legacyTLK
legacyDir := filepath.Join(referenceBuilderDir, "data", "ruleset")
if _, err := os.Stat(legacyDir); err != nil {
if os.IsNotExist(err) {
return 0, nil
}
return 0, err
}
targetDir := filepath.Join(dataDir, "ruleset")
targetPath := filepath.Join(targetDir, "ruleset.json")
if _, err := os.Stat(targetPath); err == nil {
obj, err := loadJSONObject(targetPath)
if err == nil && countLegacyTLKRefsInValue(obj) == 0 {
return 0, nil
}
}
baseObj, err := loadJSONObject(filepath.Join(legacyDir, "base.json"))
if err != nil {
return 0, err
}
rows, err := mergeRulesetRows(baseObj, filepath.Join(legacyDir, "modules"))
if err != nil {
return 0, err
}
if err := os.MkdirAll(targetDir, 0o755); err != nil {
return 0, err
}
if err := removePathIfExists(filepath.Join(targetDir, "base.json")); err != nil {
return 0, err
}
if err := removePathIfExists(filepath.Join(targetDir, "lock.json")); err != nil {
return 0, err
}
if err := removePathIfExists(filepath.Join(targetDir, "modules")); err != nil {
return 0, err
}
out := map[string]any{
"output": "ruleset.2da",
"columns": baseObj["columns"],
"rows": rows,
}
raw, err := json.MarshalIndent(out, "", " ")
if err != nil {
return 0, err
}
raw = append(raw, '\n')
if err := os.WriteFile(targetPath, raw, 0o644); err != nil {
return 0, err
}
return 1, nil
}
func mergeRulesetRows(baseObj map[string]any, modulesDir string) ([]any, error) {
rawRows, ok := baseObj["rows"].([]any)
if !ok {
return nil, nil
}
rows := make([]map[string]any, 0, len(rawRows))
byLabel := map[string]map[string]any{}
for _, raw := range rawRows {
row, ok := deepCopyValue(raw).(map[string]any)
if !ok {
continue
}
delete(row, "key")
if rawID, ok := row["id"]; ok {
id, err := asInt(rawID)
if err != nil {
return nil, err
}
row["id"] = id
}
if label, ok := row["Label"].(string); ok && label != "" && label != "****" {
if _, exists := byLabel[label]; exists {
return nil, fmt.Errorf("duplicate ruleset label %q", label)
}
byLabel[label] = row
}
rows = append(rows, row)
}
modulePaths, err := collectModulePaths(modulesDir)
if err != nil {
return nil, err
}
for _, path := range modulePaths {
obj, err := loadJSONObject(path)
if err != nil {
return nil, err
}
overrideList, ok := obj["overrides"].([]any)
if !ok {
continue
}
for _, raw := range overrideList {
override, ok := raw.(map[string]any)
if !ok {
continue
}
matchObj, ok := override["match"].(map[string]any)
if !ok {
return nil, fmt.Errorf("%s: ruleset override is missing match object", path)
}
rawLabel, ok := matchObj["Label"]
if !ok {
return nil, fmt.Errorf("%s: ruleset override match is missing Label", path)
}
labels, err := normalizeRulesetMatchLabels(rawLabel)
if err != nil {
return nil, fmt.Errorf("%s: %w", path, err)
}
for _, label := range labels {
row, ok := byLabel[label]
if !ok {
return nil, fmt.Errorf("%s: ruleset override target label %q was not found", path, label)
}
for key, value := range override {
if key == "id" || key == "key" || key == "_tlk" || isMetadataField(key) || key == "match" {
continue
}
row[key] = deepCopyValue(value)
}
}
}
}
sort.Slice(rows, func(i, j int) bool {
return rows[i]["id"].(int) < rows[j]["id"].(int)
})
out := make([]any, 0, len(rows))
for _, row := range rows {
out = append(out, row)
}
return out, nil
}
func normalizeRulesetMatchLabels(raw any) ([]string, error) {
switch typed := raw.(type) {
case string:
return []string{typed}, nil
case []any:
out := make([]string, 0, len(typed))
for _, item := range typed {
label, ok := item.(string)
if !ok {
return nil, fmt.Errorf("ruleset match.Label entries must be strings")
}
out = append(out, label)
}
return out, nil
default:
return nil, fmt.Errorf("ruleset match.Label must be a string or string array")
}
}
-140
View File
@@ -1,140 +0,0 @@
package topdata
import (
"encoding/json"
"os"
"path/filepath"
)
func importLegacySkills(referenceBuilderDir, dataDir string, legacyTLK *legacyTLKData) (int, error) {
_ = legacyTLK
legacyDir := filepath.Join(referenceBuilderDir, "data", "skills")
if _, err := os.Stat(legacyDir); err != nil {
if os.IsNotExist(err) {
return 0, nil
}
return 0, err
}
targetDir := filepath.Join(dataDir, "skills")
targetBasePath := filepath.Join(targetDir, "base.json")
targetLockPath := filepath.Join(targetDir, "lock.json")
targetModulesDir := filepath.Join(targetDir, "modules")
moduleNames := []string{
"add_athletics.json",
"add_disguise.json",
"add_linguistics.json",
"add_perception.json",
"add_scriptcraft.json",
"add_sensemotive.json",
"add_stealth.json",
"add_survival.json",
"add_userope.json",
filepath.Join("craft", "add_craftalchemy.json"),
filepath.Join("craft", "add_craftcooking.json"),
filepath.Join("craft", "add_craftjewelry.json"),
filepath.Join("craft", "add_craftleatherworking.json"),
filepath.Join("craft", "add_craftstonework.json"),
filepath.Join("craft", "add_crafttextiles.json"),
filepath.Join("craft", "ovr_craftarmorsmithing.json"),
filepath.Join("craft", "ovr_crafttinkering.json"),
filepath.Join("craft", "ovr_craftweaponsmithing.json"),
filepath.Join("craft", "ovr_craftwoodworking.json"),
filepath.Join("knowledge", "add_knowledgearcana.json"),
filepath.Join("knowledge", "add_knowledgearchitecture.json"),
filepath.Join("knowledge", "add_knowledgedungeoneering.json"),
filepath.Join("knowledge", "add_knowledgegeography.json"),
filepath.Join("knowledge", "add_knowledgehistory.json"),
filepath.Join("knowledge", "add_knowledgelocal.json"),
filepath.Join("knowledge", "add_knowledgenature.json"),
filepath.Join("knowledge", "add_knowledgenobility.json"),
filepath.Join("knowledge", "add_knowledgeplanar.json"),
filepath.Join("knowledge", "add_knowledgereligion.json"),
"ovr_acrobatics.json",
"ovr_animalhandling.json",
"ovr_appraise.json",
"ovr_concentration.json",
"ovr_disabledevice.json",
"ovr_heal.json",
"ovr_hiddenskills.json",
"ovr_influence.json",
"ovr_openlock.json",
"ovr_parry.json",
"ovr_searchtowis.json",
"ovr_sleightofhand.json",
"ovr_taunttointimidate.json",
"rmv_removedskills.json",
}
targetModulePaths := make([]string, 0, len(moduleNames))
for _, name := range moduleNames {
targetModulePaths = append(targetModulePaths, filepath.Join(targetModulesDir, name))
}
if fileExists(targetBasePath) && fileExists(targetLockPath) {
allModulesPresent := true
for _, path := range targetModulePaths {
if !fileExists(path) {
allModulesPresent = false
break
}
}
if allModulesPresent {
return 0, nil
}
}
baseObj, err := loadJSONObject(filepath.Join(legacyDir, "base.json"))
if err != nil {
return 0, err
}
baseObj["output"] = "skills.2da"
canonicalizeWikiMetadataDocument(baseObj)
lockObj, err := loadJSONObject(filepath.Join(legacyDir, "lock.json"))
if err != nil {
return 0, err
}
moduleObjs := make([]map[string]any, 0, len(moduleNames))
for _, name := range moduleNames {
obj, err := loadJSONObject(filepath.Join(legacyDir, "modules", name))
if err != nil {
return 0, err
}
canonicalizeWikiMetadataDocument(obj)
moduleObjs = append(moduleObjs, obj)
}
if err := os.MkdirAll(targetModulesDir, 0o755); err != nil {
return 0, err
}
writes := []struct {
path string
obj map[string]any
}{
{path: targetBasePath, obj: baseObj},
{path: targetLockPath, obj: lockObj},
}
for i, path := range targetModulePaths {
if err := os.MkdirAll(filepath.Dir(path), 0o755); err != nil {
return 0, err
}
writes = append(writes, struct {
path string
obj map[string]any
}{path: path, obj: moduleObjs[i]})
}
for _, write := range writes {
raw, err := json.MarshalIndent(write.obj, "", " ")
if err != nil {
return 0, err
}
raw = append(raw, '\n')
if err := os.WriteFile(write.path, raw, 0o644); err != nil {
return 0, err
}
}
return len(writes), nil
}
-6
View File
@@ -1,6 +0,0 @@
package topdata
func importLegacySkyboxes(referenceBuilderDir, dataDir string, legacyTLK *legacyTLKData) (int, error) {
_ = legacyTLK
return importLegacyDatasetMirror(referenceBuilderDir, dataDir, "skyboxes", "skyboxes.2da", nil)
}
-103
View File
@@ -1,103 +0,0 @@
package topdata
import (
"encoding/json"
"os"
"path/filepath"
)
func importLegacySpells(referenceBuilderDir, dataDir string, legacyTLK *legacyTLKData) (int, error) {
_ = legacyTLK
legacyDir := filepath.Join(referenceBuilderDir, "data", "spells")
if _, err := os.Stat(legacyDir); err != nil {
if os.IsNotExist(err) {
return 0, nil
}
return 0, err
}
targetDir := filepath.Join(dataDir, "spells")
targetBasePath := filepath.Join(targetDir, "base.json")
targetLockPath := filepath.Join(targetDir, "lock.json")
targetModulesDir := filepath.Join(targetDir, "modules")
moduleNames := []string{
"specialattacks.json",
"ovr_fixduplicates.json",
filepath.Join("bardicmusic", "fascinate.json"),
filepath.Join("bardicmusic", "inspirecompetence.json"),
filepath.Join("bardicmusic", "inspirecourage.json"),
filepath.Join("bardicmusic", "inspiregreatness.json"),
filepath.Join("bardicmusic", "inspireheroics.json"),
filepath.Join("bardicmusic", "songoffreedom.json"),
}
targetModulePaths := make([]string, 0, len(moduleNames))
for _, name := range moduleNames {
targetModulePaths = append(targetModulePaths, filepath.Join(targetModulesDir, name))
}
if fileExists(targetBasePath) && fileExists(targetLockPath) {
allModulesPresent := true
for _, path := range targetModulePaths {
if !fileExists(path) {
allModulesPresent = false
break
}
}
if allModulesPresent {
return 0, nil
}
}
baseObj, err := loadJSONObject(filepath.Join(legacyDir, "base.json"))
if err != nil {
return 0, err
}
baseObj["output"] = "spells.2da"
lockObj, err := loadJSONObject(filepath.Join(legacyDir, "lock.json"))
if err != nil {
return 0, err
}
moduleObjs := make([]map[string]any, 0, len(moduleNames))
for _, name := range moduleNames {
obj, err := loadJSONObject(filepath.Join(legacyDir, "modules", name))
if err != nil {
return 0, err
}
moduleObjs = append(moduleObjs, obj)
}
if err := os.MkdirAll(targetModulesDir, 0o755); err != nil {
return 0, err
}
writes := []struct {
path string
obj map[string]any
}{
{path: targetBasePath, obj: baseObj},
{path: targetLockPath, obj: lockObj},
}
for i, path := range targetModulePaths {
if err := os.MkdirAll(filepath.Dir(path), 0o755); err != nil {
return 0, err
}
writes = append(writes, struct {
path string
obj map[string]any
}{path: path, obj: moduleObjs[i]})
}
for _, write := range writes {
raw, err := json.MarshalIndent(write.obj, "", " ")
if err != nil {
return 0, err
}
raw = append(raw, '\n')
if err := os.WriteFile(write.path, raw, 0o644); err != nil {
return 0, err
}
}
return len(writes), nil
}
-86
View File
@@ -1,86 +0,0 @@
package topdata
import (
"encoding/json"
"os"
"path/filepath"
)
func importLegacyTailmodel(referenceBuilderDir, dataDir string, legacyTLK *legacyTLKData) (int, error) {
_ = legacyTLK
legacyDir := filepath.Join(referenceBuilderDir, "data", "tailmodel")
if _, err := os.Stat(legacyDir); err != nil {
if os.IsNotExist(err) {
return 0, nil
}
return 0, err
}
targetDir := filepath.Join(dataDir, "tailmodel")
targetBasePath := filepath.Join(targetDir, "base.json")
targetLockPath := filepath.Join(targetDir, "lock.json")
targetModulesDir := filepath.Join(targetDir, "modules")
targetModulePaths := []string{
filepath.Join(targetModulesDir, "add.json"),
filepath.Join(targetModulesDir, "ovr_tailprefixes.json"),
}
if fileExists(targetBasePath) && fileExists(targetLockPath) {
allModulesPresent := true
for _, path := range targetModulePaths {
if !fileExists(path) {
allModulesPresent = false
break
}
}
if allModulesPresent {
return 0, nil
}
}
baseObj, err := loadJSONObject(filepath.Join(legacyDir, "base.json"))
if err != nil {
return 0, err
}
baseObj["output"] = "tailmodel.2da"
lockObj, err := loadJSONObject(filepath.Join(legacyDir, "lock.json"))
if err != nil {
return 0, err
}
moduleNames := []string{"add.json", "ovr_tailprefixes.json"}
moduleObjs := make([]map[string]any, 0, len(moduleNames))
for _, name := range moduleNames {
obj, err := loadJSONObject(filepath.Join(legacyDir, "modules", name))
if err != nil {
return 0, err
}
moduleObjs = append(moduleObjs, obj)
}
if err := os.MkdirAll(targetModulesDir, 0o755); err != nil {
return 0, err
}
writes := []struct {
path string
obj map[string]any
}{
{path: targetBasePath, obj: baseObj},
{path: targetLockPath, obj: lockObj},
{path: targetModulePaths[0], obj: moduleObjs[0]},
{path: targetModulePaths[1], obj: moduleObjs[1]},
}
for _, write := range writes {
raw, err := json.MarshalIndent(write.obj, "", " ")
if err != nil {
return 0, err
}
raw = append(raw, '\n')
if err := os.WriteFile(write.path, raw, 0o644); err != nil {
return 0, err
}
}
return len(writes), nil
}
+117 -3
View File
@@ -13,6 +13,7 @@ import (
"strings"
"golang.org/x/text/encoding/charmap"
"gopkg.in/yaml.v3"
)
const (
@@ -74,6 +75,16 @@ type tlkEntryData struct {
SoundLength float32
}
type standaloneTLKDocument struct {
Schema string `yaml:"schema"`
BaseStrref int `yaml:"base_strref"`
Strings []struct {
Key string `yaml:"key"`
Text string `yaml:"text"`
ID *int `yaml:"id"`
} `yaml:"strings"`
}
type tlkStateDocument struct {
Version int `json:"version"`
Language string `json:"language"`
@@ -98,6 +109,7 @@ type tlkCompiler struct {
active map[string]tlkEntryData
activeKeys map[string]struct{}
reservedByID map[int]string
pinnedByID map[int]string
nextID int
}
@@ -147,6 +159,7 @@ func newTLKCompiler(sourceDir string, legacy *legacyTLKData) (*tlkCompiler, erro
active: map[string]tlkEntryData{},
activeKeys: map[string]struct{}{},
reservedByID: reserved,
pinnedByID: map[int]string{},
nextID: nextID,
}
if legacy != nil {
@@ -157,6 +170,42 @@ func newTLKCompiler(sourceDir string, legacy *legacyTLKData) (*tlkCompiler, erro
return compiler, nil
}
func loadStandaloneTLKStrings(sourceDir string, compiler *tlkCompiler) error {
path := filepath.Join(sourceDir, "tlk", "custom.tlk.yml")
raw, err := os.ReadFile(path)
if err != nil {
if os.IsNotExist(err) {
return nil
}
return fmt.Errorf("read %s: %w", path, err)
}
var document standaloneTLKDocument
if err := yaml.Unmarshal(raw, &document); err != nil {
return fmt.Errorf("parse %s: %w", path, err)
}
if document.Schema != "sow-topdata/tlk/v1" {
return fmt.Errorf("%s: unsupported schema %q", path, document.Schema)
}
if document.BaseStrref != customTLKBase {
return fmt.Errorf("%s: base_strref must be %d", path, customTLKBase)
}
seen := map[string]struct{}{}
for index, entry := range document.Strings {
entry.Key = strings.TrimSpace(entry.Key)
if entry.Key == "" || entry.Text == "" || entry.ID == nil {
return fmt.Errorf("%s: strings[%d] requires key, text, and id", path, index)
}
if _, ok := seen[entry.Key]; ok {
return fmt.Errorf("%s: duplicate string key %q", path, entry.Key)
}
seen[entry.Key] = struct{}{}
if err := compiler.registerInlineAtID(entry.Key, *entry.ID, tlkEntryData{Text: entry.Text}); err != nil {
return err
}
}
return nil
}
func loadBaseDialogData(path string) (*legacyTLKData, error) {
if _, err := os.Stat(path); err != nil {
if os.IsNotExist(err) {
@@ -329,6 +378,33 @@ func (c *tlkCompiler) registerInline(key string, entry tlkEntryData) (tlkCompile
}, nil
}
func (c *tlkCompiler) registerInlineAtID(key string, id int, entry tlkEntryData) error {
if id < 0 {
return fmt.Errorf("TLK key %q has negative id %d", key, id)
}
// The pin is authoritative over the per-machine .tlk_state.json cache: a
// stale mapping that dynamically grabbed this id on an older build must
// yield so the pinned key can take it. Only a genuine clash between two
// pins in custom.tlk.yml is an author error.
if owner, ok := c.pinnedByID[id]; ok && owner != key {
return fmt.Errorf("TLK id %d is pinned by both %q and %q", id, owner, key)
}
if mapping, ok := c.state.Entries[key]; ok && mapping.ID != id {
// This key held a different cached id; release it so the pin wins.
if c.reservedByID[mapping.ID] == key {
delete(c.reservedByID, mapping.ID)
}
}
if owner, ok := c.reservedByID[id]; ok && owner != key {
// Evict the stale owner; it gets a fresh id when next made active.
delete(c.state.Entries, owner)
}
c.pinnedByID[id] = key
c.state.Entries[key] = tlkStateMapping{ID: id}
c.reservedByID[id] = key
return c.markActive(key, entry)
}
func (c *tlkCompiler) customStrrefForKey(key string) int {
return customTLKBase + c.state.Entries[key].ID
}
@@ -338,6 +414,7 @@ func (c *tlkCompiler) markActive(key string, entry tlkEntryData) error {
if !ok {
mapping = tlkStateMapping{ID: c.allocateID(), Retired: false}
c.state.Entries[key] = mapping
c.reservedByID[mapping.ID] = key
}
existing, ok := c.active[key]
if ok && existing != entry {
@@ -351,9 +428,13 @@ func (c *tlkCompiler) markActive(key string, entry tlkEntryData) error {
}
func (c *tlkCompiler) allocateID() int {
id := c.nextID
c.nextID++
return id
for {
id := c.nextID
c.nextID++
if _, reserved := c.reservedByID[id]; !reserved {
return id
}
}
}
func (c *tlkCompiler) finish(outputDir, tlkName string) (int, error) {
@@ -727,3 +808,36 @@ func sortedKeys[M ~map[string]V, V any](input M) []string {
func almostEqualFloat32(a, b float32) bool {
return math.Abs(float64(a-b)) < 0.0001
}
func mergeLegacyTLK(base, override *legacyTLKData) *legacyTLKData {
if base == nil && override == nil {
return nil
}
merged := &legacyTLKData{
Language: "en",
Entries: map[string]tlkEntryData{},
Lock: map[string]int{},
}
if base != nil {
if base.Language != "" {
merged.Language = base.Language
}
for key, entry := range base.Entries {
merged.Entries[key] = entry
}
for key, id := range base.Lock {
merged.Lock[key] = id
}
}
if override != nil {
if override.Language != "" {
merged.Language = override.Language
}
for key, entry := range override.Entries {
merged.Entries[key] = entry
}
for key, id := range override.Lock {
merged.Lock[key] = id
}
}
return merged
}
+31 -3
View File
@@ -2,6 +2,7 @@ package topdata
import (
"bytes"
"encoding/json"
"errors"
"fmt"
"io"
@@ -13,6 +14,7 @@ import (
"time"
"git.westgate.pw/ShadowsOverWestgate/sow-tools/internal/erf"
"git.westgate.pw/ShadowsOverWestgate/sow-tools/internal/gff"
"git.westgate.pw/ShadowsOverWestgate/sow-tools/internal/project"
)
@@ -215,10 +217,15 @@ func collectTopPackageResources(p *project.Project, compiled2DADir string) ([]er
if err != nil {
return err
}
if skipTopPackageAsset(rel) {
return nil
var resource erf.Resource
if strings.HasSuffix(strings.ToLower(rel), ".itp.json") {
resource, err = topPackageITPResourceFromJSON(path)
} else {
if skipTopPackageAsset(rel) {
return nil
}
resource, err = topPackageResourceFromPath(path)
}
resource, err := topPackageResourceFromPath(path)
if err != nil {
return err
}
@@ -251,6 +258,27 @@ func collectTopPackageResources(p *project.Project, compiled2DADir string) ([]er
return resources, assetFiles, nil
}
func topPackageITPResourceFromJSON(path string) (erf.Resource, error) {
raw, err := os.ReadFile(path)
if err != nil {
return erf.Resource{}, fmt.Errorf("read %s: %w", path, err)
}
var document gff.Document
if err := json.Unmarshal(raw, &document); err != nil {
return erf.Resource{}, fmt.Errorf("parse %s: %w", path, err)
}
if document.FileType != "ITP " {
return erf.Resource{}, fmt.Errorf("%s: file_type must be ITP", path)
}
var payload bytes.Buffer
if err := gff.Write(&payload, document); err != nil {
return erf.Resource{}, fmt.Errorf("compile %s: %w", path, err)
}
resourceType, _ := erf.HAKResourceTypeForExtension("itp")
name := strings.TrimSuffix(filepath.Base(path), ".itp.json")
return erf.Resource{Name: strings.ToLower(name), Type: resourceType, Data: payload.Bytes(), Size: int64(payload.Len())}, nil
}
func shouldSkipTopDataSourceDir(path string, skipDirs map[string]struct{}) bool {
_, ok := skipDirs[filepath.Clean(path)]
return ok
+9 -4
View File
@@ -1867,8 +1867,10 @@ func validateGeneratedFeatFamilies(dataDir string, report *ValidationReport) {
}
required := []requiredFeatFamily{
{FamilyKey: "skillfocus", Dataset: "skills", Predicate: "accessible"},
{FamilyKey: "greaterskillfocus", Dataset: "skills", Predicate: "accessible"},
// skill families: the source dataset is data-driven (family spec), only
// the accessibility predicate is required
{FamilyKey: "skillfocus", Predicate: "accessible"},
{FamilyKey: "greaterskillfocus", Predicate: "accessible"},
{FamilyKey: "weaponfocus", Dataset: "baseitems", Column: "WeaponFocusFeat"},
{FamilyKey: "weaponspecialization", Dataset: "baseitems", Column: "WeaponSpecializationFeat"},
{FamilyKey: "improvedcritical", Dataset: "baseitems", Column: "WeaponImprovedCriticalFeat"},
@@ -1882,7 +1884,7 @@ func validateGeneratedFeatFamilies(dataDir string, report *ValidationReport) {
if !ok {
continue
}
if spec.ChildSource.Dataset != requirement.Dataset ||
if (requirement.Dataset != "" && spec.ChildSource.Dataset != requirement.Dataset) ||
spec.ChildSource.Column != requirement.Column ||
spec.ChildSource.Predicate != requirement.Predicate {
report.Diagnostics = append(report.Diagnostics, Diagnostic{
@@ -2081,7 +2083,10 @@ func validateGeneratedFeatFamilyCompleteness(path string, spec familyExpansionSp
}
if include {
if familyAllowlist {
slug := strings.TrimPrefix(sourceKey, spec.ChildSource.Dataset+":")
slug := sourceKey
if idx := strings.Index(slug, ":"); idx >= 0 {
slug = slug[idx+1:]
}
featKey, _, _, err := ctx.resolveGeneratedFeatIdentity(spec, slug, row)
if err != nil {
report.Diagnostics = append(report.Diagnostics, Diagnostic{
File diff suppressed because it is too large Load Diff
-198
View File
@@ -1,198 +0,0 @@
package topdata
import (
"os"
"path/filepath"
"strings"
)
func importLegacyVFXPersistent(referenceBuilderDir, dataDir string, legacyTLK *legacyTLKData) (int, error) {
_ = legacyTLK
if strings.TrimSpace(referenceBuilderDir) == "" {
return 0, nil
}
legacyDir := filepath.Join(referenceBuilderDir, "data", "vfx_persistent")
if _, err := os.Stat(legacyDir); err != nil {
if os.IsNotExist(err) {
return 0, nil
}
return 0, err
}
targetDir := filepath.Join(dataDir, "vfx_persistent")
targetBasePath := filepath.Join(targetDir, "base.json")
targetLockPath := filepath.Join(targetDir, "lock.json")
targetModulesDir := filepath.Join(targetDir, "modules")
targetModulePath := filepath.Join(targetModulesDir, "freeformaoes.json")
if fileExists(targetBasePath) && fileExists(targetLockPath) && fileExists(targetModulePath) {
baseObj, baseErr := loadJSONObject(targetBasePath)
lockObj, lockErr := loadJSONObject(targetLockPath)
if baseErr == nil && lockErr == nil && vfxPersistentImportComplete(baseObj, lockObj) {
legacyModulePaths, err := collectJSONRelativePaths(filepath.Join(legacyDir, "modules"))
if err == nil {
targetModulePaths, err := collectJSONRelativePaths(targetModulesDir)
if err == nil && equalStringSlices(legacyModulePaths, targetModulePaths) {
return 0, nil
}
}
}
}
baseObj, err := loadJSONObject(filepath.Join(legacyDir, "base.json"))
if err != nil {
return 0, err
}
lockObj, err := synthesizeVFXPersistentKeys(baseObj)
if err != nil {
return 0, err
}
legacyLockObj, err := loadJSONObject(filepath.Join(legacyDir, "lock.json"))
if err != nil {
return 0, err
}
for key, value := range legacyLockObj {
lockObj[key] = value
}
if err := os.MkdirAll(targetModulesDir, 0o755); err != nil {
return 0, err
}
updated := 0
for _, write := range []struct {
path string
obj map[string]any
}{
{path: targetBasePath, obj: baseObj},
{path: targetLockPath, obj: lockObj},
} {
changed, err := writeJSONObjectIfChanged(write.path, write.obj)
if err != nil {
return 0, err
}
if changed {
updated++
}
}
moduleRelPaths, err := collectJSONRelativePaths(filepath.Join(legacyDir, "modules"))
if err != nil {
return 0, err
}
for _, relPath := range moduleRelPaths {
moduleObj, err := loadJSONObject(filepath.Join(legacyDir, "modules", filepath.FromSlash(relPath)))
if err != nil {
return 0, err
}
targetPath := filepath.Join(targetModulesDir, filepath.FromSlash(relPath))
if err := os.MkdirAll(filepath.Dir(targetPath), 0o755); err != nil {
return 0, err
}
changed, err := writeJSONObjectIfChanged(targetPath, moduleObj)
if err != nil {
return 0, err
}
if changed {
updated++
}
}
targetModulePaths, err := collectJSONRelativePaths(targetModulesDir)
if err != nil {
return 0, err
}
legacyModules := make(map[string]struct{}, len(moduleRelPaths))
for _, relPath := range moduleRelPaths {
legacyModules[relPath] = struct{}{}
}
for _, relPath := range targetModulePaths {
if _, ok := legacyModules[relPath]; ok {
continue
}
if err := os.Remove(filepath.Join(targetModulesDir, filepath.FromSlash(relPath))); err != nil && !os.IsNotExist(err) {
return 0, err
}
updated++
}
return updated, nil
}
func vfxPersistentImportComplete(baseObj, lockObj map[string]any) bool {
if !vfxPersistentRowsHaveKeys(baseObj) {
return false
}
expected := map[string]int{
"vfx_persistent:blankradius5": 47,
"vfx_persistent:blankradius10": 48,
"vfx_persistent:blankradius15": 49,
"vfx_persistent:blankradius20": 50,
"vfx_persistent:blankradius25": 51,
"vfx_persistent:blankradius30": 52,
}
for key, want := range expected {
raw, ok := lockObj[key]
if !ok {
return false
}
got, err := asInt(raw)
if err != nil || got != want {
return false
}
}
return true
}
func synthesizeVFXPersistentKeys(baseObj map[string]any) (map[string]any, error) {
rawRows, ok := baseObj["rows"].([]any)
if !ok {
return map[string]any{}, nil
}
lockObj := map[string]any{}
for _, raw := range rawRows {
row, ok := raw.(map[string]any)
if !ok {
continue
}
label, _ := row["LABEL"].(string)
if label == "" {
continue
}
key := "vfx_persistent:" + strings.ToLower(label)
row["key"] = key
if rawID, ok := row["id"]; ok {
lockObj[key] = rawID
}
}
return lockObj, nil
}
func vfxPersistentRowsHaveKeys(baseObj map[string]any) bool {
rawRows, ok := baseObj["rows"].([]any)
if !ok {
return false
}
for _, raw := range rawRows {
row, ok := raw.(map[string]any)
if !ok {
return false
}
key, _ := row["key"].(string)
if key == "" {
return false
}
}
return true
}
func equalStringSlices(left, right []string) bool {
if len(left) != len(right) {
return false
}
for i := range left {
if left[i] != right[i] {
return false
}
}
return true
}
@@ -1,6 +0,0 @@
package topdata
func importLegacyVisualeffects(referenceBuilderDir, dataDir string, legacyTLK *legacyTLKData) (int, error) {
_ = legacyTLK
return importLegacyDatasetMirror(referenceBuilderDir, dataDir, "visualeffects", "visualeffects.2da", nil)
}
+8 -6
View File
@@ -569,11 +569,13 @@ func loadWikiContext(dataDir, sourceDir string) (*wikiContext, error) {
if err != nil {
return nil, err
}
loadBase := func(name string) (nativeCollectedDataset, bool, error) {
for _, dataset := range datasets {
if dataset.Name == name {
collected, err := collectNativeDataset(dataset)
return collected, true, err
loadBase := func(names ...string) (nativeCollectedDataset, bool, error) {
for _, name := range names {
for _, dataset := range datasets {
if dataset.Name == name {
collected, err := collectNativeDataset(dataset)
return collected, true, err
}
}
}
return nativeCollectedDataset{}, false, nil
@@ -583,7 +585,7 @@ func loadWikiContext(dataDir, sourceDir string) (*wikiContext, error) {
if err != nil {
return nil, err
}
skillDataset, skillOK, err := loadBase("skills")
skillDataset, skillOK, err := loadBase("skills", "skills/core")
if err != nil {
return nil, err
}
-86
View File
@@ -1,86 +0,0 @@
package topdata
import (
"encoding/json"
"os"
"path/filepath"
)
func importLegacyWingmodel(referenceBuilderDir, dataDir string, legacyTLK *legacyTLKData) (int, error) {
_ = legacyTLK
legacyDir := filepath.Join(referenceBuilderDir, "data", "wingmodel")
if _, err := os.Stat(legacyDir); err != nil {
if os.IsNotExist(err) {
return 0, nil
}
return 0, err
}
targetDir := filepath.Join(dataDir, "wingmodel")
targetBasePath := filepath.Join(targetDir, "base.json")
targetLockPath := filepath.Join(targetDir, "lock.json")
targetModulesDir := filepath.Join(targetDir, "modules")
targetModulePaths := []string{
filepath.Join(targetModulesDir, "add.json"),
filepath.Join(targetModulesDir, "ovr_wingprefixes.json"),
}
if fileExists(targetBasePath) && fileExists(targetLockPath) {
allModulesPresent := true
for _, path := range targetModulePaths {
if !fileExists(path) {
allModulesPresent = false
break
}
}
if allModulesPresent {
return 0, nil
}
}
baseObj, err := loadJSONObject(filepath.Join(legacyDir, "base.json"))
if err != nil {
return 0, err
}
baseObj["output"] = "wingmodel.2da"
lockObj, err := loadJSONObject(filepath.Join(legacyDir, "lock.json"))
if err != nil {
return 0, err
}
moduleNames := []string{"add.json", "ovr_wingprefixes.json"}
moduleObjs := make([]map[string]any, 0, len(moduleNames))
for _, name := range moduleNames {
obj, err := loadJSONObject(filepath.Join(legacyDir, "modules", name))
if err != nil {
return 0, err
}
moduleObjs = append(moduleObjs, obj)
}
if err := os.MkdirAll(targetModulesDir, 0o755); err != nil {
return 0, err
}
writes := []struct {
path string
obj map[string]any
}{
{path: targetBasePath, obj: baseObj},
{path: targetLockPath, obj: lockObj},
{path: targetModulePaths[0], obj: moduleObjs[0]},
{path: targetModulePaths[1], obj: moduleObjs[1]},
}
for _, write := range writes {
raw, err := json.MarshalIndent(write.obj, "", " ")
if err != nil {
return 0, err
}
raw = append(raw, '\n')
if err := os.WriteFile(write.path, raw, 0o644); err != nil {
return 0, err
}
}
return len(writes), nil
}
+67
View File
@@ -0,0 +1,67 @@
#!/usr/bin/env bash
# Delete the binary assets of every release except the newest KEEP ones.
#
# Gitea has no asset retention, so 9 assets x ~57 MB per tag pile up forever on
# the host disk (sow-tools #52). Nothing consumes an old asset: CI takes the
# Crucible binary from the Nix flake, and every deleted file is reproducible
# from its tag. Tags, source archives, release notes and the releases
# themselves are never touched — only attachments.
#
# Nothing is excluded, SHA256SUMS and the wrappers included: the checksums are
# only meaningful next to the binaries they cover, and the drift-check reads
# the wrappers from the latest release, which always stays complete.
#
# Best-effort by design: a stale asset is cheaper than a blocked release, so
# every API failure is a warning, never a non-zero exit.
#
# Env:
# API repo API base, e.g. https://git.example/api/v1/repos/owner/repo [required]
# TOKEN Gitea token with releases:write [required]
# KEEP how many newest releases stay complete (default 2)
set -uo pipefail
: "${API:?set API}"
: "${TOKEN:?set TOKEN}"
keep="${KEEP:-2}"
auth="Authorization: token ${TOKEN}"
per_page=50
warn() { echo "prune-release-assets: $*" >&2; }
# Walk every page so an old backlog is cleared, not only the tag that fell out
# of the window on this run. Each page already embeds its releases' assets, so
# no follow-up request per release is needed.
releases='[]'
page=1
while :; do
body="$(curl -fsS -H "$auth" "${API}/releases?limit=${per_page}&page=${page}&draft=false")" || {
warn "listing releases failed on page ${page}; pruning what was listed so far"
break
}
count="$(jq 'length' <<<"$body")" || { warn "unreadable release page ${page}"; break; }
releases="$(jq -s 'add' <(printf '%s' "$releases") <(printf '%s' "$body"))"
# A short page is the last one; this also stops a server that ignores `page`.
(( count < per_page )) && break
page=$(( page + 1 ))
done
# Sort here rather than trusting the response order, and skip drafts in case
# the server ignored `draft=false` — a draft must not consume a keep slot and
# strip the binaries off the newest real release.
mapfile -t stale < <(jq -r --argjson keep "$keep" '
[.[] | select(.draft != true)]
| sort_by(.created_at) | reverse | .[$keep:]
| .[] | "\(.id) \(.assets // [] | map(.id) | join(","))"
' <<<"$releases")
(( ${#stale[@]} )) || { echo "prune-release-assets: nothing older than the newest ${keep} releases"; exit 0; }
for line in "${stale[@]}"; do
id="${line%% *}"
assets="${line#* }"
for asset in ${assets//,/ }; do
echo "deleting asset ${asset} of release ${id}"
curl -fsS -X DELETE -H "$auth" "${API}/releases/${id}/assets/${asset}" >/dev/null \
|| warn "deleting asset ${asset} of release ${id} failed"
done
done
+78
View File
@@ -0,0 +1,78 @@
#!/usr/bin/env bash
# Contract for scripts/prune-release-assets.sh, driven by a stub curl on PATH:
# only assets of releases outside the keep window go, the window is decided by
# release date rather than response order, drafts never consume a keep slot,
# releases and tags stay, and an API failure never fails the run.
set -euo pipefail
repo_root="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
script="$repo_root/scripts/prune-release-assets.sh"
tmp="$(mktemp -d)"
trap 'rm -rf "$tmp"' EXIT
# Stub curl: one short page of releases, deliberately out of date order and
# with a draft that is newer than every published release. Each release embeds
# two assets. Every request is appended to $CALLS for the assertions below.
mkdir -p "$tmp/bin"
cat >"$tmp/bin/curl" <<'STUB'
#!/usr/bin/env bash
url="${!#}"
printf '%s\n' "$*" >>"$CALLS"
rel() { printf '{"id":%s,"created_at":"%s","draft":%s,"assets":[{"id":%s01},{"id":%s02}]}' "$1" "$2" "$3" "$1" "$1"; }
case "$url" in
*releases\?*)
[[ "${FAIL_LIST:-0}" == 1 ]] && exit 22
printf '[%s,%s,%s,%s,%s]\n' \
"$(rel 12 2024-01-02T00:00:00Z false)" \
"$(rel 10 2024-01-04T00:00:00Z false)" \
"$(rel 9 2024-06-01T00:00:00Z true)" \
"$(rel 13 2024-01-01T00:00:00Z false)" \
"$(rel 11 2024-01-03T00:00:00Z false)"
;;
*/assets/*) [[ "${FAIL_DELETE:-0}" == 1 ]] && exit 22 ;;
esac
exit 0
STUB
chmod +x "$tmp/bin/curl"
export PATH="$tmp/bin:$PATH" API=https://git.example/api/v1/repos/o/r TOKEN=t
run() { CALLS="$tmp/calls" bash "$script" >"$tmp/out" 2>"$tmp/err"; }
# Default keep=2: the two newest published releases (10, 11) stay whole, the
# older two (12, 13) lose their assets, and the newer draft is ignored.
: >"$tmp/calls"; run
deletes="$(grep -c -- '-X DELETE' "$tmp/calls" || true)"
[[ "$deletes" == 4 ]] || { echo "expected 4 asset deletes, got $deletes" >&2; exit 1; }
grep -q 'assets/1201' "$tmp/calls" && grep -q 'assets/1302' "$tmp/calls" || {
echo "expected assets of releases 12 and 13 to be deleted" >&2; exit 1; }
if grep -q 'releases/10/assets/\|releases/11/assets/' "$tmp/calls"; then
echo "kept releases lost assets" >&2; exit 1
fi
if grep -q 'releases/9/assets/' "$tmp/calls"; then
echo "a draft release was pruned" >&2; exit 1
fi
# A release or tag must never be deleted, only attachments under /assets/.
if grep -- '-X DELETE' "$tmp/calls" | grep -qv '/assets/'; then
echo "a non-asset DELETE was issued" >&2; exit 1
fi
# A short page ends the sweep: no second page, so a server ignoring `page`
# cannot spin the job until the job timeout.
[[ "$(grep -c 'releases?' "$tmp/calls")" == 1 ]] || {
echo "a short release page did not end the sweep" >&2; exit 1; }
# KEEP=4 covers every published release: nothing to delete.
: >"$tmp/calls"; KEEP=4 run
if grep -q -- '-X DELETE' "$tmp/calls"; then echo "KEEP=4 still deleted" >&2; exit 1; fi
# Failures are warnings, never a non-zero exit.
: >"$tmp/calls"; FAIL_DELETE=1 run || { echo "delete failure failed the run" >&2; exit 1; }
grep -q 'failed' "$tmp/err" || { echo "delete failure was not warned about" >&2; exit 1; }
: >"$tmp/calls"; FAIL_LIST=1 run || { echo "listing failure failed the run" >&2; exit 1; }
grep -q 'listing releases failed' "$tmp/err" || {
echo "listing failure was not warned about" >&2; exit 1; }
if grep -q -- '-X DELETE' "$tmp/calls"; then
echo "deleted assets despite an unreadable release list" >&2; exit 1
fi
echo "prune-release-assets: prunes by release date outside the keep window, ignores drafts, never deletes a release, never fails the build"
+86
View File
@@ -0,0 +1,86 @@
#!/usr/bin/env bash
set -euo pipefail
contract_script="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)/$(basename "${BASH_SOURCE[0]}")"
default_root="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
repo_root="${WORKFLOW_ROOT:-$default_root}"
cd "$repo_root"
ci=.gitea/workflows/ci.yml
release=.gitea/workflows/build-binaries.yml
sync=.gitea/workflows/sync-wrappers.yml
checkout='actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683'
assert_exact_events() {
local workflow="$1" expected="$2" actual
actual="$(awk '
/^on:$/ { in_on = 1 }
in_on && /^$/ { next }
in_on && $0 != "on:" && $0 !~ /^[[:space:]]/ { exit }
in_on { print }
' "$workflow")"
[[ "$actual" == "$expected" ]] || {
echo "workflow-contract: unexpected event scope in $workflow" >&2
return 1
}
}
[[ -f "$ci" ]] || { echo "missing consolidated CI workflow" >&2; exit 1; }
[[ ! -e .gitea/workflows/test.yml ]] || { echo "legacy test workflow still present" >&2; exit 1; }
assert_exact_events "$ci" $'on:\n pull_request:'
grep -Fqx 'permissions: read-all' "$ci"
grep -Fqx ' timeout-minutes: 60' "$ci"
[[ "$(grep -Fc "$checkout" "$ci")" -eq 1 ]]
grep -Fqx ' fetch-depth: 0' "$ci"
grep -Fq 'go vet ./...' "$ci"
grep -Fq 'go test ./...' "$ci"
grep -Fq 'shellcheck scripts/*.sh' "$ci"
grep -Fq 'yamllint .gitea' "$ci"
grep -Fq 'make smoke' "$ci"
grep -Fq 'for target in linux/amd64 linux/arm64 darwin/amd64 darwin/arm64 windows/amd64 windows/arm64; do' "$ci"
assert_exact_events "$release" $'on:\n push:\n tags: [\'v*\']'
grep -Fqx ' code: read' "$release"
grep -Fqx ' releases: write' "$release"
grep -Fqx ' timeout-minutes: 30' "$release"
[[ "$(grep -Fc "$checkout" "$release")" -eq 1 ]]
grep -Fq 'secrets.GITEA_TOKEN' "$release"
grep -Fq 'scripts/prune-release-assets.sh' "$release"
grep -Fq 'continue-on-error: true' "$release"
if grep -Fq 'secrets.GITHUB_TOKEN' "$release"; then
echo "workflow-contract: release uses the GitHub token alias" >&2
exit 1
fi
grep -Fqx 'permissions: read-all' "$sync"
grep -Fqx ' timeout-minutes: 30' "$sync"
[[ "$(grep -Fc "$checkout" "$sync")" -eq 1 ]]
assert_exact_events "$sync" $'on:\n push:\n branches: [main]\n paths:\n - \'wrappers/crucible.sh\'\n - \'wrappers/crucible.ps1\''
expect_mutation_rejected() {
local name="$1" workflow="$2" expression="$3" tmp
tmp="$(mktemp -d)"
mkdir -p "$tmp/.gitea"
cp -R "$default_root/.gitea/workflows" "$tmp/.gitea/workflows"
sed -i "$expression" "$tmp/$workflow"
if WORKFLOW_ROOT="$tmp" WORKFLOW_CONTRACT_MUTATION=1 bash "$contract_script" >/dev/null 2>&1; then
echo "workflow-contract: accepted forbidden mutation: $name" >&2
rm -rf "$tmp"
return 1
fi
rm -rf "$tmp"
}
if [[ "${WORKFLOW_CONTRACT_MUTATION:-0}" != 1 ]]; then
mutations_ok=0
expect_mutation_rejected 'CI workflow_dispatch event' "$ci" '/^ pull_request:$/a\ workflow_dispatch:' || mutations_ok=1
expect_mutation_rejected 'release develop branch' "$release" "/^ tags:/a\\ branches: [develop]" || mutations_ok=1
expect_mutation_rejected 'release schedule event' "$release" "/^ tags:/a\\ schedule:\n - cron: '0 0 * * *'" || mutations_ok=1
expect_mutation_rejected 'release extra tag pattern' "$release" "s/tags: \['v\*'\]/tags: ['v*', 'release-*']/" || mutations_ok=1
expect_mutation_rejected 'wrapper sync pull_request event' "$sync" '/^ push:$/i\ pull_request:' || mutations_ok=1
expect_mutation_rejected 'wrapper sync broad path' "$sync" "/^ - 'wrappers\/crucible.ps1'$/a\\ - 'wrappers/**'" || mutations_ok=1
(( mutations_ok == 0 )) || exit 1
fi
echo "workflow-contract: CI is PR-only; release and wrapper sync retain their narrow triggers"