Compare commits

...
8 Commits
Author SHA1 Message Date
archvillainette 4d03085996 feat(ci): prune release assets outside the newest two tags (#52) (#70)
Closes #52.

Adds `scripts/prune-release-assets.sh` and a best-effort step at the end of `build-binaries.yml`.

- Lists every release page, sorts by creation date, skips the newest two published releases, deletes the rest's attachments via `DELETE /releases/{id}/assets/{asset_id}`.
- Drafts never consume a keep slot, so a draft cannot strip the newest real release.
- Releases, tags, source archives and release notes are never touched.
- Nothing is excluded: SHA256SUMS and the wrappers go with the binaries. Checksums are only meaningful next to the files they cover, and the consumer drift-check reads the wrappers from the latest release, which stays complete.
- Failures warn and the step has `continue-on-error: true` — a stale asset is cheaper than a blocked publish.

`tests/prune-release-assets.sh` drives the script with a stub `curl`: keep window, out-of-order responses, drafts, no non-asset DELETE, short-page pagination stop, and both failure paths. Wired into `make check`.

v0.2.0 still needs its 9 assets cleared by hand in the web UI, as the issue notes.

🤖 Generated with [Claude Code](https://claude.com/claude-code)Reviewed-on: #70

Co-authored-by: vickydotbat <vickydotbat@tutamail.com>
2026-07-29 08:51:25 +00:00
archvillainette 2f4685e470 fix(erf): align restype numbers with upstream neverwinter.nim (#69)
Closes #64.

Five entries in `internal/erf/erf.go`'s restype table disagreed with upstream neverwinter.nim (`neverwinter/restype.nim`). `HAKResourceTypeForExtension` packs arbitrary source files, so a `.jpg` packed into a HAK today is written as restype 2076, which the game reads as `tml`.

| ext | was | now |
|-----|-----|-----|
| gff | 2039 (upstream `bte`) | 2037 |
| ltr | 2067 (upstream `bak`) | 2036 |
| jpg | 2076 (upstream `tml`) | 2081 |
| dfa | 2045, name typo | `dft`, 2045 (number was already right) |
| mdb | 2070 (upstream `xbc`) | removed |

Upstream registers no restype for `mdb`, so there is no correct number to give it, and leaving it squatting on `xbc` silently mislabels the resource. It is dropped from `AssetExtensions` too, which turns a `.mdb` source into a loud "unsupported extension" build error instead of a corrupt HAK entry. **This is the one judgment call the issue left open** — if `mdb` should instead stay as a deliberate local addition like `lyt`/`vis`/`gr2`, say so and I will move it to a free number in the 0x0BB8+ local range.

No asset in the current corpus uses any of these paths, so nothing in shipped content changes.

The six local additions upstream does not have (`lyt` 3000, `vis` 3001, `mdx` 3008, `wlk` 3020, `xml` 3021, `gr2` 4003) are left alone and now carry a comment saying they are deliberate.

## Root cause

The `init()` consistency guard only panicked when a number was missing from the reverse map. Its `if canonicalExt == ext { continue }` branch did nothing when the two maps disagreed, so a mismatch was never caught. The guard now panics on disagreement and also checks the reverse direction.

## Tests

- `TestRestypeTableMatchesUpstream` pins every number shared with upstream, and asserts the four numbers upstream reserves for other extensions (2039 `bte`, 2067 `bak`, 2070 `xbc`, 2076 `tml`) stay unclaimed.
- `TestRestypeTablesAreMutualInverses` is the check `init()` is meant to enforce.

`make check` passes.Reviewed-on: #69

Co-authored-by: vickydotbat <vickydotbat@tutamail.com>
2026-07-28 22:18:03 +00:00
archvillainette 00f467b932 docs(agents): note that tea reads stdin and hangs without </dev/null (#68)
`tea comment` reads stdin to EOF and appends whatever it finds to the comment body. In a non-interactive shell — which is every agent — stdin is an open pipe that never sends EOF, so the call hangs forever instead of posting.

Verified on tea 0.14.0: with `</dev/null` it posts instantly; with an open pipe it blocks until killed; with `echo "x" | tea comment N "y"` it posts `y` followed by `x`.

Documents the redirect in the tracker guide, and applies it to the wayfinder resolve step. Same trap exists on `tea issues create --description` and `tea pr create`.

🤖 Generated with [Claude Code](https://claude.com/claude-code)Reviewed-on: #68

Co-authored-by: vickydotbat <vickydotbat@tutamail.com>
2026-07-28 13:18:38 +00:00
archvillainette 22eecd1a41 depot: replace --target local with --out DIR (#67)
Closes #66.

`depot status` and `depot get` chose a depot tree on disk with `--target local`, taking the root from `DEPOT_DIR`. One decision, two flags — and `--target local` with no directory anywhere was valid but meaningless.

Now they take `--out DIR`, matching the crucible nwsync surface. `--target` names remote backends only (`bunny|cdn`). Passing both exits 64.

- `--target local` stays as a deprecated alias for `--out $DEPOT_DIR`, listed in the hidden-alias table in `docs/command-surface.md`. Nothing in `wrappers/` calls depot, but the three repos in `wrappers/consumers.txt` are out of reach from here, so the alias stays.
- `status --source` was already unused; it now warns "ignored; use --out DIR" instead of accepting silently.
- Fixes a real parse bug found on the way: the documented `depot get <sha> <dest> --target cdn` form never parsed its flags, because Go's `flag` package stops at the first positional. Positionals are split off by hand now; both orders are tested.
- Registry usage strings and `docs/command-surface.md` synced — that doc still said depot was unwired.

Not converted: `push --source` and `pull --dest`. Neither is `--target local`; one names a read source, the other a remote pull's destination. Say the word if they should become `--out` too.

`go test ./...` green.

🤖 Generated with [Claude Code](https://claude.com/claude-code)Reviewed-on: #67

Co-authored-by: vickydotbat <vickydotbat@tutamail.com>
2026-07-28 11:07:58 +00:00
archvillainette ed6945d308 Adopt workspace-wide agent standards; retire sow-docs (#51)
Part of the workspace-wide standards rollout. Establishes where work lives (issues vs ADRs vs standing law), scaffolds `docs/agents/` for the engineering skills, and adopts ADR-0001 locally. See `sow-platform` ADR-0021 for the `sow-docs` retirement.Reviewed-on: #51
Reviewed-by: xtul <mpiasecki720@protonmail.com>
Co-authored-by: vickydotbat <vickydotbat@tutamail.com>
2026-07-24 18:43:31 +00:00
archvillainette 3f500dabc8 Generate palette projections at module build (#50)
build-binaries / build-binaries (push) Successful in 2m13s
*palcus.itp files are Toolset-derived snapshots (category skeleton + blueprint descriptors). This makes Crucible the generator: module builds strip stale descriptors from the committed skeletons and inject descriptors computed from the source-tree blueprints (NAME/STRREF + RESREF, plus CR/FACTION for creatures, faction names resolved via repute.fac). Compare applies the same projection; extract never writes *palcus.itp back into source, so local Toolset palette mess stays local.

Companion PR in sow-module strips the committed palcus files down to skeletons.

🤖 Generated with [Claude Code](https://claude.com/claude-code)Reviewed-on: #50

Co-authored-by: vickydotbat <vickydotbat@tutamail.com>
2026-07-23 09:35:45 +00:00
archvillainette 8e7cead5c0 Keep model compilation headless (#49)
build-binaries / build-binaries (push) Successful in 2m14s
## Summary

- always wrap NWN model compilation with `xvfb-run`, even when the caller has `DISPLAY`
- fail closed with an actionable error when `xvfb-run` is unavailable instead of opening the client UI
- update the compiler contract and regression coverage

## Verification

- focused red/green regression tests
- `nix develop -c make check`
- real NWN compile with `DISPLAY=:0` under transient Xvfb; binary MDL output verified

Generated with Claude CodeReviewed-on: #49

Co-authored-by: vickydotbat <vickydotbat@tutamail.com>
2026-07-23 07:27:37 +00:00
archvillainette 4d38967078 fix(topdata): let pinned TLK ids evict stale state owners (#48)
build-binaries / build-binaries (push) Successful in 2m14s
## Problem

The custom palette taxonomy (#46) pins display strings to fixed TLK ids in `tlk/custom.tlk.yml`, and `registerInlineAtID` demanded each pinned id be free. But `.tlk_state.json` is **gitignored and per-machine** — each dev grows their own copy, and before #46 every ref got its id dynamically (first-come-first-served).

On any machine whose state predates #46, a ref could have already parked on a now-pinned id. That fails the build:

```
topdata validation failed with 1 error(s): error: native topdata buildability check failed:
TLK id 2689 is already reserved by "feat:yuanti/alternate_form.feat"
```

It only passes on machines whose state was regenerated after #46 (pinned window already clean). The `allocateID` reserved-skip that #46 added protects a *fresh* build, but does nothing about a *stale cache*.

## Fix

Make the pin authoritative over the per-machine cache. A stale cached owner sitting on a pinned id is evicted and reallocated a fresh id when next made active; only two pins fighting over the same id in `custom.tlk.yml` is now an error. This self-heals on the next build — no manual `.tlk_state.json` deletion needed.

Caveat: the evicted ref's strref shifts on affected machines. That's unavoidable (something must move off the pinned id), and dynamic strrefs were never stable across machines anyway.

## Tests

Added `TestBuildStandaloneTLKPinEvictsStaleStateOwner` (seeds a pre-#46 state with a feat parked on the pinned id, asserts the build succeeds, the pin owns it, and the feat is reallocated). Full `internal/topdata` suite + `go vet` pass.

🤖 Generated with [Claude Code](https://claude.com/claude-code)Reviewed-on: #48
Reviewed-by: xtul <mpiasecki720@protonmail.com>
Co-authored-by: vickydotbat <vickydotbat@tutamail.com>
2026-07-22 18:17:04 +00:00
31 changed files with 1359 additions and 72 deletions
+14
View File
@@ -65,3 +65,17 @@ jobs:
"${api}/releases/${id}/assets?name=$(basename "$f")" "${api}/releases/${id}/assets?name=$(basename "$f")"
done done
' '
# Gitea has no asset retention (#52): without this every tag keeps its
# ~57 MB binary set forever. Best-effort — a stale asset is cheaper than
# a blocked publish, so a failure here never fails the release.
- name: Prune assets of older releases
continue-on-error: true
env:
TOKEN: ${{ secrets.GITEA_TOKEN }}
REPO: ${{ github.repository }}
SERVER: ${{ github.server_url }}
run: |
nix develop --command bash -c '
API="${SERVER}/api/v1/repos/${REPO}" scripts/prune-release-assets.sh
'
+18
View File
@@ -87,3 +87,21 @@ make smoke # assert fail-closed contract
## Tests ## Tests
Tests must survive harmless changes to constants, defaults, wording, ordering, fixture data, and internal implementation details. A test that fails merely because a basic value changed is usually a bad test. Only assert exact values when the value is part of a documented public contract, external protocol, compatibility requirement, security rule, migration, or business rule. Tests must survive harmless changes to constants, defaults, wording, ordering, fixture data, and internal implementation details. A test that fails merely because a basic value changed is usually a bad test. Only assert exact values when the value is part of a documented public contract, external protocol, compatibility requirement, security rule, migration, or business rule.
## Agent skills
### Issue tracker
Issues live in Gitea at git.westgate.pw (`ShadowsOverWestgate/sow-tools`), managed with the `tea` CLI. Issues follow ownership — file work in the repo that owns it, not the one you happen to be standing in. See `docs/agents/issue-tracker.md`.
### Triage labels
Default label vocabulary (`needs-triage`, `needs-info`, `ready-for-agent`, `ready-for-human`, `wontfix`). See `docs/agents/triage-labels.md`.
### Domain docs
Single-context: `CONTEXT.md` at the repo root plus `docs/adr/`. See `docs/agents/domain.md`.
### Where work lives
Markdown here is **reference, law, or an ADR — nothing else** (`sow-codebase` ADR-0001). Live work -> wayfinder maps + Gitea issues (closeable, assignable, queryable). Settled decisions -> ADR files in `docs/adr/` (immutable, never closed, only superseded). Standing law -> `DOCTRINE.md` / `AGENTS.md` / `CONTEXT.md`. Current-state reference -> docs describing what the code does now. Everything else — plans, specs, concepts, handoffs, trackers — is process: it belongs in a Gitea issue, not a file. Harvest unfinished intent to an issue before deleting a process doc. `sow-docs` is deprecated and read-only.
+1
View File
@@ -7,6 +7,7 @@ check: vet test
shellcheck scripts/*.sh shellcheck scripts/*.sh
yamllint .gitea yamllint .gitea
bash tests/workflow-contract.sh bash tests/workflow-contract.sh
bash tests/prune-release-assets.sh
vet: vet:
go vet ./... go vet ./...
+3 -1
View File
@@ -85,7 +85,9 @@ https://git.westgate.pw/ShadowsOverWestgate/sow-docs).
- `ci.yml` — vet, test, shellcheck, yamllint, binary smoke, and cross-build all - `ci.yml` — vet, test, shellcheck, yamllint, binary smoke, and cross-build all
targets once per pull request. targets once per pull request.
- `build-binaries.yml` — on a `v*` tag, cross-build and upload the binaries, - `build-binaries.yml` — on a `v*` tag, cross-build and upload the binaries,
`SHA256SUMS`, and the wrappers to the Gitea release. `SHA256SUMS`, and the wrappers to the Gitea release, then delete the assets
of every release except the newest two — Gitea keeps them forever otherwise,
and every binary is reproducible from its tag.
- `sync-wrappers.yml` — on a `main` push that touches `wrappers/`, auto-PR the - `sync-wrappers.yml` — on a `main` push that touches `wrappers/`, auto-PR the
canonical wrappers to the consumer repos in `wrappers/consumers.txt`. canonical wrappers to the consumer repos in `wrappers/consumers.txt`.
Consumer drift checks run after those PRs merge to `main`, not on the PRs Consumer drift checks run after those PRs merge to `main`, not on the PRs
@@ -0,0 +1,55 @@
# ADR-0001: Markdown in this repo is reference, law, or an ADR — nothing else
- **Status:** Accepted
- **Date:** 2026-07-24
- **Origin:** Adopted workspace-wide from `sow-codebase` ADR-0001, which records
the full context (a `docs/` tree that had grown to 434 files, ~27MB, most of
it dead process artifacts). This repo adopts the same law so the rule is
local, not a cross-repo reference.
## Context
Markdown that *claims to describe current state or a plan* rots, because
reality diverges and nobody edits the file. Markdown that claims neither — a
dated, immutable decision record, or a standing rule — does not rot.
Two different things get conflated:
- **Rationale** — why a system is shaped the way it is. Worth keeping.
- **Process artifacts** — plans, specs, concepts, handoffs, trackers. A record
of how work happened, not what is true now.
Left unchecked, every completed effort leaves its planning behind and the repo
accumulates a "historical" pile that agents and humans must route around to
find the few live docs.
## Decision
Markdown may exist in this repo only as one of three genres:
1. **Current-state reference** — describes what the code does now, kept honest
by code review touching it.
2. **Standing law**`DOCTRINE.md`, root and folder-scoped `AGENTS.md`,
`CONTEXT.md`. States rules and vocabulary, not plans.
3. **Immutable decision record** — ADRs under `docs/adr/`. Append-only; never
edited, only superseded by a later ADR that points back.
Anything else — implementation plans, design specs, concepts, handoffs,
progress trackers, scratch — is **process** and does not live in the repo. It
lives in Gitea issues and wayfinder maps, where it can be assigned, closed, and
superseded. Small tasks need no written plan at all.
Deleting a process document is not destroying history: `git log -- <path>`
recovers it. The exception is *unfinished intent* (a design never built, an
open question still wanted) — that is live, not history, and must be harvested
to a Gitea ticket before its file is deleted. A citation from a living doc or
from code must be resolved before the cited file is deleted.
## Consequences
- The documentation map lists only current truth; there is no "historical" pile
to route around.
- No agent, under any plugin or skill, writes process-markdown into the repo.
The rule is plugin-agnostic on purpose — it binds the agent, not a named tool.
- History of deleted process docs is in git; unfinished intent is in the issue
tracker; rationale is in ADRs and law. Each thing has exactly one home.
@@ -0,0 +1,12 @@
# ADR-0002: `sow-tools` becomes the Crucible suite
- **Status:** Accepted
- **Date:** 2026-06-11
- **Migrated from:** `sow-docs` `07-decisions-log.md` entry **D11**, on the
retirement of `sow-docs`. Content is the original decision, unchanged.
- **Decision:** Rename the future toolchain surface to Crucible: one Go module,
multiple `cmd/` binaries, plus a dispatcher so wrapper commands can stay
stable.
- **Rationale:** Depot, HAK, module, topdata, and wiki tooling share internals
but have different command surfaces. One module avoids premature repo splits.
@@ -0,0 +1,29 @@
# ADR-0003: Crucible binary contract: standalone shims, fail-closed scaffold, no committed binaries
- **Status:** Accepted
- **Date:** 2026-06-11
- **Migrated from:** `sow-docs` `07-decisions-log.md` entry **D19**, on the
retirement of `sow-docs`. Content is the original decision, unchanged.
- **Decision:** Phase 5 builds `migration/sow-tools` (Crucible, D11) as a
multi-binary Go scaffold:
- One `crucible` dispatcher plus standalone `crucible-{depot,hak,module,topdata,wiki}`
binaries sharing one registry (`internal/dispatch`). The dispatcher is for
humans/CI; the **standalone shims are canonical for consumers** so wrapper
scripts resolve a single-token command and `"$builder" args` quoting stays
correct.
- Consumer resolution order: explicit env override
(`$SOW_MODULE_BUILD`/`$SOW_TOPDATA_BUILD`/`$CRUCIBLE`) → `crucible-<name>`
legacy `sow-<name>-build`. CI runs inside the pinned `crucible:<sha>` image.
- Every builder is **unwired** in the scaffold and fails closed (exit 70);
it never fakes an artifact. The `internal/` logic is migrated from
`gitea/sow-tools` by the operator at cutover (hard rule: no source
transplant by tooling).
- **Binaries are never committed** — they are CI artifacts / image layers.
Retires the committed `nwn-tool` / `tools/sow-toolkit`.
- Builders take `NWN_ROOT` only via explicit env/flag; no `$HOME` defaulting.
- **Rationale:** Locks the command surface, container, and CI shape so migrated
logic drops into a stable frame; aligns the three artifact repos onto one
Crucible contract while keeping back-compat with the pre-D11 skeletons.
- **Image name:** `registry.westgate.pw/deployment/crucible:<git-sha>` (the bare
`crucible:<sha>` is the local/dev tag).
@@ -0,0 +1,24 @@
# ADR-0004: Nix-built OCI images start server-side with Crucible; local Anvil images deferred
- **Status:** Accepted
- **Date:** 2026-06-12
- **Migrated from:** `sow-docs` `07-decisions-log.md` entry **D29**, on the
retirement of `sow-docs`. Content is the original decision, unchanged.
- **Decision:** Introduce Nix-built OCI images as an artifact-production option,
starting with `sow-tools`/Crucible in server-side CI. `sow-tools` will grow a
Nix-built `crucible` package plus `crucible-image`; PR CI builds/smokes it and
`main` publishes the normal pinned OCI image
`registry.westgate.pw/deployment/crucible:<sha>`. `docker/Dockerfile` stays during
parity and as the client-compatible fallback.
- **Local rule:** Nix users may get optional wrappers that provide tools and
call the existing Dockerfiles for local Anvil/NWServer testing. A true
`nix build .#nwserver-test-image` is explicitly deferred until real
`sow-codebase/src` exists and the Dockerfile image is proven.
- **Rationale:** Server-side image build shape is harder to change once deploy
promotion and registry gates are active, so prove Nix image builds before
cutover. Crucible is the lowest-risk pilot because it is a Go toolchain image;
NodeBB and Anvil/NWServer depend more heavily on upstream container filesystem
semantics.
- **Non-goals:** no source builds in `sow-platform`; no `nix-sidecar` or
Kubernetes-style runtime cache layer; no removal of client Dockerfiles.
+51
View File
@@ -0,0 +1,51 @@
# Domain Docs
How the engineering skills should consume this repo's domain documentation when exploring the codebase.
## Before exploring, read these
- **`CONTEXT.md`** at the repo root, or
- **`CONTEXT-MAP.md`** at the repo root if it exists — it points at one `CONTEXT.md` per context. Read each one relevant to the topic.
- **`docs/adr/`** — read ADRs that touch the area you're about to work in. In multi-context repos, also check `src/<context>/docs/adr/` for context-scoped decisions.
If any of these files don't exist, **proceed silently**. Don't flag their absence; don't suggest creating them upfront. The `/domain-modeling` skill (reached via `/grill-with-docs` and `/improve-codebase-architecture`) creates them lazily when terms or decisions actually get resolved.
## File structure
Single-context repo (most repos):
```
/
├── CONTEXT.md
├── docs/adr/
│ ├── 0001-event-sourced-orders.md
│ └── 0002-postgres-for-write-model.md
└── src/
```
Multi-context repo (presence of `CONTEXT-MAP.md` at the root):
```
/
├── CONTEXT-MAP.md
├── docs/adr/ ← system-wide decisions
└── src/
├── ordering/
│ ├── CONTEXT.md
│ └── docs/adr/ ← context-specific decisions
└── billing/
├── CONTEXT.md
└── docs/adr/
```
## Use the glossary's vocabulary
When your output names a domain concept (in an issue title, a refactor proposal, a hypothesis, a test name), use the term as defined in `CONTEXT.md`. Don't drift to synonyms the glossary explicitly avoids.
If the concept you need isn't in the glossary yet, that's a signal — either you're inventing language the project doesn't use (reconsider) or there's a real gap (note it for `/domain-modeling`).
## Flag ADR conflicts
If your output contradicts an existing ADR, surface it explicitly rather than silently overriding:
> _Contradicts ADR-0007 (event-sourced orders) — but worth reopening because…_
+88
View File
@@ -0,0 +1,88 @@
# Issue tracker: Gitea (via tea)
Issues for this repo live in Gitea at `git.westgate.pw`, repo
`ShadowsOverWestgate/sow-tools`. Use the `tea` CLI for all operations —
`gh` does not work here. For anything `tea` lacks a subcommand for, use
`tea api <endpoint>` (Gitea's API mirrors GitHub's closely).
Authenticate with your own `tea` login (`tea login add`); never commit tokens
or tea config into this repo. Note Gitea blocks self-review, so approving a PR
needs a different account than the one that opened it.
## Where work lives
Markdown in this repo is **reference, law, or an ADR — nothing else**
(`sow-codebase` ADR-0001). Four homes, no overlap:
- **Live work** → wayfinder maps + Gitea issues. Closeable, assignable,
queryable. Never a markdown file.
- **Settled decisions** → ADR files in `docs/adr/`. Immutable, findable, never
closed, never edited — only superseded by a later ADR pointing back. When an
issue ends in a durable decision, write the ADR, then close the issue
pointing at it. Decisions spanning repos go to `sow-platform/docs/adr/`.
- **Standing law** → `DOCTRINE.md`, `AGENTS.md`, `CONTEXT.md`. Rules that are
always true and vocabulary everyone shares — not the record of one decision.
- **Current-state reference** → docs describing what the code does now, kept
honest by review touching them.
Anything else — implementation plans, design specs, concepts, handoffs,
progress trackers, scratch — is **process**. It does not live in this repo. It
lives in a Gitea issue or a wayfinder map, where it can be assigned, closed,
and superseded. Small tasks need no written plan at all.
Deleting a process doc is not destroying history — `git log -- <path>` recovers
it. But *unfinished intent* (a design never built, an open question still
wanted) is live, not history: harvest it to a Gitea issue before deleting.
`sow-docs` is deprecated and read-only. Never add to it, never send work there.
## Which repo gets the issue
Issues follow ownership. File the issue in the repo that **owns the work**
see the Repo/Owns/Produces table in the workspace root `AGENTS.md`. Standing
in one repo is not a reason to file there.
If work spans repos, file it in the repo that owns the *outcome* and reference
the others from it. A wrong-repo issue is a routing bug, not a filing
preference — move it.
## Conventions
- **Create an issue**: `tea issues create --title "..." --description "..."`
- **Read an issue**: `tea issues <number>` and
`tea api repos/ShadowsOverWestgate/sow-tools/issues/<number>/comments` for comments.
- **List issues**: `tea issues list --state open` (add `--labels ...` to filter).
- **Comment**: `tea comment <number> "..." </dev/null`
Always redirect stdin. `tea` reads stdin to EOF and appends it to the body,
so any non-interactive shell (every agent) hangs forever without
`</dev/null`. Same trap on `tea issues create --description` and
`tea pr create`.
- **Apply / remove labels**: `tea api --method PATCH` on the issue, or
`tea api repos/ShadowsOverWestgate/sow-tools/issues/<number>/labels` endpoints.
- **Close**: `tea issues close <number>`
`tea` infers the repo from the git remote when run inside the clone.
Gitea shares one number space across issues and PRs.
## Pull requests as a triage surface
**PRs as a request surface: no.**
## When a skill says "publish to the issue tracker"
Create a Gitea issue with `tea issues create`.
## When a skill says "fetch the relevant ticket"
Run `tea issues <number>` plus the comments API call above.
## Wayfinding operations
Used by `/wayfinder`. The **map** is a single issue with **child** issues as tickets.
- **Map**: a single issue labelled `wayfinder:map`, holding the Notes / Decisions-so-far / Fog body. `tea issues create --title "..." --description "..." --labels wayfinder:map`.
- **Child ticket**: this Gitea instance (v1.27) has no native sub-issue hierarchy, so a child issue carries `Part of #<map>` at the top of its description, and is also added to a task list in the map body. Labels: `wayfinder:<type>` (`research`/`prototype`/`grilling`/`task`). Once claimed, the ticket is assigned to the driving dev.
- **Blocking**: Gitea's **native dependencies API** — the canonical, UI-visible representation (shows as "Depends on" / "Blocks" on the issue page). Add an edge with `tea api -X POST repos/ShadowsOverWestgate/sow-tools/issues/<child>/dependencies -f owner=ShadowsOverWestgate -f repo=sow-tools -F index=<blocker>`, where `<blocker>` is the blocker's issue **index** (its `#number` — Gitea's dependency API takes the index directly, unlike GitHub's numeric database id). Check status with `tea api repos/ShadowsOverWestgate/sow-tools/issues/<child>/dependencies` (GET) — a ticket is unblocked when every returned issue's `state` is `closed`.
- **Frontier query**: list the map's open children (`tea issues list --state open`, keep the ones whose description contains `Part of #<map>`), drop any with an open dependency (per the GET above) or an assignee; first in map order wins.
- **Claim**: `tea issues edit <n> --add-assignees <username>` — the session's first write.
- **Resolve**: `tea comment <n> "<answer>" </dev/null`, then `tea issues close <n>`, then append a context pointer (gist + link) to the map's Decisions-so-far.
+15
View File
@@ -0,0 +1,15 @@
# Triage Labels
The skills speak in terms of five canonical triage roles. This file maps those roles to the actual label strings used in this repo's issue tracker (Gitea — see `issue-tracker.md` for how to apply labels with `tea`).
| Label in mattpocock/skills | Label in our tracker | Meaning |
| -------------------------- | -------------------- | ---------------------------------------- |
| `needs-triage` | `needs-triage` | Maintainer needs to evaluate this issue |
| `needs-info` | `needs-info` | Waiting on reporter for more information |
| `ready-for-agent` | `ready-for-agent` | Fully specified, ready for an AFK agent |
| `ready-for-human` | `ready-for-human` | Requires human implementation |
| `wontfix` | `wontfix` | Will not be actioned |
When a skill mentions a role (e.g. "apply the AFK-ready triage label"), use the corresponding label string from this table.
Edit the right-hand column to match whatever vocabulary you actually use.
+9 -2
View File
@@ -29,9 +29,15 @@ aliases.
| `assets` | `fix-mdl` | Lowercase `.mdl` names and rewrite model identity to match. | | `assets` | `fix-mdl` | Lowercase `.mdl` names and rewrite model identity to match. |
| `assets` | `check-dupes` | Report runtime-name (basename) collisions across dirs. | | `assets` | `check-dupes` | Report runtime-name (basename) collisions across dirs. |
| `assets` | `clean-dupes` | Delete clean-tree files whose basename collides with primary. | | `assets` | `clean-dupes` | Delete clean-tree files whose basename collides with primary. |
| `depot` | `status` | Report referenced-vs-present drift against a backend. |
| `depot` | `push` | Upload referenced-but-absent blobs from a local depot. |
| `depot` | `verify` | Existence sweep plus sampled download re-hash. |
| `depot` | `get` | Fetch one blob with sha re-verify. |
| `depot` | `pull` | Incremental verified pull of every referenced blob. |
`crucible-depot` remains registered but unwired. It fails closed with exit `70` `depot status` and `depot get` pick their backend either with `--out DIR`, a
and never emits placeholder artifacts. depot tree on disk, or with `--target bunny|cdn`, a remote backend. The two
flags are mutually exclusive.
## Hidden compatibility aliases ## Hidden compatibility aliases
@@ -49,6 +55,7 @@ but omitted from routine help and the interactive menu:
| `topdata` | `build-top-package` | `build-top-package` | | `topdata` | `build-top-package` | `build-top-package` |
| `topdata` | `compare-topdata` | `compare-topdata` | | `topdata` | `compare-topdata` | `compare-topdata` |
| `topdata` | `convert-topdata` | `convert-topdata` | | `topdata` | `convert-topdata` | `convert-topdata` |
| `depot` | `--target local` | `--out $DEPOT_DIR` on `status`/`get` |
| `wiki` | `build-wiki` | `build-wiki` | | `wiki` | `build-wiki` | `build-wiki` |
| `wiki` | `deploy-wiki` | `deploy-wiki` | | `wiki` | `deploy-wiki` | `deploy-wiki` |
@@ -126,8 +126,9 @@ exactly as the reference does:
- Beamdog install dirs. - Beamdog install dirs.
- `--nwn <install>` overrides (path to the install root or directly to the - `--nwn <install>` overrides (path to the install root or directly to the
binary). binary).
- **Headless.** The engine is a GUI binary. If there is no `DISPLAY` and - **Headless.** The engine is a GUI binary. Require `xvfb-run` and wrap the call
`xvfb-run` is present, wrap the call: regardless of the caller's `DISPLAY` so compilation never opens the client
UI; fail before invoking the engine when `xvfb-run` is unavailable:
`xvfb-run -a --server-args=-screen 0 1024x768x24 nwmain-linux compilemodel <stem>`. `xvfb-run -a --server-args=-screen 0 1024x768x24 nwmain-linux compilemodel <stem>`.
- **Per model (one at a time — the engine's `development/` and `modelcompiler/` - **Per model (one at a time — the engine's `development/` and `modelcompiler/`
folders are flat and single-slot):** folders are flat and single-slot):**
+6 -6
View File
@@ -40,13 +40,13 @@ func runCompile(args []string, stdout, stderr io.Writer, getenv func(string) str
} }
binDir := filepath.Dir(nwmain) binDir := filepath.Dir(nwmain)
// Headless wrap: no DISPLAY + xvfb-run present -> run under a virtual X. // Always use a virtual X so compilation never opens the client UI.
var wrap []string xvfb := look("xvfb-run")
if getenv("DISPLAY") == "" { if xvfb == "" {
if xvfb := look("xvfb-run"); xvfb != "" { fmt.Fprintln(stderr, "assets compile: xvfb-run not found — install it to run the NWN model compiler headlessly")
wrap = []string{xvfb, "-a", "--server-args=-screen 0 1024x768x24"} return exitTool
}
} }
wrap := []string{xvfb, "-a", "--server-args=-screen 0 1024x768x24"}
files, err := walk(dirs, mdlExt, !*nonRecursive) files, err := walk(dirs, mdlExt, !*nonRecursive)
if err != nil { if err != nil {
+63 -2
View File
@@ -4,6 +4,7 @@ import (
"bytes" "bytes"
"os" "os"
"path/filepath" "path/filepath"
"strings"
"testing" "testing"
) )
@@ -23,13 +24,19 @@ func TestCompileDrivesEngineAndReplacesInPlace(t *testing.T) {
if err := os.WriteFile(nwmain, []byte("#!/bin/sh\n"), 0o755); err != nil { if err := os.WriteFile(nwmain, []byte("#!/bin/sh\n"), 0o755); err != nil {
t.Fatal(err) t.Fatal(err)
} }
xvfbDir := t.TempDir()
xvfb := filepath.Join(xvfbDir, "xvfb-run")
if err := os.WriteFile(xvfb, []byte("#!/bin/sh\n"), 0o755); err != nil {
t.Fatal(err)
}
t.Setenv("PATH", xvfbDir+string(os.PathListSeparator)+os.Getenv("PATH"))
getenv := func(k string) string { getenv := func(k string) string {
switch k { switch k {
case "HOME": case "HOME":
return home return home
case "DISPLAY": case "DISPLAY":
return ":0" // pretend a display exists so no xvfb wrap is needed return ":0" // a desktop display must not make compilation interactive
} }
return "" return ""
} }
@@ -38,7 +45,11 @@ func TestCompileDrivesEngineAndReplacesInPlace(t *testing.T) {
orig := runner orig := runner
defer func() { runner = orig }() defer func() { runner = orig }()
runner = func(dir string, env []string, name string, args ...string) ([]byte, error) { runner = func(dir string, env []string, name string, args ...string) ([]byte, error) {
// args: compilemodel <stem> if name != xvfb || len(args) != 5 || args[0] != "-a" ||
args[1] != "--server-args=-screen 0 1024x768x24" || args[2] != nwmain ||
args[3] != "compilemodel" {
t.Fatalf("engine command = %q %q, want xvfb-run wrapping nwmain", name, args)
}
stem := args[len(args)-1] stem := args[len(args)-1]
compiled := filepath.Join(mc, stem+".mdl") compiled := filepath.Join(mc, stem+".mdl")
return nil, os.WriteFile(compiled, []byte("\x00\x00compiled"), 0o644) return nil, os.WriteFile(compiled, []byte("\x00\x00compiled"), 0o644)
@@ -77,6 +88,11 @@ func TestCompileAbortsOnNameMismatch(t *testing.T) {
if err := os.WriteFile(nwmain, []byte("#!/bin/sh\n"), 0o755); err != nil { if err := os.WriteFile(nwmain, []byte("#!/bin/sh\n"), 0o755); err != nil {
t.Fatal(err) t.Fatal(err)
} }
xvfbDir := t.TempDir()
if err := os.WriteFile(filepath.Join(xvfbDir, "xvfb-run"), []byte("#!/bin/sh\n"), 0o755); err != nil {
t.Fatal(err)
}
t.Setenv("PATH", xvfbDir+string(os.PathListSeparator)+os.Getenv("PATH"))
getenv := func(k string) string { getenv := func(k string) string {
if k == "HOME" { if k == "HOME" {
return home return home
@@ -108,3 +124,48 @@ func TestCompileAbortsOnNameMismatch(t *testing.T) {
t.Fatal("engine should not run for a name-mismatched model") t.Fatal("engine should not run for a name-mismatched model")
} }
} }
func TestCompileFailsClosedWithoutXvfb(t *testing.T) {
home := t.TempDir()
userData := filepath.Join(home, ".local", "share", "Neverwinter Nights")
for _, d := range []string{filepath.Join(userData, "development"), filepath.Join(userData, "modelcompiler")} {
if err := os.MkdirAll(d, 0o755); err != nil {
t.Fatal(err)
}
}
nwmain := filepath.Join(t.TempDir(), "nwmain-linux")
if err := os.WriteFile(nwmain, []byte("#!/bin/sh\n"), 0o755); err != nil {
t.Fatal(err)
}
t.Setenv("PATH", t.TempDir())
getenv := func(k string) string {
if k == "HOME" {
return home
}
return ""
}
orig := runner
defer func() { runner = orig }()
engineCalled := false
runner = func(string, []string, string, ...string) ([]byte, error) {
engineCalled = true
return nil, nil
}
srcDir := t.TempDir()
if err := os.WriteFile(filepath.Join(srcDir, "foo.mdl"),
[]byte("newmodel foo\nbeginmodelgeom foo\n node dummy foo\n parent null\n endnode\nendmodelgeom foo\ndonemodel foo\n"), 0o644); err != nil {
t.Fatal(err)
}
var stdout, stderr bytes.Buffer
if code := runCompile([]string{"--nwn", nwmain, srcDir}, &stdout, &stderr, getenv); code != exitTool {
t.Fatalf("compile exit = %d, want %d\n%s", code, exitTool, stderr.String())
}
if engineCalled {
t.Fatal("engine must not run without xvfb-run")
}
if !strings.Contains(stderr.String(), "xvfb-run") {
t.Fatalf("missing actionable xvfb-run error: %s", stderr.String())
}
}
+43 -25
View File
@@ -11,6 +11,7 @@ import (
"os" "os"
"path/filepath" "path/filepath"
"sort" "sort"
"strings"
"sync" "sync"
"time" "time"
) )
@@ -51,13 +52,15 @@ func Run(args []string, stdout, stderr io.Writer, getenv func(string) string) in
func printRunUsage(w io.Writer) { func printRunUsage(w io.Writer) {
fmt.Fprint(w, `usage: fmt.Fprint(w, `usage:
depot status [--manifests DIR] [--source LOCAL_DEPOT] --target bunny|cdn|local depot status [--manifests DIR] --target bunny|cdn | --out DIR
depot push [--manifests DIR] --source LOCAL_DEPOT --target bunny depot push [--manifests DIR] --source LOCAL_DEPOT --target bunny
depot verify [--manifests DIR] --target bunny|cdn [--sample N] depot verify [--manifests DIR] --target bunny|cdn [--sample N]
depot get <sha> <dest> --target cdn|bunny|local depot get <sha> <dest> --target cdn|bunny | --out DIR
depot pull [--manifests DIR] --dest DIR --target cdn|bunny depot pull [--manifests DIR] --dest DIR --target cdn|bunny
--target local uses the DEPOT_DIR environment variable as the local root. --out DIR reads and writes a depot tree on disk at DIR instead of a remote
backend. --target names a remote backend only; the two are mutually exclusive.
--target local is a deprecated alias for --out $DEPOT_DIR.
`) `)
} }
@@ -65,16 +68,27 @@ func printRunUsage(w io.Writer) {
// than internal/backend failures (exit 70). // than internal/backend failures (exit 70).
var errUsage = errors.New("usage error") var errUsage = errors.New("usage error")
// resolveBackend builds the named backend, resolving "local" against DEPOT_DIR. // resolveBackend picks the backend for a command that can work either against a
func resolveBackend(target string, getenv func(string) string, cfg Config) (Backend, error) { // depot tree on disk (--out DIR) or a remote backend (--target bunny|cdn).
root := "" // "--target local" stays as a deprecated alias for --out $DEPOT_DIR so older
if target == "local" { // callers keep working.
root = getenv("DEPOT_DIR") func resolveBackend(out, target string, getenv func(string) string, cfg Config) (Backend, error) {
switch {
case out != "" && target != "":
return nil, fmt.Errorf("--out and --target are mutually exclusive: %w", errUsage)
case out != "":
return NewBackend("local", out, cfg)
case target == "local":
root := getenv("DEPOT_DIR")
if root == "" { if root == "" {
return nil, fmt.Errorf("--target local requires DEPOT_DIR to be set: %w", errUsage) return nil, fmt.Errorf("--target local requires DEPOT_DIR to be set (prefer --out DIR): %w", errUsage)
} }
return NewBackend("local", root, cfg)
case target == "":
return nil, fmt.Errorf("--out DIR or --target bunny|cdn is required: %w", errUsage)
default:
return NewBackend(target, "", cfg)
} }
return NewBackend(target, root, cfg)
} }
// backendErrExit maps a resolveBackend error onto the exit contract. // backendErrExit maps a resolveBackend error onto the exit contract.
@@ -119,18 +133,18 @@ func runStatus(args []string, stdout, stderr io.Writer, getenv func(string) stri
fs := flag.NewFlagSet("status", flag.ContinueOnError) fs := flag.NewFlagSet("status", flag.ContinueOnError)
fs.SetOutput(stderr) fs.SetOutput(stderr)
manifests := fs.String("manifests", "assets", "directory of *.yml manifests") manifests := fs.String("manifests", "assets", "directory of *.yml manifests")
_ = fs.String("source", "", "local depot root (unused for status target=local; see DEPOT_DIR)") source := fs.String("source", "", "deprecated and ignored (use --out DIR for a depot tree on disk)")
target := fs.String("target", "", "bunny|cdn|local") out := fs.String("out", "", "depot tree on disk to read instead of a remote backend")
target := fs.String("target", "", "bunny|cdn")
if err := fs.Parse(args); err != nil { if err := fs.Parse(args); err != nil {
return exitUsage return exitUsage
} }
if *target == "" { if *source != "" {
fmt.Fprintln(stderr, "depot status: --target is required") fmt.Fprintln(stderr, "depot status: --source is ignored; use --out DIR")
return exitUsage
} }
cfg := LoadConfig(getenv) cfg := LoadConfig(getenv)
backend, err := resolveBackend(*target, getenv, cfg) backend, err := resolveBackend(*out, *target, getenv, cfg)
if err != nil { if err != nil {
fmt.Fprintln(stderr, "depot status:", err) fmt.Fprintln(stderr, "depot status:", err)
return backendErrExit(err) return backendErrExit(err)
@@ -309,13 +323,21 @@ func runVerify(args []string, stdout, stderr io.Writer, getenv func(string) stri
func runGet(args []string, stdout, stderr io.Writer, getenv func(string) string) int { func runGet(args []string, stdout, stderr io.Writer, getenv func(string) string) int {
fs := flag.NewFlagSet("get", flag.ContinueOnError) fs := flag.NewFlagSet("get", flag.ContinueOnError)
fs.SetOutput(stderr) fs.SetOutput(stderr)
target := fs.String("target", "", "cdn|bunny|local") out := fs.String("out", "", "depot tree on disk to read instead of a remote backend")
if err := fs.Parse(args); err != nil { target := fs.String("target", "", "cdn|bunny")
// Positionals come first in the documented usage, and Go's flag package
// stops parsing at the first one, so split them off by hand.
split := 0
for split < len(args) && !strings.HasPrefix(args[split], "-") {
split++
}
positional := args[:split]
if err := fs.Parse(args[split:]); err != nil {
return exitUsage return exitUsage
} }
positional := fs.Args() positional = append(positional, fs.Args()...)
if len(positional) != 2 { if len(positional) != 2 {
fmt.Fprintln(stderr, "depot get: usage: depot get <sha> <dest> --target cdn|bunny|local") fmt.Fprintln(stderr, "depot get: usage: depot get <sha> <dest> --target cdn|bunny | --out DIR")
return exitUsage return exitUsage
} }
sha, dest := positional[0], positional[1] sha, dest := positional[0], positional[1]
@@ -323,13 +345,9 @@ func runGet(args []string, stdout, stderr io.Writer, getenv func(string) string)
fmt.Fprintf(stderr, "depot get: invalid sha %q\n", sha) fmt.Fprintf(stderr, "depot get: invalid sha %q\n", sha)
return exitUsage return exitUsage
} }
if *target == "" {
fmt.Fprintln(stderr, "depot get: --target is required")
return exitUsage
}
cfg := LoadConfig(getenv) cfg := LoadConfig(getenv)
backend, err := resolveBackend(*target, getenv, cfg) backend, err := resolveBackend(*out, *target, getenv, cfg)
if err != nil { if err != nil {
fmt.Fprintln(stderr, "depot get:", err) fmt.Fprintln(stderr, "depot get:", err)
return backendErrExit(err) return backendErrExit(err)
+96
View File
@@ -54,6 +54,91 @@ func TestRunUsage(t *testing.T) {
}) })
} }
func TestOutFlag(t *testing.T) {
t.Run("status --out reads the given tree", func(t *testing.T) {
sha := shaOf("out-blob")
manifests := t.TempDir()
writeManifest(t, manifests, sha, 8)
depotDir := t.TempDir()
writeBlob(t, depotDir, sha, "out-blob")
var out, errb bytes.Buffer
code := Run([]string{"status", "--manifests", manifests, "--out", depotDir}, &out, &errb, testGetenv(nil))
if code != 0 {
t.Fatalf("expected 0, got %d (stdout=%s stderr=%s)", code, out.String(), errb.String())
}
if !bytesContains(out.String(), "present=1") {
t.Fatalf("expected present=1, got %s", out.String())
}
})
t.Run("get --out fetches from the given tree", func(t *testing.T) {
sha := shaOf("get-blob")
depotDir := t.TempDir()
writeBlob(t, depotDir, sha, "get-blob")
dest := filepath.Join(t.TempDir(), "fetched")
var out, errb bytes.Buffer
code := Run([]string{"get", sha, dest, "--out", depotDir}, &out, &errb, testGetenv(nil))
if code != 0 {
t.Fatalf("expected 0, got %d (stderr=%s)", code, errb.String())
}
got, err := os.ReadFile(dest)
if err != nil {
t.Fatal(err)
}
if string(got) != "get-blob" {
t.Fatalf("got %q", got)
}
})
t.Run("--out with any --target is rejected", func(t *testing.T) {
for _, target := range []string{"bunny", "cdn", "local"} {
var out, errb bytes.Buffer
code := Run([]string{"status", "--manifests", t.TempDir(), "--out", t.TempDir(), "--target", target}, &out, &errb, testGetenv(nil))
if code != 64 {
t.Fatalf("--target %s: expected 64, got %d (stderr=%s)", target, code, errb.String())
}
}
})
t.Run("get accepts flags before the positionals", func(t *testing.T) {
sha := shaOf("flags-first-blob")
depotDir := t.TempDir()
writeBlob(t, depotDir, sha, "flags-first-blob")
dest := filepath.Join(t.TempDir(), "fetched")
var out, errb bytes.Buffer
code := Run([]string{"get", "--out", depotDir, sha, dest}, &out, &errb, testGetenv(nil))
if code != 0 {
t.Fatalf("expected 0, got %d (stderr=%s)", code, errb.String())
}
})
t.Run("neither --out nor --target is rejected", func(t *testing.T) {
var out, errb bytes.Buffer
code := Run([]string{"status", "--manifests", t.TempDir()}, &out, &errb, testGetenv(nil))
if code != 64 {
t.Fatalf("expected 64, got %d (stderr=%s)", code, errb.String())
}
})
t.Run("--target local still works as a hidden alias", func(t *testing.T) {
sha := shaOf("alias-blob")
manifests := t.TempDir()
writeManifest(t, manifests, sha, 11)
depotDir := t.TempDir()
writeBlob(t, depotDir, sha, "alias-blob")
var out, errb bytes.Buffer
code := Run([]string{"status", "--manifests", manifests, "--target", "local"}, &out, &errb,
testGetenv(map[string]string{"DEPOT_DIR": depotDir}))
if code != 0 {
t.Fatalf("expected 0, got %d (stdout=%s stderr=%s)", code, out.String(), errb.String())
}
})
}
func TestGetInvalidSHA(t *testing.T) { func TestGetInvalidSHA(t *testing.T) {
var out, errb bytes.Buffer var out, errb bytes.Buffer
dir := t.TempDir() dir := t.TempDir()
@@ -172,6 +257,17 @@ func bytesContains(s, substr string) bool {
return bytes.Contains([]byte(s), []byte(substr)) return bytes.Contains([]byte(s), []byte(substr))
} }
func writeBlob(t *testing.T, root, sha, content string) {
t.Helper()
path := filepath.Join(root, BlobKey(sha))
if err := os.MkdirAll(filepath.Dir(path), 0755); err != nil {
t.Fatal(err)
}
if err := os.WriteFile(path, []byte(content), 0644); err != nil {
t.Fatal(err)
}
}
func writeManifest(t *testing.T, dir, sha string, size int64) { func writeManifest(t *testing.T, dir, sha string, size int64) {
t.Helper() t.Helper()
content := fmt.Sprintf("assets:\n - path: foo\n sha256: %s\n size: %d\n", sha, size) content := fmt.Sprintf("assets:\n - path: foo\n sha256: %s\n size: %d\n", sha, size)
+3 -3
View File
@@ -94,12 +94,12 @@ var Registry = []Builder{
{ {
Name: "depot", Name: "depot",
Bin: "crucible-depot", Bin: "crucible-depot",
Summary: "content-addressed asset depot (local/cdn/bunny)", Summary: "content-addressed asset depot (disk/cdn/bunny)",
Commands: []Command{ Commands: []Command{
{Name: "status", Summary: "report referenced-vs-present drift against a backend", Usage: "crucible depot status [--manifests DIR] [--source DIR] --target bunny|cdn|local"}, {Name: "status", Summary: "report referenced-vs-present drift against a backend", Usage: "crucible depot status [--manifests DIR] --target bunny|cdn | --out DIR"},
{Name: "push", Summary: "upload referenced-but-absent blobs from a local depot", Usage: "crucible depot push [--manifests DIR] --source DIR --target bunny"}, {Name: "push", Summary: "upload referenced-but-absent blobs from a local depot", Usage: "crucible depot push [--manifests DIR] --source DIR --target bunny"},
{Name: "verify", Summary: "existence sweep plus sampled download re-hash", Usage: "crucible depot verify [--manifests DIR] --target bunny|cdn [--sample N]"}, {Name: "verify", Summary: "existence sweep plus sampled download re-hash", Usage: "crucible depot verify [--manifests DIR] --target bunny|cdn [--sample N]"},
{Name: "get", Summary: "fetch one blob with sha re-verify", Usage: "crucible depot get <sha> <dest> --target cdn|bunny|local"}, {Name: "get", Summary: "fetch one blob with sha re-verify", Usage: "crucible depot get <sha> <dest> --target cdn|bunny | --out DIR"},
{Name: "pull", Summary: "incremental verified pull of every referenced blob", Usage: "crucible depot pull [--manifests DIR] --dest DIR --target cdn|bunny"}, {Name: "pull", Summary: "incremental verified pull of every referenced blob", Usage: "crucible depot pull [--manifests DIR] --dest DIR --target cdn|bunny"},
}, },
Wired: true, Wired: true,
+28 -18
View File
@@ -67,6 +67,19 @@ type resourceEntry struct {
Size uint32 Size uint32
} }
// extensionTypes and typeExtensions must stay exact inverses of each other;
// init() panics if they drift apart.
//
// Numbers below 0x0BB8 follow upstream neverwinter.nim
// (neverwinter/restype.nim). The 0x0BB8 and up entries are lyt/vis/mdx: real
// Aurora archive types that NWN1 ships in its own data/*.bif but upstream
// happens not to register. xoreos corroborates those three numbers
// (src/aurora/types.h).
//
// This table is NWN:EE only. Do not add NWN2 formats (mdb, gr2, wlk, xml):
// NWN:EE either gives that number to something else (2070 is xbc here and mdb
// in NWN2) or has no number for it at all, so packing one into a HAK writes a
// resource the game misreads. See the reserved list in erf_test.go.
var extensionTypes = map[string]uint16{ var extensionTypes = map[string]uint16{
"res": 0x0000, "res": 0x0000,
"bmp": 0x0001, "bmp": 0x0001,
@@ -96,12 +109,13 @@ var extensionTypes = map[string]uint16{
"utt": 0x07F0, "utt": 0x07F0,
"dds": 0x07F1, "dds": 0x07F1,
"uts": 0x07F3, "uts": 0x07F3,
"ltr": 0x07F4,
"gff": 0x07F5,
"fac": 0x07F6, "fac": 0x07F6,
"gff": 0x07F7,
"ute": 0x07F8, "ute": 0x07F8,
"utd": 0x07FA, "utd": 0x07FA,
"utp": 0x07FC, "utp": 0x07FC,
"dfa": 0x07FD, "dft": 0x07FD,
"gic": 0x07FE, "gic": 0x07FE,
"gui": 0x07FF, "gui": 0x07FF,
"utm": 0x0803, "utm": 0x0803,
@@ -117,20 +131,15 @@ var extensionTypes = map[string]uint16{
"ndb": 0x0810, "ndb": 0x0810,
"ptm": 0x0811, "ptm": 0x0811,
"ptt": 0x0812, "ptt": 0x0812,
"ltr": 0x0813,
"shd": 0x0815, "shd": 0x0815,
"mdb": 0x0816,
"mtr": 0x0818, "mtr": 0x0818,
"jpg": 0x081C,
"lod": 0x081E, "lod": 0x081E,
"gif": 0x081F, "gif": 0x081F,
"png": 0x0820, "png": 0x0820,
"jpg": 0x0821,
"lyt": 0x0BB8, "lyt": 0x0BB8,
"vis": 0x0BB9, "vis": 0x0BB9,
"mdx": 0x0BC0, "mdx": 0x0BC0,
"wlk": 0x0BCC,
"xml": 0x0BCD,
"gr2": 0x0FA3,
} }
var typeExtensions = map[uint16]string{ var typeExtensions = map[uint16]string{
@@ -162,12 +171,13 @@ var typeExtensions = map[uint16]string{
0x07F0: "utt", 0x07F0: "utt",
0x07F1: "dds", 0x07F1: "dds",
0x07F3: "uts", 0x07F3: "uts",
0x07F4: "ltr",
0x07F5: "gff",
0x07F6: "fac", 0x07F6: "fac",
0x07F7: "gff",
0x07F8: "ute", 0x07F8: "ute",
0x07FA: "utd", 0x07FA: "utd",
0x07FC: "utp", 0x07FC: "utp",
0x07FD: "dfa", 0x07FD: "dft",
0x07FE: "gic", 0x07FE: "gic",
0x07FF: "gui", 0x07FF: "gui",
0x0803: "utm", 0x0803: "utm",
@@ -183,20 +193,15 @@ var typeExtensions = map[uint16]string{
0x0810: "ndb", 0x0810: "ndb",
0x0811: "ptm", 0x0811: "ptm",
0x0812: "ptt", 0x0812: "ptt",
0x0813: "ltr",
0x0815: "shd", 0x0815: "shd",
0x0816: "mdb",
0x0818: "mtr", 0x0818: "mtr",
0x081C: "jpg",
0x081E: "lod", 0x081E: "lod",
0x081F: "gif", 0x081F: "gif",
0x0820: "png", 0x0820: "png",
0x0821: "jpg",
0x0BB8: "lyt", 0x0BB8: "lyt",
0x0BB9: "vis", 0x0BB9: "vis",
0x0BC0: "mdx", 0x0BC0: "mdx",
0x0BCC: "wlk",
0x0BCD: "xml",
0x0FA3: "gr2",
} }
func init() { func init() {
@@ -205,8 +210,13 @@ func init() {
if !ok { if !ok {
panic(fmt.Sprintf("missing canonical extension for resource type 0x%04X", resourceType)) panic(fmt.Sprintf("missing canonical extension for resource type 0x%04X", resourceType))
} }
if canonicalExt == ext { if canonicalExt != ext {
continue panic(fmt.Sprintf("resource type 0x%04X is %q in extensionTypes but %q in typeExtensions", resourceType, ext, canonicalExt))
}
}
for resourceType, ext := range typeExtensions {
if registered, ok := extensionTypes[ext]; !ok || registered != resourceType {
panic(fmt.Sprintf("extension %q is 0x%04X in typeExtensions but 0x%04X in extensionTypes (registered=%v)", ext, resourceType, registered, ok))
} }
} }
} }
+69 -5
View File
@@ -134,14 +134,10 @@ func TestExtensionMappingsSupportModernAssetTypes(t *testing.T) {
"mtr": 0x0818, "mtr": 0x0818,
"shd": 0x0815, "shd": 0x0815,
"txi": 0x07E6, "txi": 0x07E6,
"jpg": 0x081C, "jpg": 0x0821,
"mdb": 0x0816,
"lyt": 0x0BB8, "lyt": 0x0BB8,
"vis": 0x0BB9, "vis": 0x0BB9,
"mdx": 0x0BC0, "mdx": 0x0BC0,
"xml": 0x0BCD,
"wlk": 0x0BCC,
"gr2": 0x0FA3,
} }
for ext, wantType := range cases { for ext, wantType := range cases {
gotType, ok := ResourceTypeForExtension(ext) gotType, ok := ResourceTypeForExtension(ext)
@@ -191,3 +187,71 @@ func TestExtensionMappingsSupportAllUTBlueprintTypes(t *testing.T) {
} }
} }
} }
// TestRestypeTableMatchesUpstream pins the restype numbers Crucible shares with
// neverwinter.nim's restype.nim. The values below are upstream's; a mismatch
// means a HAK we write is mislabelled for the game.
func TestRestypeTableMatchesUpstream(t *testing.T) {
upstream := map[string]uint16{
"res": 0, "bmp": 1, "mve": 2, "tga": 3, "wav": 4, "plt": 6, "ini": 7,
"bmu": 8, "txt": 10, "mdl": 2002, "nss": 2009, "ncs": 2010, "are": 2012,
"set": 2013, "ifo": 2014, "bic": 2015, "wok": 2016, "2da": 2017,
"tlk": 2018, "txi": 2022, "git": 2023, "uti": 2025, "utc": 2027,
"dlg": 2029, "itp": 2030, "utt": 2032, "dds": 2033, "uts": 2035,
"ltr": 2036, "gff": 2037, "fac": 2038, "ute": 2040, "utd": 2042,
"utp": 2044, "dft": 2045, "gic": 2046, "gui": 2047, "utm": 2051,
"dwk": 2052, "pwk": 2053, "utg": 2055, "jrl": 2056, "utw": 2058,
"ssf": 2060, "hak": 2061, "nwm": 2062, "bik": 2063, "ndb": 2064,
"ptm": 2065, "ptt": 2066, "shd": 2069, "mtr": 2072, "lod": 2078,
"gif": 2079, "png": 2080, "jpg": 2081,
}
for ext, want := range upstream {
got, ok := ResourceTypeForExtension(ext)
if !ok {
t.Errorf("%s: not registered, upstream has %d", ext, want)
continue
}
if got != want {
t.Errorf("%s: registered as %d, upstream has %d", ext, got, want)
}
}
// Extensions upstream registers that Crucible must not reuse for anything else.
reserved := map[uint16]string{2039: "bte", 2067: "bak", 2070: "xbc", 2076: "tml"}
for number, upstreamExt := range reserved {
if ext, ok := ExtensionForResourceType(number); ok {
t.Errorf("%d: registered as %s, upstream reserves it for %s", number, ext, upstreamExt)
}
}
}
// TestNWN2FormatsAreNotRegistered keeps NWN2 file formats out of an NWN:EE
// table. mdb and gr2 have Aurora numbers that mean something else (or nothing)
// in NWN:EE; wlk and xml have no archive number in any Aurora game, so the
// values Crucible used for them were invented. Packing any of these into a HAK
// writes a resource the game misreads.
func TestNWN2FormatsAreNotRegistered(t *testing.T) {
for _, ext := range []string{"mdb", "gr2", "wlk", "xml"} {
if number, ok := ResourceTypeForExtension(ext); ok {
t.Errorf("%s is an NWN2 format but is registered as 0x%04X", ext, number)
}
}
}
// TestRestypeTablesAreMutualInverses is the check init() is meant to enforce.
func TestRestypeTablesAreMutualInverses(t *testing.T) {
for ext, number := range extensionTypes {
canonical, ok := typeExtensions[number]
if !ok {
t.Errorf("%s: number 0x%04X missing from typeExtensions", ext, number)
continue
}
if canonical != ext {
t.Errorf("%s: maps to 0x%04X, which maps back to %s", ext, number, canonical)
}
}
for number, ext := range typeExtensions {
if got, ok := extensionTypes[ext]; !ok || got != number {
t.Errorf("0x%04X: maps to %s, which maps back to 0x%04X (ok=%v)", number, ext, got, ok)
}
}
}
+10 -2
View File
@@ -593,9 +593,14 @@ func envBool(name string) bool {
func collectModuleResources(p *project.Project, moduleHakOrder []string) ([]erf.Resource, error) { func collectModuleResources(p *project.Project, moduleHakOrder []string) ([]erf.Resource, error) {
var moduleResources []erf.Resource var moduleResources []erf.Resource
palettes, err := collectPaletteDescriptors(p)
if err != nil {
return nil, err
}
for _, rel := range p.Inventory.SourceFiles { for _, rel := range p.Inventory.SourceFiles {
abs := filepath.Join(p.SourceDir(), filepath.FromSlash(rel)) abs := filepath.Join(p.SourceDir(), filepath.FromSlash(rel))
resource, err := resourceFromJSON(abs, moduleHakOrder) resource, err := resourceFromJSON(abs, moduleHakOrder, palettes)
if err != nil { if err != nil {
return nil, err return nil, err
} }
@@ -1030,7 +1035,7 @@ func compareResourceKeys(a, b erf.Resource) int {
return 0 return 0
} }
func resourceFromJSON(path string, moduleHakOrder []string) (erf.Resource, error) { func resourceFromJSON(path string, moduleHakOrder []string, palettes paletteProjection) (erf.Resource, error) {
name, extension, err := splitSourceName(path) name, extension, err := splitSourceName(path)
if err != nil { if err != nil {
return erf.Resource{}, err return erf.Resource{}, err
@@ -1059,6 +1064,9 @@ func resourceFromJSON(path string, moduleHakOrder []string) (erf.Resource, error
if extension == ".ifo" && name == "module" && len(moduleHakOrder) > 0 { if extension == ".ifo" && name == "module" && len(moduleHakOrder) > 0 {
setModuleHAKList(&document, moduleHakOrder) setModuleHAKList(&document, moduleHakOrder)
} }
if extension == ".itp" && isPaletteProjectionResref(name) {
projectPaletteDocument(&document, palettes[strings.ToLower(name)])
}
var buf bytes.Buffer var buf bytes.Buffer
if err := gff.Write(&buf, document); err != nil { if err := gff.Write(&buf, document); err != nil {
+7
View File
@@ -117,6 +117,10 @@ func expectedResources(p *project.Project) (map[string]resourceExpectation, erro
if err != nil { if err != nil {
return nil, err return nil, err
} }
palettes, err := collectPaletteDescriptors(p)
if err != nil {
return nil, err
}
for _, rel := range p.Inventory.SourceFiles { for _, rel := range p.Inventory.SourceFiles {
abs := filepath.Join(p.SourceDir(), filepath.FromSlash(rel)) abs := filepath.Join(p.SourceDir(), filepath.FromSlash(rel))
@@ -136,6 +140,9 @@ func expectedResources(p *project.Project) (map[string]resourceExpectation, erro
if extension == ".ifo" && strings.EqualFold(name, "module") && len(moduleHakOrder) > 0 { if extension == ".ifo" && strings.EqualFold(name, "module") && len(moduleHakOrder) > 0 {
setModuleHAKList(&document, moduleHakOrder) setModuleHAKList(&document, moduleHakOrder)
} }
if extension == ".itp" && isPaletteProjectionResref(name) {
projectPaletteDocument(&document, palettes[strings.ToLower(name)])
}
canonical, err := json.Marshal(document) canonical, err := json.Marshal(document)
if err != nil { if err != nil {
+7
View File
@@ -124,6 +124,13 @@ func extractArchiveResources(p *project.Project, archive erf.Archive, desired ma
skippedCount++ skippedCount++
continue continue
} }
// *palcus.itp are Toolset-generated palette projections; the module
// build regenerates them from source blueprints, so extraction never
// writes them back into source.
if ext == "itp" && isPaletteProjectionResref(resource.Name) {
skippedCount++
continue
}
target, data, err := extractedFile(p, resource, ext) target, data, err := extractedFile(p, resource, ext)
if err != nil { if err != nil {
+298
View File
@@ -0,0 +1,298 @@
package pipeline
// Custom palette projections (*palcus.itp) are generated artifacts: the Toolset
// derives them from the category frameworks plus the blueprints present in the
// module. Crucible reproduces that projection deterministically at build time so
// the module source only carries descriptor-free category skeletons and the
// blueprint files themselves. Extract never writes palcus files back.
import (
"encoding/json"
"fmt"
"os"
"path/filepath"
"sort"
"strings"
"git.westgate.pw/ShadowsOverWestgate/sow-tools/internal/gff"
"git.westgate.pw/ShadowsOverWestgate/sow-tools/internal/project"
)
const noStrref = 0xFFFFFFFF
// hiddenPaletteID suppresses a blueprint from the Custom palette (engine
// convention; see docs in sow-module's palette research notes).
const hiddenPaletteID = 255
var paletteFamilyByExtension = map[string]string{
".utc": "creaturepalcus",
".utd": "doorpalcus",
".ute": "encounterpalcus",
".uti": "itempalcus",
".utm": "storepalcus",
".utp": "placeablepalcus",
".uts": "soundpalcus",
".utt": "triggerpalcus",
".utw": "waypointpalcus",
}
type paletteDescriptor struct {
name string
strref uint32
resref string
creature bool
cr float32
faction string
}
func (d paletteDescriptor) sortKey() string {
if d.strref != noStrref || d.name == "" {
return strings.ToLower(d.resref)
}
return strings.ToLower(d.name)
}
// palette resref (e.g. "itempalcus") -> terminal category ID -> descriptors.
type paletteProjection map[string]map[uint8][]paletteDescriptor
func isPaletteProjectionResref(name string) bool {
return strings.HasSuffix(strings.ToLower(name), "palcus")
}
func collectPaletteDescriptors(p *project.Project) (paletteProjection, error) {
projection := paletteProjection{}
factions, err := loadFactionNames(p)
if err != nil {
return nil, err
}
for _, rel := range p.Inventory.SourceFiles {
abs := filepath.Join(p.SourceDir(), filepath.FromSlash(rel))
name, extension, err := splitSourceName(abs)
if err != nil {
return nil, err
}
family, ok := paletteFamilyByExtension[extension]
if !ok {
continue
}
raw, err := os.ReadFile(abs)
if err != nil {
return nil, fmt.Errorf("read %s: %w", abs, err)
}
var document gff.Document
if err := json.Unmarshal(raw, &document); err != nil {
return nil, fmt.Errorf("parse gff json %s: %w", abs, err)
}
descriptor, paletteID, ok := blueprintDescriptor(document.Root, name, extension, factions)
if !ok {
continue
}
if projection[family] == nil {
projection[family] = map[uint8][]paletteDescriptor{}
}
projection[family][paletteID] = append(projection[family][paletteID], descriptor)
}
for _, byID := range projection {
for _, descriptors := range byID {
sort.SliceStable(descriptors, func(i, j int) bool {
a, b := descriptors[i], descriptors[j]
if a.sortKey() != b.sortKey() {
return a.sortKey() < b.sortKey()
}
return a.resref < b.resref
})
}
}
return projection, nil
}
func blueprintDescriptor(root gff.Struct, fileName, extension string, factions []string) (paletteDescriptor, uint8, bool) {
descriptor := paletteDescriptor{
strref: noStrref,
resref: strings.ToLower(fileName),
creature: extension == ".utc",
}
paletteID := -1
for _, field := range root.Fields {
switch field.Label {
case "PaletteID":
if v, ok := field.Value.(gff.ByteValue); ok {
paletteID = int(v)
}
case "TemplateResRef":
if v, ok := field.Value.(gff.ResRefValue); ok && v != "" {
descriptor.resref = strings.ToLower(string(v))
}
case "LocalizedName", "LocName", "FirstName":
if v, ok := field.Value.(gff.LocString); ok {
name, strref := locStringLabel(v)
if field.Label == "FirstName" {
descriptor.name = strings.TrimSpace(descriptor.name + " " + name)
if descriptor.strref == noStrref {
descriptor.strref = strref
}
} else {
descriptor.name = name
descriptor.strref = strref
}
}
case "LastName":
if v, ok := field.Value.(gff.LocString); ok {
name, _ := locStringLabel(v)
descriptor.name = strings.TrimSpace(descriptor.name + " " + name)
}
case "ChallengeRating":
if v, ok := field.Value.(gff.FloatValue); ok {
descriptor.cr = float32(v)
}
case "FactionID":
if v, ok := field.Value.(gff.WordValue); ok && int(v) < len(factions) {
descriptor.faction = factions[v]
}
}
}
if paletteID < 0 || paletteID == hiddenPaletteID {
return paletteDescriptor{}, 0, false
}
return descriptor, uint8(paletteID), true
}
func locStringLabel(value gff.LocString) (string, uint32) {
if value.StringRef != noStrref {
return "", value.StringRef
}
for _, entry := range value.Entries {
if entry.Value != "" {
return entry.Value, noStrref
}
}
return "", noStrref
}
func loadFactionNames(p *project.Project) ([]string, error) {
for _, rel := range p.Inventory.SourceFiles {
if !strings.HasSuffix(strings.ToLower(rel), "repute.fac.json") {
continue
}
abs := filepath.Join(p.SourceDir(), filepath.FromSlash(rel))
raw, err := os.ReadFile(abs)
if err != nil {
return nil, fmt.Errorf("read %s: %w", abs, err)
}
var document gff.Document
if err := json.Unmarshal(raw, &document); err != nil {
return nil, fmt.Errorf("parse gff json %s: %w", abs, err)
}
var names []string
for _, field := range document.Root.Fields {
if field.Label != "FactionList" {
continue
}
list, ok := field.Value.(gff.ListValue)
if !ok {
continue
}
for _, faction := range list {
name := ""
for _, f := range faction.Fields {
if f.Label == "FactionName" {
if v, ok := f.Value.(gff.StringValue); ok {
name = string(v)
}
}
}
names = append(names, name)
}
}
return names, nil
}
return nil, nil
}
// projectPaletteDocument strips every blueprint descriptor from the palette
// tree and re-inserts the descriptors derived from module source. Category
// structure (branches, terminal IDs, labels) passes through untouched.
func projectPaletteDocument(document *gff.Document, byID map[uint8][]paletteDescriptor) {
for i, field := range document.Root.Fields {
if field.Label != "MAIN" {
continue
}
if list, ok := field.Value.(gff.ListValue); ok {
document.Root.Fields[i] = gff.NewField("MAIN", projectPaletteNodes(list, byID))
}
}
}
func projectPaletteNodes(nodes gff.ListValue, byID map[uint8][]paletteDescriptor) gff.ListValue {
out := make(gff.ListValue, 0, len(nodes))
for _, node := range nodes {
if structHasField(node, "RESREF") {
continue // blueprint descriptor — regenerated below
}
terminalID := -1
fields := make([]gff.Field, 0, len(node.Fields))
for _, field := range node.Fields {
if field.Label == "LIST" {
continue // rebuilt for terminals, recursed for branches
}
if field.Label == "ID" {
if v, ok := field.Value.(gff.ByteValue); ok {
terminalID = int(v)
}
}
fields = append(fields, field)
}
switch {
case terminalID >= 0:
if descriptors := byID[uint8(terminalID)]; len(descriptors) > 0 {
fields = append(fields, gff.NewField("LIST", descriptorList(descriptors)))
}
default:
for _, field := range node.Fields {
if field.Label == "LIST" {
if list, ok := field.Value.(gff.ListValue); ok {
fields = append(fields, gff.NewField("LIST", projectPaletteNodes(list, byID)))
}
}
}
}
out = append(out, gff.Struct{Type: node.Type, Fields: fields})
}
return out
}
func descriptorList(descriptors []paletteDescriptor) gff.ListValue {
list := make(gff.ListValue, 0, len(descriptors))
for _, d := range descriptors {
fields := make([]gff.Field, 0, 4)
if d.strref != noStrref {
fields = append(fields, gff.NewField("STRREF", gff.DWordValue(d.strref)))
} else {
fields = append(fields, gff.NewField("NAME", gff.StringValue(d.name)))
}
fields = append(fields, gff.NewField("RESREF", gff.ResRefValue(d.resref)))
if d.creature {
fields = append(fields, gff.NewField("CR", gff.FloatValue(d.cr)))
fields = append(fields, gff.NewField("FACTION", gff.StringValue(d.faction)))
}
list = append(list, gff.Struct{Fields: fields})
}
return list
}
func structHasField(s gff.Struct, label string) bool {
for _, field := range s.Fields {
if field.Label == label {
return true
}
}
return false
}
+191
View File
@@ -0,0 +1,191 @@
package pipeline
import (
"bytes"
"encoding/json"
"os"
"path/filepath"
"testing"
"git.westgate.pw/ShadowsOverWestgate/sow-tools/internal/erf"
"git.westgate.pw/ShadowsOverWestgate/sow-tools/internal/gff"
"git.westgate.pw/ShadowsOverWestgate/sow-tools/internal/project"
)
const paletteTestSkeleton = `{
"file_type": "ITP ",
"file_version": "V3.2",
"root": {
"struct_type": 4294967295,
"fields": [
{
"label": "MAIN",
"type": "List",
"value": [
{
"struct_type": 0,
"fields": [
{"label": "STRREF", "type": "DWord", "value": 500},
{
"label": "LIST",
"type": "List",
"value": [
{
"struct_type": 0,
"fields": [
{"label": "STRREF", "type": "DWord", "value": 6699},
{"label": "ID", "type": "Byte", "value": 23},
{
"label": "LIST",
"type": "List",
"value": [
{
"struct_type": 0,
"fields": [
{"label": "NAME", "type": "CExoString", "value": "Stale Junk"},
{"label": "RESREF", "type": "ResRef", "value": "stalejunk"}
]
}
]
}
]
},
{
"struct_type": 0,
"fields": [
{"label": "STRREF", "type": "DWord", "value": 6753},
{"label": "ID", "type": "Byte", "value": 24}
]
}
]
}
]
}
]
}
]
}
}
`
func paletteTestItem(resref, name string, paletteID int) string {
return `{
"file_type": "UTI ",
"file_version": "V3.2",
"root": {
"struct_type": 4294967295,
"fields": [
{"label": "TemplateResRef", "type": "ResRef", "value": "` + resref + `"},
{
"label": "LocalizedName",
"type": "CExoLocString",
"value": {"string_ref": 4294967295, "entries": [{"id": 0, "value": "` + name + `"}]}
},
{"label": "PaletteID", "type": "Byte", "value": ` + itoa(paletteID) + `}
]
}
}
`
}
func itoa(v int) string {
data, _ := json.Marshal(v)
return string(data)
}
func TestBuildProjectsPaletteDescriptorsAndExtractSkipsThem(t *testing.T) {
root := t.TempDir()
mustMkdir(t, filepath.Join(root, "src", "module"))
mustMkdir(t, filepath.Join(root, "src", "palettes"))
mustMkdir(t, filepath.Join(root, "src", "blueprints", "items"))
mustMkdir(t, filepath.Join(root, "build"))
mustWriteFile(t, filepath.Join(root, "nwn-tool.json"), `{
"module": {"name": "Test Module", "resref": "testmod"},
"paths": {"source": "src", "build": "build"}
}
`)
mustWriteFile(t, filepath.Join(root, "src", "module", "module.ifo.json"), `{
"file_type": "IFO ",
"file_version": "V3.2",
"root": {"struct_type": 4294967295, "fields": [{"label": "Mod_Name", "type": "CExoString", "value": "Test Module"}]}
}
`)
mustWriteFile(t, filepath.Join(root, "src", "palettes", "itempalcus.itp.json"), paletteTestSkeleton)
mustWriteFile(t, filepath.Join(root, "src", "blueprints", "items", "i_b.uti.json"), paletteTestItem("i_b", "Bravo Item", 23))
mustWriteFile(t, filepath.Join(root, "src", "blueprints", "items", "i_a.uti.json"), paletteTestItem("i_a", "Alpha Item", 23))
mustWriteFile(t, filepath.Join(root, "src", "blueprints", "items", "i_hidden.uti.json"), paletteTestItem("i_hidden", "Hidden Item", 255))
p, err := project.Load(root)
if err != nil {
t.Fatalf("load project: %v", err)
}
if err := p.Scan(); err != nil {
t.Fatalf("scan: %v", err)
}
if _, err := BuildModule(p); err != nil {
t.Fatalf("build: %v", err)
}
archiveFile, err := os.Open(p.ModuleArchivePath())
if err != nil {
t.Fatalf("open module archive: %v", err)
}
defer archiveFile.Close()
archive, err := erf.Read(archiveFile)
if err != nil {
t.Fatalf("read module archive: %v", err)
}
var palette *gff.Document
for _, resource := range archive.Resources {
if resource.Name == "itempalcus" {
document, err := gff.Read(bytes.NewReader(resource.Data))
if err != nil {
t.Fatalf("decode itempalcus: %v", err)
}
palette = &document
}
}
if palette == nil {
t.Fatal("itempalcus missing from built module")
}
canonical, err := json.Marshal(palette)
if err != nil {
t.Fatalf("marshal palette: %v", err)
}
text := string(canonical)
if bytes.Contains(canonical, []byte("stalejunk")) {
t.Fatalf("stale descriptor survived projection: %s", text)
}
for _, resref := range []string{"i_a", "i_b"} {
if !bytes.Contains(canonical, []byte(resref)) {
t.Fatalf("descriptor %s missing from projection: %s", resref, text)
}
}
if bytes.Contains(canonical, []byte("i_hidden")) {
t.Fatalf("PaletteID 255 blueprint leaked into projection: %s", text)
}
if a, b := bytes.Index(canonical, []byte("i_a")), bytes.Index(canonical, []byte("i_b")); a > b {
t.Fatalf("descriptors not name-sorted: %s", text)
}
if _, err := Compare(p); err != nil {
t.Fatalf("compare after build: %v", err)
}
// Extraction must never write palcus files back into source.
if err := os.Remove(filepath.Join(root, "src", "palettes", "itempalcus.itp.json")); err != nil {
t.Fatalf("remove skeleton: %v", err)
}
if err := p.Scan(); err != nil {
t.Fatalf("rescan: %v", err)
}
if _, err := Extract(p); err != nil {
t.Fatalf("extract: %v", err)
}
if _, err := os.Stat(filepath.Join(root, "src", "palettes", "itempalcus.itp.json")); !os.IsNotExist(err) {
t.Fatalf("extract wrote palcus file back (stat err: %v)", err)
}
}
+1 -1
View File
@@ -31,7 +31,7 @@ var SourceExtensions = []string{
} }
var AssetExtensions = []string{ var AssetExtensions = []string{
".2da", ".bik", ".bmp", ".bmu", ".dds", ".dwk", ".gr2", ".itp", ".jpg", ".lod", ".lyt", ".mdb", ".mdl", ".mdx", ".mtr", ".plt", ".png", ".pwk", ".set", ".shd", ".tga", ".txi", ".uti", ".vis", ".wav", ".wlk", ".wok", ".xml", ".2da", ".bik", ".bmp", ".bmu", ".dds", ".dwk", ".itp", ".jpg", ".lod", ".lyt", ".mdl", ".mdx", ".mtr", ".plt", ".png", ".pwk", ".set", ".shd", ".tga", ".txi", ".uti", ".vis", ".wav", ".wok",
} }
var BuiltinScriptPrefixes = []string{ var BuiltinScriptPrefixes = []string{
+16 -2
View File
@@ -109,6 +109,7 @@ type tlkCompiler struct {
active map[string]tlkEntryData active map[string]tlkEntryData
activeKeys map[string]struct{} activeKeys map[string]struct{}
reservedByID map[int]string reservedByID map[int]string
pinnedByID map[int]string
nextID int nextID int
} }
@@ -158,6 +159,7 @@ func newTLKCompiler(sourceDir string, legacy *legacyTLKData) (*tlkCompiler, erro
active: map[string]tlkEntryData{}, active: map[string]tlkEntryData{},
activeKeys: map[string]struct{}{}, activeKeys: map[string]struct{}{},
reservedByID: reserved, reservedByID: reserved,
pinnedByID: map[int]string{},
nextID: nextID, nextID: nextID,
} }
if legacy != nil { if legacy != nil {
@@ -380,12 +382,24 @@ func (c *tlkCompiler) registerInlineAtID(key string, id int, entry tlkEntryData)
if id < 0 { if id < 0 {
return fmt.Errorf("TLK key %q has negative id %d", key, id) return fmt.Errorf("TLK key %q has negative id %d", key, id)
} }
// The pin is authoritative over the per-machine .tlk_state.json cache: a
// stale mapping that dynamically grabbed this id on an older build must
// yield so the pinned key can take it. Only a genuine clash between two
// pins in custom.tlk.yml is an author error.
if owner, ok := c.pinnedByID[id]; ok && owner != key {
return fmt.Errorf("TLK id %d is pinned by both %q and %q", id, owner, key)
}
if mapping, ok := c.state.Entries[key]; ok && mapping.ID != id { if mapping, ok := c.state.Entries[key]; ok && mapping.ID != id {
return fmt.Errorf("TLK key %q changed id from %d to %d", key, mapping.ID, id) // This key held a different cached id; release it so the pin wins.
if c.reservedByID[mapping.ID] == key {
delete(c.reservedByID, mapping.ID)
}
} }
if owner, ok := c.reservedByID[id]; ok && owner != key { if owner, ok := c.reservedByID[id]; ok && owner != key {
return fmt.Errorf("TLK id %d is already reserved by %q", id, owner) // Evict the stale owner; it gets a fresh id when next made active.
delete(c.state.Entries, owner)
} }
c.pinnedByID[id] = key
c.state.Entries[key] = tlkStateMapping{ID: id} c.state.Entries[key] = tlkStateMapping{ID: id}
c.reservedByID[id] = key c.reservedByID[id] = key
return c.markActive(key, entry) return c.markActive(key, entry)
+49
View File
@@ -2449,6 +2449,55 @@ strings:
} }
} }
func TestBuildStandaloneTLKPinEvictsStaleStateOwner(t *testing.T) {
root := testProjectRoot(t)
mkdirAll(t, filepath.Join(root, "topdata", "data", "feat"))
mkdirAll(t, filepath.Join(root, "topdata", "tlk"))
writeFile(t, filepath.Join(root, "topdata", "base_dialog.json"), "{}\n")
writeFile(t, filepath.Join(root, "topdata", "data", "feat", "base.json"), `{
"output": "feat.2da",
"columns": ["LABEL", "FEAT", "DESCRIPTION"],
"rows": [{
"id": 0,
"key": "feat:test",
"LABEL": "TEST_LABEL",
"FEAT": {"tlk": {"key": "feat:test.name", "text": "Test Feat"}},
"DESCRIPTION": "****"
}]
}`+"\n")
writeFile(t, filepath.Join(root, "topdata", "tlk", "custom.tlk.yml"), `schema: sow-topdata/tlk/v1
base_strref: 16777216
strings:
- key: sow.module.name
text: Shadows Over Westgate
id: 50
`)
// Simulate a per-machine state that predates the pinned taxonomy: the feat
// ref dynamically grabbed id 50 on an older build, exactly where the pin
// now lives. The build must self-heal instead of failing.
writeFile(t, filepath.Join(root, "topdata", tlkStateFile),
`{"version":1,"language":"en","entries":{"feat:test.name":{"id":50}}}`+"\n")
if _, err := BuildNative(testProject(root), nil); err != nil {
t.Fatalf("BuildNative failed on stale pin collision: %v", err)
}
stateRaw, err := os.ReadFile(filepath.Join(root, "topdata", tlkStateFile))
if err != nil {
t.Fatalf("read tlk state: %v", err)
}
var state tlkStateDocument
if err := json.Unmarshal(stateRaw, &state); err != nil {
t.Fatalf("parse tlk state: %v", err)
}
if state.Entries["sow.module.name"].ID != 50 {
t.Fatalf("pin must own id 50, got %#v", state.Entries["sow.module.name"])
}
if got := state.Entries["feat:test.name"].ID; got == 50 {
t.Fatalf("stale feat ref should have been reallocated off id 50, got %d", got)
}
}
func TestBuildPreservesTLKStateAcrossTextChanges(t *testing.T) { func TestBuildPreservesTLKStateAcrossTextChanges(t *testing.T) {
root := testProjectRoot(t) root := testProjectRoot(t)
mkdirAll(t, filepath.Join(root, "topdata", "data", "skills")) mkdirAll(t, filepath.Join(root, "topdata", "data", "skills"))
+67
View File
@@ -0,0 +1,67 @@
#!/usr/bin/env bash
# Delete the binary assets of every release except the newest KEEP ones.
#
# Gitea has no asset retention, so 9 assets x ~57 MB per tag pile up forever on
# the host disk (sow-tools #52). Nothing consumes an old asset: CI takes the
# Crucible binary from the Nix flake, and every deleted file is reproducible
# from its tag. Tags, source archives, release notes and the releases
# themselves are never touched — only attachments.
#
# Nothing is excluded, SHA256SUMS and the wrappers included: the checksums are
# only meaningful next to the binaries they cover, and the drift-check reads
# the wrappers from the latest release, which always stays complete.
#
# Best-effort by design: a stale asset is cheaper than a blocked release, so
# every API failure is a warning, never a non-zero exit.
#
# Env:
# API repo API base, e.g. https://git.example/api/v1/repos/owner/repo [required]
# TOKEN Gitea token with releases:write [required]
# KEEP how many newest releases stay complete (default 2)
set -uo pipefail
: "${API:?set API}"
: "${TOKEN:?set TOKEN}"
keep="${KEEP:-2}"
auth="Authorization: token ${TOKEN}"
per_page=50
warn() { echo "prune-release-assets: $*" >&2; }
# Walk every page so an old backlog is cleared, not only the tag that fell out
# of the window on this run. Each page already embeds its releases' assets, so
# no follow-up request per release is needed.
releases='[]'
page=1
while :; do
body="$(curl -fsS -H "$auth" "${API}/releases?limit=${per_page}&page=${page}&draft=false")" || {
warn "listing releases failed on page ${page}; pruning what was listed so far"
break
}
count="$(jq 'length' <<<"$body")" || { warn "unreadable release page ${page}"; break; }
releases="$(jq -s 'add' <(printf '%s' "$releases") <(printf '%s' "$body"))"
# A short page is the last one; this also stops a server that ignores `page`.
(( count < per_page )) && break
page=$(( page + 1 ))
done
# Sort here rather than trusting the response order, and skip drafts in case
# the server ignored `draft=false` — a draft must not consume a keep slot and
# strip the binaries off the newest real release.
mapfile -t stale < <(jq -r --argjson keep "$keep" '
[.[] | select(.draft != true)]
| sort_by(.created_at) | reverse | .[$keep:]
| .[] | "\(.id) \(.assets // [] | map(.id) | join(","))"
' <<<"$releases")
(( ${#stale[@]} )) || { echo "prune-release-assets: nothing older than the newest ${keep} releases"; exit 0; }
for line in "${stale[@]}"; do
id="${line%% *}"
assets="${line#* }"
for asset in ${assets//,/ }; do
echo "deleting asset ${asset} of release ${id}"
curl -fsS -X DELETE -H "$auth" "${API}/releases/${id}/assets/${asset}" >/dev/null \
|| warn "deleting asset ${asset} of release ${id} failed"
done
done
+78
View File
@@ -0,0 +1,78 @@
#!/usr/bin/env bash
# Contract for scripts/prune-release-assets.sh, driven by a stub curl on PATH:
# only assets of releases outside the keep window go, the window is decided by
# release date rather than response order, drafts never consume a keep slot,
# releases and tags stay, and an API failure never fails the run.
set -euo pipefail
repo_root="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
script="$repo_root/scripts/prune-release-assets.sh"
tmp="$(mktemp -d)"
trap 'rm -rf "$tmp"' EXIT
# Stub curl: one short page of releases, deliberately out of date order and
# with a draft that is newer than every published release. Each release embeds
# two assets. Every request is appended to $CALLS for the assertions below.
mkdir -p "$tmp/bin"
cat >"$tmp/bin/curl" <<'STUB'
#!/usr/bin/env bash
url="${!#}"
printf '%s\n' "$*" >>"$CALLS"
rel() { printf '{"id":%s,"created_at":"%s","draft":%s,"assets":[{"id":%s01},{"id":%s02}]}' "$1" "$2" "$3" "$1" "$1"; }
case "$url" in
*releases\?*)
[[ "${FAIL_LIST:-0}" == 1 ]] && exit 22
printf '[%s,%s,%s,%s,%s]\n' \
"$(rel 12 2024-01-02T00:00:00Z false)" \
"$(rel 10 2024-01-04T00:00:00Z false)" \
"$(rel 9 2024-06-01T00:00:00Z true)" \
"$(rel 13 2024-01-01T00:00:00Z false)" \
"$(rel 11 2024-01-03T00:00:00Z false)"
;;
*/assets/*) [[ "${FAIL_DELETE:-0}" == 1 ]] && exit 22 ;;
esac
exit 0
STUB
chmod +x "$tmp/bin/curl"
export PATH="$tmp/bin:$PATH" API=https://git.example/api/v1/repos/o/r TOKEN=t
run() { CALLS="$tmp/calls" bash "$script" >"$tmp/out" 2>"$tmp/err"; }
# Default keep=2: the two newest published releases (10, 11) stay whole, the
# older two (12, 13) lose their assets, and the newer draft is ignored.
: >"$tmp/calls"; run
deletes="$(grep -c -- '-X DELETE' "$tmp/calls" || true)"
[[ "$deletes" == 4 ]] || { echo "expected 4 asset deletes, got $deletes" >&2; exit 1; }
grep -q 'assets/1201' "$tmp/calls" && grep -q 'assets/1302' "$tmp/calls" || {
echo "expected assets of releases 12 and 13 to be deleted" >&2; exit 1; }
if grep -q 'releases/10/assets/\|releases/11/assets/' "$tmp/calls"; then
echo "kept releases lost assets" >&2; exit 1
fi
if grep -q 'releases/9/assets/' "$tmp/calls"; then
echo "a draft release was pruned" >&2; exit 1
fi
# A release or tag must never be deleted, only attachments under /assets/.
if grep -- '-X DELETE' "$tmp/calls" | grep -qv '/assets/'; then
echo "a non-asset DELETE was issued" >&2; exit 1
fi
# A short page ends the sweep: no second page, so a server ignoring `page`
# cannot spin the job until the job timeout.
[[ "$(grep -c 'releases?' "$tmp/calls")" == 1 ]] || {
echo "a short release page did not end the sweep" >&2; exit 1; }
# KEEP=4 covers every published release: nothing to delete.
: >"$tmp/calls"; KEEP=4 run
if grep -q -- '-X DELETE' "$tmp/calls"; then echo "KEEP=4 still deleted" >&2; exit 1; fi
# Failures are warnings, never a non-zero exit.
: >"$tmp/calls"; FAIL_DELETE=1 run || { echo "delete failure failed the run" >&2; exit 1; }
grep -q 'failed' "$tmp/err" || { echo "delete failure was not warned about" >&2; exit 1; }
: >"$tmp/calls"; FAIL_LIST=1 run || { echo "listing failure failed the run" >&2; exit 1; }
grep -q 'listing releases failed' "$tmp/err" || {
echo "listing failure was not warned about" >&2; exit 1; }
if grep -q -- '-X DELETE' "$tmp/calls"; then
echo "deleted assets despite an unreadable release list" >&2; exit 1
fi
echo "prune-release-assets: prunes by release date outside the keep window, ignores drafts, never deletes a release, never fails the build"
Regular → Executable
+2
View File
@@ -46,6 +46,8 @@ grep -Fqx ' releases: write' "$release"
grep -Fqx ' timeout-minutes: 30' "$release" grep -Fqx ' timeout-minutes: 30' "$release"
[[ "$(grep -Fc "$checkout" "$release")" -eq 1 ]] [[ "$(grep -Fc "$checkout" "$release")" -eq 1 ]]
grep -Fq 'secrets.GITEA_TOKEN' "$release" grep -Fq 'secrets.GITEA_TOKEN' "$release"
grep -Fq 'scripts/prune-release-assets.sh' "$release"
grep -Fq 'continue-on-error: true' "$release"
if grep -Fq 'secrets.GITHUB_TOKEN' "$release"; then if grep -Fq 'secrets.GITHUB_TOKEN' "$release"; then
echo "workflow-contract: release uses the GitHub token alias" >&2 echo "workflow-contract: release uses the GitHub token alias" >&2
exit 1 exit 1