Compare commits

..
13 Commits
Author SHA1 Message Date
archvillainette 87feaf96b7 fix(wiki): purge through the plugin page actions and count reset deletions (#101)
build-binaries / build-binaries (push) Successful in 2m27s
Closes #99. Closes #100.

## #99 — purge used the core topic API

`deploy-wiki --stale-policy purge` deleted pages with `DELETE /api/v3/topics/{tid}`. On a NodeBB running `nodebb-plugin-westgate-wiki` that is refused for topics in wiki categories — revision history is plugin-owned — so every purge failed with HTTP 400 and the deploy exited 1.

Purge now goes through the plugin's own page actions:

1. `PUT /api/v3/plugins/westgate-wiki/page/tombstone`
2. `DELETE /api/v3/plugins/westgate-wiki/page/hard-purge`

in that order, because a page must be tombstoned before it can be purged. A page that is already gone answers 404 on the tombstone and is treated as a completed purge, as before.

**Archive was audited and needs no change.** It rewrites the page through `updatePost`, which is an ordinary post edit the plugin allows; only delete, restore, and purge are reserved to the page actions.

**The wiki home topic.** A namespace reset enumerates every topic in the category, including the home page, which the plugin excludes from tombstone, restore, and purge alike. Those are now skipped instead of aborting the reset. NodeBB answers 403 for that and for a token without purge privileges alike, and the response body cannot tell the two apart — what can is scope. A category where nothing at all could be deleted is a privilege problem, so the run still fails there rather than writing a manifest that claims a fresh start over pages that are all still present.

**The fakes.** Every fake NodeBB in `wiki_deploy_test.go` now goes through one constructor that refuses native topic mutation exactly the way the plugin does. The old fakes answered the core API, which is how a purge path that has never worked in production stayed green in CI.

## #100 — `stale: 0` above `purged: 1213`

The reset purge never went through stale computation, so the preview reported zero deletions on a run that would delete every topic in the managed categories.

Reset deletions are now counted in `stale`, which is the number callers word their destructive-policy warning around, and the summary gains a line naming the reset and how many of its targets the manifest has no record of writing:

```
stale:       1213
purged:      1213
namespace reset: 1213 (unrecognized: 13)
  unrecognized pages were not written by this deployer; recreating them is not possible
```

The unrecognized subset is the number worth surfacing, since those are the deletions a re-seed cannot undo. The `--reset-managed-namespaces` help text now says plainly that the flag deletes every page in the managed categories, not only the ones this deployer wrote.

`DeployResult` is exported so the console reads named fields instead of eleven positional ints.

## Verification

`go vet ./...` and `go test ./...` pass. New tests cover the plugin purge order, the already-missing page, the skipped undeletable topic, the per-category privilege failure, and the reset counts.Reviewed-on: #101

Co-authored-by: vickydotbat <vickydotbat@tutamail.com>
2026-08-05 16:01:51 +00:00
archvillainette a02e06d644 docs(agents): record that tea 0.14 cannot apply org-level labels (#98)
`tea issues edit --add-labels` and `--remove-labels` resolve label names against
the repository's own label set only. Every `Kind/*`, `Priority/*`, `Reviewed/*`
and `Status/*` label lives at the org level, so none of them match: the command
exits 0, prints the issue, and applies nothing. Repo-local labels
(`wayfinder:*`, `ready-for-agent`) work normally, which makes the failure look
intermittent instead of categorical.

Upstream tea fixed this in v0.15 — `modules/task/labels.go` now queries
`ListOrgLabels` alongside the repo labels. nixpkgs pins 0.14.0, so this note
expires when tea is bumped. `tea labels` has the same repo-only limitation and
will not show the org set; `tea api orgs/ShadowsOverWestgate/labels` will.

The second bullet is the more durable half: re-read a label change before
believing it. An earlier version of this branch confidently documented the
opposite cause, because a label that a human had applied by hand was read back
as proof that the CLI had applied it. `tea issues ls -o json` exposes a `labels`
field; a zero exit code does not.

Identical addition in all seven repos carrying `docs/agents/issue-tracker.md`.Reviewed-on: #98

Co-authored-by: vickydotbat <vickydotbat@tutamail.com>
2026-08-05 11:38:35 +00:00
archvillainette f395d86db5 Replace the stale triage label vocabulary with the org label taxonomy (#97)
`docs/agents/triage-labels.md` named five labels that do not exist in the tracker (`needs-triage`, `needs-info`, `ready-for-human`, `wontfix`). Meanwhile the org has an 18-label taxonomy at https://git.westgate.pw/org/ShadowsOverWestgate/settings/labels that agents never touched, because no doc pointed at it — `Kind/Bug` had been used twice across every repo.

This replaces the doc with the real taxonomy and makes one `Kind/*` label required on every issue and PR at creation time. `AGENTS.md` gets the short version.

Enforcement lands separately in `sow-platform` (`ops/policy/labels.yml` + a nightly `ops/checks/check-labels.sh` drift check). Identical doc change in every `sow-*` repo.Reviewed-on: #97

Co-authored-by: vickydotbat <vickydotbat@tutamail.com>
2026-08-04 21:55:47 +00:00
archvillainette 9747384838 fix(extract): keep committed palette skeletons out of stale cleanup (#96)
build-binaries / build-binaries (push) Successful in 2m50s
Extract skips writing *palcus.itp back to source (build projects them from blueprints, #50), but the skip branch never marked the skeleton target as desired, so cleanupStaleFiles deleted the committed source/palettes/*.itp.json on every extract — shipping modules with empty custom palettes. Fix marks the skipped target desired; regression test builds a module with a committed skeleton and asserts extract leaves it in place.

🤖 Generated with [Claude Code](https://claude.com/claude-code)Reviewed-on: #96

Co-authored-by: vickydotbat <vickydotbat@tutamail.com>
2026-08-04 08:51:12 +00:00
archvillainette b47a8a7afd Generate racial radial rows from race UsableFeat (#94) (#95)
build-binaries / build-binaries (push) Successful in 2m26s
Closes #94.

Generates the racial radial rows at build time from the `UsableFeat` column in the race feats tables, replacing the hand-maintained racial block in `sow-topdata` `data/classes/feats/global.json`.

## Why

Racial spell-like abilities reach the in-game radial only via an `OnMenu` row in every `cls_feat_<class>.2da`. Racial feats are granted by race, never by a class, so nothing adds them automatically — they were hand-written and prepended into every class table. The list drifts: this build already had **24** usable racial feats in the race tables but only **23** hand rows, so one activatable feat was silently missing.

## What

- `racialUsableFeatRules()` scans `race_feat_*.2da` datasets, collects `UsableFeat=1` feats, emits one rule each: `List=3, GrantedOnLevel=99, OnMenu=1`. Deduped across races, sorted for deterministic output.
- Row shape is exactly the in-game-verified hand rows. `List=3` keeps it off every level-up selection list; `GrantedOnLevel=99` is above the level cap so no class ever actually grants it (`nLevelGranted` is `uint8_t`); `OnMenu=1` renders the button once the creature possesses the feat. Possession stays chargen / the login racial-feat sync (`sow-codebase#359`).
- Reuses the existing `globalRules` injection path (same dedup, feat-existence check, label lookup) but applies **unconditionally**, so a leftover hand row in `global.json` deduplicates to a no-op — the `sow-topdata` cleanup lands separately.
- Not sourced from `feat.2da`: a global feat flag would inject unrelated class abilities (e.g. a shadowdancer ability) into every class radial via multiclass.

## Tests

- `racial_feat_rules_test.go`: usable-only, non-usable excluded, cross-race dedup, non-`race_feat_` ignored, deterministic order.
- Existing `cls_feat` global-injection build tests still pass with the new parameter.
- End-to-end: with the racial block removed from `global.json`, a real `build-topdata` emits `TieflingDarkness 3 99 1` once per table across all 21, 24 racial rows total.

## Merge order

The `sow-topdata` `global.json` racial-row deletion depends on this — it must ship first (or the racial radial vanishes on the next topdata build with the released tool).

🤖 Generated with [Claude Code](https://claude.com/claude-code)Reviewed-on: #95

Co-authored-by: vickydotbat <vickydotbat@tutamail.com>
2026-08-03 18:58:43 +00:00
archvillainette f23009ed50 verify is what tells you which keys to purge (#91)
build-binaries / build-binaries (push) Successful in 2m48s
Running the repair disproved the advice #90 landed an hour earlier.

"Purge the zone, then believe `verify`" assumed the stale set was unknowable. It is not. `verify` reads the edge, so the run straight after a repair names every key the edge is still serving stale — a survey, not a verdict. Purge those, re-run, and the second run is the verdict.

The measured numbers are the whole argument:

| | |
| --- | --- |
| blobs rewritten at the origin | 2,603 |
| blobs stale at the edge | **8** |

All eight were ones a failed player sync had pulled ninety minutes before the backfill. The edge only caches what someone fetched, so purging the zone would have cooled 69,169 objects to fix 8.

Full sweep after the targeted purge: `verified 69177 of 69177 blobs behind 72544 resources: 0 failures, 14887519535 bytes checked`. #75's gate is met.

Docs only. Runbook side in sow-platform.

Refs #88, #89, #75.

🤖 Generated with [Claude Code](https://claude.com/claude-code)Reviewed-on: #91
Reviewed-by: xtul <mpiasecki720@protonmail.com>
Co-authored-by: vickydotbat <vickydotbat@tutamail.com>
2026-08-01 09:20:16 +00:00
archvillainette 3cac6e9484 fix(project): a module resref names a file, not a resource (#93)
`module.resref` is the name of the built `.mod` on disk, so the 16-byte resref limit never applied to it — NWN:EE module file names are routinely longer. The blanket check rejected `ShadowsOverWestgate` (19 characters) and blocked sow-module#60:

```
crucible module build
  module.resref "ShadowsOverWestgate" exceeds 16 characters
```

## What changed

`internal/project/project.go` — `module.resref` is validated as a **file name** now, which still rejects a resref that is a path or empty.

The 16-character limit is kept where the value really does become a resref: a project with `paths.assets` and no `haks[]` names its single generated HAK after the module resref (`build.go:1144`), and a HAK name is a resref the engine loads. That case now says what to do about it instead of refusing every long module name.

## Verified

- 3 new tests in `internal/project`: a long module name validates and produces `ShadowsOverWestgate.mod`; a resref containing a path is rejected; a long resref that would name a generated HAK is still rejected.
- `make check` green.
- `crucible module build` in sow-module writes `module/ShadowsOverWestgate.mod` (70 resources).
- `crucible topdata validate` in sow-topdata still passes — its assets live under `topdata.assets`, not `paths.assets`, so the HAK guard does not bite.

Merge this **first**: sow-module's rename PR cannot go green in CI until this lands and its `flake.lock` is bumped.

Refs ShadowsOverWestgate/sow-module#60

🤖 Generated with [Claude Code](https://claude.com/claude-code)Reviewed-on: #93

Co-authored-by: vickydotbat <vickydotbat@tutamail.com>
2026-08-01 08:59:25 +00:00
archvillainette 3f78197f0a Purge the edge after a repair, or verify answers per PoP (#89) (#90)
build-binaries / build-binaries (push) Successful in 2m33s
#89 asked for a decision. This is it, and it is the laziest of the three options listed there: **purge the whole pull zone by hand after a repair, one call, documented in the repair procedure.**

Why not the other two:

- Purging from `emit --verify` needs a CDN credential `emit` deliberately does not hold, and `emit` reports how many blobs it wrote, never which ones — so it could not target the keys anyway.
- Waiting out the TTL means 30 days.

Whole-zone rather than per-key costs a cold cache on a zone whose objects are mostly cold, and a repair scatters thousands of keys across the tree regardless.

Also answers the question #89 left open: **the zone does not negative-cache.** A missing key answers 404 with `cache-control: no-cache` and `cdn-cache: MISS`, still MISS on an immediate retry (checked credential-free, 2026-08-01).

Docs only — `docs/command-surface.md` and `nwsync`'s usage text. The procedure itself lives in sow-platform's NWSync runbook, next to the zone it acts on.

Closes #89.

🤖 Generated with [Claude Code](https://claude.com/claude-code)Reviewed-on: #90
Reviewed-by: xtul <mpiasecki720@protonmail.com>
Co-authored-by: vickydotbat <vickydotbat@tutamail.com>
2026-07-31 23:13:55 +00:00
archvillainette 7cc53aeb68 fix(nwsync): declare Frame_Content_Size on every blob, and verify what is published (#87)
build-binaries / build-binaries (push) Successful in 2m40s
Closes #86. Closes #85.

These land together on purpose. Fixing the encoder alone changes nothing for the blobs already in the zone, because `emit` skips whatever is already present.

## #86 — the framing fix

`klauspost/compress` omits the zstd `Frame_Content_Size` field for inputs under 256 bytes, which the format permits. Reference libzstd never does, so the NWN client — which sizes its output buffer from `ZSTD_getFrameContentSize` and has therefore never met a frame without one — rejected roughly 6% of our blobs outright. Any single one stops a sync dead, so no client could complete a sync of the live manifest.

No encoder option changes this, so `compressBlob` re-headers the affected frames into the shape libzstd itself emits: `Single_Segment_flag` set, `Window_Descriptor` dropped, and the freed byte spent on a one-byte `Frame_Content_Size`. Same length in, same length out, and the same descriptor byte (`0x24`) the issue recorded from libzstd.

`compressBlob` then asserts its own output. An encoder upgrade that finds another way to omit the field would otherwise reproduce #86 in silence, and a blob is skipped by every later emit once written.

`emitter_version` goes to `2`, so `assemble` refuses to merge an index written by the encoder that omitted the field.

**Proved against the reference decoder, not just a round trip.** A real emitted 175-byte blob:

```
Frames  Skips  Compressed  Uncompressed  Ratio  Check  Filename
     1      0      48   B       175   B  3.646  XXH64  frame.zst
c59d6620d4ffd4bf3fe73df43b19b7afcfe8fea4  -            <- zstd -dc | sha1sum
c59d6620d4ffd4bf3fe73df43b19b7afcfe8fea4               <- the blob's own name
```

Before the fix that `Uncompressed` column was blank.

## #85 — `nwsync verify`

`crucible nwsync verify <manifest-sha1>` reads a manifest and its blobs back through the **public pull zone**, with no credential, because what matters is the bytes a client is served, edge behaviour included. Every distinct blob is decompressed and hashed; failures are reported per blob as missing / malformed framing / size mismatch / hash mismatch, and the exit code is 1.

- `--sample N` makes a routine check cheap against a manifest that is ~69,000 blobs and 15 GB; the default is a full sweep.
- `--base URL` / `NWSYNC_PULL_BASE` overrides the public host.
- The manifest is checked against its own sha1 before a single blob is fetched.
- `emit --verify` applies the same check where `emit` would otherwise trust presence, and replaces a stored blob that is not what its name claims. This is what makes the #86 blobs repairable.

## Why the existing checks missed this

Both new checks assert the **frame property**, not just a round trip. The conformance suite (#59) compares decompressed bytes, so a frame that decodes correctly passes regardless of its header; and the earlier zone audit decompressed 68 blobs with the `zstd` CLI, a *more* capable decoder than the client's, which certified exactly the blobs the client rejects.

## Checks

`make check` and `make smoke` green. Second commit is the fixes from a two-axis review of the first.

## Not in this PR

Three follow-ups, filed separately: the backfill has not been run, replacing a blob does not purge the pull-zone edge cache, and #85's runbook line belongs to `sow-platform`.

🤖 Generated with [Claude Code](https://claude.com/claude-code)Reviewed-on: #87

Co-authored-by: vickydotbat <vickydotbat@tutamail.com>
2026-07-31 22:33:12 +00:00
archvillainette 682f920114 fix(module): run scripts/fetch-upstream-manifests (#84)
build-binaries / build-binaries (push) Successful in 2m30s
Closes #83. Part of #54; implements #63 Decision 5 ("Coordinated rename flip").

## What changed

One string literal in `internal/app/app.go:388`:

```go
[]string{"scripts", "fetch-hak-manifest"} -> []string{"scripts", "fetch-upstream-manifests"}
```

sow-module renamed its half in ShadowsOverWestgate/sow-module#57
(`scripts/fetch-hak-manifest.sh` -> `scripts/fetch-upstream-manifests.sh`,
extended to resolve the topdata channel as well). The script name is a
hardcoded path convention shared by the two repos, not config, so both sides
only work when they carry the same name.

No doc in this repo named the old script (`grep` over the tree found the one
call site only), so nothing else needed touching.

## No fallback, by decision

Per #63 Decision 5: no transitional symlink, no Go-side fallback. Both sides
flip and the short broken window is accepted, because the failure is loud and
unmistakable (`required project script is missing`). A fallback would keep
both names alive forever.

## Merge order matters

`sow-module/.gitea/workflows/release.yml` runs `nix flake update sow-tools`, so
it always builds against the latest crucible, unpinned. Every crucible module
build in sow-module fails between sow-module#57 merging and a crucible release
carrying this flip. **Merge sow-module#57 and cut a crucible release back to
back** to keep that gap short.

## Checks

- `go build ./...`, `go vet ./internal/app/`, and the full `go test ./...` suite pass.
- No test covers this call path, and none was added: it is a single hardcoded
  constant that has to match the other repo, so a test here could only assert
  the literal against itself. The real check is the sow-module build.

## Still open on the issue's "done when"

- [x] app.go runs `scripts/fetch-upstream-manifests`
- [x] no doc in this repo names `fetch-hak-manifest`
- [ ] a crucible release is cut, and the crucible module build succeeds in a
      sow-module checkout at #57's head — needs a release after this merges

🤖 Generated with [Claude Code](https://claude.com/claude-code)Reviewed-on: #84
Reviewed-by: xtul <mpiasecki720@protonmail.com>
Co-authored-by: vickydotbat <vickydotbat@tutamail.com>
2026-07-31 19:00:25 +00:00
archvillainette 2f860ca9e4 fix(nwsync): write emit and assemble summaries to stderr (#82)
build-binaries / build-binaries (push) Successful in 2m35s
Fixes #81.

`crucible nwsync emit` and `assemble` printed their summary line to **stdout**. Any caller that captures a script's stdout as a value gets the summary glued onto it — `pack-haks.sh` does `release_dir="$(...)"`, so all 11 emit summaries landed in `$release_dir` and `publish-release.sh` died with "release dir not found".

Both lines move to stderr, where `lib.sh`'s own `nwsync: emitted $key` log already goes. Neither line is a machine-readable contract: sow-topdata's contract tests grep an `EMIT_LOG` their own fake-crucible stub writes, not real stdout, so nothing parses these.

`runEmit`/`runAssemble` no longer take the stdout writer — a leak in those two functions is now impossible to write by accident. `Run` still passes stdout to `printRunUsage` for explicit `-h`/`help`, which is correct.

Adds `TestRunKeepsSummariesOffStdout`: runs both verbs end to end against a local tree and asserts stdout stays empty while the summary reaches stderr. It asserts emptiness, not wording, so the summary text stays free to change.

Full `go test ./...` green.

Follow-up, outside this repo: sow-assets-manifest needs a `flake.lock` bump, then a re-run of the v0.2.1-rc1 tag.

🤖 Generated with [Claude Code](https://claude.com/claude-code)Reviewed-on: #82
Reviewed-by: xtul <mpiasecki720@protonmail.com>
Co-authored-by: vickydotbat <vickydotbat@tutamail.com>
2026-07-31 18:42:20 +00:00
archvillainette 1c2acc5530 feat(nwsync): emit blobs in parallel with a bounded worker pool (#80)
build-binaries / build-binaries (push) Successful in 2m21s
Closes #79.

Emit is latency-bound, not CPU-bound. Every blob costs two serial round-trips to the zone — a `ProbeKey` HEAD, then a `PutReader` PUT — so a hak with a few thousand resources pays a few thousand serialised latencies. Measured on the live sow-assets-manifest backfill: 26 s of CPU across 9.5 minutes of wall clock, on a 4-core host with 5 GB free and peak RSS of 51 MB.

`emit` now hashes, compresses and stores `--jobs N` resources at once, default 16 — matching `DEPOT_JOBS` and the transport's `MaxIdleConnsPerHost`, so a worker per connection needs no fresh TLS handshake. `--jobs 1` is exactly the old behaviour.

### Three properties had to survive

Each has a test in `internal/nwsync/jobs_test.go`:

- **Deterministic manifest bytes.** `emitterVersion` promises a manifest is a function of its artifact, so `entries` is index-addressed rather than appended to — a worker owns `entries[i]` alone and the slice comes back in artifact order whatever order uploads finish in. `TestEmitProducesTheSameIndexAtEveryJobCount` diffs the `.nsym` and its sidecar between `-jobs 1` and `-jobs 16`.
- **Index still lands last.** Any worker's failure aborts before a manifest is written. `TestEmitLeavesNoIndexWhenAParallelUploadFails` fails every blob PUT with 16 workers in flight and asserts no `.nsym` appears. Under `-race` it also covers the shared counters.
- **Identical content still shares one blob.** This one bit during development and is the reason to read the diff carefully: serially, the sink's existence check absorbed two resrefs with identical bytes. In parallel both workers probe, both miss, and both upload — `TestEmitWritesBlobsAndManifest` caught it as "wrote 2 blobs, want 1". Claiming the sha1 in-process restores the dedupe and skips a probe round-trip as well.

### Memory

Peak now tracks the resources in flight rather than one resource. The ceiling is `N` × the 15 MB `fileSizeLimit` plus its compressed copy — bounded by a constant this package enforces itself, and still not tracking the archive. `TestEmitPeakMemoryIsBoundedByJobCount` re-runs the #76 regression check at `-jobs 8`: a hak 8× bigger still costs the same.

This is only cheap because of #78. Before streaming emit, N workers would have meant N whole archives resident.

### Not done

Skipping the `ProbeKey` HEAD on a first-time emit would halve round-trips, but doubles uploaded bytes on a re-run — which is exactly what a backfill is. Noted in #79 so it is not rediscovered; parallelism is the better lever and this PR takes it.

Worker compression still serialises on `blobEncoder`, which is `WithEncoderConcurrency(1)` for the memory reason in `compressedbuf.go`. At 26 s of CPU per hak that is not worth trading memory for, but it is where to look if the numbers ever say otherwise.

🤖 Generated with [Claude Code](https://claude.com/claude-code)Reviewed-on: #80

Co-authored-by: vickydotbat <vickydotbat@tutamail.com>
2026-07-31 15:25:33 +00:00
archvillainette fa32dd411f fix(nwsync): stream emit so peak memory tracks the largest resource (#76) (#78)
build-binaries / build-binaries (push) Successful in 2m13s
Closes #76. Part of #54.

`nwsync emit` held roughly 5× the artifact size in RAM, so ovh-main (7 GB, no swap) OOM-killed it on any hak over ~1.4 GB. That blocked the backfill in sow-assets-manifest and would have killed the next release rebuilding a large hak.

## What it does

- `erf.ReadIndex` / `erf.ReadPayload` — parse the header and resource table only, read one payload on demand. `erf.Read` keeps its shape but returns payloads as subslices of the buffer instead of fresh copies, which removes one full copy for the `pipeline` callers too. Callers must not mutate `Resource.Data`; the doc comment says so and no caller does.
- `nwsync.Emit` opens the artifact, hashes it by streaming for the key check (through a section reader, so the file offset stays put), then hashes, compresses and stores one resource at a time. The archive is never resident. Shadowed duplicate resrefs are now never read at all.
- Bounds checks in `ReadIndex` moved to `int64`, so key/resource-list offsets can no longer overflow.

## Not in the issue, but memory-motivated

The zstd blob encoder ran at the default concurrency, which is one encoder per CPU, each holding a window-sized history — about 200 MB of live heap doing nothing on a 24-core runner. `EncodeAll` is single-threaded per call, so concurrency 1 costs nothing. `TestSingleThreadedEncoderMatchesDefault` pins the claim that blobs come out byte-identical.

## Measured

Peak heap during emit, sampled 1 ms:

| hak | before | after |
|-----|--------|-------|
| 8 MB | 155 MB | 21 MB |
| 64 MB | 289 MB | 22 MB |

Flat, as the acceptance asks. `TestEmitPeakMemoryDoesNotScaleWithArtifactSize` fails if the 64 MB fixture costs more than the 8 MB one plus 24 MB of slack.

## Gaps

- The regression check measures Go heap, not RSS, and its largest fixture is 64 MB — a multi-GB run was not done here. A 2.15 GB hak now needs about the same ~22 MB the 64 MB one does, so the 5 GB budget is not close, but that is inference from the flat curve, not a measurement.
- mmap was suggested in the issue and skipped. Payload buffers are still anonymous, but they are one resource each (≤15 MiB), so making them file-backed buys nothing now.

🤖 Generated with [Claude Code](https://claude.com/claude-code)Reviewed-on: #78
Reviewed-by: xtul <mpiasecki720@protonmail.com>
Co-authored-by: vickydotbat <vickydotbat@tutamail.com>
2026-07-31 11:16:23 +00:00
29 changed files with 2011 additions and 193 deletions
+2 -2
View File
@@ -94,9 +94,9 @@ Tests must survive harmless changes to constants, defaults, wording, ordering, f
Issues live in Gitea at git.westgate.pw (`ShadowsOverWestgate/sow-tools`), managed with the `tea` CLI. Issues follow ownership — file work in the repo that owns it, not the one you happen to be standing in. See `docs/agents/issue-tracker.md`.
### Triage labels
### Labels
Default label vocabulary (`needs-triage`, `needs-info`, `ready-for-agent`, `ready-for-human`, `wontfix`). See `docs/agents/triage-labels.md`.
Every issue and PR gets exactly one org-wide `Kind/*` label at creation (`Kind/Bug`, `Kind/Feature`, `Kind/Enhancement`, `Kind/Documentation`, `Kind/Testing`, `Kind/Security`); `Priority/*`, `Status/*`, `Reviewed/*` and `Compat/Breaking` are optional. `tea issues create -L "Kind/Bug"`. See `docs/agents/triage-labels.md`.
### Domain docs
+1 -1
View File
@@ -19,7 +19,7 @@ Crucible is how the artifact repos turn source into artifacts.
| `crucible-depot` | `crucible depot` | content-addressed depot blob verify/move |
| `crucible-hak` | `crucible hak` | ERF/HAK pack/unpack + hak manifests |
| `crucible-module` | `crucible module` | build/extract/validate/compare the `.mod` |
| `crucible-nwsync` | `crucible nwsync` | NWSync blob emit + manifest assemble |
| `crucible-nwsync` | `crucible nwsync` | NWSync blob emit + manifest assemble + verify |
| `crucible-topdata` | `crucible topdata` | compile 2da/tlk topdata + packages |
| `crucible-wiki` | `crucible wiki` | render + deploy mechanical wiki pages |
+26 -2
View File
@@ -57,8 +57,32 @@ preference — move it.
so any non-interactive shell (every agent) hangs forever without
`</dev/null`. Same trap on `tea issues create --description` and
`tea pr create`.
- **Apply / remove labels**: `tea api --method PATCH` on the issue, or
`tea api repos/ShadowsOverWestgate/sow-tools/issues/<number>/labels` endpoints.
- **Apply / remove labels**: `tea issues edit <number> --add-labels "Kind/Bug"`
(and `--remove-labels`). This handles org-level labels (`Kind/*`,
`Priority/*`, `Reviewed/*`, `Status/*`) from tea 0.15 onwards. On 0.14 it did
not: name resolution searched only this repo's own label set, so an org label
matched nothing and the command exited 0, printed the issue, and changed
nothing. Upstream fixed it in v0.15 (`modules/task/labels.go` also queries
`ListOrgLabels`). Note `tea labels` lists repo labels only and will not show
you the org set — `tea api orgs/ShadowsOverWestgate/labels` does.
- **`tea api` needs a token in the login; SSH auth is not enough.** It sends
only the login's `token:` field and does not sign requests with your SSH key,
so an SSH-key-only login gets `{"message":"token is required"}` on every call
that needs auth. Reads against these public repos still succeed anonymously,
which hides the gap until the first write. Add a token to the login in
`~/.config/tea/config.yml` (Settings > Applications; `write:issue` covers
labels, comments and dependencies) and `tea api` works for reads and writes
alike. The `tea issues` / `tea pr` subcommands authenticate either way, so
they keep working with no token at all — that asymmetry is what makes this
confusing to diagnose.
- **Verify every label change by re-reading it.** A label command exiting 0 is
not evidence it applied — that is exactly how the 0.14 silent no-op above hid
for so long, and assuming otherwise has already cost one investigation
several wrong turns. Read the resulting set back with
`tea api repos/ShadowsOverWestgate/sow-tools/issues/<number>` and check its
`labels` field, or `tea issues ls -o json`. The read-back reflects the write
immediately; if it comes back empty, the write genuinely failed. Do not
explain an empty read-back away as replication lag.
- **Close**: `tea issues close <number>`
`tea` infers the repo from the git remote when run inside the clone.
+105 -11
View File
@@ -1,15 +1,109 @@
# Triage Labels
# Issue and PR labels
The skills speak in terms of five canonical triage roles. This file maps those roles to the actual label strings used in this repo's issue tracker (Gitea — see `issue-tracker.md` for how to apply labels with `tea`).
Labels are **org-wide**. They are defined once, for the whole
`ShadowsOverWestgate` org, at
<https://git.westgate.pw/org/ShadowsOverWestgate/settings/labels>, and every
repo in the org can use them. Never create a per-repo copy of a label that
already exists at org level.
| Label in mattpocock/skills | Label in our tracker | Meaning |
| -------------------------- | -------------------- | ---------------------------------------- |
| `needs-triage` | `needs-triage` | Maintainer needs to evaluate this issue |
| `needs-info` | `needs-info` | Waiting on reporter for more information |
| `ready-for-agent` | `ready-for-agent` | Fully specified, ready for an AFK agent |
| `ready-for-human` | `ready-for-human` | Requires human implementation |
| `wontfix` | `wontfix` | Will not be actioned |
## The rule
When a skill mentions a role (e.g. "apply the AFK-ready triage label"), use the corresponding label string from this table.
**Every issue and every pull request gets exactly one `Kind/*` label, at the
moment it is created.** Not later, not "when someone triages it". If you open
it, you label it.
Edit the right-hand column to match whatever vocabulary you actually use.
An issue with no `Kind/*` label is untriaged. That is the only meaning of
"untriaged" here — there is no `needs-triage` label.
The other groups are optional, and each one is *exclusive*: an issue can carry
at most one `Priority/*`, one `Status/*`, and one `Reviewed/*`. Gitea enforces
this.
```sh
# always from inside the owning repo's clone
tea issues create --title "..." --description "..." --labels "Kind/Bug" </dev/null
tea pr create --title "..." --description "..." --labels "Kind/Feature" </dev/null
# add a label to something that already exists (needs tea >= 0.15 for org labels)
tea issues edit <n> --add-labels "Kind/Bug,Priority/High"
# read the result back — exit 0 is not evidence the label applied
tea api "repos/ShadowsOverWestgate/<repo>/issues/<n>"
```
`tea api` sends only the login's `token:` and never your SSH key, so on an
SSH-only login every authenticated call fails with
`{"message":"token is required"}` while anonymous reads still work. Add a token
to `~/.config/tea/config.yml` if you want the raw endpoints. `tea issues edit`
needs no token.
## Kind — what this is (pick exactly one)
| Label | Use it when |
| -------------------- | --------------------------------------------------------------- |
| `Kind/Bug` | Something that used to work, or is documented to work, does not |
| `Kind/Feature` | New functionality that does not exist yet |
| `Kind/Enhancement` | Existing functionality gets better, faster, or nicer |
| `Kind/Documentation` | Docs, ADRs, runbooks, agent guides |
| `Kind/Testing` | Tests, CI checks, contract scripts |
| `Kind/Security` | Secrets, auth, permissions, hardening, a vulnerability |
Bug vs Enhancement, when it is unclear: if the current behaviour is wrong, it
is a bug. If the current behaviour is right but weak, it is an enhancement.
## Priority — how urgent (optional, at most one)
`Priority/Critical`, `Priority/High`, `Priority/Medium`, `Priority/Low`.
Leave it off if you do not know. A wrong priority is worse than none.
## Status — why it is not moving (optional, at most one)
| Label | Meaning |
| ----------------------- | -------------------------------------------- |
| `Status/Blocked` | Waiting on another issue, PR, or decision |
| `Status/Need More Info` | Waiting on the reporter or on a human answer |
| `Status/Abandoned` | Work started and stopped; nobody is on it |
## Reviewed — the verdict (optional, at most one)
`Reviewed/Confirmed`, `Reviewed/Duplicate`, `Reviewed/Invalid`,
`Reviewed/Won't Fix`. Apply one of these when closing without a fix, so the
reason survives.
## Compat
`Compat/Breaking` — add it on top of the `Kind/*` label when the change breaks
something that already works for a player, an operator, or another repo.
## Workflow labels (repo-level, not org-level)
These live in the repo, not the org, and are orthogonal to the groups above:
- `ready-for-agent` — the spec is complete; an AFK agent may pick this up.
No label means it needs a human.
- `wayfinder:map`, `wayfinder:task`, `wayfinder:research`,
`wayfinder:prototype`, `wayfinder:grilling` — set by `/wayfinder`. Leave
them alone unless you are running a wayfinder operation.
- `sow-nodebb` also has `package/*` labels naming the plugin or theme a ticket
touches.
## When a skill names a label we do not have
Skills written elsewhere (mattpocock/skills and friends) use a different
vocabulary. Translate it:
| Skill says | Do this here |
| ----------------- | ------------------------------------------------- |
| `needs-triage` | Nothing — no `Kind/*` label already means this |
| `needs-info` | `Status/Need More Info` |
| `ready-for-agent` | `ready-for-agent` |
| `ready-for-human` | Nothing — absence of `ready-for-agent` means this |
| `wontfix` | `Reviewed/Won't Fix` |
## Drift check
`ops/checks/check-labels.sh` in `sow-platform` runs nightly. It compares the
live org labels to `ops/policy/labels.yml` and lists every open issue and PR in
the org that does not have exactly one `Kind/*` label. That check is the
enforcement; this file is the rule.
+62 -1
View File
@@ -36,6 +36,7 @@ aliases.
| `depot` | `pull` | Incremental verified pull of every referenced blob. |
| `nwsync` | `emit` | Explode one artifact into NWSync blobs plus its own NSYM manifest. |
| `nwsync` | `assemble` | Merge per-artifact NSYM manifests into one merged manifest. |
| `nwsync` | `verify` | Decompress and hash a published manifest's blobs through the pull zone. |
`depot status` and `depot get` pick their backend either with `--out DIR`, a
depot tree on disk, or with `--target bunny|cdn`, a remote backend. The two
@@ -48,10 +49,20 @@ beside the artifact itself with the extension replaced, so `emit` and
`assemble` agree on where it is without being told.
```
nwsync emit [--as NAME] [--out DIR] <artifact-key> <file>
nwsync emit [--as NAME] [--out DIR] [--jobs N] [--verify] <artifact-key> <file>
nwsync assemble --group-id N [--tlk-key KEY] [--out DIR] <artifact-key>...
nwsync verify [--sample N] [--base URL] [--jobs N] <manifest-sha1>
```
`emit` is latency-bound, not CPU-bound: every blob costs an existence probe
plus an upload, and a measured backfill spent 26 seconds of CPU across 9.5
minutes of wall clock. `--jobs N` (default 16) sets how many resources are in
flight at once. The manifest is byte-identical at any value — the number of
workers is never observable in the output. Peak memory is `N` times the
per-resource limit of 15 MB plus its compressed copy, so raising `N` far past
the default costs real memory for little gain: the transport keeps 16 idle
connections per host, and past that a worker pays a fresh TLS handshake.
Both verbs upload by default; nothing bulky is ever written to the runner's
disk. `--out DIR` writes a local repository tree instead, which is the
conformance path against upstream `nwn_nwsync_write`. The zone comes from
@@ -64,6 +75,45 @@ resolves it. `--tlk-key` has its own slot because the TLK shadows nothing.
`--group-id` is per channel — 1 is current, 2 is testing, and 0 leaves the field
out of the sidecar.
`nwsync verify` is the only check on a published blob upstream of a player's
client. It reads the **pull zone**, not the storage API, and needs no
credential: what matters is the bytes a client is served, edge behaviour
included. Every blob is decompressed and hashed, and the zstd frame is asserted
to declare its content size. Neither half is optional — a `Content-Length` check
passes a byte-correct-looking object whose contents are short, and a round-trip
check alone passes a frame the game client cannot decode but Go's decoder can.
Failures are reported per blob as missing, malformed framing, size mismatch or
hash mismatch, and the exit code is 1.
A full sweep of the live manifest is roughly 69,000 blobs and 15 GB, so
`--sample N` exists to make verifying routine; the default is a full sweep.
`--base URL` (or `NWSYNC_PULL_BASE`) overrides the public host.
`emit --verify` applies the same check where `emit` would otherwise skip. `emit`
normally reads a blob's presence as proof of its contents, decided by a 1-byte
range GET, so an object written truncated — or written by an emitter since found
broken — is skipped by every later run forever and no backfill repairs it. With
`--verify` the stored copy is read back, unwrapped, hashed against its own name,
and replaced when it does not match. It costs a full GET per existing blob, so
it is a repair pass, not the default.
**After a repair, `verify` is what tells you which keys to purge.** A repair is
the one thing that makes a key serve different bytes than it did before, and the
edge caches these objects for 30 days precisely because that normally cannot
happen. The two commands look at different copies on purpose: `emit --verify`
repairs the **origin**, `verify` reads the **edge**. So a `verify` run straight
after a repair is not a verdict — it is a survey, and every blob it still calls
bad is one the edge is serving stale. Purge exactly those, then re-run it; only
that second run is the verdict.
Purging the keys `verify` names beats purging the zone, because the edge only
ever cached what somebody actually fetched: the 2026-08-01 repair rewrote 2,603
blobs at the origin and left 8 stale at the edge. The purge belongs in the
repair procedure rather than in `emit`, which reports how many blobs it wrote
and never which ones — so it could not target one even with a CDN credential,
which it deliberately does not hold (#89; the procedure itself is in
sow-platform's NWSync runbook).
`emit` uploads blobs first and the index last, so the presence of an index is
the publication marker: an artifact whose emit died halfway leaves real blobs in
the zone and no index. Blob names are content hashes, so re-running skips
@@ -99,6 +149,17 @@ saving on hak content: a 250 MB hak emitted 2296 blobs totalling 59 MB on disk
against 249 MB of resources, as recorded in the sidecar's `on_disk_bytes` and
`total_bytes`.
The zstd frame always declares its `Frame_Content_Size`. The game client sizes
its output buffer from that field and cannot decode a frame without one, but the
Go encoder omits it below 256 bytes, so `emit` re-headers those frames into the
shape reference libzstd emits: `Single_Segment_flag` set, `Window_Descriptor`
dropped, and a one-byte content size in its place. `zstd -l <frame>` must print a
decompressed size; a blank column there is the fault, and it is invisible to any
check that only decompresses, because both `zstd -dc` and Go's decoder stream
such a frame happily. This is what the sidecar's `emitter_version` counts:
version 1 omitted the field and no client could sync past such a blob, version 2
declares it. `assemble` refuses to merge indexes that disagree.
## Hidden compatibility aliases
Existing scripts may continue using these names indefinitely. They are accepted
+21 -12
View File
@@ -385,7 +385,7 @@ func refreshBuildModuleManifest(ctx context, p *project.Project, progress func(s
}
progress("Refreshing hak list from the latest published sow-assets manifest...")
if err := runProjectScript(ctx, p, []string{"scripts", "fetch-hak-manifest"}, manifestPath); err != nil {
if err := runProjectScript(ctx, p, []string{"scripts", "fetch-upstream-manifests"}, manifestPath); err != nil {
return "", "", err
}
if _, err := pipeline.ApplyHAKManifest(p, manifestPath); err != nil {
@@ -927,19 +927,28 @@ func (c *topdataConsole) emitWikiBuildResult(outputDir string, pageCount int, st
fmt.Fprintf(c.stdout, "wiki status: %s\n", status)
}
func (c *topdataConsole) emitWikiDeployResult(localPages, created, updated, skipped, stale, archived, purged, drifted int, manifest string) {
func (c *topdataConsole) emitWikiDeployResult(result topdata.DeployResult) {
spin.linebreak()
fmt.Fprintln(c.stdout, "Deploy Wiki ----------")
fmt.Fprintf(c.stdout, "project: %s\n", c.projectName)
fmt.Fprintf(c.stdout, "local pages: %d\n", localPages)
fmt.Fprintf(c.stdout, "created: %d\n", created)
fmt.Fprintf(c.stdout, "updated: %d\n", updated)
fmt.Fprintf(c.stdout, "skipped: %d\n", skipped)
fmt.Fprintf(c.stdout, "stale: %d\n", stale)
fmt.Fprintf(c.stdout, "archived: %d\n", archived)
fmt.Fprintf(c.stdout, "purged: %d\n", purged)
fmt.Fprintf(c.stdout, "drifted: %d\n", drifted)
fmt.Fprintf(c.stdout, "manifest: %s\n", c.relPath(manifest))
fmt.Fprintf(c.stdout, "local pages: %d\n", result.LocalPages)
fmt.Fprintf(c.stdout, "created: %d\n", result.Created)
fmt.Fprintf(c.stdout, "updated: %d\n", result.Updated)
fmt.Fprintf(c.stdout, "skipped: %d\n", result.Skipped)
fmt.Fprintf(c.stdout, "stale: %d\n", result.Stale)
fmt.Fprintf(c.stdout, "archived: %d\n", result.Archived)
fmt.Fprintf(c.stdout, "purged: %d\n", result.Purged)
fmt.Fprintf(c.stdout, "drifted: %d\n", result.Drifted)
if result.ResetPurged > 0 || result.ResetSkipped > 0 {
fmt.Fprintf(c.stdout, "namespace reset: %d (unrecognized: %d)\n", result.ResetPurged, result.ResetUnrecognized)
}
if result.ResetUnrecognized > 0 {
fmt.Fprintln(c.stdout, " unrecognized pages were not written by this deployer; recreating them is not possible")
}
if result.ResetSkipped > 0 {
fmt.Fprintf(c.stdout, " reset skipped: %d (NodeBB refused deletion, e.g. the wiki home page)\n", result.ResetSkipped)
}
fmt.Fprintf(c.stdout, "manifest: %s\n", c.relPath(result.Manifest))
}
func (c *topdataConsole) relPath(path string) string {
@@ -1579,7 +1588,7 @@ func runDeployWiki(ctx context) error {
return err
}
console.emitWikiDeployResult(result.LocalPages, result.Created, result.Updated, result.Skipped, result.Stale, result.Archived, result.Purged, result.Drifted, result.Manifest)
console.emitWikiDeployResult(result)
return nil
}
+58 -1
View File
@@ -10,6 +10,7 @@ import (
"time"
"git.westgate.pw/ShadowsOverWestgate/sow-tools/internal/pipeline"
"git.westgate.pw/ShadowsOverWestgate/sow-tools/internal/topdata"
)
func TestParseBuildHAKArgsContentAddressedRoot(t *testing.T) {
@@ -218,7 +219,16 @@ func TestTopdataConsoleDebugProgressAndRelativePaths(t *testing.T) {
}
console.progress("NodeBB wiki plan: create 1, update 2, skip 3, stale 4, archive 5, purge 6, drift 0")
console.emitWikiDeployResult(10, 1, 2, 3, 4, 5, 6, 0, "/workspace/project/build/wiki/deploy-manifest.json")
console.emitWikiDeployResult(topdata.DeployResult{
LocalPages: 10,
Created: 1,
Updated: 2,
Skipped: 3,
Stale: 4,
Archived: 5,
Purged: 6,
Manifest: "/workspace/project/build/wiki/deploy-manifest.json",
})
output := stdout.String()
if !strings.Contains(output, "NodeBB wiki plan") {
@@ -229,6 +239,53 @@ func TestTopdataConsoleDebugProgressAndRelativePaths(t *testing.T) {
}
}
func TestTopdataConsoleReportsManagedNamespaceReset(t *testing.T) {
var stdout bytes.Buffer
console := &topdataConsole{
stdout: &stdout,
projectRoot: "/workspace/project",
projectName: "Test Module",
commandName: "deploy-wiki",
commandLabel: "Deploy Wiki",
}
console.emitWikiDeployResult(topdata.DeployResult{
LocalPages: 1200,
Created: 1200,
Stale: 1213,
Purged: 1213,
ResetPurged: 1213,
ResetUnrecognized: 13,
ResetSkipped: 1,
Manifest: "/workspace/project/build/wiki/deploy-manifest.json",
})
// The counts are what an operator reads to decide whether a destructive run
// is safe, so each has to reach the output on a line that names what it
// counts; the wording around them is free to change.
output := stdout.String()
var staleLine, resetLine, skippedLine string
for _, line := range strings.Split(output, "\n") {
switch {
case strings.Contains(line, "stale"):
staleLine = line
case strings.Contains(line, "skipped"):
skippedLine = line
case strings.Contains(line, "reset"):
resetLine = line
}
}
if !strings.Contains(staleLine, "1213") {
t.Fatalf("expected reset deletions counted as stale, got %q", output)
}
if !strings.Contains(resetLine, "1213") || !strings.Contains(resetLine, "13") {
t.Fatalf("expected the reset line to carry both the total and the unrecognized count, got %q", output)
}
if !strings.Contains(skippedLine, "1") {
t.Fatalf("expected refused reset deletions to be reported, got %q", output)
}
}
func TestParseDeployWikiHelpListsPurgeStalePolicy(t *testing.T) {
_, err := parseDeployWikiArgs("deploy-wiki", []string{"--help"})
if err == nil || !strings.Contains(err.Error(), "--stale-policy <report|archive|purge>") {
+4 -2
View File
@@ -108,10 +108,11 @@ var Registry = []Builder{
{
Name: "nwsync",
Bin: "crucible-nwsync",
Summary: "publish NWSync blobs and manifests (emit/assemble)",
Summary: "publish NWSync blobs and manifests (emit/assemble/verify)",
Commands: []Command{
{Name: "emit", Summary: "explode one artifact into blobs plus its own NSYM manifest", Usage: "crucible nwsync emit <artifact> --out DIR"},
{Name: "assemble", Summary: "merge per-artifact NSYM manifests into one", Usage: "crucible nwsync assemble --order NAMES --entries DIR --out DIR [--group-id N]"},
{Name: "verify", Summary: "read a published manifest's blobs back through the pull zone and hash them", Usage: "crucible nwsync verify <manifest-sha1> [--sample N]"},
},
Wired: true,
},
@@ -288,7 +289,8 @@ var Registry = []Builder{
"--dry-run report changes without writing",
"--create allow missing pages to be created",
"--force update unchanged pages",
"--reset-managed-namespaces reset managed namespace state",
"--reset-managed-namespaces delete every page in the managed categories, including",
" pages this deployer never wrote, then recreate from source",
},
Aliases: []CommandAlias{{Name: "deploy-wiki"}},
},
+1 -1
View File
@@ -160,7 +160,7 @@ func TestCanonicalCommandSurface(t *testing.T) {
"module": {"build", "extract", "validate", "compare", "manifest"},
"topdata": {"validate", "build", "package", "compare", "convert"},
"wiki": {"build", "deploy"},
"nwsync": {"emit", "assemble"},
"nwsync": {"emit", "assemble", "verify"},
}
for _, builder := range Registry {
got := builder.subcommands()
+100 -1
View File
@@ -3,6 +3,7 @@ package nwsync
import (
"bytes"
"encoding/binary"
"encoding/hex"
"fmt"
"github.com/klauspost/compress/zstd"
@@ -31,6 +32,19 @@ var (
blobDecoder, _ = zstd.NewReader(nil, zstd.WithDecoderConcurrency(1))
)
// zstd frame header bits we care about. A frame starts with the magic, then a
// one-byte Frame_Header_Descriptor: bits 7-6 size the Frame_Content_Size field,
// bit 5 is Single_Segment_flag, bits 1-0 size the Dictionary_ID field.
const (
zstdFrameMagic = "\x28\xb5\x2f\xfd"
frameSingleSegment = 1 << 5
frameDictionaryMask = 0x03
// oneByteContentSizeCeiling is the size above which a Frame_Content_Size no
// longer fits in one byte. Below it the field's size flag is 0, which is
// what lets klauspost/compress leave the field out entirely.
oneByteContentSizeCeiling = 256
)
// compressBlob wraps data in NWCompressedBuffer framing.
func compressBlob(data []byte) []byte {
var out bytes.Buffer
@@ -38,10 +52,95 @@ func compressBlob(data []byte) []byte {
for _, field := range header {
_ = binary.Write(&out, binary.LittleEndian, field)
}
out.Write(blobEncoder.EncodeAll(data, nil))
frame := declareFrameContentSize(blobEncoder.EncodeAll(data, nil), len(data))
// Fail closed rather than publish a blob no client can decode. An encoder
// upgrade that finds a new way to omit the field would otherwise reproduce
// #86 in silence, and a blob is skipped by every later emit once written.
if !frameDeclaresContentSize(frame) {
panic(fmt.Sprintf("nwsync: refusing to emit a %d-byte blob whose zstd frame declares no content size (descriptor %#x)",
len(data), frame[4]))
}
out.Write(frame)
return out.Bytes()
}
// inspectBlob unwraps a stored blob the way the game client reads it, and is the
// only reader that should be trusted to judge a published blob.
//
// It asserts the frame property on top of the round trip. Go's decoder — like
// the zstd CLI — streams a frame that declares no content size, so a check that
// only decompresses and hashes is a *more* capable decoder than the client's: it
// certifies exactly the blobs the client rejects, which is how #86 reached
// production and survived an audit.
func inspectBlob(blob []byte) ([]byte, error) {
data, err := decompressBlob(blob)
if err != nil {
return nil, fmt.Errorf("malformed framing: %w", err)
}
if len(blob) > blobHeaderBytes && !frameDeclaresContentSize(blob[blobHeaderBytes:]) {
return nil, fmt.Errorf("malformed framing: the zstd frame declares no content size, which the game client cannot decode")
}
return data, nil
}
// blobMatchesName holds a stored blob to its own file name: a blob is named
// after the sha1 of its uncompressed bytes, so the name is a complete statement
// about the contents and nothing else is needed to check it.
func blobMatchesName(blob []byte, sha1Hex string) error {
data, err := inspectBlob(blob)
if err != nil {
return err
}
if got := hex.EncodeToString(sha1Sum(data)); got != sha1Hex {
return fmt.Errorf("blob %s holds the contents of %s", sha1Hex, got)
}
return nil
}
// frameDeclaresContentSize reports whether a zstd frame states how many bytes it
// decompresses to. A frame with a zero-sized Frame_Content_Size field declares
// one only when Single_Segment_flag is set; otherwise the size is unknown.
func frameDeclaresContentSize(frame []byte) bool {
if len(frame) < 5 || string(frame[:4]) != zstdFrameMagic {
return false
}
descriptor := frame[4]
return descriptor>>6 != 0 || descriptor&frameSingleSegment != 0
}
// declareFrameContentSize rewrites a frame that does not declare its
// Frame_Content_Size so that it does, and returns any other frame unchanged.
//
// klauspost/compress omits the field for inputs under 256 bytes, which the spec
// permits. Reference libzstd never does, so the NWN client — which sizes its
// output buffer from ZSTD_getFrameContentSize and has therefore never met a
// frame without one — rejects the blob outright with an empty "potential
// compression error" (#86). No encoder option changes this, so the frame is
// re-headered here.
//
// The result is the shape libzstd itself emits for the same input: setting
// Single_Segment_flag drops the Window_Descriptor byte, and the freed byte pays
// for a one-byte Frame_Content_Size. Window_Size then equals the content size,
// which is sound because the content is under 256 bytes and every match in it
// therefore falls inside that window. Same length in, same length out.
func declareFrameContentSize(frame []byte, size int) []byte {
if size <= 0 || size >= oneByteContentSizeCeiling || len(frame) < 6 || string(frame[:4]) != zstdFrameMagic {
return frame
}
descriptor := frame[4]
// Rewrite only the exact shape a small input produces: no declared size, no
// single segment, no dictionary. Anything else either declares a size
// already or is not a frame this reinterpretation is safe on.
if descriptor>>6 != 0 || descriptor&frameSingleSegment != 0 || descriptor&frameDictionaryMask != 0 {
return frame
}
reframed := make([]byte, len(frame))
copy(reframed, frame)
reframed[4] = descriptor | frameSingleSegment
reframed[5] = byte(size) // replaces Window_Descriptor
return reframed
}
// decompressBlob unwraps NWCompressedBuffer framing. It exists so a blob this
// package wrote — or one upstream wrote — can be compared by its uncompressed
// bytes, which is the only comparison that is meaningful across zstd
+101 -14
View File
@@ -12,6 +12,7 @@ import (
"sort"
"strconv"
"strings"
"sync"
"time"
"git.westgate.pw/ShadowsOverWestgate/sow-tools/internal/buildinfo"
@@ -32,7 +33,9 @@ var skippedTypes = resTypes("nss", "ndb", "gic")
// manifest quietly disagrees with. Bump it only when emitted bytes change — it
// is deliberately not the build revision, which would invalidate every
// published index on every unrelated commit.
const emitterVersion = "1"
// Version 2 declares Frame_Content_Size on every blob (#86); version 1 omitted
// it below 256 bytes and no client could sync past such a blob.
const emitterVersion = "2"
// serverTypes are loaded only server-side; a manifest holding nothing else
// has no client contents. Mirrors upstream's GlobalResTypeServerList, whose
@@ -62,12 +65,22 @@ type EmitResult struct {
BlobsWritten int
}
// defaultEmitJobs is how many resources are hashed, compressed and stored at
// once. Emit is latency-bound, not CPU-bound: a blob costs a probe round-trip
// plus an upload round-trip, and a measured backfill spent 26 s of CPU across
// 9.5 minutes of wall clock. The figure matches depot's DEPOT_JOBS default and
// the transport's MaxIdleConnsPerHost, so a worker per connection needs no new
// TLS handshake.
const defaultEmitJobs = 16
// EmitOptions describes one emit run.
type EmitOptions struct {
ArtifactKey string // depot key of the artifact; the NSYM key is derived from it
ArtifactPath string // the file on disk
As string // name override, for a TLK whose filename is not its published name
OutDir string // write locally instead of uploading — the conformance path
Jobs int // resources in flight at once; 0 means defaultEmitJobs
Verify bool // hash what would be skipped instead of trusting presence
Sink sink // test seam; nil means OutDir or the zone
}
@@ -115,7 +128,11 @@ func Emit(options EmitOptions) (EmitResult, error) {
return EmitResult{}, err
}
entries, blobs, onDiskBytes, err := emitResources(artifact, index, target)
jobs := options.Jobs
if jobs < 1 {
jobs = defaultEmitJobs
}
entries, blobs, onDiskBytes, err := emitResources(artifact, index, target, jobs, options.Verify)
if err != nil {
return EmitResult{}, err
}
@@ -176,11 +193,16 @@ func readArtifactIndex(path string, artifact io.ReaderAt, size int64, name strin
return []erf.IndexEntry{{Name: name, Type: restype, Offset: 0, Size: size}}, nil
}
// emitResources hashes, compresses and stores one resource at a time, reading
// each payload from the artifact only when its turn comes. Peak memory
// therefore tracks the largest single resource, not the archive: a 2 GB hak
// must emit inside a runner's few spare GB.
func emitResources(artifact io.ReaderAt, index []erf.IndexEntry, target sink) ([]Entry, int, int64, error) {
// emitResources hashes, compresses and stores resources, reading each payload
// from the artifact only when its turn comes. Peak memory tracks the resources
// in flight, not the archive: a 2 GB hak must emit inside a runner's few spare
// GB. jobs of them are in flight at once, so the ceiling is jobs multiplied by
// fileSizeLimit and its compressed copy — bounded, and bounded by a constant
// this package enforces itself.
//
// The returned entries are in artifact order whatever order the workers finish
// in, because a manifest's bytes are promised deterministic by emitterVersion.
func emitResources(artifact io.ReaderAt, index []erf.IndexEntry, target sink, jobs int, verify bool) ([]Entry, int, int64, error) {
// A resref appearing twice inside one artifact resolves to the last one,
// the way resman lets the last container added win.
order := make([]Identity, 0, len(index))
@@ -208,30 +230,95 @@ func emitResources(artifact io.ReaderAt, index []erf.IndexEntry, target sink) ([
return nil, 0, 0, fmt.Errorf("resources exceed the file size limit:\n %s", strings.Join(tooBig, "\n "))
}
entries := make([]Entry, 0, len(order))
// Index-addressed, never appended to: a worker owns entries[i] alone, so
// the slice comes back in artifact order and needs no lock.
entries := make([]Entry, len(order))
var blobs int
var onDiskBytes int64
for _, identity := range order {
var mu sync.Mutex
var firstErr error
// Two resrefs in one artifact can hold identical bytes, and therefore one
// blob. Serially the sink's existence check absorbed that; in parallel both
// workers would probe, both miss, and both upload. Claiming the sha1 here
// restores the dedupe and skips the probe round-trip as well.
claimed := make(map[[20]byte]bool, len(order))
failed := func() bool {
mu.Lock()
defer mu.Unlock()
return firstErr != nil
}
store := func(i int) {
identity := order[i]
payload, err := erf.ReadPayload(artifact, latest[identity])
if err != nil {
return nil, 0, 0, err
mu.Lock()
if firstErr == nil {
firstErr = err
}
mu.Unlock()
return
}
sum := sha1.Sum(payload)
entries = append(entries, Entry{
entries[i] = Entry{
SHA1: sum,
Size: uint32(len(payload)),
ResRef: identity.ResRef,
ResType: identity.ResType,
})
written, err := target.putBlob(fmt.Sprintf("%x", sum), func() []byte { return compressBlob(payload) })
}
mu.Lock()
duplicate := claimed[sum]
claimed[sum] = true
mu.Unlock()
if duplicate {
return
}
written, err := target.putBlob(fmt.Sprintf("%x", sum), verify, func() []byte { return compressBlob(payload) })
mu.Lock()
defer mu.Unlock()
if err != nil {
return nil, 0, 0, err
if firstErr == nil {
firstErr = err
}
return
}
if written > 0 {
blobs++
onDiskBytes += written
}
}
if jobs < 1 {
jobs = 1
}
work := make(chan int)
var wg sync.WaitGroup
for range jobs {
wg.Add(1)
go func() {
defer wg.Done()
for i := range work {
// After a failure the run is over — the caller discards
// everything and no index is written. Draining the rest of the
// channel cheaply, rather than returning, keeps the feeder from
// blocking on workers that have gone away.
if failed() {
continue
}
store(i)
}
}()
}
for i := range order {
work <- i
}
close(work)
wg.Wait()
if firstErr != nil {
return nil, 0, 0, firstErr
}
return entries, blobs, onDiskBytes, nil
}
+134
View File
@@ -0,0 +1,134 @@
package nwsync
import (
"bytes"
"fmt"
"path/filepath"
"runtime/debug"
"strings"
"testing"
)
// manyResources is a hak body with enough distinct resources that a worker pool
// actually interleaves. Payloads differ so nothing is deduplicated away.
func manyResources(count int) map[string][]byte {
contents := make(map[string][]byte, count)
for i := range count {
contents[fmt.Sprintf("res%05d.tga", i)] = []byte(fmt.Sprintf("payload %d", i))
}
return contents
}
// TestEmitProducesTheSameIndexAtEveryJobCount is the contract that lets emit be
// parallel at all: emitterVersion promises a manifest's bytes are a function of
// its artifact, so the number of workers must not be observable in the output.
func TestEmitProducesTheSameIndexAtEveryJobCount(t *testing.T) {
// The sidecar stamps a wall-clock time unless this is set, which would make
// two runs differ for a reason that has nothing to do with job count.
t.Setenv("SOURCE_DATE_EPOCH", "1700000000")
dir := t.TempDir()
hak := filepath.Join(dir, "sow_test_01.hak")
writeHak(t, hak, manyResources(64))
key := artifactKey(t, hak)
emit := func(jobs int) (manifest, sidecar []byte, result EmitResult) {
out := filepath.Join(t.TempDir(), "out")
result, err := Emit(EmitOptions{
ArtifactKey: key,
ArtifactPath: hak,
OutDir: out,
Jobs: jobs,
})
if err != nil {
t.Fatalf("emit at -jobs %d: %v", jobs, err)
}
manifest, sidecar, err = dirSink{root: out}.getIndex(filepath.Base(result.ManifestPath))
if err != nil {
t.Fatalf("read index at -jobs %d: %v", jobs, err)
}
return manifest, sidecar, result
}
serialManifest, serialSidecar, serial := emit(1)
parallelManifest, parallelSidecar, parallel := emit(16)
if !bytes.Equal(serialManifest, parallelManifest) {
t.Errorf("manifest bytes differ between -jobs 1 and -jobs 16")
}
if !bytes.Equal(serialSidecar, parallelSidecar) {
t.Errorf("sidecar bytes differ between -jobs 1 and -jobs 16:\n %s\n %s", serialSidecar, parallelSidecar)
}
if serial.Entries != parallel.Entries || serial.BlobsWritten != parallel.BlobsWritten {
t.Errorf("-jobs 1 wrote %d entries/%d blobs, -jobs 16 wrote %d/%d",
serial.Entries, serial.BlobsWritten, parallel.Entries, parallel.BlobsWritten)
}
}
// TestEmitLeavesNoIndexWhenAParallelUploadFails is the fail-closed check with
// workers in flight: several uploads are in the air when the first one fails,
// and the index must still never appear. Run under -race this also covers the
// shared counters.
func TestEmitLeavesNoIndexWhenAParallelUploadFails(t *testing.T) {
fixture := newZoneFixture(t)
fixture.zone.failOn = func(key string) bool { return strings.HasPrefix(key, "data/sha1/") }
dir := t.TempDir()
hak := filepath.Join(dir, "sow_test_01.hak")
writeHak(t, hak, manyResources(64))
if _, err := Emit(EmitOptions{
ArtifactKey: artifactKey(t, hak),
ArtifactPath: hak,
Sink: fixture.sink,
Jobs: 16,
}); err == nil {
t.Fatal("emit reported success after an upload failed")
}
fixture.zone.mu.Lock()
defer fixture.zone.mu.Unlock()
for key := range fixture.zone.objects {
if strings.HasSuffix(key, ".nsym") {
t.Errorf("a half-emitted artifact published an index: %s", key)
}
}
}
// TestEmitPeakMemoryIsBoundedByJobCount pins the ceiling the parallel emit
// rests on. Peak still must not track the archive — it tracks the resources in
// flight, so a bigger hak at the same job count costs the same.
func TestEmitPeakMemoryIsBoundedByJobCount(t *testing.T) {
if testing.Short() {
t.Skip("writes a 64 MB fixture")
}
defer debug.SetGCPercent(debug.SetGCPercent(10))
measure := func(count, jobs int) uint64 {
dir := t.TempDir()
hak := filepath.Join(dir, "big.hak")
writeStreamedHak(t, hak, count)
options := EmitOptions{
ArtifactKey: artifactKey(t, hak),
ArtifactPath: hak,
As: filepath.Base(hak),
OutDir: filepath.Join(dir, "out"),
Jobs: jobs,
}
return peakHeapDuring(func() {
if _, err := Emit(options); err != nil {
t.Fatalf("emit %d resources at -jobs %d: %v", count, jobs, err)
}
})
}
const jobs = 8
small := measure(8, jobs) // 8 MB
large := measure(64, jobs) // 64 MB
// Each worker may hold one resourceSize payload plus its compressed copy,
// so the pool itself is the slack — not the archive.
const slack = 24 << 20
t.Logf("peak heap at -jobs %d: 8 MB hak %d bytes, 64 MB hak %d bytes", jobs, small, large)
if large > small+slack {
t.Fatalf("peak heap scaled with artifact size at -jobs %d: 8 MB hak peaked at %d bytes, 64 MB hak at %d", jobs, small, large)
}
}
+10 -2
View File
@@ -7,6 +7,7 @@ import (
"encoding/json"
"fmt"
"io"
"path"
"path/filepath"
"sort"
"strings"
@@ -194,7 +195,14 @@ func marshalSidecar(sidecar Sidecar) ([]byte, error) {
return append(body, '\r', '\n'), nil
}
// blobPath is the data store path for a blob, hash tree depth 2.
// blobKey is where a blob lives in a zone, hash tree depth 2. emit writes it,
// verify reads it and the game client requests it, so the rule lives here and
// nowhere else.
func blobKey(sha1Hex string) string {
return path.Join("data", "sha1", sha1Hex[0:2], sha1Hex[2:4], sha1Hex)
}
// blobPath is the same location inside a local repository tree.
func blobPath(root, sha1Hex string) string {
return filepath.Join(root, "data", "sha1", sha1Hex[0:2], sha1Hex[2:4], sha1Hex)
return filepath.Join(root, filepath.FromSlash(blobKey(sha1Hex)))
}
+55
View File
@@ -458,6 +458,33 @@ func TestAssembleFailsClosedOnMissingIndex(t *testing.T) {
}
}
// Callers capture a script's stdout as a value: `dir="$(pack-haks.sh)"`. A
// summary line on stdout gets glued onto that value, so both summaries belong
// on stderr.
func TestRunKeepsSummariesOffStdout(t *testing.T) {
dir := t.TempDir()
path := filepath.Join(dir, "sow_top.hak")
writeHak(t, path, map[string][]byte{"appearance.2da": []byte("2da from sow_top")})
key := artifactKey(t, path)
out := filepath.Join(dir, "out")
for _, args := range [][]string{
{"emit", "--out", out, "--as", "sow_top.hak", key, path},
{"assemble", "--out", out, key},
} {
var stdout, stderr bytes.Buffer
if code := Run(args, &stdout, &stderr); code != exitOK {
t.Fatalf("Run(%v) exit=%d: %s", args, code, stderr.String())
}
if stdout.Len() != 0 {
t.Errorf("Run(%v) wrote to stdout: %q", args, stdout.String())
}
if stderr.Len() == 0 {
t.Errorf("Run(%v) reported no summary on stderr", args)
}
}
}
func TestRunUsageErrors(t *testing.T) {
cases := [][]string{
nil,
@@ -474,3 +501,31 @@ func TestRunUsageErrors(t *testing.T) {
}
}
}
// TestEveryBlobDeclaresItsFrameContentSize guards the fault that stopped every
// client sync (#86): klauspost/compress omits Frame_Content_Size for inputs
// under 256 bytes, and the game client cannot decode a frame without it. This
// asserts a frame property, not a round trip — the zstd CLI and Go's decoder
// both stream such a frame happily, so round-tripping cannot see the defect.
func TestEveryBlobDeclaresItsFrameContentSize(t *testing.T) {
// 230 and 175 are real sizes from the manifest that failed to sync; 255/256
// straddle the encoder's threshold.
for _, size := range []int{1, 32, 175, 230, 255, 256, 257, 1024, 5000} {
payload := make([]byte, size)
for i := range payload {
payload[i] = byte('a' + i%26)
}
blob := compressBlob(payload)
if !frameDeclaresContentSize(blob[blobHeaderBytes:]) {
t.Errorf("blob of %d bytes declares no frame content size (descriptor %#x)",
size, blob[blobHeaderBytes+4])
}
got, err := decompressBlob(blob)
if err != nil {
t.Fatalf("decompress %d-byte blob: %v", size, err)
}
if !bytes.Equal(got, payload) {
t.Errorf("%d-byte blob did not round trip", size)
}
}
}
+69 -7
View File
@@ -12,12 +12,17 @@ import (
"flag"
"fmt"
"io"
"os"
)
const (
exitOK = 0
exitUsage = 64
exitInternal = 70
// exitDrift says the command worked and the zone is wrong, which is a
// different thing for CI to act on than the command failing. It matches
// depot's code for the same meaning.
exitDrift = 1
)
// Run executes an nwsync subcommand. args[0] is the subcommand (emit|assemble);
@@ -29,9 +34,11 @@ func Run(args []string, stdout, stderr io.Writer) int {
}
switch args[0] {
case "emit":
return runEmit(args[1:], stdout, stderr)
return runEmit(args[1:], stderr)
case "assemble":
return runAssemble(args[1:], stdout, stderr)
return runAssemble(args[1:], stderr)
case "verify":
return runVerify(args[1:], stdout, stderr, os.Getenv)
case "-h", "--help", "help":
printRunUsage(stdout)
return exitOK
@@ -44,17 +51,30 @@ func Run(args []string, stdout, stderr io.Writer) int {
func printRunUsage(w io.Writer) {
fmt.Fprint(w, `usage:
nwsync emit [--as NAME] [--out DIR] <artifact-key> <file>
nwsync emit [--as NAME] [--out DIR] [--verify] <artifact-key> <file>
nwsync assemble --group-id N [--tlk-key KEY] [--out DIR] <artifact-key>...
nwsync verify [--sample N] [--base URL] <manifest-sha1>
emit explodes one .hak/.erf or one loose file (the TLK) into NWSync blobs plus
a NSYM index covering only that artifact, and uploads both. assemble merges
those indexes into one manifest, reading no bulk data. Artifact keys are depot
keys; an index lives beside its artifact, with the extension replaced.
verify reads a published manifest and its blobs back through the public pull
zone, with no credential, and decompresses and hashes every one. It is the only
check on a published blob upstream of a player's client.
--verify makes emit hash what it would otherwise skip. emit normally treats a
blob's presence as proof of its contents, so without this an object written
truncated, or written by an emitter since found broken, is skipped forever.
--verify repairs the storage zone, while verify reads the edge in front of it.
So a verify run right after a repair is a survey, not a verdict: it names the
keys the edge still serves stale. Purge those, then run it again.
--out DIR writes to a local repository tree instead of uploading, which is the
conformance path against upstream nwn_nwsync_write. Without it, the zone comes
from NWSYNC_STORAGE_ZONE, NWSYNC_STORAGE_PASSWORD and BUNNY_STORAGE_HOST.
verify needs none of those; its base comes from --base or NWSYNC_PULL_BASE.
`)
}
@@ -76,11 +96,13 @@ func parseArgs(fs *flag.FlagSet, args []string) ([]string, error) {
}
}
func runEmit(args []string, stdout, stderr io.Writer) int {
func runEmit(args []string, stderr io.Writer) int {
fs := flag.NewFlagSet("emit", flag.ContinueOnError)
fs.SetOutput(stderr)
as := fs.String("as", "", "published name of the artifact, when it differs from the key")
out := fs.String("out", "", "write to a local repository tree instead of uploading")
jobs := fs.Int("jobs", defaultEmitJobs, "resources to hash, compress and store at once")
verify := fs.Bool("verify", false, "read back and hash blobs that already exist instead of trusting their presence")
positional, err := parseArgs(fs, args)
if err != nil {
return exitUsage
@@ -89,23 +111,63 @@ func runEmit(args []string, stdout, stderr io.Writer) int {
fmt.Fprintf(stderr, "nwsync emit: <artifact-key> and <file> are both required\n")
return exitUsage
}
if *jobs < 1 {
fmt.Fprintf(stderr, "nwsync emit: -jobs must be at least 1, got %d\n", *jobs)
return exitUsage
}
result, err := Emit(EmitOptions{
ArtifactKey: positional[0],
ArtifactPath: positional[1],
As: *as,
OutDir: *out,
Jobs: *jobs,
Verify: *verify,
})
if err != nil {
fmt.Fprintf(stderr, "nwsync emit: %v\n", err)
return exitInternal
}
fmt.Fprintf(stdout, "emitted %s: %d resources, %d new blobs, index %s\n",
fmt.Fprintf(stderr, "emitted %s: %d resources, %d new blobs, index %s\n",
result.Name, result.Entries, result.BlobsWritten, result.ManifestPath)
return exitOK
}
func runAssemble(args []string, stdout, stderr io.Writer) int {
func runVerify(args []string, stdout, stderr io.Writer, getenv func(string) string) int {
fs := flag.NewFlagSet("verify", flag.ContinueOnError)
fs.SetOutput(stderr)
base := fs.String("base", getenv("NWSYNC_PULL_BASE"), "pull zone base URL to read through")
sample := fs.Int("sample", 0, "check this many random blobs instead of all of them")
jobs := fs.Int("jobs", defaultEmitJobs, "blobs to fetch and hash at once")
positional, err := parseArgs(fs, args)
if err != nil {
return exitUsage
}
if len(positional) != 1 {
fmt.Fprintf(stderr, "nwsync verify: exactly one <manifest-sha1> is required\n")
return exitUsage
}
result, err := Verify(VerifyOptions{
ManifestSHA1: positional[0],
Base: *base,
Sample: *sample,
Jobs: *jobs,
Log: stderr,
})
if err != nil {
fmt.Fprintf(stderr, "nwsync verify: %v\n", err)
return exitInternal
}
fmt.Fprintf(stdout, "verified %d of %d blobs behind %d resources: %d failures, %d bytes checked\n",
result.Checked, result.Blobs, result.Entries, result.Failures, result.Bytes)
if result.Failures > 0 {
return exitDrift
}
return exitOK
}
func runAssemble(args []string, stderr io.Writer) int {
fs := flag.NewFlagSet("assemble", flag.ContinueOnError)
fs.SetOutput(stderr)
tlkKey := fs.String("tlk-key", "", "depot key of the TLK, which shadows nothing and merges last")
@@ -134,7 +196,7 @@ func runAssemble(args []string, stdout, stderr io.Writer) int {
fmt.Fprintf(stderr, "nwsync assemble: %v\n", err)
return exitInternal
}
fmt.Fprintf(stdout, "assembled manifest %s: %d resources, %s\n",
fmt.Fprintf(stderr, "assembled manifest %s: %d resources, %s\n",
result.SHA1, result.Entries, result.ManifestPath)
return exitOK
}
+39 -10
View File
@@ -20,11 +20,17 @@ import (
// upstream's output and ours can be diffed on a developer machine.
type sink interface {
// putBlob stores one NWCompressedBuffer blob under its sha1 name and
// returns the bytes stored, or 0 if the blob was already there. Blob names
// are content hashes, so an existing name is existing content — which is
// why body is a thunk: compression is the expensive part of emit and a
// blob that is already stored must not pay for it.
putBlob(sha1Hex string, body func() []byte) (int64, error)
// returns the bytes stored, or 0 if a good copy was already there. Blob
// names are content hashes, so an existing name is normally taken as
// existing content — which is why body is a thunk: compression is the
// expensive part of emit and a blob that is already stored must not pay
// for it.
//
// verify stops trusting presence: the stored copy is read back, unwrapped
// and hashed, and replaced when it is not what its name claims. Without it
// an object written truncated, or written by an emitter since found broken,
// is skipped by every later emit forever and no backfill can repair it.
putBlob(sha1Hex string, verify bool, body func() []byte) (int64, error)
// putIndex stores a NSYM manifest and its sidecar under key, which is
// either an artifact-derived object key or a local path.
putIndex(key string, manifest, sidecar []byte) error
@@ -37,9 +43,15 @@ type sink interface {
// dirSink writes a local NWSync repository tree.
type dirSink struct{ root string }
func (s dirSink) putBlob(sha1Hex string, body func() []byte) (int64, error) {
func (s dirSink) putBlob(sha1Hex string, verify bool, body func() []byte) (int64, error) {
blob := blobPath(s.root, sha1Hex)
if _, err := os.Stat(blob); err == nil {
if !verify {
// Stat, not read: the common path must not pay to open every blob that
// is already there.
if _, err := os.Stat(blob); err == nil {
return 0, nil
}
} else if stored, err := os.ReadFile(blob); err == nil && blobMatchesName(stored, sha1Hex) == nil {
return 0, nil
}
if err := os.MkdirAll(filepath.Dir(blob), 0o755); err != nil {
@@ -91,8 +103,8 @@ type zoneSink struct {
zone string
}
func (s zoneSink) putBlob(sha1Hex string, body func() []byte) (int64, error) {
key := path.Join("data", "sha1", sha1Hex[0:2], sha1Hex[2:4], sha1Hex)
func (s zoneSink) putBlob(sha1Hex string, verify bool, body func() []byte) (int64, error) {
key := blobKey(sha1Hex)
// A throttled probe must never be read as "missing, re-upload" or as
// "present, skip", so only a confirmed Present skips the upload.
state, _, err := s.store.ProbeKey(s.ctx, key)
@@ -100,7 +112,24 @@ func (s zoneSink) putBlob(sha1Hex string, body func() []byte) (int64, error) {
return 0, fmt.Errorf("probe blob %s: %w", sha1Hex, err)
}
if state == depot.Present {
return 0, nil
if !verify {
return 0, nil
}
// The probe only proved the object exists. Read it back and hold it to
// its own name.
//
// This reads the storage API rather than the pull zone: emit holds the
// write credential, and a repair decision has to be made against the
// copy it is about to overwrite, not against an edge cache of it. A
// read that fails outright is a fault, not a verdict — treating it as
// "bad, re-upload" would turn a throttled zone into a full backfill.
stored, err := s.store.GetKey(s.ctx, key)
if err != nil {
return 0, fmt.Errorf("read back blob %s: %w", sha1Hex, err)
}
if blobMatchesName(stored, sha1Hex) == nil {
return 0, nil
}
}
data := body()
if err := s.put(key, data); err != nil {
+233
View File
@@ -0,0 +1,233 @@
package nwsync
import (
"crypto/sha1"
"encoding/hex"
"errors"
"fmt"
"io"
"math/rand/v2"
"net/http"
"path"
"sort"
"strconv"
"sync"
"time"
"git.westgate.pw/ShadowsOverWestgate/sow-tools/internal/erf"
)
// defaultPullBase is the public NWSync host, which is a Bunny pull zone fronting
// the storage zone. Verify reads through it rather than through the storage API
// on purpose: what matters is the bytes a client is served, edge behaviour
// included, not what the origin believes it holds.
const defaultPullBase = "https://nwsync.westgate.pw"
// errBlobMissing marks an object the zone does not serve at all, as distinct
// from one it serves badly.
var errBlobMissing = errors.New("missing")
// blobSource reads one object out of the zone by key. Verify never writes and
// never authenticates, so this is deliberately narrower than sink.
type blobSource interface {
get(key string) ([]byte, error)
describe(key string) string
}
// pullZone reads the zone over plain HTTP, with no credential.
type pullZone struct {
base string
client *http.Client
}
func newPullZone(base string) blobSource {
if base == "" {
base = defaultPullBase
}
return pullZone{
base: base,
// A full sweep is tens of thousands of small requests, so connections
// have to be reused; the default transport does that already.
client: &http.Client{Timeout: 60 * time.Second},
}
}
func (z pullZone) describe(key string) string { return z.base + "/" + key }
func (z pullZone) get(key string) ([]byte, error) {
resp, err := z.client.Get(z.describe(key))
if err != nil {
return nil, err
}
defer resp.Body.Close()
if resp.StatusCode == http.StatusNotFound || resp.StatusCode == http.StatusGone {
_, _ = io.Copy(io.Discard, resp.Body)
return nil, errBlobMissing
}
if resp.StatusCode < 200 || resp.StatusCode >= 300 {
_, _ = io.Copy(io.Discard, resp.Body)
return nil, fmt.Errorf("unexpected status %d", resp.StatusCode)
}
return io.ReadAll(resp.Body)
}
// VerifyOptions describes one verify run.
type VerifyOptions struct {
ManifestSHA1 string // the merged manifest to verify
Base string // pull zone base URL; empty means defaultPullBase
Sample int // check this many random blobs; 0 means all of them
Jobs int // blobs in flight at once; 0 means defaultEmitJobs
Source blobSource // test seam; nil means the pull zone at Base
Log io.Writer // per-blob failures land here; nil discards them
}
// VerifyResult reports what one verify run found.
type VerifyResult struct {
Entries int // resources the manifest names
Blobs int // distinct blobs behind those resources
Checked int // blobs actually fetched
Failures int // blobs that failed a check
Bytes int64 // uncompressed bytes verified
}
// Verify reads a published manifest and its blobs the way a client reads them,
// and reports every blob that is not what the manifest says it is.
//
// Presence is not correctness. emit skips an object that already exists on the
// strength of a 1-byte range GET, so a truncated or wrongly framed object is
// skipped by every later emit forever and the backfill cannot repair it. This is
// the only thing upstream of a player's client that can tell that has happened.
//
// Every blob is decompressed and hashed. A Content-Length check would pass the
// exact failure mode being hunted — a byte-correct-looking object whose contents
// are wrong — and a round-trip check alone would pass a frame that omits its
// content size, because Go's decoder is more capable than the client's (#86).
func Verify(options VerifyOptions) (VerifyResult, error) {
// The argument is interpolated straight into a URL path, so it is checked
// rather than trusted: exactly 20 bytes of hex, nothing else.
if sum, err := hex.DecodeString(options.ManifestSHA1); err != nil || len(sum) != sha1.Size {
return VerifyResult{}, fmt.Errorf("%q is not a manifest sha1", options.ManifestSHA1)
}
source := options.Source
if source == nil {
source = newPullZone(options.Base)
}
log := options.Log
if log == nil {
log = io.Discard
}
manifestKey := path.Join("manifests", options.ManifestSHA1)
data, err := source.get(manifestKey)
if err != nil {
return VerifyResult{}, fmt.Errorf("%s: %w", source.describe(manifestKey), err)
}
// A manifest is named after its own sha1, so this catches the zone serving
// a different manifest — or a truncated one — before any blob is fetched.
if got := hex.EncodeToString(sha1Sum(data)); got != options.ManifestSHA1 {
return VerifyResult{}, fmt.Errorf("%s hashes to %s, not the manifest asked for",
source.describe(manifestKey), got)
}
entries, err := readManifest(data)
if err != nil {
return VerifyResult{}, fmt.Errorf("%s: %w", source.describe(manifestKey), err)
}
// A manifest names one blob many times over: mappings share a sha1, and so
// do resrefs with identical contents. Fetch each blob once.
blobs := make([]Entry, 0, len(entries))
seen := make(map[[20]byte]bool, len(entries))
for _, entry := range entries {
if seen[entry.SHA1] {
continue
}
seen[entry.SHA1] = true
blobs = append(blobs, entry)
}
result := VerifyResult{Entries: len(entries), Blobs: len(blobs)}
checking := blobs
if options.Sample > 0 && options.Sample < len(blobs) {
// A full sweep of the live manifest is ~69,000 objects and ~15 GB, so
// sampling is what makes verifying a routine act rather than an event.
picks := rand.Perm(len(blobs))[:options.Sample]
checking = make([]Entry, 0, options.Sample)
for _, i := range picks {
checking = append(checking, blobs[i])
}
}
result.Checked = len(checking)
jobs := options.Jobs
if jobs < 1 {
jobs = defaultEmitJobs
}
var (
mu sync.Mutex
failures []string
)
work := make(chan Entry)
var wg sync.WaitGroup
for range jobs {
wg.Go(func() {
for entry := range work {
fault := checkEntry(source, entry)
mu.Lock()
if fault != "" {
failures = append(failures, fault)
} else {
result.Bytes += int64(entry.Size)
}
mu.Unlock()
}
})
}
for _, entry := range checking {
work <- entry
}
close(work)
wg.Wait()
// Workers finish in any order; a report an operator can diff must not.
sort.Strings(failures)
for _, fault := range failures {
fmt.Fprintln(log, fault)
}
result.Failures = len(failures)
return result, nil
}
// checkEntry fetches one blob and returns a one-line fault, or "" if it is
// exactly what the manifest entry says it is.
func checkEntry(source blobSource, entry Entry) string {
// Name the resource, not just the hash: an operator has to find the thing
// in a hak, and a bare sha1 says nothing about where to look.
extension, ok := erf.ExtensionForResourceType(entry.ResType)
if !ok {
extension = strconv.Itoa(int(entry.ResType))
}
where := fmt.Sprintf("%s (%s.%s)", entry.sha1Hex(), entry.ResRef, extension)
blob, err := source.get(blobKey(entry.sha1Hex()))
if err != nil {
if errors.Is(err, errBlobMissing) {
return where + ": missing"
}
return where + ": unreadable: " + err.Error()
}
data, err := inspectBlob(blob)
if err != nil {
return where + ": " + err.Error()
}
if uint32(len(data)) != entry.Size {
return fmt.Sprintf("%s: size mismatch: %d bytes, manifest says %d", where, len(data), entry.Size)
}
if sha1.Sum(data) != entry.SHA1 {
return fmt.Sprintf("%s: hash mismatch: contents hash to %x", where, sha1.Sum(data))
}
return ""
}
func sha1Sum(data []byte) []byte {
sum := sha1.Sum(data)
return sum[:]
}
+285
View File
@@ -0,0 +1,285 @@
package nwsync
import (
"bytes"
"crypto/sha1"
"encoding/hex"
"os"
"path/filepath"
"strings"
"testing"
"github.com/klauspost/compress/zstd"
)
// verifyFixture emits two haks and a TLK into a fake zone, assembles them, and
// hands back a verifier reading that zone the way a client would.
type verifyFixture struct {
*zoneSinkFixture
manifestSHA1 string
}
func newVerifyFixture(t *testing.T) *verifyFixture {
t.Helper()
zone := newZoneFixture(t)
dir := t.TempDir()
hak := filepath.Join(dir, "sow_test_01.hak")
// A payload under 256 bytes is the one the frame-header check exists for.
writeHak(t, hak, map[string][]byte{
"bloodstain1.tga": []byte("small"),
"appearance.2da": bytes.Repeat([]byte("2DA V2.0\n"), 200),
})
tlk := filepath.Join(dir, "sow_tlk.tlk")
if err := os.WriteFile(tlk, []byte("TLK V3.0 payload"), 0o644); err != nil {
t.Fatal(err)
}
zone.emit(t, hak)
if _, err := Emit(EmitOptions{
ArtifactKey: artifactKey(t, tlk), ArtifactPath: tlk, As: "sow_tlk.tlk", Sink: zone.sink,
}); err != nil {
t.Fatalf("emit tlk: %v", err)
}
assembled, err := Assemble(AssembleOptions{
ArtifactKeys: []string{artifactKey(t, hak)}, TLKKey: artifactKey(t, tlk), Sink: zone.sink,
})
if err != nil {
t.Fatalf("assemble: %v", err)
}
return &verifyFixture{zoneSinkFixture: zone, manifestSHA1: assembled.SHA1}
}
func (f *verifyFixture) verify(t *testing.T, sample int) (VerifyResult, string, error) {
t.Helper()
var log bytes.Buffer
result, err := Verify(VerifyOptions{
ManifestSHA1: f.manifestSHA1,
Sample: sample,
Source: f.zone.pullZone(),
Log: &log,
})
return result, log.String(), err
}
// keyOf is where a resource's blob lives, addressed by the sha1 of its
// uncompressed bytes — the same path the client requests.
func keyOf(body []byte) string {
sum := sha1.Sum(body)
return blobKey(hex.EncodeToString(sum[:]))
}
func TestVerifyPassesACleanZone(t *testing.T) {
fixture := newVerifyFixture(t)
result, log, err := fixture.verify(t, 0)
if err != nil {
t.Fatalf("verify: %v", err)
}
if result.Failures != 0 {
t.Errorf("verify reported %d failures on a clean zone: %s", result.Failures, log)
}
// A default run is a full sweep, so it must reach every blob the manifest
// names — not some of them.
if result.Checked != result.Blobs || result.Blobs == 0 {
t.Errorf("checked %d of %d blobs; a full sweep must check all of them", result.Checked, result.Blobs)
}
}
func TestVerifyReportsAMissingBlob(t *testing.T) {
fixture := newVerifyFixture(t)
key := keyOf([]byte("small"))
fixture.zone.mu.Lock()
delete(fixture.zone.objects, key)
fixture.zone.mu.Unlock()
result, log, err := fixture.verify(t, 0)
if err != nil {
t.Fatalf("verify: %v", err)
}
if result.Failures != 1 {
t.Fatalf("reported %d failures, want 1: %s", result.Failures, log)
}
if !strings.Contains(log, "missing") {
t.Errorf("a deleted blob was not reported as missing: %s", log)
}
}
func TestVerifyReportsATruncatedBlob(t *testing.T) {
fixture := newVerifyFixture(t)
key := keyOf([]byte("small"))
fixture.zone.mu.Lock()
fixture.zone.objects[key] = fixture.zone.objects[key][:blobHeaderBytes+4]
fixture.zone.mu.Unlock()
result, log, err := fixture.verify(t, 0)
if err != nil {
t.Fatalf("verify: %v", err)
}
if result.Failures != 1 {
t.Fatalf("reported %d failures, want 1: %s", result.Failures, log)
}
if !strings.Contains(log, "framing") {
t.Errorf("a truncated blob was not reported as malformed framing: %s", log)
}
}
// TestVerifyRejectsABlobWithNoDeclaredFrameContentSize is the check that #86
// slipped past: the blob decompresses to exactly the right bytes, so a verifier
// that only round-trips certifies it, yet the client cannot decode it.
func TestVerifyRejectsABlobWithNoDeclaredFrameContentSize(t *testing.T) {
fixture := newVerifyFixture(t)
body := []byte("small")
key := keyOf(body)
fixture.zone.mu.Lock()
good := fixture.zone.objects[key]
encoder, err := zstd.NewWriter(nil, zstd.WithEncoderConcurrency(1))
if err != nil {
t.Fatal(err)
}
bad := append(append([]byte{}, good[:blobHeaderBytes]...), encoder.EncodeAll(body, nil)...)
fixture.zone.objects[key] = bad
fixture.zone.mu.Unlock()
if frameDeclaresContentSize(bad[blobHeaderBytes:]) {
t.Fatal("the fixture blob declares a content size; it cannot exercise the check")
}
if got, err := decompressBlob(bad); err != nil || !bytes.Equal(got, body) {
t.Fatalf("the fixture blob must round trip, or it proves nothing: %v", err)
}
result, log, err := fixture.verify(t, 0)
if err != nil {
t.Fatalf("verify: %v", err)
}
if result.Failures != 1 {
t.Fatalf("reported %d failures, want 1: %s", result.Failures, log)
}
if !strings.Contains(log, "content size") {
t.Errorf("undeclared frame content size was not the reported reason: %s", log)
}
}
func TestVerifyReportsWrongContents(t *testing.T) {
fixture := newVerifyFixture(t)
key := keyOf([]byte("small"))
fixture.zone.mu.Lock()
// Valid framing, valid zstd, wrong bytes: only decompressing and hashing
// can see this, which is why Content-Length is not enough.
fixture.zone.objects[key] = compressBlob([]byte("wrong"))
fixture.zone.mu.Unlock()
result, log, err := fixture.verify(t, 0)
if err != nil {
t.Fatalf("verify: %v", err)
}
if result.Failures != 1 {
t.Fatalf("reported %d failures, want 1: %s", result.Failures, log)
}
if !strings.Contains(log, "hash mismatch") {
t.Errorf("wrong contents were not reported as a hash mismatch: %s", log)
}
}
func TestVerifyReportsAShortBlob(t *testing.T) {
fixture := newVerifyFixture(t)
key := keyOf([]byte("small"))
fixture.zone.mu.Lock()
// Well-formed all the way down and simply too short — the shape a killed
// upload leaves behind, and the one a Content-Length check would pass.
fixture.zone.objects[key] = compressBlob([]byte("sma"))
fixture.zone.mu.Unlock()
result, log, err := fixture.verify(t, 0)
if err != nil {
t.Fatalf("verify: %v", err)
}
if result.Failures != 1 {
t.Fatalf("reported %d failures, want 1: %s", result.Failures, log)
}
if !strings.Contains(log, "size mismatch") {
t.Errorf("a short blob was not reported as a size mismatch: %s", log)
}
}
func TestVerifyFailsWhenTheManifestIsNotTheOneAsked(t *testing.T) {
fixture := newVerifyFixture(t)
fixture.zone.mu.Lock()
fixture.zone.objects["manifests/"+fixture.manifestSHA1] = []byte("NSYM garbage")
fixture.zone.mu.Unlock()
if _, _, err := fixture.verify(t, 0); err == nil {
t.Fatal("verify accepted a manifest that is not the one requested")
}
}
func TestVerifySampleChecksFewerBlobs(t *testing.T) {
fixture := newVerifyFixture(t)
result, log, err := fixture.verify(t, 1)
if err != nil {
t.Fatalf("verify: %v", err)
}
if result.Checked != 1 {
t.Errorf("--sample 1 checked %d blobs, want 1: %s", result.Checked, log)
}
if result.Blobs <= result.Checked {
t.Errorf("sampling %d of %d blobs is not a sample", result.Checked, result.Blobs)
}
}
// TestEmitVerifyReplacesABlobThatIsNotItsName covers the reason #86 could not be
// fixed by the encoder alone: emit skips whatever is already present, so every
// blob published by the broken encoder stays broken until emit stops trusting
// presence.
func TestEmitVerifyReplacesABlobThatIsNotItsName(t *testing.T) {
fixture := newZoneFixture(t)
dir := t.TempDir()
hak := filepath.Join(dir, "sow_test_01.hak")
body := []byte("blood")
writeHak(t, hak, map[string][]byte{"bloodstain1.tga": body})
emit := func(verify bool) EmitResult {
t.Helper()
result, err := Emit(EmitOptions{
ArtifactKey: artifactKey(t, hak),
ArtifactPath: hak,
Sink: fixture.sink,
Verify: verify,
})
if err != nil {
t.Fatalf("emit (verify=%v): %v", verify, err)
}
return result
}
emit(false)
key := keyOf(body)
encoder, err := zstd.NewWriter(nil, zstd.WithEncoderConcurrency(1))
if err != nil {
t.Fatal(err)
}
fixture.zone.mu.Lock()
good := fixture.zone.objects[key]
fixture.zone.objects[key] = append(append([]byte{}, good[:blobHeaderBytes]...), encoder.EncodeAll(body, nil)...)
fixture.zone.mu.Unlock()
if plain := emit(false); plain.BlobsWritten != 0 {
t.Fatalf("a plain re-emit wrote %d blobs; it is supposed to trust presence", plain.BlobsWritten)
}
if verified := emit(true); verified.BlobsWritten != 1 {
t.Fatalf("--verify wrote %d blobs, want 1 (the bad copy must be replaced)", verified.BlobsWritten)
}
fixture.zone.mu.Lock()
repaired := fixture.zone.objects[key]
fixture.zone.mu.Unlock()
if !bytes.Equal(repaired, good) {
t.Error("the replaced blob is not what the current encoder produces")
}
if _, err := inspectBlob(repaired); err != nil {
t.Errorf("the replaced blob still fails inspection: %v", err)
}
// A second verifying run has nothing left to repair.
if again := emit(true); again.BlobsWritten != 0 {
t.Errorf("--verify rewrote %d good blobs", again.BlobsWritten)
}
}
+8
View File
@@ -21,6 +21,13 @@ type fakeZone struct {
objects map[string][]byte
puts []string
failOn func(key string) bool // when true, the PUT fails
url string // base the same objects are readable at
}
// pullZone reads the fake zone the way the public pull zone is read: plain
// unauthenticated GETs, no storage API.
func (z *fakeZone) pullZone() blobSource {
return newPullZone(z.url)
}
func newFakeZone(t *testing.T) (*fakeZone, func(string) string) {
@@ -28,6 +35,7 @@ func newFakeZone(t *testing.T) (*fakeZone, func(string) string) {
zone := &fakeZone{objects: map[string][]byte{}}
server := httptest.NewServer(zone)
t.Cleanup(server.Close)
zone.url = server.URL + "/sow-nwsync"
getenv := func(name string) string {
switch name {
case "NWSYNC_STORAGE_ZONE":
+7 -1
View File
@@ -126,8 +126,14 @@ func extractArchiveResources(p *project.Project, archive erf.Archive, desired ma
}
// *palcus.itp are Toolset-generated palette projections; the module
// build regenerates them from source blueprints, so extraction never
// writes them back into source.
// writes them back into source. The committed skeleton still has to
// survive stale cleanup, so mark its target as desired.
if ext == "itp" && isPaletteProjectionResref(resource.Name) {
effective := p.EffectiveConfig()
target, err := extractionTarget(p, "paths.source", effective.Paths.Source, p.SourceDir(), sourceSubdir(ext), strings.ToLower(resource.Name)+"."+ext+".json")
if err == nil {
desired[target] = struct{}{}
}
skippedCount++
continue
}
+82
View File
@@ -3312,6 +3312,88 @@ func TestExtractOverwritesAndRemovesStaleFiles(t *testing.T) {
}
}
func TestExtractKeepsPaletteSkeletonsThroughStaleCleanup(t *testing.T) {
root := t.TempDir()
mustMkdir(t, filepath.Join(root, "src", "module"))
mustMkdir(t, filepath.Join(root, "src", "palettes"))
mustMkdir(t, filepath.Join(root, "assets"))
mustMkdir(t, filepath.Join(root, "build"))
mustWriteFile(t, filepath.Join(root, "nwn-tool.json"), `{
"module": {
"name": "Test Module",
"resref": "testmod"
},
"paths": {
"source": "src",
"assets": "assets",
"build": "build"
}
}
`)
mustWriteFile(t, filepath.Join(root, "src", "module", "module.ifo.json"), `{
"file_type": "IFO ",
"file_version": "V3.2",
"root": {
"struct_type": 0,
"fields": [
{
"label": "Mod_Name",
"type": "CExoString",
"value": "Original Module"
}
]
}
}
`)
skeletonPath := filepath.Join(root, "src", "palettes", "creaturepalcus.itp.json")
mustWriteFile(t, skeletonPath, `{
"file_type": "ITP ",
"file_version": "V3.2",
"root": {
"struct_type": 0,
"fields": [
{
"label": "MAIN",
"type": "List",
"value": []
}
]
}
}
`)
p, err := project.Load(root)
if err != nil {
t.Fatalf("load project: %v", err)
}
if err := p.ValidateLayout(); err != nil {
t.Fatalf("validate layout: %v", err)
}
if err := p.Scan(); err != nil {
t.Fatalf("scan: %v", err)
}
if _, err := BuildModule(p); err != nil {
t.Fatalf("build module: %v", err)
}
if err := p.Scan(); err != nil {
t.Fatalf("rescan before extract: %v", err)
}
result, err := Extract(p)
if err != nil {
t.Fatalf("extract: %v", err)
}
if result.Removed != 0 {
t.Fatalf("expected no removed files, got %d", result.Removed)
}
if _, err := os.Stat(skeletonPath); err != nil {
t.Fatalf("expected palette skeleton to survive extract, stat err=%v", err)
}
}
func TestExtractMergesConfiguredGFFJSONFieldsAndLists(t *testing.T) {
root := t.TempDir()
mustMkdir(t, filepath.Join(root, "src", "module"))
+12 -2
View File
@@ -560,8 +560,18 @@ func (p *Project) ValidateLayout() error {
if strings.TrimSpace(p.Config.Module.ResRef) == "" {
failures = append(failures, errors.New("module.resref is required"))
}
if len(p.Config.Module.ResRef) > 16 {
failures = append(failures, fmt.Errorf("module.resref %q exceeds 16 characters", p.Config.Module.ResRef))
// module.resref names the built .mod FILE, so the 16-byte resref limit does not
// apply to it — NWN:EE module file names are routinely longer. It is validated as
// a file name instead. The limit still binds when the same value has to be a real
// resref: with no haks configured, an asset project names its single generated HAK
// after it, and a HAK name is a resref the engine loads.
if err := validateOutputFileName("module.resref", p.Config.Module.ResRef+".mod", ".mod"); err != nil {
failures = append(failures, err)
}
if len(p.Config.Module.ResRef) > 16 && strings.TrimSpace(p.Config.Paths.Assets) != "" && len(p.Config.HAKs) == 0 {
failures = append(failures, fmt.Errorf(
"module.resref %q exceeds 16 characters and would name this project's generated HAK; configure haks[] with a shorter name",
p.Config.Module.ResRef))
}
if strings.TrimSpace(p.Config.Paths.Source) == "" && strings.TrimSpace(p.Config.Paths.Assets) == "" && !p.HasTopData() {
failures = append(failures, errors.New("at least one of paths.source, paths.assets, or topdata.source is required"))
+72
View File
@@ -1093,6 +1093,78 @@ func TestValidateLayoutAllowsMissingAssetsDir(t *testing.T) {
}
}
// module.resref names the built .mod FILE, not a resource inside an archive, so the
// 16-byte resref limit does not apply to it. NWN:EE module file names are commonly
// longer (ShadowsOverWestgate.mod is 19). The limit still binds everywhere a resref
// really is a resref — see TestValidateLayoutRejectsLongResRefWhenItNamesAHAK.
func TestValidateLayoutAllowsLongModuleResRef(t *testing.T) {
root := t.TempDir()
mkdirAll(t, filepath.Join(root, "src"))
mkdirAll(t, filepath.Join(root, "build"))
proj := &Project{
Root: root,
Config: Config{
Module: ModuleConfig{Name: "Shadows Over Westgate", ResRef: "ShadowsOverWestgate"},
Paths: PathConfig{Source: "src", Build: "build"},
},
}
if err := proj.ValidateLayout(); err != nil {
t.Fatalf("ValidateLayout rejected a 19-character module file name: %v", err)
}
if got, want := filepath.Base(proj.ModuleArchivePath()), "ShadowsOverWestgate.mod"; got != want {
t.Fatalf("ModuleArchivePath() = %q, want %q", got, want)
}
}
// A module.resref that is not a usable file name is still rejected.
func TestValidateLayoutRejectsModuleResRefThatIsAPath(t *testing.T) {
root := t.TempDir()
mkdirAll(t, filepath.Join(root, "src"))
proj := &Project{
Root: root,
Config: Config{
Module: ModuleConfig{Name: "Test", ResRef: "../escape/mod"},
Paths: PathConfig{Source: "src", Build: "build"},
},
}
err := proj.ValidateLayout()
if err == nil {
t.Fatal("ValidateLayout accepted a module.resref containing a path")
}
if !strings.Contains(err.Error(), "module.resref") {
t.Fatalf("error does not name the offending field: %v", err)
}
}
// When a project declares no haks, the module resref becomes the name of the single
// generated HAK — and a HAK name IS a resref the engine loads. The limit applies
// there, so a long name is only allowed for projects that build no HAKs.
func TestValidateLayoutRejectsLongResRefWhenItNamesAHAK(t *testing.T) {
root := t.TempDir()
mkdirAll(t, filepath.Join(root, "src"))
mkdirAll(t, filepath.Join(root, "assets"))
proj := &Project{
Root: root,
Config: Config{
Module: ModuleConfig{Name: "Shadows Over Westgate", ResRef: "ShadowsOverWestgate"},
Paths: PathConfig{Source: "src", Assets: "assets", Build: "build"},
},
}
err := proj.ValidateLayout()
if err == nil {
t.Fatal("ValidateLayout accepted a 19-character name for a generated HAK")
}
if !strings.Contains(err.Error(), "16") {
t.Fatalf("error does not explain the resref limit: %v", err)
}
}
// paths.build is an OUTPUT dir the builder creates (MkdirAll) before writing, so
// a bare clone with no build dir yet must still validate/build with no pre-step
// (R2/parity). Only a build path that exists but is not a directory is an error.
+2 -1
View File
@@ -98,8 +98,9 @@ func buildGenerated2DAAssetGroup(p *project.Project, cfg project.GeneratedTopDat
}
results := make([]Generated2DAAsset, 0, len(collected))
racialFeatRules := racialUsableFeatRules(collected)
for _, dataset := range collected {
compiled, err := resolveNativeDataset(dataset, keyToID, rowByKey, tableRegistry, nil, project.TopDataClassFeatInjectionConfig{}, nil)
compiled, err := resolveNativeDataset(dataset, keyToID, rowByKey, tableRegistry, nil, project.TopDataClassFeatInjectionConfig{}, racialFeatRules, nil)
if err != nil {
return nil, err
}
+59 -4
View File
@@ -530,6 +530,7 @@ func buildNativeUnchecked(p *project.Project, opts NativeBuildOptions, progress
groupStats := nativeCompileGroupStats(collected)
currentGroup := ""
sidecars := newNativeSidecarCollector()
racialFeatRules := racialUsableFeatRules(collected)
for _, dataset := range collected {
group := nativeCompileGroup(dataset.Dataset.Name)
if group != currentGroup {
@@ -542,7 +543,7 @@ func buildNativeUnchecked(p *project.Project, opts NativeBuildOptions, progress
stats.SourceFragments,
))
}
compiled, err := resolveNativeDataset(dataset, globalKeyToID, globalRowByKey, tableRegistry, compiler, p.EffectiveConfig().TopData.ClassFeatInjections, sidecars)
compiled, err := resolveNativeDataset(dataset, globalKeyToID, globalRowByKey, tableRegistry, compiler, p.EffectiveConfig().TopData.ClassFeatInjections, racialFeatRules, sidecars)
if err != nil {
return BuildResult{}, err
}
@@ -3996,13 +3997,13 @@ func normalizeGlobalReferenceID(value string) string {
return parts[0] + ":" + strings.ReplaceAll(parts[1], "_", "")
}
func resolveNativeDataset(dataset nativeCollectedDataset, keyToID map[string]int, globalRowByKey map[string]map[string]any, tableRegistry resolvedTableRegistry, compiler *tlkCompiler, classFeatInjections project.TopDataClassFeatInjectionConfig, sidecars *nativeSidecarCollector) (map[string]any, error) {
func resolveNativeDataset(dataset nativeCollectedDataset, keyToID map[string]int, globalRowByKey map[string]map[string]any, tableRegistry resolvedTableRegistry, compiler *tlkCompiler, classFeatInjections project.TopDataClassFeatInjectionConfig, racialFeatRules []project.TopDataClassFeatGlobalRule, sidecars *nativeSidecarCollector) (map[string]any, error) {
rows := dataset.Rows
if strings.HasPrefix(filepath.ToSlash(dataset.Dataset.Name), "classes/feats/") {
classKey := "classes:" + dataset.Dataset.Name[strings.LastIndex(dataset.Dataset.Name, "/")+1:]
featSuccessors := buildFeatSuccessorsIndex(globalRowByKey, keyToID)
classSkills := buildClassSkillsIndex(tableRegistry, classKey)
expanded, err := expandClassesFeatRows(rows, keyToID, globalRowByKey, featSuccessors, classSkills, globalRowByKey, classKey, classFeatInjections, !dataset.Dataset.HasGlobalInjections)
expanded, err := expandClassesFeatRows(rows, keyToID, globalRowByKey, featSuccessors, classSkills, globalRowByKey, classKey, classFeatInjections, racialFeatRules, !dataset.Dataset.HasGlobalInjections)
if err != nil {
return nil, fmt.Errorf("dataset %s: %w", dataset.Dataset.Name, err)
}
@@ -4051,12 +4052,18 @@ var (
}
)
func expandClassesFeatRows(rows []map[string]any, keyToID map[string]int, rowByKey map[string]map[string]any, featSuccessors map[string]string, classSkills map[string]bool, allRowByKey map[string]map[string]any, classKey string, classFeatInjections project.TopDataClassFeatInjectionConfig, useConfiguredInjections bool) ([]map[string]any, error) {
func expandClassesFeatRows(rows []map[string]any, keyToID map[string]int, rowByKey map[string]map[string]any, featSuccessors map[string]string, classSkills map[string]bool, allRowByKey map[string]map[string]any, classKey string, classFeatInjections project.TopDataClassFeatInjectionConfig, racialFeatRules []project.TopDataClassFeatGlobalRule, useConfiguredInjections bool) ([]map[string]any, error) {
globalRules, classSkillRules := []project.TopDataClassFeatGlobalRule{}, []project.TopDataClassFeatMasterfeatRule{}
if useConfiguredInjections {
globalRules, classSkillRules = effectiveClassFeatInjectionRules(classFeatInjections)
}
// Racial usable-feat rows are generated from the race feats tables, not the
// hand-authored class-feat injections, so they apply to every class table
// regardless of whether that dataset carries its own global.json (which is
// what gates useConfiguredInjections). Deduped below against rows already
// present, so a leftover hand row in global.json is a no-op, not a double.
globalRules = append(globalRules, racialFeatRules...)
injected := make([]map[string]any, 0, len(globalRules)+len(classSkillRules))
presentRefIDs := make(map[string]struct{}, len(rows))
for _, row := range rows {
@@ -4134,6 +4141,54 @@ func expandClassesFeatRows(rows []map[string]any, keyToID map[string]int, rowByK
return combined, nil
}
// racialUsableFeatRules builds one class-feat injection rule per feat marked
// UsableFeat=1 in any race feats table (race_feat_*.2da). Racial feats are
// granted by race, never by a class, so an activatable one needs a menu-only
// cls_feat row to reach the client radial: List=3 keeps it off every level-up
// selection list, GrantedOnLevel=99 sits above the level cap so no class ever
// actually grants it, and OnMenu=1 renders the button once the creature holds
// the feat (possession comes from chargen / the login racial-feat sync). This
// replaces the hand-maintained racial rows in classes/feats/global.json - mark
// UsableFeat in the race table and the radial row follows automatically.
func racialUsableFeatRules(collected []nativeCollectedDataset) []project.TopDataClassFeatGlobalRule {
seen := map[string]struct{}{}
rules := []project.TopDataClassFeatGlobalRule{}
for _, ds := range collected {
if !strings.HasPrefix(ds.Dataset.OutputName, "race_feat_") || !strings.HasSuffix(ds.Dataset.OutputName, ".2da") {
continue
}
for _, row := range ds.Rows {
if usable, err := asInt(fieldValue(row, "UsableFeat")); err != nil || usable != 1 {
continue
}
featRef, ok := row["FeatIndex"].(map[string]any)
if !ok {
continue
}
featID, _ := featRef["id"].(string)
if featID == "" {
continue
}
if _, dup := seen[featID]; dup {
continue
}
seen[featID] = struct{}{}
rules = append(rules, project.TopDataClassFeatGlobalRule{
Feat: featID,
List: "3",
GrantedOnLevel: "99",
OnMenu: "1",
})
}
}
// Discovery order across files and the dedup map are both unordered; sort so
// the injected rows (and the resulting 2DA row numbering) are deterministic.
slices.SortFunc(rules, func(a, b project.TopDataClassFeatGlobalRule) int {
return strings.Compare(a.Feat, b.Feat)
})
return rules
}
func effectiveClassFeatInjectionRules(config project.TopDataClassFeatInjectionConfig) ([]project.TopDataClassFeatGlobalRule, []project.TopDataClassFeatMasterfeatRule) {
if len(config.GlobalFeats) == 0 && len(config.ClassSkillMasterfeats) == 0 {
return defaultClassFeatGlobalRules, defaultClassFeatClassSkillMasterfeatRules
@@ -0,0 +1,54 @@
package topdata
import (
"reflect"
"testing"
"git.westgate.pw/ShadowsOverWestgate/sow-tools/internal/project"
)
func raceFeatDataset(output string, rows ...map[string]any) nativeCollectedDataset {
return nativeCollectedDataset{
Dataset: nativeDataset{OutputName: output},
Rows: rows,
}
}
func usableRow(featID string, usable any) map[string]any {
row := map[string]any{"FeatIndex": map[string]any{"id": featID}}
if usable != nil {
row["UsableFeat"] = usable
}
return row
}
func TestRacialUsableFeatRules(t *testing.T) {
collected := []nativeCollectedDataset{
raceFeatDataset("race_feat_ddrw.2da",
usableRow("feat:keen_sense", nil), // passive, no UsableFeat -> skipped
usableRow("feat:darkvision", 1), // usable
usableRow("feat:use_poison", 0), // explicitly not usable -> skipped
usableRow("feat:drow/faerie_fire", "1"),// usable, string form
),
raceFeatDataset("race_feat_tief.2da",
usableRow("feat:darkvision", 1), // duplicate across races -> collapses to one
usableRow("feat:tiefling/darkness", 1),
),
raceFeatDataset("feat.2da", // not a race feats table -> ignored entirely
usableRow("feat:power_attack", 1),
),
}
got := racialUsableFeatRules(collected)
want := []project.TopDataClassFeatGlobalRule{
{Feat: "feat:darkvision", List: "3", GrantedOnLevel: "99", OnMenu: "1"},
{Feat: "feat:drow/faerie_fire", List: "3", GrantedOnLevel: "99", OnMenu: "1"},
{Feat: "feat:tiefling/darkness", List: "3", GrantedOnLevel: "99", OnMenu: "1"},
}
// Slice is sorted by Feat, so order is deterministic.
if !reflect.DeepEqual(got, want) {
t.Fatalf("rules: got %+v, want %+v", got, want)
}
}
+1
View File
@@ -1523,6 +1523,7 @@ func TestResolveNativeDatasetPreservesScalarTableReferenceBehavior(t *testing.T)
nil,
project.TopDataClassFeatInjectionConfig{},
nil,
nil,
)
if err != nil {
t.Fatalf("resolveNativeDataset failed: %v", err)
+127 -28
View File
@@ -8,6 +8,7 @@ import (
"fmt"
"html"
"io"
"maps"
"net/http"
"net/url"
"os"
@@ -52,7 +53,7 @@ type DeployWikiOptions struct {
TitlePrefixMinLength int
}
type deployResult struct {
type DeployResult struct {
LocalPages int
Created int
Updated int
@@ -63,6 +64,19 @@ type deployResult struct {
Drifted int
Renamed int
Manifest string
// ResetPurged counts the deletions queued by --reset-managed-namespaces.
// They are also included in Stale and Purged, because callers warn about
// destructive policies in terms of the stale count: sow-topdata's
// deploy-wiki wrapper prints "of the pages counted as 'stale' below, they
// will be DELETED from NodeBB" directly above this block.
ResetPurged int
// ResetUnrecognized counts the subset of ResetPurged that the deploy
// manifest has no record of writing. Those deletions are the ones a
// re-seed cannot undo.
ResetUnrecognized int
// ResetSkipped counts reset targets NodeBB refused to delete, such as the
// wiki home topic, which the plugin excludes from tombstone and purge.
ResetSkipped int
}
type wikiDeployPage struct {
@@ -100,6 +114,12 @@ type wikiDeployPlan struct {
Content string
RemoteHash string
Title string
// Reset marks a purge queued by --reset-managed-namespaces rather than by
// stale computation over the manifest.
Reset bool
// Unrecognized marks a reset purge whose topic the manifest has no record
// of writing.
Unrecognized bool
}
type wikiNamespacesDocument struct {
@@ -116,7 +136,7 @@ type wikiNamespaceDeclaration struct {
EditPolicy string `json:"edit_policy" yaml:"edit_policy"`
}
func DeployWikiWithOptions(p *project.Project, opts DeployWikiOptions, progress func(string)) (deployResult, error) {
func DeployWikiWithOptions(p *project.Project, opts DeployWikiOptions, progress func(string)) (DeployResult, error) {
if progress == nil {
progress = func(string) {}
}
@@ -132,19 +152,19 @@ func DeployWikiWithOptions(p *project.Project, opts DeployWikiOptions, progress
}
}
if _, err := os.Stat(opts.SourceDir); err != nil {
return deployResult{}, fmt.Errorf("wiki source directory not found: %w", err)
return DeployResult{}, fmt.Errorf("wiki source directory not found: %w", err)
}
if opts.Endpoint == "" {
return deployResult{}, errors.New("NODEBB_API_ENDPOINT is required")
return DeployResult{}, errors.New("NODEBB_API_ENDPOINT is required")
}
if opts.Token == "" {
return deployResult{}, errors.New("NODEBB_API_TOKEN is required")
return DeployResult{}, errors.New("NODEBB_API_TOKEN is required")
}
if opts.Username != "" || opts.Password != "" || opts.NotesDelimiter != "" {
return deployResult{}, errors.New("DokuWiki deployment options are no longer supported; use NodeBB endpoint, token, and category mappings")
return DeployResult{}, errors.New("DokuWiki deployment options are no longer supported; use NodeBB endpoint, token, and category mappings")
}
if opts.StalePolicy != "" && opts.StalePolicy != "report" && opts.StalePolicy != "archive" && opts.StalePolicy != "purge" {
return deployResult{}, fmt.Errorf("wiki stale policy %q is not supported", opts.StalePolicy)
return DeployResult{}, fmt.Errorf("wiki stale policy %q is not supported", opts.StalePolicy)
}
if opts.TitlePrefixMinLength <= 0 {
opts.TitlePrefixMinLength = p.EffectiveConfig().TopData.Wiki.TitlePrefixMinLength
@@ -154,7 +174,7 @@ func DeployWikiWithOptions(p *project.Project, opts DeployWikiOptions, progress
if len(namespaces) == 0 {
declarations, err := loadWikiNamespaceDeclarations(p)
if err != nil {
return deployResult{}, err
return DeployResult{}, err
}
for _, declaration := range declarations {
namespaces = append(namespaces, declaration.ID)
@@ -164,7 +184,7 @@ func DeployWikiWithOptions(p *project.Project, opts DeployWikiOptions, progress
}
envCategories, err := categoryIDsFromNamespaceEnv(declarations)
if err != nil {
return deployResult{}, err
return DeployResult{}, err
}
if len(envCategories) > 0 {
merged := envCategories
@@ -183,7 +203,7 @@ func DeployWikiWithOptions(p *project.Project, opts DeployWikiOptions, progress
progress(fmt.Sprintf("Collecting local wiki pages from %s", opts.SourceDir))
pages, err := collectLocalPages(opts.SourceDir, opts.PageIndexPath, namespaces)
if err != nil {
return deployResult{}, err
return DeployResult{}, err
}
progress(fmt.Sprintf("Loaded %d local wiki page(s)", len(pages)))
manifest := loadDeployManifest(manifestPath)
@@ -193,7 +213,7 @@ func DeployWikiWithOptions(p *project.Project, opts DeployWikiOptions, progress
progress(fmt.Sprintf("Planning NodeBB wiki deploy for %d local page(s)", len(pages)))
plans, result, nextManifest, err := planNodeBBDeploy(pages, manifest, opts, client, progress)
if err != nil {
return deployResult{}, err
return DeployResult{}, err
}
result.LocalPages = len(pages)
result.Manifest = manifestPath
@@ -212,6 +232,8 @@ func DeployWikiWithOptions(p *project.Project, opts DeployWikiOptions, progress
}
orderedPlans := orderNodeBBDeployPlans(plans)
progress(fmt.Sprintf("Executing NodeBB wiki actions: total %d, create %d, update %d, rename %d, archive %d, purge %d", len(orderedPlans), result.Created, result.Updated, result.Renamed, result.Archived, result.Purged))
resetPurgedByCID := map[int]int{}
resetSkippedByCID := map[int]int{}
for i, plan := range orderedPlans {
if shouldReportNodeBBDeployActionProgress(i, len(orderedPlans)) {
progress(fmt.Sprintf("Executing NodeBB wiki action %d/%d: %s %s", i+1, len(orderedPlans), plan.Action, plan.Page.PageID))
@@ -242,19 +264,52 @@ func DeployWikiWithOptions(p *project.Project, opts DeployWikiOptions, progress
return result, fmt.Errorf("deploy wiki page %q: update NodeBB post %d: %w", plan.Page.PageID, plan.Entry.PID, err)
}
case "archive":
// Archiving rewrites the page rather than removing it, so it goes
// through the ordinary post edit the wiki plugin allows; only
// delete, restore, and purge are reserved to the page actions.
if err := client.updatePost(plan.Entry.TID, plan.Entry.PID, plan.Content, summary); err != nil {
return result, fmt.Errorf("deploy wiki page %q: archive NodeBB post %d: %w", plan.Page.PageID, plan.Entry.PID, err)
}
case "purge":
if err := client.purgeTopic(plan.Entry.TID); err != nil {
if err := client.purgeWikiPage(plan.Entry.TID); err != nil {
// A namespace reset sweeps every topic in the category, so it
// can reach pages NodeBB will not delete at all — the wiki home
// topic above all. Skip those rather than abandoning the reset.
if plan.Reset && isNodeBBWikiPageUndeletable(err) {
result.ResetSkipped++
result.ResetPurged--
result.Purged--
result.Stale--
if plan.Unrecognized {
result.ResetUnrecognized--
}
resetSkippedByCID[plan.Entry.CID]++
progress(fmt.Sprintf("NodeBB refused to delete wiki topic %d during managed namespace reset; skipping it", plan.Entry.TID))
continue
}
return result, fmt.Errorf("deploy wiki page %q: purge NodeBB topic %d: %w", plan.Page.PageID, plan.Entry.TID, err)
}
if plan.Reset {
resetPurgedByCID[plan.Entry.CID]++
}
case "rename":
if err := client.renameWikiPage(plan.Entry.TID, plan.Entry.CID, plan.Title); err != nil {
return result, fmt.Errorf("deploy wiki page %q: rename NodeBB topic %d to %q: %w", plan.Page.PageID, plan.Entry.TID, plan.Title, err)
}
}
}
// NodeBB answers 403 both for the pages it will never delete — the wiki
// home topic — and for a token without purge privileges, and the two are
// not distinguishable from the response. What tells them apart is scope: a
// category where nothing at all could be deleted is a privilege problem,
// not a home page. Reporting that as a completed reset would leave the
// manifest claiming a fresh start over pages that are all still there, so
// the next deploy would recreate every one of them as a duplicate.
for _, cid := range slices.Sorted(maps.Keys(resetSkippedByCID)) {
if resetPurgedByCID[cid] == 0 {
return result, fmt.Errorf("NodeBB refused every managed namespace reset deletion in category %d (%d topic(s)); check that the deploy token has wiki purge privileges there", cid, resetSkippedByCID[cid])
}
}
if err := saveDeployManifest(manifestPath, nextManifest); err != nil {
return result, err
}
@@ -486,7 +541,7 @@ func isDir(path string) bool {
return err == nil && info.IsDir()
}
func planNodeBBDeploy(pages map[string]wikiDeployPage, manifest wikiDeployManifest, opts DeployWikiOptions, client *nodeBBClient, progress func(string)) ([]wikiDeployPlan, deployResult, wikiDeployManifest, error) {
func planNodeBBDeploy(pages map[string]wikiDeployPage, manifest wikiDeployManifest, opts DeployWikiOptions, client *nodeBBClient, progress func(string)) ([]wikiDeployPlan, DeployResult, wikiDeployManifest, error) {
if progress == nil {
progress = func(string) {}
}
@@ -494,17 +549,23 @@ func planNodeBBDeploy(pages map[string]wikiDeployPage, manifest wikiDeployManife
next := wikiDeployManifest{Version: "nodebb-v1", Pages: map[string]wikiDeployManifestPage{}}
remotePagesByCID := map[int][]nodeBBWikiPage{}
var plans []wikiDeployPlan
var result deployResult
var result DeployResult
if opts.ResetManagedNamespaces {
if !opts.AllowCreates && len(pageIDs) > 0 {
return nil, result, next, errors.New("wiki managed namespace reset requires --create so current generated pages can be recreated")
}
resetPlans, purged, err := planManagedNamespaceReset(opts, client, progress)
resetPlans, purged, unrecognized, err := planManagedNamespaceReset(opts, manifest, client, progress)
if err != nil {
return nil, result, next, err
}
plans = append(plans, resetPlans...)
result.Purged += purged
// A namespace reset deletes remote pages the same way stale purge does,
// so it is counted as stale: that is the number callers word their
// destructive-policy warning around.
result.Stale += purged
result.ResetPurged = purged
result.ResetUnrecognized = unrecognized
manifest = wikiDeployManifest{Version: "nodebb-v1", Pages: map[string]wikiDeployManifestPage{}}
for _, cid := range opts.CategoryIDs {
if cid != 0 {
@@ -682,7 +743,11 @@ func planNodeBBDeploy(pages map[string]wikiDeployPage, manifest wikiDeployManife
return plans, result, next, nil
}
func planManagedNamespaceReset(opts DeployWikiOptions, client *nodeBBClient, progress func(string)) ([]wikiDeployPlan, int, error) {
// planManagedNamespaceReset queues a purge for every topic in each managed
// category, not only the ones the manifest says we wrote. The second return is
// the plan count; the third is how many of those topics the manifest has no
// record of, which is the subset a re-seed cannot put back.
func planManagedNamespaceReset(opts DeployWikiOptions, manifest wikiDeployManifest, client *nodeBBClient, progress func(string)) ([]wikiDeployPlan, int, int, error) {
namespaces := slices.Clone(opts.Namespaces)
if len(namespaces) == 0 {
for namespace := range opts.CategoryIDs {
@@ -691,7 +756,15 @@ func planManagedNamespaceReset(opts DeployWikiOptions, client *nodeBBClient, pro
}
slices.Sort(namespaces)
manifestTIDs := map[int]struct{}{}
for _, entry := range manifest.Pages {
if entry.TID != 0 {
manifestTIDs[entry.TID] = struct{}{}
}
}
seenTIDs := map[int]struct{}{}
unrecognized := 0
var plans []wikiDeployPlan
for _, namespace := range namespaces {
cid := opts.CategoryIDs[namespace]
@@ -701,7 +774,7 @@ func planManagedNamespaceReset(opts DeployWikiOptions, client *nodeBBClient, pro
progress(fmt.Sprintf("Listing NodeBB wiki namespace category %d for managed reset", cid))
remotePages, err := client.listNamespacePages(cid)
if err != nil {
return nil, 0, err
return nil, 0, 0, err
}
slices.SortFunc(remotePages, func(a, b nodeBBWikiPage) int {
return a.TID - b.TID
@@ -714,19 +787,25 @@ func planManagedNamespaceReset(opts DeployWikiOptions, client *nodeBBClient, pro
continue
}
seenTIDs[remotePage.TID] = struct{}{}
_, known := manifestTIDs[remotePage.TID]
if !known {
unrecognized++
}
title := strings.TrimSpace(remotePage.Title)
if title == "" {
title = strings.TrimSpace(remotePage.TitleLeaf)
}
pageID := fmt.Sprintf("%s:reset-topic-%d", namespace, remotePage.TID)
plans = append(plans, wikiDeployPlan{
Page: wikiDeployPage{PageID: pageID, Title: title, Namespace: namespace},
Entry: wikiDeployManifestPage{TID: remotePage.TID, CID: cid, Namespace: namespace, Title: title},
Action: "purge",
Page: wikiDeployPage{PageID: pageID, Title: title, Namespace: namespace},
Entry: wikiDeployManifestPage{TID: remotePage.TID, CID: cid, Namespace: namespace, Title: title},
Action: "purge",
Reset: true,
Unrecognized: !known,
})
}
}
return plans, len(plans), nil
return plans, len(plans), unrecognized, nil
}
func findMappedRemoteTopic(entry wikiDeployManifestPage, client *nodeBBClient) (nodeBBPost, bool, error) {
@@ -1611,16 +1690,36 @@ func (c *nodeBBClient) renameWikiPage(tid, cid int, title string) error {
return c.request("PUT", "/api/v3/plugins/westgate-wiki/page/move", body, nil)
}
func (c *nodeBBClient) purgeTopic(tid int) error {
// purgeWikiPage deletes a wiki topic through the wiki plugin's own page
// actions. The core DELETE /api/v3/topics/{tid} route is refused for topics in
// wiki categories, because revision history is plugin-owned, so a purge has to
// tombstone the page first and then hard-purge it.
func (c *nodeBBClient) purgeWikiPage(tid int) error {
if tid == 0 {
return fmt.Errorf("NodeBB topic purge requires topic id")
return fmt.Errorf("NodeBB wiki page purge requires topic id")
}
err := c.request(http.MethodDelete, fmt.Sprintf("/api/v3/topics/%d", tid), nil, nil)
body := map[string]any{"tid": tid}
if err := c.request(http.MethodPut, "/api/v3/plugins/westgate-wiki/page/tombstone", body, nil); err != nil {
if isNodeBBMissingResource(err) {
return nil
}
return err
}
if err := c.request(http.MethodDelete, "/api/v3/plugins/westgate-wiki/page/hard-purge", body, nil); err != nil {
if isNodeBBMissingResource(err) {
return nil
}
return err
}
return nil
}
// isNodeBBWikiPageUndeletable reports whether NodeBB refused to delete the page
// outright rather than failing transiently. The wiki home topic answers this
// way: the plugin excludes it from tombstone, restore, and purge alike.
func isNodeBBWikiPageUndeletable(err error) bool {
var httpErr nodeBBHTTPError
if errors.As(err, &httpErr) && (httpErr.Status == http.StatusNotFound || httpErr.Status == http.StatusGone) {
return nil
}
return err
return errors.As(err, &httpErr) && httpErr.Status == http.StatusForbidden
}
func (c *nodeBBClient) acquireEditLock(tid int) (nodeBBEditLock, error) {
+281 -90
View File
@@ -2,6 +2,7 @@ package topdata
import (
"encoding/json"
"fmt"
"net/http"
"net/http/httptest"
"os"
@@ -35,7 +36,7 @@ func TestDeployWikiDryRunDoesNotWriteRemoteOrManifest(t *testing.T) {
}
updateCalls := 0
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
server := newFakeNodeBB(t, func(w http.ResponseWriter, r *http.Request) {
if got := r.Header.Get("Authorization"); got != "Bearer nodebb-token" {
t.Fatalf("unexpected authorization header %q", got)
}
@@ -50,7 +51,7 @@ func TestDeployWikiDryRunDoesNotWriteRemoteOrManifest(t *testing.T) {
default:
t.Fatalf("unexpected request %s %s", r.Method, r.URL.Path)
}
}))
})
defer server.Close()
result, err := DeployWikiWithOptions(&project.Project{}, DeployWikiOptions{
@@ -92,7 +93,7 @@ func TestDeployWikiReportsPlanningProgressBeforeRemoteWork(t *testing.T) {
}
progress := []string{}
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
server := newFakeNodeBB(t, func(w http.ResponseWriter, r *http.Request) {
switch {
case r.Method == http.MethodGet && r.URL.Path == "/api/v3/plugins/westgate-wiki/namespace/3/pages":
w.Header().Set("Content-Type", "application/json")
@@ -100,7 +101,7 @@ func TestDeployWikiReportsPlanningProgressBeforeRemoteWork(t *testing.T) {
default:
t.Fatalf("unexpected request %s %s", r.Method, r.URL.String())
}
}))
})
defer server.Close()
_, err := DeployWikiWithOptions(&project.Project{}, DeployWikiOptions{
@@ -174,7 +175,7 @@ func TestDeployWikiReportsLiveExecutionProgress(t *testing.T) {
}
progress := []string{}
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
server := newFakeNodeBB(t, func(w http.ResponseWriter, r *http.Request) {
switch {
case r.Method == http.MethodGet && r.URL.Path == "/api/v3/topics/7":
w.Header().Set("Content-Type", "application/json")
@@ -194,7 +195,7 @@ func TestDeployWikiReportsLiveExecutionProgress(t *testing.T) {
default:
t.Fatalf("unexpected request %s %s", r.Method, r.URL.String())
}
}))
})
defer server.Close()
result, err := DeployWikiWithOptions(&project.Project{}, DeployWikiOptions{
@@ -228,7 +229,7 @@ func TestDeployWikiReportsLiveExecutionProgress(t *testing.T) {
func TestNodeBBNamespacePaginationStopsWhenRepeatedCursorReturnsNoNewPages(t *testing.T) {
requests := 0
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
server := newFakeNodeBB(t, func(w http.ResponseWriter, r *http.Request) {
requests++
w.Header().Set("Content-Type", "application/json")
_ = json.NewEncoder(w).Encode(map[string]any{
@@ -240,7 +241,7 @@ func TestNodeBBNamespacePaginationStopsWhenRepeatedCursorReturnsNoNewPages(t *te
"nextCursor": "same-cursor",
},
})
}))
})
defer server.Close()
client := newNodeBBClient(server.URL, "nodebb-token")
@@ -258,7 +259,7 @@ func TestNodeBBNamespacePaginationStopsWhenRepeatedCursorReturnsNoNewPages(t *te
func TestNodeBBNamespacePaginationRejectsRepeatedCursorWithNewPages(t *testing.T) {
requests := 0
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
server := newFakeNodeBB(t, func(w http.ResponseWriter, r *http.Request) {
requests++
w.Header().Set("Content-Type", "application/json")
pages := []map[string]any{
@@ -276,7 +277,7 @@ func TestNodeBBNamespacePaginationRejectsRepeatedCursorWithNewPages(t *testing.T
"nextCursor": "same-cursor",
},
})
}))
})
defer server.Close()
client := newNodeBBClient(server.URL, "nodebb-token")
@@ -480,7 +481,7 @@ func TestDeployWikiDryRunReadoptsMissingMappedPost(t *testing.T) {
t.Fatalf("write deploy manifest: %v", err)
}
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
server := newFakeNodeBB(t, func(w http.ResponseWriter, r *http.Request) {
switch {
case r.Method == http.MethodGet && r.URL.Path == "/api/v3/posts/42":
w.Header().Set("Content-Type", "application/json")
@@ -508,7 +509,7 @@ func TestDeployWikiDryRunReadoptsMissingMappedPost(t *testing.T) {
default:
t.Fatalf("unexpected request %s %s", r.Method, r.URL.String())
}
}))
})
defer server.Close()
result, err := DeployWikiWithOptions(&project.Project{}, DeployWikiOptions{
@@ -731,7 +732,7 @@ func TestDeployWikiCreatesNodeBBTopicAndWritesManifest(t *testing.T) {
}
createCalls := 0
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
server := newFakeNodeBB(t, func(w http.ResponseWriter, r *http.Request) {
if got := r.Header.Get("Authorization"); got != "Bearer nodebb-token" {
t.Fatalf("unexpected authorization header %q", got)
}
@@ -769,7 +770,7 @@ func TestDeployWikiCreatesNodeBBTopicAndWritesManifest(t *testing.T) {
},
},
})
}))
})
defer server.Close()
manifestPath := filepath.Join(root, "deploy-manifest.json")
@@ -848,7 +849,7 @@ func TestDeployWikiRepairsManifestedPageMissingSourceContentSync(t *testing.T) {
}
updateCalls := 0
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
server := newFakeNodeBB(t, func(w http.ResponseWriter, r *http.Request) {
switch {
case r.Method == http.MethodGet && r.URL.Path == "/api/v3/posts/42":
w.Header().Set("Content-Type", "application/json")
@@ -876,7 +877,7 @@ func TestDeployWikiRepairsManifestedPageMissingSourceContentSync(t *testing.T) {
default:
t.Fatalf("unexpected request %s %s", r.Method, r.URL.String())
}
}))
})
defer server.Close()
result, err := DeployWikiWithOptions(&project.Project{}, DeployWikiOptions{
@@ -951,7 +952,7 @@ func TestDeployWikiCreatesNodeBBTopicWithoutFallbackForDefaultThreeCharacterTitl
t.Fatalf("write source page: %v", err)
}
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
server := newFakeNodeBB(t, func(w http.ResponseWriter, r *http.Request) {
if r.Method == http.MethodGet && r.URL.Path == "/api/v3/plugins/westgate-wiki/namespace/5/pages" {
w.Header().Set("Content-Type", "application/json")
_ = json.NewEncoder(w).Encode(map[string]any{"response": map[string]any{"pages": []any{}, "hasMore": false}})
@@ -983,7 +984,7 @@ func TestDeployWikiCreatesNodeBBTopicWithoutFallbackForDefaultThreeCharacterTitl
},
},
})
}))
})
defer server.Close()
_, err := DeployWikiWithOptions(&project.Project{}, DeployWikiOptions{
@@ -1010,7 +1011,7 @@ func TestDeployWikiCreatesNodeBBTopicWithFallbackForTitleShorterThanConfiguredMi
t.Fatalf("write source page: %v", err)
}
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
server := newFakeNodeBB(t, func(w http.ResponseWriter, r *http.Request) {
if r.Method == http.MethodGet && r.URL.Path == "/api/v3/plugins/westgate-wiki/namespace/5/pages" {
w.Header().Set("Content-Type", "application/json")
_ = json.NewEncoder(w).Encode(map[string]any{"response": map[string]any{"pages": []any{}, "hasMore": false}})
@@ -1038,7 +1039,7 @@ func TestDeployWikiCreatesNodeBBTopicWithFallbackForTitleShorterThanConfiguredMi
},
},
})
}))
})
defer server.Close()
_, err := DeployWikiWithOptions(&project.Project{}, DeployWikiOptions{
@@ -1080,7 +1081,7 @@ func TestDeployWikiRenamesExistingPrefixedTopicWhenTitleIsLongEnough(t *testing.
}
renameCalls := 0
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
server := newFakeNodeBB(t, func(w http.ResponseWriter, r *http.Request) {
switch {
case r.Method == http.MethodGet && r.URL.Path == "/api/v3/topics/7":
w.Header().Set("Content-Type", "application/json")
@@ -1109,7 +1110,7 @@ func TestDeployWikiRenamesExistingPrefixedTopicWhenTitleIsLongEnough(t *testing.
default:
t.Fatalf("unexpected request %s %s", r.Method, r.URL.String())
}
}))
})
defer server.Close()
result, err := DeployWikiWithOptions(&project.Project{}, DeployWikiOptions{
@@ -1166,7 +1167,7 @@ func TestDeployWikiRenamesManagedTopicWhenGeneratedTitleChanges(t *testing.T) {
}
renameCalls := 0
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
server := newFakeNodeBB(t, func(w http.ResponseWriter, r *http.Request) {
switch {
case r.Method == http.MethodGet && r.URL.Path == "/api/v3/topics/7":
w.Header().Set("Content-Type", "application/json")
@@ -1195,7 +1196,7 @@ func TestDeployWikiRenamesManagedTopicWhenGeneratedTitleChanges(t *testing.T) {
default:
t.Fatalf("unexpected request %s %s", r.Method, r.URL.String())
}
}))
})
defer server.Close()
result, err := DeployWikiWithOptions(&project.Project{}, DeployWikiOptions{
@@ -1258,9 +1259,9 @@ func TestDeployWikiDoesNotRenameHeadinglessPageToPageIDFallback(t *testing.T) {
t.Fatalf("write deploy manifest: %v", err)
}
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
server := newFakeNodeBB(t, func(w http.ResponseWriter, r *http.Request) {
t.Fatalf("headingless unchanged page must not call NodeBB, got %s %s", r.Method, r.URL.String())
}))
})
defer server.Close()
result, err := DeployWikiWithOptions(&project.Project{}, DeployWikiOptions{
@@ -1320,7 +1321,7 @@ func TestDeployWikiRenamesBrokenHeadinglessFallbackTitleBackToPageIndexTitle(t *
}
renameCalls := 0
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
server := newFakeNodeBB(t, func(w http.ResponseWriter, r *http.Request) {
switch {
case r.Method == http.MethodGet && r.URL.Path == "/api/v3/topics/7":
w.Header().Set("Content-Type", "application/json")
@@ -1345,7 +1346,7 @@ func TestDeployWikiRenamesBrokenHeadinglessFallbackTitleBackToPageIndexTitle(t *
default:
t.Fatalf("unexpected request %s %s", r.Method, r.URL.String())
}
}))
})
defer server.Close()
result, err := DeployWikiWithOptions(&project.Project{}, DeployWikiOptions{
@@ -1396,7 +1397,7 @@ func TestDeployWikiAdoptsExistingNodeBBPageWhenManifestIsMissingWithoutCreate(t
createCalls := 0
var updated string
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
server := newFakeNodeBB(t, func(w http.ResponseWriter, r *http.Request) {
switch {
case r.Method == http.MethodGet && r.URL.Path == "/api/v3/plugins/westgate-wiki/namespace/9/pages":
w.Header().Set("Content-Type", "application/json")
@@ -1442,7 +1443,7 @@ func TestDeployWikiAdoptsExistingNodeBBPageWhenManifestIsMissingWithoutCreate(t
default:
t.Fatalf("unexpected request %s %s", r.Method, r.URL.String())
}
}))
})
defer server.Close()
manifestPath := filepath.Join(root, "deploy-manifest.json")
@@ -1509,7 +1510,7 @@ func TestDeployWikiMergesHTMLManagedAndManualRegions(t *testing.T) {
}
var updated string
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
server := newFakeNodeBB(t, func(w http.ResponseWriter, r *http.Request) {
switch {
case r.Method == http.MethodGet && r.URL.Path == "/api/v3/posts/42":
w.Header().Set("Content-Type", "application/json")
@@ -1533,7 +1534,7 @@ func TestDeployWikiMergesHTMLManagedAndManualRegions(t *testing.T) {
default:
t.Fatalf("unexpected request %s %s", r.Method, r.URL.Path)
}
}))
})
defer server.Close()
result, err := DeployWikiWithOptions(&project.Project{}, DeployWikiOptions{
@@ -1589,7 +1590,7 @@ func TestDeployWikiUpdateAcquiresWestgateWikiEditLock(t *testing.T) {
lockAcquired := false
var updateToken string
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
server := newFakeNodeBB(t, func(w http.ResponseWriter, r *http.Request) {
switch {
case r.Method == http.MethodGet && r.URL.Path == "/api/v3/posts/42":
w.Header().Set("Content-Type", "application/json")
@@ -1621,7 +1622,7 @@ func TestDeployWikiUpdateAcquiresWestgateWikiEditLock(t *testing.T) {
default:
t.Fatalf("unexpected request %s %s", r.Method, r.URL.Path)
}
}))
})
defer server.Close()
result, err := DeployWikiWithOptions(&project.Project{}, DeployWikiOptions{
@@ -1668,7 +1669,7 @@ func TestDeployWikiCreateCollisionAdoptsExistingNodeBBPage(t *testing.T) {
createCalls := 0
var updated string
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
server := newFakeNodeBB(t, func(w http.ResponseWriter, r *http.Request) {
switch {
case r.Method == http.MethodGet && r.URL.Path == "/api/v3/plugins/westgate-wiki/namespace/9/pages" && r.URL.Query().Get("q") == "":
w.Header().Set("Content-Type", "application/json")
@@ -1719,7 +1720,7 @@ func TestDeployWikiCreateCollisionAdoptsExistingNodeBBPage(t *testing.T) {
default:
t.Fatalf("unexpected request %s %s", r.Method, r.URL.String())
}
}))
})
defer server.Close()
manifestPath := filepath.Join(root, "deploy-manifest.json")
@@ -1771,7 +1772,7 @@ func TestDeployWikiCreateCollisionSearchesCanonicalTitleSegment(t *testing.T) {
createCalls := 0
var updated string
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
server := newFakeNodeBB(t, func(w http.ResponseWriter, r *http.Request) {
switch {
case r.Method == http.MethodGet && r.URL.Path == "/api/v3/plugins/westgate-wiki/namespace/58/pages" && r.URL.Query().Get("q") == "":
w.Header().Set("Content-Type", "application/json")
@@ -1828,7 +1829,7 @@ func TestDeployWikiCreateCollisionSearchesCanonicalTitleSegment(t *testing.T) {
default:
t.Fatalf("unexpected request %s %s", r.Method, r.URL.String())
}
}))
})
defer server.Close()
manifestPath := filepath.Join(root, "deploy-manifest.json")
@@ -1868,7 +1869,7 @@ func TestDeployWikiReportsAndArchivesStalePages(t *testing.T) {
}
var archived string
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
server := newFakeNodeBB(t, func(w http.ResponseWriter, r *http.Request) {
if r.Method == http.MethodPut && r.URL.Path == "/api/v3/plugins/westgate-wiki/edit-lock" {
respondWikiEditLock(t, w, r, 7, "archive-lock")
return
@@ -1889,7 +1890,7 @@ func TestDeployWikiReportsAndArchivesStalePages(t *testing.T) {
archived = req.Content
w.Header().Set("Content-Type", "application/json")
_ = json.NewEncoder(w).Encode(map[string]any{"response": map[string]any{"pid": 42, "tid": 7}})
}))
})
defer server.Close()
report, err := DeployWikiWithOptions(&project.Project{}, DeployWikiOptions{
@@ -1945,9 +1946,9 @@ func TestDeployWikiDryRunPlansTrackedStalePagePurge(t *testing.T) {
}); err != nil {
t.Fatalf("write deploy manifest: %v", err)
}
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
server := newFakeNodeBB(t, func(w http.ResponseWriter, r *http.Request) {
t.Fatalf("dry-run purge must not call NodeBB, got %s %s", r.Method, r.URL.Path)
}))
})
defer server.Close()
result, err := DeployWikiWithOptions(&project.Project{}, DeployWikiOptions{
@@ -1984,9 +1985,9 @@ func TestDeployWikiPurgeRefusesStaleManifestEntryWithoutTopicID(t *testing.T) {
}); err != nil {
t.Fatalf("write deploy manifest: %v", err)
}
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
server := newFakeNodeBB(t, func(w http.ResponseWriter, r *http.Request) {
t.Fatalf("missing-topic-id purge must not call NodeBB, got %s %s", r.Method, r.URL.Path)
}))
})
defer server.Close()
_, err := DeployWikiWithOptions(&project.Project{}, DeployWikiOptions{
@@ -2017,15 +2018,12 @@ func TestDeployWikiPurgesTrackedStaleGeneratedTopic(t *testing.T) {
}); err != nil {
t.Fatalf("write deploy manifest: %v", err)
}
purgeCalls := 0
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if r.Method != http.MethodDelete || r.URL.Path != "/api/v3/topics/7" {
var calls []string
server := newFakeNodeBB(t, func(w http.ResponseWriter, r *http.Request) {
if !answerWikiPagePurge(t, w, r, &calls) {
t.Fatalf("unexpected request %s %s", r.Method, r.URL.Path)
}
purgeCalls++
w.Header().Set("Content-Type", "application/json")
_ = json.NewEncoder(w).Encode(map[string]any{"status": map[string]any{"code": "ok"}})
}))
})
defer server.Close()
result, err := DeployWikiWithOptions(&project.Project{}, DeployWikiOptions{
@@ -2041,8 +2039,8 @@ func TestDeployWikiPurgesTrackedStaleGeneratedTopic(t *testing.T) {
if result.Stale != 1 || result.Purged != 1 {
t.Fatalf("expected one stale purge, got %#v", result)
}
if purgeCalls != 1 {
t.Fatalf("expected one NodeBB topic purge call, got %d", purgeCalls)
if strings.Join(calls, ",") != "tombstone:7,hard-purge:7" {
t.Fatalf("expected the wiki plugin page actions to purge topic 7, got %#v", calls)
}
if _, ok := loadDeployManifest(manifestPath).Pages["skills:retired"]; ok {
t.Fatalf("expected purged stale manifest entry to be removed")
@@ -2075,18 +2073,17 @@ func TestDeployWikiPurgesTrackedStaleTopicsBeforeCreatingReplacementPages(t *tes
}
var calls []string
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
server := newFakeNodeBB(t, func(w http.ResponseWriter, r *http.Request) {
if answerWikiPagePurge(t, w, r, &calls) {
return
}
switch {
case r.Method == http.MethodGet && r.URL.Path == "/api/v3/plugins/westgate-wiki/namespace/3/pages":
w.Header().Set("Content-Type", "application/json")
_ = json.NewEncoder(w).Encode(map[string]any{"response": map[string]any{"pages": []any{}, "hasMore": false}})
case r.Method == http.MethodDelete && r.URL.Path == "/api/v3/topics/7":
calls = append(calls, "purge")
w.Header().Set("Content-Type", "application/json")
_ = json.NewEncoder(w).Encode(map[string]any{"status": map[string]any{"code": "ok"}})
case r.Method == http.MethodPost && r.URL.Path == "/api/v3/topics":
calls = append(calls, "create")
if len(calls) != 2 || calls[0] != "purge" {
if strings.Join(calls, ",") != "tombstone:7,hard-purge:7,create" {
t.Fatalf("expected stale purge before create, got calls %#v", calls)
}
w.Header().Set("Content-Type", "application/json")
@@ -2102,7 +2099,7 @@ func TestDeployWikiPurgesTrackedStaleTopicsBeforeCreatingReplacementPages(t *tes
default:
t.Fatalf("unexpected request %s %s", r.Method, r.URL.String())
}
}))
})
defer server.Close()
result, err := DeployWikiWithOptions(&project.Project{}, DeployWikiOptions{
@@ -2120,7 +2117,7 @@ func TestDeployWikiPurgesTrackedStaleTopicsBeforeCreatingReplacementPages(t *tes
if result.Created != 1 || result.Purged != 1 {
t.Fatalf("expected one create and one purge, got %#v", result)
}
if strings.Join(calls, ",") != "purge,create" {
if strings.Join(calls, ",") != "tombstone:7,hard-purge:7,create" {
t.Fatalf("expected purge before create, got %#v", calls)
}
}
@@ -2150,8 +2147,12 @@ func TestDeployWikiResetManagedNamespacesPurgesRemotePagesBeforeCreatingFreshMan
t.Fatalf("write deploy manifest: %v", err)
}
const resetCalls = "tombstone:7,hard-purge:7,tombstone:8,hard-purge:8"
var calls []string
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
server := newFakeNodeBB(t, func(w http.ResponseWriter, r *http.Request) {
if answerWikiPagePurge(t, w, r, &calls) {
return
}
switch {
case r.Method == http.MethodGet && r.URL.Path == "/api/v3/plugins/westgate-wiki/namespace/3/pages":
w.Header().Set("Content-Type", "application/json")
@@ -2164,17 +2165,9 @@ func TestDeployWikiResetManagedNamespacesPurgesRemotePagesBeforeCreatingFreshMan
"hasMore": false,
},
})
case r.Method == http.MethodDelete && r.URL.Path == "/api/v3/topics/7":
calls = append(calls, "purge:7")
w.Header().Set("Content-Type", "application/json")
_ = json.NewEncoder(w).Encode(map[string]any{"status": map[string]any{"code": "ok"}})
case r.Method == http.MethodDelete && r.URL.Path == "/api/v3/topics/8":
calls = append(calls, "purge:8")
w.Header().Set("Content-Type", "application/json")
_ = json.NewEncoder(w).Encode(map[string]any{"status": map[string]any{"code": "ok"}})
case r.Method == http.MethodPost && r.URL.Path == "/api/v3/topics":
calls = append(calls, "create")
if strings.Join(calls, ",") != "purge:7,purge:8,create" {
if strings.Join(calls, ",") != resetCalls+",create" {
t.Fatalf("expected namespace reset purges before create, got %#v", calls)
}
w.Header().Set("Content-Type", "application/json")
@@ -2190,7 +2183,7 @@ func TestDeployWikiResetManagedNamespacesPurgesRemotePagesBeforeCreatingFreshMan
default:
t.Fatalf("unexpected request %s %s", r.Method, r.URL.String())
}
}))
})
defer server.Close()
result, err := DeployWikiWithOptions(&project.Project{}, DeployWikiOptions{
@@ -2205,10 +2198,21 @@ func TestDeployWikiResetManagedNamespacesPurgesRemotePagesBeforeCreatingFreshMan
if err != nil {
t.Fatalf("DeployWikiWithOptions reset-managed-namespaces failed: %v", err)
}
if result.Created != 1 || result.Purged != 2 || result.Stale != 0 {
if result.Created != 1 || result.Purged != 2 {
t.Fatalf("expected one create and two namespace purges, got %#v", result)
}
if strings.Join(calls, ",") != "purge:7,purge:8,create" {
// Callers word their destructive-policy warning around the stale count, so
// reset deletions have to show up there rather than on a separate path the
// preview never mentions.
if result.Stale != 2 || result.ResetPurged != 2 {
t.Fatalf("expected reset deletions counted as stale, got %#v", result)
}
// tid 7 is the manifest's own page; tid 8 is a topic the deployer never
// wrote, and re-seeding cannot bring it back.
if result.ResetUnrecognized != 1 {
t.Fatalf("expected one unrecognized reset deletion, got %#v", result)
}
if strings.Join(calls, ",") != resetCalls+",create" {
t.Fatalf("expected reset purges before create, got %#v", calls)
}
manifest := loadDeployManifest(manifestPath)
@@ -2236,9 +2240,9 @@ func TestDeployWikiResetManagedNamespacesRequiresCreateForLocalPages(t *testing.
t.Fatalf("write source page: %v", err)
}
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
server := newFakeNodeBB(t, func(w http.ResponseWriter, r *http.Request) {
t.Fatalf("reset without --create must fail before remote calls, got %s %s", r.Method, r.URL.String())
}))
})
defer server.Close()
_, err := DeployWikiWithOptions(&project.Project{}, DeployWikiOptions{
@@ -2270,21 +2274,21 @@ func TestDeployWikiPurgeTreatsAlreadyMissingTrackedTopicAsSuccess(t *testing.T)
t.Fatalf("write deploy manifest: %v", err)
}
seen := map[string]int{}
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if r.Method != http.MethodDelete {
server := newFakeNodeBB(t, func(w http.ResponseWriter, r *http.Request) {
call, ok := wikiPagePurgeCall(t, r)
if !ok {
t.Fatalf("unexpected request %s %s", r.Method, r.URL.Path)
}
seen[r.URL.Path]++
seen[call]++
w.Header().Set("Content-Type", "application/json")
switch r.URL.Path {
case "/api/v3/topics/7":
// Topic 7 is already gone, so the plugin cannot find the page to
// tombstone; topic 8 still exists and purges normally.
if call == "tombstone:7" {
http.Error(w, `{"status":{"message":"topic not found"}}`, http.StatusNotFound)
case "/api/v3/topics/8":
_ = json.NewEncoder(w).Encode(map[string]any{"status": map[string]any{"code": "ok"}})
default:
t.Fatalf("unexpected purge path %s", r.URL.Path)
return
}
}))
_ = json.NewEncoder(w).Encode(map[string]any{"status": map[string]any{"code": "ok"}})
})
defer server.Close()
result, err := DeployWikiWithOptions(&project.Project{}, DeployWikiOptions{
@@ -2300,8 +2304,11 @@ func TestDeployWikiPurgeTreatsAlreadyMissingTrackedTopicAsSuccess(t *testing.T)
if result.Stale != 2 || result.Purged != 2 {
t.Fatalf("expected two stale purges, got %#v", result)
}
if seen["/api/v3/topics/7"] != 1 || seen["/api/v3/topics/8"] != 1 {
t.Fatalf("expected one purge call for each tracked topic, got %#v", seen)
if seen["tombstone:7"] != 1 || seen["hard-purge:7"] != 0 {
t.Fatalf("expected the already-missing topic to stop after tombstone, got %#v", seen)
}
if seen["tombstone:8"] != 1 || seen["hard-purge:8"] != 1 {
t.Fatalf("expected the surviving topic to be tombstoned then purged, got %#v", seen)
}
manifest := loadDeployManifest(manifestPath)
if _, ok := manifest.Pages["skills:retired"]; ok {
@@ -2339,9 +2346,9 @@ func TestDeployWikiPurgeDoesNotTargetCurrentGeneratedPages(t *testing.T) {
}); err != nil {
t.Fatalf("write deploy manifest: %v", err)
}
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
server := newFakeNodeBB(t, func(w http.ResponseWriter, r *http.Request) {
t.Fatalf("current generated page must not call NodeBB during matching-hash deploy, got %s %s", r.Method, r.URL.Path)
}))
})
defer server.Close()
result, err := DeployWikiWithOptions(&project.Project{}, DeployWikiOptions{
@@ -2385,3 +2392,187 @@ func respondWikiEditLock(t *testing.T, w http.ResponseWriter, r *http.Request, e
w.Header().Set("Content-Type", "application/json")
_ = json.NewEncoder(w).Encode(map[string]any{"response": map[string]any{"status": "ok", "tid": expectedTID, "token": token}})
}
// newFakeNodeBB stands up a fake NodeBB that behaves like one running
// nodebb-plugin-westgate-wiki: native topic mutation on wiki categories is
// refused, because wiki revision history is plugin-owned. Every deploy test
// goes through here, so a deployer that reaches for the core topic API fails in
// CI the way it fails in production rather than passing against a fake that is
// more permissive than the real thing.
func newFakeNodeBB(t *testing.T, handler http.HandlerFunc) *httptest.Server {
t.Helper()
return httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if strings.HasPrefix(r.URL.Path, "/api/v3/topics/") && (r.Method == http.MethodDelete || r.Method == http.MethodPut) {
http.Error(
w,
`{"status":{"code":"bad-request","message":"Use the wiki page actions to delete, restore, or purge wiki pages."}}`,
http.StatusBadRequest,
)
return
}
handler(w, r)
}))
}
// wikiPagePurgeCall recognizes the plugin page actions a purge goes through and
// reports them as "<action>:<tid>" so tests can assert the order.
func wikiPagePurgeCall(t *testing.T, r *http.Request) (string, bool) {
t.Helper()
var action string
switch {
case r.Method == http.MethodPut && r.URL.Path == "/api/v3/plugins/westgate-wiki/page/tombstone":
action = "tombstone"
case r.Method == http.MethodDelete && r.URL.Path == "/api/v3/plugins/westgate-wiki/page/hard-purge":
action = "hard-purge"
default:
return "", false
}
var req struct {
TID int `json:"tid"`
}
if err := json.NewDecoder(r.Body).Decode(&req); err != nil {
t.Fatalf("decode wiki %s request: %v", action, err)
}
if req.TID == 0 {
t.Fatalf("expected wiki %s request to carry a topic id", action)
}
return fmt.Sprintf("%s:%d", action, req.TID), true
}
// answerWikiPagePurge records and acknowledges a plugin purge action, which is
// what most fakes want to do with one.
func answerWikiPagePurge(t *testing.T, w http.ResponseWriter, r *http.Request, calls *[]string) bool {
t.Helper()
call, ok := wikiPagePurgeCall(t, r)
if !ok {
return false
}
*calls = append(*calls, call)
w.Header().Set("Content-Type", "application/json")
_ = json.NewEncoder(w).Encode(map[string]any{"status": map[string]any{"code": "ok"}})
return true
}
// resetPurgeFixture builds a source tree, a deploy manifest, and the reset
// options the namespace-reset tests share.
func resetPurgeFixture(t *testing.T, namespaces map[string]int, tidsByCID map[int][]int) (DeployWikiOptions, func(w http.ResponseWriter, r *http.Request) bool) {
t.Helper()
root := t.TempDir()
sourceDir := filepath.Join(root, "pages")
if err := os.MkdirAll(filepath.Join(sourceDir, "skills"), 0755); err != nil {
t.Fatalf("create source dir: %v", err)
}
manifestPath := filepath.Join(root, "deploy-manifest.json")
if err := saveDeployManifest(manifestPath, wikiDeployManifest{Version: "nodebb-v1", Pages: map[string]wikiDeployManifestPage{}}); err != nil {
t.Fatalf("write deploy manifest: %v", err)
}
listPages := func(w http.ResponseWriter, r *http.Request) bool {
if r.Method != http.MethodGet {
return false
}
var cid int
if _, err := fmt.Sscanf(r.URL.Path, "/api/v3/plugins/westgate-wiki/namespace/%d/pages", &cid); err != nil {
return false
}
tids, ok := tidsByCID[cid]
if !ok {
t.Fatalf("unexpected namespace listing for category %d", cid)
}
pages := make([]map[string]any, 0, len(tids))
for _, tid := range tids {
pages = append(pages, map[string]any{"tid": tid, "title": fmt.Sprintf("Page %d", tid)})
}
w.Header().Set("Content-Type", "application/json")
_ = json.NewEncoder(w).Encode(map[string]any{"response": map[string]any{"pages": pages, "hasMore": false}})
return true
}
return DeployWikiOptions{
SourceDir: sourceDir,
ManifestPath: manifestPath,
Token: "nodebb-token",
CategoryIDs: namespaces,
AllowCreates: true,
ResetManagedNamespaces: true,
}, listPages
}
func TestDeployWikiResetSkipsTopicsNodeBBRefusesToDelete(t *testing.T) {
opts, listPages := resetPurgeFixture(t, map[string]int{"skills": 3}, map[int][]int{3: {5, 6}})
var calls []string
server := newFakeNodeBB(t, func(w http.ResponseWriter, r *http.Request) {
if listPages(w, r) {
return
}
call, ok := wikiPagePurgeCall(t, r)
if !ok {
t.Fatalf("unexpected request %s %s", r.Method, r.URL.Path)
}
calls = append(calls, call)
// Topic 5 stands in for the wiki home page, which the plugin excludes
// from tombstone, restore, and purge alike.
if call == "tombstone:5" {
http.Error(w, `{"status":{"message":"[[error:no-privileges]]"}}`, http.StatusForbidden)
return
}
w.Header().Set("Content-Type", "application/json")
_ = json.NewEncoder(w).Encode(map[string]any{"status": map[string]any{"code": "ok"}})
})
defer server.Close()
opts.Endpoint = server.URL
result, err := DeployWikiWithOptions(&project.Project{}, opts, nil)
if err != nil {
t.Fatalf("DeployWikiWithOptions reset with an undeletable topic failed: %v", err)
}
if result.ResetSkipped != 1 || result.ResetPurged != 1 {
t.Fatalf("expected one skipped and one completed reset deletion, got %#v", result)
}
if result.Purged != 1 || result.Stale != 1 {
t.Fatalf("expected the skipped topic to be dropped from the counts, got %#v", result)
}
if result.ResetUnrecognized != 1 {
t.Fatalf("expected only the purged unrecognized topic to be counted, got %#v", result)
}
if strings.Join(calls, ",") != "tombstone:5,tombstone:6,hard-purge:6" {
t.Fatalf("expected the reset to continue past the refused topic, got %#v", calls)
}
}
// NodeBB answers 403 both for the wiki home page and for a token without purge
// privileges. A category where nothing could be deleted at all is the second
// case, and reporting it as a completed reset would leave the manifest claiming
// a fresh start over pages that are all still there.
func TestDeployWikiResetFailsWhenACategoryRefusesEveryDeletion(t *testing.T) {
opts, listPages := resetPurgeFixture(t,
map[string]int{"skills": 3, "feats": 4},
map[int][]int{3: {5, 6}, 4: {7, 8}},
)
server := newFakeNodeBB(t, func(w http.ResponseWriter, r *http.Request) {
if listPages(w, r) {
return
}
call, ok := wikiPagePurgeCall(t, r)
if !ok {
t.Fatalf("unexpected request %s %s", r.Method, r.URL.Path)
}
// Category 4 holds topics 7 and 8, and the token can purge neither.
if strings.HasSuffix(call, ":7") || strings.HasSuffix(call, ":8") {
http.Error(w, `{"status":{"message":"[[error:no-privileges]]"}}`, http.StatusForbidden)
return
}
w.Header().Set("Content-Type", "application/json")
_ = json.NewEncoder(w).Encode(map[string]any{"status": map[string]any{"code": "ok"}})
})
defer server.Close()
opts.Endpoint = server.URL
_, err := DeployWikiWithOptions(&project.Project{}, opts, nil)
if err == nil || !strings.Contains(err.Error(), "wiki purge privileges") {
t.Fatalf("expected a category that deleted nothing to fail loudly, got %v", err)
}
if !strings.Contains(err.Error(), "4") {
t.Fatalf("expected the failure to name the refusing category, got %v", err)
}
}