Compare commits

..
2 Commits
Author SHA1 Message Date
archvillainetteandClaude Opus 5 ba671a76ed fix(nwsync): hash through a section reader, pin the encoder claim (#76)
ci / ci (pull_request) Successful in 3m30s
Review follow-ups. checkArtifactKey drained the *os.File to EOF, so the
file offset was left at the end; correct only because everything after it
uses ReadAt. Hash a section reader instead, so no later sequential read
can silently see nothing.

The zstd concurrency comment claimed byte-identical output but nothing
asserted it, so add the test. erf.Read now says outright that Data must
not be mutated, since payloads alias one buffer.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-31 13:08:35 +02:00
archvillainetteandClaude Opus 5 59b233e3db fix(nwsync): stream emit so peak memory tracks the largest resource (#76)
Emit read the whole artifact into memory and erf.Read then allocated a
second full copy of every payload, so a hak cost about 5x its size in
RAM. The 7 GB runner was OOM-killed on any hak over ~1.4 GB, which
blocks the NWSync backfill and every release that rebuilds a large hak.

Emit now opens the artifact, hashes it by streaming for the key check,
parses only the header and resource table via erf.ReadIndex, and reads,
hashes, compresses and stores one payload at a time. erf.Read keeps its
old shape but returns payloads as subslices instead of fresh copies,
which removes the second copy for the other callers too.

The zstd encoder pool also held one window-sized history per CPU — about
200 MB of live heap on a 24-core runner. EncodeAll is single-threaded
per call, so concurrency 1 gives byte-identical blobs for far less
memory.

Peak heap is now flat at ~22 MB for both an 8 MB and a 64 MB hak, and a
regression test asserts it does not scale with artifact size.

Closes #76

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-31 13:06:29 +02:00
29 changed files with 193 additions and 2011 deletions
+2 -2
View File
@@ -94,9 +94,9 @@ Tests must survive harmless changes to constants, defaults, wording, ordering, f
Issues live in Gitea at git.westgate.pw (`ShadowsOverWestgate/sow-tools`), managed with the `tea` CLI. Issues follow ownership — file work in the repo that owns it, not the one you happen to be standing in. See `docs/agents/issue-tracker.md`. Issues live in Gitea at git.westgate.pw (`ShadowsOverWestgate/sow-tools`), managed with the `tea` CLI. Issues follow ownership — file work in the repo that owns it, not the one you happen to be standing in. See `docs/agents/issue-tracker.md`.
### Labels ### Triage labels
Every issue and PR gets exactly one org-wide `Kind/*` label at creation (`Kind/Bug`, `Kind/Feature`, `Kind/Enhancement`, `Kind/Documentation`, `Kind/Testing`, `Kind/Security`); `Priority/*`, `Status/*`, `Reviewed/*` and `Compat/Breaking` are optional. `tea issues create -L "Kind/Bug"`. See `docs/agents/triage-labels.md`. Default label vocabulary (`needs-triage`, `needs-info`, `ready-for-agent`, `ready-for-human`, `wontfix`). See `docs/agents/triage-labels.md`.
### Domain docs ### Domain docs
+1 -1
View File
@@ -19,7 +19,7 @@ Crucible is how the artifact repos turn source into artifacts.
| `crucible-depot` | `crucible depot` | content-addressed depot blob verify/move | | `crucible-depot` | `crucible depot` | content-addressed depot blob verify/move |
| `crucible-hak` | `crucible hak` | ERF/HAK pack/unpack + hak manifests | | `crucible-hak` | `crucible hak` | ERF/HAK pack/unpack + hak manifests |
| `crucible-module` | `crucible module` | build/extract/validate/compare the `.mod` | | `crucible-module` | `crucible module` | build/extract/validate/compare the `.mod` |
| `crucible-nwsync` | `crucible nwsync` | NWSync blob emit + manifest assemble + verify | | `crucible-nwsync` | `crucible nwsync` | NWSync blob emit + manifest assemble |
| `crucible-topdata` | `crucible topdata` | compile 2da/tlk topdata + packages | | `crucible-topdata` | `crucible topdata` | compile 2da/tlk topdata + packages |
| `crucible-wiki` | `crucible wiki` | render + deploy mechanical wiki pages | | `crucible-wiki` | `crucible wiki` | render + deploy mechanical wiki pages |
+2 -26
View File
@@ -57,32 +57,8 @@ preference — move it.
so any non-interactive shell (every agent) hangs forever without so any non-interactive shell (every agent) hangs forever without
`</dev/null`. Same trap on `tea issues create --description` and `</dev/null`. Same trap on `tea issues create --description` and
`tea pr create`. `tea pr create`.
- **Apply / remove labels**: `tea issues edit <number> --add-labels "Kind/Bug"` - **Apply / remove labels**: `tea api --method PATCH` on the issue, or
(and `--remove-labels`). This handles org-level labels (`Kind/*`, `tea api repos/ShadowsOverWestgate/sow-tools/issues/<number>/labels` endpoints.
`Priority/*`, `Reviewed/*`, `Status/*`) from tea 0.15 onwards. On 0.14 it did
not: name resolution searched only this repo's own label set, so an org label
matched nothing and the command exited 0, printed the issue, and changed
nothing. Upstream fixed it in v0.15 (`modules/task/labels.go` also queries
`ListOrgLabels`). Note `tea labels` lists repo labels only and will not show
you the org set — `tea api orgs/ShadowsOverWestgate/labels` does.
- **`tea api` needs a token in the login; SSH auth is not enough.** It sends
only the login's `token:` field and does not sign requests with your SSH key,
so an SSH-key-only login gets `{"message":"token is required"}` on every call
that needs auth. Reads against these public repos still succeed anonymously,
which hides the gap until the first write. Add a token to the login in
`~/.config/tea/config.yml` (Settings > Applications; `write:issue` covers
labels, comments and dependencies) and `tea api` works for reads and writes
alike. The `tea issues` / `tea pr` subcommands authenticate either way, so
they keep working with no token at all — that asymmetry is what makes this
confusing to diagnose.
- **Verify every label change by re-reading it.** A label command exiting 0 is
not evidence it applied — that is exactly how the 0.14 silent no-op above hid
for so long, and assuming otherwise has already cost one investigation
several wrong turns. Read the resulting set back with
`tea api repos/ShadowsOverWestgate/sow-tools/issues/<number>` and check its
`labels` field, or `tea issues ls -o json`. The read-back reflects the write
immediately; if it comes back empty, the write genuinely failed. Do not
explain an empty read-back away as replication lag.
- **Close**: `tea issues close <number>` - **Close**: `tea issues close <number>`
`tea` infers the repo from the git remote when run inside the clone. `tea` infers the repo from the git remote when run inside the clone.
+11 -105
View File
@@ -1,109 +1,15 @@
# Issue and PR labels # Triage Labels
Labels are **org-wide**. They are defined once, for the whole The skills speak in terms of five canonical triage roles. This file maps those roles to the actual label strings used in this repo's issue tracker (Gitea — see `issue-tracker.md` for how to apply labels with `tea`).
`ShadowsOverWestgate` org, at
<https://git.westgate.pw/org/ShadowsOverWestgate/settings/labels>, and every
repo in the org can use them. Never create a per-repo copy of a label that
already exists at org level.
## The rule | Label in mattpocock/skills | Label in our tracker | Meaning |
| -------------------------- | -------------------- | ---------------------------------------- |
| `needs-triage` | `needs-triage` | Maintainer needs to evaluate this issue |
| `needs-info` | `needs-info` | Waiting on reporter for more information |
| `ready-for-agent` | `ready-for-agent` | Fully specified, ready for an AFK agent |
| `ready-for-human` | `ready-for-human` | Requires human implementation |
| `wontfix` | `wontfix` | Will not be actioned |
**Every issue and every pull request gets exactly one `Kind/*` label, at the When a skill mentions a role (e.g. "apply the AFK-ready triage label"), use the corresponding label string from this table.
moment it is created.** Not later, not "when someone triages it". If you open
it, you label it.
An issue with no `Kind/*` label is untriaged. That is the only meaning of Edit the right-hand column to match whatever vocabulary you actually use.
"untriaged" here — there is no `needs-triage` label.
The other groups are optional, and each one is *exclusive*: an issue can carry
at most one `Priority/*`, one `Status/*`, and one `Reviewed/*`. Gitea enforces
this.
```sh
# always from inside the owning repo's clone
tea issues create --title "..." --description "..." --labels "Kind/Bug" </dev/null
tea pr create --title "..." --description "..." --labels "Kind/Feature" </dev/null
# add a label to something that already exists (needs tea >= 0.15 for org labels)
tea issues edit <n> --add-labels "Kind/Bug,Priority/High"
# read the result back — exit 0 is not evidence the label applied
tea api "repos/ShadowsOverWestgate/<repo>/issues/<n>"
```
`tea api` sends only the login's `token:` and never your SSH key, so on an
SSH-only login every authenticated call fails with
`{"message":"token is required"}` while anonymous reads still work. Add a token
to `~/.config/tea/config.yml` if you want the raw endpoints. `tea issues edit`
needs no token.
## Kind — what this is (pick exactly one)
| Label | Use it when |
| -------------------- | --------------------------------------------------------------- |
| `Kind/Bug` | Something that used to work, or is documented to work, does not |
| `Kind/Feature` | New functionality that does not exist yet |
| `Kind/Enhancement` | Existing functionality gets better, faster, or nicer |
| `Kind/Documentation` | Docs, ADRs, runbooks, agent guides |
| `Kind/Testing` | Tests, CI checks, contract scripts |
| `Kind/Security` | Secrets, auth, permissions, hardening, a vulnerability |
Bug vs Enhancement, when it is unclear: if the current behaviour is wrong, it
is a bug. If the current behaviour is right but weak, it is an enhancement.
## Priority — how urgent (optional, at most one)
`Priority/Critical`, `Priority/High`, `Priority/Medium`, `Priority/Low`.
Leave it off if you do not know. A wrong priority is worse than none.
## Status — why it is not moving (optional, at most one)
| Label | Meaning |
| ----------------------- | -------------------------------------------- |
| `Status/Blocked` | Waiting on another issue, PR, or decision |
| `Status/Need More Info` | Waiting on the reporter or on a human answer |
| `Status/Abandoned` | Work started and stopped; nobody is on it |
## Reviewed — the verdict (optional, at most one)
`Reviewed/Confirmed`, `Reviewed/Duplicate`, `Reviewed/Invalid`,
`Reviewed/Won't Fix`. Apply one of these when closing without a fix, so the
reason survives.
## Compat
`Compat/Breaking` — add it on top of the `Kind/*` label when the change breaks
something that already works for a player, an operator, or another repo.
## Workflow labels (repo-level, not org-level)
These live in the repo, not the org, and are orthogonal to the groups above:
- `ready-for-agent` — the spec is complete; an AFK agent may pick this up.
No label means it needs a human.
- `wayfinder:map`, `wayfinder:task`, `wayfinder:research`,
`wayfinder:prototype`, `wayfinder:grilling` — set by `/wayfinder`. Leave
them alone unless you are running a wayfinder operation.
- `sow-nodebb` also has `package/*` labels naming the plugin or theme a ticket
touches.
## When a skill names a label we do not have
Skills written elsewhere (mattpocock/skills and friends) use a different
vocabulary. Translate it:
| Skill says | Do this here |
| ----------------- | ------------------------------------------------- |
| `needs-triage` | Nothing — no `Kind/*` label already means this |
| `needs-info` | `Status/Need More Info` |
| `ready-for-agent` | `ready-for-agent` |
| `ready-for-human` | Nothing — absence of `ready-for-agent` means this |
| `wontfix` | `Reviewed/Won't Fix` |
## Drift check
`ops/checks/check-labels.sh` in `sow-platform` runs nightly. It compares the
live org labels to `ops/policy/labels.yml` and lists every open issue and PR in
the org that does not have exactly one `Kind/*` label. That check is the
enforcement; this file is the rule.
+1 -62
View File
@@ -36,7 +36,6 @@ aliases.
| `depot` | `pull` | Incremental verified pull of every referenced blob. | | `depot` | `pull` | Incremental verified pull of every referenced blob. |
| `nwsync` | `emit` | Explode one artifact into NWSync blobs plus its own NSYM manifest. | | `nwsync` | `emit` | Explode one artifact into NWSync blobs plus its own NSYM manifest. |
| `nwsync` | `assemble` | Merge per-artifact NSYM manifests into one merged manifest. | | `nwsync` | `assemble` | Merge per-artifact NSYM manifests into one merged manifest. |
| `nwsync` | `verify` | Decompress and hash a published manifest's blobs through the pull zone. |
`depot status` and `depot get` pick their backend either with `--out DIR`, a `depot status` and `depot get` pick their backend either with `--out DIR`, a
depot tree on disk, or with `--target bunny|cdn`, a remote backend. The two depot tree on disk, or with `--target bunny|cdn`, a remote backend. The two
@@ -49,20 +48,10 @@ beside the artifact itself with the extension replaced, so `emit` and
`assemble` agree on where it is without being told. `assemble` agree on where it is without being told.
``` ```
nwsync emit [--as NAME] [--out DIR] [--jobs N] [--verify] <artifact-key> <file> nwsync emit [--as NAME] [--out DIR] <artifact-key> <file>
nwsync assemble --group-id N [--tlk-key KEY] [--out DIR] <artifact-key>... nwsync assemble --group-id N [--tlk-key KEY] [--out DIR] <artifact-key>...
nwsync verify [--sample N] [--base URL] [--jobs N] <manifest-sha1>
``` ```
`emit` is latency-bound, not CPU-bound: every blob costs an existence probe
plus an upload, and a measured backfill spent 26 seconds of CPU across 9.5
minutes of wall clock. `--jobs N` (default 16) sets how many resources are in
flight at once. The manifest is byte-identical at any value — the number of
workers is never observable in the output. Peak memory is `N` times the
per-resource limit of 15 MB plus its compressed copy, so raising `N` far past
the default costs real memory for little gain: the transport keeps 16 idle
connections per host, and past that a worker pays a fresh TLS handshake.
Both verbs upload by default; nothing bulky is ever written to the runner's Both verbs upload by default; nothing bulky is ever written to the runner's
disk. `--out DIR` writes a local repository tree instead, which is the disk. `--out DIR` writes a local repository tree instead, which is the
conformance path against upstream `nwn_nwsync_write`. The zone comes from conformance path against upstream `nwn_nwsync_write`. The zone comes from
@@ -75,45 +64,6 @@ resolves it. `--tlk-key` has its own slot because the TLK shadows nothing.
`--group-id` is per channel — 1 is current, 2 is testing, and 0 leaves the field `--group-id` is per channel — 1 is current, 2 is testing, and 0 leaves the field
out of the sidecar. out of the sidecar.
`nwsync verify` is the only check on a published blob upstream of a player's
client. It reads the **pull zone**, not the storage API, and needs no
credential: what matters is the bytes a client is served, edge behaviour
included. Every blob is decompressed and hashed, and the zstd frame is asserted
to declare its content size. Neither half is optional — a `Content-Length` check
passes a byte-correct-looking object whose contents are short, and a round-trip
check alone passes a frame the game client cannot decode but Go's decoder can.
Failures are reported per blob as missing, malformed framing, size mismatch or
hash mismatch, and the exit code is 1.
A full sweep of the live manifest is roughly 69,000 blobs and 15 GB, so
`--sample N` exists to make verifying routine; the default is a full sweep.
`--base URL` (or `NWSYNC_PULL_BASE`) overrides the public host.
`emit --verify` applies the same check where `emit` would otherwise skip. `emit`
normally reads a blob's presence as proof of its contents, decided by a 1-byte
range GET, so an object written truncated — or written by an emitter since found
broken — is skipped by every later run forever and no backfill repairs it. With
`--verify` the stored copy is read back, unwrapped, hashed against its own name,
and replaced when it does not match. It costs a full GET per existing blob, so
it is a repair pass, not the default.
**After a repair, `verify` is what tells you which keys to purge.** A repair is
the one thing that makes a key serve different bytes than it did before, and the
edge caches these objects for 30 days precisely because that normally cannot
happen. The two commands look at different copies on purpose: `emit --verify`
repairs the **origin**, `verify` reads the **edge**. So a `verify` run straight
after a repair is not a verdict — it is a survey, and every blob it still calls
bad is one the edge is serving stale. Purge exactly those, then re-run it; only
that second run is the verdict.
Purging the keys `verify` names beats purging the zone, because the edge only
ever cached what somebody actually fetched: the 2026-08-01 repair rewrote 2,603
blobs at the origin and left 8 stale at the edge. The purge belongs in the
repair procedure rather than in `emit`, which reports how many blobs it wrote
and never which ones — so it could not target one even with a CDN credential,
which it deliberately does not hold (#89; the procedure itself is in
sow-platform's NWSync runbook).
`emit` uploads blobs first and the index last, so the presence of an index is `emit` uploads blobs first and the index last, so the presence of an index is
the publication marker: an artifact whose emit died halfway leaves real blobs in the publication marker: an artifact whose emit died halfway leaves real blobs in
the zone and no index. Blob names are content hashes, so re-running skips the zone and no index. Blob names are content hashes, so re-running skips
@@ -149,17 +99,6 @@ saving on hak content: a 250 MB hak emitted 2296 blobs totalling 59 MB on disk
against 249 MB of resources, as recorded in the sidecar's `on_disk_bytes` and against 249 MB of resources, as recorded in the sidecar's `on_disk_bytes` and
`total_bytes`. `total_bytes`.
The zstd frame always declares its `Frame_Content_Size`. The game client sizes
its output buffer from that field and cannot decode a frame without one, but the
Go encoder omits it below 256 bytes, so `emit` re-headers those frames into the
shape reference libzstd emits: `Single_Segment_flag` set, `Window_Descriptor`
dropped, and a one-byte content size in its place. `zstd -l <frame>` must print a
decompressed size; a blank column there is the fault, and it is invisible to any
check that only decompresses, because both `zstd -dc` and Go's decoder stream
such a frame happily. This is what the sidecar's `emitter_version` counts:
version 1 omitted the field and no client could sync past such a blob, version 2
declares it. `assemble` refuses to merge indexes that disagree.
## Hidden compatibility aliases ## Hidden compatibility aliases
Existing scripts may continue using these names indefinitely. They are accepted Existing scripts may continue using these names indefinitely. They are accepted
+12 -21
View File
@@ -385,7 +385,7 @@ func refreshBuildModuleManifest(ctx context, p *project.Project, progress func(s
} }
progress("Refreshing hak list from the latest published sow-assets manifest...") progress("Refreshing hak list from the latest published sow-assets manifest...")
if err := runProjectScript(ctx, p, []string{"scripts", "fetch-upstream-manifests"}, manifestPath); err != nil { if err := runProjectScript(ctx, p, []string{"scripts", "fetch-hak-manifest"}, manifestPath); err != nil {
return "", "", err return "", "", err
} }
if _, err := pipeline.ApplyHAKManifest(p, manifestPath); err != nil { if _, err := pipeline.ApplyHAKManifest(p, manifestPath); err != nil {
@@ -927,28 +927,19 @@ func (c *topdataConsole) emitWikiBuildResult(outputDir string, pageCount int, st
fmt.Fprintf(c.stdout, "wiki status: %s\n", status) fmt.Fprintf(c.stdout, "wiki status: %s\n", status)
} }
func (c *topdataConsole) emitWikiDeployResult(result topdata.DeployResult) { func (c *topdataConsole) emitWikiDeployResult(localPages, created, updated, skipped, stale, archived, purged, drifted int, manifest string) {
spin.linebreak() spin.linebreak()
fmt.Fprintln(c.stdout, "Deploy Wiki ----------") fmt.Fprintln(c.stdout, "Deploy Wiki ----------")
fmt.Fprintf(c.stdout, "project: %s\n", c.projectName) fmt.Fprintf(c.stdout, "project: %s\n", c.projectName)
fmt.Fprintf(c.stdout, "local pages: %d\n", result.LocalPages) fmt.Fprintf(c.stdout, "local pages: %d\n", localPages)
fmt.Fprintf(c.stdout, "created: %d\n", result.Created) fmt.Fprintf(c.stdout, "created: %d\n", created)
fmt.Fprintf(c.stdout, "updated: %d\n", result.Updated) fmt.Fprintf(c.stdout, "updated: %d\n", updated)
fmt.Fprintf(c.stdout, "skipped: %d\n", result.Skipped) fmt.Fprintf(c.stdout, "skipped: %d\n", skipped)
fmt.Fprintf(c.stdout, "stale: %d\n", result.Stale) fmt.Fprintf(c.stdout, "stale: %d\n", stale)
fmt.Fprintf(c.stdout, "archived: %d\n", result.Archived) fmt.Fprintf(c.stdout, "archived: %d\n", archived)
fmt.Fprintf(c.stdout, "purged: %d\n", result.Purged) fmt.Fprintf(c.stdout, "purged: %d\n", purged)
fmt.Fprintf(c.stdout, "drifted: %d\n", result.Drifted) fmt.Fprintf(c.stdout, "drifted: %d\n", drifted)
if result.ResetPurged > 0 || result.ResetSkipped > 0 { fmt.Fprintf(c.stdout, "manifest: %s\n", c.relPath(manifest))
fmt.Fprintf(c.stdout, "namespace reset: %d (unrecognized: %d)\n", result.ResetPurged, result.ResetUnrecognized)
}
if result.ResetUnrecognized > 0 {
fmt.Fprintln(c.stdout, " unrecognized pages were not written by this deployer; recreating them is not possible")
}
if result.ResetSkipped > 0 {
fmt.Fprintf(c.stdout, " reset skipped: %d (NodeBB refused deletion, e.g. the wiki home page)\n", result.ResetSkipped)
}
fmt.Fprintf(c.stdout, "manifest: %s\n", c.relPath(result.Manifest))
} }
func (c *topdataConsole) relPath(path string) string { func (c *topdataConsole) relPath(path string) string {
@@ -1588,7 +1579,7 @@ func runDeployWiki(ctx context) error {
return err return err
} }
console.emitWikiDeployResult(result) console.emitWikiDeployResult(result.LocalPages, result.Created, result.Updated, result.Skipped, result.Stale, result.Archived, result.Purged, result.Drifted, result.Manifest)
return nil return nil
} }
+1 -58
View File
@@ -10,7 +10,6 @@ import (
"time" "time"
"git.westgate.pw/ShadowsOverWestgate/sow-tools/internal/pipeline" "git.westgate.pw/ShadowsOverWestgate/sow-tools/internal/pipeline"
"git.westgate.pw/ShadowsOverWestgate/sow-tools/internal/topdata"
) )
func TestParseBuildHAKArgsContentAddressedRoot(t *testing.T) { func TestParseBuildHAKArgsContentAddressedRoot(t *testing.T) {
@@ -219,16 +218,7 @@ func TestTopdataConsoleDebugProgressAndRelativePaths(t *testing.T) {
} }
console.progress("NodeBB wiki plan: create 1, update 2, skip 3, stale 4, archive 5, purge 6, drift 0") console.progress("NodeBB wiki plan: create 1, update 2, skip 3, stale 4, archive 5, purge 6, drift 0")
console.emitWikiDeployResult(topdata.DeployResult{ console.emitWikiDeployResult(10, 1, 2, 3, 4, 5, 6, 0, "/workspace/project/build/wiki/deploy-manifest.json")
LocalPages: 10,
Created: 1,
Updated: 2,
Skipped: 3,
Stale: 4,
Archived: 5,
Purged: 6,
Manifest: "/workspace/project/build/wiki/deploy-manifest.json",
})
output := stdout.String() output := stdout.String()
if !strings.Contains(output, "NodeBB wiki plan") { if !strings.Contains(output, "NodeBB wiki plan") {
@@ -239,53 +229,6 @@ func TestTopdataConsoleDebugProgressAndRelativePaths(t *testing.T) {
} }
} }
func TestTopdataConsoleReportsManagedNamespaceReset(t *testing.T) {
var stdout bytes.Buffer
console := &topdataConsole{
stdout: &stdout,
projectRoot: "/workspace/project",
projectName: "Test Module",
commandName: "deploy-wiki",
commandLabel: "Deploy Wiki",
}
console.emitWikiDeployResult(topdata.DeployResult{
LocalPages: 1200,
Created: 1200,
Stale: 1213,
Purged: 1213,
ResetPurged: 1213,
ResetUnrecognized: 13,
ResetSkipped: 1,
Manifest: "/workspace/project/build/wiki/deploy-manifest.json",
})
// The counts are what an operator reads to decide whether a destructive run
// is safe, so each has to reach the output on a line that names what it
// counts; the wording around them is free to change.
output := stdout.String()
var staleLine, resetLine, skippedLine string
for _, line := range strings.Split(output, "\n") {
switch {
case strings.Contains(line, "stale"):
staleLine = line
case strings.Contains(line, "skipped"):
skippedLine = line
case strings.Contains(line, "reset"):
resetLine = line
}
}
if !strings.Contains(staleLine, "1213") {
t.Fatalf("expected reset deletions counted as stale, got %q", output)
}
if !strings.Contains(resetLine, "1213") || !strings.Contains(resetLine, "13") {
t.Fatalf("expected the reset line to carry both the total and the unrecognized count, got %q", output)
}
if !strings.Contains(skippedLine, "1") {
t.Fatalf("expected refused reset deletions to be reported, got %q", output)
}
}
func TestParseDeployWikiHelpListsPurgeStalePolicy(t *testing.T) { func TestParseDeployWikiHelpListsPurgeStalePolicy(t *testing.T) {
_, err := parseDeployWikiArgs("deploy-wiki", []string{"--help"}) _, err := parseDeployWikiArgs("deploy-wiki", []string{"--help"})
if err == nil || !strings.Contains(err.Error(), "--stale-policy <report|archive|purge>") { if err == nil || !strings.Contains(err.Error(), "--stale-policy <report|archive|purge>") {
+2 -4
View File
@@ -108,11 +108,10 @@ var Registry = []Builder{
{ {
Name: "nwsync", Name: "nwsync",
Bin: "crucible-nwsync", Bin: "crucible-nwsync",
Summary: "publish NWSync blobs and manifests (emit/assemble/verify)", Summary: "publish NWSync blobs and manifests (emit/assemble)",
Commands: []Command{ Commands: []Command{
{Name: "emit", Summary: "explode one artifact into blobs plus its own NSYM manifest", Usage: "crucible nwsync emit <artifact> --out DIR"}, {Name: "emit", Summary: "explode one artifact into blobs plus its own NSYM manifest", Usage: "crucible nwsync emit <artifact> --out DIR"},
{Name: "assemble", Summary: "merge per-artifact NSYM manifests into one", Usage: "crucible nwsync assemble --order NAMES --entries DIR --out DIR [--group-id N]"}, {Name: "assemble", Summary: "merge per-artifact NSYM manifests into one", Usage: "crucible nwsync assemble --order NAMES --entries DIR --out DIR [--group-id N]"},
{Name: "verify", Summary: "read a published manifest's blobs back through the pull zone and hash them", Usage: "crucible nwsync verify <manifest-sha1> [--sample N]"},
}, },
Wired: true, Wired: true,
}, },
@@ -289,8 +288,7 @@ var Registry = []Builder{
"--dry-run report changes without writing", "--dry-run report changes without writing",
"--create allow missing pages to be created", "--create allow missing pages to be created",
"--force update unchanged pages", "--force update unchanged pages",
"--reset-managed-namespaces delete every page in the managed categories, including", "--reset-managed-namespaces reset managed namespace state",
" pages this deployer never wrote, then recreate from source",
}, },
Aliases: []CommandAlias{{Name: "deploy-wiki"}}, Aliases: []CommandAlias{{Name: "deploy-wiki"}},
}, },
+1 -1
View File
@@ -160,7 +160,7 @@ func TestCanonicalCommandSurface(t *testing.T) {
"module": {"build", "extract", "validate", "compare", "manifest"}, "module": {"build", "extract", "validate", "compare", "manifest"},
"topdata": {"validate", "build", "package", "compare", "convert"}, "topdata": {"validate", "build", "package", "compare", "convert"},
"wiki": {"build", "deploy"}, "wiki": {"build", "deploy"},
"nwsync": {"emit", "assemble", "verify"}, "nwsync": {"emit", "assemble"},
} }
for _, builder := range Registry { for _, builder := range Registry {
got := builder.subcommands() got := builder.subcommands()
+1 -100
View File
@@ -3,7 +3,6 @@ package nwsync
import ( import (
"bytes" "bytes"
"encoding/binary" "encoding/binary"
"encoding/hex"
"fmt" "fmt"
"github.com/klauspost/compress/zstd" "github.com/klauspost/compress/zstd"
@@ -32,19 +31,6 @@ var (
blobDecoder, _ = zstd.NewReader(nil, zstd.WithDecoderConcurrency(1)) blobDecoder, _ = zstd.NewReader(nil, zstd.WithDecoderConcurrency(1))
) )
// zstd frame header bits we care about. A frame starts with the magic, then a
// one-byte Frame_Header_Descriptor: bits 7-6 size the Frame_Content_Size field,
// bit 5 is Single_Segment_flag, bits 1-0 size the Dictionary_ID field.
const (
zstdFrameMagic = "\x28\xb5\x2f\xfd"
frameSingleSegment = 1 << 5
frameDictionaryMask = 0x03
// oneByteContentSizeCeiling is the size above which a Frame_Content_Size no
// longer fits in one byte. Below it the field's size flag is 0, which is
// what lets klauspost/compress leave the field out entirely.
oneByteContentSizeCeiling = 256
)
// compressBlob wraps data in NWCompressedBuffer framing. // compressBlob wraps data in NWCompressedBuffer framing.
func compressBlob(data []byte) []byte { func compressBlob(data []byte) []byte {
var out bytes.Buffer var out bytes.Buffer
@@ -52,95 +38,10 @@ func compressBlob(data []byte) []byte {
for _, field := range header { for _, field := range header {
_ = binary.Write(&out, binary.LittleEndian, field) _ = binary.Write(&out, binary.LittleEndian, field)
} }
frame := declareFrameContentSize(blobEncoder.EncodeAll(data, nil), len(data)) out.Write(blobEncoder.EncodeAll(data, nil))
// Fail closed rather than publish a blob no client can decode. An encoder
// upgrade that finds a new way to omit the field would otherwise reproduce
// #86 in silence, and a blob is skipped by every later emit once written.
if !frameDeclaresContentSize(frame) {
panic(fmt.Sprintf("nwsync: refusing to emit a %d-byte blob whose zstd frame declares no content size (descriptor %#x)",
len(data), frame[4]))
}
out.Write(frame)
return out.Bytes() return out.Bytes()
} }
// inspectBlob unwraps a stored blob the way the game client reads it, and is the
// only reader that should be trusted to judge a published blob.
//
// It asserts the frame property on top of the round trip. Go's decoder — like
// the zstd CLI — streams a frame that declares no content size, so a check that
// only decompresses and hashes is a *more* capable decoder than the client's: it
// certifies exactly the blobs the client rejects, which is how #86 reached
// production and survived an audit.
func inspectBlob(blob []byte) ([]byte, error) {
data, err := decompressBlob(blob)
if err != nil {
return nil, fmt.Errorf("malformed framing: %w", err)
}
if len(blob) > blobHeaderBytes && !frameDeclaresContentSize(blob[blobHeaderBytes:]) {
return nil, fmt.Errorf("malformed framing: the zstd frame declares no content size, which the game client cannot decode")
}
return data, nil
}
// blobMatchesName holds a stored blob to its own file name: a blob is named
// after the sha1 of its uncompressed bytes, so the name is a complete statement
// about the contents and nothing else is needed to check it.
func blobMatchesName(blob []byte, sha1Hex string) error {
data, err := inspectBlob(blob)
if err != nil {
return err
}
if got := hex.EncodeToString(sha1Sum(data)); got != sha1Hex {
return fmt.Errorf("blob %s holds the contents of %s", sha1Hex, got)
}
return nil
}
// frameDeclaresContentSize reports whether a zstd frame states how many bytes it
// decompresses to. A frame with a zero-sized Frame_Content_Size field declares
// one only when Single_Segment_flag is set; otherwise the size is unknown.
func frameDeclaresContentSize(frame []byte) bool {
if len(frame) < 5 || string(frame[:4]) != zstdFrameMagic {
return false
}
descriptor := frame[4]
return descriptor>>6 != 0 || descriptor&frameSingleSegment != 0
}
// declareFrameContentSize rewrites a frame that does not declare its
// Frame_Content_Size so that it does, and returns any other frame unchanged.
//
// klauspost/compress omits the field for inputs under 256 bytes, which the spec
// permits. Reference libzstd never does, so the NWN client — which sizes its
// output buffer from ZSTD_getFrameContentSize and has therefore never met a
// frame without one — rejects the blob outright with an empty "potential
// compression error" (#86). No encoder option changes this, so the frame is
// re-headered here.
//
// The result is the shape libzstd itself emits for the same input: setting
// Single_Segment_flag drops the Window_Descriptor byte, and the freed byte pays
// for a one-byte Frame_Content_Size. Window_Size then equals the content size,
// which is sound because the content is under 256 bytes and every match in it
// therefore falls inside that window. Same length in, same length out.
func declareFrameContentSize(frame []byte, size int) []byte {
if size <= 0 || size >= oneByteContentSizeCeiling || len(frame) < 6 || string(frame[:4]) != zstdFrameMagic {
return frame
}
descriptor := frame[4]
// Rewrite only the exact shape a small input produces: no declared size, no
// single segment, no dictionary. Anything else either declares a size
// already or is not a frame this reinterpretation is safe on.
if descriptor>>6 != 0 || descriptor&frameSingleSegment != 0 || descriptor&frameDictionaryMask != 0 {
return frame
}
reframed := make([]byte, len(frame))
copy(reframed, frame)
reframed[4] = descriptor | frameSingleSegment
reframed[5] = byte(size) // replaces Window_Descriptor
return reframed
}
// decompressBlob unwraps NWCompressedBuffer framing. It exists so a blob this // decompressBlob unwraps NWCompressedBuffer framing. It exists so a blob this
// package wrote — or one upstream wrote — can be compared by its uncompressed // package wrote — or one upstream wrote — can be compared by its uncompressed
// bytes, which is the only comparison that is meaningful across zstd // bytes, which is the only comparison that is meaningful across zstd
+14 -101
View File
@@ -12,7 +12,6 @@ import (
"sort" "sort"
"strconv" "strconv"
"strings" "strings"
"sync"
"time" "time"
"git.westgate.pw/ShadowsOverWestgate/sow-tools/internal/buildinfo" "git.westgate.pw/ShadowsOverWestgate/sow-tools/internal/buildinfo"
@@ -33,9 +32,7 @@ var skippedTypes = resTypes("nss", "ndb", "gic")
// manifest quietly disagrees with. Bump it only when emitted bytes change — it // manifest quietly disagrees with. Bump it only when emitted bytes change — it
// is deliberately not the build revision, which would invalidate every // is deliberately not the build revision, which would invalidate every
// published index on every unrelated commit. // published index on every unrelated commit.
// Version 2 declares Frame_Content_Size on every blob (#86); version 1 omitted const emitterVersion = "1"
// it below 256 bytes and no client could sync past such a blob.
const emitterVersion = "2"
// serverTypes are loaded only server-side; a manifest holding nothing else // serverTypes are loaded only server-side; a manifest holding nothing else
// has no client contents. Mirrors upstream's GlobalResTypeServerList, whose // has no client contents. Mirrors upstream's GlobalResTypeServerList, whose
@@ -65,22 +62,12 @@ type EmitResult struct {
BlobsWritten int BlobsWritten int
} }
// defaultEmitJobs is how many resources are hashed, compressed and stored at
// once. Emit is latency-bound, not CPU-bound: a blob costs a probe round-trip
// plus an upload round-trip, and a measured backfill spent 26 s of CPU across
// 9.5 minutes of wall clock. The figure matches depot's DEPOT_JOBS default and
// the transport's MaxIdleConnsPerHost, so a worker per connection needs no new
// TLS handshake.
const defaultEmitJobs = 16
// EmitOptions describes one emit run. // EmitOptions describes one emit run.
type EmitOptions struct { type EmitOptions struct {
ArtifactKey string // depot key of the artifact; the NSYM key is derived from it ArtifactKey string // depot key of the artifact; the NSYM key is derived from it
ArtifactPath string // the file on disk ArtifactPath string // the file on disk
As string // name override, for a TLK whose filename is not its published name As string // name override, for a TLK whose filename is not its published name
OutDir string // write locally instead of uploading — the conformance path OutDir string // write locally instead of uploading — the conformance path
Jobs int // resources in flight at once; 0 means defaultEmitJobs
Verify bool // hash what would be skipped instead of trusting presence
Sink sink // test seam; nil means OutDir or the zone Sink sink // test seam; nil means OutDir or the zone
} }
@@ -128,11 +115,7 @@ func Emit(options EmitOptions) (EmitResult, error) {
return EmitResult{}, err return EmitResult{}, err
} }
jobs := options.Jobs entries, blobs, onDiskBytes, err := emitResources(artifact, index, target)
if jobs < 1 {
jobs = defaultEmitJobs
}
entries, blobs, onDiskBytes, err := emitResources(artifact, index, target, jobs, options.Verify)
if err != nil { if err != nil {
return EmitResult{}, err return EmitResult{}, err
} }
@@ -193,16 +176,11 @@ func readArtifactIndex(path string, artifact io.ReaderAt, size int64, name strin
return []erf.IndexEntry{{Name: name, Type: restype, Offset: 0, Size: size}}, nil return []erf.IndexEntry{{Name: name, Type: restype, Offset: 0, Size: size}}, nil
} }
// emitResources hashes, compresses and stores resources, reading each payload // emitResources hashes, compresses and stores one resource at a time, reading
// from the artifact only when its turn comes. Peak memory tracks the resources // each payload from the artifact only when its turn comes. Peak memory
// in flight, not the archive: a 2 GB hak must emit inside a runner's few spare // therefore tracks the largest single resource, not the archive: a 2 GB hak
// GB. jobs of them are in flight at once, so the ceiling is jobs multiplied by // must emit inside a runner's few spare GB.
// fileSizeLimit and its compressed copy — bounded, and bounded by a constant func emitResources(artifact io.ReaderAt, index []erf.IndexEntry, target sink) ([]Entry, int, int64, error) {
// this package enforces itself.
//
// The returned entries are in artifact order whatever order the workers finish
// in, because a manifest's bytes are promised deterministic by emitterVersion.
func emitResources(artifact io.ReaderAt, index []erf.IndexEntry, target sink, jobs int, verify bool) ([]Entry, int, int64, error) {
// A resref appearing twice inside one artifact resolves to the last one, // A resref appearing twice inside one artifact resolves to the last one,
// the way resman lets the last container added win. // the way resman lets the last container added win.
order := make([]Identity, 0, len(index)) order := make([]Identity, 0, len(index))
@@ -230,95 +208,30 @@ func emitResources(artifact io.ReaderAt, index []erf.IndexEntry, target sink, jo
return nil, 0, 0, fmt.Errorf("resources exceed the file size limit:\n %s", strings.Join(tooBig, "\n ")) return nil, 0, 0, fmt.Errorf("resources exceed the file size limit:\n %s", strings.Join(tooBig, "\n "))
} }
// Index-addressed, never appended to: a worker owns entries[i] alone, so entries := make([]Entry, 0, len(order))
// the slice comes back in artifact order and needs no lock.
entries := make([]Entry, len(order))
var blobs int var blobs int
var onDiskBytes int64 var onDiskBytes int64
var mu sync.Mutex for _, identity := range order {
var firstErr error
// Two resrefs in one artifact can hold identical bytes, and therefore one
// blob. Serially the sink's existence check absorbed that; in parallel both
// workers would probe, both miss, and both upload. Claiming the sha1 here
// restores the dedupe and skips the probe round-trip as well.
claimed := make(map[[20]byte]bool, len(order))
failed := func() bool {
mu.Lock()
defer mu.Unlock()
return firstErr != nil
}
store := func(i int) {
identity := order[i]
payload, err := erf.ReadPayload(artifact, latest[identity]) payload, err := erf.ReadPayload(artifact, latest[identity])
if err != nil { if err != nil {
mu.Lock() return nil, 0, 0, err
if firstErr == nil {
firstErr = err
}
mu.Unlock()
return
} }
sum := sha1.Sum(payload) sum := sha1.Sum(payload)
entries[i] = Entry{ entries = append(entries, Entry{
SHA1: sum, SHA1: sum,
Size: uint32(len(payload)), Size: uint32(len(payload)),
ResRef: identity.ResRef, ResRef: identity.ResRef,
ResType: identity.ResType, ResType: identity.ResType,
} })
mu.Lock() written, err := target.putBlob(fmt.Sprintf("%x", sum), func() []byte { return compressBlob(payload) })
duplicate := claimed[sum]
claimed[sum] = true
mu.Unlock()
if duplicate {
return
}
written, err := target.putBlob(fmt.Sprintf("%x", sum), verify, func() []byte { return compressBlob(payload) })
mu.Lock()
defer mu.Unlock()
if err != nil { if err != nil {
if firstErr == nil { return nil, 0, 0, err
firstErr = err
}
return
} }
if written > 0 { if written > 0 {
blobs++ blobs++
onDiskBytes += written onDiskBytes += written
} }
} }
if jobs < 1 {
jobs = 1
}
work := make(chan int)
var wg sync.WaitGroup
for range jobs {
wg.Add(1)
go func() {
defer wg.Done()
for i := range work {
// After a failure the run is over — the caller discards
// everything and no index is written. Draining the rest of the
// channel cheaply, rather than returning, keeps the feeder from
// blocking on workers that have gone away.
if failed() {
continue
}
store(i)
}
}()
}
for i := range order {
work <- i
}
close(work)
wg.Wait()
if firstErr != nil {
return nil, 0, 0, firstErr
}
return entries, blobs, onDiskBytes, nil return entries, blobs, onDiskBytes, nil
} }
-134
View File
@@ -1,134 +0,0 @@
package nwsync
import (
"bytes"
"fmt"
"path/filepath"
"runtime/debug"
"strings"
"testing"
)
// manyResources is a hak body with enough distinct resources that a worker pool
// actually interleaves. Payloads differ so nothing is deduplicated away.
func manyResources(count int) map[string][]byte {
contents := make(map[string][]byte, count)
for i := range count {
contents[fmt.Sprintf("res%05d.tga", i)] = []byte(fmt.Sprintf("payload %d", i))
}
return contents
}
// TestEmitProducesTheSameIndexAtEveryJobCount is the contract that lets emit be
// parallel at all: emitterVersion promises a manifest's bytes are a function of
// its artifact, so the number of workers must not be observable in the output.
func TestEmitProducesTheSameIndexAtEveryJobCount(t *testing.T) {
// The sidecar stamps a wall-clock time unless this is set, which would make
// two runs differ for a reason that has nothing to do with job count.
t.Setenv("SOURCE_DATE_EPOCH", "1700000000")
dir := t.TempDir()
hak := filepath.Join(dir, "sow_test_01.hak")
writeHak(t, hak, manyResources(64))
key := artifactKey(t, hak)
emit := func(jobs int) (manifest, sidecar []byte, result EmitResult) {
out := filepath.Join(t.TempDir(), "out")
result, err := Emit(EmitOptions{
ArtifactKey: key,
ArtifactPath: hak,
OutDir: out,
Jobs: jobs,
})
if err != nil {
t.Fatalf("emit at -jobs %d: %v", jobs, err)
}
manifest, sidecar, err = dirSink{root: out}.getIndex(filepath.Base(result.ManifestPath))
if err != nil {
t.Fatalf("read index at -jobs %d: %v", jobs, err)
}
return manifest, sidecar, result
}
serialManifest, serialSidecar, serial := emit(1)
parallelManifest, parallelSidecar, parallel := emit(16)
if !bytes.Equal(serialManifest, parallelManifest) {
t.Errorf("manifest bytes differ between -jobs 1 and -jobs 16")
}
if !bytes.Equal(serialSidecar, parallelSidecar) {
t.Errorf("sidecar bytes differ between -jobs 1 and -jobs 16:\n %s\n %s", serialSidecar, parallelSidecar)
}
if serial.Entries != parallel.Entries || serial.BlobsWritten != parallel.BlobsWritten {
t.Errorf("-jobs 1 wrote %d entries/%d blobs, -jobs 16 wrote %d/%d",
serial.Entries, serial.BlobsWritten, parallel.Entries, parallel.BlobsWritten)
}
}
// TestEmitLeavesNoIndexWhenAParallelUploadFails is the fail-closed check with
// workers in flight: several uploads are in the air when the first one fails,
// and the index must still never appear. Run under -race this also covers the
// shared counters.
func TestEmitLeavesNoIndexWhenAParallelUploadFails(t *testing.T) {
fixture := newZoneFixture(t)
fixture.zone.failOn = func(key string) bool { return strings.HasPrefix(key, "data/sha1/") }
dir := t.TempDir()
hak := filepath.Join(dir, "sow_test_01.hak")
writeHak(t, hak, manyResources(64))
if _, err := Emit(EmitOptions{
ArtifactKey: artifactKey(t, hak),
ArtifactPath: hak,
Sink: fixture.sink,
Jobs: 16,
}); err == nil {
t.Fatal("emit reported success after an upload failed")
}
fixture.zone.mu.Lock()
defer fixture.zone.mu.Unlock()
for key := range fixture.zone.objects {
if strings.HasSuffix(key, ".nsym") {
t.Errorf("a half-emitted artifact published an index: %s", key)
}
}
}
// TestEmitPeakMemoryIsBoundedByJobCount pins the ceiling the parallel emit
// rests on. Peak still must not track the archive — it tracks the resources in
// flight, so a bigger hak at the same job count costs the same.
func TestEmitPeakMemoryIsBoundedByJobCount(t *testing.T) {
if testing.Short() {
t.Skip("writes a 64 MB fixture")
}
defer debug.SetGCPercent(debug.SetGCPercent(10))
measure := func(count, jobs int) uint64 {
dir := t.TempDir()
hak := filepath.Join(dir, "big.hak")
writeStreamedHak(t, hak, count)
options := EmitOptions{
ArtifactKey: artifactKey(t, hak),
ArtifactPath: hak,
As: filepath.Base(hak),
OutDir: filepath.Join(dir, "out"),
Jobs: jobs,
}
return peakHeapDuring(func() {
if _, err := Emit(options); err != nil {
t.Fatalf("emit %d resources at -jobs %d: %v", count, jobs, err)
}
})
}
const jobs = 8
small := measure(8, jobs) // 8 MB
large := measure(64, jobs) // 64 MB
// Each worker may hold one resourceSize payload plus its compressed copy,
// so the pool itself is the slack — not the archive.
const slack = 24 << 20
t.Logf("peak heap at -jobs %d: 8 MB hak %d bytes, 64 MB hak %d bytes", jobs, small, large)
if large > small+slack {
t.Fatalf("peak heap scaled with artifact size at -jobs %d: 8 MB hak peaked at %d bytes, 64 MB hak at %d", jobs, small, large)
}
}
+2 -10
View File
@@ -7,7 +7,6 @@ import (
"encoding/json" "encoding/json"
"fmt" "fmt"
"io" "io"
"path"
"path/filepath" "path/filepath"
"sort" "sort"
"strings" "strings"
@@ -195,14 +194,7 @@ func marshalSidecar(sidecar Sidecar) ([]byte, error) {
return append(body, '\r', '\n'), nil return append(body, '\r', '\n'), nil
} }
// blobKey is where a blob lives in a zone, hash tree depth 2. emit writes it, // blobPath is the data store path for a blob, hash tree depth 2.
// verify reads it and the game client requests it, so the rule lives here and
// nowhere else.
func blobKey(sha1Hex string) string {
return path.Join("data", "sha1", sha1Hex[0:2], sha1Hex[2:4], sha1Hex)
}
// blobPath is the same location inside a local repository tree.
func blobPath(root, sha1Hex string) string { func blobPath(root, sha1Hex string) string {
return filepath.Join(root, filepath.FromSlash(blobKey(sha1Hex))) return filepath.Join(root, "data", "sha1", sha1Hex[0:2], sha1Hex[2:4], sha1Hex)
} }
-55
View File
@@ -458,33 +458,6 @@ func TestAssembleFailsClosedOnMissingIndex(t *testing.T) {
} }
} }
// Callers capture a script's stdout as a value: `dir="$(pack-haks.sh)"`. A
// summary line on stdout gets glued onto that value, so both summaries belong
// on stderr.
func TestRunKeepsSummariesOffStdout(t *testing.T) {
dir := t.TempDir()
path := filepath.Join(dir, "sow_top.hak")
writeHak(t, path, map[string][]byte{"appearance.2da": []byte("2da from sow_top")})
key := artifactKey(t, path)
out := filepath.Join(dir, "out")
for _, args := range [][]string{
{"emit", "--out", out, "--as", "sow_top.hak", key, path},
{"assemble", "--out", out, key},
} {
var stdout, stderr bytes.Buffer
if code := Run(args, &stdout, &stderr); code != exitOK {
t.Fatalf("Run(%v) exit=%d: %s", args, code, stderr.String())
}
if stdout.Len() != 0 {
t.Errorf("Run(%v) wrote to stdout: %q", args, stdout.String())
}
if stderr.Len() == 0 {
t.Errorf("Run(%v) reported no summary on stderr", args)
}
}
}
func TestRunUsageErrors(t *testing.T) { func TestRunUsageErrors(t *testing.T) {
cases := [][]string{ cases := [][]string{
nil, nil,
@@ -501,31 +474,3 @@ func TestRunUsageErrors(t *testing.T) {
} }
} }
} }
// TestEveryBlobDeclaresItsFrameContentSize guards the fault that stopped every
// client sync (#86): klauspost/compress omits Frame_Content_Size for inputs
// under 256 bytes, and the game client cannot decode a frame without it. This
// asserts a frame property, not a round trip — the zstd CLI and Go's decoder
// both stream such a frame happily, so round-tripping cannot see the defect.
func TestEveryBlobDeclaresItsFrameContentSize(t *testing.T) {
// 230 and 175 are real sizes from the manifest that failed to sync; 255/256
// straddle the encoder's threshold.
for _, size := range []int{1, 32, 175, 230, 255, 256, 257, 1024, 5000} {
payload := make([]byte, size)
for i := range payload {
payload[i] = byte('a' + i%26)
}
blob := compressBlob(payload)
if !frameDeclaresContentSize(blob[blobHeaderBytes:]) {
t.Errorf("blob of %d bytes declares no frame content size (descriptor %#x)",
size, blob[blobHeaderBytes+4])
}
got, err := decompressBlob(blob)
if err != nil {
t.Fatalf("decompress %d-byte blob: %v", size, err)
}
if !bytes.Equal(got, payload) {
t.Errorf("%d-byte blob did not round trip", size)
}
}
}
+7 -69
View File
@@ -12,17 +12,12 @@ import (
"flag" "flag"
"fmt" "fmt"
"io" "io"
"os"
) )
const ( const (
exitOK = 0 exitOK = 0
exitUsage = 64 exitUsage = 64
exitInternal = 70 exitInternal = 70
// exitDrift says the command worked and the zone is wrong, which is a
// different thing for CI to act on than the command failing. It matches
// depot's code for the same meaning.
exitDrift = 1
) )
// Run executes an nwsync subcommand. args[0] is the subcommand (emit|assemble); // Run executes an nwsync subcommand. args[0] is the subcommand (emit|assemble);
@@ -34,11 +29,9 @@ func Run(args []string, stdout, stderr io.Writer) int {
} }
switch args[0] { switch args[0] {
case "emit": case "emit":
return runEmit(args[1:], stderr) return runEmit(args[1:], stdout, stderr)
case "assemble": case "assemble":
return runAssemble(args[1:], stderr) return runAssemble(args[1:], stdout, stderr)
case "verify":
return runVerify(args[1:], stdout, stderr, os.Getenv)
case "-h", "--help", "help": case "-h", "--help", "help":
printRunUsage(stdout) printRunUsage(stdout)
return exitOK return exitOK
@@ -51,30 +44,17 @@ func Run(args []string, stdout, stderr io.Writer) int {
func printRunUsage(w io.Writer) { func printRunUsage(w io.Writer) {
fmt.Fprint(w, `usage: fmt.Fprint(w, `usage:
nwsync emit [--as NAME] [--out DIR] [--verify] <artifact-key> <file> nwsync emit [--as NAME] [--out DIR] <artifact-key> <file>
nwsync assemble --group-id N [--tlk-key KEY] [--out DIR] <artifact-key>... nwsync assemble --group-id N [--tlk-key KEY] [--out DIR] <artifact-key>...
nwsync verify [--sample N] [--base URL] <manifest-sha1>
emit explodes one .hak/.erf or one loose file (the TLK) into NWSync blobs plus emit explodes one .hak/.erf or one loose file (the TLK) into NWSync blobs plus
a NSYM index covering only that artifact, and uploads both. assemble merges a NSYM index covering only that artifact, and uploads both. assemble merges
those indexes into one manifest, reading no bulk data. Artifact keys are depot those indexes into one manifest, reading no bulk data. Artifact keys are depot
keys; an index lives beside its artifact, with the extension replaced. keys; an index lives beside its artifact, with the extension replaced.
verify reads a published manifest and its blobs back through the public pull
zone, with no credential, and decompresses and hashes every one. It is the only
check on a published blob upstream of a player's client.
--verify makes emit hash what it would otherwise skip. emit normally treats a
blob's presence as proof of its contents, so without this an object written
truncated, or written by an emitter since found broken, is skipped forever.
--verify repairs the storage zone, while verify reads the edge in front of it.
So a verify run right after a repair is a survey, not a verdict: it names the
keys the edge still serves stale. Purge those, then run it again.
--out DIR writes to a local repository tree instead of uploading, which is the --out DIR writes to a local repository tree instead of uploading, which is the
conformance path against upstream nwn_nwsync_write. Without it, the zone comes conformance path against upstream nwn_nwsync_write. Without it, the zone comes
from NWSYNC_STORAGE_ZONE, NWSYNC_STORAGE_PASSWORD and BUNNY_STORAGE_HOST. from NWSYNC_STORAGE_ZONE, NWSYNC_STORAGE_PASSWORD and BUNNY_STORAGE_HOST.
verify needs none of those; its base comes from --base or NWSYNC_PULL_BASE.
`) `)
} }
@@ -96,13 +76,11 @@ func parseArgs(fs *flag.FlagSet, args []string) ([]string, error) {
} }
} }
func runEmit(args []string, stderr io.Writer) int { func runEmit(args []string, stdout, stderr io.Writer) int {
fs := flag.NewFlagSet("emit", flag.ContinueOnError) fs := flag.NewFlagSet("emit", flag.ContinueOnError)
fs.SetOutput(stderr) fs.SetOutput(stderr)
as := fs.String("as", "", "published name of the artifact, when it differs from the key") as := fs.String("as", "", "published name of the artifact, when it differs from the key")
out := fs.String("out", "", "write to a local repository tree instead of uploading") out := fs.String("out", "", "write to a local repository tree instead of uploading")
jobs := fs.Int("jobs", defaultEmitJobs, "resources to hash, compress and store at once")
verify := fs.Bool("verify", false, "read back and hash blobs that already exist instead of trusting their presence")
positional, err := parseArgs(fs, args) positional, err := parseArgs(fs, args)
if err != nil { if err != nil {
return exitUsage return exitUsage
@@ -111,63 +89,23 @@ func runEmit(args []string, stderr io.Writer) int {
fmt.Fprintf(stderr, "nwsync emit: <artifact-key> and <file> are both required\n") fmt.Fprintf(stderr, "nwsync emit: <artifact-key> and <file> are both required\n")
return exitUsage return exitUsage
} }
if *jobs < 1 {
fmt.Fprintf(stderr, "nwsync emit: -jobs must be at least 1, got %d\n", *jobs)
return exitUsage
}
result, err := Emit(EmitOptions{ result, err := Emit(EmitOptions{
ArtifactKey: positional[0], ArtifactKey: positional[0],
ArtifactPath: positional[1], ArtifactPath: positional[1],
As: *as, As: *as,
OutDir: *out, OutDir: *out,
Jobs: *jobs,
Verify: *verify,
}) })
if err != nil { if err != nil {
fmt.Fprintf(stderr, "nwsync emit: %v\n", err) fmt.Fprintf(stderr, "nwsync emit: %v\n", err)
return exitInternal return exitInternal
} }
fmt.Fprintf(stderr, "emitted %s: %d resources, %d new blobs, index %s\n", fmt.Fprintf(stdout, "emitted %s: %d resources, %d new blobs, index %s\n",
result.Name, result.Entries, result.BlobsWritten, result.ManifestPath) result.Name, result.Entries, result.BlobsWritten, result.ManifestPath)
return exitOK return exitOK
} }
func runVerify(args []string, stdout, stderr io.Writer, getenv func(string) string) int { func runAssemble(args []string, stdout, stderr io.Writer) int {
fs := flag.NewFlagSet("verify", flag.ContinueOnError)
fs.SetOutput(stderr)
base := fs.String("base", getenv("NWSYNC_PULL_BASE"), "pull zone base URL to read through")
sample := fs.Int("sample", 0, "check this many random blobs instead of all of them")
jobs := fs.Int("jobs", defaultEmitJobs, "blobs to fetch and hash at once")
positional, err := parseArgs(fs, args)
if err != nil {
return exitUsage
}
if len(positional) != 1 {
fmt.Fprintf(stderr, "nwsync verify: exactly one <manifest-sha1> is required\n")
return exitUsage
}
result, err := Verify(VerifyOptions{
ManifestSHA1: positional[0],
Base: *base,
Sample: *sample,
Jobs: *jobs,
Log: stderr,
})
if err != nil {
fmt.Fprintf(stderr, "nwsync verify: %v\n", err)
return exitInternal
}
fmt.Fprintf(stdout, "verified %d of %d blobs behind %d resources: %d failures, %d bytes checked\n",
result.Checked, result.Blobs, result.Entries, result.Failures, result.Bytes)
if result.Failures > 0 {
return exitDrift
}
return exitOK
}
func runAssemble(args []string, stderr io.Writer) int {
fs := flag.NewFlagSet("assemble", flag.ContinueOnError) fs := flag.NewFlagSet("assemble", flag.ContinueOnError)
fs.SetOutput(stderr) fs.SetOutput(stderr)
tlkKey := fs.String("tlk-key", "", "depot key of the TLK, which shadows nothing and merges last") tlkKey := fs.String("tlk-key", "", "depot key of the TLK, which shadows nothing and merges last")
@@ -196,7 +134,7 @@ func runAssemble(args []string, stderr io.Writer) int {
fmt.Fprintf(stderr, "nwsync assemble: %v\n", err) fmt.Fprintf(stderr, "nwsync assemble: %v\n", err)
return exitInternal return exitInternal
} }
fmt.Fprintf(stderr, "assembled manifest %s: %d resources, %s\n", fmt.Fprintf(stdout, "assembled manifest %s: %d resources, %s\n",
result.SHA1, result.Entries, result.ManifestPath) result.SHA1, result.Entries, result.ManifestPath)
return exitOK return exitOK
} }
+10 -39
View File
@@ -20,17 +20,11 @@ import (
// upstream's output and ours can be diffed on a developer machine. // upstream's output and ours can be diffed on a developer machine.
type sink interface { type sink interface {
// putBlob stores one NWCompressedBuffer blob under its sha1 name and // putBlob stores one NWCompressedBuffer blob under its sha1 name and
// returns the bytes stored, or 0 if a good copy was already there. Blob // returns the bytes stored, or 0 if the blob was already there. Blob names
// names are content hashes, so an existing name is normally taken as // are content hashes, so an existing name is existing content — which is
// existing content — which is why body is a thunk: compression is the // why body is a thunk: compression is the expensive part of emit and a
// expensive part of emit and a blob that is already stored must not pay // blob that is already stored must not pay for it.
// for it. putBlob(sha1Hex string, body func() []byte) (int64, error)
//
// verify stops trusting presence: the stored copy is read back, unwrapped
// and hashed, and replaced when it is not what its name claims. Without it
// an object written truncated, or written by an emitter since found broken,
// is skipped by every later emit forever and no backfill can repair it.
putBlob(sha1Hex string, verify bool, body func() []byte) (int64, error)
// putIndex stores a NSYM manifest and its sidecar under key, which is // putIndex stores a NSYM manifest and its sidecar under key, which is
// either an artifact-derived object key or a local path. // either an artifact-derived object key or a local path.
putIndex(key string, manifest, sidecar []byte) error putIndex(key string, manifest, sidecar []byte) error
@@ -43,15 +37,9 @@ type sink interface {
// dirSink writes a local NWSync repository tree. // dirSink writes a local NWSync repository tree.
type dirSink struct{ root string } type dirSink struct{ root string }
func (s dirSink) putBlob(sha1Hex string, verify bool, body func() []byte) (int64, error) { func (s dirSink) putBlob(sha1Hex string, body func() []byte) (int64, error) {
blob := blobPath(s.root, sha1Hex) blob := blobPath(s.root, sha1Hex)
if !verify { if _, err := os.Stat(blob); err == nil {
// Stat, not read: the common path must not pay to open every blob that
// is already there.
if _, err := os.Stat(blob); err == nil {
return 0, nil
}
} else if stored, err := os.ReadFile(blob); err == nil && blobMatchesName(stored, sha1Hex) == nil {
return 0, nil return 0, nil
} }
if err := os.MkdirAll(filepath.Dir(blob), 0o755); err != nil { if err := os.MkdirAll(filepath.Dir(blob), 0o755); err != nil {
@@ -103,8 +91,8 @@ type zoneSink struct {
zone string zone string
} }
func (s zoneSink) putBlob(sha1Hex string, verify bool, body func() []byte) (int64, error) { func (s zoneSink) putBlob(sha1Hex string, body func() []byte) (int64, error) {
key := blobKey(sha1Hex) key := path.Join("data", "sha1", sha1Hex[0:2], sha1Hex[2:4], sha1Hex)
// A throttled probe must never be read as "missing, re-upload" or as // A throttled probe must never be read as "missing, re-upload" or as
// "present, skip", so only a confirmed Present skips the upload. // "present, skip", so only a confirmed Present skips the upload.
state, _, err := s.store.ProbeKey(s.ctx, key) state, _, err := s.store.ProbeKey(s.ctx, key)
@@ -112,24 +100,7 @@ func (s zoneSink) putBlob(sha1Hex string, verify bool, body func() []byte) (int6
return 0, fmt.Errorf("probe blob %s: %w", sha1Hex, err) return 0, fmt.Errorf("probe blob %s: %w", sha1Hex, err)
} }
if state == depot.Present { if state == depot.Present {
if !verify { return 0, nil
return 0, nil
}
// The probe only proved the object exists. Read it back and hold it to
// its own name.
//
// This reads the storage API rather than the pull zone: emit holds the
// write credential, and a repair decision has to be made against the
// copy it is about to overwrite, not against an edge cache of it. A
// read that fails outright is a fault, not a verdict — treating it as
// "bad, re-upload" would turn a throttled zone into a full backfill.
stored, err := s.store.GetKey(s.ctx, key)
if err != nil {
return 0, fmt.Errorf("read back blob %s: %w", sha1Hex, err)
}
if blobMatchesName(stored, sha1Hex) == nil {
return 0, nil
}
} }
data := body() data := body()
if err := s.put(key, data); err != nil { if err := s.put(key, data); err != nil {
-233
View File
@@ -1,233 +0,0 @@
package nwsync
import (
"crypto/sha1"
"encoding/hex"
"errors"
"fmt"
"io"
"math/rand/v2"
"net/http"
"path"
"sort"
"strconv"
"sync"
"time"
"git.westgate.pw/ShadowsOverWestgate/sow-tools/internal/erf"
)
// defaultPullBase is the public NWSync host, which is a Bunny pull zone fronting
// the storage zone. Verify reads through it rather than through the storage API
// on purpose: what matters is the bytes a client is served, edge behaviour
// included, not what the origin believes it holds.
const defaultPullBase = "https://nwsync.westgate.pw"
// errBlobMissing marks an object the zone does not serve at all, as distinct
// from one it serves badly.
var errBlobMissing = errors.New("missing")
// blobSource reads one object out of the zone by key. Verify never writes and
// never authenticates, so this is deliberately narrower than sink.
type blobSource interface {
get(key string) ([]byte, error)
describe(key string) string
}
// pullZone reads the zone over plain HTTP, with no credential.
type pullZone struct {
base string
client *http.Client
}
func newPullZone(base string) blobSource {
if base == "" {
base = defaultPullBase
}
return pullZone{
base: base,
// A full sweep is tens of thousands of small requests, so connections
// have to be reused; the default transport does that already.
client: &http.Client{Timeout: 60 * time.Second},
}
}
func (z pullZone) describe(key string) string { return z.base + "/" + key }
func (z pullZone) get(key string) ([]byte, error) {
resp, err := z.client.Get(z.describe(key))
if err != nil {
return nil, err
}
defer resp.Body.Close()
if resp.StatusCode == http.StatusNotFound || resp.StatusCode == http.StatusGone {
_, _ = io.Copy(io.Discard, resp.Body)
return nil, errBlobMissing
}
if resp.StatusCode < 200 || resp.StatusCode >= 300 {
_, _ = io.Copy(io.Discard, resp.Body)
return nil, fmt.Errorf("unexpected status %d", resp.StatusCode)
}
return io.ReadAll(resp.Body)
}
// VerifyOptions describes one verify run.
type VerifyOptions struct {
ManifestSHA1 string // the merged manifest to verify
Base string // pull zone base URL; empty means defaultPullBase
Sample int // check this many random blobs; 0 means all of them
Jobs int // blobs in flight at once; 0 means defaultEmitJobs
Source blobSource // test seam; nil means the pull zone at Base
Log io.Writer // per-blob failures land here; nil discards them
}
// VerifyResult reports what one verify run found.
type VerifyResult struct {
Entries int // resources the manifest names
Blobs int // distinct blobs behind those resources
Checked int // blobs actually fetched
Failures int // blobs that failed a check
Bytes int64 // uncompressed bytes verified
}
// Verify reads a published manifest and its blobs the way a client reads them,
// and reports every blob that is not what the manifest says it is.
//
// Presence is not correctness. emit skips an object that already exists on the
// strength of a 1-byte range GET, so a truncated or wrongly framed object is
// skipped by every later emit forever and the backfill cannot repair it. This is
// the only thing upstream of a player's client that can tell that has happened.
//
// Every blob is decompressed and hashed. A Content-Length check would pass the
// exact failure mode being hunted — a byte-correct-looking object whose contents
// are wrong — and a round-trip check alone would pass a frame that omits its
// content size, because Go's decoder is more capable than the client's (#86).
func Verify(options VerifyOptions) (VerifyResult, error) {
// The argument is interpolated straight into a URL path, so it is checked
// rather than trusted: exactly 20 bytes of hex, nothing else.
if sum, err := hex.DecodeString(options.ManifestSHA1); err != nil || len(sum) != sha1.Size {
return VerifyResult{}, fmt.Errorf("%q is not a manifest sha1", options.ManifestSHA1)
}
source := options.Source
if source == nil {
source = newPullZone(options.Base)
}
log := options.Log
if log == nil {
log = io.Discard
}
manifestKey := path.Join("manifests", options.ManifestSHA1)
data, err := source.get(manifestKey)
if err != nil {
return VerifyResult{}, fmt.Errorf("%s: %w", source.describe(manifestKey), err)
}
// A manifest is named after its own sha1, so this catches the zone serving
// a different manifest — or a truncated one — before any blob is fetched.
if got := hex.EncodeToString(sha1Sum(data)); got != options.ManifestSHA1 {
return VerifyResult{}, fmt.Errorf("%s hashes to %s, not the manifest asked for",
source.describe(manifestKey), got)
}
entries, err := readManifest(data)
if err != nil {
return VerifyResult{}, fmt.Errorf("%s: %w", source.describe(manifestKey), err)
}
// A manifest names one blob many times over: mappings share a sha1, and so
// do resrefs with identical contents. Fetch each blob once.
blobs := make([]Entry, 0, len(entries))
seen := make(map[[20]byte]bool, len(entries))
for _, entry := range entries {
if seen[entry.SHA1] {
continue
}
seen[entry.SHA1] = true
blobs = append(blobs, entry)
}
result := VerifyResult{Entries: len(entries), Blobs: len(blobs)}
checking := blobs
if options.Sample > 0 && options.Sample < len(blobs) {
// A full sweep of the live manifest is ~69,000 objects and ~15 GB, so
// sampling is what makes verifying a routine act rather than an event.
picks := rand.Perm(len(blobs))[:options.Sample]
checking = make([]Entry, 0, options.Sample)
for _, i := range picks {
checking = append(checking, blobs[i])
}
}
result.Checked = len(checking)
jobs := options.Jobs
if jobs < 1 {
jobs = defaultEmitJobs
}
var (
mu sync.Mutex
failures []string
)
work := make(chan Entry)
var wg sync.WaitGroup
for range jobs {
wg.Go(func() {
for entry := range work {
fault := checkEntry(source, entry)
mu.Lock()
if fault != "" {
failures = append(failures, fault)
} else {
result.Bytes += int64(entry.Size)
}
mu.Unlock()
}
})
}
for _, entry := range checking {
work <- entry
}
close(work)
wg.Wait()
// Workers finish in any order; a report an operator can diff must not.
sort.Strings(failures)
for _, fault := range failures {
fmt.Fprintln(log, fault)
}
result.Failures = len(failures)
return result, nil
}
// checkEntry fetches one blob and returns a one-line fault, or "" if it is
// exactly what the manifest entry says it is.
func checkEntry(source blobSource, entry Entry) string {
// Name the resource, not just the hash: an operator has to find the thing
// in a hak, and a bare sha1 says nothing about where to look.
extension, ok := erf.ExtensionForResourceType(entry.ResType)
if !ok {
extension = strconv.Itoa(int(entry.ResType))
}
where := fmt.Sprintf("%s (%s.%s)", entry.sha1Hex(), entry.ResRef, extension)
blob, err := source.get(blobKey(entry.sha1Hex()))
if err != nil {
if errors.Is(err, errBlobMissing) {
return where + ": missing"
}
return where + ": unreadable: " + err.Error()
}
data, err := inspectBlob(blob)
if err != nil {
return where + ": " + err.Error()
}
if uint32(len(data)) != entry.Size {
return fmt.Sprintf("%s: size mismatch: %d bytes, manifest says %d", where, len(data), entry.Size)
}
if sha1.Sum(data) != entry.SHA1 {
return fmt.Sprintf("%s: hash mismatch: contents hash to %x", where, sha1.Sum(data))
}
return ""
}
func sha1Sum(data []byte) []byte {
sum := sha1.Sum(data)
return sum[:]
}
-285
View File
@@ -1,285 +0,0 @@
package nwsync
import (
"bytes"
"crypto/sha1"
"encoding/hex"
"os"
"path/filepath"
"strings"
"testing"
"github.com/klauspost/compress/zstd"
)
// verifyFixture emits two haks and a TLK into a fake zone, assembles them, and
// hands back a verifier reading that zone the way a client would.
type verifyFixture struct {
*zoneSinkFixture
manifestSHA1 string
}
func newVerifyFixture(t *testing.T) *verifyFixture {
t.Helper()
zone := newZoneFixture(t)
dir := t.TempDir()
hak := filepath.Join(dir, "sow_test_01.hak")
// A payload under 256 bytes is the one the frame-header check exists for.
writeHak(t, hak, map[string][]byte{
"bloodstain1.tga": []byte("small"),
"appearance.2da": bytes.Repeat([]byte("2DA V2.0\n"), 200),
})
tlk := filepath.Join(dir, "sow_tlk.tlk")
if err := os.WriteFile(tlk, []byte("TLK V3.0 payload"), 0o644); err != nil {
t.Fatal(err)
}
zone.emit(t, hak)
if _, err := Emit(EmitOptions{
ArtifactKey: artifactKey(t, tlk), ArtifactPath: tlk, As: "sow_tlk.tlk", Sink: zone.sink,
}); err != nil {
t.Fatalf("emit tlk: %v", err)
}
assembled, err := Assemble(AssembleOptions{
ArtifactKeys: []string{artifactKey(t, hak)}, TLKKey: artifactKey(t, tlk), Sink: zone.sink,
})
if err != nil {
t.Fatalf("assemble: %v", err)
}
return &verifyFixture{zoneSinkFixture: zone, manifestSHA1: assembled.SHA1}
}
func (f *verifyFixture) verify(t *testing.T, sample int) (VerifyResult, string, error) {
t.Helper()
var log bytes.Buffer
result, err := Verify(VerifyOptions{
ManifestSHA1: f.manifestSHA1,
Sample: sample,
Source: f.zone.pullZone(),
Log: &log,
})
return result, log.String(), err
}
// keyOf is where a resource's blob lives, addressed by the sha1 of its
// uncompressed bytes — the same path the client requests.
func keyOf(body []byte) string {
sum := sha1.Sum(body)
return blobKey(hex.EncodeToString(sum[:]))
}
func TestVerifyPassesACleanZone(t *testing.T) {
fixture := newVerifyFixture(t)
result, log, err := fixture.verify(t, 0)
if err != nil {
t.Fatalf("verify: %v", err)
}
if result.Failures != 0 {
t.Errorf("verify reported %d failures on a clean zone: %s", result.Failures, log)
}
// A default run is a full sweep, so it must reach every blob the manifest
// names — not some of them.
if result.Checked != result.Blobs || result.Blobs == 0 {
t.Errorf("checked %d of %d blobs; a full sweep must check all of them", result.Checked, result.Blobs)
}
}
func TestVerifyReportsAMissingBlob(t *testing.T) {
fixture := newVerifyFixture(t)
key := keyOf([]byte("small"))
fixture.zone.mu.Lock()
delete(fixture.zone.objects, key)
fixture.zone.mu.Unlock()
result, log, err := fixture.verify(t, 0)
if err != nil {
t.Fatalf("verify: %v", err)
}
if result.Failures != 1 {
t.Fatalf("reported %d failures, want 1: %s", result.Failures, log)
}
if !strings.Contains(log, "missing") {
t.Errorf("a deleted blob was not reported as missing: %s", log)
}
}
func TestVerifyReportsATruncatedBlob(t *testing.T) {
fixture := newVerifyFixture(t)
key := keyOf([]byte("small"))
fixture.zone.mu.Lock()
fixture.zone.objects[key] = fixture.zone.objects[key][:blobHeaderBytes+4]
fixture.zone.mu.Unlock()
result, log, err := fixture.verify(t, 0)
if err != nil {
t.Fatalf("verify: %v", err)
}
if result.Failures != 1 {
t.Fatalf("reported %d failures, want 1: %s", result.Failures, log)
}
if !strings.Contains(log, "framing") {
t.Errorf("a truncated blob was not reported as malformed framing: %s", log)
}
}
// TestVerifyRejectsABlobWithNoDeclaredFrameContentSize is the check that #86
// slipped past: the blob decompresses to exactly the right bytes, so a verifier
// that only round-trips certifies it, yet the client cannot decode it.
func TestVerifyRejectsABlobWithNoDeclaredFrameContentSize(t *testing.T) {
fixture := newVerifyFixture(t)
body := []byte("small")
key := keyOf(body)
fixture.zone.mu.Lock()
good := fixture.zone.objects[key]
encoder, err := zstd.NewWriter(nil, zstd.WithEncoderConcurrency(1))
if err != nil {
t.Fatal(err)
}
bad := append(append([]byte{}, good[:blobHeaderBytes]...), encoder.EncodeAll(body, nil)...)
fixture.zone.objects[key] = bad
fixture.zone.mu.Unlock()
if frameDeclaresContentSize(bad[blobHeaderBytes:]) {
t.Fatal("the fixture blob declares a content size; it cannot exercise the check")
}
if got, err := decompressBlob(bad); err != nil || !bytes.Equal(got, body) {
t.Fatalf("the fixture blob must round trip, or it proves nothing: %v", err)
}
result, log, err := fixture.verify(t, 0)
if err != nil {
t.Fatalf("verify: %v", err)
}
if result.Failures != 1 {
t.Fatalf("reported %d failures, want 1: %s", result.Failures, log)
}
if !strings.Contains(log, "content size") {
t.Errorf("undeclared frame content size was not the reported reason: %s", log)
}
}
func TestVerifyReportsWrongContents(t *testing.T) {
fixture := newVerifyFixture(t)
key := keyOf([]byte("small"))
fixture.zone.mu.Lock()
// Valid framing, valid zstd, wrong bytes: only decompressing and hashing
// can see this, which is why Content-Length is not enough.
fixture.zone.objects[key] = compressBlob([]byte("wrong"))
fixture.zone.mu.Unlock()
result, log, err := fixture.verify(t, 0)
if err != nil {
t.Fatalf("verify: %v", err)
}
if result.Failures != 1 {
t.Fatalf("reported %d failures, want 1: %s", result.Failures, log)
}
if !strings.Contains(log, "hash mismatch") {
t.Errorf("wrong contents were not reported as a hash mismatch: %s", log)
}
}
func TestVerifyReportsAShortBlob(t *testing.T) {
fixture := newVerifyFixture(t)
key := keyOf([]byte("small"))
fixture.zone.mu.Lock()
// Well-formed all the way down and simply too short — the shape a killed
// upload leaves behind, and the one a Content-Length check would pass.
fixture.zone.objects[key] = compressBlob([]byte("sma"))
fixture.zone.mu.Unlock()
result, log, err := fixture.verify(t, 0)
if err != nil {
t.Fatalf("verify: %v", err)
}
if result.Failures != 1 {
t.Fatalf("reported %d failures, want 1: %s", result.Failures, log)
}
if !strings.Contains(log, "size mismatch") {
t.Errorf("a short blob was not reported as a size mismatch: %s", log)
}
}
func TestVerifyFailsWhenTheManifestIsNotTheOneAsked(t *testing.T) {
fixture := newVerifyFixture(t)
fixture.zone.mu.Lock()
fixture.zone.objects["manifests/"+fixture.manifestSHA1] = []byte("NSYM garbage")
fixture.zone.mu.Unlock()
if _, _, err := fixture.verify(t, 0); err == nil {
t.Fatal("verify accepted a manifest that is not the one requested")
}
}
func TestVerifySampleChecksFewerBlobs(t *testing.T) {
fixture := newVerifyFixture(t)
result, log, err := fixture.verify(t, 1)
if err != nil {
t.Fatalf("verify: %v", err)
}
if result.Checked != 1 {
t.Errorf("--sample 1 checked %d blobs, want 1: %s", result.Checked, log)
}
if result.Blobs <= result.Checked {
t.Errorf("sampling %d of %d blobs is not a sample", result.Checked, result.Blobs)
}
}
// TestEmitVerifyReplacesABlobThatIsNotItsName covers the reason #86 could not be
// fixed by the encoder alone: emit skips whatever is already present, so every
// blob published by the broken encoder stays broken until emit stops trusting
// presence.
func TestEmitVerifyReplacesABlobThatIsNotItsName(t *testing.T) {
fixture := newZoneFixture(t)
dir := t.TempDir()
hak := filepath.Join(dir, "sow_test_01.hak")
body := []byte("blood")
writeHak(t, hak, map[string][]byte{"bloodstain1.tga": body})
emit := func(verify bool) EmitResult {
t.Helper()
result, err := Emit(EmitOptions{
ArtifactKey: artifactKey(t, hak),
ArtifactPath: hak,
Sink: fixture.sink,
Verify: verify,
})
if err != nil {
t.Fatalf("emit (verify=%v): %v", verify, err)
}
return result
}
emit(false)
key := keyOf(body)
encoder, err := zstd.NewWriter(nil, zstd.WithEncoderConcurrency(1))
if err != nil {
t.Fatal(err)
}
fixture.zone.mu.Lock()
good := fixture.zone.objects[key]
fixture.zone.objects[key] = append(append([]byte{}, good[:blobHeaderBytes]...), encoder.EncodeAll(body, nil)...)
fixture.zone.mu.Unlock()
if plain := emit(false); plain.BlobsWritten != 0 {
t.Fatalf("a plain re-emit wrote %d blobs; it is supposed to trust presence", plain.BlobsWritten)
}
if verified := emit(true); verified.BlobsWritten != 1 {
t.Fatalf("--verify wrote %d blobs, want 1 (the bad copy must be replaced)", verified.BlobsWritten)
}
fixture.zone.mu.Lock()
repaired := fixture.zone.objects[key]
fixture.zone.mu.Unlock()
if !bytes.Equal(repaired, good) {
t.Error("the replaced blob is not what the current encoder produces")
}
if _, err := inspectBlob(repaired); err != nil {
t.Errorf("the replaced blob still fails inspection: %v", err)
}
// A second verifying run has nothing left to repair.
if again := emit(true); again.BlobsWritten != 0 {
t.Errorf("--verify rewrote %d good blobs", again.BlobsWritten)
}
}
-8
View File
@@ -21,13 +21,6 @@ type fakeZone struct {
objects map[string][]byte objects map[string][]byte
puts []string puts []string
failOn func(key string) bool // when true, the PUT fails failOn func(key string) bool // when true, the PUT fails
url string // base the same objects are readable at
}
// pullZone reads the fake zone the way the public pull zone is read: plain
// unauthenticated GETs, no storage API.
func (z *fakeZone) pullZone() blobSource {
return newPullZone(z.url)
} }
func newFakeZone(t *testing.T) (*fakeZone, func(string) string) { func newFakeZone(t *testing.T) (*fakeZone, func(string) string) {
@@ -35,7 +28,6 @@ func newFakeZone(t *testing.T) (*fakeZone, func(string) string) {
zone := &fakeZone{objects: map[string][]byte{}} zone := &fakeZone{objects: map[string][]byte{}}
server := httptest.NewServer(zone) server := httptest.NewServer(zone)
t.Cleanup(server.Close) t.Cleanup(server.Close)
zone.url = server.URL + "/sow-nwsync"
getenv := func(name string) string { getenv := func(name string) string {
switch name { switch name {
case "NWSYNC_STORAGE_ZONE": case "NWSYNC_STORAGE_ZONE":
+1 -7
View File
@@ -126,14 +126,8 @@ func extractArchiveResources(p *project.Project, archive erf.Archive, desired ma
} }
// *palcus.itp are Toolset-generated palette projections; the module // *palcus.itp are Toolset-generated palette projections; the module
// build regenerates them from source blueprints, so extraction never // build regenerates them from source blueprints, so extraction never
// writes them back into source. The committed skeleton still has to // writes them back into source.
// survive stale cleanup, so mark its target as desired.
if ext == "itp" && isPaletteProjectionResref(resource.Name) { if ext == "itp" && isPaletteProjectionResref(resource.Name) {
effective := p.EffectiveConfig()
target, err := extractionTarget(p, "paths.source", effective.Paths.Source, p.SourceDir(), sourceSubdir(ext), strings.ToLower(resource.Name)+"."+ext+".json")
if err == nil {
desired[target] = struct{}{}
}
skippedCount++ skippedCount++
continue continue
} }
-82
View File
@@ -3312,88 +3312,6 @@ func TestExtractOverwritesAndRemovesStaleFiles(t *testing.T) {
} }
} }
func TestExtractKeepsPaletteSkeletonsThroughStaleCleanup(t *testing.T) {
root := t.TempDir()
mustMkdir(t, filepath.Join(root, "src", "module"))
mustMkdir(t, filepath.Join(root, "src", "palettes"))
mustMkdir(t, filepath.Join(root, "assets"))
mustMkdir(t, filepath.Join(root, "build"))
mustWriteFile(t, filepath.Join(root, "nwn-tool.json"), `{
"module": {
"name": "Test Module",
"resref": "testmod"
},
"paths": {
"source": "src",
"assets": "assets",
"build": "build"
}
}
`)
mustWriteFile(t, filepath.Join(root, "src", "module", "module.ifo.json"), `{
"file_type": "IFO ",
"file_version": "V3.2",
"root": {
"struct_type": 0,
"fields": [
{
"label": "Mod_Name",
"type": "CExoString",
"value": "Original Module"
}
]
}
}
`)
skeletonPath := filepath.Join(root, "src", "palettes", "creaturepalcus.itp.json")
mustWriteFile(t, skeletonPath, `{
"file_type": "ITP ",
"file_version": "V3.2",
"root": {
"struct_type": 0,
"fields": [
{
"label": "MAIN",
"type": "List",
"value": []
}
]
}
}
`)
p, err := project.Load(root)
if err != nil {
t.Fatalf("load project: %v", err)
}
if err := p.ValidateLayout(); err != nil {
t.Fatalf("validate layout: %v", err)
}
if err := p.Scan(); err != nil {
t.Fatalf("scan: %v", err)
}
if _, err := BuildModule(p); err != nil {
t.Fatalf("build module: %v", err)
}
if err := p.Scan(); err != nil {
t.Fatalf("rescan before extract: %v", err)
}
result, err := Extract(p)
if err != nil {
t.Fatalf("extract: %v", err)
}
if result.Removed != 0 {
t.Fatalf("expected no removed files, got %d", result.Removed)
}
if _, err := os.Stat(skeletonPath); err != nil {
t.Fatalf("expected palette skeleton to survive extract, stat err=%v", err)
}
}
func TestExtractMergesConfiguredGFFJSONFieldsAndLists(t *testing.T) { func TestExtractMergesConfiguredGFFJSONFieldsAndLists(t *testing.T) {
root := t.TempDir() root := t.TempDir()
mustMkdir(t, filepath.Join(root, "src", "module")) mustMkdir(t, filepath.Join(root, "src", "module"))
+2 -12
View File
@@ -560,18 +560,8 @@ func (p *Project) ValidateLayout() error {
if strings.TrimSpace(p.Config.Module.ResRef) == "" { if strings.TrimSpace(p.Config.Module.ResRef) == "" {
failures = append(failures, errors.New("module.resref is required")) failures = append(failures, errors.New("module.resref is required"))
} }
// module.resref names the built .mod FILE, so the 16-byte resref limit does not if len(p.Config.Module.ResRef) > 16 {
// apply to it — NWN:EE module file names are routinely longer. It is validated as failures = append(failures, fmt.Errorf("module.resref %q exceeds 16 characters", p.Config.Module.ResRef))
// a file name instead. The limit still binds when the same value has to be a real
// resref: with no haks configured, an asset project names its single generated HAK
// after it, and a HAK name is a resref the engine loads.
if err := validateOutputFileName("module.resref", p.Config.Module.ResRef+".mod", ".mod"); err != nil {
failures = append(failures, err)
}
if len(p.Config.Module.ResRef) > 16 && strings.TrimSpace(p.Config.Paths.Assets) != "" && len(p.Config.HAKs) == 0 {
failures = append(failures, fmt.Errorf(
"module.resref %q exceeds 16 characters and would name this project's generated HAK; configure haks[] with a shorter name",
p.Config.Module.ResRef))
} }
if strings.TrimSpace(p.Config.Paths.Source) == "" && strings.TrimSpace(p.Config.Paths.Assets) == "" && !p.HasTopData() { if strings.TrimSpace(p.Config.Paths.Source) == "" && strings.TrimSpace(p.Config.Paths.Assets) == "" && !p.HasTopData() {
failures = append(failures, errors.New("at least one of paths.source, paths.assets, or topdata.source is required")) failures = append(failures, errors.New("at least one of paths.source, paths.assets, or topdata.source is required"))
-72
View File
@@ -1093,78 +1093,6 @@ func TestValidateLayoutAllowsMissingAssetsDir(t *testing.T) {
} }
} }
// module.resref names the built .mod FILE, not a resource inside an archive, so the
// 16-byte resref limit does not apply to it. NWN:EE module file names are commonly
// longer (ShadowsOverWestgate.mod is 19). The limit still binds everywhere a resref
// really is a resref — see TestValidateLayoutRejectsLongResRefWhenItNamesAHAK.
func TestValidateLayoutAllowsLongModuleResRef(t *testing.T) {
root := t.TempDir()
mkdirAll(t, filepath.Join(root, "src"))
mkdirAll(t, filepath.Join(root, "build"))
proj := &Project{
Root: root,
Config: Config{
Module: ModuleConfig{Name: "Shadows Over Westgate", ResRef: "ShadowsOverWestgate"},
Paths: PathConfig{Source: "src", Build: "build"},
},
}
if err := proj.ValidateLayout(); err != nil {
t.Fatalf("ValidateLayout rejected a 19-character module file name: %v", err)
}
if got, want := filepath.Base(proj.ModuleArchivePath()), "ShadowsOverWestgate.mod"; got != want {
t.Fatalf("ModuleArchivePath() = %q, want %q", got, want)
}
}
// A module.resref that is not a usable file name is still rejected.
func TestValidateLayoutRejectsModuleResRefThatIsAPath(t *testing.T) {
root := t.TempDir()
mkdirAll(t, filepath.Join(root, "src"))
proj := &Project{
Root: root,
Config: Config{
Module: ModuleConfig{Name: "Test", ResRef: "../escape/mod"},
Paths: PathConfig{Source: "src", Build: "build"},
},
}
err := proj.ValidateLayout()
if err == nil {
t.Fatal("ValidateLayout accepted a module.resref containing a path")
}
if !strings.Contains(err.Error(), "module.resref") {
t.Fatalf("error does not name the offending field: %v", err)
}
}
// When a project declares no haks, the module resref becomes the name of the single
// generated HAK — and a HAK name IS a resref the engine loads. The limit applies
// there, so a long name is only allowed for projects that build no HAKs.
func TestValidateLayoutRejectsLongResRefWhenItNamesAHAK(t *testing.T) {
root := t.TempDir()
mkdirAll(t, filepath.Join(root, "src"))
mkdirAll(t, filepath.Join(root, "assets"))
proj := &Project{
Root: root,
Config: Config{
Module: ModuleConfig{Name: "Shadows Over Westgate", ResRef: "ShadowsOverWestgate"},
Paths: PathConfig{Source: "src", Assets: "assets", Build: "build"},
},
}
err := proj.ValidateLayout()
if err == nil {
t.Fatal("ValidateLayout accepted a 19-character name for a generated HAK")
}
if !strings.Contains(err.Error(), "16") {
t.Fatalf("error does not explain the resref limit: %v", err)
}
}
// paths.build is an OUTPUT dir the builder creates (MkdirAll) before writing, so // paths.build is an OUTPUT dir the builder creates (MkdirAll) before writing, so
// a bare clone with no build dir yet must still validate/build with no pre-step // a bare clone with no build dir yet must still validate/build with no pre-step
// (R2/parity). Only a build path that exists but is not a directory is an error. // (R2/parity). Only a build path that exists but is not a directory is an error.
+1 -2
View File
@@ -98,9 +98,8 @@ func buildGenerated2DAAssetGroup(p *project.Project, cfg project.GeneratedTopDat
} }
results := make([]Generated2DAAsset, 0, len(collected)) results := make([]Generated2DAAsset, 0, len(collected))
racialFeatRules := racialUsableFeatRules(collected)
for _, dataset := range collected { for _, dataset := range collected {
compiled, err := resolveNativeDataset(dataset, keyToID, rowByKey, tableRegistry, nil, project.TopDataClassFeatInjectionConfig{}, racialFeatRules, nil) compiled, err := resolveNativeDataset(dataset, keyToID, rowByKey, tableRegistry, nil, project.TopDataClassFeatInjectionConfig{}, nil)
if err != nil { if err != nil {
return nil, err return nil, err
} }
+4 -59
View File
@@ -530,7 +530,6 @@ func buildNativeUnchecked(p *project.Project, opts NativeBuildOptions, progress
groupStats := nativeCompileGroupStats(collected) groupStats := nativeCompileGroupStats(collected)
currentGroup := "" currentGroup := ""
sidecars := newNativeSidecarCollector() sidecars := newNativeSidecarCollector()
racialFeatRules := racialUsableFeatRules(collected)
for _, dataset := range collected { for _, dataset := range collected {
group := nativeCompileGroup(dataset.Dataset.Name) group := nativeCompileGroup(dataset.Dataset.Name)
if group != currentGroup { if group != currentGroup {
@@ -543,7 +542,7 @@ func buildNativeUnchecked(p *project.Project, opts NativeBuildOptions, progress
stats.SourceFragments, stats.SourceFragments,
)) ))
} }
compiled, err := resolveNativeDataset(dataset, globalKeyToID, globalRowByKey, tableRegistry, compiler, p.EffectiveConfig().TopData.ClassFeatInjections, racialFeatRules, sidecars) compiled, err := resolveNativeDataset(dataset, globalKeyToID, globalRowByKey, tableRegistry, compiler, p.EffectiveConfig().TopData.ClassFeatInjections, sidecars)
if err != nil { if err != nil {
return BuildResult{}, err return BuildResult{}, err
} }
@@ -3997,13 +3996,13 @@ func normalizeGlobalReferenceID(value string) string {
return parts[0] + ":" + strings.ReplaceAll(parts[1], "_", "") return parts[0] + ":" + strings.ReplaceAll(parts[1], "_", "")
} }
func resolveNativeDataset(dataset nativeCollectedDataset, keyToID map[string]int, globalRowByKey map[string]map[string]any, tableRegistry resolvedTableRegistry, compiler *tlkCompiler, classFeatInjections project.TopDataClassFeatInjectionConfig, racialFeatRules []project.TopDataClassFeatGlobalRule, sidecars *nativeSidecarCollector) (map[string]any, error) { func resolveNativeDataset(dataset nativeCollectedDataset, keyToID map[string]int, globalRowByKey map[string]map[string]any, tableRegistry resolvedTableRegistry, compiler *tlkCompiler, classFeatInjections project.TopDataClassFeatInjectionConfig, sidecars *nativeSidecarCollector) (map[string]any, error) {
rows := dataset.Rows rows := dataset.Rows
if strings.HasPrefix(filepath.ToSlash(dataset.Dataset.Name), "classes/feats/") { if strings.HasPrefix(filepath.ToSlash(dataset.Dataset.Name), "classes/feats/") {
classKey := "classes:" + dataset.Dataset.Name[strings.LastIndex(dataset.Dataset.Name, "/")+1:] classKey := "classes:" + dataset.Dataset.Name[strings.LastIndex(dataset.Dataset.Name, "/")+1:]
featSuccessors := buildFeatSuccessorsIndex(globalRowByKey, keyToID) featSuccessors := buildFeatSuccessorsIndex(globalRowByKey, keyToID)
classSkills := buildClassSkillsIndex(tableRegistry, classKey) classSkills := buildClassSkillsIndex(tableRegistry, classKey)
expanded, err := expandClassesFeatRows(rows, keyToID, globalRowByKey, featSuccessors, classSkills, globalRowByKey, classKey, classFeatInjections, racialFeatRules, !dataset.Dataset.HasGlobalInjections) expanded, err := expandClassesFeatRows(rows, keyToID, globalRowByKey, featSuccessors, classSkills, globalRowByKey, classKey, classFeatInjections, !dataset.Dataset.HasGlobalInjections)
if err != nil { if err != nil {
return nil, fmt.Errorf("dataset %s: %w", dataset.Dataset.Name, err) return nil, fmt.Errorf("dataset %s: %w", dataset.Dataset.Name, err)
} }
@@ -4052,18 +4051,12 @@ var (
} }
) )
func expandClassesFeatRows(rows []map[string]any, keyToID map[string]int, rowByKey map[string]map[string]any, featSuccessors map[string]string, classSkills map[string]bool, allRowByKey map[string]map[string]any, classKey string, classFeatInjections project.TopDataClassFeatInjectionConfig, racialFeatRules []project.TopDataClassFeatGlobalRule, useConfiguredInjections bool) ([]map[string]any, error) { func expandClassesFeatRows(rows []map[string]any, keyToID map[string]int, rowByKey map[string]map[string]any, featSuccessors map[string]string, classSkills map[string]bool, allRowByKey map[string]map[string]any, classKey string, classFeatInjections project.TopDataClassFeatInjectionConfig, useConfiguredInjections bool) ([]map[string]any, error) {
globalRules, classSkillRules := []project.TopDataClassFeatGlobalRule{}, []project.TopDataClassFeatMasterfeatRule{} globalRules, classSkillRules := []project.TopDataClassFeatGlobalRule{}, []project.TopDataClassFeatMasterfeatRule{}
if useConfiguredInjections { if useConfiguredInjections {
globalRules, classSkillRules = effectiveClassFeatInjectionRules(classFeatInjections) globalRules, classSkillRules = effectiveClassFeatInjectionRules(classFeatInjections)
} }
// Racial usable-feat rows are generated from the race feats tables, not the
// hand-authored class-feat injections, so they apply to every class table
// regardless of whether that dataset carries its own global.json (which is
// what gates useConfiguredInjections). Deduped below against rows already
// present, so a leftover hand row in global.json is a no-op, not a double.
globalRules = append(globalRules, racialFeatRules...)
injected := make([]map[string]any, 0, len(globalRules)+len(classSkillRules)) injected := make([]map[string]any, 0, len(globalRules)+len(classSkillRules))
presentRefIDs := make(map[string]struct{}, len(rows)) presentRefIDs := make(map[string]struct{}, len(rows))
for _, row := range rows { for _, row := range rows {
@@ -4141,54 +4134,6 @@ func expandClassesFeatRows(rows []map[string]any, keyToID map[string]int, rowByK
return combined, nil return combined, nil
} }
// racialUsableFeatRules builds one class-feat injection rule per feat marked
// UsableFeat=1 in any race feats table (race_feat_*.2da). Racial feats are
// granted by race, never by a class, so an activatable one needs a menu-only
// cls_feat row to reach the client radial: List=3 keeps it off every level-up
// selection list, GrantedOnLevel=99 sits above the level cap so no class ever
// actually grants it, and OnMenu=1 renders the button once the creature holds
// the feat (possession comes from chargen / the login racial-feat sync). This
// replaces the hand-maintained racial rows in classes/feats/global.json - mark
// UsableFeat in the race table and the radial row follows automatically.
func racialUsableFeatRules(collected []nativeCollectedDataset) []project.TopDataClassFeatGlobalRule {
seen := map[string]struct{}{}
rules := []project.TopDataClassFeatGlobalRule{}
for _, ds := range collected {
if !strings.HasPrefix(ds.Dataset.OutputName, "race_feat_") || !strings.HasSuffix(ds.Dataset.OutputName, ".2da") {
continue
}
for _, row := range ds.Rows {
if usable, err := asInt(fieldValue(row, "UsableFeat")); err != nil || usable != 1 {
continue
}
featRef, ok := row["FeatIndex"].(map[string]any)
if !ok {
continue
}
featID, _ := featRef["id"].(string)
if featID == "" {
continue
}
if _, dup := seen[featID]; dup {
continue
}
seen[featID] = struct{}{}
rules = append(rules, project.TopDataClassFeatGlobalRule{
Feat: featID,
List: "3",
GrantedOnLevel: "99",
OnMenu: "1",
})
}
}
// Discovery order across files and the dedup map are both unordered; sort so
// the injected rows (and the resulting 2DA row numbering) are deterministic.
slices.SortFunc(rules, func(a, b project.TopDataClassFeatGlobalRule) int {
return strings.Compare(a.Feat, b.Feat)
})
return rules
}
func effectiveClassFeatInjectionRules(config project.TopDataClassFeatInjectionConfig) ([]project.TopDataClassFeatGlobalRule, []project.TopDataClassFeatMasterfeatRule) { func effectiveClassFeatInjectionRules(config project.TopDataClassFeatInjectionConfig) ([]project.TopDataClassFeatGlobalRule, []project.TopDataClassFeatMasterfeatRule) {
if len(config.GlobalFeats) == 0 && len(config.ClassSkillMasterfeats) == 0 { if len(config.GlobalFeats) == 0 && len(config.ClassSkillMasterfeats) == 0 {
return defaultClassFeatGlobalRules, defaultClassFeatClassSkillMasterfeatRules return defaultClassFeatGlobalRules, defaultClassFeatClassSkillMasterfeatRules
@@ -1,54 +0,0 @@
package topdata
import (
"reflect"
"testing"
"git.westgate.pw/ShadowsOverWestgate/sow-tools/internal/project"
)
func raceFeatDataset(output string, rows ...map[string]any) nativeCollectedDataset {
return nativeCollectedDataset{
Dataset: nativeDataset{OutputName: output},
Rows: rows,
}
}
func usableRow(featID string, usable any) map[string]any {
row := map[string]any{"FeatIndex": map[string]any{"id": featID}}
if usable != nil {
row["UsableFeat"] = usable
}
return row
}
func TestRacialUsableFeatRules(t *testing.T) {
collected := []nativeCollectedDataset{
raceFeatDataset("race_feat_ddrw.2da",
usableRow("feat:keen_sense", nil), // passive, no UsableFeat -> skipped
usableRow("feat:darkvision", 1), // usable
usableRow("feat:use_poison", 0), // explicitly not usable -> skipped
usableRow("feat:drow/faerie_fire", "1"),// usable, string form
),
raceFeatDataset("race_feat_tief.2da",
usableRow("feat:darkvision", 1), // duplicate across races -> collapses to one
usableRow("feat:tiefling/darkness", 1),
),
raceFeatDataset("feat.2da", // not a race feats table -> ignored entirely
usableRow("feat:power_attack", 1),
),
}
got := racialUsableFeatRules(collected)
want := []project.TopDataClassFeatGlobalRule{
{Feat: "feat:darkvision", List: "3", GrantedOnLevel: "99", OnMenu: "1"},
{Feat: "feat:drow/faerie_fire", List: "3", GrantedOnLevel: "99", OnMenu: "1"},
{Feat: "feat:tiefling/darkness", List: "3", GrantedOnLevel: "99", OnMenu: "1"},
}
// Slice is sorted by Feat, so order is deterministic.
if !reflect.DeepEqual(got, want) {
t.Fatalf("rules: got %+v, want %+v", got, want)
}
}
-1
View File
@@ -1523,7 +1523,6 @@ func TestResolveNativeDatasetPreservesScalarTableReferenceBehavior(t *testing.T)
nil, nil,
project.TopDataClassFeatInjectionConfig{}, project.TopDataClassFeatInjectionConfig{},
nil, nil,
nil,
) )
if err != nil { if err != nil {
t.Fatalf("resolveNativeDataset failed: %v", err) t.Fatalf("resolveNativeDataset failed: %v", err)
+28 -127
View File
@@ -8,7 +8,6 @@ import (
"fmt" "fmt"
"html" "html"
"io" "io"
"maps"
"net/http" "net/http"
"net/url" "net/url"
"os" "os"
@@ -53,7 +52,7 @@ type DeployWikiOptions struct {
TitlePrefixMinLength int TitlePrefixMinLength int
} }
type DeployResult struct { type deployResult struct {
LocalPages int LocalPages int
Created int Created int
Updated int Updated int
@@ -64,19 +63,6 @@ type DeployResult struct {
Drifted int Drifted int
Renamed int Renamed int
Manifest string Manifest string
// ResetPurged counts the deletions queued by --reset-managed-namespaces.
// They are also included in Stale and Purged, because callers warn about
// destructive policies in terms of the stale count: sow-topdata's
// deploy-wiki wrapper prints "of the pages counted as 'stale' below, they
// will be DELETED from NodeBB" directly above this block.
ResetPurged int
// ResetUnrecognized counts the subset of ResetPurged that the deploy
// manifest has no record of writing. Those deletions are the ones a
// re-seed cannot undo.
ResetUnrecognized int
// ResetSkipped counts reset targets NodeBB refused to delete, such as the
// wiki home topic, which the plugin excludes from tombstone and purge.
ResetSkipped int
} }
type wikiDeployPage struct { type wikiDeployPage struct {
@@ -114,12 +100,6 @@ type wikiDeployPlan struct {
Content string Content string
RemoteHash string RemoteHash string
Title string Title string
// Reset marks a purge queued by --reset-managed-namespaces rather than by
// stale computation over the manifest.
Reset bool
// Unrecognized marks a reset purge whose topic the manifest has no record
// of writing.
Unrecognized bool
} }
type wikiNamespacesDocument struct { type wikiNamespacesDocument struct {
@@ -136,7 +116,7 @@ type wikiNamespaceDeclaration struct {
EditPolicy string `json:"edit_policy" yaml:"edit_policy"` EditPolicy string `json:"edit_policy" yaml:"edit_policy"`
} }
func DeployWikiWithOptions(p *project.Project, opts DeployWikiOptions, progress func(string)) (DeployResult, error) { func DeployWikiWithOptions(p *project.Project, opts DeployWikiOptions, progress func(string)) (deployResult, error) {
if progress == nil { if progress == nil {
progress = func(string) {} progress = func(string) {}
} }
@@ -152,19 +132,19 @@ func DeployWikiWithOptions(p *project.Project, opts DeployWikiOptions, progress
} }
} }
if _, err := os.Stat(opts.SourceDir); err != nil { if _, err := os.Stat(opts.SourceDir); err != nil {
return DeployResult{}, fmt.Errorf("wiki source directory not found: %w", err) return deployResult{}, fmt.Errorf("wiki source directory not found: %w", err)
} }
if opts.Endpoint == "" { if opts.Endpoint == "" {
return DeployResult{}, errors.New("NODEBB_API_ENDPOINT is required") return deployResult{}, errors.New("NODEBB_API_ENDPOINT is required")
} }
if opts.Token == "" { if opts.Token == "" {
return DeployResult{}, errors.New("NODEBB_API_TOKEN is required") return deployResult{}, errors.New("NODEBB_API_TOKEN is required")
} }
if opts.Username != "" || opts.Password != "" || opts.NotesDelimiter != "" { if opts.Username != "" || opts.Password != "" || opts.NotesDelimiter != "" {
return DeployResult{}, errors.New("DokuWiki deployment options are no longer supported; use NodeBB endpoint, token, and category mappings") return deployResult{}, errors.New("DokuWiki deployment options are no longer supported; use NodeBB endpoint, token, and category mappings")
} }
if opts.StalePolicy != "" && opts.StalePolicy != "report" && opts.StalePolicy != "archive" && opts.StalePolicy != "purge" { if opts.StalePolicy != "" && opts.StalePolicy != "report" && opts.StalePolicy != "archive" && opts.StalePolicy != "purge" {
return DeployResult{}, fmt.Errorf("wiki stale policy %q is not supported", opts.StalePolicy) return deployResult{}, fmt.Errorf("wiki stale policy %q is not supported", opts.StalePolicy)
} }
if opts.TitlePrefixMinLength <= 0 { if opts.TitlePrefixMinLength <= 0 {
opts.TitlePrefixMinLength = p.EffectiveConfig().TopData.Wiki.TitlePrefixMinLength opts.TitlePrefixMinLength = p.EffectiveConfig().TopData.Wiki.TitlePrefixMinLength
@@ -174,7 +154,7 @@ func DeployWikiWithOptions(p *project.Project, opts DeployWikiOptions, progress
if len(namespaces) == 0 { if len(namespaces) == 0 {
declarations, err := loadWikiNamespaceDeclarations(p) declarations, err := loadWikiNamespaceDeclarations(p)
if err != nil { if err != nil {
return DeployResult{}, err return deployResult{}, err
} }
for _, declaration := range declarations { for _, declaration := range declarations {
namespaces = append(namespaces, declaration.ID) namespaces = append(namespaces, declaration.ID)
@@ -184,7 +164,7 @@ func DeployWikiWithOptions(p *project.Project, opts DeployWikiOptions, progress
} }
envCategories, err := categoryIDsFromNamespaceEnv(declarations) envCategories, err := categoryIDsFromNamespaceEnv(declarations)
if err != nil { if err != nil {
return DeployResult{}, err return deployResult{}, err
} }
if len(envCategories) > 0 { if len(envCategories) > 0 {
merged := envCategories merged := envCategories
@@ -203,7 +183,7 @@ func DeployWikiWithOptions(p *project.Project, opts DeployWikiOptions, progress
progress(fmt.Sprintf("Collecting local wiki pages from %s", opts.SourceDir)) progress(fmt.Sprintf("Collecting local wiki pages from %s", opts.SourceDir))
pages, err := collectLocalPages(opts.SourceDir, opts.PageIndexPath, namespaces) pages, err := collectLocalPages(opts.SourceDir, opts.PageIndexPath, namespaces)
if err != nil { if err != nil {
return DeployResult{}, err return deployResult{}, err
} }
progress(fmt.Sprintf("Loaded %d local wiki page(s)", len(pages))) progress(fmt.Sprintf("Loaded %d local wiki page(s)", len(pages)))
manifest := loadDeployManifest(manifestPath) manifest := loadDeployManifest(manifestPath)
@@ -213,7 +193,7 @@ func DeployWikiWithOptions(p *project.Project, opts DeployWikiOptions, progress
progress(fmt.Sprintf("Planning NodeBB wiki deploy for %d local page(s)", len(pages))) progress(fmt.Sprintf("Planning NodeBB wiki deploy for %d local page(s)", len(pages)))
plans, result, nextManifest, err := planNodeBBDeploy(pages, manifest, opts, client, progress) plans, result, nextManifest, err := planNodeBBDeploy(pages, manifest, opts, client, progress)
if err != nil { if err != nil {
return DeployResult{}, err return deployResult{}, err
} }
result.LocalPages = len(pages) result.LocalPages = len(pages)
result.Manifest = manifestPath result.Manifest = manifestPath
@@ -232,8 +212,6 @@ func DeployWikiWithOptions(p *project.Project, opts DeployWikiOptions, progress
} }
orderedPlans := orderNodeBBDeployPlans(plans) orderedPlans := orderNodeBBDeployPlans(plans)
progress(fmt.Sprintf("Executing NodeBB wiki actions: total %d, create %d, update %d, rename %d, archive %d, purge %d", len(orderedPlans), result.Created, result.Updated, result.Renamed, result.Archived, result.Purged)) progress(fmt.Sprintf("Executing NodeBB wiki actions: total %d, create %d, update %d, rename %d, archive %d, purge %d", len(orderedPlans), result.Created, result.Updated, result.Renamed, result.Archived, result.Purged))
resetPurgedByCID := map[int]int{}
resetSkippedByCID := map[int]int{}
for i, plan := range orderedPlans { for i, plan := range orderedPlans {
if shouldReportNodeBBDeployActionProgress(i, len(orderedPlans)) { if shouldReportNodeBBDeployActionProgress(i, len(orderedPlans)) {
progress(fmt.Sprintf("Executing NodeBB wiki action %d/%d: %s %s", i+1, len(orderedPlans), plan.Action, plan.Page.PageID)) progress(fmt.Sprintf("Executing NodeBB wiki action %d/%d: %s %s", i+1, len(orderedPlans), plan.Action, plan.Page.PageID))
@@ -264,52 +242,19 @@ func DeployWikiWithOptions(p *project.Project, opts DeployWikiOptions, progress
return result, fmt.Errorf("deploy wiki page %q: update NodeBB post %d: %w", plan.Page.PageID, plan.Entry.PID, err) return result, fmt.Errorf("deploy wiki page %q: update NodeBB post %d: %w", plan.Page.PageID, plan.Entry.PID, err)
} }
case "archive": case "archive":
// Archiving rewrites the page rather than removing it, so it goes
// through the ordinary post edit the wiki plugin allows; only
// delete, restore, and purge are reserved to the page actions.
if err := client.updatePost(plan.Entry.TID, plan.Entry.PID, plan.Content, summary); err != nil { if err := client.updatePost(plan.Entry.TID, plan.Entry.PID, plan.Content, summary); err != nil {
return result, fmt.Errorf("deploy wiki page %q: archive NodeBB post %d: %w", plan.Page.PageID, plan.Entry.PID, err) return result, fmt.Errorf("deploy wiki page %q: archive NodeBB post %d: %w", plan.Page.PageID, plan.Entry.PID, err)
} }
case "purge": case "purge":
if err := client.purgeWikiPage(plan.Entry.TID); err != nil { if err := client.purgeTopic(plan.Entry.TID); err != nil {
// A namespace reset sweeps every topic in the category, so it
// can reach pages NodeBB will not delete at all — the wiki home
// topic above all. Skip those rather than abandoning the reset.
if plan.Reset && isNodeBBWikiPageUndeletable(err) {
result.ResetSkipped++
result.ResetPurged--
result.Purged--
result.Stale--
if plan.Unrecognized {
result.ResetUnrecognized--
}
resetSkippedByCID[plan.Entry.CID]++
progress(fmt.Sprintf("NodeBB refused to delete wiki topic %d during managed namespace reset; skipping it", plan.Entry.TID))
continue
}
return result, fmt.Errorf("deploy wiki page %q: purge NodeBB topic %d: %w", plan.Page.PageID, plan.Entry.TID, err) return result, fmt.Errorf("deploy wiki page %q: purge NodeBB topic %d: %w", plan.Page.PageID, plan.Entry.TID, err)
} }
if plan.Reset {
resetPurgedByCID[plan.Entry.CID]++
}
case "rename": case "rename":
if err := client.renameWikiPage(plan.Entry.TID, plan.Entry.CID, plan.Title); err != nil { if err := client.renameWikiPage(plan.Entry.TID, plan.Entry.CID, plan.Title); err != nil {
return result, fmt.Errorf("deploy wiki page %q: rename NodeBB topic %d to %q: %w", plan.Page.PageID, plan.Entry.TID, plan.Title, err) return result, fmt.Errorf("deploy wiki page %q: rename NodeBB topic %d to %q: %w", plan.Page.PageID, plan.Entry.TID, plan.Title, err)
} }
} }
} }
// NodeBB answers 403 both for the pages it will never delete — the wiki
// home topic — and for a token without purge privileges, and the two are
// not distinguishable from the response. What tells them apart is scope: a
// category where nothing at all could be deleted is a privilege problem,
// not a home page. Reporting that as a completed reset would leave the
// manifest claiming a fresh start over pages that are all still there, so
// the next deploy would recreate every one of them as a duplicate.
for _, cid := range slices.Sorted(maps.Keys(resetSkippedByCID)) {
if resetPurgedByCID[cid] == 0 {
return result, fmt.Errorf("NodeBB refused every managed namespace reset deletion in category %d (%d topic(s)); check that the deploy token has wiki purge privileges there", cid, resetSkippedByCID[cid])
}
}
if err := saveDeployManifest(manifestPath, nextManifest); err != nil { if err := saveDeployManifest(manifestPath, nextManifest); err != nil {
return result, err return result, err
} }
@@ -541,7 +486,7 @@ func isDir(path string) bool {
return err == nil && info.IsDir() return err == nil && info.IsDir()
} }
func planNodeBBDeploy(pages map[string]wikiDeployPage, manifest wikiDeployManifest, opts DeployWikiOptions, client *nodeBBClient, progress func(string)) ([]wikiDeployPlan, DeployResult, wikiDeployManifest, error) { func planNodeBBDeploy(pages map[string]wikiDeployPage, manifest wikiDeployManifest, opts DeployWikiOptions, client *nodeBBClient, progress func(string)) ([]wikiDeployPlan, deployResult, wikiDeployManifest, error) {
if progress == nil { if progress == nil {
progress = func(string) {} progress = func(string) {}
} }
@@ -549,23 +494,17 @@ func planNodeBBDeploy(pages map[string]wikiDeployPage, manifest wikiDeployManife
next := wikiDeployManifest{Version: "nodebb-v1", Pages: map[string]wikiDeployManifestPage{}} next := wikiDeployManifest{Version: "nodebb-v1", Pages: map[string]wikiDeployManifestPage{}}
remotePagesByCID := map[int][]nodeBBWikiPage{} remotePagesByCID := map[int][]nodeBBWikiPage{}
var plans []wikiDeployPlan var plans []wikiDeployPlan
var result DeployResult var result deployResult
if opts.ResetManagedNamespaces { if opts.ResetManagedNamespaces {
if !opts.AllowCreates && len(pageIDs) > 0 { if !opts.AllowCreates && len(pageIDs) > 0 {
return nil, result, next, errors.New("wiki managed namespace reset requires --create so current generated pages can be recreated") return nil, result, next, errors.New("wiki managed namespace reset requires --create so current generated pages can be recreated")
} }
resetPlans, purged, unrecognized, err := planManagedNamespaceReset(opts, manifest, client, progress) resetPlans, purged, err := planManagedNamespaceReset(opts, client, progress)
if err != nil { if err != nil {
return nil, result, next, err return nil, result, next, err
} }
plans = append(plans, resetPlans...) plans = append(plans, resetPlans...)
result.Purged += purged result.Purged += purged
// A namespace reset deletes remote pages the same way stale purge does,
// so it is counted as stale: that is the number callers word their
// destructive-policy warning around.
result.Stale += purged
result.ResetPurged = purged
result.ResetUnrecognized = unrecognized
manifest = wikiDeployManifest{Version: "nodebb-v1", Pages: map[string]wikiDeployManifestPage{}} manifest = wikiDeployManifest{Version: "nodebb-v1", Pages: map[string]wikiDeployManifestPage{}}
for _, cid := range opts.CategoryIDs { for _, cid := range opts.CategoryIDs {
if cid != 0 { if cid != 0 {
@@ -743,11 +682,7 @@ func planNodeBBDeploy(pages map[string]wikiDeployPage, manifest wikiDeployManife
return plans, result, next, nil return plans, result, next, nil
} }
// planManagedNamespaceReset queues a purge for every topic in each managed func planManagedNamespaceReset(opts DeployWikiOptions, client *nodeBBClient, progress func(string)) ([]wikiDeployPlan, int, error) {
// category, not only the ones the manifest says we wrote. The second return is
// the plan count; the third is how many of those topics the manifest has no
// record of, which is the subset a re-seed cannot put back.
func planManagedNamespaceReset(opts DeployWikiOptions, manifest wikiDeployManifest, client *nodeBBClient, progress func(string)) ([]wikiDeployPlan, int, int, error) {
namespaces := slices.Clone(opts.Namespaces) namespaces := slices.Clone(opts.Namespaces)
if len(namespaces) == 0 { if len(namespaces) == 0 {
for namespace := range opts.CategoryIDs { for namespace := range opts.CategoryIDs {
@@ -756,15 +691,7 @@ func planManagedNamespaceReset(opts DeployWikiOptions, manifest wikiDeployManife
} }
slices.Sort(namespaces) slices.Sort(namespaces)
manifestTIDs := map[int]struct{}{}
for _, entry := range manifest.Pages {
if entry.TID != 0 {
manifestTIDs[entry.TID] = struct{}{}
}
}
seenTIDs := map[int]struct{}{} seenTIDs := map[int]struct{}{}
unrecognized := 0
var plans []wikiDeployPlan var plans []wikiDeployPlan
for _, namespace := range namespaces { for _, namespace := range namespaces {
cid := opts.CategoryIDs[namespace] cid := opts.CategoryIDs[namespace]
@@ -774,7 +701,7 @@ func planManagedNamespaceReset(opts DeployWikiOptions, manifest wikiDeployManife
progress(fmt.Sprintf("Listing NodeBB wiki namespace category %d for managed reset", cid)) progress(fmt.Sprintf("Listing NodeBB wiki namespace category %d for managed reset", cid))
remotePages, err := client.listNamespacePages(cid) remotePages, err := client.listNamespacePages(cid)
if err != nil { if err != nil {
return nil, 0, 0, err return nil, 0, err
} }
slices.SortFunc(remotePages, func(a, b nodeBBWikiPage) int { slices.SortFunc(remotePages, func(a, b nodeBBWikiPage) int {
return a.TID - b.TID return a.TID - b.TID
@@ -787,25 +714,19 @@ func planManagedNamespaceReset(opts DeployWikiOptions, manifest wikiDeployManife
continue continue
} }
seenTIDs[remotePage.TID] = struct{}{} seenTIDs[remotePage.TID] = struct{}{}
_, known := manifestTIDs[remotePage.TID]
if !known {
unrecognized++
}
title := strings.TrimSpace(remotePage.Title) title := strings.TrimSpace(remotePage.Title)
if title == "" { if title == "" {
title = strings.TrimSpace(remotePage.TitleLeaf) title = strings.TrimSpace(remotePage.TitleLeaf)
} }
pageID := fmt.Sprintf("%s:reset-topic-%d", namespace, remotePage.TID) pageID := fmt.Sprintf("%s:reset-topic-%d", namespace, remotePage.TID)
plans = append(plans, wikiDeployPlan{ plans = append(plans, wikiDeployPlan{
Page: wikiDeployPage{PageID: pageID, Title: title, Namespace: namespace}, Page: wikiDeployPage{PageID: pageID, Title: title, Namespace: namespace},
Entry: wikiDeployManifestPage{TID: remotePage.TID, CID: cid, Namespace: namespace, Title: title}, Entry: wikiDeployManifestPage{TID: remotePage.TID, CID: cid, Namespace: namespace, Title: title},
Action: "purge", Action: "purge",
Reset: true,
Unrecognized: !known,
}) })
} }
} }
return plans, len(plans), unrecognized, nil return plans, len(plans), nil
} }
func findMappedRemoteTopic(entry wikiDeployManifestPage, client *nodeBBClient) (nodeBBPost, bool, error) { func findMappedRemoteTopic(entry wikiDeployManifestPage, client *nodeBBClient) (nodeBBPost, bool, error) {
@@ -1690,36 +1611,16 @@ func (c *nodeBBClient) renameWikiPage(tid, cid int, title string) error {
return c.request("PUT", "/api/v3/plugins/westgate-wiki/page/move", body, nil) return c.request("PUT", "/api/v3/plugins/westgate-wiki/page/move", body, nil)
} }
// purgeWikiPage deletes a wiki topic through the wiki plugin's own page func (c *nodeBBClient) purgeTopic(tid int) error {
// actions. The core DELETE /api/v3/topics/{tid} route is refused for topics in
// wiki categories, because revision history is plugin-owned, so a purge has to
// tombstone the page first and then hard-purge it.
func (c *nodeBBClient) purgeWikiPage(tid int) error {
if tid == 0 { if tid == 0 {
return fmt.Errorf("NodeBB wiki page purge requires topic id") return fmt.Errorf("NodeBB topic purge requires topic id")
} }
body := map[string]any{"tid": tid} err := c.request(http.MethodDelete, fmt.Sprintf("/api/v3/topics/%d", tid), nil, nil)
if err := c.request(http.MethodPut, "/api/v3/plugins/westgate-wiki/page/tombstone", body, nil); err != nil {
if isNodeBBMissingResource(err) {
return nil
}
return err
}
if err := c.request(http.MethodDelete, "/api/v3/plugins/westgate-wiki/page/hard-purge", body, nil); err != nil {
if isNodeBBMissingResource(err) {
return nil
}
return err
}
return nil
}
// isNodeBBWikiPageUndeletable reports whether NodeBB refused to delete the page
// outright rather than failing transiently. The wiki home topic answers this
// way: the plugin excludes it from tombstone, restore, and purge alike.
func isNodeBBWikiPageUndeletable(err error) bool {
var httpErr nodeBBHTTPError var httpErr nodeBBHTTPError
return errors.As(err, &httpErr) && httpErr.Status == http.StatusForbidden if errors.As(err, &httpErr) && (httpErr.Status == http.StatusNotFound || httpErr.Status == http.StatusGone) {
return nil
}
return err
} }
func (c *nodeBBClient) acquireEditLock(tid int) (nodeBBEditLock, error) { func (c *nodeBBClient) acquireEditLock(tid int) (nodeBBEditLock, error) {
+90 -281
View File
@@ -2,7 +2,6 @@ package topdata
import ( import (
"encoding/json" "encoding/json"
"fmt"
"net/http" "net/http"
"net/http/httptest" "net/http/httptest"
"os" "os"
@@ -36,7 +35,7 @@ func TestDeployWikiDryRunDoesNotWriteRemoteOrManifest(t *testing.T) {
} }
updateCalls := 0 updateCalls := 0
server := newFakeNodeBB(t, func(w http.ResponseWriter, r *http.Request) { server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if got := r.Header.Get("Authorization"); got != "Bearer nodebb-token" { if got := r.Header.Get("Authorization"); got != "Bearer nodebb-token" {
t.Fatalf("unexpected authorization header %q", got) t.Fatalf("unexpected authorization header %q", got)
} }
@@ -51,7 +50,7 @@ func TestDeployWikiDryRunDoesNotWriteRemoteOrManifest(t *testing.T) {
default: default:
t.Fatalf("unexpected request %s %s", r.Method, r.URL.Path) t.Fatalf("unexpected request %s %s", r.Method, r.URL.Path)
} }
}) }))
defer server.Close() defer server.Close()
result, err := DeployWikiWithOptions(&project.Project{}, DeployWikiOptions{ result, err := DeployWikiWithOptions(&project.Project{}, DeployWikiOptions{
@@ -93,7 +92,7 @@ func TestDeployWikiReportsPlanningProgressBeforeRemoteWork(t *testing.T) {
} }
progress := []string{} progress := []string{}
server := newFakeNodeBB(t, func(w http.ResponseWriter, r *http.Request) { server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
switch { switch {
case r.Method == http.MethodGet && r.URL.Path == "/api/v3/plugins/westgate-wiki/namespace/3/pages": case r.Method == http.MethodGet && r.URL.Path == "/api/v3/plugins/westgate-wiki/namespace/3/pages":
w.Header().Set("Content-Type", "application/json") w.Header().Set("Content-Type", "application/json")
@@ -101,7 +100,7 @@ func TestDeployWikiReportsPlanningProgressBeforeRemoteWork(t *testing.T) {
default: default:
t.Fatalf("unexpected request %s %s", r.Method, r.URL.String()) t.Fatalf("unexpected request %s %s", r.Method, r.URL.String())
} }
}) }))
defer server.Close() defer server.Close()
_, err := DeployWikiWithOptions(&project.Project{}, DeployWikiOptions{ _, err := DeployWikiWithOptions(&project.Project{}, DeployWikiOptions{
@@ -175,7 +174,7 @@ func TestDeployWikiReportsLiveExecutionProgress(t *testing.T) {
} }
progress := []string{} progress := []string{}
server := newFakeNodeBB(t, func(w http.ResponseWriter, r *http.Request) { server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
switch { switch {
case r.Method == http.MethodGet && r.URL.Path == "/api/v3/topics/7": case r.Method == http.MethodGet && r.URL.Path == "/api/v3/topics/7":
w.Header().Set("Content-Type", "application/json") w.Header().Set("Content-Type", "application/json")
@@ -195,7 +194,7 @@ func TestDeployWikiReportsLiveExecutionProgress(t *testing.T) {
default: default:
t.Fatalf("unexpected request %s %s", r.Method, r.URL.String()) t.Fatalf("unexpected request %s %s", r.Method, r.URL.String())
} }
}) }))
defer server.Close() defer server.Close()
result, err := DeployWikiWithOptions(&project.Project{}, DeployWikiOptions{ result, err := DeployWikiWithOptions(&project.Project{}, DeployWikiOptions{
@@ -229,7 +228,7 @@ func TestDeployWikiReportsLiveExecutionProgress(t *testing.T) {
func TestNodeBBNamespacePaginationStopsWhenRepeatedCursorReturnsNoNewPages(t *testing.T) { func TestNodeBBNamespacePaginationStopsWhenRepeatedCursorReturnsNoNewPages(t *testing.T) {
requests := 0 requests := 0
server := newFakeNodeBB(t, func(w http.ResponseWriter, r *http.Request) { server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
requests++ requests++
w.Header().Set("Content-Type", "application/json") w.Header().Set("Content-Type", "application/json")
_ = json.NewEncoder(w).Encode(map[string]any{ _ = json.NewEncoder(w).Encode(map[string]any{
@@ -241,7 +240,7 @@ func TestNodeBBNamespacePaginationStopsWhenRepeatedCursorReturnsNoNewPages(t *te
"nextCursor": "same-cursor", "nextCursor": "same-cursor",
}, },
}) })
}) }))
defer server.Close() defer server.Close()
client := newNodeBBClient(server.URL, "nodebb-token") client := newNodeBBClient(server.URL, "nodebb-token")
@@ -259,7 +258,7 @@ func TestNodeBBNamespacePaginationStopsWhenRepeatedCursorReturnsNoNewPages(t *te
func TestNodeBBNamespacePaginationRejectsRepeatedCursorWithNewPages(t *testing.T) { func TestNodeBBNamespacePaginationRejectsRepeatedCursorWithNewPages(t *testing.T) {
requests := 0 requests := 0
server := newFakeNodeBB(t, func(w http.ResponseWriter, r *http.Request) { server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
requests++ requests++
w.Header().Set("Content-Type", "application/json") w.Header().Set("Content-Type", "application/json")
pages := []map[string]any{ pages := []map[string]any{
@@ -277,7 +276,7 @@ func TestNodeBBNamespacePaginationRejectsRepeatedCursorWithNewPages(t *testing.T
"nextCursor": "same-cursor", "nextCursor": "same-cursor",
}, },
}) })
}) }))
defer server.Close() defer server.Close()
client := newNodeBBClient(server.URL, "nodebb-token") client := newNodeBBClient(server.URL, "nodebb-token")
@@ -481,7 +480,7 @@ func TestDeployWikiDryRunReadoptsMissingMappedPost(t *testing.T) {
t.Fatalf("write deploy manifest: %v", err) t.Fatalf("write deploy manifest: %v", err)
} }
server := newFakeNodeBB(t, func(w http.ResponseWriter, r *http.Request) { server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
switch { switch {
case r.Method == http.MethodGet && r.URL.Path == "/api/v3/posts/42": case r.Method == http.MethodGet && r.URL.Path == "/api/v3/posts/42":
w.Header().Set("Content-Type", "application/json") w.Header().Set("Content-Type", "application/json")
@@ -509,7 +508,7 @@ func TestDeployWikiDryRunReadoptsMissingMappedPost(t *testing.T) {
default: default:
t.Fatalf("unexpected request %s %s", r.Method, r.URL.String()) t.Fatalf("unexpected request %s %s", r.Method, r.URL.String())
} }
}) }))
defer server.Close() defer server.Close()
result, err := DeployWikiWithOptions(&project.Project{}, DeployWikiOptions{ result, err := DeployWikiWithOptions(&project.Project{}, DeployWikiOptions{
@@ -732,7 +731,7 @@ func TestDeployWikiCreatesNodeBBTopicAndWritesManifest(t *testing.T) {
} }
createCalls := 0 createCalls := 0
server := newFakeNodeBB(t, func(w http.ResponseWriter, r *http.Request) { server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if got := r.Header.Get("Authorization"); got != "Bearer nodebb-token" { if got := r.Header.Get("Authorization"); got != "Bearer nodebb-token" {
t.Fatalf("unexpected authorization header %q", got) t.Fatalf("unexpected authorization header %q", got)
} }
@@ -770,7 +769,7 @@ func TestDeployWikiCreatesNodeBBTopicAndWritesManifest(t *testing.T) {
}, },
}, },
}) })
}) }))
defer server.Close() defer server.Close()
manifestPath := filepath.Join(root, "deploy-manifest.json") manifestPath := filepath.Join(root, "deploy-manifest.json")
@@ -849,7 +848,7 @@ func TestDeployWikiRepairsManifestedPageMissingSourceContentSync(t *testing.T) {
} }
updateCalls := 0 updateCalls := 0
server := newFakeNodeBB(t, func(w http.ResponseWriter, r *http.Request) { server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
switch { switch {
case r.Method == http.MethodGet && r.URL.Path == "/api/v3/posts/42": case r.Method == http.MethodGet && r.URL.Path == "/api/v3/posts/42":
w.Header().Set("Content-Type", "application/json") w.Header().Set("Content-Type", "application/json")
@@ -877,7 +876,7 @@ func TestDeployWikiRepairsManifestedPageMissingSourceContentSync(t *testing.T) {
default: default:
t.Fatalf("unexpected request %s %s", r.Method, r.URL.String()) t.Fatalf("unexpected request %s %s", r.Method, r.URL.String())
} }
}) }))
defer server.Close() defer server.Close()
result, err := DeployWikiWithOptions(&project.Project{}, DeployWikiOptions{ result, err := DeployWikiWithOptions(&project.Project{}, DeployWikiOptions{
@@ -952,7 +951,7 @@ func TestDeployWikiCreatesNodeBBTopicWithoutFallbackForDefaultThreeCharacterTitl
t.Fatalf("write source page: %v", err) t.Fatalf("write source page: %v", err)
} }
server := newFakeNodeBB(t, func(w http.ResponseWriter, r *http.Request) { server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if r.Method == http.MethodGet && r.URL.Path == "/api/v3/plugins/westgate-wiki/namespace/5/pages" { if r.Method == http.MethodGet && r.URL.Path == "/api/v3/plugins/westgate-wiki/namespace/5/pages" {
w.Header().Set("Content-Type", "application/json") w.Header().Set("Content-Type", "application/json")
_ = json.NewEncoder(w).Encode(map[string]any{"response": map[string]any{"pages": []any{}, "hasMore": false}}) _ = json.NewEncoder(w).Encode(map[string]any{"response": map[string]any{"pages": []any{}, "hasMore": false}})
@@ -984,7 +983,7 @@ func TestDeployWikiCreatesNodeBBTopicWithoutFallbackForDefaultThreeCharacterTitl
}, },
}, },
}) })
}) }))
defer server.Close() defer server.Close()
_, err := DeployWikiWithOptions(&project.Project{}, DeployWikiOptions{ _, err := DeployWikiWithOptions(&project.Project{}, DeployWikiOptions{
@@ -1011,7 +1010,7 @@ func TestDeployWikiCreatesNodeBBTopicWithFallbackForTitleShorterThanConfiguredMi
t.Fatalf("write source page: %v", err) t.Fatalf("write source page: %v", err)
} }
server := newFakeNodeBB(t, func(w http.ResponseWriter, r *http.Request) { server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if r.Method == http.MethodGet && r.URL.Path == "/api/v3/plugins/westgate-wiki/namespace/5/pages" { if r.Method == http.MethodGet && r.URL.Path == "/api/v3/plugins/westgate-wiki/namespace/5/pages" {
w.Header().Set("Content-Type", "application/json") w.Header().Set("Content-Type", "application/json")
_ = json.NewEncoder(w).Encode(map[string]any{"response": map[string]any{"pages": []any{}, "hasMore": false}}) _ = json.NewEncoder(w).Encode(map[string]any{"response": map[string]any{"pages": []any{}, "hasMore": false}})
@@ -1039,7 +1038,7 @@ func TestDeployWikiCreatesNodeBBTopicWithFallbackForTitleShorterThanConfiguredMi
}, },
}, },
}) })
}) }))
defer server.Close() defer server.Close()
_, err := DeployWikiWithOptions(&project.Project{}, DeployWikiOptions{ _, err := DeployWikiWithOptions(&project.Project{}, DeployWikiOptions{
@@ -1081,7 +1080,7 @@ func TestDeployWikiRenamesExistingPrefixedTopicWhenTitleIsLongEnough(t *testing.
} }
renameCalls := 0 renameCalls := 0
server := newFakeNodeBB(t, func(w http.ResponseWriter, r *http.Request) { server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
switch { switch {
case r.Method == http.MethodGet && r.URL.Path == "/api/v3/topics/7": case r.Method == http.MethodGet && r.URL.Path == "/api/v3/topics/7":
w.Header().Set("Content-Type", "application/json") w.Header().Set("Content-Type", "application/json")
@@ -1110,7 +1109,7 @@ func TestDeployWikiRenamesExistingPrefixedTopicWhenTitleIsLongEnough(t *testing.
default: default:
t.Fatalf("unexpected request %s %s", r.Method, r.URL.String()) t.Fatalf("unexpected request %s %s", r.Method, r.URL.String())
} }
}) }))
defer server.Close() defer server.Close()
result, err := DeployWikiWithOptions(&project.Project{}, DeployWikiOptions{ result, err := DeployWikiWithOptions(&project.Project{}, DeployWikiOptions{
@@ -1167,7 +1166,7 @@ func TestDeployWikiRenamesManagedTopicWhenGeneratedTitleChanges(t *testing.T) {
} }
renameCalls := 0 renameCalls := 0
server := newFakeNodeBB(t, func(w http.ResponseWriter, r *http.Request) { server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
switch { switch {
case r.Method == http.MethodGet && r.URL.Path == "/api/v3/topics/7": case r.Method == http.MethodGet && r.URL.Path == "/api/v3/topics/7":
w.Header().Set("Content-Type", "application/json") w.Header().Set("Content-Type", "application/json")
@@ -1196,7 +1195,7 @@ func TestDeployWikiRenamesManagedTopicWhenGeneratedTitleChanges(t *testing.T) {
default: default:
t.Fatalf("unexpected request %s %s", r.Method, r.URL.String()) t.Fatalf("unexpected request %s %s", r.Method, r.URL.String())
} }
}) }))
defer server.Close() defer server.Close()
result, err := DeployWikiWithOptions(&project.Project{}, DeployWikiOptions{ result, err := DeployWikiWithOptions(&project.Project{}, DeployWikiOptions{
@@ -1259,9 +1258,9 @@ func TestDeployWikiDoesNotRenameHeadinglessPageToPageIDFallback(t *testing.T) {
t.Fatalf("write deploy manifest: %v", err) t.Fatalf("write deploy manifest: %v", err)
} }
server := newFakeNodeBB(t, func(w http.ResponseWriter, r *http.Request) { server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
t.Fatalf("headingless unchanged page must not call NodeBB, got %s %s", r.Method, r.URL.String()) t.Fatalf("headingless unchanged page must not call NodeBB, got %s %s", r.Method, r.URL.String())
}) }))
defer server.Close() defer server.Close()
result, err := DeployWikiWithOptions(&project.Project{}, DeployWikiOptions{ result, err := DeployWikiWithOptions(&project.Project{}, DeployWikiOptions{
@@ -1321,7 +1320,7 @@ func TestDeployWikiRenamesBrokenHeadinglessFallbackTitleBackToPageIndexTitle(t *
} }
renameCalls := 0 renameCalls := 0
server := newFakeNodeBB(t, func(w http.ResponseWriter, r *http.Request) { server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
switch { switch {
case r.Method == http.MethodGet && r.URL.Path == "/api/v3/topics/7": case r.Method == http.MethodGet && r.URL.Path == "/api/v3/topics/7":
w.Header().Set("Content-Type", "application/json") w.Header().Set("Content-Type", "application/json")
@@ -1346,7 +1345,7 @@ func TestDeployWikiRenamesBrokenHeadinglessFallbackTitleBackToPageIndexTitle(t *
default: default:
t.Fatalf("unexpected request %s %s", r.Method, r.URL.String()) t.Fatalf("unexpected request %s %s", r.Method, r.URL.String())
} }
}) }))
defer server.Close() defer server.Close()
result, err := DeployWikiWithOptions(&project.Project{}, DeployWikiOptions{ result, err := DeployWikiWithOptions(&project.Project{}, DeployWikiOptions{
@@ -1397,7 +1396,7 @@ func TestDeployWikiAdoptsExistingNodeBBPageWhenManifestIsMissingWithoutCreate(t
createCalls := 0 createCalls := 0
var updated string var updated string
server := newFakeNodeBB(t, func(w http.ResponseWriter, r *http.Request) { server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
switch { switch {
case r.Method == http.MethodGet && r.URL.Path == "/api/v3/plugins/westgate-wiki/namespace/9/pages": case r.Method == http.MethodGet && r.URL.Path == "/api/v3/plugins/westgate-wiki/namespace/9/pages":
w.Header().Set("Content-Type", "application/json") w.Header().Set("Content-Type", "application/json")
@@ -1443,7 +1442,7 @@ func TestDeployWikiAdoptsExistingNodeBBPageWhenManifestIsMissingWithoutCreate(t
default: default:
t.Fatalf("unexpected request %s %s", r.Method, r.URL.String()) t.Fatalf("unexpected request %s %s", r.Method, r.URL.String())
} }
}) }))
defer server.Close() defer server.Close()
manifestPath := filepath.Join(root, "deploy-manifest.json") manifestPath := filepath.Join(root, "deploy-manifest.json")
@@ -1510,7 +1509,7 @@ func TestDeployWikiMergesHTMLManagedAndManualRegions(t *testing.T) {
} }
var updated string var updated string
server := newFakeNodeBB(t, func(w http.ResponseWriter, r *http.Request) { server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
switch { switch {
case r.Method == http.MethodGet && r.URL.Path == "/api/v3/posts/42": case r.Method == http.MethodGet && r.URL.Path == "/api/v3/posts/42":
w.Header().Set("Content-Type", "application/json") w.Header().Set("Content-Type", "application/json")
@@ -1534,7 +1533,7 @@ func TestDeployWikiMergesHTMLManagedAndManualRegions(t *testing.T) {
default: default:
t.Fatalf("unexpected request %s %s", r.Method, r.URL.Path) t.Fatalf("unexpected request %s %s", r.Method, r.URL.Path)
} }
}) }))
defer server.Close() defer server.Close()
result, err := DeployWikiWithOptions(&project.Project{}, DeployWikiOptions{ result, err := DeployWikiWithOptions(&project.Project{}, DeployWikiOptions{
@@ -1590,7 +1589,7 @@ func TestDeployWikiUpdateAcquiresWestgateWikiEditLock(t *testing.T) {
lockAcquired := false lockAcquired := false
var updateToken string var updateToken string
server := newFakeNodeBB(t, func(w http.ResponseWriter, r *http.Request) { server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
switch { switch {
case r.Method == http.MethodGet && r.URL.Path == "/api/v3/posts/42": case r.Method == http.MethodGet && r.URL.Path == "/api/v3/posts/42":
w.Header().Set("Content-Type", "application/json") w.Header().Set("Content-Type", "application/json")
@@ -1622,7 +1621,7 @@ func TestDeployWikiUpdateAcquiresWestgateWikiEditLock(t *testing.T) {
default: default:
t.Fatalf("unexpected request %s %s", r.Method, r.URL.Path) t.Fatalf("unexpected request %s %s", r.Method, r.URL.Path)
} }
}) }))
defer server.Close() defer server.Close()
result, err := DeployWikiWithOptions(&project.Project{}, DeployWikiOptions{ result, err := DeployWikiWithOptions(&project.Project{}, DeployWikiOptions{
@@ -1669,7 +1668,7 @@ func TestDeployWikiCreateCollisionAdoptsExistingNodeBBPage(t *testing.T) {
createCalls := 0 createCalls := 0
var updated string var updated string
server := newFakeNodeBB(t, func(w http.ResponseWriter, r *http.Request) { server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
switch { switch {
case r.Method == http.MethodGet && r.URL.Path == "/api/v3/plugins/westgate-wiki/namespace/9/pages" && r.URL.Query().Get("q") == "": case r.Method == http.MethodGet && r.URL.Path == "/api/v3/plugins/westgate-wiki/namespace/9/pages" && r.URL.Query().Get("q") == "":
w.Header().Set("Content-Type", "application/json") w.Header().Set("Content-Type", "application/json")
@@ -1720,7 +1719,7 @@ func TestDeployWikiCreateCollisionAdoptsExistingNodeBBPage(t *testing.T) {
default: default:
t.Fatalf("unexpected request %s %s", r.Method, r.URL.String()) t.Fatalf("unexpected request %s %s", r.Method, r.URL.String())
} }
}) }))
defer server.Close() defer server.Close()
manifestPath := filepath.Join(root, "deploy-manifest.json") manifestPath := filepath.Join(root, "deploy-manifest.json")
@@ -1772,7 +1771,7 @@ func TestDeployWikiCreateCollisionSearchesCanonicalTitleSegment(t *testing.T) {
createCalls := 0 createCalls := 0
var updated string var updated string
server := newFakeNodeBB(t, func(w http.ResponseWriter, r *http.Request) { server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
switch { switch {
case r.Method == http.MethodGet && r.URL.Path == "/api/v3/plugins/westgate-wiki/namespace/58/pages" && r.URL.Query().Get("q") == "": case r.Method == http.MethodGet && r.URL.Path == "/api/v3/plugins/westgate-wiki/namespace/58/pages" && r.URL.Query().Get("q") == "":
w.Header().Set("Content-Type", "application/json") w.Header().Set("Content-Type", "application/json")
@@ -1829,7 +1828,7 @@ func TestDeployWikiCreateCollisionSearchesCanonicalTitleSegment(t *testing.T) {
default: default:
t.Fatalf("unexpected request %s %s", r.Method, r.URL.String()) t.Fatalf("unexpected request %s %s", r.Method, r.URL.String())
} }
}) }))
defer server.Close() defer server.Close()
manifestPath := filepath.Join(root, "deploy-manifest.json") manifestPath := filepath.Join(root, "deploy-manifest.json")
@@ -1869,7 +1868,7 @@ func TestDeployWikiReportsAndArchivesStalePages(t *testing.T) {
} }
var archived string var archived string
server := newFakeNodeBB(t, func(w http.ResponseWriter, r *http.Request) { server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if r.Method == http.MethodPut && r.URL.Path == "/api/v3/plugins/westgate-wiki/edit-lock" { if r.Method == http.MethodPut && r.URL.Path == "/api/v3/plugins/westgate-wiki/edit-lock" {
respondWikiEditLock(t, w, r, 7, "archive-lock") respondWikiEditLock(t, w, r, 7, "archive-lock")
return return
@@ -1890,7 +1889,7 @@ func TestDeployWikiReportsAndArchivesStalePages(t *testing.T) {
archived = req.Content archived = req.Content
w.Header().Set("Content-Type", "application/json") w.Header().Set("Content-Type", "application/json")
_ = json.NewEncoder(w).Encode(map[string]any{"response": map[string]any{"pid": 42, "tid": 7}}) _ = json.NewEncoder(w).Encode(map[string]any{"response": map[string]any{"pid": 42, "tid": 7}})
}) }))
defer server.Close() defer server.Close()
report, err := DeployWikiWithOptions(&project.Project{}, DeployWikiOptions{ report, err := DeployWikiWithOptions(&project.Project{}, DeployWikiOptions{
@@ -1946,9 +1945,9 @@ func TestDeployWikiDryRunPlansTrackedStalePagePurge(t *testing.T) {
}); err != nil { }); err != nil {
t.Fatalf("write deploy manifest: %v", err) t.Fatalf("write deploy manifest: %v", err)
} }
server := newFakeNodeBB(t, func(w http.ResponseWriter, r *http.Request) { server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
t.Fatalf("dry-run purge must not call NodeBB, got %s %s", r.Method, r.URL.Path) t.Fatalf("dry-run purge must not call NodeBB, got %s %s", r.Method, r.URL.Path)
}) }))
defer server.Close() defer server.Close()
result, err := DeployWikiWithOptions(&project.Project{}, DeployWikiOptions{ result, err := DeployWikiWithOptions(&project.Project{}, DeployWikiOptions{
@@ -1985,9 +1984,9 @@ func TestDeployWikiPurgeRefusesStaleManifestEntryWithoutTopicID(t *testing.T) {
}); err != nil { }); err != nil {
t.Fatalf("write deploy manifest: %v", err) t.Fatalf("write deploy manifest: %v", err)
} }
server := newFakeNodeBB(t, func(w http.ResponseWriter, r *http.Request) { server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
t.Fatalf("missing-topic-id purge must not call NodeBB, got %s %s", r.Method, r.URL.Path) t.Fatalf("missing-topic-id purge must not call NodeBB, got %s %s", r.Method, r.URL.Path)
}) }))
defer server.Close() defer server.Close()
_, err := DeployWikiWithOptions(&project.Project{}, DeployWikiOptions{ _, err := DeployWikiWithOptions(&project.Project{}, DeployWikiOptions{
@@ -2018,12 +2017,15 @@ func TestDeployWikiPurgesTrackedStaleGeneratedTopic(t *testing.T) {
}); err != nil { }); err != nil {
t.Fatalf("write deploy manifest: %v", err) t.Fatalf("write deploy manifest: %v", err)
} }
var calls []string purgeCalls := 0
server := newFakeNodeBB(t, func(w http.ResponseWriter, r *http.Request) { server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if !answerWikiPagePurge(t, w, r, &calls) { if r.Method != http.MethodDelete || r.URL.Path != "/api/v3/topics/7" {
t.Fatalf("unexpected request %s %s", r.Method, r.URL.Path) t.Fatalf("unexpected request %s %s", r.Method, r.URL.Path)
} }
}) purgeCalls++
w.Header().Set("Content-Type", "application/json")
_ = json.NewEncoder(w).Encode(map[string]any{"status": map[string]any{"code": "ok"}})
}))
defer server.Close() defer server.Close()
result, err := DeployWikiWithOptions(&project.Project{}, DeployWikiOptions{ result, err := DeployWikiWithOptions(&project.Project{}, DeployWikiOptions{
@@ -2039,8 +2041,8 @@ func TestDeployWikiPurgesTrackedStaleGeneratedTopic(t *testing.T) {
if result.Stale != 1 || result.Purged != 1 { if result.Stale != 1 || result.Purged != 1 {
t.Fatalf("expected one stale purge, got %#v", result) t.Fatalf("expected one stale purge, got %#v", result)
} }
if strings.Join(calls, ",") != "tombstone:7,hard-purge:7" { if purgeCalls != 1 {
t.Fatalf("expected the wiki plugin page actions to purge topic 7, got %#v", calls) t.Fatalf("expected one NodeBB topic purge call, got %d", purgeCalls)
} }
if _, ok := loadDeployManifest(manifestPath).Pages["skills:retired"]; ok { if _, ok := loadDeployManifest(manifestPath).Pages["skills:retired"]; ok {
t.Fatalf("expected purged stale manifest entry to be removed") t.Fatalf("expected purged stale manifest entry to be removed")
@@ -2073,17 +2075,18 @@ func TestDeployWikiPurgesTrackedStaleTopicsBeforeCreatingReplacementPages(t *tes
} }
var calls []string var calls []string
server := newFakeNodeBB(t, func(w http.ResponseWriter, r *http.Request) { server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if answerWikiPagePurge(t, w, r, &calls) {
return
}
switch { switch {
case r.Method == http.MethodGet && r.URL.Path == "/api/v3/plugins/westgate-wiki/namespace/3/pages": case r.Method == http.MethodGet && r.URL.Path == "/api/v3/plugins/westgate-wiki/namespace/3/pages":
w.Header().Set("Content-Type", "application/json") w.Header().Set("Content-Type", "application/json")
_ = json.NewEncoder(w).Encode(map[string]any{"response": map[string]any{"pages": []any{}, "hasMore": false}}) _ = json.NewEncoder(w).Encode(map[string]any{"response": map[string]any{"pages": []any{}, "hasMore": false}})
case r.Method == http.MethodDelete && r.URL.Path == "/api/v3/topics/7":
calls = append(calls, "purge")
w.Header().Set("Content-Type", "application/json")
_ = json.NewEncoder(w).Encode(map[string]any{"status": map[string]any{"code": "ok"}})
case r.Method == http.MethodPost && r.URL.Path == "/api/v3/topics": case r.Method == http.MethodPost && r.URL.Path == "/api/v3/topics":
calls = append(calls, "create") calls = append(calls, "create")
if strings.Join(calls, ",") != "tombstone:7,hard-purge:7,create" { if len(calls) != 2 || calls[0] != "purge" {
t.Fatalf("expected stale purge before create, got calls %#v", calls) t.Fatalf("expected stale purge before create, got calls %#v", calls)
} }
w.Header().Set("Content-Type", "application/json") w.Header().Set("Content-Type", "application/json")
@@ -2099,7 +2102,7 @@ func TestDeployWikiPurgesTrackedStaleTopicsBeforeCreatingReplacementPages(t *tes
default: default:
t.Fatalf("unexpected request %s %s", r.Method, r.URL.String()) t.Fatalf("unexpected request %s %s", r.Method, r.URL.String())
} }
}) }))
defer server.Close() defer server.Close()
result, err := DeployWikiWithOptions(&project.Project{}, DeployWikiOptions{ result, err := DeployWikiWithOptions(&project.Project{}, DeployWikiOptions{
@@ -2117,7 +2120,7 @@ func TestDeployWikiPurgesTrackedStaleTopicsBeforeCreatingReplacementPages(t *tes
if result.Created != 1 || result.Purged != 1 { if result.Created != 1 || result.Purged != 1 {
t.Fatalf("expected one create and one purge, got %#v", result) t.Fatalf("expected one create and one purge, got %#v", result)
} }
if strings.Join(calls, ",") != "tombstone:7,hard-purge:7,create" { if strings.Join(calls, ",") != "purge,create" {
t.Fatalf("expected purge before create, got %#v", calls) t.Fatalf("expected purge before create, got %#v", calls)
} }
} }
@@ -2147,12 +2150,8 @@ func TestDeployWikiResetManagedNamespacesPurgesRemotePagesBeforeCreatingFreshMan
t.Fatalf("write deploy manifest: %v", err) t.Fatalf("write deploy manifest: %v", err)
} }
const resetCalls = "tombstone:7,hard-purge:7,tombstone:8,hard-purge:8"
var calls []string var calls []string
server := newFakeNodeBB(t, func(w http.ResponseWriter, r *http.Request) { server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if answerWikiPagePurge(t, w, r, &calls) {
return
}
switch { switch {
case r.Method == http.MethodGet && r.URL.Path == "/api/v3/plugins/westgate-wiki/namespace/3/pages": case r.Method == http.MethodGet && r.URL.Path == "/api/v3/plugins/westgate-wiki/namespace/3/pages":
w.Header().Set("Content-Type", "application/json") w.Header().Set("Content-Type", "application/json")
@@ -2165,9 +2164,17 @@ func TestDeployWikiResetManagedNamespacesPurgesRemotePagesBeforeCreatingFreshMan
"hasMore": false, "hasMore": false,
}, },
}) })
case r.Method == http.MethodDelete && r.URL.Path == "/api/v3/topics/7":
calls = append(calls, "purge:7")
w.Header().Set("Content-Type", "application/json")
_ = json.NewEncoder(w).Encode(map[string]any{"status": map[string]any{"code": "ok"}})
case r.Method == http.MethodDelete && r.URL.Path == "/api/v3/topics/8":
calls = append(calls, "purge:8")
w.Header().Set("Content-Type", "application/json")
_ = json.NewEncoder(w).Encode(map[string]any{"status": map[string]any{"code": "ok"}})
case r.Method == http.MethodPost && r.URL.Path == "/api/v3/topics": case r.Method == http.MethodPost && r.URL.Path == "/api/v3/topics":
calls = append(calls, "create") calls = append(calls, "create")
if strings.Join(calls, ",") != resetCalls+",create" { if strings.Join(calls, ",") != "purge:7,purge:8,create" {
t.Fatalf("expected namespace reset purges before create, got %#v", calls) t.Fatalf("expected namespace reset purges before create, got %#v", calls)
} }
w.Header().Set("Content-Type", "application/json") w.Header().Set("Content-Type", "application/json")
@@ -2183,7 +2190,7 @@ func TestDeployWikiResetManagedNamespacesPurgesRemotePagesBeforeCreatingFreshMan
default: default:
t.Fatalf("unexpected request %s %s", r.Method, r.URL.String()) t.Fatalf("unexpected request %s %s", r.Method, r.URL.String())
} }
}) }))
defer server.Close() defer server.Close()
result, err := DeployWikiWithOptions(&project.Project{}, DeployWikiOptions{ result, err := DeployWikiWithOptions(&project.Project{}, DeployWikiOptions{
@@ -2198,21 +2205,10 @@ func TestDeployWikiResetManagedNamespacesPurgesRemotePagesBeforeCreatingFreshMan
if err != nil { if err != nil {
t.Fatalf("DeployWikiWithOptions reset-managed-namespaces failed: %v", err) t.Fatalf("DeployWikiWithOptions reset-managed-namespaces failed: %v", err)
} }
if result.Created != 1 || result.Purged != 2 { if result.Created != 1 || result.Purged != 2 || result.Stale != 0 {
t.Fatalf("expected one create and two namespace purges, got %#v", result) t.Fatalf("expected one create and two namespace purges, got %#v", result)
} }
// Callers word their destructive-policy warning around the stale count, so if strings.Join(calls, ",") != "purge:7,purge:8,create" {
// reset deletions have to show up there rather than on a separate path the
// preview never mentions.
if result.Stale != 2 || result.ResetPurged != 2 {
t.Fatalf("expected reset deletions counted as stale, got %#v", result)
}
// tid 7 is the manifest's own page; tid 8 is a topic the deployer never
// wrote, and re-seeding cannot bring it back.
if result.ResetUnrecognized != 1 {
t.Fatalf("expected one unrecognized reset deletion, got %#v", result)
}
if strings.Join(calls, ",") != resetCalls+",create" {
t.Fatalf("expected reset purges before create, got %#v", calls) t.Fatalf("expected reset purges before create, got %#v", calls)
} }
manifest := loadDeployManifest(manifestPath) manifest := loadDeployManifest(manifestPath)
@@ -2240,9 +2236,9 @@ func TestDeployWikiResetManagedNamespacesRequiresCreateForLocalPages(t *testing.
t.Fatalf("write source page: %v", err) t.Fatalf("write source page: %v", err)
} }
server := newFakeNodeBB(t, func(w http.ResponseWriter, r *http.Request) { server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
t.Fatalf("reset without --create must fail before remote calls, got %s %s", r.Method, r.URL.String()) t.Fatalf("reset without --create must fail before remote calls, got %s %s", r.Method, r.URL.String())
}) }))
defer server.Close() defer server.Close()
_, err := DeployWikiWithOptions(&project.Project{}, DeployWikiOptions{ _, err := DeployWikiWithOptions(&project.Project{}, DeployWikiOptions{
@@ -2274,21 +2270,21 @@ func TestDeployWikiPurgeTreatsAlreadyMissingTrackedTopicAsSuccess(t *testing.T)
t.Fatalf("write deploy manifest: %v", err) t.Fatalf("write deploy manifest: %v", err)
} }
seen := map[string]int{} seen := map[string]int{}
server := newFakeNodeBB(t, func(w http.ResponseWriter, r *http.Request) { server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
call, ok := wikiPagePurgeCall(t, r) if r.Method != http.MethodDelete {
if !ok {
t.Fatalf("unexpected request %s %s", r.Method, r.URL.Path) t.Fatalf("unexpected request %s %s", r.Method, r.URL.Path)
} }
seen[call]++ seen[r.URL.Path]++
w.Header().Set("Content-Type", "application/json") w.Header().Set("Content-Type", "application/json")
// Topic 7 is already gone, so the plugin cannot find the page to switch r.URL.Path {
// tombstone; topic 8 still exists and purges normally. case "/api/v3/topics/7":
if call == "tombstone:7" {
http.Error(w, `{"status":{"message":"topic not found"}}`, http.StatusNotFound) http.Error(w, `{"status":{"message":"topic not found"}}`, http.StatusNotFound)
return case "/api/v3/topics/8":
_ = json.NewEncoder(w).Encode(map[string]any{"status": map[string]any{"code": "ok"}})
default:
t.Fatalf("unexpected purge path %s", r.URL.Path)
} }
_ = json.NewEncoder(w).Encode(map[string]any{"status": map[string]any{"code": "ok"}}) }))
})
defer server.Close() defer server.Close()
result, err := DeployWikiWithOptions(&project.Project{}, DeployWikiOptions{ result, err := DeployWikiWithOptions(&project.Project{}, DeployWikiOptions{
@@ -2304,11 +2300,8 @@ func TestDeployWikiPurgeTreatsAlreadyMissingTrackedTopicAsSuccess(t *testing.T)
if result.Stale != 2 || result.Purged != 2 { if result.Stale != 2 || result.Purged != 2 {
t.Fatalf("expected two stale purges, got %#v", result) t.Fatalf("expected two stale purges, got %#v", result)
} }
if seen["tombstone:7"] != 1 || seen["hard-purge:7"] != 0 { if seen["/api/v3/topics/7"] != 1 || seen["/api/v3/topics/8"] != 1 {
t.Fatalf("expected the already-missing topic to stop after tombstone, got %#v", seen) t.Fatalf("expected one purge call for each tracked topic, got %#v", seen)
}
if seen["tombstone:8"] != 1 || seen["hard-purge:8"] != 1 {
t.Fatalf("expected the surviving topic to be tombstoned then purged, got %#v", seen)
} }
manifest := loadDeployManifest(manifestPath) manifest := loadDeployManifest(manifestPath)
if _, ok := manifest.Pages["skills:retired"]; ok { if _, ok := manifest.Pages["skills:retired"]; ok {
@@ -2346,9 +2339,9 @@ func TestDeployWikiPurgeDoesNotTargetCurrentGeneratedPages(t *testing.T) {
}); err != nil { }); err != nil {
t.Fatalf("write deploy manifest: %v", err) t.Fatalf("write deploy manifest: %v", err)
} }
server := newFakeNodeBB(t, func(w http.ResponseWriter, r *http.Request) { server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
t.Fatalf("current generated page must not call NodeBB during matching-hash deploy, got %s %s", r.Method, r.URL.Path) t.Fatalf("current generated page must not call NodeBB during matching-hash deploy, got %s %s", r.Method, r.URL.Path)
}) }))
defer server.Close() defer server.Close()
result, err := DeployWikiWithOptions(&project.Project{}, DeployWikiOptions{ result, err := DeployWikiWithOptions(&project.Project{}, DeployWikiOptions{
@@ -2392,187 +2385,3 @@ func respondWikiEditLock(t *testing.T, w http.ResponseWriter, r *http.Request, e
w.Header().Set("Content-Type", "application/json") w.Header().Set("Content-Type", "application/json")
_ = json.NewEncoder(w).Encode(map[string]any{"response": map[string]any{"status": "ok", "tid": expectedTID, "token": token}}) _ = json.NewEncoder(w).Encode(map[string]any{"response": map[string]any{"status": "ok", "tid": expectedTID, "token": token}})
} }
// newFakeNodeBB stands up a fake NodeBB that behaves like one running
// nodebb-plugin-westgate-wiki: native topic mutation on wiki categories is
// refused, because wiki revision history is plugin-owned. Every deploy test
// goes through here, so a deployer that reaches for the core topic API fails in
// CI the way it fails in production rather than passing against a fake that is
// more permissive than the real thing.
func newFakeNodeBB(t *testing.T, handler http.HandlerFunc) *httptest.Server {
t.Helper()
return httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if strings.HasPrefix(r.URL.Path, "/api/v3/topics/") && (r.Method == http.MethodDelete || r.Method == http.MethodPut) {
http.Error(
w,
`{"status":{"code":"bad-request","message":"Use the wiki page actions to delete, restore, or purge wiki pages."}}`,
http.StatusBadRequest,
)
return
}
handler(w, r)
}))
}
// wikiPagePurgeCall recognizes the plugin page actions a purge goes through and
// reports them as "<action>:<tid>" so tests can assert the order.
func wikiPagePurgeCall(t *testing.T, r *http.Request) (string, bool) {
t.Helper()
var action string
switch {
case r.Method == http.MethodPut && r.URL.Path == "/api/v3/plugins/westgate-wiki/page/tombstone":
action = "tombstone"
case r.Method == http.MethodDelete && r.URL.Path == "/api/v3/plugins/westgate-wiki/page/hard-purge":
action = "hard-purge"
default:
return "", false
}
var req struct {
TID int `json:"tid"`
}
if err := json.NewDecoder(r.Body).Decode(&req); err != nil {
t.Fatalf("decode wiki %s request: %v", action, err)
}
if req.TID == 0 {
t.Fatalf("expected wiki %s request to carry a topic id", action)
}
return fmt.Sprintf("%s:%d", action, req.TID), true
}
// answerWikiPagePurge records and acknowledges a plugin purge action, which is
// what most fakes want to do with one.
func answerWikiPagePurge(t *testing.T, w http.ResponseWriter, r *http.Request, calls *[]string) bool {
t.Helper()
call, ok := wikiPagePurgeCall(t, r)
if !ok {
return false
}
*calls = append(*calls, call)
w.Header().Set("Content-Type", "application/json")
_ = json.NewEncoder(w).Encode(map[string]any{"status": map[string]any{"code": "ok"}})
return true
}
// resetPurgeFixture builds a source tree, a deploy manifest, and the reset
// options the namespace-reset tests share.
func resetPurgeFixture(t *testing.T, namespaces map[string]int, tidsByCID map[int][]int) (DeployWikiOptions, func(w http.ResponseWriter, r *http.Request) bool) {
t.Helper()
root := t.TempDir()
sourceDir := filepath.Join(root, "pages")
if err := os.MkdirAll(filepath.Join(sourceDir, "skills"), 0755); err != nil {
t.Fatalf("create source dir: %v", err)
}
manifestPath := filepath.Join(root, "deploy-manifest.json")
if err := saveDeployManifest(manifestPath, wikiDeployManifest{Version: "nodebb-v1", Pages: map[string]wikiDeployManifestPage{}}); err != nil {
t.Fatalf("write deploy manifest: %v", err)
}
listPages := func(w http.ResponseWriter, r *http.Request) bool {
if r.Method != http.MethodGet {
return false
}
var cid int
if _, err := fmt.Sscanf(r.URL.Path, "/api/v3/plugins/westgate-wiki/namespace/%d/pages", &cid); err != nil {
return false
}
tids, ok := tidsByCID[cid]
if !ok {
t.Fatalf("unexpected namespace listing for category %d", cid)
}
pages := make([]map[string]any, 0, len(tids))
for _, tid := range tids {
pages = append(pages, map[string]any{"tid": tid, "title": fmt.Sprintf("Page %d", tid)})
}
w.Header().Set("Content-Type", "application/json")
_ = json.NewEncoder(w).Encode(map[string]any{"response": map[string]any{"pages": pages, "hasMore": false}})
return true
}
return DeployWikiOptions{
SourceDir: sourceDir,
ManifestPath: manifestPath,
Token: "nodebb-token",
CategoryIDs: namespaces,
AllowCreates: true,
ResetManagedNamespaces: true,
}, listPages
}
func TestDeployWikiResetSkipsTopicsNodeBBRefusesToDelete(t *testing.T) {
opts, listPages := resetPurgeFixture(t, map[string]int{"skills": 3}, map[int][]int{3: {5, 6}})
var calls []string
server := newFakeNodeBB(t, func(w http.ResponseWriter, r *http.Request) {
if listPages(w, r) {
return
}
call, ok := wikiPagePurgeCall(t, r)
if !ok {
t.Fatalf("unexpected request %s %s", r.Method, r.URL.Path)
}
calls = append(calls, call)
// Topic 5 stands in for the wiki home page, which the plugin excludes
// from tombstone, restore, and purge alike.
if call == "tombstone:5" {
http.Error(w, `{"status":{"message":"[[error:no-privileges]]"}}`, http.StatusForbidden)
return
}
w.Header().Set("Content-Type", "application/json")
_ = json.NewEncoder(w).Encode(map[string]any{"status": map[string]any{"code": "ok"}})
})
defer server.Close()
opts.Endpoint = server.URL
result, err := DeployWikiWithOptions(&project.Project{}, opts, nil)
if err != nil {
t.Fatalf("DeployWikiWithOptions reset with an undeletable topic failed: %v", err)
}
if result.ResetSkipped != 1 || result.ResetPurged != 1 {
t.Fatalf("expected one skipped and one completed reset deletion, got %#v", result)
}
if result.Purged != 1 || result.Stale != 1 {
t.Fatalf("expected the skipped topic to be dropped from the counts, got %#v", result)
}
if result.ResetUnrecognized != 1 {
t.Fatalf("expected only the purged unrecognized topic to be counted, got %#v", result)
}
if strings.Join(calls, ",") != "tombstone:5,tombstone:6,hard-purge:6" {
t.Fatalf("expected the reset to continue past the refused topic, got %#v", calls)
}
}
// NodeBB answers 403 both for the wiki home page and for a token without purge
// privileges. A category where nothing could be deleted at all is the second
// case, and reporting it as a completed reset would leave the manifest claiming
// a fresh start over pages that are all still there.
func TestDeployWikiResetFailsWhenACategoryRefusesEveryDeletion(t *testing.T) {
opts, listPages := resetPurgeFixture(t,
map[string]int{"skills": 3, "feats": 4},
map[int][]int{3: {5, 6}, 4: {7, 8}},
)
server := newFakeNodeBB(t, func(w http.ResponseWriter, r *http.Request) {
if listPages(w, r) {
return
}
call, ok := wikiPagePurgeCall(t, r)
if !ok {
t.Fatalf("unexpected request %s %s", r.Method, r.URL.Path)
}
// Category 4 holds topics 7 and 8, and the token can purge neither.
if strings.HasSuffix(call, ":7") || strings.HasSuffix(call, ":8") {
http.Error(w, `{"status":{"message":"[[error:no-privileges]]"}}`, http.StatusForbidden)
return
}
w.Header().Set("Content-Type", "application/json")
_ = json.NewEncoder(w).Encode(map[string]any{"status": map[string]any{"code": "ok"}})
})
defer server.Close()
opts.Endpoint = server.URL
_, err := DeployWikiWithOptions(&project.Project{}, opts, nil)
if err == nil || !strings.Contains(err.Error(), "wiki purge privileges") {
t.Fatalf("expected a category that deleted nothing to fail loudly, got %v", err)
}
if !strings.Contains(err.Error(), "4") {
t.Fatalf("expected the failure to name the refusing category, got %v", err)
}
}