Compare commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
87feaf96b7 | ||
|
|
a02e06d644 | ||
|
|
f395d86db5 | ||
|
|
9747384838 | ||
|
|
b47a8a7afd | ||
|
|
f23009ed50 | ||
|
|
3cac6e9484 | ||
|
|
3f78197f0a | ||
|
|
7cc53aeb68 | ||
|
|
682f920114 | ||
|
|
2f860ca9e4 | ||
|
|
1c2acc5530 | ||
|
|
fa32dd411f | ||
|
|
7437653f14 |
@@ -94,9 +94,9 @@ Tests must survive harmless changes to constants, defaults, wording, ordering, f
|
|||||||
|
|
||||||
Issues live in Gitea at git.westgate.pw (`ShadowsOverWestgate/sow-tools`), managed with the `tea` CLI. Issues follow ownership — file work in the repo that owns it, not the one you happen to be standing in. See `docs/agents/issue-tracker.md`.
|
Issues live in Gitea at git.westgate.pw (`ShadowsOverWestgate/sow-tools`), managed with the `tea` CLI. Issues follow ownership — file work in the repo that owns it, not the one you happen to be standing in. See `docs/agents/issue-tracker.md`.
|
||||||
|
|
||||||
### Triage labels
|
### Labels
|
||||||
|
|
||||||
Default label vocabulary (`needs-triage`, `needs-info`, `ready-for-agent`, `ready-for-human`, `wontfix`). See `docs/agents/triage-labels.md`.
|
Every issue and PR gets exactly one org-wide `Kind/*` label at creation (`Kind/Bug`, `Kind/Feature`, `Kind/Enhancement`, `Kind/Documentation`, `Kind/Testing`, `Kind/Security`); `Priority/*`, `Status/*`, `Reviewed/*` and `Compat/Breaking` are optional. `tea issues create -L "Kind/Bug"`. See `docs/agents/triage-labels.md`.
|
||||||
|
|
||||||
### Domain docs
|
### Domain docs
|
||||||
|
|
||||||
|
|||||||
@@ -19,7 +19,7 @@ Crucible is how the artifact repos turn source into artifacts.
|
|||||||
| `crucible-depot` | `crucible depot` | content-addressed depot blob verify/move |
|
| `crucible-depot` | `crucible depot` | content-addressed depot blob verify/move |
|
||||||
| `crucible-hak` | `crucible hak` | ERF/HAK pack/unpack + hak manifests |
|
| `crucible-hak` | `crucible hak` | ERF/HAK pack/unpack + hak manifests |
|
||||||
| `crucible-module` | `crucible module` | build/extract/validate/compare the `.mod` |
|
| `crucible-module` | `crucible module` | build/extract/validate/compare the `.mod` |
|
||||||
| `crucible-nwsync` | `crucible nwsync` | NWSync blob emit + manifest assemble |
|
| `crucible-nwsync` | `crucible nwsync` | NWSync blob emit + manifest assemble + verify |
|
||||||
| `crucible-topdata` | `crucible topdata` | compile 2da/tlk topdata + packages |
|
| `crucible-topdata` | `crucible topdata` | compile 2da/tlk topdata + packages |
|
||||||
| `crucible-wiki` | `crucible wiki` | render + deploy mechanical wiki pages |
|
| `crucible-wiki` | `crucible wiki` | render + deploy mechanical wiki pages |
|
||||||
|
|
||||||
|
|||||||
@@ -57,8 +57,32 @@ preference — move it.
|
|||||||
so any non-interactive shell (every agent) hangs forever without
|
so any non-interactive shell (every agent) hangs forever without
|
||||||
`</dev/null`. Same trap on `tea issues create --description` and
|
`</dev/null`. Same trap on `tea issues create --description` and
|
||||||
`tea pr create`.
|
`tea pr create`.
|
||||||
- **Apply / remove labels**: `tea api --method PATCH` on the issue, or
|
- **Apply / remove labels**: `tea issues edit <number> --add-labels "Kind/Bug"`
|
||||||
`tea api repos/ShadowsOverWestgate/sow-tools/issues/<number>/labels` endpoints.
|
(and `--remove-labels`). This handles org-level labels (`Kind/*`,
|
||||||
|
`Priority/*`, `Reviewed/*`, `Status/*`) from tea 0.15 onwards. On 0.14 it did
|
||||||
|
not: name resolution searched only this repo's own label set, so an org label
|
||||||
|
matched nothing and the command exited 0, printed the issue, and changed
|
||||||
|
nothing. Upstream fixed it in v0.15 (`modules/task/labels.go` also queries
|
||||||
|
`ListOrgLabels`). Note `tea labels` lists repo labels only and will not show
|
||||||
|
you the org set — `tea api orgs/ShadowsOverWestgate/labels` does.
|
||||||
|
- **`tea api` needs a token in the login; SSH auth is not enough.** It sends
|
||||||
|
only the login's `token:` field and does not sign requests with your SSH key,
|
||||||
|
so an SSH-key-only login gets `{"message":"token is required"}` on every call
|
||||||
|
that needs auth. Reads against these public repos still succeed anonymously,
|
||||||
|
which hides the gap until the first write. Add a token to the login in
|
||||||
|
`~/.config/tea/config.yml` (Settings > Applications; `write:issue` covers
|
||||||
|
labels, comments and dependencies) and `tea api` works for reads and writes
|
||||||
|
alike. The `tea issues` / `tea pr` subcommands authenticate either way, so
|
||||||
|
they keep working with no token at all — that asymmetry is what makes this
|
||||||
|
confusing to diagnose.
|
||||||
|
- **Verify every label change by re-reading it.** A label command exiting 0 is
|
||||||
|
not evidence it applied — that is exactly how the 0.14 silent no-op above hid
|
||||||
|
for so long, and assuming otherwise has already cost one investigation
|
||||||
|
several wrong turns. Read the resulting set back with
|
||||||
|
`tea api repos/ShadowsOverWestgate/sow-tools/issues/<number>` and check its
|
||||||
|
`labels` field, or `tea issues ls -o json`. The read-back reflects the write
|
||||||
|
immediately; if it comes back empty, the write genuinely failed. Do not
|
||||||
|
explain an empty read-back away as replication lag.
|
||||||
- **Close**: `tea issues close <number>`
|
- **Close**: `tea issues close <number>`
|
||||||
|
|
||||||
`tea` infers the repo from the git remote when run inside the clone.
|
`tea` infers the repo from the git remote when run inside the clone.
|
||||||
|
|||||||
+105
-11
@@ -1,15 +1,109 @@
|
|||||||
# Triage Labels
|
# Issue and PR labels
|
||||||
|
|
||||||
The skills speak in terms of five canonical triage roles. This file maps those roles to the actual label strings used in this repo's issue tracker (Gitea — see `issue-tracker.md` for how to apply labels with `tea`).
|
Labels are **org-wide**. They are defined once, for the whole
|
||||||
|
`ShadowsOverWestgate` org, at
|
||||||
|
<https://git.westgate.pw/org/ShadowsOverWestgate/settings/labels>, and every
|
||||||
|
repo in the org can use them. Never create a per-repo copy of a label that
|
||||||
|
already exists at org level.
|
||||||
|
|
||||||
| Label in mattpocock/skills | Label in our tracker | Meaning |
|
## The rule
|
||||||
| -------------------------- | -------------------- | ---------------------------------------- |
|
|
||||||
| `needs-triage` | `needs-triage` | Maintainer needs to evaluate this issue |
|
|
||||||
| `needs-info` | `needs-info` | Waiting on reporter for more information |
|
|
||||||
| `ready-for-agent` | `ready-for-agent` | Fully specified, ready for an AFK agent |
|
|
||||||
| `ready-for-human` | `ready-for-human` | Requires human implementation |
|
|
||||||
| `wontfix` | `wontfix` | Will not be actioned |
|
|
||||||
|
|
||||||
When a skill mentions a role (e.g. "apply the AFK-ready triage label"), use the corresponding label string from this table.
|
**Every issue and every pull request gets exactly one `Kind/*` label, at the
|
||||||
|
moment it is created.** Not later, not "when someone triages it". If you open
|
||||||
|
it, you label it.
|
||||||
|
|
||||||
Edit the right-hand column to match whatever vocabulary you actually use.
|
An issue with no `Kind/*` label is untriaged. That is the only meaning of
|
||||||
|
"untriaged" here — there is no `needs-triage` label.
|
||||||
|
|
||||||
|
The other groups are optional, and each one is *exclusive*: an issue can carry
|
||||||
|
at most one `Priority/*`, one `Status/*`, and one `Reviewed/*`. Gitea enforces
|
||||||
|
this.
|
||||||
|
|
||||||
|
```sh
|
||||||
|
# always from inside the owning repo's clone
|
||||||
|
tea issues create --title "..." --description "..." --labels "Kind/Bug" </dev/null
|
||||||
|
tea pr create --title "..." --description "..." --labels "Kind/Feature" </dev/null
|
||||||
|
|
||||||
|
# add a label to something that already exists (needs tea >= 0.15 for org labels)
|
||||||
|
tea issues edit <n> --add-labels "Kind/Bug,Priority/High"
|
||||||
|
|
||||||
|
# read the result back — exit 0 is not evidence the label applied
|
||||||
|
tea api "repos/ShadowsOverWestgate/<repo>/issues/<n>"
|
||||||
|
```
|
||||||
|
|
||||||
|
`tea api` sends only the login's `token:` and never your SSH key, so on an
|
||||||
|
SSH-only login every authenticated call fails with
|
||||||
|
`{"message":"token is required"}` while anonymous reads still work. Add a token
|
||||||
|
to `~/.config/tea/config.yml` if you want the raw endpoints. `tea issues edit`
|
||||||
|
needs no token.
|
||||||
|
|
||||||
|
## Kind — what this is (pick exactly one)
|
||||||
|
|
||||||
|
| Label | Use it when |
|
||||||
|
| -------------------- | --------------------------------------------------------------- |
|
||||||
|
| `Kind/Bug` | Something that used to work, or is documented to work, does not |
|
||||||
|
| `Kind/Feature` | New functionality that does not exist yet |
|
||||||
|
| `Kind/Enhancement` | Existing functionality gets better, faster, or nicer |
|
||||||
|
| `Kind/Documentation` | Docs, ADRs, runbooks, agent guides |
|
||||||
|
| `Kind/Testing` | Tests, CI checks, contract scripts |
|
||||||
|
| `Kind/Security` | Secrets, auth, permissions, hardening, a vulnerability |
|
||||||
|
|
||||||
|
Bug vs Enhancement, when it is unclear: if the current behaviour is wrong, it
|
||||||
|
is a bug. If the current behaviour is right but weak, it is an enhancement.
|
||||||
|
|
||||||
|
## Priority — how urgent (optional, at most one)
|
||||||
|
|
||||||
|
`Priority/Critical`, `Priority/High`, `Priority/Medium`, `Priority/Low`.
|
||||||
|
|
||||||
|
Leave it off if you do not know. A wrong priority is worse than none.
|
||||||
|
|
||||||
|
## Status — why it is not moving (optional, at most one)
|
||||||
|
|
||||||
|
| Label | Meaning |
|
||||||
|
| ----------------------- | -------------------------------------------- |
|
||||||
|
| `Status/Blocked` | Waiting on another issue, PR, or decision |
|
||||||
|
| `Status/Need More Info` | Waiting on the reporter or on a human answer |
|
||||||
|
| `Status/Abandoned` | Work started and stopped; nobody is on it |
|
||||||
|
|
||||||
|
## Reviewed — the verdict (optional, at most one)
|
||||||
|
|
||||||
|
`Reviewed/Confirmed`, `Reviewed/Duplicate`, `Reviewed/Invalid`,
|
||||||
|
`Reviewed/Won't Fix`. Apply one of these when closing without a fix, so the
|
||||||
|
reason survives.
|
||||||
|
|
||||||
|
## Compat
|
||||||
|
|
||||||
|
`Compat/Breaking` — add it on top of the `Kind/*` label when the change breaks
|
||||||
|
something that already works for a player, an operator, or another repo.
|
||||||
|
|
||||||
|
## Workflow labels (repo-level, not org-level)
|
||||||
|
|
||||||
|
These live in the repo, not the org, and are orthogonal to the groups above:
|
||||||
|
|
||||||
|
- `ready-for-agent` — the spec is complete; an AFK agent may pick this up.
|
||||||
|
No label means it needs a human.
|
||||||
|
- `wayfinder:map`, `wayfinder:task`, `wayfinder:research`,
|
||||||
|
`wayfinder:prototype`, `wayfinder:grilling` — set by `/wayfinder`. Leave
|
||||||
|
them alone unless you are running a wayfinder operation.
|
||||||
|
- `sow-nodebb` also has `package/*` labels naming the plugin or theme a ticket
|
||||||
|
touches.
|
||||||
|
|
||||||
|
## When a skill names a label we do not have
|
||||||
|
|
||||||
|
Skills written elsewhere (mattpocock/skills and friends) use a different
|
||||||
|
vocabulary. Translate it:
|
||||||
|
|
||||||
|
| Skill says | Do this here |
|
||||||
|
| ----------------- | ------------------------------------------------- |
|
||||||
|
| `needs-triage` | Nothing — no `Kind/*` label already means this |
|
||||||
|
| `needs-info` | `Status/Need More Info` |
|
||||||
|
| `ready-for-agent` | `ready-for-agent` |
|
||||||
|
| `ready-for-human` | Nothing — absence of `ready-for-agent` means this |
|
||||||
|
| `wontfix` | `Reviewed/Won't Fix` |
|
||||||
|
|
||||||
|
## Drift check
|
||||||
|
|
||||||
|
`ops/checks/check-labels.sh` in `sow-platform` runs nightly. It compares the
|
||||||
|
live org labels to `ops/policy/labels.yml` and lists every open issue and PR in
|
||||||
|
the org that does not have exactly one `Kind/*` label. That check is the
|
||||||
|
enforcement; this file is the rule.
|
||||||
|
|||||||
+91
-1
@@ -36,6 +36,7 @@ aliases.
|
|||||||
| `depot` | `pull` | Incremental verified pull of every referenced blob. |
|
| `depot` | `pull` | Incremental verified pull of every referenced blob. |
|
||||||
| `nwsync` | `emit` | Explode one artifact into NWSync blobs plus its own NSYM manifest. |
|
| `nwsync` | `emit` | Explode one artifact into NWSync blobs plus its own NSYM manifest. |
|
||||||
| `nwsync` | `assemble` | Merge per-artifact NSYM manifests into one merged manifest. |
|
| `nwsync` | `assemble` | Merge per-artifact NSYM manifests into one merged manifest. |
|
||||||
|
| `nwsync` | `verify` | Decompress and hash a published manifest's blobs through the pull zone. |
|
||||||
|
|
||||||
`depot status` and `depot get` pick their backend either with `--out DIR`, a
|
`depot status` and `depot get` pick their backend either with `--out DIR`, a
|
||||||
depot tree on disk, or with `--target bunny|cdn`, a remote backend. The two
|
depot tree on disk, or with `--target bunny|cdn`, a remote backend. The two
|
||||||
@@ -48,10 +49,20 @@ beside the artifact itself with the extension replaced, so `emit` and
|
|||||||
`assemble` agree on where it is without being told.
|
`assemble` agree on where it is without being told.
|
||||||
|
|
||||||
```
|
```
|
||||||
nwsync emit [--as NAME] [--out DIR] <artifact-key> <file>
|
nwsync emit [--as NAME] [--out DIR] [--jobs N] [--verify] <artifact-key> <file>
|
||||||
nwsync assemble --group-id N [--tlk-key KEY] [--out DIR] <artifact-key>...
|
nwsync assemble --group-id N [--tlk-key KEY] [--out DIR] <artifact-key>...
|
||||||
|
nwsync verify [--sample N] [--base URL] [--jobs N] <manifest-sha1>
|
||||||
```
|
```
|
||||||
|
|
||||||
|
`emit` is latency-bound, not CPU-bound: every blob costs an existence probe
|
||||||
|
plus an upload, and a measured backfill spent 26 seconds of CPU across 9.5
|
||||||
|
minutes of wall clock. `--jobs N` (default 16) sets how many resources are in
|
||||||
|
flight at once. The manifest is byte-identical at any value — the number of
|
||||||
|
workers is never observable in the output. Peak memory is `N` times the
|
||||||
|
per-resource limit of 15 MB plus its compressed copy, so raising `N` far past
|
||||||
|
the default costs real memory for little gain: the transport keeps 16 idle
|
||||||
|
connections per host, and past that a worker pays a fresh TLS handshake.
|
||||||
|
|
||||||
Both verbs upload by default; nothing bulky is ever written to the runner's
|
Both verbs upload by default; nothing bulky is ever written to the runner's
|
||||||
disk. `--out DIR` writes a local repository tree instead, which is the
|
disk. `--out DIR` writes a local repository tree instead, which is the
|
||||||
conformance path against upstream `nwn_nwsync_write`. The zone comes from
|
conformance path against upstream `nwn_nwsync_write`. The zone comes from
|
||||||
@@ -64,12 +75,91 @@ resolves it. `--tlk-key` has its own slot because the TLK shadows nothing.
|
|||||||
`--group-id` is per channel — 1 is current, 2 is testing, and 0 leaves the field
|
`--group-id` is per channel — 1 is current, 2 is testing, and 0 leaves the field
|
||||||
out of the sidecar.
|
out of the sidecar.
|
||||||
|
|
||||||
|
`nwsync verify` is the only check on a published blob upstream of a player's
|
||||||
|
client. It reads the **pull zone**, not the storage API, and needs no
|
||||||
|
credential: what matters is the bytes a client is served, edge behaviour
|
||||||
|
included. Every blob is decompressed and hashed, and the zstd frame is asserted
|
||||||
|
to declare its content size. Neither half is optional — a `Content-Length` check
|
||||||
|
passes a byte-correct-looking object whose contents are short, and a round-trip
|
||||||
|
check alone passes a frame the game client cannot decode but Go's decoder can.
|
||||||
|
Failures are reported per blob as missing, malformed framing, size mismatch or
|
||||||
|
hash mismatch, and the exit code is 1.
|
||||||
|
|
||||||
|
A full sweep of the live manifest is roughly 69,000 blobs and 15 GB, so
|
||||||
|
`--sample N` exists to make verifying routine; the default is a full sweep.
|
||||||
|
`--base URL` (or `NWSYNC_PULL_BASE`) overrides the public host.
|
||||||
|
|
||||||
|
`emit --verify` applies the same check where `emit` would otherwise skip. `emit`
|
||||||
|
normally reads a blob's presence as proof of its contents, decided by a 1-byte
|
||||||
|
range GET, so an object written truncated — or written by an emitter since found
|
||||||
|
broken — is skipped by every later run forever and no backfill repairs it. With
|
||||||
|
`--verify` the stored copy is read back, unwrapped, hashed against its own name,
|
||||||
|
and replaced when it does not match. It costs a full GET per existing blob, so
|
||||||
|
it is a repair pass, not the default.
|
||||||
|
|
||||||
|
**After a repair, `verify` is what tells you which keys to purge.** A repair is
|
||||||
|
the one thing that makes a key serve different bytes than it did before, and the
|
||||||
|
edge caches these objects for 30 days precisely because that normally cannot
|
||||||
|
happen. The two commands look at different copies on purpose: `emit --verify`
|
||||||
|
repairs the **origin**, `verify` reads the **edge**. So a `verify` run straight
|
||||||
|
after a repair is not a verdict — it is a survey, and every blob it still calls
|
||||||
|
bad is one the edge is serving stale. Purge exactly those, then re-run it; only
|
||||||
|
that second run is the verdict.
|
||||||
|
|
||||||
|
Purging the keys `verify` names beats purging the zone, because the edge only
|
||||||
|
ever cached what somebody actually fetched: the 2026-08-01 repair rewrote 2,603
|
||||||
|
blobs at the origin and left 8 stale at the edge. The purge belongs in the
|
||||||
|
repair procedure rather than in `emit`, which reports how many blobs it wrote
|
||||||
|
and never which ones — so it could not target one even with a CDN credential,
|
||||||
|
which it deliberately does not hold (#89; the procedure itself is in
|
||||||
|
sow-platform's NWSync runbook).
|
||||||
|
|
||||||
`emit` uploads blobs first and the index last, so the presence of an index is
|
`emit` uploads blobs first and the index last, so the presence of an index is
|
||||||
the publication marker: an artifact whose emit died halfway leaves real blobs in
|
the publication marker: an artifact whose emit died halfway leaves real blobs in
|
||||||
the zone and no index. Blob names are content hashes, so re-running skips
|
the zone and no index. Blob names are content hashes, so re-running skips
|
||||||
whatever already landed, and `assemble` fails closed on an artifact with no
|
whatever already landed, and `assemble` fails closed on an artifact with no
|
||||||
index rather than publishing a manifest that is missing a hak.
|
index rather than publishing a manifest that is missing a hak.
|
||||||
|
|
||||||
|
### What an emitted tree looks like
|
||||||
|
|
||||||
|
`--out DIR` produces the same tree `emit` would upload, which makes it the way
|
||||||
|
to check a zone by hand without touching one:
|
||||||
|
|
||||||
|
```
|
||||||
|
<artifact-sha>.nsym binary index
|
||||||
|
<artifact-sha>.nsym.json the same index, readable
|
||||||
|
data/sha1/a7/4a/a74aa84a... one blob per resource, two-level fanout
|
||||||
|
```
|
||||||
|
|
||||||
|
A blob's name is the SHA-1 of the resource's **original** bytes, but the file on
|
||||||
|
disk is not those bytes: each blob is wrapped in NWCompressedBuffer framing, a
|
||||||
|
24-byte `NSYC` header followed by a zstd frame. Hashing the file directly will
|
||||||
|
not match its name, which is the obvious
|
||||||
|
first thing to try and the obvious first thing to be confused by. Strip the
|
||||||
|
header first:
|
||||||
|
|
||||||
|
```
|
||||||
|
tail -c +25 <blob> | zstd -dc | sha1sum # == the blob's filename
|
||||||
|
```
|
||||||
|
|
||||||
|
The header carries the uncompressed length as a little-endian `uint32` at offset
|
||||||
|
12, so the decompressed size is checkable without decompressing. Compression is
|
||||||
|
worth roughly a 4:1
|
||||||
|
saving on hak content: a 250 MB hak emitted 2296 blobs totalling 59 MB on disk
|
||||||
|
against 249 MB of resources, as recorded in the sidecar's `on_disk_bytes` and
|
||||||
|
`total_bytes`.
|
||||||
|
|
||||||
|
The zstd frame always declares its `Frame_Content_Size`. The game client sizes
|
||||||
|
its output buffer from that field and cannot decode a frame without one, but the
|
||||||
|
Go encoder omits it below 256 bytes, so `emit` re-headers those frames into the
|
||||||
|
shape reference libzstd emits: `Single_Segment_flag` set, `Window_Descriptor`
|
||||||
|
dropped, and a one-byte content size in its place. `zstd -l <frame>` must print a
|
||||||
|
decompressed size; a blank column there is the fault, and it is invisible to any
|
||||||
|
check that only decompresses, because both `zstd -dc` and Go's decoder stream
|
||||||
|
such a frame happily. This is what the sidecar's `emitter_version` counts:
|
||||||
|
version 1 omitted the field and no client could sync past such a blob, version 2
|
||||||
|
declares it. `assemble` refuses to merge indexes that disagree.
|
||||||
|
|
||||||
## Hidden compatibility aliases
|
## Hidden compatibility aliases
|
||||||
|
|
||||||
Existing scripts may continue using these names indefinitely. They are accepted
|
Existing scripts may continue using these names indefinitely. They are accepted
|
||||||
|
|||||||
+21
-12
@@ -385,7 +385,7 @@ func refreshBuildModuleManifest(ctx context, p *project.Project, progress func(s
|
|||||||
}
|
}
|
||||||
|
|
||||||
progress("Refreshing hak list from the latest published sow-assets manifest...")
|
progress("Refreshing hak list from the latest published sow-assets manifest...")
|
||||||
if err := runProjectScript(ctx, p, []string{"scripts", "fetch-hak-manifest"}, manifestPath); err != nil {
|
if err := runProjectScript(ctx, p, []string{"scripts", "fetch-upstream-manifests"}, manifestPath); err != nil {
|
||||||
return "", "", err
|
return "", "", err
|
||||||
}
|
}
|
||||||
if _, err := pipeline.ApplyHAKManifest(p, manifestPath); err != nil {
|
if _, err := pipeline.ApplyHAKManifest(p, manifestPath); err != nil {
|
||||||
@@ -927,19 +927,28 @@ func (c *topdataConsole) emitWikiBuildResult(outputDir string, pageCount int, st
|
|||||||
fmt.Fprintf(c.stdout, "wiki status: %s\n", status)
|
fmt.Fprintf(c.stdout, "wiki status: %s\n", status)
|
||||||
}
|
}
|
||||||
|
|
||||||
func (c *topdataConsole) emitWikiDeployResult(localPages, created, updated, skipped, stale, archived, purged, drifted int, manifest string) {
|
func (c *topdataConsole) emitWikiDeployResult(result topdata.DeployResult) {
|
||||||
spin.linebreak()
|
spin.linebreak()
|
||||||
fmt.Fprintln(c.stdout, "Deploy Wiki ----------")
|
fmt.Fprintln(c.stdout, "Deploy Wiki ----------")
|
||||||
fmt.Fprintf(c.stdout, "project: %s\n", c.projectName)
|
fmt.Fprintf(c.stdout, "project: %s\n", c.projectName)
|
||||||
fmt.Fprintf(c.stdout, "local pages: %d\n", localPages)
|
fmt.Fprintf(c.stdout, "local pages: %d\n", result.LocalPages)
|
||||||
fmt.Fprintf(c.stdout, "created: %d\n", created)
|
fmt.Fprintf(c.stdout, "created: %d\n", result.Created)
|
||||||
fmt.Fprintf(c.stdout, "updated: %d\n", updated)
|
fmt.Fprintf(c.stdout, "updated: %d\n", result.Updated)
|
||||||
fmt.Fprintf(c.stdout, "skipped: %d\n", skipped)
|
fmt.Fprintf(c.stdout, "skipped: %d\n", result.Skipped)
|
||||||
fmt.Fprintf(c.stdout, "stale: %d\n", stale)
|
fmt.Fprintf(c.stdout, "stale: %d\n", result.Stale)
|
||||||
fmt.Fprintf(c.stdout, "archived: %d\n", archived)
|
fmt.Fprintf(c.stdout, "archived: %d\n", result.Archived)
|
||||||
fmt.Fprintf(c.stdout, "purged: %d\n", purged)
|
fmt.Fprintf(c.stdout, "purged: %d\n", result.Purged)
|
||||||
fmt.Fprintf(c.stdout, "drifted: %d\n", drifted)
|
fmt.Fprintf(c.stdout, "drifted: %d\n", result.Drifted)
|
||||||
fmt.Fprintf(c.stdout, "manifest: %s\n", c.relPath(manifest))
|
if result.ResetPurged > 0 || result.ResetSkipped > 0 {
|
||||||
|
fmt.Fprintf(c.stdout, "namespace reset: %d (unrecognized: %d)\n", result.ResetPurged, result.ResetUnrecognized)
|
||||||
|
}
|
||||||
|
if result.ResetUnrecognized > 0 {
|
||||||
|
fmt.Fprintln(c.stdout, " unrecognized pages were not written by this deployer; recreating them is not possible")
|
||||||
|
}
|
||||||
|
if result.ResetSkipped > 0 {
|
||||||
|
fmt.Fprintf(c.stdout, " reset skipped: %d (NodeBB refused deletion, e.g. the wiki home page)\n", result.ResetSkipped)
|
||||||
|
}
|
||||||
|
fmt.Fprintf(c.stdout, "manifest: %s\n", c.relPath(result.Manifest))
|
||||||
}
|
}
|
||||||
|
|
||||||
func (c *topdataConsole) relPath(path string) string {
|
func (c *topdataConsole) relPath(path string) string {
|
||||||
@@ -1579,7 +1588,7 @@ func runDeployWiki(ctx context) error {
|
|||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
|
|
||||||
console.emitWikiDeployResult(result.LocalPages, result.Created, result.Updated, result.Skipped, result.Stale, result.Archived, result.Purged, result.Drifted, result.Manifest)
|
console.emitWikiDeployResult(result)
|
||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@@ -10,6 +10,7 @@ import (
|
|||||||
"time"
|
"time"
|
||||||
|
|
||||||
"git.westgate.pw/ShadowsOverWestgate/sow-tools/internal/pipeline"
|
"git.westgate.pw/ShadowsOverWestgate/sow-tools/internal/pipeline"
|
||||||
|
"git.westgate.pw/ShadowsOverWestgate/sow-tools/internal/topdata"
|
||||||
)
|
)
|
||||||
|
|
||||||
func TestParseBuildHAKArgsContentAddressedRoot(t *testing.T) {
|
func TestParseBuildHAKArgsContentAddressedRoot(t *testing.T) {
|
||||||
@@ -218,7 +219,16 @@ func TestTopdataConsoleDebugProgressAndRelativePaths(t *testing.T) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
console.progress("NodeBB wiki plan: create 1, update 2, skip 3, stale 4, archive 5, purge 6, drift 0")
|
console.progress("NodeBB wiki plan: create 1, update 2, skip 3, stale 4, archive 5, purge 6, drift 0")
|
||||||
console.emitWikiDeployResult(10, 1, 2, 3, 4, 5, 6, 0, "/workspace/project/build/wiki/deploy-manifest.json")
|
console.emitWikiDeployResult(topdata.DeployResult{
|
||||||
|
LocalPages: 10,
|
||||||
|
Created: 1,
|
||||||
|
Updated: 2,
|
||||||
|
Skipped: 3,
|
||||||
|
Stale: 4,
|
||||||
|
Archived: 5,
|
||||||
|
Purged: 6,
|
||||||
|
Manifest: "/workspace/project/build/wiki/deploy-manifest.json",
|
||||||
|
})
|
||||||
|
|
||||||
output := stdout.String()
|
output := stdout.String()
|
||||||
if !strings.Contains(output, "NodeBB wiki plan") {
|
if !strings.Contains(output, "NodeBB wiki plan") {
|
||||||
@@ -229,6 +239,53 @@ func TestTopdataConsoleDebugProgressAndRelativePaths(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func TestTopdataConsoleReportsManagedNamespaceReset(t *testing.T) {
|
||||||
|
var stdout bytes.Buffer
|
||||||
|
console := &topdataConsole{
|
||||||
|
stdout: &stdout,
|
||||||
|
projectRoot: "/workspace/project",
|
||||||
|
projectName: "Test Module",
|
||||||
|
commandName: "deploy-wiki",
|
||||||
|
commandLabel: "Deploy Wiki",
|
||||||
|
}
|
||||||
|
|
||||||
|
console.emitWikiDeployResult(topdata.DeployResult{
|
||||||
|
LocalPages: 1200,
|
||||||
|
Created: 1200,
|
||||||
|
Stale: 1213,
|
||||||
|
Purged: 1213,
|
||||||
|
ResetPurged: 1213,
|
||||||
|
ResetUnrecognized: 13,
|
||||||
|
ResetSkipped: 1,
|
||||||
|
Manifest: "/workspace/project/build/wiki/deploy-manifest.json",
|
||||||
|
})
|
||||||
|
|
||||||
|
// The counts are what an operator reads to decide whether a destructive run
|
||||||
|
// is safe, so each has to reach the output on a line that names what it
|
||||||
|
// counts; the wording around them is free to change.
|
||||||
|
output := stdout.String()
|
||||||
|
var staleLine, resetLine, skippedLine string
|
||||||
|
for _, line := range strings.Split(output, "\n") {
|
||||||
|
switch {
|
||||||
|
case strings.Contains(line, "stale"):
|
||||||
|
staleLine = line
|
||||||
|
case strings.Contains(line, "skipped"):
|
||||||
|
skippedLine = line
|
||||||
|
case strings.Contains(line, "reset"):
|
||||||
|
resetLine = line
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if !strings.Contains(staleLine, "1213") {
|
||||||
|
t.Fatalf("expected reset deletions counted as stale, got %q", output)
|
||||||
|
}
|
||||||
|
if !strings.Contains(resetLine, "1213") || !strings.Contains(resetLine, "13") {
|
||||||
|
t.Fatalf("expected the reset line to carry both the total and the unrecognized count, got %q", output)
|
||||||
|
}
|
||||||
|
if !strings.Contains(skippedLine, "1") {
|
||||||
|
t.Fatalf("expected refused reset deletions to be reported, got %q", output)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
func TestParseDeployWikiHelpListsPurgeStalePolicy(t *testing.T) {
|
func TestParseDeployWikiHelpListsPurgeStalePolicy(t *testing.T) {
|
||||||
_, err := parseDeployWikiArgs("deploy-wiki", []string{"--help"})
|
_, err := parseDeployWikiArgs("deploy-wiki", []string{"--help"})
|
||||||
if err == nil || !strings.Contains(err.Error(), "--stale-policy <report|archive|purge>") {
|
if err == nil || !strings.Contains(err.Error(), "--stale-policy <report|archive|purge>") {
|
||||||
|
|||||||
@@ -108,10 +108,11 @@ var Registry = []Builder{
|
|||||||
{
|
{
|
||||||
Name: "nwsync",
|
Name: "nwsync",
|
||||||
Bin: "crucible-nwsync",
|
Bin: "crucible-nwsync",
|
||||||
Summary: "publish NWSync blobs and manifests (emit/assemble)",
|
Summary: "publish NWSync blobs and manifests (emit/assemble/verify)",
|
||||||
Commands: []Command{
|
Commands: []Command{
|
||||||
{Name: "emit", Summary: "explode one artifact into blobs plus its own NSYM manifest", Usage: "crucible nwsync emit <artifact> --out DIR"},
|
{Name: "emit", Summary: "explode one artifact into blobs plus its own NSYM manifest", Usage: "crucible nwsync emit <artifact> --out DIR"},
|
||||||
{Name: "assemble", Summary: "merge per-artifact NSYM manifests into one", Usage: "crucible nwsync assemble --order NAMES --entries DIR --out DIR [--group-id N]"},
|
{Name: "assemble", Summary: "merge per-artifact NSYM manifests into one", Usage: "crucible nwsync assemble --order NAMES --entries DIR --out DIR [--group-id N]"},
|
||||||
|
{Name: "verify", Summary: "read a published manifest's blobs back through the pull zone and hash them", Usage: "crucible nwsync verify <manifest-sha1> [--sample N]"},
|
||||||
},
|
},
|
||||||
Wired: true,
|
Wired: true,
|
||||||
},
|
},
|
||||||
@@ -288,7 +289,8 @@ var Registry = []Builder{
|
|||||||
"--dry-run report changes without writing",
|
"--dry-run report changes without writing",
|
||||||
"--create allow missing pages to be created",
|
"--create allow missing pages to be created",
|
||||||
"--force update unchanged pages",
|
"--force update unchanged pages",
|
||||||
"--reset-managed-namespaces reset managed namespace state",
|
"--reset-managed-namespaces delete every page in the managed categories, including",
|
||||||
|
" pages this deployer never wrote, then recreate from source",
|
||||||
},
|
},
|
||||||
Aliases: []CommandAlias{{Name: "deploy-wiki"}},
|
Aliases: []CommandAlias{{Name: "deploy-wiki"}},
|
||||||
},
|
},
|
||||||
|
|||||||
@@ -160,7 +160,7 @@ func TestCanonicalCommandSurface(t *testing.T) {
|
|||||||
"module": {"build", "extract", "validate", "compare", "manifest"},
|
"module": {"build", "extract", "validate", "compare", "manifest"},
|
||||||
"topdata": {"validate", "build", "package", "compare", "convert"},
|
"topdata": {"validate", "build", "package", "compare", "convert"},
|
||||||
"wiki": {"build", "deploy"},
|
"wiki": {"build", "deploy"},
|
||||||
"nwsync": {"emit", "assemble"},
|
"nwsync": {"emit", "assemble", "verify"},
|
||||||
}
|
}
|
||||||
for _, builder := range Registry {
|
for _, builder := range Registry {
|
||||||
got := builder.subcommands()
|
got := builder.subcommands()
|
||||||
|
|||||||
+92
-45
@@ -311,63 +311,110 @@ func Write(w io.Writer, archive Archive) error {
|
|||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// IndexEntry locates one resource inside an archive without holding its
|
||||||
|
// payload. Streaming callers read one payload at a time from these, so peak
|
||||||
|
// memory tracks the largest resource instead of the whole archive.
|
||||||
|
type IndexEntry struct {
|
||||||
|
Name string
|
||||||
|
Type uint16
|
||||||
|
Offset int64
|
||||||
|
Size int64
|
||||||
|
}
|
||||||
|
|
||||||
|
// Index is the header plus the resource table of an ERF: everything except the
|
||||||
|
// payloads.
|
||||||
|
type Index struct {
|
||||||
|
FileType string
|
||||||
|
Version string
|
||||||
|
Entries []IndexEntry
|
||||||
|
}
|
||||||
|
|
||||||
|
// ReadIndex parses the tables of an ERF of the given size, reading only the
|
||||||
|
// header, the key list and the resource list.
|
||||||
|
func ReadIndex(r io.ReaderAt, size int64) (Index, error) {
|
||||||
|
if size < headerSize {
|
||||||
|
return Index{}, fmt.Errorf("erf file too small: %d bytes", size)
|
||||||
|
}
|
||||||
|
|
||||||
|
var hdr header
|
||||||
|
if err := binary.Read(io.NewSectionReader(r, 0, headerSize), binary.LittleEndian, &hdr); err != nil {
|
||||||
|
return Index{}, fmt.Errorf("decode erf header: %w", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
if int64(hdr.KeyListOffset)+int64(hdr.EntryCount)*24 > size {
|
||||||
|
return Index{}, fmt.Errorf("erf key list exceeds file bounds")
|
||||||
|
}
|
||||||
|
keys := make([]keyEntry, hdr.EntryCount)
|
||||||
|
keyReader := io.NewSectionReader(r, int64(hdr.KeyListOffset), int64(hdr.EntryCount)*24)
|
||||||
|
if err := binary.Read(keyReader, binary.LittleEndian, &keys); err != nil {
|
||||||
|
return Index{}, fmt.Errorf("decode key list: %w", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
if int64(hdr.ResourceListOffset)+int64(hdr.EntryCount)*8 > size {
|
||||||
|
return Index{}, fmt.Errorf("erf resource list exceeds file bounds")
|
||||||
|
}
|
||||||
|
entries := make([]resourceEntry, hdr.EntryCount)
|
||||||
|
entryReader := io.NewSectionReader(r, int64(hdr.ResourceListOffset), int64(hdr.EntryCount)*8)
|
||||||
|
if err := binary.Read(entryReader, binary.LittleEndian, &entries); err != nil {
|
||||||
|
return Index{}, fmt.Errorf("decode resource list: %w", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
index := Index{
|
||||||
|
FileType: string(hdr.FileType[:]),
|
||||||
|
Version: string(hdr.Version[:]),
|
||||||
|
Entries: make([]IndexEntry, 0, hdr.EntryCount),
|
||||||
|
}
|
||||||
|
for position, key := range keys {
|
||||||
|
entry := entries[position]
|
||||||
|
if int64(entry.Offset)+int64(entry.Size) > size {
|
||||||
|
return Index{}, fmt.Errorf("resource %d exceeds file bounds", position)
|
||||||
|
}
|
||||||
|
index.Entries = append(index.Entries, IndexEntry{
|
||||||
|
Name: string(bytes.TrimRight(key.ResRef[:], "\x00")),
|
||||||
|
Type: key.ResourceType,
|
||||||
|
Offset: int64(entry.Offset),
|
||||||
|
Size: int64(entry.Size),
|
||||||
|
})
|
||||||
|
}
|
||||||
|
return index, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// ReadPayload returns one resource's bytes.
|
||||||
|
func ReadPayload(r io.ReaderAt, entry IndexEntry) ([]byte, error) {
|
||||||
|
payload := make([]byte, entry.Size)
|
||||||
|
if _, err := r.ReadAt(payload, entry.Offset); err != nil {
|
||||||
|
return nil, fmt.Errorf("read resource %q: %w", entry.Name, err)
|
||||||
|
}
|
||||||
|
return payload, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// Read materialises a whole archive. Payloads are subslices of the buffer the
|
||||||
|
// archive was read into, so nothing is copied twice: a caller must not mutate
|
||||||
|
// Data. Callers that only need one resource at a time should use ReadIndex
|
||||||
|
// instead, which never holds the archive at all.
|
||||||
func Read(r io.Reader) (Archive, error) {
|
func Read(r io.Reader) (Archive, error) {
|
||||||
data, err := io.ReadAll(r)
|
data, err := io.ReadAll(r)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return Archive{}, fmt.Errorf("read erf: %w", err)
|
return Archive{}, fmt.Errorf("read erf: %w", err)
|
||||||
}
|
}
|
||||||
if len(data) < headerSize {
|
index, err := ReadIndex(bytes.NewReader(data), int64(len(data)))
|
||||||
return Archive{}, fmt.Errorf("erf file too small: %d bytes", len(data))
|
if err != nil {
|
||||||
|
return Archive{}, err
|
||||||
}
|
}
|
||||||
|
|
||||||
var hdr header
|
resources := make([]Resource, 0, len(index.Entries))
|
||||||
if err := binary.Read(bytes.NewReader(data[:headerSize]), binary.LittleEndian, &hdr); err != nil {
|
for _, entry := range index.Entries {
|
||||||
return Archive{}, fmt.Errorf("decode erf header: %w", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
keyStart := int(hdr.KeyListOffset)
|
|
||||||
keyEnd := keyStart + int(hdr.EntryCount)*24
|
|
||||||
if keyEnd > len(data) {
|
|
||||||
return Archive{}, fmt.Errorf("erf key list exceeds file bounds")
|
|
||||||
}
|
|
||||||
keys := make([]keyEntry, hdr.EntryCount)
|
|
||||||
if err := binary.Read(bytes.NewReader(data[keyStart:keyEnd]), binary.LittleEndian, &keys); err != nil {
|
|
||||||
return Archive{}, fmt.Errorf("decode key list: %w", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
resourceStart := int(hdr.ResourceListOffset)
|
|
||||||
resourceEnd := resourceStart + int(hdr.EntryCount)*8
|
|
||||||
if resourceEnd > len(data) {
|
|
||||||
return Archive{}, fmt.Errorf("erf resource list exceeds file bounds")
|
|
||||||
}
|
|
||||||
entries := make([]resourceEntry, hdr.EntryCount)
|
|
||||||
if err := binary.Read(bytes.NewReader(data[resourceStart:resourceEnd]), binary.LittleEndian, &entries); err != nil {
|
|
||||||
return Archive{}, fmt.Errorf("decode resource list: %w", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
resources := make([]Resource, 0, hdr.EntryCount)
|
|
||||||
for index, key := range keys {
|
|
||||||
entry := entries[index]
|
|
||||||
start := int(entry.Offset)
|
|
||||||
end := start + int(entry.Size)
|
|
||||||
if end > len(data) {
|
|
||||||
return Archive{}, fmt.Errorf("resource %d exceeds file bounds", index)
|
|
||||||
}
|
|
||||||
|
|
||||||
resref := string(bytes.TrimRight(key.ResRef[:], "\x00"))
|
|
||||||
payload := make([]byte, entry.Size)
|
|
||||||
copy(payload, data[start:end])
|
|
||||||
resources = append(resources, Resource{
|
resources = append(resources, Resource{
|
||||||
Name: resref,
|
Name: entry.Name,
|
||||||
Type: key.ResourceType,
|
Type: entry.Type,
|
||||||
Data: payload,
|
Data: data[entry.Offset : entry.Offset+entry.Size],
|
||||||
Size: int64(entry.Size),
|
Size: entry.Size,
|
||||||
})
|
})
|
||||||
}
|
}
|
||||||
|
|
||||||
return Archive{
|
return Archive{
|
||||||
FileType: string(hdr.FileType[:]),
|
FileType: index.FileType,
|
||||||
Version: string(hdr.Version[:]),
|
Version: index.Version,
|
||||||
Resources: resources,
|
Resources: resources,
|
||||||
}, nil
|
}, nil
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -3,6 +3,7 @@ package nwsync
|
|||||||
import (
|
import (
|
||||||
"bytes"
|
"bytes"
|
||||||
"encoding/binary"
|
"encoding/binary"
|
||||||
|
"encoding/hex"
|
||||||
"fmt"
|
"fmt"
|
||||||
|
|
||||||
"github.com/klauspost/compress/zstd"
|
"github.com/klauspost/compress/zstd"
|
||||||
@@ -22,9 +23,26 @@ const (
|
|||||||
blobHeaderBytes = 24
|
blobHeaderBytes = 24
|
||||||
)
|
)
|
||||||
|
|
||||||
|
// EncodeAll/DecodeAll are single-threaded per call, so the default pool of one
|
||||||
|
// encoder per CPU only buys idle memory: each holds a window-sized history, so
|
||||||
|
// on a 24-core runner that is ~200 MB of live heap doing nothing. Concurrency 1
|
||||||
|
// produces byte-identical output.
|
||||||
var (
|
var (
|
||||||
blobEncoder, _ = zstd.NewWriter(nil)
|
blobEncoder, _ = zstd.NewWriter(nil, zstd.WithEncoderConcurrency(1))
|
||||||
blobDecoder, _ = zstd.NewReader(nil)
|
blobDecoder, _ = zstd.NewReader(nil, zstd.WithDecoderConcurrency(1))
|
||||||
|
)
|
||||||
|
|
||||||
|
// zstd frame header bits we care about. A frame starts with the magic, then a
|
||||||
|
// one-byte Frame_Header_Descriptor: bits 7-6 size the Frame_Content_Size field,
|
||||||
|
// bit 5 is Single_Segment_flag, bits 1-0 size the Dictionary_ID field.
|
||||||
|
const (
|
||||||
|
zstdFrameMagic = "\x28\xb5\x2f\xfd"
|
||||||
|
frameSingleSegment = 1 << 5
|
||||||
|
frameDictionaryMask = 0x03
|
||||||
|
// oneByteContentSizeCeiling is the size above which a Frame_Content_Size no
|
||||||
|
// longer fits in one byte. Below it the field's size flag is 0, which is
|
||||||
|
// what lets klauspost/compress leave the field out entirely.
|
||||||
|
oneByteContentSizeCeiling = 256
|
||||||
)
|
)
|
||||||
|
|
||||||
// compressBlob wraps data in NWCompressedBuffer framing.
|
// compressBlob wraps data in NWCompressedBuffer framing.
|
||||||
@@ -34,10 +52,95 @@ func compressBlob(data []byte) []byte {
|
|||||||
for _, field := range header {
|
for _, field := range header {
|
||||||
_ = binary.Write(&out, binary.LittleEndian, field)
|
_ = binary.Write(&out, binary.LittleEndian, field)
|
||||||
}
|
}
|
||||||
out.Write(blobEncoder.EncodeAll(data, nil))
|
frame := declareFrameContentSize(blobEncoder.EncodeAll(data, nil), len(data))
|
||||||
|
// Fail closed rather than publish a blob no client can decode. An encoder
|
||||||
|
// upgrade that finds a new way to omit the field would otherwise reproduce
|
||||||
|
// #86 in silence, and a blob is skipped by every later emit once written.
|
||||||
|
if !frameDeclaresContentSize(frame) {
|
||||||
|
panic(fmt.Sprintf("nwsync: refusing to emit a %d-byte blob whose zstd frame declares no content size (descriptor %#x)",
|
||||||
|
len(data), frame[4]))
|
||||||
|
}
|
||||||
|
out.Write(frame)
|
||||||
return out.Bytes()
|
return out.Bytes()
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// inspectBlob unwraps a stored blob the way the game client reads it, and is the
|
||||||
|
// only reader that should be trusted to judge a published blob.
|
||||||
|
//
|
||||||
|
// It asserts the frame property on top of the round trip. Go's decoder — like
|
||||||
|
// the zstd CLI — streams a frame that declares no content size, so a check that
|
||||||
|
// only decompresses and hashes is a *more* capable decoder than the client's: it
|
||||||
|
// certifies exactly the blobs the client rejects, which is how #86 reached
|
||||||
|
// production and survived an audit.
|
||||||
|
func inspectBlob(blob []byte) ([]byte, error) {
|
||||||
|
data, err := decompressBlob(blob)
|
||||||
|
if err != nil {
|
||||||
|
return nil, fmt.Errorf("malformed framing: %w", err)
|
||||||
|
}
|
||||||
|
if len(blob) > blobHeaderBytes && !frameDeclaresContentSize(blob[blobHeaderBytes:]) {
|
||||||
|
return nil, fmt.Errorf("malformed framing: the zstd frame declares no content size, which the game client cannot decode")
|
||||||
|
}
|
||||||
|
return data, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// blobMatchesName holds a stored blob to its own file name: a blob is named
|
||||||
|
// after the sha1 of its uncompressed bytes, so the name is a complete statement
|
||||||
|
// about the contents and nothing else is needed to check it.
|
||||||
|
func blobMatchesName(blob []byte, sha1Hex string) error {
|
||||||
|
data, err := inspectBlob(blob)
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
if got := hex.EncodeToString(sha1Sum(data)); got != sha1Hex {
|
||||||
|
return fmt.Errorf("blob %s holds the contents of %s", sha1Hex, got)
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// frameDeclaresContentSize reports whether a zstd frame states how many bytes it
|
||||||
|
// decompresses to. A frame with a zero-sized Frame_Content_Size field declares
|
||||||
|
// one only when Single_Segment_flag is set; otherwise the size is unknown.
|
||||||
|
func frameDeclaresContentSize(frame []byte) bool {
|
||||||
|
if len(frame) < 5 || string(frame[:4]) != zstdFrameMagic {
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
descriptor := frame[4]
|
||||||
|
return descriptor>>6 != 0 || descriptor&frameSingleSegment != 0
|
||||||
|
}
|
||||||
|
|
||||||
|
// declareFrameContentSize rewrites a frame that does not declare its
|
||||||
|
// Frame_Content_Size so that it does, and returns any other frame unchanged.
|
||||||
|
//
|
||||||
|
// klauspost/compress omits the field for inputs under 256 bytes, which the spec
|
||||||
|
// permits. Reference libzstd never does, so the NWN client — which sizes its
|
||||||
|
// output buffer from ZSTD_getFrameContentSize and has therefore never met a
|
||||||
|
// frame without one — rejects the blob outright with an empty "potential
|
||||||
|
// compression error" (#86). No encoder option changes this, so the frame is
|
||||||
|
// re-headered here.
|
||||||
|
//
|
||||||
|
// The result is the shape libzstd itself emits for the same input: setting
|
||||||
|
// Single_Segment_flag drops the Window_Descriptor byte, and the freed byte pays
|
||||||
|
// for a one-byte Frame_Content_Size. Window_Size then equals the content size,
|
||||||
|
// which is sound because the content is under 256 bytes and every match in it
|
||||||
|
// therefore falls inside that window. Same length in, same length out.
|
||||||
|
func declareFrameContentSize(frame []byte, size int) []byte {
|
||||||
|
if size <= 0 || size >= oneByteContentSizeCeiling || len(frame) < 6 || string(frame[:4]) != zstdFrameMagic {
|
||||||
|
return frame
|
||||||
|
}
|
||||||
|
descriptor := frame[4]
|
||||||
|
// Rewrite only the exact shape a small input produces: no declared size, no
|
||||||
|
// single segment, no dictionary. Anything else either declares a size
|
||||||
|
// already or is not a frame this reinterpretation is safe on.
|
||||||
|
if descriptor>>6 != 0 || descriptor&frameSingleSegment != 0 || descriptor&frameDictionaryMask != 0 {
|
||||||
|
return frame
|
||||||
|
}
|
||||||
|
reframed := make([]byte, len(frame))
|
||||||
|
copy(reframed, frame)
|
||||||
|
reframed[4] = descriptor | frameSingleSegment
|
||||||
|
reframed[5] = byte(size) // replaces Window_Descriptor
|
||||||
|
return reframed
|
||||||
|
}
|
||||||
|
|
||||||
// decompressBlob unwraps NWCompressedBuffer framing. It exists so a blob this
|
// decompressBlob unwraps NWCompressedBuffer framing. It exists so a blob this
|
||||||
// package wrote — or one upstream wrote — can be compared by its uncompressed
|
// package wrote — or one upstream wrote — can be compared by its uncompressed
|
||||||
// bytes, which is the only comparison that is meaningful across zstd
|
// bytes, which is the only comparison that is meaningful across zstd
|
||||||
|
|||||||
+142
-41
@@ -1,10 +1,10 @@
|
|||||||
package nwsync
|
package nwsync
|
||||||
|
|
||||||
import (
|
import (
|
||||||
"bytes"
|
|
||||||
"context"
|
"context"
|
||||||
"crypto/sha1"
|
"crypto/sha1"
|
||||||
"fmt"
|
"fmt"
|
||||||
|
"io"
|
||||||
"os"
|
"os"
|
||||||
"path"
|
"path"
|
||||||
"path/filepath"
|
"path/filepath"
|
||||||
@@ -12,6 +12,7 @@ import (
|
|||||||
"sort"
|
"sort"
|
||||||
"strconv"
|
"strconv"
|
||||||
"strings"
|
"strings"
|
||||||
|
"sync"
|
||||||
"time"
|
"time"
|
||||||
|
|
||||||
"git.westgate.pw/ShadowsOverWestgate/sow-tools/internal/buildinfo"
|
"git.westgate.pw/ShadowsOverWestgate/sow-tools/internal/buildinfo"
|
||||||
@@ -32,7 +33,9 @@ var skippedTypes = resTypes("nss", "ndb", "gic")
|
|||||||
// manifest quietly disagrees with. Bump it only when emitted bytes change — it
|
// manifest quietly disagrees with. Bump it only when emitted bytes change — it
|
||||||
// is deliberately not the build revision, which would invalidate every
|
// is deliberately not the build revision, which would invalidate every
|
||||||
// published index on every unrelated commit.
|
// published index on every unrelated commit.
|
||||||
const emitterVersion = "1"
|
// Version 2 declares Frame_Content_Size on every blob (#86); version 1 omitted
|
||||||
|
// it below 256 bytes and no client could sync past such a blob.
|
||||||
|
const emitterVersion = "2"
|
||||||
|
|
||||||
// serverTypes are loaded only server-side; a manifest holding nothing else
|
// serverTypes are loaded only server-side; a manifest holding nothing else
|
||||||
// has no client contents. Mirrors upstream's GlobalResTypeServerList, whose
|
// has no client contents. Mirrors upstream's GlobalResTypeServerList, whose
|
||||||
@@ -62,12 +65,22 @@ type EmitResult struct {
|
|||||||
BlobsWritten int
|
BlobsWritten int
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// defaultEmitJobs is how many resources are hashed, compressed and stored at
|
||||||
|
// once. Emit is latency-bound, not CPU-bound: a blob costs a probe round-trip
|
||||||
|
// plus an upload round-trip, and a measured backfill spent 26 s of CPU across
|
||||||
|
// 9.5 minutes of wall clock. The figure matches depot's DEPOT_JOBS default and
|
||||||
|
// the transport's MaxIdleConnsPerHost, so a worker per connection needs no new
|
||||||
|
// TLS handshake.
|
||||||
|
const defaultEmitJobs = 16
|
||||||
|
|
||||||
// EmitOptions describes one emit run.
|
// EmitOptions describes one emit run.
|
||||||
type EmitOptions struct {
|
type EmitOptions struct {
|
||||||
ArtifactKey string // depot key of the artifact; the NSYM key is derived from it
|
ArtifactKey string // depot key of the artifact; the NSYM key is derived from it
|
||||||
ArtifactPath string // the file on disk
|
ArtifactPath string // the file on disk
|
||||||
As string // name override, for a TLK whose filename is not its published name
|
As string // name override, for a TLK whose filename is not its published name
|
||||||
OutDir string // write locally instead of uploading — the conformance path
|
OutDir string // write locally instead of uploading — the conformance path
|
||||||
|
Jobs int // resources in flight at once; 0 means defaultEmitJobs
|
||||||
|
Verify bool // hash what would be skipped instead of trusting presence
|
||||||
Sink sink // test seam; nil means OutDir or the zone
|
Sink sink // test seam; nil means OutDir or the zone
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -79,11 +92,18 @@ type EmitOptions struct {
|
|||||||
// real blobs in the zone and no index, which is unambiguous. Blob names are
|
// real blobs in the zone and no index, which is unambiguous. Blob names are
|
||||||
// content hashes, so re-running skips whatever already landed.
|
// content hashes, so re-running skips whatever already landed.
|
||||||
func Emit(options EmitOptions) (EmitResult, error) {
|
func Emit(options EmitOptions) (EmitResult, error) {
|
||||||
artifact, err := os.ReadFile(options.ArtifactPath)
|
artifact, err := os.Open(options.ArtifactPath)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return EmitResult{}, fmt.Errorf("read artifact: %w", err)
|
return EmitResult{}, fmt.Errorf("read artifact: %w", err)
|
||||||
}
|
}
|
||||||
if err := checkArtifactKey(options.ArtifactKey, artifact); err != nil {
|
defer artifact.Close()
|
||||||
|
info, err := artifact.Stat()
|
||||||
|
if err != nil {
|
||||||
|
return EmitResult{}, fmt.Errorf("read artifact: %w", err)
|
||||||
|
}
|
||||||
|
// A section reader, not the file itself: hashing must not move the file
|
||||||
|
// offset out from under everything that reads the artifact afterwards.
|
||||||
|
if err := checkArtifactKey(options.ArtifactKey, io.NewSectionReader(artifact, 0, info.Size())); err != nil {
|
||||||
return EmitResult{}, err
|
return EmitResult{}, err
|
||||||
}
|
}
|
||||||
name := options.As
|
name := options.As
|
||||||
@@ -98,7 +118,7 @@ func Emit(options EmitOptions) (EmitResult, error) {
|
|||||||
return EmitResult{}, err
|
return EmitResult{}, err
|
||||||
}
|
}
|
||||||
|
|
||||||
resources, err := readArtifact(options.ArtifactPath, artifact, name)
|
index, err := readArtifactIndex(options.ArtifactPath, artifact, info.Size(), name)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return EmitResult{}, err
|
return EmitResult{}, err
|
||||||
}
|
}
|
||||||
@@ -108,7 +128,11 @@ func Emit(options EmitOptions) (EmitResult, error) {
|
|||||||
return EmitResult{}, err
|
return EmitResult{}, err
|
||||||
}
|
}
|
||||||
|
|
||||||
entries, blobs, onDiskBytes, err := emitResources(resources, target)
|
jobs := options.Jobs
|
||||||
|
if jobs < 1 {
|
||||||
|
jobs = defaultEmitJobs
|
||||||
|
}
|
||||||
|
entries, blobs, onDiskBytes, err := emitResources(artifact, index, target, jobs, options.Verify)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return EmitResult{}, err
|
return EmitResult{}, err
|
||||||
}
|
}
|
||||||
@@ -137,87 +161,164 @@ func openSink(outDir string, injected sink) (sink, error) {
|
|||||||
return newZoneSink(context.Background(), os.Getenv)
|
return newZoneSink(context.Background(), os.Getenv)
|
||||||
}
|
}
|
||||||
|
|
||||||
// readArtifact returns the resources of an ERF/HAK/MOD, or the single resource
|
// readArtifactIndex locates the resources of an ERF/HAK/MOD, or the single
|
||||||
// a loose file represents. Upstream's resman does the same dispatch on the
|
// resource a loose file represents, without reading any payload. Upstream's
|
||||||
// file's first three bytes. name is the artifact's published name, which for a
|
// resman does the same dispatch on the file's first three bytes. name is the
|
||||||
// loose file is also its resref.
|
// artifact's published name, which for a loose file is also its resref.
|
||||||
func readArtifact(path string, data []byte, name string) ([]erf.Resource, error) {
|
func readArtifactIndex(path string, artifact io.ReaderAt, size int64, name string) ([]erf.IndexEntry, error) {
|
||||||
if len(data) >= 3 {
|
magic := make([]byte, 3)
|
||||||
switch string(data[:3]) {
|
if size >= 3 {
|
||||||
|
if _, err := artifact.ReadAt(magic, 0); err != nil {
|
||||||
|
return nil, fmt.Errorf("%s: %w", path, err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
switch string(magic) {
|
||||||
case "ERF", "HAK":
|
case "ERF", "HAK":
|
||||||
archive, err := erf.Read(bytes.NewReader(data))
|
index, err := erf.ReadIndex(artifact, size)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return nil, fmt.Errorf("%s: %w", path, err)
|
return nil, fmt.Errorf("%s: %w", path, err)
|
||||||
}
|
}
|
||||||
return archive.Resources, nil
|
return index.Entries, nil
|
||||||
case "MOD":
|
case "MOD":
|
||||||
// A persistent world never publishes module contents, so the .mod
|
// A persistent world never publishes module contents, so the .mod
|
||||||
// contributes no bytes to a manifest — it only says which haks and
|
// contributes no bytes to a manifest — it only says which haks and
|
||||||
// which TLK the manifest covers.
|
// which TLK the manifest covers.
|
||||||
return nil, fmt.Errorf("%s: a module is never emitted; a manifest is haks plus the TLK", path)
|
return nil, fmt.Errorf("%s: a module is never emitted; a manifest is haks plus the TLK", path)
|
||||||
}
|
}
|
||||||
}
|
|
||||||
extension := filepath.Ext(filepath.Base(path))
|
extension := filepath.Ext(filepath.Base(path))
|
||||||
restype, ok := erf.ResourceTypeForExtension(extension)
|
restype, ok := erf.ResourceTypeForExtension(extension)
|
||||||
if !ok {
|
if !ok {
|
||||||
return nil, fmt.Errorf("%s: unknown resource type %q", path, extension)
|
return nil, fmt.Errorf("%s: unknown resource type %q", path, extension)
|
||||||
}
|
}
|
||||||
return []erf.Resource{{
|
return []erf.IndexEntry{{Name: name, Type: restype, Offset: 0, Size: size}}, nil
|
||||||
Name: name,
|
|
||||||
Type: restype,
|
|
||||||
Data: data,
|
|
||||||
}}, nil
|
|
||||||
}
|
}
|
||||||
|
|
||||||
func emitResources(resources []erf.Resource, target sink) ([]Entry, int, int64, error) {
|
// emitResources hashes, compresses and stores resources, reading each payload
|
||||||
|
// from the artifact only when its turn comes. Peak memory tracks the resources
|
||||||
|
// in flight, not the archive: a 2 GB hak must emit inside a runner's few spare
|
||||||
|
// GB. jobs of them are in flight at once, so the ceiling is jobs multiplied by
|
||||||
|
// fileSizeLimit and its compressed copy — bounded, and bounded by a constant
|
||||||
|
// this package enforces itself.
|
||||||
|
//
|
||||||
|
// The returned entries are in artifact order whatever order the workers finish
|
||||||
|
// in, because a manifest's bytes are promised deterministic by emitterVersion.
|
||||||
|
func emitResources(artifact io.ReaderAt, index []erf.IndexEntry, target sink, jobs int, verify bool) ([]Entry, int, int64, error) {
|
||||||
// A resref appearing twice inside one artifact resolves to the last one,
|
// A resref appearing twice inside one artifact resolves to the last one,
|
||||||
// the way resman lets the last container added win.
|
// the way resman lets the last container added win.
|
||||||
order := make([]Identity, 0, len(resources))
|
order := make([]Identity, 0, len(index))
|
||||||
latest := make(map[Identity]erf.Resource, len(resources))
|
latest := make(map[Identity]erf.IndexEntry, len(index))
|
||||||
var tooBig []string
|
var tooBig []string
|
||||||
for _, resource := range resources {
|
for _, entry := range index {
|
||||||
if _, ok := erf.ExtensionForResourceType(resource.Type); !ok {
|
if _, ok := erf.ExtensionForResourceType(entry.Type); !ok {
|
||||||
return nil, 0, 0, fmt.Errorf("resref %s is not resolvable (unknown restype %d)", resource.Name, resource.Type)
|
return nil, 0, 0, fmt.Errorf("resref %s is not resolvable (unknown restype %d)", entry.Name, entry.Type)
|
||||||
}
|
}
|
||||||
if slices.Contains(skippedTypes, resource.Type) {
|
if slices.Contains(skippedTypes, entry.Type) {
|
||||||
continue
|
continue
|
||||||
}
|
}
|
||||||
if len(resource.Data) > fileSizeLimit {
|
if entry.Size > fileSizeLimit {
|
||||||
tooBig = append(tooBig, fmt.Sprintf("%s: %d bytes > %d", resource.Name, len(resource.Data), fileSizeLimit))
|
tooBig = append(tooBig, fmt.Sprintf("%s: %d bytes > %d", entry.Name, entry.Size, fileSizeLimit))
|
||||||
continue
|
continue
|
||||||
}
|
}
|
||||||
identity := Identity{ResRef: strings.ToLower(resource.Name), ResType: resource.Type}
|
identity := Identity{ResRef: strings.ToLower(entry.Name), ResType: entry.Type}
|
||||||
if _, seen := latest[identity]; !seen {
|
if _, seen := latest[identity]; !seen {
|
||||||
order = append(order, identity)
|
order = append(order, identity)
|
||||||
}
|
}
|
||||||
latest[identity] = resource
|
latest[identity] = entry
|
||||||
}
|
}
|
||||||
if len(tooBig) > 0 {
|
if len(tooBig) > 0 {
|
||||||
sort.Strings(tooBig)
|
sort.Strings(tooBig)
|
||||||
return nil, 0, 0, fmt.Errorf("resources exceed the file size limit:\n %s", strings.Join(tooBig, "\n "))
|
return nil, 0, 0, fmt.Errorf("resources exceed the file size limit:\n %s", strings.Join(tooBig, "\n "))
|
||||||
}
|
}
|
||||||
|
|
||||||
entries := make([]Entry, 0, len(order))
|
// Index-addressed, never appended to: a worker owns entries[i] alone, so
|
||||||
|
// the slice comes back in artifact order and needs no lock.
|
||||||
|
entries := make([]Entry, len(order))
|
||||||
var blobs int
|
var blobs int
|
||||||
var onDiskBytes int64
|
var onDiskBytes int64
|
||||||
for _, identity := range order {
|
var mu sync.Mutex
|
||||||
resource := latest[identity]
|
var firstErr error
|
||||||
sum := sha1.Sum(resource.Data)
|
// Two resrefs in one artifact can hold identical bytes, and therefore one
|
||||||
entries = append(entries, Entry{
|
// blob. Serially the sink's existence check absorbed that; in parallel both
|
||||||
|
// workers would probe, both miss, and both upload. Claiming the sha1 here
|
||||||
|
// restores the dedupe and skips the probe round-trip as well.
|
||||||
|
claimed := make(map[[20]byte]bool, len(order))
|
||||||
|
|
||||||
|
failed := func() bool {
|
||||||
|
mu.Lock()
|
||||||
|
defer mu.Unlock()
|
||||||
|
return firstErr != nil
|
||||||
|
}
|
||||||
|
|
||||||
|
store := func(i int) {
|
||||||
|
identity := order[i]
|
||||||
|
payload, err := erf.ReadPayload(artifact, latest[identity])
|
||||||
|
if err != nil {
|
||||||
|
mu.Lock()
|
||||||
|
if firstErr == nil {
|
||||||
|
firstErr = err
|
||||||
|
}
|
||||||
|
mu.Unlock()
|
||||||
|
return
|
||||||
|
}
|
||||||
|
sum := sha1.Sum(payload)
|
||||||
|
entries[i] = Entry{
|
||||||
SHA1: sum,
|
SHA1: sum,
|
||||||
Size: uint32(len(resource.Data)),
|
Size: uint32(len(payload)),
|
||||||
ResRef: identity.ResRef,
|
ResRef: identity.ResRef,
|
||||||
ResType: identity.ResType,
|
ResType: identity.ResType,
|
||||||
})
|
}
|
||||||
written, err := target.putBlob(fmt.Sprintf("%x", sum), func() []byte { return compressBlob(resource.Data) })
|
mu.Lock()
|
||||||
|
duplicate := claimed[sum]
|
||||||
|
claimed[sum] = true
|
||||||
|
mu.Unlock()
|
||||||
|
if duplicate {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
written, err := target.putBlob(fmt.Sprintf("%x", sum), verify, func() []byte { return compressBlob(payload) })
|
||||||
|
mu.Lock()
|
||||||
|
defer mu.Unlock()
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return nil, 0, 0, err
|
if firstErr == nil {
|
||||||
|
firstErr = err
|
||||||
|
}
|
||||||
|
return
|
||||||
}
|
}
|
||||||
if written > 0 {
|
if written > 0 {
|
||||||
blobs++
|
blobs++
|
||||||
onDiskBytes += written
|
onDiskBytes += written
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
if jobs < 1 {
|
||||||
|
jobs = 1
|
||||||
|
}
|
||||||
|
work := make(chan int)
|
||||||
|
var wg sync.WaitGroup
|
||||||
|
for range jobs {
|
||||||
|
wg.Add(1)
|
||||||
|
go func() {
|
||||||
|
defer wg.Done()
|
||||||
|
for i := range work {
|
||||||
|
// After a failure the run is over — the caller discards
|
||||||
|
// everything and no index is written. Draining the rest of the
|
||||||
|
// channel cheaply, rather than returning, keeps the feeder from
|
||||||
|
// blocking on workers that have gone away.
|
||||||
|
if failed() {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
store(i)
|
||||||
|
}
|
||||||
|
}()
|
||||||
|
}
|
||||||
|
for i := range order {
|
||||||
|
work <- i
|
||||||
|
}
|
||||||
|
close(work)
|
||||||
|
wg.Wait()
|
||||||
|
|
||||||
|
if firstErr != nil {
|
||||||
|
return nil, 0, 0, firstErr
|
||||||
|
}
|
||||||
return entries, blobs, onDiskBytes, nil
|
return entries, blobs, onDiskBytes, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@@ -0,0 +1,134 @@
|
|||||||
|
package nwsync
|
||||||
|
|
||||||
|
import (
|
||||||
|
"bytes"
|
||||||
|
"fmt"
|
||||||
|
"path/filepath"
|
||||||
|
"runtime/debug"
|
||||||
|
"strings"
|
||||||
|
"testing"
|
||||||
|
)
|
||||||
|
|
||||||
|
// manyResources is a hak body with enough distinct resources that a worker pool
|
||||||
|
// actually interleaves. Payloads differ so nothing is deduplicated away.
|
||||||
|
func manyResources(count int) map[string][]byte {
|
||||||
|
contents := make(map[string][]byte, count)
|
||||||
|
for i := range count {
|
||||||
|
contents[fmt.Sprintf("res%05d.tga", i)] = []byte(fmt.Sprintf("payload %d", i))
|
||||||
|
}
|
||||||
|
return contents
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestEmitProducesTheSameIndexAtEveryJobCount is the contract that lets emit be
|
||||||
|
// parallel at all: emitterVersion promises a manifest's bytes are a function of
|
||||||
|
// its artifact, so the number of workers must not be observable in the output.
|
||||||
|
func TestEmitProducesTheSameIndexAtEveryJobCount(t *testing.T) {
|
||||||
|
// The sidecar stamps a wall-clock time unless this is set, which would make
|
||||||
|
// two runs differ for a reason that has nothing to do with job count.
|
||||||
|
t.Setenv("SOURCE_DATE_EPOCH", "1700000000")
|
||||||
|
|
||||||
|
dir := t.TempDir()
|
||||||
|
hak := filepath.Join(dir, "sow_test_01.hak")
|
||||||
|
writeHak(t, hak, manyResources(64))
|
||||||
|
key := artifactKey(t, hak)
|
||||||
|
|
||||||
|
emit := func(jobs int) (manifest, sidecar []byte, result EmitResult) {
|
||||||
|
out := filepath.Join(t.TempDir(), "out")
|
||||||
|
result, err := Emit(EmitOptions{
|
||||||
|
ArtifactKey: key,
|
||||||
|
ArtifactPath: hak,
|
||||||
|
OutDir: out,
|
||||||
|
Jobs: jobs,
|
||||||
|
})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("emit at -jobs %d: %v", jobs, err)
|
||||||
|
}
|
||||||
|
manifest, sidecar, err = dirSink{root: out}.getIndex(filepath.Base(result.ManifestPath))
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("read index at -jobs %d: %v", jobs, err)
|
||||||
|
}
|
||||||
|
return manifest, sidecar, result
|
||||||
|
}
|
||||||
|
|
||||||
|
serialManifest, serialSidecar, serial := emit(1)
|
||||||
|
parallelManifest, parallelSidecar, parallel := emit(16)
|
||||||
|
|
||||||
|
if !bytes.Equal(serialManifest, parallelManifest) {
|
||||||
|
t.Errorf("manifest bytes differ between -jobs 1 and -jobs 16")
|
||||||
|
}
|
||||||
|
if !bytes.Equal(serialSidecar, parallelSidecar) {
|
||||||
|
t.Errorf("sidecar bytes differ between -jobs 1 and -jobs 16:\n %s\n %s", serialSidecar, parallelSidecar)
|
||||||
|
}
|
||||||
|
if serial.Entries != parallel.Entries || serial.BlobsWritten != parallel.BlobsWritten {
|
||||||
|
t.Errorf("-jobs 1 wrote %d entries/%d blobs, -jobs 16 wrote %d/%d",
|
||||||
|
serial.Entries, serial.BlobsWritten, parallel.Entries, parallel.BlobsWritten)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestEmitLeavesNoIndexWhenAParallelUploadFails is the fail-closed check with
|
||||||
|
// workers in flight: several uploads are in the air when the first one fails,
|
||||||
|
// and the index must still never appear. Run under -race this also covers the
|
||||||
|
// shared counters.
|
||||||
|
func TestEmitLeavesNoIndexWhenAParallelUploadFails(t *testing.T) {
|
||||||
|
fixture := newZoneFixture(t)
|
||||||
|
fixture.zone.failOn = func(key string) bool { return strings.HasPrefix(key, "data/sha1/") }
|
||||||
|
dir := t.TempDir()
|
||||||
|
hak := filepath.Join(dir, "sow_test_01.hak")
|
||||||
|
writeHak(t, hak, manyResources(64))
|
||||||
|
|
||||||
|
if _, err := Emit(EmitOptions{
|
||||||
|
ArtifactKey: artifactKey(t, hak),
|
||||||
|
ArtifactPath: hak,
|
||||||
|
Sink: fixture.sink,
|
||||||
|
Jobs: 16,
|
||||||
|
}); err == nil {
|
||||||
|
t.Fatal("emit reported success after an upload failed")
|
||||||
|
}
|
||||||
|
fixture.zone.mu.Lock()
|
||||||
|
defer fixture.zone.mu.Unlock()
|
||||||
|
for key := range fixture.zone.objects {
|
||||||
|
if strings.HasSuffix(key, ".nsym") {
|
||||||
|
t.Errorf("a half-emitted artifact published an index: %s", key)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestEmitPeakMemoryIsBoundedByJobCount pins the ceiling the parallel emit
|
||||||
|
// rests on. Peak still must not track the archive — it tracks the resources in
|
||||||
|
// flight, so a bigger hak at the same job count costs the same.
|
||||||
|
func TestEmitPeakMemoryIsBoundedByJobCount(t *testing.T) {
|
||||||
|
if testing.Short() {
|
||||||
|
t.Skip("writes a 64 MB fixture")
|
||||||
|
}
|
||||||
|
defer debug.SetGCPercent(debug.SetGCPercent(10))
|
||||||
|
|
||||||
|
measure := func(count, jobs int) uint64 {
|
||||||
|
dir := t.TempDir()
|
||||||
|
hak := filepath.Join(dir, "big.hak")
|
||||||
|
writeStreamedHak(t, hak, count)
|
||||||
|
options := EmitOptions{
|
||||||
|
ArtifactKey: artifactKey(t, hak),
|
||||||
|
ArtifactPath: hak,
|
||||||
|
As: filepath.Base(hak),
|
||||||
|
OutDir: filepath.Join(dir, "out"),
|
||||||
|
Jobs: jobs,
|
||||||
|
}
|
||||||
|
return peakHeapDuring(func() {
|
||||||
|
if _, err := Emit(options); err != nil {
|
||||||
|
t.Fatalf("emit %d resources at -jobs %d: %v", count, jobs, err)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
const jobs = 8
|
||||||
|
small := measure(8, jobs) // 8 MB
|
||||||
|
large := measure(64, jobs) // 64 MB
|
||||||
|
// Each worker may hold one resourceSize payload plus its compressed copy,
|
||||||
|
// so the pool itself is the slack — not the archive.
|
||||||
|
const slack = 24 << 20
|
||||||
|
|
||||||
|
t.Logf("peak heap at -jobs %d: 8 MB hak %d bytes, 64 MB hak %d bytes", jobs, small, large)
|
||||||
|
if large > small+slack {
|
||||||
|
t.Fatalf("peak heap scaled with artifact size at -jobs %d: 8 MB hak peaked at %d bytes, 64 MB hak at %d", jobs, small, large)
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -7,6 +7,7 @@ import (
|
|||||||
"encoding/json"
|
"encoding/json"
|
||||||
"fmt"
|
"fmt"
|
||||||
"io"
|
"io"
|
||||||
|
"path"
|
||||||
"path/filepath"
|
"path/filepath"
|
||||||
"sort"
|
"sort"
|
||||||
"strings"
|
"strings"
|
||||||
@@ -194,7 +195,14 @@ func marshalSidecar(sidecar Sidecar) ([]byte, error) {
|
|||||||
return append(body, '\r', '\n'), nil
|
return append(body, '\r', '\n'), nil
|
||||||
}
|
}
|
||||||
|
|
||||||
// blobPath is the data store path for a blob, hash tree depth 2.
|
// blobKey is where a blob lives in a zone, hash tree depth 2. emit writes it,
|
||||||
|
// verify reads it and the game client requests it, so the rule lives here and
|
||||||
|
// nowhere else.
|
||||||
|
func blobKey(sha1Hex string) string {
|
||||||
|
return path.Join("data", "sha1", sha1Hex[0:2], sha1Hex[2:4], sha1Hex)
|
||||||
|
}
|
||||||
|
|
||||||
|
// blobPath is the same location inside a local repository tree.
|
||||||
func blobPath(root, sha1Hex string) string {
|
func blobPath(root, sha1Hex string) string {
|
||||||
return filepath.Join(root, "data", "sha1", sha1Hex[0:2], sha1Hex[2:4], sha1Hex)
|
return filepath.Join(root, filepath.FromSlash(blobKey(sha1Hex)))
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -0,0 +1,150 @@
|
|||||||
|
package nwsync
|
||||||
|
|
||||||
|
import (
|
||||||
|
"bytes"
|
||||||
|
"fmt"
|
||||||
|
"math/rand"
|
||||||
|
"os"
|
||||||
|
"path/filepath"
|
||||||
|
"runtime"
|
||||||
|
"runtime/debug"
|
||||||
|
"testing"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"github.com/klauspost/compress/zstd"
|
||||||
|
|
||||||
|
"git.westgate.pw/ShadowsOverWestgate/sow-tools/internal/erf"
|
||||||
|
)
|
||||||
|
|
||||||
|
// TestSingleThreadedEncoderMatchesDefault pins the claim the blob encoder's
|
||||||
|
// concurrency setting rests on: it saves memory only, and a published blob is
|
||||||
|
// the same bytes either way.
|
||||||
|
func TestSingleThreadedEncoderMatchesDefault(t *testing.T) {
|
||||||
|
standard, err := zstd.NewWriter(nil)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
defer standard.Close()
|
||||||
|
|
||||||
|
body := make([]byte, 4<<20)
|
||||||
|
random := rand.New(rand.NewSource(1))
|
||||||
|
random.Read(body[:len(body)/2])
|
||||||
|
for _, size := range []int{0, 1, 4 << 10, len(body)} {
|
||||||
|
if !bytes.Equal(blobEncoder.EncodeAll(body[:size], nil), standard.EncodeAll(body[:size], nil)) {
|
||||||
|
t.Fatalf("%d bytes compress differently at concurrency 1", size)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// resourceSize is one payload in the memory fixtures. Real haks hold a few MB
|
||||||
|
// per resource, and peak memory is meant to track that, not the archive.
|
||||||
|
const resourceSize = 1 << 20
|
||||||
|
|
||||||
|
// writeStreamedHak builds a hak of count resources without ever holding the
|
||||||
|
// archive in memory, so the fixture itself does not decide the measurement.
|
||||||
|
// Payloads are distinct, so no blob is deduplicated away.
|
||||||
|
func writeStreamedHak(t *testing.T, path string, count int) {
|
||||||
|
t.Helper()
|
||||||
|
payload := filepath.Join(t.TempDir(), "payload.bin")
|
||||||
|
body := make([]byte, resourceSize)
|
||||||
|
for index := range body {
|
||||||
|
body[index] = byte(index)
|
||||||
|
}
|
||||||
|
|
||||||
|
resources := make([]erf.Resource, 0, count)
|
||||||
|
for index := range count {
|
||||||
|
// A distinct first byte per resource is enough to give every payload
|
||||||
|
// its own sha1 while still streaming from one file per resource.
|
||||||
|
unique := filepath.Join(filepath.Dir(payload), fmt.Sprintf("p%d.bin", index))
|
||||||
|
body[0] = byte(index)
|
||||||
|
body[1] = byte(index >> 8)
|
||||||
|
if err := os.WriteFile(unique, body, 0o644); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
resources = append(resources, erf.Resource{
|
||||||
|
Name: fmt.Sprintf("res%05d", index),
|
||||||
|
Type: restype(t, "tga"),
|
||||||
|
SourcePath: unique,
|
||||||
|
Size: resourceSize,
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
file, err := os.Create(path)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
defer file.Close()
|
||||||
|
if err := erf.Write(file, erf.New("HAK", resources)); err != nil {
|
||||||
|
t.Fatalf("write hak: %v", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// peakHeapDuring runs work while sampling the heap, and returns the largest
|
||||||
|
// live heap it saw.
|
||||||
|
func peakHeapDuring(work func()) uint64 {
|
||||||
|
runtime.GC()
|
||||||
|
done := make(chan struct{})
|
||||||
|
peak := make(chan uint64, 1)
|
||||||
|
go func() {
|
||||||
|
var highest uint64
|
||||||
|
var stats runtime.MemStats
|
||||||
|
for {
|
||||||
|
select {
|
||||||
|
case <-done:
|
||||||
|
peak <- highest
|
||||||
|
return
|
||||||
|
default:
|
||||||
|
}
|
||||||
|
runtime.ReadMemStats(&stats)
|
||||||
|
if stats.HeapAlloc > highest {
|
||||||
|
highest = stats.HeapAlloc
|
||||||
|
}
|
||||||
|
time.Sleep(time.Millisecond)
|
||||||
|
}
|
||||||
|
}()
|
||||||
|
work()
|
||||||
|
close(done)
|
||||||
|
return <-peak
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestEmitPeakMemoryDoesNotScaleWithArtifactSize is the regression check for
|
||||||
|
// the OOM kills on large haks: emit used to hold the whole archive (twice), so
|
||||||
|
// a 2 GB hak needed about 10 GB. Emitting an archive 8× bigger must not cost
|
||||||
|
// meaningfully more memory.
|
||||||
|
func TestEmitPeakMemoryDoesNotScaleWithArtifactSize(t *testing.T) {
|
||||||
|
if testing.Short() {
|
||||||
|
t.Skip("writes a 64 MB fixture")
|
||||||
|
}
|
||||||
|
// A lazy GC lets garbage pile up in proportion to the live heap, which
|
||||||
|
// hides the thing under test. Collecting eagerly makes the sampled heap
|
||||||
|
// track what emit actually holds.
|
||||||
|
defer debug.SetGCPercent(debug.SetGCPercent(10))
|
||||||
|
|
||||||
|
measure := func(count int) uint64 {
|
||||||
|
dir := t.TempDir()
|
||||||
|
hak := filepath.Join(dir, "big.hak")
|
||||||
|
writeStreamedHak(t, hak, count)
|
||||||
|
// The key is computed outside the measurement: the test helper reads
|
||||||
|
// the whole file to hash it, which emit itself no longer does.
|
||||||
|
options := EmitOptions{
|
||||||
|
ArtifactKey: artifactKey(t, hak),
|
||||||
|
ArtifactPath: hak,
|
||||||
|
As: filepath.Base(hak),
|
||||||
|
OutDir: filepath.Join(dir, "out"),
|
||||||
|
}
|
||||||
|
return peakHeapDuring(func() {
|
||||||
|
if _, err := Emit(options); err != nil {
|
||||||
|
t.Fatalf("emit %d resources: %v", count, err)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
small := measure(8) // 8 MB
|
||||||
|
large := measure(64) // 64 MB
|
||||||
|
const slack = 24 << 20
|
||||||
|
|
||||||
|
t.Logf("peak heap: 8 MB hak %d bytes, 64 MB hak %d bytes", small, large)
|
||||||
|
if large > small+slack {
|
||||||
|
t.Fatalf("peak heap scaled with artifact size: 8 MB hak peaked at %d bytes, 64 MB hak at %d", small, large)
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -458,6 +458,33 @@ func TestAssembleFailsClosedOnMissingIndex(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// Callers capture a script's stdout as a value: `dir="$(pack-haks.sh)"`. A
|
||||||
|
// summary line on stdout gets glued onto that value, so both summaries belong
|
||||||
|
// on stderr.
|
||||||
|
func TestRunKeepsSummariesOffStdout(t *testing.T) {
|
||||||
|
dir := t.TempDir()
|
||||||
|
path := filepath.Join(dir, "sow_top.hak")
|
||||||
|
writeHak(t, path, map[string][]byte{"appearance.2da": []byte("2da from sow_top")})
|
||||||
|
key := artifactKey(t, path)
|
||||||
|
out := filepath.Join(dir, "out")
|
||||||
|
|
||||||
|
for _, args := range [][]string{
|
||||||
|
{"emit", "--out", out, "--as", "sow_top.hak", key, path},
|
||||||
|
{"assemble", "--out", out, key},
|
||||||
|
} {
|
||||||
|
var stdout, stderr bytes.Buffer
|
||||||
|
if code := Run(args, &stdout, &stderr); code != exitOK {
|
||||||
|
t.Fatalf("Run(%v) exit=%d: %s", args, code, stderr.String())
|
||||||
|
}
|
||||||
|
if stdout.Len() != 0 {
|
||||||
|
t.Errorf("Run(%v) wrote to stdout: %q", args, stdout.String())
|
||||||
|
}
|
||||||
|
if stderr.Len() == 0 {
|
||||||
|
t.Errorf("Run(%v) reported no summary on stderr", args)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
func TestRunUsageErrors(t *testing.T) {
|
func TestRunUsageErrors(t *testing.T) {
|
||||||
cases := [][]string{
|
cases := [][]string{
|
||||||
nil,
|
nil,
|
||||||
@@ -474,3 +501,31 @@ func TestRunUsageErrors(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// TestEveryBlobDeclaresItsFrameContentSize guards the fault that stopped every
|
||||||
|
// client sync (#86): klauspost/compress omits Frame_Content_Size for inputs
|
||||||
|
// under 256 bytes, and the game client cannot decode a frame without it. This
|
||||||
|
// asserts a frame property, not a round trip — the zstd CLI and Go's decoder
|
||||||
|
// both stream such a frame happily, so round-tripping cannot see the defect.
|
||||||
|
func TestEveryBlobDeclaresItsFrameContentSize(t *testing.T) {
|
||||||
|
// 230 and 175 are real sizes from the manifest that failed to sync; 255/256
|
||||||
|
// straddle the encoder's threshold.
|
||||||
|
for _, size := range []int{1, 32, 175, 230, 255, 256, 257, 1024, 5000} {
|
||||||
|
payload := make([]byte, size)
|
||||||
|
for i := range payload {
|
||||||
|
payload[i] = byte('a' + i%26)
|
||||||
|
}
|
||||||
|
blob := compressBlob(payload)
|
||||||
|
if !frameDeclaresContentSize(blob[blobHeaderBytes:]) {
|
||||||
|
t.Errorf("blob of %d bytes declares no frame content size (descriptor %#x)",
|
||||||
|
size, blob[blobHeaderBytes+4])
|
||||||
|
}
|
||||||
|
got, err := decompressBlob(blob)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("decompress %d-byte blob: %v", size, err)
|
||||||
|
}
|
||||||
|
if !bytes.Equal(got, payload) {
|
||||||
|
t.Errorf("%d-byte blob did not round trip", size)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|||||||
+69
-7
@@ -12,12 +12,17 @@ import (
|
|||||||
"flag"
|
"flag"
|
||||||
"fmt"
|
"fmt"
|
||||||
"io"
|
"io"
|
||||||
|
"os"
|
||||||
)
|
)
|
||||||
|
|
||||||
const (
|
const (
|
||||||
exitOK = 0
|
exitOK = 0
|
||||||
exitUsage = 64
|
exitUsage = 64
|
||||||
exitInternal = 70
|
exitInternal = 70
|
||||||
|
// exitDrift says the command worked and the zone is wrong, which is a
|
||||||
|
// different thing for CI to act on than the command failing. It matches
|
||||||
|
// depot's code for the same meaning.
|
||||||
|
exitDrift = 1
|
||||||
)
|
)
|
||||||
|
|
||||||
// Run executes an nwsync subcommand. args[0] is the subcommand (emit|assemble);
|
// Run executes an nwsync subcommand. args[0] is the subcommand (emit|assemble);
|
||||||
@@ -29,9 +34,11 @@ func Run(args []string, stdout, stderr io.Writer) int {
|
|||||||
}
|
}
|
||||||
switch args[0] {
|
switch args[0] {
|
||||||
case "emit":
|
case "emit":
|
||||||
return runEmit(args[1:], stdout, stderr)
|
return runEmit(args[1:], stderr)
|
||||||
case "assemble":
|
case "assemble":
|
||||||
return runAssemble(args[1:], stdout, stderr)
|
return runAssemble(args[1:], stderr)
|
||||||
|
case "verify":
|
||||||
|
return runVerify(args[1:], stdout, stderr, os.Getenv)
|
||||||
case "-h", "--help", "help":
|
case "-h", "--help", "help":
|
||||||
printRunUsage(stdout)
|
printRunUsage(stdout)
|
||||||
return exitOK
|
return exitOK
|
||||||
@@ -44,17 +51,30 @@ func Run(args []string, stdout, stderr io.Writer) int {
|
|||||||
|
|
||||||
func printRunUsage(w io.Writer) {
|
func printRunUsage(w io.Writer) {
|
||||||
fmt.Fprint(w, `usage:
|
fmt.Fprint(w, `usage:
|
||||||
nwsync emit [--as NAME] [--out DIR] <artifact-key> <file>
|
nwsync emit [--as NAME] [--out DIR] [--verify] <artifact-key> <file>
|
||||||
nwsync assemble --group-id N [--tlk-key KEY] [--out DIR] <artifact-key>...
|
nwsync assemble --group-id N [--tlk-key KEY] [--out DIR] <artifact-key>...
|
||||||
|
nwsync verify [--sample N] [--base URL] <manifest-sha1>
|
||||||
|
|
||||||
emit explodes one .hak/.erf or one loose file (the TLK) into NWSync blobs plus
|
emit explodes one .hak/.erf or one loose file (the TLK) into NWSync blobs plus
|
||||||
a NSYM index covering only that artifact, and uploads both. assemble merges
|
a NSYM index covering only that artifact, and uploads both. assemble merges
|
||||||
those indexes into one manifest, reading no bulk data. Artifact keys are depot
|
those indexes into one manifest, reading no bulk data. Artifact keys are depot
|
||||||
keys; an index lives beside its artifact, with the extension replaced.
|
keys; an index lives beside its artifact, with the extension replaced.
|
||||||
|
|
||||||
|
verify reads a published manifest and its blobs back through the public pull
|
||||||
|
zone, with no credential, and decompresses and hashes every one. It is the only
|
||||||
|
check on a published blob upstream of a player's client.
|
||||||
|
|
||||||
|
--verify makes emit hash what it would otherwise skip. emit normally treats a
|
||||||
|
blob's presence as proof of its contents, so without this an object written
|
||||||
|
truncated, or written by an emitter since found broken, is skipped forever.
|
||||||
|
--verify repairs the storage zone, while verify reads the edge in front of it.
|
||||||
|
So a verify run right after a repair is a survey, not a verdict: it names the
|
||||||
|
keys the edge still serves stale. Purge those, then run it again.
|
||||||
|
|
||||||
--out DIR writes to a local repository tree instead of uploading, which is the
|
--out DIR writes to a local repository tree instead of uploading, which is the
|
||||||
conformance path against upstream nwn_nwsync_write. Without it, the zone comes
|
conformance path against upstream nwn_nwsync_write. Without it, the zone comes
|
||||||
from NWSYNC_STORAGE_ZONE, NWSYNC_STORAGE_PASSWORD and BUNNY_STORAGE_HOST.
|
from NWSYNC_STORAGE_ZONE, NWSYNC_STORAGE_PASSWORD and BUNNY_STORAGE_HOST.
|
||||||
|
verify needs none of those; its base comes from --base or NWSYNC_PULL_BASE.
|
||||||
`)
|
`)
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -76,11 +96,13 @@ func parseArgs(fs *flag.FlagSet, args []string) ([]string, error) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func runEmit(args []string, stdout, stderr io.Writer) int {
|
func runEmit(args []string, stderr io.Writer) int {
|
||||||
fs := flag.NewFlagSet("emit", flag.ContinueOnError)
|
fs := flag.NewFlagSet("emit", flag.ContinueOnError)
|
||||||
fs.SetOutput(stderr)
|
fs.SetOutput(stderr)
|
||||||
as := fs.String("as", "", "published name of the artifact, when it differs from the key")
|
as := fs.String("as", "", "published name of the artifact, when it differs from the key")
|
||||||
out := fs.String("out", "", "write to a local repository tree instead of uploading")
|
out := fs.String("out", "", "write to a local repository tree instead of uploading")
|
||||||
|
jobs := fs.Int("jobs", defaultEmitJobs, "resources to hash, compress and store at once")
|
||||||
|
verify := fs.Bool("verify", false, "read back and hash blobs that already exist instead of trusting their presence")
|
||||||
positional, err := parseArgs(fs, args)
|
positional, err := parseArgs(fs, args)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return exitUsage
|
return exitUsage
|
||||||
@@ -89,23 +111,63 @@ func runEmit(args []string, stdout, stderr io.Writer) int {
|
|||||||
fmt.Fprintf(stderr, "nwsync emit: <artifact-key> and <file> are both required\n")
|
fmt.Fprintf(stderr, "nwsync emit: <artifact-key> and <file> are both required\n")
|
||||||
return exitUsage
|
return exitUsage
|
||||||
}
|
}
|
||||||
|
if *jobs < 1 {
|
||||||
|
fmt.Fprintf(stderr, "nwsync emit: -jobs must be at least 1, got %d\n", *jobs)
|
||||||
|
return exitUsage
|
||||||
|
}
|
||||||
|
|
||||||
result, err := Emit(EmitOptions{
|
result, err := Emit(EmitOptions{
|
||||||
ArtifactKey: positional[0],
|
ArtifactKey: positional[0],
|
||||||
ArtifactPath: positional[1],
|
ArtifactPath: positional[1],
|
||||||
As: *as,
|
As: *as,
|
||||||
OutDir: *out,
|
OutDir: *out,
|
||||||
|
Jobs: *jobs,
|
||||||
|
Verify: *verify,
|
||||||
})
|
})
|
||||||
if err != nil {
|
if err != nil {
|
||||||
fmt.Fprintf(stderr, "nwsync emit: %v\n", err)
|
fmt.Fprintf(stderr, "nwsync emit: %v\n", err)
|
||||||
return exitInternal
|
return exitInternal
|
||||||
}
|
}
|
||||||
fmt.Fprintf(stdout, "emitted %s: %d resources, %d new blobs, index %s\n",
|
fmt.Fprintf(stderr, "emitted %s: %d resources, %d new blobs, index %s\n",
|
||||||
result.Name, result.Entries, result.BlobsWritten, result.ManifestPath)
|
result.Name, result.Entries, result.BlobsWritten, result.ManifestPath)
|
||||||
return exitOK
|
return exitOK
|
||||||
}
|
}
|
||||||
|
|
||||||
func runAssemble(args []string, stdout, stderr io.Writer) int {
|
func runVerify(args []string, stdout, stderr io.Writer, getenv func(string) string) int {
|
||||||
|
fs := flag.NewFlagSet("verify", flag.ContinueOnError)
|
||||||
|
fs.SetOutput(stderr)
|
||||||
|
base := fs.String("base", getenv("NWSYNC_PULL_BASE"), "pull zone base URL to read through")
|
||||||
|
sample := fs.Int("sample", 0, "check this many random blobs instead of all of them")
|
||||||
|
jobs := fs.Int("jobs", defaultEmitJobs, "blobs to fetch and hash at once")
|
||||||
|
positional, err := parseArgs(fs, args)
|
||||||
|
if err != nil {
|
||||||
|
return exitUsage
|
||||||
|
}
|
||||||
|
if len(positional) != 1 {
|
||||||
|
fmt.Fprintf(stderr, "nwsync verify: exactly one <manifest-sha1> is required\n")
|
||||||
|
return exitUsage
|
||||||
|
}
|
||||||
|
|
||||||
|
result, err := Verify(VerifyOptions{
|
||||||
|
ManifestSHA1: positional[0],
|
||||||
|
Base: *base,
|
||||||
|
Sample: *sample,
|
||||||
|
Jobs: *jobs,
|
||||||
|
Log: stderr,
|
||||||
|
})
|
||||||
|
if err != nil {
|
||||||
|
fmt.Fprintf(stderr, "nwsync verify: %v\n", err)
|
||||||
|
return exitInternal
|
||||||
|
}
|
||||||
|
fmt.Fprintf(stdout, "verified %d of %d blobs behind %d resources: %d failures, %d bytes checked\n",
|
||||||
|
result.Checked, result.Blobs, result.Entries, result.Failures, result.Bytes)
|
||||||
|
if result.Failures > 0 {
|
||||||
|
return exitDrift
|
||||||
|
}
|
||||||
|
return exitOK
|
||||||
|
}
|
||||||
|
|
||||||
|
func runAssemble(args []string, stderr io.Writer) int {
|
||||||
fs := flag.NewFlagSet("assemble", flag.ContinueOnError)
|
fs := flag.NewFlagSet("assemble", flag.ContinueOnError)
|
||||||
fs.SetOutput(stderr)
|
fs.SetOutput(stderr)
|
||||||
tlkKey := fs.String("tlk-key", "", "depot key of the TLK, which shadows nothing and merges last")
|
tlkKey := fs.String("tlk-key", "", "depot key of the TLK, which shadows nothing and merges last")
|
||||||
@@ -134,7 +196,7 @@ func runAssemble(args []string, stdout, stderr io.Writer) int {
|
|||||||
fmt.Fprintf(stderr, "nwsync assemble: %v\n", err)
|
fmt.Fprintf(stderr, "nwsync assemble: %v\n", err)
|
||||||
return exitInternal
|
return exitInternal
|
||||||
}
|
}
|
||||||
fmt.Fprintf(stdout, "assembled manifest %s: %d resources, %s\n",
|
fmt.Fprintf(stderr, "assembled manifest %s: %d resources, %s\n",
|
||||||
result.SHA1, result.Entries, result.ManifestPath)
|
result.SHA1, result.Entries, result.ManifestPath)
|
||||||
return exitOK
|
return exitOK
|
||||||
}
|
}
|
||||||
|
|||||||
+45
-11
@@ -6,6 +6,7 @@ import (
|
|||||||
"crypto/sha256"
|
"crypto/sha256"
|
||||||
"encoding/hex"
|
"encoding/hex"
|
||||||
"fmt"
|
"fmt"
|
||||||
|
"io"
|
||||||
"os"
|
"os"
|
||||||
"path"
|
"path"
|
||||||
"path/filepath"
|
"path/filepath"
|
||||||
@@ -19,11 +20,17 @@ import (
|
|||||||
// upstream's output and ours can be diffed on a developer machine.
|
// upstream's output and ours can be diffed on a developer machine.
|
||||||
type sink interface {
|
type sink interface {
|
||||||
// putBlob stores one NWCompressedBuffer blob under its sha1 name and
|
// putBlob stores one NWCompressedBuffer blob under its sha1 name and
|
||||||
// returns the bytes stored, or 0 if the blob was already there. Blob names
|
// returns the bytes stored, or 0 if a good copy was already there. Blob
|
||||||
// are content hashes, so an existing name is existing content — which is
|
// names are content hashes, so an existing name is normally taken as
|
||||||
// why body is a thunk: compression is the expensive part of emit and a
|
// existing content — which is why body is a thunk: compression is the
|
||||||
// blob that is already stored must not pay for it.
|
// expensive part of emit and a blob that is already stored must not pay
|
||||||
putBlob(sha1Hex string, body func() []byte) (int64, error)
|
// for it.
|
||||||
|
//
|
||||||
|
// verify stops trusting presence: the stored copy is read back, unwrapped
|
||||||
|
// and hashed, and replaced when it is not what its name claims. Without it
|
||||||
|
// an object written truncated, or written by an emitter since found broken,
|
||||||
|
// is skipped by every later emit forever and no backfill can repair it.
|
||||||
|
putBlob(sha1Hex string, verify bool, body func() []byte) (int64, error)
|
||||||
// putIndex stores a NSYM manifest and its sidecar under key, which is
|
// putIndex stores a NSYM manifest and its sidecar under key, which is
|
||||||
// either an artifact-derived object key or a local path.
|
// either an artifact-derived object key or a local path.
|
||||||
putIndex(key string, manifest, sidecar []byte) error
|
putIndex(key string, manifest, sidecar []byte) error
|
||||||
@@ -36,11 +43,17 @@ type sink interface {
|
|||||||
// dirSink writes a local NWSync repository tree.
|
// dirSink writes a local NWSync repository tree.
|
||||||
type dirSink struct{ root string }
|
type dirSink struct{ root string }
|
||||||
|
|
||||||
func (s dirSink) putBlob(sha1Hex string, body func() []byte) (int64, error) {
|
func (s dirSink) putBlob(sha1Hex string, verify bool, body func() []byte) (int64, error) {
|
||||||
blob := blobPath(s.root, sha1Hex)
|
blob := blobPath(s.root, sha1Hex)
|
||||||
|
if !verify {
|
||||||
|
// Stat, not read: the common path must not pay to open every blob that
|
||||||
|
// is already there.
|
||||||
if _, err := os.Stat(blob); err == nil {
|
if _, err := os.Stat(blob); err == nil {
|
||||||
return 0, nil
|
return 0, nil
|
||||||
}
|
}
|
||||||
|
} else if stored, err := os.ReadFile(blob); err == nil && blobMatchesName(stored, sha1Hex) == nil {
|
||||||
|
return 0, nil
|
||||||
|
}
|
||||||
if err := os.MkdirAll(filepath.Dir(blob), 0o755); err != nil {
|
if err := os.MkdirAll(filepath.Dir(blob), 0o755); err != nil {
|
||||||
return 0, fmt.Errorf("create blob directory: %w", err)
|
return 0, fmt.Errorf("create blob directory: %w", err)
|
||||||
}
|
}
|
||||||
@@ -90,8 +103,8 @@ type zoneSink struct {
|
|||||||
zone string
|
zone string
|
||||||
}
|
}
|
||||||
|
|
||||||
func (s zoneSink) putBlob(sha1Hex string, body func() []byte) (int64, error) {
|
func (s zoneSink) putBlob(sha1Hex string, verify bool, body func() []byte) (int64, error) {
|
||||||
key := path.Join("data", "sha1", sha1Hex[0:2], sha1Hex[2:4], sha1Hex)
|
key := blobKey(sha1Hex)
|
||||||
// A throttled probe must never be read as "missing, re-upload" or as
|
// A throttled probe must never be read as "missing, re-upload" or as
|
||||||
// "present, skip", so only a confirmed Present skips the upload.
|
// "present, skip", so only a confirmed Present skips the upload.
|
||||||
state, _, err := s.store.ProbeKey(s.ctx, key)
|
state, _, err := s.store.ProbeKey(s.ctx, key)
|
||||||
@@ -99,8 +112,25 @@ func (s zoneSink) putBlob(sha1Hex string, body func() []byte) (int64, error) {
|
|||||||
return 0, fmt.Errorf("probe blob %s: %w", sha1Hex, err)
|
return 0, fmt.Errorf("probe blob %s: %w", sha1Hex, err)
|
||||||
}
|
}
|
||||||
if state == depot.Present {
|
if state == depot.Present {
|
||||||
|
if !verify {
|
||||||
return 0, nil
|
return 0, nil
|
||||||
}
|
}
|
||||||
|
// The probe only proved the object exists. Read it back and hold it to
|
||||||
|
// its own name.
|
||||||
|
//
|
||||||
|
// This reads the storage API rather than the pull zone: emit holds the
|
||||||
|
// write credential, and a repair decision has to be made against the
|
||||||
|
// copy it is about to overwrite, not against an edge cache of it. A
|
||||||
|
// read that fails outright is a fault, not a verdict — treating it as
|
||||||
|
// "bad, re-upload" would turn a throttled zone into a full backfill.
|
||||||
|
stored, err := s.store.GetKey(s.ctx, key)
|
||||||
|
if err != nil {
|
||||||
|
return 0, fmt.Errorf("read back blob %s: %w", sha1Hex, err)
|
||||||
|
}
|
||||||
|
if blobMatchesName(stored, sha1Hex) == nil {
|
||||||
|
return 0, nil
|
||||||
|
}
|
||||||
|
}
|
||||||
data := body()
|
data := body()
|
||||||
if err := s.put(key, data); err != nil {
|
if err := s.put(key, data); err != nil {
|
||||||
return 0, fmt.Errorf("upload blob %s: %w", sha1Hex, err)
|
return 0, fmt.Errorf("upload blob %s: %w", sha1Hex, err)
|
||||||
@@ -193,14 +223,18 @@ func resolveIndexKey(artifactKey, outDir string) (string, error) {
|
|||||||
// checkArtifactKey fails closed when the key's embedded digest is not the
|
// checkArtifactKey fails closed when the key's embedded digest is not the
|
||||||
// digest of the bytes being emitted. Publishing an index under the wrong key
|
// digest of the bytes being emitted. Publishing an index under the wrong key
|
||||||
// silently pairs a manifest with the wrong artifact.
|
// silently pairs a manifest with the wrong artifact.
|
||||||
func checkArtifactKey(artifactKey string, artifact []byte) error {
|
// artifact is hashed by streaming, so a multi-gigabyte hak is never resident.
|
||||||
|
func checkArtifactKey(artifactKey string, artifact io.Reader) error {
|
||||||
base := path.Base(artifactKey)
|
base := path.Base(artifactKey)
|
||||||
digest := strings.TrimSuffix(base, path.Ext(base))
|
digest := strings.TrimSuffix(base, path.Ext(base))
|
||||||
if len(digest) != 64 {
|
if len(digest) != 64 {
|
||||||
return fmt.Errorf("artifact key %q does not name a sha256", artifactKey)
|
return fmt.Errorf("artifact key %q does not name a sha256", artifactKey)
|
||||||
}
|
}
|
||||||
sum := sha256.Sum256(artifact)
|
hash := sha256.New()
|
||||||
if got := hex.EncodeToString(sum[:]); got != digest {
|
if _, err := io.Copy(hash, artifact); err != nil {
|
||||||
|
return fmt.Errorf("hash artifact: %w", err)
|
||||||
|
}
|
||||||
|
if got := hex.EncodeToString(hash.Sum(nil)); got != digest {
|
||||||
return fmt.Errorf("artifact key %q names digest %s but the file hashes to %s", artifactKey, digest, got)
|
return fmt.Errorf("artifact key %q names digest %s but the file hashes to %s", artifactKey, digest, got)
|
||||||
}
|
}
|
||||||
return nil
|
return nil
|
||||||
|
|||||||
@@ -0,0 +1,233 @@
|
|||||||
|
package nwsync
|
||||||
|
|
||||||
|
import (
|
||||||
|
"crypto/sha1"
|
||||||
|
"encoding/hex"
|
||||||
|
"errors"
|
||||||
|
"fmt"
|
||||||
|
"io"
|
||||||
|
"math/rand/v2"
|
||||||
|
"net/http"
|
||||||
|
"path"
|
||||||
|
"sort"
|
||||||
|
"strconv"
|
||||||
|
"sync"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"git.westgate.pw/ShadowsOverWestgate/sow-tools/internal/erf"
|
||||||
|
)
|
||||||
|
|
||||||
|
// defaultPullBase is the public NWSync host, which is a Bunny pull zone fronting
|
||||||
|
// the storage zone. Verify reads through it rather than through the storage API
|
||||||
|
// on purpose: what matters is the bytes a client is served, edge behaviour
|
||||||
|
// included, not what the origin believes it holds.
|
||||||
|
const defaultPullBase = "https://nwsync.westgate.pw"
|
||||||
|
|
||||||
|
// errBlobMissing marks an object the zone does not serve at all, as distinct
|
||||||
|
// from one it serves badly.
|
||||||
|
var errBlobMissing = errors.New("missing")
|
||||||
|
|
||||||
|
// blobSource reads one object out of the zone by key. Verify never writes and
|
||||||
|
// never authenticates, so this is deliberately narrower than sink.
|
||||||
|
type blobSource interface {
|
||||||
|
get(key string) ([]byte, error)
|
||||||
|
describe(key string) string
|
||||||
|
}
|
||||||
|
|
||||||
|
// pullZone reads the zone over plain HTTP, with no credential.
|
||||||
|
type pullZone struct {
|
||||||
|
base string
|
||||||
|
client *http.Client
|
||||||
|
}
|
||||||
|
|
||||||
|
func newPullZone(base string) blobSource {
|
||||||
|
if base == "" {
|
||||||
|
base = defaultPullBase
|
||||||
|
}
|
||||||
|
return pullZone{
|
||||||
|
base: base,
|
||||||
|
// A full sweep is tens of thousands of small requests, so connections
|
||||||
|
// have to be reused; the default transport does that already.
|
||||||
|
client: &http.Client{Timeout: 60 * time.Second},
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func (z pullZone) describe(key string) string { return z.base + "/" + key }
|
||||||
|
|
||||||
|
func (z pullZone) get(key string) ([]byte, error) {
|
||||||
|
resp, err := z.client.Get(z.describe(key))
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
defer resp.Body.Close()
|
||||||
|
if resp.StatusCode == http.StatusNotFound || resp.StatusCode == http.StatusGone {
|
||||||
|
_, _ = io.Copy(io.Discard, resp.Body)
|
||||||
|
return nil, errBlobMissing
|
||||||
|
}
|
||||||
|
if resp.StatusCode < 200 || resp.StatusCode >= 300 {
|
||||||
|
_, _ = io.Copy(io.Discard, resp.Body)
|
||||||
|
return nil, fmt.Errorf("unexpected status %d", resp.StatusCode)
|
||||||
|
}
|
||||||
|
return io.ReadAll(resp.Body)
|
||||||
|
}
|
||||||
|
|
||||||
|
// VerifyOptions describes one verify run.
|
||||||
|
type VerifyOptions struct {
|
||||||
|
ManifestSHA1 string // the merged manifest to verify
|
||||||
|
Base string // pull zone base URL; empty means defaultPullBase
|
||||||
|
Sample int // check this many random blobs; 0 means all of them
|
||||||
|
Jobs int // blobs in flight at once; 0 means defaultEmitJobs
|
||||||
|
Source blobSource // test seam; nil means the pull zone at Base
|
||||||
|
Log io.Writer // per-blob failures land here; nil discards them
|
||||||
|
}
|
||||||
|
|
||||||
|
// VerifyResult reports what one verify run found.
|
||||||
|
type VerifyResult struct {
|
||||||
|
Entries int // resources the manifest names
|
||||||
|
Blobs int // distinct blobs behind those resources
|
||||||
|
Checked int // blobs actually fetched
|
||||||
|
Failures int // blobs that failed a check
|
||||||
|
Bytes int64 // uncompressed bytes verified
|
||||||
|
}
|
||||||
|
|
||||||
|
// Verify reads a published manifest and its blobs the way a client reads them,
|
||||||
|
// and reports every blob that is not what the manifest says it is.
|
||||||
|
//
|
||||||
|
// Presence is not correctness. emit skips an object that already exists on the
|
||||||
|
// strength of a 1-byte range GET, so a truncated or wrongly framed object is
|
||||||
|
// skipped by every later emit forever and the backfill cannot repair it. This is
|
||||||
|
// the only thing upstream of a player's client that can tell that has happened.
|
||||||
|
//
|
||||||
|
// Every blob is decompressed and hashed. A Content-Length check would pass the
|
||||||
|
// exact failure mode being hunted — a byte-correct-looking object whose contents
|
||||||
|
// are wrong — and a round-trip check alone would pass a frame that omits its
|
||||||
|
// content size, because Go's decoder is more capable than the client's (#86).
|
||||||
|
func Verify(options VerifyOptions) (VerifyResult, error) {
|
||||||
|
// The argument is interpolated straight into a URL path, so it is checked
|
||||||
|
// rather than trusted: exactly 20 bytes of hex, nothing else.
|
||||||
|
if sum, err := hex.DecodeString(options.ManifestSHA1); err != nil || len(sum) != sha1.Size {
|
||||||
|
return VerifyResult{}, fmt.Errorf("%q is not a manifest sha1", options.ManifestSHA1)
|
||||||
|
}
|
||||||
|
source := options.Source
|
||||||
|
if source == nil {
|
||||||
|
source = newPullZone(options.Base)
|
||||||
|
}
|
||||||
|
log := options.Log
|
||||||
|
if log == nil {
|
||||||
|
log = io.Discard
|
||||||
|
}
|
||||||
|
|
||||||
|
manifestKey := path.Join("manifests", options.ManifestSHA1)
|
||||||
|
data, err := source.get(manifestKey)
|
||||||
|
if err != nil {
|
||||||
|
return VerifyResult{}, fmt.Errorf("%s: %w", source.describe(manifestKey), err)
|
||||||
|
}
|
||||||
|
// A manifest is named after its own sha1, so this catches the zone serving
|
||||||
|
// a different manifest — or a truncated one — before any blob is fetched.
|
||||||
|
if got := hex.EncodeToString(sha1Sum(data)); got != options.ManifestSHA1 {
|
||||||
|
return VerifyResult{}, fmt.Errorf("%s hashes to %s, not the manifest asked for",
|
||||||
|
source.describe(manifestKey), got)
|
||||||
|
}
|
||||||
|
entries, err := readManifest(data)
|
||||||
|
if err != nil {
|
||||||
|
return VerifyResult{}, fmt.Errorf("%s: %w", source.describe(manifestKey), err)
|
||||||
|
}
|
||||||
|
|
||||||
|
// A manifest names one blob many times over: mappings share a sha1, and so
|
||||||
|
// do resrefs with identical contents. Fetch each blob once.
|
||||||
|
blobs := make([]Entry, 0, len(entries))
|
||||||
|
seen := make(map[[20]byte]bool, len(entries))
|
||||||
|
for _, entry := range entries {
|
||||||
|
if seen[entry.SHA1] {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
seen[entry.SHA1] = true
|
||||||
|
blobs = append(blobs, entry)
|
||||||
|
}
|
||||||
|
|
||||||
|
result := VerifyResult{Entries: len(entries), Blobs: len(blobs)}
|
||||||
|
checking := blobs
|
||||||
|
if options.Sample > 0 && options.Sample < len(blobs) {
|
||||||
|
// A full sweep of the live manifest is ~69,000 objects and ~15 GB, so
|
||||||
|
// sampling is what makes verifying a routine act rather than an event.
|
||||||
|
picks := rand.Perm(len(blobs))[:options.Sample]
|
||||||
|
checking = make([]Entry, 0, options.Sample)
|
||||||
|
for _, i := range picks {
|
||||||
|
checking = append(checking, blobs[i])
|
||||||
|
}
|
||||||
|
}
|
||||||
|
result.Checked = len(checking)
|
||||||
|
|
||||||
|
jobs := options.Jobs
|
||||||
|
if jobs < 1 {
|
||||||
|
jobs = defaultEmitJobs
|
||||||
|
}
|
||||||
|
var (
|
||||||
|
mu sync.Mutex
|
||||||
|
failures []string
|
||||||
|
)
|
||||||
|
work := make(chan Entry)
|
||||||
|
var wg sync.WaitGroup
|
||||||
|
for range jobs {
|
||||||
|
wg.Go(func() {
|
||||||
|
for entry := range work {
|
||||||
|
fault := checkEntry(source, entry)
|
||||||
|
mu.Lock()
|
||||||
|
if fault != "" {
|
||||||
|
failures = append(failures, fault)
|
||||||
|
} else {
|
||||||
|
result.Bytes += int64(entry.Size)
|
||||||
|
}
|
||||||
|
mu.Unlock()
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
for _, entry := range checking {
|
||||||
|
work <- entry
|
||||||
|
}
|
||||||
|
close(work)
|
||||||
|
wg.Wait()
|
||||||
|
|
||||||
|
// Workers finish in any order; a report an operator can diff must not.
|
||||||
|
sort.Strings(failures)
|
||||||
|
for _, fault := range failures {
|
||||||
|
fmt.Fprintln(log, fault)
|
||||||
|
}
|
||||||
|
result.Failures = len(failures)
|
||||||
|
return result, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// checkEntry fetches one blob and returns a one-line fault, or "" if it is
|
||||||
|
// exactly what the manifest entry says it is.
|
||||||
|
func checkEntry(source blobSource, entry Entry) string {
|
||||||
|
// Name the resource, not just the hash: an operator has to find the thing
|
||||||
|
// in a hak, and a bare sha1 says nothing about where to look.
|
||||||
|
extension, ok := erf.ExtensionForResourceType(entry.ResType)
|
||||||
|
if !ok {
|
||||||
|
extension = strconv.Itoa(int(entry.ResType))
|
||||||
|
}
|
||||||
|
where := fmt.Sprintf("%s (%s.%s)", entry.sha1Hex(), entry.ResRef, extension)
|
||||||
|
blob, err := source.get(blobKey(entry.sha1Hex()))
|
||||||
|
if err != nil {
|
||||||
|
if errors.Is(err, errBlobMissing) {
|
||||||
|
return where + ": missing"
|
||||||
|
}
|
||||||
|
return where + ": unreadable: " + err.Error()
|
||||||
|
}
|
||||||
|
data, err := inspectBlob(blob)
|
||||||
|
if err != nil {
|
||||||
|
return where + ": " + err.Error()
|
||||||
|
}
|
||||||
|
if uint32(len(data)) != entry.Size {
|
||||||
|
return fmt.Sprintf("%s: size mismatch: %d bytes, manifest says %d", where, len(data), entry.Size)
|
||||||
|
}
|
||||||
|
if sha1.Sum(data) != entry.SHA1 {
|
||||||
|
return fmt.Sprintf("%s: hash mismatch: contents hash to %x", where, sha1.Sum(data))
|
||||||
|
}
|
||||||
|
return ""
|
||||||
|
}
|
||||||
|
|
||||||
|
func sha1Sum(data []byte) []byte {
|
||||||
|
sum := sha1.Sum(data)
|
||||||
|
return sum[:]
|
||||||
|
}
|
||||||
@@ -0,0 +1,285 @@
|
|||||||
|
package nwsync
|
||||||
|
|
||||||
|
import (
|
||||||
|
"bytes"
|
||||||
|
"crypto/sha1"
|
||||||
|
"encoding/hex"
|
||||||
|
"os"
|
||||||
|
"path/filepath"
|
||||||
|
"strings"
|
||||||
|
"testing"
|
||||||
|
|
||||||
|
"github.com/klauspost/compress/zstd"
|
||||||
|
)
|
||||||
|
|
||||||
|
// verifyFixture emits two haks and a TLK into a fake zone, assembles them, and
|
||||||
|
// hands back a verifier reading that zone the way a client would.
|
||||||
|
type verifyFixture struct {
|
||||||
|
*zoneSinkFixture
|
||||||
|
manifestSHA1 string
|
||||||
|
}
|
||||||
|
|
||||||
|
func newVerifyFixture(t *testing.T) *verifyFixture {
|
||||||
|
t.Helper()
|
||||||
|
zone := newZoneFixture(t)
|
||||||
|
dir := t.TempDir()
|
||||||
|
hak := filepath.Join(dir, "sow_test_01.hak")
|
||||||
|
// A payload under 256 bytes is the one the frame-header check exists for.
|
||||||
|
writeHak(t, hak, map[string][]byte{
|
||||||
|
"bloodstain1.tga": []byte("small"),
|
||||||
|
"appearance.2da": bytes.Repeat([]byte("2DA V2.0\n"), 200),
|
||||||
|
})
|
||||||
|
tlk := filepath.Join(dir, "sow_tlk.tlk")
|
||||||
|
if err := os.WriteFile(tlk, []byte("TLK V3.0 payload"), 0o644); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
zone.emit(t, hak)
|
||||||
|
if _, err := Emit(EmitOptions{
|
||||||
|
ArtifactKey: artifactKey(t, tlk), ArtifactPath: tlk, As: "sow_tlk.tlk", Sink: zone.sink,
|
||||||
|
}); err != nil {
|
||||||
|
t.Fatalf("emit tlk: %v", err)
|
||||||
|
}
|
||||||
|
assembled, err := Assemble(AssembleOptions{
|
||||||
|
ArtifactKeys: []string{artifactKey(t, hak)}, TLKKey: artifactKey(t, tlk), Sink: zone.sink,
|
||||||
|
})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("assemble: %v", err)
|
||||||
|
}
|
||||||
|
return &verifyFixture{zoneSinkFixture: zone, manifestSHA1: assembled.SHA1}
|
||||||
|
}
|
||||||
|
|
||||||
|
func (f *verifyFixture) verify(t *testing.T, sample int) (VerifyResult, string, error) {
|
||||||
|
t.Helper()
|
||||||
|
var log bytes.Buffer
|
||||||
|
result, err := Verify(VerifyOptions{
|
||||||
|
ManifestSHA1: f.manifestSHA1,
|
||||||
|
Sample: sample,
|
||||||
|
Source: f.zone.pullZone(),
|
||||||
|
Log: &log,
|
||||||
|
})
|
||||||
|
return result, log.String(), err
|
||||||
|
}
|
||||||
|
|
||||||
|
// keyOf is where a resource's blob lives, addressed by the sha1 of its
|
||||||
|
// uncompressed bytes — the same path the client requests.
|
||||||
|
func keyOf(body []byte) string {
|
||||||
|
sum := sha1.Sum(body)
|
||||||
|
return blobKey(hex.EncodeToString(sum[:]))
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestVerifyPassesACleanZone(t *testing.T) {
|
||||||
|
fixture := newVerifyFixture(t)
|
||||||
|
result, log, err := fixture.verify(t, 0)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("verify: %v", err)
|
||||||
|
}
|
||||||
|
if result.Failures != 0 {
|
||||||
|
t.Errorf("verify reported %d failures on a clean zone: %s", result.Failures, log)
|
||||||
|
}
|
||||||
|
// A default run is a full sweep, so it must reach every blob the manifest
|
||||||
|
// names — not some of them.
|
||||||
|
if result.Checked != result.Blobs || result.Blobs == 0 {
|
||||||
|
t.Errorf("checked %d of %d blobs; a full sweep must check all of them", result.Checked, result.Blobs)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestVerifyReportsAMissingBlob(t *testing.T) {
|
||||||
|
fixture := newVerifyFixture(t)
|
||||||
|
key := keyOf([]byte("small"))
|
||||||
|
fixture.zone.mu.Lock()
|
||||||
|
delete(fixture.zone.objects, key)
|
||||||
|
fixture.zone.mu.Unlock()
|
||||||
|
|
||||||
|
result, log, err := fixture.verify(t, 0)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("verify: %v", err)
|
||||||
|
}
|
||||||
|
if result.Failures != 1 {
|
||||||
|
t.Fatalf("reported %d failures, want 1: %s", result.Failures, log)
|
||||||
|
}
|
||||||
|
if !strings.Contains(log, "missing") {
|
||||||
|
t.Errorf("a deleted blob was not reported as missing: %s", log)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestVerifyReportsATruncatedBlob(t *testing.T) {
|
||||||
|
fixture := newVerifyFixture(t)
|
||||||
|
key := keyOf([]byte("small"))
|
||||||
|
fixture.zone.mu.Lock()
|
||||||
|
fixture.zone.objects[key] = fixture.zone.objects[key][:blobHeaderBytes+4]
|
||||||
|
fixture.zone.mu.Unlock()
|
||||||
|
|
||||||
|
result, log, err := fixture.verify(t, 0)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("verify: %v", err)
|
||||||
|
}
|
||||||
|
if result.Failures != 1 {
|
||||||
|
t.Fatalf("reported %d failures, want 1: %s", result.Failures, log)
|
||||||
|
}
|
||||||
|
if !strings.Contains(log, "framing") {
|
||||||
|
t.Errorf("a truncated blob was not reported as malformed framing: %s", log)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestVerifyRejectsABlobWithNoDeclaredFrameContentSize is the check that #86
|
||||||
|
// slipped past: the blob decompresses to exactly the right bytes, so a verifier
|
||||||
|
// that only round-trips certifies it, yet the client cannot decode it.
|
||||||
|
func TestVerifyRejectsABlobWithNoDeclaredFrameContentSize(t *testing.T) {
|
||||||
|
fixture := newVerifyFixture(t)
|
||||||
|
body := []byte("small")
|
||||||
|
key := keyOf(body)
|
||||||
|
|
||||||
|
fixture.zone.mu.Lock()
|
||||||
|
good := fixture.zone.objects[key]
|
||||||
|
encoder, err := zstd.NewWriter(nil, zstd.WithEncoderConcurrency(1))
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
bad := append(append([]byte{}, good[:blobHeaderBytes]...), encoder.EncodeAll(body, nil)...)
|
||||||
|
fixture.zone.objects[key] = bad
|
||||||
|
fixture.zone.mu.Unlock()
|
||||||
|
|
||||||
|
if frameDeclaresContentSize(bad[blobHeaderBytes:]) {
|
||||||
|
t.Fatal("the fixture blob declares a content size; it cannot exercise the check")
|
||||||
|
}
|
||||||
|
if got, err := decompressBlob(bad); err != nil || !bytes.Equal(got, body) {
|
||||||
|
t.Fatalf("the fixture blob must round trip, or it proves nothing: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
result, log, err := fixture.verify(t, 0)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("verify: %v", err)
|
||||||
|
}
|
||||||
|
if result.Failures != 1 {
|
||||||
|
t.Fatalf("reported %d failures, want 1: %s", result.Failures, log)
|
||||||
|
}
|
||||||
|
if !strings.Contains(log, "content size") {
|
||||||
|
t.Errorf("undeclared frame content size was not the reported reason: %s", log)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestVerifyReportsWrongContents(t *testing.T) {
|
||||||
|
fixture := newVerifyFixture(t)
|
||||||
|
key := keyOf([]byte("small"))
|
||||||
|
fixture.zone.mu.Lock()
|
||||||
|
// Valid framing, valid zstd, wrong bytes: only decompressing and hashing
|
||||||
|
// can see this, which is why Content-Length is not enough.
|
||||||
|
fixture.zone.objects[key] = compressBlob([]byte("wrong"))
|
||||||
|
fixture.zone.mu.Unlock()
|
||||||
|
|
||||||
|
result, log, err := fixture.verify(t, 0)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("verify: %v", err)
|
||||||
|
}
|
||||||
|
if result.Failures != 1 {
|
||||||
|
t.Fatalf("reported %d failures, want 1: %s", result.Failures, log)
|
||||||
|
}
|
||||||
|
if !strings.Contains(log, "hash mismatch") {
|
||||||
|
t.Errorf("wrong contents were not reported as a hash mismatch: %s", log)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestVerifyReportsAShortBlob(t *testing.T) {
|
||||||
|
fixture := newVerifyFixture(t)
|
||||||
|
key := keyOf([]byte("small"))
|
||||||
|
fixture.zone.mu.Lock()
|
||||||
|
// Well-formed all the way down and simply too short — the shape a killed
|
||||||
|
// upload leaves behind, and the one a Content-Length check would pass.
|
||||||
|
fixture.zone.objects[key] = compressBlob([]byte("sma"))
|
||||||
|
fixture.zone.mu.Unlock()
|
||||||
|
|
||||||
|
result, log, err := fixture.verify(t, 0)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("verify: %v", err)
|
||||||
|
}
|
||||||
|
if result.Failures != 1 {
|
||||||
|
t.Fatalf("reported %d failures, want 1: %s", result.Failures, log)
|
||||||
|
}
|
||||||
|
if !strings.Contains(log, "size mismatch") {
|
||||||
|
t.Errorf("a short blob was not reported as a size mismatch: %s", log)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestVerifyFailsWhenTheManifestIsNotTheOneAsked(t *testing.T) {
|
||||||
|
fixture := newVerifyFixture(t)
|
||||||
|
fixture.zone.mu.Lock()
|
||||||
|
fixture.zone.objects["manifests/"+fixture.manifestSHA1] = []byte("NSYM garbage")
|
||||||
|
fixture.zone.mu.Unlock()
|
||||||
|
|
||||||
|
if _, _, err := fixture.verify(t, 0); err == nil {
|
||||||
|
t.Fatal("verify accepted a manifest that is not the one requested")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestVerifySampleChecksFewerBlobs(t *testing.T) {
|
||||||
|
fixture := newVerifyFixture(t)
|
||||||
|
result, log, err := fixture.verify(t, 1)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("verify: %v", err)
|
||||||
|
}
|
||||||
|
if result.Checked != 1 {
|
||||||
|
t.Errorf("--sample 1 checked %d blobs, want 1: %s", result.Checked, log)
|
||||||
|
}
|
||||||
|
if result.Blobs <= result.Checked {
|
||||||
|
t.Errorf("sampling %d of %d blobs is not a sample", result.Checked, result.Blobs)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestEmitVerifyReplacesABlobThatIsNotItsName covers the reason #86 could not be
|
||||||
|
// fixed by the encoder alone: emit skips whatever is already present, so every
|
||||||
|
// blob published by the broken encoder stays broken until emit stops trusting
|
||||||
|
// presence.
|
||||||
|
func TestEmitVerifyReplacesABlobThatIsNotItsName(t *testing.T) {
|
||||||
|
fixture := newZoneFixture(t)
|
||||||
|
dir := t.TempDir()
|
||||||
|
hak := filepath.Join(dir, "sow_test_01.hak")
|
||||||
|
body := []byte("blood")
|
||||||
|
writeHak(t, hak, map[string][]byte{"bloodstain1.tga": body})
|
||||||
|
|
||||||
|
emit := func(verify bool) EmitResult {
|
||||||
|
t.Helper()
|
||||||
|
result, err := Emit(EmitOptions{
|
||||||
|
ArtifactKey: artifactKey(t, hak),
|
||||||
|
ArtifactPath: hak,
|
||||||
|
Sink: fixture.sink,
|
||||||
|
Verify: verify,
|
||||||
|
})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("emit (verify=%v): %v", verify, err)
|
||||||
|
}
|
||||||
|
return result
|
||||||
|
}
|
||||||
|
|
||||||
|
emit(false)
|
||||||
|
key := keyOf(body)
|
||||||
|
encoder, err := zstd.NewWriter(nil, zstd.WithEncoderConcurrency(1))
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
fixture.zone.mu.Lock()
|
||||||
|
good := fixture.zone.objects[key]
|
||||||
|
fixture.zone.objects[key] = append(append([]byte{}, good[:blobHeaderBytes]...), encoder.EncodeAll(body, nil)...)
|
||||||
|
fixture.zone.mu.Unlock()
|
||||||
|
|
||||||
|
if plain := emit(false); plain.BlobsWritten != 0 {
|
||||||
|
t.Fatalf("a plain re-emit wrote %d blobs; it is supposed to trust presence", plain.BlobsWritten)
|
||||||
|
}
|
||||||
|
if verified := emit(true); verified.BlobsWritten != 1 {
|
||||||
|
t.Fatalf("--verify wrote %d blobs, want 1 (the bad copy must be replaced)", verified.BlobsWritten)
|
||||||
|
}
|
||||||
|
|
||||||
|
fixture.zone.mu.Lock()
|
||||||
|
repaired := fixture.zone.objects[key]
|
||||||
|
fixture.zone.mu.Unlock()
|
||||||
|
if !bytes.Equal(repaired, good) {
|
||||||
|
t.Error("the replaced blob is not what the current encoder produces")
|
||||||
|
}
|
||||||
|
if _, err := inspectBlob(repaired); err != nil {
|
||||||
|
t.Errorf("the replaced blob still fails inspection: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
// A second verifying run has nothing left to repair.
|
||||||
|
if again := emit(true); again.BlobsWritten != 0 {
|
||||||
|
t.Errorf("--verify rewrote %d good blobs", again.BlobsWritten)
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -21,6 +21,13 @@ type fakeZone struct {
|
|||||||
objects map[string][]byte
|
objects map[string][]byte
|
||||||
puts []string
|
puts []string
|
||||||
failOn func(key string) bool // when true, the PUT fails
|
failOn func(key string) bool // when true, the PUT fails
|
||||||
|
url string // base the same objects are readable at
|
||||||
|
}
|
||||||
|
|
||||||
|
// pullZone reads the fake zone the way the public pull zone is read: plain
|
||||||
|
// unauthenticated GETs, no storage API.
|
||||||
|
func (z *fakeZone) pullZone() blobSource {
|
||||||
|
return newPullZone(z.url)
|
||||||
}
|
}
|
||||||
|
|
||||||
func newFakeZone(t *testing.T) (*fakeZone, func(string) string) {
|
func newFakeZone(t *testing.T) (*fakeZone, func(string) string) {
|
||||||
@@ -28,6 +35,7 @@ func newFakeZone(t *testing.T) (*fakeZone, func(string) string) {
|
|||||||
zone := &fakeZone{objects: map[string][]byte{}}
|
zone := &fakeZone{objects: map[string][]byte{}}
|
||||||
server := httptest.NewServer(zone)
|
server := httptest.NewServer(zone)
|
||||||
t.Cleanup(server.Close)
|
t.Cleanup(server.Close)
|
||||||
|
zone.url = server.URL + "/sow-nwsync"
|
||||||
getenv := func(name string) string {
|
getenv := func(name string) string {
|
||||||
switch name {
|
switch name {
|
||||||
case "NWSYNC_STORAGE_ZONE":
|
case "NWSYNC_STORAGE_ZONE":
|
||||||
|
|||||||
@@ -126,8 +126,14 @@ func extractArchiveResources(p *project.Project, archive erf.Archive, desired ma
|
|||||||
}
|
}
|
||||||
// *palcus.itp are Toolset-generated palette projections; the module
|
// *palcus.itp are Toolset-generated palette projections; the module
|
||||||
// build regenerates them from source blueprints, so extraction never
|
// build regenerates them from source blueprints, so extraction never
|
||||||
// writes them back into source.
|
// writes them back into source. The committed skeleton still has to
|
||||||
|
// survive stale cleanup, so mark its target as desired.
|
||||||
if ext == "itp" && isPaletteProjectionResref(resource.Name) {
|
if ext == "itp" && isPaletteProjectionResref(resource.Name) {
|
||||||
|
effective := p.EffectiveConfig()
|
||||||
|
target, err := extractionTarget(p, "paths.source", effective.Paths.Source, p.SourceDir(), sourceSubdir(ext), strings.ToLower(resource.Name)+"."+ext+".json")
|
||||||
|
if err == nil {
|
||||||
|
desired[target] = struct{}{}
|
||||||
|
}
|
||||||
skippedCount++
|
skippedCount++
|
||||||
continue
|
continue
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -3312,6 +3312,88 @@ func TestExtractOverwritesAndRemovesStaleFiles(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func TestExtractKeepsPaletteSkeletonsThroughStaleCleanup(t *testing.T) {
|
||||||
|
root := t.TempDir()
|
||||||
|
mustMkdir(t, filepath.Join(root, "src", "module"))
|
||||||
|
mustMkdir(t, filepath.Join(root, "src", "palettes"))
|
||||||
|
mustMkdir(t, filepath.Join(root, "assets"))
|
||||||
|
mustMkdir(t, filepath.Join(root, "build"))
|
||||||
|
|
||||||
|
mustWriteFile(t, filepath.Join(root, "nwn-tool.json"), `{
|
||||||
|
"module": {
|
||||||
|
"name": "Test Module",
|
||||||
|
"resref": "testmod"
|
||||||
|
},
|
||||||
|
"paths": {
|
||||||
|
"source": "src",
|
||||||
|
"assets": "assets",
|
||||||
|
"build": "build"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
`)
|
||||||
|
|
||||||
|
mustWriteFile(t, filepath.Join(root, "src", "module", "module.ifo.json"), `{
|
||||||
|
"file_type": "IFO ",
|
||||||
|
"file_version": "V3.2",
|
||||||
|
"root": {
|
||||||
|
"struct_type": 0,
|
||||||
|
"fields": [
|
||||||
|
{
|
||||||
|
"label": "Mod_Name",
|
||||||
|
"type": "CExoString",
|
||||||
|
"value": "Original Module"
|
||||||
|
}
|
||||||
|
]
|
||||||
|
}
|
||||||
|
}
|
||||||
|
`)
|
||||||
|
skeletonPath := filepath.Join(root, "src", "palettes", "creaturepalcus.itp.json")
|
||||||
|
mustWriteFile(t, skeletonPath, `{
|
||||||
|
"file_type": "ITP ",
|
||||||
|
"file_version": "V3.2",
|
||||||
|
"root": {
|
||||||
|
"struct_type": 0,
|
||||||
|
"fields": [
|
||||||
|
{
|
||||||
|
"label": "MAIN",
|
||||||
|
"type": "List",
|
||||||
|
"value": []
|
||||||
|
}
|
||||||
|
]
|
||||||
|
}
|
||||||
|
}
|
||||||
|
`)
|
||||||
|
|
||||||
|
p, err := project.Load(root)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("load project: %v", err)
|
||||||
|
}
|
||||||
|
if err := p.ValidateLayout(); err != nil {
|
||||||
|
t.Fatalf("validate layout: %v", err)
|
||||||
|
}
|
||||||
|
if err := p.Scan(); err != nil {
|
||||||
|
t.Fatalf("scan: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
if _, err := BuildModule(p); err != nil {
|
||||||
|
t.Fatalf("build module: %v", err)
|
||||||
|
}
|
||||||
|
if err := p.Scan(); err != nil {
|
||||||
|
t.Fatalf("rescan before extract: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
result, err := Extract(p)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("extract: %v", err)
|
||||||
|
}
|
||||||
|
if result.Removed != 0 {
|
||||||
|
t.Fatalf("expected no removed files, got %d", result.Removed)
|
||||||
|
}
|
||||||
|
if _, err := os.Stat(skeletonPath); err != nil {
|
||||||
|
t.Fatalf("expected palette skeleton to survive extract, stat err=%v", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
func TestExtractMergesConfiguredGFFJSONFieldsAndLists(t *testing.T) {
|
func TestExtractMergesConfiguredGFFJSONFieldsAndLists(t *testing.T) {
|
||||||
root := t.TempDir()
|
root := t.TempDir()
|
||||||
mustMkdir(t, filepath.Join(root, "src", "module"))
|
mustMkdir(t, filepath.Join(root, "src", "module"))
|
||||||
|
|||||||
@@ -560,8 +560,18 @@ func (p *Project) ValidateLayout() error {
|
|||||||
if strings.TrimSpace(p.Config.Module.ResRef) == "" {
|
if strings.TrimSpace(p.Config.Module.ResRef) == "" {
|
||||||
failures = append(failures, errors.New("module.resref is required"))
|
failures = append(failures, errors.New("module.resref is required"))
|
||||||
}
|
}
|
||||||
if len(p.Config.Module.ResRef) > 16 {
|
// module.resref names the built .mod FILE, so the 16-byte resref limit does not
|
||||||
failures = append(failures, fmt.Errorf("module.resref %q exceeds 16 characters", p.Config.Module.ResRef))
|
// apply to it — NWN:EE module file names are routinely longer. It is validated as
|
||||||
|
// a file name instead. The limit still binds when the same value has to be a real
|
||||||
|
// resref: with no haks configured, an asset project names its single generated HAK
|
||||||
|
// after it, and a HAK name is a resref the engine loads.
|
||||||
|
if err := validateOutputFileName("module.resref", p.Config.Module.ResRef+".mod", ".mod"); err != nil {
|
||||||
|
failures = append(failures, err)
|
||||||
|
}
|
||||||
|
if len(p.Config.Module.ResRef) > 16 && strings.TrimSpace(p.Config.Paths.Assets) != "" && len(p.Config.HAKs) == 0 {
|
||||||
|
failures = append(failures, fmt.Errorf(
|
||||||
|
"module.resref %q exceeds 16 characters and would name this project's generated HAK; configure haks[] with a shorter name",
|
||||||
|
p.Config.Module.ResRef))
|
||||||
}
|
}
|
||||||
if strings.TrimSpace(p.Config.Paths.Source) == "" && strings.TrimSpace(p.Config.Paths.Assets) == "" && !p.HasTopData() {
|
if strings.TrimSpace(p.Config.Paths.Source) == "" && strings.TrimSpace(p.Config.Paths.Assets) == "" && !p.HasTopData() {
|
||||||
failures = append(failures, errors.New("at least one of paths.source, paths.assets, or topdata.source is required"))
|
failures = append(failures, errors.New("at least one of paths.source, paths.assets, or topdata.source is required"))
|
||||||
|
|||||||
@@ -1093,6 +1093,78 @@ func TestValidateLayoutAllowsMissingAssetsDir(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// module.resref names the built .mod FILE, not a resource inside an archive, so the
|
||||||
|
// 16-byte resref limit does not apply to it. NWN:EE module file names are commonly
|
||||||
|
// longer (ShadowsOverWestgate.mod is 19). The limit still binds everywhere a resref
|
||||||
|
// really is a resref — see TestValidateLayoutRejectsLongResRefWhenItNamesAHAK.
|
||||||
|
func TestValidateLayoutAllowsLongModuleResRef(t *testing.T) {
|
||||||
|
root := t.TempDir()
|
||||||
|
mkdirAll(t, filepath.Join(root, "src"))
|
||||||
|
mkdirAll(t, filepath.Join(root, "build"))
|
||||||
|
|
||||||
|
proj := &Project{
|
||||||
|
Root: root,
|
||||||
|
Config: Config{
|
||||||
|
Module: ModuleConfig{Name: "Shadows Over Westgate", ResRef: "ShadowsOverWestgate"},
|
||||||
|
Paths: PathConfig{Source: "src", Build: "build"},
|
||||||
|
},
|
||||||
|
}
|
||||||
|
|
||||||
|
if err := proj.ValidateLayout(); err != nil {
|
||||||
|
t.Fatalf("ValidateLayout rejected a 19-character module file name: %v", err)
|
||||||
|
}
|
||||||
|
if got, want := filepath.Base(proj.ModuleArchivePath()), "ShadowsOverWestgate.mod"; got != want {
|
||||||
|
t.Fatalf("ModuleArchivePath() = %q, want %q", got, want)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// A module.resref that is not a usable file name is still rejected.
|
||||||
|
func TestValidateLayoutRejectsModuleResRefThatIsAPath(t *testing.T) {
|
||||||
|
root := t.TempDir()
|
||||||
|
mkdirAll(t, filepath.Join(root, "src"))
|
||||||
|
|
||||||
|
proj := &Project{
|
||||||
|
Root: root,
|
||||||
|
Config: Config{
|
||||||
|
Module: ModuleConfig{Name: "Test", ResRef: "../escape/mod"},
|
||||||
|
Paths: PathConfig{Source: "src", Build: "build"},
|
||||||
|
},
|
||||||
|
}
|
||||||
|
|
||||||
|
err := proj.ValidateLayout()
|
||||||
|
if err == nil {
|
||||||
|
t.Fatal("ValidateLayout accepted a module.resref containing a path")
|
||||||
|
}
|
||||||
|
if !strings.Contains(err.Error(), "module.resref") {
|
||||||
|
t.Fatalf("error does not name the offending field: %v", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// When a project declares no haks, the module resref becomes the name of the single
|
||||||
|
// generated HAK — and a HAK name IS a resref the engine loads. The limit applies
|
||||||
|
// there, so a long name is only allowed for projects that build no HAKs.
|
||||||
|
func TestValidateLayoutRejectsLongResRefWhenItNamesAHAK(t *testing.T) {
|
||||||
|
root := t.TempDir()
|
||||||
|
mkdirAll(t, filepath.Join(root, "src"))
|
||||||
|
mkdirAll(t, filepath.Join(root, "assets"))
|
||||||
|
|
||||||
|
proj := &Project{
|
||||||
|
Root: root,
|
||||||
|
Config: Config{
|
||||||
|
Module: ModuleConfig{Name: "Shadows Over Westgate", ResRef: "ShadowsOverWestgate"},
|
||||||
|
Paths: PathConfig{Source: "src", Assets: "assets", Build: "build"},
|
||||||
|
},
|
||||||
|
}
|
||||||
|
|
||||||
|
err := proj.ValidateLayout()
|
||||||
|
if err == nil {
|
||||||
|
t.Fatal("ValidateLayout accepted a 19-character name for a generated HAK")
|
||||||
|
}
|
||||||
|
if !strings.Contains(err.Error(), "16") {
|
||||||
|
t.Fatalf("error does not explain the resref limit: %v", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
// paths.build is an OUTPUT dir the builder creates (MkdirAll) before writing, so
|
// paths.build is an OUTPUT dir the builder creates (MkdirAll) before writing, so
|
||||||
// a bare clone with no build dir yet must still validate/build with no pre-step
|
// a bare clone with no build dir yet must still validate/build with no pre-step
|
||||||
// (R2/parity). Only a build path that exists but is not a directory is an error.
|
// (R2/parity). Only a build path that exists but is not a directory is an error.
|
||||||
|
|||||||
@@ -98,8 +98,9 @@ func buildGenerated2DAAssetGroup(p *project.Project, cfg project.GeneratedTopDat
|
|||||||
}
|
}
|
||||||
|
|
||||||
results := make([]Generated2DAAsset, 0, len(collected))
|
results := make([]Generated2DAAsset, 0, len(collected))
|
||||||
|
racialFeatRules := racialUsableFeatRules(collected)
|
||||||
for _, dataset := range collected {
|
for _, dataset := range collected {
|
||||||
compiled, err := resolveNativeDataset(dataset, keyToID, rowByKey, tableRegistry, nil, project.TopDataClassFeatInjectionConfig{}, nil)
|
compiled, err := resolveNativeDataset(dataset, keyToID, rowByKey, tableRegistry, nil, project.TopDataClassFeatInjectionConfig{}, racialFeatRules, nil)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return nil, err
|
return nil, err
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -530,6 +530,7 @@ func buildNativeUnchecked(p *project.Project, opts NativeBuildOptions, progress
|
|||||||
groupStats := nativeCompileGroupStats(collected)
|
groupStats := nativeCompileGroupStats(collected)
|
||||||
currentGroup := ""
|
currentGroup := ""
|
||||||
sidecars := newNativeSidecarCollector()
|
sidecars := newNativeSidecarCollector()
|
||||||
|
racialFeatRules := racialUsableFeatRules(collected)
|
||||||
for _, dataset := range collected {
|
for _, dataset := range collected {
|
||||||
group := nativeCompileGroup(dataset.Dataset.Name)
|
group := nativeCompileGroup(dataset.Dataset.Name)
|
||||||
if group != currentGroup {
|
if group != currentGroup {
|
||||||
@@ -542,7 +543,7 @@ func buildNativeUnchecked(p *project.Project, opts NativeBuildOptions, progress
|
|||||||
stats.SourceFragments,
|
stats.SourceFragments,
|
||||||
))
|
))
|
||||||
}
|
}
|
||||||
compiled, err := resolveNativeDataset(dataset, globalKeyToID, globalRowByKey, tableRegistry, compiler, p.EffectiveConfig().TopData.ClassFeatInjections, sidecars)
|
compiled, err := resolveNativeDataset(dataset, globalKeyToID, globalRowByKey, tableRegistry, compiler, p.EffectiveConfig().TopData.ClassFeatInjections, racialFeatRules, sidecars)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return BuildResult{}, err
|
return BuildResult{}, err
|
||||||
}
|
}
|
||||||
@@ -3996,13 +3997,13 @@ func normalizeGlobalReferenceID(value string) string {
|
|||||||
return parts[0] + ":" + strings.ReplaceAll(parts[1], "_", "")
|
return parts[0] + ":" + strings.ReplaceAll(parts[1], "_", "")
|
||||||
}
|
}
|
||||||
|
|
||||||
func resolveNativeDataset(dataset nativeCollectedDataset, keyToID map[string]int, globalRowByKey map[string]map[string]any, tableRegistry resolvedTableRegistry, compiler *tlkCompiler, classFeatInjections project.TopDataClassFeatInjectionConfig, sidecars *nativeSidecarCollector) (map[string]any, error) {
|
func resolveNativeDataset(dataset nativeCollectedDataset, keyToID map[string]int, globalRowByKey map[string]map[string]any, tableRegistry resolvedTableRegistry, compiler *tlkCompiler, classFeatInjections project.TopDataClassFeatInjectionConfig, racialFeatRules []project.TopDataClassFeatGlobalRule, sidecars *nativeSidecarCollector) (map[string]any, error) {
|
||||||
rows := dataset.Rows
|
rows := dataset.Rows
|
||||||
if strings.HasPrefix(filepath.ToSlash(dataset.Dataset.Name), "classes/feats/") {
|
if strings.HasPrefix(filepath.ToSlash(dataset.Dataset.Name), "classes/feats/") {
|
||||||
classKey := "classes:" + dataset.Dataset.Name[strings.LastIndex(dataset.Dataset.Name, "/")+1:]
|
classKey := "classes:" + dataset.Dataset.Name[strings.LastIndex(dataset.Dataset.Name, "/")+1:]
|
||||||
featSuccessors := buildFeatSuccessorsIndex(globalRowByKey, keyToID)
|
featSuccessors := buildFeatSuccessorsIndex(globalRowByKey, keyToID)
|
||||||
classSkills := buildClassSkillsIndex(tableRegistry, classKey)
|
classSkills := buildClassSkillsIndex(tableRegistry, classKey)
|
||||||
expanded, err := expandClassesFeatRows(rows, keyToID, globalRowByKey, featSuccessors, classSkills, globalRowByKey, classKey, classFeatInjections, !dataset.Dataset.HasGlobalInjections)
|
expanded, err := expandClassesFeatRows(rows, keyToID, globalRowByKey, featSuccessors, classSkills, globalRowByKey, classKey, classFeatInjections, racialFeatRules, !dataset.Dataset.HasGlobalInjections)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return nil, fmt.Errorf("dataset %s: %w", dataset.Dataset.Name, err)
|
return nil, fmt.Errorf("dataset %s: %w", dataset.Dataset.Name, err)
|
||||||
}
|
}
|
||||||
@@ -4051,12 +4052,18 @@ var (
|
|||||||
}
|
}
|
||||||
)
|
)
|
||||||
|
|
||||||
func expandClassesFeatRows(rows []map[string]any, keyToID map[string]int, rowByKey map[string]map[string]any, featSuccessors map[string]string, classSkills map[string]bool, allRowByKey map[string]map[string]any, classKey string, classFeatInjections project.TopDataClassFeatInjectionConfig, useConfiguredInjections bool) ([]map[string]any, error) {
|
func expandClassesFeatRows(rows []map[string]any, keyToID map[string]int, rowByKey map[string]map[string]any, featSuccessors map[string]string, classSkills map[string]bool, allRowByKey map[string]map[string]any, classKey string, classFeatInjections project.TopDataClassFeatInjectionConfig, racialFeatRules []project.TopDataClassFeatGlobalRule, useConfiguredInjections bool) ([]map[string]any, error) {
|
||||||
|
|
||||||
globalRules, classSkillRules := []project.TopDataClassFeatGlobalRule{}, []project.TopDataClassFeatMasterfeatRule{}
|
globalRules, classSkillRules := []project.TopDataClassFeatGlobalRule{}, []project.TopDataClassFeatMasterfeatRule{}
|
||||||
if useConfiguredInjections {
|
if useConfiguredInjections {
|
||||||
globalRules, classSkillRules = effectiveClassFeatInjectionRules(classFeatInjections)
|
globalRules, classSkillRules = effectiveClassFeatInjectionRules(classFeatInjections)
|
||||||
}
|
}
|
||||||
|
// Racial usable-feat rows are generated from the race feats tables, not the
|
||||||
|
// hand-authored class-feat injections, so they apply to every class table
|
||||||
|
// regardless of whether that dataset carries its own global.json (which is
|
||||||
|
// what gates useConfiguredInjections). Deduped below against rows already
|
||||||
|
// present, so a leftover hand row in global.json is a no-op, not a double.
|
||||||
|
globalRules = append(globalRules, racialFeatRules...)
|
||||||
injected := make([]map[string]any, 0, len(globalRules)+len(classSkillRules))
|
injected := make([]map[string]any, 0, len(globalRules)+len(classSkillRules))
|
||||||
presentRefIDs := make(map[string]struct{}, len(rows))
|
presentRefIDs := make(map[string]struct{}, len(rows))
|
||||||
for _, row := range rows {
|
for _, row := range rows {
|
||||||
@@ -4134,6 +4141,54 @@ func expandClassesFeatRows(rows []map[string]any, keyToID map[string]int, rowByK
|
|||||||
return combined, nil
|
return combined, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// racialUsableFeatRules builds one class-feat injection rule per feat marked
|
||||||
|
// UsableFeat=1 in any race feats table (race_feat_*.2da). Racial feats are
|
||||||
|
// granted by race, never by a class, so an activatable one needs a menu-only
|
||||||
|
// cls_feat row to reach the client radial: List=3 keeps it off every level-up
|
||||||
|
// selection list, GrantedOnLevel=99 sits above the level cap so no class ever
|
||||||
|
// actually grants it, and OnMenu=1 renders the button once the creature holds
|
||||||
|
// the feat (possession comes from chargen / the login racial-feat sync). This
|
||||||
|
// replaces the hand-maintained racial rows in classes/feats/global.json - mark
|
||||||
|
// UsableFeat in the race table and the radial row follows automatically.
|
||||||
|
func racialUsableFeatRules(collected []nativeCollectedDataset) []project.TopDataClassFeatGlobalRule {
|
||||||
|
seen := map[string]struct{}{}
|
||||||
|
rules := []project.TopDataClassFeatGlobalRule{}
|
||||||
|
for _, ds := range collected {
|
||||||
|
if !strings.HasPrefix(ds.Dataset.OutputName, "race_feat_") || !strings.HasSuffix(ds.Dataset.OutputName, ".2da") {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
for _, row := range ds.Rows {
|
||||||
|
if usable, err := asInt(fieldValue(row, "UsableFeat")); err != nil || usable != 1 {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
featRef, ok := row["FeatIndex"].(map[string]any)
|
||||||
|
if !ok {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
featID, _ := featRef["id"].(string)
|
||||||
|
if featID == "" {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
if _, dup := seen[featID]; dup {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
seen[featID] = struct{}{}
|
||||||
|
rules = append(rules, project.TopDataClassFeatGlobalRule{
|
||||||
|
Feat: featID,
|
||||||
|
List: "3",
|
||||||
|
GrantedOnLevel: "99",
|
||||||
|
OnMenu: "1",
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
// Discovery order across files and the dedup map are both unordered; sort so
|
||||||
|
// the injected rows (and the resulting 2DA row numbering) are deterministic.
|
||||||
|
slices.SortFunc(rules, func(a, b project.TopDataClassFeatGlobalRule) int {
|
||||||
|
return strings.Compare(a.Feat, b.Feat)
|
||||||
|
})
|
||||||
|
return rules
|
||||||
|
}
|
||||||
|
|
||||||
func effectiveClassFeatInjectionRules(config project.TopDataClassFeatInjectionConfig) ([]project.TopDataClassFeatGlobalRule, []project.TopDataClassFeatMasterfeatRule) {
|
func effectiveClassFeatInjectionRules(config project.TopDataClassFeatInjectionConfig) ([]project.TopDataClassFeatGlobalRule, []project.TopDataClassFeatMasterfeatRule) {
|
||||||
if len(config.GlobalFeats) == 0 && len(config.ClassSkillMasterfeats) == 0 {
|
if len(config.GlobalFeats) == 0 && len(config.ClassSkillMasterfeats) == 0 {
|
||||||
return defaultClassFeatGlobalRules, defaultClassFeatClassSkillMasterfeatRules
|
return defaultClassFeatGlobalRules, defaultClassFeatClassSkillMasterfeatRules
|
||||||
|
|||||||
@@ -0,0 +1,54 @@
|
|||||||
|
package topdata
|
||||||
|
|
||||||
|
import (
|
||||||
|
"reflect"
|
||||||
|
"testing"
|
||||||
|
|
||||||
|
"git.westgate.pw/ShadowsOverWestgate/sow-tools/internal/project"
|
||||||
|
)
|
||||||
|
|
||||||
|
func raceFeatDataset(output string, rows ...map[string]any) nativeCollectedDataset {
|
||||||
|
return nativeCollectedDataset{
|
||||||
|
Dataset: nativeDataset{OutputName: output},
|
||||||
|
Rows: rows,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func usableRow(featID string, usable any) map[string]any {
|
||||||
|
row := map[string]any{"FeatIndex": map[string]any{"id": featID}}
|
||||||
|
if usable != nil {
|
||||||
|
row["UsableFeat"] = usable
|
||||||
|
}
|
||||||
|
return row
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestRacialUsableFeatRules(t *testing.T) {
|
||||||
|
collected := []nativeCollectedDataset{
|
||||||
|
raceFeatDataset("race_feat_ddrw.2da",
|
||||||
|
usableRow("feat:keen_sense", nil), // passive, no UsableFeat -> skipped
|
||||||
|
usableRow("feat:darkvision", 1), // usable
|
||||||
|
usableRow("feat:use_poison", 0), // explicitly not usable -> skipped
|
||||||
|
usableRow("feat:drow/faerie_fire", "1"),// usable, string form
|
||||||
|
),
|
||||||
|
raceFeatDataset("race_feat_tief.2da",
|
||||||
|
usableRow("feat:darkvision", 1), // duplicate across races -> collapses to one
|
||||||
|
usableRow("feat:tiefling/darkness", 1),
|
||||||
|
),
|
||||||
|
raceFeatDataset("feat.2da", // not a race feats table -> ignored entirely
|
||||||
|
usableRow("feat:power_attack", 1),
|
||||||
|
),
|
||||||
|
}
|
||||||
|
|
||||||
|
got := racialUsableFeatRules(collected)
|
||||||
|
|
||||||
|
want := []project.TopDataClassFeatGlobalRule{
|
||||||
|
{Feat: "feat:darkvision", List: "3", GrantedOnLevel: "99", OnMenu: "1"},
|
||||||
|
{Feat: "feat:drow/faerie_fire", List: "3", GrantedOnLevel: "99", OnMenu: "1"},
|
||||||
|
{Feat: "feat:tiefling/darkness", List: "3", GrantedOnLevel: "99", OnMenu: "1"},
|
||||||
|
}
|
||||||
|
|
||||||
|
// Slice is sorted by Feat, so order is deterministic.
|
||||||
|
if !reflect.DeepEqual(got, want) {
|
||||||
|
t.Fatalf("rules: got %+v, want %+v", got, want)
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -1523,6 +1523,7 @@ func TestResolveNativeDatasetPreservesScalarTableReferenceBehavior(t *testing.T)
|
|||||||
nil,
|
nil,
|
||||||
project.TopDataClassFeatInjectionConfig{},
|
project.TopDataClassFeatInjectionConfig{},
|
||||||
nil,
|
nil,
|
||||||
|
nil,
|
||||||
)
|
)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatalf("resolveNativeDataset failed: %v", err)
|
t.Fatalf("resolveNativeDataset failed: %v", err)
|
||||||
|
|||||||
+122
-23
@@ -8,6 +8,7 @@ import (
|
|||||||
"fmt"
|
"fmt"
|
||||||
"html"
|
"html"
|
||||||
"io"
|
"io"
|
||||||
|
"maps"
|
||||||
"net/http"
|
"net/http"
|
||||||
"net/url"
|
"net/url"
|
||||||
"os"
|
"os"
|
||||||
@@ -52,7 +53,7 @@ type DeployWikiOptions struct {
|
|||||||
TitlePrefixMinLength int
|
TitlePrefixMinLength int
|
||||||
}
|
}
|
||||||
|
|
||||||
type deployResult struct {
|
type DeployResult struct {
|
||||||
LocalPages int
|
LocalPages int
|
||||||
Created int
|
Created int
|
||||||
Updated int
|
Updated int
|
||||||
@@ -63,6 +64,19 @@ type deployResult struct {
|
|||||||
Drifted int
|
Drifted int
|
||||||
Renamed int
|
Renamed int
|
||||||
Manifest string
|
Manifest string
|
||||||
|
// ResetPurged counts the deletions queued by --reset-managed-namespaces.
|
||||||
|
// They are also included in Stale and Purged, because callers warn about
|
||||||
|
// destructive policies in terms of the stale count: sow-topdata's
|
||||||
|
// deploy-wiki wrapper prints "of the pages counted as 'stale' below, they
|
||||||
|
// will be DELETED from NodeBB" directly above this block.
|
||||||
|
ResetPurged int
|
||||||
|
// ResetUnrecognized counts the subset of ResetPurged that the deploy
|
||||||
|
// manifest has no record of writing. Those deletions are the ones a
|
||||||
|
// re-seed cannot undo.
|
||||||
|
ResetUnrecognized int
|
||||||
|
// ResetSkipped counts reset targets NodeBB refused to delete, such as the
|
||||||
|
// wiki home topic, which the plugin excludes from tombstone and purge.
|
||||||
|
ResetSkipped int
|
||||||
}
|
}
|
||||||
|
|
||||||
type wikiDeployPage struct {
|
type wikiDeployPage struct {
|
||||||
@@ -100,6 +114,12 @@ type wikiDeployPlan struct {
|
|||||||
Content string
|
Content string
|
||||||
RemoteHash string
|
RemoteHash string
|
||||||
Title string
|
Title string
|
||||||
|
// Reset marks a purge queued by --reset-managed-namespaces rather than by
|
||||||
|
// stale computation over the manifest.
|
||||||
|
Reset bool
|
||||||
|
// Unrecognized marks a reset purge whose topic the manifest has no record
|
||||||
|
// of writing.
|
||||||
|
Unrecognized bool
|
||||||
}
|
}
|
||||||
|
|
||||||
type wikiNamespacesDocument struct {
|
type wikiNamespacesDocument struct {
|
||||||
@@ -116,7 +136,7 @@ type wikiNamespaceDeclaration struct {
|
|||||||
EditPolicy string `json:"edit_policy" yaml:"edit_policy"`
|
EditPolicy string `json:"edit_policy" yaml:"edit_policy"`
|
||||||
}
|
}
|
||||||
|
|
||||||
func DeployWikiWithOptions(p *project.Project, opts DeployWikiOptions, progress func(string)) (deployResult, error) {
|
func DeployWikiWithOptions(p *project.Project, opts DeployWikiOptions, progress func(string)) (DeployResult, error) {
|
||||||
if progress == nil {
|
if progress == nil {
|
||||||
progress = func(string) {}
|
progress = func(string) {}
|
||||||
}
|
}
|
||||||
@@ -132,19 +152,19 @@ func DeployWikiWithOptions(p *project.Project, opts DeployWikiOptions, progress
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
if _, err := os.Stat(opts.SourceDir); err != nil {
|
if _, err := os.Stat(opts.SourceDir); err != nil {
|
||||||
return deployResult{}, fmt.Errorf("wiki source directory not found: %w", err)
|
return DeployResult{}, fmt.Errorf("wiki source directory not found: %w", err)
|
||||||
}
|
}
|
||||||
if opts.Endpoint == "" {
|
if opts.Endpoint == "" {
|
||||||
return deployResult{}, errors.New("NODEBB_API_ENDPOINT is required")
|
return DeployResult{}, errors.New("NODEBB_API_ENDPOINT is required")
|
||||||
}
|
}
|
||||||
if opts.Token == "" {
|
if opts.Token == "" {
|
||||||
return deployResult{}, errors.New("NODEBB_API_TOKEN is required")
|
return DeployResult{}, errors.New("NODEBB_API_TOKEN is required")
|
||||||
}
|
}
|
||||||
if opts.Username != "" || opts.Password != "" || opts.NotesDelimiter != "" {
|
if opts.Username != "" || opts.Password != "" || opts.NotesDelimiter != "" {
|
||||||
return deployResult{}, errors.New("DokuWiki deployment options are no longer supported; use NodeBB endpoint, token, and category mappings")
|
return DeployResult{}, errors.New("DokuWiki deployment options are no longer supported; use NodeBB endpoint, token, and category mappings")
|
||||||
}
|
}
|
||||||
if opts.StalePolicy != "" && opts.StalePolicy != "report" && opts.StalePolicy != "archive" && opts.StalePolicy != "purge" {
|
if opts.StalePolicy != "" && opts.StalePolicy != "report" && opts.StalePolicy != "archive" && opts.StalePolicy != "purge" {
|
||||||
return deployResult{}, fmt.Errorf("wiki stale policy %q is not supported", opts.StalePolicy)
|
return DeployResult{}, fmt.Errorf("wiki stale policy %q is not supported", opts.StalePolicy)
|
||||||
}
|
}
|
||||||
if opts.TitlePrefixMinLength <= 0 {
|
if opts.TitlePrefixMinLength <= 0 {
|
||||||
opts.TitlePrefixMinLength = p.EffectiveConfig().TopData.Wiki.TitlePrefixMinLength
|
opts.TitlePrefixMinLength = p.EffectiveConfig().TopData.Wiki.TitlePrefixMinLength
|
||||||
@@ -154,7 +174,7 @@ func DeployWikiWithOptions(p *project.Project, opts DeployWikiOptions, progress
|
|||||||
if len(namespaces) == 0 {
|
if len(namespaces) == 0 {
|
||||||
declarations, err := loadWikiNamespaceDeclarations(p)
|
declarations, err := loadWikiNamespaceDeclarations(p)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return deployResult{}, err
|
return DeployResult{}, err
|
||||||
}
|
}
|
||||||
for _, declaration := range declarations {
|
for _, declaration := range declarations {
|
||||||
namespaces = append(namespaces, declaration.ID)
|
namespaces = append(namespaces, declaration.ID)
|
||||||
@@ -164,7 +184,7 @@ func DeployWikiWithOptions(p *project.Project, opts DeployWikiOptions, progress
|
|||||||
}
|
}
|
||||||
envCategories, err := categoryIDsFromNamespaceEnv(declarations)
|
envCategories, err := categoryIDsFromNamespaceEnv(declarations)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return deployResult{}, err
|
return DeployResult{}, err
|
||||||
}
|
}
|
||||||
if len(envCategories) > 0 {
|
if len(envCategories) > 0 {
|
||||||
merged := envCategories
|
merged := envCategories
|
||||||
@@ -183,7 +203,7 @@ func DeployWikiWithOptions(p *project.Project, opts DeployWikiOptions, progress
|
|||||||
progress(fmt.Sprintf("Collecting local wiki pages from %s", opts.SourceDir))
|
progress(fmt.Sprintf("Collecting local wiki pages from %s", opts.SourceDir))
|
||||||
pages, err := collectLocalPages(opts.SourceDir, opts.PageIndexPath, namespaces)
|
pages, err := collectLocalPages(opts.SourceDir, opts.PageIndexPath, namespaces)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return deployResult{}, err
|
return DeployResult{}, err
|
||||||
}
|
}
|
||||||
progress(fmt.Sprintf("Loaded %d local wiki page(s)", len(pages)))
|
progress(fmt.Sprintf("Loaded %d local wiki page(s)", len(pages)))
|
||||||
manifest := loadDeployManifest(manifestPath)
|
manifest := loadDeployManifest(manifestPath)
|
||||||
@@ -193,7 +213,7 @@ func DeployWikiWithOptions(p *project.Project, opts DeployWikiOptions, progress
|
|||||||
progress(fmt.Sprintf("Planning NodeBB wiki deploy for %d local page(s)", len(pages)))
|
progress(fmt.Sprintf("Planning NodeBB wiki deploy for %d local page(s)", len(pages)))
|
||||||
plans, result, nextManifest, err := planNodeBBDeploy(pages, manifest, opts, client, progress)
|
plans, result, nextManifest, err := planNodeBBDeploy(pages, manifest, opts, client, progress)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return deployResult{}, err
|
return DeployResult{}, err
|
||||||
}
|
}
|
||||||
result.LocalPages = len(pages)
|
result.LocalPages = len(pages)
|
||||||
result.Manifest = manifestPath
|
result.Manifest = manifestPath
|
||||||
@@ -212,6 +232,8 @@ func DeployWikiWithOptions(p *project.Project, opts DeployWikiOptions, progress
|
|||||||
}
|
}
|
||||||
orderedPlans := orderNodeBBDeployPlans(plans)
|
orderedPlans := orderNodeBBDeployPlans(plans)
|
||||||
progress(fmt.Sprintf("Executing NodeBB wiki actions: total %d, create %d, update %d, rename %d, archive %d, purge %d", len(orderedPlans), result.Created, result.Updated, result.Renamed, result.Archived, result.Purged))
|
progress(fmt.Sprintf("Executing NodeBB wiki actions: total %d, create %d, update %d, rename %d, archive %d, purge %d", len(orderedPlans), result.Created, result.Updated, result.Renamed, result.Archived, result.Purged))
|
||||||
|
resetPurgedByCID := map[int]int{}
|
||||||
|
resetSkippedByCID := map[int]int{}
|
||||||
for i, plan := range orderedPlans {
|
for i, plan := range orderedPlans {
|
||||||
if shouldReportNodeBBDeployActionProgress(i, len(orderedPlans)) {
|
if shouldReportNodeBBDeployActionProgress(i, len(orderedPlans)) {
|
||||||
progress(fmt.Sprintf("Executing NodeBB wiki action %d/%d: %s %s", i+1, len(orderedPlans), plan.Action, plan.Page.PageID))
|
progress(fmt.Sprintf("Executing NodeBB wiki action %d/%d: %s %s", i+1, len(orderedPlans), plan.Action, plan.Page.PageID))
|
||||||
@@ -242,19 +264,52 @@ func DeployWikiWithOptions(p *project.Project, opts DeployWikiOptions, progress
|
|||||||
return result, fmt.Errorf("deploy wiki page %q: update NodeBB post %d: %w", plan.Page.PageID, plan.Entry.PID, err)
|
return result, fmt.Errorf("deploy wiki page %q: update NodeBB post %d: %w", plan.Page.PageID, plan.Entry.PID, err)
|
||||||
}
|
}
|
||||||
case "archive":
|
case "archive":
|
||||||
|
// Archiving rewrites the page rather than removing it, so it goes
|
||||||
|
// through the ordinary post edit the wiki plugin allows; only
|
||||||
|
// delete, restore, and purge are reserved to the page actions.
|
||||||
if err := client.updatePost(plan.Entry.TID, plan.Entry.PID, plan.Content, summary); err != nil {
|
if err := client.updatePost(plan.Entry.TID, plan.Entry.PID, plan.Content, summary); err != nil {
|
||||||
return result, fmt.Errorf("deploy wiki page %q: archive NodeBB post %d: %w", plan.Page.PageID, plan.Entry.PID, err)
|
return result, fmt.Errorf("deploy wiki page %q: archive NodeBB post %d: %w", plan.Page.PageID, plan.Entry.PID, err)
|
||||||
}
|
}
|
||||||
case "purge":
|
case "purge":
|
||||||
if err := client.purgeTopic(plan.Entry.TID); err != nil {
|
if err := client.purgeWikiPage(plan.Entry.TID); err != nil {
|
||||||
|
// A namespace reset sweeps every topic in the category, so it
|
||||||
|
// can reach pages NodeBB will not delete at all — the wiki home
|
||||||
|
// topic above all. Skip those rather than abandoning the reset.
|
||||||
|
if plan.Reset && isNodeBBWikiPageUndeletable(err) {
|
||||||
|
result.ResetSkipped++
|
||||||
|
result.ResetPurged--
|
||||||
|
result.Purged--
|
||||||
|
result.Stale--
|
||||||
|
if plan.Unrecognized {
|
||||||
|
result.ResetUnrecognized--
|
||||||
|
}
|
||||||
|
resetSkippedByCID[plan.Entry.CID]++
|
||||||
|
progress(fmt.Sprintf("NodeBB refused to delete wiki topic %d during managed namespace reset; skipping it", plan.Entry.TID))
|
||||||
|
continue
|
||||||
|
}
|
||||||
return result, fmt.Errorf("deploy wiki page %q: purge NodeBB topic %d: %w", plan.Page.PageID, plan.Entry.TID, err)
|
return result, fmt.Errorf("deploy wiki page %q: purge NodeBB topic %d: %w", plan.Page.PageID, plan.Entry.TID, err)
|
||||||
}
|
}
|
||||||
|
if plan.Reset {
|
||||||
|
resetPurgedByCID[plan.Entry.CID]++
|
||||||
|
}
|
||||||
case "rename":
|
case "rename":
|
||||||
if err := client.renameWikiPage(plan.Entry.TID, plan.Entry.CID, plan.Title); err != nil {
|
if err := client.renameWikiPage(plan.Entry.TID, plan.Entry.CID, plan.Title); err != nil {
|
||||||
return result, fmt.Errorf("deploy wiki page %q: rename NodeBB topic %d to %q: %w", plan.Page.PageID, plan.Entry.TID, plan.Title, err)
|
return result, fmt.Errorf("deploy wiki page %q: rename NodeBB topic %d to %q: %w", plan.Page.PageID, plan.Entry.TID, plan.Title, err)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
// NodeBB answers 403 both for the pages it will never delete — the wiki
|
||||||
|
// home topic — and for a token without purge privileges, and the two are
|
||||||
|
// not distinguishable from the response. What tells them apart is scope: a
|
||||||
|
// category where nothing at all could be deleted is a privilege problem,
|
||||||
|
// not a home page. Reporting that as a completed reset would leave the
|
||||||
|
// manifest claiming a fresh start over pages that are all still there, so
|
||||||
|
// the next deploy would recreate every one of them as a duplicate.
|
||||||
|
for _, cid := range slices.Sorted(maps.Keys(resetSkippedByCID)) {
|
||||||
|
if resetPurgedByCID[cid] == 0 {
|
||||||
|
return result, fmt.Errorf("NodeBB refused every managed namespace reset deletion in category %d (%d topic(s)); check that the deploy token has wiki purge privileges there", cid, resetSkippedByCID[cid])
|
||||||
|
}
|
||||||
|
}
|
||||||
if err := saveDeployManifest(manifestPath, nextManifest); err != nil {
|
if err := saveDeployManifest(manifestPath, nextManifest); err != nil {
|
||||||
return result, err
|
return result, err
|
||||||
}
|
}
|
||||||
@@ -486,7 +541,7 @@ func isDir(path string) bool {
|
|||||||
return err == nil && info.IsDir()
|
return err == nil && info.IsDir()
|
||||||
}
|
}
|
||||||
|
|
||||||
func planNodeBBDeploy(pages map[string]wikiDeployPage, manifest wikiDeployManifest, opts DeployWikiOptions, client *nodeBBClient, progress func(string)) ([]wikiDeployPlan, deployResult, wikiDeployManifest, error) {
|
func planNodeBBDeploy(pages map[string]wikiDeployPage, manifest wikiDeployManifest, opts DeployWikiOptions, client *nodeBBClient, progress func(string)) ([]wikiDeployPlan, DeployResult, wikiDeployManifest, error) {
|
||||||
if progress == nil {
|
if progress == nil {
|
||||||
progress = func(string) {}
|
progress = func(string) {}
|
||||||
}
|
}
|
||||||
@@ -494,17 +549,23 @@ func planNodeBBDeploy(pages map[string]wikiDeployPage, manifest wikiDeployManife
|
|||||||
next := wikiDeployManifest{Version: "nodebb-v1", Pages: map[string]wikiDeployManifestPage{}}
|
next := wikiDeployManifest{Version: "nodebb-v1", Pages: map[string]wikiDeployManifestPage{}}
|
||||||
remotePagesByCID := map[int][]nodeBBWikiPage{}
|
remotePagesByCID := map[int][]nodeBBWikiPage{}
|
||||||
var plans []wikiDeployPlan
|
var plans []wikiDeployPlan
|
||||||
var result deployResult
|
var result DeployResult
|
||||||
if opts.ResetManagedNamespaces {
|
if opts.ResetManagedNamespaces {
|
||||||
if !opts.AllowCreates && len(pageIDs) > 0 {
|
if !opts.AllowCreates && len(pageIDs) > 0 {
|
||||||
return nil, result, next, errors.New("wiki managed namespace reset requires --create so current generated pages can be recreated")
|
return nil, result, next, errors.New("wiki managed namespace reset requires --create so current generated pages can be recreated")
|
||||||
}
|
}
|
||||||
resetPlans, purged, err := planManagedNamespaceReset(opts, client, progress)
|
resetPlans, purged, unrecognized, err := planManagedNamespaceReset(opts, manifest, client, progress)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return nil, result, next, err
|
return nil, result, next, err
|
||||||
}
|
}
|
||||||
plans = append(plans, resetPlans...)
|
plans = append(plans, resetPlans...)
|
||||||
result.Purged += purged
|
result.Purged += purged
|
||||||
|
// A namespace reset deletes remote pages the same way stale purge does,
|
||||||
|
// so it is counted as stale: that is the number callers word their
|
||||||
|
// destructive-policy warning around.
|
||||||
|
result.Stale += purged
|
||||||
|
result.ResetPurged = purged
|
||||||
|
result.ResetUnrecognized = unrecognized
|
||||||
manifest = wikiDeployManifest{Version: "nodebb-v1", Pages: map[string]wikiDeployManifestPage{}}
|
manifest = wikiDeployManifest{Version: "nodebb-v1", Pages: map[string]wikiDeployManifestPage{}}
|
||||||
for _, cid := range opts.CategoryIDs {
|
for _, cid := range opts.CategoryIDs {
|
||||||
if cid != 0 {
|
if cid != 0 {
|
||||||
@@ -682,7 +743,11 @@ func planNodeBBDeploy(pages map[string]wikiDeployPage, manifest wikiDeployManife
|
|||||||
return plans, result, next, nil
|
return plans, result, next, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
func planManagedNamespaceReset(opts DeployWikiOptions, client *nodeBBClient, progress func(string)) ([]wikiDeployPlan, int, error) {
|
// planManagedNamespaceReset queues a purge for every topic in each managed
|
||||||
|
// category, not only the ones the manifest says we wrote. The second return is
|
||||||
|
// the plan count; the third is how many of those topics the manifest has no
|
||||||
|
// record of, which is the subset a re-seed cannot put back.
|
||||||
|
func planManagedNamespaceReset(opts DeployWikiOptions, manifest wikiDeployManifest, client *nodeBBClient, progress func(string)) ([]wikiDeployPlan, int, int, error) {
|
||||||
namespaces := slices.Clone(opts.Namespaces)
|
namespaces := slices.Clone(opts.Namespaces)
|
||||||
if len(namespaces) == 0 {
|
if len(namespaces) == 0 {
|
||||||
for namespace := range opts.CategoryIDs {
|
for namespace := range opts.CategoryIDs {
|
||||||
@@ -691,7 +756,15 @@ func planManagedNamespaceReset(opts DeployWikiOptions, client *nodeBBClient, pro
|
|||||||
}
|
}
|
||||||
slices.Sort(namespaces)
|
slices.Sort(namespaces)
|
||||||
|
|
||||||
|
manifestTIDs := map[int]struct{}{}
|
||||||
|
for _, entry := range manifest.Pages {
|
||||||
|
if entry.TID != 0 {
|
||||||
|
manifestTIDs[entry.TID] = struct{}{}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
seenTIDs := map[int]struct{}{}
|
seenTIDs := map[int]struct{}{}
|
||||||
|
unrecognized := 0
|
||||||
var plans []wikiDeployPlan
|
var plans []wikiDeployPlan
|
||||||
for _, namespace := range namespaces {
|
for _, namespace := range namespaces {
|
||||||
cid := opts.CategoryIDs[namespace]
|
cid := opts.CategoryIDs[namespace]
|
||||||
@@ -701,7 +774,7 @@ func planManagedNamespaceReset(opts DeployWikiOptions, client *nodeBBClient, pro
|
|||||||
progress(fmt.Sprintf("Listing NodeBB wiki namespace category %d for managed reset", cid))
|
progress(fmt.Sprintf("Listing NodeBB wiki namespace category %d for managed reset", cid))
|
||||||
remotePages, err := client.listNamespacePages(cid)
|
remotePages, err := client.listNamespacePages(cid)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return nil, 0, err
|
return nil, 0, 0, err
|
||||||
}
|
}
|
||||||
slices.SortFunc(remotePages, func(a, b nodeBBWikiPage) int {
|
slices.SortFunc(remotePages, func(a, b nodeBBWikiPage) int {
|
||||||
return a.TID - b.TID
|
return a.TID - b.TID
|
||||||
@@ -714,6 +787,10 @@ func planManagedNamespaceReset(opts DeployWikiOptions, client *nodeBBClient, pro
|
|||||||
continue
|
continue
|
||||||
}
|
}
|
||||||
seenTIDs[remotePage.TID] = struct{}{}
|
seenTIDs[remotePage.TID] = struct{}{}
|
||||||
|
_, known := manifestTIDs[remotePage.TID]
|
||||||
|
if !known {
|
||||||
|
unrecognized++
|
||||||
|
}
|
||||||
title := strings.TrimSpace(remotePage.Title)
|
title := strings.TrimSpace(remotePage.Title)
|
||||||
if title == "" {
|
if title == "" {
|
||||||
title = strings.TrimSpace(remotePage.TitleLeaf)
|
title = strings.TrimSpace(remotePage.TitleLeaf)
|
||||||
@@ -723,10 +800,12 @@ func planManagedNamespaceReset(opts DeployWikiOptions, client *nodeBBClient, pro
|
|||||||
Page: wikiDeployPage{PageID: pageID, Title: title, Namespace: namespace},
|
Page: wikiDeployPage{PageID: pageID, Title: title, Namespace: namespace},
|
||||||
Entry: wikiDeployManifestPage{TID: remotePage.TID, CID: cid, Namespace: namespace, Title: title},
|
Entry: wikiDeployManifestPage{TID: remotePage.TID, CID: cid, Namespace: namespace, Title: title},
|
||||||
Action: "purge",
|
Action: "purge",
|
||||||
|
Reset: true,
|
||||||
|
Unrecognized: !known,
|
||||||
})
|
})
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
return plans, len(plans), nil
|
return plans, len(plans), unrecognized, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
func findMappedRemoteTopic(entry wikiDeployManifestPage, client *nodeBBClient) (nodeBBPost, bool, error) {
|
func findMappedRemoteTopic(entry wikiDeployManifestPage, client *nodeBBClient) (nodeBBPost, bool, error) {
|
||||||
@@ -1611,17 +1690,37 @@ func (c *nodeBBClient) renameWikiPage(tid, cid int, title string) error {
|
|||||||
return c.request("PUT", "/api/v3/plugins/westgate-wiki/page/move", body, nil)
|
return c.request("PUT", "/api/v3/plugins/westgate-wiki/page/move", body, nil)
|
||||||
}
|
}
|
||||||
|
|
||||||
func (c *nodeBBClient) purgeTopic(tid int) error {
|
// purgeWikiPage deletes a wiki topic through the wiki plugin's own page
|
||||||
|
// actions. The core DELETE /api/v3/topics/{tid} route is refused for topics in
|
||||||
|
// wiki categories, because revision history is plugin-owned, so a purge has to
|
||||||
|
// tombstone the page first and then hard-purge it.
|
||||||
|
func (c *nodeBBClient) purgeWikiPage(tid int) error {
|
||||||
if tid == 0 {
|
if tid == 0 {
|
||||||
return fmt.Errorf("NodeBB topic purge requires topic id")
|
return fmt.Errorf("NodeBB wiki page purge requires topic id")
|
||||||
}
|
}
|
||||||
err := c.request(http.MethodDelete, fmt.Sprintf("/api/v3/topics/%d", tid), nil, nil)
|
body := map[string]any{"tid": tid}
|
||||||
var httpErr nodeBBHTTPError
|
if err := c.request(http.MethodPut, "/api/v3/plugins/westgate-wiki/page/tombstone", body, nil); err != nil {
|
||||||
if errors.As(err, &httpErr) && (httpErr.Status == http.StatusNotFound || httpErr.Status == http.StatusGone) {
|
if isNodeBBMissingResource(err) {
|
||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
|
if err := c.request(http.MethodDelete, "/api/v3/plugins/westgate-wiki/page/hard-purge", body, nil); err != nil {
|
||||||
|
if isNodeBBMissingResource(err) {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// isNodeBBWikiPageUndeletable reports whether NodeBB refused to delete the page
|
||||||
|
// outright rather than failing transiently. The wiki home topic answers this
|
||||||
|
// way: the plugin excludes it from tombstone, restore, and purge alike.
|
||||||
|
func isNodeBBWikiPageUndeletable(err error) bool {
|
||||||
|
var httpErr nodeBBHTTPError
|
||||||
|
return errors.As(err, &httpErr) && httpErr.Status == http.StatusForbidden
|
||||||
|
}
|
||||||
|
|
||||||
func (c *nodeBBClient) acquireEditLock(tid int) (nodeBBEditLock, error) {
|
func (c *nodeBBClient) acquireEditLock(tid int) (nodeBBEditLock, error) {
|
||||||
if tid == 0 {
|
if tid == 0 {
|
||||||
|
|||||||
@@ -2,6 +2,7 @@ package topdata
|
|||||||
|
|
||||||
import (
|
import (
|
||||||
"encoding/json"
|
"encoding/json"
|
||||||
|
"fmt"
|
||||||
"net/http"
|
"net/http"
|
||||||
"net/http/httptest"
|
"net/http/httptest"
|
||||||
"os"
|
"os"
|
||||||
@@ -35,7 +36,7 @@ func TestDeployWikiDryRunDoesNotWriteRemoteOrManifest(t *testing.T) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
updateCalls := 0
|
updateCalls := 0
|
||||||
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
server := newFakeNodeBB(t, func(w http.ResponseWriter, r *http.Request) {
|
||||||
if got := r.Header.Get("Authorization"); got != "Bearer nodebb-token" {
|
if got := r.Header.Get("Authorization"); got != "Bearer nodebb-token" {
|
||||||
t.Fatalf("unexpected authorization header %q", got)
|
t.Fatalf("unexpected authorization header %q", got)
|
||||||
}
|
}
|
||||||
@@ -50,7 +51,7 @@ func TestDeployWikiDryRunDoesNotWriteRemoteOrManifest(t *testing.T) {
|
|||||||
default:
|
default:
|
||||||
t.Fatalf("unexpected request %s %s", r.Method, r.URL.Path)
|
t.Fatalf("unexpected request %s %s", r.Method, r.URL.Path)
|
||||||
}
|
}
|
||||||
}))
|
})
|
||||||
defer server.Close()
|
defer server.Close()
|
||||||
|
|
||||||
result, err := DeployWikiWithOptions(&project.Project{}, DeployWikiOptions{
|
result, err := DeployWikiWithOptions(&project.Project{}, DeployWikiOptions{
|
||||||
@@ -92,7 +93,7 @@ func TestDeployWikiReportsPlanningProgressBeforeRemoteWork(t *testing.T) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
progress := []string{}
|
progress := []string{}
|
||||||
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
server := newFakeNodeBB(t, func(w http.ResponseWriter, r *http.Request) {
|
||||||
switch {
|
switch {
|
||||||
case r.Method == http.MethodGet && r.URL.Path == "/api/v3/plugins/westgate-wiki/namespace/3/pages":
|
case r.Method == http.MethodGet && r.URL.Path == "/api/v3/plugins/westgate-wiki/namespace/3/pages":
|
||||||
w.Header().Set("Content-Type", "application/json")
|
w.Header().Set("Content-Type", "application/json")
|
||||||
@@ -100,7 +101,7 @@ func TestDeployWikiReportsPlanningProgressBeforeRemoteWork(t *testing.T) {
|
|||||||
default:
|
default:
|
||||||
t.Fatalf("unexpected request %s %s", r.Method, r.URL.String())
|
t.Fatalf("unexpected request %s %s", r.Method, r.URL.String())
|
||||||
}
|
}
|
||||||
}))
|
})
|
||||||
defer server.Close()
|
defer server.Close()
|
||||||
|
|
||||||
_, err := DeployWikiWithOptions(&project.Project{}, DeployWikiOptions{
|
_, err := DeployWikiWithOptions(&project.Project{}, DeployWikiOptions{
|
||||||
@@ -174,7 +175,7 @@ func TestDeployWikiReportsLiveExecutionProgress(t *testing.T) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
progress := []string{}
|
progress := []string{}
|
||||||
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
server := newFakeNodeBB(t, func(w http.ResponseWriter, r *http.Request) {
|
||||||
switch {
|
switch {
|
||||||
case r.Method == http.MethodGet && r.URL.Path == "/api/v3/topics/7":
|
case r.Method == http.MethodGet && r.URL.Path == "/api/v3/topics/7":
|
||||||
w.Header().Set("Content-Type", "application/json")
|
w.Header().Set("Content-Type", "application/json")
|
||||||
@@ -194,7 +195,7 @@ func TestDeployWikiReportsLiveExecutionProgress(t *testing.T) {
|
|||||||
default:
|
default:
|
||||||
t.Fatalf("unexpected request %s %s", r.Method, r.URL.String())
|
t.Fatalf("unexpected request %s %s", r.Method, r.URL.String())
|
||||||
}
|
}
|
||||||
}))
|
})
|
||||||
defer server.Close()
|
defer server.Close()
|
||||||
|
|
||||||
result, err := DeployWikiWithOptions(&project.Project{}, DeployWikiOptions{
|
result, err := DeployWikiWithOptions(&project.Project{}, DeployWikiOptions{
|
||||||
@@ -228,7 +229,7 @@ func TestDeployWikiReportsLiveExecutionProgress(t *testing.T) {
|
|||||||
|
|
||||||
func TestNodeBBNamespacePaginationStopsWhenRepeatedCursorReturnsNoNewPages(t *testing.T) {
|
func TestNodeBBNamespacePaginationStopsWhenRepeatedCursorReturnsNoNewPages(t *testing.T) {
|
||||||
requests := 0
|
requests := 0
|
||||||
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
server := newFakeNodeBB(t, func(w http.ResponseWriter, r *http.Request) {
|
||||||
requests++
|
requests++
|
||||||
w.Header().Set("Content-Type", "application/json")
|
w.Header().Set("Content-Type", "application/json")
|
||||||
_ = json.NewEncoder(w).Encode(map[string]any{
|
_ = json.NewEncoder(w).Encode(map[string]any{
|
||||||
@@ -240,7 +241,7 @@ func TestNodeBBNamespacePaginationStopsWhenRepeatedCursorReturnsNoNewPages(t *te
|
|||||||
"nextCursor": "same-cursor",
|
"nextCursor": "same-cursor",
|
||||||
},
|
},
|
||||||
})
|
})
|
||||||
}))
|
})
|
||||||
defer server.Close()
|
defer server.Close()
|
||||||
|
|
||||||
client := newNodeBBClient(server.URL, "nodebb-token")
|
client := newNodeBBClient(server.URL, "nodebb-token")
|
||||||
@@ -258,7 +259,7 @@ func TestNodeBBNamespacePaginationStopsWhenRepeatedCursorReturnsNoNewPages(t *te
|
|||||||
|
|
||||||
func TestNodeBBNamespacePaginationRejectsRepeatedCursorWithNewPages(t *testing.T) {
|
func TestNodeBBNamespacePaginationRejectsRepeatedCursorWithNewPages(t *testing.T) {
|
||||||
requests := 0
|
requests := 0
|
||||||
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
server := newFakeNodeBB(t, func(w http.ResponseWriter, r *http.Request) {
|
||||||
requests++
|
requests++
|
||||||
w.Header().Set("Content-Type", "application/json")
|
w.Header().Set("Content-Type", "application/json")
|
||||||
pages := []map[string]any{
|
pages := []map[string]any{
|
||||||
@@ -276,7 +277,7 @@ func TestNodeBBNamespacePaginationRejectsRepeatedCursorWithNewPages(t *testing.T
|
|||||||
"nextCursor": "same-cursor",
|
"nextCursor": "same-cursor",
|
||||||
},
|
},
|
||||||
})
|
})
|
||||||
}))
|
})
|
||||||
defer server.Close()
|
defer server.Close()
|
||||||
|
|
||||||
client := newNodeBBClient(server.URL, "nodebb-token")
|
client := newNodeBBClient(server.URL, "nodebb-token")
|
||||||
@@ -480,7 +481,7 @@ func TestDeployWikiDryRunReadoptsMissingMappedPost(t *testing.T) {
|
|||||||
t.Fatalf("write deploy manifest: %v", err)
|
t.Fatalf("write deploy manifest: %v", err)
|
||||||
}
|
}
|
||||||
|
|
||||||
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
server := newFakeNodeBB(t, func(w http.ResponseWriter, r *http.Request) {
|
||||||
switch {
|
switch {
|
||||||
case r.Method == http.MethodGet && r.URL.Path == "/api/v3/posts/42":
|
case r.Method == http.MethodGet && r.URL.Path == "/api/v3/posts/42":
|
||||||
w.Header().Set("Content-Type", "application/json")
|
w.Header().Set("Content-Type", "application/json")
|
||||||
@@ -508,7 +509,7 @@ func TestDeployWikiDryRunReadoptsMissingMappedPost(t *testing.T) {
|
|||||||
default:
|
default:
|
||||||
t.Fatalf("unexpected request %s %s", r.Method, r.URL.String())
|
t.Fatalf("unexpected request %s %s", r.Method, r.URL.String())
|
||||||
}
|
}
|
||||||
}))
|
})
|
||||||
defer server.Close()
|
defer server.Close()
|
||||||
|
|
||||||
result, err := DeployWikiWithOptions(&project.Project{}, DeployWikiOptions{
|
result, err := DeployWikiWithOptions(&project.Project{}, DeployWikiOptions{
|
||||||
@@ -731,7 +732,7 @@ func TestDeployWikiCreatesNodeBBTopicAndWritesManifest(t *testing.T) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
createCalls := 0
|
createCalls := 0
|
||||||
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
server := newFakeNodeBB(t, func(w http.ResponseWriter, r *http.Request) {
|
||||||
if got := r.Header.Get("Authorization"); got != "Bearer nodebb-token" {
|
if got := r.Header.Get("Authorization"); got != "Bearer nodebb-token" {
|
||||||
t.Fatalf("unexpected authorization header %q", got)
|
t.Fatalf("unexpected authorization header %q", got)
|
||||||
}
|
}
|
||||||
@@ -769,7 +770,7 @@ func TestDeployWikiCreatesNodeBBTopicAndWritesManifest(t *testing.T) {
|
|||||||
},
|
},
|
||||||
},
|
},
|
||||||
})
|
})
|
||||||
}))
|
})
|
||||||
defer server.Close()
|
defer server.Close()
|
||||||
|
|
||||||
manifestPath := filepath.Join(root, "deploy-manifest.json")
|
manifestPath := filepath.Join(root, "deploy-manifest.json")
|
||||||
@@ -848,7 +849,7 @@ func TestDeployWikiRepairsManifestedPageMissingSourceContentSync(t *testing.T) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
updateCalls := 0
|
updateCalls := 0
|
||||||
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
server := newFakeNodeBB(t, func(w http.ResponseWriter, r *http.Request) {
|
||||||
switch {
|
switch {
|
||||||
case r.Method == http.MethodGet && r.URL.Path == "/api/v3/posts/42":
|
case r.Method == http.MethodGet && r.URL.Path == "/api/v3/posts/42":
|
||||||
w.Header().Set("Content-Type", "application/json")
|
w.Header().Set("Content-Type", "application/json")
|
||||||
@@ -876,7 +877,7 @@ func TestDeployWikiRepairsManifestedPageMissingSourceContentSync(t *testing.T) {
|
|||||||
default:
|
default:
|
||||||
t.Fatalf("unexpected request %s %s", r.Method, r.URL.String())
|
t.Fatalf("unexpected request %s %s", r.Method, r.URL.String())
|
||||||
}
|
}
|
||||||
}))
|
})
|
||||||
defer server.Close()
|
defer server.Close()
|
||||||
|
|
||||||
result, err := DeployWikiWithOptions(&project.Project{}, DeployWikiOptions{
|
result, err := DeployWikiWithOptions(&project.Project{}, DeployWikiOptions{
|
||||||
@@ -951,7 +952,7 @@ func TestDeployWikiCreatesNodeBBTopicWithoutFallbackForDefaultThreeCharacterTitl
|
|||||||
t.Fatalf("write source page: %v", err)
|
t.Fatalf("write source page: %v", err)
|
||||||
}
|
}
|
||||||
|
|
||||||
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
server := newFakeNodeBB(t, func(w http.ResponseWriter, r *http.Request) {
|
||||||
if r.Method == http.MethodGet && r.URL.Path == "/api/v3/plugins/westgate-wiki/namespace/5/pages" {
|
if r.Method == http.MethodGet && r.URL.Path == "/api/v3/plugins/westgate-wiki/namespace/5/pages" {
|
||||||
w.Header().Set("Content-Type", "application/json")
|
w.Header().Set("Content-Type", "application/json")
|
||||||
_ = json.NewEncoder(w).Encode(map[string]any{"response": map[string]any{"pages": []any{}, "hasMore": false}})
|
_ = json.NewEncoder(w).Encode(map[string]any{"response": map[string]any{"pages": []any{}, "hasMore": false}})
|
||||||
@@ -983,7 +984,7 @@ func TestDeployWikiCreatesNodeBBTopicWithoutFallbackForDefaultThreeCharacterTitl
|
|||||||
},
|
},
|
||||||
},
|
},
|
||||||
})
|
})
|
||||||
}))
|
})
|
||||||
defer server.Close()
|
defer server.Close()
|
||||||
|
|
||||||
_, err := DeployWikiWithOptions(&project.Project{}, DeployWikiOptions{
|
_, err := DeployWikiWithOptions(&project.Project{}, DeployWikiOptions{
|
||||||
@@ -1010,7 +1011,7 @@ func TestDeployWikiCreatesNodeBBTopicWithFallbackForTitleShorterThanConfiguredMi
|
|||||||
t.Fatalf("write source page: %v", err)
|
t.Fatalf("write source page: %v", err)
|
||||||
}
|
}
|
||||||
|
|
||||||
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
server := newFakeNodeBB(t, func(w http.ResponseWriter, r *http.Request) {
|
||||||
if r.Method == http.MethodGet && r.URL.Path == "/api/v3/plugins/westgate-wiki/namespace/5/pages" {
|
if r.Method == http.MethodGet && r.URL.Path == "/api/v3/plugins/westgate-wiki/namespace/5/pages" {
|
||||||
w.Header().Set("Content-Type", "application/json")
|
w.Header().Set("Content-Type", "application/json")
|
||||||
_ = json.NewEncoder(w).Encode(map[string]any{"response": map[string]any{"pages": []any{}, "hasMore": false}})
|
_ = json.NewEncoder(w).Encode(map[string]any{"response": map[string]any{"pages": []any{}, "hasMore": false}})
|
||||||
@@ -1038,7 +1039,7 @@ func TestDeployWikiCreatesNodeBBTopicWithFallbackForTitleShorterThanConfiguredMi
|
|||||||
},
|
},
|
||||||
},
|
},
|
||||||
})
|
})
|
||||||
}))
|
})
|
||||||
defer server.Close()
|
defer server.Close()
|
||||||
|
|
||||||
_, err := DeployWikiWithOptions(&project.Project{}, DeployWikiOptions{
|
_, err := DeployWikiWithOptions(&project.Project{}, DeployWikiOptions{
|
||||||
@@ -1080,7 +1081,7 @@ func TestDeployWikiRenamesExistingPrefixedTopicWhenTitleIsLongEnough(t *testing.
|
|||||||
}
|
}
|
||||||
|
|
||||||
renameCalls := 0
|
renameCalls := 0
|
||||||
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
server := newFakeNodeBB(t, func(w http.ResponseWriter, r *http.Request) {
|
||||||
switch {
|
switch {
|
||||||
case r.Method == http.MethodGet && r.URL.Path == "/api/v3/topics/7":
|
case r.Method == http.MethodGet && r.URL.Path == "/api/v3/topics/7":
|
||||||
w.Header().Set("Content-Type", "application/json")
|
w.Header().Set("Content-Type", "application/json")
|
||||||
@@ -1109,7 +1110,7 @@ func TestDeployWikiRenamesExistingPrefixedTopicWhenTitleIsLongEnough(t *testing.
|
|||||||
default:
|
default:
|
||||||
t.Fatalf("unexpected request %s %s", r.Method, r.URL.String())
|
t.Fatalf("unexpected request %s %s", r.Method, r.URL.String())
|
||||||
}
|
}
|
||||||
}))
|
})
|
||||||
defer server.Close()
|
defer server.Close()
|
||||||
|
|
||||||
result, err := DeployWikiWithOptions(&project.Project{}, DeployWikiOptions{
|
result, err := DeployWikiWithOptions(&project.Project{}, DeployWikiOptions{
|
||||||
@@ -1166,7 +1167,7 @@ func TestDeployWikiRenamesManagedTopicWhenGeneratedTitleChanges(t *testing.T) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
renameCalls := 0
|
renameCalls := 0
|
||||||
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
server := newFakeNodeBB(t, func(w http.ResponseWriter, r *http.Request) {
|
||||||
switch {
|
switch {
|
||||||
case r.Method == http.MethodGet && r.URL.Path == "/api/v3/topics/7":
|
case r.Method == http.MethodGet && r.URL.Path == "/api/v3/topics/7":
|
||||||
w.Header().Set("Content-Type", "application/json")
|
w.Header().Set("Content-Type", "application/json")
|
||||||
@@ -1195,7 +1196,7 @@ func TestDeployWikiRenamesManagedTopicWhenGeneratedTitleChanges(t *testing.T) {
|
|||||||
default:
|
default:
|
||||||
t.Fatalf("unexpected request %s %s", r.Method, r.URL.String())
|
t.Fatalf("unexpected request %s %s", r.Method, r.URL.String())
|
||||||
}
|
}
|
||||||
}))
|
})
|
||||||
defer server.Close()
|
defer server.Close()
|
||||||
|
|
||||||
result, err := DeployWikiWithOptions(&project.Project{}, DeployWikiOptions{
|
result, err := DeployWikiWithOptions(&project.Project{}, DeployWikiOptions{
|
||||||
@@ -1258,9 +1259,9 @@ func TestDeployWikiDoesNotRenameHeadinglessPageToPageIDFallback(t *testing.T) {
|
|||||||
t.Fatalf("write deploy manifest: %v", err)
|
t.Fatalf("write deploy manifest: %v", err)
|
||||||
}
|
}
|
||||||
|
|
||||||
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
server := newFakeNodeBB(t, func(w http.ResponseWriter, r *http.Request) {
|
||||||
t.Fatalf("headingless unchanged page must not call NodeBB, got %s %s", r.Method, r.URL.String())
|
t.Fatalf("headingless unchanged page must not call NodeBB, got %s %s", r.Method, r.URL.String())
|
||||||
}))
|
})
|
||||||
defer server.Close()
|
defer server.Close()
|
||||||
|
|
||||||
result, err := DeployWikiWithOptions(&project.Project{}, DeployWikiOptions{
|
result, err := DeployWikiWithOptions(&project.Project{}, DeployWikiOptions{
|
||||||
@@ -1320,7 +1321,7 @@ func TestDeployWikiRenamesBrokenHeadinglessFallbackTitleBackToPageIndexTitle(t *
|
|||||||
}
|
}
|
||||||
|
|
||||||
renameCalls := 0
|
renameCalls := 0
|
||||||
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
server := newFakeNodeBB(t, func(w http.ResponseWriter, r *http.Request) {
|
||||||
switch {
|
switch {
|
||||||
case r.Method == http.MethodGet && r.URL.Path == "/api/v3/topics/7":
|
case r.Method == http.MethodGet && r.URL.Path == "/api/v3/topics/7":
|
||||||
w.Header().Set("Content-Type", "application/json")
|
w.Header().Set("Content-Type", "application/json")
|
||||||
@@ -1345,7 +1346,7 @@ func TestDeployWikiRenamesBrokenHeadinglessFallbackTitleBackToPageIndexTitle(t *
|
|||||||
default:
|
default:
|
||||||
t.Fatalf("unexpected request %s %s", r.Method, r.URL.String())
|
t.Fatalf("unexpected request %s %s", r.Method, r.URL.String())
|
||||||
}
|
}
|
||||||
}))
|
})
|
||||||
defer server.Close()
|
defer server.Close()
|
||||||
|
|
||||||
result, err := DeployWikiWithOptions(&project.Project{}, DeployWikiOptions{
|
result, err := DeployWikiWithOptions(&project.Project{}, DeployWikiOptions{
|
||||||
@@ -1396,7 +1397,7 @@ func TestDeployWikiAdoptsExistingNodeBBPageWhenManifestIsMissingWithoutCreate(t
|
|||||||
|
|
||||||
createCalls := 0
|
createCalls := 0
|
||||||
var updated string
|
var updated string
|
||||||
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
server := newFakeNodeBB(t, func(w http.ResponseWriter, r *http.Request) {
|
||||||
switch {
|
switch {
|
||||||
case r.Method == http.MethodGet && r.URL.Path == "/api/v3/plugins/westgate-wiki/namespace/9/pages":
|
case r.Method == http.MethodGet && r.URL.Path == "/api/v3/plugins/westgate-wiki/namespace/9/pages":
|
||||||
w.Header().Set("Content-Type", "application/json")
|
w.Header().Set("Content-Type", "application/json")
|
||||||
@@ -1442,7 +1443,7 @@ func TestDeployWikiAdoptsExistingNodeBBPageWhenManifestIsMissingWithoutCreate(t
|
|||||||
default:
|
default:
|
||||||
t.Fatalf("unexpected request %s %s", r.Method, r.URL.String())
|
t.Fatalf("unexpected request %s %s", r.Method, r.URL.String())
|
||||||
}
|
}
|
||||||
}))
|
})
|
||||||
defer server.Close()
|
defer server.Close()
|
||||||
|
|
||||||
manifestPath := filepath.Join(root, "deploy-manifest.json")
|
manifestPath := filepath.Join(root, "deploy-manifest.json")
|
||||||
@@ -1509,7 +1510,7 @@ func TestDeployWikiMergesHTMLManagedAndManualRegions(t *testing.T) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
var updated string
|
var updated string
|
||||||
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
server := newFakeNodeBB(t, func(w http.ResponseWriter, r *http.Request) {
|
||||||
switch {
|
switch {
|
||||||
case r.Method == http.MethodGet && r.URL.Path == "/api/v3/posts/42":
|
case r.Method == http.MethodGet && r.URL.Path == "/api/v3/posts/42":
|
||||||
w.Header().Set("Content-Type", "application/json")
|
w.Header().Set("Content-Type", "application/json")
|
||||||
@@ -1533,7 +1534,7 @@ func TestDeployWikiMergesHTMLManagedAndManualRegions(t *testing.T) {
|
|||||||
default:
|
default:
|
||||||
t.Fatalf("unexpected request %s %s", r.Method, r.URL.Path)
|
t.Fatalf("unexpected request %s %s", r.Method, r.URL.Path)
|
||||||
}
|
}
|
||||||
}))
|
})
|
||||||
defer server.Close()
|
defer server.Close()
|
||||||
|
|
||||||
result, err := DeployWikiWithOptions(&project.Project{}, DeployWikiOptions{
|
result, err := DeployWikiWithOptions(&project.Project{}, DeployWikiOptions{
|
||||||
@@ -1589,7 +1590,7 @@ func TestDeployWikiUpdateAcquiresWestgateWikiEditLock(t *testing.T) {
|
|||||||
|
|
||||||
lockAcquired := false
|
lockAcquired := false
|
||||||
var updateToken string
|
var updateToken string
|
||||||
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
server := newFakeNodeBB(t, func(w http.ResponseWriter, r *http.Request) {
|
||||||
switch {
|
switch {
|
||||||
case r.Method == http.MethodGet && r.URL.Path == "/api/v3/posts/42":
|
case r.Method == http.MethodGet && r.URL.Path == "/api/v3/posts/42":
|
||||||
w.Header().Set("Content-Type", "application/json")
|
w.Header().Set("Content-Type", "application/json")
|
||||||
@@ -1621,7 +1622,7 @@ func TestDeployWikiUpdateAcquiresWestgateWikiEditLock(t *testing.T) {
|
|||||||
default:
|
default:
|
||||||
t.Fatalf("unexpected request %s %s", r.Method, r.URL.Path)
|
t.Fatalf("unexpected request %s %s", r.Method, r.URL.Path)
|
||||||
}
|
}
|
||||||
}))
|
})
|
||||||
defer server.Close()
|
defer server.Close()
|
||||||
|
|
||||||
result, err := DeployWikiWithOptions(&project.Project{}, DeployWikiOptions{
|
result, err := DeployWikiWithOptions(&project.Project{}, DeployWikiOptions{
|
||||||
@@ -1668,7 +1669,7 @@ func TestDeployWikiCreateCollisionAdoptsExistingNodeBBPage(t *testing.T) {
|
|||||||
|
|
||||||
createCalls := 0
|
createCalls := 0
|
||||||
var updated string
|
var updated string
|
||||||
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
server := newFakeNodeBB(t, func(w http.ResponseWriter, r *http.Request) {
|
||||||
switch {
|
switch {
|
||||||
case r.Method == http.MethodGet && r.URL.Path == "/api/v3/plugins/westgate-wiki/namespace/9/pages" && r.URL.Query().Get("q") == "":
|
case r.Method == http.MethodGet && r.URL.Path == "/api/v3/plugins/westgate-wiki/namespace/9/pages" && r.URL.Query().Get("q") == "":
|
||||||
w.Header().Set("Content-Type", "application/json")
|
w.Header().Set("Content-Type", "application/json")
|
||||||
@@ -1719,7 +1720,7 @@ func TestDeployWikiCreateCollisionAdoptsExistingNodeBBPage(t *testing.T) {
|
|||||||
default:
|
default:
|
||||||
t.Fatalf("unexpected request %s %s", r.Method, r.URL.String())
|
t.Fatalf("unexpected request %s %s", r.Method, r.URL.String())
|
||||||
}
|
}
|
||||||
}))
|
})
|
||||||
defer server.Close()
|
defer server.Close()
|
||||||
|
|
||||||
manifestPath := filepath.Join(root, "deploy-manifest.json")
|
manifestPath := filepath.Join(root, "deploy-manifest.json")
|
||||||
@@ -1771,7 +1772,7 @@ func TestDeployWikiCreateCollisionSearchesCanonicalTitleSegment(t *testing.T) {
|
|||||||
|
|
||||||
createCalls := 0
|
createCalls := 0
|
||||||
var updated string
|
var updated string
|
||||||
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
server := newFakeNodeBB(t, func(w http.ResponseWriter, r *http.Request) {
|
||||||
switch {
|
switch {
|
||||||
case r.Method == http.MethodGet && r.URL.Path == "/api/v3/plugins/westgate-wiki/namespace/58/pages" && r.URL.Query().Get("q") == "":
|
case r.Method == http.MethodGet && r.URL.Path == "/api/v3/plugins/westgate-wiki/namespace/58/pages" && r.URL.Query().Get("q") == "":
|
||||||
w.Header().Set("Content-Type", "application/json")
|
w.Header().Set("Content-Type", "application/json")
|
||||||
@@ -1828,7 +1829,7 @@ func TestDeployWikiCreateCollisionSearchesCanonicalTitleSegment(t *testing.T) {
|
|||||||
default:
|
default:
|
||||||
t.Fatalf("unexpected request %s %s", r.Method, r.URL.String())
|
t.Fatalf("unexpected request %s %s", r.Method, r.URL.String())
|
||||||
}
|
}
|
||||||
}))
|
})
|
||||||
defer server.Close()
|
defer server.Close()
|
||||||
|
|
||||||
manifestPath := filepath.Join(root, "deploy-manifest.json")
|
manifestPath := filepath.Join(root, "deploy-manifest.json")
|
||||||
@@ -1868,7 +1869,7 @@ func TestDeployWikiReportsAndArchivesStalePages(t *testing.T) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
var archived string
|
var archived string
|
||||||
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
server := newFakeNodeBB(t, func(w http.ResponseWriter, r *http.Request) {
|
||||||
if r.Method == http.MethodPut && r.URL.Path == "/api/v3/plugins/westgate-wiki/edit-lock" {
|
if r.Method == http.MethodPut && r.URL.Path == "/api/v3/plugins/westgate-wiki/edit-lock" {
|
||||||
respondWikiEditLock(t, w, r, 7, "archive-lock")
|
respondWikiEditLock(t, w, r, 7, "archive-lock")
|
||||||
return
|
return
|
||||||
@@ -1889,7 +1890,7 @@ func TestDeployWikiReportsAndArchivesStalePages(t *testing.T) {
|
|||||||
archived = req.Content
|
archived = req.Content
|
||||||
w.Header().Set("Content-Type", "application/json")
|
w.Header().Set("Content-Type", "application/json")
|
||||||
_ = json.NewEncoder(w).Encode(map[string]any{"response": map[string]any{"pid": 42, "tid": 7}})
|
_ = json.NewEncoder(w).Encode(map[string]any{"response": map[string]any{"pid": 42, "tid": 7}})
|
||||||
}))
|
})
|
||||||
defer server.Close()
|
defer server.Close()
|
||||||
|
|
||||||
report, err := DeployWikiWithOptions(&project.Project{}, DeployWikiOptions{
|
report, err := DeployWikiWithOptions(&project.Project{}, DeployWikiOptions{
|
||||||
@@ -1945,9 +1946,9 @@ func TestDeployWikiDryRunPlansTrackedStalePagePurge(t *testing.T) {
|
|||||||
}); err != nil {
|
}); err != nil {
|
||||||
t.Fatalf("write deploy manifest: %v", err)
|
t.Fatalf("write deploy manifest: %v", err)
|
||||||
}
|
}
|
||||||
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
server := newFakeNodeBB(t, func(w http.ResponseWriter, r *http.Request) {
|
||||||
t.Fatalf("dry-run purge must not call NodeBB, got %s %s", r.Method, r.URL.Path)
|
t.Fatalf("dry-run purge must not call NodeBB, got %s %s", r.Method, r.URL.Path)
|
||||||
}))
|
})
|
||||||
defer server.Close()
|
defer server.Close()
|
||||||
|
|
||||||
result, err := DeployWikiWithOptions(&project.Project{}, DeployWikiOptions{
|
result, err := DeployWikiWithOptions(&project.Project{}, DeployWikiOptions{
|
||||||
@@ -1984,9 +1985,9 @@ func TestDeployWikiPurgeRefusesStaleManifestEntryWithoutTopicID(t *testing.T) {
|
|||||||
}); err != nil {
|
}); err != nil {
|
||||||
t.Fatalf("write deploy manifest: %v", err)
|
t.Fatalf("write deploy manifest: %v", err)
|
||||||
}
|
}
|
||||||
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
server := newFakeNodeBB(t, func(w http.ResponseWriter, r *http.Request) {
|
||||||
t.Fatalf("missing-topic-id purge must not call NodeBB, got %s %s", r.Method, r.URL.Path)
|
t.Fatalf("missing-topic-id purge must not call NodeBB, got %s %s", r.Method, r.URL.Path)
|
||||||
}))
|
})
|
||||||
defer server.Close()
|
defer server.Close()
|
||||||
|
|
||||||
_, err := DeployWikiWithOptions(&project.Project{}, DeployWikiOptions{
|
_, err := DeployWikiWithOptions(&project.Project{}, DeployWikiOptions{
|
||||||
@@ -2017,15 +2018,12 @@ func TestDeployWikiPurgesTrackedStaleGeneratedTopic(t *testing.T) {
|
|||||||
}); err != nil {
|
}); err != nil {
|
||||||
t.Fatalf("write deploy manifest: %v", err)
|
t.Fatalf("write deploy manifest: %v", err)
|
||||||
}
|
}
|
||||||
purgeCalls := 0
|
var calls []string
|
||||||
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
server := newFakeNodeBB(t, func(w http.ResponseWriter, r *http.Request) {
|
||||||
if r.Method != http.MethodDelete || r.URL.Path != "/api/v3/topics/7" {
|
if !answerWikiPagePurge(t, w, r, &calls) {
|
||||||
t.Fatalf("unexpected request %s %s", r.Method, r.URL.Path)
|
t.Fatalf("unexpected request %s %s", r.Method, r.URL.Path)
|
||||||
}
|
}
|
||||||
purgeCalls++
|
})
|
||||||
w.Header().Set("Content-Type", "application/json")
|
|
||||||
_ = json.NewEncoder(w).Encode(map[string]any{"status": map[string]any{"code": "ok"}})
|
|
||||||
}))
|
|
||||||
defer server.Close()
|
defer server.Close()
|
||||||
|
|
||||||
result, err := DeployWikiWithOptions(&project.Project{}, DeployWikiOptions{
|
result, err := DeployWikiWithOptions(&project.Project{}, DeployWikiOptions{
|
||||||
@@ -2041,8 +2039,8 @@ func TestDeployWikiPurgesTrackedStaleGeneratedTopic(t *testing.T) {
|
|||||||
if result.Stale != 1 || result.Purged != 1 {
|
if result.Stale != 1 || result.Purged != 1 {
|
||||||
t.Fatalf("expected one stale purge, got %#v", result)
|
t.Fatalf("expected one stale purge, got %#v", result)
|
||||||
}
|
}
|
||||||
if purgeCalls != 1 {
|
if strings.Join(calls, ",") != "tombstone:7,hard-purge:7" {
|
||||||
t.Fatalf("expected one NodeBB topic purge call, got %d", purgeCalls)
|
t.Fatalf("expected the wiki plugin page actions to purge topic 7, got %#v", calls)
|
||||||
}
|
}
|
||||||
if _, ok := loadDeployManifest(manifestPath).Pages["skills:retired"]; ok {
|
if _, ok := loadDeployManifest(manifestPath).Pages["skills:retired"]; ok {
|
||||||
t.Fatalf("expected purged stale manifest entry to be removed")
|
t.Fatalf("expected purged stale manifest entry to be removed")
|
||||||
@@ -2075,18 +2073,17 @@ func TestDeployWikiPurgesTrackedStaleTopicsBeforeCreatingReplacementPages(t *tes
|
|||||||
}
|
}
|
||||||
|
|
||||||
var calls []string
|
var calls []string
|
||||||
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
server := newFakeNodeBB(t, func(w http.ResponseWriter, r *http.Request) {
|
||||||
|
if answerWikiPagePurge(t, w, r, &calls) {
|
||||||
|
return
|
||||||
|
}
|
||||||
switch {
|
switch {
|
||||||
case r.Method == http.MethodGet && r.URL.Path == "/api/v3/plugins/westgate-wiki/namespace/3/pages":
|
case r.Method == http.MethodGet && r.URL.Path == "/api/v3/plugins/westgate-wiki/namespace/3/pages":
|
||||||
w.Header().Set("Content-Type", "application/json")
|
w.Header().Set("Content-Type", "application/json")
|
||||||
_ = json.NewEncoder(w).Encode(map[string]any{"response": map[string]any{"pages": []any{}, "hasMore": false}})
|
_ = json.NewEncoder(w).Encode(map[string]any{"response": map[string]any{"pages": []any{}, "hasMore": false}})
|
||||||
case r.Method == http.MethodDelete && r.URL.Path == "/api/v3/topics/7":
|
|
||||||
calls = append(calls, "purge")
|
|
||||||
w.Header().Set("Content-Type", "application/json")
|
|
||||||
_ = json.NewEncoder(w).Encode(map[string]any{"status": map[string]any{"code": "ok"}})
|
|
||||||
case r.Method == http.MethodPost && r.URL.Path == "/api/v3/topics":
|
case r.Method == http.MethodPost && r.URL.Path == "/api/v3/topics":
|
||||||
calls = append(calls, "create")
|
calls = append(calls, "create")
|
||||||
if len(calls) != 2 || calls[0] != "purge" {
|
if strings.Join(calls, ",") != "tombstone:7,hard-purge:7,create" {
|
||||||
t.Fatalf("expected stale purge before create, got calls %#v", calls)
|
t.Fatalf("expected stale purge before create, got calls %#v", calls)
|
||||||
}
|
}
|
||||||
w.Header().Set("Content-Type", "application/json")
|
w.Header().Set("Content-Type", "application/json")
|
||||||
@@ -2102,7 +2099,7 @@ func TestDeployWikiPurgesTrackedStaleTopicsBeforeCreatingReplacementPages(t *tes
|
|||||||
default:
|
default:
|
||||||
t.Fatalf("unexpected request %s %s", r.Method, r.URL.String())
|
t.Fatalf("unexpected request %s %s", r.Method, r.URL.String())
|
||||||
}
|
}
|
||||||
}))
|
})
|
||||||
defer server.Close()
|
defer server.Close()
|
||||||
|
|
||||||
result, err := DeployWikiWithOptions(&project.Project{}, DeployWikiOptions{
|
result, err := DeployWikiWithOptions(&project.Project{}, DeployWikiOptions{
|
||||||
@@ -2120,7 +2117,7 @@ func TestDeployWikiPurgesTrackedStaleTopicsBeforeCreatingReplacementPages(t *tes
|
|||||||
if result.Created != 1 || result.Purged != 1 {
|
if result.Created != 1 || result.Purged != 1 {
|
||||||
t.Fatalf("expected one create and one purge, got %#v", result)
|
t.Fatalf("expected one create and one purge, got %#v", result)
|
||||||
}
|
}
|
||||||
if strings.Join(calls, ",") != "purge,create" {
|
if strings.Join(calls, ",") != "tombstone:7,hard-purge:7,create" {
|
||||||
t.Fatalf("expected purge before create, got %#v", calls)
|
t.Fatalf("expected purge before create, got %#v", calls)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -2150,8 +2147,12 @@ func TestDeployWikiResetManagedNamespacesPurgesRemotePagesBeforeCreatingFreshMan
|
|||||||
t.Fatalf("write deploy manifest: %v", err)
|
t.Fatalf("write deploy manifest: %v", err)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
const resetCalls = "tombstone:7,hard-purge:7,tombstone:8,hard-purge:8"
|
||||||
var calls []string
|
var calls []string
|
||||||
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
server := newFakeNodeBB(t, func(w http.ResponseWriter, r *http.Request) {
|
||||||
|
if answerWikiPagePurge(t, w, r, &calls) {
|
||||||
|
return
|
||||||
|
}
|
||||||
switch {
|
switch {
|
||||||
case r.Method == http.MethodGet && r.URL.Path == "/api/v3/plugins/westgate-wiki/namespace/3/pages":
|
case r.Method == http.MethodGet && r.URL.Path == "/api/v3/plugins/westgate-wiki/namespace/3/pages":
|
||||||
w.Header().Set("Content-Type", "application/json")
|
w.Header().Set("Content-Type", "application/json")
|
||||||
@@ -2164,17 +2165,9 @@ func TestDeployWikiResetManagedNamespacesPurgesRemotePagesBeforeCreatingFreshMan
|
|||||||
"hasMore": false,
|
"hasMore": false,
|
||||||
},
|
},
|
||||||
})
|
})
|
||||||
case r.Method == http.MethodDelete && r.URL.Path == "/api/v3/topics/7":
|
|
||||||
calls = append(calls, "purge:7")
|
|
||||||
w.Header().Set("Content-Type", "application/json")
|
|
||||||
_ = json.NewEncoder(w).Encode(map[string]any{"status": map[string]any{"code": "ok"}})
|
|
||||||
case r.Method == http.MethodDelete && r.URL.Path == "/api/v3/topics/8":
|
|
||||||
calls = append(calls, "purge:8")
|
|
||||||
w.Header().Set("Content-Type", "application/json")
|
|
||||||
_ = json.NewEncoder(w).Encode(map[string]any{"status": map[string]any{"code": "ok"}})
|
|
||||||
case r.Method == http.MethodPost && r.URL.Path == "/api/v3/topics":
|
case r.Method == http.MethodPost && r.URL.Path == "/api/v3/topics":
|
||||||
calls = append(calls, "create")
|
calls = append(calls, "create")
|
||||||
if strings.Join(calls, ",") != "purge:7,purge:8,create" {
|
if strings.Join(calls, ",") != resetCalls+",create" {
|
||||||
t.Fatalf("expected namespace reset purges before create, got %#v", calls)
|
t.Fatalf("expected namespace reset purges before create, got %#v", calls)
|
||||||
}
|
}
|
||||||
w.Header().Set("Content-Type", "application/json")
|
w.Header().Set("Content-Type", "application/json")
|
||||||
@@ -2190,7 +2183,7 @@ func TestDeployWikiResetManagedNamespacesPurgesRemotePagesBeforeCreatingFreshMan
|
|||||||
default:
|
default:
|
||||||
t.Fatalf("unexpected request %s %s", r.Method, r.URL.String())
|
t.Fatalf("unexpected request %s %s", r.Method, r.URL.String())
|
||||||
}
|
}
|
||||||
}))
|
})
|
||||||
defer server.Close()
|
defer server.Close()
|
||||||
|
|
||||||
result, err := DeployWikiWithOptions(&project.Project{}, DeployWikiOptions{
|
result, err := DeployWikiWithOptions(&project.Project{}, DeployWikiOptions{
|
||||||
@@ -2205,10 +2198,21 @@ func TestDeployWikiResetManagedNamespacesPurgesRemotePagesBeforeCreatingFreshMan
|
|||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatalf("DeployWikiWithOptions reset-managed-namespaces failed: %v", err)
|
t.Fatalf("DeployWikiWithOptions reset-managed-namespaces failed: %v", err)
|
||||||
}
|
}
|
||||||
if result.Created != 1 || result.Purged != 2 || result.Stale != 0 {
|
if result.Created != 1 || result.Purged != 2 {
|
||||||
t.Fatalf("expected one create and two namespace purges, got %#v", result)
|
t.Fatalf("expected one create and two namespace purges, got %#v", result)
|
||||||
}
|
}
|
||||||
if strings.Join(calls, ",") != "purge:7,purge:8,create" {
|
// Callers word their destructive-policy warning around the stale count, so
|
||||||
|
// reset deletions have to show up there rather than on a separate path the
|
||||||
|
// preview never mentions.
|
||||||
|
if result.Stale != 2 || result.ResetPurged != 2 {
|
||||||
|
t.Fatalf("expected reset deletions counted as stale, got %#v", result)
|
||||||
|
}
|
||||||
|
// tid 7 is the manifest's own page; tid 8 is a topic the deployer never
|
||||||
|
// wrote, and re-seeding cannot bring it back.
|
||||||
|
if result.ResetUnrecognized != 1 {
|
||||||
|
t.Fatalf("expected one unrecognized reset deletion, got %#v", result)
|
||||||
|
}
|
||||||
|
if strings.Join(calls, ",") != resetCalls+",create" {
|
||||||
t.Fatalf("expected reset purges before create, got %#v", calls)
|
t.Fatalf("expected reset purges before create, got %#v", calls)
|
||||||
}
|
}
|
||||||
manifest := loadDeployManifest(manifestPath)
|
manifest := loadDeployManifest(manifestPath)
|
||||||
@@ -2236,9 +2240,9 @@ func TestDeployWikiResetManagedNamespacesRequiresCreateForLocalPages(t *testing.
|
|||||||
t.Fatalf("write source page: %v", err)
|
t.Fatalf("write source page: %v", err)
|
||||||
}
|
}
|
||||||
|
|
||||||
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
server := newFakeNodeBB(t, func(w http.ResponseWriter, r *http.Request) {
|
||||||
t.Fatalf("reset without --create must fail before remote calls, got %s %s", r.Method, r.URL.String())
|
t.Fatalf("reset without --create must fail before remote calls, got %s %s", r.Method, r.URL.String())
|
||||||
}))
|
})
|
||||||
defer server.Close()
|
defer server.Close()
|
||||||
|
|
||||||
_, err := DeployWikiWithOptions(&project.Project{}, DeployWikiOptions{
|
_, err := DeployWikiWithOptions(&project.Project{}, DeployWikiOptions{
|
||||||
@@ -2270,21 +2274,21 @@ func TestDeployWikiPurgeTreatsAlreadyMissingTrackedTopicAsSuccess(t *testing.T)
|
|||||||
t.Fatalf("write deploy manifest: %v", err)
|
t.Fatalf("write deploy manifest: %v", err)
|
||||||
}
|
}
|
||||||
seen := map[string]int{}
|
seen := map[string]int{}
|
||||||
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
server := newFakeNodeBB(t, func(w http.ResponseWriter, r *http.Request) {
|
||||||
if r.Method != http.MethodDelete {
|
call, ok := wikiPagePurgeCall(t, r)
|
||||||
|
if !ok {
|
||||||
t.Fatalf("unexpected request %s %s", r.Method, r.URL.Path)
|
t.Fatalf("unexpected request %s %s", r.Method, r.URL.Path)
|
||||||
}
|
}
|
||||||
seen[r.URL.Path]++
|
seen[call]++
|
||||||
w.Header().Set("Content-Type", "application/json")
|
w.Header().Set("Content-Type", "application/json")
|
||||||
switch r.URL.Path {
|
// Topic 7 is already gone, so the plugin cannot find the page to
|
||||||
case "/api/v3/topics/7":
|
// tombstone; topic 8 still exists and purges normally.
|
||||||
|
if call == "tombstone:7" {
|
||||||
http.Error(w, `{"status":{"message":"topic not found"}}`, http.StatusNotFound)
|
http.Error(w, `{"status":{"message":"topic not found"}}`, http.StatusNotFound)
|
||||||
case "/api/v3/topics/8":
|
return
|
||||||
_ = json.NewEncoder(w).Encode(map[string]any{"status": map[string]any{"code": "ok"}})
|
|
||||||
default:
|
|
||||||
t.Fatalf("unexpected purge path %s", r.URL.Path)
|
|
||||||
}
|
}
|
||||||
}))
|
_ = json.NewEncoder(w).Encode(map[string]any{"status": map[string]any{"code": "ok"}})
|
||||||
|
})
|
||||||
defer server.Close()
|
defer server.Close()
|
||||||
|
|
||||||
result, err := DeployWikiWithOptions(&project.Project{}, DeployWikiOptions{
|
result, err := DeployWikiWithOptions(&project.Project{}, DeployWikiOptions{
|
||||||
@@ -2300,8 +2304,11 @@ func TestDeployWikiPurgeTreatsAlreadyMissingTrackedTopicAsSuccess(t *testing.T)
|
|||||||
if result.Stale != 2 || result.Purged != 2 {
|
if result.Stale != 2 || result.Purged != 2 {
|
||||||
t.Fatalf("expected two stale purges, got %#v", result)
|
t.Fatalf("expected two stale purges, got %#v", result)
|
||||||
}
|
}
|
||||||
if seen["/api/v3/topics/7"] != 1 || seen["/api/v3/topics/8"] != 1 {
|
if seen["tombstone:7"] != 1 || seen["hard-purge:7"] != 0 {
|
||||||
t.Fatalf("expected one purge call for each tracked topic, got %#v", seen)
|
t.Fatalf("expected the already-missing topic to stop after tombstone, got %#v", seen)
|
||||||
|
}
|
||||||
|
if seen["tombstone:8"] != 1 || seen["hard-purge:8"] != 1 {
|
||||||
|
t.Fatalf("expected the surviving topic to be tombstoned then purged, got %#v", seen)
|
||||||
}
|
}
|
||||||
manifest := loadDeployManifest(manifestPath)
|
manifest := loadDeployManifest(manifestPath)
|
||||||
if _, ok := manifest.Pages["skills:retired"]; ok {
|
if _, ok := manifest.Pages["skills:retired"]; ok {
|
||||||
@@ -2339,9 +2346,9 @@ func TestDeployWikiPurgeDoesNotTargetCurrentGeneratedPages(t *testing.T) {
|
|||||||
}); err != nil {
|
}); err != nil {
|
||||||
t.Fatalf("write deploy manifest: %v", err)
|
t.Fatalf("write deploy manifest: %v", err)
|
||||||
}
|
}
|
||||||
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
server := newFakeNodeBB(t, func(w http.ResponseWriter, r *http.Request) {
|
||||||
t.Fatalf("current generated page must not call NodeBB during matching-hash deploy, got %s %s", r.Method, r.URL.Path)
|
t.Fatalf("current generated page must not call NodeBB during matching-hash deploy, got %s %s", r.Method, r.URL.Path)
|
||||||
}))
|
})
|
||||||
defer server.Close()
|
defer server.Close()
|
||||||
|
|
||||||
result, err := DeployWikiWithOptions(&project.Project{}, DeployWikiOptions{
|
result, err := DeployWikiWithOptions(&project.Project{}, DeployWikiOptions{
|
||||||
@@ -2385,3 +2392,187 @@ func respondWikiEditLock(t *testing.T, w http.ResponseWriter, r *http.Request, e
|
|||||||
w.Header().Set("Content-Type", "application/json")
|
w.Header().Set("Content-Type", "application/json")
|
||||||
_ = json.NewEncoder(w).Encode(map[string]any{"response": map[string]any{"status": "ok", "tid": expectedTID, "token": token}})
|
_ = json.NewEncoder(w).Encode(map[string]any{"response": map[string]any{"status": "ok", "tid": expectedTID, "token": token}})
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// newFakeNodeBB stands up a fake NodeBB that behaves like one running
|
||||||
|
// nodebb-plugin-westgate-wiki: native topic mutation on wiki categories is
|
||||||
|
// refused, because wiki revision history is plugin-owned. Every deploy test
|
||||||
|
// goes through here, so a deployer that reaches for the core topic API fails in
|
||||||
|
// CI the way it fails in production rather than passing against a fake that is
|
||||||
|
// more permissive than the real thing.
|
||||||
|
func newFakeNodeBB(t *testing.T, handler http.HandlerFunc) *httptest.Server {
|
||||||
|
t.Helper()
|
||||||
|
return httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||||
|
if strings.HasPrefix(r.URL.Path, "/api/v3/topics/") && (r.Method == http.MethodDelete || r.Method == http.MethodPut) {
|
||||||
|
http.Error(
|
||||||
|
w,
|
||||||
|
`{"status":{"code":"bad-request","message":"Use the wiki page actions to delete, restore, or purge wiki pages."}}`,
|
||||||
|
http.StatusBadRequest,
|
||||||
|
)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
handler(w, r)
|
||||||
|
}))
|
||||||
|
}
|
||||||
|
|
||||||
|
// wikiPagePurgeCall recognizes the plugin page actions a purge goes through and
|
||||||
|
// reports them as "<action>:<tid>" so tests can assert the order.
|
||||||
|
func wikiPagePurgeCall(t *testing.T, r *http.Request) (string, bool) {
|
||||||
|
t.Helper()
|
||||||
|
var action string
|
||||||
|
switch {
|
||||||
|
case r.Method == http.MethodPut && r.URL.Path == "/api/v3/plugins/westgate-wiki/page/tombstone":
|
||||||
|
action = "tombstone"
|
||||||
|
case r.Method == http.MethodDelete && r.URL.Path == "/api/v3/plugins/westgate-wiki/page/hard-purge":
|
||||||
|
action = "hard-purge"
|
||||||
|
default:
|
||||||
|
return "", false
|
||||||
|
}
|
||||||
|
var req struct {
|
||||||
|
TID int `json:"tid"`
|
||||||
|
}
|
||||||
|
if err := json.NewDecoder(r.Body).Decode(&req); err != nil {
|
||||||
|
t.Fatalf("decode wiki %s request: %v", action, err)
|
||||||
|
}
|
||||||
|
if req.TID == 0 {
|
||||||
|
t.Fatalf("expected wiki %s request to carry a topic id", action)
|
||||||
|
}
|
||||||
|
return fmt.Sprintf("%s:%d", action, req.TID), true
|
||||||
|
}
|
||||||
|
|
||||||
|
// answerWikiPagePurge records and acknowledges a plugin purge action, which is
|
||||||
|
// what most fakes want to do with one.
|
||||||
|
func answerWikiPagePurge(t *testing.T, w http.ResponseWriter, r *http.Request, calls *[]string) bool {
|
||||||
|
t.Helper()
|
||||||
|
call, ok := wikiPagePurgeCall(t, r)
|
||||||
|
if !ok {
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
*calls = append(*calls, call)
|
||||||
|
w.Header().Set("Content-Type", "application/json")
|
||||||
|
_ = json.NewEncoder(w).Encode(map[string]any{"status": map[string]any{"code": "ok"}})
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
|
||||||
|
// resetPurgeFixture builds a source tree, a deploy manifest, and the reset
|
||||||
|
// options the namespace-reset tests share.
|
||||||
|
func resetPurgeFixture(t *testing.T, namespaces map[string]int, tidsByCID map[int][]int) (DeployWikiOptions, func(w http.ResponseWriter, r *http.Request) bool) {
|
||||||
|
t.Helper()
|
||||||
|
root := t.TempDir()
|
||||||
|
sourceDir := filepath.Join(root, "pages")
|
||||||
|
if err := os.MkdirAll(filepath.Join(sourceDir, "skills"), 0755); err != nil {
|
||||||
|
t.Fatalf("create source dir: %v", err)
|
||||||
|
}
|
||||||
|
manifestPath := filepath.Join(root, "deploy-manifest.json")
|
||||||
|
if err := saveDeployManifest(manifestPath, wikiDeployManifest{Version: "nodebb-v1", Pages: map[string]wikiDeployManifestPage{}}); err != nil {
|
||||||
|
t.Fatalf("write deploy manifest: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
listPages := func(w http.ResponseWriter, r *http.Request) bool {
|
||||||
|
if r.Method != http.MethodGet {
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
var cid int
|
||||||
|
if _, err := fmt.Sscanf(r.URL.Path, "/api/v3/plugins/westgate-wiki/namespace/%d/pages", &cid); err != nil {
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
tids, ok := tidsByCID[cid]
|
||||||
|
if !ok {
|
||||||
|
t.Fatalf("unexpected namespace listing for category %d", cid)
|
||||||
|
}
|
||||||
|
pages := make([]map[string]any, 0, len(tids))
|
||||||
|
for _, tid := range tids {
|
||||||
|
pages = append(pages, map[string]any{"tid": tid, "title": fmt.Sprintf("Page %d", tid)})
|
||||||
|
}
|
||||||
|
w.Header().Set("Content-Type", "application/json")
|
||||||
|
_ = json.NewEncoder(w).Encode(map[string]any{"response": map[string]any{"pages": pages, "hasMore": false}})
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
|
||||||
|
return DeployWikiOptions{
|
||||||
|
SourceDir: sourceDir,
|
||||||
|
ManifestPath: manifestPath,
|
||||||
|
Token: "nodebb-token",
|
||||||
|
CategoryIDs: namespaces,
|
||||||
|
AllowCreates: true,
|
||||||
|
ResetManagedNamespaces: true,
|
||||||
|
}, listPages
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestDeployWikiResetSkipsTopicsNodeBBRefusesToDelete(t *testing.T) {
|
||||||
|
opts, listPages := resetPurgeFixture(t, map[string]int{"skills": 3}, map[int][]int{3: {5, 6}})
|
||||||
|
var calls []string
|
||||||
|
server := newFakeNodeBB(t, func(w http.ResponseWriter, r *http.Request) {
|
||||||
|
if listPages(w, r) {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
call, ok := wikiPagePurgeCall(t, r)
|
||||||
|
if !ok {
|
||||||
|
t.Fatalf("unexpected request %s %s", r.Method, r.URL.Path)
|
||||||
|
}
|
||||||
|
calls = append(calls, call)
|
||||||
|
// Topic 5 stands in for the wiki home page, which the plugin excludes
|
||||||
|
// from tombstone, restore, and purge alike.
|
||||||
|
if call == "tombstone:5" {
|
||||||
|
http.Error(w, `{"status":{"message":"[[error:no-privileges]]"}}`, http.StatusForbidden)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
w.Header().Set("Content-Type", "application/json")
|
||||||
|
_ = json.NewEncoder(w).Encode(map[string]any{"status": map[string]any{"code": "ok"}})
|
||||||
|
})
|
||||||
|
defer server.Close()
|
||||||
|
opts.Endpoint = server.URL
|
||||||
|
|
||||||
|
result, err := DeployWikiWithOptions(&project.Project{}, opts, nil)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("DeployWikiWithOptions reset with an undeletable topic failed: %v", err)
|
||||||
|
}
|
||||||
|
if result.ResetSkipped != 1 || result.ResetPurged != 1 {
|
||||||
|
t.Fatalf("expected one skipped and one completed reset deletion, got %#v", result)
|
||||||
|
}
|
||||||
|
if result.Purged != 1 || result.Stale != 1 {
|
||||||
|
t.Fatalf("expected the skipped topic to be dropped from the counts, got %#v", result)
|
||||||
|
}
|
||||||
|
if result.ResetUnrecognized != 1 {
|
||||||
|
t.Fatalf("expected only the purged unrecognized topic to be counted, got %#v", result)
|
||||||
|
}
|
||||||
|
if strings.Join(calls, ",") != "tombstone:5,tombstone:6,hard-purge:6" {
|
||||||
|
t.Fatalf("expected the reset to continue past the refused topic, got %#v", calls)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// NodeBB answers 403 both for the wiki home page and for a token without purge
|
||||||
|
// privileges. A category where nothing could be deleted at all is the second
|
||||||
|
// case, and reporting it as a completed reset would leave the manifest claiming
|
||||||
|
// a fresh start over pages that are all still there.
|
||||||
|
func TestDeployWikiResetFailsWhenACategoryRefusesEveryDeletion(t *testing.T) {
|
||||||
|
opts, listPages := resetPurgeFixture(t,
|
||||||
|
map[string]int{"skills": 3, "feats": 4},
|
||||||
|
map[int][]int{3: {5, 6}, 4: {7, 8}},
|
||||||
|
)
|
||||||
|
server := newFakeNodeBB(t, func(w http.ResponseWriter, r *http.Request) {
|
||||||
|
if listPages(w, r) {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
call, ok := wikiPagePurgeCall(t, r)
|
||||||
|
if !ok {
|
||||||
|
t.Fatalf("unexpected request %s %s", r.Method, r.URL.Path)
|
||||||
|
}
|
||||||
|
// Category 4 holds topics 7 and 8, and the token can purge neither.
|
||||||
|
if strings.HasSuffix(call, ":7") || strings.HasSuffix(call, ":8") {
|
||||||
|
http.Error(w, `{"status":{"message":"[[error:no-privileges]]"}}`, http.StatusForbidden)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
w.Header().Set("Content-Type", "application/json")
|
||||||
|
_ = json.NewEncoder(w).Encode(map[string]any{"status": map[string]any{"code": "ok"}})
|
||||||
|
})
|
||||||
|
defer server.Close()
|
||||||
|
opts.Endpoint = server.URL
|
||||||
|
|
||||||
|
_, err := DeployWikiWithOptions(&project.Project{}, opts, nil)
|
||||||
|
if err == nil || !strings.Contains(err.Error(), "wiki purge privileges") {
|
||||||
|
t.Fatalf("expected a category that deleted nothing to fail loudly, got %v", err)
|
||||||
|
}
|
||||||
|
if !strings.Contains(err.Error(), "4") {
|
||||||
|
t.Fatalf("expected the failure to name the refusing category, got %v", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|||||||
Reference in New Issue
Block a user