Running the repair disproved the advice #90 landed an hour earlier.
"Purge the zone, then believe `verify`" assumed the stale set was unknowable. It is not. `verify` reads the edge, so the run straight after a repair names every key the edge is still serving stale — a survey, not a verdict. Purge those, re-run, and the second run is the verdict.
The measured numbers are the whole argument:
| | |
| --- | --- |
| blobs rewritten at the origin | 2,603 |
| blobs stale at the edge | **8** |
All eight were ones a failed player sync had pulled ninety minutes before the backfill. The edge only caches what someone fetched, so purging the zone would have cooled 69,169 objects to fix 8.
Full sweep after the targeted purge: `verified 69177 of 69177 blobs behind 72544 resources: 0 failures, 14887519535 bytes checked`. #75's gate is met.
Docs only. Runbook side in sow-platform.
Refs #88, #89, #75.
🤖 Generated with [Claude Code](https://claude.com/claude-code)Reviewed-on: #91
Reviewed-by: xtul <mpiasecki720@protonmail.com>
Co-authored-by: vickydotbat <vickydotbat@tutamail.com>
#89 asked for a decision. This is it, and it is the laziest of the three options listed there: **purge the whole pull zone by hand after a repair, one call, documented in the repair procedure.**
Why not the other two:
- Purging from `emit --verify` needs a CDN credential `emit` deliberately does not hold, and `emit` reports how many blobs it wrote, never which ones — so it could not target the keys anyway.
- Waiting out the TTL means 30 days.
Whole-zone rather than per-key costs a cold cache on a zone whose objects are mostly cold, and a repair scatters thousands of keys across the tree regardless.
Also answers the question #89 left open: **the zone does not negative-cache.** A missing key answers 404 with `cache-control: no-cache` and `cdn-cache: MISS`, still MISS on an immediate retry (checked credential-free, 2026-08-01).
Docs only — `docs/command-surface.md` and `nwsync`'s usage text. The procedure itself lives in sow-platform's NWSync runbook, next to the zone it acts on.
Closes#89.
🤖 Generated with [Claude Code](https://claude.com/claude-code)Reviewed-on: #90
Reviewed-by: xtul <mpiasecki720@protonmail.com>
Co-authored-by: vickydotbat <vickydotbat@tutamail.com>
Closes#86. Closes#85.
These land together on purpose. Fixing the encoder alone changes nothing for the blobs already in the zone, because `emit` skips whatever is already present.
## #86 — the framing fix
`klauspost/compress` omits the zstd `Frame_Content_Size` field for inputs under 256 bytes, which the format permits. Reference libzstd never does, so the NWN client — which sizes its output buffer from `ZSTD_getFrameContentSize` and has therefore never met a frame without one — rejected roughly 6% of our blobs outright. Any single one stops a sync dead, so no client could complete a sync of the live manifest.
No encoder option changes this, so `compressBlob` re-headers the affected frames into the shape libzstd itself emits: `Single_Segment_flag` set, `Window_Descriptor` dropped, and the freed byte spent on a one-byte `Frame_Content_Size`. Same length in, same length out, and the same descriptor byte (`0x24`) the issue recorded from libzstd.
`compressBlob` then asserts its own output. An encoder upgrade that finds another way to omit the field would otherwise reproduce #86 in silence, and a blob is skipped by every later emit once written.
`emitter_version` goes to `2`, so `assemble` refuses to merge an index written by the encoder that omitted the field.
**Proved against the reference decoder, not just a round trip.** A real emitted 175-byte blob:
```
Frames Skips Compressed Uncompressed Ratio Check Filename
1 0 48 B 175 B 3.646 XXH64 frame.zst
c59d6620d4ffd4bf3fe73df43b19b7afcfe8fea4 - <- zstd -dc | sha1sum
c59d6620d4ffd4bf3fe73df43b19b7afcfe8fea4 <- the blob's own name
```
Before the fix that `Uncompressed` column was blank.
## #85 — `nwsync verify`
`crucible nwsync verify <manifest-sha1>` reads a manifest and its blobs back through the **public pull zone**, with no credential, because what matters is the bytes a client is served, edge behaviour included. Every distinct blob is decompressed and hashed; failures are reported per blob as missing / malformed framing / size mismatch / hash mismatch, and the exit code is 1.
- `--sample N` makes a routine check cheap against a manifest that is ~69,000 blobs and 15 GB; the default is a full sweep.
- `--base URL` / `NWSYNC_PULL_BASE` overrides the public host.
- The manifest is checked against its own sha1 before a single blob is fetched.
- `emit --verify` applies the same check where `emit` would otherwise trust presence, and replaces a stored blob that is not what its name claims. This is what makes the #86 blobs repairable.
## Why the existing checks missed this
Both new checks assert the **frame property**, not just a round trip. The conformance suite (#59) compares decompressed bytes, so a frame that decodes correctly passes regardless of its header; and the earlier zone audit decompressed 68 blobs with the `zstd` CLI, a *more* capable decoder than the client's, which certified exactly the blobs the client rejects.
## Checks
`make check` and `make smoke` green. Second commit is the fixes from a two-axis review of the first.
## Not in this PR
Three follow-ups, filed separately: the backfill has not been run, replacing a blob does not purge the pull-zone edge cache, and #85's runbook line belongs to `sow-platform`.
🤖 Generated with [Claude Code](https://claude.com/claude-code)Reviewed-on: #87
Co-authored-by: vickydotbat <vickydotbat@tutamail.com>
Closes#79.
Emit is latency-bound, not CPU-bound. Every blob costs two serial round-trips to the zone — a `ProbeKey` HEAD, then a `PutReader` PUT — so a hak with a few thousand resources pays a few thousand serialised latencies. Measured on the live sow-assets-manifest backfill: 26 s of CPU across 9.5 minutes of wall clock, on a 4-core host with 5 GB free and peak RSS of 51 MB.
`emit` now hashes, compresses and stores `--jobs N` resources at once, default 16 — matching `DEPOT_JOBS` and the transport's `MaxIdleConnsPerHost`, so a worker per connection needs no fresh TLS handshake. `--jobs 1` is exactly the old behaviour.
### Three properties had to survive
Each has a test in `internal/nwsync/jobs_test.go`:
- **Deterministic manifest bytes.** `emitterVersion` promises a manifest is a function of its artifact, so `entries` is index-addressed rather than appended to — a worker owns `entries[i]` alone and the slice comes back in artifact order whatever order uploads finish in. `TestEmitProducesTheSameIndexAtEveryJobCount` diffs the `.nsym` and its sidecar between `-jobs 1` and `-jobs 16`.
- **Index still lands last.** Any worker's failure aborts before a manifest is written. `TestEmitLeavesNoIndexWhenAParallelUploadFails` fails every blob PUT with 16 workers in flight and asserts no `.nsym` appears. Under `-race` it also covers the shared counters.
- **Identical content still shares one blob.** This one bit during development and is the reason to read the diff carefully: serially, the sink's existence check absorbed two resrefs with identical bytes. In parallel both workers probe, both miss, and both upload — `TestEmitWritesBlobsAndManifest` caught it as "wrote 2 blobs, want 1". Claiming the sha1 in-process restores the dedupe and skips a probe round-trip as well.
### Memory
Peak now tracks the resources in flight rather than one resource. The ceiling is `N` × the 15 MB `fileSizeLimit` plus its compressed copy — bounded by a constant this package enforces itself, and still not tracking the archive. `TestEmitPeakMemoryIsBoundedByJobCount` re-runs the #76 regression check at `-jobs 8`: a hak 8× bigger still costs the same.
This is only cheap because of #78. Before streaming emit, N workers would have meant N whole archives resident.
### Not done
Skipping the `ProbeKey` HEAD on a first-time emit would halve round-trips, but doubles uploaded bytes on a re-run — which is exactly what a backfill is. Noted in #79 so it is not rediscovered; parallelism is the better lever and this PR takes it.
Worker compression still serialises on `blobEncoder`, which is `WithEncoderConcurrency(1)` for the memory reason in `compressedbuf.go`. At 26 s of CPU per hak that is not worth trading memory for, but it is where to look if the numbers ever say otherwise.
🤖 Generated with [Claude Code](https://claude.com/claude-code)Reviewed-on: #80
Co-authored-by: vickydotbat <vickydotbat@tutamail.com>
Documents what `nwsync emit --out DIR` actually produces, so a zone can be checked by hand.
The trap this removes: a blob's filename is the SHA-1 of the resource's *original* bytes, but the file on disk is NWCompressedBuffer framing — a 24-byte `NSYC` header then a zstd frame. `sha1sum <blob>` therefore never matches the name it is sitting under. The doc records the recipe that does match:
```
tail -c +25 <blob> | zstd -dc | sha1sum
```
Also notes the tree layout, that the uncompressed length is a little-endian `uint32` at offset 12, and the rough 4:1 compression ratio on hak content.
Verified two ways: against a real emit of a 250 MB hak (2296 blobs, 59 MB on disk against 249 MB of resources), and against `internal/nwsync/compressedbuf.go`, where the header is written as `[]uint32{blobMagic, blobVersion, algorithmZstd, uint32(len(data)), zstdHeaderVer, zstdDictionary}` — confirming field 3 at offset 12.
Docs only, no behaviour change. Falls out of the investigation in #76; that fix is not in this PR.
🤖 Generated with [Claude Code](https://claude.com/claude-code)Reviewed-on: #77
Reviewed-by: xtul <mpiasecki720@protonmail.com>
Co-authored-by: vickydotbat <vickydotbat@tutamail.com>
Builds the code half of #60, and closes the CLI gap the #53 sweep found: PR #71 shipped #53's original surface rather than the one #56, #62 and #65 settled.
## What lands
**`depot.KeyStore`** — `ProbeKey` / `PutReader` / `GetKey`, addressing the zone by object key rather than by depot sha. NWSync cannot use the sha-addressed path: a blob is named after the sha1 of its *uncompressed* bytes while the body uploaded is the compressed form, and Bunny's `Checksum` header is sha256 of the body. Per #55 this reuses `internal/depot`'s `httpBackend` — same IPv4-pinned transport, same retry, same tri-state probe — and the sha-addressed `Backend` is now rewritten on top of it. No second HTTP client, no per-instance hash-function fields: the caller passes the key and the checksum, which turned out simpler than #55 expected.
**A sink in `internal/nwsync`** — the zone by default, a local tree under `--out DIR` as the conformance path. Blobs upload as they are produced and the index lands last, so the presence of an index is the publication marker. A blob already in the zone is skipped via #55's probe *without* paying for compression (the body is a thunk) — which matters for the backfill, where compression is the expensive part.
**The settled CLI**
```
nwsync emit [--as NAME] [--out DIR] <artifact-key> <file>
nwsync assemble --group-id N [--tlk-key KEY] [--out DIR] <artifact-key>...
```
Artifact keys are depot keys; an index lives beside its artifact with the extension replaced (#62), derived in exactly one place so `emit` and `assemble` cannot disagree. Flags may now follow positionals — Go's `flag` stops at the first non-flag argument, which cost a run during #59.
**Fail-closed in two places** — an artifact key whose embedded digest does not match the file is refused (publishing an index under the wrong key silently pairs a manifest with the wrong artifact), and `assemble` refuses an artifact with no index rather than publishing a manifest missing a hak.
## Checks
`make check` green. Six new tests run against a Bunny-shaped `httptest` zone that verifies the `Checksum` header the way Bunny does: blobs-then-index ordering, skip-if-present, no index after a failed upload, key/file mismatch, and assemble reading indexes back out of the zone.
Conformance re-run through the new CLI against upstream `nwn_nwsync_write` 2.1.2 over `sow_vfxs_01.hak` (#59's oracle): the manifest is still **byte-identical**.
## Not in this PR
- **Live upload against the real zone.** The nwsync zone and its credential are #61, still open. Everything here is proven against a fake zone only.
- **Consumer wiring** — #65, in the three producer repos.
- **The mid-hak failure *policy*.** The mechanism is here (fail closed, orphan blobs left, re-run resumes); whether a module release may proceed when an emit failed is a human call, still open on #60.
🤖 Generated with [Claude Code](https://claude.com/claude-code)Reviewed-on: #73
Co-authored-by: vickydotbat <vickydotbat@tutamail.com>
Builds sow-tools#53. Format spec followed is the resolution comment of sow-platform#94, checked line by line against niv/neverwinter.nim at HEAD (`nwsync.nim`, `compressedbuf.nim`, `nwsync/private/libupdate.nim`).
## What lands
`crucible nwsync emit <artifact> --out DIR` — explodes one `.hak`/`.erf`, or one loose file such as the TLK, into NWSync blobs plus a NSYM v3 manifest covering only that artifact, with the same `.json` sidecar upstream writes. Blob path `data/sha1/<h0h1>/<h2h3>/<sha1>`, body in NWCompressedBuffer framing (magic `NSYC`, version 3, algorithm 2, uncompressed size, zstd header version 1, dictionary 0, raw zstd frame). The sha1 that names a blob is over the uncompressed bytes.
`crucible nwsync assemble --order NAMES --entries DIR --out DIR [--group-id N]` — merges the per-artifact manifests into one, reading no bulk data at all. Merge rule is resref shadowing, not concatenation: a resref in more than one artifact resolves to the earliest artifact in `--order`, which is how the game resolves it. `--group-id` stays caller-supplied (1 current, 2 testing; 0 is absent, matching upstream omitting a zero integer meta field).
Rules taken from upstream and not re-invented: `nss`/`ndb`/`gic` always skipped; an unresolvable restype is a hard error, not a skip; a resource over 15 MB fails closed; no `latest` file and no `.origin` file, ever. A `.mod` is refused outright — a persistent world publishes no module contents, so the module contributes no bytes.
## Two deliberate departures
- **Emitter version is its own field, not the build revision.** `emitter_version` is a constant bumped only when emitted bytes change. Keying the refuse-to-merge check on `created_with` would invalidate every published index on every unrelated crucible commit and force a re-emit of the whole 15 GB corpus — the opposite of "nothing downstream ever needs the hak again".
- **`SOURCE_DATE_EPOCH` pins the sidecar timestamp.** The manifest itself was already deterministic; the sidecar's `created` was not, against the determinism rule in `docs/consumer-contract.md`.
## Not in this PR, and why
- **Direct upload.** Only the local `--out` sink exists, which is the conformance path. The upload sink and the mid-hak-failure question are sow-tools#60, and the consumer wiring is #65.
- **The conformance run against upstream.** sow-tools#59 owns getting `nwn_nwsync_write` running and capturing reference output. The format here was read from upstream source rather than from its output, so the byte-for-byte manifest comparison and the after-decompression blob comparison still have to happen — that is what #59 is for. The tests in this PR check the layout against the spec, so a shared misreading would pass them.
- **The `artifacts/haks/sha256/<a>/<b>/<sha256>.nsym` location.** emit writes `<out>/<name>.nsym`; where a publisher puts it is the publisher's business (#65).
- **The acceptance gate** — a real client syncing from an assembled manifest — is unchanged and still open.
## Checks
`make check` green (vet, unit tests, shellcheck, yamllint, workflow contract), `make smoke` green with the new builder, `nix build .#crucible` produces `crucible-nwsync`.
🤖 Generated with [Claude Code](https://claude.com/claude-code)Reviewed-on: #71
Co-authored-by: vickydotbat <vickydotbat@tutamail.com>
`tea comment` reads stdin to EOF and appends whatever it finds to the comment body. In a non-interactive shell — which is every agent — stdin is an open pipe that never sends EOF, so the call hangs forever instead of posting.
Verified on tea 0.14.0: with `</dev/null` it posts instantly; with an open pipe it blocks until killed; with `echo "x" | tea comment N "y"` it posts `y` followed by `x`.
Documents the redirect in the tracker guide, and applies it to the wayfinder resolve step. Same trap exists on `tea issues create --description` and `tea pr create`.
🤖 Generated with [Claude Code](https://claude.com/claude-code)Reviewed-on: #68
Co-authored-by: vickydotbat <vickydotbat@tutamail.com>
Closes#66.
`depot status` and `depot get` chose a depot tree on disk with `--target local`, taking the root from `DEPOT_DIR`. One decision, two flags — and `--target local` with no directory anywhere was valid but meaningless.
Now they take `--out DIR`, matching the crucible nwsync surface. `--target` names remote backends only (`bunny|cdn`). Passing both exits 64.
- `--target local` stays as a deprecated alias for `--out $DEPOT_DIR`, listed in the hidden-alias table in `docs/command-surface.md`. Nothing in `wrappers/` calls depot, but the three repos in `wrappers/consumers.txt` are out of reach from here, so the alias stays.
- `status --source` was already unused; it now warns "ignored; use --out DIR" instead of accepting silently.
- Fixes a real parse bug found on the way: the documented `depot get <sha> <dest> --target cdn` form never parsed its flags, because Go's `flag` package stops at the first positional. Positionals are split off by hand now; both orders are tested.
- Registry usage strings and `docs/command-surface.md` synced — that doc still said depot was unwired.
Not converted: `push --source` and `pull --dest`. Neither is `--target local`; one names a read source, the other a remote pull's destination. Say the word if they should become `--out` too.
`go test ./...` green.
🤖 Generated with [Claude Code](https://claude.com/claude-code)Reviewed-on: #67
Co-authored-by: vickydotbat <vickydotbat@tutamail.com>
Part of the workspace-wide standards rollout. Establishes where work lives (issues vs ADRs vs standing law), scaffolds `docs/agents/` for the engineering skills, and adopts ADR-0001 locally. See `sow-platform` ADR-0021 for the `sow-docs` retirement.Reviewed-on: #51
Reviewed-by: xtul <mpiasecki720@protonmail.com>
Co-authored-by: vickydotbat <vickydotbat@tutamail.com>
## Summary
- always wrap NWN model compilation with `xvfb-run`, even when the caller has `DISPLAY`
- fail closed with an actionable error when `xvfb-run` is unavailable instead of opening the client UI
- update the compiler contract and regression coverage
## Verification
- focused red/green regression tests
- `nix develop -c make check`
- real NWN compile with `DISPLAY=:0` under transient Xvfb; binary MDL output verified
Generated with Claude CodeReviewed-on: #49
Co-authored-by: vickydotbat <vickydotbat@tutamail.com>
## Summary
The `registry.westgate.pw/deployment/crucible` image has no consumer: `prod.yml` no longer reserves a slot for it (contract settled in sow-platform PR #64) and no runtime or recovery path pulls it. Binaries, wrappers, and the Nix input remain the supported ways to run Crucible.
## What changed
- Deleted `.gitea/workflows/build-image.yml`, `publish-image.yml`, `test-image.yml`
- Deleted `docker/Dockerfile` and the `make image` target
- Updated README, AGENTS.md, and `docs/consumer-contract.md` to state the image is retired
Existing `deployment/crucible` images in the registry can be deleted at leisure; nothing references them.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Reviewed-on: #41
Co-authored-by: vickydotbat <vickydotbat@tutamail.com>
Co-committed-by: vickydotbat <vickydotbat@tutamail.com>
forces build parity with crucible = local builds and CI/CD builds use different tools.
Reviewed-on: #14
Co-authored-by: vickydotbat <vickydotbat@tutamail.com>
Co-committed-by: vickydotbat <vickydotbat@tutamail.com>
archvillainette
