From 7437653f143f0a8f66b72c1c8a261ff223b694b4 Mon Sep 17 00:00:00 2001 From: vickydotbat Date: Fri, 31 Jul 2026 11:00:49 +0000 Subject: [PATCH] docs: describe the on-disk shape of an emitted NWSync tree (#77) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Documents what `nwsync emit --out DIR` actually produces, so a zone can be checked by hand. The trap this removes: a blob's filename is the SHA-1 of the resource's *original* bytes, but the file on disk is NWCompressedBuffer framing — a 24-byte `NSYC` header then a zstd frame. `sha1sum ` therefore never matches the name it is sitting under. The doc records the recipe that does match: ``` tail -c +25 | zstd -dc | sha1sum ``` Also notes the tree layout, that the uncompressed length is a little-endian `uint32` at offset 12, and the rough 4:1 compression ratio on hak content. Verified two ways: against a real emit of a 250 MB hak (2296 blobs, 59 MB on disk against 249 MB of resources), and against `internal/nwsync/compressedbuf.go`, where the header is written as `[]uint32{blobMagic, blobVersion, algorithmZstd, uint32(len(data)), zstdHeaderVer, zstdDictionary}` — confirming field 3 at offset 12. Docs only, no behaviour change. Falls out of the investigation in #76; that fix is not in this PR. 🤖 Generated with [Claude Code](https://claude.com/claude-code)Reviewed-on: https://git.westgate.pw/ShadowsOverWestgate/sow-tools/pulls/77 Reviewed-by: xtul Co-authored-by: vickydotbat --- docs/command-surface.md | 29 +++++++++++++++++++++++++++++ 1 file changed, 29 insertions(+) diff --git a/docs/command-surface.md b/docs/command-surface.md index 15128c9..33162a5 100644 --- a/docs/command-surface.md +++ b/docs/command-surface.md @@ -70,6 +70,35 @@ the zone and no index. Blob names are content hashes, so re-running skips whatever already landed, and `assemble` fails closed on an artifact with no index rather than publishing a manifest that is missing a hak. +### What an emitted tree looks like + +`--out DIR` produces the same tree `emit` would upload, which makes it the way +to check a zone by hand without touching one: + +``` +.nsym binary index +.nsym.json the same index, readable +data/sha1/a7/4a/a74aa84a... one blob per resource, two-level fanout +``` + +A blob's name is the SHA-1 of the resource's **original** bytes, but the file on +disk is not those bytes: each blob is wrapped in NWCompressedBuffer framing, a +24-byte `NSYC` header followed by a zstd frame. Hashing the file directly will +not match its name, which is the obvious +first thing to try and the obvious first thing to be confused by. Strip the +header first: + +``` +tail -c +25 | zstd -dc | sha1sum # == the blob's filename +``` + +The header carries the uncompressed length as a little-endian `uint32` at offset +12, so the decompressed size is checkable without decompressing. Compression is +worth roughly a 4:1 +saving on hak content: a 250 MB hak emitted 2296 blobs totalling 59 MB on disk +against 249 MB of resources, as recorded in the sidecar's `on_disk_bytes` and +`total_bytes`. + ## Hidden compatibility aliases Existing scripts may continue using these names indefinitely. They are accepted