ShadowsOverWestgate/sow-tools
7
0
Fork 0
Code Issues 2 Pull Requests Actions Packages Projects Releases 38 Activity

docs: describe the on-disk shape of an emitted NWSync tree
ci / ci (pull_request) Successful in 3m32s
Details

Browse Source 
Blob filenames are the SHA-1 of a resource's original bytes, but the file on
disk is NWCompressedBuffer framing: a 24-byte NSYC header then a zstd frame.
Hashing the file directly is the first thing anyone tries when checking a zone
by hand, and it never matches. Record the layout and the one-line recipe that
does match, plus where the uncompressed length lives in the header.

Verified against a real emit and against internal/nwsync/compressedbuf.go.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
archvillainette
2026-07-31 12:51:14 +02:00
co-authored by Claude Opus 5
parent f9051a2c01
commit a33e39d422
1 changed files with 29 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/command-surface.md
+29
View File
@@ -70,6 +70,35 @@ the zone and no index. Blob names are content hashes, so re-running skips
whatever already landed, and `assemble` fails closed on an artifact with no whatever already landed, and `assemble` fails closed on an artifact with no
index rather than publishing a manifest that is missing a hak. index rather than publishing a manifest that is missing a hak.
### What an emitted tree looks like
`--out DIR` produces the same tree `emit` would upload, which makes it the way
to check a zone by hand without touching one:
```
<artifact-sha>.nsym binary index
<artifact-sha>.nsym.json the same index, readable
data/sha1/a7/4a/a74aa84a... one blob per resource, two-level fanout
```
A blob's name is the SHA-1 of the resource's **original** bytes, but the file on
disk is not those bytes: each blob is wrapped in NWCompressedBuffer framing, a
24-byte `NSYC` header followed by a zstd frame. Hashing the file directly will
not match its name, which is the obvious
first thing to try and the obvious first thing to be confused by. Strip the
header first:
```
tail -c +25 <blob> | zstd -dc | sha1sum # == the blob's filename
```
The header carries the uncompressed length as a little-endian `uint32` at offset
12, so the decompressed size is checkable without decompressing. Compression is
worth roughly a 4:1
saving on hak content: a 250 MB hak emitted 2296 blobs totalling 59 MB on disk
against 249 MB of resources, as recorded in the sidecar's `on_disk_bytes` and
`total_bytes`.
## Hidden compatibility aliases ## Hidden compatibility aliases
Existing scripts may continue using these names indefinitely. They are accepted Existing scripts may continue using these names indefinitely. They are accepted