sigil-wire

A typed, length-prefixed binary codec for the Sigil value model. The binary sibling to sigil-json: where JSON is the human-readable format for small control messages, sigil-wire is the compact, fast format for bulk structured payloads.

Values decode in a single O(n) forward pass, and string/bytevector bodies are copied wholesale (memcpy), avoiding the per-character O(n²) work that JSON encoding of Sigil values incurs. It is the transport-neutral wire format used by the Slate/Lantern bulk channel and inherited by Familiar over Enclave.

Usage

(import (sigil wire))

;; Encode any Sigil value to a self-describing bytevector.
(define bytes (wire-encode #{ name: "Alice" scores: #[10 20 30] }))

;; Decode it back.
(wire-decode bytes)
; => #{ name: "Alice" scores: #[10 20 30] }

The decoder is a trust boundary: bytes may be attacker-influenced (over Enclave). Pass a caps record to bound the work a hostile frame can trigger:

(wire-decode bytes (make-wire-caps max-depth: 32 max-count: 1000))

API

Procedure	Purpose
`(wire-encode value)` → bytevector	Serialize a Sigil value.
`(wire-decode bytevector [caps])` → value	Deserialize, bounded by `caps`.
`(make-wire-caps [keys])` → caps	Build a decode-limits record.
`default-wire-caps`	The default caps used when none is passed.

Caps (decoder limits)

make-wire-caps accepts keyword arguments; any omitted field takes its default:

Field	Default	Meaning
`max-bytes-len:`	64 MiB	Largest single string / bytevector body.
`max-count:`	16 M	Largest collection (list/vector/array length, dict pairs).
`max-depth:`	256	Deepest nesting of collections. Decode is recursive, so this also bounds host stack depth — keep it modest; cranking it to tens of thousands reintroduces stack-overflow risk on a hostile deeply-nested frame.
`max-total:`	256 MiB	Total decoded heap: string/bytevector body bytes plus an estimated per-element cost for every collection slot, so a collection-heavy frame of tiny elements is bounded too.
`max-int-bytes:`	1024	Largest integer payload in varint bytes (bounds bignum size).

Supported value types

Every value the seam carries: #f, #t, exact integers (including bignums), doubles, strings, bytevectors, keywords, symbols, characters, lists, dicts, vectors (R7RS #(...)) and arrays (Sigil #[...]).

A value that cannot be represented — a procedure, a port, a record without a registered codec, an improper (dotted) list, or a non-real number — is an encode error, never a silent drop. (Registered-tag record extensions are out of scope for v1.)

Wire format

A message is a fixed 4-byte header followed by exactly one root value:

message : MAGIC(2)  VERSION(1)  FLAGS(1)  <value>
          0x53 0x57  0x01        0x00

value   : TAG(1)  <payload>

Decoders reject an unknown MAGIC or an unknown major VERSION. FLAGS is reserved (0). Trailing bytes after the root value are a decode error.

Lengths and counts are unsigned LEB128 varints (little-endian base-128; the high bit of each byte flags continuation). Small collections stay a single byte.

Tag table

TAG	Type	Payload
0x00	`#f` / nil	(none)
0x01	`#t`	(none)
0x02	int	zigzag LEB128 varint (bignum-safe, single format)
0x03	float	8 bytes, IEEE-754 double, little-endian
0x04	string	varint N, then N UTF-8 bytes (memcpy)
0x05	bytevector	varint N, then N raw bytes (memcpy)
0x06	keyword	varint N, then N UTF-8 bytes
0x07	symbol	varint N, then N UTF-8 bytes
0x08	char	varint Unicode codepoint
0x09	list	varint K, then K values
0x0A	dict	varint K, then K (key value) pairs
0x0B	vector	varint K, then K values (R7RS `#(...)`)
0x0C	array	varint K, then K values (Sigil `#[...]`)

Any tag outside this table is a hard decode error (never skip-and-continue — a misparsed length would cascade).

Integers

A single zigzag LEB128 varint encodes every integer, from small values up to arbitrary-precision bignums — there is no separate bignum form. Zigzag maps signed to unsigned so small negatives stay short: n >= 0 → 2n, n < 0 → 2|n| - 1. The varint then encodes that non-negative value.

Floats

Doubles are stored as their raw 64-bit IEEE-754 bit pattern, little-endian, via the native bytevector-ieee-double-{ref,set!} accessors. Every double — normal, subnormal, ±0.0, ±inf, NaN — round-trips bit-exactly, including the sign bit of -0.0 (even though Sigil predicates cannot themselves distinguish -0.0 from +0.0). On decode, the accessor canonicalizes any hostile bit pattern to a safe quiet-NaN flonum, so a malicious float body can never produce a type-confused value.

Security

The decoder never trusts a length it reads:

Every length prefix is checked against the remaining buffer before any allocation or copy — it never pre-allocates N bytes from an untrusted varint.
All caps (body length, collection count, nesting depth, total decoded size, integer magnitude) are enforced, rejecting before allocating.
A bounds-checked cursor means a truncated or lying frame errors cleanly and never reads out of bounds; an endless varint is rejected rather than spun on.

The test suite includes a hostile-input fuzz battery (truncated frames, lying length prefixes, cap violations, unknown tags, unknown version, varint abuse, invalid UTF-8 bodies) asserting each errors cleanly (or, for malformed UTF-8, decodes leniently to a string — never a crash or out-of-bounds read).

Harden before untrusted remote (Enclave) use

One residual risk is not bounded by the per-frame caps: decoding a keyword or symbol interns its name permanently (the intern table is not garbage collected). A hostile remote peer that streams an endless supply of unique keyword/symbol names can exhaust memory over time. This is safe for local, in-process bulk payloads (the current use), but before the codec carries untrusted remote traffic over Enclave, symbol/keyword decoding should be gated — decoded as plain strings unless a schema explicitly opts in. That is a transport-level (Part 2) decision and is intentionally not enforced here, since gating it would break local round-trips.

Build

sigil deps install
sigil build

Testing

sigil test

License

BSD-3-Clause. See LICENSE.