The sigil-wire format (v1)

This is the normative, byte-level specification of the sigil-wire binary format, for anyone implementing an encoder or decoder in another language. The reference implementation is src/sigil/wire.sgl.

All multi-byte integers are little-endian. Lengths and counts are unsigned LEB128 varints. The format carries no schema: every value is self-describing via a leading tag byte.

Message framing

message : header  value
header  : MAGIC(2)  VERSION(1)  FLAGS(1)

Exactly one root value follows the header. Any bytes remaining after the root value has been fully decoded are an error (a frame carries one value).

Varints (unsigned LEB128)

A non-negative integer is encoded as a sequence of 7-bit groups, least significant first. Each byte holds 7 payload bits in its low bits; the high bit (0x80) is set on every byte except the last.

encode(u):
  loop:
    byte = u mod 128
    u    = u div 128
    if u == 0: emit(byte); break
    else:      emit(byte | 0x80)

decode:
  result = 0; mult = 1
  loop:
    b = next_byte()
    result += (b mod 128) * mult
    if b < 0x80: break
    mult *= 128

Examples: 0 → 00, 1 → 01, 127 → 7F, 128 → 80 01, 300 → AC 02.

Implementation note: the reference decoder accumulates with integer multiply/add rather than shift/or. This is deliberate — Sigil's bitwise-shift primitives corrupt values crossing the fixnum/bignum boundary. Other languages may use shift/or freely.

A decoder MUST cap varint length. The reference caps length/count varints at 10 bytes and integer-payload varints at max-int-bytes bytes, and additionally range-checks decoded lengths/counts against the remaining buffer before use.

Values

Each value begins with a 1-byte tag.

A tag not in this table is a hard error. Decoders MUST NOT attempt to skip an unknown tag: without a length the parser cannot know how many bytes to skip, and guessing cascades into misparsing the rest of the frame.

0x02 int — zigzag varint

A single format covers all integers, including arbitrary-precision bignums. Zigzag maps signed to unsigned so small-magnitude negatives stay short:

zigzag(n)   = 2n         if n >= 0
              2|n| - 1   if n < 0
unzigzag(u) = u/2            if u even
              -(u/2) - 1     if u odd   (integer division)

The zigzagged value is then written as an unsigned varint.

Examples: 0 → 02 00, -1 → 02 01, 1 → 02 02, -2 → 02 03, 63 → 02 7E.

0x03 float — IEEE-754 double

Eight bytes holding the raw IEEE-754 binary64 bit pattern, little-endian. Sign in the top bit of the last (most significant) byte.

• +inf = 7FF0000000000000, -inf = FFF0000000000000.
• NaN: any payload with the exponent field all-ones and a non-zero fraction is a NaN. The reference implementation writes whatever bits the host double carries and, on decode, canonicalizes a bit pattern that would collide with the runtime's tagged-value space to a safe quiet NaN (see below).
• -0.0 = 8000000000000000 is represented distinctly and round-trips bit-exactly (the reference implementation uses the host's native IEEE accessor, which preserves the sign bit — even though the language's numeric predicates cannot distinguish -0.0 from +0.0).

Example: 3.14 → 03 1F 85 EB 51 B8 1E 09 40 (bytes are 0x40091EB851EB851F LE).

0x04 string / 0x05 bytevector / 0x06 keyword / 0x07 symbol

A varint byte-count N followed by N bytes copied verbatim. Strings, keywords and symbols carry UTF-8; bytevectors carry raw bytes. The decoder checks N against the remaining buffer before allocating, then copies the body in one memcpy. Keywords and symbols are interned from their UTF-8 name on decode.

Example: "hi" → 04 02 68 69.

0x08 char — codepoint varint

A varint Unicode scalar value. Decoders should reject values above 0x10FFFF.

Example: #\A → 08 41.

0x09 list / 0x0B vector / 0x0C array — counted sequences

A varint element-count K followed by K encoded values in order. Lists, R7RS vectors (#(...)) and Sigil arrays (#[...]) share this shape but use distinct tags so their type is preserved across a round-trip.

Example: (1 2 3) → 09 03 02 02 02 04 02 06.

0x0A dict — counted key/value pairs

A varint pair-count K followed by 2K encoded values: key, value, key, value, … Keys are ordinary values (usually keywords) and are encoded with their own tag. Dict equality is order-independent, so the reference implementation does not commit to a key order.

Example: #{ a: 1 } → 0A 01 06 01 61 02 02 (keyword a, then int 1).

Decoder safety requirements

An implementation reading untrusted bytes MUST:

1. Validate MAGIC and VERSION before anything else.
2. Read every length/count into a bounded varint, then range-check it against the remaining buffer before allocating or copying.
3. Enforce configurable limits: maximum single-body length, maximum collection count, maximum nesting depth, maximum total decoded size, maximum integer magnitude — rejecting before allocation. The total-size limit should charge collection allocation (per-element slot cost), not only string/byte bodies, or a collection-heavy frame escapes the bound.
4. Use a bounds-checked cursor so a truncated or lying frame errors cleanly and never reads out of bounds.
5. Treat an unknown tag, unknown version, or trailing bytes as hard errors.
6. If decode is recursive, the nesting-depth limit also bounds host stack depth; keep its default modest so a deeply-nested frame cannot overflow the stack.
7. Decode a malformed UTF-8 string/keyword/symbol body with defined behavior (lenient replacement, or a clean error) — never a crash or OOB read.

Hardening for untrusted remote transport

Interning a decoded keyword/symbol name is permanent in some hosts (the intern table is not collected). A hostile peer streaming endless unique names can exhaust memory — a risk the per-frame caps do not bound. For untrusted remote traffic, gate symbol/keyword decoding (decode as strings unless a schema opts in) at the transport layer. For local in-process payloads this does not apply.

Versioning

The 1-byte VERSION is the format's major version. A decoder rejects a version it does not implement rather than guessing. New tags or payload shapes that are not backward-compatible bump the version; a capability handshake at connect time (for Familiar/Enclave) negotiates it so a new client and an old node degrade gracefully.