docs/wire.md — sigil-wire

1# The sigil-wire format (v1)

3This is the normative, byte-level specification of the sigil-wire binary format,

4for anyone implementing an encoder or decoder in another language. The reference

5implementation is `src/sigil/wire.sgl`.

7All multi-byte integers are little-endian. Lengths and counts are unsigned

8LEB128 varints. The format carries no schema: every value is self-describing via

9a leading tag byte.

11## Message framing

13```

14message : header value

15header : MAGIC(2) VERSION(1) FLAGS(1)

16```

19|-------|-------|-------|-------|

21| VERSION | 1 | `0x01` | Decoders reject an unknown major version. |

22| FLAGS | 1 | `0x00` | Reserved; currently ignored on decode. |

24Exactly one root `value` follows the header. Any bytes remaining after the root

25value has been fully decoded are an error (a frame carries one value).

27## Varints (unsigned LEB128)

29A non-negative integer is encoded as a sequence of 7-bit groups, least

30significant first. Each byte holds 7 payload bits in its low bits; the high bit

31(`0x80`) is set on every byte except the last.

33```

34encode(u):

35 loop:

36 byte = u mod 128

37 u = u div 128

38 if u == 0: emit(byte); break

39 else: emit(byte | 0x80)

41decode:

42 result = 0; mult = 1

43 loop:

44 b = next_byte()

45 result += (b mod 128) * mult

46 if b < 0x80: break

47 mult *= 128

48```

50Examples: `0 → 00`, `1 → 01`, `127 → 7F`, `128 → 80 01`, `300 → AC 02`.

52> Implementation note: the reference decoder accumulates with integer

53> multiply/add rather than shift/or. This is deliberate — Sigil's bitwise-shift

54> primitives corrupt values crossing the fixnum/bignum boundary. Other languages

55> may use shift/or freely.

57A decoder MUST cap varint length. The reference caps length/count varints at 10

58bytes and integer-payload varints at `max-int-bytes` bytes, and additionally

59range-checks decoded lengths/counts against the remaining buffer before use.

61## Values

63Each value begins with a 1-byte tag.

65| TAG | Type | Payload |

66|-----|------|---------|

67| `0x00` | `#f` / nil | none |

68| `0x01` | `#t` | none |

69| `0x02` | int | zigzag varint |

70| `0x03` | float | 8 bytes IEEE-754 LE |

71| `0x04` | string | varint N + N UTF-8 bytes |

72| `0x05` | bytevector | varint N + N raw bytes |

73| `0x06` | keyword | varint N + N UTF-8 bytes |

74| `0x07` | symbol | varint N + N UTF-8 bytes |

75| `0x08` | char | varint codepoint |

76| `0x09` | list | varint K + K values |

77| `0x0A` | dict | varint K + K (key value) pairs |

78| `0x0B` | vector | varint K + K values |

79| `0x0C` | array | varint K + K values |

81A tag not in this table is a hard error. Decoders MUST NOT attempt to skip an

82unknown tag: without a length the parser cannot know how many bytes to skip, and

83guessing cascades into misparsing the rest of the frame.

85### 0x02 int — zigzag varint

87A single format covers all integers, including arbitrary-precision bignums.

88Zigzag maps signed to unsigned so small-magnitude negatives stay short:

90```

91zigzag(n) = 2n if n >= 0

92 2|n| - 1 if n < 0

93unzigzag(u) = u/2 if u even

94 -(u/2) - 1 if u odd (integer division)

95```

97The zigzagged value is then written as an unsigned varint.

99Examples: `0 → 02 00`, `-1 → 02 01`, `1 → 02 02`, `-2 → 02 03`, `63 → 02 7E`.

100

101### 0x03 float — IEEE-754 double

102

103Eight bytes holding the raw IEEE-754 binary64 bit pattern, little-endian. Sign in

104the top bit of the last (most significant) byte.

105

106- `+inf` = `7FF0000000000000`, `-inf` = `FFF0000000000000`.

107- `NaN`: any payload with the exponent field all-ones and a non-zero fraction is

108 a NaN. The reference implementation writes whatever bits the host double

109 carries and, on decode, canonicalizes a bit pattern that would collide with the

110 runtime's tagged-value space to a safe quiet NaN (see below).

111- `-0.0` = `8000000000000000` is represented distinctly and round-trips

112 bit-exactly (the reference implementation uses the host's native IEEE accessor,

113 which preserves the sign bit — even though the language's numeric predicates

114 cannot distinguish `-0.0` from `+0.0`).

115

116Example: `3.14 → 03 1F 85 EB 51 B8 1E 09 40` (bytes are `0x40091EB851EB851F` LE).

117

118### 0x04 string / 0x05 bytevector / 0x06 keyword / 0x07 symbol

119

120A varint byte-count `N` followed by `N` bytes copied verbatim. Strings, keywords

121and symbols carry UTF-8; bytevectors carry raw bytes. The decoder checks `N`

122against the remaining buffer before allocating, then copies the body in one

123`memcpy`. Keywords and symbols are interned from their UTF-8 name on decode.

124

125Example: `"hi" → 04 02 68 69`.

126

127### 0x08 char — codepoint varint

128

129A varint Unicode scalar value. Decoders should reject values above `0x10FFFF`.

130

131Example: `#\A → 08 41`.

132

133### 0x09 list / 0x0B vector / 0x0C array — counted sequences

134

135A varint element-count `K` followed by `K` encoded values in order. Lists, R7RS

136vectors (`#(...)`) and Sigil arrays (`#[...]`) share this shape but use distinct

137tags so their type is preserved across a round-trip.

138

139Example: `(1 2 3) → 09 03 02 02 02 04 02 06`.

140

141### 0x0A dict — counted key/value pairs

142

143A varint pair-count `K` followed by `2K` encoded values: `key`, `value`, `key`,

144`value`, … Keys are ordinary values (usually keywords) and are encoded with their

145own tag. Dict equality is order-independent, so the reference implementation does

146not commit to a key order.

147

148Example: `#{ a: 1 } → 0A 01 06 01 61 02 02` (keyword `a`, then int `1`).

149

150## Decoder safety requirements

151

152An implementation reading untrusted bytes MUST:

153

1541. Validate MAGIC and VERSION before anything else.

1552. Read every length/count into a bounded varint, then range-check it against the

156 remaining buffer **before** allocating or copying.

1573. Enforce configurable limits: maximum single-body length, maximum collection

158 count, maximum nesting depth, maximum total decoded size, maximum integer

159 magnitude — rejecting **before** allocation. The total-size limit should

160 charge collection allocation (per-element slot cost), not only string/byte

161 bodies, or a collection-heavy frame escapes the bound.

1624. Use a bounds-checked cursor so a truncated or lying frame errors cleanly and

163 never reads out of bounds.

1645. Treat an unknown tag, unknown version, or trailing bytes as hard errors.

1656. If decode is recursive, the nesting-depth limit also bounds host stack depth;

166 keep its default modest so a deeply-nested frame cannot overflow the stack.

1677. Decode a malformed UTF-8 string/keyword/symbol body with defined behavior

168 (lenient replacement, or a clean error) — never a crash or OOB read.

169

170## Hardening for untrusted remote transport

171

172Interning a decoded keyword/symbol name is permanent in some hosts (the intern

173table is not collected). A hostile peer streaming endless unique names can

174exhaust memory — a risk the per-frame caps do not bound. For untrusted *remote*

175traffic, gate symbol/keyword decoding (decode as strings unless a schema opts in)

176at the transport layer. For local in-process payloads this does not apply.

177

178## Versioning

179

180The 1-byte VERSION is the format's major version. A decoder rejects a version it

181does not implement rather than guessing. New tags or payload shapes that are not

182backward-compatible bump the version; a capability handshake at connect time (for

183Familiar/Enclave) negotiates it so a new client and an old node degrade

184gracefully.