README.md — sigil-wire

1# sigil-wire

3A typed, length-prefixed **binary codec** for the [Sigil](https://codeberg.org/sigil/sigil)

4value model. The binary sibling to

5[sigil-json](https://codeberg.org/sigil/sigil-json): where JSON is the

6human-readable format for small control messages, sigil-wire is the compact,

7fast format for **bulk structured payloads**.

9Values decode in a single O(n) forward pass, and string/bytevector bodies are

10copied wholesale (`memcpy`), avoiding the per-character O(n²) work that JSON

11encoding of Sigil values incurs. It is the transport-neutral wire format used by

12the Slate/Lantern bulk channel and inherited by Familiar over Enclave.

14## Usage

16```scheme

17(import (sigil wire))

19;; Encode any Sigil value to a self-describing bytevector.

20(define bytes (wire-encode #{ name: "Alice" scores: #[10 20 30] }))

22;; Decode it back.

23(wire-decode bytes)

24; => #{ name: "Alice" scores: #[10 20 30] }

25```

27The decoder is a **trust boundary**: bytes may be attacker-influenced (over

28Enclave). Pass a caps record to bound the work a hostile frame can trigger:

30```scheme

31(wire-decode bytes (make-wire-caps max-depth: 32 max-count: 1000))

32```

34## API

36| Procedure | Purpose |

37|-----------|---------|

38| `(wire-encode value)` → bytevector | Serialize a Sigil value. |

39| `(wire-decode bytevector [caps])` → value | Deserialize, bounded by `caps`. |

40| `(make-wire-caps [keys])` → caps | Build a decode-limits record. |

41| `default-wire-caps` | The default caps used when none is passed. |

43### Caps (decoder limits)

45`make-wire-caps` accepts keyword arguments; any omitted field takes its default:

47| Field | Default | Meaning |

48|-------|---------|---------|

49| `max-bytes-len:` | 64 MiB | Largest single string / bytevector body. |

50| `max-count:` | 16 M | Largest collection (list/vector/array length, dict pairs). |

| `max-depth:` | 256 | Deepest nesting of collections. Decode is recursive, so this **also bounds host stack depth** — keep it modest; cranking it to tens of thousands reintroduces stack-overflow risk on a hostile deeply-nested frame. |

| `max-total:` | 256 MiB | Total decoded heap: string/bytevector body bytes **plus** an estimated per-element cost for every collection slot, so a collection-heavy frame of tiny elements is bounded too. |

53| `max-int-bytes:` | 1024 | Largest integer payload in varint bytes (bounds bignum size). |

55## Supported value types

57Every value the seam carries: `#f`, `#t`, exact integers (including bignums),

58doubles, strings, bytevectors, keywords, symbols, characters, lists, dicts,

59vectors (R7RS `#(...)`) and arrays (Sigil `#[...]`).

61A value that **cannot** be represented — a procedure, a port, a record without a

62registered codec, an improper (dotted) list, or a non-real number — is an

63**encode error**, never a silent drop. (Registered-tag record extensions are out

64of scope for v1.)

66## Wire format

68A message is a fixed 4-byte header followed by exactly one root value:

70```

71message : MAGIC(2) VERSION(1) FLAGS(1) <value>

72 0x53 0x57 0x01 0x00

74value : TAG(1) <payload>

75```

77Decoders reject an unknown MAGIC or an unknown major VERSION. `FLAGS` is reserved

78(0). Trailing bytes after the root value are a decode error.

80Lengths and counts are unsigned **LEB128** varints (little-endian base-128; the

81high bit of each byte flags continuation). Small collections stay a single byte.

83### Tag table

85| TAG | Type | Payload |

86|------|------------|---------|

87| 0x00 | `#f` / nil | (none) |

88| 0x01 | `#t` | (none) |

89| 0x02 | int | zigzag LEB128 varint (bignum-safe, single format) |

90| 0x03 | float | 8 bytes, IEEE-754 double, little-endian |

91| 0x04 | string | varint N, then N UTF-8 bytes (memcpy) |

92| 0x05 | bytevector | varint N, then N raw bytes (memcpy) |

93| 0x06 | keyword | varint N, then N UTF-8 bytes |

94| 0x07 | symbol | varint N, then N UTF-8 bytes |

95| 0x08 | char | varint Unicode codepoint |

96| 0x09 | list | varint K, then K values |

97| 0x0A | dict | varint K, then K (key value) pairs |

98| 0x0B | vector | varint K, then K values (R7RS `#(...)`) |

99| 0x0C | array | varint K, then K values (Sigil `#[...]`) |

100

101Any tag outside this table is a **hard decode error** (never skip-and-continue —

102a misparsed length would cascade).

103

104### Integers

105

106A single **zigzag LEB128** varint encodes every integer, from small values up to

107arbitrary-precision bignums — there is no separate bignum form. Zigzag maps

108signed to unsigned so small negatives stay short: `n >= 0 → 2n`, `n < 0 →

1092|n| - 1`. The varint then encodes that non-negative value.

110

111### Floats

112

113Doubles are stored as their raw 64-bit IEEE-754 bit pattern, little-endian, via

114the native `bytevector-ieee-double-{ref,set!}` accessors. Every double — normal,

115subnormal, `±0.0`, `±inf`, `NaN` — round-trips **bit-exactly**, including the sign

116bit of `-0.0` (even though Sigil predicates cannot themselves distinguish `-0.0`

117from `+0.0`). On decode, the accessor canonicalizes any hostile bit pattern to a

118safe quiet-NaN flonum, so a malicious float body can never produce a

119type-confused value.

120

121## Security

122

123The decoder never trusts a length it reads:

124

125- Every length prefix is checked against the remaining buffer **before** any

126 allocation or copy — it never pre-allocates N bytes from an untrusted varint.

127- All caps (body length, collection count, nesting depth, total decoded size,

128 integer magnitude) are enforced, rejecting **before** allocating.

129- A bounds-checked cursor means a truncated or lying frame errors cleanly and

130 never reads out of bounds; an endless varint is rejected rather than spun on.

131

132The test suite includes a hostile-input fuzz battery (truncated frames, lying

133length prefixes, cap violations, unknown tags, unknown version, varint abuse,

134invalid UTF-8 bodies) asserting each errors cleanly (or, for malformed UTF-8,

135decodes leniently to a string — never a crash or out-of-bounds read).

136

137### Harden before untrusted remote (Enclave) use

138

139One residual risk is **not** bounded by the per-frame caps: decoding a keyword or

140symbol *interns* its name permanently (the intern table is not garbage

141collected). A hostile remote peer that streams an endless supply of unique

142keyword/symbol names can exhaust memory over time. This is safe for local,

143in-process bulk payloads (the current use), but before the codec carries

144untrusted *remote* traffic over Enclave, symbol/keyword decoding should be gated

145— decoded as plain strings unless a schema explicitly opts in. That is a

146transport-level (Part 2) decision and is intentionally not enforced here, since

147gating it would break local round-trips.

148

149## Build

150

151```sh

152sigil deps install

153sigil build

154```

155

156## Testing

157

158```sh

159sigil test

160```

161

162## License

163

164BSD-3-Clause. See [LICENSE](LICENSE).