Use native bytevector-ieee-double accessors for the float path

Message

Swap the float encode/decode from the pure-arithmetic IEEE-754 bit derivation to sigil-stdlib's native bytevector-ieee-double-ref/set! (added in 0.17.18 alongside the bignum bitwise-correctness fix):

- encode-float: bytevector-ieee-double-set! ... 'little; decode-float: bytevector-ieee-double-ref (cur-take! c 8) 0 'little. Deletes ~85 lines (double->bits/bits->double/pack-bits/POW2/WIRE-INF/NAN). - -0.0 now round-trips bit-exactly (the accessor preserves the sign bit), superseding the earlier collapse-to-+0.0 limitation. - NaN-box safety on decode: -ref canonicalizes any hostile bit pattern to a safe quiet-NaN flonum, so a malicious float body can never yield a type-confused value. Added decode tests (tag-zone + all-ones bits). - Varint path kept as pure arithmetic on purpose: correct on any sigil (incl. older/wasm builds), the bitwise fix notwithstanding.

Verified 126/126 against a dev sigil (0.17.18-dev+0bb453df) via dev-redirects.sgl. README + docs/wire.md updated.

Changed

 README.md          |  11 +++++++----
 docs/wire.md       |  15 ++++++++-------
 src/sigil/wire.sgl | 124 +++++++++++++++++++++++-----------------------------------------------------------------------------------------------------
 test/test-wire.sgl |  24 +++++++++++++++++++++++-
 4 files changed, 61 insertions(+), 113 deletions(-)

Diff

README.mdmodified

@@ -110,10 +110,13 @@ signed to unsigned so small negatives stay short: `n >= 0 → 2n`, `n < 0 →

110

111

### Floats

112

−113

Doubles are stored as their raw 64-bit IEEE-754 bit pattern, little-endian.

−114

`+inf`, `-inf` and `NaN` round-trip bit-exactly. Note that Sigil cannot

−115

distinguish `-0.0` from `+0.0` (they compare equal under every predicate and

−116

print identically), so a negative zero encodes as `+0.0`.

+113

Doubles are stored as their raw 64-bit IEEE-754 bit pattern, little-endian, via

+114

the native `bytevector-ieee-double-{ref,set!}` accessors. Every double — normal,

+115

subnormal, `±0.0`, `±inf`, `NaN` — round-trips **bit-exactly**, including the sign

+116

bit of `-0.0` (even though Sigil predicates cannot themselves distinguish `-0.0`

+117

from `+0.0`). On decode, the accessor canonicalizes any hostile bit pattern to a

+118

safe quiet-NaN flonum, so a malicious float body can never produce a

+119

type-confused value.

120

121

## Security

122

docs/wire.mdmodified

@@ -104,13 +104,14 @@ Eight bytes holding the raw IEEE-754 binary64 bit pattern, little-endian. Sign i

104

the top bit of the last (most significant) byte.

105

106

- `+inf` = `7FF0000000000000`, `-inf` = `FFF0000000000000`.

−107

- `NaN` is written canonically as `7FF8000000000000` (any incoming NaN

−108

normalizes to this on encode); on decode any payload with exponent field all-1

−109

and non-zero fraction is a NaN.

−110

- `-0.0` is not represented distinctly by the reference implementation (Sigil

−111

cannot distinguish it from `+0.0`); it encodes as `+0.0` = `0000000000000000`.

−112

A decoder that receives `8000000000000000` should still produce `-0.0` where

−113

the host supports it.

+107

- `NaN`: any payload with the exponent field all-ones and a non-zero fraction is

+108

a NaN. The reference implementation writes whatever bits the host double

+109

carries and, on decode, canonicalizes a bit pattern that would collide with the

+110

runtime's tagged-value space to a safe quiet NaN (see below).

+111

- `-0.0` = `8000000000000000` is represented distinctly and round-trips

+112

bit-exactly (the reference implementation uses the host's native IEEE accessor,

+113

which preserves the sign bit — even though the language's numeric predicates

+114

cannot distinguish `-0.0` from `+0.0`).

115

116

Example: `3.14 → 03 1F 85 EB 51 B8 1E 09 40` (bytes are `0x40091EB851EB851F` LE).

117

src/sigil/wire.sglmodified

@@ -150,12 +150,13 @@

150

151

;; ========== Varint (unsigned LEB128) ==========

152

−153

;; NOTE: the varint and float-bit paths use only arithmetic (quotient /

−154

;; remainder / * / +), never bitwise-and / bitwise-ior / arithmetic-shift.

−155

;; Sigil's bitwise/shift primitives mis-handle values straddling the

−156

;; fixnum<->bignum boundary (e.g. round-tripping 2^62 or 7^500 through

−157

;; shift+ior corrupts the value), whereas plain integer arithmetic is

−158

;; correct for arbitrary-precision integers. See the wire-format topic note.

+153

;; NOTE: the varint path uses only arithmetic (quotient / remainder / * / +),

+154

;; never bitwise-and / bitwise-ior / arithmetic-shift. Older Sigil bitwise/

+155

;; shift primitives mis-handle values straddling the fixnum<->bignum boundary

+156

;; (round-tripping 2^62 or 7^500 through shift+ior corrupted the value); that

+157

;; VM bug is fixed as of 0.17.18, but the arithmetic form is kept because it

+158

;; is correct on ANY sigil (incl. older and wasm builds) at no real cost.

+159

;; See the wire-format topic note.

160

161

;; Write a non-negative integer as unsigned LEB128 to a port.

162

(define (write-uvarint u port)

@@ -180,90 +181,16 @@

181

(quotient u 2)

182

(- (- (quotient u 2)) 1)))

183

−183

;; ========== Float <-> IEEE-754 bits ==========

+184

;; ========== Float <-> IEEE-754 bytes ==========

185

;;

−185

;; Sigil's numeric tower is exact integers (incl. bignums) and inexact

−186

;; doubles — there are NO exact rationals — so `(exact <float>)` errors on a

−187

;; non-integer. We therefore derive the 64-bit IEEE-754 representation with

−188

;; float arithmetic + exact-integer ops, never touching a rational. Every

−189

;; scaling factor stays within the representable double range (|exponent| <=

−190

;; 1022 per step) so no intermediate overflows to infinity.

−191

−192

(define POW2-52 (expt 2 52)) ; 4503599627370496

−193

(define POW2-63 (expt 2 63)) ; sign-bit place value

−194

−195

;; Sigil has no infinity/NaN literals, but doubles CAN be inf/NaN (e.g. from

−196

;; overflow). Build them once, by arithmetic, to reconstruct on decode.

−197

(define WIRE-POS-INF (expt 2.0 2000)) ; overflows to +inf

−198

(define WIRE-NEG-INF (- (expt 2.0 2000))) ; -inf

−199

(define WIRE-NAN (- WIRE-POS-INF WIRE-POS-INF)) ; inf - inf = NaN

−200

−201

;; Assemble sign/biased-exponent/fraction into a 64-bit unsigned integer.

−202

(define (pack-bits sign biased frac)

−203

(+ (* sign POW2-63) (* biased POW2-52) frac))

−204

−205

;; Encode a double as its unsigned 64-bit IEEE-754 bit pattern.

−206

;; NaN is checked BEFORE zero: Sigil's `(= nan 0.0)` returns #t, so a

−207

;; zero-first test would mis-encode NaN as +0.0.

−208

(define (double->bits x)

−209

(cond

−210

;; NaN: canonical quiet NaN (0x7FF8000000000000); fraction = 2^51.

−211

((nan? x) (pack-bits 0 2047 (quotient POW2-52 2)))

−212

;; +/- infinity.

−213

((not (finite? x))

−214

(if (< x 0.0) (pack-bits 1 2047 0) (pack-bits 0 2047 0)))

−215

;; Zero (positive and negative zero both encode as +0.0; Sigil does not

−216

;; distinguish them under `=`, `eqv?`, `equal?`, or `number->string`).

−217

((= x 0.0) 0)

−218

(else

−219

(let* ((neg (< x 0.0))

−220

(sign (if neg 1 0))

−221

(ax (abs x)))

−222

(if (< ax (expt 2.0 -1022))

−223

;; Subnormal: significand = round(ax * 2^1074), scaled in two

−224

;; representable steps (2^1074 itself overflows a double).

−225

(let ((frac (exact (round (* (* ax (expt 2.0 1022))

−226

(expt 2.0 52))))))

−227

(pack-bits sign 0 frac))

−228

;; Normal: find unbiased exponent e with 2^e <= ax < 2^(e+1).

−229

(let loop ((e (exact (floor (/ (log ax) (log 2.0))))))

−230

(cond

−231

((<= (expt 2.0 (+ e 1)) ax) (loop (+ e 1)))

−232

((> (expt 2.0 e) ax) (loop (- e 1)))

−233

(else

−234

;; ax/2^e is exactly a double in [1,2); *2^52 is an exact

−235

;; integer significand in [2^52, 2^53).

−236

(let* ((sig (exact (round (* (/ ax (expt 2.0 e))

−237

(expt 2.0 52)))))

−238

(frac (- sig POW2-52))

−239

(biased (+ e 1023)))

−240

(pack-bits sign biased frac))))))))))

−241

−242

;; Decode an unsigned 64-bit IEEE-754 bit pattern to a double.

−243

(define (bits->double bits)

−244

(let* ((sign (quotient bits POW2-63))

−245

(rest (remainder bits POW2-63))

−246

(biased (quotient rest POW2-52))

−247

(frac (remainder rest POW2-52)))

−248

(cond

−249

;; Inf / NaN carry their own sign; return directly.

−250

((= biased 2047)

−251

(cond

−252

((not (= frac 0)) WIRE-NAN)

−253

((= sign 1) WIRE-NEG-INF)

−254

(else WIRE-POS-INF)))

−255

(else

−256

(let ((mag (cond

−257

;; Zero / subnormal.

−258

((= biased 0)

−259

(if (= frac 0)

−260

0.0

−261

(* frac (expt 2.0 -1074))))

−262

;; Normal.

−263

(else

−264

(* (+ frac POW2-52)

−265

(expt 2.0 (- biased 1075)))))))

−266

(if (= sign 1) (- mag) mag))))))

+186

;; The wire float is the raw 8-byte IEEE-754 double, LITTLE-ENDIAN (fixed by

+187

;; the format, independent of host byte order). Sigil-stdlib's native

+188

;; `bytevector-ieee-double-{ref,set!}` do the exact conversion, so every

+189

;; double — normal, subnormal, +/-0.0, +/-inf, NaN — round-trips bit-exactly.

+190

;; Crucially for the trust boundary, `-ref` is NaN-box-safe: it canonicalizes

+191

;; any hostile bit pattern (including ones that would collide with the VM's

+192

;; immediate tag space) to a genuine quiet-NaN flonum, never a type-confused

+193

;; Value. It also bounds-checks the offset/length itself.

194

195

;; ========== Encoding ==========

196

@@ -346,12 +273,10 @@

273

(error "wire-encode: value is not representable on the wire" value))))

274

275

(define (encode-float value port)

−349

(let ((bits (double->bits value)))

−350

;; 8 bytes, little-endian (arithmetic, not bitwise).

−351

(let loop ((i 0))

−352

(when (< i 8)

−353

(write-u8 (remainder (quotient bits (expt 256 i)) 256) port)

−354

(loop (+ i 1))))))

+276

;; Raw 8-byte IEEE-754 double, little-endian (exact bits, incl. -0.0).

+277

(let ((bv (make-bytevector 8 0)))

+278

(bytevector-ieee-double-set! bv 0 value 'little)

+279

(write-bytevector bv port)))

280

281

(define (encode-list value port)

282

;; Walk once: verify the list is proper and count length, collecting the

@@ -530,12 +455,9 @@

455

(cur-take! c n)))

456

457

(define (decode-float c)

−533

(cur-need! c 8)

−534

(let loop ((i 0) (bits 0))

−535

(if (= i 8)

−536

(bits->double bits)

−537

(loop (+ i 1)

−538

(+ bits (* (cur-u8! c) (expt 256 i)))))))

+458

;; cur-take! bounds-checks the 8 bytes; bytevector-ieee-double-ref

+459

;; canonicalizes any hostile bit pattern to a safe flonum.

+460

(bytevector-ieee-double-ref (cur-take! c 8) 0 'little))

461

462

(define (decode-char c)

463

(let ((cp (read-uvarint c LEN-VARINT-MAX-BYTES)))

test/test-wire.sglmodified

@@ -77,7 +77,29 @@

(test "+inf stays infinite" (assert-true (infinite? (rt (expt 2.0 2000)))))

(test "-inf stays negative" (assert-true (< (rt (- (expt 2.0 2000))) 0.0)))

(test "NaN stays NaN"

−80

(assert-true (nan? (rt (- (expt 2.0 2000) (expt 2.0 2000)))))))

+80

(assert-true (nan? (rt (- (expt 2.0 2000) (expt 2.0 2000))))))

+81

;; The native IEEE accessor preserves the sign bit, so -0.0 now round-trips

+82

;; bit-exactly. equal? cannot distinguish +/-0.0, so compare the wire bytes:

+83

;; -0.0 and +0.0 must encode to DIFFERENT frames (sign bit set only for -0.0).

+84

(test "-0.0 round-trips (decodes to a zero)"

+85

(assert-equal 0.0 (rt (- 0.0))))

+86

(test "-0.0 preserves its sign bit on the wire (distinct from +0.0)"

+87

(assert-false (equal? (wire-encode (- 0.0)) (wire-encode 0.0)))))

+88

+89

(test-group "decode - hostile float bit patterns (NaN-box safety)"

+90

;; A float body may carry any 8 bytes. bytevector-ieee-double-ref canonicalizes

+91

;; bit patterns that would collide with the VM's immediate tag space to a safe

+92

;; quiet-NaN flonum — never a type-confused Value, crash, or OOB read.

+93

(test "tag-zone NaN 0x7FFC... reads as a safe NaN flonum"

+94

(let ((r (guard (e (#t 'clean-error))

+95

(wire-decode (frame #x03 0 0 0 0 0 0 #xFC #x7F)))))

+96

(assert-true (or (eq? r 'clean-error) (and (number? r) (nan? r))))))

+97

(test "all-ones float bits read as a safe NaN flonum"

+98

(let ((r (guard (e (#t 'clean-error))

+99

(wire-decode (frame #x03 #xFF #xFF #xFF #xFF #xFF #xFF #xFF #xFF)))))

+100

(assert-true (or (eq? r 'clean-error) (and (number? r) (nan? r))))))

+101

(test "float body truncated to 4 bytes errors cleanly"

+102

(assert-true (raises? (lambda () (wire-decode (frame #x03 0 0 0 0)))))))

103

104

(test-group "round-trip - strings"

105

(test "empty" (assert-equal "" (rt "")))