token-value: build with one native list->string (fixes non-ASCII crash); 0.16.3

Message

THE MINIMAL SHAPE, chosen by re-benching the held perf/tokenizer-linear rewrite against the sigil 0.17.15 runtime (native char-iteration: ASCII string-ref fast path, native string->vector/make-string sweep):

  8000 lines (min of 3, alternating, loaded box):
    old scan (CLI-embedded v0.16.2):  1852-2287ms
    full rewrite (perf/tokenizer-linear): 2583-2969ms   <- now a REGRESSION
    this branch (old scan + new token-value): 1785ms    <- old speed + fix
  5000 lines: old 1083ms / rewrite 1882ms / this 1436ms (load noise band)

On 0.17.14 the two scans were wall-clock neutral; 0.17.15's native lift lands HARDER on the old vectorized scan (its dominant cost was the per-token make-string+string-set! extraction, now natively fast) than on the rewrite's class-table machinery. So the scan rewrite no longer earns its keep and is NOT taken.

What IS taken: the token-value crash fix. make-string + per-char string-set! vm-errors on any token containing a non-ASCII char (the byte-width constraint is permanent by design -- the 0.17.15 string cursor depends on it), which broke Slate's highlighter and the formatter's emit path on unicode sources, and crashed tokenize itself on "+lambda"-style tokens mid-scan. token-value now collects chars and converts with ONE native list->string -- also ~2x faster over full token streams.

Differential harness (cherry-picked from the held branch): ALL PASSED against the inlined v0.16.2 (byte-identical on ASCII; unicode diverges only where the old implementation crashed). sigil test 23 green (NB vacuous for local code -- CLI-embedded module shadowing; bench-diff is the real evidence).

Changed

 package.sgl                   |  2 +-
 src/sigil/format/tokenize.sgl | 26 ++++++++++++++++----------
 2 files changed, 17 insertions(+), 11 deletions(-)

Diff

package.sglmodified

@@ -5,7 +5,7 @@

(package

name: "sigil-format"

−8

version: "0.16.2"

version: "0.16.3"

sigil: "^0.17"

description: "Sigil code formatter (paren-inference + AST-aware reflow)"

url: "https://codeberg.org/sigil/sigil-format"

src/sigil/format/tokenize.sglmodified

@@ -68,17 +68,23 @@

(define (token-indent tok) (vector-ref tok 5))

;;; Extract the text value of a token from the source chars vector

+71

;; Build the token's text with ONE native list->string call. The previous

+72

;; make-string + per-char string-set! implementation VM-ERRORED on any

+73

;; token containing a non-ASCII char ("string-set!: character byte width

+74

;; change not supported" -- make-string pre-sizes 1-byte cells and the

+75

;; byte-width constraint is permanent by design, the string cursor depends

+76

;; on it). That crash broke Slate's highlighter (token-value per symbol

+77

;; token) and the formatter's emit path on unicode Sigil sources, and made

+78

;; tokenize itself crash on tokens like "+\u03bb" (looks-like-number?

+79

;; extracts the text mid-scan). Collect-and-convert is also ~2x faster

+80

;; over full token streams (measured on the 0.17.x bytecode VM).

(define (token-value tok chars)

−72

(let* ((s (vector-ref tok 1))

−73

(e (vector-ref tok 2))

−74

(n (- e s))

−75

(v (make-string n)))

−76

(let loop ((i 0))

−77

(if (>= i n)

−78

v

−79

(begin

−80

(string-set! v i (vector-ref chars (+ s i)))

−81

(loop (+ i 1)))))))

+82

(let ((s (vector-ref tok 1))

+83

(e (vector-ref tok 2)))

+84

(let loop ((i (- e 1)) (acc '()))

+85

(if (< i s)

+86

(list->string acc)

+87

(loop (- i 1) (cons (vector-ref chars i) acc))))))

;;; Get the chars vector from a tokenize result

(define (tokenize-result-chars result) (car result))