Tokenizer validation + benchmarks

The CLI-embedded-module trap (read first)

The sigil CLI embeds (sigil format ...) for its format subcommand, and the embedded modules SHADOW this package's local build in sigil test, sigil eval, and direct file runs. Importing (sigil format tokenize) in a test or bench therefore exercises the CLI's bundled version, NOT your working tree. (Verified 2026-07-16: an export added locally is unbound at runtime; SIGIL_LIB / -L / store-cache flush / version bump do not change resolution.)

Workaround used by these harnesses: create a renamed copy of the module and import that — the embedded CLI has no (sigil format tokenize2):

sed 's/(define-library (sigil format tokenize)/(define-library (sigil format tokenize2)/' \
    src/sigil/format/tokenize.sgl > src/sigil/format/tokenize2.sgl
sigil build

The tokenize2.sgl copy is gitignored scratch — regenerate it after every tokenize.sgl edit, and never commit it.

Harnesses

• bench-diff.sgl — differential correctness: compares the local build (via tokenize2) against the v0.16.2 implementation inlined verbatim in the file. Token streams must be byte-identical (all 6 token fields, the chars vector, and token-value strings) across edge cases, this repo's own source files, and a generated corpus. Run: sigil eval -f bench/bench-diff.sgl (use eval -f, NOT a direct file run — the direct runner swallows all runtime errors silently and exits 0).
• bench-perf.sgl — timing, one measurement per process: BENCH_IMPL=cli|local BENCH_LINES=n sigil eval -f bench/bench-perf.sgl. cli = the CLI's embedded tokenizer, local = the tokenize2 copy. Wall-clock on a busy box is noisy; prefer paired/min-of-N comparisons.