bench/README.md — sigil-format

1# Tokenizer validation + benchmarks

3## The CLI-embedded-module trap (read first)

5The `sigil` CLI **embeds** `(sigil format ...)` for its `format` subcommand, and

6the embedded modules SHADOW this package's local build in `sigil test`,

7`sigil eval`, and direct file runs. Importing `(sigil format tokenize)` in a

8test or bench therefore exercises the CLI's bundled version, NOT your working

9tree. (Verified 2026-07-16: an export added locally is unbound at runtime;

10SIGIL_LIB / -L / store-cache flush / version bump do not change resolution.)

12Workaround used by these harnesses: create a renamed copy of the module and

13import that — the embedded CLI has no `(sigil format tokenize2)`:

15 sed 's/(define-library (sigil format tokenize)/(define-library (sigil format tokenize2)/' \

16 src/sigil/format/tokenize.sgl > src/sigil/format/tokenize2.sgl

17 sigil build

19The `tokenize2.sgl` copy is gitignored scratch — regenerate it after every

20tokenize.sgl edit, and never commit it.

22## Harnesses

24- `bench-diff.sgl` — differential correctness: compares the local build

25 (via tokenize2) against the v0.16.2 implementation inlined verbatim in the

26 file. Token streams must be byte-identical (all 6 token fields, the chars

27 vector, and token-value strings) across edge cases, this repo's own source

28 files, and a generated corpus. Run: `sigil eval -f bench/bench-diff.sgl`

29 (use `eval -f`, NOT a direct file run — the direct runner swallows all

30 runtime errors silently and exits 0).

31- `bench-perf.sgl` — timing, one measurement per process:

32 `BENCH_IMPL=cli|local BENCH_LINES=n sigil eval -f bench/bench-perf.sgl`.

33 `cli` = the CLI's embedded tokenizer, `local` = the tokenize2 copy.

34 Wall-clock on a busy box is noisy; prefer paired/min-of-N comparisons.