Read each file once, and serialise a listing without building a node per line

Message

Measured before changing anything, and the ticket's diagnosis did not survive the measurement. t-0361 says per-page cost grows linearly with repo count. It does not: adding 32 repositories to a 1,780-page render costs -24 ms beyond what those 32 repositories cost alone, against the +131,007 ms that model predicts. Its four points varied the repo LIST over wildly heterogeneous repositories, so repo count, which repositories were in the list, and pages-per-repo all moved together. The rising ms/page is composition, not a quadratic.

Two hypotheses died before a line of this diff was written. The second was git: a shim named git at the front of PATH measures 279 calls totalling 1,951 ms against a 101,444 ms wall -- 1.9%. Batching the four spawns per commit page into one log walk would save 1.5 s in 101 s, so it is not done here, though test/git-batch-equivalence.py establishes it would be byte-identical if anyone wants it later.

Where the time actually goes, from temporary accumulators around each page type on three real repositories:

  blob pages      78,773 ms / 1059 pages   68%
  exclusion scan  25,543 ms                22%
  tree pages       5,823 ms /  330
  commit pages     5,717 ms /   60
  write path         265 ms

The filesystem is 265 ms. The cost is in-process rendering of file content, at 7.4 s per MB of tip source -- cross-checked by an isolated single-file probe at 7.3 s/MB.

So, two things:

EVERY TRACKED FILE WAS READ THREE TIMES. unrenderable-reason ran once to build the excluded list and again to build the leaves list, each call reading the whole file to inspect its first 8000 characters for a NUL, and the page loop read it a third time to render it. file-verdict replaces it and returns (reason . text), so the read happens once and the text travels with the verdict. The NUL search is now one native string-find over an 8000-character prefix rather than a Scheme loop calling string-ref and char->integer per character. The 8000 rule is kept deliberately: a NUL at position 9000 does not make a file binary, and a whole-string search would refuse a file that used to render.

A SOURCE LISTING BUILT ONE SXML NODE PER LINE -- about 335,000 of them for the sigil monorepo alone -- and sxml->xml walks a three-element tree per line while its xml-escape displays ONE CHARACTER AT A TIME into a string port. The listing is now serialised directly into a single raw node: 1,150 ms to 342 ms on a 368 KB source, outputs compared byte for byte. This is the only place in the renderer that bypasses SXML and the comment says why -- a listing is the one surface whose node count scales with the size of the input rather than with the design, and nothing else here is worth the special case.

The residue is xml-escape itself, which lives in sigil-sxml. Written up rather than fixed: another repository, its own release.

Nothing rendered changes. test/render-identical.sh compares a (path, sha256) manifest of every file of two fresh output trees, with the estate descriptor, a quarantine submission and --site-url all supplied so the estate home, the estate report page, the unconfirmed issue page and both feed kinds are in scope. GREEN at 1,794 files over the first three real estate repositories, and at 317 over five small ones. Sabotage-tested: one word changed in the listing markup turns it RED naming every affected page; restored from a hash-verified copy, GREEN again. All ten integration suites pass.

Changed

 src/docket/render.sgl    | 153 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++------------------------------------
 test/render-fourpoint.sh | 113 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 230 insertions(+), 36 deletions(-)

Diff

src/docket/render.sglmodified

@@ -1468,7 +1468,7 @@

1468

;;

1469

;; Paths that are STILL quoted after this (a newline or a quote in the

1470

;; name) are reported by name rather than dropped in silence -- see

−1471

;; unrenderable-reason.

+1471

;; file-verdict.

1472

(define (repo-tip-files dir)

1473

(let ((out (process-output->string "git" "-C" dir "-c" "core.quotePath=false"

1474

"ls-tree" "-r" "--name-only" "HEAD")))

@@ -1495,44 +1495,57 @@

1495

(car xs))

1496

(else (loop (cdr xs))))))

1497

−1498

(define (unrenderable-reason dir rel excludes)

+1498

;; EVERY TRACKED FILE IS READ EXACTLY ONCE PER RENDER, and this shape is

+1499

;; what enforces it rather than a comment asking callers to be careful.

+1500

;;

+1501

;; It used to be read THREE times: `unrenderable-reason` was called once to

+1502

;; build the excluded list and a second time to build the leaves list, each

+1503

;; call reading the WHOLE file to look at its first 8000 characters for a

+1504

;; NUL, and then the page loop read it a third time to render it. Measured

+1505

;; on three real repositories (1,059 pages): the two scanning passes alone

+1506

;; cost 25.5 s of a 116 s render, 22%, for an answer already in hand.

+1507

;;

+1508

;; So the read happens HERE, once, and the text travels with the verdict.

+1509

;; `read-file` returns a pair of (reason . text): the reason is #f when the

+1510

;; file gets a page, and the text is #f when there is nothing to render.

+1511

(define (file-verdict dir rel excludes)

1512

(define (contains? s ch)

1513

(let ((n (string-length s)))

1514

(let loop ((i 0))

1515

(cond ((>= i n) #f) ((eq? (string-ref s i) ch) #t) (else (loop (+ i 1)))))))

1516

(cond ((excluded-by? rel excludes)

1517

=> (lambda (which)

−1505

(string-append "excluded by .docket.json: " which)))

+1518

(cons (string-append "excluded by .docket.json: " which) #f)))

1519

((string-starts-with? rel "\"")

−1507

"git quotes this path, so it cannot be opened")

+1520

(cons "git quotes this path, so it cannot be opened" #f))

1521

;; `#` ends a URL path at the fragment and `?` at the query, so an

1522

;; unescaped one makes the page unreachable AND makes the index

1523

;; link somewhere else. Excluded loudly rather than emitted broken.

1524

((or (contains? rel #\#) (contains? rel #\?))

−1512

"the path contains # or ?, which cannot appear unescaped in a URL")

+1525

(cons "the path contains # or ?, which cannot appear unescaped in a URL" #f))

1526

((not (file-exists? (path-join dir rel)))

−1514

"no such file on disk at tip")

−1515

((binary-string? (read-file-string (path-join dir rel)))

−1516

"binary")

−1517

(else #f)))

+1527

(cons "no such file on disk at tip" #f))

+1528

(else

+1529

(let ((text (read-file-string (path-join dir rel))))

+1530

(if (binary-string? text) (cons "binary" #f) (cons #f text))))))

+1531

+1532

(define (verdict-reason v) (car v))

+1533

(define (verdict-text v) (cdr v))

1534

1535

;; A NUL byte means binary; rendering it as text produces garbage and

1536

;; bloats the tree. Cheaper and more honest than trusting extensions.

+1537

;;

+1538

;; ONLY THE FIRST 8000 CHARACTERS COUNT, which is the rule this has always

+1539

;; applied and is why the search is over a prefix rather than the whole

+1540

;; string: a NUL at position 9000 does not make the file binary, and a

+1541

;; whole-string search would call one that used to render. The scan is

+1542

;; `string-find` -- one native pass -- instead of a Scheme loop calling

+1543

;; `string-ref` and `char->integer` per character, which was most of the

+1544

;; 25.5 s above.

1545

(define (binary-string? text)

−1522

(let ((n (string-length text)))

−1523

(let loop ((i 0))

−1524

(cond ((>= i n) #f)

−1525

((>= i 8000) #f)

−1526

((eq? (char->integer (string-ref text i)) 0) #t)

−1527

(else (loop (+ i 1)))))))

−1528

−1529

(define (line-node i line)

−1530

(let ((anchor (string-append "L" (number->string i))))

−1531

`(div (@ (class "cline") (id ,anchor))

−1532

(a (@ (class "lno") (href ,(string-append "#" anchor))

−1533

(aria-label ,(string-append "line " (number->string i))))

−1534

,(number->string i))

−1535

(code ,line))))

+1546

(let* ((n (string-length text))

+1547

(head (if (> n 8000) (substring text 0 8000) text)))

+1548

(if (string-find head (string (integer->char 0))) #t #f)))

1549

1550

;; A file ending in a newline splits to a trailing empty element, which

1551

;; would render a phantom final line and put every citation past it off

@@ -1542,12 +1555,68 @@

1555

(let ((r (reverse lines)))

1556

(if (and (pair? r) (string-empty? (car r))) (reverse (cdr r)) lines)))

1557

+1558

;; THE SOURCE LISTING IS SERIALISED DIRECTLY, AND THIS IS THE ONE PLACE IN

+1559

;; THE RENDERER THAT BYPASSES SXML. It is worth stating why, because

+1560

;; "hand-written markup is faster" is exactly the argument that produces an

+1561

;; unmaintainable renderer if it is applied anywhere else.

+1562

;;

+1563

;; A listing is the only surface whose node count scales with the SIZE OF

+1564

;; THE INPUT rather than with the design: one `div` per line of every

+1565

;; tracked file in the estate, so the sigil monorepo alone is roughly

+1566

;; 335,000 of them. Every other page in this renderer has a node count fixed

+1567

;; by its layout, and none of them is worth a special case.

+1568

;;

+1569

;; Measured on three real repositories: source listings were 68% of the

+1570

;; render. Within one, serialisation was ~88% of the page's cost, because

+1571

;; `sxml->xml` walks a three-element tree per line and its `xml-escape`

+1572

;; displays ONE CHARACTER AT A TIME into a string port. Measured on a 368 KB

+1573

;; source file: 1150 ms through SXML, 342 ms through this, and the two

+1574

;; outputs compared byte for byte.

+1575

;;

+1576

;; THE SHAPE IS FIXED AND TRIVIAL -- three elements, five attributes, no

+1577

;; branching -- which is what makes writing it out safe to do by hand. If

+1578

;; the listing ever grows a conditional, this stops being a serialisation

+1579

;; detail and goes back through SXML.

+1580

;;

+1581

;; The escape is FIVE BULK PASSES, ampersand FIRST so the entities it

+1582

;; introduces are not re-escaped by the passes after it -- the same order,

+1583

;; and therefore the same output, as escaping character by character. It

+1584

;; runs over the WHOLE text once rather than per line, which it may do

+1585

;; because escaping neither creates nor destroys a newline.

+1586

;;

+1587

;; `test/render-identical.sh` is what holds this honest: it compares every

+1588

;; byte of the whole output tree against the previous renderer.

+1589

(define (escape-listing text)

+1590

(string-replace

+1591

(string-replace

+1592

(string-replace

+1593

(string-replace (string-replace text "&" "&") "<" "<")

+1594

">" ">")

+1595

"\"" """)

+1596

"'" "'"))

+1597

1598

(define (listing-nodes text)

−1546

(let loop ((lines (drop-trailing-blank (string-split text "\n"))) (i 1) (out '()))

−1547

(if (null? lines)

−1548

(reverse out)

−1549

(loop (cdr lines) (+ i 1)

−1550

(cons (line-node i (if (string-empty? (car lines)) " " (car lines))) out)))))

+1599

(let ((parts

+1600

(let loop ((lines (drop-trailing-blank (string-split (escape-listing text) "\n")))

+1601

(i 1) (out '()))

+1602

(if (null? lines)

+1603

(reverse out)

+1604

(let ((num (number->string i))

+1605

;; A blank line becomes a single space so the row keeps

+1606

;; its height. This runs on the ESCAPED text, which is

+1607

;; the same test either way: escaping an empty string

+1608

;; yields an empty string.

+1609

(line (if (string-empty? (car lines)) " " (car lines))))

+1610

(loop (cdr lines) (+ i 1)

+1611

(cons (string-append

+1612

"<div class=\"cline\" id=\"L" num

+1613

"\"><a class=\"lno\" href=\"#L" num

+1614

"\" aria-label=\"line " num "\">" num

+1615

"</a><code>" line "</code></div>")

+1616

out)))))))

+1617

;; `*raw*` is sxml's own escape hatch for pre-serialised markup, so the

+1618

;; listing still travels as a node and the page above it is unchanged.

+1619

(list (list '*raw* (string-join parts)))))

1620

1621

;; DIRECTORIES LIVE UNDER tree/, FILES UNDER blob/, and the split is load-

1622

;; bearing rather than decorative.

@@ -4352,12 +4421,21 @@

4421

;; emitted pages all consume this one list. Deriving it

4422

;; three times is three things that can drift apart, and

4423

;; the first version of this did exactly that.

+4424

;;

+4425

;; ONE VERDICT PER FILE, AND IT CARRIES THE FILE'S TEXT.

+4426

;; This list used to be derived by two separate passes that

+4427

;; each read every file in full, and the page loop below then

+4428

;; read each one a third time -- 22% of the render spent

+4429

;; re-answering a question already answered. The text now

+4430

;; travels with the verdict and the page loop consumes it.

+4431

(verdicts (map (lambda (rel) (cons rel (file-verdict dir rel excludes))) shown))

4432

(excluded (filter (lambda (p) (pair? p))

−4356

(map (lambda (rel)

−4357

(let ((why (unrenderable-reason dir rel excludes)))

−4358

(if why (cons rel why) #f)))

−4359

shown)))

−4360

(leaves (filter (lambda (rel) (not (unrenderable-reason dir rel excludes))) shown))

+4433

(map (lambda (v)

+4434

(let ((why (verdict-reason (cdr v))))

+4435

(if why (cons (car v) why) #f)))

+4436

verdicts)))

+4437

(renderable (filter (lambda (v) (not (verdict-reason (cdr v)))) verdicts))

+4438

(leaves (map car renderable))

4439

(dirs (tree-dirs leaves))

4440

;; The TRUE shape of the repository, from git, uncapped and

4441

;; unfiltered. Every "N of M" on a directory page is counted

@@ -4408,8 +4486,11 @@

4486

(releases-index-page name tags))))

4487

;; one page per file at tip (§12.1)

4488

(for-each

−4411

(lambda (rel)

−4412

(let ((text (read-file-string (path-join dir rel))))

+4489

(lambda (v)

+4490

;; Read once, above, alongside the verdict that let this file

+4491

;; through. There is no second read here and no second chance

+4492

;; for the two to disagree about what the file contains.

+4493

(let ((rel (car v)) (text (verdict-text (cdr v))))

4494

(if (markdown-file? rel)

4495

;; RENDERED at the file's own URL, SOURCE one click away.

4496

;; Two pages, both reachable, and the walk enforces that

@@ -4421,7 +4502,7 @@

4502

(file-page name rel text)))

4503

(emit (string-append project-slug "/repos/" name "/blob/" rel "/index.html")

4504

(file-page name rel text)))))

−4424

leaves)

+4505

renderable)

4506

;; An index page per directory, INCLUDING the root. Every index

4507

;; links to its children and every page links back up through

4508

;; the crumb, so the whole tree is walkable in both directions

test/render-fourpoint.shadded

@@ -0,0 +1,113 @@

#!/usr/bin/env bash

# render-fourpoint.sh -- t-0361'S FOUR POINTS, BEFORE AND AFTER, INTERLEAVED.

#

# t-0361 measured four prefixes of the real sigil estate (1, 3, 33 and 71

# repos) and fit `ms/page ~= 17 + 2.3 x repos`. This re-measures the SAME four

# points with two binaries so the before/after and the shape of the curve come

# out of one run.

#

# EIGHT ARMS, ROUND-ROBIN, NOT BATCHED. This host is shared and its load drifts

+10

# on a scale of minutes; running four "before" arms and then four "after" arms

+11

# would measure the difference between two half-hours as much as between two

+12

# programs. Interleaving pre,post,pre,post... at each size makes both binaries

+13

# sample the same load profile, so the RATIO survives contention even where the

+14

# absolute numbers do not.

+15

#

+16

# THE REPOS ARE A FROZEN SNAPSHOT and every HEAD is asserted unchanged at the

+17

# end: the live checkouts are shared with other sessions on this host and were

+18

# being committed to during this work, so an arm rendered before someone's push

+19

# and an arm rendered after are not two measurements of one thing.

+20

#

+21

# A POSITIVE CONTROL THAT THE TWO BINARIES ARE DIFFERENT PROGRAMS AT ALL is

+22

# printed at the end, because identical timings are equally consistent with

+23

# having run one binary twice by mistake.

+24

#

+25

# usage: PRE=<docket> POST=<docket> SNAP=<dir> test/render-fourpoint.sh [k...]

+26

set -euo pipefail

+27

pre=${PRE:?set PRE to the pre-fix docket wrapper}

+28

post=${POST:?set POST to the post-fix docket wrapper}

+29

snap=${SNAP:?set SNAP to the snapshot directory holding repos-71.json}

+30

pre=$(cd "$(dirname "$pre")" && pwd)/$(basename "$pre")

+31

post=$(cd "$(dirname "$post")" && pwd)/$(basename "$post")

+32

srcrepo=$(cd "$(dirname "$0")/.." && pwd)

+33

rounds=${ROUNDS:-1}

+34

ks=("$@"); [[ ${#ks[@]} -gt 0 ]] || ks=(1 3 33 71)

+35

+36

root=${OUT_ROOT:-$(mktemp -d)}; mkdir -p "$root"

+37

export XDG_CACHE_HOME="$root/cache"; mkdir -p "$XDG_CACHE_HOME"

+38

+39

mkdir -p "$root/record/events" "$root/state"

+40

"$srcrepo/test/materialize-fixtures.sh" "$root/record/events" >/dev/null

+41

git -C "$root/record" init -q

+42

git -C "$root/record" config user.name t; git -C "$root/record" config user.email [email protected]

+43

mkdir -p "$root/record/meta"; printf '{"schema":1,"project":"sigil"}\n' >"$root/record/meta/project.json"

+44

git -C "$root/record" add events attachments meta

+45

GIT_AUTHOR_DATE=2026-01-01T00:00:00Z GIT_COMMITTER_DATE=2026-01-01T00:00:00Z \

+46

git -C "$root/record" commit -q -m rec

+47

printf 'p\n' >"$root/state/reporter-pepper"

+48

+49

python3 - "$snap/repos-71.json" "$root" "${ks[@]}" <<'PY'

+50

import json, sys

+51

repos = json.load(open(sys.argv[1]))

+52

out = sys.argv[2]

+53

for k in sys.argv[3:]:

+54

json.dump(repos[:int(k)], open(f"{out}/prefix-{k}.json", "w"))

+55

PY

+56

+57

heads() { python3 -c "

+58

import json,subprocess

+59

for r in json.load(open('$snap/repos-71.json')):

+60

h=subprocess.run(['git','-C',r['path'],'rev-parse','HEAD'],capture_output=True,text=True).stdout.strip()

+61

t=subprocess.run(['git','-C',r['path'],'tag','-l'],capture_output=True,text=True).stdout

+62

print(r['name'],h,len(t.split()))

+63

"; }

+64

heads >"$root/heads-before"

+65

+66

run() { # binary k outdir -> ms

+67

rm -rf "$3"

+68

local s e rc

+69

s=$(date +%s%N)

+70

DOCKET_ASSETS="$srcrepo/assets" "$1" render --record "$root/record" --state "$root/state" \

+71

--project "$srcrepo/test/fixtures/chronicle/project.sgl" --repos "$root/prefix-$2.json" \

+72

--output "$3" --report-url 'http://127.0.0.1:8765/' --site-url 'https://c.example' \

+73

>"$3.log" 2>&1 && rc=0 || rc=$?

+74

e=$(date +%s%N)

+75

((rc == 0)) || { echo "render FAILED ($1, k=$2, exit $rc)" >&2; tail -3 "$3.log" >&2; exit 1; }

+76

echo $(( (e - s) / 1000000 ))

+77

}

+78

median() { sort -n | awk '{a[NR]=$1} END{print (NR%2)? a[(NR+1)/2] : int((a[NR/2]+a[NR/2+1])/2)}'; }

+79

+80

for k in "${ks[@]}"; do : >"$root/t-pre-$k"; : >"$root/t-post-$k"; done

+81

echo "warm-up on k=1 only" >&2

+82

run "$pre" 1 "$root/w" >/dev/null; run "$post" 1 "$root/w" >/dev/null; rm -rf "$root/w" "$root/w.log"

+83

for r in $(seq 1 "$rounds"); do

+84

for k in "${ks[@]}"; do

+85

run "$pre" "$k" "$root/o-pre-$k" >>"$root/t-pre-$k"

+86

echo " round $r k=$k pre $(tail -1 "$root/t-pre-$k") ms" >&2

+87

run "$post" "$k" "$root/o-post-$k" >>"$root/t-post-$k"

+88

echo " round $r k=$k post $(tail -1 "$root/t-post-$k") ms" >&2

+89

done

+90

done

+91

+92

heads >"$root/heads-after"

+93

if ! diff -q "$root/heads-before" "$root/heads-after" >/dev/null; then

+94

echo "SNAPSHOT DRIFTED -- these numbers are not comparable:" >&2

+95

diff "$root/heads-before" "$root/heads-after" >&2

+96

exit 1

+97

fi

+98

echo "snapshot frozen: all 71 HEADs and tag counts identical before and after" >&2

+99

+100

echo

+101

+102

for k in "${ks[@]}"; do

+103

mp=$(median <"$root/t-pre-$k"); mq=$(median <"$root/t-post-$k")

+104

pages=$(find "$root/o-post-$k" -name index.html | wc -l)

+105

prepages=$(find "$root/o-pre-$k" -name index.html | wc -l)

+106

  (( pages == prepages )) || { echo "PAGE COUNTS DIFFER at k=$k: pre=$prepages post=$pages -- the two are not rendering the same site" >&2; exit 1; }

+107

awk -v k="$k" -v p="$pages" -v a="$mp" -v b="$mq" \

+108

'BEGIN{printf "%d|%d|%d|%d|%.2fx|%.1f|%.1f\n", k, p, a, b, a/b, a/p, b/p}'

+109

done

+110

echo

+111

echo "raw pre : $(for k in "${ks[@]}"; do printf '%s=%s ' "$k" "$(tr '\n' ',' <"$root/t-pre-$k")"; done)"

+112

echo "raw post: $(for k in "${ks[@]}"; do printf '%s=%s ' "$k" "$(tr '\n' ',' <"$root/t-post-$k")"; done)"

+113

echo "OUT_ROOT=$root"