sigil-markdown 0.9.5: an underscore inside a word is not emphasis

Message

CommonMark treats '' and '*' differently on purpose. An underscore between two word characters is ordinary text, because snakecase is ordinary prose in technical writing. This parser applied the '*' rule to both, so every snake_case identifier written as prose was eaten:

  sokol_gfx and sokol_gp  ->  sokol<em>gfx and sokol</em>gp

Nothing in the output showed that anything had been removed, which is why it survived: the page reads as if the author wrote it that way.

Measured against the corpus a real consumer renders rather than against invented cases. Over the 1,389 commit messages and 86 READMEs the Azoth estate publishes: 97 of the commit messages and 1 of the READMEs were corrupted. That README is live at /sigil/repos/sigil-graphics/ today. After the fix, 0 READMEs and 2 commit messages still emit emphasis from an underscore, and both are accounted for rather than assumed: attribute((unused)) is CommonMark-conformant strong (the closing run is followed by punctuation), and one base64 string in a quoted example lands in punctuation-flanking territory this simplified rule does not model. Neither is identifier corruption.

The rule: a '_' run may open only when not preceded by a word character and close only when not followed by one. '' KEEPS THE OLD BEHAVIOUR -- foobar*baz is emphasis in CommonMark and remains so here. A negative control test fails if the rule is ever applied to both characters.

emphasis at a word boundary, bold, and foobar_ (which CommonMark emphasises across the inner underscore) are all unchanged and tested.

The new tests take their strings from the real corpus, not from my understanding of the bug: sokolgfx, stbimage, sigilwasmgles3, KILNARTIFACTS, groupid, run_groups. They assert TEXT IDENTITY with the source rather than the absence of an em node, because absence passes for free if the parser drops its input. Sabotage-tested: flipping the new flag back to #f turns 6 of the 9 red, and the 3 that stay green are exactly the boundary and negative-control cases that must not depend on it.

Five of the six bench goldens are byte-identical. inline-edge changed on one construct, underscore, which that corpus deliberately contains; its asterisk neighbours in the same line are untouched. Golden re-blessed.

The boundary test is ASCII-only, like this file's own char-numeric?, so a non-ASCII letter beside an underscore still reads as a boundary. Named in the source rather than left to be discovered.

Changed

 CHANGELOG.md                  |  6 ++++++
 bench/golden/inline-edge.sxml |  2 +-
 package.sgl                   |  2 +-
 src/sigil/markdown.sgl        | 62 +++++++++++++++++++++++++++++++++++++++++++++++++++-----------
 test/test-markdown.sgl        | 82 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 5 files changed, 141 insertions(+), 13 deletions(-)

Diff

CHANGELOG.mdmodified

@@ -7,6 +7,12 @@ All notable changes to this project will be documented in this file.

The format is based on [Keep a Changelog](https://keepachangelog.com/),

and this project adheres to [Semantic Versioning](https://semver.org/).

+10

## [0.9.5] - 2026-08-02

+11

+12

### Fixed

+13

+14

- An underscore between two word characters is no longer an emphasis delimiter, per CommonMark. `sokol_gfx and sokol_gp` previously rendered as `sokol<em>gfx and sokol</em>gp`: the underscores were consumed and the words silently joined, with nothing in the output to show that anything had been eaten. Every snake_case identifier written as prose was affected. Measured over the corpus one consumer actually renders, 97 of 1,389 commit messages and 1 of 86 READMEs were corrupted this way. `_emphasis_` at a word boundary, `__bold__`, and `_foo_bar_` (which CommonMark emphasises across the inner underscore) are unchanged. **`*` deliberately keeps the old behaviour** — `foo*bar*baz` is emphasis in CommonMark and remains so here; a negative-control test fails if the rule is ever applied to both characters. The boundary test is ASCII-only, like this parser's own `char-numeric?`, so a non-ASCII letter beside an underscore still reads as a boundary.

+15

## [0.9.4] - 2026-07-31

### Fixed

bench/golden/inline-edge.sxmlmodified

@@ -1 +1 @@

−1

(document (h1 "Inline edge cases") (p "Stray asterisk " (em " in the middle and a lone ! bang and a lone " "_" " underscore. A trailing asterisk word") " and " (em "leading and mid") "dle and un" (em "der") "score words. Bang before bracket " "!" "[" "not an image because no paren] and " (img (@ (src "src.png") (alt "alt"))) " real. Escaped " "*" " star, escaped " "_" " under, escaped " "`" " tick, escaped " "\\" " backslash.") (p "Text with " (code "code with * and _ and [ inside") " should stay literal. Bold " (strong "with " (code "code") " and " (em "should this nest") " inside") " end. Italic " (em "with " (a (@ (href "x")) "a link") " inside") " and " (em "with " (code "code")) " too.") (p "Empty emphasis " (strong) " and " (strong) " and lone " "*" "*" " at start. Link " (a (@ (href "http://example.com/path?a=1&b=2")) "text") " with query. Image " (img (@ (src "path/to/img.png") (alt "alt text"))) " inline.") (p "A run of specials: " (em "_" "`" "[" "!" " " "mixed together. Multiple ") (em "bold") (em " words ") (em "again") (em " and ") "em" (em " then ") "em2" "*" " close together.") (p "Unclosed " "*" (em "bold never closes here. Unclosed ") "italic and " "`" "code both dangle.") (p "Consecutive" "[" "[" "wikilink]]-style double brackets and normal " (a (@ (href "u")) "link") ". Numbers 1234567890 and symbols !@#$%^&()+=~ with no markup."))

No newline at end of file

(document (h1 "Inline edge cases") (p "Stray asterisk " (em " in the middle and a lone ! bang and a lone " "_" " underscore. A trailing asterisk word") " and " (em "leading and mid") "dle and un" "_" "der" "_" "score words. Bang before bracket " "!" "[" "not an image because no paren] and " (img (@ (src "src.png") (alt "alt"))) " real. Escaped " "*" " star, escaped " "_" " under, escaped " "`" " tick, escaped " "\\" " backslash.") (p "Text with " (code "code with * and _ and [ inside") " should stay literal. Bold " (strong "with " (code "code") " and " (em "should this nest") " inside") " end. Italic " (em "with " (a (@ (href "x")) "a link") " inside") " and " (em "with " (code "code")) " too.") (p "Empty emphasis " (strong) " and " (strong) " and lone " "*" "*" " at start. Link " (a (@ (href "http://example.com/path?a=1&b=2")) "text") " with query. Image " (img (@ (src "path/to/img.png") (alt "alt text"))) " inline.") (p "A run of specials: " (em "_" "`" "[" "!" " " "mixed together. Multiple ") (em "bold") (em " words ") (em "again") (em " and ") "em" (em " then ") "em2" "*" " close together.") (p "Unclosed " "*" (em "bold never closes here. Unclosed ") "italic and " "`" "code both dangle.") (p "Consecutive" "[" "[" "wikilink]]-style double brackets and normal " (a (@ (href "u")) "link") ". Numbers 1234567890 and symbols !@#$%^&()+=~ with no markup."))

No newline at end of file

package.sglmodified

@@ -5,7 +5,7 @@

(package

name: "sigil-markdown"

−8

version: "0.9.4"

version: "0.9.5"

sigil: "^0.17"

description: "Markdown and frontmatter parser"

url: "https://codeberg.org/sigil/sigil-markdown"

src/sigil/markdown.sglmodified

@@ -674,17 +674,54 @@

674

#f))

675

(else (loop (+ k 1))))))

676

+677

;; ---- word-boundary flanking, for '_' only ---------------------------

+678

;;

+679

;; CommonMark treats '_' and '*' differently on purpose: an underscore

+680

;; between two word characters is NOT a delimiter, because snake_case is

+681

;; ordinary prose in technical writing. '*' has no such restriction --

+682

;; `foo*bar*baz` IS emphasis in CommonMark and stays so here. This is the

+683

;; one place the two characters may legitimately diverge.

+684

;;

+685

;; Without this rule every snake_case identifier in prose was silently

+686

;; mangled, with no error and nothing in the output to show something had

+687

;; been eaten. `sokol_gfx and sokol_gp` rendered as

+688

;; `sokolgfx and sokolgp` -- live and public on azoth.works.

+689

;; Measured over the 1,389 commit messages the Azoth estate actually

+690

;; renders, 97 were corrupted this way; over its 86 READMEs, one was.

+691

;;

+692

;; ASCII-only, deliberately and like this file's own `char-numeric?`.

+693

;; A non-ASCII letter adjacent to an underscore is therefore still read

+694

;; as a boundary. Named rather than silently assumed: CommonMark uses

+695

;; Unicode classes here and this does not.

+696

(define (ascii-letter? c)

+697

(or (and (char>=? c #\a) (char<=? c #\z))

+698

(and (char>=? c #\A) (char<=? c #\Z))))

+699

+700

(define (word-char? c) (or (ascii-letter? c) (char-numeric? c)))

+701

+702

;; A delimiter run may OPEN only when nothing word-like precedes it...

+703

(define (can-open? text i)

+704

(or (= i 0) (not (word-char? (string-ref text (- i 1))))))

+705

+706

;; ...and CLOSE only when nothing word-like follows it. `after` is the

+707

;; index just past the closing run, so the caller accounts for the run's

+708

;; own width rather than this predicate guessing at it.

+709

(define (can-close? text after len)

+710

(or (>= after len) (not (word-char? (string-ref text after)))))

+711

712

;; Double-delimiter emphasis: **strong** / __strong__. Requires the

713

;; doubled opener (caller has already seen the first delimiter char),

714

;; one or more inner chars, and a doubled closer. Inner is re-parsed.

−680

(define (scan-delim-double text i len ch tag)

+715

;; `bounded?` applies the word-boundary rule above to the CLOSER.

+716

(define (scan-delim-double text i len ch tag bounded?)

717

(if (and (< (+ i 1) len) (eq? (string-ref text (+ i 1)) ch))

718

(let loop ((k (+ i 2)))

719

(cond

720

((>= k len) #f)

721

((and (eq? (string-ref text k) ch)

722

(< (+ k 1) len)

−687

(eq? (string-ref text (+ k 1)) ch))

+723

(eq? (string-ref text (+ k 1)) ch)

+724

(or (not bounded?) (can-close? text (+ k 2) len)))

725

(if (> k (+ i 2))

726

(cons (cons tag (parse-inline (substring text (+ i 2) k)))

727

(+ k 2))

@@ -695,13 +732,14 @@

732

;; Single-delimiter emphasis: *em* / _em_. Only when the opener is NOT

733

;; doubled (grammar lookahead (& (seq D (! D)))); one or more inner chars

734

;; then a closing delimiter. Inner is re-parsed.

−698

(define (scan-delim-single text i len ch tag)

+735

(define (scan-delim-single text i len ch tag bounded?)

736

(if (or (>= (+ i 1) len)

737

(not (eq? (string-ref text (+ i 1)) ch)))

738

(let loop ((k (+ i 1)))

739

(cond

740

((>= k len) #f)

−704

((eq? (string-ref text k) ch)

+741

((and (eq? (string-ref text k) ch)

+742

(or (not bounded?) (can-close? text (+ k 1) len)))

743

(if (> k (+ i 1))

744

(cons (cons tag (parse-inline (substring text (+ i 1) k)))

745

(+ k 1))

@@ -710,10 +748,12 @@

748

#f))

749

750

;; Emphasis dispatch for a delimiter char: try bold (double) then italic

−713

;; (single), matching the grammar's ordered choice.

−714

(define (scan-emphasis text i len ch)

−715

(or (scan-delim-double text i len ch 'strong)

−716

(scan-delim-single text i len ch 'em)))

+751

;; (single), matching the grammar's ordered choice. `bounded?` is #t for

+752

;; '_' and #f for '*'.

+753

(define (scan-emphasis text i len ch bounded?)

+754

(and (or (not bounded?) (can-open? text i))

+755

(or (scan-delim-double text i len ch 'strong bounded?)

+756

(scan-delim-single text i len ch 'em bounded?))))

757

758

;; Bracketed construct helper: given the position of the first inner char

759

;; (just past '[' or '!['), scan a label up to the first ']', require an

@@ -792,13 +832,13 @@

832

(loop (+ i 1) (cons "`" acc)))))

833

;; **bold** / *em*

834

((eq? c #\*)

−795

(let ((r (scan-emphasis text i len #\*)))

+835

(let ((r (scan-emphasis text i len #\* #f)))

836

(if r

837

(loop (cdr r) (cons (car r) acc))

838

(loop (+ i 1) (cons "*" acc)))))

−799

;; __bold__ / _em_

+839

;; __bold__ / _em_ -- word-bounded, unlike '*'

840

((eq? c #\_)

−801

(let ((r (scan-emphasis text i len #\_)))

+841

(let ((r (scan-emphasis text i len #\_ #t)))

842

(if r

843

(loop (cdr r) (cons (car r) acc))

844

(loop (+ i 1) (cons "_" acc)))))

test/test-markdown.sglmodified

@@ -135,6 +135,88 @@

135

(and (pair? x) (eq? (car x) 'img)))

136

result))))))

137

+138

;; ============================================================

+139

;; Intraword underscores are not emphasis delimiters

+140

;; ============================================================

+141

;;

+142

;; An underscore between two word characters is ordinary text, per

+143

;; CommonMark. Before this rule existed, every snake_case identifier written

+144

;; as prose was silently eaten: `sokol_gfx and sokol_gp` became

+145

;; `sokolgfx and sokolgp`, live on a public site.

+146

;;

+147

;; THE STRINGS BELOW ARE COPIED FROM THE REAL CORPUS -- Azoth estate READMEs

+148

;; and commit messages -- not invented. A case written from the same

+149

;; understanding that produced the bug tends to satisfy the buggy rule too;

+150

;; these cannot drift from what the estate actually renders.

+151

;;

+152

;; The assertion is TEXT-IDENTITY, not "no em node present". Absence of a

+153

;; symptom passes for free if the parser drops the input entirely; equality

+154

;; with the source string does not.

+155

+156

(define (inline-text result)

+157

(let flatten ((n result) (acc ""))

+158

(cond ((string? n) (string-append acc n))

+159

((and (pair? n) (symbol? (car n)))

+160

;; skip the tag, and an (@ ...) attribute list if present

+161

(let ((rest (if (and (pair? (cdr n)) (pair? (cadr n))

+162

(eq? (car (cadr n)) '@))

+163

(cddr n)

+164

(cdr n))))

+165

(flatten rest acc)))

+166

((pair? n) (flatten (cdr n) (flatten (car n) acc)))

+167

(else acc))))

+168

+169

(define (assert-literal source)

+170

(assert-equal source (inline-text (parse-inline source))))

+171

+172

(test-group "intraword underscores"

+173

;; --- from README.md of sigil-graphics, rendered wrong on azoth.works ---

+174

(test "sokol_gfx and sokol_gp survive"

+175

(assert-literal "Provides 2D rendering capabilities via sokol_gfx and sokol_gp"))

+176

+177

(test "stb_image and stb_truetype survive"

+178

(assert-literal "image loading (stb_image), and font rendering (stb_truetype)."))

+179

+180

;; --- from real commit messages in the estate ---

+181

(test "sigil_wasm_gles3 survives"

+182

(assert-literal "sigil_wasm_gles3 and sigil_wasm_native_start"))

+183

+184

(test "KILN_ARTIFACTS survives"

+185

(assert-literal "KILN_ARTIFACTS and KILN_DEPS"))

+186

+187

(test "group_id, node_id, run_groups survive"

+188

(assert-literal "group_id, node_id, plan_form and run_groups"))

+189

+190

;; --- the boundary cases the rule must NOT break ---

+191

(test "__init__ is still strong"

+192

(let ((result (parse-inline "__init__ method")))

+193

(assert-true (pair? (filter (lambda (x) (and (pair? x) (eq? (car x) 'strong)))

+194

result)))

+195

(assert-equal "init method" (inline-text result))))

+196

+197

(test "_emphasis_ at a word boundary still works"

+198

(let ((result (parse-inline "an _emphasis_ here")))

+199

(assert-true (pair? (filter (lambda (x) (and (pair? x) (eq? (car x) 'em)))

+200

result)))

+201

(assert-equal "an emphasis here" (inline-text result))))

+202

+203

;; CommonMark: _foo_bar_ is foo_bar. The inner underscore is

+204

;; intraword and the outer pair is not, so the run closes at the LAST one.

+205

(test "_foo_bar_ emphasises across the intraword underscore"

+206

(let ((result (parse-inline "_foo_bar_")))

+207

(assert-true (pair? (filter (lambda (x) (and (pair? x) (eq? (car x) 'em)))

+208

result)))

+209

(assert-equal "foo_bar" (inline-text result))))

+210

+211

;; The '*' rule is deliberately DIFFERENT and must not change with it.

+212

;; This is the negative control: had the fix been applied to both

+213

;; characters, this test goes red.

+214

(test "intraword ASTERISK is still emphasis"

+215

(let ((result (parse-inline "foo*bar*baz")))

+216

(assert-true (pair? (filter (lambda (x) (and (pair? x) (eq? (car x) 'em)))

+217

result)))

+218

(assert-equal "foobarbaz" (inline-text result)))))

+219

220

;; ============================================================

221

;; Front matter

222

;; ============================================================