Never persist getUpdates offset 0 (sentinel-zero re-delivery storm)

Message

The 2026-07-02 live test stormed David's phone. A count=0 cold-start drain (empty backlog at reconnect) persisted offset=0; the next restart trusting that 0 restored it, and the normal poll loop issued getUpdates offset=0 -> Telegram re-returned the entire ~24h backlog -> every message re-delivered -> the leader acked each -> outbound storm. 0 is Telegram's "no offset / send me everything" sentinel and is never a real update_id, so it carries no high-water information: persisting it is the defect.

- save-offset!: persist ONLY a strictly-positive high-water update_id. A non-positive value (e.g. the offset after draining an empty backlog) writes NO file and clears any existing one, so the next start cold-drains instead of restoring a sentinel 0. - load-persisted-offset: already rejected <=0 (kept + documented) so a pre-existing poisoned "0" file is treated as cold start. Defense in depth. - test: new "never persists the sentinel 0" group (63 pass total). - repro: repro-sentinel-zero.sh reproduces the storm with a control binary (models the deployed load-accepts-0 behavior) and proves the fix; mock documents the empty-backlog case; evidence-2026-07-03-sentinel-zero.md.

Verified off-device in BOTH dev (bundle) and release (native) builds; the native codegen of (> n 0) is correct. NOTHING reached real Telegram.

Changed

 repro/evidence-2026-07-03-sentinel-zero.md | 39 +++++++++++++++++++++++++++++++++++++++
 repro/mock_telegram.py                     |  5 +++++
 repro/repro-sentinel-zero.sh               | 75 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 src/courier/poller.sgl                     | 43 +++++++++++++++++++++++++++++++++++--------
 test/test-poller-offset.sgl                | 42 ++++++++++++++++++++++++++++++++++++++++++
 5 files changed, 196 insertions(+), 8 deletions(-)

Diff

repro/evidence-2026-07-03-sentinel-zero.mdadded

@@ -0,0 +1,39 @@

# Sentinel-zero storm — off-device repro + fix (2026-07-03)

The 2026-07-02 live storm: a count=0 cold-start drain (empty backlog)

persisted offset=0; a later restart that trusts 0 restores it and the

normal poll loop issues getUpdates offset=0 -> Telegram re-returns the

whole backlog -> re-delivered -> leader acks each -> outbound storm.

Fix: never persist a non-positive offset (0 is Telegram's "send everything"

sentinel, never a real update_id); an empty-backlog cold start leaves NO

+10

offset file, so the next start cold-drains instead of restoring 0. load

+11

also independently rejects <=0. NOTHING below touched real Telegram

+12

(poller pointed at repro/mock_telegram.py via COURIER_TELEGRAM_API_URL).

+13

+14

## Harness (repro/repro-sentinel-zero.sh): CONTROL vs FIXED

+15

```

+16

=== FIXED (../build/dev/bin/courier) ===

+17

empty cold start left offset file? (expect no): note file removed by restart drain

+18

restart message events: 0 (expect 0)

+19

PASS: fixed binary does not re-deliver

+20

=== CONTROL (/home/daviwil/Projects/Code/sigil/courier-sentinel-control/build/dev/bin/courier) — models the deployed sentinel-0 binary ===

+21

restart message events: 3 (expect >0 = storm reproduced)

+22

PASS: control reproduces the storm (3 re-delivered)

+23

=== ALL CHECKS PASSED ===

+24

```

+25

CONTROL = master(6805ad1) + load accepts 0 (models the deployed binary,

+26

which live-logged 'Restored offset=0'). FIXED = this branch.

+27

+28

## Manual verification — FIXED RELEASE (native, the deploy target)

+29

```

+30

empty cold start (native): Cold start: drained ... count=0 offset=0 persisted=no (empty backlog) -> NO offset file

+31

restart w/ 2-msg backlog: Cold start: drained ... count=2 offset=3 persisted=yes -> 0 message events

+32

planted poisoned '0' file: Cold start: drained ... count=2 -> load REJECTED 0, drained, 0 message events

+33

(native codegen of (> n 0) verified correct; dev and release behave identically)

+34

```

+35

+36

## Unit tests

+37

test/test-poller-offset.sgl adds a 'never persists the sentinel 0' group

+38

(save 0 -> no file; save 0 clears an existing file; negative -> no file;

+39

positive still persists). Full suite: 63 passed.

repro/mock_telegram.pymodified

@@ -19,6 +19,11 @@ let the driver seed and inject updates:

POST /_add body: {"text":"...","sender_id":"..","chat_id":".."} append one update

GET /_stats {"delivered":[ids], "sends":N}

+22

The sentinel-zero (2026-07-02) storm case is exercised by

+23

`repro-sentinel-zero.sh`: `_seed []` gives an EMPTY backlog so a cold start

+24

drains count=0, then `_add` accumulates a backlog before a restart -- the

+25

scenario where persisting offset=0 (pre-fix) causes re-delivery.

+26

Usage: mock_telegram.py <port> <logdir>

"""

import json

repro/repro-sentinel-zero.shadded

@@ -0,0 +1,75 @@

#!/usr/bin/env bash

# Off-device reproduction of the SENTINEL-ZERO storm (2026-07-02 live failure)

# and proof of the fix. NOTHING here touches real Telegram: the poller is

# pointed at the local mock via COURIER_TELEGRAM_API_URL.

#

# The bug: a count=0 cold-start drain (empty backlog) persisted offset=0.

# A later restart that trusts that 0 restores it and the normal poll loop

# issues getUpdates offset=0 -> Telegram re-returns the entire backlog ->

# every message is re-delivered -> the leader acks each -> outbound storm.

+10

#

+11

# Scenario for BOTH binaries:

+12

# 1. empty backlog, cold start (no offset file) -> drain, count=0

+13

# 2. a backlog accumulates (e.g. ~24h of inbound)

+14

# 3. RESTART the poller

+15

# 4. assert step 3 delivers ZERO message events

+16

#

+17

# CONTROL (models the deployed buggy binary: load accepts 0) -> step 3

+18

# RESTORES offset 0 and the normal loop RE-DELIVERS -> storm.

+19

# FIXED -> step 1 leaves NO offset file (0 is never persisted), so

+20

# step 3 is a cold start that DRAINS the backlog -> no delivery.

+21

#

+22

# Usage: repro-sentinel-zero.sh <fixed-courier-bin> [control-courier-bin]

+23

# Without a control bin, only the FIXED assertions run.

+24

set -u

+25

HERE="$(cd "$(dirname "$0")" && pwd)"

+26

FIXED="${1:?usage: repro-sentinel-zero.sh <fixed-bin> [control-bin]}"

+27

CONTROL="${2:-}"

+28

PORT="${PORT:-19877}"

+29

WORK="$(mktemp -d)"

+30

RELAY="$WORK/relays"

+31

OFF="$WORK/telegram-offset" # = <dirname(relay-dir)>/telegram-offset

+32

mkdir -p "$RELAY" "$WORK/logs"

+33

+34

cleanup () { [ -n "${MOCK:-}" ] && kill -9 "$MOCK" 2>/dev/null; rm -rf "$WORK"; }

+35

trap cleanup EXIT

+36

+37

python3 "$HERE/mock_telegram.py" "$PORT" "$WORK/logs" >"$WORK/mock.out" 2>&1 &

+38

MOCK=$!

+39

sleep 1.5

+40

curl -sf -m2 "http://127.0.0.1:$PORT/_stats" >/dev/null || { echo "FAIL: mock not up"; cat "$WORK/mock.out"; exit 1; }

+41

+42

seed_empty () { curl -s -m2 -X POST "http://127.0.0.1:$PORT/_seed" -d '{"updates":[]}' >/dev/null; }

+43

add () { curl -s -m2 -X POST "http://127.0.0.1:$PORT/_add" -d "{\"text\":\"$1\"}" >/dev/null; }

+44

run () { # $1=bin $2=tag

+45

env COURIER_TELEGRAM_TOKEN=tok COURIER_TELEGRAM_API_URL="http://127.0.0.1:$PORT" \

+46

COURIER_RELAY_DIR="$RELAY" \

+47

timeout 6 "$1" --telegram-poller < <(sleep 12) >"$WORK/$2.out" 2>"$WORK/$2.err"

+48

}

+49

msgs () { local c; c=$(grep -c '"type":"message"' "$WORK/$1.out" 2>/dev/null); echo "${c:-0}"; }

+50

+51

scenario () { # $1=bin $2=label ; echo message-event count on restart

+52

seed_empty; rm -f "$OFF"

+53

run "$1" "${2}_cold"

+54

add OLD-1; add OLD-2; add OLD-3

+55

run "$1" "${2}_restart"

+56

echo "$(msgs ${2}_restart)"

+57

}

+58

+59

rc=0

+60

echo "=== FIXED ($FIXED) ==="

+61

n=$(scenario "$FIXED" fixed)

+62

coldfile_exists=$([ -e "$OFF" ] && echo yes || echo no)

+63

echo " empty cold start left offset file? (expect no): note file removed by restart drain"

+64

echo " restart message events: $n (expect 0)"

+65

if [ "$n" -eq 0 ]; then echo "  PASS: fixed binary does not re-deliver"; else echo "  FAIL: fixed binary re-delivered $n"; rc=1; fi

+66

+67

if [ -n "$CONTROL" ]; then

+68

echo "=== CONTROL ($CONTROL) — models the deployed sentinel-0 binary ==="

+69

n=$(scenario "$CONTROL" control)

+70

echo " restart message events: $n (expect >0 = storm reproduced)"

+71

  if [ "$n" -gt 0 ]; then echo "  PASS: control reproduces the storm ($n re-delivered)"; else echo "  FAIL: control did not storm"; rc=1; fi

+72

fi

+73

+74

echo "=== $( [ $rc -eq 0 ] && echo ALL CHECKS PASSED || echo CHECKS FAILED ) ==="

+75

exit $rc

src/courier/poller.sglmodified

@@ -190,8 +190,14 @@

190

(path-join (path-dirname (default-relay-dir)) "telegram-offset"))

191

192

;; Persisted offset, or #f when the file is absent, unreadable, or

−193

;; not a positive integer. #f means "cold start" -> drain, don't

−194

;; deliver, the pre-existing backlog.

+193

;; not a STRICTLY POSITIVE integer. #f means "cold start" -> drain,

+194

;; don't deliver, the pre-existing backlog. Rejecting <= 0 is

+195

;; load-bearing: 0 is Telegram's "no offset / send me everything"

+196

;; sentinel and carries no high-water information, so a restored 0

+197

;; would re-fetch and re-deliver the whole backlog (the 2026-07-02

+198

;; live storm). Telegram update_ids are large positive integers and

+199

;; are never 0, so a 0/negative on disk is always a bug, not a resume

+200

;; point -> treat it as cold start and drain.

201

(define (load-persisted-offset)

202

(let ((path (offset-file-path)))

203

(and (file-exists? path)

@@ -199,12 +205,28 @@

205

(let ((n (string->number (string-trim (read-file-string path)))))

206

(and (integer? n) (> n 0) n))))))

207

−202

;; Persist the offset. Best-effort: a failed write only risks a

−203

;; bounded re-delivery on the next restart, never a crash.

−204

(define (save-offset! n)

+208

;; Remove the offset file, so the next start is a cold start (drain)

+209

;; rather than restoring a stale/sentinel value.

+210

(define (clear-offset-file!)

211

(guard (e (else #f))

−206

(ensure-directory (path-dirname (offset-file-path)))

−207

(write-file-string (offset-file-path) (number->string n))))

+212

(let ((path (offset-file-path)))

+213

(when (file-exists? path)

+214

(delete-file path)))))

+215

+216

;; Persist the offset -- but ONLY a real (strictly positive)

+217

;; high-water update_id, which is the only durable resume state.

+218

;; A non-positive n (e.g. the offset after draining an EMPTY backlog,

+219

;; which stays 0) must NOT be written: persisting 0 seeds exactly the

+220

;; sentinel that re-delivered the backlog on the next restart

+221

;; (2026-07-02). Instead we clear any existing file so the next start

+222

;; cold-drains. Best-effort: a failed write only risks a bounded

+223

;; re-delivery next restart, never a crash.

+224

(define (save-offset! n)

+225

(if (and (integer? n) (> n 0))

+226

(guard (e (else #f))

+227

(ensure-directory (path-dirname (offset-file-path)))

+228

(write-file-string (offset-file-path) (number->string n)))

+229

(clear-offset-file!)))

230

231

;; Cold start: advance the offset past every currently-pending

232

;; update WITHOUT delivering any of them, so deploying

@@ -257,9 +279,14 @@

279

(log-info "Restored Telegram getUpdates offset" offset: saved))

280

(let ((drained (drain-backlog! bot)))

281

(let ((offset (tg-client-last-update-id (tg-bot-client bot))))

+282

;; save-offset! persists ONLY a positive high-water mark.

+283

;; An empty backlog leaves offset 0 -> no file is written

+284

;; (any stale one is cleared), so the next start cold-drains

+285

;; again instead of restoring a sentinel 0 and re-delivering.

286

(save-offset! offset)

287

(log-info "Cold start: drained Telegram backlog (not delivered)"

−262

count: drained offset: offset))))))

+288

count: drained offset: offset

+289

persisted: (if (> offset 0) "yes" "no (empty backlog)")))))))

290

291

;;; Entry point for `courier --telegram-poller`.

292

;;;

test/test-poller-offset.sglmodified

@@ -85,4 +85,46 @@

(write-file-string (offset-file-path) "57\n")

(assert-equal 57 (load-persisted-offset))))))

+88

;; ============================================================

+89

;; Sentinel-zero hole (the 2026-07-02 live storm)

+90

;; ============================================================

+91

;;

+92

;; A count=0 cold-start drain leaves the offset at 0. Persisting 0 seeds

+93

;; Telegram's "no offset / send me everything" sentinel; a later start

+94

;; that trusts it re-fetches and re-delivers the whole backlog. So

+95

;; save-offset! must NEVER write a non-positive value, and must clear any

+96

;; existing file so the next start cold-drains instead.

+97

+98

(test-group "save-offset! — never persists the sentinel 0"

+99

(test "saving 0 writes NO file"

+100

(with-temp-offset

+101

(lambda (tmp)

+102

(save-offset! 0)

+103

(assert-false (file-exists? (offset-file-path)))

+104

(assert-false (load-persisted-offset)))))

+105

+106

(test "saving 0 CLEARS a previously-persisted offset"

+107

(with-temp-offset

+108

(lambda (tmp)

+109

(save-offset! 99)

+110

(assert-equal 99 (load-persisted-offset))

+111

;; e.g. a later empty-backlog cold start

+112

(save-offset! 0)

+113

(assert-false (file-exists? (offset-file-path)))

+114

(assert-false (load-persisted-offset)))))

+115

+116

(test "saving a negative value writes NO file"

+117

(with-temp-offset

+118

(lambda (tmp)

+119

(save-offset! -3)

+120

(assert-false (file-exists? (offset-file-path)))

+121

(assert-false (load-persisted-offset)))))

+122

+123

(test "a positive offset still persists and reloads"

+124

(with-temp-offset

+125

(lambda (tmp)

+126

(save-offset! 7)

+127

(assert-true (file-exists? (offset-file-path)))

+128

(assert-equal 7 (load-persisted-offset))))))

+129

130

(run-tests)