repro/evidence-2026-07-06-send-duplication.md

1# Send-path duplication: off-device evidence (2026-07-06)

3All runs go through `repro/mock_telegram.py`; **nothing reached real Telegram**

4(courier's SEND path is forced to the mock via `COURIER_TELEGRAM_API_URL`).

6## Setup

8- `mock_telegram.py` — records every `sendMessage`; can inject send failure

9 modes (`ok`, `reset`, `delay:N`, `status:N`, `notok`) via `POST /_mode`.

10- `send_driver.py` — a minimal MCP client (the "leader") that fires ONE

11 logical `send-message` and models the harness reaction to a send that does

12 not return a clean, timely success: a per-tool-call timeout, and on

13 timeout/death a **kill + respawn + re-issue** of the same send (the

14 `/mcp`-reconnect + agent-retry loop), up to `--retries` times. All restarts

15 share one `COURIER_RELAY_DIR`, exactly as the real leader's restarts share

16 `~/.courier`.

18## Root cause

20courier sends **exactly once per tool call** (traced: `telegram.sgl` →

21`tg-send-message` → `tg-api-call` → `http-post/json`; no retry at any layer,

22and the MCP server guards tool handlers so an exception cannot crash the

23process). The duplication is **cross-process**: the `sendMessage` ack read can

24block up to `*send-request-timeout*` (25s). If the leader's MCP client gives up

25sooner — a short tool timeout, or a human/harness `/mcp` because the call

26"looks hung" — it SIGKILLs courier and re-issues the send. Each attempt's POST

27reaches Telegram before the kill → the same message is delivered N times. The

28v0.3.7/0.3.8 `(chat-id,text)` dedup could not stop it: `recent-sends` was

29**in-memory**, so a restarted process started with an empty cache. The

30"`inv=1` then `--- courier restart ---`" in the live log is this

31SIGKILL-on-reconnect, not an internal segfault.

33## Before (pre-fix master `4a5ae2b`, in-memory dedup)

35```

36STORM (delay:8, tool-timeout 3, retries 3): 1 logical send -> 4 deliveries

37```

39Each of the 4 attempts (gen0..gen3) re-delivered because the fresh process's

40in-memory dedup was empty.

42## After (fix: persistent record-before-send dedup, `src/courier/dedup.sgl`)

44```

45baseline ok 1 send -> 1 delivery "Message sent."

46reset (ack lost) 1 send -> 1 delivery "Message sent (ack unconfirmed)."

47STORM x3 1 send -> 1 delivery (gen0 killed mid-send; gen1 suppressed)

48STORM x5 1 send -> 1 delivery

49status:500 / notok 1 send, clean error "…(message not delivered)." NO crash, key released

50never-sent (dead port) 0 deliveries, clean error, key released (retry allowed)

51```

53In the storm, gen0 records the dedup key **before** calling `tg-send-message`,

54delivers, then is SIGKILLed; gen1 (fresh process, empty memory) reads the same

55on-disk key and returns "Message sent." in ~0.1s without re-delivering.

57## Reproduce

59```

60sigil build

61# optional pre-fix control:

62git worktree add /tmp/courier-control master && (cd /tmp/courier-control && sigil deps install && sigil build)

63COURIER_CONTROL_BIN=/tmp/courier-control/build/dev/bin/courier repro/repro-send-duplication.sh

64# => ALL CHECKS PASSED (FIXED storms -> 1; CONTROL storm -> 4)

65```

67## Note on exactly-once vs at-most-once

69Telegram's `sendMessage` has no idempotency key, so exactly-once is only

70achievable by courier suppressing duplicate deliveries. The fix records a send

71**before** attempting delivery and treats a send killed mid-flight as

72delivered (at-most-once). A definite non-delivery (never-sent connect failure,

73or Telegram rejection) **releases** the key so a genuine retry can go through.

74This biases toward "never spam" over "never lose a notification" — the correct

75bias for this bot, and the whole point of the saga.