Send-path duplication: off-device evidence (2026-07-06)

All runs go through repro/mock_telegram.py; nothing reached real Telegram (courier's SEND path is forced to the mock via COURIER_TELEGRAM_API_URL).

Setup

• mock_telegram.py — records every sendMessage; can inject send failure modes (ok, reset, delay:N, status:N, notok) via POST /_mode.
• send_driver.py — a minimal MCP client (the "leader") that fires ONE logical send-message and models the harness reaction to a send that does not return a clean, timely success: a per-tool-call timeout, and on timeout/death a kill + respawn + re-issue of the same send (the /mcp-reconnect + agent-retry loop), up to --retries times. All restarts share one COURIER_RELAY_DIR, exactly as the real leader's restarts share ~/.courier.

Root cause

courier sends exactly once per tool call (traced: telegram.sgl → tg-send-message → tg-api-call → http-post/json; no retry at any layer, and the MCP server guards tool handlers so an exception cannot crash the process). The duplication is cross-process: the sendMessage ack read can block up to *send-request-timeout* (25s). If the leader's MCP client gives up sooner — a short tool timeout, or a human/harness /mcp because the call "looks hung" — it SIGKILLs courier and re-issues the send. Each attempt's POST reaches Telegram before the kill → the same message is delivered N times. The v0.3.7/0.3.8 (chat-id,text) dedup could not stop it: recent-sends was in-memory, so a restarted process started with an empty cache. The "inv=1 then --- courier restart ---" in the live log is this SIGKILL-on-reconnect, not an internal segfault.

Before (pre-fix master 4a5ae2b, in-memory dedup)

STORM (delay:8, tool-timeout 3, retries 3): 1 logical send -> 4 deliveries

Each of the 4 attempts (gen0..gen3) re-delivered because the fresh process's in-memory dedup was empty.

After (fix: persistent record-before-send dedup, src/courier/dedup.sgl)

baseline ok          1 send -> 1 delivery   "Message sent."
reset (ack lost)     1 send -> 1 delivery   "Message sent (ack unconfirmed)."
STORM x3             1 send -> 1 delivery   (gen0 killed mid-send; gen1 suppressed)
STORM x5             1 send -> 1 delivery
status:500 / notok   1 send, clean error "…(message not delivered)."  NO crash, key released
never-sent (dead port) 0 deliveries, clean error, key released (retry allowed)

In the storm, gen0 records the dedup key before calling tg-send-message, delivers, then is SIGKILLed; gen1 (fresh process, empty memory) reads the same on-disk key and returns "Message sent." in ~0.1s without re-delivering.

Reproduce

sigil build
# optional pre-fix control:
git worktree add /tmp/courier-control master && (cd /tmp/courier-control && sigil deps install && sigil build)
COURIER_CONTROL_BIN=/tmp/courier-control/build/dev/bin/courier repro/repro-send-duplication.sh
# => ALL CHECKS PASSED  (FIXED storms -> 1; CONTROL storm -> 4)

Note on exactly-once vs at-most-once

Telegram's sendMessage has no idempotency key, so exactly-once is only achievable by courier suppressing duplicate deliveries. The fix records a send before attempting delivery and treats a send killed mid-flight as delivered (at-most-once). A definite non-delivery (never-sent connect failure, or Telegram rejection) releases the key so a genuine retry can go through. This biases toward "never spam" over "never lose a notification" — the correct bias for this bot, and the whole point of the saga.