Find out why a message ran twice

Use this when the work for one message happened more than once: a double charge, a duplicate email, two rows where there should be one.

SQS standard queues deliver at least once, so a second delivery is normal. What matters is whether two deliveries both did the work, and why the second one happened.

1Step 1: List the duplicates#

Open /duplicates on Trailbeam (http://127.0.0.1:4318/duplicates), or call the API. The list is newest activity first.

# Only messages whose work was done more than once
curl -s 'http://127.0.0.1:4318/api/duplicates?reason=succeeded_more_than_once&limit=20'

# Every message received more than once, on one queue
curl -s 'http://127.0.0.1:4318/api/duplicates?reason=received_more_than_once&queue=orders'

Any other value returns 400 with {"error": "reason must be received_more_than_once or succeeded_more_than_once"}. limit is 1 to 1000 (default 100).

Each item is a message summary plus reasons:

{"messages": [{
  "message_id": "5fea7756-0ea4-451a-a703-a558b933e274",
  "state": "deleted", "source_queue": "orders", "current_queue": "orders",
  "in_dead_letter_queue": false, "dead_letter_queue_move": null,
  "deliveries": 2, "successful_deliveries": 2,
  "consumers": [{"service_name": "orders-worker", "client_id": "consumer-a"},
                {"service_name": "orders-worker", "client_id": "consumer-b"}],
  "first_seen": "2025-11-14T00:26:40.030000Z", "first_seen_clock": "host",
  "last_seen": "2025-11-14T00:27:21.000000Z",
  "reasons": ["received_more_than_once", "succeeded_more_than_once"]
}]}

Counting rules worth knowing:

• A delete retried with the same receipt handle is one success, because Trailbeam counts distinct deliveries, not delete calls.
• An accepted delete with a stale handle (stale_handle) is a success. SQS accepted it, so the consumer that sent it finished its work.
• A delete that SQS rejected (failed) is not a success, even though the consumer probably did its work before trying to delete.
• An accepted delete that could not be joined to a receive (unmatched_handle) is not counted.

2Step 2: Open the timeline#

curl -s "http://127.0.0.1:4318/api/messages/$MESSAGE_ID" | jq '{state, deliveries, successful_deliveries,
  steps: [.events[] | {id, type, inferred, delivery, receive_count, status, error_code,
                       client: .consumer.client_id}]}'

For each delivery, compare the received event's consumer (service_name, client_id, pid, thread_name) and receive_count, then the deleted event that names the same delivery. A slow consumer on Amazon SQS looks like this:

Consumer A was still working when the visibility timeout ran out. SQS made the message visible, consumer B received it, and both finished. A's delete is accepted but carries the handle of delivery 1, so its status is stale_handle. B's delete uses the handle of the latest receive, so it is final. successful_deliveries is 2.

3Step 3: Match the pattern#

4Step 4: Check what Trailbeam can and cannot order#

Each observed event has an after list that says which earlier step it is proven to follow and why (clock, process_order, receive_count or sqs_timestamps). If delivery 1's delete and the expiry are in the same group and that group has concurrent: true, Trailbeam cannot tell whether the delete came before or after the timeout ran out. It does not guess. The screen shows that group as one box labelled "concurrent, order unknown".

curl -s "http://127.0.0.1:4318/api/messages/$MESSAGE_ID" \
  | jq '.events[] | select(.type=="deleted") | {delivery, status, group, after: [.after[] | .why]}'

Prevent it#

Make your processing idempotent. Key the side effect on the SQS MessageId, which stays the same across redeliveries.

def handle(message):
    # A unique constraint on message_id makes the second insert fail
    inserted = db.execute(
        "INSERT INTO processed (message_id) VALUES (?) ON CONFLICT DO NOTHING",
        (message["MessageId"],)).rowcount
    if inserted:
        charge(message)

async function handle(message) {
  const { rowCount } = await db.query(
    'INSERT INTO processed (message_id) VALUES ($1) ON CONFLICT DO NOTHING', [message.MessageId]);
  if (rowCount) await charge(message);
}

Extend the visibility timeout while working if a run can take longer than the queue's timeout. Each call shows on the timeline as Visibility changed with its visibility_timeout_s. See Debug a visibility-timeout redelivery for a full heartbeat.

Instrument every consumer with the same key. A consumer without Trailbeam shows up only as Deliveries not observed. A consumer with a different TRAILBEAM_RECEIPT_HASH_KEY produces deletes that cannot be joined, and the timeline shows a key_mismatch notice.

export TRAILBEAM_RECEIPT_HASH_KEY="$(openssl rand -hex 32)"   # generate once, share with every service

Related#

• Debug a visibility-timeout redelivery
• Receipt handles
• Recorded and inferred steps
• HTTP API