Trace a message to the dead-letter queue

Use this when a message ended up in a dead-letter queue (DLQ) and you need to know why, or you expected it there and cannot find it.

1Step 1: Know how Trailbeam sees the move#

SQS moves a message to the DLQ without any API call that Trailbeam could record. It reports the move afterwards: the next receive of that message, on the DLQ, carries the same MessageId and the system attribute DeadLetterQueueSourceArn, naming the queue it came from.

The hooks copy that attribute to trailbeam.dead_letter_queue_source_arn on the trailbeam receive span. Trailbeam shows a Moved to dead-letter queue step when:

• a receive has trailbeam.dead_letter_queue_source_arn, and
• the last : field of that ARN (the queue name) differs from the queue of that receive.

The step is inferred: false with source: "sqs_dlq_source", because SQS reported it. SQS gives it no time, so it has time: null and a window. It is placed after the last receive on the source queue (or after the send, if no receive there was observed) and before the first DLQ receive that reports it. There is one move per source queue and DLQ pair, however many DLQ receives carry the attribute.

2Step 2: Set up the DLQ reader and Trailbeam#

Instrument whatever reads the DLQ. The client hooks add DeadLetterQueueSourceArn to MessageSystemAttributeNames on every ReceiveMessage, unless you already asked for it or for All. Nothing you asked for is removed.

import boto3, trailbeam.boto
trailbeam.boto.install()                     # same TRAILBEAM_RECEIPT_HASH_KEY as the main consumers
sqs = boto3.client("sqs")
resp = sqs.receive_message(QueueUrl=dlq_url, WaitTimeSeconds=5)

const { SQSClient, ReceiveMessageCommand } = require('@aws-sdk/client-sqs');
const trailbeam = require('@trailbeam/sdk');
const sqs = trailbeam.instrument(new SQSClient({}));
await sqs.send(new ReceiveMessageCommand({ QueueUrl: dlqUrl, WaitTimeSeconds: 5 }));

Tell Trailbeam each queue's maxReceiveCount. Spans do not carry the redrive policy. Without it, Trailbeam never says a message is due on its DLQ.

trailbeam-store --max-receive-count orders=3,payments=5     # or TRAILBEAM_MAX_RECEIVE_COUNT=orders=3,payments=5

The format is QUEUE=N, comma-separated, with N from 1 to 1000. The key is the queue name (messaging.destination.name), not its URL or ARN. A bad value stops Trailbeam at startup with expected QUEUE=N with N from 1 to 1000. Check what Trailbeam loaded:

curl -s http://127.0.0.1:4318/api/health | jq .config.max_receive_counts

3Step 3: Find the message#

Messages waiting in a DLQ are on the stuck list, because their latest receive (on the DLQ) has no final delete.

# Everything sitting in orders-dlq
curl -s 'http://127.0.0.1:4318/api/stuck?queue=orders-dlq' \
  | jq '.messages[] | select(.in_dead_letter_queue) | {message_id, source_queue, deliveries, age_s}'

# Messages that hit maxReceiveCount but have not been seen on the DLQ
curl -s 'http://127.0.0.1:4318/api/stuck?queue=orders' \
  | jq '.messages[] | select(.dead_letter_queue_move == "not_seen_on_dlq_yet") | .message_id'

On the screen, the queue column says (DLQ, from orders) or (maxReceiveCount reached; not seen on the DLQ yet).

4Step 4: Read the timeline#

A poison message on orders (visibility timeout 30 s, maxReceiveCount 3), read by a DLQ worker:

The summary has source_queue: "orders", current_queue: "orders-dlq", in_dead_letter_queue: true and dead_letter_queue_move: "observed". The move event carries from_queue, to_queue, source_arn (the full ARN SQS sent) and reported_on_delivery. Delivery numbers run across queues, so the DLQ receive is delivery 4.

Compare deliveries 1 to 3. Same consumer and no delete each time, or a different consumer each time with the same failure in your logs: the problem is the message, not one consumer.

Edge cases#

• Several DLQ receives. Every receive from the DLQ carries the attribute. The move is placed before the first of them only. Between later DLQ receives whose receive count goes up, Trailbeam adds Visibility timeout expired steps, as on any queue.
• Receive count on the DLQ. No Deliveries not observed step is added for a first DLQ receive with a count above 1. That check runs only on the source queue. Whether AWS SQS carries the receive count over to the DLQ is not verified.
• Redrive back to the source queue. Not shown as a step. A later receive on the source queue makes it current_queue again and in_dead_letter_queue becomes false.
• Deleted at maxReceiveCount. If the last delivery was deleted, dead_letter_queue_move stays null. Nothing is waiting.
• Two queues with the same name in different accounts or regions are merged in 0.1.

Prevent it#

Poison messages are a processing problem, but you can catch them before they reach the DLQ:

• Alert on the DLQ. Run the stuck query for the DLQ from monitoring:

  curl -s 'http://127.0.0.1:4318/api/stuck?queue=orders-dlq&older_than_s=0' | jq '.messages | length'

• Keep Trailbeam's maxReceiveCount equal to the redrive policy, so not_seen_on_dlq_yet fires at the right count. The demo runs TRAILBEAM_MAX_RECEIVE_COUNT=orders=3 against a queue with a redrive policy after 3 receives.

• Release transient failures explicitly. ChangeMessageVisibility to 0 shows as Visibility changed 0 s, and the expiry's reason says so. That tells a deliberate retry apart from a crash in the timeline.

Related#

• Find stuck messages
• Recorded and inferred steps
• Span contract
• Run Trailbeam