Find stuck messages
List messages that were sent but never finished, see the last step each one reached, and tell a dead consumer from a lost delete.
Use this when something "never went through", a queue's ApproximateAgeOfOldestMessage is climbing, or a support ticket names a message that should have been handled.
1Step 1: Know what "stuck" means#
A message is listed as stuck when both are true:
- It was first seen more than
older_than_sseconds ago (default: Trailbeam's--stuck-after, 300 s). - Its latest receive has no final settle. That means no accepted delete (
trailbeam.result = ok) joined to that receive, and no Lambdasuccessoutcome for that receive's invocation.
"Latest receive" is the receive with the latest end time, on any queue. A message with no receive at all always passes rule 2.
First seen is the end of the send call, on the producer host's clock. If no send was observed, it is the earliest SentTimestamp a receive carries, on SQS's clock. With neither, it is the earliest span. The two clocks are never mixed.
So the list includes:
| Kind of message | state |
Why it is listed |
|---|---|---|
| Sent, never received | not_deleted | No receive, so nothing settled it |
| Received, delete rejected or never sent | not_deleted | No accepted delete for the latest receive |
| Deleted with an old handle, then redelivered | not_deleted | The accepted delete belongs to an earlier delivery (stale_handle) |
| Waiting in a dead-letter queue | not_deleted | Latest receive is on the DLQ and is not deleted. in_dead_letter_queue: true |
| Lambda invocation still running | processing | Latest receive is a Lambda receive within the Lambda timeout, with no outcome yet |
| Lambda killed or failed | not_deleted | No success for the latest invocation |
2Step 2: Open the stuck list#
Open /stuck (oldest first). The page takes older_than_s, queue and limit:
http://127.0.0.1:4318/stuck?older_than_s=600&queue=ordersFrom a script:
curl -s 'http://127.0.0.1:4318/api/stuck?older_than_s=600&queue=orders&limit=50'{"older_than_s": 600, "messages": [{
"message_id": "b1f0c2de-1c7e-4d0a-9a55-3c1e5f0a7d42",
"state": "not_deleted", "source_queue": "orders", "current_queue": "orders",
"in_dead_letter_queue": false, "dead_letter_queue_move": null,
"deliveries": 1, "successful_deliveries": 0,
"consumers": [{"service_name": "orders-worker", "client_id": "8c6e1f2a-5b0d-4c55-9d1e-0f7a3b2c9e10"}],
"first_seen": "2026-11-14T09:12:03.418000Z", "first_seen_clock": "host",
"last_seen": "2026-11-14T09:12:04.120000Z",
"age_s": 3712.582, "age_across_clocks": false
}]}| Parameter | Range | Default |
|---|---|---|
older_than_s |
0 to 1000000000 | Trailbeam's --stuck-after (TRAILBEAM_STUCK_AFTER_S), 300 |
queue |
a queue name | all queues. Matches messages with any span on that queue (messaging.destination.name) |
limit |
1 to 1000 | 100 |
lambda_timeout_s |
1 to 86400 | Trailbeam's --lambda-timeout, 900 |
age_s is Trailbeam's clock now minus first_seen. When age_across_clocks is true, the send was not observed, so the age compares SQS's clock with Trailbeam's. Clock skew is in that age and in whether the message is listed at all.
3Step 3: Read where it stopped#
Open the message (/message?id=... or /api/messages/{id}) and look at its last steps and its notices. A delete that SQS accepted but that came after the visibility timeout ran out looks like this:
- 09:12:03.418Sentorders-api · queue orders
- 09:12:04.120Receiveddelivery 1 · receive count 1 · orders-worker
- 09:12:35.240Deletedelivery 1 · result ok · status stale_handle
- inferredVisibility timeout expiredThe visibility timeout expired before the delete: SQS accepted the delete, but the message was received again, so the delete came too late to remove it.
- 09:12:36.010Receiveddelivery 2 · receive count 2 · orders-worker
Delivery 2 has no delete, so the message is stuck. The fix is the one for a slow consumer: see Debug a visibility-timeout redelivery.
| What the timeline shows | Cause | Fix |
|---|---|---|
Sent only (deliveries: 0) |
No consumer is polling, or the consumer has no Trailbeam | Check the consumer is running. Install the hooks in it. |
| Received, then nothing | The consumer crashed, hung, or skipped the delete. Or a process without Trailbeam deleted it. | Check the consumer named in consumer (service_name, pid, thread_name) at the receive time. |
Delete failed with an error_code |
SQS rejected the delete, for example ReceiptHandleIsInvalid |
Read the code. A late delete is the usual reason. |
Delete unmatched_handle plus a key_mismatch notice |
The deleting service uses a different TRAILBEAM_RECEIPT_HASH_KEY |
Use one key everywhere. |
Delete unmatched_handle plus an unmatched_handle notice |
No stored receive has that hash: its span has not arrived, or was lost | Check the receiving service exports spans. The join runs again when the receive arrives. |
| A Lambda Received, then inferred Lambda invocation ended without an outcome | The function timed out or ran out of memory | See Debug Lambda timeouts and crashes. |
| Moved to dead-letter queue, then Received on the DLQ | The message failed maxReceiveCount times and is waiting there |
See Trace a message to the dead-letter queue. |
dead_letter_queue_move: "not_seen_on_dlq_yet" |
It reached maxReceiveCount and nothing has read the DLQ yet |
Instrument the DLQ reader, or read the DLQ. |
curl -s http://127.0.0.1:4318/api/health | jq '{unjoined: .store.unjoined_settles, ingest}'4Step 4: Watch for a pattern#
One stuck message is a bug report. Many with the same last step are an incident. Group the list by consumer and state:
curl -s 'http://127.0.0.1:4318/api/stuck?limit=1000' \
| jq -r '.messages[] | [.state, .current_queue, .deliveries, (.consumers | map(.service_name) | join(","))] | @tsv' \
| sort | uniq -c | sort -rnMany deliveries: 0 messages on one queue point at its consumers. Many not_deleted messages whose consumers are all one service point at that service.
Prevent it#
Set the threshold to your real processing time. The default 300 s lists a message whose processing legitimately takes 10 minutes. Set it when you start Trailbeam:
trailbeam-store --stuck-after 900 # or TRAILBEAM_STUCK_AFTER_S=900The demo uses TRAILBEAM_STUCK_AFTER_S=60 with a 10 s visibility timeout.
Tell Trailbeam your Lambda timeout and redrive policy, so processing ends at the right time and DLQ arrivals are expected:
trailbeam-store --lambda-timeout 30 --max-receive-count orders=3,payments=5Alert on the list. Poll the API from cron or your monitoring system:
#!/bin/sh
n=$(curl -s 'http://127.0.0.1:4318/api/stuck?older_than_s=900&queue=orders' | jq '.messages | length')
[ "$n" -gt 0 ] && echo "orders: $n stuck message(s) older than 15 min" >&2 && exit 1
exit 0Share one receipt-hash key across every service that receives or deletes on a queue, so every delete joins:
import trailbeam.boto
trailbeam.boto.install() # reads TRAILBEAM_RECEIPT_HASH_KEYconst { SQSClient } = require('@aws-sdk/client-sqs');
const trailbeam = require('@trailbeam/sdk');
const sqs = trailbeam.instrument(new SQSClient({})); // reads TRAILBEAM_RECEIPT_HASH_KEYRelated#
Queue
Amazon SQS terms in Vocabulary
Message
Amazon SQS terms in Vocabulary
Receive
ReceiveMessage call returning a message. A receive does not remove the message; it starts its visibility timeout.Amazon SQS terms in Vocabulary
Delete
DeleteMessage, made with a receipt handle. This is the only way a consumer removes a message. With the handle of an older receive, the call can still succeed while the message is received again.Amazon SQS terms in Vocabulary
Producer
SendMessage or SendMessageBatch).Amazon SQS terms in Vocabulary
SentTimestamp
Amazon SQS terms in Vocabulary
Span
OpenTelemetry terms in Vocabulary
Delivery
Dead-letter queue (DLQ)
Amazon SQS terms in Vocabulary
Visibility timeout
Amazon SQS terms in Vocabulary
Consumer
ReceiveMessage) and then deletes them. A Lambda function behind an event source mapping is a consumer too.Amazon SQS terms in Vocabulary
Hooks
trailbeam.boto.install()), the AWS SDK v3 middleware (trailbeam.instrument(client)) and the Lambda wrapper.Inferred
inferred: true. See Recorded and inferred steps.maxReceiveCount
Amazon SQS terms in Vocabulary
Message ID
MessageId, assigned by SQS when the message is sent. It stays the same on every receive, including a receive from a dead-letter queue. Trailbeam joins everything about a message by this ID.Amazon SQS terms in Vocabulary
Timeline
Change visibility
ChangeMessageVisibility: sets a new visibility timeout on an in-flight message, counted from the call. A value of 0 makes it visible at once.Amazon SQS terms in Vocabulary
Redrive policy
maxReceiveCount.Amazon SQS terms in Vocabulary
Receipt-hash key
TRAILBEAM_RECEIPT_HASH_KEY. Every service on the same queues must use the same key.