Skip to content

Find stuck messages

List messages that were sent but never finished, see the last step each one reached, and tell a dead consumer from a lost delete.

Based on 11 repo files at 54ea61b

server/src/trailbeam_store/queries.py, server/src/trailbeam_store/timeline.py, server/src/trailbeam_store/server.py, server/src/trailbeam_store/db.py, server/src/trailbeam_store/web.py, server/src/trailbeam_store/__main__.py, server/tests/test_timeline.py, server/tests/test_api.py, docs/store-api.md, server/README.md, demo/compose.yaml

On this page

Use this when something "never went through", a queue's ApproximateAgeOfOldestMessage is climbing, or a support ticket names a message that should have been handled.

1Step 1: Know what "stuck" means#

A message is listed as stuck when both are true:

  1. It was first seen more than older_than_s seconds ago (default: Trailbeam's --stuck-after, 300 s).
  2. Its latest receive has no final settle. That means no accepted delete (trailbeam.result = ok) joined to that receive, and no Lambda success outcome for that receive's invocation.

"Latest receive" is the receive with the latest end time, on any queue. A message with no receive at all always passes rule 2.

First seen is the end of the send call, on the producer host's clock. If no send was observed, it is the earliest SentTimestamp a receive carries, on SQS's clock. With neither, it is the earliest span. The two clocks are never mixed.

So the list includes:

Kind of message state Why it is listed
Sent, never received not_deleted No receive, so nothing settled it
Received, delete rejected or never sent not_deleted No accepted delete for the latest receive
Deleted with an old handle, then redelivered not_deleted The accepted delete belongs to an earlier delivery (stale_handle)
Waiting in a dead-letter queue not_deleted Latest receive is on the DLQ and is not deleted. in_dead_letter_queue: true
Lambda invocation still running processing Latest receive is a Lambda receive within the Lambda timeout, with no outcome yet
Lambda killed or failed not_deleted No success for the latest invocation

2Step 2: Open the stuck list#

Open /stuck (oldest first). The page takes older_than_s, queue and limit:

text
http://127.0.0.1:4318/stuck?older_than_s=600&queue=orders

From a script:

shell
curl -s 'http://127.0.0.1:4318/api/stuck?older_than_s=600&queue=orders&limit=50'
json
{"older_than_s": 600, "messages": [{
  "message_id": "b1f0c2de-1c7e-4d0a-9a55-3c1e5f0a7d42",
  "state": "not_deleted", "source_queue": "orders", "current_queue": "orders",
  "in_dead_letter_queue": false, "dead_letter_queue_move": null,
  "deliveries": 1, "successful_deliveries": 0,
  "consumers": [{"service_name": "orders-worker", "client_id": "8c6e1f2a-5b0d-4c55-9d1e-0f7a3b2c9e10"}],
  "first_seen": "2026-11-14T09:12:03.418000Z", "first_seen_clock": "host",
  "last_seen": "2026-11-14T09:12:04.120000Z",
  "age_s": 3712.582, "age_across_clocks": false
}]}
Parameter Range Default
older_than_s 0 to 1000000000 Trailbeam's --stuck-after (TRAILBEAM_STUCK_AFTER_S), 300
queue a queue name all queues. Matches messages with any span on that queue (messaging.destination.name)
limit 1 to 1000 100
lambda_timeout_s 1 to 86400 Trailbeam's --lambda-timeout, 900

age_s is Trailbeam's clock now minus first_seen. When age_across_clocks is true, the send was not observed, so the age compares SQS's clock with Trailbeam's. Clock skew is in that age and in whether the message is listed at all.

3Step 3: Read where it stopped#

Open the message (/message?id=... or /api/messages/{id}) and look at its last steps and its notices. A delete that SQS accepted but that came after the visibility timeout ran out looks like this:

Timeline sample
  1. 09:12:03.418Sentorders-api · queue orders
  2. 09:12:04.120Receiveddelivery 1 · receive count 1 · orders-worker
  3. 09:12:35.240Deletedelivery 1 · result ok · status stale_handle
  4. inferredVisibility timeout expiredThe visibility timeout expired before the delete: SQS accepted the delete, but the message was received again, so the delete came too late to remove it.
  5. 09:12:36.010Receiveddelivery 2 · receive count 2 · orders-worker

Delivery 2 has no delete, so the message is stuck. The fix is the one for a slow consumer: see Debug a visibility-timeout redelivery.

What the timeline shows Cause Fix
Sent only (deliveries: 0) No consumer is polling, or the consumer has no Trailbeam Check the consumer is running. Install the hooks in it.
Received, then nothing The consumer crashed, hung, or skipped the delete. Or a process without Trailbeam deleted it. Check the consumer named in consumer (service_name, pid, thread_name) at the receive time.
Delete failed with an error_code SQS rejected the delete, for example ReceiptHandleIsInvalid Read the code. A late delete is the usual reason.
Delete unmatched_handle plus a key_mismatch notice The deleting service uses a different TRAILBEAM_RECEIPT_HASH_KEY Use one key everywhere.
Delete unmatched_handle plus an unmatched_handle notice No stored receive has that hash: its span has not arrived, or was lost Check the receiving service exports spans. The join runs again when the receive arrives.
A Lambda Received, then inferred Lambda invocation ended without an outcome The function timed out or ran out of memory See Debug Lambda timeouts and crashes.
Moved to dead-letter queue, then Received on the DLQ The message failed maxReceiveCount times and is waiting there See Trace a message to the dead-letter queue.
dead_letter_queue_move: "not_seen_on_dlq_yet" It reached maxReceiveCount and nothing has read the DLQ yet Instrument the DLQ reader, or read the DLQ.
shell
curl -s http://127.0.0.1:4318/api/health | jq '{unjoined: .store.unjoined_settles, ingest}'

4Step 4: Watch for a pattern#

One stuck message is a bug report. Many with the same last step are an incident. Group the list by consumer and state:

shell
curl -s 'http://127.0.0.1:4318/api/stuck?limit=1000' \
  | jq -r '.messages[] | [.state, .current_queue, .deliveries, (.consumers | map(.service_name) | join(","))] | @tsv' \
  | sort | uniq -c | sort -rn

Many deliveries: 0 messages on one queue point at its consumers. Many not_deleted messages whose consumers are all one service point at that service.

Prevent it#

Set the threshold to your real processing time. The default 300 s lists a message whose processing legitimately takes 10 minutes. Set it when you start Trailbeam:

shell
trailbeam-store --stuck-after 900          # or TRAILBEAM_STUCK_AFTER_S=900

The demo uses TRAILBEAM_STUCK_AFTER_S=60 with a 10 s visibility timeout.

Tell Trailbeam your Lambda timeout and redrive policy, so processing ends at the right time and DLQ arrivals are expected:

shell
trailbeam-store --lambda-timeout 30 --max-receive-count orders=3,payments=5

Alert on the list. Poll the API from cron or your monitoring system:

shell
#!/bin/sh
n=$(curl -s 'http://127.0.0.1:4318/api/stuck?older_than_s=900&queue=orders' | jq '.messages | length')
[ "$n" -gt 0 ] && echo "orders: $n stuck message(s) older than 15 min" >&2 && exit 1
exit 0

Share one receipt-hash key across every service that receives or deletes on a queue, so every delete joins:

python
import trailbeam.boto
trailbeam.boto.install()   # reads TRAILBEAM_RECEIPT_HASH_KEY
javascript
const { SQSClient } = require('@aws-sdk/client-sqs');
const trailbeam = require('@trailbeam/sdk');
const sqs = trailbeam.instrument(new SQSClient({}));   // reads TRAILBEAM_RECEIPT_HASH_KEY