Trailbeam documentation
Message-level timelines for Amazon SQS. Look up any message by ID and see what happened to it, step by step.
When someone asks "what happened to this message?", the honest answer is usually scattered across three services' logs and a CloudWatch graph. Trailbeam puts it in one place. Hooks in your services record every SQS call per message, and Trailbeam, one process you run yourself, turns them into a timeline:
- 12:00:01.204Sentorders-api · SendMessage
- 12:00:01.230Received, delivery 1worker-a · receive count 1
- inferredVisibility timeout expiredafter delivery 1, before delivery 2
- 12:00:11.402Received, delivery 2worker-b · receive count 2
- 12:00:11.610Deletedworker-b · delivery 2 · final
- 12:00:13.118Delete rejectedworker-a · delivery 1 · too late
That message ran twice. Worker A was too slow, SQS handed the message to worker B, and A's late delete bounced. Every line says who recorded it, and the one step nobody could record is marked inferred.
Three things Trailbeam will not do#
- Touch your queues. Trailbeam makes no SQS calls of its own. The hooks only observe the calls your code already makes, and Trailbeam makes no AWS calls at all, so it can never raise a receive count.
- Read your messages. Hooks never read bodies or message attributes, and never export a receipt handle.
- Guess. Two steps are drawn in order only when an observed fact proves it: the hosts' clocks, the call order inside one process, SQS's receive count or SQS's timestamps. Otherwise they share a box marked concurrent, order unknown.
Trailbeam 0.1 is a pre-release: the repository is private and nothing is published yet. If you're weighing it up, start with Is Trailbeam a good fit?. It's candid about where Trailbeam falls short.
Message
Amazon SQS terms in Vocabulary
Hooks
trailbeam.boto.install()), the AWS SDK v3 middleware (trailbeam.instrument(client)) and the Lambda wrapper.Timeline
Delete
DeleteMessage, made with a receipt handle. This is the only way a consumer removes a message. With the handle of an older receive, the call can still succeed while the message is received again.Amazon SQS terms in Vocabulary
Step
Inferred
inferred: true. See Recorded and inferred steps.Queue
Amazon SQS terms in Vocabulary
Receive count
ApproximateReceiveCount: how many times SQS has returned the message. It goes up by one on every receive.Amazon SQS terms in Vocabulary
Receipt handle
Amazon SQS terms in Vocabulary
Recorded (observed)
inferred: false.Start here#
- Is Trailbeam a good fit?Where Trailbeam earns its place, where it doesn't, and what you take on by running 0.1. Worth five minutes before you instrument anything.
- QuickstartRun the local demo and open your first message timeline. About 10 minutes, no AWS account.
- Add Trailbeam to your servicesWhich package goes in which service, the minimal code for each runtime, and how to confirm spans reach Trailbeam.
Solve a problem#
Step-by-step guides for the questions people bring to Trailbeam.
- A message ran twiceA customer was charged twice or an email went out twice. Find the second delivery, see which consumer ran it, and fix the cause.
- Find stuck messagesList messages that were sent but never finished, see the last step each one reached, and tell a dead consumer from a lost delete.
- Trace a message to the DLQSee each failed delivery, the move SQS reported, and what is waiting in the DLQ, and know why a move is or is not shown.
- Visibility-timeout redeliveriesTell a slow consumer from a released message or a heartbeat that stopped, and see whose delete failed and why.
- Lambda timeouts and crashesSee how each Lambda invocation ended for each SQS record, including timeouts and out-of-memory kills the handler never saw.
- Keep SQS spans when samplingSend every Trailbeam span to Trailbeam, and every SQS span to your tracing backend, while the rest of your traces stay sampled.
- Use timelines in your toolsPull a message's timeline, the stuck list or duplicates as JSON for scripts, support tools and alerts, and read the fields correctly.