Is Trailbeam a good fit?

We'd rather you find out here than three weeks into a rollout. Trailbeam does one thing: it answers what happened to this SQS message? with a timeline you can defend. It is deliberately narrow, and 0.1 is a pre-release. Here is where we think it fits, and where it doesn't.

Trailbeam fits when#

Trailbeam fits if most of these are true for you. Each point names what Trailbeam actually records, so you can check it against your own needs.

You need to follow one message end to end, by its message ID. Trailbeam builds one timeline per messaging.message.id, across every service that touches the message: the send, every receive (with SQS's own ApproximateReceiveCount, SentTimestamp and ApproximateFirstReceiveTimestamp), every ChangeMessageVisibility, the delete and whether SQS accepted it, the outcome your handler reported, and the move to a dead-letter queue (from DeadLetterQueueSourceArn). Visibility timeouts that expired between those steps are drawn as inferred steps with their time window. See Reading a timeline.

You need to know which consumer handled each delivery: the service, the pod, the process and the thread. Every step on a timeline carries a consumer block:

So "received by orders-worker, pod orders-worker-7c9f-x2k4, pid 4242, thread worker-1, then deleted 40 s later by cleanup-job" is what you read off the timeline, not something you piece together from logs.

You need every step to point back to the span that produced it. Each observed step carries the trace_id and span_id of exactly one Trailbeam span, so you can open it in your tracing backend. Inferred steps are labelled inferred and carry a reason string that says what they were deduced from, for example the receive count jumping or the visibility timeout running out. Two steps are drawn in order only when a host clock, a per-process call counter, SQS's receive count or SQS's own timestamps prove it, and each step names the fact that placed it; otherwise they share a box marked concurrent, order unknown. If you've been burned by tools that showed a tidy story that turned out to be wrong, this is the trade we made. See How steps are ordered and Recorded and inferred steps.

Receives and deletes happen in different places. Batch deleters, cleanup jobs and Lambda's event source mapping all settle messages away from the code that received them. Trailbeam joins each delete to the exact delivery by an HMAC of the receipt handle, keyed with a secret you share across services, so a delete from another process lands on the right delivery even after redeliveries. This is the case Trailbeam handles best, not an edge case. See Receipt-handle hashing.

You don't want sampling to lose the message you're asked about. Trailbeam sends SQS spans through its own tracer provider with an always-on sampler, separately from your app's tracing, so every send, receive and delete is kept even if you sample traces at 1%. See Keep SQS spans when sampling.

Your SQS code is Python (boto3) or Node.js (AWS SDK v3), including Lambda. These are the only two packages. One call wraps the client or the handler, and the hooks never change a result or an error your code sees.

You don't run Datadog or another APM, and you'd rather not bring one in for this. Trailbeam is meant to be small enough to add for one queue:

• Hooks: only the OpenTelemetry SDK and its OTLP/HTTP exporter (plus opentelemetry-instrumentation-botocore in Python). No agent or sidecar on your hosts.
• Trailbeam: one container built on python:slim, with one runtime dependency (opentelemetry-proto), a SQLite file, and OTLP/HTTP plus the timeline UI and API on one port (127.0.0.1:4318 by default).
• No vendor account and no SaaS. Spans go from your services to the Trailbeam you run, and nowhere else.

If you do run an APM, Trailbeam still fits. It isn't meant to replace it. The Collector overlay sends the same SQS spans to Trailbeam and to your existing backend.

Trailbeam is the wrong tool when#

Some cases fall in between. Be careful with these:

• Hosts with drifting clocks. The clock source trusts each host's wall clock, and 0.1 does not correct offsets between hosts. With NTP this is fine. Without it, a delete can appear before the receive it used. The hash join still names the right delivery, but the drawn order is wrong.
• FIFO queues. The repo has no FIFO-specific handling or tests. Message groups and deduplication IDs are not recorded. We would treat FIFO as untested.
• High volume through one instance. Requests to Trailbeam are serialised over one SQLite connection, and each timeline is built on read. There are no published throughput numbers for 0.1. Trailbeam is sized for a single container, so measure before you point a busy fleet at it.
• A partly instrumented queue. If only some consumers carry the hooks, timelines have real gaps. Trailbeam says so, but it's easy to misread a gap as a bug in your code.

What you take on#

Trailbeam never records message bodies, message attributes or receipt handles (only a keyed hash of each handle). It does record message IDs, queue names and queue URLs (a URL contains your AWS account ID), the AWSTraceHeader value, service names, process IDs, Python thread names, and Lambda function names and invocation IDs. If those need a review where you work, that's the list.

Before you decide, read Known limits. It covers every limit we know of, with what it does to your timelines.

How we'd roll it out#

1. Run the Quickstart first. About ten minutes, no AWS account. The demo plays out redeliveries, a failed late delete, a dead-lettered message and duplicates, so you see what each looks like before it happens to you.
2. Pick one queue and instrument all of it. That means the producer, every consumer and anything that deletes. A whole queue gives timelines you can trust; a third of the fleet gives gaps.
3. Set a real receipt-hash key on day one. Changing it later breaks the join for any message received under the old key and settled under the new one (key_mismatch).
4. Tell Trailbeam about your queues. Set --lambda-timeout to your function's timeout and --max-receive-count to each queue's redrive setting, so inferred timeouts and dead-letter notices are right. See Configuration.
5. Wait for a real incident, then compare. Open the message in Trailbeam next to whatever you used before. If Trailbeam didn't answer faster, tell us; that is the test that matters.

How it sits next to what you have#

Trailbeam doesn't replace your observability stack. It covers the one view that stack is bad at.

Related#

• Quickstart
• Known limits
• How Trailbeam works
• Add Trailbeam to your services