How Trailbeam works
Hooks emit one span per message per SQS call, Trailbeam joins them by message ID and receipt-handle hash, and builds each timeline when you ask for it.
Trailbeam has three moving parts: hooks inside your services that turn SQS calls into OpenTelemetry spans, an optional Collector, and a server that keeps those spans in SQLite and builds a per-message timeline from them on every read.
Trailbeam never calls SQS. Everything it shows comes from spans it received.
What Trailbeam does and does not do#
- No SQS calls of its own. The packages only observe the calls your code already makes. Trailbeam never calls
ReceiveMessageon your queues, including the dead-letter queue, because a receive changes the message's receive count, visibility timeout and receipt handle. Trailbeam makes no AWS calls and needs no AWS credentials. - One change to your requests. On
ReceiveMessagethe client hooks add the system attributes Trailbeam needs (listed below). Nothing else in a request changes. - No extra IAM permissions. The hooks make no calls, so they need only the permissions your code already uses.
- No message bodies or message attributes. There is no option to capture them in 0.1.
- No receipt handles. Only a keyed hash of each handle leaves the process.
- No exception messages. On a handler exception the Lambda wrapper records the exception's class name only.
Trailbeam does record message IDs, queue names and queue URLs (a URL contains the AWS account ID), the AWSTraceHeader value, service.name, process IDs, Python thread names, and Lambda function names and invocation IDs.
The hooks: one span per message per call#
| Span name | trailbeam.kind |
Emitted for | Emitted by |
|---|---|---|---|
trailbeam send |
sqs.send |
each message SQS accepted in SendMessage / SendMessageBatch |
client hooks |
trailbeam receive |
sqs.receive |
each message ReceiveMessage returned, or each record of a Lambda SQS event |
client hooks, Lambda wrapper |
trailbeam delete |
sqs.settle |
each receipt handle in DeleteMessage / DeleteMessageBatch |
client hooks |
trailbeam change_visibility |
sqs.settle |
each receipt handle in ChangeMessageVisibility / ChangeMessageVisibilityBatch |
client hooks |
trailbeam outcome |
sqs.outcome |
each Lambda record, after the handler returns or throws | Lambda wrapper |
A batch call produces one Trailbeam span per entry. A ReceiveMessage that returns no messages produces no span. Failed SendMessageBatch entries have no message ID and produce no span.
Where the hooks sit#
- Python (boto3):
trailbeam.boto.install()registers arequest_hookandresponse_hookonopentelemetry-instrumentation-botocore. If your application already instrumented botocore, Trailbeam keeps that instrumentation and its tracer provider, and runs the Trailbeam hooks first, then yours. Otherwise Trailbeam instruments botocore with its own tracer provider, so the stock botocore span is exported to the Trailbeam endpoint too (Trailbeam counts it asignored). Each Trailbeam span is a child of the stock span for the same call. - Node.js (AWS SDK v3):
trailbeam.instrument(client)adds a middleware namedtrailbeamSqsMiddlewareat theinitializestep withpriority: 'high', ahead of serialization, signing and retries. - Lambda:
trailbeam.aws_lambda.wrap(Python) ortrailbeam.wrapHandler(Node.js) wraps the handler. Lambda's event source mapping makes theReceiveMessagecall, so the wrapper sees only the event.
What each span carries#
- An interval. The start is read just before the request is sent, the end when the response (or error) arrives. SQS acted somewhere inside. Python reads
time.time_ns()(nanoseconds). Node.js readsDate.now()(whole milliseconds, rounded down, so two intervals can look like they touch but never swap). A Lambdatrailbeam receiveis zero-length at the moment the wrapper got the event and carriestrailbeam.start_observed = false. - A call-order tick pair.
trailbeam.order.issuedandtrailbeam.order.returnedcome from one counter per Python process, or per Node.js realm (the main thread and each worker thread has its own), named bytrailbeam.order.scope. The hook takes theissuedtick before reading the start time, and thereturnedtick after reading the end time. See How steps are ordered. - SQS system attributes. On
ReceiveMessagethe hook addsApproximateReceiveCount,SentTimestamp,ApproximateFirstReceiveTimestamp,AWSTraceHeaderandDeadLetterQueueSourceArnto the request, unless you already asked for them or forAll. Nothing you asked for is removed, and no message attributes are requested. - A keyed hash of the receipt handle, never the handle:
trailbeam.receipt_handle.hmacandtrailbeam.receipt_handle.key_id. See Receipt-handle hashing. - Identity:
service.name(resource),messaging.client.id,process.pid, and in Pythonthread.name. A Lambda span also carriesfaas.name,faas.invocation_idandtrailbeam.queue.arn(the record'seventSourceARN).
Settle spans are emitted after the response, so they carry trailbeam.result (ok or failed) and, on failure, trailbeam.error_code. If the call throws before any response, one failed span per handle is still emitted.
The hooks never read the message body and never change the result or error your code sees. A hook that fails logs at debug level and lets the SQS call through.
Export#
Each package creates its own tracer provider with an always-on sampler and a batch span processor. Your application's sampling never drops Trailbeam spans, and the Node.js provider is not registered globally. Export is OTLP/HTTP protobuf to TRAILBEAM_OTLP_TRACES_ENDPOINT (default http://localhost:4318/v1/traces), straight to Trailbeam or through the Collector. The Collector configuration keeps every span with instrumentation scope trailbeam and drops every other span before Trailbeam. The Lambda wrappers force a flush after emitting receives (before your handler runs) and again after the outcomes, each bounded by a flush timeout of 2000 ms by default.
Trailbeam: ingest#
POST /v1/traces accepts application/x-protobuf (also an empty Content-Type) or application/json, optionally with Content-Encoding: gzip or deflate. The body limit is 32 MiB after decompression (413 above that).
For each span Trailbeam does this, in order:
- Scope filter. Spans whose instrumentation scope is not
trailbeamare counted asignoredand dropped. Span names are never parsed. - Version check.
trailbeam.schema_versionmust be2or3. Version 1 spans are rejected, and so is a span with no trace ID, span ID, start or end time, or one that ends before it starts. - Classify by
trailbeam.kind, and forsqs.settlebymessaging.operation.name(delete or change visibility). - Validate the fields each kind needs:
| Kind | Required |
|---|---|
| send | messaging.destination.name, messaging.message.id |
| receive | queue, message ID, trailbeam.receive_count, trailbeam.receipt_handle.hmac, trailbeam.receipt_handle.key_id |
| delete, change_visibility | queue, hash, key ID, trailbeam.result (ok or failed) |
| outcome | queue, message ID, trailbeam.outcome (success, failed_reported, failed_invalid_response, failed_exception) |
Integer attributes sent as decimal strings are read as integers. A span that fails any check is not stored. The response is still 200, with partial_success.rejected_spans and an error message listing up to five distinct reasons, and Trailbeam logs the same line as a warning.
- Insert one row per span in a single transaction, into the SQLite table
spans. The row keeps every attribute of the span as JSON; Trailbeam uses only the ones the contract names. A span already stored (same trace ID and span ID, for example an exporter retry) is skipped and counted as a duplicate.
Trailbeam: the join#
A delete or visibility change takes a receipt handle, not a message ID. The hooks keep an in-process map from handle to message ID (at most 100,000 entries, each for at most 12 hours), so a settle made in the process that received the message also carries messaging.message.id. Every row gets a mid, the message it belongs to:
- send, receive, outcome: the span's own
messaging.message.id; - delete, change visibility: the message ID of the
trailbeam receivewith the sametrailbeam.receipt_handle.hmacandtrailbeam.receipt_handle.key_id. The row is markedmid_source = 'receipt_hash'andreceive_rowpoints at that receive, so Trailbeam knows the exact delivery. If no such receive is stored yet, the settle row falls back to its own message ID (mid_source = 'span'), if it has one.
Spans arrive in any order, so the join runs both ways at insert time: a new settle looks for its receive, and a new receive claims any unjoined settles with its hash. A settle with no message ID and no matching receive has no mid and appears in no timeline. It is counted in store.unjoined_settles on /api/health.
Trailbeam: timelines on read#
Nothing about a timeline is stored. Each GET /api/messages/{id} (and each list endpoint, per listed message) loads every row with that mid and builds the timeline then:
- turn each row into an observed event (
sent,received,deleted,visibility_changed,lambda_outcome); - prove the order between them from four sources (ordering);
- add inferred events and reported dead-letter moves (recorded and inferred);
- cut the sequence into groups and mark concurrent ones (reading a timeline);
- compute the summary (
state,deliveries,successful_deliveries,first_seen) and notices.
Because the build runs on read, changing --lambda-timeout, passing lambda_timeout_s, or setting --max-receive-count changes existing timelines at once.
When something fails#
| What happens | What the timeline shows |
|---|---|
| A process exits before its batch span processor exports | Those spans are lost. A lost receive can show up as Deliveries not observed. |
| A Lambda invocation times out or runs out of memory | The receive was flushed before the handler ran. After --lambda-timeout Trailbeam infers Lambda invocation ended without an outcome. Until then, if it is the latest delivery, state is processing. |
| A producer or consumer has no Trailbeam hooks | A send_not_observed notice, or Deliveries not observed. |
| Services use different receipt-hash keys | Deletes cannot join their receives: a key_mismatch notice and deleted.status unmatched_handle. |
| No key is set | The published default key is used, with a default_key notice on every message page. |
| Host clocks are offset | clock edges between those hosts can be wrong. 0.1 does not correct clock offset. When sources disagree you get an order_conflict notice. |
| A delete uses an old receipt handle | Amazon SQS can accept it without deleting the message: deleted.status stale_handle, then a later receive. ElasticMQ rejects it instead (failed, ReceiptHandleIsInvalid). |
| Trailbeam is down, export goes through the Collector | The Collector retries until its in-memory queue of 5,000 batches is full, then drops new spans. |
| Trailbeam is down, export goes straight to it | The SDK exporter retries, then drops the batch. |
| The same span arrives twice | Skipped, and counted in ingest.duplicates. |
Why it's built this way#
Hooks in your process, not a poller. The only way to see a receive from outside is to call ReceiveMessage, which takes the message away from your consumers and raises its receive count. A tool that does that changes the thing it's observing. So Trailbeam runs where the calls already happen and records them as they pass.
Spans, not a custom protocol. You probably run OpenTelemetry already. Trailbeam spans travel through the same exporters and Collectors and can land in your own backend too. Trailbeam just reads a well-defined subset (Span contract).
Timelines on read, not on write. Spans arrive late and out of order, from many processes. Building the timeline on every read means a late span is never "too late". It also means config changes like --lambda-timeout apply to history immediately. The price is CPU per read, which is why list endpoints cap limit.
SQLite, one container. 0.1 is meant to be trivial to run next to a Collector and trivial to throw away. That choice limits scale, and Known limits is explicit about how much.
Worked example#
A worker reads one message with boto3 and deletes it 200 ms later. The handle map in that process knows the message ID, so the delete span carries it as well as the hash.
- 12:00:01.204Sentorders-api · SendMessage · 27 ms
- 12:00:01.230Receivedorders-worker · delivery 1 · receive count 1
- 12:00:01.431Deleteorders-worker · result ok · status final
In SQLite that is three rows: the send and receive with mid from their own message ID, and the delete with mid_source = 'receipt_hash' and receive_row set to the receive's row. Had a different process deleted it (empty handle map, no message ID on the span), the result would be identical: the hash alone joins it.
Edge cases#
| What you see | Cause | What to do |
|---|---|---|
/api/health shows ingest.requests at 0 |
Spans go elsewhere | Check TRAILBEAM_OTLP_TRACES_ENDPOINT in each service, including the /v1/traces path |
ingest.rejected grows |
Spans break the contract (old schema, missing attribute) | Read Trailbeam's warning log line; it names the reasons |
ingest.ignored grows |
Non-Trailbeam spans reach Trailbeam, such as stock botocore spans | Harmless. Filter them in the Collector if you want |
store.unjoined_settles stays above 0 |
Receiver not instrumented, its spans not arrived, or a different hash key | See Receipt-handle hashing |
| Spans missing from a short-lived script | The batch processor had not exported yet | Node.js: await trailbeam.forceFlush(). Python: trailbeam.boto.uninstall() flushes and shuts down |
Related#
- Reading a timeline
- Recorded and inferred steps
- How steps are ordered
- Receipt-handle hashing
- Span contract
- HTTP API
Hooks
trailbeam.boto.install()), the AWS SDK v3 middleware (trailbeam.instrument(client)) and the Lambda wrapper.Span
OpenTelemetry terms in Vocabulary
Collector
OpenTelemetry terms in Vocabulary
Server
trailbeam-store. It receives spans over OTLP/HTTP, keeps them in SQLite, and serves the JSON API and the web screen on one port.Message
Amazon SQS terms in Vocabulary
Timeline
Queue
Amazon SQS terms in Vocabulary
Dead-letter queue (DLQ)
Amazon SQS terms in Vocabulary
Receive
ReceiveMessage call returning a message. A receive does not remove the message; it starts its visibility timeout.Amazon SQS terms in Vocabulary
Receive count
ApproximateReceiveCount: how many times SQS has returned the message. It goes up by one on every receive.Amazon SQS terms in Vocabulary
Visibility timeout
Amazon SQS terms in Vocabulary
Receipt handle
Amazon SQS terms in Vocabulary
Handler
Amazon SQS terms in Vocabulary
Lambda wrapper
trailbeam.aws_lambda.wrap (Python) or trailbeam.wrapHandler (Node.js), around a handler invoked by an event source mapping.Message ID
MessageId, assigned by SQS when the message is sent. It stays the same on every receive, including a receive from a dead-letter queue. Trailbeam joins everything about a message by this ID.Amazon SQS terms in Vocabulary
Change visibility
ChangeMessageVisibility: sets a new visibility timeout on an in-flight message, counted from the call. A value of 0 makes it visible at once.Amazon SQS terms in Vocabulary
Trailbeam span
trailbeam. Trailbeam reads only these.Event source mapping
AWS Lambda terms in Vocabulary
SentTimestamp
Amazon SQS terms in Vocabulary
ApproximateFirstReceiveTimestamp
Amazon SQS terms in Vocabulary
DeadLetterQueueSourceArn
Amazon SQS terms in Vocabulary
Settle span
trailbeam delete or trailbeam change_visibility span: a call that settles what happens to an in-flight message.Sampling
OpenTelemetry terms in Vocabulary
OTLP
OpenTelemetry terms in Vocabulary
Instrumentation scope
trailbeam.OpenTelemetry terms in Vocabulary
Trace
OpenTelemetry terms in Vocabulary
Delete
DeleteMessage, made with a receipt handle. This is the only way a consumer removes a message. With the handle of an older receive, the call can still succeed while the message is received again.Amazon SQS terms in Vocabulary
Delivery
Recorded (observed)
inferred: false.Inferred
inferred: true. See Recorded and inferred steps.Producer
SendMessage or SendMessageBatch).Amazon SQS terms in Vocabulary
Consumer
ReceiveMessage) and then deletes them. A Lambda function behind an event source mapping is a consumer too.Amazon SQS terms in Vocabulary
Receipt-hash key
TRAILBEAM_RECEIPT_HASH_KEY. Every service on the same queues must use the same key.