Skip to content

How Trailbeam works

Hooks emit one span per message per SQS call, Trailbeam joins them by message ID and receipt-handle hash, and builds each timeline when you ask for it.

Based on 16 repo files at 54ea61b

docs/architecture.md, docs/glossary.md, docs/span-contract.md, docs/store-api.md, server/README.md, server/src/trailbeam_store/otlp.py, server/src/trailbeam_store/db.py, server/src/trailbeam_store/server.py, server/src/trailbeam_store/queries.py, server/src/trailbeam_store/timeline.py, python/src/trailbeam/boto.py, python/src/trailbeam/_emitter.py, python/src/trailbeam/aws_lambda.py, node/src/sqs.js, node/src/emitter.js, node/src/lambda.js

On this page

Trailbeam has three moving parts: hooks inside your services that turn SQS calls into OpenTelemetry spans, an optional Collector, and a server that keeps those spans in SQLite and builds a per-message timeline from them on every read.

Trailbeam architectureYour services call SQS as usual; the client hooks only add system attribute names to ReceiveMessage requests. The Trailbeam hooks inside them (trailbeam.boto.install, trailbeam.instrument, and the Lambda wrappers) have their own tracer provider with an always-on sampler and export one OpenTelemetry span per message per call over OTLP/HTTP, either through an optional OpenTelemetry Collector, which sends only spans with scope trailbeam on to Trailbeam, or straight to Trailbeam. The Collector can also send SQS spans to your own tracing backend. Trailbeam ingests spans at POST /v1/traces into the SQLite table spans, joins them by message ID and receipt-handle hash, and builds each timeline when it is read, for the web screen at / and /message and the JSON API under /api. Trailbeam makes no AWS calls. Amazon SQS Your SQS calls, unchanged but for ReceiveMessage attribute names Your services boto3trailbeam.boto.install() AWS SDK v3trailbeam.instrument(client) Lambda handlerthe Trailbeam wrapper Own tracer provider, always-on sampler Your tracing backend SQS spans OTel Collector optional; sends only scope trailbeam to the server OTLP/HTTP OTLP or export straight to the server Trailbeam server IngestPOST /v1/traces SQLitetable spans Timelinesbuilt when read Web screen/, /message JSON API/api/… Trailbeam only receives spans. It makes no AWS calls and needs no credentials, so it cannot change a receive count.
Figure 1 The data path. Hooks in your services turn SQS calls into spans and send them over OTLP/HTTP, through an optional Collector or straight to the server, which keeps them in SQLite and builds timelines on read. Dashed boxes are optional. Nothing flows from the server back to SQS.

Trailbeam never calls SQS. Everything it shows comes from spans it received.

What Trailbeam does and does not do#

  • No SQS calls of its own. The packages only observe the calls your code already makes. Trailbeam never calls ReceiveMessage on your queues, including the dead-letter queue, because a receive changes the message's receive count, visibility timeout and receipt handle. Trailbeam makes no AWS calls and needs no AWS credentials.
  • One change to your requests. On ReceiveMessage the client hooks add the system attributes Trailbeam needs (listed below). Nothing else in a request changes.
  • No extra IAM permissions. The hooks make no calls, so they need only the permissions your code already uses.
  • No message bodies or message attributes. There is no option to capture them in 0.1.
  • No receipt handles. Only a keyed hash of each handle leaves the process.
  • No exception messages. On a handler exception the Lambda wrapper records the exception's class name only.

Trailbeam does record message IDs, queue names and queue URLs (a URL contains the AWS account ID), the AWSTraceHeader value, service.name, process IDs, Python thread names, and Lambda function names and invocation IDs.

The hooks: one span per message per call#

Span name trailbeam.kind Emitted for Emitted by
trailbeam send sqs.send each message SQS accepted in SendMessage / SendMessageBatch client hooks
trailbeam receive sqs.receive each message ReceiveMessage returned, or each record of a Lambda SQS event client hooks, Lambda wrapper
trailbeam delete sqs.settle each receipt handle in DeleteMessage / DeleteMessageBatch client hooks
trailbeam change_visibility sqs.settle each receipt handle in ChangeMessageVisibility / ChangeMessageVisibilityBatch client hooks
trailbeam outcome sqs.outcome each Lambda record, after the handler returns or throws Lambda wrapper

A batch call produces one Trailbeam span per entry. A ReceiveMessage that returns no messages produces no span. Failed SendMessageBatch entries have no message ID and produce no span.

Where the hooks sit#

  • Python (boto3): trailbeam.boto.install() registers a request_hook and response_hook on opentelemetry-instrumentation-botocore. If your application already instrumented botocore, Trailbeam keeps that instrumentation and its tracer provider, and runs the Trailbeam hooks first, then yours. Otherwise Trailbeam instruments botocore with its own tracer provider, so the stock botocore span is exported to the Trailbeam endpoint too (Trailbeam counts it as ignored). Each Trailbeam span is a child of the stock span for the same call.
  • Node.js (AWS SDK v3): trailbeam.instrument(client) adds a middleware named trailbeamSqsMiddleware at the initialize step with priority: 'high', ahead of serialization, signing and retries.
  • Lambda: trailbeam.aws_lambda.wrap (Python) or trailbeam.wrapHandler (Node.js) wraps the handler. Lambda's event source mapping makes the ReceiveMessage call, so the wrapper sees only the event.

What each span carries#

  • An interval. The start is read just before the request is sent, the end when the response (or error) arrives. SQS acted somewhere inside. Python reads time.time_ns() (nanoseconds). Node.js reads Date.now() (whole milliseconds, rounded down, so two intervals can look like they touch but never swap). A Lambda trailbeam receive is zero-length at the moment the wrapper got the event and carries trailbeam.start_observed = false.
  • A call-order tick pair. trailbeam.order.issued and trailbeam.order.returned come from one counter per Python process, or per Node.js realm (the main thread and each worker thread has its own), named by trailbeam.order.scope. The hook takes the issued tick before reading the start time, and the returned tick after reading the end time. See How steps are ordered.
  • SQS system attributes. On ReceiveMessage the hook adds ApproximateReceiveCount, SentTimestamp, ApproximateFirstReceiveTimestamp, AWSTraceHeader and DeadLetterQueueSourceArn to the request, unless you already asked for them or for All. Nothing you asked for is removed, and no message attributes are requested.
  • A keyed hash of the receipt handle, never the handle: trailbeam.receipt_handle.hmac and trailbeam.receipt_handle.key_id. See Receipt-handle hashing.
  • Identity: service.name (resource), messaging.client.id, process.pid, and in Python thread.name. A Lambda span also carries faas.name, faas.invocation_id and trailbeam.queue.arn (the record's eventSourceARN).

Settle spans are emitted after the response, so they carry trailbeam.result (ok or failed) and, on failure, trailbeam.error_code. If the call throws before any response, one failed span per handle is still emitted.

The hooks never read the message body and never change the result or error your code sees. A hook that fails logs at debug level and lets the SQS call through.

Export#

Each package creates its own tracer provider with an always-on sampler and a batch span processor. Your application's sampling never drops Trailbeam spans, and the Node.js provider is not registered globally. Export is OTLP/HTTP protobuf to TRAILBEAM_OTLP_TRACES_ENDPOINT (default http://localhost:4318/v1/traces), straight to Trailbeam or through the Collector. The Collector configuration keeps every span with instrumentation scope trailbeam and drops every other span before Trailbeam. The Lambda wrappers force a flush after emitting receives (before your handler runs) and again after the outcomes, each bounded by a flush timeout of 2000 ms by default.

Trailbeam: ingest#

POST /v1/traces accepts application/x-protobuf (also an empty Content-Type) or application/json, optionally with Content-Encoding: gzip or deflate. The body limit is 32 MiB after decompression (413 above that).

For each span Trailbeam does this, in order:

  1. Scope filter. Spans whose instrumentation scope is not trailbeam are counted as ignored and dropped. Span names are never parsed.
  2. Version check. trailbeam.schema_version must be 2 or 3. Version 1 spans are rejected, and so is a span with no trace ID, span ID, start or end time, or one that ends before it starts.
  3. Classify by trailbeam.kind, and for sqs.settle by messaging.operation.name (delete or change visibility).
  4. Validate the fields each kind needs:
Kind Required
send messaging.destination.name, messaging.message.id
receive queue, message ID, trailbeam.receive_count, trailbeam.receipt_handle.hmac, trailbeam.receipt_handle.key_id
delete, change_visibility queue, hash, key ID, trailbeam.result (ok or failed)
outcome queue, message ID, trailbeam.outcome (success, failed_reported, failed_invalid_response, failed_exception)

Integer attributes sent as decimal strings are read as integers. A span that fails any check is not stored. The response is still 200, with partial_success.rejected_spans and an error message listing up to five distinct reasons, and Trailbeam logs the same line as a warning.

  1. Insert one row per span in a single transaction, into the SQLite table spans. The row keeps every attribute of the span as JSON; Trailbeam uses only the ones the contract names. A span already stored (same trace ID and span ID, for example an exporter retry) is skipped and counted as a duplicate.

Trailbeam: the join#

A delete or visibility change takes a receipt handle, not a message ID. The hooks keep an in-process map from handle to message ID (at most 100,000 entries, each for at most 12 hours), so a settle made in the process that received the message also carries messaging.message.id. Every row gets a mid, the message it belongs to:

  • send, receive, outcome: the span's own messaging.message.id;
  • delete, change visibility: the message ID of the trailbeam receive with the same trailbeam.receipt_handle.hmac and trailbeam.receipt_handle.key_id. The row is marked mid_source = 'receipt_hash' and receive_row points at that receive, so Trailbeam knows the exact delivery. If no such receive is stored yet, the settle row falls back to its own message ID (mid_source = 'span'), if it has one.

Spans arrive in any order, so the join runs both ways at insert time: a new settle looks for its receive, and a new receive claims any unjoined settles with its hash. A settle with no message ID and no matching receive has no mid and appears in no timeline. It is counted in store.unjoined_settles on /api/health.

Trailbeam: timelines on read#

Nothing about a timeline is stored. Each GET /api/messages/{id} (and each list endpoint, per listed message) loads every row with that mid and builds the timeline then:

  1. turn each row into an observed event (sent, received, deleted, visibility_changed, lambda_outcome);
  2. prove the order between them from four sources (ordering);
  3. add inferred events and reported dead-letter moves (recorded and inferred);
  4. cut the sequence into groups and mark concurrent ones (reading a timeline);
  5. compute the summary (state, deliveries, successful_deliveries, first_seen) and notices.

Because the build runs on read, changing --lambda-timeout, passing lambda_timeout_s, or setting --max-receive-count changes existing timelines at once.

When something fails#

What happens What the timeline shows
A process exits before its batch span processor exports Those spans are lost. A lost receive can show up as Deliveries not observed.
A Lambda invocation times out or runs out of memory The receive was flushed before the handler ran. After --lambda-timeout Trailbeam infers Lambda invocation ended without an outcome. Until then, if it is the latest delivery, state is processing.
A producer or consumer has no Trailbeam hooks A send_not_observed notice, or Deliveries not observed.
Services use different receipt-hash keys Deletes cannot join their receives: a key_mismatch notice and deleted.status unmatched_handle.
No key is set The published default key is used, with a default_key notice on every message page.
Host clocks are offset clock edges between those hosts can be wrong. 0.1 does not correct clock offset. When sources disagree you get an order_conflict notice.
A delete uses an old receipt handle Amazon SQS can accept it without deleting the message: deleted.status stale_handle, then a later receive. ElasticMQ rejects it instead (failed, ReceiptHandleIsInvalid).
Trailbeam is down, export goes through the Collector The Collector retries until its in-memory queue of 5,000 batches is full, then drops new spans.
Trailbeam is down, export goes straight to it The SDK exporter retries, then drops the batch.
The same span arrives twice Skipped, and counted in ingest.duplicates.

Why it's built this way#

Hooks in your process, not a poller. The only way to see a receive from outside is to call ReceiveMessage, which takes the message away from your consumers and raises its receive count. A tool that does that changes the thing it's observing. So Trailbeam runs where the calls already happen and records them as they pass.

Spans, not a custom protocol. You probably run OpenTelemetry already. Trailbeam spans travel through the same exporters and Collectors and can land in your own backend too. Trailbeam just reads a well-defined subset (Span contract).

Timelines on read, not on write. Spans arrive late and out of order, from many processes. Building the timeline on every read means a late span is never "too late". It also means config changes like --lambda-timeout apply to history immediately. The price is CPU per read, which is why list endpoints cap limit.

SQLite, one container. 0.1 is meant to be trivial to run next to a Collector and trivial to throw away. That choice limits scale, and Known limits is explicit about how much.

Worked example#

A worker reads one message with boto3 and deletes it 200 ms later. The handle map in that process knows the message ID, so the delete span carries it as well as the hash.

Timeline sample
  1. 12:00:01.204Sentorders-api · SendMessage · 27 ms
  2. 12:00:01.230Receivedorders-worker · delivery 1 · receive count 1
  3. 12:00:01.431Deleteorders-worker · result ok · status final

In SQLite that is three rows: the send and receive with mid from their own message ID, and the delete with mid_source = 'receipt_hash' and receive_row set to the receive's row. Had a different process deleted it (empty handle map, no message ID on the span), the result would be identical: the hash alone joins it.

Edge cases#

What you see Cause What to do
/api/health shows ingest.requests at 0 Spans go elsewhere Check TRAILBEAM_OTLP_TRACES_ENDPOINT in each service, including the /v1/traces path
ingest.rejected grows Spans break the contract (old schema, missing attribute) Read Trailbeam's warning log line; it names the reasons
ingest.ignored grows Non-Trailbeam spans reach Trailbeam, such as stock botocore spans Harmless. Filter them in the Collector if you want
store.unjoined_settles stays above 0 Receiver not instrumented, its spans not arrived, or a different hash key See Receipt-handle hashing
Spans missing from a short-lived script The batch processor had not exported yet Node.js: await trailbeam.forceFlush(). Python: trailbeam.boto.uninstall() flushes and shuts down