Skip to content
Docs

Keep every SQS span when you sample

Send every Trailbeam span to Trailbeam, and every SQS span to your tracing backend, while the rest of your traces stay sampled.

Based on 14 repo files at 54ea61b

collector/trailbeam-collector.yaml, collector/backend.yaml, collector/README.md, collector/tests/test_collector.py, python/src/trailbeam/_emitter.py, python/src/trailbeam/boto.py, node/src/emitter.js, node/README.md, server/src/trailbeam_store/otlp.py, server/src/trailbeam_store/server.py, server/src/trailbeam_store/timeline.py, demo/compose.yaml, docs/architecture.md, docs/span-contract.md

On this page

Use this when you sample traces (head sampling in the SDK, or tail sampling in a Collector) and want Trailbeam timelines to stay complete.

A timeline is built from every span of one message. Sampling drops spans per trace, and one message's send, receives and deletes are usually in different traces. A dropped receive or delete does not make a trace look thinner. It makes the timeline wrong.

1Step 1: See what a dropped span does to a timeline#

A message whose first receive span was dropped:

Timeline sample
  1. 11:40:00.002Sentorders-api · queue orders
  2. inferredDeliveries not observedThe first observed receive has receive count 2, so 1 earlier receive(s) were not observed (a consumer without Trailbeam, or lost spans).
  3. 11:40:31.200Receiveddelivery 1 · receive count 2 · orders-worker
  4. 11:40:31.540Deletedelivery 1 · result ok · status final
What the timeline shows Span that was lost What it costs you
Inferred Deliveries not observed with a count One or more trailbeam receive The delivery, its consumer and the expiry around it are gone
A send_not_observed notice, first_seen_clock: "sqs" trailbeam send first_seen falls back to SQS's SentTimestamp. The stuck age then compares two clocks.
state: not_deleted and the message on /stuck, though it was handled trailbeam delete A false stuck message
Delete unmatched_handle with an unmatched_handle notice The trailbeam receive that returned the handle The delete cannot be attached to a delivery
A Lambda invocation ended without an outcome step after a run that worked trailbeam outcome A false kill

2Step 2: Know that the packages never sample#

The Python and Node.js packages create their own tracer provider for Trailbeam spans:

Python (trailbeam) Node.js (@trailbeam/sdk)
Provider TracerProvider(sampler=ALWAYS_ON) BasicTracerProvider with AlwaysOnSampler
Registered globally No, it is only used by the hooks No
Processor BatchSpanProcessor BatchSpanProcessor
Exporter OTLP/HTTP protobuf to TRAILBEAM_OTLP_TRACES_ENDPOINT OTLP/HTTP protobuf to TRAILBEAM_OTLP_TRACES_ENDPOINT
Default endpoint http://localhost:4318/v1/traces http://localhost:4318/v1/traces

The sampler is set in code and ignores the parent's sampled flag. Your application's sampling decides what happens to your own spans only. Every Trailbeam span is still a child of the stock AWS SDK span for the same call when one exists, so it shares that trace ID.

3Step 3: Route Trailbeam's spans through the Trailbeam Collector config#

Point the packages at a Collector running trailbeam-collector.yaml. It needs otelcol-contrib (tested with 0.165.0).

shell
trailbeam-store --port 4319 &                                  # the Collector takes 4318 on this host
TRAILBEAM_STORE_ENDPOINT=http://localhost:4319 otelcol-contrib --config trailbeam-collector.yaml

What the pipeline traces/trailbeam does:

Component Setting Effect
otlp receiver HTTP on TRAILBEAM_COLLECTOR_HTTP_ENDPOINT (localhost:4318), gRPC on TRAILBEAM_COLLECTOR_GRPC_ENDPOINT (localhost:4317) Takes OTLP from the packages
filter/trailbeam_only drops spans where instrumentation_scope.name != "trailbeam" Trailbeam gets only Trailbeam spans
batch defaults Batches before export
otlp_http/trailbeam_store endpoint TRAILBEAM_STORE_ENDPOINT (http://localhost:4319), compression: gzip The exporter appends /v1/traces
retry_on_failure.max_elapsed_time: 0s Retries forever while Trailbeam is down
sending_queue.queue_size: 5000 Holds up to 5,000 batches in memory

There is no sampling processor in this pipeline. Do not add one.

4Step 4: Add your tracing backend, with SQS spans unsampled#

Layer backend.yaml on top:

shell
TRAILBEAM_BACKEND_ENDPOINT=http://jaeger:4318 \
TRAILBEAM_BACKEND_SAMPLING_PERCENTAGE=5 \
  otelcol-contrib --config trailbeam-collector.yaml --config backend.yaml

It adds two pipelines from the same otlp receiver to otlp_http/backend:

Pipeline Processors What reaches the backend
traces/backend_sqs filter/sqs_only, batch Every SQS span, unsampled
traces/backend_rest filter/drop_sqs, tail_sampling/rest, batch Everything else, at TRAILBEAM_BACKEND_SAMPLING_PERCENTAGE percent (default 10)

A span counts as an SQS span when any of these is true:

text
instrumentation_scope.name == "trailbeam"
attributes["rpc.service"] == "SQS"
attributes["rpc.system"] == "aws-api" and attributes["rpc.service"] == "Sqs"
IsMatch(attributes["messaging.system"], "^(aws.sqs|aws_sqs|AmazonSQS)$")
attributes["messaging.system.name"] == "aws.sqs"

filter/drop_sqs uses exactly the same expression, negated, so each span goes down one pipeline only. tail_sampling/rest uses one probabilistic policy with decision_wait: 5s and num_traces: 100000.

With this setup at 1% sampling of everything else, the Collector kept 467 of 467 SQS spans from Python, 231 of 231 from Node.js and 354 of 354 from Java.

5Step 5: Verify nothing is dropped#

Trailbeam's health endpoint counts what it received since it started:

shell
curl -s http://127.0.0.1:4318/api/health | jq .ingest
json
{"requests": 12, "inserted": 340, "duplicates": 0, "ignored": 55, "rejected": 0}
Counter Meaning If it is not what you expect
inserted Trailbeam spans stored Should grow with traffic
duplicates Spans already stored (same trace and span ID), for example an exporter retry Harmless. They are skipped.
ignored Spans with another instrumentation scope Should be 0 behind trailbeam-collector.yaml. Non-zero means something sends to Trailbeam directly.
rejected Trailbeam spans that break the contract Check Trailbeam log: each request logs N Trailbeam span(s) rejected: with the reasons. The exporter also gets them in the response's partial_success. The rules are in contract section 4.5.

Then pick a busy minute and check timelines for Deliveries not observed and send_not_observed. On a fully instrumented queue, both should be absent.

Prevent it#

  • Send the packages' spans to trailbeam-collector.yaml, or straight to Trailbeam (it listens on 4318 by default). Never through a pipeline with a sampler.
  • If you run your own Collector, give Trailbeam spans their own pipeline with only a filter on instrumentation_scope.name == "trailbeam" and a batch processor, as traces/trailbeam does.
  • Set TRAILBEAM_OTLP_TRACES_ENDPOINT in every instrumented service. A service that exports to an endpoint nobody listens on looks exactly like an uninstrumented one.
  • Flush before exit. Spans still in the batch processor are lost when a process exits without flushing:
python
trailbeam.boto.uninstall()      # flushes and shuts down the Trailbeam provider
javascript
await trailbeam.uninstall();    // or await trailbeam.forceFlush()