Skip to content
Docs

OpenTelemetry Collector

A ready-made Collector configuration that passes every Trailbeam span to Trailbeam, and optionally to your tracing backend.

Based on 9 repo files at 54ea61b

collector/README.md, collector/trailbeam-collector.yaml, collector/backend.yaml, collector/tests/test_collector.py, demo/compose.yaml, demo/collector-image/Dockerfile, .github/workflows/ci.yml, server/README.md, server/src/trailbeam_store/server.py

On this page

The Collector is optional. The packages can export straight to Trailbeam. Use it when you already run a Collector, or when you also want SQS spans in your own tracing backend.

Install#

It needs the contrib distribution, otelcol-contrib, because the configs use the filter and tail_sampling processors. The configs are tested with 0.165.0 (CI downloads that release; the demo uses the image otel/opentelemetry-collector-contrib:0.165.0).

Two files in collector/:

File What it does
trailbeam-collector.yaml Receives OTLP. Sends every span with instrumentation scope trailbeam to Trailbeam, unsampled. Sends nothing else to Trailbeam.
backend.yaml Optional overlay. Also sends traces to your backend: every SQS span (Trailbeam and stock AWS SDK) unsampled, everything else tail-sampled.

Run it#

Trailbeam only:

shell
TRAILBEAM_STORE_ENDPOINT=http://localhost:4319 otelcol-contrib --config trailbeam-collector.yaml

Trailbeam plus your backend:

shell
TRAILBEAM_BACKEND_ENDPOINT=http://jaeger:4318 \
  otelcol-contrib --config trailbeam-collector.yaml --config backend.yaml

Point the packages at the Collector. The default already matches when it runs on the same host:

shell
TRAILBEAM_OTLP_TRACES_ENDPOINT=http://localhost:4318/v1/traces

In Docker#

Listen on all interfaces inside a container. The demo builds a small image from demo/collector-image/Dockerfile:

text
FROM otel/opentelemetry-collector-contrib:0.165.0
COPY collector/trailbeam-collector.yaml /etc/trailbeam/trailbeam-collector.yaml
CMD ["--config", "/etc/trailbeam/trailbeam-collector.yaml"]

and sets its environment in demo/compose.yaml:

yaml
collector:
  build: { context: .., dockerfile: demo/collector-image/Dockerfile }
  environment:
    TRAILBEAM_COLLECTOR_HTTP_ENDPOINT: 0.0.0.0:4318
    TRAILBEAM_COLLECTOR_GRPC_ENDPOINT: 0.0.0.0:4317
    TRAILBEAM_STORE_ENDPOINT: http://trailbeam:4318
  depends_on: [trailbeam]

Trailbeam runs in its own container (trailbeam) on 4318, so there is no port clash. Services export to http://collector:4318/v1/traces.

Options#

All settings are environment variables read by the YAML with ${env:NAME:-default}.

Option Type Default What it does
TRAILBEAM_COLLECTOR_HTTP_ENDPOINT host:port localhost:4318 OTLP/HTTP listen address. Use 0.0.0.0:4318 in a container.
TRAILBEAM_COLLECTOR_GRPC_ENDPOINT host:port localhost:4317 OTLP/gRPC listen address.
TRAILBEAM_STORE_ENDPOINT URL http://localhost:4319 Base URL of Trailbeam. The exporter appends /v1/traces.
TRAILBEAM_BACKEND_ENDPOINT URL none, required by backend.yaml Base OTLP/HTTP URL of your tracing backend.
TRAILBEAM_BACKEND_SAMPLING_PERCENTAGE number 10 Percentage of non-SQS traces kept for the backend (probabilistic tail sampling).

How the pipelines work#

trailbeam-collector.yaml#

text
otlp receiver ─▶ filter/trailbeam_only ─▶ batch ─▶ otlp_http/trailbeam_store
  • filter/trailbeam_only drops every span where instrumentation_scope.name != "trailbeam". Trailbeam only reads trailbeam spans, so nothing else is sent.
  • otlp_http/trailbeam_store exports with compression: gzip. Trailbeam accepts Content-Encoding: gzip.
  • retry_on_failure is on with max_elapsed_time: 0s: it retries for as long as Trailbeam is down.
  • sending_queue is in memory with queue_size: 5000 batches.

There is no sampling step. The packages already use an always-on tracer provider, so your app's head sampling never drops Trailbeam spans, and this pipeline keeps them too.

backend.yaml overlay#

It adds two pipelines from the same otlp receiver:

Pipeline Processors Exporter Result
traces/backend_sqs filter/sqs_only, batch otlp_http/backend Every SQS span, unsampled
traces/backend_rest filter/drop_sqs, tail_sampling/rest, batch otlp_http/backend Everything else, sampled at TRAILBEAM_BACKEND_SAMPLING_PERCENTAGE

A span counts as SQS when any of these is true:

  • instrumentation_scope.name == "trailbeam"
  • attributes["rpc.service"] == "SQS"
  • attributes["rpc.system"] == "aws-api" and attributes["rpc.service"] == "Sqs"
  • attributes["messaging.system"] matches ^(aws.sqs|aws_sqs|AmazonSQS)$
  • attributes["messaging.system.name"] == "aws.sqs"

filter/drop_sqs uses the exact same expression, inverted, so each span goes to exactly one backend pipeline. tail_sampling/rest uses decision_wait: 5s, num_traces: 100000 and one probabilistic policy.

The trade-off: in your backend, a sampled-out trace still shows its SQS spans, without their parent HTTP or database spans. Trailbeam pipeline is unchanged by the overlay. See Keep every SQS span when you sample.

What it emits#

The Collector creates no spans. It forwards the five Trailbeam span kinds unchanged: trailbeam send, trailbeam receive, trailbeam delete, trailbeam change_visibility, trailbeam outcome. See the span contract.

Verify it works#

  1. Start Trailbeam on 4319 and the Collector with trailbeam-collector.yaml.
  2. Run an instrumented service with TRAILBEAM_OTLP_TRACES_ENDPOINT=http://localhost:4318/v1/traces.
  3. Check Trailbeam's counters:
shell
curl -s http://127.0.0.1:4319/api/health
json
{"status": "ok", "version": "0.1.0.dev0",
 "ingest": {"requests": 4, "inserted": 120, "duplicates": 0, "ignored": 0, "rejected": 0},
 "store": {"spans": {"send": 40, "receive": 40, "delete": 40}, "messages": 40, "unjoined_settles": 0},
 "config": {"stuck_after_s": 300, "lambda_timeout_s": 900, "max_receive_counts": {}}}

ingest.ignored stays at 0 behind the Collector, because the filter already removed non-Trailbeam spans. If it grows, something is exporting to Trailbeam directly.

The repo's test, collector/tests/test_collector.py, runs the real binary between two fake OTLP endpoints and checks that Trailbeam gets exactly the Trailbeam spans, and that with backend.yaml at 0% sampling the backend still gets every Trailbeam and stock SQS span:

shell
pip install pytest opentelemetry-proto
TRAILBEAM_OTELCOL_BIN=/path/to/otelcol-contrib pytest collector/tests

Without a binary the tests are skipped.

Common errors#

What you see Cause Fix
Collector rejects the config: the filter or tail_sampling processor is unknown Core otelcol distribution, which lacks them Use otelcol-contrib.
Collector or Trailbeam cannot bind 4318 Both on one host with default ports Run Trailbeam with --port 4319.
Trailbeam logs nothing, services get connection refused Collector listens on localhost inside a container Set TRAILBEAM_COLLECTOR_HTTP_ENDPOINT=0.0.0.0:4318.
Store answers 404 not found to the Collector TRAILBEAM_STORE_ENDPOINT already ends in /v1/traces, so the path is doubled Use the base URL, e.g. http://trailbeam:4318.
Backend exporter has no endpoint TRAILBEAM_BACKEND_ENDPOINT not set; it has no default Set it to your backend's base OTLP/HTTP URL.
Steps missing after Trailbeam was down Sending queue filled up; spans dropped Keep Trailbeam up; size outages against 5,000 batches.
Backend traces show SQS spans with no parent Expected trade-off of the split Raise TRAILBEAM_BACKEND_SAMPLING_PERCENTAGE if you need more full traces.