Skip to content

Add Trailbeam to your services

Which package goes in which service, the minimal code for each runtime, and how to confirm spans reach Trailbeam.

Based on 13 repo files at 54ea61b

README.md, python/README.md, node/README.md, server/README.md, collector/README.md, docs/store-api.md, python/src/trailbeam/boto.py, python/src/trailbeam/aws_lambda.py, node/src/sqs.js, node/src/lambda.js, node/index.d.ts, python/pyproject.toml, node/package.json

On this page

Add the hooks to every service that sends, receives, deletes or changes visibility on the queues you care about. A service without them leaves gaps: missing sends, deliveries_not_observed steps, or unmatched deletes.

What goes where#

Your service Package Code Setup page
Python 3.10+, boto3 (producer, worker, script) trailbeam trailbeam.boto.install() Python
Node.js 20+, AWS SDK v3 SQSClient @trailbeam/sdk trailbeam.instrument(new SQSClient({})) Node.js
Lambda with an SQS event source, Python trailbeam @trailbeam.aws_lambda.wrap Lambda
Lambda with an SQS event source, Node.js @trailbeam/sdk trailbeam.wrapHandler(handler) Lambda
Trailbeam (one per environment) server/ image docker run ... trailbeam-store Run Trailbeam
An OpenTelemetry Collector (optional) collector/trailbeam-collector.yaml otelcol-contrib --config ... Collector

A Lambda function that also calls SQS itself (for example, sends to another queue) needs both the wrapper and the client hooks.

1Step 1: Run Trailbeam#

shell
docker build -t trailbeam-store server
docker run -d --name trailbeam -p 127.0.0.1:4318:4318 -v trailbeam-data:/data trailbeam-store

It receives OTLP on /v1/traces and serves the screen and API on the same port. There is no authentication in 0.1; keep it on 127.0.0.1 or a trusted network.

2Step 2: Add the hooks#

Python (boto3)#

shell
pip install ./trailbeam/python   # from your application's environment
python
import boto3
import trailbeam.boto

trailbeam.boto.install()            # once per process, before or after creating clients
sqs = boto3.client("sqs")

The hooks make no SQS calls of their own. On ReceiveMessage they add the system attributes Trailbeam needs to your request, and keep every name you asked for.

Node.js (AWS SDK v3)#

shell
npm install ./trailbeam/node
javascript
const { SQSClient } = require('@aws-sdk/client-sqs');
const trailbeam = require('@trailbeam/sdk');

const sqs = trailbeam.instrument(new SQSClient({})); // installs with defaults on first use

Instrument every SQSClient you create. In a short-lived script, call await trailbeam.forceFlush() before exit. Spans still waiting to be exported when a process exits are lost, in either language.

Lambda#

python
import trailbeam.aws_lambda

@trailbeam.aws_lambda.wrap
def handler(event, context):
    ...
javascript
const trailbeam = require('@trailbeam/sdk');
exports.handler = trailbeam.wrapHandler(async (event, context) => {
  return { batchItemFailures: [] };
});

If the event source mapping does not have ReportBatchItemFailures on, pass report_batch_item_failures=False (Python) or { reportBatchItemFailures: false } (Node.js).

3Step 3: Set the environment#

Variable Set it to Default
TRAILBEAM_RECEIPT_HASH_KEY One secret, the same on every service on these queues the published default key, with a warning
TRAILBEAM_OTLP_TRACES_ENDPOINT Trailbeam or Collector, with the /v1/traces path http://localhost:4318/v1/traces
OTEL_SERVICE_NAME A name per service, shown on every step unknown_service
shell
export TRAILBEAM_RECEIPT_HASH_KEY=$(openssl rand -hex 32)          # once; store as a secret
export TRAILBEAM_OTLP_TRACES_ENDPOINT=http://trailbeam-store:4318/v1/traces
export OTEL_SERVICE_NAME=orders-worker

Send straight to Trailbeam for one host or a small setup. Go through the Collector if you already run one or also want traces in your own backend. See Keep every SQS span when you sample.

4Step 4: Confirm spans arrive#

Send and process one message, then:

shell
curl -s http://127.0.0.1:4318/api/health
json
{
  "status": "ok", "version": "0.1.0.dev0",
  "ingest": {"requests": 4, "inserted": 3, "duplicates": 0, "ignored": 3, "rejected": 0},
  "store": {"spans": {"send": 1, "receive": 1, "delete": 1}, "messages": 1, "unjoined_settles": 0},
  "config": {"stuck_after_s": 300, "lambda_timeout_s": 900}
}
Check Healthy If not
ingest.requests above 0 Spans are not reaching Trailbeam: check TRAILBEAM_OTLP_TRACES_ENDPOINT and the network
store.spans a count for each kind each service emits. A kind with no stored spans is left out, not shown as 0 The service with a missing kind is not instrumented
ingest.rejected 0 Trailbeam logs N Trailbeam span(s) rejected: <reasons>
store.unjoined_settles 0, or briefly above while spans are in flight Different hash keys, or the receiver is not instrumented
ingest.ignored any Non-Trailbeam spans (such as stock botocore spans). Harmless

Then open http://127.0.0.1:4318/, pick the message, and check that it has Sent, Received and Delete steps and no default_key notice.