Add Trailbeam to your services
Which package goes in which service, the minimal code for each runtime, and how to confirm spans reach Trailbeam.
Add the hooks to every service that sends, receives, deletes or changes visibility on the queues you care about. A service without them leaves gaps: missing sends, deliveries_not_observed steps, or unmatched deletes.
What goes where#
| Your service | Package | Code | Setup page |
|---|---|---|---|
| Python 3.10+, boto3 (producer, worker, script) | trailbeam |
trailbeam.boto.install() |
Python |
Node.js 20+, AWS SDK v3 SQSClient |
@trailbeam/sdk |
trailbeam.instrument(new SQSClient({})) |
Node.js |
| Lambda with an SQS event source, Python | trailbeam |
@trailbeam.aws_lambda.wrap |
Lambda |
| Lambda with an SQS event source, Node.js | @trailbeam/sdk |
trailbeam.wrapHandler(handler) |
Lambda |
| Trailbeam (one per environment) | server/ image |
docker run ... trailbeam-store |
Run Trailbeam |
| An OpenTelemetry Collector (optional) | collector/trailbeam-collector.yaml |
otelcol-contrib --config ... |
Collector |
A Lambda function that also calls SQS itself (for example, sends to another queue) needs both the wrapper and the client hooks.
1Step 1: Run Trailbeam#
docker build -t trailbeam-store server
docker run -d --name trailbeam -p 127.0.0.1:4318:4318 -v trailbeam-data:/data trailbeam-storeIt receives OTLP on /v1/traces and serves the screen and API on the same port. There is no authentication in 0.1; keep it on 127.0.0.1 or a trusted network.
2Step 2: Add the hooks#
Python (boto3)#
pip install ./trailbeam/python # from your application's environmentimport boto3
import trailbeam.boto
trailbeam.boto.install() # once per process, before or after creating clients
sqs = boto3.client("sqs")The hooks make no SQS calls of their own. On ReceiveMessage they add the system attributes Trailbeam needs to your request, and keep every name you asked for.
Node.js (AWS SDK v3)#
npm install ./trailbeam/nodeconst { SQSClient } = require('@aws-sdk/client-sqs');
const trailbeam = require('@trailbeam/sdk');
const sqs = trailbeam.instrument(new SQSClient({})); // installs with defaults on first useInstrument every SQSClient you create. In a short-lived script, call await trailbeam.forceFlush() before exit. Spans still waiting to be exported when a process exits are lost, in either language.
Lambda#
import trailbeam.aws_lambda
@trailbeam.aws_lambda.wrap
def handler(event, context):
...const trailbeam = require('@trailbeam/sdk');
exports.handler = trailbeam.wrapHandler(async (event, context) => {
return { batchItemFailures: [] };
});If the event source mapping does not have ReportBatchItemFailures on, pass report_batch_item_failures=False (Python) or { reportBatchItemFailures: false } (Node.js).
3Step 3: Set the environment#
| Variable | Set it to | Default |
|---|---|---|
TRAILBEAM_RECEIPT_HASH_KEY |
One secret, the same on every service on these queues | the published default key, with a warning |
TRAILBEAM_OTLP_TRACES_ENDPOINT |
Trailbeam or Collector, with the /v1/traces path |
http://localhost:4318/v1/traces |
OTEL_SERVICE_NAME |
A name per service, shown on every step | unknown_service |
export TRAILBEAM_RECEIPT_HASH_KEY=$(openssl rand -hex 32) # once; store as a secret
export TRAILBEAM_OTLP_TRACES_ENDPOINT=http://trailbeam-store:4318/v1/traces
export OTEL_SERVICE_NAME=orders-workerSend straight to Trailbeam for one host or a small setup. Go through the Collector if you already run one or also want traces in your own backend. See Keep every SQS span when you sample.
4Step 4: Confirm spans arrive#
Send and process one message, then:
curl -s http://127.0.0.1:4318/api/health{
"status": "ok", "version": "0.1.0.dev0",
"ingest": {"requests": 4, "inserted": 3, "duplicates": 0, "ignored": 3, "rejected": 0},
"store": {"spans": {"send": 1, "receive": 1, "delete": 1}, "messages": 1, "unjoined_settles": 0},
"config": {"stuck_after_s": 300, "lambda_timeout_s": 900}
}| Check | Healthy | If not |
|---|---|---|
ingest.requests |
above 0 | Spans are not reaching Trailbeam: check TRAILBEAM_OTLP_TRACES_ENDPOINT and the network |
store.spans |
a count for each kind each service emits. A kind with no stored spans is left out, not shown as 0 |
The service with a missing kind is not instrumented |
ingest.rejected |
0 |
Trailbeam logs N Trailbeam span(s) rejected: <reasons> |
store.unjoined_settles |
0, or briefly above while spans are in flight |
Different hash keys, or the receiver is not instrumented |
ingest.ignored |
any | Non-Trailbeam spans (such as stock botocore spans). Harmless |
Then open http://127.0.0.1:4318/, pick the message, and check that it has Sent, Received and Delete steps and no default_key notice.
Related#
Hooks
trailbeam.boto.install()), the AWS SDK v3 middleware (trailbeam.instrument(client)) and the Lambda wrapper.Queue
Amazon SQS terms in Vocabulary
Producer
SendMessage or SendMessageBatch).Amazon SQS terms in Vocabulary
Collector
OpenTelemetry terms in Vocabulary
OTLP
OpenTelemetry terms in Vocabulary
Span
OpenTelemetry terms in Vocabulary
Event source mapping
AWS Lambda terms in Vocabulary
Batch item failures
ReportBatchItemFailures turned on, your handler can return the IDs of the records that failed. Lambda deletes the rest and leaves those to become visible again. Without it, any error fails the whole batch.AWS Lambda terms in Vocabulary
Step
Timeline
Message
Amazon SQS terms in Vocabulary
In flight
Amazon SQS terms in Vocabulary
Trailbeam span
trailbeam. Trailbeam reads only these.