Node.js (AWS SDK v3)
Add Trailbeam hooks to AWS SDK v3 SQS clients, what they record, and how to check they work.
Install#
The package is @trailbeam/sdk (version 0.1.0-dev.0). It is marked "private": true and not on npm. Install it from a clone of the repository, which is private for now:
gh repo clone nick-lennox/trailbeam
npm install ./trailbeam/node # run from your application's directoryOr run npm pack ./trailbeam/node and install the tarball. It is plain CommonJS with type declarations (index.d.ts); there is no build step.
Requirements: Node.js 20 or later, and @aws-sdk/client-sqs in your app. Runtime dependencies: @opentelemetry/api ~1.9.0, @opentelemetry/exporter-trace-otlp-proto ^0.223.2, @opentelemetry/resources and @opentelemetry/sdk-trace-base ^2.12.1.
Add the hooks#
Instrument every SQSClient you create. install() is optional; instrument() installs with defaults on first use.
const { SQSClient, SendMessageCommand, ReceiveMessageCommand, DeleteMessageCommand } = require('@aws-sdk/client-sqs');
const trailbeam = require('@trailbeam/sdk');
trailbeam.install({ receiptHashKey: process.env.TRAILBEAM_RECEIPT_HASH_KEY }); // optional
const sqs = trailbeam.instrument(new SQSClient({}));
const QueueUrl = process.env.QUEUE_URL;
await sqs.send(new SendMessageCommand({ QueueUrl, MessageBody: 'hello' }));
const { Messages = [] } = await sqs.send(new ReceiveMessageCommand({ QueueUrl, WaitTimeSeconds: 5 }));
for (const m of Messages) {
await sqs.send(new DeleteMessageCommand({ QueueUrl, ReceiptHandle: m.ReceiptHandle }));
}TypeScript:
import { SQSClient } from '@aws-sdk/client-sqs';
import * as trailbeam from '@trailbeam/sdk';
const sqs = trailbeam.instrument(new SQSClient({}));For Lambda handlers use the wrapper instead. See AWS Lambda.
How the hooks attach#
instrument(client) adds one middleware, named trailbeamSqsMiddleware, to the client's middleware stack in the initialize step with priority: 'high' and override: true. Calling instrument twice on the same client is safe. A value without middlewareStack.add throws TypeError: trailbeam.instrument expects an AWS SDK v3 SQSClient.
The middleware looks up the current hooks on every call. After uninstall(), every instrumented client passes calls straight through.
- Trailbeam uses its own
BasicTracerProviderwithAlwaysOnSamplerand aBatchSpanProcessor. It does not register anything globally, and your sampling never drops Trailbeam spans. - Each Trailbeam span is a child of the active span. With
@opentelemetry/instrumentation-aws-sdkthat is the stock span for the same call, so both share a trace ID. - Parent spans are only found if the app has an OpenTelemetry context manager. Any
NodeSDKsetup registers one. - The hooks never change the result or error your code sees. Errors inside a hook go to
diag.debug.
Options#
install(options)#
Returns the Trailbeam BasicTracerProvider. Calling it again replaces the previous installation and shuts its provider down.
| Option | Type | Default | What it does |
|---|---|---|---|
receiptHashKey |
string |
undefined (then TRAILBEAM_RECEIPT_HASH_KEY, then the published default key) |
Key for the HMAC of receipt handles. An empty string is treated as unset. |
spanExporter |
SpanExporter |
OTLP/HTTP protobuf exporter to TRAILBEAM_OTLP_TRACES_ENDPOINT |
Replaces the exporter. It is wrapped in a BatchSpanProcessor. |
spanProcessor |
SpanProcessor |
undefined |
Replaces the processor and the exporter. Use SimpleSpanProcessor in tests. |
Other exports#
| Export | Signature | What it does |
|---|---|---|
instrument |
(client) => client |
Adds the middleware. Installs with defaults if needed. |
forceFlush |
() => Promise<void> |
Flushes client-hook spans waiting to be exported. |
uninstall |
() => Promise<void> |
Turns the hooks off on every client, then flushes and shuts down the provider. |
wrapHandler |
(handler, options?) |
Lambda wrapper. See AWS Lambda. |
hashReceiptHandle |
(key, receiptHandle) => string |
trailbeam.receipt_handle.hmac: first 32 hex chars of HMAC-SHA256. |
keyId |
(key) => string |
trailbeam.receipt_handle.key_id: first 8 hex chars of HMAC-SHA256 of "trailbeam key id". |
version, SCHEMA_VERSION |
constants | Package version; 3. |
Environment variables#
| Option | Type | Default | What it does |
|---|---|---|---|
TRAILBEAM_RECEIPT_HASH_KEY |
string | trailbeam-default-receipt-hash-key, with one warning |
Must be identical on every service using the same queues. Under 16 bytes: used, with one warning. |
TRAILBEAM_OTLP_TRACES_ENDPOINT |
URL | http://localhost:4318/v1/traces |
Full OTLP/HTTP (protobuf) traces URL: Trailbeam or a Collector. |
OTEL_SERVICE_NAME |
string | unknown_service |
Resource service.name. |
OTEL_RESOURCE_ATTRIBUTES |
k=v,... |
a random UUID v4 for service.instance.id, generated on each install() and wrapHandler() call |
service.instance.id becomes messaging.client.id. |
Warnings are emitted with process.emitWarning and code TRAILBEAM, once per process.
What it records#
| Command | Trailbeam span | Spans per call | Attributes added |
|---|---|---|---|
SendMessage |
trailbeam send |
1, if the output has MessageId |
messaging.message.id |
SendMessageBatch |
trailbeam send |
1 per entry in Successful |
messaging.message.id, trailbeam.batch_entry_id |
ReceiveMessage |
trailbeam receive |
1 per message | messaging.message.id, hash, trailbeam.receive_count, trailbeam.sent_timestamp_ms, trailbeam.first_receive_timestamp_ms, trailbeam.aws_trace_header, trailbeam.dead_letter_queue_source_arn |
DeleteMessage(Batch) |
trailbeam delete |
1 per receipt handle | hash, trailbeam.result, trailbeam.error_code, trailbeam.batch_entry_id (batch), messaging.message.id if known |
ChangeMessageVisibility(Batch) |
trailbeam change_visibility |
1 per receipt handle | as delete, plus trailbeam.visibility_timeout (0 is recorded) |
A send that throws gets no span. A settle command that throws still gets one failed span per receipt handle, and the error is rethrown. Missing system attributes are omitted, never sent as empty strings or -1. Other SQS commands produce no Trailbeam span.
Common attributes on every span: trailbeam.schema_version = 3, trailbeam.kind, trailbeam.source = client, messaging.system.name = aws.sqs, messaging.operation.type, messaging.operation.name, messaging.client.id, process.pid, messaging.destination.name and aws.sqs.queue.url. Node.js spans have no thread.name.
Receive request attributes#
The middleware asks ReceiveMessage for ApproximateReceiveCount, SentTimestamp, ApproximateFirstReceiveTimestamp, AWSTraceHeader and DeadLetterQueueSourceArn. It works on a copy of your input; your object is never modified. If you ask for All, nothing is added. If you use only the deprecated AttributeNames, the names go there. Message attributes are never requested.
Settle error codes#
trailbeam.error_code |
When |
|---|---|
SQS code, e.g. ReceiptHandleIsInvalid |
The command threw a service error. A namespace prefix is stripped: ElasticMQ's com.amazonaws.sqs#ReceiptHandleIsInvalid becomes ReceiptHandleIsInvalid. |
Error code, name or class name, e.g. TimeoutError |
The command threw a non-SQS error. Falls back to Error. |
Entry Code from Failed, or Failed |
One batch entry failed. |
MissingFromResponse |
A batch entry is in neither Successful nor Failed. |
Handle map, timing and order#
- Handle map: receipt handle to message ID, at most 100,000 entries, 12-hour expiry, entry dropped after a successful delete. A failed delete keeps the entry.
- Timing: the span starts when the middleware runs (before serialization, signing and retries, so it is a lower bound) and ends when the response or error arrives. Times come from
Date.now(), in milliseconds, rounded down. - Order:
trailbeam.order.issuedandtrailbeam.order.returnedare ticks of one counter per realm, shared by every instrumented client and every async task in it.trailbeam.order.scopenames it. Eachworker_threadis its own realm, with its own scope. See Ordering.
Short-lived processes#
Spans are exported in batches. Spans still waiting in the batch processor when the process exits are lost, so before a script exits:
await trailbeam.forceFlush(); // or: await trailbeam.uninstall();Verify it works#
In a test, export to memory:
const { InMemorySpanExporter, SimpleSpanProcessor } = require('@opentelemetry/sdk-trace-base');
const exporter = new InMemorySpanExporter();
trailbeam.install({ receiptHashKey: 'test-key-at-least-16-bytes', spanProcessor: new SimpleSpanProcessor(exporter) });
const sqs = trailbeam.instrument(new SQSClient({}));
// ... send, receive, delete ...
console.log(exporter.getFinishedSpans().map((s) => s.name));
// [ 'trailbeam send', 'trailbeam receive', 'trailbeam delete' ]With Trailbeam running:
curl -s http://127.0.0.1:4318/api/health
curl -s http://127.0.0.1:4318/api/messages/<message-id>A healthy flow, with a Python producer and a Node.js consumer on the same key:
- 12:00:01.204Sentorders-api · SendMessage
- 12:00:01.230Receivedbilling-node · receive count 1
- 12:00:01.388Deletebilling-node · ok · final
Common errors#
| What you see | Cause | Fix |
|---|---|---|
(node:<pid>) [TRAILBEAM] Warning: trailbeam: no receipt-hash key configured... |
No key | Set TRAILBEAM_RECEIPT_HASH_KEY or pass receiptHashKey. |
trailbeam: the receipt-hash key is shorter than 16 bytes |
Short key | Use 16 bytes or more. |
TypeError: trailbeam.instrument expects an AWS SDK v3 SQSClient |
Passed an SDK v2 client or a wrapper object | Pass the SQSClient instance. |
| Spans from one client but not another | Only one client was instrumented | Call instrument() on each SQSClient. |
| Trailbeam spans are root spans in your tracing backend | No context manager, or no AWS SDK instrumentation | Register a context manager (for example via NodeSDK). |
| Nothing in Trailbeam from a CLI or job | Process exited before the batch export | await trailbeam.forceFlush() before exit. |
| key_mismatch notice | Keys differ between services | Use one key everywhere. |
Related#
Handler
Amazon SQS terms in Vocabulary
Hooks
trailbeam.boto.install()), the AWS SDK v3 middleware (trailbeam.instrument(client)) and the Lambda wrapper.Sampling
OpenTelemetry terms in Vocabulary
Trailbeam span
trailbeam. Trailbeam reads only these.Span
OpenTelemetry terms in Vocabulary
Trace
OpenTelemetry terms in Vocabulary
Receipt handle
Amazon SQS terms in Vocabulary
OTLP
OpenTelemetry terms in Vocabulary
Lambda wrapper
trailbeam.aws_lambda.wrap (Python) or trailbeam.wrapHandler (Node.js), around a handler invoked by an event source mapping.Queue
Amazon SQS terms in Vocabulary
Collector
OpenTelemetry terms in Vocabulary
Timeline
Message
Amazon SQS terms in Vocabulary
Receive count
ApproximateReceiveCount: how many times SQS has returned the message. It goes up by one on every receive.Amazon SQS terms in Vocabulary
SentTimestamp
Amazon SQS terms in Vocabulary
ApproximateFirstReceiveTimestamp
Amazon SQS terms in Vocabulary
DeadLetterQueueSourceArn
Amazon SQS terms in Vocabulary
Message ID
MessageId, assigned by SQS when the message is sent. It stays the same on every receive, including a receive from a dead-letter queue. Trailbeam joins everything about a message by this ID.Amazon SQS terms in Vocabulary
Delete
DeleteMessage, made with a receipt handle. This is the only way a consumer removes a message. With the handle of an older receive, the call can still succeed while the message is received again.Amazon SQS terms in Vocabulary
Producer
SendMessage or SendMessageBatch).Amazon SQS terms in Vocabulary
Consumer
ReceiveMessage) and then deletes them. A Lambda function behind an event source mapping is a consumer too.