Configuration
Every setting for the Python and Node.js packages, the Lambda wrappers, the Collector and Trailbeam, with type, default and effect.
The packages read their settings when install, instrument, wrap or wrapHandler creates the tracer provider; the Lambda wrappers read AWS_LAMBDA_LOG_STREAM_NAME on each invocation. Trailbeam reads its settings once, at start.
Environment variables read by both packages#
These apply to the client hooks and the Lambda wrappers, in Python and Node.js.
| Variable | Type | Default | Effect |
|---|---|---|---|
TRAILBEAM_RECEIPT_HASH_KEY |
string (UTF-8) | trailbeam-default-receipt-hash-key, with one warning per process |
HMAC-SHA256 key for trailbeam.receipt_handle.hmac. Must be identical on every service that receives or settles messages on the same queues, or deletes cannot be joined to receives. An empty value counts as unset. A key shorter than 16 bytes is used, with one warning. |
TRAILBEAM_OTLP_TRACES_ENDPOINT |
URL | http://localhost:4318/v1/traces |
Full OTLP/HTTP (protobuf) URL, including /v1/traces. Point it at Trailbeam or at the Collector. |
OTEL_SERVICE_NAME |
string | unknown_service |
service.name on the resource. Shown as the service on each step. |
OTEL_RESOURCE_ATTRIBUTES |
k=v,... |
none | service.instance.id here becomes messaging.client.id (client hooks). Without it, the package generates a random UUID v4 each time it creates a tracer provider: on each install call, and once for each wrapped Lambda handler. |
AWS_LAMBDA_LOG_STREAM_NAME |
string | set by Lambda | Lambda wrappers only: used as messaging.client.id (one execution environment). Falls back to service.instance.id. |
AWS_LAMBDA_FUNCTION_NAME |
string | set by Lambda | Lambda wrappers only: faas.name when the context has no function name. |
Python package (trailbeam)#
trailbeam.boto.install()#
import trailbeam.boto
provider = trailbeam.boto.install(receipt_hash_key=None, span_exporter=None, span_processor=None)| Option | Type | Default | Effect |
|---|---|---|---|
receipt_hash_key |
str or None |
None (then TRAILBEAM_RECEIPT_HASH_KEY, then the default key) |
Receipt-hash key. Takes precedence over the environment variable. Positional or keyword. |
span_exporter |
SpanExporter |
OTLP/HTTP exporter to TRAILBEAM_OTLP_TRACES_ENDPOINT |
Replaces the exporter. Wrapped in a BatchSpanProcessor. Keyword only. |
span_processor |
SpanProcessor |
BatchSpanProcessor(span_exporter) |
Replaces the processor and exporter. Keyword only. |
Returns the Trailbeam TracerProvider (always-on sampler, not registered globally). Calling install again replaces the previous installation. trailbeam.boto.uninstall() removes the hooks, flushes and shuts the provider down. If the application already called BotocoreInstrumentor().instrument(...), Trailbeam keeps that setup and its tracer provider, and runs the Trailbeam hooks first, then the application's. Otherwise it instruments botocore with the Trailbeam provider, so the stock botocore spans are exported to TRAILBEAM_OTLP_TRACES_ENDPOINT too; Trailbeam counts them as ignored.
trailbeam.aws_lambda.wrap#
@trailbeam.aws_lambda.wrap(receipt_hash_key=None, flush_timeout_millis=2000, report_batch_item_failures=True)
def handler(event, context): ...| Option | Type | Default | Effect |
|---|---|---|---|
receipt_hash_key |
str or None |
None |
As for install. |
span_exporter |
SpanExporter |
OTLP/HTTP exporter | As for install. |
span_processor |
SpanProcessor |
BatchSpanProcessor |
As for install. |
flush_timeout_millis |
int |
2000 |
Longest wait for each of the two flushes per invocation (after the receives, after the outcomes). |
report_batch_item_failures |
bool |
True |
Must match ReportBatchItemFailures on the event source mapping. When False, batchItemFailures in the return value is ignored, as Lambda ignores it. |
The wrapped function has a trailbeam_tracer_provider attribute. Bare @trailbeam.aws_lambda.wrap uses every default.
Node.js package (@trailbeam/sdk)#
install(options) and instrument(client)#
const trailbeam = require('@trailbeam/sdk');
trailbeam.install({ receiptHashKey: process.env.TRAILBEAM_RECEIPT_HASH_KEY });
const sqs = trailbeam.instrument(new SQSClient({}));| Option | Type | Default | Effect |
|---|---|---|---|
receiptHashKey |
string |
undefined (then TRAILBEAM_RECEIPT_HASH_KEY, then the default key) |
Receipt-hash key. An empty string counts as unset. |
spanExporter |
SpanExporter |
OTLPTraceExporter (protobuf) to TRAILBEAM_OTLP_TRACES_ENDPOINT |
Replaces the exporter, wrapped in a BatchSpanProcessor. |
spanProcessor |
SpanProcessor |
BatchSpanProcessor(spanExporter) |
Replaces the processor and exporter. |
install returns the Trailbeam BasicTracerProvider; calling it again replaces the installation and shuts the old provider down. All its options are optional. instrument adds the middleware trailbeamSqsMiddleware (step initialize, priority high) to one AWS SDK v3 client and returns the client. It calls install() with defaults if needed, is safe to call twice on the same client, and throws TypeError if the argument has no middleware stack. Every client you want recorded must be passed to instrument. forceFlush() flushes; uninstall() turns the middleware off on every client, flushes and shuts down. Both return a promise. The package also exports hashReceiptHandle(key, receiptHandle) and keyId(key), the two hash values of the span contract, plus version and SCHEMA_VERSION (3). Types are in node/index.d.ts.
wrapHandler(handler, options)#
| Option | Type | Default | Effect |
|---|---|---|---|
receiptHashKey |
string |
undefined |
As for install. |
spanExporter |
SpanExporter |
OTLP/HTTP exporter | As for install. |
spanProcessor |
SpanProcessor |
BatchSpanProcessor |
As for install. |
flushTimeoutMs |
number |
2000 |
Longest wait for each flush. A flush that takes longer is abandoned, not failed. |
reportBatchItemFailures |
boolean |
true |
Must match ReportBatchItemFailures on the event source mapping. |
The returned handler is always async (event, context) and has a trailbeamTracerProvider property. Async, sync and callback-style handlers are accepted. wrapHandler throws TypeError if handler is not a function.
Fixed values in the packages#
Not configurable in 0.1. Listed so you know the bounds.
| Value | Python | Node.js | Effect |
|---|---|---|---|
| Receipt handle to message ID map | 100,000 entries, 12 h | 100,000 entries, 12 h | Oldest entries are dropped first. A settle whose handle is not in the map carries only the hash; Trailbeam still joins it. |
| Pending requests | 10,000 | none | Python clears its map of in-flight calls if it reaches 10,000 (only when responses were lost). |
| Minimum key length | 16 bytes | 16 bytes | Shorter keys are used, with a warning. |
| Clock | time.time_ns() (nanoseconds) |
Date.now() (whole milliseconds, rounded down) |
Wall clock for span start and end. |
thread.name |
set | not set | Python steps name the thread; Node.js steps do not. |
Store (trailbeam-store)#
Each option has an environment variable. The command-line flag wins when both are set.
| Option | Environment | Type | Default | Effect |
|---|---|---|---|---|
--db |
TRAILBEAM_STORE_DB |
path | trailbeam.sqlite |
SQLite file, WAL mode. Created if missing; older schemas are migrated on start. |
--host |
TRAILBEAM_STORE_HOST |
address | 127.0.0.1 |
Listen address. Use 0.0.0.0 in a container. There is no authentication. |
--port |
TRAILBEAM_STORE_PORT |
int | 4318 |
One port for OTLP ingest (/v1/traces), the JSON API (/api/) and the web screen. Use 4319 next to a Collector. |
--stuck-after |
TRAILBEAM_STUCK_AFTER_S |
int, seconds | 300 |
Default older_than_s for /api/stuck and /stuck. |
--lambda-timeout |
TRAILBEAM_LAMBDA_TIMEOUT_S |
int, seconds | 900 |
How long a Lambda receive may go without an outcome before lambda_invocation_ended_without_outcome is inferred. Set it to your function's timeout. Overridable per request with lambda_timeout_s. |
--max-receive-count |
TRAILBEAM_MAX_RECEIVE_COUNT |
QUEUE=N[,QUEUE=N...], N from 1 to 1000 |
none | Each queue's redrive maxReceiveCount. Enables dead_letter_queue_move: "not_seen_on_dlq_yet" and the dlq_move_not_seen notice. Spans do not carry it. Shown in /api/health as config.max_receive_counts. |
--log-level |
TRAILBEAM_STORE_LOG_LEVEL |
Python log level | INFO |
WARNING still shows rejected spans; DEBUG logs every HTTP request. |
trailbeam-store --db /var/lib/trailbeam/trailbeam.sqlite --port 4319 \
--lambda-timeout 60 --max-receive-count orders=5,payments=3A malformed --max-receive-count stops Trailbeam at start with expected QUEUE=N with N from 1 to 1000, got '...'.
Container image#
The image in server/Dockerfile sets TRAILBEAM_STORE_DB=/data/trailbeam.sqlite, TRAILBEAM_STORE_HOST=0.0.0.0 and TRAILBEAM_STORE_PORT=4318, runs as UID 10001, declares the volume /data, and has a HEALTHCHECK on /api/health every 30 s. Set any option with -e:
docker run -d --name trailbeam -p 127.0.0.1:4318:4318 -v trailbeam-data:/data \
-e TRAILBEAM_LAMBDA_TIMEOUT_S=60 -e TRAILBEAM_MAX_RECEIVE_COUNT=orders=5 trailbeam-storeCollector#
Environment variables used by collector/trailbeam-collector.yaml and collector/backend.yaml (otelcol-contrib required).
| Variable | Type | Default | Effect |
|---|---|---|---|
TRAILBEAM_COLLECTOR_HTTP_ENDPOINT |
host:port |
localhost:4318 |
OTLP/HTTP listen address. Use 0.0.0.0:4318 in a container. |
TRAILBEAM_COLLECTOR_GRPC_ENDPOINT |
host:port |
localhost:4317 |
OTLP/gRPC listen address. |
TRAILBEAM_STORE_ENDPOINT |
URL | http://localhost:4319 |
Store base URL. The exporter appends /v1/traces. |
TRAILBEAM_BACKEND_ENDPOINT |
URL | none, required by backend.yaml |
Base OTLP/HTTP URL of your tracing backend. |
TRAILBEAM_BACKEND_SAMPLING_PERCENTAGE |
number | 10 |
Share of non-SQS traces kept for the backend. SQS spans are always kept. |
Fixed in the YAML (edit the file to change them):
| Setting | Value | Effect |
|---|---|---|
filter/trailbeam_only |
drops instrumentation_scope.name != "trailbeam" |
Trailbeam only receives Trailbeam spans. |
otlp_http/trailbeam_store.compression |
gzip |
Trailbeam accepts gzip. |
retry_on_failure.max_elapsed_time |
0s |
Retry forever while Trailbeam is down. |
sending_queue.queue_size |
5000 batches, in memory |
When full, new spans are dropped. Lost on Collector restart. |
tail_sampling/rest |
decision_wait: 5s, num_traces: 100000 |
Backend overlay only: tail sampling of non-SQS spans. |
Demo#
| Variable | Default | Effect |
|---|---|---|
TRAILBEAM_DEMO_PORT |
4318 |
Host port the demo's Trailbeam is published on (127.0.0.1 only). |
COMPOSE_PROJECT_NAME |
trailbeam-demo |
Compose project and image name prefix. Change it to run a second checkout's demo. |
TRAILBEAM_RECEIPT_HASH_KEY |
empty (default key) | Passed to the demo services. |
Related#
Lambda wrapper
trailbeam.aws_lambda.wrap (Python) or trailbeam.wrapHandler (Node.js), around a handler invoked by an event source mapping.Hooks
trailbeam.boto.install()), the AWS SDK v3 middleware (trailbeam.instrument(client)) and the Lambda wrapper.Receive
ReceiveMessage call returning a message. A receive does not remove the message; it starts its visibility timeout.Amazon SQS terms in Vocabulary
Message
Amazon SQS terms in Vocabulary
Queue
Amazon SQS terms in Vocabulary
OTLP
OpenTelemetry terms in Vocabulary
Collector
OpenTelemetry terms in Vocabulary
Step
Handler
Amazon SQS terms in Vocabulary
Timeline
Span
OpenTelemetry terms in Vocabulary
Receipt-hash key
TRAILBEAM_RECEIPT_HASH_KEY. Every service on the same queues must use the same key.Sampling
OpenTelemetry terms in Vocabulary
Batch item failures
ReportBatchItemFailures turned on, your handler can return the IDs of the records that failed. Lambda deletes the rest and leaves those to become visible again. Without it, any error fails the whole batch.AWS Lambda terms in Vocabulary
Event source mapping
AWS Lambda terms in Vocabulary
Receipt handle
Amazon SQS terms in Vocabulary
Message ID
MessageId, assigned by SQS when the message is sent. It stays the same on every receive, including a receive from a dead-letter queue. Trailbeam joins everything about a message by this ID.Amazon SQS terms in Vocabulary
In flight
Amazon SQS terms in Vocabulary
Inferred
inferred: true. See Recorded and inferred steps.Function timeout
Task timed out after N seconds.AWS Lambda terms in Vocabulary
maxReceiveCount
Amazon SQS terms in Vocabulary
Trailbeam span
trailbeam. Trailbeam reads only these.