Skip to content

Configuration

Every setting for the Python and Node.js packages, the Lambda wrappers, the Collector and Trailbeam, with type, default and effect.

Based on 23 repo files at 54ea61b

python/src/trailbeam/_contract.py, python/src/trailbeam/_emitter.py, python/src/trailbeam/boto.py, python/src/trailbeam/aws_lambda.py, python/src/trailbeam/_handles.py, node/index.d.ts, node/src/contract.js, node/src/emitter.js, node/src/sqs.js, node/src/lambda.js, node/src/handles.js, server/src/trailbeam_store/__main__.py, server/src/trailbeam_store/server.py, server/Dockerfile, collector/trailbeam-collector.yaml, collector/backend.yaml, collector/README.md, demo/compose.yaml, demo/README.md, python/README.md, node/README.md, server/README.md, docs/store-api.md

On this page

The packages read their settings when install, instrument, wrap or wrapHandler creates the tracer provider; the Lambda wrappers read AWS_LAMBDA_LOG_STREAM_NAME on each invocation. Trailbeam reads its settings once, at start.

Environment variables read by both packages#

These apply to the client hooks and the Lambda wrappers, in Python and Node.js.

Variable Type Default Effect
TRAILBEAM_RECEIPT_HASH_KEY string (UTF-8) trailbeam-default-receipt-hash-key, with one warning per process HMAC-SHA256 key for trailbeam.receipt_handle.hmac. Must be identical on every service that receives or settles messages on the same queues, or deletes cannot be joined to receives. An empty value counts as unset. A key shorter than 16 bytes is used, with one warning.
TRAILBEAM_OTLP_TRACES_ENDPOINT URL http://localhost:4318/v1/traces Full OTLP/HTTP (protobuf) URL, including /v1/traces. Point it at Trailbeam or at the Collector.
OTEL_SERVICE_NAME string unknown_service service.name on the resource. Shown as the service on each step.
OTEL_RESOURCE_ATTRIBUTES k=v,... none service.instance.id here becomes messaging.client.id (client hooks). Without it, the package generates a random UUID v4 each time it creates a tracer provider: on each install call, and once for each wrapped Lambda handler.
AWS_LAMBDA_LOG_STREAM_NAME string set by Lambda Lambda wrappers only: used as messaging.client.id (one execution environment). Falls back to service.instance.id.
AWS_LAMBDA_FUNCTION_NAME string set by Lambda Lambda wrappers only: faas.name when the context has no function name.

Python package (trailbeam)#

trailbeam.boto.install()#

python
import trailbeam.boto
provider = trailbeam.boto.install(receipt_hash_key=None, span_exporter=None, span_processor=None)
Option Type Default Effect
receipt_hash_key str or None None (then TRAILBEAM_RECEIPT_HASH_KEY, then the default key) Receipt-hash key. Takes precedence over the environment variable. Positional or keyword.
span_exporter SpanExporter OTLP/HTTP exporter to TRAILBEAM_OTLP_TRACES_ENDPOINT Replaces the exporter. Wrapped in a BatchSpanProcessor. Keyword only.
span_processor SpanProcessor BatchSpanProcessor(span_exporter) Replaces the processor and exporter. Keyword only.

Returns the Trailbeam TracerProvider (always-on sampler, not registered globally). Calling install again replaces the previous installation. trailbeam.boto.uninstall() removes the hooks, flushes and shuts the provider down. If the application already called BotocoreInstrumentor().instrument(...), Trailbeam keeps that setup and its tracer provider, and runs the Trailbeam hooks first, then the application's. Otherwise it instruments botocore with the Trailbeam provider, so the stock botocore spans are exported to TRAILBEAM_OTLP_TRACES_ENDPOINT too; Trailbeam counts them as ignored.

trailbeam.aws_lambda.wrap#

python
@trailbeam.aws_lambda.wrap(receipt_hash_key=None, flush_timeout_millis=2000, report_batch_item_failures=True)
def handler(event, context): ...
Option Type Default Effect
receipt_hash_key str or None None As for install.
span_exporter SpanExporter OTLP/HTTP exporter As for install.
span_processor SpanProcessor BatchSpanProcessor As for install.
flush_timeout_millis int 2000 Longest wait for each of the two flushes per invocation (after the receives, after the outcomes).
report_batch_item_failures bool True Must match ReportBatchItemFailures on the event source mapping. When False, batchItemFailures in the return value is ignored, as Lambda ignores it.

The wrapped function has a trailbeam_tracer_provider attribute. Bare @trailbeam.aws_lambda.wrap uses every default.

Node.js package (@trailbeam/sdk)#

install(options) and instrument(client)#

javascript
const trailbeam = require('@trailbeam/sdk');
trailbeam.install({ receiptHashKey: process.env.TRAILBEAM_RECEIPT_HASH_KEY });
const sqs = trailbeam.instrument(new SQSClient({}));
Option Type Default Effect
receiptHashKey string undefined (then TRAILBEAM_RECEIPT_HASH_KEY, then the default key) Receipt-hash key. An empty string counts as unset.
spanExporter SpanExporter OTLPTraceExporter (protobuf) to TRAILBEAM_OTLP_TRACES_ENDPOINT Replaces the exporter, wrapped in a BatchSpanProcessor.
spanProcessor SpanProcessor BatchSpanProcessor(spanExporter) Replaces the processor and exporter.

install returns the Trailbeam BasicTracerProvider; calling it again replaces the installation and shuts the old provider down. All its options are optional. instrument adds the middleware trailbeamSqsMiddleware (step initialize, priority high) to one AWS SDK v3 client and returns the client. It calls install() with defaults if needed, is safe to call twice on the same client, and throws TypeError if the argument has no middleware stack. Every client you want recorded must be passed to instrument. forceFlush() flushes; uninstall() turns the middleware off on every client, flushes and shuts down. Both return a promise. The package also exports hashReceiptHandle(key, receiptHandle) and keyId(key), the two hash values of the span contract, plus version and SCHEMA_VERSION (3). Types are in node/index.d.ts.

wrapHandler(handler, options)#

Option Type Default Effect
receiptHashKey string undefined As for install.
spanExporter SpanExporter OTLP/HTTP exporter As for install.
spanProcessor SpanProcessor BatchSpanProcessor As for install.
flushTimeoutMs number 2000 Longest wait for each flush. A flush that takes longer is abandoned, not failed.
reportBatchItemFailures boolean true Must match ReportBatchItemFailures on the event source mapping.

The returned handler is always async (event, context) and has a trailbeamTracerProvider property. Async, sync and callback-style handlers are accepted. wrapHandler throws TypeError if handler is not a function.

Fixed values in the packages#

Not configurable in 0.1. Listed so you know the bounds.

Value Python Node.js Effect
Receipt handle to message ID map 100,000 entries, 12 h 100,000 entries, 12 h Oldest entries are dropped first. A settle whose handle is not in the map carries only the hash; Trailbeam still joins it.
Pending requests 10,000 none Python clears its map of in-flight calls if it reaches 10,000 (only when responses were lost).
Minimum key length 16 bytes 16 bytes Shorter keys are used, with a warning.
Clock time.time_ns() (nanoseconds) Date.now() (whole milliseconds, rounded down) Wall clock for span start and end.
thread.name set not set Python steps name the thread; Node.js steps do not.

Store (trailbeam-store)#

Each option has an environment variable. The command-line flag wins when both are set.

Option Environment Type Default Effect
--db TRAILBEAM_STORE_DB path trailbeam.sqlite SQLite file, WAL mode. Created if missing; older schemas are migrated on start.
--host TRAILBEAM_STORE_HOST address 127.0.0.1 Listen address. Use 0.0.0.0 in a container. There is no authentication.
--port TRAILBEAM_STORE_PORT int 4318 One port for OTLP ingest (/v1/traces), the JSON API (/api/) and the web screen. Use 4319 next to a Collector.
--stuck-after TRAILBEAM_STUCK_AFTER_S int, seconds 300 Default older_than_s for /api/stuck and /stuck.
--lambda-timeout TRAILBEAM_LAMBDA_TIMEOUT_S int, seconds 900 How long a Lambda receive may go without an outcome before lambda_invocation_ended_without_outcome is inferred. Set it to your function's timeout. Overridable per request with lambda_timeout_s.
--max-receive-count TRAILBEAM_MAX_RECEIVE_COUNT QUEUE=N[,QUEUE=N...], N from 1 to 1000 none Each queue's redrive maxReceiveCount. Enables dead_letter_queue_move: "not_seen_on_dlq_yet" and the dlq_move_not_seen notice. Spans do not carry it. Shown in /api/health as config.max_receive_counts.
--log-level TRAILBEAM_STORE_LOG_LEVEL Python log level INFO WARNING still shows rejected spans; DEBUG logs every HTTP request.
shell
trailbeam-store --db /var/lib/trailbeam/trailbeam.sqlite --port 4319 \
  --lambda-timeout 60 --max-receive-count orders=5,payments=3

A malformed --max-receive-count stops Trailbeam at start with expected QUEUE=N with N from 1 to 1000, got '...'.

Container image#

The image in server/Dockerfile sets TRAILBEAM_STORE_DB=/data/trailbeam.sqlite, TRAILBEAM_STORE_HOST=0.0.0.0 and TRAILBEAM_STORE_PORT=4318, runs as UID 10001, declares the volume /data, and has a HEALTHCHECK on /api/health every 30 s. Set any option with -e:

shell
docker run -d --name trailbeam -p 127.0.0.1:4318:4318 -v trailbeam-data:/data \
  -e TRAILBEAM_LAMBDA_TIMEOUT_S=60 -e TRAILBEAM_MAX_RECEIVE_COUNT=orders=5 trailbeam-store

Collector#

Environment variables used by collector/trailbeam-collector.yaml and collector/backend.yaml (otelcol-contrib required).

Variable Type Default Effect
TRAILBEAM_COLLECTOR_HTTP_ENDPOINT host:port localhost:4318 OTLP/HTTP listen address. Use 0.0.0.0:4318 in a container.
TRAILBEAM_COLLECTOR_GRPC_ENDPOINT host:port localhost:4317 OTLP/gRPC listen address.
TRAILBEAM_STORE_ENDPOINT URL http://localhost:4319 Store base URL. The exporter appends /v1/traces.
TRAILBEAM_BACKEND_ENDPOINT URL none, required by backend.yaml Base OTLP/HTTP URL of your tracing backend.
TRAILBEAM_BACKEND_SAMPLING_PERCENTAGE number 10 Share of non-SQS traces kept for the backend. SQS spans are always kept.

Fixed in the YAML (edit the file to change them):

Setting Value Effect
filter/trailbeam_only drops instrumentation_scope.name != "trailbeam" Trailbeam only receives Trailbeam spans.
otlp_http/trailbeam_store.compression gzip Trailbeam accepts gzip.
retry_on_failure.max_elapsed_time 0s Retry forever while Trailbeam is down.
sending_queue.queue_size 5000 batches, in memory When full, new spans are dropped. Lost on Collector restart.
tail_sampling/rest decision_wait: 5s, num_traces: 100000 Backend overlay only: tail sampling of non-SQS spans.

Demo#

Variable Default Effect
TRAILBEAM_DEMO_PORT 4318 Host port the demo's Trailbeam is published on (127.0.0.1 only).
COMPOSE_PROJECT_NAME trailbeam-demo Compose project and image name prefix. Change it to run a second checkout's demo.
TRAILBEAM_RECEIPT_HASH_KEY empty (default key) Passed to the demo services.