Python (boto3)

Install#

The trailbeam package (version 0.1.0.dev0) is not published. Install it into your application's environment from a clone of the repository, which is private for now:

gh repo clone nick-lennox/trailbeam
pip install ./trailbeam/python

To work on the package itself, run pip install -e 'python[test]' from the repository root instead.

Requirements: Python 3.10 or later (CI tests 3.10 and 3.12). It pulls in opentelemetry-api, opentelemetry-sdk and opentelemetry-exporter-otlp-proto-http (all >=1.27), opentelemetry-instrumentation-botocore>=0.48b0 and botocore>=1.34.

Add the hooks#

Call trailbeam.boto.install() once per process. It works before or after you create clients.

import os
import boto3
import trailbeam.boto

trailbeam.boto.install()  # reads TRAILBEAM_RECEIPT_HASH_KEY and TRAILBEAM_OTLP_TRACES_ENDPOINT

sqs = boto3.client("sqs")
url = os.environ["QUEUE_URL"]

sqs.send_message(QueueUrl=url, MessageBody="hello")
resp = sqs.receive_message(QueueUrl=url, WaitTimeSeconds=5)
for m in resp.get("Messages", []):
    sqs.delete_message(QueueUrl=url, ReceiptHandle=m["ReceiptHandle"])

Run it with the shared key and a service name:

export TRAILBEAM_RECEIPT_HASH_KEY='a-long-random-secret-shared-by-every-service'
export OTEL_SERVICE_NAME=orders-worker
python worker.py

For Lambda handlers use the wrapper instead. See AWS Lambda.

How the hooks attach#

install() registers a request_hook and response_hook on opentelemetry-instrumentation-botocore's BotocoreInstrumentor.

• botocore not instrumented yet: Trailbeam calls BotocoreInstrumentor().instrument(...) with its own tracer provider. The stock botocore span is then exported too, to the same endpoint. Trailbeam counts those spans as ignored.
• botocore already instrumented by your app: Trailbeam keeps your instrumentation and its tracer provider. It chains your existing hooks to run after its own.

Each Trailbeam span is a child of the stock botocore span for the same call, so it shares that trace ID.

Trailbeam uses its own TracerProvider with the ALWAYS_ON sampler and a BatchSpanProcessor. Your application's sampling never drops Trailbeam spans. Calling install() again shuts down the previous provider and replaces it. trailbeam.boto.uninstall() removes the hooks, flushes, and shuts the provider down.

The hooks only act on service sqs. They never raise: a failure inside a hook is logged at debug level on the trailbeam logger and your SQS call continues unchanged.

Options#

install() arguments#

Signature: install(receipt_hash_key=None, *, span_exporter=None, span_processor=None) -> TracerProvider.

The return value is the Trailbeam TracerProvider. Call .force_flush() on it when you need spans out now.

Environment variables#

What it records#

"Hash" is trailbeam.receipt_handle.hmac (first 32 hex characters of HMAC-SHA256 of the handle) and trailbeam.receipt_handle.key_id (first 8 hex characters of HMAC-SHA256 of "trailbeam key id"). The receipt handle itself and the message body are never exported.

No span is emitted for an empty receive, a send that failed, or a failed batch-send entry. Every span also carries the common attributes: trailbeam.schema_version = 3, trailbeam.kind, trailbeam.source = client, messaging.system.name = aws.sqs, messaging.operation.type, messaging.operation.name (the API action), messaging.client.id, process.pid, thread.name, messaging.destination.name (last path segment of QueueUrl) and aws.sqs.queue.url (the QueueUrl parameter, verbatim). See the span contract.

Receive request attributes#

Before ReceiveMessage is sent, the hook adds the system attributes it needs: ApproximateReceiveCount, SentTimestamp, ApproximateFirstReceiveTimestamp, AWSTraceHeader and DeadLetterQueueSourceArn. Nothing you asked for is removed. If you already ask for All, nothing is added. If you use only the deprecated AttributeNames, the names are added there; otherwise to MessageSystemAttributeNames. Message attributes are never requested.

Settle results and error codes#

trailbeam.result is ok or failed. When it is failed, trailbeam.error_code says why:

Handle map#

A receive stores receipt handle to message ID in an in-process map. A later settle span in the same process gets messaging.message.id from it. The map holds at most 100,000 entries (oldest dropped first), expires entries after 12 hours, and drops an entry after a successful delete. A delete from another process carries only the hash; Trailbeam joins it by hash anyway. See Receipt-handle hashing.

Timing and call order#

Each span starts just before the request is sent (when the request hook runs) and ends when the response or error arrives, read from time.time_ns(). Every span from one call, including each batch entry, shares that interval.

Each span also carries trailbeam.order.issued and trailbeam.order.returned, ticks of one counter shared by every thread in the process, and trailbeam.order.scope, a UUID naming that counter. A forked child gets a new scope and a counter starting at 1. If a fork happens while a call is in flight, that span has no trailbeam.order.issued. See Ordering.

Short-lived processes#

The BatchSpanProcessor exports in the background. Before a script or job exits, flush:

trailbeam.boto.uninstall()  # flushes and shuts down the Trailbeam provider

Verify it works#

In a test, capture spans in memory:

from opentelemetry.sdk.trace.export import SimpleSpanProcessor
from opentelemetry.sdk.trace.export.in_memory_span_exporter import InMemorySpanExporter
import trailbeam.boto

exporter = InMemorySpanExporter()
trailbeam.boto.install("test-key-at-least-16-bytes", span_processor=SimpleSpanProcessor(exporter))
# ... make SQS calls ...
print([s.name for s in exporter.get_finished_spans()])
# ['trailbeam send', 'trailbeam receive', 'trailbeam delete']
trailbeam.boto.uninstall()

With Trailbeam running, check that spans arrive and the message has a timeline:

curl -s http://127.0.0.1:4318/api/health
curl -s http://127.0.0.1:4318/api/messages/<message-id>

In /api/health, ingest.inserted grows and store.unjoined_settles stays at 0 when keys match. The timeline should look like this:

Common errors#

Related#

• Node.js (AWS SDK v3)
• AWS Lambda
• Run Trailbeam
• Configuration reference
• Span contract