Skip to content
Docs

Run Trailbeam

One process for OTLP ingest, SQLite storage, the web screen and the JSON API, on one port.

Based on 11 repo files at 54ea61b

server/README.md, server/Dockerfile, server/pyproject.toml, server/src/trailbeam_store/__main__.py, server/src/trailbeam_store/server.py, server/src/trailbeam_store/db.py, server/src/trailbeam_store/otlp.py, server/src/trailbeam_store/timeline.py, server/src/trailbeam_store/queries.py, docs/store-api.md, demo/compose.yaml

On this page

Trailbeam (trailbeam-store, version 0.1.0.dev0) is one Python process. It receives Trailbeam spans over OTLP/HTTP, keeps each one as a row in SQLite, joins each delete and visibility change to its delivery by receipt-handle hash, and serves the web screen and the JSON API, building each timeline when it is requested. Everything is on one port.

Trailbeam makes no AWS calls and needs no AWS credentials. It only receives spans.

Install#

Docker#

No image is published. Build it from the repository, which is private for now:

shell
gh repo clone nick-lennox/trailbeam
docker build -t trailbeam-store trailbeam/server
docker run -d --name trailbeam -p 127.0.0.1:4318:4318 -v trailbeam-data:/data trailbeam-store

Open http://127.0.0.1:4318/. Export spans to http://127.0.0.1:4318/v1/traces.

The image (Python 3.12 slim by default, build arg PYTHON_VERSION) sets TRAILBEAM_STORE_DB=/data/trailbeam.sqlite, TRAILBEAM_STORE_HOST=0.0.0.0 and TRAILBEAM_STORE_PORT=4318. It runs as UID 10001. A named volume gets the right owner automatically; a bind-mounted directory must be writable by that UID, or add --user $(id -u). Pass options with -e, for example -e TRAILBEAM_STUCK_AFTER_S=600.

The image has a HEALTHCHECK that requests /api/health every 30 s (timeout 5 s, 3 retries). docker stop sends SIGTERM, which Trailbeam handles like Ctrl-C, closing SQLite cleanly.

Without Docker#

Python 3.10 or later. The only dependency is opentelemetry-proto>=1.27.

shell
pip install -e trailbeam/server
trailbeam-store --db trailbeam.sqlite --port 4318

It logs listening on http://127.0.0.1:4318 (OTLP: /v1/traces, API: /api/), db trailbeam.sqlite.

Options#

Each flag falls back to an environment variable, then a default.

Option Type Default What it does
--db / TRAILBEAM_STORE_DB path trailbeam.sqlite SQLite file, opened in WAL mode.
--host / TRAILBEAM_STORE_HOST address 127.0.0.1 Listen address. Use 0.0.0.0 in a container.
--port / TRAILBEAM_STORE_PORT int 4318 Listen port. Use 4319 when the Collector runs on the same host.
--stuck-after / TRAILBEAM_STUCK_AFTER_S int (s) 300 Age without a final delete after which a message is listed on /stuck. Default for older_than_s.
--lambda-timeout / TRAILBEAM_LAMBDA_TIMEOUT_S int (s) 900 Age after which a Lambda receive with no outcome is inferred killed. Default for lambda_timeout_s.
--max-receive-count / TRAILBEAM_MAX_RECEIVE_COUNT QUEUE=N,... none Each queue's redrive maxReceiveCount, N from 1 to 1000. Lets a timeline say "receive count exceeded maxReceiveCount; not seen on the DLQ yet". Spans do not carry it.
--log-level / TRAILBEAM_STORE_LOG_LEVEL level INFO Python logging level. Request lines are logged at DEBUG.

Example, as the demo runs it:

shell
docker run -d --name trailbeam -p 127.0.0.1:4318:4318 -v trailbeam-data:/data \
  -e TRAILBEAM_STUCK_AFTER_S=60 -e TRAILBEAM_MAX_RECEIVE_COUNT=orders=3 trailbeam-store

Ingest#

POST /v1/traces, OTLP/HTTP. Any other POST path returns 404 not found.

Request Accepted
Content-Type application/x-protobuf, application/protobuf, empty (read as protobuf), application/json
Content-Encoding identity, gzip, deflate
Body size up to 32 MiB after decompression

Only spans with instrumentation scope trailbeam are read. Others are counted as ignored. Spans with trailbeam.schema_version 2 or 3 are accepted; version 2 just lacks call order. A span with the same trace and span ID as a stored one (an exporter retry) is skipped and counted as a duplicate.

A Trailbeam span that breaks the contract is not stored. The response is still 200, with partial_success.rejected_spans and an error_message listing up to five reasons, and Trailbeam logs a warning. Reasons you can see:

Reason Meaning
trailbeam.schema_version 1 is not one of 2, 3 Old or foreign spans.
unknown trailbeam.kind ... / messaging.operation.name ... Kind not sqs.send, sqs.receive, sqs.settle or sqs.outcome, or a settle with an unknown operation.
receive span without receive_count Required attribute missing. Receives need queue, message ID, receive count and hash.
delete span without receipt_hmac, key_id Settle spans need queue, hash and trailbeam.result.
trailbeam.result 'x' is not ok or failed Bad settle result.
unknown trailbeam.outcome 'x' Outcome not one of the four values.
span ends before it starts Bad timestamps.

HTTP errors#

Status Body Cause
400 invalid protobuf: ..., invalid JSON: ..., unsupported Content-Type '...' Body cannot be decoded.
400 body does not match its Content-Encoding Corrupt gzip or deflate.
413 request body too large or without a valid Content-Length Over 32 MiB, or no valid length.
413 decompressed request body too large Decompresses past 32 MiB.
415 unsupported Content-Encoding ... E.g. br.
500 plain text An error inside Trailbeam, logged by the server.

Storage and the join#

Every Trailbeam span is one row in table spans, unique on (trace_id, span_id), with indexes on (mid, time_ns), (receipt_hmac, key_id) and (kind, time_ns). All attributes are also kept as JSON.

Sends, receives and outcomes belong to their own messaging.message.id. A delete or visibility change is joined to the trailbeam receive with the same trailbeam.receipt_handle.hmac and trailbeam.receipt_handle.key_id, which gives its message and the exact delivery (receive_row). The join runs at insert time in both directions, because a delete can arrive before its receive. If several receives match, the latest one at or before the settle wins. Hashes made with different keys never join.

The SQLite schema is version 4. Files from versions 1 to 3 are migrated in place on start. A file with an unknown version stops Trailbeam with store schema X, this version needs 4. There is no retention: data stays until you delete the file.

Screens and API#

Path What it shows
/ Message lookup and recent messages (?queue=, ?limit=)
/message?id=<id> One message's timeline. Each step names the fact that puts it after the one above. Steps no fact orders share a dashed box with a concurrent badge; inferred steps have an inferred badge.
/duplicates Duplicate deliveries (?reason=received_more_than_once or succeeded_more_than_once)
/stuck Messages with no final delete (?older_than_s=)
GET /api/messages/{id} Timeline JSON; 404 with {"error": "no spans for message ID '...'"} if unknown
GET /api/messages, /api/duplicates, /api/stuck Lists; limit 1 to 1000, default 100; queue filter
GET /api/health Counters and config

Full fields are in the HTTP API reference.

Verify it works#

shell
curl -s http://127.0.0.1:4318/api/health
json
{"status": "ok", "version": "0.1.0.dev0",
 "ingest": {"requests": 12, "inserted": 340, "duplicates": 0, "ignored": 55, "rejected": 0},
 "store": {"spans": {"send": 100, "receive": 110, "delete": 100, "change_visibility": 3},
           "messages": 100, "unjoined_settles": 0},
 "config": {"stuck_after_s": 300, "lambda_timeout_s": 900, "max_receive_counts": {}}}

ingest counts since start. store.spans lists only kinds that have rows. unjoined_settles counts deletes and visibility changes with no matching receive yet.

To check a fresh container end to end, post the recorded spans and check the pages (standard library only):

shell
python3 trailbeam/server/smoke/smoke_test.py http://127.0.0.1:4318

Common errors#

What you see Cause Fix
Address already in use on 4318 Collector on the same host Run Trailbeam with --port 4319.
Container exits at start: SQLite cannot open /data/trailbeam.sqlite Bind mount not writable by UID 10001 Use a named volume, chown 10001, or --user $(id -u).
Services cannot connect to the container TRAILBEAM_STORE_HOST overridden to 127.0.0.1 Keep 0.0.0.0 inside the container.
expected QUEUE=N with N from 1 to 1000 Bad --max-receive-count value Use orders=3,payments=5.
ingest.rejected grows Spans break the contract Read the warning in Trailbeam log; update the packages.
ingest.ignored grows Non-Trailbeam spans sent to Trailbeam Expected without a Collector; filter with the Collector.
unjoined_settles grows, key_mismatch notice Services use different TRAILBEAM_RECEIPT_HASH_KEY Use one key everywhere.
Database grows without bound No retention in 0.1 Stop Trailbeam, then move or delete the SQLite file.