Known limits in 0.1

Every known limit of Trailbeam 0.1, what it does to your timelines, and how to work around it.

Trailbeam 0.1 is pre-release (0.1.0.dev0 for the Python package and the server, 0.1.0-dev.0 for Node.js). Nothing is published: no PyPI package, no npm package, no container image.

Release and packaging#

Limit	Consequence	Workaround
Not published	`pip install trailbeam` and `npm install @trailbeam/sdk` fail	`pip install ./trailbeam/python`, `npm install ./trailbeam/node`, `docker build -t trailbeam-store server` from a clone. The repository is private: `gh repo clone nick-lennox/trailbeam`.
Python 3.10+, `botocore>=1.34`, `opentelemetry-instrumentation-botocore>=0.48b0`	Older runtimes cannot install	Upgrade the runtime.
Node.js 20+, AWS SDK v3 only	SDK v2 clients cannot be instrumented (`instrument` throws `TypeError`)	Use `@aws-sdk/client-sqs`.
Python and Node.js only	Java, Go, .NET consumers emit no Trailbeam spans	Their receives show up as deliveries_not_observed when an instrumented consumer later sees a higher receive count.
Collector needs `otelcol-contrib`	The core distribution lacks the `filter` and `tail_sampling` processors	Use contrib. Tested with 0.165.0.

What is recorded#

Limit	Consequence	Workaround
Only `SendMessage(Batch)`, `ReceiveMessage`, `DeleteMessage(Batch)`, `ChangeMessageVisibility(Batch)`	Other SQS operations leave no trace in a timeline	None in 0.1.
No message bodies, no message attributes, no receipt handles	You cannot search or view payloads	Log payloads yourself, keyed by message ID.
Lambda errors: class name only (`trailbeam.error_type`)	No error message or stack in the timeline	Find the invocation by `faas_invocation_id` in your logs.
Python `BaseException` that is not an `Exception` (such as `SystemExit`)	Not caught by the wrapper, so no outcome is emitted	Trailbeam infers lambda_invocation_ended_without_outcome after `--lambda-timeout`.
No `thread.name` from Node.js	Node.js steps name the process, not a thread	Expected.
`trailbeam outcome` only from the Lambda wrappers	For other consumers, success is an accepted delete or nothing	Read final deletes as success.
Lambda's delete is never observed	The timeline shows deleted_by_lambda and state `deleted_inferred`	Expected; the event source mapping deletes outside your process.
Lambda receive start not observed	Zero-length span with `trailbeam.start_observed = false`; never ordered after anything by `clock` or `process_order`	Receive count and SQS timestamps still order it.
Lambda flush capped at 2000 ms per flush	An unreachable endpoint cannot use up the function's time, but its spans are not exported in time	Make the endpoint reachable; raise `flush_timeout_millis` / `flushTimeoutMs`.
Timeouts and OOM kills cannot be detected in-process	Only inferred, after Trailbeam's `--lambda-timeout` (default 900 s, the Lambda maximum). Trailbeam does not know the function's real timeout	Set `--lambda-timeout` to the function's timeout.
Spans still waiting in the `BatchSpanProcessor` when a process exits	Lost. A lost receive can show up as deliveries_not_observed	Flush before exit (`trailbeam.boto.uninstall()`, `await trailbeam.forceFlush()`).
DLQ move only when the DLQ receive carries `DeadLetterQueueSourceArn`	No move if nothing receives from the DLQ, and Trailbeam never receives from it	Consume the DLQ with an instrumented client, and set `--max-receive-count` to get the `dlq_move_not_seen` notice.
Lambda SQS records and `DeadLetterQueueSourceArn`	Whether Lambda records carry the attribute is not verified, so a DLQ consumed only by Lambda may report no move	Consume the DLQ with an instrumented client if you need the move.
Redrive back from a DLQ	A move back to the source queue shows no event	None in 0.1.
`maxReceiveCount` is not in spans	Without configuration, `dead_letter_queue_move` is `null` even past the threshold	`--max-receive-count QUEUE=N` (N from 1 to 1000).
Same-name queues merge	Queues are grouped by `messaging.destination.name`, so `orders` in two accounts or regions is one queue	Use distinct names, or a separate Trailbeam per account and region.

Clocks and ordering#

Limit	Consequence	Workaround
No clock-offset correction between hosts	The `clock` source trusts each host. A host behind can show its delete before the receive it used.	Keep clocks in sync. The `delivery` field on a delete is still correct (it comes from the hash join).
Millisecond resolution in Node.js (`Date.now()`, whole milliseconds), rounded down. Python reads `time.time_ns()`	Two Node.js calls can look like they touch; the order is never reversed	None needed.
`process_order` is per process (Python) or per realm (Node.js)	A forked Python child and each Node.js worker thread (its own realm) get their own `trailbeam.order.scope`; no order across them	Expected.
Equal SQS timestamps prove nothing	Send and first receive in the same millisecond stay in one "concurrent, order unknown" box (common on emulators)	Expected.
Contradicting sources	Steps on the cycle get no order; `order_conflict` notice and `order.conflicts`	Fix the clock that jumped.
`first_seen` on SQS's clock when the send is not observed	`/api/stuck` cutoff and `age_s` compare SQS's clock with Trailbeam's (`age_across_clocks: true`)	Instrument the producer.
Lambda timeout inference compares Trailbeam's clock now with the Lambda host's receive time	Skew between them shifts when the step appears	Keep clocks in sync; the step needs the full `lambda_timeout_s` anyway.
Schema version 2 spans have no call order	Fewer proven edges	Upgrade to packages on schema version 3. Version 1 spans are rejected.

Hashing and joins#

Limit	Consequence	Workaround
One receipt-hash key per set of queues	Different keys never join: `key_mismatch` notice, deletes stay `unmatched_handle`	Same `TRAILBEAM_RECEIPT_HASH_KEY` everywhere.
Default key `trailbeam-default-receipt-hash-key` (key ID `b5e633f7`)	No protection; `default_key` notice	Set a secret key of at least 16 bytes.
In-process handle map: 100,000 entries, 12 hours	Evicted handles produce settle spans with only the hash	Trailbeam joins by hash anyway.
Python pending-call map: 10,000	Cleared if reached (only when responses were lost)	None needed.

Storage and API#

Limit	Consequence	Workaround
No authentication	Anyone who reaches the port can read timelines and post spans	Bind to `127.0.0.1` (default) or publish the port on `127.0.0.1` only.
No retention, no delete API	The SQLite file grows forever	Stop Trailbeam and delete the file (`docker compose down -v` in the demo).
OTLP/HTTP only	No OTLP/gRPC on Trailbeam	Send gRPC to the Collector, which exports HTTP to Trailbeam.
Only scope `trailbeam`, schema versions 2 and 3	Everything else is ignored or rejected
Request body 32 MiB, before and after decompression, with a valid `Content-Length`	`413`	Smaller export batches.
`Content-Encoding` other than `gzip` or `deflate`	`415`	Use `gzip`.
One SQLite connection behind a lock	Requests are serialised	Sized for a single container in 0.1.
Order computed over every pair of a message's spans	Large timelines cost more to build	Keep `limit` small on list endpoints.
List endpoints build one full timeline per listed message	`limit` (max 1000) is not for exports	Use the screens' sizes (default 100).
Query ranges	`limit` 1 to 1000, `lambda_timeout_s` 1 to 86400, `older_than_s` 0 to 1000000000; otherwise `400`
`ingest` counters live in memory	Reset to 0 on restart	Read `store.*` for persisted counts.
Unknown database schema	Start fails: `store schema N, this version needs 4`	Use a matching Trailbeam version.

Collector#

Limit	Consequence	Workaround
`backend.yaml` keeps every SQS span, samples the rest	Sampled-out traces show SQS spans without their HTTP or database parents	Expected trade-off. Raise `TRAILBEAM_BACKEND_SAMPLING_PERCENTAGE`.