Skip to content
Docs

Troubleshooting

Symptom, cause, check and fix for missing spans, timeline gaps, confusing order, Lambda and the Collector.

Based on 20 repo files at 54ea61b

server/src/trailbeam_store/server.py, server/src/trailbeam_store/otlp.py, server/src/trailbeam_store/db.py, server/src/trailbeam_store/timeline.py, server/src/trailbeam_store/__main__.py, server/tests/test_ingest.py, server/tests/test_join.py, server/tests/test_timeline.py, server/tests/test_lambda_end_to_end.py, python/src/trailbeam/boto.py, python/src/trailbeam/aws_lambda.py, python/src/trailbeam/_contract.py, node/src/sqs.js, node/src/lambda.js, collector/README.md, collector/trailbeam-collector.yaml, server/README.md, docs/store-api.md, docs/quickstart.md, demo/README.md

On this page

Start every investigation with the health endpoint. It tells you whether spans reach Trailbeam, and what happened to them.

shell
curl -s http://127.0.0.1:4318/api/health | jq '{ingest, store}'
Counter Reading
ingest.requests is 0 Nothing reached /v1/traces since Trailbeam started
ingest.ignored grows, inserted does not Spans arrive, but none has instrumentation scope trailbeam
ingest.rejected grows Trailbeam spans break the contract. Trailbeam logs N Trailbeam span(s) rejected: <reasons> at WARNING
store.unjoined_settles grows Deletes or visibility changes whose receive never arrived

Then open the message: curl -s http://127.0.0.1:4318/api/messages/<id> | jq '.notices'.

No spans arriving#

Symptom Cause Check Fix
requests: 0 Wrong endpoint. TRAILBEAM_OTLP_TRACES_ENDPOINT is a full URL. echo $TRAILBEAM_OTLP_TRACES_ENDPOINT in the service Set it to http://<trailbeam-host>:4318/v1/traces (or the Collector's /v1/traces).
requests: 0, services in containers Trailbeam listens on 127.0.0.1 by default trailbeam-store start log: listening on http://127.0.0.1:4318 --host 0.0.0.0 (the image already sets it). Only on a trusted network: no authentication.
Port conflict on 4318 Trailbeam and the Collector both default to 4318 Which process owns the port Run Trailbeam with --port 4319; the Collector's default TRAILBEAM_STORE_ENDPOINT is http://localhost:4319.
Short-lived script records nothing Spans wait in a BatchSpanProcessor and the process exits first Spans appear only from long-running services Python: trailbeam.boto.uninstall() (or provider.force_flush()) before exit. Node: await trailbeam.forceFlush() or await trailbeam.uninstall().
Node service: no spans from one client Only clients passed to instrument are hooked grep -n instrument( const sqs = trailbeam.instrument(new SQSClient({})) for every client.
Node: TypeError: trailbeam.instrument expects an AWS SDK v3 SQSClient Passed something without middlewareStack.add (an SDK v2 client, a wrapper) Pass the v3 SQSClient itself. SDK v2 is not supported.
ingest.ignored grows, nothing stored Only stock spans arrive (for example botocore's own span) Stock spans are ignored by design. In Python, install() exports the stock botocore span too, so some ignored is normal. Make sure trailbeam.boto.install() or trailbeam.instrument() ran in that process.
rejected: trailbeam.schema_version 1 is not one of 2, 3 A package older than schema version 2 Package version Upgrade both packages from the same repository commit.
rejected: receive span without receive_count SQS (or an emulator) did not return ApproximateReceiveCount Raw ReceiveMessage response Use an SQS implementation that returns the attribute. The hooks always request it.
rejected: receive span without receipt_hmac A Lambda record with no receiptHandle The event JSON Real SQS events carry it. Fix the test event.
rejected: span ends before it starts The span was not built by the Trailbeam packages (they clamp start to end), or was rewritten on the way The sender of that span Send Trailbeam spans unmodified.
Ingest returns 413 Body over 32 MiB (before or after decompression), or no valid Content-Length Exporter batch size Export smaller batches.
Ingest returns 415 unsupported Content-Encoding br Encoding other than identity, gzip, deflate Exporter compression setting Use gzip.
Ingest returns 400 The body is not valid protobuf or OTLP/JSON, the Content-Type is not supported, or the body does not match its Content-Encoding The plain-text response body Export with the OpenTelemetry OTLP/HTTP exporter, protobuf or JSON.
An /api/ call returns 500 with "internal error" An error inside Trailbeam Trailbeam log, which records it Report it with the log line.

Timeline gaps#

What the timeline shows Cause Check Fix
Notice send_not_observed Producer not instrumented, or the send predates the database send_observed: false Install the hooks in the producer. first_seen_clock is sqs until then.
deliveries_not_observed Receive count jumped, or the first receive has a count above 1 count on the event Instrument every consumer of that queue; check for dropped spans in ingest.rejected.
Delete with unmatched_handle The receiving process is not instrumented, or its spans were lost store.unjoined_settles Instrument the receiver. The join resolves itself when the receive arrives later.
Notice key_mismatch Deleting service uses another TRAILBEAM_RECEIPT_HASH_KEY Compare trailbeam.receipt_handle.key_id on the spans One key on every service of those queues. Hashes from two keys never join.
Notice default_key No key configured (key ID b5e633f7) Warning trailbeam: no receipt-hash key configured in service logs Set TRAILBEAM_RECEIPT_HASH_KEY. Expected in the demo.
No delete at all, state not_deleted Deleted through an uninstrumented client, or never deleted Is the message on /api/stuck? Instrument the deleting client. If it really was not deleted, see stuck messages.
Delete failed with ReceiptHandleIsInvalid Late delete with an old handle on ElasticMQ, which rejects it error_code Expected on ElasticMQ. Amazon SQS accepts it and Trailbeam shows stale_handle.
Delete stale_handle Accepted delete with an earlier delivery's handle; a later receive happened delivery on the delete Processing outlived the visibility timeout. See visibility timeout.
Delete failed with MissingFromResponse A batch entry in neither Successful nor Failed of the response batch_entry_id Check the batch call's response.
No DLQ move, notice dlq_move_not_seen Receive count reached maxReceiveCount; no DLQ receive has reported the move dead_letter_queue_move Instrument the DLQ consumer. Trailbeam never receives from the DLQ itself.
No DLQ move and no notice --max-receive-count not set for the source queue config.max_receive_counts in health --max-receive-count orders=5.
Timeline sample
  1. 12:00:01.204Sentorders-api · 27 ms
  2. inferredDeliveries not observedreceive count 1 not seen
  3. 12:00:31.410Delivery 1worker-b · receive count 2
  4. 12:00:31.902Deletedworker-b · final

This shape means a consumer without Trailbeam took the first delivery.

Ordering confusion#

What the timeline shows Cause Check Fix
Send and first receive in one "concurrent, order unknown" box No source orders them. The long-poll receive was already waiting before the send, so the intervals overlap, and the two SQS timestamps are equal Same-millisecond sent_timestamp_ms and first_receive_timestamp_ms Expected. ElasticMQ stamps both in the same millisecond, which proves nothing. Amazon SQS usually tells them apart.
Delete listed before the receive whose handle it used The deleting host's clock is behind The delete's delivery still names the right receive Keep host clocks in sync. 0.1 does not correct clock offset.
Notice order_conflict Sources contradict each other, for example a clock jumped back order.conflicts No order is shown for those steps. Fix the clock.
Steps from one process not ordered Schema version 2 spans carry no call order process_order: null on events Upgrade the package to schema version 3.
Python child processes not ordered against the parent A forked child gets a new trailbeam.order.scope process_order.scope differs Expected: ticks are not comparable across a fork.
Lambda receive concurrent with earlier steps Lambda receives have no observed start start_observed: false Expected. Receive count and SQS timestamps can still order it.
/api/stuck age looks wrong first_seen is on SQS's clock age_across_clocks: true Instrument the producer so first_seen is a host time.

Lambda#

Symptom Cause Check Fix
State processing for a long time Lambda receive, no outcome, still inside lambda_timeout_s config.lambda_timeout_s Set --lambda-timeout to the function's timeout, or pass ?lambda_timeout_s=60.
lambda_invocation_ended_without_outcome Timeout, out-of-memory kill, or a Python BaseException (SystemExit) faas_invocation_id, then CloudWatch for that request Raise the timeout or memory. This step only appears once the receive is older than lambda_timeout_s.
Every record failed_invalid_response A batchItemFailures entry whose itemIdentifier is missing or not a record ID Return value Return {"batchItemFailures": [{"itemIdentifier": record.messageId}]}. Lambda fails the whole batch otherwise.
failed_reported shown but Lambda deleted the message ReportBatchItemFailures is off on the mapping Mapping's FunctionResponseTypes report_batch_item_failures=False (Python) or reportBatchItemFailures: false (Node).
No spans from the function Endpoint unreachable from the function. Each flush gives up after flush_timeout_millis / flushTimeoutMs (2000) and never fails the invocation. TRAILBEAM_OTLP_TRACES_ENDPOINT in the function's environment Make the endpoint reachable from the function.
Deleted state is deleted_inferred, no delete step Lambda's event source mapping deletes; Trailbeam cannot see it deleted_by_lambda inferred step Expected.

Collector#

Symptom Cause Check Fix
Collector fails to start on filter or tail_sampling Core distribution otelcol --version Use otelcol-contrib (tested with 0.165.0).
Store gets nothing through the Collector TRAILBEAM_STORE_ENDPOINT includes /v1/traces, or points at 4318 where the Collector itself listens Collector exporter logs Base URL only: TRAILBEAM_STORE_ENDPOINT=http://localhost:4319.
Apps in containers cannot reach the Collector Listens on localhost:4318 by default TRAILBEAM_COLLECTOR_HTTP_ENDPOINT=0.0.0.0:4318.
Spans lost after Trailbeam was down In-memory queue of 5,000 batches filled, or the Collector restarted Collector logs for dropped data Keep Trailbeam up; a persistent queue is not configured in 0.1.
Backend shows SQS spans without their HTTP parent The rest of the trace was tail-sampled out TRAILBEAM_BACKEND_SAMPLING_PERCENTAGE Expected trade-off of backend.yaml.

The demo#

Symptom Cause Fix
docker info has no Server Version: line The Docker daemon is not running. docker info prints a Server: heading even then Start Docker Desktop, or run sudo systemctl start docker.
docker: 'compose' is not a docker command Compose v2 is missing Install it. Docker Desktop includes it; on Linux, install the docker-compose-plugin package.
Cannot connect to the Docker daemon, or failed to connect to the docker API The daemon is not running Start Docker Desktop, or run sudo systemctl start docker.
The same error in WSL, or The command 'docker' could not be found in this WSL 2 distro Docker Desktop's WSL integration is off for your distribution Turn it on in Docker Desktop (Settings, Resources, WSL integration), or use docker.exe in place of docker.
fatal: could not read Username for 'https://github.com' A plain git clone of the private repository, without credentials Clone with gh repo clone nick-lennox/trailbeam, logged in with gh auth login.
port is already allocated on 4318 Something else uses the port Set TRAILBEAM_DEMO_PORT, as in the quickstart.
Another checkout's demo stops or its data shows up Every checkout uses the Compose project trailbeam-demo and its volume Set a different COMPOSE_PROJECT_NAME for the second checkout.
The start page is empty Spans are exported in batches Wait a few seconds, then run docker compose -f demo/compose.yaml logs collector trailbeam.
A build step fails with CERTIFICATE_VERIFY_FAILED A proxy on your network intercepts TLS, and the build containers do not trust its certificate None in 0.1: the Dockerfiles have no setting for it.

Start-up#

Symptom Cause Fix
store schema N, this version needs 4 Database from a newer or unknown version Use the matching Trailbeam version, or a new --db.
expected QUEUE=N with N from 1 to 1000 Malformed --max-receive-count orders=5,payments=3.
Container cannot write /data Bind mount not writable by UID 10001 Use a named volume, or --user $(id -u).