Skip to content
Docs

HTTP API

Every endpoint with its parameters, limits, response fields, error codes and an example.

Based on 9 repo files at 54ea61b

server/src/trailbeam_store/server.py, server/src/trailbeam_store/queries.py, server/src/trailbeam_store/timeline.py, server/src/trailbeam_store/otlp.py, server/src/trailbeam_store/db.py, server/src/trailbeam_store/web.py, server/tests/test_api.py, server/tests/test_ingest.py, docs/store-api.md

On this page

One port (default 4318) serves OTLP ingest, the read-only JSON API and the web screen. The full specification, with every rule behind the fields, is docs/store-api.md.

Each endpoint below has a playground. Change the parameters, copy the request as curl, Python or JavaScript, and send it to the sandbox: a Trailbeam database recorded from the local demo, 80 messages on orders and orders-dlq, frozen at one moment. Its answers come from Trailbeam's own code, errors included. The code you copy calls your own Trailbeam on 127.0.0.1:4318.

Method Path Returns
POST /v1/traces OTLP/HTTP ingest response
GET /api/messages/{message_id} One message's timeline
GET /api/messages Recent messages, newest activity first
GET /api/duplicates Messages received or processed more than once
GET /api/stuck Messages with no final delete, oldest first
GET /api/health Ingest counters, stored counts, configuration

Conventions#

  • Every /api/ response is application/json. Errors are {"error": "<message>"}.
  • Times are ISO 8601 UTC with microseconds: "2025-11-14T00:26:40.030000Z". Nanosecond fields (*_unix_nano) are strings, so JavaScript keeps them exact.
  • URL-encode the message ID in the path (encodeURIComponent). /api/messages/id%2Fwith%20spaces%3F%26 finds id/with spaces?&.
  • A repeated query parameter takes its last value.
  • No authentication in 0.1.

Query parameters#

Parameter Type Default Range Accepted by
limit int 100 1 to 1000 /api/messages, /api/duplicates, /api/stuck
queue string none the same three: only messages with at least one span whose messaging.destination.name is this queue
lambda_timeout_s int, seconds --lambda-timeout (900) 1 to 86400 every endpoint that builds timelines
reason enum both received_more_than_once, succeeded_more_than_once /api/duplicates
older_than_s int, seconds --stuck-after (300) 0 to 1000000000 /api/stuck

Errors#

Status Body When
400 {"error": "limit must be between 1 and 1000"} An integer parameter out of range
400 {"error": "older_than_s must be an integer"} A non-integer value (?older_than_s=soon)
400 {"error": "reason must be received_more_than_once or succeeded_more_than_once"} Unknown reason
404 {"error": "no spans for message ID 'abc'"} No stored span belongs to that message
404 {"error": "no such endpoint: /api/nope"} Unknown first path segment, or /api/ alone
404 {"error": "no such endpoint"} Extra path segments: /api/messages/a/b, /api/stuck/x, /api/health/x
500 {"error": "internal error"} A bug; Trailbeam logs the traceback

POST /v1/traces#

OTLP/HTTP trace export. Point TRAILBEAM_OTLP_TRACES_ENDPOINT or the Collector's TRAILBEAM_STORE_ENDPOINT here.

shell
curl -s -X POST http://127.0.0.1:4318/v1/traces \
  -H 'Content-Type: application/json' \
  --data-binary @spans.json
Header Accepted values
Content-Type application/x-protobuf, application/protobuf, empty (read as protobuf), application/json (OTLP/JSON)
Content-Encoding identity, gzip, deflate
Content-Length Required, at most 32 MiB; the decompressed body is capped at 32 MiB too

Processing per span:

  1. Spans whose instrumentation scope is not trailbeam are counted in ingest.ignored and dropped.
  2. A Trailbeam span is rejected if trailbeam.schema_version is not 2 or 3, trailbeam.kind is unknown, it has no trace ID, span ID, start or end, it ends before it starts, a required attribute is missing (for example receive span without receipt_hmac), trailbeam.result is not ok/failed, or trailbeam.outcome is unknown. The full list of required attributes per kind is in span contract §4.5.
  3. A span already stored (same trace ID and span ID) is skipped and counted in ingest.duplicates.
  4. Deletes and visibility changes are joined to the receive with the same trailbeam.receipt_handle.hmac and trailbeam.receipt_handle.key_id, whichever arrives first.

Rejections return 200 with OTLP partial_success: rejected_spans and error_message such as 1 Trailbeam span(s) rejected: trailbeam.schema_version 1 is not one of 2, 3. In JSON mode the body is {"partialSuccess": {"rejectedSpans": "1", "errorMessage": "..."}}; with no rejections it is {}.

Status Body (text/plain) When
400 invalid protobuf: ..., invalid JSON: ..., invalid OTLP/JSON: ..., unsupported Content-Type '...' The body cannot be decoded
400 body does not match its Content-Encoding Bad gzip or deflate data
404 not found Any POST path other than /v1/traces (for example /v1/metrics)
413 request body too large or without a valid Content-Length / decompressed request body too large Over 32 MiB, or no usable Content-Length
415 unsupported Content-Encoding br Encoding other than the three above
500 internal error Logged as ingest failed

GET /api/messages/{message_id}#

The message's timeline. Accepts lambda_timeout_s.

shell
curl -s "http://127.0.0.1:4318/api/messages/07b04137-3706-418a-a4a2-5574ad88d119"

The sample messages cover the cases in the guides. The poison message shows a moved_to_dead_letter_queue event with its anchors, and the slow consumer a deleted event with status: failed.

Summary fields#

Field Values
state deleted observed final delete. deleted_inferred latest delivery was a Lambda success. processing latest delivery is a Lambda invocation still inside lambda_timeout_s. not_deleted anything else.
source_queue / current_queue Queue of the first send (or first receive) / of the latest receive
in_dead_letter_queue current_queue is the to_queue of an observed moved_to_dead_letter_queue
dead_letter_queue_move "observed", "not_seen_on_dlq_yet" (needs --max-receive-count), or null
send_observed A trailbeam send span is stored
deliveries Observed receives, on any queue
successful_deliveries Distinct deliveries with an accepted delete, plus distinct Lambda invocations with success. Above 1 means the work ran twice.
first_seen, first_seen_clock Send end on the producer's clock (host); without a send, earliest SentTimestamp (sqs)
last_seen End of the latest span
concurrent_groups, groups Group count with concurrent: true; each group's group, concurrent, label ("concurrent, order unknown" or null), observed_events, inferred_events, ordered_inside, start, end
order edges (from before to, with sources), edges_by_source, conflicts
notices {"code", "message"} list

Events#

Every event has id (e1, e2, ... valid in one response), type, inferred, group and after (each entry: event, type, sources, source, why). Observed events add start, end, start_unix_nano, end_unix_nano, start_observed, time, time_unix_nano, queue, consumer, span and process_order.

type Inferred Extra fields
sent no batch_entry_id
received no delivery, receive_count, sent_timestamp, sent_timestamp_ms, first_receive_timestamp, first_receive_timestamp_ms
deleted no delivery, result, status, error_code, batch_entry_id
visibility_changed no delivery, result, visibility_timeout_s, error_code, batch_entry_id
lambda_outcome no outcome, error_type, delivery
moved_to_dead_letter_queue no (no span, no time) source (sqs_dlq_source), reported_by, from_queue, to_queue, source_arn, reported_on_delivery, window, reason, before
visibility_timeout_expired yes queue, after_delivery, before_delivery, window, reason
deliveries_not_observed yes queue, count, window, reason
deleted_by_lambda yes queue, delivery, window, reason
lambda_invocation_ended_without_outcome yes queue, delivery, faas_invocation_id, window, reason

Events placed by anchors (inferred events and the DLQ move) carry before when they have a later anchor. Order sources on edges: clock, process_order, receive_count, sqs_timestamps. Anchors of inferred events and DLQ moves use inferred_from and sqs_dlq_source; those never appear in order.edges.

deleted.status: final accepted, handle from the latest receive. stale_handle accepted, handle from an earlier delivery. failed rejected by SQS, see error_code. unmatched_handle accepted, but no stored receive has this hash.

outcome: success, failed_reported, failed_invalid_response, failed_exception.

Notice codes: order_conflict, default_key, key_mismatch, unmatched_handle, send_not_observed, dlq_move_not_seen.

GET /api/messages#

Recent messages by latest span end. Parameters: limit, queue, lambda_timeout_s.

shell
curl -s "http://127.0.0.1:4318/api/messages?limit=5"

consumers is the distinct (service_name, client_id) pairs that received the message. This summary object is the item shape of all three list endpoints.

GET /api/duplicates#

Parameters: limit, queue, reason, lambda_timeout_s. Newest activity first. Each summary adds reasons:

  • received_more_than_once: deliveries above 1.
  • succeeded_more_than_once: successful_deliveries above 1.
shell
curl -s "http://127.0.0.1:4318/api/duplicates?limit=5"

GET /api/stuck#

Messages first seen more than older_than_s ago with no accepted delete using the latest receive's handle and no Lambda success for the latest receive. Includes never-received messages and messages sitting in a dead-letter queue. Oldest first_seen first. Parameters: older_than_s, limit, queue, lambda_timeout_s.

shell
curl -s "http://127.0.0.1:4318/api/stuck?older_than_s=10&limit=5"

age_s is Trailbeam's clock minus first_seen, rounded to 3 decimals. age_across_clocks: true means first_seen is on SQS's clock, so clock skew is inside the age.

GET /api/health#

shell
curl -s "http://127.0.0.1:4318/api/health"
Field Meaning
ingest.* Counters since the process started (reset on restart)
store.spans Stored spans per kind; a kind with no rows is absent
store.messages Distinct messages
store.unjoined_settles Deletes and visibility changes with no matching receive yet
config The running --stuck-after, --lambda-timeout, --max-receive-count

The container's HEALTHCHECK calls this endpoint.