Skip to content
Docs

Use timelines in your own tools

Pull a message's timeline, the stuck list or duplicates as JSON for scripts, support tools and alerts, and read the fields correctly.

Based on 7 repo files at 54ea61b

server/src/trailbeam_store/server.py, server/src/trailbeam_store/queries.py, server/src/trailbeam_store/timeline.py, server/src/trailbeam_store/web.py, server/tests/test_api.py, docs/store-api.md, server/README.md

On this page

Use this when you want timelines in a script, a support tool, a runbook or an alert. Trailbeam's web screen is built from the same JSON API, so anything the screen shows, you can get.

1Step 1: Know the endpoints#

All endpoints are GET, on Trailbeam's port (default 4318), and return application/json. The same port also takes OTLP ingest on POST /v1/traces. Its status codes are in the HTTP API reference.

Endpoint Returns Parameters
/api/messages/{message_id} One message's full timeline lambda_timeout_s
/api/messages {"messages": [summary, ...]}, latest activity first limit, queue, lambda_timeout_s
/api/duplicates Summaries plus reasons reason, limit, queue, lambda_timeout_s
/api/stuck {"older_than_s": N, "messages": [...]}, summaries plus age_s and age_across_clocks, oldest first older_than_s, limit, queue, lambda_timeout_s
/api/health Status, version, ingest counters, stored counts, config none
Parameter Range Default
limit 1 to 1000 100
queue queue name all. Keeps messages with at least one span on that queue.
reason received_more_than_once, succeeded_more_than_once both
older_than_s 0 to 1000000000 Trailbeam's --stuck-after, 300
lambda_timeout_s 1 to 86400 Trailbeam's --lambda-timeout, 900

If a parameter is repeated, the last value wins. An empty value (?reason=) counts as not given.

Errors are {"error": "<message>"}:

Status When Example error
400 A parameter is not an integer, out of range, or an unknown reason limit must be between 1 and 1000
404 Unknown endpoint, or no span has this message ID no spans for message ID 'abc'
500 Unexpected failure, logged by Trailbeam internal error

2Step 2: Check Trailbeam is up and fed#

shell
curl -s http://127.0.0.1:4318/api/health
json
{
  "status": "ok", "version": "0.1.0.dev0",
  "ingest": {"requests": 12, "inserted": 340, "duplicates": 0, "ignored": 55, "rejected": 0},
  "store": {"spans": {"send": 100, "receive": 110, "delete": 100, "change_visibility": 3},
            "messages": 100, "unjoined_settles": 0},
  "config": {"stuck_after_s": 300, "lambda_timeout_s": 900, "max_receive_counts": {"orders": 3}}
}

ingest counts since Trailbeam started. store counts what is in the SQLite file. store.spans has a key only for span kinds with at least one stored row, so a kind with none is missing rather than 0. config shows Trailbeam's --stuck-after, --lambda-timeout and --max-receive-count values. unjoined_settles is the number of deletes and visibility changes whose receive has not arrived (or never will, for example with another hash key).

3Step 3: Get one message's timeline#

Message IDs go in the path, URL-encoded. An ID that contains / must be sent as %2F.

shell
curl -s "http://127.0.0.1:4318/api/messages/$(python3 -c 'import sys,urllib.parse;print(urllib.parse.quote(sys.argv[1],safe=""))' "$MESSAGE_ID")"

This timeline:

Timeline sample
  1. 00:26:40.030Sentorders-api · queue orders
  2. 00:26:40.020Receiveddelivery 1 · receive count 1 · after the send by sqs_timestamps
  3. inferredVisibility timeout expiredafter delivery 1, before delivery 2
  4. 00:27:20.000Receiveddelivery 2 · receive count 2
  5. 00:27:20.005Deletedelivery 2 · result ok · status final

comes back as (trimmed):

json
{
  "message_id": "5fea7756-0ea4-451a-a703-a558b933e274",
  "state": "deleted", "source_queue": "orders", "current_queue": "orders",
  "in_dead_letter_queue": false, "dead_letter_queue_move": null, "send_observed": true,
  "deliveries": 2, "successful_deliveries": 1,
  "first_seen": "2025-11-14T00:26:40.030000Z", "first_seen_clock": "host",
  "last_seen": "2025-11-14T00:27:20.005000Z",
  "concurrent_groups": 0,
  "events": [
    {"id": "e1", "type": "sent", "inferred": false, "group": 0, "end_unix_nano": "1763080040030000000", "...": "..."},
    {"id": "e2", "type": "received", "inferred": false, "group": 1, "delivery": 1, "receive_count": 1,
     "after": [{"event": "e1", "sources": ["sqs_timestamps"], "why": "SQS's timestamps show this message was sent (...) before its first receive (...)."}]},
    {"id": "e3", "type": "visibility_timeout_expired", "inferred": true, "group": null, "time": null,
     "window": {"from": "2025-11-14T00:26:39.100000Z", "to": "2025-11-14T00:27:20.000000Z"},
     "reason": "The visibility timeout expired before the message was deleted, so SQS delivered it again.",
     "after_delivery": 1, "before_delivery": 2},
    {"id": "e4", "type": "received", "inferred": false, "group": 2, "delivery": 2, "receive_count": 2},
    {"id": "e5", "type": "deleted", "inferred": false, "group": 3, "delivery": 2, "result": "ok", "status": "final"}
  ],
  "order": {"edges": ["..."], "edges_by_source": {"clock": 4, "process_order": 3, "receive_count": 1, "sqs_timestamps": 1}, "conflicts": []},
  "notices": []
}

Rules for reading it:

  • events is in display order. Each observed event is an interval: start (just before the request) to end (the response). time equals end.
  • *_unix_nano values are strings. Nanoseconds do not fit in a JavaScript number. Use BigInt or keep them as strings.
  • start is null with start_observed: false for a Lambda receive. Only its end is known.
  • Inferred events (inferred: true) have time: null, a window and a reason. They have no span. Show them differently.
  • moved_to_dead_letter_queue is inferred: false with source: "sqs_dlq_source", but also has time: null and a window.
  • id values (e1, e2, ...) are valid within one response only. Do not store them.
  • group numbers cut the list where everything before is proven before everything after. A group with concurrent: true in groups has no known order inside it.

4Step 4: Answer the support question#

"Did my message go through?" needs four fields:

Field Value Answer
state deleted Yes. A delete with the latest receipt handle was accepted.
deleted_inferred Yes. The latest Lambda invocation returned success, so Lambda deleted it.
processing Still running. The latest delivery is a Lambda invocation within lambda_timeout_s.
not_deleted No. Look at the last events and notices.
in_dead_letter_queue true It failed and is in the DLQ (current_queue).
successful_deliveries > 1 The work ran more than once.
notices[].code default_key, key_mismatch, unmatched_handle, send_not_observed, dlq_move_not_seen, order_conflict Something to fix or know before trusting the answer
shell
curl -s "http://127.0.0.1:4318/api/messages/$MESSAGE_ID" | jq -r '
  "state: \(.state)  deliveries: \(.deliveries) (\(.successful_deliveries) ok)  queue: \(.current_queue)",
  (.events[] | "\(.id) \(if .inferred then "[inferred]" else (.end // "no time") end) \(.type) \(.status // .outcome // "") \(.consumer.service_name // "")"),
  (.notices[] | "NOTICE \(.code): \(.message)")'
What the response shows Cause Fix
404 no spans for message ID No Trailbeam span carries this ID: wrong ID, uninstrumented producer and consumers, or spans not exported Check the ID. Check ingest.inserted on /api/health is growing.
404 for an ID with /, ? or spaces The ID was not URL-encoded Encode with encodeURIComponent or quote(id, safe="")
state: processing long after the function ended Trailbeam's Lambda timeout is the 900 s default Pass lambda_timeout_s, or start Trailbeam with --lambda-timeout
deliveries: 0, send_observed: true Sent, never received by an instrumented consumer See Find stuck messages

5Step 5: Use the lists in scripts#

shell
# Newest message ID
curl -s 'http://127.0.0.1:4318/api/messages?limit=1' | jq -r '.messages[0].message_id'

# Messages processed twice on orders, with their consumers
curl -s 'http://127.0.0.1:4318/api/duplicates?reason=succeeded_more_than_once&queue=orders' \
  | jq -r '.messages[] | "\(.message_id) \(.consumers | map(.client_id) | join(","))"'

# Stuck for over an hour, with ages measured across two clocks flagged
curl -s 'http://127.0.0.1:4318/api/stuck?older_than_s=3600' \
  | jq -r '.messages[] | "\(.age_s)s \(.message_id) \(.state)\(if .age_across_clocks then " (sqs clock)" else "" end)"'

Prevent it#

Most client bugs come from the encoding, the string timestamps and the 404. A small client handles all three.

python
import json, urllib.error, urllib.parse, urllib.request

BASE = "http://127.0.0.1:4318"

def timeline(message_id, lambda_timeout_s=None):
    url = f"{BASE}/api/messages/{urllib.parse.quote(message_id, safe='')}"
    if lambda_timeout_s:
        url += f"?lambda_timeout_s={lambda_timeout_s}"
    try:
        with urllib.request.urlopen(url, timeout=10) as r:
            return json.load(r)
    except urllib.error.HTTPError as e:
        if e.code == 404:
            return None                      # no spans for this ID
        raise RuntimeError(json.load(e)["error"]) from None

t = timeline("5fea7756-0ea4-451a-a703-a558b933e274")
ends = [int(e["end_unix_nano"]) for e in t["events"] if not e["inferred"] and "end_unix_nano" in e]
javascript
const BASE = 'http://127.0.0.1:4318';

async function timeline(messageId, lambdaTimeoutS) {
  const q = lambdaTimeoutS ? `?lambda_timeout_s=${lambdaTimeoutS}` : '';
  const res = await fetch(`${BASE}/api/messages/${encodeURIComponent(messageId)}${q}`);
  if (res.status === 404) return null;           // no spans for this ID
  const body = await res.json();
  if (!res.ok) throw new Error(body.error);
  return body;
}

const t = await timeline('5fea7756-0ea4-451a-a703-a558b933e274');
const ends = t.events.filter((e) => !e.inferred && e.end_unix_nano).map((e) => BigInt(e.end_unix_nano));