Reading a timeline

A timeline is the list of everything Trailbeam knows happened to one message ID: the send, each delivery, visibility changes, deletes, Lambda outcomes, steps Trailbeam inferred, and a dead-letter move SQS reported. The web screen at /message?id=<message ID> and GET /api/messages/{message_id} show the same object.

The summary#

"Latest receive" means the receive with the latest end time.

Each step is an interval#

An observed step shows two times: just before its SQS request was sent, and when the response came back. On the screen:

2026-11-14 12:00:01.204000Z to 2026-11-14 12:00:01.231000Z (27.0 ms)

SQS acted somewhere in that interval; Trailbeam cannot see exactly when. Durations under 10 s are shown in milliseconds with one decimal, longer ones in seconds.

In the API each observed event has start, end, start_unix_nano, end_unix_nano (strings, since nanoseconds overflow a JavaScript number), start_observed, and time / time_unix_nano (equal to end).

A Lambda receive has start: null and start_observed: false. Lambda's poller made the ReceiveMessage call, so the wrapper only knows it happened no later than the moment the event arrived. The screen shows no later than <time>.

Step types and their screen labels#

delivery on a delete or visibility change is the delivery whose receipt handle was used, found by the hash join. A delete's status is one of final, stale_handle, failed or unmatched_handle (see Receipt-handle hashing).

Every step also names its consumer: service_name, client_id, source (client or lambda), pid, thread_name, faas_name, faas_invocation_id, whichever the span carried. span gives trace_id and span_id to look the call up in a tracing backend.

Order is shown only when proven#

Step A is listed above step B only if an observed fact proves A came first: the clocks, the call order in one process, SQS's receive count, or SQS's timestamps. Trailbeam never orders steps by what caused what. Each step names the fact under it on the screen. In the API, each step's after list names the steps it immediately follows, with sources and a why sentence, for example:

{"event": "e2", "type": "received", "sources": ["clock", "process_order"], "source": "clock",
 "why": "Receive 1 ended (2025-11-14T00:26:40.020000Z) before this started (2025-11-14T00:27:18.900000Z), on the hosts' clocks. Same process: receive 1 returned before this call was issued."}

The screen prints each entry as After receive 1: <why>. The four sources and their exact rules are in How steps are ordered.

Groups and concurrent boxes#

Trailbeam lists all steps in one order that agrees with every proven edge, then cuts the list at every point where everything above the cut is proven before everything below it. Each piece is a group (group: 0, 1, 2, ...).

• A group of one step is drawn as a normal step. An inferred step has a dashed, striped border and an inferred badge, with its time window and reason instead of a time.
• A group of more than one step is drawn as one dashed box with a concurrent badge and the label "concurrent, order unknown".
• An inferred step that sits alone between two groups gets group: null.

Inside a box:

• each step is a bar on one time scale shared by the box (a waterfall); inferred steps and dead-letter moves are drawn as hatched windows;
• rows are sorted by request start time. That is not the order SQS acted in;
• a step whose start is unknown (a Lambda receive) is drawn from the left edge and listed first;
• some pairs may still be ordered. Their after entries say so, and ordered_inside on the group counts them;
• the box can hold two steps that do not overlap, when both overlap a third step in the box (a chain of overlaps is one group).

Each groups[] entry has group, concurrent, label, observed_events, inferred_events, ordered_inside, start (null if any member's start is unknown) and end.

Worked example#

A long-polling consumer, a visibility timeout of 30 s, and slow processing. Receive 1's poll was already waiting when the message was sent.

What Trailbeam can prove here:

• Receive 1 ended (01.229) before the send ended (01.231), and its start (00.100) is before the send's. The clocks prove nothing. If SQS's SentTimestamp is earlier than ApproximateFirstReceiveTimestamp, sqs_timestamps orders the send first. If they are the same millisecond, the send and receive 1 are one concurrent box.
• Receive 1 is before receive 2 by clock and receive_count.
• worker-b's delete is after receive 2 by process_order even if both fall in the same millisecond.
• worker-a's late delete is after receive 1 (clock, process_order) and after receive 2 by clock. It used the handle of delivery 1, so its status is stale_handle, and successful_deliveries is 2.

Edge cases#

Related#

• How steps are ordered
• Recorded and inferred steps
• Receipt-handle hashing
• HTTP API