Skip to content

Recorded and inferred steps

Recorded steps come from one span each; inferred steps are deduced by Trailbeam with a window and a reason, and the dead-letter move is shown only when SQS reports it.

Based on 11 repo files at 54ea61b

docs/architecture.md, docs/glossary.md, docs/span-contract.md, docs/store-api.md, server/src/trailbeam_store/timeline.py, server/src/trailbeam_store/order.py, server/src/trailbeam_store/web.py, server/README.md, server/tests/test_timeline.py, python/src/trailbeam/aws_lambda.py, node/src/lambda.js

On this page

A recorded step is one Trailbeam span: something a hook saw. An inferred step has no span: Trailbeam deduced it from the recorded steps around it, and always marks it inferred: true. One step type sits in between: the move to a dead-letter queue, which has no span but is reported by SQS.

The three kinds#

Kind inferred Has a time? Placed by On the screen
Recorded false start and end from the span the proven order solid step with its interval
Inferred true no: time: null and a window its anchors dashed, striped, inferred badge, between <from> and <to>, a reason
Reported dead-letter move false, source: "sqs_dlq_source" no: time: null and a window its anchors reported by SQS (DeadLetterQueueSourceArn on the DLQ receive), a window, a reason

An inferred step is never shown as recorded, and is never used to order two recorded steps.

Recorded steps#

type From span Notes
sent trailbeam send Only for messages SQS accepted
received trailbeam receive delivery counts observed receives in time order across all queues
deleted trailbeam delete result: ok means SQS accepted the call, not that the message is gone. status says which: final (accepted, with the handle of the latest receive), stale_handle, failed, unmatched_handle
visibility_changed trailbeam change_visibility visibility_timeout_s: 0 makes the message visible again at once
lambda_outcome trailbeam outcome success, failed_reported, failed_invalid_response or failed_exception (with error_type, the class name only)

Inferred steps#

Every inferred step has time: null, a window with from and to (either may be null), a reason sentence, and anchors: an earlier step it came after and, for some types, a later step it came before.

type Rule in Trailbeam Earlier anchor Later anchor window Extra fields
visibility_timeout_expired A receive on queue Q has a higher receive_count than the previous receive on Q (previous by end time). An accepted delete for the earlier delivery does not prevent it: the later receive shows the delete came too late The last accepted ChangeMessageVisibility for the earlier delivery that is proven before the later receive; else the earlier receive The later receive anchor's start to later receive's end queue, after_delivery, before_delivery
deliveries_not_observed The count jumps by more than 1 between consecutive receives on Q, or the first receive on the source queue has a count above 1 The earlier receive, or the send (if observed) The receive after the gap earlier anchor's start to that receive's end queue, count (missing receives)
deleted_by_lambda A lambda_outcome with success That outcome none outcome time to null queue, delivery
lambda_invocation_ended_without_outcome A Lambda receive with no outcome for its faas_invocation_id, and more than lambda_timeout_s since the receive (--lambda-timeout, default 900 s, the Lambda maximum) That receive none receive's end to end plus lambda_timeout_s queue, delivery, faas_invocation_id

The reason for a visibility timeout expiry#

Trailbeam picks the sentence from what happened to the earlier delivery:

Seen on the earlier delivery reason
An accepted ChangeMessageVisibility to 0 (the latest one) "ChangeMessageVisibility set the visibility timeout to 0, which made the message visible again at once."
An accepted ChangeMessageVisibility to N "The visibility timeout, last changed to N s, expired before the message was deleted."
An accepted delete, but the message was received again "The visibility timeout expired before the delete: SQS accepted the delete, but the message was received again, so the delete came too late to remove it."
Nothing "The visibility timeout expired before the message was deleted, so SQS delivered it again."

How anchors place an inferred step#

An inferred step is after a step X exactly when X is its earlier anchor or proven before that anchor, and before X exactly when X is its later anchor or proven after it. Against anything else its order is unknown, so it shares a concurrent group with those steps. A rejected delete of delivery 1, sent between delivery 1 and its redelivery, lands in the same box as the expiry: Trailbeam cannot tell which came first.

If the two anchors contradict the proven order (the later anchor is proven before the earlier one), Trailbeam keeps only the earlier anchor.

Anchors appear in the step's after entry and before field with source inferred_from. They never appear in order.edges.

The dead-letter move is reported, not inferred#

SQS makes no API call Trailbeam could observe when it moves a message to a dead-letter queue. It reports the move later: the next receive, on the dead-letter queue, carries the same MessageId and the system attribute DeadLetterQueueSourceArn. The hooks record it as trailbeam.dead_letter_queue_source_arn.

Trailbeam adds one moved_to_dead_letter_queue step per (source queue, dead-letter queue) pair when a receive carries that attribute and its last : field is a different queue:

  • Reported on the earliest such receive (reported_on_delivery), which is its later anchor.
  • Earlier anchor: the last receive on the source queue that is not proven after the report. If none was observed, the send on the source queue.
  • Fields: source: "sqs_dlq_source", reported_by, source_arn, from_queue, to_queue, window, reason.

A receive on another queue without the attribute shows no move, even with the same message ID. Trailbeam never infers a move from the message ID alone.

Not seen on the dead-letter queue yet#

If you start Trailbeam with --max-receive-count orders=5 (or TRAILBEAM_MAX_RECEIVE_COUNT=orders=5), and the highest receive count on the source queue has reached 5, the message is not deleted, and no receive reported a move, then dead_letter_queue_move is "not_seen_on_dlq_yet" and the notice dlq_move_not_seen is shown. No move step is added. Trailbeam never receives from the dead-letter queue to check, because a receive changes the message's receive count, visibility and receipt handle.

dead_letter_queue_move Meaning
observed A receive on the dead-letter queue reported the move
not_seen_on_dlq_yet Count reached the configured maxReceiveCount; no report yet
null Neither, including when maxReceiveCount is not configured for the source queue

Why the labels matter#

An inferred step is still useful. "The visibility timeout expired between delivery 1 and delivery 2" is usually exactly the answer. But it's a conclusion, not an observation, and in a postmortem the difference matters. If someone asks "how do you know?", every step on a Trailbeam timeline should have an answer. For a recorded step it's a span you can open. For an inferred step it's the reason sentence and the two steps it was deduced from.

So inferred steps look different everywhere: dashed and striped with an inferred badge on the screen, inferred: true and time: null in the API. They are never used to order two recorded steps. If you build tools on the API, we'd ask you to keep that distinction visible too.

Worked example#

A Lambda consumer throws on the first delivery and succeeds on the second.

Timeline sample
  1. 12:00:00.010Sentorders-api
  2. 12:00:01.000Receivedlambda · delivery 1 · receive count 1 · no later than
  3. 12:00:02.000Lambda outcomefailed_exception · RuntimeError
  4. inferredVisibility timeout expiredafter delivery 1, before delivery 2
  5. 12:00:40.000Receivedlambda · delivery 2 · receive count 2 · no later than
  6. 12:00:41.000Lambda outcomesuccess
  7. inferredDeleted by Lambdathe event source mapping deletes it, not observed

state is deleted_inferred. The failed outcome, the expiry and delivery 2 form one concurrent group: Lambda's poller made the second ReceiveMessage call, so nothing records whether it came before or after handler 1 failed.

Edge cases#

What you see Cause What to do
lambda_invocation_ended_without_outcome appears only 15 minutes later Trailbeam does not know your function's timeout and uses 900 s Start Trailbeam with --lambda-timeout (TRAILBEAM_LAMBDA_TIMEOUT_S) or pass lambda_timeout_s to the API (1 to 86400)
state: processing for a long time Latest Lambda receive has no outcome and is within the timeout Wait out the timeout, or lower lambda_timeout_s
deliveries_not_observed with count: 2 A consumer without Trailbeam received the message, or spans were lost Instrument every consumer; check the Collector
No deliveries_not_observed on a dead-letter queue receive with a high count The "first receive above 1" rule applies only to the source queue Expected
Two receives with equal counts and no expiry At-least-once duplicate delivery, not a timeout See duplicate processing
No expiry between receives when a host clock is skewed The "previous receive" is picked by end time; if the later count ends first, the rule does not fire Fix clock sync
A receive on the DLQ but no move step The receive lacks DeadLetterQueueSourceArn: older package, consumer without Trailbeam, or an SQS that does not set it Use packages with contract 3.1 on DLQ consumers