Recorded and inferred steps
Recorded steps come from one span each; inferred steps are deduced by Trailbeam with a window and a reason, and the dead-letter move is shown only when SQS reports it.
A recorded step is one Trailbeam span: something a hook saw. An inferred step has no span: Trailbeam deduced it from the recorded steps around it, and always marks it inferred: true. One step type sits in between: the move to a dead-letter queue, which has no span but is reported by SQS.
The three kinds#
| Kind | inferred |
Has a time? | Placed by | On the screen |
|---|---|---|---|---|
| Recorded | false |
start and end from the span |
the proven order | solid step with its interval |
| Inferred | true |
no: time: null and a window |
its anchors | dashed, striped, inferred badge, between <from> and <to>, a reason |
| Reported dead-letter move | false, source: "sqs_dlq_source" |
no: time: null and a window |
its anchors | reported by SQS (DeadLetterQueueSourceArn on the DLQ receive), a window, a reason |
An inferred step is never shown as recorded, and is never used to order two recorded steps.
Recorded steps#
type |
From span | Notes |
|---|---|---|
sent |
trailbeam send |
Only for messages SQS accepted |
received |
trailbeam receive |
delivery counts observed receives in time order across all queues |
deleted |
trailbeam delete |
result: ok means SQS accepted the call, not that the message is gone. status says which: final (accepted, with the handle of the latest receive), stale_handle, failed, unmatched_handle |
visibility_changed |
trailbeam change_visibility |
visibility_timeout_s: 0 makes the message visible again at once |
lambda_outcome |
trailbeam outcome |
success, failed_reported, failed_invalid_response or failed_exception (with error_type, the class name only) |
Inferred steps#
Every inferred step has time: null, a window with from and to (either may be null), a reason sentence, and anchors: an earlier step it came after and, for some types, a later step it came before.
type |
Rule in Trailbeam | Earlier anchor | Later anchor | window |
Extra fields |
|---|---|---|---|---|---|
visibility_timeout_expired |
A receive on queue Q has a higher receive_count than the previous receive on Q (previous by end time). An accepted delete for the earlier delivery does not prevent it: the later receive shows the delete came too late |
The last accepted ChangeMessageVisibility for the earlier delivery that is proven before the later receive; else the earlier receive |
The later receive | anchor's start to later receive's end |
queue, after_delivery, before_delivery |
deliveries_not_observed |
The count jumps by more than 1 between consecutive receives on Q, or the first receive on the source queue has a count above 1 | The earlier receive, or the send (if observed) | The receive after the gap | earlier anchor's start to that receive's end |
queue, count (missing receives) |
deleted_by_lambda |
A lambda_outcome with success |
That outcome | none | outcome time to null |
queue, delivery |
lambda_invocation_ended_without_outcome |
A Lambda receive with no outcome for its faas_invocation_id, and more than lambda_timeout_s since the receive (--lambda-timeout, default 900 s, the Lambda maximum) |
That receive | none | receive's end to end plus lambda_timeout_s |
queue, delivery, faas_invocation_id |
The reason for a visibility timeout expiry#
Trailbeam picks the sentence from what happened to the earlier delivery:
| Seen on the earlier delivery | reason |
|---|---|
An accepted ChangeMessageVisibility to 0 (the latest one) |
"ChangeMessageVisibility set the visibility timeout to 0, which made the message visible again at once." |
An accepted ChangeMessageVisibility to N |
"The visibility timeout, last changed to N s, expired before the message was deleted." |
| An accepted delete, but the message was received again | "The visibility timeout expired before the delete: SQS accepted the delete, but the message was received again, so the delete came too late to remove it." |
| Nothing | "The visibility timeout expired before the message was deleted, so SQS delivered it again." |
How anchors place an inferred step#
An inferred step is after a step X exactly when X is its earlier anchor or proven before that anchor, and before X exactly when X is its later anchor or proven after it. Against anything else its order is unknown, so it shares a concurrent group with those steps. A rejected delete of delivery 1, sent between delivery 1 and its redelivery, lands in the same box as the expiry: Trailbeam cannot tell which came first.
If the two anchors contradict the proven order (the later anchor is proven before the earlier one), Trailbeam keeps only the earlier anchor.
Anchors appear in the step's after entry and before field with source inferred_from. They never appear in order.edges.
The dead-letter move is reported, not inferred#
SQS makes no API call Trailbeam could observe when it moves a message to a dead-letter queue. It reports the move later: the next receive, on the dead-letter queue, carries the same MessageId and the system attribute DeadLetterQueueSourceArn. The hooks record it as trailbeam.dead_letter_queue_source_arn.
Trailbeam adds one moved_to_dead_letter_queue step per (source queue, dead-letter queue) pair when a receive carries that attribute and its last : field is a different queue:
- Reported on the earliest such receive (
reported_on_delivery), which is its later anchor. - Earlier anchor: the last receive on the source queue that is not proven after the report. If none was observed, the send on the source queue.
- Fields:
source: "sqs_dlq_source",reported_by,source_arn,from_queue,to_queue,window,reason.
A receive on another queue without the attribute shows no move, even with the same message ID. Trailbeam never infers a move from the message ID alone.
Not seen on the dead-letter queue yet#
If you start Trailbeam with --max-receive-count orders=5 (or TRAILBEAM_MAX_RECEIVE_COUNT=orders=5), and the highest receive count on the source queue has reached 5, the message is not deleted, and no receive reported a move, then dead_letter_queue_move is "not_seen_on_dlq_yet" and the notice dlq_move_not_seen is shown. No move step is added. Trailbeam never receives from the dead-letter queue to check, because a receive changes the message's receive count, visibility and receipt handle.
dead_letter_queue_move |
Meaning |
|---|---|
| observed | A receive on the dead-letter queue reported the move |
| not_seen_on_dlq_yet | Count reached the configured maxReceiveCount; no report yet |
null |
Neither, including when maxReceiveCount is not configured for the source queue |
Why the labels matter#
An inferred step is still useful. "The visibility timeout expired between delivery 1 and delivery 2" is usually exactly the answer. But it's a conclusion, not an observation, and in a postmortem the difference matters. If someone asks "how do you know?", every step on a Trailbeam timeline should have an answer. For a recorded step it's a span you can open. For an inferred step it's the reason sentence and the two steps it was deduced from.
So inferred steps look different everywhere: dashed and striped with an inferred badge on the screen, inferred: true and time: null in the API. They are never used to order two recorded steps. If you build tools on the API, we'd ask you to keep that distinction visible too.
Worked example#
A Lambda consumer throws on the first delivery and succeeds on the second.
- 12:00:00.010Sentorders-api
- 12:00:01.000Receivedlambda · delivery 1 · receive count 1 · no later than
- 12:00:02.000Lambda outcomefailed_exception · RuntimeError
- inferredVisibility timeout expiredafter delivery 1, before delivery 2
- 12:00:40.000Receivedlambda · delivery 2 · receive count 2 · no later than
- 12:00:41.000Lambda outcomesuccess
- inferredDeleted by Lambdathe event source mapping deletes it, not observed
state is deleted_inferred. The failed outcome, the expiry and delivery 2 form one concurrent group: Lambda's poller made the second ReceiveMessage call, so nothing records whether it came before or after handler 1 failed.
Edge cases#
| What you see | Cause | What to do |
|---|---|---|
lambda_invocation_ended_without_outcome appears only 15 minutes later |
Trailbeam does not know your function's timeout and uses 900 s | Start Trailbeam with --lambda-timeout (TRAILBEAM_LAMBDA_TIMEOUT_S) or pass lambda_timeout_s to the API (1 to 86400) |
state: processing for a long time |
Latest Lambda receive has no outcome and is within the timeout | Wait out the timeout, or lower lambda_timeout_s |
deliveries_not_observed with count: 2 |
A consumer without Trailbeam received the message, or spans were lost | Instrument every consumer; check the Collector |
No deliveries_not_observed on a dead-letter queue receive with a high count |
The "first receive above 1" rule applies only to the source queue | Expected |
| Two receives with equal counts and no expiry | At-least-once duplicate delivery, not a timeout | See duplicate processing |
| No expiry between receives when a host clock is skewed | The "previous receive" is picked by end time; if the later count ends first, the rule does not fire | Fix clock sync |
| A receive on the DLQ but no move step | The receive lacks DeadLetterQueueSourceArn: older package, consumer without Trailbeam, or an SQS that does not set it |
Use packages with contract 3.1 on DLQ consumers |
Related#
- Reading a timeline
- How steps are ordered
- Trace a message to the dead-letter queue
- Lambda failures
- Span contract
Recorded (observed)
inferred: false.Step
Trailbeam span
trailbeam. Trailbeam reads only these.Hooks
trailbeam.boto.install()), the AWS SDK v3 middleware (trailbeam.instrument(client)) and the Lambda wrapper.Inferred
inferred: true. See Recorded and inferred steps.Span
OpenTelemetry terms in Vocabulary
Dead-letter queue (DLQ)
Amazon SQS terms in Vocabulary
Message
Amazon SQS terms in Vocabulary
Queue
Amazon SQS terms in Vocabulary
Receive
ReceiveMessage call returning a message. A receive does not remove the message; it starts its visibility timeout.Amazon SQS terms in Vocabulary
Visible
ReceiveMessage call.Amazon SQS terms in Vocabulary
Delete
DeleteMessage, made with a receipt handle. This is the only way a consumer removes a message. With the handle of an older receive, the call can still succeed while the message is received again.Amazon SQS terms in Vocabulary
Delivery
Change visibility
ChangeMessageVisibility: sets a new visibility timeout on an in-flight message, counted from the call. A value of 0 makes it visible at once.Amazon SQS terms in Vocabulary
Visibility timeout
Amazon SQS terms in Vocabulary
Concurrent group
DeadLetterQueueSourceArn
Amazon SQS terms in Vocabulary
Message ID
MessageId, assigned by SQS when the message is sent. It stays the same on every receive, including a receive from a dead-letter queue. Trailbeam joins everything about a message by this ID.Amazon SQS terms in Vocabulary
Receive count
ApproximateReceiveCount: how many times SQS has returned the message. It goes up by one on every receive.Amazon SQS terms in Vocabulary
Receipt handle
Amazon SQS terms in Vocabulary
maxReceiveCount
Amazon SQS terms in Vocabulary
Timeline
Consumer
ReceiveMessage) and then deletes them. A Lambda function behind an event source mapping is a consumer too.Amazon SQS terms in Vocabulary
Handler
Amazon SQS terms in Vocabulary
Function timeout
Task timed out after N seconds.AWS Lambda terms in Vocabulary
Collector