Debug Lambda timeouts and crashes

Use this when an SQS-triggered Lambda function fails, retries, times out, or a message "disappears" between the queue and your handler.

1Step 1: Wrap the handler#

The wrapper records each SQS record twice: a trailbeam receive before your handler runs, and a trailbeam outcome after it returns or throws.

import trailbeam.aws_lambda

@trailbeam.aws_lambda.wrap
def handler(event, context):
    failed = []
    for record in event["Records"]:
        try:
            process(record)
        except RetryableError:
            failed.append({"itemIdentifier": record["messageId"]})
    return {"batchItemFailures": failed}

const trailbeam = require('@trailbeam/sdk');

exports.handler = trailbeam.wrapHandler(async (event, context) => {
  const batchItemFailures = [];
  for (const record of event.Records) {
    try { await process(record); } catch (e) { batchItemFailures.push({ itemIdentifier: record.messageId }); }
  }
  return { batchItemFailures };
});

Set TRAILBEAM_OTLP_TRACES_ENDPOINT on the function (default http://localhost:4318/v1/traces) to a Collector or Trailbeam address the function can reach.

2Step 2: Know what the wrapper records#

For each record whose eventSource is aws:sqs, before the handler runs:

• one trailbeam receive with messaging.operation.name LambdaEvent, trailbeam.source lambda, trailbeam.receive_count, trailbeam.sent_timestamp_ms, trailbeam.first_receive_timestamp_ms, the receipt-handle hash, trailbeam.queue.arn (the record's eventSourceARN), faas.name (from the context, else AWS_LAMBDA_FUNCTION_NAME) and faas.invocation_id (aws_request_id / awsRequestId);
• the span is zero-length, at the moment the event arrived, with trailbeam.start_observed = false. Lambda's poller made the ReceiveMessage call, so only an upper bound is known. The screen shows "no later than" that time;
• the wrapper flushes these spans before calling your handler, so a kill cannot lose them.

After the handler, one trailbeam outcome per record, flushed again:

messaging.client.id is AWS_LAMBDA_LOG_STREAM_NAME, so each execution environment is one consumer. If that variable is not set, it is the wrapper's service.instance.id. Events with no SQS records pass through untouched.

3Step 3: Tell Trailbeam your function's timeout#

Trailbeam infers a kill when a Lambda receive has no trailbeam outcome for the same faas.invocation_id once the Lambda timeout has passed. Trailbeam's timeout defaults to 900 s (the Lambda maximum), because spans do not carry your function's real timeout.

trailbeam-store --lambda-timeout 30           # or TRAILBEAM_LAMBDA_TIMEOUT_S=30

Or per request, from 1 to 86400:

curl -s "http://127.0.0.1:4318/api/messages/$MESSAGE_ID?lambda_timeout_s=30"

The web page takes the same parameter: /message?id=...&lambda_timeout_s=30.

4Step 4: Read the timeline#

A record whose first invocation timed out (function timeout 30 s, queue visibility timeout 60 s):

The kill's window runs from the receive's end to the receive's end plus the Lambda timeout. It carries delivery and faas_invocation_id, so you can find the invocation in CloudWatch Logs. The final state is deleted_inferred.

5Step 5: Find invocations that are still open or killed#

processing means the latest delivery is a Lambda receive, inside the timeout, with no outcome. Such messages are also on the stuck list once they pass --stuck-after (300 s), because there is no success yet.

curl -s 'http://127.0.0.1:4318/api/stuck?lambda_timeout_s=30' \
  | jq '.messages[] | {message_id, state, deliveries, age_s}'

Prevent it#

Return partial batch failures instead of throwing. A throw marks every record failed_exception and retries the whole batch. A batchItemFailures entry retries only that record.

Match the wrapper to the event source mapping. If FunctionResponseTypes does not include ReportBatchItemFailures, Lambda ignores batchItemFailures, and so must Trailbeam:

@trailbeam.aws_lambda.wrap(report_batch_item_failures=False)
def handler(event, context): ...

exports.handler = trailbeam.wrapHandler(handler, { reportBatchItemFailures: false });

Budget for the flushes. Each invocation flushes twice (receives, then outcomes). Each flush waits at most the flush timeout (default 2000 ms), so an unreachable endpoint can add up to about 4 s to an invocation. Lower it if your function's timeout is tight:

exports.handler = trailbeam.wrapHandler(handler, { flushTimeoutMs: 500 });

@trailbeam.aws_lambda.wrap(flush_timeout_millis=500)
def handler(event, context): ...

Related#

• AWS Lambda setup
• Recorded and inferred steps
• Find out why a message ran twice
• Span contract: outcome