Skip to content

Receipt-handle hashing

How the hooks hash receipt handles with a shared key, how Trailbeam joins every delete and visibility change to the exact delivery, and what breaks the join.

Based on 15 repo files at 54ea61b

docs/span-contract.md, docs/span-contract-vectors.json, docs/store-api.md, server/README.md, server/src/trailbeam_store/db.py, server/src/trailbeam_store/timeline.py, server/src/trailbeam_store/otlp.py, python/src/trailbeam/_contract.py, python/src/trailbeam/_handles.py, python/src/trailbeam/boto.py, node/src/contract.js, node/src/handles.js, node/src/sqs.js, node/index.d.ts, server/tests/test_join.py

On this page

DeleteMessage and ChangeMessageVisibility take a receipt handle, not a message ID, and are often called by a different process than the one that received. Anyone holding a valid handle can delete the message, so the hooks never export it. They export a keyed hash, and Trailbeam joins each settle span to the receive with the same hash.

The hash#

text
trailbeam.receipt_handle.hmac   = lower_hex(HMAC-SHA256(key, receipt_handle))[0:32]
trailbeam.receipt_handle.key_id = lower_hex(HMAC-SHA256(key, "trailbeam key id"))[0:8]
  • Both key and receipt_handle are encoded as UTF-8. The key is the string's bytes: a hex string from openssl rand -hex 32 is used as 64 ASCII bytes, not decoded.
  • 32 hex characters is 128 bits. key_id identifies the key without revealing it.
  • Every trailbeam receive, trailbeam delete and trailbeam change_visibility carries both attributes. Trailbeam rejects any of these spans without them.

Test vectors#

From docs/span-contract-vectors.json. Both packages are tested against them.

Key Receipt handle hmac key_id
trailbeam-default-receipt-hash-key AQEB3LQoW7GQ... (long handle in the file) 5f4b52b08d8ace8e2c54214dc75bda35 b5e633f7
example-key-0123456789abcdef same handle b8cf1504a45e5fadf383d469cfa5ae3e 05769f58
clé-ünïcode-key-0001 short-handle 27aaac5fe832aeac21be5f24b4570432 2aee8344

Configure the key#

Resolution order, in both packages: explicit argument, then TRAILBEAM_RECEIPT_HASH_KEY, then the published default key.

Where Python Node.js
Client hooks trailbeam.boto.install(receipt_hash_key=...) trailbeam.install({ receiptHashKey })
Lambda wrapper @trailbeam.aws_lambda.wrap(receipt_hash_key=...) trailbeam.wrapHandler(handler, { receiptHashKey })
Environment TRAILBEAM_RECEIPT_HASH_KEY TRAILBEAM_RECEIPT_HASH_KEY
Situation Behaviour
No key anywhere (or an empty env var) Uses trailbeam-default-receipt-hash-key and warns once per process: "trailbeam: no receipt-hash key configured (TRAILBEAM_RECEIPT_HASH_KEY); using the published default key, which gives no protection."
Key shorter than 16 bytes Used, with one warning: "trailbeam: the receipt-hash key is shorter than 16 bytes"
Empty string passed explicitly Node.js treats it as not set. Python uses it as the key, with the short-key warning

Python logs on the trailbeam logger. Node.js uses process.emitWarning with code TRAILBEAM. Neither ever logs or exports the key.

shell
export TRAILBEAM_RECEIPT_HASH_KEY=$(openssl rand -hex 32)   # store it in your secret manager

Check which key a service uses#

Compute the key ID and compare it with trailbeam.receipt_handle.key_id on that service's spans.

shell
printf '%s' 'trailbeam key id' | openssl dgst -sha256 -hmac "$TRAILBEAM_RECEIPT_HASH_KEY" | awk '{print substr($NF,1,8)}'
python
import hashlib, hmac, os
print(hmac.new(os.environ["TRAILBEAM_RECEIPT_HASH_KEY"].encode(), b"trailbeam key id", hashlib.sha256).hexdigest()[:8])
javascript
const trailbeam = require('@trailbeam/sdk');
console.log(trailbeam.keyId(process.env.TRAILBEAM_RECEIPT_HASH_KEY));

The Node.js package also exports hashReceiptHandle(key, receiptHandle).

How Trailbeam joins#

How a delete is joined to its deliveryProcess A, orders-worker, receives message 7f3c and emits a Trailbeam receive span with receipt-handle hmac b8cf…ae3e and key ID 05769f58. Process B, cleanup-job, deletes the message with that receipt handle and emits a Trailbeam delete span with the same hmac and key ID but no message ID. Both spans become rows in Trailbeam's spans table, in either order. Trailbeam joins the delete row to the receive row with the same hmac and key ID, and sets the delete row's mid to 7f3c and its receive_row to that receive, so the delete belongs to that exact delivery. Hashes made with different keys never match. orders-worker, process A Trailbeam receive messaging.message.id = 7f3c… trailbeam.receipt_handle.hmac = b8cf…ae3e trailbeam.receipt_handle.key_id = 05769f58 trailbeam.receive_count = 1 cleanup-job, process B Trailbeam delete messaging.message.id (not set) trailbeam.receipt_handle.hmac = b8cf…ae3e trailbeam.receipt_handle.key_id = 05769f58 trailbeam.result = ok hash pair hash pair Trailbeam server Table spans rows arrive in any order Join same hmac and key_id; runs in both directions The delete's row mid = 7f3c… receive_row = the receive's row Hashes made with different keys never match: the timeline gets a key_mismatch notice and the delete's status is unmatched_handle.
Figure 1 A delete from a process that never saw the message ID. The hash pair is the only link between the two spans, and it is enough: Trailbeam gives the delete row the receive's message ID and points it at that exact delivery.

Settle spans (deletes and visibility changes) are matched to receives on both receipt_hmac and key_id, through the index spans_hash. The join gives two things: the message ID, and receive_row, the exact delivery whose handle was used.

  1. Settle arrives, receive stored. Trailbeam takes the matching receive. SQS issues a new handle per delivery, so there is normally one. If an emulator reuses handles, it takes the latest receive that ended at or before the settle, else the earliest after it. The row gets mid_source = 'receipt_hash'.
  2. Settle arrives first. Its row keeps its own messaging.message.id if it has one (mid_source = 'span'), else no message. When the receive arrives, Trailbeam updates every unjoined settle with that hash and key ID.
  3. Batch calls. One span per entry; each entry joins on its own. A batch delete can hold handles from several consumers.

The in-process handle map#

The client hooks also keep a map from receipt handle to message ID, filled by ReceiveMessage responses, and put messaging.message.id on settle spans when the handle is known. It holds at most 100,000 entries (oldest dropped first), expires entries after 12 hours (the longest visibility timeout), and drops an entry after a successful delete. The map is an optimisation. A process that never saw the receive (another worker, a restart, an evicted entry) sends only the hash, and the join still works.

What the join decides#

A delete's status comes from the join:

status Rule
final result: ok, and the handle belongs to the message's latest receive
stale_handle result: ok, but the handle belongs to an earlier delivery. SQS accepts old handles without always deleting, and the later receive shows it did not
failed result: failed. error_code gives the SQS code, such as ReceiptHandleIsInvalid, or the SDK error name
unmatched_handle result: ok, but no stored receive has this hash

Only a final delete makes state: deleted. The stuck list uses the same rule.

Why a hash, and why a key#

We could have joined deletes by message ID and skipped all this. But the process that deletes often never sees the message ID. Lambda's event source mapping, batch deleters and cleanup jobs only hold the handle. And when the same message is delivered twice, the message ID can't tell you which delivery was deleted. The handle can, because SQS issues a new one per receive.

The handle itself can't leave your process, because anyone holding it can delete the message. A plain hash wouldn't help much: handles are long but structured, and we didn't want to bet on that. A keyed HMAC means a stolen span database is useless without the key. That's also why the default key prints a warning. It keeps the demo working, and it protects nothing.

Our advice: generate the key once (openssl rand -hex 32), keep it in the same secret store as your queue credentials, and give the same value to every service that touches the queue.

Worked example#

Two workers. worker-a is too slow; worker-b gets the redelivery and deletes it; then worker-a deletes with its old handle.

Timeline sample
  1. 12:00:01.230Receivedworker-a · delivery 1 · hmac 9f2c…
  2. inferredVisibility timeout expiredafter delivery 1, before delivery 2
  3. 12:00:31.410Receivedworker-b · delivery 2 · hmac 41ad…
  4. 12:00:31.902Deleteworker-b · hmac 41ad… · delivery 2 · final
  5. 12:00:33.118Deleteworker-a · hmac 9f2c… · delivery 1 · stale_handle

Each delete joined to the receive with its hash, so Trailbeam knows worker-a's delete settled delivery 1 even though it came last.

Notices#

code When Fix
default_key Any span of the message has key ID b5e633f7 Set TRAILBEAM_RECEIPT_HASH_KEY. Expected in the demo
key_mismatch The message has receives, and an unjoined settle has a key ID that none of them has Use one key everywhere
unmatched_handle An unjoined settle, and either the message has no stored receive or the key IDs match The receiving service is not instrumented, or its span has not arrived

key_mismatch and unmatched_handle can only appear on a timeline if the settle span carried the message ID. A settle with neither a match nor a message ID belongs to no message; count those with store.unjoined_settles in GET /api/health.

Edge cases#

What you see Cause What to do
unjoined_settles grows steadily A deleting service uses another key, or the receiver is not instrumented Compare key IDs per service with the commands above
key_mismatch during a deploy Key rotated on some services first: receives hashed with the old key, deletes with the new Rotate the key on all services at the same time. Spans across the change do not join
A delete with delivery: null No stored receive has its hash Same as unmatched_handle
default_key in production No key configured on at least one service The default key gives no protection: anyone can recompute hashes