Receipt-handle hashing
How the hooks hash receipt handles with a shared key, how Trailbeam joins every delete and visibility change to the exact delivery, and what breaks the join.
DeleteMessage and ChangeMessageVisibility take a receipt handle, not a message ID, and are often called by a different process than the one that received. Anyone holding a valid handle can delete the message, so the hooks never export it. They export a keyed hash, and Trailbeam joins each settle span to the receive with the same hash.
The hash#
trailbeam.receipt_handle.hmac = lower_hex(HMAC-SHA256(key, receipt_handle))[0:32]
trailbeam.receipt_handle.key_id = lower_hex(HMAC-SHA256(key, "trailbeam key id"))[0:8]- Both
keyandreceipt_handleare encoded as UTF-8. The key is the string's bytes: a hex string fromopenssl rand -hex 32is used as 64 ASCII bytes, not decoded. - 32 hex characters is 128 bits.
key_ididentifies the key without revealing it. - Every
trailbeam receive,trailbeam deleteandtrailbeam change_visibilitycarries both attributes. Trailbeam rejects any of these spans without them.
Test vectors#
From docs/span-contract-vectors.json. Both packages are tested against them.
| Key | Receipt handle | hmac |
key_id |
|---|---|---|---|
trailbeam-default-receipt-hash-key |
AQEB3LQoW7GQ... (long handle in the file) |
5f4b52b08d8ace8e2c54214dc75bda35 |
b5e633f7 |
example-key-0123456789abcdef |
same handle | b8cf1504a45e5fadf383d469cfa5ae3e |
05769f58 |
clé-ünïcode-key-0001 |
short-handle |
27aaac5fe832aeac21be5f24b4570432 |
2aee8344 |
Configure the key#
Resolution order, in both packages: explicit argument, then TRAILBEAM_RECEIPT_HASH_KEY, then the published default key.
| Where | Python | Node.js |
|---|---|---|
| Client hooks | trailbeam.boto.install(receipt_hash_key=...) |
trailbeam.install({ receiptHashKey }) |
| Lambda wrapper | @trailbeam.aws_lambda.wrap(receipt_hash_key=...) |
trailbeam.wrapHandler(handler, { receiptHashKey }) |
| Environment | TRAILBEAM_RECEIPT_HASH_KEY |
TRAILBEAM_RECEIPT_HASH_KEY |
| Situation | Behaviour |
|---|---|
| No key anywhere (or an empty env var) | Uses trailbeam-default-receipt-hash-key and warns once per process: "trailbeam: no receipt-hash key configured (TRAILBEAM_RECEIPT_HASH_KEY); using the published default key, which gives no protection." |
| Key shorter than 16 bytes | Used, with one warning: "trailbeam: the receipt-hash key is shorter than 16 bytes" |
| Empty string passed explicitly | Node.js treats it as not set. Python uses it as the key, with the short-key warning |
Python logs on the trailbeam logger. Node.js uses process.emitWarning with code TRAILBEAM. Neither ever logs or exports the key.
export TRAILBEAM_RECEIPT_HASH_KEY=$(openssl rand -hex 32) # store it in your secret managerCheck which key a service uses#
Compute the key ID and compare it with trailbeam.receipt_handle.key_id on that service's spans.
printf '%s' 'trailbeam key id' | openssl dgst -sha256 -hmac "$TRAILBEAM_RECEIPT_HASH_KEY" | awk '{print substr($NF,1,8)}'import hashlib, hmac, os
print(hmac.new(os.environ["TRAILBEAM_RECEIPT_HASH_KEY"].encode(), b"trailbeam key id", hashlib.sha256).hexdigest()[:8])const trailbeam = require('@trailbeam/sdk');
console.log(trailbeam.keyId(process.env.TRAILBEAM_RECEIPT_HASH_KEY));The Node.js package also exports hashReceiptHandle(key, receiptHandle).
How Trailbeam joins#
Settle spans (deletes and visibility changes) are matched to receives on both receipt_hmac and key_id, through the index spans_hash. The join gives two things: the message ID, and receive_row, the exact delivery whose handle was used.
- Settle arrives, receive stored. Trailbeam takes the matching receive. SQS issues a new handle per delivery, so there is normally one. If an emulator reuses handles, it takes the latest receive that ended at or before the settle, else the earliest after it. The row gets
mid_source = 'receipt_hash'. - Settle arrives first. Its row keeps its own
messaging.message.idif it has one (mid_source = 'span'), else no message. When the receive arrives, Trailbeam updates every unjoined settle with that hash and key ID. - Batch calls. One span per entry; each entry joins on its own. A batch delete can hold handles from several consumers.
The in-process handle map#
The client hooks also keep a map from receipt handle to message ID, filled by ReceiveMessage responses, and put messaging.message.id on settle spans when the handle is known. It holds at most 100,000 entries (oldest dropped first), expires entries after 12 hours (the longest visibility timeout), and drops an entry after a successful delete. The map is an optimisation. A process that never saw the receive (another worker, a restart, an evicted entry) sends only the hash, and the join still works.
What the join decides#
A delete's status comes from the join:
status |
Rule |
|---|---|
| final | result: ok, and the handle belongs to the message's latest receive |
| stale_handle | result: ok, but the handle belongs to an earlier delivery. SQS accepts old handles without always deleting, and the later receive shows it did not |
| failed | result: failed. error_code gives the SQS code, such as ReceiptHandleIsInvalid, or the SDK error name |
| unmatched_handle | result: ok, but no stored receive has this hash |
Only a final delete makes state: deleted. The stuck list uses the same rule.
Why a hash, and why a key#
We could have joined deletes by message ID and skipped all this. But the process that deletes often never sees the message ID. Lambda's event source mapping, batch deleters and cleanup jobs only hold the handle. And when the same message is delivered twice, the message ID can't tell you which delivery was deleted. The handle can, because SQS issues a new one per receive.
The handle itself can't leave your process, because anyone holding it can delete the message. A plain hash wouldn't help much: handles are long but structured, and we didn't want to bet on that. A keyed HMAC means a stolen span database is useless without the key. That's also why the default key prints a warning. It keeps the demo working, and it protects nothing.
Our advice: generate the key once (openssl rand -hex 32), keep it in the same secret store as your queue credentials, and give the same value to every service that touches the queue.
Worked example#
Two workers. worker-a is too slow; worker-b gets the redelivery and deletes it; then worker-a deletes with its old handle.
- 12:00:01.230Receivedworker-a · delivery 1 · hmac 9f2c…
- inferredVisibility timeout expiredafter delivery 1, before delivery 2
- 12:00:31.410Receivedworker-b · delivery 2 · hmac 41ad…
- 12:00:31.902Deleteworker-b · hmac 41ad… · delivery 2 · final
- 12:00:33.118Deleteworker-a · hmac 9f2c… · delivery 1 · stale_handle
Each delete joined to the receive with its hash, so Trailbeam knows worker-a's delete settled delivery 1 even though it came last.
Notices#
code |
When | Fix |
|---|---|---|
| default_key | Any span of the message has key ID b5e633f7 |
Set TRAILBEAM_RECEIPT_HASH_KEY. Expected in the demo |
| key_mismatch | The message has receives, and an unjoined settle has a key ID that none of them has | Use one key everywhere |
| unmatched_handle | An unjoined settle, and either the message has no stored receive or the key IDs match | The receiving service is not instrumented, or its span has not arrived |
key_mismatch and unmatched_handle can only appear on a timeline if the settle span carried the message ID. A settle with neither a match nor a message ID belongs to no message; count those with store.unjoined_settles in GET /api/health.
Edge cases#
| What you see | Cause | What to do |
|---|---|---|
unjoined_settles grows steadily |
A deleting service uses another key, or the receiver is not instrumented | Compare key IDs per service with the commands above |
key_mismatch during a deploy |
Key rotated on some services first: receives hashed with the old key, deletes with the new | Rotate the key on all services at the same time. Spans across the change do not join |
A delete with delivery: null |
No stored receive has its hash | Same as unmatched_handle |
default_key in production |
No key configured on at least one service | The default key gives no protection: anyone can recompute hashes |
Related#
Change visibility
ChangeMessageVisibility: sets a new visibility timeout on an in-flight message, counted from the call. A value of 0 makes it visible at once.Amazon SQS terms in Vocabulary
Receipt handle
Amazon SQS terms in Vocabulary
Message ID
MessageId, assigned by SQS when the message is sent. It stays the same on every receive, including a receive from a dead-letter queue. Trailbeam joins everything about a message by this ID.Amazon SQS terms in Vocabulary
Message
Amazon SQS terms in Vocabulary
Hooks
trailbeam.boto.install()), the AWS SDK v3 middleware (trailbeam.instrument(client)) and the Lambda wrapper.Settle span
trailbeam delete or trailbeam change_visibility span: a call that settles what happens to an in-flight message.Receive
ReceiveMessage call returning a message. A receive does not remove the message; it starts its visibility timeout.Amazon SQS terms in Vocabulary
Span
OpenTelemetry terms in Vocabulary
Lambda wrapper
trailbeam.aws_lambda.wrap (Python) or trailbeam.wrapHandler (Node.js), around a handler invoked by an event source mapping.Receipt-hash key
TRAILBEAM_RECEIPT_HASH_KEY. Every service on the same queues must use the same key.Queue
Amazon SQS terms in Vocabulary
Delivery
Consumer
ReceiveMessage) and then deletes them. A Lambda function behind an event source mapping is a consumer too.Amazon SQS terms in Vocabulary
Visibility timeout
Amazon SQS terms in Vocabulary
Delete
DeleteMessage, made with a receipt handle. This is the only way a consumer removes a message. With the handle of an older receive, the call can still succeed while the message is received again.Amazon SQS terms in Vocabulary
Event source mapping
AWS Lambda terms in Vocabulary
Timeline