Bend Message Deduplication on Azure Service Bus to Your Will
Duplicates detection functionality provided by Azure Service Bus can automatically remove duplicate messages sent to a queue or topic. Deduplication is always based on the value of the MessageId property. No other property can participate in deduplication.
In the real world, message deduplication can often depend on things that are part of the message payload itself. Let's say we process orders*. Deduplication would rather be based on the order ID and not message ID. There are a few creative solutions that allow custom deduplication. For example, perform deduplication outside of ASB broker by manually inspecting message payload and marking it as a duplicate. For example, using Azure Functions and Storage tables**. While approach like this one works, it has several drawbacks:
- Unnecessary intermediate steps
- Performance decrease
- No ability to take advantage of highly optimized and performant native deduplication
What's the solution?
Use native deduplication!
Wait, but isn't native deduplication limited to solely message ID?
Glad you've asked. Absolutely. It is. Though let's look at
the MessageId property of the
BrokeredMessage. It's a read/write property,
meaning we can set it to custom values.
Custom value you said?
Let's read a bit more of the ASB documentation on deduplication.
Solved! To deduplicate order messages on
OrderId, we'll assign brokered message
MessageId property the value of
OrderId. Done. Now order messages will be
deduplicated on order IDs***.
Hold your horses! What if I need to deduplicate based on several values from a message?
Same as with order id. Combining all property values and
assigning as MessageId. Except that there might
be a size issue.
Size issue?! Yes. BrokeredMessage.MessageId is
limited to 128 characters. Would that be a deal breaker if
generated ID needs to be more than 128 characters? Not at
all. As a matter of fact, the entire payload could be used
for deduplication. Here's an example:
var payload = serializerOfYourChoice.Serialize(payloadObject);
var msg1 = new BrokeredMessage(payload);
msg1.MessageId = CreateDeterministicIdFromHash(payload);
msg1.Label = "1st";
await sender.SendAsync(msg1).ConfigureAwait(false);
var msg2 = new BrokeredMessage(payload);
msg2.MessageId = CreateDeterministicIdFromHash(payload);
msg2.Label = "2nd";
await sender.SendAsync(msg2).ConfigureAwait(false);
The sample creates a GUID like ID by making an object hash
using serialized object. For example, using JSON.Net you
could get the serialized object and pass it to
CreateGuidLikeIdFromHash to provide the
deterministic ID that is based on a hash. As result of this
snippet, there will be only one message received when a
queue has deduplication enabled.
CreateGuidLikeIdFromHash method could be
implemented in the following way:
static string CreateDeterministicIdFromHash(string input)
{
var inputBytes = Encoding.Default.GetBytes(input);
// use MD5 hash to get a 16-byte hash of the string
using (var provider = new MD5CryptoServiceProvider())
{
var hashBytes = provider.ComputeHash(inputBytes);
return new Guid(hashBytes).ToString();
}
}
[Update: as Clemens Vasters pointed out correctly, MD5, or any other cryptography hashes, should not be used for non-cryptographic purposes. Data.HashFunction library offers number of non-cryptographic hashes that can be used instead.]
Et voilà. Now you can leverage native ASB deduplication using your custom data from the message itself without unnecessary intermediaries or performance impact.
Stay deduplicated!
* as pointed out by one of the readers, order might not be the best example. Keep in mind, this is just to serve an example, not solve world's problems :)
** deduplication with Azure Functions sample by Michael Stephenson
*** RequiresDuplicateDetection needs to be set
to true along with
DuplicateDetectionHistoryTimeWindow set to the
time span duplicates detection is taking place per message