Azure Service Bus Message: Wanted Dead or Alive

enter image description here

Running NServiceBus on Azure sometimes can be challenging. Take for example the Azure Service Bus transport. Every queue has additional queues that could contain either dead-lettered messages as a result of repeated failing processing of the poisonous messages or unsuccessful transfer.

With multiple endpoints and their queues, you want to be able to monitor your system and know when things are going south. Particular Platform offers a monitoring tool, Service Control that is designed specifically for this purpose. Except it monitors endpoints for successfully processed and failed processing messages. Not quite the whole story for the ASB’s dead-letter queues, isn’t it?

Gladly, there’s an option of Custom Checks. These checks allow periodic execution of certain tests and can report results back to the mothership, SP dashboard.

To implement a custom check, Custom Checks NuGet package needs to be a referenced. For NServiceBus version 6, the package is ServiceControl.Plugin.Nsb6.CustomChecks. With the package in place, plugin requires ServiceControl input queue.

endpointConfiguration.CustomCheckPlugin("particular.servicecontrol");

And the check class itself:

public class MonitorDeadletterQueue : CustomCheck
{
    NamespaceManager namespaceManager;
    const string endpointName = "Samples.Azure.ServiceBus.Endpoint2";

    public MonitorDeadletterQueue() : base(id: $"Monitor {endpointName} DLQ", category: "Monitor DLQ", repeatAfter: TimeSpan.FromSeconds(10))
    {
        var connectionString = Environment.GetEnvironmentVariable("AzureServiceBus.ConnectionString");
        namespaceManager = NamespaceManager.CreateFromConnectionString(connectionString);
    }

    public override async Task<CheckResult> PerformCheck()
    {
        var queueDescription = await namespaceManager.GetQueueAsync(endpointName).ConfigureAwait(false);
        var messageCountDetails = queueDescription.MessageCountDetails;

        if (messageCountDetails.DeadLetterMessageCount > 0)
        {
            return CheckResult.Failed($"{messageCountDetails.DeadLetterMessageCount} dead-lettered messages in queue {endpointName}.");
        }

        return CheckResult.Pass;
    }
}

Once implemented, DLQ custom check periodically executes and provides the status. As long as there are no dead-lettered messages, there will be no alerts.

enter image description here

However, the moment there are dead-lettered messages, the dashboard will light up.

enter image description here

In this scenario, there was indeed a dead-lettered message in the queue.

enter image description here

Once inspected and addressed, the message can be removed from the DLQ, and SP will go back to normal.

enter image description here

Or at least till the next dead-lettered message 😊 Now you can track down those dead-lettered villains an pick up your bounty.

4 Comments

  • I wrote something similar vor topics and subscriptions and this is the query

    namespaceManager
    .GetTopics()
    .SelectMany(t => namespaceManager.GetSubscriptions(t.Path), (topic, subscription) => new { topic, subscription })
    .Where(t => t.subscription.MessageCountDetails.DeadLetterMessageCount != 0)
    .Select(x => $"[T:{x.topic.Path} S:{x.subscription.Name}={x.subscription.MessageCountDetails.DeadLetterMessageCount}]");

  • @Joey,
    Indeed, that would be similar :)

  • Hi Sean, I attended and loved your session about Azure Bus Messaging Messaging at Techorama 2017 and was wondering if I could have a copy of your code samples, maybe at GitHub? If not, please provide or mail them to me at Jim.DeDonder@Spikes.be Thanks!

  • Thank you @Jim.
    You'll find the deck and the demos at https://github.com/SeanFeldman/Techorama2017

Add a Comment

As it will appear on the website

Not displayed

Your website