Azure Service Bus Message: Wanted Dead or Alive

Running NServiceBus on Azure sometimes can be challenging. Take for example the Azure Service Bus transport. Every queue has additional queues that could contain either dead-lettered messages as a result of repeated failing processing of the poisonous messages or unsuccessful transfer.

With multiple endpoints and their queues, you want to be able to monitor your system and know when things are going south. Particular Platform offers a monitoring tool, Service Control that is designed specifically for this purpose. Except it monitors endpoints for successfully processed and failed processing messages. Not quite the whole story for the ASB’s dead-letter queues, isn’t it?

Gladly, there’s an option of Custom Checks. These checks allow periodic execution of certain tests and can report results back to the mothership, SP dashboard.

To implement a custom check, Custom Checks NuGet package needs to be a referenced. For NServiceBus version 6, the package is ServiceControl.Plugin.Nsb6.CustomChecks. With the package in place, plugin requires ServiceControl input queue.

endpointConfiguration.CustomCheckPlugin("particular.servicecontrol");

And the check class itself:

public class MonitorDeadletterQueue : CustomCheck
{
    NamespaceManager namespaceManager;
    const string endpointName = "Samples.Azure.ServiceBus.Endpoint2";

    public MonitorDeadletterQueue() : base(id: $"Monitor {endpointName} DLQ", category: "Monitor DLQ", repeatAfter: TimeSpan.FromSeconds(10))
    {
        var connectionString = Environment.GetEnvironmentVariable("AzureServiceBus.ConnectionString");
        namespaceManager = NamespaceManager.CreateFromConnectionString(connectionString);
    }

    public override async Task<CheckResult> PerformCheck()
    {
        var queueDescription = await namespaceManager.GetQueueAsync(endpointName).ConfigureAwait(false);
        var messageCountDetails = queueDescription.MessageCountDetails;

        if (messageCountDetails.DeadLetterMessageCount > 0)
        {
            return CheckResult.Failed($"{messageCountDetails.DeadLetterMessageCount} dead-lettered messages in queue {endpointName}.");
        }

        return CheckResult.Pass;
    }
}

Once implemented, DLQ custom check periodically executes and provides the status. As long as there are no dead-lettered messages, there will be no alerts.

However, the moment there are dead-lettered messages, the dashboard will light up.

In this scenario, there was indeed a dead-lettered message in the queue.

Once inspected and addressed, the message can be removed from the DLQ, and SP will go back to normal.

Or at least till the next dead-lettered message 😊 Now you can track down those dead-lettered villains an pick up your bounty.