Deadlocks and Enterprise Services
The big picture - my team (Ayelet Nisan, Saar Carmi and me) is currently writing the framework for a large scale .NET project.
The problem - a few days ago while debugging our new batch framework version (a lot of working with threads, enterprise services and SWC) we have encountered a strange deadlock.
Finding the solution - being familiar with such problems our first assumption was that our applicative locks (ReaderWriterLock) was causing the problem, after a few hours we ruled it out. Our second assumption was that somehow the DB locks in conjunction with the applicative locks are causing this but while trying to debug it we saw that the actual lock is caused while the thread is creating a new instance of a class that derives from a ServicedComponent. After a few retries on different computers we were sure that we are on the problem but we could not figure what exactly is causing it. Here we stopped and did some reality check, first we know that it is always happening (sometime after 2 seconds and sometime after 30 seconds) and second (like with a lot of other locking problems) the same creation of ServicedComponent seems to have worked most of the time. To make the next major step we needed to find where exactly we are being locked inside the constructor of our ServiceComponent for that we used the debugger call stack and the Reflector tool, we were being locked inside the method Proxy.RevokeObject called from the ServicedComponentProxy.CleanupQueues and here it is
internal static void CleanupQueues(bool bGit)
{
object obj1;
if (!ServicedComponentProxy._asyncFinalizeEnabled)
{
return;
}
if (bGit && (ServicedComponentProxy._gitQueue.Count > 0))
{
obj1 = ServicedComponentProxy._gitQueue.Pop();
if (obj1 != null)
{
int num1 = (int) obj1;
try
{
Proxy.RevokeObject(num1);
}
catch
{
}
}
}
if (ServicedComponentProxy._ctxQueue.Count <= 0)
{
return;
}
obj1 = ServicedComponentProxy._ctxQueue.Pop();
if (obj1 == null)
{
return;
}
if (!Util.ExtendedLifetime)
{
ContextTrackerContainer container1 = (ContextTrackerContainer) obj1;
container1.Release();
}
else
{
ServicedComponentProxy proxy1 = (ServicedComponentProxy) obj1;
proxy1.SendDestructionEvents(false);
proxy1.ReleaseContext();
}
}
The next thing was to try and googlize our already well know methods\problem, there are not so many people that have encounter our problem further more none of them gave us a real solution to our problem.
Finally searching in the MSDN we found a clue to our problem while using GIT keyword which is mentioned a couple of times in code. The clue was in the form of KB298014 FIX: COM+ application that uses the Global Interface Table (GIT) may deadlock, following the instructions in the KB and adding two registry keys fixed our problem(the dispose solution did not seemed to be working to us).
Yet another last look at the first lines of code in the previous method
if (!ServicedComponentProxy._asyncFinalizeEnabled)
{
return;
}
seems magically linked to the registry key we just added DisableAsyncFinalization
Oh yes one last important thing, the problem was only reproduced on XP sp2 machines while not in windows 2003 servers.
.