.NET Garbage Collector and a Volatile Trading Market
It looks like we are going to have to try this fix from Microsoft. One of our servers is experiencing strange behavior between 9-10am and 3-4pm when the market is at its most volatile - and we have to process an obscene number of messages of the message bus. The symptoms we are seeing are that the .NET application runs fine outside these hours with a 10% Time in GC, and a fairly constant memory usage of 100MB. However during the volatile period, it appears as if the GC just gives up, and we start seeing the memory climb all the way up to 800MB-1.3G, at which point we blow with out of memory exceptions. We suspect this is primarily due to most message we get of the message bus as 20K in size, and thus are immediately allocated in the large object heap. Based on the work we did over the last few weeks, we are fairly sure that we are not leaking in the unmanaged code, and in the managed code, we are releasing references to object ASAP. Looks like Santomania had a similar issue.
We have also begun testing the svr GC. First signs appear to be that we spend less % Time in GC, but the memory footprint for the process is initially higher than that of the wks GC. Over time, the memory footprint appears to be pretty similar - wks vs svr.
Thanks to feedback by Nat, we will probably look at custom serialization of DataTable's
Update It appears the memory issue we were seeing above was related to running the wks GC, with the "GC Collection 2" appearing to exceed a certain threshold where it just kept on getting bigger. The Microsoft hotfix didn't even solve the problem, but moving to svr GC did. So far the svr GC is doing an excellent job, and we are constant between 95-100MB, even though the volatile market of the morning, and %Time in GC is averaging 9.
To answer Brad More's question (see feedback), we are using an custom ATL service that looks in the registry and retrieves the appropriate configuration information, and then creates the svr GC. Installation into production is always a nightmare since developers are never allowed near the boxes. However, the production support team are usually quite good, and since deploying a .NET application is just an xcopy, the only other part is to install the service, which they do via a script or from the command line, similar to this:
myService.exe -i <assemblyClass> <assembly> <configuraton location> ...
On installation of the service, all the configuration parmeters are written to the registry for use when the service is started