February 2004 - Posts
Trading applications usually consist of a some/all of the following: Trade Entry, Pricing, Position Management, Trade Matching, Confirmation and Settlement. Pricing is often the most computational intensive due to the calculations needs to generate a value (price).
Option pricing as shown in this
article often involes a considerable number of inputs (underlying value, Greeks, etc) to calculate the value of an option. Often this data will come from multiple internal applications via various types of message bus (due to legacy reasons). Due to the asynchronous nature of the input data, pricing servers often have to maintain in-memory caches of all the data from various feeds, and re-calcuate the price on a change of any one input. This leads to .NET trading servers having a different
allocation profile to maybe an ASP.NET application.
Indigo will offer a cleaner approach to
data delivering (sample source code) than .NET Remoting/DCOM offered in the object reference passing world.
Excellent error message from WsdlGen:
Microsoft (R) Web Services Description Language Generator V2
[Microsoft (R) .NET Framework, Version 1.2.30703.27]
Copyright (C) Microsoft Corporation 1998-2003. All rights reser
Error: 'this' type can not be an interface itself.
For help with command line options, please type "WsdlGen /?".
Only problem is, it doesn't really tell you the problem, the problem in this case being an interfaced inheriting from IDatagramPortTypeChannel instead of IDialogPortTypeChannel
We have two .NET trading application servers that use XmlSerializer, mostly because the market data we retrieve of the internal message bus is in XML format. Unfortunately it appears as if XmlSerializer was possibly not the best tool for the job, even with its own internal cache. We are currently processing 1000-1500 msgs/sec, which we estimate will raise to 5500-8000 msgs/sec over the next few months. However, its currently taking us on average 5mns to Deserialize() a message, which is to long - we really need it sub-millisecond.
Heartbeats Yesterday we had an unfortunate issue where a trader's UI somehow lost its .NET remoting connection to a server, and hence their view of the market was stale - in the worst case this can cause a position to go south. We resolved the issue by leveraging some existing heartbeat infrastructure, which allowed us to notify the trader when their screen had not received any messages for a certain period (implying a loss of connection), and allowing a reconnection and refresh of all his data on-screen.
Svr GC Our GC issues seem to have been resolved by the svr GC. Today we survived the morning volatile market, with memory remaining fairly constant - between 175-180MB. Remoting calls were stable (203,000 for the entire trading day), and we have an 8-10% Time in GC. The box is an IBM 8-CPU Intel Pentium 4 2.4 Mhz (Hyper-Thread) with 8G RAM. When running the wks GC on the same hardware, we saw a high usage of Gen 2, whereas with svr GC, it's the reverse. Gen 0 for the svr GC appears to run at 10Mb, whereas for the wks GC it was generally smaller. Gen 1 appears to be the same for wks and svr GC.
It looks like we are going to have to try this fix from Microsoft. One of our servers is experiencing strange behavior between 9-10am and 3-4pm when the market is at its most volatile - and we have to process an obscene number of messages of the message bus. The symptoms we are seeing are that the .NET application runs fine outside these hours with a 10% Time in GC, and a fairly constant memory usage of 100MB. However during the volatile period, it appears as if the GC just gives up, and we start seeing the memory climb all the way up to 800MB-1.3G, at which point we blow with out of memory exceptions. We suspect this is primarily due to most message we get of the message bus as 20K in size, and thus are immediately allocated in the large object heap. Based on the work we did over the last few weeks, we are fairly sure that we are not leaking in the unmanaged code, and in the managed code, we are releasing references to object ASAP. Looks like Santomania had a similar issue.
We have also begun testing the svr GC. First signs appear to be that we spend less % Time in GC, but the memory footprint for the process is initially higher than that of the wks GC. Over time, the memory footprint appears to be pretty similar - wks vs svr.
Thanks to feedback by Nat, we will probably look at custom serialization of DataTable's
Update It appears the memory issue we were seeing above was related to running the wks GC, with the "GC Collection 2" appearing to exceed a certain threshold where it just kept on getting bigger. The Microsoft hotfix didn't even solve the problem, but moving to svr GC did. So far the svr GC is doing an excellent job, and we are constant between 95-100MB, even though the volatile market of the morning, and %Time in GC is averaging 9.
To answer Brad More's question (see feedback), we are using an custom ATL service that looks in the registry and retrieves the appropriate configuration information, and then creates the svr GC. Installation into production is always a nightmare since developers are never allowed near the boxes. However, the production support team are usually quite good, and since deploying a .NET application is just an xcopy, the only other part is to install the service, which they do via a script or from the command line, similar to this:
myService.exe -i <assemblyClass> <assembly> <configuraton location> ...
On installation of the service, all the configuration parmeters are written to the registry for use when the service is started
Possibly the next book to buy
Our .NET services were using rather a lot of memory over the last few days, we tracked part of the problem down to a missing "delete" in a shared unmanaged C++ assembly. We were also leaking memory due to a missing System::Runtime::InteropService::Marshal::FreeHGlobal call for a char*
Indigo: My guess is we will see the next code drop of Indigo early next year. SOA offers an improvement over passing object references. Ryan Dawson has an interesting article on how Indigo could be used on Wall St. I really hope Indigo performs well given that its based on SOAP, as I have said many times in this blog, performance is key for Wall St. applications.
Subscribe: Financial Technology blog
Hardware: Sometimes, due to the RAD nature of trading projects, things get missed of the project plan. Today its lack of hardware - simulating a live market requires a certain amount of CPU power.
GC Update: Due to a few performance/memory leak issues we've had recently, we haven't yet had time to try out the svr GC. Hopefully, we can schedule a test of the svr GC in the next few days.
Thanks to Patrick for the solution to adding multi-instance and single-instance counters in the same category - you can't.
On the work front, the last two week's performance related work appears to be slowly paying off. We should know for sure by Tuesday/Wednesday, but so far today we have seen a reduction in CPU usage from 40-60% last week to 15-20% this week, even with a 2.5x increase in load on our servers.
A low context switch rate is less then 5,000 context switches per second, per processor. So, an 8-processor server with an overall context switch rate of 1,500 context switches per second would qualify as having a low context switch rate.
Todays useless piece of code if you to want to know the number of processors you have available:
[StructLayout(LayoutKind.Sequential)]
public struct SYSTEM_INFO
{
public uint dwOemId;
public uint dwPageSize;
public uint lpMinimumApplicationAddress;
public uint lpMaximumApplicationAddress;
public uint dwActiveProcessorMask;
public uint dwNumberOfProcessors;
public uint dwProcessorType;
public uint dwAllocationGranularity;
public uint dwProcessorLevel;
public uint dwProcessorRevision;
}
[DllImport("kernel32")]
static extern void GetSystemInfo(ref SYSTEM_INFO pSI);
SYSTEM_INFO pSI = new SYSTEM_INFO();
GetSystemInfo(ref pSI);
Console.WriteLine("Number of CPU's:" + pSI.dwNumberOfProcessors);
Since I had this problem a long time ago, and keep getting asked about it, I'll blog the answer. We originally got this problem when we had some C# code in an AppDomain, other than the default AppDomain, call some unmanaged code (C++). The unmanaged code would then perform a callback into C# at sometime in the future.
First you need to be aware of the Mixed DLL Loading Problem and linker warnings you might have seen when compiling unmanaged code.
The System.DllNotFoundException will not normally occur in the above scenario if the default AppDomain is used. This is because when the unmanaged code makes a callback, it has no idea about the managed world (and AppDomain's), and so unless otherwise told, will invoke the callback into the default AppDomain. Hence the problem, if your original C# call to the unmanaged world was via another AppDomain, then the callback will be incorrect. The simplest way to solve the problem is to compile you unmanaged code with the Visual C++ compiler option "/clr:initialAppDomain". This should cause all calls from unmanaged threads to call back into the AppDomain that loaded the unmanaged code
If a server has a MarshalByRefObject object that was create and return to a client, and the client drops its reference to the object, why does remoting still keep a hold of the object on the server? .NET Memory Profiler says one of my objects is held by ServerIdentity in the System.Runtime.Remoting namespace.
On another note this .NET Memory Profiler gives a different view of the world to this one
Marshalling DateTables is expensive, even with the binary formatter. One idea that appears to perform better for us is to do the following:
DateSet ds = new DataSet();
ds.Tables.Add(table)
ds.WriteXML(stream, System.Data.XmlWriteMode.WriteSchema)
Then take the XML, compress it, send it to the client, and reverse the process.
Finally, calling Clear() prior to Dispose() on a DataTable makes a big different to the memory usage. For us, it was making a 5Mb difference to the memory footprint. We also have this SqlClient memory leak
I hadn't come across
Modelling in Colour until a a colleague sent me
these links today.
More Posts
Next page »