Creating high performance WCF services

I had a WCF service where I wanted to be able to support over a hundred concurrent users, and while most of the service methods had small payloads which returned quickly, the startup sequence needed to pull down 200,000 records. The out of the box WCF service had no ability to support this scenario, but with some effort I was able to squeeze orders of magnitude performance increases out of the service and hit the performance goal.

Initially performance was abysmal and there was talk of ditching WCF entirely ( and as the one pushing WCF technology on the project this didn't seem like a career enhancing change )

 

Here's how performance was optimized. These are listed in the order they were implemented. Some are fairly obvious, others took some time to discover.  Each item represents, a significant increase in latency or scalability from the prior - and although I have internal measurement numbers, I'm not comfortable publishing them as the size of the data increased, and the testing approach changed.

  1. Use NetTCP binding
    This helps both throughput and the time it takes to open and close connections
  2. Use DataContract Serializer instead of XMLSerializer
    I started out using DataTables - POCO objects via Linq2Sql yielded a 6x increase
    slow: [OperationContract] MyDataTable GetData(...);
    fast: [OperationContract] IEnumerable<MyData> GetData(...);

  3. Unthrottle your service
    It's quite understanable that WCF is resistant to Denial of Service attacks out of the box, but it's too bad that it's is such a manual operation to hit the "turbo button". It would be nice if the Visual Studio tooling did this for you, or at least had some guidance (MS - hint, hint)

    The items to look at here are:
    1. <serviceBehaviors><serviceThrottling ...> set the max values high
    2. <dataContractSerializer maxItemsInObjectGraph="2147483647" />
    3. and under <netTcpBinding> setting the listenBacklog, maxConnections, and maxBuffer* value high
  4. Cache your data
    WCF, unlike ASP.Net has no built in facility to cache service responses, so you need to do it by hand. Any cache class will do.
  5. Normalize/compress your data
    this doesn't necessarily have to be done in the database, the Linq GroupBy operators make this easy to do in code. To clarify, say your data is kept in a denormalized table
    string Key1
    string Key2
    string Key3
    int val1
    int val2

    the bulk of the result set ends up being duplicate data
    LongKeyVal1 LongKeyVal2 LongKeyVal3 10 12
    LongKeyVal1 LongKeyVal2 LongKeyVal3 11 122
    LongKeyVal1 LongKeyVal2 LongKeyVal3 12 212
    so normalize this into
    LongKeyVal1 LongKeyVal2 LongKeyVal3
    10 12
    11 122
    12 212

    In code, given the following classes

    public class MyDataDenormalized
    {
        public string Key1 { get; set; }
        public string Key2 { get; set; }
        public string Key3 { get; set; }
        public int Val1 { get; set; }
        public int Val2 { get; set; }
    }
    public class MyDataGroup
    {
        public string Key1 { get; set; }
        public string Key2 { get; set; }
        public string Key3 { get; set; }
        public MyDataItem[] Values { get; set; }
    }
    public class MyDataItem
    {
        public int Val1 { get; set; }
        public int Val2 { get; set; }
    }

    you can transform an IEnumerable<MyDataDenormalized> into a IEnumerable<MyDataGroup> via the following

    var keyed = from sourceItem in source
               group sourceItem by new
               {
                   sourceItem.Key1,
                   sourceItem.Key2,
                   sourceItem.Key3,
               } into g
               select g;
    var groupedList = from kItems in keyed
                  let newValues = (from sourceItem in kItems select new MyDataItem() { Val1 = sourceItem.Val1, Val2= sourceItem.Val2 }).ToArray()
                  select new MyDataGroup()
                  {
                      Key1 = kItems.Key.Key1,
                      Key2 = kItems.Key.Key2,
                      Key3 = kItems.Key.Key3,
                      Values = newValues,
                  };
  6. Use the BinaryFormatter, and cache your serializations
    If you're willing to forgo over the wire type safety, the binary formatter is the way to go for scalability. Data caching has only a limited impact if a significant amount of CPU time is spent serializing it - which is exactly what happens with the DataContract serializer.

    The operation contract changes to
  7. [OperationContract]
    Byte[] GetData(...);

    and the implementation to

    var bf = new BinaryFormatter();
    using (var ms = new MemoryStream())
    {
        bf.Serialize(ms, groupeList);
    // and best to cache it too return ms.GetBuffer(); }

     

Before items 4,5, and 6 the service would max out at about 50 clients ( response time to go way up and CPU usage would hit 80% - on a 8 core box). After these changes were made, the service could handle of 100 + clients and CPU usage flattened out at 30%

Update: Shay Jacoby has reasonably suggested I show some code.

Update2: Brett asks about relative impact. Here's a summary

item latency scalability
2) DataContract Serializer large large
3) unthrottle small large
4) cache data small  
5) normalize data medium  
6) cache serialization small large

 

kick it on DotNetKicks.com
Published Saturday, January 3, 2009 8:14 PM by Scott Weinstein

Comments

Sunday, January 4, 2009 2:37 AM by Shay Jacoby

# re: Creating high performance WCF services

Thanks.

It could be nice if You show some code examples.

Sunday, January 4, 2009 8:32 AM by Brett

# re: Creating high performance WCF services

Scott, great post, but I've a couple comments and questions.

2.  I don't believe returning DataTable's from a web service was never a recommended practice with WCF. (which is why you had to use XmlSerializer)

4.  Often a good idea.

Did you do any testing to see if items 2 and 4 would have given you acceptable performance, and eliminated the rest of your headaches?

In what order did you perform these optimzations (in the order listed?)?

Do you have any feel for the relative impact of each optimization?

It seems Marc has some additional info not mentioned in the post (re: PSS).

I agree that this shouldn't have been so difficult to achieve, and agree with Marc that dox/guidance is probably the easiest solution.  I believe this issue is caused because WCF is so "big".  There are many many different ways to accomplish things, which means we devs are going to try all of them!

Sunday, January 4, 2009 12:18 PM by marc

# re: Creating high performance WCF services

Brett,

>> slow: [OperationContract] MyDataTable GetData(...);

>> I don't believe returning DataTable's from a web service was never a recommended practice with WCF. (which is why you had to use XmlSerializer)

I know that Scott is not a Resharper user ... but these are the types of things that a look like Resharper or CodeRush can warn you against. I am not a Resharper wiz (only a mere end-user), but I would guess that you could set up a Resharper or FxCop rule that warns about possible performance hits from a certain line of code.

One would also like to see this type of warning built into Visual Studio.

Sunday, January 4, 2009 3:45 PM by Ramon Smits

# re: Creating high performance WCF services

WCF is all about contracts and your recommendation (6) is about tweaking your contract too. This is hardly a good recommendation as why use WCF when you don't need the contract stuff at all? Seems to me that your then much better of with a custom binary serializer if you really need that kind of throughput.

Sunday, January 4, 2009 5:53 PM by Brett

# re: Creating high performance WCF services

marc,

It sure *sounds* like a feature that one of those should have!  It probably belongs more as an FxCop rule than a R# hint; mainly because it has no idea what you should change it to. An FxCop rule would flag a violation that you shouldn't return a DataSet or DataTable from a WCF service (or any class in the .svc file maybe?).  FxCop rules are an easy way for devs to learn good rules, even if you ignore most of them, eventually you'll write new code that won't cause those violations.

Sunday, January 11, 2009 12:30 PM by Mark Pollack

# re: Creating high performance WCF services

@alexandrul

You can use the ASP.NET cache outside web apps, so that is a good place to start.  

Moving up in sophistication, for the Spring.NET framework we've made a simple abstraction for caching, the ICache interface, as well as some AOP advice for caching.  worth a quick browse, but fundamentally, adding caching is pretty easy, just check if the key is present, if yes, return value from the cache, otherwise insert into the cache and return.

BTW, I've run into similar issues in the finance industry when using plain JMS messaging.  To get good performance with that middleware (TIBCO) caching, serialization (nested map messages), and compression, were the way to get acceptable performance.  I don't mean one shouldn't use WCF, just that you are going to have the same issues in any middleware stack.

Mark

Tuesday, January 20, 2009 6:43 PM by Luke Puplett

# re: Creating high performance WCF services

I'd be interested to hear the impact of the instance mode, i.e. using a singleton and synchronisation in the service.

Also, hosting outside of IIS and sliding window caching on the client as well as the server.

I've also heard that extensive caching can lead to huge >30s garbage collections on gigs of cached data expiring. I think this was on a Channel9 interview. Anyway, MS added a GC notification system to the framework (for this customer only) so their app server could take itself offline while the GC occured to ensure that no trades were held up when it struck.

Have you also experimented with the throttling settings?

Sorry, this is an important subject and I think the comments here prove there's high demand for performance tuning WCF.

Thanks for the post,

Luke

Tuesday, January 20, 2009 9:30 PM by Scott Weinstein

# re: Creating high performance WCF services

@Sandesh,

You can consume WCF with any SOAP toolkit. As for ASP.Net, last time I looked into it you needed IIS. Unless you want to consider mono

Tuesday, January 20, 2009 9:35 PM by Scott Weinstein

# re: Creating high performance WCF services

@Luke,

I used multiple instancing mode, haven’t done any testing with singleton mode, but can’t see how that would improve responsiveness.

In regards to caching, large objects, and the GC – I haven’t run into any issues along those lines, but for now the dirty little secret is that I don’t expire any objects in cache, I just take advantage of the 4 hour/week greenzone and restart the process. Something to keep in mind however.

Throttling – yes – turned all the knobs to 11.

Wednesday, January 21, 2009 6:33 AM by Luke Puplett

# re: Creating high performance WCF services

@Scott

Thanks for the prompt repsonse and your honesty. Sorry for asking about throttling when it was in the article - I was tired ;)

If you have enough ram, and you're using speedy hashtable based caching containers, then yeah, why not wait until the service window to tear down in a controlled way. It depends on personal situation I guess.

As for the context mode, my experience is drawn from Remoting. Because of the not insignificant overhead of newing-up an instance per call (Remoting didn't have PerSession), you had to go singleton, but in WCF this only happens once per client in the default PerSession mode - so you're probably right to assume it won't be a factor and the concurrency mode settings should only have an impact on the client's ability to be parallel.

Thanks again for posting this. I'm always curious to hear that WCF isn't up to extreme processing especially when some people (i.e. PlentyOfFish.com) can serve 500-600 pages per second from two IIS boxes, and each page has a payload probably more than a typical WCF's XML response. And also, MS have planned for WCF to service many, many public internet users using Silverlight apps. I think alot of people are counting on it.

Monday, March 2, 2009 9:25 PM by choy

# re: Creating high performance WCF services

The service behaviors like ConcurrencyMode and InstanceContextMode are important knobs for WCF. If initialization is costly and your service is pretty stateless, then having InstanceContextMode.Single is a good way to go. The problem with singleton and per-session is that you will then want to set the ConcurrencyMode which defaults to Single (only one thread at a time). If your service is thread-safe and re-entrant, then you want to take advantage of ConcurrencyMode.Multiple.

If you cache everything, then per-call + concurrency single can be about the same as singleton + concurrency multiple.

Wednesday, May 6, 2009 6:18 PM by kp

# re: Creating high performance WCF services

Really good tips....I was able to reduce my latency from 125 ms to 8 ms !!! Good work !!! I wish all these tips were available from Microsoft.

Thursday, August 6, 2009 6:26 PM by Marc Gravell

# re: Creating high performance WCF services

@Big - for WCF, the comparison has to be to DataContractSerializer - but it doesn't do too badly ;-p

Friday, December 18, 2009 10:41 AM by Nathani

# re: Creating high performance WCF services

Hi,

I'm new to WCF and I'm using DataSet/DataTable from my WCF service (I came to know that this is bad). How can I avoid this and make my request/response faster?

 

> SW - use objects with DataContractSerializer

Friday, July 2, 2010 5:37 PM by ThomasI

# re: Creating high performance WCF services

Realise that this is a bit stale, but i'm giving it a try:

Not sure I understand you recommendation #6. Isn't that redundant when using netTcpBinding? (#1)

Friday, July 2, 2010 11:12 PM by Scott Weinstein

# re: Creating high performance WCF services

@ThomasI

netTcp keeps type info across the wire

binaryformatter doesn't

Monday, July 26, 2010 11:13 AM by sandeep

# re: Creating high performance WCF services

plz tell me how to increase wcf performance by using basic httpbinding

Tuesday, April 19, 2011 9:45 AM by Kalpesh Hingu

# re: Creating high performance WCF services

How to use BinaryFormatter if Store procedure return DataSet to WCF. How to implement this in WCF

Thursday, June 9, 2011 7:09 AM by Etem

# re: Creating high performance WCF services

In terms of CPU usage, running the service with [ServiceBehavior(InstanceContextMode=InstanceContextMode.Single,ConcurrencyMode=ConcurrencyMode.Multiple)] provides extreme benefits

Tuesday, January 17, 2012 8:28 PM by Mark

# re: Creating high performance WCF services

I'd be interested to know your experiences Scott with regards a comparison of BasicHttpBinding and NetTcpBinding, as I have read that it is very hard to NetTcp to surpass the basic binding and I seem (anecdotal) to experience a slightly higher latency. I believe self-hosting is when it does manage to win. Does NetTcp also not share the port?