Not too long ago we had to do some performance profiling for a Commerce Server site that went into production a couple of months before. We knew we had some performance issues, but because of some inventory issues at the client, we had to rush the site into production.

The first couple of days the performance was acceptable, but as more and more customers started hitting the site, some of the nasty issues we addressed started showing up. The site was stable though, just not the fastest one out there.

The site is a pretty standard e-commerce site, not much fancy stuff on this one. The only exception is some very large categories rendering 500+ products on a single page.

First Things First: HTML Ad Libitum

Generating HTML for a category containing 500+ can be pretty tough on both the server and the client side. On the server side, the basic category layout was built using a couple of ASP.NET user controls and each product view consisted of a couple of user controls too. This provided a very flexible and configurable layout, but the performance for generating all those server controls and related markup was pretty tough for the server. The HTML output (without images) was about ~1.6 MB, which the browser wasn't too happy about either.

So removing a lot of user controls and only one user control per product resulted in an HTML output around ~400 kb tops. No need to say that this had a major impact on performance.

With the HTML all nice and cleaned up, we can start digging in to the inner workings of the site.

Next up: Tracing For Bottlenecks

To get a basic idea of where bottlenecks could be found, we turned to the ASP.NET Output Trace. It's pretty simple stuff, and I wrote some stuff the trace we I knew we had some issues. When displaying categories we had to do a lot of heavy filtering on product variants. Each product having about 10+ variants, a product list with 500 products would generate ~5.000 products to filter!

However, the trace showed that this wasn't nearly as time consuming as I had imagined. A bigger problem was in a user controlle called CatalogTopMenu.ascx.

At the top of the site the user can navigate through the catalogs available in the catalog set, implemented in CatalogTopMenu.ascx.

Bring In The Profiler

Of course, you could add stuff to the trace, compile, deploy, test, check trace, write down result, remove trace code, add more trace code, compile, deploy and so on. Instead we used ANTS Profiler from Red Gate. I really love this tool, it fairly easy to use, but it provides some real useful result. If you find yourself in need of a performance profiler I can highly recommend trying this one - there's a 14-day free trial available on their web site.

Setting Up ANTS Profiler For Commerce Server

When profiling an ASP.NET site, ANTS Profiler starts a new instance of the worker process (w3wp.exe), which runs the site you're profiling. Because Commerce Server requires a seperate application pool, running the site outside IIS proved a bit difficult. The profiling worker process is being executed by a service called "ANTS Profiler 3 Server", so setting the log on identity for this one to the same identity as the application pool used to host the site resolved this issue. You can find the service under Control Panel > Administrative Tools > Services. Here's my setup:

Next up, start the profiler from the site, and browse around. When you're done browsing, hit the "Take Snapshop" button in the profiler.

The Profiler Results

The main window displays slowest lines of code (top) and slowest method (bottom):

Looking at the slowest method you can see it's a lot slower than the second slowest! Taking a look at the source method we find the following line, which is the sinner:

It kind of surprised me that the cause of our slowest methods is a call to the Commerce Server API. We did a lot of additional tests to prove this right, and they all confirmed this issue, even though catalogs are being cached. I would imagine it would have taken a lot of Trace.Writes to find this one.

Dllhost.exe Consuming A Lot Of Memory

As a side note, we had issues with the dllhost.exe service on the server would consume a lot of memory! After a short visit to Google, it turned out that there's a known memory leak in a COM+ component, causing the dllhost.exe to run wild when performing many catalog imports. We're using BizTalk to import a lot of products, so applying Commerce Server Service Pack 2 should resolve this issue.

The Results

Fixing the issues mentioned above had a major positive impact on the performance. Other thing we did was:

• Improving the algorithm for filtering variants
• Removed some unnessecary calls to the catalog and inventory system
• A bit more caching
• And a lot more

The server side generation went from ~25 seconds to ~4 seconds the the single heaviest page on the site. With the HTML being all slim, the rendering on the client side improved a lot too.

Next up is to open up Reflector and SQL Profiler to find out why getting a catalog is such a tough operation, and maybe do a blog post about it, unless Brian beats me to it...