I noticed a flaw here. Will update the numbers in the next days.
Every once in a while, someone comes up with a benchmark comparing O/RM tools. This time, it was my friendly neighbor at weblogs.asp.net, Frans Bouma, here and here. Nothing against it, but the problem with these tests is that they tend to use the typical out-of-the-box configuration for each O/RM, which normally happens to be, well, let’s say, suboptimal. Fabio Maulo once wrote a couple of posts on this. Like him, I know little about the other O/RMs (except for Entity Framework and LINQ to SQL, I have never used any), but I know a thing or two about NHibernate, so I decided to do a simple test, but only on NHibernate.
Do note, I know very well that Frans knows this, this is not to bash him, this is just to show how misleading these benchmarks can be, if we don’t take care to optimize our tools. Of course, the same can be said about the other tools, it’s just that I don’t know them – or care, for that matter, other than Entity Framework. My purpose is to show that NHibernate performance depends heavily on the configuration used.
This will be a very simple test, so I’m going to use a basic class model:
We have a collection of Devices and Measures, and we have Values that are associated with a Measure and a Device, and store a value for a given timestamp:
For the mappings, I have used conventional automatic configuration, same as described here:
In a nutshell:
I am using IDENTITY as the primary key generator (this is for SQL Server only);
All classes are not lazy;
Defaults for everything else.
I let NHibernate generate the database schema for me and I add 100 records to the database:
I am going to execute a simple LINQ query returning all 100 entities:
I disable cache mode since I’m not using second level cache, flush mode because I’m never going to have changes that I want to submit to the database, and I set the default of read only for all entities, because I really don’t want them to change.
I am going to have two configuration settings, normal and optimized, and I will run both using regular sessions (ISession) plus another one with stateless sessions (IStatelessSession).
I ran each query 20 times and record how many milliseconds it took. Then I ran everything 10 times, discard the highest and lowest values, and calculated the average. All of this is done on my machine (x64, 4 cores with 8 GB RAM, Windows 7) and a local SQL Server 2008 Express.
I compiled the application as Release and executed it outside Visual Studio 2013. No optimizations or whatever.
Keep in mind that this is not an exhaustive, scientific, test, think of it more as a proof of concept to show that NHibernate performance can be greatly improved by tweaking a few settings.
So, to add some suspense, I’m going to show the results before I show the configuration settings used:
|Optimized (Regular Session)||68.8ms||~6.3%|
|Optimized (Stateless Session)||70.3ms||~6.4%|
The gain using optimized configuration was always enormous: the optimized configuration took only about 6% of the time;
Almost always the performance of stateless sessions was worse than that of regular sessions, but it was pretty close.
For the normal configuration I used this:
Now we get to the core of it. I selected a combination of properties that I felt, based on my experience, that could have an impact on the performance. I really didn’t test all combinations, and I may have forgotten something or chosen something that is even worse, I leave all this as an exercise to you, dear reader!
Some explanation is in order, first, a general description of the properties:
|FormatSql||Format the SQL before sending it to the database|
|GenerateStatistics||Produce statistics on the number of queries issued, entities obtained, etc|
|Hbm2ddlKeyWords||Should NHibernate automatically quote all table and column names (ex: [TableName])|
|PropertyBytecodeProvider||What bytecode provider to use for the generation of code (in this case, Lightweight Code Generator)|
|QueryStartupChecking||Check all named queries present in the configuration at startup? (none in this example)|
|ShowSql||Show the produced SQL|
|StatementFetchSize||The fetch size for resultsets|
|UseProxyValidator||Validate that mapped entities can be used as proxies|
|UseSecondLevelCache||Enable the second level cache|
|UseSqlComments||Enable the possibility to add SQL comments|
|UseQueryCache||Allows the results of a query to be stored in memory|
|WrapResultSets||Caches internally the position of each column in a resultset|
I am well aware that some of these settings have nothing to do with query performance, which was what I was looking for, but some do. Of these, perhaps the most important – no real tests, though, just empirical reasoning - were:
GenerateStatistics: since I’m not looking at them, I disabled them;
FormatSql: no need to format the generated SQL since I’m not going to look at it;
PrepareSql: prepares (compiles) the SQL before executing it;
ShowSql: not needed;
StatementFetchSize: since I’m getting 100 results at a time, and they have little data, why not process them all at the same time;
UseSecondLevelCache: I’m not using it either, this is just for reference data that is mostly immutable;
UseQueryCache: I want live results, not cached ones;
WrapResultSets: NHibernate doesn’t have look every time for the index of a specific named column.
As for the listeners, as you may know, NHibernate has a rich event system, which includes events that fire before (PreLoad) and after (PostLoad) an entity is loaded, that is, when the resultset arrives and NHibernate instantiates the entity class. NHibernate includes default event listeners for these events, which don’t do anything, but are still called for each materialized entity, so I decided to get rid of them.
What I didn’t test was:
Using LINQ projections instead of querying for entities;
Using named HQL queries;
Again, I leave it as an exercise to those interested, but I may revisit this some other time.
I think these results speak for themselves. It should be obvious that a lot can be done in terms of optimizing NHibernate performance. Like I said, I didn’t do any research prior to writing this post, I just relied on my past experience, so, it may be possible that things can even be improved – let me hear your thoughts on this! My friends, the real problem of NHibernate is not performance, trust me!
If you are interested in the actual code I used, send me an email and I will be more than happy to send it to you.
As always, I’d love to hear your opinion!