On NHibernate Performance

I noticed a flaw here. Will update the numbers in the next days.

Introduction

Every once in a while, someone comes up with a benchmark comparing O/RM tools. This time, it was my friendly neighbor at weblogs.asp.net, Frans Bouma, here and here. Nothing against it, but the problem with these tests is that they tend to use the typical out-of-the-box configuration for each O/RM, which normally happens to be, well, let’s say, suboptimal. Fabio Maulo once wrote a couple of posts on this. Like him, I know little about the other O/RMs (except for Entity Framework and LINQ to SQL, I have never used any), but I know a thing or two about NHibernate, so I decided to do a simple test, but only on NHibernate.

Do note, I know very well that Frans knows this, this is not to bash him, this is just to show how misleading these benchmarks can be, if we don’t take care to optimize our tools. Of course, the same can be said about the other tools, it’s just that I don’t know them – or care, for that matter, other than Entity Framework. My purpose is to show that NHibernate performance depends heavily on the configuration used.

Model

This will be a very simple test, so I’m going to use a basic class model:

We have a collection of Devices and Measures, and we have Values that are associated with a Measure and a Device, and store a value for a given timestamp:

   1: public class Device

   2: {

   3:     public virtual Int32 DeviceId { get; set; }

   4:     public virtual String Name { get; set; }

   5: }

   6:  

   7: public class Measure

   8: {

   9:     public virtual Int32 MeasureId { get; set; }

  10:     public virtual String Name { get; set; }

  11: }

  12:  

  13: public class Value

  14: {

  15:     public virtual Int32 ValueId { get; set; }

  16:     public virtual DateTime Timestamp { get; set; }

  17:     public virtual Device Device { get; set; }

  18:     public virtual Measure Measure { get; set; }

  19:     public virtual Double Val { get; set; }

  20: }

Mappings

For the mappings, I have used conventional automatic configuration, same as described here:

   1: var modelMapper = new ConventionModelMapper();

   2: modelMapper.IsEntity((x, y) => x.IsClass == true && x.IsSealed == false && x.Namespace == typeof(Program).Namespace);

   3: modelMapper.BeforeMapClass += (x, y, z) => { z.Id(a => a.Generator(Generators.Identity)); z.Lazy(false); };

   4:  

   5: var mappings = modelMapper.CompileMappingFor(typeof(Program).Assembly.GetTypes().Where(x => x.IsPublic && x.IsSealed == false));

   6:  

   7: cfg.AddMapping(mappings);

In a nutshell:

• I am using IDENTITY as the primary key generator (this is for SQL Server only);
• All classes are not lazy;
• Defaults for everything else.

Data

I let NHibernate generate the database schema for me and I add 100 records to the database:

   1: using (var sessionFactory = cfg.BuildSessionFactory())

   2: using (var session = sessionFactory.OpenSession())

   3: using (var tx = session.BeginTransaction())

   4: {

   5:     var device = new Device { Name = "Device A" };

   6:     var measure = new Measure { Name = "Measure A" };

   7:     var now = DateTime.UtcNow;

   8:  

   9:     for (var i = 0; i < NumberOfEntities; ++i)

  10:     {

  11:         var value = new Value { Device = device, Measure = measure, Timestamp = now.AddSeconds(i), Val = i };

  12:         session.Save(value);

  13:     }

  14:  

  15:     session.Save(device);

  16:     session.Save(measure);

  17:  

  18:     tx.Commit();

  19: }

Tests

I am going to execute a simple LINQ query returning all 100 entities:

   1: using (var session = sessionFactory.OpenSession())

   2: {

   3:     session.CacheMode = CacheMode.Ignore;

   4:     session.FlushMode = FlushMode.Never;

   5:     session.DefaultReadOnly = true;

   6:     session.Query<Value>().ToList();

   7: }

I disable cache mode since I’m not using second level cache, flush mode because I’m never going to have changes that I want to submit to the database, and I set the default of read only for all entities, because I really don’t want them to change.

I am going to have two configuration settings, normal and optimized, and I will run both using regular sessions (ISession) plus another one with stateless sessions (IStatelessSession).

I ran each query 20 times and record how many milliseconds it took. Then I ran everything 10 times, discard the highest and lowest values, and calculated the average. All of this is done on my machine (x64, 4 cores with 8 GB RAM, Windows 7) and a local SQL Server 2008 Express.

I compiled the application as Release and executed it outside Visual Studio 2013. No optimizations or whatever.

Keep in mind that this is not an exhaustive, scientific, test, think of it more as a proof of concept to show that NHibernate performance can be greatly improved by tweaking a few settings.

Results

So, to add some suspense, I’m going to show the results before I show the configuration settings used:

Some remarks:

• The gain using optimized configuration was always enormous: the optimized configuration took only about 6% of the time;
• Almost always the performance of stateless sessions was worse than that of regular sessions, but it was pretty close.

Normal Configuration

For the normal configuration I used this:

   1: var cfg = new Configuration()

   2:     .DataBaseIntegration(x =>

   3:     {

   4:         x.Dialect<MsSql2008Dialect>();

   5:         x.Driver<Sql2008ClientDriver>();

   6:         x.ConnectionStringName = "NHPerformance";

   7:     });

Optimized Configuration

Now we get to the core of it. I selected a combination of properties that I felt, based on my experience, that could have an impact on the performance. I really didn’t test all combinations, and I may have forgotten something or chosen something that is even worse, I leave all this as an exercise to you, dear reader!

   1: var cfg = Common()

   2:     .SetProperty(NHibernate.Cfg.Environment.FormatSql, Boolean.FalseString)

   3:     .SetProperty(NHibernate.Cfg.Environment.GenerateStatistics, Boolean.FalseString)

   4:     .SetProperty(NHibernate.Cfg.Environment.Hbm2ddlKeyWords, Hbm2DDLKeyWords.None.ToString())

   5:     .SetProperty(NHibernate.Cfg.Environment.PrepareSql, Boolean.TrueString)

   6:     .SetProperty(NHibernate.Cfg.Environment.PropertyBytecodeProvider, "lcg")

   7:     .SetProperty(NHibernate.Cfg.Environment.PropertyUseReflectionOptimizer, Boolean.TrueString)

   8:     .SetProperty(NHibernate.Cfg.Environment.QueryStartupChecking, Boolean.FalseString)

   9:     .SetProperty(NHibernate.Cfg.Environment.ShowSql, Boolean.FalseString)

  10:     .SetProperty(NHibernate.Cfg.Environment.StatementFetchSize, "100")

  11:     .SetProperty(NHibernate.Cfg.Environment.UseProxyValidator, Boolean.FalseString)

  12:     .SetProperty(NHibernate.Cfg.Environment.UseSecondLevelCache, Boolean.FalseString)

  13:     .SetProperty(NHibernate.Cfg.Environment.UseSqlComments, Boolean.FalseString)

  14:     .SetProperty(NHibernate.Cfg.Environment.UseQueryCache, Boolean.FalseString)

  15:     .SetProperty(NHibernate.Cfg.Environment.WrapResultSets, Boolean.TrueString);

  16:  

  17: cfg.EventListeners.PostLoadEventListeners = new IPostLoadEventListener[0];

  18: cfg.EventListeners.PreLoadEventListeners = new IPreLoadEventListener[0];

Some explanation is in order, first, a general description of the properties:

I am well aware that some of these settings have nothing to do with query performance, which was what I was looking for, but some do. Of these, perhaps the most important – no real tests, though, just empirical reasoning - were:

• GenerateStatistics: since I’m not looking at them, I disabled them;
• FormatSql: no need to format the generated SQL since I’m not going to look at it;
• PrepareSql: prepares (compiles) the SQL before executing it;
• ShowSql: not needed;
• StatementFetchSize: since I’m getting 100 results at a time, and they have little data, why not process them all at the same time;
• UseSecondLevelCache: I’m not using it either, this is just for reference data that is mostly immutable;
• UseQueryCache: I want live results, not cached ones;
• WrapResultSets: NHibernate doesn’t have look every time for the index of a specific named column.

As for the listeners, as you may know, NHibernate has a rich event system, which includes events that fire before (PreLoad) and after (PostLoad) an entity is loaded, that is, when the resultset arrives and NHibernate instantiates the entity class. NHibernate includes default event listeners for these events, which don’t do anything, but are still called for each materialized entity, so I decided to get rid of them.

Additional Work

What I didn’t test was:

• Using LINQ projections instead of querying for entities;
• Using named HQL queries;
• Using SQL.

Again, I leave it as an exercise to those interested, but I may revisit this some other time.

Conclusion

I think these results speak for themselves. It should be obvious that a lot can be done in terms of optimizing NHibernate performance. Like I said, I didn’t do any research prior to writing this post, I just relied on my past experience, so, it may be possible that things can even be improved – let me hear your thoughts on this! My friends, the real problem of NHibernate is not performance, trust me!

If you are interested in the actual code I used, send me an email and I will be more than happy to send it to you.

As always, I’d love to hear your opinion!