On NHibernate Performance

I noticed a flaw here. Will update the numbers in the next days.

Introduction

Every once in a while, someone comes up with a benchmark comparing O/RM tools. This time, it was my friendly neighbor at weblogs.asp.net, Frans Bouma, here and here. Nothing against it, but the problem with these tests is that they tend to use the typical out-of-the-box configuration for each O/RM, which normally happens to be, well, let’s say, suboptimal. Fabio Maulo once wrote a couple of posts on this. Like him, I know little about the other O/RMs (except for Entity Framework and LINQ to SQL, I have never used any), but I know a thing or two about NHibernate, so I decided to do a simple test, but only on NHibernate.

Do note, I know very well that Frans knows this, this is not to bash him, this is just to show how misleading these benchmarks can be, if we don’t take care to optimize our tools. Of course, the same can be said about the other tools, it’s just that I don’t know them – or care, for that matter, other than Entity Framework. My purpose is to show that NHibernate performance depends heavily on the configuration used.

Model

This will be a very simple test, so I’m going to use a basic class model:

image

We have a collection of Devices and Measures, and we have Values that are associated with a Measure and a Device, and store a value for a given timestamp:

   1: public class Device
   2: {
   3:     public virtual Int32 DeviceId { get; set; }
   4:     public virtual String Name { get; set; }
   5: }
   6:  
   7: public class Measure
   8: {
   9:     public virtual Int32 MeasureId { get; set; }
  10:     public virtual String Name { get; set; }
  11: }
  12:  
  13: public class Value
  14: {
  15:     public virtual Int32 ValueId { get; set; }
  16:     public virtual DateTime Timestamp { get; set; }
  17:     public virtual Device Device { get; set; }
  18:     public virtual Measure Measure { get; set; }
  19:     public virtual Double Val { get; set; }
  20: }

Mappings

For the mappings, I have used conventional automatic configuration, same as described here:

   1: var modelMapper = new ConventionModelMapper();
   2: modelMapper.IsEntity((x, y) => x.IsClass == true && x.IsSealed == false && x.Namespace == typeof(Program).Namespace);
   3: modelMapper.BeforeMapClass += (x, y, z) => { z.Id(a => a.Generator(Generators.Identity)); z.Lazy(false); };
   4:  
   5: var mappings = modelMapper.CompileMappingFor(typeof(Program).Assembly.GetTypes().Where(x => x.IsPublic && x.IsSealed == false));
   6:  
   7: cfg.AddMapping(mappings);

In a nutshell:

  • I am using IDENTITY as the primary key generator (this is for SQL Server only);
  • All classes are not lazy;
  • Defaults for everything else.

Data

I let NHibernate generate the database schema for me and I add 100 records to the database:

   1: using (var sessionFactory = cfg.BuildSessionFactory())
   2: using (var session = sessionFactory.OpenSession())
   3: using (var tx = session.BeginTransaction())
   4: {
   5:     var device = new Device { Name = "Device A" };
   6:     var measure = new Measure { Name = "Measure A" };
   7:     var now = DateTime.UtcNow;
   8:  
   9:     for (var i = 0; i < NumberOfEntities; ++i)
  10:     {
  11:         var value = new Value { Device = device, Measure = measure, Timestamp = now.AddSeconds(i), Val = i };
  12:         session.Save(value);
  13:     }
  14:  
  15:     session.Save(device);
  16:     session.Save(measure);
  17:  
  18:     tx.Commit();
  19: }

Tests

I am going to execute a simple LINQ query returning all 100 entities:

   1: using (var session = sessionFactory.OpenSession())
   2: {
   3:     session.CacheMode = CacheMode.Ignore;
   4:     session.FlushMode = FlushMode.Never;
   5:     session.DefaultReadOnly = true;
   6:     session.Query<Value>().ToList();
   7: }

I disable cache mode since I’m not using second level cache, flush mode because I’m never going to have changes that I want to submit to the database, and I set the default of read only for all entities, because I really don’t want them to change.

I am going to have two configuration settings, normal and optimized, and I will run both using regular sessions (ISession) plus another one with stateless sessions (IStatelessSession).

Configuration

Session Kind
Normal Regular Session
Optimized Regular Session
Stateless Session

I ran each query 20 times and record how many milliseconds it took. Then I ran everything 10 times, discard the highest and lowest values, and calculated the average. All of this is done on my machine (x64, 4 cores with 8 GB RAM, Windows 7) and a local SQL Server 2008 Express.

I compiled the application as Release and executed it outside Visual Studio 2013. No optimizations or whatever.

Keep in mind that this is not an exhaustive, scientific, test, think of it more as a proof of concept to show that NHibernate performance can be greatly improved by tweaking a few settings.

Results

So, to add some suspense, I’m going to show the results before I show the configuration settings used:

Configuration Time Difference
Normal 1090.3ms -
Optimized (Regular Session) 68.8ms ~6.3%
Optimized (Stateless Session) 70.3ms ~6.4%

Some remarks:

  • The gain using optimized configuration was always enormous: the optimized configuration took only about 6% of the time;
  • Almost always the performance of stateless sessions was worse than that of regular sessions, but it was pretty close.

Normal Configuration

For the normal configuration I used this:

   1: var cfg = new Configuration()
   2:     .DataBaseIntegration(x =>
   3:     {
   4:         x.Dialect<MsSql2008Dialect>();
   5:         x.Driver<Sql2008ClientDriver>();
   6:         x.ConnectionStringName = "NHPerformance";
   7:     });

Optimized Configuration

Now we get to the core of it. I selected a combination of properties that I felt, based on my experience, that could have an impact on the performance. I really didn’t test all combinations, and I may have forgotten something or chosen something that is even worse, I leave all this as an exercise to you, dear reader!Winking smile

   1: var cfg = Common()
   2:     .SetProperty(NHibernate.Cfg.Environment.FormatSql, Boolean.FalseString)
   3:     .SetProperty(NHibernate.Cfg.Environment.GenerateStatistics, Boolean.FalseString)
   4:     .SetProperty(NHibernate.Cfg.Environment.Hbm2ddlKeyWords, Hbm2DDLKeyWords.None.ToString())
   5:     .SetProperty(NHibernate.Cfg.Environment.PrepareSql, Boolean.TrueString)
   6:     .SetProperty(NHibernate.Cfg.Environment.PropertyBytecodeProvider, "lcg")
   7:     .SetProperty(NHibernate.Cfg.Environment.PropertyUseReflectionOptimizer, Boolean.TrueString)
   8:     .SetProperty(NHibernate.Cfg.Environment.QueryStartupChecking, Boolean.FalseString)
   9:     .SetProperty(NHibernate.Cfg.Environment.ShowSql, Boolean.FalseString)
  10:     .SetProperty(NHibernate.Cfg.Environment.StatementFetchSize, "100")
  11:     .SetProperty(NHibernate.Cfg.Environment.UseProxyValidator, Boolean.FalseString)
  12:     .SetProperty(NHibernate.Cfg.Environment.UseSecondLevelCache, Boolean.FalseString)
  13:     .SetProperty(NHibernate.Cfg.Environment.UseSqlComments, Boolean.FalseString)
  14:     .SetProperty(NHibernate.Cfg.Environment.UseQueryCache, Boolean.FalseString)
  15:     .SetProperty(NHibernate.Cfg.Environment.WrapResultSets, Boolean.TrueString);
  16:  
  17: cfg.EventListeners.PostLoadEventListeners = new IPostLoadEventListener[0];
  18: cfg.EventListeners.PreLoadEventListeners = new IPreLoadEventListener[0];

Some explanation is in order, first, a general description of the properties:

Setting Purpose
FormatSql Format the SQL before sending it to the database
GenerateStatistics Produce statistics on the number of queries issued, entities obtained, etc
Hbm2ddlKeyWords Should NHibernate automatically quote all table and column names (ex: [TableName])
PropertyBytecodeProvider What bytecode provider to use for the generation of code (in this case, Lightweight Code Generator)
QueryStartupChecking Check all named queries present in the configuration at startup? (none in this example)
ShowSql Show the produced SQL
StatementFetchSize The fetch size for resultsets
UseProxyValidator Validate that mapped entities can be used as proxies
UseSecondLevelCache Enable the second level cache
UseSqlComments Enable the possibility to add SQL comments
UseQueryCache Allows the results of a query to be stored in memory
WrapResultSets Caches internally the position of each column in a resultset

I am well aware that some of these settings have nothing to do with query performance, which was what I was looking for, but some do. Of these, perhaps the most important – no real tests, though, just empirical reasoning - were:

  • GenerateStatistics: since I’m not looking at them, I disabled them;
  • FormatSql: no need to format the generated SQL since I’m not going to look at it;
  • PrepareSql: prepares (compiles) the SQL before executing it;
  • ShowSql: not needed;
  • StatementFetchSize: since I’m getting 100 results at a time, and they have little data, why not process them all at the same time;
  • UseSecondLevelCache: I’m not using it either, this is just for reference data that is mostly immutable;
  • UseQueryCache: I want live results, not cached ones;
  • WrapResultSets: NHibernate doesn’t have look every time for the index of a specific named column.

As for the listeners, as you may know, NHibernate has a rich event system, which includes events that fire before (PreLoad) and after (PostLoad) an entity is loaded, that is, when the resultset arrives and NHibernate instantiates the entity class. NHibernate includes default event listeners for these events, which don’t do anything, but are still called for each materialized entity, so I decided to get rid of them.

Additional Work

What I didn’t test was:

  • Using LINQ projections instead of querying for entities;
  • Using named HQL queries;
  • Using SQL.

Again, I leave it as an exercise to those interested, but I may revisit this some other time.

Conclusion

I think these results speak for themselves. It should be obvious that a lot can be done in terms of optimizing NHibernate performance. Like I said, I didn’t do any research prior to writing this post, I just relied on my past experience, so, it may be possible that things can even be improved – let me hear your thoughts on this! My friends, the real problem of NHibernate is not performance, trust me!

If you are interested in the actual code I used, send me an email and I will be more than happy to send it to you.

As always, I’d love to hear your opinion!

                             

8 Comments

  • I did use the default, as the most performance of NH in my tests was lost in internal fk value handling. (see profile on first post on the matter). After profiling it, and I couldn't find much info on whether how to get rid of the slowness, nor did anyone step forward and correct me, I didn't bother further.

    If you could optimize the code I have, that would be great, it's on github. The query itself, the execution of it, that's not where NH loses its performance, it's solely on handling related elements per row. It might be the event handlers you speak of are the culprit, please correct me in my benchcode so it's more realistic :)

  • Hi, Frans!
    Yes, I have your code and will send you a pull request some time during the weekend (I hope!).
    Cheers! ;-)

  • Pull request: https://github.com/FransBouma/RawDataAccessBencher/pull/16

  • Great job - I honestly never expected such a difference in speed!
    I'd also like to see a comparison of different combinations of settings. It would be really interesting to see which configuration option makes the most difference.

  • Hi, Vladimir!
    Thanks! If you like to test it yourself, I can send you the code. I won't probably revisit this for now, but if I have any news, you will see them here!

  • Apparently, I made some mistakes in the test bench. I will update the post in the next few days.

  • Any update on the issues with the PR?

    I'd really like to know if there is anything that can be done OOB to increase NH's perf. Thanks.

  • Sebastien:
    Will try to update this in the next days. Have you tried any of these optimizations? Do you have your own numbers?

Comments have been disabled for this content.