Brian Ritchie's Blog

CQRS: Command Query Responsibility Segregation

Jason Gerard and I were talking about document databases the other day, and he said "Have you heard about CQRS?"  This was news to me, so I started doing some research.  Wikipedia, which usually provides good information, said the following "CQRS is simply the creation of two objects where there was previously only one. The separation occurs based upon whether the methods are a command or a query (the same definition that is used by Meyer in Command and Query Separation, a command is any method that mutates state and a query is any method that returns a value)."

Ok, that was clear as mud. Maybe a picture will help.

This diagram is from Udi Dahan's excellent presentation on CQRS at the London .NET User Group.  Udi is an amazing presenter & his presentation is well worth the watch.  He is also the guy behind NServiceBus.

Here are the basics as I understand them:

• In a multi-user environment, data on the screen is always stale. Due to this fact, we don't need a complicated ORM to pull "live" data out of our OLTP database.  Instead, we can use a persistent view model to pre-cache the data needed by the UI.
• The user interface needs to capture the user's intent, not just their input.  It can then build up commands that are submitted asynchronously to the services layer.  This is a more imperative way of doing things and provides the opportunity to inject business processes without changing the user interface.
  
• Validation is performed at both the UI & services layer using a shared component.  This decreases the chance of the command being rejected.  If it does fail, we can notify the user via email or some other out-of-band means.
• This allows our backend process to have as much time as it needs to perform the business logic & update the database. 
  

By decoupling the queries and the commands, this pattern provides great scalability & simplifies the individual components.  Of course this isn't the silver-bullet architecture, but it does solve a number of problems I've faced with scalable systems. 

One problem for this pattern seems to be how to handle user input that should be reflected in the view model.  Since the view model is read-only, Udi talks about overlaying the input over the data from the view model.  This may work in some basic scenarios, but would break down in others.  This is probably an area you'll want to evaluate before using this pattern.

Besides watching the presentation you can also read Udi's post: Clarified CQRS.

There is even a .NET framework called Ncqrs that in its words "helps build scalable, extensible and maintainable applications by supporting developers apply the CQRS architectural pattern."  It has support for some interesting technologies: NServiceBus, SQLite, Microsoft Azure Platform, StructureMap, and RavenDB.

Code Camp 2010 in Words & Pictures

The Jacksonville Developer User Group pulled off an excellent code camp this weekend.  It was a great opprotunity to catch up with friends, meet new ones, and learn some good tech! 

My day started off as a volunteer at the registration table.  We had a great group of volunteers...even if it was 6:30 in the morning...on a Saturday!

Our welcome sign...who says you don't need a code monkey?

After handing out badges, bags, and shirts for a couple hours it was off to my first class: SQL Server Complex Event Processing.  This is the new StreamInsight technology that is part of SQL Server 2008 R2.  Interesting stuff for streaming real-time data from various system...I'll need to dig into it a bit deeper.

Next it was off to learn about Functional Programming and then "Becoming a Rx Pusher with the Reactive Framework".  The Reactive Framework talk was probably the most interesting of the day.  It was presented by Jim Wooley who posts at ThinqLinq.com.  The Rx framework looks extremely interesting in terms of how it handles asynchronous processing.

Of course, I had to visit my friends over at PR'UF.  The PR'UF system was launched at this year's Code Camp and provides a better way to find the talented developers & architects you need.  What makes PR'UF different?  It uses reputable industry experts who recognize each candidate's talents and match them to projects.  Tired of getting less than stellar candidates?  Check into PR'UF.  

Next it was off to learn about the new managed metadata functionality in SharePoint 2010.  I was pretty surprised on how cool this technology actually is.  It provides easy ways to build up taxonomy hierachies, tag lists or documents by them, and then use them to filter your content.  Of course, you need the SharePoint Server edition (not just the "included" foundation edition).  I finished up the day with a little "Dynamic Code generation with IL & Expressions" followed up with "SharePoint 2010 Business Intelligence".   I think the BI speaker was a little more excited about LightSwitch than BI...so most of that talk was on LightSwitch.  There has been lots of buzz LightSwitch since it was announced.  It looks like it has potential for good, but also potential for evil...only time will tell.

To wrap up the day, we had the closing remarks & SWAG giveaway.  The high points were the iPad giveaway & the XBOX 360 + 20' flat screen giveaway!  Other SWAG included books, magazine subscriptions, and free software.  Of course, I walked away empty handed...but thats just the way it goes sometimes. 

All in all, it was a great event.  If you're in Jacksonville or the surrounding area, you should definitely check it out next year.  In the mean time, get involved in Jax DUG (Jacksonville Developer User Group) and the Jax Arg Sig (Architecture Special Interest Group).

Netflix Instant Streaming comes to iPhone & iPod touch

It's here!  Netflix has brought their awesome instant streaming technology to the iPhone & iPod touch.  I've tried it over WiFi and it is awesome.  Apparently it works over 3G too.

Why are you still reading this?  You should be over at the AppStore getting your device updated!

Jacksonville .NET Code Camp 2010

This Saturday is the 6th Annual Jacksonville .NET Code Camp!   It's an all day geek fest focusing on code and not marketing fluff.

  When: Saturday, August 28, 2010 All day (registration opens at 7:00am)
  Where: University of North Florida - 1 UNF Drive, Jacksonville FL 32246
  Cost: Free!

If you're not already going, join the masses!

  Attendees: 481
  Sessions: 37
  Speakers: 31
  Tracks: 7

Why spend your Saturday with a bunch of geeks? Sessions ranging from informal "chalk talks", hands on labs, to presentations. There are a mix of speakers ranging from MVPs, Regional Directors, Authors, and most importantly, local developers like you and me! Thanks to the generosity of the contributors there will be breakfast, snacks, lunch, and LOTS and LOTS of SWAG.

See you there!!

Adam Savage Presents Problem Solving: How I Do It

One of the key ingredients to software development is problem solving.  Knowing all the technology buzzwords is great, but if you don't have problem solving skills it is a waste.  Solving these problems is also what makes software development fun. At a recent Maker Faire, Adam Savage of MythBusters fame gave a great presentation on Problem Solving.  It is an excellent view.  He also mixes in some "behind the scenes" stories about his MythBuster experience as an added bonus.

Document Databases Compared: MongoDB, CouchDB, and RavenDB

Hopefully my last post, An Introduction to Document Databases, piqued your interest in this new frontier for database technology.

So, how do the different document databases compare?  Let's take a look.

There it is.  The most comprehensive comparison on the planet!  Of course, if you have any corrections or additions, please let me know.

UPDATED: Corrected replication

An Introduction to Document Databases

When most people say database, they mean relational database.  Edgar Codd defined and coined the term at IBM's Almaden Research Center about 40 years ago.  Since that time, relational databases have become the foundation of nearly every enterprise system.  However, Internet-scale systems have begun to push the limits of this venerable technology.  What has sprung up to fill the need?  Various next generation databases addressing some of the following points: being non-relational, distributed, and horizontal scalable.  These attributes are characteristics of the "NO SQL" movement.  In this case, NO stands for "Not Only".  So how many NO SQL databases are there?  More than I care to count.  But most of the fall into the following categories: Document, Graph, Key/Value, and Tabular/Wide Column.

Document Databases are especially interesting.  So what makes them different from the relational model? 

A document-oriented database is, unsurprisingly, made up of a series of self-contained documents. This means that all of the data for the document in question is stored in the document itself — not in a related table as it would be in a relational database. In fact, there are no tables, rows, columns or relationships in a document-oriented database at all. This means that they are schema-free; no strict schema needs to be defined in advance of actually using the database. If a document needs to add a new field, it can simply include that field, without adversely affecting other documents in the database. This also documents do not have to store empty data values for fields they do not have a value for. [ from Exploring CouchDB ]

They have some special characteristics that make them kick some serious SQL.

• Objects can be stored as documents:  The relational database impedance mismatch is gone.  Just serialize the object model to a document and go.
  
• Documents can be complex: Entire object models can be read & written at once.  No need to perform a series of insert statements or create complex stored procs.
  
• Documents are independent: Improves performance and decreases concurrency side effects
  
• Open Formats: Documents are described using JSON or XML or derivatives.  Clean & self-describing.
  
• Schema free: Strict schemas are great, until they change.  Schema free gives flexibility for evolving system without forcing the existing data to be restructured.
• Built-in Versioning: Most document databases support versioning of documents with the flip of a switch.
  

A few of the top document databases are CouchDB, RavenDB, and MongoDB.

• CouchDB is an Apache project created by Damien Katz (built using Erlang) and just reached a 1.0 status.  Damien has a background working on Lotus Notes & MySql. 
  
• RavenDB is built on using C# and has some interesting extension capabilities using .NET classes.  RavenDB was created by Ayende Rahien (the creator of Rhino Mocks & much more).
• MongoDB is written in C++ and provides some unique querying capabilities.  MongoDB was originally developed by 10gen.
  

So, where is the best place to use a document database?

• The schema-less nature makes it ideal to store dynamic data, such as CMS and CRM entities, which the end user can usually customize as necessary or semi structure data (provided by human).
• Web Related Data, such as user sessions, shopping cart, etc. - Due to its document based nature means that you can retrieve and store all the data required to process a request in a single remote call.
• Dynamic Entities, such as user-customizable entities, entities with a large number of optional fields, etc. - The schema free nature means that you don't have to fight a relational model to implement it.
• Persisted View Models - Instead of recreating the view model from scratch on every request, you can store it in its final form in a document database. That leads to reduced computation, reduced number of remote calls and improved overall performance.
• Large Data Sets - The underlying storage mechanism for Raven is known to scale in excess of 1 terabyte (on a single machine) and the non relational nature of the database makes it trivial to shard the database across multiple machines, something that Raven can do natively.

[ from About RavenDB ]

I'd be interested to hear your experiences with document databases.  I'll go more into a RavenDB in a future post.

UPDATED: Included MongoDB

Virtualization of Developer Workstations

Virtualization is everywhere.  We're using it for instancing our development, test, and production server environments.  A couple years back, I read a blog post by a development manager who virtualized his developer's PCs.  The idea made a lot of sense, so I went to our CIO and pitched the case.  We've been running like this for the past two years and it has been working great.

Here are some of the benefits:

1. Fault Tolerance. Unlike desktops, a virtualized cluster is built on SAN technology and multiple physical hosts.  A disk or server failure won't affect your developer's productivity or uncommitted code.
2. Hardware Flexibility.  Since it's a virtual machine, it is easy to increase the cores or the memory based on the needs of a given application.
3. Instance Flexibility.  Developers are no longer limited to one machine or one operating system.
   
4. Eases Upgrades.  In our recent migration to VS 2010, we stood up fresh VMs to test out & migrate to the upgraded version.
5. Mirror Production.  Your production environment isn't running Windows 7.  Allowing developers to code & test on a the same server version you're running in production removes on more variable.
6. Enhanced Tools.  Virtualized infrastructure brings additional tools like snapshots, clones, etc.

C# compiler as a service

Having the C# compiler as a service is an interesting new feature in Mono that was announced by Miguel de Icaza in his blog.  Miguel leads the Mono project for Novell.  This new feature supports the following scenarios:

• Run Mono's C# compiler on .NET
• Embed Mono's Evaluator in your C# app.
• Use Mono's REPL with the .NET framework runtime.

The compiler as a service is also being worked on by Microsoft. In this interview with Visual Studio Magazine, Lisa Feigenbaum, Microsoft's community program manager for the Visual Studio Managed Languages Team talks about the compiler as a service that Microsoft is working on.

MD: Is Microsoft still heading toward its vision of the "compiler as a service" for future versions of the C# compiler?

LF: Yes, very much so. "Compiler as a service" is a project in which we're re-writing the C# compiler in C#. In the process, we're opening up the compiler and exposing managed API's, so that you can query the compiler for information that was previously hidden behind a "black box." This change will enable several scenarios, including a richer language object model to support third-party extensions, the creation of a Read-Eval-Print-Loop, DSL embedding, and the ability to host C# in other contexts. We started on this project a couple of years ago, and plan to ship it in a future release, yet to be determined. It is a huge undertaking, and will take some time still to complete.

As she mentioned, Microsoft has been working on this for a few years.  Here's a post from Lamda the Ultimate, where he found a video from 2008 where Anders Hejlsberg talked about this functionality.

Hopefully this will be a reality soon, but for now the Mono library is a good option.

Json.NET Dynamic Extensions

I've been working with RavenDB and Json.NET lately.  RavenDB has some interesting schema-less capabilities using JSON documents.  When interacting with the API, you either get serialized objects or JSON.NET classes.  These are great, but it seemed like mixing in the Dynamic features of C# 4.0 would make things interesting.

So, I've added an extension method to JObject which allows you to get a dynamic instance back.  Here's an example:

   1:      string json = @"{  ""Username"": ""atucker"",  ""Expiration"": new Date(1230422400000),  ""AccessRights"": [    ""Search"",    ""Edit"",    ""Add""  ]}";

   2:   

   3:      JObject jo = JObject.Parse(json);

   4:      dynamic user = jo.AsDynamic();

   5:   

   6:      // Loop through dynamic list of access rights

   7:      foreach (var size in user.AccessRights)

   8:      {

   9:          if (size != null)

  10:              Console.WriteLine(size);

  11:      }

  12:   

  13:      user.FullName = "Adam Tucker";

In the above example, the code is loops through the "AccessRights" dynamic array built off the JSON objects.  The dynamic class actually wraps the JSON objects, so you can read and also update the dynamic object.  What's cool is the dynamic object changes the underlying JSON.  Here is the output of the JSON object after it is modifed:

   1:  {

   2:    "Username": "atucker",

   3:    "Expiration": new Date(

   4:      1230422400000

   5:    ),

   6:    "AccessRights": [

   7:      "Search",

   8:      "Edit",

   9:      "Add"

  10:    ],

  11:    "FullName": "Adam Tucker"

  12:  }

So, how does it work?  The DynamicJsonObject class descends from the DynamicObject class and overrides the GetDynamicMemberNames and TryGetMember methods.  To support arrays, the DynamicJsonArray overrides TryGetIndex and TrySetIndex.  It also implements IEnumerable for good measure.

Now that we have a dynamic object, you can also use my DynamicDuck code to interface it with your statically typed methods.

   1:   // Use Dynamic Duck to use our dynamic JSON via a static interface

   2:   AddUser(DynamicDuck.AsIf<IUser>(user));

Want to try code the code?

 [ Download the source ]