Unit Test Projects or Not?

image It's funny how the world works. A butterfly flaps it's wings in Brazil, and a tornado forms in Texas 1,000 miles away. Phil Haack posted a poll about unit test project structure and asked the very question we've come to on our current project. Should unit tests belong in their own project or as part of the system? I was going to post a comment on Phil's entry, but figured I would drag my explanation and description out to a full post here.

In the past I've always created a separate test project. Tree Surgeon by default does this (and now I'm looking at adding an option to let you decide at code generation time) and most projects I know of work this way. You create your MyApp.Core project (containing your domain logic) and a MyApp.Test project with all the unit tests. More recently I've been creating MyApp.Specs project but that's just a different evolution.

In the next project we're working on, we're looking to shift this approach. A shift to include unit tests in our MyApp.Core project. Here's some reasons and thinking behind it.

With unit tests (or specifications) in a separate project you end up mimicking the structure of your domain and create a namespace hierarchy. By default .NET assemblies have a default namespace for your application and then the name of any folder in the project is appended to the default namespace. So if your assemblies default namespace is MyApp.Core (and the namespace defaults to the name of the assembly) and you create a folder called Customer, all classes in that folder will be in the MyApp.Core.Customer namespace. In your test project you have a similar thing and usually you'll have the default namespace to be MyApp.Test (the name of the assembly).

Since there is only one test assembly (assuming you don't break them up that is) then you don't necessarily want to create a folder called CustomerSpecs (or CustomerTests or even Customer) so you might create a folder called Domain. After all, you're unit testing the domain but then they'll be the UI, Presenters, Factories, Data Access, etc. Do you create a separate test assembly for all of these? Probably not.

Let's see, we have an assembly (MyApp.Core), a class (Customer) in a namespace (MyApp.Core.Customer). Now you've got a test assembly (MyApp.Test), a set of Customer tests (CustomerSpecs.cs or CustomerTests.cs or whatever) in a domain namespace (MyApp.Test.Domain). This is getting a little complicated, but no big deal from a resolution perspective. You'll just bring in the namespaces you need and bang (or BAM!).

However two things seem to arise out of this setup.

First (which might kick off it's own huge debate) you need access to your Customer class and potentially other classes, enumerations, etc. that it uses which are locked away in MyApp.Core.dll. That means you have two options. Either you make the Customer class public or you use the InternalsVisibleTo attribute to let MyApp.Test.dll see the stuff inside of MyApp.Core.dll. There's another option here, slam all the files into one assembly and don't worry about it from a testing perspective. That might alleviate the problem but that's a different blog entry.

The second thing that comes out of this is a fairly deep and wide namespace hierarchy in your test assembly. That might not be a big deal unto itself, but could be an inconvenience. In addition the deep impact this might cause, let's say you have 30 domain classes and the subsequent 30 or so fixtures (or more, or less, doesn't really matter). And these are scattered around in various folders. Each time you touch the fixture to write a test or look at the domain object and create some test, you're playing hunt and peck inside your test assembly to find the right spot to match the folder structure. Of course if you don't care and toss everything into a single folder you won't care, but I think that's a different type of maintenance you don't want to get into. Then, let's say you restructure your domain (which can happen a few times throughout a project) and now some of the classes relating to Customer move into some other place in the hierarchy in your domain. That's easy enough with ReSharper and a move like this is pretty low-maintenance. Except now your test folder structure doesn't match your logical domain structure (or folder structure for that matter).

Okay, that's the side of the conversation about issues that we've come to on using a separate test project. Now the positives on including your tests with the code you're testing.

  1. I don't need a separate Test project. There's a bit of debate in the blog-o-sphere around number of projects and what's right and what's too much so keeping things lean is good.
  2. I don't have to go hunting for a fixture in some hierarchy that may (or may not) be valid or the same as my domain. With ReSharper it's easy to find files/classes, but using a separate test project I have double maintenance to deal with. If I want to keep them in sync, it's more work.
  3. I don't have to expose my domain to my tests. As everything is in one project I can use OO principles and maintain encapsulation. When you create a new class, there's a damn good reason it's marked as internal and not public. If my entire domain is internal I can choose to expose what I need outside of the system/assembly as needed rather than "make it public so the test assembly can see it". True, there are tricks to expose MyApp.Core.dll to MyApp.Test.dll but they're hacks IMHO.
  4. I can leverage my unit testing framework in my runtime environment. This is probably the biggest advantage I see when I do something like create a Test folder under my Customer folder in my domain project. I can choose to ship my unit testing framework tools (MbUnit.Framework.dll or MbUnit.Gui.exe) with my system. This would be useful say in a QA or User Acceptance environment where I can run my tests against the real environment. This might not be something you want to do all the time, but I think it's good to have the option.

Here are some arguments I've heard for including your test code with production code that I'll address.

"If my tests are in my domain, I have a reliance on my unit test framework assemblies" - Yeah? So. If I wrap log4net I have a dependency on deploying log4net.dll as well. I'm not sure I see a disadvantage to this. There's been people saying they were "bitten" by this, but I'm sure what the bite is like or what the impact of that bite might be. Optionally, when we deploy we can decide to deploy our test code and it's dependencies as needed. Just because it's there doesn't mean it needs to go out the door. If we use our NAnt build scripts, we can not include the Test code and omit the MbUnit.Framework.dll files. Clean and lean.

"I want to see my tests and only my tests in one project and what I'm testing in another" - Again, not sure the advantage to this. If anything, keeping them together reduces the amount of "jumping" around you do in your IDE from this project to a test project, then back again. I'm not convinced or sure why you "want" to see tests in one project.

"Production code is production code and not test code!" - Not sure what this means, since I consider all code production code, tests, classes, etc. The ability to unit test my "production code" in a "production environment" rather than some simulation is a bonus for me.

All in all there's no clear cut answer here. What works for you works and I think the general mass keep tests in a separate project. I want to buck the norm here and for the next project we're going to try it out differently. I think there's advantages to it (and potentially disadvantages, like having to potentially clutter up my .Core assembly with a bunch of ObjectMother classes for example) but we'll see how it goes.

I don't like not trying something because "that's the way we always did it". Doesn't make it right. So give it a shot if you want, try it out, share your experience, or leave a comment that I'm a mad coder and putting my devs through unnecessary torture.

Like Phil said, this is not "a better way" or "the right" or "wrong" way to do things. I'm going with a Test folder under my aggregate classes in my domain and we'll how that goes. YMMV.

15 Comments

  • Does that mean I'm a butterfly? :)

  • Ok.
    This CAN be a valid solution for some, but most of the time, I think that its better just to have your unit tests in another project.
    The way I structire it, is I have 1:1 class&&folder&&namespace mapping between my test project and actual project.
    If I have MyApp.Core assembly and namespace in my project, I also have Mypp.Core.Tests assembly and if I have
    MyApp.Core.Customer class in my project, (in main/Core folder) I have MyApp.Core.Tests.CustomerTests class in my test project (in main/Core) folder in test project.
    Yes - it requires me to keep both structores in sync, but its advantages pay off for this inconvenience.

    Still, I dont claim its THE way to structure your tests

  • I'll admit I scoffed at the idea of including tests with your code but those are some pretty good arguments. It's similar to the "compile to a single assembly" argument where you use your build scripts to manage compilation and deployment instead of Visual Studio. I imagine a tool like NDepend might be useful to ensure you aren't doing anything untoward with respect to maintaining OO principles.

    Very interested in seeing the results of your experiment.

  • I raised the same question(s) on the altnetconf or cli_dev or altdotnet mailing list a while back. Most of the reactions were along the lines of "HERESY!"

  • There's ways of testing private members. I'd agree this is a limitation of many unit testing frameworks; but reorganizing code to suite a tool doesn't smell right. If you listen to Oren, everything should be public, right?

    I think namespace maintenance issues in a test project are a red herring. Tools like Visual Studio and R# make finding usage and references much quicker than locating it by namespace--even in a perfect namespace hierarchy. I think the issues you've described could be addressed with added features in tools like R#.

    Separation of concerns. The entity or domain assembly shouldn't have the testing concern contained within it; especially considering it doesn't need to be deployed.

    There's something reassuring about knowing that my unit tests are testing assemblies that I'm actually deploying. If my NAnt script is creating *different* assemblies for deployment, I can't be sure that change in dependency hasn't caused a problem. I'm for reducing dependencies and concerns, AOD, DI, and IoC don't make as much sense if I'm not addressing those issues across the entire project. AOD, DI, and IoC are complex in themselves, if the goal is reduced complexity AOD, DI, and IoC get tossed too--I can still write working code with more dependencies...

  • This what happened to me:

    Could not wire up different testing assemblies in NUnit Runner to run all tests in the solution. ANSWER: One Testing project per solution.

    Started using Resharper and using it's Unit Test Runner. But didn't want to run all the tests at once. Started usng 'UnitTests' folder in each project.

    Right-click over project runs only that projects unit tests. Right-click over solution and run all unit tests.

    So you could say my unit test implementation is driven by my tool usage.

  • Since my dlls are sent to the client, I'd hate to bulk up their install with tests. How do you deal with this?

  • Some disadvantages:

    - Searching project source will also search test classes - with tests in a separate project in the same solution, you can exclude or include them

    - Increases the size of the assembly (possibly significantly)

    - Requires distributing additional assemblies (NUnit, RhinoMocks, etc.)

    - Potential for embarrassment or security breach if test code includes customer Mr. Jerk or a hardcoded login

    - Test units clutter the project hierarchy in Visual Studio

  • @Dave: "bulk up their install with unit tests". We're not talking about deploying megabytes of data here. I think the difference between an assembly with unit test code and without is pretty minimal. As I said in the post though, that's a decision you can make and choose not (via a deploy script) to include it with the build.

    @Bill: Not sure what searching your doing in project source? I agree you'll probably have duplication in code perhaps but what would you be searching for and what's the concern here? Again, the distribution might not be a big deal but you could choose not to deploy xUnit.dll assemblies. The embarassemnt factor could be big, but there needs to be some level of responsibility here.

    @Peter: I never looked at SOC at the assembly boundary but rather the class one so not sure if I agree with your argument here (but that might just be too).

    Thanks for the feedback guys!

  • You can always expose your internals to your test project via InternalsVisibleTo. I've got a recent post on my blog about getting it to work for signed assemblies, which can be a hassle.

  • Typo in last comment:
    "but that might just be too" = "but that just might be me too"

  • @Bill: All of the disadvantages you list can be avoided by using the techniques I describe in my blog post that's linked in an earlier comment. Depending what you mean by searching, if it's a text search the tests will be included even if you use these techniques but if you would use for example "find all references" only production code will be included if you have set the build mode to anything but Debug.

  • This is an interesting post. I find the idea of of embedding tests within application projects horribly messy and love things like friend assemblies as they remove the few reasons why one might be forced to do it.

    However, I've never stopped to consider whether there might actually be reasons to want to put tests in application projects. I'm not convinced by the arguments, but it's good to have ones assumptions challenged like this atg times.

  • I'm a big fan of keeping the number of projects to a bear minimum, but still - I seperate when needed.
    We have in our sln two projects called "Product.Infrastructure" and "Product.Infrastructure.UI", because we want the consumers of the first assembly to be able to safly use all code in it in server code (since there's no reference to System.Windows.Forms or any other UI dll there).
    Having the seperation between the test project and the "real" one (in all fairness - there IS a difference between test code and production code) helps me avoid using any of the testing framework (or mocking framework, or some derivitives of these) in my production code.

    Also - having the test code could hinder your ability to enforce proper (minimal) CAS policies.

    On the bright side - I do agree that most of these issues can be resolved by some compiler flags in the right places (and automatic deployment tests for the "production" dll could also help out).

  • Bil: how did these ideas work out for you in practice?

Comments have been disabled for this content.