TestDriven.NET by Jamie Cansdale

Improved support for MbUnit, xUnit and Gallio

The main focus of the TestDriven.Net 2.18 release has been to improve support for test runner plug-ins in general (not just NUnit). If you’re using xUnit, MbUnit or Gallio – I recommend you upgrade to this version.

Automatic support for 64-bit machines

The registry layout on 64-bit machines is plain weird and full of pitfalls for the unwary. The registry layout is different depending on whether you’re installing under HKLM or HKCU. Under ‘HKLM’ the ‘SOFTWARE’ key is split and test runner plug-ins needed to be registered twice in order to work in both 32 and 64-bit processes. There is no such split under ‘HKCU’ and plug-ins installed there only needed to be registered once.

This created the unfortunate situation where plug-ins installed for ‘all users’ wouldn’t work when running in a 64-bit process, but plug-ins installed ‘just for me’ would work fine. Rather than expect plug-in developers to deal with this weirdness, I’ve made some changes to automatically support plug-ins that aren’t 64-bit aware.

If you have an assembly that needs to work on 32 and 64-bit machines, you may find the following snippet useful:

    public static RegistryKey OpenSoftwareKey(bool hklm, string name)
    {
        string fullName = @"SOFTWARE";
        if (hklm)
        {
            if (Marshal.SizeOf(typeof(IntPtr)) == 8)
            {
                fullName += @"\Wow6432Node";
            }
            return Registry.LocalMachine.OpenSubKey(fullName + @"\" + name);
        }
        return Registry.CurrentUser.OpenSubKey(fullName + @"\" + name);
    }

The following ad-hoc test will display TestDriven.Net’s install directory (assuming TestDriven.Net is installed ‘for all users’):

    void test()
    {
        using(var key = OpenSoftwareKey(true, @"MutantDesign\TestDriven.NET"))
        {
            Console.WriteLine(key.GetValue("InstallDir"));
        }
    }

Better support for ah-hoc tests

In previous versions of TestDriven.Net, a test runner plug-in was required to explicitly signal when none of its tests were found for execution. This would give other test runners (such as the ad-hoc test runner) a chance to handle the test. Unfortunately most test runners have been signaling a successful test run when tests were found but none were targeted.

I’ve changed it so the ad-hoc test runner will automatically get a chance to execute if no tests were executed and the test runner plug-in indicated a successful test run. The upshot of this is that you can now have ad-hoc side-by-side with MbUnit or xUnit tests.

If you’re using xUnit, try doing ‘Run Test(s)’ on each of the following methods:

    [Fact]
    public void TestMe()
    {
        Console.WriteLine("Console output isn't displayed when using xUnit");
        Assert.True(false, "Comment out [Fact] and run as ad-hoc test! ;)");
    }
    void hello()
    {
        Trace.WriteLine("Hello, World!");
    }
    void dump()
    {
        Trace.WriteLine(AppDomain.CurrentDomain, "_verbose");
    }

(ad-hoc tests should work side-by-side with all other test framework methods as well)

Improved performance when executing with Gallio / MbUnit v3

Gallio is a test runner that supports many different test types (MbUnit, xUnit, NUnit, MSTest and more). It has its own plug-in architecture and it doesn’t use the default TestDriven.Net app domain test isolation. This makes Gallio very flexible, but it also meant it wasn’t appropriate to setup and tear down the Gallio engine for each test run.

I’ve made some changes to allow Gallio to stay resident in the test process. This has significantly improved performance (especially for short test runs). If you’re using Gallio/MbUnit v3, try upgrading to Gallio v3.0.5 build 546 and see how much of a different it makes.

Feedback…

There have been lots of other changes which you can find in the release notes. If you notice any new issues, please don’t hesitate to let me know!

TestDriven.Net Options Pane

In the latest version of TestDriven.Net you will find a new options pane. The options are as follows:

Hide trace/debug output when running all tests in project/solution

In the past this setting has always been hardwired to true. It means you can add trace information to a test without cluttering up the ‘Test’ output pane when all tests are executed. An alternative way to ensure a message will always appear in the output is to use ‘Console.WriteLine’ rather than ‘Trace.WriteLine’. This also means the verbose Gallio/MbUnit test run output will show up when executing all tests in a project.

Cache test process between test runs

By default the external test process will be cached when the ‘Run Test(s)’ command is used. This process appears in the tool tray as a rocket icon which can be used to kill the process. This is fine unless one of your tests starts leaking leaking native resources (such as leaving open a file handle). The best solution is to fix the resource leak, but you now have to option to work around the issue by killing the test process at the end of each test run. This can be useful if the resource leak is in a 3rd party DLL which can’t be easily be changed.

Categories

This option supports the most common use of test categories; you can choose to include or exclude a selection of categories. This is useful if you want to exclude long running tests or if your machine isn’t configured to execute integration tests. This feature is currently only supported by the NUnit runner that comes with TestDriven.Net, but I believe it will be supported by a a future version of Gallio.

You can specify your NUnit test categories like this:

    [Test, Category("LongRunning")]
    public void LongRunning()
    {
        Thread.Sleep(10000);
    }

There are lots of features in TestDriven.Net that could be exposed as options. I’ll try to resist this temptation as much as possible, but I’m sure a few more will creep in. ;-)

TestDriven.NET 2.18 + NUnit 2.5 Beta

I’ve just uploaded a new version of TestDriven.Net with support for NUnit 2.5 Beta.

There’s a menagerie of weird and wonderful new attributes to choose from in this point release of NUnit. Ben Hall has written a good summary of the Alpha version and I’m sure Charlie Poole will be blogging about the Beta in the coming days.

Here’s a quick summary of a few new attributes:

Generic test fixtures can be used when you need to run a batch of tests against few different implementations of a type. In the past something similar could be achieved by having an abstract base fixture and extending it for each implementation you needed to test. By using a generic fixture you keep all your test code in a single class and I think it’s more expressive.

    [TestFixture(typeof(ArrayList))]
    [TestFixture(typeof(List<int>))]
    public class IListTests<TList> where TList : IList, new()
    {
        [Test]
        public void Count()
        {
            IList list = new TList { 1, 2, 3 };
            Assert.AreEqual(3, list.Count);
        }
    }

The ‘TestCase’ attribute is similar to MbUnit’s ‘RowTest’. With this attribute you transform a single test method into multiple test cases. You can also define an expected return result, but I’d advise against using this if you want a stack trace when your test fails. It’s better to explicitly define the assert inside the test method.

    public class TestCases
    {
        [TestCase(4, 2, 2)]
        [TestCase(2, 1, 1)]
        [TestCase(5, 2, 3)]
        public void Add(int answer, int a, int b)
        {
            Assert.AreEqual(answer, a + b);
        }
    }

In previous versions of NUnit you could specify which threading model your tests required by adding some XML to your test project’s App.config file. You can now specify this directly on the test that needs it using the RequiresMTA/STA attributes.

    [Test, RequiresMTA]
    public void MTA()
    {
        Assert.AreEqual(ApartmentState.MTA,
            Thread.CurrentThread.ApartmentState);
    }
    [Test, RequiresSTA]
    public void STA()
    {
        Assert.AreEqual(ApartmentState.STA,
            Thread.CurrentThread.ApartmentState);
    }

Lastly the ‘TestFixture’ attribute is no longer required and test methods are allowed to be static. This means NUnit can now be used in a natural way for testing F# code. Note, you will need to have "Other Flags" set to "--optimize+ notailcalls" in your project’s build properties if you want to see a stack trace on any failed asserts.

    #light
    open NUnit.Framework
    [<Test>]
    let fsharp() = 
        Assert.AreEqual(2 + 2, 4)

Update: I’ve tried to highlight a few features in NUnit 2.5 which can be used to make your unit tests clearer. Xerxes Battiwalla has written a post about Assert.Throws<T>() which also falls into this category.

    [Test]
    public void CreateDomain_Null()
    {
        Assert.Throws<ArgumentNullException>(() => AppDomain.CreateDomain(null));
    }

For more information see the TestDriven.Net 2.18 release notes and the NUnit 2.5 documentation.