ISerializable - Roy Osherove's Blog

XUnit.NET aims to be the Next NUnit, but it's too soon

Update 1: Brad answers some of my concerns on his blog. Too bad you can't write comments on his posts...

Another update: There's a syntax comparison matrix between NUnit, XUnit and MbUnit.

James Newkirk, one of the original creators of NUnit, which is currently the most popular Unit Testing framework for .NET (he now works for Microsoft at the patterns and practices team if I'm not mistaken) has announced on his blog that together with Brad Wilson (also a Microsoftie) a new Unit Testing framework which is open source (it is hosted on the Microsoft CodePlex site under the Microsoft Permissive License. the new Unit Testing framework's name is XUnit.NET.

James lists some of the reasons for creating this framework as his feeling (and Brad's) that is' time to re-codify some of the rules that had become more apparent in the past year or so about Unit testing and TDD DOs and Donts that they feel are important.

some of the main differentiating properties James lists are:

• Single Object Instance per Test Method
• No [SetUp] or [TearDown].
• No [ExpectedException].
• Reducing the Number of Custom Attributes.
• Use of Generics and Anonymous Delegates.
• Assert extensibility.
• Test method extensibility.
• Test class extensibility.

I took a rather quick glance and a shot at using the new framework, and wrote some points down so you get a deeper sense of what's changed if you're familiar with NUnit:

some things James did not mention you should know or I'd like to expand on:

Update 2: The lack of setup and teardown methods is not that bad because you can still get the same behavior: just use the class constructor and implement IDisposable and use the Dispose method as your "TearDown" method. This is more of a design thing than a "Removed feature" although it will still break many tests you have that have this attribute.  There is also a [RunWithNUnit] attribute which I'm not sure how it is used yet.

• There is no Assert.Fail (overlooked? I find this very usable in some instances)
• Data Driven Testing using the TheoryAttribute and DataViaXX attributes - this is a really cool feature whis feels like the MbUnit's [RowTest] with extensibility to declare where the rows will come from. It already has "providers" for SQL, OleDB and more. I was able to put together a text file based Data Driven test in less than 5 minutes without any documentatoin. the framework API is very usable especially if you look at the source code. see example of how I did this at the bottom of this post
• It's easy to add a special test attribute that does something before or after the test by by inheriting from a special BeforeAfterTestAttribute
• There is currently only a console runner
• But there is already shipping TestDriven.NET support as part of the download! just run xunit.tdnet.installer.exe  that is next to the xunit.framework.dll
• Assert.Equals is deprecated and ou are instructed to use a generic version of Assert.Equal (without the 's') / I think this is a problem because generics make the Assert code longer to write and read while giving little contribution to the developer of the tests. You could "omit" the initial generic type declaration from the Assert.Equal (so Assert.Equal<string>("a","b") works like Assert.Equal("a","b")) but if you miss on the type of the parameter you get a less than understandable compilation error in the form of "argument type string not assignable to parameter type T"
• There is no built in Mocking framework (NUnit had Nunit.Mocks though most people didn't even know it existed until they've read this)
• Assert.Equal on two strings, if it fails, does not add that nice little touch of visual that Nunit has that shwos you with an ascii arrow "----^" where the strings differ first (it tells you the char index but that's less understandable)
• instead of TestFixtureSetup and TestFixtureTearDown, you need to implement an ITestFixture interface with an "BeforeAllTests" and "AfterAllTests" methods.
• There is no Setup and teardown. I'm not sure that's such a cool idea even though James thinks it is. Like it our not it does give a nice place to store and reuse code. yes, you can abuse it, but does that mean C# shouldn't support anonymous delegates because people could abuse it?) .  At least give us a Setupand Teardown as part of ITestFixture!
• Extensibility and creating custom test attributes is pretty easy and straightforward and I like the thinking around the API. it's usable. There is still no solution for what happens if multiple custom test attributes need to be run in a specific order.
• No Expected exception attribute. use try-catch or Assert.Throws, I find that less readable. James' explanation of why isn't convincing to me. I never came across a case where that attribute failed on the WRONG exception (especially if you set the expected message attribute on it)
• If you take a look at the tests for XUnit and XUnit.Extensions (part of the src download) you'll see that the naming conventions the use is lacking to my taste. Example: a test called "TheoryWithoutData" - it fills two out of my three required parts of a name: what are you testing, and under what scenario are you testing it under?, but the third one - What is the expected behavior under that scenario? that is missing so you have to read the test code to find out, and that sucks.
• From looking at the source code, it should be able to also run NUnit tests (backwards compatible)

Can XUnit.NET be the Next NUnit?

XUnit has some big things going for it: it is built by the original creator of NUnit who has a LOT of influence in the .NET world. Personally I really get the feeling that I should start using this framework now simply because of who built it. but that is a mistake, and I explain shortly.  the other big things which it has:

• A nice and simple API and extensibility model
• The idea of data driven tests
• hmm.. that's more or less the big stuff. now that I think about it.

It COULD be. but it's not there. simply because the user base is non existent. and what about the developers? are there only 2? how much time do they have on their hands?

Why not contribute to an existing framework?

I don't know why they had gone and built their own. perhaps it's an Ego thing. perhaps they couldn't come to grips with Charlie Poole (the current chief maintainer of Nunit) about what features go in and especially removing some of the existing features. maybe its a microsoft thing where you can't contribute to an outside open source project. perhaps James or Brad can shed some light on this.

Should you start using XUnit?

The short answer is no.

If you have an existing project with NUnit or MbUnit tests, I wouldn't recommend it until there is either feature parity or until you decide you can live without some of the features that don't exist in XUnit (Setup and teardown, test fixture setup and teardown, assembly related attributes and more. Also not that some people tend to have the tests rely on the fact that the same class fixture instance is used to run all the tests in that class so they *share* data between tests in the class (that's a mistake, but if you rely on this you'd be in a world of trouble to move). Don't move an existing project because it will take a lot of effort!

If you have a new project you should not consider using it. it's still risky since it's a new project and may not be too stable yet. the larger the user base, the more stable it is.  unless it's a proof of concept kind of thing, or if you're just testing the waters , it may be OK. right now, I think it's premature.

Example: Data Driven tests with the Theory attribute

//The test that reads lines from a text file
 [Theory]
        [DataViaTextFile("..\\..\\logicdata.txt")]
        public void SimpleTest(string s)
        {
            Console.WriteLine(s);
        }

//My custom data reader attribute:
 class DataViaTextFileAttribute:DataAttribute
    {
        private string file;

        public string FileName
        {
            get { return file; }
            set { file = value; }
        }

        public DataViaTextFileAttribute(string file)
        {
            this.file = file;
        }

        public override DataTable GetDataTable(Type typeUnderTest)
        {
            DataTable dt = new DataTable();
            dt.Columns.Add("text", typeof (string));
            string[] lines = File.ReadAllLines(file);
            foreach (string line in lines)
            {
                dt.Rows.Add(line);
            }
            return dt;
        }
    }