September 2007 - Posts
Members of the MbUnit team will be on the road in the coming months. Jeremy alluded to the fact that MbUnit will have a presence at next months alt.net conf and this will in fact be in the shape of Jeff Brown, the primary lead on the MbUnit Gallio framework. Jeff will of course be talking all about Gallio and the next generation of MbUnit so his session will be well worth catching.
In November the annual 1 day DDD conference will be held here in the UK. Sessions require voting and if you are interested in learning from team member (and red-gate test engineer) Ben Hall about testing, improving your testing and what Gallio is all about then be sure to vote for Ben's session.
I was away on vacation this week and when ever I go away the week is eventful, I came back to the news that Phil Haack has taken the red pill and James\Brad have been busy. My congrats to Phil on his news, I look foward to working with Phil as part of the ASP Insiders. James and Brads (met Brad last year, not sure if Brad remembers but he's a nice guy) new framework, xUnit.net is an interesting addition to the current set of test frameworks in .NET land. Roy, Scott and Bill all have views and Brad has addressed some questions and I'll let you look there for further information. I'll add my thoughts on XUnit only where MbUnit is involved, I'll start with Bill's points first.
Like Roy, I'm not about to jump on this bandwagon and I don't suggest you do either. If you're looking to start writing new tests with it, I guess it's okay (other than the things mentioned above) but that's for you to decide. If you're looking to replace NUnit or MbUnit with XUnit.NET forget it. It's not worth it as a lot of your tests might break or worse case, you'll have to actually just replace code via rename because of the syntax.
XUnit does add something new and different to the mix and its good to see a different path being tread. For those of you looking at making a switch as always you need to balance the fact that XUnit is still new, may have bugs and will not have not matured yet. Of course if your using NUnit then you must consider in NUnit you have plenty of docs/books an examples coupled with unqiue features like rich Mono support.
In both MbUnit\NUnit XUnit.Net covers some ground but also looses some ground too. No support for Collection or Array asserts (and in MbUnit that extends to Compiler, Reflection and File asserts). Sure you could write these for XUnit and prehaps based on the same mech as NUnit means your sharing some code. However the features and tools in both MbUnit\NUnit are ready baked\tested and used for you to put to use right away in addressing your testing needs.
On the positive side, there is extensibility but I have to ask. Why oh why can't we all get along? NUnit is great but I've recently (the last couple of months) shifted over to MbUnit which seems to have a little more life and there are some nice features I'm getting productivity out of. Why couldn't say the 3 cheeses (Andrew, James, and Charlie) get together to say "Let's build a unified framework". On unit testing framework for .NET to rule them all.
I speak to Charlie as often as I can and we are both seeking to expand co-op across MbUnit\NUnit for the good of our use bases. For example I am keen to expand our reporting format co-op so that tools like CI tools can work with one format reguardless of test framework. It is areas such as this that do deserve the most attention, prehaps Microsoft could mediate this, who knows.
Data driven testing
One thing I have heard a little about from Roys posts is the XUnits datadriven testing attribute Theory, datadriven tests are not soley unique to MbUnit. MSTest has database and file based data fixtures and outside of .NET you can also find them in TestNG. The row attribute is popular in MbUnit but she also has fixtures for the even more powerful Pair-Wise testing approach. We are going further with this in Gallio and alpha 1 is edging every nearer.
While you wait Jeff has today announced his extensions for MbUnit 2.4 that are in the vein of Gallio but allow you to enpower your row tests using MbUnit 2.4.
XUnit Test matrix
A test matrix between NUnit\MbUnit\XUnit and MSTest was brought to my attention and it's only fair that I add corrections if your looking at it also.
I based this post from a post by Sebastien Pouliot, I was so thrilled that I wanted to write it up here.
Chris Gameweld was a student on this years Mono SoC project and has worked on creating a tool to improve the unit test coverage within the Mono code base. The end result was a tool called FieldStat that checks the frequency of how often methods are called, when used with code coverage results in a test you can see where tests need be shored up. Chris has an interesting proposal.
An approach for favoring the important but uncovered code involves an algorithm that is capable of ranking the importance of code. CodeRank is a technique that is similar in spirit with Google’s PageRank –- important methods link to other important methods. The CodeRank creates an ordered ranking of all the methods where each rating assigned to a method gives its relative percent importance. This rating can be scaled by other factors including call frequency.
FieldStat is using CodeRank and producing some interesting results
A very useful tool in a testers tool box.
Master Jedi Jeff had an itch in between Gallio coding and over on the Gallio contrib coded up a Instrumentation API using Mono Cecil, very sweet. The contrib section is just something that is not on any channel and we keep to throw out our ideas. As Jeff states in his post if other folks like his ideas then it may get developed further, do check out his post and let him know what you think.
Wihin Jeffs post I noticed Jason Bock left a comment and I thought 'I know that name', then it struck me, his written two great books that I own, one on IL and the other on Attributes. I also know Jason from his .NET Languages website and ran into a post from Craig Andera on the site . As Craig states
I believe it's a good idea to learn at least one new programming language a year. If nothing else, it keeps me from getting bored. You might think I'd choose Ruby, as that's pretty hot right now (in certain circles). And in truth I'm interested. I've even read some of The Pickaxe Book. But the language I've decided to devote my time to is Common Lisp.
I can see why Craig wants to do this, he has its own requirements but more so you can find many concepts from Lisp in Ruby as well as those that Lisp holds as its own. While Ruby is a very powerful and very popular language, personally Lisp draws me in more.
James Avery has a link to Jeff Eaton post that asks "Why not ASP.NET [for startup's]?".
Let's consider that if this is a cost thing then it's running Windows (costs) as opposed to Linux (free, unless your paying for support options but its relative). ASP.NET\Visual Studio Express (web edition and c# edition) are free. Jeff's remaining points can be addressed as Webform issues not ASP.NET issues, if your looking for Rails style in ASP.NET then projects like Castle MonoRail shoild be your first port of call. You can find all the power, flexability, community and all that Rails goodness and it's built onto of ASP.NET framework. To be fair to webforms, it's short comings are well know, it's product team is hard working in solving those problems and as a great community (which extends to the ASP.NET framework as a whole) of users and problem solvers\experts like the MVPs and ASP Insiders. To address problem 1, if you want to cut costs and run Linux you can still use ASP.NET using Mono. Mono is cross platform (Linux/Mac/Windows) with cross platform tools and is free as in beer (web host availability issues aside). Couple it with MonoRail and you have Rails like development in ASP.NET and costs you nothing.
So why not just use Rails then?
Rails has convention, baked in patterns, baked in testing, powerful language and a powerful framework and yes its all for free. You can hire Rails devs fairly easily and also find Apache/MySql experts easily too. Would you really every need ASP.NET then, you would be able to find webforms coders easily enough but it would trickier to find ASP.NET devs. Finding devs who understand and use things like patterns and testing would narrow your choices down yet more, the problem is that Rails baked this in from the start and the Microsoft way simply does not so in the general case you will struggle. So general availability of devs that grok good software pratice to build your startup app as durable, changeable and scaleable, it's a no brainer.
So what next
As much as I grok Ruby as a language I do love C#, I know each can do things better than the other but technologies like LINQ are exciting. While the spec space in .NET (NSpec for example) is not as complete as Ruby (RSpec) the unit test space is more mature IMHO, RUnit does the work as Rubys xUnit workhorse but MbUnit and NUnit are really pushing at the bounds. The question of patterns and structure will come and over time the style of devs will (hopefully) change too. It is prehaps what I have now and what the potential is for the future, as a startup all of these things need consideration.
I want to extend my thanks to the following commerical software projects who have donated a license to each member of the MbUnit team to aid in it's development.
R# and TD.Net need little introduction and indeed NDepend is fast becoming a hot tool. I've only just started putting NDepend to use and the tool blows me away, CQL is a very, very powerful way of adjusting queries across the various metrics that are produced in your code. Patrick is also a nice guy and really knows his stuff, I look foward to seeing how we can worth with Patrick in the future.
The MbUnit 2.4.1 release was released tonight, this adds some bug fixes and additional functionality on the 2.4.release. In full the following was fixed or added in this release.
- [MBUNIT-97] - Assembly setup method run after test suites are generated
- [MBUNIT-108] - Test setup and teardown methods are not run for every execution of a repeated test
- [MBUNIT-113] - SetUp and TearDown methods not executed for each test in a ProcessTestFixture
- [MBUNIT-116] - .mbunit project files not associated by Windows to MbUnit GUI
- [MBUNIT-126] - Assert.AreEqual() should support arrays the way NUnit does
- [MBUNIT-128] - DirectoryNotFoundException when application exists
- [MBUNIT-129] - The Assert.Greater overloads behave differently
- [MBUNIT-132] - GenericAssert.FailIsEmpty & FailIsNotEmpty use the unformatted message
- [MBUNIT-134] - ThreadedRepeat attribute cause high cpu utilization.
- [MBUNIT-146] - Problem with AreNotEqual and object
- [MBUNIT-147] - Remove assemblies... menu item shouldn't have ellipsis
- [MBUNIT-148] - Unhandled exception 'previous state not found' when shutting down MbUnit having run no previous tests
- [MBUNIT-115] - RowTest cannot be used with a function that have a paramarray argument
- [MBUNIT-127] - Tree state should save the selected node
- [MBUNIT-130] - Provide Version Numbers in the Assembly Metadata
This is the last release to use Jira, for future issue tracking we will use google code.
My thanks to the following team members who helped in this release, Ben Hall, Dave Griffiths, Joey Calisay, Julian Hidalgo, Vadim Kreynin, Marc Stober and Jeff Brown. In addition Richard Louapre provided a patch. Dave Griffiths in answer to my recent post provided several patches and then after joiniung the team several commits, if like Dave you want to help on MbUnit then let me know. The MbUnit team is growing in numbers each day, other recent additions include Albert Weinert (who created the R# runner for MbUnit) and Mark A. Haley.
Last months blogging went off the boil, looking after my wife, day job and a very intensive time of MbUnit all have a part to play These last few months have been a very busy time for all of the team, improving our docs, making new releases like this and working towards the release of the next generation of MbUnit. Exciting times.
Phils latest point is a cracker. Microsoft cannot for a lot of reasons bundle OSS and if they do what harm would it pose to OSS projects left out in the cold.
If Microsoft had chosen to not write its own test framework, I fear they would have chosen NUnit over MbUnit simply because it’s more well known or for political reasons. Such a choice would have the potential to hurt a project like MbUnit in the never ending competition for users and contributors.
It's worth adding that Charlie and the NUnit guys and me and the MbUnit team are not in this to out do each other. Each project treads a different path and while we have some goals that are the same, each project has different paths. If nothing else the two teams are looking to co-op more on things such as reporting formats to make life easier for our users and applying the frameworks to CI tools etc. A bundle of this kind would have adverse effects on the MbUnit user base as much as a MbUnit bundle would damage NUnit, ZaneBug and CsUnit. In the interests of all our users we are more interested in co-op. Phil has some additional thoughts.
What I would have liked to have seen is for Team System to provide extensibility points which make it extremely easy to swap out MS Test for another testing framework. MS Test isn’t the money maker for Microsoft, it’s the whole integrated suite that brings in the moolah, so being able to replace it doesn’t hurt the bottom line.
I have heard that extensibility points may be a possibility in future versions but this may have several problems, if its wrapped in VSIP it may present a license issue, will the API's be useable as runner API's in each framework. I'd welcome a extension API in team system so MbUnit\NUnit and all the other folks and teams can plug into that eco system, but if its on the plan then Microsoft needs to start talking to us all. I once attended a great workshop at Redmond that exposed all the teams to a group of PHP folks and who in there brutal honesty helped Microsoft make a better product. Prehaps it is time Microsoft consider that to make this kind of thing a reality, or we may never really get any where.
Scott's last post had a link to a new opensource Lisp project for the DLR, called IronLisp (in homeage to IronRuby and IronPython). I've been a fan of Lisp for a number of years and I'll keep my eye on IronLisp to see how it goes. If your interesting in running Lisp on the CLR then should also check out fellow brit Rob Blackwell's L# and Common Larency too. If your interested in trying out Lisp then you should really download a Lisp compiler, Common Lisp (CL) is an ANSI standard (X3.226-1994) and will contain all the language features and object system. The CLR\DNL compilers will still be short of features so to get a true grasp of the language i'd start with a native compiler.
It struck me thinking that if Ruby and Lisp have similar requirements in how there object models work then a MOP makes a lot of sense and what about the DLR. I've mailed John with my thoughts and have since run into a post of Jim's about the OO support in the DLR.
One very obvious thing missing from the current DLR is a standard default implementation of this interface. This would be useful to people who wanted to implement simple scripting languages on the DLR who weren't trying to precisely match the semantics of an existing language. This would also be useful for libraries that wanted to create a generic expando object to pass to consumers. I expect that we'll provide this in the future, but at the moment our attention remains focused on the cases where we need to capture the exact semantics of existing dynamic languages where the interface approach provides the needed flexibility.
Atilla has a comment that suggests this breaks away fron a MOP approach
instead of providing interfaces on objects (and thus internalizing the knowledge required for cross-language interop into object implementations), it is a more flexible approach to define interfaces for metaobject protocols, have each language ship its implementation of it (I call these metaobject protocol implementations "object navigators" or just "navigators"), and allow languages to use other languages' metaobject protocol implementations to access and manipulate those languages' native objects, in addition to using its own MOP implementation.
I look foward to hearing Johns thoughts on this.