MSBuild, NAnt, NUnit, MSTest, and frustration

Oh bother. Visual Studio 2003 and Cruise Control.NET. Simple and elegant. A basic NAnt script to build the solution and you're good to go. Run NUnit, output goes to a format CC can understand and Bob's yer uncle. Let me quantify this. Our cruise server has a subversion client (command line) and the .NET 1.1 SDK. Visual Studio isn't installed because, duh, it's a server and cruise just needs something to build the system with.

Enter Visual Studio 2005. I just recently setup CI for our 2005 projects but it's just plain ugly, in so many ways. First there was trying to get the system to build using MSBuild. That was fine because you can simply enter this:

msbuild /t:rebuild solutionname.sln

(or whatever target you want like Debug or Release)

Like I said, if that's all it was no problem but it gets real ugly real fast.

First there's VSTS unit test projects. The team is all equipped with Visual Studio Team Suite. An expensive proposition, but one made long ago by someone wiser than me. No problem though. We're not really using the Test Manager much, there are no automated web tests, so we just write unit tests (and run them with Jamie Cansdales excellent TestDriven.NET). However when MSBuild gets ahold of a solution that contains a VS unit test project it needs some additional set of assemblies (assemblies buried in the GAC_MSIL and other places). The snippet in the ccnet.config file to get MSBuild going was pretty straight forward:

<msbuild>
    <executable>C:\WINDOWS\Microsoft.NET\Framework\v2.0.50727\MSBuild.exe</executable>
    <workingDirectory>D:\ccnet\projects\ProjectName</workingDirectory>
    <projectFile>SolutionName.sln</projectFile>
    <buildArgs>/noconsolelogger /p:Configuration=AutomatedBuild /v:diag</buildArgs>
    <targets>Build</targets>
    <timeout>600</timeout>
    <logger>C:\Program Files\CruiseControl.NET\server\ThoughtWorks.CruiseControl.MsBuild.dll</logger>
</msbuild>

Through brute force (run MSBuild, figure out what assembly it needs, go find it, lather, rinse, repeat) I was able to get a 2005 solution (with unit test projects) to compile. However actually running the tests is another story. Again brute force reigned supreme here as I trodded through an hour or two of running MSTest.exe to try to coax a couple hundred unit tests to run. The cc.config entry for getting some unit tests looks something like this:

<exec>
    <executable>C:\Program Files\Microsoft Visual Studio 8\Common7\IDE\MSTest.exe</executable>
    <baseDirectory>d:\ccnet\projects\ProjectName</baseDirectory>
    <buildArgs>/testcontainer:Source\Tests\UnitTests\bin\AutomatedBuild\UnitTests.dll /runconfig:localtestrun.Testrunconfig /resultsfile:testResults.trx</buildArgs>
    <buildTimeoutSeconds>600</buildTimeoutSeconds>
</exec>

Remember that I said this sever did not have Visual Studio installed. For the VS2005 solutions, I just installed the .NET SDK 2.0 and that was good enough. Although MSTest.exe isn't included, I snagged the file (and it's crazy set of additional DLLs scattered all over the hard drive) from another system and stuck it where the EXE was so it would be happy. 

No dice on running MSTest against a unit test project. It started down the path of running the test, but then hit a crazy COM error (CLSID not registered) and that was enough for me. No way I'm going to track down all the stupid COM registry settings for Visual Studio 2005. And what the hell was this? COM? I thought we got rid of that about 3 compilers ago?

So I was stuck to installing a full Team Suite on the server. Okay, I'll bite. I mean, I don't need everything so it'll be okay (I keep telling myself as I watch a million registry entries fill up). A few hours later and I'm staring at my command prompt again and type in my MSTest.exe command. And a dialog box pops up. Yup, a modal dialog box that tells me something is wrong (I don't remember the specifics but it wasn't very informative).

Visual Studio was installed fine and I could compile and run and build the solution I was trying to, but testing from the command line with MSTest.exe was a no-go. No matter how hard I tried, how much I cleaned up, or how many virgins I sacrificed it just won't go. And there's another problem. With 2003 and our NUnit tests, we go some nice stats. Timings, informative messages, all that good stuff. With MSTest we get nothing (other than x number of tests ran and maybe a failure, but I couldn't get that far to see if it would give the details of the failure). On top of that, the MSBuild logger bites and produces about 10,000 lines of gobbly-gook that isn't very useful. Yes, I could spend my time writing new xslt to make it pretty, trim out the "Everything was okay" lines that fill up the MSBuild task logger but I think that's not value-added at my rates.

Here's a sample email I got from the build which gives you an idea of how useless a CC.NET/MSBuild/MSTest combo is:

It's ugly. Really ugly. MSBuild produces a ton of crazy output. Writing a new MSBuild file is no walk in the park and even for someone like me that knows NAnt quite well, I'm still wrapping my head around how MSBuild does stuff. I had to create a special configuration inside of Visual Studio to omit our Report project because MSBuild can't build them (hence the target AutomatedBuild above which is not the same configuration that our developers use and something I don't like doing because that's one point of a CI server, consistent builds for everyone).

From the output above, Cruise said the build was okay but there's an error message that came out of the MSBuild logger (our report project). This is a 2005 project so the information makes no sense (I belive it's a bug that's logged but who knows when it might get fixed). And I really can't integrate the MSTest output into the email because, well, there is none. There's a hundred tests or so in this project but the logger doesn't produce anything. 

Additionaly, there are some project types MSBuild can't handle and again, it takes a rocket scientist to create an MSBuild file (even using something cool like MSBuild Sidekick) that can call another MSBuild task and exclude certain projects. This certainly isn't as easy as it was in 2003 where you just created an exclusion list (we excluded our client app as there was no extra license for the grid control and a modal dialog came up when the trial version was even looked at by the compiler).

Okay, this is mostly a rant but there's some wisdom here. Continuous Integration does not need to be this hard. CruiseControl.NET is an excellent tool and very flexible with new tools and integrating output from those tools. However when those tools require a million registry settings and even more DLLs (put in very specific places, trust me, you can't just toss them in the GAC and call it a day) and dump gobs of XML that no mere mortal (well maybe DonXml could) would be able to translate, it's just wrong. And as for the built-in Team Build that you could us, that's equally as useless as a) you can't schedule builds to trigger off of code checkins and b) again it requires a full Team Suite client to be installed on the server. Worst case scenario when I first started setting up CC.NET servers it took 4 hours. Now I can get it done in an hour (which includes testing the build of the first project). I've already spent a good day on just trying to get something to compile. It's compiling now but the output is crap and no running of tests and thats just not good enough for me. You can get MSBuild and (maybe) MSTest running with CruiseControl.NET. It's not impossible. If you install the full client and your projects are "just right" it'll work but the output isn't that hot and you'll watch your server CPU slam when compiles happen.

My advice, avoid trying to combine MSBuild, MSTest, and CruiseControl and stick to NAnt, and NUnit (using MSBuild if you have to when building 2005 solutions).

Needless to say I'm looking at one option right now. Dumping the install of VS2005 on the server, changing all our unit tests back to NUnit and using NAnt to do everything (and just calling MSBuild as an exec task for VS2005 projects). I still need to run 2003 projects so our CI server is going to look like a Frankenstein monster by the time I'm done, building VS2003 and VS2005 projects, running NUnit, MSBuild, NAnt, Simian, FxCop and whatever else we have in our mix. Even given that, using NAnt the output will be simpler (and integrate well with CC.NET), test output will be useful and informative, and I don't need to install a $15,000 piece of software on a server that has the distinct possibility to pop-up a modal dialog someday when it can't find some registry key.

Grrr. Argh.