CruiseControl, MSBuild, and Unit Tests
While converting our continuous integration process from Nant to MSBuild, I ran into a problem with unit tests. The build script runs our unit tests by calling the nunit-console.exe test runner with the Exec task. The tests reside in a number of assemblies, so I used an ItemGroup to list test assemblies, and batched the call to nunit-console.exe (as described in Peter Provost's blog posting). I wanted to run all of the tests, even if some of them failed, and my first thought was to set the ContinueOnError attribute of the Exec task to true. Unfortunately, there's a problem with this - it turns out that "Continue On Error" really means "Pretend task succeeded". In other words, even with test failures the outcome of the build was "success", and CruiseControl wouldn't report the build as failed. No good.
The solution to this turns out to be trickier than it probably should be, but it is possible. The idea is to store the exit code of the Exec task into an item group (one item per Exec), then use an Error task to check if any of the Exec's returned errors. Building on Mr Provost's example, the UnitTest target becomes:
<Target Name="UnitTest" DependsOnTargets="Compile">
<Exec ContinueOnError='true' Command='$(NUnitEXE) $(NUnitArgs) @(TestAssembly)' WorkingDirectory='%(WorkingDirectory)'>
<Output TaskParameter="ExitCode" ItemName="ExitCodes"/>
</Exec>
<Error Text="Test error occurred" Condition="'%(ExitCodes.Identity)'>0"/>
</Target>