CruiseControl, MSBuild, and Unit Tests
While converting our continuous integration process from Nant
to MSBuild, I ran into a problem with unit tests. The build
script runs our unit tests by calling the nunit-console.exe
test runner with the Exec task. The tests reside in a number
of assemblies, so I used an ItemGroup to list test assemblies,
and batched the call to nunit-console.exe (as described in
Peter Provost's blog
posting). I wanted to run all of the tests, even if some of them
failed, and my first thought was to set the ContinueOnError
attribute of the Exec task to true. Unfortunately, there's a
problem with this - it turns out that "Continue On Error"
really means "Pretend task succeeded". In other words, even
with test failures the outcome of the build was "success", and
CruiseControl wouldn't report the build as failed. No good.
The solution to this turns out to be trickier than it probably
should be, but it is possible. The idea is to store the exit
code of the Exec task into an item group (one item per Exec),
then use an Error task to check if any of the Exec's returned
errors. Building on Mr Provost's example, the UnitTest target
becomes:
<Target Name="UnitTest" DependsOnTargets="Compile">
<Exec ContinueOnError='true' Command='$(NUnitEXE) $(NUnitArgs) @(TestAssembly)' WorkingDirectory='%(WorkingDirectory)'>
<Output TaskParameter="ExitCode" ItemName="ExitCodes"/>
</Exec>
<Error Text="Test error occurred" Condition="'%(ExitCodes.Identity)'>0"/>
</Target>