Specifying Test Run Configuration in Team Build
I recently had a client who was using different test run configurations during their interactive unit testing to deploy different configuration files to use during testing. It is an interesting idea and one that I was surprised that I hadn’t heard of before. My challenge was to help them understand how to replicate this process using Team Build.
As defined in the Microsoft supported targets definition, the TestToolsTask supports the use of test lists and test run configurations. Unfortunately, the way it is used in Team Build, you have a single test run configuration regardless of how many test lists and VSMDI files you specify.
My client’s requirements were to support multiple configurations for a single test list; Team Build out-of-the-box supports multiple test lists with a single configuration. There are a couple of approaches to handle this but both will require the ability to specify multiple configurations. I proposed that the existing MetaDataFile item can be built upon with additional metadata to capture the test run configuration. This will allow any number of VSMDI files, each with one or more test lists AND a test run configuration. For my client they used the same VSMDI and test lists twice with different test run configurations. The following sample shows the concept that could be added to your TFSBuild.proj file:
<ItemGroup>
<MetaDataFile Include="$(SolutionRoot)\MySolution.vsmdi">
<TestList>BuildVerificationTests;DatabaseTests</TestList>
<TestRunConfig>$(SolutionRoot)\SqlTests.testrunconfig</TestList>
</MetaDataFile>
<MetaDataFile Include="$(SolutionRoot)\MySolution.vsmdi">
<TestList>DatabaseTests</TestList>
<TestRunConfig>$(SolutionRoot)\OracleTests.testrunconfig</TestList>
</MetaDataFile>
</ItemGroup>
As I said above, the existing TestToolsTask is capable of specifying test run configurations but is hard-coded in the Microsoft.TeamFoundation.Build.Targets file to only use the one specified in the property RunConfigFile. To use our newly defined TestRunConfig metadata in exchange for the RunConfigFile property, we can override the existing RunTestWithConfiguration target from the Team Build targets file; this target is called by the CoreTests target to actually perform the testing. You can copy the existing block from the Microsoft Team Build targets into your own TFSBuild.proj or your own custom targets file that you import into your TFSBuild.proj.
NOTE: You might be tempted to modify the original Microsoft Team Builds file. Please do NOT do this.If you modify the Microsoft.TeamFoundation.Build.Targets file, your changes may be wiped out during the next service pack upgrade. You’ll also impact all builds on that server including those that may want the default functionality. Instead, override the required targets in your own custom targets file or your TFSBuild.proj file!
The following snippet shows the code copied from the Team Build targets file as modified to support the updated MetaDataFile items shown above:
<Target Name="RunTestWithConfiguration" >
<TeamBuildMessage
Tag="Configuration"
Condition=" '$(IsDesktopBuild)'!='true' "
Value="$(Flavor)" />
<TeamBuildMessage
Tag="Platform"
Condition=" '$(IsDesktopBuild)'!='true' "
Value="$(Platform)" />
<!-- SearchPathRoot for not Any CPU -->
<CreateProperty
Condition=" '$(Platform)'!='Any CPU' "
Value="$(BinariesRoot)\$(Platform)\$(Flavor)\" >
<Output TaskParameter="Value" PropertyName="SearchPathRoot" />
</CreateProperty>
<!-- SearchPathRoot for Any CPU -->
<CreateProperty
Condition=" '$(Platform)'=='Any CPU' "
Value="$(BinariesRoot)\$(Flavor)\" >
<Output TaskParameter="Value" PropertyName="SearchPathRoot" />
</CreateProperty>
<!-- Test task for the end-to-end build -->
<TestToolsTask
Condition=" '$(IsDesktopBuild)'!='true' and '%(MetaDataFile.Identity)'!=''"
BuildFlavor="$(Flavor)"
Platform="$(Platform)"
PublishServer="$(TeamFoundationServerUrl)"
PublishBuild="$(BuildNumber)"
SearchPathRoot="$(SearchPathRoot)"
PathToResultsFilesRoot="$(TestResultsRoot)"
MetaDataFile="%(MetaDataFile.Identity)"
RunConfigFile="%(MetaDataFile.TestRunConfig)"
TestLists="%(MetaDataFile.TestList)"
TeamProject="$(TeamProject)"
ContinueOnError="true" />
</Target>
There are of course many variations on how you can achieve this, but hopefully this post will point you in the right direction. If your requirement does not require the use of test lists but you’d rather use a test container (i.e., a DLL containing all of the tests you’d like to run), the same approach can be applied to the TestToolsTask from the Team Foundation Power Tools.