Mehfuz's WebLog : VS2008

Deploy, test and pack your code - Part 2 Using MsBuild

mehfuzh — Sat, 06 Sep 2008 17:41:46 GMT

In part one of this packing series I have shown, how you can automate your build process, test and optionally mock by using the popular .net deployment tool NAnt. If you don't know what I am talking about and just landed here by help of search engine. Let me put the link below again.

http://weblogs.asp.net/mehfuzh/archive/2008/08/30/deploy-test-and-pack-your-code-part-1-using-nant.aspx

In this post, I will do a replay to mostly what I have talked in my last post but with MsBuild. Let's start with building the code. If you have a chance to see the above link you must have seen that I have to do strict maps of source files under the csc task as well as the references to make things work. Now, searching the NAnt project I have found that it has a cool solution task, which even let you define your own output path for libraries but unfortunately it does not work with .net 3.5 and Visual studio 2008.

Typically , the solution task is defined under NAnt.VSNetTasks.dll also depends on the following assemblies

To use the task inside build script you need to do something like

<loadtasks assembly="ThirdParty\Nant\NAnt.VSNetTasks.dll" />
<solution solutionfile="LinqToFlickr.sln" configuration="release" outputdir="Bin" />

But , it ends up with "Solution format not supported" exception for VS 2008 solution. We can obviously use the csc task instead and someday it will support the solution task but you can do things with MsBuild more easily. Let's see the snippet below

<Project xmlns="http://schemas.microsoft.com/developer/msbuild/2003">

  <Target Name="Build">
    <MSBuild Projects="LinqToFlickr.sln"
          Properties="Configuration=Release"  />
  </Target>
  
</Project>

I have put the snippet under Linq.flickr.Targets. Now, if your solution has multiple projects and you want the output to go to a specific folder you can easily do that using the project's "property page". The advantage here is that, if a developer adds a new file, the build guy don't need to worry about syncing them up and thus if you have a nice set of rules in your team, it can be far more time saving than strict mapping.

The next step is to test and deploy the library. I have defined another .target file for holding the named variables , which I will be using later. I named it include.Targets.

For , those who are curious, declaring property in MsBuild looks like

<Project xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
  <PropertyGroup>
    <TaskDirectory>Tasks</TaskDirectory>
    <NUnit>ThirdParty\NUnit\nunit-console.exe</NUnit>
  </PropertyGroup>

  <UsingTask AssemblyFile="$(TaskDirectory)\MSBuild.Community.Tasks.dll"
           TaskName="MSBuild.Community.Tasks.Zip"
               />
</Project>

As you can see it is possible to reference the properties later on (in this file or other files that includes it) by $(...) format. UsingTask is the MsBuild way of loadtasks in NAnt which also mimics with the namespace directives in C#.

To wrap up everything, I have created a master.proj file which is the entry point. it includes the targets and does the test and additionally zip things up. You can open up the master.proj in VS 2008. This will give you the advantage of intellisense, but here to mention that the intellisense wont work nicely for custom made tasks. Inside the project node of master.proj you can specify default target like NAnt and can make other targets depend on it. So, even if the target is in one of files included, it will still fire up everything else before playing the final target.

<Project DefaultTargets="Deploy" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
....
</Project>

We have built the project, its time to copy them in a folder to do something on them before doing the final packing. I have created a tiny Task called BatchCopy (I really missed one in the community tasks). The usage is pretty simple and it works pretty nice in my project scope

<BatchCopy source="$(SourceDir)\Bin\Release" DestinationFolder="$(BinDir)" ExtensionToExclude=".pdb;" />
<BatchCopy source="$(TestDir)\Bin\Release" DestinationFolder="$(BinDir)" ExtensionToExclude=".pdb;" />

As you can see, it requires a source and destination path and there is another property where I can specify the extensions which I want to skip separated by commas.

Till this line, I have built the project , copied assemblies to a folder, next cool thing is to test them out. Testing definitely to be done by NUnit. In Athena(Flickr API), I have used Typemock to fake request, responses, authentication, etc out so that I can test the whole API while I am on a plane to PDC 08 :-).

In the previous post, I have mentioned Typemock 5 made auto deploy possible for open source projects. Let's get that going.

Step one is to include three tasks

<UsingTask TaskName="TypeMock.MSBuild.TypeMockRegister" AssemblyFile="$(TypeMockLocation)\TypeMock.MSBuild.dll" />
<UsingTask TaskName="TypeMock.MSBuild.TypeMockStart" AssemblyFile="$(TypeMockLocation)\TypeMock.MSBuild.dll" />
<UsingTask TaskName="TypeMock.MSBuild.TypeMockStop" AssemblyFile="$(TypeMockLocation)\TypeMock.MSBuild.dll" />

TypemockLocation is not the application folder rather where I have copied necessary dlls to make the auto deploy possible. Basically to run it as a standalone your deployment folder should have the following dlls in place copied from the Typemock installation folder.

Once everything is in place, the steps are the same that I have mentioned for NAnt. Only, few syntax that you need to watch out with wide eyes.

<TypeMockRegister Company ="Open Source" License="Get one for you" AutoDeploy="True"/>

<TypeMockStart/>
      <Exec ContinueOnError="false" Command="$(NUnit) $(BinDir)\Linq.Flickr.Test.dll"/>
<TypeMockStop/>

Don't forget to add the ContinueOnError="false" that halts the script as soon as there is an error. Once, things are right running the build script will register auto deploy and run over your test project giving you the results.

Lastly, moving to master.proj, I just wanted to zip everything up with the proper content.

<CreateItem Include="$(BinDir)\**\Linq.Flickr.dll;$(BinDir)\**\LinqExtender.dll;readme.txt;" >
      <Output ItemName="FilesToZip" TaskParameter="Include"/>
</CreateItem>

<Zip Files="@(FilesToZip)" Flatten="true" 
                ZipFileName="$(BinDir)\Linq.Flickr.Lib.zip" />

The CreateItem task lets you prepare an item collection based on the parameters you passed in the Include/Exclude attribute and finally outputs the list to a variable that you can use later on (in this case, it is used with the Files attribute of the Zip task).

This is just a simple way of getting things around with MsBuild. To spice a bit more, you can wrap the msbuild command in a batch file, which requires the following two lines

@ECHO ON
C:\Windows\Microsoft.NET\Framework\v3.5\msbuild.exe Master.proj /m:2 /fileLogge

fileLogger will create an output file (msbuild.log) after the build that contains the detail of execution. /m defines that msbuild will run in two process in parallel, this option is pretty good for large solution (for ex, Telerik has plenty of projects in a solution and I found building with msbuild much faster than using the VS IDE), of course having quad core processor with /m:4 will give more boost than in two cores . In my other project which is LinqExtender, I created a script that download source code and setup working base with a click of a button. Therefore, those who check in/out things in codeplex or any other source control environment could find this task useful.

  <TFSGet
            Server="tfs05.codeplex.com"
            Port="443"
            Secured="true"
            Repository="$/LinqExtender/somthing"
            UseUICredentials="false/true"
            Username="$(User)"
            Password="$(Pass)"
            Domain ="snd"
            LocalPath="local path to host the source"
            UnBind = "true/false"
            />

Here, UseUICredential = true, will bring up the TFS login prompt every time, so if you don't want to type in user/pass every time then just leave as it is and populate the username and password parameter with proper values. Unbind will remove the source control binding after downloading , if set to true.

Last but not the least, there is another task that I want to share with you is called XmlFindReplace, this is useful if you are automating VSI package creation which I did in my LinqExtender project (I hate doing this manually every time for deployment :-)) or want to create a starter project with predefined template.

<XmlFindReplace
      FilePath=".\Template\project.xml"
      Ns ="http://schemas.microsoft.com/developer/msbuild/2003"
      Attribute = "Include"
      Element ="Reference"
      Text ="#ASSEMBLY#"
      ReplaceText="LinqExtender, Version=1.4.0.0, Culture=neutral, processorArchitecture=MSIL"
      DestinationFile="$(TempDir)\LINQProvider.csproj"
/>

Here you can do two sets of find and replace. Either replace the attribute value of an element which requires both the Element and its Attribute or just replace the element text in which case you need to provide only the Element name.

So far, you can see that it is possible to do same things with Msbuild that is possible with NAnt, but MsBuild gives more integration to VS environment which sometimes give more power over how the build goes. Earlier, in a post I have shown that at the end of each project file there is a "AfterBuild" and "BeforeBuild" target where you can put your own script to do some special tasks. Also, Msbuild comes as part of .net framework so that you don't ever need to bother about the script runtime and distributes.

Download MsBuildTasks.zip

Hope all these info are useful for you to get going, I have added the tasks with test classes so that you can play, use or extend it as you like.

Bundle SQL script with build process using MSBuild

mehfuzh — Thu, 14 Aug 2008 19:29:49 GMT

We are making a sample application, demo or startup kit or a toolkit with test project that requires some SQL scripts. We generally ship in a readme.txt that says it all. But hey, how about having the script under automated process , like if my SQL Server and VS 2008 is in place, with the click of Ctrl + Shift + B everything gets ready. Here, we will do just right that.

MSBuild provides a lot of community tasks that includes zip, XmlUpdater to SqlServer. You can get it all from tigris.org. After having it installed you can either copy it to your local app directory or reference it using $(MSBuildExtensionsPath). In my case I have copied the dll into the Tasks directory of my application (which is the example app supplied with LinqExtender). Anyway, there is a nice task called ExecuteDDL that will take in one or more sql script files , join with GO separator if required and run it against your database.

To start things off I have created a Sql.Scripts.Targets. About *.Targets file, you can either include them in other targets or *.proj or *.csproj files where you will be using them. Its almost the same with NAnt scripts that terminate with .build extensions.

<Project xmlns="http://schemas.microsoft.com/developer/msbuild/2003" >
    <PropertyGroup>
        <TaskDirectory>Tasks</TaskDirectory>
        <BatchSql>Sql\script.sql</BatchSql>
       <ConnectionString>Server=localhost;Integrated Security=True</ConnectionString>      
    </PropertyGroup>

    <UsingTask AssemblyFile="$(TaskDirectory)\MSBuild.Community.Tasks.dll"
              TaskName="MSBuild.Community.Tasks.SqlServer.ExecuteDDL"/> 
</Project>

As you can see inside Sql.Scripts.Targets , I have added a block of PropertyGroup which declares the variables which I can use later in the script. I have referenced the tasks directory where the task DLL resides. Also, added the batch SQL path (can add multiple separated by ";") and the default connection string. UsingTask is just like any other using statements in your c# code that lets you include the reference of the task which you will be using in this or other files that is dependent on this.

Now, the interesting part. If you ever opened the c# project file by a notepad or any text editor, you must have seen this

<!-- To modify your build process, add your task inside one of the targets below and uncomment it. 
       Other similar extension points exist, see Microsoft.Common.targets.
  <Target Name="BeforeBuild">
  </Target>
  <Target Name="AfterBuild">
  </Target>
-->

As it says, you can add your custom tasks under it and before or after build, it will fire up your task to do the things you want it to do. So, our final touch is to include the target file and to add the ExecuteDDL task under AfterBuild target.

With this let's un-comment the above code and add our block there.

<Import Project="Sql.Scripts.Targets" />
<Target Name="AfterBuild">
   <ExecuteDDL ConnectionString="$(ConnectionString)" Files="$(BatchSql)" ContinueOnError="false" /> 
</Target>

That's it, next time you build the code, right after it will run the necessary scripts to setup your database.

As you can see that any error occurs gives you a prompt message with VS task list window that also gives the user a clue what to do next. Finally, I have added a hollow class library project with the scripts tied up, please download it here.

Hope that helps

REST with LINQ to XML

mehfuzh — Fri, 11 Jan 2008 12:30:32 GMT

With LINQ to XML, what I have learned so far, it's awesome, it is not only the first class concept of processing XML data, but also let you process a complex XML doc in the most simplest and readable way.

I have a project called, Linq.Flickr , and I often need to process XML data for REST query.

To show a simple example of LINQ to XML, lets take a REST response from a Flickr method called flickr.people.getUploadStatus and see how it is processed.

<user id="12037949754@N01" ispro="1">  
<username>Bees</username> 
<bandwidth maxbytes="2147483648" maxkb="2097152" 
usedbytes="383724" usedkb="374" remainingbytes="2147099924" remainingkb="2096777" /> 
<filesize maxbytes="10485760" maxkb="10240" /> 
<sets created="27" remaining="lots" /> 
</user>

Before .net 3.5. it could have been done by GetElementyByTagName("users") - > inside the while loop getting the element, reading value, storing it in private variables , going to next sibling and repeat, if nested , then take the first child and so forth. But , in .net 3.5 using LINQ , the translation is not only just like XML itself, moreover , in XElement.Load , we can directly pass the REST query string.

So, Two, steps to do a REST with LINQ

Build the URL and pass it to XElement.Load
Parse the response and take necessary actions.

For example, for the above XML, I wrote something like below, in PhotoRepository.cs, that does a selective parsing.

try
{

    XElement element = GetElement(##REQUEST URL##);

    People people = (from p in element.Descendants("user")
      select new People
      {
            Id = p.Attribute("id").Value ?? string.Empty,
            IsPro =
            Convert.ToInt32(p.Attribute("ispro").Value) == 0 ? false : true,
            BandWidth =
            (from b in element.Descendants("bandwidth")
               select new BandWidth
                {
                      RemainingKB = 
                      Convert.ToInt32(b.Attribute("remainingkb").Value),
                      UsedKB = Convert.ToInt32(b.Attribute("usedkb").Value)
                 }).Single<BandWidth>()
               }).Single<People>();

    return people;
}
catch (Exception ex)
{
    throw new ApplicationException("Could not process response", ex);
}

Note :GetElement is a local method that does a XElement.Load, with some validation check.

Here my Client-Side object looks like

People
Id
IsPro
BandWidth
RemainingKB
UsedKB

The most interesting thing is, everything is done on the fly , no loop, no intermediate result tracking, no headache of sibling and first child.Overall, the code has a high point in terms of readability.

Take that !!

[New Article] Creating custom LINQ provider using LinqExtender

mehfuzh — Mon, 07 Jan 2008 23:32:39 GMT

First of all happy new year, secondly my new article is just published at Dotnetslackers. This explains the detail of creating custom LINQ providers using Linqextender.

You can check that out at http://dotnetslackers.com/articles/csharp/CreatingCustomLINQProviderUsingLinqExtender.aspx

Moreover, I would like to add some tiny enhancements to the extender.

1. All the extender descendant can now do something like.

var query = (from q in _queryContext
             orderby q.LastUpdated descending
             select new { q.Id, q.Title, q.LastUpdated }).Take(1);
 
int count = query.Count();

2. OpenLinqToSql, which is a part of Linqextender, now, supports SQLCE database.

3. Easy OnError event for exception trapping.

In the end, there are plenty of things that needs to done and to be told, but the article explains exactly how to get started and create a provider in no time. Please take a look , and if possible give some votes.

IOrderedQueryable VS IQueryable

mehfuzh — Wed, 05 Dec 2007 18:27:51 GMT

We have seen that , while making general LINQ query provider , we have to implement IQueryable along with IQueryProvider.

But what about the following query.

var query = (from ph in context.Photos
             where ph.User == User && ph.PhotoSize == PhotoSize.Medium
 && ph.SearchText == "pocketpc, windows mobile" && 
ph.SearchMode == SearchMode.TagsOnly orderby PhotoOrder.Date_Posted descending
             select ph ).Take(10).Skip(0);

As, this query has orderby clause, this will require IOrderedQueryable instead of IQueryable, or less it will end up with build error.

Now, digging deeper, what is the methodcall for orderby "item" descending/ascending

Simple order by , like orderby ph.id / orderby ph.id ascending, the internal method call is OrderBy(ph=>ph.id)

Query

var query = from ph in context.Photos where ph.user = "jcl"
 order by ph.Date_Taken select ph;

but if we use, orderby p.id descending , the internal method call will be made as OrderByDescending(ph => ph.id).Therefore, in CreateQuery<T>, we have to check for OrderByDescending and OrderBy method separately to process data accordingly.

Which could be

MethodCallExpression curentMethodcall = _expression as MethodCallExpression;

_asc = curentMethodcall.Method.Name == CallType.ORDERBYDESC ? false : true;

ProcessExpression(curentMethodcall.Arguments[1], _dummyPhotoObject);

And during ProcessExpression , the expression breakdown will be

UnaryExpression

LamdaExpression

Memberexpresion ( for p = > p.id) or direct ConstantExpression (for p=> "Dake_take")

Source : LinqExtender

Hope this is useful