Mehfuz's WebLog

Recently, I was bit playing around with amazon S3. This is said to be the cheapest storage for people who don't want to spend much of their brain cells thinking about DB issues. Now, Amazon S3 primarily supports Web service (SOAP) to store and query data out of the S3 server. The concept behind S3 is that you have to create a bucket and under it you can store as many objects you want. Generally for a particular account 100 is the threshold value for buckets. In this post, I will basically focus on configuring your proxy for uploading large files to amazon using WSE and optionally I will point out a cool browser extension for analyzing your S3 space and third party library alternative.

Amazon depends on WSE 2.0 if you want to upload files larger than 1MB. The large file is transferred in the form of DIME (Direct Internet Message Encapsulation) attachment. Microsoft has released WSE 3.0 that uses MTOM which is a W3C recommendation for replacing DIME. But the time of writing this post it is not yet supported by Amazon.

How to prepare your proxy to support DIME  ?  Well, first you have to get the WSE 2.0 SP3 from Microsoft downloads. Once the step is completed, you can copy the Microsoft.Web.Services2.dll to your local folder. Basically, once the WSE 2.0 pack is installed the DLL is added to the GAC so it is not a required step, but if you want to ship it along with your distribution then you can consider doing that.

Now once you have created the proxy from http://s3.amazonaws.com/doc/2006-03-01/AmazonS3.wsdl. You need to do some manual tweaks before your project is ready to go with WSE.

From the Diff for Reference.cs we can see that we have to add a reference to Microsoft.Web.Services2 and inherit from WebServicesClientProtocol instead of SoapHttpClientProtocol

Pretty easy but hang on a bit. While you are instantiating the proxy class you have to add/remove some standard WS filters. Therefore, if we have a method called CreateProxyInstance , with the OutputFilters it will look like

private AmazonS3 CreateProxyInstance()
{
    AmazonS3 proxy = new AmazonS3();

    /// Remove the standard WSE soap headers.
    proxy.Pipeline.OutputFilters.Remove(typeof(Microsoft.Web.Services2.Security.SecurityOutputFilter));
    proxy.Pipeline.OutputFilters.Remove(typeof(Microsoft.Web.Services2.Referral.ReferralOutputFilter));
    proxy.Pipeline.OutputFilters.Remove(typeof(Microsoft.Web.Services2.Policy.PolicyEnforcementOutputFilter));

    /// Add our custom filter to remove the unwanted WSE soap headers.
    proxy.Pipeline.OutputFilters.Add(new HeaderOutputFilter("wsa:"));

    return proxy;
}

Here the HeaderOuputFilter is a inherited from SoapOutputFilter , where we build the SoapEnvolop header without wsa node. Pretty much that's it for setting up the proxy , now let's upload something to the S3 server.

MetadataEntry[] metadataEntries = new MetadataEntry[2];

metadataEntries[0] = mContentType;
metadataEntries[1] = contentLength;

DateTime timestamp = GetCurrentTimeInMilliseconds();
string signature = GenerateSignature("PutObject", timestamp);

using (AmazonS3 proxy = CreateProxyInstance())
{
    MemoryStream stream = new MemoryStream(content);
    Microsoft.Web.Services2.Dime.DimeAttachment dimeAttachment = new 
    Microsoft.Web.Services2.Dime.DimeAttachment("S3Object",   
    Microsoft.Web.Services2.Dime.TypeFormat.Unknown, stream);
    dimeAttachment.ContentType = contentType; 

    /// add the object in wire.
    proxy.RequestSoapContext.Attachments.Add(dimeAttachment);
    proxy.PutObject(_bucketName, key, metadataEntries, content.LongLength, grants, 
    StorageClass.STANDARD, true, _accessKey, timestamp, true, signature, null);
}

From the fragmented code block we can see that "PutObject" is the method for large content not "PutObjectInLine" and along passing the stream, we have to pass the content type as well or less it wont recognized properly even you specify it in the MimeDataEntry array.

So far that is all. I have joined in a tiny class library project that will be helpful if you are getting started with S3 using SOAP. You can download it HERE. About the tool there is a cool firefox addon called S3Fox that gives you an explorer look for analyzing and even let you to add/delete items directly from S3.

Finally, WSE 2.0 does not work in medium trust mode which is an absolute show stopper for CMS and blog solutions. There is a nice S3 library at codeplex can be found at www.codeplex.com/ThreeSharp that don't requires any of these and can be a good alternative for medium trust problem.

Enjoy !!!

This happens to be the first PDC for me. This year a lot of new technologies has been unveiled during the keynotes. One of the most focused is the Windows Azure. This happens to be the one stop shop for building and deploying your service oriented application from local to remote. We all know how slow it is to build service oriented application these days and Windows Azure is just the right tool that can make life easier over the course of time. There is a breakout session by Steve Mark that you might like to checkout if you have missed it. The session is a starting point for setting up your first Windows Azure app and it also shows a way to work with Asp.net MVC for things like uploading photos. Also, I found that you can use Azure for building async based queue service easily with asp.net.

Secondly , with C# 4.0 one of the cool feature that I found is the dynamic keyword. Let's say you have the following simple class

public class Math
{
    public int Add (int x, int y)
    {
         return x + y;
    }
    public float Add (float x, float y)
    {
         return x + y;
    }
}

Now , as you can see there are two overloads for Add. One is with int and another with float. With the current C# 3.5 (for non generic classes) you need to do

Math math = new Math();
Type mType = math.GetType();

object result = mType.InvokeMember("Add", BindingFlags.InvokeMethod, null, math, new object[] {10, 20});
object resultFloat = mType.InvokeMember("Add", BindingFlags.InvokeMethod, null, math, new object[] { 10.5f, 20f });

Console.WriteLine(result);
Console.WriteLine(resultFloat);

Which of course the only way to call float and int overload depending on value type with existing solution. But with C# 4.0 you can also do the following

dynamic math= new Math();

Console.WriteLine (math.Add (10, 20));
Console.WriteLine (math.Add (10.5f, 20f));

As we can see that no invoke method or generics is ever required to call overloads with different type.You can find  more interesting stuffs on C# 4.0 from the cool session by Anders Hejlsberg, if you at any chance missed it.

PDC 2008 is the event for windows 7. It is supplied to all the attendees with the 160GB mobile hard-drive and of course as a DVC copy. The most cool thing is to be able to add vhds as logical drives and being able to set them as primary drive to boot in native mode. Most of the feature shown in the PDC keynotes are by default in protected mode, so you might need to do tweaks for it.

Finally, "M" is the language for defining your domain model and "Quadrant" is a nice tool that harnesses the "M" to design and persist complex domain model visually. Together they are the "Oslo" project. Try the three parts session from microsoftPdc dot com.

In the end, with Microsoft Surface setup in different places of the convention center with people participating in "Scavenger Hunt" challenge, cool product showcase by various players at Microsoft ecosystem and PDC dinner at Universal Studios really made things rocking. May be next PDC will be even more exciting with beta products coming out to life ;).