January 2011 - Posts

Using Google Protocol Buffers Hypermedia Type with WCF RESTful Services: A media type processor sample

Protocol Buffers is language neutral format for serializing structured data in a very optimal format. You can think about protocol buffers as XML or JSON but lighter and smaller. This format its widely used at Google to exchange data between different systems.

Given its optimal representation of structured data, protocol buffers it’s a great option for representing data processed by RESTful services. To follow the principles of REST, protocol buffers should be a represented as a new hypermedia type. In the current version of Windows Communication Foundation (WCF), incorporating a new media type requires a significant amount of effort. Fortunately, the new version of the WCF HTTP stack makes media type a first class citizen of the WCF programming model. You can find more details about this new stack on Glenn Block’s weblog.

The following section will illustrate how to leverage Google’s protocol buffers as a native media type in the WCF stack.

Implementing a protocol buffers media type processor with WCF-HTTP

The new WCF HTTP programming model abstracts the processing of a hypermedia type by using a component known as media type processors. Essentially, media type processors are represented by classes that inherit from the MediaTypeProcessor class and are invoked by the WCF runtime to handle the serialization incoming and outgoing messages in a specific media type representation.

The first important thing we need to figure out in order to implement a protocol buffers media type processors is the correct API to use. From the existing implementations, I believe the Marc Gravell’s implementation provides the cleanest programming model. Serializing an object into protocol buffers is as simple as illustrated in the following code.

   1: MemoryStream stream= new MemoryStream();
   2: SampleObj obj= new SampleObj();
   3: Serializer.Serialize(stream, obj);

We can leverage the same technique to implement a protocol buffers media type processor as highlighted in the following code:

   1: public class ProtocolBufferProcessor: MediaTypeProcessor
   2:     {
   3:  
   4:         public ProtocolBufferProcessor(HttpOperationDescription operation, 
   5:                                        MediaTypeProcessorMode mode)
   6:             : base(operation, mode)
   7:         {
   8:         }
   9:  
  10:         public override IEnumerable<string> SupportedMediaTypes
  11:         {
  12:             get
  13:             {
  14:                 yield return "application/x-protobuf";
  15:             }
  16:         }
  17:  
  18:          public override object ReadFromStream(Stream stream, 
  19:                                                HttpRequestMessage request)
  20:          {
  21:              throw new NotImplementedException();
  22:          }
  23:  
  24:          public override void WriteToStream(object instance, Stream stream,
  25:                                             HttpRequestMessage request)
  26:          {
  27:              Serializer.Serialize(stream, instance);
  28:              stream.Position = 0;
  29:          }
  30:  
  31:     }

As illustrated above, the WriteToStream operation handles the serialization of the .NET objects into the protocol buffers format.

Let’s explore how to use our media type processor with the following WCF service:

   1: [ServiceContract]
   2: [AspNetCompatibilityRequirements(RequirementsMode 
   3: = AspNetCompatibilityRequirementsMode.Allowed)]
   4: public class AccountResource
   5: {
   6:     private List<Account> accounts = new List<Account>() { 
   7:         new Account(){AccountNumber= "12345", Id= 1, 
   8:                      Balance= 10000000,
   9:                       Description= "Sample Account"},
  10:         new Account(){AccountNumber= "12345", Id= 1, 
  11:                      Balance= 10000000,
  12:                       Description= "Sample Account"},
  13:         new Account(){AccountNumber= "12345", Id= 1, 
  14:                      Balance= 10000000,
  15:                       Description= "Sample Account"}};
  16:  
  17:     public AccountResource()
  18:     { }
  19:  
  20:     [QueryComposition]
  21:     [WebGet(UriTemplate= "")]
  22:     public Account GetAccounts()
  23:     {
  24:         return accounts.AsQueryable<Account>();
  25:     }
  26:  
  27: }
  28:  
  29: public class Account
  30: {
  31:     private string accountNumber;
  32:     private int id;
  33:     private string description;
  34:     private double balance;
  35:  
  36:     public string AccountNumber
  37:     {
  38:         get { return accountNumber; }
  39:         set { accountNumber = value; }
  40:     }
  41:  
  42:     public int Id
  43:     {
  44:         get { return id; }
  45:         set { id = value; }
  46:     }
  47:  
  48:     public string Description
  49:     {
  50:         get { return description; }
  51:         set { description = value; }
  52:     }
  53:  
  54:     public double Balance
  55:     {
  56:         get { return balance;}
  57:         set{balance= value;}
  58:     }
  59: }
  60:  

We can inject the protocol buffers media processor into the WCF runtime by implementing the following host configuration as illustrated in the following code.

 

   1: public class AccountResourceConfiguration : HttpHostConfiguration, 
   2:                                            IProcessorProvider
   3: {
   4:     public void RegisterRequestProcessorsForOperation(
   5:                               HttpOperationDescription operation,
   6:                               IList<Processor> processors,
   7:                               MediaTypeProcessorMode mode)
   8:     {
   9:         
  10:         processors.Add(new JsonProcessor(operation, mode));
  11:         processors.Add(new FormUrlEncodedProcessor(operation, mode));
  12:          
  13:     }
  14:  
  15:     public void RegisterResponseProcessorsForOperation(
  16:                                  HttpOperationDescription operation,
  17:                                  IList<Processor> processors,
  18:                                  MediaTypeProcessorMode mode)
  19:     {
  20:         processors.Add(new JsonProcessor(operation, mode));
  21:         processors.Add(new ProtocolBufferProcessor(operation, mode));
  22:     }
  23: }

We can initialize the host configuration in the route tables structure as shown in the following code.

   1: public class Global : System.Web.HttpApplication
   2:    {
   3:  
   4:        protected void Application_Start(object sender, EventArgs e)
   5:        {
   6:            var configuration = new AccountResourceConfiguration();
   7:            RouteTable.Routes.AddServiceRoute<AccountResource>("accounts", 
   8:                                      new AccountResourceConfiguration());
   9:        }
  10:  
  11:        
  12:    }

At this point, our WCF service is ready to leverage the protocol buffers hypermedia format. The client should indicate the expected media type by using the Accept HTTP header as shown in the following code.

GET /ProtocolBuffers/accounts">/ProtocolBuffers/accounts">/ProtocolBuffers/accounts">http://<service endpoint>/ProtocolBuffers/accounts HTTP/1.1
User-Agent: Fiddler
Accept: application/x-protobuf
Host: localhost:8081

The response message using the protocol buffers format looks like the following:

HTTP/1.1 200 OK
Cache-Control: private
Content-Length: 108
Content-Type: application/x-protobuf
Server: Microsoft-IIS/7.5
X-AspNet-Version: 4.0.30319
X-Powered-By: ASP.NET
Date: Fri, 21 Jan 2011 04:15:32 GMT

"
12345Sample Account!?????cA
"
12346Sample Account!?????9?A
"
12347Sample Account!???`P??A

 

Even though Google’s protocol buffer is nowhere near in popularity as other serialization formats such as XML or JSON it is, undoubtedly, an very efficient hypermedia type to use in your RESTful services.

We are hiring (take a minute to read this, is not another BS talk ;) )

I really wanted to wait until our new website was out to blog about this but I hope you can put up with the ugly website for a few more days J.

Tellago keeps growing and, after a quick break at the beginning of the year, we are back in hiring mode J. We are currently expanding our teams in the United States and Argentina and have various positions open in the following categories.

  • .NET developers: If you are an exceptional .NET programmer with a passion for creating great software solutions working on great teams we want to hear from you! Don't be afraid if you are not skillful with server-side technologies such as BizTalk or SharePoint, you will have the opportunity of learning those.
  • BizTalk architects: Would you like to team up with some of the top BizTalk-WCF architects in the world? As part of our SOA practice, we have several open positions for BizTalk architects in both the United States and Argentina.
  • SharePoint architects: Our SharePoint practice is growing aggressively and expanding into hot areas such as business intelligence or enterprise search. Currently, we have several positions open for SharePoint architects in both the United States and Argentina
  • Program managers (agile coaches): Tellago is innovating in agile techniques for distributed teams. We are currently looking for program managers who have a passion for agile methodologies and who are crazy and dynamic (in that order) enough to work with us.

Why work with us (not for us)?

With so many Microsoft partners you have to rightfully question what in the world makes Tellago different. Here is a first shot?

  • Do things different at a bigger scale: At Tellago, we are not interested on building another average consultancy floated with unoriginal ideas. We are really trying to do things differently by building a team that can innovate while still delivering great solutions and always do the right thing by the customer.
  • Work on the latest technologies: With us you are always going to be playing with the latest releases of Microsoft and several open source technologies even before they hit the market.
  • Work for a company that values talent, team work and passion: If you have the talent and are passionate about building great software then you are guarantee to grow as we grow.
  • Be challenged: Learn, learn, learn. With us, you are guaranteed to be challenged every day to improve your technical skillset and to innovate in the solutions you deliver for our customers.
  • Build stuff that matters: At Tellago we are spending a lot of time focusing on technologies that are changing the world even though if we are not using some of those technologies in our current projects.
  • Team up with the best: Join a team that already includes technology thought leaders such as Pablo Cibraro, Chris Love, Dwight Goins, Don Demsak and others.
  • Contribute to Tellago Studios: While you work at Tellago, you will have the opportunity to collaborate on the Tellago Studios products as well to get involved in the future roadmaps.
  • Contribute to open source: We want you to spend time contributing to open source initiatives. Check out Tellago Dev Labs.
  • Build up your career: We want you to use Tellago to make your career better. We will encourage you to complement your daily work in projects with community activities such as writing, blogging or speaking at several conferences or user groups.
  • Make us better: We are far from being a perfect company, but we try really hard to improve every day. We want you to bring your passion to this company and help us grow, succeed and sometimes fail together.

Of course, chances are that you might have heard similar pitches before so I will let some of our results in 2010 speak for themselves http://weblogs.asp.net/gsusx/archive/2010/12/28/tellago-amp-amp-tellago-studios-2010.aspx.

If you are interested in our current positions please email me at jesus dot rodriguez at tellago dot com. If you are interested to join our team in Argentina, you can reach out to Pablo Cibraro at pablo dot cibraro at tellago dot com.

WE DO NOT CARE IF YOU DON'T HAVE A GREAT RESUME!!!!!! We care about your talent and passion.

Using a service registry that doesn’t suck part II: Dear registry, do you have to be a message broker?

Continuing our series of posts about service registry patterns that suck, we decided to address one of the most common techniques that Service Oriented (SOA) governance tools use to enforce policies.

Scenario

Service registries and repositories serve typically as a mechanism for storing service policies that model behaviors such as security, trust, reliable messaging, SLAs, etc. This makes perfect sense given that SOA governance registries were conceived as a mechanism to store and manage the policies associated with a service. When implementing a SOA governance strategy, you quickly need to decide how are you going to author and enforce policies across the services and clients in your SOA infrastructure.

The screwed-up traditional solution: Centralized policy enforcement

In recent years, it has become more obvious than ever that traditional SOA Governance tools have major challenges managing and enforcing policies on services built on the recent releases of the major web services stacks.

Why is that?

Well, the answer has to do with the fact that web services stacks such as Windows Communication Foundation(WCF), Oracle Metro or Apache Axis2 have evolved way faster than SOA governance technologies such as SOA software or Systinet. To think on a good analogy, think about trying to fix a modern airplane in a car garage; whereas you might be able to pull it off, you will certainly be very constrained to the types of problems you can fix ;)

airplane[1]

Similarly to this somewhat silly analogy, the policies that are available with traditional SOA governance tools are very limited compared to the capabilities of modern web services engines. What is an even worse, traditional SOA governance frameworks rarely provide native integration with web service stacks in order to apply these type policies.

Think about it, when a tool built on a very limited web service stack attempts to manage all web service frameworks in the market chances are that will end up not managing any of them very well J Don’t you think?

To overcome these limitations, SOA governance tools have taken a very simplistic approach of not only being the single policy repository but also acting as the centralize policy enforcement point. The following diagram illustrates this concept.

CentralizeMessaging[1]

In this model, the SOA governance engine will enforce the service policies by brokering the communication between client and services. Looks beautiful to me J Do you see anything wrong with this approach? One simple thought, IT DOES NOT SCALE!!!!! Time and time again we’ve witnessed SOA governance tools that follow this pattern become a performance and scalability bottleneck in SOA infrastructures. Bottom line, SOA governance tools should act as metadata repositories and NEVER as message brokers.

Can we do better?

The SO-Aware way

In SO-Aware, we decided to take a different approach to this problem by not acting as a message broker and instead let developers host and scale their web services using traditional hosting infrastructures such as IIS, WAS or the Windows Server AppFabric. How do we accomplish this? Very simple, SO-Aware exposes policy artifacts such as WCF configuration files as RESTful resources accessible via an HTTP GET. We then have specific extensiblity points for the various Web Services technology stacks that are responsible for downloading the policies and applying to the respective services and clients. The following figure illustrates this concept.

FedPolicies[1]

In the case of WCF, this approach allows to completely centralize the configuration of WCF service and client applications. For instance, we have current production deployments on which SO-Aware is centralizing the binding and behavior configurations for hundreds of WCF services and clients across different environments. This approach enforces the configuration management and consistency without really imposing constraints in the performance or scalability of the solution.

I hope this makes some sense? I would love to hear your thoughts ;)

Tellago 2011: Dwight, Chris and Don are MVPs

It’s been a great start of 2011. Tellago’s Dwight Goins has been awarded as a Microsoft BizTalk Server MVP for 2011. I’ve always said that Dwight should have been an MVP a long time ago. His contributions to the BizTalk Server community are nothing but remarkable.

In addition to Dwight, my colleagues Don Demsak and Chris Love also renewed their respective MVP award. A few other of us are up for renewal later in the year.

As a recognition to Dwight’s award, we have made him the designated doorman for our Tellago party at the upcoming MVP Summit!!!!

More Posts