Jesus Rodriguez's WebLog

Tellago announces BizTalk Server RFID Data Services

The series of open source releases from our team at Tellago continues….

The newest addition to the RESTful Services APIs for BizTalk Server is here. Over the last few weeks, my colleagues Uri Katsir and Suresh Girirajan have been working on building an REST/OData API for BizTalk Server RFID. The results of this work are the first version of the BizTalk Server RFID API that we’ve released in our Tellago DevLabs workspace.

Using this API, developers can manage the entire BizTalk RFID infrastructure using simple HTTP messages. Here are some examples:

Querying all RFID components

Querying all RFID Process:

http://<RFID OData Endpoint>/BTSRFIDManagementService/BTSRFIDManagementService.svc/RFProcesss

Querying all Devices:

http://<RFID OData Endpoint>/BTSRFIDManagementService/BTSRFIDManagementService.svc/Devices

Querying all Device Groups:

http://<RFID OData Endpoint>/BTSRFIDManagementService/BTSRFIDManagementService.svc/DeviceGroups

Querying all Device Providers:

http://<RFID OData Endpoint>/BTSRFIDManagementService/BTSRFIDManagementService.svc/DeviceProviders

Navigating the hierarchy

Querying a specific resource:

http://<RFID OData Endpoint>/BTSRFIDManagementService/BTSRFIDManagementService.svc/Devices('DEVICE1')

Querying all subgroups for a specific Group:

http://<RFID OData Endpoint>/BTSRFIDManagementService/BTSRFIDManagementService.svc/DeviceGroups('Group1')/DeviceSubGroups

Querying all Devices for a specific Subgroup:

http://<RFID OData Endpoint>/BTSRFIDManagementService/BTSRFIDManagementService.svc/DeviceGroups('Group1')/DeviceSubGroups('Forklift1')/AllSubDevice

Filtering:

http://<RFID OData Endpoint>/BTSRFIDManagementService/BTSRFIDManagementService.svc/RFProcess?$filter=Status%20eq%20'Started'

http://<RFID OData Endpoint>/BTSRFIDManagementService/BTSRFIDManagementService.svc/Devices?$filter=Status%20eq%20'ManuallyDisabled'

As part of our work with customers, we’ve seen the practical applicability of this type of API. RFID solutions are notorious from being deployed across heterogeneous locations that can influence the runtime behavior of the hardware. In those scenarios, having a lightweight mechanism to effectively monitor and configure the RFID hardware is a key element of a successful RFID solution.

If you are working with BizTalk Server RFID, I would encourage trying out BizTalk Server RFID Data Services. Don’t forget to send us your feedback.

NOSQL Databases for the .NET Developer. What’s the fuss all about?

A few days ago, Microsoft Research (MSR) announced Trinity, a graph database that is powering some important initiatives within Microsoft such as Probase. This is the second serious initiative of MSR in the NOSQL space. Last year, MSR released a map reduce engine under the code name Dryad which is now available as part of MSDN Dev Labs.

Regardless of whether Trinity becomes an official Microsoft product or not, this announcement is a testimony to the emerging influence of NOSQL databases for both on-premise and cloud solutions. There are many explanations to the increasing popularity of the NOSQL approach but, essentially, they all boil down to the fact that the world needs new alternatives for storing and indexing the increasingly amounts of semi-structured and unstructured data.

We can’t keep thinking to model the entire world with rows and columns.

We can definitely find alternatives to store unstructured data with relational databases, but we, most of the time, we end up paying a price in terms of complexity and, to some extent, scalability.

As many other evolutionary approaches in the software development industry in recent few years, NOSQL databases have found a home within the open source community but (understandably so) have experience some resistance from the big relational DB vendors such as Microsoft, Oracle or IBM. Despite that, the adoptions of NOSQL databases have been growing exponentially even within big enterprise companies. To cite a recent example, last December the US Federal Government publicly announced its commitment to Apache Cassandra for real time analytics.

Types of NOSQL Databases

When thinking on adopting NOSQL databases, we should be aware that they vary depending on the aspects such as the storage model, APIs and consistency model. Here is a list of some of the most popular models in the current market:

• Wide Column Store (Map-Reduce): This type of databases specializes on large data computation and batch processing. Some of the most notable technology on this space are : Hadoop, Cassandra and Dryad
• Document Stores: This type of databases are optimized to store semi-structure data in the form of documents encoded in formats like JSON, BSON, YAM, etc. Some of the top technologies on this space include MongoDB, CouchDB, RavenDB
• Key Value Stores: Key-value stores are schema-less data stores that allow the application to store its data. The data is usually stored in a data type of a programming language or an object. Thus, there is no need for a fixed data mode. Some great data stores on this space are Windows Server AppFabric Caching, MemcacheDB, Tokyo Cabinet.
• Eventually Consistent Key Value Stores: These type of databases evolve from the eventually consistent transaction model. Some technologies on this space are Project Voldemort and Amazon Dynamo
• Graph databases: These type of databases uses graph structures with nodes, edges, and properties to represent and store information. Some great technologies on this space include Neo4J, Infinite Graph, Microsoft Trinity

A more exhaustive list could be found here.

Can I use NOSQL databases in a .NET solution?

Not only can you but I would encourage you to start thinking about. Currently, there are many NOSQL databases that enable native .NET interface or interoperable APIs such as HTTP. MongoDB, CouchDB, RavenDB, Riak are some of the NOSQL databases that are becoming increasingly popular within .NET enterprise customers.

What about Windows Azure?

Azure enables a NOSQL database as a first class citizen in the form of the Table Service which allows to store large volumes of semi structured data (still modeled as rows and columns though). Additionally, it is possible to get a lot of the existing NOSQL databases working as Azure worker/web roles leveraging the Blob storage as the fundamental persistence infrastructure. When implementing the correct mechanism for partitioning and distributing the data within the Blob storage, this approach can be as efficient (if not sometimes more efficient) than relying on the Table Service. Additionally, you can enable a nice consistency between on-premise and cloud solution in the Windows Azure platform.

We can find similar initiatives for the Amazon AWS cloud using product such as Amazon S3 and Amazon EC2.

Nice theory! Have you done this before?

Absolutely! At Tellago, we are really passionate about adopting new technology trends that, we believe, can make a significant difference in our solutions for our customers. In that sense, last year we started adopting and evangelizing NOSQL technologies with our .NET customers. As a result, we have currently implemented and deployed several solutions that rely on NOSQL infrastructures for various Fortune 500 companies. I will be sharing a lot of the lessons we have learned on those implementations in future posts but I can tell you that we got to experience a lot of the benefits of NOSQL databases in terms of simplicity, agility and scalability. We also use NOSQL databases heavily as part of our work at Tellago Studios.

Load Testing your WCF Service in Two Clicks

A few months back I was talking to a VP of Architecture from one of our customers about service testing practices and he expressed a very blunt viewpoint about load testing: “Developers don’t load test their web services because it's to F…. difficult”.

Hearing this from a very experienced architect who has led some impressive SOA implementations might appear shocking as first but we have to recognize that there is a lot of truth in that statement. However, I don’t believe that the complexity has anything to do with load testing itself. In my opinion, this perception is a consequence of the ridiculous complexity of the web service load testing tools in the market.

In our recently released SO-Aware Test Workbench product, we literally obsessed about the user experience to the point that  WE ENABLED THE CREATION AND EXECUTION OF A LOAD TEST IN TWO CLICKS!

That’s right! In order to create a load test with the SO-Aware Test Workbench you can start by selecting the load profile you would like to execute for your test as illustrated in the following figure (that’s going to cost you one click :) ).

After configuring the parameters of your test, you will have to use a second click (sorry) to run the test itself as highlighted in the following figure.

At that point, you can see the real time behavior of your service based the load test profile.

If you want to share the results of your tests with your colleagues, you can easily export the load test results to .pdf or excel as illustrated below. Unfortunately, this activity might require a couple of extra clicks :)

SO-Aware at the Atlanta Connected Systems User Group

Today my colleague Don Demsak will be presenting a session about WCF management, testing and governance using SO-Aware and the SO-Aware Test Workbench at the Connected Systems User Group in Atlanta. Don is a very engaging speaker and has prepared some very cool demos based on lessons of real world WCF solutions. If you are in the ATL area and interested in WCF, AppFabric, BizTalk you should definitely swing by Don’s session. Don’t forget to heckle him a bit (you can blame it for it ;) )

How fast are my services? Comparing basicHttpBinding and ws2007HttpBinding using the SO-Aware Test Workbench

When working on real world WCF solutions, we become pretty aware of the performance implications of the binding and behavior configuration of WCF services. However, whether it’s a known fact the different binding and behavior configurations have direct reflections on the performance of WCF services, developers often struggle to figure out the real performance behavior of the services. We can attribute this to the lack of tools for correctly testing the performance characteristics of WCF services under different load profiles. As sad as this sound, the fact of the matter is that the existing web service load test tools in the market can barely test anything other than the basicHttpBinding.

With the recent release of the SO-Aware Test Workbench, we attempted to provide a simple solution to the load and performance testing of WCF/Web Services. SO-Aware Test Workbench enables developers to model and execute load tests with only a couple of clicks.

Let’s take a real example:

BasicHttpBinding and ws2007HttpBinding are, undoubtedly, the two most common bindings in WCF solutions in the real world. At some point, we’ve all heard that the ws2007HttpBinding can slightly impact the performance of a service given that, on its default configuration, it leverages WS-SecureConversation to secure the communication between client and service. However, I’ve never seen a good benchmark that details how real is the performance impact of using the ws2007HttpBinding.

Time to find the answer!!!

Using the SO-Aware Test Workbench, we can run a series of load tests against services configured with the basicHttpBinding and ws2007HttpBinding and compare the results. For the purpose of this sample, let’s use the following WCF service.

basicHttpBinding vs. ws2007HttpBinding: Simple Test Strategy

The simple test strategy simulates a fixed concurrent number of clients during a specified period of time.

When we run a simple load profile against that simulates our sample client we obtain the following results. Please notice the low average response time and large number of test generated as well as the high thread count which indicates the high performance of the service.

Running the same load profile against the service configured using the ws2007HttpBinding produces the results highlighted in the following figure. Notice that there the response time are a bit higher and the number of threads used a bit lower which clearly illustrates the performance impact introduced by the use of the ws2007HttpBinding.

basicHttpBinding vs. ws2007HttpBinding: Linear Test Strategy

The linear strategy is small variation of the simple strategy on which the concurrent number of clients fluctuates between two limits. When our service is configured with the basicHttpBinding, it exhibits the following behavior when running a linear strategy simulating between 100 to 125 concurrent clients.

The same test ran against the service using the ws2007HttpBinding produces the following results.

Again, notice the slight performance penalty introduced when using the ws2007HttpBinding.

basicHttpBinding vs. ws2007HttpBinding: Burst Strategy

The bust strategy simulates stress peaks in the number of concurrent clients. This test is typically used to simulate stress/recovery conditions for our target services. The following results were obtained when we ran a burst load profile against our sample service using the basicHttpBinding.

When switching to the ws2007HttpBinding the burst test produced the following results.

What did I learn?

I hope the results before clearly highlights the performance differences between the basicHttpBinding and the ws2007HttpBinding. More importantly, I hope this gives you an idea of all the crazy performance test you can do with the SO-Aware Test Workbench.

Announcing SO-Aware Test Workbench

Yesterday was a big day for Tellago Studios. After a few months hands down working, we announced the release of the SO-Aware Test Workbench tool which brings sophisticated performance testing and test visualization capabilities to theWCF world. This work has been the result of the feedback received by many of our SO-Aware and Tellago customers in terms of how to improve the WCF testing. More importantly, with the SO-Aware Test Workbench we are trying to address what has been one of the biggest challenges for WCF and BizTalk applications for years: LOAD TESTING!

That’s right, the SO-Aware Test Workbench includes a load testing engine that allows developers to execute load tests in just a couple of clicks and visualize the results in real time.

Here are some of the things you can do with SO-Aware Test Workbench!

Test Visualization

SO-Aware Test Workbench visualizes the execution records of a specific test, giving the tester or QA professional a very intuitive perspective of the behavior of the service.

Test Group Execution

With SO-Aware Test Workbench, testers or QA professionals can execute entire groups of tests as single unit. This feature takes full advantage of SO-Aware’s test categorization capabilities.

Load Testing

The SO-Aware Test Workbench provides a complete load testing suite that facilitates the modeling and execution of load tests with only a couple of clicks. Even more importantly, the SO-Aware Test Workbench provides real time visualization of test execution, giving the tester visually detailed information of the test runtime behavior.

The current version of the SO-Aware Test Workbench implements four fundamental load testing algorithms:

• Simple Rate Strategy:This strategy simulates a fixed number of clients with a specific delay between calls. For instance, we can use this strategy to simulate a service being consumed by a 100 clients with 2 milliseconds delay between calls.
• Linear Strategy:This strategy is very similar to the simple rate technique but it simulates a random number of clients between two limits. For instance, we can use this strategy to simulate our service being invoked by 100-200 clients during a specific period of time.
• Burst Strategy: This strategy is specifically designed for Recovery Testing. Essentially, the burst strategy simulates a concurrent number of clients during “Burst” followed by a “Burst Delay” on which the service experiences no activity. For instance, we can use this strategy to simulate spike patterns of web service consumption.
• Sawtooth Strategy:This strategy simulates a concurrent number of clients that increase or decrease abruptly based on a specific variance. This strategy can be used to stress test the service by simulating large but predictable changes in the number of concurrent clients.

If you are interested on learning more about Load Testing, you should check out this fantastic blog post from my colleague Gustavo Machado where he goes deep into the details of the different load testing strategies.

Local Test Projects

SO-Aware Test Workbench helps testers and QA professionals model and execute web service tests in their local environment without having to deploy them to the SO-Aware repository. This feature is particularly relevant to create variations of tests created in SO-Aware for specific business cases.

Test Report

The SO-Aware Test Workbench complements its rich visualization capabilities with the ability to export the results of the tests to various formats such as Adobe PDF or Excel. Testers or QA professionals can take advantage of this feature to represent the results of the tests in a format that can be easily understood by developers and architects.

Intrigued? Would like to learn more? Check out this Endpoint TV episode with Ron Jacobs on which we demo end to end the capabilities of the SO-Aware Test Workbench!

Agile SOA Governance: SO-Aware and Visual Studio Integration

One of the major limitations of traditional SOA governance platforms is the lack of integration as part of the development process. Tools like HP-Systinet or SOA Software are designed to operate by models on which the architects dictate the governance procedures and policies and the rest of the team members follow along. Consequently, those procedures are frequently rejected by developers and testers given that they can’t incorporate it as part of their daily activities.

Having SOA governance products that don’t integrate into IDEs is just another way to exclude developers and testers from the governance process.

With SO-Aware, we are trying to democratize SOA governance and management and make it a natural part of the development process. We accomplished that by providing native integration with Visual Studio 2008-2010 which are the fundamental IDEs developers rely on for building WCF services. We enable that integration in the form of two Visual Studio plugins explained in the following sections.

For WCF Service Developers: SO-Aware Service Explorer

The Service Explorer is the latest addition to the SO-Aware SDK. This explorer helps developers working on WCF services to directly interact with SO-Aware. The main user interface of the explorer lists all WCF services present in the current Visual Studio solution as illustrated in the following figure.

Publishing a WCF Service to SO-Aware directly from Visual Studio

Using the Services Explorer a WCF developer can publish a service directly to SO-Aware as highlighted in the following figure.

This is a very common scenario used by development teams to publish different versions of the services as the development process evolves.

Automatically generating WCF configuration

The coolest feature of the WCF Service Explorer is the fact that developers can import the WCF configuration stored in SO-Aware directly into their WCF project. This guarantees the consistency of the configuration across the different service projects.

In order to import the WCF configuration for a specific WCF service, we can select the Associate to SO-Aware section highlighted in the following figure.

After that we can select the specific WCF service we would like to import the configuration from.

Now we need to select the configuration file to which we would like to import the configuration as illustrated in the following figure. On this menu, we can select whether we would like to import the SO-Aware simplified configuration or the WCF native configuration as part of your WCF project.

If we select the native WCF configuration option, the wizard generates the complete system.ServiceModel section as illustrated in the following code.

   1: <?xml version="1.0" encoding="utf-8"?>

   2: <configuration>

   3:   <system.serviceModel>

   4:     <services>

   5:       <service name="Tellago.ServiceModel.Governance.Samples.SimpleService"

   6:             behaviorConfiguration="Throttling_CredentialsBehavior">

   7:         <endpoint address="address" 

   8:                 contract="ISimpleService"

   9:                 binding="ws2007HttpBinding" 

  10:                 bindingConfiguration="MutualCertBinding" />

  11:       </service>

  12:     </services>

  13:     <behaviors>

  14:       <serviceBehaviors>

  15:         <behavior name="Throttling_CredentialsBehavior">

  16:           <serviceThrottling maxConcurrentCalls="150"

  17:                              maxConcurrentSessions="150"

  18:                              maxConcurrentInstances="150" />

  19:           <serviceMetadata httpGetEnabled="true" />

  20:           <serviceDebug includeExceptionDetailInFaults="true" />

  21:           <serviceCredentials>

  22:             <serviceCertificate certinffo />

  23:             <clientCertificate>

  24:               <authentication certificateValidationMode="None" />

  25:             </clientCertificate>

  26:           </serviceCredentials>

  27:         </behavior>

  28:       </serviceBehaviors>

  29:       <endpointBehaviors />

  30:     </behaviors>

  31:     <bindings>

  32:       <ws2007HttpBinding>

  33:         <binding name="MutualCertBinding">

  34:           <security mode="Message">

  35:             <message clientCredentialType="Certificate" 

  36:                      establishSecurityContext="false"

  37:                      algorithmSuite="Basic256"

  38:                      negotiateServiceCredential="false" />

  39:           </security>

  40:         </binding>

  41:       </ws2007HttpBinding>

  42:     </bindings>

  43:   </system.serviceModel>

  44: </configuration>

Alternatively, we can select the simplified SO-Aware configuration and the wizard will generate the configuration containing a reference to the service version stored in SO-Aware. You can read more about how we simplified WCF configuration on this blog post from the SO-Aware product team.

   1: <?xml version="1.0" encoding="utf-8"?>

   2: <configuration>

   3:   <configSections>

   4:     <section name="serviceRepository" 

   5:              type="Tellago.ServiceModel.Governance.

   6:                    ServiceConfiguration.ServiceRepositoryConfigurationSection, 

   7:              Tellago.ServiceModel.Governance.ServiceConfiguration" />

   8:   </configSections>

   9:   <serviceRepository url="&lt;SO-Aware Endpoint>">

  10:     <services>

  11:       <service name="ref:SimpleService(1.0)@dev" type="SampleHost.SampleService"/>

  12:     </services>

  13:   </serviceRepository>

  14: </configuration>

After we generate the configuration we can keep it up to date by periodically upgrading it as illustrated in the following figure.

For WCF client developers: Add Service Reference from SO-Aware

Similarly to the service side, we wanted to make SO-Aware an intrinsic part of the WCF client development experience. In that sense, we extended the traditional WCF Add Service Reference dialog with a new option that allows developers to generate a service reference directly from SO-Aware. The following figure illustrates this concept.

Using this dialog, we can select the specific service version we are trying to consume as highlighted in the following figure.

After clicking generate, the runtime generates a WCF client configured to resolve the configuration from SO-Aware. In that sense, the client configuration file looks as simple as the following.

You can think about this technique as “service discovery on steroids”. Behind the covers, we used a WCF channel factory that not only resolves the client endpoint address but its entire configuration (bindings and behaviors) using SO-Aware.