Jesus Rodriguez's WebLog

Business Activity Monitoring (BAM) is one of the most attractive features of the BizTalk Server product family. Since its first release in 2004, BAM have been positioned as a mechanism to enable real time visibility across business processes implemented using BizTalk Server or other technologies such as Windows Communication Foundation (WCF) and Windows Workflow Foundation (WF).

Whether BAM has over delivered on this promise, we should admit that it is still challenging to implement and maintain medium to large BAM-based solutions in enterprise applications. Why? Well, in my opinion, BizTalk Server BAM is still based on a traditional business intelligence model on which OLAP cubes and ETL packages are at the center of the data analysis process. In a typical BAM implementation, a business user (which always ends up being a BizTalk architect) defines the activity model and that gets translated into a set of tables in the BAMPrimaryImport as well as multidimensional models defined in the BAMStarSchema and BAMAnalysis databases.

As mentioned before, BAM’s model is based on the traditional BI techniques such as cubing and ETL. Whether this type of models very effective to enable long terms analysis or large data volumes is has proven to be really unpractical to visualize the information near real time and find answers quickly which, ironically, is one of the fundamental principles of BAM! Let’s try to explain this reasoning using some of the major challenges of BAM implementations:

· Processing time: BAM SSIS packages take time to run and calculate the multidimensional measures and dimensions.

· Model versioning: Adding new dimensions and measures to an existing BAM model without affecting the existing data is extremely challenging

· Rapid Visualization: Given the dynamic characteristics BAM activity data (aka the data processed by BAM models), it is very often to find users wanting to visualize, sort, search the data in different ways. Enabling reports for all these views is completely unpractical from a long term perspective.

· Maintenance: Traditional BI infrastructures based on cubes and ETL packages are really expensive to maintain both from the infrastructure as well as from the people standpoint.

Some of these challenges were the fundamental catalyst for the adoption of self-service BI technologies a few years ago. In a self-service BI model, the data is made available to the user in an optimized manner (very often using an in-memory engine) so that it can be processed according that the user’s specific perspective. This is a very popular model in financial systems or government solutions on which the user dictates how the data is visualize instead of accessing via a preconceived report.

With the release of SQL Server 2008 R2, Microsoft gave a big step towards adopting self-service BI technologies with the release of PowerPivot for Excel and SharePoint respectively. PowerPivot allows users to load incredibly large amounts of data using an in-memory analysis service engine that runs within Excel or a SharePoint Server farm. You can find more details about this technology here: http://www.powerpivot.com/

From a BizTalk developer perspective, PowerPivot can modernize and extend the scenarios for using BAM within an enterprise. At a first glance, I can see some immediate benefits:

• Simplification: The use of PowerPivot will enable the creation of  simpler BAM models that don’t necessarily have to include all the multidimensional information.
• Personalization: Users will be able to analyze the BAM data according to their specific perspective.
• Maintenance: Simpler BAM model will translate into less of nightmare from the maintenance standpoint.
• Real time visualization: By using PowerPivot with BAM, users will be able to visualize the information near real time and modify it according to their perspective.
• Performance: Having the BAM data loaded into the PowerPivot engine will prevent from users to repeatedly hit the BAM databases when accessing the information.
• Collaboration: When using PowerPivot for SharePoint, users can take advantage of the features included in SharePoint 2010 to enable a more collaborative approach as part of their BI initiatives.

Well, enough talking…. Let’s take a look at a sample. Suppose that we have the following BAM model that describes a purchase order activity.

PO_CREATED              Milestone
PO_PROCESSED         Milestone
Amount                         Double
Customer_Country         Text
Customer_State             Text

Notice that the previous model doesn’t include any multi-dimensional information in order to keep it extremely simple.

The following code creates A MILLION instances of the purchase order activity.

   1: private static void PopulateBAMModel()

   2:      {

   3:          var r = new Random();

   4:          BufferedEventStream bamStream = new BufferedEventStream(BTS MsgBox, 0);

   5:          for (int index= 0; index <= 1000000; index++)

   6:          {

   7:              var activityID = Guid.NewGuid().ToString();

   8:              bamStream.BeginActivity(cActivityName, activityID);

   9:              var country = GetCustomerCountry();

  10:              var state = GetCustomerState(country);

  11:              bamStream.UpdateActivity(cActivityName, activityID, "PO_CREATED",

  12:                  DateTime.Now.Subtract(new TimeSpan(r.Next(1, 11),

  13:                  r.Next(1, 24), 0, 0)),

  14:                  "PO_PROCESSED", DateTime.Now,

  15:                  "Amount", GetPOAmount(), "Customer_Country",

  16:                  country, "Customer_State", state);

  17:              bamStream.EndActivity(cActivityName, activityID);

  18:          }

Using PowerPivot for Excel we can point it to the PO_AllInstances view and load the entire data set in just a few seconds. The following two figures illustrate that concept.

Having the data loaded, we can now take advantage of the entire Excel calculation engine to aggregate the data according our perspective. For instance, the following figure illustrates the process of creating a new chart that renders our BAM activity instances using the Excel Pivot control.

Using these techniques, we can analyze the data without accessing the BAM databases. At any point we can reload the data using PowerPivot’s data refresh mechanisms.

In order to enable collaboration and guarantee the scalability of the solution, we can publish the PowerPivot BAM workbook to SharePoint and access it without the need to having Excel 2010 installed in your local machine.

If you are a BizTalk developer, I truly hope these type of techniques makes as much sense to you as they do to me. Bottom line, PowerPivot is an ideal complement to BizTalk Server BAM and by no means a replacement to the existing infrastructure. When applied correctly, the combination of PowerPivot and BAM can enable scenarios have been prohibited for BizTalk Server until now.

Hello readers,

It's time to announce our monthly SO-Aware webinar. This time we are going to focus on BizTalk Server governance using SO-Aware!!!!!!! As many of you know that topic is very close to my heart :) I got to admit that this wasn't our original plan but given the increasing number of BizTalk customers that have been adopting SO-Aware we thought that it might be a good idea to dedicate our second webinar to the integration patterns between the two platforms.

In that sense, the session is going to focus on how to govern BizTalk Server artifacts using SO-Aware. Specifically, we would are planning on covering the following topics:

• Centralizing WCF adapters configuration using SO-Aware
• Centralizing WCF-LOB adapters configuration using SO-Aware
• Using the SO-Aware resolver for the BizTalk ESB toolkit
• Dynamically resolving WCF configuration using SO-Aware
• Monitoring BizTalk-hosted WCF services using SO-Aware
• Testing BizTalk-hosted WCF services using SO-Aware
• And more....

If you are interested on SOA governance with BizTalk Server I really encourage you to attend this webinar. You can register for the webinar here: http://www.tellagostudios.com/blog/biztalk-governance-using-so-aware ...If you have any suggestions about topics you would like us to cover please shoot me an email to jesus dot rodriguez at tellago dot com

Centralized configuration is one of the most attractive features of SO-Aware. The ability of managing WCF bindings and behaviors from a central location without imposing the scalability limitations of a message broker, addresses some of the key challenges of WCF enterprise solutions. This capability can be leveraged by any technology that relies on WCF configuration. Examples of those technologies are Windows Azure, SharePoint Server, BizTalk Server and specifically the BizTalk Adapter Pack.

The BizTalk Adapter Pack based on the WCF LOB Adapter SDK abstracts the interaction with LOB systems using WCF as the underlying protocol layer. Essentially, WCF adapters are implemented as WCF bindings which abstract the configuration and communication with LOB systems such as SAP or PeopleSoft. Although adapters are particularly popular on BizTalk applications, they can be used by any .NET application that requires to interact with a LOB system.

As typical WCF bindings, the adapters rely heavily on configuration to express the behavior required to interact with the specific LOB system. This configuration is quite often very complex and challenging to manage. Using SO-Aware, we could centralize the configuration of WCF-based adapters so that they can be used by the different client applications. Let's take a first look at how will this pattern work using SO-Aware. For the simplicity of this sample, we've decided to use the SQL Server WCF adapter included in the current version of the BizTalk Adapter pack.

The first step to use a WCF LOB Adapter with SO-Aware is to create a new binding type that corresponds to the adapter. The following figure illustrates this concept using the WCF-based SQL Server adapter.

After we've successfully configured the binding type, we can create new configuration for that specific adapter as illustrated in the following figure.

At this point, we can use the specific configuration on service endpoints such as BizTalk Server send ports. The following figure shows this concept.

Well, that's all we need to do to centralize the configuration of WCF-based LOB adapters. At this point, we can create new client applications (BizTalk or not) that interact with the LOB system without the need of managing complex sets of configuration.

Using SO-Aware, we can interact with all adapter with 0 configuration!!! We can accomplish that by using our configurable channel factory included as part of the current release as illustrated in the following code.

   1:  

   2:  private static void SOAwareSQLAdapterTest()

   3:  {

   4:    ConfigurableProxyFactory<TableOp_dbo_Contact> factory =

   5:                 new ConfigurableProxyFactory<TableOp_dbo_Contact>

   6:                 (new Uri("http://localhost/SOAware/ServiceRepository.svc/"),

   7:                 "SQLAdapterService(1.0)", "dev");

   8:    TableOp_dbo_Contact channel= factory.CreateProxy();

   9:    SelectResponse contactsResponse= channel.Select(new SelectRequest("*", ""));

  10:  }

Aren't you seeing the benefit already. Let's compare it to the traditional approach. The following code highlights the configuration required by a .NET client that needs to interact with SQL Server using our sample adapter.

   1: <system.serviceModel>

   2:         <bindings>

   3:             <sqlBinding>

   4:                 <binding name="SqlAdapterBinding" 

   5:                     closeTimeout="00:01:00" openTimeout="00:01:00"

   6:                     receiveTimeout="00:10:00"

   7:                     sendTimeout="00:01:00" maxConnectionPoolSize="100"

   8:                     encrypt="false" workstationId="" 

   9:                     useAmbientTransaction="true"

  10:                     batchSize="20" 

  11:                     polledDataAvailableStatement="" pollingStatement=""

  12:                     pollingIntervalInSeconds="30"

  13:                     pollWhileDataFound="false" notificationStatement=""

  14:                     notifyOnListenerStart="true" 

  15:                     enableBizTalkCompatibilityMode="true"

  16:                     chunkSize="4194304"

  17:                     inboundOperationType="Polling"

  18:                     useDatabaseNameInXsdNamespace="false"

  19:                     allowIdentityInsert="false" 

  20:                     enablePerformanceCounters="false"

  21:                     xmlStoredProcedureRootNodeName="" 

  22:                     xmlStoredProcedureRootNodeNamespace="" />

  23:             </sqlBinding>

  24:         </bindings>

  25:         <client>

  26:             <endpoint address="mssql://.//Demos?" binding="sqlBinding" 

  27:                       bindingConfiguration="SqlAdapterBinding"

  28:                 contract="TableOp_dbo_Contact" 
                      name="SqlAdapterBinding_TableOp_dbo_Contact" />

  29:         </client>

  30:     </system.serviceModel>

Now multiply this by dozens of endpoints and you can start picturing the management nightmare that we typically experience on medium to large BizTalk deployments. Centralizing the configuration also has the advantage that semi-sensitive information about the LOB systems such as the connection strings etc are saved somewhere else outside the client application

Today we are excited to have a major announcements about SO-Aware: Our Visual Studio plug-in and a preconfigured SO-Aware Virtual Image.

SO-Aware VPC

As Pablo announced last week, now you can download a virtual pc image with SO-Aware Express fully installed and configured. This image could be used by customers that are interested on testing SO-Aware without having to request specific hardware within their organizations. The VPC image is really small and can be downloaded from the following links.

• http://s3.amazonaws.com/SOAware/SO-Aware.part01.exe
• http://s3.amazonaws.com/SOAware/SO-Aware.part02.rar
• http://s3.amazonaws.com/SOAware/SO-Aware.part03.rar

SO-Aware Visual Studio plug-in

In order to enhance the development experience with SO-Aware, we've just made available a Visual Studio plug-in that allows client applications to generate a WCF service reference directly from SO-Aware!!!!!!

After you install the SO-Aware plug-in, you will find a new dialog item when you right click on a Visual Studio project.

When selecting that option, you will get a new dialog box to browse a specific instance of the SO-Aware repository and select the specific service version you would like to generate a proxy for. The SO-Aware dialog box is illustrated in the following figure.

What does this means? Well, in general, you can use WCF clients with 0 CONFIGURATION AND 0 EXTRA CODE!!!!! The experiencing for consuming a completely secured WCF service will be as simple as:

   1: private static void Test()

   2:       {

   3:           CRMServiceClient service = new CRMServiceClient();

   4:           var accounts= service.GetAccounts();

   5:       }

You can download the Visual Studio for SO-Aware as part of our SDK: http://www.tellagostudios.com/resources/so-aware-sdk-samples-and-utilities . You can find more information about the plug-in in our product weblog: http://www.tellagostudios.com/add-service-reference-so-aware-visual-studio

I hope you enjoy this two new announcements. As always feel free to email us with your feedback.

Tellago is going to rock the Buenos Aires Code Camp tomorrow!!! Four of our architects are presenting on some of the newest Microsoft technologies. Here is a quick summary of the topics our guys are presenting on.

Pablo Cibraro (http://weblogs.asp.net/cibrax/): An overview of WCF 4.0 and the Windows Server AppFabric

Gustavo Machado (http://thegsharp.wordpress.com/): Building RESTful services in the Microsoft platform. When to use what?

Mariano Rodriguez (http://weblogs.asp.net/marianor/): Windows Server AppFabric Caching deep dive

Adrian Lopez (http://adglopez.wordpress.com/): Applied Cloud Computing with Windows Azure

Some of our other architects are also attending the event. Tellago is one of the main sponsors of the Code Camp and we are actively growing our team in Argentina. If you are attending the Code Camp you should swing by Tellago's sessions and have a chat with our guys.