Microsoft’s StreamInsight first thoughts
I spent this weekend toying with Microsoft StreamInsight bits. I have a few blog posts planned for the next few days that tackle very specific CEP patterns but I figure it might make sense to start by highlighting some of the features I think differentiate Microsoft's StreamInsight from other CEP technologies in the market. Here is a n initial summary of the features I really like about Microsoft's StreamInsight:
What I like
LINQ
A continuous query language is an essential component of any CEP platform. The use of LINQ as the mechanism for querying and coordinating the processing of event streams is one of the fundamental benefits of Microsoft's StreamInsight. The continuously growth of LINQ-enabled data sources (SharePoint or Twitter might be some of the latest examples) guarantees the applicability of Microsoft's StreamInsight across highly heterogeneous environments. Furthermore, the use of a query language that is a core part of .NET guarantees great levels of extensibility and flexibility of CEP solutions.
This approach highly contrasts with other major CEP stacks in the market which rely on proprietary languages that are not part of the underlying platform. For instance, Oracle CEP uses the Continuous Query Language (CQL) as its stream query mechanism. However, COQL can't be really used from any other Java application that is not hosted within the CEP server.
Adapter model
Enabling an extensible programming model for creating input and output adapters should be a key factor for the viability of StreamInsight on highly heterogeneous environments. One of the common characteristics of CEP scenarios, is the interaction with non standards APIs such as the ones enabled by RFID readers, temperature sensors, etc. Abstracting those endpoints through a consistent programming model will enhance the management and extensibility of StreamInsight-based solutions.
Again, some of the lead CEP technologies in the market don't offer that level of extensibility which forces developers to only interact with data sources accesible through some of the adapters provided by the vendor.
Event Models
StreamInsight uses three fundamental types to describe the temporal characteristics of an event:
· Interval: The interval event model represents an event whose payload is valid for a given period of time.
· Point: A point event model represents an event occurrence as of a single point in time.
· Edge: An edge event model represents an event occurrence whose payload is valid for a given interval of time, however, only the start time is known upon arrival to the CEP server.
This feature forces the developer to declare up front the type of event model used by the application optimizing important aspects such as the query processing mechanisms.
Query Binding Model
StreamInsight uses the notion of a QueryBinder to abstract the query definition (template) from the physical streams to which the query is applied. This mechanism allows the reusability of query templates across different streams as well as it enables changes on the query templates without affecting the event streams that are being processed by the application.
Areas that still need work
Server hosting environment
At this point, StreamInsight is just a framework for implementing CEP application. More sophisticated hosting environments are obviously needed in order to apply StreamInsight to complex CEP scenarios.
Management
Management and tooling is an area that still requires some improvement in order to make StreamInsight a world class CEP platform.
Monitoring & Replay
StreamInsight includes some basic features for monitoring the flow of events for ability on a specific server endpoint. However, it should not come as a surprise that real world CEP solution will, most likely, require more robust monitoring capabilities.
Caching
Taking a lesson from other CEP platforms such as StreamBase or Oracle CEP I have the feeling that there is a lot of room for improving the processing of events with highly scalable caching infrastructures such as Microsoft Velocity or Memcache.
Integration with other Microsoft technologies
Flexibility and extensibility are by far the two fundamental benefits of Microsoft's StreamInsight. Those capabilities tremendously facilitate the integration of StreamInsight with other Microsoft technologies in order to implement real world CEP solutions. WCF, BizTalk RFID, Microsoft Sync Framework or SQL Server Business Intelligence stack are just some examples of technologies that can really both complement and benefit from the integration with Microsoft's StreamInsight. Providing best practices and samples around those integration models is going to be key for the adoption of Microsoft StreamInsight by the .NET community.