Exploring StreamInsight's adapter model

Adapters are a fundamental component of Complex Event Processing (CEP) applications. In a nutshell, adapters provide the interfaces that abstracts how events are produced or consumed by the CEP infrastructure. Most CEP frameworks leverage the concept of an adapter as the fundamental mechanism for interacting with heterogeneous systems. Following the same principles, Microsoft's StreamInsight uses adapters to model the flow of events in or out of the CEP host. Furthermore, StreamInsight enables a flexible programming model that allows developers to extend the core infrastructure by implementing custom adapters. From a programming model standpoint, StreamInsight classifies adapters based on the direction of event flow and on the event model used.

An StreamInsight adapter can be classified as input or output depending on whether events flow in or out of the hosting application. An input adapters accepts a set of events from data source on its native format, translate them and flow them into a StreamInsight application. Similarly, an output adapter receives events from an StreamInsight applications, translate them into its native format and send them to the target application.

In addition to the directionality of events, StreamInsight adapters can also be classified based on the event model they support. In that regard, StreamInsight supports three types of adapters: interval, point and edge.

• The point model is used to indicate when the event is valid for a single instant of time. Examples are the arrival of an e-mail, a meter reading, a user Web click, a stock tick, or an entry into the Windows Event Log.
• The interval model is used to indicate when an event is valid for a given period of time and both the start time and end time of the interval can be provided in the event type at the time of enqueueing the event into the server. Examples include the width of an electronic pulse or the duration of (validity of) an auction bid.
• The edge model is used to indicate when the event is valid for a given interval of time; however, only the start time is known when it is enqueued into to the CEP server. The end time of the event is provided at a later time. Examples are Windows processes, trace events from Event Tracing for Windows (ETW), a Web user session, or quantization of an analog signal.

Another important characteristic of a StreamInsight adapter is the type of data it can process. Some adapter process that can be model by strongly typed .NET classes while other adapters require the ability to process heterogeneous event payloads. StreamInsight classifies the adapter as typed or untyped based on whether the event types are well-known and strongly defined at design time or generated dynamically at runtime.

Based on the event kind and model, StreamInsight abstracts adapters using a series of based classes as illustrated in the following table.

Given the long running nature of most CEP application, input and output adapters go through different stages that represent their ability of producing or consuming events respectively. For instance, while an adapter is actively producing events its state can be set to "running". However, that state can be reset "suspended" when to indicate that there are no events immediately available in the source system.

StreamInsight models the lifecycle of an adapter using a sophisticated but flexible state machine illustrated in the following figure.

An interesting detail of the StreamInsight programming model is that the previous state machine if used by both input and output adapters. On a future post I plan to cover in more details the adapter programming model including its state transitions.

As you can notice, the adapter state transitions are controlled by the CEP server and by the adapter itself. Specifically, a state transition occurs before the CEP server calls Start() or Resume() and after the adapter calls Enqueue(), Dequeue(), Ready(), and Stopped(). The events received by the input adapter are made available to the CEP application by using the Enqueue() operation. Similarly, the output adapter uses the Dequeue() operation to retrieve events that need to be sent to the target system.

I hope this explanation will help you to get more familiar with the StreamInsight adapter model. In a future following post we will cover the details of implementing a custom adapter using the StreamInsight programming model.