Ralf's Sudelbücher

You´re fluent in object oriented programming. But now and again you´re wondering what the fuzz about component orientation is? There is supposed to be more to it than just using 3rd party controls in your user interfaces. But, what and how?

Component orientation is about higher productivity, easier maintenance, better testability, more flexibility, and - if you´re fond of it - reusability.

But how´s that? How does component orientation reach all those lofty goals? The trick is pretty simple: component orientation takes the basic design principle of loose coupling very seriously. But instead of now explaining contract-first design (CfD), IoC containers, binary code units, and component workbenches, let me demonstrate component orientation in a more tangible way with a musical example.

Requirements

The requirements for my musical project are: Produce a recording of the simple piece "Bell-ringers" (Source: "Bell-ringers" by Katherine and Hugh Colledge, (c) 1988 by Boosey and Hawkes Music Publishers Ltd.) as depicted below:

The requirements are clear, since a complete requirements document has been provided by the imaginary customer. Additionally the customer has stated, he´d prefer the piece played on the violin.

However... although I can read and understand the requirements I can´t play the violin. But that´s not really much different from software development, isn´t it? Often a solution needs to be developed with a technology you´re not familiar with; or you are supposed to adopt a process you´ve no experience with. So I guess the requirements are pretty realistic, even though it´s just an analogy.

Product

Out of didactic considerations now let me already present the product I developed according to the above requirements. So first just listen to my "Bell-ringers" recording:

Maybe it´s not exactly what you or the customer expected, but it´s on its way to what a true violin expert like Nigel Kennedy whould have produced ;-) In that, though, it´s also close to reality, isn´t it? Who has ever given a customer what she had expected for a first release?

Component oriented development

Now that you know the requirements and what I delivered to the customer, let me take you backstage. How did my component oriented development process work?

1. Decomposition into components

First I determined the components the final product should be build from. For you to understand this let me define component in a somewhat unorthodox way as:

A component is a part of the product that can be produced independently of other parts.

For a musical composition like "Bell-ringers" these parts or basic building blocks are all the different musical notes. I identified a, h, c'#, e, d', a', g'#, f'# and e' to be needed for a "Bell-ringers" production. To the right you see my original "analysis document".

In order to compose something from such components, though, more is needed: the relationships between the components have to clear. Components are not just dump parts but serve a purpose. They provide a service to other components. Here´s an addition for the above definition of component:

Components have a clear specification as to what services they provide and which other components´ services they depend on. This specification needs to be separate from any component´s realization.

Unfortunately here the analogy somewhat breaks. The relationships between musical notes are obvious from the musical score and are very, very simple. Their services are self-contained, so to speak.

Nevertheless there are relationships between the musical components. Each musical component (note) has a predecessor and a successor. That´s at least two relationships. And there can be more, e.g. in a chord with several musical notes played at the same time.

2. Component implementation

After I had identified all the components necessary to build the requested product I produced them independently. This was possible due to their loose coupling. I was able to arrange production in any order I saw fit. And if I had wanted to employ other musicians/developers I could have done so. They could have worked completely independent from me and in parallel. Thus components not only make the order of production flexible, but they also allow for very high productivity.

Here you see the "implementation" of two of the components. Yes, I implemented them not just as mp3-files, but videos. Above you just listened to the sound. But at the end you'll get the full monty :-)

Sample component a:

Sample component d':

Now you might think this kind of isolated production of components can only be done for musical components. But that´s not the case. My customers and I do it all the time in software projects. Admittedly then it´s a tad more difficult to determine which components to produce and what their dependencies are - but nevertheless it´s possible and it´s feasible. With a little component planning the stated benefits can easily be reaped.

(A word on the surroundings you can see in the video clips: Since I can´t play the violin I had to ask my little daughter, Verena (7 years old), to show me how to play the notes. So I´m sitting in her room while I´m implementing the components. And by the way: this all happened quite ad hoc some on some Sunday morning ;-)

3. Component integration

In monolithic projects, i.e. projects whose source code is just composed of classes but no components, the solution is finished as soon as all classes have been developed. With component orientation that´s different. The price to pay for higher productivity and increased flexibility is a separate integration phase. The components developed independently need to be integrated into a whole.

For the musical production this step consisted of connecting the components according to the relationships defined by the musical score. Here you see the components in the story board view of my video editing software:

From left to right it´s the components "a", "b"/"h", "c'#", "a" etc. But for now they are just roughly put together. To make them really cooperate their services need to be finetuned. This means I also need to adjust the lengths of the musical components. That´s what´s happening in the main editing view of the video software:

The duration of each "component instance" (remember: all components and even sequences of components (composite components) are reused several times) is adjusted to the requirements. Also you can see I added some "infrastructure" like a trailer and an intro.

While I tried to implement each component by itself with highest quality - call this unit testing if you like ;-) - it was only during integration I was able to see (or hear), if the individual qualities really added up to an overall acceptable quality.

As it turned out during integration, I had forgotten to implement one of the components. So I had to set up the video equipment again and do a retake. If you look closely you´ll be able to spot this component ;-)

But - thanks to component oriented production - this was the only thing I needed to do to save the whole production. Had I chosen the usual monolithic approach I would have had to redo everything. But component orientation made it possible to insert the additional implementation within the "network of building blocks" just where it was needed. That´s what I call flexibility!

4. Deployment

Finally, once all components were arranged, I let the video editing software encode the whole sequence into a single mp3-file (listen above) and a video file (see below). These files I then could deliver to the imaginary customer.

Here´s the whole product not only to hear, but also to see:

It´s the same sound as above - but since you´re looking under the hood you can actually see (!) it´s not monolithic, but component oriented. The customer experience is like it should be: smooth - although not yet perfect ;-) But the architecture of the whole is flexible. There is no tightly knit fabric of sounds, but sound bits (components) produced independently and then integrated in a way so that they form a seamless product.

The blessings of component orientation

Let me point out again the blessings of component orientation for this production:

• Components let me produce the whole in separate bits according to my liking. I could have arranged component implementation with regard to difficulty or availability of my coach Verena. Or I could have outsourced production of individual components. Components made the implementation flexibile and highly productive.
• Component quality could be checked individually. In order to attain high overall quality I did not need to produce a whole right from the start like in a live concert. I would have had to rehearse a lot for that. Rather I was able to hone the quality of small parts of the whole in isolation. Components made the implementation much easier to test.
• As the integration phase showed, components made it easy to modify the whole by inserting (or replacing or deleting or changing) just isolated parts. Again I did not need to play the whole piece again just to compensate for a mistake. I just implemented the component I had forgotten and inserted it where needed. The component oriented architecture was easy to maintain.

Well, what can I say? Component orientation rocks!

PS: As you can see from the trailer of the video, the video was planned as a hommage to some Lasse Gjertsen. You can find the at YouTube with several videos which inspired me. He´s done some awsome stuff. Check out his list of productions. But if you don´t have much time and are mainly interested in component orientation, view at least his video "Amateur". It shows a "distributed application" developed by "two teams" ;-)

As explained in my previous postings, I implemented a local/embeddable version of the Amazon SimpleDB data model and API in C#. You can download the sources from my NSimpleDB Google Code Project and build the tuple space engine yourself, or you download the demo application which includes the engine as a single assembly: NSimpleDB.dll.

Using the SimpleDB API then can be as easy as referencing the engine assembly and opening a local tuple space file like this:

See my previous posting for detailed examples.

Access to Amazon SimpleDB

The API I devised for the SimpleDB purposedly was quite "service oriented", although the implementation was just local. I did this so my implementation and Amazon´s eventually could be used interchangeably. Back then, though, I did not have access to SimpleDB due to the limited beta.

But that has changed in the meantime. I was able to use SimpleDB online and thus have now implemented access to it through the same ISimpleDBService interface. Just instanciate a different service implementation:

Instead of the placeholders pass in your Amazon access key id and your secret access key and you´re done. From then on all operations through the interface will run on your online SimpleDB space.

Here´s a small example of what you could do: create a domain, store some items into that domain, query the items, retrieve the found items, delete the domain.

(With regard to my previous API descriptions only a minor changes has occurred: you need to always pass in the nextToken as a ref parameter instead of an out parameter. But that´s a detail, I guess.)

This code runs locally as well as against SimpleDB online without change.

Under the hood I´m using Amazon´s own C# API for accessing SimpleDB. It was easy to program against - but I would not want to expose it as an API. For that I don´t find its usability too low. Too much meta data in the way of getting your problem solved.

Access to SimpleDB operation meta data

In order provide access to the meta data the Amazon SimpleDB API provides without compromising the simplicity of my ISimpleDBService interface, I decided to define a second interface: ISimpleDBDashboard.

Like a dashboard in your car this interface optionally "displays" what´s going on inside an ISimpleDBService implementation. AmazonSimpleDBService implements this interfaces and through it provides access to the response and exception meta data returned from the internally used Amazon API. To use it, just cast the service object like this:

There are only two properties on the dashboard interface: LastResponseMetaData and LastExceptionData. They give access to the meta data/exception data of the last operation issued on the current thread. So even in case you use the same service implementation on different threads, you always will be able to exactly see, what the status of the individual operations is. This is how I tried to avoid always returning some meta data object like Amazon does. It makes for a simpler API, I´d say. Call the dashboard properties right after any operation, if you like, e.g.

What´s next?

Well, that´s it for now. I have reached my initial goals of 1) provide a local version of the SimpleDB data model and API for all who want to play with it more easily than online, and 2) make the use the local tuple space and the real online SimpleDB equally possible and simple.

Of course I still have some ideas as to what can be improved (see the issue list online), but I now need to give the project a little rest. Please feel free, though, to use it and give me feedback on it via info [at] ralfw [dot] de, if you like.

Enjoy!

In my previous postings about Amazon´s SimpleDB data model and API I explained, what Amazon´s online database service - or to be more precise: tuple space - has to offer in general. If this sounds interesting to you, then now welcome to the desktop. Because it´s the desktop on which you can actually experience what it´s like to use such a tuple space. SimpleDB currently (as of Jan 08) is just in limited beta and you have to line up to get one of the limited test accounts.

But you don´t need to wait for Amazon to open up more. I implemented the SimpleDB data model and API in C# for you to integrate in your desktop or web applications. It´s an Open Source project at Google Code called NSimpleDB as short for .NET SimpleDB. I´d say it pretty much offers all features SimpleDB - but as an embeddable database engine instead of an online service.

Installing NSimpleDB

There are two ways for you to use NSimpleDB: Either you download the source code from the subversion repository of the NSimpleDB site. Then you can browse the sources and compile it yourself. But please note, you need to also install a VistaDb database engine. NSimpleDB internally is based on VistaDb and needs its libraries. As you can imagine I did not want do develop a whole persistence engine just to implement SimpleDB´s tuple space. But you can download an eval copy of VistaDb and need not fear to incur any costs right away. Also VistaDb is working on a free community edition, which I will of course use for NSimpleDB once it´s available.

Or, if you don´t want to mess around with the NSimpleDB source code, you can download the small demo application from the download area, which comes with a complete NSimpleDB engine as just one assembly: NSimpleDB.dll.

Using NSimpleDB

If you´re using the precompiled version of NSimpleDB just reference NSimpleDB.dll from your .NET project. If you compiled NSimpleDB yourself, though, reference NSimpleDB.Service.Contract.dll as well as NSimpleDB.Service.VistaDb.dll from the global bin folder of the source code tree. Also be sure to either have VistaDb installed on the same machine or copy VistaDB.NET20.dll into your projects´s output folder.

In any case you then should "open" the following namespaces in your source code:

Opening a NSimpleDB database

To work with NSimpleDB you need to manage a connection to a database file like with any regular RDBMS product. You can do it with the method pair Open()/Close() like this:

Just be sure to call Close() at the end. Pass in any name you like to give to your database file. NSimpleDB does not require a certain filename extension.

Or you can rely on the compile to generate the code to automatically close the connection. VistaDbSimpleDBSevice implements IDisposable:

Working with domains

In order to store any data in your NSimpleDB tuple space you need to create a domain first. It´s as simple as this:

No return value, nothing. This operation is idempotent. You can´t create a domain twice. From then on you can use the domain name in other operations.

To delete a domain, call the opposite operation:

This operation also is idempotent, don´t worry. And it´s asynchronous. Although the domain will become inaccessible right away for future operations, current operations are not interrupted and the actual deletion will take place at a later time.

Reflecting on domains then can be as easy as this:

But this is just the NSimpleDB way of doing it. Amazon´s SimpleDB does not provide such a simple way for retrieving all domain names. Instead the resultset is returned in pages. You determine the page size on the initial call to ListDomains():

Also you pass a token variable into which the method puts an identifier for the next page. To retrieve it, call ListDomains() again just with this token:

If there are no more pages, the token is set to null.

Here´s how you can retrieve all domain names in a pagewise manner using these operations:

Working with items and attributes

Storing items

Once you created a domain you can start storing items in it. Each item is uniquely identified by the domain name and its item name. The attributes to put into the item are passed as ISimpleDBAttribute objects. NSimpleDB for this reason provides the SimpleDBAttribute class:

You can freely choose an item name. You even need to choose one. It´s the primary key of the item, so to speak.

To later add an attribute to an item, just call PutAttributes() again and pass it the additional attribute:

Adding attributes is so easy, because they can contain multiple values. In the relational data model you would need to define more than one column or even set up a second table for this kind of 1:n relationship. Not so with (N)SimpleDB:

But then, how do you replace an attribute´s value? Pass it by explicitly stating you want it to be replaced:

Create the attribute object with true for the replace parameter.

Retrieving attributes

Retrieving an item is even easier than storing it. What you get is an array of ISimpleDBAttribute objects:

The default is to return all attributes with all their values. But you can limit the number of attributes by passing in explicit attribute names:

Deleting attributes and items

You can delete attributes from an item at any time. Just specify their names. They´ll be purged from the item with all their values:

Once all attributes are gone, the item is gone as well. To make this easier, there is a shortcut. Just don´t specify any attribute name at all:

Querying for items

Use GetAttributes() to retrieve a single item. But for that you have to know the item´s name. Where do you get this item name from, though? Just use a query:

SimpleDB´s query syntax is just about conditions the items, you´re looking for, have to fulfill. So it´s not a full blown data retrieval/manipulation language like SQL, but more just like a syntax for logical expressions. Here´s a second example to illustrate this:

To load the attributes for the found items, just call GetAttributes() in a second step:

That´s it. Easy - but different from working with SQL.

Read more about the query syntax in my previous posting.

Since a query potentially matches a large number of items, SimpleDB´s querying is paged. NSimpleDB sports an unpaged version (see above), but of course also paged querying. It works like paged domain retrieval:

Conclusion

This is NSimpleDB. A simple data persistence API - but unlike your regular RDBMS. Nevertheless I find it very interesting an am looking forward for Amazon to open up its beta program for SimpleDB. But until then you can get acquaintant with SimpleDB´s data model by using NSimpleDB locally. Enjoy!

PS: Let me know any questions and suggestions you might have. Just email me via my homepage.

Amazon´s SimpleDB is an exciting new player in the database world. It´s free, it´s online, it´s not relational. SimpleDB is a dynamic database implementing a tuple space. Currently SimpleDB (as of Jan 08) is in beta - but not everyone can get his hands on it. You have to apply and line up for one of the limited test accounts.

Nevertheless it´s worthwhile to take a closer look at SimpleDB. It´s a brave step forward by Amazon to offer an online database (accessible via a web service) that´s deviating from the mainstream data model of RDBMS.

In part 1 of the series of postings I described this data model: You store tuples (aka items) consisting of name-value pairs (aka attributes) in a SimpleDB "data space" without the need of any configuration. No schema design necessary. No tuple needs to look like an other. Just so called domains are a structuring concept to group tuples. But it´s nowhere written you have to use more than one domain. Even different kinds of items don´t force you to distribute them across domains. Domains that way are more of a concern regarding scalability and quantitative constraints Amazon put on them.

A simple SimpleDB API

The data model of SimpleDB is simple, so is its API. It´s not based on a query language (although it provides set selection, see below), but rather follows the tuple space concept in that it defines just a small number of methods to read item from and write item to the "data space".

Following I´ll use pseudo code to describe the API. I think will be pretty self explaining. In reality Amazon offers a web service to work with SimpleDB, so you´ll use some kind of proxy class in your code. Amazon even published a .NET binding - but hasn´t gotten rave reviews so far. There is much room for improvement.

Attributes as smallest data units

The smallest piece of data with SimpleDB is an attribute. An attribute is a name value pair like "Name"("Peter") or "Amount due"("000000300.00") or "DOB"("2000-05-12") or "Marked for deletion"("1").

As you can see, values are just strings. It´s like with XML. Attribute names are also strings - and they can contain white space. This makes them easier to read and use as labels in frontends.

In addition - and in stark deviation from the relational data model - attributes can have multiple values, e.g. "Phone numbers"("05195-7234", "040-413 823 090", "0170-233 4439").

Amazon suggests, you don´t try to store large pieces of data in attributes, e.g. a multi-MB image. Rather you should put such byte-blobs into some other store - e.g. a file on an FTP-server or Amazon´s S3 - and use the attribute value as a reference.

Items as containers for attributes

Attributes belong to items. In principle items can contain any number of attributes, but Amazon put some limitations on them. Currently only 256 attributes are allowed in each item.

Items can be written as tuples and are identified by an explicit id you have to provide, e.g. "123"["Name"("Peter"), "City"("Berlin")]. The id is called "item name" an again is a string.

As you can see, attributes are tuples with unnamed elements, but items are tuples whose elements are named.

Domains as containers for items

Items are stored in domains. Like them, domains have an id, the domain name. No schema needs to be defined for them. Just pour items of any structure into them as you like, e.g. "contacts"{"123"["Name"("Peter"), "Addresses"("a", "b")], "a"["City"("London"), "Country"("GB")], "b"["City"("Hamburg"), "Country"("Germany")]}.

As you can see, domains are tuples, too. Their elements are named tuples, the items.

Writing data

Roughly you can say, domains are like tables, items are like records in a table, attributes are table columns. So storing data with SimpleDB means: write items with their attributes to a domain. That´s like writing records with their column data to a table.

SimpleDB provides a single operation for writing data: PutAttributes(). Identify where you want to put the attributes - into which item in which domain -, hand in the attributes - and you´re done.

This command would write a single attribute to the item with name "123" in domain "contacts":

PutAttributes("contacts", "123", ["Name"("Peter")])

But now watch! If you then issue this command

PutAttribute("contacts", "123", ["Addresses"("a")])

you don´t overwrite what´s been stored in the item, but add to it! The same is true for this command:

PutAttribute("contacts", "123", ["Addresses"("b")])

Remember that attributes can have several values. Item "123" now looks like this: "123"["Name"("Peter"), "Addresses"("a", "b")]. So you better also write the referenced addresses to the domain:

PutAttribute("contacts", "a", ["City"("London"), "Country"("GB")])
PutAttribute("contacts", "b", ["City"("Hamburg"), "Country"("Germany")])

But how then can you overwrite data, e.g. change the name of tis contact? If you just issue a PutAttributes() with the new name, the name will be added as a second value to the existing attribute. To overwrite you need to add a replace-flag to an attribute (I´ll denote it with a "!" after the attribute name):

PutAttributes("contacts", "123", ["Name"!("Paul")])

Replacing an attribute like this deletes all (!) existing attribute values and replaces them with the new value.

A word of caution: Amazon´s SimpleDB is supposed to scale. That´s why they distribute it across many servers and need to replicate data all the time. That in turn means, it will take some time until changes you made by PutAttributes() and the other operations ripple through to all relevant servers. So don´t expect to see changes right after you applied them! Otherwise, if you issue a PutAttributes() followed right away by a GetAttributes() for the same data - this could run on a different thread - you might be in for a surprise.

Reading data

Reading items back from the SimpleDB "data space" is even easier than writing them. Just send the GetAttributes() command addressing an item in a domain and pass the names of the attributes to retrieve:

GetAttributes("contacts", "123", "Name")

will return ["Name"("Paul")]. Of course you can specify more attributes to be retrieved. And since you only state their name, they´ll be returned with all their values.

Item data can only be retrieved like this! Queries (see below) just return item names, but no attributes. Think of them as SQL statements like this:

select attributeName1, attributeName2, ... from domainName where itemName="..."

Looking up data thus always is a two step process: 1. Issue query and receive a list of matching items, 2. retrieve item´s attributes with an item name from the query result.

Deleting data

You can´t delete items explicitly. You can only delete attributes from them - and if none are left in the item, the item is deleted automatically.

DeleteAttributes("contacts", "123", "Addresses")

only deletes the references to the other items, but the contact item remains in the domain. You also need to delete its name attribute, plus, of course, the parentless addresses:

DeleteAttributes("contacts", "123", "Name")
DeleteAttributes("contacts", "a", "City", "Country")
DeleteAttributes("contacts", "b", "City", "Country")

Creating a domain

Working with domains as the containers for items is easy. You can create a domain at any time. Just call

CreateDomain("contacts")

and that´s it. Just pass in a unique domain name. From then on, you can use this domain name in item-operations.

Deleting a domain

Deleting a domain is as easy as creating it:

DeleteDomain("contacts")

The items and attributes in that domain will be gone then. But this might take up to 10 seconds, Amazon says, due to the distributed nature of SimpleDB.

Querying domains

If you want to get an overview of the domains in your SimpleDB "data space", just call ListDomains():

ListDomains(10, &nextToken)

It returns a list of domain names. This resultset is paged, though. The first parameter to ListDomains() specifies the size of these pages, e.g. 10 domain names per page, the second parameter is a token you can use to retrieve the next page.

Passing in a token to ListDomains() returns that page´s domain names and sets the token to the next page, if there is any.

nextToken = ""
domainNames = ListDomains(10, &nextToken)
// process first page of domain names
domainNames = ListDomains(10, &nextToken)
// process second page of domain names
...

Querying data

Finally, there is also a way to query items. SimpleDB sports a simple query language. You can think of the queries as the where-clause of a SQL select statement, e.g.

select itemName from domainName where simpleDB-query

Queries are limited to a single domain and return just item names as paged resultsets like ListDomains().

The building blocks of queries are predicates. A predicate is a logical expression made up of attribute comparisons, e.g.

['City' = 'Hamburg' OR 'City' = 'London']

Both attribute name and attribute value need to be put in single quotes. SimpleDB sports the usual comparison operators like =, != etc. and a STARTS-WITH which resembles the SQL like, e.g. like 'A%'.

['Name' STARTS-WITH 'A']

Remember, all comparisons are alphanumeric, since SimpleDB only stores texts.

The logical operators within predicates are AND, OR and NOT.

You may only query for a single attribute name with one predicate! ['City'='Hamburg' OR 'City'='London'] is ok, but not ['Name'='Peter' AND 'City'='London']!

To state queries on attributes with different names, you need to separate predicates for each:

['Name'='Peter'] INTERSECT ['City'='London']

The set-operations to combine the resultsets of each predicate into one are INTERSECT, UNION and NOT. INTERSECT calculates the common set of item names of two predicates, UNION merges the item name sets of two predicates. INTERSECTS thus works like the logical AND operator, UNION like the OR.

Why does Amazon deviate like this from the well established SQL way of defining queries? The reason probably lies with the internal structure of the SimpleDB "data space". Grouping the constraints on attributes with the same name probably makes query execution faster. Maybe SimpleDB is based on a column store?

EBNF SimpleDB query syntax

Query ::= ItemSetTerm { "UNION" ItemSetTerm }.

ItemSetTerm ::= ItemSetFactor { "INTERSECTION" ItemSetFactor }.
ItemSetFactor ::= [ "NOT" ] "[" PredicateExpression "]".

PredicateExpression ::= PredicateTerm { "OR" PredicateTerm }.
PredicateTerm ::= PredicateFactor { "AND" PredicateFactor }.
PredicateFactor ::= [ "NOT" ] PredicateComparison.
PredicateComparison ::= AttributeName ComparisonOperator AttributeValue.

AttributeName ::= Chars enclosed in single quotes, e.g. 'Name'.
                  All AttributeNames in a PredicateExpression need to be the same.
                  All quotes in AttributeName need to be properly escaped.
AttributeValue ::= Chars enclosed in single quotes, e.g. '003.14'.
                   All quotes in AttributeValue need to be properly escaped.

ComparisonOperator ::= "=" | "!=" | ">" | ">=" | "<" | "<=" | "STARTS-WITH".

What´s missing?

SimpleDB´s API is simple. That´s the beauty of it. A simple, dynamic data model plus a simple API sounds like a powerful combination for today´s fast moving software business.

But this simplicity comes at a price. Common operations like looking up data, are more cumbersome than with SQL. It´s a two step process due to SimpleDB´s queries returning just item names. Also currently transactions are missing completely.

Another aspect to get used to is the "eventual consistency" model, that means, changes take time to ripple through to all replicas of your data. Thus after a change there might be a short time where different clients might see the "data space" in a different state.

But overall, Amazon´s effort is very exciting nevertheless.

What´s next?

I deem SimpleDB even so exciting, that I wanted to be able to use it now and on my desktop. But there is no desktop/local version of SimpleDB and I don´t know when Amazon will grant me a test account of SimpleDB.

That´s why I sat down and developed my own Open Source version of SimpleDB: the .NET SimpleDB or NSimpleDB for short. I believe in the growing importance of tuple spaces in general and thus also am working with the University Vienna on bringing this paradigm to the hands of .NET developers. We call the basic technology "XVSM" for "eXtensible Virtual Shared Memory"; and it´s somewhat like SimpleDB. But on top we place more elaborate data structures so our space is not just partioned into domains but collections and other high level data structures. We envison them to allow for true "Space Based Collaboration" (SBC), which is in our view the foundation for "serverless real-time online collaboration". But I digress.

Back to SimpleDB: In my next posting I´ll show you, how you can use SimpleDB or the C# implementation of the SimpleDB API in your applications today without reliance on Amazon.

Have you heard about Amazon´s online "database service" SimpleDB? They describe it like this: "Amazon SimpleDB is a web service for running queries on structured data in real time." So it´s not a RDBMS, because Amazon does not call the data "relational", but just "structured". And you use a web service based API to access the data, not good old ADO.NET. Currently SimpleDB is in beta. You can get a test account to play around with it - if you´re patient. As of this writing (Jan 08) evaluation is limited; you need to apply and queue up to be assigned a test account. I have about 2 weeks ago, but haven´t heard from Amazon since then.

But why should you care? Well, SimpleDB would allow you to store data in a database without any setup costs. You don´t have to care about backup or moving to another ISP. You´re data, lots of data, can just stay with Amazon. Just add a web service proxy to you web (or desktop) application and off you go. This certainly make some (or a lot, or at least a growing number of) applications easier to implement.

Another reason to care about Amazon´s SimpleDB is its simplicity. In an age where dynamic programming becomes ever more popular and static whatever (e.g. typing, binding) loses value, making persistence more dynamic sure should look attractive. But exactly this is what Amazon´s SimpleDB is about: highly dynamic persistence of structured data.

SimpleDB data model

With SimpleDB you don´t define a database schema anymore. Your "data space" with Amazon is structured in a very simple way: it´s devided into sub-spaces called "domains" which each contain so called "items" which each contain so called "attributes". That´s it. And you can change the structure of this "data space" at any time. There is no distinction between meta data and data. Creating a domain (which resembles a table in a relational database) is a web service operation like storing an item in a domain.

To make it very clear: You divide your "data space" into domains at your leisure. (Amazon currently just artificially limits the number of domains to 100.) And you stuff items of any structure into these domains. You never define a schema for a domain. The items stored in a domain don´t have to look the same. They can contain any number of attributes; all can differ in their number of attributes.

Attributes are name-value-pairs. So items are tuples of arbitrary aritiy. That means, SimpleDB is not a relational database, but a tuplespace. Just throw items/tuples into your SimpleDB instance at your leisure. That´s all their is to SimpleDB persistence. If you like, separate tuples into different domains - but if you do it or not does not make a big difference. For distinguishing between, say, customers and invoices that´s not necessary. It might even be contraproductive, since querying items is limited to one domain at a time. There is no such thing as a SQL Join.

The use of domains

So why are there domains at all? Probably they help Amazon to make replication of items between servers easier. And it might speed up queries if you distribute your data across domains. So think of domains as easy to set up data partitions in case you have to deal with huge amounts of data.

Multi-valued attributes

But not only don´t you have to define a schema for a domain and all items/tuples can have a different structure, there´s another deviation from relational thinking: Attributes can have multiple values! So items don´t even comply with the relational first normal form. See the "Phone" attribute in the following item:

It´s not just several phone numbers separated by commas. No! The "Phone" attribute is really structured. You can retrieve (and query for) each phone number separatley. SimpleDB would return the item like this:

Think of what this means: Finally you can set up "natural references" between persistent data like in memory. A parent objekt points to its children. But when you persist these objects in a relational database, you usually invert the references. The child records will contain a foreign key to denote their parent record.

But with SimpleDB you can let parent items point to their child tuples:

See how all children are referenced with their id values from their parents? See how the number of attributes differ across the items? That´s all just fine with SimpleDB.

Just text

A drawback of SimpleDB might be its limitation to text. All attribute values are stored as just text. So comparison is alphanumeric and leads to effects like this: 20 > 100 because in fact the the comparison is "20" > "100". So be sure to take this into account when storing your non-text values like numbers or dates. Pad numbers with leading zeros (e.g. store "00012" instead of "12"), use a sortable date format (e.g. "2008-01-18"). If you expect to store negative numbers, move them into the positive range of numbers, e.g. instead of "-12" and "12" store "0" and "24" if you expect the value range to start from -12.

On the other hand SimpleDB in this regard does not differ from XML. Text simply is the least common denominator for storing data. Also, this makes SimpleDB more efficient, since it can be optimized for handling text (e.g. in terms of indices).

What´s next?

That´s pretty much all there is to say about SimpleDB´s data model. It´s simple. It´s dynamic.

In my next posting I´ll introduce you to the SimpleDB API. It´s simple, too. Just a couple of easy operations.

But if you want to move forward more quickly, have a look at the SimpleDB documentation on the Amazon website. You can also try out my Open Source implementation of the SimpleDB data model and API. It´s called NSimpleDB and is hosted with Google. More about this too in a future posting.

When writing more complex code you cannot really step through during debugging, it´s helpful to put stud it with statements tracing the execution flow. The .NET Framework provides for this purpose the System.Diagnostics namespace. But whenever I just quickly wanted to use it, it turned out to be a hassle to get the tracing running properly. That´s why I wrote down the following, to make it easier next time.

How to instrument the code?

In the code set up different System.Diagnostics.TraceSource objects. For each area to watch define a trace source name, e.g. "BusinessLogic" or "Validation" or "TextFileAdapter". Each such trace source later can be switched on/off separatedly.

...

A TraceSource object can be instantiated for just one method or can be kept around for a long time as a global (or even static) reference.

To trace the execution use one of the TraceXYZ() methods of TraceSource. Most of the times one of the following will do. TraceInformation() writes an informational message, i.e. with TraceEventType Information:

That´s the same as this:

But TraceEvent() can do more. With it you can issue message on different levels, e.g. just informational messages, warnings, error messages. The id (10 above) will show up in the event log in its own column ("Event").

Since tracing can sometimes produce an overwhelming amount of messages, you can filter them. For example you can restrict the output of messages to errors only. But you should not do so in your code. That´s why you should pass SourceLevels.All to the TraceSource ctor. (Without setting a source level explicitly, the default is Off, so you´d see no messages at all!)

However, if you want to limit the tracing imperatively to certain levels of messages, you can do so by passing a combination of levels to the ctor, e.g.

How to attach tracing to different message sinks?

Tracing messages are written to any sink attached to the tracing infrastructure. A sink can be a text file or the console or an event log. Each sink is represented by a TraceListener object. A trace source can have any number of listeners listening for messages and write them to its sink.

Listeners can be attached to trace sources imperatively in your code - or using the App.Config. I prefer the App.Config, because it allows to you add and remove listeners without touching your code.

But as long as you´re satisfied with tracing messages being sent to the debug output window of VS.NET, you don´t need an App.Config at all. There is a default listener attached to each trace source.

If you want to direct the message to other sinks, though, or filter them, then you need to tweak the App.Config. Here is the most simple trace source defined in the App.Config - but its sink still is the debug output.

The name needs to match the name passed to a TraceSource object in your code.

Now, you can add listeners to this source element to have the source´s tracing messages sent to different sinks:

This listener´s sink is the console window. By adding it to the source all tracing messages are sent to the console window - plus the debug output window. The default listener is still attached to the source. If you want messages to go to just the sinks you define, remove the default listener:

Now you can add any number and kind of TraceListener object as within your reach. Here are the ones provided by the .NET Framework:

What the sinks of the listeners are, should be pretty obvious from their names. The DefaultTraceListener is the one writing to debug output, though.

Three of the listeners need to be primed with additional data specifying the message sink. Pass a filename to the XmlWriterTraceListener and the TextWriterTraceListener. Pass an event log name to the EventLogTraceListener. Use the attribute initializeData for these parameters.

Please note: The TextWriterTraceListener works best, if you set the autoflush attribute of the <Trace/> element to true:

Otherwise the log file might be empty, because the listener did not get properly closed. autoflush lets the trace source call Flush() on all listeners after each message, so no messages get lost.

How to share listeners between sources?

Adding a listener to a source by putting its definition into the source is a bit cumbersome, if you have several sources and want to use the same listener in all of them. But there´s a simpler way. Define a shared listener and reference it in the <add/> element.

If you want to (temporarily) limit a (shared) sink to the messages from just one source, then you can set up a filter in the listener´s definition:

Just name the only source the listener should listen to.

How to limit tracing to certain message levels?

The number of messages produced by instrumented code can be huge sometimes. So what you want to do pretty often is, filter the messages according to some criteria. The tracing API lets you do that quite easily for the trace level of the messages.

As was shown above regarding the TraceSource ctor and TraceEvent() method, each message belongs to a trace message level or category. To block messages belonging to certain levels you can do two things: set up a switch or set up a filter.

The switch switches on/off a source according to the message level. That´s most simple by using the switchValue attribute of the <source/> element:

Enter any of the SourceLevels enum names shown in the following table:

The correspond to the SourceLevels parameter of the TraceSource ctor. The table shows which trace message levels are let through or are blocked by each switch value.

Alternatively use the TraceLevel integer values shown in the following table:

If you want to share switches among several sources, you can set up a switch definition and reference it:

Now, what if you want to send some messages to a certain sink and others to another sink? For exeample you want to separate error messages from informational messages? Then you use filters on listeners again:

Pass to an EventTypeFilter a SourceLevels enum name and it will listen only for messages on that level. 

That´s pretty much it to get code basically instrumented:

• Use several trace sources to be able to switch on/off tracing for different parts of your code independently.
• Choose trace listeners to direct messages where you can easily analyse the execution flow.
• Choose switches and filters to limit the messages written to the sinks so you are not burried under a pile of irrelevant details.

My previous posting on Software Transactional Memory (STM) I concluded with the remark, NSTM was not finished. How true! Here is the next release of NSTM with a couple of improvements. You can download it from Google´s project hosting site. Here´s what´s new:

Validation matrix

As mentioned in an earlier posting I was not quite satisfied with the validation strategy of NSTM. Even to me it was not entirely clear, when a transactional object (txo) would be validated. I improved on this situation in the lastest release of NSTM (rel. 1.0.0.222) by implementing this validation matrix:

Now you can tweak a transaction´s independence of others very clearly: You can make it "subordinate" and fail as early as possible, i.e. as soon as it detects another transaction has changed a value it has read. Or you can make it very "dominant" by not caring for changes by other transactions and even rigorously overwriting them.

Automatic retry of failed transactions

With databases collisions of transactions are pretty rare. It´s unlikely that two transactions change data in the intersection of their working sets. There´s usually so much data so the intersection is very small or even non-existend. When you commit a database transaction you can be quite sure it will succeed. That´s why optimistic locking has become the only "locking strategy" left in ADO.NET.

In-memory transactions, though, are different. The total amount of data on which concurrent transactions work is much smaller than with databases. In addition, in-memory data structures like a queue or stack or list hinge on just a few data items (e.g. references to the first/last element) which are under heavy pressure if several transactions concurrently add/remove elements.

If for example two transactions concurrently add elements to a queue thereby updating the same data item (reference to head of queue) it´s likely that one of them fails due to a validation error. However, had this failed transaction been run just a couple of milliseconds later, it would have succeeded, since it would have read its values after the other transaction´s commit.

Since collisions/invalid transactions seem to be more likely with NSTM compared to databases, and because the remedy is easy - just repeat the failed transaction -, a remedy should be developed. This remedy is automatically running transactions again on failure due to invalidity, i.e. automatically retrying them.

That´s why I added ExecuteAtomically() to NstmMemory. In its simplest form it just executes a delegate synchronously within a transaction (the delegate does not take any parameters and is of type System.Threading.ThreadStart):

It´s the same as if you wrote

However, if you want to guarantee the success of a transaction, then you force it to be retried indefinitely:

true as the first parameter tells ExecuteAtomically() to execute the delegate again, if the Commit() on the internal transaction failed during validation. A validation failure is recognized by ExecuteAtomically() by receiving a NstmValidationFailedException from Commit().

By just requesting automatic retry like above there is no limit on the number of re-executions. So be careful! Some pattern of concurrent transactions might cause one of them to retry again and again and thus cause a thread to hang.

If you want to avoid such pitfalls, call the method with fine grained parameters to tweak the retries. Here´s the definition of the most comprehensive method overload:

In addition to the usual transaction properties you can tell ExecuteAtomically() to...

• retry just a fixed number of times (maxRetries),
• or to retry and arbitrary number of times, but not to try longer than maxProcessingTimeMsec milliseconds.

If you want to give other threads time to do their job between a transaction´s retries, pass a delay to the method (sleepAfterRetryMsec).

To switch of any of these constraints, assign int.MaxValue to maxRetries and System.Threading.Timeout.Infinite to the millisecond parameters.

Of course you can also combine the parameters, e.g. by limiting retries to a maximum number and a maximum processing time. No further executions will be tried if either of these limits is reached.

If an exception is thrown during the transaction, the retries are aborted and the execption passed up to your code. If the retry limit is reached without a valid commit a NstmRetryFailedException is thrown.

Enjoy!

The API for my .NET Software Transactional Memory (NSTM) I´ve described so far is straightforward, I´d say. It´s close to what you´re used to from transactional dabase access and it´s even integrated with System.Transactions: open a transaction, do some stuff with transactional objects, commit the transaction. All very explicit.

Despite its familiarity this explicitness kind of stands in the way of what a piece of object oriented code is supposed to accomplish. It´s something you´ve to concern yourself with although it does not add to the raw functionality you want to achieve. And by being imperative it´s prone be done incorrectly. A more declarative way to do in-memory transactions and less explicitness thus would be nice now that objects have become transactional and automatically threadsafe in general.

I thought about that from the being when I started to work on NSTM. Microsofts STM implementation SXM shows, how transactionality could be made more declarative and "invisible": With SXM you just adorn a class with the attribute [Atomic] to make its instances transactional. That´s cool - but so far I did not consider it for NSTM, since it seemed to require fiddling around with proxies. That´s why with SXM you need to instanciate transactional objects through a factory which you´ve to create for each transactional class:

The benefits of using an attribute to make classes transactional to me seem to be lost through this cumbersome instanciation process. That´s why I did not go down this road until now.

PostSharp to the rescue

But then one day - lo and behold! - a very, very cool tool was revealed to me by InfoQ (which I´ve come to like as much as theserverside.net):

PostSharp from Gael Fraiteur is a godsend! It´s one of those technologies you´ve waited for all along without really knowing. It´s a technology that will change how you approach software development. At least that´s what his done for me since I first read about it.

What PostSharp does is not new, though. It´s a kind of Aspect Oriented Programming (AOP) weaver. And it´s a tool to alter the MSIL code in assemblies. But there are already several APO frameworks out there - also for .NET -, so why another one? There are already tools to alter assemblies like Phoenix from Microsoft and Cecil from the Mono project.

What makes PostSharp so remarkable to me, though, is in which way it combines AOP with MSIL modification. It makes it so dead easy!

• You´re not forced to learn MSIL.
• The weaving is done automagically for you by an invisible post build step.
• No factories needed to instanciate classes annotated with aspects, because no proxies are used.
• For many common types of aspects (e.g. pre/post processing of method calls, field access interception) there are templates that help you get your own aspects up and running in just a few minutes.

Congratulations Gael! And thanks for being so responsive to questions and bug reports!

Aspect oriented transactions

The first thing I tried with PostSharp to make NSTM transactions declarative. This seemed to be easy to do and would result in an also familiar transactional programming model. COM+/EnterpriseServices, ASMX, and not WCF all sport some attribute to make methods transactional. Here´s an excerpt from Juval Löwy´s article [1] on  that topic:

MyMethod in the service contract is declared as to be wrapped in a mandatory transaction. The service implementation does not have to do anything imperatively for that. The WCF infrastructure will create a new System.Transactions transaction if necessary behind the scene.

That´s what I wanted for NSTM, too, and what now seemed within reach through PostSharp. And indeed it was very easy to implement. Here´s my transactional aspect whipped up in some 30 minutes after downloading PostSharp:

How does it work? First have a look at some code which uses this attribute:

The method TransferMoney() does not open explicitly a transaction. Rather this is done by the NSTM infrastructure because the method is adorned with the [NstmAtomic()] attribute. The default then is to commit the transaction when the method returns; but it´s aborted if the method throws an unhandled exception.

Of course, if you like, you can pass a transaction scope or isolationlevel etc. to the attribute to influence how the transaction is created. By default the scope is Required so nested calls don´t each created new transactions.

Now, how is this magic woven (sic!) by PostSharp? I won´t explain PostSharp in depth here, but a quick glimpse behind the curtain won´t hurt, wouldn´t it ;-)

Remember, PostSharp modifies the original assembly in a post build step like an O/R Mapper´s enhancer (e.g. Vanatec OpenAccess) or an obfuscator (e.g. Xenocode). So here´s the "enhanced" TransferMoney() method as seen through Reflector:

private static void TransferMoney(int amountToTransfer)
{
    MethodExecutionEventArgs args;
    try
    {
        object[] arguments = new object[] { amountToTransfer };
        args = new MethodExecutionEventArgs(methodof(Program.TransferMoney, Program), null, arguments);
        ~PostSharp~Laos~Implementation.~aspect~1.OnEntry(args);
        if (args.FlowBehavior != FlowBehavior.Return)
        {
            myAccount.Write(myAccount.Read() - amountToTransfer);
            yourAccount.Write(yourAccount.Read() + amountToTransfer);
            throw new ApplicationException("Money transaction failed!");
            ~PostSharp~Laos~Implementation.~aspect~1.OnSuccess(args);
        }
    }
    catch (Exception exception)
    {
        args.Exception = exception;
        ~PostSharp~Laos~Implementation.~aspect~1.OnException(args);
        switch (args.FlowBehavior)
        {
            case FlowBehavior.Continue:
            case FlowBehavior.Return:
                return;
        }
        throw;
    }
    finally
    {
        ~PostSharp~Laos~Implementation.~aspect~1.OnExit(args);
    }
}

I greyed out the original method body so you can more clearly what PostSharp has woven around it. Basically it´s a try-catch-block which calls the On...() methods of NstmAtomicAttribute. Where .NET needs context bound objects and proxies at runtime to let you intercept calls PostSharp does it by statically inserting code which calls your interception methods.

Aspect oriented transactional data

Since success was so easy to gain with PostSharp I did not want to stop with transactional methods. Wouldn´t it be nice to hide transactionality almost altogether? If enhancer based O/R Mappers can hide persistence behind an attribute and don´t require object creation through factories, why shouldn´t I be able to do the same? If SXM falls short of this, that´s not my problem ;-)

So I set down with PostSharp and had a close look at the different kinds of aspect it provides out of the box. And really there was a way to accomplish what O/R Mapper enhancers have done before. I just had to combine a "generic field" aspect with a "type composition" aspect using a "compound" aspect. And here´s the result:

Put the [NstmTransactional] on an ordinary class to make it transactional. That´s it. It´s the same as wrapping INstmObject<T> around it. NSTM will care for buffering writes to instances and committing any changes at the end of a transaction. The money transfer example thus looses almost any trait of its transactionality:

This is just ordinary object oriented code. (Please don´t get too critical with my account business logic ;-) With PostSharp´s AOP the impact of Software Transactional Memory on your code is reduced to two attributes: make classes transactional with [NstmTransactional] and wrap transactions around methods with [NstmAtomic].

If you don´t have a class you want to make transactional, but just a scalar type or a struct, then use NstmTransactional<T>:

NstmTransactional<T> works almost like Nullable<T>. It´s just for scalar and value types. However NstmTransactional<T> is a class, not a struct! This is due to the need for identity and addressability of transactional data. When a transaction commits it needs to update a memory location it knows the address of. That´s not possible for value types which live on the stack. So if you want a value type to be transactional you need to be make it into an object.

Please note: [NstmAtomic] keeps only track of changes to one level of an object hierarchy! If you want to make more than one level transactional, you need to put [NstmAtomic] on all classes on all levels!

In this example a contact has an address. If you want to work transactionally with both, it´s not sufficient to just adorn the Contact class with [NstmAtomic], though! The attribute does not cause NSTM to track changes recursively on all objects pointed to by a Contact. Rather you need to make Address also transactional.

With all levels of the object model made transactional code like this works correctly:

Neither the contact name nor the location´s city are changed, since the transaction is aborted.

This of course means, you cannot use the .NET collection classes as is. For example you could set up a contact with several addresses like this:

Then you would want to be able to do the following:

But the output at the end of this code would surprise you:

The name of the contact and the city of the first address were rolled back correctly. But there is a second city in the list of addresses without a name. That´s the object to which the code assigned "Munich" and which was added to the address collection. The city of this address also got rolled back correctly, but the entry in the ArrayList was not. That´s because standard .NET collections are not transactional.

So, how then should you model 1:n relationships between transactional objects? True transactional collections are needed. I´m already working on some, so stay tuned. Or start building your own transactional list, queue, stack, tree etc. and let me know.

Preliminary conslusion

I´ve presented you my view of Software Transactional Memory for .NET. NSTM is written in C#, so it´s fully managed code. You can download the sources and play around with it. What will it gain you in terms of ease, productivity, or performance? Find out yourself. It´s programming model at least promises to make many multithreading tasks much easier. No need to think about locking, but rather work on on graphs of objects like on databases: open a transaction, do your work, commit the transaction. If something goes wrong, no changes are visible. That´s also an interesting propsal for GUI programming where you often want to display objects in dialogs, let the user enter changes, but also let him discard all changes. Especially with nested dialogs the nested NSTM transactions could help quite a bit to keep the frontend code straightforward, where IEditableObject falls short.

Enjoy!

PS: NSTM is not finished, of course. I´ll continue working on transactional collections. And there are some concepts from STM research I´d like to add, like a Retry() operation and blocking reads and notifications.

Download

You can download NSTM (short for .NET Software Transactional Memory) from the Google project hosting site.

Enjoy - and let me know how you like it, how it performs for you, or what you think needs to be improved.

Installation

1. Download PostSharp from www.postsharp.org and install it.
2. Download NSTM and install it by unzipping the file.
3. Open the solution in the NSTM directory and try to compile it.
   (It might fail due to incorrect references to PostSharp assemblies. The remedy is to update the PostSharp.Laos and PostSharp.Public references in the NSTM projects.)

There are a lot of NUnit compatible tests in the source code. You can use Testrunner like I do to run them in VS2005, or you can use NUnit itself. If you want to throw out all tests, just compile in Release mode.

Resources

[1] Juval Löwy, WCF Transaction Propagation, http://msdn.microsoft.com/msdnmag/issues/07/05/Foundations/default.aspx?loc=en

So far I´ve described my own .NET Software Transactional Memory´s (NSTM) API for managing transactions. It´s close to what you are used to from relational databases, I´d say. But still, it´s my own API and it stands beside what .NET already provides in terms of transactions. With System.Transactions there is already a general way to work transactionally across different resources like database and message queue, so it would be nice if NSTM was another such resource.

Juval Löwy [1] has described how this can be accomplished for in-memory data structures. However, making a data structure transactional like he did with the .NET collections does not make it threadsafe. .NET transactions - although attached to a thread - are not designed help make multithreading easier. In addition they cannot be nested truely. Also the data duplication in Juval´s collections is very coarse grained so it will become pretty slow pretty quickly once they are growing larger. Nevertheless the article is worth reading and provides very helpful insights in how .NET transactions work. It helped me a great deal making NSTM compatible with System.Transactions.

From the point of view of System.Transactions a NSTM transaction is a resource. Its state - the transaction log (txlog) - is changed during a .NET transaction and either discarded at the end if the .NET transaction is rolled back, or committed which means the transaction log is applied to the transactional objects (txo).

Recognition of .NET transactions is not switched on automatically, though. To check, if a .NET transaction is running and whether a NSTM transaction is already enlisted with it, is somewhat costly. NSTM needs to walk up the stack of active transactions for this. But you can switch on System.Transaction integration easily with the flag NstmMemory.SystemTransactionMode:

• EnlistOnAccess: NSTM checks on each access to a txo whether a .NET transaction is running. If there is no NSTM enlisted with the .NET transaction it begins a new NSTM transaction, enlists it, and will commit it/roll it back, when the .NET transaction ends.
• EnlistOnBeginTransaction: NSTM checks only for a .NET transaction when a NSTM transaction is explicity started using NstmMemory.BeginTransaction(). If a .NET transaction is running, the new NSTM transaction is enlisted with it. (To be more precise, NSTM even creates two nested transactions: the outer transaction is enlisted with the .NET transaction and the inner transaction is passed back to the application. That way the application can commit/rollback the inner transaction as usual and still get the final vote on the changes automatically from the out transaction coupled to the .NET transaction.)
• Ignore: NSTM does not care if a .NET transaction is already running. All NSTM transactions are independent of System.Transactions.

Here some sample code showing how to use NSTM with System.Transactions:

In line 1 .NET transaction recogniction is switched on. From then on each read/write access to a txo like o checks, if a .NET transaction is running and if so creates an implicit new NSTM transaction. That way the change to o get committed when the .NET transaction is finished in line 7. Please note, this option - although most convenient - is quite expensive since it requires transactional objects to check upon each and every call whether to create a NSTM transaction or not.

This is why the second example is more economic: It just checks for a .NET transaction in line 14. The price to pay for this option is an explicit NSTM transaction, i.e. a little less convenient programming model.

What´s next?

In my previous posting I said, System.Transactions integration was the final piece missing for your understanding of NSTM. But that´s not true anymore. Since then I stumbled across a very cool tool and have spiced up NSTM a bit. Stay tuned for automatic transactions like with COM+ and truely transparent transactionality for your classes.

Resources

[1] Juval Löwy, Can´t Commit: Volatile Resource Managers in .NET Bring Transactions to the Common Type, http://msdn.microsoft.com/msdnmag/issues/05/12/transactions/default.aspx

I´ve explained in my previous posting, how a single transaction weaves its magic of isolating changes to transactional objects (txo) and atomically making them visible on commit. But what´s the "reach" or "scope" of a NSTM transaction? How many transaction can be open at the same time?

Transactions are kept in TLS

To answer these questions it´s vital to understand where transactions are stored: in thread local storage (TLS). NSTM transactions are bound to a single thread, the thread they were created on. This is different from System.Transactions which are thread-independent (see [1] for details). But binding NSTM transactions to a thread is on purpose. NSTM is supposed to make multithreaded programming easier by isolating the work done in parallel on shared in-memory data structures. Thus transactions need to thread-local to keep changes made to txos in local transaction logs (txlog).

Application code in each thread works with its own transactions which of course can work on the same txos:

A txo is just briefly locked during access to avoid inconsistencies during single reads/writes. Since a txo is not just a singel value but containes at least a value and a version number which must not get out of step that´s necessary. But this kind of locking is hidden from you. Don´t worry about it. Nothing is permanently locking for the duration of a transaction. No deadlocks can occur. Rather NSTM bets on optimistic locking as already explained: txos are validated during commit (at latest) to check if they were changed by some other transaction and if so, the transaction fails. It´s the same as with ADO.NET when saving changes made to a DataSet.

When you call Read() or Write() on a INstmObject the object retrieves the current NSTM transaction from TLS and delegates further processing of your request to the transaction.

The usual transaction scopes

TLS does not just point to a single transaction, though. Rather it contains a stack of transactions (txstack) because you can nest them. Check out this code:

In line 3 there is one transaction on the TLS txstack, in line 6 there are two on txstack with txInner being the topmost. In line 8 it´s again just one transaction: txOuter.

Each transaction of course keeps its own transaction log. By default they are independent. However, they need to be ended in reverse order despite their independence. A transaction opened while another was active needs to be committed or rolled back first. That´s why you should nest transactions like above with using statements. They ensure the right order.

But there are several transaction scope options. Above the inner transaction is opened with RequiresNew. This ensures a new transaction is opened although another one is already active. The default however is just Required:

The inner transaction is opened with the default scope Required which means, a new transaction is only created if none is active yet. But in line 13 there is already txOuter active so no new transaction is started and txInner equals txOuter. In line 15 there´s still only one transaction on the txstack. This mirrors what EnterpriseServices or System.Transactions have offered all the time.

Truely nested transactions

NSTM goes beyond that, though, by providing for real nested transactions:

Using RequiresNested not only ensures there is a new transaction active in line 105, but also that any changes committed with txInner can be undone by rolling back txOuter! In the first sample above txOuter and txInner were independent. Whatever txInner committed txOuter could not undo. Although syntactically nested the transactions had no more to do with each other than transactions on different threads. But nested transactions are different: When txInner is started in line 103 is clones the txlog of txOuter and thus sees all changes already made to txos. And when txInner it commits, it does not write any changes to the txos, but just to its parent transaction txOuter. That means, changes of inner transactions are not visible "to the public" until its parent transactions "reaffirms" them by also committing.

Finally there is the RequiredOrNested scope option. If there is already a transaction running no new one is created, as long the options (scope, isolation level, clone mode) of this transaction are "compatible" with the options stated for the new transaction. If the existing transaction´s options are less strict, though, then a nested transaction is created. This option seems useful for library developers who don´t want to necessarily open new transactions for whatever their library code does - but also don´t want to sacrifice transactional strictness. Consider this scope option an experiment and food for thought ;-)

Summary

See, there answers to the initial questions were easy: A transactions scope is the thread. And you can have as many transactions open as you like. They just cannot overlap, but the can truely be nested. Threadsafety is guaranteed for transactions as a whole and INstmObject transactional objects. If you use CloneOnRead you´re pretty much on the safe side also for the values of txos, though. But this depends on how deep your clones are. For scalar types and value types containing just scalars you don´t have to worry. Deep object hierarchies, though, are not locked in any way! Take this into account when you implement ICloneable. Nevertheless NSTM is a very convenient and efficient way to parallelize work.

What´s next?

You know almost everything about my Software Transactional Memory implementation. There´s only the integration with System.Transactions missing - but not for long ;-)

Resources

[1] Juval Löwy, Can´t Commit: Volatile Resource Managers in .NET Bring Transactions to the Common Type, http://msdn.microsoft.com/msdnmag/issues/05/12/transactions/default.aspx