Mark Jen [ex-MSFT]

what happened to MarkJen?

Just as I promised... ;)

Today I’m blogging about two interesting things that people may have noticed in my last entry: the flag “enlistDuringPrepareRequired” and the significance of the byte[] named recoveryInformation passed to your code in IEnlistmentNotification.Prepare.

First up: What does the flag enlistDuringPrepareRequire (EDPR) do?

When a user wants to commit a transaction, it typically enters the two phase commit protocol (2PC). Normally, as soon as the transaction enters phase 1 and is in the process of committing, no more work should be done for the transaction and thus, no more enlistments can be made. However, let’s say you are creating a resource manager (RM) where enlistment is very expensive and thus, wants to delay work as far as possible. In other words, you don’t want to actually do work and enlist in the transaction until you are sure the transaction is about to commit. Perhaps sometimes, by delaying you can determine that you don’t actually need to do any work at all – the entire expensive enlistment operation is saved!

To facilitate this scenario, we have developed a Prepare phase that occurs after the transaction owner requests Commit but before the transaction actually enters Phase 1 of 2PC. Getting to this behavior is easy; simply do an enlistment in the transaction passing in true for the EDPR Boolean parameter. This tells System.Transactions that for this particular enlistment, you want the Prepare call to happen during this special time.

During this special time you know that the transaction is about to start 2PC and any work that you need to do must be done now. At this point, the RM can do work and enlist appropriately in the transaction; remember, normally once you’ve received Prepare, you cannot enlist in the transaction. Here is where if the RM doesn’t need to do any work, it saves the creation of a potentially expensive durable enlistment. Depending on the frequency of that scenario, this could be a huge performance gain. However, in the case that it does need to do work, this pattern incurs the relatively small cost of an extra EDPR=true enlistment.

Those who are astute will point out that during such an EDPR=true Prepare notification, the user code could enlist again with EDPR=true. In that particular scenario, there will be “waves” of pre-2PC Prepare notifications. What this means is that System.Transactions will batch the EDPR=true Prepare notifications up according to generation. All the EDPR=true enlistments that happened before the client called Commit are the first wave. Then, during this first wave, any additional EDPR=true enlistments are batched up into a second wave that happens only after all the notifications from the first wave are finished (enlistments signal that they are finished by calling back to System.Transactions using the IPreparingEnlistment provided). Within resource constraints, there is no limit to the number of EDPR=true Prepare waves you can have.

Next: What does this byte[] named recoveryInformation do?

If you decide that you need a durable enlistment (see my previous blog entry), then you need to account for failure conditions – specifically what happens if the RM goes down in the middle of a transaction that is committing. The pattern for correct RM behavior during 2PC transaction commit is well established and those steps (from the RM’s perspective) are:

1. Wait to receive the IEnlistmentNotification.Prepare notification
2. Make sure that it is ready to finalize the “work” done for the transaction
3. Write a log record containing the recoveryInfo byte and some information pertaining to the work that was done so that it can be undone in case the transaction aborts
4. Callback using IPreparingEnlistment.Prepared()
5. Wait to receive the IEnlistmentNotification.Commit or Rollback notification
6. Either finalize or revert the “work” done for the transaction
7. Write a record that nulls the record created in step 3
8. Callback using IEnlistment.EnlistmentDone()

The most classic scenario is what happens if the RM somehow fails between steps 4 and 5. In this scenario, the “work” is half finalized and is in limbo. To preserve data integrity, the RM needs to reconcile its log records when it recovers. To find out what the outcome of a transaction was, a recovering RM reads in recoveryInfo byte arrays from its log and calls:

IResourceManager myRM = new MyResourceManager();

      for (int i = 0; i < numRecoveryInfoInLog; i++)

      {

byte[] recoveryInformation = GrabRecoveryInfoFromLog(i);

IEnlistmentNotification en = new MyEnlistmentNotification();

TransactionManager.Reenlist(rm, recoveryInformation, en);

}

TransactionManager.RecoveryComplete(rm);

Then, the instances of MyEnlistmentNotification will receive the transaction outcomes for each recovery record. The actual contents of recoveryInformation are opaque and are an implementation detail of System.Transactions. However, you can safely assume that it contains at least the type of Transaction Manager you are connecting to, your RM’s identifier, and the transaction’s identifier.

To end this entry, there are a few different things I can cover next time, so I’ll take a vote. Leave me feedback or e-mail me (markjen@microsoft.com) and let me know which of the following topics most interests you:

• How do I use System.Transactions with my existing components?
• How does serialization work (a.k.a. what happened to the complicated transaction marshaling I used to have to do)?
• What are all the details involved with creating a correct durable Resource Manager?
• What happens when MSDTC goes down?
• What is this LTM (Lightweight Transaction Manager) that I’ve heard so much about?

Don’t worry, all these topics and more will be covered eventually, but if you give me a priority, I’ll be able to get to the info you need faster! Also feel free to suggest any other topic you’d like to hear about J

enlistment anyone?

So, you know you need a transaction and you've created a transaction object. Next, you most likely want to do some transacted work. At this point, there are two possibilities:

1.      you need to use a resource that understands transactions

2.      you need to create a resource that understands transactions

First of all, don’t let the term “resource” scare you; it simply means creating a piece of code that wishes to be informed when a transaction is going through its commit process. Any components you are building that want to know the outcome of a transaction, want to vote during the preparation of the transaction, or want to do durable work in the scope of a transaction all fall into the second category.

For today’s blog, I’ll be focusing on getting started developing a component in the second category. To get stuff rolling, you’ll need to decide whether or not your transacted functionality needs to be durable. Normally, you need to be durable if you have a resource for which you need to protect data integrity. If you don’t have anything that requires you to be able to guarantee data integrity in the face of system failures, then your resource is volatile.

Two main examples of durable resources are databases and file systems. For these resources, data integrity is paramount. Also, things that are built on top of these resources are also typically durable since they are backed by durable storage. In contrast,  a few examples of volatile resources include dialog boxes, diagnostic logging tools and in-memory caches. For the most part, these components are ok if they don’t receive the outcome of the transaction or if they crash and don’t maintain data integrity.

So let’s say you need to protect data integrity. That means you are creating a durable resource and you need to implement an IResourceManager object. This interface gives you a way to expose an identifier to System.Transactions; - it uses this identifier to track entities that require their notifications to be guaranteed. Think of it as a tracking number for your component so that in the event of a crash, the TMs and System.Transactions can make sure you still get notifications for your transactions.

The most basic IResourceManager looks like this:

    class MyResourceManager : IResourceManager

    {

        Guid myGuid;

        public MyResourceManager()

        {

            // ... grab myGuid from a durable store (file, databsae, etc) ...

        }

        #region IResourceManager Members

        Guid IResourceManager.Identifier

        {

            get { myGuid; }

        }

        #endregion

    }

You’ll notice that the identifier Guid doesn’t change. This is very important because if you change your identifier, System.Transactions will think you are a new and different resource and your notifications are no longer guaranteed.

Next, regardless of whether you are creating a volatile or durable resource, you'll need to create an object that implements IEnlistmentNotification. IEnlistmentNotification is an interface that defines how the Transaction Manager (TM) talks to you through System.Transactions. For those that are familiar with the 2 Phase Commit (2PC) protocol, you'll see methods that have familiar names. Here's a sample object that implements IEnlistmentNotificaiton:

    class MyEnlistmentNotification : IEnlistmentNotification

    {

        #region IEnlistmentNotification Members

        void IEnlistmentNotification.Prepare(IPreparingEnlistment preparingEnlistment, byte[] recoveryInformation)

        {

            // ... prepare unit of work for durable storage ...

            // ... save recoveryInformation ...

            preparingEnlistment.Prepared();

        }

        void IEnlistmentNotification.Commit(IEnlistment enlistment)

        {

            // ... log that unit of work is committed ...

            enlistment.EnlistmentDone();

        }

        void IEnlistmentNotification.Rollback(IEnlistment enlistment)

        {

            // ... log that unit of work has rolled back ...

            enlistment.EnlistmentDone();

        }

        void IEnlistmentNotification.InDoubt()

        {

            // ... log that unit of work is in doubt

        }

        #endregion

    }

What is happening here is you are implementing an interface so that System.Transactions can contact you during each phase in a transaction's commit protocol. This gives you a chance to do whatever work you need to do. When you are finished, you use the callback object passed to you to tell System.Transactions that you are done.

Finally, once you have an object that does what you need it to do, you'll want to enlist it in a transaction. Essentially, what you are doing here is registering your IEnlistmentNotification object with System.Transactions so that it knows to notify you when the transaction is committing. The syntax for doing this is:

    ITransaction tx = Transaction.Create();

    IResourceManager myRM = new MyResourceManager();

    IEnlistmentNotification myEN = new MyEnlistmentNotification();

    tx.VolatileEnlist(myEN, false);

    tx.DurableEnlist(myRM, myEN, false);

Confused yet? Let’s review.

·        You have a transaction and you want to do transacted work

·        You need to decide whether the work you are about to do is durable or volatile

·        If your work is durable, you need to create an object implementing IResourceManager

·        Either way, you’ll need an object implementing IEnlistmentNotificaiton

·        Use the DurableEnlist and VolatileEnlist methods on the ITransaction object to inform System.Transactions that you want to receive notifications when a transaction is committing

·        When the transaction commits, your object implementing IEnlistmentNotificaiton will get its corresponding methods called

Ta-da! That’s all there is to it. If you have specific questions or need more detail on a step, please visit the newsgroups and post up your requests.

Before I sign off for today, those of you who are really paying attention will have noticed that the DurableEnlist and VolatileEnlist functions have a Boolean parameter at the end called “enlistDuringPrepareRequired.” You may also be wondering what is in the recoveryInformation byte array passed to you when IEnlistmentNotification.Prepare is called. This info and more will be covered in later blog entries so stay tuned!

Whidbey newsgroups are available

It looks like there are newsgroups available for people with specific questions about Whidbey and related technologies. It's slightly hidden becuase it's not in the normal Microsoft.public.* newsgroup space, but there's web access (and instructions on how to use your favorite news reader if you prefer) at:

http://communities.microsoft.com/newsgroups/default.asp?icp=whidbey&amp;slcid=us

So if you're having trouble with Whidbey features, fire away in the newsgroups! MSFT employees are encouraged to check them regularly and will respond to specific inquiries if they can :)

System.Transactions is ready to change the way you write distributed code!

Hi everyone! It's been a while since I started up this blog and for the most part, it's been dormant. This is because the product I work on, System.Transactions, had never been shipped so there were concerns about us talking about it publicly. However, now that Whidbey Beta 1 is available for download, the cat is out of the bag, and it's time to get rockin'!

First order of business is: What is System.Transactions? The answer: System.Transactions is a new framework library that helps you write distributed, transacted applications in .NET. System.Transactions seeks to be fast, lightweight and easy to use.

So, let's jump right in! To get started, you'll need to add “using System.Transactions“ to the top of your source code file (and reference System.Transactions.dll when building your assemblies). Then, to create a transaction, simply do:

ICommittableTransaction myTx = Transaction.Create();

Let's take a few seconds to check out what is going on here. First, you'll notice that instead of using a constructor, I'm calling a static factory method on the Transaction class. This is because System.Transactions implements a layer of abstraction between the user program and the underlying transaction infrastructure. It allows us to update, enhance and extend the underlying implementation in the future without changing any of your code on top.

Next, you'll notice that I am getting back an object that implements the ICommittableTransaction interface. This interface has many of the standard properties and methods you'd expect on a transaction (and probably a few that you're unfamiliar with). The most important one that we'll look at today is the Commit() method. As the initiator of the transaction, your code has control of when the transaction commit will be requested of the transaction manager. Thus, you get an ICommittableTransaction. Other components that you pass your transaction to will naturally want to do work in the transaction, but you will not want them to have commit ability on your transaction. To this end, they are given an ITransaction object; ICommittableTransaction and ITransaction look identical except for one major difference: ICommittableTransaction has the Commit() method.

Now that you have a transaction, you'll most likely want to pass it to other components so that they can do work on it. We have made the objects backing your ICommittableTransaction serializable. This means you can directly pass your transactions through .NET Remoting and your transaction will be automatically marshaled to the other side. No more playing with whereabouts, transaction cookies and/or propagation tokens; everything happens through the magic of the ISerializable interface!

Congratulations, you've just had your first 30 second intro to playing with System.Transactions. I know what you're all thinking: “thanks Mark, this is all well and good, but how do I get some work done in this transaction?” To that, I say: stay tuned for my next installment, where I'll cover the details on how to enlist in the transaction and we'll get down to business - business logic that is :-p

this is the sound of inevitability...

Hello and welcome to my blog! Almost everyone I know keeps one of these things and has been telling me for years to start one up. Looks like I've finally jumped on the bandwagon and I'm able to use it for work related purposes too!

So a little about myself, I'm a Software Development Engineer in Test (SDE/T) at Microsoft in the Indigo/XML Enterprise Services group. I work on the System.Transactions team, a new v1 product that seeks to bring a unified transactions object model to Microsoft and beyond. If you were fortunate enough to attend PDC2003, you already have our bits in hand. Soon, we'll ship as part of the .NET framework and all your distributed application atomicity issues will be solved!

So come and visit often; I'll have tips and tricks for using transactions, random musings about life, and other tidbits straight from Redmond, WA.