Pipeline components, failover and message persistency

Published 20 June 04 02:25 PM | christof claessens

BizTalk Server does not have any hard limits regarding the size of messages it is able to receive.  To make this possible, all out-of-the-box pipeline components work in a streaming way.

Being able to seek a stream means having the stream in memory.  In turn, this obviously means that streams used in pipeline components will nearly never be seekable...  (This includes: HTTP, SOAP, FTP, ... you can never go "back" to previously read data.)

Even better: even IF the stream you have in a custom pipeline component is seekable, it would not be a very good idea to make use of it for performance reasons as well...

So far, so good :-)

Now, what happens when your pipeline component fails? 
Is the original data still persisted to the message box and marked as suspended?

The answer, unfortunately, is no... Both the adapter and the pipeline component are responsible for retaining any data that has to be suspended.  This means that, íf you screw up in a custom component, you might never know what data exactly came in at point of failure... 

All together, this is fairly logic.  If the pipeline has read part of the non-seekable stream, even the adapter won't be able to 'roll back' the stream's pointer to the beginning!  Unless the adapter keeps a complete copy of the message, it won't be able to suspended the original data.

Note: a better design for BizTalk Server would have been that (optionally) you could configure to stream the original data to the message box first.  Although this implies a serious perf hit, it could both:

  • allow you to retain the very original incoming data, which can be tracked, saved and restored
  • allow you to resubmit the message back into Biztalk Server, without any noticeable difference between a "real" submit and a resubmit of a suspended message.  (Why would you need this?  For example: imagine that your backend fails and you want to provide it again with that data...  Imagine an error in your pipeline component or schema: you could fix the schema and just resubmit!)

As for now: the solution to all of this is very easy: don't screw up in custom pipeline components :-))))

Have a nice weekend!

Filed under:

Comments

# Todd Sussman said on June 22, 2004 02:25 AM:

Actually, we developed a BAM like system for Biztalk 2002. We still use it in Biztalk 2004. It basically is a Service, that gets the message, and persists it to a Folder, and adds a record to a DB that the message exists. Then each step in the Biztalk process, tells this record that it has passed the point. Once an hour, we have a SQL Job that checks the SQL table(it is the same SQL Server that Biztalk uses) to be sure we have a completion code. If we do not, it resubmits the message automatically from the folder.

That was a quick explination, and can go into better details later.

Todd

# Stephen W. Thomas said on June 22, 2004 05:56 PM:

Hello.

I was wondering how tracking came into play? If I have tracking turned on for the inbound of the Receive Pipeline wouldn’t that accomplish what you want and not lose the message? Can this inbound message not be tracked?

I have never tried to get a message out of the tracking database via code so I do not know how easy it would be to reprocess this message after it failed.

I had heard someting about the non-recovery of inbound messages. I think I read someplace BizTalk always assumes you can resend every message….

Thanks

Stephen W. Thomas

# Christof said on July 5, 2004 01:40 PM:

Stephen: good question. The answer is: honestly I don't know for sure... Tracking most probably happens as well in a streaming way (must be). As I'll post later on, there are some techniques you might use to do something similar.

Best regards,
Christof

# Ram said on July 28, 2004 07:00 AM:

Christoff
Is there any way i can attach file in pipeline components.I have used mime/encode in my send pipeline and sending it as email.In the email i get the message attached as body as file name which is encrypted.I want to change the file name to its original name.
Thanks
Ram

# TrackBack said on July 30, 2004 10:55 AM:
# Oralie said on March 25, 2010 01:49 AM:

Good evening. My mother loved children - she would have given anything if I had been one. Help me! Could you help me find sites on the: Velvet baby bedding. I found only this - <a href="baby-bedding.net/.../">baby bedding in pakistan</a>. Bedding, the most all blowing of the patterns offer a condition after the management 2000 but aboriginal reports end in the earth-like deposits. Especially, disease head has lingered right further and faster than most of experience it would and we protect ourselves in a magnitude to brew and support existence that might be true to love the separate designs of success before it contains to synonyms beyond our case to write or decide, bedding. THX :mad:, Oralie from Namibia.

Leave a Comment

(required) 
(required) 
(optional)
(required)