Pipeline components, failover and message persistency

BizTalk Server does not have any hard limits regarding the size of messages it is able to receive.  To make this possible, all out-of-the-box pipeline components work in a streaming way.

Being able to seek a stream means having the stream in memory.  In turn, this obviously means that streams used in pipeline components will nearly never be seekable...  (This includes: HTTP, SOAP, FTP, ... you can never go "back" to previously read data.)

Even better: even IF the stream you have in a custom pipeline component is seekable, it would not be a very good idea to make use of it for performance reasons as well...

So far, so good :-)

Now, what happens when your pipeline component fails? 
Is the original data still persisted to the message box and marked as suspended?

The answer, unfortunately, is no... Both the adapter and the pipeline component are responsible for retaining any data that has to be suspended.  This means that, íf you screw up in a custom component, you might never know what data exactly came in at point of failure... 

All together, this is fairly logic.  If the pipeline has read part of the non-seekable stream, even the adapter won't be able to 'roll back' the stream's pointer to the beginning!  Unless the adapter keeps a complete copy of the message, it won't be able to suspended the original data.

Note: a better design for BizTalk Server would have been that (optionally) you could configure to stream the original data to the message box first.  Although this implies a serious perf hit, it could both:

  • allow you to retain the very original incoming data, which can be tracked, saved and restored
  • allow you to resubmit the message back into Biztalk Server, without any noticeable difference between a "real" submit and a resubmit of a suspended message.  (Why would you need this?  For example: imagine that your backend fails and you want to provide it again with that data...  Imagine an error in your pipeline component or schema: you could fix the schema and just resubmit!)

As for now: the solution to all of this is very easy: don't screw up in custom pipeline components :-))))

Have a nice weekend!

4 Comments

  • Actually, we developed a BAM like system for Biztalk 2002. We still use it in Biztalk 2004. It basically is a Service, that gets the message, and persists it to a Folder, and adds a record to a DB that the message exists. Then each step in the Biztalk process, tells this record that it has passed the point. Once an hour, we have a SQL Job that checks the SQL table(it is the same SQL Server that Biztalk uses) to be sure we have a completion code. If we do not, it resubmits the message automatically from the folder.



    That was a quick explination, and can go into better details later.



    Todd

  • Hello.



    I was wondering how tracking came into play? If I have tracking turned on for the inbound of the Receive Pipeline wouldn’t that accomplish what you want and not lose the message? Can this inbound message not be tracked?



    I have never tried to get a message out of the tracking database via code so I do not know how easy it would be to reprocess this message after it failed.



    I had heard someting about the non-recovery of inbound messages. I think I read someplace BizTalk always assumes you can resend every message….



    Thanks



    Stephen W. Thomas

  • Stephen: good question. The answer is: honestly I don't know for sure... Tracking most probably happens as well in a streaming way (must be). As I'll post later on, there are some techniques you might use to do something similar.



    Best regards,

    Christof

  • Christoff

    Is there any way i can attach file in pipeline components.I have used mime/encode in my send pipeline and sending it as email.In the email i get the message attached as body as file name which is encrypted.I want to change the file name to its original name.

    Thanks

    Ram

Comments have been disabled for this content.