-
-
Due to TechEd 2004, there will be some radio silence next week on both this blog and the contributions to the BizTalk Server 2004 public newsgroups. So, see you all there!!
Since there has been quite some activity on several postings lately, I'll be addressing each of those comments after next week! In the mean time: have fun - let's meet in Amsterdam!
Best regards!
Christof
-
-
BizTalk Server does not have any hard limits regarding the size of messages it is able to receive. To make this possible, all out-of-the-box pipeline components work in a streaming way.
Being able to seek a stream means having the stream in memory. In turn, this obviously means that streams used in pipeline components will nearly never be seekable... (This includes: HTTP, SOAP, FTP, ... you can never go "back" to previously read data.)
Even better: even IF the stream you have in a custom pipeline component is seekable, it would not be a very good idea to make use of it for performance reasons as well...
So far, so good :-)
Now, what happens when your pipeline component fails?
Is the original data still persisted to the message box and marked as suspended?
The answer, unfortunately, is no... Both the adapter and the pipeline component are responsible for retaining any data that has to be suspended. This means that, íf you screw up in a custom component, you might never know what data exactly came in at point of failure...
All together, this is fairly logic. If the pipeline has read part of the non-seekable stream, even the adapter won't be able to 'roll back' the stream's pointer to the beginning! Unless the adapter keeps a complete copy of the message, it won't be able to suspended the original data.
Note: a better design for BizTalk Server would have been that (optionally) you could configure to stream the original data to the message box first. Although this implies a serious perf hit, it could both:
- allow you to retain the very original incoming data, which can be tracked, saved and restored
- allow you to resubmit the message back into Biztalk Server, without any noticeable difference between a "real" submit and a resubmit of a suspended message. (Why would you need this? For example: imagine that your backend fails and you want to provide it again with that data... Imagine an error in your pipeline component or schema: you could fix the schema and just resubmit!)
As for now: the solution to all of this is very easy: don't screw up in custom pipeline components :-))))
Have a nice weekend!
-
-
Very frequently, in the BizTalk Server newsgroups, I answer questions related to filenames used by the BizTalk Server FTP and FILE adapter. More in particular it seems that for many people it is hard to figure out:
- how to change the FILE/FTP adapter's output file name
- how to acces the FILE/FTP adapter's input file name
Let's change this and dive into the subject a little deeper:
...starting with the more easy thing: the FILE adapter's input and output filename.
When the FILE adapter receives a message, it promotes context on the message that contains information about the orginal filename. That context is accessible in:
- a custom pipeline (where you'll have to program)
- use the IBaseMessage you get passed by the pipeline framework
- that interface provides you with a context property, which you can use to read promoted properties from
- to give you a feeling, here's a (non-tested!) example:
public IBaseMessage Execute(IPipelineContext pc, IBaseMessage inmsg)
{
//following namespace contains the FILE adapter properties:
const string FILEadapterTargetNamespace = "http://schemas.microsoft.com/BizTalk/2003/file-properties";
// retrieve the inbound file name that was promoted by the FILE adapter:
string sourcePath = inmsg.Context.Read("ReceivedFileName", FILEadapterTargetNamespace ) as string;
return inmsg;
}
- an orchestration (here you have intellisense)
- drop in there an expression shape
- you can access the context of you inbound message, using the context syntax (regular brackets after the messagename with in between the propertyname)
The FTP adapter does same thing upon receival of messages but obviously uses another context property to promote that information on.
It's namespace: http://schemas.microsoft.com/BizTalk/2003/ftp-properties
It's file name context property: ReceivedFileName
So far on the receival side of things, but what about sending? When sending files, the FILE and FTP differ somewhat in their approach...
Let's start again with the FILE adapter, where you basically have two options when you want to specify the outbound file name:
- Using a static port: specify the outbound file name using one of the supported macro's
This one's easy: the FILE adapter offers you some macros that you can use when binding the BizTalk Server messaging port.
MSDN explains this very well, so I will not go into any details further here... (In addition I remember Jan already posted on this subject as well!)
- Using a dynamic port: specify the outbound file name by setting context on the outbound message
Setting context can be done both in a custom pipeline component as well as in an orchestrations expression shape. In any case, when using a dynamic send port in an orchestration, you should set the outbound URL. This can be done in any expression shape (you have intellisense for this). The engine will parse the URL and based upon the prefix (file:// or ftp://) it will use the correct adapter. In addition you may add other context as well (for example specifying the message's retry count).
Setting context using a custom pipeline component is perhaps less straight forward, so let's look into this using a little sample:
IBaseMessage Execute(IPipelineContext pc, IBaseMessage inmsg)
{
//following namespace contains the FILE adapter properties:
const string FILEadapterTargetNamespace = "http://schemas.microsoft.com/BizTalk/2003/file-properties";
// set the the outbound file name for the FILE adapter:
inmsg.Context.Write("ReceivedFileName", FILEadapterTargetNamespace , "myOwnNewFileName.xml");
return inmsg;
}
So... that's it :-) Easy huh! The previous method can as well be used when having a static send port! If you explicitly set the filename in the context the FILE adapter will take this into account and won't use the "default" values specified for that BizTalk Server messaging port.
Now how does this work with the FTP adapter you might wonder? Unfortunately the FTP adapter does not have any macros you can use to set the outbound file name. However both the approach using the dynamic port as well as setting context manually works very well.
For example, inside an orchestration, I might set the location of a dynamic send port like this:
//Set dynamic ports location:
orchprtSndMyMessagePort(Microsoft.XLANGs.BaseTypes.Address) = "ftp://myserver/mydirectory/myfilename.xml";
//Set context on the message you want to send:
msgMyMessage(FTP.UserName) = "myFTPUsername";
msgMyMessage(FTP.Password) = "myFTPPassword";
msgMyMessage(BTS.RetryCount) = 20;
msgMyMessage(BTS.RetryInterval) = 2;
As illustrated, you'll need to specify the username and password for the connection as well. All together I guess this is not a bad approach at all. (Could be more user friendly but once you know, it dóes makes sense, right? In the end, évery adapter gets it's configuration from the message context!)
Very nice, very nice... but, what if you want to be able to transmit files in a particular order, or what if I want to grasp the return value from the FTP adapter's send operation?
There's no way you can grasp the return value of the FTP operation directly. Such kind of things is completely hidden by the adapter itself. BUT there's a way that allows you to know inside your orchestration wether the transmit operation succeeded. This is implemented using a switch on the level of your orchestration's send port, called: "DeliveryNotification". If set to "Transmitted", this will ensure that your orchestration stops working, until the adapter has succesfully transmitted the message. It's the same switch that allows you to send your messages in-order. In addition: if the message fails, an exception will be thrown of type: "XLang.DeliveryFailureException".
So far for files :-) If you have any further questions related to custom pipeline components... watch this blog! I will come back on this subject, showing you some very cool things!! (Obviously the newsgroups are still there as well.)
Have a nice day!
-
-
Carlo adds to my previous blog on custom BAM, that it is of course possible to combine these techniques with the things that are provided by the TPE (Tracking Profile Editor). Thanks for pointing this out Carlo.
Even better: I would suggest using the TPE for whatever it can provide you! The described techniques are only useful if the out-of-the-box tools don't fit your needs. (As always: custom code requires your own support, maintenance, deployment, testing, ... while anything out of the box is supported by a great team of specialists, available 24/24h, 7 days a week. You got my point.)
If even the suggested approach does not fit your BAMming needs, don't worry: BAM has even more advanced API's!
Just as the orchestration engine does, it is possible to "instrument" your own orchestration steps and .NET code with BAMmable data. This can be done by what is called the "BAM interceptor". (Same object as the orchestration engine internally uses.) The main advantage of this (way more advanced) technique is that you may configure the BAM needs afterwards. (While in what I explained previously in the BAMming blog entry, you kind of "hardcode" what is BAMmed exactly to which kind of activity items...)
So if you'd like to create your own TPE: go ahead, it's all possible! Just let me know if you succeeded please :-))))
-
-
To me, one of BizTalk Server's most exciting features is definitely BAM (Business Activity Monitoring)! In a very intuïtive way it allows you to do lots of great things... Of course, out of the box, BizTalk can't anticipate on every need you'll ever have regarding business monitoring... For example: as for now - out of the box - it is not possible to include variables as data in your BAM views and activities. Let's change this!! BizTalk Server actually DOES support inclusion of custom data in your BAM infrastructure! So... let's get started!
(Note: following description is rather short, very short! However it's just ment as a little reminder for those of you who occasionaly have used BAM already... For full explanations and in depth coverage please use the docs and of course, buy Scott's book!! :-))))
Ok, so now you've created a file that explains to BizTalk Server what kind of data you are expecting to be BAMmed. Let's give it to the server and tell it to prepare for that data, using the the "BAM Management Utility"
So, what exactly have we done now? The bm utility actually has created all views, tables and a bunch of SQL Server constructs to make BizTalk ready to receive and compute your BAM data. Accessing those constructs directly is only supported using the SQL Server views that end on: "_View". (This does not include the views like: "_CompletedView" or "_ActiveView".) All other access would be unsupported.
What you would normally do now is to open up the TPE (BizTalk Server Tracking Profile Editor) and configure which data in your orchestrations map to which BAM fields. Which is exactly what we are not going to do for now :-). Let's do somewhat more funky things: use BAM from within an orchestration expression shape! (Note: following applies just as well for any .NET component you'd wich to leverage BAM in!)
The most exciting part actually turns out not being difficult at all, watch this:
- I'll be a good .NET citizen and store the connection string to the BAM database in a .NET configuration file. (Curious how .config files apply to BizTalk Server orchestrations?? BizTalk actually allows you to tune it's appdomain configuration! Use it!!!!) You might fetch this like:
var_BAMConnectString = System.Configuration.ConfigurationSettings.AppSettings.Get("BAMConnectionString");
The connection string might look something like this:
"Integrated Security=SSPI; Data Source=.;Initial Catalog=BAMPrimaryImport"
-
Let's assume I'd want to track the contents of 2 variables inside my orchestration:
var_myDate and var_myString.
-
We need to initialise the BAM event stream like this:
obj_eventstream = new Microsoft.BizTalk.Bam.EventObservation.DirectEventStream(var_BAMConnectString,0);
The "0" in this case is the so called "flush threshold", which determines after how many BAM events need to happen before a commit will be done to the BAM databases. By setting this value to zero, we'll need to call "flush" explicitly.
- To begin doing cool things, just initiate the BAM activity like this:
obj_eventstream.BeginActivity("MyActivityName", var_newUniqueGuid.ToString());
- Further, we're going to need a new unique ID in a variable. A GUID will do fine:
var_newUniqueGuid = System.Guid.NewGuid();
-
Update the activity with any custom data you want:
obj_eventstream.UpdateActivity("MyActivityName", var_newUniqueGuid.ToString(), "MyDateValue", var_myDate , "MyStringValue", var_myString);
Obviously the first parameter is the name of your activity. The second one is the activity instance ID or the contiuation token. In this case, since we don't reference any other activities, I just created a new GUID. All parameters after those initial two, is data that is stored for this activity. BizTalk requires you to give this data as name-value pairs. In this case I've added 2 fields called "MyDateValue" and "MyStringValue" that have their actual data values stored in
var_myDate and
var_myString.
- End the activity like this:
obj_eventstream.EndActivity("MyActivityName", var_newUniqueGuid.ToString());
- Last, but certainly not least, don't forget to commit this data by using the flush method:
obj_eventstream.Flush();
Surprisingly easy huh? Great stuff in my humble opinion!!!!!!!!
-
-
Just spoiled by my litlle... she actually rules doing the CSS for this blog! Really like these colors :-)
As for now reading: "XQuery - The XML Query Language". While I only managed to find two books on the subject that seemed somewhat decent, this one definiately was a good choice. Michael Brundage (technical lead for XQuery within MS) does a terrific job in explaining these concepts! This guy really should get a blog up and running as fast as possible! (If he actually has one, let me know please, couldn't find anything up till now.)
If you're into XML (and you should!), this one's a must read. I'm confident that XQuery will change the way we (programmaticaly!) deal with XML in a great way. What SQL is for any RDMBS, XQuery will be for XML.
"You're not serious, are you?"
Yes I am!! Just check out the MSDN XML Developer Center and take a 10 second look at the two articles on SQL Server 2005 Dare has published over there. One is more of a feature overview with regards to XML, while the other is a best practices guide. You'll quickly realise that SQL Server will heavily rely on the XQuery syntax for al kind of XML processing, selections, ... Even better: in the .NET framework 2.0, XQuery will be a first class citizen!! Just can't wait to get my hands dirty...
First impressions? XQuery has quite a charming syntax. Almost seems like kind of a blend between a lot of languages! As an example: prolog expressions use a CSharp like ";" while if-then-else constructs use more of a VB syntax. Variables seem directly to be taken from the XSLT specs and are written with a dollar sign in front...
I was very happy to see how all concepts used in XML techologies like: basic XML, XML Schema, XSLT, XPath were carefully taken, reused and integrated into the language! If you master any of these technologies, you'll definitely benefit from this...
More on this as I make progression in my XQuery journey!
-
-
Really need to thank some people:
- First of all to my sweet for doing the great css customisation to this blog and for the past weeks which were really great! Thanks for all this. Love ya!
- Secondly, to Jan, for introducing me to the blogging community. Jan is a tremendously experienced and respected blogger, and has some high quality sharepoint content online. If you're into SPS: make sure to check out his brand new Sharepoint 2004 smartpart!!
- Further, of course, to Scott, for getting my blog up and running!
- And last - but certainly not least - to Tom, for getting my previous article online on MSDN and for introducing me as well :-)
- To Carlo, thanks for the welcoming :-) - I'll be in Amsterdam as well :-)))
So, to all of you: thanks :-)
-
-
HAT is a really cool tool, but as always: sometimes you'd just wish you could control everything yourself. (If you don't agree: you are not a developer are you? ;-) ) Especially for monitoring the health of your system and for automating certain operations, being able to code a little HAT yourself would be quite handy, wouldn't it? (Note: Microsoft Operations Manager is there as well to help you, provided that you are working with this application.)
Even more: in a real life scenario, the people monitoring your systems probably won't be BizTalk experts! Being able to give them some simplified views while:
- hiding all BizTalk "internals"
- written to show the health for just a particular application
- offering the very basic operations like "suspend" or "save this message", specialised for your application
would be a very cool thing to do won't it?
Now: here you are!! I just opened the GotDotNet floor for a little community collaboration in this area. And to make it even better: I even added a fully functional sample, showing some basics!
Now, what does the sample exactly offer you?
The BizTalk Server 2004 Tracking Playground is a little HAT-like winform application that supports:
- enumeration of orchestration types
- enumeration of pipeline types (including custom ones)
- enumeration of orchestration instances, filtered on the type of orchestration
- enumeration of pipeline instances, filtered on the type of pipeline
- enumeration of message instances related to a set of given orchestration instances or pipeline instances - that set of instances can be selected using a multi-select listbox
- saving both tracked and suspended messages to disk
Please note that the BizTalk Server 2004 Tracking Playground is not targetted at creating a full featured production quality coded framework for doing tracking operations. Rather its intention is to provide people who are planning to write their own customised tracking or querying, with a very good head start and some decent sample code!
Note: Since a lot of people on the BTS newgroups keep asking for more samples written in VB.NET, this one was written in VB.NET. Anyone who would like to rewrite: please let me know and we'll work something out!
Interested? Please find these cool things on the GotDotNet Workspace for this project or download the V1 version directly. As always I'd really like to receive tons of feedback and collaboration!!
So that'll be it for today :-)
-
-
Earlier on, I wrote a little introduction on messaging and loose coupling for those of you rather new to BizTalk Server 2004. Please feel free to give comments feedback. The article was published on the Belgium MSDN site as well.
Hope you enjoy!
-
-
Ok, here we are! Finally! After reading a couple of thousands of blogs from al kinds of friends, collegues and other IT experts: here I am, ready to join the blogosphere... Which is in fact is about right on time because I've recently finished a really cool BizTalk Server sample! (Still in progress of writing some docs and additional information, so I expect this to be published somewhere later this month.)
Best regards, please enjoy what's about to come :-)
Christof