Daniel Cazzulino's Blog

Note: this entry has moved.

As Dare noticed, this is the month of the XmlReader. Here's a new player: the XPathIteratorReader. This time, the scenario is the following: you have an XPathDocument or XPathNavigator, and need to get a reader containing a subset resulting from an XPath query. For example, you may need to return all feeds items from an XML file that contain a certain word in the title:

Just like the XmlFragmentStream, this reader fakes a root node for the iterator. Additional constructor overloads allow you to change the default root node which is <root>:

This reader also inherits from XmlTextReader, using the same technique of the XPathNavigatorReader. This means you can validate subsets of nodes against selective XML Schemas too. It also implements IXmlSerializable, so you can directly return this subset of nodes from a web service for example.

Subsets can be written down to disk easily using XmlWriter.WriteNode() method:

There are a couple interesting things inside this class:

• It leverages the XPathNavigatorReader for each item in the iterator. So it basically passes through property and method calls to it.
• Depth is increased by one all the time, except for the faked root element.
• Instead of having ifs on all XmlTextReader overrides checking whether it's at the faked root or not, I decided to go for the more elegant approach of creating a FakedRootReader class. So the code in XPathIteratorReader becomes drastically simpler. It's mostly passing calls down to the current reader and that's it. Therefore, the only braching code exists in the Read method, and it's really trivial, basically checking with the current ReadState and creating the FakedRootReader if necessary: public override bool Read() <br /> {<br /> // Return fast if state is no appropriate.<br /> if (_current.ReadState == ReadState.Closed || _current.ReadState == ReadState.EndOfFile)<br /> return false;<br /> <br /> bool read = _current.Read();<br /> if (!read)<br /> {<br /> read = _iterator.MoveNext();<br /> if (read)<br /> {<br /> // Just move to the next node and create the reader.<br /> _current = new XPathNavigatorReader(_iterator.Current);<br /> return _current.Read();<br /> }<br /> else<br /> {<br /> if (_current is FakedRootReader &amp;&amp; _current.NodeType == XmlNodeType.EndElement)<br /> {<br /> // We're done!<br /> return false;<br /> }<br /> else<br /> {<br /> // We read all nodes in the iterator. Return to faked root end element.<br /> _current = new FakedRootReader(_rootname.Name, _rootname.Namespace, XmlNodeType.EndElement);<br /> return true;<br /> }<br /> }<br /> }<br /> <br /> return read;<br /> }
• The IXmlSerializable implementation uses the following trick: it loads the incoming document, moves to the root and makes this the new "faked" root, and uses an iterator over all root node children as its new internal state :D. Here it is: void IXmlSerializable.ReadXml(XmlReader reader)<br /> {<br /> XPathDocument doc = new XPathDocument(reader);<br /> XPathNavigator nav = doc.CreateNavigator();<br /> <br /> // Pull the faked root out.<br /> nav.MoveToFirstChild();<br /> _rootname = new XmlQualifiedName(nav.LocalName, nav.NamespaceURI);<br /> <br /> // Get iterator for all child nodes.<br /> _iterator = nav.SelectChildren(XPathNodeType.All);<br /> }

Note: this entry has moved.

So, the following code showed up in that post, *unescaped* inside that xmp element:

<XPathNavigatorReader>
  <titles>
    <title_id>BU2075</title_id>
    <title>You Can Combat Computer Stress!</title>
  ...

Bottom line: escape content to avoid buggy sniffers... Damn, I was so happy to avoid harmful escaped markup...

Note: this entry has moved.

Page views: 1.421
Downloads: 243
Last 30 days we've had a steady rate of increase in page views of 200% a day, and 150% in downloads.

Note: this entry has moved.

Back at the 2004 MVP Global Summit, I met fellow XML fan Kirk, who was seeking a solution to the following problem: you have a (several) megabytes file containing multiple XML fragments, and you want to read it (in his case specially through the SgmlReader). The problem is, of course, that the XmlTextReader will throw an exception as soon as it finds the second fragment, unless you use the special ctor. overload that takes an XmlParsingContext. Dare shows an alternate solution based on XML inclusion techniques, either DTD external entities or XInclude.

These techniques effectively expose a fully well-formed document to your application, which has a number of benefits, including the ability to transform it if you need to, for example. But I was thinking more along the lines of providing a class that could actually read the fragments without resorting to those mechanisms. I couldn't cheat the XmlTextReader, so I decided to go one step lower. The result is the XmlFragmentStream, a class that wraps any System.IO.Stream and fakes the missing root element, so that an XmlTextReader layered on top of it, will think the document is well-formed. Here's how to use it:

Given the following XML fragments:

You can read (and even validate with an XmlValidatingReader) using this code:

The XmlFragmentStream class also contain two contructor overloads that allow you to specify the name and namespace of the enclosing root element (by default <root>):

public XmlFragmentStream(Stream innerStream, string rootName)
public XmlFragmentStream(Stream innerStream, string rootName, string namespaceURI)

This technique is proven by a real world (surely happy) customer Kirk helped ;). What's more, he even contributed a bug-fix he found when using it.
The performance impact of this approach in negligible because the class is basically an intermediary with minimal processing.

As Oleg noted pointed in a comment (and motivated a slight editing in this post), as well as showed in his weblog, you can do this with the aforementioned special XmlTextReader constructor overload, passing an XmlParsingContext. This is more cumbersome, in my opinion, and still leaves you with the problem of not having a valid XML document.

#region using

using System;
using System.IO;
using System.Text;

#endregion using

namespace Mvp.Xml.Common
{
 /// <summary>
 /// Allows streams without a root element (i.e. multiple document 
 /// fragments) to be passed to an <see cref="System.Xml.XmlReader"/>.
 /// </summary>
 /// <remarks>A faked root element is added at the stream 
 /// level to enclose the fragments, which can be customized 
 /// using the overloaded constructors.
 /// <para>Author: Daniel Cazzulino, kzu@aspnet2.com</para>
 /// See: http://weblogs.asp.net/cazzu/archive/2004/04/23/119263.aspx.
 /// </remarks>
 public class XmlFragmentStream : Stream
 {
 #region Fields

 // Holds the inner stream with the XML fragments.
 Stream _stream;

 bool _first = true;
 bool _done = false;
 bool _eof = false;

 // TODO: there's a potential encoding issue here.
 byte[] _rootstart = UTF8Encoding.UTF8.GetBytes("<root>");
 byte[] _rootend = UTF8Encoding.UTF8.GetBytes("</root>");
 int _endidx = -1;

 #endregion Fields

 #region Ctors

 /// <summary>
 /// Initializes the class with the underlying stream to use, and 
 /// uses the default <root> container element. 
 /// </summary>
 /// <param name="innerStream">The stream to read from.</param>
 public XmlFragmentStream(Stream innerStream)
 {
 if (innerStream == null)
 throw new ArgumentNullException("innerStream");
 _stream = innerStream;
 }

 /// <summary>
 /// Initializes the class with the underlying stream to use, with 
 /// a custom root element. 
 /// </summary>
 /// <param name="innerStream">The stream to read from.</param>
 /// <param name="rootName">Custom root element name to use.</param>
 public XmlFragmentStream(Stream innerStream, string rootName) : this (innerStream)
 {
 _rootstart = UTF8Encoding.UTF8.GetBytes("<" + rootName + ">");
 _rootend = UTF8Encoding.UTF8.GetBytes("</" + rootName + ">");
 }

 /// <summary>
 /// Initializes the class with the underlying stream to use, with 
 /// a custom root element. 
 /// </summary>
 /// <param name="innerStream">The stream to read from.</param>
 /// <param name="rootName">Custom root element name to use.</param>
 /// <param name="ns">The namespace of the root element.</param>
 public XmlFragmentStream(Stream innerStream, string rootName, string ns) : this (innerStream)
 {
 _rootstart = UTF8Encoding.UTF8.GetBytes("<" + rootName + " xmlns=\"" + ns + "\">");
 _rootend = UTF8Encoding.UTF8.GetBytes("</" + rootName + ">");
 }

 #endregion Ctors

 #region Stream abstract implementation

 /// <summary>See <see cref="Stream.Flush"/>.</summary>
 public override void Flush()
 {
 _stream.Flush();
 }

 /// <summary>See <see cref="Stream.Seek"/>.</summary>
 public override long Seek(long offset, SeekOrigin origin)
 {
 return _stream.Seek(offset, origin);
 }

 /// <summary>See <see cref="Stream.SetLength"/>.</summary>
 public override void SetLength(long value)
 {
 _stream.SetLength(value);
 }

 /// <summary>See <see cref="Stream.Write"/>.</summary>
 public override void Write(byte[] buffer, int offset, int count)
 {
 _stream.Write(buffer, offset, count);
 }

 /// <summary>See <see cref="Stream.CanRead"/>.</summary>
 public override bool CanRead { get { return _stream.CanRead; } }

 /// <summary>See <see cref="Stream.CanSeek"/>.</summary>
 public override bool CanSeek { get { return _stream.CanSeek; } }
 
 /// <summary>See <see cref="Stream.CanWrite"/>.</summary>
 public override bool CanWrite { get { return _stream.CanWrite; } }
 
 /// <summary>See <see cref="Stream.Length"/>.</summary>
 public override long Length { get { return _stream.Length; } }
 
 /// <summary>See <see cref="Stream.Position"/>.</summary>
 public override long Position 
 { 
 get { return _stream.Position; }
 set { _stream.Position = value; }
 }

 #endregion Stream abstract implementation

 #region Read method 

 /// <summary>See <see cref="Stream.Read"/>.</summary>
 public override int Read(byte[] buffer, int offset, int count)
 {
 if (_done) 
 {
 if(!_eof)
 {
 _eof = true;
 return 0;
 }
 else
 {
 throw new System.IO.EndOfStreamException(SR.GetString(SR.XmlFragmentStream_EOF)); 
 }
 }

 // If this is the first one, return the wrapper root element.
 if (_first)
 {
 _rootstart.CopyTo(buffer, 0);

 _stream.Read(buffer, _rootstart.Length, count - _rootstart.Length);

 _first = false;
 return count;
 }

 // We have a pending closing wrapper root element.
 if (_endidx != -1)
 {
 for (int i = _endidx; i < _rootend.Length; i++)
 {
 buffer[i] = _rootend[i];
 }
 return _rootend.Length - _endidx;
 }

 int ret = _stream.Read(buffer, offset, count);

 // Did we reached the end?
 if (ret < count)
 {
 _rootend.CopyTo(buffer, ret);
 if (count - ret > _rootend.Length)
 {
 _done = true;
 return ret + _rootend.Length;
 }
 else
 {
 _endidx = count - ret;
 return count;
 }
 }

 return ret;
 }

 #endregion Read method 
 }
}

Note: this entry has moved.

The full project source code can be downloaded from SourceForge

Check out the Roadmap to high performance XML.

Note: this entry has moved.

There are many reasons why developers don't use the XPathDocument and XPathNavigator APIs and resort to XmlDocument instead. I outlined some of them with regards to querying functionality in my posts about how to take advantage of XPath expression precompilation, and How to get an XmlNodeList from an XPathNodeIterator (reloaded).

XPathNavigator is a far superior way of accessing and querying data because it offers built-in support for XPath querying independently of the store, which automatically gain the feature and more importantly, because it abstracts the underlying store mechanism, which allows multiple data formats to be accessed consistently. The XML WebData team has seriously optimized the internal storage of XPathDocument, which results in important improvents both in loading time and memory footprint, as well as general performance. This was possible because the underlying store is completely hidden from the developer behind the XPathNavigator class, therefore, even the most drastic change in internal representation does not affect current applications.

However, some useful features of the XmlDocument and XmlReader classes are not available. Basically, I've created an XmlReader facade over the XPathNavigator class, which allows you to work against either an streaming or a cursor API. I'll discuss how the missing features are enabled by the use of the new XPathNavigatorReader class, part of the opensource Mvp.Xml project.

Examples use an XML document with the structure of the Pubs database.

Serialization as XML

Both the XmlDocument (more properly, the XmlNode) the and XmlReader offer built-in support to get a raw string representing the entire content of any node. XmlNode exposes InnerXml and OuterXml properties, whereas the XmlReader offers ReadInnerXml and ReadOuterXml methods.

Once you go the XPathDocument route, however, you completely loss this feature. The new XPathNavigatorReader is an XmlReader implementation over an XPathNavigator, thus providing the aforementioned ReadInnerXml and ReadOuterXml methods. Basically, you work with the XPathNavigator object, and at the point you need to serialize it as XML, you simply construct this new reader over it, and use it as you would with any XmlReader:

Another useful scenario is directly writing a fragment of the document by means of the XmlWriter.WriteNode method:

This code saves each book with a price bigger than 10 to a file named after the title id. You can note that the reader works in the scope defined by the navigator passed to its constructor, effectively providing a view over a fragment of the entire document. It's also important to observe that even when an evaluation will cause a cursor movement to the navigator in it.Current, the reader we're using will not be affected, as the constructor clones it up-front. Also, note that it's always a good idea to precompile an expression that is going to be executed repeatedly (ideally, application-wide).

XmlSerializer-ready

The reader implements IXmlSerializable, so you can directly return it from WebServices for example. You could have a web service returning the result of an XPath query without resorting to hacks like loading XmlDocument s or returning an XML string that will be escaped. XPathDocument is not XML-serializable either. Now you can simply use code like the following:

This web service response will be:

<XPathNavigatorReader>
 <titles>
 <title_id>BU2075</title_id>
 <title>You Can Combat Computer Stress!</title>
 <type>business </type>
 <pub_id>0736</pub_id>
 <price>2.99</price>
 <advance>10125</advance>
 <royalty>24</royalty>
 <ytd_sales>18722</ytd_sales>
 <notes>The latest medical and psychological techniques for living with the electronic office. Easy-to-understand explanations.</notes>
 <pubdate>1991-06-30T00:00:00.0000000-03:00</pubdate>
 </titles>
</XPathNavigatorReader>

XML Schema Validation

Imagine the following scenario: you are processing a document, where only certain elements and their content need to be validated against an XML Schema, such as the contents of an element inside a soap:Body. If you're working with an XmlDocument, a known bug in XmlValidatingReader prevents you from doing the following:

The validating reader will throw an exception because it expects an instance of an XmlTextReader object. This will be fixed in Whidbey, but no luck for v1.x. You're forced to do this:

Of course, you're paying the parsing cost twice here. The XPathNavigatorReader, unlike the XmlNodeReader, derives directly from XmlTextReader, therefore, it fully supports fragment validation. You can validate against XML Schemas that only define the node where you're standing. The following code validates all expensive books with a narrow schema, instead of a full-blown Pubs schema:

This opens the possiblity for modular validation of documents, which is specially useful when you have generic XML processing layers that validate selectively depending on namespaces, for example. What's more, this feature really starts making the XPathDocument/XPathNavigator combination a more feature-complete option to XmlDocument when you only need read-only access to the document.

Finally, I imagine you could even think about loading an XmlDocument from an XPathNavigator using the XPathNavigatorReader... although can't think of any good reason why would you want to do such a thing :S...

The full project source code can be downloaded from SourceForge .

Enjoy and please give us feedback on the project!

Special credits: the idea of a reader over a navigator isn't new. Aaron Skonnard did an implementation quite some time ago, as well as Don Box (you'll need to search the page for "XPathNavigatorReader". Mine is not based on theirs, and has features lacking on them, but they came first, that's for sure ;).

Check out the Roadmap to high performance XML.

Note: this entry has moved.

Finally I got tired of creating Windows Forms applications by mistake. You know, when you select Add New Project to a solution, it's the first item. I create FAR more class library projects than UI clients!

So, I went to the C:\Program Files\Microsoft Visual Studio .NET 2003\VC#\CSharpProjects folder, open CSharp.vsdir in notepad, and switch the priority of CSharpEXE.vsz with the one for CSharpDLL.vsz. This is the fouth component of each line. It should look like the following:

That's it. It will be the first item in the list forever... (until I hit one of those mondays at least)

BTW, there's no magic hacking here, it's documented.

Note: this entry has moved.

The whole week I've been digging into a pretty large configuration file and its schema. It's Shadowfax (Sfx), which I already introduced in a previous post. I see some points that allow for improvements, which mainly have to do with namespaces and extensibility.
Let's recap about what namespaces are for. Here's what the W3C Namespaces in XML specification says in the motivation section:

We envision applications of Extensible Markup Language (XML) where a single XML document may contain elements and attributes (here referred to as a "markup vocabulary") that are defined for and used by multiple software modules. One motivation for this is modularity; if such a markup vocabulary exists which is well-understood and for which there is useful software available, it is better to re-use this markup rather than re-invent it.

Such documents, containing multiple markup vocabularies, pose problems of recognition and collision. Software modules need to be able to recognize the tags and attributes which they are designed to process, even in the face of "collisions" occurring when markup intended for some other software package uses the same element type or attribute name.

So, namespaces should be used when you expect a document to be extended by aggregating elements from multiple disparate schemas. This motivation has to drive the design of the schema, to allow for easy extensibility while retaining XML-friendliness with regards to the format. The following concrete points could be improved:

1. Element prefixing: this is just a fragment of the file as it is now:
   <referenceArchitecture> <ra:settings xmlns:ra=".../ReferenceArchitectureSection.xsd"> <ra:targets> <ra:target ... /> </ra:targets> <ra:pipelines> <ra:serviceInterfacePipelines> <ra:pipeline ... /> </ra:serviceInterfacePipelines> <ra:serviceImplementationPipelines> <ra:pipeline ... /> </ra:pipelines> </ra:settings> </referenceArchitecture> Clearly, using a prefix is not necessary here. The Sfx namespace is the only one used in the whole document, so a default unprefixed namespace could be used on the root of the hierarchy, and the namespace rules of XML would propagate it to its children. Therefore, the fragment above is absolutely equivalent from the point of view of XML to the following one: <referenceArchitecture> <settings xmlns=".../ReferenceArchitectureSection.xsd"> <targets> <target ... /> </targets> <pipelines> <serviceInterfacePipelines> <pipeline ... /> </serviceInterfacePipelines> <serviceImplementationPipelines> <pipeline ... /> </pipelines> </settings> </referenceArchitecture> Consistency of namespaces and their use is also desired across the schemas for <referenceArchitecture>, <businessActionsDefinition>, <eventConfiguration>, etc.
2. Attributes with namespace: attributes shouldn't be assigned namespaces. It's common practice (and the W3C default) to leave attributes without namespaces. This also makes for more readable files. This default is changed in the Sfx schema by setting the attributeFormDefault= "qualified". What this means is that all attributes in the instance document (the actual configuration file) must be prefixed, as the attributes are now part of the targetNamespace: <ra:pipelines> <ra:serviceInterfacePipelines> <ra:pipeline ra:name="Default" ra:serviceActionName="*" ra:targetName="inproc"> </ra:pipeline> </ra:serviceInterfacePipelines> ... This is pretty cumbersome to read and author, and doesn't really add value to the extensibilty/usability of the schema and the config file. This may be a valid (and even necessary) approach for a highly composed document such as a SOAP message is, where every WS-* spec defines its own attributes and elements, and almost everything is prefixed. But I wonder if this actually necessary in a config file... Leaving the default attributeFormDefault  (or omitting it) in the schema, gives you the following valid instance: <pipelines> <serviceInterfacePipelines> <pipeline name="Default" serviceActionName="*" targetName="inproc" /> </serviceInterfacePipelines> ... I believe this is far better and more familiar. Extensibility isn't hurt, as the xs:anyAttribute can still be used, but now you only force attribute prefixing on extensions, not built-in values, which are the more commonly used. This brings us to the last point.
3. Schema and configuration extensibility: Sfx is meant to be flexible and allow a wide range of applications. With this idea in mind, almost everything is configurable... to an extent. One of the key pieces in this architecture (and any other SOA-like) is to provide a platform of common services where your services (let's call them business actions -BA- as in Sfx) run. I envision that some BAs may need additional configuration in order to perform their work. I've worked on such an architecture and BA developers started developing custom configuration mechanisms for their libraries because the infrastructure didn't provide it, which led to serious maintenance and deploy problems. 
   So, the schema for BA configuration should allow for open content in order to accomodate extensibility elements/attributes.
4. Configuration versioning: given the current target namespace for the configuration schema (http://www.microsoft.com/practices/referencearchitecture/services/03-08-2004/ReferenceArchitectureSection.xsd) it's only natural to infer that versioning will be handled through namespace changes, according to the release date. There's a lot of discussion in the community about schema versioning, but most agree that versioning through namespace changes is not recommended. This document explains in a short and consice manner the available options. My suggesion to make the migration path in the future when configuration is upgraded as easy as possible for developers (and an optional upgrade tool) is to use the optional XSD version attribute in the schema, together with a new schemaVersion attribute in the configuration file. The schema would look like the following: <xs:schema id="ReferenceArchitectureSection" targetNamespace="http://www.microsoft.com/practices/referencearchitecture" version="1.0" ...> <xs:complexType name="settings"> <xs:sequence> ...etc... </xs:sequence> <xs:attribute name="schemaVersion" type="xs:decimal" use="required"/> </xs:complexType> While the configuration would include the appropriate version attribute: <referenceArchitecture> <settings xmlns="http://www.microsoft.com/practices/referencearchitecture" schemaVersion="1.0"> <targets> <target ... /> </targets> Now when v2 comes out, a tool can detect the version in the configuration file, and perform any relevant upgrade (for example through an XSLT transformation to accommodate elements to the new format).

Of course configuration is just the tip of the iceberg of such a comprehensive product. Shadowfax is a very interesting architecture to build applications on top. MS is very open on feedback from the community, so I expect it to become more and more polished and sleek over time. These are my 2 cents with regards to its configuration file.

If I misunderstood some points in the schema design, I'd be glad to hear from the Sfx dev guys!

Note: this entry has moved.

Recently, I started working with Shadowfax, a reference architecure coming from the Patterns & Practices group at Microsoft. You have a look at an overview of the architecture at the GDN workspace home. I also recommend Hernan de Lahitte's blog, one of the guys working on it. He has a nice introduction as as well as a closer inspection of the processing pipeline.

I don't to repeat those intros. I'll just say that if you're developing SOA-like architectures, or you need one for your project, you should definitely take a look. It's being developed in a very open manner, with source releases pretty often, with open feedback through the workspace, and even if it's not ready for prime time yet, it's a good indication of where MS thinks you should be going with your projects. It closely follows the advises from Application Architecture for .NET: Designing Applications and Services, and generally represents a pretty good compendium of best practices. It makes heavy use of several application blocks from PAG, su as Configuration Management Application Block, Authorization and Profile Application Block an the Logging Application Block.

As this is going to take most of my working day now, I'll start a new category for these posts. I hope ScottW adds support for subscribing to a single category soon (as dasBlog does since quite a while...). This way, you can subscribe to this single category if you want.

I have the task of helping developers have a smooth experience when programming against Shadowfax. I'll drop my ideas here, as well as the dev. aids I think could be useful. Even if you don't use Shadowfax, it would be cool to get your feedback... because it's possible that some of my ideas seem useful only to me! Stay tunned, lots of posts are coming ;)

Note: this entry has moved.

My previous post has been uploaded for more than 4 hours and I still don't see it in the feed! (try it at http://weblogs.asp.net/cazzu/Rss.aspx)
Highly dissapointing (to say the least).