CodeSmith Generator 6.0 Template Parser Progress Report - Part 2

In this post I’m going to talk about how the new template parser in CodeSmith Generator 6.0 creates an AST (Abstract Syntax Tree) and then uses the visitor pattern to iterate over the nodes in the AST and do transformations.

The grammar rules I talked about in my previous post are used to both validate the syntax of the template code and to create typed AST nodes.  Here is a very simple template:

<%@ CodeTemplate Language="C#" %>
<%= DateTime.Now %>

The resulting AST from the parse looks like this:

DirectiveNode Name=CodeTemplate
  AttributeNode Language=C#
ExpressionNode
  ValueNode DateTime.Now

Each node in the AST implements a visitor pattern and I’m actually using a visitor to generate the AST visualization above.  The visitor recursively goes through each node in the hierarchy and calls the Visit method for that specific node type.  So I end up with a visitor class that looks like this:

public class VisualizeTemplateVisitor : TemplateAstVisitorBase
{
    …
    public override void Visit(DirectiveNode node)
    {
        _builder.Append("DirectiveNode Name=");
        _builder.AppendLine(node.Value.Trim());
    }
    public override void Visit(AttributeNode node)
    {
        _builder.Append("  AttributeNode ");
        _builder.Append(node.Name.Trim());
        _builder.Append("=");
        _builder.AppendLine(Format(node.Value));
    }
    …
}

I simply call the Accept method on the visitor and pass it the root node that I want to visit and it creates the output above:

string nodeVisualization = VisualizeTemplateVisitor.Accept(templateNode)

This pattern is used extensively in the new parser.  Here are some of the visitors I have so far in the new parser:

• CollapseWhitespaceVisitor – used to traverse the AST and remove whitespace and line break nodes from lines that only contain template code.  This visitor actually modifies the AST while it is visiting it.
• PopulateParseResultVisitor – used to gather information about the template like what sub-templates are referenced, what include files are referenced, what assembly references are declared, what language the template is using (ie. C#, VB, JS) and many other things.
• GenerateCompileUnitVisitor – used to generate a CodeDOM compile unit from the AST that can be fed into the .NET compiler and create an assembly.

By creating an AST and using the visitor pattern, I’m able to do any sort of transformation that I want by simply creating a new visitor and plugging it into the template compilation pipeline.  As you can imagine, this ends up being a much cleaner, simpler and extensible solution than what we had before.  We may even expose the compilation pipeline with MEF in CodeSmith Generator 6.0 so that anyone could make changes to the generated code or implement new template directives.