Tales from the Evil Empire : Security

How to choose a client template engine

Bertrand Le Roy — Thu, 05 Feb 2009 08:03:00 GMT

Disclaimer: I worked on the Microsoft Ajax 4.0 template engine, so my criteria are of course heavily influenced by our own design.

Templates are a data rendering method that server-side developers have enjoyed since the old days of classic ASP and PHP. The idea was quite simple (add code blocks and dynamic expressions directly into HTML markup) but it revolutionized web development, which before that relied on the opposite method (spitting HTML from CGI code).

On the client-side, the browser provides two ways to generate HTML: innerHTML and the DOM API. Template rendering is of course possible, but only using a JavaScript library. To be honest, one should mention XSLT here, which is standard and widely supported but whose somewhat unusual syntax has had limited success convincing web developers.

There are literally dozens of template engines available today, using many different markup conventions and algorithms. Template engines are not equal, they have different levels of complexity, different feature sets, different security models. It can be quite hard to pick one, and it really depends on the requirements of your application and also on taste.

I’ll try to go over a few things that you might want to check on a template engine. You will want to cherry-pick what’s important to you and what’s not so that you can pick the tool that’s best adapted to your problem.

1. Expression delimiter

A template system is adding new semantics to an existing language. This is not completely harmless as you have to give a new meaning to sequences of characters that were understood before as plain literals. That means that an escaping mechanism must exist to express the literal contents that were given a new meaning. For example, if the expression delimiters are <%= and %>, how do you express the “<%=” and “%>” sequences of characters as literals?

<%= and %> are by far the most common and familiar delimiters (they have been used by ASP, PHP, JSP, ASP.NET and many others), but they have one flaw that couldn’t be seen back in 1996: XHTML compliance. Granted, you may not care at all about XHTML. If you don’t, just skip this. But if you do, any template engine that uses <%= %> simply can’t have template code that is XHTML-compliant. Now this may still not be a big deal if you don’t care that the template code is compliant, but only care about the generated markup.

But there is another reason why <%= %> is a bad choice: it’s conflicting with the server-side. A client-side engine must be able to coexist peacefully with whatever server-side technology you’re using. If your server technology of choice is using <%= %> like it very likely does, it will be conflicting with your client templates. The server-side engine will kick in when it sees these delimiters and will fail one way or another.

We chose {{ }} as the expression delimiter (and also {expr } for extensible markup expressions such as {binding }), which happens to be the same choice that Django made (and Dojo, as they implement Django on the client-side, and by the way I don’t know how they manage conflict with server-side Django; if you know, drop me a comment). It works fine with XHTML and it doesn’t conflict with ASP or PHP. If you want to express the “{{“ literal, we don’t provide an escape sequence per se, but you can still do it by writing {{ “{{“ }}.

On a final note, there are a few template engines that do not have expression delimiters but instead rely on microformats. While this absolutely preserves the HTML semantics, it’s often clumsier to use and quite limiting. It’s a matter of taste, but microformats also can look like markup within markup.

2. Expression language

In the simplest cases, you’ll want to inject a simple data field into the template, like {{ Price }}. But many times, you’ll want to go beyond those simple cases and embed a more complex expression into the markup. For example, you might want to display that price with two decimal places, like $42.00. In order to handle that case and more complex ones, it is necessary to have a full expression language.

In the same way, if the template engine allows for code blocks to introduce control structures such as conditional execution or loops, it needs a full programming language.

Some template engines, such as Django, introduce their own new language. Others, such as ours, just use what’s already available and familiar: JavaScript. In addition to being familiar, it eliminates the need to write a parser and interpreter, which reduces library code size and gives better performance.

3. Protection against injection attacks

If you don’t know what an injection attack is… wow. Just wow. Go read this, now. Seriously.
http://weblogs.asp.net/bleroy/archive/2004/08/18/please-please-please-learn-about-injection-attacks.aspx

The frightening thing about almost any single one of the client template engines out there is that they have no mechanism in place to protect against injection attacks, which puts the responsibility to encode and check contents in the hands of the application developer. That’s you.

Most of the engines out there are using a very simple algorithm to build HTML: array joining, and then injecting the result using innerHTML. This is very similar to building SQL by concatenating strings: in a nutshell, you shouldn’t do that. In the same way that SQL is better built using parameterized queries, HTML is more securely built using the DOM API which just eliminates the need for encoding.

Even when using encoding or the DOM API, there is a number of attributes that still present a fair amount of danger if user data gets injected in them. For example, if you bind the href attribute of a link to a piece of user data, no amount of encoding will protect the application against a user injecting “javascript:doSomeEvil();”.

To prevent that kind of attack, we white-list the protocols in all known URL attributes to relative URLs, http and https. It is possible for the application to extend this white list as needed (available in Microsoft Ajax 4.0 Preview 4 and later).

One thing to understand is that the dangerous contents here is the data, not the template itself, which is trusted as application code is.

4. Template location

Depending on the library, the template can be embedded in comments, in script tags or just be part of the DOM. It can also be included from a separate file.

Putting the template in comments or script tags easily hides its markup from initial rendering, but it makes it harder to design with existing tools and puts it out of the reach of markup validation tools.

Including from a separate file adds one more request to the server, which might be a problem or not: it might also improve performance by allowing partial caching. The best is to have a choice here, so I’d reject an engine where external files is the only possibility, knowing that all other engines allow external files with minimal effort by feeding a simple XHR’s result into the system.

Instead of using script tags or comments, we chose to embed the template as real HTML contents anywhere in the document and just hide it from initial rendering using CSS. This makes the code designable using any existing tool, and enables us to use the browser’s native HTML parser to understand the structure of the markup.

5. Hooking up events and instantiating components

In a modern Web application, you won’t just create plain HTML, you’ll want to hook up events and instantiate components. The template engine might let you do this inline, which is obtrusive but convenient, or it might require you to enrich the DOM after template instantiation through code, or it might give you a choice between the two.

One thing to look for is how much knowledge of the template markup’s shape must be injected into the code in order to enrich the DOM.

For a picture of the different possibilities using our engine, read this post:
http://weblogs.asp.net/bleroy/archive/2008/11/28/instantiating-components-on-template-markup.aspx

6. Live bindings

One big advantage of client rendering is that the data remains under raw data form until it reaches the template engine. That means that you have a representation of the data on the client. This in turn means there is an opportunity to monitor changes to the data and to bind UI directly to it, eliminating round-trips to the server along the way.

We call this ability live bindings and it makes for very responsive applications that are still easy to build.

7. Constraints on markup shape

One thing you should try with any template engine is to repeat a row in a table, using IE. This may sound trivial, but innerHTML doesn’t work on IE in a number of places, which puts a constraint on the markup that you need to write to render certain shapes of HTML. There is almost always a workaround but it can be more or less painful to implement. Try and compare, see what you find reasonable.

8. Ease of data field access

This one is quite simple. Some template engines force you to access the current data item’s field through some special syntax such as “$T.price”. In our engine and several others, you just write “price”.

9. Readability

This one is quite subjective, but when choosing a template engine, just look at a few samples. Does the code hurt your eyes? Can you understand it right away?

10. Performance

A template engine must be able to render a reasonable number of data items with a reasonably complex template under the perception threshold. What’s “reasonable” pretty much depends on your application design, but I’d say a hundred items in a typical grid layout should be hardly noticeable. So before you choose, do a mock-up of your design and data and try it. Increase the number of items and see where rendering starts to get noticeable. Usually rendering performance is not linear so this should give you a good idea of how far you can go with each engine. Better to determine that before you invest heavily in one engine.

11. XHTML compliance

As I said above, this is one you might not care about. Don’t leave yet, it actually goes both ways, as an engine, by enforcing XHTML compliance, might potentially get in the way of writing HTML the way you want.

The lowest level of compliance is that the rendered output is compliant. Being template engines, most if not all engines enable you to create compliant or horribly broken markup, depending on your own style. What it must absolutely not do is prevent you from generating compliant markup.

The second level of compliance is that the template code itself, before data is injected, is compliant. This is much less common. Even less common is that the compliance requirement does not impose too strong a constraint on the markup you have to write.

In other words, a good template engine lets you write XHTML but doesn’t force you to.

12. Nested templates

Rendering a template within a template might look like a whacky scenario at first if the engine understands control structures such as loops. In reality, if you include something like live bindings into the mix, you will want for example an inner template to be re-drawn when the outer data item changes. This will be much easier to achieve with nested templates.

So if this is a scenario that you care about, make sure that your engine handles that case gracefully.

In conclusion

There are many engines to choose from, and many criteria to check. I’m quite convinced that our own offering is pretty solid for all these criteria and offers a good compromise of features against simplicity and performance.

There is another engine that I quite like because it’s an impressively small bit of code that remains useful, and that is John Resig’s micro-template engine, which is only 20 lines of code. Like our own engine, it “compiles” the template into a reusable function that takes the data as a parameter. But unlike ours, it has no security mechanism and has issues with some markup, which may or may not be an issue for you. It’s a different kind of compromise that puts code size and simplicity before any other consideration.

I hope this helps in making a choice.

A good resource on client templates:
http://ajaxpatterns.org/Browser-Side_Templating

John Resig’s micro-templates:
http://ejohn.org/blog/javascript-micro-templating/

ASP.NET 4.0 Ajax Preview 3:
http://www.codeplex.com/aspnet/Wiki/View.aspx?title=AJAX

UPDATE: added performance and XHTML compliance to the list from Dave Reed’s feedback, and nested templates from Keith’s.

New tools to prevent SQL injection attacks

Bertrand Le Roy — Wed, 25 Jun 2008 17:00:00 GMT

I've blogged in the past about injection attacks. Microsoft publishes additional new tools to detect and protect against injection attacks. The first tool, developed by HP, crawls web sites to automatically detect possible attacks, the second blocks dangerous requests from being executed, and the last one analyzes code to look for dangerous practice.

http://www.microsoft.com/technet/security/advisory/954462.mspx

Please, please, please, learn about injection attacks!

Bertrand Le Roy — Thu, 19 Aug 2004 00:37:00 GMT

I answer a lot of posts on the forums of the ASP.NET site. And more often than I would like to, I answer a different question than the one the poster asked, because I happened to easily spot a potential injection attack in the posted code.

Now, what is an injection attack? If you don't know and you're a web developer, you're in trouble. Read on.

There are mainly two types of injection attacks, but both use the same vector of penetration: unvalidated user input.

Rule #1: User input is EVIL (pronounced eyveel, like the doctor of the same name) and should never be trusted. Validate all user input. In the case of a web application, user input is form fields, headers, cookies, query strings, or any thing that was input or sent by users (that may include some database data, or other sometimes more exotic input like mail or ftp).

The first type of injection attack, and the most deadly for most web sites are SQL Injection Attacks. It happens most of the time when the developer injects user input into a SQL query using string concatenation. For example:

SqlCommand cmd = new SqlCommand(
"SELECT ID, FullName FROM User WHERE Login='"
+ Login.Text
+ "' AND Password='"
+ Password.Text
+ "'");

This is C#, but I'm sure our VB programmer friends will get the idea (+ means &). This code is simply frightening, but I've seen it or a variation on it so often I just can't count. OK, why is it frigtening? Well, try to enter these strings into the login and password textboxes:

' OR ''='

There, you're authenticated! What happens is simple. Instead of being the simple text that you expected, the user input some evil text that contains the string delimiter and some SQL code that you're very generously executing.

Of course, this is not the worse that could happen. Any SQL command could be executed, especially if you've been careless enough not to restrict the rights of the ASP.NET user on your database. For example, the user could very well steal all the information in your database, completely obliterate it or even take complete control of the server. This leads us to rule #2, our fisrt counter measure:

Rule #2: Secure your database: don't use the sa user to connect to the database from your application, have a strong password on the sa user (and on any user), preferably use integrated authentication to keep all secrets out of your connection string and config file, and restrict the authorizations on your database objects to what's absolutely necessary (don't give writing rights to internet users except on log tables or forum tables, for example). This way, even if you accidentally write injectable code, the database will refuse to execute anything harmful beyond information disclosure (which could still be pretty bad). Please note that the above injection example will still work even if the database is secured.

So it may seem at first that the quotes are the usual suspects in this case. Actually, if you use this kind of code, you may have noticed a few glitches for example if people have legitimate quotes in their names. So what many people have been doing for a long time is to double the quotes in the input strings, something like Login.Text.Replace("'", "''"), or replace them with another harmless character (you can recognize these sites usually because they use the ` character instead of quotes). This gives a false sense of security, which is sometimes worse than no security at all. Consider this request:

"SELECT FullName FROM User WHERE ID=" + Identity.Text

Here, no need for quotes to inject code, all you need is space and letters. Enter 0 DELETE * FROM User into the textbox, and there goes your User table. And I'm sure a hacker creative enough could come up with wicked injections that don't even need spaces. Escape sequences in particular are a usual way to pass characters that were thought to be invalid in many applications (including, yes, Microsoft products whose name have two letters, begin with I and end with E). This leads us to the third rule:

Rule #3: Black lists are always incomplete (because hackers are many and potentially smarter than you). If you have to, rely on white lists, but never black lists.

A black list is a list of all characters you consider evil (like the quote). There will always be something missing in it. Consider this as a fact (even though of course it can be complete, but you should act as if this was not the case).

A white list is a list of authorized characters. What's great about it is that you know precisely what's permitted (what's in the list) and what's is forbidden (everything else). In the last example, restricting ID.Text to numeric characters is enough to secure the request.

And now for the good news. While it is useful to know all this about SQL Injections, the .NET framework (and all modern development frameworks, like Java) provide an excellent way to prevent injections: parametrized queries. Parametrized queries are safer, cleaner and make your code easier to read. Here are the two previous examples, rewritten as parametrized queries:

SqlCommand cmd = new SqlCommand("SELECT ID, FullName FROM User WHERE Login=@Login AND Password=@Password");
cmd.Parameters.Add("@Login", SqlDbType.NVarChar, 50).Value = Login.Text;
cmd.Parameters.Add("@Password", SqlDbType.NVarChar, 50).Value = Password.Text;

SqlCommand cmd = new SqlCommand("SELECT FullName FROM User WHERE ID=@ID");
cmd.Parameters.Add("@ID", SqlDbType.Int).Value = int.Parse(Identity.Text);

This way, there is no need to escape any characters because the parameter values are directly communicated in a strongly typed manner to the database.

N.B. In the second example, you may also want to use validator controls on the Identity TextBox, and check the validity of the page server-side before you build and execute the SQL query using Page.IsValid.

Rule #4: Use parametrized queries whenever possible.

Whenever possible? Does that mean that it's not always possible? Well, here's a little problem I got from the ASP.NET forums: you have a list of checkboxes on a page that have numeric identifiers as their values. Let's say that you must extract all the database rows that have the checked values. You'd want to write something like that (pseudo-code here):

SqlCommand cmd = new SqlCommand("SELECT FullName FROM User WHERE ID IN(@IdArray)");
cmd.Parameters.Add("@IdArray", SqlDbType.IntArray).Value = Convert.ChangeType(Request.Form["Identities"], typeof(int[]));

Unfortunately, there is no such thing as an array type for Sql. So in this case, unfortunately, unless someone comes up with something better, you have to rely on concatenation:

string idList = Request.Form["Identities"];
if (IntListValidate(idList)) {
SqlCommand cmd = new SqlCommand("SELECT FullName FROM User WHERE ID IN(" + idList + ")");
}
else {
throw new InvalidDataException("The posted data contains illegal characters.");
}
...
private bool IntListValidate(string input) {
for (int i = 0; i < input.Length; i++) {
if (!Char.IsDigit(input, i) && input[i] != ',') return false;
}
return true;
}

Of course, here, you have to use a white list, digits and comma in this case. Not even space is allowed. That's pretty safe, but I wish you could do that with parameters.

Update 8/19/2004: Kyle Heon just pointed me to this great Sql article that explains just how to do that with a parameter. Thanks for the link, Kyle! So now, there's one less reason not to use parameters everywhere.
http://msdn.microsoft.com/en-us/library/aa496058.aspx

The second common type of injection attack is Cross-Site Scripting, or X-Site Scripting. Consider this simple piece of asp page:

Bonjour, <%= Request.Form("Name") %>.

What if the user enters <script>alert("5tUp1dH4k3R ownz you");</script> in the Name form field? Well, he successfully displayed an alert on his browser using your page. Nothing to be afraid about for the moment, as he'll be the only one to see it. But what if instead of directly displaying the user input we store it in a database for other users to see, in a forum application, for example? What if the parameter is passed in the querystring parameters of a hyperlink in a mail that seems to come from your bank?

Well, then a lot of nasty things can happen (by the way, these are real scenarios, stuff that happened and continues to happen every day). For example, the attacker can inject script that will post all your cookies or some confidential information that's displayed on the page to some remote site that he owns. This can include your social security number, your authentication cookies, your credit card number, or any sensitive information that may be displayed on the page.

It is usually relatively harmless for many sites because they just don't have any information that's confidential or that could allow for other, more dangerous attacks.

Once again, ASP.NET gives a first line of defense out of the box. Since v1.1, all form and querystring data is validated on the server before the page is executed. So the above example does not work on ASP.NET 1.1, it will just throw an exception. Now, this feature is sometimes deactivated by controls such as rich text editors.

The second thing we're doing, which is what you should do in your own pages and controls, is to HtmlEncode any property that comes from user input. That includes the value of an input tag that's rendered from a TextBox. It protects this particular textbox from script injections and also makes it robust against legitimate characters in the contents, such as quotes.

Rule #5: Encode all rendered properties that come from user input when rendering them.

The above example would then become:

Bonjour, <%= Server.HtmlEncode(Request.Form("Name")) %>.

There's another often overlooked rule:

Rule #6: Don't display any secrets in error messages.

ASP.NET limits by default the complete error messages to calls from the local machine. A complete error message sent to any machine can reveal table names or other secrets that could give clues for some attacker to use. And usually, an error message gives an indication as to how to make this application fail, which can be repeated and improved on the basis of all the information the error message contains.

And of course, probably the most important rule:

Rule #7: Encrypt any page that contains sensitive data.

Of course, these rules are all important and the order in which they are presented here is irrelevant. Did I forget something?

If you need more information, here's some more reading:

On Sql Injections: http://www.governmentsecurity.org/articles/SQLInjectionModesofAttackDefenceandWhyItMatters.php

On Cross-Site Scripting: http://www.net-security.org/dl/articles/xss_anatomy.pdf

And of course, Google is your friend.

UPDATE: I've used the word "quote" in this article for both the apostrophe (or single quote) and double quote. Todd pointed out in the comments that was somehow ambiguous. The point is that anything that can be used as a string delimiter, or as a delimiter in general, should be considered suspicious. Double-quotes are more frequent, but some languages such as JavaScript use both single and double quotes. SQL uses single quotes. Bottom line: beware of delimiters, and remember you may not even know the list of possible delimiters.

UPDATE: we just released new tools that aim at helping developers scan their code for potential injection attacks.
http://www.microsoft.com/technet/security/advisory/954462.mspx