Part 1: Dynamic vs. Static
Part 2: Creating Dynamic Controls
Part 3: Adding Dynamic Controls to the Control Tree
Part 4: Because you don't know what to render at design time 

PART 2

Creating Dynamic Controls

Creating a dynamic control isn't just about "newing" one up. There are several different types of dynamic controls. There are also several different ways that dynamic controls can manifest themselves in your page's control tree, sometimes when you don't even realize it. As promised, understanding this will not only increase your understanding of dynamic controls, but of ASP.NET in general.

A Dynamic Control can be a:

1. Server Control
2. User Control
3. Parsed Control

But no matter what kind of control it is (dynamic or otherwise), they all share the same base class: System.Web.UI.Control. All built-in controls, custom server controls, and user controls share this base class. Even the System.Web.UI.Page class derives from it.

This section will focus very specifically on how dynamic controls are created. So the following code samples aren't really complete. Just creating a control dynamically isn't enough to get it participating in the page life cycle because it must be added to the control tree. But I like focusing in one thing at a time. Fully understanding something makes it easier to understand something that builds on it, because you can make assumptions and focus on the new functionality.

 

Dynamically creating Server Controls

There's not much to them. All the built-in controls are Server Controls, and so are any controls you create yourself that inherit from Control, WebControl, CompositeControl, etc.

 

Dynamically creating User Controls

First lets create a hypothetical user control:

A user control is not unlike a regular ASPX webform in that the ASCX contains markup that declares "static controls", and inherits from a code-behind class, and that code-behind class itself may or may not load dynamic controls.

Assuming this ASCX file were placed into the root of the application (denoted by "~/"), we could dynamically load it like this:

You might wonder why there's a difference in loading server controls and user controls. Why can't we create a user control just like a server control? This user control has a code-behind class "MyUserControl", so why can't we do this:

I invented a new code sample CSS style just to illustrate this. Blue means good. Red means bad. You definitely can't create a user control like this.

In Part 1 remember we examined how the page parser converts your markup into a class that inherits from your code-behind class? And remember that the generated code creates the controls, adds them to the control tree, and assigns them to your code-behind's control variables that are declared as protected? User Controls work the same way.

By creating the code-behind class directly you have effectively by-passed this whole process. The result will be an instance of your control with no control tree. Your control definitely won't work, and will probably result in a NullReferenceException the first time your code actually tries to use any of the controls (because, well, they're null).

That is why the "LoadControl" method exists, and that is why the actual ASCX file must stick around. LoadControl retrieves and instantiates the type of the auto-generated code created from your markup.

The path given to LoadControl must be the virtual path to the ASCX file. Physical paths aren't allowed. The virtual path can take on one of three forms. Lets assume the application is running under virtual directory name "Truly" (in other words, http://myserver/truly represents the root of the application). The three forms are:

Whatever the case, the virtual path must map inside the current application. No loading user controls that reside in other applications. So a root-relative path better map back into the current application ("Truly"), or you will get an error.

Relative paths are relative to the current page, or the current user control. That is important to know, because if you write a user control that loads another user control dynamically, you should know the difference between these:

The path is relative to the control you call the method on. The first line loads a control that exists in the same directory as the current user control. The second line loads a control that exists in the same directory as the current page, and the two don't have to be the same. Don't assume your user control is going to live in a particular location. It might be moved around. Really you have the same problem with Page.LoadControl, because you don't really know where in the directory structure the page loading your control will be.

It's usually best to keep user controls in one location, or at least segregated into multiple "silos" that logically separate them. That way you can always use a relative path like in the first line, yet you still have the freedom to move the controls around, just as long as you move them together. Pages that reference the controls will just have to know where they are, at least relatively.

You could also consider making the path to it configurable (via a property), use a web.config setting (less favorable), or use an app-relative path (ie, "~/UserControls/MyUserControl.ascx") and make the location known and unchangeable. 

 

Dynamically creating Parsed Controls

The ParseControl method is even more dynamic. It takes a string of markup and converts it into a control:

The string can contain virtually anything you can put on a page (disclaimer: I say virtually, because while I don't know of a limitation, there may be one. If anyone reading this knows of a limitation let me know). All of the controls contained within will be put together into a single control, where it's control tree contains the parsed controls.

Parsed controls are interesting, but use them very carefully. Parsed controls carry a performance burden, because the string must be re-parsed every time. Normally the parsed result of pages or user controls is cached.

Here's a dramatic demonstration of the performance hit. Here we create the same user control over and over again (1 million times):

And here we parse the content of the user control as a string over and over again:

And here are the results:

ParseControl took 253 seconds, loading the user control only took 19 seconds. That's a huge difference. Running ParseControl 1 million times in 253 seconds is still not a major performance issue (you're likely to have much more significant bottle necks in your application), but your mileage may vary, and you can't deny the performance results. Use it very sparingly. And definitely don't parse a string that came from a potentially malicious user! You would be enabling them to create any control that is available in the application, and they could use that to attack your site.

Any virtual paths you use inside the string follow the same rules as in the user control example. Relative paths are relative to the location of the control (or page) which you call ParseControl on. So there's that same subtle but important difference between the UserControl.ParseControl method and the Page.ParseControl method.

 

Dynamic Controls and Templates

There's another sneaky way that dynamic controls can be created. You have likely used templates without even realizing it. This topic is critical because if you understand how templates are used in databound controls like the Repeater, DataGrid, GridView, etc, then you will better recognize times where you may be approaching a problem with dynamic controls when you really don't have to. Later on there will be a section all about those situations.

Templates decrease the need for you to create dynamic controls manually because they let you declaratively define a "template" by which controls will be created dynamically for you. The Repeater control for example lets you define an ItemTemplate and optionally an AlternatingItemTemplate. Within the template you declaratively define controls:

The template contains a TextBox server control. Do you think the page that this repeater sits on has this TextBox, just like in our simple UserControl above? Do you think you can access this TextBox like this?

No no no. Remember -- red means bad. The TextBox you declared does not exist on the page, at all. There is no "build control" method generated for it, and it will not be assigned to any protected class variables you define (as described in Part 1).

Think about it for a minute. It doesn't even make sense for the TextBox to be accessible from the page like a regular statically declared control. The Repeater is going to be creating this TextBox once for every DataItem that is data bound to it. So there will be many of these on the page -- anywhere between 0 and N of them in fact. So if there were a page-level reference to it, which one would it point to? It doesn't make any sense.

When you declare a regular static controls, you are telling the framework "here is a control I'd like you to create and add to the page's control tree." When you define a template, you are saying "here is a control tree that I would like added to the page's control tree when the template is used." In the case of a repeater, the template is used once for every data item. But Templates don't have to be used that way (for example, the System.Web.UI.WebControls.Login control's LayoutTemplate property is a Template that isn't repeated). It's up to the control's implementation how the template is utilized.

Instead, the markup within the Template is parsed into an object that implements the System.Web.UI.ITemplate interface. Let's take a look at what code is generated for the static repeater in the example above:

Pay particular attention to the middle of this method. The repeater control has a property named ItemTemplate (which happens to accept objects of type ITemplate). The parser's generated code is creating a new CompiledTemplateBuilder and passing in a BuildTemplateMethod Delegate to it's constructor. The delegate is pointed at a method defined on this page, which happens to be this "build control" method:

This "build control" auto-generated method is responsible for building the control tree that we declared within the repeater's ItemTemplate. As you can see, it creates a literal control, then calls the TextBox's build control method to create it, and then creates another literal control (the literal controls represent the non-server-control markup before and after the TextBox).

But this method isn't actually called from anywhere. It's simply the target of a Delegate, which was passed to the CompiledTemplateBuilder. So we haven't figured out yet exactly how the controls get into the control tree.

That's because it's up to the control containing the template to do something with the template. The page parser has done it's job. Now it's up to the repeater to do something about it. And of course, we already know the repeater "uses" it once for each data item. To see exactly how it uses it, lets look at the ITemplate interface:

That's the entire interface. Just one method. The idea is that you have this control tree template, and when you need to instantiate it -- when you need to create an actual control tree based on the template (not unlike the relationship between a Class and an Instance of a Class) -- you call InstantiateIn(). The control tree is then created and added to the container you give it.

So we can summarize Repeater's use of templates as the following:

1. Go to the next data item.
2. Determine the appropriate item template for this data item (one of: ItemTemplate, AlternatingItemTemplate, HeaderTemplate, FooterTemplate).
3. Create a container control (repeater uses a RepeaterItem).
4. Call the selected template's InstantiateIn method, passing the container.
5. Add the container to the repeater's control collection.
6. Goto step 1 if there's any data items left.

In addition to this logic, the Repeater has an optional SeparatorTemplate, which it calls InstantiateIn() on between each data item.

Remember the "build control" method that the Delegate pointed to? Calling InstantiateIn() on the template is going to call that method. So the method is executed once for each data item, and thus that txtName TextBox we declared in the ItemTemplate markup is going to be created multiple times, in a dynamic manner.

So as you can see, Templates are powerful. And it also demonstrates that the framework itself is very much involved in the creation of dynamic controls. You can also create templates programmatically instead of statically by implementing the ITemplate interface yourself, but that's for a different blog entry.

All this and I still haven't gotten to the heart of the matter. The next part will examine how and when dynamic controls are added to the control tree. By "when" I mean when during the page lifecycle. That is so important because "when" can affect how the dynamic control participates in the lifecycle, and depending on the type of control and what features of it you are relying on, doing it at the wrong time will break it...

Part 1: Dynamic vs. Static
Part 2: Creating Dynamic Controls
Part 3: Adding Dynamic Controls to the Control Tree
Part 4: Because you don't know to render at design time

Just like ViewState, dynamic controls seem to be fodder for much debate, and a source of many confusing issues. This article will be the first of a multi-part series to detail just about everything you could ever want to know about how Dynamic Controls fit in with the framework.

ASP.NET has been out for half a decade. Maybe this article is a little late in the making. But with all of the new technologies coming down the pipe (e.g., Atlas), I thought it would be nice to get back to the basics. After all, not all of us have been in the game since the beginning :)

Actually, this article should probably be titled "TRULY Understanding ASP.NET". Because as you will see, dynamic controls really aren't that special. In learning how they fit in with the page life cycle, and how they are similar to static controls, you will gain an understanding of how ASP.NET works in general.

I decided to write this article as a direct consequence of the article I wrote on ViewState. Over the course of a few months I received dozens upon dozens of comments on that article from developers needing help. They assumed the trouble they were having was a ViewState related issue. But 75% of the time, it was actually a dynamic control issue!

Part of the problem is that so many developers think they need to create controls dynamically when there's often much more elegant or easier solutions. There's a tendency for developers to try and do "too much" themselves, and that leads to complex problems that seem to have no good solution. It's frustrating and discouraging. I've been there myself -- but I've learned that, almost always, when I find myself in that situation it's because I'm failing to see a bigger picture or I'm failing to approach the problem in an "asp.net way". I hope this article helps those who are experiencing the same thing.

I also decided to break this article up into multiple episodes, because the ViewState article is quite long, and I think this article is going to be even longer. Maybe if I give it to you in more bite size pieces it will be easier to read, no?

PART 1

 

Dynamic vs. Static: What's the difference?

Normally you "declare" controls on a form via markup and the ubiquitous runat="server" attribute. Like this:

That TextBox is a "static" control. Static controls are declared in an xml-like syntax in ASPX or ASCX files. When ASP.NET parses that markup, it generates a class on the fly that does the dirty work for you -- one by one, the code creates the controls and adds them to the control tree. The TextBox declared above results in the following auto-generated code:

The code may look scary, but it's just a really careful way of creating a TextBox, setting the ID and Text properties, and returning it. It has to be careful because it's possible that user code define things that make change the intended purpose. The #line pragmas are so the compiler can alert you to any compile errors and still give you the correct line number in the declared markup, rather than the line number of the generated code (say thank you to the compiler next time that happens -- its a great feature).

There will be one of these "build control" methods for each control you statically declare on the page. If a control exists inside another control, then the parent control's "build control" method will add them to it's own control collection. In this example, the form control contains three child controls:

The child controls are:

1. A LiteralControl containing the straight-up text "Your name:"
2. The txtName TextBox
3. Another LiteralControl containing the whitespace before the end of the form tag.

Notice to create the TextBox control, this method calls that control's "build method" that we just looked at:

And notice this strange method it calls, passing the TextBox:

Because of the fact the "object" being added is a control, all that method does is call Controls.Add(@__ctrl1), thus adding the TextBox to the control tree (I say "because of the fact it is a control", because non-controls can be added as well, but how they are handled is unique to each control type).

This isn't shown in this example, but haven't you ever wondered why you declare control variables as protected? Why not private? The class that is auto-generated inherits from your code-behind (that's why you need an "inherits" attribute in the page directive). It's up to this code to create the declared controls and assign them to the variables you declare (if it exists) so that you can easily access them. Since it is derived from your code-behind class, if the variable were private, it couldn't do that!

So that is how the framework adds declared controls into the control tree. A dynamic control in the traditional sense is one that you as the page or control developer create at some point during the page lifecycle, and add to the static control tree yourself. Something like this:

This code is not nearly as scary looking as the auto-generated code from the page parser. However, it does the same thing!

So you see, this whole silly ASPX-markup-runat-server-thing exists solely to make it easier on you to create a control tree! That's all it's there for! If it didn't exist, you could still achieve all the same functionality by creating the controls and setting their properties yourself, just like the auto-generated code does! When I first realized this, it was a eureka moment for me. So if you don't feel the same way, either you don't understand what I'm saying, or I'm crazy when compared to you. Maybe it's a little of both :)

This really blurs the line between static and dynamic controls, as it should. What you think of as a static control, is really just a control that is being dynamically created by the framework instead of by you. But neither the control or the framework really care who creates it -- they can participate in the page lifecycle one and the same, and both the control and the page it sits on don't even know the difference.

But wait!

If you have ever dealt with dynamic controls before, you know that they really are different. It's true that your experience with them may differ from that of static controls, but it's important to understand exactly why there's a difference. The difference has to do with when the control enters the control tree. When the framework adds controls, it does it extremely early in the event sequence. The only thing that happens first is the control or page constructor. That's why in OnInit or OnPreInit, the controls already exist and are ready for use (well, master pages can 'mess' with that process, but this is still a true statement).

But when you dynamically create a control, you don't have the ability to do it as early as the framework does. And that has consequences...

Stay tuned. :)

Visual Studio is without a doubt a powerful tool. With every iteration, it continues to improve upon itself. But as you happily hack away at all your applications, you are blistfully unaware of it's evil dark side that has been there since the beginning. It's true.

There are those of us who embrace the dark side. But we are out numbered...

You see, the dark side isn't how it comes by default. No... it comes all happy and bright and cheery by default, and like good little jedi programmers you accept those defaults. But the dark side is there, hidden away deep within the environment settings, reaching out and corrupting those programmers who are corruptible. Why some are corruptible and some are not is a mystery that may never be solved, but each and every programmer must give pause and consider the benefits it provides.

I know what you may be thinking. It's hard to read. Are you so sure about that? Lets compare it with the happy cheery default color scheme:

The key difference of course is the pervasive black background color. This is the environment I and many others work in every day. For me, it started way back when I used a Borland C++ IDE that had preset environment color schemes. One was called Twilight, and it looked similar to this. Most people who see it for the first time are offended by it... but if you think about it, it really makes sense. It brings balance to the force.

The default scheme sports a bright white background color with dark text over it. But monitors these days are brighter than ever. You're presumably a programmer, so you've no doubt had those late but productive coding nights, nights that are lit by only the glow of your monitor. The glow is bright enough to light up the room and cast shadows. Not unlike... a light bulb.

So there you are, staring straight into a strong light source, looking for the few pixels on it which are not illuminated. Can you read the wattage and manufacturer letters on the head a light bulb while it's turned on? Ahhh... but what if the bulb were black, and only the letters on it were illuminated?

This bulb has no markings, but you'd bet they'd show up nice and bright and easy to read in the right image. Another benefit someone pointed out to me once -- if you're on a laptop, it saves your battery life! Horray for an extra 20 minutes of mobile coding!

It seems to me the only reason a black-on-white background is so standard is because the GUI was invented to be an analogy to pen and paper. Paper is white. Your screen doesn't have to be. Don't conform to the status quo! Plus, it just looks really cool... I think.

Want to join the dark side? Download the settings here..

Visual Studio 2005
Visual Studio 2008

Just import them via the Tools->Import and Export Settings menu. Don't worry. If it doesn't suit you, just revert :)

As for VS2003, to enable exporting/importing styles in VS2003 you need an add-in. There happens to be one called VSStyler, which you can download here:

VSStyler Add-In for VS2003

Once you have the add-in, download the dark side style export here:

The Dark Side for VS2003

UPDATE 03/28/2008: Added new settings for Visual Studio 2008. I've improved on the scheme a little, and it now uses the Consolas font, too.

Happy (dark) coding!

ViewState is a very misunderstood animal. I would like to help put an end to the madness by attempting to explain exactly how the ViewState mechanism works, from beginning to end, and from many different use cases, such as declared controls vs. dynamic controls.

There are a lot of great articles out there that try to dispel the myths about ViewState. You might say this is like beating a dead horse (where ViewState is the horse, and the internet is the assailant). But this horse isn't dead, let me tell you. No, he's very much alive and he's stampeding through your living room. We need to beat him down once again. Don't worry, no horses were harmed during the authoring of this article.

It's not that there's no good information out there about ViewState, it's just all of them seem to be lacking something, and that is contributing to the community's overall confusion about ViewState. For example, one of the key features that is important to understand about ViewState is how it tracks dirtiness. Yet, here is a very good, in-depth article on ViewState that doesn't even mention it! Then there's this W3Schools article on ViewState that seems to indicate that posted form values are maintained via ViewState, but that's not true. (Don't believe me? Disable ViewState on that textbox in their example and run it again). And it's the #1 Google Search Result for "ASP.NET ViewState". Here is ASP.NET Documentation on MSDN that describes how Controls maintain state across postbacks. The documentation isn't wrong per say, but it makes a statement that isn't entirely correct:

"If a control uses ViewState for property data instead of a private field, that property automatically will be persisted across round trips to the client."

That seems to imply that anything you shove into the ViewState StateBag will be round-tripped in the client's browser. NOT TRUE! So it's really no wonder there is so much confusion on ViewState. There is no where I've found on the internet that has a 100% complete and accurate explanation of how it works! The best article I have ever found is this one by Scott Mitchell. That one should be required reading. However, it does not explain the relationship of controls and their child controls when it comes to initialization and ViewState Tracking, and it is this point alone that causes a bulk of the mishandlings of ViewState, at least in the experiences I've had.

So the point of this article will be to first give a complete understanding of how ViewState basically functions, from beginning to end, hopefully filling in the holes that many other articles have. After a complete explanation of the entire ViewState process, I will go into some examples of how developers typically misuse ViewState, usually without even realizing it, and how to fix it. I should also preface this with the fact that I wrote this article with ASP.NET 1.x in mind. However, there are very few differences in the ViewState mechanism in ASP.NET 2.0. For one, ControlState is a new type of ViewState in ASP.NET 2.0, but it treated exactly like ViewState, so we can safely ignore it for the purposes of this article.

First let me explain why I think understanding ViewState to it's core is so important:

MISUNDERSTANDING OF VIEWSTATE WILL LEAD TO...
1. Leaking sensitive data
2. ViewState Attacks - aka the Jedi Mind Trick -- *waves hand* that plasma tv is for sale for $1.00
3. Poor performance - even to the point of NO PERFORMANCE
4. Poor scalability - how many users can you handle if each is posting 50k of data every request?
5. Overall poor design
6. Headache, nausea, dizziness, and irreversible frilling of the eyebrows.

If you develop an ASP.NET Application and you don't take ViewState seriously, this could happen to you:

The ViewState form field.	ViewState will add your web app's distinctiveness to it's own. Performance is futile.

I could go on but that is the gist of it. Now lets move on by starting back from the beginning:

WHAT DOES VIEWSTATE DO?
This is a list of ViewState's main jobs. Each of these jobs serves a very distinct purpose. Next we'll learn exactly how it fulfills those jobs.
1. Stores values per control by key name, like a Hashtable
2. Tracks changes to a ViewState value's initial state
3. Serializes and Deserializes saved data into a hidden form field on the client
4. Automatically restores ViewState data on postbacks

Even more important than understanding what it does, is understanding what it does NOT do:

WHAT DOESN'T VIEWSTATE DO?
1. Automatically retain state of class variables (private, protected, or public)
2. Remember any state information across page loads (only postbacks) (that is unless you customize how the data is persisted)
3. Remove the need to repopulate data on every request
4. ViewState is not responsible for the population of values that are posted such as by TextBox controls (although it does play an important role)
5. Make you coffee

While ViewState does have one overall purpose in the ASP.NET Framework, it's four main roles in the page lifecycle are quite distinct from each other. Logically, we can separate them and try to understand them individually. It is often the mishmash of information on ViewState that confuses people. Hopefully this breaks it down into more bite size nuggets. Mmmm... ViewState Nuggets.
1. VIEWSTATE STORES VALUES
If you've ever used a hashtable, then you've got it. There's no rocket science here. ViewState has an indexer on it that accepts a string as the key and any object as the value. For example:


Actually, "ViewState" is just a name. ViewState is a protected property defined on the System.Web.UI.Control class, from which all server controls, user controls, and pages, derive from. The type of the property is System.Web.UI.StateBag. Strictly speaking, the StateBag class has nothing to do with ASP.NET. It happens to be defined in the System.Web assembly, but other than it's dependency on the State Formatter, also defined in System.Web.UI, there's no reason why the StateBag class couldn't live along side ArrayList in the System.Collections namespace. In practice, Server Controls utilize ViewState as the backing store for most, if not all their properties. This is true of almost all Microsoft's built in controls (ie, label, textbox, button). This is important! You must understand this about controls you are using. Read that sentance again. I mean it... here it is a 3rd time: SERVER CONTROLS UTILIZE VIEWSTATE AS THE BACKING STORE FOR MOST, IF NOT ALL THEIR PROPERTIES. Depending on your background, when you think of a traditional property, you might imagine something like this:


What is important to know here is that this is NOT what most properties on ASP.NET controls look like. Instead, they use the ViewState StateBag, not a private instance variable, as their backing store:


And I can't stress it enough -- this is true of almost ALL PROPERTIES, even STYLES (actually, Styles do it by implementing IStateManager, but essentially they do it the same way). When writing your own controls it would usually be a good idea to follow this pattern, but thought should first be put into what should and shouldn't be allowed to be dynamically changed on postbacks. But I digress -- that's a different subject. It is also important to understand how DEFAULT VALUES are implemented using this technique. When you think of a property that has a default value, in the traditional sense, you might imagine something like the following:


The default value is the default because it is what is returned by the property if no one ever sets it. How can we accomplish this when ViewState is being used as the private backing? Like this:


Like a hashtable, the StateBag will return null as the value behind a key if it simply doesn't contain an entry with that key. So if the value is null, it has not been set, so return the default value, otherwise return whatever the value is. For you die-hards out there -- you may have detected a difference in these two implementations. In the case of ViewState backing, setting the property to NULL will result in resetting the property back to it's default value. With a "regular" property, setting it to null means it will simply be null. Well, that is just one reason why ASP.NET always tends to use String.Empty ("") instead of null. It's also not very important to the built in controls because basically all of their properties that can be null already are null by default. All I can say is keep this in mind if you write your own controls. And finally, as a footnote really, while this property-backing usage of the ViewState StateBag is how the StateBag is typically used, it isn't limited to just that. As a control or page, you can access you're own ViewState StateBag at any time for any reason, not just in a property. It is sometimes useful to do so in order to remember certain pieces of data across postbacks, but that too is another subject.

2. VIEWSTATE TRACKS CHANGES
Have you ever set a property on a control and then somehow felt... dirty? I sure have. In fact, after a twelve-hour day of setting properties in the office, I become so filthy my wife refuses to kiss me unless I'm holding flowers to mask the stench. I swear! Ok so setting properties doesn't really make you dirty. But it does make the entry in the StateBag dirty! The StateBag isn't just a dumb collection of keys and values like a Hashtable (please don't tell Hashtable I said that, he's scarey). In addition to storing values by key name, the StateBag has a TRACKING ability. Tracking is either on, or off. Tracking can be turned on by calling TrackViewState(), but once on, it cannot be turned off. When tracking is ON, and ONLY when tracking is ON, any changes to any of the StateBag's values will cause that item to be marked as "Dirty". StateBag even has a method you can use to detect if an item is dirty, aptly named IsItemDirty(string key). You can also manually cause an item to be considered dirty by calling SetItemDirty(string key). To illustrate, lets assume we have a StateBag that is not currently tracking:


Basically, tracking allows the StateBag to keep track of which of it's values have been changed since TrackViewState() has been called. Values that are assigned before tracking is enabled are not tracked (StateBag turns a blind eye). It is important to know that any assignment will mark the item as dirty -- even if the value given matches the value it already has!


ViewState could have been written to compare the new and old values before deciding if the item should be dirty. But recall that ViewState allows any object to be the value, so you aren't talking about a simple string comparison, and the object doesn't have to implement IComparable so you're not talking about a simple CompareTo either. Alas, because serialization and deserialization will be occuring, an instance you put into ViewState won't be the same instance any longer after a postback. That kind of comparison is not important for ViewState to do it's job, so it doesn't. So that's tracking in a nutshell.

But you might wonder why StateBag would need this ability in the first place. Why on earth would anyone need to know only changes since TrackViewState() is called? Why wouldn't they just utilize the entire collection of items? This one point seems to be at the core of all the confusion on ViewState. I have interviewed many professionals, sometimes with years and years of ASP.NET experience logged in their resumes, who have failed miserably to prove to me that they understand this point. Actually, I have never interviewed a single candidate who has! First, to truly understand why Tracking is needed, you will need to understand a little bit about how ASP.NET sets up declarative controls. Declarative controls are controls that are defined in your ASPX or ASCX form. Here:


I do declare that this label is declared on your form. The next thing we need to make sure you understand is ASP.NET's ability to wire up declared attributes to control properties. When ASP.NET parses the form, and finds a tag with runat=server, it creates an instance of the specified control. The variable name it assigns the instance to is based on the ID you assigned it (by the way, many don't realize that you don't have to give a control an ID at all, ASP.NET will use an automatically generated ID. Not specifying an ID has advantages, but that is a different subject). But that's not all it does. The control's tag may contain a bunch of attributes on it. In our label example up above, we have a "Text" attribute, and it's value is "Hello World". Using reflection, ASP.NET is able to detect whether the control has property by that name, and if so, sets its value to the declared value. Obviously the attribute is declared as a string (hey, its stored in a text file after all), so if the property it maps to isn't of type string, it must figure out how to convert the given string into the correct type, before calling the property setter. How it does that my friend is also an entirely different topic (it involves TypeConverters and static Parse methods). Suffice it to say it figures it out, and calls the property setter with the converted value.

Recall that all-important statement from the first role of the StateBag. Here it is again: Server Controls utilize ViewState as the backing store for most, if not all their properties. That means when you declare an attribute on a server control, that value is usually ultimately stored as an entry in that control's ViewState StateBag. Now recall how tracking works. Remember that if the StateBag is "tracking", then setting a value to it will mark that item as dirty. If it isn't tracking, it won't be marked dirty. So the question is -- when ASP.NET calls the SET on the PROPERTY that corresponds to the ATTRIBUTE that is DECLARED on the control, is the StateBag TRACKING or isn't it? The answer is no it is not tracking, because tracking doesn't begin until someone calls TrackViewState() on the StateBag, and ASP.NET does that during the OnInit phase of the page/control lifecycle. This little trick ASP.NET uses to populate properties allows it to easily detect the difference between a declaratively set value and dynamically set value. If you don't yet realize why that is important, please keep reading.

3. SERIALIZATION AND DESERIALIZATION
Aside from how ASP.NET creates declarative controls, the first two capabilities of ViewState we've discussed so far have been strictly related to the StateBag class (how it's similar to a hashtable, and how it tracks dirty values). Here is where things get bigger. Now we will have to start talking about how ASP.NET uses the ViewState StateBag's features to make the (black) magic of ViewState happen.

If you've ever done a "View Source" on an ASP.NET page, you've no doubt encountered the serialization of ViewState. You probably already knew that ViewState is stored in a hidden form field aptly named _ViewState as a base64 encoded string, because when anyone explains how ViewState works, that's usually the first thing they mention.

A brief aside -- before we understand how ASP.NET comes up with this single encoded string, we must understand the hierarchy of controls on the page. Many developers with years of experience still don't realize that a page consists of a tree of controls, because all they work on are ASPX pages, and all they need to worry about are controls that are directly declared on those pages... but controls can contain child controls, which can contain their own child controls, etc. This forms a tree of controls, where the ASPX page itself is the root of that tree. The 2nd level is all the controls declared at the top level in the ASPX page (usually that consists of just 3 controls -- a literal control to represent the content before the form tag, a HtmlForm control to represent the form and all its child controls, and another literal control to represent all the content after the close form tag). On the 3rd level are all the controls contained within those controls (ie, controls that are declared within the form tag), and so on and so forth. Each one of the controls in the tree has it's very own ViewState -- it's very own instance of a StateBag. There's a protected method defined on the System.Web.UI.Control class called SaveViewState. It returns type 'object'. The implementation for Control.SaveViewState is to simply pass the call along to the Control's StateBag (it too has a SaveViewState() method). By calling this method recursively on every control in the control tree, ASP.NET is able to build another tree that is structured not unlike the control tree itself, except instead of a tree of controls, it is a tree of data.

The data at this point is not yet converted into the string you see in the hidden form field, it's just an object tree of the data to be saved. Here is where it finally comes together... are you ready? When the StateBag is asked to save and return it's state (StateBag.SaveViewState()), it only does so for the items contained within it that are marked as Dirty. That is why StateBag has the tracking feature. That is the only reason why it has it. And oh what a good reason it is -- StateBag could just process every single item stored within it, but why should data that has not been changed from it's natural, declarative state be persisted? There's no reason for it to be -- it will be restored on the next request when ASP.NET reparses the page anyway (actually it only parses it once, building a compiled class that does the work from then on). Despite this smart optimization employed by ASP.NET, unnecessary data is still persisted into ViewState all the time due to misuse. I will get into examples that demonstrate these types of mistakes later on.

POP QUIZ
If you've read this far, congratulations, I am rewarding you with a pop quiz. Aren't I nice? Here it is: Let's say you have two nearly identical ASPX forms: Page1.aspx and Page2.aspx. Contained within each page is just a form tag and a label, like so:


They are identical except for one minor difference. In Page1.aspx, we shall declare the label's text to be "abc":


...And on Page2.aspx, we shall declare the label's text to be something much longer (the preamble to the Constitution of the United States of America):



Imagine you browse to Page1.aspx, you will see "abc" and nothing more. Then you use your browser to view the HTML source of the page. You will see the infamous hidden _ViewState hidden field with encoded data in it. Note the size of that string. Now you browse to Page2.aspx, and you see the preamble. You use your browser to view the HTML source once again, and you note the size of the encoded _ViewState field. The question is: Are the two sizes you noted the same, or are they different? Before we get to the answer, lets make it a little bit more involved. Lets say you also put a button next to the label (on each page):



There is no code in the click event handler for this button, so clicking on it doesn't do anything except make the page flicker. With this new button in place, you repeat the experiment, except this time when browsing to each page, you click the Postback button before looking at the HTML source. So the question is once again... Are the encoded ViewState values the same, or different? The correct answer to the first part the question is THEY ARE THE SAME! They are the same because in neither of the two ViewState strings are any data related to the label at all. If you understand ViewState, this is obvious. The Text property of the label is set to the declared value before it's ViewState is being tracked. That means if you were to check the dirty flag of the Text item in the StateBag, it would not be marked dirty. StateBag ignores items that aren't dirty when SaveViewState() is called, and it is the object it returns that is serialized into the hidden _ViewState field. Therefore, the text property is not serialized. Since the Text is not serialized in either case, and the forms are identical in every other way, the sizes of the encoded ViewStates on each page must be the same. In fact, no matter how large or small of a string you stuff into that text attribute, the size will remain the same.

The correct answer to the second part is again, THEY ARE THE SAME! In order for data to be serialized, it must be marked as dirty. In order to be marked as dirty, it's value must be set after TrackViewState() is called. But even when we perform a postback, ASP.NET recreates and populates the server controls in the same way. The Text property is still set to it's declared value just like it was in the first request. No other code is setting the text property, so there's no way the StateBag item could become dirty, even on a postback. Therefore, the sizes of the encoded ViewStates on each page after a postback must be the same.

So now we understand how ASP.NET determines what data needs to be serialized. But we don't know how it is serialized. That topic is outside the scope of this article (are you missing an assembly reference?), so if you're really interested in how it works, read up on the LosFormatter for ASP.NET 1.x or the ObjectStateFormatter for ASP.NET 2.0.

Finally on this topic is DESERIALIZATION. Obviously all this fancy dirty tracking and serialization wouldn't do any good if we couldn't get the data back again. That too is outside the scope of this article, but suffice it to say the process is just the reverse. ASP.NET rebuilds the object tree it serialized by reading the posted _ViewState form value and deserializing it with the LosFormatter (v1.x) or the ObjectStateFormatter (v2.0).

4. AUTOMATICALLY RESTORES DATA
This is last on our list of ViewState features. It is tempting to tie this feature in with Deserialization above, but it is not really part of that process. ASP.NET deserializes the ViewState data, and THEN it repopulates the controls with that data. Many articles out there confuse these two processes.

Defined on System.Web.UI.Control (again, the class that every control, user control, and page derive from) is a LoadViewState() method which accepts an object as parameter. This is the opposite of the SaveViewState() method we already discussed, which returns an object. Like SaveViewState(), LoadViewState() simple forwards the call on to it's StateBag object, calling LoadViewState on it. The StateBag then simply repopulates it's key/object collection with the data in the object. In case you are wondering, the object it is given is a System.Web.UI.Pair class, which is just a simple type with a First and Second field on it. The "First" field is an ArrayList of key names, and the "Second" field is an ArrayList of values. So StateBag just iterates over the lists, calling this.Add(key, value) for each item. The important thing to realize here is that the data it was given via LoadViewState() are only items that where marked dirty on the previous request. Prior to loading the ViewState items, the StateBag may already have values in it. Those values may be from declarative properties like we discussed, but they may also be values that were explicitly set by the developer prior to the LoadViewState call. If one of the items passed into LoadViewState already exists in the StateBag for some reason, it will be overwritten.

That right there is the magic of automatic state management. When the page first begins to load during a postback (even prior to initialization), all the properties are set to their declared natural defaults. Then OnInit occurs. During the OnInit phase, ASP.NET calls TrackViewState() on all the StateBags. Then LoadViewState() is called with the deserialized data that was dirty from the previous request. The StateBag calls Add(key, value) for each of those items. Since the StateBag is tracking at this point, the value is marked dirty, so that it may be persisted once again for the next postback. Brilliant! Whew. Now you are an expert on ViewState management.

IMPROPER USE OF VIEWSTATE
Now that we know exactly how ViewState works, we can finally begin to understand the problems that arise when it is used improperly. In this section I will describe cases that illustrate how a lot of ASP.NET developers misuse ViewState. But these aren't just obvious mistakes. Some of these will illustrate nuances about ViewState that will give you an even deeper understanding of how it all fits together.

CASES OF MISUSE
1. Forcing a Default
2. Persisting static data
3. Persisting cheap data
4. Initializing child controls programmatically
5. Initializing dynamically created controls programmatically

1. Forcing a Default
This is one of the most common misuses, and it is also the easiest to fix. The fixed code is also usually more compact than the wrong code. Yes, doing things the right way can lead to less code. Imagine that. This usually occurs when a control developer wants a particular property to have a particular default value, and does not understand the dirty tracking mechanism, or doesn't care. For example, lets say the Text property a control is supposed to be some value that comes from a Session variable. Developer Joe writes the following code:


This developer has committed a ViewState crime, call the ViewState police! There's two big problems with this approach. First of all, since Joe is developing a control, and he has taken the time to create a public Text property, it stands to reason that Joe may want developers that use his control to be able to set the Text property to something else. Jane is a page developer that is attempting to do just that, like so:


Jane is going to have a bad day. No matter what Jane puts into that Text attribute, Joe's control will refuse to listen to her. Poor Jane. She's using this control just like you use every other ASP.NET control, but this one works differently. Joe's control is overwriting Jane's Text value! Worse than that, since Joe sets it during the OnLoad phase, it is marked dirty in ViewState. So to add insult to injury, Jane is now incurring an increase in her page's serialized ViewState size for doing nothing more than putting Joe's Control on her page. I guess Joe doesn't like Jane very much. Maybe Joe's just trying to get back at Jane for something. Well, since we all know which sex rules this world, we can assume Jane ends up getting Joe to fix his control. Much to Jane's delight, this is what Joe comes up with:


Look at how much less code we have here. Joe doesn't even have to override OnLoad. Because the StateBag returns null if the given key does not exist, Joe can detect whether his Text property has been set already by checking for null. If it is, he can safely return his would-be default value. If it's not null, he happily returns whatever value it is. Simple as can be. Now when Jane uses Joe's control, not only is her Text attribute honored, she no longer incurs a hit on her ViewState size, either. Better behavior. Better performance. Less code. Everyone wins!

2. Persisting static data
By Static, I mean data that never changes or is not expected to change during the lifetime of a page, or even during the users session. Lets say Joe, our would-be shoddy asp.net developer, has been tasked with adding the current user's name to the top of a page in the company's eCommerce application. It's a nice way of telling the user, "hey, we know who you are!" It makes them feel special and that the site is working. Positive feedback. Lets say this eCommerce application has a business layer API that allows Joe to easily get the name of the currently authenticated user: CurrentUser.Name. Joe completes his task:




Sure enough, the current user's name will show up. Piece of cake, Joe thinks. Of course we know Joe has committed another sin. The label control he is using is tracking it's ViewState when it is assigned the current user's name. That means not only will the Label render the user name, but that user name will be encoded into the ViewState hidden form field. Why make ASP.NET go through all the work of serializing and deserializing the user name, when you are just going to reassign it after all? That's just rude! Even when confronted, Joe shrugs at the problem. It's only a few bytes! But it's a few bytes you can save so easily. There are two solutions to this problem. First... you could just disable ViewState on the label.


Problem solved. But there's an even better solution. Label has to be one of the most overused controls there are, bested only by the Panel control. It comes from the mindset of Visual Basic Programmers. To show text on a VB Form you needed a label. Labels are supposed to be the ASP.NET WebForm counterpart, so it's only nature to think you need a label to display a text value that isn't hardcoded in the webform. Fair enough, but that's not true. Label's render a <span> tag around their text content. You must ask yourself whether your really need this span tag at all. Unless you are applying a STYLE to this label, the answer is NO. This would suffice just fine:


Not only do you get to avoid having to declare a label on the form (which means less code, albeit designer-generated code), but you've followed the spirit of the code-behind model: separation of code from design! In fact, if Joe's company had a dedicated designer responsible for the look and feel of the eCommerce site, Joe could have simply passed on the task. "That's the designers job", he could have balked, and rightfully so. There is another reason why you might THINK you need a Label, and that is when something in the code behind may need to programmatically access or manipulate that label. Ok, fair enough. But you must still ask yourself whether you really need a SPAN tag surrounding the text. Introducing the most underused control in ASP.NET: The LITERAL!


No span tag here.

3. Persisting cheap data
This one is a superset of #2. Static data is definitely cheap to get. But not all cheap data is static. Sometimes we have data that may change during the lifetime of an application, possibly from moment to moment, but that data is virtually free to retrieve. By free, I mean the performance cost of looking it up is insignificant. A common instance of this mistake is when populating a dropdown list of U.S. States. Unless you are writing a web application that you plan on warping back in time to December 7, 1787 (here), the list of US States is not going to change any time soon. However, as a programmer that hates to type, you certainly wouldn't want to have to type these states by hand into your web form. And in the event a state does rebel (we can only dream... you know who you are), you wouldn't want to have to perform a code change to strike it from the list. Our proverbial programmer Joe decides he will populate his dropdown list from a USSTATES table in a database. The eCommerce site is already using a database, so its trivial for him to add the table and query it.



As is the nature of databound controls in ASP.NET, the state dropdown will be using ViewState to remember it's databound list of list items. At the time of this ranting, there are a total of 50 US States. Not only does the dropdown list contain a ListItem for each and every state, but each and every one of those states and their state codes are being serialized into the encoded ViewState. That's a lot of data to be stuffing down the pipe every time the page loads, especially over a dialup connection. I often wonder what it would be like if I explained to my grandmother the reason why her internet is so slow is because her computer is telling the server what all the US States are. I don't think she'd understand. She'd probably just start explaining how when she was young, there were only 46 states. Too bad... those extra 4 states are really wearing down your bandwidth. Damn you late comers! You know who you are!

Like the problem with static data, the general solution to this problem is to just disable ViewState on the control. Unfortunately, that is not always going to work. Whether it does depends on the nature of the control you are binding, and what features of it you are dependant on. In this example, if Joe simply added EnableViewState="false" to the dropdown, and removed the if(!this.IsPostback) condition, he would successfully remove the state data from ViewState, but he would immediately run into a troubling problem. The dropdown will no longer restore it's selected item on postbacks. WAIT!!! This is another source of confusion with ViewState. The reason the dropdown fails to remember it's selected item on postbacks is NOT because you have disabled ViewState on it. Postback controls such as dropdownlist and textbox restore their posted state (the selected item of a dropdown ist 'posted') even when ViewState is disabled, because even with ViewState disabled the control is still able to post its value. It forgets it's selected value because you are rebinding it in OnLoad, which is after the dropdown has already loaded it's posted value. When you databind it again, the first thing it does is throw that into the bit bucket (you know, digital trash). That means if a user selects California from the list, then click on a submit button, the dropdown will stubbornly return the default item (the first item if you don't specify it otherwise). Thankfully, there is an easy solution: Move the DataBind into OnInit:



The short explanation for why this works: You are populating the dropdown list with items before it attempts to load it's posted value. Now the dropdown will behave just like it did when Joe first designed it, only the rather large list of states will NOT be persisted into the ViewState hidden field! Brilliant! More importantly, this rule applies to any data that is cheap and easy to get to. You might argue that making a database query on every request is MORE costly than persisting the data through ViewState. In this case I believe you'd be wrong. Modern database systems (say, SQL Server) have sophisticated caching mechanism and are extremely efficient if configured correctly. The state list needs to be repopulated on every request no matter what you're doing. All you've done is change it from being pushed and pulled down a slow, unreliable 56kbps internet connection that may have to travel for thousands of miles, to being pulled over at worse a 10 megabit LAN connection a couple hundred feet between your internet server and database server. AND if you really wanted to improve things, you could cache the results of the database query in the application. You do the math!

4. Initializing child controls programmatically
Let's face it. You can't do everything declaratively. Sometimes you have to get logic involved. That's why we all have jobs, right? The trouble is ASP.NET does not provide an easy way to programmatically initialize properties of child controls correctly. You can override OnLoad and do it there -- but then you're persisting data that probably doesn't need to be persisted into ViewState. You can override OnInit and do it there instead, but that suffers from the same problem. Remember when we learned how ASP.NET calls TrackViewState() during the OnInit phase? It does this recursively on the entire control tree, but it does it from the BOTTOM of the tree UP! In other words, as a control or webform, the OnInit phase of your child controls occurs BEFORE your own. A control will begin tracking ViewState changes in this phase, which means by the time your own OnInit phase begins, your child controls ViewState are all already tracking! Lets say Joe would like to display the current date and time in a label declared on the form.



Even though Joe is setting the label text in the earliest event possible on his webform, it's already too late. The label is tracking ViewState changes, and the current date and time will inevitably be persisted into ViewState. This particular example could fall under the cheap data issue above. Joe could simply disable ViewState on the label to solve this problem. But here we are going to solve it a different way in order to illustrate an important concept. What would be nice is if Joe could declaratively set the label text to what he wants, something like:


You may have intuitively attempted this before. But ASP.NET will slap you in the face for it. The "<%= %>" syntax can not be used to assign values to properties of server-side controls. Joe could use the "<%# %>" syntax instead, but that isn't very different than the databinding method we've already covered (disabling ViewState and databinding it every request). The problem is we would like to be able to assign a value through code, but allow the control to continue to work in exactly the way it normally would. Perhaps some code is going to be manipulating this label, and we want any changes made to it to be persisted through ViewState like they normally would be. For example, maybe Joe wants to give the users a way to remove the date display from the form, replacing it with a blank date instead:


If the user clicks this button, the current date and time will vanish. But if we solved our original ViewState problem by disabling ViewState on the label, the date and time will magically reappear again on the next postback that occurs, because the label's ViewState being disabled means it will not automatically be restored. That's not good. What on Earth is Joe supposed to do now?

What we really want is to declaratively set a value that is based on logic, not static. If it were declared the label could continue to work like it normally does -- the initial state wouldn't be persisted since it is set before ViewState is tracking, and changes to it would be persisted in ViewState. Like I said... ASP.NET does not provide an easy way to accomplish this task. For you ASP.NET 2.0 developers out there, you do have the $ sign syntax, which allows you to use expression builders to declare values that actually come from a dynamic source (ie, resources, declared connection strings). There's no expression builder for "just run this code" so I don't think that helps you either (UPDATE: Unless you use my custom CodeExpressionBuilder!). Also for ASP.NET 2.0 developers, there's OnPreInit. That is actually a great place to initialize child control properties programmatically because it occurs before the child control's OnInit (and therefore before it is tracking ViewState) and after the controls are created. However, OnPreInit is not recursive like the other control phase methods are. That means it is only accessible on the PAGE itself. That doesn't help you what-so-ever if you are developing a CONTROL. It's too bad OnPreInit isn't recursive just like OnInit, OnLoad, and OnPreRender are, I don't see a reason for the inconsistency. The root of the problem is simply that we need to be able to assign the Text property of the label BEFORE it begins tracking its ViewState. We already know the page's OnInit event (the first event that occurs in the page) is already too late for that. So what if we could somehow hook into the Init event of the label? You can't add an event handler in code for that, because the soonest you can do it is in your OnInit which is after the source event has already occurred. And you can't do it in the constructor for the page, because declared controls are not yet created at that point. There are two possibilities:

1. Declaratively hook into the Init event:

This works because the OnInit attribute is processed before the label's own OnInit event occurs, giving us an opportunity to manipulate it before it beings tracking ViewState changes. Our event handler would set its text.

2. Create a custom control:

Then instead of a regular label on the form, a DateTimeLabel is used. Since the control is initializing it's own state, it can do so before tracking begins. It does it during the constructor if possible, so that a declared value will be honored.

5. Initializing dynamically created controls programmatically
This is the same problem as before, but since you are in more control of the situation, it is much easier to solve. Lets say Joe has written a custom control that at some point is dynamically creating a Label.


Hmmm. When do dynamically created controls begin tracking ViewState? You can create and add dynamically created controls to your controls collection at almost any time during the page lifecycle, but ASP.NET uses the OnInit phase to start ViewState tracking. Won't our dynamic label miss out on that event? No. The trick is, Controls.Add() isn't just a simple collection add request. It does much more. As soon as a dynamic control is added to the control collection of a control that is rooted in the page (if you follow its parent controls eventually you get to the page), ASP.NET plays "catch up" with the event sequence in that control and any controls it contains. So let's say you add a control dynamically in the OnPreRender event (although there plenty of reasons why you would not want to do that). At that point, your OnInit, LoadViewState, LoadPostBackData, and OnLoad events have transpired. The second the control enters your control collection, all of these events happen within the control.

That means my friends the dynamic control is tracking ViewState immediately after you add it. Besides your constructor, the earliest you can add dynamic controls is in OnInit, where child controls are already tracking ViewState. In Joe's control, he's adding them in the CreateChildControls() method, which ASP.NET calls whenever it needs to make sure child controls exist (when it is called can vary based on whether you are an INamingContainer, whether it is a postback, and whether anything else calls EnsureChildControl()). The latest this can happen is OnPreRender, but if it happens any time after or during OnInit, you will be dirtying ViewState again, Joe. The solution is simple but easy to miss:


Subtle. Instead of initializing the label's text after adding it to the control collection, Joe initializes it before it is added. This ensures without a doubt that the Label is not tracking ViewState when it is initialized. Actually you can use this trick to do more than just initialize simple properties. You can databind controls even before they are part of the control tree. Remember our US State dropdown list example? If we can create that dropdown list dynamically, we can solve that problem without even disabling its ViewState:

It works amazingly well. The dropdown list will behave as if the states are simply built-in list items. They are not persisted in ViewState, yet ViewState is still enabled on the control, meaning you can still take advantage of its ViewState dependant features like the OnSelectedIndexChanged event. You can even do this with DataGrids, although that depends on how you are using it (you will run into problems if you are using sorting, paging, or using the SelectedIndex feature).

BE VIEWSTATE FRIENDLY
Now that you have a complete understanding of how ViewState does it's magic, and how it interacts with the page lifecycle in asp.net, it should be easy to be ViewState Friendly! That is the key really... ViewState optimization is easy as pie when you understand what is going on, often times resulting in even less code than the non-friendly code. Have any suggestions, comments, error reports? Please leave a comment or send me an email!

UPDATE 02/19/2008: Take a look at TRULY Understanding ViewState, the Comment Index!