So I asked ScottGu about our particular virtualization strategy that we use at work here. I think I have extolled on the architecture before, but I will go over it again just in case.
We host about 5,500 automotive dealership's websites. Obviously those sites need to look different so we came up with the concepts of themes and variations. Themes represent a particular navigational style like leftnav, topnav, etc... Variations are just color changes of existing themes. We also generate several files for each site... things like theme headers with dealership names or OEM logo, forms created by the dealer (we abstracted forms into a data model and we allow the dealers to create their own forms, label them the way they want, add validation, etc...), and menucontrols. Everything is stored in our database and I do mean everything. Sites, urls, pages, controls, data for those controls, etc... One thing we attempted to do for performance reasons was generate these files and deploy them to a Sites folder. This sites folder contained around 5000 folders named after the site. Each of these folders contained several ascx files for the things I mentioned above like sectors (to encapsulate the logos in a header for example), forms ascx(forms are generated a site creation and placed in this folder to save performance from all of the complex logic that would have to occur at runtime) and menuitems (they are hierarchical so we figured that it would be easiest to generate these as ascx files instead of at runtime).
The problem(s) that we ran into was around number of assemblies loaded. If you had this application running on 1 server (we don't) at about 500 assemblies the memory for the w3wp.exe process spirals out of control and completely crashes. The first thing we figured out was that <compilation debug="true"/> == bad. This is really the same type of problem that we had, but because of the nature of compilation when this setting is true in the web.config we noticed it after about 2 months. Sites were being added to our architecture in this time and we didn't catch it in testing.
The second thing that we figured out was that for each directory codebehinds and aspx|ascx get compiled into different dlls. Duly noted. Although the nature of the actual compilation threw us for a loop. While all of the aspx|ascx files get compiled into 1 assembly, a seperate assembly is created for each codebehind. Even if you have debug="false in the compilation element. To me this makes absolutely no sense whatsoever. For example take a folder that has 5 ascx files and 5 codebehinds, you will actually end up with 6 assemblies in this folder. If you can compile all of the ascx files into 1 assembly, why can't you do that with the codebehinds also? We ended up working around this problem by wrapping all logic normally found in codebehinds into <script runat="server"> tags. This allowed us to effectively have 1 assembly per directory. We also did some load balancing at the network layer using F5 BigIP to distribute the load across our 28 web servers more effectively.
While there has been much talk about the new compilation model for 2.0, this really hasn't changed much at all from what Scott said. Scott said that this type of scenario was "not optimal." I don't necessarily disagree with him. He did say that the compilation changes for 2.0 were made more granular because of developers working on projects with large amounts of items in the project. He used the example of 5,000 aspx pages and 4,000 images across 36 directories with 250 items per folder. Doesn't that scenario seem less than optimal also? I mean, if you have 5,000 different pages in a website.... who would want to maintain that? At least in our architecture everything is generated. We don't have to maintain those files. I understand that your example was contrived, but if Scott Guthrie says you can do X, I guarantee people are going to try and do it. It seems like if you can tell us that our architecture is "less than optimal" you could have told the people with an especially large number of files the same thing.