Tales from the Evil Empire

I am not sure that it has to be an ideal world for the clean separation to work. Indeed, the concept needs refinement, but the core of it is good, methinks. Imagine something like this: an adapter pattern with decoratorish tendencies applied to rendering of a control.

This way, a control only represents the data and manipulation with it. It does not emit any markup, but it does know how to retrieve, process, and update information.

Parallel to it is an adapter, which doesn't know how to do any of the above with the data, but it does know how to query control's properties and how to render it using markup.

Yes, this means that for each control there has to be an adapter, but it is a compromise that we ought to be willing to make in order to escape hard-coding HTML into control's functionality.

We are essentially continuing the Page Controller paradigm into the controls, rather than stopping at the level of the page.

As a possible implementation scenario, an adapter could simply spit out control's properties as XML and allow developers write XSLT to transform it into HTML.

You could also easily imagine an adaptive control, which attempts to render properties of any control -- this is not necessarily a production environment-type trick, but something that could be useful to render orphan controls -- those without a corresponding adapter.

This is why my third role is for advanced developers -- creating adapters would not be something that a designer would do. Following requirements of a designer, yes, but probably not a designer him/herself.

I understand your points, but one of the places where the whole thing falls apart is performance. This is what I mean when I say it's an ideal world, something we should tend to, but that we should not place above everything else, despite having to solve the real-world problem in all its complexity.
Believe me, this approach where adapters are completely separated from the control has been considered, but unfortunately, adapters are expensive. As would be to have an additional XML/XSLT level.
The choice that was made is to make is as efficient as possible for the main scenario (which is HTML) and fall back to adapters only when we need to.
If you look at the present state of things, the rendering is already pretty much separated from the rest of the control: you output HTML only in Render.
Where I don't agree with you, even from an idealistic point of vue, is in your core analysis that controls should be some kind of local page controller only, leaving the rendering to some other entity. While I agree that there is a benefit in having ways to override this part of the control's behavior (as well as others), most controls are UI elements first, not data manipulating entities (exceptions are non-visual controls like DataSources and these do not render anything). The purpose of a control as I understand it is to transform some part of your data into markup AND to manage the interaction of the user back with this data (which is what XSLTs don't do).
So I think if your control does too much data manipulation, that's a sure sign that part of this code should be in a different layer and not in your control.
Don't forget also that a control's interaction with the user is dependant on the actual markup. It's a two-way thing.
So what would be left in a control if it doesn't do the rendering (and thus can't do some of the interaction either) and doesn't do stuff that belongs in other layers? Almost nothing, it's then just a structure. You've just added a useless abstraction, transferring all of the logic into the adapter.

Brant, can you elaborate? Millions of developers successfully build UIs with HTML and ASP.NET or some other server platform. How are they "failing at some point"?
I agree that XAML solves many of the problems we have to face with HTML, but HTML is very far from being the dead-end you're talking about.

Bertrand, I think your assertion that most controls are UI elements is correct and valid. There are a couple of "buts" though:

1. There is definitely a trend into making controls more than that -- look at the "login" control in Whidbey, for example. For all I can see, it's a little self-contained application.

2. Being a UI element doesn't mean that controls have to emit the actual markup. That is why I am advocating a layer of abstraction -- to provide capability to translate UI into whatever flavor of markup (even XAML, perhaps?) is required.

3. The mentioned above layer of abstraction has to work both ways -- in fact, it should be the purpose of translating results back and forth. The control doesn't need to worry about whether this was a POST request, a GET request, or an asynchronous WS callback of some sort. The control doesn't need to worry about how it would be rendered. All it has to do is to advertise to the rendering layer that it will take a one-line input of plain text titled "Login", a one-line masked input of plain text titled "Password", and a "Login" button. The control performs validation and reaction to the input. The rendering layer makes it look "pretty". I am simplifying, of course.

I realize that what I am proposing is veering off from the current concept of a Web control, but hey, that's what this communication thing is for -- thought generation :)

As far as performance is concerned, I understand your point. Balancing functionality and performance is a tough act. I do believe, however, that there are ways to minimize impact of an abstraction layer on overall application performance. You guys wrote a book on this stuff :)

Brant:

HTML is much more than RTF because of two things: Javascript and the posting mechanism. Yes, its semantics are those of a document markup language (it comes from SGML after all), because the people who conceived it couldn't foresee the importance the web would have today. But it has been improved and is today something you just can't avoid.

Of course, Windows Forms, XUL and XAML (and others) are more suited to interactive applications than HTML is. And if you have a choice, most applications should be written using a smart client technology. But you rarely have this option, even though the deployment problems are more or less solved. So there is a strong need for technologies like ASP.NET and there will still be for some more time. Of course you won't develop Excel in ASP.NET, but you can and you have to develop applications such as Outlook Web Access, which is very close to the desktop application. If only because when I'm away from my office, I still need a way to consult my mail from any computer I may find. And the only reasonable way you can develop this kind of rich application for the web is by adding a level of abstraction like what ASP.NET provides with WebControls.

I wish we all lived in a world where you don't need to develop for HTML but can always use something richer, but this is not the world we're living in. It's great to develop better rich client technologies. I see you contributed to Mozilla and XUL, this is great, but don't fall into this trap: you don't have a universal answer that applies to everything (by the way, the subtitle of my blog is highly sarcastic: ASP.NET is no universal answer).

By the way, which ASP.NET Tree control are you talking about? There are many such controls. The one we're shipping with Whidbey (and the only one we're supporting) DOES support Gecko browsers. It even does out-of-band node population without posting the whole page thanks to the excellent implementation of XmlHttp that Gecko has.

I'm a little confused also when you talk about "pixel perfect" layout. In my mind, pixel-perfect is more needed in printable document formats such as PDF, but certainly not in applications. Accessibility constraints, for example, are one reason why your UI framework should be able to gracefully adapt to very big fonts. There goes pixel-perfect layout.
Actually, all recent UI frameworks allow for dynamic resizing and positioning of the elements according to some general and relative layout.
And of course, HTML has always allowed for this by default, which makes it notoriously unprintable, but highly adaptable.

Dimitri:
I think I agree with you on the whole principle, and I think we're trying to get there, but we have to integrate our own constraints (which are huge because of our enormous developer audience). We make tradeoffs every day, we have to.
But if you read this recent article: http://www.theserverside.net/articles/showarticle.tss?id=AssemblyVersioning you can probably spot a few hints about what transformations there will be in the structure of ASP.NET in v.Next.

About the login example, yes, there is some functionality in the control, but it's still just managing the interaction of the user with the underlying authentication layer. I see what you mean, though, and it's an interesting idea.
I think we're already addressing most of this concern, though.
For example, templating is one way you can take the relevant part of the rendering to the page (or user control).
Adapters are also another way you can do that.
Themes are also very powerful (though not many people realised how much).

The main thing is that the additional layer is completely optional. The default basic scenario stays performant because it does not have these additional layers. We're trying to make the optional scenarios performant too, but there's no way they can ever be as fast as the default behavior. And believe me, we spend a lot of time on the optimization of this stuff (adapters were tough for that matter).

By the way, I love it when the comments become larger than the original message. It proves that the subject was well chosen :)

I'm a fan of Nikhil's blog and this one is a great post. Read it!

From Nikhil's post: "A key problem with adapters is exposing internal state of the control to the adapter. It is possible to expose some state, but the moment an external developer decides to write a new adapter, it is likely they will find they need some piece of internal state information that has not been exposed to them. The adapter model essentially creates a parallel inheritance hierarchy, which doesn't work all that great for extensibility in the long run."

I don't know if this will ever be read, but...

Bertrand, I don't think you, or anyone at Microsoft, for that matter, understands that HTML is just a structural markup, and does not - in fact, *MUST* not have ANYTHING to do with how the data is displayed.

In general, HTML must be written such that Netscape 1.0 (the one from 1994. or so) displays it properly, fully and completely, as well as IE6 (which should be dead) and Opera 7.x, Safari, something based on the Gecko engine, _plus_ Lynx and Links.

There are people who have disabled JavaScript. They can't do much on ASP.NET pages. There are also vision-impaired and blind people, and the story is the same because ASP.NET are mostly random tag soup, instead of a structured and valid HTML document...

The purpose of adapters should not be to fall-back functionally to older and different browsers, but to ENHANCE the *APPEARANCE* for newer ones.

There's no excuse AT ALL to do XMLHTTP requests to populate tree nodes in a tree control. None whatsoever! Someone who loads a page for offline use will get screwed by this, and will get screwed royally. The ONLY proper thing is to create a good old structure of nested unnumbered lists COMPLETELY and have JavaScript-enabled visual browsers take care of node expanding/collapsing.

Dear anonymous strict guy,

Thank you for assuming everyone at Microsoft is an idiot. What you say is more or less true of HTML + CSS, which not all browsers understand, but absolutely false for pure HTML. Take the <b> tag, for example. Why is it in every version of HTML including XHTML? It's not structural, it just means bold text, which is pure form. And it's not as if you couldn't do a span with a class that has bold font-weight. So why is it still there? Just because there are the extremists who think everyone should work the way they have decided is the right way, and there are the pragmatic people who think some freedom is better than any dogma, however "perfect" it may be.

It is absolutely correct to assume that a web site should be browsable with Lynx and when Javascript is disabled, as well as by people with disabilities. You can build such pages with ASP.NET 1.1 (which does not mean that everyone does it, most people don't bother). Now, with ASP.NET 2.0, out of the box, you'll get XHTML compliant code and accessibility by default.
And when you build a web site, having ways to achieve everything without javascript does not mean you have to force this on everyone and not have quicker ways to achieve the same things when javascript is enabled.

The purpose of adapters is to adapt the markup for _marginal_ situations. The default behavior should go to the most common situations. This is so obvious I'm appalled by your statement.

Now about XMLHTTP. First, you seem to forget that the web is not just a collection of documents. There are applications out there, for which offline use is simply meaningless. By the way, offline browsing is a very very marginal thing and will become even more so.
Now, take a site like MSDN. Hundreds of thousands of articles. Would you really like the treeview to preload all the nodes so that you can browse it offline or with a text browser? This is completely unrealistic as your page would take forever to download. The right approach is of course to adapt to a link list that can take you to the next level and that enables you to navigate back up one level (à la Mac Finder as opposed to the Windows explorer tree).
And there are of course all kinds of applications that need treeviews that are filled as nodes are open. Posting back the whole page is a waste of bandwidth, that's why the XMLHTTP mechanism is there. By the way, this is not the default behavior of our TreeView.
Actually, this is quite funny as this is one of the features that gets the most positive feedback from the community.

You know, I also get feedback all the time from people who have decided there is one and only one way to write a web page. They will tell you for example that menus should only be written using CSS and no javascript. CSS menus are great technical achievements, but they can't be keyboard accessible, for example. Ours is.
Accessibility is an absolute requirement for ASP.NET 2.0 and you can rest assured that all of our controls will be accessible by default, including TreeView and Menu.