Tales from the Evil Empire

Bertrand Le Roy's blog

News


Bertrand Le Roy

BoudinFatal's Gamercard

Tales from the Evil Empire - Blogged

Blogs I read

My other stuff

Archives

How to build a cross-browser history management system

When we built the history management feature in ASP.NET Futures, we spent considerable time experimenting with the different behaviors of the main browsers out there. The problem with such a feature is that it has to rely on a number of hacks because browser vendors basically never anticipated this need. Now they're thinking about it, so all this may be simplified in a few years, but in the meantime, it's a very complicated feature to build. One of the things that struck me was how little reliable literature is available on the subject. There is a lot of partial information, lots of false or unverified information, but very little that's really comprehensive, reliable and up to date. Good references I found include Brad Neuberg's Really Simple History and Handling Bookmarks and Back Button as well as Mike Stenhouse's Fixing the Back Button and Enabling Bookmarking for Ajax Applications. But it was a lot easier to just experiment directly on the different browsers and verify our theories directly.

This blog post will attempt to be an account of the difficulties we encountered and how we worked around them. It's also a brain dump that will help me and others maintain the feature in the long term. I hope it will evolve over time as feedback comes in, browser bugs get fixed and we ourselves learn more.

Note: all code in the pages attached to this post is pure JavaScript and XHTML and has no dependency to any script library.

How history used to work

It used to be the case that history in the browser just worked: the user navigated from page to page following links and the history stack just got filled, the back button worked as usual and life was good. With dynamic pages that post forms, things got a little less ideal as every post action would make a new entry, no matter if or how little the state of the page changed. Plus, you got this nasty "do you really want to repost this form" dialog that most users had no clue what to do with.

But Ajax applications broke the model in a more fundamental way because most actions are done without navigating away from the page or posting forms. This means that the browser basically has no way of knowing the state of the application changed in a meaningful way and nothing new gets in the history stack. Any naive user who doesn't know the implementation details of your application will expect the back button to work like it has for decades, which is catastrophic because by navigating back to the previous page it knows about, the browser will lose all the user's precious work.

Thanks to a few hacks such as the ones I will expose here, it doesn't have to be that way, and the control of the history stack can be given back to the application developer. We can even make the model better than it used to be and choose exactly what constitutes a meaningful enough state change to warrant an entry in history.

Navigating without moving

The main trick that history managers use is to have the browser believe the user navigated to a new url without the current page and all its JavaScript and DOM state being thrown away. The only part of the url that enables such a thing is the hash part. The hash part is what comes at the end of the url after a pound (#) sign. The original intent of this part of the url was to allow for navigation inside of the document. You would put a special named, href-less anchor tag in your document, and then navigating to #nameOfTheAnchor would just scroll the anchor into view. The page doesn't get reloaded, but it does enter the browser history (almost, we'll see about that in a minute). This is great for long pages with a table of contents for example.

But it's also great in that it's a way for us to manipulate the history stack without unloading the page, which is exactly what we need. So this is one more example of using something for a totally different use than what it's been designed for, but what choice do we have? And it's actually all right as this new usage doesn't conflict with the old one.

So it seems like we have an easy way to get this to work: whenever you want to create a history point, just navigate the page to #SerializedFormOfTheStateOfTheApplication; detect url changes somehow (polling works (almost) everywhere) and re-hydrate the application state from that. That should work, right? Yes, it should, but it doesn't, except on Firefox and Safari 3 for the Mac, for various reasons.

I've prepared a page that implements exactly that logic that you can try here:
HistoryHash.htm.txt (download and rename HistoryHash.htm to run on your system)

Here's a summary of the results on some popular browsers:

 

IE 7

Firefox 2

Opera 9

Safari 2 Mac

Safari 3 Windows beta

Safari 3 Mac beta
URL changes Yes Yes Yes Yes Yes Yes
Creates history entry No Yes Yes Yes Not on first page Yes
Back doesn't kill timers Yes Yes Not in 9.23 and 9.24
Yes in 9.10
Yes Yes Yes
Back enables forward Yes Yes Yes Yes Sometimes Yes
location.hash reflects changes in the url Yes Yes Yes No Yes Yes
Making it work in IE

The problem in IE comes from the fact that it doesn't create a history entry when the hash changes. This can be worked around by navigating an iframe in addition to changing the hash as IE does create a new history point every time a frame is navigated. The problem with that technique is that just like the main page, the iframe can't be just navigated to a new hash, it needs to move to a different page, which means a round-trip to the server on every new state, which is quite wasteful. I've heard of techniques to create the iframe's contents through clever scripting in the iframe url and document.writing, but I've never been able to consistently reproduce it.

<update date="Sep. 14 2007">
One of the comments over at Ajaxian pointed me to this great article from Julien Lecomte, where he explains how they built the feature over at YahooUI. It's a super-interesting read and thanks to it, the document.writing trick suddenly clicked together for me. The trick is that you need to open the document before you write to it and close it when you're done if you want the history point to be created by IE. Somehow I had missed that but today I tried again and had no difficulty making it work. The only thing that may be a little tricky is that you need to pass a url into document.open or the browser will default to about:blank. That may look ok until you try it under https, in which case you're going to get the infamous dialog about mixed contents. This can be solved by using the usual magical url javascript:'<html></html>' (thanks to the toolkit team for that trick). Please note that now we don't even have a single request for the frame, even on the first hit, again thanks to the magical url. It's also important to note that the iframe must absolutely be in the static markup of the page (you can't create it dynamically) and on the first request it must point to an existing page on the server.
I've updated the sample page for IE below to include this trick and I'll also integrate it into ASP.NET Ajax.
</update>

The frame gets the state passed to it through the search part of the url (the part after a question mark). It then calls back into a function in the main document when it loads. When the user navigates back to a previous state, it's the iframe that really navigates back, not the main page. When it does, its load event fires, which calls back into the main page again. From that callback function, we set the hash so that the browser url reflects the state for bookmarkability, and of course we update the application with that state.

Here's a simplified implementation of this. I didn't hide the iframe to make clearer what is happening here but that's of course what you'd do in production code.
HistoryIE.htm.txt (download and rename HistoryIE.htm to run on your system)
You will also need the iframe file:
HistoryFrame.htm.txt (download and rename HistoryFrame.htm to run on your system)

<update date="Mar. 31 2008">
Internet Explorer 8 finally gets rid of the need for the iframe navigation: changing the url fragment now results in a new entry being created in browser history. We were able to make our implementation of history work in IE8 by just restricting the iframe code to versions earlier than 8. Internet Explorer now follows exactly the same code path as Firefox, Safari 3 and Opera 9.5. Not all is perfect though as IE still has this weird quirk where when navigating away from the history-managed page causes its history entries to collapse to just one entry. If you do go back, it will expand again but this is really confusing to the users.
On the bright side, IE is the only browser as I write that implements an HTML5 event that gets triggered when the url fragment changes, eliminating the need to poll. I really hope the other browser follow that lead...
</update>

Making it work in Safari 2

While the workaround for IE results in a completely different implementation, at least it makes some sense and is consistent. With Safari, we're lost in bugland. In Safari 2, monitoring the hash from a timer doesn't work as location.hash is not getting updated as the url in the browser's address bar changes. This is a plain bug that is fixed in Safari 3 and the current WebKit builds. The only information we have that gets correctly updated when the user navigates through history is history.length. In other words, we know the index of the history point but we don't know the corresponding state. One thing we can do to work around this is to maintain our own array of states and use the index to retrieve the right state from this array. One caveat is that if we use a JavaScript variable to store this array, navigating to a different page will completely wipe out the state array and kill the history feature. This can be partially worked around by storing the information into a hidden form field instead of a JavaScript variable: form field values are restored by the browser when navigating through history. This works relatively well to maintain history even if the user navigates away to a different page and comes back but what it doesn't handle on the other hand is the user pressing F5.

Here's an implementation of those workarounds (I've left the history stack field visible to make it easier to understand what happens but of course in a real application it would be idden):
HistorySafari2.htm.txt (download and rename HistorySafari2.htm to run on your system)

This must be fixed in Safari 3, right?

Well, yes, Safari 3 Mac does the right thing as the unmodified hash code that works on Firefox just works. But on Windows currently other bugs cripple the feature. In the current nightly build and public beta of Safari 3 for Windows, if your page is the first that you load in the browser, no history entries get created when the hash is modified. Another weird bug that I couldn't quite find reliable repro steps for is that under some circumstances, hitting back does not enable forward, making time travel only possible to the past, without any hope of return. Somebody give them a flux capacitor, an old clock and a thunderstorm... Because of this, the Safari 3 for Windows implementation kind of works sometimes but it really needs to be fixed. I filed bug 14080 in WebKit a while ago to track this. Seriously, I don't blame them, it's a beta, betas have bugs and I'm pretty confident this will get fixed pretty soon. I've had pretty good experience with their reactivity in the past.

What about Opera then?

Well, it used to work perfectly well in Opera up until version 9.10. Then they broke it. In 9.23, and also in 9.24 which is the current version as I'm writing this (not a beta), for some reason Opera cancels all timers when hitting back. This completely breaks any history manager because there is now no way of knowing the user navigated. This problem has been reported by several persons on the Opera forums and apparently there's a bug open for it, but the closedness of Opera's bug database doesn't allow us to monitor the status of the issue.

<update date="Sep. 17 2007">
Neil Jenkins pointed me to a nice trick he's come up with to work around this bug. The idea is to have a hidden image with the following src attribute: javascript:location.href='javascript:onTick();';
Amazingly, this works and the code gets executed when the user hits back. Just doing javascript:onTick(); directly doesn't work for some reason that I don't quite understand. Apparently, Opera tries to be smart about what can be run as an img src, but it's easily worked around. If you ask me, script should simply not be allowed as image urls (as well as in a number of places) but I suppose that would break too many hacksapplications.
Anyway, the workaround does work, and it's simple enough that you should use it if you need to support Opera 9.23 and 9.24. But you also need to know that this is fixed in Opera 9.5 (still in beta as I'm writing this) so it may not be worth fixing in the long run. Kudos to Opera for their reactivity on this issue.
HistoryOpera.htm.txt (download and rename HistoryOpera.htm to run on your system).
</update>

The state of things

So things are in a pretty grim state currently. It seems like we're going back (pun intended). We used to have a collection of tricks that made possible an implementation of a history manager that worked pretty well in IE, Firefox, Opera and Safari. Now, we only have IE, Firefox and Safari Mac. I just hope this is only temporary and that both Apple and Opera repair their browsers soon.

<update date="Sep. 17 2007">
As explained in the previous section, there's now a known workaround for the Opera bug, and the bug itself will be fixed in 9.5.
</update>

<update date="Feb. 18 2008">
The WebKit bug seems to be fixed.
</update>

<update date="Apr. 18 2008">
Apparently I forgot to mention one annoying bug in Firefox that url-decodes the hash automatically, which gets in the way of using the hash as a querystring-like state bag. More details in the bug comments here: https://bugzilla.mozilla.org/show_bug.cgi?id=378962#c4.
</update>

<update date="Dec. 22 2008">
Some great additional information on ways to help refresh behave better on IE can be found here: http://www.overset.com/2008/12/21/internet-explorer-iframe-browser-history-lost-on-reload/
</update>

What's next?

In the next posts, I'll get into more details about the initial request, maintaining meaningful titles and mixing all this with asynchronous operations, which is where it really gets tricky.

Comments

Sean said:

Please consider implementing the following:

Most ajax History solutions run into problems if the user hits refresh (all history entries are lost).  

Another major headache is that the programmer has to write a lot of code to reproduce the page state of each page.

A way to fix all these problems (including ones mentioned in your article) is to have your Ajax History Control create a virtual cache of each Ajax page on the server during postback.  

This accomplish many things.  First - the programmer would not have to write code to reproduce page state when the Back Button is hit.  Second - if the user hits Refresh - and you cached history entries - you simply have the client browser rebuild the history entries.

Just a thought.

# September 7, 2007 6:38 PM

Flash said:

There's already been several history tools developed for Adobe Flash which has historically had exactly the same problems as AJAX.

# September 10, 2007 5:36 AM

Steve said:

Wow, looks like quite a hack.

The trouble with these approaches are that when browser updates come out it could break an entire application built with this history.

Seems there could be a better solution.  I'd look at it from an IIS perspective not a client browser perspective.

Utilize what is least likely to change.

# September 11, 2007 9:01 AM

Bertrand Le Roy said:

Steve: you're absolutely right those techniques are hacky and are commonly broken by new browsers (Opera 9.23 and Safari for Windows proved it recently as the article points out. On the other hand, vendors are now aware of how Ajax applications rely on url hashes to manage history and are less likely to break them in the future. Also, Ajax in general relies on a lot of browser hacks and it's perfectly ok as a developer or an organization to say that it's not suitable or too risky for a particular project.

But if you can afford to take that risk, here's how it's done.

I'm not sure what you mean by "look at it from an IIS perspective".

# September 11, 2007 1:33 PM

Steve said:

I didn't mean to be condescending - thanks for the follow up response.  It is an interesting topic.

# September 11, 2007 7:29 PM

Bertrand Le Roy said:

Steve: I didn't think you were, don't worry.

# September 11, 2007 7:34 PM

Bertrand Le Roy said:

Julien, thanks for the comment. You are absolutely right, on the first request, the frame needs to point to an existing file on the server, the magical url doesn't quite work at this point (it does later). Just to be clear, our history manager *does* work on Safari 2 and the article explains how. It's Safari 3 beta for Windows that currently has a problem.

Updating the post to take your comment into account...

# September 17, 2007 1:06 PM

Bertrand Le Roy said:

Neil: thanks for the trick. It does work. I'll update the article, but I probably won't integrate that into the product as the bug seems to be fixed in Oera 9.5.

# September 17, 2007 1:52 PM

Bertrand Le Roy said:

Milan: it's fairly easy to build an iframe dynamically, even without a physical page to point at (just use the magical url). But you can't use it for history management. Sorry about the misunderstanding.

# September 19, 2007 1:50 PM

I don't get the weird imagecrack to work said:

I've been trying to fix Opera support with this timer method for a long time now. I've noticed that timing only stops when the hash or adress changes. I think it would be prefereble to run the timer in a seperate page in a third iFrame on the page where the URL doesn't change. Reaching the timer is probably a little tricky, but for someone who can code well it's no problem i think. I have no idea if this works, but it seams logical enough to me..

# October 2, 2007 4:04 PM

wojtek said:

Bertrand,

Thanks for your work, but it wold be nice if you could break this article into parts and post updates in separate posts instead of cramming everything into this post and changing it around, it's hard to read and hard to follow.

I also believe "current state" should be a summary of how to make it work, sort of an abstract or outcome of your search - now it's just a reference to the article, which like I said is pretty hard to follow (I read the original, now I have to keep coming back to read again and again).

I wouldn't mind an edit that says click here for follow up, then the rest posted in new articles.

Cheers

Wojtek

# October 7, 2007 11:27 AM

Bertrand Le Roy said:

Well, I prefer to have all the information in a single place. Thanks for the comment.

# October 8, 2007 12:50 AM

Gabe Misura said:

How about we do what so many business people ask: Get rid of the back button? heh.

We write web_apps_, in non-webapps, are there usually back buttons? Is there still a need to have the back button?

I still use it, so I'm sure it would be very hard to break everyone's habit of using/needing it.

# October 14, 2007 1:58 AM

Bertrand Le Roy said:

Gabe: the problem is that you can't get rid of it, so you might as well make it do something useful.

# October 15, 2007 1:42 PM

Shouvik Basu said:

That was excellent, no more large libraries, great insight what actually happens. worked in IE7 Firefox 2.0.0.9 no problems as a user. i used it in my project customized a lot.

in my project I've made similar libraries for separate browsers. they all have merged beautifully

still working to merge opera and safari in my project THanks

# November 4, 2007 12:18 PM

Bertrand Le Roy said:

Matthias and Bob: each test file illustrates how to make it work for a particular browser. A file that's intended for IE is not supposed to work in Firefox, and vice-versa. The files provided here are not in the business of providing a cross-browser history library but to explain how to build one. If you're looking for a full solution, those techniques are used in the next release of ASP.NET Ajax and in the latest RSH library (which is of course totally independant from Microsoft).

# November 5, 2007 6:39 PM

Bertrand Le Roy said:

@Carlo: thanks, that's interesting but you still need to include that script at a specific place on the page, so you might as well include the iframe instead, it's just as simple.

# March 28, 2008 1:29 PM

ben said:

thanks for the javascript opera tick trick :)

# July 25, 2008 9:53 AM

Anastasius said:

Thanks for the IE hack info!

But it doesn't work on the server, when current page is being refreshed. Strangely, it works locally.

I mean - I'm creating 0,1,2,3,4 hashes. Than refresh the page - and then when I press back, it doesn't gets me back to hash number 3.

And this is right only for the remote server, when I execure your files locally, it's all ok. :(

Are there any workarounds?

# October 3, 2008 8:40 AM

Anastasius said:

Have to add.... It doesn't work on remote server, when you use this technique in .php file. With .htm it does work.

# October 3, 2008 9:02 AM

NoFear said:

I get a "Viewstate failed MAC validation" error in firefox (works in IE) after I add one or more history points and click on a link that does a response.redirect in the server-side click handler. I can provide sample code to anyone willing to assist.

# April 8, 2010 1:47 PM

Bertrand Le Roy said:

@NoFear: that's probably because of that weird behavior of Firefox regarding the hash URL encoding. It would require some investigation but it's possible that they fixed it and that broke code that was counting on the wrong behavior. I'll investigate.

# April 8, 2010 2:01 PM

Bertrand Le Roy said:

No, they still haven't fixed it and show no intention of doing it under the dubious excuse that it would break the code that works around their bug.

Ah, actually are you still getting that error if you don't create the history points?

# April 8, 2010 7:56 PM