How to build a cross-browser history management system

When we built the history management feature in ASP.NET Futures, we spent considerable time experimenting with the different behaviors of the main browsers out there. The problem with such a feature is that it has to rely on a number of hacks because browser vendors basically never anticipated this need. Now they're thinking about it, so all this may be simplified in a few years, but in the meantime, it's a very complicated feature to build. One of the things that struck me was how little reliable literature is available on the subject. There is a lot of partial information, lots of false or unverified information, but very little that's really comprehensive, reliable and up to date. Good references I found include Brad Neuberg's Really Simple History and Handling Bookmarks and Back Button as well as Mike Stenhouse's Fixing the Back Button and Enabling Bookmarking for Ajax Applications. But it was a lot easier to just experiment directly on the different browsers and verify our theories directly.

This blog post will attempt to be an account of the difficulties we encountered and how we worked around them. It's also a brain dump that will help me and others maintain the feature in the long term. I hope it will evolve over time as feedback comes in, browser bugs get fixed and we ourselves learn more.

Note: all code in the pages attached to this post is pure JavaScript and XHTML and has no dependency to any script library.

How history used to work

It used to be the case that history in the browser just worked: the user navigated from page to page following links and the history stack just got filled, the back button worked as usual and life was good. With dynamic pages that post forms, things got a little less ideal as every post action would make a new entry, no matter if or how little the state of the page changed. Plus, you got this nasty "do you really want to repost this form" dialog that most users had no clue what to do with.

But Ajax applications broke the model in a more fundamental way because most actions are done without navigating away from the page or posting forms. This means that the browser basically has no way of knowing the state of the application changed in a meaningful way and nothing new gets in the history stack. Any naive user who doesn't know the implementation details of your application will expect the back button to work like it has for decades, which is catastrophic because by navigating back to the previous page it knows about, the browser will lose all the user's precious work.

Thanks to a few hacks such as the ones I will expose here, it doesn't have to be that way, and the control of the history stack can be given back to the application developer. We can even make the model better than it used to be and choose exactly what constitutes a meaningful enough state change to warrant an entry in history.

Navigating without moving

The main trick that history managers use is to have the browser believe the user navigated to a new url without the current page and all its JavaScript and DOM state being thrown away. The only part of the url that enables such a thing is the hash part. The hash part is what comes at the end of the url after a pound (#) sign. The original intent of this part of the url was to allow for navigation inside of the document. You would put a special named, href-less anchor tag in your document, and then navigating to #nameOfTheAnchor would just scroll the anchor into view. The page doesn't get reloaded, but it does enter the browser history (almost, we'll see about that in a minute). This is great for long pages with a table of contents for example.

But it's also great in that it's a way for us to manipulate the history stack without unloading the page, which is exactly what we need. So this is one more example of using something for a totally different use than what it's been designed for, but what choice do we have? And it's actually all right as this new usage doesn't conflict with the old one.

So it seems like we have an easy way to get this to work: whenever you want to create a history point, just navigate the page to #SerializedFormOfTheStateOfTheApplication; detect url changes somehow (polling works (almost) everywhere) and re-hydrate the application state from that. That should work, right? Yes, it should, but it doesn't, except on Firefox and Safari 3 for the Mac, for various reasons.

I've prepared a page that implements exactly that logic that you can try here:
HistoryHash.htm.txt (download and rename HistoryHash.htm to run on your system)

Here's a summary of the results on some popular browsers:

Making it work in IE

The problem in IE comes from the fact that it doesn't create a history entry when the hash changes. This can be worked around by navigating an iframe in addition to changing the hash as IE does create a new history point every time a frame is navigated. The problem with that technique is that just like the main page, the iframe can't be just navigated to a new hash, it needs to move to a different page, which means a round-trip to the server on every new state, which is quite wasteful. I've heard of techniques to create the iframe's contents through clever scripting in the iframe url and document.writing, but I've never been able to consistently reproduce it.

<update date="Sep. 14 2007">
One of the comments over at Ajaxian pointed me to this great article from Julien Lecomte, where he explains how they built the feature over at YahooUI. It's a super-interesting read and thanks to it, the document.writing trick suddenly clicked together for me. The trick is that you need to open the document before you write to it and close it when you're done if you want the history point to be created by IE. Somehow I had missed that but today I tried again and had no difficulty making it work. The only thing that may be a little tricky is that you need to pass a url into document.open or the browser will default to about:blank. That may look ok until you try it under https, in which case you're going to get the infamous dialog about mixed contents. This can be solved by using the usual magical url javascript:'<html></html>' (thanks to the toolkit team for that trick). Please note that now we don't even have a single request for the frame, even on the first hit, again thanks to the magical url. It's also important to note that the iframe must absolutely be in the static markup of the page (you can't create it dynamically) and on the first request it must point to an existing page on the server.
I've updated the sample page for IE below to include this trick and I'll also integrate it into ASP.NET Ajax.
</update>

The frame gets the state passed to it through the search part of the url (the part after a question mark). It then calls back into a function in the main document when it loads. When the user navigates back to a previous state, it's the iframe that really navigates back, not the main page. When it does, its load event fires, which calls back into the main page again. From that callback function, we set the hash so that the browser url reflects the state for bookmarkability, and of course we update the application with that state.

Here's a simplified implementation of this. I didn't hide the iframe to make clearer what is happening here but that's of course what you'd do in production code.
HistoryIE.htm.txt (download and rename HistoryIE.htm to run on your system)
You will also need the iframe file:
HistoryFrame.htm.txt (download and rename HistoryFrame.htm to run on your system)

<update date="Mar. 31 2008">
Internet Explorer 8 finally gets rid of the need for the iframe navigation: changing the url fragment now results in a new entry being created in browser history. We were able to make our implementation of history work in IE8 by just restricting the iframe code to versions earlier than 8. Internet Explorer now follows exactly the same code path as Firefox, Safari 3 and Opera 9.5. Not all is perfect though as IE still has this weird quirk where when navigating away from the history-managed page causes its history entries to collapse to just one entry. If you do go back, it will expand again but this is really confusing to the users.
On the bright side, IE is the only browser as I write that implements an HTML5 event that gets triggered when the url fragment changes, eliminating the need to poll. I really hope the other browser follow that lead...
</update>

Making it work in Safari 2

While the workaround for IE results in a completely different implementation, at least it makes some sense and is consistent. With Safari, we're lost in bugland. In Safari 2, monitoring the hash from a timer doesn't work as location.hash is not getting updated as the url in the browser's address bar changes. This is a plain bug that is fixed in Safari 3 and the current WebKit builds. The only information we have that gets correctly updated when the user navigates through history is history.length. In other words, we know the index of the history point but we don't know the corresponding state. One thing we can do to work around this is to maintain our own array of states and use the index to retrieve the right state from this array. One caveat is that if we use a JavaScript variable to store this array, navigating to a different page will completely wipe out the state array and kill the history feature. This can be partially worked around by storing the information into a hidden form field instead of a JavaScript variable: form field values are restored by the browser when navigating through history. This works relatively well to maintain history even if the user navigates away to a different page and comes back but what it doesn't handle on the other hand is the user pressing F5.

Here's an implementation of those workarounds (I've left the history stack field visible to make it easier to understand what happens but of course in a real application it would be idden):
HistorySafari2.htm.txt (download and rename HistorySafari2.htm to run on your system)

This must be fixed in Safari 3, right?

Well, yes, Safari 3 Mac does the right thing as the unmodified hash code that works on Firefox just works. But on Windows currently other bugs cripple the feature. In the current nightly build and public beta of Safari 3 for Windows, if your page is the first that you load in the browser, no history entries get created when the hash is modified. Another weird bug that I couldn't quite find reliable repro steps for is that under some circumstances, hitting back does not enable forward, making time travel only possible to the past, without any hope of return. Somebody give them a flux capacitor, an old clock and a thunderstorm... Because of this, the Safari 3 for Windows implementation kind of works sometimes but it really needs to be fixed. I filed bug 14080 in WebKit a while ago to track this. Seriously, I don't blame them, it's a beta, betas have bugs and I'm pretty confident this will get fixed pretty soon. I've had pretty good experience with their reactivity in the past.

What about Opera then?

Well, it used to work perfectly well in Opera up until version 9.10. Then they broke it. In 9.23, and also in 9.24 which is the current version as I'm writing this (not a beta), for some reason Opera cancels all timers when hitting back. This completely breaks any history manager because there is now no way of knowing the user navigated. This problem has been reported by several persons on the Opera forums and apparently there's a bug open for it, but the closedness of Opera's bug database doesn't allow us to monitor the status of the issue.

<update date="Sep. 17 2007">
Neil Jenkins pointed me to a nice trick he's come up with to work around this bug. The idea is to have a hidden image with the following src attribute: javascript:location.href='javascript:onTick();';
Amazingly, this works and the code gets executed when the user hits back. Just doing javascript:onTick(); directly doesn't work for some reason that I don't quite understand. Apparently, Opera tries to be smart about what can be run as an img src, but it's easily worked around. If you ask me, script should simply not be allowed as image urls (as well as in a number of places) but I suppose that would break too many hacksapplications.
Anyway, the workaround does work, and it's simple enough that you should use it if you need to support Opera 9.23 and 9.24. But you also need to know that this is fixed in Opera 9.5 (still in beta as I'm writing this) so it may not be worth fixing in the long run. Kudos to Opera for their reactivity on this issue.
HistoryOpera.htm.txt (download and rename HistoryOpera.htm to run on your system).
</update>

The state of things

So things are in a pretty grim state currently. It seems like we're going back (pun intended). We used to have a collection of tricks that made possible an implementation of a history manager that worked pretty well in IE, Firefox, Opera and Safari. Now, we only have IE, Firefox and Safari Mac. I just hope this is only temporary and that both Apple and Opera repair their browsers soon.

<update date="Sep. 17 2007">
As explained in the previous section, there's now a known workaround for the Opera bug, and the bug itself will be fixed in 9.5.
</update>

<update date="Feb. 18 2008">
The WebKit bug seems to be fixed.
</update>

<update date="Apr. 18 2008">
Apparently I forgot to mention one annoying bug in Firefox that url-decodes the hash automatically, which gets in the way of using the hash as a querystring-like state bag. More details in the bug comments here: https://bugzilla.mozilla.org/show_bug.cgi?id=378962#c4.
</update>

<update date="Dec. 22 2008">
Some great additional information on ways to help refresh behave better on IE can be found here: http://www.overset.com/2008/12/21/internet-explorer-iframe-browser-history-lost-on-reload/
</update>

What's next?

In the next posts, I'll get into more details about the initial request, maintaining meaningful titles and mixing all this with asynchronous operations, which is where it really gets tricky.