AJAX and the Search Engine Problems

Saturday, August 6, 2005 3:02:00 PM

There are already thousands of web sites using AJAX or similar techniques. One problem all of them have is the point that web crawlers will only search for HREF, SRC or some other link tags. I never saw a web crawler that is searching for AJAX JavaScript proxies. I think web developers only see the benefit of AJAX for the speed/performance advantage of web sites. If you want to build web sites that will be reachable through search engines you have to write two sites, one with the usage of AJAX for the human visitor and one for web crawlers that will need the complete HTML/text of the page.

Backbase.com is one of the web sites that is using two versions:

Note: You cannot open the "search engine" version directly, use telnet.exe or the Google cached page at http://64.233.183.104/search?q=cache:pVNGuQ5tDvAJ:backbase.com/go/home/company/news/008_ohra_backbase.php++%22u%22+site:backbase.com&hl=de. If you click on the link for the search engines you will be redirected to the AJAX version.

We have to think about this problem when we are designing web sites, not web applications. On real web applications the need of be search engine compatible is not as high as for news and product information web sites. Or will Google change their search engine?

6 Comments

  • Thiago RS said

    That's a good point. But the problem is: Google and other SE may see this as cloaking, a method to cheat search engines delivering different contents to humans and crawlers, and you can be dropped from the result pages.

    I think another way to get your AJAX'd pages spidered is put the corresponding link in href and do not follow it, like this:

    <a href="myContent.htm" onclick="myAjaxMethod();return false;">My text</a>

    The 'return false' will make the browser not follow myLink.htm, but will call myAjaxMethod. SE spiders WILL follow myContent.htm and index contents.

    This works and you won't be penalized by search engines.

    PS: Sorry by my bad bad english :)

  • Michael Schwarz said

    @Thiago: Yes, of course, there are several ways to fix this problem, but do you really want to add such links?

    And, what do you want to put in the HREF attribute if the AJAX request depends on some other values (i.e. you have entered several values in a form and want to display only these results)? Do you want to add everytime a summery/overview page?

    I think it is not as easy as you are thinking about this. In simple web sites (like replacing the HTML content only) you are right. But in more complex web sites like shopping sites it will be more difficult.

    CIAO
    Michael

  • Oliver said

    Yes, this is a way to create site that are search engine compatible. But while Michael is talkling about web applications I will say you are not right. If you have a web site that allows you to search for your local office by entering a ZIP code you have to add one HTML page that will show all local offices or you have to add a page for each local office. Ok, we want to display more info of the local office we have decided to use the second way (for each office one page). If you want the search engine find your pages you have to add some links in a homepage. Do you really want to add thousands of A HREF tags that are hidden to the display for these local office pages? I think this will be overloaded!!! Please give me some feedback here in the page if I did not understand your way of doing this. As I can understand this is only working for your backbase solution but not for "default" HTML files. Arrrrrrg...!

  • Thiago RS said

    I am using Google Sitemaps for about 3 months and I got positive results. In my system I got a few thousands webpages (ie www.example.com/user_name) that got no inbound links, so Google cannot find then.
    I made a dynamic generated sitemap and now Google is succesfully indexing every one of then.

Comments have been disabled for this content.