Newest problem with Web Searching

I have found a new problem with my Web Search code.  It seems that there are “gravity wells” on the internet with regards to links.  The biggest “gravity well” seems to be Amazon.com.  I started watching the Sql Profiler and I noticed that the system was becoming fairly slow with regarding to inserts of new records and the queues seemed to be continually backing up.  I then started looking at the data that was being inserted.  All of the slow inserts were going against Amazon.com.  It is my guess that the number of Amazon.com inserts was creating a “hot spot” in the database with regards to the management of the indexes.  So, my next step is going to be putting some type of “governor” into the system so that it won't fill up with entries from any of the “gravity wells.”  I have a pretty good idea how to do implement this within the confines of the database and to be as efficient as possible.  The only problem?  I left my 250 gigabyte firewire drive at the W hotel in Seattle.  This is the drive with my Web Search code and database.  It should be delivered on Wednesday or Thursday.

Wally

No Comments