I have kind of put my WebSearch code off on the back burner while I worked on real customer work for the last couple of weeks. Well, I set down this morning and started looking at my data and how things were progressing with about 10 million records. Well, I found an oops, and I think it was a pretty good one. It ends up that I was taking a base url and the script and just plugging them together in a certain case. The result was that I was creating urls like http://www.domainname.com//page.asp instead of http://www.domainname.com/page.asp. Well, I took care of that this morning. I decided to purge my data and start over. I checked my data a couple of times this evening and it appears that everything is running more correct now.
I have made another change to how I am pulling data back from the database in attempt to get around another bottleneck that I am seeing. More info on this after I have a chance to see it when there is more data in the database.