Modified Web Search Algorithm
With new file space in tow, I updated my search algorithm. Instead of having a single threaded Url Dispatch, Each thread is now responsible for grabbing a set of Urls from the database to search. With that change, I appear to have gotten a bump in performance in that I no longer have a set of downtime in getting URLs. I am going to let it run for the next few hours and see how things go. So far, I like what I see.
Wally