Modified Web Search Algorithm

With new file space in tow, I updated my search algorithm. Instead of having a single threaded Url Dispatch, Each thread is now responsible for grabbing a set of Urls from the database to search.  With that change, I appear to have gotten a bump in performance in that I no longer have a set of downtime in getting URLs.  I am going to let it run for the next few hours and see how things go.  So far, I like what I see.

Wally

1 Comment

  • I'm curious: is there a possibility you will ever release the source of your pet project? =)



    I've been reading your posts for the last days, and i'm truly impressed by what your doing and it would be *great* to learn from your skills :)

Comments have been disabled for this content.