Back to playing with my Web Search Spider

Appropiate that this is my first post in 2006.  I spent several hours this evening working getting my web search spider back up and going.  I have it running on .NET 2.0 with Sql Server 2005.  I need to go back and make a stored procedure change or two and it will be ready for me to scale it back up.  I have it running on my 32 bit laptop right now.  I am going to get it going in Win64 for x64.  Things to do:

  • Update a stored procedure to be more efficient.
  • Cleanup some string processing that I do regarding urls.
  • Move my MSMQ processing code from a winforms app to a Windows Service.
  • Move my spider from a winforms app to a Windows Service.


  • What do you use the spider for?

  • Chris,

    Hi Chris. Good question. There are a couple of reasons that I first built it.

    1. My thing has always been large amounts of data. Scalability is a key issue for me. So, a database of the web is an interesting idea. It really gets you to understand things like threading, database indexes, and other important items.

    2. I just wanted to build it and understand other issues. Because its there is kindof a cool reason, I hope.


  • That makes perfect sense to me. I got on the idea of creating one a few years back, just never actually got around to working on coding it. Always have too many other projects I start and never finish ;)

  • do you have plans of sharing the winforms version you've made ?

  • Is this spider something you built from scratch yourselft? Or did you use other pre build components or libraries

  • I built everything except for the routine to get links. I found that in some magazine a few years ago. Everything else, from the threading routines, database schema, msmq integration, and other stuff is mine.

Comments have been disabled for this content.