FYI, I'm blogging most of my stuff over at More Wally now.
You might want to add my rss feed to your reader at:http://morewally.com/cs/blogs/wallym/rss.aspx
Back to playing with my Web Search Spider - Wallace B. McClure

Wallace B. McClure

All About Wally McClure - The musings of Wallym on .NET, Sql, ASP.NET, and other crazy shenanigans

News

Personal Blog

Work Blog

.NET

Book Authors

Business

Family

Friends

Georgia Tech Bloggers

Personal

Back to playing with my Web Search Spider

Appropiate that this is my first post in 2006.  I spent several hours this evening working getting my web search spider back up and going.  I have it running on .NET 2.0 with Sql Server 2005.  I need to go back and make a stored procedure change or two and it will be ready for me to scale it back up.  I have it running on my 32 bit laptop right now.  I am going to get it going in Win64 for x64.  Things to do:

  • Update a stored procedure to be more efficient.
  • Cleanup some string processing that I do regarding urls.
  • Move my MSMQ processing code from a winforms app to a Windows Service.
  • Move my spider from a winforms app to a Windows Service.
Posted: Jan 01 2006, 12:47 AM by Wallym | with 6 comment(s)
Filed under: , , ,

Comments

Chris Hammond said:

What do you use the spider for?
# January 1, 2006 5:11 PM

Wallym said:

Chris,

Hi Chris. Good question. There are a couple of reasons that I first built it.
1. My thing has always been large amounts of data. Scalability is a key issue for me. So, a database of the web is an interesting idea. It really gets you to understand things like threading, database indexes, and other important items.
2. I just wanted to build it and understand other issues. Because its there is kindof a cool reason, I hope.

Wally
# January 1, 2006 6:51 PM

Chris Hammond said:

That makes perfect sense to me. I got on the idea of creating one a few years back, just never actually got around to working on coding it. Always have too many other projects I start and never finish ;)
# January 1, 2006 10:39 PM

miwas said:

do you have plans of sharing the winforms version you've made ?
# January 4, 2006 5:00 AM

Carlos Alston said:

Is this spider something you built from scratch yourselft? Or did you use other pre build components or libraries
# January 4, 2006 1:41 PM

Wallym said:

I built everything except for the routine to get links. I found that in some magazine a few years ago. Everything else, from the threading routines, database schema, msmq integration, and other stuff is mine.
# January 4, 2006 1:49 PM
Leave a Comment

(required) 

(required) 

(optional)

(required)