As I was first building my Web Spider, i figured that the easiest thing to build the spider with would be the TP. So based on my previous ramblings, I was disappointed by the fact that the WebClient also used the TP to retrieve its results, even when used in a synchronous fashion. This effectively cut my possible performance in half. Add to this the fact that the TP in .NET only supports 25 threads per cpu at any one moment and I was double frustrated. The result was that I could only fire up 12.5 threads per cpu on my development system. I just knew that if I could just switch to managed threads, I would be able to pull in 25 threads per cpu (based on the WebClient in System.NET). While I am also constrained by the bandwidth at my office, I knew that the addition of more threads would allow me to “smooth” in the waves when the TP version wasn't able to access the network due to other work that was going on. I just knew that I could outsmart the TP scheduling mechanism that will only allocate a specific number of threads based on system resources.
Given the above, I worked this morning on implementing MTs. I got through my bugs, set the system to run with 20 MTs, hit the start button, and watched with excitement as....................the performance of the app went in the toilet compared to using the TP. How in the world could this happen? Well, as I scaled back the number of threads, I watched performance increase. It appears that the fact that I had too many threads attempting to run across my limited bandwidth was causing too many problems. It looks like, given the amount of bandwidth at my office that 8 MTs is the appropiate number of threads. Maybe I am not smarter than the TP manager in .NET..............
Just remember folks, throwing more threads at a solution does not make that solution run better.