I’ve never been a big fan of surveys or polls. Not because I don’t like them, but because I always disagree with their validity and the conclusions that are drawn from the results. Lets take a look at a couple of examples. Readers Choice awards were recently handed out. I received an email from a customer who wanted to let me know that he was contacted by a component vendor to go out and vote for them, and he didn’t even own their product. To my surprise, he wanted to know where the Infragistics email was. My answer – we make a very conscious effort to limit the number of emails we send out to our customer base. At the end of the day, our decision to keep spam out of our customer’s inboxes also meant that we were at a disadvantage in the survey/poll/award. Does the poll actually show which product is the best? Or does it show who spent more time and energy on campaigning for votes?
Now let’s look at case #2. There’s a survey out right now asking questions about AJAX tools and frameworks, and which ones developers are using. I don’t know exactly what the results of this survey look like, but here were my first impressions. The survey is hosted on a personal blog, where most readers likely share similar interests. Simone Chiaretta (the blogger) respectfully acknowledges that this blog has a limited audience and is likely biased toward the ALT.NET kind of developer. I’m sure Simone has the best intentions about publishing the survey, but it will be very difficult to not get a skewed response. In addition, what does the response actually mean?
But back to the topic at hand. Polls and surveys are just that. They represent a a sentiment of a select audience. Was the audience wide enough to capture multiple views (usually not). Was the survey done by an impartial third party (usually not). Did others have a chance to influence the results (unfortunately yes most of the time). And even when you get past all of that, it’s most important to look at what the questions were actually asking in the first place. Each time I see the results of a poll, I argue that the answer that was being deduced was not actually the question that was being asked.
My question to you – do you participate in these polls? How much weight/validity do you give the to results? And better yet, what do you think about “lobbying for votes” in readers choice type polls/awards? Is that something you would be happy to hear about, or would you prefer not to hear at all?
Perhaps I’m too analytical, but I generally tear apart survey results and the conclusions that are drawn from them.