Surveys, Polls, Voting, and why they usually stink

I’ve never been a big fan of surveys or polls.  Not because I don’t like them, but because I always disagree with their validity and the conclusions that are drawn from the results.  Lets take a look at a couple of examples.  Readers Choice awards were recently handed out.  I received an email from a customer who wanted to let me know that he was contacted by a component vendor to go out and vote for them, and he didn’t even own their product.  To my surprise, he wanted to know where the Infragistics email was.  My answer – we make a very conscious effort to limit the number of emails we send out to our customer base.  At the end of the day, our decision to keep spam out of our customer’s inboxes also meant that we were at a disadvantage in the survey/poll/award.  Does the poll actually show which product is the best?  Or does it show who spent more time and energy on campaigning for votes?

Now let’s look at case #2.  There’s a survey out right now asking questions about AJAX tools and frameworks, and which ones developers are using.  I don’t know exactly what the results of this survey look like, but here were my first impressions.  The survey is hosted on a personal blog, where most readers likely share similar interests.  Simone Chiaretta (the blogger) respectfully acknowledges that this blog has a limited audience and is likely biased toward the ALT.NET kind of developer.  I’m sure Simone has the best intentions about publishing the survey, but it will be very difficult to not get a skewed response.  In addition, what does the response actually mean?

A current snapshot of results shows that 76% of developers are using jQuery, compared with 48% using ASP.NET AJAX and 58% using AJAX Toolkit.  Does that really mean that there are more jQuery developers than WebForms/ASP.NET AJAX developers?  To confuse matters, AJAX Control Toolkit is built ONTOP of ASP.NET AJAX.  Yet these results show that there are more people using the toolkit than ASP.NET AJAX (which is actually impossible).  I know from experience that there is a huge portion of web developers out there who never want to hear the word “Javascript”.  Does that mean they’re not doing AJAX?  No, they’re simply relying on the tools that were handed to them.  Microsoft created the Microsoft ASP.NET AJAX stack and made doing things like partial rendering a breeze.  It’s all part of ASP.NET 3.5.  I don’t even call it “AJAX Extensions” anymore, it’s just ASP.NET 3.5, or Visual Studio 2008.   It’s actually so easy to do Partial Updates at this point, that a developer can ‘do AJAX’ with out even knowing how to write a line of JavaScript.  It’s very unlikely that those developers are or will be represented in this survey.  How many web developers still use the term DHTML?  That usually only shows up on resumes.  But in actuality, most web developers today are doing DHTML, and aren’t even aware of it.  I expect the same will be true of AJAX.  It’s so commonplace that there’s no reason to differentiate it from web development in the first place. 

So what spawned all of this?  Well, I saw a post from Betrand Le Roy on Twitter about this survey.  He immediately got a response asking “will the results of this survey be used to finally convince Microsoft to dump ASP.NET AJAX, and go with jQuery instead”.  I can only hope not.  jQuery is an awesome Javascript framework, but it absolutely abandons the web developer who doesn’t want to spend his days writing custom Javascript.  Microsoft created the idea of a server-side webcontrol for this very reason.  If I’m given the choice to drop an UpdatePanel on a form and have a label update on a button click via AJAX, or hand code it myself using a Javascript framework, I’m going to go with the UpdatePanel just about every time.  Will I get more satisfaction out of building it from scratch?  Certainly.  But personal satisfaction doesn’t pay the bills. And I’d be much more satisfied spending the two weeks time I saved, on the beach fishing.  The Infragistics ASP.NET controls try to find a comfortable balance between these two worlds.  They abstract away the client-side javascript for developers who don’t want to or have the time to deal with it, and they provide a client-side object model for developers who want to dive in and play.  We call it our CSOM (Client-Side Object Model) and it’s been around for years.  I think this is the path Microsoft needs to continue to take, supporting both of these key scenarios. 

But back to the topic at hand.  Polls and surveys are just that.  They represent a a sentiment of a select audience.  Was the audience wide enough to capture multiple views (usually not).  Was the survey done by an impartial third party (usually not).  Did others have a chance to influence the results (unfortunately yes most of the time).  And even when you get past all of that, it’s most important to look at what the questions were actually asking in the first place.  Each time I see the results of a poll, I argue that the answer that was being deduced was not actually the question that was being asked.  

My question to you – do you participate in these polls?  How much weight/validity do you give the to results?  And better yet, what do you think about “lobbying for votes” in readers choice type polls/awards?  Is that something you would be happy to hear about, or would you prefer not to hear at all? 

Perhaps I’m too analytical, but I generally tear apart survey results and the conclusions that are drawn from them.