Beta or not beta? Another Microsoft opinion

So busy that I missed this new and very long comment from Marie Hagman, Visual Studio, Microsoft Program Manager on the Visual Studio feedback page for a Beta 3 release. Indeed where Microsoft is wrong is that we don't really want another Beta really, the main point is to know more about the future of VS 2005 service packs.

I also added below the comment from Clint (the guy who start this debate). What can I add to that? I don't like the word Postponed, seems like synonym to trashed.

You are absolutely right, if we find Whidbey quality is not ready for a Nov 7 ship we should and will delay shipping. We feel, however, that the product has reached that key inflection point where shipping the huge progress we’ve achieved brings far greater benefit to customers than further delaying to fix a few remaining issues that won’t impact most users.

We opened the Product Feedback Center last year with the release of Beta 1 to empower customers and as a result have received a lot of high quality feedback. We’ve fixed a very high percentage of reproducible customer reported code defects. As VS Client Dev Manager Shawn Burke writes “if you're filing bugs on MSDN Feedback that you really, truely, honestly believe are ship-stopping bugs, reactivate the bug and add your justification as to why you believe that. Write the scenario that's broken and why it's so important, or ask for more information about why it doesn't meet the bar. State that well, and the right things will happen.” See Shawn Burke’s full commentary on this on his blog - http://www.shawnburke.net/Default.aspx?document=264&userinterface=9

Over the course of the Whidbey product cycle we have modified shipping schedules and put new processes in place, to make sure our customer needs are addressed. There is always more to be done and no release is 100% bug free, especially with such a complex product as Visual Studio.

Visual Studio is over 43 million lines of code, there are over 30 teams working on different pieces, with roughly 700 developers checking-in code to 11 different virtual build labs that are then integrated on a rotating schedule producing over 100 different builds of the product daily. In addition we have interdependencies with SQL and MSDN. When we ship an official release like a Beta or RTM (release to market) we lock down and are code complete several months before the actual release date to allow for a final test pass, to stabilize and hit stress goals, then get the best bits to fulfillment for mass production of media. Before code complete there is a long list of exit criteria for the product that must be met, ensuring all key features and scenarios meet expectations. We have customer bug exit criteria to ensure high priority issues are fixed.

Why are we postponing so many bugs? As Shawn says “It's a massive balancing act: how do I ship quality software that will do the right thing for my users and still close it down and get it out the door with known issues? Is this paradoxical? For large software projects, at some point this becomes a treadmill. You could literally keep at it forever if you kept fixing all the bugs. Multiply this by the complexity factor…If you keep perturbing the system, you'll never get there, especially when it comes to sensitive things like stress results.”

Bug triage, where we determine priority and resolution of bugs, can be an unbelievably difficult process in which teams must reach consensus on issues where there are often conflicting considerations. Security, usability, accessibility, integration, would it be a breaking change, is it blocking or a build break… etc. Over 30 people from across the Division meet daily to discuss bugs and fixes ensure the right calls are made in team triage. The Product Feedback Center gives customers a new visibility into this process. You can see how your bug has been assessed against the triage bar and why Microsoft has made the decision to resolve it as Postponed, Fixed, or Won’t Fix. Decisions have been reversed based on customer comments and community vote. Bottom line is we are committed to shipping the right product and through the Product Feedback Center we have an even better sense of what that is.

Thanks,
Marie Hagman
Visual Studio
Program Manager 

Answer from Clint Stotesbery:

You point out some comments in Shawn Burke's blog at http://www.shawnburke.com/Default.aspx?document=264&userinterface=9

You can read my replies there too ( number 6 and number 9) rather than me repeating what I said.

Recently there was some new information released by Soma at his blog, http://blogs.msdn.com/somasegar/archive/2005/08/22/451026.aspx

Releasing an RC for VS and Beta 3 for TFS with a Go Live license is pretty close to what I suggested so this could probably be closed as Fixed. It would be nice to have the RC and Beta 3 available to non-MSDN subscribers though.

The only other issue that I think would benefit the community would be to have a statement regarding service packs for Visual Studio. Reintroducing service packs, and releasing one within 6 to 12 months of November, would make many developers less concerned about their issues. The community consensus seems to be that waiting 2-3 years for bug fixes (in form of another VS release) isn't acceptable.
So in summary:
1. Please make VS RC and TFS Beta 3 available to non-MSDN subscribers.
2. Please make a statement regarding reintroducing service packs for VS.

Thanks.
 
 
Proposed Solution:  Put out a Beta 3 release at the end of September with a Go Live license for VS Team Suite, VS Standard/Pro, TFS, etc. Push back RTM if you have to. RTM December 31st or push it to 2006 (just keep the 2005 name then, no big deal).

Yes, people will complain about changing release dates but it isn't like this will be the first time. Developers won't care about when the RTM date was a few months after RTM if the product is full of bugs.