Evolution of the ASP.NET Test Process (Part 2)

This is the second part of a short narration that describes how the testing process has evolved in the ASP.NET QA Team. First part here.

The Feature Crew Model

After the release of .NET 2.0, all of Developer Division underwent a general change in the way we built software. This change came from top-down, and it was called the "Feature Crew Model'. I remember that all DevDiv made a lot of preparations for this new methodology, taking months out of development time to build a lot of infrastructure and tools to make it happen. Soma must be recognized for taking this bold step across the division. The basics are:

1. Each Product Unit (PU) inside DevDiv would divide their release into features. A "feature" meant a logical sub component of the product, that in theory could be developed independently.
2. A feature crew would be created from resources from the PM, Dev and QA disciplines and they would be given some degree of autonomy to shape the feature and manage their milestones.
3. A feature crew would develop everything on a separate independent branch.
4. A series of Quality Gates were put in place and only after a feature passed all of them, would the code be permitted to integrate into the main product branch.

The key part of this process was that it enforced a high level of quality before each individual piece made it into the build, and that each piece was finished. This was a big improvement because in the past there was a lot code churn happening on the main branch from all over the division that generated a lot of regressions. The second benefit was that this way of working encouraged better communication between Dev and Test, and allowed for teams that wanted to experiment with SCRUM or other agile methodologies.

Our First Feature Crews (ASP.NET Ajax 1.0 Release)

Our team's first encounter with this process was during a project code named Atlas (later to be known as ASP.NET AJAX). Back then, nobody was really sure how this would work out (and there wasn't a lot of guidelines coming from the division either), so it was understood that we would experiment and modify as we went.

Atlas was divided into the following feature crews (they were kicked off in this order, since subsequent crews relied on the previous):

• Core: which included the type system and BCL.
• Networking: everything under Sys.Net
• Partial Rendering: UpdatePanel (this is the feature crew that I worked on, along side Eilon Lipton)
• UpdateProgress/Timer: targeted crews for only these 2 controls.
• Services: script callable Membership, Profile and Roles services.

ASP.NET QA Process in a Feature Crew

Looking back, the QA team didn't made any substantial changes from our old process, we were too busy preparing for having our test sources in different branches. So the QA process ended up being:

1. QA resources would be assigned to a feature crew.
2. Dev/PM/Test would start working on the feature at the same time, on an independent branch.
3. Dev was now responsible for writing UnitTests
4. Each feature crew would meet at least 3 times a week to design the feature and provide status updates. (The crew was in charge of costing all work, and arranging it in milestones. At the end of each milestone, the crew would integrate the progress into main).
5. As the design was being developed, the PM worked on the spec, Dev on prototypes and QA on the test plan.
6. QA would have a test plan review as before.
7. QA would start testing the feature as soon as code was checked in by Dev, and would start writing automation.
8. Progress was still tracked by automation complete numbers.
9. QA was responsible of signing off a number of items on the Quality Gates check list. Of importance, is the fact a quality gate was to be automation complete, and to have run all test in a number of configuration runs.
10. After all quality gates were met, feature was integrated to Main.

How did it Work?

• Benefits
• Better collaboration between disciplines (we were all on the same boat to get our stuff integrated).
• QA was involved in the design.
• Code that entered the build was of high quality already.
• Dev's wrote unit tests which caught bugs earlier.
• Disadvantages
• Probably the biggest problem was that QA still tried to do everything that we used to do on a 2-3 year release, but now on a compressed 3-4 months iterations. Because we didn't change anything in our process, we now really became the bottle neck.
• QA was still creating full featured test plans and focused heavily on automation, for features that now changed even more often.
• Overlap between dev unit tests and QA functional tests, the old mentality of "test everything" did not go away.
• Dependencies between crews created version problems that sometimes got crews blocked.

In the end, the team pushed quality up stream, but the QA team bled and struggled a lot. And some of the problems that we had identified before began happening again, in particular, finding bugs to late. My opinion is that after Atlas was shipped, the team had a better development process that put quality first and helped the QA team, but our own QA processes still needed to adapt to this new way of doing things.

In Part 3 I'll narrate how the QA team reinvented its methodology for the next ASP.NET release, and how it did it from the bottom up.

Federico Silva Armas
SDET, ASP.NET QA Team