Evolution of the ASP.NET Test Process (part 1)

For those of you interested in testing methodologies, in these series of posts I'll go over how the QA team has changed the way we test ASP.NET over the years. I have been in the team for over 6 years now and have witnessed several transitions (most of which I think are for the better).

The Early Days (.NET 1.0, 1.1 and 2.0)

I joined the team when .NET 1.1 was about to be released and the team was getting ready to work on 2.0 (which turned out to be a major and lengthy release). Back then the general QA process was:

1. By the time the QA team finished testing and singing off on a release, the next release was well under development. There is usually a "stabilization" time at the end of a release where the QA team is busy writing automated tests, running them under all sorts of configurations and going through a list of checks called "Exit Criteria". This meant that Dev/PM were off prototyping and designing the features of the next release while we were fully engaged testing the current release.
2. The QA team divided the features between its members, taking into account interest and affinity. This was called "owning" a feature. When I very first started I was the owner of the new GridView and Login controls. At that point, the spec was pretty much finished, and the controls were already checked into the build.
3. The first thing to do was to write a test plan. What this meant was to define the totality of tests that you planned to verify and automate. These documents were very descriptive and usually amounted to many pages. We entered the tests into an internal tool that DevDiv uses called "Maddog".
4. This test plan got reviewed by the PM and the Dev of the feature and by fellow testers to try and uncover anything missing or worthy of expansion.
5. Next came the main testing phase where the tester divided his/her time between performing "ad-hoc" (which meant playing with the features trying out all sorts of scenarios to uncover defects), and writing automation.
6. Progress was tracked by the number of tests automated that were defined in the test plan. For example, a test plan defined 200 tests (with different priorities) and we would report status of how many tests were left to automate and that's how we scheduled the remaining work.
7. One thing to note is that the Developers were not writing any UnitTests back then.
8. At some point, the team would start to kick off runs. This means grabbing all tests in the repository and running them on many different platforms and configurations to uncover any possible defects. In this phase the time of a tester was divided between writing automation and analyzing the inevitable failures of each run.
9. This is were we entered the "stabilization" time for the release that I mentioned at the beginning. Making lab runs on different SKU's, languages, side by side, x32 and x64, stress, perf, etc.
10. After many months, at last we would have finished automating all the tests and have them executing on all the runs. Any bugs found were logged, and either fixed or postponed. At this point each feature owner would sign off, and the product would be shipped. (A similar process was repeated for every beta release as well).

Problems with this model

• Over-Automation
• By placing too much focus on automation, the tester spent more time writing and debugging a huge amount of regression tests rather than trying scenarios on the product.
• There was a philosophy of leaving no stone unturned and to automate everything that you could think of.
• High cost of change
• ASP.NET is a framework, and frameworks go through many revisions and changes before they are released. However, our QA process was not designed to accommodate this. The QA methodology consistently tried to lock down details too early.
• We spent a lot of time writing fully detailed test plans based on the spec for things that were bound to change a lot, and immediately after test plan creation we launched ourselves into an automation frenzy.
• What ended up happening is that changes in the design of a feature where disastrous, the cost of updating all test plans and all the previously written automation was very high.
• No UnitTests
• The QA team was responsible for testing everything, from passing every possible value to each public method of a class to end to end integration tests. This amounted to a TON of tests written by QA.
• Since the only automated mechanism to find regressions was the QA tests it was common for the builds to be broken regularly.
• QA team found bugs to late
• A side effect of our automation story was that scenarios were not verified until it came time to automate the test, which usually meant months after the feature was coded.
• All this miss placed effort meant that bugs where found later rather than sooner.
• QA team did not start at the same time as Dev/PM.
• In some ways this fact set us up to be the long pole of the team. There were occasions where the Dev had completed all his/her work, but the QA team still had months and months of work to go through.
• It was also common practice for the QA members to have little input on the design of the product. By the time we became fully engaged, most of the features were already designed and implemented.

How we survived

By reading this post you may be disappointed by our team, but keep in mind that I only focused on the "bad" things of the way we used to do things 6 years ago. I believe that regardless of the limitations of our approach the release had a high quality bar. The problems that I mention here, plus the problems with the Dev's process and with our automation infrastructure, all these together are one of the reasons why the .NET 2.0 release took such a long time. Luckily, most of these have been addressed during subsequent releases, and I'll go through the changes that we made and how well they worked in following posts. Maybe you can learn something from our experience...

I am also interested in hearing how you approach testing in your organization, what has worked well and what has not worked so well. I am looking forward to any comments or feedback that you may have. Stay tuned for part #2 :D

Federico Silva Armas
ASP.NET QA Team