ZBB Test Pass...
The ASP.NET test team is in the middle of our ZBB Test Pass. ZBB stands for Zero Bug Bounce. The release team blog has a pretty good explanation of the drive to ZBB and its meaning. The blog though does not point out that most test teams in the division try to execute a full test pass pre ZBB to ensure that all open issues are discovered and addressed during that time period. Here are some high level points about the ZBB Test Pass that teams usually execute pre-ZBB and about the ASP.NET ZBB Test Pass that is currently in progress.
- Each milestone we typically have two test passes. One before ZBB and one right towards the end of the milestone to certify and verify the final bits that are going to customers. The first test pass' purpose is to make sure we identify any bugs/regressions right before ZBB so that hitting ZBB gives us more confidence in our schedule. (i.e. we have high confidence that we have discovered all major issues/defects in the product early on and there is low probability of finding last minutes problems that would impact the schedule).
- We typically run our full automated test bed. Our current automated test bed is around 356,079 scenarios and will be growing in the few weeks past our test pass as we wrap up our automation.
- The test pass takes around 3 weeks to complete. (Just looking at our current progress we might be able to finish earlier actually :))
- The tests are executed using our internal test execution system (aka MADDOG). The run executes on around 80 machines in our lab with 30 of them used as Tiers. Tiers are basically different server configurations that allow us to execute a testcase multiple times within a different environment. (i.e. Test_A will run against a server that is running ASP.NET Process Identity as Machine and then will run again on a different server configured with ASP.NET Process Identify running as System.)
Hope that gives you more insight into what the test teams are doing right around ZBB.