This morning I presented a review of the Web Platform & Tools (WPT) Automation Bucket to the test management team to get feedback on our plans and future direction. The WPT Automation Bucket is tasked to oversee our automation challenges on the WPT test team and seek improvements in the toolset and methodology of testing. One piece of feedback that I got from that group is to do more investigation of web testing tools across the company to see how we can help those teams leverage our experience and tool set and also learn from their experience in terms of some of the challenges that they face. That got me thinking more in terms of also our ASP.NET customers out there and how they go about testing their ASP.NET applications and server controls and about the challenges they face in ensuring the quality of their applications. So I thought of blogging around this topic to get some feedback from anyone that is facing similar challenges. I would love to hear from any customers out there around this topic. Mainly:
- What challenges you face testing your ASP.NET applications or server controls.
- What methodologies\tools you have used (if you are able to share of course) in that realm and what was your experience using them.
Looking forward to hearing from you.
The ASP.NET test team is in the middle of our ZBB Test Pass. ZBB stands for Zero Bug Bounce. The release team blog has a pretty good explanation of the drive to ZBB and its meaning. The blog though does not point out that most test teams in the division try to execute a full test pass pre ZBB to ensure that all open issues are discovered and addressed during that time period. Here are some high level points about the ZBB Test Pass that teams usually execute pre-ZBB and about the ASP.NET ZBB Test Pass that is currently in progress.
- Each milestone we typically have two test passes. One before ZBB and one right towards the end of the milestone to certify and verify the final bits that are going to customers. The first test pass' purpose is to make sure we identify any bugs/regressions right before ZBB so that hitting ZBB gives us more confidence in our schedule. (i.e. we have high confidence that we have discovered all major issues/defects in the product early on and there is low probability of finding last minutes problems that would impact the schedule).
- We typically run our full automated test bed. Our current automated test bed is around 356,079 scenarios and will be growing in the few weeks past our test pass as we wrap up our automation.
- The test pass takes around 3 weeks to complete. (Just looking at our current progress we might be able to finish earlier actually :))
- The tests are executed using our internal test execution system (aka MADDOG). The run executes on around 80 machines in our lab with 30 of them used as Tiers. Tiers are basically different server configurations that allow us to execute a testcase multiple times within a different environment. (i.e. Test_A will run against a server that is running ASP.NET Process Identity as Machine and then will run again on a different server configured with ASP.NET Process Identify running as System.)
Hope that gives you more insight into what the test teams are doing right around ZBB.
My name is Faris Sweis. I'm the test lead responsible for the quality assurance of the overall ASP.NET product. My goal in blogging is to try to give our community and customers a better insight and understanding of our efforts in ensuring a high quality ASP.NET feature set. In addition to that, this is an attempt to try to connect with as many Test/QA communities out there specific to ASP.NET or software testing in general. I will talk a bit more about software testing and my views on it in a later blog but for now let me introduce myself and the ASP.NET test team.
A bit about my background
I started my career at Microsoft right out of college after a one summer internship in 1997. I graduated in summer of 1998 from Purdue University and started in Microsoft's Developer Division (which I'm still in) working on VBA in Office 2000. My role was to test the VB6 language syntax which VBA6.0 just upgraded to and to test the "Darwinization" (as it was called then referring to the Darwin installer - known today as the MSI installer) of the VBA components in Office. After shipping Office 2000, I worked on the VS Macros IDE in VS and then later in 2000 moved to work on the .NET Framework team focusing on ASP.NET and the data controls. I've been working on ASP.NET ever since...
ASP.NET Test Team
The current test team consists of 23 engineers. Our team is organized in three sub teams.
- Page Framework Team: Responsible for testing all Page level features like MasterPages, SiteNavigation, Page ...etc
- Controls Team: Responsible for testing our Server Controls. From the LiteralControl to the GridView.
- Application Services Team: Responsible for testing all the ASP.NET infrastructure and base services like HttpModules, Caching, Membership, Roles...etc
We usually get involved in testing the feature areas from their initial design phase. We participate in the brainstorming and designs of all features. Once the design is complete, we move on and start the test planning phase for our features and start designing our test automation. I will be talking more about what this entails in future blogs.