Asp.Net QA Team

News from the QA team

2009-09-14T03:40:48Z

I know there has been very little traffic on our team blog, but I recently decided to start my own blog so that I could share my personal opinions. However, here is a small news update from the ASP.NET QA team:

First and fore most: the ASP.NET QA team is hiring! If you are interested, check out our job posting here. You can also read about it in Matthew Osborn’s blog here.
The weekly CodingQA podcast is alive and kicking. Check it out, and if you like it you can subscribe using iTunes and Zune.
Jim Wang, the lead tester for ASP.NE Ajax, posted a nice tutorial using ASP.NET Ajax Preview 5 here.
Federico Silva Armas (me :P) is presenting at DevConnections this year. Check this post for more information.
Last but not least, Tales of Testing is my personal blog where I cover everything that has to do with testing. If you are into testing and want to share your ideas I am interested in listening.

Federico Silva Armas
SDET, ASP.NET QA Team

CodingQA Podcast Episodes 10-12

2009-07-03T14:01:02Z

Episode 10 Project Structure

In this show Matthew and Federico talk about how the ASP.NET team structures it’s sources and the theory behind it all. They talk about how the team handles the organization of all of it’s sources and what seems to work well along with what seems to cause hang ups.

Episode 11 Interviewing Hong Li

In the eleventh installment of Coding QA Federico is on vacation so Matthew sits down with his mentor, Hong Li, for an interview. Join Matthew as he learns a little about one of his fellow team members, servicing an released product, and working with team members that are offshore in China.

Episode 12 Jim Wang on Microsoft AJAX

In the twelfth installment of Coding QA Federico is still enjoying his vacation so Matthew sits down with his officemate Jim Wang to discuss the ins and outs of test the Microsoft AJAX libraries. Tune in as Matthew picks Jim’s brain on what it was like to test a client side framework.

Federico Silva Armas
SDET, ASP.NET QA Team

How to write a custom log in LTAF June release?

2009-06-10T19:35:12Z

If you want to write a custom log while running your test, the June 2009 release of LTAF now provides an easier way to set that up. The DriverPage gives access to a TestcaseExecutor object that exposes several lifecycle events

Here is a code snippet that shows how to write your own log (MyCustomLog.txt) by attaching to the proper events to gather information while the tests are executing.

private StringBuilder _log;
private string _logPath;

protected override void RunTestCases()
{
    // if user checked the Write Log To Disk checkbox
    if (this.WriteLogToDiskCheckBox.Checked)
    {
        // attach event handlers to write my custom log
        this.TestcaseExecutor.TestcaseExecuted += 
            new EventHandler<TestcaseExecutionEventArgs>(TestcaseExecutor_TestcaseExecuted);
        
        this.TestcaseExecutor.TestRunFinished += 
            new EventHandler<TestRunFinishedEventArgs>(TestcaseExecutor_TestRunFinished);

        this.TestcaseExecutor.TestRunStarted += new EventHandler(TestcaseExecutor_TestRunStarted);

        _logPath = System.IO.Path.Combine(Server.MapPath(""), "MyCustomLog.txt");
        _log = new StringBuilder();
    }
    
    base.RunTestCases();
}

void TestcaseExecutor_TestRunStarted(object sender, EventArgs e)
{
    _log.AppendLine("Test Run Started.");
}

void TestcaseExecutor_TestcaseExecuted(object sender, TestcaseExecutionEventArgs e)
{
    _log.AppendLine(String.Format(
        "Finished running test '{0}'. Status='{1}'", 
        e.WebTestName, 
        e.Passed));
}

void TestcaseExecutor_TestRunFinished(object sender, TestRunFinishedEventArgs e)
{   
    _log.AppendLine("Test Run Finished.");
    
    // make sure that you have write access to the test folder
    System.IO.File.WriteAllText(_logPath, _log.ToString());
}

The "MyCustomLog.txt" file will be written into the same folder where the DriverPage.aspx exists.

Federico Silva Armas
SDET, ASP.NET QA Team

What's new in LTAF June release?

2009-06-10T19:34:06Z

The June 2009 release of the Lightweight Test Automation Framework is out on CodePlex. Here is a small write up of the new features available.

Automation of frames

Now tests can automate pages that contain framesets and iFrames. For example:

[WebTestMethod]
    public void Frame()
    {
        HtmlPage page = new HtmlPage("TestFrameSet.htm");

        // get top frame
        HtmlPage frame = page.GetFramePage("topFrame");

        // verify title of frame
        Assert.AreEqual("This is the top frame", frame.Elements.Find("h1", 0).GetInnerText());
    }

Write to the test console

This is an experimental feature that is useful to debug tests. You can now write messages from your test into the test console that appears in the driver page. For example:

[WebTestMethod]
    public void PopupPageWithIndex()
    {
        // Navigate to the frameset
        HtmlPage page = new HtmlPage("TestFrameSet.htm");

        //get the top frame
        WebTestConsole.Write("Get the top frame");
        HtmlPage frame = page.GetFramePage("topFrame");

        // click on the button that opens a new page
        WebTestConsole.Write("Click the openwindow button");
        frame.Elements.Find("OpenWindow").Click();

    ...

When run, this test will output traces into the console UI:

Hide the test console

New option in the UI allows user to hide the test console. This is useful for long running test suites, to prevent the browser from maintain a grown list of commands.

Quickly verify page is rendering an asp.net error

New helper method in HtmlPage makes it easy to quickly verify if a asp.net error page is being rendered. This is useful to immediately stop the test if an error has occurred, instead of waiting for a timeout. For example:

[WebTestMethod]
    public void VerifyTheAspNetErrorPage()
    {
        HtmlPage p = new HtmlPage("PageWithError.aspx");
        bool pageFail = p.IsServerError();
        Assert.IsFalse(pageFail);

        ...
    }

Exposed test run lifecycle events

Users can now register handlers in the DriverPage.aspx to several life cycle events. This is useful, for example, to generate a custom log:

protected override void RunTestCases()
   {
       // if user checked the Write Log To Disk checkbox
       if (this.WriteLogToDiskCheckBox.Checked)
       {
           // attach event handlers to write my custom log
           this.TestcaseExecutor.TestcaseExecuted += 
               new EventHandler<TestcaseExecutionEventArgs>(TestcaseExecutor_TestcaseExecuted);
           
           this.TestcaseExecutor.TestRunFinished += 
               new EventHandler<TestRunFinishedEventArgs>(TestcaseExecutor_TestRunFinished);    
       }
       
       base.RunTestCases();
   }

Additional Resources

Visit our forum for any questions and feedback
List of known issues
Release history

Federico Silva Armas
SDET, ASP.NET QA Team

CodingQA Podcast Episode 9

2009-06-06T16:09:38Z

Download: Episode 9
Streaming: Episode 9

In this show Matthew and Federico talk about what the ASP.NET QA team does to prepare for a release. Taking into account the recent release of ASP.NET 4 Beta 1, they’ll go over several of the "exit criteria" that the product must meet before delivery to the public.

News

Lightweight Test Automation Framework June Release

Sign off on a release

Difference between Quality Gates and Exit Criteria
Accessibility. Mainly manual tests to validate feature conforms to accessibility standards.
Code Coverage. Metric used to identify problematic development areas.
Stress. Specialized automated long running tests to spot memory leaks and other stress problems.
Localization. Verification that the product can be correctly localized to several languages.
Test Passes. Functional verification of the product across many platform combinations.
Zero Active Bugs. Series of step down goals to drive feature crews to finish development.

CodingQA Podcast Episode 8

2009-06-06T16:06:30Z

Download: Episode 8
Streaming: Episode 8

This show is an open discussion about what testing means. Federico and Matthew share their opinions around the meaning of "testing", "quality" and the role of a QA team. They don't always agree but each brings something unique to the table on topics so basic yet somewhat mysterious in the testing discipline.

News

Topics of Discussion

What is the role of QA?
What is testing?
What is quality?
Should quality be measurable?
What is risk?
What is the difference between Dev and Test?

CodingQA Podcast Episode 7

2009-05-26T03:28:46Z

Download: Episode 7
Streaming: Episode 7

In this show Federico and Matthew talk about what exploratory testing (ET) is and how the ASP.NET QA team uses this methodology for testing.

News

Exploratory Testing

Definition
Differences between exploratory testing and scripted testing
Differences between exploratory testing and ad-hoc testing
The 4 principles of exploratory testing
- Creativity
- Goal oriented
- Reproducibility
- Adaptability
How the ASP.NET QA team uses exploratory testing
When is exploratory testing a good fit to test a product
Brief example of an exploratory testing session
What are the outcomes of an exploratory testing session
- Scenarios tried
- Questions
- Assumptions
- Defects
- Future scenarios

CodingQA Podcast Episode 6

2009-05-22T03:29:40Z

Download: Episode 6
Streaming: Episode 6

In this show Federico and Matthew explore the question of "when too much automation is too much?" Based on their experiences in the ASP.NET QA team, they talk about the danger of automating for the sake of automation, the problems that it can create, and the value of context driven testing.

News

New name and name website: Coding QA.
.NET Micro Framework has been open sourced.

Automation for the sake of automation

What is the goal of automation? To find bugs during development vs. to find regressions after development.
What problems can arise from having too much automation? The tale of the ASP.NET QA "boat anchor".
Context matters when deciding how to ensure a feature does not regress.
What parameters can be used to decide the testing approach
- Automation cost.
- Time of delivery.
- Resources available.
- Risk of a regression.
- Customer impact if a regression is introduced.
- Quickness between when the regression is introduced and when it is discovered.
Real examples from the team
- Testing event handler generation for web controls in VS.
- Testing picker dialogs for web controls in VS.

Federico Silva Armas
SDET, ASP.NET QA Team

CodingQA Podcast - Episode 5

2009-05-17T05:19:08Z

Download: Episode 5
Streaming: Episode 5

In this episode Matthew interviews Carl Dacosta, lead tester for the MVC 1.0 project, and talks about what it was like to develop and test a product with so many releases.

MVC crossed the 100,000 downloads mark (currently at 140,000)
MVC released under MS-PL license
Interview with Carl Dacosta
- Some history on Carl and how he got into test.
- Role of Carl as a tester in the MVC crew.
- What was the development and testing process of MVC to enable frequent releases.
- Parts of the testing process that worked well for this kind of project, and parts that didn't work so well.
- How to handle quality of releases when external partners are starting to rely on the framework.
- What did Carl enjoy the most working on the MVC crew.
- Testing the "unit testing" experience for developers working on MVC.

Federico Silva Armas
SDET, ASP.NET QA Team

CodingQA Podcast Feed

2009-05-17T05:17:26Z

We now have a proper feed for our podcast, head over to codingqa.com to subscribe. You can also find us in iTunes and very soon in Zune Marketplace as well.

For the time being I’ll keep posting here whenever an episode comes out. .

Federico Silva Armas
SDET, ASP.NE QA Team

Web Accessibility Standards

2009-04-29T20:29:55Z

Due to some test activities that I have been engaged on lately I have had to educate myself in regards to web standards. One of our PM's sent me this link that I want to share out with any testers out there that are not aware of it. This is a document found within the Massachusetts government website, and I am not sure if it is somehow special for that state (it's from 2005, so a little old). Even though it is not a replacement for Section 508, I liked the way it is presented because for each standard it explains "why" the standard is needed and "what' it means with examples. I include the list of standards that are defined:

1. Layout and Design

Standard 1.1: A state agency web page must use HTML markup tags according to industry standards.
Standard 1.2: A state agency web page should control presentation with style sheets, but the web pages must be organized so they are readable without requiring an associated style sheet.
Standard 1.3: A state agency web page should avoid using frames.
Standard 1.4: State agencies must ensure that their web pages are usable when scripts, applets, or other programmed objects are turned off or are not supported.
Standard 1.5: Forms designed to be completed online, and other interactive interface elements, must be accessible by people using assistive technology.
Standard 1.6: When a timed response is required, the user must be provided the opportunity to request additional time to continue working before the time-out occurs.

2. Navigation

Standard 2.1: A state agency web page must use clear and consistent navigation mechanisms.
Standard 2.2: A state agency must ensure that users are able to interact with web page elements in a device independent manner.

3. Graphics and Sound

Standard 3.1: A state agencyweb page must provide a text equivalent for every non-text element.
Standard 3.2: A state agency web page must provide synchronized auditory and readable text descriptions of the important information of the visual track of a multimedia presentation.
Standard 3.3: A state agency web page must provide a text equivalent for information provided in audio format.
Standard 3.4: A state agency web page that uses motion must ensure that the motion is integral to the content of the site, user-controlled, and limited to three cycles and then stopped automatically.
Standard 3.5: A state agency must ensure that the use and selection of color do not affect the information conveyed on a page.
Standard 3.6: Client-side image maps are recommended. If server-side image maps must be used, provide redundant text links for each active region.

4. Content Requiring Additional Software

Standard 4.1: All information published on a state agency web page must be published in HTML, whenever possible, to eliminate the need for additional software.
Standard 4.2: A state agency that offers files to download in compressed format (Zip files, for example) must also provide the same information in its uncompressed format or as a self extracting file.

5. File Size

Standard 5.1: A state agency must optimize files to improve download time.

6. Web Accessibility Statement

Standard 6.1: A state agency web site must provide a link to a web accessibility statement.

7. Web Site Validation

Standard 7.1: A state agency must validate web content against these Standards prior to posting and at regular intervals after posting.

Federico Silva Armas
SDET, ASP.NET QA Team

ASP.NET QA Podcast – Episode 4

2009-04-13T21:54:00Z

Download : Episode 4

In the fourth installment of the ASP.NET QA Podcast Federico and Matthew discuss “The Good,” “The Bad,” and “The Ugly” of the history of the ASP.NET QA Team. Discussion ranges from the dark ages, to the renaissance, to the industrial revolution, to the “21st and half century”. Join them as they discuss the lessons learned and how the team has improved over time.

The team released the April update to the Lightweight Test Automation Framework.
“The Dark Ages” ASP.NET 1.0/2.0
- During this time the team was very focused on automating everything.
- (disadvantage) This approach is not good with a changing Spec.
- (disadvantage) Bugs were being found to late in the process.
- (advantage) Excellent automation coverage.
“The Renaissance” Atlas/Microsoft Ajax
- The division moved to a feature crew model for development and QA.
- (advantage) The team adopted a heavy unit testing practice. Increase in code coverage.
- (disadvantage) The QA team still worked has it had before. It would automate everything is less time.
“The Industrial Revolution” ASP.NET 3.5
- The QA team began to adopt the Feature crew model and started the process of agile testing with exploratory testing.
- (advantage) Bugs were being found at the beginning of the process.
- (disadvantage) The team still wrote a large amount of automation.
“21st and half century” ASP.NET 3.5 SP1/ASP.NET 4.0/vNext
- The team has really started moving away from automating everything and is spending more time investigating user scenarios.
- The team has formed its own unique blend of agile, scrum, and any other buzz word project management theory.
- (advantage) Creating and automating sample applications provides feedback for real world customer scenarios.
- (disadvantage) The team is young and is still learning the best practices.
- (Struggle) Costing has become less of a clear cut process.
- (Struggle) The is still a barrier between the QA and Dev teams.
- (Struggle) No one wants the blame when a bug is found.

Links from the show :

Lightweight Test Automation Framework April Release

2009-04-09T19:07:00Z

The April release of the Lightweight Test Automation Framework for ASP.NET has just been posted and you can download it here. For this release, the team has worked hard to include the following bug fixes and new features. Please continue to give us your feedback as many of the fixes and features are based what we have heard from the community.

Improvements to the user interface

A new look has been given to the test name when it passes or fails. There is both color and visual queues that indicate weather a test has passed or failed. Failed test names also appear slightly larger to help them stand out.
There is now a “Run Failed Tests” button. This button will open a new browser window that will select and run only the failed tests, for easy verification of fixes.

The ability to automate popup windows

In previous versions of the framework there was no way to verify the contents of a popup window. With this release we have far better support for opening and verifying the contents of popup windows.

HtmlPage page = new HtmlPage("MyPage.aspx");
page.Elements.Find("OpenPopup").Click();
// get popup window
HtmlPage popup = page.GetPopupWindow(1);
// verify title of popup
Assert.AreEqual("This is the Popup Page", popup.Elements.Find("h1", 0).GetInnerText());

The GetPopupWindow method returns a HtmlPage object that is representative of the window at that index. This is pulled from the collection that is maintained by the framework with index zero being the main, or starting, window. In this example index zero is MyPage.aspx, while index one is the popup window.

The ability to find elements by partial attribute values

In previous versions of the framework when finding elements on a page you could only use the ID attribute to match against. So if you wanted to match all the elements that had a CSS class applied to them, as is common in jQuery, you had to supply that whole value for the class attribute. So, in previous versions of the framework if you wanted to find elements that had a CSS class applied to them you had to specify the whole value for the class attribute. Meaning that if the element has more than only class and you search based on only one class that element would not be returned. Now you have the ability to match based on any part of the value for the attribute. Below is an example of how to find elements that have the CSS Class “blue” applied to them.

HtmlElementFindParams find = new HtmlElementFindParams();
find.Attributes.Add("class", "blue", MatchMethod.Contains);
ReadOnlyCollection<HtmlElement> elements = page.Elements.FindAll(find);

Assembly name change

The assembly name has been changed from “Microsoft.Web.Testing.Light.dll” to “Microsoft.Web.Testing.Lightweight.dll” The namespaces have not changed.

ASP.NET QA Podcast – Episode 3

2009-03-31T23:07:00Z

Download : Episode 3

In the third installment of the ASP.NET QA Podcast Federico and Matthew interview Mark Berryman, a test lead on the ASP.NET QA Team. Join them as they learn the ins and outs of test management through Mark’s eyes.

Release of the Mobile Browser Database by Live Dublin.
April refresh of the Lightweight Test Automation Framework.
Interview with Mark Berryman.
- What is the difference between Test Manager and Test Lead.
- A little history of Mark and how he came to join the ASP.NET QA team.
- Responsibilities of a Test Lead / What Mark is working on.
- What Mark looks for in a tester on his team.
- Challenges Mark faced when he was an IC (individual Contributor).
- What Mark would change on items he has tested in the past.
- How Mark Manages risk on his team.
- Who is Marks favorite person in the whole world.

Links from the show :

ASP.NET QA Podcast – Episode 2

2009-03-20T18:51:00Z

Download : Episode 2

In the second installment of the ASP.NET QA Podcast Federico and Matthew discuss MIX 09 announcements and the ins and outs of supporting various browser versions on the ASP.NET QA team.

MIX 09 Conference announcements
- MVC RTM 1.0
- Silverlight 3.0
- Web Application Installer
- Web Application Gallery
What browsers the team tests on and how they decided which browsers to test.
What it means when the team says they support a browser.
How the team manages the risk of not testing on every possible browser.
The timeline for testing features on multiple browsers.
The infrastructure the team has to test on multiple browsers.
How the team keeps up with the fast growing browser market, such as the release of four major browsers in just a few months.

Links from the show :