Slides and Code from Tech·Ed 2010 session WEB01-INT
Tuesday, June 8, 2010 11:00 AM

Many thanks to everyone who attended this morning's "Web-Based Test Automation with LTAF" session at Tech·Ed 2010. As promised, I've attached the slides and source code to this 'blog post for download. I don't know if the session was recorded, but if I learn that it was, and if it is made public, I will update this 'blog post with the video's URL.

Cheers, Drew

What is New in ASP.NET 4.0 Code Access Security
Friday, April 23, 2010 3:37 PM

ASP.NET Code Access Security (CAS) is a feature that helps protect server applications on hosting multiple Web sites, ASP.NET lets you assign a configurable trust level that corresponds to a predefined set of permissions. ASP.NET has predefined ASP.NET Trust Levels and Policy Files that you can assign to applications, you also can assign custom trust level and policy files. Most web hosting companies run ASP.NET applications in Medium Trust to prevent that one website affect or harm another site etc.

As .NET Framework's Code Access Security model has evolved, ASP.NET 4.0 Code Access Security also has introduced several changes and improvements.

The main change in ASP.NET 4.0 CAS

In ASP.NET v4.0 partial trust applications, application domain can have a default partial trust permission set as opposed to being full-trust, the permission set name is defined in the <trust /> new attribute permissionSetName that is used to initialize the application domain . By default, the PermissionSetName attribute value is "ASP.Net" which is the name of the permission set you can find in all predefined partial trust configuration files.

<trust level="Something" permissionSetName="ASP.Net" />

This is ASP.NET 4.0 new CAS model. For compatibility ASP.NET 4.0 also support legacy CAS model where application domain still has full trust permission set.

You can specify new legacyCasModel attribute on the <trust /> element to indicate whether the legacy CAS model is enabled. By default legacyCasModel is false which means that new 4.0 CAS model is the default.

<trust level="Something" legacyCasModel="true|false"  />

In .Net FX 4.0 Config directory, there are two set of predefined partial trust config files for each new CAS model and legacy CAS model, trust config files with name legacy.XYZ.config are for legacy CAS model:

New CAS model: Legacy CAS model:
web_hightrust.config legacy.web_hightrust.config
web_mediumtrust.config legacy.web_mediumtrust.config
web_lowtrust.config legacy.web_lowtrust.config
web_minimaltrust.config legacy.web_minimaltrust.config

The figure below shows in ASP.NET 4.0 new CAS model what permission set to grant to code for partial trust application using predefined partial trust levels and policy files: 

Temp 

There also some benefits that comes with the new CAS model:

  • You can lock down a machine by making all managed code no-execute by default (e.g. setting the MyComputer zone to have no managed execution code permissions), it should still be possible to configure ASP.NET web applications to run as either full-trust or partial trust.
  • UNC share doesn’t require full trust with CASPOL at machine-level CAS policy.

Side effect that comes with the new CAS model:

  • processRequestInApplicationTrust attribute is deprecated  in new CAS model since application domain always has partial trust permission set in new CAS model.

In ASP.NET 4.0 legacy CAS model or ASP.NET 2.0 CAS model, even though you assign partial trust level to a application but the application domain still has full trust permission set.

The figure below shows in ASP.NET 4.0 legacy CAS model (or ASP.NET 2.0 CAS model) what permission set to grant to code for partial trust application using predefined partial trust levels and policy files:

Temp2 

What $AppDirUrl$, $CodeGen$, $Gac$ represents:

$AppDirUrl$

The application's virtual root directory. This allows permissions to be applied to code that is located in the application's bin directory.

For example, if a virtual directory is mapped to C:\YourWebApp, then $AppDirUrl$ would equate to C:\YourWebApp.

$CodeGen$

The directory that contains dynamically generated assemblies (for example, the result of .aspx page compiles). This can be configured on a per application basis and defaults to %windir%\Microsoft.NET\Framework\{version}\Temporary ASP.NET Files.

$CodeGen$ allows permissions to be applied to dynamically generated assemblies.

$Gac$

Any assembly that is installed in the computer's global assembly cache (GAC). This allows permissions to be granted to strong named assemblies loaded from the GAC by the Web application.

 

The new customization of CAS Policy in ASP.NET 4.0 new CAS model

1. Define which named permission set in partial trust configuration files

By default the permission set that will be assigned at application domain initialization time is the named "ASP.Net" permission set found in all predefined partial trust configuration files. However ASP.NET 4.0 allows you set PermissionSetName attribute to define which named permission set in a partial trust configuration file should be the one used to initialize an application domain.

Example: add "ASP.Net_2" named permission set in partial trust configuration file:

<PermissionSet
class="NamedPermissionSet"
version="1"
Name="ASP.Net_2">
<IPermission
class="FileIOPermission"
version="1"
Read="$AppDir$"
PathDiscovery="$AppDir$"
/>
<IPermission
class="ReflectionPermission"
version="1"
Flags ="RestrictedMemberAccess" />
<IPermission
class="SecurityPermission "
version="1"
Flags ="Execution, ControlThread, ControlPrincipal, RemotingConfiguration" />
</PermissionSet>

Then you can use "ASP.Net_2" named permission set for the application domain permission set:

<trust level="Something" legacyCasModel="false" permissionSetName="ASP.Net_2"  />


2. Define a custom set of Full Trust Assemblies for an application

By using the new fullTrustAssemblies element to configure a set of Full Trust Assemblies for an application, you can modify set of partial trust assemblies to full trust at the machine, site or application level.

The configuration definition is shown below:

<fullTrustAssemblies>
<add assemblyName="MyAssembly" version="1.1.2.3"
publicKey="hex_char_representation_of_key_blob" />
</fullTrustAssemblies>

3. Define <CodeGroup /> policy in partial trust configuration files

ASP.NET 4.0 new CAS model will retain the ability for developers to optionally define <CodeGroup />with membership conditions and assigned permission sets. The specific restriction in ASP.NET 4.0 new CAS model though will be that the results of evaluating custom policies can only result in one of two outcomes: either an assembly is granted full trust, or an assembly is granted the partial trust permission set currently associated with the running application domain. It will not be possible to use custom policies to create additional custom partial trust permission sets.

When parsing the partial trust configuration file:

  • Any assemblies that match to code groups associated with "PermissionSet='FullTrust'" will run at full trust.
  • Any assemblies that match to code groups associated with "PermissionSet='Nothing'" will result in a PolicyError being thrown from the CLR. This is acceptable since it provides administrators with a way to do a blanket-deny of managed code followed by selectively defining policy in a <CodeGroup /> that re-adds assemblies that would be allowed to run.
  • Any assemblies that match to code groups associated with other permissions sets will be interpreted to mean the assembly should run at the permission set of the appdomain. This means that even though syntactically a developer could define additional "flavors" of partial trust in an ASP.NET partial trust configuration file, those "flavors" will always be ignored.

Example: defines full trust in <CodeGroup /> for my strong named assemblies in partial trust config files:

<CodeGroup
class="FirstMatchCodeGroup"
version="1"
PermissionSetName="Nothing">
<IMembershipCondition
    class="AllMembershipCondition"
    version="1"
/>
<CodeGroup
    class="UnionCodeGroup"
    version="1"
    PermissionSetName="FullTrust"
    Name="My_Strong_Name"
    Description="This code group grants code signed full trust. ">
<IMembershipCondition
      class="StrongNameMembershipCondition"
version="1"
      PublicKeyBlob="hex_char_representation_of_key_blob"
/>
</CodeGroup>
<CodeGroup
   class="UnionCodeGroup"
version="1"
PermissionSetName="ASP.Net">
   <IMembershipCondition
class="UrlMembershipCondition"
version="1"
Url="$AppDirUrl$/*"
/>
</CodeGroup>
<CodeGroup
class="UnionCodeGroup"
version="1"
PermissionSetName="ASP.Net">
   <IMembershipCondition
class="UrlMembershipCondition"
version="1"
Url="$CodeGen$/*"
   />
</CodeGroup>
</CodeGroup>

4. Customize CAS policy at runtime in ASP.NET 4.0 new CAS model

ASP.NET 4.0 new CAS model allows to customize CAS policy at runtime by using custom HostSecurityPolicyResolver that overrides the ASP.NET code access security policy.

Example: use custom host security policy resolver to resolve partial trust web application bin folder MyTrustedAssembly.dll to full trust at runtime:

You can create a custom host security policy resolver and compile it to assembly MyCustomResolver.dll with strong name enabled and deploy in GAC:

public class MyCustomResolver : HostSecurityPolicyResolver
{
public override HostSecurityPolicyResults ResolvePolicy(Evidence evidence)
{
IEnumerator hostEvidence = evidence.GetHostEnumerator();

while (hostEvidence.MoveNext())
{
object hostEvidenceObject = hostEvidence.Current;

if (hostEvidenceObject is System.Security.Policy.Url)
{
string assemblyName = hostEvidenceObject.ToString();
if (assemblyName.Contains(“MyTrustedAssembly.dll”)
return HostSecurityPolicyResult.FullTrust;
}
}

//default fall-through
return HostSecurityPolicyResult.DefaultPolicy;
}
}

Because ASP.NET accesses the custom HostSecurityPolicyResolver during application domain initialization, and a custom policy resolver requires full trust, you also can add a custom policy resolver in <fullTrustAssemblies /> , or deploy in the GAC.

You also need configure a custom HostSecurityPolicyResolver instance by adding the HostSecurityPolicyResolverType attribute in the <trust /> element:

<trust level="Something"  
legacyCasModel="false"
hostSecurityPolicyResolverType="MyCustomResolver, MyCustomResolver"
permissionSetName="ASP.Net"
/>
 

Note:

If an assembly policy define in <CodeGroup/> and also in hostSecurityPolicyResolverType, hostSecurityPolicyResolverType will win.

If an assembly added in <fullTrustAssemblies/> then the assembly has full trust no matter what policy in <CodeGroup/> or in hostSecurityPolicyResolverType.

 

Other changes in ASP.NET 4.0 CAS

Use the new transparency model introduced in .Net Framework 4.0

  • Change in dynamically compiled code generated assemblies by ASP.NET:

In new CAS model they will be marked as security transparent level2 to use Framework 4.0 security transparent rule that means partial trust code is treated as completely Transparent and it is more strict enforcement.

In legacy CAS model they will be marked as security transparent level1 to use Framework 2.0 security transparent rule for compatibility.

You also can look at Security Changes in the .NET Framework 4 for more information about these security attributes.

Support conditional APTCA

  • If an assembly is marked with the Conditional APTCA attribute to allow partially trusted callers, and if you want to make the assembly both visible and accessible to partial-trust code in your web application, you must add a reference to the assembly in the partialTrustVisibleAssemblies section:
<partialTrustVisibleAssemblies>
<add assemblyName="MyAssembly"
publicKey="hex_char_representation_of_key_blob" />
/partialTrustVisibleAssemblies>
 
  • Most of ASP.NET products runtime assemblies are also changed to be marked as conditional APTCA to prevent use of ASP.NET APIs in partial trust environments such as Winforms or WPF UI controls hosted in Internet Explorer.

 

Differences between ASP.NET new CAS model and legacy CAS model:

Here list some differences between ASP.NET new CAS model and legacy CAS model

ASP.NET 4.0 legacy CAS model:

  • Asp.net partial trust appdomains have full trust permission
  • Multiple different permission sets in a single appdomain are allowed in ASP.NET partial trust configuration files Code groups
  • Machine CAS policy is honored
  • New configuration setting for legacy model:

<trust level="Something"
legacyCASModel="true" >
</trust>

<partialTrustVisibleAssemblies>
<add assemblyName="MyAssembly"
publicKey="hex_char_representation_of_key_blob" />
</partialTrustVisibleAssemblies>

ASP.NET 4.0 new CAS model:

  • ASP.NET will now run in homogeneous application domains. Only full trust or the app-domain's partial trust grant set, are allowable permission sets.
  • It is no longer possible to define arbitrary permission sets that get assigned to different assemblies. If an application currently depends on fine-tuning the partial trust permission set using the ASP.NET partial trust configuration file, this will no longer be possible.
  • Dynamically compiled assemblies output by ASP.NET build providers will be updated to explicitly mark assemblies as transparent.
  • ASP.NET partial trust grant sets will be independent from any enterprise, machine, or user CAS policy levels.
  • A simplified model for locking down web servers that only allows trusted managed web applications to run.
    • Machine policy used to always grant full-trust to managed code (based on membership conditions) can instead be configured using the new ASP.NET 4.0 full-trust assembly configuration section. The full-trust assembly configuration section requires explicitly listing each assembly as opposed to using membership conditions.
    • Alternatively, the membership condition(s) used in machine policy can instead be re-defined in a <CodeGroup /> within ASP.NET's partial trust configuration file to grant full-trust.
  • New configuration setting for new model:
<trust level="Something" 
legacyCASModel="false"
permissionSetName="ASP.Net"
hostSecurityPolicyResolverType=".NET type string" >
</trust>


<fullTrustAssemblies>
<add assemblyName=”MyAssemblyversion=”1.0.0.0
publicKey="hex_char_representation_of_key_blob" />
</fullTrustAssemblies>

<partialTrustVisibleAssemblies>
<add assemblyName="MyAssembly"
publicKey="hex_char_representation_of_key_blob" />
</partialTrustVisibleAssemblies>

 
 

Hope this post is helpful to better understand the ASP.Net 4.0 CAS.

Xiaohong Tang
ASP.NET QA Team

News from the QA team
Sunday, September 13, 2009 11:40 PM

I know there has been very little traffic on our team blog, but I recently decided to start my own blog so that I could share my personal opinions. However, here is a small news update from the ASP.NET QA team:

  • First and fore most: the ASP.NET QA team is hiring! If you are interested, check out our job posting here. You can also read about it in Matthew Osborn’s blog here.
  • The weekly CodingQA podcast is alive and kicking. Check it out, and if you like it you can subscribe using iTunes and Zune.
  • Jim Wang, the lead tester for ASP.NE Ajax, posted a nice tutorial using ASP.NET Ajax Preview 5 here.
  • Federico Silva Armas (me :P) is presenting at DevConnections this year. Check this post for more information.
  • Last but not least, Tales of Testing is my personal blog where I cover everything that has to do with testing. If you are into testing and want to share your ideas I am interested in listening.

 

Federico Silva Armas
SDET, ASP.NET QA Team

CodingQA Podcast Episodes 10-12
Friday, July 3, 2009 10:01 AM

Episode 10 Project Structure

In this show Matthew and Federico talk about how the ASP.NET team structures it’s sources and the theory behind it all. They talk about how the team handles the organization of all of it’s sources and what seems to work well along with what seems to cause hang ups.

Episode 11 Interviewing Hong Li

In the eleventh installment of Coding QA Federico is on vacation so Matthew sits down with his mentor, Hong Li, for an interview. Join Matthew as he learns a little about one of his fellow team members, servicing an released product, and working with team members that are offshore in China.

Episode 12 Jim Wang on Microsoft AJAX

In the twelfth installment of Coding QA Federico is still enjoying his vacation so Matthew sits down with his officemate Jim Wang to discuss the ins and outs of test the Microsoft AJAX libraries. Tune in as Matthew picks Jim’s brain on what it was like to test a client side framework.

 

Federico Silva Armas
SDET, ASP.NET QA Team

by farmas | with no comments
Filed under: ,
How to write a custom log in LTAF June release?
Wednesday, June 10, 2009 3:35 PM

If you want to write a custom log while running your test, the June 2009 release of LTAF now provides an easier way to set that up. The DriverPage gives access to a TestcaseExecutor object that exposes several lifecycle events

Here is a code snippet that shows how to write your own log (MyCustomLog.txt) by attaching to the proper events to gather information while the tests are executing.

private StringBuilder _log;
private string _logPath;

protected override void RunTestCases()
{
    // if user checked the Write Log To Disk checkbox
    if (this.WriteLogToDiskCheckBox.Checked)
    {
        // attach event handlers to write my custom log
        this.TestcaseExecutor.TestcaseExecuted += 
            new EventHandler<TestcaseExecutionEventArgs>(TestcaseExecutor_TestcaseExecuted);
        
        this.TestcaseExecutor.TestRunFinished += 
            new EventHandler<TestRunFinishedEventArgs>(TestcaseExecutor_TestRunFinished);

        this.TestcaseExecutor.TestRunStarted += new EventHandler(TestcaseExecutor_TestRunStarted);

        _logPath = System.IO.Path.Combine(Server.MapPath(""), "MyCustomLog.txt");
        _log = new StringBuilder();
    }
    
    base.RunTestCases();
}

void TestcaseExecutor_TestRunStarted(object sender, EventArgs e)
{
    _log.AppendLine("Test Run Started.");
}

void TestcaseExecutor_TestcaseExecuted(object sender, TestcaseExecutionEventArgs e)
{
    _log.AppendLine(String.Format(
        "Finished running test '{0}'. Status='{1}'", 
        e.WebTestName, 
        e.Passed));
}

void TestcaseExecutor_TestRunFinished(object sender, TestRunFinishedEventArgs e)
{   
    _log.AppendLine("Test Run Finished.");
    
    // make sure that you have write access to the test folder
    System.IO.File.WriteAllText(_logPath, _log.ToString());
}

The "MyCustomLog.txt" file will be written into the same folder where the DriverPage.aspx exists.

Federico Silva Armas
SDET, ASP.NET QA Team

by farmas | 1 comment(s)
Filed under: ,
What's new in LTAF June release?
Wednesday, June 10, 2009 3:34 PM

The June 2009 release of the Lightweight Test Automation Framework is out on CodePlex. Here is a small write up of the new features available.

Automation of frames

Now tests can automate pages that contain framesets and iFrames. For example:

[WebTestMethod]
    public void Frame()
    {
        HtmlPage page = new HtmlPage("TestFrameSet.htm");

        // get top frame
        HtmlPage frame = page.GetFramePage("topFrame");

        // verify title of frame
        Assert.AreEqual("This is the top frame", frame.Elements.Find("h1", 0).GetInnerText());
    }

 

Write to the test console

This is an experimental feature that is useful to debug tests. You can now write messages from your test into the test console that appears in the driver page. For example:

[WebTestMethod]
    public void PopupPageWithIndex()
    {
        // Navigate to the frameset
        HtmlPage page = new HtmlPage("TestFrameSet.htm");

        //get the top frame
        WebTestConsole.Write("Get the top frame");
        HtmlPage frame = page.GetFramePage("topFrame");

        // click on the button that opens a new page
        WebTestConsole.Write("Click the openwindow button");
        frame.Elements.Find("OpenWindow").Click();

    ...

When run, this test will output traces into the console UI:

image

 

Hide the test console

New option in the UI allows user to hide the test console. This is useful for long running test suites, to prevent the browser from maintain a grown list of commands.

image

 

Quickly verify page is rendering an asp.net error

New helper method in HtmlPage makes it easy to quickly verify if a asp.net error page is being rendered. This is useful to immediately stop the test if an error has occurred, instead of waiting for a timeout. For example:

[WebTestMethod]
    public void VerifyTheAspNetErrorPage()
    {
        HtmlPage p = new HtmlPage("PageWithError.aspx");
        bool pageFail = p.IsServerError();
        Assert.IsFalse(pageFail);

        ...
    }

 

Exposed test run lifecycle events

Users can now register handlers in the DriverPage.aspx to several life cycle events. This is useful, for example, to generate a custom log:

protected override void RunTestCases()
   {
       // if user checked the Write Log To Disk checkbox
       if (this.WriteLogToDiskCheckBox.Checked)
       {
           // attach event handlers to write my custom log
           this.TestcaseExecutor.TestcaseExecuted += 
               new EventHandler<TestcaseExecutionEventArgs>(TestcaseExecutor_TestcaseExecuted);
           
           this.TestcaseExecutor.TestRunFinished += 
               new EventHandler<TestRunFinishedEventArgs>(TestcaseExecutor_TestRunFinished);    
       }
       
       base.RunTestCases();
   }

 

Additional Resources

 

Federico Silva Armas
SDET, ASP.NET QA Team

by farmas | 3 comment(s)
Filed under: ,
CodingQA Podcast Episode 9
Saturday, June 6, 2009 12:09 PM

Download: Episode 9
Streaming: Episode 9

In this show Matthew and Federico talk about what the ASP.NET QA team does to prepare for a release. Taking into account the recent release of ASP.NET 4 Beta 1, they’ll go over several of the "exit criteria" that the product must meet before delivery to the public.

News

  • Lightweight Test Automation Framework June Release

Sign off on a release

  • Difference between Quality Gates and Exit Criteria
  • Accessibility. Mainly manual tests to validate feature conforms to accessibility standards.
  • Code Coverage. Metric used to identify problematic development areas.
  • Stress. Specialized automated long running tests to spot memory leaks and other stress problems.
  • Localization. Verification that the product can be correctly localized to several languages.
  • Test Passes. Functional verification of the product across many platform combinations.
  • Zero Active Bugs. Series of step down goals to drive feature crews to finish development.
by farmas | 1 comment(s)
Filed under: ,
CodingQA Podcast Episode 8
Saturday, June 6, 2009 12:06 PM

Download: Episode 8
Streaming: Episode 8

This show is an open discussion about what testing means. Federico and Matthew share their opinions around the meaning of "testing", "quality" and the role of a QA team. They don't always agree but each brings something unique to the table on topics so basic yet somewhat mysterious in the testing discipline.

News

Topics of Discussion

  • What is the role of QA?
  • What is testing?
  • What is quality?
  • Should quality be measurable?
  • What is risk?
  • What is the difference between Dev and Test?
by farmas | 3 comment(s)
Filed under: ,
CodingQA Podcast Episode 7
Monday, May 25, 2009 11:28 PM

Download: Episode 7
Streaming: Episode 7

In this show Federico and Matthew talk about what exploratory testing (ET) is and how the ASP.NET QA team uses this methodology for testing.

News

Exploratory Testing

  • Definition
  • Differences between exploratory testing and scripted testing
  • Differences between exploratory testing and ad-hoc testing
  • The 4 principles of exploratory testing
    • Creativity
    • Goal oriented
    • Reproducibility
    • Adaptability
  • How the ASP.NET QA team uses exploratory testing
  • When is exploratory testing a good fit to test a product
  • Brief example of an exploratory testing session
  • What are the outcomes of an exploratory testing session
    • Scenarios tried
    • Questions
    • Assumptions
    • Defects
    • Future scenarios
by farmas | 72 comment(s)
Filed under: ,
CodingQA Podcast Episode 6
Thursday, May 21, 2009 11:29 PM

Download: Episode 6
Streaming: Episode 6

In this show Federico and Matthew explore the question of "when too much automation is too much?" Based on their experiences in the ASP.NET QA team, they talk about the danger of automating for the sake of automation, the problems that it can create, and the value of context driven testing.

News

  • New name and name website: Coding QA.
  • .NET Micro Framework has been open sourced.

Automation for the sake of automation

  • What is the goal of automation? To find bugs during development vs. to find regressions after development.
  • What problems can arise from having too much automation? The tale of the ASP.NET QA "boat anchor".
  • Context matters when deciding how to ensure a feature does not regress.
  • What parameters can be used to decide the testing approach
    • Automation cost.
    • Time of delivery.
    • Resources available.
    • Risk of a regression.
    • Customer impact if a regression is introduced.
    • Quickness between when the regression is introduced and when it is discovered.
  • Real examples from the team
    • Testing event handler generation for web controls in VS.
    • Testing picker dialogs for web controls in VS.

 

Federico Silva Armas
SDET, ASP.NET QA Team

by farmas | 1 comment(s)
Filed under: ,
More Posts Next page »