Unit Testing with Silverlight

One of the important capabilities we shipped with the Beta1 release of Silverlight 2 was a unit test harness that enables you to perform both API-level and UI-level unit testing.  This testing harness is cross browser and cross platform, and can be used to quickly run and verify automated unit tests:

In addition to shipping this unit test harness for Silverlight, we also shipped the source to ~2,000 unit tests built with it that provide automated coverage for the Silverlight control source that we also shipped under a permissive license (you can take the control source, modify it, run the unit tests to verify the behavior, then re-ship the controls however you want).

Learning How to Unit Test Silverlight

Jeff Wilcox (who developed the Silverlight unit test framework and harness) has a great blog post that talks about how to add a Silverlight Unit Test project to a solution here. You can download the chat application that he shows testing from this expression blend blog post tutorial I did last month.  You can also watch this cool video post that Jeff created where he walks through the unit test framework and test cases we've shipped.

As Jeff shows in his post, you can now add a "Silverlight Test Project" to your Visual Studio solution which encapsulates unit tests for an application you are working on:

You can then add unit test classes to the test project that test APIs or simulate UI action within the Silverlight controls (simulate button clicks, etc).

You can then run the test project and execute the tests within it to verify and report their status.

Jeff's test framework automatically provides a browser based test harness and reporting system (which means you can run it on any browser/OS combination that Silverlight runs on):

Jeff's test framework supports quickly re-setting controls after each test (and avoids needing to re-launch a new browser instance for each test cases - which makes it really fast).

You can quickly rip through hundreds or thousands of automated tests in seconds:

Green results mean the tests passed.  Red results flag that a test case failed and log the assertion failure and/or runtime exceptions that occurred.

Summary

If you've ever struggled to try and come up with a strategy for doing automated unit testing or TDD with AJAX applications, I think you'll find Silverlight provides some much nicer test options.  Using Visual Studio you can also separate your tests into a separate project in your solution, and you do not need to embed the tests within your Silverlight application in order for them to run.

In addition to supporting the above unit test harness and framework, we are also going to support UI automation APIs with the final release of Silverlight 2.  These will enable accessibility scenarios (allowing screen readers to work with Silverlight and enable Section 508 compliance of Silverlight applications).  These UI automation APIs will also enable UI testing scenarios where you can build end to end browser UI automation that simulates real mouse and keyboard interactions and enables automated end to end experience testing.  The combination should enable you to build much more solid and maintainable RIA solutions.

Hope this helps,

Scott

P.S. For more tutorial posts and links on Silverlight 2, check out my new "Silverlight Tips, Tricks, Tutorials and Links" page.

Published Wednesday, April 2, 2008 2:01 AM by ScottGu

Comments

# re: Unit Testing with Silverlight

Wednesday, April 2, 2008 5:43 AM by Gill Cleeren

Really valuable post, thanks.

# re: Unit Testing with Silverlight

Wednesday, April 2, 2008 6:40 AM by Granville Barnett

Nice post :-)

Btw, you really should stop talking to yourself on the chat client :-)

# re: Unit Testing with Silverlight

Wednesday, April 2, 2008 6:55 AM by Chris Hardy

Having something that dumps unit testing results to a html page for WebForms/MVC would be great - Rspec does it with Rails and it looks great and I got rather jealous.

# re: Unit Testing with Silverlight

Wednesday, April 2, 2008 7:24 AM by Christian Pfeil

Thanks for the tutorial!

Christian Pfeil

# re: Unit Testing with Silverlight

Wednesday, April 2, 2008 8:01 AM by Håkan Forss

Nice!

Will we see something like this for standard WPF applikations as well? UI automation APIs for WPF apps and WinForm apps would be very usefull for testing.

# re: Unit Testing with Silverlight

Wednesday, April 2, 2008 8:05 AM by daveloper

Would this mean that the unit tests are also send down to the client, since the testing harness runs locally?

# re: Unit Testing with Silverlight

Wednesday, April 2, 2008 8:29 AM by chhaysambo

Good post, thanks.

# re: Unit Testing with Silverlight

Wednesday, April 2, 2008 9:01 AM by Alvin Ashcraft

Thanks Scott. This is some great info.

# re: Unit Testing with Silverlight

Wednesday, April 2, 2008 9:32 AM by nmosafi

Hey Scott

This is great for testing View based components and controls.  However ideally I would love to be able to test my Model/Controller/Utility Classes within Visual Studio / MSTest like I can with code not built on the Core CLR.

TestDriven.NET have a way of doing this, although AFAIK it doesn't work with the latest Silverlight 2 Beta.

It would be great to be able to host CoreCLR assemblies within a CLR process and have the two communicate (a bit like the SL's HTML Bridge).  Do MS have any plans to provide this?

# re: Unit Testing with Silverlight

Wednesday, April 2, 2008 10:20 AM by Gill Cleeren

Great info, thanks.

# re: Unit Testing with Silverlight

Wednesday, April 2, 2008 10:43 AM by Aaron Jensen

How can I do this from my msbuild file? On a continuous integration server? Does it *need* a browser?

# re: Unit Testing with Silverlight

Wednesday, April 2, 2008 12:52 PM by BringerOD

Good to see more example of unit testing.

# re: Unit Testing with Silverlight

Wednesday, April 2, 2008 1:00 PM by jwilcox

'daveloper',

To your question about whether unit tests are also sent down to the client, they are only if you include the tests in the same Silverlight application project.

I'd recommend creating a separate test project (details in the tutorial), your tests are then in one .Xap file - Test.Xap - and your app is in another, App.Xap.

-Jeff

# re: Unit Testing with Silverlight

Wednesday, April 2, 2008 1:28 PM by jwilcox

Håkan Forss,

The UI testing that's being done with this current framework is actually testing the wired up APIs- verifying that events sink, elements are in the visual tree, etc.  At this time there isn't an actual mouse/keyboard automation component there.  

That said, I think many people are interested in what it would take to perform UI automation for WPF and WinForms apps, I'll share your feeback with some folks.  Thanks!

# re: Unit Testing with Silverlight

Wednesday, April 2, 2008 1:36 PM by jwilcox

nmosafi,

That's good feedback about being able to run Core CLR tests from within Visual Studio.

One question I have for you: if you had a way to test "Silverlight" applications right in Visual Studio, you'd only be able to test basic APIs and class libraries: the HTML DOM bridge, the entire rich presentation framework in Silverlight, all those things would -not- be accessible to those tests.  Would that be acceptable?  

In my opinion, I'm more interested in exercising those Silverlight-specific features in the browser, since I can just as easily compile the same source files in my regular desktop CLR and run them as regular Visual Studio test projects to verify that my code is working fine.

- Jeff

# re: Unit Testing with Silverlight

Wednesday, April 2, 2008 2:36 PM by Jacek Ciereszko

Thanks for the tutorial!!!

# re: Unit Testing with Silverlight

Wednesday, April 2, 2008 3:59 PM by daveloper

jwilcox,

Thanks for clearing that up!

TDD with ASP.NET UI components isn't as intuitive compared to logic layers, this really helps out.

# re: Unit Testing with Silverlight

Wednesday, April 2, 2008 7:34 PM by Tom

How do I get the cool background of the editor that you are showing above in an example. Off the topic..sorry :-)

# re: Unit Testing with Silverlight

Wednesday, April 2, 2008 11:59 PM by Annoying Person

Why are you using a Mac? Aren't Microsoft people supposed to hate them? Are you informally admitting Microsoft's inferiority? Or did you just want to demonstrate the cross-platform capabilities of Silverlight?

# re: Unit Testing with Silverlight

Thursday, April 3, 2008 9:40 AM by Ashutosh Bhardwaj

Hi Scott,

Is there any tutorial on the UNIT TEST functionality built within VS 2008? Are you planning to take up any tutorials on the same?

# re: Unit Testing with Silverlight

Thursday, April 3, 2008 2:33 PM by dave.parslow

Hi Scott,

Great job on providing the unit test for Silverlight! I think this will really help the .net community adopt unit testing and TDD as common practices by seeing what Micrsoft themselves do.  When can we expect the unit tests for ASP.NET MVC?  Last I heard you needed to work on the mock framework that ASP.NET MVC was using.

Dave Parslow

# re: Unit Testing with Silverlight

Thursday, April 3, 2008 2:57 PM by Duckie

Helped a lot! Thanks. Back to work.

- There should be a Scott fan-site.

# re: Unit Testing with Silverlight

Friday, April 4, 2008 5:05 AM by ScottGu

Hi Ashutosh,

>>>>>> Is there any tutorial on the UNIT TEST functionality built within VS 2008? Are you planning to take up any tutorials on the same?

Stephen Walther has a great post here that should help: weblogs.asp.net/.../tdd-test-driven-development-with-visual-studio-2008-unit-tests.aspx

Thanks,

Scott

# re: Unit Testing with Silverlight

Friday, April 4, 2008 5:05 AM by ScottGu

Hi Dave,

>>>>> Great job on providing the unit test for Silverlight! I think this will really help the .net community adopt unit testing and TDD as common practices by seeing what Micrsoft themselves do.  When can we expect the unit tests for ASP.NET MVC?  Last I heard you needed to work on the mock framework that ASP.NET MVC was using.

There will be an update to ASP.NET MVC later this month that makes testing controllers much easier, and won't require mocking any objects to-do so.

Thanks,

Scott

# re: Unit Testing with Silverlight

Friday, April 4, 2008 5:06 AM by ScottGu

To Everyone,

Make sure to check out Jeff's answers above - he has jumped in and answered several of the earlier questions on unit testing.

Thanks,

Scott

# re: Unit Testing with Silverlight

Friday, April 4, 2008 7:31 AM by vik20000in

I am really linking Silverlight

# re: Unit Testing with Silverlight

Friday, April 4, 2008 9:25 AM by nmosafi

Hi Jeff

I think a combination of both types of testing would be good.  If tests can be run from MSTest then I can integrate them into my build process.  Those tests which have a dependency on the browser to run would not be able to run via MSTest and would require the browser.

Also a good point that you could compile the same source files as a normal CLR assembly and test them that way.  I might try that!

Neil

# re: Unit Testing with Silverlight

Friday, April 4, 2008 12:45 PM by ScottGu

Hi Tom,

>>>>>>> How do I get the cool background of the editor that you are showing above in an example. Off the topic..sorry :-)

Here is a pointer to my VS settings file if you want the same VS color scheme: www.scottgu.com/.../vssettings.zip

Hope this helps,

Scott

# re: Unit Testing with Silverlight

Monday, April 7, 2008 3:18 PM by Troy Gould

Why only be able to run and see results of unit tests from the browser?  It would be useful to be able run from the command line and generate a results file so that I can include in my CI process!  Please don't assume we are all vendor-locked into Team System for CI either.

# re: Unit Testing with Silverlight

Tuesday, April 8, 2008 1:28 AM by Chris Chandler (developingchris)

Nice post, ready to give it the old tire kick and test drive.

# re: Unit Testing with Silverlight

Friday, April 11, 2008 2:34 PM by Raviraj

Hi,

Great post...

I have one question though...is there a place where i can see sample code for testing something asynchronously...something like testing the play() for a MediaElement...

# re: Unit Testing with Silverlight

Monday, April 14, 2008 5:55 PM by Jeff Wilcox

Raviraj,

I'll post an asynchronous tutorial to my blog soon.  In the meantime, try:

1. Having your test class inherit from SilverlightTest base, inside Microsoft.Silverlight.Testing namespace

2. Marking the test method for asynchronous testing with the [Asynchronous] attribute, in the same namespace

3. Inside your test call the play() method on your MediaElement.  Hook up to an appropriate event for the player, OR use a DispatcherTimer, etc., to verify that everything is OK and make any assertions.  In the method with the verification at the end of testing, call TestComplete().  This is a method that is exposed by the SilverlightTest base class.

Hope this helps.

-Jeff

# re: Unit Testing with Silverlight

Tuesday, April 15, 2008 9:47 AM by Edward Miller

Do you have any estimate on when the UI automation interface will be available?  We need this ASAP to implement browser based tests of Silverlight apps which we didn't build.

# re: Unit Testing with Silverlight

Wednesday, April 16, 2008 6:56 PM by Issam Qadan

Very useful post, along with the comments. On the subject of testing Silverlight, are there any UI Test Harness tool from Microsoft that is publicly available for driving UI test Automation?

# re: Unit Testing with Silverlight

Wednesday, April 30, 2008 10:52 AM by resveratrol

I have to say, coming from adobe flash ui development, this is quite a leap regarding testing.

I am very happy with it so far.

A

# re: Unit Testing with Silverlight

Wednesday, May 21, 2008 10:17 AM by peter

This post is nice!