Tales from the Evil Empire

Bertrand Le Roy's blog

News


Bertrand Le Roy

BoudinFatal's Gamercard

Tales from the Evil Empire - Blogged

Blogs I read

My other stuff

Archives

May 2010 - Posts

Writing the tests for FluentPath

(c) Bertrand Le Roy 2003 Writing the tests for FluentPath is a challenge. The library is a wrapper around a legacy API (System.IO) that wasn’t designed to be easily testable.

If it were more testable, the sensible testing methodology would be to tell System.IO to act against a mock file system, which would enable me to verify that my code is doing the expected file system operations without having to manipulate the actual, physical file system: what we are testing here is FluentPath, not System.IO.

Unfortunately, that is not an option as nothing in System.IO enables us to plug a mock file system in. As a consequence, we are left with few options. A few people have suggested me to abstract my calls to System.IO away so that I could tell FluentPath – not System.IO – to use a mock instead of the real thing.

That in turn is getting a little silly: FluentPath already is a thin abstraction around System.IO, so layering another abstraction between them would double the test surface while bringing little or no value. I would have to test that new abstraction layer, and that would bring us back to square one.

Unless I’m missing something, the only option I have here is to bite the bullet and test against the real file system. Of course, the tests that do that can hardly be called unit tests. They are more integration tests as they don’t only test bits of my code. They really test the successful integration of my code with the underlying System.IO.

In order to write such tests, the techniques of BDD work particularly well as they enable you to express scenarios in natural language, from which test code is generated. Integration tests are being better expressed as scenarios orchestrating a few basic behaviors, so this is a nice fit.

The Orchard team has been successfully using SpecFlow for integration tests for a while and I thought it was pretty cool so that’s what I decided to use.

Consider for example the following scenario:

Scenario: Change extension
    Given a clean test directory
    When I change the extension of bar\notes.txt to foo
    Then bar\notes.txt should not exist
    And bar\notes.foo should exist

This is human readable and tells you everything you need to know about what you’re testing, but it is also executable code.

What happens when SpecFlow compiles this scenario is that it executes a bunch of regular expressions that identify the known Given (set-up phases), When (actions) and Then (result assertions) to identify the code to run, which is then translated into calls into the appropriate methods. Nothing magical. Here is the code generated by SpecFlow:

[NUnit.Framework.TestAttribute()]
[NUnit.Framework.DescriptionAttribute("Change extension")]
public virtual void ChangeExtension() {
  TechTalk.SpecFlow.ScenarioInfo scenarioInfo =
new TechTalk.SpecFlow.ScenarioInfo("Change extension",
((string[])(null))); #line 6 this.ScenarioSetup(scenarioInfo); #line 7 testRunner.Given("a clean test directory"); #line 8 testRunner.When("I change the extension of " +
"bar\\notes.txt to foo"
); #line 9 testRunner.Then("bar\\notes.txt should not exist"); #line 10 testRunner.And("bar\\notes.foo should exist"); #line hidden testRunner.CollectScenarioErrors();
}

The #line directives are there to give clues to the debugger, because yes, you can put breakpoints into a scenario:Debugging SpecFlow

The way you usually write tests with SpecFlow is that you write the scenario first, let it fail, then write the translation of your Given, When and Then into code if they don’t already exist, which results in running but failing tests, and then you write the code to make your tests pass (you implement the scenario).

In the case of FluentPath, I built a simple Given method that builds a simple file hierarchy in a temporary directory that all scenarios are going to work with:

[Given("a clean test directory")]
public void GivenACleanDirectory() {
  _path = new Path(SystemIO.Path.GetTempPath())
          .CreateSubDirectory("FluentPathSpecs")
          .MakeCurrent();
  _path.GetFileSystemEntries()
       .Delete(true);
  _path.CreateFile("foo.txt",
"This is a text file named foo."); var bar = _path.CreateSubDirectory("bar"); bar.CreateFile("baz.txt", "bar baz") .SetLastWriteTime(DateTime.Now.AddSeconds(-2)); bar.CreateFile("notes.txt",
"This is a text file containing notes."); var barbar = bar.CreateSubDirectory("bar"); barbar.CreateFile("deep.txt", "Deep thoughts"); var sub = _path.CreateSubDirectory("sub"); sub.CreateSubDirectory("subsub"); sub.CreateFile("baz.txt", "sub baz") .SetLastWriteTime(DateTime.Now); sub.CreateFile("binary.bin", new byte[] {0x00, 0x01, 0x02, 0x03,
0x04, 0x05, 0xFF}); }

Then, to implement the scenario that you can read above, I had to write the following When:

[When("I change the extension of (.*) to (.*)")]
public void WhenIChangeTheExtension(
string path, string newExtension) { var oldPath = Path.Current.Combine(path.Split('\\')); oldPath.Move(p => p.ChangeExtension(newExtension)); }

As you can see, the When attribute is specifying the regular expression that will enable the SpecFlow engine to recognize what When method to call and also how to map its parameters. For our scenario, “bar\notes.txt” will get mapped to the path parameter, and “foo” to the newExtension parameter.

And of course, the code that verifies the assumptions of the scenario:

[Then("(.*) should exist")]
public void ThenEntryShouldExist(string path) {
  Assert.IsTrue(_path.Combine(path.Split('\\')).Exists);
}

[Then("(.*) should not exist")]
public void ThenEntryShouldNotExist(string path) {
  Assert.IsFalse(_path.Combine(path.Split('\\')).Exists);
}

These steps should be written with reusability in mind. They are building blocks for your scenarios, not implementation of a specific scenario. Think small and fine-grained. In the case of the above steps, I could reuse each of those steps in other scenarios.

Those tests are easy to write and easier to read, which means that they also constitute a form of documentation.

Oh, and SpecFlow is just one way to do this. Rob wrote a long time ago about this sort of thing (but using a different framework) and I highly recommend this post if I somehow managed to pique your interest:
http://blog.wekeroad.com/blog/make-bdd-your-bff-2/

And this screencast (Rob always makes excellent screencasts):
http://blog.wekeroad.com/mvc-storefront/kona-3/
(click the “Download it here” link)

Finally, Rob (him again) tells me he did a free TekPub screencast on Specflow:
http://tekpub.com/view/concepts/5

JavaScript local alias pattern

(c) Bertrand Le Roy 2005 Here’s a little pattern that is fairly common from JavaScript developers but that is not very well known from C# developers or people doing only occasional JavaScript development.

In C#, you can use a “using” directive to create aliases of namespaces or bring them to the global scope:

namespace Fluent.IO {
    using System;
    using System.Collections;
    using SystemIO = System.IO;

In JavaScript, the only scoping construct there is is the function, but it can also be used as a local aliasing device, just like the above using directive:

(function($, dv) {
    $("#foo").doSomething();
    var a = new dv("#bar");
})(jQuery, Sys.UI.DataView);

This piece of code is making the jQuery object accessible using the $ alias throughout the code that lives inside of the function, without polluting the global scope with another variable.

The benefit is even bigger for the dv alias which stands here for Sys.UI.DataView: think of the reduction in file size if you use that one a lot or about how much less you’ll have to type…

I’ve taken the habit of putting almost all of my code, even page-specific code, inside one of those closures, not just because it keeps the global scope clean but mostly because of that handy aliasing capability.

The fastest way to resize images from ASP.NET. And it’s (more) supported-ish.

Thumb I’ve shown before how to resize images using GDI, which is fairly common but is explicitly unsupported because we know of very real problems that this can cause. Still, many sites still use that method because those problems are fairly rare, and because most people assume it’s the only way to get the job done. Plus, it works in medium trust.

More recently, I’ve shown how you can use WPF APIs to do the same thing and get JPEG thumbnails, only 2.5 times faster than GDI (even now that GDI really ultimately uses WIC to read and write images). The boost in performance is great, but it comes at a cost, that you may or may not care about: it won’t work in medium trust. It’s also just as unsupported as the GDI option.

What I want to show today is how to use the Windows Imaging Components from ASP.NET APIs directly, without going through WPF.

The approach has the great advantage that it’s been tested and proven to scale very well. The WIC team tells me you should be able to call support and get answers if you hit problems.

Caveats exist though.

First, this is using interop, so until a signed wrapper sits in the GAC, it will require full trust.

Second, the APIs have a very strong smell of native code and are definitely not .NET-friendly.

And finally, the most serious problem is that older versions of Windows don’t offer MTA support for image decoding. MTA support is only available on Windows 7, Vista and Windows Server 2008. But on 2003 and XP, you’ll only get STA support. that means that the thread safety that we so badly need for server applications is not guaranteed on those operating systems. To make it work, you’d have to spin specialized threads yourself and manage the lifetime of your objects, which is outside the scope of this article.

We’ll assume that we’re fine with al this and that we’re running on 7 or 2008 under full trust.

Be warned that the code that follows is not simple or very readable. This is definitely not the easiest way to resize an image in .NET.

Wrapping native APIs such as WIC in a managed wrapper is never easy, but fortunately we won’t have to: the WIC team already did it for us and released the results under MS-PL. The InteropServices folder, which contains the wrappers we need, is in the WicCop project but I’ve also included it in the sample that you can download from the link at the end of the article.

In order to produce a thumbnail, we first have to obtain a decoding frame object that WIC can use. Like with WPF, that object will contain the command to decode a frame from the source image but won’t do the actual decoding until necessary.

Getting the frame is done by reading the image bytes through a special WIC stream that you can obtain from a factory object that we’re going to reuse for lots of other tasks:

var photo = File.ReadAllBytes(photoPath);
var factory =
(IWICComponentFactory)new WICImagingFactory(); var inputStream = factory.CreateStream(); inputStream.InitializeFromMemory(photo,
(uint)photo.Length); var decoder = factory.CreateDecoderFromStream(
inputStream, null,
WICDecodeOptions.WICDecodeMetadataCacheOnLoad); var frame = decoder.GetFrame(0);

We can read the dimensions of the frame using the following (somewhat ugly) code:

uint width, height;
frame.GetSize(out width, out height);

This enables us to compute the dimensions of the thumbnail, as I’ve shown in previous articles.

We now need to prepare the output stream for the thumbnail. WIC requires a special kind of stream, IStream (not implemented by System.IO.Stream) and doesn’t directlyunderstand .NET streams. It does provide a number of implementations but not exactly what we need here.

We need to output to memory because we’ll want to persist the same bytes to the response stream and to a local file for caching. The memory-bound version of IStream requires a fixed-length buffer but we won’t know the length of the buffer before we resize.

To solve that problem, I’ve built a derived class from MemoryStream that also implements IStream. The implementation is not very complicated, it just delegates the IStream methods to the base class, but it involves some native pointer manipulation.

Once we have a stream, we need to build the encoder for the output format, which could be anything that WIC supports. For web thumbnails, our only reasonable options are PNG and JPEG.

I explored PNG because it’s a lossless format, and because WIC does support PNG compression. That compression is not very efficient though and JPEG offers good quality with much smaller file sizes. On the web, it matters. I found the best PNG compression option (adaptive) to give files that are about twice as big as 100%-quality JPEG (an absurd setting), 4.5 times bigger than 95%-quality JPEG and 7 times larger than 85%-quality JPEG, which is more than acceptable quality.

As a consequence, we’ll use JPEG. The JPEG encoder can be prepared as follows:

var encoder = factory.CreateEncoder(
Consts.GUID_ContainerFormatJpeg, null); encoder.Initialize(outputStream,
WICBitmapEncoderCacheOption.WICBitmapEncoderNoCache);

The next operation is to create the output frame:

IWICBitmapFrameEncode outputFrame;
var arg = new IPropertyBag2[1];
encoder.CreateNewFrame(out outputFrame, arg);

Notice that we are passing in a property bag. This is where we’re going to specify our only parameter for encoding, the JPEG quality setting:

var propBag = arg[0];
var propertyBagOption = new PROPBAG2[1];
propertyBagOption[0].pstrName = "ImageQuality";
propBag.Write(1, propertyBagOption,
new object[] { 0.85F }); outputFrame.Initialize(propBag);

We can then set the resolution for the thumbnail to be 96, something we weren’t able to do with WPF and had to hack around:

outputFrame.SetResolution(96, 96);

Next, we set the size of the output frame and create a scaler from the input frame and the computed dimensions of the target thumbnail:

outputFrame.SetSize(thumbWidth, thumbHeight);
var scaler = factory.CreateBitmapScaler();
scaler.Initialize(frame, thumbWidth, thumbHeight,
WICBitmapInterpolationMode.WICBitmapInterpolationModeFant);

The scaler is using the Fant method, which I think is the best looking one even if it seems a little softer than cubic (zoomed here to better show the defects):

Cubic
Cubic
Fant
Fant
Linear
Linear
Nearest neighbour
Nearest neighbor

We can write the source image to the output frame through the scaler:

outputFrame.WriteSource(scaler, new WICRect {
X = 0, Y = 0,
Width = (int)thumbWidth,
Height = (int)thumbHeight });

And finally we commit the pipeline that we built and get the byte array for the thumbnail out of our memory stream:

outputFrame.Commit();
encoder.Commit();
var outputArray = outputStream.ToArray();
outputStream.Close();

That byte array can then be sent to the output stream and to the cache file.

Once we’ve gone through this exercise, it’s only natural to wonder whether it was worth the trouble. I ran this method, as well as GDI and WPF resizing over thirty twelve megapixel images for JPEG qualities between 70% and 100% and measured the file size and time to resize. Here are the results:

Size of resized images
Size of resized images
 Time to resize
Time to resize thirty 12 megapixel images

Not much to see on the size graph: sizes from WPF and WIC are equivalent, which is hardly surprising as WPF calls into WIC. There is just an anomaly for 75% for WPF that I noted in my previous article and that disappears when using WIC directly.

But overall, using WPF or WIC over GDI represents a slight win in file size.

The time to resize is more interesting. WPF and WIC get similar times although WIC seems to always be a little faster. Not surprising considering WPF is using WIC. The margin of error on this results is probably fairly close to the time difference. As we already knew, the time to resize does not depend on the quality level, only the size does. This means that the only decision you have to make here is size versus visual quality.

This third approach to server-side image resizing on ASP.NET seems to converge on the fastest possible one. We have marginally better performance than WPF, but with some additional peace of mind that this approach is sanctioned for server-side usage by the Windows Imaging team.

It still doesn’t work in medium trust. That is a problem and shows the way for future server-friendly managed wrappers around WIC.

The sample code for this article can be downloaded from:
http://weblogs.asp.net/blogs/bleroy/Samples/WicResize.zip

The benchmark code can be found here (you’ll need to add your own images to the Images directory and then add those to the project, with content and copy if newer in the properties of the files in the solution explorer):
http://weblogs.asp.net/blogs/bleroy/Samples/WicWpfGdiImageResizeBenchmark.zip

WIC tools can be downloaded from:
http://code.msdn.microsoft.com/wictools

To conclude, here are some of the resized thumbnails at 85% fant:

IMG_2734_85 IMG_2744_85 IMG_2228_85
IMG_2235_85 IMG_2300_85 IMG_2305_85
IMG_2311_85 IMG_2317_85 IMG_2318_85
IMG_2325_85 IMG_2330_85 IMG_2332_85
IMG_2346_85 IMG_2351_85 IMG_2363_85
IMG_2398_85 IMG_2443_85 IMG_2445_85
IMG_2446_85 IMG_2452_85 IMG_2462_85
IMG_2504_85 IMG_2505_85 IMG_2525_85
More Posts