Tales from the Evil Empire

Bertrand Le Roy's blog

News


Bertrand Le Roy

BoudinFatal's Gamercard

Tales from the Evil Empire - Blogged

Blogs I read

My other stuff

Archives

Writing the tests for FluentPath

(c) Bertrand Le Roy 2003 Writing the tests for FluentPath is a challenge. The library is a wrapper around a legacy API (System.IO) that wasn’t designed to be easily testable.

If it were more testable, the sensible testing methodology would be to tell System.IO to act against a mock file system, which would enable me to verify that my code is doing the expected file system operations without having to manipulate the actual, physical file system: what we are testing here is FluentPath, not System.IO.

Unfortunately, that is not an option as nothing in System.IO enables us to plug a mock file system in. As a consequence, we are left with few options. A few people have suggested me to abstract my calls to System.IO away so that I could tell FluentPath – not System.IO – to use a mock instead of the real thing.

That in turn is getting a little silly: FluentPath already is a thin abstraction around System.IO, so layering another abstraction between them would double the test surface while bringing little or no value. I would have to test that new abstraction layer, and that would bring us back to square one.

Unless I’m missing something, the only option I have here is to bite the bullet and test against the real file system. Of course, the tests that do that can hardly be called unit tests. They are more integration tests as they don’t only test bits of my code. They really test the successful integration of my code with the underlying System.IO.

In order to write such tests, the techniques of BDD work particularly well as they enable you to express scenarios in natural language, from which test code is generated. Integration tests are being better expressed as scenarios orchestrating a few basic behaviors, so this is a nice fit.

The Orchard team has been successfully using SpecFlow for integration tests for a while and I thought it was pretty cool so that’s what I decided to use.

Consider for example the following scenario:

Scenario: Change extension
    Given a clean test directory
    When I change the extension of bar\notes.txt to foo
    Then bar\notes.txt should not exist
    And bar\notes.foo should exist

This is human readable and tells you everything you need to know about what you’re testing, but it is also executable code.

What happens when SpecFlow compiles this scenario is that it executes a bunch of regular expressions that identify the known Given (set-up phases), When (actions) and Then (result assertions) to identify the code to run, which is then translated into calls into the appropriate methods. Nothing magical. Here is the code generated by SpecFlow:

[NUnit.Framework.TestAttribute()]
[NUnit.Framework.DescriptionAttribute("Change extension")]
public virtual void ChangeExtension() {
  TechTalk.SpecFlow.ScenarioInfo scenarioInfo =
new TechTalk.SpecFlow.ScenarioInfo("Change extension",
((string[])(null))); #line 6 this.ScenarioSetup(scenarioInfo); #line 7 testRunner.Given("a clean test directory"); #line 8 testRunner.When("I change the extension of " +
"bar\\notes.txt to foo"
); #line 9 testRunner.Then("bar\\notes.txt should not exist"); #line 10 testRunner.And("bar\\notes.foo should exist"); #line hidden testRunner.CollectScenarioErrors();
}

The #line directives are there to give clues to the debugger, because yes, you can put breakpoints into a scenario:Debugging SpecFlow

The way you usually write tests with SpecFlow is that you write the scenario first, let it fail, then write the translation of your Given, When and Then into code if they don’t already exist, which results in running but failing tests, and then you write the code to make your tests pass (you implement the scenario).

In the case of FluentPath, I built a simple Given method that builds a simple file hierarchy in a temporary directory that all scenarios are going to work with:

[Given("a clean test directory")]
public void GivenACleanDirectory() {
  _path = new Path(SystemIO.Path.GetTempPath())
          .CreateSubDirectory("FluentPathSpecs")
          .MakeCurrent();
  _path.GetFileSystemEntries()
       .Delete(true);
  _path.CreateFile("foo.txt",
"This is a text file named foo."); var bar = _path.CreateSubDirectory("bar"); bar.CreateFile("baz.txt", "bar baz") .SetLastWriteTime(DateTime.Now.AddSeconds(-2)); bar.CreateFile("notes.txt",
"This is a text file containing notes."); var barbar = bar.CreateSubDirectory("bar"); barbar.CreateFile("deep.txt", "Deep thoughts"); var sub = _path.CreateSubDirectory("sub"); sub.CreateSubDirectory("subsub"); sub.CreateFile("baz.txt", "sub baz") .SetLastWriteTime(DateTime.Now); sub.CreateFile("binary.bin", new byte[] {0x00, 0x01, 0x02, 0x03,
0x04, 0x05, 0xFF}); }

Then, to implement the scenario that you can read above, I had to write the following When:

[When("I change the extension of (.*) to (.*)")]
public void WhenIChangeTheExtension(
string path, string newExtension) { var oldPath = Path.Current.Combine(path.Split('\\')); oldPath.Move(p => p.ChangeExtension(newExtension)); }

As you can see, the When attribute is specifying the regular expression that will enable the SpecFlow engine to recognize what When method to call and also how to map its parameters. For our scenario, “bar\notes.txt” will get mapped to the path parameter, and “foo” to the newExtension parameter.

And of course, the code that verifies the assumptions of the scenario:

[Then("(.*) should exist")]
public void ThenEntryShouldExist(string path) {
  Assert.IsTrue(_path.Combine(path.Split('\\')).Exists);
}

[Then("(.*) should not exist")]
public void ThenEntryShouldNotExist(string path) {
  Assert.IsFalse(_path.Combine(path.Split('\\')).Exists);
}

These steps should be written with reusability in mind. They are building blocks for your scenarios, not implementation of a specific scenario. Think small and fine-grained. In the case of the above steps, I could reuse each of those steps in other scenarios.

Those tests are easy to write and easier to read, which means that they also constitute a form of documentation.

Oh, and SpecFlow is just one way to do this. Rob wrote a long time ago about this sort of thing (but using a different framework) and I highly recommend this post if I somehow managed to pique your interest:
http://blog.wekeroad.com/blog/make-bdd-your-bff-2/

And this screencast (Rob always makes excellent screencasts):
http://blog.wekeroad.com/mvc-storefront/kona-3/
(click the “Download it here” link)

Finally, Rob (him again) tells me he did a free TekPub screencast on Specflow:
http://tekpub.com/view/concepts/5

Comments

Ludovic said:

Maybe I'm less "pure" in terms of TTD-ness but I don't see a problem with testing against System.IO. It's part of the framework, and I don't see people trying to abstract other framework things like collections or streams. Sure, you may want to abstract bigger framework APIs, but I guess it's a matter of personal taste whether you put System.IO in the "big stuff" or the "small stuff". And since I don't really think the side effects of the tests (manipulating the file system) are that bad, I'd consider it to be "small stuff" and I wouldn't lose sleep over it.

Now it's interesting that if you follow my opinion here, then unit tests and integration tests end up being exactly the same... which basically means that your library is unitary to begin with. And I think it's the case, so it's all good to me.

# May 29, 2010 3:33 AM

Bertrand Le Roy said:

@Ludovic: yeah, I don't want to split hair. The FluentPath integration tests run fast enough that testing against the actual file system is not a problem, but if I had a bigger system with hundreds of such tests, or if I was testing against a slower, non-mockable system (let's say something remote for example), I'd be in more trouble. All this to say that my silly little library may not expose the badness of the methodology, but more serious stuff will for sure.

# May 29, 2010 3:45 AM

DBJ said:

Testing System.IO ? RamDisk perhaps?

I reallly would like to see someone who is "inside" ;o) publishing the real life usage of ramdisk in W7 ?

Another approach perahsp would be a virtualPC on top of 64bit OS , ontop huge RAM ammount ...

--DBJ

# May 29, 2010 5:59 AM

Simon said:

Have you tried to use Moles to mock System.IO ? I never used it, but I heard it should make the job (the samples show how to mock things like DateTime.Now) for example.

# May 29, 2010 8:18 AM

Horses said:

Is there a way to stub/mock FluentPath?  Or is the technique you've outlined here the best way to write tests for code that uses FluentPath?

# May 31, 2010 12:27 AM

Bertrand Le Roy said:

@Horses: ha! Good question. I suppose as it is, it's just as testable as System.IO itself. Adding some interfaces in there would definitely help. I'll try to add them eventually.

# May 31, 2010 7:43 PM

Bertrand Le Roy said:

@Simon: Moles look interesting, thanks for the pointer, but from the manual it can't mock static methods, which is pretty much all  that we use in System.IO. It's not clear why static methods are out of the picture though: the way they do things (by rewriting the code), I don't see why you couldn't re-route static calls.

# May 31, 2010 8:02 PM

Bertrand Le Roy said:

Ah, never mind, static methods seem like they work too. Trying it.

# May 31, 2010 8:03 PM

James said:

We went with extracting operations exposed by System.IO.File and System.IO.Directory onto an interface.

All new code that accesses the file system goes through an implementation of this interface.

It's nice for speeding up tests of things which are file-system heavy (hundreds of tests at last count), as our test stub implementation is completely memory backed.

Another advantage is that you don't have to do cleanup, just let the stub be GC'.

And for integration tests, it can be useful to create strict mocks of this interface to trap unexpected file system access.

# June 1, 2010 6:39 AM

Bertrand Le Roy said:

@James: yes, and hopefully one day FluentPath can play this role as well as improve the experience of working with the file system in .NET.

# June 1, 2010 1:57 PM